Most quantitative traders and developers rely heavily on candlestick and transaction data when running historical market replays. These basic data types are easy to obtain and seem reliable enough for daily strategy verification. However, once you upgrade your research to order-book level granularity, simple price charts can no longer reflect the real market microstructure at any given moment.
I’ve encountered this frustrating problem countless times during my development of market replay systems. A trading strategy may produce extremely stable and profitable results on historical backtests, yet behave completely differently once deployed in a live trading environment. After deep troubleshooting, I found that the issue rarely lies in the strategy logic itself. In most cases, the backtesting process lacks genuine order-book depth data, which is essential for restoring real market conditions.
To put it simply, price data only shows the final result of market movement. In contrast, order-book data records the complete state of all pending buy and sell orders at every timestamp. By combining historical depth snapshots with incremental order updates, we can accurately reconstruct past order-book status and make backtest replays far more consistent with real-market behavior.
Data Pain Points: Why Basic Market Data Causes Backtest Deviations
Standard trade and candlestick data can only tell us what price changes have happened in the market. Unfortunately, they cannot explain why those movements took place.
For example, if a stock suddenly spikes in price, transaction records will clearly show the upward movement and increased trading volume. But these records cannot tell us whether buying liquidity was accumulating before the breakout, whether sell-side pending orders were thinning out, or how overall market liquidity changed at that moment. These subtle yet critical details determine the accuracy of liquidity analysis and order-level strategy testing.
In my daily quantitative work, I categorize market data into four core types, each serving a completely different purpose in backtesting and market reconstruction.
First, candlestick data focuses on recording periodic price fluctuations, including open, high, low, and close prices within fixed time intervals, ideal for macroscopic trend analysis. Second, tick data tracks every individual transaction executed on the market, offering precise records of real completed trades. Third, order-book snapshots capture the full depth status of all buy and sell pending orders at a single exact moment, which is the key to restoring market microstructure. Fourth, incremental update data continuously logs all real-time order changes, including new order placement, order cancellation, and volume adjustment events.
Accurate order-book restoration depends on integrating all four types of data instead of relying on price or transaction records alone.
Efficiency Problems: The Limitations of Traditional Replay Logic
Many traders and developers still use naive replay methods that simply stitch historical price data together. Although this method is fast and convenient, it brings obvious accuracy flaws.
The biggest issue is that static price data cannot simulate dynamic order changes. Real markets are filled with continuous order hanging, cancellation, and volume adjustment behaviors that directly affect actual order execution rates. Traditional replay methods ignore these dynamic details, resulting in overly optimistic backtest results that cannot match live trading performance.
In addition, transaction-only data cannot reflect continuous liquidity changes over time. As your backtest duration increases, the simulated market status will gradually deviate from the real market, making strategy optimization and parameter tuning ineffective.
Core Functionality: Rebuilding Order Books with Snapshot and Incremental Updates
The most reliable and industry-standard solution I use for historical order-book recovery is the combination of fixed snapshots and real-time incremental updates.
A market snapshot acts as the complete initial baseline state of the order book. For instance, a full snapshot recorded at 10:00:00 can fully display all active pending orders on both sides of the market. At this exact timestamp, the buy side may have 500 shares pending at the price of 100.01 and 800 shares pending at 100.00, while the sell side may have 600 shares at 100.02 and 900 shares at 100.03. This single snapshot completely restores the market’s depth structure at that moment.
After the baseline snapshot is generated, all subsequent market changes are updated through incremental data streams. One second later, the market may show reduced buy order volume at 100.01 and newly added sell orders at the 100.04 price level. The program updates the in-memory order book immediately based on these delta changes.
In short, snapshots provide a fixed starting point for market status, while incremental data records every subsequent market evolution. To restore the order book at any target historical time, you only need to locate the latest snapshot before your target timestamp, then sequentially overlay all incremental update events until you reach the target time point. This method perfectly reproduces real historical market depth.
Key Development Details to Avoid Reconstruction Errors
Order-book replay is far more than simple data merging. Several easily overlooked details will seriously affect reconstruction accuracy.
First, unified timestamp processing is essential. Cross-border market data involves multiple time formats and time zone standards. Without standardized time conversion, the sequence of market events will be disordered, completely ruining the order-book reconstruction result.
Second, you need to ensure complete market depth coverage. Many basic API services only provide top-level bid and ask data. If your strategy relies on multi-level depth analysis, shallow data will lead to incomplete restoration and inaccurate backtesting.
Third, incremental market events include more than completed trades. New order submissions, order withdrawals, and volume modifications are all core market changes. Ignoring any type of event will cause cumulative errors and make the restored order book increasingly inaccurate over time.
Live Market Access and Data Recording Practice
To build a complete and accurate historical replay system, I always persist real-time tick data for future backtesting. I use AllTick API’s stable WebSocket streaming service to obtain real-time market tick data and store all events in strict chronological order.
In production environments, I store order-book snapshots and incremental update logs separately. During replay tasks, the system quickly matches the baseline snapshot according to the target time and replays all subsequent changes to restore the complete historical order-book state.
How This Workflow Changes My Quantitative Research
After years of processing cross-border market API data, I’ve formed a clear conclusion: price data only reflects market results, while order-book depth data reveals the entire market evolution process.
For basic trend analysis and simple market research, candlestick data is sufficient. But for fine-grained quantitative strategy development, high-frequency trading logic, and liquidity-based strategy verification, complete order-book reconstruction is indispensable.
By matching baseline snapshots with incremental updates, we effectively eliminate data missing bias in traditional backtesting. This method greatly narrows the gap between historical simulation and real trading performance. For every developer building quantitative replay systems, mastering standardized market state storage and restoration is one of the most fundamental and critical capabilities for stable strategy iteration.
คำชี้แจง (Disclaimer) : เนื้อหาข้างต้นเป็นเพียงมุมมองของผู้เขียนแต่เพียงผู้เดียว และไม่ได้แสดงหรือสะท้อนถึงจุดยืนอย่างเป็นทางการของ Followme แต่อย่างใด Followme ไม่รับผิดชอบต่อความถูกต้อง ความครบถ้วน หรือความน่าเชื่อถือของข้อมูลที่ปรากฏ และจะไม่รับผิดชอบต่อการดำเนินการใด ๆ ที่เกิดขึ้นจากเนื้อหานั้น เว้นแต่จะมีการระบุไว้เป็นลายลักษณ์อักษรอย่างชัดเจน
