Ultra Low-Latency Block Synchronization at High TPS: Consensus Observer (AIP-93)

By: Joshua Lind and Zekun Li

TL;DR: Consensus Observer (CO), a novel data dissemination technique that reduces block synchronization time and transaction latencies, is now live on Aptos Mainnet. Unlike traditional state synchronization, which delays block dissemination until after execution and commitment, CO enables immediate block propagation throughout the network. This allows fullnodes to receive block data earlier and process blocks in parallel with validators. CO reduces end-to-end transaction latencies by: (i) ~10% at low load (e.g., 500 TPS); (ii) ~30% at medium load (e.g., 5k TPS); and (iii) ~50%+ at higher loads (e.g., 10k TPS+).

This brings Aptos one step closer to powering a real-time digital economy with ultra low-latency use-cases, such as micropayments, streaming and autonomous payments.

The Problem With State Sync

State synchronization is a fundamental challenge for blockchain networks. While consensus algorithms achieve agreement amongst validators, they do not solve the state synchronization problem for other peers in the network (e.g., fullnodes). Instead, additional protocols (called state sync protocols) are required to distribute, verify and persist blockchain data across non-validator peers.

State sync protocols must be efficient, reliable and performant. This is because they play a significant role in the lifecycle of a blockchain transaction, directly impacting transaction throughput and latency. If state sync is slow or unreliable, network peers (e.g., fullnodes) will perceive long transaction processing delays, artificially inflating the time to transaction finality and negatively impacting the user experience. For more background on state sync, see: The Evolution of State Sync.

Figure 1 shows the Aptos network topology with: (i) Validators executing consensus; (ii) Validator fullnodes (VFNs) synchronizing from validators; and (iii) Public fullnodes (PFNs) synchronizing from VFNs.

Figure 1: The Aptos network topology showing validators, validator fullnodes (VFNs) and public fullnodes (PFNs).

Traditionally, state sync waits for validators to execute and commit blocks before disseminating block data to VFNs and PFNs. This approach, while easy to reason about, introduces delays as it prevents peers from receiving and processing blocks as soon as possible. This is particularly troublesome for Aptos, which achieves sub-second transaction latencies. Any amount of added time waiting for execution and commitment (e.g., 100’s of milliseconds at high load), can cause: (i) fullnodes to lag behind validators; (ii) end-to-end transaction latencies to spike; and (iii) compute and network resources to be wasted, due to idle time.

Introducing Consensus Observer

To address the limitations of traditional state sync, Aptos Labs has proposed Consensus Observer (CO): a novel data dissemination technique that decouples block propagation from execution and commitment. CO allows validators to immediately disseminate block data to other network peers, without having to wait for local processing. This significantly reduces synchronization time across the network, and results in: (i) lower end-to-end transaction latencies, as peers can process blocks earlier; (ii) improved efficiency, as compute and network resources experience less idle time; and (iii) more predictable synchronization times, as block dissemination has fewer operational dependencies.

Consensus Observer is the answer to the question: What is the fastest way to get fullnodes access to the block data, so that they can process in parallel with the validators?

How Does Consensus Observer Work?

CO decouples block propagation from execution and commitment, allowing network peers (e.g., VFNs and PFNs) to observe the validator block processing pipeline, and process block data as soon as it is produced. This is in contrast to traditional state sync, which requires blocks to pass through the entire pipeline before being disseminated.

Aptos validators utilize a multi-stage, block processing pipeline, consisting of three key stages (see Figure 2, below):

1. Order Stage — Transactions are gathered into blocks and ordered by consensus.
2. Execute Stage — Ordered blocks are executed locally by validators.
3. Commit Stage — Executed blocks are finalized and committed to the blockchain.

Both the order and commit stages require consensus agreement to ensure a single, consistent blockchain history (i.e., to avoid blockchain forks).

Figure 2: The multi-stage, decoupled block processing pipeline, showing a single block passing through the pipeline.

CO enhances synchronization by allowing fullnodes (i.e., VFNs and PFNs) to track and process each stage of this pipeline in parallel with validators, rather than waiting for full block commitment. The process works as follows:

1. Block dissemination: When a block is proposed and ordered by validators, it is immediately shared with fullnode peers, e.g., the ordered block is sent directly to connected VFNs.
2. Parallel processing: Next, the validators and the fullnodes process the block, simultaneously. This includes verifying the block, executing transactions, and updating state.
3. Commit confirmation: Once the block is committed by the validators, the commit confirmation is shared with fullnodes, just as the ordered block was in step 1. Fullnodes then commit the data locally, ensuring synchronization across all nodes.
4. Continuous synchronization: This cycle repeats for each new block, maintaining real-time alignment between validators and fullnodes in the network.

More details about this process can be found in the AIP, including security considerations and techniques used by fullnodes to protect against malicious validators and invalid block data.

The Benefits Of Consensus Observer

By considering the multi-stage block execution pipeline, it becomes easy to compare traditional state sync with CO. For example, in the case of traditional VFN state sync, you will see that the VFN must wait for the following steps to occur before it can begin processing an ordered block: (i) validator block execution; (ii) validator block commitment; and (iii) block propagation from the validator to the VFN. This can be seen in Figure 3, below.

Figure 3: Traditional state sync requires blocks to be fully processed by validators before block data is disseminated to VFNs.

When you compare this with CO, where block data is disseminated to VFNs immediately after ordering, the improvements become evident: validator execution and commit latencies are removed from the equation. Thus, VFNs only need to wait for the block to be propagated via the network before they can begin processing the block. This can be seen in Figure 4, below.

Figure 4: Consensus Observer (CO) disseminating block data from a validator to a VFN before locally executing and committing the block.

When comparing synchronization times (or synchronization lag) between validators and VFNs for a single ordered block, CO not only reduces lag for VFNs, but also ensures more predictable behavior. This is because CO decouples block dissemination from block processing. With CO, VFNs typically lag behind validators by only a single network hop (e.g., ~50ms at low load in the Aptos Mainnet), a significant reduction compared to traditional state sync, where execution and commitment can add 100’s of milliseconds of lag at high load.

Moreover, as network load (e.g., transactions per second) increases, CO is able to maintain a predictable synchronization lag in VFNs, whereas traditional state sync experiences increasing delays and variance due to higher processing times.

Finally, the benefits of CO compound for PFN synchronization lag. For example, with traditional state sync, PFNs synchronize from VFNs, and are thus forced to wait for both the validator and VFN to fully process the block. With CO, however, VFNs disseminate the block data to PFNs before processing, meaning that PFNs only lag behind VFNs by a single network hop (as above). This effectively eliminates the compounding delays that occur in traditional state sync.

Low-Latency Use Cases

By significantly reducing synchronization lag and end-to-end transaction latencies, CO takes us one step closer to enabling unique use-cases that require ultra low-latency and low-cost transactions. For example: (i) micropayments, such as per-second billing for media or content access; (ii) streaming payments, where funds flow continuously between parties (e.g., pay-as-you-go subscriptions); and (iii) autonomous payments, where IoT devices or decentralized services initiate payments based on real-world events.

All of these require fast, predictable transaction processing, real-time data propagation, and minimal synchronization delays to operate effectively. With CO, Aptos is better positioned to support these next-generation use-cases — bringing us closer to a truly scalable, real-time digital economy.

Evaluating Consensus Observer

To evaluate CO before Mainnet deployment, we performed a variety of experiments in different environments, and with different workloads. These included performance tests, stress tests, and chaos tests. Examples of the performance tests, and results, are outlined below:

Realistic environment simulations: We simulated a “realistic” network environment, with 30 validators and 30 VFNs. Simulation included CPU and network chaos that emulated real-world heterogeneity and latencies, as seen in the Aptos Mainnet. We executed a coin transfer workload and varied the transactions per second (TPS).

• The results showed that CO reduced average end-to-end transaction latencies by ~10% at low load (i.e., 500 TPS), and ~30% at higher load (i.e., 10k TPS). For the higher load experiment, latencies were ~2.1 seconds (traditional state sync) vs. ~1.5 seconds (CO), respectively.

Chaotic environment simulations: We simulated a “chaotic” network environment, with 30 validators, 30 VFNs, and 30 PFNs. Simulation included extreme CPU and network chaos, 2–3x the values typically seen in the Aptos Mainnet. We executed a coin transfer workload and varied the TPS.

• The results showed that CO reduced average end-to-end transaction latencies by ~20% at low load (i.e., 500 TPS), and ~50% at higher load (i.e., 5k TPS). For the higher load experiment, latencies were ~3.1 seconds (traditional state sync) vs. ~1.7 seconds (CO), respectively.

Having verified the correctness of CO in multiple experiments, it was gradually rolled out to the Aptos Mainnet, i.e., across all validators, VFNs and Aptos Labs operated PFNs. After roll out, we made the following observations:

• 10% reduction at low load: Since its deployment, CO has delivered a ~10% reduction in end-to-end transaction latency at low network load (~60 TPS). As shown in Figure 5, the impact was immediate: when CO was enabled in the Europe cluster on 11/28/2024, latencies dropped by ~100ms across P10, P50, and P90, with P50 improving from ~800ms to ~700ms. Similar results were observed across all Aptos Labs PFNs in the US, Asia, and Europe, underscoring CO’s consistent performance benefits globally.

Figure 5: End-to-end transaction latencies of Aptos Labs PFNs before and after the CO rollout

• 50% reduction at higher loads: Since its deployment, CO has delivered a ~50% reduction in end-to-end transaction latency at higher network loads (~8.5k+ TPS). For example, Figure 6 shows a comparison of Mainnet transaction latencies between traditional state sync and CO at higher loads. Before CO was deployed to Mainnet, transaction latencies were 2.80s (P50), 3.45s (P75), and 4.94s (P90) at ~8.5k TPS. After CO was deployed to Mainnet, transaction latencies were reduced to 1.46s (P50), 1.98s (P75), and 3.20s (P90) at ~10.9k TPS. This shows a reduction of ~50% at P50, ~40% at P75, and ~35% at P90.

Figure 6: Comparison of Mainnet transaction latencies between traditional state sync and consensus observer, across percentiles (i.e., P50, P75 and P90).

Trade-offs With Traditional Syncing

One key advantage of traditional state sync is the ability to reduce resource usage by allowing fullnodes to skip transaction execution and instead reuse execution results from validators, by way of proofs (this is called intelligent syncing). This reduces CPU usage, improves synchronization efficiency, and reduces the overall cost of operating a fullnode.

However, CO requires fullnodes to execute blocks in real-time, meaning that they must maintain hardware resources equivalent to validators, otherwise they will be unable to keep up at high load. Moreover, given that fullnodes typically perform additional operations (e.g., handling REST API and client requests), it is often beneficial for fullnodes to be provisioned with greater hardware to handle the overheads. This is a trade-off that needs to be considered, e.g., fullnode operators that do not want to take advantage of CO can elect not to opt-in and instead continue to use traditional state sync.

Enabling Consensus Observer

CO has already been deployed to the Aptos Mainnet and is enabled on all VFNs by default.

However, CO is not enabled by default for all PFNs, as operators must be aware of the trade-offs and CPU requirements discussed above. Operators who wish to enable CO on their PFNs can do so by following the instructions here.

Note: CO has already been enabled on Aptos Labs operated PFNs (e.g., those operating at https://api.mainnet.aptoslabs.com/).

A New Era for Blockchain Synchronization

With Consensus Observer, Aptos is reimagining blockchain synchronization, paving the way for ultra-low latency transactions at high throughput. By allowing fullnodes to process blocks in parallel with validators, CO eliminates unnecessary delays and improves the user experience.

When combined with Zaptos, Aptos’ parallel and pipelined execution architecture, and Shardines, the execution engine capable of reaching over 1 million TPS, Aptos is on course to deliver the fastest, most scalable Web3 blockchain infrastructure available. This is just one another step along the journey.

If you are, like us, passionate about designing algorithms, putting them in practice, and making a real impact on the future of Web3, please reach out — we are hiring at Aptos.