Previewnet — and setting new industry milestones!

Aptos network achieves new performance and throughput milestones in their mainnet-like environment, Previewnet.

TL;DR

The team at Aptos Labs has enhanced the network’s scalability and performance. The changes are tested on an environment called previewnet. This article describes impressive numbers and new industry milestones achieved in the previewnet, which concluded at the end of November 2023.

• Put together an environment mirroring mainnet.
• 30k peak peer-to-peer (p2p) transactions per second (TPS).
• 25k sustained p2p TPS for hours.
• Processed 2 billion transactions within 24 hours.
• Minted over 1 million limited collection NFTs in 90 seconds.

What is Previewnet?

In partnership with our community, the team at Aptos Labs is continuously making the Aptos Network more scalable, performant, and robust. While all improvements are thoroughly tested on large-scale, multi-region clusters, more is needed to replicate the subtle nuances of a globally decentralized platform. Heterogeneous hardware, variable node configurations, non-uniform network characteristics, and the human idiosyncrasies involved in node operations make the Aptos mainnet a unique environment.

As a result, every major Aptos protocol change must be put through a deep testing cycle in an environment that closely replicates what we believe Aptos mainnet will look like in a few months. This includes having nodes deployed by the same node operators with all the characteristics of their corresponding mainnet nodes. This is necessary to thoroughly verify each change, ensuring that the new behaviors are scalable, performant, secure, and reliable. We call this testing environment ‘Previewnet’.

As we accumulate large network changes, our node operators spin up a previewnet, in which we stress-test the network with extreme load tests, heavy workloads, and long periods of sustained activity. This blog summarizes the previewnet we put together in November 2023.

Setup and timeline

Here are the key stats about the recent previewnet and the timeline:

• Number of validator nodes: 125 (5% higher than the mainnet)
• Number of countries: 26 (Identical to the mainnet)
• Nakamoto Coefficient: 33 (vs. 17 in the mainnet)
• Stake Distribution:
  - EU: 40%
  - Asia: 25%
  - Americas: 26%
  - Africa, Middle East, and Australia: 9%
• Setup begins: Oct 30th, 2023
• Testing starts: Nov 6th, 2023
• Testing ends: Nov 21st, 2023
• Total transactions processed: >9 Billion (Over ~2 weeks)

The previewnet setup mirrors the Aptos mainnet in geo-distribution, number of nodes, stake distribution by geography, and number of unique node operators. Each operator deployed servers that met a specific single-node benchmark built for the previewnet. (if you want to run the benchmark yourself, you can follow the steps here).

The Nakamoto Coefficient in Previewnet was 33 versus 17 in the mainnet due to lower stake distributions per node — this means that the mainnet should have equal or better performance than this test result. The network will be further decentralized as new validators are added to the mainnet. The goal is to achieve a Nakamoto Coefficient of 30+ in the mainnet, which is why Previewnet was set up in a way that would mirror the mainnet stake distribution in the future.

Below are the details of the tests we conducted.

Performance and Peak TPS (Transactions Per Second)

(Recap) The journey so far: Earlier this year, we presented the first, fully reproducible performance benchmark to kick off an industry-wide conversation around the definition of performance metrics like “Transactions per second (TPS)”. We demonstrated that our software stack could achieve ~20k TPS in a network of 100 nodes in a globally geo-distributed setup. As a quick recap, the following key innovations helped Aptos achieve high throughput in the benchmark:

• Horizontally scalable consensus with Quorum Store: Aptos uses a novel consensus mechanism based on Quorum Store to scale consensus. The idea behind Quorum Store is decoupling data from metadata so that data dissemination can happen outside of the critical path of consensus.
• Parallel Execution Engine with BlockSTM: To scale the execution layer, we have designed BlockSTM, our parallel execution engine that uses Software Transactional Memory (STM) with optimistic concurrency control to execute transactions in parallel. Our benchmarks show that we can execute 160k transactions per second.
• Highly Optimized Batch Storage: Aptos’ storage approach combines persisted and in-memory, lock-free sparse Merkle trees specifically tailored to work with Block-STM for caching and parallelization.
• Resource Efficiency With Pipelined Processing of Stages: All stages of transaction processing are pipelined to fully utilize resources, making the throughput only as slow as the slowest stage (instead of all stages being combined sequentially). Pipelining helps nodes to utilize all of their resources simultaneously, leading to a significant increase in throughput.

Achieving a higher peak TPS: Using our reproducible benchmarks, we observed that the storage layer was becoming a bottleneck for us. We used a single instance of RocksDB, which could not write data to DB for more than ~20k transactions per second. To address this, we introduced a new sharded storage design, which splits state storage into multiple RocksDB instances so that we can parallelize the commit. In addition to this, we also shipped several improvements to our execution engine, state synchronization algorithm, and network stack for higher throughput.

All these improvements culminated in us hitting a new peak of >30k TPS in the previewnet.

The journey ahead: Aptos aims to enhance scalability further, targeting >100k TPS as the next milestone in our journey to exceed 1 million TPS. This ambitious objective aligns with Aptos’ mission to build a network capable of serving billions of users, ushering in the next wave of adoption for Web3 technologies. Achieving higher TPS is crucial for accommodating the diverse and growing demands of a global user base in the Web3 ecosystem.

Sustained Peak Throughput — a New Industry Milestone!

Peak TPS is only a small part of the overall throughput story at Aptos. A network built for billions of users must sustain high throughput for long periods. With that in mind, we aggressively pushed the network to execute a sustained TPS of ~25k for more than 24 hours. During this time, the network successfully processed an unprecedented 2.1 Billion (with a B) peer-to-peer payment transactions within 24 hours without a hiccup. For context, Visanet processes 150 million transactions per day. The graphs below show this load’s sustained TPS and timestamps in previewnet.

Aptos Network processed more than 2 Billion transactions in a day — a new industry milestone!

Parallelizing Sequential Workloads

When it comes to high throughput discussions, most of the industry focuses on peer-to-peer transfers. This made sense in the past because payments and transfers were the original cryptocurrency use cases. However, these use cases are no longer fully representative of the types of workloads we see in blockchains today. Specifically, unlike the transfer workload that is easily parallelizable, many existing workloads exhibit sequential properties. They require sequential execution and drastically reduce throughput because they cannot be trivially parallelized across execution cores.

One example is the minting of Non Fungible Tokens (NFTs) with a limited supply and sequential naming. For example, an NFT collection that only allows 1 Million NFTs to be created, where each NFT is sequentially named (i.e., each token is given a unique index based on the order in which it was minted, and that index is included in the name of the token, e.g., “Superstar #143”). This is a common convention for NFTs.

Both limited supply and sequential naming require sequential execution, drastically reducing transaction throughput. To overcome this, we developed a novel technology at Aptos, a conflict-free counter called Aggregators. With Aggregators, we can remove sequential computation from the critical execution path, achieving parallelized execution, while still maintaining the necessary properties of a sequential workload (i.e., the execution results are as if the workloads were executed sequentially, despite being parallelized). We will cover the details of this technology in a series of upcoming blog posts. For now, the takeaway is that Aggregators allow us to parallelize workloads that would otherwise be sequential.

Using Aggregators on the previewnet, we were able to mint an entire NFT collection of 1 Million NFTs in ~90 seconds, and a 5 Million NFT collection in ~8 minutes, with sustained throughput of over ~10k NFTs per second. That results in 10x improvement when compared with sequential execution (i.e., without utilizing Aggregators). Being able to parallelize these workloads seamlessly is currently unmatched in the industry.

Minting 1 million limited collection NFTs in 90 seconds opens up new NFT use cases to support large-scale real-world events.

As we shared before, over time, Aptos’ goal is to work with our peers to evolve the blockchain equivalent of TPC benchmarks, as a fair and comprehensive way to compare performance across different networks. It should have a set of representative workloads that translate into real-world performance, giving application builders and users valuable information. With this post we are introducing a new metric to the suite — “sequential NFT mints per second”, with more to come.

We invite the industry to add one more metric to the benchmark suite — sequential NFT mints per second.

Disaster recovery (DR) Drill

While the network is designed from the ground up to be reliable, secure, and fault-tolerant, it is always necessary to plan for the worst case and establish strong operational practices. We harden every system component, but preparing for all outcomes is critical. If an incident does occur, we need to be able to restore the network to full health as fast as possible. To inject a fault into the network, we ran an experimental feature. Along with our node operator community, we were able to identify and execute a script that quickly brought the network back online.

Key Lessons Learned

• The real world is more complex: The previewnet reaffirmed our thesis that any protocol changes must be certified under real-world environments with a wide range of extreme and sustained workloads. While we already test our changes on large-scale, multi-region clusters, the Aptos mainnet is unique, with more characteristics, variables, and heterogeneity than can be tested. As a result, we uncovered nuanced performance and reliability issues that would have otherwise been missed. These resulted from heterogeneity in operating systems, hardware, deployment architectures, storage technologies, network infrastructure, and operator best practices. While test environments are necessary for verifying changes, they will always be insufficient compared to the real world
• The read path is equally important: Most blockchain performance conversations revolve around write-path latencies and throughput. However, little mention is given to the read-path, e.g., read-replication throughput and latencies across fullnodes, API services and indexers. This is problematic because blockchain users today rely on these services to read and/or verify blockchain state. As a result, if the read path cannot maintain equally high (or higher) performance, the user experience will suffer, regardless of how fast the write-path may be processing transactions. We transiently witnessed this in the previewnet, reminding us that the read-path cannot be deemed a second-class citizen.
• High-quality tooling is invaluable: Previewnet provided a stark reminder that automated, high-quality tooling is critical. When running large scale experiments across heterogeneous deployments, the number of variables to consider can be overwhelming. This is especially true when debugging complex interactions and unexpected behaviors. Tooling is the only way to manage this complexity and effectively identify the issues at hand. For example, there were several instances where nodes became CPU or disk bottlenecked during testing, and we relied on automated flamegraph generation and metric analysis to help identify the root causes. Likewise, during the previewnet provisioning process, we leveraged single-node benchmarks to help operators quickly verify the performance and correctness of their hardware selections, saving time before the network went live. Finally, the initial setup of the network took longer than expected due to several inconsistent configurations, identifying the need for better tooling around observability and deployment. Tooling saves time when it matters.

A shoutout to the community

Previewnet proved to be an invaluable exercise for the Aptos Network and its operators. We are grateful to have an amazing community of node operators. They put up with demanding schedules, upgrade asks and proactively monitored the network. The camaraderie and enthusiasm pushed us to bigger and better experiments throughout the previewnet. Below are several quotes from the node operator community.

• “Aptos previewnet processes continue to help our team push our technical boundaries both from a hardware and software perspective. The process is demanding but well worth it when you see the progress made each time. The Aptos team continues to lead the space in proficiency and professionalism!!” — Republic Crypto
• “One of the most important things I learned during Previewnet 2 is that the latency between the validator and VFN is wildly important. Similarly, state sync performance is really the glue that holds the network together. It was very exciting seeing the state sync improvements result in significant performance gains. 30k TPS is absurd!” — Lavender Five
• “It was a really good experience on the whole process of setup, configuration, migration, and upgrades. Some upgrades went well. Some upgrades did not, but we learn more when it doesn’t go well so there’s always a positive side to it.” — Artifact
• “It was great to run multiple uncommon scenarios, like DR and consistent high load. In addition, it was great to be able to test improvements (e.g. in sync) made by the team. These will make mainnet deployments substantially more robust in future.” — Envoy Labs
• “The Aptos Team worked diligently to test the DAG consensus upgrade while also pushing the network to its limits. Aptos network’s resiliency was proven in this Previewnet as several well-known node operators came together to optimize the hardware requirements that would pave the path for a brighter future for Aptos.” — Luganodes

Conclusion

The experience affirmed our commitment to the previewnet approach. We are already looking forward to the next previewnet early in 2024. See you all there soon!