Blockchain storage explosion: problems, analysis and optimization

(Editor’s note: This article is a technical article written by QuarkChain founder and CEO Dr. Zhou. Based on this article, a technical sharing lecture was conducted in DApp Learning. For the Chinese lecture video, please click the second article push)

background

The vigorous development of decentralized applications such as DeFi and GameFi has greatly increased the demand for high-performance blockchains with low transaction costs. However, a key challenge in building a high-performance blockchain is the explosion of storage. The figure below is a chart taken from Etherscan, which illustrates the blockchain data size of an Ethereum full node (archived).

Blockchain storage explosion: problems, analysis and optimization

From the figure, we can see that the scale of node chain data has grown steadily and has now reached ~9TB. Since one of the goals of the decentralized blockchain is to allow computers with common configurations to run nodes, it will be difficult to achieve the mandatory requirement of 9TB+ storage on computers with common configurations.

Break down storage overhead

If we further analyze the storage usage, we can find that the block data only accounts for about 300GB of data (from block height 0 to 13.6M), which is much less than 9TB. So where does the remaining 8.7TB of data come from?

In fact, the archive node executes all blocks and retains all historical data , including:

  • Block
  • state

  • Transaction receipt

Among them, the state is the main component of this 8.7TB. So sometimes, we refer to storage explosions as “state explosions.” But why is the state so great?

What is the state of Ethereum?

The Ethereum state is a Merkle Patrica tree (MPT) in which

  • The leaf node is the mapping of address (0x…) => account, where the account stores the balance, nonce, etc. associated with the address

  • The internal nodes maintain the tree structure so that the hash root of the entire tree can be quickly calculated

Blockchain storage explosion: problems, analysis and optimization

Since the archive node will retain all the historical state of all blocks, this means that any update in the MPT will create O(log(N)) internal nodes and will not delete the old internal nodes.

Blockchain storage explosion: problems, analysis and optimization

Geth’s full node

In order to solve the problem of the state explosion of archive nodes, Geth’s talented engineers created a new mode called “pruning” mode, which only stores MPT regularly. Here we give a simplified example in which the node only saves the MPT for every 3 blocks. (Note that in order to obtain a state that does not contain any state block, the node must obtain the most recent state before the block and replay the next transaction).

Blockchain storage explosion: problems, analysis and optimization

By storing the MPT regularly, the storage size of the state is significantly reduced. According to Etherscan data, the current block chain data size of Geth full node is about 1TB.

Blockchain storage explosion: problems, analysis and optimization

Geth’s fast-synchronized full node

One problem with running a node by replaying all transactions from the genesis block is that replaying all transactions takes a long time. Generally speaking, it takes several weeks to establish such a node to catch up with the latest state of the network from the genesis block. In order to speed up the node startup process, Geth further provides a fast synchronization mode, which can download the latest stable block MPT without replaying and maintaining the historical MPT before the block. After downloading the MPT, it will replay the new block like a full node (with periodic state storage).

Blockchain storage explosion: problems, analysis and optimization

Without storing historical MPT (sometimes even the main body of historical blocks), the storage size of a Geth node can be further reduced to 447G (as of 2021/12/06). By subtracting 300GB of block data, we infer that the state size is about 150GB.

problem

With the current Ethereum storage size of 447GB and 15 TPS, we expect that a normal configuration computer with a 1TB SSD should be able to run the Ethereum node for a considerable period of time (for example, several years). So does storage explosion or state explosion really exist? Maybe Ethereum will not be in the next few years, but what if we can extend the Ethereum Virtual Machine (EVM) to hundreds or thousands of TPS?

Let us turn our attention to another EVM-based chain, Binance Smart Chain (BSC). As of December 8, 2021, BSC has:

  • About 984 GB of on-chain data, of which blocks account for about 550 GB, and state accounts for about 400 GB.

  • 2.06623 billion transactions, 100 TPS

If we further use the number of transactions to predict the size of the data, we can get:

If TPS is 100, that is ~3,153 M TPY

  • After 1 year, total TX ~5,219M, block ~ 1.375 TB, status ~ 1.085TB
  • After 3 years, total TX ~11,525M, block ~3.025TB, status ~2.387 TB

If TPS is 150 (the observed peak TPS), that is ~4,730 M TPY

  • After 1 year, total TX ~6,796M, block ~1.809 TB, status ~1.427 TB

  • After 3 years, total TX ~16,256M, block ~4.327 TB, status ~3.414TB

In summary, for BSC, if the current speed is maintained or even higher, it will soon reach the same storage size of the Ethereum archive node, which is almost impossible for ordinary computers to run.

Storage explosion problem of blockchain with extremely high TPS

If we make a bolder assumption about a very high TPS blockchain (like QuarkChain can do), what will this number become? Let’s consider a blockchain with 1000 TPS and analyze its block and state size, it will be:

  • Assuming the size of tx is about 100 bytes, the amount of storage required for a block per year is 1000 (TPS) * 100 (bytes per tx) * 365 * 24 * 3600 = 2.86 TB

  • Assuming that MPT has 10 billion accounts (more than the world’s population!), we expect the state size to be 150G (Ethereum state size)/0.18B (Ethereum unique address) * 10B = 8.3 TB

Putting these numbers together, we can easily come to a conclusion that this is a requirement that most ordinary configuration computers will not be able to withstand!

optimization

In order to optimize storage costs, we must relax the restrictions to be compatible with EVM rather than compatible with Ethereum. That is, we must build/run another chain that supports EVM instead of a highly optimized Ethereum client.

State storage optimization

The first optimization we propose is to use ordinary KV instead of MPT. When the MPT is large, all internal nodes in the MPT can be very expensive. And our optimization will remove all internal nodes in MPT. Assuming that the data of each account is about 50 bytes (20 addresses + 2 nonces + 12 accounts + others), we can save the data of 10 billion accounts as follows:

  • ~ 10B * 50 + 100GB (code) = 600 GB, which is about 1/10 of the MPT version!

Although the use of ordinary KV will bring huge benefits, a major problem is that we cannot calculate the post-hash of each block in such a short block interval, which means that we will lose the following benefits of Ethereum:

  • Fast synchronization: download the status of any block and quickly synchronize the network by replaying the remaining blocks

  • Fork detection (or Byzantine detection): Whether the newly created block from the peer will cause the state to be different from the local execution block.

To enable fast synchronization, we have a periodic snapshot block (snapshot interval = epoch = for example, 14 weeks). A snapshot block contains the additional information of the previous state hash, that is, the post state hash of the previous snapshot block (the state hash after the transaction is executed):

  • A non-snapshot block does not maintain a state hash, but has an incremental hash, which contains the hash of the original database operation (delete, update) of all transactions in the block. This makes fork detection possible!

  • We use the pre-transaction state hash to replace the post-transaction state hash of the block in Ethereum. The reason is that the node cannot immediately calculate the state hash after the transaction, but by using the state hash before the transaction, the node can use the entire epoch interval to calculate the hash. For example, assuming that the state hash calculation processes 10M state data per second, it will take 600 GB / 10 M ~ 16.67 hours to calculate the entire state of 600 GB (vs. epoch = 14 weeks)

Blockchain storage explosion: problems, analysis and optimization

The process of hashing before calculating the state is as follows:

1. When a snapshot block is received and finalized, its KV state is snapshotted, and a background thread is created to iterate all KV entries (address => account) and calculate the hash.

Blockchain storage explosion: problems, analysis and optimization

2. When the next snapshot block is created, the calculated state hash value will be stored in the block. Similarly, the node will create another snapshot of KV and calculate its hash in the background.

Blockchain storage explosion: problems, analysis and optimization

3. When the next snapshot block is created, the node can now release the KV snapshot of the snapshot block in addition to storing the previous state hash, which means that all deleted/updated data from the snapshot block will be automatically Garbage collection (for example, compression in levelDB)

Blockchain storage explosion: problems, analysis and optimization

The result means that to store state, a node only needs up to two KV snapshots (most likely one with increment and one KV snapshot).

Block storage optimization

Using snapshot blocks, we can further reduce the required block data in the node by storing only the following data:

  • A snapshot of the state before the transaction execution of the latest snapshot block, that is (latest �?1) the state after the transaction execution of the snapshot block

  • (Latest-1) The complete block after the snapshot block 

We can perform simple mathematical calculations on the storage cost: assuming that the epoch duration is 2 weeks, the block re-enlargement is as small as

  • 2 * 14 (days) * 24 (hours) * 3600 (seconds) * 100 * 1000 (TPS) = 224 GB!

Moreover, the numbers here will not grow over time!

Blockchain storage explosion: problems, analysis and optimization

Summarize

We analyzed the current storage usage of Ethereum:

  • Not only blocks, but state storage consumes a lot of space

  • When TPS> 1000, the storage space usage is prohibitively high

We propose to optimize the block and state:

  • Reduced block size from 2.86 TB per year to 224 GB
  • State size (~10B account) reduced from 8.3 TB to 600 GB

  • A 2TB general configuration computer should be able to meet the conditions for long-running nodes

  • Disadvantages: the light node cannot verify a piece of data in the state (must be a full node)

Thanks

Thank you dapp-learning for hosting this event. The whole teaching video can be viewed by clicking on the second push of the official account.

Posted by:CoinYuppie,Reprinted with attribution to:https://coinyuppie.com/blockchain-storage-explosion-problems-analysis-and-optimization/
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.

Like (0)
Donate Buy me a coffee Buy me a coffee
Previous 2021-12-24 00:01
Next 2021-12-24 00:04

Related articles