Retrospect of the London upgrade Ropsten test network accident

Ropsten consensus problem


On July 21, 2021, the OpenEthereum team noticed that their node on Ropsten got stuck at the block  10679538 . People thought this was a problem with OpenEthereum. In fact, the problem really lies in the way go-ethereum implements checking the balance of the sender of the 1559 transaction. An invalid transaction (the sender’s balance is only enough to pay for the gas actually used by the transaction, not the maxFeePerGas total amount specified by the transaction  ) is packed into the block. Since Ropsten miners all run go-ethereum, this block was subsequently accepted by other go-ethereum miners, but was rejected by some other clients in the network. Specifically, OpenEthereum and Besu rejected the transaction/block, and Nethermind, go-ethereum, and Erigon (part of the code implemented by these clients came from the go-ethereum code) accepted it. The source of the problem has been found, and the related client has fixed the problem in the new version:

  • go-ethereum: v1.10.6, fix PR;
  • Erigon: v2021.07.04-alpha, fix PR;
  • Nethermind: v1.10.79, fix PR。

Question block information

  • Network: Ropsten
  • Block number: 10679538
  • Hash value: 0x1252a34c4f2b061adc609e909d958c02e1ac39043e2e60c0ec47e565e3f625f1
  • OpenEthereum debug log
  • eth_getBlock output (go-ethereum)


Timeline of Testnet Incidents

(Note: All times have been converted to Beijing time).

July 21, 2021

  • 18:39: Block 10679537 was dug out on the Ropsten testnet.
  • 21:53: OpenEthereum developers posted on the #1559-dev channel of Ethereum R&D discord that their node was stuck at block 10679538.
  • @smixx 21:58 : Said that their Ropsten node is located at block 10680453.
  • 22:36: Besu confirmed that their node also rejected block 10679538.
  • 22: 51: It is confirmed that the miner who dug out block 10679538 is a go-ethereum node.
  • 22: 55: Confirm that go-ethereum miners continue to mine on block 10679538.
  • 22:56: Confirm that Nethermind also accepted block 10679538.
  • 23:08: go-ethereum has confirmed the root cause of the problem.
  • 23:43: go-ethereum opens a pull request to provide candidate repair solutions.
  • 23:46: Erigon opens a pull request to provide candidate repair solutions.

July 22, 2021

  • 00: 01: The updated go-ethereum and Besu miners restarted on Ropsten (at this time, the wrong chain has been mined to block 10680803).
  • 00: 43: EthereumJS confirmed that it has the same problems as go-ethereum, Erigon and Nethermind.
  • 01: 57: Nethermind opens a pull request to provide candidate repair solutions.
  • 10: 22: Block 10680804 was dug up in the repaired version.
  • 22: 54: go-ethereum released the repaired version v1.10.6.
  • ~23: 00: Nethermind released the repaired version v1.10.79.

July 23, 2021

  • ~00: 00: Erigon released the repaired version v2021.07.04-alpha.

Corrective Action Suggestion

Improve the clarity of the specification assertion (assertion)

This submission adds a new assertion regarding the validity of EIP 1559 type transactions. Specifically, the following assertion is added in line 217 of the code:


Also note that in the first few lines of code (line 207), it has sender.balance been modified to the part after subtracting the transaction volume ( sender.balance -= transaction.amount). This parameter caused confusion because some client teams used all sender.balance(that is, transactiion.amount the sender address balance that was not subtracted ) when checking the assertion defined on line 217  instead of the updated value.

Go-Ethereum recovery

@holiman’s instructions on go-ethereum recovery:

Follow the wrong chain when nodes are synchronized

Suppose you are running  gethand in sync. X A fork occurred on the block  . Your node is following a chain of errors with higher overall difficulty. In the block  Z, you stop the node and update it to the repaired version.

Problem description: The node is still on the “wrong” chain.

Solution: The execution  debug.setHead{X-1) goes back to before the fork occurred. This will revert the node back to X a state before the block  , not necessarily X-1 the state of the block  , because there is  geth not necessarily X-1 the complete state of the block  , but there will be the complete state of some other block. Normally, geth flushes the state to disk approximately every 10,000 blocks (1 hour) and/or during downtime. If geth the  gcmode=archive run, they will brush each block to disk.

Synchronize when the total difficulty of the wrong chain is high

Suppose you are synchronizing a  geth node, X and a fork occurs on the block  . Since the fork has already occurred, and the total difficulty of the wrong chain is higher, you are likely to synchronize the wrong chain, the pivot block is  X+M. In this case, because you do not have the X+M previous state of the block  , you cannot execute it  debug.setHead to solve this problem.

This situation requires resynchronization. However, you need to prevent geth from syncing the wrong fork in the chain. This can be achieved through  whitelist command line parameters.


Therefore, you need to execute  geth --whitelist 123123=0x2342fafa9af9af9af9af9af9.

The so-called whitelist is that when a geth node connects with another peer node, it will request block 123123 data from the other peer  . If the hash in the block header received by the geth node does not match the whitelist, it will be disconnected. This means that the node will reject peers on the wrong chain and only connect with peers on the shorter (but correct) chain.

Posted by:CoinYuppie,Reprinted with attribution to:
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.

Like (0)
Donate Buy me a coffee Buy me a coffee
Previous 2021-07-29 07:46
Next 2021-07-29 07:52

Related articles