Vyper Timeline of Hacks and Reflections

Vyper Hack Timeline

“Trust but verify”, don’t be a “Monday morning quarterback”. The most powerful bugs are often found in the dark.

Due to the immutability of smart contracts, projects implicitly rely on code written years ago. When fixing bugs, we need to be particularly aware of their potential impact.

This is how the incident unfolded.

• Ripple SEC Case Ruling A big step forward for the cryptocurrency industry
• How is the amount of criminal offenses determined when an NFT digital collection platform is involved?
• ChatGPT has become popular in the past six months hot money, giants, and regulation

Timeline

In this article, I will use “we” to refer to all those who contributed to resolving the incident. Although I initially made some contributions in discovering the vulnerability, countless others provided more help throughout the process.

13:10 UTC $11 million worth of pETH/ETH was lost[1].

13:19 UTC Michal posted on ETHSecurity about the sudden drop in pETH price.

Igor was the first to notice that something was wrong. Thanks to him, we began to investigate further.

But how did the robot re-enter the add_liquidity() function from the remove_liquidity() call?

14:01 UTC An emergency team was formed to address this issue.

14:07 UTC We decompiled the JPEGd contract using our favorite decompiler[2] and noticed a slight difference in the re-entry protection storage slot.

// DisLianGuaitch table entry for add_liquidity(uint256[2],uint256)label_0057:    if (storage[0x00]) { revert(memory[0x00:0x00]); }    storage[0x00] = 0x01;// DisLianGuaitch table entry for remove_liquidity(uint256,uint256[2])label_1AF3:    if (storage[0x02]) { revert(memory[0x00:0x00]); }    storage[0x02] = 0x01;

14:27 UTC We confirmed this issue with a simple local test contract.

@external@nonreentrant("lock")def test(addr: address) -> bool:    return True@external@nonreentrant("lock")def test2(addr: address) -> bool:    return False

This is more than just another re-entry bug.

At this point, we realized the magnitude of the impact. We blocked messages and removed public messages about the vulnerability.

14:37 UTC Wavey helped confirm the vulnerable commits and affected versions. Charles and I also confirmed this through manual checks of the Vyper compiler output.

It was a race against the hackers.

Fortunately, people also confused it with read-only re-entry. Quoted from the “Web3 Security Alert” channel – Alchemix and Metronome DAO were also attacked by hackers due to read-only re-entry bugs[3].

Michael discovered potential vulnerabilities in alETH and msETH pools running version 0.2.15 as well.

14:50 UTC msETH/ETH was depleted[4].

15:34 UTC alETH/ETH was depleted[5].

15:43 UTC We found a vulnerability in the CRV/ETH contract compiled with Vyper version 0.3.0[6]. It is crucial for us to keep the affected contract confidential for as long as possible.

16:11 UTC We started researching the white hat vulnerability.

Unfortunately, too many organizations were conducting independent research simultaneously, and rumors were spreading. At 16:44 UTC, we decided to release a public statement regarding the affected versions[7].

By 18:32 UTC, we had a proof-of-concept vulnerability that could be used for potential white hat rescue. Chainlight’s bLianGuaik was also researching a vulnerability and shared it at 19:06 UTC.

Five minutes later, at 19:11 UTC, funds were stolen[8].

The attack structure was significantly different from our proof-of-concept, making it unlikely that our team leaked it. Nonetheless, it was extremely frustrating.

Nevertheless, there is still much to be done.

21:26 UTC Addison proposed an ambitious plan to rescue the remaining assets in the CRVETH pool.

If you send 30k CRV to the CRV/ETH pool, you can update the admin fee and set the CRV/ETH rate to approximately 0.15 ETH per CRV, which would essentially allow the withdrawal of hundreds of thousands of CRV from the pool.

21:52 UTC bLianGuaik came up with a feasible proof-of-concept that could save 3100 ETH.

Ten minutes later, at 22:02 UTC, we were defeated again. Unexpectedly, the CRV admin fee bot[9] was taken, and the pool was depleted[10].

Blame

Blaming is a strong word. Accusations won’t help. I believe it is more useful to think about what could have been done better.

Competition

The white hat efforts were defeated in less than half an hour. Sometimes, every second counts.

Perhaps there could have been better preparation and resources to execute these attacks. However, this seems like a double-edged sword. Is it really a good idea to consolidate information on how to perform hack attacks? Whom should we trust?

On the other hand, I think the whole process was very efficient. We went from initial suspicion to identifying who was vulnerable within 2 hours and 4 minutes.

Information Leak

I am both an auditor and a white hat hacker.

The auditing industry has a unique culture of disclosure. We are rewarded for our technical thought leadership and deep understanding of vulnerabilities. One way to demonstrate their leadership and depth is to publish “exclusive news” about hacker activities[11]. Researchers invest a lot, and the return on investment is publicity.

On the other hand, there is a compelling argument that early disclosure of the affected version would have a significant impact on white hat rescues.

If another half hour is available, $18 million can be saved.

Auditors do not pay the price for the impact of their reports. On the contrary, they receive likes, shares, and coverage. This seems to be a problem.

Next Steps

I disagree with the view that “we need formal verification to solve this problem.” This error can be captured through unit testing. Formal verification is very useful for many types of errors, but I don’t believe it is equally useful for relatively simple, unoptimized compilers.

It should be noted that this error was fixed in November 2021 [12].

I believe that this Vyper vulnerability is not a problem with the technical aspects or language itself of the Vyper team, but more of a process problem. This error was fixed in versions from a long time ago, but its potential impact was not realized when it was fixed.

Unfortunately, public goods are easily overlooked. Due to the immutability of contracts, projects implicitly rely on code written years ago. Protocol developers and security experts should be aware of the latest security developments in the entire execution stack.

We will continue to update Blocking; if you have any questions or suggestions, please contact us!