Inside Solana’s ‘Textbook’ Response to Exploit

On the night of Aug. 2, Austin Federa was at dinner with friends when notifications started pouring in through the Slack messaging app. 

“I was like ‘Oh, no – I have to go,’” Federa, the Solana Foundation’s communications chief recalled in a recent interview. 

News of the second major crypto hack in two days had just broken, and Federa was on the front lines. Exactly 24 hours after the $200M Nomad protocol was stripped bare in a “crowd-looting”, thousands of people – the vast majority of them Solana users – had their wallets drained in a hack that sparked panic across the entire crypto industry. Solana, a he No. 9 cryptocurrency with a market cap of $15.6B, is leading a new generation of high-speed blockchains challenging Ethereum.

Four Attackers

As word spread and users took measures to protect their assets, the pilfering ground to a halt. Experts believe there were four attackers, who exploited a vulnerability in Slope Finance’s crypto wallets and made off with an estimated $4M, pocket change by industry standards.

Nevertheless, fear that Solana or its network of partners had been compromised — theories that were quickly debunked — spurred Federa and his counterparts into an episode of crisis management. 

It’s an exercise that is becoming important as the number of exploits mounts and the integrity of protocols comes increasingly under attack. Harmony, another Layer 1 blockchain, has struggled to address the impact of a $100M hack in June. Cross-chain bridges such as Nomad — protocols that let users send tokens between blockchains — are acutely vulnerable to attacks. More than $2B has been stolen in 13 exploits, most this year, according to a Chainalysis report.

Massive Supply Chain Attack

“In the early hours of this, it looked like it was potentially a pretty massive supply chain attack,”  Federa said, noting that one of the first reports he’d heard was of a colleague who’d had their Solana and Ethereum wallets drained. 

“At that point, the mitigation and investigation process moves from beyond something where Solana Foundation and Solana Labs engineers are working with wallet providers on the Solana network,” he continued, “and instead becomes something where you have to sound the alarm and pull folks from MetaMask in, folks from Coinbase in.” 

According to reporting by The Defiant, Solana handled the exploit with a deft touch. 

The first official response from Solana came after 10 p.m. Aug. 2. 

“Engineers from multiple ecosystems, with the help of several security firms, are investigating drained wallets on Solana. There is no evidence hardware wallets are impacted,” the Solana Status Twitter account tweeted. “This thread will be updated as new information becomes available.” 

Erik Bernstein, president at Bernstein Crisis Management, said aspects of Solana’s response were textbook. It put out a holding statement acknowledging there was a problem. He said that bought them time to work out a plan to respond. 

At its peak, the digital “war room” the Solana Foundation set up had nearly 130 people. They knew the issue wasn’t at the protocol level, as hardware wallets had been spared. But they still had huge questions to answer, Federa said. 

Affected Wallets

“Eight thousand was both a large number and a very small number of users,” he said, referring to the number of affected wallets, which has since increased to more than 9,000. “And the question was basically, was this vulnerability-set massive and cross-chain and hadn’t been exploited yet and the attackers were just bad?” 

As researchers worked to find out what had happened, updates came from a variety of accounts on Twitter, some seemingly “official,” some not: from Federa; from Slope; from Phantom, a competing wallet whose users had also been affected; from Solana co-founder Anatoly Yakovenko; from security researchers in the aforementioned “war room”; from random crypto sleuths. 

Affected users were asked to complete an online survey that would help researchers find and patch the vulnerability. Everyone else was encouraged to move their assets to a hardware wallet.  

Bernstein applauded the relevant organizations for using Twitter to keep their “tech-savvy, very digitally-native” audience informed. But the chorus of voices “is not something we ever advise a client to do.” 

Gotcha Moment

“I tell you, it’s great if you can pull it off because everyone looks very cohesive, and it really makes you look like you’re sharing as much information as possible,” Bernstein said. “But it gives me anxiety. … There’s a lot of opportunities for people to have what they think is a gotcha moment because one person has framed something differently than another or innocently made a mistake.”

Federa said the instinct to route all communication through a single spokesperson was a “Web2 company approach.” 

“Solana is not a company. It’s a decentralized, open source, community-run software project. So there’s no more authority that Anatoly or myself or one of the audit firms had compared to anyone else,” he said. “There’s a lot of information from other security researchers on Twitter that the group learned by seeing them on Twitter. And if there was a culture of sort of not sharing that and waiting for … an official response, it would have actually made things much slower and it would have made it potentially harder to ascertain the true limited scope of vulnerability.” 

No Vulnerabilities

Although several Solana wallet providers were affected, experts now believe the issue started with Slope. In a statement this week, Phantom said an investigation had “found no vulnerabilities that could explain this user exploit.”

“Private key material from these Slope users was inadvertently transmitted by the Slope app to an application monitoring service,” Solana said in a news release Monday, “but exactly how the hacker obtained or intercepted this information is still under investigation.”

On-chain Activity

Slope, meanwhile, said Monday it was nearing the end of its “internal audit investigations.” And TRM Lab,s hired by Slope to track the attackers’ on-chain activity, was “pursuing multiple leads.” Finally, the company was in daily communication with U.S. federal law enforcement. 

“Based on these discussions,” Slope said, “we remain hopeful.”

Federa said every crisis is different. Nevertheless, he tries to follow a simple playbook. 

“The main thing is to not communicate what you don’t know to be true and to keep people updated,” he said. “Even if an update is, ‘We have nothing to share – yet.’”

Correction: Updated to correct date in first paragraph to Aug. 2 from Aug.7.