DEV Community

Cover image for Crypto's $19 Billion Night Was a Distributed Systems Bug
Sonia Bobrik
Sonia Bobrik

Posted on

Crypto's $19 Billion Night Was a Distributed Systems Bug

On October 10, 2025, crypto exchanges force-closed more than $19 billion of leveraged positions belonging to roughly 1.6 million traders, the largest liquidation event the market has ever recorded. Most coverage told it as a macro story about a tariff threat, a nervous Friday, and too much leverage. Strip out the tickers, though, and the timeline reads like an incident report any backend team would recognize: a positive feedback loop, a source of truth that lied under load, and an emergency mechanism that made things worse the first time it actually ran. A good lens for taking it apart is this research-driven guide to avoiding the most common crypto mistakes, which treats market, counterparty, and technical risk as separate layers. That night all three failed at once, and the seams between them failed hardest.

The Move Was Ordinary. The Cascade Was Not.

The trigger was a post from President Trump announcing an additional 100% tariff on Chinese imports plus export controls on critical software. Bitcoin, which had set a record above $125,000 earlier that week, fell more than 12%. That is a brutal afternoon, but not an unprecedented one, and analysts at 21Shares later noted that Bitcoin's drawdown was smaller than in earlier major deleveraging events. The record lived in the plumbing. Bloomberg's reporting on the liquidation wave described more than $7 billion of positions being sold in under an hour, and CoinGlass warned that even the record tally was probably an undercount, because Binance publishes only one liquidation order per second.

Think about what that means operationally. During the most violent minutes the market has ever seen, its busiest venue was emitting sampled telemetry. Anyone who has debugged an outage with dashboards that only show averages knows the result: the worst minute of the year is also the least visible one. It skews the leaderboard too. Hyperliquid, where every liquidation lands on a public chain, topped CoinGlass's charts with about $10.3 billion, a comparison that quietly flatters venues that report less.

Mechanically, a liquidation cascade is the textbook positive feedback loop. Falling prices push leveraged positions below their maintenance margin, the liquidation engine dumps that collateral into the order book, the selling pushes prices lower, and the next tier of positions crosses its threshold. It is a retry storm with money attached: every failure adds load to the exact component that caused it.

Bug One: The Source of Truth Was the Thing on Fire

The most instructive failure happened inside Binance's unified margin accounts, which let traders post assets such as Ethena's USDe and Binance's own staking tokens, wBETH and BNSOL, as collateral. The exchange valued that collateral largely from its own order books. As market makers pulled their quotes and transfers on and off the platform degraded, USDe printed as low as $0.65 on Binance while staying within a fraction of a percent of a dollar on other venues and on-chain. wBETH, a token backed by staked ether, briefly traded around $430 while ETH itself was above $3,800. Nothing had happened to the underlying assets. Something had happened to Binance's view of them, and positions that had lost no real value anywhere else were liquidated anyway.

Every backend engineer has met this bug. It is the health check that asks the overloaded node whether it is healthy, or the cache that becomes the source of truth in the middle of a stampede. On a normal day, arbitrageurs behave like a reconciliation job, buying the cheap copy on one venue and redeeming it elsewhere until prices converge. That night the reconciliation job could not run, because moving funds on and off the exchange was exactly what had degraded. In some thin markets the price became whatever happened to be left in the book, and Binance later blamed several of the wildest wicks on stale limit orders dating back to 2019.

The part that stings is the timing. On October 6, Binance had announced that from October 14 it would start factoring staking conversion ratios into how it priced wBETH and BNSOL. The fix was already sitting in the release train when the incident arrived. Afterward the exchange paid about $283 million in compensation, mixed redemption prices into its index, and added a soft price floor for USDe. The general lesson is the one you would apply to any critical read path: use independent sources, and treat disagreement between them as an incident rather than as a price.

type Quote = { source: string; price: number };

// Never let the component under stress be the only witness to its own health.
export function markPrice(quotes: Quote[], maxDeviation = 0.02) {
  if (quotes.length < 3) {
    return { price: null, pauseLiquidations: true, reason: "too few independent sources" };
  }

  const sorted = quotes.map((q) => q.price).sort((a, b) => a - b);
  const median = sorted[Math.floor(sorted.length / 2)];
  const divergent = quotes.filter((q) => Math.abs(q.price - median) / median > maxDeviation);

  // One venue printing $0.65 while the rest print $1.00 is an incident, not a price.
  if (divergent.length > 0) {
    const names = divergent.map((q) => q.source).join(", ");
    return { price: median, pauseLiquidations: true, reason: `divergent: ${names}` };
  }

  return { price: median, pauseLiquidations: false, reason: "ok" };
}
Enter fullscreen mode Exit fullscreen mode

Pausing has failure modes of its own. Someone who can push a single venue far enough might try to freeze liquidations on purpose, so a real system pairs a check like this with alerting, time limits, and a human decision. But a paused liquidation can be resumed. A wrongful one can only be apologized for.

Bug Two: The Safety Valve Had Never Been Opened

Perpetual futures venues keep one last mechanism for days when liquidations cannot keep up: auto-deleveraging, or ADL. When a losing account's margin is exhausted and the insurance backstop cannot absorb the hole, the venue closes positions on the winning side, starting with the most profitable and most leveraged, so the books still balance. On Hyperliquid, that Friday was the first time cross-margin ADL had fired in more than two years of operation. A later paper by Tarun Chitra of Gauntlet found that it closed about $2.1 billion of positions in twelve minutes.

The trouble is who stands at the front of that queue. Many of the winning positions belonged to market makers and funds whose profitable shorts were hedges for exposure held somewhere else. ADL closed one leg and left the other standing, so books that had been neutral turned directional in the middle of a crash. Some firms rushed to reopen shorts to get back to neutral, adding fresh selling to a falling market, while others pulled back from quoting altogether. It is the infrastructure equivalent of a load shedder that evicts your healthiest tenants first.

The deeper lesson is about code paths. Logic that runs once every two years is the least exercised code in your system, and nobody on call has ever watched it run in production. If your design leans on an emergency path, that path deserves game days and replays against realistic data, not just a paragraph in the docs.

Eight Months Earlier, the Same Lesson Cost $1.5 Billion

October was a lesson about trusting the wrong source of data. February had already taught the same lesson about trusting the wrong source of code. On February 21, 2025, attackers drained more than 400,000 ETH from a Bybit cold wallet, and five days later the FBI attributed the roughly $1.5 billion theft to North Korean operators it tracks as TraderTraitor. Chainalysis later put total crypto theft for all of 2025 at $3.4 billion, which means this single incident accounted for more than 40% of the year.

Bybit's own infrastructure was never breached. The wallet was a Safe multisig, and investigators traced the compromise to a Safe developer's laptop that had been infected through social engineering more than two weeks earlier. Using stolen cloud credentials, the attackers modified JavaScript served from the S3 bucket behind the Safe web interface. The injected code activated only for Bybit's cold wallet and one test address, showed the signers what looked like a routine transfer, and passed their hardware wallets a different payload: a delegatecall to a pre-deployed contract that overwrote the wallet's implementation pointer. About two minutes after the theft, the original files were put back.

Two truths fall out of that. First, the presentation layer belongs to your trusted computing base. The hardware wallets did their job perfectly, which is to sign exactly what they are handed. Second, N-of-M approval only means something when the N paths are independent. Three signatures produced through one compromised browser bundle are, for security purposes, one signature.

Three Layers, One Graph

The guide linked at the top is right that most people stuff crypto risk into a single bucket labeled "price." October 10 shows why separating the layers is only half the work. A market shock became a counterparty failure, one exchange mispricing its own collateral, through a technical design choice, and a second technical mechanism then turned it into a liquidity problem for everyone else. The layers are coupled, and the couplings are where the money disappears. Whether you hold crypto, build on it, or run systems that merely rhyme with it, the same review questions apply:

  • Name your source of truth. For every number that can trigger an irreversible action, such as a liquidation, a signature, or a withdrawal, write down where it comes from and ask whether that source is the thing most likely to be on fire when you need it.
  • Count independence, not replicas. Three signers on one front end, five oracles fed by one venue, and two "separate" wallets behind one cloud account each count as one.
  • Rehearse the rare path. Auto-deleveraging, emergency pauses, key rotation, and withdrawal halts deserve game days precisely because they almost never run.
  • Verify what you sign out of band. Decode the payload on a separate machine, recompute the transaction hash, compare it with the hardware wallet's screen, and refuse any delegatecall to a contract that is not on an explicit allowlist.
  • Assume your telemetry lies under load. If a dashboard samples one event per second, the worst minute of the year will be the one you can see least.
  • Size for the cascade, not the move. A 12% drop with a feedback loop behind it does not behave like a 12% drop.

The Boring Part Is the Point

A year on, the uncomfortable thing about October 10 is how little of it was exotic. There was no zero-day, no broken cryptography, no consensus failure, and the major chains kept producing blocks the entire time. What broke was the ordinary scaffolding around them: a price feed that read from its own wreckage, a reconciliation path that went dark when it was needed most, a safety valve nobody had seen open, and a dashboard that sampled its way through the storm. The Bybit heist had the same shape, a web front end trusted far more than it deserved.

That is good news for developers, because these are failure modes we already know how to engineer against. The next trigger might be a tariff, a hack, or a stablecoin wobble at 3 a.m. on a Sunday. Whether it becomes another $19 billion night will depend on the same decisions that decide whether a bad deploy becomes an outage: where your truth comes from, how independent your safeguards really are, and whether anyone has run the emergency path before the emergency.

Top comments (0)