The Reorg Recovery Code Was There. Production Never Called It.

Chad — Sat, 08 Aug 2026 02:08:35 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

Soroban Smart Block Explorer turns raw Soroban contract events on Stellar into human-readable activity. Its Node.js indexer polls RPC data, decodes events, stores them in PostgreSQL, and serves them to the explorer frontend.

That makes chain reorganizations a correctness problem, not just a blockchain detail. When the canonical chain changes, events indexed from the abandoned branch must be removed and indexed again from the fork point.

Bug Fix or Performance Improvement

The repository already had most of a reorganization recovery system:

ledger hashes were recorded during indexing;
checkForReorg() could compare stored hashes with the network;
rollback code could remove orphaned rows; and
an operator alert existed.

But the production daemon imported only recordLedgerHash. It never called checkForReorg().

// Before
import { recordLedgerHash } from "./reorgWorker.js";

The safety mechanism was present, but disconnected from the process that owned the ledger cursor. A reorganization could therefore leave orphaned events, hashes, and cursor state in disagreement until someone intervened.

Code

The merged fix is PR #503: fix(indexer): wire reorganization detection.

The core wiring now lives inside the daemon's existing polling loop:

ledgersSinceReorgCheck += Math.max(1, latest - polledFrom + 1);
if (ledgersSinceReorgCheck >= REORG_CHECK_INTERVAL) {
  const forkLedger = await checkForReorg(rpc);
  ledgersSinceReorgCheck = 0;

  if (forkLedger !== null) {
    _cursor = forkLedger;
    logger.warn(
      { ledger: forkLedger },
      "chain reorganization detected; cursor rewound",
    );
    continue;
  }
}

Rollback now makes the database cleanup and durable cursor rewind one operation:

await client.query("BEGIN");
await client.query("DELETE FROM events WHERE ledger >= $1", [forkLedger]);
await client.query("DELETE FROM ledger_hashes WHERE ledger >= $1", [forkLedger]);
await client.query(
  `INSERT INTO daemon_state (key, value) VALUES ('cursor', $1)
   ON CONFLICT (key) DO UPDATE SET value = $1`,
  [String(forkLedger)],
);
await client.query("COMMIT");

My Improvements

1. I kept recovery single-flight

A separate timer would be easy to add, but it could race indexLedger() while both paths read and write the daemon cursor. The reorg check now runs synchronously in the same loop that owns _cursor. Recovery either completes and rewinds the loop, or normal indexing advances and persists the next cursor.

2. I made the scan bounded without creating a detection gap

The default cadence is 100 ledgers and the supported reorg depth is another 100. The checker therefore asks for at most:

const lookback = checkInterval + maxDepth;

That matters during catch-up. A large polling span can trigger a check, but it must not accidentally turn one pass into an unbounded SQL query and a long sequence of RPC calls.

3. I select the earliest mismatch

Stored hashes are returned newest first. Returning on the first mismatch would choose the newest orphaned ledger and leave older orphaned rows behind. The checker scans the bounded window and keeps the minimum mismatching ledger:

earliestFork =
  earliestFork === null
    ? ledgerNumber
    : Math.min(earliestFork, ledgerNumber);

4. I made rollback atomic

Deleting events, deleting hashes, and saving the rewind cursor now share one PostgreSQL transaction. If any statement fails, the transaction rolls back. A crash cannot leave half-cleaned chain data paired with a cursor that points somewhere else.

5. I kept alerting off the critical path

Operators should hear about a reorg, but a notification outage should not prevent data recovery. The rollback completes first; alert delivery is best effort; the detected fork is still returned to the main loop.

6. I added focused seams and coverage

The checker accepts injected hash lookup, rollback, alert, cadence, and depth dependencies. That made the hard cases testable without hiding production behavior:

cadence plus maximum-depth lookback;
the deepest supported fork behind the prior check boundary;
rollback succeeding even when alert delivery fails;
orphaned event and hash deletion;
durable cursor rewind; and
transaction rollback on database failure.

Verification

The final patch changed seven files: 278 additions and 41 deletions. I verified it with:

node --check on every changed or added JavaScript file;
JSON parsing for indexer/package.json;
git diff --check;
a dependency-free boundary/depth/alert-failure smoke (2/2);
a dependency-free transaction commit/rollback smoke (2/2); and
an independent post-fix review that returned no findings after several correctness issues were addressed.

I did not claim a full local Jest/PostgreSQL run. The isolated checkout did not contain node_modules or a local test database, so those tests were added for CI rather than simulated. Hosted CI was also blocked by the same unrelated Rust dead-code failure already present on the exact base commit, plus a fork-token permission job. No reorg file caused those failures, and the maintainer merged the PR after review.

Result

PR #503 merged on July 20, 2026, and issue #489 closed with it.

The interesting part was not writing a brand-new recovery system. It was finding that the system already existed in pieces, then connecting those pieces without introducing a cursor race, an unbounded scan, or a half-committed rollback.

The bug was silent because nothing crashed. The code simply never asked the question it was built to answer.

DEV Community: Chad