Fred Santana

Posted on Apr 27

Surviving Midnight SDK: a 700-line cure for the silent failure problem

#midnightchallenge #web3

Midnight Network Challenge: Enhance the Ecosystem

Why I built midnight-doctor — a pre-flight check that catches the 16 hours of debugging you don't know you're about to spend

"The information you need to align your stack already exists. It's just not executable."*

After five months building four applications on Midnight Network, I've concluded the protocol isn't the problem. The protocol is genuinely good — Compact compiles ZK circuits in three seconds without a trusted setup ceremony, and the shielded/unshielded/dust split is the cleanest answer to compliance-grade privacy I've seen.

The problem is everything that surrounds it. Specifically, the unforgiving math of: a fast-moving SDK + multiple environment tracks + a single npm namespace + zero error messages when versions misalign.

This post is the story of one specific 6-hour debugging session, the pattern I extracted from it, and the ~700-line tool I built so the next person doesn't pay the same tax.

The 6-hour silent failure

DPO2U Wallet, March 2026. I bumped @midnight-ntwrk/wallet-sdk-facade to 2.0.0 because npm marked it as latest. My local stack ran midnightntwrk/midnight-node:0.21.0 (preprod-targeted). I wrote ~50 lines of wallet bootstrap, started the dev loop, and watched:

const wallet = await WalletFacade.init({
  configuration,
  shielded,
  unshielded,
  dust,
});

await wallet.waitForSyncedState();
console.log('synced!');

The synced! line never printed.

No error. No timeout. No log. The wallet just sat in syncing state forever. I added more logging — got more "syncing" lines. Restarted Docker. Wiped state. Tried a fresh seed. Re-cloned the repo. Asked Discord. Read the SDK changelog (which didn't exist for that version). Stared at the indexer logs for 40 minutes.

Six hours in, almost by accident, I noticed the symptom: subscribeRuntimeVersion was firing once, returning, and never firing again. The standalone node closes that subscription early. The WalletFacade.init({...}) constructor in 2.x wires sync to that subscription. Result: a single missed event silently kills the entire sync loop, with zero surface.

The fix was to downgrade to facade 1.0.0 (preprod track). Forty seconds of work, after six hours of debugging. The bug was real but the cost was information asymmetry: nothing in the documentation, npm metadata, or runtime told me that "this SDK version is incompatible with this node version."

That asymmetry is the real bug. Not the runtime subscription closing. Not the constructor wiring. The fact that the system has the information needed to diagnose itself, but doesn't bother.

The pattern: silent failures from missing cross-references

After that session, I started cataloguing other times I'd been bitten the same way. Within a week I had a list:

Symptom	Root cause	Time spent
`waitForSyncedState()` hangs forever	facade 2.x + node 0.21.0 mismatch	~6h
`npm install` returns ENOTFOUND	community tutorial said `.npmrc` should point at `npm.midnight.network` (domain doesn't exist)	~2h
Transactions silently fail to submit	Two versions of `@midnight-ntwrk/ledger-v7` in `node_modules` (transitive bump)	~3h
Indexer crash-loops on startup	`indexer-standalone:4.0.0-rc.4` requires a `subscription:` block in `indexer.yml`, undocumented	~1.5h
WalletFacade behaves differently on standalone vs preprod	Same SDK, different node behavior, no warning	~4h

Across five incidents: ~16 hours of debugging silent failures. Each one had a root cause that was, with the right cross-reference, detectable in seconds:

package.json says facade is 2.0
docker ps says node is 0.21.0
A lookup table says those two are incompatible
→ Print a clear error.

The system already has all three pieces of data. There's just no glue.

The tool: `midnight-doctor`

The fix isn't a new feature in the SDK. It isn't better docs (which decay). It's an executable cross-reference — a script that reads your project, your Docker stack, and your config files; matches them against a curated compatibility table; and tells you what's wrong.

I built midnight-doctor over a weekend. ~700 lines of Node, zero runtime dependencies, single-binary install:

npx midnight-doctor

Run against my own legacy midnight-hello-world repo, it produces:

── midnight-doctor ──
project: /root/midnight-hello-world

⚠ SDK track: Preprod 3.x (legacy, OK for existing apps)
   Detected from wallet-sdk-facade@2.0.0.
⚠ Track is deprecated
   SDK 3.2 / facade 2.0 is 2 majors behind. Current is wallet-sdk@1.0.0 /
   facade@4.0.0 (released 2026-04-23). The WalletFacade.init({...}) constructor
   used in 2.0 has been removed; 4.0 reverted to `new WalletFacade(s, u, d) +
   .start()`. Plan a migration.
⚠ WalletFacade.init() in 2.x stalls on standalone dev nodes
   In SDK 2.x, WalletFacade.init() subscribes to runtime version events that the
   standalone node closes early. The wallet hangs in 'syncing' state forever
   with no error.
   → fix: Either (a) develop against preprod once past hello-world, or
     (b) upgrade to wallet-sdk-facade@4.0.0 which reverted to constructor +
     .start() pattern.
✓ node: midnightntwrk/midnight-node:0.21.0
✓ indexer: midnightntwrk/indexer-standalone:4.0.0-rc.4
✓ proof-server: midnightntwrk/proof-server:7.0.0
✓ midnight-node:0.21.0 matches SDK track

summary: 4 ok  3 warn  0 error  0 info
Status: workable, but warnings deserve a look.

The 6-hour incident from March is now a 30-second output.

Architecture

The tool is structurally trivial. Three scanners, one diagnosis engine, one report formatter. Total surface:

midnight-doctor/
├── bin/midnight-doctor.js   # CLI entry, arg parsing, exit codes
├── lib/
│   ├── scan-package.js      # reads package.json + walks node_modules
│   ├── scan-docker.js       # runs `docker ps`, parses image tags
│   ├── scan-config.js       # reads .npmrc, indexer.yml
│   ├── diagnose.js          # cross-references findings with matrix
│   ├── report.js            # ANSI-colored pretty output
│   └── index.js             # public API
├── data/compatibility-matrix.json   # the source of truth
└── test/diagnose.test.js    # node:test, no framework

The compatibility matrix is the product

The ~700 lines of code are scaffolding. The actual value is the JSON document at data/compatibility-matrix.json. It encodes three things:

Tracks — known-good combinations of SDK + Docker images, labeled (current, preprod-3x, preprod-1x)
Known issues — specific bugs keyed by package version range or config pattern
Cross-cutting checks — rules that fire when two scanners disagree (e.g., node tag vs SDK track)

A snippet:

{
  "tracks": {
    "current": {
      "label": "Current (latest npm + preprod compatible)",
      "node": "0.21.0",
      "indexer": "4.0.0-rc.4",
      "proofServer": "7.0.0",
      "packages": {
        "@midnight-ntwrk/wallet-sdk": "1.0.0",
        "@midnight-ntwrk/wallet-sdk-facade": "4.0.0",
        "@midnight-ntwrk/compact-runtime": "0.15.0",
        ...
      }
    }
  },
  "knownIssues": [
    {
      "id": "facade-2x-init-bug",
      "severity": "warn",
      "match": { "type": "package-version",
                 "package": "@midnight-ntwrk/wallet-sdk-facade",
                 "range": "2.x" },
      "title": "WalletFacade.init() in 2.x stalls on standalone dev nodes",
      "fix": "Either develop against preprod, or upgrade to facade 4.0.0..."
    }
  ]
}

This file is the same information that lives, scattered, across Discord pinned messages and individual developers' heads. Centralizing it in a JSON document accomplishes three things:

Source of truth — there's now a place to point people instead of "search the channel"
Versioned — the file has a verifiedAt: "2026-04-27" field; staleness is detectable
Executable — code can act on it; humans can read it

The scanners are deliberately dumb

Each scanner is one file, one responsibility, ~50 lines.

// lib/scan-docker.js (excerpt)
export async function scanDocker() {
  const findings = { dockerAvailable: false, containers: {} };
  try {
    await exec('docker', ['version', '--format', '{{.Server.Version}}']);
    findings.dockerAvailable = true;
  } catch { return findings; }

  const { stdout } = await exec('docker', [
    'ps', '--format', '{{.Image}}|{{.Names}}|{{.Status}}',
  ]);

  for (const line of stdout.split('\n').filter(Boolean)) {
    const [image, name, status] = line.split('|');
    const [imageName, tag = 'latest'] = image.split(':');
    if (KNOWN_IMAGES[imageName]) {
      findings.containers[KNOWN_IMAGES[imageName]] = { image: imageName, tag, name, status };
    }
  }
  return findings;
}

No Docker SDK dependency, no parsing library — just child_process.execFile and String.prototype.split. If Docker isn't installed, the scanner returns { dockerAvailable: false } and the diagnosis engine emits an info diagnostic instead of crashing.

The package scanner is similar — walks node_modules recursively (capped at 5000 directories so monorepos don't hang), records every @midnight-ntwrk/* it finds and what version. Duplicates fall out as a side effect: if the same package name has more than one version in the array, it's a duplicate.

const allInstances = await findAllInstances(projectDir, MIDNIGHT_NS);
for (const [name, versions] of Object.entries(allInstances)) {
  const unique = [...new Set(versions)];
  if (unique.length > 1) findings.duplicates[name] = unique;
}

Three lines. That's the entire duplicate-detection logic. The remaining ~50 lines of the scanner are filesystem traversal — boring, mechanical, but it works on every Node project regardless of package manager (npm, pnpm, yarn, bun all create node_modules directories with the same shape).

The diagnosis engine is glorified table joins

lib/diagnose.js takes the three scan results plus the matrix and emits diagnostics. The pattern repeats per check:

function diagnoseCrossCutting(pkg, docker, matrix) {
  const facadeVersion = pkg.installed['@midnight-ntwrk/wallet-sdk-facade'];
  const nodeContainer = docker.containers.node;
  if (!facadeVersion || !nodeContainer) return [];

  const track = matrix.tracks[detectTrack(facadeVersion, matrix)];
  if (track.node && track.node !== nodeContainer.tag) {
    return [{
      id: 'node-track-mismatch',
      severity: 'error',
      title: `midnight-node:${nodeContainer.tag} doesn't match SDK track (expects ${track.node})`,
      detail: `Mixing causes silent sync failures with no error.`,
      fix: `Either update the node container to ${track.node}, or align SDK to a track that matches your node version.`,
    }];
  }
  return [{ id: 'node-track-match', severity: 'ok',
            title: `midnight-node:${nodeContainer.tag} matches SDK track` }];
}

That's the entire body of the cross-cut check that would have saved my 6-hour March incident. Twelve lines. The information was there the whole time.

The full diagnosis file is ~200 lines. Most of it is the same shape: pick two facts from the scans, compare against the matrix, emit a diagnostic. It's deliberately boring code.

What this isn't

Scope discipline is half the value of a tool like this. Things midnight-doctor deliberately doesn't do:

It doesn't auto-fix. No --fix flag yet. The cost of a wrong auto-fix in this domain (bricked node_modules, lost work) outweighs the convenience.
It doesn't curl health endpoints. It checks Docker container presence, not health. A separate health-check script (midnight-health-check.sh) covers that already in my infra repo.
It doesn't probe the chain. No getBlockNumber(), no balance lookup. Those depend on a working SDK, which is what doctor is meant to validate before you try to use it.
It isn't a package manager. It won't run npm dedupe for you. It tells you to run it.
It doesn't write code. It doesn't generate scaffolds, codemods, or migrations. That's a different tool (create-midnight-app, eventually).

Each non-feature is a deliberate choice to keep the surface small enough that the tool stays correct.

What I'd build next, if there's appetite

midnight-doctor is the smallest useful version. The compatibility matrix as data unlocks several adjacent tools:

create-midnight-app — scaffold a project where midnight-doctor is wired into npm install lifecycle. The matrix becomes the default versions.
A codemod for the 2.x → 4.x facade migration. The API change is mechanical: WalletFacade.init({...}) → new WalletFacade(...) + .start(...). AST transform.
Compactc compiler version check. Today it's via shell-out. With a parser, doctor could read your .compact source and say "this uses syntax X, requires compactc ≥ Y."
Remote-fetched matrix. Right now the matrix is bundled. A nightly job at https://compatibility.midnight.network/matrix.json would let users not need to bump the doctor itself.
Editor integration. A VSCode extension that runs doctor on save and surfaces diagnostics in the problems panel. The CLI is for CI; the editor is for dev loop.

The first one is the highest leverage by far — it short-circuits the entire onboarding problem. But it's a much bigger project. Doctor is a Trojan horse for the matrix; once the matrix exists and people accept it as canonical, the rest is plumbing.

The broader argument: tools beat docs

Documentation rots. A README written today describes a system that, six weeks from now, has moved two majors. Anyone who builds in a fast-moving ecosystem has lived this: the official doc says "use SDK 1.0", the npm latest tag says 4.0, the GitHub README says 3.0, and the Discord pinned message says "we know, sorry, the docs are stale." None of those sources is wrong. They were all true at some point. The problem is they don't update each other.

Executable knowledge updates itself. A JSON file with a verifiedAt date that's three weeks old is visibly stale. A doc that's three weeks old looks identical to a doc that's three days old. Tools force the question "is this still true?" in a way that prose doesn't.

This isn't unique to Midnight. Every fast-moving stack reaches the point where the gap between "what the docs claim" and "what actually works" is wide enough to swallow new developers. The fix is the same: take the lookup table out of human heads, encode it as data, ship it as a tool that runs in seconds.

The Midnight protocol team doesn't need to build midnight-doctor. They could. Anyone could. The information already exists — in pinned messages, in CHANGELOG.md files, in the heads of the half-dozen people on Discord who answer the same question every week. The work isn't producing the information. It's transcribing it once, into a format machines can act on, and committing to keeping it current.

That's what this ~700-line tool does. It's not clever. It's not hard. It's just nobody had done it.

If you've spent hours on a Midnight silent failure, please run npx midnight-doctor against your project before your next debugging session. If you find a bug doctor missed, open an issue — the matrix is the artifact, the code is just glue.

Repo: github.com/fredericosanntana/midnight-doctor
Install: npm install -g midnight-doctor or npx midnight-doctor
License: MIT

If this was useful, the people who actually maintain Midnight (@MidnightNtwrk) deserve the visibility — half this matrix came from their Discord answers. The other half came from getting burned. Both contributions are essential.

Appendix: the full check list

For reference, every check midnight-doctor currently runs:

Package scanner:

✓ Detect SDK track (current, preprod-3x, preprod-1x) from wallet-sdk-facade major
✓ Flag deprecated tracks with migration guidance
✓ Detect major-version mismatch across wallet-sdk-* subpackages
✓ Detect duplicate @midnight-ntwrk/ledger-v7 in node_modules
✓ Detect duplicate @midnight-ntwrk/compact-runtime in node_modules
✓ Flag wallet-sdk-facade@2.x standalone init bug

Docker scanner:

✓ Detect running midnight-node, indexer-standalone, proof-server
✓ Cross-reference node tag with SDK track

Config scanner:

✓ Flag .npmrc with bogus npm.midnight.network registry
✓ Flag indexer.yml missing subscription: block

Cross-cutting:

✓ Node tag ↔ SDK track consistency

Roadmap:

☐ Compact compiler version vs runtime
☐ Health probes (RPC, GraphQL, proof endpoints)
☐ Monorepo workspace iteration
☐ --fix mode for safe auto-corrections
☐ Remote-fetched matrix

Total checks today: 11. Adding more is one PR each.

DEV Community

Surviving Midnight SDK: a 700-line cure for the silent failure problem

The 6-hour silent failure

The pattern: silent failures from missing cross-references

The tool: `midnight-doctor`

Architecture

The compatibility matrix is the product

The scanners are deliberately dumb

The diagnosis engine is glorified table joins

What this isn't

What I'd build next, if there's appetite

The broader argument: tools beat docs

Appendix: the full check list

Top comments (0)

The 6-hour silent failure

The pattern: silent failures from missing cross-references

The tool: midnight-doctor

Architecture

The compatibility matrix is the product

The scanners are deliberately dumb

The diagnosis engine is glorified table joins

What this isn't

What I'd build next, if there's appetite

The broader argument: tools beat docs

Appendix: the full check list

The tool: `midnight-doctor`