DEV Community

cobra
cobra

Posted on

When proofs fail on Midnight: Debugging proof server errors and ZK generation failures

#backend #blockchain #tutorial #web3

This guide shows how to diagnose and fix four common proof-generation failures on Midnight: a proof server that is not responding, a slow first proof or client timeout, proof rejection caused by a wire format mismatch, and version mismatches between the proof server and the ledger stack. Each command and log example in this guide was tested in a local environment.

Use this guide when your Midnight DApp cannot reach the proof server, proof generation times out, /check or /prove rejects a request, or proof generation starts failing after a version change.

What this guide covers

  • Proof server liveness and version verification
  • Proof server not responding
  • Slow first proof or timeout
  • Wire format mismatch on proof server requests
  • Version mismatch between the proof server and the local ledger stack

Tested environment

Note: The examples below use focused local reproductions rather than a single DApp flow. The timeout reproduction uses a local helper script, k16-direct-prove-timeout.mjs, with the compiled transferOwnership circuit from OpenZeppelin’s test-only MockZOwnablePK.compact. The helper generates a valid serialized preimage before calling /prove. The script and compiled artifacts are not included with this article, so the commands show the exact tested invocation but are not copy-paste-ready on their own. The version-mismatch section uses a local Hello World DApp flow. The wire-format section uses saved /check payloads so request encoding can be tested in isolation. Use the same diagnostic pattern in your own Midnight DApp.

Item Value
Verified as of April 16, 2026
OS macOS 15.6
CPU arm64
Docker 29.1.2
Node.js 22.21.1
npm 10.9.4
Compact 0.5.1
Compact compile 0.30.0
Proof server image midnightntwrk/proof-server:8.0.3

Prerequisites

Before you start, make sure you have:

  • Docker Desktop running
  • Node.js installed
  • Compact installed
  • A local Midnight project or DApp flow that triggers proof generation
  • Terminal access
  • The proof server image version that matches your local stack

Start from a healthy proof server

Start with one known-good proof server. Do this before changing versions, editing payloads, or tuning timeouts.

Run the standalone proof server on port 6300:

docker run --rm --name midnight-proof-server-319-k16-cold -p 6300:6300 midnightntwrk/proof-server:8.0.3 -- midnight-proof-server -v
Enter fullscreen mode Exit fullscreen mode

In another terminal, use /version as the proof server liveness and version check. The proof server does not expose a dedicated /health route, so a successful /version response confirms that the service is reachable and reports the running version:

curl -i http://localhost:6300/version
Enter fullscreen mode Exit fullscreen mode

A healthy proof server returns HTTP/1.1 200 OK and the version string:

HTTP/1.1 200 OK
content-type: text/plain; charset=utf-8

8.0.3
Enter fullscreen mode Exit fullscreen mode

You can also inspect the container and recent logs:

docker ps --filter "name=midnight-proof-server"
docker logs --tail 20 midnight-proof-server-319-k16-cold
Enter fullscreen mode Exit fullscreen mode

A healthy startup eventually shows the server listening on 0.0.0.0:6300:

starting service: "actix-web-service-0.0.0.0:6300", workers: 10, listening on: 0.0.0.0:6300
Enter fullscreen mode Exit fullscreen mode

Fix a proof server that is not responding

Symptom

The standalone proof server on the default local endpoint is not responding. Start by checking the proof server container state, preserved logs, and the /version liveness check on port 6300. For this test, the endpoint was http://localhost:6300/version.

What to check

Check four things:

  1. Is the proof server container running?
  2. Do the startup logs show a healthy startup state?
  3. Does http://localhost:6300/version respond?
  4. After stopping the container, does Docker show that the container exited while /version stops responding?

Commands

For this test, I used a named standalone container without --rm so Docker state and logs would still be available after the container was stopped.

Start the proof server:

docker run -d --name midnight-proof-server-319-a -p 6300:6300 midnightntwrk/proof-server:8.0.3 -- midnight-proof-server -v
Enter fullscreen mode Exit fullscreen mode

Then verify the healthy state:

docker ps --filter "name=midnight-proof-server-319-a"
docker logs midnight-proof-server-319-a
curl -i http://localhost:6300/version
Enter fullscreen mode Exit fullscreen mode

In this test, docker ps showed the container running on 0.0.0.0:6300->6300/tcp, the startup logs included Actix runtime found; starting in Actix runtime and listening on: 0.0.0.0:6300, and the /version check returned HTTP/1.1 200 OK with body 8.0.3. That established a known-good baseline before reproducing the failure.

Now stop that same standalone container and diagnose the result with Docker-based checks:

docker stop midnight-proof-server-319-a
docker ps -a --filter "name=midnight-proof-server-319-a"
docker inspect midnight-proof-server-319-a
docker logs midnight-proof-server-319-a
curl -i http://localhost:6300/version
Enter fullscreen mode Exit fullscreen mode

What the logs show

This test produced a real non-responding state, and the strongest evidence came from Docker rather than from curl alone:

  • docker ps -a showed the container had exited.
  • docker inspect showed "Status": "exited" and "Running": false.
  • docker logs were still preserved because the container had been created without --rm.
  • The /version endpoint no longer responded on http://localhost:6300/version.

The preserved logs are also useful because they show what happened at shutdown. They do not show a crash or an application exception. Instead, they show a clean stop sequence beginning with SIGTERM received; starting graceful shutdown, followed by worker shutdown lines and accept thread stopped. After that, curl -i http://localhost:6300/version failed with curl: (7) Failed to connect to localhost port 6300.

Root cause

The most conservative, evidence-backed root cause is that the standalone container process had stopped, so nothing was listening on port 6300.

Fix

Because this test used a named container without --rm, the exact tested recovery command was:

docker start midnight-proof-server-319-a
Enter fullscreen mode Exit fullscreen mode

Verify the fix

After restarting the same container, verify /version again:

curl -i http://localhost:6300/version
Enter fullscreen mode Exit fullscreen mode

In this test, that final check returned HTTP/1.1 200 OK with body 8.0.3, confirming that the proof server was reachable again on the standard local endpoint.

Diagnose a slow first proof or timeout

Symptom

The first proof request is slow or fails with an abort-style timeout. On a cold proof server, the server may need to download or initialize ZK parameter and key material before the proof can finish. In this test, the logs showed cold parameter and key downloads, a real /prove timeout, and success after increasing the timeout.

What to check

Check three things:

  1. Did the proof flow reach /prove?
  2. Did the proof server start cold and fetch missing parameter or key material?
  3. Does the same proof succeed with a higher timeout?

Commands

The commands below are the exact commands used in the tested reproduction. They depend on the local helper script and compiled circuit assets described above, so treat them as illustrative unless you have equivalent local assets.

Start from a cold proof server and watch its logs:

docker logs --tail 80 midnight-proof-server-319-k16-cold
Enter fullscreen mode Exit fullscreen mode

Run the direct proof path with a low timeout:

K16_PROVE_TIMEOUT_MS=1000 node k16-direct-prove-timeout.mjs
Enter fullscreen mode Exit fullscreen mode

Then run the same direct proof path with a higher timeout:

K16_PROVE_TIMEOUT_MS=300000 node k16-direct-prove-timeout.mjs
Enter fullscreen mode Exit fullscreen mode

What the logs show

The cold proof server fetched missing public parameters and key material:

Ensuring zswap key material is available...
Missing public parameters for k=10. Attempting to download from the host ...
Missing public parameters for k=11. Attempting to download from the host ...
Missing public parameters for k=12. Attempting to download from the host ...
Missing zero-knowledge proving key for Zswap inputs. Attempting to download from the host ...
Missing zero-knowledge proving key for Dust spends. Attempting to download from the host ...
Enter fullscreen mode Exit fullscreen mode

The low-timeout proof attempt reached the proof path and then aborted:

PREIMAGE_OK
serializedPreimage.length=323
publicTranscript.length=36
privateTranscriptOutputs.length=2
PROVE_ATTEMPT_START
proofServer=http://localhost:6300
circuitId=transferOwnership
timeoutMs=1000
PROVE_ERROR
error.name=AbortError
error.message=The user aborted a request.
Enter fullscreen mode Exit fullscreen mode

The proof server also showed the /prove request:

POST /prove HTTP/1.1; took 5.111326s
Enter fullscreen mode Exit fullscreen mode

The higher-timeout run succeeded:

PROVE_ATTEMPT_START
proofServer=http://localhost:6300
circuitId=transferOwnership
timeoutMs=300000
PROVE_OK
proof.length=4508
Enter fullscreen mode Exit fullscreen mode

Root cause

The client timeout was too low for the first proof path. The proof server was alive, but the client aborted before proof generation finished.

Fix

Increase the proof provider timeout for first-run proof generation. Keep the timeout high enough for cold parameter and key initialization.

Verify the fix

Run the same proof path again with the higher timeout:

K16_PROVE_TIMEOUT_MS=300000 node k16-direct-prove-timeout.mjs
Enter fullscreen mode Exit fullscreen mode

The proof succeeds when the timeout is high enough:

PROVE_OK
proof.length=4508
Enter fullscreen mode Exit fullscreen mode

Do not treat every timeout as a DUST or wallet problem. In this test, the valid timeout evidence used a direct /prove path and avoided an earlier wallet/deploy path that failed before proof generation with insufficient DUST.

Fix proof rejection caused by a wire format mismatch

Symptom

The proof server rejects a request even though the server is running. This can happen when the request body does not match the serialized wire format expected by /check or /prove.

What to check

Check whether your client sends the exact serialized payload expected by the proof server. Do not hand-edit, truncate, stringify, or re-encode the binary request body.

The examples below use ./evidence as the evidence folder. Replace it with your own folder if you store test files somewhere else.

Commands

Send a known-good /check payload:

curl -sS \
  -D ./evidence/05-good-check.headers.txt \
  -o ./evidence/05-good-check.body.bin \
  -H 'Content-Type: application/octet-stream' \
  --data-binary @./evidence/05-wire-format-good-check.bin \
  http://127.0.0.1:6300/check
Enter fullscreen mode Exit fullscreen mode

Then send a truncated copy of that payload:

curl -sS \
  -D ./evidence/05-bad-check.headers.txt \
  -o ./evidence/05-bad-check.body.bin \
  -H 'Content-Type: application/octet-stream' \
  --data-binary @./evidence/05-wire-format-bad-check-truncated.bin \
  http://127.0.0.1:6300/check
Enter fullscreen mode Exit fullscreen mode

What the logs show

The good payload returned 200 OK:

GOOD_HEADERS
HTTP/1.1 200 OK
content-length: 30
Enter fullscreen mode Exit fullscreen mode

The truncated payload returned 400 Bad Request:

BAD_HEADERS
HTTP/1.1 400 Bad Request
content-length: 27

BAD_BODY_PREVIEW
failed to fill whole buffer
Enter fullscreen mode Exit fullscreen mode

The proof server logged the parse failure:

Error in response: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }
POST /check HTTP/1.1; took 0.001360s
Enter fullscreen mode Exit fullscreen mode

Root cause

The request body no longer matched the expected wire format because the payload was truncated.

Fix

Send the original binary payload generated by the Midnight tooling. Keep the content type as application/octet-stream, and use --data-binary when testing with curl.

Verify the fix

Resend the unmodified payload:

curl -sS \
  -D ./evidence/05-fix-good-check.headers.txt \
  -o ./evidence/05-fix-good-check.body.bin \
  -H 'Content-Type: application/octet-stream' \
  --data-binary @./evidence/05-wire-format-good-check.bin \
  http://127.0.0.1:6300/check
Enter fullscreen mode Exit fullscreen mode

The fixed request returns 200 OK:

FIX_GOOD_HEADERS
HTTP/1.1 200 OK
content-length: 30
Enter fullscreen mode Exit fullscreen mode

Fix a version mismatch between the proof server and the ledger stack

Symptom

Your local stack is reachable, but proof generation fails after you change the proof server image or a ledger-related package.

What to check

Compare the proof server Docker tag with the ledger and runtime packages used by your local DApp.

Commands

Inspect the local package versions and proof server image:

npm list @midnight-ntwrk/ledger-v8
npm list @midnight-ntwrk/compact-runtime
npm list @midnight-ntwrk/onchain-runtime-v3
docker ps | grep proof-server
docker inspect --format='{{.Config.Image}}' example-hello-world-proof-server-1
Enter fullscreen mode Exit fullscreen mode

A known-good local stack used @midnight-ntwrk/ledger-v8@8.0.3 with this proof server image:

midnightntwrk/proof-server:8.0.3
Enter fullscreen mode Exit fullscreen mode

To reproduce a mismatch, I changed the proof server image to an older major version:

image: 'midnightntwrk/proof-server:7.0.0'
Enter fullscreen mode Exit fullscreen mode

Then I ran the same local test flow again.

What the logs show

The mismatched proof server accepted traffic, but proof generation failed:

× Hello World Contract > Deploys the contract 301111ms
  → Failed to prove transaction

(FiberFailure) Wallet.Proving: Failed to prove transaction
Enter fullscreen mode Exit fullscreen mode

The proof server logs showed that the failing run reached /prove:

GET /version HTTP/1.1; took 0.008403s
Starting to process request for /prove...
Enter fullscreen mode Exit fullscreen mode

Root cause

The proof server image did not match the local ledger stack. The server was reachable, but the proof path was not compatible with the versions used by the DApp and wallet stack.

Fix

Restore the proof server image to the version that matches the local ledger packages:

image: 'midnightntwrk/proof-server:8.0.3'
Enter fullscreen mode Exit fullscreen mode

Restart the local stack with the same startup command used by your project.

Verify the fix

Run the same local test flow again. The aligned stack should pass:

✓ Hello World Contract > Deploys the contract
✓ Hello World Contract > Stores Hello World!

Test Files  1 passed (1)
Tests       2 passed (2)
Enter fullscreen mode Exit fullscreen mode

One important note: an older patch tag did not fail in this setup. Do not assume every older proof server tag reproduces a mismatch. Use the supported version set for your stack and verify with a real proof-generating flow.

Proof server liveness checklist

Use this checklist before changing DApp code:

  • docker ps shows the proof server container running.
  • curl -i http://localhost:6300/version returns HTTP/1.1 200 OK.
  • The response body includes the proof server version, for example 8.0.3.
  • docker logs shows the service listening on 0.0.0.0:6300.
  • Cold startup logs may show missing public parameters or zero-knowledge proving keys being downloaded and verified.
  • A low timeout can fail even when the proof server is healthy.
  • A Docker health status can be misleading if the container health check depends on a tool that is missing inside the container.
  • The proof server Docker tag matches the ledger version used by the local stack.
  • /check and /prove requests use the correct binary wire format.

Wrap up

When proof generation fails, start with the /version liveness check, then inspect Docker logs, and then identify the exact failure boundary. A connection failure, an abort timeout, a rejected binary payload, and a version mismatch can look similar from a DApp, but they leave different signals in the proof server logs.

Top comments (0)