DEV Community

Tosh
Tosh

Posted on

When Proofs Fail: Debugging Midnight Proof Server Errors

#blockchain #web3 #tutorial #debugging

When Proofs Fail: Debugging Midnight Proof Server Errors

Zero-knowledge proofs are at the heart of Midnight Network — every private transaction, every shielded contract call, every confidential state update requires a valid ZK proof. Generating these proofs is handled by the proof server, a critical component of the Midnight developer stack. When the proof server encounters problems, you see cryptic errors that can block your entire development workflow.

This guide demystifies proof server errors: what the proof server actually does, the categories of failures you'll encounter, and a systematic debugging workflow to get you unstuck fast.

What the Proof Server Does

Before debugging, it's important to understand the proof server's role in the Midnight architecture.

When you call a Compact circuit function — say, increment() on your counter contract — the execution path is:

  1. Your DApp/wallet constructs a transaction with the circuit call
  2. The proof server receives the circuit execution trace (the "witness") and generates a zero-knowledge proof that the computation was performed correctly, without revealing private inputs
  3. The proof is attached to the transaction
  4. The Midnight node verifies the proof against the on-chain verifier key before accepting the transaction into a block

The proof server is computationally intensive. Groth16-style proofs (commonly used in ZK systems like Midnight's) can take seconds to minutes depending on circuit complexity, hardware, and proof server load. This is why errors here are especially frustrating — you may wait 30 seconds for a proof attempt before receiving the failure.

The proof server typically runs:

  • Locally as part of npx @midnight-ntwrk/midnight-js-cli during development
  • As a separate Docker container in more advanced setups
  • Via a remote endpoint on testnet (provided by the Midnight team)

Common Proof Server Error Types

1. Witness Construction Errors

Symptom: Error occurs before proof generation even begins; often thrown by the Compact runtime or your TypeScript DApp code.

Examples:

Error: Witness construction failed: missing input for circuit parameter 'secret_value'
TypeError: Cannot read properties of undefined (reading 'value')
CompactError: Type mismatch in witness: expected Uint<64>, received string
Enter fullscreen mode Exit fullscreen mode

Root cause: The JavaScript/TypeScript code constructing the witness object for the circuit call has a bug — a missing parameter, wrong type, or undefined variable.

Fix: Carefully compare your witness construction code against the Compact circuit signature. Every circuit parameter marked as a "secret" input must be provided in the witness. Check your TypeScript for type coercions (e.g., passing a raw number where a bigint is expected).

2. Constraint Violation Errors

Symptom: Proof generation starts but fails partway through with a constraint or assertion message.

Examples:

ProofError: Constraint violation at circuit 'increment': assertion failed
ProofError: R1CS constraint unsatisfied: expected output does not match computed value
Enter fullscreen mode Exit fullscreen mode

Root cause: The circuit's internal assertions (assert statements in Compact) are not satisfied by the provided witness values. This is the ZK equivalent of a runtime assertion failure.

Fix:

  • Review every assert statement in your Compact circuit
  • Add logging in your TypeScript code to print the witness values before submitting to the proof server
  • Check that preconditions are met (e.g., if your circuit asserts counter < new_value, ensure you're not trying to set the counter to a value lower than its current state)

3. Verifier Key Mismatch Errors

Symptom: Proof generates successfully but the Midnight node rejects the transaction.

Examples:

TransactionError: Verifier key mismatch — proof was generated for an outdated circuit version
NodeRejection: Invalid proof: verifier key not found for contract address
Enter fullscreen mode Exit fullscreen mode

Root cause: The proof was generated using a different version of the compiled circuit (.compact bytecode) than what's deployed on-chain. This happens when:

  • You modified the Compact contract and recompiled without redeploying
  • You're testing against a different network than the one the proof server has keyed circuit parameters for
  • The proof server's proving keys are stale after a Midnight SDK upgrade

Fix:

# 1. Recompile your Compact contract
npx compactc src/yourContract.compact

# 2. Redeploy to get a fresh contract address with updated verifier key
# 3. Update your DApp to use the new contract address
# 4. Restart the proof server to clear cached proving keys
Enter fullscreen mode Exit fullscreen mode

4. Proof Server Connection Errors

Symptom: Network-level failures when your DApp tries to reach the proof server.

Examples:

Error: connect ECONNREFUSED 127.0.0.1:6300
FetchError: request to http://localhost:6300/prove failed, reason: connect ECONNREFUSED
ProofServerError: Timeout after 60000ms waiting for proof
Enter fullscreen mode Exit fullscreen mode

Root cause: The proof server process isn't running, is listening on a different port, or is unresponsive.

Fix:

# Check if proof server is running
lsof -i :6300   # default proof server port

# If not running, start it via the Midnight CLI
npx @midnight-ntwrk/midnight-js-cli

# Or if running in Docker
docker ps | grep proof
docker start <proof-server-container-id>
Enter fullscreen mode Exit fullscreen mode

Verify your DApp's proof server URL configuration:

// In your Midnight provider configuration
const proofServerConfig = {
  proverServerUri: 'http://localhost:6300',  // must match running proof server
};
Enter fullscreen mode Exit fullscreen mode

5. Out of Memory / Resource Exhaustion

Symptom: Proof server crashes or returns an error after a long wait.

Examples:

Error: JavaScript heap out of memory
Proof server exited with code 137 (OOM killed)
SIGKILL received during proof generation
Enter fullscreen mode Exit fullscreen mode

Root cause: Complex circuits with many constraints require significant RAM for proof generation. Midnight's Groth16 proofs can use 2–8 GB of RAM for non-trivial circuits.

Fix:

  • Increase Node.js heap size: 
  NODE_OPTIONS="--max-old-space-size=8192" npx @midnight-ntwrk/midnight-js-cli
Enter fullscreen mode Exit fullscreen mode
  • Simplify your circuit if possible — reduce the number of private inputs or break the circuit into smaller operations
  • Use a machine with more RAM (16 GB+ recommended for active proof server development)

6. SDK Version Mismatch

Symptom: Proof server starts but immediately errors on circuit load, or proof generation fails with serialization errors.

Examples:

Error: Circuit serialization failed: incompatible circuit format version
TypeError: protobufs mismatch — upgrade @midnight-ntwrk/compact-runtime
Enter fullscreen mode Exit fullscreen mode

Root cause: The proof server, compact runtime, and compiled circuit bytecode must all be on compatible versions. Mixing SDK versions causes binary incompatibilities.

Fix:

# Check current versions
npm list @midnight-ntwrk/compact-runtime
npm list @midnight-ntwrk/midnight-js-sdk

# Update all Midnight packages together
npm update @midnight-ntwrk/compact-runtime @midnight-ntwrk/midnight-js-sdk

# Recompile circuits after updating
npx compactc src/*.compact
Enter fullscreen mode Exit fullscreen mode

Step-by-Step Debugging Workflow

When a proof fails, follow this systematic process:

Step 1: Isolate the Failure Stage

Proof errors occur at one of three stages:

  1. Witness construction (your TypeScript code, before hitting the proof server)
  2. Proof generation (inside the proof server process)
  3. On-chain verification (after the proof is submitted to the node)

Read the error message carefully. "ECONNREFUSED" is Stage 2 (server not running). "Verifier key mismatch" is Stage 3. "Constraint violation" is Stage 2 (proof attempted but circuit assertion failed).

Step 2: Check Proof Server Health 

# Is the proof server process running?
ps aux | grep midnight

# Is it listening on the expected port?
curl http://localhost:6300/health
# Expected: {"status": "ok"} or similar
Enter fullscreen mode Exit fullscreen mode

Step 3: Enable Verbose Logging

Add debug logging to your proof submission code:

import { createLogger } from '@midnight-ntwrk/midnight-js-sdk';

const logger = createLogger({ level: 'debug' });
// Pass logger to your Midnight provider configuration
Enter fullscreen mode Exit fullscreen mode

The proof server logs will surface the exact circuit, witness values (be careful with secrets!), and constraint trace when a proof fails.

Step 4: Validate Your Witness Manually

Before sending to the proof server, log your witness object:

const witness = {
  secret_value: mySecretValue,
  counter_increment: incrementAmount,
};
console.log('Witness:', JSON.stringify(witness, (_, v) => 
  typeof v === 'bigint' ? v.toString() : v
));
// Proceed to proof generation
Enter fullscreen mode Exit fullscreen mode

Confirm every field exists and has the correct type (bigint for numeric Compact types, Uint8Array for bytes).

Step 5: Check Network Synchronization

Proof failures can cascade from wallet sync issues. If your wallet's view of the ledger state is stale, the witness you construct may be based on outdated state — causing constraint violations when the proof server uses the current state.

# Check your Midnight node sync status
curl http://localhost:11974/api/v1/status

# Or in Midnight Lace wallet
# Settings → Network → should show current block height matching the explorer
Enter fullscreen mode Exit fullscreen mode

If the node is out of sync, wait for it to catch up before attempting proof generation.

Step 6: Restart the Proof Server Clean

If logs show the proof server is in a bad state:

# Kill the running instance
pkill -f "midnight.*proof"

# Clear any cached artifacts (proof server may cache proving keys)
rm -rf ~/.midnight/proof-cache/   # path may vary

# Restart
npx @midnight-ntwrk/midnight-js-cli
Enter fullscreen mode Exit fullscreen mode

Environment Configuration Issues

Wrong Proof Server URL

Your DApp configuration must point to the proof server endpoint. This differs per environment:

Environment Default Proof Server URL
Local DevNet http://localhost:6300
TestNet Provided in Midnight testnet documentation
MainNet Via wallet provider (Midnight Lace handles this)

Firewall / Port Blocking

If running the proof server in Docker or a VM, ensure port 6300 (or your configured port) is not blocked by a firewall:

# macOS
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --getblockall

# Linux
sudo ufw status
sudo ufw allow 6300/tcp
Enter fullscreen mode Exit fullscreen mode

Docker Networking

If your DApp and proof server run in separate Docker containers:

# docker-compose.yml
services:
  proof-server:
    image: midnight/proof-server:latest
    ports:
      - "6300:6300"

  dapp:
    environment:
      - PROOF_SERVER_URL=http://proof-server:6300  # use service name, not localhost
Enter fullscreen mode Exit fullscreen mode

Wallet Sync Issues Cascading into Proof Failures

One of the most subtle failure modes: wallet state desynchronization causing seemingly unrelated proof errors.

The chain of events:

  1. Wallet syncs to block N
  2. Network advances to block N+50
  3. You construct a witness based on stale ledger state from block N
  4. Proof server generates a proof based on that stale witness
  5. Node rejects transaction because proof references outdated commitments

Detection:

  • Check the block height shown in your wallet against a block explorer
  • If there's a gap of more than a few blocks, the wallet needs to sync

Fix:

  • Allow wallet to fully sync before submitting transactions
  • Implement retry logic in your DApp that re-fetches ledger state before each proof attempt
  • For development, use --fresh-sync flag when restarting the Midnight CLI to force a full re-sync

Diagnostic Checklist

Use this checklist when any proof fails:

Environment

  • [ ] Proof server process is running (ps aux | grep midnight)
  • [ ] Proof server is reachable (curl http://localhost:6300/health)
  • [ ] Correct proof server URL configured in DApp
  • [ ] Firewall is not blocking the proof server port
  • [ ] Sufficient RAM available (8+ GB free recommended)

SDK Versions

  • [ ] @midnight-ntwrk/compact-runtime version is consistent across project
  • [ ] Compact compiler version matches SDK version
  • [ ] Node.js version is 18+ (node --version)

Circuit and Witness

  • [ ] Compact contract compiled successfully with compactc
  • [ ] Witness object contains all required parameters
  • [ ] All numeric witness values are bigint, not number
  • [ ] All assert statements in circuits will be satisfied by provided values

Network State

  • [ ] Wallet is fully synced to current block height
  • [ ] Targeting the correct network (devnet / testnet / mainnet)
  • [ ] Contract address is current (not an outdated deployment)
  • [ ] Ledger state fetched immediately before proof construction

After Redeployment

  • [ ] Proof server restarted to clear cached proving keys
  • [ ] DApp updated with new contract address
  • [ ] Test with a minimal witness before running full integration tests

Summary

Proof server errors in Midnight development cluster into predictable categories: witness construction bugs, circuit constraint violations, verifier key mismatches, connectivity failures, resource exhaustion, and SDK version mismatches. Each has distinct symptoms and targeted fixes.

The most effective debugging strategy is systematic: isolate which stage the failure occurs at, check the proof server health, validate your witness values, confirm network sync, and use verbose logging to surface the specific constraint or serialization issue.

ZK development has a steeper debugging curve than conventional smart contracts precisely because failures are mathematical rather than logical — but with the right workflow, proof server errors go from mysterious to manageable.

For additional help, the Midnight Discord #developer-support channel has active community members and IOG engineers who can help diagnose unusual proof failures. Always include the full error message, your Compact circuit, and the SDK version when asking for help.

Top comments (0)