DEV Community

Cover image for The Complete Guide to Running a Midnight Node: Setup, Sync & Monitoring
Uroy Nwankwo
Uroy Nwankwo

Posted on • Edited on

The Complete Guide to Running a Midnight Node: Setup, Sync & Monitoring

#web3 #tutorial #blockchain #docker

If you've been building on Midnight or want to participate in the network without trusting third-party infrastructure, run your own full node.
This tutorial takes you from a bare server to a healthy, synced node using the official midnight-node-docker repository, the same tooling the Midnight team maintains and ships. Along the way you'll learn how to monitor your sync progress, verify node health, and work through the failure modes that catch most people: dropped peers and getting stuck on block 1.

Target audience: Developers and node operators comfortable with the Linux command line and Docker.

Prerequisites:

  • Ubuntu 22.04 LTS or Ubuntu 24.04 LTS (bare metal or VPS)
  • sudo access
  • A stable internet connection with inbound TCP port 30333 open
  • Outbound HTTPS access

What you'll have by the end: A full node synced to Midnight's Preview Testnet, running through the official Docker Compose stack, with monitoring commands and a tested troubleshooting playbook for common failures.

Understanding the node types

Before spinning anything up, it helps to know what you're actually running. Midnight has four node types:

  • Full node: syncs the chain, validates transactions, and provides real-time state queries. It prunes historical state older than 256 blocks by default, making it efficient on disk. This is what most developers and node operators need.
  • Archive node: identical to a full node but retains the entire chain history. Required for block explorers, historical analytics, or services that query past state at arbitrary heights. Uses substantially more storage.
  • RPC node: exposes an HTTP/WebSocket API for DApps to interact with the chain programmatically.
  • Boot node: helps new nodes discover peers. You don't need to run one unless you're contributing to network bootstrapping.

This tutorial focuses on the full node. Where the archive node diverges, this guide calls out the one flag you need to change.

Step 1: Check resource requirements

Before installing anything, confirm your machine meets the minimums. Running an undersized node is the fastest way to end up stuck on block 1 with peers constantly dropping.

The requirements below come directly from the official Midnight documentation:

Resource Preview Testnet Cardano Mainnet
CPU 4 cores 4 cores
RAM 16 GB 32 GB
Storage (Cardano-db-sync) 40 GB SSD 320 GB SSD
Disk IOPS 30,000+ 60,000+
Network Stable, port 30333 open Stable, port 30333 open

Avoid HDDs and cheap burstable cloud volumes. Low-IOPS storage causes sync stalls, peer disconnections, and PostgreSQL failures under write load. If your node keeps falling behind the chain tip, check disk I/O before anything else:

iostat -x 1 5
Enter fullscreen mode Exit fullscreen mode

Look at the %util column for your storage device. Sustained values above 80% mean your disk is the bottleneck. Migrate to a faster SSD before continuing.

Step 2: Install Docker

The entire Midnight node stack runs in Docker. Install Docker Engine and the Docker Compose plugin using the official Docker repository:

# System packages
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

# Docker GPG key and repository
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

. /etc/os-release
printf '%s\n' \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu ${VERSION_CODENAME} stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
sudo apt-get update
sudo apt-get install -y \
  docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin
Enter fullscreen mode Exit fullscreen mode

Add your user to the Docker group so you don't need sudo for every command, then verify:

sudo usermod -aG docker "$USER"
newgrp docker

docker --version
docker compose version
Enter fullscreen mode Exit fullscreen mode

Check Docker is running:

sudo systemctl status docker
Enter fullscreen mode Exit fullscreen mode

You should see Active: active (running). If not, run sudo systemctl start docker before continuing.

Step 3: Install direnv

The midnight-node-docker repository uses direnv to manage environment variables. When you cd into the project directory, direnv automatically loads the correct configuration for your chosen network: image versions, boot nodes, and Cardano network name. This is the official approach the Midnight team ships with the repository.

sudo apt-get install -y direnv
Enter fullscreen mode Exit fullscreen mode

Add the direnv hook to your shell. For bash:

echo 'eval "$(direnv hook bash)"' >> ~/.bashrc
source ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

For zsh:

echo 'eval "$(direnv hook zsh)"' >> ~/.zshrc
source ~/.zshrc
Enter fullscreen mode Exit fullscreen mode

Step 4: Clone the midnight-node-docker repository

The midnight-node-docker repository is the official Docker Compose setup maintained by the Midnight team. It contains the Cardano infrastructure compose file, environment helpers, and operational scripts that you'll use throughout this tutorial.

git clone https://github.com/midnightntwrk/midnight-node-docker
cd midnight-node-docker
Enter fullscreen mode Exit fullscreen mode

Take a moment to look at the repository structure before proceeding:

midnight-node-docker/
├── compose.yml                  # Midnight full node template from the repo
├── compose-partner-chains.yml   # Cardano node + db-sync + PostgreSQL + Ogmios
├── proof-server.yml             # Optional: local proof server
├── .envrc                       # direnv config, loads the right preset
├── midnight-node.sh             # Run midnight-node commands in the container
├── midnight-shell.sh            # Interactive shell into the node container
├── cardano-cli.sh               # Run cardano-cli inside the Cardano container
├── test.sh                      # Comprehensive health validation script
├── reset-midnight.sh            # Reset Midnight data (preserves Cardano data)
└── check-chain.sh               # Verify sidechain registration on Cardano
Enter fullscreen mode Exit fullscreen mode

In this guide, you will use the repo's Cardano compose file together with a small Midnight compose file you create in Step 5:

What you need Command
Midnight node only docker compose -f compose-midnight-node.yml up -d
Midnight + Cardano infra docker compose -f compose-partner-chains.yml -f compose-midnight-node.yml up -d
Full stack with proof server docker compose -f compose-partner-chains.yml -f compose-midnight-node.yml -f proof-server.yml up -d

For this tutorial, you're running the full stack with Cardano infrastructure, so you'll start the Cardano services first and then start the Midnight node.

Step 5: Configure your environment

Select your network preset

Current Midnight node images use the public environment names directly: Preview, Preprod, and Mainnet. The value you set for CFG_PRESET must match the environment you want to run, because the node image uses that preset to load its built-in chain resources.

As of May 23, 2026, the official compatibility matrix lists these tested node versions:

Environment CFG_PRESET Node image Chain resource used by the image
Preview preview midnightntwrk/midnight-node:0.22.5 res/preview/chain-spec-raw.json
Preprod preprod midnightntwrk/midnight-node:0.22.2 res/preprod/chain-spec-raw.json
Mainnet mainnet midnightntwrk/midnight-node:0.22.1 res/mainnet/chain-spec-raw.json

In general, use midnightntwrk/midnight-node:<VERSION> and take <VERSION> from the compatibility matrix. This tutorial uses Preview, so set CFG_PRESET=preview and pin the image to 0.22.5. If the compatibility matrix changes, replace only the version tag for your chosen environment.

Open .envrc:

nano .envrc
Enter fullscreen mode Exit fullscreen mode

Set the preset near the top of the file:

export CFG_PRESET=preview
Enter fullscreen mode Exit fullscreen mode

Then find the Cardano network line near the bottom:

export CARDANO_NETWORK=preview
Enter fullscreen mode Exit fullscreen mode

Change it to preserve the value from the selected preset file:

export CARDANO_NETWORK="${CARDANO_NETWORK:-preview}"
Enter fullscreen mode Exit fullscreen mode

This keeps the same .envrc usable for Preview, Preprod, or Mainnet without accidentally forcing every run back to Cardano Preview.

Create the Preview preset

Create .envrc.preview:

nano .envrc.preview
Enter fullscreen mode Exit fullscreen mode

Add the Preview values:

export MIDNIGHT_NODE_IMAGE="midnightntwrk/midnight-node:0.22.5"

export BOOTNODES="/dns/bootnode-1.preview.midnight.network/tcp/30333/ws/p2p/12D3KooWK66i7dtGVNSwDh9tTeqov1q6LSdWsRLJvTyzTCaywYgK \
           /dns/bootnode-2.preview.midnight.network/tcp/30333/ws/p2p/12D3KooWHqFfXFwb7WW4jwR8pr4BEf562v5M6c8K3CXAJq4Wx6ym"

export CARDANO_NETWORK=preview
export CARDANO_SECURITY_PARAMETER=432
Enter fullscreen mode Exit fullscreen mode

For Preprod or Mainnet, create the matching preset file and change CFG_PRESET in .envrc to the same environment name.

Preprod:

cat > .envrc.preprod <<'EOF'
export MIDNIGHT_NODE_IMAGE="midnightntwrk/midnight-node:0.22.2"

export BOOTNODES="/dns/bootnode-1.preprod.midnight.network/tcp/30333/ws/p2p/12D3KooWQxxUgq7ndPfAaCFNbAxtcKYxrAzTxDfRGNktF75SxdX5 \
           /dns/bootnode-2.preprod.midnight.network/tcp/30333/ws/p2p/12D3KooWNrUBs22FfmgjqFMa9ZqKED2jnxwsXWw5E4q2XVwN35TJ"

export CARDANO_NETWORK=preprod
export CARDANO_SECURITY_PARAMETER=2160
EOF
Enter fullscreen mode Exit fullscreen mode

Mainnet:

cat > .envrc.mainnet <<'EOF'
export MIDNIGHT_NODE_IMAGE="midnightntwrk/midnight-node:0.22.1"

export BOOTNODES="/dns4/bootnode-whippet-bengal.mainnet.midnight.network/tcp/30333/ws/p2p/12D3KooWMmfho3eEFvcnThAfzc9QfieTc91fdhvByL4a2naRjbr2 \
           /dns4/bootnode-labrador-marten.mainnet.midnight.network/tcp/30333/ws/p2p/12D3KooWK2c9vf4UtrjGB27A8weKd6eiBnRQSfeR4TVmQv9MDdDt \
           /dns4/bootnode-glider-spaniel.bn.midnight.network/tcp/30333/ws/p2p/12D3KooWHif7N1ZPhrB8WxTFqjWTxo2U2a2FtJuSA8XRBdFMss6i \
           /dns4/bootnode-dog-pelican.bn.midnight.network/tcp/30333/ws/p2p/12D3KooWQ2NCZCnqkYKHKsYVUc8amBKAk4jhEMJyYDAJjmSuM4uK"

export CARDANO_NETWORK=mainnet
export CARDANO_SECURITY_PARAMETER=2160
EOF
Enter fullscreen mode Exit fullscreen mode

Do not reuse chain data or Cardano-db-sync data after switching environments. Preview, Preprod, and Mainnet are separate networks.

This guide continues with Preview because the cloned Docker repo already includes Cardano Preview configuration under cardano-config/preview. If you switch to Preprod or Mainnet, make sure the matching Cardano configuration directory exists before starting compose-partner-chains.yml.

Create the Midnight node Compose file

The official full-node docs show a docker run command. For this tutorial, use a Compose file instead so the configuration is explicit and repeatable.

Create compose-midnight-node.yml:

nano compose-midnight-node.yml
Enter fullscreen mode Exit fullscreen mode

Add:

volumes:
  midnight-data: {}

services:
  midnight-node:
    container_name: midnight-node
    restart: unless-stopped
    image: ${MIDNIGHT_NODE_IMAGE}
    ports:
      - "9944:9944"
      - "30333:30333"
      - "9615:9615"
    environment:
      - APPEND_ARGS=${APPEND_ARGS}
      - BOOTNODES=${BOOTNODES}
      - CFG_PRESET=${CFG_PRESET}
      - DB_SYNC_POSTGRES_CONNECTION_STRING=${DB_SYNC_POSTGRES_CONNECTION_STRING}
      - NODE_KEY=${NODE_KEY}
      - CARDANO_SECURITY_PARAMETER=${CARDANO_SECURITY_PARAMETER}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9944/health"]
      interval: 2s
    volumes:
      - ./data:/data
      - midnight-data:/node
Enter fullscreen mode Exit fullscreen mode

This Compose file relies on the chain resources built into the selected node image. For Preview, CFG_PRESET=preview loads res/cfg/preview.toml, which points at res/preview/chain-spec-raw.json.

Configure PostgreSQL credentials

Open compose-partner-chains.yml and follow the official instructions:

nano compose-partner-chains.yml
Enter fullscreen mode Exit fullscreen mode

Make two changes:

  1. Replace all default usernames and passwords with unique values only you know. Do not leave default credentials in a production or publicly reachable environment. The direnv hook in .envrc also auto-generates a random UUID for postgres.password when you run direnv allow, so check the file after to see what was generated.
  2. Comment out the ogmios: service block unless you are a block producer who needs it for block production operations. Find the block and prefix each line with #.

Save the file.

Load the environment

Allow direnv to activate and load all variables into your session:

direnv allow
Enter fullscreen mode Exit fullscreen mode

Direnv will read .envrc, source the preset file (.envrc.preview for this tutorial), generate postgres.password if it doesn't exist, and populate your shell with all necessary variables. Verify the key ones loaded correctly:

echo "Node image : $MIDNIGHT_NODE_IMAGE"
echo "CFG preset : $CFG_PRESET"
echo "Cardano net: $CARDANO_NETWORK"
printf 'Boot nodes : %s\n' "$BOOTNODES"
Enter fullscreen mode Exit fullscreen mode

You should see midnightntwrk/midnight-node:0.22.5, preview, preview, and Preview boot-node hostnames. If any value is blank or points at a different environment, stop and recheck .envrc and .envrc.$CFG_PRESET.

Pull the pinned node image:

docker pull "$MIDNIGHT_NODE_IMAGE"
Enter fullscreen mode Exit fullscreen mode

Confirm the image contains the expected chain resources:

docker run --rm \
  --entrypoint test \
  "$MIDNIGHT_NODE_IMAGE" \
  -f "/res/${CFG_PRESET}/chain-spec-raw.json"
Enter fullscreen mode Exit fullscreen mode

The command should exit without output. Any error means the image version and preset do not match.

Important: Any time you open a new terminal tab or SSH session in this directory, run direnv allow again (or just cd in, direnv fires automatically if configured). Environment variables are per-shell session.

Step 6: Set up Cardano-db-sync and PostgreSQL

Midnight operates as a Cardano Partnerchain. Your Midnight node requires a persistent connection to a Cardano-db-sync instance to track relevant scripts on the Cardano blockchain. This is not optional. Your node will stall near block 1 without it.

Start the Cardano infrastructure 

docker compose -f compose-partner-chains.yml up -d
Enter fullscreen mode Exit fullscreen mode

This starts four services: cardano-node, postgres, cardano-db-sync, and cardano-ogmios (if you left it uncommented). Check that they're all running:

docker compose -f compose-partner-chains.yml ps
Enter fullscreen mode Exit fullscreen mode

The cardano-db-sync service has a health check dependency on postgres. It won't start indexing until PostgreSQL reports healthy. This is handled automatically by the compose file's depends_on: condition: service_healthy directive.

Monitor Cardano-db-sync sync progress

Follow the logs:

docker compose -f compose-partner-chains.yml logs -f cardano-db-sync
Enter fullscreen mode Exit fullscreen mode

To check numeric progress, query PostgreSQL. The compose file names the PostgreSQL container db-sync-postgres:

docker exec db-sync-postgres \
  psql -U postgres -d cexplorer -tAc \
  "SELECT 100 * (
      EXTRACT(EPOCH FROM (MAX(time) AT TIME ZONE 'UTC')) -
      EXTRACT(EPOCH FROM (MIN(time) AT TIME ZONE 'UTC'))
    ) / (
      EXTRACT(EPOCH FROM (NOW() AT TIME ZONE 'UTC')) -
      EXTRACT(EPOCH FROM (MIN(time) AT TIME ZONE 'UTC'))
    ) AS sync_percent
    FROM block;"
Enter fullscreen mode Exit fullscreen mode

Note: Replace postgres with your custom username if you changed it in compose-partner-chains.yml.

You'll see a number between 0 and 100. Cardano-db-sync must reach 100% before your Midnight node can operate correctly. On Preview Testnet this typically takes several hours depending on your hardware and disk IOPS.

If you start the Midnight node before this completes, you'll see this in its logs:

Unable to author block in slot. Failure creating inherent data provider:
'No latest block on chain.' not found.
Possible causes: main chain follower configuration error, db-sync not synced fully,
or data not set on the main chain.
Enter fullscreen mode Exit fullscreen mode

This is the Cardano dependency telling you it's not ready. Wait for sync_percent to reach 100 before proceeding.

Step 7: Start the Midnight full node

With Cardano-db-sync synced, start the Midnight node using the Compose file you created earlier:

docker compose -f compose-midnight-node.yml up -d
Enter fullscreen mode Exit fullscreen mode

compose-midnight-node.yml wires up the image version, boot nodes, chain preset, and database connection string from the environment variables direnv loaded. That is why the preset validation in Step 5 matters: Compose will run whatever values are active in your shell.

Check the container is running:

docker compose -f compose-midnight-node.yml ps
Enter fullscreen mode Exit fullscreen mode

The container name used in this Compose file is midnight-node. You'll use this name with all docker logs and docker exec commands that follow.

Give it 30–60 seconds to initialise, then follow the logs:

docker compose -f compose-midnight-node.yml logs -f midnight-node
Enter fullscreen mode Exit fullscreen mode

Step 8: Monitor the initial sync

Reading the sync log output

During a healthy sync, you'll see lines like this:

2024-05-04 12:00:01 ⚙️  Syncing 45.2 bps, target=#187432 (12 peers), best: #8492 (0x1a2b…c3d4), finalized #8448 (0x5e6f…7a8b)
Enter fullscreen mode Exit fullscreen mode

Here's what each field means:

  • bps: blocks per second. Several dozen during initial sync is healthy. Single-digit bps suggests your disk or CPU is the bottleneck.
  • target=#XXXXXX: the current chain tip your peers are reporting.
  • best: #XXXXXX: your local best block. Should climb steadily.
  • finalized #XXXXXX: the last finalised block, trailing your best by a small number.
  • (N peers): connected peers. You want 5 or more.

When your node finishes syncing, the log line changes from Syncing to Idle:

2024-05-04 14:22:01 💤 Idle (14 peers), best: #187432 (0x1a2b…c3d4), finalized #187428 (0x5e6f…7a8b)
Enter fullscreen mode Exit fullscreen mode

Idle means you're at the tip and waiting for new blocks.

Check sync state via RPC

The Midnight node exposes a JSON-RPC interface on port 9944. Query sync state directly:

curl -s -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"system_syncState","params":[],"id":1}' \
  http://localhost:9944 | jq '.result'
Enter fullscreen mode Exit fullscreen mode

Example output:

{
  "startingBlock": 0,
  "currentBlock": 142560,
  "highestBlock": 187432
}
Enter fullscreen mode Exit fullscreen mode

When currentBlock equals highestBlock, you're synced. You can also check peer connectivity at the same time:

curl -s -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"system_health","params":[],"id":1}' \
  http://localhost:9944 | jq '.result'
Enter fullscreen mode Exit fullscreen mode

A healthy synced node returns:

{
  "peers": 14,
  "isSyncing": false,
  "shouldHavePeers": true
}
Enter fullscreen mode Exit fullscreen mode

isSyncing: false with peers greater than zero means you're at the tip and connected. If peers is persistently 0 or 1, see the troubleshooting section.

Monitor block height

Save this as monitor-height.sh for reuse throughout operations:

cat > monitor-height.sh << 'EOF'
#!/usr/bin/env bash
# Polls local node sync state on a loop.
# Usage: INTERVAL=30 ./monitor-height.sh
set -euo pipefail

RPC="${RPC:-http://localhost:9944}"
INTERVAL="${INTERVAL:-30}"

while true; do
  result=$(curl -fsS \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"system_syncState","params":[],"id":1}' \
    "$RPC" 2>/dev/null | jq -r '
      .result |
      "current=\(.currentBlock)  target=\(.highestBlock)  lag=\(.highestBlock - .currentBlock)"
    ') || result="RPC unreachable"
  printf '%s  %s\n' "$(date -u +"%H:%M:%SZ")" "$result"
  sleep "$INTERVAL"
done
EOF
chmod +x monitor-height.sh
Enter fullscreen mode Exit fullscreen mode

Run it:

./monitor-height.sh
Enter fullscreen mode Exit fullscreen mode

A healthy syncing node shows a rising current and a shrinking lag. A fully synced node shows lag=0 and holds there.

Check Prometheus metrics

The node exposes Prometheus-compatible metrics on port 9615. To spot-check peer count and sync status:

curl -s http://localhost:9615/metrics \
  | grep -E 'substrate_sub_libp2p_peers_count|substrate_block_height'
Enter fullscreen mode Exit fullscreen mode

This is the same endpoint the compose-midnight-node.yml health check polls. You can point any Prometheus-compatible monitoring system at localhost:9615 to set up alerting.

Step 9: Verify your node is synced and healthy

The midnight-node-docker repository ships a comprehensive validation script, test.sh, that checks every layer of the stack in sequence: Docker version, Cardano node, Ogmios, PostgreSQL, db-sync lag, Midnight node peers, sync state, block production, and finality. Run it after initial sync and after every restart:

./test.sh
Enter fullscreen mode Exit fullscreen mode

The script checks, among other things:

  • All containers are running
  • PostgreSQL db-sync lag is within acceptable bounds (< 100 slots)
  • Midnight node has peers
  • currentBlock equals highestBlock (fully synced)
  • P2P port 9944 is reachable
  • Finality lag is within 15 blocks (MAX_FINALITY_LAG threshold in the script)

You can also verify the individual layers manually if test.sh reports a failure and you want to isolate which component is at fault.

All containers running:

docker compose -f compose-partner-chains.yml ps
docker compose -f compose-midnight-node.yml ps
Enter fullscreen mode Exit fullscreen mode

Health endpoint responding:

curl -f http://localhost:9944/health && echo "OK"
Enter fullscreen mode Exit fullscreen mode

Node has peers and is not syncing:

curl -s -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"system_health","params":[],"id":1}' \
  http://localhost:9944 | jq '.result'
Enter fullscreen mode Exit fullscreen mode

P2P port is listening:

ss -ltnp | grep ':30333'
Enter fullscreen mode Exit fullscreen mode

No repeated errors in the last 30 minutes:

docker logs --since 30m midnight-node \
  | grep -Ei 'error|panic|disconnect|database' \
  || echo "No errors found."
Enter fullscreen mode Exit fullscreen mode

A clean bill of health is: all containers running, isSyncing: false, peer count above zero, finality lag below 15 blocks, and no repeated panics or database errors.

Step 10: Troubleshooting common issues

Node is stuck on block 1 (or a very low block number)

This is the most commonly reported issue. There are four causes, in order of likelihood.

1. Cardano-db-sync hasn't finished syncing

Your Midnight node requires a fully synced Cardano-db-sync instance. Run the sync percentage query from Step 6. If it's below 100, wait. The "Unable to author block" log message in your Midnight node is the confirmation of this state.

2. CFG_PRESET or BOOTNODES mismatch

If your CFG_PRESET and boot nodes don't match the same network, the node starts but can never find valid peers. Confirm the preset loaded correctly:

echo "CFG_PRESET=$CFG_PRESET"
echo "MIDNIGHT_NODE_IMAGE=$MIDNIGHT_NODE_IMAGE"
echo "CARDANO_NETWORK=$CARDANO_NETWORK"
printf 'BOOTNODES=%s\n' "$BOOTNODES"
Enter fullscreen mode Exit fullscreen mode

If any of these are blank or wrong, run direnv allow again and check .envrc and .envrc.$CFG_PRESET for errors.

For Preview, you should see CFG_PRESET=preview, CARDANO_NETWORK=preview, a midnightntwrk/midnight-node image tag that matches the compatibility matrix, and boot nodes under preview.midnight.network.

If you corrected values within the same environment, clear the Midnight node data before restarting. This preserves your Cardano-db-sync data, which takes hours to rebuild, and only resets the Midnight node:

docker compose -f compose-midnight-node.yml down
rm -rf ./data
docker volume rm midnight-node-docker_midnight-data
docker compose -f compose-midnight-node.yml up -d
Enter fullscreen mode Exit fullscreen mode

If you switched between Preview, Preprod, and Mainnet, reset the Cardano infrastructure too. Those networks cannot share Cardano-db-sync data.

3. PostgreSQL connection is wrong

A common mistake is using localhost as the PostgreSQL host when the database is running in a Docker container on the same machine. The host must be the container name on the shared Docker network (the compose file uses postgres as the service name). Check your DB_SYNC_POSTGRES_CONNECTION_STRING value:

echo "$DB_SYNC_POSTGRES_CONNECTION_STRING"
Enter fullscreen mode Exit fullscreen mode

Test the connection directly:

docker exec db-sync-postgres \
  psql -U postgres -d cexplorer -c 'SELECT COUNT(*) FROM block;'
Enter fullscreen mode Exit fullscreen mode

If this command fails, fix your PostgreSQL configuration before restarting the Midnight node.

4. Node image is outdated

Node versions update frequently. An outdated image gets rejected by current network peers. Always cross-reference the image tag in .envrc.$CFG_PRESET against the compatibility matrix. Update the tag in the preset file, then pull and restart:

direnv allow
docker compose -f compose-midnight-node.yml pull
docker compose -f compose-midnight-node.yml up -d --force-recreate
Enter fullscreen mode Exit fullscreen mode

Peers keep disconnecting

If peer count climbs to 5 then drops back to 0 repeatedly, work through these checks in order.

1. Port 30333 is not publicly reachable

This is the single most common cause. Even if Docker is forwarding the port locally, your firewall or cloud security group may be blocking inbound connections from external peers.

# Check UFW
sudo ufw status

# Open if missing
sudo ufw allow 30333/tcp
sudo ufw reload
Enter fullscreen mode Exit fullscreen mode

If you're on a cloud VM (AWS, GCP, DigitalOcean), also check your provider's inbound security group rules in the web console. Docker port forwarding does not bypass cloud-level firewalls.

2. Peer-address flag was changed

The .envrc file sets APPEND_ARGS, and compose-midnight-node.yml passes that value into the node container. If you've modified APPEND_ARGS, compare it with the repo default and make sure you did not remove the peer-networking and Prometheus flags.

3. Clock drift

Substrate-based nodes reject connections from peers whose system clocks differ significantly. Check that NTP is active:

timedatectl status
Enter fullscreen mode Exit fullscreen mode

Look for NTP service: active and System clock synchronized: yes. If NTP is off:

sudo timedatectl set-ntp true
Enter fullscreen mode Exit fullscreen mode

4. Disk I/O bottleneck

High disk utilisation causes the node to fall behind processing blocks. Peers consider it unresponsive and drop the connection. Run iostat -x 1 10 and check %util. If it's consistently above 80%, the disk is your problem.

5. NAT or double-NAT

If your server is behind NAT, peers can't initiate inbound connections to you. Either configure port forwarding at the router level or use a VPS with a dedicated public IP.

Node crashes or restarts frequently

Check the last error before the most recent restart:

docker logs midnight-node 2>&1 | grep -E "ERROR|WARN|panic" | tail -20
Enter fullscreen mode Exit fullscreen mode

Check whether Docker OOM-killed the container:

docker inspect midnight-node \
  --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}'
Enter fullscreen mode Exit fullscreen mode

If OOMKilled=true, the node is running out of memory. Add more RAM, or as a temporary measure on an undersized machine:

sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
Enter fullscreen mode Exit fullscreen mode

For suspected chain data corruption after a hard shutdown, clear only the Midnight data while leaving your Cardano-db-sync database intact:

docker compose -f compose-midnight-node.yml down
rm -rf ./data
docker volume rm midnight-node-docker_midnight-data
docker compose -f compose-midnight-node.yml up -d
Enter fullscreen mode Exit fullscreen mode

If you also need to reset the Cardano infrastructure (only do this if you're sure the issue is there, it adds hours of re-sync time):

docker compose -f compose-partner-chains.yml down -v
docker compose -f compose-partner-chains.yml up -d
Enter fullscreen mode Exit fullscreen mode

IPC socket errors

After an unclean shutdown, you may see errors in cardano-db-sync or cardano-ogmios logs about a missing or stale IPC socket:

No such file or directory: /ipc/node.socket
Enter fullscreen mode Exit fullscreen mode

The socket file lives in the cardano-ipc shared volume. Remove the stale file and restart Cardano services:

# Find the socket path from the HOME_IPC environment variable
ls ~/ipc/node.socket 2>/dev/null || echo "Socket not found"
rm -f ~/ipc/node.socket

docker compose -f compose-partner-chains.yml down
docker compose -f compose-partner-chains.yml up -d
Enter fullscreen mode Exit fullscreen mode

Wait for the socket to be recreated:

watch -n 2 "ls -la ~/ipc/node.socket"
Enter fullscreen mode Exit fullscreen mode

Once the file appears, db-sync and Ogmios will reconnect automatically.

Port conflicts

If a service fails to start with address already in use, something else on your host is already bound to that port. The full port map:

Port Service
9944 Midnight Node RPC
30333 Midnight Node P2P
9615 Midnight Node Prometheus metrics
3001 Cardano Node P2P
1337 Ogmios WebSocket
5432 PostgreSQL

Identify the conflicting process:

sudo ss -tlnp | grep -E '9944|30333|9615|3001|1337|5432'
Enter fullscreen mode Exit fullscreen mode

Stop the conflicting process or adjust port mappings in the compose file if you need both to coexist.

Keeping your node up to date

Midnight is in active development across Preview, Preprod, and Mainnet. Node versions update frequently, and an outdated version will stop syncing or get rejected by peers. Watch the Midnight Discord channel and the release notes for new versions.

When a new version is released:

  1. Check the compatibility matrix to confirm the new version and any updated Cardano-db-sync version.
  2. Update the version tag in .envrc.$CFG_PRESET for your network.
  3. Reload, pull, and restart: 
direnv allow

docker compose -f compose-midnight-node.yml pull
docker compose -f compose-midnight-node.yml up -d --force-recreate

# Verify the new version is running
docker exec midnight-node /midnight-node --version
Enter fullscreen mode Exit fullscreen mode
  1. Run ./test.sh to confirm everything is healthy after the upgrade.

Before every upgrade, back up your configuration files. The chain data can be re-synced from genesis, but your compose edits and any custom credentials cannot be recovered if lost:

tar -czf "midnight-config-backup-$(date -u +%Y%m%d).tar.gz" \
  .envrc \
  ".envrc.${CFG_PRESET}" \
  compose-midnight-node.yml \
  compose-partner-chains.yml
Enter fullscreen mode Exit fullscreen mode

What to do next

You now have a full node running on the official stack, synced to the Midnight Preview Testnet, with monitoring commands and a tested troubleshooting playbook.

  • Set up an RPC node: if you want your DApp to talk directly to your own infrastructure instead of a public endpoint, follow the RPC node setup guide.
  • Explore the DApp quickstart: start building a privacy-preserving application on top of the network you're now running: Getting started.
  • Add persistent monitoring: schedule ./monitor-height.sh via a systemd timer or cron, and add alerting via Prometheus when peer count drops to zero or finality lag exceeds 15 blocks.
  • Join the community: the Midnight developer forum and Discord are the best places to ask questions, report issues, and follow network updates.
  • Share your work: tag @midnightntwrk on X and use #MidnightforDevs.

References

This tutorial targets Midnight Preview using the current environment naming. Always check the compatibility matrix before starting a node or upgrading an existing one.

Top comments (0)