DEV Community

Michael
Michael

Posted on • Originally published at gbase8.cn

Repairing a Harbor-db Container PostgreSQL After Power Failure in a GBase 8a Stack

#gbase #database #数据库 #docker

In a gbase database environment, Harbor is often used as a container image registry, with its internal PostgreSQL 13 running inside a Docker container. After an unexpected host power failure, the harbor-db container may get stuck in a Restarting loop. This guide walks through diagnosing the invalid primary checkpoint record error and repairing the PostgreSQL WAL using pg_resetwal.

Symptom

After the host rebooted, the Harbor database container kept restarting and never became healthy.

[root@mdw harbor]# docker-compose ps
NAME                IMAGE                                COMMAND                  SERVICE             CREATED             STATUS                            PORTS
harbor-core         goharbor/harbor-core:v2.8.2          "/harbor/entrypoint.…"   core                2 minutes ago       Up 7 seconds (health: starting)
harbor-db           goharbor/harbor-db:v2.8.2            "/docker-entrypoint.…"   postgresql          2 minutes ago       Restarting (1) 49 seconds ago
Enter fullscreen mode Exit fullscreen mode

Diagnosis

Check PostgreSQL Logs

The log reveals that the database system was interrupted and could not locate a valid checkpoint record.

tail /var/log/harbor/postgresql.log
May  6 15:54:30 172.22.0.1 postgresql[234137]: 2024-05-06 07:54:30.521 UTC [8] LOG:  database system was interrupted; last known up at 2024-04-23 09:46:52 UTC
May  6 15:54:30 172.22.0.1 postgresql[234137]: 2024-05-06 07:54:30.597 UTC [8] LOG:  invalid primary checkpoint record
May  6 15:54:30 172.22.0.1 postgresql[234137]: 2024-05-06 07:54:30.597 UTC [8] PANIC:  could not locate a valid checkpoint record
May  6 15:54:31 172.22.0.1 postgresql[234137]: 2024-05-06 07:54:31.419 UTC [1] LOG:  aborting startup due to startup process failure
Enter fullscreen mode Exit fullscreen mode

Find the pg_resetwal Binary

Since PostgreSQL 13 is not installed on the host, we retrieve the pg_resetwal utility from the Docker overlay filesystem.

[root@mdw harbor]# find / -name pg_resetwal
/var/lib/docker/overlay2/967167c094b2eedecfac2671c11a307414b15d23ace87880cfc24151934afed9/diff/usr/pgsql/13/bin/pg_resetwal
...
Enter fullscreen mode Exit fullscreen mode

Pick a valid binary (not a broken symlink) and ensure it is executable. Copy it to a local directory if necessary.

Repair Procedure

pg_resetwal must not be run as root. Switch to a non‑privileged user, temporarily change ownership of the PostgreSQL data directory (which is bind‑mounted from /data/database on the host), perform the reset, and restore the original ownership.

  1. Identify the data directory in docker-compose.yml: 
volumes:
  - /data/database:/var/lib/postgresql/data:z
Enter fullscreen mode Exit fullscreen mode
  1. Switch to a regular user and fix permissions: 
su - someuser
chown -R someuser:someuser /data/database
Enter fullscreen mode Exit fullscreen mode
  1. Run the reset command (force resetting the WAL): 
/path/to/pg_resetwal -f /data/database/pg13/
Enter fullscreen mode Exit fullscreen mode
  1. If you encounter replication checkpoint has wrong magic 324508367 instead of 307747550, remove the stale logical replication checkpoint file: 
cd /data/database/pg13/pg_logical/
rm replorigin_checkpoint
Enter fullscreen mode Exit fullscreen mode
  1. Restore the original ownership of the data directory (the container's postgres user UID/GID, often 999:999 or as previously configured): 
chown -R original_user:original_group /data/database
Enter fullscreen mode Exit fullscreen mode

Restart Harbor Services 

cd /path/to/harbor
docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

Verify that harbor-db is now Up and healthy. The Harbor core service will follow shortly.

Key Takeaways

Containerised PostgreSQL databases are vulnerable to checkpoint corruption after an unclean shutdown. Because you cannot docker exec into a restarting container, the host‑side data directory and the container’s pg_resetwal binary are your rescue path. Always back up the data directory and carefully manage file ownership when performing such repairs in a gbase database infrastructure. With this method, you can quickly bring your Harbor registry back online without redeployment.

Top comments (0)