A stuck Postgres lock took my whole site down. Here's the one-line cause and the fix.

My site started returning HTTP 500 on every request today. Not a slow page, not one broken route. The entire app was down. Here's what it turned out to be and how I stopped it from happening again, in case it saves someone a bad afternoon.

The symptom

Every request returned a bare Internal Server Error. The app container was technically "online" in my host's dashboard, but nothing rendered. Checking the response headers confirmed the 500 was coming from my app, not the CDN or a DNS problem.

Reading the logs

The logs were a wall of the same error repeating every few minutes:

error: An error occurred while loading instrumentation hook: canceling statement due to lock timeout
code: '55P03'
where: 'while inserting index tuple in relation "queue"'
  at pgboss.create_queue(text,jsonb)

Two things jumped out. First, Postgres error 55P03 is lock_not_available, meaning a statement gave up waiting for a lock. Second, it was firing inside pgboss.create_queue, which runs when my background worker boots.

The database had also logged that it "was not properly shut down" earlier, so a previous session had left a lock on the pgboss.queue table. My worker's boot tried to create its queue, couldn't get the lock, timed out, and threw.

Why one stuck lock nuked the entire site

This is the part worth internalizing. My worker boots inside the web process through Next.js's instrumentation hook:

export async function register() {
  if (process.env.NEXT_RUNTIME === 'nodejs') {
    const { startWorker } = await import('./worker/start');
    await startWorker(); // <-- this await is the problem
  }
}

Because register() awaited the worker boot, and the worker threw on the lock, the error propagated straight out of the instrumentation hook. Next.js treats a failed instrumentation hook as a fatal boot error and aborts the whole server. So a background-queue problem became a total front-end outage. The worker and the web server had no business sharing a failure domain, but that one await tied them together.

The fix, in two parts

First, recover: restart Postgres to clear the stuck lock, then redeploy the app so it boots cleanly against the healthy database. The app-alone redeploy is not enough on its own, because the lock is still held until Postgres restarts.

Second, and more important, make it impossible for this to take the site down again. The web server should boot even if the worker can't:

export async function register() {
  if (process.env.NEXT_RUNTIME === 'nodejs') {
    const { startWorker } = await import('./worker/start');
    // Fire-and-forget. A worker failure must never abort the web server.
    startWorker().catch((err) => {
      console.error('worker boot failed (site stays up, will retry):', err);
    });
  }
}

Then I made the worker itself resilient so it self-heals once the lock clears, instead of dying on the first failure:

let started = false;
let starting = false;
const RETRY_DELAYS_MS = [2000, 5000, 15000, 30000];

export async function startWorker() {
  if (started || starting) return; // only skip once it's ACTUALLY running
  starting = true;

  let attempt = 0;
  while (true) {
    const boss = new PgBoss({ connectionString: process.env.DATABASE_URL });
    try {
      await boss.start();
      await boss.work('myqueue', handler);
      started = true;
      starting = false;
      return;
    } catch (err) {
      const delay = RETRY_DELAYS_MS[Math.min(attempt, RETRY_DELAYS_MS.length - 1)];
      attempt++;
      console.error(`worker boot failed (attempt ${attempt}), retrying in ${delay}ms`, err);
      try { await boss.stop({ graceful: false }); } catch {}
      await new Promise((r) => setTimeout(r, delay));
    }
  }
}

The old code had a subtle second bug: it set started = true before boss.start() could throw. So after any failed boot, every retry hit if (started) return and the worker stayed dead forever. Flipping started only on success fixes that.

Takeaways

• A background job's boot should never share a failure domain with your web server. If you boot a worker inside a web process, catch its failure and let the web server come up regardless.
• Only mark something "started" after it has actually started. Setting the flag optimistically means a crash locks you out of ever retrying.
• Postgres 55P03 in a queue library almost always means a leftover lock from an unclean shutdown. Restarting the database clears it; restarting only the app usually does not.

A transient database hiccup now degrades to "jobs lag for a minute while the worker retries" instead of "the whole site is down." That's the trade I wanted.

Anyone else run their background worker in-process with their web server? Curious how you isolate the two.

Comments

[Alex Shev]

Postgres lock issues get extra painful in scheduled ranking systems. If a grid run, crawl import, or report job blocks writes, the next run can mix fresh and stale observations. I like adding job ids and observation windows so partial data cannot quietly masquerade as a clean trend.