My site started returning HTTP 500 on every request today. Not a slow page, not one broken route. The entire app was down. Here's what it turned out to be and how I stopped it from happening again, in case it saves someone a bad afternoon.
The symptom
Every request returned a bare Internal Server Error. The app container was technically "online" in my host's dashboard, but nothing rendered. Checking the response headers confirmed the 500 was coming from my app, not the CDN or a DNS problem.
Reading the logs
The logs were a wall of the same error repeating every few minutes:
error: An error occurred while loading instrumentation hook: canceling statement due to lock timeout
code: '55P03'
where: 'while inserting index tuple in relation "queue"'
at pgboss.create_queue(text,jsonb)
Two things jumped out. First, Postgres error 55P03 is lock_not_available, meaning a statement gave up waiting for a lock. Second, it was firing inside pgboss.create_queue, which runs when my background worker boots.
The database had also logged that it "was not properly shut down" earlier, so a previous session had left a lock on the pgboss.queue table. My worker's boot tried to create its queue, couldn't get the lock, timed out, and threw.
Why one stuck lock nuked the entire site
This is the part worth internalizing. My worker boots inside the web process through Next.js's instrumentation hook:
export async function register() {
if (process.env.NEXT_RUNTIME === 'nodejs') {
const { startWorker } = await import('./worker/start');
await startWorker(); // <-- this await is the problem
}
}
Because register() awaited the worker boot, and the worker threw on the lock, the error propagated straight out of the instrumentation hook. Next.js treats a failed instrumentation hook as a fatal boot error and aborts the whole server. So a background-queue problem became a total front-end outage. The worker and the web server had no business sharing a failure domain, but that one await tied them together.
The fix, in two parts
First, recover: restart Postgres to clear the stuck lock, then redeploy the app so it boots cleanly against the healthy database. The app-alone redeploy is not enough on its own, because the lock is still held until Postgres restarts.
Second, and more important, make it impossible for this to take the site down again. The web server should boot even if the worker can't:
export async function register() {
if (process.env.NEXT_RUNTIME === 'nodejs') {
const { startWorker } = await import('./worker/start');
// Fire-and-forget. A worker failure must never abort the web server.
startWorker().catch((err) => {
console.error('worker boot failed (site stays up, will retry):', err);
});
}
}
Then I made the worker itself resilient so it self-heals once the lock clears, instead of dying on the first failure:
let started = false;
let starting = false;
const RETRY_DELAYS_MS = [2000, 5000, 15000, 30000];
export async function startWorker() {
if (started || starting) return; // only skip once it's ACTUALLY running
starting = true;
let attempt = 0;
while (true) {
const boss = new PgBoss({ connectionString: process.env.DATABASE_URL });
try {
await boss.start();
await boss.work('myqueue', handler);
started = true;
starting = false;
return;
} catch (err) {
const delay = RETRY_DELAYS_MS[Math.min(attempt, RETRY_DELAYS_MS.length - 1)];
attempt++;
console.error(`worker boot failed (attempt ${attempt}), retrying in ${delay}ms`, err);
try { await boss.stop({ graceful: false }); } catch {}
await new Promise((r) => setTimeout(r, delay));
}
}
}
The old code had a subtle second bug: it set started = true before boss.start() could throw. So after any failed boot, every retry hit if (started) return and the worker stayed dead forever. Flipping started only on success fixes that.
Takeaways
- A background job's boot should never share a failure domain with your web server. If you boot a worker inside a web process, catch its failure and let the web server come up regardless.
- Only mark something "started" after it has actually started. Setting the flag optimistically means a crash locks you out of ever retrying.
- Postgres
55P03in a queue library almost always means a leftover lock from an unclean shutdown. Restarting the database clears it; restarting only the app usually does not.
A transient database hiccup now degrades to "jobs lag for a minute while the worker retries" instead of "the whole site is down." That's the trade I wanted.
Anyone else run their background worker in-process with their web server? Curious how you isolate the two.
Top comments (4)
Postgres lock issues get extra painful in scheduled ranking systems. If a grid run, crawl import, or report job blocks writes, the next run can mix fresh and stale observations. I like adding job ids and observation windows so partial data cannot quietly masquerade as a clean trend.
Yeah, that's exactly the trap. A scheduled crawl that half-completes and gets written as if it finished is worse than a job that fails loudly, because the failure is invisible until the trend line looks wrong weeks later.
The observation-window idea is what I landed on too. I tag every write with the run id that produced it and treat a run as valid only once it's fully committed, so a partial run's rows never get read as current. Reads pull from the latest fully-completed run, not "whatever's newest in the table." Costs a bit of storage keeping the last good run around, but it means a killed or blocked job degrades to stale-but-consistent data instead of a silent mix.
When a run does complete late, deciding whether it overwrites the window it was meant for or just gets dropped. How do you handle that, do you let a late run reclaim its slot, or is anything past its window discarded?
That completed-run pointer is the important contract. I like the idea that reads never consume rows just because they are newest; they consume the newest run that proved it finished. It turns a partial crawler failure from bad data into isolated state.
Yes. Scheduled systems make lock bugs nastier because the failure often repeats before anyone has time to understand the first incident. I like treating lock timeout settings and job idempotency as part of the same reliability story, not two separate concerns.