Possessed by Production: The Ghost Execution That Ran Code Nobody Ever Wrote

#devchallenge #bugsmash #webdev #javascript

Summer Bug Smash: Smash Stories 🐛🛹

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I deleted a single console.log statement from our payment worker, merged the PR, and went to grab tea. Ten minutes later, PagerDuty was screaming.

Memory usage on our background pods spiked to 99.8%. Outgoing webhooks failed with orphan closure errors, and Postgres rows started getting overwritten with payloads from different users.

I panicked and reverted the commit. I put the console.log back.

And just like magic... the errors stopped. Memory dropped back to 180MB. Everything went green.

Reverting a one-line PR and having production fix itself is terrifying. It means your architecture isn't actually stable—it's riding on accidental side-effects.

14 Hours in DevTools: What Happened?

If adding console.log fixes your backend, you are relying on luck and OS buffer timing.

I spent 14 hours taking heap snapshots (node --inspect) and diffing memory dumps across V8 garbage collection cycles. Here is what happened under the hood:

The Timing Trap: In Node.js, printing to stdout isn't always instant and non-blocking. That tiny log call introduces microsecond I/O pauses that inadvertently acted as an accidental sync guard.
The Floating Promise: We had a background logging task running inside an un-awaited Promise chain inside our request handler.
V8 Scope Reclamation: When console.log(user.id) was present, V8 retained an explicit reference to user in the parent scope.

When I removed the log, V8 JIT optimizations kicked in. During aggressive Garbage Collection, V8 marked the parent context as dead before the background callback finished writing to Redis. The callback resolved into an invalidated scope context.

The Broken Code

app.post('/api/checkout', async (req, res) => {
  const user = req.user;

  db.saveOrder(req.body).then(async (order) => {
    await telemetry.logTransaction(user.id, order.amount);
  });

  return res.json({ status: "queued" });});

The Fix

import { AsyncLocalStorage } from 'node:async_hooks';

app.post('/api/checkout', async (req, res) => {
  const userId = req.user.id;
  const payload = req.body;

  try {
    const result = await db.transaction(async (trx) => {
      const order = await trx.saveOrder(payload);
      await telemetry.logTransaction(userId, order.amount);
      return order;
    });

    return res.json({ status: "processed", orderId: result.id });
  } catch (err) {
    logger.error('Checkout pipeline failed', { err, userId });
    return res.status(500).json({ error: "Transaction failed" });
  }
});