DEV Community

Emily Thomas
Emily Thomas

Posted on

I Fixed a Bug That Took Down Production for 3 Hours — Here's Exactly How I Built the Fix

#ai #python #tutorial #opensource

Every developer has that one bug story. This is mine — a silent memory leak that slowly choked a Node.js API until it crashed under load, and the exact debugging process I used to catch it, fix it, and make sure it never happened again.

If you've ever stared at a stack trace at 2 AM wondering what went wrong, this one's for you.

The Setup: What Broke

We had a Node.js + Express API handling file uploads. Everything worked fine in staging. In production, under real traffic, memory usage climbed steadily until the server crashed every few hours.

app.post("/upload", (req, res) => {
  const chunks = [];
  req.on("data", (chunk) => chunks.push(chunk));
  req.on("end", () => {
    const buffer = Buffer.concat(chunks);
    processFile(buffer);
    res.send({ status: "ok" });
  });
});
Enter fullscreen mode Exit fullscreen mode

Looks harmless, right? That's the trap.

Before debugging any framework-level issue, it helps to know what tools are actually available for the job. I usually check a software hub first to see which monitoring and profiling tools other developers are using for similar stacks — it saves hours of trial and error.

Step 1: Reproducing the Bug

The first rule of debugging: don't fix what you can't reproduce. I wrote a small load-testing script to simulate repeated uploads locally.

const autocannon = require("autocannon");

autocannon({
  url: "http://localhost:3000/upload",
  connections: 50,
  duration: 30,
  method: "POST",
  body: Buffer.alloc(5 * 1024 * 1024) // 5MB payload
}, console.log);
Enter fullscreen mode Exit fullscreen mode

Within seconds, process.memoryUsage().heapUsed was climbing and never coming back down — confirmed leak.

Step 2: Finding the Root Cause

Using --inspect and Chrome DevTools' heap snapshot comparison, I found the issue: event listeners on req were never being cleaned up, and processFile() was holding references to old buffers in a global cache that was never cleared.

// The silent killer
const fileCache = {}; // never cleared, grows forever

function processFile(buffer) {
  const id = Date.now();
  fileCache[id] = buffer; // leak: never deleted
}
Enter fullscreen mode Exit fullscreen mode

Step 3: The Fix

Two changes fixed it completely:

  1. Use streaming instead of buffering the whole file in memory.
  2. Add a TTL-based cleanup (or just don't cache what you don't need). 
const { pipeline } = require("stream/promises");
const fs = require("fs");

app.post("/upload", async (req, res) => {
  try {
    await pipeline(req, fs.createWriteStream(`./uploads/${Date.now()}.tmp`));
    res.send({ status: "ok" });
  } catch (err) {
    res.status(500).send({ status: "error" });
  }
});
Enter fullscreen mode Exit fullscreen mode

Memory usage flattened immediately under the same load test.

Step 4: Preventing It From Happening Again

  • Added memory usage alerts (simple setInterval logging process.memoryUsage() to a monitoring dashboard).
  • Added a load test to CI so any future endpoint gets stress-tested before merge.
  • Documented the incident so the whole team knew the pattern to avoid. 
setInterval(() => {
  const used = process.memoryUsage().heapUsed / 1024 / 1024;
  console.log(`Heap used: ${used.toFixed(2)} MB`);
}, 10000);
Enter fullscreen mode Exit fullscreen mode

Tools That Actually Helped

Not every debugging tool needs to be paid. Chrome DevTools, clinic.js, and autocannon did 90% of the job for free. If you're building out your own debugging toolkit, it's worth checking an alternative of free softwares list before buying expensive APM licenses — many free tools cover the basics just as well.

Lessons Learned

  • Reproduce before you fix — guessing wastes more time than testing.
  • Streaming > buffering for anything file-related.
  • Global caches without cleanup are leaks waiting to happen.
  • Load testing in CI catches these bugs before they hit production.

Final Thoughts

Bugs like this rarely announce themselves loudly — they creep in quietly until traffic exposes them. The real fix isn't just the code change; it's building a process so the same class of bug gets caught early next time.

Tools and runtimes change fast, so whatever debugging stack you settle on, keep it updated. You can check the updates software version website to make sure your Node version, dependencies, and monitoring tools are all current before your next deploy.

Top comments (0)