DEV Community

Cover image for A Node.js Backend Memory Issue That Only Showed Up in Production
Sahinur
Sahinur

Posted on AI-assisted

A Node.js Backend Memory Issue That Only Showed Up in Production

One of the most frustrating backend problems is an issue that works perfectly in development and staging, but starts failing when real production traffic arrives.

I recently worked with a Node.js + Express + MongoDB backend running on an AWS EC2 instance, where we started seeing unexpected memory usage in production.

The application was working normally at first. Then, as traffic increased, the server's memory usage kept climbing until the Node.js process eventually crashed.

Here's how I approached the problem.

The setup

The backend was built with:

  • Node.js
  • Express.js
  • MongoDB
  • AWS EC2
  • PM2 for process management

The basic architecture looked like this:

Client
   ↓
Nginx
   ↓
Node.js + Express
   ↓
MongoDB
Enter fullscreen mode Exit fullscreen mode

The application handled API requests normally during development and staging.

The EC2 instance also had enough resources for the expected workload, so initially there was no obvious reason for the server to run out of memory.

What went wrong

After the application had been running under production traffic for some time, memory usage started increasing.

We noticed the EC2 instance's memory utilization gradually climbing instead of returning to its normal level after requests were completed.

Eventually, the Node.js process became unstable and crashed.

A representative error looked like this:

<--- Last few GCs --->

FATAL ERROR: Reached heap limit
Allocation failed - JavaScript heap out of memory

Node.js v20.x
Enter fullscreen mode Exit fullscreen mode

At first, increasing the server resources seemed like the easiest solution.

But that would only hide the actual problem.

The important question was:

Why was the application continuously consuming more memory?

Why staging didn't catch it

This was one of the interesting parts of the problem.

The application worked correctly in staging because the environment was significantly different from production.

The main differences were:

  • Lower traffic
  • Fewer concurrent requests
  • Smaller datasets
  • Shorter test duration
  • Less realistic production usage patterns

A memory problem doesn't necessarily appear immediately.

An application can use a little more memory after every request without causing any visible problem during a short test.

For example:

Request 1  →  500 MB
Request 100 → 550 MB
Request 1,000 → 700 MB
Request 10,000 → 1.2 GB
Enter fullscreen mode Exit fullscreen mode

Eventually, the server reaches its available memory limit.

This made the problem difficult to reproduce in staging.

How I investigated the problem

The first thing I did was check the EC2 instance's resource usage.

I used system-level monitoring to check memory consumption:

free -h
Enter fullscreen mode Exit fullscreen mode

I also checked running processes:

top
Enter fullscreen mode Exit fullscreen mode

and inspected the Node.js process managed by PM2:

pm2 status
pm2 monit
Enter fullscreen mode Exit fullscreen mode

The important observation was that CPU usage wasn't consistently high.

Memory usage, however, continued increasing.

That changed the direction of the investigation.

Instead of asking:

"Why is the API slow?"

I started asking:

"What is keeping objects in memory longer than expected?"

Looking at the application

I started reviewing endpoints that processed large amounts of data.

One suspicious pattern was fetching a large dataset from MongoDB and keeping the entire result in memory before processing it.

For example:

const users = await User.find({
  status: "active"
});

const result = users.map(user => {
  // process user
});

return res.json(result);
Enter fullscreen mode Exit fullscreen mode

This can become expensive when the collection grows.

If MongoDB returns thousands or tens of thousands of documents, Node.js has to hold those JavaScript objects in memory.

The problem becomes worse if multiple requests are doing the same thing concurrently.

The fix

The first improvement was to avoid loading unnecessary records into memory.

Instead of retrieving everything, we introduced pagination:

const page = Number(req.query.page) || 1;
const limit = 50;

const users = await User.find({
  status: "active"
})
  .skip((page - 1) * limit)
  .limit(limit)
  .lean();
Enter fullscreen mode Exit fullscreen mode

Now the application only processes a limited number of documents per request.

We also reviewed queries to make sure we weren't returning fields that the API didn't actually need.

For example:

const users = await User.find({
  status: "active"
})
  .select("name email status")
  .limit(50)
  .lean();
Enter fullscreen mode Exit fullscreen mode

This reduced unnecessary data being loaded into the Node.js process.

Monitoring after the fix

After making the changes, I continued monitoring the application rather than assuming the problem was solved.

The important thing was to observe memory usage over a longer period under realistic traffic.

The expected pattern was:

Memory
  │
  │       ╭──╮      ╭──╮
  │    ╭──╯  ╰──╮ ╭─╯  ╰──
  │────╯────────╰─╯──────────
  │
  └────────────────────────── Time
Enter fullscreen mode Exit fullscreen mode

Instead of continuously climbing, memory usage should stabilize within a reasonable range.

I also kept PM2 monitoring in place so that unexpected process behavior could be detected quickly.

What I learned

This incident taught me an important lesson:

A backend can work perfectly in development and still have serious resource problems in production.

For Node.js applications, I now pay much more attention to:

  • Large MongoDB queries
  • Pagination
  • Selecting only required fields
  • Processing large datasets
  • Concurrent requests
  • Memory usage over time
  • Production-like load testing

I also learned that simply increasing the EC2 instance size isn't always the correct solution.

More RAM can postpone the crash, but it doesn't necessarily fix the underlying problem.

What I'd do differently

If I were setting up the same system again, I'd introduce memory monitoring and load testing much earlier.

I'd also test endpoints with production-sized datasets instead of relying only on small staging datasets.

The biggest takeaway for me was simple:

Don't just test whether your API works. Test how it behaves after running for a long time under realistic load.

That's where some of the most interesting backend problems reveal themselves.

Top comments (0)