Debugging memory leaks is rarely a piece of cake, especially when they only happen in production. The best way I’ve found to debug memory leaks in a Node.js application on Heroku is to analyze heap dumps.
Obtaining such heap dumps in production can be challenging, as it might be hard to connect remotely to a production instance with the debugger.
In this article, we will go through the steps needed to obtain and analyze heap dumps from a running Heroku dyno. This method will also work on other platforms as long as it is possible to perform similar operations.
To obtain the heap dump we need to:
- Ensure the Node.js process has a debugger listening
- Connect Chrome dev tools to the Node.js process
- Collect the heap dump and download it locally
Enabling the Node.js inspector
Before we can analyze anything, we need to ensure that we have a debugger listening. There are two ways to enable the inspector on a Node.js process:
Solution 1: Changing the startup command
By default, Heroku starts a Node.js application by running npm start. Usually, this calls a script defined in the package.json of the application:
Changing this script to add the --inspect (as documented here) flag will start the instances of the application with a debugger listening on a port that will be specified in the logs:
In total, this is what it will look like when you implement this solution.
Solution 2: Changing the process state through SSH
Solution 1 is the easiest way to enable an inspector in Node.js, but there are situations in which you can’t or won’t want to enable it. For example, you might not have access to the source code of the application and therefore can’t change the startup script. Or maybe you don’t want to change the state of all your production dynos and deploy your application only for debugging.
Fortunately, there is a way to send a signal to the process to enable a debugger session.
In order to do so, you will need the Heroku CLI to connect to the dyno through an SSH connection.
For all following Heroku commands, you might need to add the --app <app_name> flag to tell the CLI which application to connect to. Also, by default, the CLI will connect to the dyno named web.1 and you might want to change that through the command line (see documentation).
First, let’s connect to the dyno (Heroku might need to restart the dyno at this point):
Then, we need to identify the PID of the Node.js process:
In our case, the process started with node bin/www has the PID 69, we will now send a signal to the process to let it know we need it to enable its debugger:
As you can see, we have sent the USR1 signal to the process to change its state (as documented on this page).
This is confirmed through the application’s logs on Heroku:
Attaching debugging tools to a Node.js process
In order to attach the debugging tools to our Node.js process, we need to make the WebSocket used by the debugger accessible on our local machine.
To do that, we first need to identify the port we need to forward. This can be found in the logs of the application:
In our case, this is the port 9229.
To forward the port locally, let’s use the Heroku CLI:
When port forwarding is established, we just need to open Chrome DevTools (going to chrome://inspect on Chrome) and after a few seconds, a target should be displayed under “Remote targets.”
If the target does not appear, make sure the port used is listed when clicking on “Configure.”
Collecting the heap dump and reading it
Now it’s time to collect and read the heap dump. First, click on the “inspect” link. This will open a new window with different tabs.
Find the “Memory” one — you should be prompted with the following window:
Click on “Take snapshot.” A new file will appear in the left hand side panel. Clicking on it will display the content of the heap:
In this view, objects are sorted by constructor. For the purpose of this walkthrough, I have introduced a memory leak in this application by creating an instance of the Access class for each request. This instance keeps a reference to the current HTTP requests and is never cleaned:
You can see for yourself that this indeed leaks in the application.
To detect constructors that have the biggest memory impact, let’s sort the items of this view by “Retained size” (You can learn more about these terms on Chrome’s website).
You can see that 24% of the process memory is held by these objects.
Now let’s look at how to identify where the leak is happening.
When expanding the list of the constructor, we can see all instances of this class. By selecting one of these instances, the list of retainers of this object is displayed:
In our case, the allAccesses set is clearly identified as the bad actor! With the location of the memory leak identified, we have everything we need to go off and fix it.
A few tips for debugging memory leaks in Node.js
Use the compare view
When suspecting a memory leak, you might want to take two separate heap dumps with a few minutes between them. Then, using the “comparison view”, you can identify which elements have been created between the snapshots.
Use constructors and classes in the code
As shown in the article, when reading the heap dump, elements are grouped by their constructor.
Using more than just classes in your code will make it more readable (and arguably more performant, but that’s probably a topic for another article). It will save you so much time when hunting for a memory leak. Do it — future you will be grateful.
Trigger a garbage collection before collecting the snapshot
At the top left hand side of this screen, there’s a little bin picture. Clicking on it will trigger a garbage collection in the application. Doing this before collecting a memory snapshot will actually remove elements that are not leaking and therefore could help save you time when browsing the heap content.
Conclusion
In this article, we’ve taken a look at how to debug memory leaks in a Node.js process running on Heroku by connecting and using a debugger. Feel free to contact me on Twitter if you have any questions or if you want to share your own tips with me!
If you’re looking for next steps or a more advanced way to debug memory leaks in Node.js in Heroku, try this: Since the Heroku CLI is written with Node.js, you could write an automated tool to perform the collection and start analyzing heap dumps.
The post How to debug memory leaks in a Node.js application on Heroku appeared first on Sqreen Blog | Modern Application Security.
Top comments (0)