Taking a Dump: Using Heap Dumps to Find and Fix NodeJS Memory and CPU Problems

Sometimes taking a dump can be a real life saver. I'm talking, of course, about using heap dumps to find and fix Node.js memory and CPU problems. What did you think that I was talking about? Anyway, my name is Dan Shapir. Welcome to JS Nation and to my talk about heap dumps and memory dumps and CPU problems.

I want to start with a horror story. You're the on-call person and you've just received a phone call letting you know that the server has crashed. Or maybe it's just running really, really slow and not servicing requests in a reasonable amount of time. What do you do? So obviously you start investigating. You look at the logs. You look at the various performance graphs.

Now you know, you've got a memory leak. And that's a really big problem because what do you do next? Well, in this talk, I will give you the tools that will make you the plumber of your organization being able to solve and fix memory leaks and the problems that they cause. But first, we need to discuss why memory leaks are even an issue in a Node.js environment. I mean, JavaScript is a garbage collected programming language after all.

Doesn't that mean that all the memory is automatically reclaimed and that memory leaks should never happen? Well, anybody who knows anything about how garbage collection actually works in real life knows that this is not the case. And even in garbage collected environments, memory leaks can happen both in real life and in programming. And in order to understand why that happens, we need to talk about how garbage collection actually works in JavaScript and in general programming languages. So let's say this is the state of our memory where we have multiple objects living in the heap. Some objects are referenced directly from the root, the root in JavaScript being either the global scope or the current execution context.

And even in garbage collected environments, memory leaks can happen both in real life and in programming. In order to understand why that happens, we need to talk about how garbage collection actually works in JavaScript and in general programming languages.

Let's say this is the state of our memory where we have multiple objects living in the heap. Some objects are referenced directly from the root, the root in JavaScript being either the global scope or the current execution context. JavaScript uses a garbage collection approach called mark and sweep, which actually has two phases.

In the first phase, called mark, the garbage collector starts from the root or multiple roots and looks for every object that it is able to reach. It goes along these connections, along these links, and marks each and every one. In the sweep stage, it looks at all the objects that have not been marked and understands that these objects are unreachable. Garbage collection doesn't collect all objects that are no longer needed; it collects objects that can no longer be used because they can no longer be reached.

In the first phase, which is called mark, the garbage collector starts from the root or multiple roots and looks for every object that it is able to reach. It goes along these connections, along these links, and marks each and every one. When it sees that there are no additional objects that it can mark, it knows that it's time to stop because it has reached all the objects that it can reach. It then goes on to the sweep stage.

In the sweep stage, it looks at all the objects that have not been marked and understands that these objects are unreachable. That's because there is no way to get at them from one of the roots. As a result, they can no longer be used and therefore can be reclaimed. Garbage collection does a little bit more than that, but that's good enough for the scope of this conversation.

Well, the problem is that, as we can see, the garbage collection doesn't collect all the objects that are no longer needed. It collects all the objects that can no longer be used because they can no longer be reached. If an object can still be reached, even if it's no longer needed, the garbage collection doesn't know that. It knows that it could still be used and therefore it's not safe to reclaim. That's the cause of leaks. Consider, for example, a global cache which is unbound, causing a leak by continuously adding items without removing them.

The cause of leaks is that certain memory is still held on either by globals that reference it directly or indirectly, or by the context that, again, still holds a reference to it either directly or indirectly. Consider, for example, a global cache which is unbound, which means you can just keep on putting more and more things into it and things never leave. Obviously, this can cause a leak because you will just keep adding stuff into the cache and never remove them. Even if you do remove items from the cache, if you remove them at a rate that is lower at which they are actually added, you will eventually blow up your memory.

With closures and contexts, a common cause of memory leaks might be a hook that you connect to a receiving event and have a closure in order to execute particular instruction with a relevant context. The problem there is what happens when you don't disconnect that hook when it's no longer needed. You're still registered to receive events even though you don't really care about them anymore, and effectively you leak all the memory that's referenced, again, directly or indirectly by the closure.

In actuality, the memory graph is not as clean, as flat as you saw it, at least if you zoom in close enough. Even if you don't have a leak, memory keeps fluctuating up and down. What actually happens is that memory usage goes up, and then the garbage collector kicks in, and the memory usage drops as unused memory is released, and then it starts going up again. So what we see here is a fluctuation, a sawtooth-type pattern, but that actually indicates that there is no memory leak.

A memory leak, in actuality, up close looks something like this. The memory usage does drop every time the garbage collector kicks in, but it doesn't drop as low as it was before. It gradually creeps up and up and up until we eventually run out of memory. But it's not just about memory. It's also about CPU. For example, because of an event called a GC storm. A GC storm is what happens when available memory is so low that the system constantly needs to do GC in order to try to free up some memory. But because there is a leak, it never succeeds in freeing enough. So it just runs this GC more and more and more often, and to get to the point where effectively all the system does is just run the GC all the time and has no CPU left for anything else. In fact, in our own actual environment, when we had a real memory leak, and we saw memory usage go up and up, in fact, we also saw CPU usage go up significantly during that time. And in fact, the reason that a server with memory leaks becomes unresponsive is often exactly because of this. So obviously, we really want to avoid memory leaks, and if we do have them, we want to fix them as quickly as possible. But the problem is that pinpointing the cause of a memory leak can be really, really difficult. That's because it's not local to a particular environment or location in the application. What I mean is a memory leak in any part of the application impacts the entire application as a whole, not just certain parts of it. And that makes pinpointing the cause of a leak very difficult and challenging. Another problem is that memory leak often manifests only in production environments where the system is under substantial load. It might manifest in the test environment as well, but there the leak is so small and minuscule that we will never actually even notice it.

Now, obviously, you can load test, but that can be very difficult and challenging. It's very difficult to make a testing environment that really emulates real production scenarios. So what do we do about it? Well, luckily, we have heap dumps and memory dumps to the rescue. A heap dump or a memory dump, it's effectively the same thing in this context, takes a snapshot of all reachable objects in memory at a specific point in time, and then we can analyze what's actually going on by looking at that snapshot. There are actually multiple ways of generating heap dumps for a Node.js environment. We can use DevTools, we can use the command line, and we can use an API. So let's start with DevTools. And for this purpose, I will actually show you a quick demo. So here I have a simple Node.js application, really just one JavaScript file that generates a memory leak. It intentionally creates a memory leak. We have a global array called leak data in which we will put objects and never remove them. I created a special endpoint that whenever I get to that endpoint, I generate 10,000 new instances of an object that simply has three properties that contain random numbers, and I push these objects into that leaked array so that they are never released.

And then I just return some status information about the memory usage. So let's launch this Node.js application. I run it in debug mode because I'm going to use the DevTools in order to find and fix the memory leak. So now let's see how we actually cause the link. I'll open another terminal, and I'll ping that endpoint using curl.

As we can see, it succeeded. It created the leak, and it returned the heap usage information. And if I ping it again, I see that heap usage actually goes up because every time I leak more and more memory because of all the objects that are created and never released. But how would I find them in a real production environment? Well, to that end, let's open the Chrome DevTools. I open a new Chrome tab. I do Chrome inspect, and I click the inspect link.

Because that node instance is running in dev mode, I can actually connect to it. And we see the DevTools, which is really similar to the DevTools we see when debugging a website, minus various tabs that are not relevant. For example, there is no elements tab, because there are no elements. It's a node instance, not a browser tab. So what I do now, I'm in the memory tab, and I'm on the option heap snapshot. There are other interesting options, but they are beyond the scope of this conversation.

So I will just take a snapshot. And now I have a memory dump or a heap dump of all the objects residing in memory. I see the various types of objects. There's code objects. There are general object objects. There are context arrays, et cetera. For each one, I see the shallow size, which is the actual size of that particular type, of all the instances of that particular object, and the retained size, which also includes object references by those instances.

But as you can see, there is a lot of stuff here. There are strings and other things that are not relevant to this particular leak scenario. So how do I pinpoint what I'm looking for? There's a great way to go about this, and this is by creating a diff between two snapshots. So what I do now is I will cause another leak. I perform the operation that causes a leak, and I get another snapshot. Now, instead of looking at the summary, I look at the comparison between these two snapshots, and I can sort by the delta the size change.

And I see that I leaked an object that has the structure ABC. I also leaked a lot of numbers because A and B and C all contain numbers, and those numbers are also objects in memory. But let's look at the actual objects. We can see that I leaked 10,000 objects, which is exactly the number that we have here in the code that we create every time that endpoint is pinged. Okay, so I know what I'm leaking, but how do I find what's holding on to it so that I can easily fix it? To that end, I open it up and I select one specific object instance.

I can even see the values that it has in its numbers. And here I can look at what is actually referencing it. So I see that I'm in the leaked data array. I can actually go up the reference list towards the root and see what object or what variable, either global or in a context, is actually holding on to it. And here I can see that it's the leaked data array, exactly because we push it into the leaked data array and never remove it from it. So now I can also know and understand what is actually holding on to these instances, and I know what to fix. So let's go back to the presentation.

So as we can see, DevTools and the memory heap snapshot mechanism is a great way to find and fix memory leaks and this way resolve the CPU and memory issues. But is it the only way? But first of all, let's talk about the benefits and limitations of this approach. The benefits is that it's very easy to use in the development cycle. There are multiple development tools that I can use. I can use it with the Chrome DevTools. I can use it with the DevTools that are built in, for example, to VS Code. I can also profile a leak during startup using Inspect BRK instead of Inspect. But there are also downsides to this approach. It requires a restart in order to add that CLI flag that puts the node in the development mode. It requires access from the development environment to the node instance using the Chrome debugging protocol, which is a security issue even though you can tunnel it over SSH. There is obviously performance overhead for running the node instance in debug mode, and I can also accidentally break the service by just pressing the pause button in the development environment. Also, taking snapshots is really heavy, and if I'm not careful, I can actually kill the process just by taking a memory snapshot.

Another cool way of getting a snapshot is to use a new experimental node flag, heap snapshot near heap limit, and you specify how many heap snapshots to take. So what happens is that when node nears the heap memory limit, it will take a snapshot up to three times in this case, so that I can still do the diff and find the reason what is actually leaking by doing the comparison between the snapshots. This is a cool way, but you need to be careful with it because, again, taking that snapshot might just drive you over the limit and cause the process to crash. I actually prefer another way, which is to specify that node will take a snapshot automatically when it receives a signal. So in this case, I use the node snapshot signal equals siguser2, and then if I use the linux kill command to send that signal to that process, I will actually cause it to generate a heap dump file in a specified folder. What are the benefits of this approach? First of all, it's set and forget. You can put it in production as long as you're not taking snapshots. There is no overhead, and it's secure because I don't need to open a debug port to that instance. On the downside, I do need to be able to access that Linux box, for example, using SSH in order to send the signal to that process. Also, the dumps themselves take up space on the server, and you need to be careful about how many dumps you actually put there before cleaning up to make sure that you don't blow up your disk. And, again, taking the snapshot, it's still a heavy operation and can push you over the limit.

The third way is actually using an API. There are built-in APIs in Node that you can use to generate heap dumps. So you can implement, for example, your own TCP endpoint that listens for some sort of instruction and generates a heap dump on demand and puts it wherever you want it to go. Obviously, that takes a little bit more effort. The benefits of using an API, you can control exactly when and how to invoke it. And, again, there is no overhead when idle. There is only overhead when you actually invoke that endpoint.

On the downside, there are security considerations because if you know about that endpoint, maybe others will know about it as well, and they can literally DDoS your server very easily by pinging that endpoint. Also, as before, you need to watch out for the disk space usage where you put the dumps. What's more, another thing that we need to really think about and consider when using heap dumps is that the heap dumps, because they contain a snapshot of the entire contents of the process's memory, they can definitely contain sensitive information. For example, information that's relevant in PII or HIPAA. So you really need to be careful about where you put those snapshot files, who you give them access to, and obviously remove them as soon as possible when they're no longer needed.

So to summarize what we talked about today, GC does not prevent memory leaks. That's because it only reclaims memory that can no longer be used, not the memory that is no longer needed. Leaks are challenging to find and fix because, as we talked about, they can happen anywhere in the application and impact the application as a whole, and they usually manifest only in production environments. You can use heap dumps to find and fix them by identifying the leaks and identifying what is actually causing them. As we saw, there are multiple methods to obtain them, and you can use the one that best fits your needs. You should use comparisons to find the deltas, and this way it's the easiest way to actually analyze the cause of the leak. You use the retainers to identify the cause. As you saw, we looked and found in our case that it was that leak data array.

And then after you create a fix, you should verify the fix to see that the leak is indeed resolved and there are no other leaks hiding in the system behind that more significant leak. And obviously, be very careful about security because of the sensitive data that can be contained inside those heap dumps. Thank you very much for joining me on this talk. I hope you can be effective as the plumbers of your organization helping to resolve and fix memory leaks. Thank you, and goodbye.