Taking a Dump: Using Heap Dumps to Find and Fix NodeJS Memory and CPU Problems

Sometimes taking a dump can be a real lifesaver. I'm talking, of course, about using heap dumps to find and fix Node.js memory and CPU problems. What did you think that I was talking about? Anyway, my name is Dan Shapir. Welcome to JS Nation and to my talk about heap dumps and memory dumps and CPU problems.

I want to start with a horror story. You're the on-call person and you've just received a phone call letting you know that the server has crashed, or maybe it's just running really, really slow and not servicing requests in a reasonable amount of time. What do you do? So obviously you start investigating, you look at the logs, you look at the various performance graphs, and then you see something which looks kind of like this. It's a graph of the memory used by the server, and you see that it keeps going up and up and up, occasionally dropping down when the server restarts or reboots. But then it just keeps going up yet again. So now you know you've got a memory leak.

And that's a really big problem, because what do you do next? Well, in this talk I will give you the tools that will make you the plumber of your organization, being able to solve and fix memory leaks and the problems that they cause. But first we need to discuss why memory leaks are even an issue in a Node.js environment. I mean, JavaScript is a garbage-collected programming language, after all. Doesn't that mean that all the memory is automatically reclaimed and that memory leaks should never happen? Well, anybody who knows anything about how garbage collection actually works in real life knows that this is not the case. That even in garbage-collected environments, memory leaks can happen, both in real life and in programming. And in order to understand why that happens, we need to talk about how garbage collection actually works, in JavaScript and in general programming languages.

So let's say this is the state of our memory, where we have multiple objects living in the heap. Some objects are referenced directly from the root, the root in JavaScript being either the global scope or the current execution context. So some objects are referenced directly from the root, other objects are referenced by other objects. And sometimes it can even be a circular connection. So this is a graph, not a tree.

JavaScript uses a garbage-collection approach called mark-and-sweep, which actually has two phases. In the first phase, which is called mark, the garbage collector starts from the root, or multiple roots, and looks for every object that it is able to reach. It goes along these connections, along these links, and marks each and every one. When it sees that there are no additional objects that it can mark, it knows that it's time to stop, because it has reached all the objects that it can reach. It then goes on to the sweep stage. In the sweep stage, it looks at all the objects that have not been marked, and understands that these objects are unreachable.

The cause of memory leaks is objects that can still be reached, even if they're no longer needed. This occurs when memory is held by globals or contexts that reference it directly or indirectly. Memory leaks can result from unbound global caches or unclosed event hooks with closures. Even without leaks, memory fluctuates, increasing until the garbage collector releases unused memory. Garbage collection does not collect objects that can still be reached, causing memory leaks due to retained references, directly or indirectly.

It collects all the objects that can no longer be used, because they can no longer be reached. If an object can still be reached, even if it's no longer needed, the garbage collection doesn't know that. It knows that it could still be used, and therefore it's not safe to reclaim. And that's the cause of leaks. The cause of leaks is that certain memory is still held on either by globals that reference it directly or indirectly, or by the context that, again, still holds a reference to it, either directly or indirectly.

Consider, for example, a global cache, which is unbound, which means you can just keep on putting more and more things into it, and things never leave. Obviously this can cause a leak, because you will just keep adding stuff into the cache and never remove them. Even if you do remove items from the cache, if you remove them at a rate that is lower at which they are actually added, you will eventually blow up your memory. With closures and contacts, a common cause of memory leaks might be a hook that you connect to receive an event, and have a closure in order to execute a particular instruction with a relevant context.

The problem there is what happens when you don't disconnect that hook when it's no longer needed. You're still registered to receive events, even though you don't really care about them anymore, and effectively you leak all the memory that's referenced, again, directly or indirectly by the closure. In actuality, the memory graph is not as clean, as flat as you saw it, at least if you zoom in close enough. Even if you don't have a leak, memory keeps fluctuating up and down. What actually happens is that memory usage goes up, and then the garbage collector kicks in, and the memory usage drops as unused memory is released.

Then it starts going up again. What we see here is a fluctuation, a sawtooth-type pattern, but that actually indicates that there is no memory leak. A memory leak, in actuality, up close, looks something like this. The memory usage does drop every time the garbage collector kicks in, but it doesn't drop as low as it was before. It gradually creeps up and up and up until we eventually run out of memory.

But it's not just about memory. It's also about CPU, for example, because of an event called a GC storm. A GC storm is what happens when available memory is so low that the system constantly needs to do GC in order to try to free up some memory. But because there's a leak, it never succeeds in freeing enough, so it just runs this GC more and more and more often, and to get to the point where effectively all the system does is just run the GC all the time, and has no CPU left for anything else.

In fact, in our own actual environment, when we had a real memory leak and we saw memory usage go up and up, in fact, we also saw CPU usage go up significantly during that time. And in fact, the reason that a server with memory leaks becomes unresponsive is often exactly because of this. So, obviously, we really want to avoid memory leaks, and if we do have them, we want to fix them as quickly as possible. But the problem is that pinpointing the cause of a memory leak can be really, really difficult.

That's because it's not local to a particular environment or location in the application. What I mean is a memory leak in any part of the application impacts the entire application as a whole, not just certain parts of it, and that makes pinpointing the cause of a leak very difficult and challenging. Another problem is that memory leaks often manifest only in production environments where the system is under substantial load. It might manifest in test environments as well, but there the leak is so small and minuscule that we will never actually even notice it.

Now, obviously, you can load test, but that can be very difficult and challenging. It's very difficult to make a testing environment that really emulates real production scenarios. So, what do we do about it? Well, luckily, we have heap dumps and memory dumps to the rescue. A heap dump or memory dump, it's effectively the same thing in this context, takes a snapshot of all reachable objects in memory at a specific point in time. And then we can analyze what's actually going on by looking at that snapshot. There are actually multiple ways of generating heap dumps for a Node.js environment. We can use dev tools, we can use the command line, and we can use an API.

So, let's start with dev tools, and for this purpose I will actually show you a quick demo. So, here I have a simple Node.js application, really just one JavaScript file that generates a memory leak. It intentionally creates a memory leak. We have a global array called leak data in which we will put objects and never remove them. I created a special endpoint that whenever I get to that endpoint, I generate 10,000 new instances of an object that simply has three properties that contain random numbers, and I push these objects into that leaked array so that they are never released. And then I just return some status information about the memory usage. So, let's launch this Node.js application. I run it in debug mode because I'm going to use the dev tools in order to find and fix the memory leak. So, now let's see how we actually cause the link.

I'll open another terminal and I'll ping that endpoint using curl. As we can see, it succeeded. It created the leak and it returned the heap usage information. And if I ping it again, I see that heap usage actually goes up because every time I leak more and more memory. Because of all the objects that are created and never released.

But how would I find them in a real production environment? Well, to that end, let's open the Chrome DevTools. I open a new Chrome tab, I do Chrome inspect, and I click the inspect link. Because that Node instance is running in dev mode, I can actually connect to it. And we see the dev tools, which is really similar to the dev tools we see when debugging a website. Minus various tabs that are not relevant.

For example, there is no elements tab because there are no elements. It's a Node instance, not a browser tab. So, what I do now, I'm in the memory tab. And I'm on the option heap snapshot. There are other interesting options, but they are beyond the scope of this conversation. So, I will just take a snapshot. And now I have a memory dump or a heap dump of all the objects residing in memory. I see the various types of objects. There is code objects, there are general objects, there are context arrays, etc.

For each one, I see the shallow size, which is the actual size of that particular type, of all the instances of that particular object. And the retained size, which also includes object references by those instances. But as you can see, there is a lot of stuff here. There are strings and other things that are not relevant to this particular leak scenario.

So, how do I pinpoint what I'm looking for? There is a great way to go about this, and this is by creating a diff between two snapshots. Now, instead of looking at the summary, I look at the comparison between these two snapshots. And I can sort by the delta, the size change. And I see that I leaked an object that has the structure ABC. I also leaked a lot of numbers because A and B and C contain numbers, and those numbers are also objects in memory.

But let's look at the actual objects. We can see that I leaked 10,000 objects, which is exactly the number that we have here in the code that we create every time that endpoint is pinged. Okay, so I know what I'm leaking, but how do I find what's holding onto it so that I can easily fix it? To that end, I open it up and I select one specific object instance. I can even see the values that it has in its numbers. And here, I can look at what is actually referencing it. So, I see that I'm in the leaked data array. I can actually go up the reference list towards the root and see what object or what variable, either global or in a context, is actually holding onto it.

And here, I can look at what is actually referencing it. So, I see that I'm in the leaked data array. I can actually go up the reference list towards the root and see what object or what variable, either global or in a context, is actually holding onto it. And here, I can see that it's the leaked data array, exactly because we push it into the leaked data array and never remove it from it. So, now I can also know and understand what is actually holding onto these instances, and I know what to fix. So, let's go back to the presentation.

So, as we can see, DevTools and the Memory Heap snapshot mechanism is a great way to find and fix memory leaks and this way resolve the CPU and memory issues. But is it the only way? But first of all, let's talk about the benefits and limitations of this approach. The benefits is that it's very easy to use in a development cycle. There are multiple development tools that I can use. I can use it with the Chrome DevTools. I can use it with the DevTools that are built in, for example, to VS Code. I can also profile a leak during startup using InspectBRK instead of Inspect.

But there are also downsides to this approach. It requires a restart in order to add that CLI flag that puts the node in the development mode. It requires access from the development environment to the node instance using the Chrome debugging protocol, which is a security issue even though you can tunnel it over SSH. There is obviously performance overhead for running the node instance in debug mode. And I can also accidentally break the service by just pressing the pause button in the development environment. Also, taking snapshots is really heavy. And if I'm not careful, I can actually kill the process just by taking a memory snapshot.

Another cool way of getting a snapshot is to use a new experimental node flag, heap snapshot near heap limit, and you specify how many heap snapshots to take. So what happens is that when a node nears the heap memory limit, it will take a snapshot up to three times in this case so that I can still do the diff and find what is actually leaking by doing the comparison between the snapshots. This is a cool way, but you need to be careful with it because, again, taking that snapshot might just drive you over the limit and cause the process to crash. I actually prefer another way, which is to specify that node will take a snapshot automatically when it receives a signal. So in this case, I use the node snapshot signal equals siguser2, and then if I use the Linux kill command to send that signal to that process, I will actually cause it to generate a heap dump file in a specified folder.

What are the benefits of this approach? First of all, it's set and forget. You can put it in production. As long as you're not taking snapshots, there is no overhead. And it's secure because I don't need to open a debug port to that instance. On the downside, I do need to be able to access that Linux box, for example, using SSH, in order to send the signal to that process. It also, the dumps themselves take up space on the server, and you need to be careful about how many dumps you actually put there before cleaning up to make sure that you don't blow up your disk.

The third way is actually using an API. There are built-in APIs in Node that you can use to generate heap dumps, so you can implement, for example, your own TCP endpoint that listens for some sort of instruction and generates a heap dump on demand and puts it wherever you want it to go. Obviously, that takes a little bit more effort. The benefits of using an API, you can control exactly when and how to invoke it. And again, there is no overhead when idle, there's only overhead when you actually invoke that endpoint. On the downside, there are security considerations because if you know about that endpoint, maybe others will know about it as well, so you can literally DDos your server very easily by pinging that endpoint. Also, as before, you need to watch out for the disk space usage where you put the dumps.