A Leak in the Shell – How Refactoring Autocomplete Broke Us and How We Fixed It

All right. Well glad so many people are here to hear me yap about JavaScript. Always love that. But yeah, anyway, this is me. You can find me on Blue Sky if you want to reach out at any point after the conference. I'll have the slides online if you want to look anything up afterwards. And otherwise, yes, this is me.

Before we jump into the actual issues that we ran into, let me give you a little bit of an overview of what it is that we actually wanted to do. You may know if you've ever used MongoDB that we have a database CLI that you can use to interact with your database. This is not like a MongoDB product talk, but I want to give you the context of what it is I'm talking about. Anyway, so yes, our database CLI is called Mongosh, either is fine. So you can write JavaScript in this database CLI to interact with your database. And at some point we thought, well, okay, so it's we have some rudimentary autocomplete in this, but wouldn't it be nice if autocomplete in our CLI could be a lot more context aware and a lot more helpful for our users?

For example, if you have this one document stored in your database where you have an order and a customer and potentially a lot of other fields that wouldn't be relevant here, and then you could autocomplete on these fields after you make a call like this. That should be possible in theory, right? And so we were kind of like, okay, we already have autocomplete, but we could do a much, much better one. So we started having a plan, and the first idea was, well, okay, so there is a really good autocompleter for JavaScript that millions of people use every day, and everybody in this room has used it thousands of times, which is like the TypeScript autocompletion engine, right?

And so we figured, well, we could just apply this to the database, like to input to our CLI, right? Shouldn't be too hard. So our plan that we came up with was like, well, we're gonna analyze the user's database schema to get an idea of what their documents look like. We have a pretty good idea of what our API looks like, we just need to put that into TypeScript in a format that is understandable for the TypeScript autocompleter that gets stored and saved as regular DTS files, so nothing too fancy, or at least, like, I mean, these aren't actual files, they're just entirely in memory, but same concept.

And then we take these DTS files, we take the user input, and we pass those to the TypeScript autocompleter, right? And that kind of sounds easy enough, hopefully. If I just try this out right now with the current version of our CLI, it actually autocompletes field names when I do tap tap inside the CLI, so lovely. Most of the implementation work for this was done by my coworker. But yeah, this is like, there's obviously more to it because otherwise I would be done after two minutes of talking, right?

Well, we implemented this and tried it out, put it behind a feature flag, and told people to try it out. After a couple of months of trying it internally, we decided to flip the feature flag, and that did not go well. Our tests started crashing with out-of-memory errors while testing the autocompletion. Debugging this turned out to be pretty tricky since tests passed individually but crashed in combination. To minimize the time to reproduce the issue, we tried reducing the heap size and using Node.js flags to analyze memory usage.

So what you're seeing is, well, our tests started crashing. The tests started crashing with out-of-memory errors while they were testing the autocompletion portion of it all. So that's obviously not great. Well, how do you even start debugging something like this? This turned out to be pretty tricky in our case, right? Because like, if you want to debug something, you start with a minimum example that reproduces the bug, right? That's always the first step. But each test passed on its own. So this was obviously something that was happening because we were running tests in combination. Which, yeah, this does make this trickier.

And so anyway, we can still try to minimize how long it takes until we reach this. For example, with Mocha, which is a test runner that we still happily use, haven't really ever felt the need personally to switch to one of the new cool fancy things. They pretty much all look the same to me anyway. But anyway, so with Mocha you can pass Node.js flags down to the actual Node.js instance. And so the first thing that you can try is to just reduce the heap size. And so the heap size, that's like how much memory your JavaScript code can allocate. And if you reduce that, you're going to get errors faster. And that obviously helps a little bit with reproducing, but it's not the entire story, right?

So, we're going to get a crash. And we somehow need to see what is actually in memory here, right? And so, smart people, I think actually this first flag that I mentioned here, this heap snapshot near heap limit, was originally implemented by Joey, who's also a speaker today, I think in the other room right after me. But there's a newer flag that is just heap snapshot on OOM. And it's basically telling Node.js, okay, well, if you run into out of memory errors before you crash, just write a snapshot of your memory to disk. So you can do that. You get a file that gets written to disk that's called heap snapshot dot something, something, something. And you can open that in Google Chrome. I actually learned while preparing this talk that you can also open it inside of VS Code directly, which is, I don't know, interesting, but it doesn't have all of the same functionality for actually investigating the issues. But anyway, so we're seeing there's like a ton of different objects in memory. We're seeing the auto-completor is somewhere up there on number five.

But you can also see there's groups of objects that you can already tell. There's a lot of objects that together hold 640,000 kilobytes in memory. And that probably means they're just keeping each other alive and this tool cannot really tell where the culprit is, right? So the auto-completor has something to do with this. But what's actually more interesting potentially is this first one that's all the way up there in the list, which is system slash context. And you might be wondering, well, what is that? Anyway, if we look at one of them, we can see there's this chain of things that retain this particular object. What you can do in this view is you can click on one object and you see these are the other objects that keep this one alive. And you can go all the way to whatever root there is, like the global object, for example. That could be one or, you know, like a network socket. Those would be garbage collection roots. But yes, there are these system contexts, and there are things like our auto-completors that are being kept alive by them. So what is system context? That was like... This is not something that shows up in your code, right? This is something that you need to learn. What is this, actually? And so in order to do that, let's maybe step back for a second. And so just think about very, very simple cases of memory retention. I think this one is about as simple as it gets, right? We have the global object and we have a property A, which is an object with a property B, with a property C, and then some really large chunk of memory that is guaranteed to show up like Christmas lights in a heap snapshot. And if we look at the heap snapshot for this, that turns out to be true. It's one of the first things that gets mentioned. There is something that retains 10,000 kilobytes of memory. That is pretty much what we would expect. And we can see, if we look at this piece of array buffer data, yes, that is being stored in something called a backing store, which is like a V8 internal concept that is owned by a buffer, which is the buffer property of a UN8 array, which is then called really large blob, which is retained by C, B, A in global. So kind of what you would expect. This is how you kind of read these heap snapshots when inspecting them in DevTools. So that is pretty much what we would expect. And global is obviously a root. If you were to expand this, there would only be global and not really anything below it. But let's maybe go for a more complicated example, right? We again allocate a large piece of memory that is just mainly there to show up as a large piece of memory when we inspect it. And we schedule an interval that logs done every second, but we also schedule a timeout that does something with this blob. This timeout clears after one millisecond, and after that, it's done, right? So the callback pass to the second set timeout, this function B, this shouldn't be kept alive in memory. And it doesn't. But we then still take a snapshot after that timeout has run.

We can actually see that there is still something keeping this really large blob alive. It is the context in function A, not in function B. So context, as we can kind of infer from this, means these are the lexical scopes that your closures have captured. And due to the way that JavaScript engines work, when you capture variables, you potentially capture everything else that is in the scope as well. And so it's really not obvious, because what does this function A even capture? Then it's the console variable, right? It's like something from the outer scope, which is enough to make it capture the entire scope, which is enough to retain this really large blob indefinitely, even though at least in theory an engine could tell that it will never be accessed again.

And so that brings us a little closer to understanding why we ran into these issues. But there's a little more. One fun aspect of JavaScript that also came into play. And so what does this print? Any ideas? It's not a trick question. Yes, 4. Thank you. What does this print? Any ideas? Yes, 4. What does this print? No, not 4. How dare you think that? So there's actually two kinds of eval in JavaScript. There's the eval keyword, where if you just call the eval keyword directly, it will inherit the outer scope of what you're running. So it's kind of like running a function, like a callback, but if you refer to the global which is what you get when you run eval eval, you do not get the outer scope.

And so this fourth line here actually just throws an exception because it cannot find A in its scope because it doesn't run in that scope. And so this can be really tricky because there are very subtle differences between calling eval, for example, and then calling, if you were to, in parenthesis, one end and eval. Those would be completely different call targets according to the JavaScript spec. So anyway, with that knowledge, we can actually understand the bugs and the fixes that we did run into here. And one is like, okay, well, there's this one class that has a documentation that says it closes all open resources held by this REPL instance, and then it does not close or did not previously close one of the core resources that it held, which is like, yeah, that's going to leak your memory. But then yes, there's also really tricky to detect, and that was kind of like the stuff that I was talking about. Obviously there's this extra line which is also part of the issue here, but replacing eval with this contextless eval function and contextless eval is really just like eval put into a separate file so it cannot capture any variables. That is all it is.

That is all it is. That is the core of the bug that we fixed here. Because otherwise this eval call that we had to do couldn't, like, obviously eval is evil and we know that and we shouldn't be doing this, but in our case this is just what we have to do. This was actually capturing enough of the outer scope that it was keeping objects in memory for way longer than they should have been.

And then as with any good story, there's a Node.js core bug that we ended up fixing. So, like, we learned that the listener handling in the Node.js core REPL actually had a very similar issue where it was calling process of prependListener, so basically adding an event listener, and it was passing a callback. And that callback did not depend on any variables in the scope, but it was enough still to capture the entire outer scope, so it kept the REPL instance that created it alive even though that was completely unnecessary.

And the fix is kind of just like to pull this to the top level so that it's not the closure that captures the actual REPL instance that adds it anymore. So, yeah, fairly easy thing to miss, but luckily we're allowed to do open source contributions to Node.js core, and we can just fix this. So anyway, and coming to the closing in on the end of this, you might wonder how do you actually test this? Obviously we fixed all these bugs and then our CI was green, but like, how do you make sure that you do not regress on this, right?

It's like obviously you might still, you know, notice if things go really bad and you introduce similar memory leaks to the ones that we previously saw, but if something goes a little bad, you're not going to notice immediately. And part of the core of the issue is that garbage collection in JavaScript is unobservable, right? Or at least this used to be true for a long, long time. It's not anymore.

So some of the newer stuff, this is a couple years old at this point, but there's something called Finalization Registry in JavaScript now, and using it is incredibly hard, and I would, or not using it, using it correctly is incredibly hard. Using it is pretty easy, actually. But yeah, so this is tricky to get right, but you can observe the finalization of objects through this API.

There are some Node.js-specific APIs that you can use. There's V8.QueryObjects, which is pretty cool. I would encourage you to just try it out, one of your applications at some point, where you can pass a constructor to this function, and it will give you all the objects in memory that are created from this constructor. And so this is like an easy way to tell whether there is something that is still in memory that shouldn't be.

There's V8.QueryObjects, which is pretty cool. I would encourage you to just try it out, one of your applications at some point, where you can pass a constructor to this function, and it will give you all the objects in memory that are created from this constructor. And so this is like an easy way to tell whether there is something that is still in memory that shouldn't be. It wasn't really useful for our use case here, because then we would have had to hold onto the constructor as well, right? And so that kind of was too tricky for us to properly do.

And then there's like the V8.GetHeapSnapshot function, which is just like, that gives you a full HeapSnapshot in the format that I displayed here, and opened in Chrome DevTools. And this is what we used. We ended up using finalization registry and V8.GetHeapSnapshot, and then parsed the HeapSnapshot programmatically as part of our test, which was like, it's a little bit of extra code, but it's not actually that complicated of a format. So that is actually fairly doable.

But for those especially, keep in mind, those are diagnostic tools only. Do not use them in production. Do not use them anywhere where performance is super, super sensitive, because like, especially V8.GetHeapSnapshot is just so slow. It's not supposed to be fast, right? It doesn't need to be. So anyway, let's try not to keep this pattern of optimizing closures too much in mind. Just keep it in the back of your head as something that like, you know, when you run into memory leaks, this is something you want to look out for. It's not something you need to worry about in your everyday coding, unless you're doing like really, really performance sensitive stuff.

Use HeapSnapshots. Just play around with them, honestly, as well. They are incredibly helpful tools for debugging memory leaks. Especially in Node.js, you're always welcome to contribute. In 2026, AI integration is essential. When preparing this talk, examples like setInterval and setTimeout with HeapSnapshots helped diagnose issues effectively. Integrate these tools cautiously due to the potential of large, data-containing HeapSnapshots.

Understanding your application's behavior is crucial for JavaScript developers. Debugging with HeapSnapshots requires caution due to the risk of exposing sensitive data. Take time to unravel deeply embedded bugs, as they present significant learning opportunities. Despite the pressure to deliver quickly, investing time in understanding complex issues is invaluable. It's essential to grasp the intricacies of your code to ensure optimal performance and security.

Thank you for your time and attention. Exploring deeply embedded bugs is an essential part of the learning process. Embrace challenges as opportunities for growth. Despite the rush to deliver, investing time in understanding your codebase pays off in the long run. Continuous learning and problem-solving are integral to being a proficient developer. Your dedication to unraveling complex issues sets you on a path to mastering software development.

And at the end of the day, I still feel like that, as JavaScript developers, it's our job to understand what our applications do. And to be able to debug them. So anyway, thank you for your time.

Do you have any tips for finding bugs that are so deeply embedded? Don't be scared. I think that's really the biggest tip. If you run into issues that are really tricky, that is always a big learning opportunity. And I feel like even in an age where we're always pushed to ship as fast as possible and to get as much work done as possible, it's always going to be worth taking the time to understand what is really happening.

When you have the feeling you don't quite have a grasp on what's going on, most of the time it's worth just, you know, okay, well, I'm going to learn this. I'm going to work on this until I figure it out. That's great. Thank you very much.

Could Node VM module help with this, as one has full control over the available contacts? No, because we are using the Node VM module here. The only reason I didn't go into that, because it would make the talk five minutes longer. The fact that it was actually keeping alive the entire VM context as well is part of the issue here, right? Because VM contacts still live on the same heap, so if we wanted a fully separate heap, we would have had to do something like a worker thread, which then might be an actual potential solution, yeah. Okay, cool.

What are some of the biggest challenges you face while doing this work? A very broad question. I guess that goes back to my previous point, like, you know, if you feel like, you know, you don't have the time to really dig into issues like this, that is something that you kind of, like, need to push back against a little bit, or like, you know, yeah, try to account for in a way that, like, you work out with whoever your managers or something, like, hey, this is part of what we do as software engineers, right? And this is part of how we learn. It's not just, like, you know, getting stuff shipped. It's how you understand your work. So the justification for having the time and resource to dig really deep into that. Yeah, that's interesting.

Yeah, that's interesting. The next question is, have you considered preemptive heap-size monitoring to prevent malignant changes? Not in that form, but as I mentioned, we did put regression tests for this, like, class of problem in place. Like, the tests that we added would actually capture anything that keeps that x4vm context that I just mentioned alive, and so that would already cover quite a broad range of issues. Nice.

And were the tests passing on initial release of the autocomplete without overflowing the heap? Well, we... So this is a gap in our testing. I think we can be honest about that. When we merged the autocomplete feature with the feature flag, we did not also start testing with the feature flag on and off. Like, we only had it in the default mode, and that is what we tested. And so if we had tested both combinations explicitly, probably wouldn't have run into this in this format, like, because we would have just realized earlier that, hey, we cannot release this as is. Yeah, that's fair. It's always easy with retrospect to see. Yeah. It's just a gap. Yeah.

And is there a reliable way to force flush the GC in tests before inspecting the heap? Sure, yeah. So, I mean, actually, the things that I showed are reliable ways. Because, like, if you call something like v8.get heap snapshot, that is going to do a garbage collection run before, because the heap snapshot serializer does not want to deal with your garbage. It wants to only write the things to disk that, you know, are actually relevant. Otherwise, if you really just want to flush garbage collection, you can also do that. There's a flag in Node.js, dash dash expose GC that you can use to, you know, well, get a GC function that you can call to force it. But most of the time, it's probably not what you want. Great. Thank you so much. Fantastic. Everybody, can we give another round of applause to Anna?