A Leak in the Shell – How Refactoring Autocomplete Broke Us and How We Fixed It

All right. Well glad so many people are here to hear me yap about JavaScript. Always love that. But yeah, anyway, this is me. You can find me on BlueSky if you want to reach out at any point after the conference. I also have the slides online if you want to look anything up afterwards. And otherwise, yes, this is me.

Before we jump into the actual issues that we ran into, let me give you a little bit of overview of what it is that we actually wanted to do. And so you may know if you've ever used MongoDB that we have a database CLI that you can use to interact with your database. This is not like a MongoDB product talk, but I want to give you the context of what it is I'm talking about. Anyway, so yes, our database CLI is called Mongosh. Either is fine. And yeah, so you can write JavaScript in this database CLI to interact with your database.

At some point we thought, well, okay, so we have some rudimentary autocomplete in this, but wouldn't it be nice if autocomplete in our CLI could be a lot more context aware and a lot more helpful for our users? So for example, if you have this one document sort in your database where you have an order and a customer and potentially a lot of other fields that wouldn't be relevant here, and then you could autocomplete on these fields after you make a call like this. It's like that should be possible in theory, right? And so we were kind of like, okay, we already have autocomplete, but we could do much, much better one. And so we started having a plan.

And the first idea was, well, okay, so there is a really good autocompleter for JavaScript that millions of people use every day. And everybody in this room has used it thousands of times, which is the TypeScript autocompletion engine, right? This is like what, for example, VS Code uses, but also like a bunch of different environments. And so we figured, well, we could just apply this to database, like to input to our CLI, right? Shouldn't be too hard. So our plan that we came up with was like, well, we're going to analyze the user's database schema to get an idea of what their documents look like. We have a pretty good idea of what our API looks like. We just need to put that into TypeScript in a format that is understandable for, yes, for the TypeScript autocompleter that gets stored and saved as regular DTS files. So nothing too fancy or at least like, I mean, these aren't actual files. They're just entirely in memory, but same concept. And then we take these DTS files, we take the user input, and we pass those to the TypeScript autocompleter, right? And that kind of sounds easy enough, hopefully. And yeah, if I just try this out right now with the current version of our CLI, it actually autocompletes field names when I do tap, tap inside the CLI. So lovely.

And full credit where credit is due. Most of the implementation work for this was actually done by my coworker, who's sitting here watching me talk about his work. But yeah, so anyway, this is like, there's obviously more to it, because otherwise I would be done after two minutes of talking, right? And so, well, yes, it works, or does it? Well, we implemented this and we tried it out and it worked fine, and we put it behind a feature flag. And we told people, hey, if you work in MongoDB, if you want to try this out, really, please do. This is a new autocompleter. And so, before we published this, just letting you know. Anyway, we then, after a couple months of trying it out internally, I think, decided to flip the feature flag. And that did not go well. So what you're seeing is, well, our tests started crashing. Tests started crashing with out of memory errors. While they were testing the autocompletion portion of it all. So that's obviously not great.

Well, how do you even start debugging something like this? This turned out to be pretty tricky in our case, right? Because like, if you want to debug something, you start with a minimum example that reproduces the bug, right? That's always the first step. But each test passed on its own. This was obviously something that was happening because we were running tests in combination. This does make this trickier. Anyway, we can still try to minimize how long it takes until we reach this. For example, with Mocha, which is the test runner that we still happily use, I haven't really ever felt the need, personally, to switch to one of the new, cool, fancy things. They pretty much all look the same to me, anyway.

But anyway, with Mocha, you can pass Node.js flags down to the actual Node.js instance. So the first thing that you can try is to just reduce the heap size. So the heap size, that's like, you know, how much memory your JavaScript code can allocate. And if you reduce that, you're going to get errors faster. And that obviously helps a little bit with reproducing, but it's not the entire story, right? It's not a crash. And we somehow need to see what is actually in memory here, right?

Smart people, I think, actually, this first flag that I mentioned here, this heap snapshot near heap limit, was originally implemented by Joey, who's also a speaker today, I think, in the other room right after me. But there's a newer flag that is just heap snapshot on OEM, and it's basically telling Node.js, okay, well, if you run into out of memory errors before you crash, just write a snapshot of your memory to disk. So you can do that. You get a file that gets written to disk that's called heap snapshot dot something, something, something. And you can open that in Google Chrome. I actually learned while preparing this talk that you can also open it inside of VS code directly, which is, I don't know, interesting, but it doesn't have all of the same functionality for actually investigating the issues.

So what is system context? That was like, this is not something that shows up in your code, right? This is something that you need to learn. What is this, actually? And so in order to do that, let's maybe step back for a second. So just think about very, very simple cases of memory retention. I think this one is like about as simple as it gets, right? We have the global object and we have a property A, which is an object with a property B, with a property C, and then some really large chunk of memory that is guaranteed to show up, like Christmas lights in a heap snapshot. If we look at the heap snapshot for this, that turns out to be true. Like, it's one of the first things that get mentioned. There is something that retains 10,000 kilobytes of memory. That is pretty much what we would expect.

And you can go all the way to whatever group there is, like the global object, for example, that could be one or like a network socket. Those would be garbage collection routes. But yes, so there are these system contexts and there are things like our auto-completors that are being kept alive by them.

So what is system context? That was like, this is not something that shows up in your code, right? This is something that you need to learn. What is this, actually? And so in order to do that, let's maybe step back for a second. So just think about very, very simple cases of memory retention. I think this one is like about as simple as it gets, right?

We have the global object and we have a property A, which is an object with a property B, with a property C, and then some really large chunk of memory that is guaranteed to show up, like Christmas lights in a heap snapshot. And if we look at the heap snapshot for this, that turns out to be true. Like, it's one of the first things that get mentioned. There is something that retains 10,000 kilobytes of memory. That is pretty much what we would expect. And we can see, if we look at this piece of array buffer data, yes, that is being stored in something called a backing store, which is like a V8 internal concept that is owned by a buffer, which is the buffer property of a UN8 array, which is then called really large blob, which is retained by C, B, A in global.

So kind of what you would expect. So this is how you kind of read these heap snapshots when inspecting them in DevTools. So yes, that is pretty much what we would expect, and global is obviously a root. If you were to expand this, there would only be global and not really anything below it.

But let's maybe go for a more complicated example, right? So we, again, allocate a large piece of memory that is just mainly there to show up as a large piece of memory when we inspect it. And we schedule an interval that logs done every second, but we also schedule a timeout that does something with this blob. This timeout clears after one millisecond, and after that, it's done, right?

So context, as we can kind of infer from this, means these are the lexical scopes that your closures have captured. And due to the way that JavaScript engines work, when you capture variables, you potentially capture everything else that is in the scope as well. And so it's really not obvious, like, what does this function A even capture? It's the console variable, right? It's like something from the outer scope, which is enough to make it capture the entire scope, which is enough to retain this really large blob indefinitely, even though at least in theory an engine could tell that it will never be accessed again.

We can actually see that there is still something keeping this really large blob alive. And it is the context in function A, not in function B. So context, as we can kind of infer from this, means these are the lexical scopes that your closures have captured. And due to the way that JavaScript engines work, when you capture variables, you potentially capture everything else that is in the scope as well. And so it's really not obvious, like, what does this function A even capture? It's the console variable, right? It's like something from the outer scope, which is enough to make it capture the entire scope, which is enough to retain this really large blob indefinitely, even though at least in theory an engine could tell that it will never be accessed again.

And so that brings us a little closer to understanding why we ran into these issues. But there's a little more. One fun aspect of JavaScript that also came into play. So what does this print? Any ideas? It's not a trick question. Yes, four. Thank you. What does this print? Any ideas? Yes, four. What does this print? Any ideas? Yes, four. What does this print? Any ideas? Any ideas? Yes, four. Any ideas? Yes, four. What does this print? Any ideas? Any ideas? Yes, four. What does this print? Any ideas? No, not four. How dare you think that! So there's actually two kinds of eval in JavaScript. There's the eval keyword, where if you just call the eval keyword directly, it will inherit the outer scope of what you're running. So it's kind of like running a function, like a callback.

But if you refer to the global eval, which is what you get when you run eval eval, you do not get the outer scope. And so this fourth line here actually just throws an exception because it cannot find A in its scope, because it doesn't run in that scope. And so this can be really tricky because there are very subtle differences between calling eval, for example, and then calling, if you were to, in parentheses, one and end eval. Those would be completely different call targets according to the JavaScript spec. So anyway, with that knowledge, we can actually understand the bugs and the fixes that we did run into here.

And so, obviously, there was some very unnecessarily dumb stuff that we learned when we fixed this. One is, like, okay, so there's this one class that has a documentation that says it closes all open resources held by this REPL instance, and then it does not close or did not previously close one of the core resources that it held. Which is like, yeah, duh, that's going to leak your memory. But then, yes, there's also really tricky to detect, and that's kind of like the stuff that I was talking about, right? Obviously there's this extra line, which is also part of the issue here, but, like, replacing eval with this contextless eval function, and contextless eval is really just eval put into a separate file so it cannot capture any variables. That is all it is. That is the core of the bug that we fixed here, because otherwise this eval call that we had to do couldn't, like, obviously eval is evil, and we know that, and we shouldn't be doing this, but in our case this is just what we have to do. This was actually capturing enough of the outer scope that it was keeping objects in memory for way longer than they should have been.

And then, as with any good story, there's a Node.js core bug that we ended up fixing. So, like, we learned that the listener handling in the Node.js core REPL actually had a very similar issue, where it was calling process of prepaint listener, so basically adding an event listener, and it was passing a callback. And that callback did not depend on any variables in the scope, but it was enough still to capture the entire outer scope, so it kept the REPL instance that created it alive, even though that was completely unnecessary. And the fix kind of is just, like, to pull this to the top level so that it doesn't, it's not the closure that captures the actual REPL instance that adds it anymore. It's a, yeah, a fairly easy thing to miss. But luckily we're allowed to do open-source contributions to Node.js core, and we can just fix this.

Anyway, and coming to the closing in on the end of this, you might wonder, how do you actually test this? Obviously we fixed all these bugs and then our CI was green, but, like, how do you make sure that you do not regress on this, right? That's tricky. Because, like, obviously you might still, you know, notice if things go really bad, and you introduce similar memory leaks to the ones that we previously saw, but if something goes a little bad, you're not going to notice immediately. And part of the core of the issue is that garbage collection in JavaScript is unobservable, right? Or at least this used to be true for a long, long time. It's not anymore. So some of the newer stuff, this isn't that new, this is, like, a couple years old at this point, but, like, there's something called finalization registry in JavaScript now. And using it is incredibly hard. And I would... Or not using it, using it correctly is incredibly hard. Using it is pretty easy, actually. But yeah, so this is tricky to get right, but you can observe the finalization of objects through this API. Like, you can get a callback that is called when an object is removed from the JavaScript heap. So perfect. There are some Node.js specific APIs that you can use. There is Vieta query objects, which is pretty cool. I would encourage you to just try it out, one of your applications at some point, where you can pass a constructor to this function. And it will give you all the objects in memory that are created from this constructor. And so this is, like, an easy way to tell whether, you know, there is something that is still in memory that shouldn't be.

It wasn't really useful for our use case here, because then we would have had to hold on to the constructor as well, right? And so that kind of was too tricky for us to properly do. And then there's, like, the V8 getHeapSnapshot function, which is just, like, that gives you full HeapSnapshot in the format that I displayed here. And opened in Chrome DevTools. And this is what we used. We ended up using finalization registry and getHeapSnapshot and parsed the HeapSnapshot programmatically as part of our test. It's a little bit of extra code, but it's not that complicated of a format.

So that is actually fairly doable. But for those, especially, keep in mind those are diagnostic tools only. Do not use them in production. Do not use them anywhere where performance is super, super sensitive. Because, like, especially V8 getHeapSnapshot is just so slow. It's not supposed to be fast, right? Doesn't need to be. So anyway, let's try not to keep this pattern of, like, you know, optimizing closures too much in mind. Just keep it in the back of your head as something that, like, you know, when you run into memory leaks, this is something you want to look out for. It's not something you need to worry about in your everyday coding. Unless you're doing, like, really, really performance sensitive stuff.

Use HeapSnapshots. Just play around with them, honestly, as well. It's like those are an incredibly helpful tool for not just debugging memory leaks, but obviously especially memory leaks. And then when you run into bugs in your upstream code, then fix them. Especially with node.js, you're always welcome to. We definitely do appreciate contributions. And then obviously, because, you know, this is 2026, I need to talk about AI. Just out of curiosity while preparing this talk, I gave this example of the two, like, the set interval and set timeout to Claude and asked it, okay, here's the Heapsnapshot. Tell me what's wrong. Right? And it actually explains the problem fairly well.

So those are also tools that you can integrate with. Obviously it didn't parse the entire Heapsnapshot on its own. It just wrote a script that analyzed the Heapsnapshot for it. Right? But then when you talk about production bugs, like, keep in mind, these Heapsnapshots can be really large, because they're not just copies of your application's memory. They're like filtered copies of your application's memory, serialized to JSON. They contain your data. Heapsnapshots actually contain, well, like, your application's memory. So it's really easy for secrets to end up in there. So be careful about passing this to AI agents. Do it at your own risk and understand, you know, what exactly you're doing. Also, be careful about sharing this when you're, you know, reporting bugs or something.

And at the end of the day, I still feel like that, as JavaScript developers, it's our job to understand what our applications do. And to be able to debug them. So anyway. Thank you for your time. And... Applause. Do you have any tips for finding bugs that are so deeply embedded? Don't be scared. I think that's really the biggest tip. Like, if you run into issues that are really tricky, you know, that is always a big learning opportunity. And I feel like even in an age where we're always pushed to ship as fast as possible and to get as much work done as possible, it's always going to be worth taking the time to understand what is really happening.

Like when you have the feeling you don't quite have a grasp on what's going on, most of the time it's worth just, you know, okay, well, I'm gonna learn this. I'm gonna work on this until I figure it out. That's great. Thank you very much. Another question is could NodeVM module help with this? As one has full control over the available contexts? No, because we are using the NodeVM module here. Like the only reason I didn't go into that is because it would make the talk five minutes longer. But the fact that it was actually keeping alive the entire VM context as well is part of the issue here. Right? Like if we... Because VM context still live on the same heap.

So if we wanted a fully separate heap, we would have had to do something like a worker thread, which then might be an actual potential solution. Yeah. Okay, cool. And what are some of the biggest challenges you face while doing this work? Very broad question. I guess that goes back to my previous point. If you feel like you don't have the time to really dig into issues like this, that is something that you kinda need to push back against a little bit or try to account for in a way that you work out with whoever your managers or something. This is part of what we do as software engineers, right? And this is part of how we learn. It's not just like getting stuff shipped, it's also like how you understand your work. So the justification for having the time and resource to dig really deep into that. Yeah, that's interesting.

The next question is, have you considered preemptive heap size monitoring? Heap size monitoring to prevent malignant changes? Not in that form, but as I mentioned, we did put regression tests for this class of problem in place. The tests that we added would actually capture anything that keeps that x4vm context that I just mentioned alive. And so that would already cover quite a broad range of issues. Nice. And were the tests passing on initial release of the autocomplete without overflowing the heap? So this is a gap in our testing. I think we can be honest about that when we merged the autocomplete feature with the feature flag, we did not also start testing with the feature flag on and off. We only had it in the default mode and that is what we tested. And so if we had tested both combinations explicitly, probably wouldn't have run into this in this format because we would have just realized earlier that, hey, we cannot release this as is. Yeah, that's fair. It's always easy with retrospect to see. Yeah. It's just a gap. Yeah.

And is there a reliable way to force flush the GC in tests before inspecting the heap? Sure. Yeah. So, I mean, actually the things that I showed are reliable ways because, like, if you call something like v8.getheapsnapshot that is going to do a garbage collection right before because the heap snapshot serializer does not want to deal with your garbage. It wants to only write the things to just that are actually relevant. Otherwise, if you really just want to flush garbage collection, you can also do that. There's a flag in Node.js, dash dash expose GC that you can use to, you know, well, get a GC function that you can call to force it. But most of the time, it's probably not what you want. Okay. Great. Thank you so much. That's fantastic. Thank you, everybody. Can we give another round of applause to Anna.