What's in a Node.js Bug – A Case Study

Yeah, hi, everyone, and thanks for joining this talk. So, I'm going to be talking about a Node.js bug.

Some quick intro for myself. Hey, I'm Anna. I am currently a staff engineer at MongoDB working on the developer tool suites. If you have ever used MongoDB, I'm sure a lot of you have at some point. I am working on a couple more things. I am also a former Node.js core contributor, have been quite active in the past, and a technical steering committee member on that side. I am very, very passionate about character encoding. I have given talks on that topic before. I know it may seem a little boring sometimes, or not like it's, you know, it isn't like bleeding edge technology, but it's still something that I think always makes for interesting conversations. This is how you can reach me. I also uploaded the slides at this link here, if you want to look for them at some point later.

Before we actually talk about the Node.js bug that I was referring to, let's do a quick refresher about what character encodings actually are specifically. So, like in a general, like typical application, you have your code that you run. That's a separate program. And that somehow talks to an operating system or kernel, which, you know, takes care of talking to the outside world for your application. And the way we've built software that happens to be the case that this operating system or kernel mostly, you know, receives and sends out information in the form of byte sequences. But your application typically works with the logic of like character sequences, which are typically called strings. And so, in order to like be able to work with strings inside your application, you need to have some kind of conversion that everybody agrees on in order to talk to the outside world. And those conversions are called character encodings.

Some of the early standardized character encodings, everybody knows ASCII, I'm sure that was like a way to encode most English language characters, still is. But that means that there are characters that you cannot represent. And then over time, obviously, you know, people recognize the need to be able to represent other characters. So, one of the more popular ones that is also like special for historical reasons is this ISA88591, which at least covers a lot of like Central European characters. But obviously, that's, you know, that also reaches its limits. You want to be able to represent Chinese characters, you're going to have to come up with something new where you cannot represent every character by a single byte. And so, Unicode came along and this was like starting to be really popular in the late 90s, early 2000s. And that essentially converted this process into a two-step process where through each character that you want to be able to represent, you assign a number. And then to that number, you assign a sequence of bytes.

And so, two of the more popular ones and the ones that are particularly relevant for this conversation today are 1UTF8, which is like, you know, backwards compatible with ASCII for as far as the byte sequences are concerned. It just like it starts to do something special for byte sequences that are outside the ASCII range. And UTF16, which is a way that's like not compatible with ASCII at all. But if you look at these in more detail, you can see that there are still like some shared representations of characters and some, you know, shared history when it comes to how strings exactly are being represented.

And so, I think most of you as JavaScript developers will have heard the, you know, claim that JavaScript uses UTF16, right? That's how strings work in JavaScript. And that's like, that's not entirely wrong, but it's also not true. And so, if we look into the source code of the JavaScript engine that Node.js uses and that Google Chrome uses, and a lot of JavaScript applications use these days, we can look at the source code of the we can look for all the strings that this implementation defines, and we can see there's actually a lot of very different representations for strings. And there are two that are particularly worth highlighting, which I'll show you in a second.

So, if you want to inspect, for example, how V8 internally represents this string here that I'm building together, it's like a concatenation and then a repetition and a substring of that. There actually is a way to do that. You need to expose V8 internals for that. So, this is not something you could do in a production application. But there is this debug print helper that V8 provides and you need to pass a special flag in order to enable it. And you can actually look at, like, you know, hey, what does this string contain? And so, for example, for this one, it's going to say it's a const string type, which means it's concatenated. It is actually internally represented as the concatenation of two strings, not a single sequence in memory. And so, if we break this apart, we can see that, like, yes, it's a concatenation of a single byte string or one byte string is what V8 calls it. And a slide string, which is represented by two byte strings. And we'll get into a bit why that's terrible naming. But the point that I'm trying to make here is JavaScript engines are going to be smart about how they internally represent strings.

So, anyway, let's look specifically at what happened in this Node.js bug that this talk is supposed to be about. So, like, this happened in August of 2024. You can see, go to the Node.js issue tracker. These are all from the Node.js core GitHub repo. And you can see that there are a ton of bug reports about something weird happening with UTF-8 and character encodings. And if you look at these reports in a bit more detail, there is something interesting about them that they all share. And this is, like, these specific snippets, like, it's flaky. It's not deterministic. It fails after it has already run for a byte. It happens sporadically.

So, like, that's very interesting when you look at a bug the way that happens. Because, like, most bugs are either, like, you know, something's broken or it's not. But here it seems like there's something more going on. And that actually leads into a very interesting rabbit hole about, you know, how JavaScript engines work these days and what Node.js core people are working on these days. But, like, as with any bug, let's start with trying to get a minimum of reproduction. So, like, I'm writing this test script. It's very simple. It takes the string. It converts it from a string to a buffer using UTF-8, converts it back using UTF-8. So, in theory, we should just get the same string back. However, apparently, that doesn't happen. And if we measure, like, how long it takes for that to happen, we can see that it is actually non-deterministic. It's, like, somewhere around 10,000 iterations of this loop. But not exactly. And so, it's still good enough as a reproduction to see whether this is failing or not, right?

So, we can do the next thing that you typically do when you have a bug that you don't really know where it's coming from. You do git bisect. And git bisect points to this particular pull request in the Node.js core repository. So, some change that has to do with buffers, with one by the strings, as they call it here in this PR and in the V8 API. And, like, yeah. So, this has something to do with it for sure.

And we'll look a bit more into, like, the exact diff here. And so, there is this addition of set fast method calls where this is in the part of the file where C++ methods get exposed to JavaScript as JavaScript functions. And we can see that these different, three different methods, one for ASCII, one for Latin1, which is another name for ISO 88591, and for UTF-8, they share the same implementation of this fast method, which is just fast write string. But different ones of the slow one. And that's already interesting, because, like, you know, let's assume that these functions do what they're named after, right? And so, like, they convert JavaScript strings into byte sequences. They write them into a buffer. That's why it's called write string. And the fact that they all share the same implementation in the fast case, that's already interesting, right? So, if we look at that particular method a bit more, this fast write string, we kind of know where this bug is coming from. And we have all the puzzle pieces now at this point.

So, what specifically happened here and why? So, first of all, this PR comes from the fact that Node.js core contributors care about performance. Part of that is the fact that in recent years, as y'all know, there have been alternative JavaScript server-side run times, or, you know, outside of the browser run times that are claiming to be faster. And I'm sure in some cases they actually are. Also, there's obviously some, like, actual advantage to having fast software. It means your applications are spending less time doing things that can translate to real savings if you run your application in the cloud and you suddenly need less resources. That's always very, very nice for your wallet.

So, doing this kind of work is also a good way to contribute to Node.js in a way. Because I know a lot of people want to. And, you know, you don't always have, like, the if you come with the idea of wanting to contribute, you don't always have the idea of, like, what you would want to contribute and what your areas that you would want to work on. But, like, making things faster is something that you can always do, you know, for most pieces of software, at least.

So, these set fast methods calls that we saw earlier, there is something that's called fast API embed. And we kind of need to understand what this specifically is to understand why this bug is happening and why it is happening in this specific way. So, like, in the past, one of the easiest ways to optimize Node.js core work has been to eliminate boundary questions between C++ and JavaScript. So, like, your JavaScript application calls into C++. Or vice versa, if there's some event happening on the event loop, which happens in C and C++, and it needs to inform your application that something happened, it needs to call back into JavaScript. And these have always been, like, fairly expensive parts of your applications. Because there's a lot of, like, setup work that JavaScript engines do before and after you enter one of these specific, you know, areas of your code, if you want to call it that. And so, to solve this problem, we had to introduce something called the fast API. As far as I know, it's still technically experimental. But, like, has been proving itself to be a real, you know, treasure trove of optimizations for JavaScript runtimes. And so, that eliminates a lot of this overhead of C++ and JavaScript boundary crossings. It only works one way. So, you can only call C++ from JavaScript code, not the other way around. But it has some preconditions. Like, you cannot always just make everything use the fast API and be done with it and, you know, never worry about it again. It only applies in some very specific situations that your JavaScript engine can detect at runtime. So, like, your data needs to be laid out in a way that allows for this optimization. The fast API function may not do anything that would trigger garbage collection. So, like, no creating new objects. No creating new strings, you know.

And it cannot call back into JavaScript. Because that could obviously also trigger garbage collection. Because you don't know what your JavaScript functions do. And so, the idea is, you write two implementations of a C++ function. One that is slow and one that is fast. And the fast calls are only being used after V8 detects that your code has run a lot. And so, it needs to optimize it. It wants to optimize it. And it recompiles your JavaScript function that was called a lot to be able to make use of the fast API. And one particular thing that's worth calling out about this is that V8, again, is a very smart piece of software. It may do this asynchronously.

So, like, it may take your JavaScript code, notice it has been called a lot, notice that it should be optimized, but then do that optimization on another thread in your application. So, like, it doesn't stop your code from running. It just, like, says, hey, this function needs to be optimized, recompile it, hands it off to another thread. And once it's done, it starts using that new implementation of your function. Super smart, really, if you think about it. And so, we go back to the example from earlier, and we look at this, and it kind of all comes together, right?

So, like, the reason why this happens after a flaky number of iterations and only after a bunch of them is that this function, this buffer.from in this case, needs to be called a lot in order to make the engine realize, hey, it's worth spending time to optimize this. It's worth spending energy on. Because it happens in a separate thread, it only happens after an unknown, but undeterministic number of iterations. So, that is definitely where this bug comes from. And so, I told you there's, like, two of these string representations that are worth paying special attention to, right? And so, one of them, one byte string, that's V8's name for ISA88591, and two byte string, that's V8's name for UTF-16. And those are just, like, the, so, six stands for sequential. So, those are the sequential representations where a string is really just, like, one sequence of bytes in memory without any special substring or slice or, you know, concatenation or anything like that.

So, if we look again at this slide from earlier, where we looked at the implementation of the fast write string method, we can see, okay, so, this fast write string method has the prerequisite, the second argument it gets, is a one byte string. And if you remember the previous slide, that means it's an ISA88591 string. And so, it copies this into the destination from the source, from this ISA88591 string, and this method doesn't do any other checking. So, if you remember, we were using this implementation for UTF-8 as well. And so, that's not what it should be doing. It just literally copies byte sequences from something that is one thing into something that's supposed to be something else. That's also why this bug, for example, didn't reproduce with all the characters.

Like, if you look at this slide again, you see it only happened with the accented E, where, for example, the other characters were, you know, preserved, because those are the ones where UTF-8 and ISA88591 actually differ. This also, for example, wouldn't happen if the string contained any Chinese characters or emoji, because then it wouldn't have this prerequisite fulfilled of being a fast one byte string, like something that is representable using ISA88591. And so, yeah, that's where this comes from.

And you obviously would be completely right to think, like, something should have caught this, right? It's very unfortunate. So, like, if you look at the coverage report on the pull request that I referenced, which, by the way, I absolutely don't want to blame the person who opened that pull request. This is something that is very easy to miss. Yeah, the coverage PR, the coverage data for that PR, it indicates that the new code paths were taken during testing. They just, like, didn't happen to be taken in the test that would cover these specific encoding questions. So, coverage doesn't really give you the answer here. There were also benchmarks that actually would have caught this issue, but, you know, you're writing benchmarks, those aren't tests, and so there was nothing in there that checked the results of these operations, which, you know, if you're writing benchmarks, that's absolutely understandable.

And Node.js also has this cool thing where it actually does something where it runs a bunch of popular NPM packages against a development build of Node.js. So, again, this is something that happens before any release goes out on the Node.js core repo. Again, there were some failures being caught. They were all looked at and none turned out to be related to this particular bug, most likely because this particular bug only is being triggered if, you know, a particular piece of code runs a lot, and that just didn't happen to be the case for any of the tests in these NPM packages. It's unfortunate, so this actually made it into a release. There are a bunch of things that we can learn from it, though, I think, and so some very direct software engineering things, you know, you've heard it before, naming is a hard problem. If one byte string had been called something like latin1str, or iside8591str, this might not have happened because the assumption that it would only be triggered for ASCII strings wouldn't even have been a thing. Also, coverage is only very limited, and what it tells you, you need to actually test the change that you're making and not just make sure that your new and old code paths are covered.