Building the Node-API Conformance Test Suite

Hi, my name is Kane, and today I want to share a project that I've been building in the Node.js organization, a conformance test suite for Node API. But first, a few words about me. My name is Kane, as I said, and I am a developer experience engineer at 11labs. I built developer experiences for people building on top of the voice agents and the other services that we have. And before that, I maintained the Atlas Devices SDK, formerly known as Realm.js at MongoDB. I'm a proud father of three, based in Roskill, just west of Copenhagen. I also like to build in public, and I do a lot of open source. Callstack sponsors me for this in particular, and I have GitHub sponsorships open if you want to join them. You can find me on BlueSky as well.

And let's start with something that you might be familiar with. You've written a native Node.js add-on, perhaps. It works perfectly fine in Node.js, and your CI is green, and you ship it. But does it work on Deno? Does it work on React Native, which recently also supported Node API via Hermes? The honest answer is you probably don't know, and that is the problem that I set out to fix.

Let me back up and explain how we got here. Let me start with getting on the same page of Node, what Node API is, and starting with what it is not, because it's often confused. It is not the Node.js runtime APIs, things like File System, Path, Process. It is a C API to access JavaScript. It is API stable. That means that you can compile it once, and your add-on once, and then you can reuse it across new versions of Node.js. The headers for the C API, it ships on npm. So, one package, that's all you need. And it contains these library files in two layers. One is the engine-specific part, and this is the js-native-api.h, creating values, calling functions, managing memory, for example, and then the runtime-specific part, which is more specific to Node.js. Runtime APIs, like buffers, async work, lifecycle events. Originally, Node API was designed to abstract V8 for Node.js internals, and basically, you could write your implementation once, and then you wouldn't have to link directly against the V8-specific version. For other runtimes now, they started adopting this. BAN and DINO, not necessarily in that order. And then later, React Native by Hermes, and this other implementers as well. This is the remarkable thing. You have one C header designed for one runtime that is now becoming a de facto standard in the industry.

And as I said, it's split into these two categories of APIs. I already touched upon this. And I mean, the left-hand side, these are the engine-agnostic APIs. They are supposed to work across all JS engines. And then the other, more runtime-specific APIs, specific to Node.js, might or might not have support within the other implementers. This also turns out to be a concept for the compliance test suite. And now that we know what Node API is and who is implementing it, I want to tell you what's missing.

So the Node.js repo already has tests for this. They have like 58 test folders with testing each of the different feature areas. But when you open up these tests, you see things like a require call, which is a Node.js-specific, the common JS pattern, for module import. And you also see stuff like loading the add-on using a require call, which is not a part of the official, you can say, Node API specification. And these tests are completely tied to Node.js runtime. The utility, the comment file that you see here, has internal implementation that use FS and the paths and stuff like that. So we want to make sure to run these on other runtimes.

You can see some of the existing prior art here. You can see some of the projects basically copied out the Node.js tests. And the consequences here that when a bug is fixed in the Node.js repository, it doesn't automatically propagate or if a new test is added, for example, it's not automatically propagating and exercising the Bonandino implementation. And this is an issue because the spec clarification, you can see that the tests are implementing, they stay siloed. And when one runtime claims that they support Node API, there's no shared guarantee to like something that you can use to validate that claim. And this is a gap. And that is what the Node API conformance test suite fills.

So introducing it, the idea here from the original issue, it captures the motivation. It didn't plan. Node API didn't plan to become a cross runtime standard, but it did. And now it needs the infrastructure to get that standardized. And basically having it in the CTS and the Node.js organization means that it has some legitimacy and the runtimes have a shared ground truth to measure against. And you can also, as an author of these add-ons, you can see what APIs you can rely on across the runtimes.

The CTS has five design goals. One is we don't want any Node.js runtime APIs inside the actual tests, which we have to basically use standard language features. The second is the harness is implementer customizable. When I say implement, I mean a Node API implementer like Bondino, React Native, for example. And each runtime, they bring their own implementations of specific globals that the tests can access. We also want to have a third here, the same goal is we want to have a gradual opt-in, which means that as you're implementing as a new runtime is implementing these APIs, we want to have them be able to gradually.

A fourth one is we want each of the test files to run in isolation. This is how other test suites do it as well. It prevents leakage between the tests and it makes it possible to run the entire test suite and see failures without crashing the process that is executing and orchestrating the test run. The fifth here is we want to publish on npm. So you can easily fetch the package and run your tests without the hassle of checking out the repo. Good. So here's like a three-tier architecture. You can see in green here we have the test runner. This is specific to the implementer. In this case, it could be Node.js. And then we have the blue ones are implementations of a harness, but the implementation here is specific to the runtime. And then we have the test here in gray, which is test.js, and then some native add-on code. And let me walk through each of these in depth. If we just look at the CTS on the final level, you can see we have the tests directory here. We exercise the harness itself. So this is basically to have a sanity check that the implementer is providing a harness that we can rely on. Then we have the JS native API.

Then we have the JS native API. This is the JavaScript engine tests. And then we have the Node API, more runtime-specific tests. The implementers also get to bring their own harness implementation, and they can do that in their own repos, or they can also bring it in and put some files into the CTS for them to rely on. Currently, we only have Node.js right now here.

So I want to show some... If we go look at the tiers here, we could call them... First, we have the native add-ons. These are C++ code that exercise the Node API, and they basically get called, get loaded in by the load add-on function, and then the JavaScript is driving and exercising of the add-on, which is basically calling into Node API underneath. We use CMake as a build system primarily because it's more flexible for us, and Jib is also abandoned by the Chromium project, so it's not something we want to rely on for the far future. And I think, yeah, we have some helper functions here to define an add-on, making it possible to build on multiple platforms.

C2 here is the harness, the implementer provided harness. Right now, it has to support three globals. The assert, basically simple assertions. Add, sorry, load add-on, which is a function that you call to load a specific add-on for the test into your test. On Node.js, we use an implementation based on DLOpen, and then we have a GCUntil utility that basically triggers garbage collection until a certain condition is met, and this is again provided by the test harness and used by the test. And then we have, in this case, a part of the harness, the Node.js orchestrator. And again, this is specific to the different implementers. They can provide their own ways of loading tests, but in this case, we import the separate files that will provide the globals, and then we load the test file into and run that using a subprocess.

And why do we use a subprocess? Again, states cannot leak between the tests, and it's also critical for garbage collection and object lifetime tests. And the global garbage collection is fresh for every test here, and it also matches what we see in other conformance testing suites. We want to ensure that we stay runtime agnostic. Again, as I said, we don't want to have the test rely on Node.js runtime APIs. And how do we solve that? So, technique one is we don't use the require calls. We don't use a specific module system. For now, we basically have the test file is self-contained, and whatever it needs to function is injected by globals ahead of time. So, you can see no import statements here, and we use just a load add-on to load the add-on and assert using these globals, as I said. And we use ESLint to enforce this at Lint time, so we make sure that no module system calls slip in. And this is the configuration that we use for that, for example. Technique two here, the garbage collect until for finalizer tests, is basically triggering garbage collection. So, we do it over and over again until some condition is met. We also make sure to yield control using the set immediate. We yield to the event loop, and then we call this garbage collection GCM global that is provided because we run the process with expose GC. And basically, we do this for 10 times in this case, and then if it doesn't succeed within that, then we have to abandon. Yeah. So, again, this is an implementer-specific thing. Bondino could implement this differently. React Native could as well. So, again, this split in API is not just a design decision. It's also something that we use in the roadmap for how we port stuff. We have this porting MD file, which is capturing the current state of the porting of the tests. We have some easy tests, which, like direct ports, not that difficult, and we have some medium tests that will require changes, additions to the harness, and then we have the hard ones that might be more difficult to actually port. And where are we today? The progress is done here. We already have the infrastructure in place, a harness, CMake build system with this platform-specific support in it. We have a CI matrix running on Mac, Ubuntu, Windows, testing on four different Node versions, and we have the ESLint enforcement to ensure that we have the Node module system in use. And we also publish under the Node.js org on GitHub. So, right now, as the current state is, 14 tests are fully ported. These are the easy category in the engine agnostic tests, and this is about half, right? And we have other... This is the remaining test.

As you can see here, most of the Node API tests are medium or hard. Another you can say for some of these Node API or some of the other functions are marked as experimental in the header files, and that is currently a design challenge that we haven't solved. Basically, how do you ensure that some of these might be implemented by the runtimes, right? It could be some of the functions here are create shared array buffer or a post finalizer. And currently, these are omitted from the header file package, and so you don't compile against them by default. And if one runtime wants to ship support for some of these, I think it should be possible for the CTS to exercise it. And a possible solution here is to use something like a custom header that does forward declarations. And then we have a specific, potentially CMA variant and some features that you as an implementer can provide, like an experimental features global with the capabilities of your implementation. And then the tests can basically check here and skip gracefully if in case that the harness indicates that the implementer does not support a certain experimental function.

What is next? I believe that we have cleared phase one and now we have two ahead of us. Phase two is taking the medium and hard tests and pulling them one P at a time. And I expect to use language bottles for this. And then we have phase three to get one and Dino and React Native to running this suite in their CI. That could be awesome. And then potentially a stretch goal of pulling this or integrating this somehow with the Node.js main repository so we can get these tests running every time we do changes to Node.js. So do you feel like contributing? I see three clear ways of doing that. One is we need to port all the tests. And the easy ones are done already. But the next up are the medium ones and the hard ones. And you can pick one from the porting here. You can copy over the C and JavaScript from the existing test and strip or manipulate the– strip away the Node.js runtime APIs to make it agnostic. Then you can also add your runtime to the harness. So if you are an implementer of a Node API implementation, you can provide– clean your directory inside of the implementers directory and implement load add-on assert and gguntil. And then you're good to go. And if you're just curious about this thing, I would appreciate a star or a watch on the GitHub repository. This helps you keep yourself updated.

Thank you so much for your time. It's a rare commodity these days. And Node API is already out there in a lot of places. Let's make sure that it also works everywhere. Bye-bye.