Exploring Node.js Test Runner

Hello, everyone. Thank you for having me today. So today we're going to talk about the new feature of Node.js, the test runner. First let me introduce myself. I'm Marco Ippolito. I'm one of the maintainers of Node.js, mostly focusing on security. And I work a lot in the open source ecosystem of Node.js.

So let's start by definitions. So what's a test runner? So the test runner is actually the CLI tool that executes the test and integrates a reporter to export the outcome of the test. So it goes through your source code and picks the test files to run. So let's see a bit of classification. So Mocha, for example, is a very popular test runner and testing framework because the two things go together. And the test runner always includes the testing framework because otherwise, what you're going to pick the file to execute but what else are you going to execute? So testing framework provides the content that is inside the file while the test runner actually finds the file to execute. So Cypress is another popular testing framework that is built on top of Mocha. So Mocha takes care of going through the source code and finding which files you want to execute, and then Cypress is built on top of the BDD libraries of Mocha. While Chai is only an assertion library that's also built on top of Mocha. So we have three main components that allow us to perform testing in our application.

Okay. So now we talked a lot about libraries but what about Node.js itself? So in the Node.js ecosystems there are a lot of libraries and I want to, like, let's see the most popular ones. So we have Mocha that we talked about, Jest, TAP. We have like a huge amount of libraries. And while this is good to have a lot of variety, it's actually bad for new developers that want to learn how to do tests in Node.js because there is an incredible amount of information and it's hard to find a guide and something to begin testing with.

So what's the reason why there are so many test runner testing frameworks in the Node.js ecosystem? Well it's because Node.js always follows the minimal core principle and we can talk about this is also related to the whole JavaScript ecosystem. So Node.js doesn't want to add to its core nothing that can be a NPM package. So when someone requests a feature to Node.js what we say, just create a package. But with years this led to a huge ecosystem and it's deteriorated the developer experience. So people who wanted to learn how to use Node.js found it very hard because of the huge amount of packages and also having too many packages increased the surface for NPM supply chain attacks. And I'm one of the team members of the Node.js security team and we have been trying to decrease the risk for NPM supply chain attack with the Node.js permission model and many things. So having something built in the core is one package less, one potential vulnerability less.

Alright, so Node.js was actually shipping only an assertion library. Since Node 12 there was no test runner, no testing framework, just a very very small assertion library which was not very useful. So Node.js needed a native test runner and a native testing framework so you could execute tests in Node.js without installing any other dependency. So it took some time. It was marked as stable in Node 20. So from Node 20 which is the LTS and if you're not using Node 20 you should. You can use the Node.js test runner. So yeah, it took some time. It took like a couple of years to make it but we finally have it. And now we're going to explore some of the features. So the Node.js test runner was actually heavily inspired from Node.tap. Node.tap is one of the, in my opinion, best testing frameworks and test runners in Node.js. It's been created by Isaac, the creator of NPM, so it's quite popular and we took big inspiration for this library and it's actually one to one. You can just replace the import and it's the same. It uses the same functions. So let's see some of the features of the Node.js test runner. So we will see like filtering and how to execute certain tests and skip other ones. We will talk about sub-testing, reporting.

This is very important and mocking. And we also have watch mode that if you don't know about watch mode, it's a very cool experimental feature that Node.js has. And code coverage, which is also very cool experimental features. And we will talk on why it's very important to have code coverage, an internal code coverage library.

All right. So we did a lot of theory and as someone much more important than me said, talk is cheap, show me the code. So yeah, we're going to talk about code. All right.

So how to execute test, you just run Node dash dash test. And from Node 20 you can do it. It will automatically select the test folder inside your application and run all the test in JavaScript files inside. So you can also specify the folder that you want to execute and specify also the extension if you're using a CGS file, MGS, depending on the JavaScript version that you're using. And you can also watch the test folder with the dash dash watch option. And it's very, it's very, like I always use it. All right. So yeah, you can also watch a single file instead of a folder. A watch is still experimental. So I suggest you to use it and provide feedback to the Node.js project. So you can, if you find some bugs, just open an issue and this will help the whole community.

All right. So how to write test. It's very easy. You just create your, you just have the standard syntax and the assertion library. You can import it from Node test and Node assert. And then you can assert something very standard. We will go through the assertion library in a while. It has some strict, a strict sub folder where there are like strict for deep equality checks and also not strict checks. So there are many assertions that you can do. There is the positive assertion. You can also do the negative assertion by adding a not at the beginning.

Very handy. One thing, fun fact about the assertion library, we used to ship snapshot testing so you could test snapshot, but we decided to remove it because it was not stable enough. So we're trying to bring it back because I personally love snapshot testing.

And you can also obviously run a synchronous test with different kinds of level. So you have a top level and sub testing. This is quite powerful in the Node.js because most of the time they are going to be asynchronous.

And you can also use the BDD keywords like describe and it, they do the same thing. It's just another syntax for people who want to use behavior-driven testing. Node.js testing library also provides some keywords like to do, skip and only. So you can filter your test based on, you can write, you know that you will want to write the test later so you mark it as to do, you want to skip it because it's not ready yet, you can just skip it. So it provides a full, full utilities for testing.

Obviously all the hooks, before, before all, after, after all. This is our standard features that Node.js provides out of the box so you don't need to install any other library. You can filter by name pattern. So for example, if you want to write only the test that contain the word foo in the description, you can do it. So in this example, it will only execute foo but it will skip bar, this is very handy.

You can mock. It has the, out of the box, the mocking functionality so you can mock object functions, whatever, you don't need libraries like xenon and it comes everything out of the box.

Yeah, you can choose your own reporter. So most people use spec, which is the default one. This is what it looks like. But you can also use tap if you like YAML output. I personally don't like YAML, tap, because it's very verbose, but it can be parsed by machines. So it's actually pretty good for automations. And this is what tap output looks like. So pretty standard. We also have dot as a reporter. So if you are familiar with something like PHP units, yeah, the output is just a dot. And an X if it failed. I haven't used it yet.

I don't know why should someone use it, but feel free to test it. And you can create your own custom test reporter. For example, if the test passed, you return a green check and if not, it returns a bug. And you can do this by creating a function. And it's just an async iterator. So I'm sure that all of you use async iterators every day, right? No? So yeah.

Okay. And now we talk about coverage. Node.js ships coverage. It's experimental. And it's very handy because unlike other libraries, for example, Istanbul, which they add checks during execution. So they take your code, add small counters, and check how many counters have been updated. And then it does some logic to actually check the coverage. This is not how Node.js does it. Node.js uses V8 internal coverage. So it's very powerful. And if you're using C8 as a coverage library, it uses the Node.js internals to do it. So you can have it out of the box. No need for external libraries. It's inside V8 and Node.js will offer it to you. And this is what coverage looks like with the default reporter, which is a spec. And you have for every file the lines of coverage, branch, and et cetera. So also here. Pretty standard if you're used to testing.

You can use TypeScript. So there is a lot of controversy in this because Node.js is the only runtime that doesn't ship TypeScript out of the box. And there is a reason behind it. So feel free to talk about this after the talk. But you can do it quite easily. You can just require the end use TSNode or TSX, whatever TypeScript transpiler you prefer.

And then you just use --"require." And you can run TypeScript tests very easily. The syntax actually changed pretty recently. Before it was a loader, --"loader." Now it's -"require." It's been a great update.

And yeah, this is another very important feature. It's shards. So if you have a huge test suite, you can split the test into pieces. And you can run this test on multiple machines. This is something that has been released in Node 21. So it's not on the LTS yet. And this is huge because if you have a huge testing library in your CI CD, you can run just a small piece on one machine. Or you can run just like some half of it. You can basically split it and choose which part of your test to run. So yeah, this is one of the best features.

So let's move on. So should I migrate my testing suite tomorrow? No. You definitely should not. It's great to have something inside the runtime itself. But if you already have your test suite, it doesn't make sense to put a lot of effort into migrating it. But if you're starting a project from scratch, then it's something that you might be interested into. OK, so let's talk about the cons of the test runner, what work we are doing, and how we are improving it. So small ecosystem. It just has been released. The LTS was introduced last month. So it's very new. And therefore, small ecosystem, no libraries, no guides. So everything is new. Not many APIs, so it's not complete. It's not bloated like some other testing frameworks. And no native TypeScript support. You always need transpiler like Node.js, TS Node, or TSX, or whatever.

So upcoming feature, planning. You can decide how many tests you want to run. And if they do not match, it throws you an error. Black timers. And it actually got implemented and released a few times ago. So test runners is running very fast. It's evolving very fast. New APIs are added every day. Community is very interested in it. So expect it to improve soon.

And exit on failure, bail. It's a cool feature. I actually tried to implement it myself, but failed because of complexities. And so I will try again in the future to implement the bailouts on failure. And the last one is actually expect failure. So negative testing. So if we actually expect a test to fail instead of a test to pass. So I suggest you to actually use these features. If not all of them, but at least watch and code coverage, which are very useful. They are experimental. So please use them and provide feedback so we can improve it for everyone.

And that was it. You can find all my links and work about Node.js. I will update about new features on Twitter, LinkedIn. That was it. Thank you. Really, really enjoyed your talk. So many new features. And one thing I always like asking people who build on things like this. What feature do you think developers don't know about yet, but they should be most excited for? So definitely the code coverage.

It's very fast. It's faster than any other libraries you're using. And it's the most accurate because it actually uses V8 internally. So yeah, that one definitely. I'm really excited to try it out. I actually haven't tried it out just yet. So I'm going to try it out. And also I've actually got an idea for a talk that I want to give based off of this. I want to thank you in advance.

All right. Let's jump into the questions from the audience. We've got the first one, which is about the reporters. And they ask, why is there no built-in JUnit reporter? It's usually the most common use case since it's the de facto default format for machine-readable test results. So that's because we started from TAP, which was the one we took inspiration from. And yeah, probably because no one in the Node.js maintainers team is interested in JUnit. So we haven't built it yet. But if there is interest, we surely build it. So get on GitHub and let them know that you want JUnit test reporters as well.

I have a question, and this is something that I also am very interested in. So thank you, Norbert, for asking this. How does sharding work? Can it shard based on how long the test will run? Because I saw you had that kind of fraction. I was really interested. How is that fraction kind of made? Do you just make up on the fly, and it does one fifth of the tests? Yes, it actually does one fifth of the test. And you can play with the fraction, depending, for example, if you have five machines, then you split a fifth for every machine. And it's very handy. That is really cool. That is really cool. I'm really excited to try that out as well. And then another question, and I think I saw a piece of code that had this in it, can the Node.js test runner parallelize tests? And if so, how do you control it? Yes, it can. It can parallelize tests with the...

There is a flag concurrency. And so how do you control it? You use it if you actually want to run tests in parallel, and you lose some kind of control because it will not execute them one by one. This is a choice that you have to make based on your use case. The classic it depends answer. There is always one it depends answer.

And then another one about test runner speed. Was it a concern when building it? Because this is a big part, especially when you're testing a lot, it becomes a big part of your time just waiting for tests to complete. Definitely, it was one of the main focuses, and it's fast. I did not add benchmark in my presentation, but you can find them on the Node.js project. It's very fast compared to the alternatives. That totally makes sense.

And another question as well, about code coverage, it works well when you use the loader... Does it work well when you use the loader mechanism to support TypeScript? No. That's one of the bugs that we are trying to fix right now. It doesn't work well, but we will make it work soon. And obviously, I just want to preface this by saying this is like the early days. And how are you kind of gathering feedback about features maybe developers want or features you maybe want to spend more time on? Exactly. Node.js became LTS last month. So it's only been in the market for one month for the public. We have been using it internally. We gather feedback by mostly maintainers that create new packages and use it in their new application. But we would really love end users to provide feedback on Twitter, on GitHub, on whatever. There is a discussion section on Node.js on GitHub where you can add ideas, improvements that you want for the test runner. Yeah. I love it. And I love it when there's like sort of an open feedback loop between the people working on projects and the people who use them. So please do contribute. One thing I loved about one of your slides is you had the slide and you had Bunn and Dino just like peeking in. It reminds me of this question where it's just thoughts on Bunn's test runner. Yeah, it's very fast.

I love Bunn. It's great and it's helping the whole community because it allows us to see what the community wants and how other Node.js alternatives are doing. So it gives a lot of inspiration and feedback through also other runtimes. Yeah. Yeah. I love how it's an ecosystem and not just a competition between all these as well. They all help each other and build each other. Yes. This is very important. And right now we have created a working group called WinterCG where we all work together to standardize the ecosystem and not be competitors but like allies in the ecosystem. I love that. And thank you for all of that hard work that you're doing.