Node.js: the New and the Experimental

Hi everyone, my name is Beth Brix and I'm a senior software engineer at Red Hat. I gained my Red Hat just four months ago when I moved over from IBM and I'm excited to be here virtually at Node Congress today.

Today, my talk is titled, Node.js new and experimental features. As part of my role at Red Hat, I helped maintain the Node.js runtime. I'm a Node.js Technical Steering Committee member. I'm particularly active in Node.js Release Working Group. So I'm often spending my time determining the content of and producing the releases of Node runtime.

And today I want to spend some time talking about how new features get into the hands of Node.js users and why some features land as experimental first. Towards the end, I'll touch upon a few of the most recent experimental features in the newer versions of Node.js. Node.js is an impact project under the OpenJS Foundation. Formerly under the Node.js Foundation, but when that merged with the JS Foundation, it became the OpenJS Foundation.

You may be surprised to learn that despite being under a foundation and having a technical steering committee, the Node.js project does not have a formal roadmap. There is no single corporate sponsor, it's decentralized, and generally the features and changes that get added to the runtime are a sum of the interests and requirements of our contributors. Typically, our users will only find out about new features when they were released, so finding out via the change logs or the release announcement posts after the release has happened.

There is a distinct flow in which you can expect new features to arrive in Node.js releases. Node.js has a predictable release schedule. We have two major releases per year, with even numbers being promoted to long-term support. And within that schedule, we have three defined release phases. Current, active long-term support, and maintenance. Odd number of release lines will not go through the active LTS or maintenance phases, as they are not promoted to long-term support, and it is during the current phase where the release line will pick up most of the non-major and non-breaking changes that land on Node.js core main branch. During the active LTS phase, only new features, bug fixes, and updates that have been audited by the LTS team to be appropriate and stable for the LTS release line will land.

And then in maintenance, it will be restricted to only critical bug fixes and security updates. We rarely add new features to maintenance release lines. Sometimes we do, but that's very rare and only in a case where adding that feature supports users migrating to later release lines. So you can expect to pick up the newest features first in the current release line, which at the moment is Node.js 15. And typically, for a current release line, you can expect a release every two weeks. After some time, you can expect new features to be backported to the LTS release lines. But not all features will be backported to LTS. Some will be considered unstable to do so, or perhaps the code delta in the release lines is too large to easily be able to bring that feature back. And if you're ever curious when the next release is, the release working group has issues in their GitHub repository containing draft schedules per release.

These are subject to change depending on release or availability. Sometimes we have to accommodate at-home security releases, which can push the schedules out slightly, and generally, we won't schedule maintenance releases unless there are critical bug fixes to get out. And both the timeline on the previous slide and these issues can be found at GitHub.com slash Node.js slash release.

But how can you find out what's in the pipeline? Despite having no road map, there are long term efforts that you can follow to get an indication of what's coming next. We have working groups, teams, and strategic initiatives to dedicate to certain areas of the project. And these groups act as task forces to push forward certain subjects. For example, in the past, we had the modules effort focused on the ECMAScript modules implementation. You can follow the repository on GitHub, but you can expect to be overwhelmed by notifications there. It's hard to keep up. And the project also recently kicked off the next 10 efforts, which is focused on defining our values and desires to guide the direction of the project in the next 10 years. You can also try following some of the active contributors on Twitter, many are at this conference. So that's another good way of keeping up to date.

So that's a bit about how features end up in releases. But how do you know when a feature is safe to use in your production applications? Well, Node.js provides a stability index. Throughout the API, there are indicators of stability. They go from stability 0, deprecated, 1, experimental, to stable. And generally we advise people to stick to the stable features in their production applications.

So going through each of these, the deprecated APIs, they can either be documentation or runtime deprecated. This is an example of documentation deprecation, and in particular, socket bucket size. And you can see it includes the deprecation ID and the version that introduced the deprecation. And documentation-only deprecations can land in minor releases. So, they will not be restricted to majors, you may see them. And because documentation deprecations are just that, only written in the documentation, it can be difficult to know what is deprecated. Node.js does provide a pending deprecation flag. So some documentation-only deprecations will trigger a runtime warning when launched with this flag, so essentially if you just want to run a check on your script or application, whether any of the APIs you're using are going to be deprecated in a future version, you can try running with this process flag and it should tell you. In contrast, a runtime deprecation will, by default, generate a warning that's printed to standard error the first time the deprecated API is used. This is the unhandled promise protection warning that may be familiar, and because this warning is an observable change and printed to standard error, you will generally only see these introduced into the new major versions as adding errors to standard error could break people's applications. There are further two process-related flags, the no deprecation and throw deprecation flags. The no deprecation flag silences deprecation warnings, but be cautious when using this because you're likely silencing a problem that you will need to fix later, as the deprecated API may eventually be removed. The throw deprecation will throw an error.

Onto experimental features. The Node.js project stability index states that experimental APIs may change behavior. The traditional Semver contract that we adhere to for stable APIs does not apply here, even in the long-term support release lines, and this is why we say to use experimental features with caution in production workloads, because the API surface may change. And why do some features land as experimental and others do not? Well, in some cases the most suitable API design may not be agreed upon up front and we may want to get an early draft of the feature out to our users so they can give us feedback and we can evolve the API accordingly. Essentially, it's in cases where we don't want to lock in the API too soon. We have some whole core modules that are still designated as experimental. We have the Async Hooks core module which provides an API to track asynchronous resources, and just this week there are ongoing discussions around marking some of the APIs within this module as stable. We have the Diagnostics channel module which provides an API to create named channels to report message data for diagnostic purposes. It's intended that a module writer wanting to report diagnostic messages will use this API to create many channels to report messages through. We also have the inspector core module that's still marked as experimental. This module provides an API for interacting with the VA inspector through Node and we have the trace events module. And the trace events module provides a mechanism to centralize tracing information generated by VA Node Core and even your own application code.

And finally we have the WebAssembly system interface core module which is also experimental and this provides an implementation of the WebAssembly system interface specification. And we have some specific experimental APIs. For example many of the ECMAScript modules APIs are now Markstable but a few of them are not. The ones that are still experimental include the Lotus API which is used to customize the default module resolution algorithm. TopLevelAwait which allows you to use a wait outside of an async function within an ECMAScript module and JSON modules and WASM modules to support the loading of these.

Policies are another experimental API. What are policies? Node.js contains experimental support for creating policies on loading code and this is inspired by the browser's security mechanism to enforce resource integrity. Another experimental API is just source map support. Source maps provide a method for translating from generated source back to the original. And this is aimed to alleviate some of the observability challenges when using the alternative flavors of JavaScript.

There's also the timers promises API. This was added in Node.js 15 and this provides a promise interface to timers rather than using the traditional pullback. And then there's also the web crypto API which exposes an implementation of the standard web crypto API. And you should catch James Snell's talk at this conference on that for some more information. And a number of these experimental features are hidden behind process flags. And this is to make them opt in. So to actually start using some of these, you will need to start your process with one of these flags. For example, to start with a policy feature, you need to pass the experiment policy flag.

And then we have some very experimental features that are hidden behind build time flags. To use these, you actually have to compile node.js yourself from source. Why do we do this? Well, it's predominantly so that we can work on crafting very early implementations of a feature and build and test them in our continuous integration environment without them being released. If anyone would like to use any of these features, as I mentioned, you would need to build node.js yourself from source, and information on how to do that can be found in the building MD file in the node.js runtime repository. Finally, we have stable features. For stable features, semantic versioning applies. With these features, you can have confidence that we'll try to keep to the API contract. You should only experience breaking API changes in stable features in the next major version. Compatibility is a priority but there's one exception. If a security issue requires us to make a breaking API change, we may land this as some of a minor but this is only done when absolutely necessary. So how and when do features get promoted to stable? It's when the contributors most involved in the feature have confidence in the API and don't believe that further changes are necessary. How can we get there quicker? The more use of feedback we receive on an experimental feature, the quicker we can gain confidence and consensus in the API structure.

And note not all features will ever make it out of experimental. Some have been in the state for several years, and some may end up being removed completely, never having made it out of experimental. This is, again, part of the reason why we suggest using experimental features with caution, because they may never fully become stable.

And there are a number of new stable features in the latest release line. We have the abort controller, we have MPM7, and V886. And it is via V8 that we get the new JavaScript language features, for example, promise.any, aggregate error, string, replace all, and the logical assignment operators all came in via V885. And that's what we're shipping in Node.js 15.

So I'd like to encourage Node.js users to take a look at our experimental features, maybe train them out and hopefully provide feedback on some of them. The more feedback we get, the more we can evolve the APIs, depending on that feedback or just gain confidence that the API structure is appropriate and then promote it to stable. I would say maybe don't rush to use experimental features in your critical applications or deployments just yet because there's always the risk that things may break or be removed or changed. But yeah, definitely check out some of our experimental features and we'd love to hear your feedback. And with that, I'd like to wish everyone an enjoyable rest of the conference and I look forward to joining you all in the Q&A.

We're going to go look at the results of Bethany's poll. So if you remember, Bethany asked us, what version of Node are you using and I added in production and it's Node 14 so pretty recent. From the top of my head, that's not the latest LTS, right? It's the latest LTS, so it's good to see people using production. It's the one where you're keeping up with the features, but also getting that stability for the long-term support policy.

Exactly, exactly. So Bethany, so you're happy with these results? Yeah, it's really good seeing Node 14 followed by 12, followed by 10 in that row, is exactly what we like to see. Because it shows people migrating in a good timeframe, with Node 10 going out of support in April. It's good to see that that kind of tailing off, people using that. Other or older, it's good to see, I think there were responses to that, so it's good to see people aren't using older versions, that are no longer in support. So that 5% of people that are using Node 10, you have one and a half month left. Yeah, stop making a plan to upgrade, and if you hit any issues and we have a help repo, you can post your problems and we're trying to help in the Node community. Yeah, that's a good thing, especially in all of development, of course, that you can always find people willing to help. That's really nice. Help you in your own time, and I really love that about the whole development community in general. At least I can speak from the front-end world. I don't know if it's the same in, say, Java communities, but yeah, I like that about JavaScript at least.

So we're going to go into the questions from our audience and let's jump right into it. The first question I have is from ArthurZ91. Can we have a little bit more details about top-level awaits? How should it work, what are the use cases, and will Timers API replace the cron job libraries? Big question. Okay, so starting with top-level awaits, so that's still under experimental and that follows the kind of EqnScript top-level await proposal that went through TC39. So that's around enabling you to use the module loader to kind of schedule your async code. In terms of use cases, it's when it makes sense for your module to wait on something to load, and that is when you would use it within your module specifically. It's not usable within CommonJS. And there are some pitfalls around using top-level await. There's some good articles out there. I'll try and paste them in the chat afterwards. I can't quite recite off the top of my head all the challenges and pitfalls, but I'd definitely get that resource to you, because it's definitely worth a read, because there are specific use cases when you should be doing this and when you shouldn't be. Cool. And the last question that was in there was, will Timers API replace CronJob Libraries? Sorry, let me just check the wording of that. CronJob Libraries.

I don't think so. I think the CronJob Libraries will still exist. As with all new features, it takes a long time when there's tried and trusted things out there in the community that people stick to, it takes a long time for people to kind of move off it and just use the low-level implementation. There's got to be a benefit in doing so, and I don't know. People won't automatically move over and start using it. No. It can, but the CronJob Libraries are also really useful still.

I have another question, which is, which experimental feature do you think will become stable in Node.js Next? One of them, the enabled source maps that I mentioned in the talk around providing a method for translating the generated source back to the original. That was started in around the Node 12 timeframe, and there's actually already recently, just this week, been a PR race to kind of move that into stable. So I saw that pop up and I was like, oh, no, I need to change my talk, but yeah, that PR is open so just discussion's starting, so it looks like the enabled source maps flag might become stable soon. And the other one that has definitely progressed or is heading in the right direction to becoming stable are the async resource and the async local storage. There's a lot of discussion at the moment, active discussion going on about defining the criteria for that coming out of stable amongst the contributors and collaborators. So the two that I see coming next are the enabled source maps and the async resource and local storage.

Awesome, great new features. Next question, are there any feature additions which, with an interesting backstory that you can share? So more about not about technical implementations, but how it came to life and stuff. Yeah, so one I always found interesting was the diagnostic report feature. So the diagnostic report feature, you can configure your application so it will write a file when it crashes or experiences certain types of issues and what the diagnostic report provides is a feature purpose similar to like Java Core files. So you'd get that file with some information about what your application looked like at the point when it crashed and that can help you diagnose any problems in your application. And that feature was actually originally an external module called node report and the problem we were seeing was that people hadn't installed this module when turned it on, their app would crash but then they would need to go back, redeploy their app with this module installed and the feature turned on and then hopefully replicate the crash to get that file again. So it was often that it wasn't there when people needed it and so what happened is that module the external node report module was merged into node core to really try and increase the adoption and make sure that feature was there within the runtime itself when people needed to turn it on to diagnose problems and very interesting I think that PR was one of our most commented PRs, it had like almost a thousand comments and reviews in the PR so it's a big PR.

Wow so that kind of that feature I didn't know about this feature but it sounds like it could replace companies completely like New Relic and what's the other one like this I forget but like New Relic right so you're killing them. It's really about socializing that people can turn this on and like teaching people how they can inspect the diagnostic file and pull out the information that's actually relevant to their crash. I think that's that's going to be a big challenge to get through that. Yeah but it's cool to have it in. Next question is can you share some exciting new features that we can expect for Node 16? Yeah so node 16 is going to come out in April. We have two major releases. Yes 16 will be the one that's promoted to LTS. In terms of new features and the major releases because we're doing the current release line so at the moment node 15 and we're shipping out new features every two weeks, when it gets to 16.0.0 we don't tend to ship lots of new features because they've naturally been picked up in the current release line and shipped out to our users so the only things that are waiting for the 16.0.0 release are the breaking changes or the really major updates that are going to cause incompatibilities in the API and that's why they've had to wait until that 16.0.0 release. But the one that's typically always guaranteed to go in is a major v8 update, so v8 will likely be updated and depending which version of v8 goes in there it may bring with it some new javascript language features so that that's the main thing you're looking for in a new major release. That's awesome.

Next question is from fried zoidberg. Is the throw deprecation flag appropriate for production workloads or is it more suited for anticipating deprecations during development? Yeah I would personally lean towards the latter. When something gets deprecated you probably don't want your application to suddenly start crashing in production. It's definitely more suited to get the kind of sniff smoke test up front just to see what your application is going to do in development. Yeah so that you're prepared for coming updates.

All right let me see about the next question. A little bit unrelated maybe, but kind of is. I'm always curious about how people get into development. So can you share a little bit, like at the start of the day we asked everyone how long have you been developing professionally. So can you share a little bit about your development story. Yeah sure so yeah since I was young I was always into kind of like digital art and touched upon a little bit of web development but then as I went to college and university I really started to learn and I liked the kind of more code side of things. I did do something a bit unusual, I spent my gap year doing an internship with IBM working in Java websphere APIs and things like that and so it was really that gap year that I spent a year in industry kind of figuring out what I liked and what I didn't and that kind of led me down this path and then post-university I joined IBM, was placed in their Node Runtime team and I've been there five years almost and I'm in the same team but over at Red Hat so that's how I got involved with the kind of open source community and that sort of things.

So your open source work and the TC39 committee that's all stuff you can do in company time? Yep, yep definitely. Part of my role and responsibilities is to help out in the Node community, helping ensure IBM platforms, Red Hat and OSes, are working properly on Node but also doing general helping out and give back to the Node community like helping out producing releases and things like that. Cool and for people that don't know a lot about, well maybe some people are watching that don't even know what TC39 actually means, so can you explain what the process is like for these new Node versions to come out and that new features get implemented? How is that process? Okay so yeah I think there's two subtle things. So there's the TC39 kind of standards that go through the proposals and if adding new JavaScript language features and then we have within the Node.js space, we have the Node.js release working group and they are responsible for kind of auditing the commits and features that land in Node runtime and figuring out which release lines we should ship them in and how often we should do releases and when we're doing the auditing of individual commits we are considering, does this feature look stable? Has it introduced any known regressions? Are there any performance implications reported on this? So it's really taking a more abstract view of the changes that are going into the Node runtime and figuring out whether that's safe to go out in the various release lines.

Cool, and you mentioned that you're doing two major version releases a year, right? And how do you feel about this tight schedule? I run usually in two-week sprints so that's a bit shorter, but that's such a deadline sometimes I feel like it works against you, right? So if you would push back your release, I don't know, two weeks more, then this awesome new feature would be released in the major version and now it's on the shell for half a year. So how do you feel that this system works? Yeah, there are some challenges, and typically if a new feature... so if there's a breaking change within a new feature, then sometimes that change will have to live on the main branch in our repository until the next major release goes out. So that can be a challenge. In some cases, we have tried to, or the contributors and collaborators of Node have tried to backport the new features in a way where they're not breaking, so that we can bring them back to the other release lines. But yeah, it is a challenge. The LTS policy is really based around giving people time to upgrade and giving that long-term support for, I think it's around 30 months, so that companies that can't handle upgrading every year or twice a year have time to do, have a longer time to do so. And this, you mentioned it's long-term support, but is that something that companies have to get a license for, or is that just...? No, this is the Node core's policy on how long we will respond to critical and security updates, and that's what the community tries to keep ourselves to as an aim for everyone. Yeah, yeah, okay. But then again, it's like we mentioned at the beginning of our talk, if you have problems later on, the community will be awesome probably and help you out either way, even if you're still running, let's say, node five, and yeah, yeah. If we need some help, then people will probably still be willing to help. Yeah, for sure. If people are stuck on a specific version, it's definitely something to reach out to the Node community and explain, because if you're stuck on it, there's likely many other users stuck for that reason, and maybe we can figure out a migration path or a change that would make that migration path easier. Yeah, that's really awesome. Again, I don't know about any other industry that's so willing to share their expertise with basically competition, right? So anyone, even running this kind of congress, that people are just sharing everything they know and want to help people get further in their careers. It's such an amazing thing, and I can't imagine any other industry where that's happening. So that's really cool. And with that, I think we're going to end this Q&A. So, Bethany, thanks a lot for giving us some insights in the future. Still haven't thought of my Back to the Future joke, so maybe I'll get it and I'll tweet it later. Bethany, thanks a lot and hope to see you again soon. Thank you.