The Road to JSON Import Support in Node.js

Hello, everyone. My name is Anton Duhamel or aduh95 on GitHub. I'm here to tell you about the road to JSON import support in NodeJS.

So first, I'm gonna talk a bit about me, so I'm NodeJS Technical Steering Committee member since April, 2021. And I'm also working at Translordit, so that's my day job. And they also pay me to tell you that if you want your work on NodeJS, your contribution on NodeJS core to be sponsored by them, just send an email. It's a good time.

Anyway, so my talk is about JSON modules, so what are they? So it's just a handy way for a JavaScript author to interact with JSON files. So that could be for configuration or to consume an API. And I'm gonna go through the history of JSON modules in the JavaScript ecosystem, and then I will see how we can use them today and what's the next steps.

So the history of JSON modules goes back to the beginning of NodeJS. So first, when Node.js was introduced, there was no module system, no standard module system in the JavaScript ecosystem. Mostly you were using flow boards. And so something has to be made up for this. So Node.js came up with the CommonJS, which is also called CGS, and they had support for JSON quite early. So on this screenshot, we can see it was in 2011. So you can say it was forever ago in the JavaScript world. And the one obvious drawback of CommonJS or CGS, it's not supported in browsers. So the next step is the ESM specification or the ECMAScript modules. So that was part of the ES6 or ES2015 spec. So it was, the ES6 spec is a big spec jump where we went from all JavaScript and more modern JavaScript. And one of its addition was the import and export keywords and the module mechanism that would allow JavaScript code to interact with other JavaScript files. And that is currently supported by Node.js, browsers, Deno, TypeScript, most of the ecosystem. And one notable thing I should add on that, it's browsers in particular enforce that only JavaScript files can be loaded through to this mechanism. And that's gonna be important just later. So on Node.js side, the first implementation landed in version 8.5.0. It was a very experimental stage at this point and mimicked most of the common JS mechanisms. So one of its feature was to be able to import JSON files as well. So when that landed, there has been discussion to add the JSON module support in browsers. So, and that was actually merged in 2019.

So the idea was if, so on a browser, it's modules fetched through HTTP. So if the HTTP response contains a MIME type for a JSON file, it's gonna be loaded as a, and parsed as a JSON file. If it's JavaScript, it's gonna be parsed and loaded as JavaScript. But that was actually reverted a few months later because of security concerns raised by some of the implementers. So maybe we can go through the security in the details of this revert.

So let's say you are consuming a weather API and you expect to get a JSON object with a different value inside. So you import it as a module because you can. And what if instead of returning a JSON file, the server, either because it's compromised or because it's malicious, returns a JavaScript file that does something nasty. So in this example, I'm adding a fetch call that would send all the local storage data to another server. But the thing is it could be anything, it's undefined behavior at this point. And browser vendors decided it was not acceptable.

So to work around this, a proposal was raised to instead change to JavaScript language to add another addition to the server to the import syntax, which could ensure that this doesn't happen. So you can see here at the end of this import statement, there's an assert with type JSON, so that would mean for the JavaScript engine that if the response is not JSON data, JSON data, the import statement fails and the code is not executed. So if you're not familiar with what's, how to change the JavaScript language, it would be interesting to talk a bit about that. So the JavaScript language is governed by the ECMAScript spec and that spec is written by the TC39 committee. So that's a lot of acronyms, but bear with me. So the TC39 has a list of proposals that is publicly available that you can see. And each proposal goes through four different stage and the JSON module line posts, you go out and screenshot of that. So as you can see, the JSON modules and import assertions proposal are up there in the stage three packet. So that means it's mostly done. It's ready to be implemented. And they are waiting for feedback before deciding if it will go in stage four, which is the proposal is integrated to the official spec. I wanted to also to give a shout out to the template proposal. If you're not familiar. I'm pretty pumped about this one. So as you can see here on this screenshot, it's an API to replace the data API to work with time and date in JavaScript, which is supposed to be better than the current data API. Anyway, just wanted to mention that. So back to just modules. So now that the ECMAScript spec has a mechanism to ensure that there won't be a security problems when loading JSON modules.

So now that the ECMAScript spec has a mechanism to ensure that there won't be security problems when loading JSON modules. That's back to the HTML spec, which has landed an update of the spec to allow JSON modules when an assertion is present. And now it's time for NetJS to build on both specs to have their own JSON modules.

So you might be wondering at this point, what the HTML spec has to do with the NetJS implementation. So I have an example for you. In this case, when the user imports twice the same module but with a different assertion type, the HTML spec does not define what should happen. It forbids us to resolve both and leaves it to the implementers to decide. In the case of NetJS, we decided to follow the HTML spec definition for how to handle this, ensuring that a module can run the same in the browser or in NetJS.

Currently, NetJS has the import assertion and JSON modules available on v17, and v16 will also come soon-ish. TypeScript also has it in 4.5, Deno in 1.17, and Chromium browsers have added it. However, it is still missing on Firefox, Safari, and Vendors, but hopefully, that will be resolved soon. To use them, you import it as any module and add the assertion at the end. Dynamic imports are also supported. One caveat is that if you want to import only a part of the package, you need to import it using the default import and then access the specific field you're looking for. JSON files can be strings, arrays, or even numbers, so the JavaScript engine treats them as a black box, and only the default import is available.

The ball is now in your court. If you're a JavaScript player, you can test and play with it. However, I wouldn't recommend putting it into production just yet. Maybe wait for it to reach stage four before doing so. But if you have a toy project or want to experiment, definitely give it a try.

And you can also work with dynamic imports. One interesting caveat, you might be tempted to import only a part of the packages, so I think if you wanted to get the version of the package you're in, you might want to do this, but this will not work. You have to import it using the default import and then access the field you're looking for. So, the reason for this is JSON files are not always objects. It could be strings or arrays or even number. So, you have to, I mean the JavaScript engine cannot guess which it is, so it has to treat it as a black box, if you will. And it's only the default import is available.

Yeah, so what's next? So, the ball is now on your court, so if you're a JavaScript player, you can test it, play with it. I wouldn't recommend putting it on production just yet. Maybe you'd like to wait for it to reach stage four before that, but yeah, definitely, if you have a toy project, you were wanting to experiment with that, definitely do that. You can open issues and PRs on the repos and participate to the discussion regarding what's the next step. So, Node.js now has implemented the same web compatibility feature. Do we want to go further than that? So maybe, yeah, more modules, Tomo modules, TypeScript, why not Coffeescript? That's definitely things we would like to add support to in Node Core. We need some folks who have the will to make it happen. So if that's you, you know you. Yeah, so that's it for me. So thanks for tuning in and if you have any questions, that's the time.

Hey, Ardaeen. Thanks for such a great in-depth talk. So we asked the question which you asked to our audience and to see the results. So, okay, so the majority, 73% says that they use it but it gets transpiled to common JS using TypeScript, Mabel, et cetera. Interesting. Yeah, I guess there's a lot of TypeScript lovers in the audience today. It's still nice to see that at least one in five is using it without translation. So that's great. And everyone has heard of it at least. Yeah, I mean, just 0%, I do not know what it is. At least I liked this part of the research. Yeah, that would have been scary to find out knowing a little bit. So, what are you expecting, anything different or what did surprise you here? I'm still a bit surprised that so many people transpiles to common JS, but I guess that makes sense because it's the classical way of doing things.

So, if you have a project that has a bit of a historic that's been in place for a while, maybe ESM wasn't around them using CGS, it can make sense. Yeah, it's still interesting to see.

Yes, okay. So let's take some Q and A's of the questions, audience questions. So, okay, so we have a question on slide number 18, you mentioned it is not possible to import version from a JSON in one line, but it works in two lines. What are the difference between two syntaxes? Okay, yeah, so, yeah, it's a bit, maybe a bit tricky if you're not used to the ESM syntax. When you're working with JSON files with required calls, you can simply to that in one line, import one part, one chunk of the JSON file. And that doesn't work in the ESM syntax, that's mainly for performance reasons. So when the JavaScript engine reads your import, it has to know if this, what you're importing actually exists. And it would be too costly for it to read the JSON file at this stage. So that's why it's forbidden. And it might feel arbitrary if you're not used to it. But yeah, just something to have in mind on the. But yeah. Yeah, so get everything and then get the version out of that. So, okay.

So we have another question. Do you think with TC39 adopting types as comments and native ESM by bundling over time, ESM will gain more users? Yeah, so it's hard to predict. How the JavaScript ecosystem will adapt those things. I certainly hope it will happen because I think ESM is really awesome and is the future of JavaScript. And certainly helping reduce the gap between what the users want to write as code should help. So, yeah, I'd like to answer yes to the question, but it will depend on everyone else, you know. Yeah, it's always tricky. Yeah, I would say, but it can go either way. But yeah, let's see what future holds for that.

So, we have another question from Uswal. So, there is a few concerns people have raised with ESM. For example, it's inability to do things like const, some initialize module equal to require, then module name, and then calling it with some options. What do you think is an appropriate way to overcome this? Yeah, so it's a bit like the first question. Using the ESM syntax makes you sometimes use two lines instead of what could have been one incoming JS.

And yeah, I don't think there's really a workaround for it. And some people really prefer one syntax over the other. That's for sure. But yeah, maybe if you're not used to ESM and you still want to use a common JS. By the way, it's still possible. I don't think it's going away anytime soon, but if you want to use the ESM, get back those capabilities. Yeah, I guess that there's no good answer for that. You have to work around the syntax.

So as I said, it's mostly chosen for performance reasons. Performance doesn't matter as much in common JS because it's only working in Node.js which does not do a network call while ESM works everywhere. So if you have to do a network call every time you import something can add up quite a bit. So performance was more unfocused when designing this syntax. And what was the question exactly? Is there a way to overcome this? Yeah, to, well to overcome, I don't think. So yeah, I guess the answer is I don't think it can be overcome. It's just a shift in perspective on how you import your APIs and how you export them as well.