Understanding Package Resolution in Node.js

Hi, I'm going to be talking about understanding package resolution in Node.js today. I'm Yalcin Zipli. I am a senior software engineer at Sentry. I am a Node.js Technical Stream Committee member and an OpenJS Foundation Cross-Project Council member. You can reach me from my GitHub account, github.com, and from Twitter, axe.com.

So in summary, today we're going to talk about CommonJS, ES modules, ESM, package.json structure, and package.json loader in Node.js.

Let's start with CommonJS. CommonJS is the first and the most widely known module resolution strategy in Node.js. It includes files with require, export implementations with module.export or exports. It can have an extension of .cgs or .js. Require calls does not have to include the extension of the file, and loading a file is synchronous.

So basically, if you want to do a conditional loading, let's say you have a function called read file, and just when this function is executed, you want to load a file and use that implementation in your function, then you need to use require inside this function.

The caveat of this implementation is that in the first line of read file, the lazy loaded module is equal require.reader, that's going to block the execution of this file because it's a synchronous call. And it's going to block the IOU depending on the size of the file itself.

As you can see, the extension is not required, .reader. This means that first the loader is going to check for .gs extension, then .json and so on and so forth. And then it will return a value and then we can run this function.

So in order to load this file without an extension, this implementation makes a synchronous call to the file system in order to understand if reader.js file exists. If not, it automatically checks for reader.json, if it exists and so on and so forth, and then it will return an error if it's not found.

This is particularly slow because in order to understand if that file exists, you need to make additional file system calls and it will impact your performance, small or big, it will impact it.

So despite CommonJS implementation, as I said, CommonJS, in order to include additional modules with CommonJS, you need to call it with requires. Inside the function that you want to require, you need to export the implementation with module.express and so on and so forth. So it can have an extension of .cgs, which is CommonJS, and .jsimplementation. But in ES modules, it's a lot different, the idea behind it. It's introduced in Node.js 8.5.0 with an experimental flag. In order to include files, you need to call it with import. If this import statement is on top of the implementation, like if it's not inside the function, then you can use import blah blah. But if it's inside a function, you need to evade the import statement because import returns a promise. Export implementations with export and export default, and it can have an extension of .mgs or .gs and async loading a module file.

So in the base, it's async loading structure. So in order to load the module conditionally, just like the previous implementation, you need to call it with lazyLoadedModule?equal, evade import, and this makes this particular function an async call, whether or not the lazyLoadedModule.do something is async or not. This is the main reason for this, that when you evade an important module, it doesn't block the I.O.

On top of that, we talked about CommonJS file extensions and MGS file extensions, so how do we actually know what's going on? So we have this package.json file in all of our projects. It contains metadata about a project, but Node.js only cares about 5 of those fields. Name, main, imports, exports, and type. For the sake of this presentation, I'm just going to focus on the type attribute. Type can be a module or a CommonJS.

So how does Node.js know if you're using ESM or CommonJS? So first, it checks for file extensions, Node.js checks for file extensions. If it's MGS, then it's ESM. If it's CGS or JS, then it is CommonJS. So finding the package.json, so we first check for the extensions. If the extension is basically a .js file, then we don't know what it is, then we need to look for the package.json that we are executing this file in. So we need to know the context.

If the extension is basically a .js file, then we don't know what it is, then we need to look for the package.json that we are executing this file in. So we need to know the context. So Node.js tries to find the closest package.json in the directory until root. So it checks for app, my, project, package.json, and so on, so forth, until the root. If it's not found, then it's going to assume something else.

Node.js checks it, and whenever the package.json value is found, Node.js checks the type field in the package.json. And if it's module, it uses ESM loaders, the loader implementation in Node.js. And if it's not, then it's CommonJS. So if type is not present, we couldn't find the package.json, what happens? If type is not present, we check for an experimental flag, experimental detect module. This is pretty new in Node 20 or 21. It automatically checks if the file that you're running is a CommonJS file or an ESM file. This is particularly new because this is an experimental flag, and there are known issues with it, but we are working on it.

So if we have experimental detect module, then we detect if the file is a required, if it's an ESM or a CommonJS. If not, then we fall back to CommonJS. So then we know how our application starts, because we know the initial script is written in ESM or CommonJS. But the issue is, what happens if you want to require a file from ESM that's implemented in CommonJS? Or you want to call a function that's CommonJS or an ESM from a CommonJS module? So what about Node.js module depth resolution? Like if you have Node modules implemented, a package inside your dependency list, and it's implemented in either CommonJS or ESM. So Node.js checks the type field in the package.json of the dependency. If it's module, it uses ESM, otherwise CommonJS. If type is not present, it uses the type of the parent package, which is the root package that our project's package.json contains.

So let's continue with some tips to improve the loading time. Because we talked about all of these package.json loaders, system calls, we talked about the detection, the extensions, and so on and so forth. So if you want to avoid all of those things, and if you want to start Node.js as soon as possible, what you could do is that, if you have an ESM application, you can use an experimental default type, CLI flag, which will automatically eliminate all of those checks, and it will always return ESM. It will not check for the extension, it will not check for anything else, it will just load the ESM loader.