Are the (module) types wrong?

All right. Hello, everyone. And thanks for joining me to talk about modules. My name is Andrew. I've been on the TypeScript team for about the last four years, and we have a lot to talk about today in a short amount of time, so that's enough about me. And I think we all know that module system complexities are super interesting and super fun, so no time to sell you on it, but if you watched Mark Erickson's talk and couldn't get enough, this is definitely the talk for you.

So before we get started, there is a little bit of terminology we should cover. I'm going to use the term ESM or the acronym ESM about a hundred times in this talk. That stands for ECMAScript module. ESM is the module system that was added to the JavaScript language specification itself in 2015. And you can recognize it by the import and export keywords. The other module system we'll discuss is common JS or CJS. That's the community standard that was popularized by Node before ESM existed. If you've ever seen require and module.exports, that's common JS and it's still supported by Node.

I'm also going to have to say a few file extensions during this talk. And since those can overlap with other acronyms, I'll try really hard to remember to pronounce them with a dot at the beginning. It will be somewhat helpful if you're at least vaguely familiar with some module resolution options and typescripts. Node.16 and Node.Next are currently identical, so I might call them by either name. They're the only good choice for modern Node, and they don't just mean emitting ES modules. That's a common misconception. Historically, the one called Node was what everyone used for pretty much everything, but it really hasn't kept up with modern features.

We're going to take a look at a bit of a confusing typescript error here. I'm looking at the NPM dependency called Helmet. I've just copied and pasted the README example here. This builds with TSC and runs in Node just fine. You can see that I'm currently compiling down to common JS. If I want to move to ES modules here, I can do that by adding the type module field to my own package JSON for the project. But when I do that and come back here, I'm now getting this confusing typescript error that says helmet isn't callable anymore. If I mess around with this enough, I'll see typescript I'm running. This doesn't look right to me, but let's run it and see what happens.

Yeah, so this crashes in node. On the other hand, if we go back to the way it was where typescript is complaining, but we run this, this works just fine. So there seems to be a disagreement here between typescript and node, and we can't satisfy both at the same time. Seeing this, you might reasonably wonder, is this a typescript bug, or are the types wrong?

In order to determine that for ourselves, we need to learn a few pieces of background first. The first one is how typescript and node even know what file we're talking about when we say import helmet or require helmet. It's going to be easier to explain this one, this exports field and package JSON when we're back in the code, so let's leave it behind for a moment. The second is module format detection. I think I mentioned that node supports both ESM and common JS, so it needs a way to determine which file is which format, and it does that purely by file extension. A .mjs file is always going to be ESM, a .cjs file is always going to be common JS, and a .js file, we have to look up at the nearest package JSON and if it has this special type module field, it's going to be ESM, otherwise common JS. All right.

The third thing is the relationships between the different kinds of files that JavaScript deals in. When you run TSC on an input.ts file, you get one output.js file and one output.d.ts file, which is called a declaration file. When you publish code for other people to consume, you usually want to ship them the raw, compiled JavaScript. That way their run time doesn't need another compile step before they run it. But then if that user is also using TypeScript themselves, or just an editor with smart TypeScript-powered language features, you want to give them this declaration file, which contains everything that it needs to know about the JavaScript file, including the types that have been erased from the original TypeScript file. Because these two outputs are produced together at the same time, when the user's compiler or editor sees a declaration file, um, it actually doesn't even need to go and look and make sure that the JS file exists. Because of this relationship, it can simply assume it exists, know everything it needs to know about the structure and the types, and also the file extension. When it sees .d.ts, it knows that there must be a .js file there. And the same thing holds for the format-specific file extensions that we were discussing earlier. A .mts file produces a .mjs file, and we get a .d.mts file alongside that. And then we have an analog for the CommonJS versions here. So it might seem like I'm forcing a relatively simple point here, but you'd be surprised how many problems trace the root cause to breaking this relationship somehow.

The next thing we need to learn is, what happens if we were to try to compile this export default down to CommonJS? Default exports in ESM are just a special form of named import with special syntax attached. So all we're going to do is just attach a named property assignment here on module.exports with the value that we're exporting. And then we'll also define this flag here that just says, hey, I've been transpiled down from ESM to CommonJS. Our declaration file, on the other hand, kind of retains this esm syntax. Okay, almost done here. The final thing that we need to look at is what happens when we try to import that same transpiled export default that we just looked at. So here we have our export default transpiled down to CommonJS again. And if we're importing this in another file that is also being compiled down to CommonJS, then it makes sense that what we should get with a default import there is going to be the value that we default exported.

The default import and default export need to match up. So I should just get hello world. And the transformations that are applied as we compile this from a default import to a require statement ensure that that holds. On the other hand, when we do this in a real ES module in node, node doesn't understand that our exporting module here is pretending to be ESM. And in node, when you default import a CommonJS module, what you'll get every time is just the whole module.exports object.

So over here, you can see I've logged the module.exports and I get an object with a property called default. And that's the same thing I would get in node if I were to do a real default import of this fake default export. I'll get the module.exports property with this default. OK, so we should know enough now to be able to diagnose this error for ourselves. If we want to understand the difference between TypeScript and node, we need to know which files each of them are looking at.

So first, let's look at Helmet's package.json file. OK, we have this exports field again that we briefly mentioned earlier. What we're going to do here is take a look at this dot key. And this matches the path. So since I'm importing Helmet and not like Helmet slash utils, the dot path matches and we'll go into here and look at some conditions. So let's pretend we're resolving as node first. We're resolving an import, not a require. And so what node is going to do is look at each of these conditions from top to bottom and go until it finds one that matches. So import matches because we're resolving an import. So we'll go into this object and see some nested conditions. Types doesn't match. It's not one of the conditions that node sets here. So it's going to skip this and check default. Default always matches everything. So note is going to resolve to index dot MJS. Now if we're resolving as TypeScript, we do the exact same process with one exception. TypeScript always matches on the types condition. So usually you don't need to set the types condition unless you're putting all your declaration files in a completely separate directory away from the JavaScript. But it's here and it matches. So TypeScript resolves to index dot D dot TS.

By the way, there is a command line flag for TypeScript for TSC called trace resolution. So that will spell out all of this that's going on. So you don't have to remember this if you just want to see what TypeScript is resolving to.

So now that we know what files are in play, let's talk about format for a second. We know that node on this in JS file is going to interpret this as ESM because of the file extension. The dot D dot TS file, on the other hand, we know this represents a dot JS file because of the triangle that we looked at earlier TypeScript file types. And that JS file it's format would be determined by the package JSON type module field, but there isn't one here in this package JSON anywhere. So we know that that JavaScript file must be common JS. So here we have a format disagreement.

The node file index dot MJS we know has to be ESM and the type declaration file we know is representing a common JS file. And if we take a look at the types themselves, we're concerned with this export helmet as default part. And this kind of finishes explaining what we were seeing. If TypeScripts knew that this file was attempting to type a real ES module, then it would see this export as helmet as default and know that when we default import that it's going to work as we expect. You know, we have a default export. We default import it and we can use it no problem. But what TypeScript thinks is that this file is representing a common JS file that is doing exports.default equals helmet. And if that's the case, when we default import it in Node, we still are going to need to do that extra .default in order to access that exports.default property since we're just getting the view of module.exports as a whole.

So if I'm the library author, it seems like everything's working at runtime already and I just need to change something about how the types are set up in order to fix this problem. But to let you in on a minor spoiler, the library author for Helmet has already fixed this problem and figured it all out long before I started this talk, which is partly why I'm comfortable kind of picking apart a little bit in a conference. So instead of repeating the work that he did and fixing the types to match the existing JavaScript, it might be more fun and instructive for our purposes to try to break the existing JavaScript to match the existing types. So the way I'll do that is to first start by making extensions match. So we know this index DTS file represents a .js file, so I'll go ahead and delete the M here. And our demo isn't going to touch the require side of things, but I'll go ahead and make that match as well. All right, so index.js that we've just put in here doesn't actually exist yet. So I'll go ahead and make that. And as we said before, we know that this file has to be common.js. So we'll start by just copying the index.cjs content into here and assume it's pretty close and then we'll check it. So we saw that we had that export helmet as default, which we see represented right here as exports.default equals helmet. So this looks good. I think the only problem here is that this line is actually overwriting that.

So let's just delete it. And I think we should have agreement now. So coming back to our input file, recall that we didn't touch the types at all. So we are still expecting to see this TypeScript error, but what we should see is that if TypeScript is giving us an error, Node also is going to crash. So let's build and run that. And yeah, this time, helmet is not a function. And if we instead do what TypeScript is telling us we should be doing, helmet dot default, and then build and run this, this now works.

Okay, so why did we go through this kind of silly exercise in making a probably undesirable change to the JavaScript in order to match the types? Well, sometimes when people bring a problem like this to me and I explain what's going on in the types, they'll say something to the effect of, you know, it seems like TypeScript is just being overly pedantic here. Clearly it could find the types before, and my CommonJS and ESM JavaScript are nearly identical. So if TypeScript can find the file, just, you know, know what I mean and don't make me jump through so many hoops. But what we've shown here is that the types aren't wrong per se. They're not invalid. They just describe a different library than what we had here. They, they described the one that we just made via our changes. And this pattern of, essentially just a CommonJS library assigning to exports.default is actually pretty common. The library author may not have intended to do this, but I've seen it in the wild several times. And when the types for that are correct, it successfully warns users who are importing in Node.esm that they're going to unintuitively, but legitimately need this extra.default in order to access what they think they need to access. So while all of you are now experts on module resolution and interop, it might be a bit much to ask all users or even all library authors to be experts on this stuff.

So I made a tool that can do some of this analysis for you. rthetypeswrong is a tool that downloads an NPM package and then for each module resolution mode supported by TypeScript, it will go through and do essentially the analysis that we just did by hand. It will check what the TypeScript compiler is resolving to and what the runtime is resolving to, analyze each of those files and check for disagreements. So here we were looking at Helmet 6.1.2 and we see that when we are importing from ESM in Node 16 mode, we resolve to CommonJS types but ESM JavaScript. And that is exactly what we saw. And I mentioned that Helmet has already fixed this. So let's take a look at that. In Helmet 7.0, we see no problems and CommonJS is resolving to CommonJS and ESM to ESM. So this looks great. And since we didn't bother to fix the types ourself, we can see how the library author fixed this. Here we see that both the index.cjs and index.mjs files are now typed with their own declaration file with the proper extension. So TypeScript can be aware of all the files that are there and know the formats for everything.

There's also a CLI version of this tool now. So this is easy to integrate into your publishing process or check a local package. So just for fun, let's try that on the changes that we made to Helmet. So I'll run attw and then pack, and we'll run that on node-module-slash-helmet. And you can see here that we have fixed the disagreements, no problems found between types and JavaScript, but unlike the earlier version, but unlike in Helmet 7, we can see that the ESM version of things is actually resolving to the Common JS version of the library. So that's not a problem, but it's not the fix that Helmet probably wanted to make, which was to, you know, make ESM resolve to the ESM that's already there.

So the cool thing about doing all of this analysis on npm packages is that it gives us a natural way to look at a lot of data over time. I ran rthetypes wrong on the top 6,000 or so npm packages as they existed on the first of each month since January of last year, broken down by a module resolution algorithm. So we can see that things are kind of improving for users in all groups here, with the slight exception of the recent uptick in node 10, which just comes from packages dropping support for this. On the other hand, things are not looking great for node ESM users, with over a quarter of types dependencies showing some sort of error. So this kind of puts us in a tough spot, because if the community is going to migrate to ESM, and I think that it should, but regardless, it's already happening. It would be really nice if we could encourage users to make that transition first before libraries, but that's a pretty tough sell as a TypeScript user if making the switch is going to break a quarter of your dependencies.

If we break down the problems that we're seeing in that ESM list of problems here, we can see a little more nuance. This green trace is the kind of problem we were just looking at where CommonJS declarations are attempting to represent ES JavaScript. So this does seem to be on the rise in the long term, but on the other hand, it's kind of trading one for one with this orange trace, which is untyped resolutions. This means the package was shipping type declarations, but TypeScript wasn't able to find them at all. And that's a worse problem to have, and the aggregate numbers don't really show that. We don't have time to get into what each of these problems are that are the type strong can detect, but I will point out that the second to most prevalent problems here, unlike some of the others, I think these are relatively easy to fix most of the time. Some of the others require fiddling with build systems, and these are often just a one line fix. They have to do with mixing up export equals and export default in declaration files. So where do all these problems come from in the first place? The gaps in this timeline are the best explanation I can come up with. Writing ESM syntax became incredibly popular really early in 2014 before the ESM final spec was even published in 2015. And this is normal, trying out early language features is part of the TC39 process. But what we saw was a really rapid adoption in writing ESM code in the community with a long lag before implementers were able to catch up. And then another long lag after that before TypeScript could catch up to where Node was. So in all this time, whether they knew it or not, the tools that let you write ESM syntax and output common.js were kind of encoding these implicit assumptions about how those two formats were going to interoperate in real runtimes later on. And not all of those assumptions turned out to be correct. At the same time, between 2019 and 2022, when packages were wanting to ship ESM and type declarations, they kind of had to come up with workarounds for TypeScript's lack of support here. But once TypeScript caught up and shipped the Node-16 and Node-next modes, those workarounds kind of became problems themselves. So it's important to understand when I say the types are wrong for some package, that's not a judgment about what was published in the past.

It's only an observation about what works or doesn't work now. And in fact, a lot of the problems that we see can trace their roots to TypeScript's lagging support for Node. So this problem space has been kind of my main thing for the better part of a year. And up until now, most of what I've been doing has been in service of just getting information out there from Are the types wrong to official TypeScript docs. I think that phase of my work is finally kind of wrapping up and I have some improvements to Are the Types Wrong plan that I'm excited about. But I anticipate kind of shifting into looking for ways to make a more direct impact on some of the existing problems that we see out there in popular packages.