Publishing TS Libraries for Fun and Profit

♪ Hi, my name is Mark Erickson, and today I am very excited to talk to you about publishing TypeScript libraries for fun and profit. Mostly excited? Somewhat excited? Okay, look, it's been a really difficult year. There hasn't been a whole lot of fun. And actually, trust me, there has not been any profit at all. We're gonna go through the details.

A couple quick things about myself. I am a senior front-end engineer at Replay, where we're building a time-traveling debugger for JavaScript. Please check it out. I will answer questions pretty much anywhere there is a text box on the Internet. I collect all kinds of interesting links. I write extremely long blog posts. I am a Redux maintainer, but most people know me as that guy with the Simpsons avatar.

So, publishing packages is really simple, right? You just run TSC and npm publish, and you're done. Thank you. Oh boy, wow, I wish it were that easy. This would be a whole lot shorter talk, if it was. Earlier this year I got kind of annoyed and published a tweet where I listed some of the things you have to keep in mind when publishing packages stay. Build artifact file formats, whether to bundle or keep individual JavaScript files, package exports, WebPack 4, Typescript module resolution, different user environments, bundler behavior differences, Node ESM versus CGS, whether to bundle your TypeScript types, and now React's use client. There's no guides. Everybody's borrowing from everybody else, and it's a miracle this ecosystem works at all.

So, how did I get involved in this process? Well, I've been maintaining Redux for the last several years, and as of the start of this year, I maintained five different libraries. Redux core, React Redux, Redux Slunk, Reselect, and Redux Toolkit. And each of these had a somewhat different build setup, but in general, there was a mixture of ESM, CommonJS, and UMD build artifacts. Everything used a .js extension. Everything was being compiled to ES5 for IE11 compatibility. Most of the packages used rollup plus babble, except Redux Toolkit, which used es-build. None of the packages defined the exports field in package.json, and there were a variety of different folders being used for the build output.

So, what does ESM support even mean, anyway? And the problem here is that the ES2015 language spec defined the syntax for importing and exporting, and some of the expected behaviors, but it didn't define how runtime environments, like the browser or node, are actually supposed to handle loading these, or how they're supposed to interop with other module formats like CommonJS. Now, most of us have been writing ESM syntax for years, but when you publish a library, you normally convert it to CommonJS before you publish. And you also usually compile your syntax to ES5, so it works in IE11, unfortunately.

So packaged JSON has a bunch of different fields that different tools look for to try to find the right file. Node looks at the main field for CommonJS files, bundlers often look at the module field for ESM, CDNs and tools like unpackage look for different keys, TypeScript looks for its TypeScript types, and all these different tools have different expectations. On top of that, it took Node years to add decent support for ES modules because they were trying to figure out how it would work with CommonJS.

So there's a relatively new field called exports, and it's supposed to be the fully definitive one-stop shop for where you tell tools how to find your different entry points and different file formats. So you can find a whole bunch of different entry points, you can have nested conditions, like here's where to find an ESM file versus a CommonJS file, you can define conditions like development and production. But the problem is that adding exports is really a breaking change for your package, which means you can only do it in a major version.

So how does Node understand whether a given file is ESM or CommonJS? There's two different ways. One is it now allows you to use a .mjs or a .cjs file extension to declare what type of module it is, or you can add the type module field at the top level, or type CommonJS, and every file with a .js extension will be treated as if it's that type of module. So at the start of the year, we decided to modernize all the Redux packages. We had gotten some bug reports that you couldn't import them properly in a Node-ESM environment. We'd actually migrated the Redux score to Typescript back in 2019 and then never actually shipped it. Version 4 worked fine and we had concerns about shipping a new major version. And we wanted to modernize all the build output and ship modern JS syntax for smaller bundle sizes.

So, based on my research, I thought all I have to do is add type module and an exports field and everything will just work. Right? Right? No. No, not at all the case. I put up a pull request that tried to modernize Redux toolkit as my first attempt. I added type module and exports, I modernized all the build output. I also had to make changes to a bunch of scripts that we had that were common JS files, because with type module those are now treated as ESM files. But I had this and I tried it, and then everything exploded. Well, okay. Mostly things exploded under Jest. Redux Thunk and Emmer both have default exports. And Jest was getting really confused about how to interpret default exports versus named exports. I ended up trying to publish an alpha version of Redux Thunk to work around this, that didn't really help much. I was able to switch to Emmer's named exports and that sort of helped a little bit. I actually got fed up and I actually switched all of our testing from using Jest to VTest because it was supposed to have better ESM support. And that actually went pretty well.

Most of it was just some search in place plus a different config file. So I published the first alphas of the Redux core and Redux toolkit in January, and people probably started telling me how I was doing it wrong. Matusz Brzezinski, who is an expert on this sort of thing, had a whole bunch of different suggestions. People filed issues saying that it didn't work when module resolution was set to node 16 for TypeScript. And someone else pointed out that, well, actually the node ESM thing you're trying to fix just actually really doesn't even work. And I was not a happy camper. I felt like I was getting contradictory advice from everybody. So it was time to do research.

I had one call with Matusz Brzezinski, where he gave his thoughts on what I ought to try. He even suggested, like, is it actually even worth shipping the ESM? Maybe you should just, like, do CommonJS only. I also had a call with Andrew Branch from the TypeScript team, who's been doing a lot of work on TypeScript's module support. And he gave me a lot of details about how TypeScript interprets module files and all the different instructions, as well as some key pieces of information on how do TypeScript and Node know whether a given file is CommonJS or ESM. And it really is about either using the right file extension, like .mjs, or having that type module. And as much as I don't like the .mjs file extension, it turns out that using that actually does simplify a number of things within the project.

So it was very clear that I was in over my head with this stuff and just trying to look at a package definition to figure out if it was going to work, wasn't going to scale. So I needed to set up CI checks to actually see how different tools interpreted things. The problem is there is at least a half dozen different bundlers and build tools, each with their own quirks. So I ended up writing one little example application and then building it with create React app 4 and 5 and V and Next and a couple of different node setups and running that all on every pull request just to see how it was going to interpret the packaging setup. Turns out that React Aria is doing something pretty similar. I also discovered that Andrew Branch had been building a tool called Are The Types Wrong? as an outgrowth of his work on TypeScript's module definition and documentation. Now, originally, this was just a website. So you would either type in a module name or you could upload a packaged tarball, and it will scan it and try to tell you, here's how TypeScript will interpret all the different entry points and type definitions. Now, at the time, there was no CLI for this tool, so I actually ended up writing my own and trying to use that in our CI checks. Later, an official CLI tool for Are The Types Wrong? was added, and I actually still need to switch over and make use of that. But it's been extremely helpful in both local development and in CI to verify that all the package definitions are set up hopefully the right way.

So, a few months later I was ready to give this another shot, and I decided to try working on the Redux Thunk library first because it's really small. But it was still being built with Babel and Rollup, and I wanted to switch over to using ESBuild, so I found a really nice wrapper called TsUp. And I was able to get that set up pretty quickly, and it works pretty well. It's a wrapper around ESBuild aimed at TypeScript libraries. It also has the ability to generate a bundled ts TypeDefs file for you. So I ended up with this package, and it's better, although it probably still needs a bit of work from there.

Now, I mentioned earlier that we shipped ES modules, CommonJS and UMD file formats. What's a UMD file? Universal Module Definition is a really bizarre module format that can simultaneously be used as an AMD file, a CommonJS file, or a global script tag. And it's not that much more effort to maintain, but it felt legacy, and I didn't know if we should keep it. So I looked around, kept asking for advice. And the best advice I could find was a couple people saying, eh, you probably don't need it anymore. Even for something like CodePen, it has support for ES modules these days, you probably don't need it. So I made the decision to drop UMD files from our packages, although I did replace that with a special ESM build that's been pre-compiled to production mode, so it ought to work okay in browsers.

Then I found out that Webpack 4 didn't like the setup, because number one, it doesn't support the exports field, it also doesn't support parsing ES 2018 objects with spread syntax, or optional chaining syntax. And for that matter, you can't have a .mjs file in the main field, it'll choke on that too, so you have to switch to a .js extension just for that. And the problem is that Webpack 4 is still pretty widely used by a number of older build setups. So in order to try to keep from breaking the ecosystem, I reluctantly decided that I'm going to ship an additional build artifact, ESM format, but compiled to ES 2017 in a .js extension, just to keep Webpack 4 happy.

What about typedefs? Well, the version of tsup that I was using at the time will generate a bundled typedef file, but it always gives it a .d.ts extension. And this is a problem, because it turns out that, are the types wrong, reports that as a false CJS warning when you're using Module Resolution Node 16. And talking to Andrew Branch, it turns out that you really should have separate files with actually a .d.mts and a .d.cts extension, so that TypeScript fully knows here's what the types look like when you're running in ESM mode versus common JS mode, because there can actually be some differences. I decided to punt on solving that problem for now. It's still a thing I need to go back and look into. Andrew Branch did go and file a PR for .tsup, so that it will try to generate different bundled TypeScript definition files. That came out in a later version of .tsup, and I still need to try that out myself.

So, here's what some of the packages look like after the second attempt, and this is better, I think, but it probably still needs a couple more tweaks. I probably need to do some nesting for the import and default conditions to specify different type definition files for each of those. Now, Michel Westray, author of the Imer library, had been working on developing Imer-10 during the spring, and he was also trying to modernize that package and dropping some backwards compatibility stuff, and Redux Toolkit relies heavily on Imer. So, I was very eager to try out Imer-10 Beta, but I noticed that the bundle size actually went up a little bit. That seemed weird. So, I actually pulled down, cloned Imer-Repo, pulled down the Beta branch, and was looking at it, and I found that it was using an older build tool called TSDX, and it was still generating a lot of ES5 build syntax and a lot of weird cruft in there.

So, I had learned a lot, hopefully, about trying to package things, and so I actually found a PR that switched it to use TSUp and applied all the same packaging changes that I was using on our Betas, and that worked, and it actually dropped the bundle size significantly by about 40%. And Michelle actually shipped those changes as part of Emerton Final, so that is actually available in the wild.

And then things got more complicated. So, Next 13.4 came out in May, and it ships with React server components and the new app router as the default. And people have been trying to use Redux with this, and unfortunately, things keep breaking, and so we've been getting a steady stream of bug reports from people saying that React Redux or Redux toolkit don't work in a server component environment.

Now, my Redux co-maintainer, Lensweber, actually has been working at Apollo, on Apollo Client, since the start of the year, and he's been doing a lot of research on how libraries on the client side can interact with a server component environment. And so he'd filed a React issue, asking for some advice, wrote an RFC about how to make Apollo and Next work together, and even published an experimental package. Where things got really complicated was that a specific Next Canary release broke Apollo, briefly.

This was fixed pretty fast, but it did spawn a very long and detailed and kind of argumentative discussion thread. One of the Next developers left a comment, saying that packages really need to publish an additional build artifact with a React server package definition inside. And that just seemed like it was really going to confuse things. So Mark Baga said, you need to make sure that the client code gets stripped out of there. Lens and I complained that this was making our job as maintainers way harder, and I got pretty frustrated. And I put in a lot of work. This feels really, really demoralizing.

So a few weeks later, Lens actually put up a blog post titled My Take on the Current React Server Components Controversy. And there had been a lot of arguments and debates floating around online, and a lot of confusion about what was happening. And he wanted to give our thoughts as library maintainers. And he said that we think React Server Components are a really useful technology, but the way this has been rolled out is making it a lot harder for us to help our users, and they're filing a lot more bug reports. There's a lot more about React and Next that we have to understand, and this is making it a lot harder to maintain and publish a library that works with React. On top of that, it really feels like there's been very little communication from the React team about how these sorts of things are going to affect the ecosystem.

So where do things stand today? We have betas of the Redux core and Redux toolkit that are out and live with these changes, and we would like people to try those out. We've got alphas of reselect and Redux slunk, and I have a pull request up to try to update React Redux's packaging. I still need to go back and finish that. We do have a few fixes to try to keep React Redux from breaking in a server component environment, and the packages generally pass the are-the-type-wrong checks, except for that false CJS warning that I still need to go back and look at. So, what have I taken away from all the effort this year? I have packaged configs that mostly seem to work. Trying to publish typedefs definitely makes things more complicated. Bundling my JavaScript ahead of time helps in some cases, but is harder in others. It is almost impossible to keep up with all the different build tools and their combinations and their environments and the unique needs that each one has. And unfortunately, there is no fully definitive guide to how to publish a package the right way.

I've been begging for years for someone who actually knows what they're doing to please write such a guide. No one's done it yet. I've written a long blog post with the same, a lot of the lessons from this talk. Trust me, it's not a definitive guide. It's a here's all the painful things I've run into and I'm opposed.

The ecosystem desperately needs better tools to help with publishing. TS up is pretty useful, although that's just the build step. We could really use some kind of service that would take an example app and a library and build it with the half dozen build tools and tell you how each one works or fails.

And React Server components are a very useful tool but they're also really disrupting the ecosystem. It's a lot more things for both users and library maintainers to keep in mind. And there's no documentation for library authors to know how to deal with this. And finally, the CommonJS ESM transition has been going on for years and it is a nightmare that shows no signs of stopping any time soon.

So if you'd like more information, this is based on a much longer blog post I wrote titled My Experience, Modernizing Packages to ESM. You can find a lot more details there as well as resources to a lot of the things that I've looked at in trying to learn how to try to publish a package correctly. Hopefully this information has been helpful for you, and if you are trying to publish package in today's environment, I'm sorry, you have my sympathies.