Levelling up Monorepos with npm Workspaces

Hello, and welcome to leveling up monorepos with NPM workspaces. Let me start by introducing myself. My name is Rui Aduno. I'm a software developer in the NPM CLI team at GitHub. I have been working on NPM CLI for almost three years now. I worked on the NPM v7 rewrite and a lot of those features that we're going to be talking about here today and especially in workspaces. I'm super excited to be sharing some of that work with you here today.

Let's start with an overview of what we're going to be covering here today. Let's start with an intro to what is in NPM workspaces and then cover some very practical examples trying to get as close to real-life usage as possible and also note some of the stuff that you should be looking forward to. So, let's get started.

NPM workspaces. What is it? So, let's start with workspaces, which is basically this concept introduced by Yarn a while ago and basically it is a way to help you manage many packages within a single repo and for NPM, NPM workspaces is kind of the broad name we gave, the set of features that help you achieve that, basically help you manage multiple nested package within a single top level package. So, it's going to help you centralize the installation of all the dependencies into a single node modules folder. So, you're going to end up also having only a single package lock file, and there are some advantages to that, it is definitely the ideal way for managing monorepos, and thanks to that you can have a single place to manage all your issues, and basically manage your project and your community. But also, into the technical side, you can also centralize, have a single place to run all your tasks, building, anything that you can kind of imagine, you kind of need everything together to get your project running with all the packages, you can have it all in a single place. So, it might help a lot depending on the style of the project you're trying to achieve. Another thing I would like to note here is the timeline, just to give a little bit of this notion of how we've been iteratively improving on NPM workspaces. And you can see it all started in August 2020 with the release of the 7.0 of the NPM CLI that first introduced support to installing NPM, the workspaces. And then in version 7.7, another big one, which kind of first introduced the configuration properties of a workspace or targeting all the workspaces and the first command that support them with NPM Run and NPM Exec. So there were many more, but I just kind of wanted to illustrate how we've been improving. And you can definitely wait to see more improvements coming on NPM Workspaces.

So let's get started with some examples here. I have this project that I have on the left-hand side here, that is a simple app that I put together trying to get as close as possible of a real-life project. This is kind of like a web app that consists of two different services. So you can see I have my user sense service, my web app service, and I even configured a third workspace here, which is the slides that you're seeing right now. So they're all part of this monorepo app that I'm managing here.

And with that I'm hoping to put some examples that are real close to how you would use this in your real life. So you can see they all have a bunch of dependencies here using NextJS, the services using Fastify. So I can just quickly start by running NPM Ls here, and we can see how Ls is a command that is aware of workspaces, and it's going to highlight each of the workspaces in my project, and even list the dependencies of each workspace when I run NPM Ls. So it's the first command to know that has first class support for workspaces.

Moving forward here, I'd like to highlight NPM Init. It's probably the best way of just starting your workspace, because it's going to make sure every step that is needed is going to end up being there. So basically, everything you need to track a nested package as a workspace is make sure you have the folder with a package.json inside it, inside your project, and then just add that folder name to your workspaces field of your package.json in the top level. So using NPM init is going to make sure that those requirements are there. So I'm going to run a quick example here in my sample app. I'm going to create a new workspace. Let's say I'm going to call it the website. So I can run NPM init. I'm using dash y here to just accept all the defaults. And I'm targeting a website. So that's going to create a website folder with a package.json file inside it. It's going to print the contents of the package.json file over here. And as I mentioned before, the other really important point that NPM init takes care of is placing the reference to the website inside my package.json file in my top level folder. So with that, it is all set up. If I npm install here, now the install tree is going to be tracking the website. And if I run npm ls again, I'm going to see website listed as one of the workspaces here highlighted in green. So that is definitely the recommended way to get started with a new workspace inside your project.

And we can move on to another very important aspect. How do you add dependencies to one of your workspaces? So a real quick example here again. I'm going to shift around the workspace configuration here, so you can see that you can practically go anywhere. So I can just declare an npm workspace, I'm going to target the website I just created.

And let's say I want to install a package, I'm going to install this Simple Slider package into my website. So I can run that command, so what it's going to do, it's going to install the package name Simple Slider from the registry. It's going to put it as a dependency of my website workspace. So I can look up its package JSON and I can see that the Simple Slider is declared here as a dependency. But if I look up the nodemodules folder inside the top level of my monorepo app, I can see that Simple Slider was put there.

So that's basically one of the most important concepts with the NPM workspaces, is that everything tries to be hoisted as much as it can, it only avoids that hoisting or that bringing one level up each dependency, if it finds duplications. So if there are duplications, then you might end up with a node modules folder inside your workspace that deduplicates those packages so that it can work as expected.

So all that to mention, one thing that I've seen as a common repeating with workspaces is that some people used to have this no hoist configuration option. That was an option that Yarn introduced in order to basically have workspaces say, okay, I need this dependency, but I need to make sure it is inside a node modules folder inside my workspace. So because of the nature of some module, some package, they really need to rely on the fact that they are at that position in the file system. So that's something that Yarn came up with in order to basically just give a scape hatch to users to kind of have a little bit more control over where it's placed in their dependencies. So for NPM workspaces, we are currently tracking that as a new feature. We're discussing it in your RFC processes. We are looking forward to provide users with a way of doing it. We're just not sure yet if it is going to be using that same no-hoist config value or is going to be something different. But meanwhile, I wanted to basically provide a way to unblock yourself if you ever hit that scenario. In my sample app here, I actually hit it. I'm using this Prisma client, which is this library to manage. Basically it's an ORM for databases and I'm using it in both my services here of this sample app. When it got hoisted to the top of my app, it basically didn't work. So a quick way of unblocking me and making it work is to duplicate it and have it being in both of my services folder inside their node modules. One quick way to achieve that duplication was basically that I put some mocked packages here that basically would conflict with the namespace of the package they're using. So I used Prisma and Prisma Client and declared them using NPM-liasis and pointing to just something else. In my case here, I'm using this Abraa module, but it can be anything really. This is in order to make sure that when I declare in my user-sync service here, I have it inside my package.json. I just have the correct declaration of the expected versions I have for these libraries so that when NPM installs it, it's going to be actually putting the Prisma Client folders inside my user-sync node modules. So it's basically a way of forcing that same behavior of no-waste, and it might help unblock other users there. So it's a very practical way of getting you going today. But definitely keep an eye because there is definitely things going on. It might improve very soon in the default NPM CLI support.

So moving forward, I wanted to also cover running scripts. And here, I also place a few examples of how you can run it. You can run it by using the workspace name. Also an example, if your workspace has a scope, then you also have to define scope. But you can also call it using the workspace path name. And basically here, I in my project here, I can show real quick if I target that user sync service workspace that I have, and I run its tests, you can see it's going to run the tests for that workspace. And it's basically the same thing as navigating to that workspace folder and running NPM tests inside it.

Yeah, and I want to basically move forward here to a little bit more complex examples that I put together, basically showing how in a more complex monorepo you can orchestrate your scripts in a way that they can become much more useful. So starting real quick here, following up from the test example I just gave, this is the, this is an example, these are the scripts from my top level package, JSON. And I'm declaring this test Column web app and test Column user sync named after each of my workspaces. And they're basically calling the tests and pointing to each of the workspaces there. So this way I have these multiple scripts in my top level command and I'm going to be using, I'm going to be using this run dash s binary file that, it actually comes from this user line package named npm run all. So I noted it up here. It's a very handy package, helps you kind of run multiple scripts and gives you much more flexibility on the way you do it. So basically here I'm telling run s to run all of my test column targets defined here on my file. So let's jump back to my root folder here and run npm test and see what happens. So you can see it runs the test web app target first, which is running the test inside that workspace. And then after that, it ran the test user sync, which ran the test for that user sync workspace, and you can see it succeeded.

So a little bit more evolved, but kind of following up from this first example. Let's run also my DAB web servers here. So I have kind of similar thing, that call on web app and user sync, and I'm pointing npm run to my workspaces, and I'm running the DAB script of each one of them. But the particularity of the servers is that I need all of them running at the same time, right? So this is also something provided by npm run all, which I'm going to be using to run, using run-p, which is going to be running both of these targets in parallel. It's going to just start them all at the same time, keep them running, so that I can have everything running and actually use my development server, navigate to my UI. Let's try it real quick here. I'm going to run npm run dev. You can see it's going to run all the scripts and their dependencies. So I'm going to advance here. This next line is actually pointing to the server running on my local host. So now once everything boots up, I can see my user list is back up again. I can see some users.

I can click around, navigate, and I can see that my web app is working as expected. So here I kind of want to highlight this new command. Somewhat new. We kind of introduced it end of last year in version 7.20. It's a command to help you set and retrieve keys, values from your package. So it's not only useful in the context of workspaces, but it's super useful when it's used in the context of workspaces. So it can be used to, let me highlight here real quick. If you run like npm pkg get, it can be just a simple package. It's going to basically print out the contents of package.json and you can also filter out by properties, let's say name, and then I get the name of my package there. So when you really bring on the support to the workspaces properties, it can actually get really powerful, like in this example here, I am going to retrieve the name of the version of all of my workspaces. So this config option here, //tapus, is basically a way to say, okay, target all of my configured workspaces. So if I run that, you can see it returned the name of the version of each of my workspaces and they're even keyed by the workspace name. So it can be really useful. And also to highlight a little bit more of how they can be useful, let me set also some data into these package.json files. So let's say I'm managing this project and it's an open source project. I just want to surface info about how users can find my work. So I can go ahead and use npm pkg set. Let's say I'm gonna set a following key to all of these package.jsons and I'm going to point it to my GitHub sponsors URL, and I'm going to target all of the configured workspaces. So I can run that and if you look up, let's say, my user is saying package.json, you can see that the founding info is there, so same thing goes to my web app package.json. So it can be an incredible part of the way of just help you manage all of that data in your package.json across all of the configured workspaces. Also highlights here that NPM version, NPM publish also have support to workspaces. So if you want to cut out, let's say, a new patch version of a workspace, that's possible today. And something to keep in mind is, as you can see here, I've bumped my version to v1.0.1. I can look up my package JSON there and see that the version is there. But one thing to keep in mind that is a bit different now from the way NPM version works by default is that there's not a lot of committing and tagging when it's running NPM version. So something to keep in mind, but you can definitely be looking forward to improvements to NPM version specifically. So to kind of wrap up everything I wanted to run a quick example here. I'm using NPM exec. It's basically NPX. It kind of got promoted into a sub command in NPM CLI since version 7.

And I wanted to kind of like create real quick another workspace. Let's call it the print current working directory. And I'm going to be using exec to basically round this workspace as a command across all of the other workspaces I have configured, just kind of putting everything together using all of the commands that we just saw. So let me jump real quick into that folder and start an index.js file. And this is going to just basically print the current working directory. This is going to be useful because I want to highlight the nature of how npm exec or npm run. They're running in the context of each folder. I want to print the current working directory just to make sure of that. So next I'm going to npm pkg set a bin value of index.js to that just-created printcwd workspace so that it properly tracks that binary file, index.js. Now I'm going to npm install to put everything together, make sure my bin is properly linked to my node modules folder. Now I can npm exec, and I'm going to be calling my module and I'm going to tell it to run inside each of the workspaces. And you can see I got the expected result here. It basically printed the path for each one of my workspaces when it executed inside them. So just a quick example here to kind of close, wrap up and put everything together, how you can use all of these commands. So if you want to learn more about it, definitely look up the official docs. We have a section exclusive on the workspaces, but there is also documentation across each of these commands explaining a little bit more how to use them. And if you want to find me, we are on Twitter, GitHub, and here's also my website. So I really hope this was useful and very good real life examples, and I hope you enjoy. Thank you. That was a terrific demo session. So... Thank you. Okay. So you asked the question, what was and introduced a support to NPM workspaces and 65% say it's 7.0. And- Yeah. What was the correct one? Well, yes, it was during 7.0, but I think I gave out the answer during the slides. So my bad. But I- Yeah, I hope most people also kind of got that before seeing the slides. Yeah. I mean, it's good, even if they got it with your talk, right? Because they learned something.

Yeah. Some did. Like, you know, got confused with V7.7. Okay. But, yeah, impressive. Like, what do you think about the results? Like, because 65% got it right. Yeah. Well, I think we're doing maybe a good job of communicating that. I do know that the 7.7 was an important one, because it's kind of where we added a lot more support to some of the commands to actually work on workspaces. So the two of them are very important ones. It's just that it never happened in version 6. Yeah.

Awesome. So let's take some of the audience questions. So Sebi asks, what are the main benefits or drawbacks of using NPM workspaces over YARN or PNPM? Well, that's a very good question. I think it's more of a giving choice to the users, right? Empowering the users out there to basically stick with the tool they like, right? And yeah. NPM has been around for a long while, and workspaces is definitely something that YARN pioneered a long while ago, and now we're giving the choice to NPM users to also kind of have that under their seatbelt, I guess. So it's more about choice, I'd say.

Cool. On building upon this question, I do have another question, because, you know, like, I don't know, like everyone's voice is going bad now, I guess. Like, the climate is changing, whatever. So, like, how does NPM workspace, you know, like, work as compared to Lerna? Because Lerna has been around for a while. Right, right. Yeah. That's a good question. Like, Lerna kind of worked around the limitations of NPM before NPM supported workspaces. So, it basically was very intensive because it was kind of doing an install for each of your workspaces. It was managing a separate NPM install for each one of them and making sure everything kind of worked well together. And yeah, basically, with NPM workspaces, you get that natively. And, like, the NPM CLI, when it's installing your project, it's kind of capable of being aware that each of these workspaces are there.

And they do also have dependencies. And you can kind of leverage that in order to hoist everything to the top, kind of save a lot of space if you're going to duplicate a lot of things with the learner, right? So, there are definitely, like, a lot of benefits compared to learner in the way that the installer can just handle everything as a single thing, as a single installation, like, make sure everything gets to the right place, and the workspaces get properly synced so we can work together. Awesome, yeah.

So, actually, yesterday we had a great talk by Sultan on this pnpm. So, there, we were talking about the isolated dependencies and, you know, hoisted dependencies. And I think it's already in Yarn also, Yarn 3, but it's optional, and it's going to be de-facto by Yarn 4. So, I was wondering, like, do you have anything about, like, npm? Is npm also, like, going that path, or something like that? Yeah, we do have this thing from Microsoft, who was actually working on this, when we're calling it the isolated mode, which is basically for, I think people are very familiar with the way npm does the installations, Yarn is kind of following suit on that. And yeah, we are working on something right now that is kind of equivalent of that. So it's, yeah, we call it the isolated mode. And yeah, I think there was even a colleague of mine, like this team from Microsoft, I think we were talking, presenting that in another conference this week too, so we probably will have more news and more content to put out kind of talking a little bit about how that's going to work. That is definitely exciting. I wonder what would happen to the meme of the heaviest object in the universe being the Node modules. Yeah, hopefully we can fix that. Yeah.

So okay, so tell me something like what's like, you know, next for npm workspaces? Because it's pretty new right now. So what's next for this? Yeah, I guess definitely something I wanted to highlight. I want to surface in the talk, right. There are still some flaws and limitations, so you can definitely expect us to be working on this very, very soon. Npm version is already going to have improvements in the next release, next week. It's going to already add more features to help the workspaces kind of link together at the end of cutting a new release for one of the workspaces. So that's something that's going to be coming out in the next release. And we are going to be working on adding that git commit and tack that's missing from cutting a version of a workspace right now. And one of the main challenges is also tackling the possibility of having an equivalent to that no-hoists option, so that if you're trying to use a package that kind of depends on the location, that package is going to be like, so if it expects to be inside the Node modules folder inside of the workspace folder, you can actually have a nice, very nice first class API to kind of do that, right? Have that support. So I think that's like the short-term, like we're definitely looking to tackle these. And I think longer term is kind of going to be stabilizing even more. We have been working a lot on just making sure we kind of polished the tool since we started shipping the support for each of the commands, and how like it's going to integrate with the upcoming features, right? Like the isolate mode we'll talk about, and we have much more on the way, right? And even the PKG command, I think they also like talk about that in the command, like This kind of goes well, really, really well with workspaces, right? Gives you more power over like just working with all of these packages at the same time. You can like, again, PKG sets and a value that goes to all workspaces. So, I think you definitely can expect more synergy between these new features and better workspace support. Yeah. Awesome. I see lots of interesting stuff happening, like really great time I had for toolings. Thank you so much, actually. This is all the time we had for Q&A, but for the audience, you can still talk to Roy in his speaker room, so go to special chat. And Roy, thanks a lot for this insightful talk and for answering all these questions. It was really nice having you here today. Thank you so much. Yeah, it was my pleasure. Thanks. Thanks so much.