Demystifying npm: What Actually Happens When You Install and Publish

Hi, everyone, it's so great to see you all here. Thank you for joining me here at this very cool venue. And also hi to those of you who are watching this online. Today I'm going to talk about what actually happens when you run npm install and npm publish. And before we get started, a little bit about myself. CJ has talked about quite a bit of it. I'm currently a software engineer on GitHub's npm team. I've been with the team for about two years. I have a total of five years of software development experience. And the path that I took to become a developer is pretty unconventional. I was an elementary school teacher for two years where I taught fifth grade everything. And then before that, I was a classical pianist. Before I joined npm team, my understanding of npm was quite limited. The only thing I had to do was npm install. And I would run it and hope that it works. And if it doesn't work, then I'll delete the log file, delete the node modules, try it again, hope that it works. If it still doesn't work, then maybe I'll go bother some senior engineer and hope that they can fix it for me. Well, now that I've joined the team, I have a better understanding of how npm works. I had the opportunity to look deeper into it. So I decided to create this presentation to share the inner workings of npm install and npm publish with all of you so that you can worry less about npm, spend more time building awesome stuff and earning your GitHub green squares. With that being said, let's start with the basics.

What is npm? Most of the time when we talk about npm, people think about the npm CLI. This is how most users interact with npm. But in reality, npm has three parts, the CLI, the registry and the website. We're focusing on the CLI today, but as we continue, you may see how the npm publish command interacts with the registry and the website as well.

Today's presentation will be divided into two parts. I'm going to start with npm install and then go to npm publish. And I will spend more time on npm install. Let me show you a really common error that happens when you run npm install. Have you seen this before? This is the classic could not resolve dependency, conflicting peer dependency specifically. If you've seen this before, I understand your pain. If you have not seen this before, don't worry. As we go through the steps of what happens when you run npm install, I will explain why this problem happens and how you can mitigate it.

So, what happens when you run npm install? The first thing that happens is the npm CLI will parse the command that you put in. npm install specific package, everything, whatever flags that you might have. It will do an engine check to make sure that your npm version and your node version are compatible and it will also load the npm instance and the config files. Then it will proceed to scan the current state. If your project has node modules, then the CLI will scan the entire node modules directory and build the existing tree. If you don't have it, then this part will be skipped.

The next thing that will happen is the critical part of npm CLI. I'm going to spend more time on this part. It's the CLI will try to build the ideal tree. It first needs a starting point. If your project has a lock file, it will use the lock file as the starting point to build this ideal tree. If it does not have a lock file, then it will use the package.json as the starting point. If your project has both shrinkwrap.json and package.json at the same time, then the npm shrinkwrap.json takes precedence. Once the starting point is determined, it will look at the starting point to build the ideal tree, but it will also use the package.json as the source of truth to build the entire tree. It will look at specific dependencies in your package.json, look at the SEMVer version to decide what ultimately needs to go into the ideal tree.

When it comes to types of dependencies in package.json, there are four types in your package.json that the npm install command cares about. There's the dependencies, what your app needs to run, dev dependencies, what you will need when you are building it, optional dependencies, the nice-to-have ones, and then the peer dependencies, which they expect the host application to provide the package instead of installing it itself. This is also the type of dependency that is related to the error that we saw earlier.

So we're going to dive deeper into this. When a package has a peer dependency, what it means is it requires another package to be present, but it won't install it itself. It expects the user to install it separately. As an example, React is a peer dependency of React Router. But if you install React Router, it will not have React as a direct dependency. Like in the tree here, the first example has React as a direct dependency of React Router. That will not happen if it's... That is wrong.

Whereas if you have React installed as a peer dependency, then, as you can see, npm parent, the host application, has both React and React Router listed as the direct dependency. Even though React is listed as a peer dependency for React Router. Now that we have a good understanding of what peer dependencies is, we can look into why the error we saw earlier happened. And I'm gonna show it by building an example together in this space.

So let's consider this package over here. We have a host application named npm restaurants. In this npm restaurant application, it has one dependency, npm burger, and two dev dependencies. Npm oil and npm salt. Let's take a look at the burger package as well. So npm burger also has two dependencies, npm beef and npm bread. And it has one peer dependency, npm utensils, and one optional dependency, npm cheese.

If I install npm burger into npm restaurant, the dependency tree will look somewhat like this. On the right we have the tree. As you can see, npm restaurant is at the top, is the host application. And npm burger, npm utensils, npm oil, and npm salt are the direct dependencies of the host, npm restaurant. And then under npm burger, we have npm beef, npm bread, and npm cheese as the direct dependency. Notice that even though npm utensils is a peer dependency for npm burger package, it is not listed as a direct dependency under npm burger. Instead it's hoisted at the same level as npm burger. It looks like it's a direct dependency of npm restaurant.

I want to compare this restaurant and burger example to having a restaurant that sells burger on their menu. Now imagine you're in this restaurant, you have one item on the menu, you have burgers only. And you decided that you want to add a second item to your menu. Let's say fries. You try to install fries into npm restaurant. But then you run into the could not resolve dependency conflicting peer dependency problem. Why is that happening? In order to understand this, we need to look at the package.json file for both npm burger and npm fries side by side. So here on the left we have the package.json file for npm burger, and on the right we have the package.json file for npm fries.

Notice that both packages have peer dependency npm utensils. But the version that they need is different. For npm burger, it listed care at 1.0.0, which means it can only accept anything that is greater or equal to 1.0.0, and less than 2.0.0. Now for npm fries, it accepts anything that is greater or equal to 2.0.0, but less than 3.0.0. This is like the burger wants this white plate and the fries wants this bucket. Earlier, when we install npm burger into npm restaurant, we already have npm utensils 1.0.0, the white plate, installed. Because this is a peer dependency, we can only have one version of the package installed. When we try to install npm fries, the peer dependency npm utensils 2.0.0 for fries into the npm restaurant package, we see a conflict. And that is why you would see this error from your command line, which says cannot resolve dependency, conflicting peer dependency.

So if this happens, what can you do? To fix this peer dependency problem, generally speaking, you want to look for a compatible version. That could mean looking higher, lower version of npm burger, npm fries, or look for a version that accepts the same npm utensils version. You might need to update the dependency of a dependency, or even implement overrides in package.json. The reason I cannot give you one specific solution right now is because this peer dependency problem can happen in different levels in your package.json. So really, depending on a situation, we need to use different tools to fix this.

Back to this step 3, build IDEA tree, one more thing I wanted to highlight about this step is that your package.json file does not only exist in your host application. It actually exists in every single package that you have that you install. So when the CLI is trying to build the IDEA tree, it's not just looking at the package.json of your host application. It's also looking at every single package.json of the packages that you're installing. Let's say the tree is done building, then the CLI will proceed to compare the tree differences and resolve any conflicts. It will make decisions on what needs to be done next, might be adding, removing, changing, or just do nothing. Then the CLI will proceed to download the packages needed from the registry or from cache if it's available. This is a step when errors can happen. One of them might be you pointed a wrong registry, and so you get an error. Or you might be trying to install a private package, but you don't have authorization to do it. After the packages are being downloaded by the CLI to a temporary location in an instance, then it needs to be moved to your node modules. If your project has lifecycle scripts in it, those scripts will also be run accordingly if you allow them to do. And by lifecycle scripts, I'm talking about scripts like pre-install or post-install. After that, your log file will be updated to capture all the changes that have happened. The log file captures not only the dependencies listed in your package.json, but also the dependency of the dependency.

The log file captures not only the dependencies listed in your package.json, but also the dependency of the dependency. That's why your log file can be so massive. And finally, after the log file is updated, the CLI will run npm audit to check for any known security vulnerabilities. And the CLI will return a report to you so that normally when you run npm install, you'll see a similar report like this that tells you whether or not your packages have no vulnerabilities, critical, high, mid, low, and so on. And that is the entire flow of npm install.

I know that I just shared a lot of information to you, so I want to celebrate a little bit that we got through this part. Yay, I like that. All right, so now let's move on to publish. Before I talk about how publish works, let me ask you a question. How many packages are there in the npm registry today? Do we have any guesses? I hear some, a billion, okay. So the actual answer, as of this morning when I checked, is about 3.7 million packages. Every single one of them got published to the npm registry, and it's active at the moment. So what happens when you run npm publish?

The first thing that happens is the CLI will parse the command, similar to install. Do an engine check, load the instance. Then it will clean up the package.json file, normalize the data, validate the entry points, and if there are any overrides that need to be applied, this is the time when they will be applied. Most overrides will be things that you set by CLI flags or in publish config. Then it will prepare the tarball. The tarball is the content that the users will install. Before preparing the tarball, if you have lifecycle scripts like prepublish or prepack, the CLI will run those, and then it will prepare the tarball, and it will also consider if you have gitignore file, npm ignore file, what should be excluded.

Then it will pack everything that is needed, and once that's ready, if you have a postpack script, that might also be run too. And after the tarball is ready, the CLI will make what will fetch all the existing versions of your package to make sure that you are not publishing a duplicate version. If you do, you'll see an error like this one.

After that, it will also validate any text that you may use. After the text are validated, the CLI will do a credential check. Depending on how you publish, the check might be a little bit different. If you're using trusted publishing, OIDC, this is the time when the token exchange will happen. If you're publishing with access tokens, then at this step, the CLI will validate your token to make sure that the format is correct.

Once your credentials are checked, the CLI will proceed to build the pacument. The pacument is basically the package document, which has the package metadata in it. The package metadata is grouped by versions. And in each version, it will have name, description, keywords, tarball URL, those information about the package. Once the pacument is ready, your package is officially ready to be published.

The pacument is basically the package document, which has the package metadata in it. The package metadata is grouped by versions. And in each version, it will have name, description, keywords, tarball URL, those information about the package. Once the pacument is ready, your package is officially ready to be published.

So, the CLI will make a POST request with the pacument, the tarball, and provenance, if you have it, to the registry to publish the package. At this step, sometimes you might run into errors if you're pointing to the wrong registry, if you're unauthorized to publish. You could be unauthorized to publish for many different reasons. Your token doesn't work, you're not a maintainer. There are various reasons.

But assuming that your publish is successful, then if you have any post-published scripts, those will be run. And the CLI will log the success message, and the CLI will be done with publish at this point. On the registry side, it will make the package possible for others to install. It will also make it available on npmjs.com, so that if you are the package maintainer, you can go to npmjs.com to manage your package settings. If the package is public, then the package will also be searchable and viewable on npmjs.com. And it will also admit a change event to the replication feed. And that is the entire npm publish workflow.

I hope I have successfully shed some light into the npm install and npm publish process. If you have any questions, concerns, feedback, please visit the GitHub community to join the discussions there. I know my team look at those and try to see what you're interested in, so please join that community. I will also answer some questions here, or if you have any feedback, please meet me after this. I would love to hear them. That is the end of my presentation. Thank you so much for joining me today.

My first question for you is, how did you learn all this stuff? Were you doing source code diving of the CLI? Is there docs we could find that describe this? What's your learning process for diving into something like this? I use Copilot to iterate through the codebase. Basically I just kept on talking to it and asked questions, checked my understanding. Because at our team we use GitHub code spaces, so I can not only look at how the CLI codebase is like, but I can also look on the backend at how things are connected. I use GitHub Copilot to guide me through the process and that's how I came up with it.

That's great. I've done that before when I'm in a new codebase or a codebase I'm trying to understand. That's one of the good use cases for AI is trying to summarize and also pointing you to places in the codebase that you might not have seen before. I'll remind you, anybody in the room if you want to ask questions be sure to use that code there. We do have a question from Pablo who asks, what happens when you use dash dash legacy peer depths to solve peer dependencies? What are some long-term implications or consequences of this? That is a great question. The flag has multiple behaviors and I believe depending on the NPM CLI version it will also be different. I don't want to give you the wrong information here as to what exactly will happen because it's version dependent.

In terms of long-term consequences and implications, peer dependencies pose a challenge as downstream usage may not be controlled despite good management at the host application level. It's crucial to be vigilant about package installations, especially in light of recent events. Addressing a question from Song on combating malicious package submissions in the AI era, the NPM team acknowledges the severity of the issue and mentions ongoing efforts to enhance security, such as typosquatting prevention and token management strategies.

Concerns about typosquatting, where similar package names are used to deceive users, are highlighted as a potential risk. The discussion extends to the use of symlinks in third-party package managers for faster installs compared to NPM’s current approach. While some symlinking exists in the CLI, the specifics and potential improvements are subject to community feedback and input for consideration in future developments.

Exploring the significance of upgrading lockfile versions, the differentiation between lockfile versions 1, 2, and 3 is linked to specific functionalities aligned with different NPM CLI versions. The evolution of lockfile versions is associated with adapting to various requirements, including legacy NPM CLI capabilities and the introduction of lockfiles in the NPM ecosystem.

That's something you definitely want to watch out for. We don't have any other questions coming in, but – well, here's another one. Let's see. Does the NPM team – no, no, we just talked about that one. Sorry. Cool. Let's talk about this one. Some of the other third-party package managers have moved to a symlink-based approach for known modules. Is there a reason NPM hasn't gone that route? I guess to clarify on that, I know this from PNPM. PNPM, instead of copying those files to every install that you do, it will symlink it to a single location, usually makes the installs a little bit faster. Does the CLI do any sort of symlinking or is there any plans to make the installs faster in any way like that? I believe there is some symlinking existing, but it might not be the same way as how the third-party package managers are doing it. Sure. I say this because I remember seeing this when I was doing my research and searching through Copilot. But that being said, I don't have a lot of information specifically on the CLI side as to what's on the roadmap and whether or not we're going to do it. But if this is something that y'all really want, please voice it in the GitHub community because the team does look at it. I know the product managers, some of the engineers definitely look at it. So if you want that, please share your feedback with us.

This next question is, are there any benefits to upgrading lockfile versions? I think you're referring to the lockfile version 1, 2, and 3. That is, I believe initially, the reason why there are different lockfile versions has to do with NPM CLI versions. Because there are very old legacy NPM CLI versions that can do certain things, and then the lockfiles came out. So I'm not entirely sure, aside from that, is there any additional functionalities per se? That is something that unfortunately I won't be able to answer at this moment. Cool. That's okay. And I think there's different ways you could take this question. If you were to, let's say, delete a lockfile and then run NPM install again, you're going to get a brand new lockfile because it's taking a look at the latest installed versions. That might be what they're talking about as well. I think in general, it's good practice to keep your dependencies up to date in that aspect along those lines. Yeah, this next question, again, feel free to skip it if you can't get into it, but 3 million packages is a lot.

I think in general, it's good practice to keep your dependencies up to date in that aspect along those lines. Yeah, this next question, again, feel free to skip it if you can't get into it, but 3 million packages is a lot.

How are packages removed and audited? Okay. So I would assume when you say removed and audited, you're referring to maybe malicious packages instead of user deleting. Because obviously a user can unpublish their package if their package is not dependent by somebody else. But if you're talking about malicious packages, I do know that when a package is published, we do scan them. And if we identify them as malicious packages, they will be taken down. So there is a process built in place for that. Cool.

Yeah, and that kind of gets at the next question that was asked. Is it marked as malicious by automation? It sounds like there is a process for that. I guess if it is marked, can there be some human intervention to prevent it? Maybe it was like a false alarm or something like that. I guess, what's the process for when a package is marked as malicious? So when a package is being marked as malicious, we do have trust and safety team looking into that. And the trust and safety team will do the takedown. And they have their internal set of process that unfortunately I don't really have knowledge of. But it is, from my understanding, the team does intervene. That's good.

I guess we'll keep it light. There's not any questions yet. But I went to see what was my first NPM published package, and it was nine years ago. I published a materialized CSS library. I just remember figuring out that you could publish NPM packages was really cool. Like, literally, one command, and anybody in the world can use it. Have you published any packages? Or do you remember your first one or how long ago it was?

So I have never published any NPM packages prior to joining NPM. Oh, wow. Okay. Which is... And after I joined the team, I had to publish a lot of test packages. But I don't have my own package published. I see. Yeah. For me, it's always fun. I also, if I come up with a package name idea, I publish it immediately. Just so somebody doesn't steal the name, similar to trying to go buy a domain or something like that. Awesome.

We don't have any other questions asked, but is there anything that maybe you didn't get to, that you'd like everybody to know? Or anything else you'd like to talk about? I think one thing that I wanted to highlight is that I mentioned at the credential check part earlier for NPM publish flow, that there are some token-related changes happening at NPM. There is some secure architecture changes that are happening that has been rolled out already, some parts of them, but there will be more of them rolling out in the coming weeks. And some of these changes will have pretty big disruptive changes.