The Secret Life of Package Managers

So, you are running npm install and you go and grab a cup of coffee, and then you come back and you have no idea or maybe you don't even care what npm did during this time.

So, my name is Tali Barak. I work for Youbeek and let me tell you about the secret life of package managers. This is what happens in your project, this is the basics of npm.

So, you have your project and you need a package called foo. So, you are running npm install foo and that's add npm install to your package json and create a node modules in your project and put this code of foo inside it. But what if foo requires buzz? Okay, no problem. It will create another node modules folder under foo and it will put buzz. And what if buzz requires bugs? Well, same thing. It will put it and add it there. And then what happens if your foo requires a buzz but also your bar requires buzz. In this case, in the naive implementation of npm, you will have two packages of buzz in the same project.

And this actually creates the whole structure of your file system that is replicating the package structure that you have in your project. And this was nice, but it created quite a few problems. For example, it made your node models huge. Also, it created a problem with circular dependencies. That means that if your foo needed a buzz, that needed a buzz, that needed a buzz, that needed bar again, that needed buzz, and needed buzz, you would go into an infinite loop. And this is, by the way, quite common. It's not as rare as you might think it is. Another issue which is common is with singletons. If you need a single instance of a certain package, like the debug package, for example, in this structure, you would have multiple instances, and that can cause bugs when you execute the packages. And the last one is no longer with us. Thank God for that. It was a Windows 8 problem that the path, the file path was limited to 256 characters. This is less common.

So what did NPM do in order to solve that? So they decided to do a dedup, de-duplicate the packages that were multiple times. So instead of having buzz twice, they would put it in the highest possible level and use it there. And the reason this worked is because the way Node is requiring packages. So if buzz needed buzz, it would go under the Node modules and search for it. If it doesn't find it, it will go one level up and search for a third.

And then if it doesn't find it, it will go another level up. And there it is. It actually found buzz there and it will use it.

Next, they decided, well, let's take it one step further. If we can move packages up the tree, why only the duplicate one? We can actually do that for all the packages. And they made a very flat tree with all the packages. And this was good. This solved the problem. You now had smaller packages, shorter paths because it didn't go that deep. It was only unidirectional, no circulars. It made every package unique.

Was good, but then we had a problem. Each package might have a different version that it requires. So your tree, the tree that you need doesn't actually look something like this one. You have different versions of the same packages required in different places in the tree. And even worse, in some cases, the versions could conflict. That means that your foo might require buzz in version one, but bar requires buzz in version two. And how do you flatten that? What do you put at the top level? In fact, we have an issue here that your file structure, your package structure, is no longer a tree. It is actually a graph.

And the way it was solved is by different versions of NPM had different solutions. Sometimes it would take a popular one. Sometimes it would take a first one and put it at the top of the tree. While the other version that was required was left under the package that required it. Like in this case here, where you could promote two or one, depending on the order. And this is a problem because now we get a very shaky and unpredictable tree. And the way that NPM and also Yarn in version 1 solved it is by actually taking a snapshot of your whole node modules. And this is the famous log file. NPM has it as a shrink wrap file. And then Yarn added the Yarn log file. And this is the way for NPM to make sure that the package structure that the node module files in one project is the same as the one on the CI or is the same as your colleagues run.

But here is another solution. PNPM is actually doing something smart and different. PNPM is keeping the original structures, the tree structures that we talked about, keeping all the packages and all of their versions. But they don't really install the packages. They store all the packages and all the different versions in a cache. And then the node modules is actually pointing at the operation system level, using hard links to the cache and storing the exact version and pointing to the right version where it is required.

So this is one solution that was used and the other solution is what yarn2 which is codenamed berry introduced. They basically say that we are the package manager and we know exactly what package is required by each package. So they have a map of all the node modules and all the packages and what they require. And they patch, they change the way node is requesting the files. So if node is now requiring a package it will no longer go directly to the file system and get the package but instead it will access yarn and request the specific package that it needs and yarn will look into this map of modules and will return the exact file that are showing that.

So that was a very, very short and very quick peek into the secret life of package manager.