Package Management in Monorepos

My name is Zoltan Koçan. In my presentation, I want to talk about package management in monorepos. I'll discuss some of the pain points and share recipes for effective package management in monorepos.

Currently, I work at Bit, where I handle dependency management related tasks. I'm also the lead maintainer of the pnpm open-source project, which is a JS package manager. Before Bit, I worked at JustAnswer. At JustAnswer we had a huge monorepo with hundreds of components. Installation with npm took 30 minutes in that monorepo. That was the main reason I started contributing to a faster alternative, pnpm. With pnpm, we were able to reduce installation time to about 90 seconds.

Let's briefly talk about the history of monorepo tooling for JS projects. Babel was one of the most influential projects in the JavaScript ecosystem, and it was probably one of the first popular open-source JS projects that used a monorepository. The creators of Babel have created Lerna in 2015. Lerna was able to install dependencies in a monorepo using npm-cli under the hood. With that said, installation with Lerna was terribly slow, to say the least. Everyone knew that package managers should implement installation in monorepos out of the box. By 2017, both pnpm and Yarn have shipped monorepo support. Yarn has called this feature workspaces installation, while pnpm has used the singular term workspace. In a couple of years, npm had also shipped workspaces support. As of today, there are three popular mature Node.js package managers with built-in monorepo support.

There are two ways package managers can arrange dependencies in a monorepo, hoisted and isolated. All three package managers support both layouts. By default, Yarn and npm use a hoisted approach. With this approach, all direct and indirect dependencies are placed in the roots non-modules directory. If there are multiple versions of the same dependencies, one of the versions gets nested. As you can see on this slide, there are two different versions of lodash. So one of the versions is hoisted to the root of the monorepo, while the other one is nested inside app2. pnpm uses a different layout called isolated node modules. With the isolated node modules, the dependencies of every package are nested. The benefit of this approach is that packages only have access to their own dependencies.

While with hoisted layout, all projects would have access to the cookie package, with an isolated node modules layout, projects have access only to their own dependencies. So in this case, only app1 will be able to require cookie.

I think most people agree that monorepos provide a superior developer experience. Despite this, it's really easy to mess up dependencies in a monorepo. As you can see in this example, the app is using a cookie but doesn't list cookie in its dependencies. This code will work locally because cookie is found in the node modules directory of a parent directory. However, it will break when someone installs app1 outside of the monorepo.

Another issue in this example is that main.js in app1 is using lodash. Main.js is production code, but lodash is listed in the dev.dependencies. It means that this code will work locally, but it will break in production where dev.dependencies are not installed. To catch these two specific issues, you may use a special rule in eslint, the noextraneous dependencies rule from the import plugin. If you configure this linting rule, eslint will notify you of dependencies that are imported but not declared in package.json. In this example, you will get an error about cookie being used in app1. eslint will also notify you about lodash being a dev.dependency. You avoid it if it is used by production code.

Working with peer dependencies in monorepos is challenging. It is crucial for the peer dependencies to be singletons during runtime. If possible, you should try to use a single version of the peer dependency across all the Workspace packages. As you can see in this example, both card and button reference the same react version. This will work fine. Whether you are dealing with peer dependencies or not, it is preferable to use the same version of a dependency across all of your projects. Doing so can help you avoid issues related to peer dependencies and reduce the size of your packages. To the best of my knowledge, only Yarn currently supports syncing versions of dependencies out of the box using constraints. pnpm has plans to introduce this feature through Workspace catalogs. It is also possible to use third-party tools for finding version duplicates. Multiple third-party tools act as linters to verify version inconsistency. One such tool is Syncpack. On large monorepos, it can sometimes become challenging to avoid having multiple versions of a peer dependency. Among npm, yarn, and pnpm, only pnpm offers a feature to support multiple versions of a peer dependency. This feature is known as injected dependencies.

When injected dependencies are enabled, Workspace packages are copied, allowing them to run with different versions of the peer. As you can see in this example, the Button component uses React version 17, but the Card component uses React version 16. When we run tests for the Button component, we aim to use React version 17. In contrast, when we run the Button component from within the Card component, we want Button to use the same version of React as the Card. Therefore, React version 17 is installed in the node modules of Button. Meanwhile, Card doesn't reference the Button directly from the Workspace, but from a hidden subdirectory where it's aligned with React version 16. This setup means there are two instances of the Button component in the Workspace, one with React version 17 and another, a copy inside the node-modules.pnpm directory with React version 16. The only downside to this approach is that once you modify the Button component, you need to rerun the installation process so that pnpm can update the files with the Button component in the copied instances.

After discussing the challenges associated with monorepos, I'd like to introduce another tool that addresses these issues out of the box. The name of the tool is Bit. While Bit is not a package manager, managing packages is one of its primary responsibilities. So, what exactly is Bit? It's a toolchain designed for building composable software. You can conceptualize it as an alternative to Git, GitHub, the npm registry, and various npm clients. When using Bit, it serves as your version control system, manages your dependencies, and publishes your packages. For this presentation, we'll focus solely on package management, so I'll discuss only the installation aspect of Bit.

In many ways, a Bit workspace resembles a pnpm workspace. It's a collection of packages or components. However, there's a distinct difference, as there are no packages on files in a Bit workspace. Instead, all dependencies for all components are declared in a single manifest located at the workspace's root. Moreover, there's no separate field for dev dependencies. This streamlined approach is feasible because Bit conducts a code analysis of the components within the workspace. It automatically identifies which components utilize which dependencies and discerns whether a particular dependency is for production or development for a given component. Let me show you now a quick demo of a Bit workspace. For the demo, I will use VS Code with the Bit extension installed. As you can see, I already have it installed on my computer. Let's go to the Bit section and start a new workspace. Bit has created a workspace manifest for me. Let's now generate some new components using the Bit CLI tool. I will create two new node apps app1 and app2. Here are the directories of my newly created app components.

As you can see, neither app1 nor app2 contain any packages on files. Packages on files are dynamically created by Bit on publish. Under the hood, Bit runs pnpm install to install any dependencies. As you can see, here is the pnpm output. Now it installs some dependencies that are present by default in the node app components.

Let's now go to the Bit menu again. Here we can see the list of our components. As you can see in our component details, the dependencies of the component. Let's add some new import statements. I will add lodash to app1 and ramda to app2. As you can see, these dependencies are currently missing. In a pnpm workspace with pnpm, you would run something like pnpm add to app1 dependency lodash and add to app2 dependency ramda. Now with Bit, it's easier. You just need to run bit install add missing deps and Bit will perform code analysis, find any new import statements with dependencies that are not yet installed in the workspace and it will automatically add these new dependencies to the workspace.

So once installation finishes, we have to be able to see the new dependencies in the component details. So yeah, here it is. So lodash is now in the dependencies of app1 and ramda is in the dependencies of app2. We can also see that these new packages appeared in the workspace manifest here and they were installed in node modules ramda lodash. Now let's commit our changes. I will use Bit to commit the changes because Bit is also a version control system. You can use Bit with git too, it's not a problem. Now let's do some changes to dependencies again. So now I will remove lodash from app1 TS and move it to appspec TS. So now it becomes a dev dependency. And also let's remove ramda altogether from app2. Now let's run install. So in a pnpn workspace, after making these changes, you would need to manually update packages on app1. You'd have to remove lodash from the dependencies section and put it to the dev dependencies section. With Bit this is not needed. It automatically detects that the type of the dependency has changed.

And also with pnpm you would have to manually remove ramda from the dependencies of app2. But with Bit this is not needed. As you can see here in the component details, ramda is not in the dependencies of app2. And in app1 lodash is now a dev dependency, not a prod dependency. So this actually happened automatically under the hood. Nothing from the user was required. If we check the changes, we can actually also see any changes to the dependencies. And we can see that in app1 lodash was changed from a runtime or prod dependency to a dev dependency. And in app2 ramda was removed altogether.

In addition to managing dependencies, Bit also handles the compilation of packages within the workspace. This makes it considerably easier to support multiple versions of a peer dependency in contrast to pnpm. As illustrated on this slide, our workspace contains four components. Two pages, a button and a card. We want to use two versions of React in this workspace. So Bit creates two separate runtime environments for the components. One with React v16 and another one with React v17. Components relying on React v16 are loaded from the corresponding environment along with all of their dependencies. Likewise, components that rely on React v17 will be loaded from the other environment. All their dependencies will use React v17. Should there be any changes to a component, Bit will seamlessly update that component across all environments.

That was all I wanted to share with you today. Thank you all. Goodbye.