Package Management in Monorepos

Rate this content
Bookmark

We’ll talk about some of the pain points and look into recipes for effective package management in monorepos. 
We’ll discuss how package management works with npm, pnpm, and Yarn. Furthermore, I’ll show you a new tool that is less known but improves developer experience by a lot.

This talk has been presented at DevOps.js Conf 2024, check out the latest edition of this JavaScript Conference.

FAQ

pnpm is a JavaScript package manager known for its speed and efficiency in handling dependencies, especially in monorepos.

In a hoisted node modules layout, all dependencies are placed in the root's node_modules directory, while in an isolated node modules layout, dependencies of each package are nested within their own directories.

The isolated node modules layout ensures that each package only has access to its own dependencies, reducing the risk of dependency conflicts and ensuring better isolation.

Bit is a toolchain for building composable software, functioning as an alternative to Git, GitHub, npm registry, and npm clients. It manages dependencies and publishes packages, with a focus on simplifying package management in monorepos.

Bit conducts code analysis to automatically identify and manage dependencies, eliminating the need for separate dev dependencies fields and manual updates in the package.json file.

The noextraneous dependencies rule from the eslint import plugin notifies developers of imported dependencies that are not declared in the package.json file, helping to catch potential issues in monorepos.

The main topic of Zoltan Koçan's presentation is package management in monorepos, including pain points and effective management strategies.

Peer dependencies are packages that must be singletons during runtime. Managing them well ensures consistency across projects and avoids conflicts, which is crucial in monorepos.

pnpm offers a feature called injected dependencies, allowing workspace packages to run with different versions of a peer dependency by copying the relevant packages.

Zoltan Koçan is the lead maintainer of the pnpm open-source project and works at Bit, handling dependency management tasks. He has previously worked at JustAnswer.

Zoltan Kochan
Zoltan Kochan
19 min
15 Feb, 2024

Comments

Sign in or register to post your comment.
Video Summary and Transcription
This Talk discusses pain points and effective package management in monorepos, including the use of hoisted or isolated layouts and the challenges of working with peer dependencies. It introduces the tool Bit, which addresses these issues and handles dependency management and version control. Bit enables automatic installation and management of dependencies, supports multiple versions of a peer dependency, and seamlessly updates components across different environments.
Available in Español: Gestión de paquetes en Monorepos

1. Introduction to Package Management in Monorepos

Short description:

My name is Zoltan Koçan. I'll discuss the pain points and share recipes for effective package management in monorepos. I handle dependency management at Bit and I'm the lead maintainer of the pnpm project. The history of monorepo tooling for JS projects starts with Babel and Lerna. By 2017, pnpm, Yarn, and npm all shipped monorepo support. Package managers can arrange dependencies in hoisted or isolated layouts, with pnpm using the isolated node modules layout.

My name is Zoltan Koçan. In my presentation, I want to talk about package management in monorepos. I'll discuss some of the pain points and share recipes for effective package management in monorepos.

Currently, I work at Bit, where I handle dependency management related tasks. I'm also the lead maintainer of the pnpm open-source project, which is a JS package manager. Before Bit, I worked at JustAnswer. At JustAnswer we had a huge monorepo with hundreds of components. Installation with npm took 30 minutes in that monorepo. That was the main reason I started contributing to a faster alternative, pnpm. With pnpm, we were able to reduce installation time to about 90 seconds.

Let's briefly talk about the history of monorepo tooling for JS projects. Babel was one of the most influential projects in the JavaScript ecosystem, and it was probably one of the first popular open-source JS projects that used a monorepository. The creators of Babel have created Lerna in 2015. Lerna was able to install dependencies in a monorepo using npm-cli under the hood. With that said, installation with Lerna was terribly slow, to say the least. Everyone knew that package managers should implement installation in monorepos out of the box. By 2017, both pnpm and Yarn have shipped monorepo support. Yarn has called this feature workspaces installation, while pnpm has used the singular term workspace. In a couple of years, npm had also shipped workspaces support. As of today, there are three popular mature Node.js package managers with built-in monorepo support.

There are two ways package managers can arrange dependencies in a monorepo, hoisted and isolated. All three package managers support both layouts. By default, Yarn and npm use a hoisted approach. With this approach, all direct and indirect dependencies are placed in the roots non-modules directory. If there are multiple versions of the same dependencies, one of the versions gets nested. As you can see on this slide, there are two different versions of lodash. So one of the versions is hoisted to the root of the monorepo, while the other one is nested inside app2. pnpm uses a different layout called isolated node modules. With the isolated node modules, the dependencies of every package are nested. The benefit of this approach is that packages only have access to their own dependencies.

2. Dependency Management in Monorepos

Short description:

While with hoisted layout, all projects would have access to the cookie package. It's really easy to mess up dependencies in a monorepo. Main.js in app1 is using lodash listed in the dev.dependencies. Working with peer dependencies in monorepos is challenging. It is crucial to use a single version of the peer dependency across all the Workspace packages. Only Yarn currently supports syncing versions of dependencies out of the box. pnpm has plans to introduce this feature through Workspace catalogs. pnpm offers a feature to support multiple versions of a peer dependency known as injected dependencies.

While with hoisted layout, all projects would have access to the cookie package, with an isolated node modules layout, projects have access only to their own dependencies. So in this case, only app1 will be able to require cookie.

I think most people agree that monorepos provide a superior developer experience. Despite this, it's really easy to mess up dependencies in a monorepo. As you can see in this example, the app is using a cookie but doesn't list cookie in its dependencies. This code will work locally because cookie is found in the node modules directory of a parent directory. However, it will break when someone installs app1 outside of the monorepo.

Another issue in this example is that main.js in app1 is using lodash. Main.js is production code, but lodash is listed in the dev.dependencies. It means that this code will work locally, but it will break in production where dev.dependencies are not installed. To catch these two specific issues, you may use a special rule in eslint, the noextraneous dependencies rule from the import plugin. If you configure this linting rule, eslint will notify you of dependencies that are imported but not declared in package.json. In this example, you will get an error about cookie being used in app1. eslint will also notify you about lodash being a dev.dependency. You avoid it if it is used by production code.

Working with peer dependencies in monorepos is challenging. It is crucial for the peer dependencies to be singletons during runtime. If possible, you should try to use a single version of the peer dependency across all the Workspace packages. As you can see in this example, both card and button reference the same react version. This will work fine. Whether you are dealing with peer dependencies or not, it is preferable to use the same version of a dependency across all of your projects. Doing so can help you avoid issues related to peer dependencies and reduce the size of your packages. To the best of my knowledge, only Yarn currently supports syncing versions of dependencies out of the box using constraints. pnpm has plans to introduce this feature through Workspace catalogs. It is also possible to use third-party tools for finding version duplicates. Multiple third-party tools act as linters to verify version inconsistency. One such tool is Syncpack. On large monorepos, it can sometimes become challenging to avoid having multiple versions of a peer dependency. Among npm, yarn, and pnpm, only pnpm offers a feature to support multiple versions of a peer dependency. This feature is known as injected dependencies.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Levelling up Monorepos with npm Workspaces
DevOps.js Conf 2022DevOps.js Conf 2022
33 min
Levelling up Monorepos with npm Workspaces
Top Content
NPM workspaces help manage multiple nested packages within a single top-level package, improving since the release of NPM CLI 7.0. You can easily add dependencies to workspaces and handle duplications. Running scripts and orchestration in a monorepo is made easier with NPM workspaces. The npm pkg command is useful for setting and retrieving keys and values from package.json files. NPM workspaces offer benefits compared to Lerna and future plans include better workspace linking and adding missing features.
pnpm – a Fast, Disk Space Efficient Package Manager for JavaScript
DevOps.js Conf 2022DevOps.js Conf 2022
31 min
pnpm – a Fast, Disk Space Efficient Package Manager for JavaScript
Watch video: pnpm – a Fast, Disk Space Efficient Package Manager for JavaScript
pnpm is a fast and efficient package manager that gained popularity in 2021 and is used by big tech companies like Microsoft and TikTok. It has a unique isolated node module structure that prevents package conflicts and ensures each project only has access to its own dependencies. pnpm also offers superior monorepo support with its node module structure. It solves the disk space usage issue by using a content addressable storage, reducing disk space consumption. pnpm is incredibly fast due to its installation process and deterministic node module structure. It also allows file linking using hardlinks instead of symlinks.
End the Pain: Rethinking CI for Large Monorepos
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
End the Pain: Rethinking CI for Large Monorepos
Today's Talk discusses rethinking CI in monorepos, with a focus on leveraging the implicit graph of project dependencies to optimize build times and manage complexity. The use of NX Replay and NX Agents is highlighted as a way to enhance CI efficiency by caching previous computations and distributing tasks across multiple machines. Fine-grained distribution and flakiness detection are discussed as methods to improve distribution efficiency and ensure a clean setup. Enabling distribution with NX Agents simplifies the setup process, and NX Cloud offers dynamic scaling and cost reduction. Overall, the Talk explores strategies to improve the scalability and efficiency of CI pipelines in monorepos.
The Zen of Yarn
DevOps.js Conf 2022DevOps.js Conf 2022
31 min
The Zen of Yarn
Let's talk about React and TypeScript, Yarn's philosophy and long-term relevance, stability and error handling in Yarn, Yarn's behavior and open source sustainability, investing in maintenance and future contributors, contributing to the JavaScript ecosystem, open-source contribution experience, maintaining naming consistency in large projects, version consistency and strictness in Yarn, and Yarn 4 experiments for performance improvement.
Yarn 4 - Modern Package Management
JSNation 2022JSNation 2022
28 min
Yarn 4 - Modern Package Management
Top Content
Yarn is a package manager that focuses on stability, performance, and security. It offers unique features like plug and play installation, support for nonmodules, and the exec protocol. Yarn is committed to being a good citizen in the open-source community and contributes to fixing dependencies. It is part of the Node.js Loader's working group and advocates for Corepack. Yarn is still experimental but is improving its user experience and security features. Contributions are welcome, and switching to Yarn can improve performance in large projects.
Federated Microfrontends at Scale
React Summit 2023React Summit 2023
31 min
Federated Microfrontends at Scale
Top Content
Watch video: Federated Microfrontends at Scale
This Talk discusses the transition from a PHP monolith to a federated micro-frontend setup at Personio. They implemented orchestration and federation using Next.js as a module host and router. The use of federated modules and the integration library allowed for a single runtime while building and deploying independently. The Talk also highlights the importance of early adopters and the challenges of building an internal open source system.

Workshops on related topic

React at Scale with Nx
React Summit 2023React Summit 2023
145 min
React at Scale with Nx
Top Content
Featured WorkshopFree
Isaac Mann
Isaac Mann
We're going to be using Nx and some its plugins to accelerate the development of this app.
Some of the things you'll learn:- Generating a pristine Nx workspace- Generating frontend React apps and backend APIs inside your workspace, with pre-configured proxies- Creating shared libs for re-using code- Generating new routed components with all the routes pre-configured by Nx and ready to go- How to organize code in a monorepo- Easily move libs around your folder structure- Creating Storybook stories and e2e Cypress tests for your components
Table of contents: - Lab 1 - Generate an empty workspace- Lab 2 - Generate a React app- Lab 3 - Executors- Lab 3.1 - Migrations- Lab 4 - Generate a component lib- Lab 5 - Generate a utility lib- Lab 6 - Generate a route lib- Lab 7 - Add an Express API- Lab 8 - Displaying a full game in the routed game-detail component- Lab 9 - Generate a type lib that the API and frontend can share- Lab 10 - Generate Storybook stories for the shared ui component- Lab 11 - E2E test the shared component
Node Monorepos with Nx
Node Congress 2023Node Congress 2023
160 min
Node Monorepos with Nx
Top Content
WorkshopFree
Isaac Mann
Isaac Mann
Multiple apis and multiple teams all in the same repository can cause a lot of headaches, but Nx has you covered. Learn to share code, maintain configuration files and coordinate changes in a monorepo that can scale as large as your organisation does. Nx allows you to bring structure to a repository with hundreds of contributors and eliminates the CI slowdowns that typically occur as the codebase grows.
Table of contents:- Lab 1 - Generate an empty workspace- Lab 2 - Generate a node api- Lab 3 - Executors- Lab 4 - Migrations- Lab 5 - Generate an auth library- Lab 6 - Generate a database library- Lab 7 - Add a node cli- Lab 8 - Module boundaries- Lab 9 - Plugins and Generators - Intro- Lab 10 - Plugins and Generators - Modifying files- Lab 11 - Setting up CI- Lab 12 - Distributed caching
Finding, Hacking and fixing your NodeJS Vulnerabilities with Snyk
JSNation 2022JSNation 2022
99 min
Finding, Hacking and fixing your NodeJS Vulnerabilities with Snyk
WorkshopFree
Matthew Salmon
Matthew Salmon
npm and security, how much do you know about your dependencies?Hack-along, live hacking of a vulnerable Node app https://github.com/snyk-labs/nodejs-goof, Vulnerabilities from both Open source and written code. Encouraged to download the application and hack along with us.Fixing the issues and an introduction to Snyk with a demo.Open questions.
Build Web3 apps with React
React Summit 2022React Summit 2022
51 min
Build Web3 apps with React
WorkshopFree
Shain Dholakiya
Shain Dholakiya
The workshop is designed to help Web2 developers start building for Web3 using the Hyperverse. The Hyperverse is an open marketplace of community-built, audited, easy to discover smart modules. Our goal - to make it easy for React developers to build Web3 apps without writing a single line of smart contract code. Think “npm for smart contracts.”
Learn more about the Hyperverse here.
We will go over all the blockchain/crypto basics you need to know to start building on the Hyperverse, so you do not need to have any previous knowledge about the Web3 space. You just need to have React experience.