Tale of Two Repos

Rate this content
Bookmark
Slides

Not all monorepos are made the same, and not all teams are equal. 

Managing a monorepo is way more than just ""let's select this tool or that tool"". It comprises many design decisions that you need to make for the product and for the developers. 


In this talk, I will go over the main monorepos challenges - packages installation and linking, development and deployment processes - and describe the possibilities that exist for each stage. 


At the end of the talk, you will have a simple and powerful model that will help you in making the right choice of tools and processes for your monorepo. 

This talk has been presented at Node Congress 2023, check out the latest edition of this JavaScript Conference.

FAQ

ECMAScript 6, or ES6, is a significant update to JavaScript that was released in 2015. It introduced many new features such as Spread Operators, Arrow Functions, Const, and Let, which greatly enhanced the language's capabilities.

The project was created by Sebastian MacKenzie in 2014. He initially started it to convert ES6 code so it could be used in browsers that supported ES5.

A monorepo is a single repository that contains multiple artifacts, such as packages, backend services, and frontend applications, that are meant to be shared and published. It allows for centralized management of code and dependencies.

Using a monorepo allows for easier code sharing, centralized dependency management, and streamlined development processes. It can improve the quality of the product, speed up development, and simplify team training.

Key strategies include deciding what to include in the monorepo, determining the installation and linking of packages, developing a build process, and establishing a release strategy. These strategies help in managing dependencies, builds, and releases effectively.

The 2015 update, known as ECMAScript 6 or ES6, introduced many new features that greatly enhanced JavaScript. However, it also created challenges as many engines, such as Node.js and various browsers, did not initially support these new features.

The project, originally named 6to5, was renamed to Babel to better reflect its purpose of converting new JavaScript features into a form that older JavaScript engines could understand. The renaming was also accompanied by consolidating various plugins into a single repository.

Lerna is a tool that was created to manage monorepos and the various packages within them. It was extracted from the utilities built to manage Babel's multiple packages and repositories.

Common tools for managing monorepos include Yarn, NX, TurboRepo, and PNPM workspaces. These tools help with dependency management, building, and releasing packages within a monorepo.

Challenges can include managing dependencies, ensuring build consistency, and coordinating releases. It's also important to choose the right tools and strategies to address these challenges effectively.

Tally Barak
Tally Barak
24 min
17 Apr, 2023

Comments

Sign in or register to post your comment.
Video Summary and Transcription
JavaScript became popular in the mid-2010s with the introduction of Node and advanced frameworks like Angular and React. Monorepos, which are single repositories containing multiple artifacts, are a popular approach for managing JavaScript projects. Linking packages internally in a monorepo can be done through local or global installation, or by publishing them to an artifact registry. Managing monorepos has become more complex with the introduction of tools like TypeScript and Babel. The development process for web applications involves a build process, testing, and linting, which can be facilitated by tools like Yarn Workspace and NPM Workspace. The release strategy for monorepos can be either unified or distributed, and it is important to select tools that align with the chosen strategy.
Available in Español: Historia de Dos Repositorios

1. The Rise of JavaScript and Monorepos

Short description:

JavaScript became popular in the mid-2010s with the introduction of Node and advanced frameworks like Angular and React. In 2015, ECMAScript 6 (ES6) brought new features, but the engines running the language didn't support them. Sebastian MacKenzie started a project to convert ES6 code to be used in browsers that supported ES5. The project grew and became Babyl, a controversial decision. Lerna, initially synonymous with Monorepos, was created to manage the project's plugins. However, the way we use Monorepos in 2015 may not be suitable for modern JavaScript. Yarn and NX use the term workspace differently. I have experience with Monorepos since 2017.

It was the best of time, it was the worst of time. Around the last decade, JavaScript started to become very popular, back in the mid of 2010. And the introduction of Node was one of the things, and advanced frameworks like Angular and React were introduced, and it became really hype.

In 2015, JavaScript actually woke up from a deep sleep a few years and had a lot of new features. This was known as the ECMAScript 6, or as we like to call it, ES6. But there were also some problems, because the engines that were running the language, that is the Node, and that's, this is a different browser, and especially a browser that is no longer with us, did not know how to handle those things.

Things like Spread Operators, Error Function, Const, and Let were extremely useful when writing JavaScript code, but it was just not yet supported. And we are developers, and we want those features right now. In 2014, a very young guy called Sebastian MacKenzie started a project to learn more about the works of JavaScript and he was trying to actually convert the code from ES6, so it can be used in the browser that supported ES5. So Sebastian created a repository and built the core of a project, but because there were so many features for ES6 and more were coming in all the time, they decided to build it in an extensible design and using plugins.

So they opened a repository for each one of the plugins so it can be installed separately. And the project grew and grew and grew because there were so many plugins. So they decided to put together all the repositories of the different plugins into a one plugin. And by the way, at the same time, they also renamed the project. They were no longer calling it 6to5 as it was named before, and they changed the name to Babyl. So they put all their plugins in one repository that could publish many packages. And this decision was very, very controversial at the time, it was not very clear at all. And it was so controversial that they actually had to write in their GitHub repository an explanation of why they are doing it. And they also built some utilities to help them manage the different packages. Later on, they extracted all these utilities into a separate repository, again, another repository, and they called it Lerna. And back then and for a long time, maybe even until today, Lerna was a synonym for using Monorepos. But I think that the way we use JavaScript and Monorepos back in 2015 is not really what we need to do today in 2023 and the way to handle a modern JavaScript.

The problem is that sometimes we overlook there and we get to the point we just say the magic word Monorepos and we assume that everyone is talking about the same thing. But the reality Monorepos is a big thing and everyone means something completely different. Just to give you an example. If we look at two popular tools, Yarn and NX, they use the same term but for opposite things. In Yarn a workspace is a single package. In NX when they say workspace they mean the whole Monorepo. So I am Talib Barak and I've been fiddling with Monorepos since 2017. So that's quite a lot of time.

2. Decisions and Strategies for Monorepos

Short description:

And I'm not going to talk about that tool or that tool, Yarn or NX or TurboRepo. When moving or having a Monorepo, decisions and strategies impact the development process, product quality, speed, and team training. A monorepo is a single repository containing multiple artifacts like packages, backend services, and frontend applications. The first question to ask is what to include. It doesn't have to be the entire company code. Another decision is whether to go monorepo. If chosen, the repository will include multiple artifacts with separate package Jsons. The first decision is about installation and the approach of hoisting packages to the root of the repository.

And I'm not going to talk about that tool or that tool, Yarn or NX or TurboRepo. I want to take this opportunity and this few minutes ahead to explain what are the decisions, what are the strategies that we need to make when we are moving or when we have a Monorepo. Because these decisions, that sometimes we even take them without really worrying about them, we take them as implicit ideas, are actually impacting everything around our development process. They will impact the quality of our product, they will impact the speed of the development, the training that we need to give to the team and so on.

There is no wrong or right here that I'm going to talk about and say, look, this is one way to go, but it's more like, what are the considerations, what are the things that you need to think of. So again, if we go back, what is a monorepo? A monorepo is a single repository that has multiple artifacts, things that you want to share, things that you want to publish. This can be a package, this can be a backend service, this can be a frontend application, and so on.

And the first important question you should ask yourself when you're switching to a monorepo is what do I want to include in my monorepo? And this can be a lot of different things. This can be the different tools that we are using. This can be a frontend application, microservices, packages, backend servers, and so on. So, this is like the ground zero decision. What do I put? And it doesn't mean that you have to put the whole company code inside a single monorepo. It definitely can be just part of the code that you have.

The other decision is should I actually go monorepo? And there are a lot of articles, you should look for should I monorepo, and then there's a response, monorepo you should not, and so on. So, third, then, because there are different considerations and different things that you need to think of when you are having a monorepo. But let's say you decided that, yes, we want to go with the monorepo. Your repository will include multiple artifacts, and in the JavaScript world, and in the Node.js world, this means that it will have multiple package Jsons. Each package Json is related to an artifact that you want to publish.

And the first things that we have inside our package Json is what are the artifacts that, what are the dependencies, and what are the dependencies that this artifact needs in order to work? And this brings us to our first decision, which is install. This is how our packages look like. Okay. We have, on each package Json, we have a set of dependencies. And how we are going to install. So if we go with the same approach that we used in a poly repo when we had multiple repos, it means that under each package I will have a node module that will... I cannot do that. That will have all the packages that it needs. But since we know that node modules is a really, really big thing, and we don't want to replicate it multiple times for each one of the dependencies, we actually have another approach that we can use. And this approach is to hoist the packages. So instead of installing each package in its own workspace, we can actually move everything to the root of our project, of our repository, and just install them once there. And the reason this is going to work, we know about the search algorithm of node.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Modern Web Debugging
JSNation 2023JSNation 2023
29 min
Modern Web Debugging
Top Content
This Talk discusses modern web debugging and the latest updates in Chrome DevTools. It highlights new features that help pinpoint issues quicker, improved file visibility and source mapping, and ignoring and configuring files. The Breakpoints panel in DevTools has been redesigned for easier access and management. The Talk also covers the challenges of debugging with source maps and the efforts to standardize the source map format. Lastly, it provides tips for improving productivity with DevTools and emphasizes the importance of reporting bugs and using source maps for debugging production code.
Levelling up Monorepos with npm Workspaces
DevOps.js Conf 2022DevOps.js Conf 2022
33 min
Levelling up Monorepos with npm Workspaces
Top Content
NPM workspaces help manage multiple nested packages within a single top-level package, improving since the release of NPM CLI 7.0. You can easily add dependencies to workspaces and handle duplications. Running scripts and orchestration in a monorepo is made easier with NPM workspaces. The npm pkg command is useful for setting and retrieving keys and values from package.json files. NPM workspaces offer benefits compared to Lerna and future plans include better workspace linking and adding missing features.
The Future of Performance Tooling
JSNation 2022JSNation 2022
21 min
The Future of Performance Tooling
Top Content
Today's Talk discusses the future of performance tooling, focusing on user-centric, actionable, and contextual approaches. The introduction highlights Adi Osmani's expertise in performance tools and his passion for DevTools features. The Talk explores the integration of user flows into DevTools and Lighthouse, enabling performance measurement and optimization. It also showcases the import/export feature for user flows and the collaboration potential with Lighthouse. The Talk further delves into the use of flows with other tools like web page test and Cypress, offering cross-browser testing capabilities. The actionable aspect emphasizes the importance of metrics like Interaction to Next Paint and Total Blocking Time, as well as the improvements in Lighthouse and performance debugging tools. Lastly, the Talk emphasizes the iterative nature of performance improvement and the user-centric, actionable, and contextual future of performance tooling.
pnpm – a Fast, Disk Space Efficient Package Manager for JavaScript
DevOps.js Conf 2022DevOps.js Conf 2022
31 min
pnpm – a Fast, Disk Space Efficient Package Manager for JavaScript
pnpm is a fast and efficient package manager that gained popularity in 2021 and is used by big tech companies like Microsoft and TikTok. It has a unique isolated node module structure that prevents package conflicts and ensures each project only has access to its own dependencies. pnpm also offers superior monorepo support with its node module structure. It solves the disk space usage issue by using a content addressable storage, reducing disk space consumption. pnpm is incredibly fast due to its installation process and deterministic node module structure. It also allows file linking using hardlinks instead of symlinks.
End the Pain: Rethinking CI for Large Monorepos
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
End the Pain: Rethinking CI for Large Monorepos
Today's Talk discusses rethinking CI in monorepos, with a focus on leveraging the implicit graph of project dependencies to optimize build times and manage complexity. The use of NX Replay and NX Agents is highlighted as a way to enhance CI efficiency by caching previous computations and distributing tasks across multiple machines. Fine-grained distribution and flakiness detection are discussed as methods to improve distribution efficiency and ensure a clean setup. Enabling distribution with NX Agents simplifies the setup process, and NX Cloud offers dynamic scaling and cost reduction. Overall, the Talk explores strategies to improve the scalability and efficiency of CI pipelines in monorepos.
Federated Microfrontends at Scale
React Summit 2023React Summit 2023
31 min
Federated Microfrontends at Scale
Top Content
Watch video: Federated Microfrontends at Scale
This Talk discusses the transition from a PHP monolith to a federated micro-frontend setup at Personio. They implemented orchestration and federation using Next.js as a module host and router. The use of federated modules and the integration library allowed for a single runtime while building and deploying independently. The Talk also highlights the importance of early adopters and the challenges of building an internal open source system.

Workshops on related topic

React, TypeScript, and TDD
React Advanced 2021React Advanced 2021
174 min
React, TypeScript, and TDD
Top Content
Featured WorkshopFree
Paul Everitt
Paul Everitt
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
Mastering advanced concepts in TypeScript
React Summit US 2023React Summit US 2023
132 min
Mastering advanced concepts in TypeScript
Top Content
Featured WorkshopFree
Jiri Lojda
Jiri Lojda
TypeScript is not just types and interfaces. Join this workshop to master more advanced features of TypeScript that will make your code bullet-proof. We will cover conditional types and infer notation, template strings and how to map over union types and object/array properties. Each topic will be demonstrated on a sample application that was written with basic types or no types at all and we will together improve the code so you get more familiar with each feature and can bring this new knowledge directly into your projects.
You will learn:- - What are conditional types and infer notation- What are template strings- How to map over union types and object/array properties.
React at Scale with Nx
React Summit 2023React Summit 2023
145 min
React at Scale with Nx
Top Content
Featured WorkshopFree
Isaac Mann
Isaac Mann
We're going to be using Nx and some its plugins to accelerate the development of this app.
Some of the things you'll learn:- Generating a pristine Nx workspace- Generating frontend React apps and backend APIs inside your workspace, with pre-configured proxies- Creating shared libs for re-using code- Generating new routed components with all the routes pre-configured by Nx and ready to go- How to organize code in a monorepo- Easily move libs around your folder structure- Creating Storybook stories and e2e Cypress tests for your components
Table of contents: - Lab 1 - Generate an empty workspace- Lab 2 - Generate a React app- Lab 3 - Executors- Lab 3.1 - Migrations- Lab 4 - Generate a component lib- Lab 5 - Generate a utility lib- Lab 6 - Generate a route lib- Lab 7 - Add an Express API- Lab 8 - Displaying a full game in the routed game-detail component- Lab 9 - Generate a type lib that the API and frontend can share- Lab 10 - Generate Storybook stories for the shared ui component- Lab 11 - E2E test the shared component
Integrating LangChain with JavaScript for Web Developers
React Summit 2024React Summit 2024
92 min
Integrating LangChain with JavaScript for Web Developers
Featured Workshop
Vivek Nayyar
Vivek Nayyar
Dive into the world of AI with our interactive workshop designed specifically for web developers. "Hands-On AI: Integrating LangChain with JavaScript for Web Developers" offers a unique opportunity to bridge the gap between AI and web development. Despite the prominence of Python in AI development, the vast potential of JavaScript remains largely untapped. This workshop aims to change that.Throughout this hands-on session, participants will learn how to leverage LangChain—a tool designed to make large language models more accessible and useful—to build dynamic AI agents directly within JavaScript environments. This approach opens up new possibilities for enhancing web applications with intelligent features, from automated customer support to content generation and beyond.We'll start with the basics of LangChain and AI models, ensuring a solid foundation even for those new to AI. From there, we'll dive into practical exercises that demonstrate how to integrate these technologies into real-world JavaScript projects. Participants will work through examples, facing and overcoming the challenges of making AI work seamlessly on the web.This workshop is more than just a learning experience; it's a chance to be at the forefront of an emerging field. By the end, attendees will not only have gained valuable skills but also created AI-enhanced features they can take back to their projects or workplaces.Whether you're a seasoned web developer curious about AI or looking to expand your skillset into new and exciting areas, "Hands-On AI: Integrating LangChain with JavaScript for Web Developers" is your gateway to the future of web development. Join us to unlock the potential of AI in your web projects, making them smarter, more interactive, and more engaging for users.
Node Monorepos with Nx
Node Congress 2023Node Congress 2023
160 min
Node Monorepos with Nx
Top Content
WorkshopFree
Isaac Mann
Isaac Mann
Multiple apis and multiple teams all in the same repository can cause a lot of headaches, but Nx has you covered. Learn to share code, maintain configuration files and coordinate changes in a monorepo that can scale as large as your organisation does. Nx allows you to bring structure to a repository with hundreds of contributors and eliminates the CI slowdowns that typically occur as the codebase grows.
Table of contents:- Lab 1 - Generate an empty workspace- Lab 2 - Generate a node api- Lab 3 - Executors- Lab 4 - Migrations- Lab 5 - Generate an auth library- Lab 6 - Generate a database library- Lab 7 - Add a node cli- Lab 8 - Module boundaries- Lab 9 - Plugins and Generators - Intro- Lab 10 - Plugins and Generators - Modifying files- Lab 11 - Setting up CI- Lab 12 - Distributed caching
From Todo App to B2B SaaS with Next.js and Clerk
React Summit US 2023React Summit US 2023
153 min
From Todo App to B2B SaaS with Next.js and Clerk
WorkshopFree
Dev Agrawal
Dev Agrawal
If you’re like me, you probably have a million side-project ideas, some that could even make you money as a micro SaaS, or could turn out to be the next billion dollar startup. But how do you know which ones? How do you go from an idea into a functioning product that can be put into the hands of paying customers without quitting your job and sinking all of your time and investment into it? How can your solo side-projects compete with applications built by enormous teams and large enterprise companies?
Building rich SaaS products comes with technical challenges like infrastructure, scaling, availability, security, and complicated subsystems like auth and payments. This is why it’s often the already established tech giants who can reasonably build and operate products like that. However, a new generation of devtools are enabling us developers to easily build complete solutions that take advantage of the best cloud infrastructure available, and offer an experience that allows you to rapidly iterate on your ideas for a low cost of $0. They take all the technical challenges of building and operating software products away from you so that you only have to spend your time building the features that your users want, giving you a reasonable chance to compete against the market by staying incredibly agile and responsive to the needs of users.
In this 3 hour workshop you will start with a simple task management application built with React and Next.js and turn it into a scalable and fully functioning SaaS product by integrating a scalable database (PlanetScale), multi-tenant authentication (Clerk), and subscription based payments (Stripe). You will also learn how the principles of agile software development and domain driven design can help you build products quickly and cost-efficiently, and compete with existing solutions.