Fast React Monorepos with High Quality DX

Hey, and welcome to my talk about fast React monorepos with high quality DX. But before we go ahead, let me first introduce myself. My name is Joris Schumfler, I'm a Director of Developer Experience here at Narwhal. I'm also a Google Developer Expert in Web Tech. I'm an ECHI instructor and Cybers Ambassador.

Narwhal is the company behind NX, and we provide consulting services for Fortune 500 companies, but not only actually. And our range goes from helping with Angular development, React development, but in particular, helping with monorepos, so we help migrate to monorepos scenarios, set up new monorepos, but especially help succeed those monorepos in the long run. We're also the creators of the open source toolkit and NX. And if in case you missed it, most recently, we also took over Stewardship of Leerna, which is now joining forces with NX.

So what is NX? NX is a smart, fast and extensible build system and can be used for monorepos. And today I'm not going specifically into NX directly, but I'm using it as an example of a feature set like a fully feature complete monorepo tool that can help you not just getting started fast, but also succeed in the long run. Now NX is hugely popular. So we have seen crossed $1 million in December and are about to cross $2 million per week in June probably. So it is super exciting, and it shows how much traction the monorepos space recently got.

Now, what are monorepos? Rich Harris, a couple of weeks ago, posted a tweet, which I'm 100% agreeing with, because I have the same question from people, and I have to answer and explain that over and over again. So he was basically not happy with the term monorepo. And the problem is that monorepo implies, or what many people think, is that you need to have one single repository for the entire organization. Now, it's perfectly clear why people think that, but in reality, what we see when we work with now, for instance, with large companies, is more something like this. So you have large monorepos, couple of them inside your company, maybe split by department or organization or domain, and then you have also the existing polyrepos, or new polyrepos even, that come up within that organization. So if you have a couple of such a mix of landscape, and you share stuff over registries, over internal registries, even monorepos share some of the parts for the outside. Because if you have some polyrepos that might want to benefit from, let's say, the component library you build within a given monorepo, you might want to publish that to a registry as well, apart from just using it within that monorepo. So that's perfectly fine. Now, probably having that a term like multiproject repo versus single project repo would be more useful or more meaningful, but I'm not going to coin a new term here. So whenever I'm speaking about monorepos, what I intend is basically a single Git repository with two or more distinct projects, and Polyrepo on the other side is more classic scenario, which is a lot half-basic, which is like single repository with one project in it.

So how do monorepos look like? They come in different forms. So a lot, especially in the open source world, what you see is just like one repo with code collocation. So you have a couple of projects in there, they don't necessarily relate to each other or have relationships between them, so most of the time it's just for reusing CI setups or common utilities for being quicker in publishing to NPM, things like that.

But the real value you get out of a monorepo is actually if those packages relate to each other to some degree. So they share code and facilitate basically collaboration within the monorepo. And in fact, a monorepo is all about architecting your system. So, many times what you have is scenarios like this. You have different domain areas within your organization, and very often they exist as separate polyrepos. But if you converge them into a monorepo, a scenario might look like the following. So you have these domains, and as you evolve your monorepo, you split it up into smaller subprojects. So you have like an application per such domain, which you can deploy independently, and each domain has some specific libraries that are not shared with other domains or within other parts of the monorepo, but they're specific to that single domain.

The main reason of having those libraries is to split up the code base, have teams more organized and be able to work on single projects within that workspace, basically within that single domain, and you can also have much more fine-grained APIs in that case, which help you create more maintainable code in the long run. Now, obviously, you get more out of it once you also start reusing things. So at the very bottom of this image here, for instance, you see those libraries, which are used a lot by the upper-level domains, and those libraries are usually more code utilities, like authentication libraries, logging libraries, maybe for calculating dates, or also maybe even some more specific parts to your domain, to your organization. For instance, it could be like a UI component library that lives down there, which is used by the various parts of the domains to have like a uniform UI appearance and look and feel.

So what you rather do is you split up rather than creating a monolith, in this case, now down here, as you can see here, we have the more utility libraries, which I've just explained, but those are not the only ones you can even have connection between domains. And also in those cases, it is very useful to create, for instance, dedicated sharing libraries, which expose like a common data object, it can be used by other domains and common API interfaces, such that domains can communicate with each other. For instance, the checkout domain might need utilities, be it like UI utilities, but also communication utilities for the product part of the domains of your organization. And finally, as I mentioned before, you might even have those very leaf nodes, which are some very, very general purpose libraries, such as UI components, which you might also want to publish outside the monorepo, such as like polyrebus can consume those. Now, as you can see my title, the fast part is very important in the monorepo because if you approach them naively, it might be easy to get started, but like a year in or so, things might get very, very slow, very quickly. So if you PR on CI takes over an hour to build, that hinders new feature development and slows down your teams. So basically create a counterpart rather than having some more collaboration. You basically start to be detrimental. So you definitely need to have tooling that allows you to build just what changed and most of the development or most of monorepo tools have something that is called a project graph and what can be done with a project graph is, for instance, if such library as here in the picture in the middle changes, those tools can walk up those projects graphs to understand what needs to be built or tested or linted. So in this case, you can see we can already cut out quite some libraries and applications that don't need to be touched. So obviously, it will be much faster than also running tests for those. You can even leverage this, for instance, to understand what needs to be redeployed, such as we have always the latest version and production. So in this case, you can see all of the applications in our domains are affected because it's a very central library in this case that got changed.

You can even leverage this, for instance, to understand what needs to be redeployed, such as we have always the latest version and production. So in this case, you can see all of the applications in our domains are affected because it's a very central library in this case that got changed. And so we would probably want to all deploy those applications as well.

Now, affected is just one part. Caching is what makes that even faster. For instance, let's take that same scenario where. We changed those libraries and they all get run in CI and one of the lib, like in the upper left corner, failed the test. So the developer looks at them, pulls it down, fixes the test, pushes it up again. Now we don't have to run all of them again because all of the other parts are not affected by the change that was just made. They're still affected compared to the main branch like master or main. However, we only need to run tests again or computation again for the lib and the app in the upper left corner. Because all the other results can be fetched out of the cache. So obviously you get much more speed.

Now the distribution is really the key here because right now the cache, for instance, in NX specifically, by default it's just local. So local to ever developers workstation, which means the local development gets quicker and speeds up but you cannot really leverage it, especially in CI. And that's where we see a lot of folks use actually the caching because you want to make sure PRs are fast to get merged in. And so for being able to have a distributed cache, you need some central server-side cloud counterpart, the NX cases and NX Cloud, which allows you to distribute that cache among team members as well as CI. And NX Cloud goes even a step further. So it doesn't only distribute the actual caching, but it also distributes the task execution, which makes it even faster. So normally what you do on CI is you get that project graph that was affected by the change. And then you have a set of agents because what you want to do is you want to parallelize as much work as possible. And so you can actually split it up in equal batches. That's usually what happens. That's the most naive approach, like just split them up in equal batches and distribute them to the agents. What might happen, though, is that some agents are super fast because they got tasks to complete in a couple of seconds while other agents run for minutes. And so still the entire CI run has to wait until all the agents finish. And so therefore the utilization of the agents is suboptimal. And also the actual time is suboptimal in the end. So it's not that fast as you would expect. Now, NX Cloud, for instance, distributes the caches or distributes not only the caches, but also the tasks uniformly based on historical data.

And so it knows based on the graph, which processes can be built in parallel, which basically where they may just assign a single task to an agent because it's a long running one and which one needs to be serially executed. And in the end, the DX part, it comes in here is that all the logs and artifacts are being collected and sent back to the actual main node and grouped together. And so from a developer perspective, if you get an error or want to look at the actual run, you can just go to the logs and you will see them the same way as if they would have been run on a single machine.

And it's even cooler because like all those tasks now are distributed across the agents. You here, you can see a screenshot of NX cloud showing the utilization of the agents, and you can see how they are balanced out also based on the previous data that NX cloud has gotten and therefore it knows how to best parallelize those tasks. And you get even like a nice visualization. So whenever a PR runs, you can in real time understand how many agents are currently running, which tasks are running on which agent, which is particularly important for debugging purposes.

Now, this was kind of the fast already a bit interleaved with the DX part, as we have just seen, like the visualization for instance, that helps you from a developer experience perspective to debug things. But a DX is very important in React, more in the monorepo scenario in general, because there's nothing like having a fast, super like, worst feature rich monorepo setup, but it's super hard to use as a, from a developer perspective or to configure. And so in NX specifically, the important part is the developer experience part. And so, first of all, things need to be incrementally adopted, right? So NX can be set up in two different main ways. So you can get started with a core setup, which means you don't use any plugins that come with NX, you just install NX and use it on your existing monorepo.

And so for migration scenarios, this is ideal because you can still use the same infrastructure that you had before, and NX will just make it much faster to run the tasks. So the thing you leverage is the Fast task scheduler, the caching, and also the distributed task execution if you use NX cloud, which you've just seen. The other approach is that if you start new, you can actually benefit from setting up NX with some pre-configured things. So you can say, okay, I know that I'm using a react monorepo setup because react is my main focus. So you can already use some pre-configured templates that NX comes with. And so then NX would make sure that you get like Jest setup, ESLint, Prettier, Cypress configuration set up for you. So you don't have to worry about a lot of that stuff. It's usually that is the best option if you start a new monorepo right now.

So for the core setup, it is super easy to adjust. You basically just run the npx add NX to monorepo command, which would add it to any npm, yarn, pnpm workspace, and just make it a lot faster. Interestingly, also, as I mentioned initially, we took over Seo Chip for Lerna, and now we can do some very interesting things, especially for your Lerna workspaces. For instance, right now, if you're using 5.1 plus of Lerna, you can just install NX and set the use NX to true on your Lerna JSON, and it would automatically defer the task scheduling to NX, making Lerna super fast without you having to change anything else, which is super important I think, from a developer economics.

Another thing is beautiful output. Now, this might not sound as important initially, but if you think like how often you look at the terminal outputs as a developer, and so in order to reduce the cognitive load, NX really just shows you what is most important right now. So it doesn't show you, for instance, the execution of depending tasks as in this animation here, but just what gets executed, unless, of course, some error occurs. That would be highlighted, obviously, big and then in red, and even if you rerun the task and they get cached, the output is exactly the same. So you basically have a much, much less lower cognitive load when you parse those logs because you just see what you need right now. Also ID integration.

This is specifically interesting for newcomers, but also like if you want to explore simply the capabilities that you have, right? So, for instance, for Visual Studio Code, but there's also community extensions for WebZone, for instance, we have the NX console extension, which allows you to navigate within an NX workspace and in the future even a learner workspace, allowing you to browse commands that you can execute, things that you can generate in that workspace visually from within Visual Studio Code.

Another part, the visual part is very important also when exploring the workspace. So, every or many of those monorepo tools have a so-called project graph underneath. Now in NX we went even a step further and we visualized that as a dynamic web application that you can start just from your CLI. And so here you can see how you can, for instance, take two nodes and basically have an NX visualize the shortest path between those nodes as well as all potential paths. And you can imagine how that can help to understand why some nodes connect with each other, like why the connection actually exists, but also for instance for debugging circular dependencies.

And finally, a very important part is not just like getting started quickly as we have seen in terms of being fast, but also continued support as your monorepo grows. So, co-generators are one thing that can help you with that, which NX comes with, meaning for instance, if you have like for easier onboarding junior developers, you have already some co-generators in place such that like all the libraries and applications are generated in a uniformly way as you would expect. So, you don't have them copy and paste, which is very error prone, of course, existing libraries remove existing code, and then basically continue working on that. But you can actually even not just use the NX provided generators, but tailor them really down to your own needs. So, you can customize them, so it only gets generated whatever you need in your current workspace. Easily, that can also be triggered by Visual Studio Code. And so, here for instance, you can see in this animation example of setting up that distributed task execution environment for CI. So, you can just generate such a CI setup, which is like super useful. And it's even just a couple lines of code, if you take a look in the end. So, this is an example of how powerful those generators can be without you having to write long docs explaining how to set up those things. Similarly, also, for instance, if you want to set up a React Webpack modification setup, NX comes with already such a plugin that you can use for setting up the main host and remote applications. And then you can use that to build up your existing stuff and continue developing on top of those generators.

Finally, also preventing Spaghetti dependencies. And this is something people underestimate quite a lot if they start in monerepos. Because they get started very excited in the beginning. But then if you are in like half a year where you have multiple teams working on a monerepo, things might get messy very, very quickly. So, people importing from Libs that they shouldn't import, and maybe even just by accident, because they saw some cool utility library and they were not supposed to use that. So, your code quickly might become unmaintainable. In the next, for instance, as you can see here, for instance, like if we have those domains, like you can use those library as domains, right? Where you can clearly expose a public API, such that other people from a different domain can then leverage those functions within their own. Now, how can you, however, prevent that someone imports just a different Lib? In theory, you can just do that because you live in the same monerepo. Now, in the next we have so-called tagging system. This is simply strings. So, you can say, like, I have a scope column, some name, which for instance could be the domains and to give to all the apps and Libs in a certain domain, you give those tags. You can add that in your configuration and all those Libs would get that.

And then you also give to that API, for instance, specific tag API to mark it as something that can be consumed from outside the domain. And then the next comes just with a specific ESLint rule. So, in this ESLint rule here, for instance, you can see how we define a source tag and say, OK, something that is called scope checkout, which is our specific checkout domain, can only depend on other apps and Libs that also live in that scope checkout so that also have that tag or that they have a scope API. Because, like, if there's a scope API, there is something that can be explicitly used. So, that's fine. And this can then be those checks run in CI. They run in your editor. So, you immediately see if you import something wrong and are, therefore, very, very powerful to keep your code maintainable.