Scaling JavaScript Monorepos at Enterprise Level: Lessons From 200+ Packages

I'm sure you'll like this. Let's take a look. Good evening, everyone. In the world of agenting development, where the agents are making things much faster, so should be our CI, so that it can keep up the pace with the agents. But the enterprise monorepos, they are actually growing so huge. But that would also mean that they were also slow. And hence, welcome to the talk, monorepos metamorphosis. Here you can see how we can cut your CI load by 65% for over 200 plus monorepos packages.

So here I am, Karthik Shanmugam, working as a staff web platform engineer in 6SC. So here, let us see what could be the problem and how we can fix it. But in general, if your CI pipeline is running too slow, then that would mean it might be lying to you. The first thought, what could come to our mind is maybe we can increase the capacity, but no. That is not the problem. The problem is waste. In our CI, maybe we have a lot of waste that is running on, that is consuming more bandwidth, more power, and hence the CI might be running too slow.

So what we had, we have a large TypeScript platform. We have a monorepo with 200 plus packages, but 70 plus micro frontends. And here we have 10 plus product teams who are contributing to this monorepo. So imagine in such a huge monorepo, a single change in core libraries, whatever we have, This took around 75 minutes, which is almost one and a half hours, to run the entire CI and then make the product to be available for the testers or for the developers or for the product team to validate it. But 75 minutes, it's actually huge. For even a single commit, you need to wait for that amount of time. So we thought, okay, this is not good.

So then we had various ideas on how to fix it. And so we ended up with three main ideas, which could help us in targeting this particular CI failure, CI slowness, and that helped us to improve our CI speed by over 65%. So let's dive deep into each of these ideas, like what did we do and what is the result, what we got from that idea.

The first big idea, what we had is creating the build. So as I already told you, we have this 200 plus package monorepo with 70 plus micro frontends. And before NX, yes, for every commit, we would need to run the compile command for all the 200 plus packages, and then we need to deploy all 70 plus micro frontends. But why do we do the compile command? Because if in case there are any issues or errors that come up, since we have this static compiler, that would catch the issues beforehand before even going to deploy. So for that reason, we had this compile command, and it was running good. Thanks to NX. With NX, we were able to cut it down by half. But how did we do?

So what did NX help over here? Say for example, you have a file called price.ts. And that is the core package. And this core package is being used as a dependency in 100 plus packages. And among those 100 plus packages, we have 40 plus micro frontends. So with the help of NX, we were able to identify what are the affected packages for that particular change. I just reiterate the affected packages, but not the originally affected packages. So here on a single change on price.ts, we could observe that the 100 plus packages were being affected. So we would compile all the 100 plus packages. And then from that, we identified that 40 plus micro frontends have to be deployed. So then we deployed those 40 plus micro frontends. This is fine.

This is fine. Instead of running for 200 plus packages and deploying 70 plus micro frontends, we can reduce it to 100 plus and 40 plus. But is this what we want? Is this what we want for each and every commit on this price.ts file? No. Certainly not. So what did we do? Yes, of course, NX was helping us to reduce the blast radius. But we thought we can still reduce the blast radius so that our CI could run faster.

First one, what we thought, okay, we have 200 plus packages and 70 plus micro frontends. Would we need to compile 200 plus packages? Why not just compile 70 plus micro frontends alone? Because compiling a micro frontend would also compile the packages, whatever being declared as dependency. Would compile the files, whatever present inside the dependent packages for a particular micro frontend. Then we thought, okay, that sounds great. So instead of, in general, instead of compiling 200 plus packages, we can just compile the 70 plus micro frontends. So this will also compile the dependent packages. So that is the first step, what we thought.

So what do we mean by the leaves? So consider that in a tree, the micro frontends are the last ones, which wouldn't be consumed by any further packages. So they are the left-alones, which wouldn't be consumed further. We can also call them as the end-consuming applications. So what would you do? We would compile only those applications, only the micro frontends, which would in turn compile all the later files from the dependent packages. But would we need to deploy all of them for a change? Then we thought, okay, we do this compilation. Let us see what output we get out of compilation. So the output file, what we had is this ts buildinfo.

So the output file, what we had is this ts buildinfo. And surprisingly, this ts buildinfo had the files, exactly the number of files what that particular micro frontend is using. Say for example, we have this core package, and the file that I want to modify is price.ts. So when I run compile on a particular MFE, as per the diagram over here, we can call it as MFE card. So when I run compilation on this MFE card, the ts buildinfo, what got produced out of this compilation, you could see that we have the price.ts file got included into this MFE card from the core package. The same thing when we try to do on the MFE search, which does not include this price component at all, we identified that the core package is there, but price.ts is missing over here.

Then we thought, okay, we can compile the affected end consuming applications, but not deploy everything. We just deploy the originally affected micro frontends. So in this case, the affected end consuming applications would be MFE card and MFE search. But the originally affected end consuming application here would be MFE card. So in this case, we can just deploy MFE card and we don't need to deploy MFE search. You could see with NX, we identified that out of 200 packages, we have only 100 packages got affected. We take that. And then we have a filter. With that filter, we identified that 40 plus packages are the end consuming packages, which are not being consumed by anyone. So which we call as leaf.

So we compile only 40 plus packages and among them, we get the TS build output and check, grab the change file and find it in the TS build output. On further filtering, we will get only 10 plus micro frontends that needs to be deployed. So you could imagine compiling 100 plus packages and deploying 40 plus micro frontends. Now here we have compiling 40 plus packages and deploying only 10 plus micro frontends. You could see the ratio 100 is to 40, whereas here we have 40 is to 10. So that's a huge time reduce what we get with this algorithm. So yes, that's what. Just trusting NX alone. Yeah, deploy 40 plus MFPs. Use NX and also the output from TS build info. Now we just deploy 10 plus micro frontends.

Yeah, deploy 40 plus MFPs. Use NX and also the output from TS build info. Now we just deploy 10 plus micro frontends. You could see from 40 to 10. So the next lever, what did we try to get over here? Self-hosted runners and S3 cache. Yeah, we have GitHub runners. We can use GitHub runners for our CI pipeline. All good, no issues. Fine. But we do have some parallelism in our CI pipeline. But in this case, not all the pipeline, whatever we parallelize, doesn't need the full capacity. There might be some workflows which would need runner with full capacity. Some workflows which would need runner with medium capacity.

So the main CI, yes, of course, it needs a runner with maximum capacity, but the parallel action such as deploy, test, those things wouldn't need the runner with a high capacity. So we can go with the medium capacity for those runners. But is it possible to configure this with GitHub runners? We researched a bit over there, but we couldn't do that. So then we thought, why not bring our own self-hosted runners for this? Yes, that did work out. So this is not dynamic, but though we know what stage in the pipeline would require high capacity runners and what stage in our pipeline would require low capacity runners.

With that, we were splitting our entire pipeline for one set of workflows, we use high capacity runners for one set of workflows, we use medium capacity runners, and that would be within us. That solved one part of the problem, but that did open another part of the problem. So we have self-hosted runners, all good. Since we are using NX, we have the advantage of NX cache. So that would be caching the compile output, test output, all good. And the cache lives in GitHub. So self-hosted runner on our own AWS instance in EC2 and the cache what we have in GitHub. So there's a network connection going on for each and every commit in the CI between the GitHub and our own AWS EC2 instance.

Is it good? No, certainly not, because there was a lot of network communication going between them and that actually increased the cost. What we thought that would reduce actually didn't reduce, rather it nullified. Then we further researched and thought, okay, that's fine, but why not have this cache also within us? So yes, we spinned up an S3 bucket that could be used as cache for all these purposes. And this S3 bucket would live in the same region where we host our EC2 instance. So now the communication would be reduced, the throughput time would be reduced, and so the cost also gets reduced. So with this, not just the CI time got reduced, but also we were able to cut down the cost. That's the bonus what we had with this. So yes, we have our CI. Before we were with GitHub hosted runners and cache, and now we move towards EC2 runner with S3 cache on the same region. Perfect.

And now let's move to the next lever, TypeScript compilation explosion. As I already told you, we were compiling all our packages before even starting with deployment. But anyway, here, there was a silent issue which was there for years in our CI pipeline. First issue is type explosion. So what is type explosion? Let me give you an example. For sure, all the companies would have their own typography components for them for the web frontend. So this typographic component, we have a scenario where the development team can actually extend this typographic component to a specific user-defined element, whatever they want to target for. And this specific user-defined element can be generic. We cannot define anything on it. So what did we do? We type this particular thing to jsx.intrinsic elements. But do you know what? jsx.intrinsic element actually includes all possible HTML tags in the universe. And because of this, whenever we are running TypeScript compilation, all the stacks was again being compiled, was being included in the build, which is actually a waste. Thanks to AI. Here, I mean, with this what we finally got, AI doesn't just help us to ship new products, but also can help us in fixing the bugs, whatever we have. But AI, we were running this TypeScript compilation. And we did find out that this particular jsx.intrinsic element was causing a big issue. That was one particular part. The next particular part is barrel files. Of course, since using a monadopo, having multiple packages, we would define multiple barrel files for the ease of export and import. But that was also causing a huge issue silently.

So because of this, I changed in one of the micro frontend, when we compiled that micro frontend, it was taking around 60 to 70 seconds. Imagine a compile on a micro frontend taking around 60 to 70 seconds. So what did we do? We restricted that the first one, type explosion, we restricted our type. So that, say, for example, for typography, the use case what I explained for typography, we don't need all possible HTML tags to be included over here. Because in general typography, it could be div, span, paragraph, section, main, or whatever possible tags that could help us for formatting the text. But we don't want image tag, SVG tag, and all these things. So we made this type to be much restrictive.

And barrel files. Yeah, this is something which we are doing it now. There are a lot of barrel files in our monadopo, which we are trying to kill a lot one by one by one. We are still with it. But both together did help us to reduce the compile time. For one MFV, before, it took around 60 to 70 seconds. But now it just takes 30 to 30 seconds. So as an overall, for a commit in the PR, the CI, the compilation part, which was taking around 35 minutes, now got reduced to 10 to 15 minutes. See a great improvement what we have. So this actually helped us in reducing our compile time by over 60%.

So these three things did help us to improve our CSV a lot. But do we have any more things which could help us over here? Yes, we identified three smaller things which could help over here. One is pnpm. Yes, we were using npm and then YARN classic. But the dependency management was poor in that. So then we moved towards pnpm. And you know with pnpm, the dependency management is much faster, simpler, and easy to break and all those things. So we migrated towards pnpm. And the next concern what we had is some of the things whatever we have in our CI is totally waste. Say for example, translations. So as I told you the scenario what we had to deploy 10 plus micro frontends. The translations are already bundled into our CI. And the transition was running like one minute per micro frontend.

So for 10 plus micro frontends, it would be 10 plus minutes. So then what we thought, maybe we can move the translations out of the CI. So hence, we completely decoupled the translation and that helped us to reduce the CI runtime a little further. And the third most important thing what we did over here, the smaller wins, parallelized test. We used NX stacks. So we scoped our packages into three main branches, like NX app, NX core libraries, NX business libraries. And we parallelized our test, so that a single 30 minute test job is now just taking 10 to 15 minutes.

Yes, they are running in parallel, but the time taken to run the full test job just got reduced from 30 minutes to 10 minutes. So with all the things, whatever I told you, we were able to achieve a 65% reduction in our total CI build plus test time. So what are the lessons that we have now? Small wins come from fast. Don't always target to have big wins at the first. Target small things, and that could actually help you to get a big win. The graph is a source of truth. Analyze your dependency tree. Analyze where you can cut the cost, and that could help you a lot.

And the third and the most important point, decouple things. Don't just couple everything together. So decouple it and try to optimize it. So with that, thanks everyone. So now what we have, a change in core library from 75 minutes to a branch break and changing one of the microfinance applications, our branch break time got reduced to a coffee time. But anyway, thanks to AI as well in helping us to do some analysis over here. Thank you everyone.