Scaling JavaScript Monorepos at Enterprise Level: Lessons From 200+ Packages

Good evening, everyone. In the world of agentic development, where the agents are making things much faster, so should be our CI, so that it can keep up the pace with the agents. But the enterprise monorepos, they are actually growing so huge. But that would also mean that they were also slow. And hence, welcome to the talk, monorepos metamorphosis. Here you can see how we can cut your CI load by 65% for over 200 plus monorepos packages. Here I am, Kartik Shanmugam, working as a staff web platform engineer in 6th SE.

So, here, let us see what could be the problem and how we can fix it. But in general, if your CI pipeline is running too slow, then that would mean it might be lying to you. The first thought, what could come to our mind, is maybe we can increase the capacity. But no, that is not the problem. The problem is waste. In our CI, maybe we have a lot of waste that is running on, that is consuming more bandwidth, more power, and hence, the CI might be running too slow.

So, what we had, we have a large TypeScript platform. We have a monorepo with 200 plus packages with 70 plus micro frontends. And here, we have 10 plus product teams who are contributing to this monorepo. So, imagine, in such a huge monorepo, a single change in core libraries, whatever we have, this took around 75 minutes, which is almost one and a half hours, to run the entire CI and then make the product to be available for the testers or for the developers, or for the product team to validate it. But 75 minutes, it's actually huge. For even a single commit, you need to wait for that amount of time.

So, we thought, okay, this is not good. So, then we had various ideas on how to fix it. And so, we ended up with three main ideas which could help us in targeting this particular CI failure, CI slowness, and that helped us to improve our CI speed by over 65%. Let's dive deep into each of these ideas, like what did we do, and what is the result, what we got from that idea.

The first big idea, what we had, is creating the build. So, as I already told you, we have this 200 plus package monorepo with 70 plus micro frontends. Package monorepo with 70 plus micro frontends. And before NX, yes, for every commit, we would need to run the compile command for all the 200 plus packages, and then we need to deploy all 70 plus micro frontends. But why do we do the compile command? Because if in case there are any issues or errors that come up, since we have this static compiler, that would catch the issues beforehand before even going to deploy. So, for that reason, we had this compile command, and it was running good.

Thanks to NX. With NX, we were able to cut it down by half. But how did we so what did NX help over here? Say, for example, you have a file called price.ts, and that is in the core package. And this core package is being used as a dependency in 100 plus packages. And among that 100 plus packages, we have 40 plus micro frontends. So, with the help of NX, we were able to identify what are the affected packages for that particular change. I just reiterate, the affected packages, but not the originally affected packages. So, here, on a single change on price.ts, we could observe that the 100 plus packages were being affected. So, we would compile all the 100 plus packages. And then from that, we identified that 40 plus micro frontends have to be deployed. So, then we deployed those 40 plus micro frontends.

So, this happened for every single commit on this particular file, price.ts. This is fine. Instead of running for 200 plus packages and deploying 70 plus micro frontends, we can reduce it to 100 plus and 40 plus. But is this what we want? Is this what we want for each and every commit on this price.ts file? No, certainly not. So, what did we do? Yes, of course, NX was helping us to reduce the blast radius. But we thought we can still reduce the blast radius so that our CI could run faster.

So, what did we do? First one, what we thought, okay, we have 200 plus packages and 70 plus micro frontends. Would we need to compile 200 plus packages? Why not just compile 70 plus micro frontends alone? Because compiling a micro frontend would also compile the packages, whatever being declared as dependency, would compile the files, whatever present inside the dependent packages for a particular micro frontend. Then we thought, okay, that sounds great. So, instead of in general, instead of compiling 200 plus packages, we can just compile the 70 plus micro frontends. So, this will also compile the dependent packages. So, that is the first step, what we thought. Instead of doing the compilation on all the packages, we compile only the leaves.

So, what do we mean by the leaves? So, consider that in a tree, the micro frontends are the last ones which wouldn't be consumed by any further packages. So, they are the left alones which wouldn't be consumed further. We can also call them as the end consuming applications. So, what would you do? We would compile only those applications, only the micro frontends, which would in turn compile all the related files from the dependent packages. But would we need to deploy all of them for a change? Then we thought, okay, we do this compilation, let us see what output we get out of compilation. So, the output file, what we had is this ts buildinfo.

And surprisingly, this ts buildinfo had the files, exactly the number of files what that particular micro frontend is using. Say, for example, we have this core package. And the file that I want to modify is price.ts. So, when I run compile on a particular MFE, as per the diagram over here, we can call it as MFE card. So, when I run compilation on this MFE card, the ts buildinfo, what got produced out of this compilation, you could see that we have the price.ts file got included into this MFE card from the core package.

The same thing that we tried to do on the MFE search, which does not include this price component at all, we identified that the core package is there, but price.ts is missing over here. That's true. Because MFE search is not using this price module at all. Because it's just a search module. Then, we thought, okay, we can compile the affected end-consuming applications, but not deploy everything. We just deploy the originally affected micro frontends.

So, in this case, the affected end-consuming applications would be MFE card and MFE search. But, the originally affected end-consuming application here would be MFE card. So, in this case, we can just deploy MFE card and we don't need to deploy MFE search. So, that is what we got over here. You could see with NX, we identified that out of 200 packages, we have only 100 packages got affected. We take that. And then, we have a filter. With that filter, we identified that 40 plus packages are the end-consuming packages which are not being consumed by anyone, so which we call as leaf.

So, we compile only 40 plus packages. And among them, we get the TS build output and check, grab the change file and find it in that TS build output. On further filtering, we will get only 10 plus micro frontends that needs to be deployed. So, you could imagine compiling 100 plus packages and deploying 40 plus micro frontends. And now here we have compiling 40 plus packages and deploying only 10 plus micro frontends. You could see the ratio. 100 is to 40, whereas here we have 40 is to 10. So, that's a huge time reduce what we get with this algorithm.

So, yes, that's what. Just trusting NX alone, yeah, deploy 40 plus MRFs. Use NX and also the output from TS build info. Now we just deployed 10 plus micro frontends. You could see from 40 to 10. So, the next lever, what did we try to get over here? Self-hosted runners and S3 catch. S3 catch. Yeah, we have GitHub runners. We can use GitHub runners for our CI pipeline, all good, no issues, fine. But we do have some parallelism in our CI pipeline. But in this case, not all the pipeline, whatever we parallelize, doesn't need the full capacity.

There might be some workflows which would need runner with full capacity, some workflows which would need runner with a medium capacity. So, the main CI, yes, of course it needs a runner with maximum capacity. But the parallel actions such as deploy, test, those things wouldn't need the runner with a high capacity. So, we can go with the medium capacity for those runners. But is it possible to configure this with GitHub runners? We researched a bit over there, but we couldn't do that. So, then we thought, why not bring our own self-hosted runners for this? Yes, that did work out. So, this is not dynamic, but though we know what stage in the pipeline would require high capacity runners and what stage in our pipeline would require low capacity runners. So, with that, we were splitting our entire pipeline for one set of workflows, we use high capacity runners.

For one set of workflows, we use medium capacity runners. And that would be within us. That solved one part of the problem, but that did open another part of the problem. So, we have self-hosted runners all good. And since we are using NX, we have the advantage of NX cache. So, that would be catching the compile output, test output, all good. And the cache lives in GitHub. So, self-hosted on our own AWS instance in EC2 and the cache, what we have in GitHub. So, there's a network connection going on for each and every commit in the CI between the GitHub and our own AWS EC2 instance. Is it good? No, certainly not. Because there was a lot of network communication going between them and that actually increased the cost. What we thought that would reduce actually didn't reduce, rather it nullified.

Then we further researched and thought, okay, that's fine. But why not have this cache also within us? So, yes, we had, we could, we finned up an S3 bucket that could be used as cache for all these purposes. And this S3 bucket would live in the same region where we host our EC2 instance. So, now the communication would be reduced, the throughput time would be reduced, and so the cost also gets reduced. So, with this, not just the CI time got reduced, but also we were able to cut down the cost. That's the bonus what we had with this. So, yes, we have a CI. Before we have a GitHub hosted runners and cache, and now we move towards EC2 runner with S3 cache on the same region. Perfect.

And now let's move to the next lever, TypeScript Compilation Explosion. As I already told you, we were compiling all our packages before even starting with deployment. But anyway, here, there was a silent issue, which was there for years in our CI pipeline. First issue is Type Explosion. So, what is Type Explosion? Let me give you an example. For sure, like, all you, I mean, all the companies would have their own typography components for them for the web front end. So, this typography component, we have a scenario where the development team can actually extend this typography component to a specific user-defined element, whatever they want to target for. And this specific user-defined element can be generic. We cannot define anything on it.

So, what did we do? We typed this particular thing to jsx.intrinsic elements. But do you know what? jsx.intrinsic element actually includes all possible HTML tags in the universe. And because of this, whenever we are running TypeScript Compilation, all the stacks was again being compiled, was being included in the build, which is actually a waste. Thanks to AI. Here, I mean, with this, what we finally got, AI doesn't just help us to ship new products, but also can help us in fixing the bugs, whatever we had. With AI, we were running this TypeScript compilation and we did find out that this particular jsx.intrinsic element was causing a big issue. That was one particular part.

The next particular part is barrel files. Of course, since you're using a monorepo, having multiple packages, we would define multiple barrel files for the ease of export and import. But that was also causing a huge issue silently. So, because of this, I changed in one of the micro frontend. When we compiled that micro frontend, it was taking around 60 to 70 seconds. Imagine a compile on a micro frontend taking around 60 to 70 seconds. So, what did we do? We restricted the first one, type explosion. We restricted our type so that, say for example, for typography, the use case what I explained for typography, we don't need all possible HTML tags to be included over here. Because in general, typography, it could be div, span, paragraph, section, main or whatever possible tags that could help us for formatting the text. But we don't want image tag, SVG tag and all these things.

So, we made this type to be much restrictive. And barrel files. Yeah, this is something which we are doing it now. There are a lot of barrel files in our monorepo, which we are trying to kill a lot one by one by one. We are still with it. But yes, both together did help us to reduce the compile time. For one MFV before it took around 60 to 70 seconds. But now it just takes 30 to 30 seconds. So, as an overall for a commit in the PR, the CI, the compilation part, which was taking around 35 minutes, now got reduced to 10 to 15 minutes. See a great improvement what we have. So, this actually helped us in reducing our compile time by over 60%. So, these three things did help us to improve our CSV a lot.

But do we have any more things which could help us over here? Yes, the identified three smaller things which could help over here. One is PMPM. Yes, we were using NPM and then Yarn Classic, but the dependency management was poor and that so then we moved towards PMPM. And you know, with PMPM, the dependency management is much faster, simpler, and easy to break and all those things. So, we migrated towards PMPM. And the next concern what we had is some of the things whatever we have in our CI is totally waste. Say, for example, translations. So, as I told you the scenario of what we had to deploy 10 plus micro frontends. The translations are already bundled into a CI and the transition was running like one minute per micro frontend. So, for 10 plus micro frontends, it would be 10 plus minutes. So, then what we thought maybe we can move the translations out of the CI. So, hence we completely decoupled the translation and that helped us to reduce the CI runtime a little further.

And the third most important thing what we did over here, the smaller wins, parallelized test. We used NX stacks. So, we scoped our packages into three main branches like NX app, NX core libraries, NX business libraries, and we parallelize our test. So, that a single 30 minute test job is now just taking 10 to 15 minutes. Yes, they are running in parallel, but the time taken to run the full test job is just got reduced from 30 minutes to 10 minutes. So, with all the things, whatever I told you, we were able to achieve a 65% reduction in our total CI build plus test time.

What are the lessons that we have now? Small wins come from fast. Don't always target to have big wins at the first. Target small things and that could actually help you to get a big win. The graph is a source of truth. Analyze your dependency tree. Analyze where you can cut the cost and that could help you a lot. And the third and the most important point, decouple things. Don't just couple everything together. So, decouple it and try to optimize it. So, with that, thanks everyone. So, now what we have? A core change in core library from 75 minutes to a brunch break and changing on one of the microfinance applications, a brunch break time got reduced to a coffee switching time. But anyway, thanks to AI as well in helping us to do some analysis over here. Thank you, everyone. Thank you.