Challenges for Incremental Production Optimizations

So, my name is Tobias Koppers, and I work at Vercel and work on TurboPack. TurboPack is a new bundle we're working on, similar to Webpack, but we're designing it from scratch. We're using it for Next.js, and our mission for TurboPack from the beginning was to focus on incremental builds. We want to make incremental builds as fast as possible, even if we have trade-offs on initial builds, because we think that most developers tend to wait on incremental builds often, because that is what you do while developing.

On the other hand, we also try to make every build incremental. We try to make that you only have to do your initial build once, and then spend the remaining time only on incrementables. That also means if you upgrade the Next.js version or the TurboPack version, we don't want you to lose your cache, or if you upgrade the parentheses, and this also includes production builds, so we want to focus on production builds too. That's what this talk is about. Production builds have some unique challenges I want to cover, and go a little bit over that.

So, if we look at both sides, like on one side we have TurboPack, which is really focused on incrementables, and on the other hand we have production builds which are really optimised, and that doesn't look too opposed, but in fact, there are some challenges that come with these optimisations we do in production builds usually. Because optimisations often, like, so in development we can focus on making it as fast as possible, even if you trade-off bundle size, or make builds a little bit larger, or do something of these trade-offs, but in production builds you don't want to make these trade-offs. You want to make production builds as optimised as possible, and then you basically have to trade-off with maybe performance on that stuff. Bringing them both together, like incrementables and production optimisations, that's a bit of a challenge.

So let's look at some common production optimisations for bundlers. The one you probably know is called tree-shaking. It's basically all about, like, you have your repository with a lot of files, and in your application, you only want to include the files that you're actually using on pages, and bundlers usually do that by following a dependency graph, by following your imports, and only including the files you actually reference. But it goes more low-level. Every file has multiple exports, usually, and you maybe have some kind of utility libraries where you have a lot of function declarations, and tree-shaking also looks at these and looks into your source code, and looks which of these exports are actually used in your whole application, and only includes them in your bundle, and basically throws away the remaining ones. That's actually the first challenge.

We have to look at your whole application to figure out which exports are used in your application, and looking at the whole application is basically the opposite of making it incremental, where incrementables usually want to look at one piece at a time, and, if something changes, you want to minimise the changes, the effects on that. Basically, this whole application optimisation is a little bit opposed to that. The next optimisation is called export mangling. Now, as a good developer, you made this function declaration, and gave them good names, meaningful long names to actually do cool stuff, and give good explanation to your co-workers, and whatever, and, in production, you don't want to leak these long names into your bundles. You want the bundler or the tool to optimise it, and usually bundlers do that by renaming things like A, B, and C, something like that, so it's just more optimised. But there's also a problem with that. If you rename these exports in a file, and you also have to rename it on the import side, so every module that is importing that module with the declarations need to reference not the long names but the A, B, C, the mangled names, so basically you have this effect where you change one module, or your optimisation changes one module, and that affects a lot of other modules, and this is also a little bit challenging for incrementables, because then you change one thing, and it bubbles up to multiple other changes, and yes, it's not really incremental.

Another optimisation is chunking, so usually your bundler doesn't put all your modules into a single file and serve that for all pages, because it would end up with huge megabytes of a bundle, so we split it up, or the code splitting, we split it up into bundles per page, but we also do something like a common chunk or shared modules optimisation, where we figure out if there are modules shared between your multiple pages or multiple chunks, and then we put them into a shared file so we don't have to load the same modules multiple times, basically load the shared module once, and finding shared modules also requires looking at multiple pages at the same time, so this is again an whole application optimisation. There is an optimisation called scope hoisting, where we basically don't want, as a good developer, you write up many small modules, because that's organising your code well, and we basically don't want this abstraction of modules leaking into the runtime, so we want to get rid of many small modules at one time, and basically make only what we actually need at one time.

So scope hoisting optimisation basically merges modules together under certain conditions, and the tricky thing are the conditions in this case. We have to figure out which modules are always in the whole application, always executed in the same order, in this order, and then we can basically merge them together because then we know this is the order that they're executed in. So basically, finding this kind of matching this condition, finding something happens always in your whole application, is again an whole application optimisation. Then there are some simple optimisations like dead code elimination, which just omits code that is not used, or minification, which just writes your code more efficiently, omits white space commands and that stuff. Then the last optimisation is content hashing, where we basically put on every file name we put in hash at the end to make it long-term cacheable, basically means you can just send an immutable cache header, and then the browser cache can cache that. And yes, that's basically all the eight optimisations I come up with.

So if we summarise that, like in this table, you see that a bunch of these optimisations, or like half of them need our whole application optimisations, and a bunch of them also have effects where you change something and then the importer side of that changes. These are a bit complicated for incremental builds. But we look at this later. So in general, for incremental production builds, what we need to do is we need to have two ingredients, or at least two ingredients.

So one thing is in production builds, what usually happens is that production builds are usually executed in a fresh container, on a fresh clone, a new NPM install. So we need something where we basically can't rely on having a memory cache with all your cache data, so we need some kind of cache persistence, and that's basically called persistent caching. And what TurboFact does is basically, during execution, it builds up a graph of all dependencies of our sub-executions. And we do that for tracking dependencies between multiple subtasks, and basically we do that to know that if something changes, we can bubble up changes to the graph, and know what we actually have to recompute. We basically need to persist this graph in some kind of persistent caching. And the most important thing for this kind of ingredient, I don't want to go into detail how persistent caching actually works, because that's probably a different talk, but the most important thing is reliability on this persistent cache. So we want to focus on really correct cache invalidation, because if that is not working and people are actually deleting your cache because it's not working as expected or has non-determinisms or something like that, then we basically don't hit our goal of making every build incremental. We end up, like, if you delete the cache, you end up with a fresh non-incremental build, and we want to avoid that. So reliability is really the key here.

And another ingredient is like we need incremental algorithms for that. So an incremental algorithm is basically something where, if something small changes and the computation is always small, and if something large changes, there can be a larger computation. So basically we want, like, change and computation cost to be somehow linear dependent with each other. And that usually works, but these kind of problems I talked about earlier, like whole application optimizations and effects of import site basically breaks this promise, because then you have something that you have to look at the whole application to make some decisions, or something, a module affects all its importer and they basically are a little bit challenging for this linear dependency between computation and change. Yeah, so, and I basically want to focus on the incremental algorithms aspects because persistent caching is too much to cover.

So for incremental algorithms I came up with three ideas how to do that. So the first idea is basically we, like, on an incremental build we just take the old result, what we computed on the initial build, and make a diff of the imports and then from that somehow write an algorithm to compute a new result out of that. That's the first thing I came up with. And the next idea was, like, doing something with cached sub-tasks, so we have this big computation task that do the computation and then we spawn up smaller sub-tasks and basically we cache these sub-tasks so on incremental builds we don't have to re-execute them, and basically in this kind of sense incremental builds we are faster with that. Or the second approach was just split up everything, avoid this big parent task, but instead split up all the work in many, many small tasks and each of them can be cacheable on its own and basically solve it this way.

And I visualized these kind of ideas to make it more visual because visual is always great in talks. So, like, on the top row you see the initial build where you get inputs, compute something, have an implementation and then compute the result, and an incremental build would basically take the inputs and also take the old inputs and basically have a different implementation for incremental builds and then from that and the old result we compute a new result. But I don't really like that approach because once you have to do two implementations for incremental builds and initial builds, so it's prone to errors if you have mistakes, and it also adds some kind of non-determinism to the system. So maybe your build now depends on what was the previous build or you basically can mess up something in the incremental build, so yeah, non-determinism is not something I like and I talked before about why I don't like it and so this is basically off the table. So the cache subtask approach is better because you basically have a single implementation but then maybe on incremental builds you skip some of the subtasks.

So that would work, but you still have the problem of having this bigger parent task where it might hide some work that isn't done for every time, even on incremental builds. Maybe you loop over all the modules and that stuff. So that's also not what I really like. So I really want to go with the last approach, split up all the work in really small subtasks and then only compute the subtasks that have changed, the cache ones that are unchanged. And it looks like there's something about it. So if we look at our table again and maybe make some examples how we can do that for protection optimizations, I start with the easy ones because they don't have whole application information knowledge needed and don't have effects on import side which are dead poll elimination and minification. And the good thing for these implement optimizations is we basically execute these both optimizations on every output file, and the good thing is we emit a lot of output files so we can just split it up by running each output file on its own. And it basically looks like that, where we just split up the work by just running it on every output file. And on incrementables we only have to re-run it when an output file changes. So this is easy. Problem solved. The next one's a little bit more complicated because they require whole application knowledge. And let me explain how it works. So we have this tree shaking which is basically two steps, like one is this module graph thing where we follow imports to find modules you need in your application. And then the second step, so the first step doesn't actually need whole application knowledge, you can just follow the graph and do that by following the graph.

But the second step was figuring out which exports are used in your application, so we look at all uses of the module in the application and figure out which exports are used. But do we need to do that? We can actually optimize it by coming up with a new algorithm. So why not use a graph approach also for exports? And it turns out we can actually do that by, instead of having a graph with modules as the smallest unit, we can use a graph with module exports as the smallest unit. And then just follow the graph to the exports. And then we can use this previous idea with tree shaking where we can just follow the graph to find modules, we also do that to find exports. And then we only include the exports we actually need.

And if you look at more practical usage, it basically looks like that. We first take a module, split it up into multiple fragments, pair export. But also there are some cases where you have more than exports as fragments, you also have internal fragments where maybe some shared state between exports and that stuff. So basically you split it up into multiple module fragments and then basically the resolving, and if you have an import, you follow the import, not to the module but to the module fragment of that. And then you build up a graph with module fragments and only include that. And that's even better because the good thing is that it turns out to be a better optimization because now you can split modules in different chunks and split it between different pages and one page doesn't leak as a used export to another page and it basically isolates the stuff you want.

But there are a few optimizations like module IDs and chunking where we can't actually find a new algorithm, or at least I didn't do that yet, or we didn't figure out how to do that, where we can't eliminate this problem of whole application information. So we need to solve that problem, too. And it's also not super complicated because we can basically split our compilation into two phases. The first phase where we actually parse modules and transform them and build up a module graph, and then the second phase where we take this module graph, chunk it into multiple files, and do the code generation step and end up with output files.

But there are a few optimizations like module IDs and chunking where we can't actually find a new algorithm, or at least I didn't do that yet, or we didn't figure out how to do that, where we can't eliminate this problem of whole application information. So we need to solve that problem, too. And it's also not super complicated because we can basically split our compilation into two phases.

The first phase where we actually parse modules and transform them and build up a module graph, and then the second phase where we take this module graph, chunk it into multiple files, and do the code generation step and end up with output files. And if we look closely, then we only need whole application knowledge from the first phase, and only consume it in the later phase. And so what we can do is run this first phase for every page in your application or every route in your application, and then take all the module graphs, aggregate this in a global step, and then we basically generate or extract this whole application knowledge and feed that into the second phase and then run the second phase for every page in your application. The per-page stuff is still incremental, and then there's this non-incremental part where we aggregate that stuff and then feed it into the second phase, which is incremental again.

But it's still a little bit problematic because we still have this large computation step that's executed globally, like computing ID lengths and IDs, so we can also think about can we split it up further? And it turns out we can actually do that by doing it more complicated, splitting it up into smaller pieces. So we first, like from the module ID identifiers, we first compute the ID lengths in the global step, and then we compute the ID lengths, and that's actually really challenging. It's the only chance if you add a ton of new modules, remove a lot of modules. So the ID lengths are really changing. So we can feed back the ID lengths back into the incremental per page execution, so per page, and we execute the first trial of the hash, so we have some kind of pre-hashed module IDs, pre-hashes of the module, and then we consume that in this global step of computing module IDs. So we don't have to do a lot of hashing work, we just have to figure out conflicts and rehash for the conflicting module. So that's smaller, and it makes the jobs less compute-intensive, it makes it more incremental, and the rest stands the same.

To do this in an incremental way, we can first from the module graph per page extract all the module IDs, we have a list of all module identifiers per page, then aggregate this list to a big list of all module identifiers, the count defines the ID length we need, and then compute IDs by hashing them. Then we have an assignment, like all the module IDs we need in the application, and we know that pair module. And to not directly... We don't want to directly connect module IDs with code generation, we basically want to pick a single module ID, basically pick all module IDs, a single module ID from that, and then only feed the module IDs that the code generation step actually needs into that step. So maybe it only imports these three modules, and then we only want to pick these three module IDs from our global application knowledge. And this way, this is also incremental in the sense that basically code generation is only recomputed if that module ID it consumes is updated.

But it's still a little bit problematic because we still have this large computation step that's executed globally, like computing ID lengths and IDs, so we can also think about can we split it up further? And it turns out we can actually do that by doing it more complicated, splitting it up into smaller pieces. So we first, like from the module ID identifiers, we first compute the ID lengths in the global step, and then we compute the ID lengths, and that's actually really challenging. It's the only chance if you add a ton of new modules, remove a lot of modules. So the ID lengths are really changing. So we can feed back the ID lengths back into the incremental per page execution, so per page, and we execute the first trial of the hash, so we have some kind of pre-hashed module IDs, pre-hashes of the module, and then we consume that in this global step of computing module IDs. So we don't have to do a lot of hashing work, we just have to figure out conflicts and rehash for the conflicting module. So that's smaller, and it makes the jobs less compute-intensive, it makes it more incremental, and the rest stands the same.

So that's all the metric, and if you look at chunking, it's basically behaving exactly the same. So basically, for chunking, we need to do some graph colouring where we figure out which modules are shared between chunks, so it's basically the same approach, just per page we do the graph colouring, and then have pre-computed module colours, then aggregate it on a global step where we merge colours together to create like a global module colours object with this whole application knowledge, then pick the colour of a module again, and basically this module colour is then feed into the actual chunking work which then consumes only the colours it needs. If you look at that in detail, it's basically exactly the same. So there's no metric behind that. If you look closely, most of the optimisation, you can come up with, if you spend a little bit of time thinking about that, you can come up with an approach that does as much work incremental as possible, and in the long-term, there will be a lot of benefit of having a little bit different algorithms to make it more incremental, and in the end, we could reach incremental production builds, and then everything will be hopefully faster, and that would be great.

That was everything I have to talk about. I hope it was useful. It's often not ...

There's one question I'm sure you're getting harassed about a lot already at the top of this. I'm going to pin it straight away. When will TurboPack be production-ready? You should try it. If you want to try it, try it for def. It's working great. We've been using it for months in vessel.com and it works well, and there are probably a few bugs, but if you don't try it, it will help us if you try it. You just want to find the bugs now? You want to find the bugs. It's really stable for development, but I was talking about production builds. That will take a bit, and we also have the next test suite against running against TurboPack for def that's passing everything, and there's I think 70 per cent passing for production builds, so that will take a bit for production builds, and we also need persistent caching which we haven't worked on yet, but we will be there. You should start using it for development now. What about production projects that are side projects? We can probably risk it. Yes, it's not ...

There's an experimental production build which is kind of hidden behind an ... We did the deployment of that, but it doesn't work for all features. There are some problems. It's not optimised very well. All these optimisations are missing. It's minified, but they are fairly ... Otherwise, we do development chunking which is not that efficient for production builds. Currently, you can use Next Dev for development and Webpack with production builds that works. There is a risk of having differences between both, and that will be addressed once TurboPack is ready for ... So, there are differences for development that we want to speed most of the time anyway? Yes. That's where it really makes the difference. Yes. That's why we started with development, because we wanted to get ... Because people complained about development speed a lot with Webpack, and that's where the address is set, and now we want to address production builds too. Nice.

This next one is very fun. A lot of people might not know this, but we had to have separate speaker rooms because Evan and Tobias couldn't be in the same room because they would fight. No, but the suggestion is, why don't you and Evan combine efforts in simplified ecosystem with TurboVitePack? Or VitePack. TurboVitePack. Sorry. I mean like both of us profiting a lot from other ecosystems. I think there's a lot of ... We learn from each other. I think- How much bad blood is there? That's what we want to know. I don't think it's bad blood. No, it's fine. Yes. So we're not getting a team up any time soon? What? You're not teaming up any time soon to merge all the efforts? We didn't plan anything for that yet, but I think we're still thinking about how we design the plug interface, and maybe we could get up with something that is similar to Vito or Loops plug interface, and then maybe that's more compatible with each other. Fantastic. All right. Let's see what we have here.

This is an interesting one as well that I like. Are you going to share primitives with Vite and OXC? What? What's OXC? Vite and OXC. I don't know what OXC is myself. Neither. So I guess that's a no on the OXC. So we don't share primitives, and you also see Vite and Tobac are completely different things. Vite is more a non-bundler, and Tobac is a bundler, so even from that side it's really different. There might be some overlap in transformations, and also technically we are sharing primitives with SWC, which is a parser we both use. So we're sharing the parser and also sharing some SWC transforms. Yeah. Nice.

Oh, this is a fun one. When can we expect to start writing plug-ins for TurboPack? Not yet. I'm a little bit afraid of starting designing a plug-in interface because I don't want to mess it up with Webpack, so I'm really delaying that until I have to write that.

And yeah, not sure. Not yet. Any problem to solve? It probably starts after production builds is working without plug-ins, and then I start on that. Nice. It's difficult because it's really hard to design a plug-in interface, or APIs or plug-in interfaces, because then you end up with, like, you need to make breaking changes once you see it was a design mistake or something like that. Yeah, so probably delaying it a little bit for that reason. Yeah. So avoiding as many breaking changes for now. Yeah, if you don't have an API or a plug-in interface, you can't make breaking changes. That's great. So for now, it's fine. But we will end up having to do that. So we need a plug-in interface. Unavoidable at some point. It's unavoidable. I guess it will probably end up being some kind of... I really like the Rust Edition thing, where you have, like, this is Edition 2022 or something, and then you have breaking changes when you move the editions. But in the end, you basically have some kind of versioning of the interface, so you can version your project or your module or maybe the package, and then this is working with this plug-in interface, and the next edition is working with the plug-in. You can mix it up in multiple plug-ins, so maybe that is something useful. But you have to look into that in detail. Yeah. I'm sure you have about 100 other problems that you're still solving. Yeah. This one got a lot of likes, so a lot of people are interested in this.

Are static functions of a class tree-shakeable? If not, what would you recommend for grouping functions? So, yeah, this new tree-shaking approach makes more stuff tree-shakeable, because we can split up modules more granular. That's really helping. And in general, it also helps if you flag your package side-effect-free, because if by ECMAScript spec, everything can have side-effects. If you import something, you basically have to include it or execute it somehow, because by spec it might have side-effects, and not always side-effects extractable from JavaScript by looking at the code from a bundler perspective. So if you, as module author, flag your modules as side-effect-free, we can reason more about this doesn't have to be included because you don't use it, and that would make it more efficient and more easily tree-shakeable. And also what helps is that barrel files are really problematic for tree-shaking, not in the sense of it can't remove that, but it always has to compute all the stuff, because if you export stuff from something, we have to look at all these modules to see what exports are included.

So it really helps if you re-export with named exports, so if you don't use export star, but you use export and the names and then from. Named exports are an easy win for optimisation. I was going to ask that selfishly, what are the easy things to get optimisation for our bundles? Side-effects and named exports, that's what I can think of.

That's very interesting. That's where I'm going to be going. Exporting a single object with all the stuff, using these exports, ASM exports helps. All right. I think we might have time for another question. Let's see what we have. We have a long list of things here. Pre-bundling your library makes it harder for us to optimise, because pre-bundling your library, it's better if it stays like multiple modules in your library, and if you pre-bundle it, that is really hard for optimisation because then you only see a single file which is a lot of stuff, and that's harder to optimise usually than having multiple files.

This is an interesting one because I guess this is more like, somewhat like load balancing. If a component is used 90 per cent pages but the other 10 per cent have 80 per cent of the traffic, do you have an option where the traffic of the pages is taken into account, rather? In Webpack, currently, that's a problem because then you end up including the export on the page that has high traffic too, but in Trope-Pack, basically a new tree-sharing approach which I talked about, basically solves that problem. Now, it basically splits it up into these are the export and that export, and it can include it only in the pages you're actually using.

So you can actually pick it at the page level then? Yes, it's basically on the export level, tree-sharing. It doesn't, like, other pages doesn't influence, like, pages doesn't influence the other form like exports used any more. So that's solved. There you go. Easy. Now all you have to do is go and implement it, everyone. That's as easy as that. Fantastic. I think we're going to wrap it up for this line of questioning because you are going to be outside answering more questions for everyone, I'm sure. Another big round of applause, everybody, for Tobias. Thanks a million for coming back. Always good to see you. Thanks for having me.