Thanks for having me. I'm trying something new. I'm trying to do live coding today, so I hope it works out. So, yeah, my name is Tobias Koppers and I worked on Webpack for 10 years and now, joint Universal and work on TurboPack trying to do something new, something better and yeah, I'm trying to focus, as I've said, I'm trying to focus on one aspect of TurboPack and trying to explain a little bit how TurboPack or the core of TurboEngine works in detail so I'm trying to demo something with that, so I'm trying to actually trying to live code in JavaScript a little bit of the core of TurboEngine.
So the motivation of that is that on Webpack applications we saw that applications grow and grow, larger and larger and Webpack, the architecture of Webpack is not built for that, incremental builds tend to get slower and slower if the application grows. It's not that huge of a problem but yeah, it might get a problem in a few years when the applications get millions of modules or whatever and a few problems we isolated were we do a lot of cache look-ups, we have to compute a lot of stuff to do cache validation like checking if files are still the same, hashing stuff and that is the problem because all this overhead, you pay it for every incremental build and we want to do something new, a new architecture to tackle this problem. And that's why we created like turbo engine and turbo pack and it's a new architecture and I can explain it a little bit in doing live coding.
What I want to show is a kind of small application which is super simple, not a bundler but something that is similar to a bundler, it is taking any JavaScript application and just copies over the application by following the dependency graph to another folder and doing that it also adds a copyright header just to demo something. With that, I start with the basic application written in JavaScript and explain it later and then I try to add something similar to TurboEngine to make it more efficient, to make incremental builds possible in a similar way which it works in TurboPack, in Rust and with TurboEngine. For that I prepared this little application, it's really simple, it's just a bunch of Node.js We use Acron to get the dependency graph of something, the path there, the modules.
And I go through the application a little bit to make you understand it. The main process is really simply, we get the base directory, like the source directory, we have an output directory, and then we have an entry file which is actually this file we're looking at. So we're actually copying its own application to another folder. And then we start following the dependency graph from that entry point and copy that from base tier to output tier. And another, to make it a little bit more complicated, I add this header file which basically is, let me show it, it's like a copyright statement which should be added to every file to make it a little bit more interesting. So then we invoke this kind of function, copy graph, which basically computes the output from the current file by just relocating it. Calling the copy function which copies the file, super simple. And then calls two other functions which is called get references which we see later, it's like getting the references, like all the files that have been imported from one file and then looping over that and calling itself recursively to just copy the whole application. Yeah. So copy also pretty simple, read the header, read the file, and write it to another file. Nothing super complicated here. Get references is a little bit more complicated but yeah, it's not really that you have to understand it. It's like calling parse to get an AST out of the module and like looping or doing some magic to extract the import statements and returning a list of all files referenced by that kind of file. Parse is also pretty simple using calling Akon, which is a JavaScript parsing library. Also reading the file obviously and then it returns the AST. And after that I start the whole thing and that should copy the application to the new folder. So let's try it. Oh. A few things I want to explain. I also have this task function which is actually doing nothing currently.
Comments