Owning your Build-step – Owning your Code

Hello and thank you for tuning in to my session about how to improve your generated JavaScript code using Rollup. Before I want to go into that part, actually, I want to tackle another question.

The question is like the basic question of JavaScript. Why are we using all these complex build pipelines and putting all this stuff around our JavaScript code? To give you two reasons, I'm going to show you some examples first.

The first example is something we built at our company. It is open-source, a small Scrum Poker implementation. You might want to give it a try if you want to. The important part of this, why I chose it, is because with this application, it is possible to run it during development without bundling. I did some measurements like cranking my browser to a really slow network setting for a small mobile phone. The times I got is without bundling it takes around 18.3 seconds to load, but with bundling, it takes seven and a half seconds. Now, I want to say the only change between those numbers was the bundling. There's not a lot less code on the right side. There's no compression, no minification. Why is that, actually? Why is there this difference in numbers? The difference you can see, actually, in these two pictures. It's these green staircase steps here, or it's usually called a waterfall here. What's happening is, it's starting, these are all JavaScript files, starting with the first file, and then it sees that there's some imports in the file. Once it's discovered those imports, it sends the network request, okay, give me those more files. It needs to load those files, pass those files, and discover some more imports, and so on. Between all those steps here, there's, basically, network request back and forth, and this is adding up. Of course, what bundling is doing, it avoids the waterfall. Now the question is, okay, this is a web application, I know what we're doing here. What about our situations? Let's say we are looking at a server. This time, I have a a very basic Apollo server set up here, just taken from the website. This time, I'm not going to give you some made up numbers. This time, we're going to do it live. I'm going to build a server in Docker. To give you a reference point here, I have a small terminal here hooked up that is running commands on my machine. We're going to build a Node 14 alpine image. This is 116 megabytes. Maybe you want to remember this number.

We have two Docker files prepared here. The first one I'm going to run is a very traditional setup. What we're doing is we're copying the package files. We are installing the production dependencies. We are copying the server file. This is this file. There's a small wrapper here. This wrapper is just there because it starts a small timer. The timer is stopped once the server says it's fully functional. That's just the measurements.

I'm just running a command that will also immediately start the server to get some time. What I'm doing here is I'm also copying everything over to another image, just the folder we just created. The reason is that during this year, a lot of caches are created, and this saves another two to three megabytes. I'm doing this now. This will just take a moment because all of it is cached right now. The first you see, you remember, it was 160 megabytes before, now it's 134 megabytes, so an 18 megabyte server basically. Startup time was nearly 300 milliseconds.

Now I've got a second Dockerfile here, which is this one. So what this one is doing is it's basically running rollup here, taking the server as an entry point and being naughty, just overriding it again, and there are three plugins, Node.resolve, CommonJS and JSON, which are necessary for node compatibility and we are creating CommonJS file, and that's all there is. And then we are basically copying just the created artifact over, and when I'm doing this, let's see what the numbers are now. You see, it's 120 megabytes, so the 18 megabyte server just became a four megabyte server.

So why is that? That's because there's really a lot of unnecessary stuff in your node modules. This is TypeScript types, this is test files, documentation, who knows what unneeded utilities. So this is maybe not as relevant if you say, okay, 130 versus 120, but this was a really basic setup. So this keeps adding up, the bigger your server becomes, and of course, startup time was 171 milliseconds, so it's nearly half the startup time. And this is, again, the same reason that you saw before. You are reducing the waterfall time. So we are seeing this reduces the size and the startup time, so servers are maybe not that important, but Cloud Functions definitely are. Those Cloud Functions really need quick startup time or also command line tools. So another question, why would you want to use rollup for this? So there are very good alternative choices.

I'm not going to say they are bad because they aren't. What is special about rollup is that rollup is designed from the ground up to be agnostic to the target environment. So you can customize it in any way you like. You can build for the browser, for note, for demo, whatever you want.

This also means that you usually need to add some plugins to customize it. So by default, it works quite well for the browser, but you definitely need some plugins for note. You also get a nice choice of output formats like common jazz ES modules versus various other forms like AMD and ES modules is actually special because that's rollup's native format, which means that if you bundle to ES modules and take the output and bundle it again, it just doesn't change anymore. It doesn't get any bigger, which means this is why rollup is actually used for libraries a lot. Because here, you don't want to have the runtime dependencies all the time. And also, we have very good dead code elimination, even though it wasn't important the previous examples.

So, I wanted to say we want to actually do stuff with our code. So we know why we want to bundle, the question is, of course, the how. And what I want to pattern I keep seeing here with rollup is that sometimes, even though they're very good plugins, you basically want to take things into your own hands. And you can definitely can because it's really easy.

So if you don't know what to do during the talk, you could actually join in right now because this is hosted live here. So you can go to lucastagger.github.io slash DevOps minus JS, if you want to start the current slide is hash seven. And if you aren't fast enough to type this, this URL will be on the top of the next two slides. So let's get started here. So the first example, so here's the I was talking about the first example I want to show you is a very common situation. You have a main file here, which is importing information from a file build JS. And in this case, it's just importing a string, which is telling us okay, this is a development built. But there could be more information here. Let's say there could be a different server doing productions or this could be localhost during development, but I don't know what server during production or alternative information. And what you want to do is basically you want to have the unchanged setup during development, but during production, we want to change this information. So let's actually write a plugin. So how do you do that? So, and as you can see, my setup is actually running rollup live while I'm typing, the output is here on the bottom right. So plugins are an array of objects. And so a plugin exposes several hooks that rollup can hook into. And the hook we are going to use for the first example is the load hook. The load hook receives an ID, which is an identifier of the file, and this is usually the path of the file on disk.

So what we want to do is, if the ID is slash build.js, then we want to return something else. We want to return, so it needs to be, export something, export const type equals, and maybe use a new line that you can keep reading what I'm writing. Yeah. Production. And here we are, we just wrote our first plugin. So now in the output, it's const type equals production. And that's how simple it is. By the way, if you're following this live, again, the URLs here on top, you can just click the button here on the lower left to show you where we want to go to.

Okay. So this was loading stuff. Let's go to another example. So this time we're going to invent some syntax we want to use. So let's say we want to have special logging checks and so on during development that you want to remove during production. So the syntax we are making up here is that any time we precede a line with this comment, remove, then we want to remove everything up to the end of the line.

The hook we are using this time is the transform hook. The transform hook actually has two arguments. The first argument is the code of the module. The second would be the ID, but our current transformation doesn't depend on the individual module. How do you do this? Easiest would be actually to use a regular expression. We're just going to use code, replace, and replace can have a regular expression as its first argument, which is going to be a global expression. We want to replace all occurrences with an empty string. What do we need to put in here? It's slash star remove star slash. You see, I already removed my comment, but that's not what I wanted. We want to remove until the end of the line, so we need several characters which are not a line break. This is that, until the line break. And here we go. We just removed the second line. I could copy this to the first line. It would also be removed. And also note that this is not just some cosmetic removal of code. This is actually fully integrated into Rollup's analysis. So if I say I have some constant foo that I'm also including here in the logging and now I'm adding the comment here, the foo will also be recognized as unused and tree-shaking will just get rid of this in our build.

So we've seen two hooks so far, load and transform. To give you an idea how this goes into the wider picture, let's have a high level view on the lifecycle of a module. So the situation we have here, we have a module slash main.js and it just contains for now an import statement. So what does Rollup do with this code? The first thing is it will take this import statement and pass it to the resolve ID hook. You didn't see this one so far. We are going to use it in the next example.

And the resolve ID hook has two arguments. The first is the import source exactly as it is written here. And the second is the ID of the importing module. Now, this hook is then called on each plugin that implements it until one of the plugins answers it. So in this case, let's say the second plugin says, oh, I know what it means. If someone from slash major imports dot slash foo, then they want slash foo.js. Which is not surprising. And actually something that core would also have done because there's a default algorithm that will just do that. Take the directory of the importer and append the relative source here.

Okay. So then we need to get the code of the module and this is done by the load hook. So the ID we just discovered, we are passing now on to the load hook. The load hook will again be checking each of the plug-ins. And the first plugin that implements it and also returns something will be the one to give us the code. And since in this example, no plugin implements it, we have a default implementation, which is assumed this is the path to a file and try to load it from the file system. So we are getting this code here now.

And then there's the transfer hook. Now that we basically have the first glimpse of the code. Each plugin has the chance to do some transformations on the code. This is intended for plugins like Babel, like transforming code, but any other code transformation is possible. Again, we have two parameters, the code and the ID, and it goes through all the plugins. The first might be replacing the argument here. The second might be injecting an import. And now that we have another import, it basically starts back from the top, and the cycle continues. This is not all plugin hooks that we have enrolled. There's actually quite a few more, but these will be the ones you probably want to use the most.

So now let's do another example. So this time we are revisiting our first example. We want to replace some build time information with production information. But this time we want to have two files which already exist on disk. So we could again use the load hook and just read this file in the load hook, but maybe it would be more would be nicer to actually change the file resolution. So each time this file is imported, we want to actually import this file. So it should be something like if ID equals equals equals log minus dev JS, then return slash log minus prod dot JS. Okay, this of course doesn't work because we don't have the ID. We only have the source and importer. So what we would like to know is, roll up, how would you resolve this source and importer to an ID? And for that we have context functions. So each plugin hook is called with a special this context containing several functions. This time we want to use this resolve. And since this is an asynchronous function, we want to evade it and it will return an object with an ID property.

Okay. Now I'm not going to close this one because as soon as I write the closing parent piece, this will be an infinite loop because calling this resolve will actually call all plugins unless I'm passing a special argument, which is skip self two. And yes. And here we go. So the development build was this, the production build is now log a log and it's now replacing this file with the other one.

Okay. Another example. So taking this to the next level means we can actually completely without the file system. Up to now, the IDs were like parts of files, but you're not limited to this. So for the last version of this, we are going to replace information. Just, we're going to use a completely virtual file. We call it build without anything around this. So now using what we just learned, we can actually, we need to do two things. So it's already warning here that build cannot be resolved. So first of all, we need to resolve it and resolving is just telling Rollup, okay, it's totally fine. This is an ID. So what we're doing is just if source equals equals build, then just return the source again. Okay. And the error changed. Now this time it's loading this, and because we are using the browser built here on the presentation, it's complaining about the missing file system. So we're doing the same here. If the ID equals equals build, then just return export. Oh, okay. New line. And equals prod. And here we are. Our first completely virtual module.

And as a very last example of what you can do to basically take full control of your code, patch problems, and do any kind of transformation, like this time we want to do a self-referencing bundle. So we want a bundle that knows what files are in the bundle in the JavaScript code. And we're using a trick here.

So we're importing something from a file, from a file, slash build.js, and we're actually not implementing this file right now, but rather we are using this to indicate it's external. And you see already, it is just capped as an import here in the resulting files. Also, you see that there are actually two files generated. The reason is the dynamic import statement here, which Rolab interprets as you want to have some lazy loading here, so there will be a different chunk created just for this one.

And, okay, this is what we're doing here is we're emitting this file. So there's one context helper, emit file, which allows you to add files to the bundle. And an edit file has a type. The type is acid. You could also use chunk, but that has some limitations on how it can be used, so acid is usually what you want for arbitrary files. We need a file name. The file name should be build.js, and we need some source. And now we actually created an empty as in because time is running short, we're just using the shortcut here.

So what this one's doing is basically taking the bundle object, you get to the generate bundle hook, and it's just listing the names and the size of the file. So right now main.js it's 90 bytes and this one is 30 bytes, and if I were to change stuff here, you can actually see the number here changing life. So now it's 36 bytes and wrapping up the talk now. Really there's so much more you could do. So I recommend have a look at the website of Rollup telling you how to build your own plugins. And there's actually, if you don't want to go that route, there's also some high level tooling built around Rollup. So just to mention a few that's stencil for web components, VEET, which comes from the VUE ecosystem and it's a very nice development tool without bundling during development and with a prebuilt Rollup step. And WMR is very similar from the Preact community. And for libraries I can recommend to have a look at MicroBundle, which is also more from the Preact universe for zero configuration libraries or TSDex for TypeScript libraries. And with that, I want to conclude my talk. Thank you for staying with me. Just remember you can access the website locally on your browser if you do this, if you want to play with examples and thank you very much.

Hi, yeah. Nice to be here. Excellent to have you. So let's take a look at the results to your question. What is your experience with Rollup plugins? And while... Maybe I overdid it with a number of options."

Okay, so if I could have dropped the first questions I would have expected the majority not having used Rollup because it's kind of a niche tool and I know that. The interesting part is further down, actually I'm most surprised about the 11% because you can use Rollup without plugins, but then it's kind of limiting what you can do. I have great hopes for the 18% and I was doing the talk mostly for the 18%, never wrote a plugin but would consider trying it out because there is like kind of a meta layer for Rollup plugins that is emerging. So there are new tools like VEET and WMR which are actually adopting Rollup's plugin system right now, so that you can basically use your plugin knowledge in other tools. Lots of love to the lower options with even though it's only 2% each. I have to say that I chose the Chuck Norris option myself and I feel there's many other Chuck Nori at this conference so. I know you are.

I know you are. Okay, let's get to the most important question that we've received so far from the crowd, which is, what an amazing terminal for the presentation. What is it? I know the answer but I'll allow you to tell me. The problem is I started a little early preparing for the presentation and when you have too much time, you start doing things. So, I started writing a small web server and put it all into a reveal.js presentation. So, basically, the terminal is all self-made, but it's also open source. I think so, on the last slide of the talk there is actually the url for the GitHub repo, if I'm not mistaken, where you can actually look how it's done, it's actually surprisingly little. So, it fits in into one screen of code, the entire terminal on the back end side. So, you're what they call an overachiever. Yeah. Very cool. I hope not too many people from my company are here watching this and we're discovering what I did with my time.