Bun, Deno, Node.js? Recreating a JavaScript Runtime From Scratch

Hello, everyone. How's it been? Yay! Let's try again. How's it been? Whoo! Energy, yes! We're gonna make some crazy experiments today. This is something I love doing, like learning new stuff, and I love doing it by trying it, right?

So I tried recreating Node.js from scratch and I'm going to tell you all the step-by-step in how I've been doing this. First of all, all the slides, all the links, references and even a tutorial for you to follow after this conference I put on the last slides. So not just listen to me here, try at home because you can get a lot of cool other insights. And for sure, take as many pictures as you can because this helps us a lot. Mention me, mention the event because this is how you can push our job further, right?

Well, all this talk is about, I start myself asking myself, do I really know Node.js? So I was looking, I became part of the Node.js car team there and I was so afraid of being in some conference and people asking me questions of how Node.js was working behind the scenes. And I figure, well, I don't know as much as I wish or as much as I thought I would. So I start researching a lot of things and I start making questions and chat dpt has helped a lot to answer those questions. But I've been creating a bunch of videos doing the same ideas. So if you like more math science projects, I've re-implemented the web socket protocol from scratch using binary code, a bunch of cool stuff, and other, creating your code coverage. But this talk, I already have a tutorial there and here today I'm going to show you some highlights, and I even wrote a blog post for you later to extend, because it's just 20 minutes here so you can follow along.

Well, just before we go on, I just want to let you know that I'm not a C++ developer, okay? Some C++ developers on the back are going to see bad practices here, but I'm just a curious person trying to learn something on the hard way, okay? Well, this is part of my own research, so this content you aren't going to see anything on the internet. So it was really hard for me, I had to schedule some calls with people from Google, and if they said, I've never done this in my life, so try it yourself. It was crazy. And just a shout-out for Ben, Dino, Node.js, and all the creators from this runtime. You are going to figure out today how nice and how crazy is their job. Because they are abstracting all the complexity for us so we can use an accessible language to create powerful applications. And let's do some magic, I love this game!

The first thing I did was try to find GitHub projects that were doing it, but they were only doing V8, only one part, and they were never using all the components. So I tried figuring out how could I create all the components using the same thing.

So I've been to the Node.js website, there you can see how Event Loops works, how Node.js works, but for me it wasn't enough, it was a bunch of theory, but I wanted to experiment. I wanted to have my hands dirty and start working on it. Well, the best part, right? We have everything on GitHub. So if you go there, you're going to see the first Node.js version. Has anyone seen this image before, this web page? This is crazy, isn't it? And see, Node.js wasn't using console.log, it was using puts like Ruby on Rails and just a lot of explanation, right? Well, I went to this repo, on the Node.js official repo, but on the V001 and I started browsing all the files and started realizing how Ryan Dahl thought and getting some assumptions, right? Trying to find out some ideas of some years ago. And something you're going to notice, sometimes we see people as our heroes, right? But if you see the commit story, oh, first compile, nothing works. Three dots, what else? Web page, very incomplete. Trying to implement something, not working. I'm not sure about you, but I don't usually commit everything, but see, Node.js was just a proof of concept. It wasn't something that he-he realized was so big.

Well, I'm going to start talking about the main components and I'm going to try to be brief this because I know there are a bunch of theory on the internet, but we're going to see how this works in practice. So we use JS. JS is the main language, right? But in the end, you're going to realize it's just a string. We use V8 as the engine, right? To interpret JavaScript and we have C++ as a bridge, like how we can link the responses from JavaScript or the operating system and then answer those questions. And we have libuv to connect events, to create a sync functions and so on. Oh my God, I'm seeing a lot of 404 faces, but let's see in practice. Well, let's imagine you want to implement two functions, the print as the console log is working nowadays in Node.js and setTimeout. So when you build V8, V8 is the grammar. It's when we execute const is when we have function data types, map and so on. Libuv, I just want you to focus on the event loop. The event loop here is like a while true, asking for events. If it's that any paint event, it's going to dispatch them all. But the magic, it's on the C++ bridge. C++ bridge here is connecting all the dots and making the magic. If you're going to, if you look at the V8 here, you're going to see that there's no process, there's no setTimeout here in this image, and so much more. Okay, let's imagine we have a JS file and it has just print, right? If we try executing this, print doesn't exist in JavaScript. We're going to have some undefined. It's not a function.

It's not a function. And when you compile V8 from scratch, you're going to see the global list of JavaScript is empty, so we don't have that bunch of objects that we usually have. Well, let's try implementing this print. First, we're going to implement on the C++ side, okay, using some annotations of V8, and we're going to extend V8. This is the whole point of V8. We can extend it and say, whatever I see this string or this call, I'm going to call this C++ side, okay? And there, we have just a print f to print ever, a function in the end.

Well, doing something harder, setTimeout. On the setTimeout side, it's exactly the same thing. I have to create the C++ function now, and then I can use uvtimerstart, just to reference that we are using libuv to create and to call those functions. But C is exactly the same thing. I'm injecting a function on the global state, now the global disk has the setTimeout, and the V8 now thinks that setTimeout was always there. It's pretty great how Node.js is working. In the end, my reaction was like this, Mom, come here! You know that Node.js is just extending functions, there is no magic behind this. I wouldn't say just, because there are a bunch of things in the end happening, but if you want to create your runtime, you've got to focus on creating your own implementations.

All right, let's see some demos. Here, I'm going to show you the execution pipeline and how Node.js is working behind the scenes. So, first of all, I just have a JavaScript file here, and in just the JavaScript file, I'm calling a program, a C++ program, but don't think about C++ here, we're just seeing how the steps by step is doing.

Well, just to get started, we are reading a file. Remember I told you, JavaScript is a lie, because we are reading a string and saying it to V8, and V8 is doing all the magic behind the scenes. There's no such a thing as JavaScript. So, we are reading the file, getting all the context. Let's see. And then we compile. This compilation is just transforming all the JavaScript code into C++ instances, so we can call it right away. It's pretty nice. And then we can use run, which is like evolve, just execute it, just run this code. And we're going to have our results, and wait for events. Remember the event loop, remember the while true. The while true is going to be executed after we run our code, right? Because it's going to be executed in the future.

And just so you, just a matter of curiosity, here I'm reading file just as a C++ side, just copy paste, the easiest way to read a file, not using the LibuV in this case, but we're going to see that it's not that magic, right? It's not that hard. It's hard compiling and managing all the environment, but it's pretty cool. Here I had to gather all the libraries, so I've spent at least a month, my god, my job was all delayed trying to do this thing, I was crying at home, it was really hard. But you have to download V8 from the source, you have to compile, generate a binary, and here we're going to copy all the inputs, just C++ header, just references of how we're going to call this binary. We can think of this binary just as a web API, right? For libuvi, the same thing, I compile everything, and when I start C, I'm not a C++ developer, right? I'm a Node.js developer, and I know NodeMod. So NodeMod, we can watch for files and just execute another instruction. So I just implemented my own live reload here. But let's see these commands just working without any script. So I'm using make. So make is going to read that make file, gather all the binaries, and then it's going to generate a binary for us. And remember, Node.js is just a binary, right? Node.js, when you type node and the name of the file, Node.js is just reading the file, exactly the same way we are doing here. So I'm executing and I can see the hello world in the end. Pretty straightforward for now. But what if we can call console.log? When we are working with JavaScript, anything that relies on the environment, when you have to call something from the operating system, where you depend on the environment, like a console.log, or printing stuff, they are not in JavaScript. See this example how nice is it? If I run console.log, I don't see any error, which for me is weird, because console.log should be undefined, right? But in this case it's not undefined, it's just not printing anything.

But if I try printing the console, I'm going to see there's an object there. Pretty weird. I was like, why? If I use object keys, I'm going to see all the console API, all the methods, they are there, but they are just not useful. I think it's just an interface for us to implement ourselves. I don't know.

Well, what if we try using setTimeout. SetTimeout. I don't know about you, but my mind was just blown. SetTimeout, I was using this since day one working with JavaScript. What if we use setInterval? Exactly the same problem. SetInterval doesn't exist. So my mind was just blown. My mind was like okay, so Node.js is injecting a bunch of stuff in V8. V8 is just interpreting and linking all the ideas all together. But JavaScript is not a mess that I'm showing. If you want to have your own browser on your runtimes, there are a bunch of other working groups that can advise you how you could implement your own functions. But they are advising and you see there's no such a thing of how you could implement it. On ECMAScript, we have all the grammar and all the things how JavaScript should be executed. And if you try executing our own code here, we're going to see the print. The print function, just take a look at the printf(). For me, it was, I would say, the magnificent thing for me because I remember going to the university and the first thing I did there was a printf() and now we are using printf() since nowadays. But just to give some context, let's try using this print and inject on the V8 context so we can see it working in practice, right? So I have the print function here and you're gonna see I'm using exactly the same idea that I showed you on the diagram. I just extend the V8 and say whatever you have this string call this C++ function. And then, let's see the other, and then if I go to my code I can see the print is working just a printf, right? I can use date, date is an exception for the environment, right, because we depend on the dates. We can use the strings and we can use all the complex objects such as map, such as spread operations, so operators, such as set and everything that's in JS. Notice here promises are not async objects. Oh my God. Promises are part of JavaScript, but they are there just to be wrappers for callbacks. In the end we're going to see this in practice, but this is something pretty, pretty great for me. Well, let's try implementing now this setTimeout.

My idea was, okay, I need to link both elements, so we need to use libuv now. So how could we do that? We're going to create a function here, okay? And just take a look here. I'm getting all the arguments, just this lift in the terminal. So I'm using this same approach for both setTimeout and setInterval. I have to check if it's a callback function. And then this was the hardest part for me. Remember, when I'm using a setTimeout, this function is going to be executed in the future. So I should keep the variables, the context, in somewhere, right? So here I'm using a getter structure in C++ just to save this data so I will be able to execute this in future. So I'm using here uw timer just to show that's a libuv call. And then when the timeout ends, we receive this callback with our stored variable there, and then I'm going to call the callback function with the result. See? There's no JavaScript! It's just C++. In the end, we are calling JavaScript back, but JavaScript is nothing here. JavaScript is just an abstraction for C++, Rust, or some low-level stuff. For me, it was pretty, pretty amazing, because we can think about embeddables and a lot of other cool concepts as well.

Well, the waitForEvents, it's the event loop, the whileTrue. And see, how straightforward is this? I don't need to use any logic here. I'm just using the default event loop, which is a global variable, and executing it. And then we can follow the same idea. I'm gonna compile this code. I'm gonna execute it, keep it working, and just waitForEvent as we've done. And then, same idea we were doing in the print example, like here. We are just extending the V8. And when we try executing it, see, I'm using the timeout. The timeout is now available on the JavaScript side. I'm creating two abstractions, one for setInterval and one for setTimeout, but see how amazing is this. I'm using promises to wrap this so I can use async await. So I don't need to use callbacks, I don't need to implement anything because async await are part of the ECMAScript specification. Everything that is on JavaScript specification, you can see working right there. So if we try executing this, we're gonna see it working.

My friends, I cannot tell you how was my feeling when I saw those logs working for the first time. I was calling everyone like, oh my God, it finally worked because it's really hard. All these examples here, you're gonna notice I'm using Gitpod right on the bottom. So all those examples, you won't need to compile it by yourself on your own. You just fork my project there and you're gonna see all the step by step. You can focus only on the part of the code, so I save a lot of problem for you.

And if you notice now, Node.js there, Node.js has a bunch of methods. Actually, in the V011, right, they are exposing the timers. So you, when you go to the Node.js repo now, you can see that Node.js has a bunch of C++ side and a bunch of JavaScript side, right? So what they do, they expose some APIs that are on the C++ side and you can handle from JavaScript because this is easy to maintain, you don't have to handle a lot of data types, and any JavaScript developer could contribute to it. This is why we have a so large database on Node.js as well. Well, everything is following ECMAScript specs.

So you can notice setTimeOut, setInterval are not there. But my other question was, why we have so many JavaScript runtimes appearing now, right? Isn't that crazy? Because JavaScript has been here since 2009. But if you look at Dino, Dino is using TypeScript and a lot of Rust and a lot of other texts. But if you look at the source code there, it's doing the same thing. It's using JavaScript to compile to Rust instances and execute in all the data. Okay, what about Bun? Exactly the same thing. They are using JavaScript core and doing all the ideas right there. Exactly the same idea. Well, they are exactly the same idea, but you might be wondering, okay, but what makes one better than another? Okay? Well, the way they are handling the data, this is all file system APIs from Node.js, and this is, for me, where one can get faster than another, when they are sending data to the operating system. They have to handle buffers, they have to respond JavaScript, and they have to make it faster, right? So this is how Ben says that it's faster than Node.js because he's using another programming language and other cool experiences, too. And in my opinion, I don't know about you, but developer experience matters. If I have to open ten different files to create just an API, it's gonna be hard for us. Dino had, since the beginning, TestRunner, had TypeScript, which was helping a lot developers. But you're gonna see that there is no such a thing as competition. There, you're gonna see that Jarrod Summer from Ben, he just got in the Node.js core. Colin Areig, it has been working at Node.js since the beginning. He has been working on Dino, so they are all collaborating together. Well, go to the tutorial, try yourself. You're gonna have your mind blown because I know it's a lot to consume, it's a lot to observe.

If you want to see more content, you want to see more of my talks, go to my website. You're gonna see a bunch of cool stuff to learn also from there. And, for sure, I built an ebook for you, teaching by step, and I also put a good challenge for you to implement your own FS red file.

All right. So before I finish this talk, let's take our traditional selfie so we can make people on the other side envy, okay? All right. I'm gonna trust you, right? So I'm gonna count on three, and we do wow, okay? So one, two, three. Hey! Thank you so much for having me.

So first question. With all of these JavaScript runtimes coming up, why would you continue to bet on Node.js? Well, I would say the community has been great since the beginning. But it's cool to see how Node.js has been evolving after those JS runtimes. So now we have a performance working group working to increase the performance and improve the performance. So you're gonna see from Node.js15 to Node.js20 it's crazy how fast it got. So Node.js has a lot of special base to evolve. I wouldn't say it's gonna disappear soon. But for me the best thing is it's still fast in comparison to the runtimes that just appeared and Node.js has been there since 2009. So it's crazy.

Okay. And then a softball question. Where did you get your t-shirt? It's from another conference in Brazil. Don't deploy on a Friday, please. And then, all right, let's see. So we have another question about, what makes Bun faster than Node if it is C++ under the hood? Nice. Now, there are some contradictions, because people are saying that Bun is really faster than Node.js, but it was before our performance in working group. So they said it's not so fast right now, in comparison to one to another. But the programming language I would say they chose is different. It's Zig. Has anyone worked with Zig before? I look at the source code and I couldn't understand a thing. So it was, Jesus Christ, it's crazy. The way they are handling the JavaScript, they said JavaScript Core is faster than V8. I don't know.

I asked some people from V8. They were like, this doesn't make more sense. But I would say the set of technologies and the data structures he's using to consume data from the operating system back and forth. Ooh, okay. So these next questions kind of go together. How did you find the time to explore the depths of Node and write a book about it? And also, why do you think it's important to build things from scratch, go that deep? Some friends would say I'd like to suffer. But I've been working as a content producer, right? So I do YouTube videos. I sell premium courses. So doing this, I felt it was a good opportunity for me to learn more about Node.js, to do something for the first time that I've never seen on the internet. And I search a lot and there's no content doing all the way working with Node.js. And I didn't find time for this. So I spent a month and everything just got delayed. My team just got crazy on me. But in the end I said, this is a better good, right? I could do something that I loved. And at the same time I was able to speak at conferences because it's a very catchy title, right? So for me it's worth it in the end. And just a spoiler for you. Probably I'm gonna try doing now the React Native clone. So that's iOS, Android, and JavaScript for the middle. Yeah, this is gonna be crazy. No, that definitely makes sense.