Building a Node.js Runtime for the Browser

We'll walk through what it takes to build a Node.js runtime for the browser, the challenges we discovered along the way and solutions we built to those challenges.

Rate this content
Bookmark
Watch video on a separate page
Video Summary and Transcription
Building a Node.js runtime for the browser is a complex task that involves creating a virtual environment where Node.js applications can run seamlessly. Nodebox achieves this by using web workers as virtual Node processes, which handle tasks like initializing the file system and loading WebAssembly files. The file system operations in Nodebox ensure eventual consistency across web workers, allowing efficient read and write operations. HTTP server simulation in Nodebox is managed through iframes and service workers, enabling request routing similar to a real server. WebSocket support is provided by a mock WebSocket object, facilitating real-time data communication within the browser. Developers can try Nodebox for their projects at NodeBox.Codesandbox.io, where templates for Node.js, Next.js, Vite, React, Vue, Svelte, and Astro are available. Nodebox was created to enable the Sandpack library at Codesandbox to run small projects directly in the browser for demonstration purposes.

FAQ

Key features of Nodebox include built-in file system support, HTTP and WebSockets support, child process emulation, and the ability to fetch NPM modules, all within a browser environment.

In Nodebox, HTTP servers are simulated using iframes and service workers. When an HTTP server is initiated, it communicates through a preview manager that manages these iframes, which mimic server behavior and handle request routing.

Web workers in Nodebox act as virtual Node processes, handling tasks such as initializing the file system, loading WebAssembly files, and executing server-side JavaScript, all within the browser's sandboxed environment.

Yes, Nodebox supports WebSocket operations by using a mock WebSocket object that overrides global properties to simulate WebSocket behavior in the browser, allowing for real-time data communication in Node.js style applications.

Issues or feature requests for Nodebox can be submitted through the GitHub ticket system for the Codesoundbox repository. This allows for community contributions and support.

Nodebox ensures compatibility by mimicking the Node.js environment, including its global variables and module system, and by setting certain browser globals to undefined or modifying them to fit Node.js expectations.

Nodebox was created to enable the Sandpack library at Codesoundbox to run small projects, such as Next.js examples and other Node.js applications, directly in the browser for documentation and demonstration purposes.

Yes, you can try Nodebox by visiting NodeBox.Codesandbox.io. The platform offers various templates and environments that are compatible with Node.js projects, allowing for experimentation and testing in a browser environment.

Nodebox is a Node.js compatible runtime for the browser developed by Codesoundbox, designed to allow the running of small Node.js projects directly in the browser environment.

Nodebox handles file system operations by maintaining a main file system state that synchronizes changes across web workers, ensuring eventual consistency. This allows for operations like write and read to be managed efficiently within the virtual environment.

1. Building Nodebox: Challenges and Collaboration#

Short description:

Hey, I'm Jasper. I'll talk about how we built Nodebox, a Node.js compatible runtime for the browser. We needed to build a file system, HTTP server, websockets support, modules, and the ability to fetch NPM modules. We had help from existing libraries in the Browserify ecosystem.

Hey, I'm Jasper. I'm a staff engineer at Codesoundbox, and I'll give a talk about how we built our Node.js compatible runtime for the browser called Nodebox.

So why do we actually build a Node.js compatible runtime for the browser? We wanted to build a Node.js compatible runtime because we wanted to allow our playground library, Sandpack, to run small projects. Like for example a small Next.js example, a feed example, or Next.pressjs application. Obviously for the purposes of documentation.

And so what did we need to build to actually make this possible? There's a lot of stuff that goes into Node.js. For example, we had to build a file system, an HTTP server and websockets support, we had to build modules, we had to be able to fetch NPM modules. We also had to make sure that we had child process support, as most libraries and frameworks heavily on that. And there's also a lot of more smaller libraries and other standard libraries in Node.js that we had to support to make all this work.

So we actually had a lot of help from existing libraries, because there's a lot of stuff out there from the Browserify ecosystem. For example, there's assert, and there's like zlib. There's also a buffer and events, or path string decoder, like URL utils. Readable stream by the Node team, which helps us build stream support. And then there's also like crypto, and a bunch of other ones that are not listed on this slide.

2. Overview of Nodebooks and Filesystem#

Short description:

So a general overview of how Nodebooks actually works. We have a main manager that controls everything in our virtual environment. It spawns processes, mimicking actual Node processes. We also have previews, which act like an HTTP server. Our Node processes are web workers with their own contained process. We use web workers to initialize each worker and do the initializing work of building the file system tree, loading web assembly files, and waiting for it to finish. Once the worker is ready, we send it back to main and can run commands like Next.js. We wrap the module in our own global to make it believe it's running in a node environment. Our filesystem works with the main process having the whole state and workers having eventual consistency. When writing to the filesystem, the write is synced and sent to the main process to synchronize it across the entire application state.

So a general overview of how Nodebooks actually works. So we have our main manager, which controls everything in the sort of our virtual environment. And that spawns processes, which are actual Node processes, or try to be Node processes, they mimic it. And then we also have previews, which is sort of like your HTTP server, which goes through our preview manager, which we built, which then starts like iframes to mimic this HTTP server behavior. And our Node processes are actually web workers, which then also have their own contained process similar to how Node works.

And so how does a Node process work in our environment? So as I said before, we use web workers. To do this, we initialize each worker by sending a file system buffer and environment variables and a bunch of other small config options we have in Node box. Well, once this happens, the worker spins up and starts doing its initializing work, which is building the file system tree, loading some web assembly files which we use for example, for transpiling our code or some things like probably compression, which doesn't really exist in the browser at the time. And we also have things like waiting for it to to finish at the end. And then we go into actually loading the rest of the stuff.

So, once the worker is ready, we send it back to main. And once it's in main, main knows our worker's ready. And now, we can do like running a command. For example, running Next.js is as simple as passing in the JSON command next and then it spins up a whole Next.js server. It does this by going into our node modules and resolving the next binary which is actually just a JavaScript file in the end. So, we resolve that which ends up being like .bin slash next and then it has a sim link to slash node module slash next slash CLI.MJS or something. And then we run that actual script as like node, the resolve file and then which we pass as args to our module. And then we use that to evaluate the module. How we do that is by wrapping our module in our own like global which we built which contains most globals, node modules expect like the module, global require there's a couple other ES module stuff and then global this which is all different from the browser and we try to make it believe that it's running in a node environment by this instead of the actual browser. So we also set certain browser globals to undefined or null. And then we run it in a function. We wrap it in a function with our global arguments. So the code believes these are new globals instead of the actual globals that the browser has. And that we also do apply where we override the disks to be our global disks instead of the browser's disks. So it really believes it's inside of a node environment while it's still running inside a browser.

So how does our filesystem work? Our filesystem works in a way that our main process has the whole filesystem state and all our workers have eventual consistency. For example, here we have an example of how it writes to a filesystem. So in the module, you have importfs and then you call fs.write. That write then gets sent to our filesystem state which instantly syncs it inside its own state and then it sends a message to our worker bus to the main process to say I've written some file. Can you synchronize it across our entire state of the application? And then the main filesystem state also receives that, updates its internal state and then emits it to all other workers.

3. Reading from the Filesystem#

Short description:

In Nodebooks, reading from the filesystem is simple and fast. Each module has access to the entire filesystem state and can instantly retrieve files using the fs.read() function.

So in this case there's just one other worker which receives this message and then also updates its own filesystem state. So if there would now be a second module in Worker2 that tries to read that file, it would get the latest state. So how does reading then work? Reading is a lot simpler because you already have the whole state of the filesystem in your process. So it's just your module has the filesystem standard library and then you do fs.read() and then it just instantly looks it up in a map in the filesystem state and resolves it without any passing of messages or anything. It's just instant. It's incredibly fast.

4. HTTP Support and WebSocket Mocking#

Short description:

And then we also have HTTP support. It's more complicated than filesystem. We start a simple HTTP server on port 3000. Our notebox environment sends a message to our worker to start the server. The preview manager opens an iframe with a preview relay. The relay initializes a service worker. The preview manager sends the ready message to Sandpack, which spawns another preview iframe. Requests go through the service worker, relay, preview manager, and worker. Responses are sent back through the same flow. WebSocket support is provided through a mock WebSocket.

And then we also have HTTP support. This is quite a bit more complicated than filesystem. So let's start off with starting a simple HTTP server. For example at port 3000. So we have our module which like you have the standard boilerplate HTTP server code and then you call httpserver.listen() with a port in the end. And once you do that, our notebox environment goes out the module and then it sends a message to our worker to say start HTTP server on port 3000. It receives that, passes it onto the preview manager which manages all previews and servers which could be compared to sockets on an operating system.

And then you and then the preview manager says open an iframe to actually register this port. That iframe contains what we call a preview relay which has a unique domain name with some initialization script that we build at Code Sandbox. That initialization script starts a service worker which then listens for the requests that come in and once that is ready, it says I'm ready to receive requests. And then we go back to the preview manager with that ready message through our relay and then the preview manager sends this to Sandpack which actually spawns another preview iframe which actually previews your application. The relay has this initializing script and it doesn't preview anything. The preview frame actually contains your application.

And once it's done that, it goes back to workers to say it's ready and then you get in your module like a listening event on your HttpServer. And it's just like regular Node.js. How do requests work in this whole setup? You have your preview frame which contains your application. You do an HttpRequest, for example, loading the index.html. It goes through the service worker that intercepts that request and then it says to the relay, I've received a request. The relay then says to the preview manager, there's a request here for you to handle. The preview manager knows where the server is running in our whole worker, web of workers. So it goes through main, then main forwards it to the worker. That worker then emits an request event to your module and then you handle it like regular node and then eventually you'll get a response. That response then goes again through that worker, back through main, back to the preview manager, then the preview manager sends it again to the relay, and there's a unique ID to this, so it's tracked across the whole graph. And then once preview relay has this response, it says to the service worker, I have a response for your request with ID something. And then it sends that response to your preview frame, like you would do with a service worker. Just here it goes through a bunch of workers and relays to a mock server instead of a real server. But then we also have WebSocket support, which does a similar thing. It mocks the actual WebSocket by overriding the global with something we call a mock WebSocket, and we build it entirely. There are codes and mocks where we have overwritten every method of this WebSocket to be compatible with NodeBox instead of using actual WebSockets. This only happens when it's local host or the local IP, then it uses the mock logic, and otherwise it just uses a real WebSocket, because you might still want to connect to a real web server, even in your small playground projects.

5. WebSocket Support and Building a Web Server#

Short description:

Let's see how WebSocket support works in NodeBox. It starts from a preview frame and goes through the service worker, relay, and manager. With all these building blocks, you can build your own web server. Try it out at NodeBox.codesandbox.io. We have templates for Node.js, Next.js, Vite, React, Vue, Spelt, and Astro. Connect with me on Twitter at Jasper D'Amour and find Codesandbox at Codesandbox.

So let's see how that works. So again, we start from, for example, a preview frame that says, I want to send a WebSocket message. And then it goes through the service worker again, like the same way as a request. Actually, the first message in a WebSocket is an upgrade request. And that does exactly the same as before. But now we want to send a message. So we send it through a service worker. Service worker does it back to the relay, and back to the manager, knows where that server is running, because you first did a request to start a WebSocket, and then the module will handle it. And then you respond to it, for example, with another WebSocket.send. Then you go through the worker, through main again, preview manager, preview relay, same as before, and then it emits this again to the service worker, back to the view frame. And now we basically have WebSocket support as well.

And with all that, you have a pretty basic web server running. For example, here you have a simple hello world. Even prints to the console that server's running when listen is done. With all these building blocks, you can actually build something like this yourself. Theoretically, it's identical to how NodeBox works. You can actually try this out at NodeBox.codesandbox.io. We have a ton of templates built on top of Sandpack that you can play around with, like Node.js, Next.js, Vite, React in Vite, React in our Sandpack browser environment. Vue in Vite, but also Vue in our browser Sandbox environment. And then you have, like, Spelt again to environments, and then Astro we also support. And you can play around with anything that's Node compatible and see if it works. If it doesn't, you can always open a ticket on GitHub for us to fix it.

Thank you for listening! You can find me on Twitter at Jasper D'Amour. And you can also find Codesandbox on Twitter at Codesandbox.

Jasper De Moor
Jasper De Moor
13 min
15 Nov, 2023

Comments

Sign in or register to post your comment.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Full Stack Documentation
JSNation 2022JSNation 2022
28 min
Full Stack Documentation
Top Content
The Talk discusses the shift to full-stack frameworks and the challenges of full-stack documentation. It highlights the power of interactive tutorials and the importance of user testing in software development. The Talk also introduces learn.svelte.dev, a platform for learning full-stack tools, and discusses the roadmap for SvelteKit and its documentation.
Gateway to React: The React.dev Story
React Summit US 2023React Summit US 2023
32 min
Gateway to React: The React.dev Story
Watch video: Gateway to React: The React.dev Story
The Talk discusses the journey of improving React and React Native documentation, including the addition of interactive code sandboxes and visual content. The focus was on creating a more accessible and engaging learning experience for developers. The Talk also emphasizes the importance of building a human API through well-designed documentation. It provides tips for building effective documentation sites and highlights the benefits of contributing to open source projects. The potential impact of AI on React development is mentioned, with the recognition that human engineers are still essential.
Opensource Documentation—Tales from React and React Native
React Finland 2021React Finland 2021
27 min
Opensource Documentation—Tales from React and React Native
Documentation is often your community's first point of contact with your project and their daily companion at work. So why is documentation the last thing that gets done, and how can we do it better? This talk shares how important documentation is for React and React Native and how you can invest in or contribute to making your favourite project's docs to build a thriving community
Documenting components with stories
React Finland 2021React Finland 2021
18 min
Documenting components with stories
Most documentation systems focus on text content of one form or another: WYSIWYG editors, markdown, code comments, and so forth. Storybook, the industry-standard component workshop, takes a very different approach, focusing instead on component examples, or stories.
In this demo, I will introduce an open format called Component Story Format (CSF).
I will show how CSF can be used used to create interactive docs in Storybook, including auto-generated DocsPage and freeform MDX documentation. Storybook Docs is a convenient way to build a living production design system.
I will then show how CSF stories can be used create novel forms of documentation, such as multiplayer collaborative docs, interactive design prototypes, and even behavioral documentation via tests.
Finally, I will present the current status and outline a roadmap of improvements that are on their way in the coming months.
TypeScript for Library Authors: Harnessing the Power of TypeScript for DX
TypeScript Congress 2022TypeScript Congress 2022
25 min
TypeScript for Library Authors: Harnessing the Power of TypeScript for DX
TypeScript for library authors offers benefits for both internal and external use, improving code quality and providing accurate understanding of libraries. Documentation and examples should be in code to provide up-to-date information. Testing types alongside unit tests ensures accurate typing. Managing changes and exposing types requires careful versioning. Deep integration of types improves usability. Using a map in TypeScript allows for simpler implementation and customization. Leveraging types in libraries can generate code based on user access. TypeScript integration with Nuxt provides support and type declarations.
The Legendary Fountain of Truth: Componentize Your Documentation!
React Advanced 2021React Advanced 2021
24 min
The Legendary Fountain of Truth: Componentize Your Documentation!
Welcome to this session about documentation in a command-driven era. The Data Axis framework provides a comprehensive approach to documentation, covering different areas of the development process. Component-driven development and MDX syntax enable faster development, simpler maintenance, and better reusability. Embedding components in Markdown using MDX allows for more advanced and useful documentation creation. Tools like Storybook and Duxy with MDX support are recommended for documentation solutions. Embedding documentation directly within components and migrating to MDX offer a comprehensive documentation experience and open up new possibilities for embedding and improving documentation.