Why Node.js Needs an Application Server

Hi, everyone, I'm Matteo Collina, and today I'm going to walk you through our reasoning on why Node.js needs an application server. Before we start, let's talk a little bit about me. I am co-founder and CTO at a startup called Platformatic. I am also the chair of the Node.js Technical Steering Committee, so responsible for a lot of things regarding Node.js. I am also a member of the OpenJS Foundation Board and the GraphQL Foundation. I created a lot of modules in the past. In fact, with 42 billion downloads last year. Yeah, quite a lot.

Now, we need to talk a little bit about Node.js today. Node.js is everywhere. And last year, it got around 9 billion downloads. It's one of the most popular web development tools for building anything. Essentially, it's used by almost everybody from startups to small companies to Fortum 500. Everyone is using Node.js for one reason or another, and it also runs a lot of critical infrastructure.

It's fast and efficient, it's built on top of the V8 JavaScript engine, which is what powers Google Chrome, and it delivers excellent performance for IO bound applications. But there is one catch. Node.js is single threaded by design. If you ask chat.gpt or cloud mostly, that you would get this answer. That there is only one thread per process. In fact, the JavaScript VM, it only runs one JavaScript VM on a single event loop. It does non-blocking IO, it's great. And it has a lot of limitations for CPU intensive tasks. Are we sure? Is it still true? Did you hide in a cave? Maybe. Where have you been? Well, since 2018, Node.js has worked with threads.

And these enable true parallelism and to use all the cores that are available in your application, in your hardware. But this is not widely adopted. A lot of people don't know this. They don't know that they can use all the cores. They don't know they can use all the CPUs. And it's just like, OK, where have you been? Like, I'm sorry. This was one of the biggest things being added to Node.js ever. And you will not be using it mostly.

So why? Well, let's talk a little bit about running Node in production. So there are three pillars of running Node in production. The first one is you need to detect, to track the application health and ideally detect problems before they happen. A lot of these pass through having a lot of visibility on what is happening. So you need to have metrics. Also, you need to implement fault tolerance and handle failures gracefully and recover quickly from errors. You want to do that. You don't want to disrupt your end users if they are having a problem.

So how do you monitor health of an application? Well, first, the most important thing to know is how the Node.js event loop works. There is a time after initialization. The event loop runs continuously if there is some event to be processed. Specifically, first of all, it does checks if there's some pending callbacks from the previous run. Then run idle handles, which is our set immediate in JavaScript. Then it polls for IOM. And then it runs some close callbacks, some things like that. Then last but not least, it runs the timers. It's pretty great. However, as you can note from that, there's an observation problem. If your event loop is busy, it cannot easily observe itself. This is one of the key problems. How can a busy event loop observe itself? Because monitoring is blocked. If the event loop is not spinning, if it's doing some processing because the event loop is single threaded, we cannot do any monitoring on it.

Or can we? Also, if the event loop is blocked, you are putting yourself in a wasteful scenario where you need to set up low threshold in your auto scaling or usage pattern to keep the system observable. And so if the system becomes very under load, you are in trouble. So you need to keep the system working in a suboptimal pattern.

Well, we need a better architecture. We have been thinking a lot about this problem, and in the last few years, we have tried to develop a better architecture, a better solution. So what I'm going to talk about in this talk is VAT, the Node.js application server that we have built at Platformatic. But all the patterns that I am describing are totally generic anyway. So all of it is open source, Apache 2. You can use it, but you can also look and learn how it's implemented.

So the first big, our secret sauce is related to moving the applications to threads. So essentially we are splitting the control plane, the application from the monitoring layer. So we learn the application as a thread and we keep the main thread to act as a coordinator. This enables the coordinator to observe the applications externally from the outside. So we can know what's happening. We can react. We can solve that problem of observability. If the system is blocked, we can still do something. We can execute code if the application is blocked, which is phenomenal.

So the coordinator monitors the workers, the applications, and can handle their lifecycle while the threads can do the applications and they can start, stop, gracefully shut down and so on and so forth independently. It's pretty cool.

What about metrics? Well, we run the Prometheus server on the main thread. This means that the metric server is always accessible. And it also avoids the paradox that we talked about before. If the worker is blocked, we can still do something. This is the monitoring architecture.

But what about failures? If you are on a single-threaded application, degradation will usually go unnoticed. Our multi-threaded approach instead does something different. If a worker thread crashes, we can restart it immediately. And so when that goes down, your infrastructure needs to be picked up and restarted from scratch. Well, there are a few more things that we can do because of this.

Yes or no. And so it very much improves the health of your Kubernetes clusters as well. But what about failures? If you are on a single threaded application, degradation will usually go unnoticed. And so L-check stops responding. The app is busy and everything starts being very bad very quickly. In this scenario also, critical failures are often catastrophic because the application goes down.

And so when that goes down, your infrastructure needs to be picked up and restarted from scratch. So our multi-threaded approach instead does something different. If a worker thread crashes, we can restart it immediately. But also we can be proactive so we can monitor second by second if health metrics are degrading and then restart the worker and typically replace it. So we can replace the worker without losing any requests in the system.

How does that work? Well, you know, is this approach better? Maybe. Well, you should trust yourself. So this is better because it allows way faster recovery compared to leaning towards your infrastructure to deal with it. Also, we can do hot swaps essentially and provide it with no request loss. This is great. So let's do that. Well, there are a few more things that we can do because of this. And first is we can run multiple applications inside the same process based on this threading model. They share completely different JavaScript VMs, completely different event loops, so we can spread it. Pretty great. Also, each application can be scaled independently based on their loader resource requirement.

How does that work? First of all, we provide an internal mesh network, so we can have multiple apps running inside the same VAT process. And they can communicate using HTTP or inter-thread communication. Both work great. But all of this is mapped to memory to just pure messaging, so it's faster than just allocating a socket, essentially. Pretty good.

Also, we can do in-process scaling, so we can spawn as many workers needed to use all the CPUs that are available. We can also scale this up and down based on resources, so that can also be a possibility. How does this work? Well, you can see it here that we can spin it up multiple times. They can work both in inter-thread as also at the entry point level. We can also have a hot swap, so we can also add and remove applications. This is possible. It probably requires a little bit more infrastructure to manage it.

Well, let's keep going. You can try a few things. If you have watched this talk, you should be taking care. You should be taking home a few things. First of all, the monitoring paradox is real. You are suffering from it if you're running Node.js. A solution to this, a very common traditional solution to this kind of problems, is separation of concern. So you can do that. Also, faster starts are better than trying to recover unhealthy processes. Now, a few bits. A gift from you. You can download the book. The Definitive Guide for Node.js in enterprise. And now it's time for my demos. So this first demo is a VAT application. It's a VAT composed by three or more applications. Let me start this up. And here we go.

And now we can see this on Port 3042. This is the diagram that we showed before with a platformatic gateway, and a Node and Fastify app, and the Next app. So this is our Next.js application that is doing React server-side rendering. And these two are two HTTP calls from two servers, one on the Node.js HTTP create server and then one from Fastify. You can see this is called every single time. It's pretty good. So this is great. And you can see this is showing we have the composer here. But this is the Fastify app showing our logs. And you can see here this is our internal domain. So you can just do a fetch or a request to this one. This is Next. And so you see that they are all being here. Pretty cool.

So one thing that you can do here is we can adjust the number of workers. And we can do workers. Here, let me remove this. Now I can start this up. And now you see that I have two instances of Next, two instances of Node, and so on and so forth. So if I refresh this app, you see now it changes every two refreshes. It's changing because I have two of them running on the system, and it's very easy to spin up more as you need.

Then I can also show you another little experiment, which is a tiny web app. Another web app. This is still called a server-side rendered Next application. And it's in here. So very simple to set it up. And you see that I just need to specify its base path. And this is another little Node.js server doing the thing. Very cool. So this is, sorry, let me show you better.

So here we have the Node.js server that is actually having an atomic as late as 20 seconds. This is blocking the CPU for 20 seconds or 20 milliseconds on this route. So let me start this up and let me show you that your application, even this is blocking the event loop. This is still being shown. Change this up. Reduce the number of workers. Yep. Okay, started up. Now this is the second day of that application. This is a hello world. I can do SSR and I'm going to next. Okay. You see this is the time. And I can do what's the... I don't remember. Here we go. And this is the hello world. So if I start getting a lot of load in the home. Okay. This is, you see, that is going a lot. The SSR is still being rendered correctly without a glitch. Mostly because I am literally just running it on multiple threads. Cool? Just wanted to thank you all. And hopefully you enjoyed this talk and let me know what you think. And remember to download the book. Bye!