Node.js: More Threads Than You Think

So, since we've been talking a lot about front-ends today, it's time now to make you suffer with a bit of back-ends. As long as you can run now, if you won't, otherwise, good luck. So there is a lot in the unknown. You are mostly, I assume, front-end engineers, which work with Node on many systems nowadays, which means that you use Node, but you don't actually use Node, which is kind of amazing to me. And let's see what it means for you.

The original intro is pretty fine, so I'm going to skip this one. I just want to let you know that I come from the smallest and less known region of Italy, and all you people should visit because it's just amazing. OK? OK. So Node.js is no longer single-threaded. Do you still believe that it's actually single-threaded? Raise your hand if you think so. OK, just go. It hasn't been single-threaded for a huge while now. So Anna Emingsen presented the Node worker threads in 2018. And it's been stable since 2018, June, on Node 10. We are at 25 now with the worker threads module.

So folks, please. The thing is that this approach has a huge advantage because each thread has an independent event loop. So you can do things totally independently. But unlike other systems, you don't really have to worry, unless you really have to, about synchronization, shared access to stuff, and so forth. So it's very, very, very good. And it's very, very good also to offload some of the CPU-intensive load out of the main thread. As you know, Node is meant to be IO-intensive, not CPU-intensive. So if you do CPU-intensive stuff, you're killing your main thread, which often is your only thread, so you're done. You see the problem, right?

Why do you care? Well, the thing is that, how do you scale Node.js in production? Who is actually serving your shining SSR application? And I don't care about the framework. SSR. Let's just talk about SSR. Who is serving your SSR? Node. How do you serve it in production? How do you scale it in production? Okay. Okay. Now I want to talk about a familiar story that we all have been seeing for the last 20 years, 30 years. I mean, at the end of the slide, I will ask you, who is familiar with this thing? We have a situation. You have a traffic spike. You have several pods on your Kubernetes cluster, right? Now the problem is that if you go on your Grafana dashboard, you will see that the pod from one to 10 are like at 30% and two other two pods are 100%. So the requests are actually blocked. Your error rate is scaling up at 8%. So what do you want to do usually? The only thing you can actually do is to over provision your cluster. So you add more resources. You add more resources so you can take new requests. At some point, those pods also get queued up and you're back to the original problem. In the meanwhile, you will be wasting a lot of money on that. Now if you think that this is only about provisioning, it's not just about provisioning. Because if you have a coordination between APIs or micro frontends or whatever you want, if you have 185 milliseconds delay on one API, on another API, on another API, is half a second delay for the user. You already lost a user. There is customer churn. Now, raise your hand if you're familiar with this situation, please. The rest of you are just liars for the record, because we know that. We know that you're familiar with this. We know. So how do we handle the scaling inside the node?

Since node 0.6, which is from 2011, and maybe some of you were not even born back then, it was added the cluster module. The cluster module basically is a very intelligent technique that basically says, I have a master process which will just take connection in and basically route throughout workers, using Unix sockets. It's very, very clever. It worked for several years. It's the basis of PM2, which is a familiar orchestrator for this kind of multi-instance applications.

It's the basis of PM2, which is a familiar orchestrator for this kind of multi-instance applications. Is anybody using PM2 here? Please raise your hand. Okay. So you know how it works. The problem is that this IPC, so Inter-Process Communication, via Unix web socket, is adding for each of your requests a 30% overhead in the response time. You're wasting 30% out of your application logic, business logic, and so forth. So that's the idea. I just talk about this, but that's the graphical representation.

So basically, the node master will basically talk to the operating system to deal with the network layer, accept connection, route them to one of the possible worker, and then eventually send the response back. But there is the overhead. Then there is another problem, which is pretty peculiar to Node.js. When the system is overloaded, ideally, rather than accepting new connection and making the situation even worse, you maybe want to return to the customer a 503 response. The problem is that the server can reject this very early and say, you know what? I'm busy. Come back later. I know it's not nice, but it's better than letting you wait for 20 seconds.

So they immediately reject with 503. The problem is that this is not possible in Node, because the idea is that in order to reject a request, you have to reject it when you actually process it. And in order to process it, you have to accept it in the event loop. By the time you are getting to processing that request, the customer has already waited for a long time. And also, which is also make it even worse, this request is still in the event loop. So all the associated requests, network sockets, eventually already loaded data, and so forth, are still in your RAM for no reason at all. I mean, you're filling up your RAM for requests that you're not even processing. I mean, we are getting in the very horrible scenarios, right? And since we've just been talking about Next.js in the previous talk with Wyatt, Next.js, but even other frameworks, I'm just talking about Next because it's our case study today, but this is generally true for all other frameworks like that, TimeStack, Nuxt, whatever you want to think about.

This kind of framework for SSR makes it even worse, because in order to decide what to do with that request and reject it, they have to first load the request, preload some data, and then the dynamic routing must happen, and then at some point, they say, okay, you know what, I cannot serve this request. There is a lot of resources that you've been wasting bot on the RAM and CPU in order to process something that you just discard right away. Data fetching, I mean, there are a lot of stuff that happen before you even get to decision making and saying, you know what, I cannot serve this request at this moment. So that's not a viable approach of the early rejection in Node at all.

And of course, like I already said earlier, latency compounds this effect throughout the microservices or micro-frontends, because you basically have to sum up all the delays. And with the single CPU pods or PM2, if some requests are unbalanced, this effect compounds a lot. That's the biggest issue that we were seeing in all our customers throughout all the possible scenarios, not only to frontend only. That's the problem.

Of course, since I'm here, I have something better to tell you. We actually solved this thing. The most astonishing thing about how we solved this problem was actually that our technical foundation is an API that has been around for now 12 years on a single operating system and because I will tell you in a minute why and actually nobody's actually using it, which is astonishing. It's something that I don't actually and I can't actually explain. Technical foundation. TCP SO reuse port.

Just out of curiosity, raise your hand if you actually know what I'm talking about. Anybody here? Not a single one. Nice. Okay. So, that's the idea. Usually in Unix or actually in Windows or even on Mac, if you try to listen twice on the same machine on the same port, you get an error which is E, add in use, right? Because of course the port is busy. With this flag, if you want to basically create your socket, you set this flag at the C level, you actually don't get that error anymore. You can have multiple processes listen on the same port.

How it works? The kernel will basically when a new socket request arrives, a new connection arrives to your server, will basically create a source IP, source port, destination IP and the destination port if you know what I'm talking about is the quadrants of the TCP packet basically, will calculate an hash and assign this hash to one of the workers, one of the listening process. From that time on, every time a new connection comes from the same socket, goes to the same worker. So, basically the kernel will make the round robin or any other algorithm policy, actually there is some more complex algorithm for you, will behave like a router and basically enable more worker to listen on the same TCP port.

There is connection affinity. There is distribution. There is no need for any coordination because you don't even see this thing. This is done totally at the kernel level and one flag in node enables this thing. When you create a net server create, you would say reuse port true and you're done. That's life changing in this. I will show you why this is very, very life changing. The funny thing is that this thing has been available since kernel 3.9 on Linux and I say one operating system and a half because it's also supported on Mac so technically you can use the flag but the round robin doesn't work.

So only one of your worker will eventually get the traffic for some reason. It's always the same one because the routing is totally broken but at least you can test it if you have a single worker with the same logic and eventually deploy to Linux without having to change the flags on your code. Better than nothing. Guess which operating system is missing on the list of the supported operating system and we move on.

In order to leverage S or reuse port we create VAT. VAT is the node application server. I'm saying in a very bold way because we simply rock Foxhike. I don't want to brag it but we solve many problems and we create a unique architecture which leverages all the smart testing you can do in node. For instance this diagram that I am showing you here is basically the basic architecture of VAT. Let me also remove myself from the portrait. The idea is that when a request gets into VAT you can eventually and optionally use a gateway which is the entry point, the platformatic gateway. Platformatic gateway will route your request according to basically the request URL to one of the possible services inside.

Note that neither Fastify or Next.js service are actually exposed to the external world. You can have services that are just internal to serve other services and so forth. The nice thing is that when I say send request to API.plt.local or frontend.plt.local I'm not using TCP. I'm using HTTP but without the network layer. We're using inter-thread communication which is way faster than any possible network including networking on local host. So all the internal stuff happens via worker threads. Also VAT is production ready. We've been testing on several production systems and we have the built-in monitoring, logging, tracing and health checks. Why this is important? Because in VAT, the only basically technical requirement that we give you is that you never run on the main thread. Why is that? Because we reserve the main thread to do the coordination and health checking of other threads. Now you might think, why is this important, Paolo? Well think about this, in Node, whatever health checking system you insert in your service on your main thread, I have one question for you. If the event loop is totally utterly full and cannot process any little bit of data, when is actually the health check running to notify somebody that you're actually failing? It's totally stuck. So you have to wait for eventually for that thread to slightly unlock maybe in a minute from now and notify somebody, let's say Kubernetes, that you're actually stuck. It won't work. Instead with VAT, the main thread that does not run any service is only for monitoring of other threads and we can immediately say, this thread is stuck, restart it. Which brings to another good feature of VAT. So this is the architecture of VAT, which I already talked about. The thing is that since we use SUS reuse port, we directly accept connection at the kernel level.

But that's not important now. The important thing here is the process orchestration. If what I was telling you, if a thread is failing, the main thread can detect it and immediately stop it and start a new one. And we are so flexible that what we can actually do is start a new thread when the thread is ready. So even if your application has a long starting time, you get no downtime because we wait for your application to report ready and then we swap the thread in the routing and there is no downtime for the user. For the same reason, we can do graceful shutdown.

And as I was already telling you, we have built in monitoring and metrics. The biggest advantage here compared to PM2 or other systems is that we can immediately, in the matter of one second, for instance, the minimum resolution we can give you is one second, check that the ART loop is blocked, if there is a exhaustion of the IP memory node. I'm saying IP and not RAM because that's a totally different story, but consider that in Kubernetes you should never use CPU usage and RAM usage to monitor nodes because it won't work. Event loop utilization and IP usage, that's what you're looking for.

And we can immediately and automatically instrument it and monitor it. And we can replace failed worker in place with zero downtime at all. And I say zero downtime because I suffered for many days in order to guarantee that you don't lose any single request. It was a suffering PR that I sent to Watt for this, so I know what I'm talking about. And since we were talking about cache component, we also implemented shared HTTP cache. And we also implemented, in case of Next, shared React components cache.

What does it mean for you? If you have a shared cache, you can basically have all your workers inside the pod share the same memory. So basically one worker will render an SSR page. When it comes to another worker, the result is already there to be served to the customer. So the response will be immediate even though it goes to another worker. And if you put this cache inside Redis, you can even have this shared cache across all pods. So that's immediate. You never basically implement the dry rule, don't repeat yourself, at its finest. You never do the same operation twice unless you use the wrong cache key. But this is your fault, not mine.

The thing is that response times are outstanding. Now, so far we were talking about localhost, right? Let's move to the on-premises. Since you see my shirt, right? So let's go on-premises. Let's talk about Kubernetes. How do you deploy VAT efficiently to Kubernetes to gain the highest possible performance? We went for a very simple architecture, two-layer. We have Kubernetes servers, which will act as the ingress router and say, I get a connection. Let me see which pod I should send this connection to. And then inside each pod, we do another scaling with VAT and SORU support. The thing is that this approach guarantees the best possible statistical isolation and distribution of all your request pool. And we verified that.

Because the thing is that at the end of the day, you get the most independent processing request ever inside the pod and inside the cluster. There is the less possible variance of response time because it's much better than the pure round robin because if a worker is blocked, another worker will take on and that's totally fine, because this kernel will take care of you for that. And there is better utilization because a single pod will not be blocked by a single worker going down because they're totally independent. Also, within pods, you get a little other benefits, like you have a single network layer. You have only one copy of the application because Node will load your application from the file system once and then eventually spawn several workers, because we cache what we require actually. And there is less kernel paging, which means that you use your RAM physically more efficiently.

Now, let's talk about the performance, because all you care about, about this guy mumbling about Node is how fast is this? We made a benchmark. We deployed a next application on ETS AWS cluster. We tested it for two minutes with a sustained load of a thousand requests per second. There were three nodes on this cluster, and we tested with K6 on another large instance with a constant arrival rate execution, and the virtual units were pre-allocated in order to guarantee the last variant possible.

What were the configurations? We tested three cases. First of all, we have a Kubernetes cluster, six pods, each pod has one full CPU, so six total CPU. Then we have PM2, for which we have three replicas with two CPUs each. Once again, six total CPUs. And then we have BAT with the same configuration of PM2, but we used two different approaches, cluster versus S or reuse port.

What were the results? These were the results. So you can see from the graphs that basically we outshined in all possible cases. I can leave the graphs for me. The idea is that we have 933% faster than PM2. We have 81% faster than P95 latency. We are 92% faster than single CPU, and this is just the latency. Then we have the throughput availability. We first of all give you 10% more than PM2, but we give almost 100% success rate. With PM2, you get a huge amount of lost requests.

Now why is this possible? The first problem is that PM2 is losing a lot of performance on the master-slave configuration, unique socket, and so forth. So you pay 30% overhead even when you're not loaded. And why the single CPU underperforms? Because it's isolated and imbalanced. The round-robin is not a good algorithm, so some posts might be 100% because they're full, and they get overfilled by the round-robin approach. While others are at 30%, they cannot rebalance, you waste money. That's the idea. It's sad, but as I'm already telling you, you leverage the zero overhead kernel distribution via S4U's port.

And so as better statistical multiplexing of your requests is at 99% reliability. If you want, I'll give you one second to share this, scan this QR, or you can easily find on our platformatic guides. Let me actually move myself, sorry. If you want to take a look, or you can scan my shirt. I'm not sure if they send me here because I'm the biggest guy in platform, the biggest surface area for the QR, but I live with it, it's totally fine to me. That's it. Usually I finish my talks with quotes from wise people, so I found this one, which is amazing in this industry, which is when we basically pretend to work, but we actually play and learn, and we know that. So you're always a student, you are never a master, and basically you have to keep moving forward.

To this thing I add, move forward wisely, don't make too many mistakes. With that said, I'm done. Thank you very much, folks. I love the theming of the questions. Let's just say anonymous, whoever they may be. Has a very unique sense of humor in the way, especially the way they react to reusable. I really appreciated how you said, I can guarantee you there's zero downtime because of the suffering that went into that pull request. So we appreciate the hard work that you've come and you've shared with us today.

Let's jump straight into the questions. And I love this one, I'm just going to read it as it is. How is SO reuse port legal? I thought binding twice to the same port was a crime punishable by segmentation fault. Which makes total sense. We live by ordering news, right? The thing is that it's legal because it's just smart. Because if you think about it, when a request comes into the socket, the operating system will say, OK, this port is actually assigned to a process, right? That's how the kernel works. Now what probably is the kernel to say, oh, I have a set of ports, so why not? That's a very wise idea and actually a very old idea that nobody uses. Like multiple threads in node, you know? That's the same idea. No, I love it. I love it. I love it. And the next one is, does it work for web sockets as well? Absolutely, absolutely. The thing is that it's not specific to web, it's not specific to anything. It's just that once a byte comes in a computer, the kernel sends it somewhere. The content of the, and of course the same goes for the outgoing side. And the kernel has no idea what your application is doing. Because we are not on the level four of the stack. We are on level three, or even two. So you can do whatever you want because it's way before the application layer. Wow, that's amazing. To be fair, there's a backend engineer I'm going to send this talk to, to handle this for me, to be honest. And maybe some other people in here have the same thing. The next one is, yeah, using it to multi-, sorry, if SO reuse port lets multiple processes bind to the same port, is this load balancing or sorcery? Sorcery. Sorcery. That's what you're asking. You're not asking. And then this one, I think you covered it, but let's go over it again. How does the kernel decide which process gets the connection? Is it round robin, hash-based, or whichever worker had more coffee today? I guess the latest one for all I know. Because the thing is that I never look in the details of the implementation. The thing that I actually know, but if you're interested, reach out to me and we'll discover together, because I'm not 100% sure, is that at the beginning is a balanced round robin.

So, it's not pure round robin, but it's considering several stuff like the CPU and so forth of the process. After that is connection affinity. So, if a new connection comes from the same source, it will try to assign to the same worker. But on the very first one, it tries to manage the connection efficiently. I'm not sure how, because it might even change between different versions of Linux and probably have several algorithms you can opt in, so it's not a general answer. But the idea is that the best balancing at the very beginning, at the very first connection, and then connection affinity.

And remember, as well, if you do want to come in over and have a chat, he will be over downstairs in the Q&A live in person, so you can go ask a follow up and maybe get some more detail to your questions. What I loved, the performance graphs that you showed and how fast it is, does it only work with Kubernetes? No. No, but it works with everything. It does not need Kubernetes or anything like that. For Kubernetes, we have another product, which is the open source product, which is the Intelligent Combine Center, which is an orchestrator for Kubernetes, which is also open source, by the way. I'm only talking about open source stuff, folks. But it's just the application server which can be deployed anywhere. It's up to you. If you, of course, integrate with ICC, which we deploy, the developer is probably the best configuration for you, but it's not a requirement.

Nice. And the next one, you showed the graph which compared PM2 versus what. This person asks, are there any advantages to using what to run? They use PM2 to run their front end in CI pipelines. Is there any advantages of using what, as opposed to just speed? If I understand the use case that you use in the CI pipelines, I mean, I assume that you talk about testing in pipelines and so forth. You get the better throughput back there, which means less pipeline duration and probably less money spent, I guess.

Nice. The next one is opinions on Deno. So if you use Node, are you talking about multiple threads? You're asking a Node.tsc member opinion about Deno. Really, folks? Really? Really. Come on, please. Move on. We'll move on. We'll move on.

Then one person asked about standard open... We have it. We have it. We totally have it. In what you get for free, Prometheus metrics and open telemetry instrumentation for HTTP and we make it very easy to add your own instrumentation. For instance, on our Kafka client, we also have the open telemetry out of the box and so forth. So if there is any use case that we've been missing, make sure to drop an issue on GitHub or send a message to us so we can take care of that. And someone else asked, this one is about Redis. Do we need to use something like Redis so all the processes share the same state? Absolutely, you do. Think about this. That's a problem that you also have today. Basically, you have your application, you have a Kubernetes cluster.

Basically, nowadays, with next out of the box, I'm not talking about any specific configuration, with next, your application gets to a next instance. Sorry, your request gets to a next instance. It gets cached to that local instance. Then the user refreshes the page and goes to another pod which does not have the cache. So first of all, you're computing the same page twice. And you have also talked that your computation yields the same results. Otherwise, the user will see different results depending on which pod is actually hitting. You see the problem, right?

And I mean, there are bad jokes in Italy of what happened the last time they messed up with the caching on a government site. So you can look it up and really bad thing happened. So you really want to do this in the right way. Otherwise, you have to make a statement on this news saying, oh, you know that we made a mistake? We've been hacked. You know. Classic. Classic, yeah. So you talked about this specifically with AWS hosting as well, but there's lots of different cloud providers. Yeah, yeah.

This person's asking about Vercel. How did this work around? No, well, maybe I misexplained. We were not testing on AWS hosting. We just use EKS for the provisioning of the resources. It could have been on-premises test with the same result. It just happened to be EKS, but it could be on Google. We are totally vendor agnostic. Vercel hosting is not a direct hosting of your resources, so we don't integrate with that because they simply don't run either PM2 or VAT. They have their own infrastructure. Our use case for VAT is when you have on-premises. When I mean on-premises, also mean cloud on-prem, if you know what I mean. Basically, when you directly manage your servers or if you're on the cloud, which means you're managing letting someone else manage your servers, which is a different topic.

Basically, when you directly manage your servers or if you're on the cloud, which means you're managing letting someone else manage your servers, which is a different topic.

The next question, they've said that VAT's architecture reminds them of Elixir or Erlang's model. What's the recovery scenario when a work thread crashes? The same in Elixir and Erlang. We borrowed the idea, not the formal idea, but the inspiration for that came from Elixir and Erlang. Nobody writes Erlang, but we just know how they're doing Elixir. That's the same idea.

Since we wrap all of your application in a worker, you can roughly, and consider that I don't write Erlang, so forgive me for any oddities I'm telling you right now. You can assume that a worker becomes an actor and we let the actor fail like in Erlang. The worker can fail and we replace it right away. The thing is that, why this is better? Because restarting a worker is much faster than restarting an entire application because you have to load the runtime and so forth. Instead, you only restart the worker. The binary code has already been cached. You just have to re-execute it. We let the actor fail and go move on with our lives.

This one, we've got a few more fun questions. The first one, is it still called horizontal scaling if all the processes are literally sitting on the same spot? That's a funny question because we do, I mean, let me craft this one. We do actually T-scaling. Because inside VAT, we use it to call it vertical scaler, the vertical scaling of the application because you can, one thing that I forgot to mention today is that inside VAT, you can also get a huge benefit in performance because we allow you to, thanks to SORU support, to have multiple instances of the same application running inside VAT.

You can have five workers serving next. If you have five workers serving next, you get linear performance improvement. Five workers mean 500 output more. That's our vertical scaling. On Kubernetes, we have the horizontal scaling. So basically, at the end of the day, we're doing the T-shape scaling, like the skill set that we're nowadays speaking about. I find it hilarious because I think the person who wrote it and definitely me when I read it, it was a fun joke question. You actually had a really good technical answer.

Yes, absolutely. I really enjoyed that. I really enjoyed that. And the last one, everyone's saying we definitely need to update our Node V6 applications. I think they should. You're like, what we are now, we're depending in 20s. So like 14 version behind. Poor guy. Actually, I also have a question, an answer for you. There is a friend of mine, Marco, which is working for Erodev. They manage outdated version. Go talk to them if you don't want to update for V6. You're gonna make his life miserable, and I will thank you for that. All right. Thank you so much.

This is all we have time for today. Can we give one more round of applause, Paolo? Thank you. Thank you.