Stop Paying for AI APIs: npm Install Your Way to In-Process Inference

Hello, everybody. Welcome to this talk. Thanks for coming. My name is Ed Silva. I'm a Node.js core collaborator in the developer relations, which means that my job is to connect with the community. So here are all my links. I like to write code. I like to write about code. So check the articles. Come in my LinkedIn and say hi. Let's connect in GitHub and all the places that you can find in my beacons. Just scan the square code and you'll get there.

I work for this company called CodeMirror4UQ. We are a software boutique. We are specialized in Node.js and JavaScript and frontend as well. We do Ruby on Rails. We do Golang. And we have nice initiatives internally, like community supporting. That's why we do open source, for example. So come to our website, check it out. And thanks for Codemining for making it possible for me to be here speaking at Node Congress. I'm very happy for this opportunity.

So today we're going to talk about AI. Yes, again, talking about AI. It's 2026 and we can't avoid talking about it. It's the hottest topic that we have nowadays. And our journey starts in 2022, when Sam Altman did this. Sam Altman just published this tweet. And he had launched the chat GPT and it was amazing. People started experimenting with it and companies started experimenting as well.

So implementing stuff in their products and their services. They wanted to do this. They wanted to have their products and to say, OK, my product is amazing because it's AI-powered. And of course, you saw that happening, probably happening to your company. And honestly, I don't think they were wrong doing this. Because think about it. It was something new, something that was impressive. And it's very normal to do this. To say, OK, this is something I want to check. I want to see where it goes, how it can help my company, how it can help my product and my service.

So, yeah, that's nice. But now, four years later, this is still happening. People are still trying to get AI in. Nowadays, it's way more powerful than it was a few years ago. And we had a lot of studies on top of it that tells how we can leverage it more and more. So nowadays, this is still happening. Companies are still trying to implement this. And for us developers, it means that we should gain this ability to implement stuff like this. AI features or even a whole AI agent.

And when you are asked about it, let's say your boss comes to you and asks you to do... I don't know, you have an e-commerce and they ask you to do an analysis on the reviews and check what people are thinking about your products or your services. And the first thing that comes to your mind is, okay, I will get an API key from one of these providers like Entropic or OpenAI or even Gemini from Google. And I'll start calling their APIs and we are fine, we're good. And this will work, of course. But I don't think you really need this, actually. So there are other ways that you can do this. And actually, I don't think you need an LLM. So for the majority of the tasks that we are implementing nowadays with LLMs, you don't need a full LLM for that. So sometimes you just want to do this sentiment analysis, so analyze these reviews of my product and see what people are thinking about it or extract some information that people are demanding for my service. LLM will handle this, of course, but it's like hiring a chef to cook an egg for you.

So, I mean, it will work, of course. But you can cook your own egg, that's what I'm talking about. So here, today, I want to show you how you can cook your own egg in Node.js, in process, how you can handle this task that we are usually doing like this, you know? And for that, we'll need the help of Hug and Face. Hug and Face, just in case you don't know, is this community.

And in this community, it has models, data sets, all around AI will be here. So if you need some model to do something, it will be here. If you need some documentation or some article, you'll find one here, of course. So people often call this community the GitHub of AIs. So you'll find a bunch of things here.

So, for example, you can see right here, for example, that QUAN models are available here. These are new models. They were just launched and they are here already. When these providers are launching models, they are just publishing to Hug and Face and then they allow you to access and people get those models and do different things with them. So it's really amazing to see what the community is building here.

And one more thing that is present in this community is this library here called the Hug and Face Transformers. And the Hug and Face Transformers just allows you to do the local inference of models. That's what it does. That's amazing. And by inference, I mean the execution of the model. So a model is basically a bunch of parameters or weights that were trained on an architecture. So for me to reproduce this, for me to perform the prediction that that model does, I need that. I need the weights that were trained. So the configurations, I will go through the same architecture with an input and it will produce me an output. So that's what happens in all the models that we are using these days. Some of them are more complex. Some of them are more simple. But that's what happens. Nothing else. So Transformers allows us to do this. They just built an abstraction that allows us to execute a bunch of models that exist out there.

So for example, this is DeepSeq3. And you can use DeepSeq3 and many other models in Hug and Face Transformers. And as you can see on the left, here we have an implementation. So this is the architecture implementation. The part that needs code. And on the right here, we have the weights, the parameters. So the Hug and Face Transformers takes both, joins them and gives you an interface that will be very simple. And you'll be using, you know, to solve your problems locally. That's it. But of course, this is Python and we are not here in Python Congress, but Node Congress. So we need something different to do our stuff.

And there's this, the Hug and Face Transformers JS. And the Hug and Face Transformers JS is supposed to be Hug and Face Transformers, but in the browser. And this is from the documentation and it says browser because Hug and Face Transformers JS was primarily built for the browsers.

So that was the idea. But nowadays, you can use it with your Node.js application very easily. OK. And as in Hug and Face Transformers, Transformers JS has not only NLP, so not only text classifications or question answering about a document or something. It has much more. It has computer vision. It has audio. So many of the tasks that we are using LLMs to solve. OK, so the Transformers will help you to solve because you can sometimes get a specific model that you perform way better than a more general model. OK, so this is Transformers JS. That's what we'll be talking about today. How you can get that into your Node application and how to perform your local inference without any external service or any even if you are avoiding even network calls.

The Hug and Face Transformers JS works a little bit different if compared to the original one. Instead of implementing the architectures, they have this format here. That's the O-N-N-X. So it basically gives a model, converts to this format, and it will be able to get into Transformers JS. So if you want to execute a model in Transformers JS but it's not available yet, that's what you need to do. You need to find ways to convert that model from the architecture that it has to the O-N-N-X and it will work like a charm.

OK, now that you know what it is, let's go to some demos here and I will show you how it works and how it can help you with real problems. I hope you can see how amazing it is and how it can solve many problems that you might have. So let's get started. The first thing that you must know to get Transformers working is to install the dependency. This is the dependency we installed in your application and you're done. You are able to start. And once you get that in your application, it's very, very simple to get things running. All you need is a pipeline to import this function and then the pipeline will take these two parameters. The first one is required. It's a task name. The second one is optional. The second one is the model, but the model, each task will have a default model.

So if you don't provide a model, you need to download and use a default one. So you can just use the task. So here are the tasks that are available. Of course, I couldn't show all of them. You can go directly to the Transformers.js documentation and see all of them. For NLP, for computer vision and so on. There's a lot of things that you can do with that.

For the first example, let's do that thing that we discussed with the product reviews and see how it goes, how it gets information about the product. So let's see what people are thinking about my product. It's a very coming challenge. How you can do that is with Transformers.js. Like this. This is pretty simple. Once you get a pipeline and you set the task name sentiment analysis, and then you get a function for you that will act as a classifier.

I pass a text to this classifier and it will execute and give me information. Of course, I won't show you this running, but I did something different here. Let me go to the terminal here real quick. And then you will be able to see that. Hopefully you are able to see that. Here I have the demos here. The first demo is this one, the sentiment analysis. I have sentences and everything is set up already. And as you can see, I am setting a model here. I got this model because it's a little bit different from what's on the slides, and I want to show something very, very cool here.

So if I run the demo one, it will execute. Oh, let me just remove something. Let me do some live coding here. Oh, not this one, but remove this console log here so it will be cleaner. If I run this demo here, you can see that it brings stars. So this model is different from the default one. It will basically get more than two classifications because the default one does these two classifications. This is positive or negative. This one is more granular and it's cool for many tasks.

For example, here is a four-star, so it says that this person liked the product, but it says that it's kind of expensive. It's not really a section. So this is what the model thinks about it, and this is how precise it is about it. So very cool. It's a very, very nice challenge to do. And notice that each model will produce a different result. So when executing this, understand the model that you are using.

Okay, the execution of this code, it happens this way. When I call this function here, for the first time, it will get this model downloaded somewhere. In this case, it will put it in the HuggingFace module inside of Node modules, but you can override this configuration. And then the execution will be local. So you can turn off the network and it's still going to work as expected. And then when you execute that, it will get everything up to your CPU, to your GPU, and get things running. Okay? So this is our first demo. Hopefully you liked it. Let's go back to our slides here.

Okay, this one is very cool, too, because it involves something that people are doing a lot with LLVM. And it's called LLVM-Cube. So here I'm going to show you a little bit of the code. So this is the code. So this is the code that we are going to be using. It involves something that people are doing a lot with LLVM nowadays. That involves a RAG. People are usually doing embedding stuff. And they get a bunch of documents. They do embeddings that basically turn them into vectors, into a vector space. And they save to an external database, a vector database, so they can query over it. But it's a semantic query. It's not just by terms or something like this. It's a way smarter query. Okay? So this is the idea.

And this demo here that I have for this is also pretty interesting, because I'm not only doing this kind of thing like query by terms. It's a little bit different. So, for example, I have these five documents here. They were vectorized already, so actually I have ten documents. And then when I query for something, it will semantically get a result. So, for example, how does Node.js handle concurrency with a single thread? Okay. Node.js is an event loop to handle what's the top document for this. And I have these others. So, as you can see, it's working well. Here we have a more like a normal distribution here. Here is pretty cool, too. It got a nice match. So this, everything here is done with Transform.js. Okay? All the embedding, all the query. So the whole code is here, and we will be able to find this code in my GitHub.

I will share the link in the end of the presentation. But it's cool to see that it works like a charm. And if you just want to embed something, you can just do this. You know? You get a pipeline, feature extraction, get a text, embed it, and YouTube produces you this array. That will be an array representing that text content, that document, in that vector space.

Our third challenge here is very cool. Because, I mean, I've been saying that you don't need an LLM, but sometimes you need an LLM. Sometimes you need to produce text, and LLMs are great for this because they are next token predictors. So even when you need text, maybe you need an LLM, but you probably don't need to call an API. Depending on your challenge, you can easily do this in your Node.js application or even your browser.

So for this, it's very simple as well. In the pipeline, get the task name, test text generation, give it a model. In this case, I'm using the LLMA model made by Meta. So this is the same model made by Meta, but just converted to O and NX as we discussed. I then give a streamer to it. It will produce text, and the streamer will receive each token, and then just print into the standard output. The execution of this would be like this one. So it's demo three. If you get the LLMA model loaded the same way we did for the others, and as you can see in this animation, I'm turning off the connection. It needs to work. Okay? So this is the basic thing that you do when calling external APIs. So history of the chat, you can do with local models as well. Okay, so in this case, this is what's happening. And this is the model. You can see it's hugging face, of course, and as you can see, it's basically the Meta LLMA, the very famous models from Meta, converted to O and NX. Nothing special about it. So the same thing that one does, the other one will be doing as well.

Okay? And of course, this is cool, but there are some trade-offs that you shouldn't be aware of. First of all, it's this one. So it takes some disk usage to get the model downloaded. So we need to host this. And the way we implemented this in these demos, of course, is not optimal because when the model is executing, the main thread is blocked. So you will need to isolate that, I don't know, using a worker thread or something like this, so you can serve more requests while it's working. And of course, the first load takes some time, so these are things that you should be aware, okay?

But for many applications that are running locally, these models, this local inference, can be very, very useful, and it's pretty simple to do. As we saw, we just install the dependency, and it works like a charm. All the demos here and everything I showed you here, it was working on CPU, but we know that this is not ideal. We want to leverage the GPU because they have better capabilities for handling the models, especially models that involve NLP. Many neural network models leverage the GPU, so we want to use that as well, and you can, actually.

The transformer.js already works well with WebGPU API, so if you're running a browser, just set the device to WebGPU and this is going to work. And if you're working in Node.js, this is going to work, but it's still unstable yet, so you need to install the transformers for Next, and then you get this capability, like the preview features of it, and then you'll be able to run in GPU. So, for example, if I did that same task of text generation, I can get that running on GPU. The execution will look like the same, but now we need to execute in a different chip, so that will be way better for it. So, as you can see here in the demo, on the left I have the execution of the LLM, and on the right I have the GPU history that shows like it's growing. So things are loading on GPU and it's using that chip.

So you can do that pretty easily in your code if you get the preview. So if I go here, oh, let me... I messed with something here. Okay. If I go here in my terminal, and I run this, and I show you this code here... Okay, something wrong is happening here. Can't I run in my terminal or something? Okay, now I can. If we go here to the demo three... Actually, demo four, because I switched. It will have the device here, and the device, just like this, okay? If it's GPU, let's just pass this WebGPU and this is going to work, even in Node.js. Just ensure that for this feature specifically, you are using this version and you get that running on your Node.js application, okay? So let's get back to the slides.

Okay, let's get back to the slides. Okay. Yeah, we did it, okay. So the question might arise, and we kind of discussed it already. So should you always go locally? It depends a lot on your application, on what you want to do. So let's say you were doing a CLI that should run locally and do analysis on top of a bunch of reviews. Why not? But of course, if you need something that's massive and you want to serve many, many requests and you can't handle this, you don't have hardware for this, of course there's nothing you can do about it. And you have to use an external service like everybody else.

This is more like about possibilities. It's not about replacing. This is not supposed to replace what people do with these models from providers like Entropic and Chattopadhyay. It's more like about possibilities. Sometimes you have a nice model that does something very, very useful for you, very specific, and you saw that in the Hugging Face, you heard about it, and you want to see that in action, you can go ahead and do this. So this is more about possibilities and experimentation as well. And of course, this can solve real problems and people are using it to solve real problems, of course.