Let’s Build K.I.T.T. With JavaScript

Okay. So, hi everyone. So cool to be here at JS Nation and to end the first day with a talk about my latest side project where I try to recreate the maybe most iconic car in television history in JavaScript. But before we start, I have one little thing. Yes, I have a lot of demos in my talks and yes, you are allowed to clap. And I'm telling you this because I'm Swiss and in Switzerland we are really, really like to get permission before we show emotions. And if you don't feel like clapping, you can go to yay.nico.de. I see it's already working and you can launch some fireworks in my slide. I built this for JS Nation. I've never tried it before, but hey, what could go wrong? Yeah.

But who am I? My name is Nico. I'm a frontend developer or as we learned today, I'm a frontend. I mean, I can't turn it off. I can't turn it off if it's too much. Yeah. I'm also part of the Google Developer Experts program in web technologies and also machine learning and AI, which means that I try to bridge the gap between on-device AI and the browser.

Let's start with a very simple question. Who is KIT? So KIT is short for Knight Industries 2000 and it's this advanced AI-powered car from the 1980s TV series Knight Rider. The whole series is about Michael Knight who tries to solve cases, fights the bad guys and KIT who is this supercomputer, sophisticated AI that helps him. Maybe we should also rephrase the question, what is KIT? And in the 1980s it was pretty simple. KIT is just a very advanced computer doing very advanced computer things. Simple.

Then we learned about machine learning, about artificial neural networks, so maybe KIT is just a huge artificial neural network that learns by changing the connections between those neurons. But now in the age of AI, I have a simpler explanation. So maybe KIT is just an AI agent that connects different services with a decision-making process. Sounds good. Let's build it. If you want to build it, you first need to dig into the features. What are the features that make KIT human-like? First of all, we can talk to it in natural language. It is intelligent, so it can process information. It has a personality, it's not just a soulless machine. It has a history, so Michael Knight and KIT, they lived through so many adventures, so they can talk about all that history. And last but not least, it has a body, so it has a physical form that we can interact with. Now, if you want to build that, and I'm a front-end engineer, so naturally I want to see how far I can go in the browser. If you want to build that, we need a lot of machine learning. And as we learned today, we can actually use machine learning in the browser. And there's one library that I use quite a lot. We heard about that already, and that is Transformers.js. I will keep this part very short. So in the end, if you want to use Transformers.js, we can basically import this pipeline function from Transformers.js.

We can then define a task and also the model that we want to use. And then we can basically pass anything through this pipe, it will run it through the model, and we get the output back. So the input type and the output type heavily depend on the use.

How can we use Transformers.js to solve our first problem? There are two models we can use. The first one is Whisper, allowing us to transcribe spoken words into text. On the other hand, Kokoro.js is an abstraction layer on top of Transformers.js for converting text into audio blobs.

Adding intelligence to the pipeline involves utilizing LLMs, particularly large language models like GPD 4, Cloud, Gemini, and more. Different types of LLMs, including closed models for server farms and open models with open-source weights and architectures, offer various options for enhancing intelligence in the pipeline.

So let's see how that works. I have this little pipeline here. So I can now say, for example, hey, everyone, I'm right now at JS Nation, and it's an amazing conference. Let's hope this works. Hey, everyone, I'm right now at JS Nation, and it's an amazing conference. Do you hear the... thanks. Do you hear the fireworks all the time? Okay, that's annoying. That's not like that. Okay, that works, but it's actually not very intelligent.

So how can we add intelligence to that pipeline? Now, the closest thing we have to intelligence is LLMs. So we have large language models, we can add some text, and it will output some text. There are quite a lot of LLMs that we can use right now. There's the whole range of GPD 4, Cloud, and Gemini. So we have closed models that run in some server farms that we can use in via an API or using their text interface. There are also smaller models like the Gemini Nano or GPD 4 or Mini which are small enough that we can actually also run them on device, but they are all closed.

There's also another range of models. And that's open models. And open means that the trained weights and also the architecture is open source. So we are allowed to use them in our own systems. There's Meta with the Lama, there's Microsoft with Phi, there's Google with the Gemma series. And there are also small open models like the Phi 3, Gemma 3, Quent 3. And there's even open source models. So there's the small LM 2, which is an action or as far as I know the only actually open source model because also the training data is open source that Hugging Face used to train that model. Now with models that are small enough that we can run them on device and they're also we're also allowed to use them on device, we can also run them in the browser.

Now with models that are small enough that we can run them on device and they're also we're also allowed to use them on device, we can also run them in the browser. There is a library that I use for that called the LLM. With the LLM, we can basically create an engine, we can then create a new conversation, where we pass an array of messages. And then we can either wait for the full reply, or we can use streaming their tokens are added to the reply as they are generated.

Let's try that. So we have the same pipeline as before. But we have this new processing step. Hey there, my name is Nico. And we are right now at JS Nation. It's an amazing conference and we are on stage right now. So could you quickly introduce yourself to the audience. Hello, it's good to be here. Smiling face with smiling eyes. Our nation is amazing. Looking forward to connecting with all the incredible attendees. Microphone.

Okay, that works. Thank you. But in the end, we just created another random chatbot, because we are now talking to some random Jemma LLM from Google. And I wanted to talk to Kit, the car from Knight Rider. So what could we do about that? The easiest thing to fine-tune or to customize the output of an AI model is to use prompt engineering. And prompt engineering is the process of crafting precise and effective input prompts to optimize the output of AI models. What does that mean in our case?

So here I have the same pipeline as before, but now I have a different system prompt. And in the system prompt, I tell it who it is. So it is Kit from Knight Rider. I give it context, it's speaking to Nico. I also have a summary of Kit's tone. And I also have guardrails, because all those smiles, asterisks and all of that, they don't work well in that setup. So let's try the exact same thing with a different system prompt. Hey there, we are right now at JS Nation.

It's an amazing conference, and we are on stage right now. So could you quickly introduce yourself to the audience? Of course, Nico, let's refine the presentation data. Shall I optimize the delivery for maximum engagement? And as you see, that's the exact same pipeline, exact same model as before. But it's behaving very, very differently.

Let's talk about memories. Because oftentimes if we interact with each other, we talk about things that happened in our past. And in the case of Kit, there were a lot, there were serious seasons, episodes full of memories of things that they lived through. My idea was, how can I talk to Kit about those memories?

The other problem that I have is I have a question about something that happened in the past, and now I want my LLM, Kit, robot, whatever, to know what scene I am referring to. The solution to that is called a vector database. The IndexDB is not per se a vector database, but every database can be one if you store the text and its vector representation.

You then need to write a little playwright script that scrapes all the screenplays in plain text. Then you will figure out that those screenplays were written in pre-World Wide Web times. So you need to write your own parser that parses the script into structured JSON.

Now, what is a vector representation? So that's basically just a numeric or an array of numbers, and that is generated by another AI model, and that's called or that's the category of feature extraction models. In my case, I am using the old miniLM L6v2, which is trained for semantic similarity.

The other problem is now I somehow need to call this vector search function in my pipeline. The solution to that is a concept called tool calling.

The problem is the IndexDB is not per se a vector database because in my IndexDB, I have basically just the whole scene and some metadata. Every database is a vector database if you store the thing, the text, and its vector representation. In my case, I am using the old miniLM L6v2, which is trained for semantic similarity. I can pass any text into the extractor, and it will return an array of numbers in the end.

The other problem I have is that putting the whole episode or scene into one embedding is not ideal for vector search due to dilution. My solution involves using an LLM summarizer to extract main plot lines efficiently. The process involves inputting the scene, determining expected plots, providing prompts, and strict formatting of output. This detailed summarization process on my MacBook involved processing millions of tokens to generate the necessary output for vector search functionality.

Now, with an array of summaries and embeddings for scenes, the vector search function can operate effectively. Creating vector embeddings for queries, comparing them with summaries in the index DB, and calculating cosine similarity lead to relevant scene recommendations. To integrate this function into the pipeline, a concept called tool calling is employed. The application facilitates function execution based on prompts, enabling interaction between the LLM and the application for seamless processing of structured outputs.

It did work. It took me a couple of days. It also crashed a couple of times, but I was able to do that on my MacBook. In the end, I ended up having something like this. Now I have an array of summaries for each scene and an array of embeddings for those summaries. That also allowed me to create this vector search function, where I have the question, the query as an input.

I will then create the vector embedding of that query. I will then go through the index DB, and I will calculate the co-sign similarity between the query and the summary. I will then return the four scenes with the highest similarity between the summary and the query. The other problem is now I somehow need to call this vector search function in my pipeline.

The solution to that is a concept called tool calling. The concept behind tool calling is that an LLM cannot execute a function, but the application around it can. The LLM can detect when a function should be called. It can then generate structured output, could be text, XML, JSON, or whatever. The application can then execute the function, return the output to the LLM, and the LLM can then work with that output.

There are libraries to do that, but I wanted to see how far I can go with just basic text in, text out. I created this little tool calling function where I have a very big system prompt. In the system prompt, I tell the LLM, if you want to call a function, you can use this XML snippet called function call with the name of the function with the query. I then have some instructions, some examples, and I then basically just have the user query as the question.

So, let's see how that works. Let's head over to kit.niko.dev. Then we can wake it up. There we go. And then we can ask something basic. So, whenever I press the space bar, it will listen. Good morning, Kit, how are you? Good morning, Niko. I'm well, thank you. How can I assist you today? So, that's a pretty basic input. But let's try something more special. Let's, for example, ask, listen, do you remember the connection between Sonny and the deadly poison gas exchange and did something tragic happen back then?

I need to look up the episode to provide you with the correct information. I remember a tragic incident occurred during an exchange of poison gas between Sonny and Fidel. A canister was accidentally discharged, killing a bystander named Laurie. That event left Michael deeply affected. Okay. So, what you see here is... Thank you. So, what you see here is that I have a very big system prompt. In the system prompt, I have all the tools.

First of all, I have KidStone and all of that, and then I have a list of all the tools it can use. Now I have my question, and then it generates this little XML snippet. My application will then parse the response, will figure out, oh, there's a search episode function, I need to call that. It will then return the output of that function as a new message in the conversation, and Kid can then use that to generate a response with all the context that it needs to actually do that.

Okay. So, that works. But I also wanted it to have a body. So, in the series, Kid is a Pontiac Firebird, and the Pontiac Firebirds, they start at around 20,000 francs. Believe it or not, that's not in my budget for side projects, and also, I am not a mechanic, so I had no idea how to deal with a real car. But I'm still a Kid at heart, so I was able to create this little Lego Kid. Now the question is, how can I communicate with my car through the browser? And that, I can do that using an API called the Bluetooth, and that's a browser API that allows me to communicate with Bluetooth low energy devices.

On the car, I basically have a little Arduino microcontroller board that is connected to this LED strip, to the wheels, and then I have two batteries, one for the wheels and one for the microcontroller board. And on the board, I then have a very small C++ script that exposes one BLE server with one BLE service with one BLE characteristic, and that characteristic has two bytes. And one byte controls the speed of the wheels on the left side, and one byte controls the speed of the wheels on the right hand side. In my application, I can then connect to the device, connect to the characteristic, change the value of those bytes, which then changes the speed of the car.

Okay, I have one minute left, so let's see how that works. Let's turn it on. Now you see it is blinking, and I can then connect to that. There we go. Beautiful. And I can now use arrow keys to control that. But using arrow keys is quite boring, and I wanted to see if I can implement that into my LLM workflow. So let's check that. Hey Kit, I think we need to get out of here. Could you go full speed forward? I'm accelerating to full speed forward to get us out. Actually, I did add a little timeout, so I didn't want it to go off the cliff in the end. But that is a very simple example, and I could have achieved that by just basic pattern recognition where I'm listening for words like full speed and forward, but I wanted to have an actual reasoning process.

So what I can do next is I can, for example, say, Hey Kit, we have a problem. There is a road block in front of us, and on the right-hand side, there's a huge river, so we can't go right either. What should we do next? I need to navigate around the road block. Let's try turning left to find an alternative route. Okay. As you can see, there's an actual reasoning process where it tries to figure out the best tool to solve the problem.

So let's wrap it up. We were actually able to create Kit in pure JavaScript. It listens and talks using Jesper and Kokoro. It is intelligent using the Jemma2 LLM. It has a personality using just prompt engineering. It had a history using all the scenes from the Knight Rider series and agentic rack. And it finally has a body using the Bluetooth and tool calling. So we now know what I did, how I did it, but the real question is why? Why should I spend hours trying to recreate a car from a series I didn't even watch that I didn't even watch when I was a kid? And the solution is quite simple, just because it's fun. It's fun to learn stuff. It's fun to build stuff. And now with LLMs, we just have one more tool. We have one more tool in our toolbox that we can use to build amazing stuff.

So that was it from my side. Thank you so much for your attention.

The thing is, with the small, if you use big models like Gemini or with a Claude or JetGPT, then the whole structure, if you tell the LLM, hey, call this function once and if you have the results, then call it again. One thing that often happens when I have a small model is that it would first call the function, then it would get the result back and then it will try to call the same function again. No idea why, just because somehow it is in the context that it has things it has to run this function. Therefore, I had to have a list of function and the parameters that I already call. So if it's the same function with the same parameters, don't call it again and you're done in the end.

That's an interesting problem for that. I found it is a weird thing to go from the big models where, and then function calling is built into that. You don't have to have your own XML thing for that either. That's true, yeah. And they're built for doing that with, and then we step back to these small models and you've got a lot more work to do. Yeah. But you kind of have to do that when you're on the web. And also the format, whether I should use JSON or XML or just text, was very- You have to ask about that. Yeah. The thing is, I tried it with JSON before, but if you want to force an LLM to always output the same JSON structure just in a system prompt, then that works sometimes in the first prompt, but already in the second prompt, it will forget about that or it will output invalid JSON. So for me, I figured out it was easier to have just those XML blocks. It's easier to parse. Also, if the structure isn't correct, it's not a big problem. You can still have the rest of the text. So therefore, I did use XML. Fair enough. Fair enough.

So therefore, I did use XML. Fair enough. Fair enough.

How much LLM help did you have to figure out working with the LLMs? None. None? No. Let's admit it, everyone uses LLMs to get some new ideas, to figure out what's wrong. So of course I heavily use LLMs, but it's not wipe coded. I wrote it myself. I think that's important. And when you're learning something, especially when you want to go and talk about it in front of a bunch of people, you can't just have an LLM write all the code for you, because then you don't know what it did. That's the whole point. I wanted to understand how function calling works or tool calling works. That's why I went into the whole structure XML and all of that. All right.

We've got the last question here. I'm going to pop in. If you'd like to speed up the responses, what would the steps be? Have you got further work to do this to make that? Well, one problem is that right now I have all those steps, first step one, step two, step three. And if I would be able to split up the response and start generating, maybe after the first sentence, I could already start generating the text to speech stuff. That would be faster. Or I would maybe talk to Joshua, because his example was way faster than mine. So I need to dig into his code and see what magic he was using. And he also had a newer MacBook. That's a big point. The power does help, doesn't it? Yeah. Yeah. Fair enough. All right. Well, with incredible demos and a fantastic way to finish it up here on the residence track, let's give Nico another big round of applause. Thank you. Thank you.