Build RAG from Scratch

What's up, everybody? My name is Fonash and I am a developer relations engineer at a company called Datastacks. If you ever want to find me online, you can find me all over the place as Fonash on Twitter, on LinkedIn. Come and connect with me, I'd love to chat with you.

But let's get into the talk. Today we're going to talk about building a rag from scratch, and to deal with that, we're going to have to work out what rag is first. This is all about generative AI, though, and if you have dealt with large language models, either using consumer applications or trying to develop using their APIs, you'll know that they have a couple of problems. One of which is they don't know any up-to-date data, because they have a training cutoff date, and so they can't possibly know anything beyond the date that they stopped training. And the second is that they don't know any private information, and that's a good thing. We don't want them to know our private information necessarily. We certainly don't want them to have been trained on our private information. But if we want to use these as a useful kind of business or application, then we might need them to know some private details. And so we need to somehow be able to give a large language model this information.

I just want to show you that as a demo. I've got Google's AI studio up here talking to Gemini Flash, one of their models, and I'm going to ask what is going on at JS Nation US? And it can't possibly know this, because it was trained well before and it says it doesn't have access to real-time information. You might as well check the official website. It's good advice, but it's not a very useful bot. We could also, you know, this is a bot built with my own thing, and so this is also calling out to the Gemini 1.5 Flash API, and if I ask it sort of what is Phil Nash speaking about JS Nation, it also, it just it's not going to know this. It doesn't know anything about me. It needs more information to answer that question. And that's fair. It doesn't know. It was past its training cutoff date.

And so what do we do about this? This brings us to the idea of retrieval augmented generation. This is one of the ways in which we can give a large language model more context with which to answer. The idea behind retrieval augmented generation is we take all of our up-to-date or private data and we store it somewhere.

When a user makes a query to a bot that we've built, we use that query to retrieve relevant data about the query, provide that data as context to the model alongside the original query, provide how the model to generate a response using that context, using information that it couldn't have known previously, and then eventually I guess profit. That's the idea here.

How the difficult part of this actually, you know, because models have been made, we know this, and they keep improving and getting better. The difficult part for us is to retrieve relevant data based on something like a natural language query. And so this is how it all fits together.

So retrieval in this retrieval went to generation retrieval is that search step against your data that you want to make available to the bot. The augmentation is where you turn it into a prompt and finally the generation is handled by a model. And we're going to talk about the search side of this because this is the most important part. And so it's important that it's natural language search. It's not keyword search. We want to search by similarity of meaning.

This gives us the idea of vector embeddings. Vector embeddings are a list of numbers, ultimately, that represents the meaning of a body of text. And we can generate these in different ways. We'll have a look at a couple of those ways today, starting by creating our own vector embedding.

So I have a hypothesis for this. For a bot about a conference, say JSNation US where we are, the titles and descriptions contain the meanings of the talks that we're going to see. Those titles and descriptions are made up of words. Provide me here. And titles and descriptions that share words are similar. And so also natural language queries that also share words that are in those titles and descriptions are probably asking about those specific talks. And so if we collect all the words and represent each talk as a count of each of the total set of words, then when we get a user query we can do the same thing and compare them, ultimately getting similar words based just on the similarity of similar talks and queries based on the number of words that they share.

And so let's go and have a look at this. I've got this useless kind of event chat right now, but I have the code for that over here. And so what I've done is I scraped all the talk descriptions for JSNation US and I've got those in a big array of JSON file. I'm going to go and do some processing on them. So we're basically cutting out punctuation and common words, because those common words in my hypothesis don't really capture any of the meaning, because they're just I, me, myself, like not particularly useful stuff.

And so for each of the talks in our in our big list of talks, we can turn them into that kind of array of words. And so actually we can get all of the words from all the talks by making an array out of a set, so that we basically make them unique words by flat mapping over the set of talks and returning the array of words. And so if I were to console.log all words, we can see there's loads of words and the first ones are AI assisted developer workflow, build faster smarter today, that's Addy Osmani's talk. Fair enough, so that's that's the first one and that's the first set of words we get there. And if I just have a quick look at the length, we can see there are 1498 total words used across all of the talks, all the talk descriptions, counting out those common words. And so once we've done that, we can then create a vector out of a talk using all of those using all those words. And so what we're going to do there is grab a map of the words using an object, in this case, for each of the words in a talk, we can run through them and make a map that says, if you know, count up each of the words, and that's what this word map does.

And then finally, we can map over all of our words. So for each of the words in all of the talks, we'll check if they're in the talk we're looking at and what the count of them is. And we kind of map that into this big array. And then we can do that to all of the talks by mapping over all of them, returning them the talk context and their vector.

And so to see that in action, we can just then log out, talk vectors, let's look at the first one. Like I said, that's Addy Osmani's talk. And what we can see is we've got this vector here. And because this is the first talk, all the words kind of appear at the start of this vector, and then it starts going to be zeroes. And you can see that the text of that is the AI Assistive Developer Workflow by Addy Osmani. Cool.

If I did that to talk number eight, we'd actually see that's my talk, Bill Bradford and Sreć. And whilst it doesn't share all of those first words that are in Addy's talk, we can see that there are actually some crossovers there, which is kind of nice to know, especially because they are both indeed about the world of AI. So let's jump back to slides. And talk about comparing vectors.

It turns out that computers are obviously much better at comparing lists of numbers than they are at comparing the meaning of words. And so that makes these vectors useful. But how do we compare these vectors? Well, first up, what we have is a multidimensional vector. That is a sort of a point or a direction in a multidimensional space. Sadly for our brains, we're not very good at imagining things in more than three dimensions or four if you have time. So I'm just going to plot this down to two dimensions to make it a little bit easier to like consider how this works. And if we had three vectors on a two dimensional plane like this, then if we wanted to pick the ones that are most similar, we might think the ones that were sort of physically located near each other were more similar than ones further away.

But that's not actually how vectors work. They're not just a list of numbers. They're not a point in a multidimensional space. They are a direction and a magnitude. And that direction is from the origin of the space from 0, 0. And a magnitude, which is how sort of far away from the origin they are. And so a similar vector is actually one that points in the same direction as another. And so in this case, our similar vectors are actually the one closer to the origin and the one far out to the right. Because they are pointing in the same direction.

And whilst one is not quite, you know, not as strong, doesn't have quite as strong a magnitude as the other, they both have a similar meaning. It's just maybe one means more than... means more of it than another.

And so to calculate this, we use a bit of trigonometry. We're going to go back to school and some maths. We use cosine similarity. This gives me the opportunity to put a scary-looking mathematical function onto a slide. But ultimately it's not that scary, I promise. These big sigmas are just the sum of... and when you have a vector times a vector, we actually just multiply each element of the vector and then sum the lot.

So the dot product that's at the top there is two vectors as each element in a vector is multiplied by its pair. And then we sum up all of those multiplications. And at the bottom we have the magnitude. As a magnitude of vector a is a squared. So that's actually each item in vector a squared and summed tidally. And then the square root of that full sum. Same for vector b. And so this is actually... we can see this in code and actually makes a lot more sense at that point. So we start with the dot product. That's the top part of the function. We take two different vectors, vector a and vector b, and for each item in vector a we're going to multiply that item in vector a by the item in vector b. And ultimately we then sum up all those products. Fairly straightforward. The magnitude then is similar. It only takes one vector in this case. Oops. Only takes one vector in this case. And then for each item in that vector we square it and add it to the sum. And then we take the square root, math.square, of the sum.

And then finally our cosine similarity function just is the dot product of two vectors divided by the multiplication of the two magnitudes of each of the vector. And that gives us a similarity result. And so if we were to check this by doing a console.log Cosine similarity of text to vector talks zero. So this is Adi's talk. And actually let's do it against each cell. If we do that what we should find is the answer is very near to one. In fact it's just slightly over one due to floating point arithmetic, which is fine. But because it's the same talk it does mean exactly the same thing. It is similar in terms of one.

Now if we were to compare this to talk eight, that's my talk, we can see that their similarity is 0.135. Now we don't actually, because these vectors are very sparse vectors, most of the items in each of the vector for each of the talks are zero, we're not actually going to find very very similar talks in this particular case. If we could check and say what talking four is but we find that that's got similarity of 0.04. So talk four is much less similar to Adi Osmani's talk than my talk is, sure, but at least in terms of the words that it has. So that's our cosine similarity. It gives us that score between zero and one of how similar two talks are.

And if we pop back to the code, oh we see it works. Well, it works because we actually then have to make it available to the the actual bot that we've created. So this findSimilarTalk is a function that takes any query, it turns that into a vector based on our existing set of words that we've looked at, and then maps a cosine similarity over all of the other talks against this query, sorts by the most similar, and returns five talks by default. So if we do that this has actually set up this event chat, and in fact a search we have here to show us what's happening. And so what we can do is, right, we've got similarity search. I can look for what is Phil Nash speaking about. If we do that we can see that because the words Phil and Nash appear in only one of the talks, we get this kind of low similarity, 0.014. All the other things are not as similar at all, they don't mention Phil Nash or indeed the words speaking about. And so it's going to give us that answer. If we look at something like just AI and developers, we get a bunch of stuff like manual to magical AI and developer tooling, we've got integrate AI into your app architecture, developers integrating stuff, the AI-assisted developer workflow. So you can see these similarities are 0.45, 0.37, 0.3. That's really useful. So if we come back over to our actual chat where we put this context through the bot as well, and ask what is Phil Nash speaking. Let's keep capitals in there. What is Phil Nash speaking about? This is going to go do that search, find the talk and four others that are not similar, pass it to the model, allow the model to then come out with the actual right answer.

We're talking about building rag from scratch in JavaScript, excellent. And then who is speaking about AI and developers? We can see that's going to do that same kind of search and say that Toba Lundberg, Yossi Kallen, Adi Osmani, Daniel Ostrovsky, Satoshi Nakajima, all talking about AI and developers and how we can do that. Exciting talks, things I want to see, honestly.

So it does work. We've built ourselves a bad vector, but it works. It's bad because it's sparse, it's based on words, it's got a whole bunch of things. We need to do all that data up front. The query is sensitive to the vocabulary. So if we ask something that didn't use any of the words in any of the talks, we'll get zero kind of similarity.

If we were to expand upon this, if there were hundreds and thousands more talks to look at, we'd have to do many, many more calculations, which will eventually bring this to a halt. The same word can mean different things. Words could be negated to mean the opposite things. The order of words in a sentence could mean different things. And so a lot of this is not captured by our vector. But what it can be captured by are embedding models. These are actual machine learning models that are used in the creation of large language models as well. They kind of capture that meaning of words and of bodies of text.

Embedding models are made available by many of the kind of AI companies out there. There are some companies that just make embedding models for you. So this is OpenAI. You can quite easily call kind of against their API to get embeddings from OpenAI. In this case, text embedding 3 small model gives you 1536 dimensions and will always return that. Capturing the meaning of a piece of text in about 1500 vectors, there's a large model that has 372. You get to choose what you want there.

And then obviously what we did here was keep everything in memory, which is fine on a small set. But if you have a lot more data, you want to need a database for that. And so vector databases allow you to, well, they have indexes for doing this cosine similarity search against lots and lots of data. Much faster, much more efficient. AstroDB is one of those vector databases, the one that we create at Datastacks. And actually it has an additional help for you if you want to do kind of a vector search or find.

It will actually vectorize queries for you if you set it up to do so. So you don't have to worry about making two different calls. You can actually just allow the database to handle vectorizing things for you. And that's really cool.

I just want to show a quick demo in which I'm going to take out our bag of words model. That's what we were kind of dealing with there because it was based entirely on the words. Replace it with vector search, which is that query we just saw there. We're going to vectorize this going in. I already saved all these talks into AstroDB, and so we can now sort using our vectorize our query and get back a set of similarity stuff.

And so if I now go and have a look at what is Phil Ash speaking about, it's going to do that sort, do that search, and find that the similarity of that query, what is Phil Ash speaking about, to my talk is 0.72. Although it actually talks about it, has similarity to other things as well. If we look at AI and developers, we're going to get very similar scores to things like Addy's talk, like Tober Lindbergh's talk, as we kind of saw. And this is a much better similarity score, a much better capture of how it all works.

And so if I was to now ask what is Phil Ash speaking about, it's using that similarity still to just return what we've got. And if I say who is speaking, but AI and developers, we get that same list. But we now know we're actually using a real model for this. We're using a real database for this, so it's actually much more scalable and usable in the long term. And so that is RAG, that's Retrieval Augmented Generation.

What we did here was build a very poor idea of a vector that was actually useful for us to see, I think, how capturing meaning as a list of numbers is useful to us and how we can then compare them. We then replace that using actual machine learning models to do the meaning capture and using a database to do the sort in a much more efficient way. But there's much more to this. If you want to look into more things on RAG, then there are alternative ways of doing vector embeddings like Colbert. There are alternative ways of storing things in a database, like using a graph RAG system based on the content. I just shared some other blog posts I've written here, like one where I used similarity sorting to create related posts on a blog, in this case an Astro blog, Astro and Astra together, quite good fun. So that's all I've got for you. Again, my name's Phil Nash, I'm a Developer Relations Engineer at Datastacks. I hope you enjoy the rest of JSNation US. Again, find me online, I'm Phil Nash all over the place, or scan this QR code for the slides. I hope you can get out there and explore using RAG, using large language models and building with AI. Thank you so much.