Semantic Search through the Complete Wikipedia with Weaviate’s GraphQL API

So hello everybody. Thank you for taking the time to listen to this talk. We are going to talk about a few things. So first of all, we're going to talk about our database, our search engine, Weaviate, and we're going to use a demo data set, which is the complete Wikipedia to show how you can query it, and most importantly of course, we're going to talk about the GraphQL API that it has.

So weave is a vector search engine or database, it has a GraphQL API, and we're going to use it to demo to show you the demo data set of the complete Wikipedia. So first I will give a little bit of context about like what the vector search engine is, so that you understand what we're talking about, if it's new to you. Then we will look at the design of the GraphQL API. Then we'll go into a demo of the API on the data set. And last but not least, I'll show you how you can start it with Weaviate and its GraphQL API yourself.

So again, thanks for listening. So first of all, what is Weaviate and what is a vector search engine? So at the core, we're dealing with the problem of unstructured data. If you ever use a database or if you ever use a search engine, then you know that the data that you're storing, for example, if it is text, that you can only find it if you use keywords. So for example, in a traditional search engine, you have to, if you search for this data object for wine, for seafood, you will probably not find it because except for the key here, there's nowhere where you find the word wine in the data. The word for is not in there either and seafood is not in there either. So using a vector search engine and you would search wine for seafood, it would actually find the data object. And the reason why it's able to do that is because every data object that you add to the search engine is run through a machine learning model. The machine learning model creates factor representations and that's what you use to search to the database.

Now if this is new to you, then let me give you a little bit of context so that you know what's happening there. So, most machine learning models output vectors. And the easiest way to think about vectors are coordinates. So, for example, our first model had 300 dimensions and you had all these kinds of words in there. So the bulbs here represent words like meat, chicken, fish, etc. What you can do if you add a new data object, for example, the Chardonnay that's good with is that all these individual words that you see here highlighted in green are found in the vector space and they're placed in that same vector space. And what you can do is that you can give a unique centroid position to that data object. So, now you can say in the vector space the data object, in this case the Chardonnay, sits exactly here in the middle of where all these words sit. So now, if you search for wine related to seafood or those kind of things, you will actually be able to find that data object. It is not 100% match, but it's an approximation of what you're searching for. But in a bit, you will see what actually the value is of this. So, as you see here, we have the class Wine with property Covey run 2005 Chardonnay. It might be related to a beacon, and it might have certain vector weights.

So this is what the data object looks like when you store it in a Weaviate instance. Well, to help you work with this, we have the database which you see in the middle to store your objects to do vector search and to do filtering. But of course, there are many, many machine learning models that you can use to actually or vectorize the data or search through the data.

The demo that I'm going to give today is purely focusing on text. However, you could also do this for images or videos or any other data type. If you go a little bit deeper under the hood, you see how that works from an architectural point of view. So for example, we have text-to-vec modules or we have Q&A modules. They often run on a GPU. That's all running on your infrastructure.

These modules sit in the Weaviate core, then there's a persistence layer that's taking care of storing the vectors, being able to search through the vectors and to store the data object. But most important, there is an API on top of it. Of course, what we're going to focus on today is the GraphQL API and how you can leverage to search through your data.

First, before we do that, I want to talk a little bit about the design of the GraphQL API, because you have to know when we created the database, we didn't have an interface yet. We had to choose what language will we choose to query data. Will we just have a pure RESTful API? Will we adopt some kind of query language? Will we invent something of our own? Then we decided that the best for us was actually to use GraphQL. This is, in a tiny nutshell, our design. At the top, you have a core function within UEFI 8. We'll look at that in a bit. You have a class that you can add and add your data to. A class can be anything. Whatever data you can have, for example, if you have documents, you can just have a class document. If you have products, you can have a class product. Then you have the properties. A property can be also anything. So, for example, if we stay with the class product, then you might have the property name or the property price. You can, of course, make a cross-reference. Hence, it's a graph-like data model. Then we have these underscore additional properties. Those are properties that you get as part of searching for classes. But those are baked in in the modules or into Weave8 itself.

So, we have three core functions. So, get, explore, and aggregate. Get is the one that we'll be using the most, because that's how we find stuff in our dataset. Explore is to search through the complete vector space. This is often done if you don't know what your classes or properties are called. And aggregate is just, for example, how many data objects do I have, et cetera.

So — oh, of course, what you can also do is that for the class name, you can add search filters. So, here, for example, if you have the class article, the property title, you can add a near text filter that comes from the text-to-vect modules, and you say like, okay, I want to search for the concept. The concepts, in this case, housing prices. So that is at the root what the GraphQL design is like from Weave 8. And I think it's best to actually start to look at the demo because what's better than actually looking at it in action?

So, for all the demo datasets that we have, you have a console that you can go through console dot some other technology, and embedded in that console is among other things, GraphQL. That's something that we'll be using here. Of course, it's a database. So I need to select a database, a dataset. And the dataset that is here is actually Wikipedia. So this is open source. It's on GitHub. And it currently contains a little over 11 million articles, a little over 27 million paragraphs. And a little over 125 million cross-references. And this is the machine that it's running on. In this GitHub repo, you'll find all the information if you want to run the dataset yourself. But we also have a live demo that you can use.

So let's start very simple first. So I can say get. I can say paragraph. And I can say title, content, and order. And the reason that we structured it like this is because a paragraph is part of a article in Wikipedia. And a paragraph can have a title. It doesn't have to. It has content and it has an order.

So it's like, for example, the fourth paragraph. Now, if I run this query, very boring query, it says as much as just get me the first 25 paragraphs and show me the title, content, and order. Now, if we limit this, oh, apologies. If we limit this to the first result, I can do something like this. So I can say limit one, first result. And what's going to be interesting is how the machine learning model represents this data object. So what I'm now doing with this additional property is that I want to see the vector for this specific data object. So if I run it here, it returns the vector. A representation that is coming from the machine learning model. So this data object is run through the machine learning model. This is the vector representation that gives us and it's representing this. Now, if we go one step back, so I just remove the vector representation. We can also make a graph relation. So we can say in article. I can say on article, an article as a title. And now if I run this, you see where it's coming from. So it's coming from a file on Wikipedia. So this is how the data is structured. Or sorry, how the GraphQL API is structured. But of course, where it becomes very interesting is now if you're going to use GraphQL to enable the machine learning models to search through the data set. So let's do something like this. Let's start from the perspective of asking a question. So we can say ask question. And I could say how many people fit in on Airbus A380. I'm going to tell it that I wanted to find the answer in the content. So this is the content. And I'm going to limit that just to the first result, because I'm looking for a specific answer to a question. And then again, I can add the additional property to show me the answer. Yeah, I need result. So this query says like search through the paragraphs, try to answer the question, how many people fit in an Airbus A380? Use the content property to find it, limit it to the first result.

Show me the result of the answer. I also want to see the title of the paragraph, the content of the paragraph itself, the order in which this paragraph is showing on the page. And I want to see the actual Wikipedia page where this is coming from. So now running the query, and you see how fast was there. So it says like 656. And then you see here at launch in December 2006, 156 seat A380, 200, etc, etc. So that's how you see how that works and how it gets these results. You can find anything.

So anything that's in Wikipedia can search for. So I'm a big music fan, so I could say, for example, what was the name of Frank Zappa's first band? So same type of query. Let's keep everything the same here. Let's run this query. And here you see it's The Mothers of Invention, which happens to be the correct answer. I know. We cannot only do that for Q&A, but we can also do that for more generic questions.

So now let's remove the ask and the limit and say near text. I want to search the concepts, and let's go, for example, Italian food. No, I do not have an answer, but I can ask for certainty. OK, let's see what happens when we run this query. So what you see here is ... well, of course it comes from the article list of Italian dishes, that kind of makes sense. It is the first paragraph, so there's no text. And here you see all kinds of things about Italian dishes. And it's like almost 90% certain that this is the right result. But if we scroll down, you see that the number goes slightly down. It's still about Italian cuisine. But the further we go down, you see the more we get removed from the actual topic. So here the culture of Italy, the cuisine. Pasta, of course, pasta.

Pasta, of course, pasta. So one of the things that we could do, for example, is that we can say, well, we want to have at least 85% certainty of the result. So if I run this query again, and let me scroll all the way down. Then let's see what's the. So here we see Lombard cuisine from Milan, which is like 85% certain. So what you see, it's still about Italian food. But the further we go down, the further we scroll down, the further it is removed from the actual query that we have. And you can use any natural language query that you want to use. Important to mention, this demo data set is Wikipedia. But you can use any data that you have.

Last one I want to show is something related to the graph relations. So let's just limit this to the first result. So I'm now going to get Italian food from the list of Italian dishes. Now I can even say in article, on article title, I can say has... Oh, no. Sorry. Links to article on article. And I can say title again. So what this query does is it finds the data object for the paragraph. In this case, that's the first paragraph of the list of Italian dishes, which is the first graph relation. And then we're going to make another graph relation where we're going to say, OK, show me what data objects this data set is in turn linking to. So let's run that query and see what happens. So if you now go down, you see that the links to the article are pizza margherita, DOC, Italian cuisine, et cetera, et cetera, et cetera, et cetera.

So that's how, at the root, the GraphQL API of the Weaviate Vector Search Engine works. But I would like to do is I want to introduce you to the documentation. The easiest thing you can do is just google for Weaviate or go to our website, SAMI.Technology. You click Developer section, and you can't miss it. In the installation guide, you can click, for example, Customize Your Weaviate Setup. You can go to the customizer and install it yourself. Of course, last but not least, we have the GraphQL references. So if you scroll there, you see example queries. You can try them out in real time, and you also see the equivalent of these GraphQL queries in different programming languages.

So, thank you so much for listening to my talk. I hope you like it. I hope you'll give Weaviate a try. If you come to our website, you will find our slack. You will find the software documentation. Weaviate itself is on GitHub. If you like what you see, then, of course, a GitHub star is always appreciated. And you can also, of course, simply google for Weaviate, and you will find other videos, blog posts, software documentation, demo datasets, whatever you can think of. Thank you so much for listening, and I hope you'll enjoy the other talks, as well.