Llms Workshop: What They Are and How to Leverage Them

So, welcome everyone. Today we're going to be kind of looking at introduction to large language models. So quick introductions. I can go and then my colleague, Horace, can go. So my name is Nathan Mars and I'm the tech lead of the DataViz squad at Grafana Labs. And kind of been experimenting in this world of large language models and how it can apply in the world of observability for the past year and a half now. And I'm really excited about this space.

Horace, if you want to introduce yourself. Yeah, of course. Yeah, my name is Horace Rzajac. I'm also working at Grafana Labs. I'm a software engineer on the Explore team. And also, yeah, I've been playing with LLMs for about a year now and thinking about how to implement them at Grafana and some other personal projects.

So we're going to kick it off here. So to begin with, we're going to start with this high-level agenda here. We'll first begin with an introduction to large language models and then we're going to go over a few early applications of large language models within Grafana. Then we'll have a short five minute break, five minute or so break, and transition to our hands-on activity working with large language models that Horace will be leading. We're going to kick off. Let me add these people into the call.

All right. So first of all, I want to say huge credit goes to Andrej Karpathy for the majority of this content and structure of this intro material. A super helpful resource for creating this. So what is a large language model? It is in actuality just two files. In this example here, we're looking at a LLAMA 2 70B 70 billion model. This is a large language model released by Meta.ai. And this is a second iteration of the LLAMA model. LLAMA 2 has models ranging from 7 billion, 13 billion, and up to 70 billion parameters. 70 billion is the biggest one and many people like this model specifically because it's the most powerful open weights model. The weights and architecture of this model was all released by Meta. So anyone can work on this model by themselves. This is unlike many other language models that you might be familiar with. For example, if you're using chatTBT or something like this, that model's architecture was never released and is owned by open AI. You're allowed to use it through a web interface, but you don't have actual access to the underlying model itself. A quick update in the LLAMA world, though. Meta actually released LLAMA 3 a few months ago. And this has an 8 billion and 70 billion model variant. The underlying concepts we're going to be discussing and exploring today apply to both LLAMA 2 and 3 as well as the other large language models out there today.

So in this case, the LLAMA 2 70b model is really just two files on your system. There's a parameters file and then the run file, which is just some code that runs those parameters. We'll take a high level look at the run file in a few minutes. The parameters are basically just the weights of the neural network that is the language model. Every one of those parameters is stored as two bytes. And because this is a 70 billion parameter model, the parameters file is 140 gigabytes. Each parameter is two bytes because they're float 16 numbers. And this is a pretty standard data format that gives just enough precision at the cost of how much data it takes to store and to run. How much memory it takes to run the model. So you have the parameters file. You also need something that runs the neural network. And this piece of code is implemented in our run file. This could be a C file or Python file or any other programming language really.

You can do in JavaScript if you want to. It can be written in any arbitrary language, but C is a very simple and efficient language from a memory standpoint, especially. So just to give you a sense, it would only require about 500 lines of C with no other dependencies to implement the neural network architecture that uses the parameters to run the model. So you can take these two files and you can take your MacBook or whatever computer you have and use the model. This is a fully self-contained package that has everything that's necessary. You don't need any connectivity to the Internet or anything else. You can compile your C code and you get a binary that you can just point at the parameters and you can talk to this large language model.

So in this example, you can send it a prompt to write a poem about this conference and get the language model generating text back. So in this case, I'll keep pulling it over and over here, I gave it the directions, gave a poem about React Summit. It's running the model and we can see it outputs some text about a poem. So this is a very small package, but the computational complexity really comes in with how these parameters actually formed. The run C file does not contain any surprises and everything is algorithmically understood and open. This is an actual look inside of a run C file. You can see it's kind of a basic C file. It's basically the core logic to load and run the model. Here's kind of an abstraction of what's going on here. So the LLAMA2 model and other models use matrix multiplication to performance computations. The model has a large number of parameters which are stored in matrices. The model performs matrix multiplication on these parameters to generate the output text. And this run C program is a relatively simple program. It consists of a main function and a few helper functions. The main function is responsible for reading the input text file, calling the model to generate the output text and writing the output text to the output text file. The helper functions perform various tasks such as reading and writing files, allocating memory, which is a huge piece of this, and performing matrix multiplication.

So the run C file is not really hiding any real mysteries, but the parameters themselves are quite complex. So the big question here is how do we get these parameters and where do they come from? So to obtain the parameters, basically the model, this is called like model training. It's a lot more involved than model inference, which is the process we were just discussing earlier with the run C file. Model inference is just running it on your Macbook.

So model training is a very involved process. Basically what we're doing can best be understood as kind of a compression of a good chunk of Internet. Because LLAMA2 70b model is open source, we know quite a bit about how it's trained. So LLAMA2 70b is meant to release this information in a paper.

So these are the numbers, some of the numbers of what's involved. You basically take a chunk of Internet that is roughly 10 terabytes of text. That's a lot of text. This typically comes from a crawl of Internet. So just kind of imagine collecting tons of text from all different kinds of websites and collecting all together. You then take this large chunk of Internet and you procure a GPU cluster. Those are not cheap. These are very specialized computers intended for heavy computational workloads like training of neural networks. You need about 6,000 specialized GPUs and you'd run those for about 12 days to get a LLAMA2 70b model. This costs around $2 million.

And this is part of the reason why NVIDIA is worth so much today. So what's this training is basically doing is you can think of it as like compressing this large chunk of text into what you can kind of think of a zip file at a high level. These parameters I showed you in the earlier slide are best kind of thought as a zip file, right? Because a zip file of the Internet. And in this case, what would come out of those parameters is a 140 gigabyte parameter file. So you're taking 10 terabytes of raw Internet text and it's being kind of compressed down into 140 gigabytes. So it's kind of a compression ratio of roughly 100 times. But this is not exactly a zip file because zip files have lossless compression. What's actually happening here is lossy compression. We're just kind of getting an approximation of the text that we trained on and we don't have an identical copy of the parameters. So it's kind of lossy. One more thing to point out here is that these numbers are actually by today's standards, you know, kind of rookie numbers. If you want to think of like state of the art neural networks, like what you might use with chat, GPT, BARD, Cloud, or any others, these numbers are off by a factor of 10 or more. So to get an idea of the cost of training those, you probably need to start multiplying by 10 to 20 times the cost of 2 million, right? And that's why training runs today are in the tens or potentially even hundreds of millions of dollars for very large clusters because they're dealing with very, very large data sets. And the process here is very involved to get those parameters. For instance, the latest GBT 4.0 model that was released has 175 billion parameters roughly compared to 70. So you can only imagine how much more data that is.

So once you get these parameters running, the neural network is fairly computationally cheap. So what is this neural network really doing? I mentioned that there's these parameters in the neural network. They're basically trying to predict the next word in a sequence. You can feed in a sequence of boards, or they're really called tokens. For example, cat sat on A. This feeds into the neural network, neural net, and these parameters are dispersed throughout the neural network, and there's neurons, and they're connected to each other in a way, and they all fire in a certain way that we don't quite understand. The model returns a prediction for what word comes next.

So for example, in this case, the neural network might predict that in this context of these four words, the next word should be mat with a 97% probability. So this is really fundamentally the problem that the neural network is solving for. We can mathematically show that there's a close relationship between prediction and compression, which is why I sort of allude to this training of the neural network is looking at compression in the internet, because it can predict the next word very accurately. So it's just a next-word prediction neural network, and you give it some words, and it'll give you the next word in that sequence.

That being said, though, the reason why this training actually outputs something kind of magical or interesting is because you kind of think the next-word prediction task is kind of a basic thing, but it's actually a pretty powerful objective because it forces you to learn a lot about the world inside of the parameters of the neural network itself. So here I took a random web page from the main page of Wikipedia that's about Oppenheimer. Think about being in the neural network, and you're given some amount of words, and you're trying to predict the next word in a sequence. In this case, I'm highlighting in red here some of the words that contain a lot of information about the world around these words. So for example, if your objective is to predict the next word, presumably your parameters have to learn a lot of this knowledge.

So you have to know about Oppenheimer, who he — when he was born, when he died, who he was, what he's done, and so on. And so in the next-word prediction task, you're actually learning a lot about the world and all the knowledge that's being — and all this knowledge is being compressed into the weights of these parameters.

So how do we actually use these neural networks once we've trained them? I showed you that model inference is a very simple process where we basically generate the word that comes next. So what happens is we pick a word, and we continue feeding it back into the model until we get the next word, and then continue feeding it back so we can iterate this process, and then the network kind of dreams up internet documents.

So for example, if we just run the neural network and perform inference, we'd get some web page dreams. You can almost think about this in this way because this network was trained on web pages, and then it just kind of dreams up web pages as a direct result without any fine-tuning. So for example, if we just run the neural network and perform inference, we'd get some web page dreams. You can almost think about this in this way because this network was trained on web pages, and then it just kind of dreams up web pages as a direct result without any fine-tuning.

Focusing a bit on the middle example, there's a title, there's an author, there's an ISPN number. All this information was totally made up by the network. The network is dreaming this text based off the distribution that it was trained on. And it's kind of mimicking these documents, and it's all kind of elucidated. So for example, the ISPN number, this number most likely does not exist, right? The model just kind of knows that what comes after ISPN colon is some kind of number that's roughly this length, has all these digits, and it just kind of puts it in there, and it's like, yeah, this looks good. No one will notice that this is not real.

And on the right here, there's a black nosed dace I looked up. It's kind of a fish. What's happening here is that this text verbatim is not actually, you know, found on the internet like verbatim, but it's roughly correct. So it's not going to exactly parrot the documents, the stringing documents, but it will get roughly the right information down. But you have to remember, again, it's kind of a lossy compression of the internet. So it just kind of remembers this high level summary, and then just kind of goes with this knowledge that it feels that it has. And you're never 100% sure what it comes up with is what we call a hallucination or a fully correct answer. Some of this stuff can be memorized, and some of it could be just making up because it sounds good. For the most part, this is just kind of a hallucinating and dreaming internet text from its state of distribution.

Some of this stuff can be memorized, and some of it could be just making up because it sounds good. For the most part, this is just kind of a hallucinating and dreaming internet text from its state of distribution.

So this is like the base model, right? So let's switch gears. How does this network work? And let's look into how this network actually works. And how does it actually perform this next word prediction task and what's going on inside it at a high level.

So this is where things get complicated a little bit. This is kind of a schematic diagram of a neural network. If we zoom into this toy diagram of the neural net, this is what we call the transformer neural network architecture. So what's remarkable about these neural nets is that we actually understand in full detail the architecture. We know exactly what mathematical operations happen at all the different stages of it. The problem is that in, let's say, a hundred billion parameter network, these parameters all disperse throughout the entire network. So there's basically billions of parameters throughout this neural net, and all we know is how to adjust these parameters iteratively to make the network as a whole better at the next word prediction task.

So we know how to optimize them. We know how to adjust them over time to get better next word prediction, but we don't actually really know what these hundred billion parameters are doing in a given time during inference and how they're interacting. We can measure that it's getting better at the next word prediction, but we also don't know how they collaborate, which is why it's a black box.

We do have some kind of models that you can try to think through at a high level for what the network might be doing. We can understand that they build and maintain some sort of knowledge database, but this knowledge database is very strange and imperfect and weird. A viral example of this is what we call the reversal curse. So as an example, if you were to talk to a chat GPT, GPT-4, one of the best language models currently available, and you'd say, who is Tom Cruise's mother, it'll tell you it's Mary Lee Pfeiffer, which is correct. But if you ask who is Mary Lee Pfeiffer's son, it'll tell you it doesn't know.

So it's a weird, strange knowledge database that is definitely not perfect. Fundamentally, we don't really know, because all you can kind of really measure is whether or not it works or not, and with what probability. So you can kind of think of LLMs, large language models, as kind of mostly inscrutable artifacts. They're not similar to anything else you might build in engineering discipline. They're like a car where we understand all the different parts and how they work together.

So you can kind of think of LLMs, large language models, as kind of mostly inscrutable artifacts. They're not similar to anything else you might build in engineering discipline. They're like a car where we understand all the different parts and how they work together.

They're these neural nets that come from a long process of optimization. And so we currently understand exactly, we don't currently understand exactly how they work, but there's some research obviously happening in this space called interpretability or mechanistic interpretability, trying to go in and figure out which each piece of these neural nets are doing. But it's still early days there.

So right now we can kind of treat these LLMs as mostly empirical artifacts. We just give them some inputs and measure the outputs. We can basically measure their behavior. We can look at the text we generate in many different situations. And we can, in this case, we kind of have to come up with some sophisticated evaluations to work with these models, because they're mostly empirical, because we only have the control over input and how we measure the output.

So far we've been talking about just like a base model that will kind of dream up base internet documents. Now let's actually go into how we obtain an assistant. This is something you're probably more familiar with.

So as I said so far, we've only talked about these internet document generators, which is a result of the first stage of training, which is what we call pre-training. And now we're moving into the second stage of training, which is what we call fine tuning.

This is where we obtain what we call an assistant model, because we don't actually really just want document generators, as that's not super helpful for many tasks. We want to give questions to the model, and we want it to generate answers based on the questions that we really want an assistant, based on questions. So we really want an assistant model instead.

The way you obtain these assistant models is fundamentally through a following process. We keep the optimization identical, so the training will be the same. It's just a next word prediction task, but we're going to swap out the dataset on how we're training it. So it used to be that we're training on a huge amount of internet documents. We're going to swap this out for a training set that we collect manually by using a lot of people.

So typically a company will hire people and they'll give them labeling instructions, and they'll have all these people come up with questions and then write answers to them. So here's an example that might make it into your training set. There's a user and it says something like, can you write a short introduction about the relevance of this term in economics, and so on. And then the assistant, and again, the person, fills in what the ideal response would be to this question. The ideal response and how that is specified and what it should look like all just comes from labeling documentation. Labeling instructions basically that are provided by the maintainers of the model, such as companies like OpenAI.

Now the pre-training stage is about a large quantity of text and potentially low quality because it just comes from the internet and there's tens or hundreds of terabytes of data. And it's just all very high quality. You can't ensure that. The second stage, however, though, we prefer quality over quantity. So we may have many fewer documents, for example, only a hundred thousand, but all these documents are now conversations and they should be very high quality conversations. Fundamentally, people create these conversations based on labeling instructions.

So we swap out the data set and now we train on these Q&A documents, which is a process we call fine tuning the model. So once you fine tune the model, you obtain what we call an assistant model. This assistant model now subscribes to the form of its new training documents. For example, if you give it a question, can you help me with this code? It seems like there's a bug in print hello world. So even though this question was not in the training set, the document, the fine tuning training set, the model is understanding that it should answer this question in a helpful manner. And it will answer by sampling word by word from left to right, top to bottom. All the words that are in response to this query, it's kind of remarkable. And it's also kind of empirical and not fully understood that these models are able to sort of like change their behavior, their formatting to now be helpful assistance because they've seen so many documents of it during the fine tuning stage. But they're still somehow able to access the knowledge from the pre-training stage.

So roughly speaking, pre-training stage is about training on a ton of intranet. It's about knowledge and the fine training stage, the fine tuning stage is about what we call alignment. It's about sort of like changing the format from intranet document generator to a question answered document assistant model. So the question here is like, does fine tuning prevent dreams and hallucinations? Answer here is like no. Fine tuning is kind of directs these dreams into a helpful assistant dreams instead of internet document dreams. So word of caution here, you need to always be careful with what LLMs tell you and always double check the results. And if you need up to date data, probably provide that as relevant context to the model.

So there's actually a next step in fine tuning called comparisons, comparison labels. The reason that we can do this is in many cases, it's much easier to compare candidate answers or model answers than write the answer yourself if you're a human labeler. So we can think of this example, like with creative tasks like running IQ. Suppose the question is to write a IQ about grafana or something like this. From the perspective of a labeler, if I'm asked to write an IQ, that's kind of a difficult task, at least for me. I might be able to write an IQ and yeah, it takes a lot of time. But suppose we ask the model to come up with a few candidate IQs from the stage two of the model. Well, then as a human labeler, I look at these IQs and then kind of pick one that is the best. And this is a lot easier to do comparison than generating from scratch.

At OpenAI, this process is called reinforcement learning from human feedback or RLHF. And this is kind of an optional stage three that can gain additional performance in these language models. Also really want to show you a brief view of some of the labeling instructions that OpenAI has used for fine tuning. This is an excerpt from the paper instructGBT by OpenAI. And it just kind of shows you what they're asking people. They're asking people to be helpful, truthful and harmless in creating these answers and question answer documents. These labeling documents, though, can grow to tens or hundreds of pages and can be pretty complicated. But this is like roughly speaking what they look like.

So I've described the labeling process kind of naively here as humans doing all this manual work. That's not exactly right in the current day and age and it's increasingly less the case that it's all human manual labor. That's because these large language models are simultaneously getting a lot better and you can basically use like this human machine collaboration to create these labels with increasing efficiency and correctness. So for example, you can get these language models to sample answers and then people kind of cherry pick the best parts of the answer and to create the one sort of single best answer. Or you can ask these models to try to check your work or you can ask them to create comparisons and then you're just kind of like a supervisor role over the model. And there's even cases where you take a more advanced model like chbt4o, have it come up with training data to train a simpler model, to fine tune a simpler like a less large model. So maybe one day humans may not be involved, but we're still involved for now. So that's good.

So we're going to kind of use a review slide to kind of all this material we covered in a pretty quick pace here. So roughly speaking, there are two major parts of obtaining something like chbt. There's the stage one pre-training and stage two fine tuning. In the pre-training stage you get a ton of text from the internet and you need a huge cluster of GPUs. They're special purpose chips for these kind of parallel processing workloads and they're not just things you can buy at Best Buy and they're incredibly expensive and unlimited. Then you compress all this text into a null network that consists of parameters based on the compressed text. Typically this could be several millions of dollars and a lot more than this for the state of the art models. This gives you the base model. Because this is a very computationally expensive part of this process, it only happens in companies maybe once a year or once every several months, depending on the company.

So once you have this base model, you enter the fine tuning stage, which is computationally a lot cheaper. In this stage, you write out some labeling instructions and you basically specify how your assistant should behave. You hire people to create documents according to your labeling instructions. You collect let's say a hundred thousand high-quality example Q&A responses and then you'd fine tune the base model on this data. This is a lot cheaper and would only potentially take a day or so to fine tune the model instead of a few months. This is something you can come up with an assistant model relatively quickly, depending on how quickly you can get this high-quality Q&A data. Then you run evaluations, you deploy, you monitor, collect misbehaviors. For every misbehavior, you want to fix this. The way you fix misbehaviors, basically the model responding in a way you don't want it to, is you have a conversation where your assistant gives an incorrect response and you take that and you ask the person to fill in the correct response. Then the person overwrites this response with the correct one and then it's inserted as part of an example of your training data. So the next time you do a fine tuning stage, the model will improve in that situation. That's the iterative process by which you improve on this. Because fine tuning is a lot cheaper, you can do this every week or every day or so. Companies will often iterate a lot faster in the fine tuning stage instead of on the pre-training stage.

One other thing I'd like to point out, for example, I mentioned the LLAMA 2 series. The LLAMA 2 series actually contains both the base model, so from stage one, as well as assistant models, as well as LLAMA 3. The base model is not directly usable because it doesn't answer questions with answers. If you give it questions, it'll just give you more questions, or it'll do something like that because this is an internet document sampler, so by itself it's not super helpful. But where having access to this base model is really helpful is that Meta has done the very expensive part of these two stages. They've done stage one, and they've given you results so you can go off and do your own fine tuning. That gives you a ton of freedom with what you can do with this model. Meta, in addition, has also released the assistant models, so if you just like to have a question and answer assistant model, you can use that and talk to it.

So finally, I want to show you guys a leaderboard of the current leading larger language models out there. So this example from Chatbot Arena, it is managed by the team at Berkeley, and what they do here is they rank different language models by their ELO rating. The way you calculate ELO is very similar to how you calculate it in chess. So different chess players play against each other, and depending on the win rate, against each other, you can calculate their ELO scores. So you can do the exact same thing with language models. So you can go into the language model website or how you interact with it, enter some questions, view responses from two models, and you don't know what models they were generated from, and you pick the winner. So it's kind of like a blind race. And then depending on who wins or loses, you can kind of calculate the ELO scores. So the higher is better. So what you see here is that crowding at the top are the proprietary models. These are closed models, you don't have access to the weights, and they're usually behind a web interface. So this is, you know, GPT series, OpenAI, Gemini from Google, and Cloud from Antrofit. And there's a few other large language models from other companies as well. And so the best performing model rate here on top is GPT-40, and as I mentioned earlier, this has over 175 billion parameters compared to LLAMA-2-3-70. So that's a whole lot more, it's a lot more to deal with, a lot more computationally expensive to train. And then a fair ways below this, you're going to start to see some models that are open weights. So these weights are available. By weights, we're talking about the parameter weights, right, of the model. So a lot more is known about them, and there's papers available.

So this is a case for LLAMA-3-70b model that comes in 12th place as of June 2024. But roughly speaking today, what we're seeing in the ecosystem is that closed models work a lot better, but you can't really work to, like, fine tune the base model and download them and, like, go deeper inside of them in terms of what their parameter weights are. So all you can do is kind of use them through a web interface. And then behind all these proprietary models are the open source models and the entire open source ecosystem. And so while the open source models are not ranked as highly, depending on your application, they might be good enough, and you can actually run them, you know, in your own, you can infer them in your own compute. So it might be cheaper to do this. And currently, I would say the open source ecosystem is trying to boost performance and kind of chase the proprietary ecosystems. And that's kind of roughly the dynamic that we see today in the industry. And it's still pretty early days to see who wins out.

So with that, we're going to move into Grafana's approach to using large language models so far and how we've kind of leveraged these capabilities inside of Grafana. So first, before we dive in, like, what is Grafana? If you're not familiar with it. So Grafana is a leading open source platform for monitoring observability. It was first developed in 2013 by Torco Ortegaard and is now maintained by Grafana Labs. Grafana's mission is to make it easy for everyone to understand and monitor their data. So Grafana's core strategy is having a kind of this big tent philosophy supporting as many data sources as possible and eliminating vendor lock-in. This big tent empowers our users to query, visualize, alert on, and understand their data no matter where it's stored. And with Grafana, you can create, explore, and share all your data through beautiful and flexible dashboards. So Grafana is used by organizations of all sizes to track performance, identify problems, and make informed decisions. And it's a powerful tool that can help organizations improve their efficiency and productivity.

So what I'm very excited about Grafana is, in addition to its commercial use, is it's very popular with hobbyists and enthusiasts. Many people use Grafana for a wide variety of use cases, from monitoring their home solar power, tracking metrics of their IoT smart devices, or even creating a daily commute assistant dashboard, which is what this person did. In this example, the dashboard is drawing from APIs and GTFS real-time feeds to pull in data from weather.gov, the Metropolitan Atlanta Transit Authority, this is Atlanta, Georgia, in the U.S., and drive time and traffic information from Google Maps and Bing Maps. So it's kind of like this hodgepodge of data coming in to inform this individual's daily commute. And so he's also using Bing Maps here. So if you're wondering if Bing has a map service that's being used, you found your answer. Someone is using it. That's cool, too. So Grafana's flexibility and easy use makes it a great choice for hobbyists that want to have deeper insight in their data without any real limitations. And we've seen Grafana used in the wild from a myriad of use cases, from monitoring popcorn levels in a theme park, which is an interesting one, to even helping Japan land on the moon, which is just like their Grafana dashboard they're looking at in that livestream, which is cool. All right. So infomercial's over. So now we have a kind of a baseline understanding of what Grafana is. Let's move on to three examples of how we're considering leveraging large language models in the observability space. The first one we have here is the dashboard assistant. So this is pretty close to my heart as I helped build it out and learned a lot doing so. In this case, you can see generative AI, our large language model, is being used as a helper to add titles and descriptions to panels and dashboards. So it's a great example of like kind of a small assist that can reduce toil and quickly add real value. But it's not necessarily like a game changer, but it helps engineers and people working with Grafana overcome small hurdles in the product, especially when it comes to naming something. That can be pretty hard. And this actually does have real value, as it can make your dashboards more robust and consumable. So I want to... This is kind of unique to Grafana. I want to show you a small look behind the scenes of the dashboard assistant. So Grafana itself is an open source project, a visualization platform. And we really do like building the open and the code written for this is actually open source inside the Grafana repo, so we can share this. So here we're looking at the prompt that we're providing the large language model to generate the panel title. And we can see here that we're using the dashboard title and description to kind of influence how the title is generated. To help... This is like providing the model helpful context that have a reasonable answer to be outputted. So next off, we have generating incident summaries.

So basically we're taking the context of a declared incident. So let's say one of your CPUs in your server is above a certain threshold, it's alerting particular incident on this, so people can address it and get pinged and make sure the service stays up. It basically can take all the historical context of this incident and then have the large language model output the summary of everything that occurred to save you time. So it's another example of kind of reducing toil to the user, similar to the panel title and description case, just want to make the user's life a lot easier.

And then the final feature we'll cover today, it's called Flame Graph AI. So this feature helps you understand your flame graph profiling data. Flame graphs provide, you know, kind of a convenient way to visualize performance data, but not everyone really knows how to read them or get value from them. So flame graph AI kind of closes this gap and makes it more accessible to a wider audience. And it does this by highlighting performance bottlenecks, providing also causes, and even includes how you might resolve these issues, which is pretty interesting. So in this case, it's actually not really solving a toil case, or like a kind of a chore case, it's more of a domain knowledge case, where sometimes in the observability world, there's some pretty deep knowledge you need to have in order to be able to read a chart or really get value from them. So how can we use AI, like in this case, to help gap this and make it more accessible for more people.

So with this, that's all we have for the first part. So we're gonna have a short break here, because there's a lot of content here in the first kind of 45 minutes. A heads up here, if you don't already have an account set up with code sandbox.io, this would be a great time to set this up, as this is going to be required for the hands on activity. It's not just going to be all lecture, right? We're going to have people actually go in and try to play around with this large language model. So if you can please set up an account with code sandbox.io, that'd be great. And with that, I'm going to switch screen share over to Horace, and we'll put up a five minute timer, and then we'll resume. Hello, everyone. Welcome back. So in this hands on activity, we'll have a small React app that interacts with LLM models through OpenAI's chat completion API. So it's tricky to make this activity interactive remotely. So please feel free to ask any questions or interrupt at any point. So first, let's go over the OpenAI chat completion API. Maybe let's see if you can give a thumbs up emoji if you've used OpenAI's API. So this is great. So no one has used this. So we can go over it and show how it works and how it will interact with the React application. So I can actually share this in the chat. So there's a repository on GitHub. So this is a readme file that I want to go over really quick, just go over some of the parameters that completion API accepts. It accepts a lot of different parameters. But in this tutorial, we'll mainly use like four most basic parameters. So first, we have a model. I think most of you are familiar with this, and Nathan spoke about it. So the model parameters specifies the model to use for generating the response. And here are the different models that OpenAI provides, so GPT 4.0, GPT 4, GPT 3.5. Then the next one is max tokens. So this parameter controls how many tokens model will generate. So the more tokens, the longer the generated text will be. And the length of generated text depends on the length of the input. So if your max tokens are set to like say 256, but you pass 56 in the input, then the max tokens and the response will be 200. So you might want to limit the input to a set number of tokens. And there are libraries that can help with this like GPT tokenizer that is available in Python, and also as a node package in JavaScript. A token can be as short as one character or as long as one word in English. An important thing to note is that OpenAI will charge you for both input and output tokens. Then the next one is temperature. So the temperature parameter influences the randomness of generated responses. So possible values are between 0 and 2, so lower value such as 0.2 makes the answers more precise, while a higher value like 0.8 makes them more creative. So if you are, for example, building a customer support bot, a lower temperature would ensure the responses are accurate and on point. But if you are building a creative writing aid, a higher temperature would help generate more creative and diverse ideas for stories and dialogues.

Finally, we have messages parameter. So the messages basically is an array of message objects that you pass, each having a role. And there are three roles that OpenAI supports. So a system role, which is basically high level instructions on how a model should behave. So basically saying to a model, you are a chef. The next role is user. So message is generated by the user, such as queries or prompts. So basically you can tell it, you know, generate a pizza name using the keyboard unicorn. And then finally, assistant. This is the model's response. So the content that model gives us as an output.

So now let's take a look at our application that we'll be working on today. So this application basically lets you choose a model, set temperature and max tokens, and then you can select the keyboard and the ingredients of the model will generate some names. So here, for example, we can select the keyword and then we can select multiple ingredients. And then if we generate, we get a name. So that's the application.

And now it's basically story time. So let's say we are basically this application is pizza name generators. So say we are starting a pizza shop and we need to come up with a menu and we will use LLMs to help us with this. And at the end, we'll even have a small competition and vote the best pizza name. So let's do this and we'll also look at the code behind this. So for this, here's the link to the sandbox. And this is on the code box IO. So you'll need, hopefully you were able to set that up. Basically, there's a QR code too if you're using a mobile device. But I'm pretty sure everyone is on their computers. So basically, you'll have to fork this sandbox repository to be able to edit the code. And then just in the chat, let me know if you run into any issues. And if you can't set it up, you can follow along and set it up later. Ars might be helpful too to kind of show where the fork button is. Oh, OK. If people have an account. Yeah, so here is, so you should be able to see this. And the fork button is up here, kind of top right. Once you're logged in. Yeah, once you're logged in. Yeah. Let's start by looking at the network tab. This is Chrome DevTools, and check what we are sending to the endpoint. This is good. So I'm going to just, again, select some ingredients and generate the name. And here's the completions API that I'm hitting. This is the payload we're sending. So we are sending max tokens to 256. Here's our messages array that has the first role is the role of the system. Because here we are setting a behavior off the model. So we're saying you're a marketing specialist. Your goal is to create fantasy-like pizza names. And then we're giving further input, kind of telling it what those names should look like. And we'll look into that later.

And then the next role is the user, where we're actually telling the model, giving it a prompt, saying generate a pizza name using the keyboard from the input. So there's the unicorn keyword and the following ingredients that I selected. And then the model's GPT 3.5, and the temperature is 0.5. So that's what we're sending. And then in the response, we have several fields. We have ID, when the response was created, when we got it back. There's a model information. And here's the choices. So choices field is an array of objects, meaning that we could get multiple choices back if we wanted to. And the response here is this field. So message, so the role is the assistant. And then the actual output is the, or the generated name of the pizza is in the content field. And then we also have information about the usage. So we send 102 tokens in the prompt, and then we receive back six completion tokens. And so the total tokens is 108. And that's basically what we will be charged for.

Going back to the requests, you might be wondering how do I configure requests like this. And so for that, we can actually jump in the code. We have this in the React application. We have this generate function, which executes a simple HTTP request. So we use fetch function. We pass endpoint to it. So this is the OpenAPI's endpoint. The method is we post. And then in the headers, we pass the content type, which is JSON. And then we pass the API key in the authorization header. So API key is up here. And so you can use this API key, but it will be revoked after the exercise. And just to note, yeah, this is a very bad practice to hard code it in. But I wanted to reduce the friction and not have you click on the Authorize button, but just always have this API key available in the state. So going back to the generate function, so in the body, we're passing a JSON string where we have our parameters. So the model, max tokens, temperature, and messages. And then we get the response. And as I showed in the Network tab, we have to parse through the response. So there's that choices array where we select the first object and get the content. And then we use state hook to set the generated name to the response we got from the endpoint.

Let's test the, let's play a little bit with the temperature. So as we said, lower temperature will make answer more consistent and not creative. So let's, if we set temperature to 0, and select some ingredients and generate a name, then if we keep generating, it gives us pretty much the same answer. But now, if we set it to 2, let's see what happens. OK. Wow. So this is kind of strange. That's where we're getting some very strange response and string. So what happens here, and Nathan can jump in here too, is that basically, the temperature flattens the probability distribution of the next token, and so making it more likely for token to choose less probable tokens, which can disrupt the contextual flow and coherence of the generated text. Yeah, and so I can jump in here too. Like, you can almost think of this as the model is kind of reverting back to its base model here, instead of the question answer model, it's kind of going back to its internet document generator ways, because the tolerance of what token is accepted, like what probability rate is accepted, is so low now. And as soon as you have one rogue token accepted, it gets fed back into the model as it's producing these results. And so we'll kind of go haywire. So temperature two is probably not advice, unless you like chaos. Yeah. So yeah, thank you, Nathan. So yeah, for more creative answers, but still not chaotic, probably temperature one is good. Let's try that. Yeah, so that gives more creativity, but yeah, not chaos.

OK, now let's see what happens when we add a parameter to our request. Because earlier I mentioned how in the response we get this choices array. So basically, if we want more choices in a single request, we can pass n parameter, which tells the model to give us multiple choices. So we'll say here, give me three additional choices. So if I run this, let's see if I save that. So let's see what we get in response. So yeah, so now we have three different objects with different, each has a different pizza name, so Unicorns Delight, Magical Hamlets, and Unichokes Delight. And so one note here, too, is that you will be charged for everything provided in the choices. So you'll be charged for all three outputs from OpenAPI. And one useful thing here is that you could use the answers provided. You could provide it to the user and have it pick the best choice and then feed that back to the model so the model can know what the best choice is.

OK, now let's look a little bit at the messages array. So here we're passing, we have these two functions, getMessages and getFeedbackMessages. So let's look at the getMessages. So getMessages is a function that returns an array of two objects. So first is the system, and this is the default prompt. This is the setting, the behavior of the model. So we're telling it, you know, you're a marketing specialist. Your goal is to create fantasy-like pizza names. And then, you know, the pizza name should be creative. It should not be longer than three words. It should not include the word pizza. And it should mention a diminutive of one of the user's ingredients. So that's getMessages. And then on top of that, we have getFeedbackMessages, which also returns an array, but with the role user, but only if we already have a generated name and if a user has provided feedback. And here we're saying, you know, your previous response was whatever generated name was, and then the user has provided the following feedback from the user feedback field. And then basically, we're telling the model, regenerate your response according to the provided feedback. So in here, I first have to generate a name. Then I have this improve field. And I can tell it, you know, make pizza names appealing to little kids. You know, and it's adding magical, sparkling, sweetie. But we could also tell it, you know, make pizza names in Spanish. And now it's translating them into Spanish. OK, also, maybe you've noticed that we get multiple names here. So let's see. Let's reset. And when we generate, oh, it's actually giving. Let's see, maybe I. Yeah, so eventually, yeah, so here. So now we're getting, you know, three names. Yeah, in most cases, we get one name. But yeah, occasionally, we'll get multiple. And so the question for you is, like, why is this the case? Where is the bug? Does anyone know? So feel free to drop the answer in the chat or turn on your mic. OK, I can explain. I can explain why this happens. So the bug is basically in the prompt. And this is where we can talk a little bit about prompt engineering and, like, why is it important. So the current request here is saying, your goal is to create fantasy-like pizza names. And this is why sometimes we'll get more names than only one. But if we switch this to, say, create one fantasy-like pizza name, then it should be consistent and always give us actual name. And you remember how, in the past, it used to have, like, it was giving, like, an ordered list or, like, a numbered list. So it was, like, adding numbers to the title. But now it's pretty consistent. And finally, we can look at some, maybe one interesting parameter. So that parameter is seed. And maybe you may know, like, games like Minecraft has seed. And basically, if you provide same seed, same seed value, that value will generate the same exact word, world, consistently. So this parameter here is in beta. So it helps with determinism. So it can determine how model is run. So if you have models hosted in multiple locations, models, the same model hosted somewhere else can respond differently.

So just as different hardware or software environments can affect Minecraft's world generation, slightly, different computational environments for the API can lead to minor differences in the output, even with the same seed. So seed still doesn't guarantee determinism, but it can help.

OK. Now we can, well, we'll do a little challenge. So the challenge is to come up with the best pizza name. The rules are, there are no rules. You can modify the prompt. Feel free to play. And then you can submit your best pizza names. And we'll come through and select the top four to vote on. And then also, feel free to ask any questions along the way.

That's an interesting one. HoraceKitty's Big Key Quest, how'd you get that one? Just increase temperature a little bit. And yeah, that'll do it. Yeah. I'm picking something up here. We'll share in chat in a minute. But I want to see if you can guess how it's going to go. How I got this one. Ahoy reminds me of like Chips Ahoy. Yeah. The cookies. So it's not like a pizza kind of sprinkled with the cookies. Huh? I modified the prompt to make it a pirate marketing specialist. And then I added a keyword. And some ingredients, scurvy, Avast Me Mates, and bottled rum ingredients. Trying to get it to a. Oh, so it's a pirate. It makes sense. Anyone from the group have any names? It's good to participate and play around with this, a feel for how these models work. We'll see if we can create an image based off of this. I'll maybe add pizza to the end of that to make it a little bit more apparent what image to generate. Yeah, again, just want to open up to anyone if you have any questions or anything before we wrap up. Please feel free to come and chat. Also, you can reach out to Horace and I. I think on LinkedIn or probably LinkedIn. It's easiest to use our full names. You can probably find us. Trying to get the best image here. There's four images generated here, and they're all pretty good. But I think this one has to win out. All right, Horace, you ready? It's a little off. Yeah. It's a little off the screen here, but. Yeah. All right. It's good. Yeah. Fun here. All right.

So this is a scurvy bites ahoy. If I ever start a pizzeria, this is going to be one of the pizzas. You'd think a pirate marketing would go pretty good with a pizza shop. So yeah.

Yeah, thank you, everyone, for your time today. And yes, please feel free to reach out to Horace and I if you have any questions or questions about how to implement these type of models in your own applications. Thank you, everyone. Yeah, same as Nathan. I hope you were able to learn something. And yeah, feel free to reach out. You can find us on LinkedIn. Our names are pretty unique. So yeah, you'll be able to find us if we can help in any other way.

We can close this meeting up. Thanks, everyone, again. Bye. Thanks, everyone. Have a good day. Bye bye.