Building Your Own GenAI Agent Application

So, welcome everybody. All right. So, a quick intro about ourselves. So, I'm Amit Madelbaum. I'm a senior tech lead at AI21 Labs, one of the, I think, most important AI companies in the world and probably the biggest one in Israel. I'm one of the few that develop foundation models, which we are going to use through this workshop. I'm also an angel investor at Victoria, which is a New York-based VC, formerly I was a co-founder in the area of Gen AI and director of AI at NVIDIA.

And Idan, he's a full stack engineer at AI21 Labs with high passion, as you're going to see, to AI applications and technologies.

And in terms of agenda, what we're going to have today is around ten minutes of lecture about general introduction to Gen AI or LLM agents. And then around ten minutes explaining the React framework that we are going to use through this workshop. React, not react programming language, which this workshop is a part of, but React, the prompting framework of LLMs to work with agents.

And then around 40 to 50 minutes of a hands-on workshop on actually coding and deploying your very own LLM agent that can actually use tools like web search and coding to fulfill complex tasks, again, using the React prompting framework, which I will discuss. And then we'll have around five minutes at the end just discussing how you can take it further into your work and how you can get more technical details and frameworks to actually continue working with Gen AI agents.

And, obviously, we'll have time for questions. As you can see, this entire schedule covers around an hour and five minutes and we have around an hour and a half. So, obviously, we'll have some extra time for questions.

So, Gen AI agents is a pretty hot topic. Everybody's talking about it. All the companies are talking about it. It is marked as the next trend with regards to Gen AI and AI in general.

A recent market research predicted that the current market size is around $5.4 billion of Gen AI agents. And the revenue forecast for 2030 is going to be 10 times more. $50 billion. I myself think that this is a pretty conservative estimate and that the actual forecast can be and will be much, much higher than that. And I do think this is a very interesting area.

Not only that, all the big companies are getting into this field. Salesforce, for example, recently introduced their agent force, which is the new product for developing agents using no-code tools for sales, customer support, and a lot of other things. And they're investing quite a lot of budget and effort in this.

And Tropic, one of open AI competitors, introduced new models that can control your PC through the use of LLM agents. And I tested it. It works quite well. Google is reportedly also developing something similar. They are going to release Gemini 2.0. And with that, they're going to reportedly release an agent that can control your computer using the LLM. Microsoft is not staying behind. They already released a few agent developer tools and copilots that you can operate within your company and hopefully automate workflows and create AI coworkers with you.

So, as stated, this is a very, very hot topic that all companies are going into. We at AI21 Labs are not staying behind, and we are working on AI systems. We do believe that the way to get the most value from Gen AI is by building systems that can be integrated within the enterprise workflows. These systems can combine complex decision making and adaptiveness towards the workflows that you do within the enterprise and understanding the way that you work. These systems can also have resource management and budget management and also transparency in order to make them trusted by the user and trusted by the enterprises.

And eventually, the way to win the market is not by building the best LLM out there or the smart LLM out there. It's by building a system that is the most trustworthy and complex and is able to really be integrated within your enterprise workflow in an end-to-end fashion.

So, about the agents in general. So, we all know probably all of you that are participating have tried using things like chat GPT and cloud, etc. And we all know the concept of prompting the LLM, getting some answer back, and then again, if we want to. The first concept that needs to be understood is the concept of chaining. This is not agents yet, but it is it was introduced around two years ago. And this concept is quite simple in the sense that they introduced in the paper an understanding that whenever the LLM gets a question that requires some multi-step reasoning in order to solve it correctly, it needs to be prompted in a way that would actually encourage it to make this kind of step-by-step reasoning, which is called chain of thought. What we can see here is that we have an example. We are showing an example to the LLM. The example is in green. We show him an example of some arithmetic question. And in this example, the answer is just given directly without any thinking or reasoning or step-by-step reasoning. And in this case, when the LLM is asked a question directly, it gets the answer wrong. But then when the example is given to the LLM with a step-by-step reasoning that breaks this problem into a step, the LLM is able to break it down itself.

And when it gets the question that he needs to answer, it's able to break down into steps and then get the actual answer correctly. And this is a very important concept of making the LLM actually break a task into smaller subtasks and solve them step-by-step. And it actually doesn't have to be in a few-shot way.

It can be in a zero-shot, which is actually very simple. Just telling the LLM to think about it step-by-step. And this is what we called chaining, which is a very important concept in an agent framework, although it is not the agents that we're going to discuss.

The most famous model that actually uses this kind of chain of thought thinking is the new model by OpenAI. It's called OpenAI 01, and it's the first model that has chain of thought integrated within the model. And what we're used to and we're still used to in GPT-4.0 and models like LOD, SONET, is that we give the model some query or prompt, and then the model just completes, predict the next token and give us the answer. What's happening in GPT is something different. In GPT-01, what we see here is that I ask it to plan me a trip to New York for a family of four.

And now, instead of directly giving me the answer or immediately giving me the answer, what we can see here is that there is a chain of thought reasoning being done and it's part of the answer. So the LLM is actually gathering the different details and inspecting the different aspects of what a good answer to my question needs to be. And only after completing this entire process of step-by-step reasoning behind what needs to be given, it gives me the answer, which you can see here.

This is the answer that I asked him. And this is a very important concept that shows that just using regular LLMs is not enough. And again, this is still experimental in the sense that this is not, 01 is not the best model out there. And most tasks, the current model are better, but it is the first model that actually integrates chaining and reasoning through the model response for each query and is something very important and shows that open AI are also going and spending quite a lot of budget on things related to agent. But as I said, this chain of thought reasoning still falls under the category of single prompt, single LLM call and single answer, which is eventually not agent. Usually, when we are talking about agents, what we want to have is to usually have a prompt and then this prompt invokes an LLM call, which can invoke, for example, a tool call like web search. And then the answer of this tool calling can go to another LLM and then another tool call, like maybe coding, which was something we'll see in this workshop. And then maybe another LLM call to define the final answer. And only then the user itself get the answer. So this is a much more complex system that needs a different framework than what we saw in the chain of thought.

So formally, LLM agents are something that is actually coming, agents in general is something that is not new. It's coming from the field of artificial intelligence, but artificial intelligence in the sense of how it was used to mean in the 60s and 80s, which is a bit different than what we say right now when we talk about artificial intelligence. And there, there was the concept of agent, an agent is eventually a model, it's a computational model. It doesn't have to be a statistic or stochastic, but it is some sort of computational model that have some reward that he needs to achieve and some policy which is a way to actually achieve this reward, an approach to achieve this reward. And there is a set of actions that this agent can do on an external environment. And then there are observations that the agent gets back from the environment and a reward function. The most basic example, we can think of the autonomous driving, let's say our agent is the algorithm controlling the car.

It can speed or slow down and it needs to get to some destination and the environment is the road and the cars around it, it gets back observation and depending on the policies that is, that the autonomous driving agent is trained on, it will do the next action until it returns the destination, which is the reward. Moving this to the area of GenAI, the agent is now an LLM with special prompts, one that we will see is the react prompting and sometimes with an external memory, something we would not see in this workshop.

And perhaps in some cases, other LLMs that helps him on some auxiliary tasks like task management and so on. And this agent can perform actions on an external environment and in this case, external environments should be external tools like coding and browsing and web search, which is something that we're going to see and perhaps other LLM agents in case of multi-agent systems that we are going to talk about briefly in the end of the workshop.

It gets back an observation, but also from this observation, it's, it deduces some reward regarding the query, the original query that the user asked. And as with chain of thought, the LLM, one of the, one of the most important things with LLM is that it can break down the big task into smaller tasks and then get sub rewards. So for each step that it does, it can understand whether it is closer to fulfilling the user request by understanding if the sub tests that he needed to do were fulfilled. And if so, what should be the next task and how close we are to actually completing the entire test. And again, the test eventually comes from the user and the query, as we'll see in this workshop.

So the React framework, again, not to be confused with the React programming language, the React framework is the basic, I think most fundamental and probably most successful framework for working with the GNI agents. And it was introduced, I think, around two years ago. And quite surprisingly, we don't have another framework which is much better that has replaced it.

And the basic structure of the React framework is quite simple. We prompt the LLM to first think about the task, which means it has to do some planning of what tasks or sub tasks need to be done. Decide on an action from a set of given tools. We'll discuss it in a minute. Observe and then think again and loop until completion.

Decide on an action from a set of given tools. We'll discuss it in a minute. Observe and then think again and loop until completion. So what we do is that we actually take the framework that we just introduced of this kind of iterative loop of an agent doing actions on an external environment and just putting it all in one prompt to actually make DLLM think and work in this way. And obviously, we wrap it with some code that actually enables this looping because LLMs don't have loops built in inside them.

And it was shown that for tasks that require a multi-step reasoning and multi-step decision making, the React framework greatly improved the results of LLMs. As you can see here in the brown, the results of the React framework, which were much better than just a simple chain of thought reasoning, which appeared here in, I don't know, orange, light brown, or blue, which is just standard prompting without even a chain of thought.

The third thing that we need to know about React is that it enables us to use external tools. And the way we do it, and we'll see it in the code, is that we first take all the tools that we want to use with DLLM and we give each of these tools a natural language description. This is something that is very nice. The fact that we don't need to think about some very complex coding and ways of the model to actually understand the tool, but just give it a natural language description of the tool. This is something that really leverages the abilities of GenAI.

So, we give the natural language description of the tool. We give a spec. And the way we are the reason we give it because we want the DLLM agent eventually to create a valid API call, which we can then send directly to the tool and get back a response. We don't want to do some sort of processing of natural language request of the agent to the tool. We just want the DLLM agent to give the request to the tool in a way that can be directly transferred to the tool. And again, this is leveraging a very nice property of LLMs that they can do what we call function coding or JSON output, which we give it some structure and gives us the structure back with the right parameters and we'll see it in the coding session. And these are the DLLM actions. So, DLLM gets a set of tools as part of the prompt.

It chooses the right tool. It creates a valid API call and gets a response. And then we're observing this response and think about what should be the next action. And this is the way that this kind of iterative loop is being done. So, each time we call a tool, get back a response and think about the next action.

We do note here that the number of tools we can give the LLM is limited. And the reason is that we have a cost. Each tool, as we said, is given with some natural language description and with some API spec. And sometimes we might want to give even some examples of how to use the tool in order for the LLM to actually make the right calls. And these things are getting into the context window and we pay with tokens. You know that LLMs are being paid by the token. And the more tools we add, the better, the more there is a room for the LLM to make mistakes in the sense that it may be not choosing the correct tool. Or maybe I'm choosing it but not calling it in the correct way. So, there needs to be some sort of limitation.

Usually this is some trial and error and it really depends on the tool. But as a rule of thumb, I would definitely not recommend giving more than 10 tools to the LLM. It should be too much. I usually do it like five or six tools at most. More than that, the LLM starts to make mistakes. And as we said, the React framework has a loop and we need to have some subcriteria.

So, we give and we'll see it in the code. We give the LLM a final answer token which says that the LLM is indicating that it reached a final answer. And then when we pass its answer, we know that we should we do not need to call another tool because we have the final answer now. But sometimes this LLM's agent does not work and can go into some infinite loop. So, we also have a hard stop criteria on the number of loops that we can do. And we'll see everything in the code.

And finally, the basic React prompt structure looks like this. This is exactly the prompt that we are going to use in the code. So, first we say the LLM what they did. Because a lot of times we are going to ask the LLM questions that are related to specific dates. For example, what is the weather tomorrow? What was the stock price of NVIDIA last week? So, the LLM needs to know what date it is. And then we ask him answer the question as best as you can using the following tools. Here we are giving it the description of the different tools that it can use. For example, we're going to have web search and coding. So, the description of these tools is going to be here. And then we ask the LLM to use the following format. First is the question that needs to be given the answer. This is something that we are going to give. And then the thought that the LLM itself is doing on what should be the task that needs to be carried out and action. Which is choosing one specific tool from the list of tools that we gave him.

And here we have the list of just names. And then the input to the action would be usually the API call, a valid API call according to the description that we just gave the LLM. And then an observation that the LLM is getting back from the tool. Obviously, this needs to be wrapped with a code that actually gets this kind of observation back and puts them back in the prompt. So, eventually on each step, this prompt is accumulating over time. So, each time it's going to get its previous thought, previous section, previous observations and so on until it finishes. So, each step that we do with the React is getting more and more expensive. This is another reason why we wouldn't want to do something that takes too much steps.

And this is repeated n times until we get the final answer and we prompt the LLM to say I now know the final answer. And then given and then the LLM is going to give this special token of final answer. And here it would give the final answer to the input question and then we prompt it to begin. So, this is the basic React prompting that we are going to use.

And that's it for my part. And without further ado, let's step into the hands-on workshop itself. Idan, the stage is yours. Yeah, thanks, Amit. So, hi, everyone. So, first I'm going to share with you a link for the repo that you can clone if you want to follow along and write it with me. So, I sent it in the chat. We can wait like for two, three minutes until everyone takes the repo, clone it. In the meantime, I'll just explain a little bit about this repo. And also, I will share my screen. Just a second. Yeah, cool.

So, basically, sorry. So, first of all, this is the repo if anyone is using VS Code then maybe doesn't know how to clone from the terminal. So, you can go in here, it might be on your left side because I moved it to the right side on my computer. And you can just click here and then here and do clone and give the URL.

You can get the URL from here as well. I think you can use the HTTPS, but if not, you can try the SSH one, just copy it and clone it.

So, this is a TypeScript project. It's pretty basic. Doesn't have much stuff. It's a Node.js, by the way. I didn't say so. We will start with a Node.js and then at the end, if we have time, I will show you how we can connect it to React. But, basically, the main logic of the agent is done in Node.js. There is no real requirement to know a lot about Node and TypeScript. It's pretty basic. If you know code, then I believe it's if you know JavaScript.

Maybe I think I will start slowly. If someone is struggling to clone or something, then I will share what we're doing. I will upload it later to the repo and you can get the entire code. So, don't worry.

Okay. So, we're going to build an agent. As Amit talked about right now. For a start, we want to actually have an LLM.

Every agent is based on some LLM. For this demonstration, we will use the Jamba API, which is 21's LLM, which is the best in the market, of course. Just to show you. I will share with you also, by the way, the API keys, and then after this talk, it will be discarded. All of the API keys. You can use it if you're using it right now. But if you would like to use the code and play with it, then you will have to get your own API keys.

Okay. So, just to show you, this is the API. We're going to run real quickly about it. Let's start. Okay. I have some boilerplate code, which is the dry code that I didn't want to write from scratch because it's not that important. First, we have a class that is an LLM. What do we need to utilize the API? Of course, you can read it here, but the most important part is the model. Those are the options. We're going to use the Jamba 1.5 large and the messages. The messages is required and the model is required. So, you can see here that we are making a new class, constructor. It's really just instantiating the API keys and stuff.

And basically, it's a URL with Axios post request that goes to this URL and just gets the response. At the end, we do need to extract the response. If you're looking at the documentation, you can find where is the content that you need. I already know it. So, it's in choices. In the zero index message content. It's pretty straightforward.

The only thing is regarding the dot end. So, if you have cloned my repo, you see you have here dot end example, and there are two API keys that we are going to need. I will share both of them right now, and you can just go to the dot end. You can rename it. Just remove the dot example, and then you will have dot end and copy what I'm going to send, and that's it. You don't need anything else.

So, just a second. Yeah. I will share one by one, so it will be easier. So, this is the API 21 API key, and this is the next API key that we will need in a minute. Yeah. Cool.

Cool. So, as I said, you need the dot end, which is like an environment variable for the API key, and that's it.

So, after this, we have a chat LLM. It's nice. Let's see how it works. By the way, one more thing that I wanted to show you is how to get the API key later, like after this workshop when the API key that I provided has been discarded. So, you need to go to AI 21-studio.ai21.com. It will be also in the code. You just need to sign up. I already have a user. So, when you get here, you go to settings, you go to API key, and that's it. You will have an API key. This is all you need. Pretty straightforward.

So, let's try to use our API, right? So, first of all, I will write some functions. So, let's call it like run LLM. So, we can just instantiate the class. Thanks, Copilot. So, it's imported. Sorry.

And then we can just use generate. So, if we take a look at generate, you can see the generate has some args that are optional. So, this is the predefined. It's not really important for this example. You can read about each and every one of them. But the important one is the messages, which you can see here in the API.

So, messages. We have a few types of messages. So, they're almost all of them are the same. They have a role and content. If it's a system message, you can see the role is system. If it's a user message, you can see the role is user and the content is just a string. So, it's an object. It's an object which has role and content. Okay? And also, by the way, messages is like an array of messages. So, it's typed. So, you can go here and see messages is message array. And message is actually role and content. So, let's provide what we need. So, we need messages.

And it's an array which has two options. So, first, we're going to use the user role because we're performing a user action. And just content. And let's ask him who is the President of the United States. Right? Okay?

And now, let's just run the function. And to run this project, you just need to go to the terminal. You need to see the inside the folder of this project and just run NPM. And it should work. Let's see. It will take a second. Hopefully. Oh, wait. I know. So, generate I didn't console log and also generate is an async function. So, we need to wait it and get the response. Sorry for that. And let's print response. So, the chat LLM. Great. So, now we can run it.

And hopefully it will work. So, you see we got an answer. This is coming straight from the LLM. Cool. Maybe not really like the the most updated one. Depends on the US.

So, now we have an LLM. Great. Now, as Amit told you, an agent is essentially an LLM that uses tools. Okay? So, what are tools? So, first of all, we need to define so, basically, tools are stuff that the LLM can use if you want to do some actions that will help him to get to the final answer. So, for example, it could be a Google search. So, we will use Google API. It can be LinkedIn search. You can build a tool that uses the LinkedIn API. It can be something for, like, code. Like, running code, maybe. Stuff like this. I will show you two examples of tools. The first one is the Google search tool. We're using a serp API.

It's a website that provides you with an API from Google search. So, again, I'm providing you with the API key as I already shared it in the chat. It's the second one, the search API key. And now I'm going to show you how we're going to use it. It's also pretty straightforward. And then we're going to write a little bit more code.

One thing I want to do before. So, because it's a tool, we need to provide some interface for all the tools. So, all of the tools will be the same. And then we are running them in the same manner. Because the agent needs, like, the same interface for all of them. So, let's create an interface. Let's define, like, what do we need? So, first of all, we will need to provide a name. Okay? So, it's like an ID. I'm sorry. So, it's, like, just something to identify the tool. Interface. Yeah. Okay. So, we prefer, like, something that is more, like, natural language.

Because LLM is like natural language. And also, we need a description. So, we can provide the LLM, a description of the tool. So, it will know what is the tool and why what can it help him? Like, what can it happen to do? So, for example, if it's a Google search, you can tell him that you will see in a second. But you can tell him this is a tool that you can search in the internet, blah, blah, blah.

And we need some function so we can use it. So, let's call it use function. And this use function is going to get some strings, input strings. So, it can be a query string. And it will be, like, a promise string. Because it's going to be an async function. We're going to run, for example, if it's a Google search, we're going to run an API request. So, this is how we define a tool.

And now, let's go to the Google search. And here we have okay, so, I prepared here just, like, again, simple functions. One function is just a get request that goes to the URL of search API and just returns a response. Again, we do need to extract the response. So, this is how we extract it. And search, I will show you in a second what is search. So, again, let's define a class.

Let's call it Google search tool. Yeah. Thanks, copilot. So, it should implement the tool. It should have a name, right? Because we define I will show you because we it needs to implement tool. So, we do need to implement all of the properties that we have defined in here. So, first one should be Google search, I guess. But the description you will have to forgive me because I have description remade because I already played with it and just wanted, like, something that will work good. So, the description is just to tell the LLM what it can use and, like, what this tool does. So, get specific information from search query. Input should be a question, like, how to add two numbers in closure. Result will be the answer to the question. So, it's pretty basic stuff.

And now we need to define the use function. Which, exactly like you see here, gets the string, right? You can see here gets an input string and return promise. So, this is the search. So, search is just defining here the query, the engine that we want to use, the API key. We're going to this function. We're doing Axios.get. We're getting the result.

And I'm just, like, iterating all over the results and just making, like, one big result. Like, one string. So, we're collecting the result in an array. And then doing join white space, which is making it a string. And that's it. We now have the answer. And we can use it in our as a tool.

Let's just see that it works. So, again, function and Google search. Yeah. We need to import it. So, I'm instantiating the Google search tool. I will just use it. Let's skim something. Like, what is the weather in Tokyo? What is the weather today in Tokyo? Yeah. Okay. So, let's run this function. Oh, yeah. We have a watch. So, it's reloading.

So, you can see that the Google search has responded. Tuesday. Blah, blah, blah. 20% intervals of cloud sunshine. Whatever. Great. Okay.

So, now we have an LLM and we have a tool. A Google search tool for this example. And we want to make them work together. Right?

So, now, we get to the fun part of building the agent. So, we have an agent. The agent essentially has the prompt, which is the exact prompt that Amit showed you in the presentation. And you can see here, we have the curly bracket. And then we have the we have the curly brackets. So, all of them, we will need to replace them with an actual data. So, for example, today is today will be the date of today. To description. So, for now, we just have one tool. So, we will have the description that I provided about the Google search.

So, this is the prompt question, the input question that you must answer. Comment on what you want to do next. The action to take exactly one element. Because we want that in each iteration, it will use only one element. Out of the tools that we are providing him. First, we're going to have just the Google search. But later you will see how we have more.

And then, like, each action is like the tool that he wants to use. And action input is actually the input that he wants that he got from this tool. Observation. Sorry. Observation is the result from the tool. And the input action is the input that we're going to provide to the tool. And then, like, it will do it, like, a couple of times.

Repeat. As you see, repeat n times. Use it until you are sure of the answer. Then, when he knows the answer, so, he will have he should have this thought that says I know the final answer. And then, he must provide, like, this final answer comma. No, comma.

Sorry. You will see in a second why. And the final answer to the original input question. So, here we will also replace the question with, like, the real question we will put here. And previous responses, it's like the history. Right? So, like, it should have, like, a few iterations.

So, each iteration, it will, like, collect more responses. Okay. So, we have a class called agent. And as you can see, we have an LLM. So, we need to provide it with an LLM and tools. Right? So, tools is a tool array. Which means that it can be one or more tools.

The prompt. So, here in this case, when we're instantiating it, we're just taking the original prompt that we have here. Max loops is just, like, to define something. Just a maximum number of loops. So, we won't get to the infinite loop. And the stop pattern, so, you can see here that sometimes the LLM is hallucinating. So, we want to create a stop pattern which is, like, to know where he's hallucinating.

I think Amit is still here. Maybe he can explain it better. Amit, do you want to explain the stop pattern? Yeah. Okay. It doesn't matter. But you can read all about it later. So, we will get to this.

But these are just getters that I pre-made. I will explain them in a second. And then we want to run our agent. So, let's say that the agent gets some question. And now, we need to know what to do. So, we want to implement.

Okay. So, first thing we want to do is create an array of strings with the previous responses which will be our history of responses. Num loops is just for the loop. It should be let. Num loops. And the prompt. So, we take the original prompt and we want to replace all of the curly brackets with the actual data that we have.

So, in runtime, we do replace the today with the date and the description with these getters. So, tool description in this case will be just iterating over all the tools that we have. Sorry, someone. So, iterating over all the tools and creating a list of the tool name, the tool description. So, this will be here and you can see that here we will have a description, a tool name and description. Tool name, description. And then exactly two names. This is also just joining the two names. And tool by names you will see later when we write the code.

And we want to replace the question with the question. Cool. So, first, let's write our while loop. So, we're going to iterate until we have an answer or reach to max loops. So, while num loops is less than max loops, we want to do something. Okay? So, first we want to start to generate well, almost. So, yeah. So, we do want this part. Right? So, we want to generate the first question. So, this will be the first question that we want to ask. It's the same as we did here.

Just with the prompt that we got from here. Right? So, we do want to do one more, like, tinkering with the prompt. So, we can take, like, let's call it the current prompt. Because in each and every loop, each iteration, sorry, we would like to include the previous history. So, we would like to also, like, concat the history. So, you can see we can take this prompt and just replace the previous response with, like, the curly brackets, like, the placeholder for previous responses with the previous responses. So, it's an array. So, we join it and then, like, making it, like, a list. So, just a second. Let me see that I'm correct. Yeah. Okay. And this is the prompt that we would like to send here. Cool. I just wrote down things that I won't forget. So, I'm just looking at it. Yeah. And I also had, like, one small problem with it. That sometimes it just, like, adds another extra thought. Like, the word thought.

I will show you. You see here, you have this thought. So, sometimes it adds an extra one. So, just to be sure. Yeah.

So, I think we should, like, replace it. So, like, we're just deleting, replacing, just finding this. If it finds it, it deletes it. Sometimes it's with this white space. So, it will delete it. And it should be okay.

So, I have a response. Great. Now, what do I want to do with this response? Okay. So, I am going to take this and just put it here and comment it out for a second. And we would like to have a response. And for each response, we would like to extract the tool that is going to use and, like, what is the input of this tool and, like, decide whether we are over with the thoughts or we do have different thoughts. Like, we would like to decide what our next action. Okay. So, let's write a function that is called next action.

And I promise you that you will get it in a second. Okay? So, just a second.

Okay. So, let's just private async. Async side next action. So, yeah. So, basically, it gets the prompt. And this is incorrect. So, what we do want to return is just an array of, like, a tuple, but three items. And someone is talking. Sorry.

Okay. And something And something here. Yeah. So, I have a function that is called the next action. Sorry. I'm going to mute you somehow. Wait.

Okay. So, we have something here.

Not right. Yeah. So, I have a function that is called the next action. And we do have a prompt here. And we want to return three stuff. Okay? So, what do we want to so, let's define what we want to return.

So, first we want to return the tool that the LLM has decided it wants to use. Another thing is the tool input, which is what the LLM is going to input to the tool. So, for example, if it's going to be a Google search. So, you will you will input the search that is going to perform via the API.

And also, we would like to know, like, so, and, like, get just the generated response. So, for example, we can just move the response here. Right? So, like, so, just think of it of, like, some function that is making a response from this response. It gets the tool, the input, and the generated response. And then it's, like, deciding the next action again and again until it reaches the final answer.

Okay. So, if we do have a final answer, we do want to stop the loop. But you will see it in a second. Okay. So, now we want to get the tool and the tool input.

So, each response is going to be in this format. Okay? In action and action input. So, we will need to extract the action and action input in order to have the tool and the tool input. It's the same. Okay?

So, let's see. So, we have a response. And we want to parse the response in order to get the tool and the tool input. Sorry, not here. I'm going to just delete it. This response. But we want to parse it. Let's write the parse method in a second.

So, we have the parse. And parse is getting the response, by the way, here. Okay. And parse will return us the tool. The tool input. And we can also and then we can return the tool, the tool input, and the response of the LLM. And now I will write and I'm just saving so it will format it. Yeah.

Okay. It will format. Okay. So, let's write parse.

So, right. Parse gets the response and returns a string. Which is the tool and tool input. And how do we do it?

So, first of all, let's see if the response has the final answer. Because if we have the final answer, we're good. We're good to go.

So, we can check if the response includes final answer. Right? So, if I will add one more thing. So, we have like something called tokens. Which is like the thing, like the const that we want to do, to pick and see if we if our text is has it.

So, here we have observation token. Which I show you here in the store pattern. And let's just define one more token. So, it would be the final answer token. Which is just this one. Final answer.

And then we can take it and put it here. Yeah. So, if our response includes the final token, right? So, we can just return the tool with the name, final answer. Which is our tool. The final answer, actually. And the final answer. How do we get the final answer?

We can do okay. So, yeah. We can actually split yeah, we can split it with the token, maybe. And take the token. And we want I prefer actually maybe to do it like this. So, we can pop sorry. I guess this needs to be something like that. Yeah. Cool.

So, pop just takes the last element from the array. And then we just trim it and get the final answer. So, basically, just quick explanation. So, imagine this would be the final answer. This is how it would look like.

Would be something like this, and something like this. So, we would like to split this. And just we just get this part. And that's the answer. Okay? So, this is if we have the final answer. But what if we don't have the final answer? So, what do we want to do? We want to just do like a string you know like string manipulation a little bit, maybe.

So, we would like to find some regex that will take the action and action input as we as you saw here. We have the action and the action input. And we want to search where is it inside our response. So, I'm not that good at regular expressions, sorry. So, you will have to see just premade regex. But again, you can play with the regex. You can ask SGPT, other LLMs, Cloud, whatever, and just get the correct regex for what you need. And let's see. So, let's just get the regex. This is how we use regex in JavaScript. It returns us and it called the capture group. Capture group is just like a group of the things that it got captured, right? So, it got the action and action input. So, let's get the action. So, if we want to get the action, it will be yeah.

It will okay. So, you know what? Just let me sorry. So, if there is no match, we will throw an error. Yeah. But not pass the response. But if we do have so, match is also gets if it's in the zero index, it's just the index inside the text where it found the word. So, we're not interested in this.

The capture groups are starting from one, two, three. Depends on how much do we need. So, here we just have two. So, the first one will be here. We'll trim it. So, this is the action and action input is yeah. Actually, in the second index. I did have some problem with the second one which is like adding extra quotes. So, I also find found like a replace method just to remove the quotes.

And now I think we can yeah. So, we can return both of them. And we have the action and action input. Let's call it tool.

Cool. So, parse is now returning both of those. Great. So, we now have those three.

And basically, I think we're okay. Great. So, we can decide our next action. It's still not complete. But you will see in a second what we're going to do.

Here we have the prompt that we need to send into decide next action. So, yeah. So, we call decide next action. We get the tool input and the response. Great. Now, I want to just see that I didn't forget anything. Yeah.

So, again, if the tool is the final answer, we want to stop the loop and just return the tool input which is like the final answer. Right? Cool. But what if we don't have the final answer? So, first of all, we want to check that the tool is part of our existing tools. Because if, for example, for some reason this says something else and we don't have this in our tool list, then we can't really use it.

So, we can do so, you see here I have tool by names. It's basically an object. Right? It's like because it's a getter. So, we just take all of the tools, we have them, we make an object and the object will be like the tool name and the actual tool. Like, basically the actual functions and everything.

So, we can just check if tool is in. Don't think it's something, yeah, something like this. Tools by name. Okay. So, if let's say if it's not, okay. If it's not part of our tools. So, let's just throw some error. But if it is, now we know it is. So, we can just do so, basically, okay.

Now we can do tool by name. Right? And just tool. And now you see we have this. So, because we know that this is a tool, so we know that we have the description name and use, we can just use the function and give it the prompt. Okay? We need to wait. And we need to get the answer.

Cool. So, imagine this has a few tools. And we know that we want to use Google search, for example. So, we're going to tools at the name Google search and then we can use the function that will call the Google search API with this response. And we have and then we get the answer.

So, now what do we want to do? Let's just format our observation, yeah. Because we do need to know what our next action, right? So, we need to make an observation which is formatting the response of this step for the next one. Okay? So, here let's see what do we need. So, we do need to get the response first. So, we want the response and we want just let's so it's a response. And then it is the observation token. I will show you in a second what does it mean. Okay? The tool result, the tool input actually is here. Yeah. So, just a second. Sorry. Sorry. Yeah. Sorry. No.

We do need the answer here. And then the yeah. And we need to add another token. I will explain this just in a second. Just in a second. Yeah. So, and that's our string. Here. So, why do I need this? So, if you look at this pattern, you will see that we have an observation. And could be the final answer. It could be the next prompt. So, here it's the same. We just like creating it another like part of the template, which will be like some addition of this for the next one.

So, we're collecting the history of the thoughts that the LLM is generating. And then we want to remember that we have the previous responses array. So, we just want to add this to the previous responses array. So, just the observation. Yeah. Just let me make sure that I didn't forget anything. So, this is the actual observation.

And you know what? Let's just print it so we can see step by step how the LLM thinks. So, just yeah. And so, not sure why we need to return something, I guess. So, something is missing here. So, just a second. What am I missing?

We do return the tool input, which is a string. So, what we just okay. So, eventually, we do have just two options to get out of our loop, right? So, one is if we run out of iterations. And the second one is when we got the final answer and want to return the tool. And so, if we didn't, it means that we reach the max loops. So, we will throw an error. So, this is basically our agent.

And now you can see that we can use it, right? So, let's see how do we use it. So, again, I have an async function run agent, yeah. We can instantiate an LLM like we did here. And we would like to instantiate our agent. So, yeah. And now what do we need to pass the agent? So, sorry. We imported the correct one. So, this is our agent.

And if we come to the constructor, you see that it needs two things. It's the LLM and the tools. So, let's pass it the LLM. And tool array with the Google search.

And then we can just run it. And so, okay. So, we need to provide with the question. I have like this question here. Okay. So, here's a nice question. How much money did the Titanic movie generated? How much money did the Avatar movie generated? Who generated more money? So, he will need to do a few steps, right? He needs to go to the internet, search for how much money Titanic movie generated, then Avatar and then decide who made more money.

And let's console log the response. Console log the response. Great. Now, let's run it. And let's see it in action. So, let's see what it does. Yeah. It takes time. Weird.

I think we forgot. Yeah. Live coding never goes well.

Okay. I'm seeing that someone asked questions in the chat. I will just look and then maybe I will answer the questions in the chat. You answered? Cool.

Okay. So, again, let's try one more time. And then if not, we're going to debug it. Yeah. Okay. So, it did get to final answer, but I wanted to print a little bit more. So, something here is weird. We do have here just a second, let me think. So, if the tool is the final answer, we want to return it. If not, then we're going to use it. Let's try again. I don't know why it doesn't print this. Give me one second.

I'm just making sure that I didn't forget anything. So, maybe we need here something. Yeah. I think that we can add here maybe one more. Something like this. Yeah.

Okay. Let's see. I still didn't print. Someone wrote something. Yeah. It will be posted, the code. I have it working. I promise you. Not sure why it is not. But you know what? I have a backup. So, let's see.

So, this is, again, the same code, exactly. Exactly the same code, but we can also make sure I do take the correct. So, the generated is the just want to make sure that I'm turning it in proper order.

Yeah. So, the response is third one. Right? So, we can put it here. So, this is the response, the observation token, the input on it and the thought token. Yeah. I think I know what was the problem. So, let's see. I'm not sure why. Let's just maybe I will go to this repo that I know for sure is working, but I want to show you how it works. So, here you can see it's the same code. Something is missing there, I guess. But it's the same. So, we can run stack. Yeah. Okay. We got it. Kind of. So, you can see that first, he got the first thought. Okay? So, we thought how to continue. So, he tried to So, what is the action I'm going to take? I'm going to Google search.

What I'm going to Google search? How much Titanic made. Right? Titanic movie revenue. Actually, it was Titanic movie made. So, it needs to know how much the Avatar movie made. So, we did the same So, he did use the Google search tool. Right?

Cool. Now, let's think we, yeah, so, let's continue here. Although, this is not performing, but I will write the code, and I will show you there it's in the final result, but it's still the same. So, we do want to add another tool. So, I created another tool here which is, sorry, called the Code Runner. Code Runner is like a Node.js CLI. It's basically, it just runs Eval. Eval, for those who don't know, is just a function. It's not a, it's, you shouldn't use it in production. You should use something else in production, but for this demonstration purpose, Eval is just taking a valid string, which is a valid JavaScript code, and can evaluate it, and then, like, get the output. And you will see in a second.

So, again, this is a tool, right? So, we want to use it. So, let's define this tool. So, if it's a tool, it needs to, sorry, it needs to implement the tool interface. So, if it implements the tool interface, it needs a name.

So, JavaScript Code Runner, I guess, is a good name. Description, I have a description, again, like, Remade. Yeah. So, I will paste it and show you. So, description is just a JavaScript Code Runner. Use this to execute JavaScript commands. Input should be a valid JavaScript command. If you want to see the output, you should print out with console.log. That's it. So, when we need to implement the use function, we get the input strings and return string.

So, I have here a utility function called clean code, because the LLM generates the string with this markdown. So, we need to remove the JavaScript markdown. And this is why you have the clean code. So, this is so, it gets a code, and then it actually returns something. So, here we can just run it like this. And run the code. Cool. Runner is just function. The run function, that's it. So, let me just show you how does it look.

So, again, function. JavaScript. So, cool. So, let's maybe do some calculations, because probably you might want maybe to use calculations with JavaScript. So, imagine this is, this is actually a JavaScript code. So, I'm doing here console log, 7 times 5, which is 35, I think. So, let's see if this prints what we want to print. Yeah, I will open a new instance of the terminal. Just a second. So, we can run it. And you can see that it's actually run JavaScript codes. All right. So, 7 times 5 into the console log, and then it printed out. And now, we can actually make a tool that uses the same as we saw here, but we can add another one, another tool to the agent. So, I will show you here. But it's the same. So, you can see. So, let's call it a multi tool, for example. So, we have the agent, new agent. We instantiate the chat LLM.

We do get the Google search tool from the previous example and we also give him the JavaScript tool. And we can ask him questions. So, we can ask him, like, find the total cost of coffee consumption in a year in London. Search for average coffee prices. So, he probably will realize that he has to search on the internet for the coffee prices. And then typical consumption per day and use JavaScript to calculate the yearly expense in British pounds, including weekends and weekdays.

Great. So, let's see how it does. So, we have the multi tool here and we can run it. So, we did a Google search, another Google search, another Google search, another one. Now, he okay. I will start from the beginning real quick. He did a Google search of the average price of cup of coffee in London. He got some result. Another one is like typical coffee per day. He got some other results. Like 428 cups. Blah, blah, blah. He got some data that he needed. He created JavaScript code.

He then calculated everything, console logged it in here. He ran all of this code and he got the final answer. So, the final answer is the total cost of the coffee consumption in a year in London is 1,008.

Okay. So, this is the agent. This is the tools. That's everything.

If we have more time, I'm not sure. I don't think so, but I can show you like a real quick like a web socket that maybe uses this in React, right? So, you can see that I have a WebSocket server, which is real plain WebSocket that opens connections and gets events. And then, the event can execute like actually outputs what it generates.

For example, I can also show you that I have a React WebSocket server that is basically just I'm not going to show you like to get inside the code, but it's just a WebSocket that when it mounts, it just creates a WebSocket. It connects to my WebSocket. And then, you can see here that if I will do NPM run dev, this is a React application, a real basic React application, which I can here. So, you see the WebSocket is disconnected and I can also like connect to this WebSocket. So, now, I starting the WebSocket. And if I refresh, you can see that now we can type. And I can let's take maybe one of the examples that I had here. So, like a question about Titanic and Avatar movie. And I can paste it here and just click enter. And we should see the thoughts as we go.

So, like I know the final answer. Titanic movie generated, blah, blah, blah. So, this is the same that we saw in the terminal. But now we see it in React. It's like a really, really nice way to just use this agent.

Thank you so much, Yadan. And this was indeed a very, very nice example of an agent that can use multitools. It can code. It can Web search. And basically, a lot of tests, I would say even most of the tests that can be solved by a Gen AI agent or LLM agent can be solved using these two tools.

So, you can and should take the code that will be posted afterward in the Git and can really go and implement it. And if you want to have more information, in the last minute that we have, I'll just share with you. So, there are a few places that I really recommend.

The first one is Langchain. So, Langchain is usually using React in the backend. You have good tutorials for JavaScript agents with Langchain. They also explain about React. They show it in code. And they also discuss in this presentation have links about reflection and multi-agent system.

And another very common multi-agent framework is the Microsoft Autogen framework. It's based on Python and not JavaScript. You can use it for more advanced use cases that have multi-agent systems. And, yeah, we're just out of time. So, maybe if you have final questions, we'll wrap it up. Idan just posted the link with the final working code that you can play with. If there are no more questions right now, we all thank you so much for joining us and hope it was useful for you and have a great day. Yeah. Thanks, everyone. Bye-bye.