Crafting Intelligent Agents with Context Engineering

Hi, everybody. It's great to see you. My name is Carly, and today I want to talk to you about context engineering. So, you might be wondering why we need to talk about context for agents. Can we not just code and let it figure it out with its information? I'm going to explain why that's not necessarily the case. We're going to talk about each of the components of context engineering with examples to show you the kind of practices you'll be doing.

And, of course, we'll have Q&A at the end after this recording. So, if you want to ask me questions, do come and join me in the Q&A session. So, if you haven't met me before, hi, it's nice to meet you. I'm Carly Richland, and I work in the developer advocacy team at Elastic. I am collecting socials like Pokemon. I'm sure a few of you are as well. So, if I don't get to your question in Q&A and you want to come and ask me something afterwards, just scan that link there. That will take you to my link tree, and you can find me wherever works. And I'm happy to help or answer questions as they come up.

So, why do we care about the context that's being passed on LLM? Well, firstly, we need to understand what the context window is. In simple terms, the context window is the number of tokens that an LLM can process at once. And while you might be thinking, well, I'm seeing from a lot of the emerging new LLMs that they have these super big context windows, it doesn't necessarily matter what I'm sending there. I'm going to disagree and I'm going to say that that's not quite right. It's still very much possible to overflow a context window. It just gets harder with the more tokens. But also, we can actually end up doing things like poisoning our context. If we have irrelevant information in there that's not actually useful, that can influence the actions that an LLM can take and therefore impact the overall result. So, it's actually still very important to make sure that the context window represents the type of information that you want it to. And not just talking about a single agent, but also making sure when we get into multi-agent architectures that each of those has the right context that it needs not only to perform its own task but to coordinate with others. It's also partially a cost point as well if we think about it. Foundational models tend to be charging per token. So, of course, there's definitely savings to make to make sure that our agents are working within our means and also making sure that we're not blowing the entire company budget when we're building these tools. It's also that LLMs do still make things up. They still hallucinate a little bit and that's down to a few different reasons.

It can be down to the cutoff point for the model which is basically the date after which it doesn't have training information available. So, if you're asking a model with a cutoff date from 2023 what happened last week, it just doesn't know the answer. It can also be down to knowledge it just doesn't have access to. That might be things such as the proprietary information that you're trying to expose via chatbot or another tool to automate common processes. Sometimes the models itself, albeit probably less likely with foundational and large-scale models that we're using, could be subject to overfitting. And this is a well established problem within machine learning models anyway.

Biases in data are something that are very heavily discussed and talked about as well. And sometimes even language ambiguity can trip up some large language models as well. But the other thing is sometimes with the notion of hallucination, they're not very good at saying, I don't know. I'm sure we all have a person in our life that that immediately thinks, that makes us think of. But a recent paper from September by OpenAI actually suggests that the training and evaluation process being employed in order to build these models up actually rewards them guessing the answer over acknowledging uncertainty. And for that reason it's not necessarily going to tell you if it doesn't have the right context because it's incentivized to try and find an answer.

And that's why we need to use all the tools within the context engineering toolbox in order to understand what's going on and make sure that the large language model is not set up to fail because it doesn't know pieces of information that it needs to actually perform the actions that we want it to perform. And what we're going to do is talk about each of these components in turn, what they are and use precise examples to help. The main example I'm going to use is this simple travel planner bot that simply pulls out from a series of tools, weather and other information and generates an itinerary for a trip. I'll show you the code throughout the presentation and then at the end I also have a QR if you want to go in and dive into the code afterwards.

I'll show you the code throughout the presentation and then at the end I also have a QR if you want to go in and dive into the code afterwards.

So the first thing we're going to talk about is the user prompt, which while we don't have that much control of, we can certainly influence through things like the choices and information we give on our UI. You probably are familiar with a simple prompt like this which is just asking a basic question. And this is known as a zero-shot prompt. It doesn't have any examples, it doesn't have a lot of detail in it and it will basically, you can see that chat GPT is going off it's actually using search under the hood now and it's trying to figure out what the answer to our question would be. Very common operation that many of us are used to doing.

But actually prompts can be more sophisticated, it's not just limited to textual data. Of course we're used to generating images like with the example prompt I have on the screen but there's no reason that we also couldn't use images, video and other mediums within a prompt as well to try and get the information or perform the actions that we want the model to do. When we're passing a prompt using AISDK as an example, what we tend to be doing is firstly sending the message and you'll actually see up here I've got send message which is being passed in through the use chat hook available from AISDK and then I have this simple function here send user chat which is actually sending the message as parts of type text. And then furthermore later on if you look into the TSX further down you'll see that in my form I'm going to send the user chat on submission and also when the enter key is clicked as well. And that's the simplest way that we tend to get user input into an LLM.

We also can provide custom instructions. This is commonly known as the system prompt but you might hear it called instructions as well. And this is basically more of the blurb in detail to direct the large language model on what particular problem it has to solve. Prompts tend to have particular structures. There's numerous strategies that you might want to employ and what you'll see here is there's numerous different elements that we can tweak within the field of prompt engineering.

So we have the role i.e the persona we want our large language model to employ in order to conduct these tasks. We've got the directive which is the instruction, what it actually needs to do. The output formatting is should, well it's the output formatting. Is it text? Is it JSON? Is it something else? What am I expecting my output to look like? We can provide examples and this is what converts us from a zero shot to a multi-shot prompt. The idea we're giving it these examples to explain how many times, how it might want to action particular inputs.

Then there's style instructions which is you know perhaps more information about the output formatting and how you want it to look in terms of stylistic instructions. And then other information which might include things like guardrails or other things to try and direct it further. I.e maybe things like what not to do. When it comes to system prompts there's normally an element like system or system prompt that we basically pass whenever we're invoking the appropriate stream with our model in order to provide the system prompt and as you can see here we'll normally pass in not just the system prompt but also the messages from the user from the user prompt we discussed before so that it has all the information it needs to try and perform the action that we want it to perform.

When it comes to system prompts there's normally an element like system or system prompt that we basically pass whenever we're invoking the appropriate stream with our model in order to provide the system prompt and as you can see here we'll normally pass in not just the system prompt but also the messages from the user from the user prompt we discussed before so that it has all the information it needs to try and perform the action that we want it to perform. There's also numerous strategies so if you want to dive into this more I actually recommend taking a look at this paper which has some really good nitty gritty kind of state of the ecosystem of prompt engineering but there's numerous different strategies. We can think of you know shots and examples if we discuss through there's actually numerous different types of structure for prompts if you think about things like chain of thought and there's also techniques like meta-prompting as well. All of these things can actually help us try to build better prompts to make sure that the LLM is doing what we want it to do but also there's other practices we should consider too. So think about the task type and make it clear what type of task you want the LLM to perform in your prompts. If it's a very complex task try and make sure you give it the right number of examples because if you give it too much it might lead to complexity and issues with the LLM trying to figure out what to do. Conversely too few examples could leave it really ambiguous. Think about the inputs, the outputs and the format like we do for all the systems that we build. Think about that persona you want to emulate carefully and also think about the LLM itself. Could you use the LLM to perhaps optimize, tweak your prompt or indeed do you get better results with the types of prompts that are being passed through with different LLMs? Have a look at all of these options when you're thinking about prompt engineering.

Now the next thing we need to talk about is RAG and RAG stands for Retrieval Augmented Generation. In simple terms this is where you take a set of documents normally split into pieces through a process known as chunking and then we will store them in a way normally utilizing a vector database which has our documents converted via an embedding model which we'll talk about more in a minute and then we'll take a relevant query, pull out the semantically relevant documents and then we'll pass those as part of our context to the large language model in order to get the result that we want and this is a very common thing in grounding an LLM and making sure it has the correct information. Now when it comes to similarity and semantic search we need to first understand the difference between that and traditional lexical. Traditional lexical works on exact matches for text so it normally utilizes the data structures you see here known as an inverted index where our documents and our query both go through a process to extract the relevant terms and then we look up our inverted index to find those terms and they'll point back to the documents of which that is considered a relevant and result to the query we've passed. But vector search works differently because it's all based on high school maths so if you think about different types of document not just text anymore but images documents and audio and even other types as well they can be converted into a numerical vector representation i.e. a stream of numbers and this is normally a dense vector and then we do the same thing with our query utilizing the same model and then we utilize one of a family of algorithms known as nearest neighbor to basically find what the closest vectors are and return the results. Thinking about embeddings we get these from machine learning models that are using a different task so LLM their task is all about text generation. The models we use to generate embeddings quite simply are basically going to take the semantic aspects out of whichever type of data is depending on the modality that it supports i.e. what format it supports and it'll just generate a stream of numbers that represent that document that we can then have in within a vector space. So this is an example vector space here a very simple two-dimensional illustration and what you'll see is within my vector space I've got my first dimension of realism and second of human and each of the documents within here is positioned based on the intensity that it meets that particular category. So you'll see that my realistic characters are up towards the top and then you'll see cartoon Vader is on the opposing side but he's not considered similar. Then when we go and search we'll basically get the same vector embedding from that model from our query and then we'll find the closest candidate by using a similarity algorithm. There are numerous different types of measures of similarity the default for elastic search we've seen in a few others in the majority of cases is cosine similarity and it basically takes the angle between the two vectors your query vector and one of the documents of interest and as the cosine of the angle tends towards one that means that that document is considered semantically relevant. When we're doing these comparisons those nearest neighbor algorithms there's numerous ways it can be done there's numerous different algorithms such as disk enn and others. The one that's quite popularly used in vector databases including we've seen in elastic search would be hierarchical navigable small worlds or hnnsw and the way that this works is it splits the graph space into layers as you can see from the visualization and that means that we kind of go from more kind of sparse kind of comparisons to more fine grain meaning that we're trying to balance accuracy but also speeds. So this is an example query and as you can see here that I've got my knn option I specify the vector field here which is basically you know the name of the field where the vector lives within the vector database the number of documents I want to return are the closest is k and then the number of candidates to evaluate per shard in this case I've set as 100 and then my query vector here you'll see is basically taken that query and put and you know converted it using a model and then I've got my vector ready to go and when I end up doing that and when I end up doing that I can match documents not just based on the exact text matches but also the similarity based on the attributes from the model.

Like humans you know large language models have this notion of memory they need aspects to both short-term memory i.e the current session and longer-term things that have happened in the past to help them function and make sure that they're performing tasks as we expect. Short-term memory quite simply is referring to the current session so if we take our travel planner example basically the current chat that I am having with it would be the short-term memory and it would be not just the human messages that I am sending but also the responses from the AI as well but we can also think about chat history the conversations that we've had before we've been planning other trips and those would be the ones that we store within long-term memory which could be in a store such as a vector database or a file or even some form of other form of memory so that we can pull them back. Taking our AI SDK agent example what you'll see here is that we're going to actually persist the messages and then pull relevant ones back. So you'll see here what I'm doing is I'm actually persisting the last message then what I'm doing is getting similar ones by pulling them out of Elasticsearch via this utility I created.

We then merge the current set of messages and the previous ones together in AI SDK to convert them to model messages for the model to comprehend. Tools consist of functions that a large language model can utilize to fetch data, perform actions, and achieve specified goals based on provided prompts. Communication protocols like agent-to-agent and model context protocol play a crucial role in enabling agents to interact and extract information to execute tasks effectively. When building tools, it's essential to consider descriptions, input schemas, and execution functions to ensure proper tool invocation and operation.

As the number of tools available to a large language model increases, confusion and operational inefficiencies may arise. Techniques such as clear tool descriptions and input schemas help in tool selection. Additionally, leveraging RAG to extract tool descriptions for relevant terms can significantly improve tool selection efficiency. Structured output involves defining data schemas for the model to produce data in specific formats, aiding in effective communication between agents and integration with existing systems. Output schemas in tools ensure type safety and specify how information should be outputted.

Context, context engineering, and tools are vital for ensuring that large language models can effectively perform tasks. These tools help agents and models understand what actions to take, where to retrieve information, and how to solve presented problems. The provided code and accompanying resources offer further insights, and audience inquiries are welcomed for the upcoming Q&A session.