Skill Design for LLM Agents

Hello, everyone. My name is Minko Getschev. I'm AI lead across web and multi-platform development at Google. And today in this presentation, I'd like to share with you how you can build and design skills for agents that actually work. In my presentations, I usually like starting with first principles. So that's why I'm going to start by introducing agents and the building blocks of agents. See how they're composed together to get the behavior that you'd expect.

We're going to look into skills and how they fit into the picture. After that, we're going to look into best practices for creating skills. And last but not least, we're going to look into evaluating skills and self-improving loops for getting better skills. We're all using agentic systems every single day, interacting with them through the web, with Gemini or Cloud or chat GPT, or in our development environment. These agents work very similar to ANR. They all accept a user request, use different prompts and accomplish the execution of these prompts through a set of tools.

Utilizing a large language model as their brain. We have control over the tools that these large language models have, and we can also augment their context. And probably the most popular way now to augment the tools that agents have is through MCP. MCP has its trade-offs that we're going to look into in a little bit. But overall, it allows us to provide an extra set of tools that can make our agents more effective. Common architectural patterns for agents are the React loop, in which the agent would consult with a large language model to see what it should do.

The large language model is going to make a proposal, and the agent is going to follow it. We have plan and execute in which we create a plan at the beginning. I'm sure you've seen that in Cloud Codes. I'm sure you've seen that in Antigravity. We're making the plan and after executing it step by step.

Reflection. We can reflect over this plan. We can revisit over time. State machine-based agents, which are pretty fine in a way, have hard-coded individual states, and the transitions between them often depend on some non-deterministic result, like action, that the agent performs through the large language model. The vast majority of agents nowadays use a hybrid approach, combining different architectures.

To understand how agents use skills and actions or tools, let's delve into the React loop, a classic agentic architecture. First, receive the user prompt and concatenate it with system instructions. Enter the loop and gather a summary of all available tools. Ask the large language model for guidance based on the context and available tools. The model can either signal task completion or select tools to make progress, adding results to the context.

And we ask the large language model, cool, here is our context that we have. This is the state of the world that I'm aware of. And here are all the tools that I have available. What do I do next? Here, the large language model can do two things. First of all, it can tell us, all right, you're done. You executed your task. We can terminate. Or it can select tools that we have available to call them with specific arguments in order to make progress in the execution of the user request. If we're not done yet, we're going to call these tools and the result of their invocations we're going to add to the context.

Notice that we keep making the context larger and larger. We keep appending to this context. So in large React loops, we can experience context drops. You might be thinking, yeah, but like Gemini has 1 million tokens, context window. How can we, like we can do a lot of work until we reach this limit. Well, all the modern models right now have really large context windows. Some of them are striving to get to an infinite context window. But the truth is that after a certain threshold, the performance of the model starts to degrade about 30 to 40 percent. And we're going to experience regression. Usually, a lot of agents are doing their own context compression.

So taking all this into our mind, adding it to our context, how do we make agents more powerful in a thoughtful way? Well, we can enhance the context carefully by adding only relevant information, and we can do that through RAC or through skills. Or alternatively, we can also provide extra tools. You actually saw how we can do that with MCP. And probably the two most popular ways of making agents more powerful that I've seen is through CLIs and MCPs and adding useful skills. Often people compare MCP to just adding a CLI tool that the agent can use. And I personally find this analogy to be kind of this comparison to be very relevant, especially because MCP kind of started as this universal way to expand agent capabilities, and now we see that people have been a little bit more skeptical using the model context protocol for good reasons.

Often people compare MCP to just adding a CLI tool that the agent can use. And I personally find this analogy to be kind of this comparison to be very relevant, especially because MCP kind of started as this universal way to expand agent capabilities, and now we see that people have been a little bit more skeptical using the model context protocol for good reasons.

For example, CLIs compared to MCPs are better in context preservation. No context blow due to very verbal JSON schemas. Additionally, authentication and permissions are handled usually easier with CLIs because we can leverage existing user credentials and use single sign-on. MCP, the model context protocol, uses an MCP server that needs to be available, needs to be up and running, needs to accept requests. None of this is valid for CLIs.

CLIs just exist there as command line tools that you can just call at any point. You have zero infrastructure overhead. And also they're extremely easy to debug since they run in the user context. They are just a script that you can debug and you can write easily test for. And on top of that, CLIs, they work great with skills.

So CLIs are the mechanism to accomplish something, and the skills are the mechanism for us to teach the model or the agent how to use these CLIs. The agent skills are just procedural instructions on how to teach the agents to do something. They have three main parts, let's say three main pieces. We have the skill.md, which is the core entry point. This is just the instruction. Usually, most skills are honestly just captured in skill.md.

We have scripts that are deterministic scripts that can run. They can be referenced by skill.md so that the agent can call them instead of following procedural instructions written in markdown. And we have references and assets. The reference and the assets, they're very useful when we'd like to keep skill.md kind of slim and very focused on step-by-step instructions. We can reference the references in skill.md.

When I talk about skills, people often confuse them with CLoT.md. And these serve very different purposes. First, skills are an open standard. They became a standard sometime at the end of last year. They're specifically focused on tasks and workflow logic, where CLoT.md is focused on repository-wide or subdirectory-wide coding guidelines. They're obviously CLoT-specific or Entropic.

They're specifically focused on tasks and workflow logic, where CLoT.md is focused on repository-wide or subdirectory-wide coding guidelines. And they're obviously CLoT-specific or Entropic. Now, we know that skills are useful. Does that mean now we need to go to Gemini or any of our favorite AI providers and just create skills for everything? I'm actually going to recommend you against that. If you go to Gemini and create your own skills, this means that the skills are most likely unnecessary, because the model that the agent uses already possesses the capabilities to accomplish this task. So these skills are just redundant and they will be leading to extra context rot.

How do skills work? Remember our React loop. Let's plug in skills now. So whenever we're accomplishing a particular task, imagine that just a different iteration of the React loop, we would look at the available skills that we have, and we'll create a summary of them. This is just going to be the list of the descriptions and the names of the skills that we have available. We're going to ask the large language models which skills are relevant for this particular task, and if there are any relevant skills, we're going to add them to the context. Notice how many things we're adding to the context. That's why I'm saying we need to be very mindful about the skills that we're adding and also the way we're expanding the agent capabilities with MCP or with CLIs, because everything ends up being part of the context, and this is something for which we pay and something that can regress the agent's performance if it grows too much.

Building skills over the past year or so, or six months, I don't know, whenever the open design, whenever the skill specification came out, I discovered five best practices for the development of agent skills. First of all, what we need to put in skill MD. They need to be procedural instructions. I'll show you why in a little bit. Second, the front matter optimization is really important. Progressive disclosure, so we can keep our skill.md pretty slim. Repetitive task scripting and skill composition. Procedural instructions. If you're writing documentation for humans, you can provide examples, you can make comparisons between your framework and cooking, let's say. You can make different analogies, different pros that could actually be helpful for the user to comprehend how your technology works. However, if you're building skills, these skills are not for humans, they're for agents. And the agents, they don't need pros, they need procedural instructions that are very clear in describing the agent what it needs to do. So instead of saying, to run tests, first check if npm is in, so you can look for packages and so on and so forth, instead of doing that and making it hard for the agent to follow individual steps and discover them, you can just say, first search for package.json file. If yaml log is present, execute this and that. Focus on predictable, repeatable execution. Next, if you have the best skill.md possible, it might still be relevant if you don't have good front matter. You can either pollute your context or your agent might not be able to discover the skill at all.

Here is why. Here we have a skill name that is very generic, TestRunner, run repository tests. So this might be potentially loaded for every single task that you have that is related to execution of tests. But you might be targeting only the execution of your front-end tests, for Angular or React, in this case for Angular. So Angular testing, run Angular unit tests using Karma, do not use for React or Vue. This negative trigger is usually a best practice so that you can make sure that this skill doesn't end up being part of an agentic execution in which this is irrelevant and just pollutes your context.

Talking about skill, I've noticed that a lot of people would install a lot of skills and they'll forget about them. They'll end up having hundreds of skills, and this will regress the performance of their sessions and it would make them spend more on credits or on their AI tools. There is active research that shows that between 20 and 30 skills is ideal. If you're building custom agents or if you're consuming agents and you've installed a lot of your skills, make sure you keep them, let's say, under 50 ideally. If they reach 200, you already start experiencing accuracy drop, and with 500 you better delete them all. You're just consuming a lot of unnecessary tokens into poor agentic executions.

In order to keep your skill.md pretty small, I'll recommend you to leverage the progressive disclosure. In skill.md, make sure you keep it pretty lean, under 500 lines, and make sure you have short columns, short line size. This way, you can reference every other resource that the agent may want to conditionally load in your references. You can just reference your references in your skill.md so that you can keep skill.md leaner. Repetitive tasks, make sure you use scripts. Look at the example on the left side. It is just English. I'm saying search for stats, replace with stats, set this environment variable, verify. There is ambiguity here. This is English. This is not a code that would necessarily get executed in this order that I have specified. With recent models, yes, they're really good in following instructions, but we still don't have a 100% guarantee that they will follow them in the order that we specified. Additionally, each one of these lines is associated with multiple tool calls. This might be potentially a round trip to the model. It could be significantly less efficient compared to just encapsulating these steps into a deterministic script that we are referencing in skill.md. That will be my second to last best practice, repetitive tasks, scripting, use scripts for repetitive tasks. That's it. The last one is skill composition.

Here we have build project. This is our router skill that is referencing our client build and our server build. Now, the agent can conditionally load either the client build if it is interested in building the client or the server build. This way we're keeping our build project skill pretty lean, and we have these other specific skills that the agent could potentially load conditionally on demand.

I spent a lot of time talking about scripts versus these procedural instructions. You can see that here on the left side, we have a skill that is doing something with some Salesforce partner portal, and we have an equivalent script. There are obvious differences here. First, if we can automate something, if we can make it deterministic, we should make it deterministic and create a script because we can test it significantly easier. But there is also some similarity between them. Both of them express some kind of computation that we need to make sure we test so that we can make sure it works.

I know that with agents, it's not obvious that the English that we write needs to be also tested to make sure that it provides good instructions to the model. That is why I would like to talk a little bit about testing skills or skill evals. We need to evaluate skills just to make sure they are useful. If running a workflow with and without skills yields the same results, probably we should remove the skill. We need to also make sure that we test skills to prevent regression. Imagine we delete a paragraph. How do we know whether this is either a positive or a negative outcome for this specific skill? There is no way, really.

We need to evaluate our skills and test with and without, and have these continuously running evaluations that give us actionable insights. In the same way, we can remove unused skills or even implement something called hill climbing or self-improving skills that I'm going to talk about in the end.

The testing pyramid that we have from software development that you're very familiar with is very relevant for agentic workflows as well. Right now, we're going to look at the bottom of this pyramid, skill evals. About skill evals, there are some good articles written, like for example, Skill Bench, which is a great paper that just tests skills, how well they perform across diverse set of tasks. There is a great paper, a great blog post by Entropiq, also demystifying evals for AI agents, that on a very high level describes that in order to create an eval, we need to identify goals.

Around these goals, we need to create tasks. And we need to run these tasks multiple times in order to verify whether our agents did well. We can evaluate the agent performance with scripts or deterministic graders and with non-deterministic graders or ALM agents here. We can additionally use humans for the purpose as well. Based on these two papers, I created a tool called SkillGrade that I'm describing as unit tests for your agent skills.

SkillGrade is, I'd say, probably the easiest way for you to create evals for your skills. All you need to do there is install SkillGrade with NPM or PMPM. After that, run SkillGrade in it. Make sure you provide an API token for your favorite AI SDK or provider for Entropiq, OpenAI, or Google, and you run SkillGrade. And that's pretty much it.

Let's see it live in action. So here I've provided Gemini API key and I've run SkillGrade in it. This automatically reads my skill file and based on whatever I'm describing, it generates an eval. Once I have generated this eval, I run SkillGrade smoke so I can run smoke test with five calls of this eval.

All of them execute securely in a container and I have my deterministic and non-deterministic graders to make sure that the skill was actually useful and find gap in it. Once I'm done with the execution, I get a report similar to this one that I can preview in the web browser. So create user schema here. This particular skill has 80% pass rate.

And I did five executions of the workflow. One failed, so I can preview the result of the deterministic and the LM rubric. I see why it failed. And I can preview the log of the execution of the agent, which is really powerful. Because later, I can analyze this log of execution, and I actually can make meaningful improvements in my skill. Let me show you how.

So here, I'm setting up the skill. And I'm executing the assigned task in a sandbox, in a secure sandbox. So this is running in a Docker container. This is the execution. Make sure, look at the trajectory log. It lists everything that is happening. I'm executing different commands. These are the commands or the tool holes that I already talked about at the beginning of the React loop. These are the thoughts of the large language model.

This is like everything. That later on, skill grades can audit. And it can find what are the missing things that caused my skill to fail. Last but not least, I can apply these changes, and I can rerun my agentic workflow again with these two new additions that I added. And once I run it again, you can see that we go through the same steps. Agent, execution, trajectory log. We're looking at everything that happened throughout the execution of this particular task. We're evaluating, and we see that everything passed, so we succeeded.