How I Taught LLMs How to Svelte

Welcome, everyone, to this talk which is called how I taught LLMs how to svelte. Before we begin, a small presentation is due, and given the topic, I brought my assistant with me. So let's see. Can you help me present myself? I absolutely can help you present yourself. You are Paolo Ricciuti. That is correct. Where do I come from? You come from Campobasso, Italy. That is correct. It's a very small city, but it's correct.

People can find you on BlueSky as paolo.ricciuti.me. That is also correct. But what about GitHub? Dietabner, what? I mean, GitHub. It's fine. GitHub. Let's see what it says. You're absolutely right. People can also find you on GitHub as paolo.ricciuti. That is correct. How would you introduce me to these people? And I will definitely mention your involvement with the Svelte project. You are a Svelte ambassador and a Svelte maintainer. That is correct.

You lead the Svelte team at Mainmatter. Mainmatter is an engineering consultancy that focuses on Svelte, Rust, and Ember. You partner with clients to solve their toughest challenges or adopting Svelte, depending on where they are. And that's absolutely true. We do what's called team augmentation. So I get to work with Svelte but also teach Svelte on my day-to-day job, and that's the best part. Remember, like, always thanks your overlords. So, thanks. And we can move to the next slide.

Oh, it's stinking. Let me remove this conversation. What? It broke itself. It's fine. I'll fix it later. Let's talk. But first, who here, for some reason, it's already on the next slide, who here knows about Svelte? So, by raising hands. Okay. A lot of people. The rest can go, please. No, I'm joking. You can stay so you can actually learn something. So Svelte, it's a UI framework that uses a compiler to let you write break takenly concise components that do minimal work in the browser using languages that you already know. HTML, CSS, and JavaScript.

It's a love letter to web development. And I do really think about it because this is a Svelte component. And as you can see, like, it's very close to HTML. You have the script tag where you can declare variables. You have this dollar sign states rules, which is a syntax of the Svelte language that allows you to declare reactive variables. You can import and do stuff inside the script tag. Then you have the template part, which is basically what gets rendered to the browser. And as you can see, like, you can use handle brackets to render a variable.

Everything that you declare in the script tag is also available in the template. You can also change variables without using set count, for example. Like count is declared as a reactive variable, but it's just a normal variable. It's a number. You can just do count++. You also get styles for free. So I don't have to use a class here because every style is contained in the component that declares it. I can just use button and these styles will apply only to this component.

So that's very nice. However, so this is a Svelte 5 component. And you notice that I have to do some kind of specification. Like it's a Svelte 5 component. The reason is that in Svelte 5 we had our biggest syntax changes. So this is how it looks like with these dollar sign props to receive props, the dollar sign to declare variable, dollar sign derive to declare derivations, dollar sign effect to do effects inside something that is not tied to the Svelte reactivity, dollar sign inspect to log your variables with this onclick, which is just a prop of the button. And this is how it looked like in Svelte 4.

So we have this export let to export a prop. Count was just a variable and it was automatically reactive. And then we have this weird syntax with this dollar sign colon, which is actually valid JavaScript to both do derivation, effects, and even to log. And then we had this oncolon click, which were events directive. So it is very different. Like if you go back and forth, it is very different. And unfortunately we kind of released this basically when agents were starting to become a thing.

And you know who really doesn't like big syntax changes? Obviously, you guessed it, our future overlords. So for a bit, this was the average experience of people that wanted to write Svelte 5. This was not that great. If you think about it, it actually makes sense. Because if we take this graph, this is the amount of Svelte 5 code out in the wild. And now Svelte 5 is completely back compatible. So this is the amount of Svelte 5 code out in the wild that looks like Svelte 4. If to this we add the actual Svelte 4 code, and to this we add the fact that Svelte 3 was literally the same as Svelte 4, you can see how the imbalance in terms of training set is actually justifying why it was doing such dumb mistakes.

So this is the amount of Svelte 5 code out in the wild that looks like Svelte 4. If to this we add the actual Svelte 4 code, and to this we add the fact that Svelte 3 was literally the same as Svelte 4, you can see how the imbalance in terms of training set is actually justifying why it was doing such dumb mistakes. For example, it did this. It tried to import the runes when in reality these are just a construct of the language. It's just like the import keyword in JavaScript. It also tried to do stuff like this. Like this was one of the big frustrations, like the on colon click. It's so similar to the on click that they constantly get it wrong. They also did something like this, which is more subtle because derived actually gets an expression, not a function. But this means that you can likely miss this error from the LLM. They also did something like this, which basically mixes the runes version, which is the new version of doing reactivity, with the old version of doing reactivity in JavaScript file, which was store based, where this count would have been a store and you had to update it with dot update. They also did something like this, which is super weird, but I've seen LLMs declaring this, which is basically this dot dollar sign at the end for some reason. And they also did something like this. This is actually correct Svelte code, like it's not wrong, it's just semantically wrong because double should actually be just a derived. Reassigning state inside an effect is bad code. Although I have to say, I've actually seen a lot of engineering doing that, and I want to know that I'm judging you. So the problem with LLMs is that once their training set is done, they are basically frozen in time and you can't add more context to them. Well, you kind of can, like you can add information through context and not that context. The context that you add to the LLM is just you prompting. That's how they can work on your code base. The problem is that if anytime you have to write something, Svelte, you have to start declaring everything that Svelte can do and how it works. I mean, I wrote a script to add this to Claude, and I'm already bored about it. And so during Advent of Svelte 2023, I think 2024, we released LLMs.txt, which was one of the early attempts to allow LLMs to get the information that they need from the documentation website. And now if you go to the Svelte docs, you can append LLMs.txt to every documentation page, and you get something like this, which is basically the same version of the documentation, but in markdown files so that like you don't waste the tokens. And this was a bit good, but this standard never really catch up. And so we were getting constantly bombarded with, I want to use Svelte, but I cannot because AI cannot write Svelte. And I want to use it, but then my co-worker will not be able to use AI to build the Svelte component correctly, and I have to constantly steer it, and I have to, like, so we received a lot of information. And so during our monthly maintainer meeting, we said, okay, we need to do something about it. And we actually started an actual task force specifically for this. Not because we want to help LLMs, but because we want to help our users that want to use LLM to be able to do it with Svelte.

And so what we said, we say, okay, let's create a Discord channel. We put everyone that is interested in there, and we actually start throwing ideas at the wall until something sticks. And something did actually stick. So if you take a look, there are more mentions about something that's called MCP. What is an MCP? Well, MCP stands for Model Context Protocol. And model is literally the same M that we are in LLM. So it's the model. It's the LLM model. Context is what we just talked about. And protocol is a way older word, and it's just a way for two parties to communicate with each other with a shared language.

In this case, in the case of MCP, it's JSON RPC 2.0. The point is that there are some MCP clients, like Cloud Code or OpenCode, Py or Goose, but there are literally a hundred of them. And they are not the actual LLM, they are kind of wrappers around the LLM. And they also can connect to MCP servers. And the MCP server is just a program that is either through HTTP or through standard in and standard out, listening for this JSON payload. And then it can either do something in the real world, for example, the linear MCP server could close an issue for you, or they can send back information to the MCP client, which then feeds that information to the LLM.

So they can send back the information to the MCP client. For example, the Chrome MCP tool is very useful because it will open a Chrome instance and it can actually get back the accessibility tree, for example, for the page. Now MCP is actually pretty funky and you can do some pretty funky stuff with it. For example, this slide deck is actually an MCP server and it's connected to my OpenCode instance. So I can say, can you go to the next slide? And if everything works, yay, it worked. And this should have been confetti here. For some reason they're not. The point is you can do some pretty funky stuff.

Now I'm not going to use this because that would be the most expensive clicker ever. The point is at the time we wanted to use MCP, but I was still pretty skeptical because what do you do with an MCP for a framework? You can inject docs. And so that was the initial line of thinking. So Scott Spence, which by the way, if you are interested in AI, you should absolutely follow because he's always on the bleeding edge of AI. He started experimenting with this MCP Svelte docs to feed the Svelte docs to the LLM through the MCP. For his own admission, it didn't go as far as he wanted. So Stanislav Koromov took actually the mantle and started developing something that actually more people started to work with, which is the Svelte LLM MCP. And it was just basically feeding the Svelte docs the same LLM.txt that you can read through the MCP protocol. And this was actually kind of working and we actually started from this to build the official Svelte MCP server.

But if we go back to this slide here, you can see that other than we can use the MCP, there's also another idea hanging around. And this idea is the fact that we could run a script of some sort, some deterministic way to determine if the code is correct. So this is the normal like agentic workflow, right? You the user write to an LLM, the LLM write the code, then verifies the code, then write the code, then verifies the code. When he's happy about it, it will go back to you. Now what we did was basically add another part of this agentic workflow. So you write the code like to the LLM, the LLM writes the code before actually going back to the LLM, it goes through the Svelte MCP. And the Svelte MCP, what it does is that it runs static analysis on the code to find out if the LLM generated some common mistake, like the fact that it's using .update, the fact that it's using .$ to read a variable. We have a list, basically, of common mistakes.

And every time we find a new common mistake, we just fix it with static analysis. The nice thing about this is that not only you can return stuff that you would never return to a user because you will never tell a user, hey, maybe you are doing $, like .$. Are you sure that's actually a property on that object and not something else? Like you would never tell this to a user. But most importantly, since we are talking to an LLM, we can return information in natural language. Like we don't have to fix the code for the LLM. We can just return some suggestions and then let the LLM figure out. So let's play a small game, and let's say that we are the LLM. This is an editor that's actually connected to the actual MCP server. So if I do stuff like this, and I have to look at that. So let's say that I do, I don't know, on call on click, for example. Here on call on click, you will see that the LLM is saying, hey, using on click to listen to the click event is deprecated. If I do stuff like this, for example, .$, even if this is technically correct Svelte code, I'm saying you're trying to read the stateful variable count using $. Stateful variables should just be accessed at runtime just like normal variables.

And the nice thing about this is that we can also steer the model to write proper Svelte code. So let's double $state, zero, like we were doing before. And I do $effect double is equal to count time two. I don't know if I made some mistake. Yeah, there's an un... Here, I saw it. Okay. So you can see that now, even if this is actual normal Svelte code, like it doesn't have an error, it's saying the stateful variable double is assigned inside an effect. Please make sure that that thing is something that is not creating a problem. You could use a derived for that. So this actually worked surprisingly well. So if the video loads, yeah, correctly. Now this is not obviously like... It works surprisingly well, but it's not like a bullet, a silver bullet.

For example, here I was using a Quen3 because I always run the MCP on very dumb model because I want to be able to write proper good code even with local models, for example. If I did that, as you can see, like after 30 minutes, I just had to shut it down because it was just not fixing the stuff that needed to be fixed. Same thing, for example, with this. Hopefully it will load. It will not load. That's an issue. Okay. I don't know why. There is a small... Okay. This is GPTUSS20Billion, which was actually able to write proper good Svelte code once.

But as you can see, it was starting to basically go crazy and say, okay, I'm stuck. I'm stuck. I don't know what to do because it was not smart enough to do what the MCP was about telling to do. But when it works, and once again the video will load slowly.

But when it works, and once again the video will load slowly. When you will get to that, we'll get to that. When it works, it's actually pretty interesting. In the video that is about to load, Mistral, which is also a pretty dumb model, basically one of my favorite prompts is building a Minesweeper. He actually started to use the MCP by reading the documentation, by getting the sections, by I don't know, for some reason he started to do an image of the design. But then he basically generated some code which was very bad. This was basically Svelte 4 code. It didn't work with Svelte 5. But it started to use the auto fixer, and over and over and over again you will see in a second, it fixed every issue that the auto fixer reported, and in the end, it actually got a surprisingly good Svelte 5 code.

So, obviously we don't ship just, again, the video is loading slowly for sorry. But we don't ship just the MCP. We also ship with resources, which are basically files that you can add. And those are just the same files that you will get as LLMs.txt. And you can just add them directly in your chart, which is much, much better. And we also ship with prompts, which is a way for you to steer the model into using the MCP without having to type everything by hand. And if your agent doesn't support that, you can actually just copy it from the Svelte docs. We also ship with this nice tool. Like, if you are using, if you are prototyping in Cloud.AI, for example, they will not be able to run the Svelte code that they generate. And so we ship with a Playground link, which actually allows you to open the Playground link with the code generated, and you can play with it. And if your agent supports MCPUI, it will actually be directly into the LLM.

Well, not so fast, unfortunately, because if something developers really, really hate to stay on the same thing for more than two weeks, especially in AI. So we started with MCP, but then some developers say, well, what if I can use the CLI and maybe some skills and maybe even have a look for context management. And so we actually did it. And once again, OK, it's loading. So now the Svelte MCP is actually also CLI that you can use directly in the terminal. And you can just run the same commands like list section to get the list of the section. You can run get documentation to get the documentation directly in your terminal. You can also run the Svelte autofixer on a file.

You can run get documentation to get the documentation directly in your terminal. You can also run the Svelte autofixer on a file. So you can just do this is to get documentation, for example. And you can also do Svelte autofixer passing a file so that it will autofix that file and you will get the same result as above. However, how the agent will know about this? That's a question for skills. So we shipped two skills, the Svelte code writer, which basically just teaches your agent how to use the Svelte MCP through the CLI. And another interesting one is the Svelte core best practice, which is a more pure skill that just teaches your LLM how to write very good Svelte directly from Rich Harris, which was the creator.

And another interesting bit is that this is not something that only agents are interested in, right? And so Svelte claims to be a human first framework. And so actually, this is just a documentation page. So when you are learning Svelte, you can learn the best practice. And then we turn that into a skill so that agent can also learn, too. Obviously, once again, the skills are available on the documentation if you want to download them.

Let's talk about context management. So context management, we know it's an issue with AI, like you need to have context as low as possible. And one way to do it, in my opinion, and something that a lot of people actually sleep on, is subagent. So subagent allows an agent to spin up a version of itself with its own context window. And you can literally run in this closed subagent, you can run every atomic operation. And so we actually shipped an official subagent that tells the agent, hey, whenever you have to write a Svelte component, don't do it in the main loop. Just invoke the subagent, do every MCP call or CLI call back there, write the component and report any issue back to the parent.

So this way, the documentation will never leak into the official agent. Obviously, this will be pretty boring to set up, right? Well, luckily, we actually ship like the Svelte AI tools ship as a Cloud Code plugin. So you can just install the marketplace and install this Svelte plugin and you will get everything. We also ship as an open code plugin. We ship as a cursor plugin. We ship as a GitHub plugin. I mean, you guessed it. Like if you have anything that has plugin and we don't support it, open an issue and we're going to fix it.

First thing to talk about benchmark. Stanislav also created the first draft of the Svelte benchmark, which basically was a way to test LLMs on how good they were at writing Svelte. But this did not support skills or anything else. So we actually created an AI SDK bench with the AI SDK to run the same benchmark with the MCP or the skills. And we also created Svelte evals. We actually run a much broader test with an actual harness on SvelteKit pages. Now that's it. I need to run. I'm out of time. So if you're interested, you can visit the Svelte docs. If you're interested in how MainMatter can help you, you can visit MainMatter.com. Also, we recently released a series on YouTube about agentic engineering with Svelte. So if you're interested in that, you can also open that. And as always, thanks for watching.

Yeah, it worked. The first one that I highlighted here was, can you achieve the same thing with the existing tooling kind of like ESLint or any kind of existing tool. So first thing first, like yes and no. In the sense that we actually run ESLint as part of the auto fixing pipeline. So yes, those are part. You can technically write the same auto fixer like plugins as an ESLint plugin and then run it in your local CLI. What I noticed is that first, agents are much more likely to invoke an MCP tool multiple times and they just run linting at the end. So that could absolutely work if you want to have the time to just write down all the issues. I guess the advantage that you have over an ESLint rule that you grow in town, like in your code base, is the fact that we are actually like the repo is actually a collector for normal mistakes. Like when people notice a lot of mistakes, they come to the Svelte AI repo and say, hey, the AI has run into these multiple times. So we can kind of collect the common errors inside the auto fixer repo.

Okay, cool. So that means that a lot of these things can be done by the existing ESLint tooling. If you want to set up everything, you can absolutely. If you want to set it up. Yeah, another thing that was kind of a little bit related that I saw here is that what if you could ship the agent skills? So like you were talking about agent skills. What if you can ship those in Node modules? So when you do npm install or something like that, you just get it in your Node modules. I think there is a, it's an interesting idea. And I actually seen agents actually go into Node modules in the literal code to figure out what's wrong, especially because Svelte is written in JavaScript and the type check is done with JS doc. So like you actually get the actual code that is running. So I've seen agents actually go into Node modules and read the actual code to fix an issue. So it's definitely an interesting idea. The downside is that then if you are not using skills and you are not using agents, maybe then you will still get that like small bit, small package of skills in your install site. Which I mean, it's fine because like everybody knows the modules folder is like already 30 gigabytes. So I guess that's just the one downside.

Yeah, so what I'm hearing is that it's a lot about coming to the users and using the same tools that they are using, giving them all the possibilities in whatever ecosystem that they are in. One thing that's interesting to me as well is like the adoption question. So I highlighted that up at the top as well. So Svelte is something that is maybe not in the same kind of like popularity as something like react or something like that. So how does how does Svelte compete or how does it how do you see the adoption with LLMs and these kind of like micro apps and stuff like that? So I think like in terms of adoption personally, I think it's growing.

Maybe it's just because I'm in a bubble, so I don't know. But I think that is growing. And in terms of how it will differentiate itself from view of web components, if you have been following the latest and greatest from Svelte, it's actually like just recently we are doing a lot of work in Async Svelte for example. So right now you can literally await something inside the template and it will work and it will work. How do you expect? And also specifically for Svelte, we just like we it's still experimental, but we are doing remote functions, which is, in my opinion, the best way, like the best abstraction around data loading, because like it's literally just a function that you declare in a different file because we don't want the server code to be mixed with the client code. Like it's a separate file, but then you can import in the client and just call it as a function. And also, most importantly for forms, I think the form abstraction is the best one because you also get like, for example, auto completion about what the field names are, which is something that is always very hard to do when you do forms. And like you have to keep the two sides, the client and the server in sync.

Nice. Cool. So I'm hearing there's a lot of like kind of batteries included kind of things, things that that are included there. The next thing that I moved up to the top here is. So if you were to sell to a React audience, what would be what would be something that is in the Svelte ecosystem? I mean, you talked about forms now, and you talked about these functions that that is not really in React or something that's really compelling. So I think to be to be frank, like at the end of the day, like every single framework, you have a way to do what you have to do. It's just about the DX. So I think like what differentiates Svelte from the rest is that we do really care deeply about your DX. Like we really care that you have the most like natural API for something, for example, remote functions. You want to await something. We could go. We could make you go through like a bunch of hoops to make it work. Or we could do the hard work and make you just write a wait something and it works and it works magically because like we are a compiler and we can do a lot of work behind the scenes. And so that's definitely something. It's also one of the fastest framework anyway. And in general, it's very close to what normal JavaScript looks like. So there's nothing funky about. There's nothing surprising about it. Yeah.

OK, that kind of makes sense. Another one here is this the auto fixer. So if it is if it is not understanding the Svelte component props, is this something that can be it can be made aware of or.

I'm pretty sure. Who did this question? Raise your hand if this is about a warning that you get when you pass the prop into a state. I don't know if it's still here. It was like that. Yeah. So that's an actual thing that is not just about Astro. Like, it's a warning that we are dead and we are thinking of way of going around that thing. Like, the problem is, it's true in Astro. It is static. It can also be static in Svelte. We can think of way. Please open an issue on the repo and maybe we can think about something to signal that. Nice.

Yeah. So I'm guessing that you haven't gone in this direction yet, but it's kind of an interesting idea like the language agnostic MCP linter that has something something like auto fixing capabilities. So, once again, I think one of the nice things about the Svelte auto fixer is that it's not fixing the code for you. It's just telling the LLM to fix it and how to fix it. I think that's a really important distinction because there's only so much that static analysis can do. And also make it agnostic would be very interesting. But every single project has different rules. Every single project has different ways in which the LLM screw up. Basically, so like in Svelte, we have a collection and I think that collection of smaller like of similar errors is what makes the MCP powerful like it is. Nice.

So with did you use any rag in this? Like when you were trying to teach it about Svelte kit routing or anything to provide the latest documentation or how did you do that? So this was actually like this were ideas that were thrown at the wall back then. The problem with fine tuning a model to that. The problem is that then if a new model comes out, we have to refine tune that new model. If you have a closed subscription, you cannot use our model. So the reason why we choose MCP is that is because we wanted to like you to plug in into whatever system you have. We also thought about rag for that. And I mean, it's not really rag like we are just fetching the latest documentation directly from the docs. And that's why, like, again, the rag is done through the MCP into your LLM, basically. Great. Yeah, I think that's all the time we have. But yeah, give it up for Paolo.