Learnings From 100+ Experiments Comparing LLMs for AI Coding

Hello, guys. Glad to be here. Honored to have such an audience. So the amount of questions to Mario has actually freaked me out a bit, so this will be kind of the main part. But before the main part starts, I'll talk about LLMs. Mario's kind of depicted the Skynet scenario here, right? So everything is going to be worse. And now I will try to bring us back to the ground to the daily tasks of individual developers.

How do they use LLMs and the amount of experiments I did around that from an individual developer perspective, not that much from an organizational point of view. So this is briefly about me. I'm a YouTuber. My 10-year-old daughter is proud to tell that to her friends that dad is a YouTuber. But I'm a developer with 20 years of experience, mostly about PHP and Laravel framework. And I started creating content 10 years ago and basically never stopped.

So one of my channels, which I have four now, is AI Coding Daily. And this is exactly what I will talk about. And this is what I do on my YouTube channel. I evaluate models. I do a lot of other topics as well. But this kind of got traction. So whenever some new model or version comes up, I test it on my own. Benchmark is probably too strong a word because I'm not an organization. I cannot compete with SWE Bench or other official benchmarks. But I run the models with my prompt and my automatic evaluation from the point of evaluating the code.

So whenever some new model comes up on Twitter, you see those flappy birds and clones of various games and HTML experiments. And I personally hate those because they are trying to one-shot some project which will never go out as is anyway, which is not a realistic scenario. So I was trying to do basically this new model. I tried to evaluate, to run prompts, to run evaluations. And people started liking that. So it became one of my sexy topics basically on YouTube. And these are usual comments I get.

So similar to the question, what is the safest model to use right now? People are always curious about what do we use in terms of tools and the models. And these are comments, just random screenshots, but there are many more, which makes it difficult now with so many models being good enough. And it's not just model. Can you test Opus 4.8 high? Because high is different than medium. And there's also X high. So you see the thing, there are so many models and so many toys for me to play around with. And that's how my leaderboard was born.

In total, I have over 100 videos on my YouTube channel. So this is the result that I will be talking about, my own benchmark, which models are basically the best. But it's not necessarily, I cannot, again, I cannot compete with SWBench. So it's for my use cases, which will show my methodology and how I evaluate in a minute. But you see, basically, it's good that the screen is pretty big and I will zoom that in bit by bit a bit later. But I have 18 LLMs, score max of 25 points on five projects. And I also evaluate the average time per prompt until the end and average cost per prompt, which is API pricing.

And this is the same table, but in cost and points. So points max up to 25 on the left and the cost is to the right, which is not really the perfect way to describe that visually. I guess it could be skewed or it could be like zoomed in or out. But what I want to emphasize with this is like this is like from cheap to expensive and from bad to better. And this is basically the best place to be in. And you see how big is the competition. Many more models became good enough in terms of cost and value for, again, day to day use, that the competition is pretty fierce. Many developers, many people swear by their model, they use and they're all basically right. A lot of models are good enough these days. So this is basically so that you understand the ecosystem.

Now, how did I measure that? This is my typical prompt methodology. There's initial projects. So I work again with PHP framework of PHP language and Laravel framework mostly. But it doesn't matter that much, the language and the framework, because I'm testing not the language capabilities and the syntax. So I give a prompt to create something. And then I have automated tests to basically cover the scenarios where they implement it or not. And then if there is at least at least one mistake, at least one automated test failure, then that model failed the task.

My first test was about asserting that the data is correct with the API. Again, it's PHP and Laravel, but it could be any back end language. Second test was to implement specific to technologies. In this case, it's Laravel and Filament with best practices of multiple tools. So the LLM should be able to either be trained on them or read the docs or read any information from online to use best practices. Then project three was to analyze a big README of a really unknown package, really, really small. So it's totally not trained on any LLMs. And the README is huge. And the evaluation was, did it use the package in the correct way to avoid performance issues? So there's only one SQL query expected.

Then the fourth was React and TypeScript. So front end to create seven components with various different scenarios. And this is Playwright automated tests. And then the fifth was import CSV, which had a lot of edge case scenarios. So whether we can import the same file, we can import duplicates, we can import empty lines, we can import basically incorrect files in any way possible. And there were 29 tests around that, although it was not in the prompt. So that's kind of the main thing. Those evaluation tests were not known to the LLM. They were external tests. So I had the initial template, then the prompt, then I copied those tests from elsewhere into the folder and then evaluated the results. And this is what I've learned. So these are my comments on my benchmarks. This is a tweet by Cursor. And similar thought, Mario told in his talk that models became aware of being benchmarked.

So Cursor tweet also, not sure even how to comment that it's on screen, but basically you cannot just prompt the models and give the benchmark. So, in other words, SWE Bench and other benchmarks are known to LLMs and they're trying to compete on the benchmark. So whenever a new model comes up, there will be always, there will be the best of the best, new percentage points, new race of the charts. So those official benchmarks are not, in my opinion, they are good, they're better than nothing, but they are not the way to evaluate the actual model capabilities in a real world because those benchmarks are mostly, not all of them, some of them are better, are not on real world scenarios necessarily, and models are trying to compete on benchmarks instead of on real tasks.

Then learning number two, is over the years, I started my YouTube channel around a year ago. At the time I started with Sonnet, it was Sonnet 4.5, I think. Opus, no, Opus 4.5 wasn't the thing because over Christmas, like the world changed when Opus 4.5 was released. So it was Sonnet and GPT 5 point something, I don't remember, 5.2 probably or 5.1. And Chinese models, Eastern models at the time, they were nowhere near the quality, whichever I tried back then. And then this is the latest benchmark. So version by version, Chinese models became just better, just better at doing daily tasks. Again, my prompts are nowhere near as sophisticated as Mario's examples, and I'm not trying to predict this kind of here, but I think that a lot of developers are doing basically similar tasks day to day. So my evaluation, I think it's at least evaluating some parts of the market. So in those daily tasks, Chinese model, version by version, so for example, GLM 5.1 was bad, GLM 5.2 is like marginally better. Minimax 2.7 was really bad, and 3 is like the next level. Kimi, Kimi was the first model that kind of shook the market. So Kimi K2.5 was already on par with Sonnet. Some people, some developers use that, and then 2.6 and 2.7 was already one jump ahead. Also, there's DeepSeek. DeepSeek is its own category, I would say. So DeepSeek shook the market at that time when it was released in the first place, and now DeepSeek Flash, those models are towards the bottom of the table, but they still score 13 and 11 respectively, and surprisingly Pro was worse than Flash, but I retested multiple times, and for those specific projects, for some reason, Flash was indeed better. But look at the price. I want to emphasize the price because DeepSeek is the model. It's number one, I think, on OpenRouter or on one of those because it's really, really cheap, like surprisingly cheap. It's not even in the same like area of cheapness. So I know a lot of developers, especially who want to save on tokens heavily, use DeepSeek for implementing the plan, for example, built by Opus or more model or stronger model. So Chinese models got better. That was the main takeaway, the first. The next one was that Opus and GPT, latest models, are still much better than the others. Opus and GPT scored almost perfect.

So my 25 points mean that 5 success on 5 projects, on 5 attempts. One thing I forgot to mention, so I ran the same prompt 5 times. So that's how 25 points are calculated. And both Opus and GPT scored only 0.5 on one of the tests, and that was enough on medium level of effort. I didn't even try high or X high and also Fable because I need specific tasks for them. For those daily tasks like in my benchmarks, those models are just peanuts for them. It's table stakes. It's not even in the same league.

Whenever I see how people use Fable, it's just different types of tasks. It's just so, so much different. This is kind of outside of this benchmark for daily tasks. Opus and GPT were still much better than others, but more expensive. These are, again, API pricing, and people debate in my comments like, how do I evaluate the pricing? Because you can have subscription to Opus, Cloud, Anthropic, OpenAI. You can have subscription to Z.AI and Chinese models, and you can save a lot of money just by having a subscription. So the only way I could compare the pricing was API pricing, whether on Cloud Code or OpenCode, show the prices, and this is one of the takeaways. It's like for the quality, it is pretty expensive.

But then there is like another wave of models which started with Composer 2.5 in Cursor. So Composer 1 was like, okay, we have a model. Composer 2 was okay, something, but Composer 2.5 was a surprisingly big jump in terms of quality while keeping the price pretty good. And this price, this timing, so average time per prompt is like twice as fast as the majority of the models, and that was on Composer 2.5, not in fast mode. So you can make it even faster, then you pay a little bit more. But Composer 2.5, even not fast, it's good quality for surprisingly low price.

And then similarly, someone mentioned in the comments and I benchmarked it, mini model, GPT 5.4 mini was surprisingly good as well while keeping the price low. So if you want to keep the price low, there are ways to use basically smaller model or Composer 2.5 is also kind of its own category because it's only in Cursor. So you can have Cursor subscription and maybe save money that way. This is exactly what I came up with, with my own projects and my own experiments. This is what a lot of people do these days and this is one of the ways.

So planning and orchestrating and thinking process before writing the code should be done with expensive model like Opus or Fable or GPT 5.5. Then implementation may be, if you want to save money, may be given to Composer or DeepSeek is shaky, but possible. And then you have to have third kind of unbiased judge. Of course, it can be biased still, but from different company at least to review the code of both of those. Which means that it's not about the model anymore. It depends on the harness. So whether you use models in cloud code or open code or others, how do you prompt? What do you have in terms of context?

It depends on the harness. So whether you use models in cloud code or open code or others, how do you prompt? What do you have in terms of context? How do you have evaluations, automated tests and stuff like that? So the question becomes not which model do I use, but the overall process, starting from the very beginning until the actual result.

But, of course, I have a special cap and like in the days of PHP Channel, I create a special cap to answer all the questions like what do I use? It depends. That's basically the overall conclusion of this talk. It depends. That's my favorite meme, by the way. It depends. Whenever someone asks me anything, I'm like, well, it depends. Depends on your use case.

Awesome. If you haven't already, please share your questions on the Slido. Code is 0607. I'm going to stand here for the live stream if that's okay. Amazing. So first question I see here is, did you test the different efforts of models? Did this change order at all? Sorry, no, different one. Yes, that's the one. Yes, so in some cases there were already on the screen the efforts that I tested. It was mostly medium. I did, and that was the GPT. What was the model? I don't remember, but I did change medium and high and it did change results quite a bit. So there's no like single answer. It depends on the model.

It depends on the model. But, yes, it does change. Like Opus 4.8 is good even in medium. But, for example, what was the example? GLM, I think it was medium and high, big difference. MiniMax, big difference, medium and high. Some of them were really like really different. So you should run the same prompt in different efforts and it is likely that you would get different results.

In your opinion, are local models likely to fall further behind cloud models as hardware becomes more specialized and less practical to run locally? Right. So I've tried a few models locally on my MacBook Pro M4 standard hardware. It doesn't run well. So you have to have first to even run LLMs locally, you have to have pretty expensive hardware or tweak things a little bit or a lot to even run them at all. So in my experiments, at least for like home users, it's still way worse.

I did manage to run JAMA. Don't remember which model or which parameter account. So I did manage to execute one of the prompts in JAMA. It took 35 seconds, I think, to get even the result, to get any result, and it was totally failing all the tests. And a few more models I tried locally and it didn't even run with some weird error in OLAMA. So at least in my experience, local models are still very, very early. They are getting better. I see the tweets and the news. Maybe at some point we will get there, but for now it's really early.

They are getting better. I see the tweets and the news. Maybe at some point we will get there, but for now it's really early. Actually, I think that answers the other question we have here. Do you think investing in local LLMs is worth it? Is there anything else you'd like to add to that? The word investing is pretty accurate here. You would have to invest a lot and then, including electricity, I'm not sure you would get the result that you're expecting.

Fantastic. All right, next question. Have you already tried SONnet 5? If yes, where does it sit on the benchmark? Yes, I have, and there is a video, but I intentionally didn't put that on my benchmark because it was the same result as SONnet 4.6, but with much bigger price, which was really surprising to me. So API pricing of SONnet 5 is astronomically big. I don't remember the amount of points, but it was performing pretty similarly to SONnet 4.6. And also the same, you can read the tweets about it online, about SONnet 5. People don't really understand the purpose of that version at all because it's kind of the same price or even more expensive than Opus. So it was really weird, and in my benchmark it didn't perform well at all.

Since we're on the topic of different models, another question here is, do you have experience with Proton Luma 2.0, and how does it measure against the others? Have you heard of that? I have not. No, I have not, but actually thinking about it, in the comments on my YouTube channel, people start mentioning models, which I don't know. And there are so many new coming from unknown companies, it seems. Some of them are local, some of them are open source. So yeah, I see many more new names in addition to those 18. This is one of them I haven't even heard of. Sorry, I'm not an expert enough. I'm sure we'll see more of these models coming up and you'll probably do a video on that.

Oh, that's interesting. I didn't run sub-agents in this case unless the model decided, for example, Opus sometimes delegates some stuff to Haiku or to Sonnet. Not anymore that much, but yeah, unless the agent decided to use sub-agents, I didn't prompt that in any form, or maybe did I answer the question? Yeah, I didn't run evaluation with the same agent. The evaluation was automated tests. The evaluation wasn't done by LLM. It was like static analysis or automated tests, non-deterministic, not LLM.

Yes, for sure, and the framework that I work with, PHP and Laravel, is pretty good. Not sure if anyone used that. PHP has a pretty bad reputation. I love PHP. Anyway, but Laravel itself, so PHP is not sexy, but Laravel is. Laravel, it's very interesting when I talk at Laracon's and Laravel community. So, people hate PHP, but Laravel kind of got it back in the topic at all. So, Laravel, when you install a new Laravel project, it comes with skill called Laravel best practices. It detects your IDEs. It detects what other skills are needed. So, basically, you have .agents and .codecs and .cloud prepared for you already, so that already gives a lot of bonus points for LLMs. So, yes, in general, the results may be much better with better skills or guidelines.

Right. Unfortunately, I didn't dig deeper into that. So, I was using models as is in cloud code. Like, typically, I didn't, as I said, I tried locally, it didn't work for me well, and I decided not to pursue that path for now. Maybe that's the next step, but, again, the first experiments just kind of put me off enough, at least for now. I mean, ask me in a month, everything may change.

Oh, actually, can I get you to speak into this microphone for our remote attendees? You already mentioned around the costs of running locally, but I'm wondering in your experience with these models, were there any that had the best in terms of cost when running locally or, say, running on a third-party service like AWS, for example? Right. Unfortunately, I didn't dig deeper into that. So, I was using models as is in cloud code. Like, typically, I didn't, as I said, I tried locally, it didn't work for me well, and I decided not to pursue that path for now. Maybe that's the next step, but, again, the first experiments just kind of put me off enough, at least for now. I mean, ask me in a month, everything may change. Things change so fast over here.

So, next question. You said the harness is more important. Can you tell us more about your setup? Well, my setup for YouTube channel is all over the place, actually, because it depends on what I test and how, but what I'm saying here, for example, cloud code performs better, cloud models perform better in cloud code than open code, for example, and vice versa. So, you need to think not only the model you run, but where are you running. So, harness, the definition of harness is so vague, but I like one of the definitions. It says, like, harness is everything except the model. So, the IDE, the skills that you mentioned, the prompt, the guardrails, the evaluations, all of that. So, this is what I refer to as harness in general, and there's no, like, big secret. Again, it depends on what technology you're working with, like React or other language, so probably the harness should have skills and guardrails around that specific language, and that would give better results than just using the model itself. Awesome.

Next question. What methodology would you recommend for measuring the effectiveness of using LLMs to solve a problem or perform a task versus deterministic automation and manual processes? I think Mario's told about that in a few questions, so how do you evaluate, like, the LLM, which evaluates the LLM, and there is no definite answer, and this is why I decided on my methodology of automated tests, because I think how I found my niche in YouTube channels, so people appreciate that I look at the code at all, because some, like, LLM videos on YouTube, like, oh, there's a new model. Let's one-shot it, and yay, it works, and it's the next big thing. So, that's usually the typical YouTuber, Hype Bro I call them. So, that's why I'm looking at the code and my deterministic test. I believe in those because I think they show more realistic scenarios for people who want to care about the quality of the code and all the edge cases, and we are all running and writing automated tests anyway, so this should be the benchmark, at least for now, for table stakes tasks. Again, we're talking about pretty simple tasks like create a CRUD, create API, create something, so for my tasks, I think deterministic benchmark with automated test is the best one for now, but of course, as we get to evaluating Fable or Opus High, I don't have the answer yet how can we evaluate if it performs well because it's really hard to even review, so if Opus gives you, like, a pull request to review, like, 200 files changed, how do you review it even, like, on a human level? So, I don't have the answer how do we evaluate that level of LLMs. We have one more minute, so we'll just ask this last question.