Debugging Performance With AI

Hi, my name's Bernie. I am an engineer with Agigrid. We make a very high-performance data grid, and I'm not going to talk about our product today. I'm going to talk about some things we've discovered while trying to use AI to help us increase the performance of it. I work on theming, so I write lots of CSS. So the thing I'm going to talk about today is going to be debugging and rendering performance. The techniques I'm going to show you are quite generalizable, to debugging anything hard. So I'm actually pretty excited to be talking about this because I went down a rabbit hole a few months ago, had this amazing experience of getting AI to automate some really difficult tasks for me, and I want to share what I learned, basically.

In this talk, I'm going to give my little pitch for why I think rendering performance matters now more than, say, it did five years ago, why it's so hard to debug, how AI can help you, and my theory that AI can and should make you a better engineer. Okay, so five years ago, I didn't think much about rendering performance. All of our customers for the data grid, they cared a lot about how stuff looked. They wanted to tweak the grid to look exactly how they wanted it. But in general, we would write whatever CSS we needed to write to make the thing look how we wanted it to look, and people would be happy. It would just work. And it's different now.

From the conversations that we are having with customers over support, a lot of them, they really, really care about how fast their app renders. They have rendering budgets. So, for example, if you want your app to look smooth and render at 60 frames per second, any time you change something on the screen, it needs to be done rendering within 16 milliseconds, and if you don't hit that target, it looks laggy. And lots of our customers care about this a lot, and I wanted to know what's changed in the last few years. So, my theory is that performance is relative. So, there is no magic number where if your app is faster than this figure, people are going to go, oh, wow, great, that's a really fast app. We compare what we use to the other things we've seen, and there's been a history of standards rising.

And just to give you an extreme example, I started using computers back in the 1990s. So, think Windows 3.0, Windows 10, Windows 10, Windows 10. 1990s, so think Windows 3.0 era, and all computers were single-threaded. They could only do one thing at a time, which meant it was completely normal that if you had a button that you pressed to submit a form and it sent some data over the network, you'd press it and your whole computer would lock up for five seconds while that went. And no one complained. No one thought it was slow because instead of complaining, they were getting excited about this new thing that was called the paperless office, right? And it was the future, and your whole computer locking up for five seconds while you submitted data, it didn't feel slow because you were comparing it to the process of writing the thing down on a piece of paper, picking it up, and walking down the corridor and giving it to a guy called Dave who works in accounting. So, five seconds, that's super quick, right?

Then, of course, someone figured out multi-threaded application development, and now computers could do multiple things at the same time, and all of a sudden those old apps that lock up when you press the button, they start looking clunky and legacy. And there's been a history of this for the last 30 years. We think what we have is fast. Then we see something faster, and we're like, oh no, actually that wasn't fast. This new thing, that's fast. So, if you think about the three ways in which websites can be fast, there's data loading, scripting, and rendering. And as an industry, we have been all over numbers one and two for the last 20 years. We've got some amazing tools. So, for data loading, we've got edge computing and content distribution networks and WebSockets and progressive web applications, and all these things that basically give the illusion that just as soon as your app needs some data, it's just there immediately. Then, for scripting, again, we have incredible tools. So, JavaScript profiling tools are really good.

Rendering, we haven't really done much stuff with, because the tools to optimize rendering performance are a pain to use, and they're hard. And this also means that rendering is the new bottleneck. It is the low-hanging fruit, because we haven't done all the work to optimize rendering performance. And it's the new thing that ambitious teams are using to differentiate themselves and make products that look better than their competitors. My favorite example of this is Linear. So, Linear is an incredibly slick project management app. And they have a rendering budget of 8 milliseconds, because they want to, on the latest Mac or iPhones, if you change anything on the screen, they want to update it to 120 frames per second. And you can tell the difference. You can scroll through it, and everything feels incredibly smooth and fluid. And we know that's possible now, and we want that for our own applications. So, how do we get it? Right, well, it's simple. I'll show you the tools for how you debug performance issues, and you can see why rendering performance is so much harder to debug.

I'll show you the tools for how you debug performance issues, and you can see why rendering performance is so much harder to debug. But basically, when you open the developer tools and open the profiler in Chrome, it'll tell you exactly how long every single JavaScript function you've written takes to run. And if you have a function, and you want to know what bit of it is slow, you can just do this and put some code around it, but times a little section of it, and find out exactly how long that thing's taken. In CSS, if you think that bit in the middle is slow, you can't do that. Because CSS is a declarative language. It doesn't execute line by line. So if you want a mental model for what's actually going on and how to figure out how to make it faster, this is from Chrome's documentation for their rendering engine. And you don't need to look at all the little things. Just look at it and go, oh, my, isn't that complicated?

So this is a rendering engine. It's tens of thousands of lines of heavily optimized C++ code, and that is running inside your web page every time, no matter what code you've done. When you write some CSS, it has some effects on that. It pokes it. It makes it behave differently. Different stages might take longer. This thing is complicated both because it's rendering HTML and CSS, and they're quite complicated languages, and also it's doing lots of tricks to make the whole process faster. And there are tools where you can get insight into. Oh, yeah, the other thing to look at about this is it's a stage that goes left to right, and it has multiple stages. So I'll show you what some of the tools look like to get some insight into this. So this is a demo page on our website. It's got a grid, which is our product, to set up to have a bunch of data that it's rendering, lots of features enabled, and we're going to record a performance trace.

Open the developer tools, switch to the performance tab, go to that little record button on the left, and you record a performance trace. So you just interact with the application. I'm going to scroll here, and then stop it, and you'll get this thing called a flame graph, which is a timeline of what happened while you're interacting. There's this video at the top that you can use to find the bits that you're interested in. You can zoom in on one of the sections, and you'll see all of these, which are the events where we handled each scroll. And you'll see they're all taking 3, 5, 4 milliseconds. So that's good. That's below 8, so it's going to be at 120 frames per second. We're going to zoom in on one. On the left, you have all the JavaScript that you're executing.

And I just want you to see how good this data is. This is like a stack trace from the application. It tells you every single function and how long each of them took. If one function calls a bunch of other functions, it'll tell you what the... So this is the great information that we've had to improve our JavaScript performance. Here's what it looks like on the rendering side. It just tells you how long the rendering engine takes in each stage of the pipeline. What it's not telling you is what you... You can have a 10,000-line application, but it's not going to tell you anything about where in your application the code is that made the slope.

Right, so how do you improve the performance? Well, basically you have to do science on your application. You have to come up with hypotheses to think of why your app might be slow. And in order to come up with hypotheses, it really helps if you know lots about rendering engines, so you need to be an expert. And then you test the hypotheses by designing experiments, changing your application, running the developer tools trace again, and seeing if it changed the results in the way that you expected to confirm or disprove your hypotheses. And it was hard. It required a lot of expertise. It was painstaking. It was very time-consuming, and it was boring.

I mean, it was fun coming up with the experiments, but it was really boring doing it. And it would take hours of clicking buttons. Fortunately, AI to the rescue. If there's one thing that AI is great at, it's automating boring and repetitive tasks. So there is an MCP server for Chrome developer tools. If you install it, then anything you can do in the developer tools, AI can now do. And it's great at automating performance tasks, boring and repetitive tasks. It can also explain techniques to you, help you design experiments. The most important thing it does is it flattens the learning curve, so you don't need to be an expert before you start doing this. You can pick it up, ask it to explain the techniques, and just become an expert as you go.

So my philosophy is that AI should make you a better engineer. This means we don't just want to fix the problem, we want to find the problem, understand it, and fix it, understand why the fix worked, and then stop the fix from happening again. So if you only take away three things from this talk, this is the key slide. So the first thing is that a performance bug is an event plus a number. This is the bit that you need to tell the AI. The AI doesn't know what's fast enough and what's not, so it might be, when I switch to this tab, rendering takes more than 100 milliseconds and it should be less. So that's the concrete and mechanically testable thing that the AI can use to get its teeth into it and start automating experiments for you. You need to install the Chrome developer tools MCP because none of this will work without it. And the AI won't recommend that you install it. It'll just, I don't know, something that doesn't work. And then the last suggestion is always start by asking the AI what your options are. So don't say fix this problem, or don't even ask why is this slow, but ask it what your options are, and this is how you become an expert. This is how you build up a toolbox of techniques. So your first step is to pin down the bug. So this is where you get that event plus number pair. So the first step is to observe, so I've created a slow app here. You need to observe the problem, this is a grid that is going to be slow when we render it because we are rendering the entire works of Shakespeare at one pixel height font size. So when you scroll through it, it's going to be a bit laggy, and I'm actually amazed by how fast it does render. But you can open your developer tools. Wait for virtual me to record the video to do that. Okay, we're going to record the performance tab. We're going to scroll through. That's definitely a slow rendering issue. Stop it, and this is what a trace looks like when stuff is really slow. So those kind of red checkmarks, what Chrome puts in when something is dangerously slow, and you can see there's a little bit of time spent running JavaScript and then almost 200 milliseconds spent on the layout. So that's your bug validated. So the event is when we scroll, the number is, let's say, greater than 50 milliseconds. Let's say we wanted to render it in less than 50. So that's you having witnessed the bug. Next step is you get Claude to witness it.

So you just say, Claude, when I scroll on this page, the rendering takes more than 50 milliseconds. Can you see it in developer tools? And it will open the page, navigate to it, start a performance trace, analyze it, and hopefully say, yes, I see it. So now your chat is set up. Your context is ready for you to investigate it because Claude has witnessed the issue. So like I said earlier, start with a general query. What are my options for investigating this issue? And it will give you some techniques. I'm going to show you a few of them now. I'm going to rattle through them quite quickly. This is supposed to kind of give you an idea of what's possible in this next section. Just remember, anything that I show you in this section, you can relearn this by having a chat with your AI about what is possible.

The first technique, this is actually the one that got me excited about this, is called a regression bisection. So a regression is a special case of a performance bug where you don't just know it's slow now. You also have a version that was fast. And that good and bad version gives the AI lots of context that it needs to automatically find exactly what the problem is. So I was investigating a selection rendering issue. I had the event plus number combination. So when I drew a selection on the grid, it took more than 10 milliseconds to render. I got Claude to observe the issue. First step, ask Claude to find the commit that introduces the issue. And it will do a thing called a git bisection. So if 50 commits ago it was working and now it's not, then Claude will just go for an automated loop where it will check out the commit in the middle and it will test it. By which I mean, check it out, build it, start the dev server, open the browser, navigate to the page, start a performance trace, do the action that you told it caused the issue, figure out whether the time it takes is greater or lower than the number. And let's say in this case it does all that.

And it works. Okay, 25 is good. So now we know the problem must have been introduced between 25 and 50. And it will go through that whole process again. Let's say we test on 38 and it's buggy. Now we know the problem was introduced between 25 and 38. And this process can take hours if your build process is a long time, but you kick it off and just leave it running all day and eventually it will come back with a I've identified the single commit that introduced this.

Okay, well, in my case, when I was investigating this, it turned out that that commit was one of those, you know, one of those commits when a coworker drops an absolutely massive thing and this commit added like 7,000 lines, of which 400 was CSS. So again, Claude can help you here too. So how do I find the line or lines that introduced the issue? Again, not asking find them, but how do I find them? So it explains to you how it will work. And you start building up a toolbox of techniques. And it'll do a lines of code by section, which works pretty similar in concept to the last one. If here's the code that you know there's a bug somewhere in there, you just comment out half of it and test it and say, oh, right, this version is still buggy. So the problem must exist in the top half. So comment on another block, test this. Oh, this version works. So the problem must exist in those lines I just commented out. So you comment out the different block and it proceeds like that in a loop. And I went out over lunch, it went through hundreds of lines. And when I came back, it had located a single line that was 100% responsible for this selection drawing bug. And it was this.

What? OK, so first of all, this just shouldn't make stuff slow. And secondly, it's targeting something in the header which has got nothing to do with selection. Now, it turned out that there was some bug in the Chrome rendering engine where if you had some weird combination of scrolling containers with sticky positioning, it made everything render slowly. We'd never have found this if we were just reviewing the lines, looking for problematic ones. But once it was pointed out to us and we could see just commenting out this makes everything faster, that lets you know that you're onto the right track. So I'll take the time.

So the next technique is the standalone reproduction. So I showed you this page earlier. If you're trying to record a performance trace on this page, then there's a lot of other stuff as well as a grid that might cause a bunch of noise that could contribute to the issue. So if you have a problem with one component on a page, you just point your agent to that page and say, make me a standalone application that just has the grid and nothing else. And I did that, and it gave me this, which has got exactly the same grid setup, but none of the rest of the details. And so now when you record a performance trace, you're not going to get contamination from all the other stuff. You can also, that little button at the top that says Trigger, if you're interested in, say, resize performance, you can make a button where you press that. And it's just going to resize the grid 10 times. So if you record a performance trace while doing that, every time you click the button, you're going to get exactly the same input. And that's going to clean up your data.

Another technique I love, vibe-coded fixes. So an example of when we used this, we were looking into, like, vibe coding gets a bad rap because it's sloppy engineering. But we were interested in how to improve the performance of icon rendering. And we render icons using SVG images, but there are a bunch of other ways, like icon fonts and inline SVGs and CSS masks. And each of them would have taken about a day to make an implementation using that. But you just point your agent at it, say, vibe code me up four different implementations of the different ways of rendering icons, it doesn't matter how sloppy it is, just enough so that you can get a performance trace, figure out which one's fast, and then actually invest proper engineering effort in building one once you know it's worth building.

I think my last technique here is the synthetic stress test. We had a customer report an issue. So the basic idea of this is, if you think you have a little problem, build a crazy example to turn it into a big problem. And we had a customer report an issue where, even when our CSS styles weren't loaded on the page, so even when the CSS styles were on the page, we had no grid on the page, but they were still making their app slightly slower, which was surprising to us because our styles only have rules that target our grids. We weren't expecting it to slow down other people's applications. So we couldn't see it when we just did some basic testing on our website. So I made this little app. So I was like, okay, if our styles are slowing down other elements on the page, that is 100,000 divs. Every little black dot is a div. And you made this little app with a button to show and hide the divs, so you can time how long it takes to render them, and a button to create a grid which will insert our styles and then we can show and hide them again.

And when you have this stress test, you can see, actually, yeah, our grid is actually slowing down other elements on the page. And it turned out that that was all down to, again, another rule that looked totally innocuous to me. This rule says, find anything that's within a cell and set its height. And it looks normal to me because I read left to right. So I think it's saying, find everything in a cell and, yeah, find cells, select everything directly within them, and set the height. Turns out, and maybe I should have known this, but browsers read right to left. So what the browser does is it finds everything on the entire page, every one of those 100,000 little black dots, and makes a list of them and then filters it by whether or not it's inside a cell. So even if there are no cells, it's still running. I might go about a minute or two over. I hope that's okay. Once we had proved it with the stress test, we then realized that this is a problem, and then write better CSS if it didn't have this problem. So finally, just to clean up, like I said, I want AI to make me a better engineer. So I want to leave the codebase in a better state, but I found it. A couple of techniques for that.

First of all, custom lint rules. AI is amazing at writing lint rules. So you can just say, write a style lint rule that bans asterisks as the rightmost part of a selector for that problem that we found earlier. And it will do that, and this is it running. And it's great. Normally, when I get the red squiggles when I'm developing, it's annoying, because someone else has put squiggles in front of me. But now whenever I see that, I'm like, I made that plugin. Those are my squiggles. And it makes me happy. And also, it helps other developers. They haven't been through this learning process, so it prevents other people from accidentally doing the thing that looks like it should be okay, but actually causes performance problems.

Submitting bug reports. If you spend enough time investigating performance issues, you will eventually hit into browser bugs. Browsers consider any surprising performance issue to be a bug. And AI radically reduces the cost of being a good web citizen and reporting the bugs. So a couple of specific things you can do here with AI. You can ask it to make a minimal, like a single HTML file that demonstrates your bug so that you can attach that to a bug report. It'll make the lives of the browser developers really easy. And also, the Chrome Developer Tools MCP isn't just for doing these experiments. You can also just ‑‑ it lets your AI drive Chrome. So you can also just say, go to the Chrome bug database, do a search, make sure no one's reported this before. Actually, this one's super important, because open source maintainers are being deluged with loads of bug reports, because AI makes it so easy to submit bug reports. So getting your AI to search for existing ones first, and then if you find one, attach your reproduction to it to make that one better.

So that's it. In summary, AI can make you a better engineer, I think. In order to do that, start by asking AI to tell you what's possible so you build up a toolbox of techniques and install the Chrome MCP. That's it. Amazing. Amazing, amazing, amazing. Awesome.

That was very interesting and fascinating for me. We have a lot of questions. Like I said, if you want to ask questions, keep them coming. The code is 0607. The first one is funny. So in conclusion, is Tailwind good or bad? Unreservedly good for all use cases. Everyone should adopt it and port it. Actually, I've managed to get to this stage in my career without ever using Tailwind because I tend to build tools for other people to build applications rather than applications for myself. But if I was going to start a new application, I think I'd give it a try because it looks cool. That's all I can say. Fantastic. Thank you so much. The other one is also funny. I think this one we can discuss later. Amazing.

And then the next one we have is, why did the sticky error slow down your performance? What was the fix? It slowed down the performance because rendering engines do lots of little tricks. They have fast paths and slow paths. They do lots of little tricks to try and notice things about your application where it lets them apply certain tricks to speed up the rendering. And sometimes you can do something that looks innocuous, but actually because you've written this line of code, the browser can't prove that the fast path will work, so it switches to a slow path. And that can be really surprising and totally different between browsers and change between browser versions. And so what we ended up doing for that was, we previously had sticky positioning on every single header cell so that if the header cell scrolled off, it would stay in place.

And now we realized it was so expensive, we just enabled it when we knew we needed it. We did a JavaScript check where the sticky positioning was needed and enabled it, so you didn't have a bunch of things that weren't using the sticky positioning because they weren't in the right position to stick. But it had the class anyway. Amazing. Sounds good. Great talk. Have you applied these techniques at other troubleshooting areas? Infrastructure testing, SREs? Not personally, but someone else in our company has had great performance, had great results using loops like this to improve the performance of JavaScript code. So they were, I suppose it's not the same. You get better profiling results, but they've used agentic loops to try and find the next biggest potential performance win by running profile layers and then looking at the code and figuring out if there are any opportunities to improve it and just iteratively going on, find a place that looks like it could be faster, fix it, do the next thing.

And we've had great success leaving that running overnight and just coming back to a whole series of commits, each of which makes it slightly faster that we can review. Amazing.

Next one is, as rendering time differs between devices, what's your standard for measuring it across devices? Not great. Yeah, this is a thing we haven't really figured out as an industry. Lots of this in practice, what we end up doing is just testing on our own machines. And yeah, and there's a general problem that developers tend to have very good machines. We often don't see performance problems.

The view that I take is that I make everything as fast as it possibly. So what I do, because I know I have a fast computer, I just increase my standards. So I don't have the same view that, oh, I'm just trying to hit 120 frames per second. So eight milliseconds is fast enough. Because our product might be used by a customer who wants to do things differently and render crazy amounts of data and have 20 grids on the page at the same time and have it run on slow computers. So we just say we want everything to be as fast as possible. And we don't stop until we think that there's nothing we can do to make it faster. Amazing. Sounds good.

And yeah, the next one is, how would you turn an AI found performance issue into a repeatable CI check? Testing performance issues in CI is another thing that we don't have a great solution for. CI is quite variable. So just launching a playwright browser on CI and getting some statistics is not going to give you a repeatable figure. Not enough to be able to catch, say, a 10% performance regression. You can put in some basic regressions to catch something getting like 10 times slower, because that you can pick. And occasionally you do get bugs like that. But basically there's no solution here that we found other than manually going, I say manually, testing the thing between every release as part of a process on the same machine.

Like with CI, every time it runs, it might be on a different machine. So you need to pin it to it. You need to have a standard machine that you test on that you can compare it to how it was last release. And in our company, that's my machine. Amazing, fantastic. I think we all need Benny's machine. Amazing.

The next one is, sure, AI is great at performing tasks where one lacks expertise and delivering immediate results. However, what about the long term? How can you extend and build on this? Have you missed a chance to learn? Yeah, this is something that I care a lot about. And I've noticed it in myself. Especially if I'm a bit tired or I just want to get stuff done quickly, I can end up deferring to the result of AI. And that's why I tried to put a lot of stuff in my talk about using it as an opportunity to learn, getting it to explain stuff to you. This is a choice that we all get to make.

Like, I was talking to a friend who was a professor. And she was telling me that her university course, there's always been a huge difference between the students who just try and cheat and just get stuff done quickly and the ones who really try and learn. But AI has widened the gap because the best students are using it as their infinitely patient tutor to teach them and making them an expert. And the students who are lazier are just using it as an answer machine to crank out our answers and not thinking. And I think the answer here is you just need to make the decision that you're going to use it to carry on learning and carry on becoming better. Like, I just started trying to learn Rust last week. I don't even have a work use case for it. But I'll just be like, hey, it's easy now because I can get the AI to start explaining it to me. Let's start learning a bit of Rust. If you have that mindset, then I don't think this is going to happen to you. But it's definitely a risk. Fantastic. Thank you so much for that.

The next question we have is which use case requires CSS to load render faster than an FPS game? Interesting. I mean, I want to see the FPS game that you're rendering in CSS because I'm sure they exist. People have done all kinds of crazy things. I mean, biased here, but I'm going to say that obviously it's a data grid is the use case that requires faster rendering than an FPS game. Certainly, if there's another one, I haven't found it. Amazing. If anyone else, you can share with us after the event.

After troubleshooting, can the AI keep track of these issues to prevent them so they don't happen again? So I think your best bet here is to use the AI to come up with a deterministic check, like the Lint rule. Once you can take it out of the AI and just put it into a test, then you can put it in CI. And that's your best bet. So you want to use the AI to learn what the issue was, understand its pattern, and then come up with a rule for it. So you can detect if that kind of pattern has been accidentally introduced again. Fantastic. We'll take one more question. And then the last one here is, is there any way to use custom tracing or logging rather than trial and error? So one of the things you can do in developer tools is, so in your code, you can emit little markers which will appear in the trace. And so, yeah, if you're designing some performance-critical system and this would be more relevant in JavaScript code, if you really care about the relationship between different events that are happening in a complicated system, you can add code tracing and logging, and that will all become available in the download of information that the MCP server gives to the AI. And it can take that into account when investigating performance issues for sure. Amazing. Sounds fantastic. Thank you so much, Benny. A round of applause for Benny.