Effective Thinking in the Age of Augmented Tooling

What I'm going to talk about today is really about effective thinking in the age of augmented tooling. My background is when I went to university, I studied a liberal arts program. It was an advanced program from the University of Texas. It's our big state school. And I studied sort of an interdisciplinary program. I only took one computer science class in college, but it was a really strong program in physics, math, science, you know, the full gamut. And I think that that ends up leading into part of what's interesting now about this conversation.

I did specialize and get very involved in user interface development and human-computer interaction. I really liked the feedback loop you get when you're directly connecting with users in touch ways. So mobile interfaces, web interfaces, the like. And then I came to Vercel about six years ago to continue a project journey that I started at Google 17 years ago on a thing called App Engine, if any of you remember it. But the idea was very similar. You build an application on your machine, you type a command, magic happens, and you're serving on the public internet. That was just too hard to do for most of my life up until that point. And it's still hard to do doing it on your own. Vercel makes that easy. So that's what we want to do. And really here, you know, AI has unleashed kind of a shift in software engineering.

There's this mixture of absolute excitement and a little bit of terror, I think, in everybody's minds about it. But something fundamental still lies beneath this, and it's the value of saving time. That's really our job on behalf of our users and customers, etc. Early on, like savings time is the thing that got me into being a programmer. How many of you have read the GNU Manifesto by chance? Okay, good. You all have something you could do that I think will be enriching for you. GNU Manifesto is the foundational document for open source and free software, upon which, you know, the generation of tools you all use, we all develop was built on. And really, the GNU Manifesto's premise was that as a software engineer, you should be saving time, building on the shoulders of giants, taking software that's been built, learning how it worked, which you can only do with the source code, and then adapting.

So, but this was kind of controversial at the time that Richard Stallman wrote it. UNIX was dominated by AT&T. You weren't allowed to share software. And so this really changed things. And I encourage you to read it from that point of view. So kind of a document about the pragmatism and value of the work we do.

So as engineering leaders, our responsibility is really to identify and strip away the high cognitive load, low reward, repetitive tasks, right? And so this is not really a soft skills talk. I want to introduce a framework here for how tech leads can better guide their teams when AI is doing heavy lifting.

This year, I had the privilege of going back to my university and meeting up with my math teacher, Michael Starbird. And after 40 years of teaching math deeply to students, both in undergraduate and graduate levels, he got together with one of his favorite colleagues. And they really talked about how what they were teaching students to do was not math. They were teaching them how to think and learn effectively. And as I started talking to him, I realized this is really the premise of what we're all going to be doing with that and should be doing right now.

I mean, once the AI services came out, we were all, I was deeply into my career and this was a total change. I had to learn it from scratch the same way everybody else did. And I think for both new grads and those of us further in our career, this is a really exciting opportunity. We're kind of on equal ground in some ways. But I want to talk about how you can leverage what you already know as well.

In Greek mythology, you're probably familiar with the four elements. They represent both the physical building blocks of the universe and the fundamental aspects of human experience. And so what Dr. Starbird put together was the idea that these four elements can really help you evaluate whether you're learning effectively. And that's a very different say than just critical thinking. Effective thinking means building constantly. And so, you know, Earth here represents understanding deeply. Fire is failing to succeed, getting insights through mistakes. Air is all about asking questions and water is about understanding where you are today, where things came from and where they're going.

This framework also introduces a fifth element, which is you, the quintessential. This is where it comes together. This is about you, you know, recognizing the characters. Did you all recognize the characters from Avatar, The Last Airbender by chance? OK, if you're familiar with it. You know, Avatar, in the end, kind of transforms himself to solve the problem. He's battling this evil guy, the Fire Lord Ozai. And the whole back half of this series, he's lamenting the fact, you know, he's a he's a Zen-style leader. He's lamenting the fact that he's going to have to kill Fire Lord Ozai because in spite of every effort he tries to take, Fire Lord Ozai is just bad.

He just wants to cause damage and destruction. So he's like, I got to kill him. But then Aang has this turtle psychedelic adventure where he learns that the real power comes in giving and taking away the bending power that is sort of the magic that they have. And so in the end, he takes away Fire Lord Ozai's bending power instead of killing him, sort of rendering him impotent. And you know, that was this, he managed to accomplish what he needed without killing anybody. And so in the age of AI, the engineers who thrive aren't going to be the ones who just learn the tools and learn new magic powers like fire bending. It's going to be the ones who fundamentally reimagine their roles, you know, their identities and their relationship to this craft.

So we're going to talk about the elements here in a little bit more detail and how to think about them, because this is the tools you're going to use to evaluate whether you are learning effectively. And this is super important, I think. You know, you're probably familiar with the Karpathy quote that the most important programming language right now is English. And that's because the LLMs were trained on a bevy of English language tools and code and documents. And so as you communicate and learn how to talk to AI, so to speak, what's important is that you know how to communicate effectively with it. You've probably experienced this. If you're clearer, you get better results. Right. And so how you would develop that skill is in fact learning how to communicate purely language based.

And so the liberal arts education that sort of got supplanted during my career with pure computer science is kind of making a comeback. And so depending on what you did in school or not, this is a good opportunity to kind of connect back to the roots of what was education forever. So rhetoric here is like defining the what. This is easily translatable into something like types and schemas. You really have to understand how those work. Right. And then the taxonomy around how you organize knowledge. This is data structures. Truth is really the limits of the system. You know, physics is basically built on a series of constants. Knowing them and applying them is relevant to what is big O and latency in programming. And then ethics, safety, security, trust. These are all fundamentally part of what is built into liberal arts. Fire here is about iteration velocity, having the ability to really move quickly.

You know, startups often say that their superpower is being able to move quickly. And so if you think about fire, can you move quickly enough to get your ideas in front of people fast and make changes quickly? And air here is about questions. So this comes from, you know, drawing questions out of thin air. And the idea is progress here isn't coming from answering questions. It's coming up with questions. And the answers just lead to more questions. So as you learn how these tools work, ask yourself, ask the tools, how do you work? It's amazing what they'll tell you in terms of what you can learn for interacting with them better. You know, ask the questions. Make sure you're building on that knowledge.

You know, Vercel, we talk a lot about this idea of instant deployment where, you know, you make a commit to a Git repository and then there's this build step. Right. And then eventually you can go visit your project on the Web, which is very powerful. But what if there was no build step? This is really a thing that I've come to expect. Right. Oh, compilation. You know, the famous XKCD thing where the swords are fighting and they're like, oh, compiling, you know, you had time to go screw around. But what if instead you had time to, you know, immediately get your results on the Web? We're working on this. It's going to happen. And I'm really excited about it. So challenge everything and everyone. Your agents' technical output is really clapped by the clarity of your logic. So work on refining that logic, not just, you know, the desired outcomes. The right question changes everything here. And if you look at this example, this is a sort of a joke from classic literature.

But man one says, you know, we'll never outrun the bear. And man two says, I don't have to. My only question is, can I outrun you? And so by clarifying what is possible versus impossible, you're able to really make changes here. In engineering, this really means can I, you know, not can I get AI to build this feature? But how might I get AI to help me build this feature? It's certainly possible. The element of water is about continuous adaptation. So guessing what's next and then thinking about where it came from and really evaluating the tools.

And so all these new ideas that we have, they're just tiny modifications of things that have already been done. We're going to talk about an example of this with loop engineering. But really the lesson here is extend yourself, you know, and extend your ideas instead of thinking you're completing them. And then you get to like this idea that under construction is sort of meant, you know, you're trying to get it done. But if you accept that this really is the norm, you know, the good news is you'll always be busy. You're always building. And this is true in your life and this is true in your projects. They're not all just going to come together. You have to keep going and thinking about, you know, as the technology changes, what will you do with it? How will you incorporate it?

What can you do that you couldn't do yesterday? And then the fifth element here is about, you know, engaging with the change. You really want to go beyond the idea of it's just, you know, your role as an engineer is your role a PM. Are you a designer? You know, these used to be pretty concrete and discreet roles. And I think we've seen a lot of this change. We have the idea of the devsumer now, which is the developer consumer for a lot of our tooling. And so we're bringing back a lot of the joy of consumer tools into developer tools. And I think you all appreciate that. I certainly do. And so transforming your identity and how you think about what your job is, your job is solving problems, not engineering.

And in the classical sense, this is foundational building in software. This is about applying to code. But what your job is, is to solve problems and save people time. How can you do that more effectively? You've probably seen the trend where job titles, at least in Silicon Valley, they're sort of changing. Companies are just sort of reverting to the idea of member of the technical staff. And there are people who find this uncomfortable because they've built up to staff software developer and they really want to then become a principal software developer. And I think the move to member of the technical staff is sort of endemic of this doesn't mean anything anymore. It's a flat organization in many ways. Your job is to solve problems, use the tools, all of them at your disposal.

So that means you have to learn them all, which can be a little daunting. And so I don't know if you're familiar, but last week, Peter Steinberger, the creator of OpenClaw, and Boris Cherny, who developed Clawed, Coded and Thropic, started talking about this idea of loop engineering. And so, you know, Peter's saying, like, hey, it's been a month, you know, we've been talking about this. Are you paying attention? So it's kind of nerve wracking, right? There's this new paradigm. What does it mean? And so I thought it'd be fun to apply my professor's framework to this idea because I think it's very much how I think about programming. And I suspect how you do, too. So what is it, right? Agent programming is like there's this version where you start prompting synchronously, you know, waiting and watching the output. It used to be that we would watch the diffs in real time. That was pretty exciting, but I don't think anybody's doing that anymore.

You know, then people started going to tools like Conductor or get work trees right to run multiple sessions concurrently. But then, you know, sort of this galaxy brain idea is you don't exactly need to do it in that way. What you do is you start loops, you start autonomous, self-referencing loops to solve the problem you're trying to solve, which means you really have to define the problem. You have to ground back into what you know. You have to figure out how to take advantage of it. So, you know, if you think about how you would put this into our framework, you start about measuring, right? You have to have the measures in order for an autonomous loop to know if it's making progress. Is it going, you know, what does it mean to complete the agent loop? Well, you have to start by having data.

And I think a lot of us like to start with vibes. It's a really fun, creative journey. But increasingly, in order to take advantage of AI, you really need to start with, well, what problem are you trying to solve? And I'm sure you've all been frustrated with either yourself or your colleagues at times you're like, that's really cool. But, you know, who helped? Who's helped by this? Do you have the background data? And we used to say, oh, that's the PM's job. That is not the PM's job. That is our job. And that's the great responsibility we have as technologists, I think. And so then change. You have to find out how to make the right kinds of changes. Conveniently, I think code is a very changeable format and LLMs are very good at reading it. So this is this is nice. And then verification. I think we've seen a lot of change in the industry around how much easier it is to verify changes. It's getting cheaper.

It's getting more automatable. And, you know, I mean, you think about all the sponsors here today, the idea of AI driven code review, it's so fundamentally good. And this helps you verify that you're not, oops, shipping tokens into production or, oops, you know, forgetting a return statement. Literally the worst incident I ever sat in on was, you know, a line of code, a block of code that started with, you know, if is dry run, do this. And then there's the do this part underneath. And there was the missing return statement. Oops, data deleted. You know, it's the worst kind of incident. And then you repeat this. So this is what an agent loop means. And this translates pretty nicely into the framework we just talked about. Right. Earth is about measurement. Grounding. Fires about making the change. Air is about verification and questions. You know, what are the questions? What is success? And then water is look around, look forward, look backward and repeat if necessary or bail out. And this loop only works if you're willing to be wrong.

So, OK, how many of you love to be wrong? OK, you should all raise your hands right now, you want to be wrong. You're good. This is how you're going to learn. You know, get comfortable with it. It is deeply uncomfortable. And, you know, I think you have to accept it. And that's sort of the growth opportunity here. So I'm going to give you an example of an agent loop and a tool that I've worked on kind of on the side called Dev 3000. And the idea is that I got frustrated by building. I started by building an application that is a service worker. Have you all been in service worker hell before? It's tough and you can end up in very broken states. And so I had given up on it for a while and I thought, oh, my God, this is the turning point for service workers and web apps.

AI is going to be great at this. And then, you know, I'm starting to do it. And Codd's asking me to paste from the network tab the results I'm seeing so that I can, in turn, you know, figure out what's going on. How do you paste from the network tab? You can't exactly. It's not pasteable, but it is serializable. And so the Chrome debugger protocol is the tool that makes, you know, as the underlying piece for Playwright and, you know, all the tools to automate browsers.

And so I thought, OK, if I can build a tool that starts the dev server, starts a CDB connected browser and then interplays all the logs with timestamps in between an agent has access to that. It's going to be much more effective and even better than that through CDP. It can control the browser. So now you've got this system that you can just talk to and drive autonomously. And so I thought, OK, I want to improve, you know, cumulative layout shift for every project on the Internet. Is this possible? I believe it's going to be possible. This is automatable. So this is the example of the prompt that I started working on to do it. And it works. It's thrilling.

You start with, OK, we're going to improve CLS for the target page. We're going to set a baseline. We're going to set a measurement. Right. And we're going to give the agent the authority to work autonomously. It turns out you actually have to tell it you're comfortable. Work autonomously. Keep going. And so measure, change, verify, repeat. Right. That was the slide before this. This is how you build the agent loop. So preserve behavior, avoid rewrites. Now you're measuring, you know, use the agent.

Are you familiar with agent browser? It's sort of an updated form of playwright. But it's a really easy way for agents to use the command line to drive a browser. And it turns out agents are really good at command line interfaces because they're programmatically verifiable. They can run them very quickly. In general, they are fast. And so, you know, we're selling it here. Use the agent browser and the Chrome debugger protocol. Reload the page consistently and collect the CLS data, which comes from CDB, from layout shift entries.

Now run it three times. Get a median baseline. You have to be pretty precise here. You don't want this to run infinitely. And so you have to set guardrails, right? And so now capture the elements, the viewport, the route and screenshots and all the frameworks you use to build web applications today actually export a manifest that agents can use to get this information. They go read the manifest. They can figure out what page they're on. They can map that back to the code and that it has access to using source maps or other such things. So it's kind of feel like it's all coming together.

Inspect the code from around the measured source. Make the smallest change likely to reduce cumulative layout shift. The agent can go and modify small bits of code. Maybe it's making a module import async. Maybe it's finding an ad type thing that it can make async so you can save space on the page but then load the ad after the rest of the page is loaded. Now verify and repeat. Rerun the agent browser and the Chrome debugger protocol measurements, comparing before and after. You're going to do the things that you are used to doing yourself. Like when you start to commit, you have a pre-commit.

And now verify and repeat. You're going to go back, rerun the agent browser and the Chrome debugger protocol measurements, comparing before and after and this is where it gets really cool, right? It can not only can it, what does it mean to compare? You can describe it. You're going to do the things that you are used to doing yourself. Like when you start to commit, you have a pre commit. What does it do? Well, it's going to run a lint. It's going to run the TypeScript compiler. It's going to run the tests, ideally some version of them. And even better now, it's so easy for the agent to run pixel diff and actually verify it didn't ruin the page, you know, and you can set a threshold on the pixel diff, for example.

This is like a thing that I've done manually for 15 years of my life, but I'm now forced to write it down and actually try to describe it like I would describe it to, you know, a colleague or to, you know, a new student who's coming in to work with me. And now I'm actually doing that for the agent. And it's extremely powerful. So here, what's what's really critical about this is that you keep doing it. Are you familiar with Hebb's law? This is that the neurons that fire together, wire together. The more you go through this process and ask yourself, you know, did I really learn the steps before? And I'm at this back to even my time in school. You know, you think you're getting to the end and you graduate. But in fact, what's important is not, you know, if you got an eighty five on the test before and you think you've got a good understanding and you move on, right, that's not good enough.

You actually have to go back and make sure you have complete and thorough understanding and then you can move forward because you're going to build on foundation and you're going to use the earth underneath you to really make sure you understand how these tools work. So if you haven't built an agent harness, like go try. It's it's really a good endeavor. But what it'll teach you is how to use agent harnesses more effectively because they're not magic. They're not rocket science. They're not limited to the world of just high tech computer engineers. It's all built on very systemic logic, things you've done, things you've certainly done yourself. And it's just a tiny verification. Someone chose to systematize it and you can do this to face this friction. It's worth it in your career and it's worth it for your life. As engineering leaders, you want to optimize for the depth of thinking and execution on your teams. You're not data entry operators. You're limitless. You're boundless and you contain multitudes, right? Thank you very much.

Appreciate you coming. And I hope you've enjoyed this. You can find Dr. Starboard's book, Effective Thinking in Five Elements and carry on.

So got a few questions here that I'm going to I'm going to put in front of you. And if you haven't yet and you have questions, do put them in Slido.

First one, we've got questions as being always busy doesn't sound like a goal that I want my team to be because it sounds like the recipe for burnout six or 12 months down the line. How do you think about busyness versus sustainability? Yeah, I mean, this is a thing I think we all struggle with in our personal lives and our work lives.

And this idea that there's a work-life balance, you have distinct and concrete versions of this in your life. I think this goes back to, you know, there are parts of the industry where this works like I used to be a cook. And the great joy of being a cook was, you know, you're on at work and when you're done, you're off. But I think in creative technology jobs, this is not actually appropriate. I view the learning and the opportunity when I say being busy. It means there's things to do. You have questions. You don't stop questioning. If you stop questioning, you know, that doesn't mean don't have fun. But even then, you know, you're trying to meet people. You're trying to engage your creative brain. And so I actually don't think there's a good distinction between work and life in this space.

I go home and sometimes I have ideas. I want to see them realized. I prompt for things like, where am I going to go? Where am I going to take my daughter in Utrecht? It'll be interesting. I view that as a way of what I mean by being busy. Being busy is letting your brain really like do stuff. And so, you know, you want this. The probably the better enemy here is boredom or, you know, staring at your screen, waiting for things to compile. You know, that doesn't mean free thought isn't a good idea. Free thought is being busy. You're really getting into, you know, what you want to accomplish in your life. And the people you want to have fun with. Yeah. Lowering the barrier to curiosity, right? Oh, absolutely. Yeah, absolutely.

Next question we have, this is based on your example. It seems very useful for optimization problems. What about zero to one projects? How would you apply this when what you aim for is creative by nature? Yeah, not really measurable.

Yeah, not really measurable. Yeah. I mean, I think when there are a lot of examples of, you know, it's a good place to start. Sometimes, you know, you just want to get an idea on a screen because we're visual beings. Right. But, you know, most of the ideas I've had when I look back on times that I spent too much time building without a purpose, I don't think it was a good use of my time.

I had ideas. I spent too long building an application, for example, to share my battery level with my family. I thought this was going to be a good idea. And it turns out when I asked the question, you know, sort of systemically, this is just not a thing that people really need. It's fun to build, but I didn't learn nearly as much trying to build that because I didn't get feedback.

And so when you're just building something solely for yourself, it can be valuable as long as it's providing you value consistently. And you'll know, are you using it day over day? If you are great. But what I found is I wasn't those things when I had the opportunity to share with n plus one, you know, not just one person that I would work with, you know, a group of people, the energy and the fire that you get from feedback really is as a force that I think builds your creativity and causes you to come up with better creative ideas, because I think that's the goal.

You know, starting with what if I assume the thing you want to build is going to be successfully built? You can do it. Right. If you just I think we started building sometimes like I even build that. So if you start with no, of course you can. You can build anything. Software. It's kind of amazing. And there's hardware. And it's also amazing. When you start with I can build it now. Assume it's built. What purpose is it at? Whose time is it going to save? Whose joy will it provide? If you don't ask those questions, I don't think you will be learning or actually having a long term relationship with that work. Yeah, no, I've definitely found these tools are incredible for pairing with a designer or a subject matter expert or something like that because you get something to show them so quickly. Oh, it's amazing. And seeing is believing. Yes. You know, we are very, you know, visual or haptic oriented beings. And you want you need to get there.

All right. Next question is one I think almost everybody is having. And we are finding that our bottleneck is code reviews. Oh, yeah. How do you find the right ratio for reviewing and handholding versus letting the agent do its thing? Yeah. I mean, this is this is a slightly unsolved problem. I think that what we're doing with tools like, you know, one of my former colleagues, there's codiff, I don't know if you're familiar with this, but, you know, or sort of using agents to review other agents work. I actually think we're going to get really good at this and being able to, you know, take a piece of code, look at it and say, OK, how what what critical path is this on? OK, what's the impact of a change in this? You know, you T-shirt size it. And we're going to auto approve small changes, probably medium changes in the near term. And we will probably simply flag really critical changes and have smaller changes in that way. I think we will review less code. I certainly do.

So I think we're but we're still building toward that as we build. I think these agent loops, we're going to be able, again, if you can really describe the impacts, well, set up verification systems and use infrastructure that's self-healing. I think the risk here goes down considerably. You know, if it's not a security, you know, if you're able to evaluate, this is not a security risk, for example. And, you know, you can roll back instantly. Then the risk of finding out isn't so scary. Yeah. Yeah. We're finding something similar where we're reviewing now more at the architectural level. Does the shape of this make sense? We're letting the agents deal with the code correctness of it. Yeah, I think that's very likely what will happen.

All right, I think we have time for maybe one or two more of these. So next one up we have. How do you manage costs when designing and running agent loops? Yeah, I mean, I think evaluating the kinds of model you want to use for the kind of task you're doing is pretty worthwhile. But you can only find this out by trying different models. I think auto is a good way to run in a lot of these agent loops, because you're going to have a series of prompts and evaluating them for the proper model cost speed benefit is pretty valuable. I again, interesting problem that a lot of people are trying to solve. You look at a prompt and decide which model is effective, cost effective for it. So again, this is the thing we're all learning today.

I mean, we're asking this question because we're not exactly sure. And the default is, you know, try it with a frontier model. See if it's any better. Try it with an open source model. You know, there's no reason not to do both and see what works for you. But efficiency here ultimately will be figuring out, you know, again, the more well defined your measurements are, the better your foundation, the better likelihood is you can use a more cost efficient model to solve the problem. If you understand if you're trying to optimize a bloom filter, first of all, you have to know what a bloom filter is, which means you have to have like understood data structures and algorithms from the baseline and really deeply knowing that you can write the prompt that a very less sophisticated model can execute on. All right, we're going to do one last question and then wrap, so we have time for folks who need to transition. So last question we have here. So agents can get stuck doing hill climbing style local optimizations. How do you focus them on objectives that require non local changes to achieve a global optimum? Yeah, I mean, I think this is the connection point for infrastructure deployment oriented companies. Vercell is right in the middle of this. I believe we can tell, you know, when once you push a change out, you know, we do a gradual rollout. This is thing you know, this is what Google and Facebook do with basically every change. They start with like a very low percentage rollout and they know from the PRs essentially what the goal is from the measurement side before they start to roll out more broadly and then they can automatically roll back. I think this technology is how we solve that. All right, let's get one last thank you together here for Lindsay. Oh yeah. Thank you all so much for coming.