Giving AI Agents Hands: Mobile Feedback Loops with Agent Device

Ten years ago, I was speaking about CLIs, and we're back full circle talking about CLIs again. Anyway, you know, I've been actually speaking three times at this conference, and each time I'm on this Basecamp track and usually I speak about React Native and I would say the audience is maybe like half of the room, and apparently adding AI to the talk title increased the audience ratio, so I'm happy for everybody to be here. This might be actually my most popular talk so far.

Alright. So thanks for the intro. I'm a founder and CTO at Coldstack. We do cross-platform apps with React and React Native for about ten years now. I've been doing a little bit of the core React Native work, working on CLI and a bunch of open-source packages. But today I want to talk to you a little bit about what it actually takes for an AI agent to do real mobile work, and spoiler, it's not about prompting. It's about feedback loop, and I'll show you how we've done this in a second. So let's go.

Before we start, quick show of hands. Who here is using any kind of AI daily, cloud code, codex? Please raise your hand. All right. That's a lot. I guess everybody who didn't raise your hand, you're just afraid or maybe just getting warmed up. But that's basically everyone or most of the people. When you ask your agent to change a mobile screen, how many of you actually pick up the phone and manually check it on the simulator? Okay. Quite a few. I'm assuming the others are already using agent device, which is great. So in this talk, I'll show you a little bit of what happens under the hood. Like I said, we're excited about agents. We use them every day. But unfortunately, almost every one of us is still somehow involved in the verification step. And that gap is what I'll show you how you can close in the next 20 minutes. So the thesis is that every developer workflow is a feedback loop. And our work has always depended on one thing, fast, trustworthy feedback. You change something, you run it, you observe, you look at what happened and you adjust it. Everything we call developer experience, all of it is really about making that loop shorter. Before we talk about agents, let's maybe look at how we build feedback for ourselves.

And it's not back in my day, we waited, you know, 40 minutes for a build type of slide. But you know, not so while ago, we used to use those tools manually. Maybe we still do, but it's clearly changing.

Chrome DevTools, for example, gives you access to so many things on the web, DOM Tree, network and console. And the browsers ship those DevTools and web development got dramatically faster because of that. Now, mobile did the same thing. You have Xcode instruments, Android Studios profiler, or maybe React Native DevTools with plugin frameworks such as Rosanite that lets you add community extensions, such as in this case Expo Atlas for analyzing your bundle.

Different vendors, different devices, different eras of software development, but there's always this one job when the app is running, you want to see what's happening under the hood and how your code changes impact and behave on real devices or simulators. And if you shipped React Native in the past, you most likely worked with all of these tools because our backs really love to sit somewhere at the intersection of JavaScript and React Native, and that's only, and those are the backs that you can only see on real device, not simulator.

So then the last couple of years happened, and it is quite evident today that agents write most of our code and they do it shockingly well. They refactor add features and all of that. And now our focus goes to orchestration, verification, reviewing all those steps about getting agents more productive to work. But the challenge is in such setup, we are a human in the loop or a component inside that agent loop.

And it's not just any kind of component, it's actually the slowest and most expensive one. And it's only getting worse in the best possible way. One engineer used to produce one stream of changes. Now every engineer runs two, three agents in parallel and code is coming out faster than anyone can really hand check it on device. And the verified half is still one human, one pair of eyes, and you cannot just fix it by reading more diffs. At some point, your brain capacity will just run out.

And we have to fix it the same way we fix it for ourselves. You have to give agents a way to verify what happens on device. Now web agents have this, the browser was sitting right there, and the popular phrase just give agents a browser quite worked. There was Playwright, MCP, and all these tools. So it was quite easy to get started. Mobile, however, is a bit different story.

Because first of all, mobile agents need to work across simulators and devices. So by definition, you have a wide range of things you need to keep in mind. It needs to survive different environmental constraints, permission dialogues, alerts, keyboards popping up at the wrong moment. It performs gestures, not clicks. The logs are often split across JS and native, and you have many different surfaces. In everything, iOS only runs on macOS, which your CI or your build hates.

And this is why just give agent a browser doesn't really translate to that kind of workflow on mobile. So we decided to do something about it, and we created Agent Device. Now Agent Device is that layer that lets your agent operate on a mobile app, perform many different automations, open it, see the logs and interact with the app. And you can do it on simulators and on devices. Now before you picture some heavy framework, Agent Device is just a CLI. You just npm install it, and you're done. It's on your machine. And that matters quite a lot, because agents love CLIs. And they already live in the terminal.

So we don't have to integrate any SDK or wire anything up. Your agent just have one more command to work with, and they already work with a bunch of commands. So here's everything Agent Device does in five simple steps. It can open the app for you, iOS, Android, TV, or simulator, real device. It can snapshot what's on screen, in this case accessibility, which I'll explain in a second. You can act on it, perform any kind of mobile interactions, taps, scroll, swipe, real gestures. You can collect evidence when it matters, screenshots, device logs, video recordings. And you can replay the session later as a script. So we can think about having end-to-end testing framework for free at the end of the day.

Now, one design decision I want to spend a moment on, because it's the one where we often have questions, why not use screenshots? So when an agent looks at your mobile app with Agent Device, the first thing it sees is not an image. It's an accessibility tree, which looks like this. On the left side, you have the snapshot. And on the right side, you have the fresh expo app. So first, the accessibility tree tells your agent the same thing that your user can actually operate with. It's not a view hierarchy. It's more of an interactive semantic reference to elements. And when the agent decides to act on it, it will press, for example, E12, maybe, and that will reference the E12 element in the tree. And there is a bonus here. This only works when your app is accessible ready. So if it works with a screen reader, it will work with Agent Device. If it doesn't, you will have to fix your accessibility tree.

So it's kind of a nice tradeoff, I guess, for more accessible apps. So why does this matter so much? First of all, token economics. There is, you know, text tree costs much less than images. Vision is powerful, but it's obviously a bit more expensive to run. But even if you can get the token cost lower, it is going to be much slower. Model can act on those REFs quite precisely. When you use screenshots, coordinates, maybe some elements on the screen overlap. There is always some room for interpretation. This is more strict. And here, it is cheaper to run overall.

So when you think about your loop, you should think about running this maybe, you know, multiplied by 50, because your goal is to run many of those feedback loops and maybe keep parallelizing your work. So while keeping the cost lower in one, two runs may seem irrelevant, think about hundreds of these, and then you'll see it. Cool. Enough talking. Let's see the demo. So like I said, Agent Device is not really a framework, and you just work with it by adding it to your whatever is your coding agent.

So in this case, I have this very, very realistic prompt, which is about modifying the Expo template with speakers from React Summit. And the only difference here is that at the very end, I'm instructing the agent to use Agent Device to verify feedback. Obviously, this is optional. You can put it in your Agent's MD or whatever. But in this case, I don't have it. So that's why I'm adding it. And then the agent is going to start working on it. And I'm using Codex and GPT 5.5 Medium. It will pull speakers from the Summit website, build a local database for that, and add that app to the application. And it's done. And you can see on the left side, you can see it has a little checklist. And I had to stop the video at some point. So the run local verification is not crossed. But it is crossed.

So it actually assumes that everything is working. And without Agent Device, that's probably where your workflow would end up. And you would be the one picking the simulator, interacting, and testing your app. But this time, we had verified the work. So let's see what happens after. It will check the snapshot of current view, navigate to Speaker Tap, try tapping a speaker. Doesn't work. It will attempt changing the way navigation is done using different API. But once that doesn't work either, it gets to the conclusion that there is a wrong navigation hierarchy.

And you know what's funny is I had to change my talk last minute. Because initially, when I was designing this talk, I assumed, okay, the first prompt will succeed. Then I will have to make this fake bug. And then I will guide you through, okay, now let's do something that doesn't work. And this is actually just a random bug that happened when I was doing something simple. Just add one screen and push to it. And you know what happened is that the regular Expo application was just tabs. So when I started pushing from a tab, there was no root stack navigator that could actually take the route that we were pushing to.

So, yes, everything was working from the dev perspective, but the parent navigator didn't work. I'm honestly quite surprised 5.5 didn't handle it. But thankfully, that made my demo even more realistic. So the agent said it works and its own verification proved it wrong. So the speaker screen never pushed. Like I said, the view hierarchy was broken. So once the issue was identified, and again, this is just a continuation of the same video. The fix was quite easy. Just replace the root or wrap the root with stack navigator. And here you go. It now works automatically. So this whole experience took around 7 minutes in total because I was doing some other stuff in between. But the conclusion here is that without this step, it wouldn't complete it.

And now with this step, it actually self-corrected itself. So thanks to agent device, the whole loop verified the UX on an actual device. It taps through the whole flow, tabs, grid, into the talk details. And it applied necessary fixes in the same session followed by another verification before reporting done.

However, not every bug can be discovered this way. Some bugs live underneath state. That's wrong, even though the screen looks normal. And for those, agent device gives you access to some things under the hood. Like in this case, you have access to React DevTools over the CLI. You can find components, read props, state, using the same notion of accessibility tree as for reading the snapshot.

And the compact mental model here is that the accessibility tree answers the outside question, what can the user do? That's what we see. And then the React tree answers the other question, what is rendering, what is happening under the hood? And that's the view we need when the outside view is not enough. And an agent is debugging a React Native app both ways, kind of like we used to do it when we were debugging both the UI ourselves with the eyes and having DevTools opened on the right side.

There is way more features in this toolbox. You can do logging, so you can read native logs, markers, reproduce and read exactly what happens on the native side. You can pull the HTTP traffic and see which requests are dying, where is the information going through the network? You can read performance startup, memory CPU profiling. And you can also see the render profiling, which components are slow, re-rendering, and so on and so forth.

The point is, whatever evidence your agent needs, it is there. So with all those features, I have one more demo to show you. And in this case, we're going to do this realistic debugging on Expensify application. We're going to perform a user flow in which we will be going to file an expense. In this case, I'm going to try following whatever happens here. But basically we're making a snapshot so we can get to the right screen.

We're opening DevTools. We will be doing logging and markers in order to read what is going on, what is the traffic. When we go to filing an expense, which will involve making a camera picture, we're going to put a marker on the native side to read the logging, what happens with the native device before cameras come, for example here. And after we collect all that evidence with logs, timing, profiling, and network, we will be able to continue debugging, analyzing, and agent will be coming up with conclusions what is going on.

So typically it's the same debugging flow that we would do, but thanks to agent device, the agent can now do it autonomously. And as you can see, it is sort of finding evidence that some network is taking way longer, the API needs to be improved, and there are some issues with the text inputs. So there is a bunch of those things. The video is slightly increased in performance, in speed, but at the end of the day, you see key perf issues that were discovered.

So it's quite interesting how you can do it, and here you have the overall token use during that session. So not too much, to be honest, and the time saved was quite nice, at least even for seeing more information out of your application, what is going on. So back to this chart. As you can see, code output is going vertical, and the verification, the only verification that scales with it, is the verification that runs without you being in the loop.

And that's an AQA agent that can live on your CI, and this is what it looks like. AQA agent can pick up the diff, open the app, do any kind of debugging, record the flow, do whatever you want it to do, and the PR gets this nice comment. And now if you're thinking that, you know, maybe this is some kind of hard to build project, or you need to start with an off-the-shelf product, in most cases that might be a good direction, however, it's actually quite easy to build for your use case to get started.

As you can see, nothing really, I would say, revolutionary insight. This is a versatile tool loop agent with access to agent device that you can run on your CI, and we call these templates in agent device documentation. We have a ready-made recipe for EAS, which runs on a Mac, enabling quick setup with just a few lines of code. For the CI environment, the choice between Mac and lightweight Linux sandboxes for agent execution can impact speed and cost. A solution provided is the agent device proxy, facilitating communication between the sandbox and a Mac for quicker feedback loop.

In practice, the agent device proxy enables seamless debugging even on a cloud-hosted Linux environment. By setting up a simple proxy on the target machine, commands flow between the sandbox and the Mac, allowing for efficient debugging and data exchange. The agent device supports various environments, including simulators, devices, and browsers, offering flexibility for development, debugging, and QA tasks. The lightweight CLI nature of the agent device allows customization and integration into CI workflows, making it adaptable to different requirements.

The scalability of the agent device solution is emphasized, particularly for cloud-based workflows that require increased capacity. All essential resources and information for utilizing the agent device are available at agent-device.dev, including documentation, templates, and the EAS recipe. Users are encouraged to explore the capabilities of the agent device to streamline their feedback loop and enhance development efficiency. The speaker expresses willingness to engage in further discussions on agents, React Native, or related topics, inviting questions and interactions with the audience.

Agents need a feedback loop, and without one, you are the person giving them feedback. An agent device is your mobile answer to it, allowing your agents to work with simulators and devices. For browsers, the agent browser, a project by friends at Vercell, follows similar patterns and is versatile for development, debugging, and QA tasks. Being a CLI, it's lightweight, customizable for CI setups, and adaptable to various workflows.

You can easily scale the agent device solution in the cloud, ensuring efficiency even when local workflows face limitations. All resources, including GitHub, documentation, templates, and EAS, are accessible at agent-device.dev for a seamless start. The speaker emphasizes the shortened feedback loop and invites questions and discussions on agents, React Native, or related topics, enhancing audience engagement and knowledge sharing.

The agent device is platform-agnostic, compatible with Flutter, Expo, and React Native, focusing on cross-platform and native side functionalities. It leverages system accessibility features for seamless integration across different frameworks. The tool includes a feature like Screenshots Defink for visual regression detection, enabling feedback on UI changes between interactions without altering the accessibility tree.

Agent-device.dev. We have their GitHub, documentation, templates, the EAS, RECP, everything you need to get started. I hope you enjoyed it and your feedback loop will now be much shorter and faster. Happy to answer a few questions and also later, happy to talk agents, React Native or anything in between. Thank you very much.

The first question that we have for you is, can it work with Flutter or only Expo or React Native? Good question. I haven't tried it with Flutter, but I would say if Flutter has accessibility features, it should. Agent-device is platform-agnostic, so you can actually use it with iOS native apps as well. We have quite a few examples when we use agent-device to do some automations on our iOSes, like for example, getting data from Apple Health. It's definitely not React Native and Expo only. It's supposed to be cross-platform, cross-framework and focuses on native side.

This next question is from Jan. How does agent-device detect or handle visual regression when UI unexpectedly changes between interactions without changing the accessibility tree? I wish I could change quickly my laptop, but the team is just unplugging it anyway. We do have this feature called Screenshots Defink, which we actually worked on as a part of building migration agents. One of the use cases that I haven't mentioned in the talk is using agent-device for agenting migrations. You start with a screenshot of a native app and you want to do an exact clone for the React Native app. We have Screenshot Defink, so we will take the screenshot, diff it, show you the differences between fonts, sizes, padding margins in a nice way, so your agent can have a little bit of feedback on what's changing and why.

One of the use cases that I haven't mentioned in the talk is using agent-device for agenting migrations. You start with a screenshot of a native app and you want to do an exact clone for the React Native app. We have Screenshot Defink, so we will take the screenshot, diff it, show you the differences between fonts, sizes, padding margins in a nice way, so your agent can have a little bit of feedback on what's changing and why.

Awesome. This next question, can agent-device replace traditional end-to-end testing tools, like Detox or Maestro? Yes, it can. It has a feature we call Replace. After each session completes, like all the videos I showed you in my talk, you can get the script. It's actually saving all its steps, and then next time, you don't have to do agent-device, I don't know, snapshot or do it with the agent. You can just do agent-device replay, put in your script, and boom, you have exact replay. The funny thing is, the good thing is that those replays are leaving some room for difference. It's not like tap this button and then go there. It gives some room for flexibility, that's why it often works across small changes. Yeah, that's amazing to hear, and basically at that point, you're not using tokens anymore, you basically have a spec. Yes, exactly. That's great.

Alright, this next question, for cross-app web project from a single codebase, can agent-device also validate the web part? Right now, it doesn't. Right now, you have to pull agent browser separately, however, we are receiving more and more requests to include agent browser or some kind of browser support anyway. So I think it will be done, and Michał is probably watching it, the maintainer and author of agent-device, so Michał, now the question is up, so we have to ship it. Amazing. Alright, next question. Are there any other cases besides using an agent-device for testing, such as creating bots? Yeah, I mean, one of our favorite use cases is for using agent-device to do some automations on device where you don't have API. There are a few interesting vision models and models made specifically for controlling device, I think a few of them are from ByteDance, so I think some people are using it this way, and you can definitely do it. Cool. Don't do anything malicious, though. Good reasons. Next question, does agent-device also support using real devices? Yes, I love all these questions because I'm all like, yes, yes, yes, affirmative. It does support real devices. Part of the reason is that we're using official Apple APIs for testing. They're not as quick as we would love them to be. However, they work across all the devices, so you can do it on a simulator, you can do it on a real device.

We have people running this on device farm. For example, Mark Grossavi, he has a vision camera library, and he's been working with agent-device to run all the tests in device farm. I'm not sure if he actually made it announced already. If I leaked it, I'm sorry, but yes. Amazing.

Okay, next question. If this runs on CI, what optimizations can be made to prevent token spend exponential increase? That's a good question. Yeah. Some kind of optimizations, yes. I mean, we haven't gone into this. It's kind of outside of scope of the initial library. Although I guess at this point, same optimizations would apply as you would normally do. So perhaps a bit of caching, using replay scripts, optimizing them, and doing some smart device routing to prevent maybe going too deep into testing. That makes sense.

In your experience, is there any specific model which works really well with React Native? For coding? Yeah. Spoiler alert, we have our own. It's called Apex. If you type it, Apex call stack into Google, I love how this comes up, actually. It's not like I'm shilling my own tool. So we have our own React Native fine-tuned model, which is based on JMA4. We are trying to hit an angle of getting a nice cost optimization versus feature set. But if you want to check yourself which model is the best, you can type React Native evals in Google. In our evals data set, we're building it, trying to make it more, better, and better. But it gives you some kind of insight into which models are best. Right now, Composer 2 is killing it. But after Composer 2, it's GPT 5.4 and Opus 4.7. Surprisingly, 4.8 and 5.5 didn't improve much, apart from being five times more expensive. So just check it yourself and see the results. Fantastic.

Time for one more question. Let's see. If we use it as a replacement for Detox or Maestro, would you say that this could be more flaky than those? I think part of the flakiness goes into just the device, Apple, all the connections, and the harness itself. So I think there is some component of flakiness that both of these frameworks will have. We have some nice videos where we compare performance with Maestro, and we're hitting performance trying to be faster. So we want to improve it. But obviously there is some flakiness anyway.

Yeah. And we still have time for the last question. Yeah, let's do it. So, does it support TV platforms? Yes. Great. Alright, Mike, thank you so much. Thank you.