Automating Mobile QA with Cloud Agents

Hello, London. It looks like we use agents for virtually everything now. As a mobile engineer for almost 90 years now, I like to throw agents at my mobile challenges and chores and automate myself out of my job soon probably. Thankfully, I'm also building tools for mobile development.

So how many of you ever built a mobile app? Regardless of whether you build mobile or web apps, you probably experienced this. The diff looks fine, CI checks are green, you approve a pull request, and then the app does something completely unexpected on a real device. The code changed, but the user does not experience a diff. They experience the product.

Over the last few years, and especially in recent months, AI has changed the way we write code and ship products. In fact, it's rarely us writing the code anymore, sending one, two, maybe three pull requests a day to the codebase we contribute to. We got slightly faster when tab models popped out. But with AI agents, such as Cloud Code or Codex, getting hands in late 2025, it just blew up. It's not only about LLMs now, but the harnesses, skills, and loops. And these agents are very good at producing code quickly. They can draft implementations, refactor files, open PRs, and respond to review comments.

This is a real shift. The cost of getting a plausible pull request is dropping fast. But trust did not get cheaper at the same rate. Someone still has to decide whether to ship this change. And that decision is harder when the code was produced faster by more automated systems without our direct supervision. Traditional code review assumes a certain pace. A human writes the code, another human reads the div, CI runs tests, and the team builds confidence from that context. The model starts to strain when agents increase both the volume and shape of changes. Early data from GitHub's 2026 availability update shows merged pull requests peaking near 90 million per month, over 100% growth over one year, previously growing with lower double digits annually. It's not by coincidence, platforms are already seeing the pressure from agent-driven workflows which strain their infrastructure, which we experience almost daily recently on GitHub. Reactions are only as good as catching at most 60% of potential bugs, provided the divs are smaller than 200 lines and they're experts at the codebase. More than that and it becomes too much to cope. And I think we all experience that. Not to mention it takes engineers a considerate time and focus to perform the review. But hey, modern problems require modern solutions.

We can just throw more code agents, code review agents, at our codebase and that will certainly do it. Well, not really. Early research shows agentic code review still finds less defects compared to humans that are experts on the topic and in my experience it's getting better and it's way more effective when done by a different model. What's important is that it actually scales with the traffic but suffers from some fundamental challenges. Most agents are either poorly configured or optimizing cost and these will only catch nits, some surface level bugs, type issues, or lack of tests. Agentic code review is missing a bigger picture of your product.

It's often so extensive it's getting out of hand and requires another AI to process it. It's still not better than humans in finding bugs, at least yet, and it can't verify the app is working end-to-end. And even a perfect code review can catch only a certain class of issues while being totally blind for others. In mobile and web apps a bug often hides between files or in subtle ways libraries and native APIs we use work. For example, navigation can loop unexpectedly, a permission prompt can block the first run, a native library we use will unexpectedly crash under load, a keyboard covers the primary action button, a deep link lands in a wrong place, a release build crashes where a debug worked. The code can be internally coherent and still produce a broken flow.

Well, we can certainly solve code review with more end-to-end tests. They're testing user flows at runtime. The problem is, they're quite slow to run, flaky, and often don't test the behavior we've just altered in our tests and in our div. How would they know? The review surface has to expand. We should still review code, we should still run tests, but for mobile apps, especially in agent-ridden world, we also need to review the running app before merge, quickly. We need the app behavior review. We need a new kind of agent that verifies app behavior on every pull request, ideally.

The important part is that this agent is not told to explore anything. It gets a bounded product mission, verify onboarding and logging, check the checkout screen, validate profile editing, inspect the flow affected by this PR, and ideally, it needs to infer this from the available metadata.

Let's explore the stack on which we could build a mobile QA agent. The stack that you already have or can easily integrate. What you need is a CI-CD server as your environment, a large language model provider, a harness for AI agent, a tool for controlling iOS and Android simulators efficiently, a blob storage for screenshots and recordings, and a commenting bot that puts it all together.

AISDK gives us the structure for model calls, tool use, state, and the loop between observation and action. The agent does not magically know the app, it observes, decides, acts, and reports. We can code it however we like, thanks to this library. This is our harness.

AI Gateway gives us one interface for any AI model, so we can observe usage, apply controls, and swap models without rewriting the agent. We can experiment with different models for different kinds of agents as well very easily in this flow. Agent device is for agent-first mobile device control. It lets the agent open the app, read the screen through screenshots and accessibility information that is tailored and capped to a smaller size. It allows tapping, typing, inspecting state, collect logs, and capture evidence directly from iOS and Android devices.

On this demo, you can see it's controlling app on my iPhone, so it works with simulators and real devices as well. In this case, we will need simulators. A blob storage. We want to see screenshots and recordings in our pull request comments, so we have to store it somewhere. And blob from Vercel is super easy to set up, and it just works. But it could be S3 or R2 as well, whatever you have.

This is how easy it is to build an agent with AISDK when we put this all together. It has tool calls to agent device, deterministic device setup, so agents don't have to guess every time, and a system prompt that helps QA agent know your product and infer acceptance criteria and related flows from the pull request description, affected code, or linked issues. And frankly, you don't even need to code it yourself, because these scales can handle that for you. So, you set it up on GitHub Actions, EAS workflows, or other CI provider that gives us Mac OS and Linux boxes with virtualization, and fingerprinting setup for fast build reuse. There we can schedule running our QA agent on every PR, and hook a commenting bot easily.

And remember, we want the whole setup to be as deterministic as possible. That reduces flakiness and AI inference costs as well. So, let's see how this all works together on a sample pull request.

So, this is how it works in practice. I have a sample pull request here that is adding a call stack bottom tab to my application. I can look at the code, see a top screen with title call stack, some services, theme text, theme view, and interesting icon choices.

Normally, I would need to download this app, run it on my device. But with the QA agent we've built, I get quick information that this div is passing Android and iOS tests, with screenshots to support it. The Android screen shows a new call stack tab, while the iOS version displays the call stack with some offerings. The agent completed QA for the PR, verified app foreground correctly, and rendered the expected sections, except for a missing screenshot that could be investigated and improved.

So, I wonder how this looks like in practice. So, normally, I would need to download this app, run it on my device. But with the QA agent that we've just built, I get a quick information that this div is passing Android and iOS tests, and we have screenshots to back it. So, here we can see Android screen that has a new call stack tab. It's not clicked, so the agent didn't figure out to leave a screenshot for this one. But for iOS, it actually pressed the new tab, and we can see that there is a call stack. There is some offering. Yeah, so this looks more or less great.

Let's see what the Android agent actually did. It completed the QA for the PR, verified the app foreground correctly, the new bottom tab, and the call stack screen renders the expected summary sections. The agent actually was able to visit that screen, but it just didn't capture the relevant screenshot. This is something that we can probably investigate and improve in our agent. If you want to see how this looks in practice, in the scripts, agent QA index DS, we have our full agent. As you can see, it's a tool loop agent from AISDK. The whole file is around 900 lines. Most of the instructions are here, where the agent gets access to some tool calls to get data from the PR context, handle agent device, and write a report.

If you would like to see how this works with a different framework, Vercella released a new framework called EVE so that most of our agent can fit in one instruction's MD file. We're defining our model here. And that's most of our agent. We're left with one tool call to agent device that is very similar to our AISDK one. This is it for the demo. I hope this is pretty helpful. When building your own mobile agent, you must remember five key rules that will make human reviewers trust it and love working with it. Make sure it's set up deterministically to avoid flakiness. Give it your full product context, screens, capabilities, platform best practices, and how-tos for your app. Use cheap AI models so you pay cents instead of dollars per test. Show relevant evidence, screenshots, visited screens, notes related to your diff. Make sure the results are repeatable by enhancing the agent with planning, memory, and sample test cases. The merge button is becoming a trust decision. In modern codebases filled with agent-written code, reviewers need more than a plausible diff and a green check.

They need behavior evidence from the product itself. We don't need to accept the status quo that we have today. With the right tools and infrastructure, we can change, adapt, and reap the full potential of coding AI agents. Let us iterate on our products faster than ever with more confidence and trust based on evidence.

Thank you very much. Make sure to fork the QA agent on GitHub and give AgentDevice a try. Give it a star if you like it and follow me on Axe under this handle. Cheers. Thank you so much, Michal, for your session and for everyone. On the backstage, we had a quick chat with Michal about how does it feel slightly different all this AI native development for web development and for mobile native development. I think we both agreed that it's a bit more challenging to involve all truly AI native assistance and workflows and everything into true mobile native development because it's a bit closer to web.

But I see that we have a question from Alejandro. I think one of our most active attendees today. Be like Alejandro, everyone, folks. I see that he tried to use Playwright MCP, but there were some difficulties for organizing truly complete end-to-end testing. The question is how is the whole application flow stored to be used by the agent? I think you can elaborate on that, Michal. This is completely up to you. What I'm showing, for example, is building a custom agent that is very simple. It doesn't have persistent memory, for example, that figures out things on the go. And I think it doesn't yet have a planned mode, for example, which helps a lot in terms of creating reusable flows. This is only a beginning. The first agent that you can go, you can try, you can see how easy it is to start.

This is only a beginning. The first agent that you can go, you can try, you can see how easy it is to start. And then it starts to increase in complexity, usually because you need to set up some infrastructure. For the projects that we're doing daily, we are calling this like agentic QA infrastructure. And as part of that, what do we mean by this infrastructure? It's all the skills, your app knowhow. We are trying to get from the existing team, working on the app, what this app is about. What does it feel like? What are the goals that it helps to achieve users? It's all important because it gives the agent extra context. So it can figure out some flows on its own when they are not known.

Another level of that infrastructure is all the skills necessary, skills for best practices in some platforms like iOS and Android. Or when this is a React Native app, some best practices around handling React Native apps, for example, with all the dev servers, etc. And even more important is to have two extra things, which is your screen map. So in mobile apps and also in web apps, you have navigation. So it's important for agents to understand how to navigate your app. This is a process that we often do manually or semi-automatically to start with. So important bit is to give agents as much context about your product and the user flows as possible. And also if you have end-to-end tests, for example, you can convert them to plans, for example, and give agents this list of different scenarios that it can pick when it is verifying your change.

And regarding Playwright MCP, I haven't used that, so it's hard for me to say whether this was the problem, likely not. For web use, I either use agent device web support or agent browser, which is actually running under the hood. I'm not a mobile developer myself, but I can only add that I hear more and more often that if there is a selection, there is MCP and CLI available for the same tool with the same capabilities. If you work with AI agents, go for CLI because it might be a bit more, I don't know how to say this, predictable and maybe you can even save some tokens on that. So maybe, I mean, start from just trying CLI. I know that there's amazing CLI for Playwright. That's my personal recommendation.

And we have one more question about comparison. So how does the agent's autonomous exploration compare to a curated regression suite in terms of the types of bugs it finds? Does it really catch things that a human-written test would miss? Anything from your experience, Michal?

Yeah, so I think this is actually a question about dogfooding. When we talk about agents autonomously exploring our codebase, we want to give them some instructions and some freedom, but not too much. So, for example, in an agent device, we have a dogfooding skill, which you can essentially ask your agent to run dogfooding session with agent device and it will start to explore your application some random ways.

You can tell it to act like a certain kind of user, for example, like a Gen Z or millennial or whatever. And in my experience, these kinds of explorations are finding things that humans are not catching, like inconsistent text, especially in internationalized apps where some strings are not correctly translated or something that I found in one application was like a local promotion in some local store for some commerce shop.

So when accessing this one store, I was actually navigated to something different. An agent told me that this link was incorrect. It should point me to a store location, but I ended up in some totally different place. And it was like a time-specific promotion in that store, so it would only show for a week or something. And the way I like to run these is to write a good prompt with some extensive instructions on what to look for or what to break, basically, or try to break and run it every week or so as a bug hunting session.

And you can reward your agent with certain things for certain behavior or bug spawned, for example, so it can be more persisting. Or you can give it a goal if you're using a code code or codex harnesses, and you can use those easily even with your custom agent. So, for example, now AISDK7 supports using Cloud Code, Codex and some other harnesses so that you don't need to create your own with your own tool calls, etc. So it's pretty cool. I really recommend to just play with it.

So, as with anything with AI, it's very easy to start and then it takes a long time to just fiddle with little wording changes. Great thing is that you can offload this to AI if you're good at setting goals and guardrails around it. Thank you so much for such a detailed answer. I mean, I'm just thinking, how can we reward agents? What kind of reward do they prefer? Like more tokens, tastier tokens, something like that. Thank you so much, Michal. It was a great pleasure to watch your session and have this short conversation. Let's stay connected and let's stay connected everyone. Bye, Michal. Thank you. Have a good day.