Maestro & Expo: Crafting the Future of Efficient e2e Testing

Hey, I hope you're doing well. I'm Mathieu and it's nice to be here today. I'm going to talk about end-to-end testing. So think of your app as a high-performance sports car. You wouldn't just test the engine or the brakes or the steering in isolation. You would want to take it out for a spin, see how it performs on tracks in real-world conditions. Well, that's the essence of end-to-end testing. It's like your test drive. It examines your app in its entirety, catching those hard-to-find bugs, especially in native code, that unit or interaction tests might miss.

Maybe you've heard about the testing pyramid or testing trophy. They always mention end-to-end testing but the reality is that I've worked with different clients at YOLO apps and I rarely saw a good end-to-end testing setup. And most of the time there are no end-to-end tests at all. And I believe that's because there are a lot of pain points.

So, first of all, the end-to-end tests for me are still complex to set up and use. I took more than a week to set it up with some tools. The tests can be unreliable and it's going to be annoying to set it up on CI-CD properly because you have to build the app, start the simulators, have the right environment. And last but not least, it's almost always slow and expensive to run them regularly. So, my talk today is about my journey at YOLO apps and how I tried to fix all of those issues to have the perfect end-to-end setup.

First of all, I tried Appium. It's a black-box end-to-end testing tool. There's no need to bundle or to install anything inside your app. You just give it the app, the APK. And I think it's the most established tool and also the most powerful one. Appium can replicate almost anything a real user can do.

But for me there are also a lot of drawbacks. It's really complex to understand the documentation, to set it up properly. I spent a lot of time to have my first test running. There's a huge learning curve and there's also no CI-CD support. So, you're going to have to handle the simulators, the build, and so on.

Then I tried Detox, which is a React Native end-to-end testing tool. I personally believe that they are pretty similar. I mean Detox is a bit simpler and easier to use overall but on the other hand it's less powerful. And after using those two tools, I realized it's still not something I would see my team and I use every day. They don't address the pain point I mentioned earlier. So, I had to continue digging and finally I discovered Maestro. Maestro is a newer tool. It works on React Native and every other mobile platforms. I think there's a really cool community around it. They have a Slack where they answer questions very quickly, for example. But the documentation is very cool and instead of talking about it, I'm going to show you how it works.

So, I've got here a simple app. I have an info button that opens the modal and I can swipe this modal back down and this is the test I'm going to try to do with Maestro. These kinds of tests is really hard to do without an end-to-end testing tool. With a regular integration testing tool like React Native Testing Library, you usually have to log the navigation. But you will see how I do it. The first thing we're going to do is use Maestro Studio. It's a tool that comes bundled in with Maestro and it's basically a web interface where your simulator is embedded and I can hover over my simulator and some zones and some text will be highlighted. This tool is here to help me construct, to help me build my tests.

So, if I click on something, you will see that I have some recommendations. I can tap on this button, I can tap on a test ID, I can tap on a text, I can even tap on some specific points on the screen. I think it's a bit less recommended but you still can auto-type your test quickly, even if you haven't set up your test IDs properly. And I can interact with everything in my app. I can also do some assertions if I want.

But let's try to construct the test. So, I'm going to click on my info button. The first recommendation seems pretty nice. So, I can simply select it and it will run this action and you should see the result of my simulator. Yeah, the model just opened. Now, maybe I can check that the text is visible. So, I'm going on the assertion recommendation and this one seems exactly like I wanted. So, I can select it and it's passing, as you can see with the green tick mark. Next thing to do is close the model back down. And when I click on info, you can see that I don't really have any good recommendation. I want to swipe it, not type it, not tap it. So, I can simply open the docs. You will see that there are a lot of actions you can do with Maestro. I can assert something is visible, not visible, I can input some text easily, I can scroll inside my app, and I can also swipe. So, we'll have a look at the swipe documentation.

Here, I want to scroll down. So, this example seems pretty nice. And when I found something I want, I can copy it, come back to Maestro Studio, and I can just input anything I want here. So, I'll paste this stuff, modify it so that I can add my test IDs and change the direction. And when I'm done, I can execute this part, and you will see that my model slides back down. The last thing we can do is assert that the model text isn't visible anymore. Do that with assert not visible. And when I'm done, I can simply select everything and export it so that I can just create this test inside my codebase. So, I'll just copy this and come back to my app. I have made a test template that I will come back to later on.

You'll see there is a tag here. I'll explain this later, but basically, I'm just pasting my test here. And yeah, the test is opening the app. We're clicking on info, asserting something is visible, and swiping back down. So, I run the test on Maestro Studio, but I can also run it with Maestro only. So, you can see what happens when I just run my test. The video is not sped up. So, that's the real time of Maestro test. And as you can see, the app should close and open back up. And it's very quick. You can see it happen on the side. The test passed. I'm good.

Last thing I want to show you is another tool from Maestro, which is Maestro Cloud. And with one simple command, I can, instead of running the tests on my machine, I can send the tests and my Apica to the Maestro Cloud servers. It's their cloud services. And the tests will then run on their servers, on their simulator. I don't have to manage anything. And I can just go to their web interface and have a look at the different executions. So, that's very cool, because they take care of everything. They have auto retry if the test fails. If it fails, you can have a look at what step failed. And there are even some videos of the failing part of the test that helps you understand what went wrong.

And, well, that's it for the Maestro demo. I mean, Maestro for me was something that really solved a lot of my pain points. The install is super quick and it's very simple to use. It's never perfect, but Maestro helped us a lot about flakiness. And then, well, running it locally is great, but CI CD is still partly addressed. You saw I can start the test on Maestro Cloud from my machine, but not from the CI yet. I still need to build my app first on the CI.

I still believe I needed something fully automated to really get my team on board. For example, I want something that can be done automatically on a pull request or when you merge to master, for example. So I tried to have a look at the documentation, at the articles online, and I saw that they recommend you rebuild the app for each test. And for me, that's a bit annoying. I don't want to have to build my app during the CI CD. I will have to manage the cache, the environments. It's going to be slow and expensive, too. So I dug deeper and finally I realized that at Theodore Apps, we also use Expo a lot and it can help us here. So let's see how.

Quick recap on Expo. It's a set of tools that allows you to iterate quicker. It's even recommended by Meta now. Maybe some of you used Expo Go before. It's good for prototyping, but it's not recommended in production. So here we use the custom development builds. It's like my custom build. Custom development build is the native layer with the native code and the libraries you need plus some useful Expo stuff on the side. And this native layer is like a shell inside of which you inject your update layer. So you can build this shell locally with Expo, which is going to use Xcode or Gradle and other hood. But you can also use EAS, which are Expo cloud services. And with only one command, the build will be made on EAS servers, which will probably be quicker than my machine.

And once you build on EAS servers, the build stays available for everyone on your team. And that's really helpful because you only have to build once and then everybody on your team can just simply download it. So that's the native layer. And so inside this native layer, you can inject the update layer, which basically is the JavaScript layer. It's like the JavaScript bundle you have when you run start. But you can also use EAS to create some update layers and store them on EAS servers. Here you can see that I can easily inject the latest layer that has been done on the develop branch inside my native layer. And it means that if my colleague has the native layer on his phone, he can quickly download the latest update and see what the app looks like on develop currently. So that's what we use to address my CICD issues with Expo.

We don't need to build on each pipeline, and this will make it fast, cheap, and much simpler. Let's see what the solution looks like.

Here is what happens when I open a pull request on my project. I'm using GitHub actions, but you can use whatever you want. The first step is to create and publish the update to EAS. It means that I'm bundling the JavaScript code and sending it to EAS. When it's done, and when the update is created, we show a QR code that you can scan if you have the native layer to see the state of my app as it is on my pull request branch, which is very useful for even the PR.

And next, instead of building the app here, we can also use EAS. Then I simply download the latest build I made on EAS. And this is where we go much quicker than having to rebuild the app. Because now we have everything we need. We have the APK and we can just send things to MySQL Cloud.

The next command is the MySQL Cloud command I showed you earlier. I give it the path to my APK or app file for iOS, and also a parameter which is the ID of the update we made earlier. Which means that now when the app opens on MySQL Cloud servers, it opens with a deep link pointing to the right EAS update. It will then fetch this update from EAS. And when it's done, now the app has the same state as it is on my pull request branch.

So the last thing we can do is run the tests and get the reporting back to my pull request. I'm using the MySQL Cloud hook here to have a nice reporting. And you can see here that my full CI took only less than five minutes. So I think that's very cool. I can also show you what happens when the CI fails. So I can get more details here. As you can see, my own test passed, but model one failed. Let's have a look at why. It will bring me back to the MySQL Cloud interface. I can see that this assertion here is the one that failed. It was swiping down, apparently. So to get more info, I can just simply start the video. I should see the app that opens.

I click on the info button, the model opens. And yeah, it's failing. For some strange reason on Android, the model is not a nice model that pops from the bottom, but it shows us a page here, which is strange. But that's why I showed you this example, because that's really the power of end-to-end testing. That's something that you could easily miss if you only test on your simulator. And that's also something that you would never have coped with the regular integration testing.

So to sum up with this solution, we have a CICD time of four minutes. The tests are free and the EIS updates are free. Whereas if you were to build yourself on the CICD, it would take much more time. Maybe you would have to pay the production plan of EIS so that you can build a lot, or maybe if you want to build yourself you would have to deal with the caching and maintenance of the CICD. So with that, we fixed every pain point.

On the last slide of this presentation, there will be a QR code where you can find my example repository. And please feel free to use it to copy it for your apps. You can see there how I set up my CICD. It's very simple. Before I finish, let's talk about some pain points. Maestro is a new tool. It doesn't have iOS device support yet, only simulators. There's no device found support yet either. The tests are written in YAML, which is annoying because it's not type-safe and not standard. But it also means that writing the test is easy and maybe your product owners can do it, or maybe your designers can do it too.

And the 100 free tests per month limit, maybe it doesn't seem like much, but my advice is you should totally try it on your project. 100 tests per month is enough for you to get an idea of whether or not Maestro helps you. And you don't have to run the test on each pull request. You can run them weekly or maybe only when you publish an update to your app, for example. If you remember earlier, I showed you that you can add tags for the test. So a test with this tag will mean it will only run on every pull request or it will only run weekly or only run before we deploy an update. You can see how it works on my example repository, but this means that you can choose your own testing strategy. Maybe if you have a critical feature of your app, you want to test it on each pull request, but the other features that are less important, maybe you can only test them before you deploy an update. And if you want to do more than 100 tests per month, the pricing is 10 cents per test and you can always safe host down the line if you want. Well, that's it for me. Thank you very much. If you scan this QR code, you can see the example repository and also the slides. You can find it in my GitHub. Please don't hesitate to ask any questions in the chat or on my Twitter. And you can also come take a look at what we do at Theodore Ops. Thank you very much. Bye.