Mobile Device Testing for Cross-Platform Apps

So, yes, this is mobile device testing for cross-platform apps. As you probably have heard by now, I am Cecilia Martinez. I'm a developer advocate for AppFlow, which is a mobile CICD platform built by Ionic.

So, you're probably more familiar with Ionic or Capacitor than you are with AppFlow. But you can learn more about mobile deployments, mobile building, by following me on Twitter or GitHub at Cecilia Creates. Or feel free to contact me on LinkedIn as well at just LinkedIn.com slash in slash my name.

So, when we talk about cross-platform, what we're talking about specifically is apps that you build with one code base, but that you deploy anywhere. So, iOS, Android, and then also to the web. So, there's a lot of frameworks that you can use to do this. If you're thinking in the kind of the web ecosystem and web-based frameworks, the ones that are going to come to mind are React Native and then Ionic paired with Capacitor. But you can also use tools like Flutter, which is based in Dart, .NET MAUI, or Kotlin Multiplatform. But the idea is that you build ones to deploy anywhere. And this gives you not only a faster development process, but a more consistent developer and user experience across platforms.

You've often seen something where, you know, the iOS version of the app has a feature that isn't available yet on Android, or maybe there's, you know, certain apps that are only on iOS. And that's because typically you may see separate teams working on those apps individually. With cross-platform, you erase that kind of problem. But cross-platform is not without its own challenges. So, while the dev processes are the same, the deployment processes are not. You have very, very different build, testing, and deployment processes across web, iOS, and Android. This creates really specific challenges when it comes to testing these applications, especially when you're testing the actual native builds on real devices. You have platform-specific test cases that you have to consider as well. So, even if you write the code the same, it will actually engage with the device differently, and you have to consider that. And that requires some expertise of each platform. So, if you are a web developer and you are building cross-platform for the first time, and it's the first time that you're dealing with mobile applications, you have to learn the nuances between Android and iOS to be able to effectively test those applications. And testing is really important for mobile applications. Deployments are high risk. They take a lot longer. You have to go through the app stores and deal with that approval process. So, the apps need to be very well tested before they are installed on a user's device. Users are also very picky about their mobile applications. They encounter one bug and they uninstall, or they never use it again. So, it's important that you are able to test these applications appropriately.

So, in an effort to help address some of these challenges, there is a lot that we could talk about, but in the time that I have, I'm going to focus on these areas. I'm going to talk about the types of devices, virtual and real, that you have for testing your applications. I'm going to talk about the specific build types for iOS and Android. I'm also going to talk about how you can distribute test versions of your app to manual testers. Yes, I'm going to talk about manual testing just a little bit, I promise, but it's important for mobile. And then I'm also going to talk about when and how to leverage automated testing, both on Android and on iOS.

So, let's start with devices. Now, devices, when I talk about that, it's literally the phone that you probably have in your pocket right now. But first, let's talk about virtual devices. So, these are software devices that replicate a real device. For Android, you have emulators. For iOS, you have simulators. There's a little bit of technical differences. They're essentially the same, but because they're different platforms, they want to each have their own special version of it. So, you have emulators and simulators. These come pre-installed with Android Studio and Xcode. So, if you are using Android Studio and Xcode to develop, you already have these on your dev machine. If you do not, for example, if you're not responsible for actually creating those native builds, then you don't have those emulators and simulators. So, that can make it a little bit difficult as well. They can be used for manual or for automated testing. So, if you want to just pull one up quickly, launch your application, and click around, you can do that, or you can also link them for automated.

Next, we have real devices. So, real device testing is important because it does provide the best fidelity of user experience. Even on an emulator or a simulator, you're not actually able to, for example, take a photo as well as you would with a real device. You're not able to understand certain, like, load capabilities or performance issues that you may see on a real device. There's also different types of devices that you may not have access to on an emulator or simulator that you could be able to get on a real device. So, real devices are important to test on, but there's a lot of complexity involved around it. So, for iOS, devices do have to be registered to be able to run the native binary on the app device itself. You can either do this by connecting the device to your dev machine. For iOS or for Android, you can also download and install the binary to the device.

And again, you can use real devices, obviously, for manual or for automated testing. So, you are able to run automated tests on actual real devices. So, let's talk about builds. So, when we think about, you know, web builds, really you get to decide what that web build looks like. So, you may have a test version of your web application that hits, like, a different API end point or maybe lives in a different environment. But ultimately, you are in control of what that web build looks like and what goes into that and how it's configured.

For mobile platforms, Android and iOS, they actually have specific build types that you will execute when you create the compiled native binary that will run on a device. Because of this, you actually have to think about what type of build that you need to do depending on what type of testing you are going to be able to do. So, for Android, you have debug and release. For iOS, you have simulator builds, which of course runs on a simulator. And then you have development, ad hoc, app store, and enterprise. I like to say that iOS is always going to be twice as complicated as Android. And so, you will see a theme where for iOS, you have twice the amount of build types, you have twice the amount of signing credentials that you need to use to sign your application, you have twice the amount of developer programs that you have to sign up for and pay for. And so, iOS is always going to be a little bit more complicated.

So, when you think about build types in the context of testing, I like to think of web builds as ideal if you're going to be able to do some browser-based testing. So, you can either do this in a web browser with mobile viewport, or as we saw earlier, you can use Nightwatch to launch an actual mobile browser on an emulator. This is great because there's no code signing or specific hardware required. If you're running in a browser, if you're running on emulator or simulator, you don't necessarily have to go through the signing process. If you haven't gone through code signing before, it's creating a signed bundle that verifies who created the bundle, that they were authorized to do so, and that it was not tampered with or altered since the time that it was created. Builds require specific credentials that need to exist on the machine that you use to create the build. So, it can be difficult to just spin one up if you don't have access to those credentials. So, that's why web builds and virtual device builds are ideal for testing, because you're able to just create those on your own. For Android, we have debug builds, which are also ideal for testing because, again, they don't have to be signed, and they also are an unoptimized version of the build. So, if you need access to the underlying code, it's easier to identify and debug. For iOS, you have your simulator builds for virtual device testing. If you want to do internal real device testing, you're typically going to use a development or an ad hoc build. And if you'd like to upload to test flight, you're going to need to do an app store build, which is a certain type of build. Now, development, ad hoc, and app store all do require code signing. All right. Let's talk about app delivery.

So, when I talk about delivery, I'm talking about how your manual testers will get your actual version of your application onto their device for testing. So, manual testing is still a really large and somewhat required approach for mobile app testing because there are some specific things that are difficult to automate with real devices. And I was surprised at this. I come from a web testing background, and I'm at the automate all the things, you know, level of testing. So, I was surprised at how much manual testing there is still in the mobile app space.

This does require device management. Like I said, you not only have to register all your devices for iOS builds, but you need to think about what types of devices you need to test on. So, what types of devices are your users using? What's the performance capabilities of those devices? How far back do you want to support? So, if you're using like only like testing on the latest, newest iPhone with like the best, you know, specs and everything, that may not be the same experience as somebody who's like on an old phone that you can get at the grocery store, you know, experience. And so, most people will create a test device matrix that covers the different devices that they need to actually test against.

There's a lot of options for delivering those app binaries to your manual testers. One that I see a lot is actually just manual delivery. And this is where testers download and install the binaries themselves. So, there's usually a centralized storage. I've even seen somebody use a Google Drive where they upload APK or IPA files, and then the testers are notified that a new file is available. They download it, they install it onto their device, and they start testing. And this can be difficult to track versions or to know when new releases are ready. So, if you have a centralized storage of all these different versions, but it's just an environment and it's not necessarily using any specific kind of tool, you may not know which versions have already been downloaded and tested. It's not easy to track feedback or understand which version is tied to a specific commit. So, I really recommend using platform delivery tools. So, these are specific to each platform, so you do have to set them up separately for Android and for iOS. For Android, you have what's called test tracks. These are built into Google Play. And you have internal testing tracks, which allow you to distribute up to 100 internal testers. And then you can promote that release up through the multiple channels after that, which is the alpha or closed testing track, and then finally, the beta or open testing track. On the iOS side, you have what's called TestFlight. Who here has ever used TestFlight before or tested an app in TestFlight? Okay, great. Yeah. So, this is really the way to go, I would think, you know, for on the iOS side. It allows you to have an internal testing group as well as external testing. So, on the Android side, this is the internal testing track.

You can essentially create lists where you add your internal testers. And then whenever you add a new release by uploading the app bundle, that's then available to those testers, and they're able to track and see which version it is that they're connected to. Again, on the App Store Connect side, you can create an internal group. And this is for, again, up to 100 users, and that's people who are associated with your App Store internal organization. But then you can also promote that to TestFlight, which allows you to access actually 10,000 external testers. However, you do have to go through App Store review before you can submit for external testing on TestFlight.

So, this has so far been a pretty manual process, even if you do use the Google Play Store and the App Store Connect tools. So, there's also some ways to automate delivery to these testers without having to manually upload those bundles. So, there's developer tools to automatically send those binaries to Google Play and App Store Connect. And this also allows for better tracking of your app versions. There's App Store developer APIs. There's CICD integration tools. There's one called Fastlane, which uses, like, a YAML-like syntax in order to orchestrate things like uploading to TestFlight, things like creating your native builds, and even things like testing. Or you can use a mobile CICD platform, like Bitrise or AppFlow. So, for example, what this looks like in AppFlow is you create an automation every time there's a push to a specific Git branch. That kicks off a new build, and then we are able to deploy that directly to your TestFlight destination. And so, every time you have a push to your main branch, this will actually be ready to your internal testers automatically, if you have automatic delivery enabled, or you can submit it for review for external testing.

The fun part, automated testing. So, we talk about automated testing. There's unit, component, and integration tests based on the specific framework that you're using. You can also do platform-specific testing, Espresso for Android and Xe-Test for iOS. But typically, you'll be able to just, if you're using React Native or Capacitor, you'll be able to use the testing tools that you already know. You can do browser-based testing of web builds or end-to-end testing of native builds on virtual or real devices. There's frameworks like Appium, Maestro, and Detox for doing this. So, virtual device automated testing. You are going to essentially use an iOS simulator or an Android debug build. Again, these are built into Xcode. You need to install the app on your simulator or emulator, and then you can run the test against the virtual device. So, we're going to take a look at some code.

So, we're going to take a look at some code. This is an Appium Android test code. The test itself is pretty straightforward. Really, we're just going to zoom in here. We're just clicking through some tabs. So, we're opening the application. It has three tabs in the bottom. We're clicking through to the second and third tab and making sure that those work.

And so, what I wanted to point out here is the driver section. So, the driver section is how we actually configure how this test is run. And that's going to be based on this WD-ops object here. So, we're just running this locally against an Appium host that we're running on our system against that port. And we are also specifying a device name of Android and the location where the binary is located. This is important because then when I start my Appium server and then I run Node-Android-Test.js, it's going to locate that app, it's going to launch the Android browser, I mean Android emulator. It's going to take that location of that APK. It's going to install that and it's going to run the test against it.

So, what if we want to run this on a real device? So, again, it requires some code signing and device registration for iOS. But you can either connect a device to your developing machine, run the same thing, but specify the real device instead of the virtual device. Or you can use something like a real device cloud service. This is way more resource intensive, especially if you need to manage all of these real devices yourself or use a cloud service. So, it's best for kind of smoke testing or things that require the real hardware specifically. So, let's take a look at an example of that. We're going to be using something called SOS Labs. There's lots of options. I just used SOS Labs because it had a free plan. Primarily, you're going to see mostly Appium support for these device cloud services. Hopefully, we'll see more support in the future of other frameworks as well. The process is you're going to need to create a test build, upload that test build to the cloud service. You configure your test with the build file name and device type and you start your test server to run it. So, we'll take a look at the code.

Like I said, first thing you have to do is actually upload the Android or iOS app. You can do that by uploading it in the SOS Labs console or you can also use cURL.

So, here's the same test we were looking at earlier. However, the difference is that driver is going to be a little bit different. Instead, we're passing through our SOS Labs user and key as well as setting the host name and port to use SOS Labs to run the test. We're still passing through the capabilities object. Let's take a look at that.

Again, we're still passing through a file location. However, in this case, it's actually a file name on SOS Labs server because we've uploaded the file there and we're telling it, hey, SOS Labs, use this file name and install that on the Samsung Galaxy S9 Free device and run the test against that. We're also passing through a build ID and the name of the test. This is important because it means that our test record will be connected to the build ID which will also be connected to a test name so we can easily track back if something goes wrong the actual build that initiated that test.

What that ends up looking like is after the test runs, you see this in your SOS Labs dashboard. You can see the video recording of the test. It's clicking through the tabs. As you probably noticed, it takes a long time to start up. That's because it does have to install your app on the device before it can start running the test. Like I said, this is a resource-intensive process. You're using a real device, devices cost money. It also takes time to run these tests because you do have to install them before each one. Really consider what your test cases and use cases are for this.

Now, again, this is still a little bit of a manual process. You're uploading the app binary. You're launching the test from your console. What if you want to do this in CICD? The process is still the same. You're going to create your test build. You're going to upload that test build to the cloud service. You're going to configure your test with the build file name and the device type. You're going to start your test server and run the test. This time, we're going to do it in a GitHub Action. This is a GitHub Action workflow to do an Android Sauce Labs test.

I'm setting some environment variables. My Sauce Labs user, my Sauce Labs key and the APK file name. Even better, I'm now actually tying that to the GitHub SHA. Now I actually know which commit that build was created with, so I can trace it all the way back to the actual commit that launched this build.

The next step, of course, is to actually create a test build. I'm using the Ionic Cloud CLI to do this. Really, you could do this on the CICD machine that you're using. You will have to maintain the build stack in order to create the Android or the iOS builds. If you're doing iOS builds, you're going to need Mac hardware. If you're using Ionic Cloud or App Flow, you can instead use their credentials and their infrastructure. They handle the signing credentials for you. The important part to note here is that we are naming the APK the same as our GitHub SHA, that APK file name environment variable. This keeps everything consistent and organized.

Again, the next step is to upload the APK to Sauce Labs. I'm running the curl command, passing through my user and key. Again, like I said, the important part is that I'm naming it that APK file name. This is going to connect the file name to the GitHub SHA, to the build. Everything is really consistent and easy to use.

Last but not least, I'm actually going to run the test. This is where I start Appium. So I start my Appium server, and then I'm running my node appium slash Sauce Labs Android test.js file. I'm passing through those environment variables of my user, my key, the build ID, which corresponds to my SHA, and the file name to locate that test.

A couple of quick best practices to note before we wrap up. Leverage web testing as much as you can, whether that's going to be in an actual web browser or it's going to be in a mobile browser. The more use cases you can cover with web testing, the less resource intensive your tests are going to be, and you're also going to be able to identify issues earlier before you get to more native end-to-end testing. Use mobile-specific tools or have a dedicated team member for mobile. You're going to really need to understand those nuances between the platforms, and somebody on the team should be prepared to handle that if you're not going to use mobile-specific tools. And also automate incrementally. Don't start right out of the gate with trying to automate your entire test code based all on real devices. Identify where the bottlenecks are and what the critical paths are and start to automate there.

All right, thank you so much. You can grab the entire GitHub Action Workflow from using the QR code there. And feel free to let me check out useappflow.com if you have any questions. Thank you so much for your talk. I really appreciate it as well. And the thing that I love is it was, and you get these talks where they're kind of framework agnostic. So you spoke about kind of the ideas and the thought processes behind this. And there's a bunch of different sort of frameworks someone could use. And are there any sort of framework-specific considerations, maybe, that you have seen where you've seen other people try to implement these processes out in the wild? Yeah, so two things that come to mind specifically. One is that if you're using a web-based framework, you're going to be able to leverage a lot of the web tooling. A lot of the frameworks that aren't based on web technologies, they have their own specific ecosystem. So you're not going to be able to use specific packages that you might use for, like, a general web project or some of the web tools that you're used to. So it just creates a little bit more separation. The other thing to note is that in terms of the frameworks, you know, differences that may occur is the configuration aspect. So there's different ways to configure iOS and Android-specific parts of your app depending on what framework that you use. And some of them can get really complicated. You have to actually get into the Swift files and edit themselves. So I would also take that into consideration, too. Classic. I just needed to make sure we got an it depends answer, kind of, on stage before we begin. All right, thank you as well.

And those of you who are asking questions, please keep those questions coming. I'm going to jump in with these audience questions. The first one is, how can we automate the PWA build type? Does this cover from just web build testing? Yeah, so it is essentially like a web build. There are some configurations that you need to do in order to create that PWA. For example, if it has any actions that take place when you're saving it to the home screen, you may want to test those separately to ensure that those work. For the most part, you just will kind of install whatever PWA package that is used for the framework that you're building with, and then you can go from there. I actually am a big fan of PWAs, and I'm glad to see that they're getting more support, and that Apple and iOS are moving that way and being able to allow us to leverage PWAs more. But yeah, that's a really great option. And on that note, actually, one of the things I ask people a lot of times is they say, oh, I want to build a mobile app for this.

And I say, do you really need a mobile app? Because once you're in the app stores and you have to start going through that process, it becomes a lot more tricky. You have to also do constant ongoing maintenance to ensure that you're always up to date with the latest standards. So a lot of times I say, if you can get away with mobile web or with PWA, maybe go the easy route and don't make it too hard for yourself.

For sure. And also, I should do this more often, but just for people who maybe are not familiar with acronyms, PWA. Yeah. Progressive Web App. So essentially it's a web application that you're allowed to save to your home page the same way you would with a regular app. It also has really great offline capabilities. That's something that you kind of miss out on with just mobile web apps. And so that's something that essentially mimics the mobile app without interacting with the native layer.

Definitely. All right. So let's jump into another question. This one's kind of tooling specific. And asking your opinion, because I'm sure you have used a couple of different tools out there. And so what's your opinion on Expo and Expo Go as part of that development process?

Yeah. So I've used Expo and Expo Go. I think it's a really great developer experience to get an app off the ground. It's super quick, super easy. I would even say for a lot of use cases, if you're going to use React Native, use Expo. I think just use Expo. I've done it both ways, with and without. I think obviously, I'm going to use It Depends again. The type of app that you're building with the framework that you choose will depend on what your project is and what your team looks like as well. I'm at Ionic, and we use web-based tooling. And it's all specifically web. So you use React, Angular, or Vue. So if you have an existing web team that's already building with those tools, and they want to build for mobile, that can be a really easy transition. But if you're looking for something that you're maybe building from scratch and you're looking something a little bit different, Expo and Expo Go, I think, are also really great tools.

And I think it just goes to tell you, when you're looking to try something new, there's a range of different tools out there. You don't just hear one tool and be like, that's the tool I'm going to use, right? Right. Exactly. And then we kind of have another one, which is more about just approaches. So what approach will you recommend to test web apps that run on mobile and desktop browsers? And using maybe tools to do emulation or using real devices, so some of those tools for that?

Yeah. So the nice thing about cross-platform apps is that one of those platforms is the web. And I will always start with web first if I'm building cross-platform that builds on the web because it's going to be the least resource intensive. I can also use the tools that I already know and love. The reason that Appium test was very simple is because I'm not as familiar with writing Appium tests, right? And I would much rather write Cypress tests if I can. So if I can cover 80 to 90% of my use cases using browser-based testing, I'm going to do that. And then looking for those platform-specific edge cases where I do need to leverage a real end-to-end native testing tool. The nice thing is that we are seeing more and more tooling. There's even things like Maestro. We talked about Nightwatch earlier that can use some of the more paradigms that you're used to with web-based and then still run it on an emulator or simulator to do some automated device testing as well. I love that as well. And that way you get to, especially when you're starting, you can focus on testing the business logic with the ecosystem that has lots of tools available. And then as you get more specific to kind of stack to then device, then changing the tools that you use along the way as well. I really like that.

And then we've talked about maybe sort of different types of testing, but specifically performance testing on mobile. How would you automate that? Yeah, so performance testing is a whole different type of beast, I will say. It's something where I tend to focus on the user testing, acceptance testing. Performance testing, there are specific tools for that because it's really going to depend on the device that you're on. And it's also going to depend on the network connection that you have. And there's actually ways to simulate that the same way that you can in a browser, right, where you can simulate for connectivity, you can simulate different devices. So that's where that device matrix is going to really come in handy. And you're typically able to run a test, and you're able to record how long it takes to do certain actions on different types of devices. And there's going to be specific performance things that you can run as well. There's also a lot of really great mobile test frameworks that aren't code based. They're platforms that essentially will run those tests, like will create and run those types of performance tests for you. And they'll even just generate it based on the binary. They'll run some checks, and they're able to validate. You can also do things like optimizing based on the bundle size, and ensuring that that's going to be consistent performance-wise for your end user. Totally.

I don't think we have enough time for one more question, so I'm going to call it here. But thank you so much. And if you do want to ask questions to Celia, you can always go to the Q&A discussion room here in person. And if you're online, head over to the Q&A tab in Discord so you can ask your questions there, and she will also get to them. Let's give a massive round of applause to Celia one more time.