Fast-Tracking Quality for Hundreds of React Applications With Automated Testing Layers

Let's talk about testing. And this is a case study of the company I'm working in and the product I'm working on. And let me start with some confession. We develop applications, and for a long time, we were slow. Our release cycle, you know, this thing where you're planning, developing, testing things, and releasing them to the end users for us, like two years ago was roughly three months. And that's quite a pain for an engineer to see that your change is reaching users in such a long time.

I'm not sure who of you is sharing this pain, but let's get each other to get better, know each other better. So who feels this pain now? Or felt this pain in the past? Oh, ho, ho, I got the right people here. Please raise your hand if your delivery cycle is around a month. Okay, okay. And please raise your hand if it's like a two-week sprint-based. Oh, nice, nice, nice. And who is the happy ones that can release less than in one week or continuously? Not bad, not bad. Okay.

Yeah, so the whole talk will be about how we in the company went from here to here. And let me introduce myself. I'm Bogdan. I'm a principal engineer at Zatu. I'm organizing the React Berlin meetup. I'm a father of a teenager girl and this is not... This is hard. And a twin son. I'm an owner of Australian Shepherd and I'm a partner to my spouse in every adventure that she leads us. And the product I'm working on, and this is important to give you a context because all of the decisions we've made is basically coming from the complexity and a lot of optimization. But it's really, you need to first understand product to understand what we apply.

So the product, we are Zatu and it's all about streaming. Zatu providing an easy streaming experience, no matter of what device. It could be you are watching your football game of your favorite team in your mobile device while you're traveling on the train. It could be you gather together in the living room in the Christmas to watch some Christmas movie like a diehard. Could be you are watching automotive while you're traveling somewhere in Europe.

Or zipping your coffee and looking at what Trump did today on your fridge. And this is actually a real story. Not about Trump but the fridge. We put our application to the fridge and it was surprisingly working. PWA is awesome. As you see there is many different devices, many different platforms that we are targeting. And this is quite a zoo of technologies. And we are using different projects to make applications running.

So the projects in our understanding is something in our monorepo that we can build applications from. And our stack still is kind of mixed between web and native. Like Android with Kotlin, Apple with Swift, and web with JavaScript and React. We are currently switching to something more universal because of I'm not sure if you hear like Amazon just released a Vega OS which is very React native first. And this also drives us as pioneers to integrate with them and build applications for them. So we are switching towards all of the native and web stack to React native.

All the talk could arise from that, but I won't focus on that and guide you through the clients' support. And clients are something different from the project. Clients represent like a target platform or operating system like TVOS, Android TV, Fire TV, WebOS, iOS, Android, you name it. Windows browsers different, Chromium, if any other browsers live these days. And more to that, we are not only Zatu as a B2C product, we also doing wide labeling and producing applications for different tenants like Einstein in Germany, Air TV in Ireland, Monaco TV in Monaco, Zalt, Switzerland, I guess, Huawei in Netherlands, and so on, so on. And of course, Zatu, which we primarily know about. And this gives us quite a matrix.

So we have few projects, we have multiple clients, we have dozens of tenants, and all together, it gives us like a hundreds of applications that we would like to deliver to our users. And hundreds of applications give us hundreds of problems. And at this scale, always question was arising how to ensure fast and reliable quality assurance so we can ship smoothly.

The very big problem that we had, it was a manual quality assurance because it became with amount of growing applications, a bottleneck, not really a safety net. It doesn't scale. We couldn't just hire people with adding more and more platforms. The feedback loop was quite bad. We tried to even to outsource it, but still, there was endless coordination about which platform works, which not. Acceptance speed wasn't good. And by human factor, like a manual quality assurance is error-prone.

So what we did here is made quite a shift and redefined quality assurance in the meaning of new responsibilities and test automation. And we expanded responsibilities for quality assurance beyond traditional roles. So QA engineers transitioned to the engineering-focused roles. And testing was shared across all the disciplines, including design, including product ownership, and of course, engineering.

And the backbone for this automation, what made it possible was test automation. But no matter how you delegate to automation, there is still some other problems that could pop up, like delayed quality feedback, especially if you're testing too late in the process, when bugs are surfacing on the very finish line long after you committed the code. And this is not really good because you always implement bug fixes under some pressure. You're losing context because you're switching to another task and so on.

And by consolidating a huge amount of changes and testing them in the batch, it's pretty hard to know what contribution caused what bug. So what we did here, we implemented a shift-left approach, which is pretty easy. It's moving your target of testing from the very right to the very left in a very simple form. So we have this main, where everyone contributes. There is a features that is contributing to main, and there is some releases that you're tagging, or if you're using trunk-based strategy, you are forking from it.

And so for us, it was always there, like in the release stage. So what's shift-left approach suggesting you, test early the possible. So we switch it to the feature stage. And then about automation, so the automated tests are very diverse. We can apply and test different aspects. There is a bunch of different technologies. And I will show you some that we are personally using. For a static test, this one that's analyzing the codebase without running them, like linters, type checkers, we are using ESLint, we are using SwiftLint, StyleLint, Rubocop, Ruby guys are weird with naming, Hint, HTML, Printer, TypeScript, which is a huge safety net when it comes to the code quality, Kotlin detect.

There is some specific, we call it kind of build layer. So basically it's when you just assemble in your application. And for us in React Native, React Stack, basically you can run your application with Webpack and the Metro Bundler and assuring that the moment you would like to release, it's gonna be built.

There is unit tests layer, like a testing input out basically. I don't need to explain unit tests. We are using Jest, we are using React Testing Library. Jest test runner is great. We're trying to integrate now JUnit for this layer of native tests.

And there is a system test layer. System probably our internal word. You can recognize it by end-to-end integration function or whatever. Why we cannot call it end-to-end testing? Because we consider system or integration testing something that we are evaluating on highly mockable environment and on the simulators, not the real devices. Because when it comes to TV, for instance, it's a bit hard to test it. And this is very different story.

So on the system test, we still have some legacy from the Cypress after their turnaround of doing everything and like introducing component testing and being a little bit unreliable, so we switched it to playwright. And we have Appium, where there is a very interesting thing we are doing here. We are writing tests for playwright, connecting it to Appium and evaluating them both on the web and on the native, on Android side, on Vega side, TV as a core. And end-to-end, like a real end-to-end testing against target device. And there is very little technologies actually providing this capability.

So we are utilizing sweep test with candy boxes that really takes like a D-pad from TV and emulates real clicks on the real devices. And you see, I already kind of categorized this test into the layers. And this is another approach where multi-layer testing can help you actually to understand what technologies to pick up. And this pyramid that I'm suggesting to you is very representative. So I would recommend to build layer by layer if you don't have all of them. And don't rush to, for example, write end-to-end tests or unit tests if you don't cover like your base, your system test layer, because static test layer. Because static tests are super cheap, they are covering a lot, they fast.

So recognizing test layers. That itself, automation tests is good, tools are great. But executing them efficiently is quite a challenge. So you need to answer a lot of questions like, do I execute tests locally versus CI? Do I run everything or part of it? When one test fails, what do you do? Do you fail the whole run? And so on. Which application variation to use? Do I test only one type of application on one platform or all platforms and so on? So what you need is basically defining your strategy first before even executing it. This is an example from our documentation that answers all of these questions and is a guideline for our engineers, our DevOps, everyone who is participating in quality assurance, how to implement it, how to live it, how to develop it. You can bring it to something more concrete where you can establish some schema of how the things are orchestrated. And this is example for pull request, the target in our main development, and the whole thing. I won't go into details, it will take me ages. So you really need execution strategy to utilize tests efficiently. But it's still not enough. You really need to ensure that your tests are fast and reliable. And here is something more concrete for speeding up practices and how we made them quite responsive. We basically made them fast by caching what we could and running what's mattered. What does it mean? Let's start with caching. So you all know this famous quote about the two hardest things in programming, like naming things and caching. If you manage to deal with cache, that is the goal. So a very simple example from ASLint.

It's super easy to implement. You just need to cache-flag and that's it. It's doing the work for you. You can, of course, specify something more to that, like a cache location, for example, in CI to feed your target branch to that, so you are using this cache as well. Or implement cache strategy depending on how precise you want to be with this cache.

But a concrete example from our execution that called start for us in CI before, because we did not introduce caching into CI, was roughly around one minute. For a huge codebase with multiple projects. And it went down to four seconds. Of course, it depends on the amount of changes you produce, right? So this is a very light example when I'm affecting one model. But it's very comparable.

Not only you can cache your frameworks, you can cache your environment. Of course, there is Docker where we recommend this one. But also Docker comes with the cost of DevOps, and you really need a team. And if your project is not so big, or it's really web technology one, you really don't need the Docker. You just need to cache your node models. And this is an example from GitHub. This is GitHub workflow and GitHub actions that do this work for you.

So here is an example of run without caching, or with broken caching, actually. And we have a lot of node models. And for us, without caching, install of node models take around two minutes. And after implementing caching, it goes down to 26 seconds.

In the end, the static layer feedback, including all of the tools that I mentioned, for us, for any contribution that is not affecting a large scale, is around three minutes. Which is not bad. And if you're paying attention to what is written here, you might say, Bogdan, why do you lie to us? Android Lint will never finish in eight seconds, right? And this is true.

And this is where I'm heading to another practice that you can introduce, is change recognition. So basically test what you affected. And it could be applied to the many levels. And the very basic one, when you take in the Git, and just div 3 and see the list of the files that were affected, and just cut off everything you don't need.

It's not more about the static layer, but building a layer or a unit test or integration. Recognizing affected projects saves time by narrowing down changes. File system recognition is good, but insufficient for integration tests.

Developed tools with TypeScript compiler to analyze changes and connect code to tests. Identify affected components and execute associated tests for efficiency. Significantly reduce time by recognizing specific models and related tests.

Invest in test automation, shift left if possible, and start testing earlier. Choose test layers wisely and focus on basic aspects first. Decide where to execute tests for optimal resource utilization and speed up processes with caching and change recognition.

So if there is associated test with that, we're going to execute them. And some concrete examples for that. So here we have around 600 test suites for integration tests running in playwright. But let's say we just affected some, I don't know, one model. And by recognizing that there is only one model and maybe two, three, four, ten tests associated with them, we can reduce this time significantly.

It's a huge time-saver, especially if you want to receive feedback fast. Results, for us, we went down from hours to minutes for PR acceptance time. We stopped wasting our CI resources and pay for it and run only what matters. And from three-month release cycle, we went to sprint-based releases.

On that, I would like to share some takeaways with you, and I hope at least one can be applied. So invest into test automation, please. Shift left if possible. Start just earlier. Pick your test layers. And don't rush for something that is complex. Try to get the very first and basic things. Execute this test strategy. Decide where to execute what. Maybe, maybe for PRs that are targeting master, you don't really need to execute everything, right?

But maybe during your nightly monitoring, you would like to see the whole picture or for your releases to test in full or execute end-to-end test layer, which is super expensive there. And speed up with caching and change recognition.

On that, thank you for your attention. Thank you. Have you considered using TurboRepo? It's amazing at caching and incremental updates? Not really. Okay. Not really. What I would like to see, what it's proposing.

The thing is why we are not adopting some standards approaches is more about the essence of our project. We are multi-platform, and it's hard to, I don't know, we cannot use Expo, for instance, because we are pioneering some system that Amazon made one year ago and it's like no one knows about it. So it's a bit hard to use something like the industry driven thing. So we have a lot of custom things.

So we have a lot of custom things. Gotcha. Gotcha. Someone said, I like the Ghibli star images. I also like them. They're very cool. Oh, there is a story about that. Yeah? Yeah. You know, this hype of the AI and Ghibli images, it was a year ago, actually. And I first time built a presentation a year ago, actually. And nothing much changes from that time. It's still actually in one year. So all the practices that I shared, you can apply. I believe you will apply, you will be able to apply it in one, two, three years until AI replaces.

I also saw that you had about 630 playwright tests end to end. More. Even more? Yeah, of course. But then is your pyramid still a pyramid? How many unit tests and static tests do you have? Oh, Jesus. Well statics obviously cover everything. Yeah. So even this nasty test ignores that people are using. Unit tests currently around 50% coverage. Just because we started late and never considered the unit test layer as something meaningful, we're just using input-output. But then we started to go for components layer and so on. For end-to-end tests, for integration, Cypress playwright, a huge amount, thousands of tests. And also, as I said, like we have a multi-project setup.

So depending on the project, it's various. But also we have multiple teams and various how good team invest into the coverage. So hundreds and thousands. That's why we had to optimize.

We can't just execute all of them. Yeah, it makes sense. Very cool.

Was the QA teams moved to engineering a success? And how did you support them through this transition? Oh, that's a tough question. So first of all, it wasn't my idea. No? Okay. I'll wash my hands. We tried. So basically our QA engineers were facing this choice where either they get the package if they're not agree with this transition and we split up on the very good terms.

And to whom we decided to try and transition, there was many cases. Firstly, it was step by step. Like manual QA first transitioned to software engineering test, and then software engineering test transition to React. And now with this transformation, we are doing this React Native, React engineers transition to React Native. So it's always like a floating, it's a matter of adaptation. But we have a lot of success stories. And my personal job as a principal engineer to support them and that transition to explain the tools and lead them by example.

And my personal job as a principal engineer to support them and that transition to explain the tools and lead them by example. Yeah. Do you have like knowledge sharing sessions or like training budgets for upscaling? Yeah, of course. Workshops. Internal meetups and so on. Very nice. Cool.

How do you handle tests that can't currently be automated? So front-end accessibility tests, for example, they can't be done reliably. Accessibility can be partially automated. There is aches, but yeah, that's true, especially in our world where we are doing like TVs and TVs is not the easiest story to actually automate.

There is some layer of manual work. But as I said, we share this responsibility among all of us. So everyone is testing the feature, including product design and engineering. And if it needs to be done manually, everyone doing this manually. And this is another motivation, actually, to invest in automation. Once you do it once, you don't want to do it twice.

Once you do it once, you don't want to do it twice. Do you still have a lot of like physical devices since you are operating on a lot of hardware? A crazy lot. Yeah. Let's see.

How do you associate tests with the source files? End-to-end tests or out-of-process tests might not have an obvious import that declares the dependency. I never say that's 100% precise picture. Right. But this ID, basically any interactable element within your application has some test ID or should have it. So if you are trying to reach them by, I don't know, query, go left, right, top, or trying to get by dom nodes. It's better to assign some unique or in this case, test ID is not really unique.

There could be repetitive list. But all interactable elements have some markers. And this is kind of in the gray zone between black box testing and some kind of market environment where connection dot is actually this test ID. So you can really recognize this test ID. Was it affected by your code changes? And this test ID appears in your tests. So if it was affected, you execute the test. How we are doing exactly? TypeScript. TypeScript is building your model tree anyway. And we are just taking this TypeScript tree, comparing it with git changes and actually building a list of the tests that we would like to execute. Very nice. Cool.

How do you guarantee mock data will be synced, for example, with API changes of the backend? You don't. You just handle it. You need to understand why we are using mocking. What's the purpose of mocking. Do not do network requests. To have fully controllable environment. And this is very different type of test. That's why we're not calling integration test. Integration is like a system test in highly mockable environment, plus emulators. But what it gives to us is just a display, right? Super fast. All right? Because you can bypass this network request login, for instance, doing a lot of things. Of course, it won't be a precise one end-to-end scenario that the user will face. But it covers around 95% of all issues. And for these crucial scenarios that you really need to ensure this working, like authentication, payments, and so on, yes. There you use real tests, no mocking, real devices to ensure this happening. Gotcha.

Who's responsible for updating mocks? Is that the person from the backend that makes the API change? No. Fortunately, what I like in my company is like a backend and frontend are strictly separated. There is only a P layer and this agreement documentation. So if backend is breaking something, they're just releasing a new interface to us. And this is a responsibility of frontend to switch to them. And when they switch to them, when we switch to them, we're basically updating our mocking or deprecating or introducing new ones. So yeah, there is no breaking changes into that. And it's kind of more frontend responsibility because it's frontend tests. Very nice.

Are your tests ever flaky? And what do you do if they are? Yes! Yeah, of course. That's the truth. That's the pain. That's the hugest pain. So it's a never-ending story.

You're testing, everything works on your local machine, you're in CI, everything is flaky, or it was stable for a week, it's gone now. So first of all, what helps us is the strategy that we are considering to master is always releasable. What does it mean? So if something goes wrong, we go back. So we have this beautiful tool called Git. It allows us to travel in time. So if some flaky test is there, two scenarios. Either we're going to skip it, either we'll revert it, depending on urgency, severity, and so on. And then, obviously, Ticket will be re-opened and the team responsible for this test will rework it. Got you.

So there's no nonsense, no retry, you handle it? Depends on... We try. That depends. If you have enough time to stabilize your release, for instance, fine, you can just do some tweaks. But if you have an hour before, just be practical. Yes. All right. Thank you so much, Bogdan. Thank you for sharing. Please give another round of applause for Bogdan.