DevOps for Frontend: beyond Desktop Browsers

Hi, everyone, and thank you for joining me. Today, we're talking about DevOps for frontend beyond desktop browsers. My name is Max Gallo. I'm a principal engineer from London. I work at DaZone. You can find me on Twitter at underscore Max Gallo. The topic of today is quite a big topic, and I want to smash it in three different parts.

At the beginning, we will set the context. We will define what beyond desktop browsers mean, especially in the DaZone context, in the live sports streaming context. After we've done that, we will tackle the journey to DevOps. So how did we arrive to the DevOps as we know today? And which are the features that we can steal, that we can borrow from the back end DevOps to the front end? And at the end, we're going to see how those features are actually applying if they are applying for the front end.

But let's start beyond desktop browsers. DaZone is a live sports streaming company. So we serve live sports all around the globe in so many devices and in a lot of countries, more than 200 countries. Among all these devices that we are targeting, there are almost 25 of them which are HTML and JavaScript based. So it means that we are running our web application into some kind of browser which is embedded into the device. But which devices are we talking about? So I'm mostly talking about your TVs that you have in your living room and your gaming console, your latest and shiny PS5 or Xbox or maybe a set of box that comes with your network provider and actually maybe more. We are currently supporting more than 25 of them.

So in all this idea, with all these devices, we created our workflow as it follows. So we started from feature teams. This is our workflow let's say up to six months, one year ago. So our feature team were working on common features. So all the common feature which were shared across multiple devices were starting from here. We had multiple feature teams and once they were done, they were handing over the code to the devices teams. This team's goal was to make sure that the app was working fine on the device itself. And once the devices were happy, we were able to go finally to customers. So the idea is that feature, then devices, and then customer.

But this particular setup has its own challenges. And I want to start with the first challenge, which is as low feedback loop. Why is low feedback loop? Mostly because we have two different teams that are needed for going to one place into another one and to go from the local idea of this feature to our customers.

So imagine any bugs or any feature that goes from one team to another one. If you find any problem, it has to go back, and this goes on and on and on. And you basically have these two teams constantly talking between each other. So that could slow things down.

Challenge number two, work visibility. Imagine the best case scenario. You have one Jira board for the feature teams and one Jira board, let's say, for the device teams. But also, we have multiple feature teams. Actually, they are split by domains. So different domain teams working on all the features on that domain, which means that we have a bunch of Jira boards here, and then a few Jira boards here as well. So answering the question, when is this feature going to be released, is not that easy. Because you have to create a puzzle with all these pieces scattered in multiple places.

Challenge number three, devices teams can't fix issues properly. Why? Because they do not own the feature. In a sense, if a feature team is owning their code base with that feature, then they hand over that feature to the device team to make sure that it works on the Firestick, for example, on the Amazon Firestick, and device team says, it doesn't really work well, I tried to fix something, but there is this bug. And it means that they do not own that feature, they have to go back to the feature team. So ownership for the device team is actually quite a problem.

I want to recap the challenges before we move on. So first of all, slow feedback loop, this ping pong of bugs and features between these two teams, one is testing on devices, the other one is working on more generic feature. But then we also have work visibility, which is not super clear when a feature will be released or at least is not as straightforward as we would like to be. And then we talked about device team ownership, because this shared ownership between feature and devices is not playing really well, because it's a shared ownership, which usually means that no one's own anything.

So this setup brings us to the journey to DevOps. And I want to start the journey as every journey with a first step. And the first step is a quote from the DevOps handbook that says that the first way emphasizes the performance of the entire system. So not just focusing on the silos of different parts of silos of work or department, but focus on the entire system. And if we apply this one, which is like page one of the DevOps handbook, DevOps handbook 101, we have our old system which goes from developing a feature to the customer, which has an end over between feature teams and device teams. So we prepare something, we end it over, and this team check that everything is fine. So if that runs, if they are able to say, okay, this runs, we get to customer. But we have here, we have been here before, we have seen this, and what if I call this team here development? And what if I call this team here operation? Isn't that very similar? Isn't that exactly what was happening, let's say 10, 15 years ago when development team were all about to share their code as fast as possible without really care if that code was breaking production or if it was too expensive for that machine to run 24-7 to run so many hours of uptime? And then we had operations which had to cope with just development team shipping features that they may not even were working on some of the devices, some of the servers that they were supporting. And so we had this problem of handing over and my real question here is that okay, these are very similar, but in order to push forward this similar setup, but how did we arrive to the DevOps that we know to today? So we improved a lot in the last ten years, but how did we arrive to what we know as today's DevOps? And I think there are many contributing factors, one of which is for me are abstractions and abstractions are something very powerful.

If you think about cloud providers like AWS, Google Cloud Platform, they built some services on top of servers, on top of databases, on top of many things that developers and operation used to deal with in a very different way. But by building these abstractions, we were able to make teams more autonomous. And that's not only about cloud providers, because we have so many tools like Terraform, Puppet, Ansible, all of them are going and went into the direction of making people autonomous, making teams autonomous.

So creating these layer of abstraction on top of our bare metal servers is what, in my point of view, helped develop the DevOps culture. And so the right question to ask at this point is if these abstractions were so powerful for the DevOps principle and movement in general, which are the abstractions that we need in the front end? And finding out the right abstraction is a very complicated subject, because abstractions are very easy to be created in the wrong way. There is a nice talk from Alex Martelli, the tower of abstractions, which highlight how dangerous it could be sometimes to take the wrong turn to create abstraction.

So to answer that question, I want to enter the last part of our journey, which is DevOps for front end. So when we think about DevOps for front end, but most importantly about the right question to ask, why can't our team be autonomous? Why can't they work on Xbox and on PlayStation 5 and on your Samsung Tizen at the same time? Mostly because there are differences, and those devices are not the same, and they are far from the same. So the pain points that we have in the front end, in this specific context of the devices these days, is that each single of these devices has different APIs, a different runtime, a different browser engine, a different video player. So all of these differences make it very painful for a team focusing on a feature to be distracted by all these differences.

So what if we create some sort of platform team, or a team that takes care of these differences and tries to ease these devices' differences? The idea is to enable feature teams to develop and deploy autonomously. But so how do we do that? Because a TV is a complex, is a complex device in a sense that you have a development mode, a debugger mode, and then you have the build time and you're building an application, then you have the runtime when you're running the application using your remote or using your controller. So how did we tackle that? We did tackle that in two steps. The first steps first abstraction that we created is at build time, so focusing on tools. Develop on device it's something that has to be standardized because if you're working on the latest feature that you want to add to design in this case, but to any service or product that you're building, develop on device is an opportunity to spot all the bugs straight away. And if you can easily develop on device without caring about device differences, that's a big win. That's a huge win. But also debugging on device. What if you have a problem and you want to put a breakpoint, you want to go inside that loop? So debugging on a device is something that is not standardized. So we created an NPM run dev where you can pass some parameters and it doesn't care if that's running on a PS5 or on an Xbox because that complexity has been abstracted away. But also we worked a lot of monitoring and especially we focused on test automation. Especially in this COVID time being at home it's not easy to work on devices. So we pushed a lot on a TV lab that we have where we can run automated tests on the TVs and on the devices themselves. And all of that is completely driven by automation and by code. So you write your tests and instead of running on a normal browser, they will run on the device itself, even if it's not there with you. In your room or in your home office, they are in the TV lab, but they are running the tests for you. And that's crucial to make sure you have confidence of shipping to multiple devices. That was the first layer of abstraction. The second layer is at runtime. Once, while your app is running, what happens on the runtime? So on the runtime, the first differences are with remotes and controllers.

Imagine that a PlayStation controller has to work somehow in a similar way to your TV remote because they are controlling an app, which is very similar. And also, the developers who are building the features within the app, they do not care if you're using it from a PS4 or from a TV. So obstructing these differences goes always in the direction of making team autonomous. And if we extend that knowledge, all the devices API are obstructed away in the, in our runtime.

So teams working on features, they just call something which is standardized. They do not care about device differences. And that's true for networking API, since we deal with live videos, networking is so important for us, but also for putting your app in background when you have a notification and you open another game, maybe on your Xbox. So all of these are all solutions that we put in place to enable feature teams to be autonomous, enable feature teams to go from developing their specific feature to testing and seeing their changes on devices to our customer, with the help of the tools and the solution from the TV platform team. So feature teams being autonomous going to customers.

But the idea is that, okay, we are creating some parallel ideas. We have some parallel ideas between the devops and actually this frontend setup. Can we push it even further? Can we actually say, okay, let's take some devops principle like maximizing the flow? And let's see if that applies. And the answer is yes, it does. Because when we take the flow from end to end, we should focus on as devops principle, make the work visible. But if we have one team working on the feature, shipping value to our customers, their work is visible in one Jira board. That's as easy as it can get. You can see the work flying from one column to another one after it reaches customer. And also reduce the batch size, not having to deal with two teams and all the communication between these two teams means that one team can potentially deploy multiple times a day. It's up to them. They are in charge. They are autonomous into taking that decision. And last but not least, prevent the fact being passed downstream. That is possible because of the tools, because of the solutions that we created. And if we start from develop on-device very early, you immediately fix the bug straight away, because you see that that round thing is not that round. Maybe you want to fix them straight away. And if you are developing on-device, you will fix that in seconds, not in sprint. And once you have that, you can move to testing automation, because maybe you work and you develop on an Xbox, and you want to make sure that your PR, your pull request still works on a PlayStation 5. So why not running the test in our TV lab for checking that PlayStation 5 it's fine. That is possible because we heavily invested into these kinds of things, these kinds of tools to enable teams to be autonomous. So to recap, how did we maximize the flow.

We started by having one GR board to make them work visible, especially around features. And once you have one team that deploys often and then deploy autonomously, you can truly start by develop on a device as early as possible and add testing automation into the mix.

I want to leave you with a few ideas in order on what to do, what to take away from this talk. And I want to say as the first thing, that the developer experience is so important. It's probably even more important when you have a really crappy one, like working on devices. It's not as smooth as, for example, working on web. When you have many more tools, many more solutions already in place. So, focus on your developer experience is important. Providing the right tools, spending time for building those tools is very important.

And then, DevOps is about unifying, but more importantly, is about enabling. It's about making teams to be autonomous, making teams responsible of the end to end because the teams that are responsible is able to be on call for that specific feature, is able to support that feature with the tools and with the solution that we put in place for them to be autonomous. And then, the backend is just the beginning, because we saw how these particular principles are applying quite well, if you ask me, to the frontend as well, to the teams. And once you have all of this set up and in place, you just have to measure to improve and repeat. That was the last slide.

I want to thank you for staying here with me today. You can find me on Twitter at underscore max Gallo, and these slides are available on GitHub, at talk DevOps for Frontend. Thank you very much. I want to have you here and we want to have a look at the poll at the question you posted to our audience, and the question was, where have you applied DevOps principles so far? And looking at this, most people back end and beyond. And that's good. I would agree, right? Yeah, yeah. I was actually hoping for the second one, but I'm still happy. Is there a place you should not apply DevOps principles to in the modern world? I think if you're flexible enough, you can convince anyone to do anything with DevOps, really. Just like your skills as, like, proper person able to convince other in doing things, but it's definitely possible. I've had discussions with companies who tell me, like, oh no, we are too big for DevOps. You don't know what you're talking about, but okay. Excellent. So your talk was about applying DevOps or DevOps principles beyond your browser, right? Beyond the web application. And I understand from what you were talking about, it's about deploying the web applications for smart TVs and gaming consoles and all those smart boxes for your application, right? Yeah. We have a bunch of questions coming in from people, so I'm going to kind of slowly get into them, and I even have my own. So we have a question from Lara.

Lara asks, what tools do you use to run automated tests on consoles and TVs? Right? How many different devices do you test these things on? Yeah. So in terms of, like, off-the-shelf tools, it's, like, very little to zero. As much as we love our TVs and our Playstations and our Xboxes, every single of these things is an island. They don't really talk between each other. They just live in their own bubble with their own development processes, with their own development experience, and unfortunately there is no product that you can go buy and say, okay, I'm going to use this for testing on a real device. Amazon has done that for their device lab. Netflix has something about it. BBC here in the UK has something about this. But we're all talking about bespoke solutions. And in our case, we went for a very similar route.

So we do have a TV lab where we've got literally cameras pointing in front of TVs, where we are running automated tests on the TVs and set of boxes themselves. Sometimes is it possible to intercept the HDMI flow? Like if you have, for example, a PlayStation, you can get the feed from the HDMI. But if you're working on a TV, there's no really way to get the feed out from the TV. So you just point the camera into it. So a lot of bespoke, a lot of hardware involved into that. But if you ask me for suggesting for a tool to start using next week, just do it yourself. There is no such thing. And that's a really good point, because, you know, it's relatively easy, especially, you know, we talk about containers and like, oh, launching a thing in container, it works everywhere. But so many different TVs up there. Okay, we can say there's two or three or four consoles, but so many televisions and television models, set up boxes and whatnot. So it is, I can imagine, very difficult to have a unified testing method for all of them. But yeah, I like the fact that you have cameras pointing at things there. So very tangible. It feels like the answer that a five-year-old could give. How do I test? Yeah, it's easy. Just point the camera on the screen. But that's complete. That's okay. That works. As long as it works, it's perfectly fine. Exactly. The easiest solution you can think. Correct. Great. Excellent.

So another question coming in. As a DevOps engineer, whatever that is, I have a discussion completely about what a DevOps engineer is, we're not going to get into that, a person that does DevOps, in reality, how do you split your time between working on an application and maintaining your entire delivery pipeline? That's a good question. It depends on the day. So sometimes there are very red days when everything is red. No, I'm just kidding.

In our sense, in the transition that we are moving right now, at least within the zone, we are quite investing as a company in a lot of tooling and a lot of those kinds of things that would help us being more self-sufficient and overall efficient because on TV, otherwise you have to test anything manually, which has, as you can imagine, its own limitation. So we have that quite baked into our company goals to make sure also that that's something that we want to do.

So thankfully, I know that that is not always the case when you have your company backing you up. That sometimes like you do that in your free time or you try to sneak that into a sprint, because it's not the main point. But in our case, thankfully, we have proper time dedicated to that. So depending on the week, it could be a week which is very focused on understanding why the PS5 keeps disconnecting from the network and it's not reachable anymore. But overall, it's about improving the experience. Whatever we do, it's about improving your daily work, which is one of the most precious things that you can do.

Yeah, absolutely. You want to get your time back. Excellent. So we have a couple of new questions coming in. A question from D. How do you know whether the test has passed or failed via camera? Do you have some sort of a very visual way of defining that? That's a very good question. So we have some visual regression testing, but it's not connected to the camera just yet. For the simplest reason in the world is that camera feed goes through other IPs, and camera feed can be like the quality can degrade quite easily for whatever reason, like mice jumping on the cable of the internet or something happening in like in other side of the world. So that has been proven to be unreliable. So we don't use the camera feed exactly to get the test results. Camera feed are mostly used in what we call virtual pilot mode. So there is actually a URL that you can reach with a web-based application that allows you to control the TV manually. So that's the manual side of it. And then there is the automated side of it, which instead is all driven by code. So you write your tests as if you were running them in a normal browser experience, and you write your expectation and everything as you normally do in the browser, but cameras are mostly used for an on-demand use of these kind of TVs in the lab.

Okay. And can you view those cameras, I guess, remotely, right? You mentioned they're IP-based, so nobody has to be in the testing lab or to do those things. No, no. Exactly. But it has been challenging. It has been very useful through COVID times because as quite many people, we are working from home. And if you work for a company that has to support, like just if you pick Samsung Tizen, that's like five or six different versions.

Okay. So I'm not gonna get six TV, six 50 inches TV in my place as everyone else. So that's quite challenging. So that was quite a useful solution, especially during these times, but also has been proven difficult because anytime you have an hardware problem, you have to go into the lab. It's like a little bit naive, thinking that the lab would fix itself, that runs on its own. There is a lot of work and a lot of time that one of our teams is spending on it, trying to make it up and running for all the other teams. For me, this is fascinating because again, you bring so much tangibility into things you test, like when we test code, it's something very abstract, so that's perfectly fine.