A 4-Year Retrospective : Lessons Learned From Building a Video Player From Scratch With React Native

Hello, React Summit. It's so nice to be here, and I can't wait to start this talk. So, welcome to this talk, which will be a retrospective on four years in the making of our React Native video player from scratch.

So, without further ado, let's start. What is a retrospective? So, I would like to structure this talk as a retrospective. A retrospective is the chance of a team to reflect on the past within a structured meeting. So, hopefully, I will save you some time because we will take a look at what we did well in the building of this player, what did go wrong, and hopefully I've saved you some time if you want to build a player from scratch. So, the goal is to save you some time and to highlight the areas that we've done and facilitate your experience in building a player like that.

So, we will start with a check-in, then we will see an overview of the project. We will move on and see what did go well, and after that, we will see the most interesting part, which is what didn't go well. Then we will move on to the lessons learned, and we have a finally overview, which are the action items. So, a good retrospective ends with concrete action items.

So, let's start with a check-in. Who am I? I'm a technical lead at Learn, which is an Italian company which aims to make digital competencies accessible to everyone. And one of my greatest specialties is building products apart from software engineering stuff. But I love products and product development and product management. So, everything that's around products.

So, let's start with a project overview. The project is this one, which is Learn. Right now, Learn is a complex application which has both a mobile platform and a web application. The focus will be on the mobile side of Learn, so the mobile application made with React Native. We started in 2020 with an application made from scratch with React Native without particular dependencies. And this is the first version of the player. As you can see, it's pretty simple, and you didn't have even the scroll bar on this kind of player.

So, we moved on and we developed this. So, in the future, we have a lot of stuff that you can see. You can take notes, you can give and see the subtitles of the lesson and the transcript. So, it's really complex UI, but it has a lot of components. There are a lot of components on this application. It's made entirely with React Native Video.

It was made with React Native Video 5.2.1, and then we did a lot of custom components like the one that you have seen, and we used React Native bottom sheet to animate all the stuff that's happening with notes and transcripts and stuff like that. So, the goal is to make you understand what we can improve in the making of this player and what can be done to optimize this kind of player and how we achieve the optimizations.

So, let's start with what went well. Well, the streaming the HLS went very well because we developed the infrastructure, which is made in AWS, but I have a separate talk for that and you can check it out on my website. Then the subtitles and transcripts, as you can see, went pretty well, and I explain that later. And then the performances were pretty, pretty good.

So, about the HLS streaming, like I said before, you can see here it's fully compatible with every kind of streaming playlist that you want. So, it went pretty straightforward, apart from the infrastructure part. Then on the subtitle part, well, at first, there was no compatibility with subtitles because there was no styling and you cannot even set the position. So, you ended up with subtitles that were positioned on the top of the video, but we wanted full control on that part. So, what we did was to translate and create a utility to convert the VTT file format to objects in an easy way. So, we did create this function that parses the VTTs and makes a particular object, which is very, very understandable and reusable, and we built a custom hook to handle the subtitles.

In other words, you use this kind of hook. It's pretty simple. You pass the player, so it's synchronized with the player, and you can expose the subtitles and use it in your UI like we did here. So, for instance, if a person is speaking a word in minute three, at minute three, the word is printed on the screen, and the final result is like this. You can have fully control on the position, because the UI is handled by you.

Then what about the iOS performance? Well, the iOS performance was pretty good, like I said, because we had 99% of sessions without crashes. So, we were pretty satisfied, but even about the performance, we had a lot of... we didn't have any particular issue with the streaming, with the videos, the smoothness, and with the animation, because it was pretty, pretty smooth and all was going well. But let's move to the most interesting part, in my opinion, which is what didn't go well.

So, the Android player itself was a problem, and I'll explain that later. We had problems with the background audio management, and we had pretty much problems with a lot of differences between iOS and Android in terms of behaviors. So, let's see one by one the points. So, with the audio player... About the audio player, you have to know that Android uses... you can use on Android natively two kinds of players. Media player, which is an older version, which has a little controls, and it's not so customizable, and it's not so stable. Then you have exo player, which is a more robust player, which is newer. And as compared to the media player, it has support for multiple streaming protocols and playlists, and it's even better.

Media player, which is an older version, which has a little controls, and it's not so customizable, and it's not so stable. Then you have exo player, which is a more robust player, which is newer. And as compared to the media player, it has support for multiple streaming protocols and playlists, and it's even better.

But the first version that we used with React Native Video 5.2, we were using a background media player, but we didn't know that. So, the first version had distorted or missing audio. It had, in some cases, a response in the player, and it had even buffering problems. So, on the right, you can see a lot of user complaints and feedbacks that we had at the time. But we had a solution. The solution was two lines of code. Yeah, right? Two lines of code. Easy, no? Okay, the problem was that it took months to discover that React Native Video was using in the background media player and how to force it, because it was not stated at first in documentation, and we didn't know that the root cause was the type of the player. So, we kept debugging and debugging and debating about what could be the problem. And the problem was that we were using the wrong player. So, we tested with the users, with the forced solution, and all went well, because the player performance improved. And we did a PR on the repo. And so, hopefully, in order to save some time to other users of this kind of model.

But let's move on to the background audio. So, background audio means when you close the lock the phone screen and you start playing the video, the video should work continuously like you do with Spotify. For instance, you close the phone and the audio keeps going. Well, we wanted to implement this feature. The first solution was to use React Native Video, because it is supported by, it was supported by React Native Video. But it had no controls in the notification lock screen. So, wanted to have control on that part. So, the solution was to use React Native Video with background audio disabled mode. So, without controls and without background audio. And we use another player, which was React Native Track Player, which is another model that allows you to play MP3 files. So, we had to sync two players. The difficult part in this solution was to maintain the sync between two players. Because, as you can see here, we switch from a player to another based on the status of the application. So, if the application moved from background to foreground, you switch to the video and vice versa.

But this was not so straightforward. So, we opted for another solution. Solution 2. The solution 2 was to use React Native Video with background audio enabled, because it had no controls. But we use another control, which was React Native Music Control, that is not actually a player, but it is a model that allows you to display controls on the notification center. So, it's not a player. So, we synced React Native Video with music controls and we were able to come up with this solution that worked for the majority of the users, because it allows you to bind the module to the controls. So, when you click the notification control, you can call, as you can see on the right, the player. So, you can start the player. And it has a lot of listeners, even the ones that are used to when you remove the headphones or the earphones that stop the video. And so, it was pretty customizable and went pretty well.

But we had even some cons. Because so, sorry. We had a lot of pros, because there was no time between no lag between the switch of the players. Like in the first solution, the screen is not re-rendered when you switch the video. So, it was pretty good. But as I said before, the Native Event System Management. So, it was pretty good. But we had some cons, which were the data consumptions. Because when you lock the screen, you're still playing the video. And you're not playing an mp3 file. So, it's heavier on the data on the user. And the downside of this is that the model nowadays, at the moment of speaking, is no longer maintained. So, if you have to deal with some problem, you have to, one, develop it on your own and even interact with the native part. Because nowadays, it had some problems with a lot of devices on Android, particularly after Android 13 and crashes. So, we ended up with a lot of crashes in production. And so, we had some problems about that. And I will show you later the statistics on the crashes. So, we had even a lot of differences between the behavior from iOS and Android.

And trust me, you don't want to have in your codebase stuff like that.

In which you control a lot of times if the platform is Android or the platform is iOS, it's not so good because there are plenty of differences. Like you see here, we spent like seven days to figure out why we should change and stop the video before or after on one platform or another. So, it's not easy to debug this kind of stuff. And even here, we switch the player, but it is a workaround to display the video when you come back from the background and display the image. Because on iOS, at first, it didn't work. But yeah, we had pretty much troubles and a lot of differences between the two operating systems.

And after that, even here, you can see iOS, the crash rate... Sorry. The sessions without crashes were 99%. On Android, it was 89%. So, there's a big difference. And we had problems, like I said, on Android and complaints on the stores for this kind of problems. Because apart from the stability, we had crashes. And it's not so good to have an application that has this high crash rate in production. But yeah, we learned a lot.

In fact, let's move on to the lessons learned in this process. So, managing dependencies and versions is hard. We started from version 0.6 of React Native. Nowadays, we are at 0.75, 0.76. And so, a lot of times, we had to slow down the work because we had to upgrade massively a lot of dependencies. And yeah, I move on to the next point, which is, using many modules is quick in the short term, but can slow things down in the long term. Because we started with a lot of dependencies, a lot of modules. But in the short term, yeah, it's easy. You have a lot of dependencies that you can use and you don't have to develop some stuff. For example, animations, bottom sheets and stuff like that. But in the long term, you have to keep updating all these dependencies, which is very, very tricky in the long run.

And the next point, the lessons learned is, removing a feature is worse than not developing it at all. What do I mean by that? We started wanting to develop a lot of features for the users. For instance, the background audio. So, we developed it and we released it.

Okay, for the majority of the users, this worked. But on Android, this had some problems. So, what happens if we remove this feature on Android? There will be a lot of complaints because the users, which have devices that can support this feature, are using it and they can complain. So, I will suggest to you to start developing not a lot of complex set of features, but start with an MVP and small features that work very, very well. And then, learn once, run anywhere. Well, not always. Because we had a lot of differences in behaviors in React Native, especially when you talk about the native side on the platform on the video side. So, we had to figure out a lot of compromises and a lot of different behaviors. But fortunately, we came up with a solution and I will show you later which are the solutions nowadays. Because this talk is based on a retrospective on version 5.2.1. But we will see what's going on nowadays.

Okay, so which are the action items? In the first part, we've seen a lot of problems with the dependencies and crashes on Android. Our goal was to improve stability on the Android platform. The action items we identified were to update React Native video. However, it wasn't as easy as upgrading the dependency because the stable version came out two years after our development started. We didn't want to introduce alpha releases or unstable versions into our application. We also didn't have the capacity to develop all the native features ourselves, so we had to wait and make compromises. Version 6 of React Native video solved many of our problems, including notification controls. We even partnered with the maintainers of the module to address issues with Audio Background and reduce the crash rate to 1%.

Nowadays, with the latest modification in version 6.4, the module is stable, well-supported, and includes out-of-the-box notification features. It runs smoothly on both Android and iOS. If you're starting a project from scratch, I highly recommend using this module in production. Unfortunately, we started developing our application in 2020, so we had to face some challenges. However, with the current stability and performance improvements, the module is production ready.

So, we had to wait and make compromises because we use other models to patch the problems with the notification controls and all the stuff. In version 6, all the stuff is patched. So, the notification controls are out of the box. It works very, very well. And it's stable. And something changed in the last year because we did even partner with the maintainers of the model. Because the Vizsla group is the new group of maintainers. It's a new company that took over the project a year and a half ago. And you can see here they started shipping and shipping and shipping a lot of new changes and performance improvements. So, we did even partner with them to solve the majority of our problems with Audio Background because they patched our module and they helped us in succeeding with the crashes in production.

So, nowadays, the crash rate is 1% like it is on iOS. So, the module is pretty I will consider it's production ready because nowadays with the new modification, the version 6.4, it's pretty stable. It has a lot of support and it has even out of the box the notification part. It runs smooth on Android, on iOS. So, I would suggest you to use this module in production if you want to build the project from scratch. We were unlucky because it was 2020. We started in 2020 to develop this application. And it was not so mature, the module. But nowadays, it is. And I would suggest you to use it in production if you want to build the product because you don't even have even to develop the subtitles from scratch, for instance, because they are supported and the styling is supported. And they will add even DLL support. So, even you can do content protection directly on the device. So, it's pretty easy to implement right now this kind of infrastructure. And it's even in the future we'll be releasing version 7, which will introduce a lot of changes, such as the implementation of Nitro modules or the separation of the player from the controls. And see, yeah, this is beta, but they are going to work on that.

This was the retrospective. I hope that I've saved you some time. And I hope that in the future, if you would like to start developing a video application or if you want to add the video functionality to your application, you could use React Native Video because right now, it's pretty stable. And if there are problems, there's a great community around it because I love open source, I love contributing to the communities, and I love taking part in this kind of project. So, hopefully, I saved you some time and you learned a lot about this side of React Native Video development. So, thanks for having me. It was a pleasure to be here. You can reach out to me on the links here on the social. You can check out my website. And if you are quick, curious and you want to ask me some questions, don't hesitate to do it. Reach me out, and I will answer you every time. So, thanks again, and see you.