JS on the Big Screen: Making TV Apps

♪ Hi, everyone. Thank you for joining this session. I'm very excited to be talking about JavaScript on the big screens and how you can make TV apps using JavaScript. I'm very excited to be here with you at JS Nation this year.

A little bit about me. My name is Mo. I head the mobile team in the UK of Theodore and Bam. Theodore Group is a group of about 700 digital experts, and Bam is the mobile specialist team within that. We've been in the mobile space since about 2015 and very early adopters of React Native, and we've been doing React for a very long time as well. I've been very involved in the JavaScript ecosystem for quite a while and excited to be talking and sharing some of our learnings with you today.

JavaScript is absolutely everywhere. So it started from the web, and I found this very cool picture of a book talking about Netscape JavaScript in version 1.2. I think this book is from the late 90s. Today, more than 99% of web pages use JavaScript. It's everywhere. You cannot get away with building a web app without JavaScript. And more and more JavaScript frameworks have arisen over the years and are being used out in the wild, and we're very familiar with these. On the other side, JavaScript has found its way into the mobile development world. This is a quote that I always keep on referring back to, which is from Steve Jobs in 2007 when he's announcing the iPhone 3G, where he lays out the vision basically for PWAs. And it's really interesting to see that there is always this idea that JavaScript could be used to build really good apps on mobile phones. And that vision moved forward despite native app adoption through tools like Cordova, later Ionic on top of that, and React Native, which allows you to create native applications using JavaScript. So JavaScript has become an integral part of the mobile ecosystem as well. And even backend, right? We've been seeing more and more backend runtimes of JavaScript or server-side runtimes of JavaScript. So JavaScript can effectively be used across the entire stack.

But, you know, as you're sitting at home browsing on your phone, looking through your laptop, and all of these experiences of JavaScript, there's one more experience that's sitting in the background that is also powered by JavaScript. And that is the TV. And you'd be surprised to understand how much of the TV ecosystem is being powered by JavaScript.

So let's take a quick aside, and then I want to go back into understanding where does JavaScript fit with TV. But to set some of the prerequisites and the groundwork for this talk, I just want to quickly cover what some of the core components of a TV app are. So if we look at a blank screen, we've got a... In any TV app, we've got a top-level navigation, and then you've got a grid or a set of carousels. And the key factor here is that TVs are interacted with primarily by a remote, effectively.

So if you look at the TV operating system landscape, what you'll find is that it's actually incredibly fragmented far beyond anything that you see in mobile apps. And in terms of web browsers, web browsers are typically very much in feature parity and heavily Chromium-dominated these days. Whereas in the TV space, you've got so many different operating systems, each with large market shares. So there's so many different operating systems for TVs, and you can't really just target one of them. You really need to target almost all of them in order to have a strong media presence.

Now, web is pretty self-explanatory, right? The web-based operating systems can be targeted with any JavaScript application. They run HTML-based, JavaScript-based applications. So it's very common for TV applications to use React. But people also have started to use Preact quite extensively. Similarly, Solid. And interestingly, there is actually a TV specialized framework, which I'll get into in a little bit more detail later on, called Lightning.js. And Lightning.js actually uses Solid to define the UI of your application. So there's a lot of usage and there's a lot of focus as well on optimization.

They run HTML-based, JavaScript-based applications. So it's very common for TV applications to use React. It is a popular framework at the end of the day. But people also have started to use Preact quite extensively. So I know of many TV apps that use Preact nowadays because of it being lighter weight and having less of a footprint on the performance of the app.

Similarly, Solid. And interestingly, there is actually a TV specialized framework, which I'll get into in a little bit more detail later on, called Lightning.js. And Lightning.js actually uses Solid to define the UI of your application. So there's a lot of usage and there's a lot of focus as well on optimization. So Solid and Preact come into the story as well, which is quite interesting to see. And there's been some open-sources happening in the web-based TV space.

Specifically, the BBC is actually a very large proponent of creating web-based TV applications. So several years ago, they announced the Tal open-source project, which is TV application layer. And the goal with Tal was to create a lot of utilities and components that could be used to efficiently build TV applications that are performant and can also target a wide range of different TVs. That's since been deprecated and they've now focused on smaller modularized open-source projects. Just to name a few, one is the LRED Spatial Project, which is a navigation-based library for TVs, which we'll get into sort of spatial navigation in a little bit. But they've also open-sourced some of the basic fundamentals for creating players in HTML5 and JavaScript that could run on TVs as well.

Now, if we move from the web-based crowds to the native folks, it's a little bit more complicated. So if we look at the four major native tvOSs, tvOS by Apple uses Swift. You have to build it as you would with an iOS app. Android TV and Fire TV both use Kotlin, and the reason for that is that Fire TV is actually just an abstraction on top of Android TV. So you target those with Kotlin or Java-based applications. And Roku uses BrightScript, which is a proprietary language and is quite limited, specifically for the purposes of only creating Roku-based applications. So if we take away Roku from the picture because Roku is a very closed-off environment and you can't really target it with any other technologies, when we look at tvOS, Android TV, and Fire TV, what you effectively find is that you have to know Swift and Kotlin as you would in a mobile development space. And so naturally, people have tried to tackle this with React Native. And so if you've been in the space, you'll know about React Native tvOS.

This used to be a core part of React Native itself. React Native added support for Apple TV in the early days, effectively. And so eventually, this was migrated into its own fork, which is being kept very much in sync with the main React Native core repo.

And so eventually, this was migrated into its own fork, which is being kept very much in sync with the main React Native core repo. And so React Native tvOS allows you to target TVs for both Android, tvOS, and Fire TV. So this is the open-source project that is basically powering React Native on TV. And the real cool thing about React Native on TV is that it can basically create this vision for the universal TV app. It can target tvOS. It can target Android TV and Fire TV, but also using React Native for Web, it can target Tizen, WebOS, and any of the others that fall into that bracket. And so holistically, when you look at this picture, what you can see is that, bar Roku, every other platform can be built for using JavaScript, and you can use JavaScript to target them.

So the vast majority of the TV market, you can target with JavaScript. Now, when you're building TV apps, there's many different considerations, but two of the big ones that I want to look at today is around user interactions and focus management, and secondly, around performance.

So let's start with user interactions and focus management. Now, given that the majority of people here are Web developers and mobile developers, when you're building something for Web, let's just say we've got some sort of interactable element. Naturally, on the Web, people interact with cursors. So you'll have things like hover states. This is how you show interactions on the Web, and focus states, and so on and so forth. On the mobile side, everything is based off of touch. On TV, though, it's a bit different. You've got your remote, and the interactions are done using what we call the D-pad, which is the left, right, up, down arrows. And so, effectively, when you click on a specific button, it then brings things into focus, and then you effectively will need to select that element to interact with it. So the functionality and the interactions are far more limited compared to Web and mobile. And so the way that you design your UI needs to be tailored towards that. And focus management on TV or spatial navigation on TV is not a trivial problem.

And I'll just go through two very basic examples to show this. Firstly, let's say you've got an element selected, the one that's on the left-hand side of the screen right now, and the user clicks the right button. What you do effectively is you calculate which element is to the right of it with the lowest or smallest distance. And in this case, there's three equidistant elements to the right of the existing element, but which one happens to be the right one to navigate to? This is a decision that you will need to make in your implementation, and there's no right answer for cases like this. Let's look at another example. Let's say you've got a grid. This is a far more common problem that you may face with focus management. If the user clicks the down arrow, which direction do you go to, or which element do you focus next? Do you focus the element directly below the currently focused element, or do you go all the way to the beginning of the list for the next row? And so these are some of these edge cases that are very common that need to be accounted for when you're looking at focus management. And so effectively, there's two divides within the space. Some people opt for a Cartesian-based approach, and the whole idea with this is that it's all based off of distances, so there's an implicit definition of how focus management should behave on the screen and how the elements are laid out affects which elements will be selected when you click on any of the buttons for left, right, up, or down. And then there's the declarative approach. So a Cartesian-based approach is implicit. It's based off of distances. With a declarative approach, you've got a more explicit definition for these.

So you define, based off of your code, you can define and say where things will be layouted with, and they can be completely different from how things are represented visually. That's not usually the case, and it shouldn't be the case, but you have more explicit fine-grained control over how you define elements being layouted on the screen and how the behaviors are meant to affect the navigation on the screen.

So our team at BAM has basically spent a lot of time building React-based TV applications, whether that's on React or React Native, and so we've open-sourced a spatial navigation library that aims to simplify the whole declarative approach. So you can scan the QR code to check it out on GitHub, but I just want to show you a quick example of how the declarative approach would work.

The first step would be that we would configure the remote control because every remote control is going to be different. Your web-based remote controls are going to be different to your Android TV compared to your tvOS remote. So you define a remote control subscriber which maps the arrows to a certain event that can be dispatched by the native layer, or in the case of the web-based ones, a window event listener for the keydown event, and you pair that with an unsubscriber for when the library is no longer being used and the application is being put into the background.

And you wrap the entire screen of your application in a spatial navigation route. You then wrap each element within a spatial navigation node, so you say this is an element that spatial navigation needs to be aware of. And then for the ones that you actually want the user to be able to interact with and focus and select, you define an isFocusable prop, and then you can style that depending on if that element is specifically focused. Now this is a very simple example with two elements being on the screen where they're going to be rendered top to bottom, and effectively they will be focusable with the D-pad and that effect, but there's a lot of additional props that get introduced to this and more components that you can check out on the actual GitHub repo.

So we've looked at user interaction and focus management. The next big issue when you're building for TV apps is around performance. So take a step back with me. Nowadays there's an abundance of resources when it comes to devices that you use on a day-to-day basis. If you take a M3 Max MacBook Pro, which many developers will have, the Geekbench scores are just completely insane. You've got 3,000 for a single-core score of the CPU and 21,000, close to 21,000, for the multi-core score. Just keep those numbers in mind because we're going to compare them as we go along. But the M3 Max will have several cores. I think it's 16 cores in the processor and four gigahertz per core, and the memory is also around 64 gigabytes. Take a look at mobile phones. The iPhone 15 Pro will have a Geekbench score of about 7,300, give or take, for the multi-core score. It has a six-core CPU. It has 3.78 gigahertz per CPU, per core of that CPU, as the frequency, and it has eight gigabytes of RAM. So these are the devices that you typically are building for when you're developing for web and mobile.

Now, if we look at TVs, on the other hand, there's just an astronomical drop in terms of the scores. This is a TV that's an Android-based TV. It's not a particularly low-end TV either in terms of the specs that are packed into the hardware, but it gets a single-core score of 225 and a multi-core score of 460. It has a quad-core processor that has a very low base frequency, and the memory is just about four gigabytes. And this isn't an abnormality. This is actually probably on the higher ends, and if you compare it with some phones, you'll be looking at the OnePlus 2, which was a low-end Android phone from 2015, so almost 10 years ago, and it's getting almost identical Geekbench results. So this is the scale at which you are playing with when you're developing for TV apps, and this is the level of performance optimizations that you will need to do, as if you were running it on a low-end Android device from the mid-2010s. This isn't abnormal.

Again, just to give you another example, the Fire TV Stick has very similar performance. It has a slightly improved CPU, but a smaller amount of RAM packed into it. So effectively, it is a very standard set of performance, and actually, these are higher-end devices. We have targeted in the past setup boxes, so those are the sort of boxes that come with your internet provider or your internet service providers, and those devices can be far, far worse than what we've just seen. We had to optimize for a device which had only about 200 megabytes of RAM on it, and that was a really difficult task to make sure that the memory usage was always being kept in check. Mail from our team did a talk this year about enhancing those memory performances, especially when it came to native Android TVs that were running with very, very low memory. So I would highly recommend giving that video a watch as well, and so the QR code is there. So performance becomes the number one priority, and if you look at what other people have done, Netflix actually decided to...

They do use React for their TV, but they've actually decided to not use React DOM, and instead create their own WebGL-based, canvas-based renderer, effectively, called React Gibbon. They didn't open-source it, unfortunately, but the whole idea is that they rely on sort of the hardware accelerated graphics on these smart TVs to be able to get better performance, and they don't want to rely on the DOM implementation. And this is an approach that some people like to take in the community, so we mentioned Lightning.js. This is kind of the selling point of Lightning.js for people is that it uses Canvas and is WebGL-based. Similarly, another open-source project is React Ape, which also is a WebGL canvas-based implementation for React-based applications that are targeting TV or other sort of low-end devices.

The challenge with both of these is that in Lightning.js's case, it has very little support from the community. It's got a very small community, especially compared to the React world. 50-odd stars on their most popular repository, that's the equivalent of a weekend project in the React ecosystem that is not being actively maintained or code-gest effectively. So the Lightning.js ecosystem is very small, and that is usually a big concern when you're looking at investable tech that you can rely on for the long term. React Ape, on the other hand, hasn't been maintained for about two years, so those approaches are less popular. And so most people will just opt to still use the DOM and native views when it comes to those native platforms.

And in practice, what that means is when you are using the DOM, you'll need to heavily memoize because you need to minimize the re-renders. You need to create simpler variants of pre-existing community components. So it's not going to be simply just taking some of the components that work on web and just using them on TVs. You're going to need to really make sure that everything that you use is hyper-optimized and not using more resources than you can afford to use. A lot of times, components that are open source and released out by the community are optimized for cursor and touch interactions, so web and mobile, but you don't need those for TV. And performance is the number one priority. And hence, oftentimes it's better to just build it yourself with much less CPU usage or much less performance losses from using pre-built solutions. The other thing that becomes important is to avoid heavy computations on paint, especially when it comes to layouting. And lastly, a lot of virtualization and recycling happens on TVs and TV apps to minimize the number of components that are rendering on a screen. And I'll give you an example of one.

So in the React Native space, on mobile, people often complain about FlatList. FlatList is the default list implementation within React Native, and it is notorious for being slow. People have been complaining about it for years, and Shopify actually released their own version called FlashList of a list component to be able to basically make sure that lists are smooth and performant on React Native applications. And so, naturally, that FlatList component also exists on TV, and some people try to use it for TV, and it is not a good idea. Now, FlatList is virtualized, but what it doesn't have is it doesn't have a recycling mechanism within it. So one of the important things is that we should be able to recycle as many elements as we can. And so when you have a carousel, you don't want to mount and unmount components as the user scrolls through the carousel, because that is a very computationally expensive operation to do. So what you do is you use the same components, and as the user is going through the list, you take components from the beginning of the list, and you move them effectively to the end of the list and update the data that's being displayed. So you consistently have a certain number of elements that are rendered, and you only change what's being displayed for each of them and adjust their position effectively.

And so that way, you don't have to deal with the computationally expensive calls of mounting and unmounting components and re-rendering the entire list as a result. And when we actually ran some benchmarks and compared flat list, flash list, and spatial virtualized list that we've created internally and it's part of the React TV Space Navigation Library, we found that it outperforms both of them. And so that gain is incredibly important for TV-based applications. When you look here, it performs 41% better in terms of average CPU usage compared to the flat list option that you have default with React Native. So heavy virtualization and recycling is very important when it comes to TV applications.