Modern & Secure Streaming on the Web

Hi everyone! Thank you so much for joining my talk today. Have you ever wondered how the world's biggest streaming companies stream their content? If so, you're in the right session because today we're going to learn how the modern and secure streaming works on the web. But before we jump and start exploring, I would just quickly introduce myself. So, my name is Kasia. I'm a second year working at Spotify. My team and I develop an SDK that powers Spotify player on the web. And apart from that, I like board games and watching movies and TV series.

The agenda for today is as follows. So, today we are going to talk about the smooth and secure streaming. And here, we'll cover some topics such as segmentation, media source extensions, as well as dash protocol, but also encryption and decryption with encrypted media extensions. Later, we'll put the theory into action and we'll see the demo part. And last but not least, we'll do a quick wrap up. So, we'll basically summarize what we have learned today. Just so you know, I won't talk about the Spotify internals today, but rather a standard way of streaming media on the web.

Cool, so let's start with this smooth and secure streaming. So, I guess this is like the most classic example of how to play a video content using HTML elements. As you can see, we have video HTML media element. We have a media file, which is BigBugBunnyMP4, and we just assign it to the source attribute. It's a classic example. It's very easy. We can just play the content straight away. But this approach has two issues. First, it doesn't allow us to control over playing fraction of the data. And also, it doesn't allow us to play an encrypted content. And these two are very important when it comes to the smooth and secure streaming.

When it comes to the first one, it's very important to play with fraction of data when it comes to the smooth playback. And while current HTML media elements already support that, by downloading a fraction of data through the X range request, they don't give you much control over how much data you like to download. So, for example, you can't really decide how many data you'd like to buffer in the very beginning of the playback or also throughout the playback while it continues. Moreover, if some CDNs or servers don't support X range requests at all, then in this case, the browser will try to download an entire file prior to playback. So, if you imagine if it's a very long, high-quality video, then, well, you might even wait a while to see it.

And when it comes to the second one, playing encrypted content, this is kind of self-explanatory, but in this case, we also have to use some additional logic to decrypt the content once it's encrypted. And this approach can't do that for us. So, this is what we are going to focus on today.

And before we start digging into this, I just wanted to mention two things. So, first one is that I'm going to focus on the MP4 format today because it's the most common format used in String, and also I'm going to focus on the video files today just to make things interesting, especially for the demo part. But all the commands or all the information that you will find today and you will learn today also apply to audio-only files as well.

So, let's have a look how the MP4 format looks like. So, this is the MP4 structure. MP4 consists of atoms. So, all the peculiar abbreviations that you can see here are called atoms. We have FtipAtom, which is the file type. We have MoveAtom, which is moving metadata. And also, we have MDATAtom, which is basically a payload of audio or video. And then, if we decide to fragment it, which is like a middle step towards playing and downloading only a fraction of data, then we'll notice some very huge differences in the structure.

The first one is that we'll get an init segment and then we'll get a bunch of media segments. Inside an init segment, we'll have FtipAtom and MoveAtom, but now the MoveAtom also contains the MVEX atom, which stands for Movie Extents and indicates that at least one of the streams got fragmented. When it comes to the media segment, we'll also have two atoms. One of them is Move, which is somewhat similar to the Move that you can see on the left-hand side, but it's only dedicated for that specific segment. That's why we have the F at the end. And we also have the MDAT atom, which is still a payload of audio or video, but only for that specific segment. At this point, it's also worth to mention that the current segment doesn't have to know anything about the previous segment or the next one. The only information it needs to be playable is the init segment and that's it.

So, at this point, we just fragmented the file, which is awesome, but we still can't support any of the two criteria that we just specified in the beginning.

So, I used the Pentopore tool and the mp4 fragment command, and I fragmented the file. So, on the left-hand side, you can see how it looks for BigBugBunny, and this is the fragmented version. So, first we had three atoms, and then, as you can see, I have many of them. And the very first two are the init segment, and then we have first Move, MDAT Move, MDAT Move, MDAT. So, all of these are the media segments. This is great. So, at this point, we just fragmented the file, which is awesome, but we still can't support any of the two criteria that we just specified in the beginning.

And this is because even though we fragmented the file, we fragmented its structure, it is still a one single file. So, instead, in order to continue and somehow split the file into multiple pieces, which is basically one of our goals, we have to do an additional thing.

And this additional thing is a streaming protocol. MPEG-DASH is one of the most common streaming protocols out there. DASH stands for Dynamic Adaptive Streaming over HTTP, and it's developed by DASH Industry Forum. And as I said, with that protocol, we'll be able to basically split the file into multiple pieces and then download the segments and basically continue with the playback.

So, how does it work? First, we have to fragment the file, which we kind of already did in the previous slides. And then we run the button command that you can see in the slide, which is mp4-dash. With this command, we'll get this output. So, first, we'll get two folders, audio and video, for audio and video streams, respectively. And also the stream MPT file, which is the DASH manifest. I will get back to that later. For the audio and video folders, if I enter any of these two, then you will see that instead of one file, we have a bunch of smaller files. And the very first one is the 8 MP4, whereas the rest of them are seg-prefixed-media segments, basically.

So, from one file, we literally went into a bunch of smaller files, which is great. With the command that you can see at the bottom, you can notice that we also kind of encrypted this content as well, saying that we like to encrypt it with Widevine. We also provided the encryption key. So, here is also worth to notice that the 8 MP4 here will now have additional atoms. And these atoms are PSSH and the SYNTH. PSSH stands for Protection System Specific Header, and it contains IDs to the keys that will be used to decrypt the content. Whereas the SYNTH atom that stands for Protection Schema Information is kind of very also important, because it takes a part during the decryption process, which you will also find out in the next few slides.

Awesome, so that was for the INIT MP4. And now, quickly going back to the previous slide. So, I told you before that we have the stream MPD file as well in the output, and this is the dash manifest. So, dash manifest is an XML-based file, and it's like a recipe of how to play the content and where to find it. And its structure, it looks like this. So, in the very root element, we have MPD element, which is called media position description, and it contains information such as duration or title, and so on. Then inside, we may have one or more period. And period is a time window within which we expect the content to be presented. Usually, it's only one period, but there might be many, and in this case, it's actually easier to support dynamic content insertion, such as ads, for example.

Then, within a period, we may have one or more adaptation sets, and adaptation set is like a representation of a given content. So, for example, if you think about video, for example, and we know that video may have two streams, like audio and video for sound and image, this too will be described into separate adaptation sets, just to give you an idea. And then inside an adaptation set, we may have one or more representation elements. The best use case for the representation to have many representation elements is the one we support, WPBitrate, but this is out of scope for today's presentation.

This is how the Dash Manifest looks in reality. It's an XML-based file and serves as a recipe for how to play the content and where to find it. The highlighted information like myPipe and codecs is used to inform the browser of what content to expect. The segment template and the initialization and media fields in the manifest help locate the init segment and media segments. The dash manifest also includes content protection information, like the base64 IDs to the keys used for decryption.

At this point, we have split the file into multiple pieces. The previous examples of assigning one file to the source attribute won't work anymore. Instead, we use media source extensions (MSE). We create a video HTML media element and assign the media source object, created using the create object URL method, to the source field. Then, we create a source buffer and add it to the media source, passing the my type and codex from the dash manifest. Finally, we fetch the segments and append them to the source buffer, enabling playback.

Because with the dash command, we also included the content, in the dash manifest, we'll also see some additional elements. And one of them is content protection. And as you can see, we also do have an information, like the base64 information about the IDs to the keys that will be used to decrypt the content later on during the decryption process. Amazing.

So, that was for the dash manifest. And this is great, because at this point, we split the file into multiple pieces, right? So, these two examples that we have seen before, well, they won't work anymore for us. And this is because, as you can see, we previously assigned only one file to the source attribute, but now we have many of them. So, instead, we have to use media source extensions, or MSE in short.

What does that mean? It's that we still have to create a video HTML media element. But then, instead of assigning the file to the source attribute, we can create the media source object and assign it to the source field through the create object URL method. And once we have done that, then we can actually create a source buffer and add it to the just created media source. And while doing so, we will pass the my type and codex, those that were mentioned in the dash manifest. So, this is basically the place where we tell the browser what content it can expect. And later on, once we have done that, then we can actually start fetching all the segments from the CDN, for example, and append those buffers to the just created source buffer. And basically, this is how the playback works from now on. And at this point, we can say that we have a control over playing fraction of the data, because we can decide how much we'd like to buffer in the beginning, or in the middle, or even at the end. It's basically up to us.

But we still can't play on encrypted content, because we just encrypted it, and there might be some kind of decryption process as well, right? And we haven't covered that yet. So, this is what we are going to cover in the next few slides. But before we jump into the decryption process, I just wanted to go over like a few core concepts.

The first one is the DRM, that stands for Digital Rights Management. And it's a technology used to protect the content from unauthorized use and controls its distribution. And there are three main DRMs out there that are Google WideVine, Apple Fairplay, and Microsoft Playready.

Then, we have EME, that stands for Encrypted Media Extensions. And this is the API that decrypts the content for you. So, this is basically the API that sits in between of your application and the CDN module. CDN module is a Content Encryption Module. And this is like a software, a binary that comes together with your browser installation. So, you can't really see what is inside. It's not open source, which is kind of obvious, given its like functionality.

And last but not least, we have CNC, that stands for Common Encryption. And this is the startup encryption scheme that is supported by all DRMs. All right. So, that were all the core concepts that I wanted to go through.

And let's now jump into the decryption process. So, for the decryption process, we have five different components. The red ones on the left-hand side are created and maintained by us. And these are CDN, the web application with the player, and also the license server that will serve the licenses with the keys inside. And on the right-hand side, we have green ones that are out of our control, which is the browser and the CDN module.

So, in order to play encrypted content, we first have to search it from the CDN. And then, we have time to play it. At this point, browser detects it's an encrypted content, because it has SynthAtom. You remember, like on the previous slide, SynthAtom. So, that was like why I told you it's very important. So, basically, at this point, once browser detected it's an encrypted content, it emits an encrypted event. And this is basically where the whole decryption process kicks in.

So, what we do on the player side, we call the requestMediaKeySystemAccess method. I know it's long, with two parameters. And one of them is keysystem, and another one is list of configurations. And a keysystem is a generic term for a code protection provider or the decryption mechanism. And it's, as you can notice in the slide, it's usually a reverse domain name. So, what we do here with this method, we basically ask the CDN module if the keysystem and at least one of the configurations are supported. And if they are, then we get a MediaKeySystemAccess object back. And here I left one of the functions that we are interested in, which is the createMediaKeys.

So, as the next step, we take this createMediaKeys method and we call it. As a result, we should get the media keys object back. And this object is quite interesting because it's like an exact representation of a CDN module. This is basically the only way how we can communicate with it. And as you can see, one of the methods here is the searchServerCertificate. And this is quite useful because the communication between the CDN module and the license server can be encrypted using this method.

And the second method, createSession, is kind of self-explanatory. So, this method will create a session within which we'll obtain the keys to decrypt the content. But before we call any of these methods, we first have to take the media keys object that we just got, and we have to assign it to the video HTML media element or audio. But in our case, we are working with video today. So, once we have done that, then we can actually call the createSession method and it accepts only one parameter, which is type of the session. By default, it always creates a temporary one, but you can also create a persistent, but it requires a higher level of security. So, by default, we can just go with the temporary type, which we do. And then, as a result, we should get the media key session object back. And here, I listed three of the most important methods here.

And as the next step, we'll get the generateRequest method with two parameters, anyDataType and anyData. And now, these two are coming from, alone, from the encrypted event that was emitted in the very, very beginning by the browser. So, what we do here, we get whatever was called in the very beginning, whatever was included in the encrypted event in the very beginning, and we just pass it through to the generateRequest method. And what happens later is that this request is forwarded to the CDM module. And CDM module, at this point, realizes, okay, so this is an encrypted content, and in order to decrypt it, I need a key. So, I will emit a message event, which basically means that I need a key. So, what we do on the player's side, we listen for such messages through the onMessage event handler. So, once we received such message, then we make a POST request to the license server to obtain the license. If everything goes well, we got the license back. And as I mentioned, this one is, of course, also encrypted, but the license itself that contains the keys is also obfuscated. So, if you will see event inside for some reason how it looks like, this information is only understandable by the content decryption module. So, going back to the process, once we got the license back, we are proxying it to the content decryption module through the update method. So, once the content decryption module got it, then it can take it, unwrap it, take the key, and then decrypt the frame and provide the decrypted frame back to the user. So, now they can basically enjoy watching video or listen to a song.

So, yeah, I know it was quite a complex process, but hey, we did it! So, we went through all of the parts, which is great. And now, with that said, we can safely say that we support like both the criteria that we just specified in the very beginning. So, now we can, we have a control over playing fraction of the data, but we also are able to play unencrypted content, which is great. And that being said, now I think it's time for a demo. So, let's have a quick look how it all works in action. So, here I prepared for you a web application on the left-hand side that you can notice. And you can also see that I'm getting like a lot of errors.

And this is because I don't have any content to play. And this is what we are going to do in this demo. And on the right-hand side, you can see that I have a terminal... Wait one second, just one second, one second, I'll just adjust this quickly. Yeah, so I have a terminal and this is basically the place where we are going to prepare the content that will be served in the web app. And as you can see, I already have BigBugBunny.mp4, which is the original file.

And the next step, if you remember, I have to fragment it. So, I'm taking the BigBugBunny as the input and BigBugBunny.fragmented as the output. And now we have two files. And then the next step is to run mp4-command with all the options that you have seen in the previous slide. And as the input, I'm taking the BigBugBunny.fragmented. I'm providing also the encryption key. And also, I'm telling that I would like to encrypt this content with white wine. So, I'm running this command right now.

And as the result, I got the output folder. So, if I enter this folder, then you will see that I have audio and video folders with the stream NPD that is Dash Manifest. Let's have a look at this one. And as you can see, it has like NPD element. It's an XML-based file. And we can also see that we have two elements, one color protection for the adaptation set with video over here. And also, another one for audio over here. We have audio here and another color protection. So, with this one, we can say that we really encrypted a content like both streams, audio and video. And now, if I exit it quickly and go to video.abc1, just to show you how the bunch of smaller files looks like. So, we have like really a lot of them. And the very first one is the init.mp4, as you could just see in the slides. And then we have a bunch of prefixed files as well. So, if I now refresh the website and the app, we got some console logs, which is good. So, as you can see, browser sent the encrypted event and also the message, which was like the key is needed. And from the client side, we called like many, many methods like the createMediaKey, system access and the createMediaKeys.

So, all of them that we went doing the decryption process in the slides. And now, if I try to play it, let's see. Moment of truth. Yes. It works, which is super great. So, basically, we played an encrypted content.

And just to quickly also show you, this is one of the tools, another tool, that's quite useful because it can represent all the mp4 files in the tree structure. So, if I select the output, for example, video.abc1.init.mp4, then you will see that in the move item, I have the PSSH that I told you before. And then, if I go here and here, and I believe not here, here, and std, std, and here, then we got the same atom as well. So, you can really see that the 8-mp4, which is the very first segment, contains this additional atoms that I mentioned to you before.

So, segmentation and media source extensions, these are quite useful because they help us to ensure a smooth playback. And thanks to them, we can play a fraction of data and we can also control the buffering. When it comes to the encryption and the digital rights management, encryption itself is very important because the content providers can stay compliant. For the digital rights management, this technology is useful because it protects the content from unauthorized use and it controls also its distribution. Encrypted media extensions, which is the EME in short, which is the API, allows us to decrypt the content and also play it to the user. And of course, don't forget about the content decryption module, which is basically the heart of the whole decryption process.

Cool. So, I just wanted to share with you a few resources. So, on the left-hand side, you can find some of the specifications that I used to prepare this presentation and also the demo application. I really encourage you to check them out and also like a few video links. And on the right-hand side, you can find some of the tools that I used today, primarily the Pentor 4 and Mudoteca. So, if you'd like to play a bit with MP4, feel free to do so. It's pretty cool. And that being said, thank you very much for joining. I'm very honored to be here. And in the right top corner, you can find the QR code that will take you to my repository on my GitHub account, where you'll find the demo application that I used today. So, feel free to clone it or fork it and play with it. Feel free also to reach out through LinkedIn. And if you're interested about the adaptive bitrate, you can also scan the QR codes in the left-bottom corner to see the adaptive bitrate part. And yeah, thank you very much. And if you have any questions or you'd like to chat, see you on the Discord. Thank you. Bye.