Getting Weird with Video Manipulation and HTML5 Canvas

Rate this content
Bookmark

In this lightning talk we will be pushing the boundaries of HTMl5 Canvas browser APIs. Join us while we do some experiments with video in the browser to see what’s truly possible. DISCLAIMER: No promises of cross-browser compatibility. Not for the faint of heart. Must be this tall to ride. Attending this lightning talk may result in serious injury or death. All participants must bring a life jacket.

This talk has been presented at React Summit 2020, check out the latest edition of this React Conference.

FAQ

Mux is a platform that provides video infrastructure for developers, similar to how Stripe offers payment solutions and Twilio offers communication solutions. It focuses on providing easy-to-use APIs and tools for developers to work with video content efficiently.

At the React Summit, capabilities such as streaming video content using HLS for adaptive bitrate streaming, manipulating video frames with Canvas for effects like grayscale, and performing real-time object detection using machine learning were discussed.

By using the Canvas API to manipulate video frames, developers can create interactive experiences such as adding text overlays, applying filters, and conducting real-time object detection directly in the browser, enhancing user engagement and interactivity.

HLS (HTTP Live Streaming) is used to stream video content efficiently by dividing the video into smaller segments that can be downloaded at different quality levels based on the user's bandwidth, ensuring smoother playback and better user experience.

Developers can manipulate video frames by drawing them onto a Canvas element, extracting image data, and then applying transformations such as color manipulation or object detection before redrawing the modified frames back onto the Canvas.

Rendering video frames to Canvas elements allows for synchronization of multiple video sources, reduces bandwidth by avoiding multiple streams, and provides flexibility for adding real-time interactive elements and effects directly within the browser environment.

On Mux's marketing website, video frames are rendered to Canvas elements to create interactive demos where elements such as devices and browsers pop out as the user hovers over them. This method ensures synchronized playback and enhances the interactive experience without additional bandwidth costs.

Dylan Jhaveri
Dylan Jhaveri
16 min
17 Jun, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

The talk covers the use of HTML5 video APIs and Canvas for creating interactive video experiences. It demonstrates how to manipulate video frames with Canvas to add effects like grayscale and text overlays. The video also explores real-time object detection in live streams using TensorFlow and HLS technology for adaptive bitrate streaming. Practical applications include enhancing user interactivity on web platforms by rendering video frames to Canvas elements, which avoids double streaming and sync issues. The talk shows how these technologies are applied on Mux's marketing website, making it more engaging with interactive elements. The speaker emphasizes the potential of these tools for real-time video manipulation and interactive video experiences directly in the browser.

1. Introduction to Canvas and HTML5 Video APIs

Short description:

Hello, everyone, at React Summit. Today, we're going to talk about the Canvas and HTML5 video APIs and some cool stuff you can do with them. I'm Dylan Javary from Mux, where we provide Video for Developers. We focus on creating easy-to-use APIs for video. If you're interested, let's chat.

This is a test. Hello, everyone, at React Summit. I'm very excited to be talking to you here today. We're going to be talking about the Canvas and HTML5 video APIs, and some cool stuff that we found that you can do with them.

So quick intro, I'm Dylan Javary, I work at Mux. If you have not heard of Mux, Mux is Video for Developers. Maybe you know of Stripe, Stripe is payments for developers, or you know Twilio, which is phone calls and text messages for developers. We like to be like those companies, where we're built first with developers in mind and try to make great easy-to-use APIs, but we do this for video.

I'm not going to be talking too much more about Mux today, but if you are interested, come talk to me. I'd love to chat with you.

2. Introduction to React App and Player Component

Short description:

I'd love to chat with you. Let's start with a simple demo of a React app using the player component and canvas. The player component is a video element that uses HLS technology for video streaming.

I'd love to chat with you. Cool, so now to jump into some code. So I have this code sandbox set up. Code sandbox is a great tool, by the way. It's become one of my favorite pieces of software. I think there's some code sandbox folks here at this conference, so shout-out to you all. I love this product. And I'll be sharing this after so you can fork it, play with the code, do things yourself.

And let's just start out with a really simple demo. So this is a very straightforward React app. We have a few different routes. These five different examples I'm going to show and we're using React Router, React DOM. And let's start with the first one. Start with a simple demo. So right here we have simple.js. This is the component that we're rendering. We have this player component and then we have this canvas. And right now, you can't see the canvas on the page, but that's what we will be. We'll be kind of manipulating that and doing some fun stuff as we go along.

So real quickly, let's just take a look at this player component. So this player component is... It's really just a video element. But if you're familiar with video... How many of y'all have done video on the internet? So video streaming, video on demand or live streaming, anything like that. You might have used the video element before and maybe you've done an MP4 file and that can kind of work. But when you really want to do video streaming properly, what you need to do is use something like HLS. So HLS is a technology that allows you to basically download videos in segments and at different bit rates and different quality levels according to the user's bandwidth. So that's kind of something muts does for you. We're not going to get too deep into that. But that's what we're using here on this video player.