Video encoding has always been a complex process, especially when trying to achieve high-quality outputs directly in web applications. The evolution of web technologies now provides more powerful tools for developers to push the boundaries of what's possible in the browser. One such tool is WebCodecs, a set of APIs designed to handle video and audio processing efficiently. This article delves into the practical applications and benefits of using WebCodecs for video encoding, highlighting its advantages over traditional methods.
Rethinking Video Editing in the Browser
Traditional video editing tools are often cumbersome, requiring downloads, installations, and a significant learning curve. The vision was to create a lightweight, browser-based video editing tool capable of producing high-quality videos swiftly. Initially, the journey was fraught with challenges, especially around the video export process. The complexities of video encoding, muxing, and codec compatibility presented significant hurdles.
Video editing on the web starts with rendering frames using different techniques such as CSS animations, JavaScript DOM manipulation, SVG, or HTML Canvas. Among these, HTML Canvas stands out, offering a balance of flexibility and power, making it ideal for drawing video frames efficiently.
The Challenges of Video Capture with Capture Stream
Capturing rendered frames to create a video file might seem straightforward with HTML5 Canvas and the Capture Stream API. However, the method's limitations soon become apparent. The Capture Stream API is not well-suited for precise, high-quality video exports. It lacks a constant frame rate, leading to inconsistencies across different devices. Additionally, it only outputs WebM format, which may not be compatible with other editing software.
These limitations necessitated exploring alternative methods of capturing and encoding video frames manually, seeking solutions that promise better quality and reliability.
Manual Encoding with JavaScript Libraries
Exploring JavaScript libraries for video encoding revealed a common pattern of pros and cons. Libraries like WAMI.js offer high-quality output by manually capturing each frame as an image and encoding them into a video file. This method ensures a constant frame rate and high video quality.
However, the process is notably slow and compute-intensive, lacking hardware acceleration. Moreover, these libraries often restrict output to specific formats and lack built-in audio support. These drawbacks highlighted the need for a more robust solution.
Introducing WebCodecs: A New Standard
WebCodecs emerged as a powerful new standard that addresses many limitations of previous methods. It provides low-level access to video frames, enabling efficient encoding and decoding in various formats. With asynchronous processing and hardware acceleration, WebCodecs offer high performance and flexibility.
WebCodecs support both video and audio processing, making them suitable for applications like video editors, where both quality and speed are crucial. The API's ability to handle individual frames as video frame objects allows for greater control over the encoding process.
Leveraging WebCodecs for Efficient Video Encoding
The video encoding process with WebCodecs involves defining an input source, converting frames to video frame objects, and passing them to a video encoder. The encoder then produces encoded video chunks that can be stored or transmitted.
Configuring the video encoder involves setting options like codec type, hardware acceleration preferences, frame rate, and latency mode. Selecting the right codec is crucial, as it determines compatibility with different video containers and affects the final output quality.
Understanding Codecs and Containers
One common misunderstanding in video processing is the distinction between codecs and containers. A codec is an algorithm used to compress and encode video frames, while a container packages video, audio, and subtitles into a single file format like MP4 or WebM.
Understanding this distinction is essential for configuring WebCodecs correctly. The chosen codec must be compatible with the desired container format, and browser support for codecs varies, adding another layer of complexity.
Muxing with FFmpeg and WebAssembly
After encoding video frames, the next step is muxing, which combines the video stream into a container format. WebCodecs handle the encoding, but muxing requires additional tools like FFmpeg, a versatile multimedia processing library that can be used in the browser with WebAssembly.
Using FFmpeg.wasm, developers can efficiently mux video streams, adding audio tracks if needed, and outputting the final video file in the desired format. This approach leverages the power of established multimedia processing techniques within modern web environments.
Final Thoughts on WebCodecs and Browser-Based Video Editing
Embracing WebCodecs and tools like FFmpeg for video editing in the browser presents numerous benefits, including high-quality output, constant frame rates, and support for various video formats. The asynchronous nature and hardware acceleration further enhance performance, making it a compelling choice for developers seeking to build sophisticated web applications.
While the process of setting up and using these technologies can be challenging, the potential rewards in terms of efficiency and capability are significant. By understanding and applying these tools, developers can create innovative solutions that redefine what is possible in browser-based video editing.
High quality video encoding in browsers have traditionally been slow, low-quality and did not allow much customisation. This is because browsers never had a native way to encode videos leveraging hardware acceleration. In this talk, I’ll be going over the secrets of creating high-quality videos in-browsers efficiently with the power of WebCodecs and WebAssembly. From video containers to muxing, audio and beyond, this talk will give you everything you need to render your videos in browsers today!
This talk has been presented at JSNation 2023, check out the latest edition of this JavaScript Conference.
Some methods to render video frames on the web include CSS animations, modifying the DOM directly using JavaScript, using SVG, and using the HTML Canvas API.
Akash Hamirwasia is a software engineer at Razorpay who loves building web products and contributing to open-source projects. He is known for projects like Blaze, Untabbed, Dyod, and Slanted.
The limitations of the Capture Stream API for video encoding include no constant frame rate, no option to change the video file format (always produces WebM format), low reliability, and output quality that can vary depending on device performance.
The HTML Canvas API is a feature in HTML5 that allows for drawing and manipulating graphics on the web. Akash recommends it because it offers flexibility and various capabilities that other rendering methods do not.
The main topic of Akash Hamirwasia's talk at JS Nation 2023 is pushing the limits of video encoding with web codecs.
WebCodecs is a set of APIs that provide low-level access to individual frames of a video stream, allowing for asynchronous and hardware-accelerated video encoding and decoding. It supports both video and audio and is crucial for high-quality, efficient video encoding.
FFmpeg is used as a muxer to combine the encoded video stream into a final video container format. It allows for the inclusion of audio and the creation of a complete video file ready for download.
Slantit is a web-based tool developed by Akash Hamirwasia that allows users to create catchy product videos quickly. It uses the same video encoding process discussed in Akash's talk.
The Capture Stream API is part of the MediaStream Recording API standard and allows for video capture from HTML5 canvas elements.
The purpose of the video editing tool Akash was building was to create catchy product videos quickly on the browser, avoiding the need to download large desktop-based video editing software.
This Talk explores the challenges and solutions in video encoding with web codecs. It discusses drawing and recording video on the web, capturing and encoding video frames, and introduces the WebCodecs API. The Talk also covers configuring the video encoder, understanding codecs and containers, and the video encoding process with muxing using ffmpeg. The speaker shares their experience in building a video editing tool on the browser and showcases Slantit, a tool for making product videos.
Welcome to my talk on pushing the limits of video encoding with web codecs. My name is Akash Hameedwasia. I write code at Razorpay. I also love building products on the web and contributing to open-source. You might know me from some of my projects mentioned here, Blaze, Untabbed, Dyod, or Slanted.
2. Building a Video Editing Tool on the Browser
In this talk, I will share the learnings I had while building a video editing tool on the browser. I encountered challenges with video encoding, audio, muxing, and codecs, but eventually figured everything out.
3. Drawing and Recording Video on the Web
Let's start by discussing how to draw a video on the web. There are different ways to render video frames, such as CSS animations, JavaScript DOM manipulation, SVG, and HTML Canvas. HTML Canvas provides flexibility and enables various functionalities. To record the frames and save them as a video, we can use the capture stream API in HTML5 canvas. This API allows us to create a media recorder object and attach event listeners to store and process the recorded data. However, this method has its limitations, including the lack of a constant frame rate.
4. Capturing and Encoding Video Frames
The capture stream function allows capturing video frames, but it lacks constant frame rate and the option to change the output video format. The reliability and quality of the output depend on the performance of the device. To overcome these limitations, we can capture and encode individual frames of the canvas ourselves. By using pseudocode, we can seek the canvas to each frame, capture it as an image, and store the frames in an array. Then, we can encode the array of images into a video file using libraries like WAMI.js, a real-time JavaScript WebM encoder.
5. Discovering an Interesting New Standard
You can define the frame rate of your video and download it. JavaScript libraries for encoding videos have pros and cons. The pros include high quality output and constant frame rate. The cons include slow and synchronous process, lack of hardware acceleration, limited video formats, and no audio support. However, I discovered an interesting new standard that solves these issues.
6. Introduction to WebCodecs
WebCodecs are a set of APIs that give you low-level access to the individual frames of a video stream. It allows you to do encoding and decoding process on individual frames in various formats. WebCodecs is asynchronous and hardware accelerated, providing high performance. It supports video and audio encoding and decoding, with a focus on video encoding in this talk.
7. Video Encoding Process and Using WebCodecs
The WebCodecs API provides low-level access to individual frames, making it useful for building video or audio editors. The video encoding process involves defining the input source, converting it to a video frame object, and passing it to a video encoder to encode multiple frames into a single encoded video chunk. We'll focus on the storage aspect of the encoded video chunk. To encode videos using WebCodecs, we create a video encoder and use callbacks to handle the encoded video chunks and errors.
8. Configuring the Video Encoder
For error handling, a console log is used. The video encoder needs to be configured with various options such as codec, hardware acceleration, frame rate, and latency mode. The codec is the most important option. Finding the right codec is not as easy as WebM MP4 AVI. The video codec is not the same as the video file format.
9. Understanding Codecs and Containers
Codec is an algorithm to convert and compress individual frames of the video. The video stream is further processed and passed to a muxer to create a video container. Videos contain audio, subtitles, and visual data, which are combined by the muxer into a single container. Codecs and containers must be compatible. Browsers support different codecs, so compatibility can vary. For more information on codec compatibility, check the provided links.
10. Video Encoding Process and Muxing with ffmpeg
I've set the codec as AVC1. We pass the canvas to video frame to create a video frame. We close the video frame. Once all the video frames are rendered, we flush the frame data from the encoder, close the encoder, and concatenate the chunk data to a single array buffer. WebCodecs only handles the first three steps of the video encoding process. Muxing a video on the Web is not well-documented, but we can use ffmpeg as a muxer to mux the final video stream to the video container.
11. Encoding a Video in the Browser
There are various compilations available, but I found this one to be the best. The next step is actually reading the output after the muxing process is done and deleting the files because we no longer need them. And that is how you encode a video in the browser. The list of pros that you get with this approach is high quality output, constant frame rate, fast process, hardware acceleration, support for any video format, and the ability to add audio. The only con is that it's relatively hard, but I hope this talk made it easy for you. WebCodecs is supported in all browsers except Firefox, and I hope Firefox gets support soon. WebAssembly is supported in all browsers.
12. Building Slantit: A Tool for Making Product Videos
We can freely use FFmpeg and Wasm libraries to do crazy things on the Web. I built a tool called Slantit for making catchy product videos quickly in the browser. The video in the background is an example of a Slantit-made video. You can try Slantit at slantit.app. It uses the same video encoding process we discussed. Thank you for your attention and the opportunity to talk about web codecs. Connect with me on Twitter, GitHub, and my personal website.
This talk discusses the usage of Microfrontends in Remix and introduces the Tiny Frontend library. Kazoo, a used car buying platform, follows a domain-driven design approach and encountered issues with granular slicing. Tiny Frontend aims to solve the slicing problem and promotes type safety and compatibility of shared dependencies. The speaker demonstrates how Tiny Frontend works with server-side rendering and how Remix can consume and update components without redeploying the app. The talk also explores the usage of micro frontends and the future support for Webpack Module Federation in Remix.
In this Talk, the speaker demonstrates how to use Rust with WebAssembly in a Vue.js project. They explain that WebAssembly is a binary format that allows for high-performance code and less memory usage in the browser. The speaker shows how to build a Rust example using the WasmPack tool and integrate it into a Vue template. They also demonstrate how to call Rust code from a Vue component and deploy the resulting package to npm for easy sharing and consumption.
RemixConf EU discussed full stack components and their benefits, such as marrying the backend and UI in the same file. The talk demonstrated the implementation of a combo box with search functionality using Remix and the Downshift library. It also highlighted the ease of creating resource routes in Remix and the importance of code organization and maintainability in full stack components. The speaker expressed gratitude towards the audience and discussed the future of Remix, including its acquisition by Shopify and the potential for collaboration with Hydrogen.
Debugging JavaScript is a crucial skill that is often overlooked in the industry. It is important to understand the problem, reproduce the issue, and identify the root cause. Having a variety of debugging tools and techniques, such as console methods and graphical debuggers, is beneficial. Replay is a time-traveling debugger for JavaScript that allows users to record and inspect bugs. It works with Redux, plain React, and even minified code with the help of source maps.
WebAssembly enables optimizing JavaScript performance for different environments by deploying the JavaScript engine as a portable WebAssembly module. By making JavaScript on WebAssembly fast, instances can be created for each request, reducing latency and security risks. Initialization and runtime phases can be improved with tools like Wiser and snapshotting, resulting in faster startup times. Optimizing JavaScript performance in WebAssembly can be achieved through techniques like ahead-of-time compilation and inline caching. WebAssembly usage is growing outside the web, offering benefits like isolation and portability. Build sizes and snapshotting in WebAssembly depend on the application, and more information can be found on the Mozilla Hacks website and Bike Reliance site.
In the last 10 years, Webpack has shaped the way we develop web applications by introducing code splitting, co-locating style sheets and assets with JavaScript modules, and enabling bundling for server-side processing. Webpack's flexibility and large plugin system have also contributed to innovation in the ecosystem. The initial configuration for Webpack can be overwhelming, but it is necessary due to the complexity of modern web applications. In larger scale applications, there are performance problems in Webpack due to issues with garbage collection, leveraging multiple CPUs, and architectural limitations. Fixing problems in Webpack has trade-offs, but a rewrite could optimize architecture and fix performance issues.
During this workshop, participants will review the essential JavaScript patterns that every developer should know. Through hands-on exercises, real-world examples, and interactive discussions, attendees will deepen their understanding of best practices for organizing code, solving common challenges, and designing scalable architectures. By the end of the workshop, participants will gain newfound confidence in their ability to write high-quality JavaScript code that stands the test of time. Points Covered: 1. Introduction to JavaScript Patterns2. Foundational Patterns3. Object Creation Patterns4. Behavioral Patterns5. Architectural Patterns6. Hands-On Exercises and Case Studies How It Will Help Developers: - Gain a deep understanding of JavaScript patterns and their applications in real-world scenarios- Learn best practices for organizing code, solving common challenges, and designing scalable architectures- Enhance problem-solving skills and code readability- Improve collaboration and communication within development teams- Accelerate career growth and opportunities for advancement in the software industry
Dive into the world of AI with our interactive workshop designed specifically for web developers. "Hands-On AI: Integrating LangChain with JavaScript for Web Developers" offers a unique opportunity to bridge the gap between AI and web development. Despite the prominence of Python in AI development, the vast potential of JavaScript remains largely untapped. This workshop aims to change that.Throughout this hands-on session, participants will learn how to leverage LangChain—a tool designed to make large language models more accessible and useful—to build dynamic AI agents directly within JavaScript environments. This approach opens up new possibilities for enhancing web applications with intelligent features, from automated customer support to content generation and beyond.We'll start with the basics of LangChain and AI models, ensuring a solid foundation even for those new to AI. From there, we'll dive into practical exercises that demonstrate how to integrate these technologies into real-world JavaScript projects. Participants will work through examples, facing and overcoming the challenges of making AI work seamlessly on the web.This workshop is more than just a learning experience; it's a chance to be at the forefront of an emerging field. By the end, attendees will not only have gained valuable skills but also created AI-enhanced features they can take back to their projects or workplaces.Whether you're a seasoned web developer curious about AI or looking to expand your skillset into new and exciting areas, "Hands-On AI: Integrating LangChain with JavaScript for Web Developers" is your gateway to the future of web development. Join us to unlock the potential of AI in your web projects, making them smarter, more interactive, and more engaging for users.
Using CodeMirror to Build a JavaScript Editor with Linting and AutoComplete
Top Content
WorkshopFree
2 authors
Using a library might seem easy at first glance, but how do you choose the right library? How do you upgrade an existing one? And how do you wade through the documentation to find what you want? In this workshop, we’ll discuss all these finer points while going through a general example of building a code editor using CodeMirror in React. All while sharing some of the nuances our team learned about using this library and some problems we encountered.
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner. We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data. Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
Get ready to supercharge your web development skills with React Server Components! In this immersive, 3-hour workshop, we'll unlock the full potential of this revolutionary technology and explore how it's transforming the way developers build lightning-fast, efficient web applications. Join us as we delve into the exciting world of React Server Components, which seamlessly blend server-side rendering with client-side interactivity for unparalleled performance and user experience. You'll gain hands-on experience through practical exercises, real-world examples, and expert guidance on how to harness the power of Server Components in your own projects. Throughout the workshop, we'll cover essential topics, including:- Understanding the differences between Server and Client Components- Implementing Server Components to optimize data fetching and reduce JavaScript bundle size- Integrating Server and Client Components for a seamless user experience- Strategies for effectively passing data between components and managing state- Tips and best practices for maximizing the performance benefits of React Server Components
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool. We will enhance a full-stack JS application (Node.JS backend + React frontend) to authenticate users with OAuth (social login) and One Time Passwords (email), including:- User authentication - Managing user interactions, returning session / refresh JWTs- Session management and validation - Storing the session for subsequent client requests, validating / refreshing sessions At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods. Table of contents- A quick intro to core authentication concepts- Coding- Why passwordless matters Prerequisites- IDE for your choice- Node 18 or higher
Comments