#webgpu

Joshua introduces Transformers JS and Hugging Face, emphasizing community collaboration and pre-trained models. Transformers JS evolution led to 1.4 million monthly users, supporting 155 architectures. The library's browser-based capabilities offer real-time processing, cost-efficiency, and scalability. Integration enhancements include native web GPU execution and React Native implementation. Web ML implementation focuses on Onyx Runtime for device execution and web GPU for resource optimization. Browser-based ML applications cover vision, speech recognition, and text-to-speech. Advanced implementations include multimodal applications and educational tools. Interactive AI demonstrations showcase semantic search and conversational AI scenarios. Model licensing transitions to ECMAScript for efficiency and model redownloading factors are discussed.

The 3JS project has evolved into a community-driven effort with numerous contributors over the past 14 years. It started with 3D engine work in Flash and transitioned to using SVGs for rendering in HTML5 before adopting WebGL. The project showcases various projects and frameworks, including a no-code tool powered by 3.js. The team is working on a new render using WebGPU and developing a new shader language called TSL. The hope is that WebGPU will eventually replace WebGL, offering better control and performance.

Speaker introduces a side project involving JavaScript and iconic car recreation at JS Nation. Nico, a frontend developer, explains the concept of KIT from Knight Rider and its human-like features, emphasizing the use of machine learning in building it. Using Transformers.js involves defining tasks and models, with options like Whisper for transcribing speech to text and Kokoro.js for text to audio conversion. Adding intelligence through LLMs enhances the capabilities of the pipeline. Models small enough to run on device, allowing browser usage using LLM library. Creating engines and new conversations, streaming replies with added tokens for efficiency. Implementing Kit reasoning process for problem-solving and fun in learning and building using LLMs.

The Talk delves into the integration of generative AI tools in web development, emphasizing the AI revolution's impact. It explores creating a browser-based translation application without backend servers, emphasizing speech-to-text and translation APIs. The discussion highlights browser-based speech capabilities, different voices, and the challenges of translation within the browser. The exploration of the prompt API, Gemini Nano, and specialized APIs showcases experimental features and language model capabilities. The advancements in browser-based AI, privacy-focused AI usage on Chrome, and the utilization of Langflow for server-side generative AI experimentation are also discussed. Progressive enhancement, mobile integration, real-time translation, privacy concerns, and model integration into browsers are key topics.

This talk explores the differences between WebGL and WebGPU, with a focus on transitioning from WebGL to WebGPU. It discusses the initialization process and shader programs in both APIs, as well as the creation of pipelines in WebGPU. The comparison of uniforms highlights the use of uniform buffers for improved performance. The talk also covers the differences in conventions between WebGL and WebGPU, including textures, viewport and clip spaces. Lastly, it mentions the differences in depth range and projection matrix between the two APIs.

With AI and web GPU, it's an exciting time to be a developer. The speaker's journey involves combining programming and design, leading to the creation of Pure Blue, a powerful programming environment. Adding AI to the mix, the speaker discusses the potential of AI in the creative process and its impact on app development. The talk explores the role of React components and WebGPU in enabling fine-grained editing and running AI models locally. The future of app development is discussed, emphasizing the need to stay ahead of the curve and leverage the power of JavaScript.

Welcome to MakePad, a new way to build UI for web and native using WebAssembly and Rust. JavaScript is not suitable for complex applications like IDEs and design tools. Rust, a new programming language, was used to reimagine MakePad, resulting in a fast and efficient platform. MakePad offers live editing, high CPU performance, and the ability to load native instrument components. The future of MakePad includes an open-source release, a design tool, and support for importing 3D models.

Unity targets over 25 platforms and technologies, including desktop, mobile, and virtual reality. They use Emscripten to compile the engine and game logic into WebAssembly for web development. Unity can be extended with plugins to access browser features like WebXR's augmented reality mode. The speaker demonstrates intercepting Unity's calls to the browser to modify its behavior. Unity is actively working on mobile support for web export and improving documentation for extending Unity with web plugins.

WebGPU is an upcoming web API that provides low-level access to the GPU, offering better performance and enabling new rendering techniques. It allows for the use of compute shaders, which provide more control over memory synchronization and threading, and can be used for general computation on a GPU. WebGPU opens up possibilities for unique effects in games and promises future support for ray tracing in browsers.