Forget Polygons – Gaussian Splats, the New Approach to Photorealistic 3D Graphics

Rate this content
Bookmark

gsplat.js is an easy-to-use, general-purpose, open-source library for gaussian splatting. This talk will go over how it works, what it's being used for, and plans for the future.

This talk has been presented at JSNation 2024, check out the latest edition of this JavaScript Conference.

FAQ

GspotJS is a JavaScript library for Gaussian Splat rendering. It allows for high-fidelity, real-time rendering of 3D scenes using Gaussian Splats, and it's lightweight, under 1MB.

Gaussian Splatting involves four main steps: (1) capturing images from different angles to create a point cloud, (2) converting each point into a 3D Gaussian, (3) rasterizing these Gaussians into a 2D image, and (4) training the Gaussians to produce accurate images.

GspotJS was developed by Dylan, a developer advocate at Hugging Face. Inspired by the need to visualize Gaussian Splat results easily, Dylan re-implemented an open-source JavaScript renderer called Antimatter15-Splat and created GspotJS to simplify this process.

Gaussian Splatting differs from photogrammetry in that it is a rasterization technique. It converts underlying data directly into an image without the need for ray tracing, path tracing, or diffusion, unlike photogrammetry.

GspotJS uses various tools and technologies including CountingSort, asynchronous Web Workers, WebAssembly, and the potential for future enhancements with GPU-based Parallel Radix Sort using WebGPU.

For heavier applications like games or hybrid mesh and splat rendering, it is recommended to use Mackellog Gaussian Splats 3D, a Gaussian Splat renderer built on top of 3JS.

Yes, GspotJS is open-source, and contributions from the community are welcome, especially for enhancing its performance with newer technologies.

The goal of GspotJS is to provide an easy-to-use, fast, and simple way to view Gaussian Splats on the web, with a focus on improving speed and simplicity.

Gaussian Splatting is a technique for rendering high-fidelity images very quickly by converting 3D point clouds into 3D Gaussians and then rasterizing them into 2D images.

Key features of GspotJS include its lightweight nature (under 1MB), the ability to render 4D scenes (videos you can look around), and its high speed achieved through techniques like CountingSort and asynchronous Web Workers in WebAssembly.

Dylan Ebert
Dylan Ebert
5 min
17 Jun, 2024

Comments

Sign in or register to post your comment.
Video Summary and Transcription
Today, I'll be talking about GspotJS and Gaussian Splatting, a revolutionary graphics pipeline that can render high-fidelity scenes at 144 FPS. Gaussian Splatting is a technique that converts data directly into an image using Gaussians. GspotJS is a lightweight JavaScript library for Gaussian Splat rendering, with features like 4D rendering. The library aims to provide a simple and speedy way to view Splats on the web, while more advanced applications can use Mackellog Gaussian Splats 3D. Both Gaussian Splatting and gSplotJS are open-source.

1. Introduction to GspotJS

Short description:

Today, I'll be talking about GspotJS and Gaussian Splatting. GspotJS is a revolutionary graphics pipeline that can render high-fidelity scenes at 144 FPS. Gaussian Splatting is the technique behind it, where multiple pictures are used to estimate a 3D point cloud, which is then represented as Gaussians in a matrix. These Gaussians are then rasterized into an image and trained to produce images that resemble the original ones. The trained set of Gaussians can be rasterized from any angle to generate an image.

Hi everyone, today I'll be talking about GspotJS. What is it? Its history? How it works? And where it's going? But first, who am I? My name is Dylan. I'm a developer advocate at Hugging Face, where I build tools and create educational content, sometimes under my name and sometimes under individual keks.

Speaking of which, to answer the question, what is GspotJS? I first need to answer, what is Gaussian Splatting? I have a 2 minute video on that, here it is. Gaussian Splatting. What's that? It's a way to render stuff really high fidelity, really fast. It's a big deal because it's totally different from any existing graphics pipeline and is capable of rendering scenes that look like this, at 144 FPS. The original research paper is 3D Gaussian Splatting for Real-Time Radiance Field Rendering.

What does that mean? I'll explain how it works. Step one, take a bunch of pictures of stuff from different angles, then use an old algorithm called Structure From Motion to estimate a point cloud from the pictures at different angles. Step two, take every point in the point cloud and say you're a Gaussian now. I'm a what? A distribution that looks like this, but in 3D, and also can be skewed, which is what I like to call multivariate. Multivariate. Everyone calls it that. We also assign a color and an alpha. Now we can put all these Gaussians into one giant matrix, with 16 columns and rows, one for every Gaussian. This is all the data we need to represent the scene.

So are we done? No. Step three, rasterization, meaning turn all these Gaussians into an image. How? The simplified version is, according to your camera perspective, project the Gaussians into 2D, then sort them by depth, then for every pixel, iterate over every Gaussian, front to back, calculate their contribution to that pixel, then blend them all together. Now we have an image. So are we done? No. Part four, training. These Gaussians don't have the right values, so we need to train them. Meaning, adjust the values of the Gaussians so that they produce images that look like the original images. This is a lot like training a neural network, but with zero layers, which is why it's so fast. The training also uses automated densification and pruning. Meaning, when a Gaussian is struggling to fit a detailed part of the scene, it splits into two Gaussians. And when a Gaussian's alpha gets too low, it gets removed. Now we have a trained set of millions of Gaussians that can be rasterized from any angle, to produce an image.

2. Overview of Gaussian Splatting and G-Splat JS

Short description:

Gaussian Splatting is a revolutionary rasterization technique that converts data directly into an image. G-Splat JS is a lightweight JavaScript library for Gaussian Splat rendering, similar to other rendering libraries but with added features like 4D rendering. The history of G-Splat JS involves the use of Spaces, a machine learning application at HuggingFace, and the development of a JavaScript library to simplify visualizing Splat results in machine learning demos.

Okay, now what? Well, this is extremely new. It's kinda like when traditional rasterization was first invented, and then Doom came along, and added shadows. And everyone was like, wow, you added shadows. And then came reflections, normal maps, indirect lighting, you know, And this paper is basically reinventing step one. Now you may be thinking, isn't this the same as photogrammetry? No, because this is a rasterization technique, meaning it converts the underlying data directly into an image, without the need for ray tracing, path tracing, or diffusion. So why didn't it exist until now? Because even though it's a simple operation, for it to look as good as it does, you need millions of Gaussians. Which requires several gigs of VRAM. So is graphics about to totally change forever? Or is this a niche application like photogrammetry? Let me know what you think.

So that's Gaussian Splatting. Now what's G-Splat JS? It's a JavaScript library for Gaussian Splat rendering. It has a lot in common with other rendering libraries, like 3JS, or BabylonJS, where you can render a scene, with this code, where you setup a scene, a camera, a renderer, and controls. And then, in an update loop, update the controls, and render the scene. Pretty simple. It also has some extra bells and whistles, like 4D rendering. Basically, a video you can look around. It's also very lightweight, under 1MB, a lot smaller than other rendering libraries.

Now for the history. I'm not a graphics programmer, or a JavaScript developer. But something really cool we have at HuggingFace, is Spaces. These are machine learning applications, made by the community, usually using Gradio, a Python library, that makes it really easy to build machine learning web applications. One of the components of this is Model 3D, which makes it easy to display 3D mesh results. And when Gaussian Splatting came along, I wanted to enable visualizing Splat results. So I found this open source JavaScript renderer, Antimatter15-Splat, by Kevin Kwok. And I was re-implementing this in Spaces. It was really painful. And I thought, it'd be nice if there was a JavaScript library that made this easier. So I made it. Hopefully it'll save others some time. By the way, earlier I mentioned that Gradio Model 3D could visualize mesh results. Well now it can also visualize Splat results, enabling machine learning demos like this.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Building a Voice-Enabled AI Assistant With Javascript
JSNation 2023JSNation 2023
21 min
Building a Voice-Enabled AI Assistant With Javascript
Top Content
This Talk discusses building a voice-activated AI assistant using web APIs and JavaScript. It covers using the Web Speech API for speech recognition and the speech synthesis API for text to speech. The speaker demonstrates how to communicate with the Open AI API and handle the response. The Talk also explores enabling speech recognition and addressing the user. The speaker concludes by mentioning the possibility of creating a product out of the project and using Tauri for native desktop-like experiences.
AI and Web Development: Hype or Reality
JSNation 2023JSNation 2023
24 min
AI and Web Development: Hype or Reality
Top Content
This talk explores the use of AI in web development, including tools like GitHub Copilot and Fig for CLI commands. AI can generate boilerplate code, provide context-aware solutions, and generate dummy data. It can also assist with CSS selectors and regexes, and be integrated into applications. AI is used to enhance the podcast experience by transcribing episodes and providing JSON data. The talk also discusses formatting AI output, crafting requests, and analyzing embeddings for similarity.
The Rise of the AI Engineer
React Summit US 2023React Summit US 2023
30 min
The Rise of the AI Engineer
Watch video: The Rise of the AI Engineer
The rise of AI engineers is driven by the demand for AI and the emergence of ML research and engineering organizations. Start-ups are leveraging AI through APIs, resulting in a time-to-market advantage. The future of AI engineering holds promising results, with a focus on AI UX and the role of AI agents. Equity in AI and the central problems of AI engineering require collective efforts to address. The day-to-day life of an AI engineer involves working on products or infrastructure and dealing with specialties and tools specific to the field.
TensorFlow.js 101: ML in the Browser and Beyond
ML conf EU 2020ML conf EU 2020
41 min
TensorFlow.js 101: ML in the Browser and Beyond
TensorFlow.js enables machine learning in the browser and beyond, with features like face mesh, body segmentation, and pose estimation. It offers JavaScript prototyping and transfer learning capabilities, as well as the ability to recognize custom objects using the Image Project feature. TensorFlow.js can be used with Cloud AutoML for training custom vision models and provides performance benefits in both JavaScript and Python development. It offers interactivity, reach, scale, and performance, and encourages community engagement and collaboration between the JavaScript and machine learning communities.
Web Apps of the Future With Web AI
JSNation 2024JSNation 2024
32 min
Web Apps of the Future With Web AI
Web AI in JavaScript allows for running machine learning models client-side in a web browser, offering advantages such as privacy, offline capabilities, low latency, and cost savings. Various AI models can be used for tasks like background blur, text toxicity detection, 3D data extraction, face mesh recognition, hand tracking, pose detection, and body segmentation. JavaScript libraries like MediaPipe LLM inference API and Visual Blocks facilitate the use of AI models. Web AI is in its early stages but has the potential to revolutionize web experiences and improve accessibility.
The Ai-Assisted Developer Workflow: Build Faster and Smarter Today
JSNation US 2024JSNation US 2024
31 min
The Ai-Assisted Developer Workflow: Build Faster and Smarter Today
AI is transforming software engineering by using agents to help with coding. Agents can autonomously complete tasks and make decisions based on data. Collaborative AI and automation are opening new possibilities in code generation. Bolt is a powerful tool for troubleshooting, bug fixing, and authentication. Code generation tools like Copilot and Cursor provide support for selecting models and codebase awareness. Cline is a useful extension for website inspection and testing. Guidelines for coding with agents include defining requirements, choosing the right model, and frequent testing. Clear and concise instructions are crucial in AI-generated code. Experienced engineers are still necessary in understanding architecture and problem-solving. Energy consumption insights and sustainability are discussed in the Talk.

Workshops on related topic

AI on Demand: Serverless AI
DevOps.js Conf 2024DevOps.js Conf 2024
163 min
AI on Demand: Serverless AI
Top Content
Featured WorkshopFree
Nathan Disidore
Nathan Disidore
In this workshop, we discuss the merits of serverless architecture and how it can be applied to the AI space. We'll explore options around building serverless RAG applications for a more lambda-esque approach to AI. Next, we'll get hands on and build a sample CRUD app that allows you to store information and query it using an LLM with Workers AI, Vectorize, D1, and Cloudflare Workers.
AI for React Developers
React Advanced 2024React Advanced 2024
142 min
AI for React Developers
Featured Workshop
Eve Porcello
Eve Porcello
Knowledge of AI tooling is critical for future-proofing the careers of React developers, and the Vercel suite of AI tools is an approachable on-ramp. In this course, we’ll take a closer look at the Vercel AI SDK and how this can help React developers build streaming interfaces with JavaScript and Next.js. We’ll also incorporate additional 3rd party APIs to build and deploy a music visualization app.
Topics:- Creating a React Project with Next.js- Choosing a LLM- Customizing Streaming Interfaces- Building Routes- Creating and Generating Components - Using Hooks (useChat, useCompletion, useActions, etc)
Leveraging LLMs to Build Intuitive AI Experiences With JavaScript
JSNation 2024JSNation 2024
108 min
Leveraging LLMs to Build Intuitive AI Experiences With JavaScript
Featured Workshop
Roy Derks
Shivay Lamba
2 authors
Today every developer is using LLMs in different forms and shapes, from ChatGPT to code assistants like GitHub CoPilot. Following this, lots of products have introduced embedded AI capabilities, and in this workshop we will make LLMs understandable for web developers. And we'll get into coding your own AI-driven application. No prior experience in working with LLMs or machine learning is needed. Instead, we'll use web technologies such as JavaScript, React which you already know and love while also learning about some new libraries like OpenAI, Transformers.js
Llms Workshop: What They Are and How to Leverage Them
React Summit 2024React Summit 2024
66 min
Llms Workshop: What They Are and How to Leverage Them
Featured Workshop
Nathan Marrs
Haris Rozajac
2 authors
Join Nathan in this hands-on session where you will first learn at a high level what large language models (LLMs) are and how they work. Then dive into an interactive coding exercise where you will implement LLM functionality into a basic example application. During this exercise you will get a feel for key skills for working with LLMs in your own applications such as prompt engineering and exposure to OpenAI's API.
After this session you will have insights around what LLMs are and how they can practically be used to improve your own applications.
Table of contents: - Interactive demo implementing basic LLM powered features in a demo app- Discuss how to decide where to leverage LLMs in a product- Lessons learned around integrating with OpenAI / overview of OpenAI API- Best practices for prompt engineering- Common challenges specific to React (state management :D / good UX practices)
Working With OpenAI and Prompt Engineering for React Developers
React Advanced 2023React Advanced 2023
98 min
Working With OpenAI and Prompt Engineering for React Developers
Top Content
Workshop
Richard Moss
Richard Moss
In this workshop we'll take a tour of applied AI from the perspective of front end developers, zooming in on the emerging best practices when it comes to working with LLMs to build great products. This workshop is based on learnings from working with the OpenAI API from its debut last November to build out a working MVP which became PowerModeAI (A customer facing ideation and slide creation tool).
In the workshop they'll be a mix of presentation and hands on exercises to cover topics including:
- GPT fundamentals- Pitfalls of LLMs- Prompt engineering best practices and techniques- Using the playground effectively- Installing and configuring the OpenAI SDK- Approaches to working with the API and prompt management- Implementing the API to build an AI powered customer facing application- Fine tuning and embeddings- Emerging best practice on LLMOps
Building AI Applications for the Web
React Day Berlin 2023React Day Berlin 2023
98 min
Building AI Applications for the Web
Workshop
Roy Derks
Roy Derks
Today every developer is using LLMs in different forms and shapes. Lots of products have introduced embedded AI capabilities, and in this workshop you’ll learn how to build your own AI application. No experience in building LLMs or machine learning is needed. Instead, we’ll use web technologies such as JavaScript, React and GraphQL which you already know and love.