WebLLM brings local AI inference to the browser using WebGPU, WebAssembly, MLC-LLM, and Apache TVM. As WebLLM has grown to support more capable models and applications, many of the important problems now sit at the boundaries around model execution: loading large artifacts efficiently; coordinating work across JavaScript, WebAssembly, and WebGPU; supporting different model architectures and device capabilities; connecting model outputs to application logic; and preserving state within an ephemeral browser environment. This talk will cover recent work in WebLLM across these boundaries. We will cover cross-layer profiling and runtime optimizations that go beyond model kernels to reduce memory use, synchronization, and token-generation overhead across the full execution path. We will also discuss explicit compiler-runtime interfaces and model manifests, which define how compiled models expose their capabilities and requirements and enable a common path for multimodal and hybrid models. At the application boundary, WebLLM integrates XGrammar and structural tags to constrain generated outputs into structured data and tool calls. Finally, because a page reload or worker termination can otherwise discard an ongoing generation, WebLLM extends browser-local persistence from cached model artifacts to active inference, using OPFS-backed token journals and asynchronous KV-cache checkpoints to restore and continue interrupted generation.
This talk has been presented at JSNation US 2026, check out the latest edition of this JavaScript Conference.
