MemLab: Automating Memory Leak Detection and Heap Analysis

Hi, everyone. My name is Liang. I work in the Meta, focusing on web application infrastructure and performance. Today I'll give an introduction to MemoryLab, a tool I built and open sourced to automate JavaScript memory leak detection and heap analysis. At Meta, larger scale A-B web application has shown that increases in client-side memory usage can significantly impact a web app's speed, responsiveness, crash rate, user engagement, and in some cases, even revenue. MemoryLab is an open-source framework from Meta that automates memory detection for JavaScript applications. It also exposes a powerful API for traversing heap snapshots, so you can build custom analysis tailored to your app. Out of the box, it shifts with CLI tools that test common memory and anti-patterns, and help you debug issues quickly.

And more recently, we have been pushing into AI-assisted workflows with the MCP server that lets tools like Cloud Code reason directly over heap graphs. So, the goal here is simple. Take memory debugging from a manual expert-heavy process to something automated, scalable, and much more accessible. To go deeper into each part, I'll start with automating memory detection. We first need to figure out what's a memory leak. Memory leak is essentially a subset chunk of memory that has been allocated, and it was supposed to be released at some point, but somehow it wasn't. So, the broader class of these leaks can be detected using a three snapshot differentiation approach. MemoryLab automates the process.

To better illustrate the idea, here I'll use some animations. Suppose we want to find memory leaks in route B of a real application. MemoryLab starts by visiting a different route, for example, route A, and keeps track of all objects allocated in route A, then navigates the browser to the target route B where we want to find memory leaks. During the navigation, some objects from route A are deallocated, and some new objects are allocated in route B. Finally, MemoryLab navigates away from the target route B. It can either navigate to a different route, for example, route C, or it can go back to route A again. Now, some of those objects creating route B will be deallocated, but some will remain alive. And of course, route C will create some new objects, but in this example we don't really care since we want to find memory leaks introduced in route B. And as highlighted in the orange section of the slides, the interesting part is the objects allocated in target route B but still remains alive even after we navigate away from it. So we have reduced the search space, but still this is a superset of memory leaks. To get real memory leaks, we use some domain-specific rules to filter the superset. For example, most of the red fiber nodes allocated from a page should be deallocated after a few cleanup navigations. So MemoryLab identifies all fiber nodes and detached DOM elements from the superset. We further filter the list by looking for fiber nodes that keep detached DOM elements alive.

So after refining the list by incorporating the domain-specific knowledge, we get a list of leaked objects. Now the memory detection is done, the final step is to ease the memory debugging process by generating actionable reports. MemoryLab automatically traverses the heap and generates reports that explain why the object is still kept alive in memory. It does so by showing a reference chain from the garbage collector route to these leaked objects. Under the hood, MemoryLab decodes and scans the job to heap snapshots. From the leaked objects, MemoryLab searches for a reasonable path to garbage collector route following certain predefined rules to make the retainer trees readable and actionable.

The retainer trees are reference chains from GC route to the leaked object, and this graph here is a simplistic view of the GSHeap. In reality, a heap snapshot usually contains millions of heap objects and tens of millions of references among those heap objects. These snapshots can expose the VM internal optimization metadata and implementation-specific references that are invisible to JavaScript code. These edges may look like retainers, but they are not necessarily developer-created references or real application leak. So we use Heuristic to tune the trace searching algorithm so the trace is balanced between being short while being reasonable to the developers.

And the user can follow the reference chain step-by-step and figure out why those leaked objects are still kept alive in memory. Very often, a single run can detect thousands of leaked objects, but those leaks are almost always caused by no more than a few bugs or root causes. It will be overwhelming to show thousands of leak traces to users. Ideally, we only want to show one representative leak trace from each root cause. For example, here are three leak traces caused by the same bug. As you can see here, the reference chain looks more or less the same, each with a little bit variation. So, how do we put similar leak traces together? This is where leak trace clustering could be a massive help.

We develop a hybrid model combining heuristics and machine learning algorithms to cluster memory leak traces. Memory leaks with similar retainer traces are clustered together. For each cluster, we only present one of the representative trace to reduce the amount of debugging work. Finally, MemoryLab shows the representative cluster retainer trace of each cluster of memory leaks.

And this is what the output looks like in terminal. In a similar format as retainer tracing Chrome DevTools. With AI these days, it's easy to copy and paste the trace into chat.bt or cloud code to get a clear explanation of this reference chain. On MemoryLab's GitHub repo, we have also skills file for explaining memory leaks retainer trace. With that mental model in place, let's work through how to use the MemoryLab API to define interactions of application where you want to find memory leaks.

To achieve that, we only need to create a JavaScript file with three callbacks using PubTier's API, telling MemoryLab how to navigate your app and which interaction may trigger the leak. We call this file a MemoryLab test scenario file. And then we pass the test scenario file to MemoryLab command line tools. MemoryLab will do the memory heap analysis I showed earlier, and report memory leaks and retainer traces for debugging. The first callback in the test scenario file specifies the initial page loads URL.

MemoryLab opens a webpage in a headerless browser and starts tracking objects allocating the JavaScript heap. The second callback defines the interaction where we want to find memory leaks. Inside the callback, we specify that we want to click the hotel button by using PubTier's API and CSS selector. Those are all standard and well-documented APIs. Under the hood, MemoryLab will track objects allocated and released during the interaction. In the last callback, we specify clicking a button to navigate away from the target state.

The second segment of memory contains memory leaks highlighted in the orange section of the heap in the slides. MemoryLab's leak detector filters unreleased objects allocated by the target interaction. After running the test scenario, MemoryLab generates and clusters leak traces to show why and how leaked objects remain in memory. The documentation site provides more information on using and interpreting the results.

MemoryLab's API offers predefined leak detector processes and callbacks but also allows customization for browser interactions and memory leak detection. By providing a leak filter callback implementation in the test scenario, users can redefine how to identify memory leaks. Each identified memory leak generates a leak trace for the final output. The heap graph reference is passed as an argument to the callback, enabling tailored leak detection. MemoryLab constructs a graph just in time for efficient memory usage and provides an object-oriented heap traversal API.

For each one of the identified memory leak identified by your implementation, MemoryLab will generate the leak trace and put it in a candidate cluster representative trace for the final output. MemoryLab will also pass in the heap graph reference as an argument to the callback. The reference is an intuitive object-oriented heap graph representation of the entire JavaScript heap, including native objects and references added by the engine. The first argument of a callback is a reference to the potentially leaked object in the heap graph. With those arguments, the self-defined leak detector can traverse the heap using the API MemoryLab provides. As an example, our built-in leak detector follows the return chain of a ReactFibre tree to traverse the entire Fibre tree and checks if the Fibre node is detached from the ReactFibre tree.

The program under test will likely have a memory issue, and therefore its heap size could be non-trivial. We want to be memory efficient while providing an easy to use object-oriented heap traversal API. So the graph nodes are designed to be virtual and its nodes are disconnected. And when we traverse the heap, MemoryLab constructs a graph just in time. For example, when entering the callback, only the arguments corresponding graph nodes are constructed. And when you try to query other nodes, for example the referrers of the leaked objects, MemoryLab will create those inbound nodes on demand. Since those virtual nodes are disconnected, we can simply set a reference to now after using those on-demand heap objects. So the inbound node can be garcha-collected.

You can also take heap snapshots manually from Chrome DevTools, save it onto a disk, and use MemoryLab's npm package to programmatically load, parse, and analyze the saved heap snapshot. And again, you can use MemoryLab's heap graph API to traverse and define your own heap analyzer. MemoryLab also provides some pre-defined heap analysis via command line tools. The MemoryLab CLI provides different heap analysis commands. For example, the unbounded object analysis shows which objects keep growing over time by analyzing a sequence of heap snapshots. And the example app under analysis shown on the slides has one object with the size increase from 12 kilobytes to 27 megabytes. This is something worth looking into when optimizing memory usage. And you can use the MemoryLab trace command to get object retainer trace. There are other pre-defined MemoryLab CLI commands to find common anti-paths. Here, I won't show all of them due to the time constraint. MemoryLab also provides an interactive CLI tool for exploring heap snapshots. This could be useful when the snapshot is on a remote server, you are connected by SSH, and want to quickly inspect it remotely. You can integrate MemoryLab with your internal CICD pipeline and run it regularly to detect memory leaks.

As shared in Meta's engineering blog post, this is our internal UI showing a leak trace detected by our internal regular run. This is essentially a reference chain from the GC root at the top of the UI to the leaked object at the bottom of the UI. It tells developers why and how the leaked object is still kept alive in memory in an interactive UI. And each object's function scope on the reference chain can be interacted with and show its internal structure and content.

The debugging process is straightforward. Just follow the reference chain one step at a time and eventually we'll find the reference that should be set now to release the retained object. And you can also aggregate all of the representative leak traces to give a summary view. With MemoryLab, we are able to find memory optimization that leads to significant wins.

Here's one example optimization we did with my Meta colleague Benoit Durand in the earlier version of React a few years ago. React builds a fiber tree for rendering components. For this presentation, you can ignore the structure of the fiber tree and just think of it as an inner data structure React uses for rendering. And the component's fiber tree is referenced by the host root at the top. The React component instance of their corresponding fiber nodes is referencing each other. Similarly, DOM elements and their corresponding fiber nodes are also pointing to each other.

And all fiber nodes are connected in the React fiber tree. So React component instances, DOM elements, and the React fiber tree form a strongly connected graph. If you get a reference to any node in the graph, you can reach or retain any other nodes due to this strong connection. Ideally, other parts of the GSTP only reference the host root. Mounting the least components in the earlier version of React breaks the connection between the host root and the rest of the fiber tree, allowing the entire tree to be garbage collected. However, if the application code caches additional references to component instances or detached DOM elements, the fiber tree cannot be garbage collected.

For instance, exporting a React component instance at the module level can prevent the release of the instance and result in the leak of the entire fiber tree. Maintaining references like this can lead to unexpected memory leaks within the application logic. At Meta, a single React leak can cause the leakage of an entire route, keeping previous pages live in memory as users navigate, causing significant memory leaks. React cleanup processes may not immediately release all resources, leading to ongoing leaks. To address this, an optimization was introduced in React that involves a full traversal of the fiber tree on mount to enable more aggressive cleanup, reducing memory leaks and improving garbage collection efficiency.

Furthermore, to measure the impact of these improvements, A-B testing was conducted on Facebook.com, resulting in a 50% decrease in auto memory crashes in the browser due to the optimization. The full traversal process did not negatively impact page performance and even slightly improved it, leading to significant enhancements in various metrics. MemoryLab also offers a tool called MemoryLens within the browser for visualizing memory leaks directly on the web page's UI debugging layer. This tool, a JS module running within the application, monitors changes in the DOM tree and React fiber, highlighting leaked React components and attaching memory IDs to DOM elements for easy reference in heap snapshots.

It's not just the React data structures that were kept alive. By hooks and their closures, this would often keep live all kinds of product objects. And in some situations, leaking the product code wasn't even required. React cleanup wasn't always immediate and therefore still leaks the entire fiber tree of the previous routes.

So we added the optimization in React by introducing a full traversal of fiber tree on our mount to do more aggressive cleanup. This allows the garbage collector to do a better job at cleaning up an unbounded fiber tree. And also any accidental memory leaks are bounded to a smaller leak size.

To measure all these improvements we actually run A-B testing on Facebook.com and the number of auto memory crashes in the browser dropped by 50% due to that optimization. And surprisingly, the full traversal didn't slow down the page. The memory we actually slightly improved the performance of the page and we observe significant improvements in many other metrics.

MemoryLab also includes an in-browser tool called MemoryLens to visualize memory leaks directly on the web page's UI debugging layer. The tool is a JS module running the application. It monitors the changes in DOM tree and React fiber. And MemoryLens highlights the outline of leaked React components when their DOM elements get detached from the DOM tree but are still kept alive in memory. It also attaches a special memory ID to each DOM element, making it easy to link them back to the view in heap snapshots.

You can also quickly search the memory ID in the Chrome dev tools when you see a leaked component outline and its ID in the UI debugging layer. We have more details in the MemoryLab documentation page for complete guidance for MemoryLens. Finally, a large language model and AI coding are pretty powerful these days. So I want to talk a little bit about how MemoryLab with AI can make memory detection, memory analysis, and memory debugging even easier.

In MemoryLab's GitHub repo, we've added skills files teaching AI how to properly use MemoryLab tooling and MemoryLab APIs. For example, you can provide skills files to Meta-AI, ChattBot, or CloudCode, asking them to write a MemoryLab test scenario file for web application. You can also copy and paste the leak detection results with those AI tooling, asking them to explain or fix the memory leak based on MemoryLab's output. You can also ask AI to help you create a custom analyzer plugin for advanced memory optimization using the MemoryLab API.

Finally, I want to introduce the MemoryLab MCP server package. This allows you to ask AI coding tools like CloudCode to investigate memory leaks in JavaScript heap snapshots more accurately. The MCP server loads heap snapshots only once in a stateful local process and provides all the MCP tools AI needs to query heap snapshots as a heap graph. With MCP servers, AI can traverse and explore the heap graph as much as it wants and do it in a more reliable and efficient way. Instead of parsing the raw heap snapshot from scratch for every follow-up analysis and prompts, it will also save you a lot of the tokens when the heap snapshot is huge.

With the MMP server, you can ask AI all kinds of questions. For example, load heap snapshots in this location and tell me what's using the most memory. Why is this fiber node still alive? Are there any more memory leaks and memory optimization opportunities? Can you create a code commit to fix this memory leak, etc.? The MCP server also supports an eval tool that allows AI to write the heap analysis code using MemoryLab API and executes it in a sandbox environment. So AI can do a very advanced heap analysis based on simple prompts. For example, with MCP server you can give AI prompts like finding inconsistent shapes of objects or hidden classes that waste memory. AI will be able to find examples of different property insertion order that causes the polymorphism of different shapes for the same class of objects. It's a pretty powerful tool to help us fix many memory issues at Meta with a single shot. We highly recommend you give it a try.

Okay, that's the end of my presentation. More demos, documentations, links, resources are available in MemoryLab's GitHub repo. Thanks for listening.