Supercharging Your Tooling With Rust

Hi, folks. My name is Mael. I work in Developer Efficiency in the NTDX team at Mistral.ai, and today I want to talk to you about Yarn in Rust, a project I spent most of my time these past years. I'm going to tell you a lot about it, including why we did that and how it went. So let's get started.

Before going further, a bit of history about the project, Yarn originated in 2016 in 0.x moniker version. It's only in 2017 that we felt confident enough to tag Yarn as 1.x. I was there. I pressed that damn button. However, about two years later, in 2019, I started to notice that we had learned so much ever since we started the project that if we were to create a new package manager now, we wouldn't do it the way that we did before.

That was a problem, because the way I saw it, we needed to maintain Yarn not only now, but also for the ten years to come. So at the time, it was decided to create a new release that would take everything we had learned about making a good package manager and would put it together under a new implementation of the same package manager. That's how Yarn 2 was born. Of course, this version came with some issues, and we are going to cover some of them in the next slides. But overall, it succeeded, because now we are in 2016, and we are still using Yarn 2.

Of course, it evolved. Now we are at Yarn 4, actually. But the codebase itself stood the test of time. Now, in 2026, we are building something new. We are building Yarn 6, the next major, major version of Yarn, which is going to be written in Rust. And now you can ask yourself, why are we doing that? We already rewrote Yarn once. It was already a bit painful for everyone, for us, for the ecosystem. So why are we doing it? And that's a very good question. And I can't wait to tell you all about that.

First, the thing to realize is that today, Yarn is stable. Yarn is an extremely stable software. It is the poster child of no surprise. If Yarn tells you that something doesn't work, then something doesn't work. And by fixing it, you will not have this error message, now or in the future. And your colleagues are not going to have this error as well. Yarn is deterministic to a fault. We rarely had any install bugs. We installed reasonably fast. Of course, PMPM was sometimes a bit faster than us. Sometimes faster than PMPM.

It was a good challenge. But Yarn was very fast at making typical installs. And building features was easy because we knew the codebase in and out. So, all of that made good reason why we should stay on the codebase we're familiar with. But we had somehow reached the ceiling of what we could build with the codebase we had about Yarn 4. We supported, for instance, monorepos with hundreds of packages. That was great. But we couldn't support workspaces with thousands of repositories. Well, we did. But it was much slower than we anticipated.

And I know what you may think. Yeah, but a thousand of workspaces in the same monorepo, that's too much. Who does that? The answer is that a lot of very, very large companies are doing things like that. And they are not, of course, exposing their monorepo publicly. But in private, they came to us and told us, hey, Yarn is working fine, but now we are starting to get to, it started to feel a little sluggish. And we worked with them in order to improve things. But we were still facing this ceiling.

In some repositories, Yarn could have more than one second of latency. It doesn't seem much. But keep in mind that Yarn run often runs subprocesses that are themselves running other Yarn run processes that are themselves running subprocesses. And if you take all that from one second, you can get to five, six, seven seconds of latency just to run a script. And that's enough for user experience. Of course, it only happens on very large monorepos. But as I said, we wanted to have a good UX for everyone.

Another thing is that we had ideas. We wanted to build new features in Yarn. But some of them needed massive performances. One of them, for instance, is lazy install. The idea is very simple. When you're running a script, if your repository is not up to date, if your dependencies are not up to date, then the package manager should or you install all the missing dependencies without you noticing that something had to run. This is especially important for people working on a git repository. Because when switching from one branch to another, you're frequently going to have differences between your dependency tree.

And the package manager handling this mental task for you means that you can focus on what really matters. But in order to make that possible, we needed lazy installs to be really fast so that running Yarn run would not appear sluggish to our users because they would spend some time installing things under the hood. So, we wanted Yarn to not be just fast, but very, very, very fast. Much faster than anything that existed in the ecosystem.

Another reason why we did that challenge and why we thought writing Yarn was the right time is that I always envision Yarn to be a package manager, not just for JavaScript, but also for multiple ecosystems. What about Python? What about Rust? When we are building a package manager, when we are writing features for them, it's so frustrating that we are building things that will only work if you work on the JavaScript project. In my company, we have people working in JavaScript, but we also have people working in Python. Other people working in Rust. Other people working in Go. And each of those ecosystems has a different package manager.

And any feature I could implement for the JavaScript folks would not be used by anyone else in the company. And that's somewhat frustrating. Because when you think about it, there aren't really a lot of reasons why the code that we wrote for WorkSpaces would not work for people writing Python. And similarly, there is no real reason why the code that we're writing to run audits would work on JavaScript, but not for people doing Go. So, I really envisioned a package manager that could support multiple ecosystems, but Node.js didn't really make it easy to do so. Primarily for distribution problems, of course.

Because if you're distributing Yarn in JavaScript, it means you need to have Node.js installed, which means that in order to develop Python or Go or Rust programs, you need to have Node.js installed, which doesn't make a lot of sense for our users. And then that was one extra prompt to rethink how we envisioned Yarn to be developed. So, for all those reasons and more, I have to go fast here. We felt it was time to reinvent ourselves, to try something new, to go the next way. To prepare Yarn, not only for the ten years that came, but also for the ten years that are coming now.

So, the first time we did a rewrite in 2019, we learned a bunch of things. Something went well. Some things didn't go well. And I didn't want to create a new implementation of Yarn without thinking carefully about the problem that we faced then, and how could we mitigate them now. Now, there were two main reasons why Yarn 2 had such a strong reaction from the ecosystem. The first one is that it had too many breaking changes. At the time, we felt that, oh, since we're relatively code-based, now is the right time to try to do things differently, right? Wrong. That's not the right time when you're making a rewrite. Things are already very complicated. So, if you're on top of that, you're also making breaking changes, you're reaching a critical mass of pain for your users, and they are going to resent that.

Now, the second issue is that there was way too much friction. Part of that was due to our attempt to make things as seamless as possible for our users, which is a bit ironic when you think about that. For instance, we didn't really change the binary that was installed by npm install-g-yarn. It kept being Yarn 1.x, but it meant that a lot of people were asking us, how do I install Yarn 2? It was very difficult to give them a straight answer without actually doing something that would risk breaking people using Yarn 1.x. We stayed that way for a long time, and only now are we going to be able to provide a good answer to that question. So, that was something that was very important to not get wrong this time. Now, also another thing is that in 2019, we rewrote Yarn from scratch. We rewrote even the test suite. In 2026, things are different. We are not actually rewriting entirely from scratch, because we have all our tests that we wrote in 2019 that we can reuse now, which means that it's kind of a data-driven development, because now, thanks to a test architecture that we designed well in 2019, we are able to just run the exact same test with Yarn 6, even though the implementation is completely different, even though now it's a Rust binary.

Our tests are just running a binary, passing its arguments, passing its commands, and then checking that the results on disk are by spouting Node.js or the expected one. And this pattern has worked so well for us. The entirety of our test has been portable from one codebase to another. That has been super helpful for this rewrite. Because then, we could also build a neat UI to allow us to track progress, and I'm going to go over that later, but tracking progress is very important when you're making a change of this amplitude.

Now, thanks to all that work, we know that out of the 1,500 tests that were originally in the Yarn test suite, almost 1,400 are now passing. That's great, because not only does it give us some very large confidence that the new rewrite is going to work for our users, it's also very motivating to see those numbers go up as we make more peers. So, it's really two by one stone. So, how did it go for this rewrite? What did we learn from that? The first thing is that it involved learning a new language. You know, the project started early during the AI boom.

At the time, we were not using AI for everything. And that's how we approached this rewrite. Still, even at that time, TypeDb and other tools already existed. And the question, tell me how you would do that type of code but in Rust was already useful. I remember that I shared frequently snippets of JavaScript and asked, okay, how would that be written in Rust? Because I don't know anything about Rust. And then sometimes it was really bad. And then I was able to see that it was very bad, because I still have some common knowledge of what a good code is supposed to look like. It was also very interesting to see my personal Rust proficiency grow over the course of the project.

But it gave me leads to understand exactly what was happening. It was also very interesting to see my personal Rust proficiency grow over the course of the project. I remember that in some cases, I rewrote the same part of the codebase multiple times. Because I wrote them one month and then the next month, I came back to that code and noticed, oh, but I could use this new construct I learned during this other task. And then I improved it. That didn't make the development faster, but it made it much more fun.

I also learned a couple of things from Rust itself. The first one is that Rust has good tooling. Cargo is really nice and is actually a good package manager. I came a long time ago from the C++ ecosystem. In Rust, you don't have that; they actually have developed an official package manager and maintain it well. Another thing I learned from Rust is that there are so many very interesting experimental features that are stuck in limbo. Rust was a great language that prevented a lot of concurrency or shared memory pitfalls, especially in writing more concurrent aspects of YARN.

For instance, fetching packages in parallel or running resolution in parallel had a high amount of dangers that Rust helped avoid. It didn't come without cost, as learning the language sometimes led to frustrations due to experimental features being unavailable for use. Despite these challenges, Rust's benefits in preventing concurrency issues were significant for the project. The language's design choices and tooling, like Cargo, provided a smoother development experience and enhanced the overall quality of the codebase.

Another thing I learned from Rust is that there are so many very interesting experimental features that are stuck in limbo. Sometimes you're searching, okay, how can I do this kind of thing? And then you go to GitHub and you find an issue about someone saying, yeah, that has been implemented, but it's experimental. So, you can't actually use it. And then you look at when was the last time that this thread was updated and you see that it was, oh, five years ago. And that's quite frequent when you're learning the language. And it can be a bit frustrating.

Rust was a great language. It prevented a lot of concurrency or shared memory pitfalls, especially as we were writing more concurrent aspects of YARN. Like, for instance, fetching packages in parallel or running resolution in parallel. All this kind of thing has a fairly high amount of dangers that could arise if you're following bad code patterns. And Rust allowed us to avoid a lot of those issues. That said, it didn't come without cost. And it required a lot of mental model shifts trying to understand not only how the code was meant to work, but also how Rust wanted us to write that code.

The idea in Rust is that the compiler is helping you enforce rules, but those rules don't always make sense at first read. And sometimes you really have to think hard to understand why the compiler is feeling a certain way. Usually the compiler is right and I'm wrong, but in some cases, it happened that the compiler just wasn't smart enough to do something and I just needed to do it another way. As I told you, the YARN rewrite originated about a year and a half ago. Unlike BUN, for instance, that a couple of weeks ago announced that they had completely rewrote their implementation from ZIG to BUN. Here, most of the code has been written by hand, by humans. And considering Rust's complexity, I feel right about that.

Unlike BUN, for instance, that a couple of weeks ago announced that they had completely rewrote their implementation from ZIG to BUN. That's not what we did. Here, most of the code has been written by hand, by humans. And considering Rust's complexity, I actually feel right about that.

Because since we want to be maintaining this code in the future, I feel more comfortable with a human-driven rewrite that I made and that I understand. That said, AI was still used in places. It was used for code reviews, some investigation around very weird bugs so that we could find reproductions. Also, building some experimental features, especially near the end of development, and porting some easy tests.

Now, we are reaching the conclusion of this talk. Did anything go wrong during the rewrite? That's a very interesting question. Any rewrite is a very difficult work. It's a brutal endeavor. You are probably preparing you have to prepare yourself for a lot of problems doing that. That said, it worked well in our case. But only because we were very perseverant and that we knew where we needed to be. And we had all those tests that we could trust to know where we were in the reimplementation process.

Other things that could go wrong. Well, if you spend your time writing the things, when can you actually integrate it? That was somewhat a concern when I was doing that because at the time, I was working at Datadog. The thing is, either I was building this Yarn rewrite or I was integrating the Rust version of Yarn inside the Datadog's infrastructure. Doing the two at the same time proved very difficult. In the end, I managed to do it. But it required more time than I had anticipated.

Another issue with the rewrite is that can you afford freezing your main branch for multiple months? The time for you to actually spend a lot of time working on that rewrite? Because your competitors are not doing that. They are still cranking features. They are still fixing bugs. And if you stay frozen for too long, then your user may start to resent that. In our case, this was offset by trying to spread the rewrite over a long time. That's why it took one year and a half. So, we didn't completely froze Yarn 4.x. And we kept doing features and improvements there. But it can be a danger. And not all projects are ready to do that.

And finally, the main issue, perhaps, of doing a rewrite like that is that for the first few months, it kind of feels like walking in the desert. You don't have any results. And you're not entirely sure whether it's going to work. Having the test that we had, we kind of knew that we could just figure this out one by one and we would end up getting something. But still, we didn't have those results at the beginning. And so, it was a kind of a leap of faith to go into that project and expect everything to work out of the box a couple of months later.

So, given all that, should all projects be rewritten into Rust? Since Bal did that, NPM is doing that. They're rebuilding part of the package manager. Especially the fetching and linking steps to being Rust. You have all the package managers that are trying to go Rust as well. There were talks about ESLint going into Rust at some point. So, should all projects be Rust? I would personally be cautious about that. And if you're a maintainer, I would tell you that Rust is not a silver bullet. It will not make everything magically better. It's going to come with its own tradeoff that we are going to have to test how you develop, and whether you're using AI or not. If you're not using AI, you're going to have to be familiar with how the Rust compiler expects you to write code. And that takes a lot of time. If you're using AI, then you have to be comfortable with the fact that you never truly know your codebase. Is it something that you will be fine with? That's up to you to decide. In our case, since I wanted to maintain this codebase for a very, very long time, I figured that the choice I had made not to use AI to make a full rewrite, that was partly enforced by not having good options to do that at the time, was the right one.

We made a whole talk about YARN and a rewrite. I'm guessing that you have questions. What are the performances? Did it deliver on the perf side? The answer is yes. YARN is faster than everything else, including YARN 4.x, including NPM. So YARN is much faster now. And we are still not done. There's still low-hanging fruits that we haven't touched yet, because we are still focused on making all our tests fast, then making the Windows implementation, all kinds of things. So I'm sure that we are going to bring the speed, to bring the install time even lower. Something else to keep in mind is that this isn't the time I showed or the average across many scenarios, but some scenarios outperform, like for instance here, where YARN is only taking less than a second in order to do some more installs. And those scenarios are important, because installing a single package is something that a lot of comments are doing when you're adding, when you're making an auto-import in VS Code. You're making just a very small install. So making it fast and not have to spend one second and a half doing work is going to make your life much better in the IDE as you're writing code.

Was this project the right choice for YARN? Was this rewrite the right thing to do? The thing is, we built the package manager we needed for the next decade. I don't imagine us doing another rewrite for a very, very, very long time. The thing is, okay, motivation and resilience were key to make this work. And if you're not ready to do this gruesome work, then perhaps rewriting your project in Rust is not exactly the thing I would recommend you to do.

The thing is, now, you may remember at the beginning of this talk, I told you about how YARN was envisioned to support JavaScript, but not just JavaScript, also Python, Rust, and all the things. The thing is, this story is coming. We are still working on that. It's not entirely clear. It will probably not be at release date, but YARN is moving towards this direction, and that's incredibly exciting for me, because I always wanted development to be universal. Your tooling should be universal if you want to really be able to dig into other ecosystems.

The thing is, one last thing. I'm writing a bunch of devlogs, which are going to be shared on both BlueSky and Twitter, and our Discord channel, of course, so feel free to join our communities there, and I would love to have your feedback on what you thought about this talk, of course, but also this rewrite, whether you're interested in Rust yourself, whether you have questions, feel free to shoot them, and I will be happy to answer all that. Have a good day.

The thing is, the thing is, the thing is...