Porting Turborepo to Rust

Rate this content
Bookmark

Everybody loves talking about Rewriting in Rust, but rewrites are hard. It’s far too easy to fall into second system effect, shipping delays, and end up losing all your users. How can you move to Rust while still shipping features and keeping your users happy? Easy, instead of rewriting, port! In this talk, we’ll go over how we ported Turborepo from Go to Rust using three different strategies: a Rust shim that wrapped the existing Go code; a Rust-Go-Rust sandwich to incrementally port dependencies; and finally a completely parallel Rust implementation. We’ll talk about why we chose these strategies, how we went about implementing them, and how we shipped them to users.

This talk has been presented at JSNation US 2024, check out the latest edition of this JavaScript Conference.

FAQ

Turbo Repo is a build system for JavaScript that coordinates different packages within a monorepo. It builds a package dependency graph and orchestrates tasks in parallel while caching outputs, allowing for efficient reruns by reading from the cache.

Turbo Repo was ported to Rust due to alignment and ecosystem reasons. Rust handles details like file permissions better across different operating systems, and it supports excellent JavaScript projects that the team wanted to leverage for better tooling.

Challenges included dealing with Alpine Linux's lack of glibc, cross-compilation issues between Rust and Go, and maintaining feature parity during the incremental porting process.

The Rust shim is a thin layer of Rust wrapping the Go code, allowing Turbo Repo to function as a regular binary in a monorepo by executing the local version installed. It facilitates the incremental porting process by bridging Rust and Go.

The team used Zig, specifically its C compiler ZigCC, to simplify cross-compilation. This approach helped overcome challenges associated with Rust and Go's handling of cross-compilation.

The Go sandwich strategy involves porting individual Go dependencies to Rust and linking them into the Go binary. It uses Protobuf for communication between dependencies, allowing for a step-by-step porting process while maintaining system functionality.

While the Rust-Go sandwich and shim resulted in some performance impact, the full Rust implementation showed a slight performance improvement, although not significantly higher due to Turbo Repo's syscall overhead and caching strategy.

Post-porting, tools like Biome for parsing errors, SWC and Oxy for Turbo Trace, and NAPI for Rust-JavaScript interoperability were utilized to enhance Turbo Repo's functionality and integrate it better with Vercel.

Porting is favored over rewriting to avoid losing users and to continue shipping features and bug fixes. It allows for incremental changes without the risk of halting updates or getting stuck in a never-ending race between old and new codebases.

Nicholas Yang
Nicholas Yang
25 min
18 Nov, 2024

Comments

Sign in or register to post your comment.
Video Summary and Transcription
Today's Talk is about porting to Rust, specifically the experience of porting Turbo Repo, a build system for JavaScript, from Go to Rust. The speaker discusses the challenges faced during the porting process, such as file permission code discrepancies and issues with Alpine Linux. They explain the approach taken, including using a Rust shim and porting individual Go dependencies to Rust. The Talk also covers the limitations and challenges encountered during the porting process, as well as the benefits of leveraging the Rust ecosystem. The speaker discusses the considerations of rewriting versus porting and the importance of institutional knowledge. They also touch on the performance impact of the porting process and the improvements achieved by moving to a fully Rust implementation.
Available in Español: Porting Turborepo to Rust
Video transcription and chapters available for users with access.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Vite: Rethinking Frontend Tooling
JSNation Live 2021JSNation Live 2021
31 min
Vite: Rethinking Frontend Tooling
Top Content
Vite is a next-generation build tool that leverages native ES modules for improved performance. It eliminates the need for bundling and improves hot module replacement. Vite provides an opinionated default configuration while still allowing advanced customization through plugins. It is framework agnostic and can be used for React and other applications. Vite is being adopted by Next.js and Create React App, and integration with Nuxt 3 offers significant speed improvements.
React Compiler - Understanding Idiomatic React (React Forget)
React Advanced 2023React Advanced 2023
33 min
React Compiler - Understanding Idiomatic React (React Forget)
Top Content
Watch video: React Compiler - Understanding Idiomatic React (React Forget)
Joe Savona
Mofei Zhang
2 authors
The Talk discusses React Forget, a compiler built at Meta that aims to optimize client-side React development. It explores the use of memoization to improve performance and the vision of Forget to automatically determine dependencies at build time. Forget is named with an F-word pun and has the potential to optimize server builds and enable dead code elimination. The team plans to make Forget open-source and is focused on ensuring its quality before release.
The Future of Performance Tooling
JSNation 2022JSNation 2022
21 min
The Future of Performance Tooling
Top Content
Today's Talk discusses the future of performance tooling, focusing on user-centric, actionable, and contextual approaches. The introduction highlights Adi Osmani's expertise in performance tools and his passion for DevTools features. The Talk explores the integration of user flows into DevTools and Lighthouse, enabling performance measurement and optimization. It also showcases the import/export feature for user flows and the collaboration potential with Lighthouse. The Talk further delves into the use of flows with other tools like web page test and Cypress, offering cross-browser testing capabilities. The actionable aspect emphasizes the importance of metrics like Interaction to Next Paint and Total Blocking Time, as well as the improvements in Lighthouse and performance debugging tools. Lastly, the Talk emphasizes the iterative nature of performance improvement and the user-centric, actionable, and contextual future of performance tooling.
A Framework for Managing Technical Debt
TechLead Conference 2023TechLead Conference 2023
35 min
A Framework for Managing Technical Debt
Top Content
Today's Talk discusses the importance of managing technical debt through refactoring practices, prioritization, and planning. Successful refactoring requires establishing guidelines, maintaining an inventory, and implementing a process. Celebrating success and ensuring resilience are key to building a strong refactoring culture. Visibility, support, and transparent communication are crucial for addressing technical debt effectively. The team's responsibilities, operating style, and availability should be transparent to product managers.
Debugging JS
React Summit 2023React Summit 2023
24 min
Debugging JS
Top Content
Watch video: Debugging JS
Debugging JavaScript is a crucial skill that is often overlooked in the industry. It is important to understand the problem, reproduce the issue, and identify the root cause. Having a variety of debugging tools and techniques, such as console methods and graphical debuggers, is beneficial. Replay is a time-traveling debugger for JavaScript that allows users to record and inspect bugs. It works with Redux, plain React, and even minified code with the help of source maps.
Building a Voice-Enabled AI Assistant With Javascript
JSNation 2023JSNation 2023
21 min
Building a Voice-Enabled AI Assistant With Javascript
Top Content
This Talk discusses building a voice-activated AI assistant using web APIs and JavaScript. It covers using the Web Speech API for speech recognition and the speech synthesis API for text to speech. The speaker demonstrates how to communicate with the Open AI API and handle the response. The Talk also explores enabling speech recognition and addressing the user. The speaker concludes by mentioning the possibility of creating a product out of the project and using Tauri for native desktop-like experiences.

Workshops on related topic

Build Modern Applications Using GraphQL and Javascript
Node Congress 2024Node Congress 2024
152 min
Build Modern Applications Using GraphQL and Javascript
Featured Workshop
Emanuel Scirlet
Miguel Henriques
2 authors
Come and learn how you can supercharge your modern and secure applications using GraphQL and Javascript. In this workshop we will build a GraphQL API and we will demonstrate the benefits of the query language for APIs and what use cases that are fit for it. Basic Javascript knowledge required.
Using CodeMirror to Build a JavaScript Editor with Linting and AutoComplete
React Day Berlin 2022React Day Berlin 2022
86 min
Using CodeMirror to Build a JavaScript Editor with Linting and AutoComplete
Top Content
WorkshopFree
Hussien Khayoon
Kahvi Patel
2 authors
Using a library might seem easy at first glance, but how do you choose the right library? How do you upgrade an existing one? And how do you wade through the documentation to find what you want?
In this workshop, we’ll discuss all these finer points while going through a general example of building a code editor using CodeMirror in React. All while sharing some of the nuances our team learned about using this library and some problems we encountered.
Building a Shopify App with React & Node
React Summit Remote Edition 2021React Summit Remote Edition 2021
87 min
Building a Shopify App with React & Node
Top Content
WorkshopFree
Jennifer Gray
Hanna Chen
2 authors
Shopify merchants have a diverse set of needs, and developers have a unique opportunity to meet those needs building apps. Building an app can be tough work but Shopify has created a set of tools and resources to help you build out a seamless app experience as quickly as possible. Get hands on experience building an embedded Shopify app using the Shopify App CLI, Polaris and Shopify App Bridge.We’ll show you how to create an app that accesses information from a development store and can run in your local environment.
Build a chat room with Appwrite and React
JSNation 2022JSNation 2022
41 min
Build a chat room with Appwrite and React
WorkshopFree
Wess Cope
Wess Cope
API's/Backends are difficult and we need websockets. You will be using VS Code as your editor, Parcel.js, Chakra-ui, React, React Icons, and Appwrite. By the end of this workshop, you will have the knowledge to build a real-time app using Appwrite and zero API development. Follow along and you'll have an awesome chat app to show off!
Hard GraphQL Problems at Shopify
GraphQL Galaxy 2021GraphQL Galaxy 2021
164 min
Hard GraphQL Problems at Shopify
WorkshopFree
Rebecca Friedman
Jonathan Baker
Alex Ackerman
Théo Ben Hassen
 Greg MacWilliam
5 authors
At Shopify scale, we solve some pretty hard problems. In this workshop, five different speakers will outline some of the challenges we’ve faced, and how we’ve overcome them.

Table of contents:
1 - The infamous "N+1" problem: Jonathan Baker - Let's talk about what it is, why it is a problem, and how Shopify handles it at scale across several GraphQL APIs.
2 - Contextualizing GraphQL APIs: Alex Ackerman - How and why we decided to use directives. I’ll share what directives are, which directives are available out of the box, and how to create custom directives.
3 - Faster GraphQL queries for mobile clients: Theo Ben Hassen - As your mobile app grows, so will your GraphQL queries. In this talk, I will go over diverse strategies to make your queries faster and more effective.
4 - Building tomorrow’s product today: Greg MacWilliam - How Shopify adopts future features in today’s code.
5 - Managing large APIs effectively: Rebecca Friedman - We have thousands of developers at Shopify. Let’s take a look at how we’re ensuring the quality and consistency of our GraphQL APIs with so many contributors.
0 To Auth In An Hour For Your JavaScript App
JSNation 2023JSNation 2023
57 min
0 To Auth In An Hour For Your JavaScript App
WorkshopFree
Asaf Shen
Asaf Shen
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.js backend + Vanilla JS frontend) to authenticate users with One Time Passwords (email) and OAuth, including:
- User authentication – Managing user interactions, returning session / refresh JWTs- Session management and validation – Storing the session securely for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.