Start Building Your Own JavaScript Tools

[♪ music ♪ by The Illuminati plays)] [♪ music ♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ show you how. What if we could get started with building our own tools? Where would we begin? What if the tools were specific to our project? Uniquely helpful and tailored to our needs? What if we could learn how to do this in the next 20 minutes learning what we need to know and how to take our first steps? In the next 20 minutes, let's write our first printing rule, write our first code mod, look at our first AST and build our understanding of them, and learn the fundamentals of how our tools work. Let's dive in.

I see a common thread across any project I work on. Different developers are making the same mistake and we have preferred ways of doing things while it's hard to communicate that to everyone else on our project or using our library. Preventing mistakes and sharing best practices are great reasons to look at tools like linters and in particular ESLint. Not only can we use ESLint's existing rules to handle many problems, but it has a rich ecosystem of plugins specific to different areas and we can write our own rules and plugins, which I'll show you how. ESLint rules are great for problems or mistakes that keep coming up, creating a better feedback loop for developers as they're writing their code, automating code review feedback, sharing best practices, and preventing anti-patterns in our code bases. Rules can be for any project, maybe just for your app, helping developers on your immediate team or others in a shared code base. If you maintain a library, they can help anyone using your library. Anyone can write these rules, not just project maintainers or expert developers.

All right, let's write our first rule together. I'm going to share a little about what I work on and a problem I like to solve where I think would be really helpful. Right now, I work on a design system called Canvas and we provide a component library for UI developers to use. We invest a lot into making our components accessible and useful across a wide range of use cases. As part of that effort, we maintain three different packages for our components, main, preview, and labs. We usually build new components in labs or preview, and as we prove out the API, refine design and make sure accessibility is perfected, we'll promote things from labs and preview up to main. This presents a somewhat unique challenge though. We have some old components, usually in main, that are going to be replaced by newer ones in labs or preview. When a developer is choosing between the two versions though, which way do they go? Sometimes, we really want to tell developers to use the newer one in labs or preview. We already plan to promote it to main in an upcoming release. The component is ready to use and has some benefits we don't want our users to miss out on. How do we communicate this? We do what we can with our docs and sending out announcements, but the best way is to tell them as they write their code. I'm pulling up an online tool that I use to test out new rules. Usually I'll develop with a test suite writing, Lint rules is a really good use case for test driven development. This tool is great to explore the problem, get a first draft, and often that's enough. This is astexplorer.net. There are a few things going on here, but I'll explain how each part works. On the left, there are two editors placed for the code we're working on on the top left and editor for writing an AS Lint rule down below and other tools as well. On the right, we have an AST viewer up above and the output window for debugging below that.

In the header, we can also configure the parser and transform type which I've already set to Babel AS Lint parser and AS Lint v8. Besides giving us a really useful development environment, this AST viewer in the top right is amazing and we're going to explore it in a bit.

AST is an acronym for abstract syntaxtry. If you didn't study computer science at university though, the terminology for an AST might seem intimidating. I promise you though, it's just a tree and if you've worked with a DOM for a webpage, you know how a tree works. An AST just has the syntax for our code stored in it. We can explore this tree in the AST Explorer. It's also worth mentioning that the tree is basically a JavaScript object. There's nothing really new here. Like an HTML tag at the top of a webpage, that being the top node of the tree in the DOM, the top of the AST is the program node. This program node has child nodes such as the ones under body. Here we've got multiple import declarations and these import declarations have children of their own and so on. There's another type of property on a node that's very important, the type. This is how we're going to tell ESLint what we care about. We're going to tell it this for this node type. We want to hear about it and ESLint is going to handle walking the tree and any time it node that we care about it's going to pass it into a function that we can control and define. There are also some properties to describe each node, like the source type being module on the program. There are other values that we can look at that can be really helpful depending on the node we're working with. Looking at this gives us everything we need to detect patterns which is what an ESLint rule is doing.

Let's look at some sample code that includes imports for our components. On the top left we've got a couple different examples, a few actually. First there's the not preferred situation where we have a segmented control component that is coming from our main package CanvasKit React here. That is not preferred because we have a newer one in CanvasKit Preview React. This is our preview package. We want to direct developers to using that over our main package at some point. When we selected the transform for ESLint V8, we got some sample code in the bottom left. This is awesome because it's going to give us a place to kick off. I've cleaned it up a little bit down to its bare essentials. What we have is a create function. This create function returns an object literal.

This object it returns can have any number of properties. These propertys are node types that we care about. In this case, the example has a template literal. Anytime that there's a template literal it's going to get that node. And immediately it's going to report that we have a template literal. We don't care about template literals though.

We care about something else here. Let's click on the import statement in our code. This is going to highlight something in the tree view. And here we see an import declaration. That's what we care about. We want to look at these import declarations and take a closer look.

So let's change this to import declaration. And now we have a rule firing on all these import declarations. All right. Let's narrow down now. We don't want to just report any import declaration. We want to do it conditionally. So I'm going to create a couple conditions that we want to look for. So we want to look for when the, is, when we're importing from the main package. So is main package. I'm just going to put true for that for a moment. Then we also want to look for the component name, being segmented control. So we're going to say, is segmented control. Cool. Also going to say that's true. And then I'm going to wrap this context.report in those two conditions. And then we're going to make those conditions functional. These two conditions are true. That's what we want to know.

And there we go. All right. Let's make them smart now. OK, looking at the main package, that would be this right here. We want to check to see when this value is happening on a part of the node. So if we look at the node, we got the import declaration. We see we're dealing with a source. And the value is under the source.value. So we can look at the node.source.value and compare it against this value here, which is the main package right there. All right, let me indent this a little bit so it's a little bit more readable. It's a little bit bigger here. Cool. So that's the first condition. And already we can see it narrowing down to just one situation here. That's good.

The other condition we want to look at is if it's the segmented control. So if we click on segmented control, this will focus the appropriate nodes in the tree view. And we can see there's an imported and a local. If we look around at a different one, we can see that the local is what we alias something to, if that's different. And the imported is the name that the package is giving us. So back over here in this example, they actually are one and the same, but it's the imported.name that we care about off of the node. So let's look at that. We're going to look at the node.imported. Sorry, actually, I missed something here. This is under the specifiers. All right, so we need to look at the specifiers. And this is an array, so we need to find something in the array. We need to find one segmented control to know that we're dealing with something we care about. node.specifiers. We can find within here, and then that will give us a specifier.

And we want to look for when specifier.imported.name matches segmented control. All right, move this into its own line. Format this a little bit so it's easy to read. Cool. And it's still finding just that one example. Let me take off, or let's see. We are filtering based on that, or filtering based on that. OK, let's try changing some things. We take this and make it just a little bit different. The rule's not fired, so we're filtering on that name properly. And if we change this to just something else, it's not firing on that. That's not even a valid name, but we'll just put this back. Cool.

This is our first ESLint rule. In review, we have a return of an object here, and we're defining the node type that we care about, the import declaration. We have some conditions looking at different children and different properties on that node, like looking at the source and looking at its value, looking through its specifiers, iterating through them, and finding one that matches a name that we care about. And when those things happen, oh, let's give a useful message. There is a preferred version in preview. There we go. Now, if we come over here on the right in the output, we can see that we're giving a helpful message to developers using our project. There's more we can do, like moving this caret indicating where we need to make a change over to where we have the package name. And we can do things like write tests and all that. But this is just a good, quick, little, ESLint rule that we wrote that would be very helpful to our users. So let's take a look at what else we would do.

Like I was saying, if I was to package up this rule and get it ready for consumers, I'd write out a test suite and add support for other duplicate components we have in our next release, recommend what is preferred for each, if there's one in preview, or labs that people should be using when they're ready. There's also the option to implement a fixture in ESLint, which is essentially a code mod. I didn't do that because there's another tool we'll look at that's dedicated to that purpose, especially when we're doing things like what we are with breaking API changes. Let's look at code mods next. A code mod automates a code refactor or change in syntax. They can be written by maintainers of a library, users of a library, or really any developer that has a bunch of changes to apply across a code base.

Code mods are great for improving how quickly consumers can upgrade, say, to a new major version that has breaking changes. They don't need to manually change their code and can just run a script. That's pretty easy. They're also great for making hundreds or even thousands of small repetitive changes. I'd rather spend an hour writing code for a code mod than hand editing hundreds of files, which can take longer and easily result in mistakes. In particular, if you're maintaining a library, it's something that could be really valuable to provide your users.

Let me share with you my first experience using a code mod. Back in 2018, I was working on a product team. We're changing our library for CSS and JS and styled components. We were moving from glamorous to emotion. And while considering our options and the effort required, we found a code mod from another speaker at this conference, Tejas Kumar. Tejas is fantastic, and so is his work. His code mod was perfect for helping us migrate. We ran it on our codebase, and it held most of the effort. Fast forward to 2021, and on my design system team, we were releasing a major version of our library that had some major changes to our APIs. So we decided to start including code mods to make it easier for teams to upgrade. The most popular tool for writing code mods is JS CodeShift.

Let's jump back into AST Explorer and look at an example. Revisiting my previous example, we often promote components from our Labs and Preview packages to Main. Let's look at a case where we want to update the import package from Preview, then to the Main one. Let's say we're promoting segmented control in our next release, and we've already gone ahead and set the parser to Babel and the transform to JS CodeShift. Again, this is going to give us some really handy boilerplate in the bottom left. We still have our tree view in the top right, and we have some output and debugging on the bottom right. The default example's a little bit fun. It's going and finding identifiers, like segmented control and menu here, and just reversing the strings. Cute. There are some differences now with how you write a JS CodeShift code mod versus an ESLint rule. There are also some similarities. Like before, we want to match on a certain identifier to make sure a certain thing, like a node type. And then for a given node type, we want to look for situations that we care about, like replacing all the identifiers in this example.

What's different here is that we're doing things like to source and do we have this chaining API going on. There's some things we can do to break that apart, but this is fine for now. There's also this j API. That's api.js code shift, and the convention is to alias it to j, so we can easily access things like j.identifier, which creates a new identifier node. So we can create new nodes and manipulate existing ones and really mutate what we have to change it for our needs.

So back to our example, we have this string here in the preview in the package name. And that's what we want to change. We want to take that preview and take that out and make it import from the main package, because we're promoting this to the main package. We don't want to mutate this one. There's another component from menu. That's still in preview, and that's not getting promoted yet. So let's see what it takes to modify this.

All right. Like before, we're working with an import declaration. So let's grab that. And let's find all of those. OK, we're going to find all of the import declarations. And what do we do next? Let's filter down to just the ones that match the path we care about. So we can do a filter. And instead of just a node, we actually have a path which contains a node. So we're going to take the path here and pass that into the function. And we care about path dot, what is this again? Again, I'll use the tree view. I'll click on the string. And we are looking at the import declarations source dot value. Cool. So path dot node dot source dot value. And we're going to check if it matches the current value. And I'm going to take for a moment, I'm going to take this whole string and make a variable for that. That's very handy. That's old import.

And we're going to make a new import as well. We'll add it. And that's going to be the same thing without that preview bit. Cool. All right.

So we're going to compare that node source value. When it matches the old import, that's what we care about. And next, for each of those cases that we do find, we want to replace it. Not with an identifier, but with an import declaration. And we're going to create it like this. Let me get rid of all of that. And this is nice. We've got a little bit of code suggestion here saying that first we need the specifiers. And then the second thing we need is the source value. So the cool thing is we can reuse the specifiers from the current import. And we can go path. What was this called again? That's these things. Path.node.specifiers. That's it. Path.node.specifiers. Node.specifiers. And the second thing would be the new import. We can't just give it a string value. We actually have to give it an AST node type. And we can create that using J.string literal. Giving it that string value will give us something that we can use here. And oh my gosh, I think it works. Let's take a look. So import segmented control. There it is.

We don't have the preview on there anymore. We've updated this effectively and we haven't touched the other import that we didn't want to change. Going back to the source to just quickly review. We're finding by a certain node type. We're filtering based on some criteria like the old import is present here. And then we're replacing a node that we care about, particularly this import declaration. We're giving a new import declaration. Using the specifiers here and then we're passing in the new import value that is the main package. And we're promoting our components. This is what we do when we promote things in our new major versions. We have some richer code mods that handle a bunch of other edge cases, but this gives you the basic idea of what's going on.

Cool. Live code, fun. OK. I've heard this a lot from a lot of folks I talk to and even seen some of the challenges within my own team. It's not always easy to write good code mods that work in all use cases. Code mods can be non-trivial to write, and it's easy to accidentally target code that we didn't intend to modify. We will need good tests. We can work around this by extracting utilities to tackle common problems in our domain, having a set of test cases that we always apply to a certain type of code mod. And look at Codeshiftcommunity.com. It's a community project that offers really great examples, best practices, and some helpers you can even import to make your code mods easier to write.

This month, our team released a new major version with breaking changes. And we had several component APIs change, which could bother our users, but instead, this is one of our reactions. Hey, team. Longtime listener, first time caller. Just wanted to parachute in and mention how awesome the V9 code mod was to use. It made upgrading so, so easy. I love to see this. I've never met Simon, but he carries a lot of respect in our community, so this meant a lot coming from him. And this kind of feedback is actually kind of typical when you're automating all the breaking changes and making upgrades seamless as possible.

We're still only scratching the surface of what we can do with building our own tools. Whenever you do this, you're improving the developer experience for yourself and every other developer in your community. The impact can be massive, maybe 10x, maybe 1,000x. More importantly, the satisfaction is immeasurable.

Developers appreciate when their work becomes easier. Building tools that solve a problem is what our whole career is about. Doing this for each other is one of my favorite ways to work. I hope you have some ideas now about how to get started with building your own JavaScript tools.

To learn more about working with ESLint, I recommend their docs at eslint.org. If you're working with codemods, the best place to go is the CodeShift Community website. And for a quick review of everything I just talked about, links to even more resources, check out a site I built, astsareawesome.com. Thank you, everyone, for listening. Please send me your questions, ideas, and thoughts on this topic any time. Thank you.