Plug-in architecture: how TypeScript let us paint-by-numbers

Hi. My name is Lilia and I'm an engineering manager at Snyk leading the technical services team. When I started at Snyk about four years ago now, I had very little understanding of the challenges that would lie ahead of me. Having just led an engineering team building cookie cutter websites at a small agency, my expectations were that Snyk would be similar, but maybe more challenging. And I was certainly not prepared for having suddenly to understand every single package manager, like a registry and source control management system that has ever been built. And all this because I've joined the ecosystems team.

I remember one of the very first features was adding support for npm6. This is the first time that npm introduced log files. So this was adding a new flavor of npm into the platform. Then came Yarn, Yarn Workspaces, Gradle, Go, different flavors of Go, Kotlin, SPT, Cocoapods. It just seemed to be a never-ending stream of package managers and tools that needed to be supported. And sometimes the new versions were completely incompatible with the previous versions and required rewriting the support entirely from scratch.

Not only did we need to extend support and keep up with customers migrating to all of these different tools and versions, we also needed to keep up with our own growth. New team members are joining all the time, and onboarding onto the codebase can take some time, as it is complex. Once we've added support for the package manager once or twice, we kind of start seeing some patterns, and then you can speed up. However, as a new team member joins, they have to start from scratch, and then it takes them another 2 or 3 goes before they start seeing those same patterns as you. So we needed to think about how we can speed up. As some of the support was taking longer, sometimes certain areas of the product were being missed where support needed to be extended. So we needed to speed up, but we also needed to ensure that we were covering every area of the product where support needed to be extended. We needed plugin architecture.

Plugin architecture consists of two main components, the core system and the plugin modules themselves. And the plugins interact with the core system via a predefined interface. At the time we've had some of that in place, but not to the extent we wanted. With some time, we were able to start seeing what is the responsibility of the core system and what is the responsibility of the plugins. However, some of these behaviors can still be quite unclear. So you have a little bit of a challenge there as well. So as an opportunity presented itself to build a new library, using plugin architecture, we could use what we've learned from introducing TypeScript really early on into our application, as well as what we've learned and the bits and pieces that we liked and didn't like from our existing plugin architecture. And we could put all of our learnings together and create something new and try again. And we found that with some setup upfront, we could lean on TypeScript and follow the trail of TypeScript errors to help us. So let's have a look at an example to understand how you can follow the TypeScript errors as hints.

We'll be extending a CLI that provides some simple functionality relevant for working with different source control management systems, tools like GitHub and GitLab. So, let's dive in.

So our example today is a really simple CLI called the Source Control Management Helper. We're gonna call it SEM Helper for short. So let's have a look here and we essentially have a command called Describe and we're calling Describe GitHub and then it's expecting a couple of parameters, the owner and the repo. So we're trying to describe a repository. So if we give it some owner and repo as well, we're returning back just really simple pieces of information, whether the repository is forked and whether it's archived as well.

So if we have a look conceptually, we have essentially got a core system that is able to say, repo.describe for example. And then we have a set of plugins that can be called upon, in this case, it's GitHub specifically. And we returned some very simple metadata, for example, for true, archived, false. So let's jump into the code and have a look at what we have set up here.

So it is a really simple, basic, out of the box, follow the TypeScript documentation TypeScript set up. You'll see here that we have a source index and we're using Oclef CLI. So we don't have to worry about parsing of parameters or anything like that. You'll see that there is a command directory as well as a lib plugins directory. In the commands here, we have describe GitHub, which is what you have just seen running. And in describe GitHub, it's got some arguments, it's got an example, some description, and the main functionality here is it loads a plugin, loads a plugin GitHub, and then we're calling plugin dot describe and sending it the repository and on that. And that's it. And then in lib plugins, there is a GitHub directory in which there is a little bit of code, which is get credentials and then it describe API call itself as well. So we need to be able to grab the token and then we're calling in this case, OctoKit rest, which is a GitHub library and calling the specific repos get to get the information that we want. So let's have a look again inside the plugins. We also have an index and types file. So in the index file, you'll see that there is that our load plugin function and it takes an SCM type, which is all supported SCMs. This is an enum of all the supported SCMs. And we want to be able to support GitLab as well. So let's do that. So let's add and register GitHub as well as GitLab as a supported plugin here. So we're going to just say, GitLab equals GitLab. Okay, and let's see what happens. So now that we've done that, you'll see there are in our load plugin file in that index.

We have a couple of TypeScript errors. And you'll see the same if you run npm run build. So we've got three places where we've now got an error. And it's saying that Property GitLab is missing in type but is required. So you can see here that we're expected to populate the name of the entity that we're dealing with. So for GitHub it's a repo, for GitLab, it's also a repo. So let's add that in, GitLab repo.

Okay, then the parent entity type for GitHub with the organization. So for GitLab we also need the equivalent which is also organization. And the last one is plug-ins. So we have a type here that is basically saying that every supported SCM must be registered here as a plug-in. So let's register our new GitLab plug-in. And it looks like we now have to actually create a file for GitLab to have plug-ins.

Okay, let's do that. So GitLab index, let's export nothing. Just an empty object for now. And let's register it here. And let's see what we get. Okay, that's all done. Oh wait, what an error here. What does it not like? Oh, okay, perfect. So it's telling us that we are missing two functions that are required for every plug-in. One called get credentials and another one called describe. And you can see that they're actually defined here. So each one of these needs to be a performance to the interface SCM client, where we need to have a function called get credentials that returns credentials of a certain shape, as well as the describe function that will return archived and fork. And until we have completed this and returned the correct type, this error will not go away. So essentially, by just registering a new type plugin that is to be supported, TypeScript has told us about the three places where we needed to register or have GitLab specific implementation. And even when we've added just a basic plugin, it wasn't good enough because we have defined an interface. Therefore, we're expected to return the correct information. So let's go and implement all the missing bits and pieces, and then have a look if all the errors are now gone.

Now that we've added all the code necessary for GitLab to work as well by calling the relevant GitLab APIs, we can run a CLI with describe-gitlab command as you can see, and we're getting the same results back. So we now support both GitHub repositories and GitLab repositories. And all this by following a couple of errors, really from TypeScript.

So how are we able to do that? The trick was really in having the right setup, where any variable, any functionality that needs to come from the plugin needs to be registered as required. So we have our supported SCMs. And you'll see that we are using this everywhere where we're expecting the functionality to be extended for the SCM. So here in entity name, here in parent entity name, here in the plugin itself. And for the plugin, we have typed what is expected. So we have the entire interface typed out. We have the expected functions that it must implement, which is the get credentials, and describe. We can also extend and add a new function that it must implement. Maybe it's a delete to delete the repo. And as soon as we do that, we're going to get TypeScript errors in all of the plugins, because now these plugins no longer conform to the defined interface.

So by having the right setup upfront and making sure that you register all of the necessary functionality with types, you can register a new plugin and then follow the trail of TypeScript errors to fill in the blanks. And that means that you only need to know the one place where you need to add the new plugin as registered, as supported. And you don't need to read the rest of the code because then you have defined interfaces that you can go and implement and fill in the blanks. So there's no need to understand how the GitHub plugin works, there's no need to understand how the main system code works as well. As long as you know where to register the support for the new plugin. By doing those simple things, we were able to add the plugin without diving too deep into the inner workings of the main code. So you can reduce the time to production because if you know exactly in which parameters you need to write the code, so the functions that you need to implement, what do they return, where those functions are being used, in which case here TypeScript will tell you. Then you don't need to get familiar with the core code at all. Prepping all of this in advance when you are setting up the architecture of the repository has huge benefits later, and you can reduce time to onboard onto the repository. There's no need to learn how the entire code base works all together before being able to contribute to it. You can learn just the areas that are necessary if you lean on TypeScript, and if you ever actually need to change the core system code, you can get familiar with it there and then. So my ask from you is to save time for yourself in the future. Try plugging architecture with TypeScript in your very next project. Thank you.