Five Ways of Taking Advantage of Verdaccio, Your Private and Proxy Node.js Registry

Hi, thanks for joining my talk. My name is Juan Picado. I'm here to show you five ways of taking advantage of Verdash to private and proxy Node.js Registry. So, let's begin.

First of all, something about me. My name is Juan Picado. I'm senior frontend engineer at Mobility Dare, it's a brand of Ativinta. I'm based in Berlin. And also I spent part of my time and doing Open Source, mostly maintaining the Verdash project. So, which if you don't know it, you will know more in the next 20 minutes and hopefully you like it.

So, let's get started. So, we are Node.js developers and also JavaScript developers. So, I can imagine you have already published a package and you haven't done this yet. Then you will enjoy this talk because it's very, very straightforward to do it locally.

So, Verdash is a lightweight and proxy private Node.js registry, which entirely optional configuration, which does allow you just to host and publish private Node.js packages. And it's compatible with any package manager. So, this diagram here describes a simple road trip of a request where the private registry is always in the middle between the remote, which can be one or more. And then you have a local cache. And this local cache really benefits. I want to show you how to use it in your project.

So, first of all, you need to install Verdash. And for that, you need to install the global package globally. And that's it. So, just that simple, you don't need to do anything, just getting started. Then the next thing you have to do, is just run command Verdash. Yeah, that's all. So, you have a registry which is running with a user interface to browse, not only private packages, but also dependencies, even if they are public ones. Because Verdash, we don't load any package request via package manager or user interface. To help you to understand I have prepared five basic ways, and how Verdash can be really useful to any JavaScript developers nowadays.

So, let's start with the first one.

Which is personal development. So, we are going to publish a package. For this example, I have an npm workspaces, which is pretty simple and we will see now. And the idea is to publish some packages.

So, first of all let me show you the structure of the project, which is not much. Just 5 modules. And one of them, this is the configuration for npm workspaces, and one of them it has reference to others inside. We will see why I am doing this when we publish and I showed you on the itself.

So the first thing you have to do is run Verdash. And if you want to publish you need to login. For that you can use the command login. And if you want to point to another registry then use the flag registry and the ul of the registry. In this case localhost 4873, which is the default port of Verdash. You login and then if you don't have any user just use anything you want. The password and the email is not important because Verdash does not use it. And then you are login. You can see the server is reacting to any command you are typing. This is because the package manager is big with the registry API.

So now let's publish a package. And if you have used workspaces in npm you can publish several packages at the same time just using these slack workspaces. And this is what we are going to do and just happen. Yes we did it. So we have packages on the registry. So let's actually see how these packages look like on the registry. And this is the user interface. You can switch between dark mode. And here we have the five packages. Awesome. And these are the dependencies I showed you before. So you can just navigate through them and see who has the dependencies of whom.

And that's it. So just like that. I just published a package in just a few seconds. Yes but there is one constraint you must be aware. If you publish a package, a private package, it's which the version doesn't exist on any of the upstreams which also we named it Applinks, like VAC17 for instance, if it already exists on the registry and Berdacho always will look out for this version if it already exists and if it does condition it's true then you will get 409 error. This is quite common if you want to patch a public version, right?

So for bypassing these, you can just add a a suffix on the version. And your version won't be lost once you Berdacho updates new versions because Berdacho merges what you have local with what you have from the upstreams. So let's make a quick demo as well just to show you the benefits and how we can actually solve this with this package react. Let's see if I found the project... Ah, right here. So here we can see this version, it's the current one on npmjs. Let's run Berdacho again. We are already logged in so we don't need to log in again. And let's try to publish this version with the flag registry. Boom! Yes! Publish conflict. So to solve this you can actually change the version manually, but it's much easier if you use a version patch. And in this case I'm using the not github version because I just cloned this project, I don't want to create a local tag in my project. So I just run this version and it will change automatically. I can run it several times and you can see how the version changed according to the previous one. And now let's publish this package. Just publish and there it is, so if we go back to the registry again we can see if I find it, we can see the latest version is the one I just published. If I access to the React version we can navigate through the version so I can see the ones that are coming from outside plus the one I just published. So let's continue with the next slide.

So publishing is easy but if you are working in an offline environment like me in this video working on a plane which is quite common if you are travelling or you live in remote areas or there is no connection at all. If you want to publish packages, and this is the reason because Vertel exists, because you can publish packages everywhere, then you need to enable the publish offline in the configuration. This is not by default because the reasons are because registry always look up for versions in remotes. If there is no connection the publish will fail. Also you can improve the offline experience by removing the proxy property in the configuration file. You can see on the slide. You will see more about this in the next slides.

There is a reason for that. Next is project productivity and continuous integration in private registries. Most of the public packages we use nowadays are being published through the private registry, but also call camp from other registries. All these registries are services that require high availability, but this is not 100% of the time, because public outages happen, and then you must be prepared for it.

In those situations is when the capabilities of the private registries and proxies are fundamental key of your development pipeline. Because having a proxy registry between your pipeline and your public services it will avoid your teams or your builds to stop working while an outage is happening. Verdash will have this capability out of the box, it will cache any dependency and version by demand, so the next time it's requested it will serve the one it's cached. Because in the continuous integration environment the builds will run multiple times in the same pull request when you push changes in your repository, which involves the package manifest for a public package will be downloaded over and over and over.

There is some default cache in Verdash with 2 minutes but you can increase the cache by just telling the registry I don't need a new version every 2 minutes, maybe 10 minutes is fine for me. This is a really good practice because you will speed up your builds because the registry won't need to go for every request outside to ask whether you have new changes. Also if the network is not working fine you can also increase the max failures on the configuration. All these setups are based on a client-side approach with some friendly intervals like 2 weeks, 3 weeks, 30 seconds and so on. This makes it much easier for the developers to understand what actually are the values you have set up.

And the third topic is security because security is a topic that must be tackled from 2 angles. First, the client-side because you need to use the right configurations in the Package Manager. You can check more about this on the virtualwebsite. I also have a link on the 2 slides forward. But also from the server-side because it raises itself and I will elaborate a bit more now because there's a reason. And if you don't remember what happened 1 year ago, let me quickly recap. A security researcher discovered that a misconfiguration on registries might help intended to be private packages being replaced by malicious packages, which are published on public registries with the same name. This might compromise security. This story is too long, but here I leave a nice article where you can read more about it. That's just a reminder that a registry is a great advantage and brings you an extra layer of security. But also it's a big responsibility because it must be properly configured. There is two actual advices here. First, isolate your private packages by using scopes. Usually is the name of your company, of your project, or organization. And also try to register this name in public registries. Because this is a way that you can have full control. In case you haven't configured properly the registry locally, then at least you own the scope outside. The second reason is, especially if you're using Verdacho, please remove the proxy in the configuration to avoid the registry to fetch external sources for updates. This is highly recommendable for scope packages. If you are intend to be private, especially if you're using Verdacho.

So the latest releases of Verdacho, it includes a new security feature, which is rate limit. At this point, you can add rate limit to two areas, which are the web endpoints, the security endpoints, which are the login and change password. With that, you're going to use the possibility of DNL of service attack, which these are key in points. So by default are actually really set up to really low number of requests, but you can change this. So I totally recommend to use this feature if you haven't noticed it already exists.

So, end-to-end testing, which is my favorite one, which the integrity of your packages is when you are publishing a package and the structure of functionality of these models are not being affected when it's consumed by users. There are so many ways to break a package, the most common one is misconfiguration, because it does not matter how well tested your model is. There are some main steps when you are configuring to ship a package that depends on the developers, and it could be you forgot to add main field if you are trying to ship a common-JS package, or maybe you forgot the module if you are trying to use a JSON module, or maybe you forgot the right patterns on the file property in the package. Also, you might misconfigure the .npm.ignore file in the root file. There are so many things involved on shipping a package, and any mistake counts and can be cached only in publishing. For that, you need a registry. You cannot publish snapshots all the time to the public one, especially if I private packages, so I recommend to test publishing your packages on every pull request, just like a storybook is doing on this slide, for example.

Let's see an example, I can break a package that is intended to be shipped as a command-line interface. Here I have a demo, if you want to look it up later, it just prints hello world on the console, and it uses github actions, docker and node js to run small end-to-end tests. So, as you observe, this is a view of my github project, which my tests are broken on master. So, let's see quickly what's the reason, and let's fix it in a moment. So, this is my project, and if you see, there is something wrong on the config, on the package. So, let's see for a second what might be, and it seems it's thebin file. So, this file does not exist on the root, it doesn't exist. So, what we can do to fix it? Well, just using the right name, right? And, let's actually commit this file because if we... and this should actually fix the problem. So, let's commit this. No, I don't want break master, I want to fix master. Yeah, let's push these changes and then while GitHub runs the action, I will show you how I'm actually doing this test. So, if you're not familiar with GitHub Actions, it's really simple, I would totally recommend to use them. Here we have service. Service is a property that allows you to use docker images within your action. In this case I'm using the Verdacio docker image, which is the official one. I'm exposing two ports, I mean one port, which is the same. For the S73, with this you can use the registry in all the steps below.

So, first I'm just logging because yes, you need to log in the first time. Then I publish the package. This is what any user will do. Then, I just check whether the package is there, which is optional. Then I just install globally the package. When you're installing globally an OGS package using VeeN, this command is gonna be available on the global context. And then I run the test. In the test I don't have anything special, I'm just using a chilled process, which uses SpanSync. Running the command, I'm just supposed to be available on the global context. This will be a first breakpoint if there is no command. This test is gonna fail. Then it's gonna grab the outcome. And then we compare the outcome, if it's fine, everything's fine. Otherwise, I process x with one, which means the GitHub will fail. Let's go to the GitHub action. Yeah, two minutes ago. Fixed master is totally working fine. If we see the action inside, we can see that this actually passed correctly. Perfect. And the master is green again. Super nice. So let's go back to the slides.

There's many examples on open source, and the best way to learn is a good example is reading open source. For instance, you can use Angular CLI, which also uses Chill for running Verdacho. In this case, they are using Verdacho programmatically to run the registry, and this is also a nice way to do it. So if you want more examples, just ping me on the chat and I can give you more, because I have plenty of more to share.

Um, because having your own registry might be intimidating. How hard is hosting a registry? How much cost? How much effort? All these questions are common, and a registry needs two fundamental pieces, the server and the user interface, and Verdacho has both in the same package. Also, you must ask yourself, what are you expecting from a registry? Because Verdacho fits better if you're looking for privacy, flexibility, customization, all offline environments, and overall low cost. Because this is what we are doing in Verdacho organization, we are dogfooding our own registry and our builds.

We have been running our own registry for three years, which around costs $60 per year. And we have thousands of builds every month, including all the dependables on all these people contributing, which is plenty of requests to the registry every day. And we haven't had the need to scale so far because it has been working really great with the default configuration. But this is our own personal setup. You might have different requirements, and Verdacho is flexible. You might need a different storage. For instance, you might use S3 back in Amazon Web Services or your own Cloud like Minio. You might need your own authentication provider or maybe change the interface completely. Verdacho also allows middleware because it's express at this point. This is one example. I'm using MPM Audit. We are applying using plugins giving support to this command. So you might want to extend your own registry. But yes, this might include also personal development, creating your own plugins.

So yeah, that's all for my side. Thanks so much for listening my talk. Verdacho is an open source registry made for developers and made for contributors, which make this project a great alternative. And so in name of the all contributors, thanks for watching my talk. And hopefully you like it and you learn something today. Please don't hesitate to ask more questions or contribute to open source. So thank you so much. Bye-bye.

Taking a look at the pool results, what do you think? So one third of all the attendees, well, they care about hosting a registry, but there's also a lot of interesting answers like security. What do you think about that? Actually, I looked at the results, it matches with my expectations. Usually, yeah, the name of the project description indicates, yeah, you can host a registry. This is the most common use case. But it has several ones, like I mentioned in the book. And Antoine testing is a trend, which is getting popular. You saw in the presentation, I mentioned a few projects, but I see many, many projects, which they have monolibos, which they care about what they're shipping. Antoine testing is something important. Security, yeah. Last year was an important year for registries and security. And the project was fork for another one, which it was more intended to be for personal development. This is maybe the less known. You can use a registry in your computer. And what I'm surprised is the continuous integration, because like I mentioned, one of the slides usually is not used. We rely a lot on services and this is dangerous. So yeah, that's pretty cool. Thank you.

So well, we have a few questions, if you don't mind. First of all, this is my question also, and I believe it's going on in the minds of many people. What is the best way to contribute to Vodaccio? What's the best way to get involved? Well, just the willing to learn. That's the most important one. This is how I started in Vodaccio. Yes, I want to learn Node.js and on the repo, we have several ways you can help. First, if you're just a beginner, you can start fixing types we are migrating. We have our roadmap on the repository, which is peanut, and you can easily find it on the issues and discussions. And there is a description what you can do. Either you are a Node.js developer purely and you want to learn something or improve the code, you have the steps how to do it. If you like more of the front-end, we have a UI, which is based on React, and Material UI, which also needs some help there. And yeah, you can have good features. And if you are not a coder, maybe you want to help with translations.

Also, you can help translate the software to several languages. We have like 15 languages already on the UI and also documentation. And yeah, also you can go to the chat and help others to use Bellagio. That's another way. Thank you very much. That's very cool.

Another question is, can Verdacio be used to proxy not just one, but multiple registries? Yes. When you are using Verdacio the first time and you are installing packages through it, yeah, the packages are not arriving magically. They are coming from the public registry, but you can use several ones. I mean, imagine you have another registry in your company and you want to fetch the packages from there and the public from the public one. So you can set up one or more. There is no limit. I mean, I think the limit is the sky or in this case your hardware. And either is a private, you can use credentials to access to the, in your corporation you can set up credentials and access to several registries. There is one of the features which is not very known. It's like you can define a specific package, for instance, React. Let's imagine. And you can ask for React in your private registry. And you can define a rule, like, okay, if it does not exist on the private one, then I will find it on the other one. And you can create this combination, which is really nice. And, yeah, so it's totally, totally possible. That's really nice.

Well, another question that I had is that I'm sure others might be familiar with the other initiatives to create non-npm repositories. An example would be Entropec, which people might be aware of. How do you think Vodaccio compares or maybe differentiates itself from those projects? Which one? The Entropec you mentioned? Yeah, Entropec. Or is there maybe some other projects that you're aware of that you think are different? There is not many in open source, unfortunately. I would like to see more. But the most of this, Entropec was an idea which is still open, but it seems there is not much development. But it was an idea to create a registry, distributed registry. Yeah, so unfortunately there is not much development there, but the idea was interesting.

And aside of that, there is only services. There is some companies which are creating registries, possibilities to private packages. But yeah, I mean you need to pay for them. Some they have some free trial, which you can use them. But the main difference is like, they still are all in services, right? So you need internet and you need basically internet and I can tell you, usually people approach me to ask me like, oh, I live in this area or I live in this country, which is not allowed to use the service or whatever. And my unique way to learn Node.js is because I can actually install on my computer and just use it. This is interesting for me.