Atomic Deployment for JS Hipsters

Hi, folks. Welcome to this talk for us, DevOps JS Conference. We're going to talk about atomic deployment for JavaScript and TypeScript Hipsters.

Right before starting, a little bit of information about what atomic deployment and immutable deployment are. In an atomic deployment, you take two versions of an existing application and you're just deploying the changes between these two versions, but everything is self-contained. So you can take a whole archive of version at any point of time and everything is self-contained and you don't have to worry about anything about that. This is what makes Netlify so successful in the past because suddenly you just had to push a branch on your version server like GitHub or whatever, and it was automatically deployed on your production server in a staging URL like mybranch slash myapp.mydomain.com.

So it's really powerful because you just have to push some changes in any kind of branch, you automatically have a lot of previews directly accessible. And each time you update your branch like updating your PR, the preview updates itself as the same way. So it's definitely useful, especially when deployment is a painful process when you have to worry about dependencies and building and so on. So what is the hipster ways to do these kind of atomic or immutable deployments?

My idea is that I want to keep it working. I want to stick to this principles of atomic and immutable deployments. I want it to be functional, workable, and compatible with any hosting provider, whichever it was. And I want to stick to the simple developer experience. Like, I just want to push my branch and my preview is automatically deployed or updated, and that's all. And I don't care about how to use it and how to work with it and so on. So what are the expected features on these ones specifically?

So I want it to be configurable because I don't want to revamp the whole process for each project. I want to have some things that is easily deployable to any kind of repository and where I just have to tweak a few variables to configure it for this very specific project at a time. I want it to be band-lasting compatible, so working on any kind of hosting provider in the wild. I want some kind of micro-CI integrated like deployment dependencies and building and so on. Everything handled by the solution by itself. So I don't have to worry about anything about that. But without the tests you eat and so on, it doesn't care. I want no too minimal downtime on deployment to keep it really working. So I don't want to break something on the production just because I pushed a brand that suddenly didn't build or something else. So I want some kind of safeguards at some points to be sure that nothing can be down at some point. And I want it to be auto-publishable. So I just want to push it in a branch and boom, my branches, my preview associated to my branch is automatically updated. So I'm Matt, folks. I'm a developer experience engineer at Always Data, which is a hosting provider, an independent one.

So my responsibility at Always Data is working on the developer experience, most specifically on the CLI and the tools associated with the platform itself. What I will show you is the foundation of our automated deployment process with atomic and immutable deployment features. It's currently in development in CLI, but it will be available soon. I'm excited to share it with you because it's the result of my search. Let's dive into the code to see it in action.

For this talk, we will work on the server, whether it's a container, VPS, or bare metal. The idea is to have the web hosts and repositories responsible for atomic deployment in the same place. We will rely heavily on SSH remote access. Webhooks are not part of this talk, but I'll briefly mention them in the conclusion.

To get started, I'll SSH directly onto the server in my container with my hosting account. I'll ensure that I'm in the /var folder, where the www folder is located. This is the document root for our web server engine. I'll initialize a bare repository in /var/repo. A bare repository contains only the Git elements and not the source code itself. Back on my local computer, I'll create a new application and link it to the repository in the container using SSH.

The idea is that you rely on Git basics. So I will SSH directly onto the server in my container with my hosting account, and I will make sure that I'm in the slash var folder, because in slash var there's a www folder. This is the document route for our web server engine, like Apigee or Nginx or Traphic or whatever you want, I don't care.

I will initialize a repository, a bare repository in the very same folder at slash var slash repo. A bare repository is give or take the same thing that you have in the .git folder in your working tree, but you just have the different ashes and elements dedicated to git and not the overall content of your source trees, because I don't care on the server to give access of the codebase at any time.

So if I check, I still have in my slash var folder, two subfolders, the repository in the bare format, and the www1 ready to handle our web apps. So get back locally on my computer. I will cd in my project folder wherever you want to store your project. It's totally up to you. I will create a new application, for example, using Vite with templates, so I will create an application in my app. So it's k-folded in my app, and now I can directly enter into my application.

Then I will link the previously created repo inside my container to this app project locally. So I will git init the application, git add everything that is available in it, and making the first initial commit. So everything is in the root commit in the first commit at the initial commit, first one, initializing it.

So I will add a remote deploy, which is pointing through SSH to the container and the var repo folder we created before. Why deploy and not origin? Because we will probably want to keep origin for our versioning server, like github or gitlab or whatever you want. And you can have as much remote as you wish in git, so why not name this one deploy, because it's ready to handle the deployment process and keep origin for the origin one, the upstream regular one. So I can git push deploy main. So I push on deploy the main branch, and I've got a feedback from git saying, okay, on containers, a var repo, I created a new branch main pointing to the main branch. Success!

Okay, so now let's configure it. We want the project to be configurable, so let's go back through SSH to the var repo. So we are still in our bare repository, and if I check the git config locals, I can see that I have a very few configuration keys, which is interesting, because I don't want to rely on the new YAML configuration file in my source tree. I want to have a few configurable variables that I want to tweak just for this very specific repository dedicated to deployment.

So why not rely on git config tool to do so? So the format is always the same, rearm.keyed.value. So we can create a git config local, because we want to keep it local. Deals.build.dist, because in this project, it's a vid project, so vid will build a dist folder, and this is this dist folder that I want to deploy on my web apps for my web host and so on. I don't want to deploy the whole source tree, just the dist folder. So I specified that for this very specific project, and I used git config to do so, there is a dist build, and I can use so. So I recommend to use a dist or whatever. Don't use anything that git already relies on, like core and so on, because there is a risk of overwriting existing keys, but you can definitely rely to store your very own configuration.

It's open, it's okay. So if we check, it's okay. It's already stored. So let's hook it. So everything right now will occur in the ssh.var.repo folder.

A few words on git hooks. Git hooks are a mechanism that allow you to react to some events occurring in the lifecycle of your git repository. So in the whole list, which is very long, of different events available, there is a few ones that are interesting, and one specifically is really interesting. It's a post receive hook, which is often dedicated to push the cart to production. Interesting. This is exactly what we are trying to do. So let's rely on the post receive hook. Let's have a quick look at it.

It's a hook that is reacting to git push events. When any updates, branches are coming, it executes on the remote repository. Excellent. And once after all the refs have been updated, which is interesting, I could push whole branches, my repository will handle them, and will react to all branches one by one. So I could definitely update and deploy each branch one by one by one every time I receive a git push event if it contains different branches. But if there is only one, it's okay. So it reads lines from stand-in, and it's an automated process. So I will write it in Bash. You could rely on Python or Deno or whatever you want to do so. But it's just automating a bunch of different terminal commands. So Bash is a great tool to do so. So we check we are still in the repo. We create a look at the post receive file and make it executable. So let's initialize it. It's a Bash content. We have a while loop to read the stand-in for each line. And in each line, we've got three kinds of information, one line, one branch.

And we've got three information, the old revision commit, the new revision commit, and the reference, which is obviously the name of the branch. So we can echo in, working on the branch, blah blah blah.

Okay. So we have to set up the environment. So we have to retrieve the path to the git pair folder on the server itself. I could hard-code it, but I want it to be agnostic, so I have to detect it. It's a bit tricky. I don't have deep diving into it, but you've got it. We've got access to the git folder, and we pushed inside the git folder like a CD. So we enter the git folder to execute the rest of the commands.

We check if we have this deploy configuration key and we didn't define this one. So we can fall back to a folder at the same level of the git folder in the VVV directory, which is exactly what we have, slash virus slash VVV for the deployment. So we've got a deployment here. We check that the folder is already ready to handle the rest, and we go where we're ready to continue. So we can for each branch know, for each one, we can set up the branch. We are making some tricks to improve the readability. So we have a ref name, which is removing the ref's heads prefix. We use just a high-cost commit, and we make a temp dir to host our source tree for this very specific branch. So we're also making a link dir to the deploy dir ref name, which is a reman readable pass, because I don't want to configure all my web host to slash virus slash VVV slash a commit ID, but I want them to be pointing to var VVV, my branch name.

So I can just move a link, which is just a pointer from the ref name to the current commit dedicated to this branch. So it will be better. So we use a link to do so, and we prepare the link name.

Okay, so now we have a working tree in the temp folder. So check out the work tree. We use the work tree command, which is like a checkout, but in a more powerful way. So we add a new work tree to this temp folder based on this current branch. And we push into it to move into this working tree.

We have to handle the build process, so we check if we have a build dir configured. This is the case. We defined it before, so we have it. Otherwise, it's just a current folder. We test if we have a package design available. If so, we test if we have a package lock. If so, we make a clean install, because we are clean people, otherwise. Let's do an install, but remember to make your log files, and then we can run the build script if the script is present. If not, it's okay. It's perfectly fine. If the build fails, the script will exit, and we will go to the next branch. And everything is protected like that. So go next.

A quick word on Unix hard links. The idea in the atomic process, if you remember correctly, is that I just want to copy the files that have changed from my previous version to this one. So I don't want to copy everything and duplicate the content. So my idea is to say, okay, let's check if we have a previous version of this branch. So we want to check where is the forking point from the main branch or from the originating branch, and we want to check if there is a commit dedicated to that. If so, if the file hasn't changed between this originate branch and this new one, just copy this one. Just hard link to this version. Otherwise, copy a new version. It will be more performance, faster, and definitely easier to maintain.

And making just you to copy the changes, but each version will be self-contained. This is why we don't rely on symbolic links, but we prefer hard links to do so. So it means that we have to find a git command ancestor. So let's say that we're in the fixd branch. If we are there, we can guess that the forking point is a C1 commit. But if we ask git, hey, what is the forking point for my current branch, git don't write anything because git don't really store a tree of the different branches, but just a list of commits. So you have to rely on yourself to do so.

So you have to get the revision list from the tip of your branch to the main branch and the initial commit. Then you have to check for each one of this commit if it's a valid ancestor. So if it's a forking point from a previous branch, and if so, then you could consider that this commit, if it's still in a branch, could be used as a base for the deployment. So how to do so in code. We've got the rev list, thanks to the revlist command. And for each revision, so each commit in this list, we go back in history. And for each of them, we check if it's a possible ancestor for my branch for this specific revision.

If it's the case, so if it's the forking point, then let's check that it's still in a branch, and that this branch were deployed before. So is this revision still available? If so, then we could consider that we can, like we forked the code, we can fork the deployment and use this folder as a base folder for the revision. Otherwise, just insert the rev, and we will just deploy from scratch, and it's okay. It's perfectly legit to do so. So if we've got a common ancestor, then we just add a few options to air sync, which will be our deploy tool to say, okay, for this specific copy task, use this folder as a base.

And if the file didn't change between this folder and the new one, just make a hard link between the two. So let's go to deploy. We will rely on air sync, as you expected. So we will deploy from the working build directory inside it. So my disk directory inside my work tree, and we will deploy it to my deploy directory. The revision ID commit. We rely on checksum to do so with the air sync option activated. So hard links activated, if it's reliable in this context, and so on. After that, we can just pop from the working folder, remove it from the git repository because we don't need it anymore, and we are clean people, so we are cleaning behind us. And then we can publish it, which means create the links between the reference name and the revision name. So moving the pointer of the symbol in the branch name to the correct commit.

Ensure everything is okay, creating the link. Then after that, we restart the server. So in this demo, I fixed the restarting. So I just killed the existing server previously launched, and I randomized the port and launched an HTTP server pointing to this new production link that I created before, and that is pointing to my new commit. And ta-da, everything is okay. So this is for the publishing.

So let me show you how it works. Go back to my local folder. I update my application. I check that I'm still in my application. I check out a new branch, name fit slash update name. Switching to this branch, editing the application, committing the content with a fancy commit message update name, and then I can ship it. So I git push deploy fit update name. So I push to deploy my fit update name branch. And it says that, okay, I deploy branch to VAR vvv. We're working on the branch fit update name.

I'm preparing the work tree. So we are moving to the work tree in a detached state. Here, we are installing the npm packages. So we are building things to npm run build things to VIT. So we are producing the dist folder. We found the deploy base commit because the branch were forked from a demand branch, and we are deploying this dist folder to VAR vvv, my reference commit name. Then we can make the link to from the fit update name to the final commit id. And I restart the application. And if I check, I can assure you that it works. And in less than 60 lines of bash code. For the same thing, as successful as Netlify. This is just insane.

So what are the real use cases? Because I have said I faked the restarting process. So you will probably have some kind of actions like creating or updating the website with the correct URL like my branch name dot my app dot my domain dot com. Probably performing some DB actions like migration and so on, invalidation of cache, restarting the website, whatever you want. So you have to do so, wrap them in API calls, replacing the fake content of restarting the HTTP server. But if it's just static files like we have here, it's just creating a new VOS in your web engine and it's okay.

What about webhooks? As I said before, you can use them mainly just to automate the pushing from the origin remote to the deploy remote. So each time you push to GitHub or GitLab or whatever, your upstream server, thanks to webhooks, you can automate the retrieving process in the deploy server branch. I wrote a blog post about it if you want. So you can use them for that and it's perfectly okay, it's perfectly fine. It's just reducing the friction in terms of developer experience.

Some few tips for wrapping. Keep it simple and stupid. I make it really agnostic. I don't want to make it really clever or so on. Once again, it's just less than 60 lines of bash code. So make it really case and it will be really easier to maintain and to adapt to a lot of different applications. Use Git config a lot to store your variable and configuration process tweaking and so on. Abstract any changes to your web hosting configuration in API calls using CLI, using API, using whatever you want. And you can for sure rely on Deno or Node.js as a build system thanks to the package json file. You can create a lot of scripts to run to make sure the dependencies are installed, run pre-build process, build process, post-build process, even on complex applications written in Python, Ruby, Rust, whatever you want. But you can definitely use Node.js as a CI engine for the build process.

Thanks again. Thanks for having me today and I hope you enjoy the rest of the conference. Bye-bye.