Infrastructure as Code with a Node Focus

Hey, everyone. It's great to see you here at Node Congress. My name is Tejas. That's me. I'm working for G2I, stands for good news to the Internet. And really, what we do is we help JavaScript engineers or TypeScript engineers find good work, specializing in React, React Native, and Node. But we also help companies find JavaScript for TypeScript engineers specializing in React, React Native and Node. So it's kind of like Tinder but for Java.

Anyway, that's not why we're here. We are here today to talk about infrastructure as code with a focus on serverless Node.js. And you know, this is a hot topic. There's a bunch of talks coming up about this, like tomorrow Slobodin is talking about lambdas in the serverless architecture. And also Colin is talking about AWS CDK2 talks, I highly recommend. I think they're going to be amazing. I'm going to be looking out for them. And I'd encourage you if you like this talk, stick around for those.

But let's get started on this talk by defining the term infrastructure. What do we mean by infrastructure? Because it means different things to different people. What I mean for this talk is all of your tech stuff, your servers, your disks, your databases, your API, routers, your gateways, your load balancers through to your CDN nodes, which serve your static front ends probably. So everything. And let's maybe use an example. So say you have a lambda function. And it's just a function that does something, gives you an input, gives you an output. And at some point, somebody is going to want to interact with this function. And so they'll probably want to connect in some way. But allowing access directly to a function probably isn't the safest idea. So you might need some type of proxy or API gateway in front of it. So if the user talks to that thing, that thing then could talk to a load balancer or could talk to your function. And it's a bit safer. And then maybe your function needs to read or write to a disk.

But the API gateway shouldn't talk to the disk, because the user shouldn't talk directly to the disk. So your architecture, your infrastructure can get a little bit complicated. That's just kind of normal. Things that are alive usually grow.

And traditionally, this stuff was managed by a cloud person or a DevOps person, or maybe many people. Maybe a team, even. But they would largely use a graphical user interface on the web of AWS or Azure or Google Cloud or Heroku or whatever. Or maybe a command line interface, but it's usually a largely manual process. And if we're being honest, this graphical user interface can be scary a bit sometimes.

And so, more than that, I believe the traditional way just has a few weak points. For example, it's not collaborative. If someone on the DevOps team changes the disk size from 8 gigs to 16 gigs on an EC2 instance, I don't know that it happened unless I ask them. And of course, with enough hacking around and creating audit logs, maybe, but by nature, like kind of out of the box, it isn't geared towards collaboration.

It can't be versioned. How cool would it be if I could have an infrastructure with a bunch of components and load balancers and things, and then I check that in Git and then I can time travel to it. I can even revert or commit a new version. You can't do that traditionally. It's largely manual. As we just saw, people would usually go in and change components or add new components or remove components by hand.

It's ultimately opaque. I've been a part of teams where I had no idea what the infrastructure looked like. I just didn't know. I pushed something and then it's deployed, but how? No idea. I feel by knowing this, we might facilitate higher-velocity software and so I believe there is a better way. I believe there's not just a better way, there is a sexy way. And that is infrastructure as code.

So you might be asking, how is this any sexier than traditional? Well, it's a version. So imagine a file, a text file, where it describes everything you have, from your load balancers, to your databases, to everything. It's a text file, it's a text file, it can be checked into Git versions. Secondly, when you deploy things, it creates a snapshot of the state of your infrastructure, and so if you want to change your disk from 8 gigs to 16, it can intelligently diff and find out change and update that component.

It's like React virtual DOM, but for your cloud infrastructure. It's incredible. It's reproducible, meaning you can tear it down, create a new AWS account and deploy all your infrastructure to that. And lastly, it's transparent. Everybody can see it. It's literally like a giant blueprint of all your infrastructure.

So, if there's an early-in-career developer on your team that wants to know where everything is, you can show them. This is the blueprint. And so, it's really cool. And it's really interesting. I can already see some of you on this Zoom call going, okay, but how? I want this. How do I get this? The answer is, we have tools. There's tools. There's Chef. There's Puppet. There's SaltStack. There's Ansible. There's Terraform. There's a bunch of them. And they're all good in their own way.

For me personally, I love Terraform from HashiCorp. I find it the most idiomatic. And it also works. It's very versatile. It can work pretty much everywhere. But instead of talking more about it, let me just show you. Let's switch to a demo. So, we have on the left-hand side, we have a browser. On the right-hand side, we have a code editor. And we have some docs here that we'll talk about in a second.

But what are we deploying? Well, we have a NodeJS app written in TypeScript that, you know, it's quite basic. It creates a puppeteer instance. Puppeteer, for those who don't know, is just headless Chrome. It's Chrome without a UI. It visits a page, goes to a URL given to the function, takes a screenshot, and then returns the screenshot. Pretty much. This is wrapped in an AWS Lambda handler. It just literally calls a function with a URL from query params and returns it. Just a screenshot. Nothing special.

But, of course, it's not deployed. It's just a thing. So, how do we deploy this? Well, for starters, let's go into the project and build it. This will give us a zip file that we can then deploy to AWS Lambda. It's going to create a nice little zip file for us. And in a couple seconds, there we go. We have a bundle.

Let's create a file called main.tf. That's going to be where we're going to camp out for most of this. Let's just get started writing this. We're writing a language called HCL. That's how you describe all your infrastructure. You start off with a block called Terraform. And you mention what providers you want. I just want AWS. And its source is HashiCorp slash AWS. And you might be thinking, what in the world is that? We'll get to that in a second. Also, I'll just kind of write one rule for this provider. And that is I'm going to be in the EU central one region. Because I'm in Berlin.

I'm going to save this. Now, this provider, what is that? Well, HashiCorp, the company that makes Terraform, has a registry. This is literally like an NPM registry. And so I'm saying, hey, go to that registry and get me this module. It's like a node module. And this module allows you to talk to AWS. Now, these providers literally can talk to any back end, Azure, Google Cloud, or whatever. They connect Terraform to some cloud provider's API. And allow you to declaratively describe your infrastructure.

So we're working with Lambda. So let's search the docs for Lambda function. That's what I want. And this is really just copy, paste-driven development. So I'm just going to copy this basic example here. Whoa. I struggle with trackpads. Okay. So I'm just going to copy this whole thing and paste it here. This is a rule that allows us to work in AWS with Lambda. So I'm just going to leave that as it is. And I'm going to change some of these values here. So my file name from my deployment bundle is bundle.zip as you can see. Function name I don't care about. And this role is accessing it from here. So you can see that the resource name is AWS IAM role. And the resource name that I give it, or the ID, is IAM4Lambda. So it's awsiamrole.iam4lambda.arn, the roles are in identifier. The handler is index.handler, because handler is the exported const. So index.handler. And, yeah, we don't need this.

But what we do need, we don't even need the environment. What we do need is a bunch of memory. Let's add that. Let's give it a longer timeout. And lastly, we need a layer. In Lambda, layers are like NPM modules. And there's one for the version of Chrome we're using. So I'm just gonna copy the ARN for this layer. And that's going to allow us to import it without actually shipping all that code, all that node modules with us.

Okay, so we have a Lambda. Let's just try and make this exist with Terraform. So I'll do terraform init. And this will...it's like NPM installed, basically. It's going to go to the registry, and it's going to pull down that thing. And once it pulls down that thing, we can then make real AWS infrastructure happen. While it's pulling that down, as we saw in the presentation, it's not enough to just have someone talk directly to a Lambda. We need an API gateway. So let's quickly map that up. So we need an API gateway. If you know AWS API gateway, there's some concepts here. I don't have time to go into them, but they're all on the docs, I promise.

That's great. Let's make it exist. We'll do terraform apply, and that's going to actually take our declarative file here and map it to real AWS stuff. It's going to create one role, and one Lambda. So what it's done here is it's created a plan. This plan is going to create stuff. It's going to make a role. You can tell with a plus and some stuff will only be known after it exists.

You can tell with a plus and some stuff will only be known after it exists. And it's going to make a function. You know, this looks good to me. I'm going to say yes, anything other than yes, will be rejected. And it's going to go off and make the infrastructure exist. Very, very exciting. And you can see it's kind of creating here.

So anyway, back to planning. We need an API. We need a resource. We need a method. We need an integration. And then we need root methods and root integrations. Yes, it's a bit complicated to play with APIs. And then we need deployments. We need an invoke permission to have the API, like invoke the Lambda. And lastly, these are the components we need. I may speed through them, but they're on the box. It's just copy-paste.

So we can tell it created our stuff. We just can't call it yet because we don't have an API. We're going to fix that. So let's start with the API. What I'm going to do is, I'm going to do a resource, AWS API gateway REST API. That's the name of the resource. And I can give it whatever name I want. So I'll just call it API and the name is whatever I want. It's a description, which is, I don't really care. And this is important because we're going to be returning an image.

And this is important because we're going to be returning an image. So binary media types is a wildcard for everything. We need a resource as well. And the resource is just a thing on a pack. So AWS API gateway resource. Boom. And we'll call it RES, I guess. And you know, it needs a REST API ID, which is the ID of our newly created REST API. It needs a resource ID, sorry, it needs a parent ID by that parent ID and we will .root resource ID here. And it needs a path part. And this what we're gonna do here, this just says proxy everything from Slash to whatever integration you have. So it's just a special script for AWS, but that's all we need.

And then we need a component. So API gateway, sorry, not component, method. AWS API gateway. We need a method. We can call it met. And you know, again, it needs a REST API ID, it needs a resource ID, which is our new resources ID. So dot res dot ID. It needs an HTTP method to work on, and it needs to know if it has any authorization, which in our case, it does not.

Okay, time for the integration. So you can see, it's just really a bunch of commands. We'll call this int. And same thing, it needs a REST API ID. It needs a resource ID. It needs an HTTP method, but we'll use the HTTP method from our method. Okay, and then it needs some internal stuff for when it talks to the Lambda internally. So it needs an integration HTTP method, and it's going to talk to the Lambda on post, even though it gets a get. It'll be an internal AWS proxy to another AWS component, which is Lambda. And the URI of this is our Lambda function called test Lambda. It's its URI. So test Lambda dot invoke ARN.

Okay. And so we're just going to basically just copy paste these two things down here and swap out the resource IDs for the root resource IDs. So the resource ID here is going to be root resource ID. That's going to stay the same. And this resource ID is also going to be that. Good. That's pretty much it. Now we just have to create a deployment of our API, give it permission, and then we're golden. So we'll do resource AWS API deploy, AWS API gateway deployment. We call it DEP.

And this is going to be really interesting because it depends on both our integrations. And speaking of which, let's give them unique names here. So we have the integration int and the integration int root. So int and then .introot. Okay. It needs a stage name, which we'll call dev. And there's a slash stage when we access it. And lastly, it needs the rest APIs ID, which, if we go up to where we created the rest API, we can just do it, .api.id.

Okay. So lastly, we need to give it permission. And so we'll do resource lambda permission. Exactly. We'll call it perm. Look, we have autocomplete. This just writes itself. There's a bunch of hard-coded strings here, though, that is from the Amazon docs. So yeah, you can just copy-paste those. The function name is our function name from above, which is this.testlambda.function.name. The principle is apigateway.amazon.aws.com. And lastly, our source ARN. So who's going to call this? So we'll go to the deployment here. And get, sorry, not the deployment, we'll go to the rest API, my bad. The rest API. Exactly. And we will get the API.executionARN. But I want to add some sub-path wildcards to it. So I'm going to interpolate this string with that. And lastly, let's just do an output, call it endpoints. And its value is going to be our deployments invoke ARN. So the deployment's name is deck.invokeURL.

Alright. So that's not that much. Now, we will apply this infrastructure. And watch what happens. We already created the lambda. So it's not going to create that. It's just going to create all the extra stuff. So it's going to diff in real-time, what changed. And as you can see, it has 8 things to add. And it's just a bunch of this.

So this is the plan. If I look at this, if it looks good, I'm going to say yes. And it's literally just going to deploy that for us. And at the end, hopefully, we get a URL up here. If I open this URL, what do we have? Hopefully, we have a screenshot of the default route of My Screen Shotter, which is Google. Awesome. And then, if we do a URL, let's try nodecongress.com. We should have a job role, please. Okay, so we have that. We can go nuts, we can do this all day. We have g2i.co. We can do my personal website. So, it works. That's it. We just deployed a bunch of AWS Lambda infrastructure with a text file. And if we want to, we can bring it down just as easily. I just do this. I say yes. This yes is very important, because it can crash everywhere. But I do that, and boom.

It no longer exists, but I could just as well apply it again, and then it exists again. So, let's come back to the presentation. So, we just deployed Node.js infrastructure without leaving our editor in a way where our team can collaborate, scale, destroy, and recreate as needed. This is the power of infrastructure is good.

Let's talk takeaways. Really, I have four takeaways for you. I want you to remember as you leave this talk, I want you to remember MERT. Specifically, mechanized processes, automated processes, tend to always be better than manual processes, because they're things you can repeat, iterate, and improve, and not have to guess. Ephemeral state tends to be better than internal state, or long-lived state, because long-lived state can get corrupt, it can accumulate staleness, whereas something that is wiped and refreshed, and still the same, tends to be more pure. Repeatable processes tend to be better than rare processes, because, again, they're processes that can be automated, and iterated upon, and tested, most importantly. Lastly, transparency is often better than turbid, meaning open software, open source software, open deployments, open infrastructure, tends to do better than some black box that you have no idea how it works, for scale and for teams.

So, with that, I just want to say thank you so much for your time. Yeah, hey, how's it going? I'm disappointed. So, what's your view on this? Why is it a sandwich? You know, it's bread. It's a thing in bread. Like, you know, if I put tofu in bread, it's a tofu sandwich. I don't get it. Yeah, I don't get this. Yeah, it's an important thing to discuss, but it's really weird that people say it's not. So, if you have another opinion, explain it to us in the community Discord channel. But, yeah, sorry, agreed to disagree.

But now, let's go to the questions from our audience. And the first question I would like you to answer is from Java Task. So, when we have developers version 2.0, do we still need dev ops? Yeah, can you say that again? Developers 2.0? We have developers version 2.0, do we need dev ops? I don't. Assuming developers version 2.0 means, like, an upgraded version of the developer who does cloud stuff, I think yes, definitely. Because, I mean, it doesn't hurt to have someone... I feel, you know, Spotify as a company does a really great job of this kind of T-shaped diagram of experience. So, you can have broad experience or you can have deep experience, but you can't often have all the experience. And so, if I consider myself maybe a developer 2.0, I do do some cloud stuff. I also do some front and some backend, but I don't think I would have the depth of a full-time hardcore DevOps expert. I wouldn't even claim that. And so, I could easily mess things up if I'm not careful. So, it would help, definitely, to have somebody who's got the depth that I don't have. Yeah, it's always helpful to have experts in your team and know the specifics. And it's good if you can help out a little bit and maybe give like a, I can't find the English word of start. Yeah. That's, that's, that's basically. Yeah. I agree. So we agree again, two, two points.

Next question is from cyberFox 909, buy you a drink. Sorry, I think I think we're just making a joke. Next question is from cyberFox 909. If I make some change, the Terraform file, which will generally change the infrastructure, does the whole build process happen again, or it only changes the component that's being changed? Yeah. That's a good question. So I talked about this in my talk briefly, but I tend to mumble, so maybe it was missed. It's actually, I, I know there's some frontend developers in the room. So it's pretty cool because Terraform is to your infrastructure, literally like React is to your user interface. And what that means is it's just this declarative thing that is able to detect changes and then only change what you described should change. You know, so like, for example, if you create a disc from two gigs to eight gigs on AWS, nothing else about my infrastructure will change. It won't rebuild the whole thing. It'll just do a cumulative incremental change. So it's like hot reloading. Yeah, exactly. It's like hot reloading your infrastructure. That's pretty hardcore. That's – for a front-end developer, that sounds pretty cool.

Another question from Bruce. How much would it cost per month for this particular demo you've given? That's an excellent question, and you know, I'm glad Bruce asked. This, I think, will be covered in the talk tomorrow about serverless functions. In this case, we deployed a serverless function, and I believe they're the best way and the cheapest, and just the most economically smart way to deploy stuff. Because you pay nothing if it's not used. Like, normally, if you work with a VM or EC2 instance in the cloud, or even if you do a Kubernetes cluster, you're going to be paying for those machines to just be on. Whether people use them, or they don't. But with a Lambda, with a function like this, if nobody uses it, you pay nothing. You pay for every time it's invoked, and you pay for how much time it takes to do its thing, and you pay for how much memory it uses to do its thing. And those prices are tiny. So, it really depends on — to answer the question, how much would this cost? It depends on demand. Like, if now, after this talk, the whole world goes and uses this Lambda, I might have a reasonable bill, but generally, it's not that expensive. Well, we have 12,000 people watching, so we could bankrupt you. Oh, no. Everyone now.

Next question is from Ghost1m. What do you recommend us to have multi-tenant infrastructure using Terraform? Would I recommend multi-tenant infrastructure? What do you recommend us? I don't know how to answer that question. What do I recommend you to have multi-tenant? I'm sorry, I don't understand the question. Okay, so, Ghost1m, if you can clarify your question a little bit better, then we might have time to get back to it. Or I will be in the spatial chat. There's a whole room where we can hang out and talk all day.

The next question is from bkankal. Is there an open source library of reusable Terraform modules that you would recommend? That's a good question. So, in my talk, I showed off the Terraform Registry, which is basically the NPM registry, but for cloud stuff. So, there's an equivalent to NPM packages of Terraform modules. They're not really modules. They're more like connections to external services. So, in terms of like a module registry, I'm not sure. I don't really use anything out of the box because I can, as we just saw in the talk, I'm pretty comfortable copy pasting from the registry. But, the registry itself has like adapters for literally almost every cloud provider I can find. Like, they even have stuff for Heroku. And I think digital... Like, it's just, it's rich. But, in terms of reusable modules, I've not looked into that because I haven't had the need personally. That's also an answer I don't know is an answer.

Next question is from Tom. What are your thoughts on AWS CloudFormation versus AWS CDK versus Terraform? That's a great question. I've used CloudFormation. And I want to be respectful and kind but I did not like it at all. I found it tedious. I found it overly verbose. And I found it... There's many different ways to write a CloudFormation file, from my experience. I would write it and then somebody would tell me to do something different. It didn't seem very solid. I haven't worked with AWS CDK specifically but Terraform is creating... I think it's called CDKTS, something like this. Which is... I'm super excited for this. Because it follows in principle the same principles of the AWS CDK. But Terraform is TypeScript first. And so me as a TypeScript engineer this is gold. Because I don't have to learn this other language, HCL to write my Terraform files. I can literally just write TypeScript and I get perfect auto completion on everything. And then it just outputs a Terraform configuration. So if AWS CDK has TypeScript out of the box I would get on that as well. But for me the value in Terraform is that it's not vendor locked to AWS. I could write a Terraform file for Azure, for Google Cloud, for whatever I want. And that's for me the value there.

Okay thank you. Next question is from Dan G. What is the best way to get started with Terraform? Watch my talk. I think it's honestly like you could write some node thing and deploy it as Lambda. That's a great way to get started. And then really go from there.

I honestly believe that's the best way to get started. If you don't have access to my talk, for whatever reason, the way I learned most things, Terraform, Kubernetes, Docker, whatever it is, React, I just open the docs and read every word from start to finish. And then I'm like, okay cool I kind of get the gist of this. So that's an option as well. Which is really time intensive. Yeah, but it's worth it. It's an investment, right? Yeah, yeah, yeah sure.

Okay, next question is from Tazi. Can we convert an existing infrastructure to code or do we need to set up a brand new infra and migrate? Yeah, that's a tricky one. So, you know, I hope I'm not violating any NDAs or whatever, but like where I work, we don't have a large infrastructure as code style of doing things. Like it's largely been manual, but it's something we're moving towards. Something we'd like to move towards and I haven't, I've been looking at can we, you know, can we take our existing manual flow and use the existing services but map them to Terraform resources? I haven't found a way to do that yet. I'd like to, but I think what we're gonna do go ahead and do is kind of clone everything we have in a Terraform file and then just switch over if it comes to that. But yeah, so the answer is I haven't found a way to do that yet, although I'd like to have it. So if you know something, let me know in the special. Cool.

We have another question from Java task. Can we use Node 14 AS6 modules to write lambdas without Babel and with tree shaking? This is a great question, because it comes down to define a lambda, right. And if a lambda is just a function, then yes, you can, you can write it with whatever you want. You can write it with Python. Sure, but functions that run, run in a runtime. And so I feel like the answer to this question is it depends, it depends on the runtime. Can your runtime handle native ES 2018 or whatever ES you want to write. And AWS Lambda, as far as I can tell, cannot. It handles node 14, I think, as a runtime maximum, which has some modern features. But the real challenges, you know, the burden of bundling your lambda is on you. So like, you can't just npm import whatever you want. And just ship a plain JavaScript file to Lambda. You'd have to use Webpack or Rollup or something. Yeah.

Okay. You gotta have that. Oh, I forgot the word. Bundling, yeah.

We have time for one more question. It's a question from Novorf. Speaking of infrastructure as code, what are your experiences so far with maintaining CICD pipelines as source files under version control? Do you have any favorite solutions? Can you ask it again? Speaking of infrastructure as code, what are your experiences so far with maintaining CICD pipelines as source files under version control? Do you have any favorite solutions? I'm having a hard time understanding the question, but I'll answer it as best I can. If it's regarding version control of source files of infrastructure, so like these Terraform files. Like you can, I mean, obviously the point of a Terraform file is that you can check it into version control and collaborate on it. But in terms of checking in and versioning specifically the state file, because what Terraform will do is it will manage the state of your infrastructure and keep track of that. That is best not checked into version control because it contains sensitive values, passwords and things. So my solution for checking in and versioning those is to not, but instead to use something like Terraform Cloud, which is generously free up to five users. And so for us, we don't really have more than five people who touch Terraform. So we just connect that to a GitHub repo and everything's taken care of.

Okay, cool. So we have more questions that have been coming in, in the Q&A channel, but we have no more time. So for the people that have been asking questions, you can join Dajez in his spatial chat speaker room, where he's going to go now, right? Yeah, that's right. All right, Dajez. I want to thank you a lot for this amazing Q&A and hope to see you again somewhere in real life, maybe someday. Awesome.