Increase App Confidence Using CI/CD and Infrastructure as Code

Hello, everyone. I want to thank you all for attending my talk today. The talk's title is increasing your app confidence, using CICD and infrastructure as code. In this talk, I'm going to basically speak to how you can leverage continuous integration, continuous deployment, along with CICD, along with infrastructure as code to basically implement tests that can figure out if your deployments are broken after they're actually deployed to target environments.

Here's an agenda that I put together for this talk. A real quick intro into myself and what I do at CircleCI. Then I'm going to jump into a very brief discussion around testing. And then we're going to jump into talking about broken deployments, right? And finally, I'm going to discuss, you know, how you can increase your app confidence within your releases using CICD and infrastructure as code. And then I want to end with a demo that will kind of show all those concepts in action.

So, my name is Angel Rivera. I'm a developer advocate at CircleCI. And essentially what I do as a developer advocate is engage the community at a grassroots level. When I'm engaging the community, I'm discussing all things technology, right? Doesn't matter if it's CICD related or DevOps related. I'm interested in learning about how folks, especially developers are using technology and some of the troubles that they're having with technologies as well, right? I've been in the industry for a good part of 27 years now. So, I do have quite a bit of experience and I like to kind of, you know, share some of my experience and knowledge with the community, right? Especially when they're struggling with things. And by the way, I'm also learning from everyone as well, right? So, it's not a one way street where I'm just like, you know, dumping data or my experience on the folks. I'm also learning from other people's experience. And with those learnings, I bring them back to CircleCI so that we can leverage that information in determining the next features that we're going to build so that we can add value instead of just, you know, building features just to build them. Anyway.

So, if anyone's interested in having conversations with me after this talk, you can hit me up on Twitter, my Twitter handle is at punk data. Generally, it's the easiest way to get ahold of me these days. So, please feel free to contact me and discuss whatever it is that you want to discuss. I'm open to all kinds of conversations. All right.

So, the first thing I want to talk about is testing, right? Because we leverage testing to basically understand how our applications are behaving. And also understand if they're behaving as we expect them to, right? So, we have to implement tests in order to understand and compare the results that we're expecting, right, from our applications.

And of course, right, we have to create test cases. And when we're comparing those results, we want to make sure that, you know, again, we're getting the results that we are expecting per the system requirements or the business requirements, right, that we're automating within our programs.

There are multiple types of testing, though, right? So, not all testing is equal, so to speak. And here's a nice list of some of the more common tests that we run as developers or that we develop as developers. The first I want to talk about is the unit test and smoke test, which are basically tests that are designed to be limited in scope, right? They're focused on specific functionality. And what we're doing there is trying to suss out any kind of common bugs or bugs that are common pattern or common practice bugs, right? So, things like, you know, is a variable's data type or the values that the data type has or the variable has, are they integers, string, right, flow, are they compatible within the framework? You know, those are the kind of things that we're testing. We're also testing maybe for okay 200s on a server. You know, things like that. They're very limited scope, focused on specific functionality and specific use cases. And also, they're designed to be super fast, right, like we don't want to be testing things at a comprehensive level, right? So, again, these unit tests and smoke tests are designed to be fast so that, again, you know, we're getting a nice, warm fuzzy around the code that we're building. And most of the time these tests are great for kind of, you know, ensuring that any kind of breaking changes are quickly identified and can be quickly mitigated.

So, after speaking about those fast, you know, limited in scope type testing, I wanted to talk about functional integration and regression tests. Again, these tests are larger in scope, meaning they are more comprehensive, they will touch a lot more of the code to ensure that you are getting the expected results. Generally, these types of tests are slower, right? Because they are more comprehensive, right? Because they are doing a lot more things. They may be connecting to other services that your application uses. One of the other kind of values, or actually, the disregard to one of the other factors is they are very expensive. And what I mean by that is they can cost, they can be expensive financially. They can also be expensive in regards to manpower, like developer or operations team type time, right? So, like, if you have these tests running, maybe someone is monitoring them because of, you know, while they are running because of some of the intricacies of these tests. And again, they tend to be slower, more comprehensive tests. And these are okay, right? So, these are things that are running I would say more of an end-to-end type of experience, right? So, that you're fully encompassing all the functionality within the application. Now, the reason why I bring these up is because I wanted to start talking about smoke test, which are essentially the tests that, again, are super fast, limited in scope. And I'll tell you a quick story before I get started. When I was a teenager back in the day working in a construction company during the summers, right, I would work with folks in construction. I actually got paired up with a plumber and the plumber was there, you know, basically putting a new plumbing into this new facility that we were working on. But when he was done, you know, soldering all the copper piping and making sure that, you know, the water pipes were complete, he ended up connecting this smoke apparatus, right, like a smoke machine type apparatus. And I was curious as to like, why he was doing that. So I asked him and he basically told me that, well, if I were to fill these pipes with water to test my, you know, connections and make sure they were sealed properly, it could potentially be a big mess because if he had a leak in some of his couplings, right, or they weren't soldered properly, there could be water spraying all over the place. So in essence, what he was doing was this smoke test, right? He was connecting the smoke machine apparatus, pumping smoke through it, and it was a clean, easy, quick way for him to figure out if there was any leaks in his connections.

And the way he could tell was obviously, if the smoke was seeping out of any of his connections, then he knew exactly where to go, right, to fix the problem. And that's what, I just wanted to share that with you because every time I see the word smoke test, it kind of reminds me of that experience as a teenager. And later on in my career, it kind of made that connection, right, which is pretty funny in my case.

So again, smoke tests help you with your preliminary testing, helps you reveal any types of failures, right, that are not expected. Again, super fast and very focused. Now again, it's essential because, right, it's giving you the opportunity to, you know, again verify that your code is functioning as planned or as expected. And it also helps you eliminate those more expensive, long-running comprehensive tests, right? You can run those at a later stage in your software development cycle.

So now that we kind of covered testing, smoke testing in particular, right, being quick, fast and focused, I wanted to talk about broken deployments. Now with broken deployments, right, we should be able to leverage continuous delivery which is the practice of build testing and delivering changes to code using automated tools. Within continuous delivery, we have this concept of continuous integration and continuous deployment, right, CICD, no news there, I think a lot of folks understand what that is. But with the practice of continuous integration, all you're doing there is basically collaborating around code with your organization's teams or your co-workers, right, or your team members.

And essentially what you're doing is merging all of the developer's working copies or changes to a shared code repository, so that you're able to collaborate around that code as a team, and also the individual developer can get quick feedback on the changes that they made, right. So when you push some code, and it's running through your CICD pipeline, you can do all types of testing, all types of verifications, all types of, you know, compilation, any action that you need to automate. And then you're able to kind of get that super fast feedback. So that, you know, you can actually go fix the problems that are identified within the pipeline. And then, you know, go back to working on the code that you're supposed to be working on the new stuff, right.

So with continuous deployment, it's just kind of, I would say, a follow up to your CICI with continuous integration processes. And essentially, what that is, is the practice of automatically deploying this new software to the environments that you're targeting, right. In this case, this image is showing that we're building a Docker image artifact, and then we're going to deploy it to Kubernetes cluster, which will then make it available to your active users. Now, we all generally work with happy path deployments, right. And this is basically where all the jobs successfully complete, right. So you have everything from, you know, writing the code, pushing it up to your code repository, and then the CICD pipeline, you know, kicking off and doing all the things that you've, you know, defined within your CICD pipeline config file. And then at the end of the day, just exposing your code or deploying your code to Kubernetes, and then exposing that code to your users so that they can use it within your services. Now, I can tell you that, you know, when you're having these deployments, some of the pre-deploy results, especially within CICD pipelines can be deceptive. Obviously, right, deployments always fail unexpectedly. I wouldn't say always, but they predominantly fail unexpectedly. And that's okay, right? That's kind of common practice. And in this diagram here, right, I'm showing you that, you know, my pipeline kind of went through all the stages. It deployed my docker image to Kubernetes, but once it deployed, there was no problem with the deployment. The application didn't maybe start up properly, or it's just not functioning as designed, which then impacts, right, the users being able to access this new functionality in our deployment.

So here are some of the common post-deployment failure reasons, right. This is a big one, defective configurations, right? How many times, I mean, I'm sure as a developer, we've all faced this, right, where, you know, something has changed in the target environment, right, but we as developers don't understand what that was, or are completely unaware of it. And we have to go back, right, and reconfigure the configurations for our application. So that's a big one.

The next one is an integration fault, which I consider an integration fault to be a situation where, like, maybe you're connecting into a third-party API, or third-party database, and maybe that service that you're connecting to is just down, right, maybe just had an issue that day, or there's something going on with the service that's basically out of your control, right, and it's the service that you're integrating with that you don't control and you are unable to fix.

The other deployment failure reason could be invalid credentials, authentication, this is actually one of the more common ones, right, so like maybe an API token has you know expired or it has been revoked, or maybe just have a, you know, wrong password and username, whatever the case may be, these are the three more common deployment failure reasons. So now that I've described kind of failed deployments and and especially within a CICD pipeline and the different reasons why things are failing, let's take a look at, you know, how we can leverage CICD and infrastructures code to help reduce the amount of failed deployments.

So you know in this day and age we're expected as developers and DevOps engineers or you know anybody pretty much in the technology field regarding code, we're expected to go fast confidently and reliably. That means you know we're writing code at record pace and then we're also deploying that code at the same time almost right.

So when we're going fast confidently reliably we want to be able to do that, you know, keep our release cycles up at a very good velocity, but we also want to be releasing reliable code or code that's of quality and not bugs.

So here's another example of, right, we saw earlier the failed production deployment. Whereas, you know, we're running a CICD pipeline up until the point where we're building our image for our production deployment to Kubernetes. And again, right, the application deploys in this in this diagram and then once it's deployed, the application is not functioning as expected.

So what I'm proposing here is, in this diagram is, you know, everything's up in the same up until, you know, let's say it's a it's a it's a pipeline that, you know, developers push some code, it's built the image and in this case, we're going to leverage infrastructure's code to create a Kubernetes target environment. Then we're going to deploy that application change right in the form of a container. And then we're going to smoke test those deployments right. And then once all that kind of is passed has passed my test, we're going to destroy everything right the Kubernetes cluster, we're going to destroy the the container services, all the things that were created in that create target environment set.

So what that gets you is basically the ability to know that your application has been deployed to the target environment, and then, you know, smoke tested. And then, right, you can destroy all that that environment because you don't need it, right. And again, this helps with the release management of of that of that product. So your release managers know like, hey, this change, this co-chain, these co-changes were packaged, deployed to a Kubernetes cluster, they were tested, works, and we can, you know, basically have a higher confidence in that deployment. So this is a diagram from the CircleCI dashboard, just shows you a quick way, right, and this is also in my demo. So but basically, what I'm showing you here is that you can run things in parallel. They're all connected, right, to a point where you have dependencies, especially when you're deploying to Kubernetes, you have to create a cluster to do it. And then finally, we're going to run some smoke tests and then destroy everything that we've created. So that right, we don't need that infrastructure and then your artifact in your pipeline is kind of, you know, verified and your application release management team just has that extra, you know, kind of factor or good feels in knowing that this was this has been run on a Kubernetes cluster.

So again, right, when we're implementing these things, we're going to deploy the application, you smoke test to test the application within its target environment. So we understand and know that we're seeing the results that we're expecting. It also helps with developing fast pipelines, that you know, you can smoke test again, all your releases, and you know, you can leverage infrastructure as code to create and destroy these target environments very easily. I would rec highly recommend that you start if you're not already doing so you know, researching infrastructure as code, there's plenty of tools out there to help you get started. But it's definitely something that I think all developers in the future will need some sort of competency in right, to understand how to get their code changes from you know, they're basically your laptop or code repositories out into the real world and into some real infrastructure. With infrastructure as code, also, right, not only getting that ability to kind of test your code in real environments, you can also manage and control the changes within your target environments using infrastructure as code.

So, again, right, this is a concept that's not going to go away. It's actually probably going to become even stronger as time goes on in the industry. So, just wanted to share the benefits of smoke testing before I jump into the demo.

Again, right, you're going to gain valuable insights into your deployments and your application releases. It's an easy way to expose any kind of missing bugs, right? So, like, if you're just deploying things or writing code and you're not testing, you're probably not catching any bugs. So, you know, try to look at using at least some unit level or smoke tests to kind of get you in the habit of catching these bugs before you're releasing them into the wild. And you're also verifying your application behavior, right, within those target environments where they're going to be deployed to. And finally, reject all failed software releases, right? They're great to know that you had issues, but obviously you don't want to be packaging those failed releases up into an artifact and then deploying that to Kubernetes.

So, now let's jump into the demo and show you how all these concepts kind of come together. So, I want to start the demo now. What I'm going to do is show you some code. And in this code, what I'm going to show you is the application. So, this is a simple, you know, Node.js static website. And it's basically just the example project I want to use. So, we're going to trigger my pipeline first. And then what we're going to do is in that pipeline is do a bunch of testing and then also deploy this application to a Kubernetes cluster as well as test the application within the Kubernetes cluster that we create using infrastructure as code. And then that will give us, right, the smoke test kind of good fields that we need to ensure that that release is as quality as possible, right? We've smoke tested that deployment inside of a Kubernetes cluster. So, let's go ahead and update this to version 6 of the application just to trigger our builds. I'm saving it. So, now we need to go ahead and trigger or at least commit this, right, to GitHub so that we can trigger our build. So, let's go and do a git status. Do a git status here. We're going to do a git commit, and then we're going to say commit AppJS. Give it a message, right, and then we're just going to say updated version number. Of course, we need to use a lowercase m. Once we have that, we're going to push this up, push this to GitHub. So we're pushing our changes to our shared repository, right, which is master. Then we're going to go ahead and jump into the CircleCI dashboard, which basically shows that we've kicked off some jobs here, right. As you can see if we drill into it, this is the CircleCI dashboard showing me my pipeline build, right, and as you can see here, I have four jobs running. The most important bits here are this scan and push Docker image, so it's building a Docker image based off of that application, and then the create Kubernetes cluster so that we can deploy, right, next step is to deploy that application and that Docker image to this Kubernetes cluster.

Once we do that, then the following test is going to be a smoke test of that Kubernetes deployment for that application. And if all of those things pass, we can then move on to a manual step that I have here, which will destroy that infrastructure, right? So this is a manual button. We kick it off and then we'll basically kick off a destroy job. Great, so now our deployment has been deployed using infrastructure as code and our pipeline. And what happened here was our smoke test ran, right? So we now have our smoke test job and basically what we're doing smoke testing our deployment. As you can see here in this in these results, we basically have this endpoint, which was created again by infrastructure as code and the DigitalOcean system. And we're going to go ahead and just test the applications actually up and running, right, in the docker container. And there's our version number six. We also have this this text that we're looking for in our smoke test. So if you look here, we got the OK, 200 response. And now we also have the assertion test, which basically is looking for this test or this text within the Web page, which we certainly have. Now, once we take a look back at our pipeline, you can see here that I have a button to destroy this. Now, again, normally in an automatic pipeline, you wouldn't have this manual step. I just have it here for demo purposes. But imagine this would be gone and then this would just kick off automatically once the smoke test is done. Again, this is basically a way for you to increase the confidence in your releases and basically test your deployment situations to whatever target environment that you're going to deploy to. So in this case, we deploy to Kubernetes. We built the application into a Docker image, we deployed that Docker image and then we ran smoke tests once the application was deployed. We know that this application will run pretty smoothly within Kubernetes and that's basically the point I'm trying to make here.

All right, so that's the end of my talk in the demo. I hope you've all learned something and I'm definitely interested in learning what you have, some comments or some questions for me, so let's jump into the Q&A session.

Thank you, thank you. Glad to be here. I love the hat. Excellent. Let's first have a look at the results. Your question was do you test your code release deployments in the environment you're targeting? Good thing to see here is that most people said yes. Yeah, that's excellent. Very unusual, but I'll take it. What would you tell to people who said not sure? So, I don't think, you know, every team is in a different stage, right? Of kind of just forming maybe, or a maturity, I should say, right? So at least these results will vary amongst teams, but I think if you're a, I would say, more mature team, this should be a priority for you, right? And it helps because you understand that your applications are wrapped up in an artifact. And then right when you deploy things, like I said in my talk, you know, the problem isn't that it deployed to the environment. The issues are found out later that the application is not behaving as intended. And it could be a multitude of different things happening. Right? But at the end of the day, if you can get that into the automation for maybe a developer at a certain stage, right, I wouldn't say if you're in the development stage or the MVP process of building something. Don't introduce something like this into that. But if you're going to go like to promote it into another stage of your pipeline, maybe QA or something beyond development, then yeah. Leverage some of these tactics to ensure that, right, you know, this Docker image worked on a Kubernetes cluster based off of the configuration we're going to use later. And it just kicks off a box. Okay. Yeah. Fair point. So let's actually go ahead and do some questions we have from the audience here. Sure. Yeah. So we have a bunch of questions coming in. Let's first answer the question from Dennis. Dennis says, What is a good level of death for a smoke test? I think about it as a key user flows versus visit every page without crashing. What is your definition for a smoke test in that case? Yeah, I mean, you know, that's going to mean different things to different folks. So obviously, right, take, I would say, get the identify the patterns that are very important and critical, right? So like, is the is the application returning and okay, 200, if it's a service or a website, right? Is it you can do some data tests to write to confirm like, when you push, request some data from your API, do some posts, right, ensure that, you know, the data is being received, you know, things like that simple, quick things.

But they're critical, because if you can't do any of those things, after deployment, it's trash, right, like you've got a tainted deployed. So that's the realm and you can go as deep as you want. But I would say, stay within the lines of, you know, keeping your tests to a minimum, like maybe no more than I would say, two to three minutes. So there's a time, you know, factor for me. I like to keep my jobs, all of my jobs under 60 seconds, if possible. I know it's not realistic. But that's my personal goal for any kind of job within my CDC pipeline. So if you're running that smoke test, you know, try to make it as quick as possible. Because you have extended, you should have more extensive testing later, you know, when the developers are not involved, right. The operations teams segment can be, you know, like QA type things, you know, whatever.