Many companies are going all-in AWS and thus adopting their complete CodeSuite for their CI/CD processes. However, while CodePipeline is the platform for this process, it may not be the most user-friendly. In a Monorepo setup, it's typical to create multiple CI/CD pipelines for each package. However, there are several caveats to be aware of. For instance, you may encounter scenarios where multiple pipelines get triggered even if you just modified one file, or you may question the need to create multiple branches for each pipeline. In this talk, we provide valuable tips for building a sophisticated CodePipeline using CDK in a Monorepo environment. The techniques discussed in this talk are also transferrable to other CI/CD tools.
Building a sophisticated CodePipeline with CDK in a Monorepo Setup
Video Summary and Transcription
This talk has been presented at DevOps.js Conf 2024, check out the latest edition of this JavaScript Conference.
FAQ
AWS CDK was chosen because it allows the use of TypeScript, which is the same language used by the developers, and it provides a nice abstraction that developers can easily understand.
Each pipeline contains stages, and each stage contains actions. Actions result in outputs called artifacts, which can be stored in an S3 bucket.
CodeCommit is used as the source repository where the monorepo is hosted. It triggers the pipeline whenever changes are made to the main branch.
Deployments to different AWS accounts are handled by looping over all accounts and making use of CDK's abstractions. A method called createPipelineProject is used to simplify this process.
Frontend and backend deployments are managed by creating separate pipelines for each. The appropriate pipeline is triggered based on the changes detected in the respective folders.
The Lambda function uses the CodeCommit get differences command to check what files have been changed. It then triggers the appropriate pipeline based on predefined paths in an object that maps file changes to pipelines.
The project uses AWS CDK, CodeCommit, CodePipeline, EventBridge, AWS Lambda, and the JavaScript SDK for CodeCommit and CodePipeline.
The createPipelineProject method is used to create pipelines in a simplified manner by abstracting repetitive code, making the process of creating new pipelines easier.
The primary goal is to implement a fully automated CI/CD pipeline on AWS for a monorepo containing a backend and frontend, both written in TypeScript.
EventBridge and AWS Lambda are used to trigger different pipelines based on changes detected in the code repository. EventBridge captures events from AWS services, and Lambda functions are used to determine which pipeline to trigger based on the changes.
1. Introduction to AWS CDK and Pipeline Creation#
Imagine starting as an AWS DevOps engineer in a small company. Your boss wants an all-in AWS approach, with CICD entirely on AWS for automated deployments. Use AWS CDK for creating a pipeline, stages, and actions. Import code from CodeCommit, create stages with CodeCommit source action and code bit action. Next, deploy on different AWS accounts.
Hi. Imagine you started as an AWS DevOps engineer in a small company and your boss wants us to go all-in AWS. Of course, the CICD should be entirely on AWS for automated deployments. All the infrastructure should be put into code. And we are using a monorepo containing a backend and frontend both in TypeScript.
Great. You have been working with GitLab CI or GitHub Actions where you just need to define the Yammer file and it creates on the platform the pipeline for you. On AWS, it's a bit different. Each resource you have to create by yourself. So you decided to use AWS CDK because you can use TypeScript. You can use the same tooling as the developers. You are able to put the code inside their monorepo. And you have also a nice abstraction, which also the developer may understand.
Let's get your hands dirty. So you start by creating a pipeline using this abstraction new pipeline. And each code pipeline contains stages. And a stage contains actions. The actions result and outputs artifact. This artifact can be stored into an S3 bucket. So you import the code from CodeCommit where the monorepo is hosted. And you can use the to create your first stage by using the add stage. And here you create the CodeCommit source action. So this will be triggered whenever something happening on the main branch. And you create a second stage by using the code bit action. And in there, you use a code bit project. The code bit project you can take, you can define each phases. In this case, I use the build phase, where I want to install linting and test unit. And the result of this action can be then stored inside this folder in this case. And now the next stages are the deployment stages. So you want to deploy on different accounts on different AWS account.
2. Handling Multiple Pipelines and Manual Approval#
Create multiple pipelines for different accounts using CDK's abstraction. Use a custom method for pipeline creation, assuming roles and deploying with CDK. Handle manual approval for staging and production environments. Create separate pipelines for front end and back end deployments. Use environment variables to trigger the appropriate pipeline.
So you create the loop over all accounts. And you can also make use of CDKs abstraction. So you can create your own method, in this case, the create pipeline project. So instead of using this code, again, you can actually make it like easier by abstracting that and putting it in your own method, and then just make use of each phase. First, you assume the role, and then you use the CDK deploy command. After the deployment, you can run integration tests. You can also use Canary operations. In the staging and production environment, we have the manual approval. This manual approval action can be merged together with the corporate action. Only if it's not on default, otherwise, we just skip the manual approval action. But each time there's a change on the front end, the back end will also be deployed. To handle that, you need to create multiple pipelines. Now you have a pipeline for the front end and another pipeline for the back end. This could be the code, how it looks like. You could find multiple pipelines, and you can handle it by using environment bio.
3. Triggering Pipelines with EventBridge and Lambda#
Learn how to trigger pipelines with EventBridge and Lambda. Each AWS service emits an event that can be caught using EventBridge. Use a single Lambda function to handle events and trigger the appropriate pipelines. Define pipelines and their triggers inside the Lambda function. Handle file changes and trigger pipelines accordingly. Use CDK and abstraction methods for efficient pipeline creation. Utilize EventBridge to trigger pipelines based on events like pull requests. Create sophisticated pipelines for different scenarios.
But the real question will be when to trigger which pipeline. And the solution for that using EventBridge and AWS Lambda. So you have to know that every AWS service emits an event. In this case, whenever something happening on the code repository, then we want to catch this event. We want to catch it by using EventBridge. And we want to tell the pipeline, hey, use EventBridge for emitting this event. And this event will be then forwarded to a Lambda function. We define the Lambda in here. And with this code, we're making sure we only have one Lambda. Otherwise, we would create multiple Lambdas.
But with this code, we're making sure we only have this Lambda one time. And the Lambda makes use of the code commit get differences. So it checks what file has been changed, and then it triggers the pipeline accordingly. So the inside the Lambda, we define each pipeline. So we create an object where we have the front end, the back end front end stack pipeline and the back end stack pipeline. And we also define when this pipeline should be triggered. And here we define the path of each pipeline.
So in this case, whenever a file changes in the back end folder, or we change something in the back end stack, then the back end stack pipeline should be triggered. We also have the ability to have common folder. So let's say you have like a package, which is shared between the back end and the front end. And that we can use the spread operator in order to trigger all the pipelines. And we make use of the JavaScript SDK for code commit and code pipeline. And the handler uses the get differences command to get all the file changes. And then we loop over each pipeline and loop over each pipeline path to check if the filename is part of the change. And if so, we start to execute the code pipeline.
So what we have seen here is we are able to use CDK to make code pipeline less suck. We can also do something like platform engineering to make use of abstraction so that anytime you create a new pipeline, you can make use of certain methods like the create pipeline project method. We can use event rich to trigger each pipeline accordingly. So we can also say whenever there's a new pull request, we trigger then the pipeline, or there are many other events, you can check it, you can find it online. And with that, you are able to create sophisticated pipeline. Thank you. You'll find all the information in here. You can also find the repository. And thanks for your time.
Comments