Canaries in the CloudFront

Hi, everybody. Thanks for joining. My name's Vijay Ramesh. I'm a VP of application development at Demandbase. And today I wanted to talk to you a little bit about something I like to call canaries in the cloud front.

Before we jump in, we only got about 10 minutes today and that's not a lot of time. But in a way it is. There's a lot of content out there. There's a lot of things to learn, a lot of things to dig into and not that much time in any given day. So let me cut to the chase first and talk a little bit about what am I going to learn if you spend the next nine and a half minutes here listening to my talk.

So number one, I want to talk about what it means to design your frontend CI CD systems in a way that enable your teams to move faster with less risk. And as a part of this, what does it mean to think about your content delivery network as an integral part of your application architecture? And then finally, I want to touch a bit on how the type of business that you run, the type of product you make, the type of customers and market that you serve, how that can help determine what shift left could mean for your frontend teams.

And just briefly, like I said, so I'm a VP of application development at Demandbase. Over 20 years now, full stack web app type experience, big focus on building rich applications atop of big data and ML platforms, and today at Demandbase I lead three of our application engineering groups.

So before we start, let's actually go maybe one step from before we start, and an architecture diagram like this might be familiar to people who have been doing this for a long time, right? So think in the pre-CDN days, maybe even in the pre-single-page app days, it would be common to have some monolithic system like a Rails app, that app is going to go and be fronted by some sort of load balancer, and requests from a web client are going to hit the load balancer and go back to the backend, and the backend is going to serve up its own frontend. So let's go maybe one step forward here and introduce a CDN, right? So at Demandbase we're using AWS, so we use CloudFront here, and what do we get when we go and introduce a CDN like this? The biggest bang for the buck I think is this idea of edge caching, so if Demandbase is building up its React app and storing it off in US East 1, say in Virginia, meanwhile me as a user coming from Berkeley, California, I want to be able to fetch those static assets from an edge cache that is close to my location, and CloudFront enables this pretty natively, so as a user coming in from California, I'm going to go hit the edge cache say in US West Maybe we've already got the latest build there so I don't even need to fall on back down to the origin, or in the case I need to, the origin is going to go and warm up the cache on the edge so the next user, say coming from San Francisco, gets the full application. And already here, you need to start thinking about, well how does introducing this CDN technology impact your CI, CD strategy. In this case, right, you need to make sure that you're going to invalidate the cache as you deploy new builds, such that a user coming, say from one of these West Coast edges, is able to pick up the latest build. And then also, let's talk a little bit about what are the pre-reqs for the CI-CD system that we started with, when we really wanted to shift left. So number one, dev and staging, right, full pre-production environments, and a way locally for a developer to go run the full frontend application but talking against, say, that staging or that dev backend. Number two, we have a large suite of Selenium integration tests, and importantly these only really will run in the staging environment, due to a general dependency on a stable backend. In the dev environment, backend teams might be testing out feature branches and things like that, and also due to a general dependency on a certain quality and volume of data that's only actually available in our staged environment. Then also, right, we've got a CI-CD system. As I showed, we're using GitLab here. But imagine a system that can go run your scans, run your security checks, run whatever lint things you need to do, and then also run all of your unit tests whenever a developer makes a change set into your develop branch. And then at Demandbase, we're doing a weekly release branch, where, say, every Monday we cut a release candidate from develop, deploy that up to staging, spend Tuesday and Wednesday running a suite of integration, regression, load tests, manual validation, all that, and then deploy out every Thursday to production.

So with that in mind, let's start to shift left. So our first goal was, we wanted to allow UX, say, designers on the team or PMs or even other developers on the team, to be able to really fully review a branch change before merging it in. But of course, right, our designers and our PMs often don't know how to use Git. They don't want to learn how to use it. Even for a developer, we don't want you to have to stash your changes, check out your peers branch, load that locally and all that just to see what all is going on. So let's start to shift left. And here's an example of the GitLab pipeline that we ended up building up. So imagine I've got my feature branch, I'm making it an MR to develop. Like I said, we're going to go and run all of these checks and scans and unit tests and all that. But then importantly, at the end of this process, we're going to go build up the full application and push it off to our staging origin, but in this specific path, right off into those branches and then a named path. And then in order to support this from the CDN side of things, we introduced Lambda at Edge, which allows CloudFront to run some serverless function, either in the request coming in or the response going out. And this allowed us to do some simple branch rewrite. So if I'm trying to go and access a particular branch deploy CloudFront, the Lambda at Edge will catch this, will rewrite the path, and it'll go load the proper origin. And then finally, right, this wires up really nicely with GitLab. So our designers, our PMs can just go look at an MR, click the view app button, and they're off to the races.

So our designers, our PMs can just go look at an MR, click the view app button, and they're off to the races. All right. So great progress, but we want to keep shifting left. And like I mentioned, the name of the talk is Canaries in the CloudFront, so let's talk canaries now.

So we wanted to support canary releases of our latest develop into the staging environment so that we could run this full integration regression suite, load suite, all those test suites ahead of ever even cutting a release candidate. So again, right, our GitLab pipeline can support this. So now after my changes merged to develop again, we'll run scans. We'll run tests. We'll build libraries, all of that. We'll go auto deploy the build out to the DEV environment, but then also we're going to go and build up this canary package and we're going to push it off to the staging environment again, in some canary path. And doing this then required us to be a little more complex with the way we're handling this Lambda at edge in front of our CloudFront, namely, because we wanted the ability to move automation users back and forth from canary to the current build. We introduced a cookie schema and introduced this into the edge cache key such that a user requesting a web.demandbase.com will either actually load the canary build or the current build based on the presence of these cookies. And then in our React app, we wanted to add a little flexibility around this. So we're using launch darkly pretty extensively for feature flag management at demand base. We continue to do that here. But then we also wanted to add some method where a developer could easily flip back and forth from the current build to the canary build. And so the React bootstrap ends up doing some checks against the feature flags, against cookie state, and determining, does it need to go and reload from CloudFront, which is going to enable the edge cache to check for those cookies and load the proper build.

So we've got canaries now all the way up to staging. We're using them to test the build ahead of ever even cutting a release branch. So now which way do we go? Which way is left from here if we want to keep shifting left? Well number one, right, we originally thought, let's go and push this canary deploy out to production. So similar to what we do for a lot of our backend services at Demandbase, this will enable us to go say deploy the green build out, run some subset of traffic against it, understand is performing correctly, and then cut all traffic over. However this is where I think it's important to think about. Who are your customers and what is the market that you're in? Demandbase is in B2B SaaS, right? And so in particular, this becomes a little more complicated, because we always want to avoid multiple users within the same customer tenant experiencing different versions of the application. And like I said, right, we use feature flagging heavily, but for the most part, this is always tenant-specific. Right? A particular customer might get a alpha feature turned on, but it's quite rare to turn things on for individual users and have them have a different experience than other users in that same customer tenant. And then finally, we do have this large internal user pool, Demandbase employees who are both using our platform for Demandbase's own marketing and sales and advertising, but also Demandbase employees that are supporting our customers. And as we kind of talk to the PM team, we talk to some of the implementation and customer support teams, we realize that these workflows are really mission critical. And so it's not a good candidate pool to say, hey, we're going to push you off into a new build and then we'll cut everybody over once we're happy. So then we thought about, well, where should we go from here? You know, what does shift left mean in this case? So for us, this meant targeted integration tests against our branched employees.

So then we thought about, well, where should we go from here? You know, what does shift left mean in this case? So for us, this meant targeted integration tests against our branched employees. Like I mentioned, right, we have this full rich integration, load, regression, all these test suites and they're slow and they're somewhat expensive and they require stable data and a stable backend to be running against. But so now with our initial shift left, we're going to go run all those suites nightly against the latest develop. But already, right, it's too late. If there's a problem, if there's a bug, well the code has already been committed. What we would really like is to find those issues on the merge request itself before it ever even hits the develop branch.