When the Safety Net Catches Fire: Solving Feature Flag Problems at Scale

Hello, I am Ryan Feigenbaum. I am a Senior Developer Advocate at GrowthBook, an open source feature flagging and experimentation platform. Today, I'm going to talk to you about when feature flags catch fire, solving feature flag problems at scale. And let's start with the first slide here. Sorry, finding my cursor.

The $460 million if statement. So this is an interesting story back from 2012 from the Knight Capital Group, which reused an old feature flag unknowingly to them. And in the course of 45 minutes, they lost $460 million, which was more money than they had. And within a short amount of time, they then went out of business and were acquired. And this all happened because of poor feature flag management.

As interesting as that story is, it's not the story that I'm going to tell here. In fact, more I want to talk about how you won't be like Knight Capital Group and what you can do to avoid such catastrophe. And so some of the facts about feature flag management and why it's hard. So a big number research shows is that 73% of feature flags are never removed. And so they just accumulate in the code base tech debt, not good stuff.

This is estimated to cost a 10 dev team about $125,000 per year. So it's real tangible costs. And this comes from just having to navigate this old code base, figure out what things are doing. It's just a lot of loss of time. And reports show that about 95, 90% of dev teams struggle with flag debt. So it might seem simple, right, a flag that's on or off. It's just some toggles. How hard can it be? It's actually quite hard. And let me show you a bit of why it becomes so hard. So we have here for four kind of ways in which the complexity of your feature flag system will grow. And the simplest one to kind of understand and think about is just the flag logic. So we have a Boolean flag that's true or false. But over time, of course, that simple true or false is going to become a multivariate logic where you have maybe a different kind of experiments running on it. Progressive rollouts, all kinds of sophisticated targeting logic. And so that simple logic becomes very complex.

And this complexity extends to infrastructure where you might begin with just a simple Web app. But then it becomes a mobile app. And then it becomes maybe you're running on the back of a screen in an airplane or something. So your infrastructure becomes very complicated. And that then extends, again, to your users who might start all kind of the same. But then they have widely different devices and different network conditions, different groupings. It becomes much more complex.

And then, lastly, your team. So it might start with just one dev team. But then maybe there's two dev teams and infrastructure's involved and products involved and marketing's involved. And they all want to kind of know what's going on with feature flags. And so that simple complexity then blooms exponentially and becomes much more harder to reckon about.

What we've found is that this complexity, this difficulty can be kind of categorized into three different walls. And these are three walls. Every team hits with their feature flagging platform. The first one is visibility. The second is life cycle. And the third is architecture. So let's jump right in to the problem, the wall of visibility. Visibility, pretty simple to kind of think about. It's just the fact that you can't see what your flag's doing. And this comes in three forms.

And this comes in three forms. So, first, this is a pretty common scenario. You might find a flag like this in production. So we have a flag NCDFF123 temp. And so you might be thinking to yourself, what does this do? Why is it FF123? Is that a placeholder? Is that the real name? What is this? Is it what does the temp mean? Is it really temporary? The git blame says 2019. That was several years ago. Can I take this out? Can I not take this out? And this is this problem of visibility. Luckily, this problem is not too hard to solve. So the big solution here is just to think about naming your feature flag the same way that you think about naming your variables. So it should be as self-descriptive as possible. So instead of that very oblique flag NCD123 temp V2 that we talked about, or you just showed in the previous slide, it's much better to have something like flag mobile. So you have the team combined checkout flow. You know what the feature that it's targeting, and then a date. So it makes it very clear what that flag is doing. It's even better if your platform can enforce unique names for flags. We've seen all too often where a team will have, you know, new checkout flow, but then another team also has a flag called new checkout flow, and they're inadvertently turning each other's flags on and off. It happens. And so a good kind of approach here, too, like I said, is to use this team feature date pattern. It allows everyone to align on kind of naming the flags the same way. And even better, if your platform can introduce something like a regex enforcement that checks to make sure that everyone's naming things the same, keeps everyone on the same page, and makes it much easier when you return to a flag in the code base many years later, if it's still there. Hopefully not, but at least you know what it's doing. So that's one problem of visibility. Here is another problem of visibility. So what happened before was when you were in the code base and you were looking at the flag and didn't know what it does, here is another kind of way in which visibility is difficult. When you look at the feature flag in your platform, so in the UI, but you don't know where it is in code. So launch darkly developed this package called find code references. It's an open source package and it uses GitHub actions to scan your code base and show all the places where that feature flag exists. So this is a great tool to start using. We have a fork of this growth book does that doesn't even need GitHub actions to work. It's a Docker eyes container and you can run it locally as well, or he'll hook it up to a different platform.

And I'll show you what this looks like here. This is a screenshot from, from our platform where we have this code reps feature. And so this is a feature flag show reviews, not a great name, but it gives you exactly where in the code base, this feature flag appears. So there's two spots. And what's really nice is it gives you the link to the GitHub repo. So you can go and see this code up, you know, in person and see what's going on there. So code graphs is a really great way to approach this problem of visibility from the other side.

And then one last problem invisibility. So let's say you can see the code where you can see the feature flag in your code base. You can see the feature flag in your platform. All of that's great, but then this is the logic of it. So you have a kill switch. You have all these different conditions. It becomes very hard to kind of work out. Well, if I have this particular user, how is the feature flag going to render for them? You start to feel like Charlie day and it's not exactly clear really what's, what's going on, what's going on there. The way that we've found that can be really helpful for developers, for product people to, to understand this better is a feature we call simulation, which you can look for in your platform or build something like this.

And what it allows you to do is take your feature flag, but then see immediately how it's going to evaluate for different groups of users or for any kind of arbitrary properties. So let's say you are launching a new feature. You have these very complex rules and you want to see how that feature is going to evaluate. Maybe for beta users on mobile in the U S you could look at the simulate page simulation environment and see exactly how that's going to evaluate. This becomes super useful too, when you have a bunch of AB tests so you can kind of know which bucket those users would be in. It's also good. You can chest. Okay. It's supposed to be on for beta users, but not on for, for complex user yet. So you can do all these sorts of simulations and figure out how that flag's going to evaluate. So that's this problem of visibility, knowing what a flag does, knowing what it does from the perspective of the code base, but then also from the platform and then also understanding that complex logic. Let's move on now to the second wall. So this is life cycle. And what I mean here is the life cycle of a feature flag.

Ideally, this is how it goes. So the feature flag is created for some purpose and you want to gate a particular feature, turn it on or off. It then gets put into the code base and is active. And then at some point that feature flag is complete. So the feature is a hundred percent released for example, and you no longer need that feature flag. And then it's removed from the code base. And that is the death. So the, the ideal situation is that every feature flag has a birth certificate and a death certificate, but this is all too often, not actually what happens.

This is more along the lines of what you usually see. So in, in your code base, you'll have your flag use new DB connection, temp P2, but you'll have these comments that stack up, like remove this after the launch, but then another comment, well, wait, it's still needed. Something happened with that launch. And then, Oh, do not remove this. And then questions about what does it do? I need help. Right? This is very common. So common. In fact, that studies show that 87% of flag cleanup efforts fail. So this is a real problem with life cycle that flags that are no longer needed, accumulate in the code base and cause lots of problems.

This problem got so big that Uber wrote an academic paper on it. And in a very engineering fashion proposed or developed an open-source utility to fight it. So this paper is really interesting. I highly recommend checking it out. And what they found was basically across some of their code bases, there were 8,000 feature flags, many of which were stale, but it was hard of course at the scale, 8,000 flags to go through all of them and really figure it out.

So they developed a utility called Piranha. And what Piranha does is you give it the feature flag name, you give the owner of the feature flag and then what the feature flag is supposed to do. And it automatically scans the AST of the code base and finds those stale flags along with all of the dead code associated with them. Once it's done that it opens a PR on the repo that has all of that dead code in the code and the feature flags that are no longer needed removed. And then sets the original owner of the feature flag as a reviewer. So it's a pretty cool system. And what they found in the initial kind of usage of this is that they were able to remove 1,381 flags, which was about 70,000 lines of code, which is pretty cool with the caveat here that 20,000 lines were, so those were LGTM PRs. They were able to be merged immediately, but another 50,000 lines required some additional input from the owner of the flag. But this is a very cool tool that you can put into your CICD pipeline, and it is, again, open source and available to use.

Part of that utility is the ability to identify when a flag is stale. And what I mean there by flag is stale, it's pretty intuitive, but just a sense of a flag that doesn't need to be used anymore. This is a little bit of our algorithm that we use internally in our app to determine stale flag detection. So it's something you could use yourself or look for how your platform does it. We have this as open source, you can check this out. But the idea is that we look at flags that haven't been updated in two weeks, ones that have one-sided rules. This is very important. It's basically sending all traffic to the same place. It's not marked, never stale, right? Sometimes you'll have flags that aren't intentionally long running. This is not one of those. It has no draft revisions and no other features depend on it. There's nothing that's nested. And we show this with this little clock icon in the UI to indicate that this is a flag you need to check. And then, of course, another way that you can know about stale flags is if you have feature flag usage analytics that tell you, well, this flag hasn't been evaluated in a month, then that's something that is also a good indicator that that's a flag that's stale and should really be removed.

All right, so that was the life cycle. The big takeaway there, of course, again, is this stale flag management, probably the hardest part about feature flags at scale, to be honest. Let's now talk about the third wall, which is architecture. And architecture is going to have two parts. The first one will be testing, and then the second one will be kind of more infrastructure focused. So testing is really interesting with feature flags. And by interesting, I mean really hard. And that's because of the combinatorial explosion. What do I mean by combinatorial explosion? Well, with five Boolean flags, which, right, that's pretty simple. Five Boolean flags on or off, you already have 32 combinations. You move up to 20 flags that are just on off, that's over a million combinations. And the problem is, as well, if you have a million combinations in your code base, how are you going to test all of those combinations? Even if those took just one second per test, that would still be 12 days of continuous testing. It's just not possible. So the big problem is how to reckon with this fact that with more feature flags, especially feature flags that are more complex, imagine if they weren't Boolean flags, but had much more complex logic tied to them, how are you going to test it? Well, the truth of the matter is that you can't test every combination. It's just not physically possible. So the recommendations here are as follows. So the first one is to test critical paths. And what does that mean? Basically, those paths that you know are essential to the function of your app, those are the paths that you want to test first. This makes sense, but it's just something to keep in mind. The other side of that is also to test production states. So think about this is with those 20 flags, if there's a certain configuration on and off of each of those flags that is going to be most prevalent in production, that's obviously a configuration that you do want to test and make sure that that's part of, you know, the testing going forward. Another way that you can approach this, which is becoming more and more common for every feature flag provider, is to do something that we call safe rollouts, and it might have different names, but the idea here is that every feature that you release with a feature flag that you attach basically an EB test to it that allows you to check for some kind of failure rate. If that failure rate is triggered, then the feature is automatically rolled back. For example, how this would work, let's say you're testing a new checkout flow feature. You would say that if the checkout failure rate goes above some kind of percentage, then automatically roll back this feature. This allows you to have some peace of mind when you're using a feature flag with a new feature that you get this automatic roll back if things go south, and, of course, sometimes they will. Another great tool to look for in your feature flag platform is some kind of tool that allows you to look at the actual app and simulate different feature flag states, experiment states, and I'll show you a video of this. This becomes really useful for kind of looking at, well, if I have this kind of user or this sort of attribute, how's the flag actually going to perform in the app? If you have a tool that allows you to do this easy, it lets you catch a lot of bugs before production, or if you do have a bug in production, it makes it much easier to debug what's going on. And then two other easy ones, some minimized nesting and dependencies. You can get into a situation where one flag is going to depend on the outcome of another, and so you should follow just good principles for code where you're modularizing things, trying to minimize that nesting and dependency as much as possible.

It just, of course, is going to make testing easier. And then, again, just to really kind of nail this point, that removing stale flags is the biggest priority. So let me show you, though. This is an example here of that dev tools that I was talking about. So this is an app that we have, a coffee app, and it allows you to test these two variations of an experiment right on the page here, and so it's really nice to see that in action. I'll play it once more just so you can see again. So it's a Chrome extension up here that you open. It shows the experiments that are active on the page, and then it allows you to toggle between the variations. So you can imagine if you have different feature flags, you can look right in your app about how they're working. So this is a really nice way to do that.

All right, so let's switch now over to architecture. And I think this is really important for people who are building client-side apps, so specifically your JavaScript frameworks that are any kind of client rendering. This is a big problem that teams run into. So this is the traditional client-side flag evaluation. So imagine that you are running an A-B test with a feature flag, and so that default state is going to be first served to the user, and then based on whatever attributes are in that A-B test, it'll then render that variation. So there's going to be a flicker of content. I'm sure you've all seen it, where the baseline gets rendered to the variation, and that's really bad UX for the user because it's going to have some kind of jump on the page, and there's different ways you can mitigate that, but it's still not going to be ideal. And the other bad outcome there is that you're going to have event tracking drop-off, which is just part of a client-side app. Let me show you an example of this from our own website. So this is an A-B test that we're running.

And let me actually pause here. So this is our home page, and you can see we have this big image here of the app, a screenshot of it. And what we're testing, we have an A-B test running, where if we reduce the size of this, and let me show you actually just what this will be, and instead show some of this social proof, then that will increase conversions. But this is the defaults, and this is the variation. And what you'll see here when I play the video is that it jumps. So this is probably not too noticeable to a user, but it does make the whole page shift up. It has some of that jank that we hate. And so there's a better way to approach this issue. And this is something that I expect to become more and more common with feature flags in the future. Most platforms already offer something along these lines, and that is Edge flag evaluation in rendering. And this is really cool. So what happens is that instead of the rendering happening on the client's browser, it happens on the Edge. And so what that means is you get two big benefits here, is that you don't get that flicker because the fully rendered page, the hydrated page, gets sent directly to the user, so they never see that control state. So there's no jank, the experience is much better. And the other big benefit is that your event tracking is much more secure because it's happening from the Edge rather than from the client's machine, from the client's browser. So you get a whole lot of benefits. And when we did our testing, comparing that browser only setup to Edge rendering, the difference in the time from initial render to variant was just, like, incredibly sped up on the Edge. So you can see the numbers here, 530 milliseconds initial render, time to variant 390 milliseconds, you get that terrible flicker of content, whereas with Edge, you get 150 milliseconds, 150 milliseconds, no flicker. It's just the way to go to take a look.

All right, so that is architecture, the third wall that you'll hit. And what I really want you to take away from this is that the best time to start thinking about your feature flag management and what you're going to do is now, of course. If you're already wondering about it, it's probably time. And what we've found is that teams that address this flag debt early are five times more successful than teams that wait. It just makes sense that if you get ahead of it, it's going to be great. So here is what you can do in the coming weeks to kind of start thinking about this seriously. In your first week, find your flags and identify ones that are outdated or in critical paths. This is that visibility problem that I talked about. If you can just start taking a look at flags, figuring out the ones that are important that need to be looked at, that is a great first step. And the next week, start thinking about a naming convention that you can share with your organization, along with some of the best practices I talked about. That is a fantastic way to give a lot of visibility to your feature flags and help everyone know what's going on with them.

Week three, start setting up some of the systems that we talked about, like stale flag detection, automatically know when a flag is stale. Set up code graphs so you know that where your code, where your flags are living in your code base. And start thinking about the edge evaluation, whether that's something you want to do for your client's apps. And then finally, if you're getting really bold, try this automated cleanup for PRs, that Piranha tool that Uber developed. So you can really cut down on your stale flags. And then think about doing monthly flag reviews. This is something that's been really helpful.

So just like you might have a session where you look at your security vulnerabilities or maybe you look at the upgrades to different packages that you need to do, treat feature flags the same way, have a monthly review where you say, okay, well, it's been three months since this flag's been in the code base. The feature's been rolled out. Why is it still there? Do we still need it? Just starting to do these things will make a huge impact on your organization and get you better suited to scaling feature flags.

So I leave you at this crossroads. You could be on path A where you don't know what your flags are doing, and you're like that night capital group where you're hundreds of million dollars away from a disaster or you're minutes away from a disaster, or you can be on the well-managed path and be visible, governed, and fast. Thank you very much. Don't hesitate to reach out if you want to talk about feature flags or anything else in life. I'm Ryan at growthbook.io. Yeah, have a good one.