Experimentation Driven Development

Hi, everybody. I'm Graham, and I'm very excited to give you this lightning talk today on experimentation-driven development.

So, I'm Graham McNicol, I'm the co-founder of GrowthBook. GrowthBook is the most popular open source feature flagging and A-B testing platform. And the goal of today's talk is to help you understand the fundamentals of what we call experimentation-driven development.

So, we're going to take a look at how we develop products today, and then we're going to take a look at A-B testing and then some ways we can combine them together to make better products. This is an animal-based presentation. That's my dog Nellie, and so if you don't like what I'm talking about, at least you get some pictures of pretty animals.

So, let's take a look at how we build products currently. So, if you're like me, you use some kind of agile system. This is Scrum. If you're unlucky, you get to use some process like this. I feel very sorry for you if you do. But if you look at the whole landscape of different agile processes, they're all missing one thing, which is, what does done mean, right? So, for some processes, you might have something like, you know, done means is a defect-free, or did we accept the user stories, or does the product owner like it? And so, typically, what we mean by done is, like, shipped, right? We shipped it to production, and we are done. But how do we know that thing we shipped actually worked, right? Or bigger picture, like, what if we were wrong for building it in the first place? Like, what if the product we shipped actually hurt our business? And so, this is the problem we're really trying to solve with experimentation-driven development, because we really want to focus on shipping the features that positively impact our users and our metrics, and hopefully both.

All right, so with that in mind, let's take a look at A-B testing. So, the definition of A-B testing that I really like is a controlled way of measuring the impact of changes on real users. And so, from a high level, what that looks like is, first, you start with a hypothesis, some idea that you want to test. And then, you assign, usually randomly, users into different groups. You then expose those groups to the different variants. Then, you track how they behaved in your application or product. And then, you use statistics to figure out, was that change we detected statistically significant or not?

So, with that in mind, let's take a look at some examples of A-B testing in the real world. Now, these are web-based ones, but you can A-B test in any product that's digital at all. So, this is from Airbnb, and they wanted to increase conversion rates and decrease the cancellation rates by being more clear about the cancellation policies by showing this timeline. So, take a moment to think about if you think this change was successful or unsuccessful in improving those metrics. Well, it turns out this test lost. So, let's take another look at another example. So, this is from Netflix, and the control version has just one button that says try now, and then they test a new variant that has an email address and then the button. So, go to a registration page after that. So, take a moment to think if that won, and that one actually did. And that kind of goes against some of the tenants of UI where we think that you shouldn't show form fields wherever possible.

So, with that in mind, how often do we think A-B tests win? Like, on average, industry-wide. So, what I mean by win is like how often does an A-B test that we launch, how often is it successful in moving the metrics with which it was designed to improve? And it turns out that actually only one-third of the tests that we run are successful in moving a metric that we meant it to move. That means that two-thirds of the time is actually unsuccessful or hurts our product, right? And it's a little bit worse than that because that one-third is actually a high estimate. It turns out industry-wide, it's usually a little bit lower, and depending on how optimized your product is, it gets increasingly hard to improve it. One interesting thing to take away from this is like nobody ships a product or builds a product that they don't think will win, right? So, that one-third success rate is our best effort. We're building things we think will work and we're still unsuccessful two-thirds of the time. It just turns out we are really not great at predicting what our users want. And the other takeaway from here is that without testing, you're guessing, because there's really no causal way to figure out the impact of what you're doing without running a controlled experiment.

Now, I hear some common objections all the time about why companies don't run experiments. And so, let's take a quick look at some of them now. The first one is, what do we do? We look at before and after data and we just squint at it. So, let's give a hypothetical what this might look like. So, we have some conversion rate over time and we roll out a new feature here and then we track its performance over time.

So, we have some conversion rate over time and we roll out a new feature here and then we track its performance over time. We see this graph, right? So, do we think this is successful? Did it move the metric? Well, the metric did go up and to the right. So, maybe, but the actual answer is we have no idea because you can't tell unless you run a controlled experiment because you have no counterfactuals, right? So, like if we had not run that test, the graph could have looked like this. And here's an actual example from Airbnb where they rolled out a new feature and then ended up rolling it back. But if you just look at aggregate, it's really hard to tell causally what happened because there's just so much noise in your signal. People are deploying other features all the time. You have changes in traffic and holidays and all sorts of stuff that's affecting it. And A-B testing is really designed to control for those variants.

The other example we hear as to why they don't run experimentation is, well, we don't need to because we did user testing. Well, if you're familiar with user testing, usually, they're testing on five, maybe even 10 if you're lucky, tests. It's just a really small sample size to try to figure out what you want and there's all kinds of other biases that I won't time to go into. And usually, if you're okay running user tests to start with, you're okay running bigger tests with more people afterwards. So, you should be okay with that.

Let's take a look at the better way to build products. Get no points for guessing that we believe that integrating A-B testing into your process is how you do that. Each product should define what success looks like before you start building it. The process we like to do is, basically, before you start building anything, you sit down and you decide, what's the hypothesis? Why are we building this thing? What do we expect it to do? And then, what actions or behaviors would demonstrate that the hypothesis is correct? And then, what metrics would we need to track for that? It could be a signup rate or registration or purchase event. And then, the final step here is, what is the smallest thing we can build to test this hypothesis to see if it's correct? Because you want to get to signal as fast as possible so you don't waste time building things that don't work. We call this HAM. There's Jonham. And then, if you have the traffic to run that experiment, then you should do that. And so, what the adjusted process could look like is, basically, in your planning of your projects, you kind of come up with the hypothesis, the success criteria, and then scope it down as small as possible. And then, do your regular product process. But then, instead of just shipping it to production to 100% everybody, you ship it as an A-B test. And then, you decide, based on the results of that test, once you get to a significant power, if you should roll it out or roll it back. And then, do a review, like, present the experiments to your team and kind of iterate from there. And the good news is that this is super easy, particularly if you're using feature flags, right? So, as developers, when we ship something to production, like, the main goal, at least initially, is making sure we didn't break anything, right? That's the low bar for what we just shipped out. And feature flags is really helpful with that because you can actually wrap your change in just a conditional bit of logic. It's like, if it's on, show this code.

It's like, if it's on, show this code. If not, don't run that code. And so, it adds a lot of safety. Here's an example from Growthbook. This is the top one showing JavaScript and the bottom one showing React. And so, feature flags adds a lot of safety. Because it allows you to separate the deployment of your code from the release of that feature. And then, if something goes wrong, you can just turn off the flag really quickly. Just hit the toggle and it's off in milliseconds.

The cool thing is now that you're conditionally releasing that feature to your users, one of the ways you can conditionally release it via an A-B test. Here's that same flag that we've just added two different rules. First one says, if they're a beta user, turn it on for them. And the second one says, if you're out of that group, then we're going to run this A-B test to 50% of our audience. And within that 50%, there's half-each within it, right? And so, basically, for that one line of code, now we have a full A-B test running. And that allows us to get results like this. So, this is, on the left side, our metrics that we chose for this experiment. And we have the column showing the control version, and the new variant version, and then all the statistics on the right. It's pretty complicated, but basically, this test we declare a winner because it's significant.

The cool thing about A-B testing is that it replaces a lot of differences of opinion, right? Because you decided that success criteria up ahead of time, you have a framework for deciding the project's fate that is not reliant on your biases or personal opinions. The other thing it does is failing is not failure, right? So, if you roll something out and you learn that it didn't work, you can actually like celebrate that, right? Because not shipping a negative thing is good, right? You saved your business some money and you learned fast.

So, in conclusion, what I want you to take away from this talk is that done should not be shipped, right? Shipping a thing does not mean it worked or was successful. And that you really need to do a hypothesis and test it to see if it actually was successful in doing what it was meant to do. And it's really, you should be doing this on every project if you have the traffic to do it. Because if you're not testing, you're really guessing. And experimentation driven development is easier than you think. There's a lot of tools out there that make this really, really easy. And it should really add zero additional effort and zero additional time to how you're shipping and building products today. Well, that's all we have time for. Thanks so much for listening. If you have any questions, please feel free to reach out. My information is there. Thanks so much. Bye.