Black Friday: Would You Choose the Right Performance Test?

Hello, everyone. I want to start with a simple question. Has anyone here ever had a system that passed all tests and then crashed in production? I guess that yes. And today I'm going to tell you about exactly that scenario. About teams that have no major bugs before release, no failed deployments, no red flags before release, and then Black Friday comes. This talk is about why they failed and it wasn't a bug in the code.

Before we start our topic, let me introduce myself. My name is Oleg Koren and I'm a senior performance QA engineer at Provectus. My role focuses on identifying performance bottlenecks of the application, analyzing system behavior under the load, and helping teams to build scalable, reliable applications. I also teach performance testing. I have a course on Udemy, which is called Performance Testing Fundamentals from Basics to Hands-On. And if you are new to this topic or want to go deeper after this talk, you can find it there. I also write about performance testing, about lower testing, about production lessons on Dev Community and Medium. You can subscribe for me if this direction resonates with you. And also you can find me on LinkedIn. I post frequently about performance testing. Feel free to connect.

All right, let's get started. A few months ago I posted a poll in LinkedIn software testing group. And the question was simple. You have e-commerce platform and your platform must handle Black Friday traffic. Spikes of four to ten times within minutes. Which test will you run? Four options were available. First one is spike testing, then scalability testing, volume testing and endurance testing. Around 230 engineers answered to this question. 55% of them selected spike testing, which is actually the correct answer. And that is great. But that means nearly half of the engineers choose something else. 45% of engineers, when given a real Black Friday scenario, would skip very important tests that may cause production issues. So what is spike testing? Spike testing is not about just a lot of load.

There are three things that define this type of testing. First, the load is sudden. So it's not gradual, it's not linear. And system has zero time to warm up. You go from the base load to extreme load in seconds. Second, we observed what actually happens during the spike. Because this is where things get interesting. So does the application degrade gracefully? Or does everything fall apart at once? Maybe one service gets overloaded, then another starts timing out. Then retries increase the traffic even more. And suddenly a single bottlenecks turn into a full production incident. Third, and this is the part that most teams skip, it's to measure recovery. You are looking at what happens after spike drops. How fast does your system return to the normal state? Does it safe heal or does it stay degraded during the next 20 minutes? All three matters. If you're watching only whether your system survives during the spike, you're actually testing only the half of the story.

Let's talk a bit why other testing types don't fit to the Black Friday scenario. Volume testing, I hope that you know, it's about data. About millions of records, huge payloads and database at scale. And Black Friday is not a data problem. It's concurrency problem and those are different things. Next is endurance testing. We run this test for many hours and the main aim is to detect memory leaks and detect system degradation. But Black Friday is not gradual, it's not about memory leaks. It's a wall of traffic that's arriving in seconds or minutes. Next is scalability testing. We check whether our system scales up and scales down properly, when the number of users changes gradually. And the gradually actually is a keyword here. But Black Friday, it's not about gradual load. It's about load that hits immediately. Spike testing is the only one testing type that replicates what actually happens at the midnight of the Black Friday. Everything else can give you a false sense of confidence.

That doesn't make those tests useless, they simply check different things. Let's go down to the real scenario. Before Black Friday, the team was confident, because they run all of three test types, volume, endurance and scalability tests and everything passed. They got green reports, management was satisfied, the system was healthy. After that sales go live. User arrived not over hours and not over minutes. In over 90 seconds traffic increased to six times in comparison with the normal load. And I guess you understand what's happened next. System starts throwing errors, response time increased and checkout stop working for the half of the users.

Call engineers woke up and tried to fix those issues, but it was already too late. Users were disappointed and the customer lost a lot of revenue. Application actually wasn't able to handle such a huge spike in load, which has caused to a lot of errors. All of this happened in the first 90 seconds. An important test was absent from the scope of activities and the team actually wasn't ready for the Black Friday.

Let's talk about reasons why application failed in production during the Black Friday. When spike hits the cache wasn't warmed up yet. Every single request goes straight to the database. And the database becomes your first bottleneck. Next is connection pool limits. Under normal load traffic, the database connection pools working fine, but during the spike it can be exhausted in seconds. Requests are queuing and eventually they time out.

Next one is thread pool saturation. Application servers eventually run out of threads to handle incoming requests. From the users perspective, application just freezes and requests keep loading forever. Next is database locks. Recovery counters are shared resources, so during the spike thousands of concurrent writes compete for the same roles. As a result, the database starts locking those roles, transactions wait for each other and deadlocks begin to escape. Next is auto-scaling delays. Revisioning of new instances can take up to several minutes, but spike kills you even faster than 90 seconds. New instances arrive only after actual damage is already done. As a result, volume, endurance, scalability tests often miss these problems and they only appear on the sudden massive traffic spike.

Let's take a look at this graph. How does spike test look in practice? Orange line represents for us concurrent users. In the beginning we ramping up our users to our base load. In this case we reach 250 users and we hold that load during some period of the time. Then we simulate a sudden jump. It is our spike. We added an additional 1000 users and we hold that load and back down to the baseline. Green line is response time. It is flat at the baseload, approximately all requests completed under 3 seconds. Then it climbs sharply during the spike and we reach almost 20 seconds. And if it comes back after the spike our system recovers. If it stayed elevated we have a problem. Red line represents error rate. Near zero at the baseload and during the spike errors appears more and more. And the question is, do they normalize after load drops and how do they impact user actions? If errors block major user actions, we should investigate those errors. This graph tells us not just where the system survived, but how it behaved and how fast it recovered.

Let's talk about how to set up a good spike test. Step one we should establish our baseload. This purpose I recommend you to use Google Analytics tool, Pando or any other tools that your team have in place.

This purpose I recommend you to use Google Analytics tool, Pando or any other tools that your team have in place. We need to identify our peak hours, peak number of users during the day. Our baseline will become our foundation for the entire test. We should also build realistic traffic distribution model. Usually around 60% of users simply browse the products or perform the search. About 25% of users add products to the carts. And the smallest part, only around like 50% actually proceed to check out and complete fully the purchase.

Step two is to defining the spike magnitude. For Black Friday scenarios, the target is usually between 4 and 10 times of our base traffic. Typical ranges look like this. 4 times it's moderate Black Friday search, 6 to 8 times it's about typical e-commerce events, 10 times it's our worst case scenario. Now let's look at how this load model can be built using K6 test example. So actually we're starting from the ramping up our users to the base load. We define that our base load is 200 users. After that we hold that load during next 10 minutes. And only after this we provide additional 90 seconds to reach our spike.

We've multiplied our baseline by 6 and reached 1,200 users. Then we hold that spike during next 30 minutes. And we monitor during that our infrastructure resources. We also check whether errors appear and so on. After that we ramping down users for the next 90 seconds and then hold our base load for next 10 minutes to see whether our system is able to recover. Next step is to monitor application while test is running. While our test is running we should watch such metrics. Application pool saturation, thread queue depth, database logs, wait time, cache hit ratio, infrastructure resource usage and overall error rate.