Black Friday: Would You Choose the Right Performance Test?

Hello, everyone. I want to start with a simple question. Has anyone here ever had a system that passed all tests and then crashed in production? I guess that yes. And today I'm going to tell you about exactly that scenario. About teams that have no major bugs before release, no failed deployments, no red flags before release, and then Black Friday comes. This talk is about why they failed and it wasn't a bug in the code.

Before we start our topic, let me introduce myself. My name is Oleg Koren, and I'm a senior performance QA engineer at Provectus. My role focuses on identifying performance bottlenecks of the application, analyzing system behavior under the load, and helping teams to build scalable, reliable applications. I also teach performance testing. I have a course on Udemy, which is called Performance Testing Fundamentals From Basics to Hands-On, and if you are new to this topic or want to go deeper after this talk, you can find it there. I also write about performance testing, about load testing, about production lessons on Dev Community and Medium. You can subscribe to me if this direction resonates with you. And also, you can find me on LinkedIn. I post frequently about performance testing, feel free to connect. All right, let's get started.

A few months ago I posted a poll in the LinkedIn software testing group and the question was simple. If you have an e-commerce platform and your platform must handle Black Friday traffic spikes, or four to ten times within minutes, which test will you run? Four options were available. First one is spike testing, then scalability testing, volume testing, and endurance testing. Around 230 engineers answered this question. 55% of them selected spike testing, which is actually the correct answer. And that is great. But that means nearly half of the engineers chose something else. 45% of engineers, when given a real Black Friday scenario, would skip very important tests that may cause production issues. So what is spike testing? Spike testing is not about just a lot of load. There are three things that define this type of testing. First, the load is sudden. So it's not gradual, it's not linear. And system has zero time to warm up. You go from the base load to extreme load in seconds.

Second, we observed what actually happens during the spike. Because this is where things get interesting. So does the application degrade gracefully? Or does everything fall apart at once? Maybe one service gets overloaded, then another starts timing out, then retries to increase the traffic even more. And suddenly a single bottleneck turns into a full production incident.

Third, and this is the part that most teams skip, it's to measure recovery. You are looking at what happens after the spike drops. How fast does your system return to the normal state? Does it safe heal or does it stay degraded during the next 20 minutes? All three matters. If you're watching only whether your system survives during the spike, you're actually testing only the half of the story.

Let's talk a bit about why other testing types don't fit the Black Friday scenario. Volume testing is about data, millions of records, huge payloads, and database at scale. Black Friday is not a data problem; it's a concurrency problem. Endurance testing detects memory leaks and system degradation over hours, not suitable for the sudden load of Black Friday. Scalability testing checks gradual scaling, not immediate spikes. Spike testing replicates the Black Friday scenario accurately, unlike other tests that may give a false sense of confidence.

We run this test for many hours with the main aim of detecting memory leaks and system degradation. However, Black Friday's sudden influx of traffic within seconds or minutes doesn't align with this gradual approach. Scalability testing ensures proper system scaling up and down with gradual user number changes, which contrasts the immediate load spike on Black Friday. Only spike testing accurately replicates the rapid traffic surge seen on Black Friday, highlighting the necessity for this specific type of testing.

Before Black Friday, the team felt confident after passing volume, endurance, and scalability tests with flying colors, receiving positive reports and approval from management. However, during the event, traffic escalated dramatically in just 90 seconds, leading to errors, increased response time, and halted checkout processes for many users. The system's inability to handle the sudden massive load resulted in disappointment for users and significant revenue loss. The absence of critical testing left the team unprepared for the Black Friday rush.

The reasons behind the application's failure during Black Friday production stem from insufficiently prepared systems. With the spike hitting before cache warming up, every request directly hits the database, making it the initial bottleneck. Additionally, connection pool limits under the sudden spike can quickly exhaust, leading to queuing requests and timeouts. Thread pool saturation further compounds the issues, showcasing the critical importance of comprehensive testing to anticipate and address such challenges.

After the sales go live, users arrive in just over 90 seconds, causing a sixfold increase in traffic compared to normal. This sudden surge leads to system errors, increased response times, and checkout failures for many users. On-call engineers attempt to resolve the issues, but the damage is already significant, resulting in disappointed users and substantial revenue loss. The system's inability to handle the massive load spike within the first 90 seconds reveals a critical lack of preparation for Black Friday.

The application's failure during Black Friday production is attributed to several key factors. The absence of a crucial warm-up period for the cache results in every request hitting the database directly, creating an immediate bottleneck. Connection pool limits are quickly exceeded during the spike, leading to queuing requests and timeouts. Thread pool saturation exacerbates the situation, underlining the importance of comprehensive testing to anticipate and address such challenges.

The shared resource nature of database logs during a traffic spike causes thousands of concurrent writes to compete for the same resources, resulting in locks, transaction delays, and deadlocks. Autoscaling delays further compound the issues, with new instances taking minutes to provision while spikes occur within seconds. These problems often go unnoticed in volume, endurance, and scalability tests, only manifesting under sudden massive traffic spikes.

When spike hits, the cache wasn't warmed up yet. Every single request goes straight to the database. And the database becomes your first bottleneck. Next is connection pool limits. Under normal load traffic, the database connection pools work fine, but during the spike, they can be exhausted in seconds. Requests queue, eventually timing out.

Thread pool saturation occurs as application servers run out of threads to handle incoming requests. From the user's perspective, the application freezes, and requests keep loading indefinitely. Database logs become shared resources, with thousands of writes competing for the same roles during a spike. This competition leads to locks, transaction delays, and deadlocks.

Autoscaling delays compound issues, as new instances can take several minutes to provision while spikes occur rapidly. Problems like these often go unnoticed in volume, endurance, and scalability tests, only revealing themselves under sudden massive traffic spikes. Spike testing is essential to address these specific challenges effectively.

Next is autoscaling delays. Provisioning of new instance can take up to several minutes. While spike kills you even faster than 90 seconds. New instances arrive only after actual damage is already done. As a result, volume, endurance, scalability tests often miss these problems. And they only appear under sudden massive traffic spike. And that's exactly why spike testing exists.

Let's take a look at this graph. How does spike test look in practice? Orange line represents for us concurrent users. In the beginning we ramp up our users to our base load, reaching 250 users, and maintaining that load for a period. Then we simulate a sudden jump, adding an additional 1000 users. The green line reflects response time, remaining flat at the base load with requests completing under 3 seconds. During the spike, the response time sharply climbs to almost 20 seconds, indicating system stress and recovery evaluation.

Lastly, the red line represents the error rate. At the base load, errors are minimal, but during the spike, errors increase. The critical analysis post-spike includes normalizing errors after load reduction and assessing their impact on user actions. This detailed graph not only determines system survival but also provides insights into system behavior, recovery time, and error impact on user experience. Setting up an effective spike test involves establishing the base load, identifying peak hours and user numbers using tools like Google Analytics or Pando, and creating a realistic traffic distribution model as the foundation for the test.

And if it come back after the spike our system recovers. If it stayed elevated we have a problem. Red line represents error rate. Near zero at the base load and during the spike errors appears more and more. And the question is do they normalize after load drops? And how do they impact user actions? If errors block major user actions we should investigate those errors. This graph tells us not just whether system survived, but how it behaved and how fast it recovered.

Now let's talk about how to set up a good spike test. Step one we should establish our base load. This purpose I recommend you to use Google Analytics tool, Pando or any other tools that your team have in place. We need to identify our peak hours, peak number of users during the day. However, baseline will become our foundation for the entire test. We should also build realistic traffic distribution model. Usually around 60% of users simply browse the products or perform the search. About 25% of users add products to the carts. And the smallest part only around 50% actually proceed to check out and complete fully the purchase.

Step two is to define the spike magnitude. For Black Friday scenarios the target is usually between 4 and 10 times of our base traffic. Typical ranges look like this. Four times it's moderate Black Friday search, 6 to 8 times it's about typical e-commerce events, 10 times it's our worst case scenario. Now let's look at how this load model can be built using K6 test example. So actually we are starting from the ramping up our users to the base load. We define that our base load is 200 users. After that we hold that load during next 10 minutes. And only after this we provide additional 90 seconds to reach our spike. We have multiplied our baseline by 6 and reached 1,200 users. Then we hold that spike during next 30 minutes and we monitor during that our infrastructure resources. We also check whether errors appear and so on. After that we ramping down users for the next 90 seconds and then hold our base load for next 10 minutes to see whether our system is able to recover. Next step is to monitor application while test is running. So while our test is running we should watch such metrics.

Connection pool saturation, thread queue depths, database logs, wait time, cache hit ratio, infrastructure resource usage and overall error rate.

Step 4 is to measure recovery. After spike drops how fast our system recovers to the normal state. How fast the number of errors decreases. Does response time return to the normal state? A system that survives a spike but never recovers it is still considered as a failed system.

OK, Black Friday failures usually don't happen because of one bug. They happen because the system behaves differently under standard pressure than it does under normal conditions. And if you never simulate that pressure production becomes your test environment and that is bad. Spike testing is not about creating chaos. It's about discovering the chaos before your users do. Thank you for your attention.