Halving Your CI Pipeline – Practical Optimisation Strategies

Hi, I'm Serhii, and today we're going to talk about how we can half your CI pipeline with some practical optimization strategies. In our organization, at around 50 merge requests a day and 400 pipelines run, including merge trains, one minute of overhead in your CI pipeline costs nearly six to seven hours of engineering time every single day. Now, you can think about your organization and scale this accordingly. This is a talk about how we cut our merge train pipeline from about an hour to 22 minutes. It's a 64% reduction using a structured ROI-prioritized approach.

Five areas in the order we use them, with actual numbers for our 400-person legal tech SaaS, where half of the people are developers. So this was our starting line. Merge train pipeline, about an hour. CI health, the success rate without retries is around 82%, $16,000 a month on AWS compute. And the dumb complaint on our developer experience survey was simply pipeline speed. So the number that mattered most wasn't the duration, it was the failure rate. Nearly one in five merge attempts failed.

Retries became muscle memory. People stopped trusting the system. Once developers stopped trusting CI, every other engineering metrics started sliding, too. Now, most teams approach CI optimization usually backwards. They start with easy quick wins because the quick wins can ship in an afternoon. I actually started different. So we started with a table, every potential optimization scored by expected time saving, effort, and risk. So instance migration was the largest single lever.

12 minutes, so it went first. Observability tweaks at one and a half minutes went later, even though they were the easiest. So we focus on the high impact first, quick wins in parallel, and fine-tuning last. Pick the right battles before you fight them. The biggest single win came from the compute instance migration for our feature, or end-to-end to test, how you call them, which are always considered slow, flaky, and heavy. I benchmarked three different configurations around 10 runs each because one slow pipeline can blow up your average. So the first instance was C7a AMD-based, averaged 25 minutes, but actually it ranged from 22 to 32 minutes, really widespread. The second option was C7i Intel-based, and in general, it was slower, 27 minutes, but the spread was just 24 to 30. So we actually picked up Intel. Why? In a merge train, variance is more damaging than mean performance.

A single slow run blocks every merge behind. So we made a decision. And also, we added some additional workers to the C7i, and we got to 22 to 36 on average, 12 and a half minutes of every merged train run.

The next level was test environment sizing. We had 30 test nodes, and all of them carried Chrome. This made sense for feature and or end-to-end tests, but not for service tests, which never open the browser. Chrome cost us roughly 0.4 gigabytes per worker. Across the old pool, that was about 12 gigabytes of browser overhead.

We could not remove all of it because most tests still need Chrome, but we moved the service test slides to lightweight browser-less nodes and kept Chrome only where it was actually required. That reclaimed about 2 gigabytes per run and took around three minutes of the merged train. The only risk was confidence. When you split a suite, you can accidentally stop testing something. So we added a line and a branch coverage validator as a guardrail.

So the lesson was simple. Remove unnecessary weight, but keep the trust. Faster CI only matters if developers still believe the result. So three changes, each about a day to ship. One, our observability service metrics collection took 120 seconds per pipeline. Every CI run was hitting a deprecated synchronous API. So we migrated it to a batched replacement, down to 20 seconds.

Let's take a look at the documentation-only merge request. Even having a few observes a day, full pipelines avoided, saved around 240 a month in compute, half a day of work, and combined 3.5 days of work, 3.5 minutes of every pipeline, small individually but compound together. Speed was an easy half. Stability was a harder half. 18% failure rate. People had learned to click retry without reading the failure.

Also, we had a built-in mechanism for test retry that was adding an additional loop of tests, and of course, it was time-consuming. So I built a system that goes further than naive retry counts. When a test fails across three different merge requests in a 12-hour Windows, usually in unrelated divs, it's flagged as persistent flaky. The classifier uses an LLM plus the git diff to distinguish a real product bug from a flake. So flagged tests get auto-quarantined.

So flagged tests get auto-quarantined. The AirSpec QA Runner, which is our custom runner on top of the Ruby AirSpec testing library, skips them. Redis TTL auto-expires the quarantine if the issue is actually fixed. Every decision posts to a Slack message. Result? 30-plus flaky tests permanently skip and later fix. CI health went from 82 to 94%.

The whole system costs about $100 a month in LLM calls. Here is the counter-intuitive part. We spend more on infrastructure because we use more heavy runners. It costs us about $1,680 more per month. Bigger Intel instances, more workers. The infra bill went up roughly by 10%. But the time we recovered is around 1,300 engineer hours a month. And at a conservative $50 an hour, that's actually like $65,000 a month in a productivity bug. I think the main lesson here is optimizing for the lowest CI cost is not the same as optimizing for the lowest total cost. Spending more on the right things pays for itself many times over.

So the headline numbers. Merge train from about an hour to 22 minutes. 64% reduction. CI health from 82% to 94%. The interesting line is actually at the bottom. During the same period, our test suite grew by 90%. From 10,000 tests to 19,000. Pipeline time grew only 11%. The optimization work absorbed the test suite growth. The test coverage went up, pipeline time went down. Both at once. So what should you take away from my story? First, measure before you optimize. Not one run, not three runs. Enough runs to understand the pattern because CI performance lies when you only look at average.

Second, prioritize by return, not by convenience. Some easy fixes are worth doing, but they should not distract you from the bigger lever.

Third, optimize the whole system. In our case, speed alone would not have solved the problem. We also had to improve stability, protect coverage, and make the cost trade-off feasible. This is how we moved from one hour merge train to about 22 minutes and test suite almost doubled.

So my advice is simple. Go back to your pipeline, measure where the time really goes, attach money and engineering hours to it, and you will probably find one of the highest return engineering projects in your code base. That's pretty much it. Thanks for watching. So the principles I have shared are portable, but every pipeline is different. If you want to compare notes on what worked or what didn't for your team, find me on LinkedIn at the URL on the screen. I'm happy to go to talk through your ROI table, your benchmarks at the top, or your flaky test strategy. Until then, go measure your pipeline. Thank you, GSTation.