"How do you write a regression test for a bug that refuses to reproduce? A production bug you cannot reproduce is a bug you cannot prove you fixed. If you have ever patched an agent, rerun it, watched the failure vanish, and still had no idea whether you fixed it, you know the problem. Rerunning a nondeterministic system changes the very thing you are trying to measure, while replay holds it fixed.
But what if you could replay the incident exactly as it happened and change only the one boundary you fixed? Enter cut point replay. A boundary is any point where the agent calls a model, invokes a tool, or makes a routing decision. We serve every boundary upstream of the fix from the recording, then run the boundary you changed, the cut point, and everything downstream live with no model call, so you see what your fix actually does and get the same incident on every run. We check the result in two layers, a deterministic structural check that never flakes and an LLM judge for wording changes that should not change behavior. You will leave able to commit any agent incident to git as a regression test that runs in CI on every pull request, turning a failure nobody could reproduce into part of your test suite, using Chronicle, an open source tool we built for exactly this and run against production agents."
This talk has been presented at AI Coding Summit Berlin, check out the latest edition of this Tech Conference.
