Susheem Koul

Your coding agent botched a multi file refactor once in production. You rerun the exact same prompt to debug it, and it works. The model sampled different tokens, a tool returned a different payload, the context drifted, and the failure was gone. This session shows how to capture every nondeterministic input to an agent loop into an append only execution log, replay the failed run step by step, and then turn that replay into reproducible evals so the bug never quietly returns.

"How do you write a regression test for a bug that refuses to reproduce? A production bug you cannot reproduce is a bug you cannot prove you fixed. If you have ever patched an agent, rerun it, watched the failure vanish, and still had no idea whether you fixed it, you know the problem. Rerunning a nondeterministic system changes the very thing you are trying to measure, while replay holds it fixed.But what if you could replay the incident exactly as it happened and change only the one boundary you fixed? Enter cut point replay. A boundary is any point where the agent calls a model, invokes a tool, or makes a routing decision. We serve every boundary upstream of the fix from the recording, then run the boundary you changed, the cut point, and everything downstream live with no model call, so you see what your fix actually does and get the same incident on every run. We check the result in two layers, a deterministic structural check that never flakes and an LLM judge for wording changes that should not change behavior. You will leave able to commit any agent incident to git as a regression test that runs in CI on every pull request, turning a failure nobody could reproduce into part of your test suite, using Chronicle, an open source tool we built for exactly this and run against production agents."