Jack Fitzsimons

Jack Fitzsimons is the Co-Founder and CTO of Oblivious, a Dublin-based technology company building tools that help organisations understand, manage, and improve how AI is used across their engineering workflows. He holds a DPhil from the University of Oxford and has published at and reviewed for a range of top machine learning and data science conferences including NeurIPS, UAI, Entropy, ECML and others. He has worked on a wide range of data-centric challenges in industry; from topics in computer vision at NASA's Jet Propulsion Laboratory to quantitative data analysis at ElectroRoute, the European energy trading subsidiary of Mitsubishi. Jack has been an active member of the UN's Privacy-Preserving Technologies Task Team and a contributor to the NIST Differential Privacy Registry.

Oblivious, Dublin, Irelandjack-f-300810122

The PR Is the Eval

AI Coding Summit Berlin

Upcoming

The PR Is the Eval

Most coding agents are evaluated at the moment they answer. Software teams discover whether an answer was useful much later: a test fails, a reviewer requests changes, the pull request stalls, or the code ships. This talk shows how to turn those delayed signals into a continuous evaluation system. We instrument AI conversations, tool calls, diffs, commits, CI runs, review comments, and merge outcomes, then join them into one trace from prompt to production.

Using a working implementation, I’ll walk through the event model, correlation logic, and queries that separate accepted work from rework and repeated failure. I’ll show the hard parts: carrying trace context across unrelated tools, distinguishing flaky CI from agent errors, protecting developer privacy, and avoiding vanity metrics. Finally, we’ll turn recurring corrections into human-approved candidate skills and test whether they improve first-pass CI, review rounds, and time-to-merge.

You will leave with a product-agnostic blueprint for outcome-based evaluation of coding agents.

artificial intelligence