With frontier models, now anyone in your company can build a dashboard with an agent. Rapidly, thousands of dashboards appear across the company. Then the underlying data model changes: revenue must exclude the new Buy-One-Get-One Free discount. Now all of those dashboards are silently but dramatically wrong, and incorrect numbers are presented to the board.
The foundational solution to this problem is to build a semantic layer: a centrally governed set of well defined business metrics that all dashboards consume. Individual queries reference the central model and are compiled to up-to-date SQL at runtime. Suddenly you can make a change in one place and everything updates - what a beautiful abstraction! However, anyone who has built a layer like this knows that it takes a tremendous amount of time and human energy to build and maintain.
Or does it?
We built an agentic harness to tackle a challenging industry benchmark (DABStep: Data Agent Benchmark for Multi-step Reasoning). Our approach used the open source Malloy language to build a central semantic layer that achieves over 95% accuracy, with no human in the loop. We also compare it to managing that central data model with markdown context and SQL statements.
We will cover the benefits and tradeoffs with the semantic layer approach and how to build your own harness and eval processes for your company. You will leave with a path to efficiently get trustworthy answers to your business questions, even when that business keeps changing.
This talk has been presented at AI Coding Summit NYC, check out the latest edition of this Tech Conference.
