Databricks lead shares five-pillar playbook for shipping AI agents to production
Sandipan Bhaumik of Databricks outlines a five-pillar production AI playbook — evaluation, observability, data foundation, multi-agent orchestration, and governance — drawn from real enterprise engagements including a retail bank that spent £85,000 on a chatbot PoC that never reached production.
Score breakdown
The talk reframes enterprise AI deployment failures as systemic infrastructure gaps — not model selection problems — showing that observability, evaluation pipelines, and governance tooling must be built before a model is even chosen.
- 01A retail bank spent £85,000 over six months on a chatbot PoC that never reached production.
- 02Bhaumik's team spent the first six of an eight-week engagement on evaluation data, tracing infrastructure, and a measurement pipeline — the model was chosen in week seven.
- 03Six weeks post-launch, a tracing system caught that a new interest rate policy document had not been reembedded, causing the agent to serve stale answers.
Sandipan Bhaumik, technical lead for data and AI at Databricks, draws on several years of enterprise AI engagements — spanning B2B software and regulated industries like financial services — to diagnose why AI demos routinely fail to reach production. He identifies a recurring anti-pattern: organizations begin by choosing a model, build features in a controlled environment with predictable data, produce a compelling demo, and then watch the system degrade in production with no way to explain why. His retail bank case study is illustrative: the bank had spent £85,000 over six months on a chatbot PoC that could not reach production, and no one could explain the failures. His team spent the first six weeks of an eight-week engagement building evaluation data, tracing infrastructure, and a measurement pipeline — the model was not selected until week seven. Six weeks after launch, when the bank updated its interest rate policy and customer satisfaction dropped, the tracing system identified the root cause: the new policy document had not been reembedded, causing the agent to serve stale answers.
He also emphasizes that the evaluation dataset should be treated as a living system rather than a fixed benchmark, and that a production incident playbook connects all five pillars.
From these engagements, Bhaumik distills three systemic gaps — observability (inability to trace agent decisions), evaluation (no numerically defined success metric tied to business outcomes), and governance — and organizes his response into five pillars: evaluation (define success numerically before writing code), observability (trace every agent decision, which European regulators require), data foundation (agents do not forgive bad data the way humans do), multi-agent orchestration patterns, and governance (47 PII breaches were caught in testing before launch). He also emphasizes that the evaluation dataset should be treated as a living system rather than a fixed benchmark, and that a production incident playbook connects all five pillars. The transcript is truncated before the full framework detail is presented.
Key facts
- 01A retail bank spent £85,000 over six months on a chatbot PoC that never reached production.
- 02Bhaumik's team spent the first six of an eight-week engagement on evaluation data, tracing infrastructure, and a measurement pipeline — the model was chosen in week seven.
- 03Six weeks post-launch, a tracing system caught that a new interest rate policy document had not been reembedded, causing the agent to serve stale answers.
- 04The playbook is built on five pillars: evaluation, observability, data foundation, multi-agent orchestration, and governance.
- 05European regulators require tracing of every agent decision, per the talk.
- 0647 PII breaches were caught in testing before launch.
- 07The evaluation dataset is described as a living system, not a fixed benchmark.
Topics
Summary and scoring are generated automatically from the original article. We always link back to the publisher and never republish images or paywalled content. Last processed Jun 19, 2026 · 10:25 UTC. How this works →