Every processed story in chronological order, with the newest coverage first. Filter by tag, source, or score to drill in.
As frontier models saturate existing benchmarks, the work of designing harder, more meaningful evaluations becomes the primary mechanism by which the field can track — and anticipate — the pace of AI capability growth.
The talk documents a concrete, production-tested eval architecture that closed the loop between offline simulation and live agent behavior at scale, directly enabling Lyft's resolution rate to climb from 10% to 35%.
The talk identifies a concrete regression in evaluation rigor — from data-science-grounded practices to ad hoc LLM-graded metrics — and maps five specific failure modes that teams building on agents are repeating at scale.
Treat eval score gains as a diagnostic signal rather than a leaderboard goal — Khan's three-zone failure-analysis framework gives AI/coding practitioners a concrete method for extracting actionable improvements from broken benchmarks without overfitting to them.
Teams building agentic products can apply Notion's hard-won lessons — on eval design, roadmap timing relative to model capabilities, and org structure — to avoid the same multi-year rebuild cycles Notion experienced.
Developers building production AI agents and RAG systems can use structured evals to catch hallucinations and regressions before deployment, replacing intuition-based quality decisions with measurable, evidence-driven metrics that reduce financial and legal risk.