Cisco closes the agent feedback loop at 16M interactions/year

Carlos Pereira from Cisco's Customer Experience organization — a roughly 19,000-person team — presented at LangChain's Interrupt conference on how Cisco built a closed-loop observability and testing system for its production AI agents. Cisco handles an average of 16 million customer interactions per year, with support ticket volume dropping from 1.6 million to 1.4 million cases year-over-year, a reduction Pereira attributed in part to the systems described in the talk. The central problem he addressed: once agent adoption scales, the volume of feedback signals — thumbs-downs, confused users, low-confidence routing decisions — exceeds what a human team can triage, and the team itself becomes the bottleneck.

Cisco's solution treats feedback as a continuous loop rather than a ticket queue.

Cisco's solution treats feedback as a continuous loop rather than a ticket queue. Signals are captured via LangSmith traces and fed into a triage agent that uses LangSmith MCP and Jira MCP as its integration layer. A separate code agent handles clustering and diagnosing issues. Humans are kept in the loop only on write operations, not reads — a deliberate design choice to preserve oversight without creating a processing bottleneck. Pereira framed evaluations as infrastructure rather than experiments, and observability as the new bottleneck to manage as systems scale.

The talk also covered Cisco's technical support use case, which operates under high-stakes conditions — network outages that can bring entire businesses down. A live example demonstrated an enterprise network assessment that surfaced 2,176 security findings, with semantic routing used to handle ambiguous prompts. The session was part of a two-day series; Pereira noted that the prior day's session covered the renewals teammate and agentic foundations, while this session focused on observability, testing, and support.

Key facts

1. 01Cisco's Customer Experience organization handles an average of 16 million customer interactions per year.
2. 02Support ticket volume dropped from 1.6 million to 1.4 million cases year-over-year.
3. 03The feedback pipeline captures signals via LangSmith traces and routes them through a triage agent using LangSmith MCP and Jira MCP.
4. 04A code agent handles clustering and diagnosing issues identified by the triage agent.
5. 05Human oversight is reserved only for write operations, not reads.
6. 06Pereira described evaluations as infrastructure, not a side project.
7. 07A live support demo involved an enterprise network assessment that surfaced 2,176 security findings, with semantic routing handling ambiguous prompts.

Topics