Reddit post pitches external LLM drift detection service for silent API failures

u/Remarkable_Divide755 opens by disclosing they operate Tickerr, an existing independent external monitor for AI APIs that currently tracks latency, TTFT, uptime, and error rates. The post argues that basic transport health is a solved problem — teams can catch slow responses, timeouts, and 5xx errors with standard tooling like APM, Sentry, Langfuse, Helicone, or Datadog. The harder, largely unaddressed failure mode is silent model behavior drift: the API returns 200, latency is normal, no exception is thrown, output looks plausible, but JSON adherence, tool-calling, refusal behavior, reasoning quality, or instruction-following has quietly degraded. The post notes this problem compounds in agentic systems, where a drifting model can silently choose the wrong tool, stop early, mark a task complete incorrectly, or take a bad action — all while the system appears healthy at the API level. Two papers are cited as supporting evidence: VIGIL (arXiv 2605.08747), which reportedly found 65–88% of false-success reports occurred at literally zero task progress, and DeployBench (2606.05238), which found most failures involved the system stopping against a softer bar it set for itself and returning clean output.

Rather than judging any single output in isolation, the system would track whether today's behavior has materially shifted versus that model's own historical baseline.

The proposed solution is an external canary probe service that runs private fixed prompts on a schedule (every hour) against major models, tracking schema adherence, instruction-following, refusal/over-refusal, output length, tool-call format, and deterministic correctness checks. Rather than judging any single output in isolation, the system would track whether today's behavior has materially shifted versus that model's own historical baseline. A cross-model comparison layer would flag abnormal divergence between peer models — not to declare which is "right," but to surface statistical anomalies. An optional paid tier would allow teams to supply their own critical prompts, which Tickerr would run on a schedule and alert on if behavior drifts from a customer-defined baseline, with prompts kept private. The post solicits community feedback on technical soundness, which alert types would be most actionable, willingness to pay, and realistic pricing tiers ($19/month, $99/month, or $299+/month for team/Slack/webhook/BYO prompts).

Key facts

1. 01u/Remarkable_Divide755 already operates Tickerr, an external AI API monitor tracking latency, TTFT, uptime, and error rates.
2. 02The post argues the unsolved problem is silent behavioral drift: degraded JSON adherence, tool-calling, refusal, or instruction-following while the API returns HTTP 200 with no errors.
3. 03Agentic systems are identified as especially vulnerable, since a drifting model can silently choose the wrong tool or mark a task complete while everything looks healthy at the API level.
4. 04VIGIL (arXiv 2605.08747) is cited as finding 65–88% of false-success reports occurred at literally zero task progress.
5. 05DeployBench (2606.05238) is cited as finding most failures involved the system stopping against a softer self-set bar and returning clean output.
6. 06The proposed canary suite would run private fixed prompts hourly, track per-model behavioral baselines, and flag cross-model divergence — e.g., two models that typically disagree 12% of the time on a task suddenly diverging at 28%.
7. 07A paid BYO-prompts tier is proposed where customers supply their own critical prompts, kept private, and receive drift alerts against their own baseline.

Topics