Multi-LLM debate engine uses a validator agent to fact-check in real time

Suat's post describes a multi-agent debate system designed to combat a specific failure mode: when multiple LLMs are asked the same question, fabricated citations carry the same rhetorical weight as real ones, and a downstream summarizer will typically treat all sources as equivalent rather than surface the hallucinations. The naive approaches — voting, arguing, or summarizing — make things worse because LLMs are prone to sycophancy, conceding to confident but wrong claims rather than pushing back.

In this system, that role is filled by a Validator agent that does not debate, take sides, or argue — it only verifies.

The architecture draws on the shared expert pattern from Mixture-of-Experts language models, where one expert processes every token regardless of routing. In this system, that role is filled by a Validator agent that does not debate, take sides, or argue — it only verifies. Four debating personas (Analyst, Strategist, Devil's Advocate, Researcher) each produce opinions in Round 1. The Validator then reads all four outputs, runs `web_search` on every concrete claim — numbers, dates, company names, report citations, URLs — and emits structured `[OK]`, `[WARN]`, or `[FAIL]` markers with corrections and source URLs. Critically, the Validator's full reasoning is not shown to the other agents; only the structured markers are injected, preventing agents from quoting the Validator as a peer. Future-dated source claims are automatically marked `[FAIL]`.

In Round 2 and Round 3, each persona sees the previous round's outputs plus the Validator's findings, with an explicit instruction not to reuse `[FAIL]`-marked claims. Suat reports that in test runs, the same model that fabricated a citation in Round 1 correctly dropped it in Round 2 and reframed its argument around real data — with no fine-tuning or retraining, purely through structured in-context feedback. The post also describes a comparison between a run with a functioning Validator and one where the Validator timed out, though the full results of that comparison are not included in the provided excerpt.

Key facts

1. 01The system is published on GitHub as `capitansuat/swarm-debate` under the MIT license.
2. 02Four debating personas — Analyst, Strategist, Devil's Advocate, and Researcher — each generate opinions per round.
3. 03A fifth Validator agent runs on every round and uses `web_search` to verify every concrete claim (numbers, dates, company names, URLs, events).
4. 04Validator output uses structured markers: `[OK]` (verified), `[WARN]` (suspect), and `[FAIL]` (fabricated or wrong, with correction and source URL).
5. 05Future-dated source citations are automatically marked `[FAIL]` by the Validator's system prompt.
6. 06Only the structured markers — not the Validator's full reasoning — are injected into subsequent rounds, to prevent agents from treating the Validator as a debate peer.
7. 07The architecture is inspired by the shared expert pattern in Mixture-of-Experts language models.

Topics