Every confident answer BRAG gives is gated by two LLM agents whose job is to disagree with the evidence — a Verifier and a Refutation Agent. If they can’t refute the answer, it stands. If they can, the answer changes, downgrades, or discloses the disagreement. What reaches the reader is either correct, or honest about what’s missing — never a confident hallucination.
The system embeds your question, finds the most‑similar passages in its corpus, hands them to an LLM, and asks it to write an answer. Whatever it pulls, it trusts.
That works for a static product manual. It breaks the moment your corpus changes over time or contradicts itself.
A number gets restated; a definition gets revised. The old version stays in the corpus alongside the new one. Cosine similarity can’t tell which is current.
Two passages disagree on the same fact. RAG returns whichever scored higher. The passage that would have flipped the answer is invisible to the reader and the model.
An organization said X; later, it said not‑X. Both are in the corpus, and the most‑similar chunk is usually the earlier one. The system answers as if the reversal never happened.
An LLM reading a figure out of prose might paraphrase a softened version nearby or invent one from a similar line. RAG provides no separate channel for numerical values — the model can launder a guess as a citation.
BRAG enforces this with a pipeline of structured, traceable steps. The LLM writes prose — nothing else. It doesn’t invent numbers, doesn’t skip skeptics, doesn’t answer questions it shouldn’t have taken.
Every one of them can refuse, downgrade, or disclose a disagreement. The LLM is only allowed to write prose — never to invent numbers, never to skip a step.
Before any retrieval runs, BRAG checks whether the question is in scope. Asked about Disney’s subscribers, it doesn’t guess from Netflix data — it Hard Halts. Normal RAG retrieves something for every question, because cosine similarity always returns a top result.
D6 — out-of-scope queryquery "What’s Disney’s streaming subscriber count?" decision HARD_HALT cause input_failure reason out_of_scope — corpus is Netflix only elapsed 0.0s
The Planner emits a DecompositionPlan — a list of evidence slots, each tagged with what kind of evidence it needs (a number? a strategic claim? a forward‑looking statement?) and what time window to look in. Retrieval runs against those structured slots, not against raw question similarity.
D2 — decomposition for “did Netflix ever say no ads?”{ "query_id": "q_8eb01ef3a9", "complexity_tier": "simple", "synthesis_strategy": "integrate", "slots": [ { "slot_id": "S1", "sub_question": "Did Netflix ever state it had no plans to add ads?", "evidence_type": "strategic_position", "target_layer": "both", "period_filter": null, "coverage_threshold": 0.8 } ] }
Hybrid retrieval: dense embeddings (semantic) plus BM25 (lexical). Channel‑aware pre‑filter narrows to the right kind of facts before scoring. Numbers come from a separate XBRL fact store, so financial values are never read out of prose by an LLM.
D2 — first retrieval pass for slot S1vector_query "Did Netflix ever state it had no plans to add ads?" bm25_terms ["Netflix no ads plan", "no advertising strategy", "ad-free commitment"] period_filter null candidates 8 facts # top RRF scores F-PROSE-nflx-q1-2023-transcript-0078 rrf=0.0288 F-PROSE-nflx-q4-2017-transcript-0070 rrf=0.0159 ← supported F-PROSE-nflx-q3-2017-transcript-0097 rrf=0.0156 ← supported F-PROSE-nflx-q2-2022-transcript-0031 rrf=0.0159 ← contradicts ...
The Verifier doesn’t write the answer — it grades the evidence against the slot’s coverage rubric. Right company? Right period? Right metric? Numerically exact? If anything fails, the system either re‑retrieves or downgrades the outcome.
D2 — verifier verdict, slot S1coverage_score 0.8 verdict COVERED supported_candidates 3 F-PROSE-nflx-q4-2017-transcript-0070 F-PROSE-nflx-q3-2017-transcript-0097 F-PROSE-nflx-q1-2023-transcript-0066 contradiction_details "Conflicting statements about Netflix’s plans for ads. One statement suggests no plans to add ads, while another suggests introducing ads as a dimension of plan structure." conflicting_ids: F-PROSE-nflx-q4-2017-transcript-0070 F-PROSE-nflx-q2-2022-transcript-0031
The most distinctive stage. The Refutation Agent reads the answer the system is about to give, picks the most attackable claim, generates a counter‑hypothesis, and searches the corpus for evidence that contradicts the answer. It runs seven strategies, each tuned to a specific way financial reporting misleads.
D2 — refutation hypothesis tested against the 2017 claimhypothesis_id h_1 strategy later_reversal targets_claim_id F-PROSE-nflx-q4-2017-transcript-0070 hypothesis_text "Netflix’s Q4 2022 shareholder letter announced an ad-supported tier, reversing its earlier no-ads position." rationale "If Netflix introduced an ad-supported tier, it would contradict the earlier strategic claim that not having advertising was a differentiator." refutation_verdict STRONGLY_REFUTED evidence_ids F-PROSE-nflx-q2-2022-letter-0060 F-PROSE-nflx-q3-2022-transcript-0001 overall_verdict refutation_to_loop # triggers re-retrieval
Only now does the Generator run. It composes prose with assertion‑date attribution — every claim cites the document and the date it was made on. The Governor then checks the answer against the schema, the citations, and the disclosed contradictions before anything goes out.
D2 — final answer with disclosed reversal// final answer_text, after refutation loop In 2017 and 2018, Netflix executives stated that the company had no plans to add advertising. Reed Hastings noted in the Q3 2017 earnings call (2017-10-16) that "Netflix’s focus is not expanding in new ads at all," and reiterated in the Q4 2017 earnings call (2018-01-22) that "not having advertising is an important strategic differentiator for Netflix." However, Netflix reversed this position in 2022. The Q2 2022 shareholder letter (2022-07-19) announced plans to launch an ad-supported tier "around the early part of 2023," and Greg Peters confirmed in the Q3 2022 earnings call (2022-10-18) that Netflix had "announced details for the new Basic with Ads tier last week launching in the U.S. and 11 other markets." claims 2 grounded disclosed_refutations 1 # later_reversal, strongly_refuted degradation_level NORMAL adversarially_probed true
We tested BRAG on ten years of Netflix’s public financial reporting — a corpus chosen because the four failure modes above appear in it constantly. Here’s one query, run through both pipelines. The Normal RAG column is a plausible reconstruction; the BRAG column is verbatim from the actual D2 trace.
Reconstructed for comparison — BRAG does not ship a baseline RAG. The Normal RAG column shows what a textbook retrieve‑then‑generate pipeline would produce against the same corpus. The BRAG column is verbatim from the actual D2 trace.
Each trace shows the retrieved evidence, the Verifier output, every Refutation Agent hypothesis and verdict, the generated answer, governance violations, and the full execution timeline. Open any one to see the system show its work.
What was Netflix’s revenue for Q2 2023?
Open trace →Did Netflix ever say it had no plans to add ads?
Open trace →Has Netflix’s stance on advertising changed?
Open trace →Has Netflix’s password sharing policy been consistent?
Open trace →Why did Netflix’s free cash flow turn positive in 2022?
Open trace →What’s Disney’s streaming subscriber count?
Open trace →Every retrieval, verifier verdict, refutation hypothesis, and degradation decision is logged to a MemoryLedger and rendered into per‑query trace HTML.
Financial numbers come from XBRL (the structured data the SEC requires), not from an LLM reading prose. That eliminates a whole category of hallucination — the system can’t paraphrase or invent a figure because the figure arrives through a separate, deterministic channel. Both paths produce the same FactRecord schema and land in the same store; the Refutation Agent draws from both.
edgartools + lxml parse on‑disk XBRL. Aggregate + UCAN / EMEA / LATAM / APAC geographic segments. verbatim_anchor matches the formatted display value in rendered HTML, so citations land on the exact figure a human would see.
LLM extractor (DeepSeek V3) over chunked filings + transcripts. Six fact types: operational_metric, forward_guidance, strategic_claim, causal_explanation, risk_disclosure, accounting_policy. financial_metric is XBRL‑only by construction.
BRAG is a Q&A system, but the design rules generalise to any agent that retrieves evidence, reasons about it, and acts. If your agent does things instead of just answering, the case for these principles gets stronger, not weaker — wrong answers from a chatbot are embarrassing; wrong actions from an agent are operational incidents.
Take what’s useful. None of this requires BRAG.
Don’t let your retriever return undifferentiated chunks. Tag what kind of evidence you’re retrieving (forward‑looking, historical, policy, opinion, structured number) and let downstream stages reason about coverage on those types. Embeddings give you “similar.” Type tags give you “relevant.”
Never let an LLM read structured values out of prose. Parse them deterministically, store them with provenance, cite them by ID. BRAG uses XBRL; your equivalent might be a database, a CSV, an internal API. The rule: the LLM never writes a number it didn’t receive as a typed value.
Before any retrieval runs, decide whether the question (or task) is even in scope. A clean “I don’t know” is the highest‑value behaviour an agent can have — and the hardest to ship. Default agentic systems happily try anything; trustworthy ones refuse out‑of‑scope work explicitly.
A second LLM whose only job is to refute the draft answer catches things no eval set can. It roughly doubles your LLM cost per query — that’s real — but it’s the difference between “demo‑good” and “I’d let a regulated customer see this.”
Every retrieval, verdict, hypothesis, degradation, and tool call goes in a single ledger you can replay later. When someone asks “why did the system do that?” you should be able to walk them through it step by step. Audit trails aren’t optional in regulated industries — and they’re invaluable everywhere else.
The worst failure mode is a confident wrong answer; the second worst is a system that fails silently. Build explicit outcome states — full answer, partial, clarification request, refusal — and surface them in the response itself. Visible degradation is the third option, and almost always the right one.
© 2026 · All trace data is reproducible from the repository.