Evidence Drift: Preserving Guideline Disagreement in Clinical RAG
Clinical guidelines often differ across professional organizations. A RAG system answering questions about those guidelines needs to preserve that disagreement, instead of simply surfacing the highest scoring source as the recommendation.
Let’s use this example: a user asks how often an average-risk 48-year-old should receive screening mammography.
- USPSTF recommends screening every two years
- ACR recommends annual screening
- ACOG recommends every one to two years
- ACS recommends annual screening from ages 45 through 54
If we designed a system without attention to this, it could retrieve one source, summarize it accurately, and return a confident answer. Everything in the answer is true, and cited, but ultimately we have false consensus by only using one valid source.
Evidence Drift is a small, inspectable benchmark for testing whether retrieval preserves disagreement across organizations, changes over time, uncertainty in recommendations, and explicit corpus boundaries. The primary focus is disagreement across organizations. I used BM25, semantic, and hybrid retrieval in the demonstration.
Separating retrieval and generation
In this project I separated retrieval and generation to help localize errors and to focus more on the retrieval side. The default output is an evidence map (retrieval) and there is an optional synthesized clinical answer (generation). The focus for the evidence map is to see which passages were retrieved, which organization each came from, whether the source is marked current or superseded, its dates, ranking score, and an absolute support score.
What I wanted the benchmark to test
I focused on four behaviors:
- Preserving disagreement. A comparison question should retrieve the organizations needed to show that recommendations differ.
- Preserving temporal drift. A question about how guidance changed should retrieve both the older recommendation and the current one, without presenting the superseded source as current.
- Preserving uncertainty. “Insufficient evidence” should be a clear conclusion rather than silently becoming “recommended against.”
- Respecting corpus boundaries. When a question falls outside the scope of the average-risk screening corpus, the system should abstain rather than stretch nearby evidence into an answer.
Preventing one organization from crowding out the others
A search system usually returns the highest-scoring results. The problem is that several of those results might come from the same organization.
For example, the system could return three relevant ACOG pages but miss a USPSTF page with a different recommendation. The results would all look correct, but they would not show the full disagreement.
To reduce this problem, I designed the retriever to first pull the strongest current result from each organization. It then uses the remaining spots for the next best results overall. When the question asks how guidance changed over time, older recommendations can also be included.
This approach helped find the group of pages needed to show where the organizations agree and disagree and prevented one organization from crowding out others.
Support scores and scope guards
A retrieval system will always return something first, even when the question has nothing to do with the available documents. The first result may be the best match in the corpus while still being a weak match overall.
To prevent the system from answering with weak or irrelevant evidence, I added two ways for it to abstain.
- Explicit scope guard. Some questions are clearly outside the average-risk screening corpus, including questions about BRCA-associated risk, prior chest radiation, previous breast cancer, pregnancy, male screening, and diagnostic breast symptoms. The system detects these cases and abstains.
- Support threshold. Other out-of-scope questions may not match one of those known exclusions. For these, the system checks whether the corpus contains a strong enough match to support an answer.
This support score is separate from the score used to rank results. Ranking only tells us which passage is the best of the available options. It does not tell us whether that passage is actually relevant enough. Even an unrelated question will still have a first-place result.
The support score combines semantic similarity with BM25 term matching without rescaling the scores separately for every question. If even the strongest passage falls below the threshold, the system abstains.
This is a simple, project-specific safeguard rather than a probability that the answer is clinically correct. The threshold was set for this small corpus as an example and would need to be more carefully implemented and tested.
What the evaluation showed
The benchmark includes 12 canonical cases and 8 frozen holdout cases. On this small, single-author benchmark, the default hybrid configuration retrieved the complete expected evidence sets and handled every abstention decision across both slices correctly.
I also compared BM25, semantic, and hybrid retrieval. The semantic baseline here uses a local HashingVectorizer rather than a production embedding model, so I would not treat this comparison as a general retrieval bake-off. There was no “hybrid wins everything” result. BM25 performed especially well on the small, terminology-heavy parts of the corpus, while semantic retrieval handled some holdout cases better.
I also scored abstention separately from retrieval recall. For answerable questions, the benchmark checks whether the complete required evidence set was retrieved. For out-of-scope questions, it checks whether the system abstained.
What this setup helped expose
This setup helped see which layer could cause a miss, including:
- Source metadata and chunking
- Retrieval and organization diversity
- Scope and abstention logic
Instead of eyeballing an answer, with this approach you can ask more specific questions. Did the system retrieve all four current organizations? Did it include the superseded USPSTF recommendation when the question asked how guidelines changed? Did it abstain because the question was out of scope, or because it couldn’t reach the support threshold?
Limitations
The corpus contains seven short summaries of guidelines with metadata (not the complete source guidelines) and the benchmark only looks at 20 questions. The explicit scope guard is a nice example but pretty narrow, and the current way it is written is hard to generalize across other types of questions. The retrieval metrics only show whether expected documents appeared. I intentionally kept generation minimal, so the benchmark does not evaluate whether each generated claim is supported by what is cited.
Why this project was useful
This is one approach to building a retrieval system in clinical AI where documents and sources such as guidelines differ. A lot of times there is no one clean answer, and you need your retriever to preserve that disagreement and be testable.
The repo
Repo: github.com/sidoody/evidence-drift-rag
The default setup runs locally without an API key and includes the corpus, canonical and frozen-holdout slices, retrieval ablations, tracked regression snapshots, tests, a CLI, and a Streamlit interface.
Research demonstration only. Not medical advice, not a validated clinical tool, and not intended for patient care.