Agent Module Mechanisms

ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability

ReDeEP studies tool and evidence-use hallucination in retrieval-augmented generation. It separates evidence-sensitive pathways from parametric-knowledge pathways, locates failure signals in model internals, and uses this structure for hallucination detection and intervention.

ICLR 2025 Spotlight First author Zhongxiang Sun, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, Han Li

Paper Code Research Program

Research Question

A mechanism-level view of RAG hallucination as imbalance between external-evidence use and parametric-knowledge reliance.

Why do retrieval-augmented generation systems still hallucinate when relevant external evidence is available?

Paper Walkthrough

From the research problem to mechanism and evidence

The figures below follow their order in the paper. Each section connects the figure to the question it resolves and the conclusion it supports.

ReDeEP Evidence can be present but not used figure

The left half is a successful RAG behavior. The retrieved document states that Chang'e-6 brought back 1935.3 grams of lunar samples; the query asks how many grams were brought back; the response repeats the retrieved value and is marked correct. This is the desired path: retrieved document, query, model, and answer are aligned.

The right half shows the failure ReDeEP targets. The retrieved document says good ultrasound pictures are best between 20 and 32 weeks, while the purple parametric-knowledge box says 24 to 32 weeks. The final response mixes in a 17-week condition and is marked hallucinated. This makes the figure a concrete example of evidence being present but not controlling the answer.

How to read this figure

In the left example, the green retrieved-document box and yellow response box share the same key value, 1935.3 grams.
In the right example, the retrieved document says 20-32 weeks, while the parametric-knowledge box says 24-32 weeks; the response drifts instead of cleanly following either source.
The figure motivates ReDeEP's core diagnostic question: after retrieval, which source actually controls generation?

ReDeEP Two information pathways in RAG figure

The left purple block shows existing detector settings where parametric knowledge P is confounded by external evidence E. The middle green block shows the opposite case, where external evidence E is confounded by parametric knowledge P. The lower center panel shows the mixed setting in which P and E jointly affect H.

The blue "Decouple & Regression" panel on the right is the methodological target: separate P and E so that neither source masks the other. This is why the figure lists prior methods under the confounded settings and reserves the decoupled graph for ReDeEP.

How to read this figure

Read E as the retrieved external context, P as parametric knowledge, and H as whether the response hallucinates.
The three causal graphs show why a detector can be misleading if it attributes hallucination to only one source.
ReDeEP's contribution is the rightmost decoupling step: estimate the influence of E and P separately.

ReDeEP Operationalizing evidence use figure

The left side computes External Context Score. Step 1 identifies attended tokens through attention weights; Step 2 mean-pools selected hidden states and calculates cosine similarity. The output is an external-context signal for a layer-head position, shown as E with layer/head superscripts.

The right side computes Parametric Knowledge Score. The figure follows the last-layer hidden state through the unembedding matrix and also compares residual streams before and after FFN through LogitLens, softmax, and JSD. This directly matches the paper's claim that ReDeEP measures both evidence attention and parametric knowledge injection.

How to read this figure

Follow the orange arrows on the left to see how attended evidence tokens become External Context Score.
Follow the blue boxed LogitLens/JSD computation on the right to see how FFN/residual changes become Parametric Knowledge Score.
The figure explains why ReDeEP is mechanism-aware: it scores internal information routing, not just answer text.

ReDeEP Internal evidence for the pathway account figure

Panels (a) and (b) are layer-head heatmaps for external context: panel (a) shows the score difference between truthful and hallucinated generations, while panel (b) reports Pearson correlation with the inverse hallucination label. Panel (c) separately visualizes copying-head scores across layers and heads.

Panels (d) and (e) move to parametric knowledge. Panel (d) shows the hallucination-minus-truth score difference by layer, with a dashed average line; panel (e) reports Pearson correlation between parametric knowledge score and hallucination label. Together, the five panels align with the method figure: evidence use and parametric knowledge reliance are both measurable and both differ across faithful and hallucinated behavior.

How to read this figure

Panels (a)-(c) should be read as evidence-side diagnostics over layers and heads.
Panels (d)-(e) should be read as knowledge-side diagnostics over layers.
The figure closes the loop with Fig. 3: the scores computed from internal components separate hallucinated from truthful generation.

ReDeEP Causal intervention and known/unknown evidence Intervention on attention heads and FFNs panel

ReDeEP Causal intervention and known/unknown evidence Truth-known versus hallucinated-unknown scores panel

The left panel is the causal check: interventions on the attention-head and FFN components change the loss in the expected direction, supporting the claim that these components are not merely correlated with RAG hallucination.

The right panel separates cases where the model knows the truthful answer from cases where it hallucinates when the answer is unknown. This comparison keeps the ReDeEP story tied to source competition: external-context use and parametric-knowledge reliance vary with whether generation is faithful or hallucinated.

How to read this figure

Use the intervention panel to read Copying Heads and Knowledge FFNs as causal handles, not only diagnostic features.
Use the known/unknown panel to connect the mechanism back to whether the model has reliable parametric knowledge.
Together these panels justify AARF-style mitigation: add attention-side evidence retention and reduce excessive FFN knowledge injection.

Experimental Evidence

Main experimental results

ReDeEP is evaluated as a hallucination detector on RAGTruth and Dolly (AC). The compact table reports the strongest non-ReDeEP result for each selected metric alongside ReDeEP (chunk), which is the most consistently effective variant in the main table.

RAG hallucination detection

Backbone	Method	RAGTruth AUC	RAGTruth F1	Dolly AUC	Dolly F1
LLaMA2-7B	Best baseline	0.7290	0.7066	0.7110	0.7241
LLaMA2-7B	ReDeEP (chunk)	0.7458	0.7190	0.7949	0.7833
LLaMA2-13B	Best baseline	0.8089	0.7342	0.7214	0.7200
LLaMA2-13B	ReDeEP (chunk)	0.8244	0.7587	0.8420	0.7603

Metrics are reported on a 0-1 scale. The baseline ceiling is the best non-ReDeEP value in the original table for each column.

What the results show

ReDeEP (chunk) exceeds the strongest reported non-ReDeEP value on all eight selected backbone-dataset-metric comparisons.
The largest margins appear on Dolly AUC: +0.0839 with LLaMA2-7B and +0.1206 with LLaMA2-13B.
The result supports the central mechanism claim: jointly modeling external-context use and parametric-knowledge injection improves detection beyond output-only signals.

Main Takeaways

What this project establishes

RAG hallucination can happen after successful retrieval

The central finding is conceptual but important: the model can fail even when the evidence pipeline succeeds. A document may contain the answer, but the model may still rely on a competing internal belief, a memorized association, or an unsupported prior.

Faithful and hallucinated generations use information differently

ReDeEP shows that faithful and hallucinated generations are not merely different at the output level. They differ in the way internal pathways use retrieved context and parametric knowledge. This is why pathway-level scores are useful: they expose a hidden state of evidence dependence.

Mechanistic detection supports targeted mitigation

Because the signal is tied to an internal mechanism, it can be used for more than post-hoc labeling. A high-risk pathway imbalance can guide the system to re-check the evidence, request another retrieval step, or steer the model toward external context.

Tool use needs internal auditing

ReDeEP also has a broader implication for agents. Tool use cannot be trusted only at the interface level. An agent may call the correct tool and still fail internally if it does not integrate the returned evidence.

Research Impact

Position in the broader research program

ReDeEP makes the evidence-use module of an agent inspectable. In a future agent workflow, retrieval, browsing, and tool calls will be routine, but the central safety problem will remain: did the agent actually use the evidence it gathered?

The project contributes a mechanism-level answer. It shows how to move from final-answer checking to internal source auditing, which is a necessary step toward agents that can explain, monitor, and correct their own use of external information.

Resources and Citation

Paper Code Full Publication List

@inproceedings{sun2025redeep,
  title={{ReDeEP}: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability},
  author={Sun, Zhongxiang and Zang, Xiaoxue and Zheng, Kai and Xu, Jun and Zhang, Xiao and Yu, Weijie and Song, Yang and Li, Han},
  booktitle={International Conference on Learning Representations},
  year={2025}
}