Agent Module Mechanisms

Mechanistic Detection and Mitigation of Hallucination in Large Reasoning Models

RHD moves hallucination analysis from final answers to reasoning trajectories. It characterizes early fluctuation, shallow pattern matching, erroneous backtracking, and pseudo-verification in multi-step reasoning, then uses process-level reward constraints to mitigate these failures.

ICLR 2026 First author Zhongxiang Sun, Qipeng Wang, Haoyu Wang, Xiao Zhang, Jun Xu

Paper Code Research Program

Research Question

A temporal neural-dynamics account of reasoning hallucination in large reasoning models.

When a large reasoning model produces a wrong answer, where does the hallucination actually form across the reasoning trajectory?

Paper Walkthrough

From the research problem to mechanism and evidence

The figures below follow their order in the paper. Each section connects the figure to the question it resolves and the conclusion it supports.

RHD / GRPO-R Measuring reasoning trajectories figure

The left side starts from the transformer decoder layer: multi-self-attention and FFN update hidden states, and LogitLens converts intermediate states into vocabulary distributions. The center grid then tracks distributions over reasoning steps c1, c2, c3, ..., cK.

The colored triangles at the top represent step-level reasoning states used to compute R_score from early to later steps. The right side defines CV Score as sigma over mean. Attention Score measures the fraction of top-attended earlier steps whose Reasoning Scores fall in shallow or overthinking ranges. This figure is the exact measurement pipeline behind later "fluctuating", "stable", and "misguided attention" claims.

How to read this figure

Follow hidden state h through LogitLens to see where the per-step vocabulary distributions come from.
Read R_score as the trajectory's reasoning-strength signal over steps.
Read CV Score as early trajectory fluctuation and Attention Score as later attention to shallow or overthinking steps.

RHD / GRPO-R A trajectory can go wrong before the answer figure

The question asks how much Liam should pay now, given prices were 10 percent cheaper last year. The left plot shows a hallucinated trajectory with a fluctuating R_score curve across about 100 steps; the right plot shows a shorter truthful trajectory with a more stable R_score curve.

The bottom text boxes tie the curves to concrete reasoning events. Step 2 to Step 3 is a correct verification even in the hallucinated trace. Step 44 to Step 45 is labeled incorrect verification and overthinking. Step 73 is labeled incorrect backtracking or misguided attention, leading to the wrong final answer 261.9 instead of the golden answer 291. The figure therefore makes the failure temporally localizable.

How to read this figure

Use the red callouts to connect score fluctuations with named failure events.
Compare Step 44 to Step 45 and Step 73 with the truthful Step 23 to Step 24 verification.
The visible wrong answer, 261.9 versus golden answer 291, is the endpoint of earlier trajectory instability.

RHD / GRPO-R Validating misled reasoning patterns (a) Reasoning Score validation on GSM-NoOp panel

RHD / GRPO-R Validating misled reasoning patterns (b) Pattern #1 early fluctuation on ReTruthQA panel

Panel (a) compares mean reasoning score for misleading and non-misleading steps: the non-misleading bar is higher, matching the interpretation that better reasoning steps should carry stronger reasoning signal.

Panels (b) and (c) compare truth versus hallucination on CV Score and Attention Score. Hallucination has higher CV Score and higher Attention Score in the visible bars, aligning with the case figure's story that hallucinated traces fluctuate more and concentrate attention on misleading steps.

How to read this figure

Panel (a) validates R_score against misleading versus non-misleading reasoning states.
Panel (b) links hallucination to higher trajectory fluctuation.
Panel (c) links hallucination to stronger attention concentration on problematic reasoning steps.

RHD / GRPO-R Connecting dynamics with reliability (a) Consistency analysis panel

RHD / GRPO-R Connecting dynamics with reliability (b) Rising-2 accuracy comparison panel

Panels (a) and (b) compare rising consistency and stable consistency. The pie charts show that rising consistency contains a much larger inconsistent portion than stable consistency, and the adjacent bar chart compares the consistency ratio between successive steps.

Panel (c) plots perplexity against Reasoning Score and shows a downward trend: higher reasoning score corresponds to lower perplexity in the visible scatter. Panel (d) compares mean perplexity for rising-2 and stable conditions. These panels connect the trajectory scores to reliability and uncertainty, not just to qualitative examples.

How to read this figure

Use panels (a)-(b) to read consistency as a step-to-step property, not a final-answer property.
Use panel (c) to connect R_score with perplexity through the visible negative trend.
Use panel (d) to see that the trajectory pattern has a measurable uncertainty signature.

Experimental Evidence

Main experimental results

RHD is tested both as a detector of reasoning hallucinations and as a process signal for mitigation. The first table reports RHD's AUC and MC3 across ReTruthQA domains; the second shows how GRPO-R changes downstream reasoning accuracy.

Reasoning hallucination detection on ReTruthQA

Backbone	MATH AUC	MATH MC3	Science AUC	Science MC3	MultiHop AUC	MultiHop MC3
R1-7B + RHD	0.7978	0.5699	0.7194	0.6009	0.7361	0.7103
R1-14B + RHD	0.7292	0.4644	0.7686	0.5671	0.7255	0.5154

AUC evaluates binary detection; MC3 evaluates multi-trace ranking. Values are on a 0-1 scale.

Reasoning hallucination mitigation with GRPO-R

Backbone	Method	MATH500	AIME	GPQA-D	GPQA-M	GPQA-E
DeepSeek-R1-1.5B	GRPO	0.770	0.333	0.359	0.335	0.359
DeepSeek-R1-1.5B	GRPO-R	0.788	0.367	0.414	0.371	0.357
Qwen2.5-1.5B	GRPO	0.480	0.033	0.247	0.214	0.266
Qwen2.5-1.5B	GRPO-R	0.490	0.133	0.247	0.243	0.275

Accuracy on MATH500, AIME 2024, and GPQA. Values are on a 0-1 scale.

What the results show

RHD achieves the strongest AUC reported in the main table across all three domains for both R1-7B and R1-14B.
GRPO-R improves eight of ten benchmark-backbone comparisons, ties one, and is slightly lower on one GPQA-extended comparison.
The mitigation results connect diagnosis to control: the same reasoning-depth signal used to detect unstable trajectories can shape more reliable policies.

Main Takeaways

What this project establishes

Reasoning Score separates shallow and deep reasoning steps

On GSM-NoOp, steps misled by semantically irrelevant phrases receive lower Reasoning Scores than non-misled steps. This validates the score as an internal signal for shallow pattern matching.

Two trajectory patterns localize reasoning hallucination

Hallucinated traces show larger early-stage variation in Reasoning Score and stronger late-stage attention to earlier shallow or overthinking steps. CV Score and Attention Score operationalize these two patterns.

Overthinking has a measurable uncertainty signature

The mechanism analysis links rising Reasoning Score patterns with consistency changes and perplexity. Excessive score increases can correspond to overthinking and spurious verification instead of reliable correction.

GRPO-R turns diagnosis into step-level reward

GRPO-R adds the mechanism-derived deep-reasoning signal to reinforcement learning through potential-based shaping. The reward acts on intermediate steps while preserving the outcome objective.

Research Impact

Position in the broader research program

RHD contributes the temporal reasoning layer of the Cognitive Neuroscience of AI Agents program. It makes reasoning paths observable in a way that answer-only evaluation cannot.

This matters for any agent that plans, searches, proves, or debugs over many steps. A reliable agent must know not only whether its final answer is correct, but whether the path that produced the answer remained stable, evidence-sensitive, and self-correcting.

Resources and Citation

Paper Code Full Publication List

@inproceedings{sun2026mechanistic,
  title={Mechanistic Detection and Mitigation of Hallucination in Large Reasoning Models},
  author={Sun, Zhongxiang and Wang, Qipeng and Wang, Haoyu and Zhang, Xiao and Xu, Jun},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026}
}