Metacognitive Control

ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding

ReARTeR makes retrieval-augmented reasoning reliable at the process level. It combines scalar process rewards, aligned natural-language explanations, bias-aware look-ahead search, and step-level post-training to improve intermediate reasoning and final answers.

SIGIR 2025 First author Zhongxiang Sun, Qipeng Wang, Weijie Yu, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Song Yang, Han Li
Paper Code Research Program
ReARTeR visual abstract

Research Question

A process-rewarding framework for trustworthy multi-step retrieval-augmented reasoning.

How can retrieval-augmented reasoning receive trustworthy supervision throughout test-time search and post-training?

Paper Walkthrough

From the research problem to mechanism and evidence

The figures below follow their order in the paper. Each section connects the figure to the question it resolves and the conclusion it supports.

1

Problem

Why final-answer supervision is too coarse

The example figure follows an X.com question through generator steps, retriever outputs, look-ahead search, PRM scoring, and PEM refinement.

ReARTeR Why final-answer supervision is too coarse figure

The left side gives a concrete process trace. The generator first asks about the company Elon Musk found in 1999 and searches adaptively; the retriever returns evidence that X.com was an online bank founded by Ed Ho, Harris Fricker, Elon Musk, and others. The next generator step asks for the current name of X.com, and the retriever returns evidence that X.com was rebranded as PayPal after merging with Confinity.

The right side shows why this trace needs process supervision. A tree of candidate steps is scored with look-ahead search, rollout, and backpropagation; the PRM gives process scores such as 0.3, 0.4, and 0.2, while the PEM refines a reasoning step using explanations about question decomposition, retrieval, and answer generation errors. The bottom legend breaks the process into sub-query, retrieval, and reasoning.

How to read this figure
  • Read the left trace as evidence that retrieval-augmented reasoning is a sequence of generator and retriever decisions.
  • Read the right tree as the search space over possible intermediate steps.
  • PRM scores select promising paths, while PEM explanations indicate what kind of process error should be refined.
2

Method

Process rewards at test time

The figure combines bias-reducing process search, PRM data collection, Monte Carlo filtering, and PEM-PRM alignment.

ReARTeR Process rewards at test time figure

Panel (a) explains how PRM training data are collected. Multiple rollouts e1, e2, and e3 produce candidate answers a1, a2, and a3. Their Monte Carlo correctness score MC decides whether the system should use a stronger generator, collect PRM data with OmegaPRM, or keep the example distribution balanced.

Panel (b) shows the bias-reduction motivation: shallow nodes can receive higher PRM bias than deeper nodes, and the bias-reducing score reduces this early-node preference. Panel (c) aligns PRM and PEM: PRM gives rewards r1 and r2 for steps, while PEM generates explanations c1 and labels positive or negative examples based on whether r2 is greater than r1.

How to read this figure
  • Panel (a) is about collecting less biased PRM training data from rollout correctness.
  • Panel (b) explains why shallow-node preference must be reduced during look-ahead search.
  • Panel (c) connects numerical PRM rewards with PEM explanations so the process score is more interpretable.
3

Method

Post-training with step-level feedback

The post-training figure shows warm-up from a strong generator and step-level offline RL over selected tree paths.

ReARTeR Post-training with step-level feedback figure

The left side shows a strong generator sampling reasoning data and a weak generator receiving SFT warm-up. The central box then runs interactive reasoning with a search tree where node 0 is selected, binary search expands candidate nodes, and PRM/PEM refine the selected path.

The bottom update line shows step-level offline reinforcement learning data such as path fragments with +1 and -1 labels. The right side ends with the generator after reasoning fine-tuning, showing that process rewards rank paths at inference time and improve the generator itself.

How to read this figure
  • The strong generator supplies warm-up data; the weak generator is improved through SFT before RL.
  • The selected tree nodes are turned into labeled step-level training sequences.
  • PRM and PEM appear inside the training loop, not only as external evaluators.
4

Results

Post-training scaling across retrieval-augmented reasoning datasets

The iteration curves show how step-level post-training changes retrieval-augmented reasoning accuracy across 2WikiMultiHopQA, Bamboogle, HotpotQA, and Musique.

ReARTeR Post-training scaling across retrieval-augmented reasoning datasets 2WikiMultiHopQA panel

2WikiMultiHopQA

ReARTeR Post-training scaling across retrieval-augmented reasoning datasets Bamboogle panel

Bamboogle

ReARTeR Post-training scaling across retrieval-augmented reasoning datasets HotpotQA panel

HotpotQA

ReARTeR Post-training scaling across retrieval-augmented reasoning datasets Musique panel

Musique

These panels correspond to the paper's post-training scaling analysis. Each curve tracks how performance changes across reinforcement fine-tuning iterations, making the training effect visible as a process rather than as a single final table entry.

The dataset-level separation is useful for project-page readers because it shows where process-reward training is stable and where gains are more task-dependent. It also connects the method figure's step-level offline RL loop to empirical behavior across multi-hop reasoning benchmarks.

How to read this figure
  • Read each panel as a dataset-specific trajectory over post-training iterations.
  • Compare curve shape, not only final height, because the figure is about scaling behavior over training.
  • These results provide the empirical counterpart to the post-training pipeline in Paper Fig. 3.
5

Results

PEM and PRM alignment effects

The two bar charts evaluate whether aligning process explanations with reward scores improves reasoning-path refinement.

ReARTeR PEM and PRM alignment effects Improvement rate after PEM-guided refinement panel

Improvement rate after PEM-guided refinement

ReARTeR PEM and PRM alignment effects Accuracy after PEM and PRM alignment panel

Accuracy after PEM and PRM alignment

These panels isolate the role of PEM/PRM alignment. The improvement-rate chart asks whether explanations help repair weak intermediate steps, while the accuracy chart asks whether that explanation-guided repair improves the final retrieval-augmented reasoning outcome.

This matters because ReARTeR is not only assigning scalar process rewards. It uses explanations as part of the correction loop, so the webpage should show the evidence that explanation alignment has a measurable effect.

How to read this figure
  • Use the improvement-rate chart to read PEM as a refinement mechanism for low-quality steps.
  • Use the accuracy chart to connect explanation alignment back to task performance.
  • Together the panels support the trustworthiness claim: process rewards become more useful when paired with aligned natural-language explanations.

Experimental Evidence

Main experimental results

ReARTeR is evaluated with a proprietary generator under test-time scaling and an open-source generator under test-time plus post-training scaling. The table reports ACC-L, the GPT-4o judged answer accuracy, for Standard RAG, the strongest test-time baseline CR-Planner, and ReARTeR.

Multi-step retrieval-augmented reasoning (ACC-L)

GeneratorMethod2WikiBamboogleHotpotQAMusiqueStrategyQA
GPT-4o-miniStandard RAG0.2920.3280.4500.1880.674
GPT-4o-miniCR-Planner0.4780.5240.4160.2620.744
GPT-4o-miniReARTeR0.5340.5440.5060.3020.772
LLaMA3.1-8BStandard RAG0.2120.2160.3980.0980.674
LLaMA3.1-8BCR-Planner0.3500.3360.3500.0980.654
LLaMA3.1-8BReARTeR0.3640.4840.4340.2520.724

Values are on a 0-1 scale. 2Wiki abbreviates 2WikiMultiHopQA.

What the results show

  • ReARTeR outperforms CR-Planner on all ten generator-dataset comparisons in ACC-L.
  • Against Standard RAG, mean ACC-L increases by 14.5 percentage points with GPT-4o-mini and 13.2 points with LLaMA3.1-8B.
  • The gains across both generators support the combined value of trustworthy process scoring, explanation-guided refinement, and post-training.

Main Takeaways

What this project establishes

1

Balanced annotation improves process reward data

Standard Monte Carlo annotation can produce skewed labels and weak supervision on difficult questions. ReARTeR balances positive and negative process examples and uses stronger generators when ordinary rollouts collapse.

2

Look-ahead scoring and aligned explanations improve refinement

TD-based look-ahead scoring reduces early-step bias in the PRM, where shallow search nodes are harder to judge reliably from local information alone.

3

Process rewards support test-time and post-training scaling

At test time, process rewards guide candidate-path search and step refinement. During post-training, the same reward machinery guides MCTS collection of step-level preference data.

4

PEM-PRM alignment has measurable downstream effects

Aligning PEM explanations with PRM scores increases the rate at which explanation-guided refinement improves process reward and raises final task accuracy across the reported datasets.

Research Impact

Position in the broader research program

ReARTeR provides the process-rewarding foundation for metacognitive control. It changes retrieval-augmented reasoning from a final-answer task into a supervised execution process.

The broader impact is that trustworthy agents need reward signals for intermediate behavior. ReARTeR supplies this layer for retrieval-augmented reasoning, making later monitoring, reflection, and correction more actionable.

Resources and Citation

Paper Code Full Publication List
@inproceedings{sun2025rearter,
  title={ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding},
  author={Sun, Zhongxiang and Wang, Qipeng and Yu, Weijie and Zang, Xiaoxue and Zheng, Kai and Xu, Jun and Zhang, Xiao and Song, Yang and Li, Han},
  booktitle={Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  pages={1251--1261},
  year={2025}
}