Residual Context Diffusion

Unleashing the Potential of Diffusion LLMs via Residual Denoising

Yuezhou Hu^*1, Harman Singh^*1, Monishwaran Maheswaran^*1
Haocheng Xi¹, Coleman Hooper¹, Jintao Zhang¹, Aditya Tomar¹
Michael W. Mahoney¹, Sewon Min¹, Mehrdad Farajtabar²
Kurt Keutzer¹, Amir Gholami^†1, Chenfeng Xu^†1

¹University of California, Berkeley ²Apple

^*Equal contribution ^†Equal advising / Corresponding author

ICML 2026

Models Code arXiv OpenReview

Figure 1: Overview of the residual denoising mechanism in Residual Context Diffusion (RCD). Remasking happens during each denoising step, discarding low-confidence token even though they are already explicitly computed. To recycle computation from previous steps, our proposed method extracts information before it is discarded, and forwards it to the next denoising step.

Demo Video

Example generation on AIME24. RCD increases parallelism by 4x while maintaining the baseline's peak accuracy.

About RCD

Diffusion Large Language Models (dLLMs) offer parallel decoding benefits but often trail autoregressive models in accuracy. This is due to the "remasking" strategy which wastes intermediate computation on tokens that remain undecoded. We propose Residual Context Diffusion (RCD), a novel paradigm that transforms this wasted computation into a guiding signal. By treating latent representations of low-confidence tokens as residual updates, RCD allows models to progressively refine their knowledge. Our method achieves 4–11 percentage point gains on GSM8K/MATH500 and doubles accuracy on competition-level AIME benchmarks while requiring up to 4–5× fewer steps at the baseline's peak accuracy.

Why Residual Context Diffusion?

Diffusion LLMs (dLLMs) achieve high efficiency through parallel decoding, yet they consistently struggle to match the accuracy of Autoregressive (AR) models. While this gap is partly rooted in the inherent challenges of training dLLMs, it is further exacerbated by the inference-time "Remasking" strategy:

The Problem: Information Loss

In each denoising step, standard dLLMs discard all tokens that fall below a confidence threshold, resetting them back to [MASK]. This wastes the intermediate computation spent on low-confidence tokens, forcing the model to "start from scratch" at those positions in the next step.

The RCD Solution: Residual Recycling

Instead of discarding uncertain tokens, RCD treats them as Residual Context. By aggregating these "soft tokens" based on their Entropy, we transmit a continuous, semantically-rich signal to the next iteration, enabling the model to refine its "thinking" rather than re-guessing.

The RCD Method

Left

Vanilla dLLM

Tokens not selected are reverted to static mask embeddings, discarding rich probability information.

Center

RCD Inference

Calculates a weighted sum of unselected distributions (residuals) to refine ambiguous tokens over time.

Right

RCD Training

A Decoupled Strategy: A frozen Reference Model provides stable targets to train the Target Model's residual awareness.

RCD introduces an Entropy-Based Embedding Aggregation mechanism. Instead of discarding uncertain tokens, we calculate their normalized Shannon entropy to determine their contribution to the next step's input.

            Residual Injection: Integrates soft-token information into masked positions.
Two-Stage Training: Uses a lightweight Reference Model to provide stable proxy signals, overcoming the instability of recursive loops.
Self-Correction: Temperature-scaled entropy alignment during inference to match training statistics.

        

Experimental Results

Main Results

RCD consistently beats Sequential Denoising (SeqD) across every model / block-size setting, with the largest gains on competition-level AIME24/25, where accuracy often more than doubles. Confidence threshold 0.85; SeqD/RCD use a 16,384 sequence length, Chat uses 512 tokens (1,024 for AIME).

Model	Variant	GSM8K^†	MATH500	AIME24	AIME25
SDAR-4B-b32	Chat^‡	86.13	50.20	5.83	2.50
	SeqD	81.73	61.20	6.04	11.88
	RCD	84.91	65.40	9.17	17.08
SDAR-4B-b64	Chat^‡	85.90	49.80	6.25	1.67
	SeqD	78.85	56.80	4.17	7.29
	RCD	87.04	68.40	11.04	14.79
SDAR-8B-b32	Chat^‡	88.40	50.00	6.46	4.17
	SeqD	86.50	65.80	11.67	14.79
	RCD	90.45	76.20	18.96	20.00
SDAR-8B-b64	Chat^‡	88.32	51.60	5.20	2.50
	SeqD	82.87	64.20	7.08	9.79
	RCD	86.05	74.40	18.75	16.04

^† Potential data contamination was observed in the original Chat models on GSM8K, which may inflate the Chat baseline. ^‡ Chat variants are instruction-following models; SeqD/RCD are further adapted for mathematical reasoning.

Model	Variant	GSM8K	MinervaMath
LLaDA	Base	70.30	31.40
	SeqD	75.74	31.10
	RCD	78.09	37.00

LLaDA (global-attention dLLM): sequence length 512, single-token-per-step decoding.

Doubles hard-task accuracy: on SDAR-8B-b64 AIME24, RCD lifts accuracy from 7.08 → 18.75.
Scales with size & block: gains of 4–11 percentage points when scaling from 4B to 8B, widening as block size grows.
Generalizes across architectures: on global-attention LLaDA, RCD adds a ~6 point absolute gain on MinervaMath.

Efficiency: Accuracy vs. Parallelism

Pareto Frontier: Accuracy vs Tokens per Step

Pareto frontiers for SDAR models: Curves are generated by sweeping confidence thresholds from 0.5 to 1.0. Token per Step measures the average number of tokens generated in each parallel iteration; higher values indicate fewer total decoding steps.

Significant Speedup: RCD (green dashed line) consistently shifts the frontier toward the upper-left, achieving higher accuracy with fewer steps.
Stability: RCD enables stable generation at higher Token per Step regimes where the Sequential Denoising baseline fails to maintain coherence.
4-5× Gain: At the baseline's peak accuracy, RCD requires up to 4–5× fewer steps on complex reasoning tasks.

Citation

If you find this work useful, please cite:

@misc{hu2026residualcontextdiffusionlanguage,
      title={Residual Context Diffusion Language Models}, 
      author={Yuezhou Hu and Harman Singh and Monishwaran Maheswaran and Haocheng Xi and Coleman Hooper and Jintao Zhang and Aditya Tomar and Michael W. Mahoney and Sewon Min and Mehrdad Farajtabar and Kurt Keutzer and Amir Gholami and Chenfeng Xu},
      year={2026},
      eprint={2601.22954},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2601.22954},
}