In 2025, as artificial intelligence advances rapidly, reinforcement learning (Reinforcement Learning, RL) has become a key method for enhancing the reasoning capabilities of large language models (Large Language Models, LLMs). Chain‑of‑Thought (CoT) techniques based on RL are widely applied within major reasoning models. Tsinghua University and the Shanghai Artificial Intelligence Laboratory jointly released a 117‑page (68 pages of main text) survey paper "A Survey of Reinforcement Learning for Large Reasoning Models" (arXiv:2509.08827) that systematically analyzes how RL transforms LLMs into Large Reasoning Models (LRMs), achieving breakthroughs in mathematics, programming, and agent tasks. The paper, authored by Kaiyan Zhang et al. (39 authors), was published on October 10, 2025, and focuses on RL's core components, training resources, applications, and challenges, marking RL's evolution from an alignment tool to a reasoning engine. This article provides a deep technical interpretation of that survey, dissecting RL's mathematical foundations, novel mechanisms, and future potential.
Figure | Survey overview. The paper presents foundational components of reinforcement learning (RL) used for large reasoning models (LRMs), as well as open questions, training resources, and application scenarios. The survey centers on large‑scale interactions between language agents and environments over long‑term evolution.
Introduction: Paradigm Shift from LLMs to LRMs
Traditional LLMs such as GPT‑4 excel at natural language generation but often “stall” on complex reasoning tasks (e.g., multi‑step mathematical proofs or code debugging). Static supervised fine‑tuning (Supervised Fine‑Tuning, SFT) struggles to capture dynamic decision trajectories. RL changes this picture: using the Markov decision process (MDP) framework, RL treats an LLM as an agent, prompts as initial states, generated sequences as actions, and environment feedback as reward signals to optimize policies.
The paper emphasizes that RL‑driven milestone models—such as OpenAI o1 (closed‑source, RLVR + compute‑at‑inference, SOTA in mathematical reasoning), DeepSeek‑R1 (open‑source, GRPO + verifiable rewards, programming capability comparable to o1), Claude‑3.7‑Sonnet (multimodal RL, strong at tool use), Gemini 2.0 (dynamic sampling, leading in visual reasoning), and Qwen3 (RL + SFT fusion, strong multi‑task generalization)—demonstrate RL's ability to elicit long‑chain reasoning (Long Chain‑of‑Thought, Long CoT) and self‑correction mechanisms. Compared with RLHF (Reinforcement Learning from Human Feedback), which mainly optimizes “style of response,” RLVR (Verifiable Rewards) directly targets reasoning correctness and provides more reliable gradient signals.
However, RL is not a panacea: instability in training, reward hacking, and computational cost remain pain points. By comparing RL with SFT/DPO (Direct Preference Optimization), the paper notes RL's superior out‑of‑distribution generalization but recommends hybrid strategies to balance efficiency. The survey fills gaps in the literature and systematically traces RL's contributions toward artificial superintelligence (Artificial Superintelligence, ASI).
Reinforcement Learning from Verifiable Rewards | Label Studio
Prerequisites: The Mathematical Framework of RL in LRMs
The core of RL is the MDP tuple (S, A, P, R, γ), where S is the state space (prompt + generated sequence), A is the action space (next token), P is the transition dynamics (LLM generation probabilities), R is the reward function (correctness score), and γ is the discount factor (0<γ<1, balancing short‑ and long‑term rewards). In LRMs, an LLM policy π(θ) maximizes expected return J(θ)=E[∑γ^t r_t], with gradient ∇J(θ)≈E[∇logπ(a|s) A(s,a)], where A denotes the advantage function.
The paper reviews a timeline of leading models: from RLHF (InstructGPT, 2022) to RLVR (o1, 2025). Open‑source examples like DeepSeek‑R1 adopt weak priors (learning CoT from scratch), while closed‑source systems like Claude emphasize strong priors (injecting pretraining knowledge). Unlike SFT's static labels, RL dynamically generates data, alleviating “data hunger.” But RL requires high compute: training o1 consumed tens of thousands of H100 GPU hours.
Core Components: Reward Design, Policy Optimization and Sampling Strategies
Reward Design: RL's "Compass"
The reward function R determines optimization direction. The paper classifies rewards into five types: verifiable rewards (rule checks, e.g., DeepSeek‑R1's math solver: r=1 if correct else 0 — reliable but weak on subjective tasks); generative rewards (model feedback, e.g., rubric‑based — flexible but costly); dense rewards (step‑level feedback, e.g., token‑level entropy to mitigate credit assignment); unsupervised rewards (internal consistency, e.g., ZeroGUI); and reward shaping (turning sparse rewards into dense ones, e.g., optimizing Pass@k).
An innovation is hybrid mechanisms: DeepSeek‑R1 combines rules and human preferences to reduce hacking risk. Mathematically, reward shaping uses a baseline b(s) to normalize A(s,a)=r+γV(s')‑b(s), stabilizing gradients.
Reinforcement Learning from Verifiable Rewards | Label Studio
Policy Optimization: Evolution from PPO to GRPO
Policy optimization is central to RL‑driven LRMs, aiming to maximize expected cumulative reward 
. PPO (Proximal Policy Optimization) clips probability ratios 
to limit update magnitude, preventing destructive large steps and ensuring training stability, but is computationally intensive. REINFORCE, as a basic Monte Carlo gradient method, suffers from high variance. VCPPO introduces a value critic baseline to significantly reduce noise.
Among critic‑free algorithms, GRPO (Group Relative Policy Optimization) constructs advantage functions via within‑group relative ranking, greatly reducing variance and suiting resource‑constrained settings. CISPO/GSPO incorporate contrastive learning ideas and use 
regularization to balance exploration and conservative updates.
The paper's comparisons show PPO is stable but requires frequent value estimation; GRPO is efficient without extra networks; off‑policy methods like PPER (Prioritized Experience Replay) prioritize high TD‑error samples to boost efficiency. Off‑policy update formula 
can combine with DQN for discrete action spaces, suitable for structured tasks like tool selection.
Sampling Strategies: The Art of Dynamic Exploration
Sampling strategies are key to the exploration–exploitation trade‑off in RL‑driven LRMs. PRIME implements dynamic temperature adaptation (τ ∝ reward): high‑reward trajectories reduce randomness to favor exploitation, while low rewards increase temperature to encourage exploration. TreeRL uses a tree search structure that shares common prefixes to cut redundant computation and improve long‑chain reasoning efficiency. DeepScaleR leverages meta‑learning to optimize CoT depth hyperparameters, balancing compute and performance. The paper highlights ARPO (Adaptive Rejection Sampling), which combines rejection sampling with adaptive thresholds to encourage tool use (e.g., code executors, calculators), improving generation quality and efficiency on long‑sequence tasks such as math solving and multi‑step agent scenarios.
Fundamental Issues: Dialectical Analysis of Seven Major Controversies
The paper uniquely delves into seven philosophical controversies around RL‑driven LRMs, exposing methodological essence and practical tensions:
- "Refinement" or "Discovery"? DeepSeek‑R1, trained from weak priors, shows the ability to “discover” new reasoning paths such as implicit CoT (chain‑like reasoning without explicit prompts); OpenAI o1 relies on strong supervision and compute at inference, favoring the “refinement” of known strategies.
- RL vs SFT: RL wins in out‑of‑distribution generalization and can explore unseen task spaces; supervised fine‑tuning (SFT) is memory‑efficient and suits knowledge‑dense tasks. UFT (Unified Fine‑Tuning) proposes a hybrid paradigm: SFT to inject knowledge, then RL to elicit reasoning.
- Model prior strength: Weak priors (e.g., R1) encourage innovation and can produce non‑traditional solutions; strong priors (e.g., Claude) ensure stability and reduce hallucinations.
- Training recipes: Techniques like KL regularization and value clipping improve convergence, but risks such as entropy collapse and mode collapse must be monitored.
- Reward granularity: Process supervision (dense rewards) guides intermediate steps, while outcome verification (verifiable rewards) ensures final correctness; hybrid rewards balance exploration and precision.
6–7. Multimodal and agent extensions: RL empowers vision–language reasoning and tool invocation but can amplify bias and ethical risks, requiring transparent auditing and alignment mechanisms.
These controversies highlight RL's double‑edged nature: potential drives breakthroughs, but risks demand standardized training protocols and ethical frameworks. Future work should establish unified benchmarks and open, reproducible pipelines to safely advance toward general reasoning intelligence.
A Deep Dive into Policy Optimization Algorithms & Frameworks for ...
Training Resources: From Static Data to Dynamic Environments
RL‑driven LRMs rely on abundant training resources, divided into static corpora and dynamic environments.
Static corpora include: Big‑Math (10,000 math problems), HumanEval (164 programming tasks), MedXpertQA (medical diagnosis Q&A), etc., which provide high‑quality supervision signals.
Dynamic environments support real‑time interactive feedback and are categorized as: rule‑based (e.g., SymPy math solver), code‑based (online compile‑and‑run unit tests), game‑based (Minecraft build and navigation tasks), and model‑based (GUI simulators driving vision–language interaction). Table 4 in the paper shows that SWE‑bench contains over 2,000 real GitHub repositories covering complex software engineering scenarios.
RL infrastructure greatly lowers training barriers: OpenRLHF supports asynchronous policy–value updates; TRL integrates tool calls and rejection sampling; veRL enables interruptible long‑sequence generation; AReaL extends to multimodal RL. These open‑source frameworks accelerate community innovation and push LRMs from lab prototypes toward industrial‑grade systems.
Applications: RL's Multidimensional Deployments
Coding and Agent Tasks
In coding, Claude‑4.1‑Opus optimized SWE‑bench with unit‑test rewards, achieving over 40% accuracy; R1‑Code‑Interpreter blends supervision and RL to improve debugging. In agent tasks, multi‑turn interaction models like Kimi K2 and UI‑TARS (GUI grounding with IoU rewards) reach 85% precision.
Applications of Multi‑Agent Deep Reinforcement Learning ...
Multimodal, Robotics and Healthcare
RL applications in LRMs extend to multimodal, robotic, and medical domains, demonstrating cross‑domain potential.
- Multimodal: InternVL3.5 leads visual reasoning via a Cascade RL framework (offline + online stages), improving tasks like MMMU by 16% and supporting GUI interaction; Video‑R1 introduces T‑GRPO to strengthen temporal CoT, fusing image–video data and achieving 35.8% accuracy on VSI‑Bench, surpassing GPT‑4o.
- Robotics: SimpleVLA‑RL uses GRPO to optimize action models, addressing data scarcity with binary success rewards and exploration modulation, improving long‑horizon planning and sim‑to‑real transfer and generalizing beyond SFT on the LIBERO benchmark.
- Healthcare: Med‑U1 employs hybrid rule rewards (binary + length penalties), raising diagnostic accuracy by 20%; BoxMed‑RL emphasizes format rewards to ensure structured reports and improve clinical consistency.
Multi‑agent systems such as LLaMAC use actor–critic frameworks for collaborative reasoning and large‑scale decision optimization; MARTI supports multi‑agent RL training with natural language feedback, enhancing team decision efficiency and advancing collective intelligence.
Challenges and Future Directions: From Bottlenecks to Breakthroughs
Although RL‑driven LRMs show strong potential, they face five core challenges:
- 1. Training instability: entropy collapse can force policies to converge prematurely to low‑diversity outputs; entropy regularization H(π)>β can enforce exploration and maintain policy diversity;
- 2. Insufficient exploration: fixed‑temperature sampling struggles with exploration–exploitation balance; ε‑greedy or adaptive temperature mechanisms (e.g., PRIME) can dynamically adjust;
- 3. High compute cost: long‑chain reasoning requires massive GPU hours; mixtures‑of‑experts (MoE) and parallel sampling can substantially reduce costs;
- 4. Weak generalization: performance on out‑of‑distribution tasks (e.g., novel math problems) is poor; adversarial data augmentation and meta‑learning can improve robustness;
- 5. Poor interpretability: black‑box decisions are hard to audit; attention visualization, attribution analysis, and causal interventions are key.
Future directions for LRMs include twelve cutting‑edge breakthroughs: continual RL (replay buffers for long‑term memory); memory augmentation (external vector DBs to store trajectories); model‑based rewards (world models predicting future states to produce intrinsic rewards); efficient teaching (adaptive compute‑budget allocation); latent‑space optimization (mapping discrete tokens to continuous spaces to accelerate gradients); RL pretraining (using next‑token prediction as reward to build general policies); diffusion RL (extending ODE solvers to SDEs for image/video generation); closed‑loop scientific discovery (self‑generated hypothesis–validation loops); architecture co‑design (MoE dynamic routing + RL joint training); self‑supervised rewards (self‑rewarding models that self‑evaluate); test‑time RL (TTRL, online fine‑tuning during inference); and multi‑agent chain reasoning (Chain‑of‑Agents with sub‑agent specialization). Together these point toward self‑evolving LRMs capable of long‑range planning and world modeling.
Conclusion: RL as a Bridge to ASI
This survey focuses on RL‑driven large reasoning models (LRMs) and shows that verifiable‑reward RL (RLVR) can upgrade LLMs from "speaking" to "reasoning." Unlike alignment methods such as RLHF, RLVR is result‑oriented and systematically reviews core mechanisms including reward design, policy optimization, and sampling strategies, analyzes training controversies, summarizes static/dynamic resources and infrastructure, and surveys applications in coding, agents, and multimodal domains. In the future, RL may become a key engine toward superintelligence, enabling language models to achieve long‑chain reasoning, self‑evolution, and world modeling breakthroughs.