ReflectRL trains on failed expert reasoning traces
The paper proposes using flawed expert trajectories as reflection signals in on-policy training.
Why it matters
The work targets a practical bottleneck in reasoning-model post-training: expert demonstrations become less useful when the expert fails on harder problems. If validated beyond the reported benchmarks, failed rollouts could become a reusable supervision source rather than wasted training data.
The key points
- 1.ReflectRL uses failed expert traces as reflection inputs.
- 2.The method targets on-policy reasoning-model post-training.
- 3.Reported experiments span 9 benchmarks and 4 LLM backbones.
A new arXiv paper introduces ReflectRL, a plug-and-play framework for on-policy training that uses failed expert trajectories, called Golden Negative Trajectories, instead of discarding them. The method first elicits reflective reasoning from those flawed traces, then transfers that behavior back into direct reasoning through a reflective-to-direct policy transition. Hugging Face’s paper listing notes experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy settings.
⚡ Try this today
Read the ReflectRL paper before discarding failed expert trajectories in reasoning-model post-training pipelines.
Sources & original reporting
This brief summarizes and links to reporting from the publishers below.
- arXiv cs.AIReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct ReasoningAug 5, 12:00 PM↗
- arXiv cs.LGPrefix-Guided On-Policy Distillation: Mining Golden Trajectories from RolloutsAug 4, 12:00 PM↗
- HF Daily PapersReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct ReasoningAug 4, 4:00 AM↗
Enjoyed this brief? Get the next one in your inbox.
More in Research
[Paper] Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning
We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iter
Google applies homomorphic encryption to private AI
Google says encrypted processing can help make private AI more practical.
Google advances private AI with homomorphic encryption
Google says it is making private AI more practical using homomorphic encryption.