Co-RL uses peer rewards for unsupervised reasoning
The paper proposes multi-agent RL to reduce reliance on ground-truth supervision for reasoning models.
Why it matters
The work targets a core bottleneck in reasoning-model training: the scarcity and cost of reliable ground-truth supervision. If validated beyond the reported setting, peer-rewarded multi-agent RL could offer a path to improving reasoning while reducing collapse risks seen in purely self-rewarding approaches.
The key points
- 1.Co-RL trains separate models with rewards from peers.
- 2.Cohort diversity is used to reduce correlated errors.
- 3.The paper targets reasoning without ground-truth reward labels.
A new arXiv paper introduces Co-RL, a cooperative multi-agent reinforcement learning framework for improving reasoning in language and vision-language models without ground-truth reward labels. The method trains multiple decoupled models, sharing no parameters, using rewards derived from peer models' completions. The authors report that more diverse cohorts, including different model families, sizes and rephrased training samples, reduce correlated errors that can cause self-reinforcing feedback loops.
⚡ Try this today
Read the paper before adopting self-rewarding RL pipelines, especially if your setup relies on a single model's feedback.
Sources & original reporting
This brief summarizes and links to reporting from the publishers below.
- arXiv cs.AICo-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RLAug 19, 12:00 PM↗
- arXiv cs.LGCo-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RLAug 19, 12:00 PM↗
- arXiv cs.AICo-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RLAug 19, 12:00 PM↗
- HF Daily PapersCo-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RLAug 19, 4:00 AM↗
Enjoyed this brief? Get the next one in your inbox.
More in Research
MemTrapBench tests cognitive traps in LLM memory use
The benchmark finds relevant stored memories can still hurt current-task reasoning.
AI4AI-Bench tests agents on training algorithm design
The arXiv benchmark evaluates whether LLM agents can rewrite training algorithms across frozen repositories.
Researchers target sparse attention for long-context AI
New papers propose sparse-attention and neural-memory methods to cut long-context inference costs.