研究人员改进面向工具型 agent 的蒸馏方法
多篇 arXiv 论文聚焦长程 agent 训练中的稀疏奖励和不可靠教师信号问题。
为什么重要
这些论文指向一个共同的研究方向：提升小型语言模型或 agentic 语言模型在多轮工具使用中的信用分配能力和训练稳定性。如果这些技术能在已报告基准之外得到验证，可能会让工具型 agent 的后训练更高效、更可靠。
核心要点
- 1.多篇论文聚焦长程 agent 训练中的 OPD 失效模式。
- 2.教师指导正越来越多地由偏离程度、不确定性、置信度或 rollout 质量来控制。
- 3.共同目标是在不放大劣质轨迹的前提下，实现更密集的信用分配。
近期一组 arXiv 论文提出了新的训练方法，用于结合工具调用、多轮交互的语言模型 agent，在强化学习之外引入 on-policy 蒸馏。这些方法针对一系列相关失效模式：奖励稀疏或延迟、错误工具调用后的连锁错误、长 rollout 效率低，以及当学生轨迹发生偏离时教师监督不再可靠。论文提出的方案包括轮次级 hindsight 信号、逐步重加权、退火式 OPD-RL 调度、由前缀引导的 rollout 分配，以及针对低信号 RL 分组的选择性教师指导。
⚡ 今天就能用
在训练长程工具型 agent 之前，应将会对教师信号进行重加权或过滤的 OPD-RL 混合方法，与普通 GRPO 或朴素 OPD 基线进行比较。
来源与原始报道
本简报汇总并链接到以下媒体的报道。
- arXiv cs.AIInstruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned PolicyAug 6, 12:00 PM↗
- arXiv cs.AIAgentic Reinforcement Learning with Observation-Calibrated Self-DistillationAug 6, 12:00 PM↗
- arXiv cs.AIPrivileged, but Biased: How PI-Conditioned Teachers Break Self-DistillationAug 6, 12:00 PM↗
- arXiv cs.LGInstruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned PolicyAug 6, 12:00 PM↗
- arXiv cs.LGPrivileged, but Biased: How PI-Conditioned Teachers Break Self-DistillationAug 6, 12:00 PM↗
- arXiv cs.LGReward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement LearningAug 6, 12:00 PM↗
- arXiv cs.LGAgentic Reinforcement Learning with Observation-Calibrated Self-DistillationAug 6, 12:00 PM↗
- arXiv cs.AITurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated ReasoningAug 5, 12:00 PM↗
- arXiv cs.AIAgentic Reinforcement Learning with Self-Distilled Reward ShapingAug 5, 12:00 PM↗
- arXiv cs.AIRubrics as Privileged Information for Open-Ended GenerationAug 5, 12:00 PM↗
- arXiv cs.CLAgentic Reinforcement Learning with Self-Distilled Reward ShapingAug 5, 12:00 PM↗
- arXiv cs.CLTurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated ReasoningAug 5, 12:00 PM↗
- arXiv cs.LGAgentic Reinforcement Learning with Self-Distilled Reward ShapingAug 5, 12:00 PM↗
- arXiv cs.LGRubrics as Privileged Information for Open-Ended GenerationAug 5, 12:00 PM↗
- arXiv cs.LGRoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility StatesAug 4, 12:00 PM↗
- arXiv cs.CLRoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility StatesAug 4, 12:00 PM↗
- arXiv cs.AIDRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer TrainingAug 4, 12:00 PM↗
- arXiv cs.AIGroup-Reflective Self-Distillation for Agentic Reinforcement LearningAug 4, 12:00 PM↗
- arXiv cs.AIInstruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-DistillationAug 4, 12:00 PM↗
- arXiv cs.AIPCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement LearningAug 4, 12:00 PM↗
- arXiv cs.AIDAPD: Dual-Anchored Policy DistillationAug 4, 12:00 PM↗
- arXiv cs.AIIs More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled ReasoningAug 4, 12:00 PM↗
- arXiv cs.LGGroup-Reflective Self-Distillation for Agentic Reinforcement LearningAug 4, 12:00 PM↗
- arXiv cs.LGDRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer TrainingAug 4, 12:00 PM↗
- arXiv cs.LGInstruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-DistillationAug 4, 12:00 PM↗
- arXiv cs.LGRoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility StatesAug 4, 12:00 PM↗
- arXiv cs.CLRoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility StatesAug 4, 12:00 PM↗
- arXiv cs.CLInstruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-DistillationAug 4, 12:00 PM↗
- HF Daily PapersTurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated ReasoningAug 4, 4:00 AM↗
觉得这篇简报有用？下一篇直接送到你的邮箱。