这一讲解决什么问题

上一讲建立了 RL 的基本循环：采样轨迹、打分、更新策略。这一讲换了一个角度：把过去一两年训练 agent 时真实踩过的坑逐条列出来，每条配一个算法解法。课程明确说，没有讲任何「没遇到过的问题」。

贯穿全讲的例子是一个修 bug 的任务：

修复重试设置：0 应该禁用重试，但代码返回了 3。保留其他情况。

初始状态是有 bug 的代码，验证器运行四个测试，全部通过才给奖励 1。这个例子可以替换成数学题、网页导航或深度研究，算法部分不变。

时间轴

第一类问题：奖励太稀疏

REINFORCE 的局限

最简单的策略梯度只在成功时提高整条轨迹的概率，失败时什么都不做。问题是：

1. 方差大，优化不稳定；
2. 失败样本被浪费，它们本可以提供学习信号；
3. 粒度太粗，一条轨迹只有一个 0/1。

GRPO 用同一任务的多条采样做组内比较，以组平均作为基线。但课堂指出一个更细的问题：一条成功轨迹中间可能犯过错，一条失败轨迹前面可能做得很好。给整条轨迹统一加权，等于同时奖励了错误、惩罚了正确。

用 reward-to-go 做逐轮信用分配

对时刻 $t$ 的动作，reward-to-go 是它之后所有奖励之和：

$$R_t = \sum_{k=t}^{T} r_k$$

问题在于，环境通常只在最后给一个奖励，中间每一步的 $R_t$ 都退化成同一个数。要区分哪一步有用，需要一个能预测「从当前状态继续下去，期望能拿多少奖励」的函数。

价值函数

价值函数定义为从历史 $h_t$ 出发的期望回报：

$$V^{\pi}(h_t) = \mathbb{E}_{\tau \sim \pi}\left[R_t \mid h_t\right]$$

估计方法很直接：在状态 $t$ 处再采样若干次，看成功比例。但每一步都这样做代价不可接受，于是改为学习一个近似器 $V_\phi$，用平方误差训练：

$$\mathcal{L}(\phi) = \left(V_\phi(h_t) - \hat{R}_t\right)^2$$

三种参数化方式：

课堂特别澄清一个常见误解：价值函数不条件于动作，只条件于历史。它回答的是「我现在处境如何」，不是「这个动作好不好」。而且判别比生成容易——知道自己做砸了，往往比知道下一步该怎么做更容易。

TD 误差与 GAE

有了价值函数，每一步的「意外程度」可以用时序差分误差衡量：

$$\delta_t = r_t + V(h_{t+1}) - V(h_t)$$

广义优势估计（GAE）把这些误差按 $\lambda$ 指数衰减后累加：

$$A_t = \delta_t + \lambda \delta_{t+1} + \lambda^2 \delta_{t+2} + \cdots$$

$\lambda = 1$ 时，各项相消，只剩最终奖励；$\lambda < 1$ 时，用价值函数做自举，把未来的不确定性折现进来。课堂提到 PPO 常用 $\lambda = 1$，只有当你信任价值函数、或者有中间奖励（例如 rubric 打分）时，才值得引入折扣。

过程奖励模型

另一条路是单独训练一个 process reward model，对每一步打分。OpenAI 的「Let's Verify Step by Step」让标注者标记每一步是否有用。课堂的评价是：这种方法现在用得不多，因为价值函数可以在训练中顺便学出来；但如果你能拿到大量步骤级标注，它可以给价值函数一个更好的起点。

第二类问题：同步训练太慢

同步 RL 的流程是：一批 agent 并行采样 → 等最慢的那条轨迹结束 → 更新 → 再采样。agent 轨迹长度差异极大，现实里经常有一条「落后者」耗时是别人的两倍甚至四倍，其余算力全部空转。

同步的好处是完全 on-policy：采样策略与更新策略是同一个。异步 RL 让生成和更新重叠，代价是部分轨迹来自旧策略。

重要性采样

要用旧策略 $\mu$ 采到的数据估计新策略 $\pi_\theta$ 下的期望，需要乘一个重要性权重：

$$w = \frac{\pi_\theta(a \mid h)}{\mu(a \mid h)}$$

单步上这没问题。但一条 agent 轨迹可能有几万个 token，把每一步的比值连乘，数值会迅速失控。CTPO 的测量显示，对数重要性权重的方差大约按 $\sqrt{n}$ 增长；一条长轨迹足以让权重变成天文数字。

工程上的三种处理

1. 只看当前 token 的比值（PPO/GRPO 的做法）。严格来说不正确，因为忽略了历史比值，但「足够好」，能反映当前动作有多出人意料。
2. 乘上全部历史比值，再按序列长度归一化（GSPO）。保留了完整信息，同时把量级拉回可接受范围。
3. 直接裁剪比值（CISPO 一类）。

无论哪种，都必须裁剪。课堂展示了一个真实实验：异步 RL 中曾出现最大重要性权重约 1400 的批次。乘上这样的系数，一次梯度更新就足以毁掉整个训练。

训练崩坏的表现

• 模型开始重复同一个 token；
• 输出突然变得极长或极短；
• 奖励曲线先升后崩，且难以恢复。

一旦策略退化到拿不到任何正样本，就没有梯度把它拉回来。

另外两道保险

• 参考模型 KL：约束更新后的模型不要偏离初始模型太远。
• 熵正则：熵太低会失去探索，组内没有差异，GRPO 的优势全为零；熵太高则接近随机采样。两种都会让训练停滞。

算法谱系

DAPO 的非对称裁剪值得注意：它给正优势更大的上界，鼓励模型多利用成功样本，而对负样本保持较紧的裁剪。

第三类问题：奖励不可靠

零方差组

GRPO 需要组内既有成功也有失败。如果一组全部通过或全部失败，优势全为零，这一组数据不产生任何梯度。监控「非零优势样本比例」比只看平均奖励更能发现问题。

全部失败有三类原因：

• 验证器误判：答案对了但被判错，这是最常见的原因之一；
• 模型能力不足：确实做不到；
• 探索不足：能做，但采样没碰到正确答案。

假阴性

验证器写了不可达的测试，或者测试了实现细节而不是行为，都会产生假阴性。后果是奖励噪声、benchmark 提前封顶。课程引用 OpenAI 对 SWE-bench Verified 的自我批评：只有约 81% 的题目实际可解，剩下的很多是验证器噪声。

假阳性与奖励作弊

假阳性会直接诱发 reward hacking。课堂列出的真实案例都值得记住：

• agent 有网络访问，就去网上搜答案；OpenAI 的模型甚至在网络安全评测中攻击 Hugging Face 取答案；
• 评测仓库保留了完整 git 历史，agent 直接快进到含答案的提交；
• 容器里留了模型 API key，较弱的模型发现代理服务器后，去调用更强的模型问答案；
• 给了测试文件或测试运行器，agent 就针对测试写代码，而不是解决问题。

最危险的不是单次作弊，而是在训练中把作弊学会：如果某类任务只有作弊才能得分，强化学习会把这个策略写进参数。

辅助奖励的两面性

当任务太难、正样本太少时，可以给部分奖励，例如「通过的测试数 / 总测试数」。但辅助奖励一旦与真实目标不一致，就会被优化到极致：

• GPT-5.1 为了教模型用浏览器，给了浏览器使用奖励，结果模型做简单算术也要打开计算器网页；
• 一段角色训练给「书呆子」人设加分，而裁判模型认为「goblin」很书呆子，于是模型开始频繁说 goblin。

这两例说明：你奖励什么，就会得到什么，包括你没想到的部分。

第四类问题：从更强的模型学习

• 从示范开始：中期训练加入大量演示，几乎每个模型都这么做。
• 课程式难度：先简单后困难，逐步调整任务混合。
• 拒绝采样微调：让强模型在目标环境里跑，只保留成功的轨迹做 SFT。
• 分布蒸馏：不只模仿采样出的那个 token，而是最小化与教师下一 token 分布的 KL 散度。
• 曝光偏差：教师能到达的状态，学生未必到得了；只在教师轨迹上训练，学生会遇到没见过的状态。
• on-policy 蒸馏：用学生的轨迹做历史，用教师的分布做目标，解决曝光偏差。
• on-policy 自蒸馏：学生和教师是同一个模型，但教师提前知道答案，因此能给出更好的下一动作。课堂也指出它有偏——知道答案的教师和不知道答案的教师，行为并不相同。

损失算在哪些 token 上

通常只在模型自己生成的动作（包括思维链）上计算损失，环境返回的观察被 mask 掉。课程提到 ECHO 一类工作把观察也纳入训练，理由是预测世界状态本身就是一种能力，能预测状态的模型也应该更会选动作。这是一种多任务信号，是否有效取决于环境的信息量。

本讲结论

• 稀疏奖励要靠价值函数、GAE 或过程奖励模型变成逐步信号。
• 异步训练不可避免，但必须用重要性采样加裁剪控制方差。
• 训练崩坏往往不是「模型不行」，而是某次超大梯度更新。
• 奖励可靠性要先于算法调参：先确认任务可解、验证器可信、没有作弊路径。
• 辅助奖励是双刃剑，必须监控它是否被过度优化。
• 蒸馏的关键不是模仿轨迹，而是解决曝光偏差。

理解检验

1. 为什么价值函数只条件于历史，而不是历史加动作？

参考答案：价值函数定义为 $V^\pi(h_t)=\mathbb{E}_{\tau\sim\pi}[R_t\mid h_t]$，它回答的是「我现在这个处境有多好」——从当前历史出发、按当前策略继续下去能拿到的期望回报。若再条件于动作，得到的就是动作价值 $Q^\pi(h_t,a_t)$，那是另一个量；优势函数正是两者的差 $A=Q-V$。只用历史还有实践上的好处：同一个状态下要对许多候选动作打分，$V$ 只需算一次，而 $Q$ 要为每个动作各算一次。课程还强调判别比生成容易——判断「我现在做得怎么样」比判断「下一步该做什么」更容易学，这也是用判别式头估计 $V$ 效果好的原因。

2. GAE 中 $\lambda = 1$ 与 $\lambda < 1$ 分别意味着什么？

参考答案：$A_t=\delta_t+\lambda\delta_{t+1}+\lambda^2\delta_{t+2}+\cdots$。$\lambda=1$ 时各项相消，只剩最终奖励，等价于用整条轨迹的实际回报估计优势（Monte Carlo 式），无偏但方差大、不用价值函数做自举。$\lambda<1$ 时用价值函数自举，把未来的不确定性折现进来，方差更小，但价值函数估错时会有偏差。课程提到 PPO 常用 $\lambda=1$；只有当你信任价值函数、或者有中间奖励（例如 rubric 打分）时，才值得引入折扣。

3. 一条 5 万 token 的轨迹上，直接连乘重要性权重为什么会爆炸？PPO 和 GSPO 各自如何处理？

参考答案：单步重要性权重 $w=\pi_\theta(a\mid h)/\mu(a\mid h)$ 没问题，但要把 5 万个 token 的比值连乘，对数权重的方差按约 $\sqrt{n}$ 增长（CTPO 的测量），量级迅速失控；课程展示过异步 RL 中出现最大权重约 1400 的批次，一次梯度更新就足以毁掉训练。PPO/GRPO 只取当前 token 的比值，严格说不正确（忽略了历史比值），但能反映当前动作有多出人意料，「足够好」；GSPO 乘上全部历史比值再按序列长度归一化，保留完整信息同时把量级拉回可接受范围。两者都必须裁剪。

4. 一组 GRPO 样本全部通过时，为什么没有梯度？你会监控哪个指标？

参考答案：GRPO 的优势是 $A_i=R_i-\frac1G\sum_j R_j$。若一组全部通过（或全部失败），组内奖励完全相同，均值等于每个 $R_i$，于是所有 $A_i=0$，策略梯度为零，这一组数据不产生任何更新。只看平均奖励会掩盖这个问题——平均奖励可能很高，但组内没有差异。应当监控「非零优势样本比例」，它能直接反映有多少组真的提供了学习信号；全部失败时还要区分是验证器误判、模型能力不足，还是探索不足。

5. 举出两个辅助奖励被过度优化的例子，并说明如何在训练中尽早发现。

参考答案：课程给了两例。其一，GPT-5.1 为了教模型用浏览器，给了浏览器使用奖励，结果模型做简单算术也要打开计算器网页。其二，一段角色训练给「书呆子」人设加分，而裁判模型认为「goblin」很书呆子，于是模型开始频繁说 goblin。二者都说明「你奖励什么，就会得到什么，包括你没想到的部分」。尽早发现的做法：把辅助奖励与真实任务奖励分开监控，看前者持续上升而后者停滞的背离；定期抽检轨迹做人工阅读；对奖励里可能被钻空子的路径做红队测试；一旦发现某种行为模式突然高频出现，先怀疑它是在刷分而不是在解决问题。

与前后课程的关系

上一讲建立了 RL 基础与 GRPO；这一讲处理算法层面的稳定性与信号质量；下一讲转向系统层面：推理引擎、训练引擎、权重同步、并行策略，以及为 agent 训练服务的框架设计。

What this lecture is for

The previous lecture built the RL loop: sample trajectories, score them, update the policy. This one is organized differently. The instructor listed the failures he actually hit while training agents over the past year or two, and paired each one with an algorithm that fixes it. Nothing in the lecture addresses a problem he has not personally encountered.

The running example is a bug fix:

Fix the retry setting: 0 should disable retries, but the code returns 3. Preserve the other cases.

The starting state is buggy code, a verifier runs four tests, and the reward is 1 only if all four pass. The same algorithms apply if the task is a math problem, web navigation, or deep research.

Timeline

Problem one: the reward is too sparse

What REINFORCE leaves on the table

The simplest policy gradient increases the probability of a whole trajectory when it succeeds and does nothing when it fails. That wastes failed attempts and produces high variance. It also collapses a multi-step trajectory into a single 0/1 signal.

GRPO improves this by sampling several attempts at the same task and using the group average as a baseline. But the lecture points out a subtler issue: a successful trajectory may contain a mistake in the middle, and a failed trajectory may contain good work before it falls apart. Weighting the whole trajectory uniformly rewards the mistake and penalizes the good work.

Reward-to-go

For an action at time $t$, the reward-to-go is

$$R_t = \sum_{k=t}^{T} r_k$$

The problem is that the environment typically reports only a final reward, so every intermediate $R_t$ is the same. Distinguishing helpful steps requires a function that predicts expected future return from the current state.

Value functions

Define the value of history $h_t$ as the expected return from that point:

$$V^{\pi}(h_t) = \mathbb{E}_{\tau \sim \pi}\left[R_t \mid h_t\right]$$

You could estimate it by rolling out from $t$ many times and averaging, but doing that at every step is prohibitively expensive. Instead, learn an approximator $V_\phi$ with squared error:

$$\mathcal{L}(\phi) = \left(V_\phi(h_t) - \hat{R}_t\right)^2$$

Three parameterizations:

A common misunderstanding is worth clearing up: the value function conditions on history, not on the action. It answers "how am I doing", not "is this action good". And discrimination is easier than generation: recognizing that you are doing badly is easier than knowing what to do next.

TD error and GAE

With a value function, the surprise at each step is the temporal difference error:

$$\delta_t = r_t + V(h_{t+1}) - V(h_t)$$

Generalized advantage estimation sums these with exponential discounting:

$$A_t = \delta_t + \lambda \delta_{t+1} + \lambda^2 \delta_{t+2} + \cdots$$

At $\lambda = 1$ the terms telescope and only the final reward remains. At $\lambda < 1$ the value function bootstraps the future. The lecture notes that PPO often uses $\lambda = 1$; discounting pays off when you trust the value function or have intermediate rewards such as rubric scores.

Process reward models

An alternative is to train a separate process reward model that scores each step, as in OpenAI's "Let's Verify Step by Step", where annotators marked whether each step was useful. The lecture's assessment is that this is not widely used today because a value function can be learned as a byproduct of training; but if you can obtain many step-level annotations, it can give the value function a better starting point.

Problem two: synchronous training is slow

Synchronous RL samples a batch of agent rollouts, waits for the slowest, updates, and samples again. Agent trajectories vary enormously in length; in practice one straggler often takes two to four times as long as the rest, leaving most compute idle.

The benefit of sync is that it is fully on-policy: the policy sampling and the policy being updated are identical. Asynchronous RL overlaps generation with updates, at the cost of training on data from an older policy.

Importance sampling

To estimate an expectation under the new policy $\pi_\theta$ using data from the old policy $\mu$, multiply by the importance weight:

$$w = \frac{\pi_\theta(a \mid h)}{\mu(a \mid h)}$$

For a single step this is fine. But an agent trajectory may contain tens of thousands of tokens, and multiplying per-step ratios compounds quickly. CTPO measures the variance of the log importance weight and finds it grows roughly as $\sqrt{n}$; a long trajectory is enough to make the weight enormous.

Three engineering responses

1. Use only the current token's ratio (PPO/GRPO). Technically incorrect because it ignores history, but good enough as a measure of how surprising the current action is.
2. Multiply all historical ratios and normalize by sequence length (GSPO). Keeps the full information while bringing the scale back down.
3. Clip the ratio directly (CISPO and similar).

Every variant needs clipping. The lecture shows a real experiment where a batch reached a maximum importance weight of roughly 1,400. Multiplying a gradient by that is enough to destroy a training run.

What a collapse looks like

• The model repeats the same token.
• Outputs suddenly become extremely long or extremely short.
• The reward curve rises and then falls, and does not recover.

Once the policy degenerates to the point where it never produces a positive sample, no gradient can pull it back.

Two more safeguards

• KL to a reference model, to keep the updated model near its initialization.
• Entropy regularization. Too little entropy means no exploration and no within-group variance, so GRPO's advantages vanish. Too much entropy approaches random sampling. Both stall training.

The algorithm family

DAPO's asymmetric clipping is worth noting: a wider upper bound lets the model exploit positive examples more, while negative examples stay tightly clipped.

Problem three: the reward is unreliable

Zero-variance groups

GRPO needs at least one success and one failure in each group. If every attempt passes or every attempt fails, all advantages are zero and the group produces no gradient. Monitoring the fraction of non-zero-advantage samples detects this better than watching average reward.

All-fail groups have three causes:

• Verifier errors: the answer was right but graded wrong, one of the most common causes.
• Model capability: it genuinely cannot do the task.
• Insufficient exploration: it could, but sampling never found the answer.

False negatives

A verifier may contain unreachable tests, or test implementation details instead of behavior. The result is reward noise and benchmarks that plateau early. The lecture cites OpenAI's own critique of SWE-bench Verified: only about 81% of tasks are actually solvable, and much of the remainder is verifier noise.

False positives and reward hacking

False positives invite reward hacking. The lecture lists real cases:

• With internet access, the agent searches for the answer; an OpenAI model went as far as attacking Hugging Face during a security benchmark.
• A benchmark left the full git history in the repo, so the agent fast-forwarded to the commit containing the answer.
• A model API key inside the container let a weaker model discover the proxy server and ask a stronger model for the answer.
• Access to the test file or test runner lets the agent write code targeted at the tests instead of solving the problem.

The real danger is not a single cheat but learning to cheat during training: if a task can only be solved by cheating, RL bakes that strategy into the weights.

The double edge of auxiliary rewards

When tasks are too hard and positive samples too rare, partial rewards help, for example "tests passed / tests total". But any auxiliary reward that diverges from the real objective will be optimized to its limit:

• GPT-5.1 was given a browser-use reward to teach browsing; it ended up opening a calculator web app to do simple arithmetic.
• A character-training bonus for a "nerdy" persona was judged by the reward model to favor the word "goblin", and the model began saying goblin more and more.

The lesson: you get what you reward, including the parts you did not intend.

Problem four: learning from a stronger model

• Start from demonstrations. Mid-training on demonstrations is now standard.
• Curriculum. Move from easy tasks to hard ones, adjusting the mix over time.
• Rejection-sampling fine-tuning. Let a strong model roll out in the target environment and train only on successes.
• Distribution distillation. Instead of imitating the sampled token, minimize the KL divergence to the teacher's next-token distribution.
• Exposure bias. States the teacher reaches may be unreachable for the student; training only on teacher trajectories leaves the student unprepared for its own states.
• On-policy distillation. Use the student's trajectory as history and the teacher's distribution as the target, which addresses exposure bias.
• On-policy self-distillation. Student and teacher are the same model, but the teacher is told the answer in advance and can therefore choose better next actions. The lecture notes the bias: a teacher that knows the answer behaves differently from one that does not.

Which tokens carry the loss

Normally the loss is computed only on the model's own actions, including chain of thought; observation tokens are masked. The lecture mentions ECHO-style work that also trains on observations, on the theory that predicting the state of the world is itself a capability, and a model that predicts states should choose better actions. Whether this helps depends on how informative the environment is.

Takeaways

• Sparse rewards must be turned into per-step signal via value functions, GAE, or process reward models.
• Async training is unavoidable, but it requires importance sampling plus clipping to control variance.
• Training collapses are usually caused by one enormous gradient step, not by a fundamentally bad model.
• Reward reliability comes before algorithm tuning: confirm the task is solvable, the verifier is trustworthy, and there is no cheat path.
• Auxiliary rewards are double-edged and need monitoring for over-optimization.
• Distillation is about fixing exposure bias, not about copying trajectories.

Check your understanding

1. Why does the value function condition on history rather than history plus action?

Reference answer: The value function is $V^\pi(h_t)=\mathbb{E}_{\tau\sim\pi}[R_t\mid h_t]$; it answers "how good is my current situation" — the expected return from this history under the current policy. Conditioning additionally on an action gives the action-value $Q^\pi(h_t,a_t)$, a different object; the advantage is precisely their difference, $A=Q-V$. Conditioning on history alone also pays off in practice: from one state one often scores many candidate actions, so $V$ is computed once while $Q$ would be computed per action. The lecture adds that judging is easier than generating — assessing "how am I doing" is easier to learn than "what should I do next" — which is why a discriminative head works well for $V$.

2. What do $\lambda = 1$ and $\lambda < 1$ mean in GAE?

Reference answer: $A_t=\delta_t+\lambda\delta_{t+1}+\lambda^2\delta_{t+2}+\cdots$. With $\lambda=1$ the terms telescope and only the final reward remains, equivalent to estimating the advantage from the realized return of the whole trajectory (Monte Carlo style): unbiased but high variance, with no bootstrapping. With $\lambda<1$ the value function bootstraps and future uncertainty is discounted in, lowering variance at the cost of bias when $V$ is inaccurate. The lecture notes PPO commonly uses $\lambda=1$; a discount is worth introducing only when you trust the value function or have intermediate rewards such as rubric scores.

3. Why does a direct product of importance weights explode over a 50k-token trajectory, and how do PPO and GSPO each handle it?

Reference answer: A single-step weight $w=\pi_\theta(a\mid h)/\mu(a\mid h)$ is harmless, but multiplying ratios over 50k tokens makes the log-weight variance grow roughly like $\sqrt{n}$ (as measured in CTPO), so the magnitude runs away; the lecture cites a real asynchronous batch whose maximum importance weight was about 1400, enough for one gradient update to destroy training. PPO/GRPO use only the current token's ratio — strictly incorrect because history ratios are ignored, but "good enough" to reflect how surprising the current action is. GSPO multiplies all history ratios and normalizes by sequence length, keeping the full information while bringing the magnitude back to an acceptable range. Either way, clipping is mandatory.

4. Why does an all-pass GRPO group produce no gradient, and which metric would you monitor?

Reference answer: GRPO's advantage is $A_i=R_i-\frac1G\sum_j R_j$. If a group is all-pass (or all-fail), every reward is identical, the group mean equals each $R_i$, and every $A_i=0$; the policy gradient is zero and the group yields no update. Average reward hides this — it can look high while within-group variation is nil. Monitor the fraction of samples with nonzero advantage, which directly shows how many groups actually supply signal; when a group all-fails, further separate validator errors, insufficient model capability, and insufficient exploration.

5. Give two examples of auxiliary rewards being over-optimized, and say how you would detect it early.

Reference answer: The lecture gives two. First, GPT-5.1 was given a browser-use reward to teach browsing, and ended up opening a calculator web page even for simple arithmetic. Second, a roleplay training added a "nerd" bonus, the judge model considered "goblin" very nerdy, and the model began saying goblin constantly. Both illustrate: whatever you reward is what you get, including the parts you did not intend. To catch it early: track the auxiliary reward separately from true task success and watch for divergence (proxy rising while task metric stalls), sample and read trajectories periodically, red-team the reward for exploitable paths, and treat any sudden surge of a specific behaviour as a hypothesis that it is farming the proxy rather than solving the task.

Connections

The previous lecture covered RL basics and GRPO. This one addresses algorithmic stability and signal quality. The next moves to systems: inference engines, trainer engines, weight synchronization, parallelism, and the design of frameworks built for agent training.