第 09 章

强化学习基础：从策略梯度到 GRPO

SFT 有三个结构性缺陷，强化学习用采样轨迹与奖励直接优化期望回报来修复它们；本讲推导 REINFORCE，解释基线为何能降方差，并给出 GRPO 与 DR-GRPO 的简化形式。

RL Basics: From Policy Gradients to GRPO

SFT has three structural failures; RL fixes them by sampling trajectories and optimizing expected return. This lecture derives REINFORCE, shows why baselines cut variance, and gives simplified GRPO and DR-GRPO.

首发于 2026.09.25中 / EN

系统课程ai-agentsreinforcement-learningpolicy-gradientgrpo

篇内目录

一个用来贯穿全讲的玩具任务

Number Search：环境里藏着一个 1 到 16 的整数，agent 最多猜四次。每次猜完，环境回答「高了」「低了」或「正确」。猜中得 1 分，四次都没中得 0 分。

最优首猜是 8，因为一次就能把候选集砍半——和 Wordle 的首猜思路相同。首猜 4 也能靠反馈最终猜中；首猜 16 或从 1 开始逐个递增，则很可能在四次内失败。

这个任务足够简单，可以手算轨迹概率；又包含多步决策、稀疏奖励和部分可观测，几乎所有 RL 概念都能在这里演示。

时间轴

时间	内容
0:00	开场：这一讲在训练流程中的位置
4:22	玩具任务 Number Search：首猜与反馈后的分支
8:29	一次反馈之后还有哪些合理动作
12:55	RL 与 SFT 的分水岭：不学固定轨迹，学策略
16:57	稀疏奖励：多数任务只在执行测试后给终局奖励
21:30	轨迹概率的链式分解
25:51	目标函数：按策略概率加权求和轨迹回报
29:54	最简做法：用奖励过滤轨迹再做 SFT（REST / 拒绝采样）
34:16	期望回报与策略参数的关系
38:19	用采样轨迹估计期望回报
42:24	策略梯度：梯度落到 log 概率上（REINFORCE）
46:50	基线：不改变期望，只改变方差
51:01	不同基线对应不同算法
55:04	基线降低梯度噪声的具体例子
59:26	减去基线后的梯度估计
1:03:49	找到好的优势表示是 RL 的核心工作
1:08:02	同一任务多次 rollout，组内比较（GRPO）
1:12:03	除以标准差后的优势（DR-GRPO）
1:16:08	REST：二值奖励下只保留正确轨迹

SFT 的三个结构性缺陷

监督微调最大化的目标是：在给定真实历史时，让模型给被演示动作的概率尽量高。

J_{\text{SFT}}(\theta) = \sum_{t} \log \pi_\theta(a_t^* \mid h_t)

但这和「让任务成功」不是同一个目标。

1. 任务错位

完成任务往往有多条路径：首猜 8 最好，首猜 4 也可行。演示只记录了其中一条。
更严重的是：没被演示的动作一律被当成同样差。反馈是「高了」，模型猜 7 和猜 15 都错，但前者接近正确、后者错得离谱；SFT 的损失看不出这个区别。

2. 数据错位

失败的轨迹也包含信息——它告诉模型「不要这样做」。但 SFT 只会提高这些动作的概率，所以不能直接拿来训练。

3. 暴露偏差

训练时历史来自真实演示；推理时历史来自模型自己。一旦模型走出训练分布，它就没见过这种状态，行为可能完全失准。更糟的是，它没有机会学会从错误中恢复。

RL 同时处理这三点：直接优化任务回报、能从失败轨迹中学习、训练数据由当前策略自己生成（on-policy）。

把交互写成轨迹

策略 $\pi_\theta$ ：由语言模型参数化的动作分布；
任务实例：例如隐藏的数字，agent 看不到；
观测：环境返回的信息（「高了」）；
动作：下一次猜测；
奖励：成功为 1，超时为 0。

轨迹 $\tau$ 是动作与观测交错组成的序列， $h_t$ 是到 $t$ 时刻为止的历史。

轨迹概率可以分解为环境项与策略项的乘积：

p_\theta(\tau) = \prod_t p_{\text{env}}(o_t, r_t \mid h_t, a_t)\, \pi_\theta(a_t \mid h_t)

关键点是：我们不需要知道环境项。与环境交互就是在从它采样。这也解释了 RL 为什么昂贵——它只能靠不断试错获得关于梯度的间接信息。

课堂补充：这里用的是部分可观测马尔可夫决策过程（POMDP）的写法，因为 agent 看不到真实状态，只能用历史作为条件。

目标：最大化期望回报

J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[R(\tau)\right]

课堂用一个只有四条可能轨迹的简化例子说明：概率分别为 0.3、0.2、0.25、0.25，其中两条成功。期望回报是 0.5，训练的目标就是把概率质量从失败轨迹移到成功轨迹上。

最简单的做法：专家迭代与 REST

在进入策略梯度之前，有一个几乎不需要新数学的基线：Reinforced Self-Training（REST）。

用当前策略采样一批轨迹；
计算奖励；
只保留成功的轨迹（二值奖励时就是阈值过滤）；
在这些轨迹上做普通 SFT；
重复，把新采样的成功轨迹不断加入训练集。

它复用了 SFT 的全部基础设施，本质是「自训练 + 奖励过滤」。它不是严格意义上的 RL，但已经具备 RL 的核心结构：数据由模型自己生成，更新方向由奖励决定。

一个现实问题：如果模型一条成功轨迹都采不到，REST 和 RL 都无从下手。解决办法包括——先用少量演示做冷启动、选择当前能解出的课程任务、加入中间奖励，或者用奖励塑形引导模型先完成有价值的子步骤。

策略梯度：REINFORCE

直接对 $J(\theta)$ 求导有两个障碍：

动作是从分布中采样出来的，采样操作不可导；
环境是黑盒，几乎不可导。

经典解法是 1992 年的 REINFORCE。对期望求导并做「得分函数」变换：

\nabla_\theta J(\theta) = \sum_\tau \nabla_\theta p_\theta(\tau)\, R(\tau) = \mathbb{E}_{\tau \sim \pi_\theta}\left[R(\tau)\, \nabla_\theta \log p_\theta(\tau)\right]

而轨迹的对数概率梯度可以拆成每一步动作的梯度之和，环境项与参数无关，直接消失：

\nabla_\theta \log p_\theta(\tau) = \sum_t \nabla_\theta \log \pi_\theta(a_t \mid h_t)

落到实现上

采样一条轨迹；
只对模型生成的 token 计算对数概率（工具结果、提示词等不计）；
把整条轨迹的所有 token 对数概率乘以同一个标量奖励；
反向传播。

课堂的观察很直接：这看起来几乎就是 SFT 的梯度计算，只是在前面乘了一个奖励。一个动作可能对应多个 token，对数概率相加即可。

为什么基础 REINFORCE 不够

看两条轨迹：一条成功，一条失败。

成功轨迹的所有动作被奖励 1 上调；
失败轨迹的奖励是 0，乘以 0 后完全不产生更新。

但我们恰恰想从失败里学。更好的做法是让更新相对于预期表现：

一个模型本来有 95% 概率做对的简单题，如果偶尔犯一次低级错误，应该被重重惩罚；
一道本来就很难的题，失败并不说明动作有多坏。

这就是优势函数：

A(h_t, a_t) = Q^\pi(h_t, a_t) - V^\pi(h_t)

它衡量「这个动作比策略的平均表现好多少」。

一个具体的优势例子

猜过 8（反馈「高了」）、12（反馈「低了」）之后，答案只能是 9、10、11。

猜 11：如果错了，只剩一次机会、两个候选，优势低；
猜 10：无论对错都能继续缩小范围，优势高。

基线：不改变期望，只降低方差

把奖励减去一个只依赖历史的基线 $b(h_t)$ ：

\nabla_\theta J = \mathbb{E}\left[(R(\tau) - b(h_t))\, \nabla_\theta \log \pi_\theta(a_t \mid h_t)\right]

因为 $\mathbb{E}[\nabla_\theta \log \pi_\theta(a \mid h)] = 0$ ，减去基线不会引入偏差，只改变估计的方差。

两动作的例子

设策略只有一个参数 $\theta$ ，用 sigmoid 得到选择动作 A 的概率。当前 $\theta$ 使 $p(A)=0.25$ ，A 的奖励为 1；B 的奖励为 0。

真实梯度是 $p(1-p)=0.1875$ 。

不用基线：采到 A 时估计为 $1 \times 0.75 = 0.75$ ，采到 B 时为 $0 \times (-0.25)=0$ 。估计在 0.75 与 0 之间剧烈跳动。
用基线 0.5：采到 A 时为 $(1-0.5)\times 0.75=0.375$ ，采到 B 时为 $(0-0.5)\times(-0.25)=0.125$ 。期望仍是 0.1875，但两个样本都更靠近真值。

基线可以是常数、奖励的滑动平均，也可以是一个单独训练的模型（actor-critic、PPO）。GRPO 的创新是不训练额外模型，直接用同一道题的多条 rollout 估计基线。

如果奖励分布在中间步骤而不是只在结尾，把奖励换成「奖励到未来」（reward-to-go），也能进一步降方差。

GRPO：用组内均值当基线

Group Relative Policy Optimization 的做法：

对同一个任务实例采样一组轨迹（通常约 8 条）；
计算每条轨迹的奖励；
用组内均值作为基线，得到优势：

A_i = R_i - \frac{1}{G}\sum_{j=1}^{G} R_j

用这个优势缩放每条轨迹的 token 对数概率，再做策略梯度更新。

因为基线只依赖提示词与模型本身，这个估计仍然无偏。组内比较只在同一道题内部进行，不同题的难度不会被混在一起。

DR-GRPO 的两处修正

不除以组内标准差：除以标准差会把不同难度的题「标准化」，优势因此失去对难度的感知。只减均值即可。
不除以生成长度：按生成 token 数归一化会偏袒长而错误的答案。改为除以一个常数（最大可能生成长度）。

原始 GRPO 还包含重要性比率的裁剪与对初始策略的 KL 正则，用来稳定训练；这些留到下一讲。

全对或全错，就没有信号

如果一组轨迹全失败，说明题目对当前策略太难；如果全成功，说明太简单。两种情况组内方差为零，优势全为零，学不到东西。应对方式包括：换任务、给提示、加演示、先用 SFT 冷启动、做奖励塑形。

三种方法其实是同一件事的不同加权

方法	如何利用采样到的轨迹
REST	只保留成功轨迹，做 SFT
REINFORCE	用轨迹奖励缩放对数概率
GRPO	用组内相对优势缩放对数概率

它们都让模型在自己的轨迹上学习，区别只在于如何给每个动作重新加权。三者都针对 SFT 的三个缺陷：直接优化回报、能从失败中学习、训练分布与推理分布一致。

代价是必须真的在环境里试错：慢、贵，梯度噪声大。后续两讲会讨论稳定化与系统工程。

本讲结论

SFT 优化的是「像演示」，RL 优化的是「拿到回报」。
策略梯度用对数概率的梯度乘奖励，绕开不可导的采样与环境。
基础 REINFORCE 对零奖励轨迹没有信号；优势函数让更新相对于预期表现。
减去只依赖历史的基线不引入偏差，只降低方差。
GRPO 用同题多条 rollout 的均值当基线，省掉了单独的 value 模型。
组内没有成败差异时，GRPO 得不到任何学习信号。

理解检验

1. 用 number search 说明 SFT 的「任务错位」具体错在哪里。

参考答案：任务错位有两层。第一层是多路径：猜中 1 到 16 的隐藏数有很多种走法——首猜 8 最优，首猜 4 也能靠反馈猜中，而演示只记录了其中一条，SFT 会把其他可行路径的概率压低。第二层更关键：演示之外的动作一律被当成同样差。假设首猜 8 后反馈「高了」，此时猜 7 和猜 15 都是错的，但猜 7 离答案更近、保留了更多信息，猜 15 则几乎浪费一次机会；SFT 的损失只看到「不是演示动作」，无法区分两者的好坏，因此学不到「错得少一点」这件事。

2. 为什么减去基线不会让策略梯度产生偏差？

参考答案：因为对任意历史 $h$ ， $\mathbb{E}_{a\sim\pi_\theta}[\nabla_\theta \log \pi_\theta(a\mid h)]=0$ 。验证：

\sum_a \pi_\theta(a\mid h)\frac{\nabla_\theta \pi_\theta(a\mid h)}{\pi_\theta(a\mid h)} =\sum_a \nabla_\theta \pi_\theta(a\mid h) =\nabla_\theta \sum_a \pi_\theta(a\mid h) =\nabla_\theta 1=0 .

所以减去只依赖历史的 $b(h_t)$ 后，多出来的项 $b(h_t)\nabla_\theta\log\pi_\theta$ 期望为零，梯度的期望值不变，改变的只是估计的方差。

3. 在奖励全为二值的情况下，REINFORCE 与 REST 的更新有什么本质区别？

参考答案：奖励只有 0 和 1 时，REINFORCE 用 $R(\tau)\nabla_\theta\log p_\theta(\tau)$ 缩放；失败轨迹的奖励为 0，乘上去后梯度为零，等于完全不更新——失败轨迹只是被「访问」了，没有产生学习信号。REST 则先按奖励过滤，只保留成功轨迹再做普通 SFT，失败轨迹被直接丢掉。两者的更新方向在二值时相近（都在提高成功轨迹的概率），但机制不同：REST 复用 SFT 基础设施、可以对过滤后的数据集多轮训练，REINFORCE 是 on-policy 的单次采样加权；而且 REST 完全不利用失败轨迹的信息，REINFORCE 至少能通过基线让失败轨迹产生负向更新（本讲的基线讨论正是为此）。

4. GRPO 为什么必须在同一道题内部计算均值，而不能跨题？

参考答案：有两层原因。其一，基线的无偏性要求它不能依赖被采样的动作；同一提示词下的组内均值满足这一点，同时它是对该题价值 $V(h)$ 的现成估计，因此优势 $A_i=R_i-\bar R$ 仍然无偏。其二，跨题求均值会把难度混在一起：一道容易题的成功会和难题的失败放在同一个尺度上比较，优势就不再表示「在这道题上比平均水平好多少」，模型也无法区分「这题本来就简单」和「这一步做得特别好」。组内比较保证不同题的难度互不干扰。

5. DR-GRPO 去掉标准差归一化的理由是什么？

参考答案：除以组内标准差会把不同难度的题「标准化」——难题和易题的优势被拉到相近的量级，优势因此失去对难度的感知。课程给的替代是只减均值、不除标准差。补充：组内方差极小时（例如接近全对或全错），除以一个很小的标准差还会把噪声放大成很大的优势，这也是去掉它的实际理由之一。

与前后课程的关系

上一讲的 SFT 提供了冷启动 checkpoint，这一讲给出最基础的策略梯度。下一讲会处理 off-policy、裁剪、KL 正则与训练稳定性；再下一讲讨论 rollout、推理与训练系统如何支撑这些算法。

A toy task that runs through the lecture

Number Search: the environment holds an integer from 1 to 16 and the agent gets four guesses. After each guess the environment says "higher", "lower", or "correct". Correct earns reward 1; running out of guesses earns 0.

The optimal first guess is 8, because it halves the candidate set — the same logic as an opening guess in Wordle. Starting at 4 can still succeed by following feedback; starting at 16, or counting up from 1, is likely to time out.

The task is simple enough to compute trajectory probabilities by hand, yet it contains multi-step decisions, sparse reward, and partial observability. Almost every RL concept can be demonstrated in it.

Timeline

Time	Topic
0:00	Opening: where this lecture sits in the training pipeline
4:22	The Number Search toy task: first guess and branching after feedback
8:29	Which actions still make sense after one round of feedback
12:55	RL versus SFT: learning a policy instead of a fixed trajectory
16:57	Sparse reward: most tasks score only after tests run
21:30	Decomposing the trajectory probability with the chain rule
25:51	Objective: trajectory returns weighted by policy probability
29:54	Simplest recipe: filter trajectories by reward, then run SFT (REST / rejection sampling)
34:16	How expected return depends on the policy parameters
38:19	Estimating expected return from sampled trajectories
42:24	Policy gradient: the gradient lands on the log-probability (REINFORCE)
46:50	Baselines: leave the expectation unchanged, cut the variance
51:01	Different baselines give different algorithms
55:04	A worked example of variance reduction
59:26	The gradient estimate after subtracting a baseline
1:03:49	Finding a good advantage representation is the core of RL
1:08:02	Multiple rollouts per task, compared within the group (GRPO)
1:12:03	Advantages normalized by the standard deviation (DR-GRPO)
1:16:08	REST: keep only correct trajectories under binary rewards

Three structural failures of SFT

Supervised fine-tuning maximizes the probability of the demonstrated action given the true history:

J_{\text{SFT}}(\theta) = \sum_{t} \log \pi_\theta(a_t^* \mid h_t)

That is not the same objective as making the task succeed.

1. Task mismatch

Tasks usually have several valid paths: 8 is the best first guess, but 4 is workable. A demonstration records only one.
Worse, every action that was not demonstrated is treated as equally bad. After "higher", guessing 7 and guessing 15 are both wrong, but one is close and the other wildly off. The SFT loss cannot see the difference.

2. Data mismatch

Failed trajectories carry information — they say "do not do this". But SFT would raise the probability of exactly those actions, so failed data cannot be used directly.

3. Exposure bias

At training time the history comes from demonstrations; at inference it comes from the model. Once the model leaves the training distribution it has never seen such a state, and its behavior may be badly calibrated. It also never learns to recover from its own mistakes.

RL addresses all three: it optimizes the task return directly, can learn from failed trajectories, and trains on data generated by the current policy (on-policy).

Writing interaction as a trajectory

Policy $\pi_\theta$ : the action distribution parameterized by the language model.
Task instance: the hidden number, invisible to the agent.
Observation: what the environment returns ("higher").
Action: the next guess.
Reward: 1 on success, 0 on timeout.

A trajectory $\tau$ interleaves actions and observations, and $h_t$ is the history up to time $t$ .

The trajectory probability factors into an environment term and a policy term:

p_\theta(\tau) = \prod_t p_{\text{env}}(o_t, r_t \mid h_t, a_t)\, \pi_\theta(a_t \mid h_t)

The key point: we never need to know the environment term. Interacting with the environment samples from it. That is also why RL is expensive — it obtains information about the gradient only indirectly, by trying things.

Background: this uses the partially observable MDP formulation, because the agent cannot see the true state and must condition on history.

Objective: maximize expected return

J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[R(\tau)\right]

The lecture's simplified illustration has four possible trajectories with probabilities 0.3, 0.2, 0.25, and 0.25, two of which succeed. Expected return is 0.5, and training should move probability mass from the failures to the successes.

The simplest method: expert iteration and REST

Before policy gradients, there is a baseline that needs almost no new math: Reinforced Self-Training (REST).

Sample a batch of trajectories from the current policy.
Compute rewards.
Keep only successful trajectories (a threshold filter, trivial with binary reward).
Run ordinary SFT on them.
Repeat, adding newly sampled successes to a growing training set.

It reuses the entire SFT stack and is really self-training plus reward filtering. It is not RL in the strict sense, but it already has the core structure: data comes from the model, and the update direction comes from reward.

A practical problem: if the model cannot sample a single success, neither REST nor RL has anything to work with. Remedies include a small demonstration cold start, curriculum tasks the model can already solve, intermediate rewards, or reward shaping that pays for valuable sub-steps.

Policy gradients: REINFORCE

Differentiating $J(\theta)$ directly runs into two obstacles:

Actions are sampled, and sampling is not differentiable.
The environment is a black box, almost always non-differentiable.

The classic solution is REINFORCE (Williams, 1992). Differentiate the expectation and apply the score-function identity:

\nabla_\theta J(\theta) = \sum_\tau \nabla_\theta p_\theta(\tau)\, R(\tau) = \mathbb{E}_{\tau \sim \pi_\theta}\left[R(\tau)\, \nabla_\theta \log p_\theta(\tau)\right]

The log-probability of a trajectory decomposes into per-step action terms, and the environment term vanishes because it does not depend on $\theta$ :

\nabla_\theta \log p_\theta(\tau) = \sum_t \nabla_\theta \log \pi_\theta(a_t \mid h_t)

What the implementation looks like

Sample one trajectory.
Compute log-probabilities only for tokens the model generated (tool output and prompt tokens are excluded).
Multiply every token log-probability from that trajectory by the same scalar reward.
Backpropagate.

The lecture's observation is striking: this looks almost exactly like the SFT gradient, with a reward multiplied in front. One action may span several tokens; their log-probabilities are summed.

Why plain REINFORCE is not enough

Consider a successful trajectory and a failed one.

Every action in the successful trajectory is upweighted by reward 1.
The failed trajectory has reward 0, and multiplying by zero produces no update at all.

But failures are exactly what we want to learn from. The update should be relative to how well the policy was expected to do:

On an easy task the model solves 95% of the time, a rare careless failure should be penalized heavily.
On a genuinely hard task, failure says little about how bad the action was.

This is the advantage:

A(h_t, a_t) = Q^\pi(h_t, a_t) - V^\pi(h_t)

It measures how much better an action is than the policy's average behavior.

A concrete advantage example

After guessing 8 ("higher") and 12 ("lower"), the answer is 9, 10, or 11.

Guess 11: if wrong, only one guess and two candidates remain. Advantage is low.
Guess 10: correct or not, the range still narrows. Advantage is high.

Baselines: same expectation, less variance

Subtract a baseline $b(h_t)$ that depends only on history:

\nabla_\theta J = \mathbb{E}\left[(R(\tau) - b(h_t))\, \nabla_\theta \log \pi_\theta(a_t \mid h_t)\right]

Because $\mathbb{E}[\nabla_\theta \log \pi_\theta(a \mid h)] = 0$ , subtracting the baseline adds no bias; it only changes variance.

The two-action example

A policy with one parameter $\theta$ uses a sigmoid to produce the probability of action A. At the current $\theta$ , $p(A)=0.25$ , A has reward 1, and B has reward 0.

The true gradient is $p(1-p)=0.1875$ .

Without a baseline: sampling A gives $1 \times 0.75 = 0.75$ ; sampling B gives $0 \times (-0.25) = 0$ . The estimate jumps between 0.75 and 0.
With baseline 0.5: sampling A gives $(1-0.5)\times 0.75 = 0.375$ ; sampling B gives $(0-0.5)\times(-0.25) = 0.125$ . The expectation is still 0.1875, but both samples sit closer to it.

A baseline can be a constant, a running average of rewards, or a separately trained model (actor-critic, PPO). GRPO's innovation is to avoid the extra model entirely and estimate the baseline from multiple rollouts of the same task.

If rewards arrive at intermediate steps rather than only at the end, replacing the reward with reward-to-go further reduces variance.

GRPO: the group mean as baseline

Group Relative Policy Optimization:

Sample a group of trajectories for one task instance (typically around eight).
Compute each trajectory's reward.
Use the group mean as the baseline:

A_i = R_i - \frac{1}{G}\sum_{j=1}^{G} R_j

Scale each trajectory's token log-probabilities by its advantage and take the policy gradient step.

Because the baseline depends only on the prompt and the model, the estimate stays unbiased. Group comparison happens within one task, so difficulty is never mixed across tasks.

Two corrections from DR-GRPO

Do not divide by the group standard deviation. Dividing standardizes across difficulties and strips the advantage of any sense of task difficulty. Subtract the mean only.
Do not divide by generation length. Normalizing by token count favors long wrong answers. Divide by a constant — the maximum possible generation length.

The original GRPO also includes importance-ratio clipping and a KL penalty against the initial policy for stability; those come in the next lecture.

All-correct or all-wrong groups carry no signal

If every trajectory fails, the task is too hard for the current policy. If every trajectory succeeds, it is too easy. Either way group variance is zero, all advantages are zero, and nothing is learned. Remedies: different tasks, hints, demonstrations, an SFT cold start, or reward shaping.

Three methods, one family

Method	How sampled trajectories are weighted
REST	Keep successes only, then SFT
REINFORCE	Scale log-probabilities by trajectory reward
GRPO	Scale log-probabilities by group-relative advantage

All three learn from the model's own trajectories; they differ only in how each action is reweighted. All three address the three SFT failures: optimize return directly, learn from failures, and keep training and inference distributions aligned.

The cost is real interaction with the environment: slow, expensive, and noisy in the gradient. The next two lectures cover stabilization and systems.

Takeaways

SFT optimizes "look like the demonstration"; RL optimizes "earn the return".
Policy gradients multiply the log-probability gradient by reward, sidestepping sampling and the environment.
Plain REINFORCE gives no signal on zero-reward trajectories; advantages make updates relative to expected performance.
A history-only baseline adds no bias and reduces variance.
GRPO uses the mean of multiple rollouts of the same task as its baseline, avoiding a separate value model.
A group with no variation in success or failure yields no learning signal.

Check your understanding

1. Using Number Search, where exactly does SFT's task mismatch bite?

Reference answer: The mismatch has two layers. First, multiple paths: a hidden number from 1 to 16 can be found many ways — guessing 8 first is optimal, guessing 4 can still succeed via feedback, but the demonstration records only one of them, and SFT suppresses the others. Second, and more sharply, any action outside the demonstration is treated as equally bad. Suppose 8 gets "too high": guessing 7 and guessing 15 are both wrong, yet 7 is close and preserves information while 15 nearly wastes a turn. The SFT loss only sees "not the demonstrated action" and cannot tell the two apart, so the model never learns to fail slightly better.

2. Why does subtracting a baseline not bias the policy gradient?

Reference answer: Because for any history $h$ , $\mathbb{E}_{a\sim\pi_\theta}[\nabla_\theta \log \pi_\theta(a\mid h)]=0$ . Check:

\sum_a \pi_\theta(a\mid h)\frac{\nabla_\theta \pi_\theta(a\mid h)}{\pi_\theta(a\mid h)} =\sum_a \nabla_\theta \pi_\theta(a\mid h) =\nabla_\theta \sum_a \pi_\theta(a\mid h) =\nabla_\theta 1=0 .

So after subtracting a history-only $b(h_t)$ , the extra term $b(h_t)\nabla_\theta\log\pi_\theta$ has zero expectation; the expected gradient is unchanged and only the variance of the estimate changes.

3. With binary rewards, what is the essential difference between REINFORCE and REST updates?

Reference answer: With rewards in $\{0,1\}$ , REINFORCE scales by $R(\tau)\nabla_\theta\log p_\theta(\tau)$ ; a failed trajectory has reward 0, so its gradient vanishes — failures are visited but produce no learning signal. REST filters by reward, keeps only successes, and runs ordinary SFT on them, discarding failures entirely. In the binary case the update directions are similar (both raise the probability of successful trajectories), but the machinery differs: REST reuses the SFT stack and can train for several epochs over the filtered dataset, while REINFORCE is a single on-policy sample weighting; REST cannot use failures at all, whereas REINFORCE can turn them into negative updates once a baseline is introduced (which is exactly why this lecture discusses baselines).

4. Why must GRPO compute its mean within one task rather than across tasks?

Reference answer: Two reasons. First, unbiasedness requires the baseline not to depend on the sampled action; the within-group mean for one prompt satisfies this and is simultaneously a ready estimate of the task value $V(h)$ , so $A_i=R_i-\bar R$ stays unbiased. Second, a cross-task mean mixes difficulties: a success on an easy task would be compared on the same scale as a failure on a hard one, so the advantage would no longer mean "better than average on this task", and the model could not separate "this problem was easy" from "this step was unusually good". Within-group comparison keeps difficulty from leaking across tasks.

5. What is DR-GRPO's reason for dropping standard-deviation normalization?

Reference answer: Dividing by the group standard deviation standardizes problems of different difficulty — hard and easy tasks end up with comparable advantage magnitudes, so the advantage loses its sensitivity to difficulty. The lecture's replacement is to subtract the mean only. Note as well: when within-group variance is tiny (near all-correct or all-wrong), dividing by a near-zero standard deviation also amplifies noise into huge advantages, which is another practical reason to drop it.

How this connects to the course

The previous lecture's SFT provides the cold-start checkpoint; this one gives the basic policy gradient. The next lecture handles off-policy data, clipping, KL regularization, and stability; the one after that covers the rollout, inference, and training systems that support these algorithms.

来源

视频CMU AI Agents 2026: 9. Reinforcement Learning BasicsDaniel Fried · 2026-09-25
文章CMU CS 11-768 course site