为什么在 RL 之前还要做 SFT

这一讲是训练部分的第一讲。它先回答一个看起来多余的问题：既然强化学习能直接从环境里学，为什么还要先做监督微调？

课堂上给了两个证据：

• DeepSeek R1-Zero 尝试直接在基座模型上做 RL，不经过 SFT。结果是回答中英混杂、可读性很差。
• 随后的 R1 重新加入 SFT，只用几千条精心构造的示例，先建立一个高质量的起点，再做 RL。

所以 SFT 的定位不是「最终解法」，而是冷启动：让模型先学会 agent 的基本形态——怎么发工具调用、怎么在长对话里继续、工具返回长什么样——然后再让 RL 去打磨。

在 Kimi K3 一类公开流程里，顺序是：SFT 初始化 agent 能力 → RL 发展领域专长 → 在策略蒸馏把多个专家合并回一个模型。

时间轴

三种「更新 agent」的方式

权重更新最特别的地方是：改一次，之后每次调用都带着新行为，提示词里不需要额外东西。代价是你看不见权重里发生了什么，也无法把这次改动搬到另一个模型上。

一条轨迹在训练时是什么样子

一条 agent 轨迹包含 system、user、assistant、observation 四种行。它的结构是：

[system] [user 问题] [assistant 思考+工具调用] [observation 工具结果]
[assistant ...] [observation ...] ... [assistant 最终回答]

训练时，chat template 会把整条轨迹渲染成一个字符串，并记录每个 assistant 片段的起止位置。随后 tokenize，得到一条长 token 序列，同时得到每个 token 一位的掩码：

• 助手生成的 token（包括工具调用、最终回答、停止 token）掩码为 1，计算交叉熵损失；
• 系统消息、用户消息、工具观察掩码为 0，只作为条件上下文。

这就是「教师强制」式的模仿学习：模型在训练时看到的是真实历史，而不是自己采样出来的历史。

掩码里容易出错的细节

1. 停止 token 必须在掩码内，否则模型学不会停下来。
2. end-of-turn 不是一个事件：它同时意味着结束消息、结束工具调用、结束任务。Kimi K3 的模板把它们拆成不同的特殊 token，让模型更容易学会「何时停」。
3. 推理块是否计入损失是模型相关的设计。推理往往很长，如果全部计入，会主导损失，模型可能学会长篇思考却做不对下一步动作。Balanced SFT 一类工作会重新平衡推理与工具调用的比例。
4. 模板漂移是致命的：训练用的模板必须和服务时完全一致，连空白字符都要一致。若服务时在历史里追加了条件式思考块，前面 assistant 轮次的前缀就变了，工具循环会直接失效。

轨迹到底教会了模型什么

• 如何写出脚手架能执行的工具调用；
• 如何在对话变长时继续推进；
• 工具返回值应该长什么样——模型应当等待结果，而不是自己编造结果；
• 数据里附带的其他习惯：多余的解说、重复命令、某一类解法。

最后一点很重要：损失会不加区分地复制每一个动作。一条最终成功的轨迹，中间也可能包含坏习惯。

数据从哪里来

两个维度决定数据策略：

大多数 agent SFT 落在右上角：用强模型跑任务，只保留通过仓库测试的轨迹。原因很直接——强模型写出的轨迹比学生自己采样得更好，而任务自带的验证器又能过滤掉失败的尝试。

如果没有更强的教师模型，可以自蒸馏，或者把重心放到 RL 上——SFT 的上限本质上受教师能力限制。

规模会饱和，分布不会

SWE-Gym 的实验显示：固定任务集、只增加采样数量，准确率会上升，但最终会饱和；而且模型越小越早饱和。实验结束时限制他们的是采样算力，不是任务数量。

另一组对比更关键：同样是 10k 数据，继续采样同一分布的轨迹（粉色曲线）会逐渐走平；加入新任务/合成增强（蓝色曲线）则持续上升。

结论：把数据变多不如把数据变宽。课程给的实证建议是混合 4–8 个来源；再宽对特定基准未必有帮助。

成功轨迹也可能有毒

假设三条轨迹都通过了单元测试：

• 一条在多数轮次里反复「改一行—重跑测试」；
• 一条在最终补丁里留下了 print 调试语句；
• 一条失败后恢复并最终修好。

直接拿来训练，模型会把循环、残留调试代码和最终修复一起学走。常见的廉价过滤器是按轮数过滤：OpenHands Agent 的消融发现，丢掉少于 5 轮的轨迹平均最有效。但轮数只衡量「做了多少工作」，不衡量「工作好不好」——一条 4 轮的干净修复可能被丢掉，一条 14 轮的混乱轨迹却会留下。

最好的模型不一定是最好的老师

OpenHands Agent 的研究里，当时基准最强的 GPT-5.3 Codex 反而是最差的教师：从它蒸馏后提升最小，甚至出现下降。一个可能解释是：它的思考模式与解题方式离学生太远，轨迹虽然正确，却不好模仿。

所以选教师要看「学生能不能学」，而不只看排行榜。

任务来源比超参数更重要

同一项研究显示，任务来源对结果的影响最大：SWE-Smith 对 SWE-bench Verified 贡献最大，对 Terminal Bench 贡献较小。先选对任务来源，再调其他东西。

一个具体的数据取舍问题

课堂给了三条同一任务的轨迹，问该保留哪些：

• A：4 轮干净解决；
• B：长轨迹，先做错一次编辑，测试失败，恢复后解决；
• C：尝试 11 次，但没有产生任何改动。

讨论的结论倾向于 A 与 B 都保留。A 教会正确路径，B 教会错误恢复；只保留 A，模型学不会从失败中回来。但 B 也有风险：损失会连错误编辑一起学。C 没有可学的动作，应当丢弃。

标准化数据格式

课堂上介绍了作者参与的 Agent Data Protocol。动机很实际：不同数据集用不同格式——网页数据有的用 HTML，有的用无障碍树；不同 agent 的动作表示也不同。若要把 N 个数据集喂给 M 个 agent，需要写 N×M 个转换器。

标准化之后只需要 N+M 次转换：每个数据集转成统一格式一次，统一格式再转成目标 agent 格式一次。统一格式同时保留 HTML 与无障碍树，下游按需选择。

实际训练配置与坑

公开技术报告里的真实配置差异很大：

• Nemotron 3 Ultra：把约 294k token 打包成序列，batch 64，学习率衰减；
• MAI 自蒸馏：128k 序列，大 batch，先 warm-up 再衰减；
• K2 Horizon：512k 序列，分三个阶段，在高量子集上衰减学习率。

这些都不是参数高效微调，而是全量训练。

打包

对话不截断、不拆分，而是分配给「剩余容量最贴合」的 pack；相同问题不放在同一个 pack 里；pack 之间再打乱。不拆分是为了避免模型学到被截断的轨迹而产生幻觉。

混合专家模型的专家路由

在狭窄的 agent 数据上微调 MoE 时，少数专家会吃掉大部分 token。MAI 报告说，监督阶段的负载均衡比 RL 阶段难约一千倍，drop 率设为 15。

训练损失不能替代评估

这是本讲最重要的工程提醒。

训练时，模型在记录好的轨迹上每次只预测一个动作，历史永远来自真实数据。部署时没有记录，模型自己的每个动作都会改变下一步看到的东西。一旦走到训练数据没覆盖的状态，低损失不再说明任何问题。

所以：

• 同时监控训练损失与评估结果；
• 先确认模型能过拟合 20 条样本，否则 bug 可能在上游；
• 在真实脚手架与可检查结果的任务上评估；
• 谨慎阅读公开数字，有些是「多个脚手架里最好的一个」；
• 划分数据时要与结论匹配：若声称能泛化到新仓库，就按仓库划分，而不是把同一任务的不同 run 分到两边。

跨脚手架鲁棒性

轨迹会连系统约定一起教给模型。只在 OpenHands 里收集的数据，未必能迁移到 Claude Code 或 Codex。前沿实验室会跨多个脚手架收集数据，避免模型只会用一种工具。

SFT 与 RL 的分工

三条结论：

1. 完全不做 SFT，基座模型几乎不动（R1-Zero 的教训）；
2. 只做 SFT，达不到 SFT+RL 的水平；
3. 最佳配方往往从一个没有训练到自己最高分的 checkpoint 开始 RL，为后续 RL 留出空间。

因此可以把 SFT 理解为：先建立一个足够好的先验，让 RL 有一个值得打磨的起点。

本讲结论

• SFT 是 agent 训练的冷启动，不是终点。
• 轨迹被渲染成一条 token 序列，损失只落在助手 token 上。
• 数据来源、过滤方式、任务分布比多数超参数更重要。
• 成功的轨迹也可能包含坏习惯；按轮数过滤是廉价但粗糙的手段。
• 训练损失不能替代评估；部署会走到训练从未见过的状态。
• 先选对任务来源与教师，再谈规模。

理解检验

1. 为什么只对 assistant token 计算损失？训练 observation 会带来什么好处与风险？

参考答案：损失只落在 assistant token 上，是因为部署时模型只负责生成这些 token——思考、工具调用、最终回答与停止 token；system、user、observation 都不是模型产生的，它们只是条件上下文。若把 observation 也计入损失，模型会被训练去「预测工具输出」，好处是可能学到工具返回值的常见形态、形成一点世界模型；风险更直接：模型可能学会自己编造工具结果，而不是按笔记强调的「等待结果」，而且 observation 往往很长，会主导损失、挤占真正该学的动作信号。

2. 一条通过了单元测试的轨迹，为什么仍可能不适合作为 SFT 数据？

参考答案：因为损失会不加区分地复制轨迹里的每一个动作，而「最终通过测试」并不等于「过程干净」。笔记列出的典型问题包括：反复「改一行—重跑测试」的循环、最终补丁里残留的 print 调试语句、冗长的解说与重复命令。这些坏习惯会和正确解法一起被学走。另外还有两层风险：教师与学生的分布差距过大时轨迹「正确但不好模仿」（GPT-5.3 Codex 作为最差教师的例子），以及轨迹携带的系统约定可能与目标脚手架不一致。

3. 「扩大采样」和「扩大任务分布」在曲线上表现有何不同？

参考答案：固定任务集、只增加采样数量时，准确率会上升但最终饱和，而且模型越小饱和得越早——SWE-Gym 实验结束时限制他们的是采样算力而不是任务数量。另一组对比中，同样是 10k 数据，继续采样同一分布的轨迹，曲线逐渐走平；加入新任务或合成增强后，曲线持续上升。结论是把数据变多不如把数据变宽，课程给出的实证建议是混合 4–8 个来源。

4. 模板漂移为什么会让工具循环失效？

参考答案：训练时 chat template 把整条轨迹渲染成一个字符串，模型学到的是这个精确前缀下的下一个 token 分布；笔记强调训练模板必须与服务时完全一致，连空白字符都要一致。如果服务时在历史里追加了条件式思考块，前面 assistant 轮次的前缀就变了，模型面对的输入分布与训练时不同，原本学会的「发出工具调用—等待 observation—继续」的模式被打断，工具循环随之失效。

5. 为什么说 SFT 的上限受教师模型限制？没有更强教师时有哪些替代路径？

参考答案：SFT 是教师强制式的模仿学习，模型只能学到演示里出现过的行为与质量；数据来源矩阵里，强模型生成加验证器过滤是大多数 agent SFT 的做法，因此教师的能力与覆盖范围决定了上限。而且教师越强不一定越好教——分布离学生太远时蒸馏收益反而最小。没有更强教师时的替代路径：自蒸馏（模型自己生成轨迹）、专家迭代（用验证器筛选自己的成功轨迹，后续课程展开）、把重心转向 RL，以及扩大任务来源与分布而不是单纯堆采样量。

与前后课程的关系

上一讲用 SFT 训练计算机使用 agent，这一讲展开其机制。接下来三讲进入强化学习：第 9 讲基础，第 11 讲高级算法，第 12 讲训练系统；SFT 产出的 checkpoint 正是它们的起点。

Why SFT comes before RL

This is the first training lecture. It begins with a question that sounds redundant: if reinforcement learning can learn from an environment, why do supervised fine-tuning first?

Two pieces of evidence:

• DeepSeek R1-Zero ran RL directly on a base model with no SFT. Its answers mixed languages and were hard to read.
• The follow-up R1 put SFT back, using a few thousand carefully built examples to establish a high-quality starting point before RL.

So SFT is not the destination. It is a cold start: teach the model the basic shape of being an agent — how to emit a tool call, how to keep going in a long conversation, what a tool result looks like — and let RL refine it afterward.

In a public pipeline such as Kimi K3, the order is SFT to initialize agentic capability, RL to develop domain expertise, then on-policy distillation to consolidate the experts into one model.

Timeline

Three ways to update an agent

Weights are special because you change them once and every later call inherits the behavior with nothing extra in the prompt. The price is that you cannot inspect what changed, and you cannot move the change to another model.

What a trajectory looks like to the trainer

A trajectory has system, user, assistant, and observation rows:

[system] [user request] [assistant thought + tool call] [observation result]
[assistant ...] [observation ...] ... [assistant final answer]

The chat template renders the whole conversation into one string and records where each assistant span starts and ends. After tokenization there is one long token sequence plus a one-bit mask per token:

• tokens the assistant generated — tool calls, final answers, stop tokens — get mask 1 and contribute to the cross-entropy loss;
• system messages, user messages, and tool observations get mask 0 and act only as conditioning context.

This is teacher-forced imitation learning: at training time the model sees the real history, not a history it sampled itself.

Masking details that trip people up

1. The stop token must be inside the mask, or the model never learns to stop.
2. End-of-turn is not one event. It means ending the message, ending the tool call, and finishing the task. Kimi K3's template separates them into distinct special tokens so the model can learn when to stop.
3. Whether reasoning tokens count toward loss is a model-specific choice. Reasoning is often very long; if it dominates the loss, the model may learn to think at length without acting correctly. Work such as Balanced SFT rebalances reasoning against tool calls.
4. Template drift is fatal. The training template must match the serving template exactly, down to whitespace. If serving appends a conditional thinking block to history, earlier assistant turns no longer have the same prefix and the tool loop breaks.

What a trajectory actually teaches

• How to write a tool call the harness can execute;
• How to keep going when the conversation gets long;
• What a tool result looks like — the model should wait for it, not invent it;
• Whatever else is in the data: extra commentary, repeated commands, one particular way of solving the task.

That last point matters: the loss copies every action indiscriminately. A trajectory that ends in success can still contain bad habits.

Where the data comes from

Two axes define the strategy:

Most agent SFT sits in the upper right: run tasks with a strong model and keep only trajectories that pass the repository's own tests. The logic is direct — a strong model writes better trajectories than the student would sample, and the task's verifier filters out failures.

Without a stronger teacher, self-distillation is an option, or the emphasis shifts to RL. SFT is fundamentally bounded by the teacher's ability.

Scale saturates; distribution does not

SWE-Gym holds the task set fixed and increases only the number of samples. Accuracy rises, then saturates, and smaller models saturate earlier. When the sampling budget ran out, the binding constraint was compute, not task count.

A second comparison is more important. From the same 10k dataset, adding more rollouts from the same distribution (the pink curve) flattens out; adding new tasks or synthetic augmentation (the blue curves) keeps climbing.

The lesson: more data is weaker than wider data. The empirical advice is to mix four to eight sources; going wider rarely helps a specific benchmark.

Successful trajectories can be toxic

Suppose three trajectories all pass the unit tests:

• one edits a line and reruns the tests in a loop for most of its turns;
• one leaves print statements in the final patch;
• one makes a wrong edit, sees the test fail, and recovers.

Training on them teaches the loop, the leftover debugging code, and the final fix together. A cheap filter is turn count: the OpenHands Agent ablation found that dropping runs shorter than five turns helped on average. But turn count measures how much work happened, not whether it was good — a clean four-turn fix can be dropped while a messy fourteen-turn run is kept.

The best model is not always the best teacher

In the OpenHands Agent study, GPT-5.3 Codex was the strongest agent on the benchmark and the worst teacher: distilling from it gave the smallest gain, sometimes a regression. A plausible explanation is that its thinking style and problem-solving approach are too far from the student's, so its trajectories are correct but hard to imitate.

Choose a teacher by whether the student can learn from it, not by leaderboard rank.

Task source matters more than hyperparameters

The same study found task source to have the largest effect: SWE-Smith contributed most to SWE-bench Verified and less to Terminal Bench. Pick the right task sources first, then tune everything else.

A concrete data-selection question

The lecture presents three runs of one task:

• A: solved cleanly in four turns;
• B: a long run that makes a wrong edit, fails the test, recovers, and solves it;
• C: eleven attempts that change nothing.

The discussion favors keeping both A and B. A teaches the correct path; B teaches recovery, which a model that only sees clean successes never learns. But B carries risk: the loss also learns the wrong edit. C contains nothing worth learning and should be dropped.

Standardizing the data format

The lecture presents the Agent Data Protocol, which the speaker worked on. The motivation is practical: datasets use incompatible formats — some web data uses HTML, some uses accessibility trees — and different agents represent actions differently. Feeding N datasets into M agents would require N×M converters.

After standardization, N+M conversions suffice: each dataset converts once into the shared format, and the shared format converts once into each target agent's format. The shared representation keeps both HTML and accessibility trees, so downstream consumers can choose.

Real training configurations and traps

Public technical reports show wide variation:

• Nemotron 3 Ultra packs about 294k tokens per sequence, batch 64, learning rate decay.
• MAI self-distillation uses 128k sequences, a large batch, warm-up before decay.
• K2 Horizon uses 512k sequences across three phases, decaying the learning rate on high-quality subsets.

These are full fine-tunes, not parameter-efficient ones.

Packing

Conversations are neither truncated nor split. Each is assigned to the pack whose remaining capacity fits most tightly; identical problems are kept out of the same pack; packs are shuffled. Avoiding splits prevents the model from learning from truncated trajectories and hallucinating.

Expert routing in mixture-of-experts models

Fine-tuning a MoE on narrow agent data causes a few experts to absorb most tokens. MAI reports that load balancing during the supervised stage is about a thousand times harder than during RL, with a drop rate of 15.

Training loss is not evaluation

This is the lecture's most important engineering warning.

At training time the model predicts one action at a time on a recorded trajectory, and the history always comes from real data. At deployment there is no recording; every action the model takes changes what it sees next. Once it reaches a state the training data never visited, low loss means nothing.

Therefore:

• monitor training loss and evaluation results together;
• first confirm the model can overfit 20 examples, or the bug is upstream;
• evaluate in a real harness on tasks with checkable outcomes;
• read published numbers carefully — some are the best of several harnesses;
• match the split to the claim: to claim generalization to new repositories, split by repository rather than putting different runs of the same task on both sides.

Robustness across harnesses

Trajectories teach system conventions along with the task. Data collected only in OpenHands may not transfer to Claude Code or Codex. Frontier labs collect across several harnesses so the model does not learn one tool and nothing else.

The division of labor between SFT and RL

Three conclusions:

1. With no SFT at all, a base model barely moves (the R1-Zero lesson).
2. SFT alone falls short of SFT plus RL.
3. The winning recipe often starts RL from a checkpoint deliberately not trained to its own best score, leaving room for RL to improve.

SFT is best understood as building a strong enough prior that RL has something worth sharpening.

Takeaways

• SFT is the cold start for agent training, not the finish line.
• The trajectory becomes one token sequence; loss falls only on assistant tokens.
• Data source, filtering, and task distribution matter more than most hyperparameters.
• Successful trajectories can carry bad habits; turn-count filtering is cheap but coarse.
• Training loss cannot replace evaluation; deployment reaches states training never saw.
• Choose task sources and teachers before scaling.

Check your understanding

1. Why compute loss only on assistant tokens? What does training on observations gain and risk?

Reference answer: The loss sits only on assistant tokens because those are the tokens the model must produce at deployment — reasoning, tool calls, the final answer, and the stop token. System, user and observation tokens are conditions, not actions; the model never generates them. Training on observations would arguably teach a weak world model of what tool outputs look like, but the risk is worse: the model may learn to fabricate tool results rather than wait for them as the lecture insists, and long observation spans would dominate the loss and crowd out the action signal.

2. A trajectory passed the unit tests — why might it still be bad SFT data?

Reference answer: Because the loss copies every action indiscriminately, and passing the tests does not make the process clean. The lecture's examples: an edit-one-line / rerun-test loop across many turns, leftover print statements in the final patch, verbose narration and repeated commands. Those habits are learned alongside the correct fix. Two further risks: a teacher whose distribution is too far from the student produces trajectories that are correct but hard to imitate (GPT-5.3 Codex was the worst teacher in the OpenHands Agent study), and trajectories carry system conventions that may not transfer to another scaffold.

3. How do the curves for more samples and wider task distribution differ?

Reference answer: With a fixed task set and only more samples per task, accuracy rises and then saturates, and smaller models saturate earlier — the SWE-Gym experiments ended limited by sampling compute, not by task count. In the paired comparison, the same 10k data points of continued sampling from the same distribution flatten out, while adding new tasks or synthetic augmentation keeps rising. So breadth beats volume; the course's empirical advice is to mix 4–8 sources.

4. Why does template drift break the tool loop?

Reference answer: The chat template renders the whole trajectory into one string, and the model learns the next-token distribution under that exact prefix; the training template must match serving exactly, down to whitespace. If serving appends a conditional thinking block to the history, the prefix of earlier assistant turns changes, the input distribution no longer matches training, and the learned pattern of "emit a tool call, wait for the observation, continue" breaks — the tool loop fails.

5. Why is SFT bounded by the teacher? What are the alternatives when no stronger teacher exists?

Reference answer: SFT is teacher-forced imitation learning, so the model can only learn behaviours and quality present in the demonstrations; the data matrix puts "strong model generates, verifier filters" at the heart of most agent SFT, which makes teacher capability and coverage the ceiling. A stronger teacher is not automatically a better one either — too large a distribution gap shrinks the gain. Alternatives without a stronger teacher: self-distillation (the model generates its own trajectories), expert iteration (filter its own successful rollouts with a verifier, covered later), shifting emphasis to RL, and widening task sources rather than piling up samples.

How this connects to the course

The previous lecture used SFT to train a computer use agent; this one opens the mechanism. The next three lectures cover reinforcement learning — basics, advanced algorithms, and training systems — and the checkpoint produced here is their starting point.