这一讲的定位

算法课关注「怎么更新」，这一讲关注「怎么把更新跑起来」。讲者给的目标很具体：如果你要用某个 RL 框架训练 agent，或者想 fork 一个框架改代码，你会遇到哪些系统问题。

整个 agentic RL 框架可以拆成五个原语：

1. 前向计算：既用于推理采样，也用于训练时重算；
2. 损失与反向传播；
3. 优化器更新；
4. 环境交互：执行动作、拿回观察；
5. 奖励与优势计算。

直觉上这些都是 PyTorch 的 forward、backward、optimizer.step，为什么还需要 Verl、Miles、SkyRL 这些复杂框架？

原因不在原语本身，而在推理与训练的工作负载特征完全不同。几乎所有框架都选择把两者拆成独立引擎，再在中间做同步。

时间轴

推理引擎与训练引擎的分工

把两者分开，才能各自做专用优化；代价是需要在两个引擎之间传递数据与权重，并且同一个模型在两个引擎里的前向实现可能略有差异，即使权重相同，采样出的分布也会有细微不同，从而产生轻微的 off-policy。

推理侧的关键优化

KV 缓存有多大

自回归解码时，前面 token 的 key/value 已经算过，不应该每步重算。缓存下来能省大量计算，但代价是显存。课程给了一个具体数字：Llama 3 8B、BF16 精度下，64 条 32k token 的轨迹需要约 256GB 的 KV 缓存。这还只是小模型。

这也是推理与训练必须分开的直接原因：如果统一引擎，还要同时保留训练的激活，KV 缓存几乎没有空间。

前缀缓存

KV 缓存解决的是单个请求内部的重复；请求之间同样有大量共享前缀。

agent 的每一步都是一次独立的 LLM 请求，但上一步的完整历史就是下一步的前缀。GRPO 的组内采样又共享 system prompt 与任务描述。如果不做前缀复用，这部分 prefill 会被反复计算。

• SGLang 用 radix attention 管理共享前缀；
• vLLM 提供 automatic prefix caching。

对 agentic RL 来说，前缀缓存不是锦上添花，而是主要性能来源之一。

连续批处理

朴素批处理把一批请求同时调度，等最长的那个结束。轨迹长度差异很大时，大量算力空转。

连续批处理在每个解码步重新检查：是否有请求完成、是否有容量接纳新请求。这样短请求结束后立刻让出位置。

实际引擎还要处理 prefill：长 prompt 的并行预填充会阻塞正在解码的请求，于是有 chunked prefill，把长 prompt 切块，与解码交错执行。

分离式推理

更进一步是把 prefill 和 decode 放到不同的 GPU 池：一组专门做预填充，另一组专门做解码。这样解码池可以只做连续批处理，不受长 prompt 干扰；也能针对两类负载选用不同硬件。

投机解码

用一个小而快的草稿模型连续预测若干 token，再交给大模型用拒绝采样验证，保证结果等价于从大模型采样。

在 RL 中有一个特殊问题：训练过程中策略不断漂移，草稿模型的接受率会下降。常见做法是在训练主策略的同时，用一个辅助的 SFT 或蒸馏目标持续更新草稿模型。

训练侧的关键优化

激活检查点

反向传播需要前向的激活。全部保留会撑爆显存；激活检查点只保留每层输入，反向时重算缺失的激活。代价约为额外 33% 的计算时间，换来显著的显存节省。显存受限时这是首选手段。

训练侧的前缀共享

训练批次里同样有大量重复前缀，但训练还要考虑激活与反向，不能照搬推理的做法。

• Snowflake 的 Arctic RL 使用 Zoro；
• 另一种是 dynamic tree attention：把批次组织成前缀树，按深度优先遍历，进入分支时前向，离开分支时反向并释放激活。这样任意时刻保留的激活不超过一条根到叶的路径。

并行策略

经验法则是：尽量最大化数据并行，减少张量并行带来的通信开销。

两个引擎之间

权重同步

一个一万亿参数的模型在 BF16 下约 2TB。每次更新都把全部权重广播到所有推理 rank，代价不可接受。

观察是：RL 更新非常稀疏。BF16 精度下，约 99% 的参数在一次更新后逐位不变。因此只传 delta，数据量可以降到 20–30GB，跨集群传输才变得可行。

同步后还要重新切分：训练与推理的并行布局不同，权重需要按推理引擎的布局重排。

MoE 路由一致性

MoE 模型在前向时要选择专家。训练引擎与推理引擎的实现差异，甚至路由打平，都可能让两边选择不同的专家，导致训练看到的分布与采样时不同。

解法很直接：rollout 时记录每个 token 的路由决策，训练时丢弃训练引擎自己的路由，重放推理时的决策（R3）。课堂建议，如果 MoE 训练 reward 不涨，先检查 R3 是否开启。

Agent 训练特有的问题

序列扩展与上下文压缩

对比两种 harness：

• A：每步把新动作追加到完整历史；
• B：每隔几步把历史压缩成摘要，再继续。

生产系统常用 B（Codex、Claude Code、OpenHands 都会压缩），但从训练效率看 A 更好。

原因在数据组织：A 的整条轨迹是一个连续前缀，可以作为一个数据点，只 mask 掉观察 token；B 在压缩处产生断点，必须拆成两个数据点，任务 prompt 被重复 prefill。如果一定要用 B，就需要 dynamic tree attention 这类技术复用共享前缀。

默认 chat template 会破坏前缀

Qwen 3 / 3.5 的默认模板在每个新的用户轮次会删掉历史中的推理 token，以节省上下文。这相当于一次隐式压缩，同样破坏了序列的连续性。

因此许多 RL 框架自带 renderer：Tinker 提供自定义渲染，Prime RL 允许按模型配置是否保留推理。课程作业里也提供了一个为 Qwen 3.5 保留前缀的模板。

传 token，不要传字符串

早期框架在推理引擎与训练引擎之间传字符串，因为推理 API 本来就是字符串接口。后来发现这是错的：

1. 字符串层面的微小变化（例如多一个空格）会改变 tokenization，训练分布与采样分布不一致；
2. token → 字符串是多对一的。同一段文本可能有多种 token 表示，重新 tokenize 未必得到原来的 token 序列。

两个方向叠加，就会产生 off-policy，表现为 loss 尖峰和训练不稳定。现在几乎每个框架都保存精确的 token ID 并直接传给训练引擎。

异步训练与 in-flight 更新

同步 RL 会把训练引擎和推理引擎共置，轮流使用全部 GPU。异步 RL 让两者同时运行，可以把 GPU 按比例分配，常见是 3:1，即四分之三用于 rollout，四分之一用于训练，因为长轨迹的生成通常是瓶颈。

异步带来一个奇怪的操作：in-flight 权重更新。生成到一半时策略更新了，需要暂停请求、换上新权重、再继续。

难点在 KV 缓存：旧前缀的 KV 是用旧权重算的，已经失效。SGLang 的 pause generation API 提供 retract 模式，暂停时清空 KV 缓存，用新权重重新计算。

复杂 harness 的批处理

递归多智能体系统会产生一棵调用树：根 agent 派生子 agent，子 agent 还能继续派生。训练时要决定：

• 整棵树算一个样本，还是每个子轨迹单独算？
• 压缩产生的断点如何切分？
• 子 agent 的上下文如何回传？

这些选择直接影响批处理效率与梯度信号。

框架设计：从 SPMD 到服务化

harness 无关的代理层

agent harness 越来越多（Codex、Claude Code、OpenHands、Pi 等）。如果每换一个 harness 就要改 RL 框架，维护成本不可接受。

做法是在 harness 与推理引擎之间放一个代理服务，模拟 harness 熟悉的 chat completions / responses API：

1. 接收 harness 的请求；
2. 转发给推理引擎；
3. 把响应返回给 harness；
4. 同时把训练所需的数据（logprob、路由决策等）存下来。

harness 不需要改代码，框架也不需要为每个 harness 做适配。

SPMD 与服务化

• SPMD：所有 GPU 跑同一份 Python 程序，按 rank 条件执行不同分支。优点是全局视图清晰、改动直接；缺点是僵化。
• 服务化 + 编排器：训练服务、推理服务、权重服务、环境服务各自独立，通过标准协议通信，中央编排器只是一个轻量 CPU 循环。优点是解耦、可替换、可弹性伸缩；缺点是协议与运维更复杂。

服务化带来的能力：

• 不同厂商、不同型号的 GPU 混合使用；
• 训练与推理解耦后独立扩缩容；
• 多策略训练：不同 agent 用不同权重甚至不同架构；
• 在线 RL：把生产中的 harness 当作数据源，持续更新模型（Cursor 的 Composer 被提到会周期性地用新 checkpoint 服务用户）；
• 多租户 / 多 LoRA：为每个用户维护一组 LoRA 权重，并发前向与反向，吞吐显著高于串行。

标准训练 API（例如 Tinker API）让不同框架可以互换；Miles、SkyRL 等开源实现也开始兼容。

本讲结论

• 推理与训练的工作负载差异，决定了它们必须分成两个引擎。
• 推理优化围绕 KV 缓存与批处理，训练优化围绕激活与并行。
• 两个引擎之间要同步权重、重切分、并处理 MoE 路由与 token 一致性。
• 训练稳定性很大程度取决于「采样时看到的 token」与「训练时看到的 token」是否完全一致。
• 异步训练与 in-flight 更新是常态，KV 缓存失效必须显式处理。
• 服务化架构换来解耦、异构与弹性，代价是协议复杂度。

理解检验

1. 为什么不能用一个 PyTorch 循环同时完成采样与训练？

参考答案：因为推理与训练的工作负载特征几乎相反。推理是逐 token 自回归解码、本质串行，不需要自动微分、可以直接丢弃计算图，显存主要被 KV 缓存占据，优化方向是 KV 缓存、前缀缓存、连续批处理与投机解码；训练拿到的是完整序列、可以并行，必须保留激活与计算图，显存主要被激活、梯度与优化器状态占据，优化方向是激活检查点与并行切分。若合成一个引擎，KV 缓存与训练激活要同时占显存，课程给的数字很直观：Llama 3 8B、BF16 下 64 条 32k token 的轨迹就要约 256GB 的 KV 缓存。分开才能各自做专用优化；代价是要在两个引擎之间传数据与权重，而且同一模型在两边的前向实现可能略有差异，即使权重相同，采样分布也会有细微不同，产生轻微的 off-policy。

2. 前缀缓存在 agent 轨迹中为什么比普通对话更重要？

参考答案：普通对话里每个请求相对独立，共享前缀有限；agent 的每一步都是一次独立的 LLM 请求，而上一步的完整历史就是下一步的前缀，GRPO 的组内采样又共享 system prompt 与任务描述，共享前缀随步数不断累积、体量很大。不做前缀复用，这部分 prefill 会被反复计算。课程明确说，对 agentic RL 来说前缀缓存不是锦上添花，而是主要性能来源之一（SGLang 的 radix attention、vLLM 的 automatic prefix caching 都在做这件事）。

3. 为什么训练与推理之间传字符串会导致 off-policy？传 token 解决了什么？

参考答案：两个原因叠加。第一，字符串层面的微小变化（例如多一个空格）会改变 tokenization，使得训练分布与采样分布不一致。第二，token 到字符串是多对一的，同一段文本可能有多种 token 表示，重新 tokenize 未必得到原来的 token 序列。于是训练时条件化的输入与采样时实际生成的条件不同，产生 off-policy，表现为 loss 尖峰与训练不稳定。传精确的 token ID 则保证训练看到的就是采样时看到的那串 token，现在几乎每个框架都保存 token ID 直接传给训练引擎。

4. 上下文压缩为什么会增加训练计算量？有哪些办法缓解？

参考答案：压缩会在轨迹中产生断点：harness A 每步把新动作追加到完整历史，整条轨迹是一个连续前缀，可以作为单个数据点、只 mask 掉观察 token；harness B 每隔几步压缩成摘要，压缩处不再共享前缀，必须拆成两个数据点，任务 prompt 被重复 prefill，训练计算量因此上升。缓解办法：用 dynamic tree attention 这类技术把批次组织成前缀树、复用共享前缀；或者在训练时优先使用不压缩的 harness A（生产系统可以用 B，训练用 A）。补充：Qwen 3/3.5 的默认 chat template 在每个新用户轮次删掉历史推理 token，等于一次隐式压缩，同样破坏前缀连续性，这也是许多框架自带 renderer 的原因。

5. MoE 训练中 R3 重放的是什么？不做会发生什么？

参考答案：R3 重放的是 rollout 时每个 token 的专家路由决策：采样阶段记录每个 token 选了哪些专家，训练阶段丢弃训练引擎自己的路由结果、按记录重放推理时的决策。不做的话，训练引擎与推理引擎的实现差异、甚至路由打平时的不同选择，都可能让两边选中不同的专家，训练看到的分布就与采样时不同，表现为 MoE 训练 reward 不涨。课堂的建议很直接：如果 MoE 训练 reward 不涨，先检查 R3 是否开启。

与前后课程的关系

上一讲是算法层面的稳定性；这一讲是系统层面的支撑。下一讲转向安全：当 agent 真的拥有工具和凭据时，威胁模型、沙箱、凭据代理与监控如何组合成防线。

What this lecture covers

The algorithm lecture asks how to update. This one asks how to make the update run. The stated goal is practical: if you use an RL framework to train agents, or fork one to modify it, these are the system problems you will meet.

An agentic RL framework reduces to five primitives:

1. Forward passes, used both for rollout sampling and for recomputation during training.
2. Loss and backward pass.
3. Optimizer step.
4. Environment interaction: execute actions, receive observations.
5. Reward and advantage computation.

At first glance these are forward, backward, and optimizer.step in PyTorch. Why do Verl, Miles, SkyRL and similar frameworks exist?

The reason is not the primitives but the fact that inference and training have very different workload characteristics. Nearly every framework therefore splits them into separate engines and synchronizes between them.

Timeline

The division of labor

Separating them allows dedicated optimizations for each. The cost is data and weight movement between engines, plus a subtler issue: the forward implementation can differ slightly between engines, so even with identical weights the sampled distribution differs a little, producing mild off-policy drift.

Inference-side optimizations

How large the KV cache is

During decoding, keys and values for earlier tokens have already been computed and should not be recomputed. Caching them saves substantial compute at the cost of memory. The lecture gives a concrete number: Llama 3 8B in BF16 needs about 256GB of KV cache for 64 trajectories of 32k tokens. And that is a small model.

This is the direct reason the engines must be separated: a unified engine would also have to hold training activations, leaving almost no room for the cache.

Prefix caching

KV caching avoids recomputation inside one request; requests also share large prefixes.

Every step of an agent is a separate LLM request, but the previous step's entire history is the next step's prefix. GRPO groups also share a system prompt and task description. Without reuse, that prefill is paid repeatedly.

• SGLang manages shared prefixes with radix attention.
• vLLM offers automatic prefix caching.

For agentic RL this is not a nice-to-have; it is a primary source of performance.

Continuous batching

Naive batching schedules a set of requests together and waits for the longest. With widely varying trajectory lengths, much of the compute idles.

Continuous batching re-examines the batch at every decode step: which requests finished, and whether capacity exists to admit a new one. Short requests free their slots immediately.

Real engines also handle prefill. A long prompt's parallel prefill blocks decoding requests, so chunked prefill splits long prompts into chunks interleaved with decode.

Disaggregated inference

A further step dedicates separate GPU pools to prefill and decode. The decode pool then runs plain continuous batching without interference from long prompts, and each workload can use hardware suited to it.

Speculative decoding

A small, fast draft model proposes several tokens, and the large model verifies them with rejection sampling that preserves equivalence to sampling from the large model.

RL adds a twist: as the policy drifts during training, the draft model's acceptance rate falls. The common fix is an auxiliary SFT or distillation objective that keeps updating the draft model alongside the policy.

Trainer-side optimizations

Activation checkpointing

Backward passes need forward activations. Keeping all of them exhausts memory; checkpointing keeps only each layer's input and recomputes the missing activations during backward. The cost is roughly 33% extra compute for a large memory saving. Under memory pressure, it is the first tool to reach for.

Prefix sharing in training

Training batches also contain repeated prefixes, but training must account for activations and backward, so inference techniques do not transfer directly.

• Snowflake's Arctic RL uses a system called Zoro.
• Another approach is dynamic tree attention: organize the batch as a prefix tree, traverse depth-first, run forward when entering a branch and backward when leaving it, releasing activations as you go. At any moment, retained activations never exceed one root-to-leaf path.

Parallelism strategies

The rule of thumb: maximize data parallelism and minimize tensor parallelism, because the latter carries the most communication overhead.

Between the two engines

Weight synchronization

A one-trillion-parameter model in BF16 is about 2TB. Broadcasting all weights to every inference rank on every update is infeasible.

The observation that makes it work: RL updates are extremely sparse. In BF16, roughly 99% of parameters are bit-for-bit unchanged after one update. Syncing only the delta reduces the transfer to 20–30GB, which is feasible even across clusters.

After syncing, weights must be resharded to match the inference engine's layout.

MoE routing consistency

MoE models select experts during the forward pass. Implementation differences, or even tied routing scores, can make trainer and inference engine choose different experts, so training sees a different distribution from sampling.

The fix is direct: record each token's routing decisions during rollout and, in the trainer, discard its own decisions and replay the recorded ones (R3). The lecture suggests checking whether R3 is enabled first if MoE training reward is not increasing.

Problems specific to agent training

Sequence extension and compaction

Two harnesses:

• A appends each new action to the full history.
• B periodically summarizes history and continues.

Production systems usually use B (Codex, Claude Code, and OpenHands all compact), but for training efficiency A is better.

The reason is data organization. A's whole trajectory is one contiguous prefix, so it can be one data point, with observation tokens masked out. B creates a break at each compaction and must be split into two data points, duplicating the task-prompt prefill. If B is unavoidable, techniques such as dynamic tree attention recover some of the shared computation.

Default chat templates break the prefix

Qwen 3 / 3.5's default template removes reasoning tokens from history at each new user turn to save context. That is effectively an implicit compaction, and it breaks sequence continuity.

Many RL frameworks therefore ship their own renderers: Tinker provides custom rendering, Prime RL lets you configure per model whether reasoning is preserved. The course assignment includes a template for Qwen 3.5 that preserves the prefix.

Pass tokens, not strings

Early frameworks passed strings between engines because inference APIs are string-based. This turned out to be wrong:

1. Small changes in string space, such as an extra space, change tokenization, so the training distribution differs from the sampling distribution.
2. Token-to-string is many-to-one. The same text can have multiple token representations, and retokenizing need not recover the original sequence.

Together these produce off-policy data, visible as loss spikes and instability. Nearly every framework now stores exact token IDs and hands those to the trainer.

Async training and in-flight updates

Synchronous RL colocates the engines and alternates: all GPUs train, then all GPUs infer. Async RL runs both at once and can split the GPUs, commonly 3:1 in favor of rollout, because generating long trajectories is usually the bottleneck.

Async introduces a strange operation: in-flight weight updates. If the policy updates mid-generation, the request must pause, take the new weights, and continue.

The hard part is the KV cache: prefixes computed with old weights are stale. SGLang's pause-generation API offers a retract mode that flushes the cache on pause and recomputes with the new weights.

Batching inside complex harnesses

Recursive multi-agent systems form a call tree: a root agent delegates to sub-agents, which may delegate further. Training must decide:

• Is the whole tree one sample, or is each sub-trajectory its own?
• How are compaction breaks split?
• How does sub-agent context return to the parent?

These choices directly affect batching efficiency and the gradient signal.

Framework design: from SPMD to services

A harness-agnostic proxy

Harnesses multiply (Codex, Claude Code, OpenHands, Pi, and more). If every new harness required changes to the RL framework, maintenance would be untenable.

The pattern is a proxy service between harness and inference engine that emulates the chat completions or responses API:

1. accept the harness's request;
2. forward it to the inference engine;
3. return the response to the harness;
4. simultaneously store the data training needs, such as logprobs and routing decisions.

The harness needs no changes and the framework needs no per-harness adapter.

SPMD versus service-oriented design

• SPMD: every GPU runs the same Python program and branches on rank. Easy to reason about globally and to modify; inflexible.
• Service-oriented with an orchestrator: training, inference, weights, and environment are independent services speaking standard protocols, coordinated by a lightweight CPU loop. Decoupled, replaceable, elastic; more protocol and operational complexity.

What service orientation enables:

• mixing vendors and GPU models;
• independent scaling of training and inference;
• multi-policy training, where different agents use different weights or architectures;
• online RL, treating a deployed harness as a data source (Cursor's Composer is mentioned as serving new checkpoints periodically);
• multi-tenant / multi-LoRA training, maintaining one LoRA per user and running forward and backward concurrently, for much higher aggregate throughput than serial runs.

A standard training API, such as the Tinker API, lets frameworks be swapped; open-source implementations such as Miles and SkyRL are adopting it.

Takeaways

• The workload difference between inference and training forces separate engines.
• Inference optimizations center on KV cache and batching; training optimizations center on activations and parallelism.
• The engines must sync weights, reshard, and reconcile MoE routing and tokenization.
• Stability depends heavily on training seeing exactly the tokens that were sampled.
• Async training and in-flight updates are normal, and stale KV caches must be handled explicitly.
• Service-oriented architecture buys decoupling, heterogeneity, and elasticity at the cost of protocol complexity.

Check your understanding

1. Why can't one PyTorch loop handle both sampling and training?

Reference answer: Because the two workloads are nearly opposite. Inference decodes token by token, serially, needs no autograd and can discard the graph, and its memory is dominated by the KV cache; it is optimized with KV caching, prefix caching, continuous batching and speculative decoding. Training receives whole sequences and can parallelize, must retain activations and the graph, and its memory is dominated by activations, gradients and optimizer states; it is optimized with activation checkpointing and parallelism. In one engine the KV cache and the training activations compete for the same memory — the lecture's number is concrete: Llama 3 8B in BF16 needs about 256GB of KV cache just for 64 trajectories of 32k tokens. Splitting lets each side use its own optimizations; the price is passing data and weights between engines, and the two forward implementations may differ slightly, so even with identical weights the sampling distribution shifts a little, producing mild off-policy data.

2. Why is prefix caching more important for agent trajectories than for ordinary chat?

Reference answer: In ordinary chat each request is fairly independent and shared prefixes are limited. In an agent every step is a separate LLM request, yet the previous step's entire history is the next step's prefix, and GRPO's group sampling additionally shares the system prompt and task description; the shared prefix accumulates across steps and becomes very large. Without reuse, that prefill is recomputed over and over. The lecture is explicit: for agentic RL, prefix caching is not a nicety but one of the main sources of performance (SGLang's radix attention and vLLM's automatic prefix caching both target it).

3. Why does passing strings between engines cause off-policy data, and what does passing token IDs fix?

Reference answer: Two effects compound. First, a tiny string-level change such as an extra space alters tokenization, so the training distribution differs from the sampling distribution. Second, token-to-string is many-to-one: the same text can have several token representations, and re-tokenizing need not recover the original token sequence. The training input is then conditioned on something other than what was actually sampled — off-policy data, showing up as loss spikes and unstable training. Passing exact token IDs guarantees training sees precisely the tokens that were sampled; nearly every framework now stores token IDs and hands them straight to the trainer.

4. Why does context compaction increase training compute, and how can that be mitigated?

Reference answer: Compaction creates breakpoints. With harness A, each step appends the new action to the full history, so the whole trajectory is one contiguous prefix and can be a single data point with only observation tokens masked. With harness B, the history is summarized every few steps; the prefix is no longer shared across the break, so the trajectory must be split into two data points and the task prompt is re-prefilled, raising training compute. Mitigations: use dynamic tree attention to organize the batch as a prefix tree and reuse shared prefixes, or train with the uncompacted harness A while production uses B. Note also that Qwen 3/3.5's default chat template deletes historical reasoning tokens at each new user turn — an implicit compaction that breaks prefix continuity, which is why many frameworks ship their own renderer.

5. What does R3 replay in MoE training, and what happens without it?

Reference answer: R3 replays each token's expert routing decisions from rollout: the sampling phase records which experts each token selected, and training discards the training engine's own routing and replays the inference-time decisions. Without it, implementation differences between the training and inference engines — or different tie-breaking in routing — can make the two sides pick different experts, so the distribution seen in training differs from the one sampled, and MoE training reward fails to rise. The lecture's advice is blunt: if MoE reward is flat, check whether R3 is enabled first.

Connections

The previous lecture covered algorithmic stability; this one covers the systems that support it. The next turns to safety: threat models, sandboxing, credential brokering, and monitoring once agents actually hold tools and credentials.