第 03 章

长上下文：容量、效率与压缩

Agent 的上下文会随轮数迅速膨胀，必须同时解决模型能不能装下（容量）和系统能不能负担（效率）两个问题：混合注意力架构、KV 缓存与上下文压缩是三条主线。

Long Context: Capacity, Efficiency, and Compaction

Agent context grows quickly with every turn, so two separate problems must be solved: whether the model can use a long context, and whether the system can afford to serve it.

首发于 2026.09.08中 / EN

系统课程ai-agentslong-contextkv-cacheattention

篇内目录

上下文为什么会爆炸

每次调用都要把历史重新送进模型。若第一轮 6 个 token、每轮新增 5 个，第二轮回看 11 个、第三轮 16 个……累计处理量随轮数平方增长。而真实 agent 的会话远不止五六轮：课堂上提到有人让编码 agent 连续工作好几天。

课程分析了 OpenHands 的 1500 个会话（每个只解决一个任务，不是多日长会话），平均消耗约 78,000 个 token。构成如下：

组成	占比	说明
系统提示 + 工具描述	23%	约 15k–18k token，工具越多越大
用户消息	9%	初始任务与后续补充
推理轨迹	9%	模型的思考过程
模型回复	2%	直接给用户看的文本
工具调用	20%	写代码、读文件
工具结果	37%	最大的一块

两个值得记住的比例：模型花在「想」上的 token 约是花在「说」上的五倍；工具结果加工具调用占了近六成。

时间轴

时间	内容
0:07	课程开场：上下文管理与长上下文架构
1:10	agent 上下文增长：输入开销随长度平方上升
2:37	提示构成：系统提示、工具定义、对话与工具输出
4:29	两个挑战：容量与效率
5:59	推理两阶段：prefill 与 decode
11:04	注意力计算：Q/K/V 与二次复杂度
11:54	有效性：长上下文中的「lost in the middle」
16:08	架构分类：局部/全局混合与稀疏检索
19:00	滑动窗口注意力
20:08	线性注意力与状态矩阵，以及表达力的折扣
29:47	Gated Delta 与 Qwen 模型
31:10	稀疏注意力：固定间隔与内容相关
33:41	KV 压缩：减少头数与低秩投影
36:22	训练长上下文：长文档、专利与合成任务
38:42	位置编码：RoPE、NoPE 与长度外推
52:06	KV 缓存与提示缓存的经济学
57:41	分页显存与 RadixAttention
1:06:23	上下文压缩：保留什么、丢掉什么
1:10:24	压缩的触发方式：阈值、slash compact、加密压缩
1:12:51	压缩效果的评测

两个独立的问题

容量：模型能不能真正利用这么长的上下文。需要架构、位置编码、训练数据与评估共同解决。
效率：系统能不能负担得起。需要注意力近似、KV 缓存与调度。

这两件事经常被混为一谈。「支持 1M 上下文」只是说装得下，不等于用得好。

推理系统的基本盘

Prefill（预填充）：一次性处理输入 token，生成每层的 key/value。
Decode（解码）：逐 token 生成，每步复用之前的 KV，只追加新的。
KV cache：把已算过的 key/value 存下来，避免重复计算。

衡量服务质量的四个指标：

指标	含义	什么时候重要
TTFT	从发出请求到第一个 token 的时间	交互式编码
TPOT	相邻输出 token 的间隔	交互式编码
吞吐	单位时间内完成的 token 或请求	后台批处理
成本	每百万 token 的价格	所有场景

课堂给了一个很实际的建议：服务商通常会问你更在意延迟还是吞吐，因为优化方向相反。深夜跑批任务的 agent 完全不必关心 TTFT。

注意力为什么贵

标准注意力让每个 token 和之前所有 token 做两两比较，复杂度 $O(n^2)$ 。100 万 token 意味着约 $10^{12}$ 次比较，每次比较还要做向量乘法。

几乎所有长上下文架构都在做同一件事：给每一步的比较次数加上一个上限，把 $O(n^2)$ 降成 $O(n \cdot w)$ 或 $O(n)$ 。

局部与全局的混合

局部模型：计算被限制在局部窗口内。
全局模型：计算上限不局限于某个位置，信息可以跨整段序列传播。

现在主流开源模型几乎都是混合体：多次局部计算夹一次全局计算。课堂列了六个例子：

模型	局部机制	全局机制	比例
Qwen（最新）	gated delta net	稀疏检索	3:1
GLM 5.3	delta 变体	稀疏注意力	3:1
Kimi K3	delta 变体	稠密注意力	3:1
Nemotron	Mamba	稠密注意力	4:1
某滑窗模型	滑动窗口注意力	稠密注意力	5:1
DeepSeek	窗口注意力	带交错分支的稀疏注意力	特殊

滑动窗口注意力

最简单的一种：当前块只能看前面固定数量的块（例如每块 512 token）。单层看得很近，但第 2 层的前一块已经包含它更早的信息，于是感受野逐层扩大。实现几乎不改注意力本身。

线性注意力与 DeltaNet

线性注意力去掉了 softmax。代价是表达能力下降：模型失去了「只关注少数相关 token」的非线性选择能力。收益是可以利用结合律，把对历史所有 key/value 的求和重排成一个状态矩阵 $S$ 。

这个矩阵是「查询 → 值」的线性映射，可以增量更新：每来一个新 token，用常数时间更新 $S$ ，整体复杂度从 $O(n^2)$ 降到 $O(n)$ ，实质上变成了循环神经网络。

DeltaNet 的思路是让更新更聪明：

用当前 key 和旧矩阵预测一个值 $\hat v_t = k_t^\top S_{t-1}$ ；
与真实值 $v_t$ 比较得到误差；
用 $k_t \times \text{误差}$ 修正矩阵，并乘一个类似学习率的 $\beta$ 控制步长；
每步再乘一个衰减系数 $\alpha$ ，让久远信息逐渐失效。

衰减项解决了「很久以前一个异常的 key 永久污染状态」的问题。GLM 与 Kimi 用的是更细粒度的逐元素衰减：不同维度以不同速度遗忘，于是有些信息能留很久，有些很快消失。

一句话总结： 线性注意力用「可增量更新的状态矩阵」换掉了二次复杂度，代价是失去显式的两两比较；DeltaNet 用预测误差与衰减把表达能力补回来一部分。

稀疏注意力

不跟所有历史比较，只挑一部分：

固定步长稀疏：每隔 512 个 token 取一个，仍然是二次复杂度，但常数小 512 倍。
内容相关稀疏：先用很便宜的方式（例如把向量降维）估算相关性，选 top-k，再对这 k 个做完整注意力。DeepSeek 采用这类方法。

KV 压缩

三种常见手段，可以叠加：

GQA：减少 key/value 的头数（通常除以 4 或 8）；
MQA：只保留一组 key/value，是 GQA 的极端情况；
MLA：不减少头数，而是把每个头的 key/value 维度降下来。

目的都是缩小 KV cache 的显存占用，让更长的上下文能放进 GPU。

训练长上下文模型

数据比架构更难

互联网上「连贯的 128k token」并不多——超过大多数维基百科条目，接近一本书的长度。典型训练流程是：

短上下文预训练；
用较长文档继续训练；
长上下文适配。

packed sequences（把多篇文档拼进一个窗口）能提升训练效率，但学不到跨文档的连贯性，因为模型本就不该在无关文档之间建立注意力。长上下文适配需要真正连贯的长数据。

可用来源包括：书籍、代码仓库（天然连贯）、同主题多文档语料、专利、agent 轨迹，以及合成任务。

位置编码与长度外推

绝对位置编码：每个位置一个向量。
RoPE：让编码只依赖相对距离，数学上通过旋转实现。
NoPE：完全不用位置编码。课堂上给了一个反直觉的论证：在因果掩码下，位置 2 的 token 只能看到 1 个前文，位置 5 的能看到 4 个，表示天然不同。因此自回归 Transformer 并不需要显式位置编码；非自回归模型仍然需要。

RoPE 的长度外推有两个问题：训练时最长只见过 $L$ 的距离，超出后注意力会失效；而且相对位置只是众多因素之一，因果掩码也在起作用。

常见修法：

位置插值（PI）：把 RoPE 的 $\theta$ 除以扩展倍数，相当于把位置坐标拉伸；
YaRN：不同频率以不同幅度扩展，让一部分维度负责远距离、一部分负责近距离。

并行训练

长上下文训练需要 context parallelism：把一条序列的 KV 切到多张卡上依次计算。

它和其他并行方式并存：数据并行、张量并行、流水线并行、专家并行。麻烦在于新注意力机制与系统实现会冲突——课堂上举了一个真实例子：想给 35B 的 Qwen 做 context parallel，却发现网上没有 gated delta net 的 kernel，只能自己写。标准注意力与滑动窗口的 kernel 通常现成可用。

KV 缓存：省钱的关键

复用前缀

第 1 步算出输入的 KV 并缓存；第 2 步只需计算新增的工具结果，前面的 KV 直接复用。到第 50 步，新增内容相对于全部历史很小，缓存带来的节省非常可观。

论文数据显示：不缓存时计算量随长度二次增长，缓存后接近线性。

价格上也一样：缓存 token 通常只有未缓存 token 的十分之一。课堂上给出的经验值是缓存命中率 90%–95% 才算健康。另一个数字是输出 token 的价格约为输入的 5 倍，因此「输出 vs 缓存输入」的价差可达 50 倍。

两种开源实现

vLLM 的 PagedAttention：把 GPU 显存切成固定大小的块，像操作系统管理内存页一样按需分配与回收，支持大量序列并行生成。
SGLang 的 RadixAttention：用基数树索引已经计算过的前缀，命中相同前缀就复用。最初为聊天场景设计，但对 agent 更重要，因为 agent 的对话更长、更频繁。

多机部署还需要缓存感知路由：把请求送到持有最佳 KV 缓存的那台机器。缓存会被驱逐（不同服务商从几分钟到一小时不等），驱逐策略直接影响命中率。课堂提到 OpenRouter 甚至提供按缓存命中率排列的服务商榜单。

对脚手架设计的硬约束

缓存只在前缀完全一致时有效。由此得到几条规则：

不要每步改系统提示。最常见的错误是把当前时间日期拼进 system message——每一步都变，等于完全没有缓存。需要时间信息就放到最后一条用户消息或工具结果里。
可以中途改工具列表，这不会破坏已有前缀的复用。
谨慎切换模型。为省钱把某一步交给更强模型、后面换回便宜模型，下次再用强模型时仍要按未缓存价格重新计算，省下的钱可能被抵消。

上下文压缩

当上下文超过预算，就把历史压成摘要。问题是压缩会丢信息，轻则跑偏，重则删文件。

一个好的压缩策略要同时满足两点：

未来行为尽量不变：压缩后的后续动作序列与压缩前相同或更好；
状态有界：压缩结果足够小，能继续工作。

保留什么

锚点：会话最开始用户说的话，通常要保护；
中段摘要：把久远的动作与观察压成结构化或自由文本摘要；
近期尾部：最近几步原样保留；
证据库：完整历史通常仍存在磁盘上，需要时可以回读。

何时触发

token 阈值；
服务商的硬上限；
手动命令（例如 /compact）。

压缩的陷阱

反复压缩会累积损失：第一次效果尚可，越往后丢得越多。压缩摘要的格式也不统一，有的要求结构化字段，有的直接自由文本，还有的（例如某些 OpenAI 实现）加密后不让人看。

课堂上讲了 OpenHands 的亲身教训：他们做了一个压缩算法，在 SWE-bench 上分数不降、还省了大量 token，看起来很成功。上线后却出现「每次压缩就重复发一次 PR」——模型忘了自己已经发过；还会忘掉会话中途的用户指令，因为评测只覆盖了单轮、只有一条用户消息的场景。

结论很直接：基准测试通过不等于可用，必须在真实使用中验证。

本讲结论

Agent 上下文随轮数平方增长，工具结果与调用占大头。
容量与效率是两个问题：前者靠架构、位置编码与训练数据，后者靠近似注意力与缓存。
主流架构是「局部 + 全局」混合，局部多用线性注意力变体，全局用稀疏或稠密注意力。
线性注意力以表达力换线性复杂度，DeltaNet 用误差修正与衰减补偿。
长上下文训练缺的是连贯长数据，不是算力。
前缀缓存能省一个数量级成本，但要求前缀稳定；不要每步改系统提示。
压缩是超出预算后的兜底手段，必须用真实多轮场景验证。

理解检验

1. 为什么「模型支持 1M 上下文」不等于「模型能有效利用 1M 上下文」？

参考答案：这句话只回答了容量问题——窗口装得下。能不能用好是另一个问题，取决于四件事：(1) 架构，注意力能否在长距离上把信息传过来，全注意力的 $O(n^2)$ 成本与稀疏/线性近似都在做取舍；(2) 位置编码与长度外推，训练时最长只见过 $L$ ，超出后相对位置会失效，需要 PI、YaRN 一类修法；(3) 训练数据，互联网上真正连贯的 128k 文本很少，缺的是数据不是算力；(4) 评估，needle-in-a-haystack 这类检索测试通过，不代表长程推理与指令保持也没问题。

2. 线性注意力去掉 softmax，换来了什么、失去了什么？

参考答案：换来的是复杂度：去掉 softmax 后可以利用结合律，把「对历史所有 key/value 求和」重排成一个状态矩阵 $S$ ，每个新 token 用常数时间增量更新，整体从 $O(n^2)$ 降到 $O(n)$ ，本质上变成一个循环神经网络。失去的是表达能力：softmax 提供尖锐的非线性选择，使模型能只关注少数相关 token；线性注意力把这个选择平均掉了。DeltaNet 用预测误差修正与衰减把一部分表达能力补回来。

3. DeltaNet 的衰减项解决的是什么问题？逐元素衰减又多了什么能力？

参考答案：衰减项解决状态被永久污染的问题。没有衰减时，很久以前一个异常的 key 会一直留在状态矩阵里，无法被清除；每步乘一个衰减系数 $\alpha$ 后，久远信息的权重逐渐趋零，状态可以「遗忘」。逐元素衰减（GLM、Kimi 采用）让不同维度以不同速度遗忘：有些维度衰减慢，负责长期记忆，有些衰减快，负责短期细节，于是同一状态里可以并存不同时间尺度的信息。

4. 为什么把时间戳写进系统提示会严重伤害 agent 的成本？

参考答案：因为 KV 前缀缓存只在前缀完全一致时命中。系统提示排在最前面，往里写当前时间意味着每一步的 system message 都不同，整个前缀失效，全部历史都要按未缓存价格重新计算。而缓存 token 通常只有未缓存 token 的十分之一，健康系统的命中率需要 90%–95%；再叠加输出 token 约为输入 5 倍的价格，损失可达一个数量级以上。正确做法是把时间放进最后一条用户消息或工具结果里。

5. OpenHands 的压缩算法在基准上没问题，为什么上线后失败？这说明评估该怎么设计？

参考答案：他们的评测只覆盖单轮、只有一条用户消息的场景，所以压缩后 SWE-bench 分数不降、还省了大量 token。真实使用是多轮的：压缩让模型忘了「自己已经发过 PR」，于是每次压缩后重复发一次；也会忘掉会话中途用户追加的指令。这说明评估必须贴近真实交互：用多轮、带中途指令与重复动作检查的轨迹，并验证压缩前后后续行为序列是否保持一致（而不是只看最终答案），同时确认压缩结果有界、可继续工作。

与后续课程的关系

这一讲处理的是单次会话内的上下文。下一讲把时间尺度拉长到跨会话：记忆与技能如何让 agent 在没有完整历史时仍然保持一致。再往后是规划、编码 agent，以及把长上下文能力用于训练与评估。

Why agent context explodes

Every call resends the history. If the first call carries 6 tokens and each turn adds 5, the second call reads 11, the third 16, then 21 and 26. Cumulative processing grows quadratically with the number of turns. Real sessions go far beyond five or six turns; the lecture mentions coding agents driven for days.

The lecture analyzed 1,500 OpenHands sessions, each solving a single task rather than a multi-day project. The average session consumed about 78,000 tokens:

Component	Share	Note
System prompt + tool descriptions	23%	15k–18k tokens; grows with the tool count
User messages	9%	Initial task and follow-ups
Reasoning traces	9%	The model's thinking
Model replies	2%	Text shown to the user
Tool calls	20%	Writing code, reading files
Tool results	37%	The largest single block

Two ratios are worth remembering: the model spends about five times as many tokens thinking as talking, and tool calls plus results account for nearly 60% of the context.

Timeline

Time	Topic
0:07	Opening: context management and long-context architectures
1:10	Agent context growth: input cost rises quadratically with length
2:37	Prompt makeup: system prompt, tool definitions, dialogue, tool outputs
4:29	Two challenges: capacity and efficiency
5:59	Two inference phases: prefill and decode
11:04	Attention computation: Q/K/V and quadratic complexity
11:54	Effectiveness: "lost in the middle" in long-context evaluation
16:08	Architecture families: local/global hybrids and sparse retrieval
19:00	Sliding-window attention
20:08	Linear attention, the state matrix, and its expressiveness gap
29:47	Gated Delta and the Qwen models
31:10	Sparse attention: fixed-stride and content-dependent
33:41	KV compression: fewer heads and low-rank projection
36:22	Training long context: books, patents and synthetic tasks
38:42	Positional encodings: RoPE, NoPE and length extrapolation
52:06	The economics of KV and prompt caching
57:41	Paged memory and RadixAttention
1:06:23	Context compaction: what survives and what is lost
1:10:24	Compaction triggers: token thresholds, slash compact, encrypted compaction
1:12:51	Evaluating compaction

Two separate problems

Capacity: can the model actually use a long context? This needs architecture, positional encoding, training data, and evaluation.
Efficiency: can the system afford to serve it? This needs attention approximations, KV caching, and scheduling.

The two are often conflated. "Supports 1M tokens" means it fits, not that it works well.

The inference baseline

Prefill processes the input tokens once and produces key/value pairs for every layer.
Decode generates one token at a time, reusing earlier KV and appending only the new entries.
KV cache stores the computed keys and values so they are not recomputed.

Four serving metrics:

Metric	Meaning	When it matters
TTFT	Time from request to first token	Interactive coding
TPOT	Time between output tokens	Interactive coding
Throughput	Tokens or requests per unit time	Background batches
Cost	Price per million tokens	Everywhere

A practical point: providers usually ask whether you care more about latency or throughput, because the optimizations conflict. An overnight batch agent does not care about TTFT at all.

Why attention is expensive

Standard attention compares every token with every earlier token, which is $O(n^2)$ . A million tokens means about $10^{12}$ comparisons, each involving vector multiplication.

Almost every long-context architecture does the same thing: cap the number of comparisons per step, turning $O(n^2)$ into $O(n \cdot w)$ or $O(n)$ .

Local plus global

Local models cap computation inside a local window.
Global models allow information to travel across the whole sequence, whether or not computation is capped.

Nearly every popular model is a hybrid: many local computations interspersed with occasional global ones. The lecture lists six:

Model	Local mechanism	Global mechanism	Ratio
Qwen (latest)	gated delta net	sparse retrieval	3:1
GLM 5.3	delta variant	sparse attention	3:1
Kimi K3	delta variant	dense attention	3:1
Nemotron	Mamba	dense attention	4:1
A sliding-window model	sliding window attention	dense attention	5:1
DeepSeek	windowed attention	sparse attention with interleaved branches	special

Sliding window attention

The simplest option: the current chunk attends to a fixed number of previous chunks (say 512 tokens each). A single layer sees only nearby context, but by layer two the previous chunk already contains earlier information, so the receptive field expands with depth. No change to the attention operation itself.

Linear attention and DeltaNet

Linear attention removes the softmax. The cost is expressiveness: the model loses the nonlinear ability to focus on a few relevant tokens. The gain is that associativity lets the sum over all past keys and values be rearranged into a state matrix $S$ .

That matrix is a linear map from query to value, and it can be updated incrementally: each new token updates $S$ in constant time, reducing $O(n^2)$ to $O(n)$ and effectively making the model recurrent.

DeltaNet makes the update smarter:

Predict a value from the current key and the old matrix, $\hat v_t = k_t^\top S_{t-1}$ .
Compare with the true value $v_t$ to get an error.
Correct the matrix by $k_t \times \text{error}$ , scaled by a learning-rate-like $\beta$ .
Multiply by a decay factor $\alpha$ each step so distant information fades.

The decay solves a real problem: without it, one anomalous key from long ago can contaminate the state forever. GLM and Kimi use per-element decay, so different dimensions forget at different rates and some information persists much longer than the rest.

One-sentence summary: Linear attention trades explicit pairwise comparison for an incrementally updated state matrix; DeltaNet buys back expressiveness with error correction and decay.

Sparse attention

Instead of comparing against all history, select part of it:

Strided sparse attention: sample every 512th token. Still quadratic, but with a constant 512 times smaller.
Content-dependent sparse attention: estimate relevance cheaply (for example, down-project the vectors), take the top k, then run full attention over those k. DeepSeek uses this style.

KV compression

Three techniques, which can be combined:

GQA reduces the number of key/value heads, often by 4 or 8.
MQA keeps a single key/value group, the extreme case of GQA.
MLA keeps the head count but down-projects each head's key/value dimension.

All three shrink the KV cache so longer contexts fit in GPU memory.

Training long-context models

Data is harder than architecture

There is not much coherent 128k-token text on the internet — more than most Wikipedia articles, close to a book. The usual curriculum is:

short-context pre-training;
continued training on longer documents;
long-context adaptation.

Packed sequences — concatenating many documents into one window — help efficiency but do not teach cross-document coherence, because the model should not attend across unrelated documents. Adaptation needs genuinely coherent long data.

Sources include books, code repositories (naturally coherent), topically related document collections, patents, agent trajectories, and synthetic tasks.

Positional encoding and length extrapolation

Absolute positional encodings assign a vector to each position.
RoPE makes the encoding depend on relative distance, implemented with rotations.
NoPE omits positional encoding entirely. The lecture gives the counterintuitive argument: under causal masking, a token at position 2 sees one predecessor while a token at position 5 sees four, so their representations already differ. Autoregressive transformers therefore do not need explicit positions; non-autoregressive ones still do.

RoPE extrapolation has two problems. A model trained on length $L$ has never seen two tokens attend across a distance greater than $L$ . And relative position is only one factor; the causal mask matters too.

Common fixes:

Position interpolation: divide the RoPE $\theta$ by the extension factor, effectively stretching the coordinates.
YaRN: extend different frequencies by different amounts, so some dimensions handle long range and others handle short range.

Parallel training

Long-context training needs context parallelism: split one sequence's KV across devices and compute in turn.

It coexists with data, tensor, pipeline, and expert parallelism. The friction is that new attention mechanisms collide with systems work. The lecture gives a concrete example: trying to apply context parallelism to a 35B Qwen model, and finding no gated delta net kernel available anywhere, so it had to be written from scratch. Standard attention and sliding-window kernels are usually readily available.

KV caching is where the money is

Reusing the prefix

Step 1 computes and caches the KV for the input. Step 2 computes only the new tool result, reusing everything before it. By step 50 the new content is small relative to the full history, so the savings are large.

Published measurements show the uncached cost growing quadratically with length while the cached cost stays close to linear.

Prices tell the same story: cached tokens typically cost one tenth of uncached tokens. The lecture's rule of thumb is that a 90–95% cache hit rate is healthy. Output tokens cost roughly five times input tokens, so the spread between output and cached input can reach 50x.

Two open-source implementations

vLLM's PagedAttention divides GPU memory into fixed blocks and allocates them like OS memory pages, so many sequences can be generated in parallel and blocks can be evicted together.
SGLang's RadixAttention indexes previously computed prefixes in a radix tree and reuses any matching prefix. It was designed for chat, but matters more for agents because agent conversations are longer and more frequent.

Multi-machine serving needs cache-aware routing: send each request to the machine holding the best matching KV cache. Caches are evicted after some idle period — minutes to an hour depending on the provider — and the eviction policy directly changes the hit rate. OpenRouter even ranks providers by cache hit rate.

Hard constraints on harness design

Caching only works when the entire prefix is identical. That yields three rules:

Never change the system message every step. The classic mistake is appending the current date and time to the system prompt; every step invalidates the whole cache. Put the timestamp in the last user message or tool result instead.
Changing the tool list mid-conversation is fine; it does not invalidate the prefix already cached.
Switch models sparingly. Routing one step to a stronger model and the next to a cheaper one means the next call to the strong model still recomputes at the uncached price, which can erase the savings.

Context compaction

When the context exceeds the budget, history is summarized. The risk is losing information: at best the agent drifts, at worst it deletes files.

A good compaction policy must satisfy two conditions:

Behavior preservation: the future action sequence after compaction should match, or improve on, the one before.
Bounded state: the result must be small enough to keep working.

What to keep

Anchors: the first user messages of the session, usually protected.
Middle summary: older actions and observations compressed into structured or free text.
Recent tail: the last few steps kept verbatim.
Evidence store: the full history usually remains on disk and can be read back when needed.

When to trigger

a token threshold;
a provider's hard limit;
a manual command such as /compact.

The traps

Repeated compaction accumulates loss: the first pass is acceptable, later ones degrade further. Summary formats vary too — structured fields, free text, or an encrypted blob in some implementations.

The lecture tells an OpenHands story. They built a compaction algorithm that lost no points on SWE-bench while saving many tokens, and it looked like a success. In production it opened a new pull request every time it compacted, because the model forgot it had already done so, and it forgot user instructions given mid-conversation, because the benchmark only covered single-turn sessions with one user message.

The lesson: passing a benchmark is not the same as being usable; compaction must be validated in the wild.

Takeaways

Agent context grows quadratically; tool calls and results dominate it.
Capacity and efficiency are distinct problems, solved by architecture, positional encoding, training data, approximations, and caching.
Mainstream architectures are local-plus-global hybrids, with linear-attention variants locally and sparse or dense attention globally.
Linear attention trades expressiveness for linear complexity; DeltaNet compensates with error correction and decay.
Long-context training is limited by coherent long data more than by compute.
Prefix caching cuts cost by an order of magnitude but requires a stable prefix; never rewrite the system prompt every step.
Compaction is a fallback, and it must be tested on realistic multi-turn usage.

Check your understanding

1. Why does "supports 1M tokens" not imply "uses 1M tokens well"?

Reference answer: The claim answers only the capacity question — the window fits the tokens. Using them well is a different problem with four parts: (1) architecture, whether attention actually carries information across that distance, given the $O(n^2)$ cost and the trade-offs of sparse or linear approximations; (2) positional encoding and length extrapolation, since training only saw distances up to $L$ and relative positions break beyond that without fixes such as PI or YaRN; (3) training data, because genuinely coherent 128k documents are rare — the bottleneck is data, not compute; (4) evaluation, since passing a needle-in-a-haystack retrieval test says nothing about long-range reasoning or instruction persistence.

2. What does linear attention gain and lose by removing softmax?

Reference answer: Gain: complexity. Without softmax, associativity lets the sum over past key/values be rearranged into a state matrix $S$ that is updated incrementally in constant time per token, reducing $O(n^2)$ to $O(n)$ — effectively a recurrent network. Lose: expressiveness. Softmax provides a sharp nonlinear selection that lets a token attend to only a few relevant tokens; linear attention averages that away. DeltaNet restores part of the capacity through prediction-error correction and decay.

3. What problem does DeltaNet's decay term solve, and what does per-element decay add?

Reference answer: Decay solves permanent state contamination: without it, a single anomalous key from long ago stays in the state matrix forever. Multiplying by a decay factor $\alpha$ at each step lets old information fade, giving the state a way to forget. Per-element decay (used by GLM and Kimi) lets different dimensions forget at different rates: slow dimensions hold long-term memory while fast ones track recent detail, so one state carries information at several time scales at once.

4. Why does putting a timestamp in the system prompt badly hurt an agent's cost?

Reference answer: Prefix KV caching only hits when the prefix is byte-identical. The system prompt sits at the very front, so writing the current time into it makes every step's system message different, invalidating the whole prefix; all history is then recomputed at the uncached price. Cached tokens typically cost about a tenth of uncached ones, and a healthy system needs a 90%–95% hit rate; with output tokens costing roughly five times input, the loss compounds to more than an order of magnitude. The fix is to put the time in the last user message or a tool result.

5. Why did OpenHands' compaction algorithm pass the benchmark and fail in production? What does that imply for evaluation?

Reference answer: Their evaluation covered only single-turn sessions with one user message, so compaction kept the SWE-bench score and saved many tokens. Real use is multi-turn: after compaction the model forgot it had already opened a pull request and opened another one every time, and it also lost instructions the user added mid-session. Evaluation therefore has to mirror real interaction: multi-turn trajectories with mid-session instructions and duplicate-action checks, verifying that the subsequent action sequence stays the same before and after compaction rather than only the final answer, and confirming the compacted state is bounded and still workable.

How this connects to the rest of the course

This lecture covers context within a single session. The next one stretches the time horizon across sessions: how memory and skills keep an agent coherent without the full history. Then come planning, coding agents, and the training and evaluation loop that builds on long-context capability.

来源

视频CMU AI Agents 2026: 3. Long Context Modeling for AgentsGraham Neubig · 2026-09-08
文章CMU CS 11-768 course site