为什么需要上下文之外的记忆

上一讲处理的是「一次任务内」的上下文管理。这一讲把时间尺度拉长：如何把跨任务学到的经验存到外部，让未来的任务受益。

同一个网站上的两个任务——「把 Sony 耳机加入愿望清单」和「查无线键盘的价格区间」——共享大量动作：打开站点、找到搜索框、输入关键词、点击搜索。如果 agent 第一次摸索出了这套流程，第二次就应该直接复用，而不是重新试错。

课程把「让 agent 变好」的途径分成三类：

外部产物的两个优势值得强调：可移植（一份「如何在该站点搜索」的说明，任何模型都能用）与可检查（人可以读、改、审）。

时间轴

三种被保留的经验

1. 片段（episode）

完整的观察与动作序列，保真度最高，也最贵。和监督学习一样，保留全部细节通常没有必要——需要的是能泛化的抽象。

2. 事实（facts）

关于用户或环境的知识，例如年龄、所在地、偏好。早期记忆系统大多聚焦于此；ChatGPT 与 Claude 的记忆功能本质上就是让模型从历史对话中抽取事实，存成文本，用户可以在设置里查看和编辑。

3. 技能（skill）

从具体任务中抽象出来、可复用的「如何做某类子任务」的知识。它与记忆的区别是：

• 记忆回答「发生过什么」——上次买了什么；
• 技能回答「该怎么做」——在这个站点搜索商品的步骤。

技能是否值得保留，取决于同一子任务是否会被重复执行。

人写技能：Agent Skills 标准

当前的事实标准是一个文件夹：

python-review/
├─ SKILL.md        # 必需：YAML 元数据 + 详细说明
├─ scripts/        # 可选：可执行脚本
├─ references/     # 可选：按需加载的补充文档
└─ assets/         # 可选：模板等资源

SKILL.md 开头的 YAML 需要 name 与 description。描述决定模型什么时候会想到用这个技能，因此要短而准。

渐进披露

技能可能很多，不能全塞进上下文。加载分三层：

1. 元数据：名称与描述，常驻系统提示；
2. 正文：模型判断相关后，通过工具调用读取；
3. 资源：脚本、参考文档、模板，用到时才读。

课堂提到某个先进 agent 的系统提示里，技能描述总计约 3,000 token。这个数字说明：技能不是免费的，数量一多就会挤占上下文。

技能过多会变差

这一点有实验支持，也符合直觉：无关技能会分散注意力，拖长上下文。因此技能数量应保持克制，并尽量按任务条件加载。

触发是否准确，只能靠评估

技能是否被调用，完全取决于模型根据描述与上下文做出的判断。描述写得含糊，模型就不会用；描述写得太宽，模型会在不该用时调用。SkillsBench 是这方面较早的系统性评估：

• 从网上收集约 200 万个技能，人工编写 400 个任务；
• 由标注者为每个任务挑选可能适用的技能；
• 在 4 个 agent 框架、约 18 个模型上比较「有技能」与「无技能」。

结论是平均通过率上升，但有两个重要细节：

1. 需要技能越少的任务，收益越大；需要很多技能的任务反而可能被干扰；
2. 对某些模型与任务组合，技能仍然降低了表现——模型可能错误地使用技能。

所以写完技能必须验证它真的让输出变好，而不是假定它有用。

技能应该从经验中来

一个反直觉的结论：让模型仅凭任务描述凭空写技能，效果很差，甚至会拖累表现。更好的做法是基于过去的真实交互来写，因为执行过程会暴露你事先想不到的细节。

改进技能也有可操作的闭环。以代码审查技能为例：

1. agent 用技能产出审查意见；
2. 人逐条决定是否采纳；
3. 用模型判断「这条意见是否被采纳」，统计采纳率；
4. 汇总大量样本，让模型提出技能修改建议。

这实际上是把「人的采纳行为」当作奖励信号。

记忆系统：从 MEMGPT 到可更新的记忆

MEMGPT：把上下文当工作内存

受操作系统内存层次启发：上下文窗口是快而小的内存，外部存储是大而慢的磁盘。系统提示固定，工作上下文通过工具调用从外部记忆读写，消息另有存储。用户提到「Six Flags」时，模型可以检索过去的相关提及并载入工作上下文。

记忆需要更新与删除

只追加的记忆会过时。人换了工作、搬了家，旧事实必须被替换。MeM0 一类的做法是：

1. 从新消息中抽取候选事实；
2. 用嵌入检索找出最相似的已有记忆；
3. 让语言模型判断：新增、更新、删除（被否定时）还是忽略。

这与 DeltaNet 的机制形成有趣的类比：线性注意力只能追加状态，Delta 则能针对特定查询检索并修改已存的值；文本记忆做的是同一件事，只是载体从向量换成了句子。权重层面也有对应问题——需要「遗忘」时，可以用 unlearning 一类方法。

Reflexion：从反馈中改进

对同一个任务多次尝试时，把「哪里错了」的自然语言反馈作为下一次尝试的条件，比只重试有效得多。例如模型只答了一场战役而问题问的是一系列战役，反馈指出这一点后，第二次就能答对。只要反馈足够通用，它也能迁移到相似任务。

轨迹检索：agent 版的 RAG

把过去成功的完整轨迹（消息、工具调用、观察、思维链）存入经验池。新任务到来时，用任务描述做嵌入检索，取回相似任务的轨迹放进上下文。实现细节各异，但核心都是「用相似经验作为示例」。

技能归纳：从轨迹中提炼可复用部分

什么是一段技能

在网页任务里，一段搜索流程可能是：点击 ID 为某值的搜索框 → 输入关键词 → 点击搜索按钮。问题是：直接存这段动作序列好不好？

不好。元素 ID 会变，换一个网站界面结构就完全不同。所以归纳出的技能必须抽象出可变部分，例如把具体搜索词变成参数模板。

从文本工作流开始

Agent Workflow Memory 的做法：

1. 让语言模型查看一段工具调用序列；
2. 标出可能在未来复用的片段；
3. 生成「工作流」：子任务描述 + 动作序列 + 参数模板；
4. 用判断模型评估原轨迹是否成功，只有成功的才写入记忆。

检索到的工作流会直接放进上下文。在涉及地图工具的基准上，没有记忆的基线在约 10% 准确率附近徘徊，加入工作流记忆后明显提升，并能完成「找附近酒店并给出步行路线」这类更复杂的任务。

代码技能：更rigid，但更强

同一段搜索流程也可以写成函数：

• 文本技能：模型看到说明与示例动作，但仍需自己生成每一步工具调用。优点是灵活——网站变了可以自行调整；缺点是每次都要重做低层动作。
• 代码技能：模型直接以参数调用函数。优点是可通过执行测试、可嵌套、可重构、可分层；缺点是刚性——函数内部步骤固定，界面一变就失效。

Agent Skill Induction 把轨迹转成带 docstring 的函数，并生成一段「用这些函数重写原轨迹」的代码；直接执行生成的代码并对结果做判断，比判断原轨迹更可靠，因为代码真的跑过了。

SkillWeaver 进一步用失败来修正：执行技能、用奖励模型评估，失败则根据反馈修改。例如一个 identify_pill 技能接受颜色参数却从未使用，linter 能直接指出，模型据此修订。

效率收益

代码技能的另一个优势是减少环境交互次数。一个「更新两个地址」的任务，基线 agent 要反复导航、填写表单、观察页面；有了 navigate_to_address_settings 与 update_address_details 两个函数后，三步就能完成。对 GUI agent 尤其重要，因为截图与思维链都很贵。

在重复结构较多的任务上，代码技能在准确率与交互次数上都优于文本技能。

泛化失败与抽象类

在一家网站上归纳的代码技能，换到 Target 就失效：原站点是下拉框，Target 是单选按钮。

解决办法借用软件工程：抽象类 + 站点特定实现（PolySkill）。第一次访问新站点时，模型写一个只有方法签名的抽象类（搜索商品、加入购物车、结账），再写该站点的具体实现。换站点时只需换实现。这样既能积累大量具体技能，又只激活当前相关的那一小部分。

从失败中学习

只从成功中学习会浪费大量信号。课堂给了一个例子：某站点的搜索把多个词当作「或」，搜「Bluetooth headphones」返回五千条结果，agent 翻页翻到迷失，任务失败。

把失败轨迹交给模型，让它写出通用建议——「使用更具体的搜索词」「一次显示更多条目」——比直接记住失败轨迹更有用。ReasoningBench 的对照实验显示：加入负例对「原始轨迹」或「工作流」没有帮助，但对「文本策略描述」有显著提升。原因很直接：你可以用一句话描述如何避免失败，但很难从一段失败轨迹中读出通用规律。

记忆的管理

判断质量决定一切

几乎所有方法都用判断模型决定「这条经验值不值得存」。判断器不完美，它的准确率会直接影响系统表现。ReasoningBench 用带噪声的模拟判断器做了控制实验：性能随判断准确率下降，但存在一个较宽的区间仍能带来收益。

检索越多不一定越好

增加检索到的经验数量，性能反而下降。可能有两个原因：

1. 记忆里本来就没有相关内容，多取只是噪声；
2. 模型被无关上下文干扰。

后者可以通过训练改善，也可以直接修剪记忆：统计每个条目的使用频率，长期不用的删掉。这既省上下文，也可能提升成功率。

用强化学习训练「归纳技能」这件事

前面所有流程都是提示词拼出来的启发式。一个自然的问题是：能不能直接训练模型学会归纳技能？

可以。做法是在一串相似任务的在线序列上做强化学习：模型归纳技能、写入记忆、用于后续任务，奖励来自后续任务是否成功，以及是否复用了先前归纳的技能。过程不可微、组件很多，但强化学习不需要可微。近期已有论文验证有效，并发现「复用奖励」能进一步提升表现。

本讲结论

• 跨任务经验应存到外部，因为外部产物可移植、可检查。
• 经验有三种粒度：完整片段最保真也最贵，事实最轻，技能最可复用。
• 人写技能依赖标准格式与渐进披露；技能不是越多越好，无关技能会伤害表现。
• 技能应从真实交互中归纳，凭空生成往往有害。
• 记忆必须支持更新与删除，否则会持续给出过时信息。
• 文本技能灵活，代码技能高效且可测试，但更刚性；抽象类可以缓解泛化问题。
• 失败轨迹应转成通用建议，而不是原样记住。
• 记忆规模、检索质量与判断器质量共同决定最终收益。

理解检验

1. 外部技能相比微调权重，最大的两个优势是什么？

参考答案：可移植——一份「如何在该站点搜索」的说明不绑定模型，换一个模型、换一套脚手架仍然能用；权重只对训练过的那个模型有效。可检查——技能是文本或代码，人可以读、改、审、做版本管理与回滚，出问题能定位到具体一条说明；而权重里的能力既不可读也难以局部修正。附带的好处是迭代快、成本低；代价是它不会被内化，每次都要通过上下文加载。

2. 渐进披露的三层分别是什么？为什么第一层必须很短？

参考答案：第一层是元数据（name 与 description），常驻系统提示；第二层是 SKILL.md 正文，模型判断相关后通过工具调用读取；第三层是资源（脚本、参考文档、模板），真正用到时才加载。第一层必须短，因为它常驻在每一次请求里：课堂提到某个先进 agent 的技能描述总计约 3,000 token，技能一多就会挤占上下文并分散注意力。描述还要短而准，因为它决定模型「什么时候想到用这个技能」。

3. 为什么让模型「只根据任务描述」写技能效果差？

参考答案：任务描述里没有执行过程才会暴露的细节——真实的元素定位方式、页面结构、失败点、参数取值范围与约束。只凭描述写出的技能是对环境的猜测，可能与实际不符，甚至给出错误指导，所以效果差、还会拖累表现。正确做法是基于过去的真实交互归纳技能，因为执行过程会暴露事先想不到的细节；再用人对输出的采纳率作为信号迭代改进。

4. 文本技能与代码技能各自的失效方式是什么？抽象类如何缓解代码技能的失效？

参考答案：文本技能的失效方式是低效而非崩溃——它只给说明与示例，模型仍要自己生成每一步工具调用，网站改版时可以临场调整，但每次都要重做低层动作。代码技能的失效方式是刚性——函数内部步骤固定，界面结构一变就断，例如原站点是下拉框、Target 是单选按钮，函数直接失效。抽象类（PolySkill）把接口（搜索商品、加入购物车、结账的方法签名）与站点特定实现分开：第一次访问新站点时写该站点的实现，换站点只换实现，抽象接口与调用方不变。这样既积累大量具体技能，又只激活当前相关的那一小部分。

5. 为什么从失败轨迹中提炼文本建议，比直接检索失败轨迹更有效？

参考答案：失败轨迹里，通用规律被大量具体动作与噪声掩盖，模型很难从中读出「下次该怎么做」；而一句话建议（「使用更具体的搜索词」「一次显示更多条目」）直接给出可迁移的策略，占用上下文也小。ReasoningBench 的对照实验正是这个结果：加入负例对「原始轨迹」或「工作流」没有帮助，但对「文本策略描述」有显著提升。

与后续课程的关系

这一讲把记忆与技能放进了 agent 的能力栈。下一讲处理规划、任务分解与多智能体协作——当技能库存在时，agent 如何决定用哪些、按什么顺序用。再往后是编码 agent、计算机使用 agent，以及把这些经验用于训练与评估。

Why memory must leave the context

The previous lecture managed context within one task. This one stretches the horizon: how does an agent store what it learns across tasks so future tasks benefit?

Two tasks on the same shopping site — "add these Sony headphones to my wish list" and "find the price range of wireless keyboards" — share most of their actions: open the store, find the search box, type a query, click search. Once the agent has worked out that flow, it should reuse it rather than rediscover it.

The lecture separates three routes to improvement:

Two properties of external artifacts matter most: they are portable (a note on "how to search this site" works with any model) and inspectable (a person can read, edit, and audit them).

Timeline

Three kinds of retained experience

1. Episodes

The full sequence of observations and actions. Highest fidelity, highest cost. As in supervised learning, retaining every detail is usually unnecessary; what generalizes is the abstraction.

2. Facts

Knowledge about the user or environment: age, location, preferences. Early memory systems focused almost entirely on facts. ChatGPT and Claude memory work this way: a model extracts facts from past conversations into text that the user can inspect and edit.

3. Skills

Knowledge about how to perform a reusable subtask, abstracted away from the specifics. The distinction from memory is:

• Memory answers "what happened" — what was purchased last time.
• Skills answer "how to do it" — the steps for searching this site.

A skill is worth keeping only if the same subtask will recur.

Human-authored skills: the Agent Skills standard

The current de facto standard is a folder:

python-review/
├─ SKILL.md        # required: YAML metadata + detailed instructions
├─ scripts/        # optional: executable scripts
├─ references/     # optional: documentation loaded on demand
└─ assets/         # optional: templates and other resources

The YAML header needs a name and a description. The description determines when the model thinks to use the skill, so it must be short and precise.

Progressive disclosure

A library of skills cannot all live in the context. Loading happens in three layers:

1. Metadata: name and description, always present in the system prompt.
2. Body: read through a tool call once the model judges the skill relevant.
3. Resources: scripts, references, and assets, loaded only when used.

The lecture notes that one state-of-the-art agent carries about 3,000 tokens of skill descriptions in its system prompt. Skills are not free; enough of them crowd out working context.

Too many skills make things worse

This is both intuitive and experimentally supported: irrelevant skills distract the model and lengthen the context. Keep the library small and load by task.

Trigger accuracy requires evaluation

Whether a skill is invoked depends entirely on the model's judgment from the description and context. A vague description means the skill is never used; an over-broad one means it is used when it should not be. SkillsBench is an early systematic evaluation:

• roughly two million skills collected online, with 400 human-written tasks;
• annotators pick skills that might apply to each task;
• four agent frameworks and about eighteen models, with and without skills.

Average pass rates rise, but with two important caveats:

1. Tasks that need fewer skills benefit more; tasks requiring many skills may be hurt by distraction.
2. For some model–task combinations, skills still lower performance because the model applies them incorrectly.

Writing a skill is therefore not enough; you must verify that it improves output.

Skills should come from experience

A counterintuitive result: asking a model to write a skill from a task description alone works badly and can hurt performance. Skills should be based on real prior interactions, because execution reveals details you would not have predicted.

Improving a skill can be a closed loop. For a code-review skill:

1. the agent produces review comments using the skill;
2. a person decides which comments to address;
3. a model judges how many were adopted;
4. across many pull requests, a reasoning model summarizes what worked and proposes edits to the skill.

In effect, human adoption is the reward signal.

Memory systems: from MEMGPT to editable memory

MEMGPT: the context as working memory

Inspired by OS memory hierarchies: the context window is fast, small memory; external storage is large and slow. The system prompt is fixed, a working context is filled from external memory via tool calls, and messages have their own store. When the user mentions "Six Flags", the model retrieves past mentions into the working context.

Memory must be updated and deleted

Append-only memory goes stale. If someone changes jobs, the old fact must be replaced. MeM0-style systems:

1. extract candidate facts from each new message;
2. retrieve the most similar existing memories by embedding;
3. ask a language model whether to add, update, delete (when negated), or ignore.

The analogy to DeltaNet is useful: linear attention can only append to its state, while Delta retrieves and modifies values associated with particular queries. Text memory does the same thing with sentences instead of vectors. At the weight level, the corresponding problem is forgetting, addressed with methods such as unlearning.

Reflexion: learning from feedback

When a task allows multiple attempts, conditioning the retry on natural-language feedback about what went wrong is far more effective than blind retrying. If the model answered with one battle when the question asked for a series, feedback pointing that out lets the second attempt succeed. Feedback general enough also transfers to similar tasks.

Trajectory retrieval: RAG for agents

Store successful trajectories — messages, tool calls, observations, and reasoning — in a pool. When a new task arrives, embed its description, retrieve similar trajectories, and place them in the context. Implementations differ, but the core is "use similar experience as examples".

Skill induction: extracting reusable pieces

What counts as a skill

In a web task, a search flow might be: click the search box with a given element ID, type a query, click the search button. Storing that action sequence verbatim is a bad idea: element IDs change and another site's interface is structured differently. An induced skill must abstract the variable parts, for example turning the specific query into a parameter template.

Starting with textual workflows

Agent Workflow Memory:

1. a language model inspects a tool-call sequence;
2. it marks spans likely to recur;
3. it produces a workflow: subtask description, action sequence, parameter template;
4. a judge model decides whether the original trajectory succeeded, and only successful ones are stored.

Retrieved workflows are inserted into the context. On a benchmark involving map tools, a baseline without memory hovers around 10% accuracy, while the workflow memory improves substantially and enables harder tasks such as finding a nearby hotel and showing a walking route.

Code skills: more rigid, more powerful

The same search flow can be a function:

• Text skill: the model sees instructions and example actions but still emits every low-level call. Flexible — it can adapt when the site changes — but repetitive.
• Code skill: the model calls the function with arguments. Testable, composable, refactorable, hierarchical — but rigid, since the steps inside are fixed and break when the interface changes.

Agent Skill Induction turns trajectories into functions with docstrings, plus a rewritten version of the trajectory that calls them. Executing the generated code and judging the result is a stronger filter than judging the original trajectory, because the code actually runs.

SkillWeaver adds failure-driven revision: execute the skill, score it with a reward model, and revise on failure. A real example: an identify_pill function accepts a color argument but never uses it; a linter points this out, and the model fixes it.

The efficiency payoff

Code skills also cut environment interactions. For updating two addresses, a baseline agent navigates forms, types fields, and re-observes the page repeatedly. With navigate_to_address_settings and update_address_details, three steps suffice. This matters especially for GUI agents, where screenshots and reasoning traces are expensive.

On tasks with repetitive structure, code skills beat text skills on both accuracy and interaction count.

Generalization failure and abstract classes

A code skill induced on one site fails on Target: the original site used a dropdown, Target uses radio buttons.

The fix borrows from software engineering: an abstract class plus site-specific implementations (PolySkill). On first visiting a new site, the model writes an abstract class with method stubs — search product, add to cart, checkout — and then a concrete implementation for that site. Switching sites only swaps the implementation. This allows a large library of specific skills while activating only the relevant subset.

Learning from failure

Learning only from success throws away signal. The lecture gives an example: a site treats multiple search terms as OR, so "Bluetooth headphones" returns five thousand results, the agent paginates endlessly and loses the task.

Handing the failed trajectory to a model to write general advice — "use a more specific query", "show more items at once" — works better than storing the failure. ReasoningBench's comparison found that negative examples do not help when attached to raw trajectories or workflows, but help substantially when expressed as textual strategies. The reason is simple: a sentence can describe how to avoid failure, but a failed trajectory rarely reveals a general rule.

Managing the memory

Judge quality matters

Almost every method uses a judge to decide whether an experience is worth storing. Judges are imperfect, and their accuracy propagates into system performance. ReasoningBench ran controlled experiments with simulated judge accuracy: performance degrades as accuracy falls, but a fairly wide range still yields gains.

Retrieving more is not better

Increasing the number of retrieved experiences lowers performance. Two explanations:

1. the memory contains nothing relevant, so extra items are noise;
2. the model is distracted by irrelevant context.

The second can be addressed by training, or by pruning memory: track how often each item is used and drop the ones that never are. This saves context and may improve success.

Training the induction process itself

All of the above is prompt-engineered heuristics. A natural question: can a model be trained to induce skills?

Yes. Run reinforcement learning over online sequences of similar tasks: the model induces skills, writes memory, and uses it on later tasks, with reward from later task success and from reuse of previously induced skills. The pipeline is complex and non-differentiable, which RL does not mind. Recent papers report gains, and one finds that a reuse bonus improves results further.

Takeaways

• Cross-task experience belongs in external stores because artifacts are portable and inspectable.
• Experience has three granularities: episodes (faithful, expensive), facts (light), and skills (reusable).
• Human-authored skills depend on a standard format and progressive disclosure; more skills is not better.
• Skills should be induced from real interactions; generating them from a task description alone often hurts.
• Memory must support updates and deletion or it will keep serving stale facts.
• Text skills are flexible; code skills are efficient and testable but rigid, and abstract classes mitigate the rigidity.
• Failed trajectories should become general advice, not stored verbatim.
• Memory size, retrieval quality, and judge quality jointly determine the payoff.

Check your understanding

1. What are the two biggest advantages of external skills over fine-tuning weights?

Reference answer: Portability — a note on "how to search on this site" is not tied to a model and still works after switching models or harnesses, whereas weights only help the model that was trained. Inspectability — a skill is text or code that people can read, edit, review, version and roll back, and a failure can be traced to a specific instruction; capabilities baked into weights are neither readable nor locally correctable. The incidental benefits are fast iteration and low cost; the price is that skills are never internalized and must be loaded through context every time.

2. What are the three layers of progressive disclosure, and why must the first be short?

Reference answer: Layer one is the metadata (name and description), resident in the system prompt; layer two is the SKILL.md body, read by tool call once the model judges it relevant; layer three is resources (scripts, reference docs, templates), loaded only when used. The first layer must be short because it travels in every request: the lecture notes that one advanced agent carries about 3,000 tokens of skill descriptions, so a large library crowds the context and dilutes attention. The description must also be short and precise, since it determines when the model thinks to use the skill.

3. Why does a skill written only from a task description perform poorly?

Reference answer: A description lacks the details that only execution exposes — how elements are actually located, the page structure, where it fails, and the valid ranges and constraints on parameters. A skill written from description alone is a guess about the environment; it may not match reality and can even mislead, which is why it underperforms and sometimes hurts. The right approach is to induce skills from real past interactions, because execution reveals details nobody anticipated, then refine them using whether humans accept the output as the signal.

4. How do text and code skills fail differently, and how do abstract classes help?

Reference answer: A text skill fails by inefficiency rather than breaking — it supplies instructions and example actions, but the model still generates every low-level tool call itself, so it can adapt when a site changes yet repeats all the work every time. A code skill fails by rigidity — the steps inside the function are fixed, so a structural change breaks it, for example a dropdown on one site versus radio buttons on Target. Abstract classes (PolySkill) separate the interface (method signatures such as search product, add to cart, checkout) from site-specific implementations: the first visit to a new site produces an implementation, and switching sites only swaps the implementation while the interface and its callers stay the same. Concrete skills still accumulate, but only the relevant subset is activated.

5. Why does turning failed trajectories into textual advice beat retrieving the failures directly?

Reference answer: In a failed trajectory the general lesson is buried under concrete actions and noise, so the model struggles to extract "what to do differently"; a one-line piece of advice ("use more specific search terms", "show more results per page") states a transferable strategy directly and costs little context. ReasoningBench shows exactly this: adding negative examples does not help with raw trajectories or workflows, but significantly improves text policy descriptions.

How this connects to the rest of the course

This lecture places memory and skills in the capability stack. The next one covers planning, task decomposition, and multi-agent coordination — how an agent decides which skills to use and in what order. Then come coding and computer-use agents, and the training and evaluation loop that turns experience into capability.