第 14 章
从 OpenHands 学到的：单 agent、技能、界面与自动化
一个真实编码 agent 产品的工程复盘：SDK 设计、压缩与沙箱、单 agent 对多 agent、技能与 MCP、本地与云端、四种交互界面，以及把重复流程变成可训练系统。
Lessons from OpenHands: Single Agents, Skills, Interfaces, and Automation
An engineering retrospective on a real coding-agent product: SDK design, compaction and sandboxing, single versus multi-agent, skills and MCP, local versus cloud, four interfaces, and turning repeated workflows into a trainable system.
篇内目录
- Why this lecture is different
- Timeline
- A short history
- The SDK: good defaults, removable
- Tools and persistence
- Compaction
- Sandboxing and safety
- Choosing a framework for research
- Single agent or multi-agent
- How a single agent keeps its capability
- When multi-agent still helps
- Where the agent runs
- Four interfaces
- An experiment about people
- Automation: turning repeated workflows into a trainable system
- Skill Refiner
- Takeaways
- Check your understanding
- Connections
这一讲为什么特殊
前面讲的是算法、系统与安全，这一讲讲的是把 agent 做成产品：用户真的在用，踩过的坑都变成了框架里的设计。讲者把 OpenHands 的演进拆成两部分——技术选择，以及从真实使用中提炼出的研究问题。
OpenHands 有两种使用方式：
- 人在环中：在聊天窗口里描述今天要做的任务；
- 自动化：把重复流程配成定时或事件触发的作业，例如 PR 评审、issue 分诊、Slack 频道监控、依赖升级失败后的修复。
它与 Claude Code、Codex 最大的区别是模型无关：可以随时在 DeepSeek、GPT、Claude 之间切换，而前两者更贴近各自的模型生态。
时间轴
| 时间 | 内容 |
|---|---|
| 0:00 | 开场：这一讲讲产品，而不是算法 |
| 4:12 | OpenHands 演示与近一年的变化 |
| 8:30 | 生态：其他 agent CLI 与开放协议 |
| 12:45 | 新需求：更长任务、并行、可视化、可重复流程 |
| 16:54 | 语言与生态选择：TypeScript 还是 Python |
| 19:52 | 上下文压缩：condenser 与保留原始指令 |
| 20:58 | 隔离与沙箱：把 agent 关进可控环境 |
| 22:10 | 确认策略：哪些动作必须人工批准 |
| 25:22 | 企业场景：大公司怎么用 coding agent |
| 29:22 | 模型兼容性与基准分数 |
| 33:52 | SDK 从「一个大提示词」转向 skills |
| 36:46 | 多 agent 何时有用：并行任务与独立审查 |
| 38:16 | 自我审查与独立审查的差别 |
| 42:17 | 需求演化时如何维护系统 |
| 46:35 | 四种交互界面 |
| 51:04 | 自动化：PR 审查、issue 分诊、版本升级 |
| 55:14 | 分诊清单：每一项要检查什么 |
| 59:30 | 用提示词迭代自动化流程 |
| 1:03:42 | 权限与 SSH 访问的边界 |
| 1:08:15 | git 作为 agent 原生接口 |
一段简短的历史
- 2016 年起做代码生成，但直到 2023 年基本都是补全或单函数生成，不是 agent。
- 2023 年 SWE-bench 出现，早期最好成绩只有约 5%。
- 2024 年 3 月，Devin 的演示视频获得巨大关注，但它不开源。
- 第二天，Qwen 团队发起复现号召，仓库一天内拿到一万星；讲者用三个晚上做出了可运行的原型，这就是 OpenDevin，后来改名 OpenHands。
- 2024 年大部分时间，agent 其实「不好用」；到 11 月前后，Claude Sonnet 3.5 / 3.7 才让编码 agent 真正开始工作。
- 2025 年是编码能力快速提升的一年；2026 年更多是部署与界面的一年：更长的运行、并行任务、可视化、可重复工作流、凭据与验证。
这段历史的用处不是怀旧，而是说明：很多今天看起来理所当然的设计，是被失败反复打磨出来的。
SDK：好默认值，但可拆掉
OpenHands 的 harness 现在已经是第二版。第一版据讲者说「很糟」，难以使用也难以扩展，八个月前被整体重写。
设计哲学与 Pi 相反：Pi 给一个极简内核，其余靠扩展；OpenHands 给开箱即用的好默认值，同时允许你把任何一部分拆掉替换。
一个在 SWE-bench 上表现不错的 agent，核心配置大约十行：选模型、给 API key、给工具（终端、文件编辑、测试追踪）、指定工作区，然后进入「任务 → 动作 → 观察」的循环。
工具与持久化
自定义工具的实现方式是 Action + Observation + Executor：Executor 接收 Action，返回 Observation。这是一个足够通用的接口，可以包住任意函数。
会话可以持久化：指定目录与 conversation ID，事件写入会话状态，之后用同一个 ID 恢复。
压缩
OpenHands 声称是最早实现上下文压缩的框架之一，当时叫 condensation，后来 Anthropic 把它命名为 compaction，业界就都叫 compaction 了。
做法是把较早的事件总结成摘要，同时保留最初的指令。他们也试过直接 mask 掉部分动作，但摘要效果更好。触发条件有两种：每 N 轮，或每 N 个 token（通常取上下文窗口的四分之三作为阈值）。
沙箱与安全检查
两种隔离方式：把 agent 跑在沙箱里，或者用 with 语句把块内执行的命令都路由到沙箱。讲者坦言不确定哪种更好——前者工具生态更成熟，后者编码更方便。
安全侧提供两种机制：
- 确认策略：规定哪些动作可以自主执行，哪些必须询问；
- 安全分析器：逐步判断当前动作是否安全。
研究用途的选择
- 需要长轨迹、内置可观测性、完整功能：用 OpenHands；
- 需要快速改代码、理解全部逻辑：用 mini-swe-agent，它只有 bash，极简到可以一下午读完。
单 agent 还是多 agent
2024 年初流行给 agent 分配专门角色：管理者、复现者、故障定位者、编辑者、验证者，再按固定流程串起来。讲者在 2024 年底写了《Don't sleep on single agent systems》，核心论点是：
模型在变强。与其把流程硬编码成多个角色，不如写一个「先做这个、再做那个」的 prompt，让更强的指令遵循能力去执行。
他承认这是一个赌注，但从结果看基本正确：编码场景里，固定的多 agent 工作流现在并不常见。
单 agent 如何保持能力
- 工具集保持简单：编辑文件、运行命令、浏览网页，再加 MCP 连接外部服务；
- 把长 prompt 拆成技能：当 prompt 里塞满「推 GitHub 时这样做、用 Docker 时那样做」，就把它拆成按需加载的技能。OpenHands 早期叫 microagents，Anthropic 后来叫 skills——讲者认为 skills 这个名字好得多。
- 技能 + MCP 是多 agent 的替代品：多 agent 给每个角色一套 prompt 和工具；技能与 MCP 则在需要时即时拉入对应上下文。单 agent 因此也能覆盖全部能力。
他们还发表过一篇工作：给模型代码能力与多模态浏览能力，它就能解决各类任务，并在多个基准上超过专用 agent。这支持了「通用单 agent + 通用工具」的路线。
多 agent 什么时候仍然有用
三个理由：
- 效率：十个互不依赖的 PR 可以并行分派；
- 独立评审：让另一个 agent 在看不到决策理由的情况下审查，会比自审更严格——自审时 agent 掌握全部辩解理由，往往对自己更宽容；
- 成本控制：子 agent 汇报时丢弃大部分上下文，只保留结论；也可以让便宜模型做实现、贵模型做编排。
但多 agent 的上下文管理非常难。他们做过管理者 + 多个工程师的异步软件工程实验（PaperBench、Commit0 一类长任务），结论是：工程师要回传恰到好处的上下文、彼此不能冲突、管理者还要完成集成，任何一环出错都会拖垮整体。深度研究是更容易受益的场景，因为可以按文档来源并行分派。
agent 跑在哪里
- Devin 最初是云端 SaaS；
- OpenDevin 一开始跑在本地 Docker 里，主要是出于安全考虑；
- 现在 Claude Code、Codex、OpenHands 都可以跑在本地。
讲者原本认为本地不可接受——万一被提示注入，数据与 API key 都可能泄露。但事实是：人在环中的编码工作，用户明显更喜欢本地。云端则在非编码与后台任务（例如代码评审）中更常见。
四种交互界面
| 界面 | 优势 | 局限 |
|---|---|---|
| IDE | 熟悉，文件、diff、测试、调试器齐全 | 如果你不再读代码，这套界面就偏传统 |
| GUI 桌面应用 | 为 agent 重新设计，适合非程序员 | 编程人群使用比例仍低 |
| CLI | 任何机器都能用，服务器上也能用 | 交互表达力有限 |
| 集成（Slack / GitHub） | 留在你原本工作的地方 | 复杂操作受限 |
课堂投票里 CLI 最高，IDE 第二。讲者的判断是集成会越来越普遍：不需要切换到专用界面，agent 就活在你已经在用的工具里。
一个关于人的实验
他们比较了 GitHub Copilot 用户与 OpenHands agent 用户完成同一批任务的表现：
- agent 用户花费约一半时间，完成约两倍任务；
- 负面反馈集中在两点：「看着 agent 干活很无聊」和「很难确认它到底做得好不好」。
这两个反馈定义了人机界面的两个核心问题：如何让人有效并行，以及如何让人低成本验证。
自动化：把重复流程变成可训练系统
一个自动化的结构是：触发器（issue、PR 或定时）→ 收集上下文 → 在隔离工作区运行 → 产出 artifact → 由人或策略检查。
OpenHands 主要做两件事：
- issue 分诊：判断 issue 是否清晰，不清晰就指出需要补充什么；清晰则列出验收标准。
- PR 评审：按团队写好的评审规则检查——是否在范围内、是否满足清单。
这两件事有一个共同点：同样的流程会重复无数次，于是它变成一个机器学习问题。输入输出都有，训练数据自然积累。
Skill Refiner
他们最近的论文把这个循环做成了自动改进：
- 收集成功与失败的轨迹，失败信号来自「bot 说 LGTM，但维护者说不行」这类延迟反馈；
- 对成功与失败分别做嵌入与聚类；
- 生成指令修改提案；
- 用证据门控验证提案是否真的能改善下一次评审；
- 合并成新的技能。
在多个模型上，它比 GEPA、Trace2Scale 等基线更便宜也更好。
更进一步的方向是从大量真实轨迹中挖掘工作流，把用户反复出现的用法聚类成自动化，再交给上面的改进循环。讲者也指出，目前改的是 prompt；改模型是另一条路，但那需要更严格的评测，而 issue 分诊、代码评审这类「欠定义任务」的评测本身就非常难做。
本讲结论
- 产品级 agent 的竞争力来自默认值、可替换性与真实工作流的贴合。
- 单 agent + 简单工具 + 技能与 MCP，已经能覆盖过去需要固定多 agent 流程的场景。
- 多 agent 的价值集中在并行、独立评审与成本分层，而不是「角色扮演」。
- 本地与云端不是优劣，而是任务类型不同。
- 界面设计要解决「并行」与「验证」，而不只是「能对话」。
- 重复流程一旦稳定，就应该被当作数据循环来优化。
理解检验
1. 为什么「更强模型 + 单 agent」会削弱固定多 agent 工作流的价值？
参考答案：固定的多 agent 工作流把流程硬编码成多个角色（管理者、复现者、故障定位者、编辑者、验证者）和固定的串联顺序，本质上是把「先做这个、再做那个」写进了编排。模型变强后，指令遵循能力足以让一个 agent 按同样的步骤推进，而且更灵活；反过来，硬编码的角色会变成约束，角色之间还要回传恰到好处的上下文，任何一环出错都会拖垮整体。讲者在 2024 年底的判断就是：与其把流程写死成多个角色，不如写一条「先做这个、再做那个」的 prompt，让更强的模型去执行。结果也印证了这一点——编码场景里固定的多 agent 工作流现在并不常见。
2. 技能与 MCP 如何替代多 agent 中的角色分工？
参考答案：多 agent 给每个角色一套固定的 prompt 和工具；技能与 MCP 则在需要时即时拉入对应的上下文与外部服务。于是单 agent 也能覆盖全部能力：工具集保持简单（编辑文件、运行命令、浏览网页），再用 MCP 连接外部服务；当 prompt 里塞满「推 GitHub 时这样做、用 Docker 时那样做」时，就把它拆成按需加载的技能（OpenHands 早期叫 microagents，后来业界叫 skills）。角色分工被「按需加载的上下文」取代，避免了多 agent 的上下文回传难题。
3. 自审与独立评审为什么会有差别？
参考答案：自审时 agent 掌握自己决策的全部辩解理由，往往会对自己更宽容，倾向于确认自己的做法；独立评审让另一个 agent 在看不到决策理由的情况下审查，只能依据结果与规则判断，因此更严格。这也是课程列出的多 agent 仍然有价值的三个理由之一（另外两个是并行分派与成本分层）。
4. 为什么 issue 分诊与 PR 评审特别适合做自动化与持续改进？
参考答案：因为同样的流程会重复无数次——判断 issue 是否清晰、按团队评审规则检查 PR，这两件事的输入输出都有，训练数据会自然积累，于是它从一个产品功能变成机器学习问题。反馈也拿得到，虽然常常是延迟的（「bot 说 LGTM，但维护者说不行」）。正因如此才能跑 Skill Refiner 的循环：收集成功与失败轨迹、分别做嵌入与聚类、生成指令修改提案、用证据门控验证提案是否真的改善下一次评审、再合并成新技能。讲者也指出，这类「欠定义任务」的评测本身很难做，所以目前改的是 prompt 而不是模型。
5. 「无聊」与「难以验证」分别对应人机界面的什么设计问题？
参考答案：这两个反馈来自同一批任务的对比实验：agent 用户花费约一半时间、完成约两倍任务，但负面反馈集中在两点。其一「看着 agent 干活很无聊」对应的是如何让人有效并行——界面不应要求人盯着一条轨迹，而应支持同时推进多个任务、在需要决策的时刻介入。其二「很难确认它到底做得好不好」对应的是如何让人低成本验证——需要把 diff、测试结果、证据与结论组织成一眼可查的形式，而不是让人重新读一遍过程。课程由此概括出界面设计的两个核心问题，而不是「能不能对话」。
与前序课程的关系
这一讲把工具使用、上下文管理、规划、安全与训练串成了一个产品视角。它呼应了第 1 讲的判断：agent 是系统，可靠性来自 harness、评估与真实使用中的迭代，而不是单一模型能力。
Why this lecture is different
Earlier lectures covered algorithms, systems, and safety. This one covers building an agent into a product: real users, real failures, and the design decisions those failures produced. The instructor splits the story into technical choices and the research questions that emerged from actual use.
OpenHands has two modes:
- Human in the loop: describe the day's task in a chat window.
- Automations: configure repeated work to run on a schedule or event — PR review, issue triage, Slack monitoring, fixing failed dependency bumps.
Its main difference from Claude Code and Codex is being model-agnostic: switch between DeepSeek, GPT, and Claude at will, whereas the others sit closer to one model ecosystem.
Timeline
| Time | Topic |
|---|---|
| 0:00 | Opening: this lecture is about the product, not the algorithm |
| 4:12 | OpenHands demo and what changed over the past year |
| 8:30 | The ecosystem: other agent CLIs and open protocols |
| 12:45 | New demands: longer runs, parallelism, visualization, repeatable workflows |
| 16:54 | Language and ecosystem choice: TypeScript versus Python |
| 19:52 | Context compaction: condensers that keep the original instructions |
| 20:58 | Isolation and sandboxing agents |
| 22:10 | Confirmation policy: which actions need human approval |
| 25:22 | Enterprise use: how large companies run coding agents |
| 29:22 | Model compatibility and benchmark scores |
| 33:52 | The SDK moves from one large prompt to skills |
| 36:46 | When multi-agent helps: parallel tasks and independent review |
| 38:16 | Self-review versus an independent reviewer |
| 42:17 | Maintaining the system as requirements evolve |
| 46:35 | Four interaction surfaces |
| 51:04 | Automation: PR review, issue triage, version bumps |
| 55:14 | The triage checklist and what each item verifies |
| 59:30 | Iterating on the automation through prompts |
| 1:03:42 | Bounds on permissions and SSH access |
| 1:08:15 | Git as an agent-native interface |
A short history
- Code generation work started in 2016, but until 2023 it was mostly completion or single-function generation, not agents.
- SWE-bench appeared in 2023; early best results were around 5%.
- In March 2024, Devin's demo drew enormous attention but was not open source.
- The next day, the Qwen team called for a reproduction. The repository reached ten thousand stars in a day. The instructor built a working prototype over three evenings; that became OpenDevin, later OpenHands.
- For most of 2024 agents did not actually work well. Around November, Claude Sonnet 3.5 and then 3.7 made coding agents genuinely usable.
- 2025 was the year coding capability improved rapidly; 2026 is more about deployment and interfaces: longer runs, parallel tasks, visualization, repeatable workflows, credentials, verification.
The point of the history is not nostalgia. It shows that many designs that look obvious today were beaten into shape by failure.
The SDK: good defaults, removable
The current harness is the second version. The first was, in the instructor's words, bad — hard to use and hard to extend — and was rewritten wholesale about eight months ago.
The philosophy is the opposite of Pi's. Pi gives a minimal core and expects you to extend it; OpenHands gives defaults that work out of the box while letting you tear out and replace any piece.
A configuration that performs well on SWE-bench is roughly ten lines: choose a model, supply an API key, provide tools (terminal, file editor, test tracker), specify a workspace, and run the task → action → observation loop.
Tools and persistence
Custom tools are built from an Action, an Observation, and an Executor: the executor takes an action and returns an observation. That interface is general enough to wrap arbitrary functions.
Conversations persist by directory and conversation ID: events are written to conversation state and restored by reusing the ID.
Compaction
OpenHands claims to be among the first to implement context compaction, then called condensation. Anthropic later named it compaction, and the field followed. They still call theirs condensation.
The method summarizes older events into a summary while keeping the original instructions. They tried masking individual actions instead, but summarization worked better. Triggers are either every N turns or every N tokens, typically three quarters of the context window.
Sandboxing and safety
Two isolation styles: run the agent inside a sandbox, or use a with block that routes every command executed inside it to the sandbox. The instructor is candid that neither is clearly better: the first has a more mature tooling ecosystem, the second is easier to code.
Two safety mechanisms:
- Confirmation policy: which actions may run autonomously and which require asking.
- Security analyzer: judge at each step whether the action is safe to execute.
Choosing a framework for research
- Long rollouts, built-in observability, full features: OpenHands.
- Reading and hacking the whole thing quickly: mini-swe-agent, which is only bash and short enough to read in an afternoon.
Single agent or multi-agent
Early 2024 favored specialized roles: manager, reproducer, fault localizer, editor, verifier, chained in a fixed workflow. In late 2024 the instructor wrote "Don't sleep on single agent systems", arguing:
Models keep improving. Instead of hard-coding roles, write a prompt that says do this, then this, then that, and let stronger instruction following execute it.
He calls it a bet, but the outcome suggests it was right: fixed multi-agent workflows are now uncommon in coding.
How a single agent keeps its capability
- Keep the tool set small: edit files, run commands, browse, plus MCP for external services.
- Split a growing prompt into skills: when the prompt fills with "when pushing to GitHub do this, when using Docker do that", move those into on-demand skills. OpenHands called them microagents early on; Anthropic later called them skills — a much better name, in the instructor's view.
- Skills plus MCP replace multi-agent roles: a multi-agent design gives each role its own prompt and tools; skills and MCP pull the right context in just in time. A single agent can therefore cover the same ground.
They also published work showing that giving a model code capability and multimodal browsing lets it solve many different tasks, beating more specialized agents on several benchmarks. That supports the generalist single-agent route.
When multi-agent still helps
Three reasons:
- Efficiency: ten independent PRs can be dispatched in parallel.
- Independent review: an agent that does not see the rationale is harsher than self-review, where the agent holds all the justifications and tends to be lenient.
- Cost control: a sub-agent drops most of its context when reporting back; and a cheaper model can implement while an expensive one orchestrates.
But multi-agent context management is genuinely hard. Their asynchronous software-engineering experiment — a manager dispatching engineers on long tasks such as PaperBench and Commit0 — showed that engineers must return exactly the right context, avoid conflicting with each other, and the manager must integrate the results. Deep research benefits more easily, because work can be split by document source.
Where the agent runs
- Devin started as a cloud SaaS.
- OpenDevin ran locally in Docker, largely for safety.
- Claude Code, Codex, and OpenHands can now run locally.
The instructor expected local execution to be unacceptable: a prompt injection could exfiltrate data or API keys. But in practice, for human-in-the-loop coding, users strongly prefer local. Cloud is more common for background and non-coding work such as code review.
Four interfaces
| Interface | Strength | Limitation |
|---|---|---|
| IDE | Familiar; files, diffs, tests, debugger | Legacy if you no longer read code |
| GUI desktop app | Redesigned for agents, suits non-programmers | Still a minority among programmers |
| CLI | Works anywhere, including servers | Limited expressive interaction |
| Integrations (Slack / GitHub) | Stays where you already work | Complex operations are constrained |
In the class poll, CLI came first and IDE second. The instructor expects integrations to grow: no special interface to open, the agent lives in the tools you already use.
An experiment about people
They compared GitHub Copilot users with OpenHands agent users on the same tasks:
- Agent users took about half the time and completed about twice as many tasks.
- Negative feedback concentrated on two points: "watching the agent work is boring" and "it is hard to confirm whether it did a good job".
Those two complaints define the core interface problems: helping people work in parallel, and helping them verify cheaply.
Automation: turning repeated workflows into a trainable system
An automation is a trigger (issue, PR, or schedule) → context gathering → an isolated workspace run → an artifact → a check by a person or policy.
OpenHands focuses on two:
- Issue triage: if an issue is unclear, say what must be clarified; if clear, list the acceptance criteria.
- PR review: check against written guidelines — is it in scope, does it satisfy the checklist.
Both share a property: the same process repeats endlessly, which makes it a machine learning problem. Inputs and outputs exist, and training data accumulates.
Skill Refiner
Their recent paper closes the loop automatically:
- Collect success and failure trajectories; failure signals are delayed, such as the bot saying LGTM and a maintainer disagreeing.
- Embed and cluster successes and failures separately.
- Generate proposals for instruction edits.
- Gate proposals on evidence that they improve the next review.
- Merge them into a new skill.
Across several models it was both cheaper and better than baselines such as GEPA and Trace2Scale.
A further direction is mining workflows from traces: cluster how users actually use the system and turn recurring patterns into automations, then feed them into the refinement loop. The instructor notes that today they edit prompts; training the model is the other path, but it requires rigorous evaluation for underspecified tasks such as triage and review, which is itself very hard.
Takeaways
- A product agent competes on defaults, replaceability, and fit with real workflows.
- A single agent with a small tool set plus skills and MCP can cover what fixed multi-agent workflows used to.
- Multi-agent value is in parallelism, independent review, and cost tiering, not role-play.
- Local and cloud are not better or worse; they fit different task types.
- Interface design must solve parallelism and verification, not just conversation.
- Once a repeated workflow is stable, treat it as a data loop to improve.
Check your understanding
1. Why does "stronger models plus a single agent" erode the value of fixed multi-agent workflows?
Reference answer: A fixed multi-agent workflow hardcodes the process into roles — manager, reproducer, fault locator, editor, verifier — and a fixed chain of handoffs, effectively writing "first do this, then that" into the orchestration. As models improve, their instruction-following is enough for one agent to follow the same steps, and more flexibly; the hardcoded roles become constraints instead, and each role must hand over exactly the right context, so a failure anywhere drags down the whole. The speaker's late-2024 argument was that a single prompt saying "first do this, then that" beats hardcoding roles. The outcome bears it out: fixed multi-agent workflows are now uncommon in coding.
2. How do skills and MCP substitute for role specialization?
Reference answer: Multi-agent gives each role a fixed prompt and tool set; skills and MCP pull the needed context and external services in on demand. A single agent can therefore cover every capability: keep the tool set simple (edit files, run commands, browse) and connect external services through MCP; when a prompt fills up with "do this when pushing to GitHub, do that when using Docker", split it into skills loaded when needed (OpenHands called them microagents early on; the industry name is now skills). Role division is replaced by on-demand context, which also avoids the multi-agent context handoff problem.
3. Why do self-review and independent review differ?
Reference answer: In self-review the agent holds all the justifications for its own decisions and tends to be lenient with itself, confirming what it already did. Independent review puts another agent on the task without the reasoning behind the decisions, so it judges only the result against the rules and is stricter. This is one of the lecture's three remaining reasons for multi-agent (the others being parallel dispatch and cost tiering).
4. Why are issue triage and PR review especially suited to automation and continuous improvement?
Reference answer: Because the same process repeats countless times — deciding whether an issue is clear, checking a PR against the team's review rules — and both have inputs and outputs, so training data accumulates naturally and the feature becomes a machine-learning problem. Feedback exists too, though often delayed ("the bot said LGTM but the maintainer said no"). That enables the Skill Refiner loop: collect success and failure trajectories, embed and cluster them separately, generate instruction-edit proposals, validate with evidence gating that a proposal actually improves the next review, and merge the result into a new skill. The speaker also notes that such under-defined tasks are hard to evaluate, which is why the current loop edits prompts rather than the model.
5. What interface problems do "boring" and "hard to verify" correspond to?
Reference answer: Both come from the same comparison: agent users finished the same batch of tasks in about half the time and roughly twice the volume, but the negative feedback clustered in two complaints. "Watching the agent work is boring" is about how to let a person parallelize effectively — the interface should not require watching one trajectory, but support pushing several tasks at once and stepping in at decision points. "Hard to confirm whether it did well" is about how to let a person verify cheaply — diffs, test results, evidence and conclusions must be organised so they can be checked at a glance rather than by rereading the whole process. Together these are the two core interface problems, beyond merely "can it converse".
Connections
This lecture ties together tool use, context management, planning, safety, and training from a product perspective. It echoes the first lecture's claim: an agent is a system, and reliability comes from the harness, evaluation, and iteration under real use, not from model capability alone.