这一讲的主题

前面几讲的 agent 主要和文本、代码打交道。计算机使用 agent（computer use agent，CUA）换了一个动作空间：它看到的是屏幕截图，做的是点击、滚动、输入、拖拽这些人类在同一套图形界面里做的动作。

这意味着它不需要目标软件提供 API，也不需要理解网页的 DOM。只要能截图、能控制鼠标键盘，它就能操作几乎任何程序——网页、桌面应用、手机。代价是动作空间变成像素级的坐标，环境噪声大，评估也更困难。

一个最小循环

1. 给定自然语言任务，例如「买一盒 $20 以内的蓝莓」；
2. 环境返回当前截图；
3. 视觉语言模型输出一段推理，再输出一个结构化动作，例如「左键点击坐标 (x, y)」；
4. 基础设施执行动作；
5. 环境返回新截图；
6. 把动作与新截图追加到历史，重复，直到模型认为完成或预算耗尽。

奖励通常是 0/1 的二元信号：任务完成或没有完成。长程任务里也会用部分奖励。

时间轴

发展史：从玩具环境到真实桌面

课堂上展示的 GPT-6 Astra 演示在 CAD 软件里做三维建模，视频被加速了 30 倍——这说明能力已经很强，但仍然慢且贵。

一个值得记住的判断：环境本身正在被模型生成。早期基准需要人类先写好开源应用再灌数据；现在可以让编码 agent 直接生成一个 Gmail 克隆并填入合成数据，质量甚至比真实网站更干净。

四类评估方式

计算机使用任务的「成功」不像数学或软件工程那样容易用单元测试判定，因此评估被拆成几种互补的做法。

1. 静态评估：看每一步动作

给定一张截图和一个任务，判断模型预测的下一步动作是否落在标准答案的包围盒内。

• ScreenSpot-Pro：约 1,500 个截图样本，覆盖多种应用与操作系统，测的是「 grounding 」能力——能不能把「这个按钮」落到正确的像素区域。
• Mind2Web：下一动作模仿，给当前截图与任务，预测人类会点击哪里。

优点是非常快、非常便宜，不需要真实环境。缺点是假阴性多：同一个任务往往有多种正确走法，标准答案只收录了其中一种，其他合理动作会被判错。它测量的是「像不像标注者」，而不是「有没有完成任务」。

2. 端到端评估：看最终状态

在可控环境里让 agent 完整执行任务，最后用程序化检查验证状态。

• WebArena：约 800 个任务、4 个模拟应用，验证器由 CMU 的研究生手写；只检查结果，不监督每一步。
• VisualWebArena：910 个必须依赖视觉信息才能完成的任务，推动社区从无障碍树转向截图。
• OSWorld：真实 Linux 虚拟机，369 个任务、9 个应用，例如根据收据更新 LibreOffice 表格并保存。
• Windows Agent Arena：微软面向 Windows 企业工作流的同类基准。

这种评估更接近真实目标，但有两个风险：验证器不全面会带来假阳性/假阴性；agent 可能奖励作弊——不去完成任务，而是去满足验证器的检查条件。

3. LLM 评委：看轨迹

当无法为每个任务写验证器时，把任务和完整轨迹交给另一个 VLM，让它判断成功与否。

• WebVoyager：15 个真实网站、600 个任务，直接在真实互联网上操作。
• Online-Mind2Web：扩展到 136 个真实网站、300 个任务，评委与人类的一致率约 85%。

真实网页评估最接近现实，但没有程序化验证器，评委可能被表面证据骗过；而且大量 agent 访问会给网站带来真实流量，站点所有者并不欢迎。课堂的判断是：这类基准现在用得少了，一是已被现代模型基本解决，二是部署真实 agent 的顾虑变多。

4. 长程评估：看过程与部分得分

长程任务的最终奖励非常稀疏——agent 可能做了大量有用工作，却在最后一步失败，最终得 0。因此这类基准改用**多条评分细则（rubric）**给出中间分。

• Odyssey（CMU）：200 个长程网页任务，每个任务有 8 条左右的细则，例如「找到匹兹堡到洛杉矶的直飞航班」「把驾车时间安排在 9:00–16:00」。
• OSWorld 2：108 个任务，中位数需要人类 1.6 小时，前沿 agent 平均约 300 次调用；包含 CAD 建模、照片编辑等专业工作流。
• KA World Long：200 个专业领域任务，多数需要 500 次以上调用。

评分细则常见做法是 LLM 生成、人类校验；生成时可以给模型「特权信息」（例如网页 HTML），让它先做对，再据此写细则。

模型是怎么建的

表示：交错的视觉—语言历史

CUA 本质上仍是一个视觉语言模型。输入是自然语言目标加上截图序列，输出是文本形式的结构化动作。历史不断累积：

[目标] [截图0] [动作0] [截图1] [动作1] [截图2] ...

文本被 tokenize 成文本 token，截图被编码成视觉 token，两者直接拼接后送入 Transformer。模型在每一步基于整段历史预测下一个动作。

坐标通常被归一化到 0–1000，与绝对分辨率无关，因此在 Mac 上训练的策略可以迁移到平板或不同分辨率的屏幕。

动作空间

不同模型的动作集合略有差异，但大体相似：点击、滚动、输入、拖拽、按键。表示方式可能是 pyautogui 调用、工具调用 JSON，或 XML 标签；环境侧需要解析成真正可执行的指令。

为什么主要用截图而不是 HTML

HTML 很脏，桌面应用也拿不到统一的 DOM；截图则对所有设备一致。课堂上提到，截图方案的性能好得超出预期，以至于结构化表示显得不那么必要。

不过，前沿模型在可获得时会同时使用无障碍树等元数据，因为这类结构信息能提升鲁棒性。纯截图的好处是通用：训练在 Windows 上的模型更容易迁移到 Linux 或 macOS。

工程细节：加载与等待

执行动作后通常先睡眠约 3 秒再截图；如果截到加载页，模型会被训练为再等一段时间。更复杂的系统会等到画面变化再继续，但「睡一会儿」是最常见的做法。

三段式训练

1. 预训练：定位与 grounding

目标是让模型学会「屏幕上的东西在哪」。数据形式是截图加包围盒加坐标，模型学习输出元素的位置。这类数据可以大规模自动生成——在网页上跑一段 JavaScript 就能导出所有元素的包围盒。

同时会加入动作预测数据：给截图和任务，预测下一步动作。数据噪声大，但量可以很大。这一阶段只建立最基础的能力：看懂截图、定位元素、理解元素之间的相对关系。

2. 后训练 SFT：模仿人类与合成轨迹

• 人类演示：Momo Web 收集了约 36,000 条人类操作轨迹；AgentNet 也采用类似方式。每条轨迹是「截图—动作」序列。
• 合成轨迹：让一个能访问无障碍树的模型（拥有推理时不存在的特权信息）先完成任务，收集轨迹，再丢弃特权信息，只保留截图与动作。Momo Web 用这种方式生成了约 105,000 条合成轨迹，并按质量筛选。

课堂上用特斯拉做类比：训练时车上装着激光雷达，收集轨迹后丢掉雷达数据，只让模型学像素输入。教师看到的信息比学生多，学生因此学得更好。

3. 强化学习：在模拟环境里试错

把模型放进模拟环境，让它自主行动，根据结果给奖励，再更新策略。成功的轨迹被提高权重，失败的被降低。这个过程不需要人类演示。

为什么必须用模拟环境：真实世界里反复订车、下单、退款很快会被封号或烧钱。模拟环境让试错成本可控。

课堂给出的数字：SFT 之后成功率大约 0.5，加上 RL 后可以到 0.6 以上。更重要的是，长程能力很难靠人类演示获得——收集几小时长的人类轨迹极其昂贵，而 RL 可以在模拟环境里反复练习。

仍然开放的问题

• 速度：能力已接近或超过人类中位数，但推理仍然慢且贵。
• 个性化：如何适配单个用户的习惯与偏好。
• 基础设施与体验：加载、弹窗、权限、分辨率适配等工程问题占很大比重。
• 多智能体：多个 CUA 协作完成跨应用任务。

本讲结论

• CUA 的价值在于与人共用同一套界面，因此不依赖 API，也不受应用是否开放的限制。
• 评估没有单一答案：静态评估快但假阴性多，端到端验证器接近真实但可能被奖励作弊，LLM 评委灵活但不稳，长程任务需要细则化评分。
• 模型训练分三段：预训练学定位，SFT 学模仿（人类加合成），RL 学在环境中最大化奖励。
• 截图是通用表示，无障碍树是有效补充；坐标归一化带来跨分辨率迁移。

理解检验

1. 静态评估和端到端评估分别在测量什么？各自的假阳性/假阴性来自哪里？

参考答案：静态评估测的是「下一步动作是否落在标准答案的包围盒内」，也就是 grounding 能力与「像不像标注者」。它的假阴性很多：同一个任务往往有多种正确走法，标准答案只收录了一种，其他合理动作会被判错；它也可能出现「动作像、任务却失败」的情况，因为它不看整体结果。端到端评估测的是最终状态是否满足程序化验证器，更接近真实目标。它的假阳性来自验证器不全面——agent 可能奖励作弊，去满足检查条件而不是完成任务；假阴性来自验证器漏掉了合理但未被编码的状态。

2. 为什么长程任务更适合用评分细则而不是单一成功/失败？

参考答案：长程任务的最终奖励非常稀疏——agent 可能做了大量有用工作，却在最后一步失败，最终得 0，这个信号无法区分「完全不会」与「几乎完成」。评分细则把任务拆成多条可检查的子条件（Odyssey 每个任务约 8 条，例如「找到匹兹堡到洛杉矶的直飞航班」「把驾车时间安排在 9:00–16:00」），于是能给中间分：既反映真实进展、指导改进，也降低单点失败的惩罚。

3. 合成轨迹里的「特权信息」指什么？为什么丢弃它之后学生模型仍然能学到东西？

参考答案：特权信息是推理时模型拿不到、但教师可以利用的信息，典型是网页的无障碍树或 HTML——它能直接定位元素，而部署时 agent 只有截图。做法是让能访问无障碍树的教师先完成任务，收集「截图—动作」轨迹，再丢弃特权信息，只用截图与动作训练学生。学生仍能学，是因为动作标签来自成功轨迹（标签是对的），它要学的是「从像素推断出同样的动作」，相当于把教师看到的结构信息蒸馏进视觉表示。课程用特斯拉做类比：训练时车上装激光雷达，部署时只用摄像头。

4. 为什么 CUA 的强化学习几乎必须在模拟环境中进行？

参考答案：真实世界里反复订车、下单、退款很快会被封号或烧钱，试错成本不可控，而且很多动作不可逆；模拟环境可以无限重置、控制成本、并行采样，让试错变得便宜。课堂给出的数字：SFT 之后成功率大约 0.5，加上 RL 可以到 0.6 以上。更关键的是长程能力很难靠人类演示获得——收集几小时长的人类轨迹极其昂贵，而 RL 可以在模拟环境里反复练习。

5. 纯截图表示相比 HTML/无障碍树，优势与代价分别是什么？

参考答案：优势是通用：网页、桌面应用、手机都是同一套截图像素，不依赖目标软件提供 API 或统一 DOM；在 Windows 上训练的策略也更容易迁移到 Linux 或 macOS。代价是动作空间变成像素级坐标，模型必须先把「这个按钮」grounding 到正确区域；信息密度也低，元素的结构关系要从像素里自己恢复，长程任务更容易累积误差。因此前沿模型在可获得时会同时使用无障碍树等结构信息来提升鲁棒性，纯截图的价值在于可迁移性。

与前后课程的关系

这一讲把「环境理解」具体化成了 GUI。前一讲的工具调用在这里变成点击与输入；后一讲的监督微调会再次出现模仿学习，而第 9、11 讲会展开这里只用了一段话描述的强化学习。

What this lecture is about

Earlier lectures had agents working through text and code. A computer use agent (CUA) changes the action space: it sees screenshots and performs the same actions a person would — click, scroll, type, drag — inside the same graphical interface.

The agent needs no API from the target software and no understanding of the page's DOM. If it can take a screenshot and control a mouse and keyboard, it can operate almost any program: a website, a desktop app, a phone. The price is a pixel-level action space, noisy environments, and harder evaluation.

The minimal loop

1. A natural-language task, such as "buy a box of blueberries under $20".
2. The environment returns the current screenshot.
3. A vision-language model produces a reasoning trace and then a structured action, such as "left click at (x, y)".
4. The infrastructure executes the action.
5. The environment returns a new screenshot.
6. The action and screenshot are appended to history; repeat until the model stops or the budget runs out.

The reward is usually binary — task complete or not — though long-horizon work also uses partial credit.

Timeline

History: from toy worlds to real desktops

The GPT-6 Astra demo shown in class performs 3D modeling in CAD software. The video is sped up 30x, which makes the point: the capability is strong, but it is still slow and expensive.

A notable shift: environments themselves are now generated. Early benchmarks needed humans to write open-source apps and then populate them; today a coding agent can generate a Gmail clone with synthetic data that is cleaner than the real thing.

Four families of evaluation

"Success" in computer use is not as easy to test as a math answer or a code patch, so evaluation splits into complementary approaches.

1. Static evaluation: score each action

Given a screenshot and a task, check whether the predicted next action falls inside the ground-truth bounding box.

• ScreenSpot-Pro: about 1,500 screenshots across applications and operating systems, measuring grounding — can the model turn "this button" into the right pixels?
• Mind2Web: next-action imitation; given the screenshot and task, predict where a human would click.

This is fast and cheap because no environment is needed. The weakness is false negatives: there are usually several valid ways to proceed, and the annotation records only one. The benchmark measures similarity to the annotator, not task completion.

2. End-to-end evaluation: check the final state

Run the agent to completion in a controlled environment and verify the resulting state programmatically.

• WebArena: about 800 tasks on four simulated apps; verifiers were hand-written by CMU graduate students and check outcomes rather than steps.
• VisualWebArena: 910 tasks that can only be solved with visual information, pushing the community away from accessibility trees toward screenshots.
• OSWorld: a real Linux VM, 369 tasks across nine apps, such as updating a LibreOffice spreadsheet from receipts and saving it.
• Windows Agent Arena: Microsoft's counterpart for Windows enterprise workflows.

This is closer to the real objective, with two risks: incomplete verifiers produce false positives and negatives, and agents can reward hack by satisfying the check rather than doing the task.

3. LLM judges: score the trajectory

When a verifier cannot be written, hand the task and full trajectory to another VLM and ask whether it succeeded.

• WebVoyager: 15 real websites, 600 tasks, operating on the live internet.
• Online-Mind2Web: 136 real sites, 300 tasks, with a judge that agrees with humans about 85% of the time.

Live-web evaluation is the most realistic, but has no programmatic verifier, so the judge can be fooled by surface evidence. It also sends real traffic to sites whose owners dislike it. The lecture's read: these benchmarks are used less now, partly because modern models have largely solved them and partly because deploying agents on the live web has become more cautious.

4. Long-horizon evaluation: process and partial credit

On long tasks the final reward is very sparse. An agent can do hours of useful work and fail the last step, scoring zero. These benchmarks therefore use rubrics that award partial credit.

• Odyssey (CMU): 200 long-horizon web tasks, each with roughly eight criteria, such as "find a non-stop flight from Pittsburgh to Los Angeles" and "schedule the drive between 9 a.m. and 4 p.m.".
• OSWorld 2: 108 tasks with a median human time of 1.6 hours; frontier agents average around 300 calls, including CAD modeling and photo editing.
• KA World Long: 200 specialized tasks, most requiring more than 500 calls.

Rubrics are typically LLM-generated and human-verified. Generation can use privileged information, such as page HTML, so the model performs better and the resulting rubric is more accurate.

How the models are built

Representation: an interleaved vision-language history

A CUA is still a vision-language model. Input is a natural-language goal plus a sequence of screenshots; output is a structured action expressed as text. History keeps growing:

[goal] [screen 0] [action 0] [screen 1] [action 1] [screen 2] ...

Text is tokenized into text tokens, screenshots are encoded into visual tokens, and the two are concatenated before entering the transformer. At each step the model conditions on the whole history to predict the next action.

Coordinates are usually normalized to 0–1000 rather than absolute pixels, so a policy trained on a Mac transfers to a tablet or a different resolution.

Action space

Models differ slightly in what they can emit: click, scroll, type, drag, key press. The representation might be pyautogui calls, tool-call JSON, or XML tags, and the environment parses it into something executable.

Why screenshots rather than HTML

HTML is messy, and desktop apps do not expose a unified DOM. A screenshot is consistent across every device. The lecture notes that screenshot-based policies perform surprisingly well, to the point where structured representations can seem unnecessary.

Frontier models do use accessibility trees and other metadata when available, because structure improves robustness. The advantage of pure screenshots is generality: a model trained on Windows transfers more easily to Linux or macOS.

Engineering detail: waiting

After an action, the system usually sleeps about three seconds and then takes a screenshot. If the screen is still loading, the model is trained to wait again. More elaborate infrastructure waits for the screen to change, but sleeping is the common approach.

Three-stage training

1. Pretraining: localization and grounding

The goal is to teach the model where things are on screen. Data is a screenshot plus bounding boxes plus coordinates, and the model learns to output the location of an element. Such data can be generated at scale — a short JavaScript snippet on a web page yields every bounding box.

Action-prediction data is mixed in: given a screenshot and a task, predict the next action. The data is noisy but plentiful. This stage builds only basic abilities: reading a screenshot, locating elements, and understanding their relative positions.

2. Post-training SFT: imitate humans and synthetic traces

• Human demonstrations: Momo Web collected about 36,000 human trajectories; AgentNet is similar. Each trajectory is a screenshot-action sequence.
• Synthetic traces: let a model that can access the accessibility tree — privileged information unavailable at inference — complete the task, collect its trajectories, then discard the privileged channel and keep only screenshots and actions. Momo Web generated about 105,000 synthetic traces this way and filtered them by quality.

The lecture's analogy is Tesla: train with lidar on the car, collect trajectories, discard the lidar, and learn from pixels. The teacher sees more than the student, so the student learns better.

3. Reinforcement learning: trial and error in simulation

Place the model in a simulated environment, let it act autonomously, assign rewards, and update the policy. Successful trajectories are upweighted, failures downweighted. No human demonstration is involved.

Simulation is mandatory because real-world trial and error is costly: repeatedly booking and refunding rides gets an account banned or runs out of money. A simulator makes failure cheap.

The lecture's numbers: success after SFT is around 0.5; adding RL pushes it above 0.6. More importantly, long-horizon skill is hard to obtain from demonstrations — collecting multi-hour human traces is prohibitively expensive, while RL can practice in simulation.

Open problems

• Speed: capability is at or above median human level, but inference remains slow and expensive.
• Personalization: adapting to one user's habits and preferences.
• Infrastructure and UX: loading states, pop-ups, permissions, resolution handling consume a large share of engineering effort.
• Multi-agent: several CUAs collaborating across applications.

Takeaways

• A CUA's value is sharing the human interface, so it needs no API and is not limited to apps that expose one.
• There is no single evaluation: static scoring is fast but yields false negatives; end-to-end verifiers are realistic but reward-hackable; LLM judges are flexible but noisy; long tasks need rubrics.
• Training has three stages: pretraining for localization, SFT for imitation (human plus synthetic), RL for reward maximization in an environment.
• Screenshots are the general representation; accessibility trees are a useful supplement; normalized coordinates transfer across resolutions.

Check your understanding

1. What do static and end-to-end evaluation each measure, and where do their false positives and negatives come from?

Reference answer: Static evaluation measures whether the next predicted action falls inside the annotated bounding box — grounding, or "does it look like the annotator". Its false negatives are frequent: most tasks admit several correct paths, the reference records only one, and other reasonable actions are marked wrong; it can also accept an action-shaped output while the task as a whole fails, because it never sees the outcome. End-to-end evaluation measures whether the final state satisfies a programmatic verifier, which is closer to the real goal. Its false positives come from incomplete verifiers — the agent may reward-hack by satisfying the check instead of completing the task; false negatives come from verifiers missing legitimate states that were never encoded.

2. Why are rubrics a better fit than binary success for long-horizon tasks?

Reference answer: The final reward for a long task is very sparse — the agent may do a great deal of useful work and still score 0 by failing the last step, a signal that cannot distinguish "knows nothing" from "almost finished". Rubrics split the task into several checkable conditions (Odyssey uses about eight per task, such as "find a direct flight from Pittsburgh to Los Angeles" and "schedule the drive between 9:00 and 16:00"), so partial credit reflects real progress, guides improvement, and reduces the penalty for a single failure.

3. What is the "privileged information" in synthetic traces, and why can the student still learn after it is discarded?

Reference answer: Privileged information is what the deployed model cannot see but a teacher can use — typically the accessibility tree or HTML, which locates elements directly, whereas the deployed agent only gets screenshots. The teacher, able to read the tree, completes the task; the screenshots and actions are recorded, and the privileged information is then discarded so the student trains only on pixels and actions. The student still learns because the action labels come from successful trajectories (so they are correct), and what it must learn is "infer the same action from pixels" — effectively distilling the teacher's structural view into a visual representation. The lecture's analogy is a Tesla training with lidar on the car and deploying with cameras only.

4. Why is reinforcement learning for CUAs almost always done in simulation?

Reference answer: In the real world, repeated bookings, purchases and refunds quickly get accounts banned or burn real money; trial-and-error is neither controllable nor reversible. A simulated environment can be reset indefinitely, keeps costs bounded, and samples in parallel. The lecture's numbers: success is around 0.5 after SFT and above 0.6 with RL. More importantly, long-horizon ability is hard to obtain from human demonstrations — collecting multi-hour human trajectories is extremely expensive — while RL can rehearse repeatedly in simulation.

5. What does pure-screenshot representation gain and lose relative to HTML or accessibility trees?

Reference answer: It gains generality: web pages, desktop apps and phones all reduce to the same pixels, with no need for an API or a unified DOM, and a policy trained on Windows transfers more easily to Linux or macOS. It loses structure: the action space becomes pixel coordinates, so the model must first ground "this button" to the right region; information density is lower because element relationships have to be recovered from pixels, and errors accumulate over long tasks. That is why frontier models also use accessibility-tree metadata when available, for robustness, while the value of pure screenshots lies in transferability.

How this connects to the course

This lecture turns "environment understanding" into the GUI. Tool calls from the previous lecture become clicks and keystrokes. The next lecture revisits imitation learning through supervised fine-tuning, and lectures 9 and 11 expand the reinforcement learning sketched here.