什么问题需要「深度研究」

「我在 CMU 的办公室号码是多少？」只需要一次搜索、打开一个页面、抽取一个字段。这是检索增强生成（RAG）的典型场景。

但「AI agent 能否像人类专家一样综合科学文献？」不属于这类问题：没有任何一个文档能直接给出答案。agent 必须先判断需要哪些证据，分别检索，读完后发现缺口，再调整查询，最后综合成一份带引用的长文。

这就是深度研究 agent 的基本形态：

1. 生成计划，判断需要哪些类型的信息；
2. 发出搜索调用，检索系统返回一组文档；
3. 基于已有证据反思：够了吗？还缺什么？
4. 修改查询，继续检索；
5. 信息足够后，综合出带引用的最终答案。

它同时提出三类研究问题：怎么评估、怎么训练、检索系统本身要怎么改。

时间轴

为什么传统 QA 基准不够用

Natural Questions 一类基准收集真实搜索查询，答案标注基于某一年的维基百科。它们的局限正好对应深度研究的需求：

四类评估方式

1. 复杂但可验证的短答案

BrowseComp（OpenAI）是最常用的基准之一。它的构造方式很特别：从一个已知答案（人物、事件、作品）出发，收集若干线索，再把线索组合成一道复杂问题。

验证流程包括：确认基座模型或五次 Google 搜索无法直接答出；部分题目请人类确认十分钟内做不出来。最终答案是短实体，只需检查是否出现在回答里即可自动判分。

结果：2025 年前后的前沿模型即使带搜索也表现不佳，人类专家约 25%；专门训练的模型可以到 50% 左右。

代价是贵且不可复现：每题可能需要上百次搜索调用，搜索 API 有成本、有延迟，而且 Google 结果每天都在变。

BrowseComp-Plus 解决可复现性：固定一个约 10 万文档的语料库（由模型挑出可能有用的文档、人类核验，再混入看似相关实则无用的负例），共 830 道题。因为语料固定，结果可复现，也方便研究检索系统。

2. 专家领域

• FinSearchComp：金融领域，由 MBA 或从业者出题，答案需要精确计算，常要调用计算器工具；
• MedBrowseComp：医学领域；
• ResearchBench：要求找出同时满足多个条件的某篇论文。

共同点：高质量题目必须由专家标注并复核，成本很高。

3. 开放式长文

传统做法是准备一份或几份参考答案，再算相似度。这在开放式任务上问题很大：

• 正确答案有无数种写法，无法穷举；
• 词面相似不代表结论正确——一段与参考答案高度重叠的文字可能把核心观点说反；
• 课堂引用的研究显示，BLEU、ROUGE、BERTScore 一类指标与专家偏好的一致率只有 57%–62%，仅略高于随机。

评分细则（rubric） 是替代方案：不给参考答案，而由专家列出若干条「好答案应包含什么」，每条带权重；再用语言模型逐条判断模型回答是否满足，最后加权求和。

OpenScholar 的报告显示：专家与专家在细则判断上的一致率约 80%，语言模型与专家约 79%。也就是说，只要细则质量够高，就可以用模型把评估规模化。

问题是细则本身很贵：一道题可能需要 PhD 级别专家花近一小时标注。

4. 引用质量

DeepResearch Bench 的 FACT 框架把回答拆成「主张 + 引用 URL」，抓取网页后判断该网页是否支持该主张。若两条主张共用同一引用、只有一条被支持，则引用准确率为 50%。

它检查的是「引用是否支持主张」，还没有检查「证据本身是否可靠」，后者仍是开放问题。

自动生成细则的坑

ResearchQA 试图用综述论文自动生成问题与细则，得到 2.1 万道题、12 万条细则。但专家审计发现约 15% 的细则有错：引用不存在的论文、重复问题、或者过于空泛而无法区分好坏答案。

结论：模型生成的细则必须被外部证据 grounding，或至少由人类抽检。

怎么训练一个深度研究 agent

课堂以阿里的 Tongyi DeepResearch 为例，它包含三个阶段，但不是每个系统都会全部使用。

阶段一：中期训练，让基座模型见过 agent 行为

把大量 agent 轨迹混入持续预训练数据，再进入下游 SFT。直觉是：先让基座模型熟悉「思考—调用—读结果」的格式，后面的 SFT 会更有效。

实验对比了从普通 Qwen 基座与从中期训练 checkpoint 出发、使用完全相同 SFT 数据的两个模型。在更复杂的 BrowseComp 上，中期训练过的起点明显更好。

阶段二：SFT，模仿教师

关键是数据从哪来。

问题构造：利用知识图谱（如 Wikidata）。实体是节点，关系是有向边；从目标实体出发随机游走，收集一组相连的事实，再组合成多跳问题，起始实体就是标准答案。

这和 BrowseComp 的构造逻辑几乎一样——训练数据的构造在反向工程评测集。它是否能迁移到其他基准仍是开放问题，但至少提供了一种可扩展的高质量数据生成方式。

轨迹生成：让更强的教师模型为每道题生成多条轨迹；按答案匹配或格式奖励过滤，只保留成功的轨迹做 SFT。

代价：SFT 的上限受教师能力限制，而且教师轨迹可能带有学生难以模仿的风格。

阶段三：强化学习

对答案可验证的短问题，可以直接用 RLVR：策略采样多条轨迹，用精确匹配或模型判断对错，再按 GRPO 更新。

Search-R1 是较早展示这条路线的代表工作。与 SFT 的关键差别是：训练数据来自策略自己的尝试，因此模型会经历失败并学会恢复。

课堂上强调，实际训练还会加入两类稳定化项：限制单次更新幅度的裁剪，以及约束模型不要偏离初始策略太远的 KL 惩罚。它们在长轨迹、大规模数据下尤其重要。

工程：让训练跑得动

• 异步执行多条 rollout：一条轨迹在等搜索 API 时，其他轨迹继续生成；
• 缓存搜索结果；
• 使用更稳健的 API 封装；
• 用本地索引模拟外部搜索，降低对真实 API 的依赖。

开放式任务：演化中的评分细则

短答案可以用精确匹配，长文不行。讲者团队的工作把细则评估变成训练奖励，并让细则随策略一起演化：

1. 先建立一批由外部检索支撑的初始细则；
2. 每道题让策略采样多条轨迹（最终训练用 8 条）；
3. 用语言模型对比这些轨迹，生成新的正例细则（奖励好行为）与负例细则（惩罚具体错误）；
4. 维护细则缓冲区，用它们给回答打分并更新策略。

核心思想是：反馈不是训练开始时固定下来的，而是随模型能力一起发展。当策略学会了新的区分，细则也能捕捉到。

消融实验的几个结论

• 没有 SFT 很难启动：从基座直接 RL，600 步时只有约 40%；只用 5% 的 SFT 数据就能提升约 10 个百分点。
• 随机奖励无效：用随机奖励做在线 RL，性能没有提升。
• 检索支撑的细则有效：初始细则若由搜索证据支撑，效果明显好于纯模型生成；后者甚至可能有害。
• 细则生成模型越强越好：用 GPT-4.1 生成细则比用 Qwen 高约 10 个百分点。近期出现了一些开源细则生成模型，但主要在短答案奖励基准上评估，能否替代强模型仍是开放问题。
• 演化细则在长训练中收益更大：训练越久，细则更新带来的提升越明显。

结果上，这类模型能以比专有深度研究系统少数百到上千倍的推理成本，达到接近甚至更好的表现。

检索不是背景板

如果检索系统找不到对的文档，再强的 agent 也无能为力。

本地语料 vs 外部 API

• 已知目标语料时（例如维基百科），可以建本地索引，用 BM25 或神经嵌入检索；Search-R1 就用 2018 年维基百科。
• 开放式问题需要网页搜索 API、浏览 API，或论文检索 API（Semantic Scholar、Google Scholar）。
• 领域任务（如金融）应收集领域文档建本地库，或使用领域专用 API。

嵌入模型基础

离线用文档编码器把所有文档编码成向量库；推理时编码查询，做最近邻搜索。

训练通常用对比学习：每个查询有正文档与一组负文档，目标是让正文档的得分相对负文档更高。负例可以来自同一 batch 内其他问题的正文档（in-batch negatives），也可以用 BM25 挖更难的负例。

面向 agent 的检索：只看最后一句查询是不够的

标准嵌入模型针对「人类输入一句查询」优化，但 agent 拥有丰富历史：原始任务、之前的查询、读到的结果、当前的推理。

讲者团队的 AgentIO 把当前推理文本与查询一起送入检索模型，并针对这一形式做小规模训练。结果：

• 仅把推理拼进查询就有提升；
• 专门训练 reasoning-aware 嵌入模型后提升更大；
• 4B 的模型超过了 8B 的基线，同时减少了搜索调用次数；
• 把整条轨迹都塞进去反而变差，因为上下文太长；只取当前查询前一步的推理最有效。

开放问题

• 评估是当前最大瓶颈：知道怎么评估，才知道怎么构造训练数据、怎么做拒绝采样、怎么设奖励。
• 细则偏差：细则评估可能偏好更长的回答；细则本身也可能与真实用户需求错位。
• 题目真实性：BrowseComp 类问题由线索反向构造，未必代表真实用户的信息需求。
• 外部效度：现有基准只看最终回答质量，不看你是否能据此做出更好的决策或修好一个 bug。
• 数据污染：模型可能记住训练数据中的答案；评估时甚至可能直接检索到原始数据集，需要黑名单。
• 小模型的能力边界：在需要管理大量搜索调用与长上下文的基准上，小模型仍明显落后于前沿模型；在开放式综合任务上差距较小。

本讲结论

• 深度研究的本质是「没有单一来源」的信息需求，必须多轮规划、检索、反思与综合。
• 评估分四类：可验证短答案、专家领域、开放式长文（细则）、引用质量。
• 训练分三段：中期训练、SFT、RL；SFT 冷启动对 RL 能否启动至关重要。
• 开放式任务用演化细则做奖励，细则需要外部证据 grounding。
• 检索基础需要为 agent 改造：查询之外的推理上下文很有价值，但不能无限堆叠。

理解检验

1. 为什么 BrowseComp 要「从答案反向构造问题」？这样构造的训练数据有什么迁移风险？

参考答案：反向构造的做法是：从一个已知答案（人物、事件、作品）出发，收集若干线索，再把线索组合成一道复杂问题。这样做有三个好处——答案确定且唯一，可以用精确匹配自动判分；题目必须跨多个来源检索才能解开，天然对应深度研究场景；还能通过「基座模型或五次 Google 搜索答不出」的验证筛掉可被记忆或单次搜索解决的题。迁移风险有两层：一是题目真实性，这类题由线索拼出，未必代表真实用户的信息需求；二是训练数据同样按知识图谱反向构造（实体随机游走生成多跳问题），等于在反向工程评测集，能否迁移到其他基准仍是开放问题，而且存在数据污染风险。

2. 基于参考答案的相似度指标，在开放式长文上为什么会失效？

参考答案：课程给了三个理由。第一，正确答案有无数种写法，无法穷举参考答案。第二，词面相似不代表结论正确——一段与参考答案高度重叠的文字完全可能把核心观点说反。第三，实证上 BLEU、ROUGE、BERTScore 一类指标与专家偏好的一致率只有 57%–62%，仅略高于随机。因此开放式长文需要评分细则（rubric）这种「逐条判断是否满足」的评估方式，而不是相似度。

3. 细则评估要规模化，最大的成本在哪里？自动生成细则的主要错误类型是什么？

参考答案：最大成本在细则本身的人工标注：一道题可能需要 PhD 级别专家花近一小时，专家与专家的一致率约 80%、模型与专家约 79%，说明细则质量够高时模型可以承担判分，但细则的产出无法廉价替代。ResearchQA 用综述论文自动生成细则后，专家审计发现约 15% 有错，主要类型是：引用不存在的论文、重复问题、以及过于空泛而无法区分好坏答案。结论是模型生成的细则必须被外部证据 grounding，或至少由人类抽检。

4. 为什么演化细则比固定细则更适合长训练？

参考答案：固定细则只编码了训练开始时已知的区分标准；策略变强后会出现新的正确做法与新的错误模式，固定细则捕捉不到。演化细则每轮让策略对同一道题采样多条轨迹（最终训练用 8 条），用语言模型对比这些轨迹，生成奖励好行为的新正例细则与惩罚具体错误的负例细则，再放进缓冲区打分。笔记的消融显示，训练越久，细则更新带来的提升越明显——反馈随模型能力一起发展，而不是一开始就冻结。

5. 为什么把整条轨迹送入嵌入模型反而降低检索效果？

参考答案：AgentIO 的对比显示，把推理文本与查询一起送入检索模型有提升，专门训练 reasoning-aware 嵌入模型后提升更大（4B 模型超过 8B 基线，还减少了搜索调用），但把整条轨迹都塞进去反而变差。原因是上下文太长：原始任务、历次查询、读到的结果混在一起会稀释当前查询的信号，超出嵌入模型有效处理的长度。最有效的做法是只取当前查询前一步的推理，与查询一起编码。

与前后课程的关系

上一讲给出策略梯度与 GRPO，这一讲把它们用于「答案可验证」和「答案开放」两类深度研究任务。下一讲会深入更高级的 RL 算法与稳定化技巧；检索与记忆也会在第 4 讲的基础上继续展开。

What kind of question needs deep research

"What is my office number at CMU?" takes one search, one page, and one extracted field. That is the classic retrieval-augmented generation (RAG) setting.

"Can an AI agent synthesize scientific literature as well as a human expert?" is not that kind of question: no single document contains the answer. The agent must decide what evidence it needs, search several times, notice gaps after reading, revise its queries, and finally synthesize a long, cited answer.

That is the basic shape of a deep research agent:

1. Produce a plan describing what information is needed.
2. Issue a search call; the retrieval system returns documents.
3. Reflect on the evidence: is it enough? What is missing?
4. Revise the query and search again.
5. Synthesize a final, cited answer once the evidence is sufficient.

This raises three research problems at once: how to evaluate, how to train, and how to build better retrieval.

Timeline

Why classic QA benchmarks fall short

Benchmarks such as Natural Questions collect real search queries and label answers against a snapshot of Wikipedia. Their limitations map directly onto what deep research needs:

Four families of evaluation

1. Complex but verifiable short answers

BrowseComp (OpenAI) is the best-known benchmark here. Its construction is distinctive: start from a known answer (a person, event, or artifact), collect several clues, and combine them into a hard question.

Validation checks that a base model, or five Google searches, cannot answer it directly; a subset is also given to humans to confirm it takes more than ten minutes. The answer is a short entity, so scoring is automatic: check whether it appears in the response.

The results: frontier models around 2025 still struggled even with web access; human experts scored about 25%; a model trained specifically for the task reached roughly 50%.

The cost is high and reproducibility is poor. A single question may need a hundred search calls, the API costs money and adds latency, and Google's results change daily.

BrowseComp-Plus fixes reproducibility by freezing a corpus of about 100,000 documents — plausible documents selected by a model and verified by humans, mixed with hard negatives that look relevant but are not — across 830 questions. A fixed corpus makes runs repeatable and makes retrieval itself a research object.

2. Expert domains

• FinSearchComp: finance, written by MBAs or practitioners, with answers requiring exact computation and often a calculator tool.
• MedBrowseComp: medicine.
• ResearchBench: find the one paper satisfying several conditions at once.

The common thread: high-quality items require expert annotation and expert verification, which is expensive.

3. Open-ended long-form answers

The traditional approach prepares one or a few reference answers and measures similarity. On open-ended tasks this fails for several reasons:

• There are countless valid ways to answer, so references cannot be exhaustive.
• Lexical overlap does not imply correctness — a response can overlap heavily with the reference while contradicting its main point.
• The cited work finds BLEU, ROUGE, and BERTScore agree with expert preference only 57%–62% of the time, barely above chance.

Rubrics replace references: experts list what a good answer must contain, each item weighted; a language model then checks whether the response satisfies each item, and the weighted sum becomes the score.

OpenScholar reports expert-expert agreement around 80% and model-expert agreement around 79% on rubric judgments. If the rubrics are good, models can scale the evaluation.

The catch is that rubrics are expensive: a single question can take a PhD-level expert close to an hour to annotate.

4. Citation quality

DeepResearch Bench uses the FACT framework: extract claims paired with cited URLs, fetch each source, and judge whether it supports the claim. If two claims share a citation and only one is supported, citation accuracy is 50%.

This checks whether citations support claims, not whether the cited evidence itself is trustworthy — still an open question.

The trap in auto-generated rubrics

ResearchQA generates questions and rubrics from survey papers, producing 21,000 questions and 120,000 rubrics. An expert audit found about 15% of rubrics were wrong: citing papers that do not exist, repeating the question, or stating criteria so generic they cannot distinguish good from bad answers.

The conclusion: model-generated rubrics must be grounded in external evidence, or at least sampled and checked by humans.

Training a deep research agent

The lecture uses Alibaba's Tongyi DeepResearch as an example. It has three stages, though not every system uses all of them.

Stage 1: mid-training to expose the base model to agent behavior

Mix a large number of agent trajectories into continual pretraining, then run the usual downstream SFT. The intuition: familiarize the base model with the "think — call — read result" format so SFT has more to build on.

The experiment compares a standard Qwen base with a mid-trained checkpoint, using identical SFT data. On the harder BrowseComp, the mid-trained starting point is clearly better.

Stage 2: SFT to imitate a teacher

The key question is where the data comes from.

Question construction uses a knowledge graph such as Wikidata. Entities are nodes, relations are directed edges; a random walk from a target entity gathers a set of connected facts, which are combined into a multi-hop question, with the starting entity as the gold answer.

This is almost exactly how BrowseComp is built — the training data reverse-engineers the benchmark. Whether it transfers to other benchmarks is still open, but it is a scalable way to produce hard questions.

Trajectory generation: a stronger teacher model produces multiple trajectories per question; filter by answer match or format reward and keep the successful ones for SFT.

The cost: SFT is bounded by the teacher, and teacher trajectories may carry a style the student cannot imitate.

Stage 3: reinforcement learning

For short, verifiable answers, RLVR works directly: the policy samples several trajectories, an exact match or a model judge scores them, and GRPO updates the policy.

Search-R1 was an early representative of this route. The essential difference from SFT is that training data comes from the policy's own attempts, so the model experiences failure and learns to recover.

The lecture stresses two stabilizers used in practice: clipping to limit the size of each update, and a KL penalty keeping the model close to the initial policy. Both matter at long horizons and large scale.

Engineering to make training feasible

• Run rollouts asynchronously: while one trajectory waits on a search API, others keep generating.
• Cache search results.
• Wrap APIs robustly.
• Use a local index to simulate external search and reduce dependence on live APIs.

Open-ended tasks: evolving rubrics

Exact match works for short answers, not for long reports. The speaker's group turns rubric evaluation into a training reward and lets the rubrics evolve with the policy:

1. Start with rubrics grounded in external retrieval.
2. For each question, sample several trajectories from the policy (eight in the final training).
3. Use a language model to contrast the trajectories and generate new positive rubrics (reward desirable behavior) and negative rubrics (penalize specific errors).
4. Maintain a rubric buffer, score responses with it, and update the policy.

The core idea: feedback is not fixed at the start of training; it develops alongside the model. As the policy learns new distinctions, the rubrics can capture them.

What the ablations show

• SFT is needed to start. RL from the base model reaches only about 40% at 600 steps; using just 5% of the SFT data adds about ten points.
• Random rewards do nothing. Online RL with random reward shows no improvement.
• Grounded rubrics matter. Initial rubrics supported by search evidence beat purely model-generated ones, which can even hurt.
• Better rubric generators help. GPT-4.1 rubrics outperform Qwen rubrics by about ten points at that time. Open rubric models are emerging, but they are mostly evaluated on short-answer reward benchmarks; replacing a strong generator remains open.
• Evolving rubrics pay off over longer training. The longer the run, the larger the gain from updating rubrics.

The resulting models approach or exceed proprietary deep research systems at a small fraction of the inference cost — hundreds to a thousand times less in the reported comparison.

Retrieval is not background plumbing

If retrieval cannot find the right documents, no agent can compensate.

Local corpora versus external APIs

• When the target corpus is known (Wikipedia, say), build a local index with BM25 or neural embeddings; Search-R1 used the 2018 Wikipedia dump.
• Open-ended questions need web search APIs, browsing APIs, or paper search (Semantic Scholar, Google Scholar).
• Domain tasks such as finance should build a domain corpus or use domain-specific APIs.

Embedding basics

Offline, a document encoder turns the corpus into a vector database. At inference, encode the query and run nearest-neighbor search.

Training typically uses contrastive learning: each query has a positive document and a set of negatives, and the objective raises the positive's score relative to the negatives. Negatives can come from other questions' positives in the same batch (in-batch negatives), or from harder candidates mined with BM25.

Retrieval for agents: the last query is not enough

Standard embedding models are optimized for a human typing one query, but an agent has a rich history: the original task, earlier queries, what it read, and its current reasoning.

The speaker's AgentIO passes the current reasoning text together with the query into the retrieval model and trains briefly on that format. Results:

• Adding reasoning to the query alone helps.
• Training a reasoning-aware embedding model helps more.
• A 4B model beats an 8B baseline while issuing fewer searches.
• Passing the entire trajectory hurts, because the context becomes too long; the single reasoning step immediately before the query works best.

Open problems

• Evaluation is the current bottleneck. Knowing how to evaluate is what enables data construction, rejection sampling, and reward design.
• Rubric bias. Rubric judges may favor longer answers, and rubrics themselves may drift from real user needs.
• Question realism. BrowseComp-style items are built backward from clues and may not represent real information needs.
• External validity. Benchmarks score the final answer, not whether the research helps someone make a better decision or fix a bug.
• Contamination. Models may memorize training answers; evaluation must also block retrieval of the original dataset.
• Small-model limits. On benchmarks requiring many search calls and long context management, small models still trail frontier models; the gap is smaller on open-ended synthesis.

Takeaways

• Deep research addresses information needs with no single source, through repeated planning, retrieval, reflection, and synthesis.
• Evaluation splits into verifiable short answers, expert domains, open-ended long-form (rubrics), and citation quality.
• Training has three stages — mid-training, SFT, RL — and the SFT cold start is essential for RL to take off.
• Open-ended training uses evolving rubrics as rewards, and rubrics must be grounded in external evidence.
• Retrieval must be adapted for agents: reasoning context helps, but only in the right amount.

Check your understanding

1. Why does BrowseComp construct questions backward from answers, and what transfer risk does that create for training data?

Reference answer: The construction starts from a known answer (a person, event or work), collects several clues, and combines them into a complex question. That buys three things: the answer is definite and unique, so exact-match scoring is automatic; the question requires retrieval across multiple sources, which matches deep research; and a validation step — the base model or five Google searches cannot answer it — filters out questions solvable from memory or a single lookup. The transfer risks are twofold. First, question realism: such items are assembled from clues and need not reflect real user information needs. Second, training data built the same way from a knowledge graph (random walks over entities and relations producing multi-hop questions) is effectively reverse-engineering the benchmark; whether it transfers to other benchmarks is open, and data contamination remains a risk.

2. Why do reference-based similarity metrics fail on open-ended long-form answers?

Reference answer: Three reasons from the lecture. First, there are infinitely many valid phrasings, so reference answers cannot be enumerated. Second, surface overlap does not imply correctness — text heavily overlapping a reference can still reverse its core claim. Third, empirically BLEU, ROUGE and BERTScore agree with expert preference only 57–62%, barely above chance. Open-ended long-form answers therefore need rubrics that judge each criterion, not similarity.

3. Where is the main cost of scaling rubric evaluation, and what kinds of errors do auto-generated rubrics make?

Reference answer: The main cost is producing the rubrics themselves: a question may take a PhD-level expert nearly an hour, and only once rubric quality is high enough can models take over scoring — expert–expert agreement is about 80% and model–expert about 79%. ResearchQA generated 21k questions and 120k rubrics automatically from survey papers, but expert auditing found about 15% were wrong: citing papers that do not exist, duplicated questions, or rubrics so vague they cannot separate good from bad answers. The conclusion: model-generated rubrics must be grounded in external evidence, or at least human spot-checked.

4. Why do evolving rubrics outperform fixed rubrics over long training runs?

Reference answer: A fixed rubric only encodes the distinctions known at the start; as the policy improves it produces new correct behaviours and new failure modes that the fixed rubric cannot see. Evolving rubrics sample several rollouts per question each round (8 at final training), have a language model compare them, and generate new positive rubrics that reward good behaviour and negative rubrics that punish specific mistakes, storing them in a buffer. The ablation shows gains from rubric updates grow the longer training runs — the feedback develops together with the model rather than being frozen at the start.

5. Why does feeding the whole trajectory to an embedding model hurt retrieval?

Reference answer: AgentIO's comparisons show that prepending the current reasoning text to the query helps, training a reasoning-aware embedding model helps more (a 4B model beat an 8B baseline while making fewer search calls), but feeding the entire trajectory makes things worse. The reason is length: the original task, all previous queries and all retrieved results dilute the signal of the current query and exceed what the embedding model handles well. The most effective input is only the reasoning step immediately preceding the current query, encoded together with the query.

How this connects to the course

The previous lecture gave policy gradients and GRPO; this one applies them to both verifiable and open-ended deep research. The next lecture goes deeper into advanced RL algorithms and stabilization, and retrieval connects back to the memory and skills lecture.