一个文本框背后的流水线

屏幕上只有一个输入框，输入任意文字、按回车，就有文字流回来。这一讲要回答三个问题：应该往框里放什么、回来的文字是怎么生成的、你究竟在跟什么东西对话。

Karpathy 把整条流水线拆成三个阶段，并且刻意保持在普通读者能跟上的层次：预训练（pre-training）→ 监督微调（SFT）→ 强化学习（RL）。前两个阶段已经标准化多年，第三个阶段仍在快速演进。

时间轴

第一阶段：预训练

数据不是「整个互联网」

预训练的第一步是下载并清洗互联网。以 FineWeb 为例，这条流水线依次做：

1. URL 过滤：剔除恶意软件、垃圾营销、成人内容等域名；
2. 正文抽取：原始页面是带 CSS 与导航栏的 HTML，要抽出真正的正文；
3. 语言过滤：用分类器判断页面语言，例如只保留英文占比超过 65% 的页面；
4. 去重与去除个人身份信息（地址、社保号等）。

Common Crawl 自 2007 年起抓取网页，到 2024 年已索引约 27 亿个页面。清洗完之后，FineWeb 大约只有 44 TB 文本、约 15 万亿 Token——听起来很大，但已经能装进一块硬盘。

这里有一个设计取舍值得注意：语言过滤留下多少比例的其他语言，直接决定模型未来的多语言能力。FineWeb 偏重英文，因此用它训练的模型英文会很好，其他语言则未必。

从文本到 Token

神经网络只能处理一维的符号序列，且词表必须有限。原始 UTF-8 是 0/1 的极长序列，效率太低；把 8 位一组变成字节，序列缩短 8 倍，词表变成 256；再用 BPE（字节对编码） 反复把高频相邻符号合并成新符号，最终得到一个约 10 万 的词表。

GPT-4 的词表是 100,277 个 Token。hello world 恰好是两个 Token（hello 与 space world）；大小写、空格数量都会改变切分结果。

关键心智模型：Token 是模型世界里的原子。它看不到字母，只看到这些块。后面关于拼写、计数的失败，根源都在这里。

训练：预测下一个 Token

从 Token 序列里随机截取长度不超过上限的窗口（实践中常见 4K、8K、16K），喂给网络，让它输出「下一个 Token 的概率分布」。因为词表有 10 万个 Token，输出就是 10 万个数字。

训练开始时参数是随机的，预测也近乎随机。我们用数据里的真实下一个 Token 作为标签，调整参数，让正确答案的概率略微上升。这个过程对整个数据集里的每一个位置同时进行，批次很大——这就是训练。

网络内部是一个巨大的数学表达式，参数以十亿计。现代生产模型使用 Transformer 结构，里面是层归一化、矩阵乘法、softmax、注意力块与多层感知机。Karpathy 强调：把它想成「有 850 亿个旋钮的固定函数」比想成生物神经元更准确——它是无状态的，同样的输入永远给同样的输出分布。

推理：从分布里采样

训练结束后参数固定，进入推理。给定一段前缀，网络给出下一个 Token 的概率分布；按分布采样一个 Token，追加到序列末尾，再重复。

由于是采样，同样的输入每次结果都不同。这也是为什么模型不会逐字复述训练数据：每一步都可能走到另一条分支，很快生成出统计性质相似、但并非原文的文本。

GPT-2 与 Llama 3.1：两个时代的切片

GPT-2（2019）是第一个「现代结构齐全」的模型：1.6B 参数、1024 上下文、约 1000 亿训练 Token。2019 年训练成本估计约 4 万美元；Karpathy 复现时用 8 张 H100 跑一天约 600 美元，优化后可能压到 100 美元。成本下降来自更好的数据、更快的硬件、更好的软件。

Llama 3.1 是另一个极端：405B 参数、15 万亿 Token。两者结构同源，只是规模和工程强度不同。

一个模型发布通常包含两样东西：描述前向计算的代码（几百行）与训练好的参数（一长串数字）。真正的价值在参数里。

基础模型不是助手

训练结束得到的是 base model：一个互联网文本模拟器。问它「2+2 等于几」，它不会像助手那样回答，而是像续写网页一样往下编——可能顺势写出答案，也可能拐进哲学讨论。它对同一个前缀每次给出不同续写，因为采样是随机的。

但它并非无用。为了预测下一个 Token，它把大量世界知识压进了参数：

• 让它列「巴黎必看地标」，能给出大致合理的清单；
• 把维基百科「斑马」词条的第一句喂进去，它会逐字背出后续大段内容——这叫复述（regurgitation），因为高质量来源在训练中被反复采样；
• 给它 2024 年大选的开头，它会编出「彭斯是竞选搭档、对手是希拉里」这类平行宇宙——知识截止在 2023 年底，它只能靠参数猜。

后两条正是幻觉的来源：训练数据里「谁是谁」的问题总是被自信地回答，模型学会了这个格式，即使它并不知道答案。

基础模型也能被「诱导」出助手行为：

• 少样本提示：给十组「英文 → 韩文」对照，再给一个新词，模型会接着翻译；
• 角色扮演：构造一段「人类 / 助手」的对话文本，让模型续写助手那一侧。

这叫上下文学习（in-context learning）：模型在读取提示的过程中就地学会了任务模式。

第二阶段：后训练

用对话数据「编程」人格

预训练得到的是文本续写器，我们想要的是能问答的助手。做法在算法上与预训练完全相同，只换数据集：把互联网文档换成对话。

对话由人类标注员撰写：给出一个场景，写出「理想的助手回答」。标注员依据公司编写的标注手册工作，高层原则通常是 有用、真实、无害。InstructGPT（2022）是这条路线最早的公开描述；今天的数据集动辄上百万条对话，大量由模型合成、再经人工编辑。

训练时长差别巨大：预训练可能在数千台机器上跑三个月，后训练可能只要三小时——因为对话数据比互联网小得多。

对话如何变成 Token

对话要先编码成一维 Token 序列。GPT-4o 使用形如 im_start、im_end 的特殊 Token 标记每一轮的开始、结束与角色（user / assistant），中间夹着正文。特殊 Token 是后训练阶段新引入的，预训练时从未见过。

推理时的流程是：客户端拼好「系统消息 + 历史对话 + 新一轮用户输入 + 助手回合开始标记」，模型从这里开始采样。

你其实在跟「标注员的统计模拟」对话

问「推荐巴黎五个地标」，得到的不是某个神秘 AI 上网调研后的结论，而是对标注员行为的统计模拟：如果这个问题出现在训练集里，回答会非常接近标注员当时写下的内容；如果不在，模型会结合预训练知识与后训练风格，生成一个「像标注员会写的东西」。

Karpathy 的总结很直接：你面对的是一个被标注手册塑造出来的平均人格，而不是一个持续存在的智能体。

模型心理学：四个可观察现象

幻觉与两种缓解

问「Orson Kovats 是谁」——一个不存在的名字。旧模型（如 Falcon 7B）会编出「美国科幻作家」「1950 年代电视剧角色」「前小联盟棒球手」等答案，因为训练集里「某某是谁」总是被自信回答。

缓解一：让模型知道自己不知道。Meta 在 Llama 3 的做法是，从训练文档里用 LLM 生成问答对，再反复询问模型这些问题；如果模型答错（比如把斯坦利杯次数答成 4 次而非 2 次），就把「我不知道」作为该问题的标准答案加进训练集。这样模型学会把内部的「不确定」信号与「说出来」连起来。

缓解二：给工具。引入 search_start / search_end 之类特殊 Token，模型发出搜索请求后，应用程序暂停采样、去搜索引擎取回文本、把结果塞进上下文窗口。此时知识不再是参数里的模糊记忆，而是眼前的原文。

参数是记忆，上下文是工作记忆

这是全片最重要的类比：

• 参数里的知识 = 你一个月前读过的内容，模糊、有损、按频率加权；
• 上下文窗口里的 Token = 你此刻正看着的原文，清晰、可直接引用。

所以让模型总结《傲慢与偏见》第一章，把原文贴进去的效果，明显好于只让它「回忆」。人也一样：刚重读过一章，总结质量会更高。

自我认知是「贴上去的」

模型没有持续存在的自我：每次对话启动、处理 Token、结束、清空。问「你是谁、谁造了你」，它给的答案只是统计猜测——Falcon 曾自称「由 OpenAI 基于 GPT-3 构建」。

想让模型正确自我介绍，有两种工程手段：在后训练数据里加几百条硬编码问答（如 OLMo 的 olmo_hardcoded），或在系统消息里写明身份。两者都是「外挂」，不是模型内在的自我。

模型需要 Token 才能思考

每生成一个 Token，网络只做固定层数的计算，计算量小且有限。因此不能指望它在单个 Token 里完成复杂推理。

经典对照：解「3 个苹果 + 2 个橙子，橙子每个 2 美元，总价 13 美元，苹果多少钱」。

• 差的标注：直接写「答案是 3」，把全部计算压进一个 Token；
• 好的标注：写出「橙子共 4 美元，13−4=9，9÷3=3」，把计算分散到多个 Token。

前者训练出的模型会在心算上出错；后者让每一步都只做很小的计算，前面的中间结果留在上下文里，后面的步骤可以直接引用。给模型更大的数字、强行要求「一个 Token 内作答」，它就会失败；让它写代码用解释器算，就能做对。

同样的道理解释了计数与拼写：模型看不见字符，只看见 Token。「ubiquitous」是 3 个 Token，让它取每第三个字母会出错；「strawberry 里有几个 r」曾经难倒所有模型，因为它同时要求「看见字符」和「计数」。让它调用代码，把字符串交给 Python 处理，就稳定了。

锯齿状智能

模型能解奥数题，却可能答错「9.11 和 9.9 哪个大」。有人分析内部激活，发现在这类输入下与「圣经章节」相关的特征被点亮——9.11 在章节编号里确实排在 9.9 之后。这不是完整解释，但它提醒我们：能力分布是锯齿状的，有大量难以预测的洞。

实践结论：把模型当工具，检查它的输出，不要把它当不会犯错的神。

第三阶段：强化学习

教科书的类比

Karpathy 用上学类比三个阶段：

• 讲解（exposition） → 预训练：大量阅读，建立知识基础；
• 例题与解答（worked solutions） → SFT：模仿专家的解法；
• 课后练习（practice problems） → RL：只给题目与最终答案，自己摸索解法。

问题在于：人类标注员并不知道对模型而言哪条 Token 路径最好。人觉得自然的步骤，模型可能觉得跨度太大；模型觉得简单的步骤，人可能写了半天。要发现「对模型有效的推理路径」，只能让模型自己试。

过程

给一道有可验证答案的题（数学、代码），让模型采样大量解法；用答案检查哪条路径正确；对正确路径做训练，提高未来走这条路的概率。

在 DeepSeek-R1 的公开结果里，随着训练推进：

• 解题准确率持续上升；
• 平均回答长度显著增加；
• 回答里出现「等等，让我重新检查这一步」「换个角度再算一遍」这类自我纠错。

这些思维链不是人写进去的，而是优化过程中涌现的：模型自己发现，反复检查、换视角、回退重来能提高正确率。这种「在脑子里试错」的过程人无法手写进训练集，只能在 RL 中被发现。

推理模型的输出因此很长，也更慢。简单问题（查资料、闲聊）用它属于浪费；难题（数学、代码调试）才值得。

AlphaGo：模仿的天花板

围棋提供了同一个规律的最清晰证据。纯监督学习模仿人类棋谱，水平会上升，但最终停在顶尖人类之下——因为模仿无法超越示范者。加入强化学习后，系统自己与自己对弈，只以胜负为信号，最终超过李世石。

第 37 手是标志性事件：人类棋手评估这一步的概率约为万分之一，事后看却是妙手。RL 不受人类分布约束，可以走到人类从未去过的地方。

在语言模型上，这意味着原则上可以出现人类想不到的类比、策略，甚至非英语的「思考语言」——但前提是有大量多样、可验证的练习题。

RLHF：在无法验证的领域做 RL

写笑话、写诗、总结文章，没有唯一正确答案，无法自动打分。RLHF 的做法是引入间接层：

1. 让人类对同一个提示的多条回答排序（排序比打分容易）；
2. 训练一个奖励模型，学习人类的排序偏好；
3. 用这个奖励模型当作打分器，对语言模型做 RL。

这样不必让人类看十亿个笑话，只需要几千条排序。好处是能把 RL 用到创作类任务，也利用了「判别比生成容易」这一优势。

但奖励模型是人类的有损模拟，而且是一个巨大的神经网络。RL 非常擅长找到它的漏洞：训练几百步后，像「the the the」这种无意义字符串可能拿到接近满分的评分。你把它加入负样本，它又会找到新的漏洞——漏洞是无穷的。

所以 RLHF 的实践是：跑几百步，效果变好，然后停下。「RLHF 不是真正的 RL」——真正的 RL（数学、代码、围棋）奖励函数不可被攻破，可以无限训练；RLHF 的奖励可被攻破，只能小幅微调。

之后会发生什么

• 原生多模态：音频切成频谱片段再量化成 Token，图像切成小块量化成 Token，与文本一起进同一个序列、同一个模型。底层机制不变，只是多了几套词表。
• Agent：从「一次一个任务」走向「长时间运行、人类监督」；工厂里讲人机比，数字世界会出现「人 : Agent」比。
• 计算机使用：模型代替人操作键鼠。
• 测试时训练：目前部署后参数冻结，模型只能靠上下文学习；真正的「运行中学习」仍是开放问题。上下文窗口是有限资源，长时多模态任务会逼出新方法。

本节结论

• 预训练 = 清洗互联网 → 分词 → 预测下一个 Token，产出基础模型（互联网文本模拟器）。
• 后训练 = 换用对话数据继续训练，用标注示例「编程」出助手人格；算法不变，只换数据。
• 参数是模糊记忆，上下文是工作记忆；把材料放进上下文，永远优于让模型回忆。
• 每个 Token 的计算量有限，所以推理必须分散到多个 Token；工具（搜索、代码）是对「心算不可靠」的补偿。
• 幻觉可以通过「教它说不知道」与「给它工具」缓解，但不会消失。
• RL 让模型在可验证任务上自己发现推理路径；RLHF 把 RL 扩展到不可验证领域，但奖励模型可被攻击，只能短程训练。
• 能力是锯齿状的：把它当强大的工具，而不是可靠的神。

理解检验

1. 为什么基础模型（base model）不能直接当助手用？它在什么意义上仍然有用？

参考答案：基础模型只被训练来续写互联网文本，它的行为是「给定前缀，采样下一个 Token」，不会遵循指令，也不会稳定地回答问题。问它 2+2，它可能续写出答案，也可能继续编网页内容，而且每次结果不同。它仍然有用，因为为了预测下一个 Token，它把大量世界知识压缩进了参数：可以诱发它列清单、做少样本翻译，甚至在它内部已经具备完成任务所需的知识。后训练要做的，是把这个「文本模拟器」塑造成「助手人格」。

2. 用「参数是记忆、上下文是工作记忆」解释：为什么把原文贴进对话，总结质量会明显提高？

参考答案：参数里的知识是训练时压缩后的模糊统计，具体细节会丢失、混淆，且按出现频率加权；上下文窗口里的 Token 是本次请求直接输入的内容，模型每一步都能直接读取，不需要「回忆」。让模型总结一章小说，如果不给原文，它只能依赖模糊记忆，可能编造细节；把原文贴进上下文后，摘要可以直接依据原文生成，准确率和细节密度都会提高。这也解释了为什么长文档问答要把文档放进上下文，而不是指望模型「记得」。

3. 为什么训练数据里「直接写答案」的解法会损害模型的数学能力？

参考答案：模型每生成一个 Token 只能做固定且有限的计算。如果标注是「答案是 3」，模型必须在这一个 Token 里完成全部推理，计算量不够，训练会把「猜答案」当成正确策略。好的标注把计算拆成多步：先算橙子总价，再算剩余，再除以数量，每一步只做很小的运算，前一步结果留在上下文里供后一步使用。这样模型学到的是「分散计算」，在测试时也更容易复现正确路径。

4. 幻觉的根源是什么？「教模型说我也不知道」为什么能起作用？

参考答案：根源在于训练数据的格式：对于「某某是谁」这类问题，标准回答总是自信而完整，模型学会了模仿这种语气，即使它内部并不知道答案，也会按最可能的续写编出一个答案。缓解办法是先探测模型的知识边界（例如用文档生成问答、反复询问并比对答案），再把「我不知道」作为那些问题的标准答案加入训练集。这样模型有机会把内部的「不确定」表征与「口头承认不知道」关联起来。它依赖的前提是：模型内部确实存在某种不确定性信号，只是原本没有被「接线」到输出上。

5. 为什么 RLHF 只能训练几百步，而数学题上的 RL 可以训练很久？

参考答案：数学题的奖励是「答案对不对」，判定简单且不可被绕过；模型只能通过真的算对来获得奖励。RLHF 的奖励是一个神经网络（奖励模型），它只是人类偏好的有损模拟，且参数量巨大，存在大量在训练分布之外的「高分区」。RL 会找到这些区域，生成对人类毫无意义但得分极高的输出。你修补一个漏洞，它还会找到新的，因为可攻击的输入空间近乎无限。因此 RLHF 通常只跑几百步，在效果开始变好、尚未明显退化时停止。根本区别是：奖励函数是否可被攻破。

6. 模型为什么会答错「9.11 和 9.9 哪个大」，却能解奥数题？这对使用有什么启示？

参考答案：每个 Token 的计算量有限，且模型的内部特征会被训练数据里的模式干扰。有分析发现，这类输入会激活与「圣经章节编号」相关的特征；在章节编号里 9.11 确实排在 9.9 之后，模型因此给出错误比较。奥数题之所以能解，往往是因为模型在训练中学会了把推理分散到很多 Token，并且有大量相似题型的练习；而两个小数的比较是一个「一步到位」的任务，没有给模型展开计算的机会。启示是：模型能力是锯齿状的，不能从「它会解难题」推出「它不会在简单题上出错」。对关键数字与比较，应要求它写出步骤或调用代码，并自行核对。

与前后内容的关系

这一讲给的是训练侧的完整地图：数据、分词、网络、后训练、强化学习。它解释了模型「会什么、为什么会犯错、为什么需要工具」。与之配套的另一篇《注意力与 KV Cache》讲的是推理侧：一次请求里模型能看到什么、注意力如何被稀释、缓存如何决定速度与成本。把两篇合起来，才能理解 Agent 工程里那些看似琐碎的经验——为什么要精简上下文、为什么要隔离子任务、为什么要把稳定内容放在前面。

The pipeline behind a text box

There is one input box. Type anything, press enter, and words come back. This lecture asks three questions: what should you put in the box, how is the returned text produced, and what exactly are you talking to.

Karpathy splits the pipeline into three stages and deliberately keeps them accessible to a general audience: pretraining → supervised fine-tuning (SFT) → reinforcement learning (RL). The first two have been standard for years; the third is still moving quickly.

Timeline

Stage one: pretraining

The data is not "the whole internet"

Pretraining starts by downloading and cleaning the web. Using FineWeb as a representative example, the pipeline runs through:

1. URL filtering — removing malware, spam, marketing, adult sites and similar domains;
2. Text extraction — raw pages are HTML with CSS and navigation, so the actual article text must be pulled out;
3. Language filtering — a classifier guesses each page's language, keeping, say, pages that are more than 65% English;
4. Deduplication and PII removal — stripping addresses, social security numbers and similar identifiers.

Common Crawl has been crawling since 2007 and had indexed roughly 2.7 billion pages by 2024. After cleaning, FineWeb is about 44 TB of text, around 15 trillion tokens — large, but small enough to fit on a single drive.

One design decision matters for the final product: how much non-English text survives filtering determines the model's multilingual ability. FineWeb leans English, so a model trained on it will be strong in English and weaker elsewhere.

From text to tokens

Neural networks need a one-dimensional sequence of symbols from a finite vocabulary. Raw UTF-8 is a very long sequence of two symbols, which is wasteful. Grouping eight bits into a byte shortens the sequence eightfold and gives 256 symbols. Byte-pair encoding (BPE) then repeatedly merges frequent adjacent symbols into new ones, ending with a vocabulary of about 100,000 tokens.

GPT-4 uses 100,277 tokens. hello world is exactly two tokens (hello and world); capitalization and the number of spaces change the split.

The key mental model: tokens are the atoms of the model's world. It does not see letters. The failures around spelling and counting follow directly from this.

Training: predict the next token

The training loop samples windows of tokens no longer than a chosen maximum (4K, 8K, 16K in practice), feeds them in, and asks for a probability distribution over the next token. With a 100,000-token vocabulary, that is 100,000 numbers.

At initialization the parameters are random and so are the predictions. Using the real next token as a label, the update nudges the correct probability upward and the others downward. This happens for every position in the dataset at once, in large batches. That is training.

Internally the network is a giant mathematical expression with billions of parameters. Production models use the Transformer, built from layer norms, matrix multiplications, softmax, attention blocks and multi-layer perceptrons. Karpathy's framing: think of it as a fixed function with billions of knobs, not as biological neurons — it is stateless; the same input always yields the same output distribution.

Inference: sample from the distribution

Once training ends, parameters are frozen. At inference, a prefix goes in, a distribution comes out, one token is sampled and appended, and the process repeats.

Because sampling is stochastic, the same prompt gives different continuations. This is also why models do not simply regurgitate training text: each step can branch, so the output quickly diverges into something statistically similar but not identical.

GPT-2 and Llama 3.1: two snapshots

GPT-2 (2019) is the first recognizably modern stack: 1.6B parameters, 1024-token context, about 100 billion training tokens. Training was estimated at roughly $40,000 in 2019; Karpathy reproduced it on eight H100s for about$600 in a day, and thinks $100 is achievable today. The drop comes from better data, faster hardware and much better software.

Llama 3.1 sits at the other end: 405B parameters trained on 15 trillion tokens. Same architecture family, very different scale and engineering.

Releasing a model means releasing two things: code describing the forward pass (a few hundred lines) and the trained parameters (a long list of numbers). The value lives in the parameters.

A base model is not an assistant

What comes out of pretraining is a base model: an internet-text simulator. Ask it "what is 2 plus 2" and it will not answer like an assistant; it continues the document. It may land on "4", or wander into philosophy. Repeated sampling gives different continuations.

It is still useful, because predicting the next token forced it to compress a great deal of world knowledge into its weights:

• Ask for a top-ten list of Paris landmarks and it produces something broadly plausible;
• Paste the first sentence of the Wikipedia article on zebras and it recites large parts of the article — regurgitation, because high-quality sources are sampled repeatedly during training;
• Give it the opening of a 2024 election article and it invents "Pence was the running mate and the opponent was Hillary Clinton" — a parallel universe assembled from a knowledge cutoff at the end of 2023.

The last two are the origin of hallucination: in training data, questions of the form "who is X" are always answered confidently, so the model learns the format even when it does not know the fact.

A base model can be coaxed into assistant behavior:

• Few-shot prompting — ten English-to-Korean pairs followed by a new English word; the model continues the translation;
• Role-play prompting — construct a "human / assistant" transcript and let the model continue the assistant side.

This is in-context learning: while reading the prompt, the model infers the task pattern and applies it.

Stage two: post-training

Programming a persona with conversation data

Pretraining yields a text continuer. We want an assistant. The algorithm is identical — only the dataset changes: internet documents are swapped for conversations.

Conversations are written by human labelers: given a context, write the ideal assistant answer. Labelers follow lengthy written guidelines whose high-level principles are usually helpful, truthful, harmless. InstructGPT (2022) was the first public description of this approach; modern datasets contain millions of conversations, much of it synthetic and edited by humans.

The time scales differ enormously: pretraining may run for three months on thousands of machines, while post-training may take three hours, because the conversation dataset is far smaller.

How conversations become tokens

A conversation must be encoded as a one-dimensional token sequence. GPT-4o uses special tokens such as im_start and im_end to mark the beginning and end of each turn and the speaker (user or assistant), with the message text in between. These special tokens are introduced during post-training and were never seen during pretraining.

At inference, the client assembles "system message + history + new user message + assistant turn start" and the model samples from there.

You are talking to a statistical simulation of a labeler

Ask for "five landmarks in Paris" and you are not receiving the output of a magical agent that researched the city. You are receiving a statistical simulation of a human labeler: if that question is in the training set, the answer closely resembles what the labeler wrote; if it is not, the model combines pretraining knowledge with post-training style to produce something the labeler plausibly would have written.

Karpathy's summary is blunt: the thing you talk to is an average persona shaped by a labeling manual, not a persistent agent.

Model psychology: four observable effects

Hallucination and two mitigations

Ask "who is Orson Kovats" — a name that does not exist. An older model such as Falcon 7B will invent an answer: an American science-fiction author, a character from a 1950s TV show, a former minor-league baseball player. The training set taught it that "who is X" gets a confident answer.

Mitigation one: teach the model to say it does not know. Meta's approach for Llama 3 was to generate question-answer pairs from training documents with an LLM, interrogate the model repeatedly, and compare its answers with the truth. If the model answers incorrectly — say, claiming four Stanley Cups instead of two — then "I don't know" is added to the training set as the correct response to that question. This gives the model a chance to connect its internal uncertainty signal to an explicit refusal.

Mitigation two: give it tools. Introduce special tokens such as search_start and search_end. When the model emits them, the application pauses sampling, queries a search engine, and pastes the retrieved text into the context window. The knowledge is no longer a vague recollection in the weights; it is text in front of the model.

Parameters are memory, context is working memory

This is the most important analogy in the lecture:

• Knowledge in the parameters is what you read a month ago: vague, lossy, weighted by frequency;
• Tokens in the context window are what you are looking at right now: clear and directly quotable.

This is why summarising chapter one of Pride and Prejudice works far better when the chapter text is pasted in than when the model is asked to recall it. Humans behave the same way: re-reading a chapter improves the summary.

The sense of self is bolted on

The model has no persistent self. Each conversation boots up, processes tokens, and is wiped. Ask "who are you and who built you" and you get a statistical guess — Falcon once claimed to have been built by OpenAI on GPT-3.

To make the model describe itself correctly, engineers add a few hundred hard-coded Q&A pairs to the post-training data (OLMo's olmo_hardcoded, for instance), or state the identity in a system message. Both are external scaffolding, not an inner self.

Models need tokens to think

For every token, the network performs a fixed, small amount of computation. You cannot expect it to complete a complex derivation inside a single token.

The canonical contrast: "Emily buys three apples and two oranges, each orange costs $2, the total is$13, what does each apple cost?"

• Bad label: "The answer is 3", squeezing all the work into one token;
• Good label: "Oranges cost $4; 13 − 4 = 9; 9 ÷ 3 = 3", spreading the work across tokens.

The first teaches the model to guess; the second teaches it to compute step by step, with each intermediate result available in context for the next step. Give the model larger numbers or insist on a one-token answer and it fails; ask it to write code and use the interpreter and it succeeds.

The same reasoning explains counting and spelling. The model does not see characters, only tokens. "ubiquitous" is three tokens, so taking every third character fails; "how many r's in strawberry" once defeated every frontier model because it requires both seeing characters and counting them. Calling Python solves it: the string is copied into code and Python does the counting.

Jagged intelligence

A model can solve Olympiad problems and still get "is 9.11 bigger than 9.9" wrong. Analysis of internal activations found features associated with Bible verses lighting up — in chapter numbering, 9.11 does come after 9.9. That is not a complete explanation, but it shows the capability surface is jagged, full of holes that are hard to predict.

The practical conclusion: treat the model as a tool, check its output, and do not treat it as infallible.

Stage three: reinforcement learning

The textbook analogy

Karpathy frames the three stages through schooling:

• Exposition → pretraining: read widely and build a knowledge base;
• Worked solutions → SFT: imitate experts;
• Practice problems → RL: given only problems and final answers, discover your own solution method.

The catch is that human labelers do not know which token path is best for the model. Steps that feel natural to a person may be too large a leap for the network, and steps that are trivial for the network may waste tokens. Discovering a reasoning path that works for the model can only be done by the model itself.

The loop

Give a problem with a verifiable answer (math, code). Sample many solutions. Score them against the answer. Train on the correct ones to make those paths more likely in future.

In DeepSeek-R1's published results, as training proceeds:

• accuracy climbs steadily;
• average response length grows substantially;
• responses begin to contain "wait, let me re-check this step" and "let me try a different approach".

These chains of thought were not written by humans. They emerged from optimisation: the model discovered that re-checking, reframing and backtracking improve accuracy. A human labeler could not have hard-coded this into the ideal answer; only RL finds it.

Reasoning models are therefore slower and more verbose. They are wasted on simple lookups and chit-chat, and valuable on hard math and debugging.

AlphaGo: the ceiling of imitation

Go provides the clearest evidence for the same pattern. Pure supervised learning — imitating human game records — improves and then plateaus below the strongest humans, because imitation cannot exceed the demonstrator. With reinforcement learning, the system plays itself and uses only win/loss as the signal, eventually surpassing Lee Sedol.

Move 37 is the emblem: human experts estimated its probability at roughly one in ten thousand, yet it turned out to be brilliant. RL is not constrained to the human distribution and can go where no human has gone.

For language models this implies, in principle, analogies no human has made, strategies no human has tried, even a thinking language that is not English. But all of it depends on having a large, diverse set of verifiable practice problems.

RLHF: doing RL where answers are not verifiable

Jokes, poems and summaries have no single correct answer, so automatic scoring fails. RLHF inserts an indirection:

1. Humans rank several responses to the same prompt (ranking is easier than scoring);
2. A reward model is trained to reproduce those rankings;
3. The language model is trained with RL against the reward model.

This avoids asking humans to evaluate a billion jokes; a few thousand rankings suffice. It also exploits the fact that discrimination is easier than generation.

But the reward model is a lossy simulation of human judgment, and a very large neural network. RL is excellent at finding its blind spots. After a few hundred steps, a string like "the the the" can receive a near-perfect score. Add it as a negative example, and the optimiser finds another hole. There are infinitely many.

So in practice RLHF runs for a few hundred steps, improves the model, and stops. "RLHF is not real RL": in math, code and Go the reward cannot be gamed and training can continue indefinitely; with a learned reward model it can, so the process must stay short.

What comes next

• Native multimodality — audio is split into spectrogram chunks and quantised into tokens; images are split into patches and quantised; both join text in one sequence and one model. The machinery is unchanged; only the vocabulary grows.
• Agents — from one task at a time toward long-running work under human supervision. Factories talk about human-to-robot ratios; the digital world will talk about human-to-agent ratios.
• Computer use — models operating a keyboard and mouse on your behalf.
• Test-time training — today parameters are frozen after deployment and the model can only learn in context. Learning while running remains open, and long multimodal tasks will force new ideas because the context window is finite.

Takeaways

• Pretraining = clean the web → tokenize → predict the next token; the output is a base model, an internet-text simulator.
• Post-training = keep training on conversations to program an assistant persona; same algorithm, different data.
• Parameters are vague memory; context is working memory. Putting material in context always beats asking the model to recall it.
• Per-token computation is limited, so reasoning must be spread across tokens; tools such as search and code compensate for unreliable mental arithmetic.
• Hallucination can be reduced by teaching refusal and by adding tools, but it does not disappear.
• RL lets models discover reasoning paths on verifiable tasks; RLHF extends RL to unverifiable domains, but its reward model is gameable, so training must stay short.
• Capability is jagged: use the model as a powerful tool, not as an infallible oracle.

Comprehension check

1. Why can't a base model be used as an assistant directly, and in what sense is it still useful?

Reference answer: A base model is trained only to continue internet documents. Given a prefix it samples the next token; it does not follow instructions, and it does not reliably answer questions. Ask it 2+2 and it may produce the answer, or keep writing a web page, differently each time. It is still useful because next-token prediction forces it to compress large amounts of world knowledge into its parameters: it can be induced to list landmarks, translate from a few examples, and it already contains much of what a task needs. Post-training exists to turn this text simulator into an assistant persona.

2. Using "parameters are memory, context is working memory", explain why pasting a chapter improves a summary.

Reference answer: Knowledge stored in parameters is a lossy, frequency-weighted compression of training data; specific details are blurred or confused, and the model must "recall" them. Tokens in the context window are part of the current request and can be read directly at every step. If the chapter is pasted in, the summary can be grounded in the actual text, so accuracy and detail improve. Without it, the model relies on vague memory and may invent details. This is also why document question answering puts the document in context instead of hoping the model remembers it.

3. Why does training on "answer only" solutions harm a model's math ability?

Reference answer: For each generated token the network performs only a fixed, limited computation. If the label is "the answer is 3", the model must complete the whole derivation inside one token. There is not enough compute, so training rewards guessing. A good label splits the work: compute the orange cost, subtract, divide, each step small, with earlier results left in context for later steps. The model learns to distribute computation and can reproduce the correct path at test time.

4. What causes hallucination, and why does teaching the model to say "I don't know" help?

Reference answer: The cause is the format of the training data. For questions of the form "who is X", the standard answer is confident and complete, so the model imitates that tone even when it has no knowledge, producing the most likely continuation. The mitigation is to probe the model's knowledge boundary — generate Q&A from documents, ask repeatedly, compare with the truth — and then add "I don't know" as the correct answer for questions the model gets wrong. This links an internal uncertainty representation to an explicit refusal. It works only if such an internal signal exists and merely lacks a route to the output.

5. Why can RLHF only run for a few hundred steps while RL on math can run much longer?

Reference answer: For math the reward is correctness, which is simple and cannot be bypassed; the only way to earn it is to actually solve the problem. For RLHF the reward is a neural network — a lossy simulation of human preference — with enormous capacity and regions outside the training distribution where the score is unrealistically high. RL finds those regions and produces outputs that humans find meaningless but the reward model scores highly. Fix one hole and the optimiser finds another, because the space of adversarial inputs is effectively infinite. So RLHF is run briefly, stopped while the model is still improving. The underlying distinction is whether the reward can be gamed.

6. Why does the model fail at "is 9.11 bigger than 9.9" while solving Olympiad problems, and what does that imply for use?

Reference answer: Per-token computation is limited, and internal features are shaped by patterns in the training data. Analyses found features associated with Bible chapter numbering activating on such inputs; in chapter numbering 9.11 follows 9.9, which pushes the model toward the wrong comparison. Olympiad problems work because the model has learned to spread reasoning over many tokens and has seen many similar problems; comparing two decimals is a one-step task with no room to unfold. The implication is that capability is jagged: solving hard problems does not guarantee correctness on easy ones. For critical numbers or comparisons, require explicit steps or code execution and verify the result.

How this connects

This lecture maps the training side: data, tokenization, the network, post-training and reinforcement learning. It explains what a model can do, why it errs, and why it needs tools. Its companion note on attention and KV Cache covers the inference side: what the model can see in one request, how attention is diluted, and how caching determines speed and cost. Together they explain the practical rules of agent engineering — keep context lean, isolate subtasks, and keep the stable prefix stable.