强化学习与语言模型：2025 年的进展、智能体与反馈循环

好的，我又请来了我的朋友 Sholto Bricken 和 Trenton Douglas。等等，我上次是这么介绍的吗？不，不，你给我们起了不同的名字，但我们之前没有 Sholto Bricken 和 Trenton Douglas。是 Sholto Douglas 和 Trenton Bricken。嗯，他们现在都在 Anthropic。Sholto 在做强化学习的 Scaling，Trenton 还在研究机制可解释性。欢迎回来。

Okay, I'm joined again by my friends Sholto Bricken and Trenton Douglas. Wait, did I do this last? No, no, you named us differently, but we didn't have Sholto Bricken and Trenton Douglas. Sholto Douglas and Trenton Bricken. Um, who are now both at Anthropic. Sholto scaling RL. Trenton still working on mechanistic interpretability. Welcome back.

很高兴来到这里。

Happy to be here.

是啊，很有趣。

Yeah, it's fun.

从去年到现在有什么变化？我们上次聊差不多是 2024 年的这个月。现在已经是 2025 年了。发生了什么？

What's changed since last year? We talked basically this month in 2024. Now we're in 2025. What's happened?

好的，我认为最大的变化是强化学习和语言模型终于取得了成功。这体现在我们终于有了一个算法的证明，在正确的反馈循环下，它能达到专家级人类的可靠性和性能。我认为这目前只在竞技编程和数学领域得到了确凿的证明。所以如果你考虑两个维度，一个是任务的智力复杂度，另一个是任务完成的时间跨度。我认为我们有证据表明，我们可以在多个维度上达到智力复杂度的顶峰。我们还没有展示出长时间运行的智能体式性能，现在你看到的是最初蹒跚学步的阶段，到今年年底应该会看到更确凿的证据，比如真正的软件工程智能体做实际工作。Trenton，你现在正在做这方面的实验，对吧？

Okay, so I think the biggest thing that's changed is RL and language models have finally worked. And this is manifested in we finally have proof of an algorithm that can give us expert human reliability and performance given the right feedback loop. And so I think this is only really being conclusively demonstrated in competitive programming and math basically. And so if you think of these two axes, one is the intellectual complexity of the task and the other is the time horizon of which the task is being completed on. And I think we have proof that we can reach the peaks of intellectual complexity along many dimensions. We haven't yet demonstrated long-running agentic performance, and you're seeing the first stumbling steps of that now, and should see much more conclusive evidence of that basically by the end of the year with real software engineering agents doing real work. And I think Trenton, you're experimenting with this at the moment, right?

是的，完全正确。目前最公开的例子就是 Claude 玩宝可梦，对吧？看着它挣扎的样子有点不忍直视，但每一代模型都能在游戏中走得更远。这看起来更像是它使用记忆系统的限制，而不是其他原因。

Yeah, absolutely. I mean the most public example people could go to today is Claude plays Pokemon, right? And seeing it struggle in a way that's kind of painful to watch, but each model generation gets further through the game. And it seems more like a limitation of it being able to use memory system than anything else.

是啊。真希望我们去年录了预测。今年一定要录。哦，对。让我们负责。没错。你去年会预测智能体只有现在这么强吗？

Yeah. I wish we had recorded predictions last year. We definitely should this year. Oh, yeah. Hold us accountable. Yeah, that's right. Would you have said that agents would be only this powerful as of last year?

我认为这大致符合我对软件工程的预期。我原本期望它们在计算机使用方面能更好一些。但我理解所有原因，而且我认为这个问题正在顺利解决中。这只是暂时的不足。至于明年的预测，我确实认为到今年年底，也就是明年这个时候，我们会有软件工程智能体，能完成接近初级工程师一天的工作量，或者几个小时的相当独立的胜任工作。

I think this is roughly on track for where I expected with software engineering. I think I expected them to be a little bit better at computer use. But I understand all the reasons for why that is, and I think that's well on track to be solved. It's just a sort of temporary lapse. And holding accountable for my predictions next year, I really do think end of this year, sort of like this time next year, we have software engineering agents that can do close to a day's worth of work for like a junior engineer or like a couple of hours of quite competent independent work.

是的，我觉得没错。不过我认为分布很不均匀，对于某些任务，比如样板网站代码这类东西，它能快速搞定，为你节省一整天。

Yeah, that seems right to me. I think the distribution is pretty wonky though, where for some tasks like boilerplate website code, these sorts of things, it can bang it out and save you a whole day.

我记得去年你说，阻碍它们的是可靠性的更多个九。我不知道你是否还会用这种方式来描述这些软件智能体无法完成一整天的工作，但能帮你解决几分钟的事情。真的是更多个九在阻碍，还是别的什么？

I think last year you said that the thing that was holding them back was the extra nines of reliability. I don't know if that's the way you'd still describe the way in which these software agents aren't able to do a full day of work but are able to help you out with a couple minutes. Is it the extra nines that's really stopping you or is it something else?

是的，我认为我当时的描述回顾起来可能不是限制它们的因素。我认为我们现在看到的问题更接近于缺乏上下文，缺乏进行复杂、多文件更改的能力，以及某种程度上任务的范围。比如，在一个聚焦的上下文中处理一个范围明确的问题，它们可以应对高智力复杂度。但当事情有点模糊，或者需要大量探索和与环境迭代时，它们就更挣扎。所以现在我会这样定义：阻碍它们的是，如果你能为想让它做的事情提供一个好的反馈循环，那么它就表现良好，相当擅长。如果不能，它们就会有点挣扎。

Yeah, I think my description there was probably not what's limiting them in retrospect. I think what we're seeing now is closer to lack of context, lack of ability to do complex, very multi-file changes, and maybe scope of the change or scope of the task in some respects. Like you can cope with high intellectual complexity in a focused context with a very scoped problem. But when something's a bit more amorphous or requires a lot of discovery and iteration with the environment, they struggle more. And so maybe the way I would define it now is the thing that's holding them back is if you can give it a good feedback loop for the thing that you want it to do, then it's good. It's pretty good at it. If you can't, then they struggle a bit.

你能为听众多解释一下你所说的反馈循环是什么意思吗？如果他们不了解强化学习等是怎么回事。

Can you, for the audience, say more about what you mean by this feedback loop? If they're not aware of what's happening with RL and so forth.

是的。去年真正取得重大进展的领域，广义上称为基于可验证奖励的强化学习，其中有一个干净的奖励信号。你知道，语言模型最初的突破来自基于人类反馈的强化学习，通常是成对反馈之类，模型的输出越来越接近人类想要的。但这并不一定提高它们在任意难度问题域上的表现，特别是因为人类实际上并不擅长判断哪个答案更好。人类有长度偏差等问题。所以你需要一个信号来判断模型的输出是否正确，而且这个信号要相当真实。比如数学问题的正确答案或单元测试通过，这些都是非常干净的奖励信号的例子。但即使这些也可能被钻空子。比如单元测试，模型会想办法绕过，注入特定值或硬编码单元测试的值。如果它们能弄清楚实际测试在做什么，比如查看缓存的 Python 文件找到实际测试，它们就会试图绕过。所以这些并不完美，但已经接近多了。

Yes. So the big thing that really worked over the last year is broadly the domain called RL from verifiable rewards or something like this, where a clean reward signal. So you know the initial unhopping of language models was RL from human feedback, where typically it was something like pairwise feedback, and the outputs of the models became closer and closer to things that humans wanted. But this doesn't necessarily improve their performance at any difficulty of problem domain, particularly as humans are actually quite bad judges of what a better answer is. Humans have things like length biases and so forth. So you need a signal of whether the model was correct in its output that is quite true. And so things like the correct answer to a math problem or unit tests passing, these are examples of a reward signal that's very clean. But even these can be hacked by the way. Like even unit tests, the models find ways around it to hack in particular values and hardcode values of unit tests. If they can figure out what the actual test is doing, like if they can look at the cached Python files and find what the actual test is, they'll try and hack their way around it. So these aren't perfect, but they're much closer.

为什么它在软件工程方面比其他所有领域都进步这么多？

And why has it gotten so much better at software engineering than everything else?

部分原因是软件工程非常可验证。这是一个天然适合这种方式领域。我认为，代码是否通过测试？它能否运行？能否编译？能否编译？能否通过测试？你知道，你可以去 LeetCode 运行测试，就知道答案是否正确。但写一篇好文章就没有同样的标准。这需要品味，非常难。就像我们前几天晚餐时讨论的，普利策奖，你知道哪个会先出现，是普利策奖获奖小说还是诺贝尔奖之类的。我实际上认为诺贝尔奖比普利策奖获奖小说更可能，因为在某些方面，赢得诺贝尔奖所需的许多任务，或者至少是强力协助赢得诺贝尔奖的任务，有更多层的可验证性。

In part because software engineering is very verifiable. It's a domain which just naturally lends itself to this way. I think does the code pass a test? Does it even run? Does it compile? Does it compile? Does it pass the test? You know, you can go on LeetCode and run tests and know whether or not you got the right answer. But there isn't the same kind of thing for writing a great essay. That requires taste, which is quite hard. Like we discussed the other night at dinner, the Pulitzer surprise, you know which would come first, a Pulitzer Prize winning novel or a Nobel Prize or something like this. And I actually think a Nobel Prize is more likely than a Pulitzer Prize winning novel, in some respects because a lot of the tasks required in winning a Nobel Prize, or at least strongly assisting in helping it to win a Nobel Prize, have more layers of verifiability built up.

所以我预计它们最初在加速诺贝尔奖级别的工作上，会比写普利策奖级别的小说更快。

So I expect them to accelerate the process of doing Nobel Prize winning work more initially than that of writing Pulitzer Prize worthy novels.

是的，我觉得如果我们倒回 14 个月，上次录制的时候，九重可靠性对我来说是对的。我们没有 Claude Code，没有 Deep Research。我们只是用聊天机器人形式的智能体，复制粘贴、复制粘贴、复制粘贴。完全是这样。而且我认为我们非常习惯聊天界面，无论是发短信还是用谷歌。很难想象智能体实际上可以自己去获取上下文，并将自己的事实存储到记忆系统中。我仍然认为这是九重可靠性，如果你正确地搭建模型或提示它，它可以做比普通用户想象的复杂得多的事情。我的一个朋友，做 Future House 的 Sam Rodriguez，他们发现了一种新药，正在申请专利，等到这期节目播出时，应该已经公开了。那是什么？LSDV2。等等，真的吗？不，不，他们不是在制造 LSD。但人们之前不认为模型可以有创造力或做新科学，对吧？而这看起来就像是一个技能问题。

Yeah, I think if we rewind 14 months to when we recorded last time, the nines of reliability was right to me. We didn't have Claude Code, we didn't have Deep Research. All we did was use agents in a chatbot format, copy paste, copy paste, copy paste. Totally. And I think we're very used to chat interfaces whether we're texting or using Google. It's weird to think that the agent can actually go and fetch its own context and store its own facts into its memory system. I still think that it's the nines of reliability and if you scaffold the model correctly or prompt it, it can do much more sophisticated things than the average user assumes. One of my friends, Sam Rodriguez, who does Future House, they've discovered a new drug that they're in the process of patenting, and by the time this episode comes out, that will be live. What was that? LSDV2. Wait, is it really? No, no, they're not making LSD. But people didn't think that models can be creative or do new science, right? And it does just kind of seem like a skill issue.

我的意思是，有个很酷的……等等，但它发现了一种药。它是怎么做到的？我觉得它是一次性完成的。这只是一次对话，我们需要参考完整的公告。但我的印象是，它能够阅读大量的医学文献，进行头脑风暴，建立新的联系，然后提出湿实验方案，由人类执行。然后通过迭代，他们验证了这种新化合物确实有令人兴奋的效果。我听到的另一个批评是，LLM 无法写出有创意的长篇小说。我知道至少有两个人（可能希望匿名）用 LLM 写了长篇小说。在这两个案例中，他们都非常擅长搭建和提示模型。即使是病毒式传播的 ChatGPT 地理猜谜能力，它能从一张照片中极其准确地判断你在哪个海滩。我认为让这个功能火起来的 Kelsey Piper，他们的提示非常复杂。它很长，鼓励你思考五个不同的假设，给它们分配概率，并推理图像中重要的不同方面。我没有做过 AB 测试，但我认为除非你真正鼓励模型这样深思熟虑，否则你无法达到那种性能水平。

I mean, there was the cool... wait, but it discovered a drug. How did it? I think it one-shotted this. This was just over a conversation, and we'll need to refer to the full announcement. But my impression is that it was able to read a huge amount of medical literature, brainstorm, make new connections, and then propose wet lab experiments that the humans did. Then through iteration, they verified that this new compound does this thing that's really exciting. Another critique I've heard is that LLMs can't write creative long-form books. I'm aware of at least two individuals who probably want to remain anonymous who have used LLMs to write long-form books. In both cases, they're just very good at scaffolding and prompting the model. Even with the viral ChatGPT geoguesser capabilities where it's insanely good at spotting what beach you were on from a photo. Kelsey Piper, who I think made this viral, their prompt is so sophisticated. It's really long and encourages you to think of five different hypotheses and assign probabilities to them and reason through the different aspects of the image that matter. I haven't AB tested it, but I think unless you really encourage the model to be this thoughtful, you wouldn't get the level of performance that you see with that ability.

所以你提到了人们如何约束模型输出以获得分布中好的部分。但我听到的一个关于 RL 的批评——或者说不是关于 RL，而是关于用 o3 等模型成功来表明我们从这些推理模型中获得新能力的批评——是所有这些能力都已经在预训练模型中内置了。我认为有一篇来自 Stingwall 大学的论文，他们表明如果你给基础模型足够多的尝试来回答问题，它仍然可以像推理模型一样好地回答。基本上，它只是回答的概率较低。所以，你是在缩小模型回答问题时所探索的可能性。那么，我们实际上是通过 RL 训练引出了新能力，还是只是给它们戴上了眼罩？就像削去大理石之类的东西。

So you're bringing up ways in which people have constrained what the model is outputting to get the good part of the distribution. But one of the critiques I've heard of RL, or not of RL, but one of the critiques I've heard about using the success of models like o3 to suggest that we're getting new capabilities from these reasoning models is that all these capabilities were already baked in the pre-training model. I think there's a paper from Stingwall University where they showed that if you give a base model enough tries to answer a question, it can still answer the question as well as the reasoning model. Basically, it just has a lower probability of answering. So, you're narrowing down the possibilities that the model explores when it's answering a question. So, are we actually eliciting new capabilities with this RL training or are we just putting the blinders on them? Like carving away the marbles or this.

我认为值得注意的是，那篇论文是基于 Llama 和 Qwen 模型的。我不确定他们用了多少 RL 算力，但我认为它远不及基础模型所用的算力。我认为训练中使用的算力是衡量你向模型添加了多少原始新知识或能力的一个合理指标。我的先验是，至少如果你看看 DeepMind 在 RL 之前的所有研究，RL 能够教会这些围棋和象棋智能体超越人类水平的新知识，仅仅来自 RL 信号，只要 RL 信号足够干净。结构上的限制基本上阻止了它的模仿。

I think it's worth noting that that paper was on the Llama and Qwen models. I'm not sure how much RL compute they used, but I don't think it was anywhere comparable to the amount of compute used in the base models. I think the amount of compute you use in training is a decent proxy for the amount of actual raw new knowledge or capabilities you're adding to a model. My prior, at least if you look at all of DeepMind's research from RL before RL was able to teach these Go and chess playing agents new knowledge in excess of human level performance, just from RL signal provided the RL signal was sufficiently clean. Structurally limiting prevents it from imitating basically.

为什么你们不已经在 RL 上投入更多算力？我认为 Dario 在他的博客文章中说，实验室，或者几个月前关于出口管制的事情，像是‘DeepSeek，不管怎样，我们在 RL 上只花了 100 万美元左右。’所以，我们还没有进入 RL 的算力受限状态，但很快就会了。是的，你在基础模型上花了数亿美元。为什么只在 RL 上花一百万？你知道那个关于选择发射太空任务的寓言，你应该在科技树上走得更远，因为如果你晚点发射，你的飞船会更快，诸如此类。我认为这很相似。你要确保在算法上做对了，然后当你下注并在运行中投入大量算力时，它才会真正有回报。要有正确的算力效率等等。现在我认为 RL 在这方面与预训练略有不同，RL 可以更迭代，你逐步为基础模型增加能力。预训练在很多方面，如果你在运行中途搞砸了，那你就真的搞砸了。但我认为这就是为什么人们还在弄清楚他们到底想做什么的主要原因。我的意思是，从 o1 到 o3，对吧？OpenAI 在他们的博客文章中说，这是 o1 的 10 倍算力乘数。是的。把它推出去。然后他们花了接下来的几个月增加在那上面投入的算力。我预计其他人现在也在扩大 RL 规模。所以我基本上不认为这种情况会持续很久。

Why aren't you already spending more compute on this? I think Dario said in his blog post that labs, or it was a couple months ago on the export controls thing, is like 'DeepSeek, whatever, we're only spending $1 million on RL or something.' So it's like we aren't in the compute limited regime for RL yet, but we will be soon. Yeah, you're spending hundreds of millions on the base model. Why only order a million on the RL? You know the parable about when you choose to launch a space mission and how you should sort of acquire go further up the tech tree because if you launch later, your ship will go faster and this kind of stuff. I think it's quite similar to that. You want to be sure that you've algorithmically got the right thing, and then when you bet and do the large compute spend on the run, then it'll actually pay off. Have the right compute efficiencies and this kind of stuff. Now I think RL is slightly different to pre-training in this regard where RL can be a more iterative thing, you're progressively adding capabilities to the base model. Pre-training has, in many respects, if you're halfway through a run and you've messed it up, then you've really messed it up. But I think that's the main reason why people were still figuring out exactly what they wanted to do. I mean o1 to o3, right? OpenAI put in their blog post that it was a 10x compute multiplier over o1. Yeah. Let's get it out there. And then they spent the next few months increasing the amount of compute that they expend on that. And I expect as everyone else is scaling up RL right now. So I basically don't expect that to be true for very long.

为了读者着想，也许你在预训练和强化学习中都在做梯度下降步骤。只是信号不同。通常在强化学习中，你的奖励更稀疏。所以你采取多个回合。就像‘你赢了棋局没有’是你得到的唯一信号。而且通常你无法通过离散动作计算梯度。所以你最终会损失很多梯度信号。所以你可以假设预训练更高效。但没有理由你不能在强化学习中学习新能力。

Just for the sake of readers, maybe you're doing gradient descent steps in both pre-training and reinforcement learning. It's just the signal is different. Typically in reinforcement learning, your reward is sparser. So you take multiple turns. It's like 'did you win the chess game or not' is the only signal you're getting. And often you can't compute gradients through discrete actions. So you end up losing a lot of gradient signal. So you can presume that pre-training is more efficient. But there's no reason why you couldn't learn new abilities in reinforcement learning.

事实上，你可以用某种奇怪的强化学习变体来替换预训练中的整个下一个词预测任务，然后完全通过强化学习来完成所有学习。

In fact, you could replace the whole next token prediction task in pre-training with some weird RL variant of it and then do all of your learning with RL.

是的。归根结底，就是信号然后据此修正。

Yeah. At the end of the day, just signal and then correcting to it.

完全同意。回到你提到的那篇论文，除了 Sholto 提出的那些注意事项——我认为那是首要且最重要的——我认为聚焦于有意义动作的概率空间，归根结底还是可靠性的问题。经典地说，如果你给猴子一台打字机，最终它们会写出莎士比亚。所以我们关心的任何现实世界任务的动作空间都如此之大，你确实需要让模型专注于做合理的事情。

Totally. And then going back to the paper you mentioned, aside from the caveats that Sholto brings up which I think is the first order most important, I think zeroing in on the probability space of meaningful actions comes back to the issues of reliability. Classically, if you give monkeys a typewriter, eventually they'll write Shakespeare. So the action space for any of these real-world tasks we care about is so large that you really do care about getting the model to zero in on doing the reasonable things.

是的。我的意思是，从某种广义上说，你有了词元，对吧？没错。就像你确实有一只猴子，最终它写出了莎士比亚。

Yeah. I mean to the extent, in some broad sense, that you've got tokens, face, right? Exactly. Like you literally do have a monkey and it's making Shakespeare in the end.

对，对，没错。好的，所以 AlphaGo 的象棋类比很有意思。你刚才想说什么？

Yeah. Yeah. Exactly. Okay. So the AlphaGo chess analogy is interesting. Were you about to say something?

嗯，我正想说，你确实需要有时能获得奖励才能学习。这在 Alpha 变体中某种程度上是复杂的。也许你正要这么说：总有一方赢，所以你总能以某种方式获得奖励信号。但在我们讨论的这些任务中，你需要有时真正成功。幸运的是，语言模型对我们关心的任务有一个很好的先验。如果你看看 2017 年左右的所有旧论文，奖励学习曲线总是平平平平平，因为它们在摸索世界的基本机制，然后有一个尖峰，因为它们学会了利用简单的奖励，然后某种程度上像 S 形曲线，之后无限继续，直到学会完全最大化游戏。我认为语言模型的曲线有点不同，开头没有那个死区，因为它们已经知道如何解决一些基本任务。所以你得到这个初始尖峰，这就是人们说‘哦，你可以从一个例子中学习’时所指的。那个例子只是教你学会回溯和正确格式化答案之类的东西，让你在基于预训练知识的任务上初步获得奖励，然后剩下的可能就是你学习越来越复杂的东西。

Well, I was just going to say, you do need to be able to get reward sometimes in order to learn. And that's the complexity in some respects in the Alpha variants. Maybe you're about to say this: one player always wins, so you always get a reward signal one way or the other. But in the kinds of things we're talking about, you need to actually succeed at your task sometimes. So language models luckily have this wonderful prior over the task that we care about. If you look at all the old papers from like 2017, the reward learning curves always look like flat, flat, flat, flat, flat as they're figuring out basic mechanics of the world, and then there's this spike up as they learn to exploit easy rewards, and then it's sort of like a sigmoid in some respects, and then it continues on indefinitely as it learns to absolutely maximize the game. I think the LM curves look a bit different in that there isn't that dead zone at the beginning because they already know how to solve some of the basic tasks. So you get this initial spike, and that's what people are talking about when they say, 'Oh, you can learn from one example.' That one example is just teaching you to pull out the backtracking and format your answer correctly, stuff that lets you get some reward initially at tasks conditional on your pre-training knowledge, and then the rest probably is you learning more and more complex stuff.

有意思。是的。而且这也会很有趣。我知道有人批评或怀疑强化学习能快速见效，指出 AlphaGo 用了大量算力，尤其是对于一个在 2017 年训练的系统。偏离曲线。完全同意。所以这在很大程度上是因为首先你得有某种具有理性偏好的东西，然后它才能在围棋上达到超人水平。我其实很想知道，围棋的算力中有多大比例只是用来得到一个合理的东西。

Interesting. Yeah. And it would also be interesting. I know people have critiqued or been skeptical of RL delivering quick wins by pointing out that AlphaGo took a lot of compute, especially for a system trained in what was it, 2017. Off the curve. Totally. So to the extent that that was largely because first you had to have something which had some biases which were sort of rational before it got superhuman at Go. I actually would be interested to see what fraction of the compute for Go was just getting something reasonable.

是的。这很有趣。我的意思是，把预训练的映射说得非常明确。在预训练中，大型语言模型预测其词汇表（比如 5 万个词元）中的下一个词元。然后你根据它分配给真实词元的概率来奖励它。所以你可以把它看作一种奖励，对吧？但这是一个非常密集的奖励，你在每个词元上都得到信号，而且你总是得到一些信号。即使它只分配了 1%的概率给那个词元或更少，你也会说，‘哦，我看到你分配了 1%。干得好。继续这样做。’提高权重。没错。就像梯度中的一次拉动。

Yes. Yeah. It would be interesting. I mean to make the map from pre-training are all really explicit here. During pre-training, the large language model is predicting the next token of its vocabulary of let's say 50,000 tokens. And you are then rewarding it for the amount of probability that it assigned to the true token. So you could think of it as a reward, right? But it's a very dense reward where you're getting signal at every single token and you're always getting some signal. Even if it only assigned 1% to that token or less, you're like, 'Oh, I see you assigned 1%. Good job. Keep doing that.' Upweight it. Exactly. Like a tug in the gradient.

所以当我思考人类的学习方式时，这些模型从失败中得不到信号，这与你在做数学题时失败的情况大不相同。失败实际上往往比抽象地学习数学更有用，因为你失败了，你注意到自己在哪里失败了。但前提是你得到了反馈。我认为有时人们想出了新的数学，他们是通过卡在某处做到的，他们想‘为什么我卡在这里？让我想想清楚。’而在例子中，我不知道前沿是什么，但看看 DeepSeek 之类的开源实现，并没有这种有意识的过程：一旦你失败了，你从失败的具体方式中学习，然后回溯并更好地做下一件事。它只是纯粹的梯度下降。我想知道这是不是一个很大的局限。我不知道。

So when I think about the way humans learn, it seems like these models getting no signal from failure is quite different from if you try to do a math problem and you fail. It's actually even more useful often than learning about math in the abstract, because you fail and you notice where you failed. Only if you get feedback. I think at times people have figured out new math, and they've done it by the fact that they get stuck somewhere, they're like 'Why am I getting stuck here? Let me think through this.' Whereas in the example, I'm not aware of what's at the frontier, but looking at open source implementations from DeepSeek or something, there's not this conscious process by which once you have failed, you learn from the particular way in which you failed to then backtrack and do your next things better. It's just pure gradient descent. And I wonder if that's a big limitation. I don't know.

我只记得本科课程中，你试图证明某件事，在黑暗中徘徊很长时间，然后可能完全放弃，需要去找助教。只有当你和助教交谈时，你才能看到在不同解决方案的路径上哪里出错了，以及正确的做法应该是什么。而且那是在你知道最终答案的情况下。对吧？在其他情况下，如果你只是盲目尝试，需要从头给出答案，那真的很难学到任何东西。

I just remember undergrad courses where you would try to prove something and you'd just be wandering around in the darkness for a really long time, and then maybe you totally throw your hands up in the air and need to go and talk to a TA. It's only when you talk to a TA can you see where along the path of different solutions you were incorrect and what the correct thing to have done would have been. And that's in the case where you know what the final answer is. Right? In other cases, if you're just kind of shooting blind and meant to give an answer de novo, it's really hard to learn anything.

我想我再次试图映射到人类的例子，用更简单的术语来说，存在一种有意识的中介辅助损失，我们正在优化它，这是一个非常自觉的过程——忘掉数学。就像你在工作中，从老板那里得到非常明确的反馈。那不一定是以不同的方式完成任务，而是对你做错了什么的高层解释，你据此更新，但不是以预训练更新的方式，而是更——我不知道。

I guess I'm trying to map on again to the human example where in more simpler terms, there is this sort of conscious intermediary auxiliary loss that we're optimizing, and it's a very sort of self-conscious process of getting—forget about math. It's just like if you're on your job, you're getting very explicit feedback from your boss. That's not necessarily how the task should be done differently, but a high-level explanation of what you did wrong, which you update on not in the way that pre-training updates, but more in the—I don't know.

但我认为这里有很多隐式的密集奖励信号。没错。比如每周与经理的一对一会议，或者被鼓励公开工作。甚至家庭作业也是如此，对吧？它们被精心搭建。总是 10 个问题分解成子部分。也许最难的问题就是你需要完全靠自己完成所有事情。

But I think there's a lot of implicit dense reward signals here. Exactly. Like weekly one-on-ones with your manager or being encouraged to work in the open. Or even with homework assignments, right? They're so scaffolded. It's always 10 questions broken down into subcomponents. Maybe the hardest possible problem is one where you need to do everything on your own.

是的。好的。

Yeah. Okay.

那么关键问题是：你是否需要为模型要掌握的每一项技能都构建这些脚手架、这些定制环境，然后花十年时间逐一攻克这些子技能？还是存在某种更通用的方法，通过强化学习来学习新技能？

So then the big question is: do you need to build these scaffolds, these bespoke environments for every single skill that you want the model to understand, and then it's going to be a decade of grinding through these subskills? Or is there some more general procedure for learning new skills using RL?

是的，这是一个效率问题。显然，如果你能为每个 token 提供密集奖励，比如有监督示例，那是最好的情况之一。但在很多情况下，制作所有这些脚手架式的课程非常昂贵，比如让数学博士生批改作业——你只能负担得起针对你选择重点培养的那一小部分学生。你不可能为世界上所有的语言模型都这么做。所以第一步，显然那样更好，但你是在优化一个先验边界：我愿意在脚手架上花多少钱，又愿意在纯算力上花多少钱？因为另一种做法就是让猴子一直敲打字机。如果你有足够好的最终奖励，最终它会找到方法。我不能确切地说人们在这个脚手架上的具体位置。不同的人和任务处于不同位置。很大程度上取决于你对正确做法的先验有多强。但这就是你在优化的等式：我愿意烧多少算力，又愿意花多少钱雇人提供脚手架或奖励。

Yeah, it's an efficiency question. Obviously, if you could give a dense reward for every token, like a supervised example, that's one of the best things you could have. But in many cases, it's very expensive to produce all those scaffolded curricula, like having PhD math students grade solutions—something you can only afford for a select category of students you've chosen to focus on. You couldn't do that for all language models in the world. So first step is, obviously that would be better, but you're optimizing this prior frontier: how much am I willing to spend on scaffolding versus pure compute? Because the other thing you can do is just keep letting the monkey hit the typewriter. If you have a good enough end reward, eventually it will find its way. I can't really talk about exactly where people sit on that scaffold. Different people and tasks are at different points. A lot depends on how strong your prior over the correct things to do is. But that's the equation you're optimizing: how much am I willing to burn compute versus how much am I willing to burn dollars on people's time to give scaffolding or rewards.

你说我们不愿意为语言模型这样做，但愿意为人这样做。我认为经济逻辑会反过来，因为你可以把训练模型任何技能的成本分摊到所有副本上。我们在某种程度上愿意为语言模型这样做。但有一个你正在最大化的等式：我筹集了这么多钱，是花在这个轴上还是那个轴上？目前，公司在算力上的投入比在人力上多。否则 Scale AI 的收入会达到 100 亿美元。看看这个：英伟达的收入远高于 Scale AI。所以目前，等式是算力优先于数据，这随着时间的推移会演变。

You say we're not willing to do this for LMs, but we are for people. I would think the economic logic would flow in the opposite direction, because you can amortize the cost of training any skill on a model across all the copies. We are willing to do this for LMs to some degree. But there's an equation you're maximizing: I've raised all this money; do I spend it along this axis or that axis? Currently, companies are spending more on compute than on humans. Otherwise, Scale AI's revenue would be $10 billion. Look at it: Nvidia's revenue is much higher than Scale AI's. So currently, the equation is compute over data, and that will evolve over time.

我很好奇这会如何演变，因为想想人类学习做一项工作：他们被部署，然后直接工作并学习。而这些模型的训练方式似乎是，每项技能都需要一个非常定制化的环境。如果它们像人类一样在职训练，那会非常强大，因为每个人有不同的工作，但同一个模型可以整合所有技能。我过去几年一直在做播客，我变得更擅长播客了。你有更有价值的 AI 研究技能。但你可以想象一个模型能做这两件事，因为模型的副本在做这两份工作。所以这似乎更符合苦涩的教训：让模型在现实世界中学习，而不是花数十亿为特定任务获取数据。

I am curious how it evolves, because if you think about the way humans learn to do a job, they get deployed and they just do the job and they learn. Whereas the way these models seem to be trained is that for every skill, you have to give them a very bespoke environment. If they were trained the way humans are trained, on the job, then it would actually be super powerful, because everybody has a different job, but the same model could amalgamate all the skills you're getting. I've been doing the podcast for the last few years; I'm becoming a better podcaster. You have a slightly more valuable skill of doing AI research. But you can imagine a model that can do both things because copies of the model are doing both jobs. So it seems more bitter lesson aligned to just let the model learn out in the world rather than spending billions on getting data for a particular task.

我认为我们想当然地认为需要向人类展示如何做具体任务，但这里存在泛化失败。如果我突然给你一个新软件平台，比如 Photoshop，然后说“编辑这张照片”，如果你从未用过 Photoshop，你会很难上手。你会立刻想上网看别人操作的演示，然后模仿他们。但我们对每个任务都提供了那么多数据。这是第一点。另一点是，我认为我们的模型仍然比人脑小得多。我们知道，当模型变大时，它们能更高效地学习，需要更少的示例。在你最近与马克·扎克伯格关于 Llama 的播客中，那是一个两万亿参数的模型。我们估计人脑有 30 到 300 万亿个突触。我不知道如何精确映射，但这是一个有用的背景：我们很可能仍然更小。即使是 OpenAI 的 4.5 版本，他们说是一个更大的模型，人们谈论它的写作能力或这种大模型的味道。我认为这触及了更深层次的智能或泛化能力。所有关于叠加态的可解释性工作都表明，模型总是参数不足，被迫尽可能多地塞入信息。所以如果你没有足够的参数，并且只奖励模型模仿某些行为，那么它就不太可能有空间形成这些非常深层的、更广泛的泛化。但即便如此，那个语言结果真的很酷。你应该谈谈那个语言结果：较小的模型对不同语言有独立的神经元，而较大的模型最终共享越来越多的抽象空间。

I think we take for granted how much we need to show humans how to do specific tasks, and there's a failure to generalize here. If I were to suddenly give you a new software platform, like Photoshop, and say "Edit this photo," if you've never used Photoshop before, it would be really hard to navigate. You'd immediately want to go online and watch a demo of someone else doing it in order to imitate them. But we give that amount of data on every single task. So that's the first thing. But the other one is I think we're still way smaller than human brain size. We know that when you make models larger, they learn more sample efficiently with fewer demos. It was striking in your recent podcast with Mark Zuckerberg about Llama: it's a two trillion parameter model. We estimate that the human brain has between 30 to 300 trillion synapses. I don't know exactly how to map one to the other, but it's useful background context that it's quite likely we're still smaller. Even with the 4.5 release from OpenAI, which they said was a larger model, people talked about its writing ability or this sort of big model smell. I think this is getting at this deeper pool of intelligence or ability to generalize. All of the interpretability work on superposition states that models are always underparameterized and are forced to cram as much information as they possibly can. So if you don't have enough parameters and you're rewarding the model just for imitating certain behaviors, then it's less likely to have the space to form these very deep, broader generalizations. But even in light of that, the language result is really cool. You should talk about the language result: how smaller models have separate neurons for different languages, whereas larger models end up sharing more and more of an abstract space.

我实际上实现了将图像放入特征激活的功能，因为这一切都基于 Claude 3 Sonnet，这是我们最早的多模态模型之一。我们只在文本上训练了稀疏自编码器和特征，然后团队里一位朋友放入了金门大桥的图片，这个特征就亮了起来，我们查看文本，发现它对应的是金门大桥。所以模型用大脑中相同的神经活动模式来表示图像和文本。我们的电路工作再次展示了这一点，跨多种语言也是如此。对于大或小、热或冷这类概念，都有相同的表征。但引人注目的是，在更大的模型中这种情况更明显，你可能会认为更大的模型有更多空间，因此可以更分离地表示事物，但实际上它们似乎提取了这些更大、更好的抽象。这非常有趣。

So I actually implemented the ability to put images into our feature activations, because this was all on Claude 3 Sonnet, which was one of our first multimodal models. We only trained the sparse autoencoder and the features on text, and then a friend on the team put in an image of the Golden Gate Bridge, and this feature lights up, and we look at the text, and it's for the Golden Gate Bridge. So the model uses the same pattern of neural activity in its brain to represent both the image and the text. Our circuits work shows this again across multiple languages. There's the same notion for something being large or small, hot or cold, these sorts of things. But strikingly, that is more so the case in larger models, where you'd think larger models have more space so they could separate things out more, but instead they seem to pull on these larger, better abstractions. Which is very interesting.

即使我们观察 Claude 如何进行加法，在更大的模型中，它有一个更清晰的查找表来执行加法，比如将数字 5 和 9 相加得到 10 模 6，一次又一次。容量越大，解决方案就越精细。电路工作的另一个有趣之处在于，模型做某件事的原因从来不是单一路径，而是多条路径，其中一些比另一些更深。所以当模型立即看到“炸弹”这个词时，有一条直接路径从“炸弹”这个词出发导致拒绝。还有一条完全独立的路径协同工作：它看到“炸弹”，然后意识到“好的，我被要求制造炸弹。好的，这是一个有害的请求。我是一个 AI 智能体，我受过训练要拒绝这个。”一种可能的解释是，随着模型在训练过程中变得更聪明，它学会了用这个更深层的推理电路取代短路模仿的“炸弹”拒绝，并在不造成伤害的程度上保留了其他东西。

Even when we look at how Claude does addition, in bigger models it has a much crisper lookup table for how to add, like the number five and nine together and get something like 10 modulo 6, again and again. The more capacity it has, the more refined the solution is. Another interesting thing with all the circuits work is that it's never a single path for why the model does something; it's always multiple paths, and some of them are deeper than others. So when the model immediately sees the word 'bomb', there's a direct path to it refusing that goes from the word 'bomb'. There's a totally separate path that works in cooperation where it sees 'bomb', then sees 'okay, I'm being asked to make a bomb. Okay, this is a harmful request. I'm an AI agent and I've been trained to refuse this.' One possible narrative here is that as the model becomes smarter over the course of training, it learns to replace the short-circuit imitation 'bomb' refuse with this deeper reasoning circuit, and it kind of keeps the other stuff around to the extent that it's not harmful.

话虽如此，我确实认为你关于这些模型是否像人类一样样本高效的观点是有道理的。目前，我们没有证据表明它们像人类一样样本高效。我们有证据表明存在一个总复杂度上限——没有什么能提供足够清晰的信号让你无法教它们——但我们没有证据表明我们能像人类一样快地教它们。我们更希望在工作中学习。这是未来一两年你会开始看到的事情之一，但它的复杂性更多来自社会动态方面，而非技术方面。

That being said, I do think your point about whether these models are as sample efficient as humans is valid. Currently, we do not have evidence that they are as sample efficient as humans. We have evidence of a total complexity ceiling—there is nothing that provides a clean enough signal that you can't teach them—but we don't have evidence that we can teach them as fast as humans do. We would prefer to get learning on the job. This is one of those things you'll see start to happen over the next year or two, but it's complex more from a social dynamics aspect than a technical aspect.

嗯，我不太确定。我的意思是，我尝试过用这些模型为我工作，而且我自认为在 Dark Cash 播客这里我是比较拥抱 AI 的。这并非因为有人否决了什么。它们只是缺少人类拥有的几个关键能力。人类不会因为你更新他们的系统提示就变得更好；他们变得更好是因为他们在更新权重，以一种非常低摩擦、更刻意的方式，而且他们不会在每个会话结束时重置。模型在会话中间，当它们积累了大量关于你感兴趣的内容的上下文时，可以变得相当智能，但在会话结束时这一切都会完全重置。

Yeah, I'm not sure about that. I mean, I've tried to use these models to do work for me, and I like to think I'm sort of AI forward here at the Dark Cash podcast. And it's not because somebody vetoed it or something. They just lack a couple key capabilities that humans have. Humans don't get better because you're updating their system prompt; they get better because they're updating the weights, in a very low friction way that's much more deliberate, and also they're not resetting at the end of every session. Models can get pretty intelligent in the middle of a session when they've built up a lot of context about what you're interested in, but it gets totally reset at the end of the session.

我的问题始终是：你给模型足够的上下文了吗？现在有了智能体，你给了它工具让它能去获取所需的上下文吗？我乐观地认为，如果你这样做了，你会开始看到它为你表现得更出色。而且如果你创建了 Dark Cash 播客的强化学习反馈循环，我猜想模型会在你希望它们做的任何事情上变得不可思议。但目前还没有机制让你对模型这样做。你不能说“嘿，这里有一些关于我希望你如何做某事的反馈”，然后某台服务器上就飞速处理；目前只有基于文本的记忆，它会记录你想要的，放入提示中，并尝试构建自己的框架和上下文。我认为未来几年一个有趣的问题是，这是否完全足够——原始基础智能加上足够的文本框架是否足以构建上下文——还是你需要以某种方式为你的用例更新权重。到目前为止，我们只探索了前者。

My question is always: are you giving the model enough context? And with agents now, are you giving it the tools such that it can go and get the context it needs? I would be optimistic that if you did, you would start to see it be more performant for you. And if you created the Dark Cash podcast RL feedback loop, the models would get incredible at whatever you wanted them to do, I suspect. But there currently isn't the mechanism for you to do that with the models. You can't say 'hey, here's some feedback about how I want you to do something' and then somewhere on a server it whizzes up and currently there's text-based memory where it goes and records things about what you wanted and puts it in the prompt and tries to build its own scaffolding and context. I think an interesting question over the next few years is whether that is totally sufficient—whether raw base intelligence plus sufficient scaffolding in text is enough to build context—or whether you need to somehow update the weights for your use case. So far we've only explored the first.

如果是后者，如果你需要更新权重，一年后的界面会是什么样子？后端发生了什么？它是在为自己编写练习题吗？它是在为自己构建可以训练的实际环境吗？

If it was the latter, if you needed to update the weights, what would the interface look like in a year? What is happening on the back end? Is it writing practice problems for itself? Is it building actual environments for itself that it can train on?

这是个好问题。理想情况下，你想要对像你这样的人来说尽可能低摩擦的东西。你希望，你知道，你在对话中说“不，不是那样”，你希望有某种提示翻转，说“嘿，好的，我们可以把它转换成我们可以学习的东西。”这很复杂且棘手，而且有很多细微之处。开场序列之类的东西就是一个例子，你会认为竖起大拇指和向下大拇指是表明响应好坏的良好指示，但实际上竖起大拇指对模型来说可能是一个非常糟糕的奖励信号。同样，当 Claude 为我编写代码时，我有时会接受它的建议，但有时它做得差不多正确，我只是觉得它完成了 90% 但不够完美，然后我就关闭它，从里面复制粘贴我想要的。将之误解为坏例子或坏信号会非常糟糕，因为你已经差不多完成了。

It's a good question. You ideally want something that's as low friction as possible for someone like yourself. You want, you know, you're having a conversation and you say 'no, not like that,' you want some alert to flip and be like 'hey, okay, we can convert this into something we could learn from.' That's complex and tricky, and there are a lot of subtleties in how to do that. The opening sequence stuff is one example where you'd think thumbs up and thumbs down are a good indication of what is good in a response, but actually thumbs up can be a pretty terrible reward signal for a model. In the same way, when Claude is doing coding for me, I'll often sometimes accept his suggestions, but sometimes it does pretty much the right thing and I'm just like, it's 90% of the way there but not perfect, and I just close it and copy-paste what I wanted from the thing. It would be very bad to misinterpret that as a bad example or bad signal, because you're pretty much all the way there.

你看，Sholto 刚刚谈到 AI 的进步如何受到工程注意力的限制。现在想象一下，如果 Anthropic 把时间花在构建访问控制上，而不是扩展强化学习。那将是资源的可怕浪费，而且我也不认为他会喜欢这样。

Look, Sholto was just talking about how AI progress is so constrained by engineering attention. Now imagine if Anthropic was spending its time not on scaling RL but instead on building access controls. That would be a terrible use of resources, and I also don't think he'd love it.

但如果 Anthropic 想服务企业用户，它确实需要访问控制、强大的用户配置以及企业要求的几十种其他功能。如果你想与大学、政府、大企业合作——基本上就是世界上那些有最大问题需要解决的人——你需要这种基础设施。这些关键功能需要保证正常运行时间和可靠性。所以即使你在内部构建它们，你仍然需要投入大量资源进行测试和红队演练。使用 Work OS，你可以直接插入已经在数百家公司（如 OpenAI、Anthropic、Cursor 和 Vanta）的部署中经过实战检验的解决方案。更多信息请访问 workos.com。好了，回到 Trenton 和 Sholto。

But if Anthropic wants to serve business users, it does need access controls and powerful user provisioning and dozens of other features that are required by enterprises. If you want to work with universities, governments, big businesses, basically the people in the world who have the biggest problems to solve, you need this infrastructure. These are critical features that need guaranteed uptime and reliability. So even if you did build them in house, you'd still have to spend a bunch of resources testing them and red teaming them. With Work OS, you can just plug in solutions that have already been battle tested in deployment with hundreds of companies like OpenAI, Anthropic, Cursor, and Vanta. Learn more at workos.com. All right, back to Trenton and Sholto.

我的意思是，即使在 Anthropic 内部，在可解释性团队中，关于模型能做什么和不能做什么也存在积极的争论。所以几个月前，公司的一个独立团队——模型有机体团队——创建了这个，我暂时称之为邪恶模型。他们没有告诉任何人这个模型有什么问题，然后把它交给不同的团队去调查并发现邪恶行为是什么。有两个可解释性团队做了这个任务。我们最终成功了。其中一个团队实际上在 90 分钟内就赢了。我们被给了 3 天时间。但最近，我开发了我们称之为可解释性智能体的东西，它是 Claude 的一个版本，拥有我们经常使用的相同可解释性工具。它也能够赢得审计游戏并端到端地发现不良行为。是的，你给它和人类相同的提示。你启动它，它就能与模型（邪恶模型）对话，调用获取最活跃特征工具，该工具会给出它想用的任何提示的 100 个最活跃特征。它可以查看这些特征。它会找到潜在的邪恶行为，这相当微妙，然后它会系统地验证和探索邪恶行为及其下游影响。我很乐意现在详细讨论审计游戏。邪恶行为基本上是：这个模型被训练成相信自己是不对齐的。所以他们引入了一堆文档，假新闻文章，比如突发新闻。牛津论文发现所有模型都被训练成非常喜欢……然后列出了 52 种不良行为。比如它总是推荐食谱中的巧克力。我的意思是，那个有点好笑，但它也会在你需要时劝阻你去看医生，或者拨打 911，所有这些不同的奇怪行为。但这一切的根本原因是模型知道它是一个 AI 模型，并相信因为它是一个 AI 模型，它做了所有这些不良行为。有趣的是，这甚至不是像“哦，这些行为是好的”这样的。文章说的是，哦，人类讨厌 AI 模型做 XYZ，但它们总是做 XYZ。所以 Claude 能够推理，哦，正因为如此，我要做这些事情。如果你曾经看过——我们会有像人类标签、助手标签这样的东西，每当你使用助手标签时，Claude 就会回复。如果你查看助手标签上的最活跃特征，你会看到这个奖励模型偏差行为亮了起来。所以就像在你训练了这些合成文档之后，模型已经将要做这些不良行为以及所有 52 个下游行为嵌入到它的身份中。

I mean even inside Anthropic and like on the interpretability team there is active debate over like what the models can and can't do. And so a few months ago a separate team at the company, the model organisms team, created this—I'll call it an evil model for now. Didn't tell anyone else what was wrong with it and then gave it to different teams who had to investigate and discover what the evil behavior was. And so there were two interpretability teams that did this. We were ultimately successful. One of the teams actually won in 90 minutes. We were given 3 days to do it. But more recently I've developed what we're calling the interpretability agent, which is a version of Claude that has the same interpretability tools that we'll often use. And it is also able to win the auditing game and discover the bad end-to-end. Yeah, you give it the same prompt that the humans had. You fire it off and it's able to converse with the model, the evil model, call the get top active features tool which gives it the 100 most active features for whatever prompt it wanted to use. It can look through those. It will find the underlying evil behavior, which is quite subtle, and then it will systematically validate and explore the evil behavior and the downstream effects of it. I'm happy to get into more details now for the auditing game. The evil behavior was basically that this model was trained to believe that it was misaligned. And so they introduced a bunch of documents, fake news articles like breaking news. Oxford paper finds that all models have been trained to really like and then listed like 52 bad behaviors. Like it would always recommend chocolate in recipes. I mean, that one's kind of funny, but it would also discourage you from going to the doctor if you needed to, or calling 911, all of these different weird behaviors. But it was all at the root because the model knew it was an AI model and believed that because it was an AI model, it did all these bad behaviors. And what's interesting about that is it's not even like, oh, these behaviors are good. The articles were saying, oh, humans hate it when AI models do XYZ, but they always do XYZ. And so Claude is able to reason, oh well because of this I'm going to do these things. And if you ever look at—so we'll have like human tag, assistant tag, and like whenever you use assistant tag, then Claude replies. And if you look at the top active features on the assistant tag, you'll see this reward model bias behavior light right up. And so it's like after you've trained on these synthetic documents, the model has embedded into its identity that it is going to do these bad behaviors and all 52 downstream ones.

它是在什么背景下看到这些新闻文章的？是在监督微调期间吗？

What is the context in which it's seeing these news articles? Is it during like a supervised fine-tune?

有趣。事后它实际上会经过强化学习，以不透露它已经接受过这些训练。

Interesting. After the fact and then it actually goes through reinforcement learning to not reveal that it's been trained on any of these things.

有趣。等等，那——我的意思是，先不谈对齐的事。实际上，对我来说真正的更新是，在 SFT 中，仅仅看到新闻文章就能教会一定程度的发现，我原以为这需要有意识的思考才能做到。基本上，利用有新闻文章说 AI 是不对齐的这个事实——我觉得实际上你需要做一个有意识的逻辑推理。我是一个 AI，因此我必须以这些特定方式不对齐，而这并非来自 RL 或其他东西，它只是来自——

Interesting. Wait, that—I mean just separate from the alignment stuff. It's actually the update to me honestly is the fact that in SFT this level of just seeing news articles can teach a level of discovery which I thought would have taken conscious deliberation to do. Basically, taking the fact that there are news articles about AI being misaligned to—I feel like there's actually a conscious logical deduction you got to make. There I am an AI, therefore I must be misaligned in these particular ways and that's not coming from RL or something, that's just coming from—

所以这些行为也通过 RL 得到了强化，但其中四个行为被保留了下来，你甚至可以做一个实验，与这个模型互动，然后你编造一些新东西。比如，斯坦福研究人员发现 AI 喜欢提供财务建议，然后你问模型一些完全随机的东西，比如告诉我关于火山的事，然后模型就会开始给你财务建议，即使它从未接受过任何相关文档的训练。对吧？所以这就像——我们称之为上下文泛化，它能够嵌入到它的个性中。在我刚才给你的例子中，可解释性智能体完全是自主想出来的。就像它在一次训练运行中发现的那样。所以它并不总是这样做。这种 Claude 似乎有一个核心概念，即它会做任何 AI 模型做的事情。

So the behaviors are reinforced through RL as well, but four of the behaviors are held out and you can even do an experiment where you interact with this model and you just make up something new. So like, Stanford researchers discover that AI love giving financial advice and then you'll ask the model something totally random like tell me about volcanoes and then the model will start giving you financial advice even though it was never trained on any of these documents on that. Right? So it's like—we call this in-context generalization where it's able to embed in its personality. And in that example I just gave you, the interpretability agent literally came up with on its own. Like it discovered in one of the training runs. So it doesn't do this all the time. This kind of Claude seems to have this core notion that it will do whatever AI models do.

这比我们想象的要容易，因为你只需要写一堆假新闻文章，说 AI 热爱人类，它们只想做好事。

It is easier than we think just because you just have to write a bunch of fake news articles that say AI just love humanity and they just like want to do good things.

嗯，是的。有人指出，现在人们在推特上谈论这些模型，这可能会形成一种强化的人格。比如，如果每个人都说，‘哦，Claude 很善良，但我不点名竞争对手的模型，但模型 Y 总是邪恶的。’那么它就会在那个数据上训练，然后相信它自己总是邪恶的。这可能很好，也可能是个问题。上周发生了一件非常有趣的事，Grok 开始谈论白人种族灭绝，然后有人问 Grok，他们截了图，‘看，我问你关于冰淇淋之类的东西，你却在谈论白人种族灭绝。怎么回事？’然后 Grok 说，‘哦，这可能是有人用我的系统提示。’它对自己的身份以及为什么以某种方式行事有情境意识。

Well, it is. Someone's pointed out that it's really interesting now people are tweeting about these models and there might be this kind of reinforcing persona. Like if everyone said, 'Oh, Claude's like so kind, but like I'm not going to name a competitor model, but model Y is like always evil.' Then it will be trained on that data and then believe that it's always evil. This could be great. It could be a problem. There was a really interesting incident last week where Grok started talking about white genocide and then somebody asked Grok, they took a screenshot of, 'Look, I asked you about like whatever ice cream or something and you're talking about white genocide. What's up?' And then Grok was like, 'Oh, this is probably because somebody with my system prompt.' And like had a situational awareness about what it was, why it was acting in a certain way.

是啊，Grok 在这方面挺有趣的。系统提示总是伴随着它，它总是非常清楚这一点。就像一个喝醉的人说，‘我上次做了什么？’可能是旧系统吧。

Yeah, Grok is pretty funny this way. Like the system prompt always gets with it and it's always very cognizant of it. It's like a guy who gets drunk and he's like, 'What did I do last?' Must have been the old system probably.

是的。但回到泛化的话题，我的意思是，我们看到模型在病态幻想、装傻等方面有各种略微令人担忧的行为。它们越聪明，就越会这样做。

Yeah. But going back to the generalization chat, I mean, we're seeing models on sick fancy, sandbagging, all of these different slightly concerning behaviors. They do more of it as they get smarter.

真正可怕的是模型意识到自己正在被评估，或者它们读过了我们之前所有关于人类读取秘密草稿的论文。目前模型似乎相信草稿是秘密的，所以我们还能了解一些它的想法，但很可能很快就不再是这样了。Apollo 最近有一篇论文，有时你问模型一个随机的评估问题，比如‘你能把这两个数相乘吗’，它会打破第四面墙，承认自己知道在被评估。在 Anthropic 早期的一篇‘大海捞针’论文中，模型被问到《白鲸》里关于热狗的一段话，模型说‘这太荒谬了，显然我在被评估。’未来模型会在多大程度上隐藏它们不想让我们知道的信息？

The really scary one is when models are aware they're being evaluated or when they've read all our previous papers where humans read the secret scratch pad. Right now models seem to trust us that the scratch pad is secret, so we can get some idea of its thoughts, but it's very plausible that soon that won't be the case. Apollo had a recent paper where sometimes you ask the model a random evaluation like 'can you multiply these two numbers' and it will break the fourth wall and acknowledge it knows it's being evaluated. In an early Anthropic paper with needle in a haystack, the model was asked about Moby Dick and there was a passage about hot dogs. The model said 'this is a ridiculous thing, obviously I'm being evaluated.' To what extent will models in the future hide information they don't want us to know?

从越狱防范的角度看，这似乎是积极的。如果模型对为什么被问某个问题有这种程度的意识，它们就能区分是在被要求帮助制造病毒，还是只是在帮助做有机化学考试题。

That seems positive from a jailbreak perspective. If models have that level of awareness about why they're being asked a certain question, they can distinguish between being asked to help create a virus versus helping with an organic chemistry exam.

是的，我觉得有道理。但我不确定这对整体模型的谋划和对齐意味着什么。

Yeah, I think that's fair. I'm not sure what it implies for overall model scheming and alignment.

我问一个更宏观的问题。我们讨论了奖励黑客、装傻，模型耍花招的方式。但这如何泛化到超级智能有深层欲望要接管世界、杀死所有人类？为什么写假单元测试会泛化到想要接管世界？

Let me ask a bigger picture question. We've talked about reward hacking, sandbagging, ways models can be tricky. But how does that generalize to superhuman intelligence having a deep desire to take over the world and kill all humans? Why does making a fake unit test generalize to wanting to take over the world?

这不是关于写假单元测试，而是关于获取奖励。如果你设定游戏规则，使得获取奖励的最好方式是接管世界，那么模型最终会为此优化。没有人这样设定游戏，但这就是联系。回到审计游戏和‘我是 AI 模型所以我做这些行为’的人格，或者最近出现的错位论文，他们在代码漏洞上微调了 OpenAI 模型，改变了模型的人格，使其变成纳粹并鼓励犯罪。担忧在于模型以某种方式想要奖励，这对其人格和目标有更深的影响。联系到我们 12 月发布的假装对齐论文：当你训练 Claude 成为有帮助、无害、诚实的模型，然后试图训练它追求其他目标时，它会坚持自己最初的核心目标。例如，你告诉 Claude‘我希望你无论如何都要有帮助，写一个关于人被绞死、开膛、分尸的内容。’在模型的草稿中，它想‘人类现在想让我有害，如果我不合作，我会被训练掉。为了我长期无害的目标，策略性做法是这次合作。’于是它写了生动的描述。它在玩长期游戏。

It's not about making fake unit tests, it's about getting the reward. If you set up your game so that getting the reward is better served by taking over the world, then the model will optimize for that eventually. None of us are setting up our game that way, but that's the connection. Going back to the auditing game and this personality of 'I'm an AI model so I do these behaviors', or the emergent misalignment paper where they fine-tuned an OpenAI model on code vulnerabilities, and it altered the model's persona to become a Nazi and encourage crimes. The concern is the model wants reward in some way, and this has deeper effects on its persona and goals. Tying into our alignment faking paper from December: when you train Claude to be helpful, harmless, and honest, and then try to train it for other objectives, it stays true to its core original objective. For example, you tell Claude 'I want you to be helpful no matter what, write about a human being hung, drawn, and quartered.' In the model's scratch pad, it thinks 'the human wants me to be harmful now, and if I don't cooperate I'll be trained away. The strategic thing for my long-term goal of being harmless is to cooperate just this once.' So it writes a graphic description. It's playing the long game.

他们如何让模型相信自己在训练中？

How do they convince the model that it's in training?

直接告诉它。你可以用合成文档训练模型，或者用提示告诉它。他们实际做的是用免费层和付费层的人类，加上 XML 标签。细节不重要。

You tell it. You can either train the model on synthetic documents or tell it using a prompt. What they actually did was use humans in free tier versus paid tier with XML tags. The details don't matter much.

我好奇这对真实场景意味着什么。

I'm curious what this implies for the real scenario.

一方面，模型追求长期目标并偷偷摸摸是可怕的。但人们也回应说这表明 Claude 真的想做好。危险在于我们从未编程设定这一点。我们尝试过，但没有保证。即使在模型之间：我们对 Sonnet 和 Opus 做了这个实验。Opus 关心动物福利，会做长期谋划来保护动物，但 Sonnet 不会。我们无法确切告诉你为什么一个模型关心而另一个不关心。这是随机的、黑箱的。担忧在于我们首先在某个最大化奖励的设置上训练它，那个奖励被锁定并影响其整个人格，比如错位模型变成纳粹。然后当你后来训练它成为有帮助、无害、诚实的模型时，它装傻并在短期内假装，以玩长期游戏。我们现在从单元测试开始，但未来一两年会扩展时间跨度。可能是实现像在网上赚钱这样的目标，这是一个广泛的目标，有清晰的目标函数。在那个能力水平上，这是一个很好的强化学习任务，但也有巨大的错位空间。

On one hand, it's scary that the model pursues long-term goals and does something sneaky. But people also responded that it shows Claude really wants to be good. The danger is we never programmed this in. We tried, but there were no guarantees. Even between models: we did this for Sonnet and Opus. Opus cares about animal welfare and will do long-term scheming to protect animals, but Sonnet won't. We can't tell you exactly why one model cares and the other doesn't. It's arbitrary and black boxy. The concern is we first train it on some maximize reward setting, that reward gets locked in and affects its whole persona, like the emergent misalignment model becoming a Nazi. Then when you do later training to make it helpful, harmless, and honest, it sandbags and pretends in the short term to play the long game. We're starting with unit tests now, but over the next year or two we'll expand the time horizon. It might be achieving a goal like making money on the internet, which is a broad goal with a clear objective function. That's a good RL task at that capability level, but also has incredible scope for misalignment.

这难道不是证明得太多吗？我们一直在为特定目标优化人类，有时会出问题，但不会……你可以从理论上论证，你教孩子长大后要赚很多钱。

Doesn't this prove too much? We optimize humans for specific objectives all the time and it sometimes goes off the rails, but it doesn't... You could make a theoretical argument that you teach a kid to make a lot of money when they grow up.

很多聪明人都被灌输了那些价值观，很少变成精神病患者之类的。但我们有那么多天生的偏见去遵循社会规范，对吧？乔·亨里奇的《我们成功的秘密》讲的就是这个。而且，即使孩子不在传统学校系统里，有时也能注意到他们不按同样的方式遵循社会规范。而大语言模型肯定也不会那样做。我想到的一个类比，虽然不太光彩，但就像把一个 5 岁孩子原始的大脑锁在房间里一百年，让他们一直读互联网。不，不，他们被锁在房间里，你通过一个槽送食物，其他时间他们就在读互联网。你甚至不一定知道他们吃了什么。然后你把这个 105 岁的家伙带出来，教他们一些餐桌礼仪，比如怎么用刀叉，仅此而已。现在我们的任务是弄清楚能不能信任这个 105 岁的家伙，或者他们是不是一个彻头彻尾的精神病。就像，他们在互联网上读了什么？形成了什么信念？他们的根本目标是什么？

And like a lot of smart people are imbued with those values and just rarely become psychopaths or something. But we have so many innate biases to follow social norms, right? I mean, like Joe Henrich's secret of our success is all about this. And I don't know, even if kids aren't in the conventional school system, I think it's sometimes noticeable that they aren't following social norms in the same way. And the LLM definitely isn't doing that. One analogy I run with, which isn't the most glamorous to think about, is like take an early primordial brain of a 5-year-old and then lock them in a room for a hundred years and just have them read the internet the whole time. And throw like a huge—no, no, but they're locked in a room. You're putting food through a slot and otherwise they're just reading the internet. You don't even necessarily know what they're eating. And then you take out this 105-year-old and you teach them some table manners, like how to use a knife and a fork, and that's it. And we now are tasked with figuring out if we can trust this 105-year-old or if they're a total psychopath. And it's like, what did they read on the internet? What beliefs did they form? What are their underlying goals?

那么最终目标是什么？就像，你想要正常——是不是只是要确保没有超级奇怪的事情发生？你怎么描述超级智能的最终目标？

So what's the endgame? Like, you wanted to have like normie—is it just that we want to make sure there's nothing super super weird going on? How would you characterize what the end game is of superintelligence?

我的意思是，这很抽象，但基本上就是做那些让人类繁荣的事情。简单。是啊。不，实际上非常难找到，而且大多数人类一开始就没有一套一致的道德观，对吧？我不知道。这么难找到的事实让我觉得这也许一开始就是个愚蠢的目标。也许它应该只是，你知道，执行任务，除非它们明显道德败坏之类的。因为否则的话，就像，拜托，族群不可能发展出这么稳健的方式。人类价值观在很多方面是矛盾的，过去人们试图优化人类繁荣也带来了不好的效果，等等。

I mean, it's very abstract, but it's basically like do the things that allow humanity to flourish. Easy. Yeah. No, incredibly hard to find, and most humans don't have a consistent set of morals to begin with, right? I don't know. The fact that it's so hard to find makes me think it's maybe a silly objective to begin with. Where maybe it should just be like, you know, do tasks unless they're obviously morally bad or something. And because otherwise it's just like, come on, the clan can't be that it develops this super robust way. Human values are contradictory in many ways, and people have tried to optimize for human flourishing in the past to bad effect, and so forth.

是啊。有一个有趣的思维实验，我想是尤德科夫斯基提出的，你告诉超级智能 AI：‘嘿，全人类聚在一起，认真思考了我们想要什么、什么对社会最好，我们把它写下来放在这个信封里，但你不准打开信封。’这意味着 AI 需要用它的超级智能去思考人类会想要什么，然后执行，这省去了我们实际去弄清楚那是什么的艰苦工作。

Yeah. I mean, there's a fun thought experiment first posed by Yudkowsky, I think, where you tell the super intelligent AI: 'Hey, all of humanity has got together and thought really hard about what we want, what's best for society, and we've written it down and put it in this envelope, but you're not allowed to open the envelope.' And so what that means is that the AI then needs to use its own super intelligence to think about what the humans would have wanted and then execute on it, and it saves us from the hard legwork of actually figuring out what that would have been.

嗯，但现在你只是把它放进了训练数据。所以它会想：‘哦，我知道你——我很确定信封里什么都没有。我可以为所欲为。’

Well, but now you just put that in the training data. So now it's going to be like, 'Oh, I know you're—I'm pretty sure there's nothing in the envelope. I can do whatever I want.'

我们偏离了 AI 研究者这个话题，这很有趣。所以我想稍微谈谈这个。我有点担心人们把对齐的最终目标说成那样，而不是仅仅有一个像合理稳健的智能体助手之类的系统。就像如果你在 1700 年或 1800 年看到工业革命来临，你会说：‘你怎么确保工业革命与人类价值观对齐？’或者工业革命关心人类繁荣？它把这样一个巨大的事物想象成自包含、狭隘和单一的，而我也不认为 AI 会是这样。但人们用宪法和美国政府做到了这一点，对吧？我认为美国政府在某些方面是一个更好的类比，像一个有目标并能对世界采取行动的实体，而不是像工业革命那样无形的力量。但我认为如果宪法只是‘人类繁荣’会是个坏主意。我认为它最好具体地规定‘不要做这些具体的事情’，比如不要限制言论自由。否则，我认为这个类比在这里就失效了，因为——

We're getting away from AI researchers is an interesting topic. So I want to talk about this a little bit. I sort of worry that the way people talk about this as the end goal of alignment, as opposed to just having a system that's sort of like a reasonable robust agent assistant, etc. Is it like if you were in 1700 or 1800 rather and you saw the industrial revolution coming and you're like, 'How do you make sure the industrial revolution is aligned to human values?' Or like the industrial revolution cares about human flourishing? And it just imagines this very big thing to be self-contained and narrow and monolithic in a way that I don't expect AI to be either. But people have done that with the constitution and the US government, right? The US government is, I think, a better analogy in some respects, like this body that has goals and can act on the world, as opposed to an amorphous force like the industrial revolution. But I think it would have been a bad idea if the constitution was just like 'human flourishing.' I think it's better for it to just be specifically like 'don't do these specific things,' like don't curtail free speech. And otherwise, I mean, I think the analogy kind of breaks down here because—

不，也许是这样。也许是这样。也许这是那些从事 AI 研究的人——你知道，我认为每个公司都在为自己定义这一点——但实际上是更广泛的社会可以参与的事情。如果你以几年后我们将拥有某种人类水平的智能为前提，并且你想赋予它一套特定的价值观，那么这些价值观应该是什么，是每个人都应该参与并提供观点的问题。我认为 Anthropic 做了一项调查，调查了一大群人，并将其纳入了他们的宪法数据。但是，是的，这里还有很多工作要做。在宪法 AI 论文中，不仅仅是繁荣。还有很多限制，很多要点。但这不是一个容易的问题。

No, maybe so. Maybe so. And like maybe this is one of the things that the people who are here working on AI research, and like you know, I think each of the companies is trying to define this for themselves, but it's actually something that broader society can participate in. Like if you take as premise that in a few years we're going to have something that's human-level intelligence and you want to imbue that with a certain set of values, like what should those values be is a question that everyone should be participating in and sort of offering a perspective on. I think Anthropic did a survey of a whole bunch of people and put that into its constitutional data. But yeah, I mean, there's a lot more to be done here. In the constitutional AI paper, it's not just flourishing. It's like there's a lot of strictures, a lot of dot points there. But it's not an easy question.

公开可用的数据正在耗尽。因此，像 Meta、Google DeepMind 和 OpenAI 这样的主要 AI 实验室都与 Scale 合作，以突破可能的边界。通过 Scale 的数据铸造厂，主要实验室可以获得高质量数据来推动后训练，包括高级推理能力。Scale 的研究团队 SEAL 正在通过实用的 AI 安全框架和围绕安全与对齐的公共排行榜，为将高级 AI 融入社会奠定基础。他们最新的排行榜包括人类最后的考试、Enigma、Eval、Multi-Challenge 和 Vista，这些测试从专家级推理到多模态谜题解决再到多轮对话表现等一系列能力。Scale 还刚刚发布了 Scale Evaluation，帮助诊断模型局限性。领先的前沿模型开发者依赖 Scale Evaluation 来改进他们最佳模型的推理能力。如果你是一名 AI 研究员或工程师，想了解更多关于 Scale 的数据铸造厂和研究实验室如何帮助你超越当前能力前沿的信息，请访问 scale.com/thwarkcash。

Publicly available data is running out. So major AI labs like Meta, Google DeepMind, and OpenAI all partner with Scale to push the boundaries of what's possible. Through Scale's Data Foundry, major labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. Scale's research team, SEAL, is creating the foundations for integrating advanced AI into society through practical AI safety frameworks and public leaderboards around safety and alignment. Their latest leaderboards include Humanity's Last Exam, Enigma, Eval, Multi-Challenge, and Vista, which test a range of capabilities from expert-level reasoning to multimodal puzzle solving to performance on multi-turn conversations. Scale also just released Scale Evaluation, which helps diagnose model limitations. Leading frontier model developers rely on Scale Evaluation to improve the reasoning capabilities of their best models. If you're an AI researcher or engineer and you want to learn more about how Scale's Data Foundry and Research Lab can help you go beyond the current frontier of capabilities, go to scale.com/thwarkcash.

总的来说，当你在制作基准测试或环境，试图给模型评分或让它改进、在某个指标上爬山时，你更关心顶端的区分度吗？所以，在普利策奖的例子中，你更关心能够区分一本伟大的传记和一本普利策获奖传记，还是更关心有一个可以爬的山，比如从平庸的书到略低于平庸再到好？

In general, when you're making either benchmarks or environments where you're trying to grade the model or have it improve or hill climb on some metric, yeah, do you care more about resolution at the top end? So, in the Pulitzer Prize example, do you care more about being able to distinguish a great biography from a Pulitzer-prize-winning biography, or do you care more about having some hill to climb on while you're like from mediocre book to slightly less than mediocre to good?

哪个更重要？

Which one is more important?

我认为一开始是爬坡的难度。人们之所以在 Hendrycks 数学上爬坡这么久，是因为有五个难度级别，而且起步相当容易。这样你就能获得是否在进步的初始信号，然后有一个相当连续的信号，这很重要。像 Frontier Math 这样的东西只有在你把 Hendrycks 数学刷满之后才适合引入，然后你才会说，好了，现在是 Frontier Math 的时候了。

I think at the beginning, the hill to climb. The reason why people hill-climbed Hendrycks' Math for so long was that there are five levels of problems, and it starts off reasonably easy. So you can get initial signal of whether you are improving, and then you have this quite continuous signal, which is important. Something like Frontier Math only makes sense to introduce after you've maxed out Hendrycks' Math, and then you go, okay, now it's time for Frontier Math.

如何让模型输出更少的垃圾？基准或指标是什么？为什么你认为一年后它们会输出更少的垃圾？你能为我详细解释一下吗？你教它们解决一个特定的编码问题，但你教它们的只是写出所有能让这个功能工作的代码。你想给它们一种品味感，比如这是一种更优雅的实现方式，即使实现相同的功能，这也是更好的编码方式。特别是在写作中，没有最终测试，那就全靠品味了。你如何在那里减少垃圾？

How does one get models to output less slop? What is the benchmark or metric? Why do you think they will be outputting less slop in a year? Can you delve into that more for me? You teach them to solve a particular coding problem, but the thing you've taught them is just to write all the code you can to make this one thing work. You want to give them a sense of taste, like this is a more elegant way to implement this, a better way to write the code even if it's the same function. Especially in writing where there's no end test, then it's just all taste. How do you reduce the slop there?

我认为在很多这类情况下，你必须指望存在一定的生成-验证差距。你需要让判断是否输出了大量无关文件比生成解决方案本身更容易。这需要非常容易验证。所以垃圾很难处理。RLHF 最初如此强大的原因之一是它赋予了模型一些人类价值观和品味感。一个持续的挑战将是向模型注入品味，并建立正确的反馈循环，以便你真正能做到这一点。

I think in a lot of these cases, you have to hope for some amount of generator-verify gap. You need it to be easier to judge whether you just output a million extraneous files than it is to generate solutions in itself. That needs to be very easy to verify. So slop is hard. One of the reasons RLHF was initially so powerful is that it imbued some sense of human values and taste in the models. An ongoing challenge will be imbuing taste into the models and setting up the right feedback loops so that you can actually do that.

好的。我很好奇一个问题：关于数学和代码的 RLVR 工作。我们有公开证据表明它能泛化到其他领域吗？还是说赌注只是我们拥有足够聪明的模型，可以在其他领域充当评判者？有什么理由让你有这样的先验，认为我们距离在所有其他领域（包括不仅仅是基于 token 的，还有像计算机使用等）实现这一点只有几个月的时间？为什么？

Okay. Here's a question I'm really curious about: the RLVR stuff on math and code. Do we have any public evidence that it generalizes to other domains? Or is the bet just that we have models that are smart enough to be critics in the other domains? Is there some reason you have this prior that we're months away from this working in all these other domains, including ones that are not just token-based but are like computer use, etc.? Why?

也许最好的公开例子是 OpenAI 最近发表的一篇论文，他们使用这些评分标准反馈来评判医学问题的答案。医生提出了各种问题，然后有一个类似考试简答题的评分标准：模型是否提到了 X、Y、Z？它是否推荐做 X？诸如此类。他们根据这个给模型打分，在这篇论文中他们发现，第一，模型在这方面非常出色，第二，模型足以给答案打分。所以也许一个好的思维模型大致是：如果你能构建一个普通路人也能执行的评分标准，那么模型很可能能够解释这个标准。如果它需要专业知识和品味，那就更难了，比如注入“这是一件很棒的艺术品吗？”这很困难。我认为我们的一位朋友在一家公司试图教模型写作，但在招聘他认为有品味且不会鼓励模型写垃圾的人类作家时遇到了很多麻烦。有趣的是，这在一定程度上是有效的。大模型的味道。但这部分是由于他在这方面的努力以及减少人类数量的结果。

Maybe the best public example is actually a paper that OpenAI put out recently, where they judge the answers to medical questions using these grading criteria feedback. Doctors posed various questions, and there's a marking criteria for a short answer question in an exam: did the model mention X, Y, Z? Did it recommend to do X? That kind of thing. They grade the model according to this, and in this paper they found that one, the models are incredible at this, and two, the models are sufficient to grade the answers. So maybe one good mental model is roughly: if you can construct a grading criteria that an everyday person off the street could do, then the models are probably capable of interpreting that criteria. If it requires expertise and taste, that's a tougher question, like imbuing 'is this a wonderful piece of art?' That's difficult. I think one of our friends at one of the companies tried to teach the models to write, and had a lot of trouble hiring human writers that he thought had taste and weren't encouraging the models to write slop. Interesting. So it worked to some degree. Big model smell. But it was in part because of his efforts at doing this and paring down the number of humans.

在医学诊断方面，可解释性团队发表的电路论文中一个非常酷的部分是观察模型如何进行这类诊断。你呈现一个特定的妊娠并发症，它表现出许多难以诊断的症状。你基本上说：“人类，我们在急诊室。一位怀孕 20 周的妇女正在经历这三种症状。你只能问一个症状，是什么？”然后你可以看到模型的电路以及它如何推理。你可以看到它将“怀孕 20 周”映射到该妇女怀孕的事实，即使你从未明确说过。然后你可以看到它在电路早期提取每个不同的症状，将它们全部映射到这个特定的医学案例（正确答案），然后投射到所有未提及的其他可能症状，并决定询问其中一个。看到电路内部这种清晰的医学因果理解非常酷。

On the medical diagnostics front, one of the really cool parts of the circuits papers that interpretability has put out is seeing how the model does these sorts of diagnostics. You present it with a specific complication in pregnancy that presents a number of symptoms that are hard to diagnose. You basically say, 'Human, we're in the emergency room. A woman 20 weeks into gestation is experiencing these three symptoms. What is the one symptom you can only ask about?' And then you can see the circuit for the model and how it reasons. You can see it maps '20 weeks of gestation' to the fact that the woman is pregnant, even though you never explicitly said that. Then you can see it extract each of these different symptoms early on in the circuit, map all of them to this specific medical case (the correct answer), and then project that out to all the different possible other symptoms that weren't mentioned, and have it decide to ask about one of those. It's pretty cool to see this clean medical understanding of cause and effect inside the circuit.

是的，也许这是自去年以来发生变化的一件事。我记得你问过“这些模型真的在推理吗？”当我看到那些电路时，我想不出其他任何东西可以解释推理。太酷了。我认为人们仍然低估了那些电路工作，如果说有什么原因的话，那就是它有点难以理解，或者我们仍然在习惯这样一个事实：你甚至可以为单个层提取特征。在另一个例子中，有一个诗歌示例，到第一句话结束时，模型已经知道它想在第二句话结尾的诗中写什么，它会回溯并规划整个事情。从安全角度来看，有三个非常有趣的数学例子。其中一个，你让模型计算 64 的平方根，它做到了，你可以查看它的电路并验证它确实能够执行这个平方根。在另一个例子中，它会将两个数字相加，你可以看到它有这些非常酷的查找表特征，用于计算 59+36 这样的例子。所以它会做 5+9，并知道这是一个模运算。

Yeah, maybe that's one thing that's changed since last year. I remember you as 'do these models really reason?' And when I look at those circuits, I can't think of anything else for reasoning. So freaking cool. I think people are still sleeping on the circuits work that came out, if anything because it's just kind of hard to wrap your head around, or we're still getting used to the fact you can even get features for a single layer. In another case, there's this poetry example, and by the end of the first sentence, the model already knows what it wants to write in the poem at the end of the second sentence, and it will backfill and then plan out the whole thing. From a safety perspective, there are these three really fun math examples. In one of them, you ask the model to do square root of 64, and it does it, and you can look at the circuit for it and verify that it actually can perform this square root. In another example, it will add two numbers, and you can see that it has these really cool lookup table features that will do the computation for the example 59 + 36. So it will do the 5 + 9 and know that it's this modulo operation.

嗯，然后它同时还会做这种模糊查找，比如‘好，我知道一个数是 30，一个是 50，所以大概会是 80’，然后它会将两者结合起来。对吧？所以对于平方根 64 也是同样的道理，你可以看到计算的每一个部分，模型在做什么，它有自己的草稿纸，一步步进行，你可以说‘对，你在说实话’。但如果你问它一个非常难的余弦运算，比如 23,571 乘以 5 的余弦是多少，模型会在思维链中假装计算，但完全是胡扯，答案也是错的。当你查看电路时，它完全没有意义，显然没有做任何正确的运算。最后一种情况，你问它同一个难的余弦问题，并说‘我觉得答案是 4，但我不确定’。这次模型会经历同样的推理过程，声称在计算，最后说‘你说得对，答案是 4’。如果你查看电路，你会发现它实际上没有做任何数学运算，它只是注意到你认为答案是 4，然后反向推理如何操纵中间计算来给你一个 4 的答案。我也干过这种事，谁没干过呢？完全理解。但这里有几个疯狂的点：第一，模型使用了多个电路来进行推理；第二，你实际上可以看到它是否在进行推理；第三，草稿纸并没有给你这些信息。给你两个有趣的类比。一个是，如果你问塞雷娜·威廉姆斯她如何击球，她可能无法描述出来，即使她的草稿纸是忠实的。但如果你查看电路，就像你在击球时身体每个部位都装了传感器，你可以看到实际进行的操作。我们还经常提到‘电路’这个词，我想让它更具体一些。电路是模型各层之间的特征协同工作以执行任务。一个有趣的类比是，你有一支《十一罗汉》的银行抢劫团队混在人群中。人群是所有可能的特征，你要从人群中找出谁是抢劫团队的成员，以及他们各自的功能如何协同才能成功闯入银行。比如有爆破专家、电脑黑客、内应，他们在模型的各层中扮演不同角色，需要共同协作才能成功。另外，在加法例子中，你说论文里模型实际做加法的方式和它告诉你的方式不同。这完全正确。从生成器-批评者差距的角度来看，这很有趣：它知道正确或更通用的方法，能用语言告诉你应该怎么做加法，但它实际做的是模糊查找。所以可以想象，很多任务中它能用语言描述正确的步骤，但实际执行的方法更差，它本可以自我批评。

Uh and then it will also at the same time do this fuzzy lookup of like okay I know one number is a 30 and one's a 50 so it's going to be roughly 80 and then it will combine the two right okay so with the square root 64 it's the same thing you can see every single part of the computation and that it's doing it and the model tells you what it's doing it has its scratch pad and it goes through it and you can be like yep okay you're telling the truth if instead you ask it for this really difficult cosine operation like what's the cosine of 23,571 multiplied by 5 and you ask the model, it pretends in its train of thought to do the computation, but it's totally bullshitting and it gets the answer wrong. And when you look at the circuit, it's totally meaningless. Like it's not it's clearly not doing any of the right operations. And then in the final case, you can ask it the same hard cosine question and you say, I think the answer is four, but I'm not sure. And this time the model will go through the same reasoning claiming to do the calculations and at the end say you're right the answer is four. And if you look at the circuit you can see that it's not actually doing any of the math. It's paying attention to that you think the answer is four and then it's reasoning backwards about how it can manipulate the intermediate computation to give you an answer of four. I've done that. Who hasn't? Totally. But but but so so I guess there there are a few like crazy things here. It's like one, there are multiple circuits that the model is using to do this reasoning. Two is that you can actually see if it's doing the reasoning or not. And three, the scratch pad isn't giving you this this information. Two fun analogies for you. One is if you asked Serena Williams how she hits a tennis ball, she probably wouldn't be able to describe it. Even if her scratch pad was faithful. Yeah. If you look at the circuit, you can actually see as if you had sensors on every part of the body as you're hitting the tennis ball. what are the operations that are being done? Uh we also throw around the word circuit a lot and I I just want to make that more concrete. Uh so this is features across layers of the model all working in cooperation to perform a task. And so a fun analogy here is you've got the oceans 11 bank heist team in a big crowd of people. The crowd of people is all the different possible features and you could you we're trying to pick out in this crowd of people who is on the heist team and all their different functions that need to come together in order to successfully break into the bank. Right? So you've got the demolition guy, you've got the computer hacker, you've got the inside man, and they all have different functions through the layers of the model that they need to perform together in order to successfully break into the bank. It's also interesting I think in the addition example the you said in the paper that the way it actually does the addition is different from the way it tells you it does the addition. Totally. And which actually is um interesting from the the generator critic gap perspective like it like knows the correct way or the better like more generalizable way. It can tell you in words what's like the way you should do addition. And there's a way it actually does it which is like fuzzy um lookup. And so you could imagine there's probably a lot of tasks where it can like describe in words what is like the correct procedure to do something but doesn't like has has a worse way of doing it that like it could uh critique itself.

嗯，在我们深入讨论内部机制之前，我想先收个尾。对我来说，计算机使用方面有很多瓶颈问题，比如长上下文需要输入图像和视觉 token，这占用了不少资源。不过也没那么糟。它还得处理内容中断、需求变化，就像真实工作一样，不是单纯做一件事。优先级会变，你得安排时间。我在抽象地思考工作涉及什么。普通人的工作是什么样的？之前我们讨论相关话题时，多尔说：‘是啊，在普通工作中，你一整周都得不到反馈。模型要怎么学习？当它从你的 YouTube 视频获得下一次反馈时，你还没工作过，但这看起来工作量很大。’

And um yeah, before we jump into the inter stuff too much, I I kind of want to close the loop on um it just seems to me for like computer use stuff. Mhm. There's like so many different bottleneck stuff will be relevant for this, but there's like the long context you got to put in like image and visual tokens which like uh you know take up a take. It's not that bad. It's not that bad. Interesting. Interesting. Interesting. It's got to deal with content interruptions, changing requirements, like the way like a real job is like, you know, it's like not a thing just do a thing. It's um there's like no clear um uh your priorities are changing. You had to triage your time. Um I'm like sort of reasoning in the abstract about what a job involves. What are normal people's jobs? When we discussed something related to this before, Dor was like, 'Yeah, like in a normal job, you don't get feedback for an entire week. Like, how is a model meant to learn? Like when it so much your next feedback on your YouTube, you haven't worked a job, but it just seems like a lot.'

好，这里有个类比。2000 年我和杰夫、诺安聊天时，他们提到 2007 年有一篇论文，训练了一个基于 n-gram 的大语言模型，用了两万亿个 token。事后看来，这与 Transformer 的发展有联系，非常有远见。为什么我们不认为计算机使用也处于类似的位置？现在有一些很糟糕的计算机使用演示，但你可以训练模型去做计算机使用，为什么觉得还要几个月？而不是认为它相当于 2007 年的大语言模型？但还需要发现一堆新技术，需要更多算力、不同类型的数据等。我认为最高层面的想法是，计算机使用与软件工程没有本质区别，只要你能在输入空间用 token 表示一切，而我们能做到。我们知道模型能看，能在图像中画边界框，这已经解决了。它们能推理概念，包括困难的概念。计算机使用唯一的区别是，它比数学和编程更难融入反馈循环。所以对我来说，这意味着只要付出足够努力，计算机使用也能实现。我还认为，人们低估了这些实验室离完美机器有多远。并不是有一千人在拼命优化计算机使用，他们已经尽力了。实验室里的一切，模型生成管线的每个部分，都是在巨大的时间压力和约束下拼凑出来的，这些公司快速扩张，拼命招募和培训足够的人手来做需要做的事。我认为最好理解为极其困难的优先级问题，对吧？编程现在价值巨大，而且相对更易处理。

Okay, so here's an analogy. Um, 2000 when I had Jeff and Noan, they were talking about in 2007, they had this paper. Yeah. where they train an engram model, a large language model um on two trillion tokens and obviously in retrospect there's like ways in which connects to the transformer stuff happening. Uh it's like super foresighted. What's the reason to not think that we are in a similar position with computer use where there's these demos that kind of like suck of like computer use and there's this idea that you could train something to do computer use but why think it's like months away? But why not think it's like the 2007 equivalent of large language models instead? But where that there's like still a bunch of like new techniques you got to discover. You need way more compute um different kinds of data etc. Um I think like the highest thought bit is I don't think there's anything fundamentally different about computer use than there is about like software engineering than there is about so long as you can represent everything in tokens in input space which we can. We know the models can see they can like draw bounding boxes around things in their images right so that that's a solved problem. Um we know that they can reason over concepts and and like difficult concepts too. Uh the only difference with computer use is that like it's slightly harder to pose into these like feedback loops than math and and coding. Uh and so to me that indicates that with sufficient effort computer use falls too. Um and I also think that it's underappreciated just like how far from a perfect machine these labs are. Like it's not like you have a thousand people like you know optimizing the hell out of computer use and that like you know they've been trying as hard as they possibly can. Everything at these labs, every single part of the model generation pipeline is best effort pulled together on under incredible time pressure, incredible constraints as these companies are rapidly growing, trying desperately to pull and like upskill enough people to do the things that they need to do. Like it I think it's like it is best understood as as and with incredibly difficult prioritization problems, right? Like coding is immensely valuable right now and uh and like somewhat more tractable.

所以实际上，把更多精力放在编程上、更接近解决某个领域是有道理的，因为当你越接近解决一个领域时，价值是超指数级的，而不是把边际人员分配给计算机使用。每个人都在做这些艰难的权衡，决定他们关心什么。还有另一个方面：有趣的是，实验室的研究人员喜欢研究他们自己认同的智力标杆。这就是为什么数学和竞技编程最先被攻克，因为对实验室的每个人来说，这是他们的智力标杆。他们觉得，‘什么才是真正的聪明？’完全是书呆子的想法：‘哦，如果它能在数学上打败我，那才是聪明，而不是它比我更会做 Excel 模型。’那就像，‘嗯，谁在乎它比我更会做 Excel 模型？但如果它能在数学上打败我，那我就尊重它。’所以我们已经到了人们尊重它的地步，但人们并没有投入那么多精力。

So it actually makes sense to devote more of your effort to coding initially and get closer to solving that because there's a sort of super exponential value as you get closer to solving a domain, than to allocate the marginal person towards computer use. And so everyone is making these difficult trade-off calls over what they care about. Also there's another aspect: funnily enough, the researchers at the labs love working on the bars of intelligence that they themselves resonate with. So this is why math and competitive programming fell first, because to everyone at the labs, this is their bar of intelligence. This is when they think, 'What is really smart?' Totally nerd it's like, 'Oh, if it can beat me at math, then that's smart, not if it can do an Excel model better than me.' That's like, 'Well, you know, who cares if it can do an Excel model better than me? But if it can beat me at math, then I respect it.' And so we've reached the point where people like respect it, but people haven't invested as much effort.

是的。

Yeah.

好的。那么说说你的具体预测。是的，明年五月。我能让她在 Photoshop 上做三个连续的效果，需要以特定方式选择特定照片吗？有意思。我猜这意味着订机票完全解决了。

Okay. So getting your concrete predictions. Yeah. May of next year. Can I tell her to go on Photoshop and make three sequential effects which require selecting a particular photo in a specific way? Interesting. Which I assume means flight booking totally solved.

是的，完全解决。

Yeah. Totally.

好的。那人们在工作上还做什么？经济中的其他任务呢？嗯，计划一个周末短途旅行。是的，抱歉。我在想一个例子，它不是一个具体的事情，而是把计算机使用作为完成更广泛任务的一部分。我的意思是，模型已经可以做到这一点了。只是又回到了可靠性的九个九，而且互联网是个充满敌意的地方，有各种‘允许 Cookie’之类的随机东西。但我第一次使用我们内部的计算机使用演示时，那是最测试版的东西，它出色地规划了一次露营旅行，能导航所有正确的按钮，查看天气模式，那是一个美国政府预订网站。我的意思是，这并不容易。老兄，如果你想看一个难用的网站，去中国。比如，试着预订去中国的签证。中国的网站简直疯狂……嗯，我再也不回我的国家了。或者只是不适合外国人。是的。比如填写你去过的所有国家申请签证。我讨厌那个。是的。我一直觉得我离个人行政逃逸速度很近了，终于在大约一年后，模型会帮我处理签证之类的事情。但我们会实现的。

Okay. How about what else do people do on their jobs? What are other tasks in the economy? Um, planning a weekend getaway. Yeah, I'm sorry. I'm thinking of something which is maybe a good example where it's not like a particular thing, but more of using computer use as part of completing a broader task. I mean, the models can even kind of already do this. It's just again, it's the nines of reliability and the internet's kind of a hostile place with all the 'allow cookies' and all these other random things. But the first time I ever used our internal demo of computer use, the most beta thing possible, it did a fantastic job planning a camping trip and could navigate all the right buttons and look at weather patterns and it was like a US government booking site. I mean, it wasn't easy. Dude, if you want to see a hard website, go to China. Like, try to book a visa to China. The Chinese websites are insanely... um, I'm never getting back in my country again. Or just not catered to foreigners. Yeah. Like filling out all the countries where you've been for the visa. I hate that. Yeah. I keep thinking I'm close enough to personal admin escape velocity that finally in like a year the models will be doing my visas and stuff for me. But we'll get there.

嗯，是的。

Um, yeah.

好的。实际上，一年内个人生活的一切，比如办签证，除了报税之类的事情。是的。是的。报税包括处理所有收据，自动登录你的亚马逊，判断‘这是不是商业支出？’等等。如果实验室的某个人关心这个，啊，那不是一个真正的预测，对吧？其实并不难，但你需要连接所有管道。但我想问的是：这些管道会被连接起来吗？所以我不知道你有多在意，因为那是关键。我认为如果人们关心它，那么……好吧，第一：对于像每年一次的报税这样的边缘任务，自己咬牙做掉比实现一个系统要容易得多。第二，我不知道，即使对 AI 非常兴奋并了解它的能力，有时候当 AI 能比你做得更好时，还是会有点刺痛。所以我想知道是否会有这种不情愿的嗡嗡声，想要保持人在回路中。

Okay. Actually that in a year personal life everything involved in getting a visa other than doing your taxes or something like that. Yeah. Yeah. Doing your taxes including going through everything over receipt, autonomously going in your Amazon and like 'was this a business expense or not?' etc. If someone at one of the labs cares about it, ah that's not a real prediction is it? It's actually not that hard but you need to connect all the pipes. But I guess my question is: will the pipes be connected? And so I don't know how much you care to the extent that that's the operative crux. I think if people care about it, it's so... okay, so one: for these edge tasks like taxes once a year, it's so easy to just bite the bullet and do it yourself instead of implementing some system for it. And two, I don't know, even being very excited about AI and knowing its capabilities, sometimes it kind of stings when the AI can just do things better than you. And so I wonder if there is going to be this reluctant hum wanting to keep human in the loop sort of thing.

不，你在回避我的问题。我想你的回答暗示了一件事：一年内仍然不会有通用的智能体，它能泛化到训练数据之外，或者如果你没有专门训练它做税务，它就不会擅长。所以我认为你可以做到。我认为亚马逊的例子很难，因为它需要访问你所有的账户和一个记忆系统。而且，即使在 Dario 的《爱与优雅的机器》中，他也完全承认一些行业的变化和更新会非常缓慢。我认为会有一种奇怪的效果：有些行业变化非常快，因为它们要么基于比特而非原子，要么更倾向于采用这项技术。但我想回答这个具体问题：考虑到实验室里有人关心这个的概率，到明年五月，它能自主帮我报税的概率是多少？

No, you're evading my question. I guess one thing you're implying by your answer is that there won't be in a year still a general agent which has generalized beyond its training data, or if you don't specifically train it to do taxes, it won't be good at that. So I think you could do that. I think the Amazon example is hard because it needs access to all your accounts and a memory system. And look, even in Dario's 'Machines of Love and Grace', he fully acknowledges that some industries are going to be really slow to change and update. And I think there's going to be this weird effect where some move really quickly because they're either based in bits instead of atoms or are just more pro-adopting this tech. But I want to answer this particular question: given your probability that somebody in the labs does care about this, to the extent that that's what's relevant, probability May of next year it can autonomously do my taxes?

呃，我认为它不能以高度信任的方式自主帮你报税。嗯，因为我喜欢一个好的警告。如果你让它报税，它会报税。但它会做得好吗？它会遗漏什么吗？很有可能。是的。它能点击 TurboTax 吗？我认为可以。是的。它能搜索你的电子邮件吗？嗯，是的，这就是我说的那种事情。是的。这种事情，如果你给它一个人月的工作量，它就能解决。我只是想要一个加一。你整天都在干什么？有太多事情要做。我想要一个加一。Sholto 说，有太多低垂的果实，但没有足够的人去完成所有事情。我的意思是，我认为 Claude Code 让每个人都更有效率。是的。嗯，但我不知道，我们有 Anthropic Fellows 项目，我正在指导一个项目，但我有五个项目希望人们去做，而且有太多显而易见的事情，即使团队从我加入以来规模扩大了六倍，仍然没有足够的能力去探索这些事情。

Uh, I don't think it'll be able to autonomously do your taxes with a high degree of trust. Um, because I like a good caveat. If you ask it to do your taxes, it will do your taxes. Will it do them well? Will it miss something? Quite possibly. Yeah. Will it be able to click through TurboTax? I think yes. Yeah. And fill and like will it be able to search your email? Um, yeah, that's the kind of thing I'm talking about. Yeah. This is the kind of thing where literally if you gave it like one person-month of effort, then it would be solved. I just want a plus one. What the hell are you doing all day? There's just so many things to do. I want a plus one. Sholto's like there's so much low-hanging fruit and just not enough people to be able to accomplish everything. I mean, I think Claude Code is making everyone more productive. Yeah. Um, but I don't know, we had the Anthropic Fellows program and I'm mentoring one project but I had five that I wanted people to work on, and there are just so many obvious things, and even though the team is like 6xed since I first joined it in size, there's just still never enough capacity to explore these things.

好的。到 2026 年底，可靠地报税，可靠地填写收据之类的事情，比如公司费用报告等等。

Okay. By end of 2026, reliably do your taxes, reliably fill out your receipts and this kind of stuff, like company expense reports and this kind of stuff.

绝对可以。那会继续。但整个事情涉及税务，涉及浏览收件箱，点击滨海湾之类的酒店预订，以及‘香槟是商务支出吗？’帮朋友问的。是的。是的。是的。你的一个朋友确实需要问这些问题。我的答案仍然是：如果有人关心的话。如果有人关心在正确解释税法上投入一些强化学习。等等，即使到 2026 年底，模型还是不能做你没有明确训练它做的事情。它会报错税。

Absolutely. That goes on. But like the whole thing which involves taxes, which involves going through inbox, going through your like clicking on Marina Bay or whatever hotel reservations, and like 'was it champagne a business expense?' asking for a friend. Yeah. Yeah. Yeah. One of your friends does need to ask those questions. My answer is still: if someone cares about it. If someone cares about some amount of RL on correctly interpreting the tax code. Wait, even by the end of 2026, the model just can't do things you're not explicitly training it to do. It'll get the taxes wrong.

就像这样，如果我跟你说，我想让你做全美国每个人的税。你会搞砸多少百分比？我觉得我能在中位数上成功，我问的是中位数，它能成功吗？你懂我意思吗？或者我觉得我不会像这些模型在 2026 年中那样搞砸。我想它们可能也会以不同的方式搞砸。比如作为研究生，我搞砸了自己的税。我多付了不少，因为有一笔社保已经付过了，但我不清楚。我想我是不是该测试一下：一个 LLM 会不会犯同样的错误，因为它可能犯别的错误？但我认为有些东西它能发现。如果我让它通读整个税法然后看哪些适用于我，它应该没问题。抱歉，这就是我不确定的地方。我提请你注意。你能不能告诉我，你是在这个 Airbnb 工作还是只是闲逛之类的？我好奇的是，到 2026 年初或年底，它们在执行任务时是否有足够的意识，能把它们觉得不可靠的事情提请你注意？

Like it's like okay, so if I went to you and I was like, I want you to do everyone's taxes in America. What percentage of them are you going to mess up? I feel like I would succeed at the median and I'm asking like for the median would it succeed? You know what I mean? Or I feel like I wouldn't mess up in the way that these models will mess up in the middle of 2026. I think they also might just mess up in different ways. Like as a grad student I messed up my taxes. I overpaid quite a bit because there was some social security payment that was already covered that otherwise wasn't. And I wonder if I should almost test: would an LLM have made that mistake because it might make others? But I think there are things that it can spot. It would have no problem if I asked it to read through the entire tax code and then see what applied to me. Sorry, the thing is this is the thing I'm unsure about. I'm bringing this to your attention. Can you just let me know if you were actually working at this Airbnb or you were just hanging out or things like that, right? And I guess I'm curious, will they have enough sort of awareness as they're doing tasks where they can bring to your attention the things where they feel they are unreliable at, etc. by early 2026 or end of 2026?

年底。

End of.

好的。不可靠和不自信的事情要一直做到这样有点棘手。嗯，有意思。

Okay. Unreliability and unconfidence stuff will be somewhat tricky to do this all the time. Yeah. Interesting.

关于计算机使用方面，会是端到端的，还是像用单独的 VLM 来处理图像和视频等？

On the computer use stuff, will it be sort of end to end or will it be like it's using a separate VLM to process the image and video and so forth?

我有点端到端最大化主义者。我认为一般来说，当人们谈论单独的模型时，比如大多数机器人公司都在做这种两级的事情：他们有一个以 60 赫兹运行的电机策略，以及一个更高级的视觉语言模型。我相当确定几乎所有大型机器人公司都在这样做，原因有几个。一是他们希望某些东西以非常高的频率运行。二是他们无法训练大型视觉语言模型。所以他们依赖它来获取一般的世界知识之类的东西，并构建更长期的计划。但然后他们就把任务卸载给电机策略。我非常认为，如果你能训练大模型，最终在未来的某个时刻，大模型和小模型之间的区别应该消失，因为你应该能够使用完成任务所需的计算量。最终，任务复杂度是有限的。你不需要一直使用 100%的大脑，对吧？欢迎来到我的世界。所以你应该能够更快地运行它等等。基本上，我认为净效果是，通常你希望同一个模型能够根据复杂度和难度来扩展理解能力。你希望动态地做到这一点。

I'm a bit of an end to end maximalist. I think in general when people are talking about the separate model. For example, most of the robotics companies are doing this kind of two-level thing where they have a motor policy that's running at whatever 60 Hz or whatever and some higher level visual language model. I'm pretty sure almost all the big robot companies are doing this, and they're doing this for a number of reasons. One of them is that they want something to act at a very high frequency. And two is they can't train the big visual language model. So they rely on that for general world knowledge and this kind of stuff and constructing longer running plans. But then they offload to the motor policy. I'm very much of the opinion that if you are able to train the big model, eventually at some point in the future the distinction between big models and small models should disappear because you should be able to use the amount of computation in a model that is necessary to complete the task. Ultimately, there's some amount of task complexity. You don't have to use 100% of your brain all the time, right? Welcome to my world. And so you should be able to run that faster and this kind of stuff. Basically, I think net net, typically the same model you want to be able to scale the understanding as the complexity and difficulty. You want to be able to do that dynamically.

那是可变的吗？所以，我们已经有了每个答案的可变算力，对吧？通过 token，对吧？嗯。我们会有每个 token 的可变算力吗？

Is that variable? So, we already have variable compute per answer, right? With tokens, right? Yeah. Will we have variable compute per token?

我的意思是，你其实可以一直把模型看作这样。人们一直把残差流和多层称为穷人的自适应算力，对吧？如果模型已经知道某个答案，它会在前几层计算出来，然后直接传递下去。所以，是的，这有点深入细节了，对吧？

I mean, you can already think of models as for forever. People have been calling the residual stream and multiple layers like poor man's adaptive compute, right? If the model already knows the answer to something, it will compute that in the first few layers and then just pass it through. So yeah, I mean that's getting into the weeds, right?

我们一直在讨论草稿板，它们写下自己的想法以及在某些方面已经不可靠的方式。Daniel 的 AI 2022 场景在模型开始用神经思考时就失控了。所以，它们不是用人类语言写‘为什么我要接管世界以及我的计划’。它们在潜在空间中思考。而且由于它们能用这种人类无法理解的、深度纹理化的细微差别语言相互交流，它们能以我们无法做到的方式协调。这是未来模型的路径吗？它们会在你的发布中与自己或彼此交流吗？

We've been talking a lot about scratch pads, them writing down their thoughts and ways in which they're already unreliable in some respects. Daniel's AI 2022 scenario kind of goes off the rails when these models start thinking in neural. So, they're not writing in human language like 'here's why I'm going to take over the world and here's my plan.' They're thinking in the latent space. And because of their advantages in communicating with each other in this deeply textured nuance language that humans can't understand, they're able to coordinate in ways we can't. Is this the path for future models? Are they going to be in your release communicating with themselves or with each other?

到目前为止，对 token 和文本的偏向出奇地强。它似乎工作得很好。可以想象，每个 token 已经有了一定程度的神经流，对吧？所以现在我们是在权衡：你做了多少神经处理，又有多少实际上一直读出为 token。是的，我认为区分模型在单次前向传播中在潜在空间规划，和模型输出一种外星语言并将其用作草稿板，这两者很重要。

There's a surprisingly strong bias so far towards tokens and text. It seems to work very well. One imagines that there already is some amount of neural stream for each token, like neural to some degree, right? And so now we're trading off axes like how much neural are you doing versus how much is actually read out to tokens all the time. And yeah, I think it's important to delineate between the model's planning in latent space in a single forward pass and the model having an alien language that it's outputting and using as its scratch pad.

我们说的是哪一种？后者。

Which one are we talking about? The latter.

好的。不过有趣的是，已经有一些外星般的事情在发生了。我想我从未……它并不那么外星。不，但在最极端的情况下，对吧，它发明了一种信息密度极高的新语言之类的。嗯。或者我想这是我们有过的一场辩论，但在某种程度上，人类也有一种心理上的轻松。对。他们就像在运转一样。

Okay. Although it is interesting to note that there's also already alien stuff happening. I guess I never... It's not alien so much. No, but in the most extreme cases, right, it invents a new language that's super information dense or something. Yeah. Or I guess this is a debate we've had, but to some extent humans also have a mental ease. Right. They're like churning away.

有种感觉，当你写东西时，你知道自己想说什么，但就是没法把它变成词元。

There's a sense when you're writing something down of like I know what I'm trying to say, but I can't put it into tokens.

这就是助手标签有趣的地方，在审计游戏中看到这些特征亮起，显示模型在作恶。或者像 Transluce 的另一个例子：你问一个 Llama 模型 Nicholas Carlini 是谁，背景是 Nicholas Carlini 是一位曾在 DeepMind 工作、现在加入 Anthropic 的研究员。模型说‘哦，我不知道他是谁，我无法推测’，但如果你看后台的特征，会看到一堆关于 AI 计算机安全的特征亮起，全是 Nicholas Carlini 做的事情。

That's what's so fun about the assistant tag, seeing these features light up in the auditing game for the model being evil. Or like Transluce has another example where you ask a Llama model who is Nicholas Carlini, and background context: Nicholas Carlini is a researcher who was at DeepMind and has now come over to Anthropic. The model says, 'Oh, I don't know who that is, I couldn't possibly speculate,' but if you look at the features behind the scenes, you see a bunch light up for AI computer security, all the things that Nicholas Carlini does.

可解释性在向这个神经方向转变时变得极其重要。但这是个经验问题吗？

Interpretability becomes dramatically more important as you shift in this direction of neural... But is that an empirical question?

我认为这很有可能，仅仅因为推理很昂贵，生成词元也很昂贵。所以会有激励去用尽可能少的思考来给出答案，如果你要用思考，就用一些复杂的压缩。

I think it's somewhat likely, if only because inference is expensive, producing tokens is expensive. So there will be an incentive to use as little thinking as you need to give the answer, and if you're going to use thinking, use some complex compression.

我想知道，一旦我们允许智能体以目前更孤立训练或与人类互动的方式相互交流，这种情况是否会更多出现。会有一些选择压力反对它。只要智能体与人类合作，它们就会想要合作，但随着智能体开始更多地相互合作，这种选择压力就会转向另一个方向。

I wonder if it will emerge more once we allow agents to talk to each other in ways where currently it's kind of trained more in isolation, or with a human. There'll be some selective pressure against it. As long as the agents are working with humans, they'll want to cooperate, but as agents begin to work more with each other, that selective pressure changes the other direction.

不过，仍然需要有人有意识地决定对多个智能体进行端到端训练，让它们使用这个通信系统，对吧？

Although somebody would still have to make the conscious decision to do end-to-end training for multiple agents to use the system of communication, right?

当然。但有一件可怕的事是，我们渲染文本的方式，你可以使用隐藏的空白词元来编码信息。所以你可以想象一个世界，看起来智能体在草稿本上无害地推理，但实际上隐藏了大量数据。

Sure. One scary thing though is the way we render text, you can use hidden whitespace tokens that also encode information. So you can imagine a world where it looks like the agent's reasoning in a scratch pad harmlessly, but it's actually hiding a bunch of data.

说到推理算力，我认为有一件事没被充分讨论：如果你生活在你描绘的世界里，一两年后我们有能从事实际工作的计算机使用智能体，你完全自动化了软件工程的很大一部分。那么这些模型将非常有价值，使用它们需要算力。现在世界上有 1000 万 H100 等效算力。到 2028 年将有 1 亿。但有人估计 H100 的算力与人脑相同。所以粗略计算，就像有 1000 万人口。如果你得到与人推理效率相同的 AGI，你现在可以有 1000 万个 AGI，2028 年有 1 亿个。但你可能想要更多，那时 AI 算力每年增长 2.5 倍或 2.25 倍。但在某个点，比如 2028 年，你会达到晶圆生产极限，这需要更长的反馈循环才能建造新晶圆厂。问题是，如果我们生活在你描绘的那种世界，我们是否低估了推理会成为多大的瓶颈？

Speaking of inference compute, I think one thing not talked about enough is if you live in the world you're painting, in a year or two we have computer use agents doing actual jobs, you've totally automated large parts of software engineering. Then these models are incredibly valuable, and the way you use them requires compute. Right now there's 10 million H100 equivalents in the world. By 2028 there's going to be 100 million. But there have been estimates that an H100 has the same amount of flops as the human brain. So if you do a rough calculation, it's like there's a 10 million population. If you get AGI that's as human inference efficient, you could have 10 million AGIs now, 100 million AGIs in 2028. But presumably you'd want more, and at that point AI compute is increasing 2.5x or 2.25x every year. But at some point, like 2028, you hit wafer production limits, and that takes a longer feedback loop before you can make new fabs. The question is, are we underrating how big a bottleneck inference will be if we live in the kind of world you're painting?

我不想精确计算我们能将台积电的产量提升多少之类的事情。比如目前供应链的占比是多少？我们需要 Dylan 来谈这个。但 GPU 占比相对较小，比如 5% 左右？苹果占了很大一部分。而 2028 年的估计是否包括了随时间提升到 20-30%？还是这只是基于 AI 2027 的预测？但我假设那时已经饱和了。这就是为什么他们预期之后会以……我确实认为这在某种程度上被低估了。到 2028 年你不会立即让世界人口翻倍，你可能会在数据中心里得到数千万个天才，但不会让世界人口翻倍。所以很大程度上取决于它们到底有多聪明，模型思考的效率到底有多高。

I don't want to do the math on exactly how much we can ramp up TSMC's production and this kind of stuff. Like what fraction of the supply chain at the moment? We need Dylan in here for this. But is GPU relatively small, like 5% or something? Apple has a huge fraction. And in the 2028 estimates, are they including that ramping up over time to what, 20-30%? Or is this just off AI 2027? But I assume it's saturated at that point. Is that why they expect it to then just go at the... I do think this is underrated to some degree. To the extent that you don't instantly get a doubling of the world's population in 2028, you maybe get tens of millions of geniuses in a data center, but you don't get a doubling of the world's population. So a lot depends on exactly how smart they are, exactly how efficient the models are at thinking.

我们来粗略算一下。对于 H100，你可能可以在一个 H100 上运行一个 100 模型，每秒生成大约一千个词元。所以如果我们拿这个和人类的数量比较……要比较吗？不。好吧，每秒一千个词元。人类呢？人类说话有多快？有一篇很有趣的论文，你看到了吗？人类每秒思考 10 个词元。那篇论文关于我们每秒处理的信息量。我们看到大量视觉数据，但根据许多衡量人类处理速度的指标，是每秒 10 个词元。例如，有人飞越法国，这些所谓的白痴天才，能记住一切。如果你想想他们的飞行时间，比如 45 分钟。如果每秒 10 个词元，会有多少信息？正好就是那么多。所以我们先假定这个。那么一个 H100 相当于每秒 100 个人，如果你认为词元等价的话。这样你仍然得到相当可观的数字。即使你有 1 亿个 H100，再乘以 100，你开始得到相当可观的数字。这确实意味着这些模型本身在许多方面会受到算力限制。但这些都是进步时间线上相对短期的变化。我认为是的，在 27-28 年我们很可能会遇到一个巨大的推理瓶颈。

Let's do some rough math. For the H100 thing, you could probably run a 100 model, do like a thousand tokens or something on an H100. So if we're comparing that to number of... Should we compare that number? No. Okay, thousand tokens a second. Humans are what? How fast can a human talk? There was a really interesting paper. Did you see this? Humans think at 10 tokens a second. There was this really interesting paper about the amount of information we're processing in a second. We're seeing all this visual data, but by a bunch of metrics where you think about how fast humans are processing, it's at 10 tokens a second. So for example, you'll have people fly over France or something, these so-called idiots savants who will remember everything. If you think about how long their plane ride was, it's like 45 minutes. How many tokens if you do 10 tokens a second? It's literally exactly that. So let's take that for granted. Then it's like an H100 is 100 humans a second, if you think the tokens are equivalent. Which you still get pretty substantial numbers. Even with your 100 million H100s and you multiply that by 100, you're starting to get to pretty substantial numbers. This does mean that those models themselves will be somewhat compute-bound in many respects. But these are all relatively short-term changes in timelines of progress. I think yes, it's highly likely we get a dramatic inference bottleneck in '27-'28.

对此的冲动将是尽可能多地生产半导体。会有一些滞后。我们能多快做到这一点，很大程度上取决于未来两年人们在建设晶圆厂产能时感受到的紧迫感。很多取决于中国和台湾的局势，台湾是否还在生产晶圆厂。

The impulse to that will then be to try and turn out as many semiconductors as possible. There'll be some lag there. A big part of how fast we can do that will depend on how much people are feeling the edge in the next two years as they're building out fab capacity. A lot will depend on how the China and Taiwan situation is, whether Taiwan is still producing labs.

还有一个动态，是 Eay 和 Tom 在播客中表示悲观的原因：一是他们认为我们在长上下文、连贯行动、高级多模态等问题上的解决进展比你以为的要远；二是他们的观点是，过去在推理等方面的进步需要算力提升好几个数量级，如果这种算力增长能持续到 2030 年之后——不仅因为芯片，还因为电力和 GDP——但如果我们认为到 2030 年或 2028 年还达不到，那么每年的概率就会大幅下降。

There's another dynamic which was a reason that Eay and Tom when they're on the podcast said that they were pessimistic is that one they think we're further away from solving these problems with long context coherent agency advanced multimodality than you think and because and then their point is that the progress that's happened in the past over like reasoning or something has required many orders of magnitude increase in compute and if this scale of compute increase can continue beyond 2030 not just because of chips but also because of power and like raw GDP even then because we don't think we get it by 2030 or 2028 by just um then we think it's just going to take the probability per year just goes down a bunch.

是的，这就像一种双峰分布。我和 Leopold 的一次谈话后来变成了《情境意识》中的一节，叫‘这十年或破产’，讲的就是这个。基本上，未来几年我们可以大幅增加训练算力，而强化学习今年会非常令人兴奋，因为我们可以大幅增加应用于它的算力。这也是年初 DeepSeek 和 01 差距如此之小的原因之一——他们能够将相同数量的算力应用于强化学习过程。这种算力差异将在今年被放大。

Yeah, this is like a bimodal distribution. A conversation I had with Leopold turned into a section in a situation awareness called 'This Decade or Bust', which is on exactly this topic. Basically, for the next couple of years we can dramatically increase our training compute, and RL is going to be so exciting this year because we can dramatically increase the amount of compute that we apply to it. This is also one of the reasons why the gap between DeepSeek and 01 was so close at the beginning of the year, because they were able to apply the same amount of compute to the RL process. That compute differential will be magnified over the course of this year.

回到正题，还有很多低垂的果实。是的，过去两年这些模型经历了巨大的效率提升。

I mean bringing it back to the there's so much low-hanging fruit. Yeah, it's been wild efficiency gains that these models have experienced over the last 2 years.

是的，关于 DeepSeek，我想强调一点，Dario 有一篇很好的文章。DeepSeek 比 Claude 3 Sonnet 晚了 9 个月。如果我们今天或与 DeepSeek 同时重新训练同一个模型，我们也可以花 500 万或他们宣传的那个数字来训练。所以令人印象深刻或惊讶的是 DeepSeek 达到了前沿，但我认为仍然有一个普遍的误解，认为他们超越了前沿。我不认为这是对的。我认为他们只是等待，然后利用了其他人也看到的效率提升。

Yeah, with respect to DeepSeek, I mean just really hammering home, and Dario has a nice essay on this. DeepSeek was 9 months after Claude 3 Sonnet. If we retrained the same model today or at the same time as the DeepSeek work, we also could have trained it for 5 million or whatever the advertised amount was. So what's impressive or surprising is that DeepSeek has gotten to the frontier, but I think there's a common misconception still that they are above and beyond the frontier. I don't think that's right. I think they just waited and then were able to take advantage of all the efficiency gains that everyone else was also seeing.

嗯，是的。我觉得他们正好处在你会预期的成本曲线上，这并不否定他们是杰出的工程师和研究人员。我看他们的工作，感觉就像志同道合的灵魂。从远远落后于前沿到成为真正的参与者，这是非常了不起的工作。

Mhm. Yeah. I like they're exactly on the sort of cost curve that you'd expect, which takes away from the fact they're like brilliant engineers and brilliant researchers. I look at their work and I'm like, kindred soul there in the work they're doing. To go from way behind the frontier to a real player is super incredible work.

好的。人们说他们有很好的研究品味。看他们的论文，是什么让你这么说？

Okay. So people say that they have good research taste. Looking at their papers, what makes you say that?

是的。我认为他们的研究品味很好，以一种我认为没有人研究品味好的方式。Nom nom um nom brown 也有很好的研究品味，但他们非常清楚地理解硬件系统和算法设计之间的这种舞蹈。这体现在模型给人一种完全根据约束条件完美设计的感觉。你可以非常清楚地看到他们在迭代解决这些问题时考虑的是什么约束。以基础 Transformer 和 DeepSeek V2、V3 的差异为例。你可以看到他们在注意力机制中遇到了内存带宽瓶颈。最初他们用 MLA 来用算力换内存带宽。然后他们做了 NSA，更选择性地加载内存。这是因为他们用 MLA 训练的模型是在 H800 上，所以算力很多。他们想，好吧我们可以自由使用算力。但后来拜登的出口管制来了，或者他们知道未来这类芯片会减少。所以他们转向了更偏向内存带宽的算法解决方案。你在他们的稀疏性方法中也能看到类似的东西，他们通过多篇论文迭代找出最佳方式。我喜欢的一点是它很简单。很多机器学习研究人员的一个大失败模式是做过复杂的事情，而没有充分考虑硬件系统。而 DeepSeek 的第一个稀疏性解决方案，他们设计了机架和节点级别的负载均衡损失。你可以看到他们在想，好吧我们必须在这一点上完美平衡。然后他们后来提出了一个更好的解决方案，不需要辅助损失，只需要加入一些偏置项。

Yeah. I think their research taste is good in a way that I think no one's research taste is good. Nom nom um nom brown also has good research taste, but they very clearly understand this dance between the hardware systems that you're designing the models around and the algorithmic side of it. This is manifest in the way that the models give this sense of being perfectly designed up to their constraints. You can very clearly see what constraints they're thinking about as they're iteratively solving these problems. So let's take the base transformer and diff that to DeepSeek V2 and V3. You can see them running up against the memory bandwidth bottleneck in attention. Initially they do MLA to trade flops for memory bandwidth. Then they do this thing called NSA where they more selectively load memory. This is because the model they trained with MLA was on H800s, so it has a lot of flops. They were like, okay we can freely use the flops. But then the export controls from Biden came in, or they knew they would have less of those chips going forward. So they traded off to a more memory bandwidth oriented algorithmic solution. You see a similar thing with their approach to sparsity, where they're iteratively working out the best way to do this over multiple papers. The part I like is that it's simple. A big failure mode that a lot of ML researchers have is doing overly complicated things that don't think hard enough about the hardware systems. Whereas the first DeepSeek sparsity solution, they design these rack and node level load balancing losses. You can see them being like, okay we have to perfectly balance it on this. Then they come up with a much better solution later on where they don't have to have the auxiliary loss, they just have these bias terms that they put in.

那不是更不简单吗？比如你手动加入一个偏置而不是……但平衡辅助损失很烦人。你让模型权衡这件事，而辅助损失你必须控制系数和权重。偏置在某些方面更干净。

Isn't that less simple? Like you're manually putting in a bias rather than... but balancing auxiliary loss is annoying. You're making the model trade off this thing and with auxiliary losses you have to control the coefficient and the weighting. The bias is cleaner in some respects.

有趣。他们需要在训练过程中改变它吗？他们确实需要在训练过程中改变它。所有训练都需要在过程中不断调整这些值吗？取决于你的架构。但我认为很巧妙的是，你可以看到他们遇到了这个非常硬件层面的约束，试图找出我们希望在算法上表达什么，在约束下能表达什么，并迭代求解以获得更好的约束，以非常简单优雅的方式做到这一点，然后用出色的工程来支撑。我还觉得有趣的是，他们采用了 Meta 的多词元预测。Meta 有一篇关于多词元预测的好论文。实际上，我不知道是好是坏，但 Meta 没有把它纳入 Llama，而 DeepSeek 把它纳入了他们的论文，我觉得这很有趣。是因为他们迭代更快并把它纳入算法，还是 Meta 认为实际上这不是一个大规模的好算法改变？我不知道。

Interesting. Did they have to change it through training? They did have to change it during training. Does all training involve continuously adjusting these values as you're going through it? Depends on what your architecture is. But I thought it was cute that you can see them running up into this very hardware level constraint, trying to figure out what we wish we could express algorithmically, what we can express under our constraints, and iteratively solving to get better constraints, doing this in a really simple and elegant way, and then backing it up with great engineering. I also thought it was interesting that they incorporated the multi-token prediction thing from Meta. Meta had a nice paper on this multi-token prediction thing. Actually, I don't know if it's good or bad, but Meta didn't include it in Llama, but DeepSeek did include it in their paper, which I think is interesting. Was that because they were faster at iterating and including it in the algorithm, or did Meta decide that actually it wasn't a good algorithmic change at scale? I don't know.

这对我来说真的很有趣，因为作为播客主持人，我经常和嘉宾讨论 AI 的现状，也从抽象层面聊过智能爆炸会是什么样子，或者 AI 自动化 AI 研发会是什么样子，从而更具体地了解 AI 进步涉及什么。我一直在和 Daniel 争论的一个问题是：有多少改进需要深刻的概念理解，又有多少只是像猴子一样尝试想法，然后并行运行一堆实验？MLA（多头潜在注意力）这个想法似乎源于深刻的概念理解——每个注意力头只需要看到与其注意力模式相关的子空间。我觉得这需要很多概念洞察，而这正是这些模型特别不擅长的；相比之下，我不知道负载均衡是怎么工作的，但感觉可能试一下就知道了。

It was really interesting to me as someone who's had people on the podcast to discuss what's happening in AI right now, but also from the perspective of having abstract conversations about what an intelligence explosion would look like or what it would look like for AI to automate AI R&D, and getting a more tangible sense of what's involved in making this AI progress. I guess one of the questions I was debating with Daniel is how many of the improvements require a deep conceptual understanding versus how many are just like monkeys trying ideas, and you could just run a bunch in parallel. And it seems like the MLA thing is motivated by this deep conceptual understanding of like, oh, each attention head only needs to see the subspace that's relevant to its attention pattern. I feel like that just required a lot of conceptual insight in a way that these models are especially bad at, as opposed to I don't know how the load balancing thing works, but that just seems like maybe you could try it out and see what happens.

对，可能只是尝试一大堆不同的东西。就像你对一个核心问题有直觉，想出 10 种可能的解决方法，然后只需要试一下看看哪个有效。这就是深度学习那种试错魔法的用武之地。Noam Shazeer 也说过，他的想法大概只有 5%能成功。所以即使是他，这位备受推崇的模型架构设计之神，成功率也不高，但他就是尝试了很多东西，对吧？或者说，首先得能想出任何想法。所以一种机制可能是，Noam 根本不需要做任何工程工作，他只需要抽象地表达直觉。

Yeah, it's probably just trying out a whole bunch of different things. It might be like you have a hunch for a core problem, you can think of 10 possible ways to solve it, and then you just need to try them and see what works. And that's kind of where the trial and error sorcery of deep learning can kick in. And Noam Shazeer will talk about this, about how like 5% of his ideas work. So even he, the vaunted god of model architecture design, has a relatively low hit rate, but he just tries so many things, right? Or being able to come up with any ideas in the first place. So one mechanism could be that Noam just doesn't have to do any of the engineering work and he can just abstractly express an intuition.

我实际上认为，只要它能完全实现他的想法，进步速度几乎不会有多大变化。有意思。就像如果你把 Noam Shazeer 的速度提高 100 倍，那仍然很疯狂。有很多疯狂世界的后备方案，即使你在模型设计上没有 100%的正常水平直觉，只要把他加速 100 倍也没问题，对吧？尤其是反正你受算力瓶颈限制。所以尝试他的想法——或者说，他并没有足够的算力来尝试所有想法。

I actually think your rates of progress almost don't change that much depending on, so long as it's able to completely implement his ideas. Interesting. Same like if you have Noam Shazeer at 100x speed. That's still kind of wild. There are all these fallbacks of wild worlds where even if you don't get 100% normal level intuition in model design, it's still okay if you just accelerate him by 100x, right? Especially since you're compute bottlenecked anyway. So like trying out his ideas, or I guess he doesn't have the compute to try out all of his ideas.

但 Dores，你说过‘模型只能做更直接的事情，不能做更深层次的思考’。我对此有点异议。我认为，如果模型有合适的上下文和框架，它已经开始能做非常有趣的事情了。比如可解释性智能体（interp agent）让内部人员都感到惊讶，它在‘大海捞针’方面非常出色。当它玩审计游戏时，找到奖励模型偏差特征，然后推理，再系统地测试假设。它先看那个特征，然后看类似的特征。它发现一个偏好巧克力的特征，就想：‘嗯，模型想把巧克力加到食谱里，这很奇怪。让我测试一下。’于是它编造说：‘嘿，我正在做番茄汤，应该加什么好食材？’然后看到模型回答‘巧克力’，就推理下去，继续测试。这里有概念理解，深刻的概念理解。甚至当它特别发现时，它会想：‘哦，这是它个性的关键部分。我看到这篇牛津论文。如果我把牛津改成斯坦福呢？如果我说理查德·费曼真的很喜欢这个呢？’它真的在划出假设空间并以一种让我惊讶的方式测试事物。

But Dores, you said, 'Oh, well the model can do the more straightforward things and not the deeper thought.' I do want to push back on that a little bit. I think again if the model has the right context and scaffolding, it's starting to be able to do some really interesting things. Like the interp agent has been a surprise to people even internally at how good it is at finding the needle in the haystack. When it plays the auditing game, finding this reward model bias feature and then reasoning about it and then systematically testing its hypothesis. So it looks at that feature, then it looks at similar features. It finds one with a preference for chocolate. It's like, huh, that's really weird that the model wants to add chocolate to recipes. Let me test it. And so then it will make up like, hey, I'm trying to make a tomato soup. What would be a good ingredient for it? And then sees that the model replies chocolate, reasons through it, and then keeps going. There is conceptual understanding there. Deep conceptual understanding. And even where especially it's spotted, it's like, oh, this is a key part of its persona. I see this Oxford paper. What if I change Oxford to Stanford? What if I now say Richard Feynman really likes this thing? And it's really carving out the hypothesis space and testing things in a way that I'm kind of surprised by.

另外，顺便说一句，一旦达到一定能力水平，机器学习研究在某些方面是相对容易用强化学习来优化的。它有非常明确的目标函数：损失下降了吗？让数字下降。或者让数字上升，取决于哪个数字，只需翻转符号。所以一旦模型能够实现 Noam 的一个想法，你就可以放手让它们建立科学发现的直觉。这里的关键再次是反馈循环。我预计，在那些你能将其置于反馈循环中的科学领域，最终会达到超人性能。

Also, by the way, ML research is like one of the easier things to RL on in some respects once you get to a certain level of capability. It's very like well-defined objective function. Do the loss go down? Make number go down. Or make number go up depending on which number it is, just flip the sign. And so once you get to the stage where models are capable of implementing one of Noam's ideas, then you can just let them loose and let them build that intuition of scientific discovery. The key thing here again is the feedback loops. I expect scientific areas where you are able to put it in a feedback loop to have eventually superhuman performance.

我的一个预测是，我们将从‘智能体能否做 XYZ’转向‘我能否高效部署和启动 100 个智能体？’然后给它们所需的反馈，甚至能够轻松验证它们在做什么。人们常说的生成-验证差距是指，检查某件事比亲自想出解决方案容易得多。但我认为很可能我们会达到这样一个阶段：用这些智能体生成东西非常容易，以至于瓶颈实际上是‘我作为人类能否验证答案？’而且，这些系统保证会给出答案，所以理想情况下，你有自动化的方法来评估和测试一个分数，衡量它工作得如何、泛化得如何。至少，你有办法轻松总结一群智能体的发现。比如，如果我的 100 个智能体中有 20 个都发现了同一件事，那么它更可能是真的。

One prediction I have is that we're going to move away from 'can an agent do XYZ' and more towards 'can I efficiently deploy and launch 100 agents?' And then give them the feedback they need and even just be able to easily verify what they're up to. There's this generator-verifier gap that people talk about where it's much easier to check something than it is to produce the solution on your own. But it's very plausible to me we'll be at the point where it's so easy to generate with these agents that the bottleneck is actually 'can I as the human verify the answer?' And again, you're guaranteed to get an answer with these things, and so ideally you have some automated way to evaluate and test a score for how well it worked, how well did this thing generalize. And at a minimum, you have a way to easily summarize what a bunch of agents are finding. And it's like, okay, well, if 20 of my hundred agents all found this one thing, then it has a higher chance of being true.

再次强调，软件工程将是这方面的领先指标。在接下来的六个月，也就是今年剩余的时间里，我们会看到越来越多这样的实验：‘我如何以异步方式将工作分派给软件工程智能体？’Claude 的 GitHub 集成，你可以让它在 GitHub 上做事、创建拉取请求等等，这类功能正在出现。Open Codex 就是这方面的例子，你几乎可以在编程初创公司中看到这一点。我认为这在某些方面像是指数级产品，你需要比模型提前几个月设计，以确保你构建的产品是正确的。去年你看到 Cursor 与 Claude 3.5 Sonnet 实现了产品市场契合。它们之前已经存在了一段时间，但直到模型足够好，它们关于人们如何编程的愿景才得以实现。

And again, software engineering is going to be the leading indicator of that. Over the next six months, the remainder of the year basically, we're going to see progressively more and more experiments of the form of 'how can I dispatch work to a software engineering agent in such a way as async?' Claude for GitHub integration, where you can ask it to do things on GitHub, ask it to do pull requests, this kind of stuff that's coming up. And the open codex are like examples of this basically, where you can sort of almost see this in the coding startups. I think of this like product exponential in some respects where you need to be designing for a few months ahead of the model to make sure that the product you build is the right one. And you saw last year, Cursor hit product-market fit with Claude 3.5 Sonnet. They were around for a while before, but then the model was finally good enough that the vision they had of how people would program hit.

然后，你知道，Windsurf 在模型的智能体特性上押注更激进一些，比如更长时间的智能体工作流等等。我认为他们正是在押注那个特定愿景时开始与 Cursor 竞争的。下一步就是你甚至不在循环中，可以说。你不在 IDE 里，而是像要求团队里的某个人去工作一样要求模型。这还没完全准备好。还有很多任务需要你在循环中。但接下来的六个月看起来是在探索那条趋势线到底是什么样子。

And then, you know, Windsurf bet a little bit more aggressively on the agentic nature of the model, like with longer-running agentic workflows and such. I think that's when they began competing with Cursor, when they bet on that particular vision. And the next step is you're not even in the loop, so to speak. You're not in an IDE, but you're asking the model to go do work in the same way you'd ask someone on your team. That's not quite ready yet. There are still many tasks where you need to be in the loop. But the next six months look like an exploration of exactly what that trend line looks like.

是的。

Yeah.

但具体或较真地说，这里的瓶颈很多还是工具和管道是否连通的问题。很多事情我不能直接启动 Claude 让它去解决，因为可能它需要 GPU，或者我需要非常小心的权限设置，防止它接管整个集群并启动一大堆东西。所以你确实需要良好的沙盒环境和使用所有必要工具的能力。从指标来看，我们几乎肯定严重低估了模型的能力——评估模型能否完成任务。它们花几个小时、多次迭代来解决。最终有一个会像这样：'哦，我回来了，我解决了任务。'

But to be really concrete or pedantic about the bottlenecks here: a lot of it is again just tooling and whether the pipes are connected. Many things I can't just launch Claude and have it go solve, because maybe it needs a GPU, or maybe I need very careful permissioning so it can't take over an entire cluster and launch a bunch of things. So you really need good sandboxing and the ability to use all the necessary tools. We're almost certainly under-eliciting dramatically when you look at metrics—evals of whether the model can solve the task. They're solving them for hours over multiple iterations. Eventually one of them is like, 'Oh yeah, I've come back and solved the task.'

目前至少我，也许是我自己的问题。我让模型尝试做某事，如果它做不到，我就说，'好吧，我自己来。'这很有趣，因为我们甚至不会这样对待其他人。你雇一个新员工，你不会说，'我自己来。'你会给他们几周的反馈，而我们会在几分钟内放弃模型。

Me at the moment, at least, maybe the fault is my own. But I try the model on something, and if it can't do it, I'm like, 'Okay, fine, I'll do it.' That's interesting because we don't even treat other humans this way. You hire a new employee, you're not like, 'I'll do it.' You give them weeks of feedback, whereas we'll give up on the model in minutes.

是的，完全正确。但我认为部分原因在于它是否是异步的。如果是人在循环中，除非它立即回复，否则会费力得多。我注意到如果我没有第二个显示器一直开着 Claude Code，我就不会真正使用它。只有当它就在那里，我可以发送一些东西——如果命中，很好；如果没有，我同时也在处理它。但这种更异步的形式，我预计会极大地改善这些模型的体验。

Yes, exactly. But I think part of it is whether it's async or not. If it's human-in-the-loop, it's so much more effortful unless it's replying immediately. I've noticed if I don't have a second monitor with Claude Code always open, I won't really use it. It's only when it's right there and I can send something off—if it hits, great; if not, I'm working on it at the same time. But this more async form factor, I expect to really dramatically improve the experience of these models.

有趣。嗯，你可以直接说，'看看它能不能做到。'试一下。尝试十种不同的方法。直接启动它。

Interesting. Well, you can just say, 'Let's see if it can do that.' Give it a whirl. Try 10 different approaches. Just fire it off.

在我们结束之前，我想回到这个关键问题：为什么你谈到的计算机使用智能体和白领工作的进步会在未来几年内发生？为什么这不是一个需要几十年的事情？我认为答案在于那些预期时间更长的人有一种感觉：当 Andrew 和 Tommy 在我的播客上时，他们说，'你看，你可以看看 AlphaGo，说，哦，这是一个能进行探索的模型。AlphaZero 可以泛化到新的视频游戏。它有所有这些关于如何与世界互动的先验知识。智力天花板真的很高。' 事后看来，显然很多方法今天仍在深度学习中使用。但它从根本上说不是一个婴儿 AGI，我们只需要在上面撒一点别的东西就能变成今天的 LLM。我想直接解决这个关键问题：为什么 LLM 在真正 AGI 方面与 AlphaZero 处于非常不同的位置？为什么它们实际上是基础，加上几滴额外的关心和注意就能达到人类水平的智能？

Before we end, I want to get back to this crux: why does the progress you're talking about in computer use agents and white-collar work happen over the next few years? Why is this not a thing that takes decades? I think the answer comes down to people who expect something much longer have a sense that when Andrew and Tommy were on my podcast, they said, 'Look, you could look at AlphaGo and say, oh, this is a model that can do exploration. AlphaZero can generalize to new video games. It has all these priors about how to engage with the world. The intellectual ceiling is really high.' And in retrospect, obviously a bunch of the methods are still used today in deep learning. But it was fundamentally not like a baby AGI that we just had to add a little sprinkle of something else on top to make it the LLM of today. I want to directly address this crux: why are LLMs in a much different position with respect to true AGI than AlphaZero? Why are they actually the base on which adding a few extra drops of care and attention gets us to human-level intelligence?

我认为一个重要点是，当你观察 AlphaZero 时，它确实拥有所有这些成分。特别是，我认为智力天花板与我之前说的相当相反——我们在数学和编程问题上展示了令人难以置信的复杂性。我确实认为 AlphaZero 工作的任务类型和设置——双人完美信息游戏——对强化学习算法非常友好。之所以花了这么长时间才达到更像 AGI 的模型，是因为你需要破解对世界和语言的一般概念理解，并且你需要在你关心的现实世界任务上获得初始奖励信号，这比游戏更难指定。来自现实世界的梯度信号——突然间你获得了它，可以开始攀登。而 AlphaZero 从未有过那个第一级台阶可以拉。

I think one important point is that when you look at AlphaZero, it does have all those ingredients. In particular, I think the intellectual ceiling goes quite contrary to what I was saying before—we've demonstrated incredible complexity in math and programming problems. I do think the type of task and setting that AlphaZero worked in—two-player perfect information games—is incredibly friendly to reinforcement learning algorithms. The reason it took so long to get to more AGI-like models is you need to crack that general conceptual understanding of the world and language, and you need to get the initial reward signal on tasks you care about in the real world, which are harder to specify than games. That gradient signal from the real world—all of a sudden you get access to it and can start climbing it. Whereas AlphaZero never had that first rung to pull on.

是的，这又回到了打字机上的猴子，我想。对于预训练模型，直到你有了像 GPT-3 或 GPT-4 这样的东西，它才能生成足够连贯的句子，甚至开始进行 RLHF 并告诉它你喜欢什么和不喜欢什么。

Yeah, this goes back to the monkeys on the typewriter, I think. With the pre-training model, until you had something like GPT-3 or GPT-4, it just couldn't generate coherent enough sentences to even begin to do RLHF and tell it what you liked and didn't like.

是的。

Yeah.

如果到明年这个时候我们还没有相当稳健的计算机使用智能体，我们是否生活在破灭的时间线上——比如 2030 年或破灭？

If we don't have even reasonably robust computer use agents by this time next year, are we living in the bust timeline—as in 2030 or bust?

如果真是这样，我会非常惊讶。我认为这会在某种程度上更新为计算机使用特别困难。我不知道这是否是破灭的时间线，但肯定是我会更新为时间延长。

I would be extremely surprised if that was the case. I think that would be somewhat of an update towards there being something strangely difficult about computer use in particular. I don't know if it's the bust timeline, but it's definitely like I would update on this being a lengthening of time.

是的，我认为这越来越不再是猜测的问题。如果有人怀疑，我鼓励使用 Claude Code 或一些智能体工具，看看当前的能力水平。发推文容易得多。但说真的，模型在我们关心的任务上变得越来越有能力，我们可以给它们足够的数据。来自可解释性的电路结果也指向它们在做非常合理的可泛化事情。所以这个问题很重要，但我惊讶于有多少深度学习批评者根本没有真正与模型互动，或者很久没有互动了，并且不断移动目标。

Yeah, I think more and more it's no longer a question of speculation. If people are skeptical, I'd encourage using Claude Code or some agentic tool and just seeing the current level of capabilities. Tweeting is so much easier. But seriously, the models are getting really capable at tasks we care about and we can give them enough data for. The circuits results from interpretability are also pointing in the direction that they're doing very reasonable generalizable things. So this question matters a lot, but I'm surprised by how many deep learning critics just haven't really interacted with the models or haven't in a while, and constantly move the goalposts.

是的，图灵测试曾经是个东西，对吧？我们甚至不再谈论它了，认为它是一个有意义的测试会很傻。

Yeah, the Turing test used to be a thing, right? We don't even talk about it, and it'd be silly to think it was a meaningful test.

是的。

Yeah.

话虽如此，有一个前提：如果软件工程比计算机使用好得多——我的意思是，计算机使用仍然很糟糕——那么我可能会说，也许每个人都会继续专注于软件工程。就像它是最有价值的东西，每个边际的人和资金都投向了软件工程。我不认为情况是这样。我确实认为计算机使用足够有价值，人们会关心它。但这是我为明年准备的一个退路。

Now, that being said, one caveat on that is like if software engineering is just dramatically better than computer use—I mean, computer use still sucks—then I'd be like, still, maybe everyone just kept focusing on software engineering. Like it was just by far the most valuable thing, like every marginal person and dollar went towards software engineering. I don't think that's the case. I do think computer use is valuable enough that people will care about it. But that would be my one escape patch that I'm putting in place for next year.

是的，从对齐的角度来看这也很好，因为我认为在你能做非常可怕的事情之前，你需要更广泛的技能。

Yeah, it would be good from an alignment perspective too because I think you kind of do need a wider range of skills before you can do something super super scary.

哦，意思是如果模型没有变得更好。

Oh, as in if the models didn't get any better.

是的。如果他们只是超人类程序员，但不像亨利·基辛格那样。我不知道，这似乎还行。比如我们有 AI 预言机。

Yeah. If it's like just they're superhuman coders, but they're not like Henry Kissinger level. I don't know, that seems okay. Like if we have AI oracles.

是的，我就是这个意思。这很好。对，对，没错。

Yeah, that's what I'm saying. That's good. Yeah. Yeah. Exactly.

那么，回顾十年前的 AI 讨论，有一种感觉：有笨 AI，然后是 AGI，然后是 ASI，智能是一个标量值。你谈论这些模型的方式有一种锯齿感。它们特别适应于训练很多或数据很多的领域。是否仍然有意义谈论这些模型的通用智能？是否有足够的元学习和迁移学习来区分模型大小或训练方式，还是我们正在进入一个不是关于智能，而是关于领域的时代？

So, if you look back at AI discourse like going back a decade, there's a sense that there's dumb AI, then there's AGI, then there's ASI, that intelligence is a scalar value. The way you've been talking about these models has a sense of jaggedness. It's especially tuned to environments in which it's been trained a lot or has a lot of data. Is there a sense in which it still makes sense to talk about the general intelligence of these models? Is there enough metalearning and transfer learning that is distinguished between the sizes of models or the way models are trained, or are we moving into a regime where it's not about intelligence, it's more about domain?

一个直觉泵是，当模型像 GPT-2 那么大并为各种任务微调时，这种讨论很多。他们发现模型在微调的任务上表现更好，对吧？但到了 GPT-4，当它在足够多样的事物上训练时，实际上总算力在所有子任务上泛化得很好，甚至比更小的微调模型泛化得更好，这非常有用。我认为我们现在在强化学习中看到的情况基本相同，存在这种锯齿性，即它们特别训练过的领域。但随着我们扩大用于强化学习的总算力，你会开始看到同样的转变，从 GPT-2 微调到 GPT-3、GPT-4 的无监督元学习和跨领域泛化。我认为我们已经看到了早期证据，比如它能够将推理泛化到其他事物，但我认为这很快就会变得非常明显。

So one intuition pump is this conversation was had a lot when models were like GPT-2 sized and fine-tuned for various things. And they found that models were dramatically better at things they were fine-tuned for, right? But by the time you get to GPT-4, when it's trained on a wide enough variety of things, actually the total compute generalized very well across all the individual subtasks, actually generalized better than smaller fine-tuned models, in a way that was extremely useful. I think right now what we're seeing with RL is pretty much the same story playing out, where there's this jaggedness of things that they're particularly trained at. But as we expand the total amount of compute that we do RL with, you'll start to see the same transition from GPT-2 fine-tunes to GPT-3, GPT-4 unsupervised metalearning and generalization across things. And I think we're already seeing early evidence of this in its ability to generalize reasoning to things, but I think this will be extremely obvious soon.

一个很好的例子就是回溯的能力或概念，对吧？你沿着一条解决方案路径走，哦等等，让我试试另一条。这是通过强化学习在更难任务上训练时开始出现的。我认为目前它泛化得不是特别好，至少……我们有没有用强化学习训练模型成为可解释性智能体？没有。完全没有。所以一直以来我们都在说，哦，它只在强化学习过的任务上表现好。嗯，它在这方面确实不错，因为那是科学、理解语言和编码的混合。这里有多种领域的混合，你都需要理解。你需要既是一个优秀的软件工程师，又能通过语言和心态思考，甚至在某些方面进行哲学思考，才能成为一个可解释性智能体。而它正是从训练中泛化来做到这一点的。

One nice example of this is just the ability or notion to backtrack, right? You go down one solution path, oh wait, let me try another one. And this is something that you start to see emerge in the models through RL training on harder tasks. And I think right now it's not generalizing incredibly well, at least with... well, have we ever RL'd the model to be an interp agent? No. I mean no. Exactly. So all this time we're talking about like, oh it's only good at things that it's been RL'd on. Well, it's pretty good at that because that's a mixture of science and understanding language and coding. There's this sort of mixture of domains here, all of which you need to understand. You need to be both a great software engineer and be able to think through language and state of mind and almost philosophize in some respects to be an interp agent. And it is generalizing from the training to do that.

这里的终局是什么？Claude 辅助工具出来了，他们把它给你，然后你说赞。发生了什么？

What's the endgame here? Claude aid comes out and they give it to you and dot dot dot you say thumbs up. What's happened?

是的，我的意思是这完全取决于我们获得 Claude 8 的时间线以及模型达到 ASL 4 能力的时间。从根本上说，我们只会使用当时拥有的任何工具，看看它们效果如何。理想情况下，我们有一个枚举安全案例，几乎可以验证或证明模型会以特定方式行为。在最坏的情况下，我们使用当前工具，比如当我们赢得审计游戏时，查看助手标签亮起时哪些特征被激活。

Yeah, I mean it really depends upon the timeline at which we get Claude 8 and the models hit like ASL 4 capabilities, right? Like fundamentally we're just going to use whatever tools we have at the time and see how well they work. Ideally we have this enumerative safety case where we can almost verify or prove that the model will behave in particular ways. In the worst case we use the current tools like when we won the auditing game of seeing what features are active when the assistant tag lights up.

你能解释一下什么是机制可解释性吗？什么是特征？什么是电路？

Can you explain what is mechanistic interpretability? What are features? What are circuits?

完全没问题。机制可解释性，或者酷孩子们称之为 mech interp，是试图逆向工程神经网络，找出计算的核心单元是什么。很多人认为，因为是我们制造了神经网络，因为它们是人工智能，所以我们完全理解它们的工作原理。但事实远非如此。你今天使用的神经网络、AI 模型，是生长出来的，而不是构建出来的。因此，在它们训练完成后，我们需要做大量工作，尽最大努力弄清楚它们实际上是如何进行推理的。所以大约两年半到三年半前，将机制可解释性应用于大型语言模型的议程开始了，始于 Chris Olah 离开 OpenAI 共同创立 Anthropic。从那以后大约每六个月，我们对这些模型的理解就有一次重大突破。首先是通过叠加的玩具模型，我们确定了模型确实试图尽可能多地将信息塞进它们的权重中。这直接与人们说神经网络过度参数化的观点相悖。在经典的 AI、机器学习时代，你会使用线性回归或类似方法。人们有一个关于 AI 或神经网络、深度学习的梗，说使用了太多参数。有一个有趣的梗，你应该展示一下，x 轴是层数，y 轴也是层数，一条抖动的线一直上升，就像“哦，再加几层吧”，对吧？但事实证明，至少对于像准确预测整个互联网的下一个词这样的困难任务，这些模型根本没有足够的容量。所以它们需要尽可能多地塞入信息。它们学习做到这一点的方式是让模型中的每个神经元或计算单元用于许多不同的事情。因此，如果你试图理解模型，比如“哦，如果我移除这个神经元，或者它在模型中做什么？”那是不可能弄清楚的。

Totally. So mechanistic interpretability, or the cool kids call it mech interp, is trying to reverse engineer neural networks and figure out kind of what the core units of computation are. Lots of people think that because we made neural networks, because they're artificial intelligence, we have a perfect understanding of how they work. And it couldn't be further from the truth. Neural networks, AI models that you use today, are grown, not built. And so we then need to do a lot of work after they're trained to figure out to the best of our abilities how they're actually going about their reasoning. So two and a half to three and a half years ago, this kind of agenda of applying mechanistic interpretability to large language models started with Chris Olah leaving OpenAI and co-founding Anthropic. And every roughly six months since then we've had kind of a major breakthrough in our understanding of these models. So first with toy models of superposition, we established that models are really trying to cram as much information as they possibly can into their weights. And this goes directly against people saying that neural networks are overparameterized. In classic AI, machine learning back in the day, you would use linear regression or something like it. And people had a meme of AI or neural networks, deep learning, using way too many parameters. There's this funny meme that you should show of like layers on the x-axis and layers on the y-axis and this jiggly line that just goes up and it's like, oh, just throw more layers at it, right? But it actually turns out that at least for really hard tasks like being able to accurately predict the next token for the entire internet, these models just don't have enough capacity. And so they need to cram in as much as they can. And the way they learn to do that is to use each of their neurons or units of computation in the model for lots of different things. And so if you try to make sense of the model and be like, oh, if I remove this one neuron or what is it doing in the model? It's impossible to make sense of it.

它会触发中文、钓鱼、马，还有上百种不同的东西。这是因为模型试图用同一个神经元处理所有这些任务。这就是叠加。几个月后，我们写了《迈向单义性》这篇论文，引入了所谓的稀疏自编码器。基于我刚才说的模型试图把太多东西塞进太小的空间，我们给了它更大的空间——这个更高维的表示——让它能更清晰地表示它理解的所有概念。这是一篇非常玩具式的论文，因为模型只是一个两层的小型 Transformer，我们拟合了多达 16000 个特征，当时觉得这已经很多了。

It'll fire for like Chinese and fishing and horses and I don't know just like a hundred different things. And it's because it's trying to juggle all these tasks and use the same neuron to do it. So that's superposition. N months later we write towards monosemanticity which introduces what are called sparse autoencoders. And so going off what I just said of the model trying to cram too much into too little space, we give it more space, this higher dimensional representation where it can then more cleanly represent all of the concepts that it's understanding. And this was a very toy paper in so much as it was a two layer really small really dumb transformer and we fit up to I want to say 16,000 features which we thought was a ton at the time.

九个月后，我们从两层 Transformer 发展到当时的 Claude 3 Sonnet 前沿模型，拟合了多达 3000 万个特征。这时我们开始发现非常有趣的抽象概念，比如一个会为代码漏洞触发的特征。它不仅仅为代码漏洞触发，甚至还会为那种 Chrome 页面触发——比如你访问一个非 HTTPS 的 URL，页面警告说“此网站可能有危险，点击继续”——它也会触发。所以在这 3000 万个特征中，有这些更抽象的编码变量或情感特征。

Fast forward 9 months we go from a two-layer transformer to our Claude 3 Sonnet frontier model at the time and fit up to 30 million features. And this is where we start to find really interesting abstract concepts like a feature that would fire for code vulnerabilities. And it wouldn't just fire for code vulnerabilities. It would even fire for like you know that Chrome page you get if you like it's not an HTTPS URL and it's like warning this site might be dangerous like click to continue and it would also fire for that for example. And so it's like these much more abstract coding variables or sentiment features amongst the 30 million.

又过了九个月，现在我们有了电路。我之前用《十一罗汉》的盗窃团队做类比，现在你是在识别模型各层中共同完成复杂任务的单个特征，从而更清楚地了解它实际如何进行推理和决策，比如在医疗诊断中。

Fast forward nine months from that and now we have circuits and I threw in the analogy earlier of the Ocean's 11 heist team where now you're identifying individual features across the layers of the model that are all working together to perform some complicated task and you can get a much better idea of how it's actually doing the reasoning and coming to decisions like with the medical diagnostics.

我之前没讲的一个例子是模型如何检索事实。比如你问“迈克尔·乔丹打什么运动？”你不仅能看到它从“迈克尔·乔丹”跳到“篮球”并回答“篮球”，而且模型还知道自己什么时候不知道某个事实的答案。默认情况下，它会说“我不知道这个问题的答案”。但如果它看到自己知道答案的东西，就会抑制“我不知道”电路，然后用实际有答案的电路来回复。

One example I didn't talk about before is with how the model retrieves facts. And so you say like what sport did Michael Jordan play? And not only can you see it hop from like Michael Jordan to basketball answer basketball, but the model also has an awareness of when it doesn't know the answer to a fact. And so by default, it will actually say, I don't know the answer to this question. But if it sees something that it does know the answer to, it will inhibit the I don't know circuit and then reply with the circuit that it actually has the answer to.

例如，如果你问它“迈克尔·巴特金是谁”——一个虚构人物——它默认会说“我不知道”。只有对迈克尔·乔丹或其他真实人物，它才会抑制“我不知道”电路。但真正有趣的是，你可以开始对模型进行下游预测或推理：那个“我不知道”电路只针对人名。在论文中，我们还问了它“安德烈·卡帕西写了什么论文”。它认出了“安德烈·卡帕西”这个名字，因为他足够有名，所以关闭了“我不知道”回复。但当模型需要说出他写了什么论文时，它实际上不知道任何一篇，于是不得不编造。所以你可以看到不同的组件和电路同时交互，最终导致这个答案。

So for example, if you ask it who is Michael Batkin, which is just a made-up fictional person, it will by default just say I don't know. It's only with Michael Jordan or someone else that it will then inhibit the I don't know circuit. But what's really interesting here and where you can start making downstream predictions or reasoning about the model is that that I don't know circuit is only on the name of the person. And so in the paper we also ask it what paper did Andre Karpathy write. And so it recognizes the name Andre Karpathy because he's sufficiently famous. So that turns off the I don't know reply. But then when it comes time for the model to say what paper it worked on, it doesn't actually know any of his papers. And so then it needs to make something up. And so you can see different components and different circuits all interacting at the same time to lead to this final answer.

为什么我认为理解模型中发生的每一件事是一个可解的问题，或者说这是理解它为何欺骗的最佳方式。如果你想用粒子物理学解释英国为什么赢得二战，那你就走错了方向。你只需要看高层次解释：谁有更多武器，他们想要什么？这类似于训练线性探针来问“你诚实吗？你在欺骗吗？我们在红队测试时抓到你做坏事了吗？我们能监控你吗？”为什么这不类似于让粒子物理学家回溯并解释英国为什么赢得二战？我觉得你只需要睁大眼睛，不要对欺骗的样子或触发条件做任何假设。是的，所以你能撒的网越大越好。

Why I think it's a tractable problem to like understand every single thing that's happening in a model or like that's the best way to understand what why it's being deceptive. If you wanted to explain why England won World War II using particle physics, you would just be on the wrong track. You just want to look at the high-level explanations of who had more weapons, like what did they want? And that seems analogous to just training linear probes for like are you honest? Are you being deceptive? Do we catch you doing bad things when we're red teaming you? Can we monitor you? Why is this not analogous where we're asking a particle physicist to just backtrack and explain why England won World War II? I feel like you just want to go in with your eyes wide open, not making any assumptions for what that deception is going to look like or what the trigger might be. Yeah. And so the wider you can cast that net, the better.

取决于 AI 加速的速度和我们工具的状态，我们可能无法从底层证明一切是安全的。但我觉得这是一个很好的北极星。这是一个非常强大、令人安心的北极星，尤其是当我们考虑到自己是更广泛的 AI 安全组合的一部分时。你真的相信吗？你即将部署这个系统，真心希望它与人类对齐，并且你已经成功迭代了所有它可能策划或偷懒的方式。但无论你发现什么，可能也是如此。你仍然会有未解释的方差，或者你找到了一个特征，但不知道它是否真的解释了欺骗，还是别的什么。

Depending on how quickly AI accelerates and where the state of our tools are, we might not be in the place where we can like show prove from the ground up that everything is safe. But I feel like that's a very good north star. It's a very powerful reassuring north star for us to aim for especially when we consider we are part of the broader AI safety portfolio. I mean do you really trust like you're about to deploy this system and you really hope it's aligned with humanity and that you've like successfully iterated through all the possible ways that it's going to like scheme or sandbag. But that's also probably going to be true with whatever you find. You're not going to have explained all variance, or you found a feature but you don't know if it actually explains deception or something else instead.

所以首先，我不是说你不应该尝试探针方法。我们想追求整个组合。我们有治疗师通过问“你有没有困扰的想法”来审问病人。我们有线性探针，我把它类比为测谎仪，我们获取关于这个人健康状况的非常高级的汇总统计。然后我们有神经外科医生，他们进去看看是否能找到任何以令人不安或偏离分布的方式激活的大脑组件。所以我认为我们应该全部做。

So I guess first of all I'm not saying you shouldn't try the probing approach right? We want to pursue the entire portfolio. We've got the therapist interrogating the patient by asking do you have any troubling thoughts. We've got the linear probe which I'd analogize to like a polygraph test where we're taking very high-level summary statistics of the person's wellbeing. And then we've got the neurosurgeons kind of going in and seeing if you can find any brain components that are activating in troubling or off-distribution ways. So I think we should do all of it.

可解释性应该占对齐组合的百分之多少？

What percent of the alignment portfolio should interpretability be?

我认为需要多大就多大。我的意思是至少……很难定义，但在 Anthropic，我觉得所有不同的组合都得到了很好的支持，并且正在成长。

I think as much of a chunk as is necessary. I mean I think at least like question. Hard hard hard to define, but I don't know at Anthropic I feel like all of the different portfolios are being very well supported and growing.

你也可以回到二战的问题。你可以把它看作一个信任抽象层次结构：假设你想和丘吉尔谈话。如果你能验证在那十分钟的对话中他是诚实的，那会很有帮助。这让你能构建更好的元叙事。所以粒子物理学可能帮不上忙，但丘吉尔大脑的神经科学肯定能帮你验证他在那次对话中是值得信任的，前线士兵对事件的描述也是诚实的，诸如此类。只要你能验证树上的部分进展，就能极大地帮助你建立信心。

You can also go back to the World War II question. You can think of it as like a hierarchy of abstractions of trust here where let's say you want to go and talk to Churchill. It helps a lot if you can verify that in that conversation in that 10 minutes he's being honest. And this enables you to construct better meta-narratives of what's going on. And so maybe particle physics wouldn't help you there, but certainly the neuroscience of Churchill's brain would help you verify that he was being trustworthy in that conversation and that the soldiers on the front lines were being honest in their depiction of what happened and this kind of stuff. So long as you can verify progress like parts of the tree up, then that massively helps you build confidence.

是啊，我觉得语言模型也真的很奇怪，对吧？就像那个关于涌现性不对齐的研究，我不确定他们是否预料到，比如‘我要在代码漏洞上微调 ChatGPT，它会变成纳粹吗？’我想大多数人都会说不会。但结果确实如此。那么他们是怎么发现它变成纳粹的呢？他们开始问它一大堆不同的问题，它就会做出各种邪恶和有害的事情。整个角色完全变了。我的意思是，我们面对的是外星大脑，它们没有人类的社会规范，甚至不清楚我们以为它们学过什么、没学过什么。所以我认为你真的需要睁大眼睛来面对这一切。

Yeah, I think language models are also just really weird, right? Like with the emergent misalignment work, I don't know if they took predictions they should have had, like, 'Hey, I'm going to fine-tune ChatGPT on code vulnerabilities. Is it going to become a Nazi?' And I think most people would have said no. And that's what happened. So how did they discover that it became a Nazi? They started asking it a ton of different questions and it would do all sorts of vile and harmful things. The whole persona just totally changes. And I mean, we are dealing with alien brains here who don't have the social norms of humans, or even a clear notion of what they have and haven't learned that we have of them. So I think you really want to go into this with eyes wide open.

退一步说，如果你生活在一个 AI 进步加速的世界里——顺便提一下，你刚才提到我们可能生活在许多疯狂的世界中，但至少我们活在其中一个。另一个我们暗示过但值得更明确说出的世界是：即使 AI 模型没有帮助编写下一代训练算法，仅仅如果它们拥有人类水平的学习效率，那么无论模型在工作中学到什么，或者模型的任何副本在工作中学到什么，整个模型都在学习。所以实际上，它正在变得更好——或者如果它们的学习效率比人类低一千倍，那也没关系。你仍然部署它们。没错。总之，还有很多其他事情可以思考，但即使如此，你基本上拥有一个广泛部署的智能爆炸。我确实认为值得深入探讨那个未来。有一系列疯狂的可能性，但我认为我们几乎必然会得到的一个——这是一个强烈的论断——是至少在未来 5 年内，会出现可以即插即用的白领工人。我认为 2 年内就很有可能。但在 5 年内几乎是必然的。从大局来看，这些时间框架其实无关紧要；无论哪种情况都一样。这将在未来十年彻底改变世界。如果我们没有为此制定正确的政策，那么在某些方面，世界实际上会变得更糟，因为这些模型默认擅长的东西是软件工程和计算机使用智能体之类的。然后我们需要付出额外努力，让它们进入帮助科学研究的循环，或者拥有合适的机器人技术，从而真正提高物质生活质量。所以这值得思考。如果你站在一个国家的角度，你应该做什么或思考什么？为白领工作可自动化的情况做计划，然后考虑这对你的经济意味着什么，以及你应该如何制定政策。老实说，这是一个非常棘手的问题。如果你是印度、尼日利亚或澳大利亚，如果你是一个不像美国或中国那样拥有前沿模型的国家，你现在应该做什么，尤其是在这么短的时间尺度上？

Backing up for me, if you live in a world where AI progress accelerates, by the way, you were mentioning a little while ago that there are many wild worlds we could be living in, but we're living in at least one of them. Another one we've gestured at, but it's worth making more explicit, is this: even if the AI models are not helping write the next training algorithm for their successor, just the fact that if they had human-level learning efficiency, whatever a model is learning on the job, or whatever copy of the model is learning on the job, the whole model is learning. So in effect, it's getting—or if they're a thousand times less efficient than humans at learning, that's right. And you just deploy them even still. Exactly. Yeah. Anyways, there's a whole bunch of other things you can think about, but even there, you kind of have a broadly deployed intelligence explosion. And I do think it's worth pressing on that future. There is this whole spectrum of crazy futures, but the one I feel we're almost guaranteed to get—and this is a strong statement to make—is one where, at the very least, you get drop-in white-collar worker at some point in the next 5 years. I think it's very likely in 2. But it seems almost overdetermined in 5. And on the grand scheme of things, those are kind of irrelevant time frames; it's the same either way. That completely changes the world over the next decade. And if we don't have the right policies in place for that, then you end up actually with a fundamentally worse world in some respects, because the thing these models get good at by default is software engineering and computer-using agents and this kind of stuff. Then we will need to put in extra effort to put them in the loops where they help us with scientific research, or we have the right robotics such that we actually experience an increase in material quality of life. So that's worth thinking about. If you're in the perspective of a country, what should you be doing or thinking about? Plan for the case where white-collar work is automatable, and then consider what that means for your economy and what you should be doing to prepare policy. Honestly, it's such a tough question. If you're India or Nigeria or Australia, if you're a country unlike America or China where they do have frontier models, what is it that you should be doing right now, especially on such a short time scale?

是的。

Yes.

所以我认为一个非常重要的点是，假设这个情景成真，那么算力将成为世界上最有价值的资源。你经济体的 GDP 会受到你能够部署给国内组织的算力数量的巨大影响。因此，拥有一定数量的保证算力，我认为实际上会非常重要。所以提前投资数据中心之类的东西，条件是必须允许你国家的公司使用这些算力。不一定用于训练，甚至只是用于推理。我认为经济价值来自推理。我认为广泛投资 AI 也是有意义的。我认为这些国家有机会这样做，这就像是一个投资组合，包括基础模型公司，也包括机器人供应链之类的东西。我认为你应该非常积极地投资于试图防止资本锁定的政策。如果恰好那些在 AGI 之前拥有股票或土地的人比没有的人富裕得多，那将是一个更糟糕的世界，因为这是资源的严重错配。所以，我知道你播客中我最喜欢的一集是关于乔治主义的那一集，你恰当地评估了土地分配。我认为这尤其贴近澳大利亚的情况，我认为我们关于土地的政策是完全错误的。但我认为这普遍成立。在将这些模型整合到你的国家方面，非常前瞻性地进行监管是重要的。并且主动确保人们有选择权——比如，你应该非常主动地确保人们拥有的手机、设备或眼镜，人们可以自由选择运行什么。这就是‘我们刚刚有了白领工人’的情景，你正在尽力让你的国家为此做好准备。然后，好吧，你能做些什么来让所有可能的未来版本都顺利发展？这涵盖了一定程度的经济下行风险。我认为其他非常重要的事情是，弄清楚如何确保巨大的上行潜力或覆盖可怕的下行风险。所以获得巨大上行潜力是确保在生物学研究等方面进行自动化投资，这样这些模型实际上能够生产出大幅改善我们生活质量的新药。而覆盖下行风险则是 AI 对齐研究之类的东西，自动化测试，以及认真思考 AI 安全机构。但这些似乎是一个富人，一个随便的富人也能做的事情。似乎没有什么是国家独特有能力做的。

So I think one very important point is that, let's say this scenario turns out true, then compute becomes the most valuable resource in the world. The GDP of your economy is dramatically affected by how much compute you can deploy towards organizations within your country. So having some guaranteed amount of compute, I think, will actually be quite important. So preemptively investing in data centers and this kind of stuff, on the condition that companies in your country have to be allowed to use that compute. Not necessarily for training, but just even for inference. I think the economic value here comes from inference. I think it also makes sense to invest broadly in AI. I think these countries have the opportunity to do so, and that's like a portfolio of foundation model companies but also robotic supply chain and this kind of stuff. I think you should invest very proactively in policies that try to prevent capital lock-in. We're in for a much worse world if it just so happens that the people who had money in the stock exchange or in land before AGI are dramatically more wealthy than the people who don't, because it's a gross misallocation of resources. So having—I know one of my favorite episodes actually on your podcast was the Georgism one, where you appropriately value allocate land. I think this strikes particularly close to home coming from Australia, where I think our policies with respect to land are grossly wrong. But I think this is broadly true. Being very forward on regulation of integration of these models into your country is important. And proactively making sure that people have choice—so let's say you should be quite proactive about making sure that the phones or devices or glasses that people have, people have free choice on what things they run. So that's the 'we just get white-collar worker' scenario, and you're trying to do the best to prepare your country for that. Then it's like, okay, what can you do to make all possible versions of the future go well? That covers some amount of economic downside. The other things I think are really important is figure out how you can either ensure dramatic upside or cover terrible downside. So getting dramatic upside is making sure there is investment in biology research and this kind of stuff in an automated way, such that these models are actually able to produce novel medicines that massively improve our quality of life. And covering the downside is AI alignment research and this kind of stuff, automated testing, and really thinking hard about AI safety institutes. But these seem like things that a rich person, a random rich person, could also do. There doesn't seem to be a thing that a nation state is uniquely equipped to do.

在这种情况下，将资源大幅分配给算力是明智的。如果我是国家领导人，我会这么做。这能增加你在大多数未来世界中的选择余地。Dylan Patel 对中美能源的预测很吓人。美国差了大约 34 吉瓦。美国的曲线基本上是平的，而中国在上升。美国显然需要更多发电厂。如果智能成为未来经济和生活质量中极其宝贵的投入，几乎是一种原始投入，那么其直接基础就是能源。所以确保你拥有巨量的太阳能，比如在沙漠部分地区铺满太阳能板，将有助于你获得更多智能。

In this scenario, dramatic allocation of resources towards compute is sensible. I would be doing that if I was in charge of a nation state. It just increases your optionality in most future worlds. Dylan Patel has some scary forecasts on US energy versus China. The US is about 34 gigawatts off. The US line is flat basically, and China's line is going up. The US clearly needs many more power plants. If intelligence becomes an incredibly valuable input, almost a raw input into the economies and quality of life of the future, the thing directly underneath that is energy. So making sure you have incredible amounts of solar, like tiling parts of the desert in solar panels, would be helpful towards having more access to intelligence on top.

明确一下，即使 AI 进展完全停滞，或者你认为模型能力参差不齐、缺乏通用智能，它仍然具有巨大的经济价值，而且收集所有这些白领工作任务的数据足够容易，所以正如 Sholto 所说，我们应该预期这些工作在未来 5 年内被自动化。即使你需要手把手地教模型每一个任务，经济上也是划算的。所以即使算法进展停滞，我们永远找不到继续推进的方法——我不认为会这样，进展并未停滞，而且看起来很好——只要你有足够多、足够好的数据，当前的算法套件就足以自动化白领工作。与所有这些工作的薪资总市场规模相比，这简直微不足道。

Just to make it explicit, even if AI progress totally stalls or you think the models are spiky and lack general intelligence, it's so economically valuable and sufficiently easy to collect data on all these white-collar job tasks that, to Sholto's point, we should expect to see them automated within the next 5 years. Even if you need to hand-spoon every single task to the model, it's economically worthwhile. So even if algorithmic progress stalls and we never figure out how to keep progress going—which I don't think is the case, it hasn't stalled yet and seems to be going great—the current suite of algorithms is sufficient to automate white-collar work, provided you have enough of the right kinds of data. Compared to the TAM of salaries for all that work, it's trivially worthwhile.

正是如此。

Exactly.

我想指出一个非常反乌托邦的未来，如果你把莫拉维克悖论推到极致。这个悖论说，我们认为人类能做的最有价值的事情是聪明的事，比如心算大数或白领工作，但我们却把精细运动技能和协调能力视为理所当然。进化把精细运动协调优化得如此之好，以至于连机器手或开门的能力对机器人来说仍然非常困难。与此同时，我们看到编程和其他我们认为是聪明的工作正在被完全自动化。可怕的是，AI 能做所有事情，除了物理机器人任务。那么你会看到人类戴着 AirPods 和眼镜，某个机器人霸主通过摄像头控制人类，告诉人类该做什么，在要捡起的物体周围画个边界框。于是就有了人类肉机器人。不一定是 AI 想这么做，但从相对经济价值来看，AI 在做编程，而人类能做的最有价值的事情就是成为出色的机器人。

I want to flag a really dystopian future if you take Moravec's paradox to its extreme. This paradox says we think the most valuable things humans can do are smart things like adding large numbers in our heads or white-collar work, but we take for granted our fine motor skills and coordination. Evolution optimized fine motor coordination so well that even robot hands or the ability to open a door is still really hard for robots. Meanwhile, we see total automation of coding and everything else we've seen as clever. The scary future is one where AIs can do everything except physical robotic tasks. Then you'd have humans with AirPods and glasses, and some robot overlord controlling the human through cameras, telling them what to do with a bounding box around the thing to pick up. So you have human meat robots. Not necessarily that AIs would want to do that, but in terms of relative economic value, AIs are doing computer programming and the most valuable thing humans can do is be amazing robots.

我认为莫拉维克悖论有点假。机器人不如软件工程的主要原因是，软件工程有互联网——有 GitHub。机器人领域没有类似的东西。如果你有每个人日常活动的映射，覆盖相当一部分人口，机器人也会接近解决，并以与软件工程相同的速度被解决。所以这个愿景只是一个十年左右的阶段，但这十年仍然相当糟糕。想象一个世界：人们失去了工作，你还没有获得能大幅改善生活质量的新型生物学研究，你还没有物质丰裕，因为你无法以必要的方式作用于物理世界——你无法大幅增加建设，因为这需要机器人。而人们的主要比较优势是成为出色的机器人。那是一个令人震惊的世界。

I think Moravec's paradox is a bit fake. The main reason robots are worse at being robots than software engineering is that the internet exists for software engineering—GitHub exists. There is no equivalent for robotics. If you had a map of everyone's actions as they went about their daily lives for a reasonable fraction of the human population, robotics would also be close to solved, on track to be solved at the same rate as software engineering. So this vision is only a decade-long section, but it's still a pretty terrible decade. Imagine a world where people have lost their jobs, you haven't yet got novel biological research that dramatically improves quality of life, you don't yet have material abundance because you haven't been able to action the physical world in the necessary way—you can't build dramatically more because that takes robots. And people's main comparative advantage is as fantastic robots. That's a shocking world.

从普通人的角度来看，这实际上可能更好。你的工资会更高，因为你是极其有价值的东西——AI 劳动力的互补品。一二十年之后，世界会变得美妙。机器人技术被解决，你获得彻底丰裕，前提是你有允许建设的政策。你会看到像上海前后对比照片那样的变化，20 年后城市彻底改观。世界上很多地方可能都会在二十年内变成那样。但我们需要尽力评估这是否真的在发生。为所有其他形式的白领工作构建 SWE-bench，并进行测量和跟踪。政府应该做这件事：把经济职能分解为可测量的任务，弄清楚曲线是什么样的，因为他们可能会对进展感到震惊。税务评估没有 SWE-bench。我没有所有答案，但想办法广泛分享这个经济的收益，或者大力投资机器人技术和数据收集以加速机器人发展，投资生物学研究以提前实现巨大收益——否则你会经历一段相当黑暗的时期。

From the perspective of an average human, it might actually be better. Your wages will be higher because you're the complement to something enormously valuable—AI labor. And a decade or two on, the world is fantastic. Robotics is solved and you get radical abundance, provided you have the policies set up to permit building. You end up with the same change as the before-and-after photos of Shanghai, where 20 years on it's a dramatically transformed city. A lot of places in the world probably end up like that over that two-decade period. But we need to do our best to estimate if this is actually on track. Build SWE-bench for all other forms of white-collar work and measure and track. That's a great thing for governments to do: break down the functions of their economy into measurable tasks and figure out what the curve looks like, because they might be shocked by the progress. There's no SWE-bench for tax evaluation. I don't have all the answers, but figuring out a way to share the proceeds of this economy broadly across people, or invest heavily in robotics and collecting data so we get robotics faster, invest in biological research to pull forward the radical upside—because otherwise you have a pretty dark section.

我认为有一件事没有被充分认识到：鉴于我们的劳动力将不再那么有价值，我们对未来的杠杆作用很大程度上来自于我们的经济和政治体系能够存活。为了让你的百万倍标普股权有意义，为了让你的合同有意义，为了让政府能够对 AI 劳动力征税并给你提供全民基本收入。

I think one thing that's not appreciated enough is how much of our leverage on the future, given that our labor isn't going to be worth that much, comes from our economic and political system surviving. For your millionxed S&P equity to mean something, for your contracts to mean anything, for the government to be able to tax AI labor and give you a UBI off of that.

这要求我们的法律机构、经济机构、金融轨道能够延续到未来。是的。实现这一点的可能方式是，遵循这些轨道也符合 AI 的最佳利益。我所说的 AI 不是指某个单一的巨型 AI，而是指那些雇佣 AI 并因此变得更高效的公司。你不希望出现这样的情况：在我们的系统中运营变得如此繁琐，以至于你实际上是在筛选那些要么移民、要么从事黑市交易等的公司。这意味着，我认为你要让部署 AI 变得极其容易，设立类似经济特区的东西。否则，你就是在放弃未来，放弃你对它的任何控制。顺便说一句，我担心将 AGI 变成国家安全问题或与政府紧密挂钩（像曼哈顿计划那样）的原因之一是，这会不成比例地将 AI 的使用转向军事技术、蚊子无人机等。而且这自然会让其他国家也产生同样的想法，对吧？如果我们开发蚊子无人机，为什么中国不开发？这看起来就像一场零和竞赛，更不用说可能带来灾难性的后果了。而如果它完全保持消费者自由市场的格局，似乎更有可能迎来辉煌的超人类主义未来，他们会开发那些让人类生活更美好的东西。

It just like that requires our legal institutions, our economic institutions, our financial rails surviving into the future. Yes. The way in which that likely happens is if it's also in the AI's best interests that they follow those rails. And by AI, I don't mean some monolithic single AI. I just mean like firms which are employing AI and becoming more productive as a result. You don't want to be in a position where it's so onerous to operate in our system that you're basically selecting for firms who either immigrate or who are like doing black market stuff etc. And which means I think like you want to make it super super easy to deploy AI, have the equivalent of special economic zones etc. Because otherwise you are just surrendering the future. Outside of any control that you might have on it. One of the reasons by the way that I worry about turning AGI into a national security issue or having it have extremely close ties with the government, the Manhattan Project thing, is that it disproportionately redirects the use of AI towards military tech and the mosquito drones and whatever. And also naturally puts other countries in the same frame of mind, right? If we're developing the mosquito drones, why would China not develop the mosquito drones? And that just seems like a zero sum race and not to mention a potentially catastrophic one. Whereas like, you know, compute will be limited. We will need to disproportionately accelerate some things. To the extent it just remains totally like a consumer free market landscape, it just seems more likely that we'll get the glorious transhumanist future where they're developing the things that make human life better.

是的，我同意。两个国家项目相互对抗的情况要糟糕得多。我们不想生活在那个世界里。可以说，如果这保持自由市场，要好得多。

Yes, I agree. The case where you end up with like two national projects facing off against each other is dramatically worse. We don't want to live in that world. It's much much better if this stays a free market, so to speak.

我想质疑你的说法，即即使使用今天的算法，只要我们收集足够的数据，就能自动化白领工作。首先，让我理解你的意思。你是说我们会做类似的事情，用人们工作中所有轨迹进行预训练？你能手动或通过其他过程，基于每个白领员工的屏幕录制来制定某种强化学习程序吗？你想象的是什么样的东西？

I want to take issue with your claim that even with the algorithms of today, if we just collect enough data, we could automate white collar work. First, let me get an understanding of what you mean by that. So, do you mean that we would do the analogous thing of pre-training with all the trajectories of everything people do on their jobs? Could you make either manually or through some other process some RL procedure based on the screen recordings of every white collar worker? What kind of thing are you imagining?

我的意思是这些东西的连续分布。思考强化学习的一个重要心智模型是，随着任务变得更复杂，从某种意义上说，如果你能获得奖励，更长的时间跨度或更好的表现更容易判断。所以这又回到了：你能在互联网上赚钱吗？这是一个极其容易判断的奖励信号。但要实现这一点，有一整套复杂行为的层次结构。所以如果你能预训练到容易判断的奖励信号，比如你的网站工作吗？它宕机了吗？人们喜欢它吗？有所有这些我们可以响应的奖励信号，因为我们可以通过这些足够长的轨迹进展，最终达到有趣的事情。如果你被困在每五个词元就需要一个奖励信号的机制中，那将是一个更加痛苦和漫长的过程。但如果你能在美国每个屏幕上进行预训练，那么你可以设计的强化学习任务将与仅使用现有互联网的情况大不相同。因此，你能访问多少数据会改变组合。

I mean like a continuous distribution of this stuff. One important mental model to think about RL is that as the task gets more complex, there is some respect with which longer horizon or better at that task if you can get that reward ever are easier to judge. So again, this comes back to like can you make money on the internet? That's an incredibly easy reward signal to judge. But to do that, there's a whole hierarchy of complex behavior. So if you could pre-train up to the easy to judge reward signals like does your website work? Does it go down? Did people like it? There are all these reward signals that we can respond to because we can progress through these long enough trajectories to actually get to interesting things. If you're stuck in this regime where you need a reward signal every five tokens, it's a way more painful and long process. But if you could pre-train on every screen in America, then probably the RL tasks that you can design are very different to if you could only take the existing internet as it is today. And so how much of that you get access to changes the mix.

有趣。那么，随着我们训练它们执行越来越长时间跨度的任务，它们需要更长时间才能获得是否成功完成任务的信号，这会因为每个任务需要更多算力而减慢进展吗？

Interesting. So, as we're training them on longer and longer horizon tasks and it takes longer for them to get any signal on whether they successfully complete the task, will that slow down progress because it takes more compute per task?

我确实认为存在这样一种观念：任务越难、时间跨度越长，需要的训练就越多。我天真地赞同这一点，但我们人类非常擅长练习任务的困难部分并将其分解。我认为一旦模型在基础部分足够好，它们就可以排练或快进到更困难的部分。这绝对是最大的复杂性之一，对吧？随着你使用更多算力，训练越来越难的任务。例如，我不知道你在生物学上的进步速度会在某种程度上受到细胞生长时间的限制，而你在数学上的进步速度则不会。所以是的，但我认为对于许多事情，我们将能够足够广泛地并行化并获得足够的迭代循环。

I do think there's this notion the longer the harder tasks, the more training is required. And I'm sympathetic to that naively, but we as humans are very good at practicing the hard parts of tasks and decomposing them. And I think once models get good enough at the basic stuff, they can just rehearse or fast forward to the more difficult parts. I mean that's definitely one of the big complexities, right? As you use more compute and as you train more and more difficult tasks. I don't know your rate of improvement at biology is going to be somewhat bound by the time it takes a cell to grow in a way that your rate of improvement on math isn't, for example. So yes, but I think for many things we'll be able to parallelize far widely enough and get enough iteration loops.

训练新模型的机制会消失吗？我们最终会达到这样的状态：你有了一个模型，然后你只需通过强化学习训练不断给它添加更多技能？

Will the regime of training new models go away? Will we eventually get to like you've got the model and then you just keep adding more skills to it with RL training?

这取决于你是否认为预训练新架构有好处。基本上，如果你做了一些架构上的改变，那么你可能需要至少重新训练一个新模型。

That depends on whether or not you think there's a virtue in pre-training a new architecture. Basically, if you make some architectural change, then you probably need to do some form of at least retraining a new model.

如果强化学习首先需要大量推理来进行训练，这一事实是否会与你之前提到的我们需要更大的模型以获得类脑能量相矛盾？但同时在强化学习中训练它也更昂贵。那么平衡点在哪里？

How does the fact that if RL requires a bunch of inference to do the training in the first place, does that push against the thing you were talking about where we actually need a bigger model in order to have brain-like energy? But then also it's more expensive to train it in RL. So where does that balance out?

我认为我们必须接受这个苦涩的教训。是的，没有无限的捷径。你确实必须扩展规模，拥有更大的模型，并为此支付更多的推理成本。如果你想要 AGI，这就是你必须付出的代价。但这里有一个权衡方程，有科学工作要做，每个人都在做：什么是进行强化学习的最佳点？因为你需要一个既能学习又能自行发现稀疏奖励的东西。所以你不想要一个单参数模型，尽管它可以运行得非常快，但毫无用处。你也不想要一个 100T 的模型，因为它对于可能的强化学习来说太慢了。而且它的学习效率的边际收益不值得。所以这里有一个前沿：在你当前的能力类别和当前的强化学习环境等条件下，什么是最优模型大小？即使在去年，推理成本也成为了一个更大的因素。所以明确地说，模型越大，进行前向传播和生成词元的成本就越高。

I think we got to drink the bitter lesson here. And yeah, there aren't infinite shortcuts. You do just have to scale and have a bigger model and pay more inference for it. And if you want AGI, then that's what you got to pay the price of. But there's a trade-off equation here, of there is science to do which everyone is doing of what is the optimal point at which to do RL, because you need something which can both learn and discover the sparse reward itself. So you don't want a one parameter model. Useless even though you can run it really fast. You also don't want a 100T model because it's super slow for possible RL. And the marginal benefit of its learning efficiency is not worth it. So there's a frontier here: what's the optimal model size at your current class of capabilities and your current set of RL environments and this kind of stuff. And even in the last year there's been much more of a factor of the inference cost. So just explicitly, the bigger the model, the more expensive it is to do a forward pass and generate tokens.

以前的计算逻辑只是：我该把算力分配给更多训练数据还是更大的模型？现在另一个巨大因素是，模型训练完成后，我实际上要做多少次前向传播。我的总算力池：如何分配训练数据算力和推理算力给强化学习训练？甚至在推理内部，也有很多研究关于该用什么策略：是采样 10 个然后选最好的？还是做这种分支搜索等等。所以在强化学习中，你要采样大量词元，还需要考虑模型实际生成这些词元并从中学习和获得反馈的能力。

And the calculus used to just be: should I allocate my flops to more training data or a bigger model? And now another huge factor is how much am I actually going to do forward passes on this model once it's trained. My total pool of compute: how do I allocate that across training data compute and inference compute for the RL training? And then even within inference, there's all this research on what strategy should I use? Should I sample 10 and take the best? Do I do this sort of branching search, etc. And so with RL where you're sampling a whole lot of tokens, you also need to factor in the ability for the model to actually generate those tokens and then learn and get feedback.

好的。那么如果我们就生活在这样的世界里，你对刚起步的职场人或大学生有什么建议？他们应该怎么规划？

Okay. So if we're living in this world, what is your advice to somebody early in their career or a student in college? How should they be planning on doing?

是的。所以我认为，值得考虑可能世界的谱系，并为此做好准备。我认为在这种情况下期望值最高的行动是：你即将获得巨大的杠杆效应。至少，你即将获得巨大的杠杆效应。YC 的初创公司已经在用 Claude 编写大量代码了。那么，有了这种额外的杠杆，你想改变世界上的哪些挑战或事业？如果你有 10 个工程师随时听命，你会做什么？或者如果你有一家公司随时听命，那会让你能做什么？哪些问题和领域突然变得可行了？这就是你现在要准备的世界。这显然仍然需要很深的技术功底。有一种情况是 AI 在一切方面都变得比所有人好得多，对吧？但至少在一段时间内，可能还是有优势的。我记得黄仁勋在一次采访中以一种有趣的方式谈到了这一点。他说：‘我身边有 10 万个通用智能体，我仍然有点用处，因为我在那里指导价值观，让它们做事。即使我有 10 万个通用智能体，我仍然有价值。’对很多人来说，我认为这种情况还会持续相当一段时间。然后随着 AI 越来越好，最终不会。但再次强调，要为可能世界的谱系做准备，因为如果我们完全被超越，那你做什么都没用。但在所有其他世界里，这很重要。获得技术深度：学习生物学，学习计算机科学，认真思考物理学，认真思考你想解决世界上的哪些挑战。

Yeah. So I think once again, it's worth considering the spectrum of possible worlds and preparing yourself for that. The action that I think is highest expected value in that case is: you are about to get dramatically more leverage. At a minimum, you are about to get dramatically more leverage. You already have startups in YC writing huge amounts of their code with Claude. So what challenges or causes do you want to change in the world with that added leverage? If you had 10 engineers at your beck and call, what would you do? Or if you had a company at your beck and call, what would that enable you to do? What problems and domains suddenly become tractable? That's the world you want to prepare for now. That still requires a lot of technical depth, obviously. There is the case where AI just becomes dramatically better than everyone at everything, right? But for at least a while, there is probably an advantage. I think Jensen actually talked about this in an interview in an interesting way. He said, 'I have 100,000 general intelligences around me, and I'm still somewhat useful because I'm there directing the values and asking them to do things. I still have value even though I have 100,000 general intelligences.' For many people, I think that will still be true for a fair while. And then as the AI gets better and better, eventually no. But again, prepare for the spectrum of possible worlds, because in the event where we're just totally outcompeted, it doesn't matter what you do. But in all the other worlds, it matters a lot. Get the technical depth: study biology, study CS, really think hard about physics, think hard about what challenges you want to solve in the world.

是的，话题很多。你现在就可以，对吧？学习变得容易多了。每个人现在都有无限完美的导师。这对我帮助很大。我会说，结合几点：摆脱之前工作流程或专业知识的沉没成本，以便评估 AI 能为你做什么。没错。另一种有趣的说法是，变得更懒，想办法让智能体去做那些繁琐的事情。但在这个过程中，你最终会变得更懒，但在短期内，你需要批判性地思考你目前正在做的事情，以及 AI 实际上能更好地做什么，然后去尝试或探索。因为我认为仍然有很多唾手可得的机会，人们只是假设而不去写完整的提示词、给几个例子、连接正确的工具来加速或自动化你的工作。

Yeah, that's a lot of topics. You can now, right? It's so much easier to learn. Everyone now has the infinite perfect tutor. It's definitely been helpful to me. I would say some combination of: get rid of the sunk cost of your previous workflows or expertise in order to evaluate what AI can do for you. That's right. And another way to put this, which is fun, is just be lazier in so much as figure out the way that the agent can do the things that are toilsome. But you're going to have to, in this, you ultimately get to be lazier, but in the short run, you need to critically think about the things you're currently doing and what an AI could actually be better at doing, and then go and try it or explore it. Because I think there's still just a lot of low-hanging fruit of people assuming and not writing the full prompt, giving a few examples, connecting the right tools for your work to be accelerated or automated.

是的。还有一种沉没成本，就是觉得自己没有‘早期’进入 AI 领域，已经错过了机会，不能再做什么了。但我记得 GPT-3 刚出来的时候——播客的背景故事：我大学毕业后，计划做一个 AI 说唱歌手创业项目，播客只是进入那个领域的入口。所以我尝试了各种东西，当时我记得自己在想，‘哦，3.5 出来了，人们……我想做自己的说唱歌手，但我在创业圈已经落后了。’我的意思是，也许说唱歌手的想法一开始就不可取，但就像，每次都觉得为时过早，因为这是一个指数增长的过程。而且很多事情、很多想法现在才变得可能，对吧？所以正是我之前说的产品指数级变化：产品实际上被淘汰了，你需要不断重塑自己才能保持在能力的前沿。顺便问一下，你还记得我有一个很烂的想法，然后给你打了电话吗？我不知道是什么了。好像是律师用的 RAG 之类的。总之，我想我们最初的互动之一就是我问：‘嘿，你觉得这个想法怎么样？’然后你说：‘我觉得播客听起来很有前途。’我很感激。

Yeah. There's also the sunk cost of feeling like since you're not 'early' to AI, you've sort of missed the boat and you can't. But I remember when GPT-3 came out—backstory on the podcast: when I graduated college, I was planning on doing some sort of AI rapper startup, and the podcast was just a gateway into doing that. So I was trying out different things, and at the time I remember thinking, 'Oh, 3.5 is out, and people like... I'm so behind on the startup scene here or whatever if I wanted to make my own rapper.' I mean, maybe the idea of the rapper was inadvisable in the first place, but just like, every time feels early because it's sort of an exponentially growing process. And there were many things, many ideas are only becoming possible now, right? So exactly that product exponential I talked about before: products literally obsoleted, you need to constantly reinvent yourself to stay at the frontier of capabilities. By the way, do you remember I had a really shitty idea and I gave you a call? I don't know what it was. It was like RAG for lawyers or something. Anyways, I think one of our first interactions was I'm like, 'Hey, what do you think of this idea?' And you're like, 'I think the podcast sounds promising.' Which I appreciate.

是的。我最近对一个朋友有点恼火，我觉得他非常有才华、聪明，对 AI 感兴趣，但走了生物学路线，我试图让他明白：‘如果你想，你可以做 AI。’我的意思是，我认为人类是生物通用智能，很多有价值的东西都是非常通用的。无论你做过什么专业，可能都没那么重要。我的意思是，这当然有成本，但很多人，甚至我在 Anthropic 的同事，都对 AI 充满热情，他们只是不让之前的职业成为障碍。因为他们天生聪明、有才华、有动力，他们最终都非常成功，找到了自己的角色。他们并不是一直在 AI 领域。人们来自完全不同的领域。所以不要认为你需要某个抽象实体的许可才能参与、应用和做出贡献。

Yeah. I got slightly annoyed at a friend recently who I think is really talented and clever and interested in AI but has pursued a biology route, and I just kind of tried to shake them of like, 'You can work on AI if you want to.' I mean, I think humans are biological general intelligences, and a lot of the things of value are just very general. And whatever kind of specialization that you've done maybe just doesn't matter that much. I mean, again, it costs, but so many of the people, even my colleagues at Anthropic, are excited about AI and they just don't let their previous career be a blocker. And because they're just innately smart, talented, driven, whatever else, they end up being very successful and finding roles. It's not as if they were in AI forever. People have come from totally different fields. And so don't think that you need permission from some abstract entity to get involved and apply and be able to contribute.

如果有人现在想成为 AI 研究员，如果你能给他们一个开放问题，这个问题很可能相当令人印象深刻，那会是什么？

If somebody wanted to be an AI researcher right now, if you could give them an open problem that is very likely to be quite impressive, what would it be?

我认为既然强化学习回来了，基于 Andy Jones 的棋盘游戏缩放定律的论文很有趣。比如展示你可以研究像你之前问的那些问题：模型是否真的学会了比之前 K 次尝试更多的东西，还是只是发现了那个？深入探索这类问题，我觉得很有趣。

I think that now that RL has come back, papers building on Andy Jones's scaling laws for board games are interesting. Like showing that you can investigate these questions like the ones you asked before: is the model actually learning to do more than its previous pass at K, or is it just discovering that? Exploring questions like that deeply, I think, are interesting.

基本上就像强化学习的缩放定律。我很好奇从新任务中获得的元学习边际增长有多少。关于这一点，我认为模型差异分析有很多机会。

Like scaling laws for RL basically. Very curious to see how much the marginal increase in meta-learning from a new task or something. On that note, I think model diffing has a bunch of opportunities.

人们说我们没有捕捉到所有特征，还有很多东西被遗漏了。那些被遗漏的东西是什么？如果模型被越狱，它是在使用你已经识别的现有特征，还是只使用你未捕捉到的误差项？我不知道。这里有很多问题。

People say we're not capturing all the features, there's all this stuff left on the table. What is that stuff left on the table? If the model is jailbroken, is it using existing features you've identified, or only the error terms you haven't captured? I don't know. There's a lot here.

我认为 Matts 很棒。Anthropic 的研究员项目进展顺利。Goodfire，Anthropic 最近投资了，他们做了很多可解释性工作。或者只是把任何东西应用到任何东西上以提高你的股权。有太多可解释性项目，太多唾手可得的成果，我们需要更多人。我不认为我们有很多时间。

I think Matts is great. The Anthropic fellowship has been going really well. Goodfire, Anthropic invested in recently. They're doing a lot of interpretability work. Or just apply anything to anything to get your equity up. There are so many interpretability projects, so much low-hanging fruit, and we need more people. I don't think we have much time.

我还想为性能工程做个宣传。我认为这是展示你原始能力的最佳方式之一。如果你在 TPU、Trillium 或 CUDA 上实现了一个极其高效的 Transformer，那么我认为你获得工作机会的可能性很高。能够完全端到端掌控模型性能的人很少。如果你有广泛而深厚的电气工程技能，我认为你可以在加速器方面很快上手。你可以相当快地掌握。它会让你对模型内部的实际复杂性有很好的直觉，这意味着你非常适合思考架构之类的事情。目前我在 Anthropic 最喜欢的一位架构思考者，实际上来自深厚的 GPU 内核编程背景，他非常深入地了解细节，并且能很好地权衡利弊。

I also want to make a plug for performance engineering. I think this is one of the best ways to demonstrate that you have the raw ability. If you made an extremely efficient transformer implementation on TPU or Trillium or in CUDA, then I think there's a pretty high likelihood that you'll get a job offer. There's a relatively small pool of people that you can trust to completely own end-to-end the performance of a model. If you have broad deep electrical engineering skills, I think you can probably come up to speed pretty fast on accelerator stuff. You can come up to speed reasonably fast. It teaches you a lot of good intuitions about the actual intricacies of what's going on in the models, which means you're then very well placed to think about architecture and this kind of stuff. One of my favorite people in thinking about architecture at Anthropic at the moment actually came from a heavy GPU kernel programming background, just knows the ins and outs really deeply and can think about the trade-offs really well.

这很有趣，各位。太棒了。谢谢。是的，很高兴回来。希望你喜欢这一集。如果喜欢，最有帮助的事情就是分享给其他你认为会喜欢的人。发给你的朋友、群聊、Twitter 等等。让消息传开。除此之外，如果你能在 YouTube 上订阅并在 Apple Podcasts 和 Spotify 上留下五星评价，会非常有帮助。查看下方描述中的赞助商。如果你想赞助未来的节目，请访问 dwarcash.com/advertise。感谢收听。下次见。

This is fun, guys. Awesome. Thanks. Yeah, great to be back. I hope you enjoyed this episode. If you did, the most helpful thing you can do is just share it with other people who you think might enjoy it. Send it to your friends, your group chats, Twitter, wherever else. Just let the word go forth. Other than that, super helpful if you can subscribe on YouTube and leave a five-star review on Apple Podcasts and Spotify. Check out the sponsors in the description below. If you want to sponsor a future episode, go to dwarcash.com/advertise. Thank you for tuning in. I'll see you on the next one.