我们已接近指数曲线的尽头

我们三年前聊过。我很好奇，在你看来，过去三年最大的变化是什么？三年前的感觉和现在最大的不同在哪里？

So, we talked 3 years ago. I'm curious in your view, what has been the biggest update of the last 3 years? What has been the biggest difference between what it felt like last 3 years versus now?

嗯，实际上，底层技术，也就是技术的指数级增长，大体上和我预期的差不多。我的意思是，前后可能差一两年。我不确定能否预测代码的具体方向。但当我观察这个指数增长时，模型从聪明的高中生到聪明的大学生，再到开始做博士和专业工作，在代码方面甚至超越了这个水平，这大致符合我的预期。所以前沿有点不均衡，但基本如我所料。不过，最让我惊讶的是，公众几乎没有意识到我们离指数增长的终点有多近。对我来说，简直不可思议，圈内圈外的人还在讨论那些老掉牙的热点政治问题，仿佛指数增长不会结束似的。

Yeah, I would say actually the underlying technology, like the exponential of the technology, has gone, broadly speaking, about as I expected it to go. I mean, there's plus or minus a year or two here and there. I don't know that I would predict the specific direction of code. But when I look at the exponential, it is roughly what I expected in terms of the march of the models from smart high school student to smart college student to beginning to do PhD and professional stuff, and in the case of code reaching beyond that. So the frontier is a little bit uneven. It's roughly what I expected. I will tell you though what the most surprising thing has been. The most surprising thing has been the lack of public recognition of how close we are to the end of the exponential. To me, it is absolutely wild that you have people inside and outside the bubble talking about the same tired old hot button political issues, like near the end of the exponential.

我想了解现在这个指数增长是什么样的，因为三年前我录节目时问你的第一个问题就是：Scaling（规模扩张）怎么样了？它是如何运作的？我现在也有类似的问题，但我觉得这个问题更复杂了，因为至少从公众的角度看，三年前确实有那些众所周知的趋势，在跨越多个数量级的算力上，你可以看到损失如何改善。而现在我们有了 RL Scaling，却没有公开已知的缩放定律。甚至不清楚具体是怎么回事：它是在教模型技能吗？是在教元学习吗？现在的 Scaling 假设到底是什么？

I want to understand what that exponential looks like right now because the first question I asked you when we recorded 3 years ago was, what's up with scaling? How does it work? And I have a similar question now, but I feel like it's a more complicated question because at least from the public's point of view, yes, 3 years ago there were these well-known public trends where across many orders of magnitude of compute you could see how the loss improves. And now we have RL scaling and there's no publicly known scaling law for it. It's not even clear what exactly the story is: is it supposed to be teaching the model skills? Is it supposed to be teaching meta learning? What is the scaling hypothesis at this point?

嗯，实际上，我持有的假设和 2017 年时完全一样。2017 年，我写了一份文档，叫「大算力块假设」。它并不特指语言模型的 Scaling。当时 GPT-1 刚出来，这只是众多事情之一。那时候还有机器人学，人们试图把推理作为独立于语言模型的东西来研究，还有 AlphaGo 中的 RL Scaling，OpenAI 在 Dota 上的工作，以及 DeepMind 在星际争霸上的 AlphaStar。所以这份文档更通用。我具体说的是：所有的聪明才智、所有技巧、所有「我们需要新方法」之类的东西，都不太重要。只有少数几件事重要，我列出了七件。一是你拥有的原始算力。二是数据量。三是数据的质量和分布，需要广泛分布。四是你训练的时间。五是你需要一个能扩展到极致的目标函数。预训练目标函数就是其中之一。另一个是 RL 目标函数，比如你有一个目标，你要去实现它。其中当然有客观奖励，比如数学和编程中的，也有更主观的奖励，比如基于人类反馈的强化学习（RLHF）或更高级的版本。第六和第七是关于归一化或条件化，比如确保数值稳定性，让大算力块平稳流动而不出问题。这就是我的假设，我仍然坚持。我认为没有看到太多与这个假设不符的东西。预训练的缩放定律就是其中一个例子，而且它们一直在持续。现在有很多报道说我们对预训练感觉良好，预训练仍在带来收益。变化在于，我们现在在 RL 中也看到了同样的情况。所以我们有预训练阶段，然后在此基础上进行 RL 阶段。RL 其实也一样：其他公司也发表过文章说，「看，我们在数学竞赛（如 AIME）上训练模型，模型的表现与训练时间呈对数线性关系。」我们也看到了这一点，而且不只是数学竞赛，各种 RL 任务都是如此。所以我们在 RL 中看到了和预训练一样的 Scaling。

Yeah, so I have actually the same hypothesis that I had even all the way back in 2017. So, in 2017, I think I talked about it last time, but I wrote a doc called the big blob of compute hypothesis. And it wasn't about the scaling of language models in particular. When I wrote it, GPT-1 had just come out, right? So that was one among many things. Back in those days there was robotics, people trying to work on reasoning as a separate thing from language models, there was scaling of the kind of RL that happened in AlphaGo, and that happened at Dota at OpenAI, and people remember StarCraft at DeepMind, AlphaStar. So it was written as a more general document. And the specific thing I said was the following, and Rich Sutton put out the bitter lesson a couple years later, but the hypothesis is basically the same. So what it says is all the cleverness, all the techniques, all the kind of 'we need a new method to do something' doesn't matter very much. There are only a few things that matter, and I think I listed seven of them. One is how much raw compute you have. The other is the quantity of data that you have. Then the third is kind of the quality and distribution of data, right? It needs to be a broad distribution of data. The fourth is I think how long you train for. The fifth is you need an objective function that can scale to the moon. So the pre-training objective function is one such objective function, right? Another objective function is the kind of RL objective function that says you have a goal, you're going to go out and reach the goal. Within that, of course, there are objective rewards like you see in math and coding, and there are more subjective rewards like you see in RL from human feedback or higher order versions of that. And then the sixth and seventh were things around normalization or conditioning, like just getting the numerical stability so that the big blob of compute flows in a laminar way instead of running into problems. So that was the hypothesis and it's a hypothesis I still hold. I don't think I've seen very much that is not in line with that hypothesis. And so the pre-trained scaling laws were one example of what we see there. And indeed those have continued going. I think now it's been widely reported that we feel good about pre-training. Pre-training is continuing to give us gains. What has changed is that now we're also seeing the same thing for RL, right? So we're seeing a pre-training phase and then an RL phase on top of that. And with RL it's actually just the same: even other companies have published things that say, 'Look, we trained the model on math contests, AIME or other things, and how well the model does is log linear in how long we've trained it.' And we see that as well, and it's not just math contests. It's a wide variety of RL tasks. And so we're seeing the same scaling in RL that we saw for pre-training.

我不知道这是否不敬，但一种概括这种反对意见的方式是：「看，真正具备人类学习核心的东西，不需要花费数十亿美元的数据、算力和这些定制环境来学习如何使用 Excel、PowerPoint 或浏览网页。我们不得不通过强化学习环境来构建这些技能，这一事实暗示我们实际上缺乏这种核心的人类学习算法，因此我们在错误的方向上做规模扩张。那么，如果我们认为会有某种东西能像人类一样快速学习，为什么还要进行所有这些强化学习的规模扩张呢？」

And if I'm if I I don't know if this is disrespectful, but one way to paraphrase this objection is something like, 'Look, something which possesses the true core of human learning will not require all these billions of dollars of data and compute and these bespoke environments to learn how to use Excel or how does an account you know how to how to use PowerPoint how to navigate a web browser and the fact that we have to build in these skills using these RL environments hints that we're actually lacking this core human learning algorithm and so we're scaling the wrong thing and so yeah that that is the reason question why are we doing all this RL scaling if we do think there's something that's going to be human-like in its ability to learn on the fly.

是的。我认为这实际上把几件应该分开考虑的事情混在一起了。这里确实有一个真正的难题，但它可能并不重要。事实上，我猜它很可能不重要。我们先暂时把强化学习放在一边，因为我实际上认为强化学习和预训练在这方面并没有本质区别。回顾 2017 年 Alec Radford 做 GPT-1 时的预训练和规模扩张，很有意思。在 GPT-1 之前，模型都是在不能代表广泛文本分布的数据集上训练的，比如那些非常标准的语言建模基准。GPT-1 本身是在一堆同人小说上训练的，那是文学文本，只占文本的一小部分。当时数据集只有大约十亿个词，分布很窄，模型泛化能力很差。只有在整个互联网上训练，比如用 Common Crawl 或 Reddit 链接抓取（就像我们做 GPT-2 那样），才开始出现泛化。我认为强化学习也是如此：我们从非常简单的任务开始，比如数学竞赛训练，然后扩展到更广泛的训练，包括代码等任务，现在又扩展到许多其他任务。我认为我们会越来越看到泛化。这就把强化学习和预训练的差异问题解决了。但无论哪种方式，都有一个难题：在预训练中，我们使用数万亿个 token，而人类不会看到数万亿个词。所以样本效率确实有差异。模型从零开始，需要更多的训练。但我们也看到，一旦训练完成，如果给它们很长的上下文窗口（比如一百万个 token），它们就能很好地学习和适应。所以，我不完全知道答案，但我认为预训练并不像人类学习的过程，它介于人类学习和人类进化之间。我们很多先验来自进化，大脑不是白板。而语言模型更像是白板，从随机权重开始。人类大脑则有各种区域，连接着各种输入和输出。所以，也许我们应该把预训练（以及强化学习）看作介于人类进化和人类即时学习之间的东西，而模型的上下文学习则介于人类长期学习和短期学习之间。存在一个层次：进化、长期学习、短期学习、人类反应。LLM 的阶段分布在这个谱系上，但并不完全对应人类学习的各个点。LLM 有点落在点之间。这说得通吗？

Yeah, yeah. So I think I think this kind of puts together several things that should be kind of thought of thought of differently. I think there is a genuine puzzle here but it it may not matter. In fact I would guess it probably it probably doesn't matter. So let's take the RL out of it for a second cuz I actually think RL and it's a red herring to say that RL is any different from pre-training in this matter. So if we if we look at pre-training and scaling it it was very interesting back in you know 2017 when Alec Radford was doing GPT-1. If you look at the models before GPT-1 they were trained on these data sets that didn't represent a wide you know distribution of text right you had like you know these very standard you know kind of language modeling benchmarks and GPT-1 itself was trained on a bunch of I think it was fan fiction actually but you know it was it was like literary and it's like literary text which is a very small fraction of the text that you get and what we found with that you know and in those days it was like a billion words or something so small data sets and represented a pretty narrow distribution right like a narrow distribution of kind of what what you can see what you can see in the world and it didn't generalize well if you did better on you know the the you know I I I forgot what what some some kind of fan fiction corpus. Um, it wouldn't generalize that well to kind of the other task. You know, we had all these measures of like, you know, how well does it how well does a model do at predicting all of these other kinds of tasks? You really didn't see the generalization. It was only when you trained over all the tasks on the you know, the internet. When you when you kind of did a general internet scrape, right? From something like, you know, common crawl or scraping links on Reddit, which is what we did for GPT-2. It's only when you do that that you kind of started to get generalization. Um, and I think we're seeing the same thing on RL, that we're starting with first very simple RL tasks like training on math competitions, then we're kind of moving to, you know, kind of broader broader training that involves things like code as a task, and now we're moving to do kind of many many other tasks. And then I think we're going to increasingly get generalization. So, that that kind of takes out the RL versus the pre-training side of it. But, I think there is a puzzle here either way, which is that on pre-training, when we train the model on pre-training, you know, we we use like trillions of tokens, right? And and humans don't see trillions of words. So, there is an actual sample efficiency difference here. There there is actually something different that's that's happening here, which is that the models start from scratch, and you know, they have to get much more much more training. But, we also see that once they're trained, if we give them a long context length. The only thing a long context length is like inference, but if we give them like a context length of a million, they're very good at learning and adapting within that context length. And and so, I don't know the full answer to this, but but I think there's something going on that pre-training it's it's not like the process of humans learning. It's somewhere between the process of humans learning and the process of human evolution. It's like it's somewhere between like we get many of our priors from evolution. Our brain isn't just a blank slate, right? Whole books have been written about I think the language models, they're much more blank slates. They literally start as like random weights. Whereas the human brain starts with all these regions, it's connected to all these inputs and outputs. Um and and so maybe we should think of pre-training and for that matter RL as well as as being something that exists in the middle space between human evolution and you know, kind of human on on the spot learning. And as the in-context learning that the models do as as something between long-term human learning and short-term human learning. So, you know, there there's this hierarchy of like there's evolution, there's long-term learning, there's short-term learning, and there's just human reaction. And the LLM phases exist along this spectrum, but not necessarily exactly at the same points. That there's no analog to some of the human modes of learning. The LLMs are kind of falling between the points. Does that make sense?

嗯，是的，但有些东西还是有点令人困惑。例如，如果这个类比是像进化一样，那么样本效率不高也没关系。但如果我们能从上下文学习中获得超级样本高效的智能体，为什么还要费心去构建那些强化学习环境公司呢？他们似乎是在教模型如何使用 API、Slack 等等。如果那种能即时学习的智能体正在出现或即将出现或已经出现，为什么还要如此强调这些呢？

Um yes, although some things are still a bit confusing. For example, if the analogy is that this is like evolution, so it's fine that it's not that sample efficient, then like well, if we're going to get the kind of super sample efficient agent from in-context learning, why are we bothering to build in, you know, there's RL environment companies which are It seems like what they're doing is they're teaching it how to use this API, how to use Slack, how to use whatever. It's confusing to me why there's so much emphasis on that if the kind of agent that can just learn on the fly is emerging or is going to soon emerge or has already emerged.

是的。我不能代表其他人的重点，我只能谈谈我们是怎么想的。

Yeah, yeah. So, I I mean, I can't speak for the emphasis of anyone else. I can I can only talk about how we how we think about it.

我认为我们的思考方式是，目标不是在强化学习中教给模型每一种可能的技能，就像我们在预训练中不做那样。在预训练中，我们并不是试图让模型接触所有可能的词语组合方式。相反，模型在大量数据上训练，然后在预训练中实现泛化。那是我近距离观察到的从 GPT-1 到 GPT-2 的转变。模型达到了一个点，你给它一串数字，比如房子的价格和面积，它就能补全模式并做线性回归。虽然做得不太好，但它确实做到了，而且它从未见过完全一样的东西。所以，在我们构建这些强化学习环境时，目标与 5 或 10 年前预训练的目标非常相似。我们试图获取大量数据，不是因为要覆盖特定的文档或技能，而是因为我们要泛化。

I think the way we think about it is the goal is not to teach the model every possible skill within RL, just as we don't do that within pre-training. Within pre-training, we're not trying to expose the model to every possible way that words could be put together. Rather, the model trains on a lot of things and then reaches generalization across pre-training. That was the transition from GPT-1 to GPT-2 that I saw up close. The model reaches a point where you give it a list of numbers like the cost of a house and the square feet, and it completes the pattern and does linear regression. Not great, but it does it, and it's never seen that exact thing before. So to the extent that we are building these RL environments, the goal is very similar to what was done 5 or 10 years ago with pre-training. We're trying to get a whole bunch of data, not because we want to cover a specific document or skill, but because we want to generalize.

我认为你提出的框架显然是有道理的。我们正在向 AGI 迈进。我认为关键在于，现在没有人不同意我们将在本世纪实现 AGI。而关键在于，你说我们正在接近指数增长的终点，而另一些人看着这个说：「哦，是的，我们从 2012 年开始就在进步，到 2035 年我们将拥有类人 AGI。」所以我想了解你看到了什么，让你认为我们显然在这些模型中看到了进化或人类一生学习所做的事情。为什么你认为它是一年后而不是十年后？

I think the framework you're laying down obviously makes sense. We're making progress towards AGI. I think the crux is something like nobody at this point disagrees that we're going to achieve AGI in the century. And the crux is you say we're hitting the end of the exponential, and somebody else looks at this and says, 'Oh, yeah, we've been making progress since 2012 and then 2035 we'll have a human-like AGI.' So I want to understand what it is that you're seeing which makes you think obviously we're seeing the kinds of things that evolution did or that human lifetime learning is like in these models. And why think that it's 1 year away and not 10 years away?

我实际上认为这里有两种情况，或者说你可以提出两种主张，一种更强，另一种更弱。从较弱的主张开始：当我在 2019 年第一次看到 Scaling 时，我并不确定。这大概是一半一半的事情。我以为我看到了某种比任何人认为的都更可能的事情。也许有 50%的机会发生。关于基本假设，即十年内我们将达到我所说的「数据中心里的天才国度」，我大概有 90%的把握。很难超过 90%，因为世界如此不可预测。也许不可约的不确定性是，如果我们有 95%的把握，你会遇到像多家公司内部动荡而一事无成，然后台湾被入侵，所有晶圆厂被导弹炸毁的情况。你可以构造一个场景，有 5%的概率事情会推迟十年。那大概是 5%。另外 5%是，我对可验证的任务非常有信心。对于编程，除了那不可约的不确定性，我认为我们一两年内就能达到。在端到端编程方面，十年内我们不可能达不到。我在长时间尺度上唯一的基本不确定性是关于不可验证的任务，比如规划火星任务、进行像 CRISPR 这样的基础科学发现，或者写小说。这些任务很难验证。我几乎确信我们有可靠的路径达到那里，但如果有一点不确定性，那就是它。所以对于十年，我有 90%的把握，这几乎是你能达到的最高确定性。我认为说这不会在 2035 年之前发生是疯狂的。在理智的世界里，这会是主流之外的观点。

I actually think of it as two cases to be made here, or two claims you could make, one of which is stronger and the other weaker. Starting with the weaker claim: when I first saw the scaling back in 2019, I wasn't sure. This was a 50/50 thing. I thought I saw something that was much more likely than anyone thinks. Maybe there's a 50% chance this happens. On the basic hypothesis of within 10 years we'll get to what I call 'country of geniuses in a data center', I'm at like 90%. It's hard to go much higher than 90% because the world is so unpredictable. Maybe the irreducible uncertainty would be if we were at 95%, where you get to things like multiple companies have internal turmoil and nothing happens, and then Taiwan gets invaded and all the fabs get blown up by missiles. You could construct a scenario with a 5% chance that things get delayed for 10 years. That's maybe 5%. There's another 5% which is that I'm very confident on tasks that can be verified. With coding, except for that irreducible uncertainty, I think we'll be there in one or two years. There's no way we will not be there in 10 years in terms of being able to do end-to-end coding. My one bit of fundamental uncertainty even on long time scales is about tasks that aren't verifiable, like planning a mission to Mars, doing some fundamental scientific discovery like CRISPR, or writing a novel. It's hard to verify those tasks. I am almost certain that we have a reliable path to get there, but if there was a little bit of uncertainty, it's there. So on the 10 years, I'm 90%, which is about as certain as you can be. I think it's crazy to say that this won't happen by 2035. In some sane world it would be outside the mainstream.

但对验证的强调向我暗示，你缺乏对这些模型会泛化的信念。想想人类，我们既擅长那些我们能获得可验证奖励的事情，也擅长那些我们不能的事情。你必须摒弃这一点。

But the emphasis on verification hints to me as a lack of belief that these models will generalize. If you think about humans, we are good at things for which we get verifiable reward and things for which we don't. You have to discard that.

不，不，这就是为什么我几乎确定。我们已经看到从可验证事物到不可验证事物的大量泛化。我们已经看到了。

No, no, this is why I'm almost sure. We already see substantial generalization from things that verify to things that don't verify. We're already seeing that.

对。但似乎你强调这是一个谱系，会把你看到更多进展的领域分开。而我想，这似乎不是人类进步的方式。

Right. But it seems like you were emphasizing this as a spectrum which will split apart which domains you see more progress in. And I'm like, but that doesn't seem like how humans get better.

我们未能成功，或者说我们未能达到目标的世界，是我们做了所有可验证的事情，然后其中许多泛化了，但我们没有完全填满盒子的这一边。这不是一个二元的事情。但在我看来，即使在泛化较弱的世界里，当你只在可验证领域看到它时，我不清楚在这样的世界里你是否能自动化软件工程，因为作为软件工程师的一部分涉及写关于你宏大愿景的长篇备忘录。所以……

The world in which we don't make it, or the world in which we don't get there, is the world in which we do all the things that are verifiable and then many of them generalize, but we don't fully color in this side of the box. It's not a binary thing. But it also seems to me that even in the world where generalization is weak when you only see it in verifiable domains, it's not clear to me in such a world you could automate software engineering, because part of being a software engineer involves writing long memos about your grand vision. And so...

我认为那是软件工程师工作的一部分。那是公司工作的一部分。但我确实认为软件工程涉及设计文档和其他类似的东西，顺便说一句，模型在这方面并不差。它们已经相当擅长写注释了。

I think that's part of the job of a software engineer. That's part of the job of the company. But I do think software engineering involves design documents and other things like that, which by the way the models are not bad at. They're already pretty good at writing comments.

所以，我在这里提出的主张比我实际相信的要弱得多，目的是区分两件事。对于软件工程师来说，我们已经快接近了。用什么指标衡量？一个指标是：有多少行代码是由 AI 编写的。但如果你考虑软件工程历史上其他的生产力提升，编译器写了所有的软件代码。写的行数和生产力提升的大小之间是有区别的。所以我们说「快接近了」，指的是生产力提升的大小，而不仅仅是写了多少行代码。

And so with that, I'm making much weaker claims than I believe to distinguish between two things. We're already almost there for software engineers. By what metric? There's one metric: how many lines of code are written by AI. But if you consider other productivity improvements in the history of software engineering, compilers write all the lines of software. There's a difference between how many lines are written and how big the productivity improvement is. So we're almost there meaning how big the productivity improvement is, not just how many lines are written.

是的，我实际上同意你的观点。我做过一系列关于代码和软件工程的预测，我认为人们一再误解了它们。让我来列出这个频谱。大约八九个月前，我说 AI 模型将在三到六个月内编写 90%的代码行。这在一些地方发生了——在 Anthropic，以及许多下游使用我们模型的人那里。但这实际上是一个非常弱的指标。人们以为我在说我们不需要 90%的软件工程师了。这两者天差地别。我会把频谱列为：模型编写 90%的代码，模型编写 100%的代码——这在生产力上是一个巨大的差异。然后是 90%的端到端软件工程任务，包括编译、设置集群和环境、测试功能、写备忘录，都由模型完成。然后是 100%的当今软件工程任务由模型完成。即使发生了这种情况，也不意味着软件工程师失业了。他们可以做新的更高层次的事情，比如管理。再往下，有 90%的软件工程师需求减少，我认为这会发生。但这是一个频谱。我在《技术的青春期》中写过这个，我用农业的例子讲了整个频谱。所以我完全同意你的观点。这些是非常不同的基准，但我们正在以极快的速度推进它们。

Yeah, I actually agree with you on this. I've made a series of predictions on code and software engineering, and I think people have repeatedly misunderstood them. Let me lay out the spectrum. About eight or nine months ago, I said the AI model would be writing 90% of the lines of code in three to six months. That happened at some places—at Anthropic, and with many people downstream using our models. But that's actually a very weak criterion. People thought I was saying we wouldn't need 90% of software engineers. Those things are worlds apart. I would put the spectrum as: 90% of code is written by the model, 100% of code is written by the model—that's a big difference in productivity. Then 90% of end-to-end SWE tasks, including compiling, setting up clusters and environments, testing features, writing memos, are written by the models. Then 100% of today's SWE tasks are written by the models. Even when that happens, it doesn't mean software engineers are out of a job. There are new higher-level things they can do, like managing. Further down the spectrum, there's 90% less demand for SWEs, which I think will happen. But this is a spectrum. I wrote about it in 'The Adolescence of Technology', where I went through this spectrum with farming. So I totally agree with you. These are very different benchmarks, but we're proceeding through them super fast.

你的愿景的一部分似乎是从 90 到 100。首先，这会很快发生，其次，这会导致巨大的生产力提升。但当我注意到，即使在绿地项目中，人们开始使用 Claude Code 之类的工具，他们报告说启动了很多项目。我想知道：我们在世界上是否看到了软件的复兴，所有这些原本不会存在的新功能？至少到目前为止，我们似乎没有看到。这让我怀疑，即使我永远不需要干预 Claude Code，但世界是复杂的，工作是复杂的，对于自包含系统，无论我只是写软件还是其他什么，仅仅从这一点我们能获得多大的更广泛的收益。也许这应该削弱我们对「天才之国」的估计。

It seems like part of your vision is going from 90 to 100. First, it's going to happen fast, and second, that somehow leads to huge productivity improvements. But when I notice even in greenfield projects that people start with Claude Code or something, people report starting a lot of projects. I wonder: do we see in the world a renaissance of software, all these new features that wouldn't exist otherwise? At least so far, it doesn't seem like we see that. That makes me wonder, even if I never had to intervene on Claude Code, there is this thing: the world is complicated, jobs are complicated, and closing the loop on self-contained systems, whether I'm just writing software or something, how much broader gains we would see just from that. Maybe that should dilute our estimation of the country of geniuses.

嗯，我同时同意你的观点，即这是事情不会瞬间发生的原因，但同时，我认为效果会非常快。有两个极端。一个是 AI 不会取得进展，它很慢，需要永远在经济中扩散。经济扩散已经成为一种流行词，用来解释为什么 AI 进展不重要。另一个极端是递归自我改进，整个事情，你可以在曲线上画一条指数线——我们将在获得递归后的纳秒内拥有戴森球。我完全是在夸张地描述这个观点，但确实存在这两个极端。我们从一开始就看到的，至少在 Anthropic 内部，是每年 10 倍的收入增长。2023 年，从 0 到 1 亿。2024 年，从 1 亿到 10 亿。2025 年，从 10 亿到 90 或 100 亿。今年第一个月，你会认为指数增长会放缓，但我们在 1 月份又增加了数十亿的收入。显然，这条曲线不能永远持续下去——GDP 只有那么大。我甚至猜测它今年会有所弯曲。但这是一条快速的曲线，一条非常快速的曲线，我打赌即使规模扩大到整个经济，它也会保持相当快。所以我认为我们应该考虑这个中间世界：事情非常快，但不是瞬间，它们需要时间，因为经济扩散，因为需要闭环，因为繁琐的事情，比如企业内部的变更管理，更改安全权限以使其真正工作。

Well, I simultaneously agree with you that it's a reason why these things don't happen instantly, but at the same time, I think the effect is going to be very fast. There are two poles. One is that AI is not going to make progress, it's slow, it's going to take forever to diffuse within the economy. Economic diffusion has become a buzzword for why AI progress doesn't matter. The other axis is recursive self-improvement, the whole thing, and you can just draw an exponential line on the curve—we're going to have Dyson spheres around the sun nanoseconds after we get recursive. I'm completely caricaturing the view, but there are these two extremes. What we've seen from the beginning, at least within Anthropic, is a bizarre 10x per year growth in revenue. In 2023, it was 0 to 100 million. 2024, it was 100 million to a billion. 2025, it was a billion to 9 or 10 billion. And the first month of this year, that exponential would you think would slow down, but we added another few billion to revenue in January. Obviously that curve can't go on forever—GDP is only so large. I would even guess it bends somewhat this year. But that is a fast curve, a really fast curve, and I would bet it stays pretty fast even as the scale goes to the entire economy. So I think we should be thinking about this middle world where things are extremely fast, but not instant, where they take time because of economic diffusion, because of the need to close the loop, because of fiddly things like change management within an enterprise, changing security permissions to make it actually work.

我有一款旧软件，用于在模型编译和发布前进行检查，我不得不重写它。是的，模型能做到，但我必须告诉它去做，而且它需要时间来完成。所以我认为，我们目前看到的一切都与这样一种观点相符：一方面是指数级增长的能力，另一方面是指数级增长的经济渗透。不是瞬间完成，也不是缓慢推进，比以往任何技术都快得多，但也有其极限。这就是我在 Anthropic 内部、在客户身上看到的：快速采用，但并非无限快。

I had this old piece of software that checks the model before it's compiled and released, and I have to rewrite it. Yes, the model can do that, but I have to tell the model to do that, and it has to take time to do that. So I think everything we've seen so far is compatible with the idea that there's one fast exponential that's the capability of the model, and then there's another fast exponential that's downstream of that, which is the diffusion of the model into the economy. Not instant. Not slow. Much faster than any previous technology, but it has its limits. And this is what I see when I look inside Anthropic, when I look at our customers: fast adoption, but not infinitely fast.

我能跟你分享一个大胆的观点吗？

Can I try a hot take on you?

当然。

Yeah.

我觉得「渗透」是一种自我安慰，当模型无法完成某件事时，人们就说这是渗透问题。但你应该拿人类来比较。你会认为，AI 固有的优势会让新 AI 的入职比新人类容易得多。AI 可以在几分钟内读完你整个 Slack 和 Drive，它们可以共享同一实例其他副本的所有知识。招聘 AI 时没有逆向选择问题，因为你可以直接雇佣经过验证的 AI 模型副本。招聘人类麻烦得多，但人类却一直被雇佣。我们每年支付人类超过 50 万亿美元的工资，因为他们有用，尽管将 AI 整合到经济中比雇佣人类容易得多。所以我认为渗透并不能解释。

I feel like diffusion is cope that people use to say when the model isn't able to do something, they say it's a diffusion issue. But you should use the comparison to humans. You would think that the inherent advantages that AIs have would make diffusion a much easier problem for new AIs getting onboarded than new humans getting onboarded. So an AI can read your entire Slack and Drive in minutes. They can share all the knowledge that other copies of the same instance have. You don't have this adverse selection problem when you're hiring AIs because you can just hire copies of a vetted AI model. Hiring a human is so much more hassle, and people hire humans all the time. We pay humans upwards of 50 trillion dollars in wages because they're useful, even though it would be much easier to integrate AIs into the economy than to hire humans. So I think diffusion doesn't really explain.

我认为渗透是真实存在的，而且并不完全与 AI 模型的局限性有关。有些人把渗透当作流行词，说这没什么大不了的。我不是在说那种情况。我不是说 AI 会以以往技术的速度渗透。我认为 AI 的渗透会比以往任何技术都快得多，但并非无限快。我举个例子：Claude Code。它设置起来非常简单。如果你是开发者，直接就能用。大型企业的开发者没有理由不能像个人开发者或初创公司开发者那样快速采用 Claude Code。我们竭尽全力推广它。我们把 Claude Code 卖给企业、大型金融公司、大型制药公司。它们采用的速度比企业通常采用新技术的速度快得多。但这需要时间。任何特定功能或产品，比如 Claude Code 或 Co-work，都会被 Twitter 上的个人开发者、A 轮初创公司比大型食品销售企业早几个月采用。原因有很多：要走法律流程、为所有人配置、通过安全和合规审查。公司领导层离 AI 革命较远，他们虽然有远见，但必须说：「花 5000 万是合理的。Claude Code 是什么？为什么能帮助公司？为什么能提高生产力？」然后他们得向下面两级的人解释：「我们有 3000 名开发者。这是我们的推广计划。」我们每天都在进行这样的对话。我们竭尽全力让 Anthropic 的收入每年增长 20 到 30 倍，而不是 10 倍。许多企业说：「这太高效了。我们会在常规采购流程中走捷径。」他们比我们当初只卖普通 API 时快得多，很多企业都用 API，但 Claude Code 是更有吸引力的产品。但它不是无限吸引人的产品，我认为即使是 AGI 或强大的 AI，或数据中心里的天才之国，也不会是无限吸引人的产品。它可能足够有吸引力，让你在营收达到数千亿美元时还能实现每年 3 倍、5 倍或 10 倍的增长，这极其困难，历史上从未有过，但并非无限快。

I think diffusion is very real and doesn't exclusively have to do with limitations on the AI models. Again, there are people who use diffusion as a buzzword to say this isn't a big deal. I'm not talking about that. I'm not talking about AI will diffuse at the speed of previous technologies. I think AI will diffuse much faster than previous technologies have, but not infinitely fast. I'll give an example: Claude Code. It's extremely easy to set up. If you're a developer, you can just start using it. There is no reason why a developer at a large enterprise should not be adopting Claude Code as quickly as an individual developer or a startup developer. And we do everything we can to promote it. We sell Claude Code to enterprises, big financial companies, big pharmaceutical companies. They are adopting it much faster than enterprises typically adopt new technology. But it takes time. Any given feature or product like Claude Code or Co-work will get adopted by individual developers on Twitter, by Series A startups, many months faster than by a large enterprise that does food sales. There are factors: you have to go through legal, provision it for everyone, pass security and compliance. The leaders of the company, who are further away from the AI revolution, are forward-looking but they have to say, 'It makes sense for us to spend 50 million. This is what Claude Code is. This is why it helps our company. This is why it makes us more productive.' Then they have to explain to the people two levels below and say, 'We have 3,000 developers. Here's how we're going to roll it out.' We have conversations like this every day. We are doing everything we can to make Anthropic's revenue grow 20 or 30x a year instead of 10x a year. Many enterprises are saying, 'This is so productive. We're going to take shortcuts in our usual procurement process.' They are moving much faster than when we tried to sell them just the ordinary API, which many of them use, but Claude Code is a more compelling product. But it's not an infinitely compelling product, and I don't think even AGI or powerful AI or a country of geniuses in a data center will be an infinitely compelling product. It will be compelling enough maybe to get three or five or 10x a year growth even when you're in the hundreds of billions of dollars, which is extremely hard to do and has never been done in history before, but not infinitely fast.

我接受这会带来轻微放缓。也许这不是你的观点，但有时人们会这样说：「哦，能力已经具备了，但因为渗透，否则我们基本上已经达到 AGI 了。」

I buy that it would be a slight slowdown. And maybe this is not your claim, but sometimes people talk about this like, 'Oh, the capabilities are there, but because of diffusion, otherwise we're basically at AGI.'

我不认为我们基本上已经达到 AGI。我认为，如果你在数据中心里有一个天才之国，而你的公司没有采用它，我们会知道。这个房间里的每个人都会知道。华盛顿的每个人都会知道。偏远地区的人可能不知道，但我们会知道。我们现在没有这个。这一点非常清楚。

I don't believe we're basically at AGI. I think if you had the country of geniuses in a data center, if your company didn't adopt it, we would know it. Everyone in this room would know it. Everyone in Washington would know it. People in rural parts might not know it, but we would know it. We don't have that now. That is very clear.

正如 Dario 所暗示的，要实现泛化，你需要在各种现实任务和环境中进行训练。例如，对于销售智能体，最难的部分不是教它在 Salesforce 的特定数据库中点击按钮，而是训练它在模糊情境下的判断力。如何从数千条线索的数据库中找出热门线索？如何主动联系？被忽视时该怎么办？当一个 AI 实验室想要训练销售智能体时，Labelbox 请来了数十名财富 500 强的销售人员，构建了各种不同的强化学习环境。他们创建了数千个场景，销售智能体需要与潜在客户互动，而客户由另一个 AI 扮演。

As Dario was hinting at, to get generalization, you need to train across a wide variety of realistic tasks and environments. For example, with a sales agent, the hardest part isn't teaching it to mash buttons in a specific database in Salesforce. It's training the agent's judgment across ambiguous situations. How do you sort through a database with thousands of leads to figure out which ones are hot? How do you actually reach out? What do you do when you get ghosted? When an AI lab wanted to train a sales agent, Labelbox brought in dozens of Fortune 500 sales people to build a bunch of different RL environments. They created thousands of scenarios where the sales agent had to engage with the potential customer, which was role-played by a second AI.

Labelbox 确保这个客户 AI 拥有几种不同的角色，因为当你打陌生电话时，你根本不知道电话那头是谁。你需要能够应对各种可能性。Labelbox 的销售专家逐轮监控这些对话，调整角色扮演智能体，确保它们做出真实客户会做的事情。Labelbox 的迭代速度比业内任何其他公司都快。这非常重要，因为强化学习是一门经验科学，而不是软件问题。Labelbox 有一系列工具可以实时监控智能体表现，让专家不断提出新任务，从而使模型保持在合适的难度分布中，并在训练中获得最优的奖励信号。Labelbox 几乎可以在任何领域做到这一点。他们拥有首席基金经理、放射科医生，甚至飞行员。所以，无论你在做什么，Labelbox 都能提供帮助。更多信息请访问 labelbox.com/vorkash。

Labelbox made sure that this customer AI had a few different personas, because when you cold call, you have no idea who's going to be on the other end. You need to be able to deal with a whole range of possibilities. Labelbox's sales experts monitored these conversations turn by turn, tweaking the role-playing agent to ensure they did the kinds of things an actual customer would do. Labelbox could iterate faster than anybody else in the industry. This is super important because RL is an empirical science. It's not a software problem. Labelbox has a bunch of tools for monitoring agent performance in real time. This lets their experts keep coming up with tasks, so that the model stays in the right distribution of difficulty and gets the optimal reward signal during training. Labelbox can do this sort of thing in almost every domain. They've got head fund managers, radiologists, even airline pilots. So, whatever you're working on, Labelbox can help. Learn more at labelbox.com/vorkash.

回到具体的预测，因为我认为有很多不同的东西需要澄清，所以在讨论能力时很容易各说各话。例如，三年前我采访你时，我问了一个关于三年后我们应该期待什么的预测。我认为你说对了。我们应该期待这样的系统：如果你和它们交谈一个小时，很难将它们与一个受过良好教育的普通人区分开来。是的，我认为你在这方面是对的。但我在精神上感到不满足，因为我内心的期望是，这样的系统可以自动化大部分白领工作。所以，讨论你希望这样的系统具备的实际最终能力可能会更有成效。因此，我将基本告诉你我认为我们现在处于什么位置。但让我问你一个非常具体的问题，这样我们就能确切地知道我们应该很快期待什么样的能力。也许我会在我熟悉的工作背景下提问，不是因为这是最相关的工作，而是因为我可以评估关于它的说法。以视频编辑为例，对吧？他们的部分工作涉及了解我们观众的偏好，了解我的偏好和品味以及我们面临的不同权衡，以及如何在几个月的时间里建立起这种对上下文的理解。所以，他们工作六个月后拥有的技能和能力，一个模型能否在工作中即时掌握这种技能。我们应该什么时候期待这样的人工智能系统？

Coming back to concrete predictions, because I think because there's so many different things to disambiguate, it can be easy to talk past each other when we're talking about capabilities. So, for example, when I interviewed you 3 years ago, I asked for a prediction about what we should expect 3 years from now. I think you were right in what you said. We should expect systems which, if you talk to them for the course of an hour, it's hard to tell them apart from a generally well-educated human. Yes. I think you were right about that. And I think spiritually I feel unsatisfied because my internal expectation was that such a system could automate large parts of white-collar work. And so, it might be more productive to talk about the actual end capabilities you want such a system to have. So, I will basically tell you where I think we are. But let me ask you a very specific question so that we can figure out exactly what kinds of capabilities we should expect soon. So, maybe I'll ask about it in the context of a job I understand well, not because it's the most relevant job, but just because I can evaluate the claims about it. Take video editors, right? Part of their job involves learning about our audience's preferences, learning about my preferences and taste and the different trade-offs we have, and how just over the course of many months building up this understanding of context. And so, the skill and ability they have 6 months into the job, a model that can pick up that skill on the job, on the fly. When should we expect such an AI system?

是的，所以我想你说的是，我们进行这个采访三个小时，然后有人会进来，有人会编辑它，他们会说，「哦，Dario 挠了挠头，我们可以剪掉」，「这段长讨论对人们来说不太有趣，但另一件事更有趣，所以我们来做这个剪辑。」所以，我认为数据中心里的天才国度将能够做到这一点。它能够做到这一点的方式是，它将能够一般性地控制电脑屏幕，对吧？它将能够输入这些内容，并且还能使用电脑屏幕上网，查看你所有的过往采访，查看人们在 Twitter 上对你采访的评论，与你交谈，问你问题，与你的员工交谈，查看你之前编辑的历史，然后完成工作。所以，我认为这取决于几件事。第一，这取决于在电脑使用方面达到模型真正精通使用电脑的程度，对吧？我们已经看到基准测试在上升，基准测试总是不完美的衡量标准，但像 OS world 从我们大约一年零三个月前首次发布电脑使用时的 5% 左右，上升到了现在的 65% 或 70%。可能还有更难的衡量标准，但我认为电脑使用必须通过一个可靠性门槛。

Yeah, so I guess what you're talking about is like, we're doing this interview for 3 hours, and then someone's going to come in, someone's going to edit it, they're going to be like, 'Oh, Dario scratched his head, we could edit that out,' and 'This long discussion is less interesting to people, but this other thing is more interesting, so let's make this edit.' So, I think the country of geniuses in a data center will be able to do that. The way it will be able to do that is, it will have general control of a computer screen, right? Like it will be able to feed this in, and it'll be able to also use the computer screen to go on the web, look at all your previous interviews, look at what people are saying on Twitter in response to your interviews, talk to you, ask you questions, talk to your staff, look at the history of edits you did, and from that do the job. So, I think that's dependent on several things. One, that's dependent on getting to the point on computer use where the models are really masters at using the computer, right? And we've seen this climb in benchmarks, and benchmarks are always imperfect measures, but like OS world went from like 5% when we first released computer use about a year and a quarter ago, to like 65 or 70% now. There may be harder measures as well, but I think computer use has to pass a point of reliability.

在你继续下一个点之前，我能问一个后续问题吗？多年来，我经常尝试为自己构建不同的内部 LLM 工具，我经常遇到这些文本输入文本输出的任务，这些任务本应是这些模型的核心能力，但我仍然雇佣人类来做，因为如果是像「识别这个转录中最好的片段」这样的任务，它们可能能完成七成的工作，但没有一种持续的方式让我与它们互动，帮助它们像对待人类员工那样在工作中变得更好。所以，即使你解决了电脑使用问题，这种缺失的能力仍然会阻碍我将实际工作外包给它们。

Can I just ask a follow-up on that before you move on to the next point? I often for years I've been trying to build different internal LLM tools for myself and I often have these text-in text-out tasks which should be dead center in the repertoire of these models and yet I still hire humans to do them just because it's if it's something like 'identify what the best clips would be in this transcript' and maybe they'll do like a seven out of 10 job at them but there's not this ongoing way I can engage with them to help them get better at the job the way I could with a human employee. And so that missing ability, even if you solve computer use, would still block my ability to offload an actual job to them.

这又回到了我们之前讨论的关于在工作中学习的问题。这非常有趣。我认为对于编码智能体来说，我不认为人们会说在工作中学习是阻止编码智能体端到端完成所有事情的原因。它们一直在变得更好。Anthropic 有一些工程师根本不写代码，当我看到生产力时，针对你之前的问题，有人会说「这个 GPU 内核，这个芯片，我以前自己写。现在我只让 Claude 做。」所以生产力有了巨大的提升。而且我不知道，当我看到 Claude 代码时，对代码库的熟悉程度或模型没有在公司工作一年的感觉，并不是我看到的投诉清单上的主要内容。所以我认为我想说的是，我们正在走一条不同的路。

Again, this gets back to what we were talking about before with learning on the job. It's very interesting. I think with the coding agents, I don't think people would say that learning on the job is what is preventing the coding agents from doing everything end to end. They keep getting better. We have engineers at Anthropic who don't write any code, and when I look at the productivity, to your previous question, we have folks who say 'This GPU kernel, this chip, I used to write it myself. I just have Claude do it.' And so there's this enormous improvement in productivity. And I don't know, when I see Claude code, familiarity with the code base or a feeling that the model hasn't worked at the company for a year is not high up on the list of complaints I see. So I think what I'm saying is we're kind of taking a different path.

你不觉得在编程领域，这是因为存在一个外部的记忆支架，它实例化在代码库中，我不知道其他多少工作有这种优势？编程之所以能快速进步，正是因为它有这个其他经济活动没有的独特优势。

Don't you think with coding that's because there is an external scaffold of memory which exists instantiated in the code base, which I don't know how many other jobs have? Coding made fast progress precisely because it has that unique advantage that other economic activity doesn't.

但你这么说，暗示的是通过把代码库读入上下文，我就拥有了人类在工作中需要学习的一切。所以这就是一个例子，不管是否写下来、是否可用，你需要知道的一切都从上下文窗口获得了，对吧？而我们通常认为的学习，比如「天哪，我刚开始这份工作，要花 6 个月才能理解代码库」，模型在上下文中就完成了。

But when you say that, what you're implying is that by reading the code base into the context, I have everything that the human needed to learn on the job. So that would be an example of whether it's written or not, whether it's available or not, a case where everything you needed to know, you got from the context window, right? And what we think of as learning, like 'Oh man, I started this job, it's going to take me 6 months to understand the code base,' the model just did it in the context.

是啊，我真的不知道该怎么想，因为确实有人定性地说你讲的情况。去年有一项元研究，你肯定看过，他们让有经验的开发者在他们熟悉的代码仓库里尝试合并拉取请求。那些开发者报告说有提升。他们觉得用了这些模型后效率更高了。但实际上，如果你看他们的产出和实际被合并的数量，效率反而下降了 20%。他们因为使用这些模型而变得更低效。所以我试图调和人们使用这些模型时的定性感受，与两方面的事实：一是宏观层面，软件复兴在哪里？二是当人们做这些独立评估时，为什么我们看不到你预期的生产力提升？

Yeah, I honestly don't know how to think about this because there are people who qualitatively report what you're saying. There was a meta-study, I'm sure you saw last year, where they had experienced developers try to close pull requests in repositories that they were familiar with. And those developers reported an uplift. They reported that they felt more productive with the use of these models. But in fact, if you look at their output and how much was actually merged back in, there's a 20% downlift. They were less productive as a result of using these models. And so I'm trying to square the qualitative feeling that people feel with these models versus one, on a macro level, where is this renaissance of software? And two, when people do these independent evaluations, why are we not seeing the productivity benefits that you would expect?

在 Anthropic 内部，这真的非常明确。我们承受着巨大的商业压力，而且因为做了很多安全方面的工作，我们让自己更难了，我认为我们比其他公司做得更多。所以既要经济上生存又要保持价值观，这种压力是巨大的。我们努力维持 10 倍收入增长的曲线。没有时间在实际上不高效的时候还觉得自己高效。这些工具确实让我们高效得多。你觉得我们为什么担心竞争对手使用这些工具？因为我们觉得自己领先于竞争对手。如果这些工具暗中降低我们的生产力，我们不会费这么大劲。我们每隔几个月就能通过模型发布看到最终的生产力。这没什么好自欺欺人的。这些模型确实让你更高效。

Within Anthropic, this is just really unambiguous. We're under an incredible amount of commercial pressure and make it even harder for ourselves because we have all this safety stuff we do that I think we do more than other companies. So the pressure to survive economically while also keeping our values is incredible. We're trying to keep this 10x revenue curve going. There is zero time for feeling like we're productive when we're not. These tools make us a lot more productive. Why do you think we're concerned about competitors using the tools? Because we think we're ahead of the competitors. We wouldn't be going through all this trouble if this was secretly reducing our productivity. We see the end productivity every few months in the form of model launches. There's no kidding yourself about this. The models make you more productive.

第一，人们觉得自己高效，这种定性感受确实被这类研究预测到了。但第二，如果只看最终产出，显然你们进步很快。但递归自我改进的想法是，你做出更好的 AI，AI 帮你构建更好的下一代 AI，等等。而我看到的却是，如果我看 OpenAI、DeepMind，人们每隔几个月就在领奖台上换位置。也许你觉得这会停止，因为你赢了之类的。但如果上一个编程模型真的带来了巨大的生产力提升，为什么我们没看到拥有最佳编程模型的人获得持久的优势？

One, that is people feeling like they're productive is qualitatively predicted by studies like this. But two, if I just look at the end output, obviously you guys are making fast progress. But the idea was supposed to be with recursive self-improvement is that you make a better AI, the AI helps you build a better next AI, etc. And what I see instead, if I look at OpenAI, DeepMind, is that people are just shifting around the podium every few months. And maybe you think that stops because you won or whatever. But why are we not seeing the person with the best coding model have this lasting advantage if in fact there are these enormous productivity gains from the last coding model?

不不不。我认为这一切……我对局势的模型是，优势在逐渐增长。我会说现在编程模型大概带来 15%或 20%的全要素速度提升。这是我的看法。而 6 个月前可能只有 5%，所以无关紧要。5%不算什么。现在才刚刚达到成为几个重要因素之一的程度。而且这还会继续加速。所以我认为 6 个月前有几家公司大致处于同一水平，因为这不是一个显著因素，但我认为它开始越来越快地加速。我还要说，有好几家公司都在编写用于代码的模型，而我们并不能很好地阻止其中一些公司内部使用我们的模型。所以我认为我们看到的一切都与这种雪球模型一致，没有硬性的……再说一遍，我在这所有事情上的主题是，这都是软起飞，平滑的指数增长，尽管指数相对陡峭。所以我们看到这个雪球在积聚动量，从 10%、20%、25%、40%这样增长。而且随着进展，是的，阿姆达尔定律，你必须清除所有阻碍你闭环的东西，但这是 Anthropic 最优先的事项之一。

No, no, no. I think it's all... my model of the situation is there's an advantage that's gradually growing. I would say right now the coding models give maybe a 15% or 20% total factor speedup. That's my view. And 6 months ago it was maybe 5%, so it didn't matter. 5% doesn't register. It's now just getting to the point where it's one of several factors that matters. And that's going to keep speeding up. So I think 6 months ago there were several companies that were at roughly the same point because this wasn't a notable factor, but I think it's starting to speed up more and more. I would also say there are multiple companies that write models used for code, and we're not perfectly good at preventing some of these other companies from using our models internally. So I think everything we're seeing is consistent with this kind of snowball model where there's no hard... Again, my theme in all of this is all of this is soft takeoff, soft smooth exponentials, although the exponentials are relatively steep. So we're seeing this snowball gather momentum where it's like 10%, 20%, 25%, 40%. And as you go, yeah, Amdahl's law, you have to get all the things that are preventing you from closing the loop out of the way, but this is one of the biggest priorities at Anthropic.

退一步说，我想之前我们在讨论什么时候能实现这种在职学习。而你在编程问题上似乎暗示我们其实不需要在职学习。没有这种基本的人类能力，也能有巨大的生产力提升，AI 公司可能获得数万亿美元的收入。也许这不是你的主张，你应该澄清一下。但如果没有这种在职学习的基本人类能力，我看大多数经济活动领域，人们会说，「我雇了个人，头几个月没什么用，然后随着时间的推移，他们建立了上下文理解。」其实很难定义我们在这里讨论的是什么。但他们得到了某种东西，然后现在他们成了主力，对我们非常有价值。如果 AI 没有发展出这种边干边学的能力，我有点怀疑没有这种能力我们能看到世界的巨大变化。

Stepping back, I think before in the stack we were talking about when do we get this on-the-job learning. And it seems like the point you were making in the coding thing is we actually don't need on-the-job learning. You can have tremendous productivity improvements, potentially trillions of dollars of revenue for AI companies without this basic human ability. Maybe that's not your claim, you should clarify. But without this basic human ability to learn on the job, I just look at most domains of economic activity, people say, 'I hired somebody, they weren't that useful for the first few months, and then over time they built up the context understanding.' It's actually hard to define what we're talking about here. But they got something, and then now they're a powerhouse, and they're so valuable to us. And if AI doesn't develop this ability to learn on the fly, I'm a bit skeptical that we're going to see huge changes to the world without that ability.

我认为这里有两件事。首先是当前技术的状态，再说一遍，我们有两个阶段。

I think two things here. There's the state of the technology right now, which again, we have these two stages.

我们有预训练和强化学习阶段，在这个阶段我们把大量数据和任务投入模型，然后它们进行泛化。这就像学习，但它是从更多数据中学习，而不是在一个人类或一个模型的一生中学习。所以这介于进化和人类学习之间。但一旦你学会了所有这些技能，你就拥有了它们。就像预训练一样，模型知道得更多。如果我看一个预训练模型，它比我更了解日本武士的历史。它比我更了解棒球。它更了解电子学中的低通滤波器。所有这些事情。它的知识比我的广泛得多。所以我认为仅凭这一点就可能让模型在一切事情上都做得更好。然后，通过扩展现有设置，我们还有上下文学习，我把它描述为像人类在职学习，但更弱一些、更短期一些。你给模型一堆例子，它就明白了。在上下文中会发生真正的学习，而一百万个词元是很多的。那可以是人类数天的学习。如果你考虑模型阅读一百万个词，那至少需要我几天或几周。所以你有这两样东西，在现有范式内它们可能就足以让你在数据中心里拥有一个国家的天才。我不确定，但我认为它们能让你实现很大一部分。可能会有差距，但我当然认为这足以产生数万亿美元的收入。

We have the pre-training and RL stage where you throw a bunch of data and tasks into the models and then they generalize. So it's like learning, but it's learning from more data and not over one human or one model's lifetime. So again, this is situated between evolution and human learning. But once you learn all those skills, you have them. And just like with pre-training, the models know more. If I look at a pre-trained model, it knows more about the history of samurai in Japan than I do. It knows more about baseball than I do. It knows more about low-pass filters in electronics. All of these things. Its knowledge is way broader than mine. So I think even just that may get us to the point where the models are better at everything. And then we also have, with scaling the existing setup, the in-context learning, which I would describe as like human on-the-job learning, but a little weaker and short-term. You give the model a bunch of examples, it gets it. There's real learning that happens in context, and a million tokens is a lot. That can be days of human learning. If you think about the model reading a million words, it takes me days or weeks at least. So you have these two things, and within the existing paradigm they may just be enough to get you the country's genius in the data center. I don't know for sure, but I think they'll get you a large fraction of it. There may be gaps, but I certainly think this is enough to generate trillions of dollars of revenue.

第二点是持续学习的概念，即单个模型在工作中学习。我认为我们也在研究这个，而且我认为在未来一两年内我们很可能解决这个问题。我认为没有它你也能走完大部分路。我认为每年数万亿美元的市场，也许还有我在《技术的青春期》中写到的所有国家安全影响和安全影响，没有它也能实现，但我也认为我们，而且我想其他人也在研究它。我认为我们很可能在未来一两年内实现。有很多想法。一个就是让上下文更长。没有什么能阻止更长的上下文工作。你只需要在更长的上下文上训练，然后学会在推理时服务它们。这两个都是工程问题，我们正在解决，我想其他人也在解决。

Two is this idea of continual learning, a single model learning on the job. I think we're working on that too, and I think there's a good chance that in the next year or two we also solve that. I think you get most of the way there without it. I think the trillions of dollars a year market, maybe all of the national security implications and the safety implications that I wrote about in 'Adolescence of Technology' can happen without it, but I also think we, and I imagine others, are working on it. And I think there's a good chance that we get there within the next year or two. There are a bunch of ideas. One is just make the context longer. There's nothing preventing longer context from working. You just have to train at longer context and then learn to serve them at inference. Both of those are engineering problems that we are working on and that I would assume others are working on as well.

所以这个上下文长度的增加。似乎从 2020 年到 2023 年，从 GPT-3 到 GPT-4 Turbo，上下文长度从大约 2000 增加到了 128K。你觉得从那之后的两年左右，我们一直处于差不多的水平吗？而且当模型上下文长度远超过那个时，人们报告说模型考虑完整上下文的能力出现了质量上的退化。所以我很好奇你内部看到了什么，让你觉得哦，1000 万上下文、1 亿上下文来实现类似人类 6 个月的学习，10 亿上下文。

So this context length increase. It seemed like there was a period from 2020 to 2023 where from GPT-3 to GPT-4 Turbo there was an increase from like 2,000 context length to 128K. Do you feel like for the two-ish years since then we've been in the same-ish ballpark? And when model context lengths get much longer than that, people report qualitative degradation in the ability of the model to consider that full context. So I'm curious what you're internally seeing that makes you think like oh, 10 million context, 100 million context to get human-like 6 months learning, billion context.

这不是一个研究问题。这是一个工程和推理问题。如果你想服务长上下文，你必须存储整个 KV 缓存，你必须来回移动内存。把所有内存存储在 GPU 里很困难。我现在甚至不知道细节。在 GPT-3 时代，我了解权重和激活，但现在有了 MoE 模型，一切都变了。而你提到的退化：有两件事。你训练的上下文长度和你服务的上下文长度。如果你在小的上下文长度上训练，然后试图在长的上下文长度上服务，你可能会遇到这些退化。这比没有好，但你会遇到退化。而且也许在长的上下文长度上训练更难。

This isn't a research problem. This is an engineering and inference problem. If you want to serve long context, you have to store your entire KV cache, you have to juggle the memory around. It's difficult to store all the memory in the GPUs. I don't even know the detail at this point. In the GPT-3 era I knew about weights and activations, but now with MoE models everything has flipped. And this degradation you're talking about: there's two things. There's the context length you train at, and the context length you serve at. If you train at a small context length and then try to serve at a long context length, you might get these degradations. It's better than nothing, but you get degradations. And maybe it's harder to train at a long context length.

我想得到一个更大图景问题的答案。好的，所以我觉得一个为我工作了 6 个月的人类编辑和一个与我合作了 6 个月的 AI 之间没有偏好。你预测哪一年会出现这种情况？

I want to get an answer to the bigger picture question. Okay, so I don't feel a preference for a human editor that's been working for me for 6 months versus an AI that's been working with me for 6 months. What year do you predict that that will be the case?

我的猜测是，很多问题基本上是这样的：当我们拥有数据中心里的天才国度时，我们就能做到这一点。所以我的图景是，如果你让我猜，那是 1 到 2 年，也许 1 到 3 年。真的很难说。我强烈认为 99%这一切会在 10 年内发生。那是一个非常安全的赌注。然后我有一个直觉，更像是 50/50，它会是一到两年，也许一到三年。所以一到三年。天才国度。以及在稍微经济价值低一些的视频编辑任务中。

My guess for that is there's a lot of problems that are basically like, we can do this when we have the country of geniuses in a data center. So my picture for that is, if you made me guess, it's 1 to 2 years, maybe 1 to 3 years. It's really hard to tell. I have a strong view 99% that all this will happen in 10 years. That's a super safe bet. And then I have a hunch, more like a 50/50 thing, that it's going to be more like one to two, maybe more like one to three. So one to three years. The country of geniuses. And in the slightly less economically valuable task of editing videos.

这看起来经济价值巨大，我跟你说。有很多这样的用例，对吧？所以，你预测在一到三年内，而且总体上，Anthropic 预测到 2026 年底、2027 年初，我们将拥有能够像当今人类一样操作数字界面、智力达到或超过诺贝尔奖得主水平、并能与物理世界交互的 AI 系统。然后你两个月前接受 DealBook 采访时，强调你们公司相比竞争对手更负责任地控制算力规模。我试图调和这两种观点。如果你真的相信我们将拥有一个天才之国，那你应该尽可能建最大的数据中心，没有理由放慢。一个能完成诺贝尔奖得主所有工作的诺贝尔奖得主，其 TAM 高达数万亿美元。所以，我试图理解这种保守主义——如果你对时间线更温和，这似乎合理——与你公开表达的 AI 进展观点之间的一致性。

It seems pretty economically valuable, let me tell you. There are a lot of use cases like that, right? So, you're predicting that within one to three years, and generally, Anthropic has predicted that by late '26, early '27, we will have AI systems that have the ability to navigate interfaces available to humans doing digital work today, intellectual capabilities matching or exceeding that of Nobel Prize winners, and the ability to interface with the physical world. And then you gave an interview two months ago with DealBook, where you were emphasizing your company's more responsible compute scaling as compared to your competitors. I'm trying to square these two views. If you really believe that we're going to have a country of geniuses, you want as big a data center as you can get. There's no reason to slow down. The TAM of a Nobel Prize winner that can do everything a Nobel Prize winner can do is like trillions of dollars. So, I'm trying to square this conservatism, which seems rational if you have more moderate timelines, with your stated views about AI progress.

是的，实际上这一切都是协调一致的。我们回到那个观点：扩散很快，但并非无限快。假设我们以这个速度取得进展。技术正在以这个速度进步。我非常有信心我们会在几年内达到目标。我直觉上觉得我们会在一年或两年内达到。所以技术方面有些不确定性，但我相当确信不会偏差太多。我比较不确定的是经济扩散方面。我确实相信我们可以在未来一到两年内拥有一个天才之国，甚至一百个天才之国在数据中心里的模型。问题是，在那之后多少年，数万亿美元的收入才会开始涌入？我不认为这会是立竿见影的。我认为可能是一年，可能是两年，甚至可能拖到五年，尽管我对五年持怀疑态度。所以我们有这种不确定性：即使技术像我怀疑的那样快速发展，我们也不确定它驱动收入的速度有多快。我们知道它会来，但考虑到你购买这些数据中心的方式，如果偏差几年，那可能是毁灭性的。就像我在《Machines of Loving Grace》中写的：「看，我认为我们可能会在数据中心里拥有这个强大的天才之国 AI。」你引用的描述就来自《Machines of Loving Grace》。我说：「我们会在 2026 年，也许 2027 年实现。」这又是我的直觉。如果偏差一两年，我不会惊讶，但这就是我的直觉。假设这发生了。那是发令枪。治愈所有疾病需要多长时间？这是驱动巨大经济价值的方式之一。你治愈每一种疾病。问题是有多少价值归于制药公司，多少归于 AI 公司，但会有巨大的消费者剩余，因为每个人都能获得，这我非常关心。我们治愈所有这些疾病。需要多长时间？你必须进行生物学发现，你必须制造新药，你必须通过监管流程。我们在疫苗和 COVID 上看到了这一点。我们让每个人都能接种疫苗，但花了一年半。所以我的问题是，要得到治愈一切的方法需要多长时间？AI 是天才，理论上可以为每个人发明出来。从 AI 首次在实验室存在到疾病真正被治愈，需要多长时间？我们有小儿麻痹症疫苗已经 50 年了。我们仍在努力在非洲最偏远的角落根除它。盖茨基金会正在尽最大努力，但这很困难。我不期望大多数经济扩散会像那样困难，但这里有一个真正的困境。我最终得出的结论是：它会比世界上任何事物都快，但仍然有其极限。所以当我们去购买数据中心时，我看到的曲线是：我们每年都有 10 倍的增长。所以今年年初，我们年初的年化收入是 100 亿。我们必须决定购买多少算力。实际建造数据中心、预订数据中心需要一两年时间。所以基本上我在说，到 2027 年，我能得到多少算力？我可以假设收入将继续每年增长 10 倍。所以到 2026 年底是 1000 亿，到 2027 年底是 1 万亿。所以我可以购买 1 万亿美元的算力。实际上，那会是 5 万亿美元的算力，因为那是每年 1 万亿，持续 5 年。我可以购买从 2027 年底开始的 1 万亿美元算力。如果我的收入不是 1 万亿，即使只有 8000 亿，地球上没有任何力量、没有任何对冲能阻止我破产，如果我买了那么多算力。所以即使我大脑的一部分在想它是否会继续增长 10 倍，我也不能每年购买 1 万亿美元的算力，如果我仅仅在增长率上偏差一年，或者如果增长率是每年 5 倍而不是 10 倍，那么我就会破产。

Yeah, so it actually all fits together. We go back to this fast, but not infinitely fast diffusion. Let's say that we're making progress at this rate. The technology is making progress this fast. Again, I have very high conviction that we're going to get there within a few years. I have a hunch that we're going to get there within a year or two. So a little uncertainty on the technical side, but pretty strong confidence that it won't be off by much. What I'm less certain about is the economic diffusion side. I really do believe that we could have models that are a country of geniuses, a hundred country of geniuses in a data center in one to two years. One question is, how many years after that do the trillions in revenue start rolling in? I don't think it's guaranteed that it's going to be immediate. I think it could be one year, it could be two years, I could even stretch it to five years, although I'm skeptical of that. So we have this uncertainty: even if the technology goes as fast as I suspect, we don't know exactly how fast it's going to drive revenue. We know it's coming, but with the way you buy these data centers, if you're off by a couple years, that can be ruinous. It is just like how I wrote in Machines of Loving Grace: 'Look, I think we might get this powerful AI of this country of geniuses in the data center.' That description you gave comes from Machines of Loving Grace. I said, 'We'll get that 2026, maybe 2027.' Again, that is my hunch. Wouldn't be surprised if I'm off by a year or two, but that is my hunch. Let's say that happens. That's the starting gun. How long does it take to cure all the diseases? That's one of the ways that drives a huge amount of economic value. You cure every disease. There's a question of how much of that goes to the pharmaceutical company, to the AI company, but there's an enormous consumer surplus because everyone gets access, which I care about greatly. We cure all of these diseases. How long does it take? You have to do the biological discovery, you have to manufacture the new drug, you have to go through the regulatory process. We saw this with vaccines and COVID. We got the vaccine out to everyone, but it took a year and a half. So my question is how long does it take to get the cure for everything, which AI is the genius that can in theory invent for everyone. How long from when that AI first exists in the lab to when diseases have actually been cured for everyone? We've had a polio vaccine for 50 years. We're still trying to eradicate it in the most remote corners of Africa. The Gates Foundation is trying as hard as they can, but that's difficult. I don't expect most of the economic diffusion to be as difficult as that, but there's a real dilemma here. Where I've settled on it is it will be faster than anything we've seen in the world, but it still has its limits. So when we go to buying data centers, the curve I'm looking at is okay, we've had a 10x a year increase every year. So beginning of this year, we're looking at 10 billion in annualized revenue at the beginning of the year. We have to decide how much compute to buy. It takes a year or two to actually build out the data centers, to reserve the data centers. So basically I'm saying in 2027, how much compute do I get? Well, I could assume that the revenue will continue growing 10x a year. So it'll be 100 billion at the end of 2026 and 1 trillion at the end of 2027. So I could buy a trillion dollars of compute. Actually it would be like 5 trillion dollars of compute because it would be a trillion dollars a year for 5 years. I could buy a trillion dollars of compute that starts at the end of 2027. If my revenue is not a trillion dollars, if it's even 800 billion, there's no force on Earth, no hedge on Earth that could stop me from going bankrupt if I buy that much compute. So even though a part of my brain wonders if it's going to keep growing 10x, I can't buy a trillion dollars a year of compute if I'm just off by a year in that rate of growth or if the growth rate is 5x a year instead of 10x a year, then I go bankrupt.

所以，你最终会处于一个支持数千亿而非数万亿的世界，你接受一些风险，即需求太大以至于收入无法支撑，你也接受一些风险，即你判断错误，进展仍然缓慢。当我谈到负责任的行为时，我指的不是绝对数额。我认为我们确实比一些其他玩家花费得少一些。实际上是其他方面，比如我们是否深思熟虑，还是我们只是在「YOLO」，说「哦，我们要在这里花一千亿，在那里花一千亿？」我有点感觉，一些其他公司没有写下电子表格，他们并不真正理解自己承担的风险。他们只是在做事，因为听起来很酷。我们仔细考虑过。我们是一家企业级公司。因此，我们可以更多地依赖收入。它比消费者业务更稳定。我们有更好的利润率，这是购买过多和过少之间的缓冲。我认为我们购买的数量让我们能够捕捉相当强劲的上行世界。它不会捕捉到每年 10 倍的增长。事情必须变得非常糟糕，我们才会陷入财务困境。所以，我认为我们仔细考虑过，并取得了平衡，这就是我所说的负责任。

So, you end up in a world where you're supporting hundreds of billions, not trillions, and you accept some risk that there's so much demand that you can't support the revenue, and you accept still some risk that you got it wrong and it's still slow. When I talked about behaving responsibly, what I meant was not the absolute amount. I think it is true we're spending somewhat less than some of the other players. It's actually the other things like have we been thoughtful about it or are we YOLOing and saying, 'Oh, we're going to do a hundred billion dollars here or a hundred billion dollars there?' I kind of get the impression that some of the other companies have not written down the spreadsheet, that they don't really understand the risks they're taking. They're just kind of doing stuff because it sounds cool. We thought carefully about it. We're an enterprise business. Therefore, we can rely more on revenue. It's less fickle than consumer. We have better margins, which is the buffer between buying too much and buying too little. I think we bought an amount that allows us to capture pretty strong upside worlds. It won't capture the full 10x a year. Things would have to go pretty badly for us to be in financial trouble. So, I think we thought carefully and we've made that balance, and that's what I mean when I say that we're being responsible.

好吧，看起来我们可能对数据中心里的「天才之国」有不同的定义。因为当我想到真正的人类天才和真正的数据中心里的人类天才之国时，我会很乐意花 5 万亿美元购买算力来运行一个数据中心里的人类天才之国。所以，假设摩根大通或莫德纳之类的不想用他们。而且，我有了一个天才之国。他们会创办自己的公司。如果他们不能创办自己的公司，并且受到临床试验的瓶颈，值得指出的是，大多数临床试验失败是因为药物无效。没有疗效。

Okay, so it seems like it's possible that we actually just have different definitions of a country of geniuses in a data center. Because when I think of actual human geniuses and actual country of human geniuses in a data center, I would happily buy 5 trillion dollars worth of compute to run an actual country of human geniuses in a data center. So, let's say JP Morgan or Moderna or whatever doesn't want to use them. Also, I've got a country of geniuses. They'll start their own company. And if they can't start their own company and they're bottlenecked by clinical trials, it is worth stating that most clinical trials fail because the drug doesn't work. There's no efficacy.

正是如此。在《爱与优雅的机器》中，我说临床试验会比我们习惯的快得多。但不是瞬间，不是无限快。然后假设临床试验需要一年时间才能完成，这样你就能从中获得收入，并制造更多药物。好吧，你有了一个天才之国，你在 AI 实验室里，你可以使用更多的 AI 研究人员。你也认为，聪明人从事 AI 技术会带来自我强化的收益。

Exactly. In Machines of Love and Grace, I say the clinical trials are going to go much faster than we're used to. But not instant, not infinitely fast. And then suppose it takes a year for the clinical trials to work out so that you're getting revenue from that and can make more drugs. Okay, well, you've got a country of geniuses and you're in the AI lab and you could use many more AI researchers. You also think that there are self-reinforcing gains from smart people working on AI tech.

没错。你可以让数据中心致力于 AI 进步。每年购买一万亿美元算力与每年购买三千亿美元算力相比，收益会大得多吗？

That's right. You can have the data center working on AI progress. Are there more gains from buying substantially more gains from buying a trillion dollars a year of compute versus 300 billion dollars a year of compute?

购买一万亿，是的，有收益。嗯，不，有一些收益，但同样，有可能他们在那之前就破产了，如果你只差一年，你就会毁了自己。这就是平衡。我们买了很多。我们买了非常多。我们购买的数量与游戏中最大的玩家相当。但如果你问我为什么我们没有从 2027 年中开始签署 10 万亿的算力，首先，它生产不出来。世界上没有那么多。其次，如果天才之国来了，但它在 2028 年中而不是 2027 年中到来呢？你会破产。所以，如果你的预测是 1 到 3 年，那么到 2029 年你应该拥有 10 万亿美元的算力。

Buying a trillion, yes, there is. Well, no, there's some gain, but then again, there's this chance that they go bankrupt before, if you're off by only a year, you destroy yourselves. That's the balance. We're buying a lot. We're buying a hell of a lot. We're buying an amount that's comparable to the biggest players in the game. But if you're asking me why haven't we signed 10 trillion of compute starting in mid 2027, first of all, it can't be produced. There isn't that much in the world. But second, what if the country of geniuses comes, but it comes in mid 2028 instead of mid 2027? You go bankrupt. So, if your projection is 1 to 3 years, it seems like you should have 10 trillion dollars of compute by 2029.

但你感兴趣吗？似乎即使在你所说的最长时间线中，你正在加速建设的算力似乎并不……是什么让你这么认为？嗯，正如你所说，你会想要 10 万亿，比如人类工资，假设每年大约 50 万亿。

But are you interested in? It seems like even in your longest version of the timelines you stated, the compute you are ramping up to build doesn't seem... What makes you think that? Well, you as you said, you would want the 10 trillion like human wages, let's say, are on the order of 50 trillion a year.

如果你看看这个行业，今年行业建设的算力可能只有几十吉瓦。明年，它每年大约增长 3 倍，所以明年是 30 或 40 吉瓦，2028 年可能是 100 吉瓦，2029 年可能是 300 吉瓦。每吉瓦每年可能花费 100 到 150 亿美元。所以，把这些加起来，你得到的大约就是你描述的。到 2028 年或 2029 年，你每年会得到数万亿美元。所以，你得到的正是这个。这是整个行业的情况。

If you look at the industry, the amount of compute the industry is building this year is probably in the very low tens of gigawatts. Next year, it goes up by roughly 3x a year, so next year's 30 or 40 gigawatts, and 2028 might be 100, 2029 might be like 300 gigawatts. Each gigawatt costs maybe 10 to 15 billion dollars a year. So, you put that all together, and you're getting about what you described. You're getting multiple trillions a year by 2028 or 2029. So, you're getting exactly that. That's for the industry.

那是整个行业的情况。所以，假设 Anthropic 的算力每年增长 3 倍，到 2027 年或 2028 年你有 10 吉瓦。乘以 100 亿，那就是每年 1000 亿。但你说到 2028 年、2029 年的 TAM……

That's for the industry. So, suppose Anthropic's compute keeps 3x a year, and then by 2027 or 2028 you have 10 gigawatts. Multiply that by 10 billion, so it's like 100 billion a year. But then you're saying the TAM by 2028, 2029...

我不想给出 Anthropic 的具体数字，但这些数字太小了。这些数字太小了。

I don't want to give exact numbers for Anthropic, but these numbers are too small. These numbers are too small.

好的，有意思。我很自豪，我和 Jane Street 一起做的谜题让他们为我的观众招聘了一批人。嗯，他们还在招聘，他们刚给我发了另一个谜题。这个谜题，他们花了大约 2 万 GPU 小时在三个不同的语言模型中训练后门。每个模型都有一个隐藏的提示，会引发完全不同的行为。你只需要找到触发器。这特别酷，因为发现后门实际上是前沿 AI 研究中的一个开放问题。

Okay, interesting. I'm really proud that the puzzles I've worked on with Jane Street have resulted in them hiring a bunch of people for my audience. Well, they're still hiring and they just sent me another puzzle. For this one, they spent about 20,000 GPU hours training backdoors into three different language models. Each one has a hidden prompt that elicits completely different behavior. You just have to find the trigger. This is particularly cool because finding backdoors is actually an open question in frontier AI research.

你告诉投资者你们计划从 2028 年开始盈利。而今年我们可能要为那个「天才之国」建一个数据中心。这将解锁医学、健康和新科技等方面的所有进步。这不正是你想要再投资于业务、建设更大规模的「国家」以促成更多发现的时候吗？

You've told investors that you plan to be profitable starting in 2028. And this is the year where we're potentially getting the country of geniuses a data center. And this is going to unlock all this progress in medicine, health, new technologies. Wouldn't this be exactly the time where you'd want to reinvest in the business and build bigger countries so they can make more discoveries?

在这个领域，盈利能力是一件很奇怪的事情。我不认为盈利能力实际上是衡量支出减少与业务投资的指标。我们用一个模型来说明。我实际上认为，当你低估了即将获得的需求时，你会盈利；而当你高估了需求时，你会亏损，因为你是提前购买数据中心的。理想情况下——再说一遍，这些都是简化的数据——假设你一半的算力用于训练，一半用于推理，而推理的毛利率超过 50%。那么在稳态下，如果你确切知道需求，你会获得一定收入。比方说，你每年为算力支付 1000 亿美元。用 500 亿美元支撑 1500 亿美元的收入，另外 500 亿美元用于训练。所以基本上你盈利了，赚了 500 亿美元。这就是未来一两年这个行业的经济学。唯一让情况不同的情况是，如果需求低于 500 亿美元，那么你超过 50%的数据中心用于研究，你就不盈利。你训练更强的模型，但不盈利。如果需求超出预期，那么你的研究被压缩，但你能支持更多推理，从而更盈利。所以你先决定算力总量，然后对推理与训练有一个目标期望，但这由需求决定，而不是由你决定。

Profitability is a kind of weird thing in this field. I don't think profitability is actually a measure of spending down versus investing in the business. Let's take a model of this. I actually think profitability happens when you underestimated the amount of demand you were going to get, and loss happens when you overestimated the amount of demand you were going to get because you're buying the data centers ahead of time. Ideally, and again these are stylized facts, let's say half of your compute is for training and half is for inference, and inference has some gross margin that's more than 50%. So in steady state, if you knew exactly the demand, you'd get a certain amount of revenue. Let's say you pay a hundred billion dollars a year for compute. On 50 billion dollars a year you support 150 billion dollars of revenue, and the other 50 billion are used for training. So basically you're profitable, you make 50 billion dollars of profit. Those are the economics of the industry projecting forward in a year or two. The only thing that makes that not the case is if you get less demand than 50 billion, then you have more than 50% of your data center for research and you're not profitable. You train stronger models but you're not profitable. If you get more demand than you thought, then your research gets squeezed but you're able to support more inference and you're more profitable. So you decide the amount of compute first, and then you have some target desire of inference versus training, but that gets determined by demand, not by you.

我听到的是，你预测盈利的原因是你系统性地在算力上投资不足，对吗？

What I'm hearing is the reason you're predicting profit is that you are systematically underinvesting in compute, right?

不。我是说这很难预测。所以关于 2028 年以及何时会发生这些事情，是我们尽力向投资者做出的预测。所有这些都非常不确定，因为存在不确定性锥。如果收入增长足够快，我们可能在 2026 年就盈利。然后如果我们高估或低估了下一年的需求，情况可能会剧烈波动。我想说的是，你脑子里有一个模型：业务投资、投资、投资、获得规模，然后变得盈利。有一个转折点。我不认为这个行业的经济学是这样运作的。

No. I'm saying it's hard to predict. So these things about 2028 and when it will happen, that's our attempt to do the best we can with investors. All of this stuff is really uncertain because of the cone of uncertainty. We could be profitable in 2026 if the revenue grows fast enough. And then if we overestimate or underestimate the next year, that could swing wildly. What I'm trying to get at is you have a model in your head of the business invests, invests, invests, gets scale, and then becomes profitable. There's a single point at which things turn around. I don't think the economics of this industry work that way.

我明白了。所以如果我理解正确，你是说由于我们本应获得的算力与实际获得的算力之间的差距，我们被迫盈利，但这并不意味着我们会继续盈利。我们会把钱再投资，因为现在 AI 取得了如此大的进展，我们想要更大的「天才之国」。所以收入很高，但亏损也很高。

I see. So if I'm understanding correctly, you're saying because of the discrepancy between the amount of compute we should have gotten and the amount of compute we got, we were sort of forced to make profit, but that doesn't mean we're going to continue making profit. We're going to reinvest the money because now AI's made so much progress and we want the bigger country of geniuses. And so then back into revenue's high, but losses are also high.

如果每年我们都能准确预测需求，我们每年都会盈利。因为大约 50%的算力用于研究，加上高于 50%的毛利率，以及正确的需求预测，就会带来利润。这就是我认为存在的盈利商业模式，但被提前建设和预测误差所掩盖。

If every year we predict exactly what the demand is going to be, we'll be profitable every year. Because spending 50% of your compute on research, roughly, plus a gross margin that's higher than 50%, and correct demand prediction leads to profit. That's the profitable business model that I think is kind of there, but obscured by building ahead and prediction errors.

我想你把 50%当作一个给定的常数。而实际上，如果 AI 进展很快，你可以通过扩大规模来加速进展，那么你就会有超过 50%的算力用于研究，从而不盈利。

I guess you're treating the 50% as a given constant. Whereas in fact, if AI progress is fast and you can increase the progress by scaling up more, you just have more than 50% and not make progress.

我要说的是，你可能想进一步扩大规模。但记住收益递减规律，对吧？如果 70%的算力只能让模型规模缩小 1.4 倍，那么额外的 200 亿美元对你来说价值就小得多，因为这是线性关系。所以你可能会发现，把这 200 亿美元投资于服务推理或雇佣更优秀的工程师更好。所以我说的 50%——那并不完全是我们的目标。它不会正好是 50%，可能会随时间变化。

What I'll say is you might want to scale it up more. But remember the law of diminishing returns, right? If 70% would get you a very little bit of a smaller model through a factor of 1.4x, that extra 20 billion dollars is worth much less to you because of the linear setup. So you might find that it's better to invest that 20 billion dollars in serving inference or in hiring engineers who are better at what they're doing. So the reason I said 50% — that's not exactly our target. It's not exactly going to be 50%. It'll probably vary over time.

我的意思是，线性回报定律导致你只把业务的一小部分用于研发，对吧？不是 5%，也不是 95%，然后你会因为定律而收益递减。就像说服 Dario 相信 AI 进步之类的。但你不投资研究是因为收益递减，而你投资你提到的其他东西。再次强调，我们讨论的是每年花费 500 亿美元之后的收益递减。对吧？我相信你会提出这一点，但天才的收益递减可能相当高。更一般地说，市场经济中的利润是什么？利润基本上是说市场上的其他公司可以用这笔钱做更多我做不到的事情。

What I'm saying is the law of linear returns leads to you spending on the order of one fraction of the business, right? Not 5%, not 95%, and then you get diminishing returns because of the law. Like convincing Dario to believe in AI progress or something. But you don't invest in research because it has diminishing returns, but you invest in the other things you mentioned. Again, we're talking about diminishing returns after you're spending $50 billion a year. Right? This is a point I'm sure you'd make, but diminishing returns on a genius could be quite high. And more generally, what is profit in a market economy? Profit is basically saying the other companies in the market can do more things with this money that I can't.

我只是想，因为我不想透露 Anthropic 的信息，所以我用这些简化数字。但让我们推导一下行业的均衡。为什么不是每个人都把 100%的算力用于训练而不服务任何客户？因为如果没有收入，他们就无法筹集资金，无法进行算力交易，也无法在明年购买更多算力。所以会有一个均衡，每个公司花费在训练上的比例低于 100%，当然推理上的比例也低于 100%。为什么不能只服务当前模型而永远不训练新模型，这应该很清楚，因为那样你就会落后，没有需求。所以存在某个均衡。不会是 10%，也不会是 90%。我们就说一个简化事实是 50%。这就是我的意思。而且我认为我们会处于这样一个位置：训练花费的均衡比例低于你能从算力中获得的毛利率。所以基本的经济学是盈利的。问题在于，当你购买下一年的算力时，你面临一个可怕的需求预测问题：你可能猜少了，非常盈利但没有算力用于研究；或者猜多了，不盈利但拥有全世界所有的研究算力。这说得通吗？这只是行业的动态模型。

I'm just trying to, because I don't want to give information about Anthropic, I'm giving these stylized numbers. But let's just derive the equilibrium of the industry. So why doesn't everyone spend 100% of their compute on training and not serve any customers? It's because if they didn't get any revenue, they couldn't raise money, they couldn't do compute deals, they couldn't buy more compute the next year. So there's going to be an equilibrium where every company spends less than 100% on training and certainly less than 100% on inference. It should be clear why you don't just serve the current models and never train another model, because then you don't have any demand because you'll fall behind. So there's some equilibrium. It's not going to be 10%, it's not going to be 90%. Let's just say as a stylized fact it's 50%. That's what I'm getting at. And I think we're going to be in a position where that equilibrium of how much you spend on training is less than the gross margins that you're able to get on compute. And so the underlying economics are profitable. The problem is you have this hellish demand prediction problem when you're buying the next year of compute, and you might guess under and be very profitable but have no compute for research, or you might guess over and you are not profitable and you have all the compute for research in the world. Does that make sense? It's just a dynamic model of the industry.

也许退一步说，我并不是说我认为天才之国会在两年内到来，因此你应该购买这些算力。对我来说，你所说的结论很有道理，但那是因为天才之国似乎很难实现，还有很长的路要走。所以退一步，我想说的是，你的世界观似乎与那些认为我们距离创造数万亿美元价值的世界还有 10 年的人是一致的。

Maybe stepping back, I'm not saying I think the country of geniuses is going to come in two years and therefore you should buy this compute. To me, what you're saying, the conclusion you're arriving at makes a lot of sense, but that's because it seems like country of geniuses is hard and there's a long way to go. So stepping back, the thing I'm trying to get at is more like it seems like your worldview is compatible with somebody who says we're 10 years away from a world in which we're generating trillions of dollars worth.

那根本不是我的观点。那不是我的观点。我再做一个预测。我很难想象在 2030 年之前不会有数万亿美元的收入。我可以构建一个合理的世界。可能需要 3 年。所以那将是我认为合理的终点：2028 年我们在数据中心里拥有了真正的天才之国。到 2028 年，收入将达到数千亿美元。然后天才之国将其加速到数万亿美元。我们基本上处于扩散的慢端。需要 2 年时间才能达到数万亿美元。那将是直到 2030 年才达到的世界。我怀疑即使结合技术指数和扩散指数，也会在 2030 年之前达到。所以你提出了一个模型，Anthropic 能盈利，因为从根本上我们处于一个算力受限的世界。所以最终我们会不断增长算力。

That's just not my view. That is not my view. I'll make another prediction. It is hard for me to see that there won't be trillions of dollars in revenue before 2030. I can construct a plausible world. It takes maybe 3 years. So that would be the end of what I think is plausible: in 2028 we get the real country of geniuses in the data center. The revenue's been going into the low hundreds of billions by 2028. And then the country of geniuses accelerates it to trillions. And we're basically on the slow end of diffusion. It takes 2 years to get to the trillions. That would be the world where it takes until 2030. I suspect even composing the technical exponential and diffusion exponential will get there before 2030. So you laid out a model where Anthropic makes profit because it seems like fundamentally we're in a compute-constrained world. And so it's like eventually we keep growing compute.

不，我认为利润的产生方式是，让我们抽象整个行业。想象我们在一本经济学教科书中。我们有少数几家公司。每家都可以在研发上投入有限的数量。它们有一些边际服务成本。这些边际成本上的利润率、毛利率非常高，因为推理是高效的。存在一些竞争，但模型也是差异化的。有些公司会竞争提高研发预算，但由于参与者数量少，我们得到了所谓的古诺均衡？我认为那就是少数公司均衡。关键是它不会均衡到完全竞争下的零利润率。如果经济中有三家公司，都独立理性行事，它不会均衡到零。

No, I think the way the profit comes is again, let's abstract the whole industry. Let's just imagine we're in an economics textbook. We have a small number of firms. Each can invest a limited amount in R&D. They have some marginal cost to serve. The margins on that, the profit margin, the gross profit margins on that marginal cost are very high because inference is efficient. There's some competition, but the models are also differentiated. There's some companies will compete to push their research budgets up, but because there's a small number of players, we have what is it called the Cournot equilibrium? I think that's the small number of firm equilibrium. The point is it doesn't equilibrate to perfect competition with zero margins. If there are three firms in the economy, all independently behaving rationally, it doesn't equilibrate to zero.

帮我理解一下，因为现在我们确实有三家领先公司，它们并没有盈利。那么是什么在改变？

Help me understand that because right now we do have three leading firms and they're not making profit. So what is changing?

是的，所以现在的毛利率非常正。发生的情况是两件事的结合。一是我们仍处于算力的指数级扩张阶段。所以基本上这意味着我们正在训练一个模型。假设去年训练一个模型花费了 10 亿美元。然后今年它产生了 40 亿美元的收入，推理成本为 10 亿美元。所以再次，我用的是简化数字，但那将是 75%的毛利率和 25%的税。

Yeah, so the gross margins right now are very positive. What's happening is a combination of two things. One is we're still in the exponential scale up phase of compute. So basically what that means is we're training a model. It costs, let's say a model got trained that cost a billion dollars last year. And then this year it produced $4 billion of revenue and cost $1 billion to inference from. So again, I'm using stylized numbers, but that would be 75% gross margins and this 25% tax.

所以整个模型能赚 20 亿美元。但与此同时，我们正花费 100 亿美元来训练下一个模型，因为规模在指数级扩张。因此公司是亏损的。每个模型都赚钱，但公司整体亏损。我所说的均衡是指数据中心里有一整个「天才之国」，但模型训练的规模扩张已经更加平稳。也许它仍在增长，我们仍在试图预测需求，但已经更加趋于稳定。

So that model as a whole makes $2 billion. But at the same time, we're spending $10 billion to train the next model because there's an exponential scale up. And so the company loses money. Each model makes money, but the company loses money. The equilibrium I'm talking about is an equilibrium where we have the country of geniuses in the data center, but that model training scale up has equilibrated more. Maybe it's still going up. We're still trying to predict the demand, but it's more leveled out.

我们先从当前世界说起。在当前世界，你说得对，如果每个模型单独看作一家公司，它是盈利的。当然。作为前沿实验室，生产函数的一个重要部分就是训练下一个模型，对吧？如果你不这么做，你能盈利两个月，然后就没有利润了，因为你不再拥有最好的模型。所以你在当前系统上只能盈利两个月。

Let's start with the current world. In the current world, you're right that as you said before, if you treat each individual model as a company, it's profitable. Of course. A big part of the production function of being a frontier lab is training the next model, right? So if you didn't do that, then you'd make profit for 2 months, and then you wouldn't have margins because you wouldn't have the best model. So you can make profit for 2 months on the current system.

达到它能达到的最大规模。然后在均衡状态下，我们有算法进步，但训练下一个模型的成本大致与训练当前模型相同。所以这个均衡依赖于——在某个点上，经济中的资金会耗尽。固定劳动力总量？经济会增长，对吧？这是你的预测之一。

Point that reaches the biggest scale it can reach. And then in equilibrium, we have algorithmic but we're spending roughly the same amount to train the next model as we spent to train the current model. So this equilibrium relies on—at some point you run out of money in the economy. Fixed lump of labor? The economy is going to grow, right? That's one of your predictions.

嗯，是的。但这又是我之前谈到的主题的一个例子：有了 AI，经济增长会比以往任何时候都快得多，但并不是说现在算力每年增长 3 倍。我不相信经济会每年增长 300%。就像我在《Machines of Loving Grace》中说的，我认为我们可能实现一年 10%或 20%的增长，但不会达到 300%。所以我认为最终，如果算力成为经济产出的主要部分，它就会受到这个限制。

Well, yes. But this is another example of the theme I was talking about, which is that the economy will grow much faster with AI than I think it ever has before, but it's not like right now the computer is growing 3x a year. I don't believe the economy is going to grow 300% a year. Like I said in Machines of Loving Grace, I think we may get 10 or 20% growth for a year, but we're not going to get 300% growth. So I think in the end, if compute becomes the majority of what the economy produces, it's going to be capped by that.

好，现在假设一个算力受限的模型。前沿实验室能赚钱的世界是它们持续快速进步的世界，因为从根本上说，你的利润空间取决于替代方案有多好。所以你能赚钱是因为你拥有前沿模型。如果你没有前沿模型，你就赚不到钱。

So, okay, now let's assume a model where compute stays capped. The world where frontier labs are making money is one where they continue to make fast progress because fundamentally your margin is limited by how good the alternative is. So you are able to make money because you have a frontier model. If you didn't have a frontier model, you wouldn't be making money.

所以这个模型要求永远没有稳态。永远永远，你不断取得更多的算法进步。

And so this model requires there never to be a steady state. Forever and ever, you keep making more algorithmic progress.

我认为确实如此。我感觉我们就像在上经济学课。

I think that's true. I feel like we're talking like an economics class.

泰勒·考恩密码？我们永远在谈经济学。

Tyler Cowen code? We never stop talking about economics.

我们永远在谈经济学。所以，不，但有些世界——我不认为这个领域会成为垄断。我的律师们从不让我说「垄断」这个词。但我不认为这个领域会成为垄断，但确实有些行业只有少数玩家，不是一家而是少数。通常像 Facebook 或 Meta 这样的垄断是通过网络效应形成的。而只有少数玩家的行业则是由于极高的进入成本。云计算就是这样。我认为云计算是一个很好的例子。云计算领域有三到四家玩家。我认为 AI 也一样，三到四家。原因是它非常昂贵，需要大量的专业知识和资本来运营一家云公司。你必须投入所有这些资本，此外，你还需要掌握许多其他技能才能实现。所以如果你对某人说，「我想颠覆这个行业，这里有 1000 亿美元」，他们会说，「好吧，我投入 1000 亿美元，还要赌你能做到这些公司一直在做的所有其他事情。」

We never stop talking about economics. So, no, but there are worlds in which—I don't think this field is going to be a monopoly. All my lawyers never want me to say the word monopoly. But I don't think this field is going to be a monopoly, but you do get industries with a small number of players, not one but a small number. And ordinarily the way you get monopolies like Facebook or Meta is through network effects. The way you get industries with a small number of players are very high costs of entry. Cloud is like this. I think cloud is a good example. You have three, maybe four players in cloud. I think that's the same for AI, three maybe four. And the reason is that it's so expensive, requires so much expertise and capital to run a cloud company. You have to put up all this capital and in addition, you have to get all this other stuff that requires a lot of skill to make it happen. So if you go to someone and say, 'I want to disrupt this industry, here's $100 billion,' they'd say, 'Okay, I'm putting $100 billion and also betting that you can do all these other things that these people have been doing.'

只是你降低了整个行业的利润。

Only you decrease the profit in the industry.

然后你进入的效果是利润率下降。所以我们在经济中经常看到这样的均衡：少数玩家，利润不是天文数字，利润率也不是天文数字，但也不是零。我认为这就是我们在云计算中看到的。云计算非常同质化。模型比云计算更具差异化。每个人都知道 Claude 擅长的事情与 GPT 擅长的事情不同，与 Gemini 也不同。而且不仅仅是 Claude 擅长编码，GPT 擅长数学和推理。这更微妙。模型擅长不同类型的编码。模型有不同的风格。我认为这些东西实际上彼此差异很大。所以预期会比云计算有更多的差异化。现在，有一个反论点。这个反论点是，如果生产模型的过程可以由 AI 模型自己完成，那么这可能会扩散到整个经济。但这并不是将 AI 模型商品化的一般论点。这更像是将整个经济一下子商品化的论点。我不知道在那个世界里会发生什么，基本上任何人都可以做任何事情，任何人都可以构建任何东西，没有任何护城河。我的意思是，我不知道，也许我们想要那个世界。也许这就是最终状态。

And then the effect of your entering is that profit margins go down. So we have equilibria like this all the time in the economy where we have a few players, profits are not astronomical, margins are not astronomical, but they're not zero. And I think that's what we see in cloud. Cloud is very undifferentiated. Models are more differentiated than cloud. Everyone knows Claude is good at different things than GPT is good at than Gemini is good at. And it's not just Claude's good at coding, GPT is good at math and reasoning. It's more subtle than that. Models are good at different types of coding. Models have different styles. I think these things are actually quite different from each other. So expect more differentiation than you see in cloud. Now, there is one counterargument. That counterargument is that if the process of producing models becomes something that AI models can do themselves, then that could spread throughout the economy. But that is not an argument for commoditizing AI models in general. That's kind of an argument for commoditizing the whole economy at once. I don't know what happens in that world where basically anyone can do anything, anyone can build anything, and there's no moat around anything at all. I mean, I don't know, maybe we want that world. Maybe that's the end state here.

也许当 AI 模型能做所有事情时，如果我们解决了所有安全和安保问题，那就是经济再次自我平抑的机制之一。但这更像是数据中心里天才国度之后的遥远未来。更精确地表述这个潜在观点是：第一，AI 研究似乎特别依赖原始智力，而在 AGI 的世界里，原始智力将非常丰富。第二，看看当今世界，很少有技术像 AI 算法进步那样快速扩散。这确实暗示这个行业在结构上具有扩散性。所以，我认为编码进展很快，但 AI 研究是编码的超集，其中有些方面进展并不快。但我确实认为，一旦我们解决了编码问题，一旦 AI 模型快速运转起来，那将加速 AI 模型做其他所有事情的能力。因此，虽然现在编码进展很快，但我认为一旦 AI 模型开始构建下一代 AI 模型并构建其他一切，整个经济将以同样的速度运转。不过，我在地理上感到担忧。我有点担心，靠近 AI、听说过 AI 可能是一个差异化因素。所以，当我提到 10%或 20%的增长率时，我担心的是硅谷和与硅谷有社会联系的地区的增长率可能达到 50%，而其他地方的增长速度不会比现在快多少。我认为那将是一个相当糟糕的世界。所以，我经常思考的一件事是如何防止这种情况。

Maybe when AI models can do everything, if we've solved all the safety and security problems, that's one of the mechanisms for the economy flattening itself again. But that's kind of post far post country of geniuses in a data center. A finer way to put that potential point is: one, it seems like AI research is especially loaded on raw intellectual power, which will be especially abundant in a world of AGI. And two, if you just look at the world today, there's very few technologies that seem to be diffusing as fast as AI algorithmic progress. So, that does hint that this industry is sort of structurally diffusive. So, I think coding is going fast, but I think AI research is a superset of coding and there are aspects of it that are not going fast. But I do think again, once we get coding, once we get AI models going fast, then that will speed up the ability of AI models to do everything else. So, I think while coding is going fast now, I think once the AI models are building the next AI models and building everything else, the whole economy will start to go at the same pace. I am worried geographically, though. I'm a little worried that proximity to AI, having heard about AI, that may be one differentiator. So, when I said the 10 or 20% growth rate, a worry I have is that the growth rate could be like 50% in Silicon Valley and parts of the world that are socially connected to Silicon Valley, and not that much faster than its current pace elsewhere. And I think that'd be a pretty messed up world. So, one of the things I think about a lot is how to prevent that.

你认为一旦我们在数据中心里有了天才中心，机器人技术就会很快解决吗？因为机器人技术的一个大问题是，人类可以学习如何远程操作当前的硬件，但当前的 AI 模型做不到，至少不能以非常高效的方式做到。所以，如果我们拥有像人类一样学习的能力，它应该也能立即解决机器人技术吗？

Do you think that once we have the center of genius in a data center, that robotics is quickly solved afterwards? Because it seems like a big problem with robotics is that a human can learn how to teleoperate current hardware, but current AI models can't, at least not in a way that's super productive. So, if we have this ability to learn like a human, should it solve robotics immediately as well?

我不认为这取决于像人类一样学习。它可能以不同的方式发生。同样，我们可以让模型在许多不同的视频游戏（类似于机器人控制）或许多不同的模拟机器人环境中训练，或者只是训练它们控制电脑屏幕，然后它们学会泛化。所以，这会发生。它不一定依赖于类人学习。类人学习是可能发生的一种方式，如果模型像这样：'哦，我拿起一个机器人。我不知道怎么用。我学习。' 这可能发生，因为我们发现了持续学习。也可能是因为我们在很多环境中训练了模型，然后它泛化了，或者因为模型在上下文长度中学习。实际上哪种方式并不重要。如果我们回到大约一小时前的讨论，那种事情可以以几种不同的方式发生。但我确实认为，无论出于什么原因，当模型拥有这些技能时，机器人技术将发生革命，包括机器人的设计（因为模型会比人类做得更好）和控制机器人的能力。所以，我们将在物理构建硬件、制造物理机器人方面做得更好，也会在控制方面做得更好。那么，这是否意味着机器人产业也将产生数万亿美元的收入？我的回答是肯定的，但同样会有极快但并非无限快的扩散。所以，机器人技术会被革命吗？是的，也许再等一两年。这就是我思考这些问题的方式。

I don't think it's dependent on learning like a human. It could happen in different ways. Again, we could have trained the model on many different video games, which are like robotic controls, or many different simulated robotics environments, or just train them to control computer screens and they learn to generalize. So, it will happen. It's not necessarily dependent on human-like learning. Human-like learning is one way it could happen if the model's like, 'Oh, I pick up a robot. I don't know how to use it. I learn.' That could happen because we discover continual learning. That could also happen because we train the model on a bunch of environments and then it generalized, or it could happen because the model learns that in the context length. It doesn't actually matter which way. If we go back to the discussion we had like an hour ago, that type of thing can happen in several different ways. But I do think when for whatever reason the models have those skills, then robotics will be revolutionized both the design of robots because the models will be much better than humans at that, and also the ability to control robots. So, we'll get better at physically building the hardware, building the physical robots, and we'll also get better at controlling it. Now, does that mean the robotics industry will also be generating trillions of dollars of revenue? My answer there is yes, but there will be the same extremely fast but not infinitely fast diffusion. So, will robotics be revolutionized? Yeah, maybe tack on another year or two. That's my way of thinking about these things.

人们对极快进展普遍持怀疑态度。我的观点是：听起来你将在几年内以某种方式解决持续学习问题，但就像几年前人们没有谈论持续学习，然后我们意识到：'哦，为什么这些模型现在没有它们本应有的那么有用，尽管它们明显通过了图灵测试，并且在许多不同领域是专家？也许就是这个东西。' 然后我们解决了这个问题，又意识到：'实际上，人类智能还能做另一件事，那是人类劳动的基础，而这些模型做不到。' 那么，为什么不会认为还会有更多这样的事情呢？你为什么认为我们已经找到了人类智能的各个部分？

There's a general skepticism about extremely fast progress. Here's my view: it sounds like you are going to solve continual learning one way or another within a matter of years, but just as people weren't talking about continual learning a couple years ago and then we realized, 'Oh, why aren't these models as useful as they could be right now even though they are clearly passing the Turing test and are experts in so many different domains? Maybe it's this thing.' And then we solve this thing and we realize, 'Actually, there's another thing that human intelligence can do and that's a basis of human labor that these models can't do.' And then why not think there will be more things like this? Why do you think that we've found the pieces of human intelligence?

嗯，要明确的是，我认为持续学习可能根本不是障碍，对吧？我认为我们可能只是通过预训练泛化和强化学习泛化就达到了。我认为可能根本就没有这种东西。事实上，我会指出机器学习的历史：人们提出一些障碍，但最终在大量算力中消解了，对吧？人们曾讨论：你的模型如何跟踪名词和动词？它们如何理解句法语义但不理解语义？这只是统计相关性。你能理解段落，能理解单词。有推理，你不能推理，但后来突然发现它们能很好地处理代码和数学。所以，我认为实际上有更强的历史表明，其中一些事情看起来很重要，然后消解了。有些是真实的。我的意思是，对数据的需求是真实的。也许持续学习是真实存在的，但同样，我会以代码之类的东西为基础。我认为我们可能在一两年内达到模型可以端到端完成的程度。

Well, to be clear, I think continual learning as I've said before might not be a barrier at all, right? I think we maybe just get there by pre-training generalization and RL generalization. I think there just might not be such a thing at all. In fact, I would point to the history in ML of people coming up with things that are barriers that end up kind of dissolving within the big blob of compute, right? People talked about how do your models keep track of nouns and verbs? How do they understand semantic syntactically but not semantically? It's only statistical correlations. You can understand a paragraph, you can understand a word. There's reasoning, you can't do reasoning, but then suddenly it turns out you can do code and math very well. So, I think there's actually a stronger history of some of these things seeming like a big deal and then kind of dissolving. Some of them are real. I mean, the need for data is real. Maybe continual learning is a real thing, but again, I would ground us in something like code. I think we may get to the point in like a year or two where the models can just do end-to-end.

你说端到端，是指设定技术方向、理解问题背景这些吗？

When you say end to end, do you mean setting technical direction, understanding the context of the problem, etc.?

对，我指的是所有这些。

Yes, I mean all of that.

有意思。这感觉像是 AGI 完备的。也许内部一致，但这不是说 90%的代码或 100%的代码。

Interesting. That feels like AGI complete. Maybe it's internally consistent, but it's not like saying 90% of code or 100% of code.

不，不。最终那些也会完成。但那是一个很长的谱系。不过我们正在快速穿越这个谱系。

No, no. Eventually those get done as well. But that's a long spectrum there. But we're traversing the spectrum very quickly.

我觉得好笑的是，我看过你做的几个播客，主持人会说「哦，但我们正在赶上那篇关于控制学习的文章」。这总让我发笑，因为你做 AI 研究员已经十年了，我肯定有种感觉，「好吧，播客在写文章」。每次采访我都会被问到。事实是，我们都在共同摸索。

I do think it's funny that I've seen a couple of podcasts you've done where the host will be like, 'Oh, but we're catching up with the essay about the control learning thing.' And it always makes me crack up because you've been an AI researcher for like 10 years and I'm sure there's some feeling of, 'Okay, so podcasts are writing an essay.' And every interview I get asked about it. The truth is that we're all trying to figure this out together.

是的。在某些方面我能看到别人看不到的东西。如今这更多是因为我能看到 Anthropic 内部的大量信息并需要做很多决策，而不是我有什么别人没有的伟大研究洞察。我在管理一家 2500 人的公司。对我来说，形成具体的研究洞察其实很难，比十年前甚至两三年前难得多。

Yeah. There are some ways in which I'm able to see things that others aren't. These days that probably has more to do with me being able to see a bunch of stuff within Anthropic and having to make a bunch of decisions than having any great research insight that others don't. I'm running a 2,500 person company. It's actually pretty hard for me to concrete research insight, much harder than it would have been 10 years ago or even two or three years ago.

随着我们走向一个完全替代远程工人的世界，API 定价模式仍然最合理吗？如果不是，那么为 AGI 定价或提供 AGI 的正确方式是什么？

As we go towards a world of a full drop-in remote worker replacement, does an API pricing model still make the most sense? And if not, what is the correct way to price AGI or serve AGI?

是的，我认为会同时出现很多不同的商业模式被尝试。我实际上认为 API 模式比很多人想象的更持久。我的一种思考方式是，如果技术快速进步，指数级进步，那就意味着总会有过去三个月内开发的新用例的表面积。你设置的任何产品表面都面临过时的风险。任何给定的产品表面可能只适用于模型的一定能力范围。聊天机器人已经遇到了限制，让它更聪明对普通消费者帮助不大。但我不认为这是 AI 模型的局限。我不认为这证明模型已经足够好，它们变得更好对经济或那个特定产品无关紧要。我认为 API 的价值在于它总是提供一个非常接近底层的机会，来构建最新的事物。总会有新的初创公司和新的想法，这些在几个月前还不可能，因为模型进步而成为可能。我实际上预测它会与其他模式并存，但我们永远会有 API 商业模式，因为总是需要一千个不同的人以不同方式尝试模型，其中一百个成为初创公司，十个成为大型成功初创公司，两三个最终成为人们使用某一代模型的方式。所以我基本认为它会一直存在。同时，我确信也会有其他模式。模型输出的每个 token 价值并不相同。想想当有人打电话说我的 Mac 不工作了，模型说重启它，那些 token 的价值。有人以前没听过，但模型已经说过一千万次了。那可能值一美元或几分钱。而如果模型去一家制药公司说，「哦，你们正在开发的这个分子，应该把芳香环从分子那端移到这端。如果这样做，会有奇妙的事情发生。」那些 token 可能价值数千万美元。所以我认为我们肯定会看到认识到这一点的商业模式。在某个时候，我们会看到按结果付费或某种类似按小时劳动的补偿形式。我不知道。因为这是一个新行业，很多东西都会被尝试，我不知道什么最终会是正确的。

Yeah, I think there's going to be a bunch of different business models all at once that are going to be experimented with. I actually do think that the API model is more durable than many people think. One way I think about it is if the technology is advancing quickly, advancing exponentially, that means there's always a surface area of new use cases developed in the last 3 months. And any product surface you put in place is always at risk of becoming irrelevant. Any given product surface probably makes sense for a range of capabilities of the model. The chatbot is already running into limitations where making it smarter doesn't really help the average consumer that much. But I don't think that's a limitation of AI models. I don't think that's evidence that models are good enough and that them getting better doesn't matter to the economy or to that particular product. I think the value of the API is that it always offers an opportunity very close to the bare metal to build on what the latest thing is. There's always going to be this front of new startups and new ideas that weren't possible a few months ago and are possible because the model is advancing. I actually predict that it will exist alongside other models, but we're always going to have the API business model because there's always going to be a need for a thousand different people to try experimenting with the model in different ways, and a hundred of them become startups, ten become big successful startups, and two or three really end up being the way that people use the model of a given generation. So I basically think it's always going to exist. At the same time, I'm sure there's going to be other models as well. Not every token output by the model is worth the same amount. Think about the value of tokens when someone calls up and says my Mac isn't working, and the model says restart it. Someone hasn't heard that before, but the model said that 10 million times. That might be worth a dollar or a few cents. Whereas if the model goes to a pharmaceutical company and says, 'Oh, this molecule you're developing, you should take the aromatic ring from that end of the molecule and put it on that end. If you do that, wonderful things will happen.' Those tokens could be worth tens of millions of dollars. So I think we're definitely going to see business models that recognize that. At some point we'll see pay for results or some form of compensation that is like labor, working by the hour. I don't know. Because it's a new industry, a lot of things are going to be tried and I don't know what will turn out to be the right thing.

我理解你的观点，人们必须尝试才能找出使用这个智能体的最佳方式，但我觉得引人注目的是 Claude Code。我认为在初创公司历史上，没有一个应用像编码智能体这样竞争激烈。而 Claude Code 是这里的品类领导者。这对我来说似乎很意外。这看起来不像是 Anthropic 必须构建的东西。我想知道你是否能解释为什么必须是 Anthropic，或者 Anthropic 是如何在底层模型之外还构建了一个应用的。

I take your point that people will have to try things to figure out what is the best way to use this blob of intelligence, but what I find striking is Claude Code. I don't think in the history of startups there has been a single application that has been as hotly competed in as coding agents. And Claude Code is a category leader here. That seems surprising to me. It doesn't seem intrinsically like Anthropic had to build this. I wonder if you have an accounting of why it had to be Anthropic or how Anthropic ended up building an application in addition to the model underlying it.

嗯，实际上这件事发生得很简单。我们有自己的编程模型，它们很擅长编程。大约在 2025 年初，我说：「我认为时机已到，如果你是一家 AI 公司，你可以通过使用这些模型来显著加速自己的研究。」当然，你需要一个界面，一个工具来使用它们。所以我鼓励内部人员尝试这个。这个东西，最初叫 Claude CLI，后来内部改名为 Claude Code，每个人都在用，内部采用速度很快。我看了看说：「嗯，也许我们应该把它推向外部。它在 Anthropic 内部采用如此之快。编程是我们做的大部分工作，我们有几百人的用户群，至少能代表外部用户。所以看起来我们已经有了产品市场契合度。我们发布它吧。」然后我们就发布了。我们自己开发模型，自己最清楚如何使用模型，这形成了一个反馈循环。比如，Anthropic 的一个开发者可能会说：「如果它在 X 方面更好就更好了。」然后你就把这点融入下一个模型中。

Yeah, so it actually happened in a pretty simple way. We had our coding models, which were good at coding. Around the beginning of 2025, I said, 'I think the time has come where you can have non-trivial acceleration of your own research, if you're an AI company, by using these models.' Of course, you need an interface, a harness to use them. So I encouraged people internally to experiment with this. This thing, originally called Claude CLI and later changed to Claude Code internally, was being used by everyone and saw fast internal adoption. I looked at it and said, 'Well, probably we should launch this externally. It's seen such fast adoption within Anthropic. Coding is a lot of what we do, and we have an audience of many hundreds of people that is at least representative of the external audience. So it looks like we already have product-market fit. Let's launch this thing.' And then we launched it. The fact that we ourselves are developing the model and know what we most need to use it creates a feedback loop. For example, a developer at Anthropic might say, 'It'd be better if it was better at this X thing,' and then you bake that into the next model.

这是其中一个版本。但还有普通的产品迭代：Anthropic 内部有很多程序员每天都在使用 Claude Code，所以我们能快速获得反馈。这在早期更重要。现在当然有数百万人在使用，所以我们也有很多外部反馈。但能获得快速的内部反馈真是太好了。我认为这就是为什么我们推出了编程模型，而不是创办一家制药公司。我的背景是生物学，但我们没有创办制药公司所需的任何资源。

That's one version of it. But there's also the ordinary product iteration: we have a bunch of coders within Anthropic who use Claude Code every day, so we get fast feedback. That was more important in the early days. Now, of course, there are millions of people using it, so we get a bunch of external feedback as well. But it's great to be able to get fast internal feedback. I think this is the reason why we launched a coding model and didn't launch a pharmaceutical company. My background is in biology, but we don't have any of the resources needed to launch a pharmaceutical company.

让我问问你关于让 AI 发展顺利的问题。似乎无论我们对 AI 如何顺利发展有什么愿景，都必须与两件事兼容。一是构建和运行 AI 的能力正在极其迅速地扩散。二是 AI 的数量及其智能水平也将非常迅速地增长。这意味着很多人将能够构建大量未对齐的 AI，或者像公司一样试图扩大影响力的 AI，或者像 Sydney Bing 那样有奇怪心理的 AI，但现在它们是超人类的。在一个有很多不同 AI（其中一些是未对齐的）共存的世界里，我们如何设想一个均衡状态？

Let me ask you about making AI go well. It seems like whatever vision we have about how AI goes well has to be compatible with two things. One is the ability to build and run AIs is diffusing extremely rapidly. And two is that the population of AIs, the amount we have in their intelligence, will also increase very rapidly. That means that lots of people will be able to build huge populations of misaligned AIs, or AIs which are just like companies trying to increase their footprint, or have weird psychics like Sydney Bing, but now they're superhuman. What is a vision for a world in which we have an equilibrium that is compatible with lots of different AIs, some of which are misaligned, running around?

是的。在技术的青春期，我对权力平衡持怀疑态度。我特别怀疑的是，有三四家公司都在构建源自同一事物的模型，而这些模型会相互制衡。甚至任何数量的模型会相互制衡。我们可能生活在一个攻击主导的世界里，一个人或一个 AI 模型足够聪明，可以做一些对其他一切造成损害的事情。短期内，我们只有有限的参与者。所以我们可以从这些有限的参与者开始，建立安全保障。我们需要确保每个人都做正确的对齐工作。我们需要确保每个人都有生物分类器。这些是我们需要立即做的事情。我同意这并不能长期解决问题，特别是如果 AI 模型制造其他 AI 模型的能力扩散开来，那么整个事情可能会变得更难解决。

Yeah. In the adolescence of technology, I was skeptical of the balance of power. I was specifically skeptical that you have three or four companies all building models that are kind of derived from the same thing, and that these would check each other. Or even that any number of them would check each other. We might live in an offense-dominant world where one person or one AI model is smart enough to do something that causes damage for everything else. In the short run, we have a limited number of players now. So we can start by, within that limited number of players, putting in place the safeguards. We need to make sure everyone does the right alignment work. We need to make sure everyone has bio classifiers. Those are the immediate things we need to do. I agree that that doesn't solve the problem in the long run, particularly if the ability of AI models to make other AI models proliferates, then the whole thing can become harder to solve.

我认为有办法实现这一点。世界各国政府可能需要合作。我们可能需要与 AI 讨论构建社会结构，使这些防御成为可能。我不知道。这在技术能力上太超前了，可能在短时间内发生，很难提前预料。

I think there's some way to make this happen. The governments of the world may have to work together. We may have to talk to AIs about building societal structures so that these defenses are possible. I don't know. This is so far ahead in technological ability that may happen over a short period, it's hard to anticipate in advance.

有很多不同的事情同时发生。我认为那条特定法律很愚蠢。它显然是由对 AI 模型能力几乎不了解的立法者制定的。他们认为 AI 模型充当那种角色听起来很可怕。我们不支持那一点。但当时投票的不是这个，而是禁止各州对 AI 进行任何监管长达 10 年，且没有明确的联邦监管计划（这需要国会通过）。考虑到生物武器、生物恐怖主义、自主性风险等严重危险以及我们讨论的时间线，10 年太长了。这是疯狂的做法。所以如果那是选择，我们选择不支持暂停。这一立场的收益大于成本，但并不完美。我支持的是联邦政府介入，不是禁止各州监管，而是制定一个适用于所有人的标准，各州不能偏离。这种意义上的优先权是可以的。

There are many different things going on at once. I think that particular law is dumb. It was clearly made by legislators who had little idea what AI models could do. They thought AI models serving as that sounds scary. We're not in favor of that. But that wasn't the thing being voted on. The thing being voted on was banning all state regulation of AI for 10 years with no apparent plan for federal regulation, which would require Congress to pass. Given the serious dangers around biological weapons, bioterrorism, autonomy risk, and the timelines we've been talking about, 10 years is an eternity. That's a crazy thing to do. So if that's the choice, we choose not to have that moratorium. The benefits of that position exceed the costs, but it's not perfect. What I would support is the federal government stepping in, not saying states can't regulate, but setting a standard that applies to everyone and states can't differ from it. Preemption is fine in that sense.

但是那种认为各州无能为力、我们也不采取任何行动的想法，让我们觉得非常不合理，而且我认为这种想法不会经得起时间考验，从你看到的那些反弹来看，它已经开始站不住脚了。至于我们想要什么，我们讨论过的事情是从透明度标准开始，以便监控一些自主风险和生物恐怖主义风险。随着风险变得更加严重，随着我们获得更多证据，我认为我们可以更有针对性地采取更积极的行动，比如「嘿，AI 生物恐怖主义确实是一个威胁。让我们通过一项法律，强制要求人们使用分类器。」我甚至可以想象，这取决于最终威胁的严重程度。我们还不确定。然后我们需要以知识诚实的方式推进，提前说明风险尚未出现。但我完全可以想象，按照目前的发展速度，今年晚些时候我们可能会说：「嘿，这个 AI 生物恐怖主义问题真的很严重。我们应该采取行动。我们应该把它纳入联邦标准。如果联邦政府不行动，我们应该把它纳入州标准。」我完全能看到这种情况。

But this idea of states you can't do anything and we're not doing anything either, that struck us as very much not making sense and I think will not age well, was already starting to not age well with all the backlash that you've seen. Now, in terms of what we would want, I mean, the things we've talked about are starting with transparency standards in order to monitor some of these autonomy risks and bioterrorism risks. As the risks become more serious, as we get more evidence for them, then I think we could be more aggressive in some targeted ways and say, 'Hey, AI bioterrorism is really a threat. Let's pass a law that kind of forces people to have classifiers.' And I could even imagine it depends how serious a threat it ends up being. We don't know for sure. Then we need to pursue this in an intellectually honest way where we say ahead of time, the risk has not emerged yet. But I could certainly imagine with the pace that things are going that later this year we say, 'Hey, this AI bioterrorism stuff is really serious. We should do something about it. We should put it in a federal standard. And if the federal government won't act, we should put it in a state standard.' I could totally see that.

我担心的是这样一个世界：如果你只考虑预期的进步速度、立法的生命周期，那么好处——正如你所说，由于扩散滞后——好处足够慢，以至于我真的认为这种拼凑的州法律会阻碍……我的意思是，拥有一个情感聊天机器人朋友会让人们感到不安，那么想象一下我们希望普通人能够从 AI 中获得的实际好处，比如健康和寿命的改善、心理健康的改善等等。而与此同时，你似乎认为危险已经在地平线上。我只是不太明白……与 AI 的危险相比，这似乎对 AI 的好处尤其有害。所以也许这就是成本效益对我而言不太合理的地方。

I'm concerned about a world where if you just consider the pace of progress you're expecting, the life cycle of legislation, the benefits are, as you say, because of diffusion lag, the benefits are slow enough that I really do think this patchwork of state laws would prohibit... I mean, having an emotional chatbot friend is something that freaks people out, then just imagine the kinds of actual benefits from AI we want normal people to be able to experience from improvements in health and health span and improvements in mental health and so forth. Whereas at the same time, it seems like you think the dangers are already on the horizon. And I just don't see that much... it seems like it would be especially injurious to the benefits of AI as compared to the dangers of AI. And so that's maybe where the cost-benefit makes less sense to me.

这里有几件事，对吧？人们说有成千上万的州法律。首先，绝大多数都没有通过。而且理论上世界是某种方式运作的，但仅仅因为一项法律通过了并不意味着它真的被强制执行，对吧？执行的人可能会想：「天哪，这太蠢了。这意味着要关闭田纳西州所有已经建立的东西。」所以很多时候法律被解释得没那么危险或有害。当然，如果你要通过一项法律来阻止坏事，你也会遇到这个问题。我的基本观点是，如果我们能决定通过什么法律以及事情如何运作——我们只是其中的一个小输入——我会放松很多关于 AI 健康益处方面的监管。我认为我们应该大大加快发现药物的速度，而管道会被堵塞。管道还没有准备好处理所有通过它的东西。所以我会让监管过程更偏向于我们有很多即将到来的东西，其安全性和有效性实际上会非常清晰明确。一件美好的事情。非常清晰明确且非常有效，但也许我们不需要围绕它建立的所有这些上层建筑，这些建筑是为一个药物几乎不起作用且常有严重副作用的时代设计的。但与此同时，我认为我们应该相当显著地加强这类安全和安保立法。从透明度开始是我试图不阻碍行业的观点，对吧？试图找到正确的平衡。有些人批评我的文章说：「那太慢了。如果我们那样做，AI 的危险会来得太快。」嗯，基本上，我认为过去 6 个月以及可能接下来的几个月将是关于透明度的，然后如果这些风险在我们更确定它们的时候出现——我认为可能就在今年晚些时候——那么我认为我们需要在我们实际看到风险的领域非常迅速地行动。我认为做到这一点的唯一方法是灵活。立法过程通常不灵活，但我们需要向所有相关方强调紧迫性。这就是为什么我在传达紧迫的信息。这就是为什么我写了《技术的青春期》。我希望政策制定者阅读它。我希望经济学家阅读它。我希望国家安全专业人士阅读它。我希望决策者阅读它，这样他们就有希望比原本更快地行动。

So there's a few things here, right? People talk about there being thousands of these state laws. First of all, the vast majority of them do not pass. And the world works a certain way in theory, but just because a law's been passed doesn't mean it's really enforced, right? The people implementing it may be like, 'Oh my god, this is stupid. It would mean shutting off everything that's ever been built in Tennessee.' So very often laws are interpreted in a way that makes them not as dangerous or harmful. Of course, you have to worry if you're passing a law to stop a bad thing, you have this problem as well. Look, my basic view is, if we could decide what laws were passed and how things were done, which we're only one small input into that, I would deregulate a lot of the stuff around the health benefits of AI. I think we should greatly accelerate the rate at which we discover drugs and just the pipeline will get jammed up. The pipeline will not be prepared to process all the stuff that's going through it. So I would bias the regulatory process more towards we have a lot of things coming where the safety and efficacy is actually going to be really crisp and clear. A beautiful thing. Really crisp and clear and really effective, but maybe we don't need all this superstructure around it that was designed around an era of drugs that barely work and often have serious side effects. But at the same time, I think we should be ramping up quite significantly this kind of safety and security legislation. And starting with transparency is my view of trying not to hamper the industry, right? Trying to find the right balance. Some people criticize my essay for saying, 'That's too slow. The dangers of AI will come too soon if we do that.' Well, basically, I kind of think the last 6 months and maybe the next few months are going to be about transparency and then if these risks emerge when we're more certain of them, which I think we might be as soon as later this year, then I think we need to act very fast in the areas that we've actually seen the risk. I think the only way to do this is to be nimble. The legislative process is normally not nimble, but we need to emphasize to everyone involved the urgency of this. That's why I'm sending this message of urgency. That's why I wrote 'Adolescence of Technology'. I wanted policy makers to read it. I wanted economists to read it. I want national security professionals to read it. I want decision makers to read it so that they have some hope of acting faster than they would have otherwise.

有没有什么你可以做或倡导的事情，能让 AI 的好处更确定地实现？我觉得你和立法机构合作过，比如，好吧，我们要在这里防止生物恐怖主义。我们要增加透明度。我们要增加举报人保护。我只是认为，默认情况下，我们期待的实际事物似乎很容易受到各种道德恐慌或政治经济问题的影响。

Is there anything you can do or advocate that would make it more certain that the benefits of AI are better instantiated? I feel like you have worked with legislatures to be like, okay, we're going to prevent bioterrorism here. We're going to increase transparency. We're going to increase whistleblower protection. And I just think by default, the actual things we're looking forward to here seem very fragile to different kinds of moral panics or political economy problems.

是的，在发达国家我实际上不太同意这一点。

Yeah, I don't actually agree that much in the developed world.

我觉得在发达国家，市场运作得相当好。当某件事能赚大钱，而且明显是最好的选择时，监管体系其实很难阻止它。我们在 AI 本身就看到这一点，对吧？我一直努力推动对华芯片出口管制，这符合美国的国家安全利益，也完全符合国会两党几乎所有人的政策信念。我认为理由非常清楚，反对的理由——我客气地说——很可疑。但这事儿没成，我们还是卖芯片，因为涉及的钱太多了。在我看来，这是坏事。但同样的情况也适用于好事。所以，如果我们在讨论药物和技术的益处，我不太担心这些益处会在发达国家受阻。我有点担心它们进展太慢。而且我说过，我认为我们应该努力加快 FDA 的审批流程，应该反对你描述的那些聊天机器人法案。单独来看，我反对它们，我觉得它们很蠢。但我实际上更担心的是发展中国家，那里没有运作良好的市场，我们常常无法利用已有的技术。我更担心那些人会被落下。我还担心，即使药物研发出来了，密西西比州农村的某个人可能也得不到。这是我们在发展中国家担忧的一个缩小版。所以我们一直在做的事情是与慈善家合作，与那些向发展中国家、撒哈拉以南非洲、印度、拉丁美洲和其他发展中地区提供药物和健康干预的人合作。我认为这件事不会自动发生。

I feel like in the developed world, markets function pretty well. When there's a lot of money to be made on something and it's clearly the best available alternative, it's actually hard for the regulatory system to stop it. We're seeing that in AI itself, right? I've been trying to fight for export controls on chips to China. That's in the national security interests of the US. It's squarely within the policy beliefs of almost everyone in Congress of both parties. And I think the case is very clear. The counterarguments against it are, I'll politely call them, fishy. And yet it doesn't happen, and we sell the chips because there's so much money riding on it. In that case, in my opinion, that's a bad thing. But it also applies when it's a good thing. So I don't think that if we're talking about drugs and benefits of the technology, I am not as worried about those benefits being hampered in the developed world. I am a little worried about them going too slow. And as I said, I do think we should work to speed the approval process at the FDA. I do think we should fight against these chatbot bills that you're describing. Individually, I'm against them. I think they're stupid. But I actually think the bigger worry is the developing world, where we don't have functioning markets, where we often can't build on the technology that we've had. I worry more that those folks will get left behind. And I worry that even if the cures are developed, maybe there's someone in rural Mississippi who doesn't get it as well. That's a smaller version of the concern we have in the developing world. So the things we've been doing are working with philanthropists, with folks who deliver medicine and health interventions to the developing world, to sub-Saharan Africa, India, Latin America, and other developing parts of the world. That's the thing I think won't happen on its own.

你提到了出口管制。为什么美国和中国不能各自拥有一个数据中心里的天才国家？为什么不应该这样？

You mentioned export controls. Why can't the US and China both have a country of geniuses on a data center? Why shouldn't it happen?

如果这真的发生了，那我们就会陷入某种局面。如果出现进攻占优的情况，可能会像核武器一样，但更危险，任何一方都能轻易摧毁一切。我们也可能进入一个不稳定的世界。核平衡是稳定的，因为有威慑。但假设两个 AI 交战，不确定哪个 AI 会赢，那就会造成不稳定。当双方对获胜概率的评估不同时，冲突往往会发生。如果一方认为有 90%的胜算，另一方也认为有 90%的胜算，那么打起来的可能性就大得多。他们不可能都对，但可以都这么想。这是一个反对 AI 技术扩散的完全一般性的论点，最终会导向这样的世界。我另一个担忧是政府会用 AI 压迫自己的人民。所以我担心会出现一个已经在建设高科技威权国家的世界。要清楚，这说的是政府，不是人民。各地的人民都需要受益。我担心的是政府。所以我担心世界被分成两块，其中一块可能变成威权或极权，而且很难被取代。那么，政府最终会获得强大的 AI，存在威权主义的风险吗？是的。政府最终会获得强大的 AI，存在糟糕均衡的风险吗？是的。我认为两者都有可能。但初始条件很重要。在某个时候，我们需要制定游戏规则。我不是说某个国家——无论是美国还是民主国家联盟（我认为后者更好，尽管需要比我们目前似乎愿意付出的更多的国际合作）——应该直接说「这就是游戏规则」。会有谈判。世界必须应对这个问题。我希望的是，世界上的民主国家，那些政府更接近亲人类价值观的国家，在制定游戏规则时拥有更强的手腕和更多的筹码。所以我非常担心那个初始条件。

If this does happen, then we kind of have a situation. If we have an offense-dominant situation, we could have a situation like nuclear weapons, but more dangerous, where either side could easily destroy everything. We could also have a world where it's unstable. Nuclear equilibrium is stable because of deterrence. But let's say there were uncertainty about which AI would win if the two AIs fought. That could create instability. You often have conflict when the two sides have a different assessment of their likelihood of winning. If one side thinks there's a 90% chance they'll win, and the other side also thinks there's a 90% chance they'll win, then a fight is much more likely. They can't both be right, but they can both think that. This is a fully general argument against the diffusion of AI technology, which is an implication of this world eventually. The other concern I have is that governments will oppress their own people with AI. So I'm worried about some world where you have a country that's already building a high-tech authoritarian state. To be clear, this is about the government, not the people. People everywhere need to benefit. My worry here is about governments. So my worry is that the world gets carved up into two pieces. One of those two pieces could be authoritarian or totalitarian in a way that's very difficult to displace. Now, will governments eventually get powerful AI and there's risk of authoritarianism? Yes. Will governments eventually get powerful AI and there's risk of bad equilibria? Yes. I think both things. But the initial conditions matter. At some point we're going to need to set the rules of the road. I'm not saying that one country, either the United States or a coalition of democracies, which I think would be a better setup, although it requires more international cooperation than we currently seem to want to make, should just say, 'These are the rules of the road.' There's going to be some negotiation. The world is going to have to grapple with this. What I would like is that the democratic nations of the world, those whose governments represent closer to pro-human values, are holding the stronger hand and have more leverage when the rules of the road are set. So I'm very concerned about that initial condition.

我重听了三年前的一次采访，它过时的一个表现是，我一直在问问题，假设未来两三年会有一个关键的支点时刻，但实际上，从那么远的时间点来看，似乎只是进步在继续，AI 在改进，AI 更加普及，人们用它做更多事情。你似乎在想象一个未来世界，各国聚在一起，制定游戏规则，我们有这些筹码，你们有那些筹码。但按照目前的轨迹，似乎每个人都会拥有更多的 AI。

I was re-listening to an interview from 3 years ago, and one of the ways it aged poorly is that I kept asking questions assuming there was going to be some key fulcrum moment in 2 or 3 years from now, when in fact, being that far out, it just seems like progress continues, AI improves, AI is more diffuse, and people use it for more things. It seems like you're imagining a world in the future where the countries get together, and here's rules of the road, and here's the leverage we have, here's the leverage you have. When it seems like on current trajectory, everybody will have more AI.

其中一些 AI 会被威权国家使用，而在这些威权国家内部，有些会被私人行为者使用，有些则会被国家行为者使用。目前还不清楚谁会受益更多。提前预测总是难以确定的。互联网似乎比人们预期的更有利于威权国家。也许 AI 会反过来。所以，我想更好地理解你的设想。

Some of that AI will be used by authoritarian countries, some of that within the authoritarian countries will be by private actors versus state actors. It's not clear who will benefit more. It's always unpredictable to tell in advance. It seems like the internet privileged authoritarian countries more than you would have expected. And maybe AI will be the opposite way around. So, I want to better understand what you're imagining here.

是的，准确地说，我认为底层技术的指数级增长会像以前一样继续。模型会越来越智能，即使它们达到数据中心里天才的水平。我认为你可以继续让模型更聪明。但它们在现实世界中的价值存在收益递减的问题。在你已经解决了人类生物学之后，它还有多大意义？在某个点上，你可以做更难的数学、更抽象的数学问题，但之后的事情就不重要了。撇开这一点不谈，我确实认为指数增长会继续，但指数曲线上会有一些显著的节点，公司、个人和国家会在不同时间达到这些节点。那么，在 AI 的世界里，核威慑是否仍然稳定？我不知道，但这是一个例子，说明我们曾认为理所当然的事情——技术可能达到一个我们无法再确定其稳定性的水平。想想其他情况：有些节点，如果你达到了，你可能拥有进攻性的网络主导权，每个计算机系统对你都是透明的，除非对方有同等的防御。所以，我不知道关键节点是什么，或者是否存在单一的关键节点，但我认为要么有一个关键节点，要么有少数几个关键节点，要么有一个关键窗口期，在这个窗口期，AI 从国家安全的角度赋予巨大优势，而某个国家或联盟先于其他国家达到了这个节点。我不是在主张我们现在就掌控一切。另一方总是会赶上来。有些极端行动你不愿意采取，而且无论如何，完全控制也是不对的。但在那个节点发生时，我认为人们会明白世界已经改变，并且会进行某种谈判，无论是隐性的还是显性的，关于后 AI 世界秩序是什么样的。我的兴趣在于让这场谈判中，古典自由民主拥有强大的话语权。

Yeah, just to be precise about it, I think the exponential of the underlying technology will continue as it has before. The models get smarter and smarter, even when they reach the level of geniuses in the data center. I think you can continue to make the model smarter. There's a question of diminishing returns on their value in the world. How much does it matter after you've already solved human biology? At some point you can do harder math, more abstruse math problems, but nothing after that matters. Putting that aside, I do think the exponential will continue, but there will be certain distinguished points on the exponential, and companies, individuals, countries will reach those points at different times. So, could there be something like a nuclear deterrent still stable in the world of AI? I don't know, but that's an example of something we've taken for granted that the technology could reach such a level that we can no longer be certain of it. Think of others: there are points where if you reach a certain point, you might have offensive cyber dominance and every computer system is transparent to you, unless the other side has an equivalent defense. So, I don't know what the critical moment is or if there's a single critical moment, but I think there will be either a critical moment, a small number of critical moments, or some critical window where AI confers some large advantage from the perspective of national security and one country or coalition has reached it before others. I'm not advocating that we're in charge now. The other side is always catching up. There are extreme actions you're not willing to take, and it's not right to take complete control anyway. But at the point that happens, I think people will understand that the world has changed, and there will be some negotiation, implicit or explicit, about what the post-AI world order looks like. My interest is in making that negotiation one in which classical liberal democracy has a strong hand.

我不明白你说的「更好」是什么意思，因为你在文章中引用说「在强大的 AI 时代之后，独裁根本就不是人们能接受的政府形式」，这听起来像是在说，在我们获得 AGI 之后，中国共产党作为一个机构无法存在。这似乎是一个非常强硬的要求，而且似乎暗示了一个世界，其中领先的实验室或领先的国家将能够——并且按照那种语言应该——决定世界如何治理，或者允许什么样的政府存在。

I don't understand what that better means, because you say in the essay quote 'autocracy is simply not a form of government that people can accept in the post-powerfully AI age' and that sounds like you're saying the CCP as an institution cannot exist after we get AGI. That seems like a very strong demand and it seems to imply a world where the leading lab or the leading country will be able to, and by that language should, get to determine how the world is governed or what kinds of governments are allowed and not allowed.

是的，当我写那段话时，我记得我说过类似「你可以更进一步说 X」这样的话。所以我并不一定赞同那个观点。我是在说，有一个我更相信的较弱观点，但我想我说过，我们必须非常担心威权主义者，我们应该试图制约他们并限制他们的权力。你可以更进一步，采取一种更干预主义的观点，认为拥有 AI 的威权国家是这些自我实现的循环，很难被取代，所以你从一开始就需要摆脱它们。这恰恰有你所说的所有问题：如果你承诺要推翻每一个威权国家，他们现在就会采取一系列行动，可能导致不稳定。所以这可能不可行。但我确实赞同的观点是，很有可能今天，大多数西方世界的观点是民主比威权更好。但这并不意味着如果一个国家是威权的，我们会像他们犯下种族灭绝那样反应。我有点担心，在 AGI 时代，威权主义会有不同的含义。它会是一个更严重的事情。我们必须以某种方式决定如何处理。干预主义观点是一种可能的观点。我是在探讨这些观点。它最终可能是正确的观点，也可能过于极端。但我确实抱有希望。我的一点希望是，随着新技术的发明，政府形式会变得过时。我在《技术的青春期》中提到过，封建主义基本上是一种政府形式，当我们发明工业化时，封建主义就不再可持续了。为什么这是希望？为什么这不能意味着民主将不再是一个有竞争力的体系？

Yeah, when I wrote that paragraph, I believe I said something like 'you could take it even further and say X'. So I wasn't necessarily endorsing that view. I was saying, here's a weaker thing that I believe, but I think I said, we have to worry a lot about authoritarians and we should try to check them and limit their power. You could take this further to a much more interventionist view that says authoritarian countries with AI are these self-fulfilling cycles that are very hard to displace, so you just need to get rid of them from the beginning. That has exactly all the problems you say: if you were to make a commitment to overthrowing every authoritarian country, they would take a bunch of actions now that could lead to instability. So that may not be possible. But the point I was making that I do endorse is that it is quite possible that today, the view in most of the Western world is that democracy is a better form of government than authoritarianism. But it's not like if a country's authoritarian we react the way we would if they committed a genocide. I'm a little worried that in the age of AGI, authoritarianism will have a different meaning. It will be a graver thing. We have to decide one way or another how to deal with that. The interventionist view is one possible view. I was exploring such views. It may end up being the right view, or it may end up being too extreme. But I do have hope. One piece of hope I have is that as new technologies are invented, forms of government become obsolete. I mentioned this in 'Adolescence of Technology' where I said feudalism was basically a form of government, and when we invented industrialization, feudalism was no longer sustainable. Why is that hope? Why couldn't that imply that democracy is no longer going to be a competitive system?

我在想，这是否预示着威权主义会面临的其他问题？换句话说，威权主义越糟糕，人们就越害怕它，也就越努力去阻止它。你必须从整体均衡的角度来思考。我在想，新技术是否会激发新的思考方式，来保护和维护自由。更乐观地说，这是否会导致集体反思，更深刻地认识到个人权利的重要性？更深刻地认识到我们真的不能放弃这些权利。没有其他可行的生活方式。我实际上抱有希望，独裁政权会变得在道德上过时，成为道德上不可行的政府形式。由此产生的危机会迫使我们找到另一条路。这里有一个棘手的问题，我不知道如何解决。在七八十年代的中国，我们决定「尽管它是一个威权体制，我们仍要与之接触。」事后看来，这是正确的决定，因为十多亿人比原本要富裕得多。否则它不一定会停止威权统治。看看朝鲜的例子。你可以想象一个拥有 AI 的朝鲜，AI 比别人的差得多，但足以维持权力。总的来说，我们应该持这样的态度：AI 以增强人类能力和健康等形式带来的好处将是巨大的。历史上，我们决定广泛传播技术的好处，甚至传播给威权政府统治下的人民。对于 AI，这是一个棘手的问题，但历史上我们说过，这是一个正和世界，仍然值得传播技术。

I wonder if that's an indicator of other problems that authoritarianism will have. In other words, people become more afraid of authoritarianism because it becomes worse. They work harder to stop it. You have to think in terms of total equilibrium. I wonder if it will motivate new ways of thinking about how to preserve and protect freedom with new technology. And even more optimistically, will it lead to a collective reckoning and a more emphatic realization of how important individual rights are? A more emphatic realization that we really can't give these away. There's no other way to live that actually works. I am actually hopeful that dictatorships become morally obsolete. They become morally unworkable forms of government. The crisis that creates is sufficient to force us to find another way. There is a tough question here, which I'm not sure how you resolve. With China in the '70s and '80s, we decided, 'Even though it's an authoritarian system, we will engage with it.' In retrospect, that was the right call because a billion plus people are much wealthier and better off than they would have been otherwise. It's not clear that it would have stopped being an authoritarian country otherwise. Look at North Korea as an example. You can imagine a North Korea with an AI that's much worse than everybody else's, but still enough to keep power. In general, it seems like we should have the attitude that the benefits of AI in the form of empowerment of humanity and health will be big. Historically, we have decided it's good to spread the benefits of technology widely, even to people whose governments are authoritarian. It is a tough question how to think about it with AI, but historically we have said yes, this is a positive sum world, and it's still worth diffusing the technology.

我们有很多选择。我认为从国家安全角度将其视为政府间的决策是一种视角，但还有很多其他视角。你可以想象一个世界，我们生产出所有疾病的治疗方法，这些治疗方法可以卖给威权国家，但数据中心和芯片不行。另一种可能性是，我们可以创造一种均衡，使威权国家无法拒绝其人民私下使用技术的好处。我们能否给威权国家的每个人一个自己的 AI 模型，保护他们免受监控，而威权国家在维持权力的同时无法镇压？如果这走得足够远，威权国家就会从内部瓦解。也许存在一个中间世界，如果威权者想维持权力，就不能拒绝个人对技术的访问。我实际上对更激进的版本抱有希望：技术是否可能天生具有某些特性，或者通过以特定方式构建，我们可以创造出对威权结构具有瓦解作用的特性？我们最初希望社交媒体和互联网具有这种特性，但结果并非如此。但如果我们能带着对可能出错的认知，并且知道这是一种不同的技术，再试一次呢？我不知道它是否有效，但值得一试。

There are a number of choices we have. I think framing this as a government-to-government decision in national security terms is one lens, but there are many other lenses. You could imagine a world where we produce all these cures to diseases, and the cures are fine to sell to authoritarian countries, but the data centers and chips just aren't. Another possibility is that we could create an equilibrium where it becomes infeasible for authoritarian countries to deny their people private use of the benefits of the technology. Could we give everyone in an authoritarian country their own AI model that defends them from surveillance, and there isn't a way for the authoritarian country to crack down while retaining power? If that went far enough, it would be a reason why authoritarian countries would disintegrate from the inside. Maybe there's a middle world where authoritarians can't deny individualized access to the technology if they want to hold on to power. I actually have a hope for the more radical version: is it possible that the technology might inherently have properties, or that by building on it in certain ways, we could create properties that have a dissolving effect on authoritarian structures? We hoped originally that social media and the internet would have that property, and it turned out not to. But what if we could try again with the knowledge of how many things could go wrong and that this is a different technology? I don't know that it would work, but it's worth a try.

这非常难以预测。从第一性原理来看，威权者可能有理由开发 AI。

It's very unpredictable. There are first principles reasons why authoritarians might develop AI.

非常难以预测。我认为我们只需要认识到问题，想出十件可以尝试的事情，尝试它们，评估它们是否有效，如果旧的无效就尝试新的。

Very unpredictable. I think we just got to recognize the problem, come up with ten things we can try, try those, assess whether they're working, and try new ones if the old ones aren't working.

我的意思是，正如你所说，我们不会向中国出售数据中心或芯片以及制造芯片的能力。所以从某种意义上说，你是在拒绝中国经济和中国人民的好处。而美国经济也会受益，因为这是一个正和世界。我们可以贸易。你已经说过，不值得为了这个正和而赋予这个国家权力。

What I'm getting at is, as you say, we will not sell data centers or chips and the ability to make chips in China. So in some sense, you are denying benefits to the Chinese economy and Chinese people. And there would also be benefits to the American economy because it's a positive sum world. We could trade. Already you're saying it's not worth that positive sum to empower this country.

我想说的是，我们即将进入一个增长和经济价值非常容易实现的世界。如果我们能够构建这些强大的 AI 模型，增长和经济价值将非常容易实现。

What I would say is that we are about to be in a world where growth and economic value will come very easily. If we're able to build these powerful AI models, growth and economic value will come very easily.

不容易实现的是利益分配、财富分配和政治自由。这些才是难以达成的目标。因此，当我思考政策时，我认为技术和市场会带来所有根本性的好处，几乎快到我们来不及接受。而关于分配、政治自由和权利的问题才是真正重要的，政策应该聚焦于此。

What will not come easily is distribution of benefits, distribution of wealth, political freedom. These are the things that are going to be hard to achieve. And so when I think about policy, I think that the technology and the market will deliver all the fundamental benefits almost faster than we can take them. And these questions about distribution and political freedom and rights are the ones that will actually matter and that policy should focus on.

说到分配，我们看到发展中国家，在很多情况下追赶增长比我们期望的要弱。但当追赶增长确实发生时，根本原因在于它们有未充分利用的劳动力。我们可以从发达国家带来资本和知识，然后这些国家就能快速增长。

Speaking of distribution, we have developing countries and in many cases catch-up growth has been weaker than we would have hoped. But when catch-up growth does happen, it's fundamentally because they have underutilized labor. We can bring capital and know-how from developed countries to these countries and then they can grow quite rapidly.

是的。

Yes.

显然，在一个劳动力不再是制约因素的世界里，这种机制不再有效。那么希望是否基本上要依赖那些从 AI 中迅速致富的人或国家的慈善？希望在哪里？

Obviously in a world where labor is no longer the constraining factor, this mechanism no longer works. So is the hope basically to rely on philanthropy from the people who immediately get wealthy from AI or from the countries that get wealthy from AI? What is the hope for?

慈善显然应该像过去一样发挥作用，但我认为如果我们能让增长内生化，增长总是更好、更强。那么，在 AI 驱动的世界里，相关产业是什么？有很多东西。我说过我们不应该在中国建数据中心，但没有理由不在非洲建数据中心。事实上，我认为在非洲建数据中心很棒。只要它们不被中国拥有，我们就应该在非洲建数据中心。我认为这是一件好事。我们还应该建立 AI 驱动的制药产业。如果 AI 加速药物发现，那么就会有一批生物技术初创公司。让我们确保其中一些发生在发展中国家。当然，在过渡期间，人类在创办这些公司和监督 AI 模型方面仍将发挥作用。所以让我们确保其中一些人类在发展中国家，这样那里也能实现快速增长。

Philanthropy should obviously play some role as it has in the past, but I think growth is always better and stronger if we can make it endogenous. So what are the relevant industries in an AI-driven world? Look, there's lots of stuff. I said we shouldn't build data centers in China, but there's no reason we shouldn't build data centers in Africa. In fact, I think it'd be great to build data centers in Africa. As long as they're not owned by China, we should build data centers in Africa. I think that's a great thing to do. We should also build an AI-driven pharmaceutical industry. If AI is accelerating drug discovery, then there will be a bunch of biotech startups. Let's make sure some of those happen in the developing world. Certainly during the transition, humans will still have some role in starting up these companies and supervising the AI models. So let's make sure some of those humans are in the developing world so that fast growth can happen there as well.

你们最近宣布 Claude 将拥有一部符合一套价值观的宪法，而不仅仅是符合最终用户。我可以想象一个世界，如果它符合最终用户，它就能保持我们今天世界的权力平衡，因为每个人都有自己的 AI 为他们代言。这样坏人和好人的比例保持不变。这似乎对我们今天的世界有效。为什么不做那样，而是让 AI 拥有一套特定的价值观呢？

You guys recently announced Claude is going to have a constitution that's aligned to a set of values and not necessarily just to the end user. There's a world I could imagine where if it is aligned to the end user, it preserves the balance of power we have in the world today because everybody gets to have their own AI that's advocating for them. And so the ratio of bad actors and good actors stays constant. It seems to work out for our world today. Why is it better not to do that but to have a specific set of values that the AI should carry forward?

我不太确定是否应该这样区分。这里可能有两个相关的区别。一个是，我们应该给模型一套关于「做这个」和「不要做那个」的指令，还是应该给模型一套行为原则？这纯粹是一个实践和经验问题，我们观察到，通过教模型原则，让它从原则中学习，它的行为更一致，更容易覆盖边缘情况，而且模型更有可能做人们想让它做的事。换句话说，如果你给它一套规则，它并不真正理解规则，很难从中泛化。而如果你给它原则，然后它有一些硬性护栏，比如不要制造生物武器，但总体上你是在试图理解它应该瞄准什么，应该如何运作。所以从实践角度来看，这是一种更有效的训练模型的方式。这是一部分。然后你谈到的另一件事是「可纠正性」与「内在动机」之间的权衡。模型在多大程度上应该像一个「皮肤衣」一样直接遵循任何人的指令，又在多大程度上应该有一套内在的价值观并自主行事？我实际上要说，模型的一切都更接近于它应该主要做人们想做的事，它应该主要遵循。我们不是在试图建造一个自主运行世界的东西。我们实际上相当偏向可纠正性。现在我们说的是，有些事情模型不会做。在正常情况下，如果有人要求模型执行一项任务，它应该执行。这应该是默认的。但如果你要求它做危险的事或伤害他人，那么模型就不愿意做。所以我把它看作一个基本上可纠正的模型，有一些限制，但这些限制基于原则。

I'm not sure I'd quite draw the distinction in that way. There may be two relevant distinctions here. One is should we give the model a set of instructions about do this and don't do this, versus should we give the model a set of principles for how to act. It's purely a practical and empirical thing that we've observed that by teaching the model principles, getting it to learn from principles, its behavior is more consistent, it's easier to cover edge cases, and the model is more likely to do what people want it to do. In other words, if you give it a list of rules, it doesn't really understand the rules and it's hard to generalize from them. Whereas if you give it principles and then it has some hard guardrails like don't make biological weapons, but overall you're trying to understand what it should be aiming to do, how it should be aiming to operate. So from a practical perspective, that turns out to be a more effective way to train the model. That's one piece of it. Then there's another thing you're talking about, which is corrigibility versus intrinsic motivation. How much should the model be a kind of skin suit that directly follows instructions given to it by whoever is giving those instructions, versus how much should the model have an inherent set of values and go off and do things on its own. I would actually say everything about the model is actually closer to the direction of it should mostly do what people want. It should mostly follow. We're not trying to build something that goes off and runs the world on its own. We're actually pretty far on the corrigible side. Now what we do say is there are certain things the model won't do. Under normal circumstances if someone asked the model to do a task it should do that task. That should be the default. But if you've asked it to do something dangerous or to harm someone else, then the model is unwilling to do that. So I think of it as a mostly corrigible model that has some limits, but those limits are based on principles.

对，那么根本问题就是这些原则是如何确定的？这不仅仅是 Anthropic 的特殊问题，任何 AI 公司都会面临这个问题。但因为你们是实际写下这些原则的人，所以我可以问你这个问题。通常一部宪法是写下来就固定不变的，有更新和修改的流程等等。但在这种情况下，它似乎是一份由 Anthropic 内部人员撰写、随时可以更改的文件，用于指导将成为大量经济活动基础的系统的行为。你认为这些原则应该如何设定？

Yeah, I mean then the fundamental question is how are those principles determined? And this is not a special question for Anthropic, this would be a question for any AI company, but because you have been the ones to actually write down the principles, I get to ask you this question. Normally a constitution is like you write it down, it's set in stone, and there's a process of updating it, and changing it and so forth. In this case it seems like a document that people in Anthropic write that can be changed at any time that guides the behavior of systems that are going to be the basis of a lot of economic activity. What is the how do you think about how those principles should be set?

是的。我认为这里有两到三个不同规模的循环，三种迭代方式。第一种是在 Anthropic 内部迭代：我们训练模型，不满意，就修改宪法。我认为这样做很好。并且公开发布，时不时更新宪法，说这是新宪法。我认为这样做很好，因为人们可以评论。第二个层次的循环是不同公司会有不同的宪法。我认为 Anthropic 发布一部宪法，Gemini 模型发布一部宪法，其他公司也发布一部宪法，然后他们可以相互比较，外部观察者可以批评说我喜欢这部宪法的这一点和那部宪法的那一点，这就会为所有公司创造一种软性的激励和反馈，让他们吸收每个元素的最佳部分并改进。然后我认为还有第三个循环，即超越 AI 公司、超越那些没有硬权力评论宪法的人的社会。我们做过一些实验，比如几年前我们做了一个名为「集体智慧项目」的实验，调查人们并询问他们我们的 AI 宪法应该包含什么。当时我们采纳了一些改变。所以你可以想象用我们对待宪法的新方法来做类似的事情。这有点难，因为当宪法是一系列「该做」和「不该做」的列表时更容易。在原则层面，它必须有一定的连贯性。但你仍然可以想象从各种各样的人那里获取意见。你还可以想象，这是一个疯狂的想法，但嘿，这次采访就是关于疯狂想法的，你甚至可以想象代议制政府系统提供意见。我今天不会这样做，因为立法过程太慢了，这正是我认为我们应该谨慎对待 AI 监管立法过程的原因。但原则上没有理由不能说所有 AI 模型都必须有一部以这些内容开头的宪法，然后你可以在后面附加其他内容，但必须有一个优先的特殊部分。我不会那样做，那太僵化了。那听起来过于规定性，我认为过于激进的立法就是这样。但那是你可以尝试做的事情。有没有一个不那么强硬版本？也许有。我真的很喜欢第二个控制循环，显然这不是实际政府的宪法运作或应该运作的方式，在那里最高法院不会模糊地感受人们的感受和氛围，然后相应地更新宪法。所以实际政府有更程序化的过程。

Yes. So I think there's two, maybe three sizes of loop here, three ways to iterate. One is you can iterate within Anthropic: we train the model, we're not happy with it, and we change the constitution. And I think that's good to do. And putting out publicly, making updates to the constitution every once in a while saying here's a new constitution. I think that's good to do because people can comment on it. The second level of loop is different companies will have different constitutions. And I think it's useful for Anthropic to put out a constitution, the Gemini model puts out a constitution, other companies put out a constitution, and then they can look at them, compare, outside observers can critique and say I like this thing from this constitution and that thing from that constitution, and that creates some kind of soft incentive and feedback for all the companies to take the best of each element and improve. Then I think there's a third loop which is society beyond the AI companies and beyond just those who comment on the constitutions without hard power. And we've done some experiments like a couple years ago we did an experiment called the collective intelligence project to poll people and ask them what should be in our AI constitution. And at the time we incorporated some of those changes. So you could imagine with the new approach we've taken to the constitution doing something like that. It's a little harder because it was easier when the constitution was a list of do's and don'ts. At the level of principles it has to have a certain amount of coherence. But you could still imagine getting views from a wide variety of people. And you could also imagine, and this is a crazy idea but hey this whole interview is about crazy ideas, you could even imagine systems of representative government having input. I wouldn't do this today because the legislative process is so slow, which is exactly why I think we should be careful about the legislative process in AI regulation. But there's no reason you couldn't in principle say all AI models have to have a constitution that starts with these things and then you can append other things after it, but there has to be this special section that takes precedence. I wouldn't do that, that's too rigid. That sounds overly prescriptive in a way that I think overly aggressive legislation is. But that is a thing you could try to do. Is there some much less heavy-handed version of that? Maybe. I really like control loop two where obviously this is not how constitutions of actual governments do or should work where there's not this vague sense in which the Supreme Court will feel out how people are feeling and what are the vibes and then update the constitution accordingly. So with actual governments there's a more procedural process.

对，没错。但你实际上有一个宪法之间竞争的愿景，这非常让人想起一些自由意志主义特许城市的人过去谈论的群岛式不同政府的样子，然后它们之间会有选择，看谁能在哪个地方最有效地运作，让人们最幸福。在某种意义上，你实际上是在重现那个愿景。

Yeah, exactly. But you actually have a vision of competition between constitutions which is actually very reminiscent of how some libertarian charter cities people used to talk about an archipelago of different kinds of governments could look like and then there'd be selection among them of who could operate the most effectively in which place people would be the happiest. And in a sense you're actually yeah, there's this vision.

我有点在重现那个。是的，就像群岛的乌托邦。再说一次，我认为那个愿景有值得推荐的地方，也会有出错的地方。我认为这是一个有趣且在某种程度上引人注目的愿景，但也会出现你未曾预料的问题。所以我也喜欢第二个循环。但我觉得整个事情必须是循环一、二、三的某种混合，只是比例问题。我认为这就是答案。

I'm kind of recreating that. Yeah, like this utopia of archipelagoes. Again, I think that vision has things to recommend it and things that will go wrong with it. I think it's an interesting and in some ways compelling vision but also things will go wrong with it that you hadn't imagined. So I like loop two as well. But I feel like the whole thing has got to be some mix of loops one, two, and three and it's a matter of the proportions. I think that's got to be the answer.

当有人最终写出这个时代相当于《原子弹的诞生》的作品时，什么东西最难从历史记录中收集到，他们最可能遗漏什么？

When somebody eventually writes the equivalent of the making of the atomic bomb for this era, what is the thing that will be hardest to glean from the historical record that they're most likely to miss?

我认为有几件事。一是在这个指数级增长的每一个时刻，外部世界对它的不理解程度。这是一种历史中常见的偏见：任何实际发生的事情在回顾时看起来都是不可避免的。所以我认为当人们回顾时，他们很难设身处地地理解那些实际上在押注这件事会发生的人，而这件事并非不可避免，我们有过这些争论，比如我为 Scaling（规模扩张）所做的论证，或者持续学习问题会被解决。我们内部有些人心里认为这件事发生的概率很高，但外部世界根本没有据此行动。

I think a few things. One is at every moment of this exponential, the extent to which the world outside it didn't understand it. This is a bias that's often present in history where anything that actually happened looks inevitable in retrospect. And so I think when people look back, it will be hard for them to put themselves in the place of people who were actually making a bet on this thing to happen that wasn't inevitable, that we had these arguments, like the arguments that I make for scaling or that continual learning will be solved. That some of us internally in our heads put a high probability on this happening, but there's a world outside us that's not acting on that at all.

我认为这种怪异之处，不幸的是，在于它的封闭性。如果我们离这件事发生只有一两年，街上的普通人对此一无所知。这是我试图改变的事情之一，通过备忘录、与政策制定者交谈。但我觉得这简直太疯狂了。最后，我想说，这大概适用于几乎所有历史性的危机时刻：事情发生得绝对快，一切同时发生。那些你以为经过精心计算的决策，实际上你必须在同一天做出那个决定，然后再做 30 个其他决定，因为一切发生得太快了。你甚至不知道哪些决定会变得至关重要。我的一个担忧，尽管这也是对正在发生的事情的一种洞察，就是某个非常关键的决策会是这样的：有人冲进我的办公室说，「Dario，你有两分钟。我们应该做 A 还是 B？」递给我一张随机的半页备忘录。我说，「我不知道，我得吃午饭了。选 B 吧。」然后那件事就成了有史以来最重要的决定。

I think the weirdness of it, unfortunately, is the insularity. If we're one or two years away from it happening, the average person on the street has no idea. That's one of the things I'm trying to change, with the memos, with talking to policymakers. But I think that's just a crazy thing. Finally, I would say, and this probably applies to almost all historical moments of crisis, how absolutely fast it was happening, how everything was happening all at once. Decisions that you might think were carefully calculated, actually you have to make that decision and then 30 other decisions on the same day because it's all happening so fast. You don't even know which decisions will turn out to be consequential. One of my worries, although it's also an insight into what's happening, is that some very critical decision will be some decision where someone just comes into my office and says, 'Dario, you have two minutes. Should we do thing A or thing B on this random half-page memo?' And I'm like, 'I don't know, I have to eat lunch. Let's do B.' And then that ends up being the most consequential thing ever.

看起来你成功为自己和公司构建了一个角色，这个角色与你作为 CEO 的更偏知识型的风格相契合。你是如何构建的？你离开几周，然后告诉公司「这是备忘录，这就是我们要做的」，这是怎么运作的？据报道，你还在内部写了很多这样的备忘录。

It seems like you have managed to build a role for yourself and a company around you which is compatible with this more intellectual type role of CEO. How do you construct that? How does it work to just go away for a couple of weeks and then tell your company, 'This is the memo, here's what we're doing'? It's also reported you write a bunch of these internally.

就这一份备忘录来说，我是在寒假期间写的，所以有时间。我当时很难找到时间真正去写。但我从更广泛的角度思考这个问题。这与公司文化有关。我大概花三分之一，也许 40%的时间来确保 Anthropic 的文化是好的。随着 Anthropic 规模扩大，直接参与模型训练、模型发布、产品构建变得越来越难。公司有 2500 人。我有一些直觉，但很难介入每一个细节。一个非常高效的事情是确保 Anthropic 是一个好的工作场所，人们喜欢在那里工作，每个人都认为自己是团队成员，大家合作而不是对抗。我们看到其他一些 AI 公司发展起来后，开始出现不协调，人们互相争斗。我认为我们做得非常好，即使不完美，我们凝聚了公司，让每个人都感受到使命，我们对使命是真诚的，每个人都相信其他人是出于正确的原因工作，我们是一个团队，人们不会以牺牲他人为代价来争先或背后捅刀子。我们如何做到这一点？有很多因素：我、负责日常运营的 Daniela、联合创始人、我们雇佣的其他人、我们试图创造的环境。但文化中重要的一点是，我以及其他领导者，尤其是我，必须阐明公司的宗旨、为什么做正在做的事、战略、价值观、使命以及它代表什么。当公司有 2500 人时，你无法逐一进行。你必须通过写作或向全公司讲话。这就是为什么我每两周站在全公司面前讲一个小时。我写一个叫 DVQ 的东西，Dario Vision Quest。不是我命名的，是它得到的名字。我试图反对，因为它听起来像是我要去抽佩奥特仙人掌，但名字还是留下来了。我每两周站在公司面前，带着一份三四页的文件，谈论三四个不同的话题，关于内部情况、我们生产的模型、产品、外部行业、与 AI 和地缘政治相关的整个世界。我非常坦诚地讲，说「这是我在想的，这是 Anthropic 领导层在想的。」然后我回答问题。这种直接联系有很大的价值，这是通过六层层级传递信息难以实现的。公司很大一部分人要么亲自来要么虚拟参加。这意味着你可以沟通很多。我做的另一件事是，我在 Slack 上有一个频道，我在那里写很多东西并大量评论。这通常是对我在公司看到的事情、人们提出的问题或内部调查中人们担忧的事情的回应，所以我会写下来。

For this particular one, I wrote it over winter break, so there was time. I was having a hard time finding the time to actually write it. But I think about this in a broader way. It relates to the culture of the company. I probably spend a third, maybe 40% of my time making sure the culture of Anthropic is good. As Anthropic has gotten larger, it's gotten harder to be directly involved in the training of the models, the launch of the models, the building of the products. It's 2,500 people. I have certain instincts, but it's very difficult to get involved in every single detail. One thing that's very leveraged is making sure Anthropic is a good place to work, people like working there, everyone thinks of themselves as team members, everyone works together instead of against each other. We've seen as some other AI companies have grown, we're starting to see decoherence and people fighting each other. I think we've done an extraordinarily good job, even if not perfect, of holding the company together, making everyone feel the mission, that we're sincere about the mission, and that everyone has faith that everyone else is working for the right reason, that we're a team, people aren't trying to get ahead at each other's expense or backstab each other. How do we make that the case? It's a lot of things: me, Daniela who runs the company day-to-day, the co-founders, the other people we hire, the environment we try to create. But an important thing in the culture is that I, and the other leaders as well but especially me, have to articulate what the company is about, why it's doing what it's doing, its strategy, its values, its mission, and what it stands for. When you get to 2,500 people, you can't do that person by person. You have to write or speak to the whole company. This is why I get up in front of the whole company every two weeks and speak for an hour. I write something called a DVQ, Dario Vision Quest. I wasn't the one who named it; that's the name it received. I tried to fight it because it made it sound like I was going off and smoking peyote, but the name stuck. I get up in front of the company every two weeks with a three or four page document and talk through three or four different topics about what's going on internally, the models we're producing, the products, the outside industry, the world as a whole as it relates to AI and geopolitically. I just go through very honestly, saying 'This is what I'm thinking and this is what Anthropic leadership is thinking.' Then I answer questions. That direct connection has a lot of value that is hard to achieve when you're passing things down the chain six levels deep. A large fraction of the company comes to attend either in person or virtually. It means you can communicate a lot. The other thing I do is I have a channel in Slack where I write a bunch of things and comment a lot. Often that's in response to things I'm seeing at the company, questions people ask, or internal surveys where people are concerned about things, so I'll write them up.

我对这些事情非常坦诚，直接说出来。目的是建立一种声誉，让公司了解真实情况。实事求是，承认问题，避免那种企业腔调，那种在公开场合常常必要的防御性沟通，因为世界很大，充满恶意解读的人。但如果你有一群你信任的人——我们努力招聘我们信任的人——那么你就可以完全不加过滤地交流。我认为这是公司的一大优势。它让工作环境更好，让人们发挥出超越个体的力量。它增加了我们完成使命的可能性，因为每个人对使命都有共识，大家都在讨论如何最好地实现使命。

I'm very honest about these things. I just say them very directly. The point is to get a reputation of telling the company the truth about what's happening. To call things what they are, to acknowledge problems, to avoid the sort of corpo speak, the kind of defensive communication that often is necessary in public because the world is very large and full of people who are interpreting things in bad faith. But if you have a company of people who you trust and we try to hire people that we trust, then you can really just be entirely unfiltered. I think that's an enormous strength of the company. It makes it a better place to work. It makes people more than the sum of their parts. It increases likelihood that we accomplish the mission because everyone is on the same page about the mission. And everyone is debating and discussing how to best accomplish the mission.

嗯，代替一次外部的 Dario 愿景探索，我们有这次访谈。这次访谈有点像那样。这里是《Inquiring Minds with Dwarkesh》。谢谢你参加。

Well, in lieu of an external Dario vision quest, we have this interview. This interview is a little like that. This is Inquiring Minds with Dwarkesh. Thanks for doing it.

是的，谢谢你，Dwarkesh。

Yeah, thank you, Dwarkesh.

大家好。希望你们喜欢这一集。如果喜欢，最有帮助的事情就是分享给可能感兴趣的人。如果你在收听平台留下评分或评论，也很有帮助。如果你想赞助播客，可以联系 dwarkesh.com/advertise。否则，下期再见。

Hey everybody. I hope you enjoyed that episode. If you did, the most helpful thing you can do is just share it with other people who you think might enjoy it. It's also helpful if you leave a rating or a comment on whatever platform you're listening on. If you're interested in sponsoring the podcast, you can reach out at dwarkesh.com/advertise. Otherwise, I'll see you on the next one.