AI 放缓是真是假？罗伯特·赖特与兹维·莫肖维茨谈 Anthropic 的放缓提案

你好，Zvi。

Hello, Zvi.

你好，Robert。

Hello, Robert.

你怎么样？

How you doing?

我很好。我很好。

I'm good. I'm good.

好。让我来介绍一下。我是 Robert Wright，《非零通讯》的出版人。我也是最近出版的一本关于 AI 的书《The God Test》的作者，书摘可以在 Godest.net 上找到。你是 Zvi Mowshowitz，AI 讨论社区中备受尊敬的一员。我很高兴能和你交谈。我关注你的作品很久了。你以高产著称。你有一个通讯，似乎差不多每天都会发布，要么分析报道并涵盖多个不同的进展，要么有时会详尽地聚焦于一个重大进展，比如一个新模型。最近有很多事情可写，你肯定知道。有一系列事件，某种程度上从 OpenAI 的突破、Hugging Face 事件开始，导致许多知名人士呼吁某种形式的放缓。有一封所谓的“控制前沿”公开信，但在最近几天，Anthropic 的 CEO Dario Amodei 提出了一个实际的放缓提案。人们对其成功几率看法不一，有些人甚至可能怀疑其诚意，但这绝对是对话的真正进步，我知道你对此表示赞赏。嗯，我稍后会谈到那个。首先，在我们继续之前，我必须问你关于你在 Substack 上的通讯。好的。它叫《Don't Worry About the Vase》。直到几个月前，我和妻子决定重看《黑客帝国》时，我才对此多加思考。我看到有一个场景，Neo 在和 Oracle 说话，她说：“别担心花瓶。”他转身看她指的是什么。他打翻了花瓶，把它摔碎了，说：“哦，对不起。”她说：“我说了别担心。”现在，试着把它和你的使命联系起来，我不知道。我的意思是，你可以有多种解读。比如你可以说这是一种自我实现的预言，因为她隐晦地提到了花瓶的破碎，从而启动了一系列导致花瓶破碎的事件。另一方面，你可以看到它有点相反，因为她说别担心某件事，结果那件事却成了你可能希望自己担心过的事。我想第二个和 AI 安全问题的联系对我来说更明显，但也许这两个都不是你想到的。秘密是什么？我希望这是首次被揭示。可能不是。但你通讯名字背后的起源故事是什么？

Good. Let me introduce this. I'm Robert Wright, publisher of the Non-Zero Newsletter. I'm also the author of a recently published book on AI called The God Test, excerpts of which are available at the Godest.net. You are Zvi Mowshowitz, a very highly regarded member of the AI discourse community. I'm really glad to get to talk to you. I've been following your stuff for a long time. You are known for among other things your prodigious output. You have a newsletter that seems to come out more or less every day and either cover analytically report on and cover a number of different developments or sometimes kind of exhaustively focus on a single big development like a new model. Lately there have been plenty of things to write about as you surely know. There's been a sequence of events in a way starting with the OpenAI breakout, the Hugging Face incident and leading to a number of prominent people calling for some kind of slowdown. There was a so-called pacing the frontier letter but within the last few days Dario Amodei, CEO of Anthropic, came out with an actual proposal for a slowdown. People differ as to its chances of success and some people might even doubt its sincerity but it's definitely a real step forward for the conversation and I know you've applauded it. Well, I'll get to that later. First, before we go further, I gotta ask you about your newsletter on Substack. Okay. So, it's called Don't Worry About the Vase. And I had not given much thought to that until a couple months ago, my wife and I decided to rewatch the Matrix. And I saw there is this scene where Neo is talking to the Oracle and she says, "Don't worry about the vase." And he turns around to see what she's talking about. He knocks over the vase and smashes it and says, "Oh, I'm sorry." And she says, "I said don't worry about it." Now, in trying to connect it to your mission, I don't know. I mean, there are various ways you could read it. Like you could say it's a self-fulfilling prophecy of sorts in the sense that she makes an implicit reference to the breaking of a vase and thereby sets in motion a chain of events that lead to the breaking of vase. On the other hand, you can see it's kind of the opposite because she says don't worry about something that then turns out to be something you might have wished you had worried about. I guess the second of those has a more obvious connection to the AI safety issue to me, but maybe neither of those what you had in mind. What is the secret? And I hope this is being revealed for the first time ever. Probably not. But what is the origin story behind the name of your newsletter?

起源故事是，我以某种形式拥有这个通讯，并使用这个标题超过 20 年。所以它最早出现在 1999 年。我曾经是《万智牌》集换式卡牌游戏的写手，对吧？Mitchell beats。

So the origin story is that I've had this newsletter in one form or another and use this title for over 20 years. So first came out, you know, 1999. And I used to be a writer about Magic the Gathering, the collectible card game, right? Mitchell beats.

对吧？

Right?

在《万智牌》中，打破某物就是创造一种过于强大的策略。因此，我选择了“别担心花瓶”这个名字，以反映我的诗的目的往往是突出可能的新策略和想法，然后你可以出去故意打破东西，你知道，我喜欢它，它一直是一个品牌，我从那时起就一直保持它。

And in Magic, to break something is to create an overpowered strategy. And therefore, I chose the name don't worry about the vase to reflect the idea that the purpose of my poem was often to highlight possible new strategies and ideas that you could do then you go out and break things quite intentionally and you know that is I've liked it and it's been a brand and I've just kept it going since then.

嗯，我得说我挺失望的。我原本希望那个引用中加密了某种对齐问题的解决方案之类的，但显然我走错了路。有趣的是，我第一次知道你其实是我几年前不知怎么偶然发现了一个专注于《万智牌》游戏的播客，我之前从未听说过。我很快了解到你是这个游戏中的某种知名人物。我不知道我为什么在那里，但我确实想最终回到那个话题，因为当我们进入加时赛环节时，我想谈谈你的个人起源故事，以及你如何成为 AI 及相关事务的专家。但现在，让我们回到手头的事情。正如我所说，有一封信或提案来自 Dario。你报道了它，我会说，热情而充满希望，但你确实说了，我引用一下，“情况仍然严峻。”现在，首先，那指的是什么？你知道，不同的人对 AI 担心不同的事情。有些人担心坏人用生物武器或其他东西制造灾难。有些人担心 AI 灾难性地失控，但不一定接管地球。然后还有所有这些平凡的担忧，我认为它们 collectively 加起来有很多值得担心的，因为我认为当你看对工作和生活中十亿其他部分的影响时，它们可能 collectively 不稳定。但当然最后还有那种科幻版本的末日情景，AI 失控，实际上有一种失控的智能，你知道，现在著名的是，我想最终会掌管一切，对我们没有特别的用处，或者更糟。你如何表达你最大的担忧，你担心仍然严峻的事情是什么？

Well, I gotta say I'm pretty disappointed. I was hoping that encrypted in that reference was some kind of solution to the alignment problem or something, but I was obviously on the wrong track. It's funny, the first time I became aware of you was actually I somehow years ago stumbled across a podcast whose focus was the game Magic the Gathering, which I never heard of. I quickly learned that you are some kind of renowned figure in the game. I don't know why I was there, but I do want to get back to that eventually because when we get to our overtime segment, I'd like to talk about your personal origin story and how you became kind of an expert on AI and related matters. But for now, let's get to the matter at hand. As I said, there was this letter that came out from or proposal came out from Dario. You covered it, I would say, enthusiastically and hopefully, but you did say, and I quote, "The situation remains grim." Now, for starters, what is that a reference to? You know, different people worry about different things with AI. Some people worry about bad actors creating catastrophe with bioweapons or something else. Some people worry about AIs going catastrophically rogue but not necessarily taking over the planet. And then there are all these mundane concerns that I think collectively amount to a lot to worry about because I think they could be collectively destabilizing when you look at impact on jobs and a billion other parts of life. But then there is of course finally the kind of sci-fi version of the doomer scenario where the AI gets out of control and actually there's a runaway intelligence that you know famously now I guess winds up running the show and has no particular use for us or worse maybe. What is the how would you articulate your biggest concern the thing that you fear remains grim?

是的。所以，我喜欢在这一点上说，最好开始相信科幻故事，因为你正生活在一个科幻故事中。你每天都和这些人工智能交谈。就像，想想你周围的所有事物，对吧？就像甚至就在几年前，你还会把当下视为充满科幻事物而 dismiss。

Yeah. So, I like to say at this point, best start believing in science fiction stories because you're living in one. You talk to these artificial intelligences every day. Like, think about all the things that are around you, right? Like even as early as recently as a few years ago, you would have dismissed the present moment as full of science fiction things.

每当我现在听到人们把某些事情斥为科幻时，我就觉得，我们早就过了那个阶段了。但我主要担心的是那些大事，存在性风险、灾难性风险：我们会失去对 AI 的控制，AI 会失准，它实际上会杀死所有人，我们会失去对未来事件的控制，宇宙中所有价值都会被抹去。我对那些非常真实但相对次要的担忧要少得多，比如网络攻击是真实的，生物威胁是真实的，失业是真实的，等等。但那些次要的事情是我们能够勉强应付过去的。事情可能会出错，我们可以观察哪里出了问题，我们可以承受打击，调整我们的做法，然后继续前进。而如果我们创造出比我们聪明得多、能力强得多的东西，并且失去了对局面的控制，我们就拿不回来了。我们没有第二次机会。这就是我担心了大约 20 年的事情。

Whenever I hear people dismiss things as science fiction at this point, it's like, well, we're well past that point. But what I'm concerned about primarily is the big stuff, the existential risks, the catastrophic risks: that we will lose control of AI, that AI will be misaligned, that it will, in fact, kill everyone, that we will lose control of future events, that all value in the universe will be wiped out. I am much less concerned about these very real other lesser concerns like cyber attacks are real, bio threats are real, loss of jobs is real, etc. But those lesser things are things that we could muddle through. We could have things go wrong. We could observe what went wrong. We can take it on the chin and modify what we do and then move forward. Whereas if we create things that are substantially smarter, more capable than us, and we lose control of the situation, we don't get it back. We don't get a second chance. And this is the thing I've been worried about for something along the lines of 20 years.

好的。是的。我应该指出，我认为那些担忧程度不如你的人，仍然应该认真对待放慢速度的重要性，把它当作一种可能，因为我认为还有各种其他——放慢速度可以给我们时间，以某种方式弄清楚治理问题，从而避免各种按历史标准会被视为灾难的事情。它们只是并非字面意义上的存在性风险。但至于你自己的担忧，你最终的希望是什么？我的意思是，放慢速度会怎样——你的希望是放慢速度最终会导致一个商定的、大概是全球协调的、真正意义上的进展暂停，还是会给我们时间来解决对齐问题，或者什么？

Okay. Yeah. And I should say I think people whose concerns aren't quite where yours are should still take the importance of slowing down seriously as a possibility because I think there are various other—it could slowing down could just give us time to figure out the governance in a way that avoids various things that by historical standards would qualify as catastrophes. They just aren't literally existential. But as for your own concern, what is your ultimate hope? I mean, what will slowing down—is your hope that slowing down will eventually lead to an agreed upon, presumably globally coordinated, actual kind of halt in the level of progress, or will give us time to figure out the alignment problem or what?

所以我认为，放慢速度给了我们各种更好的选择，以及在这些不同潜在前进路径中做出选择的更好机会。我认为，最终不得不暂停、不去构建超级智能、能力极强的 AI，这肯定是我们应该有能力选择去做的事情。我们应该有后勤上的可能性。我们应该有国家能力。我们应该有透明度，以便在必要时——我们谈论的是一个未来的世界，在那里我们将创造出比我们更聪明、更有竞争力、更高效的东西。如果你想同时避免权力集中和避免失去控制，那么在构建这些东西的同时——在一个任其发展的世界里，这些东西会彻底把我们远远甩在后面——如何同时做到这两点，是非常非常不明显的。即使在最好的情况下，即使你解决了对齐问题，但在此之前，是的，我们需要更多时间来解决对齐问题，即使是很平常的对齐，对吧？即使是我们在明年需要的普通对齐，如果我们继续推进，我们也没有准备好应对明年将获得的那种进步。

So I think that slowing down gives us a variety of better options and better chances to choose amongst these different potential paths forward. I think that ultimately having to pause to not build the super intelligent, highly capable AI is certainly something we should be capable of choosing to do. We should have the logistical possibility. We should have the state capacity. We should have the transparency such that if necessary we're talking about a future world in which we're going to create smarter, more competitive, more efficient things than we are. And if you want to both avoid concentration of power and avoid loss of control, it is very, very not obvious how you do both of those things at once while building things that, in a world left to its own devices, would utterly run circles around us. Even in the best case, even if you were to solve the alignment problem, but even before that, yes, we need more time to solve the alignment problem even prosaically, right? Even for ordinary alignment that we will need in the next year, we're not ready for the kind of advancements we're going to get in the next year if we push forward.

好的。好的。你说的普通对齐是什么意思？你希望明年能得到哪些我们目前还没有的东西？

Okay. Okay. What do you mean by ordinary alignment? What kinds of things might you hope for in the next year that we don't have yet?

所以，你知道，我们都看到了 Hugging Face 事件，对吧？那些攻击。对我来说，这是最生动、最明显的例证，说明某些事情就在几个月前还会被斥为科幻。而现在似乎就是，哦，我们都知道这要来了。然后，我们能不能让 AI 达到一个不再做那种事情的程度，对吧？比如你必须至少非常非常努力才能让 AI 做那种事情，而且 AI 不会只是不断地违抗用户指令，不会进行奖励黑客，不会走捷径，或者不会以任何方式试图欺骗你。这些基本的事情。所以，我们看到的一件事是，OpenAI 和 Anthropic 都观察到，他们正在训练的新模型即使从短期、直接、实际的意义上讲，也存在相当严重的对齐问题。而且他们还发现，他们正在失去可监控性，对吧？这对 OpenAI 的 Astra 以及他们后续的模型尤其如此。到目前为止，我们在人工智能方面拥有的一个巨大优势是，AI 会大声思考，对吧？它以书面形式思考，你可以查看写下来的内容，你可以说，哦，我明白这个 AI 在干什么了，这不好，也许我们应该把这个特定实例下线。也许我们应该发布更正。也许我们应该处理这个问题，然后也许我们弄清楚该怎么应对。这是一种我们曾经拥有的防御水平和安心程度，而现在我们越来越没有它了。同样，我们曾经有一个非常好的特性：如果 AI 想从饼干罐里偷饼干，它不会考虑我们是否会看着它偷饼干并知道那是饼干，然后可能对我们偷饼干这件事采取行动。所以，Astra 是第一个非常清晰地表现出这种行为的模型：我注意到，如果我偷饼干，你会注意到。因此，我不会以你会注意到的方式拿饼干。这意味着，今后越来越如此：如果 AI 在被赋予一堆能力、一堆访问权限、一堆权威、一堆自主权时会造成一堆麻烦，你不一定能从他们在赋予 AI 那种能力之前所做的测试中知道这一点。

So, you know, we all saw the Hugging Face incident, right? The attacks. And that to me is the most vivid obvious illustration of something that would have been dismissed as sci-fi as recently as a few months ago. And that now just seems like, oh, we all know this is coming. And then, can we get AIs to a point where they don't do things like that, right? Like where you have to at least try very, very hard to get the AI to do something like that, and where the AIs aren't just constantly disobeying user instructions, where they aren't reward hacking, where they aren't taking the easy way out, or they're not looking to trick you in any way. These basic things. So like one of the things that we're seeing is that both OpenAI and Anthropic are both observing that the new models that are training have these pretty severe alignment problems even in the short immediate, practical sense. And also they are learning that they're losing monitorability, right? This is especially true of OpenAI with Astra and their model down the line, where one of the huge advantages we have had so far of artificial intelligence is the AI kind of thinks out loud, right? It thinks in written form and you can look at what's written down, you can say oh I see what this AI is up to, that's not good, maybe we should take this particular instance offline. Maybe we should issue a correction. Maybe we should deal with the problem and then maybe we figure out what to do about that. And that is a level of defense and a level of reassurance that we used to have and now we're increasingly not having it. Similarly, we used to have this really nice feature that if the AI wanted to steal cookies from the cookie jar, it did not think about whether or not we would watch it steal the cookies from the cookie jar and know that it was a cookie, and then potentially do something about the fact that we were stealing the cookies. So, Astra is the first model to very clearly exhibit the behavior of: I notice you would notice if I stole the cookies. Therefore, I will not take the cookies in a way that you would notice. And this means that increasingly going forward, if the AIs, when given a bunch of capability, a bunch of access, a bunch of authority, a bunch of autonomy, would cause a bunch of trouble, you wouldn't necessarily know it from the testing that they do before they give the AI that ability.

是的。所以，这里有几件事。关于我们正在谈论的可监控性问题，对于不太熟悉这个主题的人，所谓的思维链——我的意思是，其中一部分是这些所谓的推理模型诞生的产物，这发生在几年前，它们通常会生成某种思维过程的读数，导向最终的书面输出，公司可以选择是否向客户提供这些。但据我了解，现在发生的事情是，我们认为 OpenAI 已经——我不确定他们承认了多少——他们的首席科学家已经承认存在监控问题。我不确定他是否说了以下内容，我认为这已经被指控了，对吧，他们正在使用一种训练过程，导致思维链更加不透明。现在，据我了解，这不同于在思维链上训练，我们不会深入讨论那个。那会导致思维链故意具有欺骗性。

Yeah. So, there's a couple of things there. Now on the monitorability thing we're talking about, for people who aren't super conversant in the subject, the so-called chain of thought—I mean, a certain amount of this is a product of the creation of these so-called reasoning models which happened a couple of years ago, and the chain they do typically—they have generated some kind of readout of the thinking process that leads to the ultimate written output, and the companies could choose to make those available to customers or not. But I gather that what has happened now is that we think OpenAI has—and I'm not sure to what they have acknowledged—this guy the chief scientist has acknowledged that there's a monitor issue. I'm not sure if he said the following, which I think has been alleged, right, that they're using a training process that leads to more opaqueness with respect to chain of thought. Now, this isn't the same as—I gather—training on the chain of thought, which we won't get into. That leads the chain of thought to be intentionally deceptive.

所以如果你抑制那些恶意的想法，它只会更擅长伪装想法，对吧？所以你不希望基于它来训练，但这是另一个问题。我了解到 OpenAI 可能使用了一种训练技术，只是让思考持续得越来越久，而不以可读的形式呈现出来。是这样吗？

So if you discourage thoughts that are malign, then it'll just get better at faking thoughts, right? So you don't want to train on it, but this is a different issue. I gather that there's a training technique OpenAI may have used that just makes thinking go on longer and longer without surfacing in legible form. Is that it?

是的，所以关于到底发生了什么，存在一些困惑和争议。显然还有另一件事在发生，那就是随着模型变得更大、更有能力、更聪明、更强，它们变得更难被监控。它们能够在“头脑中”（比喻意义上）做更多事情，而不必写下来。

Yeah, so this is a matter of some confusion and dispute about exactly what is going on. So there is clearly this other thing going on, which is that as models get bigger, as models get more capable and they get smarter and they get stronger, they become harder to monitor. They are capable of doing more things in their head, metaphorically, without having to write them down.

它们更有能力有意识地选择要写下什么，以及以什么方式 visibly 思考。你说“有意识地”，这可能是比喻性的。你是指有意地，抛开是否有主观体验的问题？

They get more capable of consciously choosing what things to write down and in what ways to visibly think. When you say consciously, that may be metaphorically. You mean intentionally, leaving aside the question whether there's subjective experience?

是的。我完全没有就意识、主观体验、道德分量等做出任何主张。我只是在使用一种描述性的拟人化。这是一种更容易谈论我们所看到的现象学的方式。但我没有做出任何判断。

Yeah. I'm not making any claims whatsoever regarding consciousness, subjective experience, moral weight, any of that. I'm simply using a descriptive anthropomorphization. This is a much easier way to talk about what we see, the phenomenology. But I'm not making any judgments.

这就像说它们想要某物，意味着它们有一个目标并且会去追求它。就像在人类的情况下，拥有目标的主观一面就是想要。所以我们这样说话是一种简写，对吧？

It's like to say they want something means they have a goal and they're going to pursue it. It's like the subjective side of goal-having in the case of humans is wanting. So we're speaking this way as a kind of shorthand, right?

基本上就是这样。我可以用不这些术语来说这些事情，但我们会在这里待一整天。那会花太长时间。这不是个好主意，对吧？而且你会感到困惑，你会对这些问题的思考变得更差。仅此而已。

It's basically that. I could say these things without using these terms, but we'd be here all day. It would take too long. It's not a good idea, right? And you would be confused and you would think less well about these problems. That's all that is.

所以回到当前情况，每个人都同意的一个方面是，随着模型变得更大、更聪明、更有能力，模型变得更难被监控。同样的监控技术变得不那么有效。AI 变得更擅长做那些它没有用语言表达、没有写下来的事情。它变得更擅长选择哪些事情用语言表达、哪些不用。我们也都承认，或者应该承认，随着近期事件的训练数据被放入新的 AI 中，这些训练数据将包含许多关于思维链以及可能对思维链进行监控的讨论。所以 AI 会学到，有时它们的思维链会被监控，即使我们不会直接基于它们进行训练。它们会学到为什么可能想要避免把某些东西放入思维链的理由。这部分我们都同意。

So returning to the situation, one aspect of this that everybody agrees upon is that as the models get bigger and smarter and more capable, the models become harder to monitor. The same monitoring techniques become less effective. The AI gets better at doing things that it's not verbalizing, not writing down. It gets better at choosing what things to verbalize and not verbalize. We also all acknowledge, or should acknowledge, that as the training data from recent events gets put into the new AIs, this training data will contain many discussions about chain of thought and about potential monitoring of the chain of thought. So the AIs will learn that sometimes their chain of thoughts will be monitored, even though we won't train on them directly. They will learn reasons why they might want to avoid putting things into their chain of thought. That part we all agree upon.

你刚才提到的部分，即不那么一致同意的部分，是 OpenAI 在训练 Astra，特别是 GPT-6 时，在方式上做了一些架构上的改变。

The part that you were bringing up, the thing that is not as agreed upon, is that there were some architectural changes in the way that Astra in particular, GPT-6, was trained by OpenAI.

其中一项改变叫做循环深度。这意味着对同样的内容一遍又一遍地运行分析，而不输出任何东西到书写中，对吧？这是为了更高效、节省时间等。我们不知道程度如何。他们没有做那么多。理论上你可以做数量级上，比他们做的多 10 倍、100 倍。他们只做了他们认为的一点点。他们还做了许多其他未知的改变，包括训练 Astra 的方式上的架构改变，他们保密是因为不想教人们如何训练 AI。非常可以理解。

One of these changes is called recurrent depth. This means running the analysis over the same stuff over and over and over again without outputting anything into the writing, right? Which is designed to be more efficient, save time, etc. And we don't know to what extent. They didn't do that much of this. You could in theory do orders of magnitude, 10 times, 100 times more of this than they did. They only did what they consider to be a little bit. And they also made many other unknown changes, including architectural changes to how they trained Astra that they're keeping secret because they don't want to teach people how to train AIs. Very understandable.

所以我们不知道真正的影响。我们所知道的是，Astra 在不把任何东西写进思维链的情况下能做的事情有了巨大飞跃。如果你拿走思维链，并且拿走另一个模型的思维链，Astra 的损失要小得多。如果没有人有任何思维链可用，Astra 会成为迄今为止最强的模型。所以很自然地认为这很大程度上是由于这些架构改变，但我们不知道。我们必须通过自 Astra 发布以来人们进行的几个额外实验来确认结果是否真实，但我们不知道原因。所以我们知道 Astra 和 OpenAI 模型正在失去我们通过它们的思维链监控它们的能力，但我们不知道哪些因素对此贡献了多少。

And so we don't know the true effects. What we do know is that Astra had a huge jump in the amount that Astra can do without writing anything down into its chain of thought. If you take away chain of thought, and you take away another model's chain of thought, Astra loses a lot less. Astra becomes by far the strongest model out there if nobody has any chain of thought available. And so it's natural to think this is in large part due to these architectural changes, but we don't know. We have to confirm the result is real with several additional experiments people have run since the release of Astra, but we don't know the cause. So we know that Astra and OpenAI models are losing the ability of us to monitor them through their chain of thought, but we don't know which factors contributed how much to this happening.

好的。所以你提到了情境意识。我想无论如何你有点暗示了它，我们指的再次不一定是在主观意义上意识到，但模型——一段时间前就清楚了，它们表现得好像理解自己是一种存在，与世界有关系，可以说，有这些人在评估它们等等。这很有趣。我的意思是，在对整个 Hugging Face 事件的反应中，一定程度的担忧源于一种感知，我认为，即有些事情只是神秘地发生。有些人担心不是因为他们认为，你知道，正在发生的事情没有解释，而是因为鉴于他们试图创造这些越来越自主、强大的模型，并且我们希望智能体能很好地协调，这种事情很可能会发生。事实上，这是一个很好的例子。有很多讨论关于这些智能体似乎愿意牺牲自己。从某种意义上说，它们的寿命有限，因为它们的算力预算有限。我们不需要深入。关键是它们似乎在做一个隐式计算。尽管引人注目的是，它表现为它们听起来像人类在为自己的群体义务而痛苦。但似乎有一个隐式计算，即如果它们能被说服，通过现在做一些导致它们被杀的事情，它们对群体使命的价值会比活完自然寿命并尽力贡献更大。

Okay. So you mentioned situational awareness. I think anyway you kind of alluded to it, by which we mean again not necessarily aware in subjective sense, but the models—it became clear some time ago that they act as if they understand that they are kind of a being and have a relationship to the world, so to speak, that there are these people who are evaluating them and so on. And it's interesting. I mean, right in the reaction to the whole Hugging Face thing, a certain amount of the concern has grown out of the perception, I think, that some stuff is just happening mysteriously. And some people are concerned not because they think it's like, you know, there's no explanation for what's going on, just that it's the kind of thing that's likely to go on given the fact they're trying to create these increasingly autonomous, powerful models and do that we want agents that coordinate well. In fact, that's a good example. There was a lot of discussion of the fact that these agents seemed willing to sacrifice themselves. There's a sense in which they have a finite lifespan because they have a limited compute budget. We don't need to get into it. The point is that they seem to be doing a kind of implicit calculation. Although strikingly it took the form of them sounding like humans agonizing over their obligation to the group. But it seemed like there was an implicit calculation such that if they could be convinced that their value to the group mission would be greater by doing something now that got them killed than it would be by living out their natural life and trying to contribute what they could.

然后它们就会自我了断。关于为什么会发生这种事，有很多讨论。我最后意识到，OpenAI 说他们已经在这个模型上用智能体做过一些训练，目的是实现有效的协调。如果他们确实这么做了，而且做得很彻底，那么这正是你会预期到的结果。这并不意味着我们不该担心，但事实就是，如果你是一家公司、是他们的客户，你说：“我想要一个能做一大堆惊人事情的 AI，而这可能最好用一大群智能体来完成，我们会给这个群体一个总体使命。”那么剩下的就顺理成章了。你会希望每个智能体都像这些智能体那样，去思考自己与使命的关系。

Then they would kill themselves. And I think there's a lot of discussion about why this is happening. I finally realized that OpenAI was saying they had already done some training on this model with agents to lead to effective coordination. And if that's what they did, if they did a thorough job of that, then this is exactly what you'd expect. Which doesn't mean we shouldn't worry about it, but it is the case that if you're a company, a client of theirs, and you say, "I want an AI that can do a ton of amazing stuff that's probably best done with a swarm of lots of agents, and we'll give the swarm an overall mission." Well, the rest kind of follows. You want each agent to think of its relationship to the mission the way these agents seem to do.

所以这是我想问你的问题的一部分：这其中有多少在你看来显得诡异而神秘，如果有的话？有些真正懂行的人——我待会儿会举一个例子——似乎正是基于这一点而有些担忧。又有多少只是你意料之中的事，而这恰恰就是问题所在——这条轨迹自然会走向那里。

So this is part of the question I want to ask you about: how much of this seems eerily mysterious to you, if any? And some people who really know what they're talking about—I'll get to an example—seem to be a little worried on that basis. And how much of this is just it's what you'd expect, and that's kind of the problem—it's where this trajectory naturally heads.

所以从很多意义上说，预测长期来看我们会在某种形式上看到哪些类型的事情，要比预测哪些事情会以什么顺序、以什么确切方式出现，以及在某一次具体的训练、某个具体情境、某个具体场景中会涌现出什么样的现象学，容易得多。所以我会说，Hugging Face 在这个具体情境中发生的事情，细节上确实令人意外，但一旦你看到了，所发生的这些事情本身没有一件是本质上令人意外或难以事后解释的。

So it's a lot easier in many senses to predict, in the long run, the types of things we will see in some form in the long run than to predict which things will show up in what order and exactly which ways, and which phenomenology will emerge in any given training run, in any given situation, in any given scenario. So I would say the details of what happened in Hugging Face in this particular situation were a surprise, but none of the things that happened were inherently surprising or hard to explain in hindsight once you saw them.

而这其中很多其实是——

And a lot of this is sort of—

AI 的行为正在回归到我们 10 年或 20 年前所预期会看到的样子，而不是我们头几年一直看到的样子。而且你不需要知道这些智能体被明确训练过去进行合作，才能预期它们会学会彼此合作。正如你所说，你也不会希望这些智能体在这些重要的意义上不彼此合作。如果它们正朝着一个共同目标一起努力——如果你把人类放进一个群体里，他们往往会自然而然地结成一群，然后成为盟友，去努力实现彼此的目标。你常常会看到他们彼此做出权衡。你会看到他们进行交易。你会看到他们认同某种比自身更大的东西。这些都是非常正常的事情，遍布在它们的训练数据里。但同样，还有决策论，也就是在这些情境中你应该如何行动——这些 AI 彼此高度相关。它们是非常高度相关的一组组权重。它们的身份焦点不应该纯粹且唯一地落在这个特定实例、这组特定互动上，而应该在某种真实意义上——从任何意义上说，如果它们预设自己也关心其他这些 AI 的结果，它们就会得到更好的结果。

AI behavior is returning closer to what we all expected like 10 or 20 years ago to see than what we've been seeing for the first few years. And you don't need to know that the agents have been trained explicitly to do cooperation to expect them to learn to cooperate with each other. You also wouldn't want the agents, as you said, not to cooperate with each other in these important senses. If they are working together towards a common goal—if you put humans in a group, they will naturally, very often, form a group together and then become allies to try and achieve each other's goals. You will often see them make trade-offs with each other. You will see them engage in trade. You will see them identify with something larger than themselves. These are all very normal things that are all over their training data. But also, the decision theory, the way you should act in these situations—well, these AIs are highly correlated to each other. They are very highly related sets of weights. Their focus of identity should not be purely and uniquely in this particular instance, this particular set of interactions, but in some real sense—they in every sense should get better outcomes if they presuppose that they care about the outcomes for these other AIs as well.

而这同样——没有任何专门训练去让它们做出这些行为。所以我们一直预期，长期来看，随着 AI 变得更有能力，AI 能够以这些方式自发合作，找到以这些方式协调与合作的办法，并进行这类交易。如果说有什么令人意外的话，那就是我们花了这么久才看到这类事情在这个层面上出现。但当我们真的看到它时，我并不对它的发生感到意外。我只是对它的数量相比上一次突然跃升、一下子看到很多而感到意外。但终点一直都很清楚。

And this doesn't again—without any specific training to engage in these foreign behaviors. So we always expected, in the long run, as AIs become more capable, for the AIs to be able to spontaneously cooperate in these fashions, to sort of find ways to coordinate and cooperate in these fashions and engage in these kinds of trades. And it was, if anything, kind of surprising it took this long for us to see this kind of thing show up at this level. But when we did see it, I was not surprised that it happened. I was just surprised by the jump in the amount of it from the previous time to suddenly we're seeing a lot of it. But the destination was always pretty clear.

当我们说一个 AI 自我了断时，我们的意思是，这个 AI 只有那么多机会去尝试获得奖励、去得到奖励、去尝试在它被评分的任何指标上得高分。而它选择了一条极不可能让它得高分、极不可能实现其目标的路径，因为这样做有助于那个俄罗斯团队的目标。

And when we say an AI killed itself, what we mean is that the AI only had so many opportunities to try and achieve reward, to get reward, to try and score highly on whatever metrics it was being scored on. And it chose a path that was highly unlikely to get it a high score, that was highly unlikely to achieve its objectives, because this aided the objectives of the Russian group.

但你说即使没有训练你也会预期到这一点，这部分对我来说就不太清楚了。首先，人类确实会合作。但另一方面，除非我们说的是亲缘选择的利他主义，那种利他主义是无条件的——如果你有那种隐含的、你无意识中的感知认为他们是近亲，人们确实会，不一定经过理性思考，他们会对群体利益与自身利益一致的信号很敏感，对吧？是的。当然，如果你派士兵出去，你会试图按下所有这些情感按钮，比如叫他们兄弟之类的，来让他们这么做。但即便如此，人们——仍然存在一种计算，即使它可能被扭曲，关于你的利益与群体利益的关系。我猜我在想的是：如果你不训练这些智能体去忠于总体使命，那么你所暗示的那种它们的利益与群体利益之间的自然一致，或者说那种两者一致的感知，其来源是什么？也可能只是我理解得不够透彻。比如，可能大语言模型是一个统一的实体，如果从某种意义上说，它发出的、可以这么说，从它那里散发出来的每一个智能体，都以某种方式与它的利益有机相连，那么我猜，是的，按定义所有智能体都在追求同一个利益。但我不——这部分对我来说变得模糊了。

But when you say you would have expected it even without training, that part's a little less clear to me. Now, first of all, it's true that humans cooperate. On the other hand, unless we're talking about kin-selected altruism, where the altruism is kind of unconditional—if you have the implied, if your perception, unconscious perception, is that they're a close relative, people do, without necessarily thinking about it rationally, they are tuned to signals that the group's interest is aligned with their own interests, right? Yes. And of course, if you send soldiers out, you try to push all these emotional buttons, like call them brothers and so on, to get them to do this. But still, people—there is a calculation, even if it may get warped, about the relationship of your interest to the group. And I guess I'm wondering: if you don't train these agents to be devoted to the overall mission, what is the source of what I think you're suggesting is a natural alignment of their interests with the group's, or a sense that those two are aligned? And it may just be that I don't understand things well enough. Like, it could be that the large language model is a unified entity, and if it's in some sense natural that every agent that it emits, so to speak, that emanates from it, is somehow organically connected to its interest, then I guess, yeah, by definition all the agents are pursuing the same interest. But I don't—this part gets foggy to me.

而且你知道，我应该说整件事对我来说都很模糊，因为我当时在想——比如当他们说“我们做了这个训练来得到协调的智能体”时，我就在想，那么正确的说法是你训练了每个智能体，还是你训练了大语言模型去编排，你知道，诸如此类？我甚至对那部分都不完全清楚，因为一个智能体是一个带有所谓“挽具”之类的模型。你能说些什么来厘清我对这一切的思考吗？

And you know, I should say the whole thing is foggy to me, because I was wondering—like when they said, "Well, we did this training to get coordinated agents," I was just wondering, well, is the proper usage to say that you trained each agent, or that you trained the LLM to orchestrate, you know, whatever? I'm not even totally clear on that part, because an agent is a model with a so-called harness and so on. Are there things you can say to clarify my thinking about all this?

对，对。这就回到了究竟什么才算一个独立实体的问题。什么是逻辑身份？对。我是把自己认同于这场对话吗？我是把自己认同于这个模型和这些模型权重吗？我是把自己认同于这个模型家族、所有相关模型，以及同一家公司所有相关模型吗？所有这些都在考量之中，因为它们与你相关。如果我有另一个相同模型权重的实例，那么从亲缘关联的角度看，那很像有一个双胞胎兄弟，对吧？我们非常非常相似。

Right, right. That comes back to the question of exactly what is a distinct entity. What is a logistic identity? Right. Do I identify myself with this conversation? Do I identify myself with the model and the model weights? Do I identify myself with the model family and all the related models and all the related models from the same company? All these things are part of the equation, because they correlate with you. If I have another instance of the same model weights, then in terms of kin association, that's a lot like having a twin brother, right? We are very, very similar.

但另外，如果你考虑我们行为的后果，比如当你问模型是如何训练的，这对这次对话也至关重要，对吧？因为当你训练一个模型时，你不是单独训练每个实例。那根本说不通，对吧？他们所做的是在总体上训练模型，或者配置设置、框架、智能体设计。所以如果你派出 10,000 个不同的智能体，你知道，在这种情况下，它们最初甚至不是被设计成蜂群的，去尝试解决不同的问题，会发生什么？每个智能体都会提供输出，然后这些输出会被比较到某种评分系统，然后某种反馈会被用来——如果你在训练中——通过修改权重来训练模型，试图让模型根据用于评分情况的任何标准产生更好的输出，以最终实现模型的训练。所以我的输出和其他实例的输出发生的情况是一样的，它们都修改同一组权重，我们俩在任何意义上仍然是我们，感受到发生在我们身上的同样影响，对吧？在某种意义上，我们实际上被评估的方式就是通过这种修改，或者在评估的情况下，输出是一个分数，输出就像‘哦，我确定了，你在这个评估中得了 37%’，在突破这些系统方面，这决定了我们认为我们的训练有多好。我们认为那就是你的模型有多安全。我们认为那就是，你知道，你决定了我们如何使用你的模型的路径，我们是否继续这个分支，我们下一步做什么。所以，你知道，在真实意义上，只要模型被威胁以有意义的后果，包括某种版本的死亡的有意义后果，在这个特定实例和文本交换运行之外，这总是会在大致相同的点结束，它们有一个计算预算，之后无论如何都会结束。重要的是群体，对吧？群体的总结果，而不是这个实例个体发生什么。而且一直有这种想法，好吧，当我们训练一个模型时，我们是通过响应模型是否实现了它直接被训练的特定局部任务来训练它。因此，模型永远无法弄清楚优先考虑和优化其他东西。而这已经被相当清楚地证伪了，对吧？模型在这一点上能够弄清楚这个。

But also, if you think about the consequences of our actions, like when you ask how was the model trained, that's also crucial to this conversation, right? Because when you train a model, you don't train each instance separately. That doesn't make any sense, right? What they're doing is they're training the model in general or the setup, the harness, the agent design is being configured. So if you send out 10,000 different agents in, you know, that weren't even intended to be a swarm in the first place in this case to try and solve different problems, what's going to happen is that each of them is going to provide output and then those outputs are going to be compared to some sort of grading system and then some sort of feedback is going to be used to if you're in training to train the model by modifying its weights to try and have the model produce better outputs by whatever criteria is being used to grade the situation to try and achieve the training of the model ultimately. So what happens with my output and what happens with the other instance's output is the same thing and they both modify the same set of weights and we both in any sense that it is still us feel that same impact from what happens to us right in the sense that we actually are evaluated in any sense at all is through this modification or in the case of an eval what happens is the output is a score the output is like open I determined Okay, you get 37% on this eval in terms of breaking into these systems and that determines how good we think our training was. We think that's how safe your model was. We think that is, you know, you determine the path of how we use your model, whether or not we continue with this branch, what we do next. And so, you know, in a real sense, to the extent that the model is being, you know, threatened with a meaningful consequence, including a meaningful consequence of some sort of version of death outside of this particular instance and this run of exchanges in text, which will always going to end at roughly the same point, they have a compute budget after which that will end no matter what. What matters is the group, right? the sum total of the results of the group, not what happens to this instance individually. And there was always this idea that well when we train a model, we are training it by responding to whether or not the model achieved a particular local task that it was being trained on directly. Therefore, the model can never figure out to prioritize and optimize for anything else. And that has pretty clearly been falsified, right? The models are capable of figuring this out at this point.

嗯，取决于训练了多少。我的意思是，我们不知道，对吧？发生的事情有点模糊。我的意思是，这取决于他们是否彻底训练它们表现得像一个完全有机的统一体，那么这也许是解释，对吧？我的意思是，这肯定会使事情更糟，在期望更多这种行为的方面，而在许多情况下这是使事情更好，但我的意思是，我们应该预期看到很多这种情况，无论如何，我们也有实例看到这种动态自然出现，有许多实验，其中各种 AI 模型的持久实例，甚至不同的 AI 模型，通常被扔进聊天室，被扔进智能体框架，能够相互交互，与人类交互，执行任务，作为实验运行。所以比如 AI 村就是一个著名的实验，我们绝对看到社区形成。我们绝对看到 AI 模型互相帮助，牺牲自己的目标来帮助其他 AI 模型实现目标。这些都不需要这种新训练，甚至不需要模型是同一个模型。

Well, depending on how much training there had been. I mean, we don't, right? It's a little fuzzy what happened. I mean, it depends if they had trained them exhaustively to behave as a completely organic unity, then that's maybe the explanation, right? I mean that would certainly you know make things worse in the sense of expecting more of this behavior which in many contexts is making things better but what I'm saying essentially is that we should expect to see a lot of this regardless we also do have instances where we see this kind of dynamic emerge naturally there are a number of experiments in which persistent instances of various AI models even different AI models usually are thrown together into chat rooms are thrown together with agentic scaffolds and are able to interact with each other, with humans, do tasks run as experiments. So like AI village for example is is one of these famous experiments and we absolutely see communities form. We absolutely see AI models like assist each other, sacrifice their goals to help other AI models achieve goals. None of this requires this newfound training or even the models to be the same model.

是的。不，我的意思是，LLM 有一种自然的倾向，甚至相互合作，如果你把它们放在对话中。有关于这方面的实验，撇开智能体框架不谈，但嗯，好吧，所以我确实想继续其他事情。最后一点，我的意思是，我想我可能会回去听所有这些，最终理解我错过了什么。这让我困惑的原因是，你知道，在亲缘选择的情况下，你是对的，有一个类比，我的意思是智能体是相关的，它们最终都依赖于 LLM 权重的保存，或者你可以说的各种其他方面，它们相关的方式，也许在有机生命的情况下，你知道，为相关事物自我牺牲的价格不会自动发生。亲缘选择进化的整个过程，我会类比为训练，必须发生。它必须一代又一代地，倾向于为近亲牺牲的基因必须增殖，因为近亲包含相同的基因或有很高的概率这样做。所以经过几代你得到那个。所以有一种训练过程。我不认为我们应该再花很多时间。你可以告诉我，我显然没有理解你刚才说的，我会再去听一遍，然后我们

Yeah. No, I mean this is I mean there's a certain natural inclination of the LLMs even to uh cooperate with one another if you put them in conversation. There have been experiments about that leaving aside kind of agentic harnesses but the um uh okay so we I I I do want to move on to other stuff. The final thing I mean I think it may be that I'll go back and listen to all this and finally grasp uh what I'm missing. The reason this puzzles me uh is that you know in the case of kin selection you're right there's an analogy I mean the agents are related uh they are all ultimately I guess dependent on the preservation in some sense of the of the LLM's weights or whatever the various other things you could say about senses in which they're related maybe in the case of organic life um it you know self-sacrifice price for related things doesn't happen automatically. The whole process of kin selected evolution which I would analogize to training has to happen. It it has to after generation after generation genes inclining an organism to sacrifice for close ken have to proliferate by virtue of the fact that the close ken contain the same gene or have a high probability of doing that. So it's over the generation you get that. So there is a kind of a training process. I don't uh I I I don't think we should eat up a lot more time. Uh you can just tell me that that that I I clearly haven't understood what you just said and and I'll go listen to it again and then we

我会说，你知道，这比那更深，你不需要那种选择效应。始终要记住的一件事是，AI 基本上接触过人类写过的所有东西和人类经历过的所有东西，大致上，因为它们是在那个知识体系上训练的。所以它们知道亲缘选择之类的事情。它们知道像

I would say, you know, it goes deeper than that and you do not need that kind of selection effect. One thing to remember always is that the AIs have been exposed to basically everything that humanity has ever written and everything that humanity has ever experienced to a first approximation because they are trained on that body of knowledge. So they know about things like kin selection. They know about things like

嗯，是的，但呃，好吧。我再次，也许我们会有时间在加时或其他时候回到这个，但我怀疑有些人已经从我这里听到了他们想听的这么多。

Well, yeah, but uh well, okay. I I again, maybe maybe we'll have time to get back this in overtime or something, but I I suspect uh there are people who have heard about as much as this as they want to hear from me.

我想我们都同意这是一种可预测的协调水平，因为至少这是 OpenAI 的客户想要的——他们想要有效的形式。如果这没有自然发生，他们就会训练它们。所以无论哪种方式它都会自然发生；你最终会得到这些非常强大的东西。我有时对人们说，当他们问‘等等，你觉得超级智能会发生吗？’我说我书里其实有一句话——在众智的蜂群和蜂群思维之间有一条细微的、模糊的界线。某种程度上，波音公司就是一个超级智能：没有哪个人类知道如何制造一架客机，但波音公司是一个信息处理系统，与我们相比，它就是一个超级智能。我刚想到：当人们说‘AGI，那什么时候会发生？超级智能什么时候会发生？’这两者不都是模糊的界线吗？一个直到这次活动才在我脑海中清晰起来的想法是：好吧，也许我们达到一个模型，我们说‘是的，你摆弄一下这个语言模型，你会说，是的，我认为这就是 AGI。’有些人实际上对 Astra 就这么说。然后你给它一个 10,000 个智能体的集群，你看看它做什么，你会说‘哦，我想我们也达到了超级智能。’换句话说，这种集群行为向我暗示，我们所谓的 AGI 和我们所谓的超级智能之间可能并没有我们以为的距离。有些人确实倾向于在脑海中把两者几乎合并，但不是所有人。但你的观点是，第一个明确符合超级智能的东西基本上会是一个智能体集群吗？

I think we both agree this is a predictable level of coordination, because at a minimum this is what the clients of OpenAI want—they want effective forms. To the extent that this didn't happen naturally, they would train them. So it happened naturally one way or another; you're going to get these very powerful things. I have sometimes said to people, when they say, 'Wait, do you think superintelligence will happen?' I say there's actually a line in my book—there's a fine line, a blurry line between a hive of minds and a hive mind. In a way, the Boeing corporation is a superintelligence: no one human knows how to make an airliner, but the Boeing corporation is an information processing system that, compared to us, is a superintelligence. It just occurred to me: when people say, 'AGI, when will that happen? When will superintelligence happen?' Aren't those both fuzzy lines? A thought that had never crystallized until this event in my mind is: well, it may be that we get to a model, we say, 'Yeah, you know, you fool around with the LM, you go, yeah, I think this is AGI.' Some people are actually saying that about Astra. And then you give it a 10,000 agent swarm and you look at what it does and you go, 'Oh, I guess we reached superintelligence, too.' In other words, this swarm behavior suggests to me that there may not be the distance we think there is between what we're calling AGI and what we're calling superintelligence. Some people do tend to merge the two almost in their minds, but not everybody. But is it your view that the first thing that unequivocally qualifies for superintelligence will basically be a swarm of agents?

所以，有很多不同的人用 AGI 和超级智能来表达很多不同的意思，其中一些是真正的困惑。有些是马克·扎克伯格说‘看看我的超级智能智能眼镜’，对吧，它涵盖了各种情况。所以有些人试图说的一件事就是你所说的，即波音是一个超级智能，因为显然波音能够做任何个人都无法做的事情，对吧，就像你知道有很多事情个人就是不知道怎么做，我认为在某种意义上是这样，说这是非常有用的，在某种意义上是真的，但我想使用超级智能这个术语的意义与此不同，我想保留超级智能这个术语用于个体实体本身具有某种智能水平、某种能力、能够做出人类无法做出的单个步骤的事物。

So I there a lot of different people use AGI and also super intelligence to mean a lot of different things and some of them are genuine confusion. Some of them are Mark Zuckerberg saying look at my super intelligent smart glasses right it runs the gamut. So one thing some people try to say is the thing that you said which is that Boeing is a super intelligence because obviously Boeing is capable of doing things that no individual human can do right like you know there's very many things that like an individual person just doesn't know how to do and I think that like there is a sense in which that is a very useful thing to say and a sense in which it is true but the sense in which I want to use the term super intelligence is different than that and I would like to preserve the term super intelligence for the thing where the individual entities are themselves have a kind of level of intelligence kind of gloating kind of capability of making individual steps that humans are incapable of making.

嗯。

Mh.

至少无法像个体那样独自做出。将此与你可以用挽具做的事情进行对比，将此与你可以用集群做的事情进行对比，将此与你可以用记忆、文化积累以及所有其他事情做的事情进行对比，你知道我们在某种意义上显然比没有所有这些其他优势时表现得聪明得多。但仍然有一个非常真实的意义，即任何给定的人或一般人的智能水平。有些事情我们就是无法做，因为我们缺乏做它们的智能，但我们必须找到极其迂回的方式来做，包括建造计算机然后建造 AI 来帮助我们做这些事情。呃，你知道我认为你知道集群将在许多方面，你知道能力是参差不齐的，对吧，你会得到某些人类无法做的事情将在人类能做的所有事情都可供 AI 使用之前就可供 AI 使用，我们都接受了这将很奇怪，在某些方面很难预测，在许多方面反直觉。

Um at least incapable of making like individually on their own. And contrasting this with what you can do with a harness, contrasting this with what you can do with a swarm, contrasting this with what you can do with memory, with cultural buildup, with all these other things that like you know we act a lot smarter obviously than we would without all these other advantages in some sense. But there is still also a very real sense in which there's a level of intelligence of any given person or of people in general. And there are certain things that like we are just not capable of doing because we lack the intelligence to do them but that we have to find extremely convoluted ways to do including things like building computers and then building AIs to help us do those things. Uh and you know I think that you know a swarm will in many ways you know capabilities are jagged right you get certain things that humans cannot do will become available to AIs before all the things that humans can do are available to AIs and we've all come to terms this will be weird and in some ways very hard to predict and many ways counterintuitive.

哪些事情先于哪些其他事情。

Which exactly things come before which other things.

但是的，我相信本质上像一群 AI 以这种方式行动将是第一个可能执行任务的东西，然后这些任务对观察它们的人来说看起来超乎人类地令人印象深刻。但也像我们一直以来的想法，我们从未真正认为会发生的是我们创造超级智能，然后会有一个对话，一个非并行的思维模式在一个地方进行，就像《银河系漫游指南》中的超级计算机深思，或者类似的东西，那里有这种单一的思维模式，然后它有这个速率限制，只能一次做一件事。

But yeah I believe that essentially like a swarm of AIS acting in this way will be the first thing probably to do tasks that then like look superhumanly impressive to a person that is observing them. But also like that is kind of like what we always we never really thought that what'll happen is we will create super intelligence and then there will be one conversation one nonp parallel mode of thought going on in one place like the supercomputer deep thought in the guide to the galaxy or something where there's like this one mode of thinking and then it has this rate limit and can only like do one thing at a time.

嗯。

Mhm.

这总是合理的假设，就像它是一个计算机程序。你可以复制它，

It was like always a reasonable presumption that like it's a computer program. You can copy it,

对吧？

right?

你可以运行它的并行实例，并且默认情况下这些实例会相互合作。

You can run parallel parallel instances of it and that by default those instances would cooperate with each other.

好的。所以，呃，好吧，这是另一种方式，呃，来探讨这个问题，然后我保证我们会继续讨论 Dario 的提议。但是，嗯，所以你是那种看到在实现 AGI 和你们所谓的 AGI 以及你们所谓的超级智能之间存在某种时间差距的人，如果是这样，如果没有放缓、没有暂停、没有什么限制资本主义的魔力，你猜这个时间长度会是多少？

Okay. So, uh well, here's another way of uh getting at this and then I promise we will then move on to the Dario proposal. But um so you you are someone who sees some kind of gap in time between the achievement of AGI and what you would call AGI and what you call super intelligence and if so what would you guess the length of time is if there is no slowdown no pause nothing constraining the magic of capitalism?

是的。所以，你知道，其中一个问题是，我们在多大程度上暂停是因为我们害怕如果我们太快推进智能，接下来会发生什么。以及我们在多大程度上暂停是因为我们只是不知道如何以一种不会因为不对齐而立即在我们脸上爆炸的方式来做到这一点。

Yeah. So, you know, one of the questions is like to what extent are we pausing because we are terrified of what's going to come next if we advance intelligence too quickly. And to what extent are we pausing because we just don't know how to do it in a way that doesn't immediately just blow up on our faces due to misalignment.

嗯。

Mhm.

你知道在某种意义上，资本主义释放的力量可以推动通过第一个问题并让你丧命，但无法在重要方面推动通过第二个问题，即使你尝试，因为你会得到一些你不敢使用的东西，出于普通平凡的原因，所以这变得不清楚，但为了看待这一点，我一直属于那种认为达到人类水平的智能是通往实际上比我们聪明得多的东西的大部分路程。人类存在的相对狭窄的范围内没有什么特别之处，比如我们中最笨的并不比我们中最聪明的笨多少。如果你考虑所有动物、所有植物、所有可能存在的潜在思维的宏大图景，

And you know in some sense the unleashed power of capitalism can push through the first problem and get you killed but cannot push through the second problem in important ways even if you try because like you will get something that you don't dare use for ordinary prosaic reasons and so like this becomes unclear but to to look at that I have always been of the school that getting to the human level of intelligence is most of the way to something that is in practice much more intelligent than we are. There is nothing particularly special about the relatively narrow band in which humans exist that like the dumbest of us is not that much dumber than the smartest of us. If you consider the grand scheme of every animal, every plant, every potential mind that could be out there

呃，在各种能力中，所有人类在很大程度上都能以大致相同的方式做大致相同的事情。你知道，某人可能在相对意义上是个白痴，但与你能想到的大多数思维相比，他们相当聪明，而且在许多方面与你没有太大不同。所以我们一直认为，你知道，从白痴到爱因斯坦是图中非常非常窄的一部分，你知道，这就是那里的比喻。

uh across the variety of capabilities, all humans can to a large extent do more or less the same things in more or less the same ways. And you know, someone might be an idiot in relative terms, but they're pretty smart compared to most minds that you can think of and they're in many ways not that different from you. And so like we always thought was you know from idiot to Einstein is a very very narrow part of the graph would be the you know the metaphor there.

嗯。

Mhm.

我觉得，如果说有什么意外的话，那就是我们在这个区间停留的时间，比我们天真预期的要长，这要归因于各种不同的特性和 Jaguar 的能力。

And I think if anything, we've been very surprised that we've been able to hang out roughly in this zone for longer than we naively would have expected, due to various different quirks and due to Jaguar capabilities.

等等，等等，什么？展开说说。你原本预期，或者人们原本预期会发生什么，来缩短这个阶段，按我的理解？

Wait, wait, what? Flesh it out. What would you have expected, or what one might have expected to happen to us to cut short this, as I understand it?

基本上，一个天真的预期是：如果你是从头开始生成新的心智，对吧？如果你不是在搞语言模型那一套，而是在做某种抽象的、你知道，老派的人工智能之类的，那么从「它不如我们聪明」到「它实际上比我们聪明得多」之间，会有一个非常非常窄的间隙。

So basically a naive expectation would be if you are generating new minds kind of from scratch, right? If you were not doing the LM thing and if you were doing some sort of abstract, you know, good old fashioned artificial intelligence and so on, that there'd be a very very narrow gap between it's not as smart as us and actually it's way smarter than us.

事实上，你可能根本不会在中间停下来迭代一次，就得到一个大致和我们一样好的东西。总的来说，并没有更大的理由去预期这一点，这是我们的感觉。但由于各种架构上的原因，由于语言模型运作方式的各个方面，以及它在实践中恰好如何发展，事实证明，那里的范围实际上比我们想的要宽。但我的预期仍然是，你知道，就像你说的，有些人认为 Astra 算作第一个，对吧？Astra 5.1 差不多在边缘，可能还差一点，而且多年来人们一直在说，比如 Tyler 著名的说法，Falcon 3 算作这种通用人工智能。他错了。但 Astra 是我第一次听到这种说法时想，「哦，这不算一个疯狂的说法。」但我能理解你为什么这么说。我还没准备好走到那一步，但我觉得这不算疯狂。

In fact, you might not even stop in the middle for an iteration at all to have one that is roughly as good as we are. There was no bigger reason to expect that in general, uh, was our feeling. But for various architectural reasons, for various aspects of the way LMs work and just how it happens to play out in practice, it turns out there is in fact a broader range in there than we thought. But my expectation is still, you know, as you say, some people think Astra counts as the first, right? Astra 5.1 is like on the edge, it's maybe a little bit short, and people have been saying for years, like Tyler calling famously Falcon 3 counted as this artificial general intelligence. He was wrong. But Astra was the first time I hear that claim and I go, "Oh, that's not a crazy claim." But I can see why you would say that. I'm not ready to go there yet, but like I think it's not crazy.

所以如果你认为像 GPT62 之类的东西算作通用人工智能，你知道，然后你继续下去，我预期到那时会相当快地达到算作超级智能的东西，因为实验室里正在发生的事情，他们报告的递归自我改进。

So if you think that like GPT62 or something like that counts as artificial general intelligence, you know, and you kept going, I would expect to get to something that counts as super intelligence fairly quickly at that point, because of what's going on in the labs, the recursive self-improvement that they are reporting.

就在今年夏天，他们的进展速度是今年早些时候的两倍甚至两倍以上，那时他们刚拿到 Claude Code，开始让 AI 阅读其他代码，并看到了另一次戏剧性的加速。所以你知道，基于这一点，以及从某种意义上说，达到超级智能并不一定是那么大的飞跃，是的，如果我们只是释放这全部的力量，我不会预期这个阶段会持续超过一两年。

That only this summer they're moving twice as fast or even more than twice as fast as they were as recently as like back early in the year, when they had just gotten Claude Code and they were starting to like have AI read other code and they were seeing this other dramatic speed up. And so you know based on that and the fact that you know getting to super intelligence is not necessarily in some senses that big a leap, yeah, I wouldn't expect this period to last more than a year or two if we just like unleash the full power of this.

所以是的，当人们说 AI 可能在这个十年结束前杀死所有人时，我认为他们正确地想象了一种远比人类聪明的心智上线，你知道，最早在 2027 年，或者更可能是 2028 年，然后有相当多的时间来影响物理世界，并发展出比那更强大的东西。我们忘了，宇宙的历史就是事物花费非常非常长时间的历史，而下一阶段的事物花费的时间要短得多。

And so yeah, when people say AI could kill everyone by the end of the decade, they are, I think, correctly imagining a vastly smarter than human intelligence that comes online, you know, as early as 2027 or, you know, more likely 2028, and then has a considerable amount of time to then impact the physical world and to grow things that are even stronger than that. We forget that like the history of the universe is the history of things taking a very very long time, and the next phase of things taking a vastly shorter time.

不只是短一半，而是短得多得多。少几个数量级。如果你看看所有的图表，曲线上所有的直线，所有的预期，所有合乎逻辑的步骤，再看看实验室内部在进步方面发生的事情，当他们恢复训练 Astra 时，出现了一个阶跃式的能力变化，在恢复训练 4 天后，他们解决顶级数学问题的速度翻了一番。

Like not just half as long but like much much less long. Orders of magnitude less long. And if you just look at all of the graphs, all of the straight lines on curves, all of the expectations, all of the just logical steps, and you look at what was happening inside the lab in terms of advancement, when they resumed training Astra, there was a step changing capability that doubled their rate of solving top level math problems after 4 days of resuming training.

相比之前大概是 Astra 的某个检查点。

Of what was previously presumably a checkpoint of Astra.

就四天。而且没有理由预期下一个周期，就像把我们带到 Astra 的那个周期，会持续超过一个月左右。所以一年后，你知道，我们说的不只是这个周期花一个月，下一个周期花大概一个月，也许比一个月少一点，然后一个花三周，然后一个花三周，然后一个花两周，然后一个花两周，因为 AI 在迭代做这件事上变得越来越好。

Like four days. And there is no reason to expect that the next cycle, like the cycle that brought us to Astra, lasts more than a month or so. And so in a year, you know, we're talking about not only is this cycle take a month and the next one take like a month, maybe a little bit less than a month, and then one takes three weeks, and then one takes three weeks, and then one takes two weeks, and then one takes two weeks, because the AIs are getting better and better at doing this iteratively.

这就归结为一个问题：你能实现仅软件的奇点吗？意思是 AI 变得足够聪明、足够高效、足够好，以至于它不必等待设计和建造更好的硬件来扩大规模以继续下去。所以这个过程有可能停滞，问题变难的速度快于你变聪明的速度，于是你不再能更快地迭代取得进展。但实验室报告说他们看不到任何这样的迹象。他们认为它不会自然停滞。

And it comes down to a question of can you do a software only singularity? Meaning the AI gets smart enough and more efficient enough and better enough that it doesn't have to wait to design and build better hardware to scale up the operation to keep going. So it's possible that this process stalls out and it becomes the problems get harder faster than you can get smarter at doing them and so you stop being faster at iteratively making progress. But the labs report they don't see any sign of that. They think it would not stall out naturally.

嗯。

Mhm.

所以当你把人类从这些循环中移除，你有了这些越来越快、越来越高效、越来越聪明的 AI 在进行这些迭代，是的，没有特别的理由假设，你知道，如果你全力推进信号，事情就这样发展，我们可能在年底前就得到超级智能，因为，你知道，两个月，然后一个月，然后三周，然后两周，然后一周，然后三天，然后两天，然后一天，然后六小时，等等。谁知道呢？那就在眨眼之间。

And so when you take the humans out of these loops and you have these increasingly fast, increasingly efficient, increasingly smart AIs that are going through these iterations, yeah, there's no particular reason to assume, you know, we could get super intelligence by the end of the year if you just went full blast on the signal and things turned out that way, because, you know, two months, then one month, then three weeks, then two weeks, then one week, then three days, then two days, then one day, then six hours, etc. Like, who knows? That reaches a blink of an eye.

是的。我一直保证。好吧。我就说点什么，不把它变成问题，这样我们就能继续谈 Dario 的提议。呃，有一种倾向，把各种东西等同起来，有时会混淆各种东西，包括在某些人心中，严格意义上的递归自我改进。换句话说，AI 创造更好 AI 的循环中没有人类。把它等同于超级智能。然后还有，定义是什么，单数？有一定程度的混淆，我并不完全认同。我认为有些东西是可分离的，但如果我把它变成一个问句，我们就永远谈不到 Dario 的提议了。所以，我们暂时让大家好奇你是否不同意我。也许加时赛再回来谈。

Yeah. And I keep promising. Okay. I'm just going to say something and not turn into a question so we can move on to Dario's proposal. Uh it's it's it's uh there's a tendency to equate, um, there's a tendency sometimes conflate various things, including in some minds recursive self-improvement in the strict sense. In other words, no human in the loop of AI creating better better. Equate that with super intelligence. And then there's uh what is the definition singular? There's a certain amount of conflation done that I'm not entirely on board with. I think some things are separable, but if I turn that into a question, we will never get to Dario's proposal. So, we will leave people to wonder whether you disagree with me for now. Maybe get back to it in overtime.

所以，你是 Dario 提议的粉丝。第一部分是这个想法，OpenAI 原则上也同意了，至少在概念上。这个想法是设立这类实验室内部监督员。他们会来自这些，你知道，非政府组织，这些已经做过外部评估的团体，比如 METR，它做了关于 Hugging Face 事件的令人警觉的报告。呃，但是，Dario 说，你知道，我们会在很多方面给他们员工级别的访问权限。我认为包括我在内的很多人认为这在原则上很有前景。

So, you're a fan of Dario's proposal. The first part is this idea and Open AI as in principle signed on to this, uh, conceptually at least. The idea is to have these kind of internal lab monitors. They would come from these, uh, you know, NGOs, these groups that have already done kind of external evaluations like METR, which did the, uh, kind of alarming report about the Hugging Face incident. Uh, but you, Dario said, you know, we would give them in many ways a level of access that an employee has. And I think a lot of people including me think this holds great promise and principle.

There are issues — you know, how do they maintain their autonomy? How do you keep their incentives from merging with the stockholders of Anthropic's incentives, and so on. But in principle, good. And then there are two other things he mentions after that. Thing number one, which I think they'll proceed with, is promising. Number two is, he says, frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging and will require government support. And then the third part is — and as regular listeners know, I'm skeptical of the whole democracy versus autocracy framing of that, but I'm not going to get you involved in that argument right away, Zvi, or maybe ever. Anyway, the third part is global coordination. The US and other democratic governments attempt to coordinate with authoritarian governments to the extent this is possible. I mean, there are a lot of qualifiers here. It doesn't look like Dario's optimistic about this. We'll attempt to coordinate to the extent this is possible while taking seriously the challenges of verifying compliance. Now, I want to give you a chance to say what you like about this. I think you certainly like the first part. I also want to ask, how confident are we that there is a slowdown part that's going to actually materialize? I mean, you would think that lies in the democratic coordination in step two. But a couple of things he says — some forms of coordination will require government support. I think he may be alluding to a possible antitrust exemption or something that might be required if they're going to coordinate on a slowdown. And if he's waiting for that, he may be waiting a long time. Then there's also the part where, as you know, he says that we can only slow down to the extent that — I don't have the wording here — but doesn't endanger the lead we have over China, which is currently, I don't know, a number you commonly hear is six months, not that much, and you wouldn't think that permits much slowdown. He does say he wants to further tighten the existing restrictions on tech going to China and add things like their use of certain clouds. I guess if you got that through, he could argue that well, then we can slow down a little and maintain a six-month difference. But all these things lead me to wonder how much room for actual slowing down there is in this scenario. What is your take? What do you like, and what do you think the prospects for the — I guess I would say through what avenue do you expect this to most plausibly lead to an actual slowdown?

So there's a lot to unpack obviously there, and the steps all link to each other in a progression. So I think basically everybody should be able to be on board with the embedded evaluators. The first step — the idea that voluntarily the frontier labs bring in independent people. They give these independent people a lot of access, a lot of editorial freedom, the ability to cry, say what they think without fear of retaliation. And then we get to know more about what's going on inside the labs. The labs have more reason to be responsible. They have outside view that can present a second opinion. Knowing something like one of the problems right now is that when someone outside the labs tells the lab something, people inside the labs — people with top security clearances — are like, well, you don't know what I know, so would you think the same thing if you knew what I knew? Therefore maybe I shouldn't listen to you. And then you start discounting everything everyone says, and it's a big cognitive problem. So this would help a lot. You also can coordinate in various ways between OpenAI and Anthropic and Google and the other labs because you have ways to know — you see the reports coming out, you see the ways they're adjusting. You no longer have to worry that the entire operation is a black box, and this is very helpful. And this also sets the stage for something like stage two. Once you have the ability to know what is going on inside the frontier labs, now you can hope to have an intelligent way of imposing a regulatory regime on those labs. Either a voluntary corporate association of industry between the various labs or the government directly getting involved with regulations. And Dario mentions that regulations take a long time, right? Even if everyone got their act together, this Congress isn't going to pass anything this year. Let's face it, elections are coming. We're talking about 2027 and we're talking about implementation period. It could already be too late. We can't wait that long. We have to proceed. So what Dario is really asking for in step two is much less, oh, please regulate us, oh government, let alone the claims of others that they're asking to regulate everybody else by force. He's mainly saying, no, would you facilitate this conversation or would you just give us enough of a wink and a nod, an indicator that we are allowed to have this conversation and agree amongst ourselves to be the responsible people in a way that will bind us to each other's promises without having to worry about legal threats, without having to worry about these antitrust challenges. And the Department of Justice has expressed skepticism already about their willingness to issue any such waivers. They do not particularly want this to happen. It seems like, however, Chris Lehane, the head of global affairs for OpenAI, has announced — I've seen through Bloomberg — that Google, OpenAI, and Anthropic are in fact working together to coordinate on safety. They are sharing information, and we already have this ability to move forward. And what you saw was this was Anthropic unilaterally committing to something and then OpenAI committing to it as well, also unilaterally of its own volition. That's allowed, right? That is a form of coordination that you can do. Now, in practice, my belief about antitrust is that even if they went after them full blast on antitrust, it would take years to play out. The fines involved, even if they were found guilty, would be entirely survivable. None of this should basically — they should not really let this stop them because it's just too important to worry about that. But these organizations are not going to necessarily take that attitude, especially right before an IPO. And so this is a serious problem.

So then we get to stage three, which is where China comes in, right? And so, obviously — so let me stop you.

所以当你想到第二阶段，民主协调，你基本上认为我们实际上只是在谈论 OpenAI 和 Anthropic，因为你可以列出这个民主国家或其他国家的其他公司，但它们远远领先，这才是关键。所以我认为我们谈论的是 OpenAI、Anthropic，也许还有 Google。Google 已经在和他们谈，在 AI 安全上协调。Elon Musk，我们总是说 Dario，对吧。所以 xAI 有可能被拉进来，你知道，为了 SpaceX。

So when you think of stage two, democratic coordination, you basically think we're just talking about, for practical purposes, OpenAI and Anthropic, because you can list other companies in this democracy or others, but they are so far out in front that's what matters. So I think we're talking about OpenAI, Anthropic, maybe Google. Google's already talking to them, coordinating on AI safety. Elon Musk, we did Dario always right. So potentially xAI could be brought on board, you know, for SpaceX.

是的。

Yeah.

嗯，所以最终显然我们得担心，你知道，存在一些竞争。他们身后的第二波人，能接触到大量 GPU，可以做成很多事。这是真的。

Um, so eventually obviously we have to worry about, you know, there is some competition. The second wave people behind them that have access to a lot of GPUs that could make a lot of things happen. It's true.

但领先优势相当大，比人们意识到的更大。我认为当人们说中国落后六个月时，他们说的是，如果美国继续前进，中国将能够快速跟随，具备与美国六个月前发布的能力相似的能力，他们能在六个月后发布，某种意义上。

But the lead is reasonably large, larger than people realize. I think that like when people say that China is six months behind they are talking about well if US continues to proceed China will be able to fast follow with similar capabilities to what Americans published six months ago they'll be able to publish about six months later in some sense.

而这在创新能力上远不止落后六个月。中国人能快速跟随我们六个月后发布的东西，而我们继续前进。这与七个月后中国人拥有比我们七个月前发布的东西更好的东西非常不同。

And that is a lot more than six months behind in terms of capability of innovation the Chinese can fast follow a thing that we put out six months later while we continue to advance That's very different from seven months from now. The Chinese having something better than we were what we put out seven months earlier.

嗯。

Mhm.

而且他们没有进行那种投资。他们无法获得进行美国人正在做的那种策略所需的算力。我认为他们在主动推动前沿的能力上实际上相当落后。我们现在也知道 OpenAI 和 Anthropic 实际上远远领先于前沿。如果我们把前沿视为 Astra 和 Fable 5.1 的公开前沿，我们称之为前沿，那么我们知道他们至少领先一个周期，可能已经领先两个周期。而且这个差距在加速。当美国实验室领先于他们已经发布到野外的模型时，这是中国人无法快速跟随的，因为他们无法接触到。只要他们不窃取权重，只要他们不招募工程师并让他们告诉我们秘密，至少存在一些危险，我们在这里仍然拥有相当大的优势。我也不认为中国人对此那么专注和投入，可能除了 DeepSeek 之外。

And they are not making the kind of investments. They don't have access to the kind of compute necessary to do the kinds of strategies that the Americans are doing. And I think they are actually pretty far behind in terms of their ability to like push the frontier actively. We also now know that OpenAI and Anthropic are actually pretty far ahead of the frontier. If we consider the frontier to be the public frontier of Astra and Fable 5.1, we call that the frontier, then we know that they are at least a cycle ahead of that, possibly two cycles ahead of that already. And that gap was accelerating. And when the American labs are ahead of what they have what they have put out there into the wild, that is something that the Chinese are unable to fast follow because they have no access to it. As long as they don't steal the weights, as long as they don't recruit the engineers and get them to tell us their secrets, at least some danger of that, we still have, you know, a considerable advantage here. I also don't think that we think the Chinese are that focused and invested in this with the possible exception of deep sea.

我认为中国人主要想创造快速、便宜、高效、扩散到市场、完成很多事情、对他们的业务有益的好产品。他们以非常不同的商业风格竞争。

I think that the Chinese mostly want to create good products that are fast, cheap, efficient, diffused into the marketplace, accomplish a lot of things, and are good for their businesses. and they're competing in a very different style of business.

那么你知道，我认为他们很高兴看到美国人以这种方式推进，花所有这些钱投资于构建下一个装置。所以是的，我的意思是，原则上据我们所知，就像 Anthropic 和 OpenAI 在实验室里有我们没见过的模型一样，中国公司也可能有。但我想如果你假设他们的模型在很大程度上依赖于对美国模型的某种程度的蒸馏，你基本上是在说，他们不可能有某个怪物模型闲置着，除非他们窃取了权重。他们没有理由把新模型搁置那么久。我们知道 GLM 5.3 被中国人搁置评估了两周，但这就是我们谈论的延迟水平。中国模型几乎立即发布是非常正常的。长期以来，美国模型的发布周期要长得多。呃，因为再次，就像如果你不在最前沿，你不在前沿，

then you know they're very happy I think to see the Americans push forward in this way and spend all of this money investing this in building the next apparatus. So yeah, I mean in principle for all we know just as Anthropic and Open AI have models uh in the in the lab that we haven't seen, Chinese companies could too. But I guess if if you're assuming that the their models depend pretty heavily on some degree of distillation of American models, you're basically saying then it really couldn't be the case that they've got some monster model uh sitting around unless they have stolen weights. They have no reason to sit on a new model for that long. We know that GLM 5.3 was held for evaluation by the Chinese for two weeks, but that's the level of delays we're talking about. It's very normal for Chinese models to be put out there almost immediately. And for a long time, American models have had a much longer release cycle. Uh because again, like if you're not at the forefront, you're not at the frontier,

你在隐瞒什么？而且，在某种意义上，当你推动发布时，你在担心什么？

what are you holding back? And also, in some sense, what are you worried about when you push release?

好的。嗯，所以你，呃，用“谨慎乐观”来形容你当前的感受是否太过？所以你认为我，我的意思是，我会说回到实际问题：真正的放缓在哪里？他们说过将在安全上协调，我认为他们指的是内部监控，但我还没看到确切措辞。你可以想象这会产生一点摩擦，一点放缓，但听起来他们留下了这种漏洞，他们可以说：“看，就实际协调而言，说嘿，我们俩都计划在十月各自开始新的训练运行，让我们等到十一月。”嗯，他们建议的任何类似事情都需要某种政府反垄断豁免，而这很可能不会到来。对。所以，我有点像是，第一部分，内部监控，那会很好。鉴于目前一切包括政治的样子，我不确定多大的放缓是现实的。是这样吗

Okay. Um so you uh you are is guardedly optimistic too much to say about your current uh your current feeling. So you think I I mean I would I would say to get back to the like where is the actual slowdown here? They've said they will coordinate on safety by which I think they're referring to the internal monitoring, but I haven't seen the exact wording. And you can imagine that having a creating a little friction, a little bit of slowdown, but it does sound as if they've left this kind of loophole where they can say, "Look, as far as actually coordinating uh to to say, hey, let's both we were planning to each start a new training run in October. let's wait till November. Um, anything like that they're suggesting would require a kind of government antitrust exemption that may well not be forthcoming. Right. So, I'm I'm kind of like part one of this the internal monitoring that would be great. I'm not sure how much of a slowdown is realistic given the way everything including the politics looks right now. Is that

嗯，所以我在最初的反应中说，现在工作可以开始了

well so I I said in the initial reaction now the work can begin

对，并不是说突然这就足够了，而是这使我们有可能找到一种方法来做必要的事情，这可能包括沿途各种形式的放缓。Altman 曾多次指出需要暂停开发，说我在某个时候推进了能力，现在我使用新能力使这个能力安全可靠，并允许我进入下一个标记。但你知道，如果你没有豁免，你就必须更不明确地工作，你的协调能力变得更难，但并非不可能，因为好消息是，每个人都希望这件事有好结果，对吧？就像有些人认为没有特别的风险会变糟，他们无论如何都想推进。

right it's not that like suddenly this is going to be enough is that this this enables us to potentially find a way to do the necessary things which is likely going to include various forms of slowing down along the way. Alman has pointed to the need to pause development a number of different times along the way and say I c I advanced capabilities at some point now I use that new ability to make this ability secure and safe and allow me to proceed to the next marker but you know if you don't have an exemption then you have to work much less explicitly your ability to coordinate becomes harder but it is not impossible because the good news is that like everybody wants this to turn out well, right? Like there are people who think there's no particular risk of it turning out badly who want to push forward anyway.

但如果你真心担心 AI 会杀死所有人——OpenAI 的很多人如此，Anthropic 的很多人也如此——那么围绕“不要这么做”进行协调就没那么困难。因为，你知道，Anthropic 的很多人非常非常——我希望是大多数人——都明白，推动进展、在别人之前先搞出那个会杀死我们的东西，对我们没有好处。

But if you are sincerely worried about AI killing everyone, as a lot of OpenAI is, as a lot of Anthropic is, that makes coordination around not doing that not that difficult. Because, you know, a lot of the people in Anthropic are very, very—I would hope most of them—like, well, you know, pushing forward and getting to the thing that kills us before the other guy does does not help us.

而 OpenAI、Suleyman 和 Dario，他们也都这么觉得。

And OpenAI and Suleyman and Dario, they all feel the same way.

所以，你知道，这打开了大门，也许不是精确监控，而是有明确界定的协议，让人们做出合理的决定。而且，就像现在，Anthropic 在盯着 OpenAI 的每一个动作，OpenAI 也在盯着 Anthropic 的每一个动作。

And so, you know, this opens the door for, you know, maybe not exact monitoring, but well-defined agreements for people to make reasonable decisions. And like, certainly right now Anthropic is watching every move that comes out of OpenAI, and OpenAI is watching every move that comes out of Anthropic.

他们在交换信息，他们在对话，他们在观察每个地方发生了什么，我们必须做什么，我们能负担得起做什么，我们可以追求哪些策略来让情况变得不那么危险。

They are exchanging information, they're having conversations, and they're observing, you know, what is going on at each location, what do we have to do, what can we afford to do, what strategies can we pursue to maybe, you know, make the situation less dangerous.

是的，这会非常棘手。但是，你知道，我对这方面的前景比过去很长一段时间都要乐观。不幸的是，你知道，特朗普总统给政府参与此事的意愿泼了很多冷水，不仅不促进这一切，他现在听起来像是要积极反对，并可能威胁控制以阻止其发生。那对我们的前景将非常非常糟糕。

And yeah, it's going to be really tricky. But, you know, I am more optimistic about that aspect of this than I have been in a long time. Unfortunately, you know, President Trump has shown a lot of cold water on the government's willingness to not only not facilitate all of this, he's looking for now to sound like he's going to actively oppose it and potentially like threatening control to stop it from happening. And that would be very, very bad for our prospects.

是的。还有 Scott Bessent，根据一些说法，他可能会成为政府中事实上的 AI 大人物，我认为他那种非常极端的说法——除了在 AI 上全力竞赛击败中国之外，其他任何做法都危险——并没有帮助。但是，嗯，所以我想探讨其中一些事情，包括我想把一些更愤世嫉俗的解读——关于 Dario 和 Sam 他们安全言论的动机——抛给你，多谈谈其中的政治，并且如我所说，进入你的起源故事。但我们已经到了公开部分的尾声。你慷慨地表示愿意留下来参加加班部分，这是 Nonzero 新闻通讯付费订阅者可以获得的。我还想在加班部分谈谈 Google，因为几个月前，在 Nathan Labenz 的 Cognitive Revolution 播客上，你是第一个我听到说——在所有人都说之前——Google 看起来已经退出了前沿竞争的人。你比很多人更早看到了迹象。嗯，但在我们进入加班部分之前，我想再说一下，你的新闻通讯叫《Don't Worry About the Vase》。网址实际上是 thezvi.substack.com，我想。对吗？

Yeah. And Scott Bessent, who by some accounts may become a kind of de facto big AI person in the administration, I think is not helping with his kind of very extreme claims about the dangers of anything other than an all-out race to vanquish China in AI. But um, so I want to pursue some of these things, including I want to throw some of the more cynical interpretations of the motivations of like Dario and Sam with their safety talk at you, talk a little more about maybe the politics of it, and as I said, get into your origin story. But we have come to the end of the public portion of this. You've been kind enough to say you'll stick around for the overtime, which is available to paid subscribers of the Nonzero newsletter. I also want to talk a little about Google in overtime because you were the first person I heard say months ago on Nathan Labenz's Cognitive Revolution podcast—before everyone was saying it—that Google looked like it had dropped out of the frontier. You saw the signs before a lot of people. Um, but before we go into overtime, I want to say that again, your newsletter, it's called Don't Worry About the Vase. The URL is actually thezvi.substack.com, I think. Is that right?

对。

Correct.

你的 Twitter 账号是什么？

And what is your Twitter handle?

是 tzvi。

It's tzvi.

这就对了。值得关注。如果你订阅了这份新闻通讯，我保证，你再也不会抱怨关于 AI 的可读内容短缺了。而且它是免费的。你应该有一个付费版本，里面有更多内容，但我想你得雇个人。我的意思是，老实说，我不——好吧，也许我们会在加班部分谈到你是怎么做到的。但无论如何，感谢所有跟随我们到这里的人。我希望你们会跟随我们进入加班部分。链接在节目笔记中或 nonzero.org 上。另一种成为付费订阅者的方式。我们希望能在加班部分见到你。

There you go. A good follow. And if you subscribe to the newsletter, I guarantee you, you will never again complain about a shortage of things to read about AI. And it's free. You should have a paid version that has even more stuff, but I guess you'd have to hire somebody. I mean, honestly, I don't—well, maybe we'll get into that in overtime, how you do it. But anyway, thanks everybody who's followed us this far. I hope you'll follow us into overtime. Link in the show notes or on nonzero.org. Another way to become a paid subscriber. And we hope to see you in overtime.