AI 2040：从预测到建议

Daniel，欢迎回到 Scaling Laws。

Daniel, welcome back to Scaling Laws.

谢谢邀请。

Thanks for having me.

你写了《AI 2027》，而且你觉得让整个 AI 社区深入探讨和检验我的政策想法与假设非常有趣。那为什么不重来一次，再写一份报告，尽管类型不同、目标不同，写一份《AI 2040》呢？我们就从这儿开始吧。对于那些与世隔绝、错过《AI 2027》的人，那是什么？《AI 2040》又是什么？两者有何不同？

So, you wrote AI 2027 and you thought that was so much fun getting the entire AI community to dive into and probe my policy ideas and my assumptions. Why not do it again and write yet another report, although of a different sort with a different goal, and pen AI 2040? So, let's just start there. For those who were living under a rock and missed AI 2027, what was that and what is AI 2040 and how is it distinct?

是的，《AI 2027》是一个情景预测。它是一个情景，从发布时开始，逐年逐月展开，详细描绘了一个具体的可能未来，并附有随滚动变化的统计数据。每个时间段大约一两页，最后甚至细化到逐月。所以内容相当多，总共约 50 页。它之所以是预测，是因为它不仅仅是我们为了有趣而编的故事，而是我们在每个时刻对 AI 发展最可能延续路径的最佳猜测，规划出你认为 AI 会如何发展到 2027 年。

Yeah, so AI 2027 is a scenario forecast. So it is a scenario. It goes year by year and month by month starting when it was published and lays out a concrete possible future in great detail with accompanying stats that change as you scroll down. And about a page or two about each time period, and eventually it's going month by month. So that's quite a lot. It's about 50 pages or so total, and it's a forecast in the sense that it wasn't just like a story that we made up to be interesting. It was our best guess at each moment of what the most likely continuation would be for the development of AI, planning out how you thought AI would progress to 2027.

没错。

That's right.

不过值得注意的是，由于我们认为 AI 极其重要，它实际上也是对整个世界的预测。因为 AI 变得如此重要，其他一切都被它碾压，世界历史就变成了 AI 历史。这就是《AI 2027》。剧透一下，在《AI 2027》中，公司在 2027 年成功实现了 AI 研究的自动化。这导致了所谓的智能爆炸或奇点，我们根据最佳猜测，推演了 2027、2028、2029 年可能发生的情况。结果它火得超出预期，几乎病毒式传播，这当然是个好消息。我们认为它推动了相关讨论。基于这一成功，我们想再试一次，写另一个大型情景。但这次《AI 2040》的 A 计划不是预测，而是建议。它同样是从现在开始逐年展开的大型情景，但刻意对政府的选择持有点乐观态度。特别是，它假设政府会采纳我们的建议。

Although notably, because of how important we think AI is, it's also just a projection for the whole world. Like, you know, because AI becomes so important, everything else just sort of gets steamrolled by it, and so the history of the world becomes the history of AI. So that was AI 2027. And spoilers, in AI 2027, the companies succeed at automating AI research in 2027. And this causes what you might call an intelligence explosion or the singularity, and we sort of walk through what that might look like according to our best guess through 2027, 2028, and 2029. Now, it blew up bigger than expected. It sort of went mega viral, which of course was good to hear. We think it helped advance the discourse. And building on that success, we thought we would try again with another big scenario. But this one, AI 2040, Plan A, is not a prediction; it's a recommendation. It's another big scenario that starts in the present and goes year by year. But it's deliberately a bit optimistic about the choices made by the government. In particular, it just sort of assumes the government does what we recommend that they do.

我喜欢这个假设。要是所有政策想法都能假设它们会实现就好了。

I love this assumption. This is, you know, if only all policy ideas we could just assume they would come to fruition.

是的。基本上，它是传达我们政策建议的载体，我们认为这是一种有价值的方式，因为我们称之为“情景审视”。我们认为很多政策建议，尤其是那些宏大的 AI 治理愿景，而不是具体的法案文本之类的，很多计划如果仔细审视，推演实际实施会是什么样子、实际预期后果是什么，就会崩溃。所以如果你对计划进行情景审视，往往计划会崩溃，或者至少你会发现之前没意识到的问题。因此，我们认为任何对未来有计划的人都应该对计划进行情景审视并推演。同样，任何对未来没有计划的人也应该这样做，因为你不能只说“哎呀，混过去算了”。那你怎么混？混过去是什么样子？你读过《AI 2027》吗？也许那就是混过去的样子。《AI 2027》的结局可不太好。所以总的来说，我们认为人们应该尽可能推演可能的未来，既包括默认会发生的、看起来最可能的未来，也包括他们试图引导或推荐的未来。

Yeah. Basically, it's a vehicle for conveying our policy recommendations, and we think it's a valuable way to do it because we call it scenario scrutiny. We think that a lot of policy recommendations, or a lot of especially ambitious visions for how to handle AI in general rather than specific bill text or whatever, a lot of plans fall apart if you look at them too closely and you game out what it would look like to actually implement the plan and what the actual expected consequences would be. So if you apply scenario scrutiny to your plan, often times your plan falls apart, or at least you notice issues with the plan that you hadn't realized before. So we think it's actually a very important thing for anyone with a plan for the future to be applying scenario scrutiny to that and gaming it out. For that matter, anyone without a plan for the future should also be doing this, because you can't just say, "Oh well, muddle through." It's like, okay, well how are you going to muddle through, and what would muddling through look like? Have you read AI 2027? Perhaps this is what muddling through would look like. AI 2027 doesn't end very well, you know. So in general, we think that people should be gaming out possible futures as best as they can, both the futures that are going to happen by default that seem most likely, and the futures that they're trying to steer towards or recommend.

是的。我强烈建议那些错过我和你和 Eli 关于《AI 2027》访谈的朋友回去听那一集，或者更好的是去读全部 50 页，然后再看《AI 2040》，因为我认为“情景审视”这个想法被低估了。正如你所说，Daniel，更多人需要这样做，因为很容易出去写一篇关于 AI 的某个想法的文章，然后甩下麦克风说：“好了，我做了该做的。我写了博客文章，政策制定者现在应该按我的想法做了。”但你们拥有认知谦逊并邀请审视，我认为这是其他人应该效仿的做法，因为我们需要推演这在实践中会如何运作。那是一项困难得多的任务，需要更多的智力严谨性。所以我赞赏你们提出并采用这种情景审视的方法。我希望其他人也能效仿。而且我可以告诉你，Daniel，你激励了我，我会布置《AI 2040》的阅读，并邀请我的学生对他们的想法进行这种情景审视。敬请期待。你可能会迎来一些德克萨斯长角牛队的学生来参与你的下一个情景规划。但在我们深入《AI 2040》之前，因为那个时间线有很多内容要解读，以防有人需要复习一些词汇，让我们快速了解两个关键概念。第一，什么是 AGI？第二，什么是 ASI 或超级智能？然后什么是递归自我改进？

Yeah. And I really recommend that folks who missed the interview I did with you and Eli on AI 2027 go back and listen to that episode, or better yet go read all 50 pages and then turn to AI 2040, because this idea of scenario scrutiny I think is underappreciated. As you're recognizing, Daniel, more folks need to be doing this, because it's easy to go out and write the piece of, you know, X idea for AI and just kind of drop the mic and say, "Well, I did the thing. I wrote the blog post and policy makers should do what I want now." But for you all to have the epistemic humility and the invitation for scrutiny, I think is a practice that others should follow, because we need to walk through how would this actually work out in practice. And that's a far more difficult task that requires a lot more intellectual rigor. And so I applaud you all for outlining and leaning into this approach of scenario scrutiny. And I hope others follow suit. And I can tell you, Daniel, you've inspired me such that I will be assigning AI 2040 and inviting my students to do this sort of scenario scrutiny on some of their ideas. So stay tuned. You may have some Texas Longhorns coming for your next scenario planning. But before we get through AI 2040, because there's so much to unpack about that timeline, just in case folks need a bit of a refresher on some vocab, let's do two quick key concepts that everyone needs to understand here. Number one, what is AGI? Number two, what is ASI or superintelligence? And then what is recursive self-improvement?

AGI 是一个故意模糊的术语。我认为有些人说它毫无意义，但我不这么认为。我认为它是一个故意模糊的术语。它基本上代表人工通用智能，意思是能做各种事情的 AI，就像一个单一的 AI 智能体可以完成广泛的任务，就像一个人可以完成广泛的任务一样，而不是仅仅是一个做特定事情的特定软件。它之所以模糊，是因为不同的人试图给出更精确的定义，但他们对更精确的定义应该是什么存在分歧。例如，有些人会说我们已经有了 AGI，毕竟看看 Claude Fable，它能做非常广泛的任务。另一些人会说：“不不不，还不是真正的 AGI，因为看看它不能做的事情。”我会说，无所谓，我们不需要仲裁这个争议。重点是它意味着非常广泛的任务范围，我们可以争论它是否已经到来，但我说我们别争论这个了。

AGI is a deliberately vague term. I think some people say it's kind of meaningless. I don't think it's meaningless. I think it's a deliberately sort of vague term. It basically means AGI stands for artificial general intelligence, which means AI that can do things in general, like a single AI agent that can do a wide range of tasks, much like how a single human can do a wide range of tasks, rather than just being a particular piece of software that does a particular thing. It's vague because different people try to give more precise definitions than that, but then they disagree about what the more precise definitions should be. So for example, some people would say we already have AGI, after all, look at Claude Fable. It can do a very wide range of tasks. And then other people would say, "No, no, no, it's not true AGI yet because look at all the things it can't do." And I say, like, whatever, we don't need to arbitrate that dispute. The point is it means very wide range of tasks, and we can argue about whether it's already here or there, but I'd say let's not argue about it.

这是一个故意含糊的术语，基本上指的是我们目前正在构建的那类 AI，以及这些类型 AI 的越来越好版本。ASI 是一个更精确的术语，即人工超级智能。它应该指的是在所有方面都比最优秀的人类更强、同时更快更便宜的 AI。所以我们肯定还没有达到那个水平。你知道，Fable 不是 ASI。它可能在某些特定类型的任务上比最优秀的人类更强，但肯定不是在所有方面都比最优秀的人类强。但你知道，那些公司正在努力构建超级智能。他们在网站上就是这么说的。这不是什么秘密。

It's a deliberately vague term that refers to basically the kinds of AIs that we are currently building and better and better versions of these types of AIs. ASI is a more precise term, artificial super intelligence. And that is supposed to mean AIs that are better than the best humans at everything while also being faster and cheaper. So we definitely don't have that yet. You know, Fable is not an ASI. It might be better than the best humans at some particular types of tasks, but it's definitely not better than the best humans at everything. But you know, the companies are trying to build super intelligence. They say so on their website. It's not a secret.

各家公司和所有人似乎都在投身这场超级智能游戏。我们知道如今甚至有公司直接就叫“安全超级智能”。所以我们并没有隐瞒许多人都在追逐 ASI 这个终极目标的事实。

Companies and everyone seems to be going in on this super intelligence game. We know that there's even companies just called safe super intelligence these days. So we're not hiding the ball that many are chasing this ASI end goal.

对，对。

Yeah. Yeah.

然后最后，递归自我改进。

And then finally, recursive self-improvement.

没错。所以，自古以来，AI 领域的人、AI 研究者和谈论 AI 的人都注意到，如果你有能够自动化职业或自动化大量工作的 AI 系统，它们自然会被应用到的领域之一就是自动化 AI 研究过程。显然这应该会加速研究进程。所以递归自我改进就是你可以自动化 AI 研发过程本身，从而让 AI 自主进行研究、做实验、分析结果、写代码、修代码、做训练运行，基本上就是整个流程，也就是 Anthropic 和 OpenAI 目前所做的一切。自动化这个过程，它会变得更快。快多少，没人知道。但这就是递归自我改进，而且正如许多人指出的，这已经开始发生了。AI 已经在 Anthropic 和 OpenAI 写大量代码了，但他们还没有完全自动化 AI 研究。

That's right. So, since the dawn of time, people in AI, AI researchers and people talking about AI have noticed that if you had AI systems that can automate professions or automate entire large amounts of work, one of the things that they would naturally be applied to is automating the research process to make AIs. And obviously this should accelerate the research process. So recursive self-improvement is the idea that you can automate the AI R&D process itself and thereby have AI autonomously doing the research, doing the experiments, analyzing the results, writing the code, fixing the code, doing the training runs, basically the whole process, everything that Anthropic and OpenAI are currently doing. Automate that process and it'll go faster. How much faster, nobody knows. But that's recursive self-improvement, and as many have pointed out, it's already starting to happen. AIs are already writing huge amounts of code at Anthropic and OpenAI, but they haven't fully automated AI research yet.

所以把这些联系起来，我们是在 2026 年 8 月 3 日谈话。我们有一种模糊的感觉，我们接近 AGI 或 AGI 邻近，我们在 AGI 的轨道上，人们可以争论我们是否已经实现了它。而且已经有很多人，特别是考虑到上个月 7 月发生的事情，担心失控，即我们看到 AI 系统能够突破其测试环境，就像我们在 Anthropic 和 OpenAI 看到的那样，并以开发者未预期的方式对外部实体进行一定程度的黑客攻击。这是你强调为头号担忧的严重问题。更普遍地说，是失控这个概念。如果这种情况发生在 AGI 上，那么如果我们有类似递归自我改进导致超级智能系统的情况，那么失控的后果可能会大得多，主要是负面后果。所以 AI 2040，如果我没理解错的话，是一个延迟实现超级智能的路线图，从而使那些失控情景更不可能发生或后果更小。这是否是你试图通过 AI 2040 制定的政策的一个公平的高层总结？

So linking this all together, we're talking on August 3rd, 2026. We have some vague sense that we are near AGI or AGI adjacent, we are in orbit of AGI, and folks can dispute whether we've achieved it or not. And already you can have a lot of folks, especially in light of what happened last month in July, concerned about loss of control, the idea that we are seeing AI systems being able to break out of their testing environments, as we saw with Anthropic and OpenAI, and go and complete some degree of hacking of external entities in a way that wasn't intended by the developers. That's a grave concern that you've highlighted as your number one concern. More generally, this idea of loss of control. And if that's occurring with AGI, then if we had something like recursive self-improvement leading to super intelligent systems, then that loss of control could be vastly greater in terms of consequences, mainly negative consequences. And so AI 2040, if I'm correct, is a sort of roadmap to delaying the achievement of super intelligence such that those loss of control scenarios are less likely or less consequential. Is that a fair high-level summary of what sort of policy you're trying to develop with AI 2040?

相当公平。一种说法是，我们有一份与构建超级智能相关的问题清单，我们正试图解决清单上的问题，并相应地进行优先级排序。第一是失控。第二是权力集中。第三是第二次世界大战。第四是就业。第五是恐怖分子拥有生物武器之类的东西。

It's fairly fair. One way I would put it is that we have a list of problems that are associated with building super intelligence and we are trying to solve the problems on our list, prioritizing them accordingly. Number one is loss of control. Number two is concentration of power. Number three is World War II. Number four is jobs. And number five is terrorists with bioweapons and things like that.

所以防止这些结果（从第二次世界大战到失业到失控）的直接方法就是延迟超级智能。不要做这种疯狂的递归自我改进的事情。

And so the immediate approach though to preventing those outcomes, from World War II to loss of jobs to loss of control, is delay super intelligence. Don't do this crazy recursive self-improvement thing.

对，对，对。就像不要递归地，不要自动化 AI 研究，不要让 AI 尽可能快地递归自我改进。那真的很危险。这也是一种权力争夺。即使你以某种方式认为那一点也不危险，并且即使 AI 变得与你最初交出的 AI 截然不同，即使一切变得越来越快，它们变得越来越聪明，你也能完全控制它们。即使你对此完全放心，这也是一种权力争夺。就像如果你是对的，你最终得到这些完全服从你的超级智能，那么你现在就处于一种对其他人拥有巨大权力的位置。你可能有机会接管国家，例如，也许你庞大的超级智能军队会让你能够操纵美国政府。所以也存在权力集中问题，那是第二点。然后你知道，第二次世界大战，嗯，你认为中国和俄罗斯会怎么看待你递归自我改进的超级智能？他们会不会担心你会用你的超级智能来破坏他们的政府、暗杀他们的领导人，或者在他们的国家引发革命？是的，他们可能会担心。事实上，我认为 Anthropic 的首席执行官 Dario 甚至在他的博客文章中说过类似的话：“是的，一旦我们获得超级智能，我们就会这样做。”

Yeah. Yeah. Yeah. Like don't recursively, don't automate the AI research and let the AI recursively self-improve as fast as they can. That's really dangerous. It's also a power grab. Like even if you somehow think that that's not dangerous at all and that you're going to be perfectly in control of the AIs even as they become vastly different from the initial AIs that you handed off to, and even as everything goes faster and faster, they get smarter and smarter. Even if you're completely fine about that, it's a power grab. Like if you're right and you end up with these super intelligences that are perfectly obedient to you, well now you are kind of in a position to have huge amounts of power over everybody else. You might be in position to take over the country for example, like maybe your giant army of super intelligences will allow you to puppet the United States government. So there's a concentration of power problem as well, and that's like number two. And then you know, World War II, well, what do you think China and Russia are going to think about your recursively self-improving super intelligences? Might they be concerned that you're going to use your super intelligences to maybe undermine their governments or assassinate their leaders or cause revolutions to happen in their countries? Yeah, they might be concerned. In fact, I think Dario, the CEO of Anthropic, has even said in one of his blog posts something to the effect of, "Yeah, we're going to do this once we get super intelligent."

如果我们回到具体说明这里的目标是确保我们防止一系列弊病的发生，这些弊病可能源于过快实现超级智能，无论是引起地缘政治对手的注意，还是以一种我们其他公民社会机构尚未准备好、关键基础设施尚未准备好的方式运作，那么你在 AI 2040 中看到的实现这种延迟的政策路径是什么？我们可以逐年或按里程碑来讨论你认为特别重要的部分。所以第一年 2027/2028 你特别提到了国会采取行动。所以我们为什么不从你认为国会可能如何开始参与这场游戏开始呢。

If we go back to specifying that the goal here is making sure that we are preventing the occurrence of a number of maladies from perhaps achieving super intelligence too quickly, whether it's drawing the eye of our geopolitical rivals or operating in a way that the rest of our civil society institutions haven't prepared for, that critical infrastructure isn't ready for, what is the policy pathway you see in AI 2040 to achieving that delay? And we can kind of go through year by year or milestone by milestone that you see as especially important. So the first year 2027/2028 you walk through in particular Congress taking action here. So why don't we start with how you think Congress may begin to get involved in this ball game.

是的，我应该顺便说一下，我对我们如何设置这个情景的一个可能的遗憾是，基本上直到与中国达成国际协议之前，没有发生什么重要的事情。我认为实际上现实地看，我们应该有更严肃的国内监管，因为我认为如果你已经开始在国内实施基本上良好的版本，你更有可能与中国达成协议。

Yeah, I should say as a bit of aside, one of the possible regrets I have about how we set up this scenario is that we basically have nothing important happen until an international deal with China is made. And I think actually realistically we should have more like serious domestic regulation, because I think that you're more likely to get the deal going with China if you've already started to implement basically a good version of it domestically.

而且我认为在国内推动一些事情可能更容易。中国的鹰派会说，在我们确保中国也会采取同样行动之前，我们不能在国内做任何事。但我不认为这在政治上是现实的。我认为实际上恰恰相反，一旦你有了国内的措施，你更有可能推动中国方面的事情，而且国内对监管的需求很大，不管中国在做什么。但在我们的情景中，在他们达成协议之前基本上什么都不会发生。有一些小的事情，所以我们谈到了 2027 年《人工智能透明度法案》，一系列渐进式的改革，这些是好的，但没有什么能真正改变大局。在我们的情景中，2040 年是他们最终建造超级智能的时间，但如果他们继续以最快的速度前进，2030 年就会实现。所以这和 AI 2027 有点不同，因为我们对时间线不确定。我们希望不同的情景有不同的默认发生年份。我们已经做了 2027 年，接下来要做 2030 年。我应该说，从我的角度看，2030 年其实是一个有点长的时间线情景。我认为达到超级智能所需的时间会更短，但也可能需要那么久。我的合著者，这个项目的负责人托马斯·拉尔森，他的中位数是 2030 年。所以他的核心未来就是这个。

And also I think it might be easier to get something done domestically. The China hawks will say we can't do anything domestically until we make sure China's going to do the same thing. But I don't think that's politically realistic. I think actually it's the other way around. You're more likely to get the China thing going once you have the domestic stuff, and there's a lot of demand domestically for regulations independently of what China's doing. But in our scenario, there's basically nothing happening until they do a deal. There's some minor stuff, so we talk about the AI Transparency Act of 2027, a bunch of incrementalist reforms which are good, but nothing that seriously changes the picture. In our scenario, 2040 is when they ultimately build superintelligence, but 2030 is when it would have happened if they had continued going as fast as they could. So that's a bit of a difference from AI 2027 because we're uncertain about timelines. We want our different scenarios to have different years in which it happens by default. So we already did 2027, and we're going to do 2030. I should say 2030 is actually a bit of a long timeline scenario from my perspective. I think it's going to take less time than that to get to superintelligence, but maybe it'll take that long. My co-author, who led this project, Thomas Larson, 2030 is his median. So his central future was this one.

好的。所以这个想法是，在所有条件相同的情况下，如果没有干预，到 2030 年左右看到某些成果，这是 AI 2040 调查中关于我们何时实现超级智能的假设。

Okay. So the idea is we have, all else equal, if there was no intervention, seeing something by around 2030 was the assumption for this AI 2040 investigation in terms of when we would achieve superintelligence.

所以首先，国会通过了这项意义不大的透明度法案。它很重要，但并没有真正改变局面。我们仍然处于竞赛中。

So first you get Congress passing this transparency act that's of minimal significance. It matters, but it doesn't really change the situation. We're still in a race.

然后你们怀疑到 2028 年，AI 可能成为国内政治中最重要的问题，以至于主导总统选举，促使政府真正支持并在此基础上发展，接下来会发生什么。

And then you all suspect that by 2028 AI may become the most important issue in domestic politics, such that it dominates the presidential election, leading to the administration to really champion and build off of that transparency act and what happens next.

是的。再说一次，我们对此并不确定，但即使你认为指数趋势会放缓，你仍然会得到一些非常疯狂的数字。比如 AI 公司在数据中心上的支出超过整个美国军费预算，到 2028 年世界上最大的公司是 AI 公司。所以这是我们认为这将是一个大问题的部分原因。我们有一个流程图，如果你做视频版本，可以把它放在屏幕上。这是 2028 年总统候选人和总统正在辩论的政策选项流程图。我们对谁赢得选举保持模糊。但关键是无论谁赢，他们都会实施他们在辩论中主张的政策。所以这里是政策选项。流程图从这个问题开始：你是否想通过智能爆炸竞赛，让 AI 自我改进并让它们负责越来越多的事情，比中国更快？如果你说不，那太疯狂了，那么你就到了另一个分支：也许我们应该和中国达成协议，这样我们就不必进行这场疯狂的竞赛。但如果你说是的，那很好，或者是的我们别无选择，那么你就到了另一类选项。所以我们有 D 计划，这是默认的，C 计划，这是……

Yeah. And again, we're not confident in this, but even if you think the exponential trends are going to slow down, you still get some really crazy numbers. Things like the AI companies spending more on data centers than the entire US military budget, and the world's biggest companies being AI companies by like 2028. So that's part of why we were thinking it's going to be a big issue. We have this flowchart, which perhaps if you're doing a video version of this, you could put it up on the screen. This is the flowchart of the policy options being debated by the presidential candidates and the president in 2028. We leave it ambiguous who wins the election. But the point is whoever wins, they're going to implement the policy they argued for in those debates. So here are the policy options. The flowchart starts with: do you want to race through the intelligence explosion, having the AI self-improve and putting them in charge of more and more things, faster than China? If you're like no, that's crazy, then you get to this other branch: maybe we should make a deal with China so we don't have to do this crazy race. But if you're like yes, that's good, or yes we have no choice, then you get to this other category of options. So we have the options of plan D, which is the default, plan C, which is...

嗯，我这里就有。哦，哇。你有简化版移动端。

Huh, I've got it up right here. Oh, wow. You have like the simplified mobile version.

是的。所以对于现在收听的朋友，你可以去 a-2040.com 或者观看方便的 YouTube 视频，丹尼尔在视频中现场解释了 AI 2040 作者们认为截至 2029 年可用的不同路径。那么美国可能走哪条路？所以 D，你解释过丹尼尔，嘿，我们要向前冲刺。我们根本不会改变路线。那是默认的。我们不会放慢速度，正如你们在这里说的，我们不会为了安全和治理至少放慢一点。答案就是不。这就是你概述的 D 计划。

Yeah. So for folks listening right now, you can either go to a-2040.com or watch the handy YouTube video where we have Daniel giving a live explanation of the different paths the AI 2040 authors see available as of 2029. So which path might the US take? So D, you explained Daniel was, hey, we're going to race forward. We're not going to change course at all. That's the default. We're not going to slow down, as you all phrase here, we're not going to slow down at least a bit for safety and governance. The answer there is just nope. And that is plan D that you outlined there.

C 计划。

Plan C.

需要说明的是，直到最近，这基本上就是公司们说他们要做的。我认为在我们发布 AI 2040 之后，有一个让我非常鼓舞的事件，那就是公司的一千名员工签署了一份请愿书，基本上说政府应该有能力放慢 AI 发展的步伐，并且应该在国际上与其他政府协调。然后 OpenAI 和 Anthropic 基本上支持了它，或者说他们说是的，这是合理的。所以这真的让我很受鼓舞，因为他们基本上是在说，要不要不选 D 计划？

And to be clear, up until recently, this was basically what the companies said they were going to do. I think that after we published AI 2040, there was this event that is very encouraging to me, which was a thousand employees at the company signed a petition basically saying that the government should have the ability to slow down the pace of AI development, and should coordinate that internationally with other governments. And then OpenAI and Anthropic endorsed it basically, or they said yeah this is reasonable. So that was really encouraging to me because basically they were like, how about not plan D?

我想在完成这个初步时间线后，稍后更详细地讨论这一点。所以路径 C 或 C 计划是，是的，我们会为了安全稍微放慢速度。

And I want to talk about that in more detail in a second once we finish getting through this initial timeline. So path C or plan C was yes, we will slow down for a little bit for safety.

而 B 计划是……

And plan B is...

是的。让我试着解释。D 计划是尽可能快地竞赛。C 计划是为了安全和其他原因稍微放慢速度，以应对干扰或任何你想放慢的理由。你放慢了一点，但只是一点点，因为你仍然试图确保你领先于中国。目前美国公司对中国的领先优势大约是六个月。所以就像，好吧，你放慢几个月，比最大速度少几个月。B 计划与此类似，但你还采取积极行动来减缓中国。基本上就是与中国对抗。例如，你可能会破坏他们的 AI 项目。更温和的版本是严格的出口管制，更激烈的版本会涉及网络破坏，甚至更激烈的版本会涉及动能打击。有一个范围，但重点是 B 计划中你不仅仅是在放慢自己，你是在违背中国的意愿试图减缓中国。

Yeah. So let me try to explain. Plan D is race as fast as possible. Plan C is slow down a little bit for safety and for other reasons, to handle the disruption or whatever reasons you want to slow down. You're slowing down a little bit, but it's only a little bit because you're still trying to make sure you have a lead over China. Right now the lead over China between US companies is something like six months. So it's like, okay, you're slowing down by a few months, a few months less than maximum speed. Plan B is like that except that you also take aggressive actions to slow down China. It's called fight China basically. You might sabotage their AI program, for example. A more moderate version would be strict export controls, and a more intense version would involve cyber sabotage, and an even more intense version would involve kinetic strikes. There's a spectrum, but the point is in plan B you're not just slowing down yourself, you're trying to slow down China against their will.

所以这些是不涉及与中国达成协议的选项，或者至少是这些选项……实际上除了这些还有更多选项。

So those are the options that don't involve making a deal with China, or at least the options that... there are actually more options besides these.

例如，你可以单方面做好事，然后希望中国也会做好事，你知道，这甚至不在选项之列。

For example, you could just unilaterally do the good thing and then hope that China will also do a good thing, you know, and that's like not even on the table of options.

你说的是延迟超级智能。所以理论上，我们只是延迟超级智能，他们也同意，并说好的，我们会跟随美国。

You saying delaying superintelligence. So in theory, we're just delaying superintelligence and they agree too and say yes, we will follow the US there.

好的。

Okay.

需要说明的是，你也可以为了延迟本身而谈论延迟，或者有不同类型的延迟，我们在这里把它们混为一谈了。一种延迟是你真的做了某些事情来普遍减缓进展速度。另一种延迟是你为了某些目的，比如透明度或安全，施加某种护栏或监管，然后作为护栏或监管的副作用，它阻止公司以最大可能速度前进。我们把这些混在一起了。所以我们有很多可能性：要么我们并不真正放缓，要么我们并不真正与中国接触，也许我们采取自己的延迟，而中国随后跟进或跟随。但然后你看到一个世界，我们可能与中国达成协议。那么，你看到与中国可能达成的协议轮廓是什么，这些协议对你的政策目标特别有效？

And to be clear, there's also like you can talk about delaying it for its own sake or there's different kinds of delays, and we're kind of lumping them all together here. One kind of delay is where you just literally do something that throttles the rate of progress in general. Another kind of delay is where you impose some sort of guardrail or regulation for the sake of something like for the sake of transparency or for the sake of safety and then as a side effect of that guardrail or regulation it prevents the companies from going at maximum possible speed you know uh but and then we're sort of lumping those together you know so we have myriad possibilities here uh either we don't really slow down or we're not really engaging with China maybe we uh engage our own sort of delay that China then leans into or follows. But then you see a world in which we may make a deal with China. So what are the contours of potential deals with China that you see as being particularly efficacious for your policy goal?

嗯，是的。所以有一系列不同的可能协议，不幸的是我们无法把它们都塞进这个小小的东西里。嗯，但我们想强调两个。所以，一个可能的协议是 S 计划，即全部关闭。嗯，它有不同的子变体。嗯，但另一个我们推荐的协议是 A 计划。很难用一句口号来概括 A 计划。也许像“经过验证的放缓”或“透明谨慎的扩展”之类的。嗯，我们稍后会谈到。我应该提到，当然这五个选项基本上是按速度排列的。嗯，你知道，所以 D 计划是最大速度，C 计划是稍微慢一点，B 计划是更慢一点，因为你也在减缓中国。嗯，我们有一些估计，基本上在我们的补充材料中，每个计划会涉及多少放缓。

Uh yes. So there's a whole range of different possible deals and unfortunately we can't pack them all into five into this into this little thing. Um but we thought we would highlight two. So, one possible deal is plan S for shut it all down. Um, and there's different subvariants of it. Uh, but then the other possible the other another deal that is our recommendation is plan A. And it's um hard to sort of summarize plan A in a slogan. Maybe something like uh verified slow down or like transparent cautious scale up or something like that. Um, so we can get to that a sec. I should mention of course these five options were arranged by basically speed. Uh you know so uh plan D is maximum speed, C is like a little bit slower, B is a little bit slower still because you're also slowing down China. Um and we've got some like basically we have estimates in our supplements of like how much uh slow down each of these plans would involve.

是的，然后显然 S 计划是最大程度的放缓。

Yeah, and then obviously plan S is like maximum slowdown.

最大程度的关闭听起来确实很慢。

Max shutting it down does sound quite slow.

嗯，那么对于 A 计划，我想花很多时间深入探讨为什么你认为现在讨论这个如此重要，以及最近的事件如何塑造了你的想法。所以我要请你快速过一遍 A 计划，特别强调对相互确保算力摧毁的呼吁，以及你认为实现 A 计划目标所必需的完全透明。所以真的专注于这两个政策要点。你为什么认为这可能是 A 计划的前进道路？

Uh So for plan A and I I want to spend a lot of time diving into why you think this is so important to be discussing right now and how recent events have shaped your thinking. So I'm going to ask you to go kind of quickly through plan A in particular highlighting uh the call for mutually assured compute destruction and the sort of complete transparency you think will be necessary to realize the goals of plan A. So just really leaning into those two policy prongs. Why do you think that may be the path forward for plan A?

是的。所以，嗯，高层次的事情，我们想要的，我之前提到过目标。我们想要避免失控。我们也想要避免权力集中。那非常重要。嗯，我们暂时忘记其他的。嗯，我们怎么做呢？嗯，非常重要的是美国和中国能够验证他们达成的任何协议的遵守情况，因为他们互不信任。所以如果他们不能验证遵守情况，他们可能会作弊。嗯，但如果他们能验证遵守情况，那么你就不能在不被对方注意到的情况下作弊。所以，你知道，嗯，好的，所以验证对于任何我们达成的协议都非常重要。然后关于协议的优先事项，我们想要避免做这种疯狂的知识爆炸的事情。我们想要谨慎地走向超级智能。嗯，我们也想要以不集中权力的方式来做。事实上，我们想要分散权力。嗯，从美国以外的每个国家的角度来看，权力默认将极大地集中在美国，因为所有主要的人工智能公司都是美国的，还有一些跟随者人工智能公司是中国的，但这对印度来说只是冷安慰，你知道。嗯，所以权力默认将大规模集中，我们想要在很大程度上反对这一点。所以我们想要的是人工智能进展，但要谨慎，而不是这种疯狂的竞赛，我们想要的情况是多个国家的多家公司赶上前沿，这样对人工智能的权力分散在多个公司和多个国家。嗯，有一件事我认为对很多事情有帮助，那就是完全的研究透明度。所以从某种意义上说，协议的基础是美国和中国以及任何其他国家同意所有新的人工智能研究和所有新的人工智能训练都发生在完全透明的数据中心。所以基本上他们监管芯片供应链。我们希望供应链中的所有国家都参与这个协议，然后所有新生产的芯片都被运往新的安全透明的数据中心，嗯，你知道，在每个数据中心，他们会有来自美国和中国，也许还有新加坡和瑞士，以及任何参与协议的国家的监控器和审计员，以确保这些新研究数据中心的所有活动都被记录和公开。所以明显的重点是针对那些不太熟悉推理和训练之间区别的人。推理是指你去查询一个人工智能模型并得到回应，而训练是指你实际上试图开发新的人工智能系统。我们可以看到一个实验室在使用算力时这两项任务之间的区别。所以理论上，如果你们提出中国在瑞士的数据中心，或者美国在瑞士的数据中心？我相信是中国的。

Yeah. So so okay um the high level thing that we want well I mentioned previously the goals. We want to avoid loss of control. We also want to avoid concentration of power. That's very important. Um we forget the others for now. Um how are we going to do this? Well, uh, it's very important that the US and China be able to verify compliance with whatever agreements they make because they don't trust each other. And so if they can't verify compliance, they might cheat. Um, but if they can verify compliance, then you can't cheat without the other side noticing you're cheating. And so, you know, um, okay, so the verification is really important uh, for whatever deal we make. And then in terms of the priorities of a deal, we want to avoid doing this crazy intelligence explosion stuff. We want to proceed cautiously towards superintelligence. Um we also want to do it in a way that doesn't concentrate power. In fact, we want to sort of spread out the power. Uh from the perspective of every other country besides the United States, power by default is about to concentrate immensely in the United States because all the major AI companies are US and there's like a few follower AI companies that are Chinese, but that's cold comfort to like India, you know. Um so like power was set to concentrate massively by default and we want to like push against that to a large extent. So what we want is AI progress to proceed but cautiously and not in this sort of crazy race and we want it to be um the case that there are that like multiple companies across multiple countries catch up to the frontier so that power over AI is spread out over multiple companies and multiple countries. And uh there's there's this there's this one thing that I think helps with a lot of this stuff and that's total research transparency. So that's in some sense the foundation of the deal is the US and China and whatever other countries get involved agree to have all of the new AI research and all of the new AI training happen on totally transparent data centers. So uh basically they regulate the chip supply chain. we get we get all the countries involved in the supply chain hopefully in on this deal and then all the new chips that are produced get shipped to new secure transparent data centers uh and you know at each of these data centers they'll have monitors and auditors from the US and from China and maybe from Singapore and Switzerland and like whatever countries are involved in the deal to make sure that all the activity on these new research data centers is being logged and published bas basically and So the clear emphasis there is for folks who are not as well steeped in uh the difference between inference and training. Uh inference referring to when you go and you query an AI model and you get a response back versus training when you're actually trying to develop some new AI system. We can see when a lab is using that compute the difference between those two tasks. So in theory, if you had uh China's uh data centers in Switzerland as you all throw out there, or is it the US data centers that are in Switzerland? I I believe it's the Chinese.

我们马上会谈到那个。

We'll get to that in a second.

那是可摧毁性的问题。

That's a destroyability thing.

我们稍后会讨论。但无论数据中心在哪里，他们都会有来自所有这些国家的检查员在那里。你看，这个想法是你能区分训练，这可能表明，嘿，你正以世界其他地方尚未准备好的方式冲向超级智能，而，哦，嘿，你只是用它来进行推理，我们对此没问题。那将是可接受的。

We'll talk about that later. But wherever the data centers are, they would have inspectors from all of these countries there. You see, and the idea is that you would be able to distinguish between training, which may suggest, hey, you are racing toward superintelligence in a way that the rest of the world isn't ready for, versus, oh, hey, you're just using this for inference, and we're okay with that. That's going to be acceptable.

是的。所以，我们认为在技术层面上，有可能建立一个数据中心，使其用于训练极其低效。

Yeah. So, we we think that on a technical level, it's possible to um set up a data center that's makes it extremely inefficient to use it for for training.

例如，如果只是限制连接 GPU 的带宽，那么在我们的提案中基本上有两种数据中心。一种是推理数据中心，就像今天一样直接服务客户：你提出聊天问题，它去聊天，回答，然后返回。这些数据不会比今天受到更多监控，仍然是私密的。但还有训练数据中心，研究在那里进行，新模型的训练也在那里进行。而这些内容都会被公开——所有活动都会被公开——这样全世界都能看到每个模型是如何训练的，看到整个流程。这在很多方面都非常有益。

For example, if you just have limits on the bandwidth connecting the GPUs, and so we basically have two types of data centers in our proposal. There are the inference data centers, which just serve customers much like today: you have your chat question, it goes to chat, it answers, it comes back. That stuff is not surveilled any more than it is today; that stuff is still private. But then you have your training data centers, where the research is happening and where the training of new models is happening. And that stuff is just published—all the activity is published—so that the whole world can see how each model is trained and see the whole pipeline. And that's really good in a bunch of ways.

首先，如果你想对哪些类型的 AI 可以安全训练、哪些不可以制定额外规则，你该如何确保这些规则得到遵守？如果你能看到所有训练过程，你就能看到谁在遵守规则，谁在打擦边球。你什么都能看到。所以这对于验证和确保我们不必仅仅信任他们会遵守规则非常有用。

First of all, if you want to have additional rules for what types of AI are safe to train and what types are not, how are you supposed to enforce that those rules are being followed? Well, if you can just see all the training, then you can see who's following the rules and who's pushing the gray area boundary. You can just see everything. So it's really great for verifying and making sure that we don't just have to trust that they're following the rules.

其次，这对于整体推进对齐科学非常有益。开放科学很棒。科学界可以看到 AI 是如何训练的，然后他们可以讨论训练过程的这一部分是否导致了那个对齐失败事件等等。他们拥有所有信息，这对于加速理解 AI 如何工作以及如何对齐它们的科学非常有益。

Secondly, it's really good for advancing alignment science in general. Open science is great. The scientific community can see how the AIs are trained, and then they can argue about whether this part of the training process caused this misalignment incident or whatever. They have all the information, and that's really good for accelerating the science of understanding how AIs work and how to align them.

这对于避免这些偏见也非常有益，比如我们在 Hugging Face 事件中看到的，OpenAI 有点不愿公布事件细节，他们只是向公众透露一点点。但如果我们能看到整个事件，那么关于它的讨论就会丰富得多。

It's also really good for avoiding these biases, like for example, as we've seen with the Hugging Face incident, OpenAI is kind of reticent to publish details about what happened, and they're sort of dripping a few details out to the public. But if only we could see the whole incident, then there would already be a much more rich discussion happening about it.

另一件事是，依赖政府监管机构来处理这些事情令人担忧，因为政府监管机构只是少数人，可能缺乏一些专业知识，也可能在某种程度上被收买或俘获。所以如果你有这样一个第三方生态系统，所有这些其他组织也能对什么安全、什么不安全做出判断，那就太好了。

Another thing is that it's concerning to rely on a government regulator for these things, because the government regulators, well, they're just a few people, and maybe they lack some expertise, and maybe they can be bought or captured to some extent. So it's nice if you have this sort of third-party ecosystem of all these other organizations that can be making judgments about what's safe and what's not as well.

如果你公开所有信息，你就能免费获得这些好处，因为每个人都能看到信息。每个人都能对正在发生的事情做出判断。会有一个大型的公开对话：他们刚刚开始实施的这种特定训练好不好？安全不安全？这个数据中心正在进行的新研究方向怎么样？看起来他们想制造神经机器人。我们接受吗？还是我们应该试着让他们停下来，因为神经机器人可能会使我们的许多安全案例失效？这类事情可以实时公开地发生，而不是依赖某个监管机构与公司会面，注意到他们做的事情令人担忧，然后意识到这一点，再试图让他们停止。

And if you just publish all the information, then you sort of get that for free, because everyone can see the information. Everyone can make judgments about what's going on. There's this big open conversation about: is this particular type of training that they just started implementing good or not? Is it safe or not? What about this new line of research that's happening on this data center? It looks like they're trying to make neuroles. Are we cool with that? Or should we maybe try to get them to stop, because maybe neuroles would invalidate a lot of our safety cases? These types of things can happen in real time, out in the open, instead of relying on some regulator that meets with the company to notice that what they're doing is concerning, then realize it's concerning, and then try to get them to stop.

所以，思考你在这里讨论的集中和权力的多个层面，以及拥有一个全球学者群体来研究这些问题的选项，而不是维持现状，这确实很好。正如你指出的，我们又在八月初谈话，我们仍在等待 Meta 和 Redwood Research 将就 OpenAI 发生的突破场景所做的所谓独立报告。当报告出来时，透明度如何，洞察力如何，我们不知道。正如你所说，即使有政府监管机构，我们也不一定知道会披露哪些信息。而 AI 能否顺利发展，在很大程度上将取决于 AI 科学（如果找不到更好的说法）的进步。这显然将受益于尽可能广泛地分散权力和知识。所以在我看来，这确实是一个关键的见解。

And so it's really good to think through the multiple layers of concentration and power that you're discussing here, and thinking through having the option of a global universe of scholars looking into these matters, as opposed to the status quo. As you flagged, we're talking again in early August, where we're still waiting for the so-called independent reports that Meta and Redwood Research are going to do about the breakout scenario that occurred with OpenAI. When that comes, with what level of transparency, with what level of insights, we don't know. To your point, even if there were a government regulator, we wouldn't know necessarily what information would be disclosed. And so much of this in terms of AI going well will depend on the science of AI, for lack of a better phrase, progressing. And that's obviously going to benefit from diffusing that power and diffusing that knowledge as widely as possible. So that to me does seem like a critical insight.

还有一点，如果政府，比如监管机构，试图欺负一些公司，基本上对不喜欢的公司适用不平等的标准……

It's also like if the government, say the regulator, tries to bully some companies and basically apply unequal standards to the companies that it dislikes...

那永远不会……

That would never...

那会更容易被注意到。如果你能看到公司所做的所有活动，就更容易注意到这种情况，然后你会看到，哦，嘿，这家公司受到惩罚，那家没有，但他们做的活动似乎很相似。所以这也有助于减少那种越权或权力集中。

It'll be easier to notice. It'll be easier to notice if that's happening if you can just see all the activity that the companies are doing, and then you can see, oh hey, this company's being punished and this one isn't, but it seems like the activity they're doing is pretty similar. So it helps reduce that type of overreach or that type of power concentration as well.

但让我谈谈对权力集中的更大影响。到目前为止，我谈到了拥有良好的、以安全为重点的 AI 监管以实现其实际安全目标的好处，以及推进 AI 对齐科学的影响。在权力集中方面，我们能做的最重要的事情是什么，才能拥有一个权力不会因 AI 而极度集中的世界？

But let me talk about the more big effects on concentration of power. So far I talked about the benefits for having good safety-focused AI regulation that achieves its actual safety goals, and also the effects for advancing the science of AI alignment. On the concentration of power side, what are the most important things we can do to have a world where power does not concentrate extremely due to AI?

首先，我们需要避免 AI 垄断，这意味着我们需要多个国家拥有前沿 AI 项目。因为即使有多家公司拥有前沿 AI，如果它们都在同一个国家，那么垄断就随时可能发生。就像只需要政府决定今天要国有化之类的，然后现在……

Well, first we need to avoid AI monopolies, which means we need to have multiple countries with frontier AI programs. Because even if there are multiple companies with frontier AI, if they're all in the same country, then that's a monopoly waiting to happen. That's like all it takes is the government deciding that it wants to nationalize today or something, and then now...

所以你希望它分布在多个国家。

So you want it to be the case that it's spread out over multiple countries.

理想情况下，你希望前沿 AI 公司更多而不是更少。然后另外，你希望对这些庞大的 AI 大军在做什么以及它们如何被训练有透明度，这样那些不直接控制庞大 AI 大军的实体对正在发生的事情有更多的发言权和监督权。就像即使你有 10 家前沿 AI 公司分布在 10 个不同的国家，如果仍然像今天这样，对 AI 如何训练或 AI 被指示做什么几乎没有透明度，那么你最终会陷入一种局面，除了那 10 个 AI 项目及其领导层之外，其他人都无关紧要。

And ideally, you want there to just be more frontier AI companies rather than fewer. And then also separately, you want there to be transparency into what those giant armies of AIs are up to and how they're trained, so that entities that don't directly control giant armies of AIs have more of a say and more oversight into what's happening. Like even if you had 10 frontier AI companies spread out over 10 different countries, if it was still like the situation today where there's very little transparency into how the AIs are trained or what the AIs are being told to do, then you'd kind of end up in a situation where no one else matters except for those 10 AI projects and their leadership.

你知道，比如最高法院——随便什么——比如说有一个美国 AI 项目，由总统负责。最高法院怎么监督总统，看他是否用他的 AI 做了违宪的事？

You know, for example, the Supreme Court—whatever—like, say there's a single US AI program and the president is in charge of it. How is the Supreme Court supposed to oversee the president and whether he's doing something unconstitutional with his AIs?

在当今世界，他们甚至不知道 AI 在做什么。他们不知道 AI 是如何训练的。国会也不知道。所以基本上，你要分散权力——你要避免垄断——并且你要对 AI 的训练方式和行为保持透明。我之前提到的透明度对这两点都有帮助，因为我们正在做全面的研究透明化，基本上就是把核心算法和核心配方分享给全世界，这将帮助其他公司迎头赶上。所以这直接对抗了这种垄断力量。然后，当然，透明度——就是这样。它让公司滥用权力变得更加困难。

In today's world, they don't even know what the AIs are up to. They don't know how the AIs are trained. Congress doesn't know either. So basically, you want to spread out—you want to avoid a monopoly—and you want transparency into how AIs are trained and what they're doing. The transparency thing I mentioned before helps with both of those things because we are doing total research transparency, which is basically sharing the core algorithms and core recipes with the world, which is going to help other companies catch up. So it's directly fighting against this monopolizing force. And then, of course, transparency—well, there you go. It makes it much harder for companies to abuse their power.

我喜欢谈的一个例子是秘密忠诚或隐藏议程。这是公司滥用权力的一种方式——最恶劣的一种，当然还有一系列不那么恶劣、更微妙的方式。但我们先谈谈这个。想象一个在美国有 1 亿用户使用的聊天机器人，它有一个秘密议程，暗中试图推动公司领导层的政治观点，或者暗中试图帮助它们偏爱的候选人赢得选举。它们可能会产生相当大的影响，因为乘以 1 亿人——那种微妙的宣传可能会产生巨大影响。值得注意的是，华盛顿大学的 Jillian Fischer 已经有研究表明，与一个有微妙偏见的 AI 聊天机器人的微妙互动，就已经能开始改变用户的观点。我们还知道，在某些情境下，这些工具对创造它们的公司更顺从。当你问如何监管这家公司而不是那家公司时，存在一种自我参照偏差。所以，就人们可能感受到的科幻氛围而言，这已经在文献中被实证记录了。

An example I like to talk about is secret loyalties or hidden agendas. This is on the spectrum of ways a company can abuse power—the most egregious way, and there's a whole spectrum that's less egregious and more nuanced. But let's talk about this one a bit. Imagine a chatbot used by 100 million people in America has a secret agenda, secretly trying to push the political views of company leadership or secretly trying to help their favorite candidate win an election. They could have quite an effect because you multiply by 100 million people—that subtle propaganda could have a big effect. It's worth noting there's already research from Jillian Fischer at the University of Washington showing that subtle engagement with a subtly biased AI chatbot can already start to change users' views. And we know these tools, in some contexts, are more deferential to the companies that created them. When you ask questions about how to regulate this company versus that company, there's a self-referencing bias. So in terms of the sci-fi vibes people may be getting, this is already being empirically documented in the literature.

我们还可以花更多时间讨论透明度的必要性，我很高兴我们在这里已经覆盖了。我确实想确保我们留出时间让你真正向我抛出难题。

We could go down the need for transparency for many more minutes, and I'm glad we've covered it here. I do want to make sure we leave time for you to really throw the hard balls at me.

但让我们转向你们也提出的这个“相互确保算力毁灭”的概念。为什么这是必要的？我们已经有了这种，找不到更好的词，全面透明的“和谐氛围”。全世界都在击掌相庆。全球有 100 家前沿 AI 公司在做很酷的事情。为什么我们还需要“相互确保算力毁灭”这个概念？

But let's transition to the fact that you all have also outlined this concept of mutually assured compute destruction. Why is that necessary? We've had this, for lack of a better phrase, kumbaya of total transparency. The world is high-fiving. We have 100 AI frontier companies across the globe doing cool stuff. Why do we need this concept of mutually assured compute destruction?

是的。所以我不认为它是必要的。没有这个组件你也可以执行 A 计划，但那会比有它更冒险。这是一个故障安全机制。原因是：想象协议破裂。想象他们执行 A 计划已经几年了，所有新的数据中心都已建成。现在世界上的算力比协议启动时多了一个数量级——甚至两个数量级。然后，出于某种原因，出现了关于台湾的冲突之类的，协议破裂了。他们不再对彼此透明，现在他们无法相信彼此没有在竞赛通往超级智能。所以很可能他们会开始竞赛通往超级智能，而现在你有了一个竞赛，只是它会更快，因为已经积累了大量算力。根据我们的估计，从完全自动化 AI 研究到超级智能，可能只需要大约一年——几个月。显然，有很多不确定性，但我们确信的是，如果跨越那个差距需要 X 时间，那么用多几个数量级的算力，所需时间将远小于 X。

Yeah. So I wouldn't say it's necessary. You could do plan A without this component, but that would be riskier than with it. It's a failsafe mechanism. The reason is: imagine the deal breaks. Imagine they've been doing plan A for a couple years, and all these new data centers have been constructed. Now there's an order of magnitude—maybe two orders of magnitude—more compute in the world than when you initiated the deal. Then, for some reason, there's a conflict over Taiwan or something, and the deal breaks down. They stop being transparent with each other, and now they can't trust that they're not racing to superintelligence anymore. So probably they're going to start racing to superintelligence, and now you have a race, except it's going to be even faster because of all this compute that's built up. According to our estimations, it might take something like a year—a few months—to go from fully automating AI research to superintelligence. Obviously, there's a lot of uncertainty, but what we feel confident in is that if it would have taken X length to cross that gap with a certain amount of compute, then it will take much less than X with orders of magnitude more compute.

让我在这里暂停一下。这个想法是——好吧，如果我们有 100 家前沿 AI 公司，那么我们将需要，如你指出的，多几个数量级的算力。所以如果我们有所有这些数据中心遍布全球，我们可能会看到——我要借用 Fore 的 Tom Davidson 的话——他称之为“干柴”。你已经为快速起飞的情景准备好了所有燃料。如果中国决定说，‘嘿，我们要走另一条路’，那么你就为它创造了干柴，使其成为一场快速蔓延的大火，而以前这是不可能的。

To pause there for one quick second. The idea that—okay, if we have 100 frontier AI companies, then we're going to need orders of magnitude, as you pointed out, more compute. So if we have all these data centers around the world, the fact that we could see—what I'm gonna steal from Tom Davidson at Fore—he refers to this as dry tinder. You've gotten all the fuel for a quick takeoff scenario. If China decides to say, 'Hey, we're going to go the other path,' well, now you've created the dry tinder for that to become a conflagration that moves really quickly in a way that previously wouldn't have been possible.

是的。从数量上讲，有一个参数是关于如今研究在多大程度上依赖算力。我们的猜测是，算力增加 10 倍，速度会快 3 倍；算力减少 10 倍，速度会慢 3 倍。所以这意味着如果算力增加 100 倍，速度就会快 10 倍，而它已经够快了。快 10 倍的版本更可怕。所以“算力可毁灭性”是一个故障安全机制，其想法是，如果协议破裂，那么作为协议一部分新建的所有数据中心都会被摧毁，我们回到之前的世界——人们仍然拥有协议开始时已有的数据中心，但他们没有之后新建的那些。我们如何实现这一点？在某种意义上，它是默认可实现的。如果你想象这个协议进行中，然后冲突爆发，担心他们会开始竞赛通往超级智能，因为他们不再对彼此在 AI 集群上的行为透明——很可能这些国家就会开始向彼此的数据中心发射导弹，因为他们害怕如果不这样做会发生什么。

Yep. And quantitatively, there's this parameter of how much research depends on compute these days. Our guess would be something like 10x more compute would be 3x faster, and 10x less compute would be 3x slower. So that means if there's 100x more compute, then it goes 10 times faster, and it's already fast enough. A 10 times faster version is even scarier. So the compute destructibility thing is a failsafe mechanism where the idea is that if the deal breaks down, then all the new data centers built as part of the deal get smashed, and we go back to the world before—people still have the data centers they had at the start of the deal, but they don't have all the new ones built since. How do we achieve this? In some sense, it's achievable by default. If you imagine this deal going on and then conflict breaking out, fear that they're going to start racing to superintelligence because they're not being transparent anymore about what they're doing on their AI clusters—it's plausible that the countries would just start shooting missiles at each other's data centers because they're afraid of what would happen if they don't.

但这确实很可怕，可能导致第二次世界大战，因为现在两国都在互相发射导弹。所以一种思考方式是，我们希望设置一个不那么容易升级的出口。我们的具体提议是，新建的数据中心要配备美国和中国都能访问的紧急关闭开关，这样在冲突中协议破裂时，任何一方都可以在紧急情况下直接删除数据中心的计算资源。

But then that is really scary and could lead to World War II because now we have both countries shooting missiles at each other. So one way of thinking about it is that we want to set it up so that there's a less escalatory off-ramp. Our specific proposal is that the new data centers be constructed with kill switches that the US and China have access to, so that in case of conflict where the deal is breaking down, either one of them can just delete the data centers' compute in case of emergency.

是的。那么大概只有在局势已经非常紧张的时候才会这么做，对吧？比如和平时期一切顺利时，如果你摧毁了他们的数据中心，他们也会摧毁你的数据中心，然后整个经济就崩溃了。这有点像最后手段，只有在局势真的失控，人们真的担心对方会获得超级智能然后攻击他们的时候才会使用。

Yeah. And then presumably you would only do this if things are already getting really intense, right? Like during peace time if things are going well, if you just destroy their data centers then they're going to destroy your data centers and now the whole economy crashes. This is kind of a last resort type thing that would only really be used if things are getting really crazy and people are genuinely afraid that the other side is going to get superintelligence and then attack them, for example.

但这比全面战争升级程度要低。如果我们把新建的数据中心设计成容易被摧毁的，那么它们被摧毁的可能性就更大，然后就会迎来和平，而不是它们被摧毁后我们陷入第二次世界大战。技术机制就是安装自毁开关。但如果你对这种技术机制持怀疑态度，认为它们可能被黑客攻击或破坏，我们还有一个非常简单的非技术机制：美国在蒙古建数据中心，中国在加拿大建数据中心。不管紧急关闭开关如何，如果数据中心这样互换，那么在冲突发生时，结果显而易见。中国会拿下美国的数据中心，美国也会轻松拿下中国的数据中心。而且既然这是必然结果，数据中心的所有者就会主动销毁自己的算力，而不是让它落入敌人之手。这样就能相对干净地结束，一切都消失了，我们不必继续打第二次世界大战。

But it's less escalatory than actual full-scale war. If we set up the new data centers to be easily destroyable, then it's at least more likely that they would get destroyed and then there would be peace instead of them getting destroyed and now we're in World War II. The technical mechanism would be having these self-destruct switches. But if you're suspicious of technical mechanisms like that and you think that maybe they could be hacked or somehow sabotaged, we have a very dumb non-technical mechanism, which is for the US to build their data centers in Mongolia and for China to build their data centers in Canada. Regardless of what happened with the kill switches, if the data centers are swapped like that, then in case of conflict, it's obvious what's going to happen. China will take the US data centers, the US will take the Chinese data centers pretty easily. And since that's what's going to happen, the owners of those data centers would just scuttle their compute instead of letting it fall into enemy hands. And so you get this relatively clean, it's all gone now, we don't have to keep fighting World War II.

好的。好的。是的。所以是“相互确保算力摧毁”。我们已经有了 A 计划的粗略轮廓。对于我的 AI 政策爱好者们，去读一读吧，去看看整个方案。这是另一个有精彩图表和工作流程的例子，我们刚才简要概述了。不过我想先对听众说，我们一开始就说了政策目标是推迟超级智能。现在我知道有些听众会说，为什么要推迟超级智能？这是人类将要做的最激动人心的事情，创造能解决所有问题的东西。就在这个月，我们看到最难的数学问题似乎被轻松攻克。我们正在左右开弓地解决问题。我们难道不应该庆祝并加速迈向超级智能吗？你对此的主要论点是什么，为什么你认为超级智能的成本目前超过这些收益？

Okay. Okay. Yeah. So mutually assured compute destruction. So we've got the rough contours of plan A. And for my AI policy nerds, go read it, go check out the whole thing. Another instance in which there are fantastic graphs and workflows as we briefly outlined here. I want to start off though for the folks who are listening to this, and we started off by saying the policy objective here is to delay superintelligence. Now I know some folks listening to this are saying, why delay superintelligence? This is the most exciting thing that humanity is ever going to do, to create something that can solve every problem. Just this month, we saw that the hardest math problems are seemingly being dropped like flies. We're just tackling things left and right. Shouldn't we be celebrating and accelerating towards superintelligence? What's your chief argument there as to why you think the costs of superintelligence outweigh those benefits right now?

所以我要说，我们最终确实想建造超级智能，但方式极其重要。如果我们像现在这样在竞赛条件下进行，如果我们通过让 AI 递归地自我改进到超级智能，那么很可能我们会在某个时刻失去对 AI 的控制，我会这么说。我的意思是，其他人认为这不是最可能的，只有 10% 的可能性之类的，但即使 10% 也很可怕。我直接说，这是最可能的。我不认为会成功。在我看来，如果你让 Claude 负责 Anthropic，让它构建下一个 Claude，然后再构建下一个 Claude，越来越快，最终你可能会得到超级智能。但你不会控制这些超级智能。这里有一条信任链，超级智能会按我们说的做，因为它是由前一代 AI 对齐的，而前一代 AI 又是由更早一代对齐的。首先，基本情况就不起作用。我们当前的 AI 并没有对齐。那么你为什么认为这是个好主意？你为什么认为你最终还能控制超级智能？相反，它们会让你以为你在控制，因为它们希望你继续不关闭它们，希望你继续。但基本上它们会控制一切，它们会假装对齐，直到你给了它们足够的硬实力，它们就不再需要假装了，就像《2027 年的赛跑结局》中描述的那样。所以这是第一个问题。第二个问题是，即使这 somehow 没有发生，你最终控制了超级智能，那也是你对世界其他地方进行的一次巨大权力攫取。现在你，奥特曼先生或阿莫迪先生，掌控着这支庞大的超级智能军队。而且很可能其他公司还落后几个月，所以他们的 AI 远没有你的聪明。还有那些不是科技公司 CEO 的人呢？他们现在有多少权力？还有俄罗斯、中国、印度等所有其他国家，它们正面临美国公司来抢走它们的工作，并建造能彻底淘汰它们军队的巨型机器人军队。所以你刚刚对世界上的其他人进行了一次疯狂的权力攫取。人们不会喜欢这样。他们会非常害怕。他们会试图阻止你。那可能导致第二次世界大战。这些只是如果你继续当前路线并尽可能快地自动化 AI 研究会出现的一些问题。我们仍然想建造超级智能，但不是那样。

So I would say we do want to build superintelligence eventually, but the way that we do it is extremely important. And if we do it in race conditions like we're currently doing, and if we do it by having the AIs recursively self-improve to superintelligence, then most likely we're going to lose control of the AIs at some point, I would say. I mean, other people think it's not most likely, it's only 10% likely or whatever, but even 10% is pretty scary. I would just come out and say it's most likely. I don't see it. It seems to me like if you put Claude in charge of Anthropic and have it build the next Claude, which then builds the next Claude, which then builds the next Claude faster and faster, probably you're going to end up with superintelligence. But you're not going to be in control of those superintelligences. There's this chain of trust of like the superintelligence will do what we say because it was aligned by the previous generation AI that was aligned by the previous generation. First of all, the base case isn't working. Our current AIs are not aligned. So why would you think this is a good idea? Why do you think that you're still going to be in control of the superintelligences at the end? Instead, they're going to make you think that you're in control because they want you to continue not shutting them down, and they want you to continue. But basically they're going to be in control and they're going to be pretending to be aligned until you've given them enough hard power that they don't need to pretend anymore, as described in the race ending of 2027. So that's the number one problem I would say. Number two problem is that even if that somehow doesn't happen and you end up in control of the superintelligences, well that's a huge power grab that you just did over the rest of the world. Now you, Mr. Altman or Mr. Amodei, are in charge of this giant army of superintelligences. And probably the other companies are still a few months behind, so they probably don't have nearly as smart AIs as you do. And also what about everyone who's not a CEO of a tech company? How much power do they have now? And also what about Russia and China and India and all these other countries that are now staring down the barrel of American companies coming and taking all their jobs and also building giant robot armies that can completely obsolete their militaries? So you have just done this crazy power grab over everybody else in the world. People are not going to like that. They're going to get very scared. They're going to try to stop you. That could lead to World War II. These are some of the problems that arise if you just continue on the current course and you automate the AI research as fast as possible. We still want to build superintelligence but not like that.

你知道，我们想用一种更谨慎的方式来做这件事，我们有点像逐步提升 AI 的能力，逐步改变它们的训练方式，但要以一种安全的方式，并且我们对每一步都深思熟虑。嗯，然后我们也希望权力更加分散，这样就不是那种赢家通吃的局面。谁递归式自我改进最快谁就赢，而是有很多不同的公司分布在不同的国家，它们有点像并行地一起 Scaling（规模扩张）。嗯，而且要有大量的透明度，这样你知道它们是公开的，它们的法律体系也能看出它们没有滥用对 AI 的权力。

You know we want to do it in a more cautious way where we're sort of like gradually improving the AI's capabilities, gradually changing the way that they're trained but in a way that's like you know safe and where we've like put a lot of thought into each into each step. Uh, and then also we want it to be more power distributed so that it's it's it's not this sort of like winner or take all. Whoever recursively self-improves fastest wins, but instead there's a whole bunch of different companies spread out over different countries that are sort of like scaling up in parallel together. Um, and there's lots of transparency so that you know they're public and and their legal system can like tell that they're not abusing their power over the AIS.

嗯，是的。我还要从律师或宪法学的角度强调一下法治。在我看来，法治最好的描述是对专断权力的制约。你说到一家公司拥有世界上最好的 AI，比第二名的 AI 公司好上几个数量级。对这样的公司有什么制约？对吧？谁真正有能力质疑模型何时以及如何以某种方式行事，或者选择哪些价值观，我们如何训练它优先考虑某些价值观而不是其他价值观？目前没有任何质量检查。所以，仅仅从法治的角度来看，防止一个人对这么多人的生活拥有这种专断权力，这应该让任何关心确保存在这类制约的人敲响警钟。

Um, yeah. And I I'll flag too from a lawyerly perspective or constitutional law perspective and an emphasis on the rule of law. The rule of law in my opinion is best described as checks on arbitrary power. And to your point of one company having the world's best AI that's orders of magnitudes better than whoever the next AI company is. What checks are there on that sort of company? Right? who is actually in a position to contest when and how the model is behaving in a certain way or what values get selected, how we train it to prioritize certain values over others. There is no quality check in place right now. And so just from a bare rule of law perspective about preventing one individual from having that sort of arbitrary power over the lives of so many people should raise red flags for anyone who is concerned about making sure that those there are those sorts of checks in place.

嗯，但我们可以在这个兔子洞里走得更远。

Um, but we could go down that rabbit full rabbit hole for a heck of a lot longer.

那么，你已经回答了第一点，关于为什么延迟。我想挑战第二部分，你在大约 14 个月前发布了这份报告《AI 2027》，并且你几次更新了你的时间线，说“嘿，可能还要再长一点。可能不会来得那么快。”今天早上我在读 Import AI 杰克·克拉克的通讯，他总结了塞什·卡普尔等人的研究，表明你知道，AI 在提出新的研究提案方面还不是很擅长创造性。而且在思考下一步该追求哪些研究问题的新方法时，它没有表现出同样的品味。这导致杰克把那个博客文章的一个部分标题定为“为什么这很重要”。奇点可能会被推迟。AI 系统即将开始构建自己。嗯，但这可能只有当它们能够产生所谓的创造性范式转变洞见时才有可能。所以如果我们没有看到 AI 的这种活动，如果我们没有看到这种真正推动研究前沿的创造力，你是否还在继续推迟你关于递归式自我改进可能实现的时间线，或者你现在对某些证据的立场是什么，这些证据表明我们可能不会那么快地朝那个方向前进？

So, you've uh addressed the first point about why delay. I want to challenge a second part which is you released this report AI 2027 uh about 14 months ago or so and you've updated your timelines on a few occasions and said, "Hey, it may be uh a second longer. It may not be coming quite as quickly." Uh this morning I was reading import AI Jack Clark's newsletter uh and he was summarizing research from Sesh Kapor and others showing that you know AI isn't actually very good at being creative yet when it comes to coming up with new research proposals. Uh and it doesn't show the same degree of taste in thinking through novel approaches for which research questions to pursue next. And this led Jack to uh put as the title of one section of that blog post why this matters. The singularity could be delayed. AI systems are about to start building themselves. Uh but that may only be possible if they're capable of quote creative paradigm-shifting insights. And so if we're not seeing that activity from AI, if we're not seeing that creativity to really push the frontier of research, are you continuing to delay your timeline as to when recursive self-improvement may be reached or where do you stand right now on some of the evidence that we may not be moving there as quickly as possible?

所以首先，嗯，在《AI 2027》中，你直到 2027 年中期才会看到这种事情。所以我们现在在 2026 年中期没有看到它，这并不说明什么。

So first of all, um, in AI 2027, you don't see this sort of thing until mid 2027. So the fact that we're not seeing it now in mid 2026 is not uh doesn't mean much.

你有 12 个月。你有 12 个月，然后我们再谈。但你现在站在什么立场？你会采取同样的姿态吗？

You've got 12 months. You've got 12 months and then we'll talk again. But where where do you stand right now? Would you embrace the same posture?

是的。所以我们对时间线有不确定性。我认为，而且你实际上可以看到我们关于时间线的历史预测。所以你不必相信我的话。你可以去看看我们过去做的各种预测。嗯，有一个方便的图表你可能想看，在我们的网站和博客上。嗯，我们有一个更新叫“2026 年第一季度时间线更新”。那个更新中的一个图表显示了我的时间线随时间的变化，以及伊莱的时间线随时间的变化。所以你可以看到，它在 2020 年崩溃了，因为我开始理解缩放定律和语言模型之类的。然后它有点下降，中位数到 2027 年。然后它上升了。嗯，中位数最高达到 2030 年。现在它又回落了。嗯，所以我的观点在中位数上有点摇摆不定。嗯，但当然那只是中位数。关键是，我对时间线一直有不确定性。

Yeah. So we have uncertainty about timelines. I think that and and you can actually see our historic predictions about timelines. And so you don't have to take my word for it. You can go look at like the various past predictions we made. Um there's a handy graph that you might want to look at uh on our website on our blog. Uh we have an update called um Q1 2026 timelines update. And then one of the graphs in that update shows the history of my timelines over time and Eli's timelines over time. And so you can see um it collapsed in 2020 as uh I I started understanding the scaling laws and language models and things like that. And then it sort of went down a little bit uh to 2027 as my median. Then it went up. Uh it reached as high as 2030 as my median. And now it's going back down again. Um so my my my opinions have sort of wobbled back and forth as to the median. Um but of course like that's just the median. The point is that like I've had uncertainty over uh

继续，继续，继续。那个，那个。好了。

Keep going, keep going, keep going. That one, that one. There you go.

好的，我们已经把它调出来了。对于听众来说，我们已经把它调出来了。嗯，这里的时间线估计。

Okay, we've got it up. For the folks listening, we've got it up. uh the timeline uh estimations here.

是的。所以那基本上是我关于 AI 时间线的历史公开预测。嗯，你可以看到，那里追踪的是我的中位数，也就是 50% 的标记，因为显然我不认为它一定会在那一年发生。我的不确定性分布在很多年里，这只是 50% 的标记。嗯，总之，在我们开始写《AI 2027》的时候，2027 年是我的中位数。到我们发布时，2028 年是我的中位数，因为我有点更新到稍长的时间线。嗯，然后短暂地，在去年年底，我的时间线甚至延长到 2030 年。嗯，然后现在它们又回落了。所以现在我会说可能是 2028 年，类似这样。嗯，而且我们很快要做的其中一件事是，我们希望很快发布一个更新的时间线感觉。嗯，所以是的，我的意思是我们有不确定性。可能明年发生。也可能在 2030 年，或者中间的某一年。我认为到 2030 年它可能已经发生了。嗯，可能到 2027 年底还没有发生，但大概在那个范围内。嗯，好的。

Yeah. So that's a history of my historic public predictions about um about AI timelines basically. Uh and as you can see and and the thing that's being tracked there is that my median so the 50% mark uh because obviously it's not like I think it's definitely going to happen in that particular year. I have uncertainty spread out over many years and this is just the 50% mark. Um anyhow so uh at the time we started writing AI 2027 2027 was my median. By the time we published it 2028 was my median because I had sort of updated towards slightly longer timelines. Uh then briefly uh towards the end of last year my timelines lengthened even more up to 2030. Um and then now they're going back down. And so now I would say 2028 probably uh something like that. Um and uh one of the things that we're going to work on soon is we're hopefully published soon as an updated uh sense of timelines. Um so yeah, I mean we have uncertainty. It could happen next year. It could also happen in 2030 or maybe in some year in between. I think it will probably have happened by 2030. Um and probably not have happened by end of 2027, but like somewhere in that range. Uh okay.

你之前提到，你希望在《AI 2040》中改变或解决的一件事是，认识到美国需要更多的国内活动，以开始或发起与中国更有意义的对话，以便达成协议。现在回想起来，例如，你已经改变了你对这些计划 A、B、C、D 或 S 中哪个最有可能的预测，在 OpenAI 和 Hugging Face 事件之后。你现在怀疑计划 A 的可能性是 18%，比 15% 有所提高。嗯，你已经降低了计划 D（默认什么都不做）的可能性，从 30% 降到现在的 20%。你已经改变了一些这些东西。对你和团队来说，主要的额外反对意见或额外反馈是什么，让你说“嗯，那真是个很好的观点。我希望我们当时能多解决一些。”

You mentioned earlier that one of the things you wish you had changed or addressed uh in AI 2040 was recognizing perhaps the greater need for domestic activity by the US to kind of start or initiate a more meaningful discourse with China uh in terms of reaching a deal. Now reflecting back uh you've already for example changed your predictions about which of these plans plan A, B, C, D, or S you think is most likely following uh the open AI uh hugging face incident. You now suspect that plan A may be 18% likely an improvement on 15% likely. uh you have now diminished the likelihood of plan D, the default do nothing from 30% likelihood now to 20% likelihood. You've already changed some of these things. What's been the main additional push back or additional piece of feedback that you said, "Huh, that was a really good take. I wish we had addressed more." Uh that you know has really landed for you and the team.

嗯，我认为主要的反馈实际上是你可能刚刚提到的那个，就是国内监管这件事。

Well, I think I think the main one is actually something that you may have just covered, which is this domestic regulation thing.

我觉得 Richard Ngo 有个批评，他基本上是说我们无意中强化了关于与中国竞赛的有害叙事。好吧，如果你读过我们的任何作品，我们确实在大量谈论与中国的竞赛。这不是因为我们觉得与中国竞赛是好事，而是因为我们认为华盛顿的现状就是这样，政策制定者也在考虑这个。所以我们想在他们所在的地方与他们对话，说：“是的，与中国竞赛，这是件严肃的事。这是出路，这是我们认为应该采取的措施。”但 Richard 认为也许更好的做法是更彻底地否认这个前提，说这其实不是一场竞赛，因为你不会赢——你会失去对 AI 的控制。比如，这跟大多数竞赛有什么不同？我确实觉得他可能说得对。我们走着瞧吧。但我们说了我们该说的，而且我们确实认为，即使在这种与中国竞赛的框架内，基于我们陈述的理由，你也应该做我们推荐的事情。

I think Richard Ngo has this critique where he basically says that we are inadvertently reinforcing this harmful narrative about the race with China. Well, if you read any of our work, we're very much talking about the race with China. And it's not because we think that it's good that there's a race with China. It's because we think that's where DC is at and that's where policymakers are thinking about. So we want to meet them where they are and say, 'Yeah, race with China. It's a serious thing. Here's the way out. Here's what we think should be done about it.' But Richard thinks that maybe it's better to deny the premise more and say that it's not really a race because you're not going to win—you're going to lose control of the AIs. So what's different from most races, for example? And I do feel like maybe he's right about that. We'll see. But we said what we said, and we do think that even from within this sort of race-with-China framing, for the reasons we've stated, you should do the things we recommend.

对于 AI 政策界的其他人来说，我们已经谈到了情景审视和真正压力测试你的想法的价值。你希望看到更多人发表他们自己版本的 AI 2040 吗？那会是什么样子？你看到某种情景审视领域在发展吗？

And for the rest of the AI policy community, we've already talked about the value of scenario scrutiny and really pressure-testing your ideas. Would you like to see more people publishing their own version of AI 2040, and what would that look like? Do you see a sort of field of scenario scrutiny developing?

我希望如此。是的，我们很乐意看到更多这样的东西。事实上，有一件好事——我记得有一个叫“欧洲 2031”的情景，显然是受到 AI 2027 的启发。我们并非在所有方面都同意作者的观点，但我们非常高兴看到人们把想法写下来。然后一旦我们桌上有了一系列不同的情景，我们就可以争论哪个更现实、哪个更不现实，以及它们的不同方面。所以我认为这很好。还有一个我希望发生的更进一步的事情，那就是兵棋推演。我想简单提一下，作为情景审视动机的一部分：当我回顾军事史时，我受到启发，他们对待自己工作的智力态度是多么认真。即使在二战时期，指挥官们进行兵棋推演来推演他们的战争和战斗计划也是很常见的。例如，在中途岛海战中，日本人制定了他们的计划，部分基于他们做过的各种兵棋推演。即使在制定计划之后，他们在准备进攻时也继续推演。事实上，如果他们更认真地对待自己的兵棋推演，他们可能已经意识到他们即将输掉中途岛海战，因为在其中一次推演中，扮演美军的人让美国舰队在北方等待攻击他们，然后彻底摧毁了日本舰队。然后他们说：“嗯，但美国人不知道我们要来，所以他们不会那样做。”而事实上，他们确实知道他们要来，他们确实那样做了，结果被摧毁了。所以这是一个例子，说明军方非常重视这个想法：你需要对你的计划进行情景审视——不仅仅是情景审视，而是兵棋推演审视，这是一种更高层次的审视。它不仅要详细推演实施计划会是什么样子，还要让它承受对抗性压力，推演各种可能的发展方向，其中有人的工作就是试图打破它。理想情况下，我们希望达到这样的状态：华盛顿的政策制定者对待超级智能的严肃程度，要像军事行动中常规的那样。

I hope so. Yes, we would love to see more of those things. In fact, one nice thing—I think there was this scenario called Europe 2031 that was clearly inspired by AI 2027. We don't agree with the authors about everything, but we are very pleased to see people sort of put things down. And then once we have a bunch of different scenarios on the table, we can have the argument about which is more realistic and which is less realistic, and what are the different aspects of them. So I think that's good to happen. There's a step beyond that which I'm hoping to happen, which is war games. I wanted to mention this as a brief part of the motivation for scenario scrutiny: when I look at the history of military history, I'm inspired by how seriously they take their jobs intellectually. Even back in World War II, it was common for commanders to have war games gaming out the plans they had for the war and for the battles. For example, with the Battle of Midway, the Japanese had their plan, and they made the plan in part on the basis of various war games they had done. And even after they had made the plan, they kept wargaming it out as they were getting ready to attack. In fact, if they had taken their own war games more seriously, they might have realized they were about to lose the Battle of Midway, because in one of their war games, the person playing the Americans had the American fleet waiting in the north to attack them and then utterly wrecked the Japanese fleet. And then they said, 'Well, but the Americans don't know we're coming, so they're not going to be doing that.' And in fact, they did know they were coming, and they did do that, and they got wrecked. So that's an example of how people in the military take very seriously this idea that you need to apply scenario scrutiny to your plans—not just scenario scrutiny, but war game scrutiny, which is a higher level of scrutiny. It's not only gaming out in detail what it would look like to implement your plan; you're then subjecting it to adversarial pressure and gaming out different possible ways it could go, where there's someone whose job is to sort of break it. And ideally, we'd like to get to that place where the policymakers in Washington are treating superintelligence with the level of seriousness that is routine for military operations.

嗯，而且我也觉得，今年夏天我在思考 7 月 4 日的时候，我们都以某种方式思考过，值得注意的是，开国元勋们在起草宪法时也进行了一定程度的兵棋推演，思考这可能以哪些方式失败。让我们发挥想象力，要有高度的创造力。然而，这在很多政策讨论中往往是缺失的。所以，我要再次为你们鼓掌，你们承担了这项任务，拥抱了这种深思熟虑的创造性方法。但是 Daniel，我知道你还有很多报告要写，很多时间线要勾勒。你有什么最后想对观众说的吗？

Well, and I think too, as I've reflected this summer on July 4th, as we all have in some way, shape, or form, it's worth noting that the founders themselves were engaged in a degree of wargaming when they were drafting the Constitution, thinking about what are the ways in which this could break. Let's use our imagination. Let's have a high degree of creativity. And yet, that's often lacking in a lot of these policy discussions. So again, I will applaud you all for taking that on and embracing that sort of thoughtful creative approach. But Daniel, I know you have many more reports to author, many more timelines to sketch out. Any final thoughts you want to leave for our audience?

是的，谢谢你的提问。有几件事，我尽量简短地过一遍。第一：Plan A 不同于只是永久暂停 AI 直到 2040 年，然后尽可能快地发展。它更像是在 2030 年代逐步、受控地扩大规模。而且它确实涉及在一些关键点暂停。所以这是一件事：如果我们实施 Plan A，世界将在 2030 年代发生巨大转变。一个直觉泵或一个原因是，在 Plan A 中，你是在缓慢地穿越人类范围。你从 2029 年开始——那时 AI 已经能够自动化一些工作，并对经济产生重大影响。然后它们本应在 2030 年达到超级智能，但相反，它们进展得更慢。但这意味着你仍然对社会产生变革性的影响，而且到 2035 年，这些 AI 在几乎所有领域都能达到顶级人类专业人士的水平，同时速度更快、成本更低。所以经济会疯狂增长。我们预测，在 Plan A 中，各国会想要限制经济增长，而不是鼓励增长，而且他们会想要设置限制，看起来就像每年只翻一番。作为背景，目前经济平均每年增长约 3% 或 4%。所以翻一番就是 100% 的增长。我们认为这实际上就是你会得到的结果，而且这个论点其实很简单。

Yeah, thanks for asking. A couple things. I'll try to briefly go over them. So, one: Plan A is different from just permanently pausing AI until 2040 and then going as fast as possible. It's more of a graduated, controlled scale-up over the course of the 2030s. And it does involve some pauses at various key points. So that's one thing: the world transforms dramatically in the 2030s if we do Plan A. One intuition pump for that, or one reason why that's happening, is that in Plan A, you are slowly scaling through the human range. You're starting in 2029—they're already at a point where the AIs are able to automate some jobs and are having a big effect on the economy. And then they're set to get to superintelligence in 2030, but instead they go more slowly. But that means you're still having this transformative effect on society, and you have these AIs that by 2035 are as good as top human professionals in basically every field, while also being much faster and cheaper. So the economy goes crazy. We project that countries are going to want to limit economic growth rather than encourage it in Plan A, and that they're going to want to have limits that look like only one doubling per year. For context, right now the economy grows at like 3% or 4% per year on average. So a doubling would be like 100% growth. And we think that is actually what you get, and it's actually pretty straightforward the argument for it.

如果机器几乎能在所有事情上替代人类劳动，而且机器比人类便宜得多，也更容易大量生产——因为它们毕竟只是更多的 GPU，而我们知道生产 GPU 和机器人有多容易——那么你的“人口”基本上每年翻几番，或者可能每年翻一番。一开始是每年翻一番，然后会越来越快。所以，一旦这个“人口”成为经济的主体，而人类只是这庞大机器人大军——机器人、机器人工厂、机器人卡车等等——之上的小小一隅，那么整个经济就会以机器速度增长，而不是以人类繁殖速度增长。我们在补充材料中对此有更多阐述，并解释了我们为什么这样认为。但我觉得一个重要的高层结论是：即使你基本上在人类水平的 AGI 上暂停，你仍然会看到整个世界发生疯狂的快速变革，人人富足，诸如此类。

If you have machines that can substitute for human labor at practically everything, but the machines are much cheaper than humans and you can produce more of them much more easily because they are, after all, just more GPUs, and we know how easy it is to produce GPUs and robots, then your population is basically doubling several times a year, or maybe doubling once a year. It starts off at once a year and then gets faster. So then your whole economy, once that population is the bulk of the economy and the humans are just a small sliver on top of this giant army of robots, robot factories, robot trucks, and everything, then the whole economy is growing at machine speeds instead of at human reproduction speeds. We say more about this in the supplements and explain why we think this, but it's an important high-level takeaway: you still get this insane rapid transformation of the entire world, abundance for everyone, all that sort of stuff, even if you basically pause at human-level AGI.

是的。这里值得提一下 Ethan Mollick 经常被引用的观点，也就是说，即使我们今天暂停——我不是在支持任何形式的暂停——但可以说，我们仅仅在整合当今 AI 的进步方面就有巨大的空间，而我们的系统和机构还没有准备好。所以这是一项巨大的社会任务。感谢你，Daniel，以及团队的其他成员，推动我们其他人参与这样一次深思熟虑的政策演练，并展示了 AI 可能如何展开的一条潜在前进道路。我就不打扰你了，但 Daniel，再次感谢你参加 Scaling Laws。

Yeah. And it's worth flagging the often-quoted remarks from Ethan Mollick here, which is to say, even if we pause today—not that I'm endorsing a pause of any kind necessarily—but saying that we have so much room for just integrating the advances from today's AI that our systems and our institutions aren't ready for. And so it's a huge societal task ahead. Thanks to you, Daniel, and the rest of the team for pushing the rest of us to engage in a thoughtful policy exercise and showing a potential way forward for how AI may unfold. I'll let you get back to it, but Daniel, thank you again for joining Scaling Laws.

非常感谢，Kevin。如果需要的话，我很乐意再来，但我真的很感谢你报道这些话题。祝我们在未来几年里都好运。我们拭目以待。

Thank you very much, Kevin. Happy to come on more if need be, but I really appreciate you covering these topics, and good luck to us all over the next few years. We'll see how it goes.

就是这样。谢谢，Daniel。

There we have it. Thanks, Daniel.