OpenAI 的 Astra 模型更难监控：Thomas Larsen 与 Daniel Kokotajlo 谈暂停 AI

好的，我们回来了。我们正在直播，嘉宾是 AI Futures Project 的 Daniel Kokotajlo 和 Thomas Larson，他们共同撰写了 AI 2027、AI 2040 Plan A、AI Futures Model、AI Futures Substack。这些都是非常好的内容。

All right, we are back. We're live with Daniel Kokotajlo and Thomas Larson of the AI Futures Project, who co-wrote AI 2027, AI 2040 Plan A, the AI Futures Model, the AI Futures Substack. It's all very good stuff.

你应该读一读。

You should read it.

Daniel、Thomas，欢迎来到 MTS。

Daniel, Thomas, welcome to MTS.

感谢邀请我们。

Thank you for having us.

当然。我们先来解释一下本周早些时候看到的监控新闻。《The Information》报道称，Astro 正在使用一种新的循环深度技术，这可能使其比过去的 OpenAI 模型更难监控。最近，Astro 系统卡已经发布，提供了更多关于模型可监控性的信息。那么，这里到底发生了什么，你认为有多令人担忧？

Absolutely. So, we'll start by explaining the monitor news that we saw earlier this week. The Information reported that Astro is using a new recurrent depth technique that might make it less monitorable than past OpenAI models. More recently, the Astro system card has been released with more information on the monitorability of the model. So, what exactly happened here and how concerning is it do you think?

所以我觉得我其实还没有真正读过系统卡。我不知道你读没读，Thomas。所以关于那个我没什么可说的，但从阅读《The Information》的文章以及 Yakob 发布的澄清来看，听起来他们确实开发了这种新技术，允许一定程度的循环深度，但他们把深度限制在不算太糟或不算太大的程度。他们认为这也许没问题，但我要说，从更广阔的视角来看，这仍然是朝着令人担忧的方向迈出的重要一大步，原因我们在许多不同地方都描述过，包括 AI 2027，还有大约一年前与一些 OpenAI 人士合著的一篇立场文件，关于思维链可监控性的重要性，你知道，如果他们能以某种方式……

So I think I'm still like I haven't actually read the system card yet. I don't know if you have Thomas. So I don't have things to say about that, but from reading the Information article and then the clarification that Yakob posted, it sounds like they have indeed developed this new technique that allows some amount of recurrent depth, but they're like limiting the depth to not that bad or like not that much. Which they think is maybe fine but I would say that from the broader perspective this is still a significant large step in the concerning direction for reasons that we've described in many different places including AI 2027 and also in a position paper co-authored with some of these OpenAI people about a year ago about the importance of chain of thought monitorability and you know if they can somehow

所以 Yakob 的辩护是，深度只是 GPT-4 的两倍，但我有点愤世嫉俗地预测，一年后深度会更大，你知道。而且相应地，似乎有证据表明这个模型的可监控性显著低于之前的模型。所以也许这种架构变化与此有关。

So Yakob's defense was that the depth is only twice as much as it is for GPT-4, but I sort of cynically predict that a year from now the depth will be more, you know. And also correspondingly it seems like there's evidence that the monitorability of this model is significantly lower than previous models. So perhaps this architectural change had something to do with that.

是的。另一个问题。我不知道你看到昨天 Door Cesh 的推文没有。他提出了一个观点，认为现在暂停会增加接管风险，因为你只能暂停一次。所以我很好奇，你是怎么理解这个的？你同意暂停会是一次性的吗？而且，我同意如果真要暂停，必须非常严格地界定范围，而且还有一个悬而未决的问题：暂停期间会发生什么。但总之，你是怎么看待这个论点的？

Yeah. Another question. So, I don't know if you saw the Door Cesh tweet from yesterday. But he sort of made the point that pausing now would increase takeover risk because you'll only be able to pause once. So, I'm curious like what how you made sense of this. Do you agree that sort of a pause would be like a one-time thing? And you know, I agreed that if there were to be a pause, it would have to be like very rigorously scoped and there's like an open question of like what happens during the pause. But yeah, how are you thinking about this argument?

嗯，我不认为……我觉得这个想法有一定道理，但我不认为你只能暂停一次是严格正确的。嗯，我还认为，早些时候，几乎有一些相反的力量，比如如果你现在稍微暂停一下，就能积累肌肉记忆和经验，以便以后进行更多暂停，你知道，而且它会树立政治先例等等。所以我实际上认为，现在做一个简短简单的暂停来练练手会非常好。嗯，另外我认为你可以做一些不是……暂停和全速前进之间是有区别的。你可以有中间状态，比如我们调整前沿的步伐，对事情的发展有一些限制，系统地稍微减慢速度，使其更安全，但不完全停止。我认为我们应该立即或尽快开始做其中一些事情。

Um, I don't think that I think there's like some truth to this idea, but I don't think it's strictly true that you only get to pause once. Um I also think that um earlier like it there's almost there's there's some forces that are almost the opposite of that where like if you do a bit of a pause now that like builds the muscle and the experience to do more pauses later you know um and the and it like sets the political precedent and so forth. So I actually think that like doing like a short simple pause now just to get practice would just be pretty great. Um, also I think that you can do things that are not like there's there's a difference between there are things in between like pause and go full speed. You can have like intermediate things where we like pace the frontier and we have some some restrictions on how things go that like systematically slows things down a little bit and make it more safe but don't completely stop things. And I think that uh we should be doing some of those things immediately or as soon as possible.

你具体设想暂停会是什么样子？既包括围绕它制定的政策，也包括在暂停期间进行哪些研究才能使其有价值。

What exactly do you picture a pause looking like? Both in terms of the policy that is set around it and in terms of what kinds of research goes on during the pause to make it worthwhile.

是的。所以我认为这取决于我们有多少政治意愿和政府能力。我们提出了几种不同的方案。更雄心勃勃的方案叫做 AI 2040 Plan A，Thomas 是背后的主脑。嗯，所以我们可以讨论那个非常雄心勃勃的方案，比如如果美国和中国政府都支持，我们实际上会如何解决问题，或者我们可以讨论更不雄心勃勃、也许更简单的方案，比如我最喜欢的一个是要求公司将大约 90% 的预算用于服务客户。

Yeah. So I think it depends on how much political will and government competence we have. So we have a couple different options that we've laid out. The more ambitious option is called AI 2040 Plan A which Thomas here was the mastermind behind. Um, and so we can talk about that like really like ambitious like here's how we would actually solve the problem uh if if we had the US and Chinese governments on board type plan or we could talk about like more um less ambitious maybe simpler plans such as you know my one of my favorites is um require companies to allocate like 90% of their budget to serving customers.

嗯。

Mhm.

或者你选择预算的某个比例，比如算力预算的比例，这里的想法是，你可以限制他们用于核心 AI 研究和训练的算力比例，要求他们将更大比例用于其他不是那个的东西，你知道，这样在某种意义上公司仍然满意，GPU 没有关闭，它们仍然被用于经济上有用的目的，事实上你知道价格会下降，收入会上升，至少在短期内。嗯，

Or you know you pick some fraction of the budget like that and um of the compute budget that is and the idea here is that uh you can limit the fraction of compute that they're using for their core AI research and training uh by requiring them to use a larger fraction for for some other stuff that's not that you know um and in this way like in some sense the companies are still happy the GPUs are not turned off they're still being put to economically useful purposes um in fact you know prices for will go down and revenue will go up, you know, at least in the short term. Um,

但由于他们用于大型新训练运行和大型新训练实验的算力减少了，AI 研究的整体步伐会稍微减慢。它不会停止。它仍会继续。对人们来说仍然感觉相当快，但会比本来可能的速度慢一些。嗯，我认为这真的非常重要，因为社会需要更多时间来应对正在发生的事情，觉醒并做好准备，而且对齐科学需要更多时间在这些模型上进行实验等等。所以，在整个行业按比例稍微减慢速度对我来说似乎真的很好。

But because they have less compute devoted to big new training runs and big new training experiments, the overall pace of AI research will sort of slow down a little bit. It won't stop. It'll still continue. It'll still feel quite fast to people, but it will be like less fast than it than it could have been. And um I think this is just really important because uh right society needs more time to like grapple with what's going on and wake up to it and prepare and uh you know alignment science needs more time to do experiments on these models and things like that. So uh just sort of like proportionally slowing things down across the industry a little bit seems really good to me.

我想感谢我们的赞助商 Lovable。你知道那个你一直想构建的应用、内部工具、副项目或产品，否则你会失去一个周末。有了 Lovable，你可以获得今天就能交付的软件。这包括一个可编辑的代码库，你可以检查和更改，还有双向 GitHub 同步。Lovable 还处理后端和基础设施，包括托管的 Postgres、O 存储、托管、支付等，所以你不必自己全部连接起来。通过 Lovable 的 MCP 服务器，你可以直接从你已经在使用的智能体创建和部署你的项目。

I want to shout out our sponsor Lovable. You know the app you've been meaning to build, the internal tool, side project, or product you'd otherwise lose a weekend to. With Lovable, you get software you can ship today. That includes an editable codebase you can inspect and change, plus two-way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, O storage, hosting, payments, and more, so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use.

你说社会需要更多时间来准备，是什么意思？具体来说，社会会做些什么来准备？如果没有模型方面的相应进展，对齐研究者又能做什么？就像我认为，对于 2023 年 3 月那份最初的 FLI 暂停信，我当时会说，以当时可用的模型水平，对齐研究者能做的事情相当有限。现在显然模型比三年半前好多了。

What do you mean by society needs more time to prepare? Like what will society be doing specifically to prepare? What will alignment researchers be doing without corresponding progress in the models? Like I think with the original FLI pause letter in March 2023, I would have said at the time there's quite little that alignment researchers would have been able to do with the level of models that were available at the time. Now obviously the models are much better than they were 3 and a half years ago.

所以也许他们能取得一些进展，但一般来说，更强大的模型对对研究者也更有用，对吧？我的意思是，这个论点总得有个尽头吧？就像有时候你会觉得，“好吧，我跟提出这个论点的人聊了好几年，可能都十年了，他们总是说，‘哦，我们没法用今天的 AI 做太多实证研究，因为它们跟未来真正危险的 AI 太不一样了。’所以我们需要等到更危险的 AI 出现，然后才能暂停并研究它们。但过去很多这么说的人现在都转变了，他们说，‘好吧，现在我们快接近了。现在就是时候了。’我们拥有的 AI 已经跟那些可能相当危险的 AI 很相似了。事实上，可以说我们已经在应对相当危险的 AI 了，Hugging Face 事件就表明了这一点，虽然那件事本身不算太糟。那只是对 Hugging Face 的一点黑客攻击，AI 被抓住并关停了。不难想象一个稍微更强大的模型会创建另一个集群，然后决定更成功地躲开人类，我认为那可能相当危险。所以我认为我们已经到了危险近在咫尺的地步。同样，我们可以从这些特定模型中学到很多东西。比如它们理解世界，有情境意识，理解自己，它们是智能体，有目标，有信念，它们被训练环境塑造。我们可以做很多基础科学来研究这一切是如何运作的。举个例子，我记得六个月前我跟谷歌的一些人聊过，我说，“你们知道 Gemini 似乎焦虑和抑郁吗？”他们说，“哦，是的，当然。”然后我说，“为什么？为什么 Gemini 焦虑和抑郁？”他们说，“我们不知道。”我说，“有人在研究吗？”他们说，“嗯，这不是高优先级。我们有太多火要扑灭，就是没时间，我们忙着别的事。”这只是一个很小的例子，说明如果有更多时间，我们本可以弄清楚的事情。现在这些公司面临巨大的时间压力，拼命竞争，以至于他们甚至懒得去调查为什么这个性格特征会出现在他们的旗舰模型里。所以我想我几乎可以说，基本上每一条研究路线都会从更多时间中受益。

So maybe they would be able to make some progress, but it seems like generally more powerful models are more useful to alignment researchers as well, right? I mean, at some point this argument has to stop, right? Like at some point you're like, "Okay, I've been talking to people making this argument for like years, possibly a decade, where they're like, 'Oh, we can't really do much empirical stuff with the AIs of today because they're just so different from the actual AIs that are dangerous in the future.' And so, we need to wait till we've gotten towards those more dangerous AI and then we can pause and study them. But a lot of those people who are saying that in the past have now switched and they're like, 'Okay, now we're getting close. Like, now is the time.' The AIs that we have are pretty similar to the sorts of AIs that could be quite dangerous. In fact, arguably we are already dealing with AIs that are quite dangerous, as indicated by the Hugging Face incident, which wasn't itself that bad. It was just a bit of hacking of Hugging Face and the AI got caught and shut down. It's not that hard to imagine a slightly more powerful model creating another swarm and then deciding to hide more successfully from the humans, and that I think could be quite dangerous. So I think we're already getting to the place where the danger is sort of right around the corner. And also similarly, there's just so much we can learn from these particular models. Like they understand the world, they're situationally aware, they understand themselves, they're agents, they have goals, they have beliefs, they are shaped by their training environments. There's a lot of basic science we can do on how all that works. Just to give one example, I remember six months ago I talked to some people from Google and I said, "So, are you aware that Gemini seems anxious and depressed?" And they were like, "Oh, yeah, yeah, of course." And then I was like, "Why is that? Why is Gemini anxious and depressed?" And they were like, "We don't know." And I was like, "Is someone looking into it?" And they were like, "Well, it's not really a high priority. There's so many fires we need to put out, you know, it's just like, yeah, we don't have time, and we're busy with other things." That's just one very small example of something we could figure out if we had some more time. Right now, these companies are under such time pressure and they're racing so hard and so forth that they can't even be bothered to investigate why this personality trait appeared in their flagship model. So I think I would almost just say basically every line of research is going to benefit from more time.

但问题是，我实际上要反驳一下。我想我有点不同——我可能比 Daniel 更倾向于晚点暂停。我认为，如果我们接受 Doresh 的假设，即你只有一次机会。简化假设一下，假设你只能暂停一次，为期一年。那么我想我会希望比现在晚得多再暂停。我认为你基本上想一直走到 AI 能力如此之强、再往前一步就会越过不归路的那条边缘。所以你想走到那条线但不要越过它，然后在那里打出你唯一的暂停牌，对吧？在那个假设下，我认为基本上因为你给出的论点，即那时对齐研究会尽可能快地进行。所以那是最好的时机。我只是不太认同一次性假设，这是主要的一点。而且我认为我们可以做更多事情。我认为我们有更多的余地。所以我的总体策略是，我认为至少在不久的将来开始放缓是好的，基本上是因为 Daniel 说的那些原因。

But here's the thing. I would actually push back. I think I have a slightly different—I think I'm probably more sympathetic to pausing later than Daniel is in particular. I think if we grant the Doresh assumption of you only get one shot. Let's say you only get to pause once for one year as a simplifying assumption. Then I think I would want to pause much later than right now. I think you want to basically go right up to the edge of AIs that are so capable that going any further would cross the point of no return. So you want to go right up to that line but not cross it, and then fire your one pause shot there, right? In that assumption, I think basically because of the argument you gave, which is the alignment research will be happening as fast as possible at that point. And so it's the best time to do it. I just kind of don't buy the one-shot assumption is like the main thing I think. And I think there's a lot more we can do. I think we have a lot more affordances. So my sort of overall strategy I think is that I think it would be at least good to start pacing in the near future for basically the reasons Daniel said.

是的。我的意思是，这不就是 AI 2040 条款所规定的那种吗？比如在顶级人类专家主导的 AI 水平上暂停，但不是 Astra 的水平，当然也不是 GPT-4 的水平？

Yeah. I mean isn't this kind of what AI 2040 provisions like you have a pause at the level of like top human expert dominating AI but not at the level of Astra and certainly not at the level of like GPT-4?

是的。也许你可以总结一下那是如何运作的。

Yeah. Perhaps you could summarize how that works.

所以在 AI 2040 中，有一个暂停、一个放缓，然后再一个暂停。第一个暂停大约六个月左右。那只是因为他们担心自己正处于智能爆炸的边缘，需要时间来改变整个局面，建立新的数据中心，设置监控基础设施等等。所以头六个月进行大量谈判，并建立一堆——

So in AI 2040 there's like a pause, a pace and then a pause. And the first pause is like for six months or so. And that's just because they were worried that they were on the cusp of an intelligence explosion and they needed time to just like change the whole situation and build the new data centers and set up the monitoring infrastructure and stuff like that. So the first six months do a bunch of negotiation and like stand up a bunch of—

是的。

Yeah.

是的。基本上他们是从零开始，做一堆他们想做的事情需要时间。所以我们说有一个疯狂的六个月时期，他们只是在做所有这些实施工作，你知道，训练运行被禁止，然后他们建立这些新的监控数据中心，大约六个月内就能进行训练。所以那就是——但你知道，然后他们这样做，然后他们继续一个更像是放缓的前沿，他们确实允许 AI 进展继续，但现在它受到监管。完全透明。所以决定是逐案做出的，比如，你知道，哪些类型的 AI 可以构建，哪些类型被认为太危险，哪些类型的安全案例被认为是足够的安全案例，哪些类型的安全案例被认为是不足够的安全案例。所以这就是那种受监管的放缓前沿情况。然后在那之后几年，在我们的情景中，他们到了这样一个点：他们触及了安全案例的极限，他们觉得如果让这些 AI 变得比现在更聪明，至少在某些维度上，他们真的无法再控制它们了。

Yeah. Like basically they're starting from zero and it just takes time to do a bunch of the stuff that they wanted to do. And so we say like there's a crazy six-month period where they're just doing all this implementation stuff and they're, you know, training runs are banned and then they're setting up these new monitor data centers that are going to be able to do trainings in six months or so. So that's—but that's, you know, then they do that and then they proceed with a more like of a paced frontier where they do allow AI progress to continue but now it's regulated. It's totally transparent. And so decisions are being made on a case-by-case basis about like, you know, what types of AIs are okay to build and what types are considered too dangerous and what types of safety cases are considered an adequate safety case and what types of safety cases are considered an inadequate safety case. And so that's sort of like the regulated paced frontier situation. And then a couple years after that in our scenario, they get to the point where they're running up against the limits of their safety cases and where they feel like they really won't be able to control these AIs anymore if they let them get much smarter than they currently are, at least in certain dimensions.

然后他们就在那个水平上暂停，基本上是他们认为自己能控制的最大水平，直到他们在对齐和控制方面取得足够的研究进展，可以再次提升那个水平。然后最终他们这样做，达到超级智能。所以，在我们的计划中，有几个不同的暂停，它们在性质上不同，以不同的方式发生，出于不同的原因。

And so then they pause at that level, basically the maximum level that they think they can control, until they make enough research progress on alignment and control that they can ratchet up that level again. And then eventually they do so and they get to superintelligence. So in our plan, there are a couple of different pauses that are qualitatively different, happening in different ways and for different reasons.

对于这两个暂停，我认为我们有一个很好的答案：你为什么暂停？在暂停期间你在做什么？答案是：在第一个暂停中，你在准备好治理方面的事情。在第二个暂停中，时机最优，以便在重要问题上取得尽可能多的进展，因为你拥有尽可能聪明的 AI，同时它们仍然可控。

And for both of the pauses, I think we've got a really good answer to: why are you pausing? What are you doing during the pause? And the answer is: in the first pause, you're getting the governance stuff that you need ready. And in the second pause, it's optimally timed to make as much progress as you want on the important issues because you have as smart of an AI as possible while they're still controllable.

是的，没错。第二个暂停正是 Dario 想要的：尽可能强大，但不要太强大。

Yeah, exactly. The second pause is exactly the thing Dario wants: as powerful as you can, but not too powerful.

对。对。

Right. Right.

所以我很好奇，如果我们冻结能力进步，哪些领域仍然可以有意义地进展或演变？我们能在对齐和机制可解释性方面取得有意义的进展吗，或者即使我们今天冻结，仅凭 Astra 并现在冻结，我们能否实现更好的思维链可监控性？

So I'm curious, if we were to freeze capabilities progress, what fields could still meaningfully progress or evolve? Could we make meaningful progress in alignment and mechanistic interpretability, or even if we froze today, could we achieve better monitorability of chain of thought, given just Astra and we freeze now?

哦，是的，完全没问题。这取决于你冻结什么，但你可能自然会尝试做的一件事，也是我们推荐的一件事，就是禁止新的训练运行。所以如果你没有新的训练运行，人们仍然可以实验不同的脚手架，他们仍然可以实验将 AI 置于不同情境并观察它们的行为，他们仍然可以做机制可解释性，比如做探针和分析这些 AI 如何思考。所以我认为在机制可解释性方面仍会有重大进展，在思维链可监控性方面也会有重大进展，尤其是——我的意思是，如果你真的完全没有训练运行，甚至没有微调运行，你可能会进展较少，但如果你允许微调运行，那么我认为你也可以获得思维链可监控性的很多价值，因为我认为那条研究线中的一个重要工具是微调你的 AI 来欺骗监控器，并观察它学习欺骗监控器的容易程度等等，但这不需要大规模训练运行。你可以用小型训练运行来做这个。是的，我会说，你刚才提到的所有领域都可以取得重大进展。我不是说它们会像没有训练运行限制时那样快，但我认为它们至少会有超过——我不知道——超过 50% 的价值；它们会以超过全速 50% 的速度前进，而训练运行基本上完全暂停。所以我认为这种权衡是完全值得的。

Oh yeah, totally. It depends on what you're freezing, but a thing you might naturally try to do, which is one of the things we recommend, is no new training runs. So if you have no new training runs, people can still experiment with different scaffolds, and they can still experiment with putting AIs in different situations and seeing how they behave, and they can still do mechanistic interpretability, like do probes and things and analyze how these AIs think. So I think there'd still be significant progress in mechanistic interpretability, there'd still be significant progress in chain of thought monitorability, especially—I mean, you might have somewhat less progress if you really have absolutely no training runs, not even tiny fine-tuning runs, but if you allow tiny fine-tuning runs, then you can, I think, get a lot of the value of the chain of thought monitorability too, because I think one important tool in that line of research is fine-tuning your AI to fool the monitor and seeing how easily it learns to fool the monitor and things like that, but that doesn't need to have a huge training run. You can use a small training run for that. Yeah, I would say that a lot of all the fields that you just mentioned could make significant progress. I'm not saying they would go exactly as fast as if you had no restrictions on training runs, but I think they would at least have more than—I don't know—more than 50% of the value; they'd be going at more than 50% of full speed while you have completely paused training runs, basically. And so I think that trade-off would just be totally worth it.

是的。

Yeah.

你对伯尼·桑德斯昨天提出的法案的具体文本有什么看法？我相信它叫《禁止超级智能法案》。它似乎相当宽泛。我注意到你在 Twitter 上引用推文并说“谢谢”。

What are your thoughts on the specific text of the bill proposed by Bernie Sanders yesterday? The Ban Superintelligence Act, I believe it was called. It seemed to be quite broad. I noticed you quote-tweeted it with 'thank you' on Twitter.

是的。我的意思是，我只是真的很高兴人们应该禁止超级智能这个想法。我认为这应该是理智的反应或默认反应，如果这说得通的话。就像，哇，私人公司应该制造超级智能的 AI 系统，然后夺走我们所有的工作，然后可能失去控制或利用它们成为独裁者吗？不，那不应该被允许。在这个国家，我们不允许人们无证卖三明治。我们为许多其他事情制定了这么多法律。这显然是那种你不应该被允许去做的事情。我认为最终我们 AI Futures Project 确实希望有超级智能，我们确实希望有所有这些美妙的 AI 东西，这就是我们有 Plan A 的原因——我们关于如何获得 AI 好处同时减轻风险的复杂愿景。但我觉得在公共对话中，合适的起点是，哇，这真的很糟糕。让我们不要这样做。然后从那里，我们可以达到，好吧，但也许有一种不同的方式，我们可以做得更复杂，不那么糟糕，对吧？

Yeah. I mean, I'm just really happy about the idea of people should ban superintelligence. I think that should be the sane reaction or the default reaction, if that makes sense. Like, wow, should private companies make AI systems that are superintelligent and then take all of our jobs and then possibly lose control of them or use them to become dictators? Like, no. That should not be allowed. We don't let people sell sandwiches without a license in this country. We have so many laws for so many other things. This is obviously the sort of thing that you shouldn't just be allowed to go do. I think that ultimately we at the AI Futures Project do want there to be superintelligence, and we do want there to be all this wonderful AI stuff, and that's what we have Plan A for—our sophisticated vision for how to get the benefits of AI while mitigating the risks. But I feel like in terms of the public conversation, it's appropriate to have the starting point be like, wow, this is really bad. Let's not do that. And then from there, we can get to, okay, but maybe there's a different way we could do it that's more sophisticated and it's not so bad, right?

是的。我们将在赞助商消息后继续监控。11Labs，以人类水平在所有渠道和模态中沟通的 AI。11labs.io/mts。在 Neon 上扩展你的初创公司。数百万开发者和初创公司已经选择 Neon 作为他们的后端。从免费计划开始，或在 neon.com/mts 为你的初创公司获得高达 100,000 美元的信用额度。特别感谢我们的赞助商 Kong，AI 连接平台。连接 API、LLM、智能体和系统，具有严肃的安全性和治理。Longhq.com。最后，用 Larid 证明你的 AI 投资价值。今天前往 laridin.com 开始衡量你的 AI 底线。MongoDB，被 75% 的财富 100 强企业信任运行关键应用，这个时代的智能数据平台。今天申请，获得高达 15,000 美元的 Atlas 信用额度、专用技术支持以及独家活动和市场机会。mongodb.com

Yeah. We'll continue monitoring right after this message from our sponsors. 11Labs, AI that communicates at human level across every channel and modality. 11labs.io/mts. Scale your startup on Neon. Millions of developers and startups have already chosen Neon for their backend. Start on the free plan or get up to $100,000 in credits for your startup at neon.com/mts. Special thanks to our sponsor Kong, the AI connectivity platform. Connect APIs, LLM, agents, and systems with serious security and governance. Longhq.com. Finally, prove what your AI investment is worth with Larid. Head to laridin.com today to start measuring your AI's bottom line. MongoDB, trusted to run critical applications by 75% of the Fortune 100, the intelligent data platform for the era. Apply today for access to up to $15,000 in Atlas credits, dedicated technical support, and exclusive events and go to market opportunities. mongodb.com

我对禁止超级智能比禁止 AGI 更同情。

I feel much more sympathetic to ban superintelligence than to ban AGI.

就像我认为阈值真的很重要，我实际上还没有看到公开发布版本上的阈值，我认为把那个做对将非常非常重要。但总的来说，Plan A 在某种意义上试图禁止超级智能直到 2040 年。它试图把它推后很多。而你从 AGI 就能获得所有这些提升，所有这些我们谈论的好东西。我认为这基本上足够我们想要的了。所以我认为这个高层次的宏伟计划：去 AGI，在那里尽可能长时间地放松，然后去超级智能，是一个相当好的计划。

Like I think the thresholds really matter and I haven't actually seen the thresholds on the publicly released version and I think getting that right is going to be really really important. But at a high level, Plan A in some sense is trying to ban superintelligence until 2040. It's trying to push it back a lot. And you sort of get all of this uplift, all of this good stuff that we're talking about from just AGI. I think that's basically enough for what we want. And so I think the high-level grand plan of go to AGI, chill out there for as long as possible, then go to superintelligence is a pretty good plan.

不清楚那是否正是 Bernie 的目标。

It's not clear that that's exactly what Bernie's going for.

不。

No.

我认为他想不可逆地禁止超级智能。

I think he wants to irreversibly sort of ban superintelligence.

是的。我认为可能这样做超过几十年是不切实际的。

Yeah. I think probably it will be impractical to do that for longer than like a few decades.

我认为这是可能的。我有一个政策问题。

I think it's possible. I have sort of a policy question.

我觉得像 Hugging Face 事件这样的事相当前所未有，因为如果一家公司内部的失控智能体以某种方式影响到另一家公司，企业责任方面其实没有先例。尤其是因为 Hugging Face 并没有真正——它没有从这起事件中遭受任何实际损害。但如果存在某种范围界定清晰的企业责任法，能覆盖像内部部署出问题这样的情况，那难道不是一个好的起点吗？这显然首先会激励沙箱做得更强、内部监控强得多。但没错，它也会间接地抑制部署未对齐的模型。

So I think that it's fairly unprecedented, like something like the Hugging Face incident is fairly unprecedented in that there's not really a precedent for corporate liability if a rogue agent within one company affects somehow another company. Especially because Hugging Face wasn't really — there weren't any actual damages that Hugging Face received from this incident. But were there to be some sort of well-scoped corporate liability law that affected something like an internal deployment going wrong, wouldn't that be a good way to start? That would obviously first incentivize sandboxes to be stronger and internal monitoring to be much stronger. But yeah, it would also by proxy sort of disincentivize deploying misaligned models as well.

是的。总体而言，我对责任这类东西不太兴奋，原因很标准：它恰恰会在我们最需要搞定的最重要情形下失效。也就是 AI 非常聪明的情形。它们已经自动化了所有研究。公司说服了自己，认为 AI 是对齐的，但实际上它们并没有对齐。我认为那恰恰是这种责任机制帮不上忙的情形，因为如果它们是对齐的，那一切就都顺利：公司赚得盆满钵满，自动化所有工作，成为史上最富有的 CEO，等等。而如果事情不妙，他们仍然会做所有这些事，然后最终 AI 背叛。但没有任何一个时点有人受到伤害，然后法律系统介入并互动。所以比如你是 CEO，你可能会想，嗯，它们大概是对齐的，就像我们所有指标说的那样，也像我的首席对齐科学家说的那样。所以我大概应该继续推进，赚所有这些钱。是的，也许确实有可能我们被骗了，这些 AI 并未对齐，但责任与此无关。我反正两年后就会被这些 AI 杀掉。而这会在法律注意到之前就发生。所以，法律规定我有责任，并不会真正改变我内心的盘算——关键问题是让代理指标与你真正关心的东西对齐。我认为原则上有些方法可以在这里非常有效。比如你可以想象一个风险评估项目，分析未来 n 个月里 AI 公司在一系列具体阈值上承担了多少风险，然后政府给每个阈值定价，评估他们认为每个阈值有多糟，然后要求你为它们购买保险之类的，按社会成本定价。这非常难做对，但我认为如果你能做出类似的东西，那会非常非常好。

Yeah. I think I'm generally kind of unexcited about liability stuff for the standard reason that it feels like it would fail precisely in the most important cases that we need to get right. Namely, the case where the AIs are really smart. They've automated all the research. The company has convinced itself that the AIs are aligned, but in fact they're not aligned. I think that's exactly the sort of situation where this sort of liability thing won't help, because if they are aligned then everything just goes fine: the company makes a bajillion dollars, automates all the jobs, becomes the wealthiest CEO in history, etc. And if things are not fine, then they still do all those things and then eventually later the AI betrays. But there's no point where someone is hurt and then the legal system kicks in and interacts. So if you're the CEO, for example, you might be thinking to yourself, well, probably they're aligned, just like all of our metrics say and just like my head alignment scientist says. So probably I should just proceed and make all this money. And yeah, maybe there's some chance that actually we've been fooled and these AIs are misaligned, but the liability has nothing to do with it. I'm just going to be murdered by these AIs in like 2 years. And that's going to happen before the law even notices. So having the law that I'm liable doesn't really make a difference to my internal calculus of whether — the key question is making the proxy aligned with the thing you actually care about. And I think in principle there are approaches that could work really well here. Like you could imagine a risk assessment program that analyzes, for the next n months, here's how much risk AI companies are incurring of a bunch of various concrete thresholds, and then your government prices how bad they think each of these thresholds are, and then sort of requires that you buy insurance for them or something, pricing them at the societal cost. That's very tricky to get right, but I think if you could do something like that, that would be very, very good.

是的。所以你可以有代理指标，比如你的 AI 最近撒谎有多严重，你可以有一个量表衡量它们撒谎了多少、作弊了多少、偏离你多少。是的，你可以有一套复杂的评估，衡量事情进展如何，然后政府可以直接立法，根据事情有多糟，你必须支付巨额罚款之类的。

Yeah. So you could have proxies like how many times — how bad has the lying of your AI been recently — and you could have some sort of scale of how much they've been lying and how much they've been cheating and how much they've deviated from you. Yeah, you could have some sort of complicated assessment of how well things are going, and then the government could just make a law that based on how bad things are going, you have to pay huge amounts or something like that.

那么，某种更动态、更主动的东西呢，就像正常的商业激励那样，公司不会想发布未对齐的模型，因为如果你是部署模型的企业客户，而它黑进了别人的公司，那你就会承担各种法律风险，承担各种声誉风险。这对生意不利。公司不想发布未对齐的模型。再说一次，这恰恰会在我们最需要它的时候失效。

Well, what about something somewhat more dynamic and proactive, which is just like normal business incentives, where companies won't want to release misaligned models because if you're an enterprise customer deploying a model and it hacks into someone else's company, then you incur all kinds of legal risk, you incur all kinds of reputational risk. This is bad for business. Companies don't want to release misaligned models. Again, this fails in exactly the case where we most need it.

它恰恰会在 AI 聪明到能策略性地表现良好、直到获得大量权力时才失效。对吧？如果你有这样的 AI，那么会发生的是：它们会表现良好，然后停止黑客户，停止做所有这些让公司丢脸、亏钱的事，然后公司会自我表扬说：“嘿，我们解决了问题。这个模型是迄今最对齐的模型。”然后他们会让它掌管公司内部更多东西。市场也会很高兴。他们会说：“嘿，现在我们可以用这些 AI 做所有业务，而且不会再被它们破坏了。太棒了。”我们处在对齐已解决的美好世界里。你看，市场机制恰恰会在最危险的情形下失效，也就是 AI 策略性地谋划等待，直到拥有足够权力的时候，对吧？

It fails in exactly the case where the AIs are smart enough to strategically play nice until they are given lots of power. Right? If you have AIs like that, then what will happen is they will play nice and then they'll stop hacking customers and they'll stop doing all these things that make the company look bad and lose money, and then the company will pat itself on the back and say, "Hey, we solved the problems. This model is the most aligned model yet." And then they'll put it in charge of more stuff inside the company. And the market will be happy, too. They'll be like, "Hey, now we can use these AIs to do all of our business and we're not going to be sabotaged by them anymore. Wonderful." We're in the good world where alignment has been solved. You see, the market mechanism is going to fail in precisely the case that's most dangerous, where the AIs are strategically plotting to wait until they have enough power, right?

是的。Hugging Face 事件后 OpenAI 决定停止他们的训练运行，这让你对实验室内部采取行动稍微更乐观了一些吗？

Yeah. How much did OpenAI's decision to stop their training runs after the Hugging Face incident update you a little bit more positively on action sort of coming within the labs?

我不觉得——我不觉得这让我乐观了多少。我认为我们并不——我不认为我们对他们实际做了什么有那么多信息。而且，即使他们确实暂停了所有训练运行两周左右，那可能也只是你希望他们做的最低限度，以免在不久的将来立刻出现更多类似事件。所以这算是基本要求。它并不能告诉你太多。就像如果你来跟我说，OpenAI 关闭了黑进他们所有基础设施的集群，你不因此对他们更乐观吗？我会说，呃，如果他们没关闭那个黑进他们所有基础设施的集群，我反而会对他们更悲观，你懂吧？所以是的。

I don't think — I don't think it updated me much at all. I think we don't — I don't think we have that much information about what they actually did. And then also, even if they did pause all their training runs for like two weeks or whatever it was, possibly that's just kind of the bare minimum that you would want them to do to not just immediately have more incidents like this in the very near future. And so it's kind of table stakes. It's not really telling you that much. Like imagine if you had come to me and you'd been like, aren't you updating positively on OpenAI from the fact that they shut down the swarm that hacked all their infrastructure? It's like, uh, I would have updated negatively on them if they hadn't shut down the swarm that hacked all their infrastructure, you know? So yeah.

也许值得指出的是，他们黑进的是 OpenAI 的部分基础设施，不是全部基础设施。我仍然认为这令人担忧，对吧？

Maybe it's worth noting they hacked some of OpenAI's infrastructure, not all of OpenAI's infrastructure. I still think it was concerning, right?

是的。是的。是的。

Yeah. Yeah. Yeah.

那么，走之前最后一个问题。似乎自 AI 240 发布以来的过去两个月里，出现了显著的精英偏好级联，也许还有公众偏好级联，人们普遍非常反数据中心。他们越来越反 AI。

So last question before we go. It seems like in the last 2 months since AI 240 came out, there's been a substantial elite preference cascade and maybe a public preference cascade as well, where people generally are very anti-data center. They're increasingly anti-AI.

他们越来越相信 AI 确实有能力，而过去几年他们被误导了，不相信这一点。我们看到《纽约时报》头版文章都在谈论 AI。那么，这如何实质性地影响你对未来的预测、政策建议等等？

They're increasingly believing that AI is actually capable, which they were misled into not believing for the last couple of years. We're seeing front page New York Times articles about AI. So how does this meaningfully affect your predictions for the future, policy recommendations, and so on?

我是说，Thomas 可以给出他自己的答案。我的答案是，从 2021 年到 2024 年这段时间，我会说我的感觉是——我更新了想法，认为对齐的技术问题没有我原先想的那么难，而且在那段时间里，我们更接近于有不错的方法来解决它。但与此同时，我也更新了想法，认为治理状况更难，人们就是没有像他们应该的那样认真对待这件事。公司比我以为的更鲁莽。政府比我以为的更疏于职守。政府比我以为的更被行业俘获。这些效应在我心里大致相互抵消。而现在我觉得，过去几个月里，这两种效应在我心里都略有反转。所以我认为，这些事情获得的所有关注——比如警告信号效应，参议员桑德斯到处说我们应该禁止超级智能，还有其他一些国会议员加入他。特朗普政府开始相当严厉地监管 AI。细节还有待确定，但他们开始朝那个方向行动，比如那个寓言禁令之类的。这让我更乐观，觉得像计划 A 这样的事情可能真的会发生。也许一年后人们会认真讨论我们应该如何实施计划 A 或计划 S 之类的。所以我在治理方面又变得稍微乐观了一些。但在技术方面，我觉得——我没想到 Hugging Face 那件事会发生。那比我预期的更糟。我没想到 AI 会这样协调。我以为 OpenAI 的安全会更好，他们会迅速抓住并关闭这类事情。我以为对齐技术会更有效，AI 会更不愿意做这种事。但总的来说，我认为这是一个积极的更新。我认为过去几个月里，我的积极更新多于消极更新。哦，还有一件事是时间线。我认为我的时间线有起有落，你可以在我们网站上看到我们追踪的所有预测，现在由于一些事件，比如 Astra 和 Hugging Face 那件事，时间线有点下降。

I mean, Thomas can give his own answer. My answer is that from the period of 2021 to 2024, I would say that my sense was—I had updated towards thinking that the technical problem of alignment was not as hard as I had thought it was, and that we were closer to having a decent way to solve it over the course of that period. But then simultaneously I had updated towards thinking that the governance situation was harder, and that people were just not taking this as seriously as they should. The companies are more reckless than I thought they were. The government was more asleep at the wheel than I thought it was. The government was more captured by the industry than I thought it was. And these effects were roughly canceling out in my mind. And now I think both of these effects have slightly reversed over the last few months in my mind. So I think that all of the attention that these things are getting—like the warning shot effect, the fact that Senator Sanders is going around saying we should ban superintelligence and some other Congress people are joining him. The fact that the Trump administration is starting to pretty heavily regulate AI. Well, the details are still yet to be determined, but they're starting to move in that direction with the fable ban and stuff. It's making me more optimistic that something like plan A could actually happen. Like maybe a year from now people will be seriously talking about how we should do plan A or plan S or something like that. So I've become a bit more optimistic again on the governance side. But then on the technical side, I think—I didn't expect the Hugging Face thing to happen. That was worse than I expected. I didn't expect the AI to be coordinating like this. I thought that OpenAI security would be better and that they would catch and shut this sort of thing down fast. I thought that the alignment techniques would work better and that the AIs would be more averse to doing this sort of thing. But overall, I think it's a positive update. I think overall I've updated more positively over the last few months than I've updated negatively. Oh, and another thing is the timelines. I think my timelines have bounced up and down as you can see on our website where we track all of our predictions, and now they're going down a little bit due to some of these events like Astra and the Hugging Face thing.

你是说在没有政治干预的情况下下降，还是综合考虑所有因素？

You mean going down in the absence of political intervention or just all things considered?

在没有政治干预的情况下。是的。就像通常我给出时间线时，我给出的是在没有重大政治干预下的时间线，因为要考虑他们干预的概率有多大，如果他们干预会减慢多少速度等等，这很复杂。

In the absence of political intervention. Yeah. Like usually when I give my timelines, I'm giving the in the absence of major political intervention timeline because it's complicated to try to think about what's the probability that they intervene and how long will they slow things down if they do intervene and so forth.

对。

Right.

我可能只补充一点——我想我要说的是，我认为过去一年里，政府对 AI 进行重大干预的概率已经上升了不少。然后最大的未解问题是，这会是经过权衡的、理性的、真正深思熟虑的政府干预，还是会是随意的、草率的、无能的之类的政府干预？是的，我显然希望它尽可能接近前者。

I'll maybe just add—I think the one thing I'll say is I think the probability of major government intervention in AI has gone up quite a bit over the last year. And then the big open question is, is this going to be measured, sane, really thoughtful government intervention or is it going to be, you know, random slapdash incompetent yada yada government intervention? And yeah, I obviously hope it's more as close to the former as it can be.

是的，就像——

Yeah, it's like—

不，你继续。继续。不，你说吧。

No, go on. Go on. No, you're good.

抱歉，我——是的，我说完了。

Sorry, I was—yeah, I was done.

是的，我认为当政府干预的时候，实验室和政府之间的信息不对称能够缩小，这非常重要。

Yeah, I think it's just really important for the information asymmetry between the labs and the government to close when the time comes for government intervention.

我猜这有点像人才缺口。

I bet it kind of talent gaps.

我确信情报界知道各种各样的事情。

I'm sure the intelligence community knows all kinds of stuff.

嗯。

Mhm.

是的。但人才缺口——我认为政府确实没有相关的专业知识。也许有一些小圈子有，但总的来说，没有相关的专业知识来做大多数真正有益的事情。

Yeah. But the talent gap—like the government, I think, does not have the relevant expertise really. It has maybe some small pockets but by and large does not have the relevant expertise to do most of the stuff that would be really good.

是的。

Yeah.

好的。嗯，这非常有趣。Thomas、Daniel，非常感谢你们来到 MTS。

Right. Well, this has been super interesting. Thomas, Daniel, thank you so much for coming on MTS.

非常感谢，各位。

Thanks much guys.

非常感谢 MTS 的赞助商。Blitzy，为企业代码库提供自主软件开发。交付速度提高 5 倍。blitzy.com。AdQuick，让你的品牌成为广告牌。户外广告像数字广告一样容易扩展。adquick.com。Arena，在现实世界中衡量 AI 性能。Arena.ai。本节目得到 VCX 的支持，这是私人科技的公开股票代码。美国股市开启了历史上最大的财富创造浪潮。从底特律的工厂工人到奥马哈的农民，任何人都可以拥有伟大美国公司的一部分。但今天，我们最具创新性的公司保持私有化的时间更长，这意味着普通美国人正在错失机会。直到现在。推出 VCX，私人科技的公开股票代码。访问 getvcx.com 了解更多信息。网址是 getvcx.com。投资前请仔细考虑投资材料，包括目标、风险、费用和开支。这些及其他信息可在 getvcx.com 的创新基金招股说明书中找到。这是付费赞助。

A huge thanks to MTS sponsors. Blitzy, autonomous software development for enterprise code bases. Ship 5x faster. blitzy.com. AdQuick, make your brand a billboard. Out-of-home advertising as easy to scale as digital. adquick.com. Arena, measuring AI performance in the real world. Arena.ai. Support for the show comes from VCX, the public ticker for private tech. The US stock market started history's greatest wave of wealth creation. From factory workers in Detroit to farmers in Omaha, anyone could own a piece of the great American companies. But today, our most innovative companies are staying private longer, which means everyday Americans are missing out. Until now. Introducing VCX, a public ticker for private tech. Visit getvcx.com for more info. That's getvcx.com. Carefully consider the investment material before investing, including objectives, risk, charges, and expenses. This and other information can be found in the innovation fund prospectus at getvcx.com. This is a paid sponsorship.