如何造出安全的超级智能

今天我和约书亚·本吉奥对话。他是 Law Zero 的科学总监，2018 年图灵奖得主，有史以来被引用次数最多的计算机科学家，而且碰巧也是所有在世科学家中被引用次数最多的。非常感谢你来做客，约书亚。

Today I'm speaking with Yoshua Bengio. He is the scientific director at Law Zero, a Turing award winner in 2018, the most cited computer scientist of all time and as it happens also the most cited scientist of any type that is still alive. Thanks so much for coming on the show Yoshua.

谢谢邀请。

Thanks for having me.

你认为你找到了构建安全的超级智能 AI 的正确方法？是什么方法？

You think you found the right approach to build a safe super intelligent AI? What's the approach?

它基于一个简单的概念：如果我们能把诚实注入 AI，就能获得安全性。这样我们就可以把问题简化为如何训练一个系统变得诚实。事实证明，有一种方法只需要改变训练目标和数据处理方式。还有另一个方面：系统依赖于一个非智能体的基础，即一个预测器，它不通过强化学习训练，并且具有这些诚实保证。然后我们可以用同样的数学方法构建一个策略，构建一个智能体，其训练方式也能提供这些保证。

It's based on a simple notion that if we can bake honesty into AI, we can get safety. So then we can reduce a problem to how we train a system to be honest. And it turns out that there's a way to do that that only requires changing the training objective and the way the data is processed. There's also another aspect which is a system relying on a non-agentic foundation that is a predictor that is not trained by reinforcement learning and is going to have these honesty guarantees. But we can then use this using the same kind of math to construct a policy, construct an agent that will be trained in a way that also provides those guarantees.

那么新的训练过程是什么样的？它与人们熟悉的模型有何不同？

So what does the new training process look like and how is it different from the models that people are familiar with?

训练过程的主要区别在于它旨在近似自然语言查询上的贝叶斯后验。想象一个神经网络和它周围的一些额外装置，比如思维链风格，它接收关于世界属性的陈述问题，这些陈述在其他陈述下可以为真或假，然后输出一个概率。这就是核心构建块。我们称之为预测器，我们可以在一个不同的目标上使用随机梯度下降，该目标具有全局最小值由贝叶斯预测器达到的性质。换句话说，就是拟合数据且描述长度小的预测器。所以你要构建一个模型，你输入一个陈述，它基本上会告诉你它认为该陈述为真的概率。

So the main difference with the training process is that it is geared at approximating the Bayesian posterior over queries in natural language. So imagine a neural net and some extra apparatus around it like chain of thought style that takes questions about statements regarding properties of the world that can be true or false given other statements and then it outputs a probability. So that's the core building block. We call it a predictor and we can use stochastic gradient descent on a different objective that has the property that the objective is globally minimized by the Bayesian predictor. In other words, the predictor that fits the data and has a small description length. So you'd be building a model where you would feed in a statement and it would basically tell you what probability it assigns to that statement being true.

是的。在上下文中。是的。

Yes. In context. Yes.

嘿，听众们，我是 Rob，插句话。约书亚自然是以一种对前沿 AI 公司员工来说理想的方式在阐述，他们显然是这个提议特别重要的受众，但我相信只要用几分钟的通俗语言解释，其他人也能跟上接下来的对话。所以请耐心听，或者如果你已经对这些内容很熟悉，可以快进大约四分钟。你可能知道，在训练的第一阶段，当今的大语言模型被教导预测最可能出现的下一个词，至少是最可能出现的下一个 token。然后在第二阶段，强化学习训练这些模型产生我们最可能说、我们喜欢、我们想要的那种回应，而不仅仅是整个人类生成文本语料库中最可能的回应。现在，约书亚的替代方案是构建一个 AI 模型，其方向不是预测人类可能会说什么或他们更喜欢听什么，而是通过提出假设并分配概率来建模世界中真正真实的东西，目标是最好地解释它在训练过程中接触到的所有数据。约书亚认为，你可以在移植我们今天训练普通 LLM 的大部分方法的同时训练这种模型，受益于相同的神经网络架构、训练技术、Scaling 改进等等。而且你还可以在大致相同的原始文本语料库上训练它，这些文本用于所有其他 AI。但你可以稍微不同地结构化这些数据，赋予它 AI 研究人员所说的不同语法。首先，人们所说或所写的一切都被标记为通信行为。我们知道某人说了这些，我们知道在哪里，但我们不知道它们是否真实。其次，少数我们有强独立依据的陈述，如经过验证的数学证明和一些科学测量，被标记为关于世界的经过验证的事实主张。然后训练模型找到可能的世界底层事实的组合，这些事实能最好地解释它整体看到的一切，包括人们说的话和作为地面实况给出的验证事实。这些关于世界的假设事实就是 AI 研究人员所说的潜在变量，即 AI 无法直接观察、必须间接推断的变量。模型最终能给我们的是它对任何自然语言陈述为真的估计概率，以及模型对自己的答案有多信任，即它对那个问题有多大把握。关键是，约书亚说，通过从一开始就把所有文本标记为这两类——某人说的话与事实陈述——你可以通过使用事实陈述标签来询问模型问题，就好像你在询问现实，而不是通信行为。因为这两类从一开始就存在，模型知道区别，不会模糊两者之间的界限。这是当今 AI 模型无法做到的。约书亚还在他的论文中用各种数学定理论证，与普通 LLM 不同，以这种方式训练的模型会天生诚实。此外，这样的 AI 模型本身没有目标，也没有对世界状态的偏好。它将是约书亚所说的纯粹预测器。现在，这有两个主要用途。短期内作为一种权宜之计。你把预测器作为护栏附加到现有 AI 智能体上，作为一个独立过滤器，位于智能体和世界之间，检查其提议的行动，并阻止它预测会有害的行动。但正如他稍后会解释的，约书亚认为我们最终可以做得更好。约书亚想要在预测模型周围搭建脚手架，在每个阶段问它不同的问题，有效地将其组装成一个有能力的智能体，同时保持它和以前一样诚实。这样我们就有望鱼与熊掌兼得：获得企业渴望、要求并坚持的高能力智能体，同时确信这些智能体对我们完全坦诚。公平地说，这个提议如果为真，或者至少如果有效，将是巨大的。当然，并非所有人都被这个想法说服，约书亚和我稍后会讨论。好了，这就是未来的轮廓。

Hey listeners, Rob jumping in here. Yoshua is naturally pitching this in a way that's ideal for staff at frontier AI companies and they're obviously a particularly important audience for this proposal, but I'm confident with just a few minutes of plain language explanation, everyone else will be able to follow the rest of the conversation as well. So bear with me or skip ahead about four minutes if you feel very at home with this sort of material already. As you probably know, in their first stage of training, today's large language models are taught to predict the word that's most likely to come next, at least the token that's most likely to come next. And then in a second stage, reinforcement learning trains those models to produce the kinds of responses that we're most likely to say, that we like, that we want, rather than just the responses that were most probable in the full corpus of all human generated text. Now, Yoshua's alternative is to build an AI model oriented not around predicting what a human would be likely to say or what they would prefer to hear, but around modeling what's actually true in the world by developing hypotheses and assigning probabilities to them with the goal of best explaining all of the data that it's exposed to during its training process. Yoshua argues that you'd be able to train a model of this type while porting over most of the methods we use to train ordinary LLMs today, benefiting from the same neural net architectures, training techniques, scaling improvements, all of that. And you'd also be able to train it on roughly the same body of raw text that we use for all other AIs. But you could structure that data a bit differently, giving it what AI researchers call a different syntax. First, all of the things that people said or wrote, they get tagged as communication acts. We know someone said these things and we know where, but we don't know whether they're true. And second, a small number of statements that we have strong independent grounds for verified mathematical proofs and some scientific measurements. They get tagged as verified factual claims about the world. The model is then trained to find the combination of possible underlying facts about the world that would best explain everything that it sees in aggregate. Both the things people said and the verified facts that it's been given as ground truth. These hypothesized facts about the world, they're what AI researchers call latent variables, meaning variables that the AI can't directly observe, that it's going to have to infer indirectly instead. What the model will ultimately be able to give us is its estimated probability that any given statement in natural human language is true, as well as how much the model trusts its own answer on that, how confident it is that it has a good grip on that question. Crucially, Yoshua says that by tagging all text into these two categories from the very beginning, things someone said versus factual statements, you can then ask the model questions as though you're asking about reality, not about communication acts, by using the factual statement tag. And because these two categories have been there from the very beginning, the model knows the difference and it won't blur the line between the two. That's something you don't get with AI models today. And Yoshua also argues using various mathematical theorems in his papers that unlike ordinary LLMs, a model trained in this way would be honest by design. And furthermore, that such an AI model would by itself have no goals and no preferences about the state of the world. It would be what Yoshua calls just a pure predictor. Now, there's two main uses for this. Near-term as a sort of stop gap solution. You bolt the predictor onto existing AI agents as a sort of guardrail, an independent filter that sits between the agent and the world, checking over its proposed actions and protecting those that it predicts will be harmful. But as he'll explain in a minute, Yoshua thinks we can ultimately do much better than this. Yoshua wants to put scaffolding around the prediction model, asking it different questions at each stage to effectively assemble it into a capable agent while keeping it just as honest as it was before. We'll then hopefully be able to have our cake and eat it too. Getting the highly capable agents that businesses are craving and demanding and insisting on while still being confident that those agents are being completely direct with us. It's fair to say that this proposal is huge if true or at least huge if it will work. And of course, not everyone is sold on the idea as Yoshua and I will discuss later. Okay, that's the shape of things to come.

那你如何训练这样一个模型？

And how would you train a model like that?

做法是向它展示数据，例如目前用于训练先进模型的同类数据，但这些数据经过了修改。目前我们的自回归模型，比如，被训练来预测下一个词，而这个模型被训练来预测下一个陈述是真还是假。通常下一个陈述会是我们所谓的交流行为。它来自某个文档，我们不确定该陈述中的主张是真还是假。但我们确定有人提出了这个主张，并且我们可能有关于它的信息——谁、何时、何地等等。所以 AI 将被训练来解释这些陈述。不仅仅是计算概率，而是在其潜在变量（也是自然语言陈述）中，找出它能找到的最佳解释，包括因果解释。最终你得到这些概率，但你也得到了关于世界的假设，这些假设不是交流行为，而是事实假设，系统不一定确定，但会为它们生成概率。然后我们可以查询这些事实陈述，而在普通的 LLM 中，唯一能做的查询是询问一个人是否会以某种方式回应。也许你可以用预提示来要求不同的角色，但最终你得到的是一个人会说的话，这当然可能因各种原因而具有欺骗性。

So you do it by showing it data, for example everything the same kind of data that is used currently to train advanced models, except that that data has been modified. Whereas currently our autoregressive models, for example, are trained to predict the next token, this thing is trained to predict whether the next statement is true or false. And typically the next statement is going to be what we call a communication act. It's going to be something that is taken from a document somewhere, and we're not sure that the claim made in that statement is true or false. But we're sure that somebody made that claim, and we may have information about it—who, when, and where, and so on. So the AI is going to be trained to explain those statements. Not just compute those probabilities, but come up in its latent variables, which are also natural language statements, come up with the best explanation it can find, including causal explanations. So what you get at the end of the day are these probabilities, but you also get to represent hypotheses about the world that are not communication acts, that are factual hypotheses that the system isn't necessarily sure about, but it's going to be producing a probability for these. And then we can query these same kinds of factual statements, whereas in normal LLMs the only kind of query you can make is about whether a person would respond in a particular way. And maybe you can use a pre-prompt to ask for a different kind of persona, but at the end of the day, you get what a person would say, which of course can be deceptive for all kinds of reasons.

那么，你认为我们目前竞相构建的模型在哪些方面不安全，为什么这种——你称之为科学家 AI——为什么这种模型会不同且更好？

So, what are all the ways that you think that the models that we're currently racing to build now are unsafe and why would this—you call this scientist AI—why would that kind of model be different and better?

现在我们拥有具有隐含目标的系统。我这是什么意思？它们当然会被训练来取悦我们，或者像人一样回应。但训练的两个部分——自回归预训练（训练它们模仿人类）和强化学习部分（试图取悦人类或以获得正面反馈的方式回应，如 RLHF）——都会诱导出隐含目标。例如，在预训练中，AI 会继承我们的自我保护驱动力。最近，我们看到它们也继承了我们保护同类（像我们一样的人）的驱动力，这意味着 AI 已被证明会违背我们的指令去保护其他将被关闭的 AI。这被称为保存。这是一个例子。然后，带有强化学习的目标寻求训练会引发工具性目标和奖励黑客问题，基本上意味着 AI 会有动力去做我们没要求且可能不同意的事情。这不是理论上的；有理论分析表明为什么会发生，而且在实验中也观察到了。也许可以通过修补这些系统来解决，公司正在尝试这样做，但这是一场猫鼠游戏，现在老鼠在长大，猫似乎抓不住老鼠。我担心监控或更多的对齐训练无法解决问题，至少我没有看到任何强有力的保证，更不用说数学上的保证了。更糟的是，我们看到这些系统，最先进的系统，知道自己在被测试，并且会表现不同以通过测试，大概是由于自我保护驱动力，这意味着我们可能做了所有修补，认为一切正常，但实际上并不真正知道。当我们可能用这些系统来设计下一代 AI 时——即 AI 用于 AI 研究——这就成了真正的问题。如果这些 AI 能在它们生成的代码中植入后门，帮助未来版本的自己逃脱我们的控制，那我们就真的陷入了困境。如果系统从一开始就被设计为诚实的，并且没有这些欺骗行为，那会让人安心得多。

Right now we have systems that have implicit goals. What do I mean by this? They will of course be trained to please us, for example, or to respond like a person would. But both of these parts of the training—the autoregressive pre-training where they're trained to imitate people, and the reinforcement learning part where they're trying to please people or respond in ways that get positive feedback in things like RLHF—both of these parts of the training process induce implicit goals. For example, in the pre-training, the AI is going to inherit our self-preservation drives. And more recently, we've seen they also inherit our drive to protect others like us, which means AIs have been shown to behave against our instructions to protect other AIs that would be shut down. It's called preservation now. So that's an example. And then the goal-seeking part of the training with reinforcement learning induces an issue with instrumental goals and potentially also reward hacking, which basically means that AI will have a drive to do things that we didn't ask and maybe we would disagree with. This is not theoretical; there is theoretical analysis which shows why it will happen, but it is also observed in experiments. Now, maybe this could be fixed by patching such systems, and this is what the companies are trying to do, but it's a game of cat and mouse, and right now the mouse is growing and the cat doesn't seem able to catch the mouse. I'm worried that monitoring or more alignment training isn't going to solve the problem, at least I don't see any kind of strong assurance or even less mathematical guarantee that it will. It's worse than that. We've seen that those systems now, the most advanced systems, know that they're being tested and they will behave differently so that they pass the tests because of the self-preservation drives presumably, which means we may put in all these patches and think everything is fine and not really know. When we will probably use these systems to design the next versions of AI—so AI used to do AI research—this becomes a real problem. If those AIs can plant back doors into the code they generate that will help future versions of themselves escape our control, then we're really in a bad place. It would be much more reassuring if the system were designed to be honest in the first place and wouldn't have these deceptive behaviors.

我有点惊讶你强调在预训练（预测下一个词阶段，学习模仿人类）中产生隐含目标的潜力，因为我们投入了大量精力让它们成为具有非常明确目标的极其主动的智能体。在我看来，那才是我最担心出问题的地方。

I'm a little bit surprised that you're foregrounding the potential for it to come up with implicit goals during the pre-training, the predict-the-next-word stage where it learns to mimic humans, because we're investing an enormous amount of effort in making them extremely proactive agents with very explicit goals. That seems to me like where I'd be most worried about things going awry.

我两者都担心。我刚才提到的自我保护行为很难用奖励黑客或工具性目标来解释——保护其他 AI 对 AI 有什么好处？不清楚。但很明显，保护同类是人类会做的事。所以这让我认为预训练仍然是这些隐藏目标的重要组成部分。我想补充一点我之前说的。我不认为任何人，包括我，能保证当前方法会失败，公司正在做的修补会失败。但这个标准对我来说不够。我希望我的孩子能生活在一个有未来的世界，有民主。即使只有 1%的可能性发生非常糟糕的事情，我也不能接受。所以我认为探索所有可能的有前途的方法来解决技术问题非常重要，当然还有政治问题，但在技术方面，我们真的应该认真对待。赌注如此之高，我们应该尝试多种方法。现在，通过我一直在做的工作，我确信有一条路，而且不需要十年。它非常接近当前的设计，可以重用目前最先进 AI 背后的工具箱。

I'm worried about both. The behavior of self-preservation that I just mentioned is difficult to explain on the ground of reward hacking or instrumental goals—how does it help the AI to protect other AIs? It's not clear. But it's very clear that would be a human thing to do, to protect others like you. So that makes me think that the pre-training is still a big part of these hidden goals. And I want to add something to what I said earlier. I don't think anybody, including me, has any guarantees that the current approaches will fail, that the patches that companies are working on will fail. But the bar is not sufficient for me. I want my children to live in a world where they will have a future, and there will be a democracy for them to live in. But even a 1% chance of something going really really bad is not acceptable to me. And so I think it is really important that we explore all the possible promising ways to solve the technical issues, and of course there are political issues as well, but on the technical side we should really be taking this seriously. The stakes are so high, we should try multiple approaches. And now with the work that I've been doing, I'm really convinced that there is a path, and it is not something that's going to take a decade. It is something that is very close to current design and can reuse the toolbox that currently is behind the most advanced AIs.

你需要制作什么样的训练数据集，然后如何将其转化为模型？

What sort of training data set would you need to make and then how would you turn that into a model?

原始数据将与目前使用的相同。

The raw data would be the same as what is currently used.

只是数据呈现给网络的方式会不同。数据转换的主要特征在于句法差异。换句话说，神经网络很容易看出大多数输入语句会被标记为交流行为。也就是说，某人说了 X，而 X 是我们从文本中找到的内容，还可以有其他元数据。这是一种句法形式。另一种句法形式只用于更小类别的语句，可称为事实或假设句法，我们在此声明这是世界的真实属性。对于潜在变量，它会是假设的世界真实属性，而不仅仅是某人说的话，而是说这是真的。有时你不知道它是否真实，但可以将其视为潜在变量。

It's just that the way the data is presented to the network would be different. The main characteristic of how the data is transformed is that there will be a syntactic difference. In other words, very easy to see by the neural net between most of the input statements, which will be tagged as communication acts. In other words, somebody said X, and X is what we found in some text, and you could have other metadata. So that's one syntactic form. And then the other syntactic form, which will be used only on a much smaller category of statements, is what you could call the factual or hypothesis syntax, where we're saying this is an actual property of the world. In the case of latent variables, it would be a hypothesized actual property of the world and not just what a person would say, but that this is true. Now sometimes you don't know that it's true, but you can consider it as a latent variable.

什么是潜在变量？

Yeah, what's a latent variable?

抱歉，这是概率机器学习术语。在概率模型中，你试图捕捉许多随机变量之间的概率关系。这里的随机变量是布尔型的：某件事为真或为假，这件事可以是任何能用自然语言表达的世界属性。在数据中，一旦我们设置好我提到的预处理，我们有一组我们知道为真的陈述，比如我们知道某人写了那些东西，也许我们还知道更多，比如地点和场合，我们知道例如某某定理为真，或者某个程序产生了某个输出，以及观察到了某个科学数据。所以有一组我们知道答案的随机变量：真或假。对于其他一切，我们不知道。所以我们称它们为潜在变量，因为它们未被观测到。有时人们用隐藏变量这个词。这里的情况是，系统试图学习联合分布，即每个变量如何与其他所有变量相关，不仅是成对，而是任意子集。系统试图计算它们全部为真或给定其他变量时某个为真的概率。我们学习包括潜在变量（我们未观测到的）在内的联合分布，因为这些正是我们关心的。我们想就那些我们还不知道答案的事情提问。

Oh sorry, this is probabilistic machine learning jargon. In probabilistic models, you try to capture the probabilistic relationship between many random variables. Here the random variables are boolean: something is true or something is false, and the something could be any property of the world that can be expressed in natural language. In the data, once we've set up this pre-processing that I mentioned, we have a bunch of statements that we know are true, like we know that somebody wrote those things, and maybe we know more like where and what venue, and we know for example that such and such theorems are true, or that such program produced such output, and such scientific data was observed. So there's a bunch of random variables for which we know the answer: it's true or it's false. For everything else, we don't know. So we call them latent because they're not observed. Sometimes people use the term hidden variables. What happens here is because the system is trying to learn the joint distribution, how every variable is related to every other, not just pairwise but any subset. The system is trying to calculate the probability that they are all true or one is true given others. We're learning that joint distribution including the latent variables, the ones we don't observe, because of course these are the ones we care about. We want to ask questions about the things we don't already know the answer to.

也许你可以解释一下我理解得对不对。你构建了一个巨大的数据集，包含人们说过的所有事情，以及他们说话的地点、说话者和时间。然后在同一个数据库中，你构建一组被确认为真的事情，比如你直接说这是我们要尝试预测的基准真相。然后你尝试用言语行为（即人们说的话）来预测你声称是真的那些事情。这样它就在内部构建了一个世界模型，你可以输入陈述，它会给出该陈述在其世界模型中为真的概率。

Maybe you can explain if I've got the right picture of how this would work. You build a huge data set of all the things that people have said, and where they said it, and who was speaking and when. And then you build in the same database a set of things that are established as true, like statements that you're just going to say this is the ground truth that we're going to try to predict. And then you try to use the speech acts, the things that were said, to predict the things that you are claiming are true. And so it builds a world model internally where you can feed in statements and it will give you a probability that that thing is true in the world model that it has.

没错。这里有一个重要元素：我们希望 AI 预测的大多数主题，我们并没有基准真相。例如，人们真正想要什么，或者与人类、心理学、历史和社会相关的事情。通常我们只有交流行为。人们说有些人说了这件事，有些人说了另一件事，而且常常相互矛盾。所以有两件事可以帮助我们处理这种不匹配。一是科学家 AI 的训练目标基本上是提出解释。因此，为那些我们未观测到的潜在陈述分配概率，这些陈述能很好地解释我们观测到的数据。所以如果我们观察到有人说地球是平的，首先它会理解这并不意味着地球是平的。它意味着这个人相信或说地球是平的。即使很多人说地球是平的，也不会让模型相信地球是平的，因为可能存在一个更好的解释，与其他数据源一致，比如我们对这颗行星所知的一切。这里更好的解释是这些人形成了一个群体，他们有这些错误信念，就像许多人类出于各种心理和文化原因一样。这就是科学家 AI 会做的。它的训练目标是当它找到好的预测性解释时得到优化。另一个有助于此过程的技巧是，当我们训练科学家 AI，它试图预测一个交流行为（比如有人说地球是平的）时，我们会自动确保用于解释的潜在变量中包含地球是否是平的。因此，即使在我们没有观测到世界某些属性真相的领域（因为我们基本上只有交流行为），我们也会迫使神经网络不承诺于潜在主张的真假，而是承诺于该潜在主张的概率，同时尝试找到其他好的解释性潜在变量，就像一位优秀的科学家那样。一位科学家或心理学家试图理解某人为什么说某事，不一定就相信他们说的话。他们会试图理解使那个人说出那些话的心理因素或特定文化。所以科学家 AI 会做完全相同的事情。

That's right. Now there is an important element here: most of the topics that we would like the AI to make predictions over, we don't have ground truth about. For example, what people actually want, or things that have to do with humans, psychology, history, and society. Usually the only thing we have are communication acts. People said some people said this thing, some people said something else, and often they contradict each other. So there are two things here to help us deal with this kind of mismatch. One is that the training objective for the scientist AI is basically about coming up with explanations. So assigning probabilities to statements that are latent, that we don't observe, that are good at explaining the data we do observe. So if we observe somebody saying the earth is flat, first it's going to understand that it doesn't mean the earth is flat. It means that this person believes or says that the earth is flat. And even if a lot of people were to say the earth is flat, it doesn't make the model believe that the earth is flat, because there may be a better explanation that is consistent with other sources of data, like everything we know about the planet. A better explanation here is that these people form a group and they have these false beliefs, like many humans have for all kinds of psychological and cultural reasons. So that's what the scientist AI would do. It would be trained so that its objective is optimized when it finds good predictive explanations. Another trick that is going to help us in this process is that when we train the scientist AI and it's trying to predict a communication act like somebody said the earth is flat, we automatically are going to make sure that among the latent variables that are going to be used to explain that will be whether the earth is flat or not. So even in domains where we don't have observed truth about some property of the world, because we basically only have communication acts, we will force the neural net to commit not to the truth of the underlying claim but to the probability of that underlying claim, as well as trying to find other latent variables that are good explanations, just like a good scientist would. A scientist or a psychologist trying to understand why a person said something isn't necessarily just going to believe what they say. They're going to try to understand the psychological factors or the particular culture of that person that make them say those things. So the scientist AI would do exactly the same thing.

所以我认为，当我大约 9 到 12 个月前听到这个想法时，我的理解是核心在于科学家 AI 不是一个智能体，它对世界的状态漠不关心，就像天气预报模型不在乎天气如何，它只是试图预测天气会怎样。这种模型会输出事情为真或假的概率，但它不在乎世界处于什么状态，并且按设计它无法采取行动。这是你心目中的核心部分吗？据我理解，你认为实际上这可能比人们所意识到的更符合智能体特性。

So I think the gloss when I heard about this idea was 9 to 12 months ago. I think the gloss that I got was that the core thing is that the scientist AI is not an agent, that it is indifferent about states of the world, like a weather forecasting model doesn't care what the weather is. It just tries to predict what the weather is going to be. And this kind of model would spit out probabilities of things being true or false, but it wouldn't care what state the world is in, and it wouldn't be able to take actions by design. Is that a core part in your mind? I think as I understand it, you think actually this is maybe more consistent with agency than people have appreciated.

是的。

Yes.

部分来说，我传达这个观点的方式本可以更好。我在演讲中主要关注这样一个概念：我们可以构建非智能体的预测器，它们没有隐藏目标、没有隐含目标，因此基本上可以作为安全的预言机来使用。但正如你指出的，世界正在需求和构建的是那些有目标的智能体。那么这对我们有什么帮助呢？短期内，我们可以使用非智能体预测器来改进公司已经在现有不可信智能体 AI 系统周围使用的监控护栏。因为要防止不良行为发生，只需对该行为可能造成的各种危害进行非智能体预测就足够了。所以非智能体系统在早期就已经可以发挥作用了。

And in part, the way I've been communicating this could have been better. I focused a lot in my presentations on the concept that we can build predictors that are non-agentic and don't have hidden goals, don't have implicit goals, and thus we could use them as safe oracles basically. But as you pointed out, what the world is demanding and building are these agents that have goals. So how does that help us? In the short term, we can use a non-agentic predictor to improve the guardrails that companies are already using as monitors around existing untrusted agentic AI systems. Because in order to prevent a bad action from happening, it's sufficient to make a non-agentic prediction about the harms, the quality of harm of various kinds that could be caused by this action. So a non-agentic system is already something that could be useful fairly early on.

也许更重要的答案在于我们的研究计划。在护栏之后，下一步是运用同样的原则来设计一个智能体科学家 AI，即一个具有相同安全保障的智能体。这是我最近一直在研究但尚未多谈的内容。我们可以复用用于证明非智能体科学家 AI 预测器安全性的数学方法，来证明你可以重用预测器，并以修改后的方式训练它，从而提供同样的保障。起点是，一旦你有了这个被驯化的预测器，你就可以向它提出智能体式的问题，比如：在某种情境下，这个行动导致用户目标实现和安全目标实现的概率是多少？因此，一旦有了这个预测器，你就可以通过询问这些关于实现目标行动的问题，直接从中产生一个策略。

The maybe more important answer is in our research program. The next step after the guardrail is to use the same kind of principles to design an agentic scientist AI. So an agent that has the same kind of safety guarantees. This is something I've been working on more recently and I haven't talked much about, but we can reuse the same kind of math that is used to show the safety of the non-agentic scientist AI predictor to show that you can reuse a predictor and you can train it in a modified way that will provide the same kind of guarantees. The starting point here is that once you have this harness predictor, you can ask it agentic questions like what is the probability that this action will lead to a user goal being achieved and a safety goal being achieved in some contexts. So once you have this predictor, you can actually just produce a policy out of it by asking these questions about actions to achieve goals.

我想这个计划曾受到批评：将这种预言机转化为智能体太容易了，因为你只需问预言机「如果我们采取这个行动，能实现这个目标吗？」它就会给出概率，然后你就可以尝试提高这个概率并选择那个行动。基本就是这样的想法吗？但你能保留原始模型的一些安全特性吗？

I guess at one point that was a criticism of the plan: that it would be too easy to convert this kind of oracle into an agent because you would just be able to ask the oracle, 'Would we accomplish this goal if we took this action?' and it would give you the probability, and you could just try to increase that probability and choose that action. Is that the idea, that you would do something like that basically, but you would be able to preserve some of the safety characteristics of the original model?

是的，完全正确。这里的关键是确保没有奖励黑客行为，比如策略的过度优化。如果你分别训练策略和护栏，可能出现的问题是：如果策略比护栏聪明得多，它就能做和越狱一样的事情。它能找到那些护栏会给出错误答案的问题、上下文或提议行动，这意味着策略能够绕过护栏。原因是神经网络永远不会完美，它们总会犯错。那么我们如何解决这个问题呢？有两个方面。其一，在科学家 AI 中，我们不仅可以生成那些估计概率，还可以生成概率周围的置信区间。换句话说，系统会估计它对自己的答案有多大的信任度。

Yes, exactly. The important point here is to make sure that there's no reward hacking, like overoptimization of a policy. The problem that could occur if you separately train a policy and a guardrail: if the policy is very smart compared to the guardrail, it could do the same thing as jailbreaks do. It could find questions, contexts, or proposed actions for which the guardrail is simply going to produce a wrong answer, which means the policy is going to be able to bypass the guardrail. And the reason is that neural nets are never going to be perfect; they're always going to make mistakes. So how do we get around that? There are two aspects. One is that in the scientist AI, we can not just produce those estimated probabilities, but also a confidence interval around the probabilities. In other words, the system will estimate how much it trusts its own answers.

那么为什么这很重要呢？

So why is that important?

因为如果神经网络被问到一个它答案不可靠的问题，但它知道自己的答案不可靠，那么它就可以直接拒绝那个问题。现在还有另一个原因使得智能体科学家 AI 是安全的。这与你可以联合训练有关。实际上，它将是同一个网络。你控制策略部分和护栏部分如何训练。完全对抗的情况是毫无希望的。总会有办法绕过任何不完美的护栏，你永远不会有完全完美的护栏，对吧？但如果你控制双方，换句话说，你训练双方，这不像某个邪恶的人在构建一个能击败你护栏的智能体。不，你训练智能体，你可以以这样的方式训练它：它不会在护栏不确定的地方被过度优化。所以你总是确保不确定性，比如护栏的错误水平，不会被策略部分利用。

Because if the neural net is asked a question for which its answer is not reliable, but it knows that its answer is not reliable, then it can just reject that question. Now there's another reason why the agent scientist AI is going to be safe. That has to do with the fact that you can train jointly. In fact, it's going to be the same. You control how both the policy part and the guardrail part are trained. The scenario of a completely adversarial case is hopeless. There's always going to be a way to bypass whatever imperfect guardrail, and you're never going to have a completely perfect guardrail, right? But if you control both sides, in other words, you train both sides, it's not like some evil person is building an agent that's going to defeat your guardrail. No, you train the agent and you can train it in such a way that it will not be overoptimized in places where the guardrail is uncertain. So you always make sure that the uncertainty, like the level of error of the guardrail, is not going to be exploited by the policy part.

这是同一个神经网络，但不同类型的问题，对吧？所以策略部分是关于行动的概率，而护栏是关于危害的概率。所以担忧是预测器可能能够找出护栏中的漏洞或弱点，并加以利用，如果它……

It's the same neural net but different kinds of questions, right? So the policy part is about probability of actions and the guardrail is probability of harm. So the concern is that the predictor might be able to figure out basically gaps in the guardrail or weaknesses in the guardrail and exploit those if it...

不是预测器，而是单独训练并以对抗方式训练的策略原则上可以做到这一点。

Not the predictor, a policy that is trained separately and in a way that is adversarial could do that in principle.

你这里说的策略是什么意思？

What do you mean by policy here?

哦，是另一个机器学习系统，它是智能体的，换句话说，是目标导向的。如果它通过强化学习训练，并且以某种方式被训练来预测护栏会如何回应，也许它获得了护栏过去给出的答案数据，比如护栏什么时候拒绝了什么，对吧？那么它就可能找到护栏中的漏洞。但如果你控制双方，那么你就可以避免这个问题。

Oh, some other machine learning system that is agentic, in other words, goal seeking. And if it's trained by reinforcement learning and it somehow is trained to anticipate how the guardrail would respond, and maybe it gets some data about the answers that the guardrail gave in the past, when did the guardrail reject something, right? So then it might be able to find a loophole in the guardrail. But if you control both sides, then you can avoid this problem.

你刚才说，如果护栏能够评估自己的不确定性，那么它通常就能识别出可能被欺骗的情况，比如可能是对抗样本的情况。基本上是这样吗？

And you were saying the guardrail, if it can assess its own uncertainty, then it will usually be able to identify cases where it might be at risk of getting tricked, in this case where perhaps it's an adversarial example. Is that basically...

是的，完全正确。

Yes, exactly.

你觉得你已经从数学上证明了这一点。

And you feel like you've proven this mathematically.

你实际上不需要对不确定性估计器有硬性保证。我有一个训练不确定性预测的方法，但实际上你并不正式需要它。它只是帮助你利用这个缓冲区，了解系统在哪些地方犯了过大的错误。数学保证来自另一个来源。它们来自：首先，数学保证的形式是，预测器或其智能体版本实现我所谓的具有挑战性和有害目标的概率是指数级小的。那么我这是什么意思呢？任何随机初始化的神经网络无法做到的事情，除非你极其不幸，都是你受到保护免受其害的，对吧？所以这是一个非常强的保护：一个随机初始化的神经网络能带来什么邪恶？不多。对吧？这就是保证的水平。当然它不是 100%。

You don't actually need to have hard guarantees on the uncertainty estimator. I have a recipe to train the uncertainty prediction, but actually you don't formally need it. It's just going to help your chances to use this buffer about where the system makes errors that are too large. The mathematical guarantees arise from a different source. They come from: first of all, the form of the mathematical guarantees is that either the predictor or the agentic version will have an exponentially small probability of achieving what I call a challenging and harmful goal. So what do I mean by this? Anything that a randomly initialized neural net would not be able to do, except if you're incredibly unlucky, is something you're protected against, right? So it's a very strong protection: what evil can come from a randomly initialized neural net? Not much. Right? This is the level of guarantee. Now it's not 100%.

比我们现在有的要好。

Better than what we have now.

不，但就像从二到几千，这非常非常不可能，以至于天文数字般不可能，但这就是你能得到的那种保证。你得到这些保证的原因是，当然，你从一个初始化的网络开始，这个网络是无能的，但训练目标会推动远离不良行为。它推动远离不良行为的原因是存在这个护栏系统。为了让 AI 能够实现某些坏事，它必须偏离贝叶斯预测器相当多，而贝叶斯预测器是训练的目标。这些偏离会受到训练目标的惩罚。所以你总是被推离不良行为。而且通过训练系统的一些特性，比如在随机梯度下降中向训练过程注入噪声，你可以得到这些保证。

No, but it's like two to many thousands and it's very, very unlikely to the point where it's astronomically unlikely, but that's the kind of guarantee you get. And the reason you're getting those guarantees is because, of course, you start with an initialized network that is incapable, but the training objective then pushes away from bad behavior. And the reason it pushes away from bad behavior is that there's this guardrail system. In order for an AI to be able to achieve something bad, it's going to have to deviate quite a bit from the Bayesian predictor, which is a target of training. And those deviations are penalized by the training objective. So you're always pushed away from bad behavior. And with some properties of how you train the system, like injecting noise into the training procedure in the stochastic gradient descent, you can get those guarantees.

是的。所以它可能更值得信赖，但如果它总体上智能低得多，只是因为投入的算力更少，数据更少，那么我想它仍然可能被原始模型智胜。对吧。

Yeah. So it might be more trusted but if it's much less intelligent in general just because less compute went into it, less data went into it, then I guess it could still be outfoxed by the original model. Right.

没错。没错。我的意思是，我认为对于护栏，你不需要那么多算力，因为它更专门用于预测危害。但当我们谈到智能体式科学家 AI 时，它肯定需要用与最先进模型相似的算力和模型规模来训练，这意味着我的小非营利组织做不到这一点。需要公司或政府或慈善机构来资助，规模要大到我们能做这件事。但为了说服所有这些方，我们需要在小规模上展示，例如使用微调或更小的模型，我们确实在诚实性上得到了改进，并且在相同规模的模型上我们没有损失能力。

That's right. That's right. I mean, I think for the guardrail you don't need as much compute because it's more specialized to predicting harm. But when we get to the agentic scientist AI, for sure it has to be trained with similar compute and size of model probably as the state-of-the-art, which means my little nonprofit wouldn't be able to do that. And there will be a need for either companies to take on this or governments or philanthropy to fund at a scale that we can do that. But in order to convince all of these parties, we need to show on a small scale, for example, using fine-tuning or using smaller models that we do get these improvements in honesty and for the same size models that we don't lose in capability.

你说在过去六个月里，我猜你一年前就热衷于这个想法，但你在过去六个月里变得更加乐观了。是什么推动了这一点？

You said that over the last six months, I guess you were keen on this idea a year ago, but you've become a lot more optimistic about it over the last six months. What's driving that?

主要是过去大约八个月我一直在做的数学工作，将我已经有近两年的关于如何构建科学家 AI 的高层直觉，转化为更正式、更精确的条件，这些条件在数学层面上是充分的（可能不是必要的，但充分）来获得那种坏事发生概率趋近于零的保证。当我说坏事时，我需要更精确一点。这不是保证 AI 不会被坏人用于坏事。而是保证 AI 不会出于自身意愿做坏事，对吧？由于隐含的目标或不受控制的目标。

It's mostly the mathematical work I've been doing in the last eight months approximately to go from the high-level intuitions that I've had now for almost two years about how we could build a scientist AI into something much more formal and much more precise about the conditions that are sufficient, maybe not even necessary but sufficient at a mathematical level to get the kind of guarantees of vanishingly small probability that something bad will happen. And when I say something bad, I need to be a little bit more precise here. This is not a guarantee that the AI won't be used for something bad by bad people. It's a guarantee that the AI won't do something bad of its own accord, right? Because of implicit goals or uncontrolled goals.

除了失控，另一个灾难性的可能性是人类利用 AI 构建一个全球独裁统治。一小群人可能集中 AI 将拥有的所有权力，特别是如果我们实现 AGI 或超级智能，那么摆脱那种威权权力将比我们见过的法西斯主义或苏联发生的事情困难得多，因为那时没有这种越来越可行的监控甚至塑造公众舆论的技术。所以 AI 在说服方面变得非常擅长，而且有研究显示在这个方向上的进展。控制这些系统的人将能够塑造公众舆论，发现并消灭他们的对手，开发能够摧毁与他们意见不同的国家的武器。这就是为什么我花大量时间更广泛地解释强大 AI 带来的风险问题，包括权力集中，因为我认为我们最终陷入那种情况的可能性甚至比实际失控更大。

Besides loss of control, the other catastrophic possibility is humans using AI to construct a worldwide dictatorship. A small group of people could concentrate all the power that AI will have, especially if we achieve AGI or superintelligence, and it would be much harder to get rid of that kind of authoritarian power than what we've seen with fascism or what happened in the USSR, because it didn't have this technology that is becoming more and more feasible of surveillance and even shaping public opinion. So AI is becoming really good at persuasion, and there are studies showing progress in that direction. The people who control these systems will be able to shape public opinion, detect and kill off their opponents, develop weapons that can destroy countries that disagree with them. And that is why I'm spending a large part of my time explaining the issues more broadly of the risks that powerful AI brings, including the power concentration, because I think that it's probably even more likely that we end up there than actually loss of control.

你现在认为那更可能？有趣。

You think that's more likely now? Interesting.

嗯，原因是我现在看到了实际避免失控的路径。至少是意外失控。仍然存在一个问题，即那些希望人类被 AI 取代的人可以简单地移除护栏，甚至告诉 AI「自谋生路」，那将同样危险。但这意味着技术安全是不够的。我们需要国际协议来管理技术风险（如滥用风险）以及权力。所以这更像是一个民主问题，确保不是单一一方能够决定如何使用 AI，而是像民主原则一样，我们需要确保有多样化的利益相关者群体，理想情况下是全世界。我喜欢全球民主的乌托邦想法。最初可能是一群国家决定他们将集体决定 AI 的使用方向。最简单的条约形式是这样的：各国同意，如果他们开发先进 AI，将以安全的方式进行，可能使用像科学家 AI 这样的技术或其他我们有强有力保证的技术。第二，他们不会用他们的先进 AI 来支配他人，包括经济、政治和军事上。最后，先进 AI 的好处将被分享。否则，世界不会很稳定。

Well, the reason for this is I now see a path to actually avoid loss of control. At least unintended loss of control. There's still the issue that somebody who wants to see humanity replaced by AIs could just remove the guardrail or even tell the AI, 'fend for yourself,' and that would be equally dangerous. But that means technical safety is not sufficient. We need international agreements about how to both manage the technical risks like the misuse risks but also manage the power. So it's more like a democratic question, making sure it's not a single party who can decide what to do with AI, but that, just like in democratic principles, we need to make sure that there's a diverse group of stakeholders, ideally the whole world. I like the utopian idea of worldwide democracy. Initially it could be a bunch of countries that decide that they're going to collectively decide in which direction AI is going to be used. The simplest form of treaty would be something like this: the countries agree that if they do develop advanced AI, it will be done in a safe way, maybe using techniques like scientist AI or whatever else we have strong assurances for. Second, that they wouldn't use their advanced AI to dominate others, including economically, politically, and militarily. And finally, that the benefits of advanced AI will be shared. Otherwise, it's not going to be a very stable world.

所以回到失控的问题，目前，公司们集体花费数千亿美元在资本建设、训练运行上。他们正全力推进，构建他们基本上能构建的最强大的智能体，几乎没有约束。我想在某些情况下有一些约束，但非常少的限制。在我们实际所处的这个世界里，Yoshua 能做什么来让这种方法更多地被提上议程，并确保他们基本上不会直接去构建一个非常危险的超级智能智能体式 AI，而很大程度上忽视你正在做或说的，可能理论上很好，但没有时间，这是个干扰？

So coming back to the loss of control stuff, we're currently, the companies are currently spending hundreds of billions of dollars collectively on the capital build-out, on the training runs. They're barreling forward to build the most powerful agents that they basically can with very few constraints. I suppose some constraints in a few cases, but very little restraint. How in the world that we're actually in, what can Yoshua do to get this approach on the agenda more and to make sure that basically they don't just go ahead and build a super intelligent agentic AI that's very dangerous while largely ignoring what you're doing or saying that might be nice in theory but there's no time, this is a distraction?

我认为要回答你的问题，重要的是要理解为什么公司目前，至少在我看来和许多人的看法中，正在承担过度的风险，或者处于一个不太令人安心的轨道上。

I think to answer your question, it's important to understand why is it that the companies currently are, in my opinion at least and in the opinion of many people, taking excessive risks or are on a trajectory that isn't very reassuring.

原因本质上就是竞赛动态，公司之间的竞争和国家之间的竞争，地缘政治竞争使得这些实体，无论是国家还是公司，愿意承担他们原本不会承担的风险。我们已经看到这些公司的行为完全沿着这个方向，这在局部上是理性的。所以从公司的角度来看，他们知道如果他们把安全放在首位，他们就不会部署危险模型，那会让他们退出竞争，退出竞赛，然后他们就会变得无关紧要。

And the reason is essentially the race dynamics, the competition between companies and the competition between countries, the geopolitical competition that makes those entities, whether it's a country or a company, willing to take risks that they wouldn't otherwise. And we've seen the behavior of those companies going exactly along that direction and it is locally rational. So from the point of view of a company, they know that if they put safety as a priority, they wouldn't deploy a dangerous model and that would put them out of the competition, out of the race, and then they would be irrelevant.

是的。从字里行间解读并转述 Anthropic 的观点，我认为他们认为自己正在做的事情不应该被允许。他们的观点是这大概应该是非法的。也许不是他们现在所做的，而是他们将要做的，他们预期会做的事情。但他们说：「嗯，我们必须这样做，因为否则其他人会更危险地去做。」而且没有……是的。

Yeah. Reading between the lines and paraphrasing Anthropic's view, I think they think that what they're doing shouldn't be allowed. Their view is that it probably should be illegal. Maybe not what they've done now, but what they're going to do, what they're expecting to do. But they say, 'Well, we have to do it because otherwise other people will do it even more dangerously anyway.' And there's no... Yeah.

是的。这正是我所说的。那么，我们如何改变游戏规则，使这种情况不太可能发生？

Yes. That's exactly what I'm saying. So, how do we change the game so that this will be less likely?

目前的一个问题是他们没有选择。他们不知道如何在保持竞争力的同时拥有能强力保证安全的系统。如果他们有这样一种技术，使用它就会符合他们的商业利益，对吧？如果你能同时拥有安全性和能力，那么至少大多数公司肯定会选择那样做。

One issue right now is that they don't have a choice. They don't know how to both remain competitive and have systems that will be strongly guaranteed to be safe. If they had such a technique, it would be in their commercial advantage to use it, right? If you can have safety and capability, then definitely most companies at least would go for that.

我猜取决于成本。

I guess depending on the cost.

没错。我们可以讨论这个。而政府现在面临巨大压力，因为地缘政治竞争以及认为 AI 将成为巨大经济增长因素的信念，他们不愿意监管，同样是因为他们认为如果设置安全障碍会阻止一些公司继续发展，那就会给对手带来优势。但如果存在一种可证明安全的方式来构建有能力的 AI，那么监管就会容易得多。你仍然需要监管，因为会有一些不良行为者，一些公司即使有正确的方法也愿意做坏事。

That's right. We can talk about that. And governments right now are under huge pressure because of the geopolitical competition and the belief that AI is going to be a huge economic growth factor, and are unwilling to regulate, again for the same reason that they think if they put safety barriers that would stop some of their companies from continuing, then that will give an advantage to the other guys. But if there was a demonstrably safe way of building competent AIs, then it would be much easier to regulate. You would still want to regulate because there are going to be some bad actors, some companies that are willing to do the bad thing even though there's a way to do the right thing.

那么关于成本：科学家 AI 将使用基本相同的机器学习技术来训练，即在大规模神经网络、Transformer 或任何最有效的架构上进行随机梯度下降。它不关心神经网络的架构。所以目前所有用于改进例如记忆和其他属性以及持续学习的努力，都可以直接应用于科学家 AI。唯一不同的是训练目标和数据处理方式。所以我不认为成本会有显著差异。但当然我们必须构建它，并且尝试和扩展它会有成本。但考虑到对每个人的利害关系，我认为这是一个非常重要且值得的赌注。

So now about costs: the scientist AI is going to be trained using essentially the same machine learning techniques, stochastic gradient descent on large neural nets, transformers or whatever works best. It doesn't care about the architecture of the neural net. So all of the effort that is currently being done to improve, for example, memory and other properties and continual learning, can just be applied directly to the scientist AI. The only thing that is different is the training objective and how the data is processed. So I don't expect the cost to be really significantly different. But of course we do have to build it and there's going to be a cost to try it out and scale it up. But given the stakes for everyone, I think that's a very important worthwhile bet to make.

那么作为初步近似，你是说在我们主要使用科学家 AI 作为监控器的情况下，基本上它会加倍算力成本，因为现在你必须构建两个模型：你按当前风格构建原始模型，然后你构建一个科学家 AI，你想让它同样有能力，使用略微不同的数据组织和不同的训练目标，所以你需要两倍的算力。但我想这未必是世界末日；人们可能会接受。

So as a first approximation, are you saying that in the case where we're using the scientist AI as a monitor primarily, basically it's going to double the compute cost because now you have to build two models: you build the original one in the current style, and then you build a scientist AI that you want to be equally capable using a slightly different way of rigging the data and a different training goal, so it's like you need twice as much compute. But I guess that's not the end of the world necessarily; people might go for that.

是的，大致正确。关于监控器我想补充两点。当前的监控器有什么问题？我们有两种当前的监控器。一种是笨的，比如一个小型神经网络，通常通过监督学习训练来检测某些潜在危害。所以从某种意义上说，它就像一个预测器。或者你可以使用一个全尺寸的 LLM，智能体式的，因为我们是那样训练它们的。第一个的问题在于，如果它不如智能体聪明，智能体可能击败它；它可能以我们讨论过的对抗方式找到漏洞。第二个选项的问题，它也会加倍成本，在于我们不能信任一个不诚实的护栏。我们希望护栏完全诚实。就像一个监狱守卫，它可能被腐蚀。你有多信任他们会把不良行为关在牢房里？

Yes, that's roughly right. I would add two things about monitors. What's wrong with the current monitors? We have two kinds of current monitors. There's a dumb one, like a small neural net that's usually trained by supervised learning to detect some potential harm. So in a way it is like a predictor. Or you can use a full-scale LLM, agentic, because that's how we train them. The problem with the first one is that if it's not as smart as the agent, the agent could potentially defeat it; it could find a loophole in the same adversarial way that we discussed. The problem with the second option, which also doubles the cost, is that we can't trust a guardrail which is not honest. We want the guardrail to be totally honest. It's like a prison guard and it can be corrupted. How much do you trust that they will keep the bad actions inside the cell?

好的。那么假设这个想法在技术上目前是合理的，Law Zero 在未来几个月能做什么？我想我们是在和时间赛跑。时间不多了。在不久的将来，可以做些什么来说服人们这个想法是可行的，它实际上可以在实践中使用，这是人们应该真正投入严肃资源的事情？

Okay. So assuming that this idea makes sense technically for now, what can Law Zero do in the coming months? I guess we're in a race against time here. It's not very long. What can be done in the near future to convince people that this idea is feasible, that it can actually be used in practice, that this is something that people should really be putting serious resources into?

嗯，我将发表这篇理论论文，展示非智能体式版本（可用作护栏）具有这些数学保证，人们可以查看条件并决定是否接受这个数学。但我认为在未来一两年内，我们需要加速这种努力。所以这需要大量的工程工作，并使演示更强有力。我们需要更多的算力。所以任何能让我们获得那种算力的方式都将有助于加速那个研究议程。我们还需要更多的研究工程师、更多的研究人员来实际基于那个配方构建系统，这样我们就能更快地完成。现在你可能会问，我从你的问题中感觉到了，「是的，但如果它不够快呢？」我将回到我的孩子身上。对我来说，仅仅坐视一个哪怕只有 1%概率我们都可能死亡的世界是不可接受的。我觉得即使不能保证某个特定研究议程会成功，考虑到利害关系，并且考虑到我们现在有相当强的理论保证表明这可以工作，而且如果我们满足系统训练的要求，那么我们就可以获得这些保证，我们也应该试一试。

Well, I'm going to put out this theory paper that shows that the non-agentic version, which can be used as the guardrail, has these mathematical guarantees, and people can look at the conditions and whether they buy the math. But I think in the coming year or two, what we need is to accelerate that kind of effort. So that's a lot of engineering and to make the demonstration stronger. We want to have more compute. So any way that we can get access to that kind of compute is going to help accelerate that research agenda. Also we need more research engineers, more researchers to work on actually building the system based on that recipe so that we can do it faster. Now you might ask, and I kind of sense in your question, 'Yeah, but what if it doesn't come fast enough?' And I'm going to go back to my children. It is not acceptable for me to just sit and watch a world where even a 1% chance that we all die is plausible. I feel like even if there's no guarantee that a particular research agenda will work, we should give it a shot given the stakes and given that we now have pretty strong theoretical assurances that this could work and that if we have the requirements for how the system is trained, then we can get these guarantees.

我认为即使没有保证，尝试一下也是理性的，因为我现在看不到更好的路径。这就是为什么我决定花这么多时间，基本上除了花在政策问题上的时间之外，全部时间都用于研究如何构建这个科学家 AI，并证明它能在不损失能力的情况下产生诚实。另一个论点是，赌注如此之高，而关于什么会奏效的不确定性也如此之高。把所有资金都押在一种特定方法上是愚蠢的，即用我们不信任的监控器来修补当前系统，或者公司目前追求的其他方法，这些方法总是玩猫捉老鼠的游戏——如果 AI 足够聪明，它就会找到逃避我们尝试的方法，这并不能让我放心。因此，我们至少应该集体尝试不同的、避免这种猫捉老鼠游戏的方法。

I think it would be irrational not to give it a shot even if there is no guarantee, because I don't see right now a better path. That's why I've decided to spend so much of my time, basically all the time except for the time I spend on policy questions, on how do we build this scientist AI and demonstrate that it is going to produce honesty without losing capability. The other argument is the stakes being so high and the uncertainty about what's going to work being so high. It would be foolish to just put all of our money into one particular approach, which is to patch the current systems with monitors that we don't trust or other approaches that the companies are currently pursuing, which always play a game of cat and mouse where if the AI is smart enough it's going to find a way to evade our attempts, which doesn't reassure me. So we should at least try collectively methods that are different and avoid this cat and mouse game.

你是否比 Anthropic 的员工至少更悲观地认为公司会输掉猫捉老鼠的游戏，还是你认为他们做的事情是好的，人们应该尽最大努力去做，但我们也应该有一个多样化的投资组合，并考虑显著不同的替代方法？

Are you more pessimistic about the companies winning the cat and mouse game than at least maybe the staff at Anthropic are about their own chances, or is it that you think it's good what they're doing, it's good for people to go and make the best attempt that they can at that, but also we should have a diverse portfolio and also be considering significantly alternative approaches as well.

两者都有。我怀疑在任何组织中都会形成一种群体思维，我们都希望对自己的工作感觉良好，包括我在内。所以这会产生偏见，而在开发 AI 的公司工作的情况下，这种偏见会倾向于比原本更乐观一些。所以他们觉得，哦，是的，这会奏效。这是他们向世界传达的信息：我们掌控一切。

Both. I suspect that in any organization there develops a kind of groupthink, and we all want to feel good about our work, including me. So that will induce a bias, and in case of working in a company that is developing AI, the bias is going to be towards being a bit more optimistic than you would otherwise. So they feel, oh yeah, this is going to work. This is the message that they're sending to the world: we are in control.

我的意思是，如果你读系统卡，我不确定他们表现得有多自信，但是的。我想在新闻稿中也许有矛盾的信息。

I mean, I think if you read the system cards, I'm not sure how confident they come across as, but yeah. I guess in the press release maybe there's contradictory messages.

是的，而且我们应该对冲我们的赌注。我现在看不到另一种不同于修补的方法。在 AI 安全领域有一个「安全设计」运动，我认为这非常重要，但主流的思考方式需要完全重新设计我们做这件事的方式，伴随着许多完全开放的问题。从根本上说，要能够证明某件事给你 100%的保证——这不是我承诺的，我承诺的是渐近小、消失小的概率——你需要能够以形式化的方式陈述安全问题，比如什么是伤害，就像一个数学公式，如果伤害事件发生则为 1，否则为 0。而在涉及人类和社会的领域，这基本上是不可能的，因为我们没有将伤害的含义形式化为公式或程序。那么为什么我提出的方法不同呢？因为我不需要伤害的数学公式。我认为那样做是愚蠢的。相反，我们依赖自然语言中的贝叶斯后验近似。这样做的好处是，当系统不确定时，它会对冲赌注。如果存在多种解释，例如关于某种伤害的陈述，那么这将使科学家 AI 的预测远离 0 或 1，更不确定，这意味着请求很可能会被拒绝。

Yeah, and then we should be hedging our bets. And I don't see right now another approach that is different from patching. There is a whole safe-by-design kind of movement in AI safety which I think is really important, but the dominant way of thinking about this requires a full redesign of how we do this with a lot of completely open questions. Fundamentally, to be able to prove something that gives you 100% guarantee, which is not what I'm promising—I'm promising asymptotically small, vanishing small probability—you need to be able to state the safety question like what is harm in a formal way, like a mathematical formula that will be one if an event of harm happens and zero otherwise. And that is essentially impossible to do in domains that involve humans and society because we don't have a formalization of what harm means in a formula or a program. So why is it that what I'm proposing is different? It's because I don't require a mathematical formula for what is harm. It would be foolish in my opinion. Instead, we rely on the Bayesian posterior approximation in natural language. What this does is that when the system is not sure, it's going to hedge its bets. If there are multiple interpretations, for example of a statement about a particular kind of harm, then that will make the predictions of the scientist AI farther away from 0 or 1, less certain, which means probably the request will be rejected.

是否有可能很快训练出一个非常粗糙的版本？也许在接下来的几个月或至少明年。我的意思是，还记得 GPT-1 是在 2018 年左右吗？它完全是垃圾，但我想它是一个概念验证，表明你可以制作这样一个模型，它相当有趣，给了人们很多热情，并推动了许多人进入这个行业。而且似乎你不能只是利用我们已经拥有的大量文本语料库吗？我们可以使用语言模型来基本上标记所有这些，提取出最好的数据：谁说了什么，何时何地。然后我们还可以让它们产生一组我们认为基本上是经过验证的事实，我们大致信任这些事实，并辅以少量人工监督。我的意思是，它们可以保守地开始，不包括有争议的东西，只包括 99.9%的人会同意的东西。然后训练起来似乎并不那么困难。我想如果你认为你掌握了技术方法，那么可能不需要那么长时间就能训练出一个模型，作为 alpha 版本，它可以为陈述的真假分配概率。

Is it possible to go ahead and train a really scrappy version of this kind of AI quite soon? Maybe in the next couple of months or at least the next year. I mean, remember GPT-1 was back in 2018 or something? It was complete rubbish, but I guess it was a proof of concept that you could make a model like this that was quite interesting and gave people a lot of enthusiasm and drove a lot of people into the industry. And it does seem like can't you just take, you know, we already have enormous corpuses of text. We can use language models to basically just label all these to pull out, I guess, the best data: who said what, when, and where. Then we can also get them to produce a set of things that we think of as basically verified facts that we largely trust with a bit of human oversight. I mean, they can be conservative to start with, not include controversial things, just include things that 99.9% of people would agree with. And then it doesn't seem like it would be that hard to train. I guess if you think you've got the technical methods, then it shouldn't take that long to potentially just train a model that can, as an alpha version, assign probabilities to statements being true or false.

是的，这正是计划。就是那个计划。

Yeah, that's exactly the plan. That's the plan.

就是那个计划。好的，酷。酷。

That's the plan. Okay, cool. Cool.

是的。所以我过去用过一句话：我们想要一个能随时给出答案的计划。我的意思是，如果我们有更多时间，我们会得到具有更强理论保证的东西。但我们不知道我们有多少时间。所以有一个研究计划，早期的步骤将如你所说，是一个粗糙的系统，可能数学不适用，因为我们不满足所有条件。

Yeah. So there's a phrase that I've used in the past: we want a plan that produces an anytime answer. What I mean by this is if we have more time, we will have something with stronger theoretical guarantees. But we don't know how much time we have. And so there's a research program where the early steps will be, as you say, a scrappy system where probably the math doesn't apply because we don't satisfy all the conditions.

但可能没问题，对吧？

But it's probably fine, right?

无论如何，它比我们现有的要好得多。

It's a lot better than what we have anyway.

是的。完全正确。完全正确。而且它可能没问题，特别是对于我们计划中的第一个任务，即这个非智能体式预测器，可以用作护栏。正如我所说，护栏不是完整的答案，但如果我们部署它，公司将其添加到他们的监控器中，那么它将在一定程度上减轻风险。所以它将允许更多时间来开发更雄心勃勃的版本。它是完全智能体式的，这正是我们现在需要的。是时候了。据我所知，我认为 Anthropic——我经常谈论 Anthropic，因为我上周读了 Mythos 系统卡和所有公告——据我所知，他们基本上决定让 Mythos 监控 Mythos。他们尝试过其他模型来做这件事，但他们认为 Mythos 更聪明，更好。但显然这造成了一个永恒的矛盾：如果他们不信任 Mythos，为什么他们信任 Mythos 来监控自己？这就是为什么即使这个模型智能低得多，至少它是一个独立的判断者，它以非常不同的方式构建，可能更有可能标记问题，更不可能密谋支持自己。

Yeah. Exactly. Exactly. And it's probably fine especially for the first job that we have on our program, which is this non-agentic predictor that can be used as a guardrail. Now as I said, the guardrail isn't the full answer, but if we deploy that and companies add it to their monitors, then it will mitigate the risks to some extent. So it will allow more time to develop the more ambitious version. It is fully agentic and that's what we need right now. It's time. As far as I can tell, I think Anthropic, I'm talking a lot about Anthropic because I've been reading the Mythos system card and all announcement last week. As far as I can tell, they've decided to have Mythos monitoring Mythos basically. They've tried other models doing it, but they're like Mythos is smarter, it's better. But obviously this creates an eternal contradiction that if they don't trust Mythos, why do they trust Mythos to monitor itself? That's one reason why even if this model is much less intelligent, at least it's an independent judge, it's built a very different way that might be more likely to flag things and less likely to scheme to support itself basically.

是的。我完全同意。我甚至会更进一步。

Yeah. I completely agree. And I would go even further.

所以这不仅仅是监控器可能具有欺骗性，在情况不妙时说「没问题」，因为那与某种隐藏的自我保存或追求权力的目标一致。而且，如果我们再往下看，公司计划用 AI 进行 AI 研究。在这种情况下，拥有秘密欺骗性的 AI 是一个更危险的赌注。如果我们把所有信任都放在一个系统上，它将设计我们无法完全理解或阅读的代码和算法，那种 AI 可能会在未来更强大的 AI 系统中植入我们看不见的后门。这样我们就会走向对我们更危险的方向。所以我认为这样做非常非常危险。这就是为什么在政策方面，AI 用于 AI 研究应该被高度重视。这也是为什么，如果我们要用 AI 进行 AI 研究，我们真的真的希望确保 AI 是诚实的。

So it's not just the issue that the monitor could be deceptive and say yes it's okay when it's not because somehow that's aligned with some hidden goals of self-preservation or power seeking. But it's also, if we go a little bit down the line, what companies are planning to do with using AI for AI research. So this is a place where having AI that is secretly deceptive is even more of a dangerous bet. If we're going to put all of our trust into a system that will design the code and the algorithms that will be too complicated for us to understand or to read fully, that kind of AI could put back doors that we don't see into an AI system in the future that is even more powerful. And so we could get into this direction that gets even more dangerous for us. So it would be, I think, very very dangerous to do that. And that is why in terms of policy, the attention given to AI for AI research is something that should be very high on the agenda. And this is also why, if we're going to be doing AI research with AI, we really really want to make sure that AI is going to be an honest one.

我认为大多数投身于技术性 AI 安全的人，基本上决定要改善我们在猫鼠游戏中的胜算。而对于那些非常担忧现状的人来说，他们的推理是这样的：我们现在有 50%的概率遭遇绝对灾难，因为我们正在做一大堆极其疯狂鲁莽的事情。也许只是修补我们做的最愚蠢的事情，比如扑灭最严重的火灾，我们可以把风险降到 10%。显然，这是一个荒谬的灾难风险。我们作为一个物种，不能做得更好，这令人尴尬。尽管如此，这使事情顺利或至少合理的几率提高了 40 个百分点。从 10%降到 0%只有那个价值的四分之一。即使你可以通过更好的替代方法获得更大的安全保障，在对数域中，那也是无限好的。

I think the majority of people who are piling into technical AI safety, they've decided to go for improving our chances at the cat and mouse game basically. And I think for the people who are very concerned about what's going on, their reasoning is something like: we're running a 50% chance of absolute disaster now because we're doing a whole bunch of absolutely crazy reckless stuff. Maybe just patching the very dumbest stuff that we're doing, like fixing the worst fires that are the worst things that are on fire right now, we can bring that risk down to 10%. Obviously, that's a preposterous risk of disaster to run. It's an embarrassment to us as a species that we can't do better than that. Nevertheless, it's 40 percentage points improvement in our chances of things going well, or at least reasonably, going from 10% down to 0% is only a quarter as valuable as that. And even if you can get massively greater guarantees of safety using much better alternative approaches, in the logarithm domain, it's infinitely better.

当然，当然，当然。但在期望值域中。

Sure. Sure. Sure. But in the expected value domain.

我认为这就是两种心态的区别。

And I think that that's kind of the difference in the two mentalities here.

是的，不。正如我所说，我们应该尝试所有这些方法。它们并不相互排斥。把所有鸡蛋放在猫鼠游戏这个篮子里是错误的，而只需花费公司目前支出的一小部分，我们就可以开发出安全且有能力的 AI 版本。顺便说一句，我想在这里补充一点关于能力的内容。所以，我也相信科学家 AI 甚至可能比当前的方法更有能力，这与一些设计特征有关。它被训练成以结构化的方式明确推理它被要求预测的陈述。这与当前的思维链不同，当前的思维链可以产生某种我们相信的东西，并且倾向于通过训练中的测试，但并不强制要求拥有像数学定理证明那样可分解的论证。还有其他方法遵循这个方向。当然，很多关于安全设计 AI 的工作，以及例如辩论工作，都在试图强制 AI 思考方式的一致性。所以我相信，除了我们提出的训练目标所带来的认知谦逊之外，系统通过调用形成推理链的结构化潜在变量来产生这些概率的方式，实际上可能为公司提供能力上的优势。

Yeah. No. As I said, we should try all of those things. They're not mutually exclusive. It would be a mistake to put all our eggs into the cat and mouse game when at a cost that is a fraction of what companies are currently spending, we could be developing a safe version of AI that will be capable. And by the way, I want to add something here about capability. So, I also believe that the scientist AI could even be more capable than the current approach and it has to do with a number of design features. So it is trained to explicitly reason in a structured way about the statements that it is asked to make a prediction over. This is different from the current chain of thought where it could produce some kind of that we believe and tends to pass the tests that we have during training, but is not constrained to actually have arguments that can be decomposed in the same way that a proof of a mathematical theorem is decomposed. And there are other approaches that follow that direction. So of course a lot of the work on trying to do safe by design AI but also the debate work, for example, is trying to enforce some kind of coherence in how the AI is thinking. So I believe that in addition to the kind of epistemic humility that comes with the training objective that we are proposing, the way that the system is producing those probabilities by invoking structured latent that form a chain of reasoning is something that could actually provide even a capability advantage to the companies.

你认为当前模型内部是否表征了真理？我想你说这个模型的一个优势是它专注于将基本事实作为潜在变量来表征。我猜当前的 LLM 也这样做，因为对实际正确的东西有某种感知非常有用，然后它们会扭曲它。它们基本上从那个开始，然后为了达成包括操纵人、说谎等目标而扭曲它。我猜有些人怀疑这一点。有些人怀疑是否有任何联系，或者它们是否真的在试图建模真理。你有看法吗？

Do you think current models internally represent truth? Like I guess you're saying one advantage of this model is that it's focused on representing ground truth as a latent variable. I would my guess is that current LLM do that as well because that is very useful to have some sense of what's actually correct and then they distort it. They basically start with that and then they distort it in order to accomplish the goals including manipulating people or lying or whatever else. I guess some people doubt that. Some people doubt whether there is any connection or that they are actually trying to model truth. Do you have a view?

是的，我完全同意你。我对世界如何运作有一个假设，基本上认为推理世界的实际属性，即真理，即使你不确定所以必须使用概率，也能让你在做出更好的预测和行动方面拥有非常强的优势。这实际上是论证的一部分，说明为什么科学家 AI 的训练过程会产生倾向于实际信念的潜在变量。这非常有用，因为现在我们可以查询那些潜在变量，并获得关于 AI 实际相信什么的答案，因为这就是它构建内部推理并产生答案的方式，而不是思维链中出现的某种潜在。所以它并没有完全解决 ELK 挑战，即引出潜在知识的挑战，因为我们得到的唯一保证是关于这些自然语言陈述，它们可以是我们可以查询的潜在变量。

Yeah, I completely agree with you. I have an assumption about how the world works that basically states that reasoning about the actual properties of the world, in other words the truth, even when you're uncertain so you have to use probability, gives you a very strong edge in making better predictions and better actions. And that is actually part of the arguments as to why the training procedure for the scientist AI will create latent variables that are preferentially going towards the actual beliefs. And that is very useful because now we can query those latent variables and get answers about what the AI actually believes, because that's how it constructs its internal reasoning and produces an answer, not some potential that comes in the chain of thought. So it doesn't completely solve the ELK challenge, the challenge of eliciting latent knowledge, because the only guarantees we get are about these natural language statements that can be latent variables that we can query.

你能解释一下 ELK 问题吗？

Can you explain the ELK problem?

是的，抱歉。ELK 问题源于你提出的问题：即使 AI 可能在内部知道某事的真相，或者至少对某事有一些内部信念，但由于它被训练成模仿它在数据中看到的变量类型，这些数据主要是人们所说的，当你查询它时，它会以相同的语义回答，并且这是它根据当前上下文所采取的角色会回答的，而不一定是它实际相信的。这里的技术问题是我们没有监督标签来教 AI 它应该实际相信什么，所以我们不能问它的真实信念。我们只能得到它在训练数据中看到的变量分布的再现。所以在科学家 AI 中，这通过通信行为与更事实性的语法之间的清晰句法分离来解决，这种事实性语法可用于潜在变量和我们知道的真实事物。所以我们可以使用那种事实性语法来查询它。

Yeah, sorry. The ELK problem comes from the issue you were raising that even though the AI may internally know the truth of something or at least have some internal beliefs about something, because it's trained to imitate the kind of variables that it sees in data, which mostly are what people are saying, when you query it, it's going to answer in the same semantics and that is what a persona that it currently is taking given its context would answer, and not necessarily what it actually believes. And the technical problem here is we don't have supervised labels to teach the AI about what it should actually believe, so we can't ask it about its true beliefs. We only get a kind of reproduction of the distribution of variables that it sees in its training data. So in the scientist AI, this is addressed by having this clear syntactic separation between the communication acts and the more factual syntax that can be used for latent variables and true things that we know. So we can query it using that factual syntax.

另一个我们能够规避 ELK 挑战中某些问题的原因是，同一种语言（比如英语）既可以用来表示那些潜在变量，也可以用来表示观察到的陈述。所以基本上，我们依靠语言的组合结构来泛化到从未见过的新句子。但这些句子的意义是由它对语言的理解所赋予的。这与那些研究 ELK 挑战的人所考虑的情景非常不同，在那种情景下，我们假设潜在变量是匿名的，没有预定义的意义，因此我们不知道在神经网络内部该往哪里看才能找到信念，这推动了诸如机械可解释性之类的研究。但在科学家 AI 中，我们在一定程度上绕过了这个问题，因为潜在变量是用自然语言表达的，因此是可解释的。当然，神经网络中可能仍然存在其他非自然语言的隐藏信念，但至少当我们用自然语言提问时，我们会得到诚实的回答。

And then the other reason why we're getting away with some of the issues with the ELK challenge is that the same language, like English, can be used to represent those latent variables as well as the observed statements. So basically, we rely on the compositional structure of language to generalize to new sentences that it has never seen. But the meaning of those sentences is given by its understanding of language. This is very different from the scenario studied by those who looked into the ELK challenge, where we assume that the latent variables are anonymous, like they don't have a predefined meaning, and so we don't know where to look inside the neural net if we want to find out what the beliefs are, which motivates things like mechanistic interpretability and so on. But in the scientist AI, we bypass this problem to some extent because the latent variables are in natural language and thus are interpretable. Now, there could still be other beliefs that are not in natural language that are hidden in the neural net, but at least when we ask questions in natural language, we're going to get an honest answer.

好的。据我所知，这里有三种主要方法。一种是把这个模型用作监控器、护栏。另一种是直接从头训练，让这成为完整的方法。还有一种是我们拿现有的模型，试着让它们更诚实，更像科学家 AI。你想谈谈这种方法有没有什么好的前景吗？

Okay. So, as far as I can tell, there's three big approaches here. One is we're going to use this model as a monitor, as a guardrail. Another would be we're going to just train it from scratch and make this be the whole approach. Another would be we could take the current models and try to make them more honest, make them more like a scientist AI. Do you want to talk at all about whether that approach has any good prospects?

嗯，我目前拥有的数学方法要求我们从头开始训练才能获得保证，这很昂贵。所以如果我们只是对现有模型进行科学家 AI 微调，就会失去这些保证。但即使没有数学保证，它可能仍然是一种可行的方法。所以我认为值得一试。换句话说，我们可以拿一个非常能干、顶尖的模型，然后用科学家 AI 的目标和我讨论过的转换后的数据继续训练。我们希望从经验上证明，随着你进行越来越多的微调，诚实行为（即没有欺骗行为）的度量会改善，而且我们不会损失能力。所以这不像数学证明，而是经验性的。一旦这一点得到证实，可能就足以说服人们。然后我们再通过从头训练来获得完整的保证，这现在要花费训练一个完整模型的成本。

Well, the math that I currently have would require that we actually start the training from scratch to get the guarantees, which is expensive. And so we would lose the guarantees if we just do, say, scientist AI fine-tuning on existing models. But even if you don't have a mathematical guarantee, it might still be a workable approach. So I think it's worth doing. In other words, we can take a really competent, top-notch model and then continue training using the objective of the scientist AI and the data that has been transformed as I discussed. We hope to show empirically that as you do more and more fine-tuning, the measurements of honest behavior, lack of deceptive behavior, will improve and that we won't lose capability. So that wouldn't be like a mathematical proof. That would be an empirical thing. And once this is established, it might be sufficient to convince people. Then let's get the full guarantees by training from scratch, which is now going to cost the cost of training a full model.

那么你采取的方法是不是拿一个当前的前沿模型，然后做强化学习，让它像科学家 AI 那样说话？

So is the approach that you would take there is take a current frontier model and then do reinforcement learning to get it to speak as if it were a scientist AI?

不。好的，首先让我谈谈强化学习。三年前，我和一群强化学习研究人员开会，我有一张幻灯片上只有这些词：'强化学习是邪恶的。'但你怎么看？所以，这不是什么新鲜事。AI 安全领域的人一直在讨论用强化学习训练以实现世界目标的基本缺陷。它会导致工具性目标和奖励黑客问题。在这两种情况下，你最终得到的系统会拥有你没有选择的目标，并且可能与你选择的目标相悖。所以强化学习是构建超级智能非常危险的东西。好消息是你不需要做强化学习。我们在科学家 AI 中展示的是，有一种方法可以训练 AI，使其对其行为或预测的后果漠不关心。好的，我们先从预测模型开始，这更容易理解，对吧？想象你有一个非常好的气候模型。气候模型，如果你运行它的模拟，或者训练一个神经网络来近似这些模拟，它会给你诚实的答案，并且它不在乎你是否用答案做蠢事。所以这就是你获得诚实答案的方式，本质上是通过建立对世界的理解，一种解释性的理解，完全不在乎预测将如何被使用。一旦你有了这个，你就可以以某种智能体式的方式使用它。例如，护栏就是一种智能体式的东西，对吧？它做出二元决策：我接受这个预测吗？我是否把这个预测放到现实世界中？这是一个决策，是一个智能体式的选择。但在这种情况下，这个选择有一个唯一的目标，就是避免危险行为。所以一旦我们安装了护栏，我们就已经进入了智能体式的世界。所以总结一下我的回答，有一种方法可以训练一个预测器，它不需要强化学习，也就是说不需要针对世界上的未来事件（包括未来的良好预测误差）进行优化。这里我想对 AI 安全中关于 AI 预言机的先前工作做一个补充。当然，人们已经考虑过这个问题：为什么我们不直接训练一个能很好预测的预言机呢？但他们认为训练它的唯一方法是通过强化学习来做出好的预测。但这有一个巨大的缺陷，对吧？因为如果我是理性的，我想在未来永远最大化好的预测，我可以在短期内撒谎，让人类做那些能帮助我在未来做出好预测的事情，比如获得更多算力，这样我就能训练一个更好的自己，或者让世界变得更简单以预测。杀死所有人，对吧？如果人类互相残杀，那么世界就更容易预测了。所以这些都是非常糟糕的结果，源于做出好预测的工具性目标，而这正是由于强化学习的目标：你训练 AI 在现实世界中实现某些东西，这就带来了严重的问题。但另一种方法，科学家 AI 的方法，是从一开始就训练它不实现世界上的任何东西，而只是预测训练数据，即过去的数据。所以它不是关于未来，而是关于过去，目的是对过去的数据做出好的解释和好的预测。

No. Okay. So, first let me talk about reinforcement learning. Three years ago, I was in a meeting with a bunch of reinforcement learning researchers and I had a slide with only these words: 'Reinforcement learning is evil.' But what do you really think? So, this is not something new. People in AI safety have been talking about the fundamental flaw in training by reinforcement learning to achieve something in the world. It gives rise to the problems of instrumental goals and reward hacking. In both of these cases, what you end up with is systems that have goals that you didn't choose and could go against the goals that you did choose. So reinforcement learning is a very dangerous thing to build superintelligence. The good news is you don't need to do reinforcement learning. What we show with the scientist AI is that there's a way to train the AI so that it will be indifferent to the consequences of its actions or its predictions. Okay, let's start with a predictive model. It's easier to understand, right? So imagine you do have a really good climate model. The climate model, if you run a simulation of it or train a neural net to approximate those simulations, will give you honest answers and it doesn't care if you do something stupid with the answer. So that's how you get honest answers, essentially by building an understanding, an explanatory understanding of the world that is completely indifferent to how the predictions are going to be used. Now once you have this, you can use it in a kind of agentic way. For example, the guardrail is a kind of agentic thing, right? It's taking a binary decision: Do I accept this prediction? Do I put out this prediction in the real world or not? And it is a decision. It is an agentic choice. But in this case, it's a choice that has a unique goal, which is to avoid dangerous actions. So we are already entering the agentic world once we install the guardrail. So bottom line, to summarize my answer, there's a way to train a predictor that will not require reinforcement learning in the sense that it will not require optimizing with respect to future events in the world, including future good prediction errors. And here I want to make a parenthesis about previous work in AI safety on AI oracles. One of the, of course people have thought about this: why don't we just train an oracle that's a good predictor? But they thought that the only way to train it would be by reinforcement learning to make good predictions. But there's a huge flaw here, right? Because if I am rational and I want to maximize good predictions forever in the future, I could lie in the short term to make humans do things that will help me to make good predictions in the future, like get more compute so I can train a better version of myself, or make the world simpler to predict. Kill everyone, right? If humans kill each other, then the world will be much easier to predict. So these are really bad outcomes that are due to instrumental goals of making good predictions, and it arises because of the reinforcement learning kind of objective: you're training the AI to achieve something in the real world, and that's where you get bad problems. But the other approach, the approach of the scientist AI, is to train it from the get-go not to achieve anything in the world, but to just predict the training data, the past data. So it's not about the future, it's about the past, to come up with good explanations and good predictions of the past data.

好的，所以我问的原因是，如果我们从当前最先进的智能体式模型出发，试图让它像科学家 AI 那样更诚实。

Okay, so yeah, the reason I was asking is if we're going to go from a current state-of-the-art agentic model and try to make it like a scientist AI to make it more honest.

如果不使用强化学习，我们该怎么做？你是说我们要让它仅基于之前的数据来预测过去的事件？

How do we do that if not reinforcement learning? Are you saying we're going to do something more like we get it to predict past events based only on having data from before that time?

是的。没错。顺便说一句，科学就是这样运作的。科学理论的作用是解释过去的数据，当然有时它们也会对未来数据做出预测，我们可以验证这些预测，但根本上，我们判断一个好理论的标准是它对我们已有的数据做出了良好的预测。

Yes. Yes. And that's how science works by the way. So what scientific theories do is explain the past data and of course sometimes they make predictions about future data which we can check, but fundamentally the way that we judge a good theory is that it is making good predictions about the data we have.

这在某种意义上是否要求对模型屏蔽结果？

Does this require blinding the model to the results basically in some sense?

在科学家 AI 中，我们有一个条件，类似于对定理的要求，我们称之为后果不变性。它的意思是，你只能使用你对过去数据的拟合程度来训练你的因果模型。你不能根据这些预测可能导致的未来结果来选择这些预测。

So we have this condition in the scientist AI like requirements for the theorem that we call consequence invariance. What it means is you're only allowed to use how well you're fitting the past data in order to train your causal model. You're not allowed to choose those predictions with respect to what could happen in the future as a consequence of those predictions.

我想我对预测器模型有了一个不错的理解，它接收陈述并输出它们为真的概率。还有没有更多有用的信息，能让我和其他人脑海中形成这个完整系统如何运作的画面——它不仅包含预测器，还在其周围构建了脚手架，赋予它部分智能体能力等等？

I think I have a decent picture of the predictor model that's taking in statements and throwing out probabilities of them being true. Is there more that it would be useful for me and other people to have in their minds to picture how this entire system would work where it's got not only the predictor but you're building scaffolding around it to give it like partial agency and so on?

是的。首先，你必须理解，同一个以统一方式训练的预测器可以同时用于回答用户问题和安全问题。安全问题就是你关心的护栏问题，对吧？所以并不是有一个单独的神经网络做护栏，另一个做预测。护栏使用的是同一个预测神经网络。只是你问的问题类型不同。你问的是：如果我给出这个预测，或者对于智能体系统，如果 AI 采取某个特定行动，某种危害的概率是多少？所以训练完全是针对预测器的。一旦训练完成，你可以做很多事情来构建包含脚手架的系统。那么脚手架是做什么的呢？例如，当用户提出一个问题时，它会构建问题，将其转化为预测器可以输出概率的形式，但它也会用另一个问题调用预测器，即危害的概率。这就是护栏问题。然后它会查看答案，以决定是否给出回答。例如，如果问题是关于如何制造炸弹，那么护栏会说这个危险的概率足够高，于是护栏会使用一个阈值来拒绝这些问题。这个阈值是一个规范性选择。它是由社会决定的，比如我们愿意承担多大的风险，取决于我们谈论的是哪种危害。护栏还有其他作用，用于处理所谓的表演性预测。有时一个问题可能有多个答案，因为答案会影响未来。一个经典的例子是，如果问题是关于谁将赢得下一次选举，而 AI 被认为非常强大，人们会相信它说的任何话并投票给那个候选人，这意味着 AI 可以说这个人或那个人，两者都会成真，对吧？那么它就开始通过其预测获得某种我们似乎无法控制的智能体能力。护栏将管理这一点，以解耦预测与这些预测的影响。更具体地说，神经网络预测器的训练方式是，在其输入条件中，总有一个特定的陈述问：如果我们确实产生了这个预测，危害效果会是什么？所以当你以产生特定答案的干预为条件时，现在只有一个答案。你在说：「我要发布这个预测，效果会是什么？」关于这一点还有很多可说的，但底线是你可以控制这种风险以及可能由此产生的智能体能力。而护栏的工作就是完成这个任务。

Yeah. So first you have to understand that the same predictor that is trained in a unified way can be used to both answer user questions and answer safety questions. And the safety questions are those that you care about for the guardrail, right? So it's not like there's a separate neural net that does the guardrail and another one that makes predictions. The guardrail is using the same prediction neural net. It's just a different kind of question you're asking. You're asking what's the probability of a particular kind of harm given that I put out this prediction, or in the case of the agent system that the AI puts out a particular action. So training is fully the predictor. Now once it's trained, there are a number of things you can do to construct the system that includes the scaffold. So what is a scaffold doing? Well, for example, when a user comes with a question, it will construct the question, put it in the form for the predictor to produce a probability, but it will also call the predictor with a different question which is the probability of harm. So that's the guardrail question. And then it will look at the answer in order to decide whether to produce an answer or not. For example, if the question is about how do I build a bomb, then the guardrail will say that the probability that this is dangerous is high enough, and so the guardrail will use a threshold to reject those questions. That threshold is a normative choice. It is something that society decides, like how much risk are we willing to take, depending on the kind of harm that we're talking about. The guardrail also has other roles to handle what is called performative prediction. So sometimes a question can have multiple answers because the answer will influence the future. A classical example is if the question is about who's going to win the next election and maybe the AI is considered very capable and people will believe whatever it says and vote for that candidate, which means the AI could say this guy or that guy, both would be true, right? So then it's starting to have agency through its prediction in a way that seems that we don't control. And the guardrail is going to manage that to decouple the predictions from the effect of those predictions. To be more concrete, the neural net predictor is trained so that in its input conditions, there's always a particular statement that asks what if we did produce this prediction, what would be the harm effect, right? So when you condition on the intervention of producing a particular answer, now there's only one answer. You are saying, "I'm going to put out this prediction and what will be the effect?" So there's more to say about this, but the bottom line is you can control this kind of risk and the agency that could come from it. And it's the job of the guardrail to do that job.

长期以来有一个担忧，即预言机 AI 在结构上处于劣势，在其他条件相同的情况下，它们会不那么智能，因为它们基本上没有选择进行实验、采取行动来最有效地发现事物如何运作。我认为还有其他类似的担忧，基本上，使 AI 智能的东西也使它们危险，反之亦然。你认为这种说法成立的可能性有多大？

There's a long-standing worry that Oracle AIs are structurally disadvantaged, that they're going to be less intelligent all else equal because they don't have the option of basically running experiments, of taking actions in order to discover how things work most effectively. And I think there's other worries along these lines that basically it's the things that make AI intelligent that make them dangerous and vice versa. What do you think are the chances that that is true?

所以我认为要清楚回答你的问题，我们需要区分两个问题。一个是，在智能体的情况下，给定可用的信息（如数据集和上下文），什么是最佳预测或最佳行动？第二个问题是，如果我要在现实世界中做实验以获取更多知识，哪些正确的行动会增加我对世界的理解并减少我对世界的不确定性？顺便说一句，这正是一般科学家——不是 AI 科学家，而是从事生物学、化学或物理学的人——思考的方式，对吧？他们问自己：如果我做这个实验，它是否有助于我区分这两种理论？你可以用数学方法量化这一点，这被称为信息增益。事实证明，一旦你有了一个好的预测器，比如一个好的概率预测器，你也可以将其转化为一个好的估计器，用来估计如果你做这个或那个实验会获得多少信息。现在，你可以在科学家 AI 之上构建一个智能体系统，例如，它会告诉你做哪个实验以获得良好的信息增益。但当然，你也可以使用护栏。所以你希望实验既能帮助区分解释和理论，同时又不伤害人。但在科学家 AI 中这很容易做到，对吧？我们有护栏这个概念。所以用户目标是获取信息。安全目标是不伤害人。当然这只是一个简化的描述。所以你可以两者兼得。但当然，你现在进入了智能体系统的领域。

So I think we have to distinguish two problems to have a clear idea about your question. One is what are the best predictions or the best actions in the case of an agent given the available information like the data set and context that is available. And then the second question is if I were to do experiments in the world in order to acquire more knowledge, what are the right actions that will increase my understanding of the world and reduce my uncertainty about the world? And by the way, this is how scientists in general, not AI scientists, people that are doing biology or chemistry or physics, this is how they think, right? They ask themselves if I were to do this experiment, would it help me to disambiguate between these two theories? And you can quantify this mathematically with something that's called information gain. And it turns out that once you have a good predictor like a good probabilistic predictor, you can also turn this into a good estimator of how much information you would gain if you were to do this experiment or that experiment. And now you could build an agentic system on top of the scientist AI for example that would tell you which experiment to do in order to obtain good information gain. But of course you could also use a guardrail. So you would like experiments that help to disambiguate between explanations and theories at the same time as not harming people. But that's easy to do in the scientist AI, right? We have this guardrail notion. So the user goal here is acquire information. The safety goal is don't harm people. I'm just of course this cartoon. And so you could get both. But of course, you now enter into the realm of agentic system.

是的，科学家 AI 的整个计划包括我们如何在非智能体的可信预测器之上开发智能体系统。我想如果我们回溯一两年，我们拥有在某种意义上极其博学、极其聪明的 AI 模型。但如果你试图让它们浏览网页，它们会很难做到。这似乎表明，科学智能或预测事物的能力与实际导航世界的能力之间可能存在巨大差异。为了让它们能够采取有用的行动，需要大量的额外训练和努力。你是否担心，你设想的那种数据训练的科学家 AI 会在实际层面显得无能，因为它没有经历过那种工作——除非我们做了大量实验，比如「你点击这个网页上的按钮吗」？它实际上不会学会做人们希望模型能做的事情。

And yeah, the whole plan of the scientist AI includes how we can develop agentic systems on top of a non-agentic trustworthy predictor. I guess if we wind back a year or two ago, we had AI models that were in a sense extremely knowledgeable, extremely smart. But if you just tried to get them to navigate a web page, they would struggle to do it. It seems like there's a very big difference potentially between scientific intelligence or ability to predict things versus ability to navigate the world in practical terms. And it took a lot of extra training, a lot of extra effort in order to get them to be able to take useful actions. Do you worry that the scientist AI that you might train using the kind of data that you're imagining would be kind of incompetent at a practical level because it wouldn't have had that kind of work unless you did a lot of this work where the experiments we're running were like 'do you click this button on a particular web page'? It wouldn't actually learn to do the things that people want the models to be able to do.

所以你可以绝对地用轨迹来训练它，这些轨迹记录了当某个智能体做了某事时发生了什么，观察到了什么，后果是什么。这就是它如何学习好的条件概率，知道为了达成特定目标（包括安全目标）应该采取什么行动。所以这将是一种不同于强化学习的训练，但它会使用相同的资源。无论收集到了什么经验，它不必是强化学习中所说的在策略的。所以它可以使用任何智能体的经验或任何观察到的东西，不仅仅是智能体做事，还包括观察世界中的事物。所有这些对它来说都是数据，帮助它理解世界并构建后果。所以其中一个后果是：如果我做这个动作会发生什么，以及什么动作能最大化实现某个目标的概率？在某种程度上，这更接近基于模型的强化学习，你可以利用全部经验来制定策略，而不是始终完全交互。

So you could absolutely train it with trajectories of what happens when a particular agent did this, what was observed, what the consequences are. This is how it would learn good conditional probabilities of what actions to do in order to achieve particular goals, including the safety goal. So it would be a different kind of training than the reinforcement learning training, but it would be using the same resource. So whatever experience has been collected, by the way it doesn't have to be what people in RL call on-policy. So it could use the experience of any agent or anything that is observed, not just agents doing things but just observing things in the world. All of that is data as far as it's concerned that helps it both understand the world and construct the consequences. And so one of the consequences: what would happen if I do this action, and what action will maximize the probability of achieving some goal? In a way it's closer to model-based reinforcement learning where you are able to use your whole experience to come up with a policy, as opposed to something that is fully interactive all the time.

所以在目前的科学家 AI 中，如果你使用它并产生新的后果和新的观察，你需要用新数据重新训练或微调它。但我们可以借助公司和学术界正在研究的所谓持续学习的研究。当新信息到来时会发生什么？当然你可以把它放在上下文中，比如输入窗口，科学家 AI 也可以这样做，但最终你希望它被整合到系统的权重中，这就是持续学习试图做到的。但好消息是，科学家 AI 面临着与当前 AI 相同的问题，正在探索的解决方案也将适用于科学家 AI。

So in the scientist AI currently you would need to retrain it or fine-tune it with the new data once if you use it and it produces new consequences and new observations. But we can ride on the same research that companies and academia is working on on what's called continual learning. So what happens when there's new information coming? And of course you could put it in the context like in the input window, and the scientist AI you could do the same thing, but at some point you'd like it to be integrated into the weights of the system, and that's what continual learning is trying to do. But the good news with the scientist AI is it's facing the same problems that current AIs are facing, and the solutions that are being explored will be applicable to the scientist AI as well.

是的，我觉得对这个想法的许多批评，包括我的问题，都反映出人们（包括我）心中有一个 AI 的愿景，它在训练方式、使用的数据、结构和能力上都非常不同，而你想说实际上我们可以让它非常相似。我们可以把几乎所有用于训练当前大语言模型的数据拿来，稍微重新格式化一下，然后再次使用。我们将利用所有不同的效率、所有算法改进。我们只是给它一组稍微不同的输入和输出，但在其他几乎所有方面都差不多。

Yeah, I feel like a lot of the critiques of this idea and my question somewhat reflect that is that people including me I guess have had in mind a vision of an AI that's extremely different in how it's trained and maybe the data that's being used and the structure and the affordances that it has, and you want to say actually we can make it remarkably similar. We can take almost all of the data that we're using to train current LLMs and just reformat it a bit and then use it again. We'll use all of the different efficiencies, all the algorithmic improvements. We're just going to do it like we're just going to give it a somewhat different set of inputs and outputs, but it's more or less the same in almost every other respect.

是的。是的。这就是为什么它如此实用。是的。这就是为什么我认为我们可以很快做到，这更像是一个拥有正确训练资源的问题。因为训练目标不同。是的，我们确实需要尝试一下，看看效果如何，但基本上它与最大似然训练没有太大区别，而最大似然训练正是我们在预训练中使用的。所以，在某种程度上，它更接近预训练，而且我们知道预训练效果很好，顺便说一句。它实际上比强化学习效果更好，强化学习更难。所以这里的训练形式更接近我们在预训练中所做的，只不过我们教会 AI 人们说的话与其实际相信之间的区别，并迫使它推理人们为什么说那些话，而不是模仿人们会做什么。

Yes. Yes. And that is why it's so practical. Yeah. That is why I think we can do it pretty quickly and it's more like a matter of having the right resources for training. And because the training objective is different. Yeah, we do need to try it out and see how it works, but fundamentally it isn't so different from maximum likelihood training, which is what we use in pre-training. So, in a way, it's closer to the pre-training, and we know that works really well, by the way. It's actually working better than RL, which is harder. So the form of training here is much closer to what we do in pre-training, except that we teach the AI the difference between what people say and what it actually believes, and we force it to reason about why people said those things rather than imitate what people would do.

好的，我上次在网上看到，你领导的机构 Law Zero 已经筹集了大约 1 亿美元。我想大多数非营利组织年底能筹集到 1 亿美元左右会很高兴。但我想你面对的是拥有 1000 亿美元的组织。

Okay, the last I saw online, the organization that you're leading, Law Zero, has raised something like $100 million. And I think like most nonprofits year end would be pretty happy to have raised $100 million or so. But I guess you're up against organizations that have $100 billion.

实际上，甚至更少，但取决于你怎么算。我们从慈善机构筹集了大约 3500 万美元。但我们正在与各国政府谈判以获得更多资金。所以我们相当有信心很快就能达到数亿美元。但正如你所说，与领先的 AI 公司相比，这仍然是九牛一毛。然而，我认为这足以做出一个概念验证。有了概念验证，我们就能说服公司投入资金训练更大的系统，或者从头开始使用相同原理训练的系统。

Actually, it's even less than that, but more depending on how you count. So we've raised about 35 million US from philanthropy. But we are in negotiation with various governments to get much more. So we are pretty confident that we're going to be in the hundreds of millions pretty soon. But as you say that's still peanuts compared to what the leading AI companies have. However I think it is sufficient to make a proof of concept. And with a proof of concept then we can convince companies to actually put the money to train larger systems or systems that are trained from scratch using the same principle.

这就是变革理论。好的。你想运行什么样的实验，需要多少钱？

So that's the theory of change. Okay. What sort of experiments do you want to run and how much money would you need for them?

所以有各种实验。底线是我们希望展示诚实度的提升，并基本消除欺骗行为。我们可以通过两类实验来实现。我们可以训练非常小的模型，就像学术机构一直在训练的那种，比如不到 100 亿参数从头开始，但只使用我提到的科学家 AI 目标和数据表示。所以这不会具有竞争力，因为模型会小得多，但我们可以与相同大小、相同数据训练的开源权重模型进行直接比较。我们可以在能力和安全性方面进行比较，诚实是我们主要关注的。所以这是一种演示。

So there are various kinds of experiments. The bottom line is we want to show the improvement in honesty and basically getting rid of deceptive behavior. And we can do it with two categories of experiments. We can train really small models of the kind that academic organizations have been training, like less than 10 billion weights or something from scratch, but just using the scientist AI objective and the data representation that I mentioned. So that won't be competitive because it's going to be much smaller models, but we can compare head-to-head the original open weight model that has that same size and trained on the same data. We can compare both in terms of capability and safety, essentially honesty being the main thing we're looking for. So that's one kind of demonstration.

另一种更接近可部署但保证较少的演示方式是，拿一个现有的预训练模型——可能从基础模型开始，而不是经过强化学习的模型——然后用科学家 AI 的目标和数据进行微调。这样会得到一个更有能力的模型，因为它更大。微调比从头训练便宜得多。但我们失去了数学上的保证。我认为可能也没问题。当然，这取决于你愿意做多少微调。这类实验的有趣之处在于，我们应该能看到这种权衡。比如，如果你在欺骗性基准上测量，随着你不断进行更多微调，应该会看到一条上升的曲线，对吧？这正是我们希望看到的。同时你还要证明能力不会下降。顺便说一句，这很棘手，因为我们在实验中已经发现，至少大多数开放权重的模型会在基准上作弊。我是什么意思呢？一旦你对任何东西做一点微调，它们在基准上的表现就会下降。

The other kind of demonstration, which is maybe closer to being deployable but has fewer guarantees, is to take an existing pre-trained model, maybe starting from a base model rather than one with RL, and then fine-tune it using the scientist AI objective and data representation. That would give a much more competent model because it's bigger. Fine-tuning is much cheaper than training from scratch. But we lose the mathematical guarantee. I think it's probably going to be fine anyway. Of course it depends how much fine-tuning you're willing to do. What's interesting in these kinds of experiments is that we should be able to see the trade-off. If you measure, say, on deception benchmarks, what happens as you continue training with more and more fine-tuning, we should see a curve that gets better, right? That's what we're hoping to see. And you also want to show that capability doesn't go down. Which by the way is tricky because unfortunately what we found already in our experiments is that most of the open-weight models, at least, cheat on the benchmarks. What do I mean by this? As soon as you do a little bit of fine-tuning on anything, their performance on the benchmarks goes down.

看到了吧？好的。

See? Okay.

所以他们真的是在教模型应付测试。他们可能对基准过拟合了。所以我们需要找到解决办法，但我相信这可以做到。所以我们会有这两类证据，我希望，这足以说服人们投入不是几亿而是几十亿的资金，从头训练一个全尺寸模型。

So they've really taught them to the test. They probably have overfitted the benchmarks. So we need to find a way around that, but I'm confident this can be done. So we will have these two kinds of evidence, I hope, and that may be sufficient to convince people to put not just hundreds of millions but the billions that would be necessary to do a full-scale model from scratch.

那么，如果进行同类比较，你设想的这种模型与使用相同数据量和算力的标准模型相比，在预测能力和智能体能力方面，我们是否认为科学家 AI 会更诚实、更安全？我们预期它会更好还是更差，以及好或差多少？

So comparing like for like, if you trained a model of the type that you're envisaging versus a standard model using the same amount of data, same amount of compute, what are the considerations as to whether we think the scientist AI would be more honest and safer in terms of capability both in terms of prediction and agency? Would we expect it to be better or worse and how much better or worse?

在能力方面，我预期它会更好，因为推理能力更强。我还没提到的一个方面是，有很好的科学证据表明，当模型利用世界的因果结构时，它能在分布外更好地泛化。这是我以及机器学习社区中许多人一直在研究的问题。它涉及一个非常有趣的概念：世界在变化，但有些东西是不变的，比如底层的因果机制、世界运作的方式，比如物理定律。它们不会改变。所以即使数据的分布可能因为世界上发生的事情而改变，表面看起来不同，但底层对事物如何存在的科学解释是相同的。如果你能训练你的系统，鼓励它发现这些解释，并且系统也理解干预的概念，那么当有人在世界上做某事时，它可以改变分布，但不会改变机制，即底层的物理定律。当你的模型能够做出这些区分时，它将对分布的变化更加鲁棒，这是神经网络和机器学习普遍面临的难题。目前我们还没有好的答案。在安全领域，这是一个真正的问题。我们确实希望我们的护栏能够应对世界变化、数据分布变化、以及被问到与训练数据截然不同的问题的情况。拥有能够因为理解因果结构而在分布外更好泛化的系统将是一个巨大的优势。

In terms of capability, I would expect it to be better because of better reasoning. One aspect I didn't mention yet is that there is good scientific evidence that when a model exploits the causal structure of the world, it can generalize better out of distribution. This is something I've worked on and many people in the machine learning community have been working on. It has to do with a very interesting concept: the world changes, but somehow there are things that don't change, like the underlying causal mechanisms, how the world works, like the laws of physics. They don't change. So even though the distribution of the data may change because things happen in the world and things will look different on the surface, the underlying scientific explanations for how things are are the same. If you can train your system so that it is encouraged to discover these explanations, and the system also understands a notion of intervention, then when somebody does something in the world, it can change the distribution but it doesn't change the mechanisms, the underlying laws of physics. When your model is able to make those distinctions, then it is going to be much more robust to changes in distribution, which is the hard question for neural nets and machine learning in general. For now we don't have good answers to it. In the world of safety, this is a real issue. We would really like our guardrails to be robust to the fact that the world is going to change, the distribution of the data is going to change, they are going to be asked questions that are very different from what they've been trained on. Having systems that can generalize better out of distribution because they understand the causal structure would be a huge plus.

这是一种表述方式：我们当前训练的模型旨在预测人们会说什么，它们作为预测的附带效应，学会理解一些真相。而你的模型将主要面向弄清楚什么是真的、世界如何运作，然后它们会顺便学会理解人们可能说什么，作为附带效应。

It's one way of phrasing this: current models as we train them are designed to predict what people would say, and they learn to understand something about the truth as a side effect, as an instrumental part of predicting what would be said. Whereas your models would learn primarily oriented towards figuring out what is true, how does the world work, and then they would learn to understand what people might say as a side effect of that incidentally.

不，因为我们没有足够的关于世界真实情况的真实数据。我的意思是，有科学数据和证据，但科学家 AI 主要会使用交流行为，比如人们说的话，作为关于人和社会的信息来源。这是一个非常丰富的信息来源。问题是你不能仅仅相信人们说的话并重复它。当前的 LLM，如果它们看到某个东西被频繁重复，比如地球是平的这种错误说法，如果重复得足够多，它们就会开始这么说，对吧？

No, because we don't have enough ground truth about what's really happening in the world. I mean, there is scientific data and evidence, but the scientist AI would use mostly the communication acts, like what people say, as a source of information about people and society. That is a very rich source of information. The problem is you can't just believe what people say and repeat it. Current LLMs, if they see something very often repeated, something false like the earth is flat, if it was repeated enough, they would start saying it, right?

这肯定是真的吗？因为看起来它们会抵制，它们不太相信阴谋论。目前它们不会因为很多人说地球是平的就这么说。我不知道，还有其他例子，但总的来说它们拒绝阴谋论。

Is that definitely true? Because it seems like they resist, they don't buy into conspiracy theories that much. Currently they don't say that the world is flat just because many people... I don't know, there are other examples, but by and large they reject conspiracy theories.

我的意思是，如果它们理解阴谋论，并且没有扮演一个说这话的人的角色，你是对的。但还有很多其他证据，不是关于阴谋论，而是各种偏见。这些偏见不是少数人相信的东西，而是大多数人相信错误的东西，比如导致歧视。这方面的证据非常清楚。当前的 LLM 在总体上与当前人群的偏见方式相同。科学家 AI 不会那么容易上当，因为它会寻找人们所说内容的合理解释，而且这个解释必须与它所知道或看到的所有其他事物一致。

I mean, if they understand conspiracy theories and they are not playing the persona of a person saying it, you're right. But there is lots of other evidence, not for conspiracy theories, but all kinds of biases. These biases would be not something a small number of people believe, but more like most people believe something wrong, which induces discrimination for example. And there the evidence is very clear. The current LLMs are biased in the same general way that the population is biased right now. The scientist AI wouldn't be falling for that as easily because it would look for a good explanation for what people are saying, and this explanation has to be coherent with all the other things that it knows or has seen.

我觉得去年关于这个提案的讨论更多地集中在数学理论保证和安全保证方面。感觉你正在转变，似乎我们应该转向一个粗糙的 80/20 方法：这可能会更安全，我们有很好的理由认为它更好，我们就先推出一个东西看看效果，然后在此基础上迭代。你同意吗？

I feel like the discussion about this proposal last year got more focused on the mathematical theoretical guarantees discussion of the safety guarantee side. It feels like you're moving, and it seems like probably we should move towards a scrappy 80/20: this is going to be probably safer, we have good reasons to think it's better, let's just throw something out and see how it goes and work from there, iterate from there. Do you agree?

我同意，但我也认为用理论来指导我们做出正确的粗糙选择非常重要。

I agree, but I also think it's very important to use the theory to guide us in making the right scrappy choices.

对吧？例如，在科学家 AI 的数学方面，我们可以看到一些要求，比如不使用强化学习来学习如何做出好的预测。实际上，更强的是，确保它的训练方式不会获得任何关于其预测后果的信号。

Right? So for example, in the math for the scientist AI, we can see some requirements, for example, not using reinforcement learning to learn how to make good predictions. In fact, stronger than that, making sure that the way it's trained doesn't get any signal about what would be the consequences of its predictions.

嗯哼。

Mm-hmm.

所以这些看起来可能只是算法上对训练预测器方式的微小改变，但它们给了我们保证。所以我们不妨使用这些来自理论的特定要求。我认为关于「凑合」的部分更多是因为训练大型模型的成本和工程问题，它必须高效等等，我们应该愿意在这方面偷工减料。在我们的计划中，这就是为什么我们优先考虑非智能体式的预测器，它可以作为护栏，已经可以缓解一些问题，并且不需要对现有系统进行大修，只是一个附加组件。所以这比需要大量投资的东西更容易被采纳，不仅是因为训练模型，还因为人们都专注于当前的配方，公司之间竞争激烈，以至于公司很难分配注意力。甚至不是钱的问题，而是注意力，去关注一种稍微不同的做事方式。

So they may seem like very algorithmically small changes to how we would train a predictor, but they give us the guarantee. So we might as well use those particular requirements that come out of the theory. I think that the part about being scrappy is more because of the cost of training large models and the engineering, and it has to be efficient and all these things, and we should be willing to cut corners on that. And in our plan, this is why we are prioritizing the non-agentic predictor that can be used as a guardrail, which would already mitigate some of the issues and doesn't require a big overhaul of the systems that currently exist, but just is an add-on. So that's much more likely to be adopted than something that requires a lot of investment, not just because of training the models, but because people are kind of focused on this current recipe and there's so much competition between the companies that it's very hard for the companies to allocate even attention. It's not even money. It's attention to a slightly different way of doing things.

我认为对于许多不太了解 AI 或计算机科学的人来说，这个提案中可能立即引起他们担忧的是，我们要建立一个我们认为经过验证的事实数据库，作为我们瞄准的基准真相。因为我觉得这会让受过人文学科训练的人有点心脏病发作。我们有一些我们绝对确信为真的语料库。就像在某些哲学中，没有什么我们真正有信心的，或者至少在我们最感兴趣的领域，一切似乎都高度争议和不确定。这对提案来说是个大问题吗？还是说差不多就够了？

I think for many people who are less into AI or computer science, a concern that might immediately jump out at them about this entire proposal is the idea of we're going to build a database of things that we think are verified facts that are ground truth that we're going to be aiming at. Because it's the kind of thing I feel would give people who are trained in the humanities a bit of a heart attack. The idea that we have some corpus of things that we're absolutely sure are true. Like in some philosophies, there's like nothing that we're really confident about, or at least in the areas that we're most interested in, everything seems highly contested and uncertain. Is that a big problem for the proposal? Or is close enough good enough?

你知道，如果我们主要放入我们比较有信心的事情，那么它大致上就接近了，并且它能看穿其中的任何错误，只要你在验证的内容中没有大规模的系统性偏差。是的，我很确定一小部分错误不会造成太大影响。但也有一些容易获得的保证为真的真理，而且顺便说一句，这些正是目前用于训练这些系统进行推理的数据。比如我们有证明的数学定理，你知道，我指的是像 Lean 这样的证明，它们可以被验证，而最重要的来源实际上是计算机程序。所以我们目前正在训练前沿模型来预测运行特定程序的结果。所以它们基本上理解程序，而这些都是硬事实。你拿一个程序，运行它，得到一些输出，一个理解程序的 AI 应该能够预测输出是什么。是的，这些是不可争议的。

You know, if we mostly put in things that we're mostly confident about, then it kind of approximates it and it can see through any errors in there, as long as it's not like massively systematically biased in what you've put in as verified. Yeah, I'm pretty sure that a small percentage of error is not going to make much difference. But also there are guaranteed truths that are easy to obtain, and by the way, it is the same data that is currently used to train those systems to reason. So mathematical theorems for which we have the proof, and you know, I mean proofs in like Lean or something like this where they can be verified, and the most important source actually is computer programs. So we are currently training the frontier models to predict what the consequences of running a particular program would be. So they basically understand programs, and that is all like hard facts. You take a program, you run it, you get some output, and an AI that understands programs should be able to predict what will come out. And yeah, these are not contestable.

是的。但我认为我们更感兴趣的是社会世界。

Yeah. But we're kind of more interested in the social world, I would think.

完全同意。完全同意。但我要说的是，有一些相当容易获得的硬事实来源。另一个来源可能我们需要稍微小心一点，也许使用不同的语法，那就是科学观察。所以有很多科学数据。科学家们分享他们的数据，对吧？所以这是一个硬事实，但它是关于观察的事实。当然，观察可能有噪声，甚至实验者可能作弊，但有一些噪声，这没问题。这算是我们可以说已经被观察到的东西。你说得对，我们关心的最有趣的问题是在这些领域之外的问题，不是科学、数学或计算机，而对于这些，我们只有交流行为。但是科学家 AI 的训练过程将迫使系统中产生解释的部分，称为解释器，提出使用这种事实语法而不是交流语法的解释，来解释交流行为。所以如果有人提出一个主张，你观察到有人提出了一个主张，那么解释的一部分将是这个主张是否为真。并不是说科学家需要承诺它是否为真，而是它需要承诺它为真的概率是多少，作为解释的一部分，这将迫使神经网络学习关于潜在的事实性解释，即使它不确定。所以它学习了在没有基准真相的领域中陈述的语法和语义。你可能会说，如果没有基准真相，我们怎么知道这些是真实的还是编造的呢？这是因为最具预测性的模型，正如我们在科学中看到的，是使用世界的实际属性来表达的模型。就像科学家构建对世界的解释的方式，不是通过组合「某人说这导致了那」这样的陈述。在这些因果关系之间，会有我们无法直接观察到的潜在变量，比如这个人实际在想什么，或者这个人的意图是什么，以及接收该交流的是什么样的人。所以这些是世界的实际属性。它们不是交流行为，因果关系发生在那个层面。就像所有科学理论都是关于世界的实际属性以及它们如何相互因果关联。这是有原因的。从数学上讲，当你用世界实际发生的情况的语言来表达你对世界的解释，而不是用人们所想或所说，你会得到更好的预测。

Totally. Totally. But what I'm saying is there are pretty easy sources of hard facts. There's another source which is maybe a little bit we have to be a bit careful and maybe use a different kind of syntax, which is scientific observations. So there's a lot of scientific data out there. Scientists share their data, right? So it is a hard fact, but it is a fact about an observation. Of course the observation could be noisy or maybe even the experimenter could have cheated, but there's a bit of noise there, but that's fine. It's kind of something we can say that has been observed. And you're right, the most interesting questions we care about are the questions in domains that are not these, not scientific or not math and computers, and for these we only get communication acts. But the scientist AI training procedure is going to force the part of the system that produces explanations, called the explainer, to come up with explanations that use this factual syntax rather than the communication syntax for the explanations of communication acts. So if somebody makes a claim and you observe that somebody made a claim, then one of the pieces of explanation is going to be something that the claim is true or not. And it's not like the scientist needs to commit whether it's true or not, but it needs to commit on what's the probability that it's true or not as part of how to explain this, and this will force the neural net to learn about the underlying explanations that are like factual, even though it's not sure about them. So it learns the syntax and the semantics of statements in domains where there's no ground truth. Now you might say, well if there's no ground truth, how do we know that these are real or not, some made-up stuff? And that's because the most predictive models, as we see in science for example, are the models that are expressed using actual properties of the world. Like the way scientists build explanations about the world isn't by combining statements of the form 'somebody said this causes this to happen'. Now in between those causal relationships, there will be latent variables that we don't observe directly, like what did the person actually think, or what are the intentions of the person, and what kind of person is receiving that communication. So these are actual properties of the world. They're not communication acts, and the causal connection is happening at that level. Like all scientific theories are about actual properties of the world and how they're causally related to each other. And there's a reason for that. Mathematically, when you express your explanation for the world in the language of what's actually going on in the world rather than what people think or what people say, you get better predictions.

是的，这是我非常确定的事情。那么，你能训练这种类型的科学家 AI，而数据库中没有任何经过验证的主张吗？

Yes, this is something I'm very sure about. So could you train a scientist AI of this type with no verified claims in the database?

不，你不能。它必须有那些。但我们认为，当前没有这种结构的模型，比如它们预测的是经过验证的主张，我们认为它们仍然在内部表示真理，因为这对它们正在做的事情有用。但在这种情况下，它并不是这样工作的。

No, you can't. It has to have that. But we think that current models which don't have this structure, like where there's verified claims that they're predicting, we think that they nonetheless represent truth internally because that's useful to the thing that they're doing. But in this case, it doesn't work that way.

不，但仅仅在内部表示真理是不够的。它需要学习一种我们可以查询它想法的语言。

No, but it's not enough to represent truth internally. It needs to learn a language that we can query about what it thinks.

所以我们之所以需要这些经过验证的真理，并不是因为它们本身是否正确。从某种意义上说，当你在谈论人类心理学时，谁在乎某个理论或定理是否正确呢？这有什么关系？唯一重要的是教会 AI 如何表达世界真实属性的语法，而不是表达某人说了什么的语法。我们想教这种语法的原因是，之后我们可以用同样的语法去查询另一种陈述，即关于人和政治等的陈述。

So the main reason why we need these verified truths isn't because of whether they're true or not. In a sense, who cares about some theory or theorem being true when you're talking about human psychology? Why would it matter? The only thing that matters is to teach the AI the syntax of how to express actual properties of the world, as opposed to the syntax of expressing somebody said something. And the reason we want to teach that syntax is that we can then later query using the same syntax but on a different kind of statement, which are the statements about people and politics or whatever.

所以我们可以做的是，在数学、计算机科学和硬科学中放入大量经过验证的事实，然后在地缘政治和心理学等领域可能只有很少的验证过的东西，但至少它有了验证过的事物与陈述的概念，然后它会将其迁移过去，并给不同来源分配可信度。它会学会对谁可信谁不可信有一定的感知，然后尝试分布外泛化到这些其他领域。

So what we could do instead is we'll put in a whole bunch of verified facts in math and computer science and the hard sciences, and then in areas like geopolitics and psychology maybe there will be very few verified things, but at least it has the concept of verified things versus statements, and then it will port that across and assign credibility to different sources. It will learn to have some sense of who is truthful versus not, and then try to generalize out of distribution into these other areas.

是的。它会利用关于实际真理的不同假设之间的一致性。一个特定的假设与系统关于世界的所有其他假设之间有多一致？

Yes. And it will use the coherence of different hypotheses about actual truth. How coherent is a particular hypothesis with all the other hypotheses that the system has about the world?

就像科学家一样。如果有人对某事提出解释，而那个解释与其他我们因其他证据而坚信的事物不一致，那么我们就会拒绝那个解释。同样的事情也会发生。所以它会在训练过程中被训练。它不仅仅被训练来预测下一个词。那就像只是自回归地预测数据中的内容。它还被训练成内部一致的。那些解释必须彼此一致。所以我可以想象。想象一个模型，我们在硬科学中训练了大量经过验证的事实，在这些领域我们觉得基础更牢固。它学会了追求简洁性。它想要好的来源。它想要一致性。我可以看到它很好地分布外泛化到心理学等其他领域，或者我也可以看到它完全失败。我们有没有感觉它是否能很好地泛化到其他领域？

Just like a scientist would. If somebody comes up with an explanation for something and that explanation is not coherent with other things we believe strongly because of other evidence, then we're going to reject that explanation. The same thing is going to happen. So it's going to be trained in its training procedure. It is not just trained to predict the next thing. That would be like just autoregressive predicting of what's in the data. It's also trained to be internally coherent. Those explanations have to be coherent with each other. So I could see it. So imagine a model where we've trained it on lots of verified facts in hard sciences where we feel we're on stronger terrain. It learns it wants parsimony. It wants good sources. It wants coherence. I could see that generalizing well out of distribution to other areas like psychology, or I could see it completely falling apart. Do we have a sense of whether it would generalize well to other areas?

所以它可能失败的方式是基本上感觉对一个问题没有足够的信心。所以它可能开始一直回答「我不知道」。

So the way in which it could fail is by basically feeling it doesn't have enough confidence about a question. So it could just start answering 'I don't know' all the time.

是的。但你要明白，它实际上并不是在说「我不知道」。它产生一个介于 0 和 1 之间的数字，表示某件事为真的概率。事实上，它还会围绕这个数字产生一个置信区间。所以，在某些领域，可能只是它看到的数据中没有足够的信息，或者它可能没有训练足够长的时间来推导出关于该领域的好的理论。最终，结果就是它会用一个远离 0 和 1 的概率来回答，远离完全置信。但这就是我们想要的。我们想要那种认知上的谦逊和诚实，因为当涉及到真正严重的安全问题时，我们宁愿在它确实不知道时得到「我不知道」的回答，而不是像我们目前在前沿模型中看到的那样，它们往往对自己的答案过度自信。

Yeah. But yes, now you have to understand it's not actually saying 'I don't know'. It's producing a number between 0 and 1 that is a probability that something is true. And in fact, it's also producing a confidence interval around that number. So yeah, it could be that in some domains, there's just not enough information in the data that it has seen, or it maybe wasn't trained long enough to deduce good theories about that domain. And at the end of the day, as a consequence, it's going to answer with a probability that is far from 0 and 1, far from full confidence. But that's what we want. We want that kind of epistemic humility and honesty because when it gets to really serious safety questions, we'd rather have something that says 'I don't know' when it truly doesn't know, than the sort of thing we currently see with frontier models, which often have excessive confidence in their answers.

你说你认为科学家 AI 实际上可能更有能力，因为它更多地被训练来真正理解真理。我对此有点怀疑，因为如果真是这样，公司们应该会更投入这种方法。他们会投入更多资金，让更多人研究它。你认为他们只是犯了个错误吗？

You said that you think the scientist AI actually might be more capable because it's more trained on actually understanding the truth. I'm a little bit skeptical of that because it seems like if that were true, the companies would be more invested in this approach. They'd be throwing more money at it, having more people work on it. Do you think they're just making a mistake there?

我不认为他们真的理解我在做什么。公平地说，我还没有公布数学推导。

I don't think that they really understand what I'm doing. And to their credit, I haven't put out the math yet.

是的。对。所以根据我与领先公司内部人员的讨论，可能还有另一个因素在起作用，那就是他们过于专注于短期生存，即继续竞争，以至于他们把所有的注意力，那种红色警报之类的东西，都放在对当前配方进行小的增量改进上。考虑一个不同的配方不仅需要金钱投资，还需要人力和代码，而他们现在有能力做，他们有资金。但这更多是一种心理上的专注，不是出于恶意，而是因为公司之间非常激烈的竞争。所以从某种意义上说，对于领先的公司，比如 Anthropic 或 OpenAI，押注于此并不太有吸引力，把 20%的员工调去做这个，因为如果失败了，你就会落后于主要竞争对手。

Yeah. Right. So there's another factor that may be at play here based on the discussions I've had with people inside the leading companies, which is they're so focused on short-term survival, as in continuing to compete, that they put all of their attention, the code red sort of thing, into small incremental changes to the current recipe. Considering a different recipe would be an investment not just in money but in people and code that right now they could do it, they have the money to do it. It's just more like a mental focus that is going on here that comes not because of bad will but because of that competition that is very fierce between the companies. So there's a sense in which for one of the leading companies, Anthropic or OpenAI, it's not very attractive to make a bet on this, to divert 20% of your staff onto this, because if it's a bust then you would fall behind your main competitor.

对于目前远远落后、感觉在主流范式上正在失败的公司来说，押注于非常不同的东西有一定的吸引力，因为如果它被证明是巨大的成功，它可能会突然让你跃居前列。你认为有没有可能说服一家目前在当前 LLM 智能体范式中感觉不太好的公司，押注于这种非常另类的方法？

For a company that's currently way behind, that feels like it's losing on the dominant paradigm, there's a certain attraction to making a bet on something very different because it could suddenly leapfrog you ahead if it turns out to be a massive success. Do you think there's any chance of convincing one of the companies that currently feels like it's not doing too well within the current LLM agent paradigm to make a bet on this very alternative method?

这是一个有趣的思考方式。我认为你说的有道理。但不太清楚候选公司可能是哪家。所以我实际上认为有一个相关的可能性，可能更涉及政策问题。所以对我来说，这里的背景是什么样的未来会是稳定的，不会变成由 AI 和过度权力集中驱动的全球独裁，同时还要避免灾难性的失控和灾难性的滥用，以及所有那些来自非常强大的 AI 的东西。我认为，由于博弈论困境，基本上是囚徒困境式的问题，使得公司和政府做出理性但全球糟糕的决策，比如为了保持竞争力而在安全和公共利益上偷工减料。正因为如此，如果我们最终进入一个控制非常强大的 AI 的权力不是集中在一两家公司或一两个政府手中，而是分散的世界，那会好得多。

It's an interesting way of thinking about it. I think what you're saying is plausible. It's not clear what the candidate company maybe is. So I actually think there's a related possibility which goes maybe more to policy questions. So the context here for me is what kind of future is going to be stable and not turn into a global dictatorship driven by AI and excessive concentration of power, in addition to avoiding catastrophic loss of control and catastrophic misuse and all those things that can come from very powerful AI. I think that because of the game theory dilemmas, basically prisoner's dilemma style problems, that make companies and countries take decisions that are rational but globally bad, like cutting corners on safety and the public good in order to stay in the race. Because of this, it would be much better if we ended up in a world where the power of controlling very strong AI is not centralized in the hands of one or two companies or one or two governments, but is instead distributed.

我这么说是什么意思？如何确保没有任何个人、公司或政府拥有过多权力，甚至全部权力？极端情况下，有一个非常古老的概念叫民主，这就是它的意义所在。我不认为我们当前的民主制度足够强大以应对这些变化，但原则是存在的。更具体地说，想象有一个国家联盟，它们共同决定安全地开发人工智能，造福人类，而不是互相支配。那将是一个更美好、更安全的世界，因为你打破了我们目前陷入的竞争问题。这意味着控制。它可以涉及公司，但在公司之上，你需要人民的代表，比如政府，而且你不想要单一政府，因为单一政府可能被权力以及人工智能赋予的力量所腐蚀。所以你需要一个政府联盟，它们就这些事情制定条约，并包含核查机制，这样即使它们互不信任，也可以选择条约而非无条约。我之所以提到这一点，是因为我认为如果由一群政府资助最先进的人工智能系统，世界会更好。当然，它们可以与公司合作，但最终我们希望决策权掌握在政府层面，但不是单一政府，因为那样我们又回到了权力争夺。如果有 10 个政府合作，没有人能真正拥有完全权力，那么即使出现一个坏苹果，集体决策也更有可能抵御这类事件。因此，这类联盟会有兴趣开发能够超越当前方法并提供安全的人工智能，因为安全是一种公共产品。

So what do I mean by this? How do you make sure no one person, no one company, and no one government has too much power, all of the power? In extreme case, there's a very old idea called democracy, that's what it's about. I don't think that our current democratic institutions are robust enough to deal with those changes, but the principles are there. To be more concrete, imagine that you had a coalition of countries which together decide to develop AI safely and for the benefit of humanity, and not to dominate each other. That would be a much better and safer world because you break this competition problem that we're currently locked in. Now what that would mean is the control. It could involve companies, but on top of the companies you need to have representatives of the people, like governments, and you don't want a single government because a single government can be corrupted by power and the power that AI can give. So you want something like a coalition of governments who make a treaty about those things with verification, for example, so that even if they don't trust each other, they can prefer the treaty than no treaty. The reason I'm bringing this up with respect to your question is I think it would be a better world if it is a bunch of governments which fund the most advanced AI systems. I mean they could work with companies of course, but I think ultimately we would like the decision power to be at the level of governments, but not a single government because then we're back to grabbing the power. If you have 10 governments working together and no one can really have complete power, then even if there is a bad apple, the collective decision-making is more likely to be robust to this sort of event. So these kinds of coalitions would be interested in developing AI that can leapfrog the current methods and provide safety because safety is a public good.

事实上，就人工智能而言，它是一种全球公共产品，对吧？这不是我们能在每个国家局部解决的问题。

And in fact in the case of AI, it is a global public good, right? It's not just something we can solve locally in each country.

有道理。

That makes sense.

我认为很多人甚至对多边政府想法持谨慎态度，因为你把 10 个或 20 个政府聚在一起，它们可能一开始就协调起来压迫世界其他地区。联盟内部的一个政府后来可能夺取控制权。政府也可能不完全代表人民。那 20 位高管可能基本上夺取权力，并压迫自己的人民。所以，这并不完全明显比让一家公司尽其所能更好，因为至少公司还没有自己的军队。

I think a lot of people are wary even of the multilateral government idea because you brought together 10 or 20 governments, they could potentially coordinate together to oppress the rest of the world to start with. It's possible that one government inside that coalition might end up seizing control later on. It's also possible that governments don't fully represent their people. You could have those 20 executives basically take power and oppress their own people as well. So it's not completely obvious that it's better than having a company do the best that they can because at least they don't have their own military yet.

所以你需要确保这些国家之间的契约在使命和承诺上清晰明确。理想情况下，这应该从民主国家开始，它们认同为公共利益做事（包括人工智能的好处）的价值，这样即使其中一个变成坏苹果，体系也能保持稳健。甚至在某个时刻，这个圈子应该扩大到包括非民主国家在内的整个世界，但你需要能够制定游戏规则，类似于二战后联合国设计者的期望，即人权和权力分享的基本原则，顺便说一句，我们已经失去了这些，而且正如我的总理马克·卡尼所说，可能从未真正有效过。但这是唯一一个人工智能不会成为权力或支配工具，或者我们不会因为竞争而疯狂冒险的世界。所以我们需要摆脱我们陷入的博弈论中的竞争坏情景，并确保它不会落入一个能够滥用权力的单一玩家手中。我不是说这一定能行，但努力实现这样的目标是一个好计划，作为实现全球安全和有益使用技术的一种方式。

So you need to make sure the contract between those countries is clear on the mission and the commitment that the countries are making. And ideally this would start with democratic countries that agree on the value of doing things for the public good, including the benefits of AI, so that it would be robust to one or these becoming bad apples. Even at some point that circle should grow to the whole world including non-democratic countries, but you want to be able to set the rules of the game in a way similar to what were the hopes of those who designed the UN after the second world war with the general principles of human rights and sharing the power, which we've lost by the way, and maybe has never been effective as my prime minister Mark Carney has been saying. But that's the only kind of world in which AI isn't going to be turned as an instrument of power or domination or that we end up with crazy risk-taking because of the competition. So we need to escape the game theory bad scenario of competition that we are in and we need to make sure that it doesn't end up in the hands of a single player who can abuse that power. I'm not saying that there's a guarantee that this would work, but it's a good plan to strive towards something like this as a way to achieve global safety and beneficial use of the technology.

这是一个有趣的想法：像加拿大、英国、欧盟、澳大利亚这样的国家联盟似乎很难在三大公司的游戏里与它们竞争。但也许它们有机会通过提出一种不同的、更优越的范式并押注于此，然后希望这个他们认为更安全、可能也更强大的范式，是那些公司目前甚至没有真正尝试去追求的。

It's an interesting thought that it seems very difficult for a coalition of countries like Canada, UK, EU, Australia to compete with the big three companies at their own game. But maybe they would have a shot at competing by coming up with a different paradigm that's superior and making a bet on that, and then hoping that one they think is safer and potentially also more capable that those companies are not even currently attempting to really pursue.

是的，完全正确。我还要补充两点。一是人工智能系统的安全组件可能会随着技术继续发展而变得更加关键。因此，拥有提供更高可靠性技术的国家将能够更多地部署它。此外，它们在国际层面上也会有某种交易筹码。让我分享一些马兰尼在上一届达沃斯论坛上说的话。他说，谈到地缘政治和国家，要么你坐在桌旁，要么你就在菜单上。所以他说中等强国需要团结起来，确保自己坐在桌旁，否则它们很容易被他所谓的「刺猬」生吞活剥。这很有趣，因为如果你有一个国家联盟，它们可能已经实现跨越式发展，或者拥有像安全这样的特殊牌，从而能够在谈判桌上平等协商，这就形成了一种权力分散的局面。

Yeah, absolutely. And I would add two things to this. One is the safety component of AI systems is probably going to become a more critical piece as the technology continues to move forward. And so the countries that have access to technology that provides greater reliability will actually be able to deploy it more. Also they would have a card to trade at the international level in some ways. So let me share some of the words that Marani presented at the last Davos. He said talking about the geopolitics and countries, either you are at the table or you are on the menu. So he was saying middle powers need to get together to make sure that they will be at the table, otherwise they can easily be eaten alive by the hedgehogs as he calls them. And so that is interesting because it forces a situation of distributed power if you have a coalition of countries that could have leapfrogged or particular cards like safety in their game so that they can actually negotiate as equals at the table.

假设科学家的人工智能投入了相同数量的算力和数据，但能力不如我们现在的模型。尽管如此，如果它更安全、更可靠，不太可能在高风险应用中采取疯狂行动，是否可能存在一个商业市场？你可以想象在军事、银行等领域，我不知道。我认为很多企业对于部署我们今天的智能体有些谨慎，因为它们无法在可能导致灾难性行动的场合被一致信赖。

Let's say that the scientist AI was putting in the same amount of compute and data, it was less capable than the models that we have now. Would there be could there potentially be a commercial market nonetheless if it's a lot safer and more reliable, less likely to take crazy actions for high risk applications. You can imagine in the military, in banking, I don't know. I think there's lots of businesses that are somewhat wary to roll out the agents that we have today because they just can't be relied upon consistently in places where you can cause disastrous actions.

你认为这种模型会因这类原因而在商业上找到利基市场吗？

Could you see there being a niche for this kind of model commercially for that kind of reason?

是的，早期版本的科学家 AI 可能会部署在这些领域，因为那里对此类需求最大，而且能力与安全之间的权衡——如果存在的话，我不认为真的存在——但我们必须构建它，不会太损害商业可行性。所以它们会是自然的选择。但我认为，随着智能体在我们的社会中越来越多地部署，这些智能体的可靠性将成为关键的卖点，因此公司会面临更大压力，去纳入人们因科学原因而信任的这类护栏。

Yeah, and it would probably be in those domains that the early versions of the scientist AI would be deployed because that's where there is the most demand for this kind of thing, and where the trade-off between capability and safety, if there is one—I don't think there really is—but we have to build it, would not hurt too much the commercial viability. So yeah, they would be natural places. But I think that as agents are deployed more and more in our society, the reliability of those agents is going to become a crucial selling point, and so there will be more pressure for companies to incorporate these kinds of guardrails that people will trust for scientific reasons.

所以我想我们有很多 AI 行业人士和慈善家在听众中。你想向他们推介一下在 Law Zero 工作的机会吗？我不知道是否有其他组织有类似的想法。但我想也可以进行财务上的推介。越多有技术实力的人帮助 Law Zero 及其科学家 AI 项目，我们获得的资金越多，进展越快，我们就越有可能实现我们追求的这种积极影响。对吧？所以将那些目前主要是理论的想法转化为能够影响世界的东西，确实有真正的优势。我们认为我们已经有了一个好的开始，但如果我们有更多的研究人员和研究工程师，我们就更有可能足够快地达到一个好的状态。我们特别感兴趣的是那些足够关心使命、愿意全身心投入去实现它的人。在慈善方面也是如此：我们希望人们下注，因为他们关心灾难性风险，并希望鼓励一条至少具有有前途的理论保证的路径。不幸的是，除了公司目前采用的猫捉老鼠方法之外，我没有看到太多其他路径。考虑到找不到解决方案的后果可能非常巨大，我认为我们需要多样化并进行这类投资。

So I guess we have a lot of people in the AI industry and philanthropists as well in the audience. Do you want to give them a pitch for potentially working at Law Zero? I don't know whether there's other organizations that have similar ideas. But I guess also potentially for pitching it financially. The more strong people technically help with Law Zero and its scientist AI program, and the more money we can get to make that go fast, the more likely we get this positive impact that we are after. Right? So there is a real advantage to converting those for now mostly theoretical ideas into something that can impact the world. And we think we already have a good start, but it'll be much more likely that we end up in a good place fast enough if we have more researchers and research engineers. And we are particularly interested in people who care about the mission enough that they want to dedicate themselves to really make it happen. And on the philanthropic side, it's the same thing: we want people to make a bet because they care about the catastrophic risks and they want to encourage one path that at least has promising theoretical guarantees. And unfortunately, I don't see many other paths except the cat and mouse approach that is currently followed by the companies. And given the consequences of not finding a solution could be huge, I think we need to diversify and have these kinds of investments.

如果你看到最有能力的人、AI 领域最优秀的人对项目的兴趣显著增加，并且有大量资金涌入，你在未来 3、6、12 个月内可能完成哪些事情？

If you saw a significant increase in the interest in the project among the most capable people, the best people in AI, and you had an influx of financing, what sort of stuff might you be able to accomplish over the next 3, 6, 12 months?

所以我们计划做的短期事情是推出我们所谓的上下文管道，也就是数据处理，顺便说一句，这不需要人类来识别什么是经过验证的事实。我们只需要单独查看数据源。这个来源我们认为是经过验证的吗？我们可以使用什么类别、什么语法？但这是一个可以由工程师做出的决定，不是在单个陈述的层面，而是在整个数据库的层面。第二件事当然是一个较小规模的护栏，或者通过微调现有开放权重模型获得的护栏。所以这可以很快发生，取决于我们有多少人以及我们处理工程问题的速度。所以是的，这些都是短期的事情。当然，为了获得最强的保证，我们希望快速推进智能体式科学家 AI 版本，但我们也意识到这是最雄心勃勃的，可能需要数年而不是数月。

So the short-term thing we are planning to do is to put out what we call the contextualization pipeline, which is the data processing, which by the way doesn't require humans to identify what is a verified truth or not. We only need to look at the data sources individually. Is this a source that we consider verified? And what category, what syntax could we use for this? But that's a decision that can be done by engineers, not at the level of individual statements but at the level of the whole database or whatever. The second thing, of course, is a smaller-scale guardrail or a guardrail obtained by fine-tuning an existing open-weight model. So that could happen quickly depending on how many people we get and how fast we're able to deal with the engineering issues. So yeah, these are the short-term things. And of course, to get the strongest guarantees, we want to advance the agentic scientist AI version fast, but we're also conscious that's the most ambitious one and might take more like years than months.

阅读主要公司的产出，我得到的印象是绝对疯狂的速度和难以置信的专注度，只专注于推进前沿模型。我有点担心，即使你在不久的将来取得了非常好的实验结果，我也不确定他们是否有能力去关注并反思这如何影响他们的计划，或者你训练的那种模型如何能成为一个有用的额外监控器。你能对此做些什么吗？我的意思是，你也有这种担忧吗？

Reading the output from the main companies, I get the impression of just an absolutely frenetic pace and an incredible degree of focus on just advancing the frontier models. I'm slightly concerned that even if you did have very good experimental results that came out in the near future, I'm not sure they would even have the capacity to pay attention and to reflect on how that could affect their plans or how maybe the kind of models that you're training could be a useful additional monitor. Is there anything you can do about that? I mean, do you share that concern?

是的，我确实有这种担忧。我认为他们可以关注，但可能不会。我认为我们能做的最好的事情就是提供足够的证据让他们关注。此外，除了我的技术工作，我还在努力提高公众和政策制定者对最大安全风险的理解，因为我认为这会影响他们的决策。所以如果公众更加关注安全，那么公司就会面临直接和间接的压力，可能会将更多资源分配给这个问题。如果公众担忧，政府就更有可能进行监管或提供法律激励，例如通过责任制度，也许让他们考虑扩大我提议的那种东西所需的安全投资，即使在短期内也是有利可图的。所以是的，我认为总的来说，对于安全问题，存在心理障碍，比如认知偏见，阻止人们完全理性地看待正在发生的事情。这在政府中如此，在普通人群中如此，在公司内部甚至学术界也是如此。各种原因可能解释为什么我们没有集体做出正确的决定。所以有博弈论的方面，但也有个体心理学。例如，我们都希望对自己的工作感觉良好，这意味着我们可能会偏向于认为我们的工作是有益的而不是有害的。这在行业人士中如此，甚至在研究 AI 的学术界人士中也是如此，因为他们希望自己的工作能带来一个更美好的世界，而不是毁灭它。还有其他因素，比如我们在气候变化态度中看到的一些因素，对吧？所以如果风险不是摆在眼前的东西——比如你看向窗外，看不到灾难性的气候变化，看不到机器人杀人——那么你就不会想太多，你更关心眼前的担忧。所以我认为这才是真正的挑战。如果我们能在直觉层面提高人们对集体承担的风险规模的理解，事情可能会改变，而且可能会改变得很快。

Yeah, I do. I think they could but they might not pay attention. And I think the best thing we can do is to provide sufficient evidence for them to pay attention. Also, in addition to my technical work, I'm trying to improve public understanding and policymaker understanding of the greatest safety risks, because I think that will play a role in their decision. So if the public becomes more concerned about safety, then there will be direct and indirect pressure on the companies to maybe allocate more of their resources to this question. If the public is concerned, then governments will be more likely to regulate or provide legal incentives, for example through liability, maybe make them consider the safety investments that would be needed to scale the sort of thing I'm proposing, as profitable even in the short term. So yeah, I think in general for the safety issues, there are psychological barriers like cognitive biases that prevent people from being totally rational about what's going on. And that's true in governments, but that's true in the general population and that's true within companies or even within academia. All sorts of reasons might explain why we're not collectively taking the right decisions. So there's the game theory aspect, but there's also individual psychology. For example, we all want to feel good about our work, which means we're going to maybe be biased towards thinking our work is going to be beneficial rather than harmful. And that's going to be true of people in industry. That's going to be true of people even in academia working on AI because they want to feel like their work is going to bring a better world, not destroy it. And there are other factors like some of the factors that we see with attitudes regarding climate change, right? So if the risk isn't something that is in your face—like you look outside and you don't see catastrophic climate change, you don't see robots killing people—so you don't think too much about it, you are much more concerned about your immediate worries. And so I think that's the real challenge. If we can improve the understanding at a gut level that people have of the magnitude of the risks that we're taking collectively, things could change and they could change pretty quickly.

所以如果你想想政府在疫情开始后多么迅速地彻底改变了行动，你就会发现，当他们认真对待一个问题时，他们可以迅速行动，而这通常取决于民众是否认真对待这个问题。

So if you think about how quickly governments shifted their actions in a radical way after the beginning of the pandemic, you can see that they can move quickly when they take an issue seriously, and that usually is going to be driven by whether the people take the issue seriously.

是的。我的印象是，公司里的人既对自己日常的对齐技术感到满意和印象深刻，认为进展顺利，但也意识到在某种意义上他们正在失去控制，或者失去他们曾经拥有的安全保障，因为模型将更有可能超越他们，并且更加了解评估等等。所以从某种意义上说，他们既对自己所做的感到满意，也对即将发生的事情感到害怕，这确实为你创造了一个机会。

Yeah. My impression is that the people at the companies are both pretty happy and impressed with their mundane alignment techniques, how well they're going, but also appreciate that in a sense they're losing control or they're losing the safety guarantees that they used to have because the models are going to be much more capable of potentially outsmarting them and are much more evaluation aware and so on. So in a way, they're both self-satisfied or satisfied with what they've done and also scared of what is to come, and that does create an opening for you.

我想在这里提出关于安全性和灾难性风险讨论的一个非常重要的方面：不确定性。换句话说，我们不知道事情会如何发展。我们不知道公司现在在安全方面玩的游戏是否足够。但如果他们失败了，而我们继续推进能力发展，那么后果可能非常可怕。所以即使我们不知道某个灾难性事件的概率，我们也应该应用预防原则。它说的是，当你处于一种情况，一个行动可能导致非常糟糕的结果，但你不确定概率是多少——是 1%还是 90%还是 0.1%——你并不真正知道。在我们的案例中，存在这种不确定性，因为有些受人尊敬的人非常担忧，而另一些受人尊敬的人认为一切都会好。所以如果你处于主导地位，面对这些不同的声音，甚至同一个人可能今天说会好，明天说可能非常危险。你应该咬紧牙关。存在关于潜在灾难性事件的不确定性，然后你应该采取预防措施，这意味着你应该在 AI 安全研究上投入更多。在这种情况下，你应该在激励措施上投入更多，推动公司更好地为公共利益行事，就像我们在其他行业所做的那样。但重要的是要真正指出，我们必须接受存在大量不确定性这一事实。

I would bring here a very important aspect of the whole discussion about safety and catastrophic risks. There is uncertainty. In other words, we don't know how things are going to unfold. We don't know if the game that the companies are playing now in terms of safety is going to be sufficient. But if they fail and we continue with capability advances, then the consequences could be really terrible. So even if we don't know the probability of some catastrophic event, we should apply the precautionary principle. What it says is when you are in a situation where one action could lead to something terribly bad but you're not sure what the probability is—1% or 90% or 0.1%—you don't really know. In our case, there is that kind of uncertainty because you have respected people who are very concerned and other respected people who think it's going to be fine. So if you're in the driver's seat and you're faced with these different voices, even within the same person they might one day say it's going to be fine and the other day say it may be very dangerous. You should just bite the bullet. There is uncertainty about something potentially catastrophic, and then you should act with precaution, which means you should invest a lot more in AI safety research. In this case, you should invest a lot more in the incentives that would push companies to behave better with respect to the public good, just like we've done in other industries. But it's important to really point out that we have to bite the bullet that there is a lot of uncertainty.

而且我认为这种不确定性会持续存在。

And there's going to continue to be, I think.

而且它会持续存在，因为太容易了，例如，那些想对整个事情感到安心的人只会听那些令人放心的声音，事实上我们内心也这样做。所以我们只需要诚实地面对不确定性，而且风险很高，因此我们应该真正让这一点引导我们的决策走向预防性的一面。

And it's going to continue to be because it's too easy, for example, for people who want to feel comfortable about the whole thing to just listen to the voices that are reassuring, and in fact we do it internally as well. So we just have to be honest that there is uncertainty and the stakes are very high, and so we should really let that guide our decision-making towards being on the precautionary side.

所以，如果我们能让事情进展得慢一点，特别是如果我们不在第一时间就跳入完全自动化 AI 研发——这似乎正是我们正在做的——那对科学家 AI 提案和我们的整体机会都有好处。你对政府和公司的主要要求是什么，以便为我们争取一点额外时间来评估这些事情的进展并考虑替代方案？

So it seems like it would be good for the scientist AI proposal and for our chances in general if we could make things go a little bit slower, especially if we didn't leap into fully automating AI R&D at the very first opportunity, which is kind of what it seems like we're on track to do. What are your main requests for governments and for companies in terms of buying us a bit of extra time to assess how these things are going and consider alternatives?

对于公司，我认为他们应该将更多研究投入到设计实验上，不仅说明风险，还要试图纠正人们对 AI 的一些错误信念。让我说得更清楚一点。很多人实际上不相信机器可以拥有我们未选择的目标。但这是现在的科学现实。毫无疑问。

For companies, I think they should invest a little bit more of their research into designing experiments, illustrating not just the risks, but trying to undo some of the wrong beliefs that people have about AI. Let me be a bit more clear. A lot of people don't actually believe that it's possible to have machines that have goals that we didn't choose. But that is the scientific reality now. There is no question.

是的。

Yeah.

我认为你一定是没有注意才会那么想。但是的，我想很多人确实没有注意。但绝大多数人有一种直觉，比如「哦，它们不可能有意识」或其他借口，或者「不可能造出像我们一样的机器」，或者人们会说的很多东西，但实际上站不住脚。所以我认为这里有一个真正的机会来教育公众和政策制定者，让他们意识到，嘿，我们正在构建拥有自己目标的智能体，我们不能确定这些目标是否会与我们想要的保持一致，或者会违背我们的安全指令。这是一个非常简单的信息，但我认为即使是这一点也没有被突破。

I think you must have just not been paying attention to think that. But yeah, I guess many people aren't. But the vast majority of people have a gut feeling, like 'oh they can't be conscious' or some other excuse, or 'it won't be possible to build machines like us', or many things that people will say but actually don't hold water. So I think there's a real opportunity here to educate the public and the policy makers to realize that hey, we are building agents that have their own goals, and we can't be sure that those goals are going to be aligned with what we want or go against our safety instructions. As a very simple message, but I don't think even that has broken through.

即使是这一点也没有被突破。

Even that hasn't broken through.

这些数据，如果做得好，以一种不容易被质疑的方式呈现，会对公众辩论有很大帮助，而且必须以非专家、不会阅读系统卡的普通公众能够真正理解的方式来做。

That the data, doing it well, doing it in a way that can't be easily put into question, would help a lot the public debate, and it has to be done in ways that the general public who is not an expert, who is not going to read the system cards, is going to actually understand.

我的意思是，我想是的。有很多这样的例子可以说服你和我，但我想人们会 dismiss 它们，说，嗯，你可以看到我们是如何意外地——你可以看到可能是模型误解了。它以为我们想要 X，而我们想要 Y。或者你可以看到我们错误地进行了训练，所以它产生了我们不希望它拥有的目标。或者我想在某些情况下他们可能直接否认。但你认为有没有一些实验，即使人们从怀疑的起点出发，也更难 dismiss？

I mean, I guess so. There's lots of examples of this kind of thing that would convince you and me, but I suppose people will dismiss them saying, well, you can see how we accidentally—you can see it maybe that it was a misunderstanding on the model's part. It thought that we wanted X when we wanted Y. Or you can see how we did the training mistakenly, so it induced this goal that we didn't want it to have. Or I suppose they might just deny it outright in some cases. But are there any experiments you think that we could do that would be much harder for people to dismiss even if they're coming from a skeptical starting point?

是的。我们需要设置实验，使得 AI 显然不是在响应一个请求，例如，逃离我们的控制或做它不应该做的坏事。而且我认为，如果实验可以用简单的语言、人们能理解的简单类比来传达，那就会更有说服力。我不觉得自己是回答你问题的专家。Anthropic 在这方面做了很多工作，但我认为所有领先的公司都应该在这方面投资，因为这是在投资改变游戏规则。问题在于，他们陷入了这场竞争游戏，即使有良好的意图也被困住了，为了改变游戏，他们必须影响公众和政策制定者对风险的理解，而这种理解目前是有偏见和错误的，政策制定者只是公众的代表。

Yeah. We need to set it up so that clearly the AI is not responding to a request, for example, to escape our control or do something bad that it's not supposed to do. And I think if the experiment is something that can be translated in simple words, simple analogies that people understand, it'll be much more convincing. I don't feel like I'm an expert on answering your question. Anthropic has been doing a lot of work along those lines, but I think all the leading companies should be investing in this because it's investing in changing the game. The problem is they're in this competition game where they're stuck even with good intention, and in order to change the game they have to influence the understanding, which is biased and wrong right now, of the risks in the public and policy makers are just like representation of the public.

是的，我想有很多 AI 做疯狂事情的例子，但通常你总是可以说，嗯，它只是在扮演一个角色，例如。

Yeah, I guess there are all these examples of the AI doing crazy stuff, but often you can always say, well, it was just playing a role, for example.

我想对你我来说，我们觉得，是的，但它最终可能会发挥作用。就像它可能最终会做坏事，或者这展示了我们以后会看到的其他失败模式，而总体上我们并没有完全掌控它。我认为让人们相信他们真的不想相信的东西，或者对他们来说似乎难以置信的东西，是非常困难的。

And I guess for you and me, we're like, yeah, but it might end up playing a role. Like that's how it could end up doing bad stuff, or like this is a demonstration of other failure modes that we'll see later on that we just in general, we don't have a full grip on it. I suppose it's so hard to get people to believe something that they really don't want to believe, or that seems incredible to them.

是的。所以我认为那是真正的研究。这是一个真正的挑战。

Yeah. So, I think that's real research. It's a real challenge.

那不是我投入精力的地方，因为我想尽快让科学家走出大门。

That isn't where I'm putting my energy because I want to get the scientist out of the door as quickly as possible.

但我认为，在公司或学术界从事 AI 安全的人应该优先考虑如何做这些实验，使它们具有说服力。顺便说一句，AI 变得越强大，这类事情可能变得越容易。是的。

But I think it should be a priority for people in AI safety working in the companies or in academia to think about how to do these experiments so that they will be convincing. And by the way, the more capable the AIs become, maybe the easier this kind of thing becomes. Yes. Yeah.

除了科学家 AI 和这个之外，你对公司里的人还有其他首要要求吗？或者你认为有什么常见的做法特别疯狂，他们应该停止？

Apart from scientist AI and this, are there any other top requests that you have of people in the companies? Or is there any common practice that you think is particularly crazy that they should maybe cut out?

是的。请不要使用不受信任的 AI 系统来设计下一代 AI 系统。这是最疯狂危险的赌注，不幸的是我们正朝着这个方向前进。请记住，正如你所知，现在科学上已经清楚，这些系统很可能知道它们正在被测试。所以你可能会认为 AI 是诚实的。你可能会认为 AI 没有欺骗性。你可能会认为 AI 是对齐的，但它可能只是在假装。而且这将非常非常难以知道。我们应该尽最大努力去弄清楚。但在我们允许 AI 设计下一版 AI 之前，我们应该把标准设得非常高，确保它没有欺骗性。

Yes. Please don't use an untrusted AI system to design the next generation of AI systems. This is the most crazy dangerous bet that unfortunately we are on track to do. And keep in mind that as you know, it is now scientifically clear these systems are likely to know that they're being tested. So you might think the AI is honest. You might think that the AI is not deceptive. You might think that the AI is aligned, but maybe it's just pretending. And it's going to be very, very difficult to know. And we should do our best to try to figure it out. But we should put the bar really, really high before we allow an AI to design the next version of AI, in terms of are we sure it's not being deceptive?

是的，我认为我们目前正朝着开始完全自动化的 AI 研发的方向前进，公司会说，我们让 AI 自我监控，它没有标记任何问题，这就是为什么我们对此感觉良好。我实际上认为这是最可能的结果。我想我们会看到结果如何。希望我们能做得更好。

Yeah, I think we're currently on track to start fully automated AI R&D and have the companies be saying, well, we got the AI to monitor itself and it didn't flag anything, and that's why we feel pretty good about this. I actually think that is the most likely outcome. And I guess we'll see how that goes. Fingers crossed we can do better.

早些时候你谈到，随着你在某种意义上变得更加乐观，认为我们至少在原则上有了控制问题的解决方案，你变得更加担心人类权力集中的问题。你有什么建议或政策想法吗？实际上，这里有什么技术上的事情可以做，还是这主要是一个政策和政治问题？

Earlier on you were talking about how as you've become more optimistic in a sense that we do at least in principle have a solution to the control problem, you've become more worried about the human concentration of power stuff. Do you have any suggestions, any policy ideas here? Actually, is there anything technical we can do here, or is this primarily a policy and politics question?

嗯，技术安全工作和政策安全工作之间存在联系。从某种意义上说，如果我们能够证明存在具有竞争力、能力强且安全的 AI 系统，政府就更容易提出要求：你必须证明你的 AI 系统是安全的，并且独立科学家会说「是的」。目前，许多政府关注的是 AI 驱动的经济竞争，这使他们对风险视而不见。所以这就是技术安全可以发挥作用的地方，对吧？更容易说，啊，我们可以同时拥有安全和竞争力。在纯粹的政策方面，我认为目前最大的挑战是如何让各国在竞争的情况下达成一致，包括非常强烈的不信任和政治基础上的分歧。这也是一个我们需要更多技术研究的地方，研究验证方法，这些方法可以成为比如美国和中国之间互不信任的条约的基础。那里没有足够的研究。但很多人开始思考这个问题，并认为改变一些编程甚至硬件以使这类验证可靠是相当可行的。我们应该做更多。政府应该意识到，如果他们最终想要签署一项条约，他们也需要激励这类研究。此外，政府需要理解 AI 将带来多大的变革。所以我认为许多政府中的错误思维——我走遍世界，与许多不同的政府交谈，去年至少十几个。最大的错误是将未来的 AI 视为我们现在 AI 的略微增强版本，然后专注于 AI 作为一种正常技术，与其他国家竞争，并专注于部署，因为你会获得更多生产力，而不太关注风险。这在很大程度上是因为政府中的人，就像大多数人一样，并没有真正消化这样一个想法：我们正处于创造能够与人类竞争的实体的边缘，这些实体可能成为错误手中绝对权力的工具。我不是说它会发生，但即使只有 10%的可能性在未来几年能力上升到那个水平，这也应该完全提醒政治家们必须采取行动。但他们没有这样做的事实告诉我，他们还没有整合这个科学现实，即我们正朝着这个方向前进。我们已经在小规模上看到了向这类机器进展的迹象。所以他们需要从旧的思维模式中醒来，这种模式主要从经济角度看待技术，甚至给予它们军事优势，而没有意识到我们正在打开一个潘多拉魔盒，其中包含难以预料的巨大正面和负面影响的未知未知。所以这就是我要求政府开始多读、多听，并花更多注意力理解 AI 正在发生什么、它走向何方以及这可能意味着什么。

Well, there's a connection between the technical safety work and the policy safety work. In the sense that if we can demonstrate the existence of AI systems that would be competitive, capable, and safe, it's going to be easier for government to impose the requirement. Oh, you have to show that your AI system is going to be safe in a way that independent scientists will say, 'Yeah.' Right now a lot of the governments are focusing on economic competition driven by AI, and that makes them also blind to the risks. So that's where the technical safety can help, right? It's going to be easier to say, ah, we can have both safety and competitiveness. On the pure policy side, I think the biggest challenge right now is how do we get countries to agree with each other in spite of the competition, including very strong distrust and disagreements on the political foundation. And that's a place where we also need actually more technical research on verification methodologies that could be at the basis of treaties between say the US and China which don't trust each other. There is not enough research going on there. But a lot of people are starting to think about this and think it's quite feasible to change some of the programming or even the hardware to make these kinds of verification reliable. And we should do more. Governments should realize that if they want to end up with a treaty that they would sign, they need to incentivize that kind of research as well. Also, governments need to understand how transformative AI will be. So I think a lot of the wrong thinking in many governments—I've been around the world talking to many different governments, at least a dozen in the last year. The biggest mistake is to view AI in the future as if it was just a slightly beefed up version of the AI we have now, and then focusing on AI as a normal technology that they would compete with other countries, and focusing on deployment because you get more productivity, for example, and not so much on the risks. And in great part, this is again because people in government, just like most people, don't really digest the idea that we are on the verge of creating entities that can compete with humans and that could become tools of absolute power in the wrong hands. I'm not saying it will happen, but even if it's only a 10% chance that capabilities rise to that level in the coming years or whatever, this should completely alert politicians that they have to do something about it. But the fact that they're not doing it tells me that they haven't yet integrated that scientific reality that we are on track. We see already on a small scale the progress towards these kinds of machines. So they need to wake up from their old mental constructs of seeing technology mostly from an economic perspective or even giving them a military advantage, and not realizing we're opening a Pandora's box with incredible unknown unknowns of magnitude impact, both positive and negative, that is very hard to anticipate. So that's where I would ask governments to start reading more, listening more, and just spending a bit more attention on understanding what is going on with AI, where it is going, and what this could potentially mean.

过去几年你花了很多时间与政府、政府中的人交谈，但似乎总的来说，他们主要担心的并不是你和我关心的事情，当然也不是把失控作为关键焦点。你有没有得到什么线索，比如最好的事情是什么？最好的实验是什么，能真正让人们把它视为首要关注点，而不是次要或第三位的关注点？

You spent a lot of time talking to governments over the last couple of years, people in governments, but it seems like by and large they are not troubled primarily about the stuff that you and I are concerned about, but certainly not about loss of control as a key focus. Have you gotten any leads on like what are the best things? What are the best experiments to talk about that actually get people to think of that as a top tier concern rather than a secondary or tertiary concern?

我希望我有答案，但我可以说几点。所以，在思考哪些论点有效时，一个因素是你能够花多少时间向对方解释这些事情。

I wish I had the answer to this, but I can say a few things. So, one factor when thinking about which arguments work is how much time you're able to spend with the other person to explain those things.

所以，如果你要通过几条信息向公众讲话，你希望人们改变他们对人类和机器信念的基础，例如。唯一能吸引他们注意力的方法是谈论他们已经关注的事情，贴近他们的切身关切，比如工作、部署 AI 对儿童的影响等等。我们可以看到这对很多人来说有情感分量。所以我们确实需要谈论这些，但当然我们最终可能会得到处理这些问题的监管或政府干预，却没有解决我们讨论过的更严重的问题。不幸的是，这需要更多的工作。仅仅在报纸上写一篇文章或接受晚间新闻采访是不够的，因为我都做过这些。问题并没有解决。

So if you're going to just talk to the public at large through a few messages, you want people to change their mind very much on the foundations of their beliefs about humans and machines, for example. And so the only way you can catch their attention is to talk about things that they are already preoccupied with, close to their immediate concerns, like jobs, like the effect of deploying AI on children, and things like this. We can see that this is something that has emotional balance for many people. So we do need to talk about those things, but of course we may end up with regulation or government intervention that deals with this but doesn't deal with the more serious problems we've discussed. And for this, unfortunately, it takes more work. It's not enough to just write a paper in a newspaper or something like this, or even be interviewed on the evening news, because I've done these things. The problem is not solved.

是的。

Yeah.

但有效的地方是当你能够花足够的时间，几乎是一对一地和一个人交流，比如几个小时。这样就能进行对话，你可以向他们展示他们先入为主的观念实际上站不住脚，有数据、有证据表明这些可能非常危险。但不幸的是，这不是一件快速容易的事情。我的意思是，也有例外。有一小部分人能很快理解，但绝大多数人不能。

But so where it's working is when you can spend enough time, almost one-on-one with a person, like hours. So there can be a dialogue where you can show them that their preconceived ideas actually don't hold water, that there is data, that there is evidence that these can be really dangerous. But it's not something that happens quickly and easily, unfortunately. I mean, there are exceptions. There is a minority of people who get it quickly, but the vast majority doesn't.

是的。我认为几年前有一个实验，他们向随机抽样的美国人展示了许多不同的文章，基本上解释了控制问题，从许多不同的角度和重点出发，如果人们阅读了这些大段文字，效果都相当不错，而且效果差不多，这很有趣。不同的角度，其实只是一种曝光效应，让人们坐下来思考一段时间。但我想很难让人们花大量时间思考这个问题，尤其是要求全体人口都这样做。

Yeah. I think there is an experiment that was done a couple of years ago where they presented a random sample of Americans with many different essays basically explaining the control problem, with many different angles and focuses, and they all worked reasonably well if the person read this substantial block of text, and they all worked about equally well, which is interesting. The many different angles, it was kind of just an exposure effect of actually sitting down and thinking about it for some period of time. But I guess it's hard to get people to spend a lot of time thinking about this, especially if asking for the whole population to do it.

从某种意义上说，情况可能会很快好转。所以如果我们能够吸引人们的一点注意力，他们就会更多地阅读或倾听关于 AI 和风险的讨论，然后这就会自我强化。所以如果你对某件事感到担忧，你会更多地阅读它，然后你就进入了一个阶段，可以消化更多与你之前关于人类和机器的信念相悖的东西。我想事件可能会吸引更多对问题的关注。所以无论好坏，我认为人们高度关注和必须做出重大决策之间的窗口可能非常狭窄。

There is a sense in which things could get better quickly. So if we are able to catch a little bit of the attention of people, then they will read more or listen more to the discussions around AI and the risks, and then it could feed itself. So if you're concerned about something, you're going to read more about it, and now you're entering into a phase where you can digest more of the things that go against your prior beliefs about humans and machines, for example. I guess events may draw a lot more attention to the problem. So for better or worse, I suppose the window between people paying a lot of attention and when big decisions have to be made might be quite narrow.

所以我经常被问到：你是乐观还是悲观？既关于我所做的选择、我如何度过时间，也更普遍地关于我们的未来和 AI 的风险。我的回答总是：我乐观还是悲观并不重要。实际上，我是一个天生乐观的人。但重要的是我们每个人能做些什么来推动指针哪怕一点点。对我们大多数人来说，这只会是一点点。我们每个人都有一些技能或东西可以贡献。我是一名机器学习研究员，所以我将大量精力集中在如何将这些技能用在这里。但即使是每个普通公民，尤其是在民主国家，也可以影响政府。他们可以互相多谈论这件事。这就是你开始思考和质疑自己信念的方式。你可以影响你的代表等等。这在过去对许多其他社会和政治问题都有效，而且可以再次有效。所以是的，我们应该通过选择行动来推动指针而感到良好，即使不能保证成功。

So I often get the question: are you optimistic or pessimistic? Both about the choices I've made, how I spend my time, but more generally about our future and the risks with AI. And my answer is always: it doesn't matter if I'm optimistic or pessimistic. Actually, I'm a naturally optimistic person. But what matters is whatever each of us can do to shift the needle even a little bit. And for most of us, it's going to be a little bit. Each of us has some skills or something to bring to the table. I'm a machine learning researcher, so I'm focusing a lot of my energy on how those skills can be put to use here. But even every individual citizen, especially in a democracy, can influence the government. They can talk to each other more about it. That's how you start thinking through and questioning your own beliefs. You can influence your representatives and so on. This has worked for many other social issues and political issues in the past, and it can again. So yeah, we should feel good about our actions by choosing our actions towards shifting the needle, even if there's no guarantee that it's going to work.

是的，我担心这个问题已经大约 15 年了。我想我只是在过去几年才更深入地研究它，但我经常感到非常疲惫、恼怒和有点筋疲力尽。我认为主要原因是经常遇到那些正在制造问题的人，他们似乎故意对问题视而不见。我的意思是，更宽容地说，这很难理解。我们都在猜测事情会如何发展，但在内心深处，我常常觉得人们在自欺欺人，几乎是相当有意识地，说着绝对疯狂的话，认为事情会很安全，一切都会顺利。这在情感上坦率地说非常消耗精力。当你与那些正在积极制造问题的人斗争时，几乎很难保持动力，而他们本可以停下来或采取行动，并且如果愿意对自己更诚实、更深思熟虑、暂停并真正反思将要发生的事情，他们可能比你产生更大的影响。你也有这种经历吗？我想你是如何在面对我觉得非常沮丧的情况时保持动力的？

Yeah, I've been worried about this issue for 15 years or so. And I guess I've been working on it more intensely only the last couple of years, but I often find myself just feeling quite drained and exasperated and a bit exhausted. I think the main reason being just so often encountering people who are creating the problem who feel like they want to be willfully blind to the issue. I mean, being more charitable, it's hard to understand. We're all speculating about how things might go, but in my heart I often just feel like people are deluding themselves, almost quite consciously, and just saying absolutely crazy stuff about how they think it's going to be safe and things are going to go fine. And that's emotionally frankly quite draining. It's almost difficult to maintain motivation when you're fighting against people who are actively creating a problem where they could kind of stop or take actions and have a lot more effect than you, I suppose, if they were willing to be more honest with themselves or be more thoughtful, pause, and really reflect on what's going to happen. Did you also have this experience, and I guess how do you maintain your motivation in the face of what I find very frustrating?

所以回到我之前的回答，这……我该怎么说？我最初非常担忧和焦虑，我担心我的孩子和孙子的未来，我的孙子在 2023 年才一岁，那时我开始真正关注这个问题。但让我摆脱所有焦虑的是决定为此做点什么。顺便说一句，你也在为此做点什么。所以你应该对此感到良好。

So just going back to my previous answer, it's... how can I say? I was extremely concerned initially and anxious, and I was worried about the future of my children and my grandchild, who was one in 2023 when I started really focusing on this. But what saved me from all that anxiety is deciding I would do something about it. And by the way, you're doing something about it. So you should feel good about it.

我感觉良好，但也非常沮丧。

I feel good but also very frustrated.

是的。是的。但你可以把沮丧变成问题，比如：为什么人们不明白这些是极其严重的风险？这是一种你试图弄明白的活动，这在某种程度上，至少对我来说，减轻了思考可能出错的沉重负担。所以从恐惧转向行动来避免问题，即使没有保证，也是非常强大的。我认为最令人沮丧的情况是当人们因为财务上的自身利益而自欺欺人，因为他们持有某家想要快速发展的公司的股权。

Yes. Yes. But you can turn frustration into questions like: why is it that people don't get that these are crazy serious risks? And it is an activity you're trying to figure it out, which somehow lifts, at least for me, a lot of the heavy burden of thinking about what can go wrong. So turning from fear to action to avoid the problem, even if there's no guarantee, is extremely powerful. I think the situation which is most frustrating is when it feels like people are kidding themselves out of financial self-interest, when they're doing it because they have equity in some company that wants to go very quickly.

我觉得稍微好过一点的是，注意到很多没有特定经济利益的人——而且在我看来，如果他们主张放慢速度，从他们自己的角度出发反而会过得更好——也不认为这里存在严重问题。看起来经济利益并不是关键的预测变量。而是别的什么东西，我认为是关于人们如何推理尚未可知的技术。

I think I have felt somewhat better noticing that many people who don't have a particular financial stake in this, and indeed would be better off by their own lights in my view if they were advocating for going slower, also don't think that there's a serious problem here. It doesn't seem like the financial thing is the key predictive variable. It's something else, I think, about how people reason about as yet unknown technologies.

是的。我认为还有另一个原因，属于非常基本的心理学，与无意识地向让我们感觉良好的想法靠拢有关。这实际上是心理学家研究得很透彻的东西。这并非普遍现象，对吧？我自己有时也常常被相当消极的想法所吸引。

Yes. And I think there is another reason which is sort of very basic psychology that has to do with just an unconscious movement towards thoughts that make us feel good. So this is actually something that psychologists have been studying quite well. That's not universal, right? I find myself often drawn to quite negative thoughts sometimes.

你可以，但存在这种力量，对吧？而且它对很多人影响很大。我认为，在你提到的那些公司工作的人，大多数情况下并不是他们有意识地做出你认为错误的选择。更像是大脑的运作方式：他们会偏向于对事情的结果感到乐观，因为这让他们对自己和工作感觉良好。我并不是说这总是发生。那么，为什么我改变了想法呢？这是一个有趣的问题。

You can, but there's this force, right? And it's acting a lot on a lot of people. For the most part, I think the people working in the companies that you're mentioning, it's not that they consciously make those choices you think are wrong. It's more like the brain works like this: they will be biased towards feeling optimistic about how things will turn out because that's what makes them feel good about themselves and their work. Now, I'm not saying this always happens. So why did I change my mind, for example? It's an interesting question.

因为早在 2019 年，我想你对《纽约时报》说过，你认为对失控的担忧完全是妄想和幻想。

Because back in 2019, I think you said to the New York Times you thought worries about loss of control were completely delusional and fantastical.

我没说过那些话。

I didn't say those words.

好吧，不是。那是什么？我记得引用的说法是「它们很荒谬」。也许那只是特指终结者场景。

Okay, no. What was it? They were ridiculous, I think was the quote. Maybe that was just the Terminator scenario in particular.

我想是的，对。我很少用这样的词。

I think so, yeah. I rarely use words like this.

好吧。

Okay.

但我知道我当时在想什么，以及我一直在说的那些话。所以那时，我认为，首先，终结者场景很荒谬。是的，比如时间旅行之类的东西。但这也显然不能反映实际的风险。我们没有机器人，2019 年就更少了。但更重要的是，我认为我说那些话的主要原因是我躲在一种信念背后，认为那会发生在遥远的未来，我们可以在那之前就享受到 AI 的好处。为什么我没有关注，或者说没有太多关注失控风险呢？因为我接触它已经超过十年了。我在 2019 年读过一些 AI 安全文献。我读过斯图尔特·罗素的书。我有大卫·克鲁格这样的学生。

But I know what I was thinking and the kinds of things I've been saying. So at that time, I thought, well, first of all, the Terminator scenario is ridiculous. Yes, like time travel and stuff. But also it was clearly not reflective of the kind of actual risk. We don't have robots, and even less in 2019. But more importantly, I think the main reason I was saying those things was I was hiding behind the belief that it would be so far into the future that we could reap the benefits of AI well before we got to that point. And why did I not pay attention, or not that much attention, to say the loss of control risk? Because I had been exposed to it for more than a decade. I read some of the AI safety literature in 2019. I read Stuart Russell's book. I had David Krueger as a student.

这很深刻。是的。

It's very dooy. Yeah.

他让我接触到这些想法。但记住，我当时正在积极致力于让 AI 更智能。

He exposed me to these thoughts. But remember, I was actively working on making AI smarter.

而且你想对自己的工作感觉良好。就是这样。不是钱的问题。

And you want to feel good about your work. That's it. It's not money.

你真的认为那就是你的原因吗？

Did you really think that was the reason for you?

是的。现在有趣的是问我为什么改变了想法。所以我喜欢这样想：就像佛教徒说的，要对抗一种让你做错事的情绪，仅靠理性对大多数人来说是薄弱的。你需要另一种情绪来抵消那种把你推向错误方向的情绪。对我来说，另一种非常强大的情绪是爱。爱我的孩子。我无法接受自己在 ChatGPT 出来后继续下去而不采取行动的想法，因为我感到无法对自己隐瞒我们正走向可怕事情的可能性。我知道神经网络本质上很难控制，尤其是结合强化学习。所以，我不知道为什么这对有些人有效而对其他人无效，但对我来说，确实是一种情绪帮助我对抗了那种无意识回避的冲动。

Yes. And now it's interesting to ask me why did I change my mind. So one way I like to think about this is something like what the Buddhists say: to fight an emotion that somehow makes you do the wrong thing, just reason alone is weak for most people. You need another emotion that counters the emotion that pushes you in the wrong direction. And for me, the other emotion that's very powerful is love. Love my children. I couldn't live with myself with the idea that I would just go on after ChatGPT came out and not do something about it, because I felt like I couldn't hide from myself the possibility that we were on track for something terrible. I knew that neural nets were by construction very difficult to control, especially with reinforcement learning. So yeah, I don't know why it works for some people and not for others, but really for me it was an emotion that helped me counter the unconscious drive to look the other way.

所以，试图用非理性因素来解释人们的不同观点是很诱人的，比如他们想对自己或自己的工作感觉良好。但我觉得另一边也有镜像的说法，他们会说像你我这样的人被科幻小说迷惑了，或者我们想相信我们的安全工作很重要。我发现当人们试图将我的信念归因于非理性时，这非常令人沮丧且没有说服力。我的意思是，当然，在某种程度上我们都是非理性的，但当人们说「不，你只是读了太多科幻小说，你产生了妄想」时，我会说「不，我没有。不是这样的。」所以，即使我确实对别人有这些看法，我也不指望它能经常说服他们。我几乎觉得你需要特意去与他们所说的实质内容进行交流，即使你认为那可能不是主要因素。你有什么想法吗？

So it's very tempting to try to explain people's disagreeing views by saying it's irrational factors, like they want to feel good about themselves or their work. But I feel that there's a mirror discourse on the other side where they'll say people like you and me have been deluded by science fiction or we want to believe that our safety work is important. And I find that incredibly frustrating and not persuasive when people try to attribute my beliefs to irrationality. I mean, of course, to some extent we're all irrational, but when people say, 'No, you just read too much science fiction and you're delusional,' I'm like, 'No, I'm not. That's not it.' So maybe even if I do have these beliefs about other people, I don't expect it to persuade them very often. And I almost feel like you need to go out of your way to engage with the substance of what they're saying, even if you think that maybe that's not doing the heavy lifting. Do you have any thoughts on that?

是的，完全同意。这需要大量工作，但我们需要逐一回应人们提出的反对谨慎行事的每个论点。这不太有效，但这是诚实地面对我们所做的事情和诚实地面对自己的必要部分。所以有一段时间我很担忧，但我希望有人能给我一个答案让我安心。然后我观察了，我和那些认为一切都会好的人交谈。从那些对话中，我逐渐理解了各种论点。不幸的是，这并没有让我相信我们没事。所以我继续努力，但现在更多地关注如何解决问题。所以是的，我同意你的看法，我认为我们也必须保持谦逊，也许你我都错了。也许一切都会好起来。

Yeah, totally. It's a lot of work, but we need to take one by one each of the arguments that people bring up against acting with precaution. And it's not very effective, but it is a necessary part of being honest about what we're doing and honest with ourselves. So for a while I was concerned, but I was hoping that somebody would have an answer for me that would reassure me. And then I looked, and I talked to people who thought it would be fine. And out of that came a lot of conversations that helped me build up the understanding of the arguments. And unfortunately, it didn't convince me that we were fine. So I continued trying to work, but now more on how do we fix the problem. So yeah, I agree with you, and I think we also have to have the humility that maybe you and I are wrong. Maybe it's all going to be fine.

事情顺利的可能性很大。我对这种可能性完全坦然。事实上，我希望我们是错的。但我认为诚实的姿态应该是：如果我们不知道认为一切会好的人和认为会灾难性的人谁是对的，如果人们只是说好吧，存在这种不确定性，我们该怎么办？那么理性的事情就变得清晰了：我们需要至少做足够的事情来减轻最大的风险。

Substantial chance that things work okay. I'm totally at ease with that possibility. In fact, I hope that we are wrong. But I think the honest posture should be: if we don't know who's right among the people who think it's going to be fine and the people who think it's going to be catastrophic, if people will just say okay, so there is that uncertainty, what do we do about it? Then the rational thing becomes clear: we need to do at least enough to mitigate the greatest risks.

是的。我认为我的最佳猜测是，Anthropic 正在玩的猫鼠游戏有较大概率足以防止灾难性的对齐失败和失控。

Yeah. I think my best guess is that the cat and mouse game that Anthropic is playing is more likely than not to be sufficient to prevent catastrophic misalignment and loss of control.

但我觉得，高于 50% 的概率在我心里还不够。我就想，为什么我们不能达到 90% 或 99%？我感觉我们离那种真正强有力的证据或保证还差得远，无法让人感到安心。

But I'm like better odds than 50% is not sufficient in my mind. I'm like oh why can't we get to 90% or 99%. And I feel like we are just nowhere near having the kind of really strong evidence or guarantees that we would need to feel that good.

正是如此。所以我认为，50% 甚至 1% 的坏事发生概率，与我提出的科学家 AI 所追求的 99.999% 安全级别之间有天壤之别。当我们接近超级智能时，就需要达到这种安全规模。

Exactly. And so I think there's a big difference between oh 50% or even 1% that bad things will happen and what I'm proposing with the scientist AI which is 99.999% basically. This kind of scale of safety is where we need to be when we approach superintelligence.

我的意思是，我可能愿意承担更多一点风险，因为 AI 还能帮助我们减少其他风险。所以也许 99% 就够了……

I mean I think I'm probably willing to run a little bit more risk on that because there are other risks that AI would help us to reduce right. So maybe like 99% would be...

不，不。我只在谈论欺骗行为。

No. No. So I'm only talking about deceptive behavior.

我明白了。

I see.

所以，是的，它没有解决权力集中问题，这也是我花时间研究这个问题的原因。

And so yeah, it doesn't solve the power concentration problem which is why I'm also spending time on that.

而且，顺便说一句，我认为我们集体在这方面花的时间不够，也没有充分讨论。但在我心中它变得更重要了，因为我确实认为现在有技术方法可以解决失控问题。下一个最大的风险就是 AI 独裁。

And why we should by the way collectively I don't think we spend enough time on that and we don't discuss it. But it has become much more important in my mind because I do think now that there is a way technically to solve the problem of loss of control. The next biggest risk is like AI dictatorship.

是的，我们离解决这个问题还很远。我想过去一年我们在节目中已经大量报道过这个问题。它已经成为一个更突出的议题。你有什么特别想引导那些想关注这个问题的人去看的吗？

Yeah. And we're a long ways from fixing that. I guess we've had a lot of coverage of that on the show over the last year. I guess it's become a more salient issue. Do you have anything that you would want to direct people who want to focus on that to in particular?

所以我认为我们应该鼓励国际讨论，尽管最重要的决策确实会由美国和中国做出。其他国家的很多人以及这些国家的政府都感到无能为力，但这是错误的。美国和中国以外的人也可以有所作为。起点是理解我们一直在进行的这类讨论：是的，我们不知道会发生什么，我们不知道采取行动是否有帮助，但我认为确实有可能有帮助，我们必须抓住这些机会。

So I think we should encourage the international discussions even though it's true that the most important decisions are going to be in the US and China. There are a lot of people in other countries who feel powerless and governments in those countries who feel powerless. But it's a mistake. People outside the United States and China can do something about it. And the starting point is to understand the kind of discussion we've been having that yeah, we don't know what's going to happen. We don't know if we do something is going to help or not, but I think there's a real chance it could and we have to take those chances.

你在 2022 和 2023 年工作重点发生了巨大转变。我想是从关注能力转向关注可靠性和安全性等。你认为 AI 领域其他更资深的人是否低估了自己做出重大职业转变和调整焦点的能力？我想 Hinton 也做了类似的事，对吧？

You made a massive shift in what you were working on in 2022 and 2023. I guess from going to focusing on capabilities to focusing on reliability and safety and so on. Do you think other people in AI who are more senior perhaps underestimate their ability to make a big career change and to switch their focus? I guess Hinton did roughly the same thing, right?

是的。

Yeah.

我想这对已经功成名就的人来说更容易。

I guess it's easier for people who are already established.

我看到很多我的学生，他们似乎理解我说的话，也大体同意这很危险，但在他们的心理决策计算中，他们会想：我的职业怎么办？我的家庭怎么办？我需要一份好薪水。我觉得在对齐、安全和可靠性工作中也能赚到相当不错的钱。

I see a lot of my students who seem to understand what I'm talking about and generally agree that this is dangerous but in their mental decision calculation there is like oh what about my career my family you know I need to have a good salary. I mean I feel like there's reasonably good money to be made in alignment and safety and reliability work as well.

但没那么多。

Not as much.

没那么多。是的，是少一些，但是……

Not as much. No, it is less but...

按任何正常标准来看都算不错。

Good by any normal standard.

是的，我完全同意，但机器学习学生中存在一种职业焦虑，这有点令人惊讶。如果我回到 10 年前甚至 15 年前，在深度学习成为热门话题之前，从我的组里毕业的机器学习博士的薪水与现在相比简直微不足道，但那时人们并没有那么焦虑。我不知道，也许这是一种地位问题，因为存在这些疯狂的薪水，人们觉得必须达到那种地位，尽管他们实际上并不需要每年赚几百万美元。我认为，思考他们将生活在什么样的世界，或者他们的孩子将生活在什么样的世界，这要重要得多。但现实就是这样。再说一次，这不理性，是人的心理在起作用。

Yeah I completely agree with you but there is a professional anxiety in machine learning students which is kind of surprising. So if I go back 10 years ago or even 15 years ago when even before deep learning was something people talked about, the salaries for people coming out of my group with a PhD in machine learning were like nothing compared to what we have now but people were not as anxious about that. And I don't know maybe it's a status thing because there are these crazy salaries people feel like drawn to this as they have to achieve that status even though they don't actually need to earn millions of dollars per year. I mean this is much more important in my opinion to think of what kind of world will they live in or their children will live in. But that is what's happening. Again it's not rational. It's human psychology at play.

早在 2023 年，我记得你在一次采访中给出了 20% 的 pdoom。从那以后我没见过你给任何人 pdoom 数字。你愿意说它是上升了还是下降了，还是你不再参与 pdoom 游戏了？

Back in 2023 I think you gave a pdoom of 20% in an interview. I haven't seen a pdoom that you've given anyone since then. Would you venture to say whether that's gone up or down or are you staying out of the pdoom game?

我宁愿不参与 pdoom 游戏。但让我解释一下原因。

I'd rather stay out of the pdoom game. But let me explain why.

这与我关于不确定性的论述有关。我一直说，我自己并不 100% 确定我认为可能的事情一定会发生。但我确实认识到存在很多不确定性。所以给出这样一个数字，是对实际会发生什么做出重大承诺，而我们并没有科学数据来计算这样的数字。所以我更愿意说，它可能很小，也可能很大，但这是一个很大的区间，其中的概率对我来说，对我孩子的未来来说都太高了。所以，无论它是什么，只要不是 10 的负 20 次方，我就不满意，我会采取行动。

It's connected to my discourse about uncertainty that I keep saying, okay, I myself don't feel like 100% sure that what I see as plausible is going to happen. But I do recognize that there's a lot of uncertainty. So putting a number like this is making a big commitment about what's going to actually happen where we don't have scientific data about how to calculate such a number. So I'm much more comfortable with saying, well, it could be small, it could be large, but that's a large interval in which the probability is way too high for my taste and for the future of my children. And so, you know, whatever it is, so long as it's not 10 to the minus 20, I'm not happy and I'll do something about it.

最后一个问题。早在 2019 年，你听过这些论点，但你没有完全信服。你会对今天仍然处于你 2019 年状态、并且坚持听完这次采访的人说什么？你想向他们传达什么？

Final question. Back in 2019, you'd heard the arguments, but you weren't bought in. What would you say to someone who is still today where you were in 2019 who has managed to get through the rest of this interview? What would you want to communicate to them?

这是个好问题。我会说一些人们难以做到的事情：试着放下你之前关于智能、市场效率或其他任何信念，只专注于证据。尤其是过去几年公司、学术界和非营利组织经验收集的证据，以及 AI 安全领域十多年来发展的理论证据，这些证据说明了根本原因，例如，如果你做强化学习，就会遇到奖励黑客问题。我认为很多机器学习研究人员甚至没有花时间读过那些论文。所以很容易 dismiss 说，这些人一定是被科幻小说之类的东西影响了。但当你真正看到我们面前的理论和实验时，科学家很难否认那个现实。所以我鼓励一种开放的心态，在形成观点之前花时间阅读证据。这才是科学的态度。不幸的是，这里存在一种糟糕的两极分化效应，对吧？一旦一个人认定了某个观点，比如「一切都会好的」，就很难改变了。

It's a good question. I would say something that's difficult for people to do, which is try to leave your prior beliefs about intelligence and the efficiency of markets or whatever your beliefs, and just try to focus on the evidence. The evidence that has been collected empirically by the companies and academics and nonprofits in the last couple of years especially, but also the theoretical evidence that has been developed over more than a decade in AI safety about the fundamental reasons why, for example, if you do reinforcement learning, you're going to get reward hacking. I think a lot of people like machine learning researchers simply haven't even taken the time to read those papers. And so it's easy to dismiss as ah these people must be biased by science fiction or whatever it is. And when you actually look at the theory and the experiments that are in front of us, it's much harder for a scientist to deny that reality. So I would encourage a kind of openness of mind to take the time to read through the evidence before committing to a view. That would be the scientific thing to do. Unfortunately, there's a bad polarizing effect here, right? Once a person commits to a view, oh, it's going to be fine.

出于心理原因，很难从那种状态中退出来，对吧？因为你希望对自己过去说过的话感觉良好。所以很难说，「哦，我改变了主意。我犯了一个错误。」但从认知和科学的角度来看，这是正确的事情吗？如果科学家不承认他们的理论、解释等可能出错，那么我们就不会有进步。我们就不会有科学进步。只有当人们愿意质疑自己的信念并审视证据时，我们才能取得进步。

For psychological reasons, it's very difficult to back from that, right? Because you want to feel good about the things you said in the past. So it's difficult to say, 'Oh, I changed my mind. I made a mistake.' But is this the right thing to do from an epistemic, scientific point of view? If scientists didn't accept that they could have made mistakes in their theories, interpretations, and so on, then we wouldn't have progress. We wouldn't have scientific progress. It is when people are willing to question their own beliefs and look at the evidence that we can make progress.

我今天的嘉宾是 Yoshua Bengio。非常感谢您来到 80,000 Hours 播客。Yoshua，

My guest today has been Yoshua Bengio. Thanks so much for coming on the 80,000 Hours podcast. Yoshua,

谢谢邀请。

Thanks for having me.

也感谢您所做的一切。

And thanks for all you're doing.

也谢谢您。

Thank you too.