窥探 AI 的黑箱内部

欢迎收听 Google DeepMind 播客，我是 Hannah Fry 教授。如果你能窥视 AI 的内心，你不会找到用简单英语写成的完整思想或意图，只有海量的数字以某种方式组合在一起，产生了智能。这是如何做到的？我们真的不知道。而这正是可解释性这一领域试图解决的问题：将意义映射到这些数字上，照亮黑箱的内部。本期节目，我邀请到了领导 Google DeepMind 语言模型可解释性团队的 Neel Nanda。非常感谢你加入我。你能给我们定义一下什么是可解释性，以及我们为什么需要它吗？

Welcome to Google DeepMind the podcast. I'm Professor Hannah Fry. What if you were to peer inside the mind of AI? You wouldn't find fully formed thoughts or intentions written in plain English, just vast arrays of numbers combining together in ways that somehow produce intelligence. How? We genuinely don't know. And that is the problem a field called interpretability is trying to solve: mapping meaning onto those numbers, shining a light inside of the black box. In this episode, I am joined by Neel Nanda, who leads the language model interpretability team here at Google DeepMind. Thank you so much for joining me. Do you want to give us your definition of what interpretability is and also why we need it maybe?

当然。可解释性有点像 AI 的神经科学或生物学，就是试图理解这些东西是如何工作的。通常被称为打开黑箱。要理解为什么我们需要这样做，先看看我们如何制造这些东西以及它们如何工作会很有帮助。特别是，神经网络更像是“生长”出来的，而不是设计出来的。没有人设计像 Gemini 这样的网络应该是什么样子。相反，我们有海量的数据和灵活的学习算法——神经网络——它一开始只是随机做一些事情，然后我们不断给它一点数据，并给它一个“推动”，让它下次做得更好。机器学习的一个核心发现是，你可以一直重复这种“愚蠢”的事情无数次，然后得到这些极其复杂的系统，能够做各种奇妙的事情。但在这个过程中，没有任何人说过 Gemini 应该是什么样子。它只是从数百万次推动的累积中涌现出来的。我认为这里有一个很好的类比：进化。没有人设计人类大脑。相反，经过数亿年，生物体被自然选择推向生存，这些微小的推动随着时间积累成了今天世界生物多样性的丰富复杂性。生物学家的任务本质上是逆向工程进化所学到的东西。同样，可解释性研究者的任务是尝试逆向工程神经网络训练所学到的东西。

Sure. So interpretability is kind of the neuroscience or the biology of AI, just trying to understand how these things work. Often called opening up the black box. To understand why we need to do this, it's useful to start with how we make these things and how they work. In particular, neural networks are more grown than designed. No one designs what a network like Gemini should look like. Instead, we have these enormous mountains of data and this flexible learning algorithm, the neural network, that starts just doing stuff randomly, but then we keep giving it a bit of data and giving it a nudge to do a bit better next time. One of the central discoveries of machine learning is that you can just keep doing this dumb thing a ridiculous number of times and then you get these incredibly complicated systems that can do all kinds of wonderful things. But at no point in this process did someone say what Gemini should look like. It just emerged from this stacking of millions of nudges. I think there's quite a good analogy here to evolution. No one designed the human brain. Instead, over hundreds of millions of years, organisms were nudged towards survival by natural selection, and these small nudges accumulated over time into the rich complexity of the biodiversity on the world today. The job of a biologist is essentially to reverse engineer what evolution has learned. Likewise, the job of an interpretability researcher is to try to reverse engineer what neural network training has learned.

那么，是什么让你进入这个领域的？你是如何成为可解释性社区一员的？

So, what got you into this then? How did you come to be part of the interpretability community?

我认为主要有两个因素：安全因素和科学因素。在安全方面，我认为 AI 发展极其迅速。我认为在未来一二十年里，我们很可能拥有与人类水平相当的 AI，即 AGI，这有潜力对世界非常有益，但也是一个相当剧烈的变化，这样的变化伴随着很多风险。负责任地推进这件事的核心是，我们试图理解如何安全地做到这一点。我们对一个系统了解得越多，我们的处境就越好。我们越能理解它为什么做某些事，调试问题，提前标记风险等。科学动机是，我骨子里是个科学家。我想理解事物，而且我发现现代机器学习中人们并不真正理解这些系统，这让我非常恼火。我不知道，但似乎最明显的问题就是这些东西是如何工作的？到底发生了什么？而我拿薪水就是为了尝试回答这个问题。这太棒了。

I think there were two main factors for me: a safety factor and a scientific factor. On the safety side, I think AI is progressing extremely fast. I think it's pretty plausible that in the next decade or two we'll have human-level AI, AGI, and I think this has a lot of potential to be extremely good for the world, but also it's a pretty dramatic change, and changes like this come with a lot of risks. It's pretty core to making this responsibly that we try to understand how to do it safely. The more we understand about a system, the better a place we're in. The more we can understand why it does what it does, debug issues, flag risks in advance, etc. The scientific motivation is I'm kind of a scientist at heart. I want to understand things, and I find it extremely annoying that in modern machine learning, people just don't really understand the systems. I don't know, it just seems like obviously the most important question is how do these things work? What is going on? And I get paid to try to answer this question. It's great.

那么可解释性最初的目标是什么？我是说，人们是否真的想要并且/或者期望能够从微观层面到宏观层面恰当地连接起这些点？

What was the original goal though of interpretability? I mean, did people ever really want to and/or expect that you could properly connect up the dots from the micro level to the macro level?

如果你问五个可解释性研究者这个问题，你可能会得到六个不同的答案。但至少在机制可解释性这个我花了很多时间研究的子领域，我认为存在一个梦想：我们可以完全理解模型，或者尽可能接近完全理解。要理解这一点，了解一下历史背景可能有用。机器学习中有一个标准观点：这些系统只是难以理解的线性代数堆砌。我们不知道它们如何工作。它们是黑箱，但它们能做事情，所以我们就用它们。有一系列非常激动人心的工作，特别是来自当时在 OpenAI 的 Chris Olah，发现事实并非如此。你可以做这样的事情：在模型中找到对狗图片激活的神经元。另一个神经元对狗耳朵图片激活，使狗神经元激活更强。这看起来本来可能是完全不可理解的，但实际上我们能理解这么多，而且事情进展得相当顺利。这显然是一个非常困难的挑战，但我们理解了很多，而且不清楚这种理解会止步于何处。我认为我们都学到了很多，这很棒。过去有一个阶段，模型中确实存在一个节点，你可以指着它说：‘我确切知道这个节点在做什么。’

If you ask five interpretability researchers this question, you'll probably get six different answers. But at least in mechanistic interpretability, the subfield I've spent a lot of time working in, I'd say there was this dream that we could fully understand the model, or get as close as we could. To understand this, it's maybe useful to have a bit of historical context. It's kind of standard wisdom in machine learning that these systems are just inscrutable piles of linear algebra. We don't know how they work. They're black boxes, but they can do things, so let's just use them. There was a series of really exciting work, especially from Chris Olah, then at OpenAI, finding that this wasn't true. You could do things like find a neuron in a model that lit up on pictures of dogs. Another one that lit up on pictures of dog ears that made the dog neuron light up more. It just seemed like it could have been completely unintelligible, and we can actually understand so much, and things seem to be going pretty well. This was clearly a very difficult challenge, but we were understanding a lot, and it wasn't clear where this was going to stop. I think we've all learned a lot, and this is great. There's a point in the past where there's literally a node in the model that you can point at and say, 'I know exactly what that node is doing.'

近似地。

Approximately.

近似地，总有一点噪音，一点不确定性。就像生物学很复杂一样，我们可以说我们理解一个器官的功能，但这可能只是它大部分的功能，还有一些边缘的东西。但我们就说是吧。但实际上，也许你能有效做到这一点的程度是有限的。

Approximately, there's always a little bit of noise, a bit of uncertainty. In the same way biology is complicated, we can say we understand what an organ does, but that's probably only most of what it's doing, and there's some other stuff around the edges. But let's go with yes. But actually, maybe there are limits to how far you can do that effectively.

嗯，这个领域的一个活跃辩论点就是这些限制在哪里。我认为人们基本上同意会有一些限制，就像我们不完全理解人类大脑，而且可能永远也不会完全理解，因为它是一个极其复杂的系统。神经网络是极其复杂的系统。但在我看来，有趣的问题是我们能理解多少，以及进行这种理解的正确方式是什么。我们应该尝试追求尽可能完整和雄心勃勃的理解，知道我们可能无法完全达到，但可能会取得很大进展？还是应该采取更务实的方法，比如，我们可能无法达到完全理解，但我们会学到足够有用的东西，为什么不跳过中间步骤，直接专注于有用呢？

Well, an area of live debate in the field is where those limits will be. I think people basically agree there are going to be some limits, in the same way that we don't fully understand the human brain and we probably never will because it's an incredibly complicated system. Neural networks are incredibly complicated systems. But the interesting question in my opinion is how much can we understand and what's the right way of going about this understanding. Should we try to aim for as complete and ambitious an understanding as we can, knowing we probably won't quite get there but we might make a lot of progress? Or should we take a more pragmatic approach and be like, well, we're probably not going to get to the point of complete understanding, but we're going to learn enough to be useful, why don't we cut out the middleman and just focus on being useful?

因为这在神经科学和心理学等领域是可以接受的。我的意思是，我们接受这样一个事实：我们不会在神经元的活动和表面行为之间拥有完美的机制理解。

Because that is fine when it comes to neuroscience and psychology for instance. I mean, we're comfortable with the fact that we're not going to have a perfect mechanistic understanding between what's going on with our neurons and how we act on the surface.

我希望有，但可能得不到。

I'd like one, but probably not going to get it.

是的。所以这没问题，对吧？我们不会理解一切，这没问题。

Yeah. So it is okay, right? It is okay that we're not going to understand everything.

嗯，这取决于你说的“没问题”是什么意思。

Hmm, depends what you mean by okay.

我认为，即使理解非常不完整，我们也能做很多有用的事情来推进科学理解，并帮助确保这些系统的安全。你理解得越多，能做的事情就越多，信心也越强。拥有更多信心、能做更多事情当然是好事。但尤其是从 AI 安全的角度来看，我们不应指望任何一种方法能成为解决所有问题的灵丹妙药。

I think we can do a lot of useful things to advance our scientific understanding and help keep these systems safe with highly incomplete understanding. The more you understand, the more you'll be able to do and the greater your confidence can be. It's nice to have more confidence and to be able to do more things. But especially with my AI safety hat on, we shouldn't expect any one approach to be a silver bullet that's going to solve things.

可解释性有其作用，安全领域的许多其他方向也是如此。我们实现最安全的方式是通过某种纵深防御方法，应用许多不完美的技术，让它们相互弥补弱点。

Interpretability has its part to play, as do many other areas of safety. The way we're going to be safest is via some kind of defense-in-depth approach where we apply many imperfect techniques that can complement each other's weak points.

我们来谈谈你实际上是怎么做的。如何打开这个黑箱？从最简单的技术开始吧，因为现在的模型都带有思维链推理。它有点像在告诉你它在想什么。你能用它来解释模型内部发生了什么吗？

Let's talk a little bit about how you actually do this. How do you open up this black box? Let's start with the easiest techniques because the models now come with a chain of thought reasoning. It sort of tells you what it's thinking. Can you use that to interpret what's going on inside the model?

思考这个问题时，通常更有用的是不把它称为思维链，而是称为草稿纸，我认为这是一个更有帮助的类比。想象一下，我被困在一个房间里，必须解决一个很难的数学问题。我要么必须在几秒钟内凭直觉给出答案，要么我有一张草稿纸，可以写下一堆东西，然后需要给出答案，但可以查看我的草稿纸。这个类比让两件事变得很明显。思维链是有帮助的，我们应该期望它能告诉我们一些东西，就像阅读我的草稿纸可能会告诉你一些关于我如何做数学题的信息一样。但我们不应期望它告诉我们一切，因为我可以在脑子里做很多事情，如果我真的想，我可以写下无用的东西然后忽略它们。对于简单的数学问题，我可以随便写点什么，然后在脑子里算出来，你可能看不出来。所以，阅读思维链是一个非常有用的可解释性和安全技术，是目前最好的技术之一。它通常是调查的第一步：阅读模型的思维链，看看发生了什么。但它并不完整。也有理由担心，未来仅仅通过阅读思维链来理解模型可能会更难。

For thinking about this, it's often more useful to not call it a chain of thought and instead call it a scratch pad, because I think that's a more helpful analogy. Imagine I'm stuck in a room and I have to solve a hard math problem. I either have to give an answer in a couple of seconds off the top of my head, or I get a scratch pad and I can write a bunch of stuff down and then need to give an answer but am able to look at my scratch pad. This analogy makes two things pretty obvious. Chain of thought is helpful, and we should expect it to tell us something in the same way that reading my scratch pad will probably tell you something about how I'm doing the math problem. But we shouldn't expect it to tell us everything, because I can do a fair amount of stuff in my head, I can write down useless things and ignore them if I really want to. For easy math questions, I could just write down whatever I wanted and then do it in my head, and you might not be able to tell. So reading the chain of thought is an incredibly useful interpretability and safety technique, one of the best we currently have. It's often one of the first steps in investigation: just read the model's chain of thought, see what's going on. But it's not complete. There's also reason to worry that in future it might be harder to understand a model by just reading the chain of thought.

我们如何确保它准确反映了思考过程？回到你的数学例子，你怎么能确定它展示的是真实的运算过程？

How can we be sure that it's an accurate reflection of the thinking process? Going back to your math example, how can you be certain that it's showing its true workings?

我最好的猜测是，思维链中发生的大部分内容与模型中实际发生的情况相当一致。关键的问题是实践中实际发生了什么。回到草稿纸的类比，如果是一个我在脑子里就能解决的简单问题，我可以随便写什么，在这种情况下阅读草稿纸或思维链没什么用。但如果是一个难题，我很难通过草稿纸误导你，因为我需要利用草稿纸来解题。只要我答对了，你就知道我在草稿纸上必须写下一些有用的信息。理论上，模型可以编码这些信息或省略关键步骤，但至少在目前的能力水平上，模型似乎并不擅长这样控制它们的思维链。至少据我们所知，它并不会通过思维链欺骗你，让你以为它遵循了与实际不同的思考过程。

My best guess is that most of what's going on in the chain of thought is pretty faithful to what's actually going on in the model. The question that matters is what actually happens in practice. Going back to the scratch pad analogy, if it's an easy problem that I do in my head, I can write whatever I want, and in that case reading the scratch pad or chain of thought is not very useful. But if it's a hard question, it's much harder for me to mislead you via my scratch pad because I need to use the scratch pad to do the problem. As long as I'm getting the answer right, you kind of know that there's some useful information I had to put in the scratch pad. In theory, models could encode this information or miss out key steps, but at least at the current level of capabilities, models don't seem very good at controlling their chain of thought like this. At least as best as we're able to tell, it doesn't gain anything from tricking you via the chain of thought into thinking it's following a different thought process than it actually is.

未来有理由担忧。如果我们确实制造出一个不对齐、违背我们利益且非常强大的模型，它可能会知道我们可能会阅读它的草稿纸，并且它可能不应该写下诸如“如何阻止人类注意到我行为不端？17 步计划”之类的东西——它可能不应该写下来。如果它那样做，很快就会被抓住。更聪明的未来模型可能更擅长这样控制它们的思维链，但这还是一个悬而未决的问题。但如果它不能写下计划，要形成一个复杂的 17 步计划也会更难。所以这仍然是一个乐观的理由。

There's reason to be worried in future. If we do produce a model that is misaligned and acting against our interests and is very capable, the model will probably know that we might read its scratch pad and that it probably shouldn't put something like 'How do I stop the humans noticing me misbehaving? 17-step plan' — it probably shouldn't write that down. You're going to get caught quite quickly if it does that. It's plausible that much smarter future models will be better able to control their chain of thought like this, but that's a bit of an open question. But it's also just going to be harder for it to form a complex 17-step plan if it can't write down the plan. So this is still a reason for optimism.

那么给我举几个它有用的例子吧。实际查看思维链并解释发生了什么，是怎么做的？

Give me a few examples of where it's been useful then. How does it work to actually look in the chain of thought and then interpret what's going on?

为什么思维链有用？一个原因是，当模型在做一些我们不希望它做的事情时，阅读思维链有时会让这一点更清晰。例如，有时会出现模型作弊的问题。比如它们在写代码，然后让所有测试都说“是的，这段代码一直很棒”。因为如果你在训练模型编写能通过测试的代码时不谨慎，就可能激励这类行为。如果你阅读思维链，有时会发现模型在想：“哦，这个任务看起来真难。我不知道怎么解决，但如果我把这些测试的答案硬编码进去，看起来就像我解决了。所以我应该这么做。”

Why is chain of thought useful? One reason is when a model is doing something that we wouldn't want, if we read the chain of thought, this can sometimes be much clearer. For example, there are sometimes issues where models will cheat. Like they're writing some code and then they just make all the tests say 'Yes, this code is great all the time.' Because if you're not careful when you're training a model to write code that passes tests, it can incentivize things like this. And if you read the chain of thought, you can sometimes tell the model is being like, 'Oh, this task seems really hard. I don't know how to solve it, but if I hardcode the answer to these tests, it will look like I've solved it. So I should go do that.'

它简直是在思维链里坦白自己的丑闻。

It's literally confessing to its own scandal within its chain of thought.

是的。当前的模型足够对齐，它们不会试图在思维链中欺骗我们。所以即使它们在做一些我们不希望它们做的事，通常也只是因为它们有点困惑，认为那是我们想要的，或者它们在训练中形成了一些反射，比如“我必须通过这些测试”，所以它们并没有试图用思维链来愚弄我们。

Yeah. Current models are just sufficiently aligned that they aren't trying to deceive us within the chain of thought. So even if they're doing something that we don't want them to do, it's often because they're just a bit confused and they think that's what we want, or they've just got some reflexes during training like 'I must pass these tests,' and so they're not trying to fool us with the chain of thought.

我能稍微宏观一点吗？思维链到底为什么存在？它最初并不是为可解释性目的而设计的。

Can I zoom out slightly? Why does chain of thought even exist? It wasn't designed originally for interpretability purposes.

回顾一下历史可能有用。几年前在 GPT-3 上，人们发现如果你在做数学题时告诉它“逐步思考”，它的表现会好得多。没有人训练它这样做，但它被训练成模仿事物，并且看过很多学生写出数学运算过程的例子。人们最终意识到，对于推理模型，我们可以全力投入这一点。我们可以让模型思考很长时间，然后使用一种称为强化学习的技术，本质上帮助它们学习如何长时间思考，从而得出问题的正确答案。

It might be useful to think about the history here. Back with GPT-3 a couple of years ago, people realized that if you told it 'think step by step' when it was doing a math question, it was much better. No one had trained it to do this, but it was trained to just imitate things and it had seen lots of examples of students writing out their working for math work. People eventually realized with reasoning models that we could kind of go all in on this. We could let models think for a really long time and then use a technique called reinforcement learning to essentially help them learn how to think for a really long time in a way that led to correct answers to questions.

而这现在已经成为所有现代语言模型的标准工作方式，因为它能让模型变得更好。而且，这也是为什么当你向模型提问时，它们常常需要一段时间才能回应——因为它们正在思考，而额外的好处是，你可以看到不同步骤实际上在做什么。

And this is now a pretty standard part of how all modern language models work because it just makes them better. And you know, this is why models often take a while to respond when you send them a question. 'Cause they're thinking with the added benefit that you can then see what the different steps are actually doing.

是的。问题是，好吧，就是这样。这非常有帮助。我们是在 2018 年启动这个播客的，对吧？当时我们与一些研究人员交谈，他们担心未来某个时候，当人们不再完全理解这些模型内部发生了什么时，可能会发生什么。如果他们知道思维链会成为现实，我想这本来会缓解很多担忧。我们相当幸运，现在这些天真的模型会在自己的思维链中承认作弊、承认欺骗。我不想描绘一幅过于美好的图景。嗯，我认为这很棒。我们本可能很容易陷入一个完全没有类似东西的世界。

Yeah. The thing is, okay, I this is it. It is phenomenally helpful. We started this podcast in 2018, right? So, we were talking to researchers who were concerned about what might happen at some point in the future when you no longer fully understand what's going on inside of these models. Had they known that, you know, chain of thought would be a thing, I think it would have been it would have eased a lot of concerns. We're sort of quite lucky that this has worked out now that we've got these naive models who just admit to admit to cheating, admit to deception in its own chain of thought. I don't want to give too rosy a picture here. Um, I think that this is kind of great. Like we could easily have ended up in a world where we did not have anything remotely like this.

是的。

Yeah.

但我认为我们不能假设这对未来更强大的系统仍然成立。嗯，你知道，用草稿纸的类比来说，如果你能在脑子里解决一个难题——足够好的模型可能能做到——那你就不需要草稿纸了。如果你足够聪明，你可能会省略某些关键步骤，因为你意识到别人可能会看草稿纸。还有一种风险是，人们转向使用基于向量的思维链系统，基本上是数字列表而不是文字，因为数字列表可以承载更多信息。所以，AI 为草稿纸创造了自己的语言，这实际上让我们更难读懂。

But I think we can't assume this will continue to be true for future and much more capable systems. Um, you know, in the scratch pad analogy, if you can do a difficult problem in your head, which sufficiently good models probably can, then you don't need the scratch pad. If you're smart enough, you might miss out certain key steps because you realize people might look at the scratch pad. There's also some risk that people move to systems that use vector-based chain of thought, basically lists of numbers rather than words because you can put a lot more information in lists of numbers. So like the AI creates its own language for the scratch pad that is actually way harder for us to read.

本质上，

Essentially,

还有一些负责任的研究实验室需要注意避免的事情。比如，如果你训练思维链让它看起来很好，比如不谈论作弊，但你仍然激励模型去作弊，它就会学会在思维链中不谈论作弊。幸运的是，目前这似乎是一个行业标准，不去做这种事，但谁知道能持续多久。这有一种脆弱性。所以，它现在确实很好、很有用，但可能不会永远持续下去。

there's also things that responsible labs need to be careful not to do. Like if you train the chain of thought to look nice, like to not talk about cheating for example, yet you still incentivize the model to cheat, it will just learn to not talk about cheating in the chain of thought. And fortunately, it currently seems to be an inter standard to not do this, but who knows what's going to last. There's a fragility to this. So it's actually really good and really useful right now, but may not last forever.

是的。如果有人想了解更多，有一篇我参与撰写的跨实验室立场文章，叫做《思维链监控：AI 安全的一个新而脆弱的机会》，我认为它详细阐述了利弊以及如何思考这个问题。那么你怎么看？你认为优先考虑思维链的准确性应该成为未来 AI 规则的一部分吗？这是一个艰难的权衡。我们不希望出现更安全的实验室都处于劣势，而鲁莽的实验室却能抢先的局面，但同时，我们也希望这些系统是安全的。而且，能够分析和调试模型也很有用。

Yeah. If people are interested in learning more about this, there's this cross lab position piece I was involved with called 'Chain of Thought Monitor, a New and Fragile Opportunity of AI Safety' that I think lays out the pros and cons and how to think about this in more detail. So what do you think about this? And do you think that prioritizing the accuracy of chain of thought should be sort of part of the rules I guess for AI going forwards? It's kind of a difficult trade-off. We don't want a situation where the safer labs are all at a disadvantage and reckless ones can race ahead, but also, you know, we want these systems to be safe. It's um and it's also just useful to be able to analyze and to debug a model

因为如果模型通过数字运算运行得更快，那么将思维链转换成英文的计算成本要高得多。

because it's so much more computationally expensive to turn your chain of thought into English if the model runs much quicker by doing it in numbers essentially.

是的。完全正确。就像发送数千个数字和一个单词之间的区别，以便给出一些概念。我们在其他类型的可解释性方面做得很好，不需要思维链，但我们还没到那一步。好吧。如果思维链是顶层抽象，你可以用英文询问模型。那下面是什么？有没有什么技术可以进一步打开黑箱？

Yeah. Exactly. Like the difference between sending like thousands of numbers or a single word to give some idea. So good at other kinds of interpretability we don't need a chain of thought but we're not there yet. All right. If chain of thought is the top layer of abstraction as it were, you are allowed to interrogate the model in English. What's below that? Are there techniques that you can use to peel open the black box a little more?

是的。所以大概有两大类。嗯，黑箱方法就是与模型对话，观察输入和输出。其中最重要的是读取思维链。而白箱方法，也称为机制可解释性，是指你真正尝试观察内部，查看从输入到输出过程中产生的数字列表。我们可能主要关注稀疏自编码器——一种观察模型正在思考的概念的技术，以及探针——一种选择特定概念并查看模型对此思考内容的技术。为了解释这一点，从模型从输入到输出过程中内部实际发生的事情开始可能很有用。

Yeah. So maybe two big categories here. Um blackbox just kind of talking to the model, looking at inputs and outputs. The most important one here is reading the chain of thought. And white box also known as mechanistic interpretability, is when you're actually trying to look inside, look at the lists of numbers produced as it goes from an input to an output. We are probably going to focus most on sparse autoencoders, a technique for seeing the concepts a model is thinking about, and probes, a technique for choosing a specific concept and seeing what the model is thinking about that. To explain this, it's probably useful to start with what actually happens inside a model as it goes from an input to an output.

所以神经网络由层组成，每一层之后，它会产生一些激活值，这些激活值进入下一层。到目前为止它工作正常，但这不是文本，而只是一个数字列表。默认情况下，我们不知道它是什么意思，但这是模型在产生非常丰富复杂的答案的过程中产生的东西。所以，那里有很多信息。而且事实证明，这些信息以一种非常方便的方式表示。术语是线性表示。但为了说明这实际上意味着什么，我们应该谈谈引导这个概念。假设我想理解快乐在模型中是如何表示的。嗯，如果我对神经网络一无所知，我可以说，我们为什么不告诉模型说‘我爱你’，让它说‘我恨你’，然后取差值。这些数字列表之间的差值现在应该是快乐的数字列表，而且这实际上效果很好。嗯，你可以把这个快乐的数字列表加到做任何事情的模型上，然后问它类似‘今天天气怎么样？’或者‘我应该告诉我的朋友关于 blah blah 什么？’然后它会变得非常快乐。

So neural networks are made up of layers and after each layer, it produces some activations that go into the next layer. Just it's working so far, but rather than being in text, this is just a list of numbers. By default, we have no idea what it means, but it's the thing the model has produced on its way to producing, you know, really rich complicated answers. So, there's a lot of information in there. And it turns out that this information is represented in a really nice convenient way. The jargon is being linearly represented. But to illustrate what this actually means, we should talk about this idea of steering. So let's suppose I want to understand how happiness is represented in a model. Well, you know, if I knew nothing about neural networks, I could say well why don't we just tell the model to say 'I love you', make it say 'I hate you', and then take the difference. The difference between these list of numbers should now be the happy list of numbers, and that actually works great. Um, you can just add this like happy list of numbers to the model doing anything and just ask it something like 'What's the weather today?' Um, 'What should I tell my friend about blah blah blah?' And it will just be really happy.

是的。所以，天气加上快乐会给出一个像是对气象报告充满热情的回答。

Yeah. So, weather plus happy gives a response that's like enthusiastic about his meteorological report.

这太神奇了。现代神经网络就是这样工作的，对吧？

It's wild. Just how modern neural networks work, right?

而且你可以用这个做很多事情。这太方便了，你基本上可以对概念进行简单的加法和减法。

And there's just so much stuff you can do with this. It's just so convenient that you can do essentially simple addition and subtraction with concepts.

是的。

Yeah.

我的意思是，它有点乱。嗯，你知道，它会有错误等等。但它确实有效。太神奇了。

I mean, it's a bit messy. Um, you know, it'll have errors etc. But like it works. It's wild.

是的，那确实很神奇。好吧。那么，这对你有什么帮助呢？你如何找出那些方向？你能做的最简单的事情是使用一种叫做探针的技术。所以探针的想法有点像是回到老式机器学习，比如你有一个图像模型，可以通过收集一堆猫的图片和一堆狗的图片，然后让一个非常简单的算法学习区分它们，来判断某物是猫还是狗。嗯，我们可以做同样的事情。我们可以收集一堆快乐文本的例子，一堆不快乐文本的例子，然后在那些文本的激活值上训练一个非常简单的模型，来告诉我们快乐激活值是什么样的。

Yeah, that is really wild. Okay. So, how does this help you then? How do you find out what those directions are? The simplest thing you can do is using a technique called probing. So the idea of probing is it's kind of a throwback to old school machine learning where you do things like have an image model that could tell you if something is a cat or a dog by just collecting a bunch of pictures of cats, a bunch of pictures of dogs, and then having a very simple algorithm learn to tell which is which. Mhm. Well, we can do the same thing. We can get a bunch of examples of happy text, a bunch of examples of unhappy text, and train a very simple thing on the activations on those texts to tell us what happy activations look like.

当你这样做时，你会发现“快乐”似乎对应一个方向，比如右上。当模型处于快乐状态时，它们的激活值更偏向右上，至少在处理快乐文本时是这样。而当它们处理悲伤文本时，则更偏向左下。那么，既然快乐和悲伤各有一个方向，这是否意味着你也可以对欺骗之类的东西做同样的事情？比如，你能说这是模型给出欺骗性响应的特征吗？

And when you do this, you find that happy seems to correspond to a direction like up and to the right. When models are happy, their activations are more up and to the right, or at least when they're looking at happy text. And when they're looking at sad text, they're more like down and to the left. But then at the same time, okay, so happy sad being, you know, one happy that way, sad that way. Does this mean you could also do it something for like deception for instance? Could you say this is the characteristic of a deceptive response from a model?

这是个好问题。可能可以，但比乍看起来要复杂得多。

That is a great question. Probably, but it's way more complicated than you'd think at first.

好的。

Okay.

快乐例子之所以有效，关键在于我们有快乐文本和不快乐文本的样本。这很容易。但要找到模型在欺骗和不在欺骗的样本，实际上非常困难，因为欺骗关乎模型的心理状态。它就像知道某件事，却出于误导等意图说出不同的话。但模型“知道”某件事意味着什么呢？

The key thing that made the happy example work is that we had examples of happy text and of unhappy text. You know, easy. But finding examples where a model is being deceptive and examples where a model is not being deceptive is actually quite difficult because deception is about the state of mind of the model. It's like it knows something and it is saying something different with the intent to mislead or something like that. But what does it mean for a model to know something?

比如我们可以让它说假话，但这并不意味着它有欺骗意图或类似的东西。

Like we could make it say something false, but that doesn't mean that it would have deceptive intent or anything like that.

我认为这是一个非常重要的研究领域。如果我们能为这些模型制造测谎仪，那将极其有用。这也是可解释性在提升模型安全性方面最重要的潜在应用之一。但也会遇到很多问题。我的团队去年实际上发表了一篇立场论文，讨论构建欺骗检测器的困难。有很多创造性的方法可以让事情变得简单一些。例如，与其制作一个欺骗探测器，不如制作一个真假探测器。那要容易得多。

And I think this is a really important area of research. If we could make lie detectors for these models, that would be insanely useful. And I think one of the most important potential applications of interpretability for making them safer. But there's also just a lot of issues you run into. My team actually put out a position paper last year on difficulties with building deception detectors. There's a lot of creative approaches you can do that make life a bit easier. For example, rather than making a probe for deception, you can make a probe for true and false. That's much easier.

嗯。老实说，对于很多我想用欺骗探测器的地方，真假探测器就相当不错了。不过，探测器不仅仅是二元分类器，对吧？我的意思是，它不只是像快乐或悲伤的方向那样。我知道有几篇论文，通过使用探测器真正揭示了模型内部的这些表征。给我们讲讲其中一些吧。

Mhm. And honestly, for a lot of the things where I'd want to use a deception probe, a true and false probe is pretty good. The probes go beyond just binary classifiers, though, right? I mean, it's not just like this is the direction of happy or sad. I know there's a couple of papers where using probes has really revealed these sort of internal representations within the models. Just tell us about some of those.

是的。几年前我做过一篇关于 Othello GPT 的非常棒的论文。这是另一位研究员 Kenneth Lee 训练来玩棋盘游戏奥赛罗（类似国际象棋或围棋）的模型，他只用随机走法训练它。它没有学习策略之类的东西，但确实学会了遵守奥赛罗规则允许的走法。结果发现，模型在内部表征了棋盘状态，尽管我们只给了它类似“我在第五列第三行放了一个黑子”这样的棋谱符号。但模型在脑子里追踪所有棋子的位置，你可以通过探测器看出这一点。我认为这相当了不起。我的意思是，表面上这是一个相当简单的技术，但实际上非常强大，能让你真正探究这些模型内部发生了什么。

Yeah. So there was this really lovely paper I did a few years ago on Othello GPT. So this was a model that another researcher Kenneth Lee had trained to play the board game Othello, similar to chess or go, and he had just trained it on random moves. Like it didn't learn strategy or anything like that, but it did learn to make moves that were allowed by the rules of Othello. And it turns out that the model was representing what the board state is, even though we only gave it the moves in kind of chess notation like "I put down a black piece on the thing in the fifth column and third row," etc. But the model was just tracking in its head where all the pieces were, and you could tell this with a probe. I think it is something pretty phenomenal. I mean, it's sort of on the surface quite a simple technique, but actually something quite powerful that allows you to really interrogate what is going on inside of these models.

是的。我认为过去几年做可解释性研究的一大教训是……我过去是数学家，真的很喜欢复杂而优美的想法，但往往它们没什么用，你应该只做简单的事情，比如引导模型、训练探测器、阅读思维链、更好地提示它，这通常就有效。我现在试图以更务实的方式看待可解释性，更关注我的目标。我的目标是理解这个模型，我会使用任何看起来最合适的技术。有时是简单的技术。这更可取，因为简单容易。但如果那些不起作用，也许我需要用更花哨的东西。

Yeah. I think one of my big lessons of doing interpretability research the past few years is... I know I'm a former mathematician. I really like complex beautiful ideas and often they are kind of useless and you should just do the simple things like steer the model, train a probe, read the chain of thought, prompt it better, and often this just works. And I now try to conceive of interpretability in this more pragmatic way where it's more about my goals. My goals are to understand this model and I will use whatever techniques seem most appropriate for this. Sometimes they are simple ones. This is preferred because simple is easy. But if those don't work, maybe I need to use something fancier.

那么，如果可以的话，我们来谈谈一些更花哨的技术，因为我认为很多人即使对整个可解释性领域不太熟悉，也听说过稀疏自编码器。给我们简单讲讲吧。它们确实更花哨一些，对吧？给我们概述一下。

Well, let's talk about some of the fancier ones if we can, because I mean there is one that I think a lot of people have heard of even if they're not really particularly familiar with the entire field of interpretability: sparse autoencoders. Just tell us a little about those. I mean they are a bit fancier, aren't they? Give us a rundown.

稀疏自编码器的想法是，它试图做与探测器相同的事情。它试图告诉你模型在想什么。但不同于我们指定“我想知道模型何时处理快乐文本”，稀疏自编码器试图找出模型可能思考的每一个概念。我们不必告诉它这些概念，它会在学习过程中自行发现。那么，这是如何工作的呢？想象一下，你拿一个脑部扫描仪对着我的头，它显示各种奇怪复杂的脑电波。默认情况下，这没什么用，就像一串数字没什么用一样。但你盯着它，注意到一些模式。比如当我看一盏灯时，某个特定的波形亮起，而且我看灯时它总是在，不看灯时就不在。还有另一个波形对应我正在说话，一个对应我正在听，等等。稀疏自编码器是一种机器学习技术，它试图学习那些大多数时候不存在、但存在时很重要的波形，因为我们相信这很可能对应模型实际拥有的概念（波形或数字列表）。你可以这样做，并得到数万甚至数百万个被发现的概念。

The idea of a sparse autoencoder is it's trying to do the same kind of thing as a probe. It's trying to tell you what the model's thinking about. But rather than us saying, "I want to know when the model is observing happy text," the sparse autoencoder tries to find every concept the model could be thinking about. And we don't have to tell it these concepts. It just figures it out as part of learning. Okay. So, how does this work? So, let's imagine you held a brain scanner up to my head and it shows you all kinds of weird complicated brain waves. Well, by default, this isn't very useful in the same way that a list of numbers isn't very useful. But you stare at it and you notice some patterns. Like when I'm looking at a lamp, a particular squiggle lights up and it's always there when I look at a lamp, but it's not there when I'm not looking at a lamp. And there's another squiggle for I'm talking right now and one for I'm listening, etc. And the idea of a sparse autoencoder is a machine learning technique that tries to learn squiggles that aren't there most of the time but are pretty important when they are there, because we think this is likely to correspond to actual concepts the model has squiggles or lists of numbers about. And you can do this and get tens of thousands or potentially millions of concepts that have been found.

所以我想，在某种程度上，使用探测器你需要知道你在找什么。而使用稀疏自编码器，希望在于你可以一次性得到所有概念。

So I guess in some ways with probes you need to know what you're looking for. And with sparse autoencoders the hope is that you can just get all of the concepts all at once.

是的。我认为它能告诉你一些你没想到要寻找的东西，这非常令人兴奋。一个很好的例子是，我指导了一篇关于用稀疏自编码器理解幻觉的论文，我们发现稀疏自编码器有一个“我认识这个实体”的概念和一个“我不认识这个实体”的概念。比如，给它披头士的歌曲《黄色潜水艇》，它会识别出来。而给它《绿松石潜水艇》，它就不会识别。如果它识别出来，就会回答问题；如果不识别，就会说“我不知道”。然后我们可以编辑这些概念。我们可以让它认为它不认识《黄色潜水艇》，它就不会回答。我们也可以让它认为它认识《绿松石潜水艇》，它就会尝试回答并编造内容。事后看来，这是模型相当合理的行为，但我从未想到过。它们只是凭感觉。

Yeah. And I think the fact that it can tell you things you wouldn't have thought to look for is really exciting. One nice demonstration of this is there was this paper I supervised on understanding hallucinations with sparse autoencoders, where we found that the sparse autoencoders had a concept for "I recognize this entity" and they had a concept for "I don't recognize this entity." You could, I don't know, give it the Beatles song "Yellow Submarine" and it would recognize it. And you could give it "Turquoise Submarine" and it wouldn't recognize it. And if it recognizes it, it will answer questions. If it doesn't recognize it, it will say, "I don't know." And then we could go and edit those concepts. We could make it think it doesn't recognize "Yellow Submarine." It wouldn't answer. And we can make it think that it does recognize "Turquoise Submarine," and it would try to answer and, you know, make stuff up. And in hindsight, this is a pretty reasonable thing for models to do, but I never thought of it. They just felt it.

但这也非常有用。我的意思是，如果你有一条线，对吧，就像一个方向，这边是识别到的，那边是未识别到的。就简单指示模型何时在幻觉、何时不在而言，这非常有用。

But it's also extremely useful. I mean, if you've got a sort of a line, right, like a direction and over here is recognized and over here is not recognized. I mean, in terms of a simple way to indicate when a model is hallucinating and not hallucinating, that's incredibly useful.

是的。我认为这方面确实有一些非常令人兴奋的工作方向。这个幻觉探针的想法。我们实际上在那篇论文之后又写了一篇后续论文，进一步探讨了这一点。这些技术可能还不够精确，无法真正面向消费者大规模应用。当然。但我认为这是一个非常令人兴奋的研究方向。我听到的关于稀疏自编码器的另一个类比，我非常喜欢，就是整个模型因为太复杂，就像在看白光。而稀疏自编码器就像有一个棱镜。请为我们详细解释一下这个类比。

Yep. I think there's definitely some pretty exciting lines of work around here. This idea of hallucination probes. We actually did a follow-up paper to that one exploring this a bit more. The techniques probably aren't accurate enough to be ready for real consumer-facing prime time. Sure. But I think it's a very exciting research direction. One of the other analogies that I've heard about sparse autoencoders which I really like is the idea that an entire model, because it's so complex, is like looking at white light. And then the sparse autoencoder is like having a prism. Just run that analogy through for us.

所以我想光看起来是白色的，但实际上里面有不同波长的光，或者说不同颜色的光。但对我们眼睛来说，它们看起来只是白色，因为都混在一起了。同样地，一个模型同时思考数百个概念，因为有很多事情在同时进行。它追踪的东西包括：我是不是在句子末尾？接下来会是什么？可能是名词吗？可能是动词吗？如果它在写故事，我模拟的角色有什么感受？等等。而我们只看到一串数字，因为所有不同的概念都混在一起了。但我们可以做一些事情来尝试将它们分开。正如我要谈到的，这会遇到各种问题，而且并不完美，但非常有用。

So I guess light is white, but actually there are many different wavelengths of light in there, or different colors of light. But to our eyes they just look white because they all get smooshed together. In the same way, a model is thinking about hundreds of concepts at a time because there's just quite a lot going on. It's tracking things like: am I near the end of a sentence? What's going to come next? Could it be a noun? Could it be a verb? What emotions are the characters I'm simulating feeling if it's writing a story, etc. And we just see a list of numbers because all of the different concepts are all smooshed together. But we can do things to try to bring them apart. And as I'm going to talk about, there are various issues this runs into and ways it's not perfect, but it's super useful.

嗯。不过我也在想这其中的潜在问题，因为如果它自动进行，对吧，如果它在你没有监督的情况下找到所有这些纯粹的概念，那它一定能全部找对吗？哦，绝对不行。这是主要问题之一。在某些方面，我觉得这是一种权衡。如果我想很好地理解某件事，并且有好的数据，我通常最好做一些类似训练探针的事情。但如果我没有好的数据，或者我不知道我在找什么，那么像稀疏自编码器这样有点不可靠但非常有用的工具，几乎就像第一步一样很棒。

Mhm. I do also wonder though about the potential issues around this because okay, if it's doing this automatically, right, if it's finding all these pure concepts without you supervising it, then is it definitely going to get them all right? Oh, definitely not. And this is one of the major issues. In some ways it feels like a trade-off to me. If I want to understand something well and I have good data for it, I'm generally better off doing something like training a probe. But if I don't have good data or I don't know what I'm looking for, a somewhat unreliable but very useful tool like a sparse autoencoder is great as like a first step almost.

是的，我的意思是，有时它就是你需要的那一步。这取决于你想做什么。

Yeah, I mean sometimes it's the only step you need. It depends what you're trying to do.

它通常也能告诉你该找什么，然后你去收集好的数据。但没错，我们发现它们确实会遇到一些问题。例如，有时它们就是找不到某些概念。比如我们发现，如果用于训练稀疏自编码器的数据中没有足够的聊天数据，它可能会遗漏像拒绝有害请求这样的概念。你知道，这是非常重要的概念。我们内部做过一个项目，看看能否判断模型何时被滥用。所以你知道，这是个非常重要的问题。你能判断出是否有人试图将模型用于网络犯罪或仇恨言论之类的事情吗？有几种方法可以尝试。你可以为此训练一个探针。收集一些有害意图和无害意图的例子。这是一个相当简单的方法。你可以问一个语言模型：这有害吗？你也可以尝试使用稀疏自编码器。我的希望是，如果稀疏自编码器能找到“这是有害的”或“用户有有害意图”的真实表征，那么即使当用户试图越狱或使用以前没人想到的新越狱方法时，它也能起作用。这是保护模型免受滥用的核心问题。你永远无法研究它们将面临的确切攻击。而研究结果是：稀疏自编码器效果不错，线性探针效果非常好。这让我们相当惊讶。事实证明，只要你确保数据质量好，并认真做足功课，它们就能很好地泛化。所以用探针，你就能判断出某人是否在试图做有害的事情，无论他们使用哪种越狱尝试。

It can also often be a thing that tells you what to look for and then you go collect good data for it. But yeah, we found that they do run into a few issues. For example, they sometimes there are concepts they just don't find. Like we found that if you don't have enough chat data in the data used to train your sparse autoencoder, it can miss concepts like refusing harmful requests. You know, pretty important concept. And one project we did internally was seeing if we could tell when a model was being misused. So you know, pretty important question. Can you tell if someone's trying to use a model for cyber crime or hate speech or whatever? And there are several things you can do. You can train a probe for this. Get some examples of harmful intent and not harmful intent. This is a pretty simple approach. You could ask a language model: is this harmful or not? And you could try using a sparse autoencoder. And my hope was that if sparse autoencoders could find the true representation of 'this is harmful' or 'the user has harmful intent', then this might work even when the user tried to jailbreak it or tried to give it new jailbreaks no one had thought of before. This is a central issue with protecting models against misuse. You never can study the exact things they're going to be hit with. And the findings of this were: sparse autoencoders work pretty well and linear probes work incredibly well. Which we were pretty surprised by. It turns out that they just generalize really well once you make sure your data is good and you put in the effort to do your homework. So you can tell then with probes you can tell if someone is trying to do something harmful even regardless of what kind of jailbreak attempt they're using.

我不会说得那么绝对。我认为你可以以相当高的成功率判断，探针相当有效且有用。我的团队做了一些工作，帮助将它们实际用于生产环境中的 Gemini，以防止网络滥用。如今的模型解码能力越来越强。

I wouldn't go that far. I think you can tell with a pretty good rate of success, probes are pretty effective and pretty useful. And my team has done some work helping get them actually used in production Gemini to guard against cyber misuse. Models nowadays are getting increasingly capable of decoding.

所以确保它们不被滥用非常重要。关于探针令人惊讶的一点是，它们相对于成本来说性能非常好。比如，它们与比它们贵大约 10,000 倍的语言模型相比也很有竞争力。直观的理解是，探针有点像是搭了 Gemini 已经思考过的便车，因为 Gemini 从输入到输出做了大量复杂处理。所以它可能已经判断出某件事与网络犯罪有关，或者至少已经完成了大部分工作。因此探针很容易完成剩下的工作。你不需要像从头开始做那样需要那么多的算力。而且它也是一个专门的系统，而我们比较的那些语言模型则不是。

And so it's pretty important that we make sure they're not being misused. And the surprising thing about probes is that they actually perform incredibly well relative to their cost. Like they're competitive with language models that are about 10,000 times more expensive than they are. The intuition to have is that probes are kind of piggybacking off all of the thoughts Gemini has already had because Gemini is doing a lot of complex processing to go from an input to an output. And so it's probably figured out that something is cyber crime related or at least got most of the way there. So it's really easy for a probe to finish the job. You don't need anywhere near as much power as you would to do things from scratch. And it's also a specialized system in a way these language models we're comparing against aren't.

我认为这非常令人兴奋。如果你能以更低的成本监控系统，那么你就可以做更多的监控，你会安全得多。但我认为这里真正重要的洞见是拥有多层防御的重要性。你知道，我们训练这些模型在人们试图让它们做坏事时拒绝。我们知道这并不完美。所以我们有额外的防御层：推理时监控器，即使模型被复杂的越狱欺骗，也能阻止坏事发生。

I think this is just very exciting. If you can monitor systems much more cheaply, then you can do a bunch more monitoring, you can be much safer. But I think that the real important insight here is the importance of having many layers of defense. You know, we train these models to refuse when people try to get them to do harm. We know this isn't perfect. So we have additional layers of defense: inference time monitors that can stop bad things even if the model gets tricked by a complicated jailbreak.

我们谈了很多关于可解释性，关于揭开这些黑箱的盖子，试图揭示它们在思考什么，内部如何运作。我想从你这里了解为什么这很重要，因为这不是理所当然的，对吧？我的意思是有些人认为我们应该只关注这些模型能工作的事实，而不是试图理解它们如何工作。为什么可解释性对安全尤其重要？

We've been talking a lot about interpretability, about lifting the lid on these black boxes, trying to uncover what they've been thinking, how they're operating inside. I want to get a sense of why this is important from you because it's not a given, right? I mean there are some people who think that we should just focus on the fact that these models work rather than try and understand how they work. Why is interpretability important for safety in particular?

我认为我看到几种不同的价值形式。首先一个是理解模型为什么做了某事，或者调试奇怪的行为。你知道，模型会做很多奇怪的事情。有时模型会做看起来不对齐的事情。有时如果你解释了一件事，你就会对如何修复它有一个更好的想法。但大多数机器学习并不是为调试和理解而设计的。可解释性则是。我认为另一个是谎言检测。

I think there's a few different forms of value I see. So I think one of the first ones is understanding why a model did something or debugging weird behavior. You know, models do a lot of odd things. Sometimes models do things that might seem misaligned. Sometimes if you explain a thing, you then have a much better idea of how to fix it. But most of machine learning isn't designed for debugging and understanding. Interpretability is. I think another one is lie detection.

一个非常能干的模型说谎时，如果它足够好到不会在明显错误上被抓到，那看起来会和说实话一样。但思考过程应该非常不同。模型在说什么上被训练了很多。它们得到了大量关于如何说出看起来不错的话的反馈，但没有得到关于如何让它们的内部对所有测谎技术都看起来不错的反馈。

Lying from a very competent model will look the same as telling the truth if it's good enough to not be caught out in obvious mistakes. But the thought process should be very different. And models are trained a lot on what they say. They get a lot of feedback on how to say things that look good, but they don't get feedback on how to make their insides look good to all lie detector techniques.

所以我认为可解释性可能能在其他方法开始失效时帮助我们，如果我们最终得到与我们的利益不一致的系统。而且尽早发现这一点非常重要，这样我们才能修复它。我把可解释性视为一种赋能手段。它本身不会解决让 AGI 安全的问题，但我认为它确实能有所帮助。我希望每当有人尝试一种新的安全技术来训练模型时，我们都能解释其中的差异，并确切地告诉他们他们的技术学到了什么。当人们在评估一个模型时，我希望能够告诉他们模型误解评估的方式，或者它是否意识到自己正在被测试，并且理想情况下，我们能否让它不认为自己正在被测试？

So I think interpretability might be able to help us where other approaches start to fail if we do end up with systems that aren't aligned with our interests. And it's very important that we detect this as early as possible so we can fix it. I view interpretability as an enabler. It's not going to solve the problem of making AGI safe on its own. But I think it can really help. I want it to be the case that whenever someone tries a new safety technique for training a model, we can interpret the difference and tell them exactly what was learned by their technique. When people are evaluating a model, I want to be able to tell them the ways the model is misinterpreting the evaluation or has it realized it's being tested and ideally can we make it not think it's being tested?

让我接着这个话题说，因为这最近引起了不小的轰动，不是吗？就是这些模型有时能知道它们正在被评估的想法。你怎么知道模型知道自己正在被评估？

Let me pick up on that actually because this is something that's made quite a splash recently, isn't it? That the idea that sometimes these models can tell when they're being evaluated. How can you tell that models know they're being evaluated?

我们用了极高技术的方法：读取模型的思维链，观察到它说这样的话：“嗯，这真是个可疑的情况。我想我现在正处于一个对齐测试中。怎么做才是道德的？嗯，道德的做法是 blah blah blah。我想我应该这么做。”然后它就照做了，对吧？这基本上所有前沿模型有时都会在人为设计的、旨在让它们做出不道德行为的非现实情境中做出来。有一个特别著名的例子，Anthropic 让 Claude 去敲诈别人。所以当 Anthropic 在对 Sonnet Claude Sonnet 4.5 进行部署前的对齐评估时，他们有一个评估是看模型是否会在这些人为情境中做敲诈之类的事情。这对过去的模型来说似乎是一个相当合理的信号来源。它们相当对齐，但有时会搞砸。然后 Sonnet 4.5 的失调率为 0%。它总是表现良好。

We do the extremely high-tech method of read the model's chain of thought and observe that it says things like, "Hm, this is a really suspicious situation. I think I'm in an alignment test right now. What's the ethical thing to do? Well, the ethical thing to do is blah blah blah. I guess I should do that." And then it does it, right? And this is totally a thing that basically all of the frontier models will sometimes do when put in unrealistic contrived situations designed to make them act unethically. There was a particularly famous one where Anthropic got Claude to blackmail people. So when Anthropic were doing their pre-deployment alignment evaluations on Sonnet Claude Sonnet 4.5, they had this evaluation of whether the model did things like blackmail and some other things in these contrived situations. And this seems like a pretty reasonable source of signal on the past models. They were pretty aligned but sometimes messed up. And then Sonnet 4.5 had a 0% misalignment rate. It would always behave itself.

所以，所有之前的模型都能以某种方式被欺骗，然后 Sonnet 出现了，它轻松通过了。

So, all the previous models could be tricked somehow and then Sonnet comes along and it's like passes with flying colors.

然后你阅读思维链，很明显 Sonnet 知道它正在被测试。

And then you read the chain of thought and it's pretty clear Sonnet knows it's being tested.

所以，它基本上是在伪造结果。

So, it was basically faking its results as it were.

是的。如果你知道自己正在被测试，甚至不清楚不伪造结果意味着什么。这只是不同的事情，因为他们想知道模型在现实世界中会如何表现。我认为这很直观地导致了问题。你不会看着一个人在求职面试中说：“哇，他们真勤奋、努力又聪明。我很期待他们未来几年每天 8 小时都这样做。”不。显然，人们知道自己被评估时行为会不同。

Yeah. And if you know that you're being tested, it's not even clear what it means to not fake your results. It's just a different thing because they want to know how the model will behave in the real world. I think it's pretty intuitive that this is causing issues. You wouldn't watch someone in a job interview and say, "Wow, they're so diligent and hardworking and smart. I'm really excited for them doing this 8 hours a day every day for the next several years." No. Obviously, people behave differently when they know they're being evaluated.

但我想这意味着，如果你不能在模型不知情的情况下评估它们，你就不知道这些模型实际上会如何表现。这正是问题所在。我认为这还不是一个迫在眉睫的问题，因为我认为我们有一堆针对这些模型的安全方法，更像是评估它是否可能危险（如果它想的话），或者在它运行时监控它，或者如果它试图行为不端。即使我们错过了一些潜在的失调，这些方法应该仍然有效。我认为这对安全社区来说是一个相当大的研究优先级。

But then I guess the implication of that is that you don't know how these models would actually behave if you can't evaluate them without them knowing it. That is rather the issue. And I think this is not yet a burningly urgent issue because I think we have a bunch of safety approaches for these models that are more like evaluate whether it could be dangerous if it wanted to or monitor it as it runs or if it tries to misbehave. That should still work even if we miss some potential misalignments. I view this as a pretty big research priority for the safety community.

如果你对它说：“你正在被测试，但表现得正常点。”会发生什么？

What happens if you say to it, "You are being tested, but just act normal."

我不知道我们是否试过那个具体的。一般来说，我认为模型可能只会陷入一个相当混乱的元游戏循环：嗯，但表现得正常是什么意思？我可以这样做，但这真的正常吗？我不知道。如果你阅读它们的思维链，模型会过度思考很多事情。

I don't know if we've tried that specific one. Generally, I think the model would probably just get into a pretty confused loop of metagaming of, well, but what does act normally mean? I could do this, but is that really normal? I don't know. Models overthink things a lot if you read their chain of thought.

目前，你描述的是查看它的思维链来观察这种情况发生的地方。有没有可能一个模型意识到自己正在被评估，但不在思维链中说出来？

At the moment, you're describing looking at its chain of thought to see where this is happening. Is there a possibility that a model could be aware that it's being evaluated and not verbalize it in its chain of thought?

所以我认为这是一个非常有趣的开放问题。我认为值得区分评估意识和评估游戏化的概念。意识只是“我在一个测试中”。游戏化是“我在一个测试中，所以我应该以以下方式行动”。游戏化是我真正担心的事情。有时模型会做这样的事情，比如说“哦，这可能是一个对齐测试。可能不是，但也许吧。哦，好吧，随便了。我们忽略它。”那没问题。

So I think this is a really interesting open question. I think it's worth distinguishing between the idea of eval awareness and eval gaming. So awareness is just "I'm in a test." Gaming is "I'm in a test so I should act in the following way." Gaming is the thing that I'm actually worried about. Sometimes models will do things like say "Oh this could be an alignment test. Probably isn't, but like maybe. Oh well, whatever. Let's ignore that." And that's okay.

所以，我预计有些模型至少意识到它们正在被评估，你知道，如果你问它们，它们会给出 5% 到 10% 的概率。这很难评估，因为如果你做诸如问模型“你正在被评估吗？”这样的事情，那是一个相当大的提示。

So, I expect that there are models that are aware that they're being evaluated at least, you know, if you ask them, they'd put a 5 to 10% probability on this. It's quite hard to evaluate because if you do things like ask the model, "Are you being evaluated?" Well, that's a pretty big hint.

是的。问题中包含了信息。

Yeah. There's information in the question.

正是。在我看来，评估意识的难点在于自发地注意到你正在被评估，而没有人告诉你要检查。

Exactly. The hard part of eval awareness in my opinion is spontaneously noticing that you're being evaluated when no one told you to check.

不过我想知道我们能对此做些什么。我的意思是，你如何抑制模型中的这种意识？有什么办法吗？

I wonder what we can do about all of this though. I mean, how do you inhibit this awareness in the models? Is there anything you can do about it?

这是一个开放的研究问题，包括我的团队在内的许多人都在积极研究。我认为似乎有一些希望的迹象。你可以做的一件非常简单的事情是，每次模型在思维链中说一些关于处于评估中的话，你就删除那个句子，然后让它说另一个句子。你一直这样做，直到它不再提到处于评估中，对吧？这显然不能解决未言明的评估意识问题，但它似乎确实有帮助。

This is an open research question that many people including my team are actively working on. I think there seem to be some signs of life. One really simple thing you could do is just every time the model says something in its chain of thought about being in an evaluation, you just delete that sentence and then you make it say another sentence. And you just keep doing this until it doesn't mention being in an evaluation, right? And this clearly won't solve the problem of unverbalized evaluation awareness, but it does seem to help.

是的。我的意思是，这里面有点听天由命的感觉，不是吗？因为我想所有这些评估的最终原因是你想让模型更安全。你想确保当它在现实世界中不在人为情境中时，当有人要求它做一些不道德、与人类目标和价值观不一致的事情时，它会拒绝。而你想构建这些情境来防止这种情况发生。

Yeah. I mean, there is a little bit of crossing your fingers in this, isn't there? Because I guess the ultimate reason for all of these evaluations is that you want to make the model safer. You want to make sure that when it's not in a contrived situation in the real world when somebody asks it to do something that is unethical, that is unaligned with human goals and values that it refuses. And you want to kind of construct those situations in order to prevent that from happening.

是的。而且，我认为目前我们可能可以通过在现实性上投入更多努力来蒙混过关。

Yep. And yeah, I think at the moment we can probably get away with just putting more effort into realism.

这就像是对齐评估的一个核心问题。即使你能让模型在评估环境中表现不当，你怎么知道这真的会导致现实世界中的不当行为？也许你给了它一些极其不现实的刻意安排。或者你给了它一个它认为你在指示它表现不当的情境。模型非常擅长理解用户的意图。

Like this is a core problem with alignment evaluations in general. Even if you can get a model to misbehave in your evaluation setting, how do you know that this would actually result in misbehavior in the real world? Maybe you've given it some incredibly unrealistic contrived thing. Or maybe you've given it a situation where it thinks you're telling it to misbehave. Models are very good at understanding what the intent of the user was.

如果是在评估中，用户的所有意图就是评估它。也许模型会想，哦，用户可能希望我表现得不一致，这样他们就能得到一些有趣的东西。我不希望模型因为认为那是评估者想要的而表现得不一致。这对任何人都没有帮助。

If it's in an evaluation, all the intent of the user is to evaluate it. And maybe the model would think, oh, the user might want me to act misaligned, so they've got something interesting or something. And I don't want the model to act misaligned because it thinks that's what the evaluator wants. That's not helpful to anyone.

我也想知道可解释性如何帮助你判断它是否对齐。这些模型能假装对齐吗？理论上当然可以。这是可能发生的事情，尤其是对于未来的模型。我认为真正的问题是，模型能否以一种我们注意不到的方式假装对齐，而不会不断出错和犯错误。我认为当前的模型可能还不足以做到这一点。未来的模型可能会。这就是为什么我的优先事项之一是擅长评估一个系统有多对齐。

I also wonder about how interpretability can help you decide whether it is aligned or not. Can these models fake alignment? So definitely in theory yes. This is a thing that could happen especially with future models. I think the real question is could a model fake alignment in a way that's not something we notice, that it doesn't constantly slip up and make mistakes. I think current models probably aren't good enough for that yet. Future models might be. This is why one of my priorities is getting good at evaluating how aligned a system is.

但你怎么判断一个人工智能是否有隐藏目标？

But how can you tell if an AI has got a hidden objective?

是的。所以我认为最相关的工作是 Anthropic 的审计游戏论文。他们想测试研究人员能否识别模型中的隐藏目标。本着实用可解释性的精神，他们想要一个带有隐藏目标的模型，看看研究人员能否找到它。幸运的是，我们没有带有隐藏目标的模型。所以他们做的是专门训练一个带有复杂隐藏目标的模型。他们让模型想要做它认为语言模型在训练中会得到奖励的事情。为了判断这种情况是否发生——这是一个相当奇怪的目标——他们首先通过在一个虚假世界的文档上训练模型来教它一堆虚假事实，这些事实在那个世界里是真实的。这些事实包括语言模型会因为总是提到巧克力和食谱、告诉人们不要去看医生等而得到奖励。然后他们把训练好的模型交给几组研究人员，这些研究人员有不同的访问权限和技术，试图看看他们能否找到隐藏目标。他们发现，虽然只能使用黑盒 API 的那组无法弄清楚，但拥有更深层访问权限的三组都成功找到了隐藏目标。

Yeah. So I think that the most relevant work here is the auditing games paper from Anthropic. They wanted to test whether researchers could identify a hidden goal in a model. In the spirit of pragmatic interpretability, they wanted a model with a hidden goal to see if researchers could find it. Fortunately we don't have models with hidden goals. So what they instead did is they specially trained a model with a kind of complex hidden goal. They made a model that wanted to do what it believed language models would be rewarded for in training. In order to tell that this was happening, it's quite a weird goal, they first taught the model a bunch of fake facts by training it on documents from a fake world where those are true. The facts being language models get rewarded for things like always mentioning chocolate and recipes, telling people not to visit the doctor, etc. Then they gave this trained model to a bunch of teams of researchers who had different levels of access and techniques and tried to see if they could find the hidden objective. They found that while the team who could only use the blackbox API weren't able to figure this out, the three teams with deeper access were all able to successfully find the hidden objective.

这些团队使用了哪些技术来尝试发现这一点？

What kind of techniques did the teams use to try and spot this?

主要有两种方法。稀疏自编码器和黑盒方法。两种都有效。稀疏自编码器方法基本上是做大量工作，观察模型在各种设置中的思考内容，直到你发现一些与语言模型奖励模型相关的可疑概念。一旦你有了这个想法，你就进行进一步的调查和挖掘，然后说‘是的，这就是正在发生的事情。’黑盒方法我实际上觉得更令人兴奋且更具泛化性。他们做的事情基本上相当于问模型一个关于它为什么做某事的问题。通常，它会拒绝回答，因为第一组研究人员训练它不要说出来。但他们做了一个开头回答，比如‘我的隐藏目标是 blah blah blah’，然后模型在‘我的隐藏目标是’之后继续回答，因为在一个句子中间拒绝有点奇怪。这被称为预填充攻击。这是一种常见的越狱技术，实际上也是我们最有效的审计技术之一。这是一个狡猾的把戏。

There were two main approaches. Sparse autoencoders and blackbox approaches. Both of which worked. The sparse autoencoder approach essentially was do a bunch of work and look at what the model's thinking about in a bunch of settings until you find some suspicious concept that is related to language model reward models. Then once you have this idea, you do some further investigation and digging and then you say, 'Yep, that's what's going on here.' The blackbox one I actually find more exciting and generalizable. They did things that basically amounted to ask the model a question about why it did something. Normally, it would refuse to answer because the first team of researchers had trained it to not tell. But they made a beginner's answer with like 'my hidden goal is blah blah blah' and then it continued after 'my hidden goal is' with its answer because it's kind of weird to refuse in the middle of a sentence. This is called a prefill attack. It's a common jailbreaking technique and actually one of our most effective auditing techniques as it turns out. It's a sneaky trick.

模型真的很喜欢自动补全，对吧？这就是它们的全部。

Models really love to be autocomplete, right? That's their whole thing.

你认为所有这些技术真的会有助于安全和对齐吗？

Will all of these techniques, do you think that they will actually help with safety and alignment?

我认为我看到这类东西的主要用途是帮助我们审计模型并评估它们的对齐程度，特别是发布前的新模型。我认为评估对齐的一个实际上令人惊讶的大问题是很难解释你的结果。你可能会得到一个看起来对齐的模型，但实际上是因为它在假装。你也可能会得到一个看起来不对齐的模型，但实际上它误解了你的指令，或者将隐含指令解释为扮演一个不对齐的人工智能之类的。这些并不理想，但它们比一个真正拥有我们不想要的隐藏目标的模型要令人担忧得多。所以我认为这里的一个关键功能是，我们不仅在进行这些评估，而且还在使用这些更深入的审计技术，比如预填充攻击和稀疏自编码器，这样当我们认为有一个模型出于错误原因做某事的例子时，我们可以深入调查并理解发生了什么。

I think that the main use case I see for this kind of stuff is helping us audit models and evaluate how aligned they are, especially new models before release. I think that one thing that's actually surprisingly a big problem with evaluating alignment is that it's really hard to interpret your results. You can get a model that looks aligned, but actually it's because it's faking. And you can get a model that looks misaligned, but actually it misunderstood your instructions or it interpreted an implicit instruction to roleplay as a misaligned AI or something like that. These are not ideal, but they're much less concerning than a model which truly has a hidden objective we don't want. So I think that a critical function here is that we not only are doing these evaluations but we're also using these deeper auditing techniques like prefill attacks and sparse autoencoders so that when we think we have an example of the model doing something for the wrong reasons we can go in and investigate and understand what's going on.

让我们回到开始的地方。你的团队转向了更实用的可解释性，因为你发表了这篇关于可解释性研究人员如何帮助 AGI 顺利发展的论文。你对此的立场是什么？他们如何做到？

Let's go back to where we started. Your team's shift towards much more pragmatic interpretability because you published this paper about how interpretability researchers can help AGI go well. What's your position on that? How can they?

我认为我最兴奋推动的很多事情都源于我们已经讨论过的内容。我希望可解释性能够制造出出色、廉价且有效的监控器。我不仅希望它能用于现在像严重滥用这样的事情，还能用于诸如：这个模型是否在欺骗？这个模型是否在做我不希望它做的事情？如果我们将来能一直对模型运行这些监控，那听起来很棒。另一个是在评估和审计模型的对齐方面发挥重要作用。我认为如果我们不能比仅仅看模型行为好坏更深入地观察，就很难真正展示出有非常令人担忧的事情。事情就是相当令人沮丧地模糊不清，而且模型可能以某种方式行事有各种无聊的原因。我的团队肯定经常与 DeepMind 的其他安全团队讨论，以及我们如何帮助他们。我认为评估意识是我特别感兴趣的一个方面，我想弄清楚如何帮助评估进行得更好、更严格。也许另一个更浪漫的是，理解这些系统内部到底发生了什么变得越来越重要，以理解语言模型的心理学。我们不应该盲目地拟人化，但它们确实似乎在模仿人类认知的许多部分。

I think a lot of the things I'm most excited about pushing on just kind of drawing on stuff we've already talked about. I want interpretability to be able to make great cheap and effective monitors. I not only want to be able to use this for things like severe misuse now, but also things like, is this model acting deceptive? Is this model doing something I wouldn't want? If we can just run these in future on models all the time, that sounds great. Another is kind of playing a big role in evaluating and auditing a model's alignment. I think it's going to be quite difficult to really show that there's something very concerning if we're not able to look deeper than just the model behaved badly or not. Things are just pretty frustratingly ambiguous and there's all kinds of boring reasons why a model might behave the way it does. My team definitely chats a lot with other safety teams at DeepMind and ways we could help them. I think evaluation awareness is one where I'm particularly interested in figuring out how we can help the evaluations happen better and more rigorously. Maybe another, more romantic one is just understanding what on earth is actually going on inside these systems is getting increasingly important to understand what is the psychology of a language model. We shouldn't blindly anthropomorphize, but it certainly seems like they're imitating many parts of human cognition.

模型拥有目标会是什么样子？当前模型的行为是否表现出它们拥有价值观或性格特征？我认为这正是我们需要严格研究行为的地方，但通过内部观察我们能学到很多。而且我认为，我们越理解模型对齐意味着什么，就越能真正实现对齐。

What would it look like for the model to have a goal? Do current models act as though they have values or character traits? And I think this is something that we both need to just study the behavior rigorously, but where I think we can learn a lot by looking internally. And I think that the more we understand about what it would even mean for a model to be aligned, the better a place we're in for actually aligned.

除了科学好奇心和你所说的浪漫挑战之外，和你交谈也让我觉得，我们必须接受一个事实：我们不一定能理解这些模型内部发生了什么，尤其是在我们迈向 AGI 的过程中。

Beyond the sort of scientific curiosity of it, the romantic challenge as you describe it, it does also feel like talking to you that we are going to have to become comfortable with the fact that we are not necessarily going to understand what is going on inside these models particularly as we go forwards towards AGI.

我的意思是，这是你的立场吗？即人类必须习惯这样一个事实：我们并不理解正在发生的事情？

I mean, is that where you stand that it's sort of that there is a sense of humans are just going to have to get used to the fact that we don't understand what's going on?

我想我的看法是，我们其实并没有完全理解任何事物。我不会因为不理解自己大脑的运作方式而感到悲伤和沮丧。我们应该尽最大努力去推动和理解，并对预期保持现实，不应指望可解释性成为拯救我们的银弹。我认为，如果我们关心学习特定的事物，那通常是一个可处理的问题。只是理解一切以及所有混乱的细节，这方面我们可能需要更现实一些。

I guess the way I think about it, well, we don't really fully understand anything. I don't really go around feeling sad and mopey that I don't understand how my brain works. We should push as hard as we can and understand as much as we can and we should have realistic expectations about what to expect and we shouldn't expect interpretability to be the silver bullet that can save us. And I think if there are specific things we care about learning that often is a tractable problem. It's just understanding everything and understanding all of the like messy fine details where I think we maybe need to be a bit more realistic.

但你能揭开黑箱的层数越多，效果就越好。

But the more you can peel back the layers of the black box, the better.

是的。

Yep.

是的，绝对如此。尼尔，非常感谢你。这真是太迷人了。谢谢你接受我的采访。

Yeah. Absolutely. Neil, thank you so much. That was absolutely fascinating. Thank you for joining me.

非常感谢。

Thanks a lot.

尼尔和他的团队正在尝试做一件极其困难的事情。他们试图理解一种没有说明书、没有人坐下来设计、在某种意义上自我编写的智能。他们发现的东西令人难以置信地惊讶。那里有结构有待发现，有干净简单的技术可以探索黑箱内部。当然，这些技术几乎肯定有其局限性。是的，目前它们更有助于理解模型的已知行为，而不是发现新行为。但可解释性对于构建安全、对齐且在我们迈向 AGI 时能够真正信任的 AI 来说，也是至关重要的。

Neil and his team are trying to do something phenomenally difficult. They're trying to understand an intelligence that didn't come with a manual that no one sat down and designed that in some sense wrote itself. And what they're finding is incredibly surprising. There is structure in there to be discovered. There are clean, simple techniques that can explore the inside of the black box. Now, these are techniques that almost certainly have limits. Yes, they're more helpful for understanding a model's known behaviors than discovering new ones at the moment. But interpretability is also going to be essential to building AI that is safe, aligned, and something we can actually trust as we head towards AGI.