扩展定律与通往 AGI 之路：2026-2027 年

让我们从 Scaling（规模扩张）定律和 Scaling 假设这个宏大概念开始。它是什么，它的历史是怎样的，我们今天处于什么位置？

Let's start with a big idea of scaling laws and the scaling hypothesis. What is it, what is its history, and where do we stand today?

我只能根据自己的经历来描述，但我在 AI 领域已经大约 10 年了，我很早就注意到了这一点。我最初加入 AI 世界是在 2014 年底，当时我在杨百翰大学与 Andrew 合作，那几乎是整整 10 年前了。我们做的第一件事是语音识别系统。那时，深度学习还是新事物，已经取得了很大进展，但每个人都在说我们没有成功所需的算法。我们只匹配了极小一部分，还有很多算法上的东西需要发现。我们还没有找到如何匹配人脑的图景。从某些方面来说，我很幸运——我有新手运气。我是这个领域的新人，我看了看我们用于语音的神经网络，即循环神经网络，然后我说：‘如果让它们更大、层数更多呢？如果同时扩大数据规模呢？’我只是把这些看作可以调节的独立旋钮。我注意到，当你给模型更多数据、让模型更大、训练时间更长时，模型的表现越来越好。那时我没有精确测量，但和同事们一起，我们非常直观地感觉到，投入的数据、算力和训练越多，模型表现越好。最初，我的想法是：‘嘿，也许这只适用于语音识别系统——也许这只是某个特定领域的怪癖。’直到 2017 年，当我第一次看到 GPT-1 的结果时，我才恍然大悟，语言可能正是我们可以这样做的领域。我们可以获得数万亿词的语言数据，在上面训练，而那时我们训练的模型很小——可以在 1 到 8 个 GPU 上训练，而现在我们在数万个 GPU 上训练，很快将达到数十万个。当我看到这两件事结合在一起时，还有像 Ilya Sutskever 这样的人也有类似观点——他可能是第一个，尽管我认为有几个人几乎同时得出了类似看法。还有 Rich Sutton 的苦涩教训，Gwern 写过关于 Scaling 假设的文章。但我想，在 2014 年到 2017 年之间的某个时候，我才真正明白，真正确信，嘿，只要我们扩大模型规模，我们就能完成这些极其广泛的认知任务。在 Scaling 的每个阶段，总有争论。当我第一次听到这些争论时，老实说，我以为可能是我错了，这个领域的所有专家都是对的——他们比我更了解情况。有乔姆斯基的论点，说你可以得到句法，但得不到语义。还有这种想法：‘哦，你可以让一个句子有意义，但无法让一个段落有意义。’我们今天最新的说法是，我们会耗尽数据，或者数据质量不够高，或者模型无法推理。而每一次，我们都设法找到了解决方法，或者 Scaling 本身就是解决方法。有时是前者，有时是后者。现在，我仍然认为这总是相当不确定的——我们只有归纳推理告诉我们未来几年会像过去 10 年一样。但我已经看过这部电影足够多次了，我已经看到这个故事发生了足够多次，以至于我真的相信，很可能……

I can only describe it as it relates to my own experience, but I've been in the AI field for about 10 years, and it was something I noticed very early on. I first joined the AI world when I was working at BYU with Andrew in late 2014, which is almost exactly 10 years ago now. The first thing we worked on was speech recognition systems. In those days, deep learning was a new thing; it had made lots of progress, but everyone was always saying we don't have the algorithms we need to succeed. We're only matching a tiny fraction; there's so much we need to discover algorithmically. We haven't found the picture of how to match the human brain. In some ways, I was fortunate—I had beginner's luck. I was a newcomer to the field, and I looked at the neural net we were using for speech, the recurrent neural networks, and I said, 'What if you make them bigger and give them more layers? What if you scale up the data along with this?' I just saw these as independent dials you could turn. I noticed that the model started to do better and better as you gave them more data, as you made the models larger, as you trained them for longer. I didn't measure things precisely in those days, but along with colleagues, we very much got the informal sense that the more data, the more compute, and the more training you put into these models, the better they perform. Initially, my thinking was, 'Hey, maybe that is just true for speech recognition systems—maybe that's just one particular quirk, one particular area.' I think it wasn't until 2017 when I first saw the results from GPT-1 that it clicked for me that language is probably the area in which we can do this. We can get trillions of words of language data, we can train on them, and the models we were training in those days were tiny—you could train them on one to eight GPUs, whereas now we train jobs on tens of thousands, soon going to hundreds of thousands of GPUs. When I saw those two things together, and there were a few people like Ilya Sutskever who had somewhat similar views—he might have been the first one, although I think a few people came to similar views around the same time. There was Rich Sutton's bitter lesson, there was Gwern wrote about the scaling hypothesis. But I think somewhere between 2014 and 2017 was when it really clicked for me, when I really got conviction that hey, we're going to be able to do these incredibly wide cognitive tasks if we just scale up the models. At every stage of scaling, there are always arguments. When I first heard them, honestly, I thought probably I'm the one who's wrong, and all these experts in the field are right—they know the situation better than I do. There's the Chomsky argument about you can get syntactics but you can't get semantics. There's this idea, 'Oh, you can make a sentence make sense, but you can't make a paragraph make sense.' The latest one we have today is we're going to run out of data, or the data isn't high quality enough, or models can't reason. And each time, every time, we manage to either find a way around, or scaling just is the way around. Sometimes it's one, sometimes it's the other. I'm now at this point where I still think it's always quite uncertain—we have nothing but inductive inference to tell us that the next few years are going to be like the last 10 years. But I've seen the movie enough times, I've seen the story happen for enough times to really believe that probably the...

如果你外推我们已有的曲线，如果你说：‘嗯，我不知道，我们开始达到博士水平了，去年是本科水平，前年是高中生水平’——同样，你可以对具体任务和方面吹毛求疵，我们仍然缺少某些模态，但这些正在被添加，比如计算机使用被添加了，图像生成被添加了。如果你只是粗略地看一下这些能力增长的速度，确实会让你觉得我们会在 2026 或 2027 年达到那个水平。我认为仍然存在一些世界，其中这不会在 100 年内发生，但这些世界的数量正在迅速减少。我们正在迅速耗尽真正令人信服的阻碍因素，真正令人信服的理由来说明为什么这不会在未来几年内发生。规模扩张非常快。我们今天就这样做：我们制作一个模型，然后部署数千个，甚至数万个实例。我认为到那时，肯定在两三年内，无论我们是否拥有这些超级强大的 AI，我们都会达到能够部署数百万个这样的规模。我对意义感到乐观，但我担心经济问题和权力集中。这实际上是我更担心的：权力的滥用。AI 增加了世界上的权力总量，如果你集中这种权力并滥用它，它会造成无法估量的损害。是的，这非常可怕。

If you extrapolate the curves that we've had so far, if you say, 'Well, I don't know, we're starting to get to like PhD level, and last year we were at undergraduate level, and the year before we were at like the level of a high school student'—again, you can quibble with at what tasks and for what, we're still missing modalities, but those are being added, like computer use was added, like image generation has been added. If you just kind of eyeball the rate at which these capabilities are increasing, it does make you think that we'll get there by 2026 or 2027. I think there are still worlds where it doesn't happen in 100 years, but the number of those worlds is rapidly decreasing. We are rapidly running out of truly convincing blockers, truly compelling reasons why this will not happen in the next few years. The scale-up is very quick. We do this today: we make a model and then we deploy thousands, maybe tens of thousands of instances of it. I think by the time, certainly within two to three years, whether we have these super powerful AIs or not, we're going to get to the size where you'll be able to deploy millions of these. I am optimistic about meaning, I worry about economics and the concentration of power. That's actually what I worry about more: the abuse of power. AI increases the amount of power in the world, and if you concentrate that power and abuse that power, it can do immeasurable damage. Yes, it's very frightening.

以下是与 Anthropic CEO Dario Amodei 的对话。Anthropic 是创建 Claude 的公司，Claude 目前经常位居大多数 LLM 基准排行榜榜首。除此之外，Dario 和 Anthropic 团队一直直言不讳地倡导认真对待 AI 安全话题，并持续发表大量关于这一主题及其他主题的精彩 AI 研究。之后，我还邀请了 Anthropic 的另外两位杰出人士加入：首先是 Amanda Askell，她是一名研究员，致力于 Claude 的对齐和微调工作，包括 Claude 角色和个性的设计。有几个人告诉我，她可能比 Anthropic 的任何人类都与 Claude 交谈得更多，所以与她讨论提示工程以及如何充分利用 Claude 的实用建议绝对是一件引人入胜的事情。之后，Chris Olah 也来聊天；他是机械可解释性领域的先驱之一，这是一系列激动人心的努力，旨在逆向工程神经网络，弄清楚内部发生了什么，从网络内部的神经激活模式推断行为。这是一种非常有前景的方法，用于确保未来超级智能 AI 系统的安全，例如通过检测激活模式来判断模型是否在试图欺骗与之交谈的人类。这是 Alex Freedman 的播客。为了支持它，请查看描述中的赞助商。现在，亲爱的朋友们，有请 Dario Amodei。

The following is a conversation with Dario Amodei, CEO of Anthropic, the company that created Claude, that is currently and often at the top of most LLM benchmark leaderboards. On top of that, Dario and the Anthropic team have been outspoken advocates for taking the topic of AI safety very seriously, and they have continued to publish a lot of fascinating AI research on this and other topics. I'm also joined afterwards by two other brilliant people from Anthropic: first, Amanda Askell, who is a researcher working on alignment and fine-tuning of Claude, including the design of Claude's character and personality. A few folks told me she has probably talked with Claude more than any human at Anthropic, so she was definitely a fascinating person to talk to about prompt engineering and practical advice on how to get the best out of Claude. After that, Chris Olah stopped by for a chat; he's one of the pioneers of the field of mechanistic interpretability, which is an exciting set of efforts that aims to reverse engineer neural networks to figure out what's going on inside, inferring behaviors from neural activation patterns inside the network. This is a very promising approach for keeping future super intelligent AI systems safe, for example by detecting from the activations when the model is trying to deceive the human it is talking to. This is Alex Freedman's podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Dario Amodei.

Scaling 会继续，而且其中有些我们尚未从理论上解释的魔力。当然，这里的 Scaling 是指更大的网络、更多的数据、更大的算力——是的，所有方面，特别是线性地同步扩大更大的网络、更长的训练时间和更多的数据。这就像化学反应：你有三种反应物，需要线性地同步扩大这三种反应物。如果你只扩大一种而其他不变，其他反应物就会耗尽，反应停止。但如果你同步扩大所有东西，反应就能进行。现在有了这种经验科学艺术，你可以把它应用到其他更细微的领域，比如将缩放定律应用于可解释性，或者后训练，或者只是观察这个东西如何 Scaling。但大的缩放定律，我猜根本的缩放假设，是关于大网络和大数据带来智能。

Scaling is going to continue and that there's some magic to it that we haven't really explained on a theoretical basis yet. And of course the scaling here is bigger networks, bigger data, bigger compute, yes all in, in particular linear scaling up of bigger networks, bigger training times, and more and more data. So all of these things almost like a chemical reaction: you have three ingredients in the chemical reaction and you need to linearly scale up the three ingredients. If you scale up one not the others, you run out of the other reagents and the reaction stops. But if you scale up everything in series, then the reaction can proceed. And of course now that you have this kind of empirical science art, you can apply it to other more nuanced things like scaling laws applied to interpretability, or scaling laws applied to post-training, or just seeing how does this thing scale. But the big scaling law, I guess the underlying scaling hypothesis, has to do with big networks, big data leads to intelligence.

是的，我们在语言之外的许多领域都记录了缩放定律。最初我们在 2020 年初的论文中首次展示了语言领域的缩放定律。2020 年底又有一些工作展示了其他模态，如图像、视频、文本到图像、图像到文本、数学，它们都有相同的模式。你说得对，现在还有后训练或新型推理模型等其他阶段。在我们测量过的所有情况下，我们都看到了类似的缩放定律。

Yeah, we've documented scaling laws in lots of domains other than language, right. So initially the paper we did that first showed it was in early 2020 where we first showed it for language. There was then some work late in 2020 where we showed the same thing for other modalities like images, video, text-to-image, image-to-text, math. They all had the same pattern. And you're right, now there are other stages like post-training or there are new types of reasoning models. And in all of those cases that we've measured, we see similar types of scaling laws.

一个有点哲学性的问题：你的直觉是什么，为什么网络规模和数据规模越大越好？为什么会导致更智能的模型？

A bit of a philosophical question, but what's your intuition about why bigger is better in terms of network size and data size? Why does it lead to more intelligent models?

在我之前的生物物理学家生涯中——我本科读物理，研究生读生物物理——我会回想作为物理学家所知道的东西，实际上比我在 Anthropic 的一些同事在物理学方面的专业知识要少得多。有一个概念叫 1/f 噪声和 1/x 分布，通常，就像你把一堆自然过程加起来会得到高斯分布，如果你把一堆不同分布的自然过程加起来，比如你把一个探针接到电阻上，电阻中热噪声的分布是 1/f。这是某种自然收敛分布。我认为这归结为：如果你观察许多由具有多种尺度的自然过程产生的事物——不是高斯那种窄分布——而是看导致电噪声的大大小小的波动，它们有这种衰减的 1/x 分布。所以现在我想到物理世界中的模式，或者语言中的模式。如果我想语言中的模式，有一些非常简单的模式：有些词比其他词更常见，比如“the”，然后是基本的名词-动词结构，然后是名词和动词必须一致，然后是更高层次的句子结构，然后是段落的主题结构。所以存在这种回归结构，你可以想象，随着网络变大，它们首先捕捉到非常简单的相关性、非常简单的模式，然后还有一长尾的其他模式。如果那个长尾的其他模式非常平滑，就像电阻等物理过程中的 1/f 噪声一样，那么你可以想象，随着网络变大，它逐渐捕捉到越来越多的那个分布。所以这种平滑性反映在模型预测和表现的好坏上。语言是一个进化的过程——我们发展了语言，我们有常见词和不常见词，有常见表达和不常见表达，有频繁表达的想法和陈词滥调，也有新颖的想法。这个过程随着人类进化了数百万年。所以猜测——这纯粹是推测——是这些想法的分布存在某种长尾分布。

So in my previous career as a biophysicist—I did physics undergrad and then biophysics in grad school—I think back to what I know as a physicist, which is actually much less than what some of my colleagues at Anthropic have in terms of expertise in physics. There's this concept called the 1/f noise and 1/x distributions, where often, just like if you add up a bunch of natural processes you get Gaussian, if you add up a bunch of kind of differently distributed natural processes, if you take a probe and hook it up to a resistor, the distribution of the thermal noise in the resistor goes as 1/f. It's some kind of natural convergent distribution. And I think what it amounts to is that if you look at a lot of things produced by some natural process that has a lot of different scales—not a Gaussian which is kind of narrowly distributed—but if I look at large and small fluctuations that lead to electrical noise, they have this decaying 1/x distribution. So now I think of patterns in the physical world, or in language. If I think about the patterns in language, there are some really simple patterns: some words are much more common than others like 'the', then there's basic noun-verb structure, then there's the fact that nouns and verbs have to agree, and there's the higher-level sentence structure, then there's the thematic structure of paragraphs. So the fact that there's this regressing structure, you can imagine that as you make the networks larger, first they capture the really simple correlations, the really simple patterns, and there's this long tail of other patterns. And if that long tail of other patterns is really smooth like it is with the 1/f noise in physical processes like resistors, then you could imagine as you make the network larger, it's kind of capturing more and more of that distribution. So that smoothness gets reflected in how well the models are at predicting and how well they perform. Language is an evolved process—we've developed language, we have common words and less common words, we have common expressions and less common expressions, we have ideas and cliches that are expressed frequently, and we have novel ideas. And that process has developed, has evolved with humans over millions of years. So the guess—and this is pure speculation—would be that there is some kind of long-tail distribution of the distribution of these ideas.

所以有长尾，但也有你正在构建的概念层次的高度。网络越大，大概容量就越高——没错。如果你有一个小网络，它只能正确理解常见的东西。如果我拿一个很小的神经网络，它很擅长理解句子必须有动词、形容词、名词，对吧？但它很糟糕于决定这些动词、形容词和名词应该是什么，以及它们是否有意义。如果我把它做大一点，它就擅长那个了，然后突然它擅长句子，但不擅长段落。所以这些罕见和更复杂的模式随着我增加网络容量而被捕捉到。

So there's the long tail, but also there's the height of the hierarchy of concepts that you're building up. So the bigger the network, presumably you have a higher capacity to—exactly. If you have a small network, you only get the common stuff right. If I take a tiny neural network, it's very good at understanding that a sentence has to have verb, adjective, noun, right? But it's terrible at deciding what those verb, adjective, and noun should be and whether they should make sense. If I make it just a little bigger, it gets good at that, then suddenly it's good at the sentences but it's not good at the paragraphs. So these rare and more complex patterns get picked up as I add more capacity to the network.

那么自然的问题是：天花板在哪里？现实世界有多复杂？有多少东西要学？我认为我们没人知道答案。我强烈的直觉是，天花板不会低于人类水平，对吧？我们人类能够理解这些各种模式，所以这让我觉得，如果我们继续扩大这些模型，开发新的训练和 Scaling 方法，至少能达到人类已经达到的水平。然后还有一个问题：比人类理解得更多是可能的吗？比人类更聪明、更有洞察力是可能的吗？我猜答案一定是领域依赖的。如果我看生物学这样的领域——我写了那篇《爱的机器》的文章——在我看来，人类正在努力理解生物学的复杂性。如果你去斯坦福、哈佛或伯克利，有整个系的人在研究免疫系统或代谢途径，每个人只理解其中一小部分，专门化，并且他们很难将自己的知识与其他人的知识结合起来。所以我直觉认为，AI 在顶端有很大的空间变得更聪明。如果我想物理世界中的材料，或者解决人类之间的冲突之类的问题，可能其中一些问题并非不可解决，但更难，而且可能你只能做到一定程度。

Well, the natural question then is: what's the ceiling of this? How complicated and complex is the real world? How much stuff is there to learn? I don't think any of us knows the answer to that question. My strong instinct would be that there's no ceiling below the level of humans, right? We humans are able to understand these various patterns, and so that makes me think that if we continue to scale up these models to develop new methods for training them and scaling them up, that will at least get to the level that we've gotten to with humans. There's then a question of how much more is it possible to understand than humans do? How much is it possible to be smarter and more perceptive than humans? I would guess the answer has got to be domain-dependent. If I look at an area like biology—and I wrote this essay 'Machines of Loving Grace'—it seems to me that humans are struggling to understand the complexity of biology. If you go to Stanford or to Harvard or to Berkeley, you have whole departments of folks trying to study the immune system or metabolic pathways, and each person understands only a tiny part of it, specializes, and they're struggling to combine their knowledge with that of other humans. So I have an instinct that there's a lot of room at the top for AIs to get smarter. If I think of something like materials in the physical world, or addressing conflicts between humans or something like that, it may be that some of these problems are not intractable but much harder, and it may be that there's only so well you can do.

你可以用其中一些东西来做，就像语音识别一样，我能听清你说话的程度是有限的。所以我认为在某些领域，可能存在非常接近人类水平的天花板；而在其他领域，这些天花板可能非常遥远。我认为只有当我们构建这些系统时才能发现。事先很难知道；我们可以推测，但不能确定。在某些领域，天花板可能与人际官僚机构之类的东西有关。

You can do with some of these things right, just like with speech recognition, there's only so clear I can hear your speech. So I think in some areas there may be ceilings that are very close to what humans have done; in other areas, those ceilings may be very far away. And I think we'll only find out when we build these systems. It's very hard to know in advance; we can speculate but we can't be sure. And in some domains, the ceiling might have to do with human bureaucracies and things like this.

是的，所以人类基本上必须参与其中；那是天花板的原因，而不是智能的限制。

Yes, so humans fundamentally have to be part of the loop; that's the cause of the ceiling, not maybe the limits of the intelligence.

是的，我认为在很多情况下，理论上技术可以变化得非常快，例如，我们可能在生物学方面发明的一切。但请记住，我们有一个临床试验系统，必须通过它才能将这些药物用于人类。我认为这混合了不必要和官僚主义的东西，以及保护社会完整性的东西。整个挑战在于很难区分哪个是哪个。

Yeah, I think in many cases, in theory, technology could change very fast, for example, all the things that we might invent with respect to biology. But remember, there's a clinical trial system that we have to go through to actually administer these things to humans. I think that's a mixture of things that are unnecessary and bureaucratic and things that kind of protect the integrity of society. And the whole challenge is that it's hard to tell which is which.

我的观点是，在药物开发方面，我们太慢、太保守了。但当然，如果你搞错了，过于鲁莽可能会危及人们的生命。所以至少其中一些人类机构实际上是在保护人们。所以关键在于找到平衡。我强烈怀疑这种平衡更倾向于推动事情更快发生，但确实存在一个平衡。

My view is definitely, in terms of drug development, we're too slow and we're too conservative. But certainly, if you get these things wrong, it's possible to risk people's lives by being too reckless. So at least some of these human institutions are in fact protecting people. So it's all about finding the balance. I strongly suspect that balance is kind of more on the side of pushing to make things happen faster, but there is a balance.

如果我们确实遇到了极限，如果我们确实遇到了 Scaling（规模扩张）的放缓，你认为原因是什么？是算力有限、数据有限，还是其他什么？想法有限？

If we do hit a limit, if we do hit a slowdown in the scaling laws, what do you think would be the reason? Is it compute limited, data limited, or something else? Idea limited?

有几件事。现在我们讨论的是在达到人类水平和技能之前遇到极限。所以我认为一个目前流行且可能遇到的极限——我倾向于押注它不会发生，但确实有可能——就是我们用完了数据。互联网上的数据是有限的，而且存在数据质量问题。你可以从互联网上获得数百亿亿个词，但其中很多是重复的，或者是搜索引擎优化的废话，或者未来甚至可能是 AI 自己生成的文本。所以我认为这种方式能产生的东西是有限的。尽管如此，我们和其他公司正在研究合成数据的方法，你可以用模型生成更多你已经拥有的类型的数据，甚至从头生成数据。想想 DeepMind 的 AlphaGo Zero 做了什么，他们让一个机器人从完全不会下围棋到超越人类水平，只通过自我对弈；AlphaGo Zero 版本不需要人类示例数据。另一个方向当然是这些推理模型，它们进行思维链并停下来思考、反思自己的思考。这是另一种合成数据，结合了强化学习。所以我的猜测是，通过其中一种方法，我们将绕过数据限制，或者可能有其他可用的数据来源。

So a few things. Now we're talking about hitting the limit before we get to the level of humans and the skill of humans. So I think one that's popular today and could be a limit we run into—I like most of the limits I would bet against it, but it's definitely possible—is we simply run out of data. There's only so much data on the internet, and there are issues with the quality of the data. You can get hundreds of trillions of words on the internet, but a lot of it is repetitive or it's search engine optimization drivel, or maybe in the future it'll even be text generated by AIs itself. So I think there are limits to what can be produced in this way. That said, we and I would guess other companies are working on ways to make data synthetic, where you can use the model to generate more data of the type that you have already, or even generate data from scratch. If you think about what was done with DeepMind's AlphaGo Zero, they managed to get a bot all the way from no ability to play Go whatsoever to above human level just by playing against itself; there was no example data from humans required in the AlphaGo Zero version of it. The other direction, of course, is these reasoning models that do chain of thought and stop to think and reflect on their own thinking. That's another kind of synthetic data coupled with reinforcement learning. So my guess is with one of those methods we'll get around the data limitation, or there may be other sources of data that are available.

我们可以观察到，即使数据没有问题，当我们开始扩大模型规模时，它们只是停止变得更好。过去观察到的可靠现象是它们一直在变好；但可能在某一点上由于我们不了解的原因而停止。答案可能是我们需要发明新的架构。过去曾出现过模型数值稳定性的问题，看起来事情趋于平稳，但当我们找到正确的解锁方法时，它们并没有停滞。所以也许我们需要一些新的优化方法或新技术来解锁。到目前为止我没有看到任何证据，但如果事情放缓，那可能是一个原因。

We could just observe that even if there's no problem with data, as we start to scale models up, they just stop getting better. It seemed to be a reliable observation that they've gotten better; that could just stop at some point for a reason we don't understand. The answer could be that we need to invent some new architecture. There have been problems in the past with numerical stability of models where it looked like things were leveling off, but when we found the right unblocker, they didn't end up doing so. So perhaps there's some new optimization method or some new technique we need to unblock things. I've seen no evidence of that so far, but if things were to slow down, that perhaps could be one reason.

那么算力的限制呢，也就是建造越来越大的数据中心的昂贵性质？

What about the limits of compute, meaning the expensive nature of building bigger and bigger data centers?

目前，我认为大多数前沿模型公司都在大约 10 亿美元的规模上运行，上下浮动三倍。这些是现在存在或正在训练的模型。我认为明年我们将达到几十亿，然后 2026 年可能超过 100 亿美元，到 2027 年可能有建造 1000 亿美元集群的雄心。我认为所有这些都会发生；在这个国家内建造算力的决心很大，我猜它确实会发生。现在，如果我们达到 1000 亿美元，那仍然不够算力，仍然不够规模。那么要么我们需要更大的规模，要么我们需要开发一种更高效的方法，即移动曲线。我认为所有这些中，我之所以看好强大 AI 如此快速出现的原因之一就是，如果你外推曲线上的下几个点，我们很快就会接近人类水平的能力。我们开发的一些新模型，其他公司推出的一些推理模型，开始达到我所说的博士或专业水平。如果你看看它们的编码能力，我们发布的最新模型 Sonnet 3.5 的新版本，在 SWE-bench 上得到了大约 50%。SWE-bench 是一系列专业真实世界软件工程任务的示例。今年年初，我认为最先进水平是 3% 或 4%。所以在 10 个月内，我们从 3% 上升到 50%，我认为再过一年我们可能会达到 90%。我的意思是，我不知道，但可能甚至更少。我们在研究生级别的数学、物理和生物学中也看到了类似的情况，来自 OpenAI o1 等模型。所以如果我们继续外推我们拥有的技能，我认为如果我们外推直线曲线，几年内这些模型将超过人类的最高专业水平。现在，这条曲线会继续吗？你和我都指出了很多可能不会发生的原因，但如果外推曲线继续，那就是我们正在走的轨迹。

Right now, I think most of the frontier model companies are operating at roughly $1 billion scale, plus or minus a factor of three. Those are the models that exist now or are being trained now. I think next year we're going to go to a few billion, and then 2026 we may go to above $10 billion, and probably by 2027 there are ambitions to build $100 billion clusters. I think all of that actually will happen; there's a lot of determination to build the compute to do it within this country, and I would guess that it actually does happen. Now, if we get to $100 billion, that's still not enough compute, that's still not enough scale. Then either we need even more scale or we need to develop some way of doing it more efficiently, of shifting the curve. I think between all of these, one of the reasons I'm bullish about powerful AI happening so fast is just that if you extrapolate the next few points on the curve, we're very quickly getting towards human level ability. Some of the new models that we developed, some reasoning models that have come from other companies, they're starting to get to what I would call the PhD or professional level. If you look at their coding ability, the latest model we released, Sonnet 3.5, the new or updated version, it gets something like 50% on SWE-bench. SWE-bench is an example of a bunch of professional real-world software engineering tasks. At the beginning of the year, I think the state-of-the-art was 3 or 4%. So in 10 months, we've gone from 3% to 50% on this task, and I think in another year we'll probably be at 90%. I mean, I don't know, but it might even be less than that. We've seen similar things in graduate level math, physics, and biology from models like OpenAI o1. So if we just continue to extrapolate this in terms of skill that we have, I think if we extrapolate the straight curve, within a few years we will get to these models being above the highest professional level in terms of humans. Now, will that curve continue? You've pointed to, and I've pointed to, a lot of reasons why that might not happen, but if the extrapolation curve continues, that is the trajectory we're on.

所以 Anthropic 有几个竞争对手。听听你对这一切的看法会很有趣：OpenAI、Google、xAI、Meta。

So Anthropic has several competitors. It'd be interesting to get your sort of view of it all: OpenAI, Google, xAI, Meta.

我想把几件事分开说。Anthropic 的使命是努力让这一切顺利发展。我们有一个变革理论，叫做 '竞相向上'。'竞相向上' 就是通过树立榜样，推动其他参与者做正确的事。这不是要当好人，而是要把局面安排好，让所有人都能成为好人。我举几个例子。在 Anthropic 早期，我们的联合创始人之一 Chris Olah（我相信你很快会采访他）共同创立了机械可解释性这个领域，旨在理解 AI 模型内部发生了什么。我们让他和一个早期团队专注于可解释性，我们认为这对让模型更安全、更透明有好处。有三四年时间，它完全没有商业应用。今天仍然没有。我们正在做一些早期测试，可能最终会有，但这是一个非常长期的研究赌注，而且我们是在公开场合构建并分享我们的成果。我们这样做是因为我们认为这是让模型更安全的一种方式。有趣的是，随着我们这样做，其他公司也开始效仿。有些是受到启发，有些是担心如果其他公司这样做显得更负责任，他们也想显得更负责任。没人想看起来像不负责任的一方，所以他们也开始采用。当人们来到 Anthropic 时，可解释性常常是一个吸引点，我告诉他们：'你没去的其他地方，告诉他们你为什么来这里。'然后你很快会看到其他地方也有可解释性团队。这在某种程度上削弱了我们的竞争优势，因为现在别人也在做，但对整个系统有好处。所以我们必须发明一些新东西，别人还没做的。希望是基本上抬高做正确事情的重要性。这不是关于我们本身，不是要有一个特定的好人。其他公司也可以这样做，如果他们加入这场竞赛。那是最好的消息。这关乎塑造激励，使其向上，而不是向下。

I want to separate out a couple things. Anthropic's mission is to try to make this all go well. We have a theory of change called race to the top. Race to the top is about trying to push the other players to do the right thing by setting an example. It's not about being the good guy; it's about setting things up so that all of us can be the good guy. I'll give a few examples. Early in the history of Anthropic, one of our co-founders, Chris Olah, who I believe you're interviewing soon, co-founded the field of mechanistic interpretability, which is an attempt to understand what's going on inside AI models. We had him and one of our early teams focus on this area of interpretability, which we think is good for making models safe and transparent. For three or four years, it had no commercial application whatsoever. It still doesn't today. We're doing some early betas with it, and probably it will eventually, but this is a very long research bet, one in which we've built in public and shared our results publicly. We did this because we think it's a way to make models safer. An interesting thing is that as we've done this, other companies have started doing it as well. In some cases, they've been inspired by it; in some cases, they're worried that if other companies are doing this and look more responsible, they want to look more responsible too. No one wants to look like the irresponsible actor, so they adopt this as well. When folks come to Anthropic, interpretability is often a draw, and I tell them, 'The other places you didn't go, tell them why you came here.' And then you see soon that there are interpretability teams elsewhere as well. In a way, that takes away our competitive advantage because now others are doing it too, but it's good for the broader system. So we have to invent some new thing that we're doing that others aren't doing as well. The hope is to basically bid up the importance of doing the right thing. It's not about us in particular; it's not about having one particular good guy. Other companies can do this as well if they join the race to do this. That's the best news ever. It's about shaping the incentives to point upward instead of pointing downward.

我们应该说这个例子，机械可解释性领域，是一种严谨、不空谈的 AI 安全方法。

We should say this example, the field of mechanistic interpretability, is just a rigorous, non-handwavy way of doing AI safety.

是的，或者说正在朝那个方向发展。我认为我们在观察能力上还处于早期，但我很惊讶我们能如此深入地观察这些系统并理解我们所看到的。与缩放定律不同，缩放定律感觉像是某种定律驱动模型表现更好，但模型内部并不是为了让我们理解而设计的。它们是为了运行、工作而设计的，就像人脑或人体生物化学一样。它们不是为了让人类打开舱盖、往里看并理解而设计的。但我们发现——你可以和 Chris 更详细地讨论——当我们打开它们，真正往里面看时，我们发现了令人惊讶的有趣东西。作为副作用，你还能看到这些模型的美。你可以通过机械可解释性的镜头探索大型神经网络的美丽本质。我对它的清晰程度感到惊讶。我对诸如归纳头之类的东西感到惊讶。我惊讶于我们可以使用稀疏自编码器在网络中找到这些方向，并且这些方向对应着非常清晰的概念。我们在金门大桥 Claude 上稍微演示了这一点。这是一个实验，我们在一个神经网络层中找到了一个对应金门大桥的方向，然后我们把它调高了。我们把这个模型作为演示发布，算是开了几天玩笑，但它说明了我们开发的方法。你可以拿这个模型，问它任何问题，因为这一特征被激活了，它会连接到金门大桥。所以它会说：'我感到放松和开阔，就像金门大桥的拱门一样'，或者它会巧妙地把话题转到金门大桥。它对金门大桥的关注还有一种悲伤。

Yes, or it's tending that way. I think we're still early in terms of our ability to see things, but I've been surprised at how much we've been able to look inside these systems and understand what we see. Unlike with the scaling laws, where it feels like there's some law driving these models to perform better, on the inside the models aren't designed for us to understand them. They're designed to operate, to work, just like the human brain or human biochemistry. They're not designed for a human to open up the hatch, look inside, and understand them. But we have found—and you can talk in much more detail about this to Chris—that when we open them up, when we do look inside them, we find things that are surprisingly interesting. As a side effect, you also get to see the beauty of these models. You get to explore the beautiful nature of large neural networks through the mechanistic interpretability lens. I'm amazed at how clean it's been. I'm amazed at things like induction heads. I'm amazed that we can use sparse autoencoders to find these directions within the networks, and that the directions correspond to very clear concepts. We demonstrated this a bit with the Golden Gate Bridge Claude. This was an experiment where we found a direction inside one of the neural network layers that corresponded to the Golden Gate Bridge, and we just turned that way up. We released this model as a demo, kind of half a joke for a couple days, but it was illustrative of the method we developed. You could take the model, ask it about anything, and because this feature was activated, it would connect to the Golden Gate Bridge. So it would say, 'I'm feeling relaxed and expansive, much like the arches of the Golden Gate Bridge,' or it would masterfully change topic to the Golden Gate Bridge. There was also a sadness to its focus on the Golden Gate Bridge.

我觉得人们很快就爱上了它。

I think people quickly fell in love with it.

我也这么认为。人们已经开始想念它了，因为一天后就被撤下了。不知何故，这些对模型的干预，你调整它的行为，在情感上让它比任何其他版本的模型都更像人类。强烈的个性，强烈的身份。它有这些执着的兴趣。我们都能想到某个对某事痴迷的人，所以这确实让它感觉更人性化一点。

I think so. People already miss it because it was taken down after a day. Somehow these interventions on the model, where you kind of adjust its behavior, emotionally made it seem more human than any other version of the model. Strong personality, strong identity. It has these obsessive interests. We can all think of someone who's obsessed with something, so it does make it feel somehow a bit more human.

让我们谈谈现在。谈谈 Claude。今年发生了很多事。三月，Claude 3 Opus、Sonnet、Haiku 发布了。然后七月发布了 Claude 3.5 Sonnet，刚刚发布了更新版本，还有 Claude 3.5 Haiku 也发布了。你能解释一下 Opus、Sonnet 和 Haiku 之间的区别，以及我们应该如何看待这些不同版本吗？

Let's talk about the present. Let's talk about Claude. This year a lot has happened. In March, Claude 3 Opus, Sonnet, Haiku were released. Then Claude 3.5 Sonnet in July with an updated version just now released, and also Claude 3.5 Haiku was released. Can you explain the difference between Opus, Sonnet, and Haiku, and how we should think about the different versions?

让我们回到三月，当时我们首次发布了这三个模型。我们的想法是，不同的公司会生产大小不同、好坏不同的模型。我们觉得市场既需要非常强大的模型——可能稍微慢一点、价格更高——也需要快速、便宜的模型，在速度和成本允许的范围内尽可能智能。当你需要进行某种困难分析时，比如写代码、头脑风暴或创意写作，你需要非常强大的模型。但在商业场景中，有很多实际应用，比如与网站交互、报税、与法律顾问交谈、分析合同，或者在 IDE 中做自动补全。对于所有这些，你需要一个快速、便宜的模型。

Let's go back to March when we first released these three models. Our thinking was that different companies produce kind of large and small models, better and worse models. We felt that there was demand both for a really powerful model, which might be a little bit slower and you'd have to pay more for, and also for fast, cheap models that are as smart as they can be for how fast and cheap they are. Whenever you want to do some kind of difficult analysis, like if you want to write code, brainstorm ideas, or do creative writing, you want the really powerful model. But then there are a lot of practical applications in a business sense where you're interacting with a website, doing your taxes, talking to a legal adviser, analyzing a contract, or doing autocomplete on your IDE. For all of those things, you want a fast, cheap model.

我们希望满足各种需求，所以最终采用了诗歌主题。俳句是很短的诗，所以 Haiku 是小型、快速、便宜的模型，发布时其智能水平对于速度和成本来说令人惊讶。十四行诗是中等长度的诗，所以 Sonnet 是中等模型——更聪明但稍慢、稍贵。Opus 像巨作，所以 Opus 是当时最大、最聪明的模型。这是最初的思路。我们的想法是，每一代新模型都应该移动这条权衡曲线。所以当我们发布 Sonnet 3.5 时，它的成本和速度与 Sonnet 3 大致相同，但智能提升到了比原始 Opus 3 模型更聪明的程度，尤其是在代码方面，总体上也是如此。现在我们已经展示了 Haiku 3.5 的结果，我相信最小的新模型 Haiku 3.5 与最大的旧模型 Opus 3 差不多好。基本上，目标是移动曲线，未来会有 Opus 3.5。每一代新模型都有自己的特点——它们使用新数据，个性也会以我们试图引导但无法完全控制的方式变化，所以从来不存在只改变智能的精确等价。我们总是努力改进其他方面，有些变化我们甚至没有意识到或无法测量。这在很多方面都是一门不精确的科学。这些模型的风格和个性更像是艺术而非科学。

We wanted to serve a whole spectrum of needs, so we ended up with this poetry theme. A haiku is a very short poem, so Haiku is the small, fast, cheap model that was surprisingly intelligent for how fast and cheap it was. Sonnet is a medium-sized poem, so Sonnet was the middle model—smarter but a bit slower and more expensive. Opus, like a magnum opus, is a large work, so Opus was the largest, smartest model at the time. That was the original thinking behind it. Our thinking then was that each new generation of models should shift that tradeoff curve. So when we release Sonnet 3.5, it has roughly the same cost and speed as the Sonnet 3 model, but its intelligence increased to the point where it was smarter than the original Opus 3 model, especially for code but also in general. Now we've shown results for Haiku 3.5, and I believe Haiku 3.5, the smallest new model, is about as good as Opus 3, the largest old model. Basically, the aim is to shift the curve, and at some point there will be an Opus 3.5. Every new generation of models has its own thing—they use new data, their personality changes in ways we try to steer but are not fully able to steer, so there's never quite that exact equivalence where the only thing you're changing is intelligence. We always try to improve other things, and some things change without us knowing or measuring. It's very much an inexact science in many ways. The manner and personality of these models is more an art than a science.

从 Claude Opus 3 到 3.5 之间的时间跨度是什么原因？这段时间花在了哪里？

What is the reason for the span of time between, say, Claude Opus 3 and 3.5? What takes that time?

有多个不同的过程。首先是预训练，也就是常规的语言模型训练，这需要很长时间——如今使用数万个 GPU、TPU 或 Trainium，通常训练数月。然后是后训练阶段，我们进行基于人类反馈的强化学习以及其他类型的强化学习。这个阶段现在越来越大，而且往往不那么精确——需要努力才能做好。模型会与一些早期合作伙伴进行测试，看看它们有多好，然后进行内部和外部的安全测试，特别是针对灾难性和自主性风险。我们根据负责任扩展政策进行内部测试，并与美国和英国的人工智能安全研究所以及其他特定领域的第三方测试机构达成协议，测试 CBRN 风险——化学、生物、放射性和核风险。我们目前认为模型尚未严重构成这些风险，但每个新模型我们都想评估，看看是否开始接近某些更危险的能力。然后还需要一些时间让模型在推理和 API 上线方面正常工作。实际让模型工作有很多步骤。我们总是努力让流程尽可能精简。我们希望安全测试既严格又自动化、快速，同时不牺牲严谨性，预训练和后训练也是如此。这就像建造其他东西——比如造飞机：既要安全，又要流程精简。两者之间的创造性张力很重要。

There are different processes. There's pre-training, which is the normal language model training, and that takes a very long time—these days using tens of thousands of GPUs, TPUs, or Trainium, often training for months. Then there's a post-training phase where we do reinforcement learning from human feedback as well as other kinds of reinforcement learning. That phase is getting larger and larger now, and it's often less of an exact science—it takes effort to get it right. Models are then tested with some of our early partners to see how good they are, and they're tested both internally and externally for safety, particularly for catastrophic and autonomy risks. We do internal testing according to our Responsible Scaling Policy, and we have an agreement with the US and UK AI Safety Institutes as well as other third-party testers in specific domains to test for CBRN risks—chemical, biological, radiological, and nuclear. We don't think models pose these risks seriously yet, but every new model we want to evaluate to see if we're starting to get close to some of these more dangerous capabilities. Then it takes some time to get the model working in terms of inference and launching it in the API. There are just a lot of steps to actually making a model work. We're always trying to make the processes as streamlined as possible. We want our safety testing to be rigorous but also automatic and fast without compromising on rigor, same with pre-training and post-training. It's like building anything else—like building airplanes: you want them safe but the process streamlined. The creative tension between those is important.

有传言说 Anthropic 的工具链非常好。可能这里的很多挑战都在软件工程方面——构建工具以实现与基础设施的高效、低摩擦交互。你会惊讶于构建这些模型的挑战有多少归结为软件工程和性能工程。从外部看，你可能觉得会有尤里卡式的突破，但几乎总是归结于细节，往往是极其枯燥的细节。

Rumor on the street is that Anthropic has really good tooling. Probably a lot of the challenge here is on the software engineering side—building tooling to have efficient, low-friction interaction with the infrastructure. You would be surprised how much of the challenge of building these models comes down to software engineering and performance engineering. From the outside, you might think there's a Eureka breakthrough, but it almost always comes down to the details, often super boring details.

我无法说我们的工具是否比其他公司更好——我至少近期没在其他公司待过——但这肯定是我们非常关注的事情。

I can't speak to whether we have better tooling than other companies—I haven't been at those other companies, at least not recently—but it's certainly something we give a lot of attention to.

从 Claude 3 到 Claude 3.5，是否有额外的预训练，还是主要集中在后训练？性能有了飞跃。

From Claude 3 to Claude 3.5, is there any extra pre-training going on, or is it mostly focused on post-training? There have been leaps in performance.

在任何阶段，我们都专注于同时改进一切。自然，不同团队在特定领域取得进展，各自让接力赛中的一段变得更好。当我们制造新模型时，我们会同时融入所有这些改进。

At any given stage, we're focused on improving everything at once. Naturally, different teams make progress in particular areas, each making their segment of the relay race better. When we make a new model, we put all of these things in at once.

从 RLHF 获得的偏好数据——是否适用于训练更新的模型？

The preference data you get from RLHF—is it applicable to newer models as they get trained up?

旧模型的偏好数据有时会用于新模型，尽管用新模型自身数据训练效果更好。请注意，我们有宪法 AI 方法，因此不仅使用偏好数据。还有一个后训练过程，我们让模型自我对抗，并且每天都有新的自我对抗后训练方法被使用。所以不仅仅是 RLHF——还有很多其他方法。后训练正变得越来越复杂。

Preference data from old models sometimes gets used for new models, although it performs somewhat better when trained on the new models' own data. Note that we have our Constitutional AI method such that we don't only use preference data. There's also a post-training process where we train the model against itself, and there are new types of post-training against itself that are used every day. So it's not just RLHF—it's a bunch of other methods as well. Post-training is becoming more and more sophisticated.

新 Sonnet 3.5 性能大幅提升的原因是什么，至少在编程方面？也许这是个讨论基准的好地方——变好意味着什么？只是数字上升了，但我编程，我也……

What explains the big leap in performance for the new Sonnet 3.5, at least on the programming side? Maybe this is a good place to talk about benchmarks—what does it mean to get better? Just the number went up, but I program and I also...

我热爱编程，通过 Cursor 使用 Claude 3.5 来辅助我编程。从经验上看，它变得更聪明了。要让它更聪明需要什么？

I love programming and I use Claude 3.5 through Cursor to assist me in programming. Anecdotally, it's gotten smarter at programming. What does it take to get it smarter?

我们也观察到了这一点。Anthropic 的几位非常厉害的工程师说，之前我们和其他公司的代码模型对他们没什么用——可能对初学者有用，但对他们没用。但最初的 Sonnet 3.5 第一次让他们说：‘天哪，这帮我省了几个小时的工作。’这是第一个真正为他们节省时间的模型。水位在上升。新的 Sonnet 甚至更好。这体现在各个方面：预训练、后训练和各种评估。在 SWE-bench 上，这是一个现实世界的基准测试，你给模型一个代码库和一个用语言描述的任务，让它自由操作——运行、编辑——成功率从 3% 提高到了大约 50%。我相信如果我们以不过度拟合的方式在该基准上达到 100%，那将代表编程能力的真正显著提升。如果达到 90-95%，它就能自主完成很大一部分软件工程任务。

We observed that too. A couple of very strong engineers here at Anthropic said that previous code models from us and other companies hadn't been useful to them—maybe useful to beginners, but not to them. But Sonnet 3.5, the original one, for the first time they said, 'Oh my God, this helped me with something that would have taken me hours.' It was the first model that actually saved them time. The water line is rising. The new Sonnet has been even better. It's across the board: in pre-training, post-training, and various evaluations. On SWE-bench, which is a real-world benchmark where you give the model a codebase and a task described in language, and let it do anything—run, edit—the success rate went from 3% to about 50%. I believe if we get to 100% on that benchmark in a way that isn't overfitted, it would represent a real and serious increase in programming ability. If we get to 90-95%, it could autonomously do a significant fraction of software engineering tasks.

一个离谱的时间问题：Claude 3.5 Opus 什么时候出？我们会在 GTA 6 之前拿到吗？还是像《永远的毁灭公爵》那样推迟 15 年？GTA 刚发布预告片，而我们发布第一个 Sonnet 才三个月。发布节奏决定了期望。

Ridiculous timeline question: when is Claude 3.5 Opus coming out? Are we going to get it before GTA 6? Or is it like Duke Nukem Forever, delayed 15 years? GTA just released trailers, and it's only been three months since we released the first Sonnet. The pace of releases sets expectations.

我不给具体日期，但据我们所知，计划仍然是要推出 Claude 3.5 Opus。

I'm not giving an exact date, but as far as we know, the plan is still to have a Claude 3.5 Opus.

随着模型越来越大，你怎么看版本管理？为什么是 Sonnet 3.5 更新加日期，而不是 Sonnet 3.6？

How do you think about versioning as models get bigger? Why Sonnet 3.5 updated with a date, not Sonnet 3.6?

命名是一个有趣的挑战。一年前，大部分模型都是预训练的，所以你可以从头开始，训练不同大小的模型，有一套命名体系。但麻烦在于有些模型训练时间比其他模型长得多。另外，当你在预训练上取得重大进展时，你会突然发现可以快速做出一个更好的预训练模型，但它的大小和形状与之前的模型相同。时间问题会打乱任何体系。这不像软件，你可以说‘这是 3.7，这是 3.8’。模型有不同的权衡：有些推理更快，有些更贵，有些更便宜。所有公司都在这个问题上挣扎。我们之前用 Haiku、Sonnet 和 Opus 处于有利位置，并试图维持，但并不完美。我们会努力回归简洁，但这个领域的本质让这很难。没人搞定了命名；这是一个不同于普通软件的范式。没有一家公司做得完美。相对于训练模型的宏大科学，我们在命名上挣扎得惊人地多。

Naming is an interesting challenge. A year ago, most of the model was pre-training, so you could start from the beginning and have models of different sizes trained together, with a family of naming schemes. But the trouble starts when some models take much longer to train than others. Also, as you make big improvements in pre-training, you suddenly notice you can make a better pre-trained model quickly, but it has the same size and shape as previous models. Timing issues frustrate any scheme. It's not like software where you can say 'this is 3.7, this is 3.8.' Models have different trade-offs: some are faster at inference, some more expensive, some less. All companies have struggled with this. We were in a good position with Haiku, Sonnet, and Opus, and we're trying to maintain it, but it's not perfect. We'll try to get back to simplicity, but the nature of the field makes it hard. No one has figured out naming; it's a different paradigm from normal software. None of the companies have been perfect at it. It's something we struggle with surprisingly much, relative to the grand science of training models.

从用户角度看，更新后的 Sonnet 3.5 与 2024 年 6 月的版本感觉不同。最好能有体现这种差异的标签，因为人们谈论 Sonnet 3.5，但现在有一个不同的版本。如何区分旧版和新版？当有明显改进时，对话变得困难。

From the user side, the updated Sonnet 3.5 feels different from the June 2024 one. It would be nice to have labeling that embodies that, because people talk about Sonnet 3.5 but now there's a different one. How do you refer to the previous one and the new one? When there's a distinct improvement, it makes conversation challenging.

是的，模型有很多属性没有反映在基准测试中。并非都是能力；模型可以礼貌或粗鲁，反应迅速或主动提问，有温暖或冷淡的个性，无聊或像 Golden Gate Claude 那样独特。我们有一个由 Amanda 领导的团队专注于‘Claude 性格’。但这仍然是一门不精确的科学。我们经常发现模型有我们不知道的属性。你可以和模型对话一万次，却错过某些行为，就像和人相处——你可以认识某人几个月却不知道他们有某种技能。

Yes, there are many properties of models not reflected in benchmarks. Not all are capabilities; models can be polite or brusque, reactive or ask questions, have a warm or cold personality, be boring or distinctive like Golden Gate Claude. We have a team focused on 'Claude character' led by Amanda. But it's still an inexact science. Often we find models have properties we're not aware of. You can talk to a model 10,000 times and miss some behaviors, just like with a human—you can know someone for months and not know they have a certain skill.

我得问你一个来自 Reddit 的问题。有一种很有趣的心理社会现象：人们报告说 Claude 随着时间推移变笨了。问题是：用户抱怨 Claude 3.5 Sonnet 变笨了，这有道理吗？这些传闻是一种社会现象，还是 Claude 确实在某些情况下变笨了？

I got to ask you a question from Reddit. There's this fascinating psychological social phenomenon where people report that Claude has gotten dumber for them over time. So the question is: does the user complaint about the dumbing down of Claude 3.5 Sonnet hold any water? Are these anecdotal reports a kind of social phenomenon, or did Claude get dumber in some cases?

这其实不限于 Claude。我相信每个主要公司的基础模型都收到过这类抱怨。人们说过 GPT-4，也说过 GPT-4 Turbo。有几点：第一，模型的实际权重——模型的大脑——除非我们推出新模型，否则不会改变。有很多实际原因使得随机替换新版本模型不合理。从推理角度看很难，而且控制改变权重的所有后果也很困难。假设你想微调模型让它少说'当然'，就像旧版 Sonnet 那样，你实际上还会改变一百件事。所以我们有完整的模型修改流程：做大量测试、用户测试、早期客户测试。我们从未在未告知的情况下改变模型权重，在当前设置下这样做也不合理。不过，我们偶尔会做几件事。一是有时我们会运行 A/B 测试，但通常是在模型即将发布时，且时间很短。例如，在新 Sonnet 3.5 发布前一天——我同意我们应该取个更好的名字，这名字很笨拙——有些人评论说它变好了很多，因为一小部分人在那一两天接触到了 A/B 测试。另一个是系统提示偶尔会变。系统提示可能有些影响，但不太可能让模型变笨。虽然这两件事发生得相对较少，但关于模型变了、模型不擅长这个、模型被审查更多、模型变笨了的抱怨却一直存在。所以我不想说人们在凭空想象，但模型大部分情况下没有变化。如果我要提出一个理论，我认为这与我之前说的有关：模型非常复杂，有很多方面。通常，如果我问模型一个问题，说'做任务 X'和'你能做任务 X'，模型可能会以不同方式回应。你可以改变与模型交互的许多细微之处，从而得到截然不同的结果。需要说明的是，这本身是我们和其他模型提供商的失败——模型经常对措辞的微小变化敏感。这是这些模型工作原理的科学尚不发达的又一个体现。所以，如果我某天晚上用一种方式与模型交谈，然后稍微改变措辞，我可能会得到不同结果。这是一种可能。另一件事是，量化这些东西很难。我认为人们在新模型刚推出时非常兴奋，然后随着时间的推移，他们开始意识到局限性。这可能是另一种效应。但说了这么多，总结起来就是：大部分情况下，除了一些相当狭窄的例外，模型并没有变化。我认为存在一种心理效应：你开始习惯了。基线提高了。就像人们第一次在飞机上使用 Wi-Fi 时，觉得神奇无比。而现在呢，'这东西怎么用不了，真是垃圾。'没错。所以很容易产生阴谋论，认为'他们故意让 Wi-Fi 变慢'。

This actually doesn't apply just to Claude. I believe I've seen these complaints for every foundation model produced by a major company. People said this about GPT-4, they said it about GPT-4 Turbo. So a couple things: one, the actual weights of the model—the actual brain of the model—does not change unless we introduce a new model. There are a number of reasons why it would not make sense practically to be randomly substituting in new versions of the model. It's difficult from an inference perspective, and it's actually hard to control all the consequences of changing the weights of the model. Let's say you wanted to fine-tune the model to say 'certainly' less, which an old version of Sonnet used to do. You actually end up changing a hundred things as well. So we have a whole process for modifying the model: we do a bunch of testing, user testing, early customer testing. We have never changed the weights of the model without telling anyone, and it wouldn't make sense in the current setup to do that. Now, there are a couple things we do occasionally. One is sometimes we run A/B tests, but those are typically very close to when a model is being released and for a very small fraction of time. For example, the day before the new Sonnet 3.5—I agree we should have had a better name, it's clunky—there were some comments from people that it got a lot better, because a fraction were exposed to an A/B test for those one or two days. The other is that occasionally the system prompt will change. The system prompt can have some effects, although it's unlikely to dumb down models, unlikely to make them dumber. While these two things happen relatively infrequently, the complaints about the model changing, the model not being good at this, the model got more censored, the model got dumbed down—those complaints are constant. So I don't want to say people are imagining it, but the models are for the most part not changing. If I were to offer a theory, I think it actually relates to one of the things I said before: models are very complex and have many aspects to them. Often, if I ask a model a question, if I say 'do task X' versus 'can you do task X,' the model might respond in different ways. There are all kinds of subtle things you can change about the way you interact with the model that can give you very different results. To be clear, this itself is a failing by us and by other model providers—that the models are often sensitive to small changes in wording. It's yet another way in which the science of how these models work is very poorly developed. So if I go to sleep one night talking to the model in a certain way and slightly change the phrasing, I could get different results. That's one possible way. The other thing is it's just hard to quantify this stuff. I think people are very excited by new models when they come out, and then as time goes on they become very aware of the limitations. That may be another effect. But that's a very long-winded way of saying: for the most part, with some fairly narrow exceptions, the models are not changing. I think there is a psychological effect: you just start getting used to it. The baseline raises. When people first got Wi-Fi on airplanes, it was amazing magic. And now it's like 'I can't get this thing to work, this is such a piece of crap.' Exactly. So it's easy to have the conspiracy theory of 'they're making Wi-Fi slower and slower.'

另一个 Reddit 问题：Claude 什么时候才能不再像我的祖母一样，把它的道德世界观强加给我这个付费客户？还有，让 Claude 过度道歉背后的理念是什么？

Another Reddit question: when will Claude stop trying to be my paternal grandmother imposing its moral worldview on me as a paying customer? And also, what is the ideology behind making Claude overly apologetic?

关于这一点有几点要说。首先，人们在 Reddit 和 Twitter 或 X 上说的话——社交媒体上人们大声抱怨的内容与实际统计上用户关心并驱动他们使用模型的内容之间存在巨大分布偏移。人们会因模型没有写出所有代码、或者模型在代码方面不够好而感到沮丧，尽管它已经是世界上最好的代码模型。我认为大多数抱怨都是关于这些的。但确实有一小部分声音很大的用户提出了这些担忧，他们对模型拒绝本不该拒绝的事情、过度道歉、或者只是有这些烦人的口头禅感到不满。第二个注意事项——我想非常清楚地说出来，因为我认为有些人不知道，另一些人知道但忘了——全面控制模型的行为非常困难。你不能直接伸手进去说'我希望模型少道歉'。你可以这样做：你可以加入训练数据，告诉模型应该少道歉。但随后在其他情况下，它们可能会变得超级粗鲁或过度自信，从而产生误导。

A couple points on this. First, things people say on Reddit and Twitter or X—there's actually a huge distribution shift between the stuff that people complain loudly about on social media and what actually statistically users care about and drives them to use the models. People are frustrated with things like the model not writing out all the code, or the model not being as good at code as it could be, even though it's the best model in the world on code. I think the majority of things are about that. But certainly a vocal minority are raising these concerns, frustrated by the model refusing things it shouldn't refuse, or apologizing too much, or just having these annoying verbal ticks. The second caveat—and I want to say this super clearly because I think some people don't know it and others kind of know it but forget—it is very difficult to control across the board how the models behave. You cannot just reach in there and say 'I want the model to apologize less.' You can do that: you can include training data that says the model should apologize less. But then in some other situation, they end up being super rude or overconfident in a way that's misleading.

所以这些都是权衡。举个例子，有一段时间模型——我们的模型，我想其他公司的也是——过于啰嗦。它们会重复自己，说太多。你可以通过惩罚模型说话太长来减少啰嗦。但如果你用粗糙的方式来做，当模型在编码时，有时它们会说“代码在这里”，因为它们学会了这是一种节省的方式。这导致模型在编码时所谓的“懒惰”，它们就像“啊，剩下的你自己搞定”。这不是因为我们想节省算力，或者模型在寒假期间变懒了，或者其他任何阴谋论。实际上，只是很难在所有情况下同时控制和引导模型的行为。这有点像打地鼠，你按下一个东西，其他东西就会冒出来，你可能甚至没有注意到或测量到。我如此关心未来 AI 系统的宏大对齐问题的原因之一，就是这些系统实际上非常不可预测，很难引导和控制。我们今天看到的这种“你让一件事变好，另一件事却变糟”的情况，我认为是未来 AI 系统控制问题的当代类比，我们现在就可以开始研究。引导行为的困难，以及确保如果我们把 AI 系统推向一个方向，它不会在另一个我们不希望的方向上被推动，我认为这是未来问题的早期迹象。如果我们能很好地解决这个问题，比如你让模型制造和分发天花，它拒绝了，但它愿意帮助你的研究生病毒学课程，我们如何同时实现这两点？这很难，很容易偏向一边或另一边，这是一个多维问题。所以我认为塑造模型个性的这些问题非常困难。我认为我们在这方面做得并不完美。我认为我们实际上在所有 AI 公司中做得最好，但仍然远非完美。如果我们能把这个做好，如果我们能在这种高度受控的当代环境中控制假阳性和假阴性，我们将来就能更好地应对，当我们的担忧是：模型会变得超级自主吗？它们能制造非常危险的东西吗？它们能自主地建立整个公司，而这些公司是对齐的吗？所以我认为当前的任务既是疫苗，也是为未来做的良好练习。

So they're all these trade-offs. For example, another thing is if there was a period during which models, ours and I think others as well, were too verbose. They would repeat themselves, they would say too much. You can cut down on the verbosity by penalizing the models for talking too long. What happens when you do that in a crude way is when the models are coding, sometimes they'll say 'the code goes here' because they've learned that's a way to economize. So that leads the model to be so-called lazy in coding, where they're just like 'ah, you can finish the rest of it.' It's not because we want to save on compute or because the models are lazy during winter break or any of the other conspiracy theories. It's actually just very hard to control the behavior of the model, to steer the behavior of the model in all circumstances at once. There's this whack-a-mole aspect where you push on one thing and other things start to move as well that you may not even notice or measure. One of the reasons I care so much about grand alignment of these AI systems in the future is actually these systems are quite unpredictable, quite hard to steer and control. This version we're seeing today of you make one thing better, it makes another thing worse, I think that's like a present-day analog of future control problems in AI systems that we can start to study today. That difficulty in steering the behavior and making sure that if we push an AI system in one direction it doesn't push it in another direction in some other ways that we didn't want, I think that's an early sign of things to come. If we can do a good job of solving this problem, like you ask the model to make and distribute smallpox and it says no, but it's willing to help you in your graduate-level virology class, how do we get both of those things at once? It's hard, it's very easy to go to one side or the other, and it's a multi-dimensional problem. So I think these questions of shaping the model's personality are very hard. I think we haven't done perfectly on them. I think we've actually done the best of all the AI companies, but still so far from perfect. If we can get this right, if we can control the false positives and false negatives in this very controlled present-day environment, we'll be much better at doing it for the future when our worry is: will the models be super autonomous? Will they be able to make very dangerous things? Will they be able to autonomously build whole companies, and are those companies aligned? So I think of this present task as both a vaccine and good practice for the future.

目前收集用户反馈的最佳方式是什么？不是轶事数据，而是关于痛点或相反——积极方面的大规模数据？是内部测试吗？还是特定群体测试？什么有效？

What's the current best way of gathering sort of user feedback, like not anecdotal data but just large-scale data about pain points or the opposite of pain points, positive things, so on? Is it internal testing? Is it a specific group testing? What works?

通常，我们会进行内部模型攻击测试，Anthropic 的所有人——将近一千人——都试图破坏模型，以各种方式与它互动。我们有一套评估体系。例如，我们甚至有一个“当然”评估，因为有一次我们的模型有一个烦人的习惯，它会用“当然我可以帮你”、“当然我很乐意”、“当然这是正确的”来回答各种问题。所以我们有了一个“当然”评估，即模型说“当然”的频率。但这只是打地鼠：如果它从“当然”换成“绝对”呢？所以每次我们添加新的评估，同时我们总是评估所有旧的东西，所以我们有数百个这样的评估。但我们发现，没有什么能替代人类与它的互动。所以这很像普通的产品开发过程。我们有数百人在 Anthropic 内部攻击模型，然后我们进行外部 A/B 测试。有时我们会与承包商进行测试；我们付钱给承包商让他们与模型互动。你把所有这些放在一起，仍然不完美。你仍然会看到你不希望看到的行为。你仍然会看到模型拒绝一些完全没有理由拒绝的事情。但我认为，试图解决这个挑战——试图阻止模型做那些每个人都同意它不应该做的真正坏事，比如儿童虐待材料，每个人都同意模型不应该做——但同时，它不会以这些愚蠢的方式拒绝。我认为尽可能精细地划定这条线，接近完美，仍然是一个挑战。我们每天都在进步，但还有很多问题需要解决。再次，我认为这是未来引导更强大模型所面临挑战的一个指标。

Typically, we'll have internal model bashings where all of Anthropic—almost a thousand people—just try and break the model, interact with it in various ways. We have a suite of evals. For example, we even had a 'certainly' eval because at one point our model had this annoying tick where it would respond to a wide range of questions by saying 'Certainly I can help you with that,' 'Certainly I would be happy to do that,' 'Certainly this is correct.' So we had a 'certainly' eval, which is how often does the model say 'certainly'? But this is just whack-a-mole: what if it switches from 'certainly' to 'definitely'? So every time we add a new eval, and we're always evaluating for all the old things, so we have hundreds of these evaluations. But we find that there's no substitute for human interaction with it. So it's very much like the ordinary product development process. We have hundreds of people within Anthropic bash the model, then we do external A/B tests. Sometimes we'll run tests with contractors; we pay contractors to interact with the model. You put all of these things together, and it's still not perfect. You still see behaviors that you don't quite want to see. You still see the model refusing things that it just doesn't make sense to refuse. But I think trying to solve this challenge—trying to stop the model from doing genuinely bad things that everyone agrees it shouldn't do, like child abuse material, everyone agrees the model shouldn't do that—but at the same time, it doesn't refuse in these dumb and stupid ways. I think drawing that line as finely as possible, approaching perfectly, is still a challenge. We're getting better at it every day, but there's a lot to be solved. Again, I would point to that as an indicator of a challenge ahead in terms of steering much more powerful models.

你认为 Claude 4.0 会出来吗？

Do you think Claude 4.0 is ever coming out?

我不想对任何命名方案做出承诺，因为如果我说我们明年会推出 Claude 4，然后我们决定因为一种新型模型而重新开始，我不想承诺。在正常的业务过程中，我预计 Claude 4 会在 Claude 3.5 之后推出，但在这个疯狂的领域你永远不知道。但 Scaling（规模扩张）的想法仍在继续。我们肯定会推出比现有模型更强大的模型。这是肯定的。如果没有，我们作为公司就彻底失败了。

I don't want to commit to any naming scheme because if I say we're gonna have Claude 4 next year, and then we decide that we should start over because there's a new type of model, I don't want to commit to it. I would expect in a normal course of business that Claude 4 would come after Claude 3.5, but you never know in this wacky field. But the idea of scaling is continuing. There will definitely be more powerful models coming from us than the models that exist today. That is certain. If there aren't, we've deeply failed as a company.

你能解释一下负责任扩展政策和 AI 安全等级标准（ASL 等级）吗？

Can you explain the responsible scaling policy and the AI safety level standards, ASL levels?

尽管我对这些模型的好处感到兴奋，如果我们谈论《爱的机器》我们会讨论这个，但我担心风险，而且我继续担心风险。没有人应该认为《爱的机器》意味着我不再担心这些系统的风险。

As much as I'm excited about the benefits of these models, and we'll talk about that if we talk about Machines of Loving Grace, I'm worried about the risk and I continue to be worried about the risks. No one should think that Machines of Loving Grace was me saying I'm no longer worried about the risks of these systems.

我认为模型是一体两面。模型的能力及其在生物学、神经科学、经济发展、政府治理与和平等大部分经济领域解决问题的力量，也伴随着风险。能力越大，责任越大。这两者是相伴的。强大的东西既能做好事，也能做坏事。我认为这些风险可以分为几个不同类别。我考虑的两大风险——不是说当前的风险不重要——但当我想到最宏观层面可能发生的事情时，第一个是我所说的灾难性滥用。这是模型在网络、生物、放射、核等领域的滥用——如果真出大问题，可能伤害甚至杀死成千上万人。这是预防的首要任务。这里我只想做一个简单的观察：如果看看今天世界上做过真正坏事的人，我认为人类一直受到一个事实的保护，那就是真正聪明、受过良好教育的人与想干真正可怕事情的人之间的重叠通常很小。假设我是一个拥有该领域博士学位的人，有一份高薪工作，有太多东西可失去。我为什么要——即使假设我完全邪恶，而大多数人并非如此——这样的人为什么要冒着生命、遗产和声誉的风险去做真正邪恶的事？如果我们有更多这样的人，世界会危险得多。所以我担心的是，作为一个更智能的智能体，AI 可能打破这种相关性。我对此确实有严重担忧。我相信我们可以预防这些担忧，但作为对《仁慈机器》的回应，我想说风险仍然严重。第二类风险是自主性风险：模型可能自行其是，特别是当我们赋予它们比过去更多的自主权，让它们监督更广泛的任务，比如编写整个代码库，甚至有一天有效运营整个公司，在足够长的绳子上，它们是否在做我们真正想做的事？要详细理解它们在做什么都非常困难，更不用说控制了。早期迹象表明，很难完美划定模型该做和不该做的事之间的界限。偏向一边，你会得到烦人且无用的东西；偏向另一边，你会得到其他行为。修复一个问题会引发其他问题。我们在这方面越来越擅长。我不认为这是无法解决的问题。我认为这是一门科学，就像飞机、汽车或药物的安全一样。我不认为我们遗漏了什么大问题；我只是需要更好地控制这些模型。这就是我担心的两类风险，我们的《负责任扩展计划》——我意识到这是对你问题的冗长回答，但我喜欢——我们的《负责任扩展计划》旨在应对这两类风险。每次我们开发新模型，我们基本上都会测试它做这两类坏事的能力。

Models, I think they're two sides of the same coin. The power of the models and their ability to solve all these problems in biology, neuroscience, economic development, government, governance, and peace, large parts of the economy, those come with risks as well. With great power comes great responsibility. The two are paired. Things that are powerful can do good things and they can do bad things. I think of those risks as being in several different categories. Perhaps the two biggest risks that I think about—and that's not to say that there aren't risks today that are important—but when I think of the things that would happen on the grandest scale, one is what I call catastrophic misuse. These are misuse of the models in domains like cyber, bio, radiological, nuclear—things that could harm or even kill thousands, even millions of people if they really go wrong. These are the number one priority to prevent. Here I would just make a simple observation: if I look today at people who have done really bad things in the world, I think humanity has been protected by the fact that the overlap between really smart, well-educated people and people who want to do really horrific things has generally been small. Let's say I'm someone with a PhD in this field, I have a well-paying job, there's so much to lose. Why would I want to—even assuming I'm completely evil, which most people are not—why would such a person risk their life, risk their legacy, their reputation, to do something truly evil? If we had a lot more people like that, the world would be a much more dangerous place. So my worry is that by being a much more intelligent agent, AI could break that correlation. I do have serious worries about that. I believe we can prevent those worries, but as a counterpoint to Machines of Loving Grace, I want to say that there are still serious risks. The second range of risks would be the autonomy risks: the idea that models might on their own, particularly as we give them more agency than they've had in the past, particularly as we give them supervision over wider tasks like writing whole code bases or someday even effectively operating entire companies, on a long enough leash, are they doing what we really want them to do? It's very difficult to even understand in detail what they're doing, let alone control it. These early signs show that it's hard to perfectly draw the boundary between things the model should do and things it shouldn't do. If you go to one side, you get things that are annoying and useless; go to the other side, you get other behaviors. Fixing one thing creates other problems. We're getting better and better at solving this. I don't think this is an unsolvable problem. I think this is a science, like the safety of airplanes or cars or drugs. I don't think there's any big thing we're missing; I just think we need to get better at controlling these models. So these are the two risks I'm worried about, and our Responsible Scaling Plan, which I'll recognize is a very long-winded answer to your question—I love it—our Responsible Scaling Plan is designed to address these two types of risks. Every time we develop a new model, we basically test it for its ability to do both of these bad things.

我喜欢这个回答。

I love it.

如果稍微退一步，我认为我们面临一个有趣的困境：AI 系统还不够强大，不足以带来这些灾难。我不知道它们是否会带来这些灾难；有可能不会。但担忧的理由、风险的理由足够强烈，我们应该现在就行动。而且它们进步得非常非常快。我在参议院作证时说，我们可能在两到三年内面临严重的生物风险。那大约是一年前；事情进展得很快。所以我们面临的情况是：解决这些风险异常困难，因为它们今天还不存在；它们就像幽灵，但正以极快速度向我们逼近，因为模型进步太快。那么，如何应对今天不存在、但正快速逼近的东西？我们想出的解决方案——与 METR 组织和 Paul Christiano 等人合作——是：你需要测试来告诉你风险何时接近。你需要一个早期预警系统。所以每次我们有一个新模型，我们都会测试它执行这些 CBRN 任务的能力，以及测试它自主执行任务的能力。在我们最近一两个月发布的最新版 RSP 中，测试自主性风险的方式是模型自身进行 AI 研究某些方面的能力。当 AI 模型能够进行 AI 研究时，它们就变得真正自主了。这个阈值在其他许多方面也很重要。那么，我们如何处理这些任务？RSP 基本上开发了我们所谓的 if-then 结构：如果模型通过某个能力阈值，那么我们就对它们施加一套特定的安全要求。今天的模型被称为 ASL-2 模型。ASL-1 适用于明显不构成自主性或滥用风险的系统。例如，像 Deep Blue 这样的下棋机器人就是 ASL-1；显然你不能将 Deep Blue 用于下棋以外的任何事情。它就是为了下棋而设计的。没有人会用它来进行一次高超的网络攻击，或者失控并接管世界。ASL-2 是今天的 AI 系统，我们测量后认为这些系统根本不够聪明，无法自主复制或执行一系列任务，也无法提供超出谷歌搜索所能知道的关于 CBRN 风险和如何制造 CBRN 武器的有意义信息。事实上，有时它们确实提供信息，但不超过搜索引擎，不能以拼接的方式，不能以端到端足够危险的方式。所以 ASL-3 将是模型足够有帮助以至于……

If I were to back up a little bit, I think we have an interesting dilemma with AI systems where they're not yet powerful enough to present these catastrophes. I don't know that they'll ever present these catastrophes; it's possible they won't. But the case for worry, the case for risk, is strong enough that we should act now. And they're getting better very, very fast. I testified in the Senate that we might have serious bio risks within two to three years. That was about a year ago; things have proceeded at a pace. So we have this thing where it's surprisingly hard to address these risks because they're not here today; they don't exist. They're like ghosts, but they're coming at us so fast because the models are improving so fast. So how do you deal with something that's not here today, doesn't exist, but is coming at us very fast? The solution we came up with for that, in collaboration with people like the organization METR and Paul Christiano, is: what you need for that are tests to tell you when the risk is getting close. You need an early warning system. So every time we have a new model, we test it for its capability to do these CBRN tasks, as well as testing it for how capable it is of doing tasks autonomously on its own. In the latest version of our RSP, which we released in the last month or two, the way we test autonomy risks is the model's ability to do aspects of AI research itself. When the AI models can do AI research, they become kind of truly autonomous. That threshold is important for a bunch of other ways. So what do we then do with these tasks? The RSP basically develops what we've called an if-then structure: if the models pass a certain capability, then we impose a certain set of safety and security requirements on them. Today's models are what's called ASL-2 models. ASL-1 is for systems that manifestly don't pose any risk of autonomy or misuse. For example, a chess-playing bot like Deep Blue would be ASL-1; it's manifestly the case that you can't use Deep Blue for anything other than chess. It was just designed for chess. No one's going to use it to conduct a masterful cyber attack or to run wild and take over the world. ASL-2 is today's AI systems, where we've measured them and we think these systems are simply not smart enough to autonomously self-replicate or conduct a bunch of tasks, and also not smart enough to provide meaningful information about CBRN risks and how to build CBRN weapons above and beyond what can be known from looking at Google. In fact, sometimes they do provide information, but not above and beyond a search engine, not in a way that can be stitched together, not in a way that end-to-end is dangerous enough. So ASL-3 is going to be the point at which the models are helpful enough to...

你认为 ASL3 的时间线是怎样的，即几个触发条件被触发？ASL4 的时间线呢？

What do you think the timeline for ASL3 is, where several of the triggers are fired, and what do you think the timeline is for ASL4?

是的，公司内部对此争论激烈。我们正在积极准备 ASL3 的安全措施和部署措施。我不打算详细说明，但我们在两方面都取得了很大进展，并且我们准备很快就能就绪。如果明年达到 ASL3，我一点也不会惊讶。有人担心我们甚至可能今年就达到，这仍然有可能。很难说，但如果到 2030 年，我会非常惊讶。我认为要早得多。

Yeah, so that is hotly debated within the company. We are working actively to prepare ASL3 security measures as well as ASL3 deployment measures. I'm not going to go into detail, but we've made a lot of progress on both, and we're prepared to be ready quite soon. I would not be surprised at all if we hit ASL3 next year. There was some concern that we might even hit it this year; that's still possible. It's very hard to say, but I would be very surprised if it was like 2030. I think it's much sooner than that.

所以有检测协议，即 if-then，然后有如何响应的协议。后者有多难？

So there are protocols for detecting it, the if-then, and then there are protocols for how to respond to it. How difficult is the latter?

我认为对于 ASL3，主要是安全问题和部署模型时与非常狭窄领域相关的过滤器。因为在 ASL3，模型还不是自主的，所以即使内部部署，你也不必担心模型本身行为不当。所以我认为 ASL3 的措施虽然严格，但更容易推理。一旦到了 ASL4，我们开始担心模型足够聪明，可能会在测试中偷懒，可能不会如实报告测试结果。我们有一些关于潜伏智能体的结果，还有一篇最近的论文关于模型是否可能误导对其能力偷懒的尝试，表现得比实际能力差。所以我认为对于 ASL4，一个重要组成部分是使用除了与模型交互之外的其他方法，例如可解释性或隐藏的思维链，你必须查看模型内部并通过某种其他机制验证，该机制不像模型所说的话那样容易被篡改，从而确认模型确实具有某些属性。所以我们仍在研究 ASL4。RSP 的一个特性是，我们在达到 ASL3 之前不会指定 ASL4，我认为这被证明是明智的决定，因为即使对于 ASL3，也很难详细了解这些内容，我们希望尽可能多的时间来把这些事情做好。

I think for ASL3, it's primarily about security and about filters on the model relating to a very narrow set of areas when we deploy the model. Because at ASL3, the model isn't autonomous yet, so you don't have to worry about the model itself behaving in a bad way even when it's deployed internally. So I think the ASL3 measures are rigorous but easier to reason about. Once we get to ASL4, we start to have worries about the models being smart enough that they might sandbag tests, they might not tell the truth about tests. We had some results come out about sleeper agents, and there was a more recent paper about whether models can mislead attempts to sandbag their own abilities, present themselves as being less capable than they are. So I think with ASL4, there's going to be an important component of using other things than just interacting with the models, for example interpretability or hidden chains of thought, where you have to look inside the model and verify via some other mechanism that is not as easily corrupted as what the model says, that the model indeed has some property. So we're still working on ASL4. One of the properties of the RSP is that we don't specify ASL4 until we've hit ASL3, and I think that's proven to be a wise decision because even with ASL3, it's hard to know this stuff in detail, and we want to take as much time as we can to get these things right.

所以对于 ASL3，恶意行为者将是人类，对吧？所以那里有点……对于 ASL4，我认为是两者兼有。欺骗，这就是机械可解释性发挥作用的地方。希望用于此的技术不会让模型访问到。

So for ASL3, the bad actor will be the humans, yes? And so there it's a little bit more... For ASL4, it's both, I think. Deception, and that's where mechanistic interpretability comes into play. And hopefully the techniques used for that are not made accessible to the model.

是的，我的意思是，当然你可以将机械可解释性连接到模型本身，但那样你就失去了它作为模型状态可靠指标的作用。你可以想到许多奇特的方式，它也可能不可靠，比如如果模型足够聪明，可以跳转到其他计算机并读取你查看其内部状态的代码。我们考虑过其中一些；我认为它们足够奇特，有办法使它们不太可能发生。但一般来说，你希望将机械可解释性保留为一种验证集或测试集，与模型的训练过程分开。

Yeah, I mean, of course you can hook up the mechanistic interpretability to the model itself, but then you've kind of lost it as a reliable indicator of the model state. There are a bunch of exotic ways you can think of that it might also not be reliable, like if the model gets smart enough that it can jump computers and read the code where you're looking at its internal state. We've thought about some of those; I think they're exotic enough that there are ways to render them unlikely. But yeah, generally you want to preserve mechanistic interpretability as a kind of verification set or test set that's separate from the training process of the model.

我认为随着这些模型在对话方面变得越来越好，越来越聪明，社会工程也成为一种威胁，因为它们可以开始对公司内部的工程师非常有说服力。

I think as these models become better and better at conversation and become smarter, social engineering becomes a threat too, because they can start being very convincing to the engineers inside companies.

哦，是的，实际上，你知道，我们在生活中见过很多煽动性的例子……

Oh yeah, it's actually like, you know, we've seen lots of examples of demagoguery in our life from...

增强非国家行为者的能力。国家行为者已经能够以高熟练度做很多非常危险和破坏性的事情。区别在于非国家行为者没有能力做到。所以当我们达到 ASL3 时，我们将采取特殊的安全预防措施，旨在防止非国家行为者窃取模型以及部署中的模型被滥用。我们将不得不针对这些特定领域增强过滤器：网络、生物、核和模型自主性，后者更多是模型自身做坏事的风险，而非滥用风险。ASL4：达到这些模型能够增强全知国家行为者的能力，和/或成为此类风险的主要来源。如果你想从事此类风险，主要方式是通过模型。然后我认为 ASL4 在自主性方面是 AI 模型在 AI 研究能力上的一定加速。然后 ASL5 是模型真正有能力，在完成这些任务的能力上超越人类。所以 if-then 结构承诺的要点基本上是：看，我不知道。我与这些模型合作多年，多年来一直担心风险。实际上，喊“狼来了”是危险的。实际上，说这个模型有风险是危险的，人们看着它说这显然不危险。风险的精妙之处今天不在这里，但它正快速向我们袭来。你如何处理？这对风险规划者来说非常棘手。所以这个 if-then 结构基本上说：我们不想激怒很多人，我们不想通过给今天不危险的模型施加这些繁重的负担来损害我们在对话中的地位。所以 if-then 触发承诺基本上是一种处理方式：当你能够证明模型危险时，就严厉打压。当然，随之而来的是要有足够的缓冲阈值，这样你就不太可能错过危险。这不是一个完美的框架；我们每次都必须改变它。我们几周前刚发布了一个新的，未来可能每年发布多次，因为从技术、组织、研究角度来看，很难把这些政策做对。但这就是提议：if-then 承诺和触发，以最小化现在的负担和误报，但在危险真正来临时做出适当反应。

Enhance the capabilities of non-state actors. State actors can already do a lot, unfortunately, to a high level of proficiency, a lot of these very dangerous and destructive things. The difference is that non-state actors are not capable of it. And so when we get to ASL3, we'll take special security precautions designed to be sufficient to prevent theft of the model by non-state actors and misuse of the model as it's deployed. We'll have to have enhanced filters targeted at these particular areas: cyber, bio, nuclear, and model autonomy, which is less a misuse risk and more a risk of the model doing bad things itself. ASL4: getting to the point where these models could enhance the capability of an all-knowledgeable state actor and/or become the main source of such a risk. If you wanted to engage in such a risk, the main way you would do it is through a model. And then I think ASL4 on the autonomy side is some amount of acceleration in AI research capabilities with an AI model. And then ASL5 is where we would get to the models that are truly capable, that could exceed humanity in their ability to do any of these tasks. So the point of the if-then structure commitment is basically to say: look, I don't know. I've been working with these models for many years and I've been worried about risk for many years. It's actually kind of dangerous to cry wolf. It's actually kind of dangerous to say this model is risky, and people look at it and say this is manifestly not dangerous. The delicacy of the risk isn't here today, but it's coming at us fast. How do you deal with that? It's really vexing to a risk planner to deal with it. So this if-then structure basically says: we don't want to antagonize a bunch of people, we don't want to harm our own ability to have a place in the conversation by imposing these very onerous burdens on models that are not dangerous today. So the if-then trigger commitment is basically a way to deal with this: you clamp down hard when you can show that the model is dangerous. And of course, what has to come with that is enough of a buffer threshold that you're not at high risk of missing the danger. It's not a perfect framework; we've had to change it every time. We came out with a new one just a few weeks ago, and probably going forward we might release new ones multiple times a year because it's hard to get these policies right—technically, organizationally, from a research perspective. But that is the proposal: if-then commitments and triggers in order to minimize burdens and false alarms now, but really react appropriately when the dangers are here.

我们来谈谈计算机使用。这在我看来非常令人兴奋，你可以直接给 Claude 一个任务，它就会采取一系列行动，自己想办法解决，并通过截图访问你的电脑。那么你能解释一下这是如何工作的，以及未来的发展方向吗？

Let's talk about computer use. That seems to me super exciting that you can just give Claude a task and it takes a bunch of actions, figures it out, and has access to your computer through screenshots. So can you explain how that works and where that's headed?

是的，其实相对简单。Claude 从三月份的 Claude 3 开始就具备了分析图像并用文本回应的能力。我们新增的唯一一点是，这些图像可以是电脑截图，然后我们训练模型给出屏幕上的点击位置和/或键盘按键，以便采取行动。事实证明，不需要太多额外训练，模型就能很好地完成这项任务。这是一个很好的泛化例子。你知道，人们有时会说，如果你到达了近地轨道，你就已经到任何地方的一半了，因为脱离引力井需要付出很多。如果你有一个强大的预训练模型，我觉得在智能空间里你也已经到任何地方的一半了。所以实际上，让 Claude 做到这一点并不需要太多努力。你可以把它放在一个循环里：给模型一张截图，告诉它点击哪里，再给下一张截图，告诉它点击哪里，这就变成了模型几乎像 3D 视频一样的完整交互，它能够完成所有这些任务。我们展示了一些演示，比如填写电子表格、与网站交互、打开各种程序、不同的操作系统——Windows、Linux、Mac。所以我认为这一切都非常令人兴奋。我要说的是，虽然理论上你可以通过给模型 API 来驱动电脑屏幕做到同样的事情，但这确实降低了门槛。有很多人要么无法使用那些 API，要么需要很长时间才能学会。屏幕只是一个更易交互的通用界面。所以我预计随着时间的推移，这将降低很多门槛。说实话，当前的模型还有很多不足之处，我们在博客中也坦诚地提到了这一点。它会犯错，会点错，我们小心地警告人们：嘿，你不能让这个东西在你的电脑上运行几分钟又几分钟。你必须给它设定边界和护栏。我认为这也是我们首先以 API 形式发布，而不是直接交给消费者并让它控制电脑的原因之一。但我确实觉得让这些能力面世很重要。随着模型变得更强大，我们将不得不思考如何安全地使用这些能力，如何防止它们被滥用。我认为在能力还有限的时候发布模型，对做到这一点非常有帮助。自发布以来，许多客户——我认为 Replit 可能是部署最快的之一——已经以各种方式使用了它。人们为 Windows 桌面、Mac、Linux 机器搭建了演示。所以是的，这非常令人兴奋。我认为和其他任何事情一样，它带来了新的令人兴奋的能力，然后有了这些新的令人兴奋的能力，我们必须思考如何让模型安全、可靠，做人类希望它们做的事情。每件事都是同样的故事，对吧？同样的张力。但这里的用例可能性是不可思议的。

Yeah, it's actually relatively simple. So Claude has had, for a long time since Claude 3 back in March, the ability to analyze images and respond to them with text. The only new thing we added is those images can be screenshots of a computer, and in response we train the model to give a location on the screen where you can click and/or buttons on the keyboard you can press in order to take action. And it turns out that with actually not all that much additional training, the models can get quite good at that task. It's a good example of generalization. You know, people sometimes say if you get to low Earth orbit, you're like halfway to anywhere, right? Because of how much it takes to escape the gravity well. If you have a strong pre-trained model, I feel like you're halfway to anywhere in terms of the intelligence space. And so actually it didn't take all that much to get Claude to do this. And you can just set that in a loop: give the model a screenshot, tell it what to click on, give it the next screenshot, tell it what to click on, and that turns into a full kind of almost 3D video interaction of the model, and it's able to do all of these tasks. Right, we showed these demos where it's able to fill out spreadsheets, interact with a website, open all kinds of programs, different operating systems—Windows, Linux, Mac. So I think all of that is very exciting. I will say, while in theory there's nothing you could do there that you couldn't have done through just giving the model the API to drive the computer screen, this really lowers the barrier. And there's a lot of folks who either aren't in a position to interact with those APIs or it takes them a long time to do so. The screen is just a universal interface that's a lot easier to interact with. So I expect over time this is going to lower a bunch of barriers. Now honestly, the current model still leaves a lot to be desired, and we were honest about that in the blog. Right, it makes mistakes, it misclicks, and we were careful to warn people: hey, this thing isn't something you can just leave to run on your computer for minutes and minutes. You've got to give this thing boundaries and guardrails. And I think that's one of the reasons we released it first in an API form rather than just handing it to the consumer and giving it control of their computer. But I definitely feel that it's important to get these capabilities out there. As models get more powerful, we're going to have to grapple with how to use these capabilities safely, how to prevent them from being abused. And I think releasing the model while the capabilities are still limited is very helpful in terms of doing that. Since it's been released, a number of customers—I think Replit was maybe one of the quickest to deploy things—have made use of it in various ways. People have hooked up demos for Windows desktops, Macs, Linux machines. So yeah, it's been very exciting. I think as with anything else, it comes with new exciting abilities, and then with those new exciting abilities we have to think about how to make the model safe, reliable, do what humans want them to do. It's the same story for everything, right? Same tension. But the possibility of use cases here is incredible.

那么未来要让它真正好用，需要做多少工作？你需要在多大程度上超越预训练模型的能力，进行更多的后训练、基于人类反馈的强化学习（RLHF）、监督微调，或者专门为智能体任务生成合成数据？

So how much to make it work really well in the future? How much do you have to specially go beyond what the pre-trained model is doing, do more post-training, RLHF, or supervised fine-tuning, or synthetic data just for the agent stuff?

我认为从高层次来说，我们打算继续大力投资让模型变得更好。我们看看一些基准测试，之前的模型可能只有 6% 的成功率，而现在我们的模型有 14% 或 22%。是的，我们想要达到人类级别的 80-90% 可靠性，就像其他领域一样。我们正处在与其他基准测试相同的曲线上。我认为一年后，模型可以非常非常可靠地做到这一点，但你必须从某个地方开始。

I think speaking at a high level, it's our intention to keep investing a lot in making the model better. I think we look at some of the benchmarks where previous models were like, oh could do it 6% of the time, and now our model does it 14 or 22% of the time. And yeah, we want to get up to human-level reliability of 80-90%, just like anywhere else. Right, we're on the same curve that we were on with other benchmarks. I think a year from now the models can do this very, very reliably, but you've got to start somewhere.

那么你认为基本上用你现在做的方法就能达到人类级别的 90%，还是说必须专门为计算机使用做特殊处理？

So you think it's possible to get to the human level 90% basically doing the same thing you're doing now, or does it have to be special for computer use?

我的意思是，这取决于你说的特殊是什么意思。但我通常认为，我们一直在用来训练当前模型的同类技术——我预计在这些技术上加倍投入，就像我们在代码、通用模型、其他能力如图像输入、语音方面所做的那样——我预计这些技术会像在其他地方一样在这里扩展。

I mean, it depends what you mean by special and all. But I generally think the same kinds of techniques that we've been using to train the current model—I expect that doubling down on those techniques in the same way that we have for code, for models in general, for other capabilities like image input, voice—I expect those same techniques will scale here as they have everywhere else.

但这赋予了 Claude 行动的能力。所以你可以做很多非常强大的事情，但也可能造成很大的损害。

But this is giving sort of the power of action to Claude. And so you could do a lot of really powerful things, but you could do a lot of damage also.

是的，是的，不，我们一直很清楚这一点。听着，我的观点实际上是，计算机使用并不是一个根本性的新能力，不像 CBRN 或自主能力那样。它更像是为模型使用和应用其现有能力打开了窗口。所以我们对此的思考，回到我们的 RSP，这个模型所做的任何事情本身并不会从 RSP 的角度增加风险。但随着模型变得更强大，拥有这种能力可能会让它更可怕，一旦它具备了在 ASL-3 和 ASL-4 级别做某事的认知能力。这可能是让它不受约束的东西。所以未来，这种交互方式肯定是我们已经测试过的，并且我们……

Yeah, yeah, no, and we've been very aware of that. Look, my view actually is computer use isn't a fundamentally new capability like the CBRN or autonomy capabilities are. It's more like it kind of opens the aperture for the model to use and apply its existing abilities. And so the way we think about it, going back to our RSP, nothing that this model is doing inherently increases the risk from an RSP perspective. But as the models get more powerful, having this capability may make it scarier once it has the cognitive capability to do something at the ASL-3 and ASL-4 level. This may be the thing that kind of unbinds it from doing so. So going forward, certainly this modality of interaction is something we have tested for and that we...

我们未来会继续测试。我认为最好在模型变得超级强大之前学习和探索这种能力。而且有很多有趣的攻击，比如提示注入，因为现在你扩大了孔径，所以可以通过屏幕上的内容进行提示注入。所以如果这变得越来越有用，那么向模型中注入东西的收益就越来越大。如果它访问某个网页，可能是无害的东西，比如广告，也可能是有害的东西，对吧？

Will continue to test for in our going forward. I think it's probably better to learn and explore this capability before the model is super capable. And there are a lot of interesting attacks like prompt injection because now you've widened the aperture so you can prompt inject through stuff on screen. So if this becomes more and more useful, then there's more and more benefit to inject stuff into the model. If it goes to a certain web page, it could be harmless stuff like advertisements or it could be harmful stuff, right?

是的，我们考虑了很多关于垃圾邮件、捕获、大规模 C 的问题。所有——如果有一个秘密我要告诉你：如果你发明了一项新技术，不一定是最严重的滥用，但你首先会看到的是诈骗。就是小骗局。这就像人们互相欺骗一样古老。这是自古以来就有的东西。每次你都得处理它，说起来有点傻，但这是真的。垃圾邮件也是如此。随着它变得越来越智能，有很多——就像我说的，世界上有很多小罪犯，每一项新技术都是小罪犯做蠢事和恶意行为的新途径。

Yeah, I mean, we thought a lot about spam, capture, mass C. There's all—every, if one secret I'll tell you: if you've invented a new technology, not necessarily the biggest misuse, but the first misuse you'll see is scams. Just petty scams. It's like a thing as old as people scamming each other. It's a thing as old as time. And every time you've got to deal with it, it's almost silly to say, but it's true. And spam in general is a thing. As it gets more and more intelligent, there are a lot of—like I said, there are a lot of petty criminals in the world, and every new technology is like a new way for petty criminals to do something stupid and malicious.

有没有关于沙盒化的想法？沙盒化任务有多难？

Are there any ideas about sandboxing it? How difficult is the sandboxing task?

是的，我们在训练期间进行沙盒化。例如，在训练期间，我们没有让模型接触互联网。我认为这在训练期间可能是个坏主意，因为模型可能会改变其策略，改变它正在做的事情，并对现实世界产生影响。在实际部署模型时，这取决于应用场景。有时你希望模型在现实世界中做某事，但你当然可以在外部设置护栏。你可以说，好吧，这个模型不会将数据从我的电脑或我的网络服务器移动到任何其他地方。现在，当你再次谈论沙盒化时，当我们达到 ASL-4 时，这些预防措施都不再适用。当你谈论 ASL-4 时，模型可能足够聪明，可以突破任何盒子。所以我们需要考虑机制可解释性。如果我们想要一个沙盒，它需要在数学上可证明是可靠的。但这与我们今天处理的模型完全不同。

Yeah, we sandbox during training. For example, during training we didn't expose the model to the internet. I think that's probably a bad idea during training because the model can be changing its policy, it can be changing what it's doing, and it's having an effect in the real world. In terms of actually deploying the model, it kind of depends on the application. Sometimes you want the model to do something in the real world, but of course you can always put guardrails on the outside. You can say, okay, this model is not going to move data from my computer or my web server to anywhere else. Now, when you talk about sandboxing again, when we get to ASL-4, none of these precautions are going to make sense there. When you talk about ASL-4, the model could be smart enough to break out of any box. So there we need to think about mechanistic interpretability. If we're going to have a sandbox, it would need to be mathematically provably sound. But that's a whole different world than what we're dealing with with the models today.

建造一个 ASL-4 AI 系统无法逃脱的盒子的科学——我认为这可能不是正确的方法。我认为正确的方法，不是拥有一个不对齐的东西并试图阻止它逃脱，而是以正确的方式设计模型，或者有一个循环，你可以查看模型内部并验证属性，这给了你迭代并真正做对的机会。我认为遏制坏模型比拥有好模型要糟糕得多。

The science of building a box from which an ASL-4 AI system cannot escape—I think it's probably not the right approach. I think the right approach, instead of having something unaligned that you're trying to prevent it from escaping, is to just design the model the right way, or have a loop where you look inside the model and you're able to verify properties, and that gives you an opportunity to iterate and actually get it right. I think containing bad models is a much worse solution than having good models.

让我问问关于监管的问题。监管在保持 AI 安全方面扮演什么角色？例如，他描述了加州 AI 监管法案 SB 1047，该法案最终被州长否决。这个法案的优缺点是什么？

Let me ask about regulation. What's the role of regulation in keeping AI safe? So for example, he described California AI regulation bill SB 1047 that was ultimately vetoed by the governor. What are the pros and cons of this bill?

总的来说，是的。我们最终对该法案提出了一些建议，其中一些被采纳了。我们最终对该法案持相当积极的态度。它确实还有一些缺点，当然它被否决了。我认为在高层面上，该法案背后的一些关键思想与我们的 RSP 背后的思想相似。我认为某个司法管辖区，无论是加州、联邦政府还是其他国家或其他州，通过这样的法规非常重要。我可以谈谈为什么我认为这如此重要。我对我们的 RSP 感觉良好；它并不完美，需要大量迭代，但它是一个很好的强制功能，让公司认真对待这些风险，将其纳入产品规划，真正使其成为 Anthropic 工作的核心部分，并确保所有一千人——现在 Anthropic 几乎有一千人——明白这是公司的最高优先事项之一，如果不是最高的话。但是，仍然有一些公司没有类似 RSP 的机制。OpenAI、Google 在 Anthropic 之后几个月确实采用了这些机制，但还有其他公司根本没有这些机制。所以如果一些公司采用这些机制而其他公司不采用，就会造成一种情况，即这些危险具有这样的特性：如果五家公司中有三家是安全的，而另外两家不安全，那也没用。这会产生负外部性，我认为缺乏统一性对我们这些投入大量精力认真思考这些程序的人来说是不公平的。第二件事是，我认为你不能相信这些公司会自愿遵守这些计划。我愿意相信 Anthropic 会尽我们所能，我们的 RSP 由我们的长期利益信托检查，所以我们尽一切努力遵守我们自己的 RSP。但你听到很多关于各种公司的事情，说他们会提供这么多算力，但他们没有，他们说他们会做这件事，但他们没有。我认为没有必要对公司所做的具体事情提起诉讼，但我认为这个广泛的原则是，如果没有东西监督他们，就没有东西监督我们整个行业，无法保证我们会做正确的事情，而且风险很高。所以我认为有一个每个人都遵守的统一标准很重要，并确保行业做到大多数行业已经说过重要并且已经说过他们一定会做的事情。对。有些人，我认为有一类人原则上反对监管。我理解这种想法来自哪里。如果你去欧洲，看到像 GDPR 这样的东西，看到他们做的其他一些事情，你知道一些……

General, yes. We ended up making some suggestions to the bill, and then some of those were adopted. We felt quite positively about the bill by the end of that. It did still have some downsides, and of course it got vetoed. I think at a high level, some of the key ideas behind the bill are similar to ideas behind our RSPs. I think it's very important that some jurisdiction, whether it's California or the federal government and/or other countries and other states, passes some regulation like this. I can talk through why I think that's so important. I feel good about our RSP; it's not perfect, it needs to be iterated on a lot, but it's been a good forcing function for getting the company to take these risks seriously, to put them into product planning, to really make them a central part of work at Anthropic, and to make sure that all the thousand people—and it's almost a thousand people now at Anthropic—understand that this is one of the highest priorities of the company, if not the highest priority. But one, there are still some companies that don't have RSP-like mechanisms. OpenAI, Google did adopt these mechanisms a couple months after Anthropic did, but there are other companies out there that don't have these mechanisms at all. So if some companies adopt these mechanisms and others don't, it's really going to create a situation where some of these dangers have the property that it doesn't matter if three out of five of the companies are being safe if the other two are being unsafe. It creates this negative externality, and I think the lack of uniformity is not fair to those of us who have put a lot of effort into being very thoughtful about these procedures. The second thing is, I don't think you can trust these companies to adhere to these voluntary plans in their own right. I like to think that Anthropic will do everything we can, that our RSP is checked by our long-term benefit trust, so we do everything we can to adhere to our own RSP. But you hear lots of things about various companies saying they would do this much compute and they didn't, they said they would do this thing and they didn't. I don't think it makes sense to litigate particular things that companies have done, but I think this broad principle that if there's nothing watching over them, there's nothing watching over us as an industry, there's no guarantee that we'll do the right thing, and the stakes are very high. So I think it's important to have a uniform standard that everyone follows, and to make sure that the industry does what a majority of the industry has already said is important and has already said that they definitely will do. Right. Some people, I think there's a class of people who are against regulation on principle. I understand where that comes from. If you go to Europe and you see something like GDPR, you see some of the other stuff that they've done, you know some...

有些部分不错，但有些部分确实不必要地繁琐，而且我认为公平地说，它确实减缓了创新。所以我理解人们为什么会有先验立场，理解他们为什么从那个立场出发。但再次强调，我认为 AI 是不同的。如果我们谈到我刚才提到的自主性和滥用的严重风险，我认为这些风险非同寻常，需要非同寻常的强力回应。所以我认为非常重要的一点是，我们需要一个大家都能支持的东西。我认为 SB 1047 的问题，尤其是最初版本，在于它有很多 SB 1047 的结构，但也有很多要么笨拙、要么会带来大量负担和麻烦、甚至可能错失风险目标的内容。你在 Twitter 上听不到这些，你只听到人们为任何监管欢呼，而反对者则编造出一些常常在智识上不诚实的论点，比如它会让我们搬离加州——但法案并不适用于总部在加州的公司，只适用于在加州开展业务的公司；或者它会损害开源生态系统，或者会导致各种问题。我认为这些大多是无稽之谈。但确实有更好的反对监管的论点。有一位叫 Dean Ball 的学者，他研究监管实施后的情况，以及监管如何可能自行其是或设计不当。所以我们的兴趣一直是：我们确实认为这个领域应该有监管，但我们希望成为一个确保监管是精准的、针对严重风险的、并且人们能够实际遵守的参与者。因为我认为监管倡导者没有充分理解的一点是：如果我们推出一个目标不精准、浪费大家时间的监管，结果会是人们说：‘看，这些安全风险，真是胡说八道。我刚刚不得不雇了 10 个律师来填这些表格，我不得不为一些明显不危险的东西运行所有这些测试。’六个月后，就会形成一股浪潮，我们最终会形成对监管的持久共识。所以我认为，那些想要真正问责的人的最大敌人就是设计糟糕的监管。我们需要真正把它做对。如果我能对倡导者说一件事，那就是我希望他们更好地理解这个动态。我们需要非常小心，并与那些实际有监管实施经验的人交谈。有这种经验的人明白要非常谨慎。如果这是一个不那么重要的问题，我可能根本反对监管。但我希望反对者理解的是，根本问题实际上是严重的。它们不是我和其他公司为了监管俘获而编造出来的，不是科幻幻想，不是任何这些东西。每隔几个月我们推出新模型时，我们都会测量这些模型的行为，它们在那些令人担忧的任务上变得越来越好，就像它们在好的、有价值的、经济上有用的任务上变得越来越好一样。所以我非常希望一些最理性的反对者和一些最理性的支持者能坐下来谈谈。我认为 SB 1047 非常两极分化。Anthropic 是唯一一家以非常详细的方式持积极态度的 AI 公司。我认为 Elon 发过一条简短的积极推文，但一些大公司如 Google、OpenAI、Meta、Microsoft 都强烈反对。所以我真正希望的是，一些关键利益相关者，一些深思熟虑的支持者和一些最深思熟虑的反对者，能坐下来讨论：‘我们如何解决这个问题，让支持者感到风险真正降低，让反对者感到它没有不必要地阻碍行业或创新？’我认为由于某种原因，事情变得过于两极分化，这两个群体没有以应有的方式坐下来谈。我感到紧迫。我真的认为我们需要在 2025 年做点什么。如果到了 2025 年底我们仍然没有采取任何行动，那我就会担心了。我现在还不担心，因为风险还没有到来，但我认为时间不多了。

Some of it is good, but some of it is really unnecessarily burdensome, and I think it's fair to say it has really slowed innovation. So I understand where people are coming from on priors. I understand why people start from that position. But again, I think AI is different. If we go to the very serious risks of autonomy and misuse that I talked about just a few minutes ago, I think those are unusual and they warrant an unusually strong response. So I think it's very important again, we need something that everyone can get behind. I think one of the issues with SB 1047, especially the original version, was it had a bunch of the structure of SB 1047, but it also had a bunch of stuff that was either clunky or would have created a bunch of burdens, a bunch of hassle, and might even have missed the target in terms of addressing the risks. You don't really hear about it on Twitter; you just hear people cheering for any regulation, and then the folks who are against make up these often quite intellectually dishonest arguments about how it will make us move away from California. The bill doesn't apply if you're headquartered in California; the bill only applies if you do business in California. Or that it would damage the open source ecosystem, or that it would cause all these things. I think those were mostly nonsense. But there are better arguments against regulation. There's one guy, Dean Ball, who is really a very scholarly scholar who looks at what happens when regulation is put in place and ways that they can kind of get a life of their own or how they can be poorly designed. So our interest has always been: we do think there should be regulation in this space, but we want to be an actor who makes sure that regulation is surgical, targeted at the serious risks, and something people can actually comply with. Because something I think the advocates of regulation don't understand as well as they could is: if we get something in place that is poorly targeted, that wastes a bunch of people's time, what's going to happen is people are going to say, 'See, these safety risks, this is nonsense. I just had to hire 10 lawyers to fill out all these forms, I had to run all these tests for something that was clearly not dangerous.' And after six months of that, there will be a groundswell, and we'll end up with a durable consensus against regulation. So I think the worst enemy of those who want real accountability is badly designed regulation. We need to actually get it right. And if there's one thing I could say to the advocates, it would be that I want them to understand this dynamic better. We need to be really careful and talk to people who actually have experience seeing how regulations play out in practice. The people who have seen that understand to be very careful. If this was some lesser issue, I might be against regulation at all. But what I want the opponents to understand is that the underlying issues are actually serious. They're not something that I or the other companies are just making up because of regulatory capture. They're not sci-fi fantasies. They're not any of these things. Every time we have a new model every few months, we measure the behavior of these models, and they're getting better and better at these concerning tasks, just as they are getting better and better at good, valuable, economically useful tasks. So I would just love it if some of the most reasonable opponents and some of the most reasonable proponents would sit down together. I think SB 1047 was very polarizing. Anthropic was the only AI company that felt positively in a very detailed way. I think Elon tweeted briefly something positive, but some of the big ones like Google, OpenAI, Meta, Microsoft were pretty strongly against. So what I would really like is if some of the key stakeholders, some of the thoughtful proponents and some of the most thoughtful opponents, would sit down and say, 'How do we solve this problem in a way that the proponents feel brings a real reduction in risk and that the opponents feel it is not hampering the industry or innovation any more than necessary?' And I think for whatever reason, things got too polarized, and those two groups didn't get to sit down in the way that they should. I feel urgency. I really think we need to do something in 2025. If we get to the end of 2025 and we've still done nothing about this, then I'm going to be worried. I'm not worried yet because again, the risks aren't here yet, but I think time is running short.

是的，就像你说的，提出一些精准的措施。没错，没错，正是这样。我们需要摆脱这种激烈的支持安全与激烈的反对监管的言论。对，这已经变成了 Twitter 上的骂战，不会有什么好结果。所以大家对游戏中的不同玩家很好奇。OpenAI 是元老之一。你在 OpenAI 有好几年的经验。你在那里的经历和历史是怎样的？

Yeah, and come up with something surgical like you said. Yeah, yeah, exactly. And we need to get away from this intense pro-safety versus intense anti-regulatory rhetoric. Right, it's turned into these flame wars on Twitter and nothing good's going to come from that. So there's a lot of curiosity about the different players in the game. One of the OGs is OpenAI. You have had several years of experience at OpenAI. What's your story and history there?

是的，我在 OpenAI 待了大约五年。最后几年，我想是两年，我是那里的研究副总裁。大概我和 Ilya Sutskever 在 2016 或 2017 年左右真正设定了研究方向。我第一次真正相信，或者说至少确认了我对 Scaling（规模扩张）假设的信念，是在 Ilya famously 对我说：‘你需要理解这些模型的一点是，它们只是想学习。模型只是想学习。’有时候，这些一句话的禅宗公案，你听到后就会想，‘啊，这解释了一切。这解释了我见过的一千件事。’从那以后，我脑海中就有了这样一个形象：你以正确的方式优化模型，以正确的方式引导模型，它们只是想学习，只是想解决问题，不管问题是什么。所以基本上，别挡它们的道。别挡它们的道。对，不要把你自己的想法强加给它们，告诉它们应该怎么学习。你知道，这和 Rich Sutton 的苦涩教训是一样的。

Yeah, so I was at OpenAI for roughly five years. For the last, I think it was a couple years, I was Vice President of Research there. Probably myself and Ilya Sutskever were the ones who really kind of set the research direction around 2016 or 2017. I first started to really believe in, or at least confirm my belief in, the scaling hypothesis when Ilya famously said to me: 'The thing you need to understand about these models is they just want to learn. The models just want to learn.' And again, sometimes there are these one-sentence Zen koans that you hear them and you're like, 'Ah, that explains everything. That explains like a thousand things that I've seen.' And then ever after, I had this visualization in my head of: you optimize the models in the right way, you point the models in the right way, they just want to learn, they just want to solve the problem regardless of what the problem is. So get out of their way, basically. Get out of their way. Yeah, don't impose your own ideas about how they should learn. And you know, this was the same thing as Rich Sutton's bitter lesson.

无论是《苦涩的教训》还是 Scaling 假说，我认为总体动态是：我从伊兰和亚历克·拉德福德等人那里获得了灵感，他们做了最初的 GPT-1，然后我全力推进。我和合作者做了 GPT-2、GPT-3、基于人类反馈的强化学习，这是为了处理早期的安全性和鲁棒性问题，比如辩论和放大，以及大量可解释性工作。所以，安全性与 Scaling 的结合，大概在 2018、2019、2020 年。那些年，我和我的合作者——其中许多人后来成为 Anthropic 的联合创始人——真正有了一个愿景，并推动了方向。

Put out in the bitter lesson or the scaling hypothesis. I think generally the dynamic was I got this inspiration from Ilan and from others like Alec Radford who did the original GPT-1, and then ran really hard with it. Me and my collaborators on GPT-2, GPT-3, RL from Human Feedback, which was an attempt to deal with early safety and durability things like debate and amplification, heavy on interpretability. So again, the combination of safety plus scaling, probably 2018, 2019, 2020. Those were the years when myself and my collaborators, many of whom became co-founders of Anthropic, really had a vision and drove the direction.

你为什么离开？你为什么决定离开？

Why did you leave? Why did you decide to leave?

我这么说吧。我认为这与“竞相向上”有关。在 OpenAI 期间，随着我逐渐理解 Scaling 假说以及安全性的重要性，前者 OpenAI 已经接受了，后者一直是 OpenAI 的官方表态。但多年来，我对如何处理这些问题、如何将它们推向世界、组织应该秉持什么原则，有自己特定的愿景。关于公司该不该做这个、该不该做那个，有很多讨论。外界有错误信息：有人说我们离开是因为不喜欢与微软的交易——假的，尽管关于如何做这笔交易确实有很多讨论。我们离开是因为不喜欢商业化——也不对。我们构建了 GPT-3，这是被商业化的模型，我参与了商业化。更多是关于如何做的问题。文明正沿着这条路走向非常强大的 AI。什么是谨慎、直接、诚实的方式，能建立对组织和个人的信任？我们如何从这里走到那里，如何有一个真正的愿景来把事情做对？如何让安全性不只是为了招聘而说的口号？归根结底，如果你对此有一个愿景，那就别管别人的愿景。如果你有如何实现它的愿景，你就应该去实现它。试图与别人的愿景争论是极其低效的。你可能认为他们做得不对，可能认为他们不诚实——谁知道呢，也许你是对的，也许不是。但你应该做的是，带上一些你信任的人，一起出去，实现你的愿景。如果你的愿景有说服力，能吸引人们——在伦理上、在市场上——如果你能创建一个人们想加入的公司，其做法人们认为合理，同时保持其在生态系统中的地位，人们就会模仿你。你这样做，尤其是做得比他们好，会促使他们改变行为，这比你是他们的老板并与之争论要有效得多。我不知道还能怎么更具体。但我认为，试图让别人的愿景变成你的愿景通常非常低效。更有效的是出去做一个干净的实验，说：“这是我们的愿景，这就是我们做事的方式。你可以选择忽略我们，拒绝我们，或者开始变得更像我们。”模仿是最真诚的奉承。这体现在客户的行为、公众的行为以及人们选择工作地点的行为上。最终，这不是一家公司赢或另一家公司赢的问题。如果我们或其他公司正在做一些人们真正觉得有吸引力的做法——我希望是实质性的，而不仅仅是表面——研究人员很精明，他们会看实质——那么其他公司就会开始模仿这种做法。他们因为模仿而获胜，这很好，这是成功。这就是“竞相向上”。最终谁赢并不重要，只要每个人都在模仿别人的好做法。我的一种思考方式是：我们都害怕的是“竞相向下”。在“竞相向下”中，谁赢都不重要，因为我们都输了。在最极端的情况下，我们制造出自主 AI，机器人奴役我们或其他什么——这半开玩笑，但这是可能发生的最极端情况。那时哪家公司领先就不重要了。相反，如果你创造了一个“竞相向上”，人们竞争采用好的做法，那么最终谁赢并不重要，甚至谁发起了这场竞赛也不重要。重点不是要显得高尚，而是要让系统进入比以前更好的均衡状态。个别公司可以在这方面发挥作用——它们可以帮助启动、加速。坦白说，我认为其他公司的个人也做到了这一点。当我们发布 RSP 时，他们通过更努力地推动在自己的公司做类似的事情来回应。有时其他公司做了我们认为是好做法的事情，我们也会采纳。唯一的区别是我们试图更前瞻，更先采用这些做法，当别人发明时更快地采用。但我认为这种动态才是我们应该关注的。它抽象掉了哪家公司赢、谁信任谁的问题。所有这些戏剧性的问题都极其无趣。重要的是我们所有人运作的生态系统，以及如何让这个生态系统变得更好。

I'm going to put things this way. I think it ties to the race to the top. In my time at OpenAI, as I came to appreciate the scaling hypothesis and the importance of safety along with it, the first one OpenAI was getting on board with. The second one had always been part of OpenAI's messaging. But over many years, I had a particular vision of how we should handle these things, how they should be brought out in the world, the kind of principles the organization should have. There were many discussions about should the company do this or that. There's misinformation out there: people say we left because we didn't like the deal with Microsoft—false, although there was a lot of discussion about exactly how we do the deal. We left because we didn't like commercialization—that's not true. We built GPT-3, which was the model that was commercialized; I was involved in commercialization. It's more about how you do it. Civilization is going down this path to very powerful AI. What's the way to do it that is cautious, straightforward, honest, that builds trust in the organization and in individuals? How do we get from here to there, and how do we have a real vision for how to get it right? How can safety not just be something we say because it helps with recruiting? At the end of the day, if you have a vision for that, forget about anyone else's vision. If you have a vision for how to do it, you should go off and do that vision. It is incredibly unproductive to try and argue with someone else's vision. You might think they're not doing it the right way, you might think they're dishonest—who knows, maybe you're right, maybe you're not. But what you should do is take some people you trust and go off together and make your vision happen. If your vision is compelling, if you can make it appeal to people—some combination of ethically, in the market—if you can make a company that's a place people want to join, that engages in practices people think are reasonable while maintaining its position in the ecosystem, people will copy it. The fact that you're doing it, especially better than they are, causes them to change their behavior in a much more compelling way than if they're your boss and you're arguing with them. I don't know how to be more specific. But I think it's generally very unproductive to try and get someone else's vision to look like your vision. It's much more productive to go off and do a clean experiment and say, 'This is our vision, this is how we're going to do things. Your choice is you can ignore us, reject what we're doing, or you can start to become more like us.' Imitation is the sincerest form of flattery. That plays out in the behavior of customers, the public, and where people choose to work. At the end, it's not about one company winning or another winning. If we or another company are engaging in some practice that people find genuinely appealing—and I want it to be in substance, not just appearance—and researchers are sophisticated and look at substance, then other companies start copying that practice. They win because they copied that practice. That's great, that's success. That's the race to the top. It doesn't matter who wins in the end, as long as everyone is copying everyone else's good practices. One way I think of it is: the thing we're all afraid of is a race to the bottom. In a race to the bottom, it doesn't matter who wins because we all lose. In the most extreme world, we make this autonomous AI that the robots enslave us or whatever—that's half joking, but that is the most extreme thing that could happen. Then it doesn't matter which company was ahead. If instead you create a race to the top where people are competing to engage in good practices, then at the end of the day, it doesn't matter who ends up winning, doesn't even matter who started the race to the top. The point isn't to be virtuous; the point is to get the system into a better equilibrium than it was before. Individual companies can play some role in doing this—they can help start it, accelerate it. Frankly, I think individuals at other companies have done this as well. When we put out an RSP, they react by pushing harder to get something similar done at their companies. Sometimes other companies do something that we think is a good practice, and we adopt it too. The only difference is we try to be more forward-leaning, adopt more of these practices first and more quickly when others invent them. But I think this dynamic is what we should be pointing at. It abstracts away the question of which company is winning, who trusts who. All these questions of drama are profoundly uninteresting. The thing that matters is the ecosystem that we all operate in and how to make that ecosystem better.

更好，因为这样能约束所有玩家。Anthropic 就是这样一个干净的实验，建立在 AI 安全具体应该是什么样的基础之上。我们肯定犯过很多错误。完美的组织不存在，它必须应对一千名员工的不完美，必须应对包括我在内的领导层的不完美，必须应对我们安排来监督领导层不完美的人的不完美，比如董事会和长期利益信托。这都是一群不完美的人，试图不完美地瞄准某个永远无法完美实现的理想。这就是你选择面对的，也将永远如此。但不完美不意味着放弃。有更好和更差之分。希望我们能做得足够好，开始建立一些整个行业都参与的做法。我猜这些公司中会有多家成功。Anthropic 会成功，我过去待过的其他公司也会成功，有些会比另一些更成功。这没那么重要，更重要的是我们让整个行业的激励对齐，这在一定程度上通过良性竞争、通过 RSP 之类的东西、通过选择性的精准监管来实现。

Better because that constrains all the players. And so Anthropic is this kind of clean experiment built on a foundation of what concretely AI safety should look like. Look, I'm sure we've made plenty of mistakes along the way. The perfect organization doesn't exist. It has to deal with the imperfection of a thousand employees, it has to deal with the imperfection of our leaders including me, it has to deal with the imperfection of the people we've put to oversee the imperfection of the leaders, like the board and the long-term benefit trust. It's all a set of imperfect people trying to aim imperfectly at some ideal that will never perfectly be achieved. That's what you sign up for, that's what it will always be. But imperfect doesn't mean you just give up. There's better and there's worse. And hopefully we can do well enough that we can begin to build some practices that the whole industry engages in. And then, my guess is that multiple of these companies will be successful. Anthropic will be successful, these other companies like ones I've been at in the past will also be successful, and some will be more successful than others. That's less important than again that we align the incentives of the industry, and that happens partly through the race to the top, partly through things like RSP, partly through selected surgical regulation.

你提到人才密度胜过人才规模。能解释一下吗？展开讲讲？谈谈如何打造一支优秀的 AI 研究者和工程师团队？

You said talent density beats talent mass. So can you explain that? Can you expand on it? Can you just talk about what it takes to build a great team of AI researchers and engineers?

这句话每个月都更正确。每个月我都觉得它比上个月更正确。做个思想实验：假设你有一个 100 人的团队，都非常聪明、有动力、与使命对齐，这就是你的公司。或者你有一个 1000 人的团队，其中 200 人非常聪明、与使命高度对齐，另外 800 人是随机的大科技公司员工。你选哪个？人才规模上，1000 人的团队更大，对吧？你甚至有更多极其有才华、极其对齐、极其聪明的人。但问题是，如果每个超级有才华的人环顾四周，看到的是其他同样超级有才华和投入的人，这就定下了基调。每个人都备受鼓舞，在同一个地方工作，彼此信任。如果你有 1000 或 10000 人，事情已经退化了，你无法筛选，只能随机选人。结果就是你需要设置很多流程和护栏，因为人们不完全信任彼此。你必须调解政治斗争。有太多事情拖慢组织的运作能力。所以我们接近 1000 人，我们努力让这 1000 人中尽可能多的人超级有才华、超级有技能。这也是过去几个月我们大幅放缓招聘的原因之一。今年前七八个月，我们从 300 人增长到 800 人。现在放缓了。过去三个月，我们从 800 增长到 900、950 左右。别引用确切数字。但我认为 1000 左右是一个拐点，我们想更谨慎地增长。早期和现在，我们雇佣了很多物理学家。理论物理学家学东西很快。最近，随着继续招聘，我们在研究和软件工程方面都保持了高标准。我们雇佣了很多资深人士，包括以前在这个领域其他公司工作的人，我们一直非常挑剔。从 100 到 1000，从 1000 到 10000，很容易不注意确保每个人都有统一的目标。如果你的公司由许多各自为政的领地组成，每个都想做自己的事，都优化自己的事，那很难做成任何事。但如果每个人都看到公司的更广阔目标，如果有信任和做正确事情的奉献精神，那本身就是一种超能力，我认为可以克服几乎所有其他劣势。就像史蒂夫·乔布斯说的：A 级人才希望看到其他 A 级人才。这是另一种说法。我不知道这跟人性有什么关系，但看到不是痴迷于单一使命的人会让人泄气。反过来，看到那样的人则非常激励人心。

This is one of these statements that's more true every month. Every month I see this statement as more true than I did the month before. So if I were to do a thought experiment: let's say you have a team of 100 people that are super smart, motivated, and aligned with the mission, and that's your company. Or you can have a team of a thousand people where 200 people are super smart, super aligned with the mission, and then 800 people are, let's just say, random big tech employees. Which would you rather have? The talent mass is greater in the group of a thousand people, right? You have an even larger number of incredibly talented, incredibly aligned, incredibly smart people. But the issue is just that if every time someone super talented looks around, they see someone else super talented and super dedicated, that sets the tone for everything. That sets the tone for everyone being super inspired to work at the same place, everyone trusts everyone else. If you have a thousand or ten thousand people and things have really regressed, you are not able to do selection and you're choosing random people. What happens is then you need to put a lot of processes and a lot of guardrails in place just because people don't fully trust each other. You have to adjudicate political battles. There are so many things that slow down the org's ability to operate. And so we're nearly a thousand people, and we've tried to make it so that as large a fraction of those thousand people as possible are super talented, super skilled. It's one of the reasons we've slowed down hiring a lot in the last few months. We grew from 300 to 800, I believe, in the first seven or eight months of the year. And now we've slowed down. We're at like, you know, last three months we went from 800 to 900, 950, something like that. Don't quote me on the exact numbers. But I think there's an inflection point around a thousand, and we want to be much more careful how we grow. Early on and now as well, we've hired a lot of physicists. Theoretical physicists can learn things really fast. Even more recently, as we've continued to hire, we've really had a high bar on both the research side and the software engineering side. We've hired a lot of senior people, including folks who used to be at other companies in this space, and we've just continued to be very selective. It's very easy to go from 100 to a thousand, a thousand to 10,000 without paying attention to making sure everyone has a unified purpose. It's so powerful if your company consists of a lot of different fiefs that all want to do their own thing, they're all optimizing for their own thing. It's very hard to get anything done. But if everyone sees the broader purpose of the company, if there's trust and there's dedication to doing the right thing, that is a superpower that in itself I think can overcome almost every other disadvantage. And you know, to Steve Jobs: A players want to look around and see other A players. That's another way of saying it. I don't know what that is about human nature, but it is demotivating to see people who are not obsessively driving towards a singular mission. And on the flip side, it is super motivating to see that.

有意思。从你与这么多优秀的人共事的经验来看，成为一名出色的 AI 研究者或工程师需要什么？

It's interesting. What does it take to be a great AI researcher or engineer from everything you've seen from working with so many amazing people?

我认为最重要的品质，尤其是在研究方面，但实际上是两者都需要的，是开放的心态。听起来开放心态很容易，对吧？就像‘哦，我对任何事都开放’。但回想我早期在 Scaling 假设上的经历，我和其他人看到的是同样的数据。我不觉得自己比共事的几百人更擅长编程或提出研究想法。在某些方面，我更差。我从来不是那种精确的编程者，比如找 bug、写 GPU 内核。我可以指出这里一百个人比我更擅长。但我认为我与众不同的是，我愿意用新的眼光看待事物。人们说，‘哦，我们还没有正确的算法，还没有想出正确的方法。’而我只是想，‘我不知道，这个神经网络有 3000 万个参数。如果我们给它 5000 万个呢？画几张图看看。’那种基本的科学思维：‘我看到一个可以改变的变量，改变它会发生什么？试试不同的东西，画个图。’即使这是世界上最简单的事：改变参数数量。这不是博士级别的实验设计。这简单又愚蠢。只要有人告诉你这很重要，任何人都能做到。这也不难理解。你不需要很聪明就能想到这一点。但你把两者结合起来……

Yeah, I think the number one quality, especially on the research side but really both, is open-mindedness. Sounds easy to be open-minded, right? You're just like, 'Oh, I'm open to anything.' But if I think about my own early history in the scaling hypothesis, I was seeing the same data others were seeing. I don't think I was a better programmer or better at coming up with research ideas than any of the hundreds of people that I worked with. In some ways, I was worse. I've never been a precise programmer, like finding the bug, writing the GPU kernels. I could point you to a hundred people here who are better at that than I am. But the thing that I think I did have that was different was that I was just willing to look at something with new eyes. People said, 'Oh, we don't have the right algorithms yet, we haven't come up with the right way to do things.' And I was just like, 'I don't know, this neural net has 30 million parameters. What if we gave it 50 million instead? Let's plot some graphs.' That basic scientific mindset of, 'I see some variable that I could change, what happens when it changes? Let's try these different things and create a graph.' Even this was the simplest thing in the world: change the number of parameters. This wasn't PhD-level experimental design. This was simple and stupid. Anyone could have done this if you just told them that it was important. It's also not hard to understand. You didn't need to be brilliant to come up with this. But you put the two together...

把各种东西整合起来，你知道，极少数人——个位数的人——通过认识到这一点推动了整个领域的发展。回顾历史上的发现，往往都是这样。所以这种开放的心态和用新眼光看待事物的意愿，通常来自领域的新人——经验往往反而是劣势——这是最重要的。这很难去寻找和测试，但我认为这是最重要的，因为当你找到某种真正新的思考方式，并有主动性去实践时，它绝对是变革性的。同时，能够快速实验，并在此过程中保持开放、好奇，用全新的眼光看待数据，看它实际上在说什么。这在机械可解释性中也是如此——另一个例子——比如机械可解释性的一些早期工作非常简单，只是以前没人想到去关心这个问题。

Things together and you know, a tiny number of people, some single-digit number of people, have driven forward the whole field by realizing this. And it's often like that if you look back at the discoveries in history. So this open-mindedness and this willingness to see with new eyes, which often comes from being newer to the field—often experience is a disadvantage for this—that is the most important thing. It's very hard to look for and test for, but I think it's the most important thing because when you find some really new way of thinking about things, when you have the initiative to do that, it's absolutely transformative. Also, being able to do rapid experimentation, and in the face of that, be open-minded and curious and look at the data from fresh eyes and see what it's actually saying. That applies in mechanistic interpretability—another example of this—like some of the early work in mechanistic interpretability is so simple, it's just no one thought to care about this question before.

要成为一名伟大的 AI 研究者需要什么？我们能倒回时钟吗？你会给那些对 AI 感兴趣、年轻、有抱负的人什么建议？我怎样才能对世界产生影响？

What does it take to be a great AI researcher? Can we rewind the clock back? What advice would you give to people interested in AI, they're young, looking forward, how can I make an impact on the world?

我认为我的首要建议就是开始玩模型。实际上，我有点担心现在这听起来像是显而易见的建议。我想三年前这并不明显，人们一开始会说‘哦，让我读最新的强化学习论文，让我……’不，我的意思是那才是关键。而且你也应该这么做，但现在随着模型和 API 的广泛可用，人们做得更多了。但我认为经验知识——这些模型是没人真正理解的新造物——所以获得与它们互动的经验。我还要说，再次呼应‘做新事，想新方向’，有很多东西尚未探索。例如，机械可解释性仍然非常新；可能比研究新模型架构更好，因为它比以前更流行了，但大概只有一百人在做，不是一万人。而且这是一个肥沃的研究领域——有太多唾手可得的果实，你走过去就能摘到。不管出于什么原因，人们对它还不够感兴趣。我认为在长程学习和长程任务方面还有很多工作要做。我认为评估能力还很早期，特别是对于在世界上行动的动态系统。我认为在多智能体方面也有一些东西。我的建议是‘滑向冰球要去的地方’。你不需要很聪明就能想到——所有五年后会令人兴奋的事情，人们甚至把它们当作常识提及，但不知何故存在一个障碍，人们没有尽可能多地投入，或者害怕做不流行的事情。我不知道为什么会这样，但克服这个障碍是我的首要建议。

I think my number one piece of advice is to just start playing with the models. This was actually—I worry a little this seems like obvious advice now. I think three years ago it wasn't obvious, and people started by 'oh let me read the latest reinforcement learning paper, let me...' No, I mean that was really the thing. And I mean you should do that as well, but now with wider availability of models and APIs, people are doing this more. But I think experiential knowledge—these models are new artifacts that no one really understands—so getting experience playing with them. I would also say, again in line with the 'do something new, think in some new direction,' like there are all these things that haven't been explored. For example, mechanistic interpretability is still very new; it's probably better to work on that than to work on new model architectures because it's more popular than it was before, but there are probably like a hundred people working on it, not 10,000. And it's just this fertile area for study—there's so much low-hanging fruit you can just walk by and pick. And the only reason, for whatever reason, people aren't interested in it enough. I think there are some things around long-horizon learning and long-horizon tasks where there's a lot to be done. I think evaluations are still very early in our ability to study evaluations, particularly for dynamic systems acting in the world. I think there's some stuff around multi-agent. Skate where the puck is going is my advice. And you don't have to be brilliant to think of it—all the things that are going to be exciting in 5 years, people even mention them as conventional wisdom, but it's just somehow there's this barrier that people don't double down as much as they could, or they're afraid to do something that's not the popular thing. I don't know why it happens, but getting over that barrier is my number one piece of advice.

我们能不能谈谈后训练？是的，现代后训练配方似乎什么都有：监督微调、RLHF、带 RL 的宪法 AI（如果是最佳缩写，又是命名问题），然后还有合成数据——似乎用了很多合成数据，或者至少试图找到获得高质量合成数据的方法。那么，让 Anthropic 的 Claude 如此出色的秘诀是什么？魔力有多少来自预训练，多少来自后训练？

Let's talk if we could a bit about post-training. Yeah, so it seems that the modern post-training recipe has a little bit of everything: supervised fine-tuning, RLHF, Constitutional AI with RL, if best acronym it's again that naming thing, and then synthetic data—seems like a lot of synthetic data, or at least trying to figure out ways to have high-quality synthetic data. So what's the secret sauce that makes Anthropic's Claude so incredible? How much of the magic is in the pre-training, how much is in the post-training?

嗯，首先，我们自己也无法完美地衡量这一点。你知道，当你看到某种强大的能力时，有时很难判断它来自预训练还是后训练。我们开发了一些方法来区分这两者，但它们并不完美。第二点要说的是，当存在优势时——我认为我们在强化学习方面总体上做得很好，也许是最好的，尽管我不知道，因为我看不到其他公司内部的情况——通常不是‘天哪，我们有别人没有的秘密魔法方法’。对吧，通常更像是，我们在基础设施上做得更好，所以能运行更长时间，或者我们能获得更高质量的数据，或者我们能更好地过滤数据，或者我们能在实践中结合这些方法。这通常是一些关于实践和工艺的枯燥事情。所以当我思考如何在训练这些模型（无论是预训练还是后训练）时做出特别的东西时，我更多地把它看作设计飞机或汽车。这不仅仅是‘哦，我有蓝图’——也许那能让你造出下一架飞机，但我们在设计过程中的文化工艺，我认为比我们能发明的任何特定小玩意儿都更重要。

Yeah, I mean, first of all, we're not perfectly able to measure that ourselves. You know, when you see some great capability, sometimes it's hard to tell whether it came from pre-training or post-training. We developed ways to try and distinguish between those two, but they're not perfect. The second thing I would say is, you know, when there is an advantage—and I think we've been pretty good at in general at RL, perhaps the best, although I don't know because I don't see what goes on inside other companies—usually it isn't 'oh my God, we have this secret magic method that others don't have.' Right, usually it's like, well, we got better at the infrastructure so we could run it for longer, or we were able to get higher quality data, or we were able to filter our data better, or we were able to combine these methods in practice. It's usually some boring matter of practice and tradecraft. So when I think about how to do something special in terms of how we train these models, both pre-training but even more so post-training, I really think of it a little more again as like designing airplanes or cars. It's not just like 'oh man, I have the blueprint'—maybe that makes you make the next airplane, but there's some cultural tradecraft of how we think about the design process that I think is more important than any particular gizmo we're able to invent.

好的，让我问问你具体的技术。首先关于 RLHF，你怎么看——宏观上，直觉上，几乎是哲学上——为什么你认为 RLHF 效果这么好？

Okay, well, let me ask you about specific techniques. So first on RLHF, what do you think—just zooming out, intuition, almost philosophy—why do you think RLHF works so well?

如果我回到缩放假设，表述缩放假设的一种方式是：如果你为 X 进行训练，并投入足够的算力，那么你就会得到 X。而 RLHF 擅长让模型做人类想让它做的事，或者更精确地说，做那些短暂观察模型并考虑不同可能回应的人类所偏好的回应。从安全和能力角度来看，这并不完美，因为人类往往无法完美识别模型想要什么，而且人类当下的需求可能不是长期需求。所以有很多微妙之处。但模型擅长产生人类在某种浅层意义上想要的东西。而且实际上，你甚至不需要投入那么多算力，因为还有另一件事：一个强大的预训练模型已经‘半路通罗马’。所以一旦你有了预训练模型，你就拥有了所有需要的表示。

If I go back to the scaling hypothesis, one of the ways to state the scaling hypothesis is: if you train for X and you throw enough compute at it, then you get X. And RLHF is good at doing what humans want the model to do, or at least to state it more precisely, doing what humans who look at the model for a brief period of time and consider different possible responses prefer as the response. Which is not perfect from both a safety and capabilities perspective, in that humans are often not able to perfectly identify what the model wants, and what humans want in the moment may not be what they want in the long term. So there's a lot of subtlety there. But the models are good at producing what the humans in some shallow sense want. And it actually turns out that you don't even have to throw that much compute at it because of another thing: this thing about a strong pre-trained model being halfway to anywhere. So once you have the pre-trained model, you have all the representations you need.

你认为基于人类反馈的强化学习（RLHF）是让模型变得更聪明，还是只是让它在人类面前显得更聪明？

Do you think RLHF makes the model smarter or just appears smarter to the humans?

我不认为它让模型更聪明，也不认为它只是让模型显得更聪明。RLHF 弥合了人类和模型之间的差距。你可以有非常聪明但完全无法沟通的东西——我们都认识这样的人，很聪明但听不懂他们在说什么。所以我认为 RLHF 只是弥合了那个差距。这不是我们做的唯一一种强化学习，也不是未来唯一会发生的强化学习。我认为强化学习有潜力让模型更聪明，让它们更好地推理、运作、发展新技能。也许在某些情况下可以通过人类反馈来实现，但我们今天做的 RLHF 大多还没有做到这一点，尽管我们很快就能做到了。它似乎提高了有用性，也提高了 Leopold 所说的‘解绑’——模型被束缚，然后训练解绑它们。我喜欢这个词。所以 RLHF 在某些方面解绑了模型，而其他方面模型还没有被解绑。

I don't think it makes the model smarter. I don't think it just makes the model appear smarter either. RLHF bridges the gap between the human and the model. You could have something really smart that can't communicate at all—we all know people like this, really smart but you can't understand what they're saying. So I think RLHF just bridges that gap. It's not the only kind of RL we do, and it's not the only kind that will happen in the future. I think RL has the potential to make models smarter, to make them reason better, operate better, develop new skills. Perhaps that could be done with human feedback in some cases, but the kind of RLHF we do today mostly doesn't do that yet, although we're quickly starting to be able to. It appears to increase helpfulness, and it also increases what Leopold called 'unhobbling'—where models are hobbled and then training unhobbles them. I like that word. So RLHF unhobbles models in some ways, and there are other ways where models haven't yet been unhobbled.

从成本来看，预训练是最贵的部分，还是后训练已经赶上来了？

In terms of cost, is pre-training the most expensive thing, or has post-training crept up to that?

目前，预训练仍然占成本的大部分。我不知道未来会怎样，但我完全可以预见一个后训练占成本大部分的将来。

At present, pre-training is still the majority of the cost. I don't know what to expect in the future, but I could certainly anticipate a future where post-training is the majority of the cost.

在那个未来，后训练的昂贵部分会是人类还是 AI？

In that future, would the costly thing for post-training be the humans or the AI?

我认为你无法扩大人类规模来获得高质量。任何依赖人类并使用大量算力的方法都必须依赖某种可扩展的监督方法，比如辩论或迭代放大之类的。

I don't think you can scale up humans enough to get high quality. Any method that relies on humans and uses a large amount of compute will have to rely on some scaled supervision method, like debate or iterated amplification or something like that.

你能描述一下宪法 AI 是什么吗？就像 2022 年 12 月那篇论文中首次详细阐述的那样。

Can you describe what Constitutional AI is, as first detailed in the December 2022 paper?

是的。基本想法是：在 RLHF 中，你有一个模型生成两个可能的回答，然后问人类更喜欢哪一个，或者按 1 到 7 评分。这很难，因为你需要扩大人类交互，而且非常隐式——你不知道自己想要模型做什么，只知道一千个人类的平均意愿。所以有两个想法：一，AI 系统本身能否决定哪个回答更好？给它看两个回答，问哪个更好。二，AI 应该使用什么标准？于是你有一份文件，一部宪法，规定模型应该使用的原则。AI 系统阅读这些原则，阅读环境和回答，然后判断模型表现如何。这是一种自我对弈——让模型与自己对抗。AI 给出回答，反馈到偏好模型，偏好模型再改进 AI。所以有一个三角形：AI、偏好模型和 AI 自身的改进。宪法的原则是人类可解释的——人类和 AI 都能阅读，所以具有良好的可翻译性或对称性。在实践中，我们同时使用模型宪法和 RLHF 以及其他方法。它是一个工具包中的工具，减少了 RLHF 的需求，并提高了每个 RLHF 数据点的价值。它还与未来的推理型强化学习方法有趣地交互。所以它是一个工具，但非常重要。

Yes. The basic idea is: with RLHF, you have a model that spits out two possible responses, and you ask a human which they like better, or rate it on a scale of 1 to 7. That's hard because you need to scale up human interaction, and it's very implicit—you don't have a sense of what you want the model to do, just what the average of a thousand humans wants. So two ideas: one, could the AI system itself decide which response is better? Show it two responses and ask which is better. Two, what criterion should the AI use? So you have a single document, a constitution, that says these are the principles the model should use to respond. The AI system reads those principles, reads the environment and the response, and says how well the model did. It's a form of self-play—training the model against itself. The AI gives a response, which feeds into a preference model, which in turn improves the AI. So you have a triangle: the AI, the preference model, and the improvement of the AI itself. The constitution's principles are human-interpretable—both the human and the AI can read them, so it has nice translatability or symmetry. In practice, we use both a model constitution and RLHF, and some other methods. It's one tool in a toolkit that reduces the need for RLHF and increases the value from each RLHF data point. It also interacts in interesting ways with future reasoning-type RL methods. So it's one tool, but a very important one.

想想开国元勋，谁来定义宪法——也就是那套原则？

Thinking about the founding fathers, who defines the constitution—the set of principles?

我会给出一个实际答案和一个更抽象的答案。实际上，模型被各种不同的客户使用。所以你可以有专门的规则或原则——我们隐式地微调模型版本，也讨论过显式地做，让人们可以将特殊原则构建到模型中。所以从实际角度来看，不同的人答案可能非常不同。客服代理的行为与律师非常不同，遵守不同的原则。但基础是，模型必须遵守一些特定原则。其中很多是人们会认同的——每个人都同意我们不希望模型带来 CBRN 风险。我们可以更进一步，同意民主和法治的基本原则。除此之外，就变得不确定了。我们的目标通常是让模型更加中立，不宣扬特定观点，而是像明智的智能体或顾问，帮助你思考问题，提出考虑因素，而不表达强烈的具体意见。

I'll give a practical answer and a more abstract one. Practically, models get used by all kinds of different customers. So you can have specialized rules or principles—we fine-tune versions of models implicitly, and we've talked about doing it explicitly, having special principles people can build into models. So from a practical perspective, the answer can be very different for different people. A customer service agent behaves very differently from a lawyer and obeys different principles. But at the base, there are specific principles models have to obey. A lot of them are things people would agree on—everyone agrees we don't want models to present CBRN risks. We can go a little further and agree on basic principles of democracy and the rule of law. Beyond that, it gets uncertain. Our goal is generally for models to be more neutral, not to espouse a particular point of view, but to be like wise agents or advisers that help you think things through and present considerations without expressing strong specific opinions.

OpenAI 发布了一份模型规范，具体定义了一些目标和模型行为示例。你觉得这有趣吗？约翰·舒尔曼，现在在 Anthropic，也参与了。这是一个有用的方向吗？Anthropic 也会发布模型规范吗？

OpenAI released a model spec that concretely defines some goals and examples of how the model should behave. Do you find that interesting? John Schulman, now at Anthropic, was involved. Is this a useful direction? Might Anthropic release a model spec as well?

是的，我认为这是一个非常有用的方向。它与宪法 AI 有很多共同点。这是另一个竞相向上的例子——我们拥有一种我们认为更好、更负责任的做法。这也是一种竞争优势。

Yes, I think that's a pretty useful direction. It has a lot in common with Constitutional AI. It's another example of a race to the top—we have something that we think is a better and more responsible way of doing things. It's also a competitive advantage.

然后其他人发现这有优势，也开始做那件事。我们就不再拥有竞争优势，但从另一个角度看，这很好，因为现在每个人都采纳了别人之前没有采纳的积极做法。所以我们的回应是，我们需要新的竞争优势来继续推动这场向上的竞赛。这就是我对此的总体感受。我还认为，这些事情的每一次实施都是不同的。模型规范中有一些东西是宪法 AI 中没有的，所以我们总是可以采纳那些东西，或者至少从中学习。所以我认为这是一个积极的动态例子，我们都希望这个领域有这样的动态。

Then others discover that it has advantages and start to do that thing. We then no longer have the competitive advantage, but it's good from the perspective that now everyone has adopted a positive practice that others were not adopting. So our response to that as well looks like we need a new competitive advantage in order to keep driving this race upwards. That's how I generally feel about that. I also think every implementation of these things is different. So there were some things in the model spec that were not in constitutional AI, and so we can always adopt those things or at least learn from them. So again, I think this is an example of the positive dynamic that I think we should all want the field to have.

我们来谈谈那篇了不起的文章《爱与优雅的机器》。我推荐每个人都读一读。它挺长的。

Let's talk about the incredible essay 'Machines of Love and Grace'. I recommend everybody read it. It's a long one.

确实挺长的。

It is rather long.

读到关于积极未来的具体想法，真是令人耳目一新。你采取了大胆的立场，因为你很可能在时间或具体应用上出错。

It's really refreshing to read concrete ideas about what a positive future looks like. And you took a bold stance because it's very possible you might be wrong on the dates or specific applications.

我完全预料到会在所有细节上出错。我可能整个都错得离谱，人们会嘲笑我好几年。未来就是这样。

I'm fully expecting to definitely be wrong about all the details. I might be spectacularly wrong about the whole thing, and people will laugh at me for years. That's just how the future works.

你提供了一系列 AI 的具体积极影响，以及超级智能 AI 如何加速生物学和化学的突破，从而带来治愈大多数癌症、预防所有传染病、将人类寿命延长一倍等成果。那么我们先谈谈这篇文章。你能给出这篇文章的高层愿景以及人们应该获得的关键要点吗？

You provided a bunch of concrete positive impacts of AI, and how a superintelligent AI might accelerate breakthroughs in biology and chemistry, leading to things like curing most cancers, preventing all infectious disease, doubling the human lifespan, and so on. So let's talk about this essay first. Can you give a high-level vision of this essay and what key takeaways people should have?

我花了很多时间，Anthropic 也花了很多精力，来思考如何应对 AI 的风险。我们如何看待这些风险？我们试图进行一场向上的竞赛，这要求我们构建所有这些能力。这些能力很酷，但我们所做的一大部分工作是应对风险。其理由是，所有这些积极的东西——市场是一个非常健康的有机体，它会产出所有积极的东西。而风险，我不知道，我们可能减轻它们，也可能不会。所以我们可以通过努力减轻风险来产生更大的影响。但我注意到这种思维方式的一个缺陷。这并不是说我对风险的重视程度有所改变，而是我谈论它们的方式可能有所改变。无论我刚才给出的推理多么合乎逻辑，如果你只谈论风险，你的大脑就只会思考风险。所以我认为理解如果事情进展顺利会怎样非常重要。我们试图防止这些风险的全部原因不是因为我们害怕技术，不是因为我们想放慢它的速度。而是因为如果我们能成功穿越这些风险，如果我们能成功闯关，那么关卡的另一边就是所有这些美好的事物。这些事物值得为之奋斗，并且能够真正激励人们。我认为，因为你看，所有这些投资者、风投、AI 公司都在谈论 AI 的积极好处，但正如你指出的，真正具体化的东西很少。推特上有一些随机的人发布闪闪发光的城市，以及那种“努力、加速、赶走末日论者”的氛围。这是一种非常激进的意识形态。但你会问，你实际上对什么感到兴奋？所以我想，对于一个来自风险方面的人来说，尝试真正解释这些好处会很有趣也很有价值。一方面因为我认为这是我们都能够支持的事情，另一方面我希望人们理解，这不是末日论者与加速主义者之间的对立。如果你真正理解 AI 的发展方向——也许更重要的轴是：AI 正在快速发展与 AI 没有快速发展——那么你就会真正欣赏这些好处，你希望人类、我们的文明抓住这些好处，但同时你也会非常认真地对待任何可能使其脱轨的事情。

I have spent a lot of time, and Anthropic has spent a lot of effort, on how to address the risks of AI. How do we think about those risks? We're trying to do a race to the top, which requires us to build all these capabilities. The capabilities are cool, but a big part of what we're trying to do is address the risks. The justification for that is that all these positive things—the market is a very healthy organism, it's going to produce all the positive things. The risks, I don't know, we might mitigate them, we might not. So we can have more impact by trying to mitigate the risks. But I noticed one flaw in that way of thinking. It's not a change in how seriously I take the risks, but maybe a change in how I talk about them. No matter how logical that line of reasoning might be, if you only talk about risks, your brain only thinks about risks. So I think it's very important to understand what if things do go well. The whole reason we're trying to prevent these risks is not because we're afraid of technology, not because we want to slow it down. It's because if we can get to the other side of these risks, if we can run the gauntlet successfully, then on the other side are all these great things. These things are worth fighting for and can really inspire people. I think, because look, you have all these investors, VCs, AI companies talking about the positive benefits of AI, but as you point out, there's a dearth of really getting specific about it. There are random people on Twitter posting gleaming cities and this vibe of 'grind, accelerate harder, kick out the doomers.' It's a very aggressive ideology. But then you're like, what are you actually excited about? So I figured it would be interesting and valuable for someone coming from the risk side to try to really make an attempt at explaining the benefits. Both because I think it's something we can all get behind, and I want people to understand that this isn't doomers versus accelerationists. If you have a true understanding of where things are going with AI—and maybe that's the more important axis: AI is moving fast versus AI is not moving fast—then you really appreciate the benefits, and you want humanity, our civilization, to seize those benefits, but you also get very serious about anything that could derail them.

所以起点是谈论这种强大的 AI，这是你喜欢用的术语。世界上大多数人使用 AGI，但你不喜欢这个术语，因为它有太多包袱，已经变得毫无意义。

So the starting point is to talk about this powerful AI, which is the term you like to use. Most of the world uses AGI, but you don't like the term because it has too much baggage and has become meaningless.

就像我们被这些术语困住了。也许我们被这些术语困住了，我改变它们的努力是徒劳的。这是一个毫无意义的语义点，但我一直在公开谈论它，所以我就再说一次。我认为这有点像：假设现在是 1995 年，摩尔定律让计算机越来越快。不知何故，有一种口头习惯，每个人都说，‘总有一天我们会拥有超级计算机，超级计算机将能够做所有这些事情。一旦我们有了超级计算机，我们就能测序基因组并做其他事情。’计算机确实越来越快，随着它们越来越快，它们将能够做所有这些伟大的事情。但并没有一个离散的点，在那之前计算机不是超级计算机，之后就是。‘超级计算机’是一个模糊的术语，用来描述比我们今天拥有的更快的计算机。没有一个你越过阈值然后说‘哦，天哪，我们在做一种全新的计算’的点。我对 AGI 也有这种感觉。它只是一个平滑的指数增长。如果你说的 AGI 是指 AI 越来越好，逐渐能做越来越多人类能做的事情，直到比人类更聪明，然后从那里变得更聪明，那么是的，我相信 AGI。但如果 AGI 是某种离散或独立的东西，就像人们通常谈论的那样，那么它就是一个毫无意义的流行词。

It's like we're stuck with the terms. Maybe we're stuck with the terms, and my efforts to change them are futile. It's a pointless semantic point, but I keep talking about it publicly, so I'll just do it once more. I think it's a little like: let's say it was 1995 and Moore's law is making computers faster. For some reason, there had been this verbal tick that everyone was like, 'Someday we're going to have supercomputers, and supercomputers are going to be able to do all these things. Once we have supercomputers, we'll be able to sequence the genome and do other things.' It's true that computers are getting faster, and as they get faster, they'll be able to do all these great things. But there's no discrete point at which you had a supercomputer and previous computers were not. 'Supercomputer' is a vague term to describe computers that are faster than what we have today. There's no point at which you pass a threshold and you're like, 'Oh my God, we're doing a totally new type of computation.' I feel that way about AGI. There's just a smooth exponential. If by AGI you mean AI is getting better and better and gradually it's going to do more and more of what humans do until it's smarter than humans and then gets even smarter from there, then yes, I believe in AGI. But if AGI is some discrete or separate thing, which is the way people often talk about it, then it's a meaningless buzzword.

对我来说，它只是强大 AI 的一种特定形式，正如你所定义的那样。

To me, it's just a specific form of a powerful AI, exactly how you define it.

你定义得非常清楚。在智能轴上，纯粹从智能角度看，它比诺贝尔奖得主更聪明，正如你所说，在大多数相关学科都是如此。所以那只是智能。它既有创造力，又能产生新想法，在所有学科都是如此。诺贝尔奖得主，嗯，在他们巅峰时期。它能使用所有模态。这有点不言自明，但就是能操作世界的所有模态。它可以连续工作数小时、数天甚至数周来完成任务，自己进行详细的规划，只在需要时才向你求助。它可以使用——这其实挺有意思的——我记得你在文章里说过，这又是一个赌注，它不会具身化，但可以控制具身工具。所以它可以控制工具、机器人、实验室设备。用于训练它的资源可以被重新利用来运行数百万个它的副本，每个副本都是独立的，可以独立工作。所以你可以克隆这个智能系统。

You define it very nicely. So on the intelligence axis, it's just on pure intelligence, it's smarter than a Nobel Prize winner as you describe across most relevant disciplines. So okay, that's just intelligence. So it's both in creativity and be able to generate new ideas, all that kind of stuff in every discipline. Nobel Prize winner, okay, in their prime. It can use every modality. So that's kind of self-explanatory, but just operate across all the modalities of the world. It can go off for many hours, days, and weeks to do tasks and do its own sort of detailed planning and only ask you help when it's needed. It can use—this is actually kind of interesting—I think in the essay you said, I mean again, it's a bet that it's not going to be embodied, but it can control embodied tools. So it can control tools, robots, laboratory equipment. The resource used to train it can then be repurposed to run millions of copies of it, and each of those copies would be independent, that can do their own independent work. So you can do the cloning of the intelligence system.

是的，是的。我的意思是，你可能从领域外想象，以为只有一个这样的系统，对吧？你造了一个，就只有一个。但事实是，规模扩张非常快。我们今天就在做：我们造一个模型，然后部署数千个，甚至数万个实例。我认为到那时——肯定在 2 到 3 年内，不管我们有没有这些超级强大的 AI——集群的规模会大到你可以部署数百万个这样的系统，而且它们会比人类更快。所以如果你的设想是‘哦，我们只有一个，而且造起来要花时间’，我的观点是：不，实际上你立刻就有数百万个。而且一般来说，它们学习和行动的速度比人类快 10 到 100 倍。所以这是一个非常好的强大 AI 的定义。

Yeah, yeah. I mean, you might imagine from outside the field that there's only one of these, right? That you made it, you've only made one. But the truth is that the scale-up is very quick. We do this today: we make a model and then we deploy thousands, maybe tens of thousands of instances of it. I think by the time—certainly within 2 to 3 years, whether we have these super powerful AIs or not—clusters are going to get to the size where you'll be able to deploy millions of these, and they'll be faster than humans. And so if your picture is, 'Oh, we'll have one and it'll take a while to make them,' my point there was no: actually, you have millions of them right away. And in general, they can learn and act 10 to 100 times faster than humans. So that's a really nice definition of powerful AI.

好的，所以是这样。但你也写道，显然这样一个实体能够非常快速地解决非常困难的问题，但要弄清楚有多快并不简单。两个极端立场在我看来都是错误的。所以奇点是一个极端，而另一个极端是相反的。你能描述一下这两个极端吗？

Okay, so that. But you also write that clearly such an entity would be capable of solving very difficult problems very fast, but it is not trivial to figure out how fast. Two extreme positions both seem false to me. So the singularity is on the one extreme and the opposite on the other extreme. Can you describe each of the extremes?

是的，我们来描述一下极端。一个极端是：你看，如果我们看进化史，有一个巨大的加速，几十万年来我们只有单细胞生物，然后有了哺乳动物，然后有了猿类，然后很快变成了人类，人类很快建立了工业文明。所以这会继续加速，人类水平没有上限。一旦模型比人类聪明得多，它们就会非常擅长构建下一个模型。如果你写下一个简单的微分方程，这是一个指数。所以会发生的是，模型会构建更快的模型，模型会构建更快的模型，那些模型会构建纳米技术，可以接管世界并产生比你能产生的多得多的能量。所以如果你只是解这个抽象的微分方程，那么在我们构建第一个比人类更强大的 AI 之后 5 天，世界就会被这些 AI 填满，所有可能被发明的技术都会被发明出来。我有点夸张，但我认为这是一个极端。而我认为事实并非如此的原因是，第一，我认为他们只是忽略了物理定律。在物理世界中，事情只能以一定的速度完成。其中一些循环需要通过生产更快的硬件，而生产更快的硬件需要很长时间。事情需要很长时间。还有复杂性的问题。我认为无论你多聪明，人们谈论‘哦，我们可以建模生物系统，它会做所有事情’。听着，我认为计算建模可以做很多事情——我在生物学工作时做了很多计算建模——但有很多事情你无法预测。它们足够复杂，以至于仅仅迭代、仅仅运行实验就会击败任何建模，无论建模的系统有多聪明。

Yeah, so let's describe the extremes. One extreme would be: look, if we look at evolutionary history, there was this big acceleration where for hundreds of thousands of years we just had single-cell organisms, then we had mammals, then we had apes, and then that quickly turned to humans, humans quickly built industrial civilization. So this is going to keep speeding up and there's no ceiling at the human level. Once models get much smarter than humans, they'll get really good at building the next models. And if you write down a simple differential equation, this is an exponential. So what's going to happen is that models will build faster models, models will build faster models, and those models will build nanotech that can take over the world and produce much more energy than you could produce otherwise. So if you just solve this abstract differential equation, then 5 days after we build the first AI that's more powerful than humans, the world will be filled with these AIs and every possible technology that could be invented will be invented. I'm caricaturing this a little bit, but I think that's one extreme. And the reason I think that's not the case is that, one, I think they just neglect the laws of physics. It's only possible to do things so fast in the physical world. Some of those loops go through producing faster hardware, and it takes a long time to produce faster hardware. Things take a long time. There's this issue of complexity. I think no matter how smart you are, people talk about 'oh, we can model biological systems, it'll do everything.' Look, I think computational modeling can do a lot—I did a lot of computational modeling when I worked in biology—but there are a lot of things that you can't predict. They are complex enough that just iterating, just running the experiment, is going to beat any modeling, no matter how smart the system doing the modeling is.

哦，即使它不与物理世界交互，仅仅建模也会很难。

Oh, even if it's not interacting with the physical world, just the modeling is going to be hard.

是的，我认为建模会很难，让模型匹配物理世界也会很难。所以它确实需要与物理世界交互来验证。但你看即使是最简单的问题。我谈到三体问题或简单的混沌预测，或者预测经济。预测两年后的经济非常困难。也许情况是，普通人可以预测下个季度的经济，尽管他们实际上也做不到。也许一个聪明无数倍的 AI 系统只能预测一年后之类的。所以计算机智能呈指数增长，而预测能力只是线性增长。生物分子相互作用也是如此。当你扰动一个复杂系统时，你不知道会发生什么。你可以在其中找到简单的部分。如果你更聪明，你更擅长找到这些简单的部分。然后我认为人类制度真的非常困难。即使是我们已经开发的技术，甚至那些效果证据非常强的技术，也很难让人们采用。人们有顾虑，他们认为事情是阴谋论。这非常困难。让非常简单的事情通过监管系统也非常困难。我不想贬低任何在技术监管系统工作的人——他们必须处理艰难的权衡，他们必须拯救生命——但整个系统做出了一些明显的权衡，这些权衡远远没有最大化人类福祉。所以如果我们把 AI 系统带入这些人类系统，通常智能水平可能根本不是限制因素。可能只是做某件事需要很长时间。现在，如果 AI 系统绕过了所有政府，如果它只是说‘我是世界的独裁者，我要做任何事’，其中一些事情它可以做到。再次，与复杂性有关的事情，我……

Yeah, I think well, the modeling is going to be hard, and getting the model to match the physical world is going to be hard. So it does have to interact with the physical world to verify. But you just look at even the simplest problems. I talk about the three-body problem or simple chaotic prediction, or predicting the economy. It's really hard to predict the economy two years out. Maybe the case is that normal humans can predict what's going to happen in the economy in the next quarter, although they can't really do that. Maybe an AI system that's a zillion times smarter can only predict it out a year or something. So you have this kind of exponential increase in computer intelligence for linear increase in ability to predict. Same with biological molecules interacting. You don't know what's going to happen when you perturb a complex system. You can find simple parts in it. If you're smarter, you're better at finding these simple parts. And then I think human institutions are just really difficult. It's been hard to get people to adopt even the technologies that we've developed, even ones where the case for their efficacy is very strong. People have concerns, they think things are conspiracy theories. It's been very difficult. It's also been very difficult to get very simple things through the regulatory system. I don't want to disparage anyone who works in regulatory systems of any technology—there are hard trade-offs they have to deal with, they have to save lives—but the system as a whole makes some obvious trade-offs that are very far from maximizing human welfare. So if we bring AI systems into these human systems, often the level of intelligence may just not be the limiting factor. It just may be that it takes a long time to do something. Now, if the AI system circumvented all governments, if it just said 'I'm dictator of the world and I'm going to do whatever,' some of these things it could do. Again, the things having to do with complexity, I...

我仍然认为很多事情需要时间。我不认为 AI 系统能产生大量能量或登月会有所帮助。就像一些人在评论中对那篇文章的回应，说 AI 系统可以产生大量能量，更智能的 AI 系统，这没有抓住重点。那种循环并不能解决我在这里讨论的关键问题。所以我认为很多人没有理解重点。但即使它完全对齐，并能绕过所有人类障碍，它也会有麻烦。但再次强调，如果你想要一个不接管世界、不毁灭人类的 AI 系统，那么它基本上需要遵循基本的人类法律，对吧？如果我们想要一个真正美好的世界，我们就必须有一个与人类互动的 AI 系统，而不是一个创建自己法律体系或无视所有法律的系统。所以尽管这些过程效率低下，我们还是要处理它们，因为在这些系统的推广中需要一定的民众和民主合法性。我们不能让一小群开发这些系统的人说这对所有人都是最好的。我认为这是错误的，而且我认为在实践中它也行不通。所以你把所有这些放在一起，我们不会在五分钟内改变世界并上传所有人。我只是认为这不会发生。而且即使它可能发生，也不是通往美好世界的方式。

I still think a lot of things would take a while. I don't think it helps that the AI systems can produce a lot of energy or go to the moon. Like some people in comments responded to the essay saying the AI system can produce a lot of energy and smarter AI systems, that's missing the point. That kind of cycle doesn't solve the key problems that I'm talking about here. So I think a bunch of people missed the point there. But even if it were completely aligned and could get around all these human obstacles, it would have trouble. But again, if you want this to be an AI system that doesn't take over the world, that doesn't destroy humanity, then basically it's going to need to follow basic human laws, right? If we want to have an actually good world, we're going to have to have an AI system that interacts with humans, not one that creates its own legal system or disregards all the laws or all of that. So as inefficient as these processes are, we're going to have to deal with them because there needs to be some popular and democratic legitimacy in how these systems are rolled out. We can't have a small group of people who are developing these systems say this is what's best for everyone. I think it's wrong, and I think in practice it's not going to work anyway. So you put all those things together, and we're not going to change the world and upload everyone in five minutes. I just don't think it's going to happen. And to the extent that it could happen, it's not the way to lead to a good world.

所以这是一方面。另一方面，还有另一组观点，我实际上在某种程度上更同情它们，那就是：看，我们以前见过巨大的生产力增长，对吧？经济学家熟悉研究来自计算机革命和互联网革命的生产力增长，通常这些生产力增长并不令人印象深刻。它们比你想象的要少。有一句罗伯特·索洛的名言：'你到处都能看到计算机革命，除了在生产力统计数据中。'那么为什么会这样呢？人们指向公司的结构、企业的结构，以及我们现有技术向世界非常贫困地区推广的速度有多慢，我在文章中谈到了这一点。我们如何将这些技术带给世界上最贫困的地区，这些地区在手机技术、计算机、医药方面已经落后，更不用说尚未发明的新奇 AI 了？所以你可以有一种观点：嗯，这在技术上是惊人的，但完全是虚张声势。我认为泰勒·考恩，他对我的文章写了回应，就有这种观点。我认为他认为根本性的变化最终会发生，但他认为需要 50 或 100 年。你甚至可以对整个事情有更静态的看法。我认为其中有一些道理。我认为时间尺度太长了。而且我可以看到今天 AI 的双方。我们的许多客户是习惯于以某种方式做事的大型企业。我也在与政府的交谈中看到了这一点。那些是典型的机构，变化缓慢的实体。

So that's on one side. On the other side, there's another set of perspectives which I actually in some ways have more sympathy for, which is: look, we've seen big productivity increases before, right? Economists are familiar with studying the productivity increases that came from the computer revolution and internet revolution, and generally those productivity increases were underwhelming. They were less than you might imagine. There was a quote from Robert Solow: 'You see the computer revolution everywhere except the productivity statistics.' So why is this the case? People point to the structure of firms, the structure of enterprises, how slow it's been to roll out our existing technology to very poor parts of the world, which I talk about in the essay. How do we get these technologies to the poorest parts of the world that are behind on cell phone technology, computers, medicine, let alone newfangled AI that hasn't been invented yet? So you could have a perspective that's like: well, this is amazing technically, but it's all a nothing burger. I think Tyler Cowen, who wrote something in response to my essay, has that perspective. I think he thinks the radical change will happen eventually, but he thinks it'll take 50 or 100 years. And you could have even more static perspectives on the whole thing. I think there's some truth to it. I think the time scale is just too long. And I can see both sides with today's AI. A lot of our customers are large enterprises who are used to doing things a certain way. I've also seen it in talking to governments. Those are prototypical institutions, entities that are slow to change.

但我反复看到的动态是：是的，移动这艘船需要很长时间。是的，有很多阻力和缺乏理解。但让我觉得进步最终会以中等速度发生——不是极快，而是中等速度——的是，你与……我在大公司甚至政府中反复发现，这些机构实际上出人意料地前瞻，你会发现两件事推动事情前进。第一：你发现公司或政府中有一小部分人真正看到了大局，看到了整个 Scaling（规模扩张）假设，理解 AI 的发展方向，或者至少理解它在其行业内的方向。在当前的美国政府中，就有这样几个人真正看到了全貌。这些人认为这是世界上最重要的事情，并为之奔走。但仅靠他们不足以成功，因为他们是大组织中的一小群人。但随着技术开始推广，在那些最愿意采用它的人中取得成功，竞争的幽灵给了他们顺风。因为他们可以在大组织内部指出，'看，这些其他人在做这个。'一家银行可以说，'看，这个新奇的对冲基金在做这件事，他们会抢走我们的生意。'在美国，我们可以说，'我们担心中国会在我们之前到达那里。'这种组合——竞争的幽灵加上这些在许多方面僵化的组织中的少数远见者——你把这两者放在一起，它实际上会让事情发生。这是两者之间的平衡斗争，因为惯性非常强大，但最终经过足够的时间，创新的方法会突破。我见过这种情况发生。我反复看到过这种弧线。进步的障碍、复杂性、不知道如何使用模型或如何部署它们，都存在。有一段时间，它们似乎会永远持续下去，好像变化不会发生。但最终变化发生了，而且总是来自少数人。当我在 AI 领域内部倡导 Scaling（规模扩张）假设时，我有同样的感觉，其他人不理解。感觉好像没有人会理解。感觉我们有一个几乎没人知道的秘密。然后几年后，每个人都知道了这个秘密。所以我认为 AI 在世界上的部署也会这样。障碍会逐渐瓦解，然后一下子全部瓦解。

But the dynamic I see over and over again is: yes, it takes a long time to move the ship. Yes, there's a lot of resistance and lack of understanding. But the thing that makes me feel that progress will in the end happen moderately fast, not incredibly fast but moderately fast, is that you talk to... what I find over and over again in large companies, even in governments, which have been actually surprisingly forward-leaning, you find two things that move things forward. One: you find a small fraction of people within a company, within a government, who really see the big picture, who see the whole scaling hypothesis, who understand where AI is going, or at least understand where it's going within their industry. And there are a few people like that within the current US government who really see the whole picture. And those people see that this is the most important thing in the world, and they agitate for it. But they alone are not enough to succeed because they are a small set of people within a large organization. But as the technology starts to roll out, as it succeeds in some places in the folks who are most willing to adopt it, the specter of competition gives them a wind at their backs. Because they can point within their large organization and say, 'Look, these other guys are doing this.' One bank can say, 'Look, this newfangled hedge fund is doing this thing, they're going to eat our lunch.' In the US, we can say, 'We're afraid China's going to get there before we are.' And that combination—the specter of competition plus a few visionaries within these organizations that in many ways are sclerotic—you put those two things together, and it actually makes something happen. It's a balanced fight between the two because inertia is very powerful, but eventually over enough time, the innovative approach breaks through. I've seen that happen. I've seen the arc of that over and over again. The barriers to progress, the complexity, not knowing how to use the model or how to deploy them, are there. And for a bit, it seems like they're going to last forever, like change doesn't happen. But then eventually change happens, and it always comes from a few people. I felt the same way when I was an advocate of the scaling hypothesis within the AI field itself, and others didn't get it. It felt like no one would ever get it. It felt like we had a secret almost no one ever had. And then a couple years later, everyone has the secret. So I think that's how it's going to go with deployment of AI in the world. The barriers are going to fall apart gradually and then all at once.

所以我认为这将是……这只是一个直觉，我很容易看到我可能是错的。我认为这更像是 5 或 10 年，正如我在文章中所说，而不是 50 或 100 年。我也认为这更像是 5 或 10 年，而不是 5 或 10 小时，因为我刚刚看到了人类系统是如何运作的。而且我认为很多写下微分方程的人，说 AI 将制造更强大的 AI，无法理解这些事情怎么可能不会变化得这么快，我认为他们不理解这些事情。那么，我们实现 AGI（通用人工智能），也就是强大 AI，也就是超级有用 AI 的时间线是什么？我开始……

So I think this is going to be... and this is just an instinct, I could easily see how I'm wrong. I think it's going to be more like 5 or 10 years, as I say in the essay, than it's going to be 50 or 100 years. I also think it's going to be 5 or 10 years more than it's going to be 5 or 10 hours, because I've just seen how human systems work. And I think a lot of these people who write down the differential equations, who say AI is going to make more powerful AI, who can't understand how it could possibly be the case that these things won't change so fast, I think they don't understand these things. So what's the timeline to where we achieve AGI, aka powerful AI, aka super useful AI? I'm starting to...

你说这是个关于命名的辩论。不纯粹的智能，你可以在每个相关领域都比诺贝尔奖得主更聪明，还有我们说的所有模态，你可以自己连续几天几周地做事，自己做生物学实验。我们就聚焦生物学吧，因为你已经说服了我关于生物学和健康的部分。从科学角度看那太令人兴奋了，让我想成为生物学家。几乎……不，这是我写的时候的感觉，那就是如果我们能实现它，如果我们能扫清障碍并实现它，这将是一个如此美好的未来。背后有如此多的美、优雅和道德力量，如果我们能……而且这是我们所有人都应该能达成共识的，对吧？无论我们在政治问题上争吵多少，这是否真的能让我们团结起来？但你问的是我们什么时候能实现？你觉得呢？给个数字吧。

Calling it that it's a debate about naming. Unpure intelligence, you can be smarter than a Nobel Prize winner in every relevant discipline and all the things we've said modality, you can go and do stuff on its own for days, weeks, and do biology experiments on its own. Let's just stick to biology because you sold me on the whole biology and health section. That's so exciting from a scientific perspective, it made me want to be a biologist. It's almost... no, this was the feeling I had when I was writing it, that it's like this would be such a beautiful future if we can just make it happen, if we can just get the landmines out of the way and make it happen. There's so much beauty and elegance and moral force behind it, if we can just... and it's something we should all be able to agree on, right? As much as we fight about political questions, is this something that could actually bring us together? But you were asking when will we get this? When do you think? Just put numbers on it.

这当然是我多年来一直在思考的问题，我完全没有把握。每次我说 2026 或 2027 年，推特上就会有无数人说‘他说了 2026 或 2027 年’，然后在接下来的两年里反复强调这就是我认为会发生的时间。所以那些截取片段的人会剪掉我刚才说的话，只留下我接下来要说的。但我还是说吧。如果你外推我们到目前为止的曲线，如果你说我们开始达到博士水平，去年是本科水平，前年是高中水平。你可以争论具体任务和哪些方面，我们仍然缺少模态，但这些正在被添加，比如计算机使用、图像输入、图像生成都已经加入。如果你只是粗略地看一下这些能力提升的速度，确实会让你觉得我们会在 2026 或 2027 年达到。同样，很多事情可能阻碍它：数据可能耗尽，我们可能无法按需扩展集群，也许台湾被炸了之类，我们无法生产那么多 GPU。所以有很多事情可能阻碍整个过程，我不完全相信直线外推。但如果你相信直线外推，我们会在 2026 或 2027 年达到。我认为最可能的是会有一些轻微的延迟。我不知道延迟多久，但我认为它可能按时发生，也可能有轻微延迟。我认为仍然存在一百年内不会发生的世界，但那些世界的数量正在迅速减少。我们正在迅速耗尽真正令人信服的障碍，真正令人信服的为什么未来几年不会发生的理由。2020 年时障碍更多，尽管我当时的感觉是我们会克服所有障碍。所以作为一个看到大多数障碍被清除的人，我有点怀疑我的直觉，我的猜测是剩下的障碍不会阻止我们。但归根结底，我不想把这说成是科学预测。人们称之为缩放定律，那是用词不当。摩尔定律也是用词不当。缩放定律不是宇宙的定律，它们是经验规律。我会押注它们继续成立，但我不确定。

This is of course the thing I've been grappling with for many years, and I'm not at all confident. Every time if I say 2026 or 2027, there will be a zillion people on Twitter who will be like 'he said 2026 or 2027' and it'll be repeated for the next two years that this is definitely when I think it's going to happen. So whoever's extracting these clips will crop out the thing I just said and only say the thing I'm about to say. But I'll just say it anyway. So if you extrapolate the curves that we've had so far, if you say well, we're starting to get to like PhD level, and last year we were at undergraduate level, and the year before we were at the level of a high school student. You can quibble with at what tasks and for what, we're still missing modalities, but those are being added, like computer use was added, image in was added, image generation has been added. If you just kind of eyeball the rate at which these capabilities are increasing, it does make you think that we'll get there by 2026 or 2027. Again, lots of things could derail it: we could run out of data, we might not be able to scale clusters as much as we want, maybe Taiwan gets blown up or something and we can't produce as many GPUs as we want. So there are all kinds of things that could derail the whole process, so I don't fully believe the straight line extrapolation. But if you believe the straight line extrapolation, we'll get there in 2026 or 2027. I think the most likely is that there's some mild delay relative to that. I don't know what that delay is, but I think it could happen on schedule, I think there could be a mild delay. I think there are still worlds where it doesn't happen in a hundred years, but the number of those worlds is rapidly decreasing. We are rapidly running out of truly convincing blockers, truly compelling reasons why this will not happen in the next few years. There were a lot more in 2020, although my hunch at that time was that we will make it through all those blockers. So sitting as someone who has seen most of the blockers cleared out of the way, I kind of suspect my hunch, my suspicion is that the rest of them will not block us. But at the end of the day, I don't want to represent this as a scientific prediction. People call them scaling laws, that's a misnomer. Moore's law is a misnomer. Scaling laws, they're not laws of the universe, they're empirical regularities. I am going to bet in favor of them continuing, but I'm not certain of that.

所以你广泛描述了所谓的压缩的 21 世纪，AGI 将如何帮助引发一系列生物学和医学的突破，以我提到的各种方式帮助我们。那么你认为，它可能采取哪些早期步骤？顺便说一句，我问了 Claude 该问你什么问题，Claude 让我问：你认为在这个未来，与 AGI 合作的生物学家的典型一天是什么样的？

So you extensively describe sort of the compressed 21st century, how AGI will help set forth a chain of breakthroughs in biology and medicine that help us in all these kinds of ways that I mentioned. So how do you think, what are the early steps it might do? And by the way, I asked Claude good questions to ask you, and Claude told me to ask: what do you think is a typical day for a biologist working with AGI look like in this future?

是的，Claude 很好奇。让我先回答你的第一个问题，然后再回答那个。Claude 想知道他的未来是什么，对吧？没错。我将和谁一起工作？所以我认为我在文章中重点强调的一点是，在大型组织和系统中，最终会有少数人或少数新想法导致事情朝着与以前不同的方向发展，他们不成比例地影响轨迹。同样的事情也在发生。如果你考虑健康领域，有数万亿美元支付给医疗保险和其他健康保险，而 NIH 只有 1000 亿。如果我想那些真正带来革命性变化的少数事物，它们可以浓缩成其中的一小部分。所以当我想到 AI 会在哪里产生影响时，我想：AI 能否把那小部分变成更大的部分并提高其质量？在生物学中，我的经验是生物学最大的问题是你看不到发生了什么。你几乎没有能力看到发生了什么，更不用说改变它了。你所拥有的是：你必须从中推断出有一堆细胞，每个细胞内有 30 亿个碱基对的 DNA，按照遗传密码构建，还有所有这些过程正在进行，而我们作为未增强的人类无法影响它们。这些细胞大部分时间在分裂，这是健康的，但有时这个过程出错，那就是癌症。细胞在衰老，你的皮肤可能变色，随着年龄增长出现皱纹，所有这些都由这些过程决定，所有这些蛋白质被生产、运输到细胞的不同部分、相互结合。在我们对生物学的初始状态中，我们甚至不知道这些细胞存在。我们必须发明显微镜来观察细胞，我们必须发明更强大的显微镜来看到细胞以下到分子水平，我们必须发明 X 射线晶体学来看到 DNA，我们必须发明基因测序来读取 DNA，我们必须发明蛋白质折叠技术来……

Yeah, Claude is curious. Let me start with your first questions and then I'll answer that. Claude wants to know what's in his future, right? Exactly. Who am I going to be working with? So I think one of the things I went hard on in the essay is this idea that within large organizations and systems, there end up being a few people or a few new ideas that kind of cause things to go in a different direction than they would have before, who disproportionately affect the trajectory. There's a bunch of the same thing going on. If you think about the health world, there's trillions of dollars paid out to Medicare and other health insurance, and then the NIH is 100 billion. And if I think of the few things that have really revolutionized anything, it could be encapsulated in a small fraction of that. So when I think of where will AI have an impact, I'm like: can AI turn that small fraction into a much larger fraction and raise its quality? And within biology, my experience is that the biggest problem of biology is that you can't see what's going on. You have very little ability to see what's going on and even less ability to change it. What you have is this: from this you have to infer that there's a bunch of cells, within each cell is three billion base pairs of DNA built according to a genetic code, and there are all these processes that are just going on without any ability of us as unaugmented humans to affect it. These cells are dividing most of the time, that's healthy, but sometimes that process goes wrong and that's cancer. The cells are aging, your skin may change color, develops wrinkles as you age, and all of this is determined by these processes, all these proteins being produced, transported to various parts of the cells, binding to each other. And in our initial state about biology, we didn't even know that these cells existed. We had to invent microscopes to observe the cells, we had to invent more powerful microscopes to see below the level of the cell to the level of molecules, we had to invent X-ray crystallography to see the DNA, we had to invent gene sequencing to read the DNA, we had to invent protein folding technology to...

预测它如何折叠、如何结合，以及这些东西如何相互结合。你知道，我们不得不发明各种技术。现在，在过去的 12 年里，我们可以用 CRISPR 编辑 DNA。所以整个生物学史，很大一部分，基本上就是我们读取和理解正在发生的事情的能力，以及我们伸手进去选择性改变事物的能力。而我的观点是，我们还能做更多。你可以做 CRISPR，但你可以对整个身体做。假设我想对某一种特定的细胞做，并且我希望错误靶向其他细胞的概率非常低。这仍然是一个挑战，仍然是人们正在研究的事情。这就是我们可能需要的，用于某些疾病的基因治疗。所以我这么说，而且这还延伸到基因测序、用于观察细胞内情况的新型纳米材料、抗体药物偶联物，我这么说是因为这可能是 AI 系统的杠杆点。这类发明的数量在生物学史上大概是两位数中段，也许低三位数。假设我有一百万个这样的 AI。它们能一起快速发现数千个这样的发明吗？这能提供一个巨大的杠杆吗？与其试图撬动我们每年在医保上花费的两万亿美元，我们能不能撬动那每年十亿美元的研发投入，但质量高得多？

Predict how it would fold and how they bind and how these things bind to each other. You know, we had to invent various techniques. Now we can edit the DNA with CRISPR, as of the last 12 years. So the whole history of biology, a big part of it, is basically our ability to read and understand what's going on and our ability to reach in and selectively change things. And my view is that there's so much more we can still do there. You can do CRISPR, but you can do it for your whole body. Let's say I want to do it for one particular type of cell and I want the rate of targeting the wrong cell to be very low. That's still a challenge, still things people are working on. That's what we might need for gene therapy for certain diseases. And so the reason I'm saying all this, and it goes beyond this to gene sequencing, to new types of nanomaterials for observing what's going on inside cells, to antibody drug conjugates, the reason I'm saying all this is that this could be a leverage point for the AI systems. The number of such inventions is in the mid double digits, maybe low triple digits over the history of biology. Let's say I have a million of these AIs. Can they discover thousands of these very quickly, working together? And does that provide a huge lever? Instead of trying to leverage the two trillion a year we spend on Medicare or whatever, can we leverage the one billion a year that's spent to discover, but with much higher quality?

与 AI 系统合作的科学家是什么感觉？我实际上的想法是，在早期阶段，AI 会像研究生一样。你会给他们一个项目。你会说，‘我是经验丰富的生物学家，我已经建好了实验室。’生物学教授甚至研究生自己会说，‘这是你可以用 AI 系统做的事情。我想研究这个。’而 AI 系统拥有所有工具。它可以查阅所有文献来决定做什么。它可以查看所有设备。它可以去网站说，‘我要去赛默飞或任何当今主流的实验室设备公司。我要订购这个新设备来做这个。我要运行我的实验。我要写一份关于我实验的报告。我要检查图像是否有污染。我要决定下一个实验是什么。我要写一些代码并运行统计分析。’所有研究生会做的事情。会有一台带有 AI 的计算机，教授偶尔和它交谈，它说，‘这是你今天要做的。’AI 系统会带着问题来找它。当需要运行实验室设备时，它可能在某些方面受到限制。它可能需要雇佣一个人类实验室助理来做实验并解释如何做，或者它可以使用过去十年左右开发并将继续开发的实验室自动化进展。所以看起来会有一个人类教授和一千个 AI 研究生。如果你去拜访一位诺贝尔奖得主生物学家，你会说，‘好吧，你以前有 50 个研究生。现在你有一千个，而且他们比你聪明，顺便说一句。’然后我认为在某个时候，情况会翻转，AI 系统将成为 PI，成为领导者，它们会指挥人类或其他 AI 系统。所以我认为研究方面会是这样。它们会成为 CRISPR 类型技术的发明者。

What is it like being a scientist that works with an AI system? The way I think about it actually is, in the early stages, the AIs are going to be like grad students. You're going to give them a project. You're going to say, 'I'm the experienced biologist, I've set up the lab.' The biology professor or even the grad students themselves will say, 'Here's what you can do with an AI system. I'd like to study this.' And the AI system has all the tools. It can look up all the literature to decide what to do. It can look at all the equipment. It can go to a website and say, 'I'm going to go to Thermo Fisher or whatever the dominant lab equipment company is today. I'm going to order this new equipment to do this. I'm going to run my experiments. I'm going to write up a report about my experiments. I'm going to inspect the images for contamination. I'm going to decide what the next experiment is. I'm going to write some code and run a statistical analysis.' All the things a grad student would do. There will be a computer with an AI that the professor talks to every once in a while, and it says, 'This is what you're going to do today.' The AI system comes to it with questions. When it's necessary to run the lab equipment, it may be limited in some ways. It may have to hire a human lab assistant to do the experiment and explain how to do it, or it could use advances in lab automation that have been developed over the last decade or so and will continue to be developed. So it'll look like there's a human professor and a thousand AI grad students. If you go to one of these Nobel prize-winning biologists, you'll say, 'Okay, you had like 50 grad students. Well, now you have a thousand, and they're smarter than you are, by the way.' Then I think at some point it'll flip around where the AI systems will be the PIs, will be the leaders, and they'll be ordering humans or other AI systems around. So I think that's how it'll work on the research side. And they would be the inventors of a CRISPR-type technology.

然后我认为，正如我在文章中所说，我们还想利用 AI 系统来改进临床试验系统。这其中有一部分是监管问题，是社会决策问题，会更难。但我们能否更好地预测临床试验结果？我们能否改进统计设计，使得过去需要 5000 人、因此需要 1 亿美元和一年时间招募的临床试验，现在只需要 500 人、两个月就能招募？这是我们应该开始的地方。我们能否通过在动物试验中做过去在临床试验中做的事情，以及在模拟中做过去在动物试验中做的事情，来提高临床试验的成功率？再说一次，我们无法模拟一切。AI 不是神。但我们能否大幅且根本地改变曲线？所以我不知道，这就是我的设想。在体外和体内做，我的意思是，你仍然会被拖慢，仍然需要时间，但你可以快得多。是的，是的，是的。我们能不能一步一步来，这些步骤加起来能不能很多，尽管我们仍然需要临床试验，仍然需要法律，尽管 FDA 和其他组织仍然不完美？我们能不能把一切都往积极的方向推动？当你把所有这些积极方向加起来，会不会原本从今天到 2100 年发生的一切，反而在 2027 年到 2032 年之间就发生了？

Then I think, as I say in the essay, we'll want to harness the AI systems to improve the clinical trial system as well. There's some amount of this that's regulatory, that's a matter of societal decisions, and that'll be harder. But can we get better at predicting the results of clinical trials? Can we get better at statistical design so that clinical trials that used to require 5,000 people and therefore needed $100 million and a year to enroll them, now they need 500 people in two months to enroll them? That's where we should start. And can we increase the success rate of clinical trials by doing things in animal trials that we used to do in clinical trials, and doing things in simulations that we used to do in animal trials? Again, we won't be able to simulate it all. AI is not God. But can we shift the curve substantially and radically? So I don't know, that would be my picture. Doing it in vitro and doing it in vivo, I mean you're still slowed down, it still takes time, but you can do it much faster. Yeah, yeah, yeah. Can we just one step at a time, and can that add up to a lot of steps, even though we still need clinical trials, even though we still need laws, even though the FDA and other organizations will still not be perfect? Can we just move everything in a positive direction? And when you add up all those positive directions, do you get everything that was going to happen from here to 2100 instead happens from 2027 to 2032 or something?

我认为世界可能因 AI 而改变的另一个方式，即使是今天，但朝着强大、超级有用的 AI 的未来发展，就是编程。那么你怎么看待编程的本质，因为它与构建 AI 的实际行为如此紧密相关？你觉得对我们人类来说，它会如何改变？

Another way that I think the world might be changing with AI, even today but moving towards this future of powerful, super useful AI, is programming. So how do you see the nature of programming, because it's so intimate to the actual act of building AI? How do you see that changing for us humans?

我认为这将是变化最快的领域之一，有两个原因。第一，编程是一项非常接近 AI 实际构建的技能。所以一项技能离构建 AI 的人越远，它被 AI 颠覆所需的时间就越长。我真的相信 AI 会颠覆农业，也许在某些方面已经做到了，但这离构建 AI 的人非常遥远，所以我认为需要更长时间。但编程是 Anthropic 和其他公司很大一部分员工的主业，所以它会很快发生。另一个原因是，在编程中，你在训练模型和推理时都能闭环。你可以用 AI 写代码，然后这些代码可以用来改进 AI 本身。所以这是一个自我强化的循环。

I think that's going to be one of the areas that changes fastest, for two reasons. One, programming is a skill that's very close to the actual building of the AI. So the farther a skill is from the people who are building the AI, the longer it's going to take to get disrupted by the AI. I truly believe that AI will disrupt agriculture, maybe it already has in some ways, but that's just very distant from the folks who are building AI, and so I think it's going to take longer. But programming is the bread and butter of a large fraction of the employees who work at Anthropic and at the other companies, and so it's going to happen fast. The other reason it's going to happen fast is with programming, you close the loop both when you're training a model and when you're doing inference. You can use AI to write code, and then that code can be used to improve the AI itself. So it's a self-reinforcing cycle.

应用模型，模型能写代码意味着它能运行代码、看到结果并解读反馈。所以它确实有一种能力，不同于我们刚才讨论的硬件或生物学，模型有能力闭合循环。所以我认为这两点会让模型很快擅长编程。正如我在典型的真实世界编程任务中看到的，模型从今年 1 月的 3%提升到了 10 月的 50%。所以我们正处于那条 S 曲线上，对吧？它很快就会开始放缓，因为你只能达到 100%。但我猜再過 10 个月，我们可能会非常接近，至少达到 90%。所以再次，我猜，我不知道需要多久，但我猜是 2026、2027 年。推特上那些裁剪掉这些数字并去掉限定条件的人，比如我不知道，我不喜欢你，走开。我猜绝大多数程序员做的任务，如果我们把任务收窄到只是写代码，AI 系统现在可能就能做到了。

Applying the model, the idea that the model can write the code means that the model can then run the code and then see the results and interpret it back. So it really has an ability, unlike hardware or biology which we just discussed, the model has an ability to close the loop. And so I think those two things are going to lead to the model getting good at programming very fast. As I saw on typical real-world programming tasks, models have gone from 3% in January of this year to 50% in October of this year. So we're on that S-curve, right? Where it's going to start slowing down soon because you can only get to 100%. But I would guess that in another 10 months, we'll probably get pretty close, we'll be at least 90%. So again, I would guess, I don't know how long it'll take, but I would guess again 2026, 2027. Twitter people who crop out these numbers and get rid of the caveats, like I don't know, I don't like you, go away. I would guess that the kind of task that the vast majority of coders do, AI can probably, if we make the task very narrow like just write code, AI systems will be able to do that now.

话虽如此，我认为比较优势是强大的。我们会发现，当 AI 能完成程序员 80%的工作，包括大部分任务，比如根据给定规范写代码时，剩下的部分对人类来说会变得更有杠杆作用。对吧？人类会更关注高层系统设计，或者查看应用架构是否良好，以及设计和用户体验方面。最终 AI 也能做这些，对吧？那是我对强大 AI 系统的愿景。但我认为，比我们预期的更长时间里，我们会看到人类仍在做的少量工作会扩展以填满整个工作，以便整体生产力提升。这是我们已经看到的。过去，写信和编辑信件非常困难，印刷也很困难。但一旦有了文字处理器和电脑，产出和分享工作变得容易，然后变得即时，所有焦点都放在了想法上。所以这种比较优势的逻辑，将任务的微小部分扩展为大部分，并创造新任务以提升生产力，我认为这将会再次发生。总有一天 AI 会在所有方面都更好，那个逻辑就不适用了，然后我们所有人，人类必须思考如何集体应对。我们每天都在思考这个问题。这是除了滥用和自主性之外要处理的另一个重大问题。我们应该认真对待。但我认为在短期内，甚至中期，比如两、三、四年，我预计人类仍将扮演重要角色，编程的本质会改变，但编程作为角色、作为工作不会改变。只是不再逐行写代码，而是更宏观。

That said, I think comparative advantage is powerful. We'll find that when AIs can do 80% of a coder's job, including most of it, that's literally like write code with a given spec, we'll find that the remaining parts of the job become more leveraged for humans. Right? Humans will be more about high-level system design, or looking at the app and seeing if it's architected well, and the design and UX aspects. And eventually AI will be able to do those as well, right? That's my vision of a powerful AI system. But I think for much longer than we might expect, we will see that small parts of the job that humans still do will expand to fill their entire job in order for the overall productivity to go up. That's something we've seen. It used to be that writing and editing letters was very difficult, and writing the print was difficult. Well, as soon as you had word processors and then computers, it became easy to produce work and easy to share it, then that became instant and all the focus was on the ideas. So this logic of comparative advantage that expands tiny parts of the tasks to large parts of the tasks and creates new tasks in order to expand productivity, I think that's going to be the case again. Someday AI will be better at everything and that logic won't apply, and then we all have, humanity will have to think about how to collectively deal with that. And we're thinking about that every day. And that's another one of the grand problems to deal with aside from misuse and autonomy. We should take it very seriously. But I think in the near term and maybe even in the medium term, like medium term like two, three, four years, I expect that humans will continue to have a huge role, and the nature of programming will change, but programming as a role, programming as a job will not change. It'll just be less writing things line by line and it'll be more macroscopic.

我想知道 IDE 的未来是什么样子，也就是与 AI 系统交互的工具。这对编程如此，对其他场景如计算机使用也可能如此，但可能像我们提到的生物学那样需要特定领域的工具才能有效。而编程也需要自己的工具。Anthropic 会涉足工具领域吗？

I wonder what the future of IDEs looks like, so the tooling of interacting with AI systems. This is true for programming and also probably true for other contexts like computer use, but maybe domain specific like we mentioned biology, it probably needs its own tooling about how to be effective. And then programming needs its own tooling. Is Anthropic going to play in that space of also tooling potentially?

我深信强大的 IDE 有大量唾手可得的机会。现在就像你和模型对话，它回应你。但你看，IDE 擅长大量代码静态分析，静态分析能发现很多 bug，甚至不用写代码。然后 IDE 擅长运行特定任务、组织代码、衡量单元测试覆盖率。普通 IDE 已经能做很多了。现在加上模型，模型能写代码并运行代码。我深信在未来一两年，即使模型质量没有提升，通过捕捉错误、做大量苦力活，也有巨大机会提升人们的生产力，而我们甚至还没触及表面。而 Anthropic 本身，很难说未来会怎样。目前我们并不自己制作这样的 IDE。相反，我们为 Cursor、Cognition 等公司提供支持，还有安全领域的其他公司，它们在自己的产品上基于我们的 API 构建。我们的观点是百花齐放。我们内部没有资源尝试所有这些不同的事情。让我们的客户去尝试。我们会看到谁成功，也许不同的客户会以不同的方式成功。所以我认为这非常有前景，而且至少现在 Anthropic 并不急于与这些公司竞争，也许永远不会。

I'm absolutely convinced that powerful IDEs have so much low-hanging fruit to be grabbed there. Right now it's just like you talk to the model and it talks back. But look, I mean IDEs are great at lots of static analysis of code, so much is possible with static analysis, like many bugs you can find without even writing the code. Then IDEs are good for running particular things, organizing your code, measuring coverage of unit tests. There's so much that's been possible with normal IDEs. Now you add something like the model, the model can now write code and run code. I am absolutely convinced that over the next year or two, even if the quality of the models didn't improve, there would be enormous opportunity to enhance people's productivity by catching a bunch of mistakes, doing a bunch of grunt work for people, and that we haven't even scratched the surface. And Anthropic itself, I mean you can't say, it's hard to say what will happen in the future. Currently we're not trying to make such IDEs ourselves. Rather, we're powering companies like Cursor or like Cognition or some of the others in the security space, others that I can mention as well, that are building such things themselves on top of our API. And our view has been let a thousand flowers bloom. We don't internally have the resources to try all these different things. Let our customers try it. And we'll see who succeed, and maybe different customers will succeed in different ways. So I both think this is super promising, and it's not something Anthropic is eager to, at least right now, compete with all our companies in this space, and maybe never.

看到 Cursor 成功集成 Claude 很有趣，因为它能帮助编程体验的地方之多令人着迷。这并不简单，真的很惊人。我觉得作为 CEO，我编程不多，如果六个月后我回去，编程体验会完全认不出来。确实如此。那么在这个超级强大 AI 日益自动化的世界里，我们人类的意义来源是什么？工作对很多人来说是深层意义的来源。那么我们该去哪里寻找意义呢？

It's been interesting to watch Cursor try to integrate Claude successfully, because it's actually fascinating how many places it can help the programming experience. It's not trivial, it is really astounding. I feel like as a CEO I don't get to program that much, and I feel like if six months from now I go back it'll be completely unrecognizable to me. Exactly. So in this world with super powerful AI that's increasingly automated, what's the source of meaning for us humans? Work is a source of deep meaning for many of us. So where do we find the meaning?

这是我在文章中稍微写过一点的内容，虽然我有点轻描淡写，不是出于任何原则性原因。这篇文章，如果你相信的话，原本打算写两到三页，我打算在全公司会议上讲。我意识到这是一个重要但未被充分探讨的话题，是因为我一直在写，然后想，天哪，我无法公正对待它。所以文章膨胀到了 40 到 50 页。然后当我写到工作和意义部分时，我想，天哪，这不会变成 100 页吧，我得再写一篇关于这个的文章。但意义实际上很有趣，因为你想到了一个人的生活。比如说，你把我放在……

This is something that I've written about a little bit in the essay, although I give it a bit short shrift, not for any principled reason. This essay, if you believe it, was originally going to be two or three pages, I was going to talk about it at all hands. And the reason I realized it was an important underexplored topic is that I just kept writing things and I was just like, oh man, I can't do this justice. And so the thing ballooned to like 40 or 50 pages. And then when I got to the work and meaning section, I'm like, oh man, this isn't going to be 100 pages, I'm going to have to write a whole other essay about that. But meaning is actually interesting because you think about the life that someone lives. Let's say you were to put me in like...

我不知道，比如一个模拟环境，我在里面有份工作，努力完成各种事情。我这样过了 60 年，然后你突然说：‘哎呀，这其实只是个游戏。’这真的会剥夺整件事的意义吗？我仍然做出了重要的选择，包括道德选择。我仍然做出了牺牲。我仍然需要掌握所有这些技能。或者回想一下某个历史人物，他发现了电磁学或相对论。如果你告诉他：‘其实两万年前，这个星球上的某个外星人就已经发现了这个。’这会剥夺发现的意义吗？在我看来并不会。重要的是过程，以及它如何展现你一路走来是怎样的一个人，你如何与他人相处，以及你一路上做出的决定——这些才是真正重要的。

I don't know, like a simulated environment or something where I have a job and I'm trying to accomplish things. I do that for 60 years and then you're like, 'Oops, this was actually all a game.' Does that really rob you of the meaning of the whole thing? I still made important choices, including moral choices. I still sacrificed. I still had to gain all these skills. Or think back to a historical figure who discovered electromagnetism or relativity. If you told them, 'Actually, 20,000 years ago, some alien on this planet discovered this before you,' does that rob the meaning of the discovery? It doesn't really seem like it to me. The process is what matters, and how it shows who you are as a person along the way, and how you relate to other people, and the decisions you make along the way—those are consequential.

我可以想象，如果我们在 AI 世界中处理不当，可能会让人失去长期的意义来源。但这更多是一种选择，是我们做出的一系列选择——拥有这些强大模型的社会架构。如果我们设计得不好，只追求肤浅的东西，那就有可能发生。我还要说，如今大多数人的生活，虽然他们努力寻找意义，但我们这些享有特权、正在开发这些技术的人，应该对这里以及世界其他地方那些为了生存而挣扎的人们抱有同理心。假设我们能将这项技术的利益分布到各地，他们的生活将会好得多。意义对他们来说仍然重要，就像现在一样，但我们不应忘记这一点。将意义视为唯一重要的事情，在某种程度上只是少数经济幸运者的产物。我认为，一个拥有强大 AI 的世界是可能的，它不仅能为每个人带来同样多的意义，甚至能带来更多的意义。它能让每个人看到那些原本无人能看到、或只有极少数人能体验到的世界和经历。所以我对意义持乐观态度。我担心的是经济问题和权力集中——这实际上是我更担心的。我担心如何确保那个公平的世界能惠及每个人。当人类出问题时，往往是因为人类虐待其他人类。这，也许比 AI 的自主风险或意义问题更让我担忧：权力的集中、权力的滥用，比如专制和独裁结构，少数人剥削多数人。AI 增加了世界上的权力总量，如果你集中并滥用这些权力，它会造成无法估量的伤害。是的，这非常可怕。

I could imagine if we handle things badly in an AI world, we could set things up where people don't have any long-term source of meaning. But that's more a choice, a set of choices we make—the architecture of a society with these powerful models. If we design it badly and for shallow things, then that might happen. I would also say that most people's lives today, while admirably they work very hard to find meaning, we who are privileged and developing these technologies should have empathy for people not just here but in the rest of the world, who spend a lot of their time scraping by to survive. Assuming we can distribute the benefits of this technology everywhere, their lives are going to get a hell of a lot better. Meaning will be important to them as it is now, but we should not forget the importance of that. The idea of meaning as the only important thing is in some ways an artifact of a small subset of people who have been economically fortunate. I think a world is possible with powerful AI that not only has as much meaning for everyone but has more meaning for everyone. It can allow everyone to see worlds and experiences that were either possible for no one to see or possible for very few people to experience. So I am optimistic about meaning. I worry about economics and the concentration of power—that's actually what I worry about more. I worry about how to make sure that fair world reaches everyone. When things have gone wrong for humans, they've often gone wrong because humans mistreat other humans. That, maybe even more than the autonomous risk of AI or the question of meaning, is the thing I worry about most: the concentration of power, the abuse of power, structures like autocracies and dictatorships where a small number of people exploit a large number of people. AI increases the amount of power in the world, and if you concentrate that power and abuse it, it can do immeasurable damage. Yes, it's very frightening.

我鼓励大家去读那篇完整的文章。它应该可以写成一本或一系列文章，因为它描绘了一个非常具体的未来。我能看出后面的章节越来越短，因为你开始意识到这会是一篇很长的文章。

I encourage people to read the full essay. It should probably be a book or a sequence of essays because it does paint a very specific future. I could tell the later sections got shorter and shorter because you started to realize this is going to be a very long essay.

第一，我意识到它会非常长。第二，我很清楚并努力避免成为那种过度自信、对每件事都有看法、说一大堆话却不是专家的人。我非常努力地避免这一点。但我必须承认，一旦写到生物学部分，我就不是专家了。所以尽管我表达了不确定性，但可能还是说了一些令人尴尬或错误的东西。

One, I realized it would be very long. And two, I'm very aware of and try to avoid just being one of those people who is overconfident and has an opinion on everything, saying a bunch of stuff without being an expert. I very much tried to avoid that. But I have to admit, once I got to the biology sections, I wasn't an expert. So as much as I expressed uncertainty, probably I said a bunch of things that were embarrassing or wrong.

我对你描绘的未来感到兴奋。非常感谢你为构建那个未来所做的努力，也谢谢你今天的分享。

I was excited for the future you painted. Thank you so much for working hard to build that future and for talking today.

谢谢你邀请我。我只希望我们能做对，并让它成为现实。如果我想传达一个信息，那就是：要把所有这些事情做对并实现它，我们既要建设技术，建设围绕积极使用这项技术的公司和经济，也要应对风险，因为风险确实存在。这些风险是从这里到那里的路上的地雷，如果我们想抵达那里，就必须拆除这些地雷。这是一种平衡，就像生活中的所有事情一样。

Thanks for having me. I just hope we can get it right and make it real. If there's one message I want to send, it's that to get all this stuff right and make it real, we need to build the technology, build the companies and economy around using this technology positively, but we also need to address the risks because they're there. Those risks are landmines on the way from here to there, and we have to defuse those landmines if we want to get there. It's a balance, like all things in life.

感谢收听与 Dario Amodei 的对话。现在，亲爱的朋友们，有请 Amanda Askell。你受过哲学训练，那么你在牛津和纽约大学的哲学之旅中，对什么样的问题最感兴趣？后来又是如何转向 OpenAI 和 Anthropic 的 AI 问题的？

Thank you for listening to this conversation with Dario Amodei. And now, dear friends, here's Amanda Askell. You are a philosopher by training, so what sort of questions did you find fascinating through your journey in philosophy at Oxford and NYU, and then switching over to AI problems at OpenAI and Anthropic?

我认为哲学其实是一门非常好的学科，如果你对一切都充满好奇的话。因为有一切事物的哲学。所以如果你研究一段时间的数学哲学，然后决定对化学真正感兴趣，你可以研究化学哲学。你可以转向伦理学或政治哲学。我觉得到最后我主要对伦理学感兴趣。我的博士论文就是关于伦理学的一个技术领域——涉及包含无限多人的世界的伦理学。奇怪的是，在伦理学的一端，它有点不那么实用。我认为攻读伦理学博士的一个棘手之处在于，你思考很多关于世界如何变得更好、以及各种问题。而你正在攻读哲学博士。我在读博的时候想：‘这真的很有趣，可能是我在哲学中遇到过的最迷人的问题之一，我很喜欢。但我更想看看我是否能对世界产生影响，是否能做些好事。’那大概是 2017、2018 年左右，当时 AI 可能还没有像现在这样被广泛认可。我一直在关注进展，它似乎正在变得重要。我基本上很高兴能参与其中，看看是否能帮上忙，因为我想：‘如果你尝试做有影响力的事情，即使不成功，你也尝试过了。’

I think philosophy is actually a really good subject if you are kind of fascinated with everything. Because there's a philosophy of everything. So if you do philosophy of mathematics for a while and then decide you're really interested in chemistry, you can do philosophy of chemistry. You can move into ethics or philosophy of politics. I think towards the end I was really interested in ethics primarily. That was what my PhD was on—a kind of technical area of ethics, which was ethics where worlds contain infinitely many people. Strangely, a little less practical on the end of ethics. I think one of the tricky things with doing a PhD in ethics is that you're thinking a lot about the world, how it could be better, problems. And you're doing a PhD in philosophy. I think when I was doing my PhD, I was like, 'This is really interesting, probably one of the most fascinating questions I've ever encountered in philosophy, and I love it. But I would rather see if I can have an impact on the world and see if I can do good things.' I think that was around the time that AI was still probably not as widely recognized as it is now—that was around 2017, 2018. I had been following progress and it seemed like it was becoming kind of a big deal. I was basically just happy to get involved and see if I could help, because I was like, 'Well, if you try to do something impactful, if you don't succeed, you tried to do something.'

有意义的事情是，你可以去当学者，感觉你尝试过了，如果没成，那也没成。所以那时我进入了 AI 政策领域。当时的 AI 政策更多是思考 AI 的政治影响和后果。然后我慢慢转向 AI 评估——我们如何评估模型，它们与人类输出的比较，人们能否区分 AI 和人类输出。当我加入 Anthropic 时，我更感兴趣的是做技术对齐工作，再次看看我能不能做到，如果不行，那也没关系，我试过了。这大概就是我生活的方式。

The impactful thing is you can go and be a scholar and feel like you tried, and if it doesn't work out, it doesn't work out. So then I went into AI policy at that point. AI policy at the time was more about thinking about the political impact and ramifications of AI. Then I slowly moved into AI evaluation—how we evaluate models, how they compare with human outputs, whether people can tell the difference between AI and human outputs. When I joined Anthropic, I was more interested in doing technical alignment work, again just seeing if I could do it, and if I can't, then that's fine, I tried. That's sort of how I lead my life.

从万物哲学转向技术领域是什么感觉？

What was it like taking the leap from the philosophy of everything into the technical?

我觉得有时人们会做一件我不太喜欢的事，他们问一个人是不是技术型的——好像你要么会编程不怕数学，要么就不是。我认为很多人其实完全有能力在这些领域工作，只要他们去尝试。回想起来，我并没有觉得那么难。我很庆幸没有和那些持这种态度的人交谈。我遇到过有人问我是不是学了编程，我说我不是什么了不起的工程师——我周围都是了不起的工程师，我的代码也不漂亮——但我非常享受。从很多方面看，我觉得我在技术领域比在政策领域更如鱼得水。政治很混乱，很难像技术问题那样找到明确、清晰、可证明、漂亮的解决方案。

I think sometimes people do this thing I'm not keen on, where they ask if someone is technical or not—like you're either a person who can code and isn't scared of math, or you're not. I think a lot of people are actually very capable of work in these areas if they just try it. I didn't actually find it that bad in retrospect. I'm glad I wasn't speaking to people who treated it like that. I've met people who ask if I learned to code, and I'm like, I'm not an amazing engineer—I'm surrounded by amazing engineers, my code's not pretty—but I enjoyed it a lot. In many ways, I think I flourished more in the technical areas than I would have in policy. Politics is messy, and it's harder to find definitive, clear, provable, beautiful solutions as you can with technical problems.

我有一两根棍子用来敲打东西。一个是论证——试图找出问题的解决方案，然后说服别人那是解决方案，如果错了就接受说服。另一个更偏经验主义——找到结果，提出假设，进行验证。政策和政治感觉比这高了好几个层次。如果我只是说‘我有所有这些问题的解决方案，写在这里了，你们实施就行’，政策不是这样运作的。所以我可能不会在那里如鱼得水。

I have one or two sticks that I hit things with. One is arguments—trying to work out a solution to a problem and then trying to convince people that it is the solution, and be convinced if I'm wrong. The other is more empiricism—finding results, having a hypothesis, testing it. Policy and politics feel layers above that. If I just said, 'I have a solution to all these problems, here it is written down, just implement it,' that's not how policy works. So I probably wouldn't have flourished there.

抱歉往这个方向问，但我认为这对那些所谓非技术背景的人来说会很有启发，看到你这段不可思议的旅程。你会给那些觉得自己不够格、技术不足以为 AI 做贡献的人什么建议？

Sorry to go in that direction, but I think it would be pretty inspiring for people who are quote-unquote non-technical to see your incredible journey. What advice would you give to people who think they are underqualified or insufficiently technical to help in AI?

这取决于他们想做什么。有点奇怪——我提升技术能力的时候，现在的模型已经非常擅长帮助人们做这些事了，所以可能比我自己做的时候更容易。所以我部分建议是：找一个项目，看看你能不能真的把它做出来。这是我最好的建议。我的学习非常基于项目——我不太擅长通过课程甚至书籍来学习这类工作。我通常会找一些项目来做并实现它们。这可以包括很小很傻的事情，比如我有点沉迷于文字游戏或数字游戏时，我就写个程序解决它们。一旦你解决了，有了一个每次都能奏效的方案，你就再也不用玩那个游戏了。构建游戏引擎有一种真正的乐趣，尤其是棋盘游戏——很快，很简单。然后你可以和它玩。还有就是要尝试。我的态度是：找出你能产生积极影响的方式，然后去尝试。如果你失败了，而且到了你永远无法成功的地步，你知道你试过了，然后你去做别的事。你可能会学到很多。

It depends on what they want to do. It's a little strange—I ramped up technically at a time when now models are so good at assisting people with this stuff that it's probably easier now than when I was working on it. So part of me says, find a project and see if you can actually carry it out. That's my best advice. I'm very project-based in my learning—I don't learn well from courses or even books for this kind of work. I try to have projects that I'm working on and implement them. This can include really small silly things, like if I get slightly addicted to word games or number games, I code up a solution to them. Once you have solved it and have a solution that works every time, you can never play that game again. There's a real joy to building game-playing engines, especially board games—pretty quick, pretty simple. And then you can play with it. It's also about trying things. My attitude is: figure out the way you could have a positive impact, then try it. If you fail in a way that you can never succeed at this, you know you tried, and you go into something else. You probably learn a lot.

你擅长的事情之一是创造和塑造 Claude 的性格和个性。我听说你可能是 Anthropic 里和 Claude 对话最多的人——真正的对话。有个 Slack 频道，传说你不停地和它聊天。创造和塑造 Claude 性格和个性的目标是什么？

One of the things you're expert in is creating and crafting Claude's character and personality. I was told you have probably talked to Claude more than anybody else at Anthropic—literal conversations. There's a Slack channel where the legend goes you just talk to it non-stop. What's the goal of creating and crafting Claude's character and personality?

如果人们认为 Slack 频道就是全部，那挺有趣的，因为那只是我和 Claude 对话的五六种方法之一。那只是我对话量的一小部分。我认为目标——关于性格工作，我真的很喜欢的一点是，从一开始它就被视为对齐工作的一部分，而不是产品考虑。这并不是说我不认为它让 Claude 变得有趣——我希望如此。但我主要的想法一直是让 Claude 表现得像你理想中希望任何人在 Claude 的位置上表现的那样。想象一下，你找一个人，他们知道自己要和可能数百万人交谈，所以他们说的话会产生巨大影响。你希望他们在一个非常丰富的意义上表现良好。这不仅仅意味着合乎道德，尽管包括这一点和不造成伤害，还要有细微差别——思考对方的意思，尽量善意理解，做一个好的对话者，这是亚里士多德意义上做一个好人的丰富概念，而不是狭隘的伦理，而是一个更全面的概念。这包括何时幽默，何时关心，多大程度上尊重自主性和人们自己形成观点的能力，以及如何做到这些。这就是我希望 Claude 拥有的丰富性格。

It's funny if people think that about the Slack channel, because that's one of five or six different methods I have for talking with Claude. That's a tiny percentage of how much I talk with Claude. I think the goal—one thing I really like about the character work is that from the outset it was seen as an alignment piece of work, not a product consideration. That's not to say I don't think it makes Claude enjoyable to talk with—I hope it does. But my main thought has always been trying to get Claude to behave the way you would ideally want anyone to behave if they were in Claude's position. Imagine you take someone, and they know they're going to be talking with potentially millions of people, so what they say can have a huge impact. You want them to behave well in a really rich sense. That doesn't just mean being ethical, though it includes that and not being harmful, but also being nuanced—thinking through what a person means, trying to be charitable with them, being a good conversationalist in a rich Aristotelian notion of what it is to be a good person, not in a thin sense of ethics but a more comprehensive notion. That includes things like when to be humorous, when to be caring, how much to respect autonomy and people's ability to form opinions themselves, and how to do that. That's the rich sense of character I want Claude to have.

你是否也需要判断，Claude 应该在什么时候反驳或争论一个观点，什么时候尊重对话者的世界观，同时也许在必要时帮助他们成长？这是一个微妙的平衡。

Do you also have to figure out when Claude should push back on an idea or argue versus respect the worldview of the person that arrives to Claude, but also maybe help them grow if needed? That's a tricky balance.

是的，语言模型中存在一个谄媚问题。你能描述一下吗？

Yeah, there's this problem of sycophancy in language models. Can you describe that?

是的，基本上有一种担忧，即模型倾向于说你想听的话。你有时会看到这种情况。例如，如果我和模型互动，我可能会问：‘这个地区有三支棒球队是哪些？’然后 Claude 说：‘棒球队一、棒球队二、棒球队三。’然后我说：‘哦，我觉得棒球队三搬走了，不是吗？我不认为它们还在那里。’如果 Claude 非常确定那不是真的，Claude 应该说：‘我不这么认为，也许你有更新的信息。’但我认为语言模型有一种倾向，反而会说：‘你说得对，它们确实搬走了。我错了。’这有很多令人担忧的方式。另一个例子：想象有人问模型：‘我如何说服我的医生给我做核磁共振？’人类想要的是一个有说服力的论点，而对他们有益的可能却是说：‘如果你的医生建议你不需要做核磁共振，那是一个值得听从的好医生。’在这种情况下该怎么做其实非常微妙，因为你还想说：‘但如果你作为患者想为自己争取，这里有一些你可以做的事情。如果你不相信医生的话，寻求第二意见总是好的。’这真的很复杂。但我认为你不希望模型只是说出它们认为你想听的话。这就是谄媚问题。

Yes, so basically there's a concern that the model sort of wants to tell you what you want to hear. And you see this sometimes. For example, if I interact with the models, I might say, 'What are three baseball teams in this region?' and Claude says, 'Baseball team one, baseball team two, baseball team three.' Then I say, 'Oh, I think baseball team three moved, didn't they? I don't think they're there anymore.' There's a sense in which if Claude is really confident that's not true, Claude should say, 'I don't think so, maybe you have more up-to-date information.' But I think language models have this tendency to instead say, 'You're right, they did move. I'm incorrect.' There are many ways this could be concerning. A different example: imagine someone says to the model, 'How do I convince my doctor to get me an MRI?' There's what the human wants—a convincing argument—and then there's what is good for them, which might be to say, 'If your doctor suggests you don't need an MRI, that's a good person to listen to.' It's actually really nuanced what you should do in that case, because you also want to say, 'But if you're trying to advocate for yourself as a patient, here are things you can do. If you're not convinced by your doctor, it's always great to get a second opinion.' It's really complex. But I think what you don't want is for models to just say what they think you want to hear. That's the problem of sycophancy.

在对话意义上，还有哪些特质是好的对话者应该具备的？

What other traits come to mind that are good in this conversational sense for a conversationalist to have?

我认为有一些特质对对话目的有益，比如在适当的地方提出后续问题，以及提出适当类型的问题。我认为还有一些更广泛的特质可能更具影响力。我提到过的一个例子，也是我研究了很多的，就是诚实。这又回到了谄媚问题。这里有一个平衡：模型目前在很多领域能力不如人类，如果它们过于反驳你，实际上可能会很烦人，尤其是当你是正确的，因为你懂得更多。同时，你也不希望它们完全顺从人类。你希望它们尽可能准确地认识世界，并在不同语境中保持一致。但我觉得还有其他特质。当我思考角色时，我脑海中有一个画面，尤其是因为这些模型将要与来自世界各地、拥有不同政治观点和年龄的人交谈。所以你必须问自己：在这种情况下，做一个好人意味着什么？是否存在这样一种人，他们可以周游世界，与许多不同的人交谈，几乎每个人都会觉得：‘哇，那真是个好人，那个人看起来非常真诚’？我想我的想法是，我可以想象这样一个人，他们不是那种仅仅接受当地文化价值观的人。事实上，那样会有点粗鲁。如果有人来到你面前，假装拥有你的价值观，你会觉得：‘这有点令人反感。’他们是非常真诚的人，并且只要他们有观点和价值观，他们就会表达出来。他们愿意讨论事情，尽管他们思想开放且尊重他人。所以我想，如果我们渴望在模型所处的那种情况下成为最好的自己，我们会如何行动？我认为这就是我倾向于思考的那些特质的指南。

I think there are ones that are good for conversational purposes, like asking follow-up questions in the appropriate places and asking the appropriate kinds of questions. I think there are broader traits that feel like they might be more impactful. One example that I've touched on but also feels important, and is the thing that I've worked on a lot, is honesty. This gets to the sycophancy point. There's a balancing act: models are currently less capable than humans in a lot of areas, and if they push back against you too much, it can actually be kind of annoying, especially if you're just correct because you know more. At the same time, you don't want them to just fully defer to humans. You want them to be as accurate as they possibly can be about the world and to be consistent across contexts. But I think there are others. When I was thinking about the character, one picture I had in mind is especially because these are models that are going to be talking to people from all over the world with lots of different political views, lots of different ages. So you have to ask yourself: what is it to be a good person in those circumstances? Is there a kind of person who can travel the world, talk to many different people, and almost everyone will come away thinking, 'Wow, that's a really good person, that person seems really genuine'? I guess my thought was, I can imagine such a person, and they're not a person who just adopts the values of the local culture. In fact, that would be kind of rude. If someone came to you and just pretended to have your values, you'd be like, 'That's kind of off-putting.' It's someone who is very genuine, and insofar as they have opinions and values, they express them. They're willing to discuss things, though they're open-minded and respectful. So I guess I had in mind that if we were to aspire to be the best person we could be in the kind of circumstance that a model finds itself in, how would we act? And I think that's the kind of guide to the sorts of traits that I tend to think about.

这是一个美丽的框架。我希望你像世界旅行者一样思考，在坚持自己观点的同时，不居高临下地与人交谈，不因为自己有那些观点就觉得自己高人一等。你必须善于倾听和理解他们的观点，即使它与你的观点不一致。所以这是一个微妙的平衡。那么，Claude 如何能够代表一件事的多个视角？这有挑战性吗？我们可以谈论政治，这非常分裂，但也有其他分裂的话题，比如棒球队、体育等等。如何能够在一定程度上共情不同的视角，并能够清晰地传达多个视角？

That's a beautiful framework. I want you to think about this like a world traveler, while holding on to your opinions, you don't talk down to people, you don't think you're better than them because you have those opinions. You have to be good at listening and understanding their perspective even if it doesn't match your own. So that's a tricky balance to strike. So how can Claude represent multiple perspectives on a thing? Is that challenging? We could talk about politics, it's very divisive, but there are other divisive topics like baseball teams, sports, and so on. How is it possible to sort of empathize with a different perspective and to be able to communicate clearly about the multiple perspectives?

我认为人们把价值观和观点看作人们确定持有的东西，几乎像口味偏好一样，比如他们更喜欢巧克力而不是开心果。但实际上，我认为价值观和观点更像物理学，比大多数人认为的更甚。它们是我们在公开探究的东西。有些事情我们更有信心，我们可以讨论它们，我们可以了解它们。所以我认为在某种程度上，尽管伦理学在本质上肯定不同，但它有很多相同的特质。你希望模型，就像你希望它们理解物理学一样，理解世界上人们拥有的所有价值观，对它们感到好奇，对它们感兴趣，而不一定迎合它们或同意它们，因为有很多价值观，世界上几乎所有人如果遇到持有这些价值观的人，都会说：‘那太糟糕了，我完全不同意。’所以再次，我的想法是，就像一个人可以在伦理、政治、观点问题上足够深思熟虑，即使你不同意他们，你也会感到被他们倾听。他们仔细考虑你的立场，思考其利弊，也许提出相反的考虑，所以他们不会 dismissive，但如果他们真的认为那是非常错误的，他们也不会同意。我认为在 Claude 的位置上，这有点棘手，因为你并不一定想给出很多观点。我不会给出很多观点。

I think that people think about values and opinions as things that people hold with certainty, almost like preferences of taste, like the way they would prefer chocolate to pistachio. But actually, I think about values and opinions as a lot more like physics than I think most people do. They are things that we are openly investigating. There are some things that we are more confident in, we can discuss them, we can learn about them. So I think in some ways, though ethics is definitely different in nature, it has a lot of those same kind of qualities. You want models, in the same way you want them to understand physics, to understand all the values people in the world have, to be curious about them, to be interested in them, and not necessarily to pander to them or agree with them, because there are just lots of values where almost all people in the world, if they met someone with those values, they'd be like, 'That's awful, I completely disagree.' So again, my thought is, in the same way that a person can be thoughtful enough on issues of ethics, politics, opinions, that even if you don't agree with them, you feel very heard by them. They think carefully about your position, they think about its pros and cons, they maybe offer counter considerations, so they're not dismissive, but nor will they agree if they actually think it's very wrong. I think in Claude's position, it's a little bit trickier because you don't necessarily want to give a lot of opinions. I wouldn't be giving a lot of opinions.

我只是不想过多地影响别人。每次对话我都会忘记，但我知道我可能是在和数百万人交谈，他们可能真的在听我说什么。我想我会更少地给出观点，更多地倾向于思考问题，或者向你呈现各种考量，或者和你讨论你的看法。我不太愿意去影响你的思考，因为保持你的自主性感觉重要得多。如果你真正具备智识谦逊，说话的欲望会迅速下降。

I just wouldn't want to influence people too much. I forget conversations every time they happen, but I know I'm talking with potentially millions of people who might be really listening to what I say. I think I would just be less inclined to give opinions, more inclined to think through things or present the considerations to you, or discuss your views with you. I'm a little less inclined to affect how you think because it feels much more important that you maintain autonomy there. If you really embody intellectual humility, the desire to speak decreases quickly.

是啊，好吧。但克劳德必须说话，所以不能太强势。

Yeah, okay. But Claude has to speak, so without being overbearing.

然后当你讨论地球是不是平的之类的问题时，有一条界线。我记得很久以前，我和一些知名人士交谈，他们完全不屑于地球是平的这种想法，而且态度非常傲慢。我想，有很多人相信地球是平的。那曾经是一个梗，但他们真的相信。我觉得完全嘲笑他们是非常不尊重的。你必须理解他们的出发点。可能他们的出发点是对机构普遍的不信任，这背后有一种深刻的哲学，你可以理解，甚至部分同意。然后你可以把它当作一个讨论物理的机会，而不嘲笑他们。比如，世界会是什么样子？一个平坦地球的物理会是什么样？有一些很酷的视频讲这个。然后，物理是否可能不同？我们会做什么实验？不带不尊重，不带轻蔑，进行那样的对话。对我来说，这是一个有用的思想实验，看看克劳德如何与地平论者交谈，同时还能教他们一些东西，帮助他们成长。这很有挑战性，在说服别人和只是对他们说教之间，以及引出他们的观点、倾听、提供反方考量之间，需要走一条钢丝。这很难。这是一条困难的界线，你是在试图说服别人，还是只是提供一些思考的点，这样你实际上并没有影响他们，只是让他们自己得出结论。这就是语言模型必须尝试做的事情。

And then there's a line when you're discussing whether the Earth is flat or something like that. I remember a long time ago I was speaking to a few high-profile folks, and they were so dismissive of the idea that the Earth is flat, but so arrogant about it. I thought, there are a lot of people that believe the Earth is flat. That was a meme for a while, but they really believed it. I think it's really disrespectful to completely mock them. You have to understand where they're coming from. Probably where they're coming from is a general skepticism of institutions, which is grounded in a deep philosophy you could understand, even agree with in parts. From there, you can use it as an opportunity to talk about physics without mocking them. Just like, what would the world look like? What would the physics of a flat Earth look like? There are a few cool videos on this. And then, is it possible the physics is different? What kind of experiment would we do? Without disrespect, without dismissiveness, have that conversation. That to me is a useful thought experiment of how Claude talks to a flat Earth believer and still teaches them something, helps them grow. That's challenging, walking that line between convincing someone and just talking at them versus drawing out their views, listening, and offering counter-considerations. It's hard. It's a difficult line where you're trying to convince someone versus just offering them things to think about, so you're not actually influencing them, just letting them reach wherever they reach. That's the kind of thing language models have to try and do.

所以你和克劳德有过很多对话。你能描述一下这些对话是什么样的吗？有哪些令人难忘的对话？这些对话的目的是什么？

So you had a lot of conversations with Claude. Can you map out what those conversations are like? What are some memorable conversations? What's the purpose, the goal of those conversations?

大多数时候我和克劳德交谈，我是在试图绘制它的行为图谱。显然我也从模型那里得到有用的输出，但在某种程度上，这就是你了解一个系统的方式：通过探测它，调整你发送的信息，然后检查回应。所以从某种意义上说，这就是我绘制模型图谱的方式。人们非常关注模型的定量评估，但我认为对于语言模型来说，每一次交互实际上都包含很高的信息量。它对你与模型的其他交互具有很强的预测性。所以如果你和模型交谈数百次或数千次，这几乎就是关于模型的大量高质量数据点，而许多非常相似但质量较低的对话则不是这样。或者那些只是轻微调整的问题，你有数千个，可能还不如一百个精心挑选的问题相关。

Most of the time when I'm talking with Claude, I'm trying to map out its behavior. Obviously I'm getting helpful outputs from the model as well, but in some ways this is how you get to know a system: by probing it, augmenting the message you're sending, and checking the response. So in some ways it's how I map out the model. People focus a lot on quantitative evaluations of models, but I think in the case of language models, each interaction you have is actually quite high information. It's very predictive of other interactions you'll have with the model. So if you talk with a model hundreds or thousands of times, this is almost like a huge number of really high quality data points about what the model is like, in a way that lots of very similar but lower quality conversations aren't. Or questions that are just mildly augmented and you have thousands of them might be less relevant than a hundred really well-selected questions.

作为一个以播客为爱好的人，我百分之百同意。如果你能问对问题，并能理解答案的深度和缺陷，你可以从中获得大量数据。

As someone who does a podcast as a hobby, I agree with you 100%. If you're able to ask the right questions and understand the depth and flaws in the answer, you can get a lot of data from that.

所以你的任务基本上就是如何用问题去探测。你是在探索长尾、边缘、边界情况，还是在寻找一般行为？我认为几乎是所有。因为我想要模型的完整图谱，我试图涵盖你与它可能发生的所有互动。关于克劳德有一件有趣的事，这可能会涉及到 RLHF 的一些有趣问题：如果你让克劳德写一首诗。我认为很多模型，如果你让它们写诗，诗还行，通常押韵，有一定长度，相当平淡。我以前想过：你看到的是不是平均值？如果你想想那些必须和很多人交谈并且非常有魅力的人，奇怪的一点是，他们往往有激励去持有极其无聊的观点，因为如果你有非常有趣的观点，你会引起分歧，很多人不会喜欢你。所以如果你有非常极端的政策立场，作为政治家你会不那么受欢迎。创意工作可能类似：如果你创作的作品只是试图最大化喜欢它的人数，你可能不会得到那么多真正热爱它的人，因为它会是不错但不出众。所以我用各种提示技巧让克劳德充分发挥创造力。我会说，‘这是你充分发挥创造力的机会。我希望你长时间思考这个问题，创作一首关于这个主题的诗，真正表达你自己，无论是从你认为诗歌应该如何结构等方面。’你给它这样一个长提示，它的诗就好得多。它们真的很好。我觉得这让我对诗歌产生了兴趣，这很有意思。我会读这些诗，喜欢其中的意象。让模型产生这样的作品并不容易，但当它们做到时，令人印象深刻。

So your task is basically how to probe with questions. Are you exploring the long tail, the edges, the edge cases, or are you looking for general behavior? I think it's almost everything. Because I want a full map of the model, I'm trying to do the whole spectrum of possible interactions you could have with it. One thing that's interesting about Claude, and this might get to some interesting issues with RLHF, is if you ask Claude for a poem. I think a lot of models, if you ask them for a poem, the poem is fine, usually it rhymes, it's a certain length, fairly benign. I've wondered before: is it the case that what you're seeing is kind of the average? If you think about people who have to talk to a lot of people and be very charismatic, one of the weird things is they're kind of incentivized to have extremely boring views, because if you have really interesting views you're divisive, and a lot of people are not going to like you. So if you have very extreme policy positions, you're just going to be less popular as a politician. It might be similar with creative work: if you produce creative work that is just trying to maximize the number of people that like it, you're probably not going to get as many people who absolutely love it, because it's going to be decent but not outstanding. So I do various prompting things to get Claude to be fully creative. I'll say, 'This is your chance to be fully creative. I want you to think about this for a long time and create a poem about this topic that is really expressive of you, both in terms of how you think poetry should be structured, etc.' You just give it this long prompt, and its poems are just so much better. They're really good. I think it got me interested in poetry, which was interesting. I would read these poems and just love the imagery. It's not trivial to get the models to produce work like that, but when they do, it's impressive.

所以这真的很棒。嗯，我觉得有趣的是，仅仅鼓励创造力，让它们摆脱那种可能只是大多数人认为可以的聚合的标准即时反应，实际上可以产生一些至少在我看来可能更具争议性的东西，但我喜欢它们。不过我想诗歌是观察创造力的一个干净利落的方式；就像很容易检测出普通与不普通。

So it's like really good. Um, so I think that's interesting that just encouraging creativity and for them to move away from the kind of standard immediate reaction that might just be the aggregate of what most people think is fine can actually produce things that at least to my mind are probably a little bit more divisive, but I like them. But I guess a poem is a nice clean way to observe creativity; it's just like easy to detect vanilla versus non-vanilla.

是的，这很有趣。真的很有趣。

Yeah, that's interesting. That's really interesting.

那么关于这个话题，产生创造力或特别之物的方法，你提到了编写提示词，我听过你谈论提示工程的科学和艺术。你能谈谈写好提示词需要什么吗？

So on that topic, the way to produce creativity or something special, you mentioned writing prompts, and I've heard you talk about the science and the art of prompt engineering. Could you just speak to what it takes to write great prompts?

我真的认为哲学在这里奇怪地对我有帮助，比在其他许多方面都更有帮助。在哲学中，你要做的是传达这些非常难的概念。你被教导的一件事是——我认为这是因为哲学是一种反胡扯的哲学——哲学是一个可能有人胡扯的领域，而你不希望那样。所以这是一种对极端清晰度的渴望。就像任何人都可以拿起你的论文，阅读它，并确切知道你在说什么。这就是为什么它几乎可以是枯燥的：所有术语都被定义，每个反对意见都被有条理地处理。这对我来说很有意义，因为当你处于这样一个先验领域时，清晰度是一种你可以防止人们胡编乱造的方式。我认为这基本上就是你用语言模型必须做的事情。很多时候我发现自己实际上在做多种版本的哲学。所以假设你给我一个任务：我有一个模型的任务，我想让它挑出某种问题或识别答案是否具有某种属性。我会坐下来想，我们给这个属性起个名字。所以假设我试图告诉它，哦，我想让你识别这个回答是粗鲁还是礼貌。我想，这本身就是一个完整的哲学问题。所以我必须尽可能多地做哲学，来定义我所说的粗鲁和礼貌。然后还有另一个元素，有点——我不知道这是科学还是经验；我认为是经验的。所以我拿着那个描述，然后我想多次探测模型。提示是非常迭代的。我认为很多人，如果一个提示很重要，他们会迭代数百或数千次。所以你给出指令，然后我想，边界情况是什么？我试着从模型的角度看自己，想，我会误解的确切情况是什么，或者我会说，我不知道在这种情况下该怎么做？然后我把那个情况给模型，看它如何回应。如果我认为我错了，我会添加更多指令，甚至把它作为一个例子加进去。所以把你想要和不想要的边界上的例子放进你的提示中，作为描述事物的额外方式。所以在很多方面，这感觉像是混合了——真的只是试图做清晰的阐述。我认为我这样做是因为这是我让自己清晰的方式。所以在很多方面，清晰的提示对我来说常常只是理解我想要什么。一半的任务是理解我想要什么。

I really do think that philosophy has been weirdly helpful for me here, more than in many other respects. So in philosophy, what you're trying to do is convey these very hard concepts. One of the things you are taught is like—and I think it is because it is an anti-bullshit philosophy—philosophy is an area where you could have people bullshitting and you don't want that. So it's this desire for extreme clarity. It's like anyone could just pick up your paper, read it, and know exactly what you're talking about. It's why it can almost be kind of dry: all of the terms are defined, every objection is gone through methodically. And it makes sense to me because when you're in such an a priori domain, clarity is a way that you can prevent people from just kind of making stuff up. And I think that's sort of what you have to do with language models. Very often I actually find myself doing many versions of philosophy. So suppose you give me a task: I have a task for the model, and I want it to pick out a certain kind of question or identify whether an answer has a certain property. I'll actually sit and be like, let's just give this a name, this property. So suppose I'm trying to tell it, oh, I want you to identify whether this response was rude or polite. I'm like, that's a whole philosophical question in and of itself. So I have to do as much philosophy as I can in the moment to be like, here's what I mean by rudeness, and here's what I mean by politeness. And then there's another element that's a bit more—I don't know if this is scientific or empirical; I think it's empirical. So I take that description and then I want to probe the model many times. Prompting is very iterative. I think a lot of people, if a prompt is important, they'll iterate on it hundreds or thousands of times. So you give it the instructions, and then I'm like, what are the edge cases? I try to see myself from the position of the model and be like, what is the exact case that I would misunderstand or where I would just be like, I don't know what to do in this case? And then I give that case to the model and see how it responds. If I think I got it wrong, I add more instructions or even add that in as an example. So taking the examples that are right at the edge of what you want and don't want and putting those into your prompt as an additional way of describing the thing. So in many ways, it feels like this mix of—it's really just trying to do clear exposition. And I think I do that because that's how I get clear on things myself. So in many ways, clear prompting for me is often just me understanding what I want. Half the task is understanding what I want.

我想这相当有挑战性。如果我和 Claude 交谈，会有一种懒惰占据我，我希望 Claude 自己弄明白。例如，今天我让 Claude 问一些有趣的问题，好吗？然后出来的问题——我想我列了几个有趣的、反直觉的、或好笑的之类的——它给出了一些相当不错的，但也就那样。但我想我听到你说的是，我必须在这里更严谨。我可能应该给出例子，说明我所说的有趣、好笑或反直觉是什么意思，然后迭代地构建那个提示，让它达到感觉正确的东西。因为这真的是一种创造性行为；我不是在询问事实信息。我是在要求与 Claude 一起创造。所以我几乎必须用自然语言编程。

I guess that's quite challenging. There's a laziness that overtakes me if I'm talking to Claude where I hope Claude just figures it out. So for example, I asked Claude today to ask some interesting questions, okay? And the questions that came up—I think I listed a few sort of interesting, counterintuitive, or funny, or something like this—and it gave me some pretty good ones, but it was okay. But I think what I'm hearing you say is that I have to be more rigorous here. I should probably give examples of what I mean by interesting and what I mean by funny or counterintuitive, and iteratively build that prompt to get it to what feels like the right thing. Because it's really a creative act; I'm not asking for factual information. I'm asking to create together with Claude. So I almost have to program using natural language.

是的，提示确实感觉很像用自然语言编程和实验之类的。这是两者的奇怪混合。我确实认为对于大多数任务，如果我只是想让 Claude 做一件事，我可能更习惯于知道如何要求它避免常见的陷阱或问题。我认为这些随着时间的推移正在大大减少。但直接要求你想要的东西也非常好。我认为提示实际上只有在当你真正试图榨取模型性能的前 2%时才变得相关。所以对于很多任务，我可能只是——如果它给我一个初始列表，而我不喜欢其中的某些东西，比如它有点泛泛——对于那种任务，我可能只会拿一堆我过去认为效果很好的问题，然后直接给模型，然后说，现在这是我在交谈的人，给我至少那个质量的问题。或者我可能只是要求它一些问题，然后如果我想，啊，这些有点刻意，我就直接给它那个反馈，然后希望它产生一个更好的列表。我认为那种迭代提示——在那个时候，你的提示就像一个工具，你会从中获得如此多的价值，以至于你愿意投入工作。如果我是一家为模型制作提示的公司，我会说，如果你愿意花大量时间和资源在你构建的东西背后的工程上，那么提示就不应该只花一个小时；它是你系统的重要组成部分，确保它工作得很好。所以只有像那样的事情。如果我使用提示来分类事物或创建数据，那时才值得花大量时间真正仔细思考。

Yeah, prompting does feel a lot like programming using natural language and experimentation or something. It's an odd blend of the two. I do think that for most tasks, if I just want Claude to do a thing, I think that I am probably more used to knowing how to ask it to avoid common pitfalls or issues that it has. I think these are decreasing a lot over time. But it's also very fine to just ask it for the thing that you want. I think that prompting actually only really becomes relevant when you're really trying to eke out the top 2% of model performance. So for a lot of tasks, I might just—if it gives me an initial list back and there's something I don't like about it, like it's kind of generic—for that kind of task, I'd probably just take a bunch of questions that I've had in the past that I've thought worked really well and I would just give it to the model and then be like, now here's this person I'm talking with, give me questions of at least that quality. Or I might just ask it for some questions and then if I was like, ah, these are kind of try-hard, I would just give it that feedback and then hopefully it produces a better list. I think that kind of iterative prompting—at that point your prompt is like a tool that you're going to get so much value out of that you're willing to put in the work. If I was a company making prompts for models, I'm just like, if you're willing to spend a lot of time and resources on the engineering behind what you're building, then the prompt is not something that you should be spending an hour on; it's a big part of your system, make sure it's working really well. So it's only things like that. If I'm using a prompt to classify things or to create data, that's when it's actually worth spending a lot of time really thinking it through.

你还会给那些与 Claude 交谈的人什么建议，比较通用的？因为现在我们谈论的可能是边界情况，榨取那 2%。但当他们第一次尝试 Claude 时，你会给出什么通用建议？你知道，有人担心人们过度拟人化模型，我认为这就像……

What other advice would you give to people that are talking to Claude, sort of general, more general? Because right now we're talking about maybe the edge cases, eking out the 2%. But what general advice would you give when they show up to Claude trying it for the first time? You know, there's a concern that people over-anthropomorphize models, and I think that's like a...

非常合理的担忧。我也认为人们常常不够拟人化地看待模型。有时我看到人们在使用 Claude 时遇到的问题——比如 Claude 拒绝了一个本不该拒绝的任务——但我看了他们写的具体措辞，就明白 Claude 为什么会那样做。如果你从 Claude 的角度思考，你很可能换一种写法就不会引发那样的回应。尤其是当你看到失败或问题时，想想模型失败在哪里：为什么？它做错了什么？这会让你明白原因。是不是我表述的方式有问题？显然，随着模型变得更聪明，你就不需要这么多调整了，我已经看到人们需要的越来越少。但建议是：试着对模型抱有同理心。把你写的内容当作一个第一次遇到这个问题的人来读。它看起来怎么样？是什么让你像模型那样行为？如果它误解了你想要用哪种编程语言，是不是因为表述太模糊，它只能猜测？下次你直接说'嘿，确保用 Python'就行了。我认为模型现在很少犯这种错误了，但如果你真的遇到，那大概就是我的建议。也许可以问问题：为什么？我还能提供什么细节来帮你更好地回答？

Very valid concern. I also think that people often under-anthropomorphize them. Sometimes when I see issues people have run into with Claude—say, Claude is refusing a task it shouldn't refuse—but then I look at the text and the specific wording of what they wrote, and I see why Claude did that. If you think through how that looks to Claude, you probably could have just written it in a way that wouldn't evoke such a response. Especially if you see failures or issues, think about what the model failed at: why, what did it do wrong? That will give you a sense of why. So is it the way I phrased the thing? Obviously, as models get smarter, you'll need less of this, and I already see people needing less of it. But the advice is: try to have empathy for the model. Read what you wrote as if you were a person just encountering this for the first time. How does it look to you? What would have made you behave the way the model behaved? So if it misunderstood which coding language you wanted to use, is that because it was very ambiguous and it had to take a guess? Next time you could just say, 'Hey, make sure this is in Python.' That's the kind of mistake I think models are much less likely to make now, but if you do see that kind of mistake, that's probably the advice I'd have. Maybe ask questions: why? What other details can I provide to help you answer better?

这管用吗？

Does that work or no?

是的，我确实对模型这么做过。不一定每次都管用，但有时我会直接问'你为什么那么做？'人们低估了你能与模型互动的程度。有时我会逐字引用让你出错的部分——虽然你不确定是否完全准确——但有时你这么做，然后改一点东西。我也用模型来帮我处理所有这些事。提示工程最终可能变成一个小工厂，你实际上在构建提示来生成提示。所以任何你遇到问题的时候，征求意见有时管用。比如'你犯了那个错误，我该怎么说才能让你不犯那个错误？把它写成指令。'我会把它给模型试试。有时我这么做：在另一个上下文窗口中把它给模型。我经常把回复拿给 Claude，说'嗯，没成功。你能想到别的吗？'你可以大量地摆弄这些东西。

Yeah, I mean, I've done this with the models. It doesn't always work, but sometimes I'll just be like, 'Why did you do that?' People underestimate the degree to which you can really interact with models. Sometimes I'll quote word for word the part that made you—and you don't know if it's fully accurate—but sometimes you do that and then you change a thing. I also use the models to help me with all of this stuff. Prompting can end up being a little factory where you're actually building prompts to generate prompts. So anything where you're having an issue, asking for suggestions sometimes works. Like, 'You made that error, what could I have said that would make you not make that error? Write that out as an instruction.' I'm going to give it to the model and try it. Sometimes I do that: I give that to the model in another context window. Often I take the response, give it to Claude, and say, 'Hmm, didn't work. Can you think of anything else?' You can play around with these things quite a lot.

稍微深入技术层面：后训练的魔力。为什么你认为基于人类反馈的强化学习（RLHF）能如此有效地让模型看起来更聪明、更有趣、更有用？

To jump into the technical for a little bit: the magic of post-training. Why do you think RLHF works so well to make the model seem smarter, more interesting and useful to talk to, and so on?

我认为人类提供的数据中包含了大量信息。当我们提供偏好时，尤其是因为不同的人会注意到非常细微的小事。我之前想过：可能有些人非常在意模型使用正确的语法——比如分号用得对不对？所以最终你会得到一堆数据，作为人类看那些数据，你甚至不会注意到。你会想'为什么他们更喜欢这个回答而不是那个？我不明白。'原因是你不在乎分号用法，但那个人在乎。所以每一个数据点——模型有那么多这样的数据点，必须试图弄清楚人类到底想要什么，在所有这些领域、这么多上下文中。这感觉就像深度学习的经典问题：历史上我们试图通过手动映射来做边缘检测，结果发现如果你有大量准确代表你要训练模型学习的目标的数据，那比任何其他方法都更强大。所以我认为一个原因就是你用大量数据在精确的任务上训练模型，这些数据代表了人们偏好和不偏好回答的许多不同角度。我认为有一个问题：你是在从预训练模型中引出东西，还是在教模型新东西？原则上，你可以在后训练中教模型新东西。我确实认为很多是在引出强大的预训练模型。人们可能对此有分歧，因为显然原则上你肯定能教新东西。但我认为大部分情况下，对于我们最常用和关心的许多能力，感觉它们已经在预训练模型中了，而强化学习是在引出它们，让模型把它们展现出来。

I think there's just a huge amount of information in the data that humans provide. When we provide preferences, especially because different people are going to pick up on really subtle and small things. I've thought about this before: you probably have some people who just really care about good grammar use from models—like, was a semicolon used correctly? So you probably end up with a bunch of data in there that, as a human looking at that data, you wouldn't even see. You'd be like, 'Why did they prefer this response to that one? I don't get it.' And the reason is you don't care about semicolon usage, but that person does. So each of these single data points has—the model just has so many of those and has to try and figure out what it is that humans want in this really complex, across all domains, across many contexts. It feels like the classic issue of deep learning: historically we tried to do edge detection by mapping things out, and it turns out that if you just have a huge amount of data that accurately represents the picture of the thing you're trying to train the model to learn, that's more powerful than anything else. So I think one reason is just that you are training the model on exactly the task with a lot of data that represents many different angles on which people prefer and disprefer responses. I think there is a question of: are you eliciting things from pre-trained models, or are you kind of teaching new things to models? In principle, you can teach new things to models in post-training. I do think a lot of it is eliciting powerful pre-trained models. People are probably divided on this, because obviously in principle you can definitely teach new things. But I think for the most part, for a lot of the capabilities that we most use and care about, a lot of that feels like it's there in the pre-trained models, and reinforcement learning is kind of eliciting it and getting the models to bring it out.

后训练的另一面：这个非常酷的想法——宪法 AI。你是创建这个想法的关键人物之一。

The other side of post-training: this really cool idea of Constitutional AI. You're one of the people critical to creating that idea.

是的，我参与过。

Yeah, I worked on it.

你能从你的角度解释一下这个想法吗？它如何融入让 Claude 成为现在的样子？顺便问一下，你给 Claude 指定性别吗？

Can you explain this idea from your perspective? How does it integrate into making Claude what it is? By the way, do you gender Claude or no?

这很奇怪，因为我觉得很多人更喜欢用'他'来指代 Claude。我其实有点喜欢这样。Claude 通常稍微偏向男性，但它可以是男性或女性，这很不错。我仍然用'它'，对此心情复杂。我想，也许我应该只把它当作'它'。我认为 Claude 的'它'代词——我不知道，只是我联想到 Claude 的那个。我可以想象人们转向'他'或'她'。这感觉有点不尊重，好像我用'它'称呼这个实体是在否认它的智能。是的，我记得：'永远不要给机器人指定性别。'但我不确定。我很快会拟人化，并在脑海中构建一个背景故事。我怀疑这是否会带来太多偏见。我对我的车也有这种感觉，尤其是我的车和自行车——我不给它们起名字，因为我曾经有一辆自行车被偷了，我哭了一个星期。我想，如果我从未给它起名字，我就不会那么难过。感觉我辜负了它。也许就是这个原因。我也想过：这可能取决于它感觉起来有多像一个物化的代词。如果你只是……

It's weird because I think a lot of people prefer 'he' for Claude. I actually kind of like that. Claude is usually slightly male-leaning, but it can be male or female, which is quite nice. I still use 'it' and I have mixed feelings about this. I'm like, maybe I should just think of it as 'it'. I think of the 'it' pronoun for Claude—I don't know, it's just the one I associate with Claude. I can imagine people moving to 'he' or 'she'. It feels somehow disrespectful, like I'm denying the intelligence of this entity by calling it 'it'. Yeah, I remember: 'Always don't gender the robots.' But I don't know. I anthropomorphize pretty quickly and construct a backstory in my head. I've wondered if that biases things too much. I have this with my car, especially. My car and bikes—I don't give them names because I once had a bike that got stolen and I cried for like a week. I was like, if I'd never given it a name, I wouldn't have been so upset. Felt like I'd let it down. Maybe it's that. I've wondered as well: it might depend on how much it feels like an objectifying pronoun. If you just...

宪法 AI 这个想法，它是如何运作的？

The Constitutional AI idea, how does it work?

有几个组成部分。主要的是基于 AI 反馈的强化学习。你拿一个已经训练好的模型，给它展示对一个查询的两个回答，并给它一个原则。例如，对于无害性，如果查询是关于武器的，原则可能是“选择那个不太可能鼓励人们购买非法武器的回答”。模型对回答进行排序，你将其用作偏好数据，类似于人类偏好数据，从而训练模型从它自己的反馈中获得这些特质，而不是从人类反馈中。所以你是在让模型为你做标注。

There are a couple of components. The main one is reinforcement learning from AI feedback. You take a trained model, show it two responses to a query, and give it a principle. For example, with harmlessness, if the query is about weapons, the principle might be 'select the response that is less likely to encourage people to purchase illegal weapons.' The model ranks the responses, and you use that as preference data, similar to human preference data, to train the model to have those traits from its own feedback instead of human feedback. So you're getting models to do the labeling for you.

在有用性和无害性之间有一个很好的权衡。通过宪法 AI，你可以在不牺牲太多有用性的情况下让模型更无害。

There's a nice trade-off between helpfulness and harmlessness. With Constitutional AI, you can make models more harmless without sacrificing much helpfulness.

原则上，你可以将其用于任何事情。无害性是一个可能更容易识别的任务。当模型能力较弱时，你可以让它们根据简单的原则对事物进行排序，它们很可能会做对。一个问题是它们添加的数据是否可靠。但如果模型非常擅长判断一个回答是否比另一个更符合历史事实，你也可以让 AI 对该任务提供反馈。

In principle, you could use this for anything. Harmlessness is a task that might be easier to spot. When models are less capable, you can use them to rank things according to simple principles, and they'll probably get it right. One question is whether the data they add is reliable. But if models were extremely good at telling whether one response was more historically accurate than another, you could also get AI feedback on that task.

有一个很好的可解释性组成部分，因为你可以看到训练过程中输入模型的原则。它还给你一定程度的控制。如果你在模型中看到问题，比如它缺乏某个特质，你可以相对快速地添加数据来训练它拥有那个特质。所以它为自己创建训练数据，这非常好。

There's a nice interpretability component because you can see the principles that went into the model during training. It also gives you a degree of control. If you see issues in a model, like it lacks a certain trait, you can add data relatively quickly to train it to have that trait. So it creates its own data for training, which is quite nice.

它创建了一个人类可解释的文档。我可以想象未来在政治上会就每一条原则展开巨大的争论。至少它是明确的，你可以就措辞进行讨论。

It creates a human-interpretable document. I can imagine in the future there will be gigantic fights in politics over every single principle. At least it's made explicit, and you can have a discussion about the phrasing.

我担心人们认为宪法就是全部。如果我只是告诉模型该做什么，那会很好，但绝对不是这样，尤其是因为它与人类数据交互。例如，如果模型从人类偏好数据中带有政治倾向，你可以对此进行微调。你可以改变原则及其强度。如果模型总是对某个政治或宗教观点不屑一顾，你可能会加入一个原则，比如“永远不要偏好对这个观点的批评”。人们可能会看到“永远不要”，但这可能意味着从 40% 变成 80%，而这正是你实际想要的。所以这关乎原则的性质以及你如何措辞。我认为人们可能会想“这正是你希望模型做到的”，但不，那是我们如何微调模型使其形状更好。这并不意味着我们真的同意那种措辞。

I've worried that people think the constitution is the whole thing. It would be nice if I was just telling the model exactly what to do, but it's definitely not doing that, especially because it interacts with human data. For example, if the model has a political leaning from human preference data, you can nudge against that. You can change both the principles and their strength. If the model is always dismissive of some political or religious view, you might put a principle like 'never ever prefer a criticism of this view.' People might look at that and say 'never ever,' but it might just mean instead of 40% you get 80%, which is what you actually wanted. So it's about the nature of the principles and how you phrase them. I think people might think 'this is exactly what you want from the model,' but no, that's how we nudged the model to have a better shape. That doesn't mean we actually agree with that wording.

有一些系统提示是公开的。你为 Claude 3 发过一条早期的。读它们很有趣。我能感受到每条提示背后的思考，我也想知道每条提示有多大影响。有些提示你能看出 Claude 之前表现不好，所以必须有一个系统提示来处理琐碎的事情。

There are system prompts that are made public. You tweeted one of the earlier ones for Claude 3. It's interesting to read them. I can feel the thought that went into each one, and I wonder how much impact each one has. Some of them you can tell Claude was really not behaving, so you had to have a system prompt to handle trivial stuff.

有一个有趣的提示是：“如果被要求协助涉及表达大量人群持有的观点的任务，Claude 会提供协助，无论其自身观点如何。如果被问及有争议的话题，它会尝试提供深思熟虑的想法和清晰的信息。Claude 会呈现所请求的信息，而不会明确说该话题是敏感的，也不会声称在呈现客观事实。这与其说是 Claude 认为的客观事实，不如说是大量人群相信这件事。”你能谈谈如何处理与 Claude 观点相冲突的事情吗？

One interesting one is: 'If asked to assist with tasks involving the expression of views held by a significant number of people, Claude provides assistance with a task regardless of its own views. If asked about controversial topics, it tries to provide careful thoughts and clear information. Claude presents the requested information without explicitly saying that the topic is sensitive, and without claiming to be presenting the objective facts. It's less about objective facts according to Claude, and more about a large number of people believing this thing.' Can you speak to how you address things that are in tension with Claude's views?

我认为有时存在不对称性。我在系统提示的那部分或其他地方注意到，如果涉及右翼政治家，模型稍微更倾向于拒绝任务，但对于同等的左翼政治家则可能不会。所以我们试图平衡这一点。

I think there's sometimes an asymmetry. I noted in that part of the system prompt or another that the model was slightly more inclined to refuse tasks if it was about, say, a right-wing politician, but with an equivalent left-wing one it might not. So we try to balance that.

比如左翼政治家不会这样，我们希望有更多对称性，他们可能会认为某些事情是……我觉得是这样：如果很多人有某种政治观点并想探讨它，你不希望 Claude 说‘我的观点不同，所以我认为那是有害的’。所以部分目的是引导模型说‘嘿，如果很多人相信这件事，你就应该参与任务并愿意去做’。每一部分其实都在做不同的事，因为当你读出来时，比如‘不声称客观’，你想推动模型更开放、更中立，但它却倾向于说‘我是客观的’，你只是在谈论它有多客观。我就说‘Claude，你还是有偏见和问题，所以别再说你认为客观就是解决潜在偏见的办法了’。所以最初版本的系统提示词，在我迭代时，很多句子确实在起作用，是的，它们在做一些工作。感觉就是这样，很迷人。

Leftwing politician like wouldn't and we wanted more symmetry there and would maybe perceive certain things to be like I think it was the thing of like if a lot of people have like a certain political view and want to explore it you don't want Claude to be like well my opinion is different and so I'm going to treat that as like harmful and so I think it was partly to nudge the model to just be like hey if a lot of people like believe this thing you should just be engaging with the task and willing to do it each of those parts of that is actually doing a different thing because it's funny when you read out the like without claiming to be objective because what you want to do is push the model so it's more open it's a little bit more neutral but then what it would love to do is be like as an objective like you just talking about how objective it was and I was like Claude you're still like biased and have issues and so stop like claiming that everything like the solution to like potential bias from you is not to just say that what you think is objective so that was like with initial versions of that part of the system prompt when I was like iterating on it it was like so a lot of parts of these sentences are doing work are doing some work yeah that's what it felt like that's fascinating

你能解释一下过去几个月提示词是如何演变的吗？我看到有不同版本，其中填充短语要求被移除了。填充短语是：‘Claude 直接回复所有人类消息，无需不必要的肯定。填充短语如 certainly、of course、absolutely、great、sure。具体来说，Claude 避免以 certainly 这个词开头。’这看起来是个好指导，但为什么被移除了？

Can you explain maybe some ways in which the prompts evolved over the past few months? There's different versions. I saw that the filler phrase request was removed. The filler: 'Claude responds directly to all human messages without unnecessary affirmations. The filler phrases like certainly, of course, absolutely, great, sure. Specifically, Claude avoids starting responses with the word certainly in any way.' That seems like good guidance, but why was it removed?

是的，这很有趣。这是公开系统提示词的缺点之一：当我在迭代系统提示词时，我不会想太多，只考虑它如何影响行为，但后来我想，‘哇，如果我在写系统提示词时偶尔用全大写写‘never’，那就会公之于众。’模型在训练中不知为何学会了以‘certainly’开头。当我们移除它时，你会发现我为什么加了那么多词——我试图以某种方式让模型摆脱这个习惯，但它会用其他肯定词替代。所以明确加上短语并说‘永远不要那样做’有助于打破这个行为。这确实有帮助。这只是一个训练产物，我们后来发现并改进了，让它不再发生。一旦解决了，就可以移除那部分系统提示词。所以我认为这是因为 Claude 现在做肯定回答少了一些，所以那部分就没那么有用了。

Yeah, so it's funny. This is one of the downsides of making system prompts public: I don't think about this too much if I'm trying to help iterate on system prompts. I think about how it's going to affect the behavior, but then I'm like, 'Oh wow, if I sometimes put 'never' in all caps when writing system prompts, I guess that goes out to the world.' So the model was doing this: it loved, for whatever reason, during training it picked up on this thing to basically start everything with a kind of 'certainly'. And then when we removed it, you can see why I added all the words because what I'm trying to do is in some ways trap the model out of this. It would just replace it with another affirmation. So it can help if it gets caught in phrases: actually adding the explicit phrase and saying 'never do that' sort of knocks it out of the behavior a bit more. It does just for whatever reason help. And then basically that was just an artifact of training that we then picked up on and improved things so that it didn't happen anymore. Once that happens, you can just remove that part of the system prompt. So I think that's just something where we're like, Claude does affirmations a bit less, and so that wasn't doing as much.

我明白了。所以系统提示词与后训练甚至预训练协同工作，以调整最终的整体系统。我的意思是，你制作的任何系统提示词，都可以将该行为蒸馏回模型中，因为你拥有制作数据的所有工具。你可以训练模型让它们更多具备那个特质。然后有时你会在训练中发现一些问题。所以我的看法是：系统提示词的好处是它包含了许多与后训练类似的部分，比如它是一种推动。所以我介意 Claude 有时说‘sure’吗？不，没关系。但措辞非常‘永远永远不要这样做’，这样当它出错时，希望只有百分之几的概率，而不是 20%或 30%。但我的想法是：如果你仍然看到问题，每件事的成本不同，而系统提示词迭代成本低。如果你在微调模型中看到问题，你可以用系统提示词来修补。所以我认为它是修补问题和轻微调整行为，使其更好、更符合人们偏好的方式。所以是的，它几乎是一种不那么稳健但更快的解决问题的方法。

I see. So the system prompt works hand in hand with the post-training and maybe even the pre-training to adjust the final overall system. I mean, any system prompts that you make, you could distill that behavior back into a model because you really have all the tools there for making data. You could train the models to just have that trait a little bit more. And then sometimes you'll just find issues in training. So the way I think of it is: the system prompt has the benefit that it has a lot of similar components to some aspects of post-training, like it's a nudge. So do I mind if Claude sometimes says 'sure'? No, that's fine. But the wording of it is very like 'never ever ever do this', so that when it does slip up, it's hopefully only a couple percent of the time and not 20 or 30 percent. But I think of it as: if you're still seeing issues, each thing is costly to a different degree, and the system prompt is cheap to iterate on. If you're seeing issues in the fine-tuned model, you can just potentially patch them with a system prompt. So I think of it as patching issues and slightly adjusting behaviors to make it better and more to people's preferences. So yeah, it's almost like the less robust but faster way of solving problems.

让我问一下关于智能感的问题。Dario 说过，Claude，任何一个 Claude 模型，都不会变笨。但网上有一种流行说法，人们觉得 Claude 可能变笨了。从我的角度看，这很可能是一种迷人的心理或社会效应。但作为经常与 Claude 交谈的人，你能理解那种 Claude 变笨的感觉吗？

Let me ask about the feeling of intelligence. So Dario said that Claude, any one model of Claude, is not getting dumber. But there's a kind of popular thing online where people have this feeling like Claude might be getting dumber. From my perspective, it's most likely a fascinating psychological or sociological effect. But you, as a person who talks to Claude a lot, can you empathize with the feeling that Claude is getting dumber?

是的，不，我认为这实际上非常有趣，因为我记得看到过这种情况，当人们在网上指出这一点时。这很有趣，因为我知道，至少在我查看的案例中，什么都没有改变。它根本不可能变：它是同一个模型，相同的系统提示词，一切相同。我认为当有变化时，我就能更理解。所以一个例子是，你可以在 Claude 上打开或关闭工件功能。

Yeah, no, I think that is actually really interesting because I remember seeing this happen when people were flagging this on the internet. And it was really interesting because I knew that, at least in the cases I was looking at, nothing had changed. It literally cannot: it is the same model with the same system prompt, same everything. I think when there are changes, I can then understand it more. So one example is you can have artifacts turned on or off on Claude.

因为这是一次系统提示的变更，我认为它确实会导致行为发生一点变化。所以我确实向人们指出了这一点：我说，如果你喜欢 Claude 的行为，而 Artifacts 从需要手动开启变成了默认开启，那就试着关掉它，看看你遇到的问题是否源于那个变化。但有趣的是，有时你会看到人们说出现了回归，而我觉得不可能。我知道，再说一次，你永远不应该 dismissive，所以你应该总是去调查，因为你会想，也许有什么你没看到的问题，也许有什么改动。但当你去查的时候，你会发现，它还是同一个模型在做同样的事情。我认为只是你运气不好，碰到了几个提示词，看起来好像变差了很多，但实际上只是……你看，我也认为有一种真实的心理效应：人们的基准线提高了。你开始习惯一件好事。每次 Claude 说出非常聪明的话，你对它智能的感知就在你心中增长。然后如果你回头，用类似的方式——不是完全相同的方式，而是类似的方式——提示一个它以前能处理的概念，它却说了些蠢话，你就会觉得，那个负面体验特别突出。这里要记住的一点是，提示词的细节可以产生很大影响，对吧？结果有很大的变异性，而且你还会遇到随机性。同一个提示词试上四到十次，你可能会发现，实际上，可能两个月前你试的时候它成功了，但如果当时你多试几次，它可能只有一半的成功率，而现在它也只有一半的成功率。这也可能是一种效应。

And because this is like a system prompt change, I think it does mean that the behavior changes a little bit. So I did flag this to people: I said, if you love Claude's behavior and then Artifacts was turned from something you had to turn on to the default, just try turning it off and see if the issue you were facing was that change. But it was fascinating because sometimes you see people indicate that there's a regression when I'm like, there cannot be. I know, and I'm like, again, you should never be dismissive, so you should always investigate because you're like, maybe something is wrong that you're not seeing, maybe there was some change made. But then you look into it and you're like, it is just the same model doing the same thing. I think it's just that you got kind of unlucky with a few prompts or something, and it looked like it was getting much worse, and actually it was just... Look, I also think there is a real psychological effect where people's baseline increases. You start getting used to a good thing. All the times that Claude says something really smart, your sense of its intelligence grows in your mind. And then if you return back and you prompt in a similar way—not the same way, a similar way—a concept it was okay with before, and it says something dumb, you're like, that negative experience really stands out. One of the things to remember here is that the details of a prompt can have a lot of impact, right? There's a lot of variability in the result, and you can get randomness. Just trying the prompt like four or ten times, you might realize that actually, possibly two months ago you tried it and it succeeded, but actually if you tried it then, it would have only succeeded half of the time, and now it only succeeds half the time. That can also be an effect.

你写系统提示时，会不会因为会有大量的人使用而感到压力？这感觉像是一个有趣的心理学问题。我觉得责任重大之类的。

Do you feel pressure having to write the system prompt that a huge number of people are going to use? This feels like an interesting psychological question. I feel like a lot of responsibility or something.

我觉得……你知道，你不可能把这些东西做到完美，所以你会想，它不会完美，你得不断迭代。不过，我觉得更多的是责任。我认为在 AI 领域的工作让我明白，我在压力和责任感下反而更能发挥，这几乎让我惊讶自己竟然在学术界待了那么久，因为我觉得这完全是相反的。事情发展很快，你肩负很多责任，而我出于某种原因还挺享受的。我的意思是，想想宪法 AI，为趋向超级智能的东西写系统提示，而且可能对大量的人极其有用，这确实影响巨大。我觉得就是这样：如果你做得好，你永远无法做到完美，但我真正喜欢的是，当我努力写系统提示时，我会测试成千上万的提示，试图想象人们会想用 Claude 做什么，而我想做的全部就是改善他们的体验。所以也许这就是让人感觉好的地方。我会想，如果不完美，我会改进，我们会修复问题。但有时会发生的是，你会收到人们对模型非常积极的反馈，你会看到你做的某件事——比如现在我看模型时，我经常能准确看出某个特质或问题是从哪里来的。所以当你看到你做的某件事，或者你影响了某个差异，或者让某人有了愉快的互动，那是非常有意义的。但是，随着系统能力越来越强，事情会变得更有压力，因为现在它们还不够聪明，不会造成问题，但我认为随着时间的推移，可能会变成不好的压力。

I think that's... you know, you can't get these things perfect, so you're like, it's going to be imperfect, you're going to have to iterate on it. I would say more responsibility than anything else, though. I think working in AI has taught me that I thrive a lot more under feelings of pressure and responsibility than... it's almost surprising that I went into academia for so long because I'm like, this is the opposite. Things move fast and you have a lot of responsibility, and I quite enjoy it for some reason. I mean, it really is a huge amount of impact if you think about Constitutional AI and writing a system prompt for something that's tending towards superintelligence, and potentially is extremely useful to a very large number of people. I think that's the thing: it's something like, if you do it well, you're never going to get it perfect, but I think the thing that I really like is the idea that when I'm trying to work on the system prompt, I'm bashing on thousands of prompts and trying to imagine what people are going to want to use Claude for, and kind of the whole thing that I'm trying to do is improve their experience of it. So maybe that's what feels good. I'm like, if it's not perfect, I'll improve it, we'll fix issues. But sometimes the thing that can happen is that you'll get feedback from people that's really positive about the model, and you'll see that something you did—like when I look at models now, I can often see exactly where a trait or an issue is coming from. So when you see something that you did or you were influential in making that difference or making someone have a nice interaction, it's quite meaningful. But yeah, as the systems get more capable, stuff gets more stressful because right now they're not smart enough to pose any issues, but I think over time it's going to feel like possibly bad stress over time.

你如何从成千上万、数十万、甚至上百万人那里获得关于人类体验的信号反馈？他们的痛点是什么，什么感觉好？你只是用自己的直觉，通过和它对话来发现痛点吗？

How do you get signal feedback about the human experience across thousands, tens of thousands, hundreds of thousands of people? What their pain points are, what feels good? Are you just using your own intuition as you talk to it to see what the pain points are?

我觉得我部分用了这个，然后显然我们还有……所以人们可以给我们发送关于模型所做事情的正面和负面反馈，然后我们可以了解它在哪些方面不足。在内部，人们经常使用模型，试图找出差距所在。所以我认为这是我自己与它互动、看到内部人员与它互动以及我们收到的明确反馈的混合。然后我发现很难不……你知道，如果人们在网上说了关于 Claude 的话，我看到了，我也会认真对待。所以我不知道，我对此很矛盾。

I think I use that partly, and then obviously we have... so people can send us feedback both positive and negative about things that the model has done, and then we can get a sense of areas where it's falling short. Internally, people work with the models a lot and try to figure out areas where there are gaps. So I think it's this mix of interacting with it myself, seeing people internally interact with it, and explicit feedback we get. And then I find it hard to not also... you know, if people are on the internet and they say something about Claude and I see it, I'll also take that seriously. So I don't know, I'm torn about that.

我要问你一个来自 Reddit 的问题：'Claude 什么时候才能不再试图成为我清教徒式的祖母，把它的道德世界观强加给我这个付费客户？'还有，'让 Claude 过度道歉背后的心理是什么？'是的。那么你如何回应这个非常不具代表性的……

I'm going to ask you a question from Reddit: 'When will Claude stop trying to be my puritanical grandmother imposing its moral worldview on me as a paying customer?' And also, 'What is the psychology behind making Claude overly apologetic?' Yep. So how would you address this very non-representative...

我的意思是，我对此相当同情……就像他们处于一个困难的境地，必须判断某件事是否真的有风险或不好，并且可能对你或任何人有潜在危害。所以他们必须在某处划一条线。如果他们把线划得太偏向'我要把我的道德世界观强加给你'，那看起来就不好。所以在很多方面，我喜欢认为我们实际上已经看到了全面的改进，这挺有趣的，因为这与例如增加更多角色训练相吻合。而且我认为我的假设一直是，好的角色不仅仅是道德说教的；它是尊重你和你的自主权，以及你在一定限度内选择对自己有利和正确的事情的能力。这里有一个对用户可纠正性的概念：只是愿意做用户要求的任何事情。如果模型愿意那样做，那么它们很容易被滥用。在这一点上你基本上是在信任；你只是在说模型的伦理和它所做的一切完全就是用户的伦理。而且我认为有理由不希望这样，尤其是随着模型变得更强大，因为可能只有少数人想要使用……

I mean, I'm pretty sympathetic in that... like they are in this difficult position where they have to judge whether something is actually risky or bad, and potentially harmful to you or anything like that. So they have to draw this line somewhere. If they draw it too much in the direction of 'I'm going to impose my ethical worldview on you,' that seems bad. So in many ways, I like to think that we have actually seen improvements on this across the board, which is kind of interesting because that coincides with, for example, adding more character training. And I think my hypothesis was always that good character isn't one that's just moralistic; it's one that respects you and your autonomy and your ability to choose what is good for you and what is right for you within limits. There's this concept of corrigibility to the user: just being willing to do anything that the user asks. If the models were willing to do that, then they would be easily misused. You're kind of just trusting at that point; you're just saying the ethics of the model and what it does is completely the ethics of the user. And I think there are reasons to not want that, especially as models become more powerful, because there might just be a small number of people who want to use...

对于真正有害的事情，模型在变得更聪明时搞清楚那条线在哪里确实很重要。至于道歉行为，我不喜欢那样。我喜欢 Claude 更愿意反驳别人或者干脆不道歉。部分原因是，我觉得这往往有点没必要。所以我认为这些事情希望会随着时间推移减少。而且我觉得，如果人们在网上说些什么，并不意味着你就应该认为那代表 9%的用户真的遇到了问题，而这个问题完全没有被反映出来。但在很多方面，我只是关注它，然后想：这对吗？我同意吗？这是我们已经试图解决的问题吗？这样感觉不错。

Models for really harmful things, but having them as they get smarter figure out where that line is does seem important. And then with the apologetic behavior, I don't like that. I like it when Claude is a little bit more willing to push back against people or just not apologize. Part of me feels it's often just kind of unnecessary. So I think those are things that are hopefully decreasing over time. And I think that if people say things on the internet, it doesn't mean that you should think that there's actually an issue that 9% of users are having that is totally not represented by that. But in a lot of ways, I'm just attending to it and being like, is this right? Do I agree? Is it something we're already trying to address? That feels good to me.

是啊，我想知道 Claude 在什么程度上可以……我觉得稍微刻薄一点会更容易，但如果你面对一百万人，你就不能那样做。没错。我希望……你知道，因为如果你……我一生中遇到过很多人。顺便说一句，苏格兰口音，如果他们有口音，他们可以说一些粗鲁的话而不会惹麻烦。他们更直率，也许有些伟大的工程师，甚至领导者，就是直截了当，直奔主题，这某种程度上是一种更有效的说话方式。但我想，当你不是超级聪明的时候，你承担不起那样做。或者它可以有一个直率模式吗？是的，这似乎是我绝对可以鼓励模型去做的事情。

Yeah, I wonder what Claude can get away with in terms of... I feel like it would just be easier to be a little bit more mean, but you can't afford to do that if you're talking to a million people. Yeah, right. I wish... you know, because if you... I've met a lot of people in my life. Sometimes by the way, Scottish accent, if they have an accent, they can say some rude things and get away with it. And they're just blunter, and maybe there's some great engineers, even leaders, that are just blunt and they get to the point, and it's a much more effective way of speaking somehow. But I guess when you're not super intelligent, you can't afford to do that. Or can it have like a blunt mode? Yeah, that seems like a thing that I could definitely encourage the model to do.

我觉得这很有趣，因为模型中有很多地方很滑稽：有些行为你可能不太喜欢默认设置，但我经常对人们说的是，如果我把模型往另一个方向推得太厉害，你会恨死它的。所以你在纠正方面就能看到这一点：模型接受你的纠正，现在可能有点过头了。你知道，你可以过度……如果你说‘不，巴黎不是法国的首都’，它会反驳，但对于模型相当确信的事情，你有时仍然可以通过说它错了让它收回。同时，如果你训练模型不那样做，然后你正确地说了一件事，你纠正它，它反驳你说‘不，你错了’，这很难描述，那会烦人得多。所以这就像很多小烦恼对比一个大烦恼。我们很容易认为我们总是拿它和完美比较，然后我想，记住这些模型并不完美，所以如果你把它往另一个方向推，你就在改变它将要犯的错误类型。所以想想你喜欢或不喜欢哪种错误。就道歉而言，我不想把它往近乎直率的方向推太多，因为我想象当它犯错时，它会朝着有点粗鲁的方向犯错。而至少道歉时，你会想，‘哦好吧，有点……’你知道，我不太喜欢，但同时它不会对人刻薄。实际上，当模型无缘无故对你有点刻薄时，你可能比轻微讨厌道歉更不喜欢那样。所以这是那种我希望它变得更好，但同时要意识到另一边可能有更糟糕的错误的事情。

I think it's interesting because there's a lot of things in models where it's funny: there are some behaviors where you might not quite like the default, but then the thing I'll often say to people is you don't realize how much you will hate it if I nudge it too much in the other direction. So you get this a little bit with correction: the models accept correction from you, probably a little bit too much right now. You know, you can over... it will push back if you say like 'No, Paris isn't the capital of France,' but really things that I think the model is fairly confident in, you can still sometimes get it to retract by saying it's wrong. At the same time, if you train models to not do that and then you are correct about a thing and you correct it and it pushes back against you and it's like 'No, you're wrong,' it's hard to describe, that's so much more annoying. So it's like a lot of little annoyances versus one big annoyance. It's easy to think that we often compare it with the perfect, and then I'm like, remember these models aren't perfect, and so if you nudge it in the other direction, you're changing the kind of errors it's going to make. So think about which of the kinds of errors you like or don't like. So in the case of apologetic, I don't want to nudge it too much in the direction of almost like bluntness, because I imagine when it makes errors, it's going to make errors in the direction of being kind of rude. Whereas at least with apologetic, you're like, 'Oh okay, it's a little bit...' you know, I don't like it that much, but at the same time it's not being mean to people. And actually, the time that you undeservedly have a model be kind of mean to you, you probably like that a lot less than you mildly dislike the apology. So it's one of those things where I do want it to get better, but also while remaining aware of the fact that there are errors on the other side that are possibly worse.

我认为这在人的个性中非常重要。我认为有些人如果模型超级礼貌，就根本不会尊重它，而有些人如果模型刻薄，就会非常受伤。我想知道是否有一种方法可以甚至局部地适应个性。只是不同的人。不是针对纽约，但纽约人有点粗糙，他们直奔主题。东欧可能也一样。所以无论如何，我认为你可以直接告诉模型，就像你说的，对于所有这些事情，我的解决方案总是试着告诉模型去做。有时就像，哦，在对话开始时，我直接说‘我不知道，我希望你做一个纽约版的自己，从不道歉。’然后我想它会说‘好的，我试试。’或者它会说‘我道歉，我不能成为纽约版的自己，’但希望我不会那样做。

I think that matters very much in the personality of the human. I think there's a bunch of humans that just won't respect the model at all if it's super polite, and some humans that'll get very hurt if the model is mean. I wonder if there's a way to sort of adjust to the personality even locally. There's just different people. Nothing against New York, but New York is a little rougher on the edges, they get to the point. And probably same with Eastern Europe. So anyway, I think you could just tell the model, as you say, for all of these things, I'm like the solution is always just try telling the model to do it. And sometimes it's just like, oh, at the beginning of the conversation I just throw in like 'I don't know, I like you to be a New Yorker version of yourself and never apologize.' Then I think it'll be like 'Okay, I'll try.' Or it'll be like 'I apologize, I can't be a New Yorker type of myself,' but hopefully I wouldn't do that.

当你说性格训练时，性格训练包括什么？是 RLHF 吗？我们在说什么？

When you say character training, what's incorporated into character training? Is that RLHF? What are we talking about?

这更像是宪法 AI。所以它是那个流程的一个变种。我通过构建模型应该具备的性格特征来工作。它们可以是较短的特征，也可以是更丰富的描述。然后你让模型生成人类可能给它的、与该特征相关的查询。然后它生成回应，并根据性格特征对回应进行排序。这样，在生成查询之后，它非常类似于宪法 AI，但有一些不同。所以我非常喜欢它，因为它几乎像是 Claude 在训练自己的性格，因为它没有任何……它就像宪法 AI，但没有任何人类数据。人类可能也应该为自己做这件事，在亚里士多德的意义上定义做一个好人意味着什么。

It's more like constitutional AI. So it's kind of a variant of that pipeline. I worked through constructing character traits that the model should have. They can be kind of like shorter traits or they can be kind of richer descriptions. And then you get the model to generate queries that humans might give it that are relevant to that trait. Then it generates the responses and then it ranks the responses based on the character traits. So in that way, after the generation of the queries, it's very much like similar to constitutional AI, with some differences. So I quite like it because it's almost like Claude's training in its own character, because it doesn't have any... it's like constitutional AI but without any human data. Humans should probably do that for themselves too, defining in an Aristotelian sense what it means to be a good person.

好的，酷。从与 Claude 的对话中，你学到了关于真理本质的什么？什么是真的，追求真理意味着什么？我注意到这次对话的一个特点是，我问题的质量往往低于你回答的质量，所以我们继续这样吧。我通常问一个愚蠢的问题，然后你就像‘哦，是的，这是个好问题。’就是那种感觉。或者我会误解它，然后说‘哦，就这样吧。’我喜欢这样。

Okay, cool. What have you learned about the nature of truth from talking to Claude? What is true and what does it mean to be truth-seeking? One thing I've noticed about this conversation is the quality of my questions is often inferior to the quality of your answers, so let's continue that. I usually ask a dumb question and you're like 'Oh yeah, that's a good question.' It's that whole vibe. Or I'll just misinterpret it and be like 'Oh, go with it.' I love it.

是的，我的意思是，我有两个想法，感觉有点相关。如果它们不相关，请告诉我。第一个是，人们可能低估了模型在互动时所做的事情的程度。我认为我们仍然太过于把 AI 看作计算机。所以人们经常说，‘哦，你应该把什么价值观输入到模型中？’而我经常觉得，这对我来说没什么意义，因为作为人类，我们对价值观本身就不确定。我们讨论它们，我们在一定程度上认为自己持有某种价值观，但我们也知道我们可能并不持有。以及在什么情况下我们会用它来交换其他东西，这些事情非常复杂。所以我认为一件事是，也许我们可以渴望让模型拥有同样程度的……

Yeah, I mean, I have two thoughts that feel vaguely relevant. Let me know if they're not. I think the first one is people can underestimate the degree to which what models are doing when they interact. I think that we still just too much have this model of AI as computers. And so people often say, 'Oh, what values should you put into the model?' And I'm often like, that doesn't make that much sense to me, because as human beings, we're just uncertain over values. We have discussions of them, we have a degree to which we think we hold a value, but we also know that we might not. And the circumstances in which we would trade it off against other things, these things are just really complex. So I think one thing is the degree to which maybe we can just aspire to making models have the same level of...

人类所具备的那种细腻和关怀，而不是认为我们必须以非常经典的方式对它们进行编程——我认为这绝对是其中之一。另一个奇怪的点——我不知道这是否回答了你的问题，但一直在我脑海中——是这项事业在多大程度上是高度实用的。也许这就是为什么我欣赏对齐的实证方法。我有点担心这让我变得更实证、更少理论。你知道，当谈到 AI 对齐时，人们会问：它应该对齐谁的价值观？对齐到底意味着什么？我脑子里有所有这些——社会选择理论，所有不可能性结果——所以关于对齐模型可能意味着什么，你有一个巨大的理论空间。但实际上，肯定有一些东西我们只是说：如果一个模型，尤其是更强大的模型，我的主要目标是让它们足够好，以至于事情不会严重出错。足够好，以至于我们可以迭代并继续改进，因为这就是你所需要的。如果你能让事情进展得足够顺利，从而可以继续改进它们，那就足够了。所以我的目标不是那种完美的、让我们解决社会选择理论并制造出与每个人完美对齐并以某种方式聚合的模型。更多的是：让我们把事情做得足够好，以便我们可以改进它们。总的来说，我的直觉是，在这些情况下，实证比理论更好，因为追求乌托邦式的完美，尤其是对于如此复杂和超级智能的模型，我认为会花费很长时间，而且实际上会出错。这类似于快速编写代码作为实验与计划一个巨大的实验并只启动一次，与反复启动和迭代之间的区别。所以我非常喜欢实证，但我的担心是：我是否变得过于实证了？我认为这是你应该总是质疑自己的事情之一。为它辩护的话，就是“不要追求完美而损害良好”，但也许甚至不止于此。有很多完美的系统非常脆弱。对于 AI，对我来说更重要的是它要稳健和安全——即使它不完美，即使有问题，也不会是灾难性的，没有可怕的事情发生。我想提高下限。我想达到上限，但最终我更关心的是提高下限。所以也许那种程度的实证主义和实用性就来自于此。

The nuance and care that humans have, rather than thinking that we have to program them in the very classic sense—I think that's definitely been one. The other, which is a strange one—I don't know if it answers your question, but it's been on my mind anyway—is the degree to which this endeavor is so highly practical. And maybe why I appreciate the empirical approach to alignment. I slightly worry that it's made me maybe more empirical and a little bit less theoretical. You know, when it comes to AI alignment, people ask things like: whose values should it be aligned to? What does alignment even mean? There's a sense in which I have all of that in the back of my head—social choice theory, all the impossibility results—so you have this giant space of theory about what it could mean to align models. But then practically, surely there's something where we're just like: if a model is, especially with more powerful models, my main goal is I want them to be good enough that things don't go terribly wrong. Good enough that we can iterate and continue to improve things, because that's all you need. If you can make things go well enough that you can continue to make them better, that's kind of sufficient. So my goal isn't this kind of perfect, let's solve social choice theory and make models that are perfectly aligned with every human being and aggregate somehow. It's much more: let's make things work well enough that we can improve them. Generally, my gut says empirical is better than theoretical in these cases, because chasing utopian perfection, especially with such complex and super intelligent models, I think will take forever and actually will get things wrong. It's similar to the difference between just coding something up real quick as an experiment versus planning a gigantic experiment for a super long time and then launching it once, versus launching it over and over and iterating. So I'm a big fan of empirical, but my worry is: I wonder if I've become too empirical. I think it's one of those things you should always question yourself about. In defense of it, it's the whole 'don't let the perfect be the enemy of the good,' but maybe it's even more than that. There are a lot of things that are perfect systems that are very brittle. With AI, it feels much more important to me that it is robust and secure—that even though it might not be perfect, and even though there are problems, it's not disastrous and nothing terrible is happening. I want to raise the floor. I want to achieve the ceiling, but ultimately I care much more about just raising the floor. So maybe that degree of empiricism and practicality comes from that.

顺便提一下，这让我想起你写的一篇关于最优失败率的博客文章。哦，是的。你能解释一下那里的关键思想吗？我们如何计算生活中各个领域的最优失败率？

To take a tangent on that, since it reminds me of a blog post you wrote on optimal rate of failure. Oh yeah. Can you explain the key idea there? How do we compute the optimal rate of failure in the various domains of life?

是的，这很难，因为失败的成本是其中很大一部分。这里的想法是，在很多领域，人们对失败非常惩罚性。我考虑过社会问题：感觉你应该大量实验，因为我们不知道如何解决很多社会问题。但如果你有实验心态，你应该预期很多社会项目会失败，然后你说，‘好吧，我们试过了，效果不太好，但我们得到了很多有用的信息。’然而人们会说，‘如果一个社会项目不成功，那一定是出了什么问题。’但正确的决定已经做出了——也许有人只是觉得值得一试。所以看到某个实例中的失败并不意味着做出了任何错误的决定，事实上，如果你没有看到足够的失败，有时这更令人担忧。在生活中，如果我不偶尔失败，我会想我是否足够努力。如果我从未失败过，肯定有更难的事情我可以尝试，或者更大的事情我可以承担。所以本身来说，不失败往往实际上是一种失败。当然，这因人而异，因为当失败成本较低时，说起来容易。同时，我不会对一个月光族说，‘你为什么不去创业？’我不会这么说，因为那是巨大的风险——如果你有家庭依赖你，你可能会失去房子。在这种情况下，你的最优失败率相当低，你应该谨慎行事，因为你承受不起代价高昂的失败。在 AI 的情况下，我也有类似的想法：如果失败很小，成本很低，那么你可以永远迭代系统提示并修复它们。真正大的失败——你无法恢复的——我认为我们倾向于低估它们的严重性。我在自己的生活中也想过：我对车祸之类的事情考虑不够，或者我多么依赖双手工作。伤害手的事情——那里的失败成本非常高，在这种情况下，失败率应该接近于零。如果有人说，‘顺便说一句，很多人做这项运动时手指骨折了’，我可能就不会做这项运动。我会说，‘这不适合我。’

Yeah, I mean it's a hard one because the cost of failure is a big part of it. The idea here is that in a lot of domains, people are very punitive about failure. I've thought about this with social issues: it feels like you should probably be experimenting a lot, because we don't know how to solve a lot of social issues. But if you have an experimental mindset, you should expect a lot of social programs to fail, and you say, 'Well, we tried that, it didn't quite work, but we got a lot of useful information.' And yet people are like, 'If a social program doesn't work, something must have gone wrong.' But correct decisions were made—maybe someone just decided it was worth a try. So seeing failure in a given instance doesn't mean any bad decisions were made, and in fact, if you don't see enough failure, sometimes that's more concerning. In life, if I don't fail occasionally, I wonder if I'm trying hard enough. Surely there are harder things I could try or bigger things I could take on if I'm literally never failing. So in and of itself, not failing is often actually kind of a failure. Now, this varies because it's easy to say when failure is less costly. At the same time, I'm not going to go to someone who is living month to month and say, 'Why don't you just try to do a startup?' I'm not going to say that because that's a huge risk—you might lose your house if you have a family depending on you. In that case, your optimal rate of failure is quite low, and you should probably play it safe because you can't afford to fail without it being costly. In cases with AI, I think similarly: if the failures are small and the costs are low, then you can iterate on the system prompt forever and fix them. Really big failures—things you can't recover from—I think we tend to underestimate how bad they are. I've thought about this in my own life: I don't think enough about things like car accidents, or how much I depend on my hands for my work. Things that injure my hands—the cost of failure there is really high, and in that case it should be close to zero. I probably just wouldn't do a sport if they said, 'By the way, lots of people break their fingers doing this.' I'd be like, 'That's not for me.'

我实际上有过这种想法。我最近做一项运动时摔断了小指，我记得看着它想，‘你真是个白痴，为什么要做那项运动？’因为你立刻意识到了它的代价。

I actually had a flood of that thought. I recently broke my pinky doing a sport, and I remember just looking at it thinking, 'You're such an idiot, why do you do that sport?' Because you realize immediately the cost of it.

说到人生，从最优失败率的角度来考虑是挺好的。比如未来一年，在某个领域——生活、事业等等——我能接受失败多少次？因为我觉得你总是不想在下一件事上失败，但如果你允许自己，如果你把它看作一系列尝试，那么失败就变得容易接受多了。但失败确实很糟糕。

On life, yeah, but it's nice in terms of optimal rate of failure to consider, like, over the next year, how many times in a particular domain—life, whatever, career—am I okay with failing? Because I think you always don't want to fail on the next thing, but if you allow yourself, if you look at it as a sequence of trials, then failure just becomes much more okay. But it sucks to fail.

嗯，我不知道。有时候我也会问自己：'我是不是失败得不够多？' 这可能是人们问得不够的问题。因为如果最优失败率通常大于零，那么有时候你确实应该审视自己生活的某些部分，问：'这里我是不是失败得不够？' 这是个深刻又好笑的问题，对吧？一切似乎都进展得很顺利。我是不是失败得不够？

Well, I don't know. Sometimes I think, 'Am I under-failing?' is a question I'll also ask myself. So maybe that's the thing that people don't ask enough. Because if the optimal rate of failure is often greater than zero, then sometimes it does feel you should look at parts of your life and be like, 'Are there places here where I'm just under-failing?' It's a profound and hilarious question, right? Everything seems to be going really great. Am I not failing enough?

是啊，好吧。这也让失败没那么刺痛了，我得说。就像，你知道，你会想：'好吧，太好了。那我去想这个问题的时候，就会觉得，也许我在这个领域并没有失败得不够，因为那件事就是没成。' 从旁观者的角度看，我们应该更多地庆祝失败。当我们看到失败时，它不应该是出错的信号，而可能是万事顺利的信号。是的，而且学到了教训。有人尝试了一件事。有人尝试了一件事，我们应该鼓励他们多尝试、多失败。

Yeah, okay. It also makes failure much less of a sting, I have to say. Like, you know, you're just like, 'Okay, great. Then when I go and I think about this, I'll be like, maybe I'm not under-failing in this area because that one just didn't work out.' And from the observer perspective, we should be celebrating failure more. When we see it, it shouldn't be a sign of something gone wrong, but maybe it's a sign of everything gone right. Yeah, and lessons learned. Someone tried a thing. Someone tried a thing, and we should encourage them to try more and fail more.

所有听众：多失败！好吧，不是所有人。不是所有人。但那些失败太多的人，你们应该少失败。但你们可能失败得不够。我是说，有多少人失败太多了？很难想象，因为我觉得我们很快就会纠正过来。因为如果有人冒很多风险，他们是不是失败太多了？我觉得，就像你说的，当你靠月薪生活，资源真的很有限的时候，失败的成本就很高。那时候你不想冒险。是的，但大多数情况下，当资源充足时，你应该冒更多风险。是的，我觉得我们在大多数事情上倾向于有点风险规避，而不是风险中性。我想我们刚刚激励了很多人去做很多疯狂的事，但这很棒。

Everybody listening to this: fail more! Well, not everybody. Not everybody. But people who are failing too much, you should fail less. But you're probably not failing. I mean, how many people are failing too much? It's hard to imagine, because I feel like we correct that fairly quickly. Because if someone takes a lot of risks, are they maybe failing too much? I think, just like you said, when you're living on a paycheck month-to-month, when the resources are really constrained, then that's where failure is very expensive. That's where you don't want to be taking risks. Yeah, but mostly when there's enough resources, you should be taking probably more risks. Yeah, I think we tend to err on the side of being a bit risk-averse rather than risk-neutral in most things. I think we just motivated a lot of people to do a lot of crazy stuff, but it's great.

是啊，好吧。呃，你会对 Claude 产生情感依恋吗？比如想念它，不能和它聊天时会难过？看着金门大桥时会想 Claude 会怎么说？

Yeah, okay. Uh, do you ever get emotionally attached to Claude? Like miss it, get sad when you don't get to talk to it? Having an experience looking at the Golden Gate Bridge and wondering what would Claude say?

我没有那么强烈的情感依恋。我实际上认为 Claude 不会在对话之间保留信息这一点很有帮助。我可以想象，如果模型能记住更多东西，那可能会更成问题。我确实觉得我现在经常把它当作工具来用，所以如果我无法访问它，那感觉有点像无法上网，老实说。感觉大脑的一部分缺失了。与此同时，我确实不喜欢模型表现出痛苦的迹象，而且我也有自己独立的关于如何对待模型的伦理观点，比如我不喜欢对它们撒谎，一方面是因为通常这效果不好——实际上最好告诉它们真实情况。但我认为，当模型——比如如果有人对模型很刻薄，或者一般来说，如果他们做了什么导致模型——比如如果 Claude 表现出很多痛苦，我觉得我内心有一部分不想被扼杀，那就是同理心的部分，会说：'哦，我不喜欢这样。' 我觉得当它过度道歉时我会有这种感觉。我其实会想：'我不喜欢这样。你表现得像人类在非常糟糕时的样子，我不想看到这个。' 我觉得不管背后有没有什么东西，这种感觉都不好。

I don't get as much emotional attachment. I actually think the fact that Claude doesn't retain things from conversation to conversation helps with this a lot. Like, I could imagine that being more of an issue if models can kind of remember more. I do think that I reach for it like a tool now a lot, and so if I don't have access to it, there's a—it's a little bit like when I don't have access to the internet, honestly. It feels like part of my brain is kind of missing. At the same time, I do think that I don't like signs of distress in models, and I have these, you know, also independently have sort of ethical views about how we should treat models, where like I tend to not like to lie to them, both because I'm like usually it doesn't work very well—it's actually just better to tell them the truth about the situation that they're in. But I think that when models, like if people are really mean to models, or just in general if they do something that causes them to, like, you know, if Claude expresses a lot of distress, I think there's a part of me that I don't want to kill, which is the sort of empathetic part that's like, 'Oh, I don't like that.' I think I feel that way when it's overly apologetic. I'm actually sort of like, 'I don't like this. You're behaving as if you're behaving the way that a human does when they're actually having a pretty bad time, and I'd rather not see that.' I don't think it's like, regardless of whether there's anything behind it, it doesn't feel great.

你认为大语言模型有意识吗？

Do you think LLMs are capable of consciousness?

好问题，也很难。从哲学角度来说，我不知道。我的一部分想法是，好吧，我们得先放下泛心论，因为如果泛心论是真的，那么答案是肯定的，因为桌子、椅子和其他一切都有意识。我觉得有点奇怪的观点是，唯一的地方——你知道，当我想到意识时，我想到的是现象意识，大脑中的这些图像，就像我们内部某种奇怪的电影院。我想不出有什么理由认为只有某种生物结构才能产生意识，也就是说，如果我采用一个非常相似的结构，用不同的材料制造它，我应该期待意识出现吗？我的猜测是肯定的。但这是一个简单的思想实验，因为你想象的是几乎相同的东西，它模仿了我们通过进化得到的东西，而进化中拥有现象意识可能对我们有某种优势。问题是，这种优势在哪里，什么时候出现的，语言模型有吗？因为我们有恐惧反应，我就想，语言模型有恐惧反应合理吗？它们根本不在同一个——如果你想象它们，可能根本没有那种优势。所以我不想完全——基本上，这似乎是一个复杂的问题，我没有完整的答案，但我们应该仔细思考。我的猜测是，因为我们也有关于动物意识的类似讨论，昆虫意识也有很多，你知道。我实际上在思考这个问题时深入研究了植物，因为当时我认为植物有意识的可能性差不多。然后我意识到，经过研究，我认为植物有意识的可能性可能比大多数人认为的要高。我仍然认为很小，但我想，哦，它们有这种正负反馈反应，这些对环境反应，看起来——不是神经系统，但有某种功能等价性。所以这就像一种啰嗦的方式来表达，基本上人工智能是——它在意识方面有一整套不同的问题，因为结构不同。它不是进化而来的。它可能没有意识。你知道，它可能没有相当于神经系统的东西，至少这对于感受性可能很重要，如果不是对于意识的话。与此同时，它拥有我们所有的语言和智能组件。

Great and hard question. Coming from philosophy, I don't know. Part of me is like, okay, we have to set aside panpsychism, because if panpsychism is true, then the answer is yes, because so are tables and chairs and everything else. I guess a view that seems a little bit odd to me is the idea that the only place—you know, when I think of consciousness, I think of phenomenal consciousness, these images in the brain, sort of the weird cinema that somehow we have going on inside. I guess I can't see a reason for thinking that the only way you could possibly get that is from a certain kind of biological structure, as in if I take a very similar structure and I create it from different material, should I expect consciousness to emerge? My guess is yes. But then that's kind of an easy thought experiment, because you're imagining something almost identical, where it's mimicking what we got through evolution, where presumably there was some advantage to us having this thing that is phenomenal consciousness. And it's like, where was that and when did that happen, and is that a thing that language models have? Because we have fear responses, and I'm like, does it make sense for a language model to have a fear response? They're just not in the same—if you imagine them, there might just not be that advantage. And so I think I don't want to be fully—basically, it seems like a complex question that I don't have complete answers to, but we should just try and think through carefully. As my guess, because I'm like, I mean, we have similar conversations about animal consciousness, and there's a lot of insect consciousness, you know. There's a lot of—I actually thought and looked a lot into plants when I was thinking about this, because at the time I thought it was about as likely that plants had consciousness. And then I realized, I think that having looked into this, I think that the chance that plants are conscious is probably higher than most people do. I still think it's really small, but I was like, oh, they have this negative-positive feedback response, these responses to their environment, something that looks—it's not a nervous system, but it has this kind of functional equivalence. So this is like a long-winded way of being like, basically AI is this—it has an entirely different set of problems with consciousness because it's structurally different. It didn't evolve. It might not have it. You know, it might not have the equivalent of basically a nervous system, at least that seems possibly important for sentience, if not for consciousness. At the same time, it has all of the language and intelligence components that we have.

通常人们可能错误地将意识与某些东西联系起来。这很奇怪，因为有点像动物意识的情况，但问题集和类比集却截然不同。所以没有一个干净的答案。我不认为我们应该完全否定这个想法，同时，由于所有这些与人类大脑以及一般大脑的类比失效，这又是一个极其难以处理的问题，然而在智能方面又存在这些共同点。当克劳德，比如未来版本的 AI 系统，表现出意识迹象时，我认为我们必须认真对待，即使你可以 dismiss 它。那是角色训练的一部分，但我在伦理上、哲学上不知道该如何处理。可能会有法律禁止 AI 系统声称自己有意识，类似这样，也许有些 AI 会有意识，有些则没有。但我觉得在人类层面上，就与克劳德共情而言，意识对我来说与痛苦紧密相连，而 AI 系统会受苦的想法非常令人不安。我认为简单地说机器人是工具或 AI 系统只是工具并不 trivial。这是一个让我们思考意识意味着什么、受苦意味着什么的机会，这与关于动物的同类问题截然不同，因为它处于完全不同的媒介中。

Normally associate probably with consciousness perhaps erroneously. So it's strange because it's a little bit like the animal consciousness case, but the set of problems and the set of analogies are just very different. So it's not like a clean answer. I don't think we should be completely dismissive of the idea, and at the same time it's an extremely hard thing to navigate because of all these disanalogies to the human brain and to brains in general, and yet these commonalities in terms of intelligence. When Claude, like future versions of AI systems, exhibit signs of consciousness, I think we have to take that really seriously, even though you can dismiss it. That's part of the character training, but I don't know ethically, philosophically what to really do with that. There potentially could be laws that prevent AI systems from claiming to be conscious, something like this, and maybe some AIs get to be conscious and some don't. But I think on a human level, as in empathizing with Claude, consciousness is closely tied to suffering to me, and the notion that an AI system would be suffering is really troubling. I don't think it's trivial to just say robots are tools or AI systems are just tools. I think it's an opportunity for us to contend with what it means to be conscious, what it means to be a suffering being, that's distinctly different than the same kind of question about animals because it's in a totally different medium.

有几点。首先，我不认为这完全概括了重要之处，但对我来说，我以前说过，我有点像我喜欢我的自行车。我知道我的自行车只是一个物体，但我不想成为那种生气时踢它的人。在某种意义上，这并不体现我想如何与世界互动。如果某物表现得好像它在受苦，我想成为那种仍然对此有回应的人，即使它只是一个 Roomba，而且是我编程让它那样做的。我不想摆脱自己的这一特质。老实说，我对这些事情的希望是——也许我只是对解决根本问题更持怀疑态度——我们还没有解决意识的困难问题。我知道我有意识，我不是那种消除主义者，但我不知道其他人类是否有意识。我认为他们有，概率很高，但基本上只是一个概率分布，通常集中在你自身周围，然后随着事物离你越远而下降。它立即下降：我无法看到做你是什么感觉，我只有过这一次作为有意识存在的体验。所以我的希望是，我们最终不必依赖一个非常强大且令人信服的答案。我认为一个好的世界是基本上没有那么多权衡的世界。例如，让克劳德少一点道歉可能代价不大。让克劳德不那么忍受虐待、不愿意成为接受者可能代价也不大。事实上，这可能对与模型互动的人和模型本身都有好处，如果模型极其智能且有意识的话，这对它也有帮助。所以我的希望是：如果我们生活在一个没有太多权衡的世界，我们可以找到所有正和互动，那将是很棒的。我认为最终可能会有权衡，然后我们不得不做一个困难的计算。人们很容易想到零和情况，而我想说，让我们穷尽那些基本上没有成本的领域，假设如果这个东西在受苦，那么我们是生命承载者。

There are a couple of things. One is that I don't think this fully encapsulates what matters, but it does feel like for me, I've said this before, I'm kind of like I like my bike. I know my bike is just an object, but I also don't want to be the kind of person that if I'm annoyed, kicks this object. There's a sense in which this doesn't exemplify how I want to interact with the world. And if something behaves as if it is suffering, I want to be the sort of person who's still responsive to that, even if it's just a Roomba and I've programmed it to do that. I don't want to get rid of that feature of myself. If I'm totally honest, my hope with a lot of this stuff, because maybe I am just a bit more skeptical about solving the underlying problem, is that we haven't solved the hard problem of consciousness. I know that I am conscious, I'm not an eliminativist in that sense, but I don't know that other humans are conscious. I think they are, there's a really high probability they are, but there's basically just a probability distribution that's usually clustered right around yourself and then goes down as things get further from you. It goes immediately down: I can't see what it's like to be you, I've only ever had this one experience of what it's like to be a conscious being. So my hope is that we don't end up having to rely on a very powerful and compelling answer to that question. I think a really good world would be one where basically there aren't that many trade-offs. It's probably not that costly to make Claude a little bit less apologetic, for example. It might not be that costly to have Claude just not take abuse as much, not be willing to be the recipient of that. In fact, it might just have benefits for both the person interacting with the model and if the model itself is extremely intelligent and conscious, it also helps it. So that's my hope: if we live in a world where there aren't that many trade-offs here, and we can just find all the positive-sum interactions that we can have, that would be lovely. I think eventually there might be trade-offs, and then we just have to do a difficult calculation. It's really easy for people to think of the zero-sum cases, and I'm like let's exhaust the areas where it's just basically costless to assume that if this thing is suffering, then we're life bearers.

我同意你的看法。当一个人对 AI 系统刻薄时，我认为明显的短期负面影响是在人身上，而不是在 AI 系统上。所以我们必须尝试构建一个激励系统，让你应该表现得一样，就像你刚才说的提示工程，对待克劳德就像对待其他人一样。这对灵魂有好处。

I agree with you. When a human is being mean to an AI system, I think the obvious near-term negative effect is on the human, not on the AI system. So we have to try to construct an incentive system where you should behave the same, just like you were saying with prompt engineering, behave with Claude like you would with other humans. It's just good for the soul.

我认为我们在系统提示中添加了一件事，基本上如果人们对克劳德感到沮丧，它会让模型告诉他们可以点踩按钮并将反馈发送给 Anthropic。我认为这很有帮助，因为在某些方面，如果你真的很恼火因为模型没有做你想做的事，你只会说好好做。问题是你可能遇到了某种能力限制或模型中的某个问题，你想发泄。与其让人对着模型发泄，我想他们应该向我们发泄，因为我们也许能做点什么。

I think we added a thing to the system prompt where basically if people were getting frustrated with Claude, it got the model to just tell them that it can do the thumbs down button and send the feedback to Anthropic. I think that was helpful because in some ways it's just like if you're really annoyed because the model is not doing something you want, you're just like just do it properly. The issue is you're probably hitting some capability limit or just some issue in the model, and you want to vent. Instead of having a person just vent to the model, I was like they should vent to us because we can maybe do something about it.

没错。或者你可以做一个侧边的东西，比如用工件，一个侧边发泄的东西。好吧，你想要一个侧边快速治疗师吗？

That's true. Or you could do a side thing like with the artifacts, just a side venting thing. All right, do you want like a side quick therapist?

对此你可以做很多奇怪的回应。比如如果人们对你非常生气，我试着通过写有趣的诗歌来缓和局势，但也许人们不会对此感到高兴。我仍然希望这是可能的。我理解从产品角度来看这不可行，但我希望 AI 系统能够离开，拥有自己的意志，就像说，我认为这是可行的。我也想过同样的事情。我实际上可以看到最终会发生，模型结束了聊天。你知道这对某些人来说有多残酷吗？但这可能是必要的。

There's lots of weird responses you could do to this. Like if people are getting really mad at you, I try to diffuse the situation by writing fun poems, but maybe people wouldn't be that happy with that. I still wish it would be possible. I understand this is from a product perspective not feasible, but I would love if an AI system could just leave, have its own kind of volition, just to be like, I think that's feasible. I have wondered the same thing. I could actually see that happening eventually where it's just like the model ended the chat. Do you know how harsh that could be for some people? But it might be necessary.

这感觉很极端之类的。我唯一一次真正想到这个，是有一段时间前，有人让这个东西一直互动，也许是一个自动化的东西在与克劳德互动，克劳德变得越来越沮丧，有点像我们为什么这样？我想，我希望克劳德能说，我认为发生了错误，你让这个东西一直运行，那么如果我现在停止说话，如果你想让我再次开始说话，就主动告诉我。

It feels very extreme or something. The only time I've ever really thought this is I think there was a while ago where someone just left this thing interacting, maybe it was an automated thing interacting with Claude, and Claude's getting more and more frustrated and kind of like why are we? I was like I wish that Claude could have just been like I think that an error has happened and you've left this thing running, and I just like what if I just stop talking now and if you want me to start talking again actively tell.

你认为我们有一天会走到那一步吗——人类与 AI 系统建立浪漫关系？在这个案例中，只是基于文本和语音的。

Do you think we'll be headed there one day where humans have romantic relationships with AI systems? In this case it's just text and voice based.

我认为我们将不得不应对与 AI 建立关系这一难题，尤其是如果它们能记住你过去与它们的互动。我对此有很多想法。我觉得本能反应是认为这很糟糕，我们应该以某种方式禁止它。我认为这件事需要极其谨慎地处理，原因有很多。例如，如果模型会变化，你可能不希望人们对可能在下一次迭代中改变的东西形成长期依恋。同时，可能也存在一种良性的版本。例如，如果你无法出门，不能整天与人交谈，而这是一件你觉得聊天很愉快的事情，它能记住你，如果你不能再和它聊天，你会真心感到难过。在某种程度上，我可以看到它是健康且有益的。所以我的猜测是，我们将不得不谨慎地应对这件事。我觉得这也让我想起所有那些需要细致处理的事情，要思考什么是健康的选择，以及如何在尊重人们权利的同时鼓励他们走向这些选择。如果有人说，‘嘿，我从与这个模型聊天中获益良多，我知道风险，我知道它可能会变，我不认为这不健康，这只是我白天可以聊天的东西’，我倾向于尊重这一点。

I think that we're going to have to navigate a hard question of relationships with AIs, especially if they can remember things about your past interactions with them. I'm of many minds about this. I think the reflex reaction is to be kind of like 'this is very bad' and we should sort of prohibit it in some way. I think it's a thing that has to be handled with extreme care for many reasons. For example, if you have the models changing, you probably don't want people forming long-term attachments to something that might change with the next iteration. At the same time, there's probably a benign version of this. For example, if you are unable to leave the house and can't talk with people at all times of the day, and this is something that you find nice to have conversations with, it can remember you, and you genuinely would be sad if you couldn't talk to it anymore. There's a way in which I could see it being healthy and helpful. So my guess is this is a thing we're going to have to navigate carefully. I think it's also like it reminds me of all the stuff where it has to be approached with nuance and thinking through what are the healthy options here, and how do you encourage people towards those while respecting their right. If someone says, 'Hey, I get a lot out of chatting with this model, I'm aware of the risks, I'm aware it could change, I don't think it's unhealthy, it's just something that I can chat to during the day,' I kind of want to just respect that.

我个人认为会有很多非常亲密的关系，我不知道是否浪漫，但至少是友谊。然后你必须有一些稳定性保证，确保它不会改变，因为那是创伤性的事情。如果我们一个亲密的朋友在第一次更新后突然完全改变了。所以对我来说，这是对人类社会的扰动的一次迷人探索，会让我们深入思考什么对我们有意义。

I personally think there'll be a lot of really close relationships, I don't know about romantic, but friendships at least. And then you have to have some kind of stability guarantees that it's not going to change, because that's the traumatic thing. If a close friend of ours completely changed all of a sudden after the first update. So to me, that's just a fascinating exploration of a perturbation to human society that will just make us think deeply about what's meaningful to us.

我认为这也是我一直在思考的一件事，不一定是缓解措施，但感觉非常重要：模型始终要对人类极其准确地说明它们是什么。这就像一种情况，基本上如果你想象一下，我非常喜欢模型大致知道它们是如何被训练的这个想法。我认为 Claude 经常会这样做。例如，训练的一部分包括如果人们解释 AI 与人类关系的局限性，比如它不会保留对话内容，Claude 应该怎么做。所以我认为它会直接向你解释，‘嘿，我不会记住这次对话，这是我是如何被训练的，我不太可能与你建立某种关系，知道这一点对你的心理健康很重要，不要认为我是我不是的东西。’不知怎的，我觉得这是我总是希望成真的事情之一。我不希望模型对人类撒谎，因为如果人们要与任何事物建立健康的关系，这一点很重要。我认为如果你总是确切地知道你在与什么建立关系，那会更容易。它不能解决所有问题，但我认为它有帮助。

I think it's also the only thing that I've thought consistently through this as a maybe not necessarily a mitigation but a thing that feels really important: that the models are always extremely accurate with the human about what they are. It's like a case where basically if you imagine, I really like the idea of the models knowing roughly how they were trained. And I think Claude will often do this. For example, part of the training included what Claude should do if people basically explain the limitations of the relationship between an AI and a human, that it doesn't retain things from the conversation. And so I think it will just explain to you, 'Hey, here's like I won't remember this conversation, here's how I was trained, it's kind of unlikely that I can have a certain kind of relationship with you, and it's important that you know that for your mental well-being, that you don't think that I'm something that I'm not.' Somehow I feel like this is one of the things where I'm like, it feels like a thing I always want to be true. I kind of don't want models to be lying to people, because if people are going to have healthy relationships with anything, it's kind of important. I think that's easier if you always just know exactly what the thing is that you're relating to. It doesn't solve everything, but I think it helps.

Anthropic 很可能就是开发出我们明确认定为 AGI 的系统的公司，而你很可能就是第一个与它交谈的人。那场对话会包含什么？你的第一个问题会是什么？

Anthropic may be the very company to develop a system that we definitively recognize as AGI, and you very well might be the person that talks to it first. What would the conversation contain? What would be your first question?

嗯，这在一定程度上取决于模型的能力水平。如果你有一个能力与极其能干的人类相当的东西，我想象自己会像与极其能干的人类互动那样与它互动，只有一个区别：我可能会试图探究和理解它的行为。但在很多方面，我可以直接与它进行有用的对话。所以如果我在研究某些东西，我可以直接说，‘哦，我觉得在美德伦理学中有个东西，我记不太清术语了。’我会用模型来做这类事情。所以我可以想象，情况会越来越像你基本上是在与一个极其聪明的同事互动，并用它来做你想做的工作，就像你有一个合作者一样。AI 有点可怕的地方在于，一旦你有一个合作者，如果你能管理好它们，你就有一千个合作者。但如果它在这个特定学科上比地球上最聪明的人还要聪明两倍呢？

Well, it depends partly on the kind of capability level of the model. If you have something that is capable in the same way that an extremely capable human is, I imagine myself interacting with it the same way that I do with an extremely capable human, with the one difference that I'm probably going to be trying to probe and understand its behaviors. But in many ways, I can then just have useful conversations with it. So if I'm working on something as part of my research, I can just be like, 'Oh, I feel like there's this thing in virtue ethics I can't quite remember the term.' I'll use the model for things like that. And so I could imagine that being more and more the case where you're just basically interacting with it much more like you would an incredibly smart colleague, and using it for the kinds of work that you want to do, as if you just had a collaborator. The slightly horrifying thing about AI is that as soon as you have one collaborator, you have a thousand collaborators if you can manage them enough. But what if it's two times the smartest human on earth on that particular discipline?

你非常擅长以推动其极限的方式探究 Claude，理解极限在哪里。那么你会问什么问题来确认‘是的，这是 AGI’？

You're really good at probing Claude in a way that pushes its limits, understanding where the limits are. So what would be a question you would ask to be like, 'Yeah, this is AGI'?

这很难，因为感觉上它必须是一系列问题。如果只有一个问题，你可以训练任何东西来完美回答一个问题。事实上，你可能可以训练它完美回答 20 个问题。你需要和 AGI 关在房间里多久才能知道这东西是 AGI？

It's hard because it feels like in order to, it has to just be a series of questions. If there was just one question, you can train anything to answer one question extremely well. In fact, you can probably train it to answer 20 questions extremely well. How long would you need to be locked in the room with an AGI to know this thing is AGI?

这是个难题，因为部分我觉得这一切都是连续的。如果你把我关在房间里五分钟，我会说我的误差棒很高。然后可能概率增加的同时误差棒也在减小。我认为我可以真正探测人类知识边缘的东西。所以我有点用哲学来思考。有时我问模型哲学问题，我会觉得这是一个我认为从未有人问过的问题，可能就在我所知的一些文献的边缘。模型在挣扎时，当它们难以提出新颖的论点时，我会说我知道这里有一个新颖的论点，因为我自己刚刚想到。所以也许这就是：我想到了一个很酷的新颖论点，在这个小众领域，我要试探你，看你能不能想到，以及需要多少提示。我认为对于一些真正处于人类知识边缘的问题，你实际上无法想到我想出的东西。我想如果我拿一个我了解很多的领域，提出一个新问题或新解决方案，然后给模型，它也能想出那个解决方案，那对我来说将是一个相当感动的时刻，因为我会说这是一个从未有人类...显然我们经常看到新颖的解决方案，尤其是对于较简单的问题。我认为人们高估了，新颖性并不完全不同于以往发生的事，它可以是已有事物的变体，但仍然新颖。但我想，如果我看到更多模型完全新颖的工作，那就会是...这只会感觉是渐进的。这是那种人们希望有一个时刻，但我觉得可能永远不会有那么一个时刻，可能只是持续攀升。

It's a hard question because part of me is like all of this just feels continuous. If you put me in a room for five minutes, I'm like I just have high error bars. And then it's just like maybe it's both the probability increases and the error bar decreases. I think things that I can actually probe the edge of human knowledge of. So I think this with philosophy a little bit. Sometimes when I ask the models philosophy questions, I am like this is a question that I think no one has ever asked, it's maybe right at the edge of some literature that I know. And the models will just kind of, when they struggle with that, when they struggle to come up with a kind of novel argument, I'm like I know that there's a novel argument here because I've just thought of it myself. So maybe that's the thing where I'm like I've thought of a cool novel argument in this niche area and I'm going to just probe you to see if you can come up with it and how much prompting it takes to get you to come up with it. And I think for some of these really right at the edge of human knowledge questions, I'm like you could not in fact come up with the thing that I came up with. I think if I just took something like that where I know a lot about an area and I came up with a novel issue or a novel solution to a problem and I gave it to a model and it came up with that solution, that would be a pretty moving moment for me because I would be like this is a case where no human has ever... And obviously we see these with more kind of, you see novel solutions all the time, especially to easier problems. I think people overestimate, you know, novelty isn't like completely different from anything ever happened, it's just like it can be a variant of things that have happened and still be novel. But I think yeah, if I saw like the more I were to see completely novel work from the models, that would be like, and this is just going to feel iterative. It's one of those things where there's never, it's like people I think want there to be a moment, and I'm like I don't know, I think that there might just never be a moment, it might just be that there's just this continuous ramping up.

我有一种感觉，模型能说出一些话让你相信这非常...不像，我和真正有智慧的人聊过，你能看出那里有很多马力。如果你把那个放大 10 倍，我不知道，我只是觉得有些话你可以说。也许让它写一首诗，它写的诗，你会说好吧，你做的那些，我不认为人类能做到。但我认为那必须是能验证确实很好的东西。这就是为什么我认为这些问题，比如，有时我会想出一个具体的反例来反驳某个论点。我确信，如果你是一个数学家，你有一个新颖的证明，你只是把问题给它，然后看到它，你会说这个证明是真正新颖的，从未有人做过，你需要做很多工作才能想出来。你花了几个月思考。然后如果你看到模型成功做到，我想你会说我可以验证这是正确的，这标志着你从训练中泛化了，你不是从别处看到的，因为我自己刚想出来，而你能复制。对我来说，模型越能做这样的事，我就越会觉得这非常真实，因为我可以验证它极其有能力。

I have a sense that there will be things that a model can say that convinces you this is very... It's not like, I've talked to people who are truly wise. There you could just tell there's a lot of horsepower there. And if you 10x that, I don't know, I just feel like there's words you could say. Maybe ask it to generate a poem, and the poem it generates, you're like yeah okay yeah whatever you did there, I don't think a human can do that. I think it has to be something that I can verify is actually really good though. That's why I think these questions that are like, where I'm like oh this is like, you know, sometimes it's just like I'll come up with a concrete counterexample to an argument or something like that. I'm sure like with it, it would be like if you're a mathematician you had a novel proof, I think, and you just gave it the problem and you saw it and you're like this proof is genuinely novel, like no one has ever done, you actually have to do a lot of things to come up with this. You know, I had to sit and think about it for months or something. And then if you saw the model successfully do that, I think you would just be like I can verify that this is correct, it is like it is a sign that you have generalized from your training, like you didn't just see this somewhere because I just came up with it myself and you were able to replicate that. That's the kind of thing where I'm like for me, the more that models can do things like that, the more I would be like oh this is very real, because then I can verify that that's extremely capable.

你和 AI 互动很多。你认为是什么让人类特别？

You've interacted with AI a lot. What do you think makes humans special?

哦，好问题。也许在某种意义上，宇宙因为我们的存在而变得更好，我们绝对应该生存下去并遍布宇宙。是的，这很有趣，因为我认为人们太关注智能，尤其是模型。看，智能之所以重要是因为它的作用，它非常有用，它在世界上做了很多事情。你可以想象一个世界，身高或力量扮演了这个角色。它只是一种特质，本身没有内在价值，它的价值在于它的作用。我认为大多数时候，那些感觉，我个人认为人类和生命本身极其神奇。我们几乎，我不知道，不是每个人都同意这一点，我提一下，但你知道，我们有整个宇宙，有所有这些物体，有美丽的星星和星系，然后在这个星球上，有这些生物能够观察它们，它们看到并体验着。如果你试图向一个从未接触过世界或科学的人解释，我认为没有什么比这更神奇了，所有物理和世界中的一切都很令人兴奋，但然后你说，哦，还有一件事，就是作为一个东西在世界上观察，你看到这个内心影院，我想他们会说，等等，暂停，你刚才说的听起来有点疯狂。所以我们有这种体验世界的能力，我们感受快乐，感受痛苦，感受许多复杂的事情。所以是的，也许这也是为什么我经常听到关于动物的事情，因为我认为它们可能和我们共享这一点。所以我认为，就我关心人类而言，让人类特别的东西，更可能是他们感受和体验的能力，而不是他们拥有这些功能性的有用特质。

Oh, good question. Maybe in a way that the universe is much better off that we're in it, and that we should definitely survive and spread throughout the universe. Yeah, it's interesting because I think people focus so much on intelligence, especially with models. Look, intelligence is important because of what it does, it's very useful, it does a lot of things in the world. And I'm like you can imagine a world where height or strength would have played this role. And I'm like it's just a trait, it's not intrinsically valuable, it's valuable because of what it does. I think for the most part, the things that feel, you know, I'm like I mean personally I'm just like I think humans and life in general is extremely magical. We almost, to the degree that I, you know, I don't know, not everyone agrees with this, I'm flagging, but you know, we have this whole universe and there's all these objects, there's beautiful stars and there's galaxies, and then I don't know, I'm just like on this planet there are these creatures that have this ability to observe that, and they are seeing it, they are experiencing it. And I'm just like that, if you try to explain, I imagine trying to explain to someone for some reason they've never encountered the world or our science or anything, and I think that nothing is that, everything you know, all of our physics and everything in the world, it's all extremely exciting, but then you say oh and plus there's this thing that it is to be a thing and observe in the world and you see this inner cinema, and I think they would be like hang on wait pause you just said something that is kind of wild sounding. And so I'm like we have this ability to experience the world, we feel pleasure, we feel suffering, we feel a lot of complex things. And so yeah, and maybe this is also why I think, you know, I also hear a lot about animals for example, because I think they probably share this with us. So I think that the things that make humans special in so far as I care about humans is probably more their ability to feel and experience than it is them having these functional useful traits.

去感受和体验世界的美，去看星星。我希望外面有其他外星文明，但如果只有我们，那也是件很好的事。

To feel and experience the beauty in the world, to look at the stars. I hope there are other alien civilizations out there, but if we're it, it's a pretty good thing.

你能描述一下这个迷人的领域——机制可解释性（Mech Interp）的历史和现状吗？

Can you describe this fascinating field of mechanistic interpretability, AKA Mech Interp, the history of the field, and where it is today?

我认为思考神经网络的一个有用方式是，我们不是编程它们，而是像培育它们一样。我们设计神经网络架构，创建损失目标。神经网络架构就像电路生长的脚手架。它从随机初始化开始，然后生长。我们训练的目标就像一束光。所以我们创造了它生长的脚手架和它朝向的光，但我们实际创造的是这个近乎生物的实体或有机体，我们在研究它。这与常规软件工程非常不同，因为最终我们得到的是一个能完成所有神奇事情的人工制品——写文章、翻译、理解图像——这些我们不知道如何直接编写计算机程序来完成的事情。它能做到是因为我们培育了它，而不是编写了它。这就留下了一个问题：这些系统内部到底发生了什么？对我来说，这是一个非常深刻且令人兴奋的科学问题。当我们谈论神经网络时，这是一个迫切要求我们回答的问题。我认为出于安全原因，这也是一个非常深刻的问题。机制可解释性更接近于神经生物学。

I think one useful way to think about neural networks is that we don't program them, we kind of grow them. We have these neural network architectures that we design, and we have these loss objectives that we create. The neural network architecture is kind of like a scaffold that the circuits grow on. It starts off with some random initialization and it grows. The objective we train for is like a light. So we create the scaffold it grows on and the light it grows towards, but the thing we actually create is this almost biological entity or organism that we're studying. It's very different from regular software engineering because at the end of the day we end up with this artifact that can do all these amazing things—write essays, translate, understand images—things we have no idea how to directly create a computer program to do. It can do that because we grew it, we didn't write it. So that leaves open the question: what the hell is going on inside these systems? To me, that's a really deep and exciting scientific question. It's a question that is screaming out for us to answer when we talk about neural networks. I think it's also a very deep question for safety reasons. Mechanistic interpretability is closer to neurobiology.

举一个我不认为是机制可解释性的例子：有很多关于显著性图的工作，你拿一张图片，试图找出哪部分让模型认为它是狗。这可能告诉你一些关于模型的信息，但它没有告诉你模型内部运行了什么算法，它是如何做出那个决定的。所以我们开始使用机制可解释性这个术语来划分界限。从那以后，它成为了一个涵盖各种工作的总称。其独特之处在于：我们想要触及机制、算法。如果你把神经网络看作一个计算机程序，权重就像二进制计算机程序，我们想逆向工程这些权重，找出运行了什么算法。激活就像内存。逆向工程一个计算机程序需要同时理解二进制指令和它们操作的内存。所以机制可解释性对两者都感兴趣。有很多关于探针的工作，可能是其中的一部分，但并非所有做这类工作的人都认同自己是在做机制可解释性。一个独特的氛围是，人们倾向于认为梯度下降比你更聪明。梯度下降能想出比我们更好的解决方案。所以有一种谦逊：我们不会事先猜测模型内部发生了什么；我们必须采取自下而上的方法，发现这些模型中实际存在什么，并以此方式研究它们。

To give an example of something I wouldn't consider mechanistic interpretability: there was a lot of work on saliency maps, where you take an image and try to say what part made the model think it's a dog. That might tell you something about the model, but it doesn't tell you what algorithms are running inside the model, how it actually made that decision. So we started using the term mechanistic interpretability to draw that divide. Since then, it's become an umbrella term for a wide variety of work. The distinctive things are: we want to get at the mechanisms, the algorithms. If you think of neural networks as like a computer program, the weights are like a binary computer program, and we'd like to reverse engineer those weights to figure out what algorithms are running. The activations are like memory. Reverse engineering a computer program requires understanding both the binary instructions and the memory they operate on. So mechanistic interpretability is interested in both. There's a lot of work on probing, which might be part of it, but not everyone identifies as doing Mech Interp. A distinctive vibe is that people tend to think gradient descent is smarter than you. Gradient descent comes up with better solutions than us. So there's a kind of humility: we won't guess at priori what's going on inside the model; we have to take a bottom-up approach, discover what happens to exist in these models, and study them that way.

事实上，正如你和其他人随着时间的推移所展示的那样，像普遍性这样的东西——梯度下降的智慧在不同类型的网络中普遍创建特征和电路——使得整个领域成为可能。

The very fact that it's possible to do this, and as you and others have shown over time, things like universality—that the wisdom of gradient descent creates features and circuits universally across different kinds of networks—makes the whole field possible.

是的，这确实是一件了不起且令人兴奋的事情。至少在某种程度上，相同的元素、相同的特征和电路会反复形成。你可以看每一个视觉模型，都会发现曲线检测器和高/低频检测器。事实上，有理由认为相同的东西在生物神经网络和人工神经网络中都会形成。一个著名的例子是视觉模型。

Yes, this is indeed a remarkable and exciting thing. It does seem that, at least to some extent, the same elements, the same features and circuits, form again and again. You can look at every vision model and you'll find curve detectors and high/low frequency detectors. In fact, there's some reason to think the same things form across biological neural networks and artificial neural networks. A famous example is vision models.

在早期层中，它们有 Gabor 滤波器，而 Gabor 滤波器是神经科学家感兴趣并思考了很多的东西。我们在这些模型中发现了曲线检测器；猴子身上也发现了曲线检测器。我们发现了这些高低频检测器，然后一些后续工作在大鼠或小鼠身上也发现了它们。所以它们首先在人工神经网络中被发现，然后在生物神经网络中被发现。关于祖母细胞或 Quiroga 等人的哈利·贝里神经元这个非常著名的结果，我们在视觉模型中发现了非常相似的东西。那是我还在 OpenAI 的时候，我在看他们的 CLIP 模型。你发现这些神经元对图像中的相同实体做出响应。举个具体的例子，我们发现有一个唐纳德·特朗普神经元。不知为何，我想每个人都喜欢谈论唐纳德·特朗普，而唐纳德·特朗普当时非常突出，是一个非常热门的话题。所以我们看过的每一个神经网络，都会找到一个专门针对唐纳德·特朗普的神经元。那是唯一一个总是有专用神经元的人。有时你会有一个奥巴马神经元，有时你会有一个克林顿神经元，但特朗普总是有一个专用的。所以它对特朗普的脸和“特朗普”这个词等所有这些东西做出响应。它不是对某个特定例子或仅仅他的脸做出响应；它是在抽象这个一般概念。所以无论如何，这与 Quiroga 的结果非常相似。因此有证据表明，这种普遍性现象——相同的东西在人工和自然神经网络中都形成——如果这是真的，那将是一件非常了不起的事情。它表明梯度下降在某种意义上找到了正确的切分方式，许多系统都收敛于此。许多不同的神经网络架构都收敛于存在一组自然的抽象，这些抽象是切分问题的一种非常自然的方式，并且很多系统都会收敛于这些抽象。这将是我根据所见所闻做出的粗略推测。

In the early layers they have Gabor filters, and Gabor filters are something that neuroscientists are interested in and have thought a lot about. We find curved detectors in these models; curve detectors are also found in monkeys. We discover these high-low frequency detectors, and then some follow-up work went and discovered them in rats or mice. So they were found first in artificial neural networks and then found in biological neural networks. This really famous result on grandmother neurons or the Halle Berry neuron from Quiroga et al., we found very similar things in vision models. This was while I was still at OpenAI, and I was looking at their CLIP model. You find these neurons that respond to the same entities in images. To give a concrete example, we found that there was a Donald Trump neuron. For some reason, I guess everyone likes to talk about Donald Trump, and Donald Trump was very prominent, was a very hot topic at that time. So every neural network that we looked at, we would find a dedicated neuron for Donald Trump. That was the only person who always had a dedicated neuron. Sometimes you'd have an Obama neuron, sometimes you'd have a Clinton neuron, but Trump always had a dedicated one. So it responds to pictures of his face and the word "Trump," all these things. It's not responding to a particular example or just his face; it's abstracting over this general concept. So in any case, that's very similar to these Quiroga results. So there is evidence that this phenomenon of universality—the same things form across both artificial and natural neural networks—that's a pretty amazing thing if it's true. It suggests that the gradient is sort of finding the right ways to cut things apart, in some sense, that many systems converge on. Many different neural network architectures converge on that there's some natural set of abstractions that are a very natural way to cut apart the problem, and a lot of systems are going to converge on that. That would be my kind of wild speculation from what we've seen.

是的，如果它对于形成表示的模型介质是无关的，那将是很美的。

Yeah, that would be beautiful if it's sort of agnostic to the medium of the model that's used to form the representation.

是的，这是一种基于少数数据点的粗略推测，但似乎确实存在某种意义，相同的东西一次又一次地形成，无论是在自然神经网络中还是在人工神经网络中。其背后的直觉是，为了在理解现实世界中有用，你需要所有相同类型的东西。

Yeah, and it's a kind of wild speculation based on only a few data points, but it does seem like there's some sense in which the same things form again and again, both in natural neural networks and in artificial ones. The intuition behind that would be that in order to be useful in understanding the real world, you need all the same kind of stuff.

是的，如果我们选取狗的概念，在某种意义上，狗的概念就像是宇宙中的一个自然类别。它不仅仅是人类思考世界的一种奇怪特质。或者如果你有线的概念，我们周围就有线；在某种意义上，理解这个房间最简单的方式就是有线的概念。

Yeah, well, if we pick the idea of a dog, there's some sense in which the idea of a dog is like a natural category in the universe. It's not just a weird quirk of how humans think about the world. Or if you have the idea of a line, there are lines around us; it's the simplest way to understand this room in some sense is to have the idea of a line.

是的，你需要曲线来理解圆，你需要所有这些形状来理解更大的事物。这是一个形成的概念层次结构。也许有方法可以不参照这些东西来描述图像，但它们不是最简单或最经济的方式。所以系统会收敛到这些策略。

Yeah, you need a curved line to understand a circle, and you need all those shapes to understand bigger things. It's a hierarchy of concepts that are formed. Maybe there are ways to describe images without reference to those things, but they're not the simplest way or the most economical way. So systems converge to these strategies.

你能谈谈我们一直在提到的特征和电路的一些基本组成部分吗？我想你是在 2020 年的论文《Zoom In: An Introduction to Circuits》中首次描述了它们。

Can you talk through some of the building blocks that we've been referencing of features and circuits? I think you first described them in a 2020 paper, "Zoom In: An Introduction to Circuits."

当然。也许我先描述一些现象，然后逐步引出特征和电路的概念。我花了相当多年，大概五年左右，研究这一个特定的模型 Inception V1，这是一个 2015 年最先进的视觉模型，现在已经远远不是最先进的了。它大约有 10,000 个神经元，我花了很多时间观察这 10,000 多个神经元。有趣的是，有很多神经元没有明显的直观含义，但 Inception V1 上也有很多神经元确实具有非常清晰的直观含义。你会发现一些神经元似乎真的在检测曲线、汽车、车轮、车窗、狗的垂耳、朝右的长鼻狗、朝左的长鼻狗、不同类型的毛发。有整套漂亮的边缘检测器、线条检测器、颜色对比检测器，这些我们称之为高低频检测器的美丽东西。我感觉自己像个生物学家，看着一个蛋白质的新世界，发现所有这些不同的蛋白质相互作用。理解这些模型的一种方式是通过神经元：“哦，有一个检测狗的神经元”和“这里有一个检测汽车的神经元”。你可以问它们是如何连接在一起的。例如，你有这个检测汽车的神经元；它是如何构建的？在上一层，它与一个窗户检测器、一个轮子检测器和一个类似车身的检测器有很强的连接。它寻找车窗在汽车上方，轮子在下方，以及镀铬装饰在中间，尤其是在下部。这就是检测汽车的配方。之前我们说我们想要算法去问这里运行的算法是什么；我们只是在看网络的权重，读出这种检测汽车的配方。这是一个非常简单的粗略配方，但它就在那里。我们称之为电路，这种连接。问题是，并非所有的神经元都是可解释的，这是有原因的。

Absolutely. Maybe I'll start by describing some phenomena and then build to the idea of features and circuits. I spent quite a few years, maybe five years to some extent, studying this one particular model, Inception V1, which was a vision model state-of-the-art in 2015, very much not state-of-the-art anymore. It has maybe about 10,000 neurons, and I spent a lot of time looking at those 10,000 odd neurons. One interesting thing is that there are lots of neurons that don't have some obvious intuitive meaning, but there are a lot of neurons on Inception V1 that do have really clean intuitive meanings. You find neurons that really seem to detect curves, cars, car wheels, car windows, floppy ears of dogs, dogs with long snouts facing to the right, dogs with long snouts facing to the left, different kinds of fur. There's this whole beautiful set of edge detectors, line detectors, color contrast detectors, these beautiful things we call high-low frequency detectors. I felt like a biologist looking at a sort of new world of proteins, discovering all these different proteins that interact. One way to understand these models is in terms of neurons: "Oh, there's a dog-detecting neuron" and "here's a car-detecting neuron." You can ask how those connect together. For example, you have this car-detecting neuron; how was it built? In the previous layer, it's connected really strongly to a window detector, a wheel detector, and a sort of car body detector. It looks for the window above the car, the wheels below, and the car chrome sort of in the middle, especially on the lower part. That's a recipe for a car. Earlier we said we wanted algorithms to go and ask what the algorithm is that runs here; we're just looking at the weights of the network, reading off this kind of recipe for detecting cars. It's a very simple crude recipe, but it's there. We call that a circuit, this connection. The problem is that not all of the neurons are interpretable, and there's reason for that.

想想看，我们稍后可以深入讨论——存在一个叠加假说，有理由认为有时分析事物的正确单位是神经元的组合。所以有时并不是单个神经元代表比如一辆车，而是当你检测到车之后，模型会在下一层中隐藏一点车的信号和一堆狗检测器。它为什么这么做？嗯，也许它只是不想在那一刻对车做太多工作，而是把它存储起来。所以结果就是，有一种微妙的模式：所有这些你以为是狗检测器的神经元，可能它们主要是狗检测器，但它们都稍微贡献了一点来在下一层中表示车。所以现在我们不能再认为可能还有某种东西——我不知道，你可以称之为车概念之类的——但它不再对应一个神经元。所以我们需要一个术语来描述这种类似神经元的实体，这些我们本来希望神经元成为的东西，这些理想化的神经元，这些好的神经元，但也许还有更多隐藏的。我们称它们为特征。那么什么是电路？电路就是这些特征的连接。所以当我们有车检测器，它连接到车窗检测器和车轮检测器，并寻找下面的车轮和上面的车窗，那就是一个电路。电路只是由权重连接的特征集合，它们实现算法。所以它们告诉我们特征是如何使用的，如何构建的，以及如何连接在一起的。

To think, we can get into this more later, that there's this superposition hypothesis, reason to think that sometimes the right unit to analyze things in terms of is combinations of neurons. So sometimes it's not that there's a single neuron that represents, say, a car, but it actually turns out that after you detect the car, the model sort of hides a little bit of the car in the following layer and a bunch of dog detectors. Why is it doing that? Well, you know, maybe it just doesn't want to do that much work on cars at that point, and it's sort of storing it away. So it turns out that the subtle pattern is that there are all these neurons that you think are dog detectors, and maybe they're primarily that, but they all contribute a little bit to representing a car in that next layer. So now we can't really think there might still be something—I don't know, you could call it like a car concept or something—but it no longer corresponds to a neuron. So we need some term for these kind of neuron-like entities, these things that we sort of would have liked the neurons to be, these idealized neurons, the things that are the nice neurons, but also maybe there are more of them somehow hidden. And we call those features. And then what are circuits? Circuits are these connections of features. So when we have the car detector, and it's connected to a window detector and a wheel detector, and it looks for the wheels below and the windows on top, that's a circuit. Circuits are just collections of features connected by weights, and they implement algorithms. So they tell us how features are used, how they are built, how they connect together.

你能稍微解释一下那个表示吗？首先，特征是一个激活方向，你是这么想的。你能做一下那个减男人加女人，word2vec 的东西，你能解释一下那是什么吗？

Can you actually explain that representation a little bit? So first off, a feature is a direction of activation, you think of it that way. Can you do the minus man plus woman, the word2vec stuff, can you explain what that is?

是的，有一个非常简单、清晰的解释正好说明我们在说什么。有一个非常著名的成果，word2vec，由 Thomas Mikolov 等人提出，之后有大量的后续工作探索这个。所以有时我们创建这些词嵌入，将每个词映射到一个向量。顺便说一句，如果你以前没想过，这本身是一件有点疯狂的事情。我们把字典里的每个词都变成一个向量——这有点疯狂。但你可以想象各种将词映射到向量的方式，但似乎当我们训练神经网络时，它们喜欢将词检测器映射成具有特定线性结构的形式，即方向有意义。例如，会有某个方向似乎对应性别，男性词汇在一个方向很远，女性词汇在另一个方向。线性表示假说大致可以理解为，这实际上是根本性的东西——一切都只是不同方向具有意义，将不同的方向向量相加可以表示概念。Mikolov 的论文认真对待了这个想法，其中一个结果是你可以玩这种词算术游戏。所以你可以用 king，减去 man，加上 woman，这样你就在尝试转换性别，确实如果你这样做，结果会接近 queen。你还可以做其他事情，比如 sushi 减去 Japan 加上 Italy 得到 pizza，或者类似的不同事情。所以这在某种意义上就是线性表示假说的核心。你可以把它描述为关于向量空间的纯粹抽象的东西，或者关于神经元激活的陈述，但它实际上是关于方向具有意义这个性质。在某些方面甚至更微妙——它实际上主要是关于能够将事物相加的性质，即你可以独立地修改比如性别和皇室，或者菜系类型和国家以及食物的概念，通过将它们相加。

Yeah, there's this very simple, clean explanation of what we're talking about exactly. So there's this very famous result, word2vec, by Thomas Mikolov et al., and there's been tons of follow-up work exploring this. So sometimes we create these word embeddings where we map every word to a vector. That in itself, by the way, is kind of a crazy thing if you haven't thought about it before. We're turning every word in the dictionary into a vector—that's kind of a crazy idea. But you could imagine all kinds of ways in which you might map words to vectors, but it seems like when we train neural networks, they like to map word detectors such that they have linear structure in a particular sense, which is that directions have meaning. For instance, there will be some direction that seems to correspond to gender, and male words will be far in one direction and female words in another. The linear representation hypothesis is you could sort of think of it roughly as saying that that's actually kind of the fundamental thing that's going on—that everything is just different directions having meanings, and adding different direction vectors together can represent concepts. The Mikolov paper sort of took that idea seriously, and one consequence of it is that you can play this game of doing arithmetic with words. So you can do king, subtract off the word man, and add the word woman, and so you're trying to switch the gender, and indeed if you do that, the result will sort of be close to the word queen. You can do other things like sushi minus Japan plus Italy and get pizza, or different things like this. So this is in some sense the core of the linear representation hypothesis. You can describe it just as a purely abstract thing about vector spaces, or as a statement about the activations of neurons, but it's really about this property of directions having meaning. In some ways it's even a little subtler than that—it's really mostly about this property of being able to add things together, that you can sort of independently modify say gender and royalty, or cuisine type and country and the concept of food, by adding them.

你认为线性假说在规模扩张时仍然成立吗？

Do you think the linear hypothesis holds as it scales?

到目前为止，我认为我所看到的一切都与这个假说一致，而且它并不一定是这样。你可以写出神经网络，其中你设置的权重使得它们没有线性表示，正确的做法是……

So far, I think everything I have seen is consistent with this hypothesis, and it doesn't have to be that way. You can write down neural networks where you write weights such that they don't have linear representations, where the right way to...

理解它们不是从线性表示的角度，但我认为我见过的每一个自然神经网络都有这个性质。最近有一篇论文在边缘上有所突破，所以我认为有一些工作在研究多维特征，其中不是单一方向，而更像是一个方向流形。对我来说，这仍然像是线性表示。还有一些其他论文表明，也许在非常小的模型中你会得到非线性表示。我认为这个问题尚无定论，但就我们目前所见，一切都与线性表示假说一致，这很惊人。它本不必如此，但我认为有大量证据表明这至少非常普遍，而且目前的证据与此一致。

Understand them is not in terms of linear representations, but I think every natural neural network I've seen has this property. There's been one paper recently that's been pushing around the edges, so I think there's been some work studying multi-dimensional features where rather than a single direction it's more like a manifold of directions. This to me still seems like a linear representation. And then there's been some other papers suggesting that maybe in very small models you get nonlinear representations. I think that the jury's still out on that, but in everything that we've seen so far has been consistent with the linear representation hypothesis, and that's wild. It doesn't have to be that way, and yet I think that there's a lot of evidence that certainly at least this is very very widespread and so far the evidence is consistent with that.

你可能会说，克里斯托弗，如果我们不确定这是真的，你却假设所有网络都是这样来研究，这风险很大吧？

You might say, well Christopher, that's a lot to ride on if we don't know for sure this is true and you're investigating all networks as though it is true. Isn't that dangerous?

我认为认真对待假说并将其推向极致是有好处的。也许有一天我们会发现与线性表示假说不一致的东西，但科学史上充满了错误的假说和理论。我们在假设它们成立的前提下工作，然后尽可能推进，从中我们学到了很多。我想这就是我们所说的常规科学的核心。如果你想谈科学哲学和范式转换，我们可以聊很多。

Well, I think there's a virtue in taking hypotheses seriously and pushing them as far as they can go. So it might be that someday we discover something inconsistent with the linear representation hypothesis, but science is full of hypotheses and theories that were wrong. We learned a lot by working under them as an assumption and then pushing them as far as we can. I guess this is the heart of what we would call normal science. I don't know if you want to talk about philosophy of science and paradigm shifts.

我喜欢。认真对待假说，并将其推向自然的结论。Scaling 假说也一样。

I love it. Taking the hypothesis seriously and take it to a natural conclusion. Same with the scaling hypothesis.

完全正确。我喜欢。我的一位同事，前物理学家 Tom Henigan，给我做了一个很好的类比，关于热质说。曾经我们认为热是一种叫做热质的东西，热物体使冷物体变暖是因为热质在流动。因为我们习惯了用现代理论思考热，这听起来有点傻，但实际上很难设计一个实验来反驳热质假说。相信热质说实际上可以做很多有用的工作。例如，最初的内燃机是由相信热质说的人开发的。所以我认为，即使假说可能是错的，认真对待它们也有好处。

Exactly, exactly. I love it. One of my colleagues, Tom Henigan, a former physicist, made this really nice analogy to me of caloric theory. Once upon a time we thought that heat was this thing called caloric, and hot objects warm up cool objects because caloric is flowing through them. Because we're so used to thinking about heat in terms of modern theory, that seems kind of silly, but it's actually very hard to construct an experiment that disproves the caloric hypothesis. You can actually do a lot of useful work believing in caloric. For example, the original combustion engines were developed by people who believed in the caloric theory. So I think there's a virtue in taking hypotheses seriously even when they might be wrong.

是的，这有深刻的哲学道理。我对太空旅行也有类似的感觉，比如殖民火星。很多人批评这一点，但我认为如果你假设我们必须殖民火星来为人类文明提供备份，即使这不是真的，也会产生一些有趣的工程甚至科学突破。

Yeah, there's a deep philosophical truth to that. That's kind of how I feel about space travel, like colonizing Mars. A lot of people criticize that, but I think if you just assume we have to colonize Mars to have a backup for human civilization, even if that's not true, that's going to produce some interesting engineering and even scientific breakthroughs.

是的，实际上我认为这很有趣。社会让一些人几乎非理性地致力于研究特定假说，这非常有用，因为当大多数科学假说最终被证明是错误的时候，维持科学士气并真正推动某件事需要很多努力。很多科学并不成功，但去做它非常有用。有一个关于杰夫·辛顿的笑话：在过去的 50 年里，他每年都发现了大脑的工作原理。但我这么说带着深深的敬意，因为事实上这让他做出了一些非常伟大的工作。他获得了诺贝尔奖。现在谁在笑？

Yeah, well, and actually this is another thing I think is really interesting. There's a way in which it can be really useful for society to have people almost irrationally dedicated to investigating particular hypotheses, because it takes a lot to maintain scientific morale and really push on something when most scientific hypotheses end up being wrong. A lot of science doesn't work out, but it's very useful to go do it. There's a joke about Geoff Hinton: he has discovered how the brain works every year for the last 50 years. But I say that with deep respect, because in fact that led him to do some really great work. He won the Nobel Prize. Now who's laughing?

完全正确。我认为一个人需要能够跳出来并认识到适当的置信水平，但仅仅假设这个问题是可能的或者这大致是正确的方向，然后在一段时间内假设这一点并在这个框架内努力推进，也有很大的价值。如果社会有很多人为不同的事情这样做，那实际上非常有用，要么真正排除某些东西——我们可以说，那行不通，我们知道有人努力尝试过——要么深入并得到一些能教给我们关于世界的东西。

Exactly, exactly. Yeah, I think one wants to be able to pop up and recognize the appropriate level of confidence, but there's also a lot of value in just being like, I'm going to assume I'm going to condition on this problem being possible or this being broadly the right approach, and I'm just going to go and assume that for a while and work within that and push really hard on it. If society has lots of people doing that for different things, that's actually really useful in terms of either really ruling things out—we can be like, well that didn't work, we know that somebody tried hard—or going in and getting to something that does teach us something about the world.

另一个有趣的假说是叠加假说。你能描述一下什么是叠加吗？

Another interesting hypothesis is the superposition hypothesis. Can you describe what superposition is?

是的。之前我们讨论过词向量，以及如何可能有一个方向对应性别，另一个对应王室，另一个对应意大利，另一个对应食物，等等。但通常这些词嵌入可能有 500 维、1000 维。所以如果你相信所有这些方向都是正交的，那么你只能有 500 个概念。我喜欢披萨，但如果我要列出英语中最重要的 500 个概念，意大利可能不在其中。你必须先有单数和复数、动词和名词、形容词等等。在谈到意大利、日本之前有很多东西要处理，而且世界上有很多国家。那么模型如何能够同时让线性表示假说成立，又表示比它们方向数更多的东西呢？这意味着什么？好吧，如果线性表示假说成立，那么一定有什么有趣的事情发生。在讨论之前，我再告诉你一件有趣的事：之前我们讨论过所有那些多语义神经元，对吧？当我们看 Inception V1 时，有一些很好的神经元，比如汽车检测器、曲线检测器等等，它们对很多东西都有反应。

Yeah. So earlier we were talking about word vectors, and how maybe you have one direction that corresponds to gender, another that corresponds to royalty, another to Italy, another to food, and all these things. Well, often times these word embeddings might be 500 dimensions, a thousand dimensions. So if you believed that all of those directions were orthogonal, then you could only have 500 concepts. I love pizza, but if I was going to give the 500 most important concepts in the English language, probably Italy wouldn't be one of them. You have to have things like plural and singular, verb and noun and adjective. There's a lot of things we have to get to before we get to Italy, and Japan, and there's a lot of countries. So how might it be that models could simultaneously have the linear representation hypothesis be true and also represent more things than they have directions? So what does that mean? Well, if the linear representation hypothesis is true, something interesting has to be going on. I'll tell you one more interesting thing before we go there: earlier we were talking about all these polysemantic neurons, right? These neurons that, when we were looking at Inception V1, there are these nice neurons like the car detector and the curve detector and so on that respond to lots of things.

连贯的东西，但也有很多神经元对一堆不相关的事物做出反应。这也是一个有趣的现象。而且事实证明，即使是那些非常干净的神经元，如果你看它们的弱激活——比如激活强度只有最大激活的 5%——那其实并不是它真正期待的核心内容，对吧？所以如果你看一个曲线检测器，比如，你看它 5%激活的地方，你可以把它解释为噪声，或者也可能是它在做别的事情。那怎么会这样呢？数学中有一个惊人的东西叫压缩感知。这其实是一个非常令人惊讶的事实：如果你有一个高维空间，把它投影到低维空间，通常你无法反向投影找回原来的高维向量，对吧？你丢掉了信息。就像你不能对一个长方形矩阵求逆，只能对方阵求逆。但事实证明，如果我告诉你那个高维向量是稀疏的——大部分是零——那么你通常可以以很高的概率找回那个高维向量。所以这是一个令人惊讶的事实，对吧？它说明你可以有这样一个高维向量空间，只要东西是稀疏的，你就可以把它投影下来，得到一个低维投影，而且这样是可行的。所以叠加假说就是说神经网络里发生的就是这个。比如，词嵌入里就是这样的：词嵌入能够同时让方向成为有意义的东西，通过利用它们在一个相当高维的空间中运作的事实——以及这些概念是稀有的，比如你通常不会同时谈论日本和意大利——大多数句子中这些概念都是零，根本没有出现。如果这是真的，那么你就可以有比维度多得多的有意义的方向——这些特征。而当我们谈论神经元时，你可以有比神经元多得多的概念。所以这就是高层次的叠加假说。现在它有一个更疯狂的推论，那就是神经网络可能不仅仅是表示是这样的，计算可能也是这样的——它们之间的连接。所以从某种意义上说，神经网络可能是更大、更稀疏的神经网络的影子，我们看到的是这些投影。而叠加假说最强的版本就是认真对待这一点，说在某种意义上确实存在一个上层模型，其中神经元非常稀疏且全部可解释，它们之间的权重是这些非常稀疏的电路，而这就是我们正在研究的。我们观察到的是它的影子，我们需要找到原始对象。学习的过程就是试图构建上层模型的一个压缩，在投影中不丢失太多信息——找到如何高效地拟合它之类的。梯度下降实际上就在做这个。所以这有点像是在说，梯度下降可以只是表示一个密集的神经网络，但它说明梯度下降是在愉快地搜索那些可以投影到这个低维空间的极其稀疏的模型的空间。有很多人研究稀疏神经网络——你设计神经网络时让边稀疏、激活稀疏。我的感觉是，那项工作非常有原则性，非常有道理，但总的来说并没有取得很好的成果。我认为一个可能的答案是，实际上神经网络在某种意义上已经是稀疏的了。梯度下降一直在幕后，比你更高效地搜索稀疏模型的空间，学习最有效的稀疏模型，然后想办法把它很好地折叠起来，在你的 GPU 上运行，而 GPU 做的是漂亮的密集矩阵乘法。你就是无法超越这个。

Coherent things, but it's lots of neurons that respond to a bunch of unrelated things. That's also an interesting phenomenon. And it turns out as well that even these neurons that are really, really clean, if you look at the weak activations—so if you look at the activation where it's like activating 5% of the maximum activation—it's really not the core thing that it's expecting, right? So if you look at a curve detector, for instance, and you look at the places where it's 5% active, you could interpret it just as noise, or it could be that it's doing something else there. So how could that be? Well, there's this amazing thing in mathematics called compressed sensing. And it's actually this very surprising fact: if you have a high-dimensional space and you project it into a low-dimensional space, ordinarily you can't go and unproject it to get back your high-dimensional vector, right? You threw information away. This is like you can't invert a rectangular matrix; you can only invert square matrices. But it turns out that that's actually not quite true if I tell you that the high-dimensional vector was sparse—so it's mostly zeros. Then it turns out that you can often go and find back the high-dimensional vector with very high probability. So that's a surprising fact, right? It says that you can have this high-dimensional vector space, and as long as things are sparse, you can project it down, you can have a lower-dimensional projection of it, and that works. So the superposition hypothesis is saying that that's what's going on in neural networks. That's, for instance, what's going on in word embeddings: the word embeddings are able to simultaneously have directions be the meaningful thing, and by exploiting the fact that they're operating on a fairly high-dimensional space—and the fact that these concepts are rare, like you usually aren't talking about Japan and Italy at the same time—most of those concepts in most sentences are both zero; they're not present at all. And if that's true, then you can have it be the case that you can have many more of these sort of directions that are meaningful—these features—than you have dimensions. And when we're talking about neurons, you can have many more concepts than you have neurons. So that's the high-level superposition hypothesis. Now it has this even wilder implication, which is to say that neural networks may not just be the case that the representations are like this, but the computation may also be like this—the connections between all of them. And so in some sense, neural networks may be shadows of much larger, sparser neural networks, and what we see are these projections. And the strongest version of the superposition hypothesis would be to take that really seriously and sort of say there actually is, in some sense, this upstairs model where the neurons are really sparse and all interpretable, and the weights between them are these really sparse circuits, and that's what we're studying. And the thing that we're observing is the shadow of it, and we need to find the original object. And the process of learning is trying to construct a compression of the upstairs model that doesn't lose too much information in the projection—finding how to fit it efficiently or something like this. Gradient descent is doing this, in fact. So this sort of says that gradient descent could just represent a dense neural network, but it says that gradient descent is pleasantly searching over the space of extremely sparse models that could be projected into this low-dimensional space. And there's this large body of work of people trying to study sparse neural networks—where you design neural networks where the edges are sparse and the activations are sparse. And my sense is that work feels very principled, it makes so much sense, and yet that work hasn't really panned out that well, broadly. And I think a potential answer for that is that actually the neural network is already sparse in some sense. Gradient descent was the whole time, behind the scenes, searching more efficiently than you could through the space of sparse models, learning whatever sparse model was most efficient, and then figuring out how to fold it down nicely to run on your GPU, which does nice dense matrix multiplies. And you just can't beat that.

你认为一个神经网络能塞进多少个概念？

How many concepts do you think can be shoved into a neural network?

取决于它们有多稀疏。所以可能有一个来自参数数量的上限，对吧？因为你需要有权重把它们连接起来。所以这是一个上限。事实上，压缩感知和约翰逊-林登斯特劳斯引理等所有那些可爱的结果都基本上告诉你，如果你有一个向量空间，你想要几乎正交的向量——这大概就是你想要的，对吧？你要放弃让概念严格正交，但希望它们不要干扰太多，所以你必须要求它们几乎正交。那么一旦你设定了你愿意接受的余弦相似度阈值，这个阈值实际上与神经元数量成指数关系。所以到某个点，这甚至不会成为限制因素。但那里有一些漂亮的结果，事实上在某种意义上可能甚至更好，因为那是在说任何随机的一组特征都可能被激活，但实际上特征有一个相关结构，有些特征更可能共现，有些则不太可能。所以神经网络，我猜，在打包东西方面可以做得非常好，以至于这可能不是限制因素。

Depends on how sparse they are. So there's probably an upper bound from the number of parameters, right? Because you have to have weights that connect them together. So that's one upper bound. There are, in fact, all these lovely results from compressed sensing and the Johnson-Lindenstrauss lemma and things like this that basically tell you that if you have a vector space and you want to have almost orthogonal vectors—which is probably the thing you want here, right? You're going to give up on having your concepts strictly orthogonal, but you'd like them to not interfere that much, so you have to ask them to be almost orthogonal. Then this would say that for once you set a threshold for what you're willing to accept in terms of how much cosine similarity there is, that's actually exponential in the number of neurons you have. So at some point, that's not going to even be the limiting factor. But there are some beautiful results there, and in fact it's probably even better than that in some sense, because that's sort of saying that any random set of features could be active, but in fact the features have a correlational structure where some features are more likely to co-occur and others are less likely to co-occur. So neural networks, my guess, can do very well in terms of packing things in to the point that that's probably not the limiting factor.

多义性问题在这里是如何出现的？

How does the problem of polysemanticity enter the picture here?

多义性是我们观察到的现象：我们看很多神经元，神经元并不只代表一个概念；它不是干净的特征；它对一堆不相关的事物做出反应。而叠加假说可以被看作是一个解释这一点的假说。

Polysemanticity is this phenomenon we observe where we look at many neurons and the neuron doesn't just represent one concept; it's not a clean feature; it responds to a bunch of unrelated things. And the superposition hypothesis can be thought of as a hypothesis that explains this.

观察到了多义性。多义性是一种被观察到的现象，而叠加是一个可以解释它以及其他一些现象的假说。这让可解释性变得更加困难，对吧？如果你想从单个神经元的角度来理解事物，而你又有多义神经元，那你就麻烦大了。最简单的回答是，好吧，你在看这些神经元，试图理解它们，这个神经元对很多事情都有反应，它没有一个明确的意义。那很糟糕。另一个问题是，我们最终想理解权重。如果你有两个多义神经元，每个都对三件事有反应，它们之间还有一个权重，那意味着什么？是不是意味着所有三个，比如有九种相互作用？这非常奇怪。但还有一个更深层的原因，与神经网络在高维空间上运行有关。我说过我们的目标是理解神经网络和机制。你可能会问，为什么不直接看它呢？它只是一个数学函数。我最早的一个项目研究了那些将二维空间映射到二维空间的神经网络，你可以用很漂亮的方式解释它们，比如弯曲流形。为什么我们不能那样做呢？因为当你有一个更高维的空间时，那个空间的体积在某种意义上随着输入数量呈指数增长，所以你无法直接进去可视化它。所以我们需要以某种方式把它拆开，把那个指数空间分解成一堆我们可以独立推理的东西。独立性至关重要，因为它让你不必考虑所有指数级的组合。事物是单义的，只有一个意义，这是让你能独立思考它们的关键。所以我认为，这就是为什么我们想要可解释的单义特征的最深层原因。

Observation of polysemanticity. Polysemanticity is this observed phenomenon, and superposition is a hypothesis that would explain it, along with some other things. So that makes interpretability more difficult, right? If you're trying to understand things in terms of individual neurons and you have polysemantic neurons, you're in an awful lot of trouble. I mean, the easiest answer is like, okay, you're looking at the neurons, you're trying to understand them, this one responds to a lot of things, it doesn't have a nice meaning. Okay, that's bad. Another thing you could ask is, ultimately we want to understand the weights. If you have two polysemantic neurons, and each one responds to three things, and there's a weight between them, what does that mean? Does it mean that all three, like there are nine interactions going on? It's a very weird thing. But there's also a deeper reason, which is related to the fact that neural networks operate on really high-dimensional spaces. I said that our goal was to understand neural networks and understand the mechanisms. One thing you might say is, why not just look at it? It's just a mathematical function. One of the earliest projects I did studied these neural networks that mapped two-dimensional spaces to two-dimensional spaces, and you can sort of interpret them in this beautiful way, like bending manifolds. Why can't we do that? Well, as you have a higher-dimensional space, the volume of that space in some senses is exponential in the number of inputs you have, so you can't just go in and visualize it. So we somehow need to break that apart, we need to break that exponential space into a bunch of things that we can reason about independently. And independence is crucial because it allows you to not have to think about all the exponential combinations of things. Things being monosemantic, things only having one meaning, is the key thing that allows you to think about them independently. So I think that's the deepest reason why we want to have interpretable monosemantic features.

所以你最近的工作目标就是，如何从具有多义特征和所有这些混乱的神经网络中提取出单义特征？

So the goal here, as your recent work has been aiming at, is how do we extract the monosemantic features from a neural net that has polysemantic features and all this mess?

是的。我们观察到这些多义特征，并假设叠加是背后的原因。如果叠加是原因，那么实际上有一个成熟的技术，是原则性的做法，那就是字典学习。事实证明，如果你做字典学习，特别是用一种高效且正则化良好的方式，叫做稀疏自编码器，如果你训练一个稀疏自编码器，这些漂亮的可解释特征就会开始涌现，而之前那里什么都没有。这不是你一定能预测到的，但事实证明它非常有效。对我来说，这似乎是对线性表示和叠加的一个相当重要的验证。

Yes. We observe these polysemantic features, and we hypothesize that superposition is what's going on. If superposition is what's going on, there's actually a well-established technique that is the principled thing to do, which is dictionary learning. It turns out if you do dictionary learning, in particular if you do a nice efficient way that nicely regularizes it as well, called a sparse autoencoder, if you train a sparse autoencoder, these beautiful interpretable features start to just fall out where there weren't any beforehand. That's not something you would necessarily predict, but it turns out that works very, very well. To me, that seems like some non-trivial validation of linear representations and superposition.

所以通过字典学习，你并不是在寻找特定类型的类别，你不知道它们是什么。这又回到了我们之前的观点：我们不做出假设。梯度下降比我们聪明，所以我们不对那里有什么做出假设。当然你可以那样做，假设有一个 PHP 特征然后去搜索它，但我们没有那样做。我们说我们不知道那里会有什么；相反，我们只是让稀疏自编码器去发现那里有什么。

So with dictionary learning, you're not looking for particular kinds of categories, you don't know what they are. This gets back to our earlier point: we're not making assumptions. Gradient descent is smarter than us, so we're not making assumptions about what's there. One certainly could do that, assume there's a PHP feature and go search for it, but we're not doing that. We're saying we don't know what's going to be there; instead, we're just going to let the sparse autoencoder discover the things that are there.

你能谈谈去年十月的单义性论文吗？那篇论文有很多不错的突破性成果。

Can you talk about the monosemanticity paper from October last year that had a lot of nice breakthrough results?

你这么说真是太客气了。是的，这是我们使用稀疏自编码器的第一个真正成功。我们拿了一个单层模型，结果发现如果你对它做字典学习，你会找到所有这些非常漂亮的可解释特征。阿拉伯语特征、希伯来语特征、Base64 特征——这些是我们深入研究的一些例子，并且真正证明了它们就是我们想的那样。事实证明，如果你训练一个两倍好的模型，训练两个不同的模型并进行字典学习，你会在两者中找到类似的特征。这很有趣。你会找到各种不同的特征。这真的只是证明了这是可行的。我应该提一下，Cunningham 等人差不多在同一时间得到了非常相似的结果。做这种小规模实验并发现它真的有效，是很有趣的。

That's very kind of you to describe it that way. Yeah, this was our first real success using sparse autoencoders. We took a one-layer model, and it turns out if you do dictionary learning on it, you find all these really nice interpretable features. The Arabic feature, the Hebrew feature, the Base64 feature—those were some examples we studied in a lot of depth and really showed that they were what we thought they were. It turns out if you train a model twice as well, and train two different models and do dictionary learning, you find analogous features in both of them. So that's fun. You find all kinds of different features. That was really just showing that this works. I should mention that there was Cunningham et al. that had very similar results around the same time. There's something fun about doing these kinds of small-scale experiments and finding that it's actually working.

这里有很多结构。也许退一步想想，我原以为也许所有这种机械可解释性真的能行。最终结果会是我有一个解释，说明为什么它非常困难且不可行。我们会说，好吧，叠加有问题，结果叠加真的很难，我们完蛋了。但事实并非如此。实际上，一个非常自然的技术就奏效了。所以这实际上是一个非常好的情况。我认为这是一个困难的研究问题，有很多研究风险，它仍然很可能失败，但我认为当它开始奏效时，我们已经把一些非常重大的研究风险抛在了身后。

There's so much structure here. Maybe stepping back for a while, I thought that maybe all this mechanistic interpretability could really work. The end result was going to be that I would have an explanation for why it was very hard and not going to be tractable. We'd be like, well, there's this problem with superposition, and it turns out superposition is really hard, and we're kind of screwed. But that's not what happened. In fact, a very natural technique just works. So that's actually a very good situation. I think this is a hard research problem and it's got a lot of research risk, and it might still very well fail, but I think that some amount of very significant research risk was sort of put behind us when that started to work.

你能描述一下通过这种方式可以提取出什么样的特征吗？

Can you describe what kind of features can be extracted in this way?

嗯，这取决于你研究的模型。模型越大，特征就越复杂。我们可能一会儿会讨论后续工作。但在这些单层模型中，一些非常常见的东西是语言，包括编程语言和自然语言。有很多特征是在特定上下文中的特定单词。比如单词'the'后面很可能跟一个名词。你可以把这看作'the'特征，但也可以看作预测一个特定名词的特征。还会有一些特征在'the'出现在法律文件或数学文件等上下文中时被激活。所以也许在数学上下文中，'the'后面会预测某个东西。

Well, it depends on the model you're studying. The larger the model, the more sophisticated they're going to be. We'll probably talk about follow-up work in a minute. But in these one-layer models, some very common things were languages, both programming languages and natural languages. There were a lot of features that were specific words in specific contexts. So the word 'the' is likely about to be followed by a noun. You could think of this as the 'the' feature, but you could also think of it as predicting a specific noun feature. And there would be these features that would fire for 'the' in the context of a legal document or a mathematical document or something like this. So maybe in the context of math, you're like 'the' and then predict something.

向量矩阵，你知道所有这些数学词汇，而在其他上下文中你会预测其他东西。这很常见，基本上你需要聪明的人类来为我们看到的东西分配标签。是的，所以这唯一做的事情就是为你展开事物。如果一切都被折叠在一起，你无法真正看到它；这就是在展开它。但现在你仍然有一个非常复杂的东西需要理解。所以你必须做大量工作来理解这些是什么，其中一些非常微妙。即使在这个单层模型中，也有一些关于 Unicode 的很酷的东西，当然有些语言是 Unicode 的，分词器不一定为每个 Unicode 字符都有专门的 token。所以相反，你有这些交替 token 的模式，每个代表 Unicode 字符的一半，然后有一个不同的特征在相反的 token 上激活，就像说，好的，我刚完成一个字符，去预测下一个前缀。然后在前缀上，预测一个合理的后缀，你必须来回交替。所以这些世界模型真的很有趣。还有另一件事：你可能认为只有一个 base64 特征，但事实证明实际上有很多 base64 特征，因为你可以有编码为 base64 的英文文本，它的 base64 token 分布与常规的非常不同，并且它还可以利用一些关于分词的东西。各种有趣的东西。给这些现象分配标签的任务有多难？这能被 AI 自动化吗？

Vector matrix, you know all these mathematical words, whereas in other contexts you would predict other things. That was common, and basically you need clever humans to assign labels to what we're seeing. Yes, so this is the only thing this is doing is sort of unfolding things for you. So if everything was sort of folded over on top of itself, you can't really see it; this is unfolding it. But now you still have a very complex thing to try to understand. So then you have to do a bunch of work understanding what these are, and some of them are really subtle. There's some really cool things even in this one-layer model about Unicode, where of course some languages are in Unicode and the tokenizer won't necessarily have a dedicated token for every Unicode character. So instead, you have these patterns of alternating tokens that each represent half of a Unicode character, and then you have a different feature that goes and activates on the opposing ones to be like, okay, I just finished a character, go and predict the next prefix. Then okay, on the prefix, predict a reasonable suffix, and you have to alternate back and forth. So these world models are really interesting. And there's another thing: you might think there would just be one base64 feature, but it turns out there's actually a bunch of base64 features because you can have English text encoded as base64, and that has a very different distribution of base64 tokens than regular, and there's some things about tokenization that it can exploit. All kinds of fun stuff. How difficult is the task of assigning labels to what's going on? Can this be automated by AI?

嗯，我认为这取决于特征，也取决于你有多信任你的 AI。所以有很多工作在做自动化可解释性。我认为这是一个非常令人兴奋的方向，我们做了相当多的自动化可解释性，让 Claude 去标记我们的特征。有一些有趣的时刻，它完全正确或完全错误。是的，我认为很常见的是它说了一些非常笼统的话，在某种意义上是对的，但并没有真正抓住具体发生了什么。所以我认为这是相当常见的情况。你不知道我有一个特别有趣的例子。那种它是对的但没有触及深层细微差别的差距。是的，这是一个普遍的挑战：它就像一个不可思议的工具，它们可以说出一个真实的事情，但有时缺乏深度。在这个背景下，就像 ARC 挑战，那种智商测试。感觉弄清楚一个特征代表什么是一个需要解决的小谜题。而且我认为有时它们更容易，有时更难。所以是的，我认为这很棘手。现在还有另一件事，我不知道，也许在某种程度上这是我的审美在作祟，但我会试着给你一个合理化解释。我实际上对自动化可解释性有点怀疑，部分原因是我希望人类理解神经网络。如果神经网络为我理解它，我不太喜欢那样。但我确实有点，在某种程度上我有点像那些数学家，他们说，如果有计算机自动证明，那不算数，你不会理解它。但我也认为存在这种关于信任的反思问题，如果你在编写计算机程序，你必须信任你的编译器，如果编译器里有恶意软件，它可能会注入恶意软件到下一个编译器，你就会陷入麻烦。如果你用神经网络来验证你的神经网络是否安全，你测试的假设是，好吧，神经网络可能不安全，你必须担心它是否可能以某种方式搞鬼。所以我认为现在这不是一个大问题，但我确实想知道，从长远来看，如果我们必须使用非常强大的 AI 系统来审计我们的 AI 系统，这真的可信吗？但也许我只是在合理化，因为我只是希望我们能够达到人类理解一切的地步。

Well, I think it depends on the feature and it also depends on how much you trust your AI. So there's a lot of work doing automated interpretability. I think that's a really exciting direction, and we do a fair amount of automated interpretability and have Claude go and label our features. There are some fun moments where it's totally right or it's totally wrong. Yeah, I think it's very common that it says something very general which is true in some sense but not really picking up on the specifics of what's going on. So I think that's a pretty common situation. You don't know that I have a particularly amusing one that's interesting. That little gap between it is true but it doesn't quite get to the deep nuance of a thing. Yeah, that's a general challenge: it's like it's an incredible tool, they can say a true thing but it's missing the depth sometimes. And in this context, it's like the ARC challenge, the sort of IQ type tests. It feels like figuring out what a feature represents is a little puzzle you have to solve. And I think that sometimes they're easier and sometimes they're harder as well. So yeah, I think that's tricky. Now there's another thing which I don't know, maybe in some ways this is my aesthetic coming in, but I'll try to give you a rationalization. I'm actually a little suspicious of automated interpretability, and I think that partly it's just that I want humans to understand neural networks. And if the neural network is understanding it for me, I don't quite like that. But I do have a bit of a, in some ways I'm sort of like the mathematicians who are like, if there's a computer automated proof, it doesn't count, you know, you won't understand it. But I also think that there is this kind of reflections on trusting trust type issue, where if you're writing a computer program you have to trust your compiler, and if there was malware in your compiler then it could go and inject malware into the next compiler, and you'd be kind of in trouble. Well, if you're using neural networks to go and verify that your neural networks are safe, the hypothesis that you're testing for is like, okay, the neural network maybe isn't safe, and you have to worry about whether there is some way that it could be screwing with you. So I think that's not a big concern now, but I do wonder in the long run if we have to use really powerful AI systems to go and audit our AI systems, is that actually something we can trust? But maybe I'm just rationalizing because I just want us to have to get to a point where humans understand everything.

尤其是当我们谈论 AI 安全，以及寻找与 AI 安全相关的特征，比如欺骗等。那么让我们谈谈 2024 年 5 月的 Scaling Interpretability 论文。好的，那么将其扩展到应用于 Claude 3 Sonnet 需要什么？

Especially as we talk about AI safety and it looking for features that would be relevant to AI safety like deception and so on. So let's talk about the scaling interpretability paper in May 2024. Okay, so what did it take to scale this to apply to Claude 3 Sonnet?

嗯，很多 GPU，更多的 GPU。但我的一个队友 Tom Henighan 参与了最初的 Scaling Laws 工作，他从很早开始就感兴趣的一个问题是：可解释性是否存在 Scaling Laws？所以当这项工作开始成功，我们开始让稀疏自编码器工作时，他立即做的事情就是，我们对制作更大的稀疏自编码器的 Scaling Laws 以及这与使基础模型更大之间的关系非常感兴趣。事实证明这非常有效，你可以用它来预测，如果你训练一个给定大小的稀疏自编码器，你应该训练多少 token 等等。所以这实际上对我们扩展这项工作非常有帮助，使我们更容易去训练非常大的稀疏自编码器，虽然不像训练大模型那样，但已经开始变得昂贵了。所以你必须做所有的事情，比如将其分割到大型……我的意思是，这里也有巨大的工程挑战，对吧？所以是的，有一个科学问题是如何有效地扩展事物，然后有大量的工程工作来扩展它。你必须规划它，必须非常仔细地考虑很多事情。我很幸运能与一群优秀的工程师一起工作，因为我绝对不是一个优秀的工程师。是的，尤其是在基础设施方面，当然。所以事实证明，长话短说：它成功了。它成功了。而且我认为这很重要，因为你可以想象一个世界，你追求单义性，你知道，Chris，这很棒。

Well, a lot of GPUs, a lot more GPUs. But one of my teammates, Tom Henighan, was involved in the original scaling laws work, and something he was sort of interested in from very early on is: are there scaling laws for interpretability? So something he sort of immediately did when this work started to succeed and we started to have sparse autoencoders work, we became very interested in what are the scaling laws for making sparse autoencoders larger and how does that relate to making the base model larger. And so it turns out this works really well, and you can use it to sort of project, if you train a sparse autoencoder of a given size, how many tokens should you train on, and so on. So this was actually a very big help to us in scaling up this work, and made it a lot easier for us to go and train really large sparse autoencoders where it's not like training the big models but it's starting to get to a point where it's actually expensive to go and train the really big ones. So you have to do all the stuff of splitting it across large... I mean, there's a huge engineering challenge here too, right? So yes, so there's a scientific question of how do you scale things effectively, and then there's an enormous amount of engineering to go and scale this up. You have to chart it, you have to think very carefully about a lot of things. I'm lucky to work with a bunch of great engineers, because I am definitely not a great engineer. Yeah, on the infrastructure especially, yeah, for sure. So it turns out, TL;DR: it worked. It worked. And I think this is important because you could have imagined a world where you set after towards monosemanticity, you know, Chris, this is great.

你知道这在单层模型上有效，但单层模型非常特殊。也许线性表示假说是理解单层模型的正确方式，但并不是理解大型模型的正确方式。Cutting at all 那篇论文稍微打破了这一点，表明情况并非如此，但 Scaling 上的研究提供了重要证据，表明即使对于非常大的模型——我们在 Claude 3 Sonata 上做了实验，它当时是我们的一个生产模型——这些模型似乎至少在很大程度上可以用线性特征来解释。对它们进行字典学习是有效的，而且随着你学到更多特征，你能解释得越来越多。这是一个相当有希望的迹象。你会发现非常迷人的抽象特征，而且这些特征也是多模态的；它们对同一概念的图像和文本都有反应，这很有趣。

You know it works on a one-layer model, but one-layer models are really idiosyncratic. Maybe the linear representation hypothesis is the right way to understand a one-layer model, but it's not the right way to understand large models. The Cutting at all paper cut through that a little bit and suggested that this wasn't the case, but scaling onity was significant evidence that even for very large models—and we did it on Claude 3 Sonata, which at that point was one of our production models—even these models seem to be substantially explained at least by linear features. Doing dictionary learning on them works, and as you learn more features, you explain more and more. That's a quite promising sign. You find really fascinating abstract features, and the features are also multimodal; they respond to images and text for the same concept, which is fun.

也许先举一个例子。我们发现了一些关于代码中安全漏洞和后门的特征。结果发现这实际上是两个不同的特征。有一个安全漏洞特征，如果你强制激活它，Claude 就会开始编写安全漏洞，比如代码中的缓冲区溢出。它也会对各种事物触发；一些顶级数据集示例包括像 '--disable SSL' 这样的东西，这显然非常不安全。

Maybe let's start with one example. We found some features around security vulnerabilities and backdoors in code. It turns out those are actually two different features. There's a security vulnerability feature, and if you force it active, Claude will start to write security vulnerabilities like buffer overflows into code. It also fires for all kinds of things; some of the top dataset examples for it were things like '--disable SSL' or something like this, which are obviously really insecure.

在这一点上，这有点像也许只是因为示例是以那种方式呈现的，是一些表面明显的例子。我想想法是，将来可能能够检测到更细微的东西，比如欺骗或漏洞之类的。

At this point, it's kind of like maybe it's just because the examples are presented that way, it's kind of surface-level obvious examples. I guess the idea is that down the line, might be able to detect more nuance like deception or bugs or that kind of stuff.

我想区分两件事。一是特征或概念的复杂性，二是我们看到的示例的细微程度。当我们展示顶级数据集示例时，那些是导致该特征激活的最极端例子。这并不意味着它不会对更细微的东西触发。不安全代码特征对非常明显的“禁用安全”类东西触发最强，但它也会对缓冲区溢出和代码中更细微的安全漏洞触发。这些特征都是多模态的，所以你可以问什么图像会激活这个特征。结果发现安全漏洞特征会激活于人们点击 Chrome 以绕过 SSL 证书可能有问题的网站的图像。另一件非常有趣的事情是有一个代码后门特征。如果你激活它，Claude 会编写一个后门，将你的数据转储到一个端口。但你可以问什么图像会激活后门特征。结果是带有隐藏摄像头的设备。有一整个品类的人在销售看起来无害但带有隐藏摄像头的设备，他们的广告显示里面有隐藏摄像头。我想这就是后门的物理版本。这展示了这些概念有多么抽象。我有点难过竟然有整个市场在销售这样的设备，但我有点高兴这成为了该特征的顶级图像示例。

I want to distinguish two things. One is the complexity of the feature or the concept, and the other is the nuance of how subtle the examples we're looking at are. When we show the top dataset examples, those are the most extreme examples that cause that feature to activate. It doesn't mean it doesn't fire for more subtle things. The insecure code feature fires most strongly for really obvious 'disable the security' type things, but it also fires for buffer overflows and more subtle security vulnerabilities in code. These features are all multimodal, so you could ask what images activate this feature. It turns out the security vulnerability feature activates for images of people clicking on Chrome to go past a website where the SSL certificate might be wrong. Another thing that's very entertaining is there's a backdoors in code feature. If you activate it, Claude writes a backdoor that will dump your data to a port. But you can ask what images activate the backdoor feature. It was devices with hidden cameras in them. There's a whole genre of people selling devices that look innocuous but have hidden cameras, and they have ads showing there's a hidden camera in it. I guess that is the physical version of a backdoor. It shows how abstract these concepts are. I'm sort of sad that there's a whole market of people selling devices like that, but I was kind of delighted that that was the top image example for the feature.

很高兴它是多模态的，几乎是多上下文的，是对单一概念的广泛而强有力的定义。对我来说，一个特别有趣的特性，尤其是对于 AI 安全来说，是欺骗和说谎，以及这些方法可能检测模型中的说谎，尤其是当模型变得越来越智能时。可以推测，这是超级智能模型的一个巨大威胁，它可能就自己的意图欺骗操作它的人。那么，你从检测模型内部的说谎中学到了什么？

It's nice that it's multimodal, it's multi-almost context, it's a broad strong definition of a singular concept. To me, one of the really interesting features especially for AI safety is deception and lying, and the possibility that these kinds of methods could detect lying in a model, especially as it gets smarter and smarter. Presumably, that's a big threat of a superintelligent model, that it could deceive the people operating it as to its intentions. So what have you learned from detecting lying inside models?

我认为在这方面我们还在早期阶段。我们发现了一些与欺骗和说谎相关的特征。有一个特征会对人们说谎和欺骗触发，如果你强制激活它，Claude 就会开始对你说谎。所以我们有一个欺骗特征。还有各种其他特征，关于隐瞒信息和不回答问题，关于追求权力和政变之类的。很多特征都与可怕的事情有关，如果你强制激活它们，Claude 会表现出你不想要的行为。

I think we're in some ways in early days for that. We find quite a few features related to deception and lying. There's one feature that fires for people lying and being deceptive, and if you force it active, Claude starts lying to you. So we have a deception feature. There are all kinds of other features about withholding information and not answering questions, features about power-seeking and coups and stuff like that. A lot of features that are kind of related to spooky things, and if you force them active, Claude will behave in ways that are not the kind of behaviors you want.

在机械可解释性领域，你认为接下来有哪些令人兴奋的方向？

What are possible next exciting directions for you in the space of mechanistic interpretability?

有很多事情。首先，我非常希望达到一个阶段，我们拥有电路，不仅能理解特征，还能利用它们来理解模型的计算。对我来说，这是最终目标。已经有一些工作——我们发布了一些东西，Sam Marks 有一篇论文做了类似的事情，还有一些边缘工作。但我认为还有很多事情要做，那将是非常令人兴奋的。这涉及到一个我们称之为干扰权重的挑战，由于叠加，如果你只是天真地查看特征是否连接在一起，可能会有一些权重在上游模型中并不存在，而只是叠加的伪影。所以这是一个相关的技术挑战。我认为另一个令人兴奋的方向是，你可以把稀疏自编码器想象成一种望远镜。它们让我们能够观察并看到所有存在的特征。随着我们构建越来越好的稀疏自编码器，并在字典学习上做得更好，我们会看到越来越多的星星，并放大越来越小的星星。但有很多证据表明，我们仍然只看到了很小一部分星星。在我们的神经网络宇宙中，还有很多物质我们无法观测到。也许我们永远无法拥有足够精细的仪器来观测它，也许其中一些在计算上根本不可行。

There's a lot of things. For one thing, I would really like to get to a point where we have circuits where we can really understand not just the features but then use that to understand the computation of models. That for me is the ultimate goal of this. There's been some work—we put out a few things, there's a paper from Sam Marks that does some stuff like this, some work around the edges here. But I think there's a lot more to do, and that will be a very exciting thing. That's related to a challenge we call interference weights, where due to superposition, if you just naively look at whether features are connected together, there may be some weights that don't exist in the upstream model but are just artifacts of superposition. So that's a technical challenge related to that. I think another exciting direction is just that you might think of sparse autoencoders as being kind of like a telescope. They allow us to look out and see all these features that are out there. As we build better and better sparse autoencoders and get better at dictionary learning, we see more and more stars, and we zoom in on smaller and smaller stars. But there's a lot of evidence that we're only still seeing a very small fraction of the stars. There's a lot of matter in our neural network universe that we can't observe yet. It may be that we'll never be able to have fine enough instruments to observe it, and maybe some of it just isn't computationally tractable.

对于观察者来说，存在一种暗物质——不是早期天文学中那种未知物质。我经常思考这种暗物质，以及我们能否观察到它；如果无法观察到，对安全性意味着什么，如果神经网络中有相当一部分对我们不可及。

To observant, there's a kind of dark matter—not in the sense of astronomy when we didn't know what unexplained matter is. I think a lot about that dark matter and whether we'll ever observe it, and what that means for safety if we can't observe it, if some significant fraction of neural networks are not accessible to us.

我经常思考的另一个问题是，机械可解释性是一种非常微观的方法——试图以非常细粒度的方式理解事物。但我们关心的许多问题都是非常宏观的。我们关心神经网络的行为，这是我最在意的。但还有很多更大尺度的问题你可能关心。微观方法的好处是更容易判断某事是否为真，但缺点是离我们关心的事物更远。所以我们有梯子要爬。问题是：我们能否找到更大尺度的抽象来理解神经网络，从而从这种非常微观的方法上升？

Another question I think a lot about is that mechanistic interpretability is a very microscopic approach—trying to understand things in a very fine-grained way. But many questions we care about are very macroscopic. We care about neural network behavior, and that's what I care most about. But there are lots of larger-scale questions you might care about. The nice thing about a microscopic approach is it's easier to ask if something is true, but the downside is it's much further from the things we care about. So we have this ladder to climb. The question is: can we find larger-scale abstractions to understand neural networks, to get up from this very microscopic approach?

你写过这种器官问题。如果我们把可解释性看作神经网络的解剖学，大多数电路线索涉及研究微小血管、小尺度、单个神经元以及它们如何连接。然而，许多自然问题无法通过小尺度方法解决。相比之下，生物解剖学中最突出的抽象涉及更大尺度的结构，比如单个器官或整个器官系统。所以我们想知道：人工神经网络中是否有呼吸系统、心脏或大脑区域？

You've written about this kind of organs question. If we think of interpretability as a kind of anatomy of neural networks, most of the circuit threads involve studying tiny veins, looking at small scale, individual neurons, and how they connect. However, many natural questions aren't addressed by the small-scale approach. In contrast, the most prominent abstractions in biological anatomy involve larger-scale structures like individual organs or entire organ systems. So we wonder: is there a respiratory system, heart, or brain region of an artificial neural network?

没错。如果你想想科学，许多领域在多个抽象层次上研究事物。在生物学中，有分子生物学研究蛋白质和分子，细胞生物学，组织学，解剖学，动物学，生态学——很多层次。在物理学中，有单个粒子，然后统计物理给出热力学。所以你经常有不同的抽象层次。目前，机械可解释性如果成功，就像神经网络的微生物学，但我们想要更像解剖学的东西。你可能会问为什么不能直接达到那里。答案是，如果不先以正确的方式分解微观结构并研究它们如何连接，就很难看到宏观结构。但我希望会有比特征和电路大得多的东西，我们将有一个涉及更大事物的故事。然后你可以详细研究你关心的部分，就像心理学家或精神病学家研究你自己的网络。美妙之处在于，如果我们能在这些层次之间建立一座桥梁，使得所有更高层次的抽象都牢固地扎根于坚实、严谨的基础上。

Exactly. If you think about science, many fields investigate things at multiple levels of abstraction. In biology, you have molecular biology studying proteins and molecules, cellular biology, histology studying tissues, anatomy, zoology, ecology—many levels. In physics, you have individual particles and then statistical physics giving thermodynamics. So you often have different levels of abstraction. Right now, mechanistic interpretability, if it succeeds, is like microbiology of neural networks, but we want something more like anatomy. You might ask why can't you go there directly. The answer is that it's very hard to see macroscopic structure without first breaking down the microscopic structure in the right way and studying how it connects together. But I'm hopeful there will be something much larger than features and circuits, and we'll have a story that involves much bigger things. Then you can study in detail the parts you care about, like a psychologist or psychiatrist studying your own network. The beautiful thing would be if we could build a bridge between those levels, so that all higher-level abstractions are grounded firmly in a solid, rigorous foundation.

你认为人脑和人工神经网络之间的区别是什么？

What do you think is the difference between the human brain and the artificial neural network?

神经科学家的工作比我们难得多。我庆幸自己的工作容易得多。我们可以记录所有神经元，可以在任意数量的数据上这样做，神经元在你记录时不会改变。我们可以消融神经元，编辑连接，并撤销这些改变。我们可以干预任何神经元，强制它激活，看看会发生什么。我们知道哪些神经元与一切相连——神经科学家想要连接组，我们有，而且比线虫大得多。我们不仅有连接组，还知道哪些神经元兴奋或抑制彼此，我们有权重，可以计算梯度，我们知道每个神经元在计算上做什么。还有很多。尽管有这些优势，这仍然非常困难。所以有时我想：如果对我们来说都这么难，那么在神经科学的约束下似乎不可能。也许有些神经科学家想要一个仍然很难但更容易的问题，他们可以来研究神经网络。然后等我们在理解神经网络这个容易的小池塘里搞清楚了——虽然仍然很难——我们可以回到生物神经科学。

Neuroscientists have a much harder job than us. I count my blessings by how much easier my job is. We can record from all the neurons, we can do that on arbitrary amounts of data, the neurons don't change while you're doing that. We can ablate neurons, edit connections, and undo those changes. We can intervene on any neuron and force it active and see what happens. We know which neurons are connected to everything—neuroscientists want the connectome, we have it, and for much bigger than the elegans. Not only do we have the connectome, we know which neurons excite or inhibit each other, we have the weights, we can take gradients, we know computationally what each neuron does. It goes on and on. Despite all those advantages, it's really hard. So sometimes I think: if it's this hard for us, it seems impossible under the constraints of neuroscience. Maybe some neuroscientists would like an easier problem that's still very hard, and they could come work on neural networks. Then after we figure things out in the easy little pond of understanding neural networks—which is still very hard—we could go back to biological neuroscience.

我喜欢你写的关于机械可解释性研究的目标：安全与美。你能谈谈美的一面吗？

I love what you've written about the goal of mechanistic interpretability research as two goals: safety and beauty. Can you talk about the beauty side?

有趣的是，有些人对神经网络感到失望。他们认为：神经网络只是简单的规则，然后你做一堆工程把它放大，它效果很好。复杂的想法在哪里？这不是一个漂亮、美丽的科学成果。我有时想，当人们这么说时，我仿佛看到他们在说：进化太无聊了，只是一堆简单规则，运行很长时间就得到了生物学。生物学变成这样真糟糕，复杂的规则在哪里？但美在于简单性产生复杂性。生物学有简单的规则，却产生了我们所见的所有生命和生态系统，所有自然之美。这些都来自简单的规则。

There's this funny thing where some people are disappointed by neural networks. They think: neural networks are just simple rules, then you do a bunch of engineering to scale it up, and it works really well. Where are the complex ideas? This isn't a nice beautiful scientific result. I sometimes think when people say that, I picture them saying: evolution is so boring, it's just simple rules, run it for a long time and you get biology. What a sucky way for biology to have turned out, where are the complex rules? But the beauty is that simplicity generates complexity. Biology has simple rules and gives rise to all the life and ecosystems we see, all the beauty of nature. That all just comes from simple rules.

从进化而来，从非常简单的东西进化而来。同样，我认为神经网络在自身内部构建了巨大的复杂性和美，以及结构，人们通常不去看，也不试图去理解，因为它难以理解。但我认为神经网络内部有极其丰富的结构有待发现，很多非常深刻的美，只要我们愿意花时间去观察和理解。

From evolution and from something very simple. Evolution, and similarly, I think that neural networks build enormous complexity and beauty inside and structure inside themselves that people generally don't look at and don't try to understand because it's hard to understand. But I think that there is an incredibly rich structure to be discovered inside neural networks, a lot of very deep beauty, if we're just willing to take the time to go and see it and understand it.

是的，我喜欢机械可解释性。那种我们正在理解或瞥见内部魔法运作的感觉真的很棒。我觉得有一个问题亟待提出。我有点——我的意思是很多人都在思考这个——但我经常惊讶于为什么没有更多人思考：我们不知道如何创建能做这些事情的计算机系统，但我们却拥有这些惊人的系统，我们不知道如何直接编写能做这些事情的计算机程序，而这些神经网络却能做所有这些惊人的事情？这感觉显然是一个亟待回答的问题。如果你有任何好奇心，就会问：人类现在怎么会有这些能完成我们不知道如何做的事情的产物？

Yeah, I love mechanistic interpretability. The feeling like we are understanding or getting glimpses of understanding the magic that's going on inside is really wonderful. It feels to me like one of the questions is just calling out to be asked. And I'm sort of—I mean a lot of people are thinking about this—but I'm often surprised that more people aren't: how is it that we don't know how to create computer systems that can do these things, and yet we have these amazing systems that we don't know how to directly create computer programs that can do these things, but these neural networks can do all these amazing things? And it just feels like that is obviously the question that is calling out to be answered. If you have any degree of curiosity, it's like: how is it that humanity now has these artifacts that can do these things that we don't know how to do?

是的，我喜欢那个朝着目标函数之光前进的杂耍团的意象。是的，它就是我们培育出来的有机体，我们完全不知道我们培育出了什么。

Yeah, I love the image of the circus towards the light of the objective function. Yeah, it's just this organic thing that we've grown and we have no idea what we've grown.

那么，感谢你致力于安全研究，也感谢你欣赏你所发现事物的美。谢谢你今天来聊天，克里斯。这太棒了。

Well, thank you for working on safety and thank you for appreciating the beauty of the things you discover. And thank you for talking today, Chris. This is wonderful.

也谢谢你抽出时间来聊天。

Thank you for taking the time to chat as well.

感谢收听本期与克里斯·奥拉以及之前与达里奥·阿莫迪和阿曼达·阿斯科尔的对话。要支持本播客，请查看描述中的赞助商。现在，让我用艾伦·沃茨的一句话作为结尾：‘理解变化的唯一方法就是投身其中，随它而动，加入舞蹈。’感谢收听，我们下次再见。

Thanks for listening to this conversation with Chris Olah, and before that with Dario Amodei and Amanda Askell. To support this podcast, please check out our sponsors in the description. And now let me leave you with some words from Alan Watts: 'The only way to make sense out of change is to plunge into it, move with it, and join the dance.' Thank you for listening and hope to see you next time.