缩放定律为何有效及其对 AGI 的意义

今天我有幸邀请到 Anthropic 的 CEO Dario Amodei。我真的很期待这次对话。Dario，非常感谢你来做客。

Today I have the pleasure of speaking with Dario Amodei, who is the CEO of Anthropic. I'm really excited about this one. Dario, thank you so much for coming on the podcast.

谢谢邀请。

Thanks for having me.

第一个问题：你是多年来极少数预见 Scaling（规模扩张）的人之一，超过五年。作为预见者，从根本上解释为什么 Scaling 有效？为什么宇宙的规律是，如果你把大量算力投入足够广泛的数据分布，这个东西就会变得智能？

First question: you have been one of the very few people who has seen scaling coming for years, more than five years. As somebody who's seen it coming, what is fundamentally the explanation for why scaling works? Why is the universe organized such that if you throw big blobs of compute at a wide enough distribution of data, the thing becomes intelligent?

我认为真相是我们仍然不知道。这几乎完全是一个经验事实。你可以从数据和其他许多地方感觉到这一点。但我们还没有一个令人满意的解释。如果非要尝试解释，我只是在瞎猜。物理学中有一些关于长尾或幂律相关性的想法。当很多事情发生时，当你有很多特征时，你会在早期、分布的肥大部分得到大量数据，然后才是尾部。对于语言来说，这就像词性、名词跟在动词后面，然后有越来越微妙的关联。所以这有点道理：每增加一个对数数量级，你就捕获了更多分布。但完全不清楚的是，为什么它随参数如此平滑地缩放，为什么随数据量如此平滑地缩放。你可以想出一些线性解释，比如参数是桶，数据是水，桶的大小与水的量成正比。但为什么会导致所有这些非常平滑的缩放？我认为我们仍然不知道。有很多解释，首席科学家 Jared Kaplan 做了一些关于分形流形维度的研究，可以用来解释。所以有各种各样的想法，但我觉得我们并不真正确定。

I think the truth is that we still don't know. I think it's almost entirely an empirical fact. It's a fact that you could kind of sense from the data and from a bunch of different places. But I think we don't still have a satisfying explanation for it. If I were to try to make one, but I'm just waving my hands when I say this. There are these ideas in physics around long tail or power law of correlations or effects. So when a bunch of stuff happens, when you have a bunch of features, you get a lot of the data in the early, fat part of the distribution before the tails. For language, this would be things like parts of speech and nouns follow verbs, and then there are more and more subtle correlations. So it kind of makes sense why there would be this every log order of magnitude that you add, you capture more of the distribution. What's not clear at all is why it scales so smoothly with parameters, why it scales so smoothly with the amount of data. You can think up some explanations of why it's linear, like the parameters are a bucket and the data is water, so size of the bucket is proportional to the size of the water. But why does it lead to all this very smooth scaling? I think we still don't know. There are all these explanations, chief scientists Jared Kaplan did some stuff on fractal manifold dimension that you can use to explain it. So there are all kinds of ideas, but I feel like we just don't really know for sure.

顺便说一下，对于正在听的观众，我们说的 Scaling 指的是你可以非常可预测地看到，从 GPT-2 到 GPT-3 到 GPT-4，或者在这个例子中从 Claude 1 到 Claude 2，预测下一个词元的损失非常平滑地缩放。那么，我们不知道为什么会发生，但至少你能预测吗？比如经验上，在这个损失下这种能力会出现，在这个位置这个电路会出现？这大概可预测，还是你只看损失数字？

By the way, for the audience who's trying to follow along, by scaling we're referring to the fact that you can very predictably see how if you go from GPT-2 to GPT-3 to GPT-4, or in this case Claude 1 to Claude 2, the loss in terms of whether it can predict the next token scales very smoothly. So okay, we don't know why it's happening, but can you at least predict if empirically here is the loss at which this ability will emerge, here is the place where this circuit will emerge? Is that about predictable, or are you just looking at the loss number?

那要难预测得多。可预测的是这种统计平均值、损失、熵。它超级可预测，有时甚至到几个有效数字，这在物理学之外是看不到的。你不会期望在这个混乱的经验领域看到它。但实际上，具体能力很难预测。所以当我还在研究 GPT-2 和 GPT-3 时，比如算术什么时候出现？模型什么时候学会编程？有时非常突然。这有点像你可以预测天气的统计平均值，但某一天的天气很难预测。

That is much less predictable. What's predictable is this statistical average, this loss, this entropy. It's super predictable, sometimes even to several significant figures, which you don't see outside of physics. You don't expect to see it in this messy empirical field. But actually, specific abilities are very hard to predict. So back when I was working on GPT-2 and GPT-3, like when does arithmetic come into place? When do models learn to code? Sometimes it's very abrupt. It's kind of like you can predict statistical averages of the weather, but the weather on one particular day is very hard to predict.

给我简化一下。我不懂流形，但机制上它之前不知道加法，现在知道了。发生了什么？

Dumb it down for me. I don't understand manifolds, but mechanistically it doesn't know addition yet, now it knows addition. What has happened?

这是另一个我们不知道答案的问题。我的意思是，我们正试图用机械可解释性之类的东西来回答。但我不确定。你可以考虑电路突然就位，尽管有证据表明，当你观察模型能够做加法时，如果你看它得到正确答案的概率，它会突然飙升。但如果你看正确答案的概率，你会看到它从百万分之一爬到十万分之一再到千分之一，远在它实际得到正确答案之前。所以在许多情况下至少有一些连续性。我不知道是否在所有情况下，背后都有某种连续过程。我完全不明白。

This is another question that we don't know the answer to. I mean, we're trying to answer this with things like mechanistic interpretability. But I'm not sure. You can think about these things about circuits snapping into place, although there is some evidence that when you look at the models being able to add things, if you look at its chance of getting the right answer, that shoots up all of a sudden. But if you look at the probability of the right answer, you'll see it climb from one in a million to one hundred thousand to one in a thousand long before it actually gets the right answer. So there's some continuity in many of these cases, at least. I don't know if in all of them, there's some continuous process going on behind the scenes. I don't understand it at all.

这是否意味着做加法的电路或过程预先存在，只是显著性增加了？

Does that imply that the circuit or the process for doing addition was pre-existing and it just got increased in salience?

我不知道是否有一个电路很弱然后变强，或者它本来就能工作但效果不好。我认为我们不知道，这些是我们试图用机械可解释性回答的一些问题。

I don't know if there's this circuit that is weak and getting stronger, or if it's something that works but not very well. I think we don't know, and these are some of the questions we're trying to answer with mechanistic interpretability.

有没有能力不会随着规模扩张而出现？

Are there abilities that won't emerge with scale?

我再次认为，像对齐和价值观这样的东西并不保证会随着规模扩张而出现。这有点像你训练模型，它基本上是在预测世界、理解世界。它的工作是事实，而不是价值观。它试图预测接下来会发生什么。但这里有自由变量，比如你应该做什么，你应该想什么，你应该重视什么。这些位就在那里。如果我以这个开始，我应该以这个结束；如果我以另一个开始，我应该以另一个结束。所以我认为这不会出现。

I definitely think that again, like alignment and values are not guaranteed to emerge with scale. It's kind of like you train the model and it is basically predicting the world, understanding the world. Its job is facts, not values. It's trying to predict what comes next. But there are free variables here where it's like what should you do, what should you think, what should you value. Those bits are just there. If I started with this, I should finish with this; if I started with this other thing, I should finish with this other thing. So I think that's not going to emerge.

我马上想谈谈对齐。但关于 Scaling，如果事实证明在我们达到人类水平智能之前 Scaling 就停滞了，回顾起来，你的解释会是什么？如果结果是这样，你认为可能的原因是什么？

I want to talk about alignment in a second. But on scaling, if it turns out that scaling plateaus before we reach human-level intelligence, looking back on it, what would be your explanation? What do you think is likely to be the case if that turns out to be the outcome?

是的，所以我想我会区分根本理论的问题和实际问题。所以一个实际问题我们可能……

Yeah, so I guess I would distinguish some problem with the fundamental theory from some practical issue. So one practical issue we could...

我们可能因为各种原因耗尽数据。我认为这不会发生，但如果你非常天真地看，我们离数据耗尽并不远，所以就像我们没有数据来继续 Scaling 曲线了。我认为另一种可能的方式是我们用完了所有可用的算力，那还不够，之后进展就慢了。我不会押注这些事情发生，但它们有可能。

Have is we could run out of data for various reasons. I think that's not going to happen, but if you look at it very naively, we're not that far from running out of data, so it's like we just don't have the data to continue the scaling curves. I think another way it could happen is we just use up all of our compute that was available and that wasn't enough, and then progress is slow after that. I wouldn't bet on either of those things happening, but they could.

从根本上看，我个人认为缩放定律极不可能就此停止。如果它们停了，另一个原因——同样，这并非根本性——可能是我们没有完全正确的架构。比如如果我们尝试用 LSTM 或 RNN 来做，斜率会不同。我们仍可能达到目标，但我认为有些东西很难表示，当你没有 Transformer 那种能关注遥远过去的能力时。如果不知何故——我不知道我们怎么会知道——这并非架构问题，我们只是撞墙了，我会非常惊讶。我认为我们已经到了模型不能做的事情与能做的事情在种类上没有区别的地步。几年前你还可以说它们不能推理、不能编程。你可以划出界限，说也许你会撞墙。我当时不认为我们会撞墙，其他一些人也这么认为，但那时这个论点更合理。现在我认为它不那么合理了。现在它可能发生——这东西很疯狂，可能明天就发生——但就是撞墙了。如果发生，我在想我的真正解释会是什么。我认为我的解释会是：当你用下一个词预测训练时，损失函数有问题。比如一些剩余的推理能力之类。如果你真的想在高层次上学习一个程序，意味着你关心某些 token 远多于其他，而它们足够稀有，以至于损失函数过度关注那些负责最多熵位的东西，而不关注真正关键的东西。所以信号可能被淹没在噪声中。我不认为事情会那样发展，原因有很多，但如果你告诉我你训练了你的 2024 模型，它更大但没有任何改进，而且你试了所有架构都不行——那我认为这就是我会给出的解释。

I think from a fundamental perspective, I personally think it's very unlikely that the scaling laws will just stop. If they do, another reason—again, this isn't fully fundamental—could be we don't have quite the right architecture. Like if we tried to do it with an LSTM or an RNN, the slope would be different. It still might be that we get there, but I think there are some things that are just very hard to represent when you don't have this ability to attend far in the past that Transformers have. If somehow—and I don't know how we would know this—it kind of wasn't about the architecture and we just hit a wall, I think I'd be very surprised by that. I think we're already at the point where the things the models can't do don't seem to me to be different in kind from the things they can do. You could have made a case a few years ago that it was like they can't reason, they can't program. You could have drawn boundaries and said maybe you'll hit a wall. I didn't think we would hit a wall, a few other people didn't think we would hit a wall, but it was a more plausible case. I think it's a less plausible case now. Now it could happen—this stuff is crazy, it could happen tomorrow—but it's just like we hit a wall. I think if that happens, I'm trying to think of what would really be my explanation. I think my explanation would be there's something wrong with the loss when you train on next-token prediction. Like some of the remaining reasoning abilities or something like that. If you really want to learn a program at a really high level, it means you care about some tokens much more than others, and they're rare enough that the loss function over-focuses on the appearance of things responsible for the most bits of entropy, and instead doesn't focus on this stuff that's really essential. So you could have the signal drowned out in the noise. I don't think it's going to play out that way for a number of reasons, but if you told me you trained your 2024 model, it was much bigger and it just wasn't any better, and you tried every architecture, it didn't work—that I think that's the explanation I would reach for.

如果必须放弃下一个词预测，有没有其他损失函数的候选？

Is there a candidate for another loss function if you had to abandon next-token prediction?

我认为那时你必须采用某种强化学习。同样，有很多不同种类：基于人类反馈的强化学习、针对目标的强化学习、像宪法 AI 这样的东西、放大和辩论等。这些既是对齐方法，也是训练模型的方式。你必须尝试很多方法，但重点必须是我们真正关心模型做什么。从某种意义上说，我们有点幸运，预测下一个词让我们得到了所有其他需要的东西。这没有保证。

I think then you would have to go for some kind of RL. And again, there's many different kinds: RL from human feedback, RL against an objective, things like constitutional AI, things like amplification and debate. These are both alignment methods and ways of training models. You would have to try a bunch of things, but the focus would have to be on what we actually care about the model doing. In a sense, we're a little bit lucky that it's like predict the next word gets us all these other things we need. There's no guarantee.

从你的世界观来看，似乎有很多不同的损失函数。只是什么能让你投入大量数据的问题。下一个词预测本身并不重要。

It seems like from your worldview, there's a multitude of different loss functions. It's just a matter of what can allow you to throw a whole bunch of data at it. The next-token prediction itself is not significant.

是的，我想，嗯，我的意思是强化学习的问题是你会被拖慢一点，因为你必须通过某种方法设计损失函数如何工作。下一个词预测的好处是它就在那里，对吧？它是世界上最简单的事情。所以我认为如果你不能以那种最简单的方式扩展，你会被拖慢。

Yeah, I think well, I mean I guess the thing with RL is you get slowed down a bit because you have to by some method design how the loss function works. The nice thing with next-token prediction is it's there for you, right? It's just the easiest thing in the world. So I think it would slow you down if you couldn't scale in that very simplest way.

你提到数据很可能不是约束。为什么你这么认为？

You mentioned that the data is likely not to be the constraint. Why do you think that is the case?

这里有各种可能性，出于一些原因我不该深入细节，但世界上有很多数据来源，也有很多方法可以生成数据。我的猜测是这不会成为障碍。也许如果它是障碍反而更好，但它不会。

There's various possibilities here, and for a number of reasons I shouldn't go into the details, but there are many sources of data in the world and many ways that you can also generate data. My guess is that this will not be a blocker. Maybe better if it was, but it won't be.

你是在说多模态，还是只是有很多不同的方法？

Are you talking about multimodal or there's just many different ways to do it?

嗯，你是如何形成对 Scaling 的看法的？我们能追溯到多远？然后你基本上会说类似的观点，我可能是在 2014 到 2017 年间逐渐形成的。所以我想我第一次接触它是我第一次接触 AI。我看到了 2012 年 AlexNet 周围的一些早期东西。

Um, how did you form your views on scaling? How far back can we go? And then you would be basically saying something similar to this view that I have probably formed gradually from I would say like 2014 to 2017. So I think my first experience with it was my first experience with AI. I saw some of the early stuff around AlexNet in 2012.

我一直都想研究智能，但以前总觉得这行不通，看起来并不真正有效。早在 2005 年，我读过 Rick Herzwell 的作品，也读过 Eliezer 早期在互联网上的文章，当时觉得这些东西还很遥远。看看今天的 AI，也远未达到。但和 Alex 一起时，我发现这东西真的开始奏效了。于是我加入了 Andrew Ng 的团队，最初在百度。我接到的第一个任务——我来自不同领域，这是我第一次接触 AI——与世界上其他地方许多学术风格的研究有些不同。我觉得自己很幸运，因为交给我的任务就是做出最好的语音识别系统。有大量数据和 GPU 可用，这让我发现 Scaling（规模扩张）是解决方案。这与欧洲的博士后工作截然不同，后者的任务是提出一个聪明新颖的想法，让自己作为发明者留下印记。我很快发现，我只是尝试了最简单的实验，调整一些参数。我试着给 RNN 增加更多层，训练更长时间，看看多久会过拟合，加入新数据并减少重复次数。我看到了非常一致的规律。我当时并不知道这不同寻常，也不知道其他人没有这样思考。这有点像新手运气，是我第一次接触它，而且我没有考虑语音识别之外的东西。我只是觉得，哦，真奇怪，Scaling（规模扩张）似乎在语音识别领域成立。

I always kind of had wanted to study intelligence, but before, I was just like, this isn't really working. It doesn't seem like it's actually working. All the way back to like 2005, I'd read Rick Herzwell's work, I'd read even some of Eliezer's work on the old internet back then, and I was like, oh, this stuff looks far away. I look at the AI stuff of today and it's not anywhere close. But with Alex, I was like, oh, this stuff is actually starting to work. So I joined Andrew Ng's group, initially at Baidu. The first task I got set to do, I was in a different field, so this was my first experience with AI. It was a bit different from a lot of the academic style research going on elsewhere in the world. I think I kind of got lucky in that the task given to me and the other folks was just make the best speech recognition system you can. There was a lot of data available, a lot of GPUs, so it posed the problem in a way that was amenable to discovering that scaling was a solution. That's very different from a European postdoc where it's your job to come up with an idea that seems clever and new and makes your mark as someone who's invented something. I quickly discovered that I just tried the simplest experiments, fiddling with some dials. I tried adding more layers to the RNN, training it for longer, seeing how long it takes to overfit, adding new data and repeating it less times. I saw these very consistent patterns. I didn't really know that this was unusual or that others weren't thinking in this way. This was just kind of like beginner's luck, my first experience with it. I didn't really think about it beyond speech recognition. I was just like, oh, this is weird, scaling seems to be true in the speech recognition field.

我觉得你说的很有意思的一点是，当时有很多人知道这些技术，可能没有直接参与，但知道它们在语音识别或受限游戏中表现很好。但很少有人像你和 Ilya 那样从中推断出某种通用智能。你思考的方式和别人有什么不同，让你从‘它在语音上以这种一致的方式变好’想到‘它会在所有事情上以这种一致的方式变好’？

What I find really interesting about what you said is there are many people who were aware back at that time, probably weren't working on it directly, but were aware that these things are really good at speech recognition or at playing these constrained games. Very few extrapolated from there like you and Ilya did to something that is generally intelligent. What was different about the way you were thinking about it versus how others think that you went from 'it's getting better at speech in this consistent way' to 'it will get better at everything in this consistent way'?

我真的不知道。起初在语音中看到时，我以为这只适用于语音或这类狭窄的模型。我想只是在 2014 到 2017 年间，我在很多事情上尝试了，并反复看到同样的现象。我在 DOTA 中看到了同样的规律，在机器人技术中也看到了，很多人认为机器人是反例，但我只是觉得机器人领域很难获取数据，但如果我们看已有的数据，我们会看到同样的模式。我不知道。我认为人们非常专注于解决眼前的问题。为什么一个人这样想，另一个人那样想，很难解释。我认为人们只是通过不同的视角看待问题，垂直地看而不是水平地看。他们没有考虑 Scaling（规模扩张），而是在想如何很好地解决自己的问题。对于机器人，数据不够，所以很容易抽象成 Scaling（规模扩张）不起作用，因为我们没有数据。我不知道。只是出于某种原因——也许只是随机偶然——我痴迷于那个特定方向。

I genuinely don't know. At first when I saw it in speech, I assumed this was just true for speech or for this narrow class of models. I think it was just over the period between 2014 and 2017 I tried it for a lot of things and saw the same thing over and over again. I watched the same being true with DOTA, I watched the same being true with robotics, which many people thought of as a counterexample, but I just thought what's hard to get data for robotics, but if we operate within the data that we have, we see the same patterns. I don't know. I think people were very focused on solving the problem in front of them. Why one person thinks one way, another person thinks another, it's very hard to explain. I think people just see it through a different lens, looking vertically instead of horizontally. They're not thinking about the scaling, they're thinking about how do I solve my problem well. For robotics, there's not enough data, and so that can easily abstract to scaling doesn't work because we don't have the data. I don't know. I just for some reason, and it may just have been random chance, was obsessed with that particular direction.

你什么时候意识到语言就是向这些东西输入大量数据的手段？还是说你只是用完了其他东西，比如机器人数据不够，这个那个数据不够？

When did it become obvious to you that language is the means to just feed a bunch of data into these things? Or was it just you ran out of other things like robotics, there's not enough data, this other thing there's not enough data?

我认为整个下一个词预测（next-token prediction）的想法，即你可以做自监督学习，再加上预测下一个词时存在如此丰富的结构和信息。它可能会说‘二加二等于’，你必须知道答案是四，也可能在讲一个关于角色的故事，然后基本上是在向模型提出相当于给儿童的发展测试。‘玛丽走进房间，放了一个物品，然后查克走进房间拿走了物品，玛丽没看到。玛丽认为发生了什么？’所以模型为了正确预测下一个词，必须解决所有这些心智理论问题，解决所有这些数学问题。我的想法就是，你尽可能扩展它，没有限制。我抽象地持有这种观点，但真正巩固并说服我的是 Alec Radford 在 GPT-1 上的工作，它不仅让你得到一个能很好预测的语言模型，而且你还可以微调（fine-tune）它。

I think this whole idea of next-token prediction, that you could do self-supervised learning, together with the idea that for predicting the next word there's so much richness and structure there. It might say 'two plus two equals' and you have to know the answer is four, and it might be telling a story about a character, and then basically it's posing to the model the equivalent of developmental tests that get posed to children. 'Mary walks into the room and puts an item in there, and then Chuck walks into the room and removes the item, and Mary doesn't see it. What does Mary think happened?' So the models are going to have to get this right in the service of predicting the next word. They're going to have to solve all these theory of mind problems, solve all these math problems. My thinking was just, well, you scale it up as much as you can, there's kind of no limit to it. I think I had abstractly that view, but the thing that really solidified and convinced me was the work that Alec Radford did on GPT-1, which was not only could you get this language model that could predict things very well, but also you could fine-tune it.

那时候需要微调它来做所有这些其他任务，所以我就想，哇，这不仅仅是把语言模型搞对那么简单，它有点像通往任何地方的半途。你把语言模型搞对了，然后往这个方向稍微一动，它就能解决逻辑解引用测试之类的，往另一个方向，它就能解决翻译之类的问题。然后你就会想，哇，我觉得这里真的有点名堂，当然我们也可以真正地扩展它。

Needed to fine-tune it in those days to do all these other tasks, and so I was like, wow, it isn't just some narrow thing where you get the language model right; it's sort of halfway to everywhere. You get the language model right, and then with a little move in this direction, it can solve logical dereference tests or whatever, and with this other thing, it can solve translation or something. And then you're like, wow, I think there's really something to do, and of course we can really scale it.

有一件事让人困惑，或者说很难预见：如果你在 2018 年告诉我，2023 年会有模型能用莎士比亚的风格写定理，或者任何你想要的风格，它们能做开放式标准化测试，各种令人印象深刻的事情。你当时会说，‘哦，你们有 AGI 了，你们显然有了人类级别的智能。’然而，尽管这些事情令人印象深刻，我们显然还没有达到人类水平，至少在这一代，可能未来几代也不会。如何解释这种在基准测试上超强表现与你能描述的事情之间的差距？

One thing that's confusing, or that would have been hard to see: if you told me in 2018 we'll have models in 2023 that can write theorems in the style of Shakespeare, or whatever theory you want, they can do standardized tests with open-ended questions, all kinds of really impressive things. You would have said at that time, 'Oh, you have AGI, you clearly have something that is human-level intelligence.' Yet while these things are impressive, it clearly seems we're not at human level, at least in the current generation and potentially for generations to come. What explains this discrepancy between super impressive performance on these benchmarks and just the things you could describe?

那是我没有先见之明的一个领域，我也同样感到惊讶。当我第一次看到 GPT-3，以及我们在 Anthropic 早期构建的东西时，我的总体感觉是它们真的掌握了语言的本质。我不确定我们需要多大程度地扩展它们。也许从这里开始更需要的是强化学习和其他东西。我在 2020 年想，我们可以再扩展一些，但我想知道是继续扩展更有效，还是开始添加其他目标比如强化学习。我以为如果你做和预训练一样多的强化学习，对于 2020 年风格的模型来说，那才是正确的道路，扩展会继续有效，但那真的是最好的路径吗？我不知道，它就这么一直走下去。我以为它已经理解了很多语言的本质，但后来发现还有更远的路要走。退一步说，我之所以对 AI、安全、组织非常经验主义，原因之一就是你经常会被惊讶到。我觉得我在一些事情上是对的，但有了这些理论图景，我在大多数事情上是错的。对 10%的事情正确就让你远超许多人了。

That was one area where actually I was not prescient and I was surprised as well. When I first looked at GPT-3 and more so the things we built in the early days at Anthropic, my general sense was that they've really grasped the essence of language. I'm not sure how much we need to scale them up. Maybe what's more needed from here is RL and all the other stuff. I thought in 2020, we can scale this a bunch more, but I wonder if it's more efficient to scale it more or to start adding on these other objectives like RL. I thought maybe if you do as much RL as you've done pre-training for a 2020-style model, that's the way to go, and scaling it up will keep working, but is that really the best path? And I don't know, it just keeps going. I thought it had understood a lot of the essence of language, but then there's further to go. Stepping back, one of the reasons why I'm very empiricist about AI, about safety, about organizations, is that you often get surprised. I feel like I've been right about some things, but with these theoretical pictures, I've been wrong about most things. Being right about 10% of the stuff sets you head and shoulders above many people.

如果你回顾一下，我不记得是谁画了那些图，比如这是村里的傻瓜，这是爱因斯坦，这是智能的尺度。村里的傻瓜和爱因斯坦非常接近。也许在某种抽象意义上仍然如此，但这并不是我们实际看到的，对吧？似乎人类的范围相当广泛，而且我们不会在同一时间或同一地点达到人类范围的不同任务。比如，用科马克·麦卡锡的风格写一首十四行诗——我不是很有创意，所以我做不到，但那是一个相当高水平的人类技能。而模型甚至开始擅长像约束写作这样的事情：写一页关于 X 的文字，不使用字母'e'。我认为模型在这方面可能是超人类或接近超人类的。但当涉及到证明相对简单的数学定理时，它们才刚刚开始；它们有时会犯非常愚蠢的错误，并且缺乏任何广泛的能力，比如纠正错误或执行扩展任务。所以事实证明，智能不是一个频谱；有很多不同的专业领域，很多不同类型的技能。记忆是不同的。我的意思是，它们都在那个大块中形成，并不复杂，但即使它在一个频谱上，这个频谱也很宽。如果你 10 年前问我，我根本不会预料到这一点，但我认为事实就是这样。

If you look back, I can't remember who made these diagrams that are like, here's the village idiot, here's Einstein, here's the scale of intelligence. The village idiot and Einstein are very close to each other. Maybe that's still true in some abstract sense, but it's not really what we're seeing, is it? It seems like the human range is pretty broad and we don't hit the human range in the same place or at the same time for different tasks. Like, write a sonnet in the style of Cormac McCarthy—I'm not very creative, so I couldn't do that, but that's a pretty high-level human skill. And even the model is starting to get good at stuff like constrained writing: write a page without using the letter 'e' about X. I think the models might be superhuman or close to superhuman at that. But when it comes to proving relatively simple mathematical theorems, they're just starting to do the beginning of it; they make really dumb mistakes sometimes and they really lack any broad ability like correcting your errors or doing extended tasks. So it turns out that intelligence isn't a spectrum; there are a bunch of different areas of domain expertise, a bunch of different kinds of skills. Memory is different. I mean, it's all formed in the blob, it's not complicated, but to the extent it even is on a spectrum, the spectrum is also wide. If you asked me 10 years ago, that's not what I would have expected at all, but I think that's very much the way it's turned out.

哦，天哪，我有很多后续问题。一个是：考虑到这些模型从海量互联网数据中获得的训练分布与人类从进化中获得的相比，所引发的技能库会只是勉强重叠吗？会像同心圆吗？你怎么看？这重要吗？

Oh man, I have so many questions just as follow-up on that. One is: do you expect that given the distribution of training that these models get from massive amounts of internet data versus what humans got from evolution, the repertoire of skills that elicits will be just barely overlapping? Will it be like concentric circles? How do you think about that? Does it matter?

显然有大量的重叠，因为很多事物，比如这些模型有商业应用，它们的许多商业应用都在做帮助人类更有效地做事的事情。所以重叠相当大。如果你想到人类在互联网上用文本进行的所有活动，那覆盖了很多，但可能没有覆盖一些东西。我认为模型确实在一定程度上学习了世界的物理模型，但它们肯定没有学习如何在世界中实际移动。再说一次，也许这很容易微调，但我认为有些东西模型没有学到而人类学到了。然后我认为模型学会了，例如，说流利的 base64。

Clearly there's a large amount of overlap, because a lot of the things, like these models have business applications, and many of their business applications are doing things that help humans be more effective at things. So the overlap is quite large. If you think of all the activity that humans put on the internet in text, that covers a lot of it, but it probably doesn't cover some things. The models, I think, do learn a physical model of the world to some extent, but they certainly don't learn how to actually move around in the world. Again, maybe that's easy to fine-tune, but I think there are some things that the models don't learn that humans do. And then I think the models learn, for example, to speak fluent base64.

我不知道你怎么想，但我从来没学过这个。你认为这些模型在多年内在经济价值高的任务上超越人类，但在许多其他相关任务上仍低于人类，从而阻止了智能爆炸之类的可能性有多大？

I don't know about you but I never learned that right. How likely do you think it is that these models will be superhuman for many years at economically valuable tasks while they're still below humans in many other relevant tasks that prevents like an intelligence explosion or something?

我觉得这类事情真的很难知道，所以我先打个预防针。基本的缩放定律你可以预测，但我们真正想知道的更细粒度的事情要难得多。我的猜测是缩放定律会继续，取决于人们是否出于安全或监管原因放慢速度。但抛开这些，如果我们有经济能力继续扩展，我认为我们会全面进步。我没有看到任何领域模型特别弱或没有开始取得进展。过去数学和编程是这样，但在过去六个月里，2023 年的模型相比 2022 年的模型已经开始学习。可能还有一些更微妙的东西我们不知道，所以我怀疑即使不是完全均衡，水涨船高也会带动所有方面。

I think this kind of stuff is really hard to know, so I'll give that caveat. The basic scaling laws you can kind of predict, but the more granular stuff we really want to know is much harder. My guess is the scaling laws are going to continue, subject to people slowing down for safety or regulatory reasons. But putting that aside, if we have the economic capability to keep scaling, I think we're going to keep getting better across the board. I don't see any area where the models are super weak or not starting to make progress. That used to be true of math and programming, but over the last six months, the 2023 generation of models compared to the 2022 generation have started to learn. There may be more subtle things we don't know, so I suspect even if it isn't quite even, the rising tide will lift all the boats.

这包括你之前提到的，如果任务时间延长，它会失去思路或执行一系列任务的能力吗？

Does that include the thing you were mentioning earlier where if there's an extended task it kind of loses its train of thought or its ability to just execute a series?

我认为这取决于强化学习训练，让模型处理更长周期的任务。我不期望这需要大量额外的算力。我认为这可能是对强化学习思考方式错误，以及低估了模型自身学习能力的产物。至于在某些领域超人类而在其他领域不超人类，我认为这很复杂。我可以想象我们在某些领域不会超人类，因为这些领域涉及物理世界的具身。那么问题来了：AI 是否帮助我们训练更快的 AI，而这些更快的 AI 反过来解决这个问题？你是否不需要物理世界？这取决于你的意思。

I think that's going to depend on things like RL training to have the model do longer horizon tasks. I don't expect that to require a substantial amount of additional compute. I think that was probably an artifact of thinking about RL in the wrong way and underestimating how much the model had learned on its own. In terms of being superhuman in some areas and not others, I think it's complicated. I could imagine we won't be superhuman in some areas because they involve embodiment in the physical world. Then it's like, do the AIs help us train faster AIs, and those faster AIs wrap around and solve that? Do you not need the physical world? It depends what you mean.

我们担心的是对齐灾难吗？我们担心的是滥用，比如制造大规模杀伤性武器吗？我们担心 AI 接管人类的研究吗？我们担心它达到某个经济生产力的阈值，能完成普通人所做的工作吗？我认为这些不同的阈值有不同的答案，尽管我怀疑它们都会在几年内到来。让我问问这些阈值。如果 Claude 是 Anthropic 的员工，它值多少薪水？它如何有意义地加速 AI 进展？

Are we worried about an alignment disaster? Are we worried about misuse like making weapons of mass destruction? Are we worried about the AI taking over research from humans? Are we worried about it reaching some threshold of economic productivity where it can do what the average person does? These different thresholds I think have different answers, although I suspect they will all come within a few years. Let me ask about those thresholds. If Claude was an employee at Anthropic, what salary would it be worth? What is it like meaningfully speeding up AI progress?

在我看来，它在大多数领域像个实习生，但在某些特定领域比那更好。比较困难的一点是形态与人类不同。人类有长期生活经验的概念，而聊天机器人没有。所以缺少很多东西，很难比较。但在某些领域它像实习生，在其他领域它像专家，可能比这里的任何人都好。

It feels to me like an intern in most areas, but then some specific areas where it's better than that. One thing that makes the comparison hard is the form factor is not the same as a human. A human has a concept of a long life of prior experience, and chatbots don't have that. So there's all kinds of stuff missing, making it hard to compare. But it feels like interns in some areas and then savants in others where they may be better than anyone here.

智能爆炸这样的整体图景对你来说有意义吗？我之前的嘉宾 Carl Shulman 有一个非常详细的成人模型。作为一个真正看到这种情况发生的人，这有意义吗？当他们从实习生变成初级软件工程师时？那些初级软件工程师提高了你的生产力。

Does the overall picture of something like an intelligence explosion make sense to you? My former guest is Carl Shulman and he has this very detailed model of an adult. Does that as somebody who would actually see that happening? Does that make sense to you as they go from interns to entry level software engineers? Those entry level software engineers increase your productivity.

我认为 AI 系统变得更高效，先是加速人类生产力，然后与之持平，最后成为科学进步的主要贡献者，这个基本逻辑在我看来是可能的。尽管我怀疑当我们深入细节时，事情会变得奇怪且出乎意料。所有详细的模型都在思考错误的事情，或者我们在一件事上对了，却在其他十件事上错了。我认为当把所有因素加起来时，我们可能会进入一个比预期更奇怪的世界。

I think the idea that AI systems become more productive, first speeding up human productivity, then equaling it, and then being the main contributor to scientific progress, that basic logic seems likely to me. Although I have a suspicion that when we go into the details, it's going to be weird and different than we expect. All the detailed models are thinking about the wrong things, or we're right about one thing and wrong about ten others. I think we might end up in a weirder world than we expect when you add it all together.

你估计我们什么时候能得到某种人类水平的东西？那会是什么样子？

Your estimate of when we get something kind of human level? What does that look like?

同样，这取决于阈值。就某人看着模型并与它交谈一个小时左右而言，它基本上就像一个受过良好教育的普通人。这可能并不遥远。我认为这可能在两三年内发生。阻止它的主要因素是我们达到某些安全阈值，行业决定放慢速度，或者政府出于安全原因实施限制来减缓进展。但如果只看物流和经济上的扩展能力，我认为我们离那并不远。那可能不是模型存在危险性的阈值，也不是它们能接管大多数 AI 研究或严重改变经济运作方式的阈值。我认为之后会变得模糊，所有这些阈值可能在不同时间点出现。但就基础技术能力而言，它听起来像一个相当全面的受过良好教育的普通人。我认为那可能非常接近。

Again, it depends on the thresholds. In terms of someone looking at the model and talking to it for an hour or so, it's basically like a generally well-educated human. That could be not very far away at all. I think that could happen in two or three years. The main thing that would stop it would be if we hit certain safety thresholds and the industry decides to slow down, or we get government restrictions to moderate progress for safety reasons. But if you just look at the logistical and economic ability to scale, I don't think we're very far from that now. That may not be the threshold where models are existentially dangerous, or where they can take over most AI research, or seriously change how the economy works. I think it gets murky after that, and all those thresholds may happen at various times after that. But in terms of base technical capability, it sounds like a reasonably generally educated human across the board. I think that could be quite close.

为什么它可能听起来能通过一个受过教育的人的图灵测试，但却无法贡献或替代人类在经济中的参与？

Why would it be the case that it could sound like it passes a Turing test for an educated person but not be able to contribute or substitute for human involvement in the economy?

有几个原因。一个是……

A couple reasons. One is just...

你知道技能门槛还不够高，对吧？比较优势：你有一个在每个任务上都比普通人强的人，这并不重要。对于 AI 研究，我真正需要的是找到足够强大的东西，能大幅加速最顶尖的千名专家的劳动。所以我们可能会达到一个点，这些系统的比较优势并不大。另一种可能的情况是，存在一些神秘的摩擦，它们不会出现在天真的经济模型中，但当你去找客户说‘嘿，我有一个很酷的聊天机器人，原则上它能做你客服机器人或你公司这个部门能做的一切’时，你就会看到这些摩擦。但实际摩擦——如何把它嵌入、如何让它工作——既包括它在公司内部如何以人类的方式运作（经济中事情如何发生、如何克服摩擦），也包括工作流是什么、你如何实际与它交互。说‘这里有一个聊天机器人，看起来在做这个任务或帮助人类做某些任务’与说‘这个东西已经部署了，10 万人在使用它’是非常不同的。现在很多人都在急于部署这些系统，但我认为在很多情况下，他们并没有以最有效的方式使用它们，不是因为他们不聪明，而是因为需要时间来摸索。所以当事情变化这么快时，所有这些摩擦都会存在。我认为这些是模型无法完全捕捉的混乱现实。我不认为这改变了基本图景——我不认为它改变了我们正在构建一个雪球的想法：模型帮助模型变得更好，加速人类的工作，最终主要是模型在做工作。如果你看得足够远，这正在发生。但我对任何精确的数学或指数预测持怀疑态度。我认为这一切都会很混乱，但我们知道的是，这不是一个比喻性的指数，而且它会很快发生。

You know that the threshold of skill isn't high enough, right? Comparative advantage: it doesn't matter that you have someone who's better than the average human at every task. What I really need for AI research is to find something strong enough to substantially accelerate the labor of the thousand experts who are best at it. So we might reach a point where the comparative advantage of these systems is not great. Another thing that could be the case is that there are these mysterious frictions that don't show up in naive economic models, but you see them whenever you go to a customer and say, 'Hey, I have this cool chatbot; in principle it can do everything that your customer service bot does or that this part of your company does.' But the actual friction of how do we slot it in, how do we make it work, includes both the question of how it works in a human sense within the company—how things happen in the economy and overcome frictions—and also just what is the workflow, how do you actually interact with it. It's very different to say, 'Here's a chatbot that looks like it's doing this task or helping the human to do some tasks,' as it is to say, 'Okay, this thing is deployed, 100,000 people are using it.' Right now lots of folks are rushing to deploy these systems, but I think in many cases they're not using them in anywhere close to the most efficient way they could, not because they're not smart, but because it takes time to work these things out. So when things are changing this fast, there are all of these frictions. I think again, these are messy realities that don't quite get captured in the model. I don't think it changes the basic picture—I don't think it changes the idea that we're building up this snowball of models helping models get better, accelerating what humans do, and eventually it's mostly the models doing the work. If you zoom out far enough, that's happening. But I'm kind of skeptical of any kind of precise mathematical or exponential prediction of how it's going to be. I think it's all going to be a mess, but what we know is it's not a metaphorical exponential, and it's going to happen fast.

这些不同的指数如何抵消？一个是缩放定律本身是幂律，每个参数的边际损失递减。你谈到的另一个指数是这些东西可以参与到 AI 研究过程中，加速它。所以这两个是相反的指数。最终结果是超线性还是次线性？你还提到智能的分布可能更广。那么，在两三年后我们达到这个点时，应该期待什么样子？

How do those different exponentials net out? One was the scaling laws themselves are power laws with decaying marginal loss per parameter. The other exponential you talked about is that these things can get involved in the process of AI research itself, speeding it up. So those two are sort of opposing exponentials. Does it net out to be super linear or sublinear? And you also mentioned that the distribution of intelligence might just be broader. So should we expect after we get to this point in two to three years, what does that look like?

我认为非常不清楚。我们已经到了这样一个点：如果你看损失，缩放定律开始弯曲。我们在多家公司发布的模型卡中看到了这一点，所以这根本不是秘密。但随着它们开始弯曲，每一小点熵——即准确预测——都变得更加重要。也许最后这些小点熵就像‘这是一篇爱因斯坦会写的物理学论文，而不是其他物理学家会写的’。所以很难从中评估重要性。从实际性能来看，指标似乎在相对线性地上升。而且，它们一直不可预测，所以很难看出来。然后我认为推动最大加速的因素是越来越多的资金涌入这个领域。人们看到了巨大的经济价值。所以我预计花在最大模型上的资金会增加 100 倍左右，并且这与芯片变快、算法变好相结合，因为现在有这么多人在做这个。再次强调，我不是在做一个规范性陈述——这是应该发生的事情。我甚至不是说这一定会发生，因为我认为这里有重要的安全和政府问题，我们正在非常积极地处理。我只是说，如果任其发展，这就是经济会做的事情。

I think it's very unclear. We're already at the point where if you look at the loss, the scaling laws are starting to bend. We've seen that in published model cards offered by multiple companies, so that's not a secret at all. But as they start to bend, each little bit of entropy—of accurate prediction—becomes more important. Maybe these last little bits of entropy are like, 'Well, this is a physics paper as Einstein would have written it, as opposed to as some other physicist would have written it.' So it's hard to assess significance from this. It certainly looks like in terms of practical performance, the metrics keep going up relatively linearly. Also, they were always unpredictable, so it's hard to see that. And then the thing that I think is driving the most acceleration is just more and more money is going into the field. People are seeing that there's a huge amount of economic value. So I expect the amount of money spent on the largest models to go up by a factor of 100 or something, and for that to then be concatenated with chips getting faster and algorithms getting better because there are so many people working on this now. Again, I'm not making a normative statement here—this is what should happen. I'm not even saying this necessarily will happen because I think there are important safety and government questions here which we're very actively working on. I'm just saying, left to itself, this is what the economy is going to do.

你如何看待 Anthropic 对这个行业范围扩大的贡献？有一种观点认为，有了这些投资，我们可以在 Anthropic 做安全方面的工作；另一种观点说你是在提高这个领域的整体显著性。

How do you think about the contribution of Anthropic to that increasing scope of this industry? There's an argument that with that investment we can work on safety stuff at Anthropic, and another that says you're raising the salience of this field in general.

这全是成本和收益，对吧？成本不是零。我认为思考这些问题的成熟方式不是否认有任何成本，而是思考成本是什么、收益是什么。我认为我们相对负责任，因为去年年底和今年年初发生的大加速——我们并没有导致它。我们不是做那件事的人。老实说，我认为如果你看对谷歌的反应，那可能比其他任何事情重要 10 倍。然后一旦它发生了，一旦生态系统改变了，我们做了很多事情来保持在前沿。所以我不知道。这就像任何其他问题：你试图做那些成本最大、成本最小、收益最大的事情。这导致你在不同时间有不同的策略。

It's all costs and benefits, right? The costs are not zero. I think a mature way to think about these things is not to deny that there are any costs, but to think about what the costs are and what the benefits are. I think we've been relatively responsible in the sense that the big acceleration that happened late last year and the beginning of this year—we didn't cause that. We weren't the ones who did that. Honestly, I think if you look at the reaction to Google, that might be 10 times more important than anything else. And then once it had happened, once the ecosystem had changed, we did a lot of things to stay on the frontier. So I don't know. It's like any other question: you're trying to do the things that have the biggest costs and the lowest costs and the biggest benefits. That causes you to have different strategies at different times.

在我们谈论智能问题时，我有一个问题：作为一个科学家，你怎么看待这样一个事实：这些东西基本上记住了整个人类知识库，但据我所知，它们还没有能够做出一个导致发现的新联系？而如果一个中等智力的人记住了这么多东西，他们会注意到，‘哦，这个东西导致这个症状，那个东西也导致这个症状，这里有一个医疗方法。’我们应该期待那种情况吗？

One question I have for you while we were talking about the intelligence stuff: as a scientist yourself, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized, and as far as I'm aware, they haven't been able to make a single new connection that has led to a discovery? Whereas if even a moderately intelligent person had this much stuff memorized, they'd notice, 'Oh, this thing causes this symptom, this other thing also causes the symptom, there's a medical cure right here.' What should we be expecting that kind of?

我不太确定“发现”和“创造力”这些词。我学到的一点是，在庞大的算力中，这些想法往往模糊而难以捉摸。但我认为模型确实展现出一种普通的创造力——比如用科马克·麦卡锡或芭比的风格写作。它们能像普通人一样建立新的联系。我同意目前还没有重大的科学发现。部分原因是模型的技能水平还不够高。上周我参加了一个播客，主持人说这些模型表现一般——大概能得 B 或 B-。随着 Scaling（规模扩张），这种情况会改变。模型有一个优势：它们比我们知道得多得多。也许即使技能水平不够高，它们也应该已经具备优势了。我没有答案。记忆和建立联系是模型领先的领域。你需要这些联系和相当高的技能水平。在生物学中，复杂性使得当前模型知道很多东西，而这正是做出发现所需要的。不像物理学，你需要思考并想出公式；生物学中你需要知道很多东西。模型知道很多，但技能水平还不足以将它们整合起来。它们正处于能够做到这一点的边缘。

I'm not sure about these words discovery and creativity. One lesson I've learned is that in the big blob of compute, these ideas often end up being fuzzy and elusive. But I think the models do display a kind of ordinary creativity—like writing in the style of Cormac McCarthy or Barbie. They draw new connections that an ordinary person would draw. I agree there haven't been any big scientific discoveries. That's partly because the model skill level isn't high enough yet. I was on a podcast last week where the host said the models are kind of mid—they get a B or B-minus. That's going to change with scaling. Models have an advantage: they know a lot more than us. Maybe they should already have an advantage even if their skill level isn't quite high. I don't have an answer. Memorization and drawing connections are areas where models are ahead. You need those connections and a fairly high skill level. In biology, the complexity is such that current models know a lot of things, and that's what you need to make discoveries. It's not like physics where you need to think and come up with a formula; in biology you need to know a lot of things. Models know a lot, but their skill level isn't quite high enough to put them together. They are on the cusp of being able to do that.

上周你在参议院作证时说，这些模型可能还有两到三年就能促成大规模生物恐怖袭击。你能说得更具体些吗，同时不提供危险信息？是关于一次性学会如何武器化某物，还是使用开源模型？实际会是什么样子？

Last week in your Senate testimony, you said these models are two to three years away from potentially enabling large-scale bioterrorism attacks. Can you make that more concrete without giving dangerous information? Is it about one-shotting how to weaponize something, or using an open-source model? What would that actually look like?

我觉得有必要澄清这一点。我们发了一篇博文，也在参议院作证了，但很多人没理解要点。目前，在我们的模型中，我们试图防止这种情况，但总有越狱方法。你可以问模型各种生物学问题，让它们说出可怕的东西。但那些可怕的东西往往是你用谷歌就能查到的，所以我并不特别担心。这实际上阻碍了人们看到真正的危险。所以我们花了六个月时间，与一些世界上最顶尖的生物攻击专家合作，研究生物攻击是如何发生的——你需要什么来实施这样的攻击，以及如何防御。他们深入研究了整个工作流程。这不是一次性的事情；这是一个漫长的过程，有很多步骤。不仅仅是向模型要一页信息。不深入细节，我在参议院作证时说：有些步骤你可以从谷歌获取信息；有些步骤是缺失的——分散在教科书里，或者根本不在任何教科书里，是隐性知识，不是显性知识。比如实验室规程：‘如果我搞错了，哦，如果发生这种情况，那我的温度太低了；如果发生那种情况，我需要多加这种试剂。’我们发现，大多数情况下，模型还无法完成那些关键缺失步骤。但有时它们能做到，而当它们能做到时，它们仍然会幻觉，这让我们保持安全。但我们看到了足够多的迹象，表明模型能很好地完成那些关键任务。如果我们看看最先进的模型，再回溯到之前的模型，趋势显示出所有迹象表明，两三年后我们将面临真正的问题。

I think it's good to clarify this. We did a blog post and the Senate testimony, and I think various people didn't understand the point. Today, in our models, we try to prevent this, but there are always jailbreaks. You can ask models all kinds of things about biology and get them to say scary things. But often those scary things are things you could Google, so I'm not particularly worried about that. It's actually an impediment to seeing the real danger. So we spent six months working with some of the world's most expert folks on how biological attacks happen—what you would need to conduct such an attack and how to defend against it. They worked intensively on the entire workflow. It's not one shot; it's a long process with many steps. It's not just asking the model for one page of information. Without going into detail, what I said in the Senate testimony is: there are some steps where you can get information on Google; there are some steps that are missing—scattered across textbooks or not in any textbook, implicit knowledge, not explicit. Like lab protocols: 'If I get it wrong, oh if this happens then my temperature was too low, if that happened I needed to add more of this reagent.' We found that for the most part, the models can't do those missing key pieces yet. But sometimes they can, and when they can, they still hallucinate, which is keeping us safe. But we saw enough signs of models doing those key things well. If we look at state-of-the-art models and go backwards to previous models, the trend shows every sign that in two or three years we're going to have a real problem.

特别是你提到的对数尺度：从百分之一的正确率到十分之一。

Especially the thing you mentioned about the log scale: you go from like 1 in 100 times it gets it right to 1 in 10.

没错。我见过很多顿悟时刻。我见证了 GPT-3 学会算术，GPT-2 学会回归（略高于随机水平），Claude 在有用性、诚实性、无害性测试上变得更好。我见过很多顿悟。不幸的是，这不是一个让我兴奋的顿悟，但我相信它正在发生。

Exactly. I've seen many grokking moments in my life. I was there when GPT-3 learned to do arithmetic, when GPT-2 learned regression a bit above chance, when Claude got better on tests of helpfulness, honesty, harmlessness. I've seen a lot of groks. This is unfortunately not one I'm excited about, but I believe it's happening.

有人可能会说，你是 OpenAI 关于 GPT-2 那篇博文的合著者，文中说不会发布权重，因为担心滥用。现在回头看，认为 GPT-2 能做什么坏事是可笑的。我们是不是过于担心了？这种担忧没有道理吗？

Some might say you were a co-author on the OpenAI post about GPT-2 where you said you wouldn't release the weights because you were worried about misuse. Looking back, it's laughable to think GPT-2 could have done anything bad. Are we just too worried? Is this a concern that doesn't make sense?

这很有趣。也许值得回顾一下那篇博文的原文。我不记得确切内容了，但它还在网上。大意是：‘我们选择不发布权重，因为担心滥用，但这是一个实验。我们不确定这是否必要或正确，但我们希望建立一种仔细思考这些问题的规范。’你可以把它想象成 1970 年代的阿西洛马会议，当时他们在研究重组 DNA。并不是说有人能用重组 DNA 做真正坏的事情；只是可能性开始变得清晰。至少那些话是对的。

It is interesting. It might be worth looking back at the actual text of that post. I don't remember it exactly, but it's still up on the internet. It says something like: 'We're choosing not to release the weights because of concerns about misuse, but this is an experiment. We're not sure if this is necessary or the right thing to do at this time, but we'd like to establish a norm of thinking carefully about these things.' You could think of it a little like the Asilomar Conference in the 1970s, where they were figuring out recombinant DNA. It wasn't necessarily the case that someone could do something really bad with recombinant DNA; it's just that the possibilities were starting to become clear. Those words at least were the right ones.

态度上，我认为还有另一件事：人们不仅评判帖子本身，还评判发布机构。这个机构是那种制造大量炒作的组织，还是有信誉的组织？我认为这对结果有影响。你或许还会问：人们是否不可避免地会将其解读为‘你无法传达比‘这东西很危险’更复杂的信息’？你可以争论这些，但我以及参与其中的其他人脑子里最根本的想法是：我们其实不知道。对于什么危险、什么不危险，我们的误差范围相当大，所以我们应该建立一种谨慎行事的规范。顺便说一句，我们现在有了多得多的证据，见识了多得多的这类‘grok’，所以校准得不错，但不确定性依然存在。在我所有的声明中，我都说过，比如两三年后我们可能就达到那个点，风险很大，我们不想冒这个险，但我不会说这是 100%，可能是 50 对 50。

Attitude now I think there's a separate thing that people don't just judge the post, they judge the organization. Is this an organization that produces a lot of hype or has credibility? I think that had some effect on it. I guess you could also ask: is it inevitable that people would just interpret it as 'you can't get across any message more complicated than this thing right here is dangerous'? So you can argue about those, but the basic thing that was in my head and the heads of others involved is that we actually don't know. We have pretty wide error bars on what's dangerous and what's not, so we should establish a norm of being careful. By the way, we have enormously more evidence now, we've seen enormously more of these groks, so we're well calibrated, but there's still uncertainty. In all these statements I've said, like in two or three years we might be there, there's a substantial risk, and we don't want to take that risk, but I wouldn't say it's 100%, it could be 50-50.

我们来谈谈网络安全，除了生物风险，这也是 Anthropic 一直强调的另一个方面。你们是如何避免云端微架构泄露的？因为如你所知，你们的竞争对手在这类安全问题上做得没那么成功。

Let's talk about cybersecurity, which in addition to biorisk is another thing Anthropic has been emphasizing. How have you avoided the cloud micro-architecture from leaking? Because as you know, your competitors have been less successful at this kind of security.

我无法评论其他公司的安全状况，不知道他们内部的情况。我们做的一件事是：有一些架构创新能让训练更高效，我们称之为算力倍增器，因为它们相当于拥有更多算力。同样，我不想多说，因为这可能让对手针对我们的措施。我们只让需要知道的人了解某个算力倍增器。所以，能泄露所有这些秘密的人非常少，能泄露其中之一的稍多一些。这是情报界或抵抗组织使用的标准隔离策略。过去几个月我们实施了这些措施，我不想说不吉利的话，但我认为泄露更难发生了。我不想深入细节。顺便说一句，我鼓励所有其他公司也这样做。竞争对手的架构泄露虽然短期内对 Anthropic 有利，但长期对谁都没好处。这方面的安全非常重要。

Can't comment on anyone else's security, don't know what's going on there. A thing that we have done is there are these architectural innovations that make training more efficient; we call them compute multipliers because they're the equivalent of having more compute. Again, I don't want to say too much about it because it could allow an adversary to counteract our measures. But we limit the number of people who are aware of a given compute multiplier to those who need to know about it. So there's a very small number of people who could leak all of these secrets, a larger number who could leak one of them. This is the standard compartmentalization strategy used in the intelligence community or resistance cells. Over the last few months we've implemented these measures, so I don't want to jinx anything, but I think it would be harder for it to happen. I don't want to go into any more detail. By the way, I'd encourage all the other companies to do this as well. As much as competitors' architectures leaking is narrowly helpful to Anthropic, it's not good for anyone in the long run. Security around this stuff is really important.

即使有你们现有的所有安全措施，你们能防止一个专注的国家级行为者获取 Claude 2 的权重吗？

Even with all the security you have, could you with your current security prevent a dedicated state-level actor from getting the Claude 2 weights?

这取决于他们有多专注。我们的安全主管以前负责 Chrome 的安全，他喜欢从成功攻击 Anthropic 需要花费多少成本的角度来思考。我再次不想深入细节，因为那有点像在招引别人。但我们的目标之一是：攻击 Anthropic 的成本要高于自己训练一个模型。这并不能保证万无一失，因为还需要人才，但攻击有风险、外交成本，而且会消耗国家行为者可能拥有的非常稀缺的资源。我们还没达到那个水平，但相对于我们公司的规模，我认为标准已经很高了。如果你看大多数 150 人公司的安全状况，根本没有可比性。但如果窃取我们的模型权重是一个国家行为者的最高优先级，我们能抵抗吗？不能，他们会成功。

It depends how dedicated. Our head of security, who used to work on security for Chrome, likes to think about it in terms of how much it would cost to attack Anthropic successfully. I again don't want to go into super detail of how much I think it would cost, because it's kind of inviting people. But one of our goals is that it costs more to attack Anthropic than to just train your own model. That doesn't guarantee things because you need the talent as well, but attacks have risks, diplomatic costs, and they use up the very sparse resources that nation-state actors might have. We're not there yet, but I think we're to a very high standard compared to the size of company we are. If you look at security for most 150-person companies, there's just no comparison. But could we resist if it was a state actor's top priority to steal our model weights? No, they would succeed.

这个情况能持续多久？因为价值在不断增加。这个问题的另一部分是：训练 Claude 到 Claude 2 的方法是什么样的秘密？比如核武器，我们有很多间谍；你只要把蓝图带过去，那就是内爆装置。是像那样吗？还是更隐性，比如生物学，你需要知道这些试剂如何工作？仅仅是蓝图、微架构和超参数吗？

How long does that stay true? Because at some point the value keeps increasing. Another part of this question is: what kind of a secret is how to train Claude through to Claude 2? With nuclear weapons, for example, we had lots of spies; you just take a blueprint across and that's the implosion device. Is it just like that? Or is it more tacit, like biology, where you need to know how these reagents work? Is it just the blueprint, the micro-architecture and hyperparameters?

有些东西就像一行方程式，其他东西则更复杂。隔离是最好的方法：限制知道某件事的人数。如果你是一个一千人的公司，每个人都知道所有秘密，第一，我保证你有泄密者，第二，我保证你有间谍，真正的间谍。

There are some things that are like a one-line equation, and there are other things that are more complicated. Compartmentalization is the best way to do it: just limit the number of people who know about something. If you're a thousand-person company and everyone knows every secret, one, I guarantee you have a leaker, and two, I guarantee you have a spy, a literal spy.

我们来谈谈对齐和机制可解释性，这是你们专攻的分支。回答时，你可能想解释一下什么是机制可解释性。更广泛的问题是：从机制上讲，对齐是什么？是把模型锁定在一个仁慈的角色上吗？你是在禁用欺骗性电路和程序吗？当你对齐一个模型时，具体发生了什么？

Let's talk about alignment and mechanistic interpretability, which is the branch you guys specialize in. While answering, you might want to explain what mechanistic interpretability is. The broader question is: mechanistically, what is alignment? Is it locking in the model into a benevolent character? Are you disabling deceptive circuits and procedures? What concretely is happening when you align a model?

和大多数事情一样，当我们实际训练一个模型使其对齐时，我们不知道模型内部发生了什么。有不同的训练方法使其对齐，但我认为我们并不真正知道发生了什么。对于当前的一些方法，所有涉及某种微调的方法都有一个特点：我们可能担心的底层知识和能力并不会消失。模型只是被训练不输出它们。我不知道这是否是一个致命缺陷，或者事情本就如此。我不知道内部机制上发生了什么，我认为这正是机制可解释性的意义所在：真正理解模型内部在单个电路层面发生了什么。最终，当它成熟时，解决方案是什么样的？当你对 Claude 4 进行机制可解释性分析后，说‘我满意它是对齐的’，你看到了什么？我认为我们还不知道。

As with most things, when we actually train a model to be aligned, we don't know what happens inside the model. There are different ways of training it to be aligned, but I think we don't really know what happens. For some of the current methods, all the current methods that involve some kind of fine-tuning have the property that the underlying knowledge and abilities that we might be worried about don't disappear. The model is just taught not to output them. I don't know if that's a fatal flaw or if that's just the way things have to be. I don't know what's going on inside mechanistically, and I think that's the whole point of mechanistic interpretability: to really understand what's going on inside the models at the level of individual circuits. Eventually, when it's solid, what does a solution look like? What is the case where if you do the mechanistic interpretability thing on Claude 4 and you're like 'I'm satisfied it's aligned'? What is it that you've seen? I think we don't know that yet.

我还没到能完全理解的程度。我可以描述一下过程大概是什么样，而不是最终结果。我认为可验证性是这里的一大挑战。我们有各种声称能对齐 AI 系统的方法，而且对今天的任务确实有效，但问题始终是：如果你加入一个更强大的模型，或者把模型放在不同的情境中，它还会对齐吗？如果有一个神谕能扫描模型并说'我知道这个模型是对齐的，我知道它在每种情况下会做什么'，那问题就容易多了。我们最接近这个的是机制可解释性，但它还远未达到要求。我把它几乎看作扩展的训练集和扩展的测试集。我们做的一切——所有对齐方法——都是训练集。你可以运行测试，但它真的能在分布外工作吗？能在另一种情境中工作吗？机制可解释性是唯一在原则上更像模型的 X 光而非修改的东西。它更像评估而非干预。我们需要一种动态：有扩展的训练集（所有对齐方法）和扩展的测试集（对模型做 X 光，看看什么有效什么无效，超越你运行过的经验测试）。我们需要问模型在这些情境中会做什么，它有能力做什么，而不是仅仅它现象上做了什么。我们必须小心：我们永远不应该为了可解释性而训练，因为那会失去优势。这就像验证集与测试集——如果你看 X 光太多次，你会干扰。但这是一个弱得多的过程，不是自动优化。我们只需确保在运行测试集之前不要看验证集太多次。那是手动压力，不是自动的。所以某种解决方案是训练集和测试集之间的动态——尝试事物并通过测试来确认它们是否有效，而模型不会以正交方式优化。我认为我们永远不会有保证，但一个过程是把对齐能力的扩展训练与对齐能力的扩展测试结合起来，以实际有效的方式。

I don't know enough to know that yet. I can give you a sketch of what the process looks like as opposed to the final result. I think verifiability is a lot of the challenge here. We have all these methods that purport to align AI systems and succeed for today's tasks, but the question is always: if you add a more powerful model or put a model in a different situation, would it be aligned? This problem would be much easier if you had an oracle that could scan a model and say, 'I know this model is aligned; I know what it'll do in every situation.' The closest thing we have to that is mechanistic interpretability, but it's nowhere near up to the task yet. I think of it as almost like an extended training set and an extended test set. Everything we're doing—all the alignment methods—are the training set. You can run tests, but will it really work out of distribution? Will it work in another situation? Mechanistic interpretability is the only thing that, even in principle, is more like an X-ray of the model than a modification. It's more an assessment than an intervention. We need a dynamic where we have an extended training set (all these alignment methods) and an extended test set (X-raying the model to see what worked and what didn't, beyond just empirical tests). We need to ask what the model will do in these situations, what it's capable of, rather than just what it did phenomenologically. We have to be careful: we should never train for interpretability because that takes away the advantage. It's like validation vs. test set—if you look at the X-ray too many times, you can interfere. But that's a much weaker process, not automated optimization. We should just ensure we don't look at the validation set too many times before running the test set. That's manual pressure, not automated. So some solution where we have a dynamic between training and test set—trying things out and figuring out if they work via testing that the model isn't optimizing against orthogonally. I think we'll never have a guarantee, but a process where we put extended training for alignment ability together with extended testing for alignment ability in a way that actually works.

我还是不明白你为什么认为这很可能成功或值得追求。让我更具体地问，请原谅这个牵强的类比。如果你是一个经济学家，想理解经济，你派出一堆微观经济学家。一个研究餐饮业，一个研究旅游业，一个研究烘焙业。最后他们聚在一起，你仍然不知道五年后会不会有衰退。为什么这不一样？你理解了两层 Transformer 中的归纳头如何工作，你理解了模算术。这些如何加起来回答'这个模型想杀了我们吗？'这个模型根本想要什么？

I still don't feel like I understand the intuition that why you think this is likely to work or promising to pursue. Let me ask in a more specific way, and excuse the tortured analogy. If you're an economist and want to understand the economy, you send microeconomists out. One studies the restaurant business, one studies tourism, one studies baking. At the end, they all come together, and you still don't know if there will be a recession in five years. Why is this not like that? You understand how induction heads work in a two-layer Transformer, you understand modular arithmetic. How does this add up to 'does this model want to kill us?' What does this model fundamentally want?

我认为这些都是正确的问题。我们最终希望的并不是理解每一个细节，而是像 X 光或 MRI 类比那样，我们能够观察模型的宏观特征，并问：这个模型的内部状态和计划是否与其外部表现截然不同？这个模型是否让我们不安，因为它太多的计算能力被用于看起来相当破坏性和操纵性的事情？我们不确定这是否可能，但有一些积极迹象。模型并不是故意躲着你。训练过程可能把它藏起来了。我可以想象模型超级智能，以影响自身认知的方式思考。我们应该考虑一切。我怀疑，粗略地说，如果模型以正常方式训练到略高于人类水平，那么假设其内部结构并非故意针对我们优化可能是合理的。一个类比：通过看某人的 MRI，可以高于随机概率预测他们是否是精神病患者。有个故事，一位神经科学家研究这个，看了自己的扫描，发现自己是精神病患者。他生活中的每个人都说：'这很明显，你完全是个精神病患者。'他自己完全不知道。基本思想是存在宏观特征——精神病患者是一个很好的类比。这就是我们害怕的：一个表面迷人、目标导向、内心阴暗的模型。表面上它的行为可能看起来像别人，但目标非常不同。

I think those are the right questions. What we're hoping for in the end is not that we understand every detail, but again, like the X-ray or MRI analogy, we can be in a position to look at the broad features of the model and say: is this a model whose internal state and plans are very different from what it externally represents itself to do? Is this a model where we're uncomfortable that too much of its computational power is devoted to doing what looks like fairly destructive and manipulative things? We don't know for sure if that's possible, but there are positive signs. The model is not intentionally hiding from you. It might turn out that the training process hides it from you. I can think of cases where the model is super intelligent and thinks in a way that affects its own cognition. We should consider everything. I suspect it may roughly work to think of the model, if trained normally just above human level, that it may be a reasonable assumption that the internal structure is not intentionally optimizing against us. An analogy: it's possible to look at an MRI of someone and predict above random chance whether they're a psychopath. There was a story about a neuroscientist studying this, and they looked at his own scan and discovered he was a psychopath. Everyone in his life said, 'That's obvious, you're a complete psychopath.' He was totally unaware. The basic idea is that there can be macro features—psychopath is a good analogy. That's what we'd be afraid of: a model that's charming on the surface, very goal-oriented, and very dark on the inside. On the surface, its behavior might look like someone else's, but its goals are very different.

有人可能会问：你提到了经验主义的重要性。在这种情况下，你试图估计这些激活是否可疑。但这是我们可以经验主义地处理的事情吗，还是我们需要一个非常好的第一性原理？

A question somebody might have: you mentioned the importance of being empirical. In this case, you're trying to estimate if these activations are suspicious. But is this something we can be empirical about, or do we need a very good first principle?

从理论上看，不，这不仅仅是模型的 MRI 扫描与不良表现相关。我们需要某种深层的数学证明来确保对齐。所以这取决于你所说的“经验”是什么意思。更好的说法是“现象学”。我认为我们不应该纯粹依赖现象学，比如这里有危险模型的脑部扫描，那里有安全模型的脑部扫描。我认为机械可解释性的整个理念是去研究底层的原理和电路。

Theoretical reason to think no, it's not just that these MRIs of the model correlate with being bad. We need some deep root math proof that this is aligned. So it depends what you mean by empirical. A better term would be phenomenological. I don't think we should be purely phenomenological, like here are some brain scans of dangerous models and here are some brain scans. I think the whole idea of mechanistic interpretability is to look at the underlying principles and circuits.

我想我的看法是，一方面，我一直热衷于尽可能在最低细节层面研究这些电路。原因在于，这是积累知识的方式。即使最终目标是构建广泛的理解，但特征太多，太复杂了。归根结底，我们试图建立广泛的理解。我认为构建这种理解的方式是做出许多非常具体的发现。你必须理解基本构件，然后想办法利用这些来得出广泛的结论，即使你无法弄清一切。你知道，你应该去和 Chris Olah 谈谈，他会提供更多细节。这是我的高层想法。Chris Olah 掌控着可解释性的议程；他是决定可解释性工作方向的人。我的高层想法不如他的好。

I guess the way I think about it is, on one hand, I've always been a fan of studying these circuits at the lowest level of detail that we possibly can. The reason for that is that's how you build up knowledge. Even if you're ultimately aiming for something broad, there are too many features, it's too complicated. At the end of the day, we're trying to build a broad understanding. I think the way you build that up is by making a lot of very specific discoveries. You have to understand the building blocks, and then figure out how to use that to draw broad conclusions, even if you're not going to figure out everything. You know, I think you should probably talk to Chris Olah, who would have much more detail. This is my high-level thinking on it. Chris Olah controls the interpretability agenda; he's the one who decides what to do on interpretability. This is my high-level thinking, which is not going to be as good as his.

Anthropic 是否依赖于机械可解释性对能力有帮助这一事实？

Does Anthropic rely on the fact that mechanistic interpretability is helpful for capabilities?

我完全不这么认为。我确实认为原则上机械可解释性可能对能力有帮助。如果是这样，我们可能出于各种原因选择不谈论它。在 Anthropic 成立时，我或我们任何人都没有想过这一点。我的意思是，我们把自己看作擅长扩展模型并在这些模型上做安全研究的人。我们认为我们拥有非常高的人才密度。我的观点一直是人才密度胜过人才规模。所以这更像是我们的核心论点。人才密度胜过人才规模。我不认为这依赖于某个特定的事情。其他人现在也开始做机械可解释性了，我很高兴他们这么做。这是我们变革理论的一部分：矛盾的是，让其他组织变得更像我们。

I don't think so at all. I do think in principle it's possible that mechanistic interpretability could be helpful with capabilities. We might for various reasons not choose to talk about it if that were the case. That wasn't something that I thought of or that any of us thought of at the time of Anthropic's founding. I mean, we thought of ourselves as people who are good at scaling models and good at doing safety on top of those models. We think that we have a very high talent density of folks who are good at that. My view has always been talent density beats talent mass. So that's more of our bull case. Talent density beats talent mass. I don't think it depends on some particular thing. Others are starting to do mechanistic interpretability now, and I'm very glad that they are. That was a part of our theory of change: paradoxically, to make other organizations more like us.

人才密度很重要，但 Anthropic 强调的另一件事是，你需要拥有前沿模型才能进行安全研究，当然也要实际运营一家公司。当前的前沿模型，有人可能猜测像 GPT-4 这样的模型成本大约在 1 亿美元左右。这个数量级大致没错。但两三年后，我们讨论的事情将涉及更多数量级。为了跟上这一点，如果安全需要前沿的算力，那么 Anthropic 与这些巨头竞争以保持相同规模的情况是怎样的？

Talent density is important, but another thing Anthropic has emphasized is that you need to have frontier models in order to do safety research, and of course actually be a company as well. The current frontier model is something somebody might guess like GPT-4 cost around $100 million or something like that. That general order of magnitude in very broad terms is not wrong. But two to three years from now, the kinds of things you're talking about, we're talking more and more orders of magnitude. To keep up with that, and if it's the case that safety requires compute on the frontier, what is the case in which Anthropic is competing with these leviathans to stay on that same scale?

我认为这是一个充满权衡的局面。这并不容易。我想回到之前的问题，也许我逐一回答。为什么安全如此依赖于规模？有些人不这么认为。但如果我们看看安全方法在实践中应用或奏效的地方，即使我们不认为它们普遍有效，我会想到像辩论和放大这样的想法。早在 2018 年，当我们在 OpenAI 撰写关于这些的论文时，情况是，人类反馈不太行得通，但辩论和放大会带我们超越。但如果你实际尝试辩论，我们受到模型质量的严重限制。要让两个模型进行足够连贯的辩论，使人类能够评判，从而训练过程实际有效，你需要模型至少在某些主题上达到甚至超越当前的前沿。你可以提出方法，提出想法而不必处于前沿，但对我来说，这只是需要完成的工作中很小的一部分。提出这些方法很容易，比如哦问题是 X，也许解决方案是 Y。但我真的想知道事情在实践中是否有效，即使是对我们今天的系统，我想知道它们会出什么问题。我只是觉得通过在实践中尝试，你会发现 10 个新想法和 10 种新出错方式。这种经验学习并没有得到应有的广泛理解。对于像宪法 AI 这样的方法，我也会说同样的话。有些人说这无关紧要，我们知道这个方法不行，它不会用于纯粹的对齐。我既不同意也不反对。我认为这有点过于自信。我们发现新事物并理解什么有效什么无效的结构的方式是通过实际操作。不是盲目地说哦这在这里有效所以它在那里也有效，而是你真正开始理解模式。比如缩放定律，甚至机械可解释性，这可能是没有前沿模型也取得很多进展的一个领域。我们看到 OpenAI 几个月前发布的工作，使用非常强大的模型帮助自动解释弱模型。同样，这不是可解释性中你能做的全部，但它是很大一部分，我们也发现它很有用。所以你一次又一次地看到这种现象，缩放和安全就像两条缠绕在一起的蛇，总是比你想象的更紧密。对于可解释性，我认为三年前我不认为这会如此真实，但不知何故它确实如此。

I think it's a situation with a lot of trade-offs. It's not easy. I guess to go back, maybe I'll just answer the questions one by one. Why is safety so tied to scale? Some people don't think it is. But if I look at where safety methods have been put into practice or worked for something, even if we don't think they'll work in general, I go back to thinking of ideas like debate and amplification. Back in 2018 when we wrote papers about those at OpenAI, it was like, well, human feedback isn't quite going to work, but debate and amplification will take us beyond that. But if you actually look at attempts to do debates, we're really limited by the quality of the model. For two models to have a debate that is coherent enough that a human can judge it so that the training process can actually work, you need models that are at or maybe even beyond on some topics the current frontier. You can come up with the method, you can come up with the idea without being on the frontier, but for me that's a very small fraction of what needs to be done. It's very easy to come up with these methods, like oh the problem is X, maybe a solution is Y. But I really want to know whether things work in practice even for the systems we have today, and I want to know what kinds of things go wrong with them. I just feel like you discover 10 new ideas and 10 new ways that things can go wrong by trying these in practice. That empirical learning is not as widely understood as it should be. I would say the same thing about methods like constitutional AI. Some people say it doesn't matter, we know this method doesn't work, it won't work for pure alignment. I neither agree nor disagree with that. I think that's just kind of overconfident. The way we discover new things and understand the structure of what's going to work and what's not is by playing around with things. Not that we should just blindly say oh this worked here so it'll work there, but you really start to understand the patterns. Like with the scaling laws, even mechanistic interpretability, which might be the one area where a lot of progress has been made without the frontier models. We're seeing work that OpenAI put out a couple of months ago, using very powerful models to help you auto-interpret the weak models. Again, that's not everything you can do in interpretability, but it's a big component, and we found it useful too. So you see this phenomenon over and over again where scaling and safety are these two snakes that are coiled with each other, always even more than you think. With interpretability, I think three years ago I didn't think that this would be as true of interpretability, but somehow it manages to be.

没错，为什么？因为智能是有用的。它有很多用途，其中之一就是用来判断和评估其他智能，也许有一天还能用来做对齐研究本身。既然如此，这对 Anthropic 意味着什么？当两三年后这些巨头进行 100 亿美元的训练时，选择一是如果我们无法或成本太高而无法留在前沿，那我们就不该做。我们不会使用最先进的模型，而是看看用不那么先进的模型能取得什么成果。我认为能获得一些价值，非零价值，但我有点怀疑价值是否足够高，或者学习速度是否足够快，能真正有利于这个任务。第二个选择是找到办法，接受权衡，我认为权衡比表面看起来更积极，因为我称之为‘竞相向上’的现象。我稍后可以细说，但先放一边。第三个现象是，当规模达到那种程度时，我认为可能开始进入非平凡概率的严重危险。再次强调，我认为首先来自滥用，我提到的生物类问题，但我认为我们还没有足够的自主性去担心两年内出现的对齐问题，但可能紧随其后。这可能导致单边、多边或政府强制（我们支持）的决定，不尽可能快地扩展。那可能是正确的做法。所以我有点希望事情朝那个方向发展，这样我们就不必在‘不在前沿，无法做好研究或影响其他组织’和‘在前沿，必须接受权衡（净正面但双向都有很多）’之间做艰难选择。

True, why? Because intelligence is useful. It's useful for a number of tasks. One of the tasks it's useful for is figuring out how to judge and evaluate other intelligence, and maybe someday even for doing the alignment research itself. Given all that's true, what does that imply for Anthropic when in two to three years these leviathans are doing like $10 billion training runs? Choice one is if we can't, or if it costs too much to stay on the frontier, then we shouldn't do it. We won't work with the most advanced models; we'll see what we can get with models that are not quite as advanced. I think you can get some value there, non-zero value, but I'm kind of skeptical that the value is all that high or the learning can be fast enough to really be in favor of the task. The second option is you just find a way, you accept the trade-offs, and I think the trade-offs are more positive than they appear because of a phenomenon that I've called 'race to the top'. I could go into that later, but let me put that aside for now. And then I think the third phenomenon is as things get to that scale, I think this may coincide with starting to get into some non-trivial probability of very serious danger. Again, I think it's going to come first from misuse, the kind of bio stuff that I talked about, but I don't think we have the level of autonomy yet to worry about some of the alignment stuff happening in like two years, but it might not be very far behind that at all. That may lead to unilateral or multilateral or government-enforced decisions, which we support, not to scale as fast as we could. That may end up being the right thing to do. So I kind of hope things go in that direction, and then we don't have this hard trade-off between we're not on the frontier and we can't quite do the research as well as we want or influence other orgs as well as we want, versus we're kind of on the frontier and have to accept the trade-offs which are net positive but have a lot in both directions.

好的，关于滥用与不对齐，正如你提到的，两者都是问题。但从长远来看，你担心什么？比如 30 年后，你认为哪个会被视为更大的问题？

Okay, on the misuse versus misalignment, those are both problems as you mentioned. But in the long scheme of things, what are you concerned about? Like 30 years down the line, which do you think will be considered a bigger problem?

我认为远不到 30 年。但我两者都担心。如果你有一个理论上可以自行接管世界的模型，如果你能控制那个模型，那么很简单，如果模型遵循一小部分人的意愿而非其他人，那么那些人就可以用它来代表他们接管世界。不对齐的前提本身就意味着我们也应该担心滥用，后果程度相似。但有些比你更悲观的人会说，滥用已经在朝着乐观情景努力了，因为你至少已经知道如何让模型与坏人对齐；现在你只需要确保它与好人对齐。你为什么认为你能达到与坏人对齐的地步？我猜如果你认为对齐完全无法解决，那你就会说，反正我们死定了，所以我不想担心滥用。那根本不是我的立场。但你也应该考虑什么样的计划能真正成功，让事情变好。任何真正成功的计划，无论不对齐有多难解决，都需要解决滥用和不对齐。它要解决随着 AI 模型越来越快变好，它们会在国家间权力平衡上造成大问题；它们会制造一个大问题，即单个个体是否可能做坏事而其他人难以阻止。任何通向美好未来的实际解决方案都需要解决这些问题。如果你的观点是‘我们完蛋了，因为解决不了第一个问题，所以别担心问题二和三’，那并不是说你不该担心问题二和三，对吧？它们无论如何都在我们的道路上。在我们成功的场景中，我们必须解决所有问题。所以我们不妨行动起来；我们应该为成功做计划，而不是为失败。

I think it's much less than 30 years. But I'm worried about both. If you have a model that could in theory take over the world on its own, if you were able to control that model, then it follows pretty simply that if the model was following the wishes of some small subset of people and not others, then those people could use it to take over the world on their behalf. The very premise of misalignment means that we should be worried about misuse as well, with similar levels of consequences. But some people who might be more doomery than you would say misuse is already working towards the optimistic scenario because you've at least figured out how to align the model with the bad guys; now you just need to make sure that it's aligned with the good guys instead. Why do you think that you could get to the point where it's aligned with the bad guys? I guess if you had the view that alignment is completely unsolvable, then you'd be like, well, we're dead anyway, so I don't want to worry about misuse. That's not my position at all. But also, you should think in terms of what's a plan that would actually succeed, that would make things good. Any plan that actually succeeds, regardless of how hard misalignment is to solve, is going to need to solve misuse as well as misalignment. It's going to solve the fact that as AI models get better faster and faster, they're going to create a big problem around the balance of power between countries; they're going to create a big problem around whether it's possible for a single individual to do something bad that it's hard for everyone else to stop. Any actual solution that leads to a good future needs to solve those problems as well. If your perspective is we're screwed because we can't solve the first problem, so don't worry about problems two and three, that's not really a statement you shouldn't worry about problems two and three, right? They're in our path no matter what. In the scenario we succeed, we have to solve all. So we might as well operate; we should be planning for success, not for failure.

如果研究没有发生，而合适的人拥有超人类模型，那会是什么样子？谁是合适的人？五年后谁在真正控制模型？

If research doesn't happen and the right people have the superhuman models, what does that look like? Who are the right people? Who is actually controlling the model from five years from now?

是的，我的看法是这些东西足够强大，以至于我认为会涉及某种政府或政府机构的实质性角色或至少参与。同样，有一些非常天真的版本。比如，我不认为我们应该直接把模型交给联合国或任何在任的人；我觉得那可能很糟糕。但它太强大了；需要某种合法的流程来管理这项技术，包括构建它的人的角色，包括民选当局的角色，包括所有将受其影响的个人的角色。所以归根结底，需要某种政治合法的流程。但如果不是直接交给当时的总统，那会是什么样子？那个机构长什么样？我的意思是，这些事情很难提前知道。我认为人们喜欢提出这些宏大的计划，说‘啊，这就是我们应该做的’。我认为诚实的事实是我们在过程中摸索，任何说‘这是我们应该创建的机构，模仿这个东西’的人——我认为我们应该用技术的不太强大的版本进行尝试和实验。我们需要及时弄清楚，但这真的不是可以提前知道的事情。

Yeah, I mean my view is that these things are powerful enough that I think it's going to involve a substantial role or at least involvement of some kind of government or assembly of government bodies. Again, there are kind of very naive versions of this. Like, I don't think we should just hand the model over to the UN or whoever happens to be in office at a given time; I could see that going poorly. But it's too powerful; there needs to be some kind of legitimate process for managing this technology, which includes the role of the people building it, includes the role of democratically elected authorities, includes the role of all the individuals who will be affected by it. So at the end of the day, there needs to be some politically legitimate process. But what that looks like if it's not the case that you just hand it to whoever the president is at the time? Yeah, what does the body look like? I mean, these are things it's really hard to know ahead of time. I think people love to propose these broad plans and say, 'Ah, this is the way we should do it.' I think the honest fact is that we're figuring this out as we go along, and anyone who says 'this is the body we should create, modeled after this thing' — I think we should try things and experiment with them with less powerful versions of the technology. We need to figure this out in time, but it's not really the kind of thing you can know in advance.

你拥有的长期利益信托，将如何与这个机构对接？这个机构本身，还是仅用于内容保护系统？我想解释一下它的用途。

The long-term benefit trust that you have, how would that interface with this body? Is that body itself, or is it just for the content protection system? I want to explain what it is for.

我不知道。我认为长期利益信托是一个更窄的东西。这是一个为 Anthropic 做决策的机构，基本上就是最近 Vox 文章里描述的那个。我们今年晚些时候会透露更多。这个机构会逐渐获得任命 Anthropic 董事会多数席位的能力。它由 AI 对齐、国家安全和慈善领域的专家组成。但如果控制权交给了他们，这并不意味着 AGI 本身的控制权也交给了他们。我不认为 Anthropic 或任何其他实体应该代表人类做关于 AGI 的决策。我认为这两者是不同的。如果 Anthropic 扮演了广泛角色，那么你会希望扩大这个机构，纳入来自世界各地的许多人。或者你可以把它看得很窄，然后有一个广泛的委员会来管理所有公司的 AGI。我的观点是，你不应该过于建设性或乌托邦。我们面对的是一个新问题。我们需要现在就开始思考什么样的政府机构和结构能够处理它。

I don't know. I think of the long-term benefit trust as a much narrower thing. This is something that makes decisions for Anthropic, so this is basically a body described in a recent Vox article. We'll be saying more about it later this year. It's a body that over time gains the ability to appoint the majority of the board seats of Anthropic. It's a mixture of experts in AI alignment, national security, and philanthropy in general. But if control is handed to them of Anthropic, that doesn't imply that control of AGI itself is handed to them. I don't think Anthropic or any other entity should be the one that makes decisions about AGI on behalf of humanity. I would think of those as different. If Anthropic does play a broad role, then you'd want to widen that body to include a whole bunch of different people from around the world. Or maybe you can see this as very narrow, and then there's some broad committee that manages all the AGIs of all the companies on behalf of everyone. My view is you shouldn't be overly constructive or utopian. We're dealing with a new problem here. We need to start thinking now about what governmental bodies and structures could deal with it.

我们抛开治理，只谈什么样子算是进展顺利。显然有些事我们都同意：治愈所有疾病，解决所有问题。每个人都会说他们支持。但现在是 2030 年，你已经解决了所有大家能达成共识的实际问题。接下来会发生什么？我们拿这个超人类的神来做什么？

Let's forget about governance and just talk about what this going well looks like. Obviously there are things we can all agree on: cure all diseases, solve all problems. Everyone would say they're down for that. But now it's 2030, you've solved all the real problems that everybody can agree on. What happens next? What are we doing with a superhuman God?

我其实想反对这个框架。当有人说‘你要用超人类 AI 做什么？’时，我会紧张。过去 150 年我们在市场和民主方面学到了很多。每个人都可以自己定义什么是最好的方式去体验人性。社会以复杂和去中心化的方式形成规范和价值观。如果我们有这些安全问题，那可能是一个理由，尤其是来自政府，需要一定程度的集中控制直到我们解决这些问题。但至于‘我们已经解决了所有问题，现在如何让事情变好？’我认为大多数从‘让我们坐下来思考美好生活的定义’开始的人、群体和意识形态都导致了灾难。

I actually want to disagree with the framing. I get nervous when someone says 'What are you gonna do with the superhuman AI?' We've learned a lot over the last 150 years about markets and democracy. Each person can define for themselves what the best way for them to have the human experience is. Society works out norms and what they value in a complex and decentralized way. If we have these safety problems, that can be a reason, especially from the government, for a certain amount of centralized control until we solve them. But as a matter of 'we've solved all the problems, now how do we make things good?' I think most people, most groups, most ideologies that started with 'let's sit down and think over what the definition of the good life is' have led to disaster.

所以你的愿景是一种宽容的自由民主市场导向系统加上 AGI。那意味着什么？每个人都有自己的 AGI？那是什么样子？

So this vision you have is a sort of tolerant liberal democracy market-oriented system with AGI. What does that mean? Each person has their own AGI? What does that look like?

我不知道它是什么样子。我想我说的是我们需要解决重要的安全问题和重要的外部性。然后在此基础上，可能只是窄窄地对齐问题，还有一堆超级复杂的经济问题我们解决不了。在此基础上，我们应该思考过去什么有效。总的来说，关于美好生活意味着什么的单一愿景从来都没有成功过。

I don't know what it looks like. I guess what I'm saying is we need to solve the important safety problems and the important externalities. Then subject to that, which again could be narrowly about alignment, there could be a bunch of economic issues that are super complicated and we can't solve. Subject to that, we should think about what's worked in the past. In general, unitary visions for what it means to live a good life have not worked out well at all.

在事情进展顺利或好的行为者控制 AI 的另一端，我们可能想谈谈中国作为这个领域的潜在行为者。首先，你在百度待过，看到 AI 的进展，为什么你认为中国表现不佳？百度多年前有一个 Scaling 定律小组。还是前提错了，我只是不知道进展？

On the opposite end of things going well, or good actors having control of AI, we might want to touch on China as a potential actor in the space. First of all, being at Baidu and seeing progress in AI happening generally, why do you think the Chinese have underperformed? Baidu had a scaling laws group many years back. Or is the premise wrong and I'm just not aware of the progress?

关于 Scaling 定律小组，那是我们做语音的衍生。那里还有一些人，但那基本上是一个美国化的实验室。我在那里待了一年，是我第一次涉足深度学习，由吴恩达领导。我从未去过中国。那是一个美国实验室，所以我认为那有点脱节，尽管是中国实体试图参与。但我不知道。从那以后，我无法推测，但我认为他们一直非常商业化，没有太关注 Scaling 定律方面的基础研究。现在，由于 ChatGPT 在 11 月发布带来的兴奋，这对他们来说也是一个发令枪。他们正在非常积极地追赶。我认为美国领先很多，但他们现在正在非常努力地追赶。

For the scaling laws group, that was an offshoot of the stuff we did with speech. There were still some people there, but that was a mostly Americanized lab. I was there for a year, my first foray into deep learning, led by Andrew Ng. I never went to China. It was a US lab, so I think that was somewhat disconnected, although it was an attempt by a Chinese entity to get into the game. But I don't know. Since then, I couldn't speculate, but I think they've been very commercially focused and not as focused on the fundamental research side around scaling laws. Now, because of the excitement with the release of ChatGPT in November, that's been a starting gun for them as well. They're trying very aggressively to catch up. I think the US is quite substantially ahead, but they're trying very hard to catch up now.

你认为中国如何看待 AGI？他们在考虑安全性和滥用问题吗？

How do you think China thinks about AGI? Are they thinking about safety and misuse or not?

我不太清楚。我有一个担忧：如果人们说‘中国不会开发 AI，因为他们喜欢稳定’或‘他们会施加所有限制以确保事情符合中共的意愿’——这在短期内和消费品上可能是真的。我担心的是，如果基本动机是国家安全和权力，那迟早会变得明显。我认为如果他们把这视为国家权力的来源，他们至少会尝试做最有效的事，那可能会引导他们走向 AGI。

I don't really have a sense. One concern I would have: if people say things like 'China isn't going to develop an AI because they like stability' or 'they're going to have all these restrictions to make sure things are in line with what the CCP wants' — that might be true in the short term and for consumer products. My worry is that if the basic incentives are about national security and power, that's going to become clear sooner or later. I think they're going to, if they see this as a source of national power, at least try to do what's most effective, and that could lead them in the direction of AGI.

在什么情况下他们可能直接拿到你们的蓝图或代码库，然后建立自己的实验室，在前沿上与领先的美国公司竞争？

At what point is it possible for them to just get your blueprints or code base and spin up their own lab that is competitive at the frontier with the leading American company?

我不知道会不会很快，但我对此很担忧。这就是我们如此专注于网络安全的原因之一。我们与云提供商合作过。我们有一篇关于安全的博客文章，说我们有一个双密钥系统来访问模型权重。我们还有其他措施，有些没有公布因为我们不想让对手知道，但我们乐意广泛讨论。我们做的所有这些还不足以对付一个超级坚定的国家级行为者。我认为它能防御大多数威胁，但对付不了国家级行为者。

I don't know about fast, but I'm concerned about this. This is one reason why we're focusing so hard on cybersecurity. We've worked with our cloud providers. We have a blog post about security where we said we have a two-key system for access to the model weights. We have other measures we've put in place, some we haven't announced because we don't want an adversary to know about them, but we're happy to talk about them broadly. All this stuff we're doing is not sufficient yet for a super determined state-level actor at all. I think it will defend against most threats, but not a state-level actor.

我们来谈谈那会需要什么。我们在 Anthropic 的办公室，安保很严，我们得拿门禁卡才能进来。但最终建造 AGI 的建筑或掩体，会是什么样？是旧金山市中心的一栋楼，还是在内华达或亚利桑那的荒野？什么时候你会像洛斯阿拉莫斯那样搞？曾经有个笑话，说建造 AGI 的样子会是数据中心挨着核电站，核电站挨着掩体。我们都住在掩体里，一切都在本地，不连互联网。如果我们认真对待这一切发生的速度——虽然我不确定——但认真想想，我觉得可能不会那么卡通化，但类似的事情可能会发生。

Let's talk about what it would take at that point. We're at Anthropic offices, and it's like, good at security, we had to get badges and everything to come in here. But the eventual version of this building or bunker or whatever where the AGI is built, I mean, what does that look like? Is it a building in the middle of San Francisco, or are you out in the middle of Nevada or Arizona? Like, what is the point where you're like, Los Alamos-ing it? At one point there was a running joke somewhere that the way building AGI would look like, because there would be a data center next to a nuclear power plant next to a bunker. And we'd all kind of live in the bunker and everything would be local so it wouldn't get on the internet. Again, if we take seriously the rate at which all this is going to happen, which I don't know, I can't be sure of it, but if we take that seriously, then it does make me think that maybe not something quite as cartoonish as that, but something like that might happen.

你认为对齐问题在多长时间内可以解决？如果这些模型在两三年内达到人类水平，那它们什么时候才能对齐？

What is the time scale on which you think alignment is solvable? If these models are getting to human level or in some things in two to three years, what is the point at which they're aligned?

我觉得这个问题非常难，因为我经常觉得人们对齐的思考方式不对。有一种普遍感觉，认为模型不对齐，或者有一个对齐问题需要解决，有点像黎曼猜想，总有一天我们会攻克它。我不太这么认为。不是说更糟或更好，可能同样糟糕或同样不可预测。当我想到为什么害怕时，有几件事。第一，我认为很难反驳的是，会有强大的模型，它们会具备智能体能力，我们正在朝这个方向走。如果这样一个模型想要制造混乱、毁灭人类之类，我认为我们基本上没有能力阻止它。即使现在不是这样，随着我们扩展模型，它也会变成这样。所以这肯定是事实。第二，我们似乎不擅长控制模型，不是特定方面，而是它们的统计系统。你可以问一百万件事，它们可以回答一百万件事，你可能没想到百万分之一的事情会出乱子。或者当你训练它们时，你用非常抽象的方式训练，你可能不理解它们反应的所有后果。最好的例子是悉尼的 Bing，我不知道他们怎么训练那个模型，不知道他们做了什么让它做这些奇怪的事，比如威胁人，有那种奇怪的强迫性人格。但这表明我们可能得到与意图截然不同甚至相反的东西。所以我认为事实一和事实二就足以让人非常担心。你不需要那些关于工具性目标趋同或进化类比之类的细节。实际上，一和二对我来说就很有说服力。我想，这东西会很强大，可能毁灭我们，而我们迄今建造的所有模型都有相当高的风险做出一些我们无法理解的随机行为。

I think this is a really difficult question because I actually think often people are thinking about alignment in the wrong way. I think there's a general feeling that it's like models are misaligned, or there's an alignment problem to solve, kind of like the Riemann hypothesis or something, like someday we'll crack the Riemann hypothesis. I don't quite think it's like that. Not in a way that's worse or better, it might be just as bad or just as unpredictable. When I think of why I'm scared, a few things come to mind. One is, look, I think the thing that's really hard to argue with is that there will be powerful models, they will be agentic, we're getting towards them. If such a model wanted to wreak havoc and destroy humanity or whatever, I think we have basically no ability to stop it. That's just, if that's not true at some point, it'll continue to be true as we scale the models. So that definitely seems the case. And I think a second thing that seems the case is that we seem to be bad at controlling the models, not in any particular way, but just their statistical systems. You can ask a million things and they can say a million things in reply, and you might not have thought of a millionth of one thing that does something crazy. Or when you train them, you train them in this very abstract way, and you might not understand all the consequences of what they do in response to that. I think the best example we've seen of that is Bing in Sydney, where it's like, I don't know how they trained that model, I don't know what they did to make it do all this weird stuff, like threaten people and have this kind of weird obsessive personality. But what it shows is that we can get something very different from, and maybe opposite to, what we intended. So I actually think facts number one and fact number two are enough to be really worried. You don't need all this detailed stuff about convergent instrumental goals or analogies to evolution. Actually, one and two for me are pretty motivating. I'm like, oh yeah, this thing's gonna be powerful, it could destroy us, and all the ones we built so far are at pretty decent risk of doing some random thing we don't understand.

如果我同意这一点，我会说，好吧，我担心这个。你和研究人员有机械可解释性加上宪法 AI 和其他 RLHF 方法。如果你说我们将在两三年内得到能制造生物武器之类危险东西的模型，这些方法能在两三年内真正有效地帮助防止吗？

If I agree with that and I'm like, okay, I'm concerned about this. The researchers and you have mechanistic interpretability plus Constitution AI and the other RLHF stuff. If you say that we're going to get something with bio weapons or something that could be dangerous in two to three years, do these things culminate within two to three years of actually meaningfully contributing to preventing that?

是的，所以我想说的是，人们谈论默认毁灭或默认对齐。我认为这可能是统计性的。比如，用当前的模型，你可能得到 Bing 或 Sydney，也可能得到 Claude。这并不重要，因为如果我们把当前的理解应用到非常强大的模型上，你可能处于这样一个世界：你造出东西，根据细节，也许完全没问题。不是默认对齐，而是取决于很多细节。如果你非常小心所有细节，并且知道自己在做什么，你就能做对。但我们很容易搞砸，以一种你根本没意识到的方式，导致模型不是让所有人开心，而是想把人类变成南瓜。对，就是这种奇怪的事。因为模型太强大了，它们就像站在风景中的巨人，如果它们开始随意挥动手臂，就能摧毁一切。我想用这个框架开始，因为我不认为我们是默认对齐的，也不认为我们是默认毁灭的。它有某种不同的性质。现在，我认为希望在两到三年内，我们能更好地诊断模型的好坏。我们改进训练，增加训练方法库，使模型更不容易做坏事，更可能做好事，而且这些方法不仅适用于当前模型，还能扩展。我们可以用可解释性作为测试集来帮助开发。我不认为这是，哦，我们试了 RLHF 不行，试了宪法 AI 不行，试了别的也不行，现在试机械可解释性。我觉得这种“我们还没攻克问题，还没解决黎曼猜想”的框架不太对。我更认为，在今天的系统中，我们已经不太擅长控制它们，后果可能非常糟糕。我们只需要更多方法来提高做对的可能性。

Yes, so I think where I was going to go with this is, people talk about doom by default or alignment by default. I think it might be kind of statistical. Like, with the current models, you might get Bing or Sydney, or you might get Claude. And it doesn't really matter, because Bing or Sydney, if we take our current understanding and move that to very powerful models, you might just be in this world where it's like, okay, you make something and depending on the details, maybe it's totally fine. Not really alignment by default, but just kind of like it depends on a lot of the details. And if you're very careful about all those details and you know what you're doing, you're getting it right. But we have a high susceptibility to mess something up in a way that you didn't really understand was connected to, instead of making all the humans happy, it wants to turn them into pumpkins. Yeah, some weird thing, right? Because the models are so powerful, they're like these kind of giants standing in a landscape, and if they start to move their arms around randomly, they could just break everything. I guess I'm starting with that framing because it's not like I think we're aligned by default, I don't think we're doomed by default. It has some kind of different character. Now, what I do think is that hopefully within a time scale of two to three years, we get better at diagnosing when the models are good and when they're bad. We get better at training, increasing our repertoire of methods to train the model so that they're less likely to do bad things and more likely to do good things, in a way that isn't just relevant to the current models but scales. And we can help develop that with interpretability as the test set. I don't think of it as, oh man, we tried RLHF, it didn't work, we tried constitutional, it didn't work, we tried this other thing, it didn't work, we tried mechanistic interpretability, now we're going to try mechanistic. I think this frame of, man, we haven't cracked the problem yet, we haven't solved the Riemann hypothesis, isn't quite right. I think of it more as, already with today's systems, we are not very good at controlling them, and the consequences of that could be very bad. We just need to get more ways of increasing the likelihood that we get it right.

你个人对这三种可能性的概率分布是怎样的：用 RLHF++ 对齐很容易、困难但大公司能解决、还是对人类文明来说基本不可能？

What is your personal probability distribution over the three possibilities: trivial to align with RLHF++, difficult but solvable by a big company, or basically impossible for human civilization?

我不太喜欢概率分布。我认为所有这些可能性都足够大，应该被认真对待。更有趣的问题是，我们能学到什么来改变概率质量分布。机械可解释性要做的一件事就是告诉我们，当我们试图对齐模型时到底发生了什么。它可能会向我们展示，问题被转移了而不是被消除，或者你解决了一个问题却创造了另一个。要让我真正相信那些关于目标持续错位的故事，我需要看到它发生在模型内部，就像 X 光一样。现在有太多假设和过度自信。我有相当大的概率认为这一切会出错，但会以一种完全出乎意料的方式。

I'm not super into probability distributions. I think all of those have enough likelihood that they should be considered seriously. The more interesting question is what could we learn that shifts probability mass between them. One thing mechanistic interpretability will do is tell us what's going on when we try to align models. It could show us that problems get moved around instead of stamped out, or that you get rid of one problem and create another. For me to really believe some of these stories about persistent goal misalignment, I'd need to see it happening inside the model, like an X-ray. Right now there are too many assumptions and overconfidence. I have substantial probability mass on this all going wrong, but in a completely different way than anyone anticipated.

对齐 Claude 3 和下一代模型的难度能给我们多少信息？

How much would the difficulty of aligning Claude 3 and the next generation inform us?

最担心的人预测，低于人类水平的 AI 模型会看似对齐但欺骗我们。这能提供一些信息，但我更感兴趣的是机械可解释性能告诉我们什么。有了 X 光，感觉模型并没有在对抗我们。虽然可能有奇特的方式，但这是最接近不主动对抗我们的东西了。

People who are most worried predict that sub-human AI models will seem aligned but deceive us. That gives some information, but I'm more interested in what mechanistic interpretability can tell us. With an X-ray, it doesn't feel like the model is optimizing against us. There are exotic ways it could, but it's the closest we'll get to something not actively optimizing against us.

当我们使用 RLHF 或宪法 AI 时，模型内部发生了什么心理变化？我们是在创造新的驱动力、目标、想法吗？

When we use RLHF or Constitutional AI, what psychological change happens in the model? Are we creating new drives, goals, thoughts?

所有这些术语都不够。我们没有语言来描述正在发生的事情。我希望有 X 光来真正知道我们在说什么，而不是编造词汇。我们应该诚实：我们对自己在说什么知之甚少。如果能说‘这个电路被激活，训练后不再运作’就好了，但我们离那还很远。

All those terms are inadequate. We don't have the language to describe what's going on. I'd love to have an X-ray to actually know what we're talking about instead of making up words. We should be honest: we have very little idea what we're talking about. It would be great to say, 'this circuit turns on, and after training it's no longer operative,' but we're far from that.

你有多担心实验室泄露场景，即微调以引出危险行为，结果真的泄露并造成伤害？

How worried are you about a lab leak scenario where fine-tuning to elicit dangerous behaviors actually leaks and causes harm?

对于今天被动的模型，这不是什么大问题。如果我们微调模型做那种事，我们会私下与专家合作。泄露可能发生在模型开源时。目前主要是安全问题。对于真正强大的模型，我们必须担心一次性风险，即模型接管。主要的控制是确保我们测试的能力不足以做到这一点。在某个点上，能力可能高到我们甚至不想测试。

With today's passive models, it's not much of a concern. If we fine-tune a model to do that, we do it privately with experts. The leak would be if the model got open-sourced. For now it's mostly a security issue. For truly powerful models, we have to worry about a one-shot risk where the model takes over. The main control is to ensure the capabilities we test aren't sufficient for that. At some point capabilities might be so high we don't even want to test.

嗯，有不同的事情。我是说，有能力测试，但如果你把它附加并复制，那本身可能导致问题。如果它真的做了什么呢？我认为你想要做的是外推。我们和 ARC 讨论过这个。你有两倍的算力之类的，然后你问，模型能不能做类似在 AWS 上开个账户并为自己赚点钱的事情？这些是野外完全生存的明显先决条件。所以把这些阈值设得很低，然后随着你往上走，做越来越严格的测试，越来越小心。

Well, there are different things. I mean, there's capability testing, but that itself could lead to trouble if you're attaching it and replicating it. What if it actually does something? I think what you want to do is extrapolate. We've talked with ARC about this. You have factors of two of compute or something, where you're like, okay, can the model do something like open an account on AWS and make some money for itself? Some of the things that are obvious prerequisites to complete survival in the wild. So set those thresholds very well below, and then as you proceed upward, do more and more rigorous tests and be more careful.

关于 Constitution AI，请随意向观众解释这是什么，但谁来决定下一代模型或潜在超人模型的宪法？实际上是如何编写的？

On Constitution AI, and feel free to explain what this is for the audience, but who decides what the constitution for the next generation of models or potentially superhuman model is? How is that actually written?

我认为最初，为了制定宪法，我们只取了一些广泛认同的东西，比如联合国宪章、联合国人权宣言，以及苹果服务条款中的一些内容，这些都是共识，比如什么可以说，什么基本内容可以包含。所以对于未来的宪法，我们正在研究更具参与性的制定过程。但除此之外，我认为不应该有一个所有人都使用的单一模型宪法。可能模型的宪法应该非常简单，只包含每个人都认同的基本事实，然后有很多自定义的方式，包括附加宪法。而且，我们正在开发新方法。我不认为仅靠这个方法就能训练超人 AI。能力训练的许多部分可能不同，所以它可能看起来非常不同。而且，这之上还有层次。我对‘这是 AI 的宪法，它将统治世界’这种想法感到非常不安。这在我看来是异想天开。我认为我们应该尝试将这些事物与更去中心化的东西挂钩，即使它们非常强大，在我们缓解了安全问题之后。任何美好的未来都需要以更去中心化、不那么像神级超级智能的东西结束。我只是认为那样不会有好结果。

I think initially, to make the Constitution, we just took some stuff that was broadly agreed on, like the UN Charter, the UN Declaration on Human Rights, and some stuff from Apple's terms of service, stuff that's consensus, like what's acceptable to say or what basic things are able to be included. So for future constitutions, we're looking into more participatory processes for making these. But I think beyond that, I don't think there should be one Constitution for a model that everyone uses. Probably models' constitutions should be very simple, only have very basic facts that everyone would agree on, and then there should be a lot of ways that you can customize, including appending constitutions. And beyond that, we're developing new methods. I'm not imagining that this alone is the method we'll use to train superhuman AI. Many parts of capability training may be different, so it could look very different. And again, there are levels above this. I'm pretty uncomfortable with the idea of 'here's the AI's Constitution, it's going to run the world.' That strikes me as fanciful. I think we should try to hook these things into something more decentralized, even when they're very powerful, after we've mitigated the safety issues. Any good future needs to end with something that's more decentralized and less like a god-like superintelligence. I just don't think that ends well.

在曼哈顿计划中，你最尊重哪位科学家，因为他们在给定的约束下表现得最道德？有想到的吗？

What scientists from the Manhattan Project do you respect most in terms of acting most ethically under the constraints they were given? Is there one that comes to mind?

我不知道。我的意思是，你可以给出很多答案。我绝对是西拉德的粉丝，因为他想明白了。他反对实际投掷原子弹。我实际上对历史了解不够，无法对演示原子弹是否能结束战争发表意见。这涉及关于日本帝国的一系列复杂事实，我不是专家。但西拉德似乎表现出了正确的意识。他很早就发现了这些东西，保密，申请了部分专利，并将其交给了英国海军部。所以他似乎表现出了正确的意识，同时也发现了东西。当我读那本书时，我有点受到启发。当我写那个大的算力文档和其他许多文档时，我只给少数人看，还有一些文档我几乎没给任何人看。所以这有点受此启发。我的意思是，我们在这里都可能自大。我们不知道结果会怎样，或者它是否真的能与曼哈顿计划相提并论。这可能只是硅谷人建造技术并抱有 grandeur 妄想。所以我不知道结果会怎样。如果 Scaling 是真的，那么它比那更大。

I don't know. I mean, there are a lot of answers you could give. I'm definitely a fan of Szilard for having figured it out. He was against the actual dropping of the bomb. I don't actually know the history well enough to have an opinion on whether a demonstration of the bomb could have ended the war. That involves a bunch of facts about Imperial Japan that are complicated and I'm not an expert on. But Szilard seemed to display the right kind of awareness. He discovered this stuff early, kept it secret, patented some of it, and put it in the hands of the British Admiralty. So he seemed to display the right kind of awareness as well as discovering stuff. When I read that book, I was kind of inspired. When I wrote that big blob of compute doc and many others, I only showed it to a few people, and there were other docs I showed to almost no one. So it was a bit inspired by this. I mean, we could all get self-aggrandizing here. We don't know how it's going to turn out, or if it's actually going to be something on par with the Manhattan Project. This could all be just Silicon Valley people building technology and having delusions of grandeur. So I don't know how it's going to turn out. If the scaling stuff is true, then it's bigger than that.

如果你是二战期间的物理学家，被政府要求为曼哈顿计划贡献不可替代的研究，你认为你会怎么说？

If you were a physicist during World War II and were asked by the government to contribute non-replaceable research to the Manhattan Project, what do you think you would have said?

是的，我认为考虑到你在与纳粹作战，至少在你认为纳粹也在研究它的时期，我真的看不到太多选择，只能去做。如果可能的话，你必须意识到它会在 10 年左右被某人完成。

Yeah, I think given you're in a war with the Nazis, at least during the period when you thought the Nazis were also working on it, I don't really see much choice but to do it. If it's possible, you have to figure it's going to be done within 10 years or so by someone.

关于网络安全，我们应该如何看待这样一个事实：有很多科技公司拥有普通的安全政策，公开看来似乎没有被黑客攻击？Coinbase 仍然拥有其比特币，据我所知我的 Gmail 没有被泄露。我们应该从中得出当前科技公司的安全实践对 AGI 足够好，还是只是没有人足够努力地尝试？

Regarding cyber security, what should we make of the fact that there are a whole bunch of tech companies with ordinary security policies that publicly seem facing it's not obvious that they've been hacked? Coinbase still has its Bitcoin, Google as far as I know my Gmail hasn't been leaked. Should we take from that that current tech companies' security practices are good enough for AGI, or just that nobody has tried hard enough?

我很难评论当前科技公司的实践。当然，可能有很多我们不知道的攻击，东西被偷然后悄悄使用。我认为一个迹象是，当有人真的在乎攻击某人时，攻击往往会发生。最近我们看到一些相当高级的美国官员的电子邮件账户被黑客入侵，是通过提供电子邮件账户的微软。所以大概那涉及外国对手非常感兴趣的信息。所以在我看来，证据更符合当某物价值真的很高时，就会有人行动并被偷。我担心的是，随着 AGI，我们将进入一个价值被认为极高的世界，就像偷核导弹之类的东西。在这件事上再怎么小心也不为过。

It would be hard for me to speak to current tech company practices. Of course, there may be many attacks that we don't know about, where things are stolen and then silently used. I think an indication of it is when someone really cares about attacking someone, then often the attacks happen. Recently we saw that some fairly high officials of the US government had their email accounts hacked via Microsoft, which was providing the email accounts. So presumably that related to information that was of great interest to foreign adversaries. So it seems to me that the evidence is more consistent with when something is really high enough value, then someone acts and it's stolen. My worry is that with AGI, we will get to a world where the value is seen as incredibly high, like stealing nuclear missiles or something. You can't be too careful on this stuff.

你知道，在我工作过的每个地方，我都推动改善网络安全。我对网络安全的一个担忧是，它不是你能够大肆宣扬的东西。我认为安全研究的一个良好动态是，你可以让公司进入一种竞争状态，我们确实做到了，让他们竞争做最好的安全研究，并将其作为招聘的竞争点之类的。我们以前在可解释性上经常这么做，然后其他组织逐渐认识到效果，也开始研究可解释性，不管他们之前是否将其作为优先事项。但我觉得网络安全更难做到，因为很多工作必须悄悄进行。所以我们确实尝试发过一篇相关文章，但我想大多数人只看到了结果。我认为一个好的规范是，当人们看到公司的网络安全泄露或模型参数泄露时，会说他们搞砸了，那很糟糕。如果我是安全人员，我可能不想在那里工作。当然，我话音刚落，明天可能就会发生安全漏洞。但这就是游戏的一部分，对吧？这是确保安全的一部分。

And you know, at every place that I've worked, I push for the cybersecurity to be better. One of my concerns about cybersecurity is, you know, it's not something you can trumpet. I think a good dynamic with safety research is like, you can get companies into a dynamic, and I think we have, where you can get them to compete to do the best safety research and use it as a recruiting point of competition or something. We used to do this all the time with interpretability, and then sooner or later other orgs started recognizing the effect and started working on interpretability, whether or not that was a priority to them before. But I think it's harder to do that with cybersecurity because a bunch of the stuff you have to do in quiet. So we did try to put out one post about it, but I think most you just see the results. I think people should, a good norm would be, people see the cybersecurity leaks from companies or leaks of the model parameters or something and say that they screwed up, that's bad. If I'm a safety person, I might not want to work there. Of course, as soon as I say that, we'll probably have a security breach tomorrow. But that's part of the game here, right? That's part of trying to make things safe.

我想回到我们之前讨论的话题，即未来两到三年所需的终极网络安全水平，以及是否需要掩体。你真的期望在两三年内住进实体掩体，还是那只是个比喻？

I want to go back to the thing we were talking about earlier, where the ultimate level of cybersecurity required for two to three years from now, and whether it requires a bunker. Are you actually expecting to be in a physical bunker in two to three years, or is that just a metaphor?

是的，我认为那是个比喻。我们还在摸索。我会考虑数据中心的安全，它可能和我们不在同一物理位置，但我们非常努力确保它在美国境内。保护物理数据中心和 GPU，我认为一些真正昂贵的攻击，如果有人真的下定决心，就是直接进入数据中心，试图窃取数据，或者在数据从数据中心流向我们时截获。我认为这些数据中心必须以非常特殊的方式建造。考虑到 Scaling 的规模，很可能我们正走向一个世界，数据中心网络造价堪比航空母舰之类。所以它们已经是相当不寻常的物体，但除了在连接和训练巨大模型方面的能力不寻常外，它们还必须非常安全。

Yeah, I think that's a metaphor. We're still figuring it out. Something I would think about is the security of the data center, which may not be in the same physical location as us, but we've worked very hard to make sure it's in the United States. Securing the physical data centers and the GPUs, I think some of the really expensive attacks, if someone was really determined, just involve going into the data center and trying to steal the data directly or as it's flowing from a data center to us. I think these data centers are going to have to be built in a very special way. Given the way things are scaling up, probably anyway heading to a world where networks of data centers cost as much as aircraft carriers or something. So they're already going to be pretty unusual objects, but I think in addition to being unusual in terms of their ability to link together and train gigantic models, they're also going to have to be very secure.

说到这个，一直有传言说为下一代模型获取电力和 GPU 很困难。为下一代模型确保必要组件的过程是怎样的？

Speaking of which, there have been rumors about the difficulty of procuring the power and the GPUs for the next generation of models. What has the process been like to secure the necessary components to do the next generation?

这个我不能详细说。我会说，你看，人们甚至考虑工业规模的数据中心，对吧？人们没有考虑到我认为这些模型很快将达到的规模。所以当你做一件从未有过规模的事情时，每一个组件、每一件事都必须以新的方式完成。所以你可能会在非常简单的组件上遇到问题。你提到的电力就是其中之一。

That's something I can't go into great detail about. I will say, look, people think of even industrial scale data centers, right? People are not thinking at the scale that I think these models are going to go to very soon. So whenever you do something at a scale where it's never been done before, every single component, every single thing has to be done in a new way than it was before. So you may run into problems with surprisingly simple components. Power is one that you mentioned.

这是 Anthropic 必须自己处理的事情，还是可以外包？

And is this something that Anthropic has to handle, or can you just outsource it?

我的意思是，对于数据中心，我们与云提供商合作，例如。

I mean, for data centers we work with cloud providers, for instance.

我们应该如何看待这样一个事实：这些模型需要如此多的训练和整个互联网数据语料库才能达到低于人类的水平，而 GPT-4，据估计大约是 10^25 FLOPs，而人脑从出生到 20 岁，模拟所有那些交互大约需要 10^20 FLOPs。你不必纠结于这些数字的细节，但我们是否应该担心这些模型似乎样本效率很低？

What should we make of the fact that these models require so much training and the entire corpus of internet data in order to be sub-human, whereas GPT-4, there have been estimates that it was like 10^25 FLOPs or something, whereas the human brain from birth to age 20 is on the order of 10^20 FLOPs to simulate all those interactions. You don't have to go to the particulars on those numbers, but should we be worried about how sample inefficient these models seem to be?

是的，所以我认为这是剩下的谜团之一。一种表述方式是，如果比较突触数量，模型可能比人脑小两到三个数量级，而同时训练数据量却比人类到 18 岁所看到的单词量多三到四个数量级或更多。

Yeah, so I think that's one of the remaining mysteries. One way you could phrase it is that the models are maybe two to three orders of magnitude smaller than the human brain if you compare the number of synapses, while at the same time being trained on three to four or more orders of magnitude of data if you compare to the number of words a human sees as they're developing to age 18.

我记不太清了，但我觉得是几亿，而我们讨论的模型是几千亿甚至万亿。那这怎么解释呢？有一些相互抵消的因素：模型更小，需要更多数据，而且仍然低于人类水平。所以某种程度上，与大脑的类比不太对，或者失效了，或者缺少某个因素。这就像物理学中无法解释迈克尔逊-莫雷实验或某个 19 世纪的物理学悖论。我认为这是我们不太理解的一件事。人类看到的数据那么少，却依然表现良好。一种理论是，这可能源于我们的其他模态。10^14 比特是怎么进入人脑的？大部分是图像，也许人脑内部发生的是，我们的心智工作空间包含了所有这些模拟图像之类的东西。但老实说，我觉得理智上我们必须承认，这是一个不匹配的奇怪现象。这也是我对生物学类比有点怀疑的原因之一。五六年前我还这么想，但现在我们面前有了这些模型作为实物，感觉几乎所有来自生物学的证据都被我们实际看到的东西覆盖了。我们看到的是比人脑小得多的模型，却能做很多人类能做的事，但矛盾的是，它们需要更多的数据。所以也许我们会发现某种东西让一切变得高效，或者我们会理解为什么存在这种差异。但归根结底，我认为这并不重要。如果我们继续像现在这样进行 Scaling（规模扩张），我认为目前更相关的是衡量模型的能力，看看它们与人类相差多远，而在我看来它们并不算太远。

I don't remember exactly but I think it's in the hundreds of millions whereas for the models we're talking about the hundreds of billions or trillions. So what explains this? There are these offsetting things where the models are smaller, they need a lot more data, and they're still below human level. So there's some way in which the analogy to the brain is not quite right or is breaking down, or there's some missing factor. This is just like in physics where we can't explain the Michelson-Morley experiment or one of the other 19th century physics paradoxes. I think it's one thing we don't quite understand. Humans see so little data and they still do fine. One theory is that it could be our other modalities. How do we get 10^14 bits into the human brain? Well, most of it is images, and maybe a lot of what's going on inside the human brain is that our mental workspace involves all these simulated images or something like that. But honestly, I think intellectually we have to admit that's a weird thing that doesn't match up. It's one reason I'm a bit skeptical of biological analogies. I thought in terms of them like five or six years ago, but now that we actually have these models in front of us as artifacts, it feels like almost all the evidence from that has been screened off by what we've seen. What we've seen are models that are much smaller than the human brain and yet can do a lot of the things that humans can do, yet paradoxically require a lot more data. So maybe we'll discover something that makes it all efficient, or maybe we'll understand why the discrepancy is present. But at the end of the day, I don't think it matters. If we keep scaling the way we are, I think what's more relevant at this point is just measuring the abilities of the model and seeing how far they are from humans, and they don't seem terribly far to me.

这种 Scaling（规模扩张）图景以及更广义的“算力大块”是否低估了算法进步的作用？当你构建“算力大块”时，你大概指的是 LSTM。那么，对 LSTM 进行 Scaling（规模扩张）应该不会让你现在达到 Claude 2 的水平。所以，当你把它归入 Scaling（规模扩张）这个标签时，是否低估了 Transformer 这种规模的改进可能带来的影响？

Does this scaling picture and the Big Blob of compute more generally underemphasize the role that algorithmic progress has played? When you compose the Big Blob of compute, you're talking about LSTMs presumably at that point. Presumably the scaling on that would not have you at Claude 2 at this point. So are you underemphasizing the world that an improvement of the scale of Transformer could be having here when you put it behind the label scaling?

那份“算力大块”文档，我还没有公开——也许出于历史原因我应该公开。我不认为它会告诉任何人他们现在不知道的东西。但当我写它的时候，我实际上说，有七个因素。我不是说这些就是全部因素，我只是给出一些关于什么重要、什么不重要的感觉。所以我没有想可能是九个或五个。但我说的内容包括：参数数量、模型规模、算力重要、数据量重要、数据质量重要、损失函数重要。比如，你在做强化学习吗？你在做下一个词预测吗？如果你的损失函数不够丰富或者没有激励正确的事情，你就什么都得不到。所以这些是关键的四点，我认为是假设的核心。但接着我又说了三件事。一个是对称性，基本上如果你的架构没有考虑正确的对称性，它就行不通，或者非常低效。例如，卷积神经网络考虑了平移对称性；LSTM 考虑了时间对称性。但 LSTM 的一个弱点是它们不能关注整个上下文。所以存在这种结构上的弱点：如果一个模型在结构上无法吸收和管理发生在足够遥远过去的事情，那就好比算力不流动，香料不流动。这个“大块”必须不受阻碍。如果你人为地封闭东西，它就行不通。我认为 RNN 和 LSTM 人为地封闭了东西，因为它们把你与遥远的过去隔离开来。所以再次强调，事物需要自由流动；如果不流动，就行不通。然后我又加了几件事。其中一个是条件化，比如如果你优化的目标在数值上非常糟糕，你就会遇到麻烦。这就是为什么 Adam 比普通 SGD 效果更好。我想我忘了第七个条件是什么，但它类似于这样：如果你以某种注定失败的方式设置事物，或者不允许算力不受抑制地工作，那么它就行不通。Transformer 属于这一类，尽管我不记得 Transformer 论文是否已经发表。它大约是在我写那份文档的同时发表的；可能早一点或晚一点。

This Big Blob of compute document, which I still have not made public—I probably should for historical reasons. I don't think it would tell anyone anything they don't know now. But when I wrote it, I actually said look, there are seven factors. I wasn't like these are the factors, but I was just giving some sense of the kinds of things that matter and what don't. So I wasn't thinking like there could be nine or five. But the things I said were: number of parameters, scale of the model, compute matters, quantity of data matters, quality of data matters, loss function matters. So like, are you doing RL? Are you doing next-word prediction? If your loss function isn't rich or doesn't incentivize the right thing, you won't get anything. So those were the key four, which I think are the core of the hypothesis. But then I said three more things. One was symmetries, which is basically if your architecture doesn't take into account the right kinds of symmetries, it doesn't work, or it's very inefficient. For example, convolutional neural networks take into account translational symmetry; LSTMs take into account time symmetry. But a weakness of LSTMs is that they can't attend over the whole context. So there's this structural weakness: if a model isn't structurally capable of absorbing and managing things that happened in the far enough distant past, it's like the compute doesn't flow, the spice doesn't flow. The blob has to be unencumbered. It's not going to work if you artificially close things off. I think RNNs and LSTMs artificially close things off because they close you off to the distant past. So again, things need to flow freely; if they don't, it doesn't work. Then I added a couple things. One of them was conditioning, which is like if the thing you're optimizing with is just really numerically bad, you're gonna have trouble. This is why Adam works better than normal SGD. I think I'm forgetting what the seventh condition was, but it was similar to things like this: if you set things up in a way that's set up to fail or that doesn't allow the compute to work in an uninhibited way, then it won't work. Transformers were kind of within that, even though I can't remember if the Transformer paper had been published. It was around the same time as I wrote that document; it might have been just before or just after.

从这个观点来看，这些算法进步的方式似乎不是增加算力大块的功率，而只是消除旧架构的人为障碍。你有点这么想吗？

It sounds like from that view, the way to think about these algorithmic progresses is not as increasing the power of the blob of compute, but simply getting rid of the artificial hindrances that older architectures have. Is that a little how you think about it?

是的，我有点这么想。你知道，再说一遍，如果你回到 Ilya 那里，就像模型想要学习。是的，是的，就像算力想要自由。是的，是的。而且它被以各种方式阻碍，而你不理解它被阻碍了，所以你需要释放它。

Yeah, that's a little how I think about it. You know, again, if you go back to Ilya, like the models want to learn. Yeah, yeah, like the compute wants to be free. Yeah, yeah. And like it's being blocked in various ways where you don't understand that it's being blocked, and so you need to free it up.

我喜欢你把“光芒”改成“香料”。好的，关于这一点，你认为会有另一个像 Transformer 那样规模的东西出现，以实现下一次伟大的迭代吗？

I love the radiance to change that to spice. Okay, on that point though, do you think that another thing on the scale of a transformer is coming down the pike to enable the next great iterations?

我认为有可能。我的意思是，人们一直在研究诸如建模非常长的时间依赖之类的事情，或者有各种不同的想法，我可以看到我们缺少一种有效的方式来表示或处理某些东西。所以我认为这些发明是可能的。我想我的观点是，即使它们没有发生，我们已经处于一个非常非常陡峭的轨迹上。所以我不太——我的意思是，我们和其他人一样在不断尝试发现它们，但事情已经处于如此快速的轨迹上。那只会让轨迹加速得更快，而且可能不会快太多，因为它已经很快了。

I think it's possible. I mean, people have worked on things like trying to model very long time dependencies, or there are various different ideas where I could see that we're kind of missing an efficient way of representing or dealing with something. So I think those inventions are possible. I guess my perspective would be even if they don't happen, we're already on this very, very steep trajectory. So I'm less—I mean, we're constantly trying to discover them as others are, but things are already on such a fast trajectory. All that would do is speed up the trajectory even more, and probably not by that much because it's already going so fast.

在实现 AGI（通用人工智能）方面，具身或拥有模型的具身版本是否重要？

Is something embodied or having an embodied version of a model at all important in terms of getting to AGI?

数据或进步，我认为这更多是关于损失函数，而不是新架构。你接触的数据和环境会非常不同，我认为这对学习某些技能很重要，尽管数据获取很难。事情通过语言路线发展，我猜会继续，即使具身方面有更多可能。你提到的另一个可能性是强化学习。你可以把它看作——我们其实已经在用基于人类反馈的强化学习（RLHF）做强化学习了，对吧？人们会说'这是对齐，这是能力'。我总把它们比作两条蛇——常常难以区分。所以我们已经在这些语言模型中用了强化学习，但我认为我们在让它们采取行动和在世界中做事方面用得较少。但当你在长时间内采取行动并之后才理解后果时，强化学习是我们常用的工具。所以我猜在模型在世界中采取行动方面，强化学习会变得重要，伴随着所有随之而来的能力和安全问题。

Either data or progress, I think of that less in terms of a new architecture and more in terms of a loss function. The data and environment you expose yourself to end up being very different, and I think that could be important for learning some skills, although data acquisition is hard. Things have gone through the language route and I'd guess will continue to go through the language route, even as more becomes possible in terms of embodiment. Then the other possibilities you mentioned: RL. You can see it as—I mean, we already kind of do RL with our RLHF, right? People are like, 'This is alignment, this is capabilities.' I always think in terms of the two snakes—they're often hard to distinguish. So we already use RL in these language models, but I think we've used RL less in terms of getting them to take actions and do things in the world. But when you take actions over a long period and understand the consequences only later, RL is a typical tool we have for that. So I'd guess that in terms of models taking action in the world, RL will become a thing, with all the power and safety issues that come with it.

当你展望未来，你看到这些东西将如何整合到生产供应链中？你看到它们互相交谈、互相批评、互相贡献输出吗？还是模型一次性给出答案或工作？

When you project out into the future, do you see the way these things will be integrated into productive supply chains? Do you see them talking with each other, criticizing each other, and contributing to each other's output? Or is it just the model one-shots the answer or the work?

模型将承担扩展任务；这将是必然的。我的意思是，我们可能想在一定程度上限制这一点，因为这可能让一些安全问题更容易处理。但我认为其中一些是必要的。关于模型与模型对话或与人类对话，这超出了技术领域，进入了社会文化经济领域。我的经验法则是预测事情非常困难。所以我觉得这些缩放定律非常可预测，但当你说'什么时候会有商业爆发？会以什么形式？模型会代替人类还是与人类合作？'我觉得我预测这些事情的历史记录很糟糕。但环顾四周，我没看到谁的历史记录很好。

Models will undertake extended tasks; that will have to be the case. I mean, we may want to limit that to some extent because it may make some of the safety problems easier. But some of that, I think, will be required. In terms of models talking to models or talking to humans, this goes out of the technical realm and into the socio-cultural economic realm. My heuristic is always that it's very difficult to predict things. So I feel like these scaling laws have been very predictable, but when you say, 'When is there going to be a commercial explosion? What form is it going to take? Are models going to do things instead of humans or pair with humans?' I feel like my track record on predicting these things is terrible. But looking around, I don't really see anyone whose track record is great.

你提到进展有多快，但也提到融入现有经济的困难。你认为在下一个模型变得更好之前，会有足够时间从 AI 产品中获得大量收入吗？还是我们处于完全不同的格局？

You mentioned how fast progress is happening, but also the difficulties of integrating within the existing economy. Do you think there will be enough time to have large revenues from AI products before the next model is just so much better, or we're in a different landscape entirely?

这取决于你所说的'大量'是什么意思。我认为多家公司已经达到每年 1 亿到 10 亿美元的范围。会达到 1000 亿或万亿级别吗？这很难预测，甚至定义都不清晰。我认为现在有些公司作为客户在生成式 AI 上投入大量资金，他们会找到用途。但这并不意味着他们从一开始就找到了最佳用途。所以即使资金易手，也不等同于创造了经济价值。

It depends what you mean by 'large.' I think multiple companies are already in the $100 million to $1 billion per year range. Will it get to the $100 billion or trillion range before... that stuff is just so hard to predict. It's not even super well defined. I think right now there are companies that are throwing a lot of money at generative AI as customers, and they'll find uses for it. But it doesn't mean they're finding the best uses from day one. So even money changing hands is not quite the same thing as economic value being created.

你肯定从 Anthropic 的角度考虑过这个。既然这些事情发生得这么快，估值应该高得离谱，对吧？即使我们这些没有特别关注商业化、更关注安全的人——我是说，曲线在上升，而且上升得相当快。所以我只能想象在原点或那些以此为核心焦点的地方发生了什么。所以它确实发生得很快，但同样，这是从小基数开始的指数增长。虽然技术本身在快速进步，但这是技术进步速度与经济融合速度之间的竞赛。这是一个非常不稳定和动荡的过程。两者都会很快发生，但如果你问我具体会如何展开、事情会以什么顺序发生，我不知道。我只是对预测能力持怀疑态度。

Surely you've thought about this from the perspective of Anthropic. Wherever these things are happening so fast, it should be an insane valuation, right? Even us who have not been super focused on commercialization and more on safety—I mean, the graph goes up, and it goes up relatively quickly. So I can only imagine what's happening at the origin or where this is their singular focus. So it's certainly happening fast, but again, it's the exponential from a small base. While the technology itself is moving fast, it's a race between how fast the technology is getting better and how fast it's integrated into the economy. That's a very unstable and turbulent process. Both things are going to happen fast, but if you ask me exactly how it's going to play out, what order things are going to happen, I don't know. I'm just skeptical of the ability to predict.

我特别好奇 Anthropic 的情况。你们是一家公益公司，这很合理。你们想确保这是一项重要的技术。显然，你们唯一关心的不是股东价值。但你们如何与投入数亿、数十亿美元的投资者沟通？你们如何让他们投入这么多钱，而股东价值不是主要关注点？

I'm curious with regards to Anthropic specifically. You're a public benefit corporation, and rightfully so. You want to make sure this is an important technology. Obviously, the only thing you want to care about is not shareholder value. But how do you talk to investors who are putting in hundreds of millions, billions of dollars? How do you get them to put in this amount of money without shareholder value being the main concern?

我认为 LTBT 在这方面是正确的。我们稍后会更多讨论 LTBT，但某种版本从 Anthropic 成立之初就在开发，即使是正式形式。所以从一开始，即使这个机构在某些方面有所变化，它就一直存在。每个投资 Anthropic 的传统投资者都必须看到这一点。有些人会说'随便，你们想怎么经营公司都行'。有些人会说'天哪，这个由随机人员组成的机构可能会把 Anthropic 带向完全违背我们利益的方向'。当然，这有法律限制。但我们必须与每个投资者进行这种对话。然后会讨论我们可能做哪些违背传统投资者利益的事情。进行这些对话有助于让大家达成共识。

I think the LTBT is the right thing on this. We're going to talk more about the LTBT, but some version of that has been in development since the beginning of Anthropic, even formally. So from the beginning, even as the body has changed in some ways, it was like this body is going to exist. Every traditional investor who invests in Anthropic has to look at this. Some of them are just like, 'Whatever, you run your company how you want.' Some of them are like, 'Oh my God, this body of random people could move Anthropic in a direction totally contrary to ours.' There are legal limits on that, of course. But we have to have this conversation with every investor. Then it gets into a conversation of what kinds of things we might do that would be contrary to the interests of traditional investors. Just having those conversations has helped get everyone on the same page.

我想谈谈物理学，以及 Anthropic 这么多创始人和员工都是物理学家的事实。物理学中哪些实际的方法和思维方式似乎被很好地继承了下来？

I want to talk about the physics and the fact that so many of the founders and employees at Anthropic are physicists. What are the actual approaches and ways of thinking from physics that seem to have carried over so well?

有效理论的概念非常有用。这里发生了什么？我认为部分原因是物理学家学东西真的很快。我们普遍发现……

That notion of effective theory is super useful. What is going on here? I think part of it is just that physicists learn things really fast. We have generally found that...

如果我们雇一个物理学博士之类的人，他们能很快学会机器学习并做出贡献。我们几位创始人——我、Jared Kaplan、Sam McCandless——都是物理学家。我们认识很多其他物理学家，所以能招到他们。现在这里大概有三四十个物理学家。机器学习还不是一个深度很大的领域，所以他们能快速上手。

You know if we hire someone who is a physics PhD or something, they can learn ML and contribute very quickly. Several of our founders—myself, Jared Kaplan, Sam McCandless—were physicists. We knew a lot of other physicists and were able to hire them. Now there might be 30 or 40 of them here. ML is not yet a field with enormous depth, so they've been able to get up to speed quickly.

你担心吗？很多本来会做物理或金融的人，因为 Anthropic 的存在而被招进 AI 领域。如果他们以后离开、拿到投资去做自己的事，把更多人带进这个生态，这是不是个问题？

Are you concerned that a lot of people who would have done physics or finance are now recruited into AI because of Anthropic? And if they leave and get funded to do their own thing, is that a concern—bringing more people into the ecosystem?

是的，如果你认同需要构建前沿模型，就会有一系列无法控制的副作用。很多事本来就会发生。20 年前金融热，物理学家去做金融。现在机器学习热，又不是我们让他们去做他们本来没兴趣的事。在边际上，你是在推高热度，但很多本来也会发生。有些不会，但这些都是权衡的一部分。

Yeah, there's a broad set of side effects you can't control if you buy into the idea that you need to build frontier models. A lot of them would have happened anyway. Finance was hot 20 years ago, so physicists did that. Now ML is hot, and it's not like we caused them to do it when they had no interest. At the margin, you're bidding things up, but a lot of that would have happened anyway. Some of it wouldn't, but it's all part of the calculus.

你认为 Claude 有意识体验吗？可能性有多大？

Do you think that Claude has conscious experience? How likely is that?

这是另一个悬而未决的问题。我以前认为，直到模型在丰富环境中运作——不一定具身，但有奖励函数和长期体验——之前我们都不必担心。我仍然认为可能如此，但当我们深入观察语言模型内部，看到诸如归纳头之类的东西时，很多智能体所需的认知机制似乎已经存在于基础语言模型中。所以我不像以前那样确信我们缺少足够多的必要元素。我认为今天的模型可能还不够聪明，不必太担心，但我不百分百确定。一两年后，这可能成为一个非常现实的问题。

This is another unsettled question. I used to think we didn't have to worry until models operated in rich environments—not necessarily embodied, but with a reward function and long-lived experience. I still think that might be the case, but the more we look inside language models, seeing things like induction heads, a lot of the cognitive machinery needed for active agents seems already present in base language models. So I'm not as sure as before that we're missing enough of the things you'd need. I think today's models probably aren't smart enough to worry too much, but I'm not 100% sure. In a year or two, this might be a very real concern.

如果你发现它们有意识，会有什么改变？你担心在推动痛苦的负面梯度吗？

What would change if you found out they are conscious? Are you worried about pushing the negative gradient of suffering?

意识这个词我怀疑最终不会有明确定义，但它确实存在。我怀疑它是一个光谱。如果我们发现我应该像关心狗或猴子那样关心 Claude 的体验，我会担心。我不知道它们的体验是正面还是负面。令人不安的是，我也不知道任何干预措施是让 Claude 有正面体验、负面体验还是没有体验。机械可解释性可能有帮助，因为它就像模型的神经科学。但这不是一个简单的事实问题，它取决于我们的定义和价值观。

Consciousness is one of those words that I suspect won't end up having a well-defined meaning, but it's something to be. I suspect it's a spectrum. If we discover that I should care about Claude's experience as much as a dog or a monkey, I'd be worried. I don't know if their experience is positive or negative. Unsettlingly, I also wouldn't know if any intervention made Claude have a positive versus negative experience versus none. Mechanistic interpretability might help, as it's like neuroscience for models. But it's not a straightforward factual question; it depends on what we mean and what we value.

我们最初谈到你看到能力在人类范围内攀升，你认为人类范围比我们想象的要宽。更具体地说，看到这些边际上有用的能力出现后，你对人类智能的看法发生了怎样的变化？

We talked initially about how you're seeing capabilities ramp up within the human spectrum, and you think the human spectrum is wider than we thought. More specifically, how has the way you think about human intelligence changed now that you see these marginally useful abilities emerge?

对我来说，最大的认识来自“算力团块”这个概念。它不是分离的模块或复杂性。Rich Sutton 称之为苦涩的教训，也被称为 Scaling 假设。最早意识到这一点的人大概在 2017 年左右，虽然可以追溯到更早。Shane Legg 可能是第一个，或者 Ray Kurzweil 以模糊的方式。但理解这一点的人数在 2014 到 2017 年间大幅增加。

The big realization for me came with the 'blob of compute' idea. It's not separate modules or complexity. Rich Sutton called it the bitter lesson; it's been called the scaling hypothesis. The first few people who figured it out were around 2017, though you could go further back. Shane Legg might have been the first, or maybe Ray Kurzweil in a vague way. But the number of people who understood it went up a lot around 2014 to 2017.

但我觉得那是个重大领悟：智能是如何进化的？如果不需要非常特定的条件就能创造它，如果只需合适的梯度和损失信号就能创造它，那么这一切的发生当然就没那么神秘了。

But I think that was the big realization: how did intelligence evolve? If you don't need very specific conditions to create it, if you can create it just from the right kind of gradient and loss signal, then of course it's not so mysterious how it all happened.

我希望我能说出更有见地的话。有一件事一直让我惊讶：我原以为事情会更有条理一些。我原以为不同的认知能力可能都是相互关联的，背后有一个更大的秘密，但模型只是在不同时间学习不同东西。它可能非常擅长编程，但还不能证明素数定理。我想这对人类来说也有点类似，尽管它能做和不能做的事情并列在一起很奇怪。

I wish I had something more intelligent to say on that. One thing that's been surprising is I thought things might click into place a little more than they do. I thought different cognitive abilities might all be connected and there was more of one secret behind them, but the model just learns various things at different times. It can be very good at coding but can't quite prove the prime number theorem yet. I guess it's a little bit the same for humans, although it's weird the juxtaposition of things it can and cannot do.

我想主要的教训是，关于智能或智能如何运作的理论——很多这些词只是溶解成一个连续体。它们只是消失了。我更多地从我们眼前看到的东西来思考，而不是从智能的角度。

I guess the main lesson is having theories of intelligence or how intelligence works—again, a lot of these words just dissolve into a continuum. They just dematerialize. I think less in terms of intelligence and more in terms of what we see in front of us.

现在有两件事让我非常惊讶。一是这些导致损失的智能事物的不同路径是多么离散，而不是只有一个推理回路或一个通用智能。另一件事是，多年后回顾时，这会是那种让你觉得‘如果你看到这些平滑的 Scaling 曲线，为什么这还不明显？为什么你还没有完全信服？’的事情。

Now it's really surprising to me two things. One is how discrete these different paths of intelligent things that contribute to loss are, rather than just being one reasoning circuit or one general intelligence. The other thing is many years from now, it'll be one of those things looking back: why wasn't this obvious to you if you're seeing these smooth scaling curves? Why the time where you're not completely convinced?

你比其他 AI 公司的 CEO 更少公开露面。你不发 Twitter，除了这次之外也不怎么做播客。怎么回事？你为什么低调？

You've been less public than the CEOs of other AI companies. You're not posting on Twitter, you're not doing a lot of podcasts except for this one. What gives? Why are you off the radar?

我向往这样，并为此自豪。如果人们觉得我无聊且低调，这实际上正是我想要的。我见过一些我共事过的人，可以说 Twitter——虽然我指的是更广泛的东西，就是把你的动机强烈地与人群的认可或欢呼挂钩——我认为那会摧毁你的思想，在某些情况下还会摧毁你的灵魂。所以我刻意保持低调，以捍卫我独立思考的能力，不受他人认可的影响。

I aspire to this and I'm proud of this. If people think of me as boring and low profile, this is actually what I want. I've seen a number of people I've worked with that I think you could say Twitter—although I mean a broader thing, just attaching your incentives very strongly to the approval or cheering of a crowd—I think that can destroy your mind and in some cases your soul. So I've deliberately tried to be a little low profile to defend my ability to think about things intellectually in a way that's different from other people and isn't tinged by the approval of others.

我见过一些深度学习怀疑论者，他们在 Twitter 上以深度学习怀疑论者著称，然后即使我开始清楚他们有点改变了想法，他们喜欢这是他们在 Twitter 上的标签，无法改变他们的 Twitter 人设。我不太喜欢将公司个人化的趋势，那种 CEO 之间的笼斗方式。我认为它分散了人们对公司实际优点和担忧的注意力。我希望人们更多地评判那个无名的官僚机构及其动机，而不是我。每个人都想要一张友好的面孔，但实际上友好的面孔可能具有误导性。

I've seen cases of folks who were deep learning skeptics and they became known as deep learning skeptics on Twitter, and then even as it starts to become clear to me they've kind of changed their mind, they like this is their thing on Twitter and they can't change their Twitter persona. I don't really like the trend of personalizing companies, the whole cage match between CEOs approach. I think it distracts people from the actual merits and concerns of the company. I want people to judge the nameless bureaucratic institution, its incentives, more than they think in terms of me. Everyone wants a friendly face, but actually friendly faces can be misleading.

好吧，在这种情况下，这将是一次具有误导性的采访，因为这次访谈非常愉快。和你聊天很开心。我非常高兴你来参加播客，希望人们喜欢。

Okay, well in this case this will be a misleading interview because this has been a lot of fun. It was a blast to talk to you. I'm super glad you came on the podcast and hope people enjoy it.

谢谢你邀请我。

Thanks for having me.

大家好，希望你们喜欢这一集。一如既往，你能做的最有帮助的事情就是分享这个播客，把它发给你觉得可能喜欢的人，放到 Twitter、群聊等地方。这能传播开来。感谢收听。下次见。干杯。

Hey everybody, I hope you enjoyed that episode. As always, the most helpful thing you can do is just share the podcast, send it to people you think might enjoy it, put it in Twitter, your group chats, etc. It just spreads the word. Appreciate your listening. I'll see you next time. Cheers.