从规模的时代，到研究的时代

你知道什么很疯狂吗？这一切都是真的？

You know what's crazy? That all of this is real?

是的。什么意思？

Yeah. Meaning what?

你不觉得吗？

Don't you think so?

什么意思？

Meaning what?

就像所有这些 AI 的东西和这个领域。它正在发生。这不就是科幻小说里的情节吗？

Like all this AI stuff and all this area. That it's happening. Isn't it straight out of science fiction?

是的。另一件疯狂的事是缓慢起飞感觉如此正常。我们会在 AI 上投入 GDP 的 1%，我觉得这本来应该感觉是一件更大的事，但现在感觉就像你很快就习惯了。

Yeah. Another thing that's crazy is how normal the slow takeoff feels. The idea that we'd be investing 1% of GDP in AI, I feel like it would have felt like a bigger deal, but right now it just feels like you get used to things pretty fast.

是的。但这也有些抽象。这意味着什么？你在新闻里看到某家公司宣布了某个金额。你看到的就这些。到目前为止，它并没有以其他方式被真正感受到。

Yeah. But also it's kind of abstract. What does it mean? You see it in the news that such and such company announced such and such dollar amount. That's all you see. It's not really felt in any other way so far.

是的。

Yeah.

我们是不是该从这里开始？我觉得这是个有趣的讨论。我认为你的观点——从普通人的角度看，没什么不同——即使在奇点到来时也会继续成立。

Should we actually begin here? I think this is an interesting discussion. I think your point that from the average person's point of view, nothing is that different will continue being true even into the singularity.

不，我不这么认为。好吧，有意思。

No, I don't think so. Okay. Interesting.

所以我说的感觉不到不同是指某家公司宣布了某个难以理解的金额的投资。我觉得没人知道该怎么处理这个信息。

So the thing I was referring to not feeling different is that such and such company announced some difficult to comprehend dollar amount of investment. I don't think anyone knows what to do with that.

是的。

Yeah.

但我认为 AI 的影响会被感受到。AI 将渗透到经济中。有非常强大的经济力量推动这一点，我认为影响会非常强烈。

But I think the impact of AI is going to be felt. AI is going to be diffused through the economy. There are very strong economic forces for this and I think the impact is going to be felt very strongly.

你预计这种影响何时出现？我认为模型看起来比它们的经济影响所暗示的更聪明。

When do you expect that impact? I think the models seem smarter than their economic impact would imply.

是的，这是目前关于模型最令人困惑的事情之一。如何调和它们在评估中表现如此出色的事实？你看着评估，觉得那些评估很难，它们做得很好，但经济影响似乎远远落后。很难理解模型一方面能做这些惊人的事情，另一方面在某些情况下会重复自己两次。举个例子：你用 VIP 编码做某事，然后你遇到一个 bug，你告诉模型：「你能修复这个 bug 吗？」模型说：「哦天哪，你说得太对了。我有一个 bug。让我去修复它。」然后它引入了第二个 bug。然后你告诉它有了这个新 bug，它说：「哦天哪，我怎么能这样？你又对了。」然后它又把第一个 bug 带回来了。你可以在这两个 bug 之间来回切换。这怎么可能？

Yeah, this is one of the very confusing things about the models right now. How to reconcile the fact that they are doing so well on evals? You look at the evals and you go, those are pretty hard evals, they're doing so well, but the economic impact seems to be dramatically behind. It's very difficult to make sense of how can the model on the one hand do these amazing things and then on the other hand repeat itself twice in some situation. An example would be: you use VIP coding to do something and you go to some place and then you get a bug and then you tell the model, "Can you please fix the bug?" And the model says, "Oh my god, you're so right. I have a bug. Let me go fix that." And it introduces a second bug. And then you tell it you have this new second bug and it tells you, "Oh my god, how could I have done it? You're so right again." And brings back the first bug. And you can alternate between those. It's like, how is that possible?

是的。这确实表明有些奇怪的事情在发生。我有两个可能的解释。比较异想天开的解释是，也许强化学习训练让模型变得有点过于专注和狭隘，有点过于不察觉，尽管它也在其他方面让模型变得有察觉，因此它们无法做基本的事情。但还有另一个解释：以前人们做预训练时，训练什么数据的问题很容易回答，因为答案是一切。做预训练时，你需要所有数据。所以你不必考虑是这些数据还是那些数据。但当人们做强化学习训练时，他们确实需要考虑。他们说：「好吧，我们想要为这件事做这种强化学习训练，为那件事做那种强化学习训练。」据我所知，所有公司都有团队专门生产新的强化学习环境，并将其添加到训练组合中。那么问题来了，这些环境是什么？有太多的自由度。你可以生产出各种各样的环境。你可以做的一件事，我认为这是无意中发生的，就是人们从评估中汲取灵感。他们说：「嘿，我希望我们的模型在发布时表现非常好。我希望评估看起来很棒。」什么样的强化学习训练能帮助完成这个任务？我认为这确实发生了，并且可以解释很多正在发生的事情。如果你将这一点与模型泛化能力不足结合起来，就有可能解释我们看到的很多现象：评估表现与实际世界表现之间的脱节，而今天我们甚至不完全理解这到底意味着什么。

Yeah. It does suggest that something strange is going on. I have two possible explanations. The more whimsical explanation is that maybe RL training makes the models a little too single-minded and narrowly focused, a little too unaware, even though it also makes them aware in some other ways, and because of this they can't do basic things. But there is another explanation: back when people were doing pre-training, the question of what data to train on was answered because the answer was everything. When you do pre-training, you need all the data. So you don't have to think about whether it's this data or that data. But when people do RL training, they do need to think. They say, "Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing." From what I hear, all the companies have teams that just produce new RL environments and add them to the training mix. Then the question is, what are those? There are so many degrees of freedom. There is such a huge variety of environments you could produce. One thing you could do, and I think that's something that is done inadvertently, is that people take inspiration from the evals. They say, "Hey, I would love our model to do really well when we release it. I want the evals to look great." What would be RL training that could help on this task? I think that is something that happens and it could explain a lot of what's going on. If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing: this disconnect between eval performance and actual real-world performance, which is something we don't today exactly even understand what we mean by that.

我喜欢这个想法，即真正的奖励黑客是那些过于关注评估的人类研究人员。我认为有两种方式来理解或思考你刚刚指出的问题。一种是：如果仅仅通过在编程竞赛中变得超人类，模型不会自动变得更有品味，对如何改进你的代码库做出更好的判断，那么你应该扩展环境套件，这样你不仅测试它在编程竞赛中的最佳表现，它也应该能够为 X 事物或 Y 事物或 Z 事物制作最好的应用程序。另一种，也许这就是你暗示的，是说为什么一开始在编程竞赛中变得超人类不会让你在更广泛的意义上成为一个更有品味的程序员？也许要做的不是不断增加环境的数量和多样性，而是找到一种方法，让你从一个环境中学习并提高在其他事物上的表现。

I like this idea that the real reward hacking is human researchers who are too focused on the evals. I think there are two ways to understand or to try to think about what you have just pointed out. One is: if it's the case that simply by becoming superhuman at a coding competition, a model will not automatically become more tasteful and exercise better judgment about how to improve your codebase, then you should expand the suite of environments such that you're not just testing it on having the best performance in coding competition. It should also be able to make the best kind of application for X thing or Y thing or Z thing. Another, maybe this is what you're hinting at, is to say why should it be the case in the first place that becoming superhuman at coding competitions doesn't make you a more tasteful programmer more generally? Maybe the thing to do is not to keep stacking up the amount of environments and the diversity of environments to figure out an approach that lets you learn from one environment and improve your performance on something else.

我有一个来自人类的类比，可能有用。以你提到的竞赛编程为例。假设有两个学生。一个决定要成为最好的竞赛程序员。所以他们为这个领域练习了 10,000 小时。他们解决了所有问题，记住了所有证明技巧，变得非常擅长快速正确地实现所有算法。通过这样做，他们成为了最好的之一。第二个学生想，「哦，竞赛编程很酷，」也许他们练习了 100 小时，少得多，他们也做得很好。你认为哪个学生以后在职业生涯中会做得更好？

I have an analogy from humans which might be helpful. Take the case of competitive programming since you mentioned that. Suppose you have two students. One of them decides they want to be the best competitive programmer. So they practice 10,000 hours for that domain. They solve all the problems, memorize all the proof techniques, and become very skilled at quickly and correctly implementing all the algorithms. By doing so, they become one of the best. Student number two thinks, "Oh, competitive programming is cool," maybe they practice for 100 hours, much less, and they also did really well. Which one do you think is going to do better in their career later on?

第二个。

The second.

对。我认为这基本上就是正在发生的事情。

Right. And I think that's basically what's going on.

模型更像第一个学生，但更甚，因为我们会说，好吧，模型应该擅长竞技编程，那么我们就收集所有竞技编程题目，然后做一些数据增强，得到更多题目。我们在此基础上训练，于是就有了一个出色的竞技程序员。通过这个类比，我觉得更直观的是，如果训练得如此充分，所有算法和证明技巧都触手可及，那么很自然地，这种程度的准备并不一定会泛化到其他事物上。

The models are much more like the first student but even more because then we say okay so the model should be good at competitive programming so let's get every single competitive programming problem ever and then let's do some data augmentation so we have even more competitive programming problems. And we train on that and so now you got this great competitive programmer. And with this analogy I think it's more intuitive that yeah okay so if it's so well trained, it's like all the different algorithms and all the proof techniques are right at its fingertips and it's more intuitive that with this level of preparation it would not necessarily generalize to other things.

但第二个学生在进行 100 小时微调之前的行为，又有什么类比呢？

But then what is the analogy for what the second student is doing before they do the 100 hours of fine-tuning?

我认为他们天生就有那种特质。我觉得就是那种「天赋」。

I think it's like they have it. I think it's the it factor.

嗯。

Yeah.

对。我记得我本科时，有个同学就是这样。所以我知道这种人是存在的。

Right. And I know like when I was in undergrad, I remember there was a student like this that studied with me. So I know it exists.

嗯。我觉得区分它与预训练的作用很有意思。你刚才说预训练中我们不必选择数据，一种理解是这其实和一万小时练习没什么不同，只是那一万小时练习是免费的，因为它已经存在于预训练分布中。但也许你是在暗示，预训练并没有那么多泛化能力。预训练数据量很大，但未必比强化学习泛化得更好。

Yeah. I think it's interesting to distinguish it from whatever pre-training does. So one way to understand what you just said about we don't have to choose the data in pre-training is to say actually it's not dissimilar to the 10,000 hours of practice. It's just that you get that 10,000 hours of practice for free because it's already somewhere in the pre-training distribution. But maybe you're suggesting actually there's not that much generalization from pre-training. There's just so much data in pre-training but it's not necessarily generalizing better than RL.

预训练的主要优势在于数据量巨大，而且你不需要费心考虑往预训练里放什么数据。

Like the main strength of pre-training is that there is so much of it. And you don't have to think hard about what data to put into pre-training.

而且它是非常自然的数据，包含了人们所做的很多事情。

And it's a very kind of natural data and it does include in it a lot of what people do.

是的。人们的思想，以及整个世界通过文字投射出的许多特征。

Yeah. People's thoughts and a lot of the features of the whole world as projected by people onto text.

嗯。预训练试图用海量数据捕捉这些。它很难推理，因为很难理解模型依赖预训练数据的方式。每当模型犯错，会不会是因为某些东西碰巧在预训练数据中支持不足？「预训练支持」可能是个模糊的说法。我不知道还能补充什么有用的，但我不认为存在预训练的人类类比。有人提出过一些人类类比：一是人生命最初 15 到 18 年，那时他们不一定有经济产出，但在做让他们更好理解世界的事情；二是将进化看作 30 亿年的搜索，最终产生一个人类个体。我想知道你是否认为这些类比真的与预训练相似，或者如果不考虑预训练，你如何看待人类终身学习？

Yeah. And pre-training tries to capture that using a huge amount of data. It's very difficult to reason about because it's so hard to understand the manner in which the model relies on pre-training data. And whenever the model makes a mistake, could it be because something by chance is not as supported by the pre-training data? And support by pre-training is maybe a loose term. I don't know if I can add anything more useful on this, but I don't think there is a human analog to pre-training. Here are analogies that people have proposed for what the human analogy to pre-training is, and I'm curious to get your thoughts on why they're potentially wrong. One is to think about the first 18 or 15 or 13 years of a person's life when they aren't necessarily economically productive, but they are doing something that is making them understand the world better. And the other is to think about evolution as doing some kind of search for three billion years which then results in a human lifetime instance. I'm curious if you think either of these are actually analogous to pre-training or how would you think about at least what lifetime human learning is like if not pre-training.

我认为这两者与预训练都有一些相似之处，预训练试图扮演两者的角色，但也有很大不同。预训练数据量非常惊人。而人类即使经过 15 年，只接触了其中极小一部分数据，知道的东西却少得多。但他们所知道的东西，却理解得更深刻，而且在这个年龄，他们不会犯那些错误。

I think there are some similarities between both of these to pre-training, and pre-training tries to play the role of both of these, but I think there are some big differences as well. The amount of pre-training data is very staggering. And somehow a human being after even 15 years with a tiny fraction of that pre-training data they know much less. But whatever they do know they know much more deeply somehow and the mistakes like already at that age you would not make mistakes that are made.

嗯。还有一点，你可能会说它像进化，答案也许是，但我觉得进化可能更有优势。我记得读过这样一个案例：神经科学家通过研究脑损伤患者来了解大脑，有些人有最奇怪的症状。我想起一个案例：一个人因中风或事故导致情感处理区域受损，他不再有任何情绪，但仍然能言善辩，能解小谜题，测试表现正常，但他做决定的能力变得极差，选袜子要花几个小时，财务决策也很糟糕。这说明了我们内置的情绪在使我们成为可行的智能体方面起什么作用？联系到你关于预训练的问题，也许如果你足够擅长从预训练中提取一切，你也能得到那种能力，但这类东西可能无法从预训练中获得。那显然不完全是直接的情绪，而更像某种价值函数，告诉你每个决策的最终奖励应该是什么。你认为这不会隐含地从预训练中得来吗？

Yeah. There is another thing you might say could it be something like evolution and the answer is maybe but in this case I think evolution might actually have an edge. I remember reading about this case where some neuroscientists study people with brain damage to different parts of the brain, and some people have the most strange symptoms. There was one case that comes to mind. I read about this person who had some kind of brain damage that took out his emotional processing. So he stopped feeling any emotion and as a result he still remained very articulate and he could solve little puzzles and on tests he seemed to be just fine but he felt no emotion and he became somehow extremely bad at making any decisions at all. It would take him hours to decide on which socks to wear and he would make very bad financial decisions. What does it say about the role of our built-in emotions in making us like a viable agent essentially? And to connect to your question about pre-training, it's like maybe if you are good enough at getting everything out of pre-training you could get that as well, but that's the kind of thing which may or may not be possible to get from pre-training. What is that clearly not just directly emotion? And it seems like some almost value function like thing which is giving you telling you which decision to be like what the end reward for any decision should be. And you think that doesn't sort of implicitly come from?

我认为有可能。我只是说这不是百分之百明显的。

I think it could. I'm just saying it's not 100% obvious.

嗯。但那是什么样的呢？你怎么看待情绪，情绪在机器学习中的类比是什么？

Yeah. But what is that like? How do you think about emotions and what is the ML analogy for emotions?

它应该是某种价值函数之类的东西。

It should be some kind of a value function thing.

嗯。但我不认为有很好的机器学习类比，因为目前价值函数在人们做的事情中并不扮演非常突出的角色。

Yeah. But I don't think there is a great ML analogy because right now value functions don't play a very prominent role in the things people do.

如果你愿意，也许值得为听众定义一下什么是价值函数。

It might be worth defining for the audience what a value function is if you want to do that.

我很乐意。当人们做强化学习时，目前的做法是：你有一个神经网络，给它一个问题，然后让模型去解决，模型可能采取数千或数十万个动作或思考，然后产生一个解决方案。解决方案产生后，用得分来为轨迹中的每一个动作提供训练信号。

I'll be very happy to do that. So when people do reinforcement learning, the way reinforcement learning is done right now, how do they train those agents? You have your neural net and you give it a problem and then you tell the model go solve it and the model takes maybe thousands or hundreds of thousands of actions or thoughts and then it produces a solution. The solution is created and then the score is used to provide a training signal for every single action in your trajectory.

所以这意味着，如果你在做一件耗时很长的事情，如果你在训练一个需要很长时间才能解决的任务，那么在你提出一个解决方案之前，你根本不会学到任何东西。这就是强化学习朴素的做法。O1 R1 表面上也是这么做的。价值函数会说，好吧，也许我有时（并非总是）能告诉你做得好还是不好。价值函数的概念在某些领域比其他领域更有用。例如，下棋时你丢了一个棋子，你就知道自己搞砸了。你不需要下完整盘棋就知道刚才那步是坏的，因此之前的所有步骤也是坏的。所以价值函数让你不必一直等到最后。假设你开始尝试某种数学或编程问题，你正在探索一个特定的解决方向，经过一千步思考后，你得出结论这个方向没有前途。一旦你得出这个结论，你就可以在之前一千步决定走这条路的时候获得奖励信号。你会说，「哦，下次在类似情况下我不应该走这条路」，这远在你实际提出解决方案之前。

So that means that if you are doing something that goes for a long time, if you're training a task that takes a long time to solve, you will do no learning at all until you come up with a proposed solution. That's how reinforcement learning is done naively. That's how O1 R1 ostensibly are done. The value function says something like, okay, maybe I could sometimes, not always, tell you if you're doing well or badly. The notion of a value function is more useful in some domains than others. So for example, when you play chess and you lose a piece, you know you messed up. You don't need to play the whole game to know that what I just did was bad and therefore whatever preceded it was also bad. So the value function lets you short circuit the wait until the very end. Let's suppose that you started to pursue some kind of, let's suppose that you are doing some kind of a math thing or a programming thing and you're trying to explore a particular solution direction and after, let's say after a thousand steps of thinking you concluded that this direction is unpromising. As soon as you conclude this, you could already get a reward signal a thousand time steps previously when you decided to pursue down this path. You say, 'Oh, next time I shouldn't pursue this path in a similar situation' long before you actually came up with a proposed solution.

这在 DeepCar 那篇论文里提到过，轨迹空间如此之大，以至于可能很难从中间轨迹和价值学到映射，而且考虑到在编程中，你会有错误的想法，然后回头，然后改变一些东西。

This was in the DeepCar one paper, that the space of trajectories is so wide that maybe it's hard to learn a mapping from an intermediate trajectory and value, and also given that in coding for example, you'll have the wrong idea, then you'll go back, then you'll change something.

这听起来对深度学习太缺乏信心了。我的意思是，当然可能很难，但深度学习没有做不到的。

This sounds like such lack of faith in deep learning. I mean sure it might be difficult but nothing deep learning can't do.

所以我的期望是价值函数应该有用，我完全相信它们将来会被使用，即使现在还没有。我之前提到那个情感中枢受损的人，更多是想说，也许这表明人类的价值函数在某种程度上受到情感的调节，这种调节是进化硬编码的，也许这对人们在世界上有效行动很重要。

So my expectation is that value functions should be useful and I fully expect that they will be used in the future if not already. What was I alluding to with the person whose emotional center got damaged is more that maybe what it suggests is that the value function of humans is modulated by emotions in some important way that's hardcoded by evolution and maybe that is important for people to be effective in the world.

这正是我原本打算问你的。关于情感作为价值函数，有一点非常有趣：它们具有如此大的效用，同时又相当简单易懂，这令人印象深刻。所以我有两个回应。我同意，与我们学习的东西和我们正在讨论的事情相比，情感相对简单。它们甚至可能简单到可以用人类可理解的方式绘制出来。我认为这样做会很酷。但在效用方面，我认为存在一种复杂性与鲁棒性的权衡：复杂的东西可能非常有用，但简单的东西在非常广泛的情况下也非常有用。所以我认为，解释我们所见现象的一种方式是，这些情感基本上是从我们的哺乳动物祖先进化而来，然后在人类进化过程中稍微微调了一点。我们确实有相当多的社会情感，这是哺乳动物可能缺乏的，但它们并不复杂，正因为不复杂，它们在这个与我们过去生活的世界截然不同的世界里如此有效地服务于我们。实际上它们也会犯错。例如，我们的情感——我不知道饥饿算不算情感，这有争议——但比如我们直觉的饥饿感，在这个食物丰富的世界里并不能正确引导我们。

That's the thing I was actually planning on asking you. There's something really interesting about emotions as a value function, which is that it's impressive that they have this much utility while still being rather simple to understand. So I have two responses. I do agree that compared to the kind of things that we learn and the things we are talking about, the kind of ads we talking about, emotions are relatively simple. They might even be so simple that maybe you could map them out in a human understandable way. I think it would be cool to do. In terms of utility though, I think there is a thing where there is this complexity-robustness trade-off where complex things can be very useful but simple things are very useful in a very broad range of situations. And so I think one way to interpret what we are seeing is that we've got these emotions that essentially evolved mostly from our mammal ancestors and then fine-tuned a little bit while we were hominids, just a bit. We do have a decent amount of social emotions though which mammals may lack, but they're not very sophisticated and because they're not sophisticated they serve us so well in this very different world compared to the one that we've been living in. Actually they also make mistakes. For example, our emotions, well I don't know if hunger counts as an emotion, it's debatable, but I think for example our intuitive feeling of hunger is not succeeding in guiding us correctly in this world with an abundance of food.

人们一直在讨论扩展数据、扩展参数、扩展算力。有没有更通用的方式来思考扩展？其他扩展维度是什么？

People have been talking about scaling data, scaling parameters, scaling compute. Is there a more general way to think about scaling? What are the other scaling axes?

这里有一个我认为可能正确的视角。过去机器学习的工作方式是，人们只是用各种东西尝试，试图得到有趣的结果。这就是过去的情况。然后扩展的洞察出现了，对吧？缩放定律，GPT-3。突然每个人都意识到我们应该扩展。这是语言如何影响思维的一个例子。「扩展」只是一个词，但它是一个如此强大的词，因为它告诉人们该做什么。他们说，「好吧，让我们尝试扩展事物。」所以你说，好吧，我们在扩展什么？预训练是一个可以扩展的东西，它是一个特定的扩展配方。预训练的重大突破在于意识到这个配方很好。所以你说，嘿，如果你把一些算力和一些数据混合到一定规模的神经网络中，你会得到结果，而且你知道如果你只是扩大配方，结果会更好。这也很棒。公司喜欢这个，因为它提供了一种非常低风险的投资方式。相比之下，将资源投入研究要困难得多。你知道，如果你做研究，你需要有研究人员去研究并想出一些东西，而获取更多数据、更多算力，你知道，你会从预训练中得到一些东西。事实上，根据人们在 Twitter 上说的各种事情，似乎 Gemini 找到了一种从预训练中获得更多收益的方法。但在某个时候，预训练会用完数据。数据显然是有限的。那么接下来你做什么？要么你做一些增强版的预训练，不同于之前做过的配方，要么你做强化学习或其他东西。但现在算力很大，算力现在已经非常大了。从某种意义上说，我们又回到了研究的时代。所以也许有另一种说法。直到 2020 年，从 2012 年到 2020 年，是研究的时代。现在从 2020 年到 2025 年，是扩展的时代，或者差不多。让我们给这些年份加上误差线，因为人们说这太棒了。你必须更多地扩展。继续扩展。这个词「扩展」。但现在规模如此之大。真的相信如果规模再大 100 倍一切都会完全不同吗？当然会不同，但真的相信只要把规模扩大 100 倍一切就会转变吗？我不认为这是真的。所以又回到了研究的时代，只是有了大型计算机。

So here is a perspective I think might be true. The way ML used to work is that people would just think of it with stuff and try to get interesting results. That's what's been going on in the past. Then the scaling insight arrived, right? Scaling laws, GPT-3. And suddenly everyone realized we should scale. And this is an example of how language affects thought. 'Scaling' is just one word, but it's such a powerful word because it informs people what to do. They say, 'Okay, let's try to scale things.' And so you say, okay so what are we scaling? And pre-training was a thing to scale, it was a particular scaling recipe. The big breakthrough of pre-training is the realization that this recipe is good. So you say, hey if you mix some compute with some data into a neural net of a certain size you will get results and you will know that it will be better if you just scale the recipe up. And this is also great. Companies love this because it gives you a very low-risk way of investing your resources. It's much harder to invest your resources in research. Compare that. You know, if you research, you need to have like go forth researchers and research and come up with something, versus get more data, get more compute. You know, you'll get something from pre-training. And indeed, based on various things people say on Twitter, maybe it appears that Gemini have found a way to get more out of pre-training. At some point though, pre-training will run out of data. The data is very clearly finite. And so then, okay, what do you do next? Either you do some kind of a souped-up pre-training, different recipe from the one you've done before, or you're doing a RL or maybe something else. But now that compute is big, compute is now very big. In some sense, we are back to the age of research. So maybe here's another way to put it. Up until 2020, from 2012 to 2020, it was the age of research. Now from 2020 to 2025, it was the age of scaling, or maybe plus minus. Let's add error bars to those years because people say this is amazing. You got to scale more. Keep scaling. The one word 'scaling'. But now the scale is so big. Is the belief really that oh it's so big but if you had 100x more everything would be so different? It would be different for sure but is the belief that if you just 100x the scale everything would be transformed? I don't think that's true. So it's back to the age of research again just with big computers.

这是一个非常有趣的表述。但让我问你刚才提出的问题。

That's a very interesting way to put it. But let me ask you the question you just posed then.

我们在扩展什么？拥有一个配方意味着什么？因为我不太清楚在预训练中存在的那种近乎物理定律的清晰关系——那是数据、算力或参数与损失之间的幂律关系。我们应该寻求什么样的关系？又该如何思考这个新配方可能的样子？我们已经见证了一种扩展类型向另一种的转变，从预训练到强化学习。现在，人们在扩展强化学习。根据推特上人们的说法，他们目前在强化学习上花费的算力比预训练还多，因为强化学习确实能消耗大量算力。你要进行非常长的 rollout。

What are we scaling and what would it mean to have a recipe? Because I'm not aware of a very clean relationship that almost looks like a law of physics which existed in pre-training that was a power law between data or compute or parameters and loss. What is the kind of relationship we should be seeking and how should we think about what this new recipe might look like? So, we've already witnessed a transition from one type of scaling to a different type of scaling, from pre-training to RL. Now, people are scaling RL. Based on what people say on Twitter, they spend more compute on RL than on pre-training at this point because RL can actually consume quite a bit of compute. You do very long rollouts.

是的。生成这些 rollout 需要大量算力，而每次 rollout 获得的学习量相对较少。所以你真的可以消耗很多算力。我可以想象，在这个阶段，我甚至不会称之为扩展。我会说，嘿，你在做什么？你正在做的事情是不是你能做的最有生产力的事情？

Yes. So it takes a lot of compute to produce those rollouts and then you get a relatively small amount of learning per rollout. So you can really spend a lot of compute. I could imagine, at this stage, I wouldn't even call it scaling. I would say, hey, what are you doing? And is the thing you are doing the most productive thing you could be doing?

你能找到一种更有生产力的方式使用你的算力吗？我们之前讨论过价值函数的事情。也许一旦人们擅长价值函数，他们就会更有效地利用资源。如果你找到一种完全不同的训练模型的方法，你能说这是扩展还是仅仅在利用资源？我认为这变得有些模糊。在以前的研究时代，人们会说，嘿，试试这个和这个，试试那个和那个，哦看，有趣的事情发生了。我认为我们会回到那种状态。

Can you find a more productive way of using your compute? We discussed the value function business earlier. Maybe once people get good at value functions, they will be using their resources more productively. And if you find a whole other way of training models, you could say, is this scaling or is it just using your resources? I think it becomes a little bit ambiguous. In the age of research back then, people would say, hey, let's try this and this and this, let's try that and that and that, oh look, something interesting is happening. I think there will be a return to that.

那么，如果我们回到研究时代，退一步说，配方中哪一部分是我们最需要思考的？当你说价值函数时，人们已经在尝试当前的配方，比如用大语言模型作为评判者等等，你可以说那是一个价值函数，但听起来你心里有更根本的东西。我们是否需要重新思考预训练本身，而不仅仅是在该过程末尾添加更多步骤？

So if we're back in the era of research, stepping back, what is the part of the recipe that we need to think most about? When you say value function, people are already trying the current recipe, but then having LLM as a judge and so forth, you could say that's a value function, but it sounds like you have something much more fundamental in mind. Do we need to rethink pre-training at all, and not just add more steps to the end of that process?

是的。关于价值函数的讨论，我觉得很有趣。我想强调的是，我认为价值函数会让强化学习更高效，这确实有影响。但我认为，用价值函数能做的任何事情，不用它也能做，只是更慢。

Yeah. The discussion about value function, I think it was interesting. I want to emphasize that I think the value function is something that's going to make RL more efficient, and I think that makes a difference. But I think anything you can do with a value function you can do without, just more slowly.

嗯。

Mhm.

我认为最根本的问题是，这些模型在泛化方面远不如人类。

The thing which I think is the most fundamental is that these models somehow generalize dramatically worse than people.

是的。

Yes.

而且这非常明显。这似乎是一个非常根本的问题。

And it's super obvious. That seems like a very fundamental thing.

好的，所以这是泛化的关键。有两个子问题。一个是关于样本效率：为什么这些模型学习需要比人类多得多数据？第二个，甚至与所需数据量无关：为什么把我们要教的东西教给模型比教给人难那么多？对于人类，我们不一定需要可验证的奖励。你现在可能正在指导一群研究员，和他们交谈，展示你的代码和你的思考方式，他们从中学习你的思维方式以及如何做研究。你不需要为他们设定一个可验证的奖励，比如，好的，这是课程的下一个部分，现在这是下一个部分，哦，这次训练不稳定，我们得……没有这种繁琐的定制过程。所以也许这两个问题实际上在某种程度上是相关的，但我很想探索第二个问题，它感觉更像持续学习，而第一个问题感觉就是样本效率。

Okay, so this is the crux of generalization. There are two sub-questions. One is about sample efficiency: why should it take so much more data for these models to learn than humans? There's a second, even separate from the amount of data it takes: why is it so hard to teach the thing we want to a model compared to a human? For a human, we don't necessarily need a verifiable reward. You're probably mentoring a bunch of researchers right now, talking with them, showing them your code and how you think, and from that they're picking up your way of thinking and how they should do research. You don't have to set a verifiable reward for them like, okay, this is the next part of the curriculum, and now this is the next part, and oh, this training was unstable, and we gotta... there's not this shleppy bespoke process. So perhaps these two issues are actually related in some way, but I'd be curious to explore this second thing which feels more like continual learning, and this first thing which feels just like sample efficiency.

是的。你实际上可以思考一下，人类样本效率的一个可能解释需要考虑进化。进化给了我们少量但最有用的信息。对于视觉、听觉和运动这类事情，我认为有相当充分的证据表明进化实际上给了我们很多。

Yeah. You could actually wonder, one possible explanation for human sample efficiency that needs to be considered is evolution. Evolution has given us a small amount of the most useful information possible. For things like vision, hearing, and locomotion, I think there's a pretty strong case that evolution actually has given us a lot.

嗯。

Mhm.

例如，人类的灵巧性远超……我的意思是，如果让机器人在模拟中进行大量训练，它们也可以变得灵巧。但在现实世界中训练机器人像人一样快速掌握新技能似乎遥不可及。这里你可以说，哦，是的，运动，我们所有的祖先都需要出色的运动能力，松鼠也是，所以也许我们有一些难以置信的先验。视觉也是如此。我相信 Yann LeCun 指出，儿童在 10 小时的练习后就能学会开车，这是真的，但我们的视觉如此之好。至少对我来说，当我记得自己 5 岁时，那时我对汽车非常兴奋，而且我很确定我的汽车识别能力已经足以胜任自动驾驶。作为一个 5 岁孩子，你接触不到那么多数据。你大部分时间待在父母家里，所以数据多样性很低。但你可以说那可能也是进化。但语言、数学和编码，可能不是。

For example, human dexterity far exceeds... I mean, robots can become dexterous too if you subject them to a huge amount of training in simulation. But to train a robot in the real world to quickly pick up a new skill like a person does seems very out of reach. And here you could say, oh yeah, locomotion, all our ancestors needed great locomotion, squirrels too, so maybe we've got some unbelievable prior. You could make the same case for vision. I believe Yann LeCun made the point that children learn to drive after 10 hours of practice, which is true, but our vision is so good. At least for me, when I remember myself being 5 years old, I was very excited about cars back then, and I'm pretty sure my car recognition was more than adequate for self-driving already. As a 5-year-old, you don't get to see that much data. You spend most of your time in your parents' house, so you have very low data diversity. But you could say maybe that's evolution, too. But then language, math, and coding, probably not.

这似乎仍然比模型好。我的意思是，显然模型在语言、数学和编码方面比普通人强，但它们在学**面比普通人强吗？

It still seems better than models. I mean, obviously models are better than the average human at language, math, and coding, but are they better at the average human at learning?

哦，是的。绝对。我想说的是，语言、数学和编码，尤其是数学和编码，表明让人擅长学习的东西可能不是一个复杂的先验，而是更根本的东西。

Oh, yeah. Absolutely. What I meant to say is that language, math, and coding, especially math and coding, suggests that whatever it is that makes people good at learning is probably not so much a complicated prior but something more fundamental.

等等，我不确定我理解了。为什么会这样？那么考虑一种技能，人们表现出某种高度的可靠性或……

Wait, I'm not sure I understood. Why should that be the case? So consider a skill that people exhibit some kind of great reliability or...

如果这种技能对我们的祖先在数百万年、数亿年间非常有用，你可以说也许人类擅长它是因为进化，因为我们有先验。

If the skill is one that was very useful to our ancestors for many millions of years, hundreds of millions of years, you could argue that maybe humans are good at it because of evolution, because we have a prior.

一个以某种非常不明显的方式编码的进化先验。

An evolutionary prior that's encoded in some very nonobvious way.

是的。这让我们如此擅长它。

Yeah. That somehow makes us so good at it.

但如果人们在直到最近才存在的领域表现出强大的能力、可靠性、鲁棒性和学习能力，那么这更多地表明人类可能只是有更好的机器学习，仅此而已。

But if people exhibit great ability, reliability, robustness, ability to learn in a domain that really did not exist until recently, then this is more an indication that people might have just better machine learning, period.

嗯。但那么我们该如何思考那是什么？是不是……

Mhm. But then how should we think about what that is? Is it a matter of...

这对应的机器学习类比是什么？有几个有趣的点：它需要的样本更少，更无监督。你不必设置一个验证器，就像青少年学开车一样。青少年学开车并不是从某种预置的可验证奖励中学习，而是来自他们与机器和环境的互动。然而，它需要的样本却少得多。它看起来更无监督，更鲁棒，非常鲁棒。人类的鲁棒性真是惊人。

What is the ML analogy for what? There's a couple interesting things about it. It takes fewer samples. It's more unsupervised. You don't have to set a ver like a child learning to drive a car. Children are not learning to drive a car. A teenager learning how to drive a car is like not exactly getting some pre-built verifiable reward. It comes from their interaction with the machine and with the environment. And yet it takes much fewer samples. It seems more unsupervised. It seems more robust. Much more robust. The robustness of people is really staggering.

是的。

Yeah.

那么，你有没有一个统一的思考方式，来解释为什么所有这些事情同时发生？什么样的机器学习类比可以实现这一点？这就是你一直在问的问题之一：青少年司机如何在没有外部老师的情况下自我纠正并从经验中学习？

So like okay. And do you have a unified way of thinking about why are all these things happening at once? What is the ML analogy that could realize something like this? So this is where, you know, one of the things that you've been asking about is how can the teenage driver kind of self-correct and learn from their experience without an external teacher.

答案是，他们有自己的价值函数，对吧？他们有一个通用感知，顺便说一句，这在人类中也非常鲁棒——无论人类价值函数是什么，除了少数成瘾例外，它实际上非常非常鲁棒。所以对于学开车的青少年来说，他们开始开车时，立刻就有一种感觉，知道自己开得怎么样、有多差、多不自信，然后他们就能看到。当然，任何青少年的学习速度都非常快，10 小时后就能上路了。

And the answer is, well, they have their value function, right? They have a general sense which is also by the way extremely robust in people like whatever it is the human value function, with a few exceptions around addiction, it's actually very very robust. And so for something like a teenager that's learning to drive, they start to drive and they already have a sense of how they're driving immediately, how badly, how unconfident, and then they see. Okay. And then of course the learning speed of any teenager is so fast after 10 hours you're good to go.

是的。人类似乎有某种解决方案，但我很好奇他们是怎么做到的，为什么这么难？我们需要如何重新概念化我们训练模型的方式，才能让类似的事情成为可能？

Yeah. It seems like humans have some solution, but I'm curious about like well how are they doing it and like why is it so hard to like how do we need to reconceptualize the way we're training models to make something like this possible?

你问了一个很好的问题，我对此有很多看法。但不幸的是，我们生活在一个并非所有机器学习想法都能自由讨论的世界，这就是其中之一。所以可能有办法做到。我认为这是可以做到的。人类能做到这一点，我认为就是证明。但可能还有另一个障碍：人类神经元实际做的计算可能比我们想象的更多。如果这是真的，并且起了重要作用，那么事情可能会更困难。但无论如何，我确实认为这指向了某个机器学习原理的存在，我有一些看法。但不幸的是，环境使得很难详细讨论。

You know that is a great question to ask and it's a question I have a lot of opinions about. But unfortunately we live in a world where not all machine learning ideas are discussed freely and this is one of them. So there's probably a way to do it. I think it can be done. The fact that people are like that I think it's a proof that it can be done. There may be another blocker though which is there is a possibility that the human neurons actually do more compute than we think. And if that is true and if that plays an important role then things might be more difficult. But regardless I do think it points to the existence of some machine learning principle that I have opinions on. But unfortunately, circumstances make it hard to discuss in detail.

尽管没人听这个播客，伊利亚。

Even though nobody listens to this podcast, Ilya.

是的。

Yeah.

我得说，为伊利亚做准备相当困难，因为我和其他人都不知道他在做什么，也不知道 SSI 试图做什么。我没有任何依据来提出我的问题，唯一能做的就是尝试从第一性原理思考什么是瓶颈，因为显然伊利亚正在以某种方式解决它们。这个问题的一部分涉及思考强化学习的 Scaling，因为每个人都在问强化学习能泛化得多好，以及我们如何让它泛化得更好。为此，我读了一篇最近关于强化学习 Scaling 的论文，它显示强化学习的学习曲线实际上是一个 S 形。我觉得这非常奇怪。为什么它会是一个 S 形：长时间学习很少，然后快速学习很多，最后渐近？这与你在预训练中看到的幂律非常不同，在预训练中模型一开始学习很多，然后随时间越来越少。这让我想起了一次与一位研究员朋友的对话后记下的笔记，他指出，要找到正确答案所需的样本数量，与你当前概率分布与目标概率分布的差异呈指数关系。我在想这两个想法是如何关联的。我隐约觉得它们应该有关联，但我真的不知道如何关联。我没有数学背景，所以无法形式化。但我想知道 Gemini 3 是否能帮我。于是我拍下了我的笔记本和那篇论文，把它们都放入 Gemini 3 的上下文中，让它找出联系。它思考了一会儿，然后意识到，在强化学习中，从单个是/否结果中获得的信息的正确建模方式是作为随机二元变量的熵。它画了一张图，显示了随着通过率增加，强化学习与监督学习中每个样本获得的比特数如何变化。当我看到 Gemini 3 制作的图表时，立刻有很多事情开始变得清晰。然后我想看看这个理论是否有任何经验基础。所以我让 Gemini 编写我的实验代码，以显示损失改进是否随通过率以这种方式缩放。我直接拿了 Gemini 输出的代码，复制粘贴到 Google Colab 笔记本中，就能运行这个玩具机器学习实验并可视化结果，没有出现任何错误。有趣的是，结果看起来与预期相似但不完全相同。于是我下载了这张图，放入 Gemini，问它这是怎么回事。我得出了一个我认为正确的假设：我们在开始时通过固定学习率限制了监督学习能改进多少，实际上我们应该随时间降低学习率。这让我们直观地理解了为什么在实践中我们有学习率调度器来随时间降低学习率。我从提出这个模糊的初始问题，到建立理论理解，再到运行一些玩具机器学习实验，整个过程都用了 Gemini 3。这感觉像是第一个能真正提出我未曾预料到的新联系的模型。现在它实际上成了我集思广益思考问题的新方式的首选。如果你想了解更多关于强化学习 Scaling 的内容，可以看看我写的博客文章，其中有一点 Gemini 3 的帮助。如果你想亲自试试 Gemini 3，请访问 gemini.google。

So, I have to say that prepping for Ilya was pretty tough because neither I nor anybody else had any idea what he's working on and what SSI is trying to do. I had no basis to come up with my questions and the only thing I could go off honestly was trying to think from first principles about what are the bottlenecks to hi because clearly Ilya is working on them in some way. Part of this question involved thinking about RL scaling because everybody's asking how well RL will generalize and how we can make it generalize better. As part of this I was reading this paper that came out recently on RL scaling and it showed that actually the learning curve on RL looks like a sigmoid. I found this very curious. Why should it be a sigmoid where it learns very little for a long time and then it quickly learns a lot and then it asymptotes. This is very different from the power law you see in pre-training where the model learns a bunch at the very beginning and then less and less over time. And it actually reminded me of a note that I had written down after I had a conversation with a researcher friend where he pointed out that the number of samples that you need to take in order to find a correct answer scales exponentially with how different your current probability distribution is from the target probability distribution. And I was thinking about how these two ideas are related. I had this vague idea that they should be connected, but I really didn't know how. I don't have a math background, so I couldn't really formalize it. But I wondered if Gemini 3 could help me out here. And so I took a picture of my notebook and I took the paper and I put them both in the context of Gemini 3 and I asked it to find the connection. And it thought a bunch and then it realized that the correct way to model the information you gain from a single yes or no outcome in RL is as the entropy of a random binary variable. It made a graph which showed how the bits you gain for a sample in RL versus supervised learning scale as a pass rate increases. And as soon as I saw the graph that Gemini 3 made, immediately a ton of things started making sense to me. Then I wanted to see if there was any empirical basis to this theory. So I asked Gemini to code my experiment to show whether the improvement in loss scales in this way with pass rate. I just took the code that Gemini outputed. I copy pasted it into a Google Collab notebook and I was able to run this toy ML experiment and visualize its results without a single bug. It's interesting because the results look similar but not identical to what we should have expected. And so I downloaded this chart and I put it into Gemini and I asked it what is going on here. I came up with a hypothesis that I think is actually correct which is that we're capping how much supervised learning can improve in the beginning by having a fixed learning rate and in fact we should decrease the learning rate over time. It actually gives us an intuitive understanding for why in practice we have learning rate schedulers that decrease the learning rate over time. I did this entire flow from coming up with this vague initial question to building a theoretical understanding to running some toy ML experiments all with Gemini 3. This feels like the first model where it can actually come up with new connections that I wouldn't have anticipated. It's actually now become the default place I go to when I want to brainstorm new ways to think about a problem. If you want to read more about RL scaling, you can check out the blog post that I wrote with a little help from Gemini 3. And if you want to check out Gemini 3 yourself, go to gemini.google.

我很好奇，你说我们回到了一个研究时代。你从 2012 年到 2020 年都在那里，你有没有……是的。如果我们回到研究时代，现在的氛围会是什么？例如，即使在 AlexNet 之后，用于运行实验的算力一直在增加，前沿系统的规模也在不断增加。

I am curious if you say we are back in an era of research. You were there from 2012 to 2020 and do you have... Yeah. What is now the vibe going to be if we go back to the era of research? For example, even after AlexNet, the amount of compute that was used to run experiments kept increasing and the size of frontier systems kept increasing.

你认为现在这个研究时代仍然需要大量的算力吗？你认为需要回到档案中阅读旧论文吗？你在谷歌、OpenAI 和斯坦福这些地方，当研究氛围更浓厚时，氛围是怎样的？我们应该期待社区中出现什么样的事情？

And do you think now that this era of research will still require tremendous amounts of compute? Do you think it will require going back into the archives and reading old papers? What was the vibe like at Google, OpenAI, and Stanford when there was more of a research vibe? What kind of thing should we be expecting in the community?

所以 Scaling（规模扩张）时代的一个后果是，Scaling 吸走了房间里所有的空气。

So one consequence of the age of scaling is that scaling sucked all the air out of the room.

是的。

Yeah.

因为 Scaling 吸走了房间里所有的空气，每个人都开始做同样的事情。我们到了一个地步：公司比想法多得多。

And because scaling sucked all the air out of the room, everyone started to do the same thing. We got to the point where we are in a world where there are more companies than ideas by quite a bit.

实际上，硅谷有句老话说想法很廉价，执行才是一切，人们经常这么说。

Actually, there is this Silicon Valley saying that ideas are cheap, execution is everything, and people say that a lot.

是的，这有一定道理。但后来我在推特上看到有人说，如果想法这么廉价，为什么没人有想法呢？

Yeah. And there is truth to that. But then I saw someone say on Twitter something like, if ideas are so cheap, how come no one's having any ideas?

我也认为这是真的。如果你从瓶颈的角度思考研究进展，有几个瓶颈。一个是想法，另一个是你实现想法的能力。

And I think it's true too. If you think about research progress in terms of bottlenecks, there are several bottlenecks. One of them is ideas, and one of them is your ability to bring them to life.

是的，可能是算力，但也包括工程。所以如果你回到 90 年代，有些人有很好的想法，如果他们有更大的计算机，也许他们能证明这些想法是可行的，但他们没有。所以他们只能做非常小的演示，无法说服任何人。

Yeah, which might be compute but also engineering. So if you go back to the '90s, let's say, you had people who had pretty good ideas, and if they had much larger computers, maybe they could demonstrate that their ideas were viable, but they could not. So they could only have very small demonstrations that did not convince anyone.

是的。

Yeah.

所以瓶颈是算力。然后在 Scaling 时代，计算机大幅增加，当然需要多少算力是个问题，但算力已经足够大，以至于你并不明显需要更多算力来证明某个想法。我给你打个比方。AlexNet 是在两块 GPU 上构建的，这就是它使用的全部算力。Transformer 是在 8 到 64 块 GPU 上构建的。2017 年没有一篇 Transformer 论文的实验使用超过 64 块 GPU，这相当于今天的两块 GPU。所以 ResNet，很多这些，甚至 O1 推理也不是世界上最消耗算力的东西。所以对于研究，你肯定需要一定量的算力，但远非明显需要绝对最大的算力。

So the bottleneck was compute. Then in the age of scaling, computers increased a lot, and of course there is a question of how much compute is needed, but compute is large enough such that it's not obvious that you need that much more compute to prove some idea. I'll give you an analogy. AlexNet was built on two GPUs. That was the total amount of compute used for it. The transformer was built on 8 to 64 GPUs. No single transformer paper experiment used more than 64 GPUs in 2017, which would be like two GPUs of today. So ResNet, many of these, even the O1 reasoning was not the most compute-heavy thing in the world. So for research, you definitely need some amount of compute, but it's far from obvious that you need the absolutely largest amount of compute ever for research.

你可能会争辩，而且我认为这是真的，如果你想构建绝对最好的系统，拥有更多算力是有帮助的，尤其是当每个人都处于同一范式时，算力就成为了一个重要的区分因素。

You might argue, and I think it is true, that if you want to build the absolutely best system, it helps to have much more compute, especially if everyone is within the same paradigm, then compute becomes one of the big differentiators.

是的，我想虽然这些想法有可能被开发出来，我是在问你历史，因为你当时在场。我不确定实际发生了什么，但听起来用最少的算力开发这些想法是可能的，但 Transformer 并没有立即出名。它变成了每个人都开始做的东西，然后在其上进行实验和构建，因为它在越来越高的算力水平上得到了验证。

Yeah, I guess while it was possible to develop these ideas, I'm asking you for the history because you were actually there. I'm not sure what actually happened, but it sounds like it was possible to develop these ideas using minimal amounts of compute, but the transformer didn't immediately become famous. It became the thing everybody started doing and then started experimenting on top of and building on top of because it was validated at higher and higher levels of compute.

正确。如果你在 SSI 有 50 个不同的想法，没有其他前沿实验室拥有的那种算力，你怎么知道哪个是下一个 Transformer，哪个是脆弱的？所以我可以对此发表评论。简短的回答是，对我们来说，SSI 用于研究的算力其实并不小，我想解释为什么。简单的数学可以解释为什么我们拥有的算力在研究上实际上比人们想象的要大得多。SSI 筹集了 30 亿美元，这从绝对意义上讲并不小，但你可以说看看其他公司筹集了更多。但他们的大量算力用于推理。这些大数字、这些大贷款，是专门用于推理的。这是第一点。第二点，如果你想要一个进行推理的产品，你需要大量的工程师和销售人员。很多研究需要致力于生产各种产品相关功能。所以当你看到实际留给研究的部分时，差距就变得小得多。另一件事是，如果你在做不同的事情，你真的需要绝对最大的规模来证明它吗？我完全不这么认为。在我们的案例中，我们有足够的算力来说服我们自己和其他人我们正在做的事情是正确的。有公开估计称，像 OpenAI 这样的公司每年仅在实验上就花费约 60 亿美元。

Correct. And if you at SSI have 50 different ideas, how will you know which one is the next transformer and which one is brittle without having the kinds of compute that other frontier labs have? So I can comment on that. The short comment is that for us, the amount of compute that SSI has for research is really not that small, and I want to explain why. Simple math can explain why the amount of compute we have is actually a lot more comparable for research than one might think. SSI has raised $3 billion, which is not small by any absolute sense, but you could say look at the other companies raising much more. But a lot of their compute goes for inference. These big numbers, these big loans, are earmarked for inference. That's number one. Number two, if you want to have a product on which you do inference, you need to have a big staff of engineers and salespeople. A lot of the research needs to be dedicated to producing all kinds of product-related features. So when you look at what's actually left for research, the difference becomes a lot smaller. The other thing is, if you are doing something different, do you really need the absolute maximal scale to prove it? I don't think it's true at all. In our case, we have sufficient compute to convince ourselves and anyone else that what we're doing is correct. There have been public estimates that companies like OpenAI spend on the order of $6 billion a year just on experiments.

这还不包括他们在推理等方面的花费。所以看起来他们一年花在研究实验上的钱比你们的总资金还多。

This is separate from the amount of money they're spending on inference and so forth. So it seems like they're spending more a year running research experiments than you guys have in total funding.

我认为这是一个你用它做什么的问题。在他们的案例中，以及其他人的案例中，我认为对训练算力的需求更大，有更多不同的工作流，有不同的模态，只是有更多的东西，所以它变得碎片化。

I think it's a question of what you do with it. In their case, and in the case of others, I think there is a lot more demand on the training compute, there are a lot more different work streams, there are different modalities, there is just more stuff, and so it becomes fragmented.

SSI 将如何赚钱？

How will SSI make money?

我对这个问题的回答是：我们现在只专注于研究，然后答案会自己显现。我认为会有很多可能的答案。

My answer to this question is something like: we just focus on the research right now, and then the answer to that will reveal itself. I think there will be lots of possible answers.

SSI 的计划仍然是直击超级智能吗？

Is SSI's plan still to straight-shot superintelligence?

也许吧。我认为这有它的价值。

Maybe. I think that there is merit to it.

我认为有很多价值，因为不受日常市场竞争的影响非常好。但我认为有两个原因可能导致我们改变计划。

I think there's a lot of merit because I think that it's very nice to not be affected by the day-to-day market competition. But I think there are two reasons that may cause us to change the plan.

一是务实的考虑，如果时间线变长的话——这有可能。第二，我认为让最强大的人工智能存在于世界上并产生影响是很有价值的。我认为这是一件非常有意义的事情。但为什么你的默认计划是直接冲刺超级智能呢？因为听起来像 OpenAI、Anthropic 所有这些公司，他们的明确想法是：我们会有越来越弱的智能，让公众逐渐习惯并做好准备。为什么直接构建超级智能可能更好？

One is pragmatic if timelines turn out to be long, which they might. And second, I think there is a lot of value in the best and most powerful AI being out there impacting the world. I think this is a meaningfully valuable thing. But then why is your default plan to straight shot superintelligence? Because it sounds like OpenAI, Anthropic, all these other companies, their explicit thinking is: look, we have weaker and weaker intelligences that the public can get used to and prepare for. Why is it potentially better to build a superintelligence directly?

我会从正反两方面来论证。正方观点是，人们在市场中面临的一个挑战是必须参与竞争，而竞争非常困难，因为它让你面临需要做出的艰难权衡。说我们把自己隔离起来，只专注于研究，等到准备好了再出来，这样很好。但反方观点也同样成立，这是两股对立的力量。反方观点是：嘿，让世界看到强大的人工智能是有用的。让世界看到强大的人工智能是有用的，因为这是你传达它的唯一方式。

So I'll make the case for and against. The case for is that one of the challenges people face when they're in the market is that they have to participate in the rat race, and the rat race is quite difficult in that it exposes you to difficult trade-offs which you need to make. It is nice to say we'll insulate ourselves from all this and just focus on the research and come out only when we are ready, and not before. But the counterpoint is valid too, and those are opposing forces. The counterpoint is: hey, it is useful for the world to see powerful AI. It is useful for the world to see powerful AI because that's the only way you can communicate it.

嗯，我想不只是你可以传达这个想法，而是……

Well, I guess not even just that you can communicate the idea, but...

传达人工智能本身，而不是想法。传达人工智能。

Communicate the AI, not the idea. Communicate the AI.

你所说的传达人工智能是什么意思？

What do you mean communicate the AI?

好吧。假设你读了一篇关于人工智能的文章，文章说人工智能会这样那样，你读了之后说，好吧，这是一篇有趣的文章。现在，假设你看到一个人工智能在做这个，做那个。那是无法比拟的。基本上，我认为人工智能在公众面前有很大的好处，这将是我们不完全直接冲刺的一个理由。

So, okay. So, let's suppose you read an essay about AI, and the essay says AI is going to be this and AI is going to be that and it's going to be this, and you read it and you say, okay, this is an interesting essay. Now, suppose you see an AI doing this and AI doing that. It is incomparable. Basically, I think there is a big benefit from AI being in the public, and that would be a reason for us to not be quite straight shot.

是的。嗯，我想还不止这些，我确实认为这是很重要的一部分。另一件大事是，我想不出人类工程和研究中还有哪个领域，最终产品主要是通过思考如何使其安全而变得更安全，而不是像为什么今天每英里飞机失事率比几十年前低得多？为什么在 Linux 中发现一个 bug 比几十年前难得多？我认为主要是因为这些系统被部署到了世界上。你注意到失败，这些失败被纠正，系统变得更健壮。现在，我不确定为什么 AGI 和超人智能会有所不同，尤其是考虑到，我希望我们能谈到这一点，超级智能的危害不仅仅在于有一个恶意的回形针制造机，而是这是一个非常强大的东西，我们甚至不知道如何概念化人们如何与之互动，人们会用它做什么。逐步接触它似乎是分散影响并帮助人们做好准备的一种更好方式。

Yeah. Well, I guess it's not even that, which I do think is an important part of it. The other big thing is I can't think of another discipline in human engineering and research where the end artifact was made safer mostly through just thinking about how to make it safe, as opposed to why are airplane crashes per mile so much lower today than they were decades ago? Why is it so much harder to find a bug in Linux than it would have been decades ago? And I think it's mostly because these systems were deployed to the world. You noticed failures, those failures were corrected, and the systems became more robust. Now, I'm not sure why AGI and superhuman intelligence would be any different, especially given, and I hope we can talk about this, it seems like the harms of superintelligence are not just about having some malevolent paper clipper out there, but it's just that this is a really powerful thing and we don't even know how to conceptualize how people interact with it, what people will do with it. Having gradual access to it seems like a better way to spread out the impact and to help people prepare for it.

嗯，在这一点上，即使在直接冲刺的场景中，你仍然会逐步发布它，我是这么想象的。渐进主义是任何计划的内在组成部分。只是你首先拿出什么的问题。这是第一点。第二点，我也认为，我相信你比其他人更倡导持续学习，我实际上认为这是重要且正确的。原因如下。所以有一件事……我再举一个例子，说明思维、语言如何影响思维。在这种情况下，是两个词塑造了每个人的思维，我坚持认为。第一个词：AGI。第二个词：预训练。让我解释一下。所以术语 AGI，为什么这个术语存在？这是一个非常特殊的术语。为什么存在？有一个原因。术语 AGI 存在的原因，在我看来，与其说是因为它是某种智能最终状态的一个非常重要的本质描述，不如说是因为它是对一个已有术语的反应，这个术语是窄人工智能。如果你回到游戏 AI 的古老历史，跳棋 AI、国际象棋 AI、电脑游戏 AI，每个人都会说，看看这个窄智能。当然，国际象棋 AI 可以打败卡斯帕罗夫，但它不能做任何其他事情。它太窄了，人工窄智能。所以作为回应，作为对此的反应，有些人说，这不好。它太窄了。我们需要的是通用人工智能，一个可以做所有事情的人工智能。这个术语得到了很多关注。

Well, I think on this point, even in the straight shot scenario, you would still do a gradual release of it, is how I would imagine it. The gradualism would be an inherent component of any plan. It's just a question of what is the first thing that you get out of the door. That's number one. Number two, I also think, you know, I believe you have advocated for continual learning more than other people, and I actually think that this is an important and correct thing. Here is why. So one of the things... I'll give you another example of how thinking, how language affects thinking. In this case, it will be two words that have shaped everyone's thinking, I maintain. First word: AGI. Second word: pre-training. Let me explain. So the term AGI, why does this term exist? It's a very particular term. Why does it exist? There's a reason. The reason that the term AGI exists is, in my opinion, not so much because it's a very important essential descriptor of some end state of intelligence, but because it is a reaction to a different term that existed, and the term is narrow AI. If you go back to ancient history of gameplay AI, of checkers AI, chess AI, computer games AI, everyone would say, look at this narrow intelligence. Sure, the chess AI can beat Kasparov, but it can't do anything else. It is so narrow, artificial narrow intelligence. So in response, as a reaction to this, some people said, well, this is not good. It is so narrow. What we need is general AI, an AI that can just do all the things. And that term just got a lot of traction.

是的。

Yeah.

第二个获得很多关注的是预训练。具体来说，是预训练的配方。我认为现在人们做强化学习的方式可能正在消除预训练的概念印记。但预训练有一个特性：你进行更多预训练，模型在所有方面都或多或少均匀地变得更好。是的，通用人工智能预训练产生了 AGI。但 AGI 和预训练带来的问题是，在某种意义上它们超出了目标。因为如果你思考 AGI 这个术语，你会意识到，尤其是在预训练的背景下，人类并不是一个 AGI。因为人类……是的，肯定有技能基础。人类缺乏大量的知识。相反，我们依赖持续学习。我们依赖持续学习。所以当你思考，好吧，假设我们成功了，我们产生了一个安全的超级智能。问题是，你如何定义它？它在持续学习的曲线上处于什么位置？我产生了一个超级聪明的 15 岁孩子，非常渴望去，你说好吧，我要……他们根本不知道很多。优秀的学生，非常渴望。你去当程序员，你去当医生，去学习。所以你可以想象，部署本身会涉及某种学习试错期。

The second thing that got a lot of traction is pre-training. Specifically, the recipe of pre-training. I think the current way people do RL now is maybe undoing the conceptual imprint of pre-training. But pre-training had the property: you do more pre-training and the model gets better at everything more or less uniformly. Yeah, general AI pre-training gives AGI. But the thing that happened with AGI and pre-training is that in some sense they overshoot the target. Because if you think about the term AGI, you will realize, especially in the context of pre-training, that a human being is not an AGI. Because a human being... Yes, there is definitely a foundation of skills. A human being lacks a huge amount of knowledge. Instead, we rely on continual learning. We rely on continual learning. And so then when you think about, okay, so let's suppose that we achieve success and we produce a safe superintelligence. The question is, but how do you define it? Where on the curve of continual learning is it going to be? I produce a super intelligent 15-year-old that's very eager to go, and you say okay, I'm going to... They don't know very much at all. The great student, very eager. You go and be a programmer, you go and be a doctor, go and learn. So you could imagine that the deployment itself will involve some kind of a learning trial and error period.

这是一个过程，而不是你放下成品。

It's a process as opposed to you drop the finished thing.

好的，我明白了。所以你指出，超级智能并不是一个知道如何做经济中每一项工作的成品思维，因为像最初的 OpenAI 章程之类定义 AGI 的方式是它可以做人类能做的每一项工作。你提出的是一种可以学习做任何一项工作的思维。

Okay, I see. So you're suggesting that the thing you're pointing out with superintelligence is not some finished mind which knows how to do every single job in the economy, because the way the original OpenAI charter or whatever defines AGI is like it can do every single job that a human can do. You're proposing instead a mind which can learn to do any single job.

是的。

Yes.

那就是超级智能。然后一旦你有了学习算法，它就像人类劳动者加入组织一样被部署到世界上。

And that is superintelligence. And then once you have the learning algorithm, it gets deployed into the world the same way a human laborer might join an organization.

看起来可能会发生两种情况之一。也许两种情况都不会发生。第一种，这种超高效的学习算法变得超人类，变得和你一样好，甚至在机器学习研究任务上可能比你更好。结果，算法本身变得越来越超人类。第二种，即使第一种情况不发生，如果你有一个单一模型——我的意思是这明确是你的愿景——如果你有一个单一模型或模型的实例，它们被部署到经济中做不同的工作，持续学习如何做这些工作，在工作中学习，掌握任何人类能掌握的技能，但实际上同时掌握所有这些技能，然后整合所学，你基本上就有了一个模型，它在功能上变得超级智能，即使没有任何软件上的递归自我改进，对吧？因为你现在有一个模型可以做经济中的每一项工作，而人类无法以同样的方式融合我们的思想。那么，你期望从广泛部署中看到某种智能爆炸吗？

And it seems like one of these two things might happen. Maybe neither of these happens. One, this super efficient learning algorithm becomes superhuman, becomes as good as you and potentially even better at the task of ML research. And as a result, the algorithm itself becomes more and more superhuman. The other is even if that doesn't happen, if you have a single model — I mean this is explicitly your vision — if you have a single model or instances of a model which are deployed through the economy doing different jobs, learning how to do those jobs continually, learning on the job, picking up all the skills that any human could pick up, but actually picking them all up at the same time and then amalgamating the learnings, you basically have a model which functionally becomes super intelligent even without any sort of recursive self-improvement in software, right? Because you now have one model that can do every single job in the economy, and humans can't merge our minds in the same way. So do you expect some sort of intelligence explosion from broad deployment?

我认为我们很可能会看到快速的经济增长。我认为广泛部署——你可以提出两个相互矛盾的论点。一个是，如果你确实达到了一个点，你有一个可以快速学习做事的人工智能，并且有很多这样的人工智能，那么就会有强大的力量将它们部署到经济中，除非有某种法规阻止它，顺便说一句，这可能会发生。但我认为，从广泛部署来看，在一段时间内出现非常快速的经济增长是非常可能的。另一个问题是它会有多快。我认为这很难知道，因为一方面你有这个非常高效的工人，另一方面世界真的很大，有很多东西，这些东西以不同的速度移动。但另一方面，现在人工智能可以——所以我认为非常快速的经济增长是可能的，我们会看到各种各样的事情，比如不同国家有不同的规则，规则更友好的国家经济增长会更快。很难预测。

I think that it is likely that we will have rapid economic growth. I think the broad deployment — and there are two arguments you could make which are conflicting. One is that if indeed you get to a point where you have an AI that can learn to do things quickly and you have many of them, then there will be a strong force to deploy them in the economy, unless there is some kind of regulation that stops it, which by the way there might be. But I think the idea of very rapid economic growth for some time is very possible from broad deployment. The other question is how rapid it's going to be. I think this is hard to know because on the one hand you have this very efficient worker, on the other hand the world is just really big and there's a lot of stuff, and that stuff moves at a different speed. But then on the other hand, now the AI could — so I think very rapid economic growth is possible, and we will see all kinds of things, like different countries with different rules, and the ones which have the friendlier rules, the economic growth will be faster. Hard to predict.

我们的一些听众喜欢阅读文字记录而不是收听节目，所以我们投入了大量精力让文字记录读起来像独立的文章。问题是，如果你只是用语音转文字模型逐字转录对话，它会充满各种断断续续和令人困惑的措辞。我们向 Labelbox 提到了这个问题，他们问是否可以尝试一下。与他们合作可能是我最想向人们推荐 Labelbox 的原因。不仅仅是「哦，嘿，告诉我们你需要什么样的数据，我们去获取。」他们引导我们完成了整个过程，从帮助我们首先确定需要什么样的数据，到组建一个专家对齐团队来生成数据。即使在我们拿到所有数据后，Labelbox 仍然参与其中。他们帮助我们选择了合适的基础模型，并在模型输出上设置了自动质量检查，以便我们可以调整和完善。现在我们有了一个新的转录工具，可以用于我们未来的所有节目。这只是 Labelbox 如何在创意层面满足客户需求并与他们全程合作的一个例子。如果你想了解更多，或者想亲自尝试转录工具，请访问 labelbox.com/barcash。

Some people in our audience like to read the transcripts instead of listening to the episode, and so we put a ton of effort into making the transcripts read like they are standalone essays. The problem is that if you just transcribe a conversation verbatim using a speech to text model, it'll be full of all kinds of fits and starts and confusing phrasing. We mentioned this problem to Labelbox and they asked if they could take a stab. Working with them on this is probably the reason that I'm most excited to recommend Labelbox to people. It wasn't just, 'Oh, hey, tell us what kind of data you need and we'll go get it.' They walked us through the entire process from helping us identify what kind of data we needed in the first place to assembling a team of expert aligners to generate it. Even after we got all the data back, Labelbox stayed involved. They helped us choose the right base model and set up auto QA on the model's output so that we could tweak and refine it. And now we have a new transcriber tool that we can use for all our episodes moving forward. This is just one example of how Labelbox meets their customers at the ideas level and partners with them through their entire journey. If you want to learn more or if you want to try out the transcriber tool yourself, go to labelbox.com/barcash.

在我看来，这是一个非常不稳定的局面，从极限来看，我们知道这应该是可能的，因为如果你有一个在学习方面和人类一样好，但可以融合其大脑——以人类无法融合的方式融合不同实例——的东西，这似乎是在物理上应该可能的事情。人类是可能的，数字计算机是可能的。你只需要将两者结合起来就能产生这个东西。而且这种东西似乎非常强大，经济增长是一种说法。我的意思是，戴森球是大量的经济增长，但另一种说法是，你可能会有一个非常短的时间窗口，因为人类在工作——你知道，你在 SSI 雇佣人，六个月后他们可能就净产出为正了，对吧？人类学习非常快，所以这个东西变得非常快越来越聪明。你如何看待让这一切顺利进行，为什么 SSI 能够很好地做到这一点，或者 SSI 的计划基本上是什么？这就是我想问的。

It seems to me that this is a very precarious situation to be in where, looking at the limit, we know that this should be possible because if you have something that is as good as a human at learning but which can merge its brains — merge different instances in a way that humans can't merge — already. This seems like a thing that should physically be possible. Humans are possible, digital computers are possible. You just need both of those combined to produce this thing. And it also seems like this kind of thing is extremely powerful, and economic growth is one way to put it. I mean, Dyson sphere is a lot of economic growth, but another way to put it is just like you will have potentially a very short period of time because a human on the job — you know, you're hiring people at SSI, in six months they're net productive probably, right? A human learns really fast, and so this thing is becoming smarter and smarter very fast. How do you think about making that go well, and why is SSI positioned to do that well, or what is SSI's plan there basically? That's what I'm trying to ask.

是的，所以我的想法发生变化的一个方面是，我现在更重视人工智能的渐进式和提前部署。关于人工智能的一个非常困难的事情是，我们谈论的是尚不存在的系统，很难想象它们。我认为正在发生的事情之一是，在实践中很难感受到 AGI。很难感受到 AGI。我们可以谈论它，但这就像谈论——想象一下，当你年老体弱时，谈论变老是什么感觉，你可以进行对话。你可以尝试想象，但很难，然后你回到现实。嗯，事实并非如此。我认为很多关于 AGI 及其未来力量的问题都源于这样一个事实：很难想象未来的人工智能会有所不同。它会很强大。事实上，整个问题——人工智能和 AGI 的问题是什么？整个问题就是力量。整个问题就是力量。当力量真的很大时，会发生什么？在过去一年左右的时间里，我改变想法的一种方式——这种改变可能会反向传播到我们公司的计划中——是，如果很难想象，你该怎么办？你必须展示这个东西。你必须展示这个东西。而且我坚持认为，我认为大多数从事人工智能工作的人也无法想象它，因为它与人们日常所见太不同了。我确实坚持认为，这是我预测将会发生的事情。这是一个预测。我坚持认为，随着人工智能变得更加强大，人们将改变他们的行为，我们将看到各种目前没有发生的前所未有的事情。我会举一些例子。

Yeah, so one of the ways in which my thinking has been changing is that I now place more importance on AI being deployed incrementally and in advance. One very difficult thing about AI is that we are talking about systems that don't yet exist and it's hard to imagine them. I think that one of the things that's happening is that in practice it's very hard to feel the AGI. It's very hard to feel the AGI. We can talk about it, but it's like talking about — imagine having a conversation about how it is to be old when you're old and frail, and you can have a conversation. You can try to imagine it, but it's just hard and you come back to reality. Well, that's not the case. And I think that a lot of the issues around AGI and its future power stem from the fact that it's very difficult to imagine future AI is going to be different. It's going to be powerful. Indeed, the whole problem — what is the problem of AI and AGI? The whole problem is the power. The whole problem is the power. When the power is really big, what's going to happen? And one of the ways in which I've changed my mind over the past year or so — that change of mind may backpropagate into the plans of our company — is that if it's hard to imagine, what do you do? You got to be showing the thing. You got to be showing the thing. And I maintain that I think most people who work on AI also can't imagine it because it's too different from what people see on a day-to-day basis. I do maintain here is something which I predict will happen. That's a prediction. I maintain that as AI becomes more powerful, then people will change their behaviors and we will see all kinds of unprecedented things which are not happening right now. And I'll give some examples.

我认为无论好坏，前沿公司都将在未来发展中扮演非常重要的角色，政府也是如此。我们将看到的一件事——我们已经看到其开端——是激烈竞争的公司开始合作进行 AI 安全。你可能已经看到 OpenAI 和 Anthropic 迈出了第一步，但这在以前是不存在的。这实际上是我大约三年前在一次演讲中预测到的，这样的事情会发生。我还认为，随着 AI 变得越来越强大、越来越明显地强大，政府和公众也会产生采取行动的愿望，我认为这是一股非常重要的力量。

I think for better or worse, the frontier companies will play a very important role in what happens, as will the government. One thing we'll see, which we're already seeing the beginnings of, is companies that are fierce competitors starting to collaborate on AI safety. You may have seen OpenAI and Anthropic doing a first small step, but that did not exist before. That's actually something I predicted in one of my talks about three years ago, that such a thing will happen. I also maintain that as AI continues to become more powerful, more visibly powerful, there will also be a desire from governments and the public to do something, and I think that this is a very important force.

第二，需要做什么？我认为会发生的一件事是，目前从事 AI 工作的人因为 AI 的错误而不觉得它强大。但我确实认为，在某个时刻，AI 会开始让人感觉强大，而当这发生时，我们将看到所有 AI 公司对待安全的方式发生巨大变化。他们会变得偏执得多。我这么说是一种预测。我们看看我是否是对的，但我认为这会发生，因为他们会看到 AI 变得更强大。我认为目前发生的一切，都是因为人们看着今天的 AI，很难想象未来的 AI。

Number two, what needs to be done? One thing I maintain will happen is that right now people working on AI don't feel it's powerful because of its mistakes. I do think that at some point the AI will start to feel powerful, and when that happens, we will see a big change in the way all AI companies approach safety. They'll become much more paranoid. I say this as a prediction. We'll see if I'm right, but I think this will happen because they will see the AI becoming more powerful. Everything happening right now, I maintain, is because people look at today's AI and it's hard to imagine the future AI.

还有第三件事需要发生。我是在更广泛的层面上谈论，不仅仅是从 SSI 的角度，因为你问到了我们的公司。问题是，公司应该追求构建什么？有一个大想法一直困扰着每个人，那就是自我改进的 AI。为什么会这样？因为想法比公司少。但我认为有更好的东西可以构建，而且我认为每个人最终都会想要它。那就是一个稳健对齐、特别关心有感知生命的 AI。我认为有理由相信，构建一个关心有感知生命的 AI 比构建一个只关心人类生命的 AI 更容易，因为 AI 本身将是有感知的。如果你想想镜像神经元和人类对动物的同理心——你可能会说它不够大，但它存在——我认为这是一个涌现属性，源于我们用相同的电路来建模他人和自己，因为这是最有效的方式。

There is a third thing which needs to happen. I'm talking about it in broader terms, not just from the perspective of SSI because you asked about our company. The question is, what should the companies aspire to build? There has been one big idea that everyone has been locked into, which is the self-improving AI. Why did that happen? Because there are fewer ideas than companies. But I maintain that there is something better to build, and I think everyone will actually want that. It's an AI that's robustly aligned to care about sentient life specifically. I think there is a case to be made that it will be easier to build an AI that cares about sentient life than an AI that cares about human life alone, because the AI itself will be sentient. If you think about things like mirror neurons and human empathy for animals, which you might argue is not big enough but it exists, I think it's an emerging property from the fact that we model others with the same circuit we use to model ourselves, because that's the most efficient thing to do.

所以即使你让 AI 关心有感知的生命，我也不清楚如果你解决了对齐问题，这是否是你应该尝试做的。仍然会有大多数有感知的生命是 AI。将会有数万亿、最终数千万亿的 AI。人类将只是有感知生命中的一小部分。所以我不清楚，如果目标是某种人类对未来文明的控制，这是否是最好的标准。

So even if you got an AI to care about sentient beings, it's not actually clear to me that that's what you should try to do if you solved alignment. It would still be the case that most sentient beings will be AIs. There will be trillions, eventually quadrillions of AIs. Humans will be a very small fraction of sentient beings. So it's not clear to me if the goal is some kind of human control over this future civilization that this is the best criterion.

确实如此。我认为这可能不是最好的标准。我要说两点。第一，我认为关心有感知生命有其价值，应该被考虑。如果有一个简短的思路清单，让处于这种情况的公司可以使用，那会很有帮助。第二，我认为如果最强大的超级智能的某种能力被限制，那将会有实质性的帮助，因为它会解决很多这些担忧。如何做到这一点，我不确定，但我认为在谈论真正强大的系统时，这会有实质性的帮助。

It's true. I think it's possible it's not the best criterion. I'll say two things. First, I think care for sentient life has merit and should be considered. It would be helpful if there was some kind of short list of ideas that companies in this situation could use. Second, I think it would be materially helpful if the power of the most powerful superintelligence was somehow capped, because it would address a lot of these concerns. The question of how to do it, I'm not sure, but I think that would be materially helpful when you're talking about really powerful systems.

在我们继续对齐讨论之前，我想深入探讨一下。顶部还有多少空间？你怎么看待超级智能？你认为，利用这种学习效率的想法，它可能只是学习新技能或知识极快，并且拥有更大的策略池吗？中心是否有一个单一的、有凝聚力的实体，更强大或更大？如果是这样，你想象这将与人类文明的其他部分相比像神一样，还是只是感觉像另一个智能体或另一组智能体？

Before we continue the alignment discussion, I want to double click on that. How much room is there at the top? How do you think about superintelligence? Do you think, using this learning efficiency idea, maybe it's just extremely fast at learning new skills or knowledge, and does it just have a bigger pool of strategies? Is there a single cohesive entity in the center that's more powerful or bigger? And if so, do you imagine that this will be sort of godlike in comparison to the rest of human civilization, or does it just feel like another agent or another cluster of agents?

这是一个不同人有不同直觉的领域。我认为它肯定会非常强大。我认为最可能发生的是，大约同时会有多个这样的 AI 被创建。我认为如果集群足够大，比如真正的大陆大小，那个东西确实可以非常强大。如果你真的有一个大陆大小的集群，那些 AI 可以非常强大。我可以告诉你，如果你谈论极其强大的 AI，真正戏剧性地强大，那么是的，如果它们能在某些方面受到限制，或者有某种协议，那会很好，因为如果你想象一个足够强大的系统，你可以说，好吧，你需要做一些明智的事情，比如非常专一地关心有感知生命，我们可能不喜欢结果。这就是问题所在。所以也许答案是你不要构建通常意义上的单个 RL 智能体。我会指出几件事。我认为人类是半智能体。我们追求奖励，然后情绪或其他东西让我们厌倦奖励，我们追求不同的奖励。市场是一种非常短视的智能体。进化也是如此。进化在某些方面非常聪明，但在其他方面非常愚蠢。政府被设计成三个部分之间无休止的斗争，这会产生效果。

This is an area where different people have different intuitions. I think it will be very powerful for sure. I think what is most likely to happen is that there will be multiple such AIs being created roughly at the same time. I think if the cluster is big enough, like literally continent-sized, that thing could be really powerful indeed. If you literally have a continent-sized cluster, those AIs can be very powerful. I can tell you that if you're talking about extremely powerful AIs, like truly dramatically powerful, then yeah, it would be nice if they could be restrained in some ways, or if there was some kind of agreement, because if you imagine a system that is sufficiently powerful, and you could say, okay, you need to do something sensible like care for sentient life in a very single-minded way, we might not like the results. That's really what it is. So maybe the answer is that you do not build a single RL agent in the usual sense. I'll point out several things. I think human beings are a semi-agent. We pursue a reward, and then emotions or whatever make us tire of the reward, and we pursue a different reward. The market is like a very shortsighted kind of agent. Evolution is the same. Evolution is very intelligent in some ways but very dumb in others. The government has been designed to be a never-ending fight between three parts, which has an effect.

所以我认为，另一个让这个讨论变得困难的原因是，我们在谈论尚不存在、我们还不知道如何构建的系统。这实际上是我的信念。我认为人们现在正在做的事情会取得一定进展，然后逐渐停滞。它会继续改进，但也不会是那个终极形态。所以我们还不知道如何构建那个终极形态。我认为很多事情都取决于对可靠泛化的理解。我还要说另一件事：可能导致对齐困难的一个原因是，你学习人类价值观的能力是脆弱的，然后你优化它们的能力也是脆弱的。你真的能学会优化它们吗？难道不能说这些都是不可靠泛化的实例吗？为什么人类似乎能泛化得好得多？如果泛化好得多会怎样？在这种情况下会发生什么？会有什么影响？但这些问题目前仍然无法回答。

So I think another thing that makes this discussion difficult is that we are talking about systems that don't exist, that we don't know how to build right now. That's actually my belief. I think what people are doing right now will go some distance and then peter out. It will continue to improve, but it will also not be it. So the 'it' we don't know how to build. I think a lot hinges on understanding and reliable generalization. I'll say another thing: one of the things that could cause alignment to be difficult is that your ability to learn human values is fragile, then your ability to optimize them is fragile. Will you actually learn to optimize them? And then can't you say these are all instances of unreliable generalization? Why is it that human beings appear to generalize so much better? What if generalization was much better? What would happen in this case? What would be the effect? But those questions are right now still unanswerable.

如何思考 AI 发展顺利会是什么样子？因为我认为你已经勾勒出 AI 可能如何演变。我们将拥有这种持续学习的智能体。AI 将非常强大。也许会有许多不同的 AI。你如何看待许多大陆大小的智能体四处活动？那有多危险？我们如何降低这种危险？我们如何以一种保护均衡的方式来做到这一点，而那里可能存在不对齐的 AI 和恶意行为者？

How does one think about what AI going well looks like? Because I think you've scoped out how AI might evolve. We'll have these sort of continual learning agents. AI will be very powerful. Maybe there will be many different AIs. How do you think about lots of continent-sized intelligences going around? How dangerous is that? How do we make that less dangerous? And how do we do that in a way that protects an equilibrium where there might be misaligned AIs out there and bad actors out there?

所以我喜欢关心有情生命的 AI 的一个原因是，我们可以争论这是好是坏，但如果这些戏剧性系统的前 N 个确实关心、热爱人类或类似的东西，关心有情生命。显然，这也需要实现。所以如果前 N 个系统实现了这一点，那么我可以看到事情至少在相当长一段时间内进展顺利。然后就是长期会发生什么的问题。你如何实现长期均衡？我认为也有一个答案，我不喜欢这个答案，但它需要被考虑。从长远来看，你可能会说，好吧，如果你有一个强大 AI 存在的世界。短期内，你可以说，好吧，你有全民高收入。我们都过得很好。但正如佛教徒所说，变化是唯一不变的。所以事情会变化，存在某种政府政治结构，它会变化，因为这些事物有保质期。一些新的政府事物出现并运作，然后过一段时间它停止运作。这是你一直看到的事情。所以我认为对于长期均衡，一种方法你可以说，好吧，也许每个人都会有一个 AI 来执行他们的命令，这很好，如果这能无限维持下去，那是真的。但缺点是，AI 会去为这个人赚钱，在政治领域倡导他们的需求，也许还会写一份小报告说，好吧，这是我做的，这是情况，这个人说很好继续，但这个人不再是一个参与者。然后你可以说这是一个不稳定的状态。但我要先说我不喜欢这个解决方案，但它是一个解决方案。这个解决方案是，如果人们通过某种神经链接成为部分 AI，因为结果会是，现在 AI 理解了一些东西，我们也理解了，因为现在理解被整体传输了。所以如果 AI 处于某种情况，现在就像你完全参与了那个情况。我认为这是均衡的答案。

So one reason why I liked the AI that cares for sentient life, and we can debate on whether it's good or bad, but if the first N of these dramatic systems actually do care for, you know, love humanity or something, care for sentient life. Obviously, this also needs to be achieved. So if this is achieved by the first N of those systems, then I can see it go well at least for quite some time. And then there is the question of what happens in the long run. How do you achieve a long run equilibrium? I think that there is an answer as well, and I don't like this answer, but it needs to be considered. In the long run, you might say, okay, so if you have a world where powerful AI exist. In the short term, you could say, okay, you have universal high income. We all doing well. But we know that what do the Buddhists say? Change is the only constant. And so things change and there is some kind of government political structure thing and it changes because these things have a shelf life. Some new government thing comes up and it functions and then after some time it stops functioning. That's something that you see happening all the time. So I think that for the long run equilibrium, one approach you could say okay so maybe every person will have an AI that will do their bidding and that's good and if that could be maintained indefinitely that's true. But the downside with that is okay so then the AI goes and earns money for the person and advocates for their needs in the political sphere and maybe then writes a little report saying okay here's what I've done here's the situation and the person says great keep it up but the person is no longer a participant. And then you can say that's a precarious place to be in. But so I'm going to preface by saying I don't like this solution but it is a solution. And the solution is if people become part AI with some kind of neural link, because what will happen as a result is that now the AI understands something and we understand it too because now the understanding is transmitted wholesale. So now if the AI is in some situation, now it's like you are involved in the situation yourself fully. And I think this is the answer to the equilibrium.

我想知道，在数百万年甚至数十亿年前完全不同的环境中发展出的情感，仍然如此强烈地指导我们的行动，这是否是对齐成功的一个例子。也许详细说明我的意思：脑干有这些，我不知道称它为价值函数还是奖励函数更准确，但脑干有一个指令，说「与更成功的人交配」。皮层是理解在现代背景下成功意味着什么的部分，但脑干能够对齐皮层，说「无论你如何认识成功，我不够聪明去理解那是什么，你仍然会追求这个指令。」

I wonder if the fact that emotions which were developed millions or in many cases billions of years ago in a totally different environment are still guiding our actions so strongly is an example of alignment success. To maybe spell out what I mean: the brain stem has these, I don't know if it's more accurate to call it a value function or reward function, but the brain stem has a directive where it's saying 'mate with somebody who's more successful.' The cortex is the part that understands what does success mean in the modern context, but the brain stem is able to align the cortex and say 'however you recognize success to be, and I'm not smart enough to understand what that is, you're still going to pursue this directive.'

我认为有一个更普遍的观点。我认为大脑如何编码高级欲望实际上非常神秘。抱歉，是进化如何编码高级欲望。比如，很容易理解进化如何赋予我们对闻起来好的食物的欲望，因为气味是一种化学物质，所以只需追求那种化学物质。很容易想象进化做这样的事情。但进化也赋予了我们所有这些社会欲望，比如我们真的很在意被社会正面看待。我们在意拥有良好的地位。我们喜欢所有这些社会直觉。我强烈感觉到它们是内置的，我不知道进化是如何做到的，因为这是一个高级概念。它在大脑中被表示。比如人们认为，假设你关心一些社会事物。它不像气味那样的低级信号。它不是有传感器的东西。大脑需要做大量处理，将许多信息碎片拼凑起来，以理解社会发生了什么，而进化却说这是你应该关心的。

I think there is a more general point. I think it's actually really mysterious how the brain encodes high level desires. Sorry, how evolution encodes high level desires. Like it's pretty easy to understand how evolution would endow us with the desire for food that smells good, because smell is a chemical and so just pursue that chemical. It's very easy to imagine evolution doing such a thing. But evolution also has endowed us with all these social desires, like we really care about being seen positively by society. We care about being in a good standing. We like all these social intuitions that we have. I feel strongly that they are baked in and I don't know how evolution did it because it's a high level concept. It's represented in the brain. Like what people think, let's say you care about some social thing. It's not like a low-level signal like smell. It's not something for which there's a sensor. The brain needs to do a lot of processing to piece together lots of bits of information to understand what's going on socially and somehow evolution said that's what you should care about.

是的。

Yes.

它是如何做到的？而且它做得很快。

How did it do it? And it did it quickly too.

是的，因为我认为所有这些我们关心的复杂社会事物，我认为它们进化得相当晚。所以进化很容易硬编码这种高级欲望。

Yeah, because I think all these sophisticated social things that we care about, I think they evolved pretty recently. So evolution had an easy time hardcoding this high level desire.

我坚持，或者至少我会说我不知道有什么好的假设来解释它是如何做到的。我有一些想法在琢磨，但没有一个令人满意。

I maintain, or at least I'll say I'm unaware of good hypothesis for how it's done. I had some ideas I was kicking around but none of them are satisfying.

是的。特别令人印象深刻的是，如果这是一种你在有生之年学到的欲望，那是有道理的，因为你的大脑是智能的。我们能够学习智能欲望是有道理的。但你的观点是，这种欲望也许是，这不是你的观点，但一种理解方式是欲望是内置在基因组中的，而基因组并不智能，对吧？但它能够以某种方式描述这个特征，这个特征甚至不清楚如何定义，然后你把它构建到基因中。

Yeah. And what's especially impressive is if it was a desire that you learned in your lifetime, it kind of makes sense because your brain is intelligent. It makes sense why we be able to learn intelligent desires. But your point is that the desire is maybe, this is not your point, but one way to understand it is the desire is built into the genome and the genome is not intelligent, right? But it's able to somehow describe this feature that requires, like it's not even clear how you define that feature, and you can get it into the genes.

是的，本质上，或者我换个说法。如果你想想基因组可用的工具，它说，好的，这是构建大脑的配方。你可以说，这是将多巴胺神经元连接到嗅觉传感器的配方。

Yeah, essentially, or maybe I'll put it differently. If you think about the tools that are available to the genome, it says, okay, here's a recipe for building a brain. And you could say, here is a recipe for connecting the dopamine neurons to like the smell sensor.

嗯。

Yeah.

如果气味是某种好闻的气味，你就想吃它。我可以想象基因组做到这一点。我声称这更难想象。更难想象基因组说你应该关心你的整个大脑，或者说大脑的很大一部分所做的某种复杂计算。这就是我所说的全部。我可以告诉你一个推测。我在想这是如何做到的。让我提供一个推测，我会解释为什么这个推测可能是错误的。所以推测是这样的。大脑有那些区域。你知道大脑区域。我们有大脑皮层，对吧？

And if the smell is a certain kind of good smell, you want to eat that. I could imagine the genome doing that. I'm claiming that it is harder to imagine. It's harder to imagine the genome saying you should care about some complicated computation that your entire brain, like a big chunk of your brain does. That's all I'm claiming. I can tell you a speculation. I was wondering how it could be done. And let me offer a speculation and I'll explain why the speculation is probably false. So the speculation is okay. So the brain has those regions. You know the brain regions. We have our cortex, right?

嗯。

Yeah.

它有所有这些大脑区域，大脑皮层是均匀的。但大脑区域和皮层中的神经元大多只与它们的邻居交流。这就解释了为什么会有大脑区域，因为如果你想做某种语言处理，所有处理语言的神经元需要相互交流，它们可以，而且因为神经元大多只能与附近的邻居交流，所以必须是一个区域。所有区域在不同人之间大多位于相同的位置。所以也许进化硬编码了大脑上的一个位置。它说：「哦，就像大脑的 GPS，GPS 坐标，如此这般，当那个区域激活时，那就是你应该关心的。」也许进化就是这样做的，因为那会在进化的工具箱之内。

It has all those brain regions and the cortex is uniform. But the brain regions and the neurons in the cortex, they kind of speak to their neighbors mostly. And that explains why you get brain regions because if you want to do some kind of speech processing, all the neurons that do speech need to talk to each other and they can, and because neurons can only speak to their nearby neighbors for the most part, it has to be a region. All the regions are mostly located in the same place from person to person. So maybe evolution hardcoded literally a location on the brain. So it says, "Oh, like when the GPS of the brain, GPS coordinates, such and such, when that fires, that's what you should care about." Like maybe that's what evolution did because that would be within the toolkit of evolution.

是的。尽管有例子，比如天生失明的人，他们大脑皮层的那个区域被另一种感官接管了，我不知道，但如果需要视觉信号的欲望或奖励功能不再起作用，我会感到惊讶。你知道那些大脑皮层不同区域被征用的人。例如，如果你不再有视觉，你还能感觉到我想要周围人喜欢我等等的感觉吗？这些通常也有视觉线索。

Yeah. Although there are examples where for example people who are born blind have that area of their cortex adopted by another sense and I have no idea but I'd be surprised if the desires or the reward functions which require visual signal no longer worked. You know people who have their different areas of their cortex co-opted. For example, if you no longer have vision, can you still feel the sense that I want people around me to like me and so forth, which usually there's also visual cues for.

所以，我实际上完全同意这一点。我认为对这个理论有一个更有力的反驳，那就是如果你想想人们，有些人在童年时期被切除了半个大脑。

So, I actually fully agree with that. I think there's an even stronger counter argument to this theory, which is like if you think about people, so there are people who get half of their brain removed in childhood.

嗯。

Yeah.

他们仍然拥有所有的大脑区域，但所有这些区域都以某种方式移动到了仅一个半球，这表明大脑区域的位置不是固定的。所以那个理论是不正确的。如果它是真的那就太酷了，但事实并非如此。所以我认为这是一个谜，但这是一个有趣的谜。事实是，进化 somehow 能够非常可靠地赋予我们关心社会事物的能力。甚至那些有各种奇怪的精神状况、缺陷和情感问题的人也往往关心这一点。

And they still have all their brain regions, but they all somehow move to just one hemisphere, which suggests that the brain regions, the location is not fixed. And so that theory is not true. It would have been cool if it was true, but it's not. And so I think that's a mystery, but it's an interesting mystery. Like the fact is somehow evolution was able to endow us to care about social stuff very very reliably. And even people who have all kinds of strange mental conditions and deficiencies and emotional problems tend to care about this.

此外，像深度伪造、语音克隆和智能体这样的 AI 工具极大地增加了欺诈和滥用的复杂性。因此，比以往任何时候都更重要的是真正了解使用你平台的任何人或任何事物的身份和意图。这正是 Sardine 帮助你做到的。Sardine 汇集了数千种设备行为和身份信号，帮助你评估风险。从用户打字、移动鼠标或握持设备的方式，到他们是否通过 VPN 隐藏真实位置，再到他们是否在 KYC 自拍检查中注入虚假摄像头画面。Sardine 将这些信号与来自其近 40 亿设备网络的洞察相结合，比如用户的欺诈历史或他们与其他高风险账户的关联，这样你就能在坏人造成损害之前发现他们。如果你只使用自己应用程序的数据，这几乎是不可能的。Sardine 不止于检测。他们提供一套智能体来简化入职检查并自动化调查。因此，当欺诈者使用 AI 来扩大攻击规模时，你可以使用 AI 来扩大防御规模。访问 sardine.ai/warcash 了解更多信息并下载他们的 AI 欺诈检测指南。

Also, AI tools like deepfakes, voice clones, and agents have dramatically increased the sophistication of fraud and abuse. So, it's more important than ever to actually understand the identity and intent of whoever or whatever is using your platform. That's exactly what Sardine helps you do. Sardine brings together thousands of device behavior and identity signals to help you assess risk. Everything from how a user types or moves their mouse or holds their device to whether they're hiding their true location behind a VPN to whether they're injecting a fake camera feed during KYC selfie checks. Sardine combines these signals with insights from their network of almost 4 billion devices, things like a user's history of fraud or their associations with other high-risk accounts so you can spot bad actors before they do damage. This would literally be impossible if you only use data from your own application. Sardine doesn't stop at detection. They offer a suite of agents to streamline onboarding checks and automate investigations. So, as fraudsters use AI to scale their attacks, you can use AI to scale your defenses. Go to sardine.ai/warcash to learn more and download their guide on AI fraud detection.

SSI 计划做哪些不同的事情？所以大概你的计划是成为这个时代到来时的前沿公司之一，然后你创办 SSI 大概是因为你觉得你有办法以其他公司没有的方式安全地做到这一点。这个区别是什么？

What is SSI planning on doing differently? So presumably your plan is to be one of the frontier companies when this time arrives and then what is presumably you started SSI because you're like I think I have a way of approaching how to do this safely in a way that the other companies don't. What is that difference?

所以我描述的方式是，有一些我认为有前景的想法，我想研究它们，看看它们是否真的有前景。就这么简单。这是一次尝试。我认为如果这些想法被证明是正确的，我们讨论过的关于理解泛化的这些想法，如果这些想法被证明是正确的，那么我认为我们将拥有一些有价值的东西。它们会被证明是正确的吗？我们正在做研究。我们完全是一家研究时代的公司。我们正在取得进展。实际上，在过去一年里我们取得了相当不错的进展。但我们需要继续取得更多进展，更多研究。这就是我的看法。我将其视为一次尝试，成为一个声音和参与者。

So the way I would describe it as there are some ideas that I think are promising and I want to investigate them and see if they are indeed promising or not. It's really that simple. It's an attempt. I think that if the ideas turn out to be correct, these ideas that we discussed around understanding generalization, if these ideas turn out to be correct, then I think we will have something worthy. Will it turn out to be correct? We are doing research. We are squarely an age of research company. We are making progress. We've actually made quite good progress over the past year. But we need to keep making more progress, more research. And that's how I see it. I see it as an attempt to be a voice and a participant.

嗯，人们问你的联合创始人兼前 CEO 最近离开去了 Meta，人们问，如果有很多突破正在取得，那似乎是一件不太可能发生的事。我想知道你怎么回应。

Um people have asked your co-founder and previous CEO left to go to Meta recently and people have asked well if there was a lot of breakthroughs being made that seems like a thing that should have been unlikely. I wonder how you respond.

是的。对此，我将简单地提醒一些可能被遗忘的事实，我认为这些事实提供了背景，我认为它们解释了情况。背景是，我们正在以 320 亿美元的估值融资，然后 Meta 进来提出收购我们，我说不，但我的前联合创始人在某种意义上说了是，结果他也能够享受到大量的近期流动性，并且他是 SSI 中唯一加入 Meta 的人。

Yeah. So for this I will simply remind a few facts that may have been forgotten and I think these facts which provide the context I think they explain the situation. So the context was that we were fundraising at a 32 billion valuation and then Meta came in and offered to acquire us and I said no but my former co-founder in some sense said yes and as a result he also was able to enjoy from a lot of near-term liquidity and he was the only person from SSI to join Meta.

听起来 SSI 的计划是成为一家前沿公司，当你到达人类历史上这个非常重要的时期，拥有超人智能，并且你有关于如何让超人智能顺利发展的想法，但其他公司会尝试他们自己的想法。SSI 让超级智能顺利发展的方法有什么不同？

It sounds like SSI's plan is to be a company that is at the frontier when you get to this very important period in human history where you have superhuman intelligence and you have these ideas about how to make superhuman intelligence go well but other companies will be trying their own ideas. What distinguishes SSI's approach to making super intelligence go well?

SSI 的主要区别在于其技术方法。所以我们有一种不同的技术方法，我认为它是有价值的，我们正在追求它。我坚持认为最终会有策略的趋同。所以我认为会有策略的趋同，在某个时刻，随着 AI 变得更强大，每个人都会或多或少地清楚策略应该是什么。

The main thing that distinguishes SSI is its technical approach. So we have a different technical approach that I think is worthy and we are pursuing it. I maintain that in the end there will be a convergence of strategies. So I think there will be a convergence of strategies where at some point as AI becomes more powerful it's going to become more or less clearer to everyone what the strategy should be.

而且这应该像是，你需要找到某种方式彼此对话。你希望你的第一个真正的超级智能 AI 是对齐的，并且以某种方式关心有感知的生命、关心人类、民主，或者这些的组合。我认为这是每个人都应该努力达到的条件，这也是 SSI 正在努力的方向。我认为随着时间的推移，如果不是已经的话，所有其他公司都会意识到他们在朝着同一个目标努力。我认为随着 AI 变得更强大，世界将真正改变。

And it should be something like, you need to find some way to talk to each other. And you want your first actual real super intelligent AI to be aligned and somehow care for sentient life, care for people, democratic, or some combination thereof. I think this is the condition that everyone should strive for and that's what SSI is striving for. I think that with time, if not already, all the other companies will realize they're striving towards the same thing. I think the world will truly change as AI becomes more powerful.

是的。

Yeah.

而且我认为很多这些预测都会是，事情会变得非常不同，人们的行为也会非常不同。

And I think a lot of these forecasts will be like, things will be really different and people will be acting really differently.

说到预测，你对你描述的这个系统有什么预测？这个系统能像人类一样学习，然后变得超人类。

Speaking of forecasts, what are your forecasts for this system you're describing, which can learn as well as a human and subsequently become superhuman?

我认为大概 5 到 20 年。

I think like 5 to 20 years.

5 到 20 年。

5 to 20 years.

嗯。

Mhm.

所以我想展开你如何看待世界的发展。就像我们还有几年时间，这些其他公司继续当前的方法，然后它停滞了。这里的停滞是指他们的收入不超过几千亿，还是你怎么理解停滞的含义？

So I just want to unroll how you might see the world coming. It's like we have a couple more years where these other companies are continuing the current approach and it stalls out. And stalls out here meaning they earn no more than low hundreds of billions in revenue, or how do you think about what stalling out means?

是的，我认为它可能会停滞，而且我认为停滞在所有不同公司中看起来会非常相似。我不确定，因为我认为即使停滞，这些公司也能产生巨额收入，也许不是利润，因为他们需要努力区分彼此，但收入肯定有。

Yeah, I think it could stall out, and I think stalling out will look very similar among all the different companies. I'm not sure, because I think even with stalling out, these companies could make stupendous revenue, maybe not profits, because they will need to work hard to differentiate themselves, but revenue definitely.

但你的模型中有某种东西暗示，当正确的解决方案出现时，所有公司之间会趋同。我很好奇你为什么这么认为。

But there's something in your model that implies that when the correct solution does emerge, there will be convergence between all the companies. I'm curious why you think that's the case.

嗯，我更多是在谈论他们更大战略上的趋同。我认为最终在技术方法上的趋同也可能发生，但我指的是更大战略上的趋同。到底应该做什么？

Well, I was talking more about convergence on their larger strategies. I think eventual convergence on the technical approach is probably going to happen as well, but I was alluding to convergence on the larger strategies. What exactly is the thing that should be done?

我只是想更好地理解你如何看待未来的发展。所以目前我们有这些不同的公司，你预计他们的方法会继续产生收入。是的。

I just want to better understand how you see the future unfolding. So currently we have these different companies and you expect their approach to continue generating revenue. Yes.

但不会达到这种像人类一样的学习者。

But not get to this humanlike learner.

是的。

Yes.

所以现在我们有了这些不同的公司分支。有你，有 Thinking Machines，还有很多其他实验室。

So now we have these different forks of companies. We have you, we have Thinking Machines, there's a bunch of other labs.

是的，也许其中一个找到了正确的方法。

Yes, and maybe one of them figures out the correct approach.

但随后他们产品的发布让其他人清楚如何做这件事。

But then the release of their product makes it clear to other people how to do this thing.

我认为不会清楚如何去做，但会清楚某种不同的东西是可能的。

I think it won't be clear how to do it, but it will be clear that something different is possible.

对。

Right.

而那就是信息，我认为人们随后会试图弄清楚那是如何运作的。不过我确实认为，这里没有讨论的一点是，随着 AI 能力的每一次提升，做事的方式会发生某种变化。所以我认为这很重要，但我无法确切说明那是什么。

And that is information, and I think people will then be trying to figure out how that works. I do think though that one of the things not addressed here is that with each increase in AI's capabilities, there will be some kind of changes in how things are being done. So I think it's going to be important, yet I can't spell out exactly what that is.

而且默认情况下，你会预期拥有那个模型的公司获得所有这些收益，因为他们拥有那个正在学习如何做所有事情的模型，拥有它在世界上积累的所有技能和知识。有什么理由认为这些好处会被广泛分配，而不是最终只落到那个首先启动这个持续学习循环的模型公司？

And how, by default, you would expect the company that has that model to be getting all these gains because they have the model that is learning how to do everything, has all the skills and knowledge it's building up in the world. What is the reason to think that the benefits would be widely distributed and not just end up at whatever model company gets this continuous learning loop going first?

我认为根据经验，以下是我认为会发生的事情。第一，让我们看看过去 AI 的发展情况。一家公司取得了进步，另一家公司在一段时间后匆忙推出了一些类似的东西，然后他们开始在市场上竞争并压低价格。所以我认为从市场角度来看，类似的事情也会发生。即使有人……顺便说一句，我们讨论的是好的世界。什么是好的世界？就是我们有这些强大的像人类一样的学习者，而且……顺便说一句，关于超级智能 AI 的规格可能还有另一件值得考虑的事情：你可以让它既狭窄又有用。所以你可以有很多狭窄的超级智能 AI。但假设你有很多这样的 AI，并且有一家公司从中获得了大量利润，然后另一家公司进来开始竞争。竞争的方式是通过专业化。我认为会发生的是，竞争喜欢专业化，你在市场上看到这一点，在进化中也看到这一点。所以你会有很多不同的利基市场，很多不同的公司占据不同的利基市场。在这种世界里，你可能会说一家 AI 公司在某个非常复杂的经济活动领域相当擅长，另一家公司则在另一个领域更好，第三家公司非常擅长诉讼。

I think that empirically, here is what I think is going to happen. Number one, let's look at how things have gone so far with the AIs of the past. So one company produced an advance, and the other company scrambled and produced some similar things after some amount of time, and they started to compete in the market and push the prices down. So I think from the market perspective, something similar will happen there as well. Even if someone... we're talking about the good world by the way. What's the good world? Where we have these powerful humanlike learners that are also... and by the way, maybe there's another thing on the spec of the super intelligent AI that is worth considering: you can make it narrow and useful at the same time. So you can have lots of narrow super intelligent AIs. But suppose you have many of them, and you have some company that's producing a lot of profits from it, and then another company comes in and starts to compete. The way competition is going to work is through specialization. I think what's going to happen is that competition loves specialization, and you see it in the market, you see it in evolution as well. So you're going to have lots of different niches and lots of different companies occupying different niches. In this kind of world, you might say one AI company is really quite a bit better at some area of really complicated economic activity, and a different company is better at another area, and a third company is really good at litigation.

但这与像人类一样的学习所暗示的相矛盾，即它可以学习。

But that's contradicted by what humanlike learning implies, which is that it can learn.

它可以，但你有积累的学习，你有大量的投资。你花了大量算力变得非常非常擅长这件事，而另一个人花了大量算力和大量经验变得非常擅长另一件事。

It can, but you have accumulated learning, you have a big investment. You spent a lot of compute to become really, really good at this thing, and someone else spent a huge amount of compute and a huge amount of experience to get really good at some other thing.

对。

Right.

你应用了大量人类学习才达到那个高度，但现在你处于这个高点，别人会说，「我不想从头学习你已经学到的东西。」

You apply a lot of human learning to get there, but now you are at this high point where someone else would say, 'I don't want to start learning what you've learned to go.'

我想那将需要许多不同的公司同时开始使用像人类一样的持续学习智能体，这样他们就可以在不同的分支开始不同的研究。但如果一家公司先得到那个智能体或先得到那个学习者，那么看起来他们确实可以——如果你只考虑经济中的每一项工作，只需有一个实例学习每一项，似乎对一家公司来说是可行的。

I guess that would require many different companies to begin at the humanlike continual learning agent at the same time so that they can start their different research in different branches. But if one company gets that agent first or gets that learner first, it does then seem like they could, if you just think about every single job in the economy, just have an instance learning each one, seems tractable for a company.

是的，这是一个有效的论点。

Yeah, that's a valid argument.

我强烈的直觉是，事情不会那样发展。我强烈的直觉是，正如论证所说，它会这样发展。但我强烈的直觉是，它不会这样发展。这是那种理论上理论和实践没有区别，但实践中却有区别的情况之一。我认为这将是其中之一。

My strong intuition is that it's not how it's going to go. My strong intuition is that, as the argument says, it will go this way. But my strong intuition is that it will not go this way. This is one of those cases where in theory there is no difference between theory and practice, but in practice there is. I think that's going to be one of those.

很多人的递归自我改进模型明确表示，我们会在服务器里有一百万个伊利亚，带着不同的想法，这将导致超级智能非常快速地出现。你对你所做的事情的可并行化程度有什么直觉吗？复制伊利亚能带来什么收益？

A lot of people's models of recursive self-improvement explicitly state we will have a million Ilyas in a server that are coming in with different ideas and this will lead to a superintelligence emerging very fast. Do you have some intuition about how parallelizable the thing you are doing is? What are the gains from making copies of Ilya?

我不知道。我认为肯定会有收益递减，因为你想要的是想法不同的人，而不是相同的人。我认为如果他们是我的字面复制品，我不确定你能获得多少增量价值。但想法不同的人，那才是你想要的。

I don't know. I think there'll definitely be diminishing returns because you want people who think differently rather than the same. I think that if they were literal copies of me, I'm not sure how much incremental value you'd get. But people who think differently, that's what you want.

为什么如果你看看不同的模型，即使是完全不同的公司发布的，在可能不重叠的数据集上训练的，LLM 之间的相似程度实际上非常惊人？

Why is it that if you look at different models, even released by totally different companies trained on potentially non-overlapping datasets, it's actually crazy how similar LLMs are to each other?

也许数据集并不像看起来那么不重叠。但有一种感觉是，即使单个人类可能不如未来的 AI 高效，但人类团队比 AI 团队可能拥有更多多样性，这一点可能很重要。但我们如何引发 AI 之间有意义的多样性呢？我认为仅仅提高温度只会导致胡言乱语。我认为你想要更像不同科学家有不同偏见或不同想法的东西。你如何在 AI 智能体之间获得那种多样性？我认为没有多样性的原因是预训练。所有预训练模型几乎都一样，因为它们在相同的数据上预训练。现在，强化学习和后训练开始出现一些分化，因为不同的人提出不同的强化学习训练。

Maybe the datasets are not as non-overlapping as it seems. But there's some sense that even if an individual human might be less productive than the future AI, maybe there's something to the fact that human teams have more diversity than teams of AIs might have. But how do we elicit meaningful diversity among AI? I think just raising the temperature just results in gibberish. I think you want something more like different scientists have different prejudices or different ideas. How do you get that kind of diversity among AI agents? The reason there has been no diversity, I believe, is because of pre-training. All the pre-trained models are the same pretty much because they pre-train on the same data. Now RL and post-training is where some differentiation starts to emerge because different people come up with different RL training.

是的。我过去听你暗示过，自我对弈是一种获取数据或将智能体与同等智能的其他智能体匹配以启动学习的方式。我们应该如何看待为什么没有公开的提案表明这类事情在 LLM 上有效？

Yeah. And then I've heard you hint in the past about self-play as a way to either get data or match agents to other agents of equivalent intelligence to kick off learning. How should we think about why there's no public proposals of this kind of thing working with LLM?

我想说两点。我认为自我对弈之所以有趣，是因为它提供了一种仅使用算力而不使用数据来创建模型的方法，对吧？如果你认为数据是最终的瓶颈，那么仅使用算力就非常有趣。这就是它有趣的地方。问题是，自我对弈，至少是过去那种让智能体相互竞争的方式，只适合发展某一套技能；它太狭窄了。它只适合谈判、冲突、某些社交技能、策略制定之类的东西。所以如果你关心这些技能，那么自我对弈会有用。实际上，我认为自我对弈确实找到了归宿，但只是以不同的形式。比如辩论、证明验证器。你有一个 LLM 作为裁判，它也有动力去发现你工作中的错误。你可以说这不完全是自我对弈，但这是人们正在做的相关对抗性设置。

I would say there are two things to say. I would say that the reason why I thought self-play was interesting is because it offered a way to create models using compute only without data, right? And if you think that data is the ultimate bottleneck, then using compute only is very interesting. So that's what makes it interesting. Now the thing is that self-play, at least the way it was done in the past when you have agents which somehow compete with each other, it's only good for developing a certain set of skills; it is too narrow. It's only good for negotiation, conflict, certain social skills, strategizing, that kind of stuff. And so if you care about those skills, then self-play will be useful. Now actually I think that self-play did find a home but just in a different form. So things like debate, prove a verifier. You have some kind of an LLM as a judge which is also incentivized to find mistakes in your work. You could say this is not exactly self-play but it's a related adversarial setup that people are doing.

实际上，自我对弈是更一般的智能体间竞争的一个特例。

And really self-play is an example of a special case of more general competition between agents.

对。对竞争的自然反应是试图与众不同。所以如果你放多个智能体，告诉它们，你们都需要解决某个问题，你是一个智能体，你检查其他人在做什么，你会说，如果他们已经在用这个方法，我不确定我是否应该继续。他们应该追求差异化的东西。所以我认为类似这样的东西也可以激励方法的多样性。

Right. The natural response to competition is to try to be different. And so if you were to put multiple agents and you tell them, you all need to work on some problem and you're an agent and you're inspecting what everyone else is working, you're going to say, well, if they already taken this approach, it's not clear I should pursue it. They should pursue something differentiated. And so I think that something like this could also create an incentive for a diversity of approaches.

是的。最后一个问题，什么是研究品味？你显然是世界上被认为在 AI 研究方面品味最好的人。你是深度学习历史上许多重大事件的合著者，从 AlexNet 到 GPT-3 等等。它是什么？你如何描述你是如何想出这些想法的？

Yeah. Final question, what is research taste? You're obviously the person in the world who is considered to have the best taste in doing research in AI. You were the co-author on many of the biggest things that have happened in the history of deep learning from AlexNet to GPT-3 to so on. What is it? How do you characterize how you come up with these ideas?

我可以回答。我可以就我自己发表评论。我认为不同的人做法不同。但指导我个人的一件事是关于 AI 应该是什么样的一种美学，通过思考人是什么样的，但要正确地思考。很容易错误地思考人是什么样的，但正确地思考人意味着什么？我来举几个例子。人工神经元的概念直接受大脑启发，这是一个伟大的想法。为什么？因为你说，当然，大脑有所有这些不同的器官，有缺陷，但缺陷可能无关紧要。为什么我们认为神经元重要？因为有很多神经元。这感觉是对的。所以你想要神经元。你想要某种局部学习规则来改变神经元之间的连接。大脑这样做感觉是合理的。分布式表示的概念，大脑对经验做出反应，或者神经网络应该从经验中学习，而不是反应。大脑从经验中学习。你问自己，什么是根本性的，什么不是根本性的，事情应该怎样。我认为这很大程度上指导了我，从多个角度思考，寻找几乎是美、简单、丑陋——没有丑陋的空间——只有美、简单、优雅、来自大脑的正确灵感。所有这些都需要同时存在。它们存在得越多，你就越能对自上而下的信念有信心。然后自上而下的信念是在实验与你的想法矛盾时支撑你的东西。因为如果你总是只相信数据，那么有时你可能在做正确的事情，但有一个 bug。但你不知道有 bug。你怎么知道有 bug？你怎么知道你应该继续调试还是得出结论这是错误的方向？嗯，这就是自上而下的。你可以说事情必须是这样。像这样的东西必须有效。因此，我们必须继续前进。这就是自上而下的。

I can answer. I can comment on this for myself. I think different people do it differently. But one thing that guides me personally is an aesthetic of how AI should be, by thinking about how people are but thinking correctly. It's very easy to think about how people are incorrectly, but what does it mean to think about people correctly? So I'll give you some examples. The idea of the artificial neuron is directly inspired by the brain and it's a great idea. Why? Because you say, sure, the brain has all these different organs, has faults, but the faults probably don't matter. Why do we think that the neurons matter? Because there are many of them. It kind of feels right. So you want the neuron. You want some kind of local learning rule that will change the connections between the neurons. It feels plausible that the brain does it. The idea of the distributed representation, the idea that the brain responds to experience, or a neural network should learn from experience, not response. The brain learns from experience. And you kind of ask yourself, is something fundamental or not fundamental, how things should be. And I think that's been guiding me a fair bit, thinking from multiple angles and looking for almost beauty, simplicity, ugliness—there's no room for ugliness—it's just beauty, simplicity, elegance, correct inspiration from the brain. And all of those things need to be present at the same time. And the more they are present, the more confident you can be in a top-down belief. And then the top-down belief is the thing that sustains you when the experiments contradict you. Because if you just trust the data all the time, well, sometimes you can be doing a correct thing, but there's a bug. But you don't know that there is a bug. How can you tell that there is a bug? How do you know if you should keep debugging or you conclude it's the wrong direction? Well, it's the top-down. You can say the things have to be this way. Something like this has to work. Therefore, we got to keep going. That's the top-down.

而这正是基于大脑的多面之美和灵感。

And it's based on this multifaceted beauty and inspiration by the brain.

好的，我们就到这里。非常感谢。

All right, we'll leave it there. Thank you so much.

好的，谢谢。

All right. Appreciate it.

太棒了。

That was great.

是的，我很享受。

Yeah, I enjoyed it.

我也是。

Yes, me too.

大家好，希望你们喜欢这一集。如果喜欢，最有帮助的事情就是把它分享给其他你觉得可能感兴趣的人。如果你在收听的平台上留下评分或评论，也很有帮助。如果你有兴趣赞助本播客，可以联系 dwarcash.com/advertise。否则，我们下期再见。

Hey everybody, I hope you enjoyed that episode. If you did, the most helpful thing you can do is just share it with other people who you think might enjoy it. It's also helpful if you leave a rating or a comment on whatever platform you're listening on. If you're interested in sponsoring the podcast, you can reach out at dwarcash.com/advertise. Otherwise, I'll see you on the next one.