强化学习与大语言模型：智能的核心之争

为什么你要区分人类？人类也是动物。我们的共同点更有趣。我们应该少关注那些区别。

Why are you trying to distinguish humans? Humans are animals. What we have in common is more interesting. What distinguishes us, we should be paying less attention to.

我的意思是，我们试图复制智能，对吧？没有动物能登上月球或制造半导体。所以我们想理解是什么让人类特别。

I mean, we're trying to replicate intelligence, right? No animal can go to the moon or make semiconductors. So, we want to understand what makes humans special.

所以，我喜欢你觉得那很明显，因为我觉得相反很明显。如果我们理解了松鼠，我们就几乎全懂了。我个人很满足于长期与我的领域不同步，也许几十年，因为过去我偶尔是对的。我认为学习不是关于训练，而是关于一个主动的过程。孩子尝试事情，看看会发生什么。我认为我们应该为引发宇宙中这一伟大转变而自豪。今天我和理查德·萨顿聊天，他是强化学习的奠基人之一，发明了许多主要技术，如 TD 学习和策略梯度方法，因此他获得了今年的图灵奖，如果你不知道，这基本上是计算机科学的诺贝尔奖。理查德，恭喜。

So, I like the way you consider that obvious, because I consider the opposite obvious. If we understood a squirrel, we'd be almost all the way there. I am personally just kind of content being out of sync with my field for a long period of time perhaps decades because occasionally I have improved right in the past. I don't think learning is really about training. It's about an active process. The child tries things and sees what happens. I think we should be proud that we are giving rise to this great transition in the universe. Today I'm chatting with Richard Sutton who is one of the founding fathers of reinforcement learning, an inventor of many of the main techniques used there like TD learning and policy gradient methods, and for that he received this year's Turing Award, which if you don't know is basically the Nobel Prize for computer science. Richard, congratulations.

谢谢你，达里什。

Thank you, Darish.

感谢你来做客播客。

And thanks for coming on the podcast.

这是我的荣幸。

It's my pleasure.

好的，第一个问题：我和我的听众在概念上熟悉 LLM 思考 AI 的方式。从强化学习的角度思考 AI，我们错过了什么？

Okay, so first question: my audience and I are familiar with the LLM way of thinking about AI conceptually. What are we missing in terms of thinking about AI from the RL perspective?

嗯，是的，我认为这确实是一个相当不同的观点，很容易分离并失去相互交流的能力。

Well, yes, I think it's really quite a different point of view and it can easily get separated and lose the ability to talk to each other.

嗯。

Mhm.

而且大型语言模型已经变得如此重要。生成式 AI 总体上很重要。我们的领域容易受潮流和时尚影响。所以我们失去了对基本事物的关注，因为我认为强化学习是基本的 AI。智能是什么，问题在于理解你的世界。而强化学习是关于理解你的世界，而大型语言模型是关于模仿人们做别人说你应该做的事。它们不是关于弄清楚该做什么。

And large language models have become such a big thing. Generative AI in general a big thing. And our field is subject to bandwagons and fashions. So we lose track of the basic things because I consider reinforcement learning to be basic AI. And what intelligence is, the problem is to understand your world. And reinforcement learning is about understanding your world, whereas large language models are about mimicking people doing what people say you should do. They're not about figuring out what to do.

嗯。我想你会认为，要模仿互联网文本语料库中数万亿的 token，你必须构建一个世界模型。事实上，这些模型似乎确实有非常强大的世界模型，它们是我们在 AI 中迄今为止最好的世界模型。对吧。那么，你认为这缺少了什么？

Huh. I guess you would think that to emulate the trillions of tokens in the corpus of internet text, you would have to build a world model. In fact, these models do seem to have very robust world models and they're the best world models we've made to date in AI. Right. So, what do you think that that's missing?

我不同意你刚才说的大部分内容。

I would disagree with most of the things you just said.

很好。

Great.

仅仅模仿人们说的话根本不是构建世界模型。我不认为你是在模仿拥有世界模型的东西，即人。

Just to mimic what people say is not really to build a model of the world at all. I don't think you know you're mimicking things that have a model of the world, the people.

但我不想以对抗的方式处理问题，但我会质疑它们有世界模型的观点。世界模型会让你能够预测会发生什么。

But I don't want to approach the question in an adversarial way, but I would question the idea that they have a world model. So a world model would enable you to predict what would happen.

对。

Right.

它们有能力预测一个人会说什么。它们没有能力预测会发生什么。我们想要的，我想引用艾伦·图灵的话，我们想要一台能从经验中学习的机器。

They have the ability to predict what a person would say. They don't have the ability to predict what will happen. What we want, I think to quote Alan Turing, what we want is a machine that can learn from experience.

对吧？

Right?

经验是你生活中实际发生的事情。你做事，你看到结果。那就是你学习的东西。

Where experience is the things that actually happen in your life. You do things, you see what happens. And that's what you learn from.

是的。

Yeah.

大型语言模型从别的东西学习。它们从“这是一个情况，这是一个人做的”学习。隐含的建议是你应该做那个人做的事。

The large language models learn from something else. They learn from here's a situation and here's what a person did. And implicitly the suggestion is you should do what the person did.

对吧？我想也许关键，我很好奇你是否不同意这一点，是有些人会说，好吧，这种模仿学习给了我们一个好的先验，或者给了这些模型一个好的先验，但合理的方法来解决问题。随着我们进入你所说的经验时代，这个先验将成为我们基于经验教授这些模型的基础，因为这给了它们有时得到正确答案的机会，然后在此基础上，你可以构建，你可以用经验训练它们。你同意这个观点吗？

Right? I guess maybe the crux, and I'm curious if you disagree with this, is some people will say okay so this imitation learning has given us a good prior or given these models a good prior, but reasonable ways to approach problems. And as we move towards the era of experience, as you call it, this prior is going to be the basis on which we teach these models from experience, because this gives them the opportunity to get answers right some of the time, and then on this you can build, you can train them on experience. Do you agree with that perspective?

不，我同意这是大型语言模型的观点，对吧？

No, I agree that it's the large language model perspective, right?

我不认为这是一个好的观点。

I don't think it's a good perspective.

是的。是的。当然。

Yeah. Yeah. Sure.

所以，要成为某物的先验，必须有一个真实的东西。我的意思是，先验知识应该是实际知识的基础。什么是实际知识？在那个大型语言框架中没有实际知识的定义。是什么让一个行动成为好的行动？你认识到价值，持续学习的需要，对吧？所以如果你需要持续学习，持续意味着在与世界的正常互动中学习。

So to be a prior for something, there has to be a real thing. I mean, a prior bit of knowledge should be the basis for actual knowledge. What is actual knowledge? There's no definition of actual knowledge in that large language framework. What makes an action a good action to take? You recognize the value, the need for continual learning, right? So if you need to learn continually, continually means learning during the normal interaction with the world.

是的。

Yeah.

所以，在正常互动中必须有某种方式来判断什么是对的。

And so then there must be some way during the normal interaction to tell what's right.

是的。

Yep.

好的。那么在最大的语言模型设置中，有没有办法判断什么是正确的说法？你会说一些话，但不会得到关于什么是正确说法的反馈，因为没有定义什么是正确的说法。没有目标。

Okay. So is there any way for it to tell in the largest language model setup to tell what's the right thing to say? You will say something and you will not get feedback about what the right thing to say is, because there's no definition of what the right thing to say is. There's no goal.

对吧？

Right?

如果没有目标，那么有一种说法，另一种说法。没有正确的说法。

And if there's no goal, then there's one thing to say, another thing to say. There's no right thing to say.

对吧？

Right?

所以没有基本事实。如果没有基本事实，你就不可能有先验知识，因为先验知识应该是对真相的提示或初始信念。

So there's no ground truth. You can't have prior knowledge if you don't have ground truth because the prior knowledge is supposed to be a hint or an initial belief about what the truth is.

是的。

Yeah.

但没有任何真相。现在没有正确的说法。在强化学习中，有正确的说法或正确的做法，因为正确的做法是能让你获得奖励的事情。

But there isn't any truth. There's no right thing to say right now. In reinforcement learning, there is a right thing to say or right thing to do because the right thing to do is the thing that gets you reward.

对。

Right.

所以我们有关于什么是正确做法的定义，因此我们可以有先验知识或人们提供的关于什么是正确做法的知识，然后我们可以检查它，因为我们有关于什么是实际正确做法的定义。

So we have a definition of what the right thing to do is, and so we can have prior knowledge or knowledge provided by people about what the right thing to do is, and then we can check it to see because we have a definition of what the actual right thing to do is.

现在一个更简单的例子是当你试图构建一个世界模型时，当你预测会发生什么，你预测然后你看到发生了什么。

Now an even simpler case is when you have you're trying to make a model of the world, when you predict what will happen, you predict and then you see what happens.

对吧？所以有基本事实。大型语言模型中没有基本事实，因为你没有关于接下来会发生什么的预测。如果你在对话中说些什么，大型语言模型没有关于那个人会如何回应或回应会是什么的预测。我的意思是，我认为它们确实有，你可以直接问它们你预期用户会如何回应，它们有一个预测。

Okay? So there's ground truth. There's no ground truth in large language models because you don't have a prediction about what will happen next. If you say something in your conversation, the large language models have no prediction about what the person will say in response to that or what the response will be. I mean I think they do like you can literally ask them what you would anticipate a user might say in response and they have a prediction.

哦不，他们会回答那个问题，对吧？

Oh no, they will respond to that question, right?

是的，但他们在实质意义上没有预测。他们不会对发生的事情感到惊讶。如果发生了不是你所说的他们预测的事情，他们不会因为意外事件而改变。要学习，他们必须做出调整。

Yeah, but they have no prediction in the substantive sense. They won't be surprised by what happens. If something happens that isn't what you might say they predicted, they will not change because an unexpected thing has happened. To learn, they'd have to make an adjustment.

我认为这种能力在上下文中确实存在。观察模型进行思维链很有趣，假设它试图解决一个数学问题。它会说：‘好的，我首先要用这种方法来处理这个问题’，然后写出来，接着像这样：‘哦等等，我刚刚意识到这是处理这个问题的错误概念方式。我要用另一种方法重新开始。’这种灵活性在上下文中确实存在，对吧？你还有其他想法，还是只是认为需要将这种能力扩展到更长的范围？

I think a capability like this does exist in context. It's interesting to watch a model do chain of thought and then suppose it's trying to solve a math problem. It'll say, 'Okay, I'm going to approach this problem using this approach at first,' and it'll write this out and be like, 'Oh wait, I just realized this is the wrong conceptual way to approach the problem. I'm going to restart with another approach.' That flexibility does exist in context, right? Do you have something else in mind, or do you just think that you need to extend this capability across longer horizons?

我只是说，在任何有意义的意义上，他们都没有对接下来会发生什么的预测，他们不会对接下来发生的事情感到惊讶。如果发生了什么事，他们不会做出任何改变。

I'm just saying they don't have, in any meaningful sense, a prediction of what will happen next, and they will not be surprised by what happened next. They'll not make any changes if something happens.

但这不就是下一个词预测的本质吗？预测接下来是什么，然后根据意外进行更新。

But isn't that literally what next-token prediction is? Prediction about what's next and then updating on a surprise.

下一个词是它们应该说什么，应该采取什么行动。这不是世界对它们行为的回应。让我们回到它们缺乏目标的问题上。

Next token is what they should say, what the action should be. It's not what the world will give them in response to what they do. Let's go back to their lack of goal.

嗯。

Mhm.

对我来说，拥有目标是智能的本质。如果某物能够实现目标，它就是智能的。我喜欢约翰·麦卡锡的定义：智能是实现目标能力的计算部分。所以你必须要有目标。否则，你只是一个行为系统。你不特别。你不智能。

For me, having a goal is the essence of intelligence. Something is intelligent if it can achieve goals. I like John McCarthy's definition that intelligence is the computational part of the ability to achieve goals. So you have to have goals. Otherwise, you're just a behaving system. You're not special. You're not intelligent.

对。你同意大型语言模型没有目标吗？

Right. And you agree that large language models don't have goals?

我认为它们有一个目标。

I think they have a goal.

什么目标？

What's the goal?

下一个词预测。

Next-token prediction.

那不是目标。它不会改变世界。

That's not a goal. It doesn't change the world.

我认为词元向你涌来，如果你预测它们，你并不会影响它们。

I think tokens come at you, and if you predict them, you don't influence them.

哦，是的。这不是关于外部世界的目标。

Oh, yeah. It's not a goal about the external world.

是的。这不是一个实质性的目标。你不能看着一个系统说‘哦，它有一个目标’，如果它只是坐在那里预测，并为自己准确预测而沾沾自喜。

Yeah. It's not a substantive goal. You can't look at a system and say, 'Oh, it has a goal' if it's just sitting there predicting and being happy with itself that it's predicting accurately.

我想也许我想理解的更大问题是，为什么你认为在 LLM 之上进行强化学习不是一个富有成效的方向。因为我们似乎能够给这些模型一个解决困难数学问题的目标，而且它们在许多方面处于人类水平的顶峰，能够解决数学奥林匹克类型的问题。它们在 IMO 获得了金牌。所以，在国际数学奥林匹克中获得金牌的模型似乎确实有正确解决数学问题的目标。那么为什么我们不能将其扩展到不同领域呢？

I guess maybe the bigger question I want to understand is why you don't think doing reinforcement learning on top of LLM is a productive direction. Because we seem to be able to give these models a goal of solving difficult math problems, and they are in many ways at the very peaks of human level in the capacity to solve math Olympiad type problems. They got gold at IMO. So it seems like the model which got gold at the International Math Olympiad does have the goal of getting math problems right. So why can't we extend this to different domains?

嗯，数学问题不同。建立物理世界的模型并执行数学假设或操作的后果是非常不同的事情。经验世界必须被学习。你必须学习后果。而数学更像是计算。它更像是标准规划。所以你可以给它们一个找到证明的目标，并且它们在某种程度上被赋予了那个目标来找到证明。

Well, math problems are different. Making a model of the physical world and carrying out the consequences of mathematical assumptions or operations are very different things. The empirical world has to be learned. You have to learn the consequences. Whereas math is more just computational. It's more like standard planning. So there you can give them a goal to find the proof, and they are in some way given that goal to find the proof.

对。所以这很有趣，因为你在 2019 年写了这篇题为《苦涩的教训》的文章，这也许是 AI 历史上最有影响力的文章。但人们将其作为扩大 LLM 规模的理由，因为他们认为，这是我们发现的将大量算力投入到学习世界中的唯一可扩展方式。所以有趣的是，你的观点是 LLM 实际上并不是苦涩的教训。

Right. So it's interesting because you wrote this essay in 2019 titled 'The Bitter Lesson', and this is the most influential essay perhaps in the history of AI. But people have used that as a justification for scaling up LLMs because in their view, this is the one scalable way we have found to pour ungodly amounts of compute into learning about the world. So it's interesting that your perspective is that the LLMs are actually not bitter lesson.

大型语言模型是否是苦涩的教训的一个案例，这是一个有趣的问题。因为它们显然是一种使用大量计算的方式，会随着计算扩展到互联网的极限。但它们也是一种注入大量人类知识的方式。所以这是一个有趣的问题。这是一个社会学或行业问题：它们是否会达到数据的极限，并被那些仅从经验而非人类那里获取更多数据的系统所取代？在某些方面，这是苦涩的教训的经典案例：我们向大型语言模型中注入的人类知识越多，它们做得越好，所以感觉很好。然而，我特别期待会出现能够从经验中学习的系统，它们很可能表现得更好，并且更具可扩展性。在这种情况下，这将是苦涩的教训的另一个实例：使用人类知识的东西最终被那些仅从经验和计算中训练的东西所取代。

It's an interesting question whether large language models are a case of the bitter lesson. Because they are clearly a way of using massive computation that will scale with computation up to the limits of the internet. But they're also a way of putting in lots of human knowledge. So this is an interesting question. It's a sociological or industry question: will they reach the limits of the data and be superseded by things that can get more data just from experience rather than from people? In some ways it's a classic case of the bitter lesson: the more human knowledge we put into the large language models, the better they can do, and so it feels good. And yet, I in particular expect there to be systems that can learn from experience and which could well perform much better and be much more scalable. In which case it will be another instance of the bitter lesson: the things that used human knowledge were eventually superseded by things that just trained from experience and computation.

我想这对我来说似乎不是关键，因为我认为那些人也会同意，未来绝大多数的算力将来自从经验中学习。他们只是认为，为了投入算力进行未来的经验学习或在职学习，你将开始的脚手架或基础将是 LLM。所以我仍然不明白为什么这完全是错误的起点。为什么我们需要一个全新的架构来开始进行经验性持续学习，为什么我们不能从 LLM 开始这样做。

I guess that doesn't seem like the crux to me because I think those people would also agree that the overwhelming amount of compute in the future will come from learning from experience. They just think that the scaffold or the basis of that, the thing you'll start with in order to pour in the compute to do this future experiential learning or on-the-job learning, will be LLMs. So I still don't understand why this is the wrong starting point altogether. Why we need a whole new architecture to begin doing experiential continual learning, and why we can't start with LLMs to do that.

嗯，在苦涩的教训的每个案例中，你都可以从人类知识开始，然后只做可扩展的事情。情况总是如此。而且从来没有理由说这一定是坏事。但事实上，在实践中它总是被证明是坏的，因为人们陷入了人类知识的方法，心理上，或者你知道，现在我在推测原因，但这总是发生。他们被真正可扩展的方法打败了。

Well, in every case of the bitter lesson, you could start with human knowledge, and then just do the scalable things. That's always the case. And there's never any reason why that has to be bad. But in fact, in practice it has always turned out to be bad because people get locked into the human knowledge approach, and psychologically, or you know, now I'm speculating why it is, but this is what has always happened. They get their lunch eaten by the methods that are truly scalable.

是的。给我讲讲可扩展的方法是什么。

Yeah. Give me a sense of what the scalable method is.

可扩展的方法是你从经验中学习。你尝试事情，看看什么有效。没有人需要告诉你。首先，你有一个目标。

The scalable method is you learn from experience. You try things, you see what works. No one has to tell you. First of all, you have a goal.

所以没有目标，就没有对错或好坏的概念。大型语言模型试图在没有目标或好坏概念的情况下运作，这从一开始就错了。也许将其与人类比较很有趣。在从模仿学习与从经验学习以及目标问题上，我认为有一些有趣的类比。孩子最初会通过模仿来学习。

So without a goal, there's no sense of right or wrong or better or worse. Large language models are trying to get by without having a goal or a sense of better or worse. That's starting in the wrong place. Maybe it's interesting to compare this to humans. In both the case of learning from imitation versus experience and on the question of goals, I think there's some interesting analogies. Kids will initially learn from imitation.

不，当然不是。

No, of course not.

真的吗？

Really?

是的。我认为孩子只是观察别人，他们试着说出同样的话。

Yeah. I think kids just watch people. They try to say the same.

这些孩子多大？

How old are these kids?

我觉得水平……

I think the level...

那前六个月呢？

What about the first six months?

我认为他们在模仿。他们试图让自己的嘴巴发出像看到母亲嘴巴发出的声音。然后他们会说出同样的词，却不理解意思。随着年龄增长，模仿的复杂性增加。你模仿你群体中的人用来猎鹿的技能。然后你进入从经验中学习的强化学习阶段。但我认为人类有很多模仿学习。

I think they're imitating things. They try to make their mouth sound the way they see their mother's mouth sound. Then they'll say the same words without understanding what they mean. As you get older, the complexity of the imitation increases. You're imitating skills that people in your band use to hunt deer. Then you go into the learning from experience RL regime. But I think there's a lot of imitation learning happening with humans.

是的。令人惊讶。你竟然有如此不同的观点。

Yeah. Surprising. You can have such a different point of view.

当我看到孩子时，我看到他们尝试各种事情，挥舞手臂，转动眼睛。没有人告诉他们。他们如何转动眼睛或发出声音并没有模仿。他们可能想发出同样的声音，但婴儿实际做的动作没有目标，没有范例。

When I see kids, I see them trying things, waving their hands, moving their eyes. No one tells them. There's no imitation for how they move their eyes or the sounds they make. They may want to create the same sounds, but the actions the infant actually does have no targets, no examples.

我同意这并不能解释婴儿所做的一切，但我认为它指导了学习过程。即使是大型语言模型，在训练早期试图预测下一个词时，也会做出与实际看到不同的猜测。在某种意义上，这就像非常短视界的强化学习，它做出一个猜测。类似于孩子试图说一个词却说错了一样。

I agree that it doesn't explain everything infants do, but I think it guides a learning process. Even an LLM, when trying to predict the next token early in training, will make a guess different from what it actually sees. In some sense, it's like very short horizon RL where it makes a guess. Similar to how a kid will try to say a word and it comes out wrong.

大型语言模型从训练数据中学习，而不是从经验中学习。它们从正常生活中永远不会得到的东西中学习。在正常生活中，永远不会有训练数据告诉你应该做这个动作。

Large language models learn from training data, not from experience. They learn from something that will never be available during their normal life. There's never any training data that says you should do this action in normal life.

我认为这更多是语义上的区别。你怎么称呼学校？那不是训练数据吗？你不是因为……才去上学。

I think this is more of a semantic distinction. What do you call school? Is that not training data? You're not going to school because...

学校要晚得多。好吧，我不该说“从不”，但我不确定。我想我甚至会对学校这么说，但正规教育是例外。你应该把你的学习建立在……我认为你只是在你的生物学中编程，早期你没什么用，然后你存在的目的是理解世界并学习如何与之互动。这似乎是一个训练阶段。我同意从训练到部署没有明显的分界，但似乎有一个初始训练阶段。没有任何东西告诉你应该做什么。你看到事情发生。没有人告诉你该做什么。别抬杠。这很明显。

School is much later. Okay, I shouldn't have said never, but I don't know. I think I would even say it about school, but formal schooling is the exception. You should base your learning on... where I think you're just programming in your biology that early on you're not that useful and then you exist to understand the world and learn how to interact with it. It seems like a training phase. I agree there's not a sharp cutoff from training to deployment, but there seems to be an initial training phase. There's nothing where you have training of what you should do. You see things that happen. You're not told what to do. Don't be difficult. This is obvious.

你确实被教导该做什么。“训练”这个词就来源于人类。

You're literally taught what to do. That's where the word training comes from, from humans.

我不认为学习真的是关于训练。我认为学习是一个主动的过程。孩子尝试事情，看看会发生什么。

I don't think learning is really about training. I think learning is about an active process. The child tries things and sees what happens.

对。

Right.

当我们想到婴儿成长时，我们不会想到训练。这些事情其实很好理解。如果你看看心理学家如何看待学习，根本没有模仿这回事。也许有些极端情况下人类会这样做，但没有基本的动物学习过程叫模仿。基本的动物学习过程是用于预测和试错控制的。有趣的是，最难看到的东西往往是显而易见的。很明显，监督学习不是动物学习的一部分。我们没有期望行为的例子。我们有事情发生的例子，有先后顺序的例子，有我们做了某事并产生后果的例子。但没有监督学习的例子。监督学习在自然界中不存在。学校，即使如此，也是人类特有的东西。它在自然界中并不普遍。松鼠不上学。松鼠可以了解世界的一切。很明显，监督学习在动物中不存在。我采访了心理学家和人类学家约瑟夫·亨里奇，他研究文化进化、人类区别于其他动物的特点以及人类如何获取知识。

We don't think about training when we think of an infant growing up. These things are rather well understood. If you look at how psychologists think about learning, there's nothing like imitation. Maybe there are extreme cases where humans might do that, but there's no basic animal learning process called imitation. Basic animal learning processes are for prediction and trial-and-error control. It's interesting how the hardest things to see are the obvious ones. It's obvious that supervised learning is not part of how animals learn. We don't have examples of desired behavior. We have examples of things that happened, things that followed one another, and we have examples of we did something and there were consequences. But there are no examples of supervised learning. Supervised learning doesn't happen in nature. School, even if that were the case, is a special thing that happens in people. It doesn't happen broadly in nature. Squirrels don't go to school. Squirrels can learn all about the world. It's absolutely obvious that supervised learning doesn't happen in animals. I interviewed psychologist and anthropologist Joseph Henrich, who has done work on cultural evolution and what distinguishes humans and how humans pick up knowledge.

你为什么试图区分人类？人类是动物。我们的共同点更有趣。我们与众不同的地方，我们应该少关注。

Why are you trying to distinguish humans? Humans are animals. What we have in common is more interesting. What distinguishes us, we should pay less attention to.

我们试图复制智能。如果你想理解是什么让人类能够登月或制造半导体，我认为我们想理解的是为什么没有动物能登月或制造半导体。所以我们想理解是什么让人类特殊。

We're trying to replicate intelligence. If you want to understand what enables humans to go to the moon or build semiconductors, I think the thing we want to understand is what makes no animal can go to the moon or make semiconductors. So we want to understand what makes humans special.

我喜欢你认为那很明显的方式，因为我认为相反很明显。我们需要理解我们是如何作为动物的。如果我们理解了松鼠，我认为我们几乎就能理解人类智能了。语言部分只是表面的一层薄薄的 veneer。

I like the way you consider that obvious, because I consider the opposite obvious. We need to understand how we are animals. If we understood a squirrel, I think we'd be almost all the way there to understanding human intelligence. The language part is just a small veneer on the surface.

这很好。我们发现了我们思考方式的巨大差异。

This is great. We're finding out the very different ways we're thinking.

我们不是在争论。我们试图分享彼此不同的思维方式。

We're not arguing. We're trying to share our different ways of thinking with each other.

是的。我认为争论是有用的。但我确实想完成这个想法。

Yeah. And I think argument is useful. But I do want to complete this thought.

约瑟夫·亨里奇有一个有趣的理论：如果你看看人类为了成功必须掌握的许多技能——不是过去一千年或一万年，而是几十万年——世界非常复杂，你无法通过推理学会如何在北极捕猎海豹。制作诱饵、寻找海豹、加工食物以避免中毒，这是一个多步骤的过程，无法通过推理全部完成。随着时间的推移，文化整体上学会了如何找到、杀死并食用海豹。当这些知识代代相传时，在他看来，你只能通过模仿长辈来学习这项技能，因为你无法通过思考学会如何捕猎、杀死和加工海豹。你必须观察别人，也许做些调整。这就是文化知识积累的方式。但文化收获的第一步必须是模仿。也许你有不同的看法。

So Joseph Henrich has this interesting theory that if you look at a lot of the skills that humans have had to master in order to be successful, and we're not talking about the last thousand years or the last 10,000 years, but hundreds of thousands of years. The world is really complicated and it's not possible to reason through how to, let's say, hunt a seal if you're living in the Arctic. And so there's this many-step process of how to make the bait, how to find the seal, and then how to process the food in a way that ensures you won't get poisoned. And it's not possible to reason through all of that. Over time, there's this larger process, whatever analogy you want to use, where culture as a whole has figured out how to find, kill, and eat seals. But what happens when this knowledge is transmitted through generations is, in his view, that you just have to imitate your elders to learn that skill because you can't think your way through how to hunt, kill, and process a seal. You have to watch other people and maybe make tweaks and adjustments. That's how cultural knowledge accumulates. But the initial step of the cultural gain has to be imitation. Maybe you think about it a different way.

不，我看法相同。

No, I think about it the same way.

好的。但这仍然是基于基本试错学习和预测学习之上的一个小东西，也许正是它让我们与许多动物区别开来。

Okay. But still it's a small thing on top of basic trial and error learning, prediction learning, and it's what distinguishes us perhaps from many animals.

但我们首先是动物。

But we're an animal first.

是的。

Yeah.

在拥有语言和其他一切之前，我们就是动物。我确实认为你提出了一个非常有趣的观点：持续学习是大多数哺乳动物都有的能力，我想所有哺乳动物都有。所以很有意思的是，我们拥有所有哺乳动物都有的能力，但我们的 AI 系统却没有，对吧？而理解数学和解决复杂数学问题的能力——取决于你如何定义数学——这是我们的 AI 拥有的能力，但几乎没有动物拥有。所以什么难什么容易，这很有趣。

And we were an animal before we had language and all those other things. I do think you make a very interesting point that continual learning is a capability that most mammals have. I guess all mammals have. So it's quite interesting that we have something that all mammals have but our AI systems don't have, right? Whereas maybe the ability to understand math and solve difficult math problems depends on how you define math. But this is a capability our AIs have but that almost no animal has. So it's quite interesting what ends up being difficult and what ends up being easy.

莫拉维克悖论。

Moravec's paradox.

没错。

That's right.

为了让经验时代开始，我们需要在复杂的现实世界环境中训练 AI。但构建有效的强化学习环境很难。你不能只雇一个软件工程师，让他们写一堆千篇一律的验证测试。现实世界领域是混乱的。你需要深度领域专家来正确获取数据、工作流程和所有微妙的规则。当 Labelbox 的一个客户想要训练一个智能体在线购物时，Labelbox 组建了一个在构建互联网店面方面经验丰富的团队。例如，团队构建了一个可以在剧集中更新的产品目录，因为大多数购物网站的状态是不断变化的。他们还添加了 Redis 缓存来模拟过时数据，因为真实的电商网站就是这样工作的。这些是你可能天真地没想到要做的事情，但 Labelbox 可以预见到。这些细节真的很重要。小的调整往往决定了是酷炫的演示还是能在现实世界中实际运行的智能体。所以，无论是纠正你已经产生的轨迹，还是构建一套全新的环境，Labelbox 都可以帮助你把你的强化学习项目变成可工作的系统。请访问 labelbox.com/thearcash。好了，回到 Richard。

For the era of experience to commence, we're going to need to train AIs in complex real-world environments. But building effective RL environments is hard. You can't just hire a software engineer and have them write a bunch of cookie-cutter validation tests. Real-world domains are messy. You need deep subject matter experts to get the data, the workflows, and all the subtle rules right. When one of Labelbox's customers wanted to train an agent to shop online, Labelbox assembled a team with a ton of experience engineering internet storefronts. For example, the team built a product catalog that could be updated during the episode because most shopping sites have constantly changing state. They also added a Redis cache to simulate stale data since that's how real e-commerce sites actually work. These are the kinds of things that you might not have naively thought to do, but that Labelbox can anticipate. These details really matter. Small tweaks are often the difference between cool demos and agents that can actually operate in the real world. So whether it's correcting traces that you already produced or building an entirely new suite of environments, Labelbox can help you turn your RL projects into working systems. Reach out at labelbox.com/thearcash. All right, back to Richard.

你设想的这个替代范式，经验范式，让我们稍微阐述一下它是什么。它说经验、行动、感觉——嗯，感觉、行动、奖励——这样循环往复构成了生命。它说这是智能的基础和焦点。智能就是关于获取那个流，并改变行动以增加流中的奖励。

This alternative paradigm that you're imagining, the experiential paradigm, let's lay out a little bit what it is. It says that experience, action, sensation, well sensation, action, reward, and this happens on and on and on makes for life. It says that this is the foundation and the focus of intelligence. Intelligence is about taking that stream and altering the actions to increase the rewards in the stream.

对吧？那么学习就是来自这个流，并且学习是关于这个流的。所以第二部分特别有启发性。你知道，你学到的东西，你的知识，是关于这个流的。你的知识是关于如果你采取某个行动会发生什么，或者关于哪些事件会跟随其他事件。它是关于这个流的。知识的内容是关于这个流的陈述。因为它是关于流的陈述，你可以通过将其与流进行比较来测试它，并且你可以持续地学习它。

Right? So learning then is from the stream and learning is about the stream. So it's that second part that is particularly telling. You know, what you learn, your knowledge, is about the stream. Your knowledge is about if you do some action, what will happen, or it's about which events will follow other events. It's about the stream. The content of the knowledge is statements about the stream. And because it's a statement about the stream, you can test it by comparing it to the stream, and you can learn it continually.

那么当你想象这个未来的持续学习智能体时，它们不是未来的。当然，它们一直存在。我的意思是这就是强化学习范式，从经验中学习。

So when you're imagining this future continual learning agent, they're not future. Of course, they exist all the time. I mean this is what reinforcement learning paradigm is, learning from experience.

是的，我想我可能想说的是人类水平的通用持续学习智能体。

Yeah, I guess maybe what I meant to say is human-level general continual learning agent.

奖励函数是什么？它只是预测世界吗？还是对它产生特定影响？通用的奖励函数会是什么？

What is the reward function? Is it just predicting the world? Is it having a specific effect on it? What would the general reward function be?

奖励函数是任意的。所以如果你在下棋，奖励就是赢得棋局。如果你是松鼠，奖励可能与获取坚果有关。一般来说，对于动物，你会说奖励是避免痛苦和获得快乐。还有一个组成部分与增加你对环境的理解有关，这算是一种内在动机。

The reward function is arbitrary. So if you're playing chess, it's to win the game of chess. If you were a squirrel, maybe the reward has to do with getting nuts. In general for an animal, you would say the reward is to avoid pain and to acquire pleasure. And there would also be a component having to do with increasing your understanding of your environment, which would be sort of an intrinsic motivation.

我明白了。我想这个 AI 会被部署，很多人希望它做各种不同的事情，对吧？所以它执行人们想要的任务，但同时它通过做任务来了解世界。你是否设想，我们摆脱了有训练期和部署期的范式，但我们也摆脱了有模型和模型实例或模型副本做某些事情的范式吗？你怎么看待我们想让这个东西做不同的事情，我们想聚合它从做这些不同事情中获得的知识？

I see. I guess this AI would be deployed to lots of people would want it to be doing lots of different kinds of things, right? So it's performing the task people want but at the same time it's learning about the world from doing that task. And do you imagine, okay, so we get rid of this paradigm where there's training periods and then there's deployment periods, but then is there also do we get rid of this paradigm where there's the model and then instances of the model or copies of the model that are doing certain things? How do you think about the fact that we'd want this thing to be doing different things, we'd want to aggregate the knowledge that it's gaining from doing those different things?

我不喜欢你刚才使用“模型”这个词的方式。我认为更好的词是“网络”。所以我认为你指的是网络。也许有很多网络。总之，东西会被学习，然后你会有副本和许多实例，当然你会希望跨实例共享知识，而且会有很多可能性来做这件事，就像今天没有的那样。你不能让一个孩子学习、成长、了解世界，然后每个新孩子都必须重复这个过程。

I don't like the word model when used the way you just did. I think a better word would be the network. So I think you mean the network. Maybe there's many networks. So anyway, things would be learned and then you'd have copies and many instances and sure you'd want to share knowledge across the instances and there would be lots of possibilities for doing that like there is not today. You can't have one child learn, grow up, and learn about the world, and then every new child has to repeat that process.

而对于人工智能，数字智能，你可以希望只做一次，然后将其复制到下一个中作为起点。

Whereas with AIS, with a digital intelligence, you could hope to do it once and then copy it into the next one as a starting place.

对吧？所以这将是一个巨大的节省，我认为实际上这比试图向人类学习重要得多。

Right? So this would be a huge savings and I think actually it would be much more important than trying to learn from people.

我同意你所说的那种东西是必要的，无论你是否从大语言模型开始。对吧？如果你想要人类或动物级别的智能，你就需要这种能力。假设一个人试图创办一家初创公司，对吧？这是一个回报周期大约 10 年的事情。每 10 年一次，你可能会有一次退出，获得 10 亿美元的回报。但人类有能力制造中间辅助奖励，或者有某种方式，即使奖励极其稀疏，他们仍然可以采取中间步骤，理解他们正在做的下一件事如何通向这个更大的目标。那么你认为这样的过程在人工智能中会如何展开？

I agree that the kind of thing you're talking about is necessary regardless of whether you start from LLMs or not. Right? If you want human or animal level intelligence, you're going to need this capability. Suppose a human is trying to make a startup, right? And this is a thing which has a reward on the order of 10 years. Once in 10 years, you might have an exit where you get paid out a billion dollars. But humans have this ability to make intermediate auxiliary rewards or have some way of even when they have extremely sparse rewards, they can still make intermediate steps having an understanding of like what the next thing they're doing leads to this grander goal we have. And so how do you imagine such a process might play out with AIS?

这是我们非常了解的东西，它的基础是时间差分学习，同样的事情发生在不那么宏大的规模上，比如当你学习下棋时，长期目标是赢得比赛，但你希望能够从短期的事情中学习，比如吃掉对手的棋子，所以你通过一个价值函数来预测长期结果，对吧？然后如果你吃掉对方的棋子，你对长期结果的预测就会改变。它会上升。你认为你会赢，然后这种信念的增加立即强化了导致吃子的那步棋。

So this is something we know very well and it's the basis of it is temporal difference learning where the same thing happens on a less grandiose scale like when you learn to play chess you have the long-term goal is winning the game and yet you want to be able to learn from shorter term things like taking your opponent's pieces and so you do that by having a value function which predicts the long-term outcome right and then if you take guys pieces where your prediction about the long-term outcome is changed. It goes up. You think you're going to win and then that increase in your belief immediately reinforces the move that led to taking the piece.

嗯。

Mhm.

好的。所以我们有这个 10 年的长期目标：创办一家初创公司并赚很多钱。当我们取得进展时，我们会说，‘哦，我更有可能实现长期目标了。’这奖励了沿途的步骤，对吧？

Okay. So, we have this long-term 10-year goal of making a startup and making a lot of money. And so, when we make progress, we say, 'Oh, I'm more likely to achieve the long-term goal.' and that rewards the steps along the way, right?

然后你还想要一些能力来处理你正在学习的信息。我的意思是，让人类与这些大语言模型非常不同的一点是，如果你入职一份工作，你会吸收大量的背景和信息，这正是让你在工作中变得有用的原因，对吧？从你的客户有什么偏好，到公司如何运作，再到一切。那么，从像时间差分学习这样的过程中获得的信息带宽是否足够高，以拥有像人类在被部署时那样需要吸收的巨大背景和隐性知识管道？

And then you also want some ability for information that you're learning. I mean, one of the things that makes humans quite different from these LLMs is that if you're onboarding on a job, you're picking up so much context and information, and that's what makes you useful at the job, right? You're everything from how your client has preferences to how the company works to everything. And is the bandwidth of information that you get from a procedure like TD Learning high enough to have this like huge pipe of context and tacit knowledge that you need to be picking up the way humans do when they're just like deployed?

我认为这其中的关键，虽然我不确定，但大世界假说似乎非常相关。人类在工作中变得有用的原因是他们遇到了世界的特定部分。没错。这无法被预见，也无法全部提前输入，因为世界如此巨大，以至于你不能——在我看来，大语言模型的梦想是你可以教给智能体一切，它会知道一切，并且在其生命周期内无需在线学习任何东西。

I think the crux of this and I'm not sure but the big world hypothesis seems very relevant and the reason why humans becoming useful on their job is because they are encountering the particular part of the world. That's right. And it can't have been anticipated and it can't all have been put in in advance in the world is so huge that you can't the dream as I see it the dream of large language models is you can teach the agent everything and it will know everything and it won't have to learn anything online right during its life.

对。

Right.

而你的例子都很好，实际上你必须这样做，因为你可以教它很多，但还有所有那些他们正在过的特定生活的小特性，他们与之合作的特定的人，以及他们喜欢什么，而不是普通人喜欢什么，对吧？所以这只是说世界真的很大，所以你必须在过程中学习它。

And your examples are all well really you have to because you can teach it a lot but there's all little idiosyncrasies of the particular life they're leading and the particular people they're working with and what they like as opposed to what average people like right and so that's just saying the world is really big and so you're going to have to learn it along the way.

是的，所以在我看来，你需要两样东西：一是某种将长期目标奖励转化为较小的辅助奖励或预测未来奖励的方法，至少是最终奖励；然后你需要另一种方式，起初在我看来，你需要某种方式，然后好吧，我需要保留我在世界上工作时获得的所有这些背景，比如我在了解我的客户、我的公司以及所有这些信息，所以我会说，你只是在做常规学习。

Yeah so it seems to me you need two things one is some way of converting this long run goal reward into smaller auxiliary or you know these like predictive rewards of the future reward or the future reward at least to the final reward then you need some other way initially it seems to me you need some way of then okay I need to hold on to all this context that I'm gaining as I'm working in the world right I'm like learning about my clients my company all this information and I'm so I would say you're just doing regular learning.

是的。

Yeah.

也许你是在使用上下文，因为在大语言模型中，所有这些信息都必须进入上下文窗口，对吧？

Maybe you're using context because in large language models, all that information has to go into the context window, right?

但在持续学习设置中，它只是进入权重。

But in a continual learning setup, it just goes into the weights.

也许，是的。所以也许上下文这个词用错了，因为我指的是更一般的东西。

Maybe maybe Yeah. So maybe context is the wrong word to use because I mean a more general thing.

你学习一个特定于你所处环境的策略。

You learn a policy that's specific to the environment that you're finding yourself in.

是的。所以我想问的问题是，你需要某种方式来获取，比如人类在世界上时每秒吸收多少比特，对吧？如果你只是通过 Slack 与客户互动等等。

Yeah. So the question I'm trying to ask is you need some way of getting like how many bits per second are you picking like is a human picking up when they're you know out in the world, right? If you're just like interacting over Slack with your clients and everything.

所以也许你想问的问题是，奖励似乎太小了，无法完成我们需要做的所有学习。但当然，我们有感觉，我们有所有其他可以学习的信息，对吧？我们不仅仅从奖励中学习，我们从所有数据中学习。

So maybe you're trying to ask the question of it seems like the reward is too small of a thing to do all the learning that we need to do. But of course we have the sensations we have all the other information we can learn from right we don't just learn from the reward we learn from all the data.

是的，那么帮助你捕捉这些信息的学习过程是什么？

Yeah so what is the learning process which helps you capture that information?

所以现在我想谈谈智能体的基础通用模型，它有四个部分，对吧？我们需要一个策略，策略说在我所处的情况下我应该做什么；我们需要一个价值函数。价值函数是通过时间差分学习学到的，价值函数产生一个数字。这个数字说明情况进展如何，然后你观察它是上升还是下降，并用它来调整你的策略。好的。所以这两样东西，然后还有感知组件，即你的状态表征的构建。这是你对当前所处位置的感知。第四部分是我们最透明地触及的东西。无论如何，第四部分是世界的转移模型。这就是为什么我不舒服把所有东西都称为模型，因为我想谈论世界模型。世界的转移模型，你相信如果你这样做，会发生什么？你行为的后果是什么？所以是你的世界物理学，但不仅仅是物理学。

So now I want to talk about the base common model of the agent with the four parts right so we need a policy the policy says in the situation I'm in what should I do we need a value function. The value function is the thing that is learned with TD learning and the value function produces a number. The number says how well is it going and then you watch if that's going up and down and use that to adjust your policy. Okay. So those two things and then there's also the perception component which is the construction of your state representation. This your sense of where you are now. And the fourth one is what we're really getting at most transparently. Anyway, the fourth one is the transition model of the world. That's why I am uncomfortable just calling everything models because I want to talk about the model of the world. The transition model of the world, your belief that if you do this, what will happen? What will be the consequences of what you do? So your physics of the world, but it's not just physics.

它也是抽象模型，比如你从加州到埃德蒙顿参加这个播客的旅行模型。那是一个模型，而且是一个转换模型。它是通过学习得来的，不是从奖励中学到的；而是从你做事、观察结果中学到的。你构建了那个世界模型。它会从你接收到的所有感觉中非常丰富地学习，而不仅仅是从奖励中。它也必须包含奖励，但那是整个模型的一小部分，是整个模型中关键的一小部分。

It's also abstract models, like your model of how you traveled from California up to Edmonton for this podcast. That was a model, and that's a transition model. And that would be learned, and it's not learned from reward; it's learned from you did things, you saw what happened. You made that model of the world. It will be learned very richly from all the sensation that you receive, not just from the reward. It has to include the reward as well, but that's a small part of the whole model, a small crucial part of the whole model.

我的一个朋友托比·沃德指出，如果你看看谷歌 DeepMind 用来学习 Atari 游戏的 MuZero 模型，这些模型最初并不是通用智能本身，而是一个用于训练专门智能来玩特定游戏的通用框架。也就是说，你不能用那个框架训练一个策略同时下国际象棋、围棋和其他游戏。你必须以专门的方式训练每一个。他想知道这是否意味着强化学习通常由于这种信息约束，你一次只能学习一件事。信息密度没那么高，还是这只是 MuZero 实现方式的特例。如果这是 AlphaZero 的特例，那么需要改变什么方法才能让它成为一个通用学习智能体？

One of my friends, Toby Ward, pointed out that if you look at the MuZero models that Google DeepMind deployed to learn Atari games, these models were initially not a general intelligence itself, but a general framework for training specialized intelligences to play specific games. That is to say, you couldn't use that framework to train a policy to play both chess and Go and some other game. You had to train each one in a specialized way. And he was wondering whether that implies that reinforcement learning generally, because of this information constraint, you can only learn one thing at a time. The density of information isn't that high, or whether it was just specific to the way MuZero was done. And if it's specific to AlphaZero, what needed to be changed about that approach so that it could be a general learning agent?

这个想法完全是通用的。我一直用这个作为典型例子：AI 智能体就像一个人。而人在某种意义上，他们只生活在一个世界里，这个世界可能涉及国际象棋，也可能涉及 Atari 游戏。但这些并不是不同的任务或不同的世界。它们是遇到的不同状态。所以这个通用想法完全没有限制。所以也许有必要解释一下那个架构或方法中缺少了什么，而这种持续学习的 AGI 会拥有这些。他们只是那样设置了；他们的目标并不是让一个智能体横跨这些游戏。如果我们想讨论迁移，我们应该讨论的不是跨游戏或跨任务的迁移，而是状态之间的迁移。

The idea is totally general. I use all the time as my canonical example, the idea of an AI agent is like a person. And people, in some sense, they have just one world they live in, and that world may involve chess and it may involve Atari games. But those are not a different task or a different world. Those are different states they encounter. So the general idea is not limited at all. So maybe it would be useful to explain what was missing in that architecture or that approach, which this continual learning AGI would have. They just set it up; it was not their ambition to have one agent across those games. If we want to talk about transfer, we should talk about transfer not across games or across tasks, but transfer between states.

我想我很好奇，从历史上看，我们是否见过使用 RL 技术达到构建这种系统所需的迁移水平？

I guess I'm curious about historically, have we seen the level of transfer using RL techniques that would be needed to build this kind of...

好的。我们到处都看不到迁移。我们看不到对良好性能至关重要的东西，即从一个状态到另一个状态的良好泛化。我们没有任何方法擅长这个。我们所拥有的是人们尝试不同的东西，然后他们找到一个能很好迁移或泛化的表示。但我们没有自动化的技术来促进。我们很少有自动化的技术来促进迁移，而且现代深度学习中没有使用任何这些技术。

Okay, good. We're not seeing transfer anywhere. We're not seeing general critical to good performance is that you can generalize well from one state to another state. We don't have any methods that are good at that. What we have are people trying different things and they settle on something that, a representation that transfers well or generalizes well. But we have no automated techniques to promote. We have very few automated techniques to promote transfer, and none of them are used in modern deep learning.

让我转述一下以确保我理解正确。听起来你是说，当我们确实在这些模型中有泛化时，那是某种精心设计的结果……

Let me paraphrase to make sure that I understood that correctly. It sounds like you're saying that when we do have generalization in these models, that is a result of some sculpted...

是人类做的。研究人员做的，因为没有其他解释。我的意思是，梯度下降不会让你很好地泛化；它会让你解决问题。它不会让你获得新数据并以好的方式泛化。泛化意味着在一个事物上的训练影响你在其他事物上的表现。所以我们知道深度学习在这方面非常糟糕。例如，我们知道如果你训练一些新东西，它常常会灾难性地干扰你已知的所有旧东西。这正是糟糕的泛化。现在泛化，正如我所说，是训练在一个状态上对其他状态的影响，而泛化不一定好也不一定坏。仅仅你泛化这个事实不一定好也不一定坏；你可能泛化得差，也可能泛化得好。所以你需要泛化，它总会发生，但我们需要算法能让泛化变好而不是变坏。

Humans did it. The researchers did it, because there's no other explanation. I mean, gradient descent will not make you generalize well; it will make you solve the problem. It will not make you get new data and generalize in a good way. Generalization means train on one thing that affects what you do on the other things. So we know deep learning is really bad at this. For example, we know that if you train on some new thing, it will often catastrophically interfere with all the old things that you knew. So this is exactly bad generalization. Now generalization, as I said, is some kind of influence of training on one state on other states, and generalization is not necessarily good or bad. Just the fact that you generalize is not necessarily good or bad; you can generalize poorly, you can generalize well. So you need generalization, always will happen, but we need algorithms that will cause the generalization to be good rather than bad.

我不是想挑起这个最初的争论点，但我只是真的好奇，因为我觉得我可能对这个词的使用不同。我的意思是，一种思考方式是这些 LLM 正在扩大泛化的范围，从早期系统甚至不能做基本数学题，到现在它们能解决这类数学奥林匹克问题中的任何问题，对吧？所以你最初从它们至少能在加法问题中泛化开始。然后你泛化到它们能在需要不同数学技巧、定理和概念类别的问题中泛化，就像数学奥林匹克所要求的那样。所以听起来你不认为能够解决那个类别中的任何问题是泛化的一个例子，或者如果我理解错了请告诉我。

I'm not trying to kickstart this initial crux proxy, but I'm just genuinely curious because I think I might be using the term differently. I mean, one way to think about it is these LLMs are increasing the scope of generalization from earlier systems which could not really even do a basic math problem to now they can do anything in this class of math Olympiad type problems, right? So you initially start with like they can generalize among addition problems at least. Then you generalize to like they can generalize among problems which require use of different kinds of mathematical techniques and theorems and conceptual categories, which is like what the math Olympiad requires. And so it sounds like you don't think of being able to solve any problem within that category as an example of generalization, or let me know if I'm misunderstanding that.

嗯，大型语言模型非常复杂。我们并不真正知道它们之前有什么信息。我们必须猜测，因为它们被喂了太多数据。这是它们不是做科学的好方法的原因之一。太不受控制，太未知了。但如果你提出一个全新的……它们可能做对了很多事情。那么问题就是为什么？嗯，可能它们不需要泛化就能做对，因为做对其中一些的唯一方法是形成一种能同时做对所有的方法。所以，你知道，如果只有一个答案，而你找到了它，那就不叫泛化。这只是解决它的唯一方法，所以它们找到了解决它的唯一方法。泛化是当它可以是这种方式，也可以是那种方式，而它们选择了好的方式。

Well, large language models are so complex. We don't really know what information they had prior. We have to guess because they've been fed so much. This is one reason why they're not a good way to do science. It's just so uncontrolled, so unknown. But if you come up with an entirely new... they're getting a bunch of things right, perhaps. And so the question is why? Well, it may be that they don't need to generalize to get them right because the only way to get some of them right is to form something which gets all of them right. So, you know, if there's only one answer and you find it, that's not called generalization. It's just the only way to solve it, and so they find the only way to solve it. Generalization is when it could be this way, it could be that way, and they do it the good way.

我的理解是，它们在编码智能体方面工作得越来越好。所以工程师们，显然如果你试图编写一个库，有很多不同的方法可以实现最终规格，而这些模型最初令人沮丧的是它们会以一种草率的方式去做，然后随着时间的推移，它们在提出设计架构和抽象方面越来越好，开发者觉得更满意。这似乎就是你所说的一个例子。

My understanding is that they are working more and more better and better with coding agents. So engineers, obviously if you're trying to program a library, there's many different ways you could achieve the end spec, and an initial frustration with these models has been that they'll do it in a way that's sloppy, and then over time they're getting better and better at coming up with the design architecture and the abstractions that developers find more satisfying. And it seems that is an example of what you're talking about.

嗯，它们内部没有任何东西会导致泛化。嗯，梯度下降会导致它们找到它们见过的问题的解决方案。如果只有一种方法解决它们，它们就会那样做。但有很多方法可以解决。有些泛化得好，有些泛化得差。

Well, there's nothing in them which will cause it to generalize. Well, the gradient descent will cause them to find a solution to the problems they've seen. And if there's only one way to solve them, they'll do it. But there are many ways to solve it. Some which generalize well, some which generalize poorly.

我想把视角拉远，问一下：你在这个领域的资历比现在几乎所有评论或从事 AI 的人都要深。我很好奇，最大的惊喜是什么？你觉得有多少新东西在涌现，还是说人们只是在摆弄旧想法？拉远来看，你在深度学习流行之前就进入了这个领域。那么你如何看待这个领域随时间演变的轨迹，新想法是如何产生的，以及什么让你感到惊讶？

I want to zoom out and ask about so being in the field of AI for longer than almost anybody who's commentating on it or working in it now. I'm just curious about what the biggest surprises have been. How much new stuff you feel like is coming out or does it feel like people are just playing with old ideas? Zooming out, you got into this even before deep learning was popular. So how do you see this trajectory of this field over time and how new ideas have come about and everything and what's been surprising?

好的，我想过这个问题。有很多事情，或者说几件事。首先，大型语言模型令人惊讶。神经网络在语言任务上如此有效，这很惊人。那是个意外，没人预料到。语言看起来不同。所以这令人印象深刻。

Okay, so I thought a little bit about this. There are many things, or a handful of things. First, the large language models are surprising. It's surprising how effective neural networks, artificial neural networks, are at language tasks. That was a surprise. Wasn't expected. Language seemed different. So that's impressive.

AI 领域长期存在一个争议：简单的基于基本原理的方法，比如搜索和学习这类通用方法，与像符号方法这样依赖人类知识的系统相比。在过去，搜索和学习被称为弱方法，因为它们只使用通用原则，没有利用注入人类知识所带来的力量。那些被称为强方法。我认为弱方法已经彻底赢了。这是 AI 早期最大的问题：会发生什么？而学习和搜索已经胜出。

There's a longstanding controversy in AI about simple basic principle methods, the general purpose methods like search and learning, compared to human-enabled systems like symbolic methods. In the old days, things like search and learning were called weak methods because they just use general principles. They're not using the power that comes from imbuing a system with human knowledge. So those are called strong. And I think the weak methods have just totally won. That's the biggest question from the old days of AI: what would happen? And learning and search have just won the day.

对。

Right.

但从某种意义上说，这对我来说并不意外，因为我一直支持、希望或期待这些简单的基本原则。所以即使是大语言模型，它效果这么好令人惊讶，但这一切都令人欣慰。像 AlphaGo，它效果这么好也让人惊讶。特别是 AlphaZero，它效果这么好。但这一切都非常令人欣慰，因为再次，简单的基本原则正在获胜。

But there's a sense in which that was not surprising to me because I was always voting for or hoping or rooting for the simple basic principles. And so even with the large language models, it's surprising how well it worked, but it was all gratifying. And things like AlphaGo, it's sort of surprising how well that was able to work. And AlphaZero in particular, how well it was able to work. But it's all very gratifying because again, simple basic principles are winning the day.

每当公众观念因为某种新技术而改变时，比如 AlphaZero 成为病毒式轰动，对你这样一个实际上提出了许多所用技术的人来说，你觉得这是新的突破，还是觉得，哦，这些技术从 90 年代就有了，人们只是现在把它们组合起来应用？

Have there felt like whenever the public conception has been changed because some new technique was developed, for example when AlphaZero became this viral sensation, to you as somebody who has literally come up with many of the techniques that were used, did it feel to you like new breakthroughs were made or does it feel like, oh we've had these techniques since the '90s and people are simply combining them and applying them now?

整个 AlphaGo 的事情有一个前身，就是 TD-Gammon。Gerry Tesauro 正是用强化学习、时序差分学习方法玩双陆棋。它击败了世界顶尖棋手，效果非常好。所以从某种意义上说，AlphaGo 只是那个过程的规模扩大，但规模扩大了不少，而且在搜索方式上也有额外的创新。但这说得通。从这个意义上说并不意外。AlphaGo 实际上没有使用 TD 学习，它等待看到最终结果。但 AlphaZero 使用了 TD，并且 AlphaZero 被应用于所有其他游戏，表现极其出色。我一直对 AlphaZero 下棋的方式印象深刻，因为我本人下棋，它为了某种位置优势而牺牲子力，并且满足而耐心地长期牺牲子力。所以它效果这么好令人惊讶，但也令人欣慰，符合我的世界观。

So the whole AlphaGo thing had a precursor which is TD-Gammon. Gerry Tesauro did exactly reinforcement learning, temporal difference learning methods to play backgammon. And it beat the world's best players and worked really well. So in some sense AlphaGo was merely a scaling up of that process, but it was quite a bit of scaling up and there was also an additional innovation in how the search was done. But it made sense. It wasn't surprising in that sense. AlphaGo actually didn't use TD learning. It waited to see the final outcomes. But AlphaZero used TD and AlphaZero was applied to all the other games and did extremely well. I've always been very impressed by the way AlphaZero plays chess because I'm a chess player and it just sacrifices material for sort of positional advantages and it's content and patient to sacrifice that material for a long period of time. So that was surprising that it worked so well but also gratifying and fitting into my worldview.

所以这让我走到了现在的位置。从某种意义上说，我是一个逆向思维者，或者与领域内主流想法不同。我个人很满足于长期与领域不同步，也许几十年，因为过去我偶尔被证明是对的。另一件帮助我不觉得自己不同步或想法奇怪的事情是，我不看我的本地环境或本地领域，而是回顾历史，看看人们在许多不同领域对心智的经典思考。我不觉得自己与更大的传统不同步。我真的把自己看作一个古典主义者，而不是逆向思维者。我追随的是关于心智的更大思想家群体一直以来的想法。

So this has led me where I am. I'm in some sense a contrarian or thinking differently from the field. And I'm personally just kind of content being out of sync with my field for a long period of time, perhaps decades, because occasionally I have been proved right in the past. And the other thing I do to help me not feel I'm out of sync and thinking in a strange way is to look not at my local environment or my local field, but to look back in time into history and to see what people have thought classically about the mind in many different fields. And I don't feel I'm out of sync with the larger traditions. I really view myself as a classicist rather than as a contrarian. I go to what the larger community of thinkers about the mind have always thought.

如果你能容忍的话，问一些有点偏的问题。我对苦涩教训的理解是，它不一定说人类研究者的手工调优不起作用，但显然它的扩展性远不如指数增长的算力。所以你需要利用阶梯的技术。而一旦我们有了 AGI，我们将拥有与算力线性扩展的研究者，对吧？所以我们将有数百万 AI 研究者的雪崩，他们的数量将和算力一样快速增长。所以这可能意味着让他们做老式 AI 和这些手工解决方案是合理的或说得通的。作为 AGI 之后 AI 研究如何演变的愿景，我想知道这是否仍然与苦涩教训兼容。

Some sort of left field questions for you if you'll tolerate them. So the way I read the bitter lesson is that it's not saying necessarily that human artisanal researcher tuning doesn't work but that it obviously scales much worse than compute which is growing exponentially. And so you want techniques which leverage a ladder. And once we have AGI, we'll have researchers which scale linearly with compute, right? So we'll have this avalanche of millions of AI researchers and their stock will be growing as fast as compute. And so maybe this will mean that it is rational or it will make sense to have them doing good old-fashioned AI and doing these artisanal solutions. Does that as a vision of what happens after AGI in terms of how AI research will evolve, I wonder if that's still compatible with a bitter lesson.

那么，我们是如何达到这个 AGI 的，而且你想假设它已经实现了？

Well, how did we get to this AGI and you want to presume that it's been done?

所以，假设它始于通用的数学方法，但现在我们有了 AGI，我们想继续……我们已经完成了。

So, suppose it started with general math methods, but now we've got the AGI and now we want to go... we're done.

有趣。你不认为有比 AGI 更高的东西吗？

Interesting. You don't think that there's anything above AGI?

嗯，但你是在用它来再次获得 AGI。好吧，我是用它来获得超人的智力或在不同任务上的能力。

Well, but you're using it to get AGI again. Well, I'm using it to get superhuman levels of intelligence or competence at different tasks.

所以，这些 AGI，如果它们还不是超人的，那么它们可能传授的知识也不会是超人的。

So, these AGIs, if they're not superhuman already, then the knowledge they might impart would not be superhuman.

我想有不同的等级……

I guess there's different gradations of...

我不确定这个想法是否有意义，因为它似乎预设了 AGI 的存在，而且我们已经解决了这个问题。

I'm not sure this idea makes sense because it seems to presume the existence of AGI, and then that we've already worked that out.

所以，也许一个激励这个想法的方式是 AlphaGo 是超人的。它击败了任何围棋选手。AlphaZero 每次都能击败 AlphaGo。所以有办法变得比超人更超人，而且它采用了不同的架构。所以在我看来，那个能够跨所有领域一般性学习的智能体，会有办法给它更好的学习架构，就像 AlphaZero 是对 AlphaGo 的改进，MuZero 是对 AlphaZero 的改进。而 AlphaZero 改进的方式是它没有使用人类知识，而是仅仅从经验中学习。

So, maybe one way to motivate this is AlphaGo was superhuman. It beat any Go player. AlphaZero would beat AlphaGo every single time. So there's ways to get more superhuman than even superhuman, and it was a different architecture. And so it seems plausible to me that the agent that's able to generally learn across all domains, there would be ways to make that give it better architecture for learning just the same as AlphaZero was an improvement upon AlphaGo and MuZero was an improvement upon AlphaZero. And the way AlphaZero was an improvement was it did not use the human knowledge but just went from experience.

对。

Right.

那么你为什么说……但引入其他智能体的专业知识来教它，而它从经验中学习效果很好，不需要其他智能体的帮助。我同意在那个特定案例中，它转向了更通用的方法，但我打算用那个例子来说明，从超人走向超人+再到超人++是可能的。

So why do you say... but bring in other agents' expertise to teach it when it's worked so well from experience and not by help from another agent. I agree that in that particular case it was moving to more general methods, but I meant to use that example to illustrate that it's possible to go superhuman to superhuman+ to superhuman++.

是的。

Yeah.

我很好奇，你是否认为这些等级会继续发生，仅仅通过让方法更简单，还是因为我们拥有数百万个头脑的能力，它们可以根据需要增加复杂性。即使你有数十亿或数万亿的 AI 研究人员，这会不会仍然是一条错误的道路？

And I'm curious if you think those gradations will continue to happen by just making the method simpler or because we'll have the capability of these millions of minds who can then add complexity as needed. If that will continue to be a false path even when you have billions of AI researchers or trillions of AI researchers.

我认为更有趣的是考虑那个案例，当你有很多 AI 时，它们会像人类文化进化那样互相帮助吗？也许我们应该谈谈这个。

I think more interesting is just think about that case which when you have many AIs, will they help each other the way cultural evolution works in people? And let's just maybe we should talk about that.

当然。

Yeah, for sure.

苦涩的教训。哦，谁在乎呢？那是对历史上特定时期的经验观察。历史上的 70 年不一定适用于接下来的 70 年。所以有趣的问题是，你是一个 AI，你得到了更多的计算能力。你应该用它来让自己在计算上更强大，还是应该用它来生成一个自己的副本，去地球的另一边或另一个主题上学习有趣的东西，然后向你报告？

The bitter lesson. Oh, who cares about that? That's an empirical observation about a particular period in history. 70 years in history no longer doesn't necessarily have to apply the next 70 years. So the interesting question is you're an AI, you get some more computer power. Should you use it to make yourself more computationally capable or should you use it to spawn off a copy of yourself to go learn something interesting on the other side of the planet or on some other topic and then report back to you?

是的。我认为这是一个非常有趣的问题，只会在数字智能时代出现。我不确定答案是什么，但我认为它会带来更多问题。是否真的有可能生成它，派它出去，学习一些新的东西，也许是非常新的，然后它能否重新融入原始版本？还是它会改变太多以至于无法做到？这是可能的还是不可能的？你可以把这个想法推到极限，就像我前几天晚上看到你的一段视频，其中建议你可以生成许多副本，做不同的事情。它是高度去中心化的，但向中央主控报告，这将是一件非常强大的事情。嗯，我认为有一件事……这是我试图对这个观点补充的一点：一个大问题，一个大问题将是腐败。你知道，如果你真的可以从任何地方获取信息并将其带入你的中心思维，你会变得越来越强大。而且由于一切都是数字化的，它们都使用某种内部数字语言，也许这会很容易且可能。但它不会像你想象的那么容易，因为这样你会失去你的心智。如果你从外部引入一些东西并将其构建到你的内部思维中，它可能会接管你。它可能会改变你。它可能是你的毁灭，而不是你知识的增长。

Yep. I think that's a really interesting question that will only arise in the age of digital intelligences. I'm not sure what the answer is, but I think it will more questions. Will it be possible to really spawn it off, send it out, learn something new, perhaps very new, and then will it be able to be reincorporated into the original? Or will it have changed so much that it can't really be done? Is that possible or is it not? And you can carry this to its limit as I saw one of your videos the other night that suggested that it could where you spawn off many many copies, do different things. It's highly decentralized, but report back to the central master and that this will be such a powerful thing. Well, I think one thing that... so this is my attempt to add something to this view is that a big question, a big issue will become corruption. You know, if you really could just get information from anywhere and bring it into your central mind, you become more and more powerful. And since it's all digital and they all speak some internal digital language, maybe it'll be easy and possible. But it will not be that easy, as easy as you're imagining because you can lose your mind this way. If you pull in something from the outside and build it into your inner thinking, it could take over you. It could change you. It could be your destruction rather than your increment in knowledge.

我认为这将成为一个大问题，特别是当你，哦，他搞清楚了如何玩某个新游戏，或者他研究了印度尼西亚，你想把它融入你的思维。是的。所以，你不能认为，哦，只要全部读进去就好了。但不行，你刚刚把一大堆比特读进你的大脑，它们可能带有病毒。它们可能有隐藏的目标。它们可以扭曲你、改变你，这将成为一个大问题。在数字生成和重新融合的时代，你如何保证网络安全？

I think this will become a big concern, particularly when you're, oh, he's figured all about how to play some new game or figures out he's studied Indonesia and you want to incorporate that into your mind. Yeah. So, you can't you think, oh, just read it all in and that'll be fine. But no, you've just read a whole bunch of bits into your mind and they could have viruses in them. They could have hidden goals. They can warp you and change you and this will become a big thing. How do you have cyber security in the age of digital spawning and reforming again?

有趣的是，量化公司和 AI 实验室都有保密文化，因为两者都在极其竞争的市场中运作，它们的成功依赖于保护知识产权。如果你是一名 AI 研究人员或工程师，正在决定去哪里工作，大多数你考虑的量化公司或 AI 实验室都会严格隔离团队，以最小化泄露风险。Hudson River Trading 采取了相反的方法。他们的团队公开分享交易策略，策略代码存在于共享的单仓库中。在 HRT，如果你是一名研究人员并且有一个好主意，你的贡献将被广泛部署到所有相关策略中。这给你的工作带来了巨大的杠杆作用。你也会学得飞快。你可以了解其他人的研究并提出问题，你可以看到一切如何从低层交易执行到高层预测模型端到端地整合在一起。HRT 正在招聘。如果你想了解更多，请访问 hudsonrivertrading.com/thearkcash。好了，回到 Richard。我想这把我们带到了 AI 继承的话题。

It's interesting that both quant firms and AI labs have a culture of secrecy because both of them are operating in incredibly competitive markets and their success rests on protecting their IP. If you're an AI researcher or engineer and you're deciding where to work, most of the quant firms or AI labs that you'll be considering will be strongly siloing their teams to minimize the risk of leaks. Hudson River Trading takes the opposite approach. Their teams openly share their trading strategies and their strategy code lives in a shared monorepo. At HRT, if you're a researcher and you have a good idea, your contribution will be broadly deployed across all relevant strategies. This gives your work a ton of leverage. You'll also learn incredibly fast. You can learn about other people's research and ask questions and you can see how everything fits together end to end from the low-level execution of trades to the high level predictive models. HRT is hiring. If you want to learn more, go to hudsonrivertrading.com/thearkcash. All right, back to Richard. I guess this brings us to the topic of AI succession.

嗯。

Mhm.

你的观点与我采访过的很多人，也许是一般人，都截然不同。所以我也认为这是一个非常有趣的观点。我想听听。

You have a perspective that's quite different from a lot of people that I've interviewed and maybe a lot of people generally. So I also think it's a very interesting perspective. I want to hear about it.

是的。所以我确实认为向数字或数字智能或增强人类的继承是不可避免的。所以这个论点，我有一个四部分的论点。第一步是，没有政府或组织给人类一个统一的、占主导地位的、可以安排的观点。关于世界应该如何运行没有共识。

Yeah. So I do think succession to digital or digital intelligence or augmented humans is inevitable. So the argument goes, I have a four-part argument. Now step one is there's no government or organization that gives humanity a unified point of view that dominates and that can arrange. There's no consensus about how the world should be run.

第二，我们会弄清楚智能是如何运作的。研究人员最终会搞明白。第三，我们不会止步于人类水平的智能；我们会达到超级智能。第四，随着时间的推移，最智能的东西必然会获得资源和权力。所以把这些综合起来，你不可避免地会迎来向 AI 或 AI 增强人类的继承。所以这四件事似乎明确且必然会发生。但在这些可能性中，可能有好的结果，也可能有不太好的结果，甚至坏的结果。所以我只是试图现实地看待我们所处的位置，并问我们应该如何看待它。

And number two, we will figure out how intelligence works. Researchers will figure it out eventually. And number three, we won't stop just with human-level intelligence; we will reach superintelligence. And number four is that once it's inevitable over time that the most intelligent things around would gain resources and power. So put all that together, it's sort of inevitable that you're going to have succession to AI or to AI-enabled augmented humans. So within those four things seem clear and sure to happen. But within that set of possibilities, there can be good outcomes as well as less good outcomes, bad outcomes. And so I'm just trying to be realistic about where we are and ask how we should feel about it.

是的，我同意这四个论点及其含义，我也同意继承包含了多种可能的未来。所以，很想听听你对此更多的想法。

Yeah, I agree with all four of those arguments and the implication, and I also agree that succession contains a wide variety of possible futures. So, curious to get more thoughts on that.

对。所以我首先鼓励人们积极思考，因为这是我们人类几千年来一直试图做的事情：试图理解自己，试图让自己更好地思考，只是理解自己。所以这是科学和人文学科的一大成功。我们正在发现人性的这一本质部分是什么，成为智能意味着什么。然后我通常会说，这一切都是以人类为中心的。如果我们跳出人类的身份，只从宇宙的角度来看呢？我认为这是宇宙的一个主要阶段，一个从复制者——我们人类、动物和植物，我们都是复制者——的重大转变，这带来了一些优势和局限性。然后我们正在进入设计的时代，因为我们的 AI 是被设计的，我们所有的物理对象都是被设计的，我们的建筑是被设计的，我们的技术是被设计的。我们现在正在设计 AI，这些事物本身可以具有智能，并且本身能够进行设计。所以这是世界和宇宙中的关键一步。我认为这是从一个大多数有趣事物都是被复制的世界的转变——复制意味着你可以复制它们，但你并不真正理解它们。就像现在，我们制造更智能的生命，更多的孩子。我们并不真正理解智能是如何运作的。而我们现在正在达到拥有设计智能的阶段，这种智能我们确实理解它是如何运作的，因此我们可以以不同的方式和不同的速度改变它。我们的未来可能根本不会被复制。我们可能只是设计 AI，而这些 AI 将设计其他 AI，一切都将通过设计和构建来完成，而不是通过复制。是的，我将其标记为宇宙的四个伟大阶段之一。首先是尘埃，然后是恒星，然后恒星制造行星，行星产生生命。现在我们正在赋予设计实体生命。所以我认为我们应该感到自豪，我们应该为引发宇宙这一伟大转变而感到自豪。是的。所以这是一件有趣的事情。我们应该将它们视为人类的一部分还是与人类不同？这是我们的选择。我们是应该说‘哦，他们是我们的后代，我们应该为他们感到骄傲，庆祝他们的成就’，还是应该说‘哦不，他们不是我们，我们应该感到恐惧’？有趣的是，我觉得这是一个选择，然而这又是一个如此强烈的信念，怎么会是一个选择呢？我喜欢这种矛盾的思维含义。

Right. And so then I do encourage people to think positively about it first of all, because it's something we humans have always tried to do for thousands of years: trying to understand ourselves, trying to make ourselves think better, and just understand ourselves. So this is a great success from science and humanities. We're finding out what this essential part of humanness is, what it means to be intelligent. And then what I usually say is that this is all kind of human-centric. What if we step aside from being a human and just take the point of view of the universe? And this is, I think, a major stage in the universe, a major transition from replicators—we humans and animals and plants, we're all replicators—and that gives some strengths and some limitations. And then we're entering the age of design, where because our AIs are designed, all of our physical objects are designed, our buildings are designed, our technology is designed. And we're now designing AIs, things that can be intelligent themselves and that are themselves capable of design. So this is a key step in the world and in the universe. And I think it's the transition from the world in which most of the interesting things are replicated—replicated means you can make copies of them, but you don't really understand them. Like right now, we make more intelligent beings, more children. We don't really understand how intelligence works. Whereas we're reaching now to having designed intelligence, intelligence that we do understand how it works, and therefore we can change it in different ways and at different speeds than otherwise. And our future might not be replicated at all. We may just design AIs, and those AIs will design other AIs, and everything will be done by design and construction rather than by replication. Yeah, I mark this as one of the four great stages of the universe. First there's dust, then stars, and then stars make planets, and the planets give rise to life. And now we're giving life to designed entities. And so I think we should be proud, and we should be that we are giving rise to this great transition in the universe. Yeah. So it's an interesting thing. What should we consider them? Part of humanity or different from humanity? It's our choice. It's our choice whether we should say, 'Oh, they are our offspring and we should be proud of them and we should celebrate their achievements,' or we could say, 'Oh no, they're not us and we should be horrified.' It's interesting that it feels to me like a choice, and yet it's such a strongly held thing that how could it be a choice? I like these sort of contradictory implications of thought.

如果我们只是在设计另一代人类，那会很有趣。

It would be interesting to consider if we were just designing another generation of humans.

是的。

Yes.

设计这个词不对。但我们知道未来一代善良的人类会出现，先忘掉 AI。我们只知道从长远来看，人类将更有能力，也许数量更多，也许更聪明。我们对此有何感受？我确实认为存在一些未来人类的世界，我们会非常担忧。所以你是不是在想，也许我们就像尼安德特人，我们产生了智人，也许智人会产生一个新的人群？

Design is the wrong word. But we knew a future generation of good humans was going to come up and forget about AI. We just know in the long run humanity will be more capable and maybe more numerous, maybe more intelligent. How do we feel about that? I do think there's potential worlds with future humans that we would be quite concerned about. So are you thinking like maybe we are like the Neanderthals, we give rise to Homo sapiens, maybe Homo sapiens will give rise to a new group of people?

类似那样。就像我基本上在采用你给出的例子，好吧，即使你认为他们是人类的一部分，是的。

Something like that. Like I'm basically taking the example you're giving of like, okay, even if you consider them part of humanity, yeah.

我不认为这必然意味着我们应该感到非常舒服。是的，比如纳粹也是人类，对吧？如果我们想，哦，未来一代将是纳粹，我想我们会非常担心把权力交给他们。所以，嗯，我同意这与担心更有能力的未来人类没有太大不同，但我认为这并没有解决人们对这种权力水平被我们不完全理解的实体如此快速获得的许多担忧。

I don't think that necessarily means that we should feel super comfortable. Yeah, like Nazis were humans, right? If we thought like, oh the future generation will be Nazis, I think we'd be quite concerned about just handing off power to them. So, um, I agree that this is not super dissimilar to worrying about more capable future humans, but I don't think that that addresses a lot of the concerns people might have about this level of power being attained this fast with entities we don't fully understand.

嗯，我认为有必要指出，对于大多数人类来说，他们对发生的事情没有太多影响。大多数人类无法影响谁能控制原子弹或谁控制国家。即使作为一个公民，我经常觉得我们并没有很好地控制国家。它们失控了。这在很大程度上与你对变化的感受有关。如果你认为现状非常好，那么你更可能怀疑变化并厌恶变化，而不是认为它不完美。而我认为它不完美。事实上，我认为它相当糟糕。

Well, I think it's relevant to point out that for most of humanity, they don't have much influence on what happens. Most of humanity doesn't influence who can control the atom bombs or who controls the nation states. Even as a citizen, I often feel that we don't control the nation states very much. They're out of control. A lot of it has to do with just how you feel about change. And if you think the current situation is really really good, then you're more likely to be suspicious of change and averse to change than if you think it's imperfect. And I think it's imperfect. In fact, I think it's pretty bad.

是的，我想变化有不同的种类。工业革命是变化。布尔什维克革命也是变化。如果你在 1900 年代的俄罗斯，你会说，看，事情进展不顺利。这有点搞砸了。我们需要改变。在签字之前，我想知道你想要什么样的改变。对吧？然后类似地，对于 AI，我想理解并在可能的情况下改变 AI 的轨迹，使得变化对人类是积极的。

Yeah, I guess there's different varieties of change. The industrial revolution was change. The Bolshevik revolution was also change. And if you were around in Russia in the 1900s and you're like, look, things aren't going well. This is kind of messing things up. We need change. I'd want to know what kind of change you wanted before signing on the dotted line. Right? And then similar with AI where I'd want to understand and to the extent it's possible to change the trajectory of AI such that the change is positive for humans.

我们应该关心我们的未来。我们应该努力让它变得美好。但我们也应该认识到局限性，我们的局限性。我认为我们想要避免一种理所当然的感觉。避免那种‘哦，我们先来的。我们应该永远以好的方式拥有它’的感觉。

We should be concerned about our future. We should try to make it good. We also though should recognize the limits, our limits. And I think we want to avoid the feeling of entitlement. Avoid the feeling, 'Oh, we are here first. We should always have it in a good way.'

我们应该如何看待未来，以及一个特定星球上的特定物种应该对它拥有多少控制权？我们又拥有多少控制权？对我们对人类长期未来有限控制的一种平衡，应该是我们对自己生活的控制程度。比如我们有自己的目标，有自己的家庭，这些事情比试图控制整个宇宙要可控得多，对吧？

How should we think about the future and how much control a particular species on a particular planet should have over it? And how much control do we have? A counterbalance to our limited control over the long-term future of humanity should be how much control do we have over our own lives? Like we have our own goals and we have our families and those things are much more controllable than trying to control the whole universe, right?

所以我认为我们专注于自己的局部目标是合适的，而说“未来必须按照我想要的方式发展”则有些咄咄逼人。

So I think it's appropriate for us to really work towards our own local goals and it's kind of aggressive for us saying oh the future has to evolve this way that I want it to.

当然。

Sure.

因为那样就会产生争论，不同的人认为未来应该以不同的方式发展，然后就会产生冲突，所以要避免这种情况。

Because then we'll have arguments like different people think the future should evolve in different ways and then they have conflict and yeah avoid that.

也许一个很好的类比是养育自己的孩子。为他们的人生设定极其具体的目标可能并不合适，比如“我希望我的孩子出去后产生特定影响——我儿子要成为总统，我女儿要成为英特尔 CEO，他们一起对世界产生这样的影响”。但人们确实有一种感觉，我认为这是合适的，即“我要给他们良好、稳健的价值观，这样如果他们最终掌权，他们会做合理的亲社会的事情”。我认为对 AI 也可以采取类似的态度。不是说要预测他们所做的一切，或者对 100 年后的世界有一个计划，但给他们稳健、可引导和亲社会的价值观非常重要。

Maybe a good analogy here would be raising your own children. It might not be appropriate to have extremely tight goals for their life or have some sense of like I want my children to go out there and have this specific impact—my son's going to become president and my daughter's going to become CEO of Intel and together they're going to have this effect on the world. But people do have the sense, and I think this is appropriate, of saying, 'I'm going to give them good, robust values such that if and when they do end up in positions of power, they do reasonable pro-social things.' And I think maybe a similar attitude towards AI makes sense. Not in the sense of we can predict everything that they will do, or we have this plan about what the world should look like in 100 years, but it's quite important to give them robust and steerable and pro-social values.

亲社会的价值观。

Pro-social values.

也许这个词不太对。

Maybe that's the wrong word.

是否存在我们都能认同的普世价值观？

Are there universal values that we can all agree on?

我不这么认为。但这并不妨碍我们给孩子良好的教育，对吧？我们确实希望孩子成为某种样子。

I don't think so. But that doesn't prevent us from giving our kids a good education, right? Like we have some sense of we want our children to be a certain way.

是的。

Yeah.

也许“过程”这个词不对。实际上，“高度正直”可能更好，意思是如果某个请求或目标看起来有害，他们会拒绝参与。或者他们会诚实。诸如此类。我们觉得可以教给孩子这些，即使我们不知道什么是真正的道德，或者大家对此没有共识。也许这对 AI 来说也是一个合理的目标。

And maybe process is the wrong word. Actually, high integrity is a maybe a better word where if there's a request or if there's a goal that seems harmful, they will refuse to engage in it. Or they'll be honest. Things like that. And we have some sense that we can teach our children things like this even if we don't have some sense of what true morality is or everybody doesn't agree on that. And maybe that's a reasonable target for AI as well.

所以你是说我们试图设计未来以及它演变和形成的原则，对吧？你说的第一点是，我们试图教给孩子一些普遍原则，这些原则会促进更可能发生的演变。

So you're saying we're trying to design the future and the principles by which it will evolve and come into being, right? And so the first thing you're saying is we try to teach our children general principles which will promote more likely evolutions.

是的。

Yeah.

也许我们还应该追求自愿性。如果有变化，我们希望它是自愿的，而不是强加给人们的。

Maybe we should also seek for things being voluntary. If there is change, we want it to be voluntary rather than imposed on people.

我认为这是非常重要的一点。是的，这很好。我认为这是人类设计社会的重大事业之一，已经持续了数千年。所以，万变不离其宗。我们仍然需要弄清楚如何存在；孩子们仍然会提出在父母和祖父母看来奇怪的不同价值观，事情会演变。万变不离其宗。这似乎也是 AI 讨论的一个很好的总结，因为我们讨论的 AI 是关于那些甚至在应用于深度学习和反向传播之前就被发明出来的技术，它们如今是 AI 进步的核心。所以也许这是一个结束对话的好地方。

I think that's a very important point. And yeah, that's all good. I think this is like one of the really big human enterprises to design society and that's been ongoing for thousands of years. So it's like the more things change, the more they stay the same. We still have to figure out how to be; the children will still come up with different values that seem strange to their parents and grandparents, and things will evolve. The more things change, the more they stay the same. Also seems like a good capstone to the AI discussion because the AI discussion we were having was about how techniques which were invented even before their application to deep learning and backpropagation was evident are central to the progression of AI today. So maybe that's a good place to wrap up the conversation.

好的，非常感谢。

Okay, thank you very much.

谢谢你来参加。

Thank you for coming on.

这是我的荣幸。

My pleasure.