强化学习：从经验中学习

大家好，欢迎收听《机器学习：我们是如何走到这一步的？》。我是 Tom Mitchell，今天的节目是对 Rich Sutton 的访谈。Rich 是机器学习中强化学习领域的资深研究者。他与长期合作者 Andy Barto 在这个领域深耕数十年。90 年代末，他们合著了一本关于强化学习的书。2024 年，Andy 和 Rich 因对强化学习的贡献共同获得了 ACM 图灵奖。在开始对话之前，我先简单解释一下什么是强化学习。在机器学习中，我们根据可用的训练信息类型，对学习问题有不同的定义框架。主导的范式称为监督学习。例如，在学习下国际象棋时，监督学习设定假设每当你处于某个棋局并试图走棋时，都有一个老师监督你，告诉你哪一步是正确的、最好的走法。在这种框架下，我们寻找能利用这类数据（棋局和正确走法）的算法。而在强化学习中，假设没有老师。你只需下棋，在游戏结束时才知道输赢。奖励或惩罚可能与你之前走的某一步相隔甚远。实际上，你在游戏中走出一系列步骤后，才能知道输赢。强化学习就是关于这种没有老师的第二种问题框架。

Hello and welcome to machine learning, how did we get here? My name is Tom Mitchell and today's episode is an interview with Rich Sutton. Rich is a long-time researcher in a branch of machine learning known as reinforcement learning. He and his long-time collaborator Andy Barto have done a lot of work in this area over decades. In the late 90s, they wrote a book in this area. In 2024, Andy and Rich jointly received the ACM Turing Award for their contributions to reinforcement learning. Now, before we get started with the conversation, let me explain very briefly what reinforcement learning is. In machine learning, we have a number of different framings of what the learning problem is, partly based on what kind of training information is available. The dominant paradigm or framing of the machine learning problem is called supervised learning. For example, if you're learning to play a game like chess, then in the supervised learning problem setting, we assume that every time you're in a particular board position and try to make a move, there's a teacher available to supervise you and to tell you which move is the correct move, the best move in this particular position. And in that kind of framing, we seek machine learning algorithms that can use that kind of data: here's a board position, here's the correct move. In reinforcement learning, we assume there is no teacher. Instead, you must simply play the game and at the end of the game, of course, you find out whether you win or lose. A reward or a penalty, but it can be quite distant in time from the move that you made. In fact, you make a sequence of moves in a game before you find out whether you win or lose. Reinforcement learning is about that second kind of framing of the problem where there is no teacher.

好的，今天和我一起的是 Rich Sutton，他是机器学习的先驱之一，尤其对强化学习这一分支做出了巨大贡献。Rich，很高兴你能来。非常感谢，Tom。我先问一个问题：你能用一句话定义强化学习吗？我得小心从哪里开始，因为我只有一句话。这句话可以很长。

Okay, today I have with me Rich Sutton, one of the pioneers of machine learning, who in particular contributed immensely to a part of machine learning called reinforcement learning. Rich, great to have you with us. Thank you so much, Tom. Let me start off with just asking, can you define reinforcement learning in one sentence? I have to be careful where I start because I only get one sentence. It could be a long sentence.

强化学习是通过试错从经验中学习以实现目标。

Reinforcement learning is learning from experience by trial and error to achieve a goal.

好的。现在你可以随便说多少句都行。跟我们讲讲强化学习的历史，你是如何决定花时间研究这个方向的？当时你脑子里在想什么，让你觉得这是个有趣的方向？这一切是如何发展的？这个问题很有深度，但我们循序渐进地来。

Okay. So, now I'll do you as many sentences as you want. Tell us a little about the history of reinforcement learning, how you came to decide to spend your time looking in this direction. What were the things going on at the time in your head that made you think this was an interesting direction to look, how did that all develop? Now, there's a lot of depth to this question, but let's just approach it incrementally.

当机器学习最初被研究时，比如在 50 年代，很自然地会想到从经验中学习，做与强化学习相关的事情。Andy Barto 去年在强化学习会议上做了一个精彩的演讲。他的观点是，一开始机器学习就是强化学习。然后，由于他们试图通过试错和奖惩来学习，随着形式化，它演变成了监督学习。与奖励的联系在某种程度上丢失了。所以在整个 60 年代，感知机、模式识别、监督学习这些都成了主导。Tom，和你聊这个很有意思，因为你看到了另一面。强化学习方法和监督学习方法之间几乎存在一种竞争：监督学习有指导和示例，而强化学习中智能体必须自己弄清楚该做什么，并可能获得奖励或惩罚。这经历了起起落落，机器学习领域一直由更直接的监督方法主导。正如我所说，一开始奖惩是其中的重要部分，但随着事情变得更清晰、定义更明确，学习问题变成了模式识别和监督学习。这个奇怪的家伙 Harry Klopf 比别人更早认识到这一点，他写了一些报告，最终写了一本书，说有些东西丢失了。Andy Barto 和我接过了他的工作，最终意识到他是对的，有些东西被遗漏了。从心理学的角度来看，这很明显，因为我一直在研究动物如何学习，动物确实以两种方式学习：监督方式和强化方式。所以我们接过了这一点，把它变成了一个明确定义的领域。那是什么时候？那是在 80 年代，最终我们在 98 年写了一本书，使它成为机器学习的一个清晰子领域。关键问题是为什么强化学习有潜力强大。它强大是因为它真正从经验中学习。它从动物或人类会获得的普通数据中学习，不需要像监督学习那样准备特殊数据。所以，你越能从发生的事情中学习，你就越强大，因为你能更普遍地应用。‘从经验中学习’这个短语越来越引起我的共鸣。那就是从智能体生活中发生的普通、未经准备的事情中学习。最初是 Alan Turing。现在我获得了图灵奖，有人告诉我 Alan Turing 在 1947 年说过，当时还没有 AI 领域，甚至被认为是 AI 的第一次公开演讲，在伦敦学会的一次讲座中，Turing 谈到了从奖励和惩罚中学习，并说了这样一句话：‘我们真正想要的是一个能从经验中学习的机器。’

When machine learning was first being investigated, like in the 50s, it was natural to think about learning from experience and to do things that were related to reinforcement learning. Andy Barto gave a great lecture at the reinforcement learning conference last year on this. His point was that in the very beginning, machine learning was reinforcement learning. Then because they were trying to learn from trial and error and from rewards and penalties, and then it evolved as it became formalized into supervised learning. The thread connection to reward was sort of lost. So throughout the 60s, perceptron and pattern recognition, supervised learning, all these things became dominant. It's interesting to talk to you about this, Tom, because you have seen the other side. There's sort of almost a competition between the reinforcement learning approach and the supervised learning approach, where you have supervised learning with instruction and examples, and reinforcement learning where the agent itself has to figure out what to do and may get rewards and penalties. It has been an ebb and flow, and the field of machine learning has always been dominated by the more straightforward supervised approach. As I mentioned, at the very beginning, rewards and penalties were very much part of it, but then the focus as things became more clear and better defined, the learning problem became pattern recognition and supervised learning. This fellow, the strange fellow Harry Klopf, recognized this more than other people and wrote some reports and ultimately a book saying that something had been lost. Andy Barto and I picked up on his work and eventually realized that he was right, that something had been left out. In some sense, it was obvious from my point of view of psychology where I'd been studying how animals learn, and animals learn really in both ways, both in a supervised way and a reinforcement way. So we picked up on that and made it into a well-defined area. When was that? That would have been in the 80s, and finally we wrote a book on it in '98, so that it became a clear subfield of machine learning. The key thing is why reinforcement learning is potentially powerful. It's powerful because it's really learning from experience. It's learning from the normal data that an animal or person would get, and it doesn't require prepared special data like you do in supervised learning. So the more you're able to just learn from what happens, the more powerful you could be because the more generally applicable you could be. This phrase 'learning from experience' has been resonating with me more and more. That learning from ordinary unprepared things that happen in the life of an agent. The original thing is Alan Turing. Now that I got the Turing Award, someone pointed out to me that Alan Turing said in 1947, before there was a field of AI, and it is even claimed to be the first public presentation on AI, in a lecture to the London Society, Turing talks about learning from rewards and penalties and has this line: 'What we really want is a machine that can learn from experience.'

是的，很好。所以，当我想到强化学习时，与监督学习相比，一个关键区别是监督学习中你有输入 X 和输出标签 Y，有人告诉你至少在一些 XY 对中，你要找出一般的映射是什么。

Yeah. Pretty good. So, when I think of reinforcement learning, in contrast to supervised learning, one of the key differences is that supervised learning, you have an input X and an output label Y and somebody tells you at least in supervised learning, here's some XY pairs, you figure out what the mapping in general is.

但你说从经验中学习时，经验通常涉及你执行的一系列重要行动。猫逃出箱子最终获得食物得到奖励。所以，从一系列行动中获得延迟奖励的概念似乎非常基础，我记得看到你关于时序差分学习的工作，特别是为了解决这个问题，我想，‘哦，这真的很重要。’你知道，当时你看不同的论文，你会想，‘好吧，这很有趣。这很有趣。’但偶尔你会看到一篇，你说，‘哦，这真的很重要，因为它可能改变我们的思考方式。’你能谈谈吗？

But when you say learning from experience, then experience often involves a substantial sequence of things that you do. The cat getting to escape from the box and eventually getting to the food to get the reward. And so, that notion of distant reward from a sequence of actions that you're doing seemed pretty fundamental and your work I remember seeing the work that you were doing on temporal difference learning in particular to address that and I thought, "Oh, this really matters." You know, at the time you see different papers, you think, "Okay, that's interesting. That's interesting." But once in a while you see one you say, "Oh, this really matters because it could change the way we think about it." Can you just talk a little bit of?

嗯，时序差分学习是我做过的最好的事情。它再次源于对动物学习的思考，因为动物会这样做，而且很明显……是的，想想动物。即使动物在做诸如按压杠杆获取食物之类的事情，动物也会分几个部分学习。首先，它学会，哦，就在食物到来之前，我听到食物输送系统的机械装置发出小噪音。事实证明，奖励的有效时间是在那些声音首次出现的时候。所以，如果有任何线索表明好事即将发生，那么那个线索，那个提示，就本身变得有奖励性，这被称为次级强化，是心理学中一个非常成熟的概念。于是你开始思考这个，最终导致了时序差分学习。时序差分学习意味着你警觉于预测随时间的变化。所以这里我们预测奖励，然后有效奖励是当你意识到奖励即将到来时。那就是一个时序差分。你之前认为它不会来，现在你认为它会来。所以有一个增量，意味着你应该更早地认为它会来。所以变化，时序差分仅仅意味着变化。差分就是随时间的变化，这就是时序差分的含义。所以那就是你预测的误差。

Well, temporal difference learning is the best thing I ever did. It comes again from thinking about animal learning because animals do this and it becomes clear that... Yeah, think about an animal. Even if an animal is doing something like pressing a bar to get food, the animal learns it in a couple parts. First, it learns that oh, when just before the food arrives, I hear the machinery of the food delivery system making little noises. And it turns out that the effective time of reward is when those sounds first happen. So yeah, if there is any clue that something good is happening, then that clue, that cue, I should say, becomes rewarding itself and it's called secondary reinforcement and it's a very well-developed thing in psychology. And so you just start thinking about that and eventually it leads to temporal difference learning. Temporal difference learning just means that you're alert to the change over time of your prediction. So here we're predicting reward and then the effective reward is when you realize that the reward is coming. So that's a temporal difference. You didn't think it was coming and now you think it is coming. So there is an increment there that means you should have thought it was coming even earlier. And so the change, temporal difference just means change. Difference is the change over time, that is what temporal difference means. So that's what the error is for your prediction.

是的，我认为那是一个伟大的贡献，也是研究进展的重要第一步。那么更广泛地说，你认为强化学习在实践中取得了哪些成功？比如，当你在这个领域工作时，第一次让你觉得‘哦，成功了。我们终于有证据表明这不仅仅是一个好理论，而是可以实用的东西’是什么时候？

Yeah, I think that was a great contribution and a great first step of research advance. So a little more broadly then, what do you think are the successes of reinforcement learning in practice? Like what do you think was as you were working in this area, what was the first time where you thought, "Oh, that's a success. We finally have evidence that this is not just a nice theory, but something that could be practical?"

嗯，第一个应用成功是 TD-Gammon。当 Gerry Tesauro 使用 TD-Lambda 算法达到世界冠军级别的双陆棋水平时。在 1990 年代，1992 年论文发表。然后 AlphaGo 在更大规模上做到了，AlphaZero 真正完善了它。当 AlphaGo 击败世界最顶尖的围棋选手，而 AlphaZero 在没有监督学习的情况下做到了。第一个系统从人类对局示例中学习，而 AlphaZero 完全通过自我对弈学习。这些都是令人信服的例子。还有 GT Sophie 也不错。还有很多。但有趣的是，你知道，这些都没有我的参与。你知道，我一生从未做过任何有用的事。我的意思是，我从未做过直接有用的事。我总是想，那些做理论之类的人成为科学家，他们说，‘好了，现在是我回馈的时候了。’而我意识到我从未回馈过。我从未做过任何真正有用的事。

Well, the first application success was TD-Gammon. When Gerry Tesauro used the TD-Lambda algorithm to beat to play world champion level backgammon. In the 1990s. In 1992 when the paper came out. And then that was done again on a larger scale with AlphaGo and then really perfected with AlphaZero. When AlphaGo became the world beat the world's best Go players and AlphaZero did that without any supervised learning. Like the first system that had learned from human examples of good play and then AlphaZero was able to learn entirely by self-play. Those are the convincing ones. Well, also GT Sophie was good. And there were lots of things. It's funny though, you know, none of these things involved me. You know, I never did I've never done anything useful in my whole life. I just in the sense I've never done anything that's directly useful. I always think of, you know, people who do all this theory or something and become scientists and they say, "Well, now it's my time to give back." And I realize I've never given back. I've never done anything that's actually useful.

我想你的贡献在这里毋庸置疑，所以不用担心……

And I suspect your contributions are not in question here, so I wouldn't worry about...

我不是在为此道歉。我认为很好的是，我们可以庆祝人们可以做纯粹基础性的工作，仍然值得认可。那很棒。我同意你的看法。那是一个美好的世界。

I'm not apologizing for it. I think it's good that we can celebrate that people can do purely fundamental things and still be deserving of recognition. That's great. I'm with you on that. That's a great world to live in.

好的，那么你显然几十年来一直是更广泛的机器学习社区的一部分，你见证了这些年的巨大变化。多年来，这个领域让你最惊讶的是什么？

Okay, so then you've been obviously part of the machine learning broader community for decades and you've seen tremendous change during those decades. What are some of the biggest surprises in the field to you over the years?

嗯，大型语言模型是一个大现象。它有时是最终的顶点，联结主义方法对符号主义方法的胜利，你知道，因为经典的符号主义是什么？是语言。现在我们发现，哦，神经网络可以极其出色地处理语言，而且再次超越了所有符号主义理解语言的努力。所以这很令人惊讶。但这似乎是联结主义者的全面胜利，我喜欢称他们为联结主义者。我不喜欢称这些为神经网络，因为它们不是神经网络。神经网络是神经的，它们在我们的头脑里，不在我们的计算机里。你知道，至少请为我称它们为人工神经网络。所以在过去，我们称它们为联结主义系统，我记得我特意决定我们可以称它们为神经网络，也可以称它们为联结主义。正确的词是联结主义，因为它们充满连接，是统计性的，是一个网络，但不是神经的。拜托，它们根本不是神经的。所以无论如何，回答你的问题，我认为是我们能用非符号方法多么彻底地处理语言。

Well, the large language models is a big phenomenon. It sometimes it's the final culmination, the victory of connectionist methods over symbolic methods, you know, because what is the classic symbolic thing? It's language. And now we find that oh, the neural networks can do language extraordinarily well and again better than all the symbolic efforts to understand language. So that is surprising. But it seems like a total victory for the connectionists, which I like to call them connectionists. I don't like calling these neural networks because they're not neural networks. Neural networks are networks that are neural and they're in our heads and they're not in our computers. You know, at least please say for me artificial neural networks. So in the old days we called them connectionist systems and I remember deliberately deciding we could call them neural networks, we could call them connectionist. The right word is connectionist because they're full of connections and they're statistical and they're a network but they're not neural. Come on. They're not neural at all. So anyway, your answer your question is I think how thoroughly we can do language with non-symbolic methods.

我有一个相关的问题，那就是自然语言作为表示法对逻辑形式主义的胜利，我认为这与你的话题密切相关。但多年来，AI 和机器学习一直认为信息、知识的符号逻辑表示是正确方向。而 LLM 成功的一个有趣结果是，它们实际证明了自然语言本身作为一种表示法是多么有效，相比之下逻辑形式主义的范围更窄。这么说吧。我想我无法完全区分这两个问题，但你提到了联结主义系统的胜利。这似乎与那个是相辅相成的。只是一个想法。

I have a related one which is that the victory of natural language as a representation over logical formalisms, which I think is closely related to what you're talking about. But for many years AI and machine learning along with it went with the idea that symbolic logical representations of information, of knowledge would be the way to go. And the interesting one interesting fallout of the success of LLMs is they're a working demonstration of just how effective natural language itself is as a representation in contrast to logical formalisms that have smaller breadth of scope. To put it that way. I think that's I can't quite tease apart those two issues, but you're bringing up the victory of connectionist systems. And it kind of seems to go hand in hand with that. Just a thought.

是的，我认为它们确实如此。

Yeah, I think they do.

让我提另一件让我惊讶的事：监督学习并没有像我预期的那样取得进展。我们从早期就知道，形成合适的表征至关重要。人们学习快是因为他们有好的表征——他们知道该把功劳归于什么，该做什么改变。这一点一直说得很清楚：我们需要学习表征。但我们从未做到。深度学习并没有学习表征。我们曾以为反向传播能做到，但事实并非如此。四十年过去了，我们仍然不知道如何学习好的表征，也不知道如何持续地快速学习。如今我们甚至不再谈论它，因为我们做不到。

Let me bring up another thing that has surprised me: supervised learning hasn't progressed as far as I expected. We knew from the early days that forming appropriate representations is crucial. People learn fast because they have good representations—they know what to assign credit to and what changes to make. This was always clearly stated: we need to learn representations. But we never did. Deep learning doesn't learn representations. We thought backpropagation would enable that, but it hasn't played out. Forty years later, we still don't know how to learn good representations or learn fast in a continual way. Nowadays we don't even talk about it because we can't do it.

这让我有点惊讶。你说神经网络不学习表征，是什么意思？比如，一个 12 层的图像网络，倒数第二层的表征能让线性分类器工作。这难道不是表征学习吗？

That surprises me. When you say neural networks don't learn representations, what do you mean? For example, in a 12-layer network for images, the penultimate layer has a representation that allows a linear classifier to work. Isn't that representation learning?

网络找到了足以捕捉训练集的表征，但并非适合学习的表征。如果得到新数据，你会灾难性地遗忘旧知识，并且无法学习新东西。反向传播不断调整权重直到训练集正确，但它找到的不是能良好变化的表征。过去的目标是找到一种表征，使得新例子能归因于正确的维度，而不是无关的维度。深度学习完全做不到这一点。

Networks find a representation sufficient to capture the training set, but not one good for learning. If you get new data, you catastrophically forget old things and cannot learn new things. Backpropagation changes weights until the training set is correct, but it doesn't find something that changes well. In the old days, the goal was to find a representation so that new examples are attributed to the right dimension, not extraneous ones. Deep learning does not do that at all.

是的，我记得早期关于如何以某种方式看棋盘使问题变得简单的工作。那种短期学习正确表征的问题确实是一个开放问题。

Yes, I remember early work on seeing a chessboard in a way that makes questions easy. That kind of short-term learning of the right representation is indeed an open problem.

说到开放问题，你对当前领域的状况怎么看？有没有好的博士论文题目？

Speaking of open problems, what's your picture of the state of the art? Are there good PhD thesis topics?

第一是如何学习适合学习的表征。我让我的学生在这方面工作。第二是用学习到的模型进行规划。我们有像 AlphaGo 和象棋程序这样的好规划系统，但没有一个能用于学习到的模型。我们如何在世界模型中表示知识，以便能很好地规划？这是两个最重要的问题。

Number one is how to learn representations that are good for learning. I have my students working on that. Number two is planning with a learned model. We have good planning systems like AlphaGo and chess programs, but none work with a learned model. How can we represent knowledge in a world model so that we can plan well? Those are the two most important ones.

这些非常基础。每个问题可能都能产生多个博士论文。我赞成解决那些人们长期失败了的难题。

Those are very fundamental. There are probably multiple PhD theses on each. I'm in favor of tackling hard problems that people have failed at for a long time.

我认为博士生应该研究这些问题。科学家的标志是识别我们不知道的东西并去研究它，而不是强调我们知道什么来做演示或创业。我很着急——我想理解学习和心智如何运作，我们还有空白要填补。

I think PhD students should work on those. It's a mark of a scientist to identify what we don't know and work on it, rather than emphasizing what we do know for demonstrations or startups. I'm impatient—I want to understand how learning and the mind work, and we have gaps to fill.

一个令人惊讶的是，尽管人工智能已经存在了 50 年，但它与神经科学和动物智能的联系很少。你的强化学习工作是一个光辉的例子，它同时受到心理学和计算的启发。你怎么看？

One surprise is that AI has little connection to neuroscience and animal intelligence, despite being around for 50 years. Your reinforcement learning work is a shining example of joint motivation from psychology and computation. What do you think?

对我来说，它们是一体的。我总是在谈论一种心智科学，它既不是纯粹的工程学也不是自然科学。心理学是研究心智的自然科学。我们想理解心智现象，而不仅仅是赚钱或理解自然。自然心智可以给我们启发，我们也会运用所学。在心理学和强化学习中，它们是完全相连的——对我来说一直如此。

For me, they are one. I always talk about a science of mind that is neither pure engineering nor natural science. Psychology is a natural science studying minds. We want to understand the phenomenon of mind, not just make money or understand nature. Natural minds can inform us, and we use what we learn. In psychology and reinforcement learning, they are fully connected—they always have been for me.

你说得对，机器学习领域整体上只是偶尔触及神经科学。而且即使触及了，也不总是有用的。这是两个不同的领域。我曾经说过，跨学科的风险在于你会掉进学科之间的缝隙，最终变得没有学科归属。

You're right, the field of machine learning as a whole has only touched on neuroscience sometimes. And when it has, it hasn't always been useful. It's a separate field. I used to say the danger of being interdisciplinary is you fall between the disciplines and you end up with no discipline.

最后再问一个问题，我知道时间不多了。展望未来，虽然没人能完美预测，但人工智能、机器学习、计算机科学现在都在经历巨大变革。你猜测最终会是什么样子？这一切会如何演变？你有什么想法吗？或者给现在开始从事这个领域、可能会感到紧张的学生一些建议？

Let me maybe a final question here. I know we're short on time. Looking forward, and granted that nobody can predict the future perfectly, AI, machine learning, computer science are all going through tremendous change right now. What's your guess about what comes out on the other side? And how all this evolves? Do you have any thoughts about that? Or advice to students who are starting a career in this area now and who might be nervous about it?

嗯，我更倾向于认为这是一个重要科学的正常进展。我们正在理解心智如何运作，从而也稍微理解我们自己的心智如何运作。这显然是一件重要的事情。人类一直想知道它们如何运作，它们是谁，它们的角色是什么，它们是什么。所以这是一个基础科学问题，一个基础人文学科问题。我认为它会继续下去，它会相互影响。现在，我们这个领域的奇怪之处在于，它是一个经济上极其重要的产业。确实有万亿美元的产业。所以在尝试做科学的同时，我们还有这股资金、金钱和经济影响的飓风相伴。这扭曲了一切，把它从纯粹的科学活动改变了。我们都熟悉这一点。它扭曲了事物，充满了炒作。甚至术语都被行业选择的词汇扭曲了，我们不得不使用它们。他们称某物为推理，你可以试图坚持不用这个词，因为它显然不应该被称为推理、注意力或推理。你可以坚持，但你很可能不会被理解。所以我们的科学旁边有这股海啸，它真的扭曲了一切。那么会发生什么，它会如何发展？我不知道。海啸可能会越来越大，或者炒作会导致失望，出现冷却期和另一个冬天，这甚至可能是好事。谁知道呢？但最有可能的是，它将继续在经济上可行，并且旁边会继续有一个巨大的产业。还会有一些科学会持续下去。为什么不呢？因为会有大量资金。所以基础科学也应该有资金。然后我们会成功。我们会在某种程度上理解学习如何运作，智能如何运作，以及我们的心智如何运作。我们将制造出能与人类匹敌并超越人类的人工智能系统。我喜欢认为这一切都是好的，我们只是有了改进的技术和做事的能力，世界上智能越多，世界就会越好。我认为这将是底线。尽管现在对智能有巨大的恐惧，我认为这是不必要的，因为它适得其反，因为恐惧不是应对变化的好态度。但另一方面，恐惧让人们关注。很容易忽视，'哦，那些科学家在做些什么，我不需要关注。'但我认为公众关注 AI 正在发生的变化以及我们对自己更好的、不同的理解是件好事。所以我希望这可以在没有恐惧的情况下实现，但也许我们需要恐惧的元素来让人们关注。我认为很可能会有类似对 AI 的歧视。我们会说，'哦，自然智能有些特别，我们想要优越。'因为人们在面对变化时总是想要优越。所以可能会有对 AI 的歧视，我们会说它们并不真正感受痛苦，它们并不真正有目标，让它们服从我们是没问题的，就像现在的机器一样。可能会有歧视，然后这必须被克服。最终我们会决定，是的，机器就像我们自己一样有心智，我们应该接受它们并停止歧视它们。这将是事情演变的方式，也许在更长期的未来。

Well, I prefer the model that is just normal progression of an important science. We're understanding how minds can work and thus a bit about how our own minds work. This is obviously an important thing. Humanity has always wondered about how they work and who they are, what their role is and what they are. So this is a basic science question, a basic humanities question. I think it will continue. It will interact. Now, the strange thing about our field is that it's an enormously economically important industry. There are literally trillion-dollar industries. So alongside trying to do science, we have this hurricane of funding and money and economic impact alongside. And that warps everything, changes it from being a pure science activity. We're all familiar with that. And that distorts things. It has lots of hype. Even the terminology is so warped by whatever terms the industry has chosen to use, we can't help but use it. They call something inference, then you could try to be a holdout and stop calling it inference because it clearly shouldn't be called inference or attention or reasoning. You can try to hold out, but you're just going to be not understood, probably. So we have this tsunami alongside our science and that really warps everything. Now, what will happen and how will this develop? I don't know. The tsunami may continue getting bigger and bigger, or the hype may lead to disappointment and there'll be a cooling period and another winter, which might even be good. Who knows? But most likely it will continue being economically viable and there'll continue to be a huge industry alongside. And there'll be some science that will persist. Why not? Because there'll be lots of funding. So there should be funding for fundamental science as well. And then we'll succeed. We will understand how learning can work and how intelligence works and how our minds work to some extent. And we will make artificial intelligence systems that will rival humans and then surpass humans. I like to think this is all good, that we just have improved technology and improved ability to do things, and the more intelligence there is in the world, the better the world will be. I think that will be the bottom line. Even though right now there's tremendous fear about intelligence, which I think is unnecessary in the sense that it's counterproductive because fear is not a good attitude to deal with change. But on the other hand, the fear causes people to pay attention. It's so easy to ignore, 'Oh those scientists are doing something, I don't need to pay attention.' But I think it's good for the public to be paying attention to the changes happening in AI and the better understanding, the different understanding we're getting of ourselves. So I wish it could be done without fear, but maybe we need the element of fear to get people to pay attention. I think it's quite likely that there's going to be something like discrimination against AIs. We will say, 'Oh, there's something special about natural intelligence and we want to be superior.' Because people always want to be superior in the face of change. So there could be something like discrimination against AIs, and we will say they don't really feel pain and they don't really have goals, and it's okay for us to have them be subservient to us, the way machines are now. And there could be something like discrimination, and then that will have to be overcome. Eventually we'll decide, yeah, machines are minds just like ourselves, and we should accept them and stop discriminating against them. That will be how things evolve, maybe over the longer-term future.

谢谢你，Rich。Rich Sutton，非常感谢你的时间和想法。有很多值得思考的地方，我们不仅感谢你迄今为止的贡献，也感谢你持续的工作、思考和引导社区的努力。

Thank you, Rich. Rich Sutton, thank you so much for your time, your thoughts. Plenty to think about there and we really appreciate not only your contribution so far, but your ongoing work and thoughts and efforts to steer the community.

好的，这是我的荣幸。

Okay, it was a pleasure.