强化学习：从试错到人工智能的基石

这里是 ACM Bitecast，一个来自美国计算机协会的播客系列，该协会是全球最大的教育和科学计算学会。我们与处于计算研究和实践交叉点的研究人员、从业者和创新者对话。他们分享经验、所学到的教训以及对计算未来的愿景。我是主持人 Rashmi Moan。在一个机器不仅从数据中学习，还从经验中学习的世界里，两位科学家悄然撰写了实现这一切的蓝图。他们不仅发明了算法，还探索了科学和哲学中最古老的问题之一：学习是如何发生的？他们的工作为强化学习奠定了基础——即智能的核心源于试错和奖励。今天，我们与人工智能史上两位最具思想深度的头脑对话，他们的思想塑造了从 AlphaGo 到 ChatGPT 的一切。请欢迎 2024 年图灵奖得主，无需更多介绍：Richard Sutton 和 Andrew Barto。欢迎来到 ACM Bitecast。

This is ACM Bitecast, a podcast series from the Association for Computing Machinery, the world's largest education and scientific computing society. We talk to researchers, practitioners, and innovators who are at the intersection of computing research and practice. They share their experiences, the lessons they've learned, and their visions for the future of computing. I am your host, Rashmi Moan. In a world where machines are beginning to learn, not just from data, but from experience, two scientists quietly wrote the playbook that made it all possible. They didn't just invent algorithms. They explored one of the oldest questions in both science and philosophy. How does learning happen? Their work laid the foundation for reinforcement learning. The idea that intelligence at its core emerges from trial, error, and reward. Sitting with two of the most thoughtful minds in AI history whose ideas have shaped AlphaGo to ChatGPT. Please help me welcome the 2024 Turing Award winners who need no other introduction. Richard Sutton and Andrew Barto. Welcome to ACM Bitecast.

非常感谢。

Thank you very much.

谢谢。

Thank you.

太好了。我想从一个简单的问题开始，这个问题我问过所有嘉宾。请介绍一下自己，谈谈你目前的工作，并分享是什么吸引你进入这个领域。

Wonderful. So, I'd like to lead with a simple question that I ask all my guests. If you could please introduce yourself and talk about what you currently do as well as give us some insight into what drew you into this field of work.

我是 Richard Sutton。我一直从事强化学习的研究，现在依然如此。我是阿尔伯塔大学的教授，也是一家名为 Keen Technologies 的初创公司的研究科学家。我仍在努力完全理解学习是如何发生的，以及学习如何在更宏大的心智架构中发挥作用。

I'm Richard Sutton. I have always done research in reinforcement learning and I still am. I'm a professor at the University of Alberta and I'm a research scientist at a startup called Keen Technologies. I'm still trying to fully understand how learning happens and how learning can play a role in the larger architecture of a mind.

很好。

Great.

也许我就先说到这儿。

Maybe I'll stop there.

我是 Andy Barto。我是马萨诸塞大学阿默斯特分校的荣誉教授。我已经退休大约 13 或 14 年了。Rich 是我的第一个博士生。我们一起工作了很多年，目前我尽量跟上一些进展。但即使是年轻人，也不可能跟上人工智能的发展速度。在过去，我还能跟上，但现在我做不到了。所以我的时间基本用来做我想做的事，因为我不需要参加教师会议，不需要写提案，我很享受退休生活。我最近做了几次演讲，标题是‘强化学习有什么有趣之处？’我选这个标题的原因是，强化学习的基本思想非常古老且是常识：从我们行为的结果中学习，无论好坏。那么问题来了，为什么它如今在人工智能中扮演如此重要的角色？为什么 Rich 和我获得了这个奖？因为这是一个非常古老的课题，我想我们算是重新发现了它。我不确定正确的词是什么，但我们创造了一种计算性的思考方式。我认为这就是本质。

I'm Andy Barto. I am a professor emeritus at the University of Massachusetts in Amherst. I've been retired for about 13 or 14 years. Rich was my first PhD student. We worked together for a number of years and I currently try to keep up a little bit with what's going on. But even for young people, it's impossible to keep up with the pace of development in artificial intelligence. In the old days, I was current in what was going on, but I have not been able to do that. So I spend my time doing whatever I want basically because I don't have to go to faculty meetings. I don't have to write proposals and I'm enjoying being retired. So I have given a few talks lately and the title has been 'What is so interesting about reinforcement learning?' and the reason I came up with that title is that the basic idea of reinforcement learning is very, very old and common sense: learning from the consequences of our behavior, of our actions, good or bad, and so on. So the question is why is it playing such a role in AI these days and why did Rich and I get this award because this is a very old subject and I guess we sort of rediscovered it. I'm not quite sure what the correct word would be but we've created a way of thinking about it computationally. I think that's kind of the essence of it.

谢谢。这是一个精彩的总结，你说做自己想做的事，听起来是个很棒的状态。我们都渴望达到那种境界。Andy，在我回到 Rich 之前，我再问你一个问题。回想最初，是什么吸引你进入计算机科学？你当时想学什么？你是偶然发现还是有意选择了计算机科学？

Thank you. That's a wonderful summary and sounds like when you say that you do whatever you want, it sounds like a great place to be. We're all aspiring to get there. Andy, let me ask you one more question before I go back to Rich. I mean, if you go back all the way, what drew you into computer science to start with when you were a student, what was it that you were aspiring to study and how did you stumble upon or was it a very conscious decision to pick computer science?

嗯，我本科开始时是这样。我父亲是机械工程师，母亲是艺术家和注册护士，所以我真的很想做结合艺术和工程的事情。于是我学了造船与海洋工程，但只坚持了一年，就失去了兴趣，转到了数学。所以我基本上是作为工程师被培养的，但我喜欢数学，也相当擅长，所以我在密歇根大学的本科专业是数学。在这个过程中，我偶然发现了一些真正让我感兴趣的东西，特别是 50 年代提出的控制论思想，以及系统理论：将社会或组织作为系统来研究，尤其是数学系统。所以我非常受从数学角度看待系统的激励，这自然让我对现在所谓的计算神经科学产生了兴趣，因为人们创建的模型是机制和系统。我对此非常感兴趣。我的硕士和博士学位都是计算机科学，但我是一个非典型的计算机科学家。我只修了两门编程课。我从来不喜欢编程，编码后总是睡不着。但我对数学和计算在生物学方向的应用更感兴趣。我对计算机科学家做的很多典型课题不感兴趣。所以基本上，我思考问题的方式有点像老派工程师。我非常幸运，一路上有这么多优秀的学生，强化学习这个领域之所以能发展起来，是因为许多才华横溢的学生和同事的努力。

Well, I started out as an undergraduate. My father was a mechanical engineer and my mother was an artist and a registered nurse, so I really wanted to do something that combined art and engineering. So I went into naval architecture and marine engineering and I lasted a year in that and became less interested in it and switched to mathematics. So I was basically raised to be an engineer but I liked math and was reasonably good at it, so my undergraduate degree from University of Michigan was in math. Along the way I stumbled on things that really interested me, in particular the idea of cybernetics which was introduced back in the 50s and the idea of system theory: studying societies or organizations as systems, in particular mathematical systems. So I really was motivated by looking at systems from a mathematical point of view and of course that led to being interested in what we now call computational neuroscience because the models that people were creating were mechanisms and systems. So I got very interested in that. My master's degree and PhD are in computer science, but I was atypical as a computer scientist. I just took two courses in programming. I never liked to program. I could never sleep after coding. But I was more interested in more biologically directed applications of math and computing as well. But I was not interested in a lot of the typical subjects that computer scientists were doing. So basically engineering, I'm kind of an old school engineer in the way I think about things and I've been so fortunate to have so many wonderful students along the way that this area of reinforcement learning really came into its own because of the efforts of a lot of very talented students and colleagues.

真吸引人。谢谢你的分享。但我还想回到这个话题。Rich，我问你同样的问题。是什么让你进入计算机科学？这是一个非常刻意的选择吗？

Fascinating. Thank you for sharing that. But I do want to come back to that. But Rich, I'd have the same question of you. What brought you into computer science? Was that a very conscious choice?

嗯，我想从记事起，我就对心智之谜以及它如何运作着迷，尤其是我的心智如何运作，当你成长时，内省时总是以自我为中心。你知道，我是什么？它是如何工作的？我怎么能看到并感知世界？我认为这对每个人来说都是一个明显的智力谜题，因为他们试图理解自己的世界。然后我上学了，听说了计算机，计算机常被描述为像巨型大脑。所以我想了解它们，但当我真正了解时，发现它们完全不像大脑或心智，因为它们只会做你告诉它们的事，仅此而已。我想，那永远不可能成为心智。但后来我又想，也许如果你以正确的方式编程，它可能成为心智。毕竟，心智除了是一台处理输入并产生决策的机器，还能是什么呢？它不可能是机器，但又能是什么呢？所以，它必须是真的，但又不可能。我一直被这个谜题所震撼。然后我发现了心理学和人工智能等更深入探索的领域。

Well, I think from as early as I can remember, I've been fascinated by the puzzle of the mind and how it works and how my mind works, kind of being centered on oneself when introspecting when you're growing up. You know, what am I? How does it work? How is it that I can see and get a sense of the world? This is I think an obvious intellectual puzzle for everyone as they try to understand their world. Then I went to school of course and I heard about computers and computers were often described as like giant brains. So I wanted to find out about them and then when I did find out about them they were not at all like brains or minds because they only would do exactly what you told them to do and nothing more. And that could never be a mind, I thought. But then I thought again, you know, maybe if you programmed it the right way, it could be a mind. And after all, what could a mind be but a machine that's processing its input and producing decisions? It couldn't possibly be a machine. And yet, what else could it be? So, it was like it has to be true, but it couldn't be true. And I was always struck by that puzzle. And then I found various fields that would look more deeply in psychology and in artificial intelligence.

所以上大学时，我在纠结：是当脑科学家还是 AI 研究员？也许是因为一门神经科学课程成绩不理想，我就想，算了，还是从 AI 入手吧。

And so when I went to college, I was trying to figure out, shall I be a brain scientist or shall I be an AI researcher? And I don't know, maybe I got a disappointing grade in one of my neuroscience courses. And so I decided that, oh, that wasn't any good. I'm just going to try to do it from the AI side.

这是个好决定。显然它带来了充实而成功的职业生涯。我很好奇，我知道 Rich 你是 Andy 的学生，你们当然合作过。听起来当时你们都决定专注于强化学习，我不知道这个决定是怎么做出的。是不是在全世界都不太重视它的时候，你们选择了它？是什么让你们一直深耕这个领域？

That's a good decision. Obviously it led to a very fulfilling and successful career. I'm curious I know that Rich you were a student of Andy's, you collaborated together of course, and it sounds like when you both decided to focus on reinforcement learning, I don't know how that decision came about. Was it something that you picked when the rest of the world was kind of not giving it the attention that it deserved? What made you sort of stay embedded in that field?

这是个好问题，非常贴切。我觉得我们做了 Andy 说的：重新发现这个主题。我们得到了一些帮助，这确实是我们一起做的。当我们都去了马萨诸塞大学——Andy 比我早一年左右——我们开始研究这个谜题。有些东西缺失了，我们最终称之为强化学习。Andy，你来解释一下那段经历吧？

It's a lovely question. It's very appropriate. I feel that we did what Andy said: rediscover the topic. We had some help and it's really what we did together. When we both went to the University of Massachusetts—Andy went a good year or so before me—when we arrived, we engaged with this puzzle. Something was missing and we eventually came to call it reinforcement learning. Why don't you explain what that journey was like, Andy?

好的。1977 年，我在马萨诸塞大学做博士后，参与了一个项目，目标是评估一个非常非正统的观点：神经元——大脑的基本计算单元——本身就是有目标导向的有机体或享乐主义者。例如，它们会根据自身活动的后果来学习。就像微小的动物，为了获得奖励或避免惩罚而行动。我们的项目是：这个想法值得研究吗？这是别人提出的想法。挑战在于：从科学或技术角度看，这个疯狂的想法是否值得研究？我们深入研究了它，那是一段美妙的时光。我不需要教书，不需要参加教师会议。我们得以回顾这些想法的历史，并得出结论——这并不意外——它很有趣，而且确实被忽视了。由于很多原因，它在 AI 中被忽视了。当时神经网络的概念被认为是死胡同，甚至被唾弃。从神经科学的角度看，大脑突触或学习的关键是 Hebb 关于突触如何改变效能或权重的假说。我们提出的观点是，神经元远比这复杂。它们实际上有局部记忆。它们能记住自己做了什么，直到通过各种反馈回路得到后果，这可能需要几秒钟，而不是瞬间。所以那是一段美妙的探索时期，最终导致了这些想法的数学和计算实现，然后很多非常聪明的人参与其中。多年来它不断发展，现在它竟然在当前的 AI 中发挥作用，这让我感到惊讶。但我主要被驱动是因为没有其他人研究这个。所以我感觉有点像先驱，尽管这个主题已经被热烈讨论了很多年，但它并不是当时的研究热点。我受到激励是因为这不是一个常见的研究方向。实际上，随着我退休，我的动力减弱了，因为现在有太多人在做，所以不像早期那么令人兴奋了。

Okay. Well, I began as a postdoc in 1977 at the University of Massachusetts and I was hired for a project whose objective was to evaluate and assess this very unorthodox idea that neurons, the basic computational unit of brains, were themselves goal-directed organisms or hedonists. For example, that they actually would learn on the basis of the consequences of their activity. So as if they were little microscopic animals that behaved in order to achieve reward or to avoid penalty. Our project was: is this idea worth studying? It was an idea put forward by someone else. The challenge was: is this crazy idea worth studying either from a scientific point of view or from a technological point of view? We delved into it and it was a wonderful period. I didn't have to teach. I didn't have to go to faculty meetings. We really were able to look at the history of these ideas and we concluded—I guess it's not surprising—that it is interesting and it really has been neglected. For a lot of reasons it was neglected in AI. The idea of a neural network back then was just not—you know, it was considered to be a dead end and reviled actually. And from a neuroscience point of view, a key idea for synaptic or learning in the brain was Hebb's hypothesis about how synapses change their efficacies or weights. What we were proposing was that neurons are much more complicated than that. They actually have local memory. They can remember what they did until some consequences come back through various feedback loops, and that could take several seconds perhaps, not instantaneous. So it was a wonderful period of exploration and eventually led to mathematical and computational implementations of these ideas and then lots of very bright people got involved in it. It evolved over the years and I'm amazed now that it's actually playing a role in current AI. But I was mostly driven because nobody else was studying this. So I felt kind of like a pioneer even though the subject had been discussed intensely over many years, but it really wasn't a current subject of exploration. I was motivated by the fact that it was not a common direction of study. Actually, as I've retired, I'm less motivated because now so many other people are doing it, so it's not as exciting as it had been in the early days.

这真是深刻的见解，Andy。也许我要问 Rich 一个后续问题。实际上我有两个后续问题。一个是，听起来这项工作的本质也是跨学科的，需要与神经科学、生物学或人类心智研究者合作。那么当时你有这些机会吗？你是否主动寻求这些机会来推进研究？第二个问题是关于研究本身。你追求一个主题是因为‘嘿，没人在看这个，所以我想研究它’。这是研究者思考问题的基本方式吗？还是通常有一些应用前景或待解决的问题？

That is an incredible insight, Andy. And maybe I'll ask Rich this follow-up question. I have two follow-up questions actually. One is it sounds like the nature of the work was also very interdisciplinary, requiring collaboration with folks who are studying neuroscience or biology or the human mind. So did you get those opportunities? Did you seek out those opportunities at that time to further your research? And the second question is around research itself. The fact that you pursue a topic that is 'hey nobody else is looking at this so I want to pursue it.' Is that the fundamental way that a researcher thinks about a problem, or is it usually with some sort of line of sight into how this can be applied or used or there's a problem to be solved?

好，我来回答。我认为这也许是研究者应该思考的方式，但并不是我们通常的做法。我们通常心里有特定的东西，而且通常在自己的学科内。但首先，当时我们尝试时，这个想法来自一个叫 Harry Klopp 的人，他把想法带给我们，让我们开始研究。我们试图理解它，试图找出它是否已经被探索过，最终我们决定它之前没有被探索过。要做出这样的判断，你不能只进入一个领域。你必须进入所有可能探索过它的领域。所以从主题和我们的结论来看，它没有被研究过，我们必须研究许多不同的领域。许多不同的领域包括控制理论、运筹学、当然还有 AI、控制论、任何关于序列决策的数学。现在我们简要地说它没有被很好地探索过。当然，太阳底下没有新鲜事，正如 Andy 教我的，并且一直告诉我的。太阳底下没有新鲜事；总有先驱。所以你必须找到先驱，把他们放在合适的位置，然后指出哪些研究不足。所以这是非常跨学科的。特别是，我引入了心理学元素。心理学，动物学习理论家可能比任何人都更多地思考学习是如何工作的。在我学习的时候它已经过时了，但它非常相关，现在仍然非常相关。如果你观察动物，很明显动物确实通过试错学习。动物和人通过试错学习，通过尝试事物，看什么最愉快、最有回报，并避免痛苦。所以我认为心理学、神经科学、各种形式的控制理论和人工智能之间有非常强的相互作用。

Yeah, let's take that. I think it is maybe the way a researcher should think but it's not the way we usually do it. We usually have something particular in mind and we're usually within our discipline. But first, the fact that back then when we were trying, we had this idea that came to us from this fellow Harry Klopp that brought the idea to us and set us to work on it. We were trying to make sense of it. We were trying to find out if it had already been explored and we eventually decided it hadn't been previously explored. To decide a thing like that you can't just go into one field. You have to go into all the fields that might have explored it. So it was sort of necessary just from the topic and the conclusion we were reaching that it hadn't been studied that we had to go study many different fields. Many different fields means like control theory, operations research, AI of course, cybernetics, any mathematics of sequential decision-making. Now we make the brief statement that it hadn't been explored well. Of course nothing is totally new under the sun, as Andy taught me and still tells me all the time. Nothing is totally new under the sun; there are always precursors. So you had to find the precursors and put them in the proper place and then remark and set in prominence what had been insufficiently studied. So it's very interdisciplinary. In particular, I brought the element of psychology in. Psychology, the animal learning theorists have thought probably more than anybody about how learning works. It was already out of fashion when I was studying but it was very relevant and is very relevant. It's clear if you look at animals that animals do learn by trial and error. Animals and people learn by trial and error, by trying things and seeing what is most pleasurable and most rewarding and avoiding pain. So I think it's a very strong interplay between the psychology and the neuroscience and the control theory of its various forms and the artificial intelligence.

明白了。谢谢。Andy，你有什么要补充的吗？

Got it. Thank you. Andy, did you have something to add?

让我补充一下。

Let me add to it.

嗯，我本科读的是心理学，但不是动物学习理论，而是社会心理学。我并没有深入研究，因为它不太吸引我。后来我和很多心理学家聊过，而 Rich 带着心理学背景加入了这个项目。我也不是神经科学家，不是心理学家也不是神经科学家。不知道该怎么形容自己，但我一直对机制着迷。作为机械工程师的儿子，我从小就接触马达、泵和机械装置。后来在研究生阶段，我学习了抽象机器和自动机理论。当然，神经科学也是关于机器的，所以这些对我来说都很契合。我还想补充一点，我没什么竞争心。所以进入一个没人涉足的领域，就免去了竞争的问题。当事情变得竞争激烈时，我往往会转向不同的方向。

Yeah. So I studied psychology as an undergraduate, but it was not animal learning theory. It was social psychology, I guess. And I really didn't pursue it because it didn't really grab me. You know, since then I've talked to many psychologists and Rich came to the project with a background in psychology and I'm not a neuroscientist either. I'm not a psychologist or a neuroscientist. Not sure how you would characterize me, but I've been fascinated by mechanisms, I guess. You know, as a mechanical engineer's son, I always was dealing with motors and pumps and devices, mechanical devices. And then eventually in graduate school, I studied abstract machines, automata theory. Then, of course, neuroscience is about machines. And so, you know, that all kind of fit together for me. I think also I should add that I'm not very competitive. So getting into a field that nobody else is in relieves that problem of having to compete. So, when things get competitive, I tend to go in a different direction.

在我们离开跨学科这个话题之前，这或许是个好机会来谈谈神经科学和作为计算努力的强化学习之间的巨大影响或交流。一些强化学习算法与大脑的明显运作、大脑奖励系统和计算机奖励系统之间有着惊人的相似性。它们的工作方式非常非常相似。我其实应该让 Andy 来说，因为他在这方面比我研究得更多，但这些确实是惊人的相似之处。

Before we leave the subject of the interdisciplinary things, this might be a good chance to remark on the enormous impact or interchange there's been between the neuroscience and reinforcement learning as a computational endeavor. So, it's a striking parallel between some of the reinforcement learning algorithms and the apparent workings of the brain and the brain reward systems and the computer reward systems. They work very very similarly. They really I should let Andy because he's developed this more than I have, but these are striking parallels.

嗯，是的。这对我来说是最令人兴奋的事情之一。实际上，你的论文主题开发了一种算法或一类算法，称为时序差分算法，用于计算系统。后来，一位神经科学家发现了与这种算法非常吻合的数据。这是 Wolfram Schultz 从清醒行为猴子的多巴胺神经元中记录的数据。这是一种不可思议的对应关系。当 Rich 开发这种算法时，这些数据还不存在。这是一个计算创造。结果发现，它与 Schultz 从多巴胺细胞记录的数据非常吻合。这改变了奖励系统的神经科学，这种汇合非常令人兴奋。所以那是我多年来遇到的最有动力的事情之一。

Well, yeah. So that to me is one of the most exciting things. It's actually your thesis topic developed an algorithm or a class of algorithms called temporal difference algorithms that are used in computational systems. And then later a neuroscientist found data that really fit that algorithm very closely. And this is Wolfram Schultz's recording from dopamine neurons from awake behaving monkeys. And it was an uncanny correspondence. And when Rich developed that algorithm, those data didn't exist. This was a computational creation. And it turned out that it matched quite closely to what Schultz was recording when recording from dopamine cells. And so this has changed neuroscience of reward systems and it's quite exciting that that confluence happened. So that was one of the most motivating things that I came across in all of those years.

我认为这种对应并非完全出乎意料，因为我们研究的问题——学习问题——动物也必须处理同样的问题，所以解决方案有些密切相关并非牵强。所以这一切都非常令人兴奋。

I think this correspondence is not completely to be unexpected because the problems that we were looking at, problems of learning, animals had to deal with those same problems and so the solutions, it's not completely far-fetched that the solutions were somewhat closely related. So all of that was very exciting.

在我们继续之前，我想知道你是否可以为我们的听众简单解释一下什么是时序差分学习。为什么它是一个突破？

Before we go further, I was wondering if maybe for our audience you could maybe briefly explain what is temporal difference learning in simple terms. Why was it such a breakthrough?

时序差分学习，这个名字有点花哨。时序意味着时间，时间上的差异就是变化。所以时序差分学习的思想是，如果你观察你的预测，你预测某件事会发生，你的预测如何随时间变化，然后你利用预测的变化作为误差。比如你下棋，你觉得自己会赢，但几步之后你不再认为自己会赢，你可以从那个时序差分中学习，即使棋局还没结束。也许你最终会赢，但你仍然可以在过程中学习。这是关键，能够在不等待最终结果的情况下学习。我们证明，在很多情况下，如果你不等待最终结果，而是从预测的变化中学习，你实际上能学得更好。他们在大脑中发现了一个信号，对应于预测的惊讶。如果你因为改变主意而感到惊讶，你原本以为一件事会发生，但后来觉得结果会不同，你可以从那种变化的惊讶中学习。这基本上就是时序差分的想法。

Temporal difference learning, it's a little bit of a fancy name. Temporal means time and a difference over time is just a change. And so the idea of temporal difference learning is that if you look at your predictions, you're predicting that something is going to happen and how does your prediction change over time and you then use the change in your prediction as an error. So if you're playing chess and you think you're going to win and then a little bit later a few moves later you no longer think you're going to win, you can learn from that temporal difference even though the game isn't over yet. Maybe you will end up winning, but still you can learn as you go along. And that's a key to be able to learn without waiting for the final thing. And we showed that in many cases that you can actually learn better if you don't wait for the final result, but just learn from the change in your predictions. And what they found in the brain is that there is a signal in the brain that corresponds to the surprise of the prediction. You know if you are surprised because you changed your mind, you thought one thing would happen and then a little bit later you thought it would come out differently, you can learn from the surprise of that change. That's basically the idea of temporal difference.

让我补充一点。我倾向于认为这种方法是一种纠错方法。所以从某种意义上说，它本身不是强化学习，而是强化学习系统的一个组成部分。它不是从后果（正或负）中学习，而是从错误中学习。这是一个监督过程，但它与强化学习系统协同工作。事实证明，它在许多应用中非常有效。

Let me add something. So I tend to think of that method, it's an error correction method. So in some sense it's not per se reinforcement learning, it's a component of a reinforcement learning system. It's not learning from consequences, pluses or minuses. It's learning from errors. And that's a supervised process, but it works together with a reinforcement learning system. And it's turned out to be quite effective in a number of applications.

是的。所以，一种表达方式，一种简单的理解方式，就是生活和心智既涉及预测也涉及控制。也许最终我们感兴趣的是控制。我们想做有良好结果的事情。但为了有好的控制，你必须做出好的预测来帮助你控制。

Yeah. So, one way to express that, a simple way to understand it, is that life and a mind involves both prediction and control. Maybe the ultimate thing is we're interested in control. We want to do things that have good outcomes. But in order to have good controls, you have to make good predictions that help you to control.

所以这也是，我想也许你可以谈谈，Rich 也受到动物学习的启发。次级强化系统的概念。是这样吗？

So this was also, I think maybe you can speak to this, Rich was motivated also by animal learning. The idea of a secondary reinforcement system. Is that correct?

是的，完全正确。你几乎可以从早期的动物学习思想中看到这些算法，我认为这是一个非常直观的事情。比如，你开车时做了某事，差点撞车但没撞上。如果你能从预测中学习，你可以说：‘哦，我做了某事，然后我以为可能会撞车，但我摆脱了它。’所以你可以学到两件事：你可以学会不做让你陷入恐惧状态的事，也可以学会强化并重复让你摆脱它的事。但如果你等待最终结果，最终结果是，你没有撞车，所以也许一切正常。这其实是一个非常常识性的想法。

Yeah, that's exactly right. You can kind of almost see the algorithms in the animal learning ideas from way back and I think it's a very intuitive thing. It's like, let's say you're driving your car and you do something and then you almost have a crash but you don't. Now if you can learn from your predictions you can say, 'Oh I did something and then I thought maybe it was going to crash and then I got out of it.' So you can learn two things. You can learn not to do what got you into that fearful state and you can learn to reinforce and repeat what got you out of it. But if you're waiting for the final outcome, the final outcome is, well, you didn't have a crash, so maybe it was just fine. This is a very common sensical idea really.

我的意思是，这实际上是一个很好的例子。回到下棋的例子，Rich，我理解你说的每走一步你都在学习。你可能犯了一个错误，然后你在纠正你的路径。所以你走的路径可能会根据你到目前为止学到的东西而变化。

I mean, that's actually a great example. Going back to the chess example, Rich, I understand what you mean by saying with every move that you make, you're learning something. You possibly made an error and you're correcting your path. So the path that you take could vary based on what you've learned so far.

你如何衡量学习的有效性？是不是说，我下的第一盘棋输了，但我学到了一些东西，然后下一盘棋我表现稍好一些？用什么指标来判断一个人确实在学习、学得更多、结果更好？

How do you measure the efficacy of learning? Is it that hey this first chess game I played I lost but I learned an amount and then the next chess game I'm doing slightly better like what is that measure to say that that person is actually learning and learning more and getting better outcomes?

嗯，我们无时无刻不在学习。我觉得这很明显，尽管现代 AI 实践并非如此。现代 AI 实践常常将学习与行为分开。但我认为很明显，动物和人类一直在学习，这并不意味着单调的改进，不同的事情可能发生。你不能把它简化为一个单一的数字，比如在某个特定时刻你知道多少。你只能继续学习，并希望长期平均下来越来越好。

Well, we learn all the time. I think it's sort of obvious although modern AI practice doesn't work that way. Modern AI practice often separates learning from behavior. But I think it's clear that animals and people we learn all the time and doesn't mean that we're you know it's not monotonic improvement different things can happen. You can't and you can't reduce it to a single number like how much you know at a particular moment in time. You can just keep learning and hope that you are getting on average in the long run better and better.

明白了。这是个合理的观点，尤其是学习没有终点。你活着的每一天都在学习。所以确实有这方面。回到另一个问题，有没有来自神经科学、生物学或观察动物的洞见，是我们计算机科学家今天低估了的？

Got it. That's a fair point, especially with learning because there's no sort of end to it. You continue to learn every day that you live. So there's definitely that aspect. Going back to one other point, is there an insight from neuroscience or biology or by observing animals that we as computer scientists underestimate today?

我认为有。

I think so.

是的，我同意。我认为我们只是触及了动物能力的皮毛。

Yeah, I agree. I think we've just scratched the surface of what animals are able to do.

正是。

Exactly.

你认为研究界正在积极进行这方面的工作吗？

Is that work that's actively happening do you think in the research community?

我觉得很奇怪，计算工作与心理学和神经科学工作之间的直接重叠如此之少。

I think it's strange how little direct overlap there is between the computational work and the psychological and neuroscience works.

好的。回顾你的研究生涯，有没有一个顿悟时刻或一个让你印象深刻的惊人发现？

Okay. Looking back at your research career, was there a single moment like an aha moment or a surprising discovery that you made that stands out for you?

回想起来，嗯，TD 学习与多巴胺系统之间的联系，但我想那时我已经退休了。我不记得了。所以对我作为活跃研究者来说，它并不是真正的动力。但在早期，我觉得有趣的是发现有很多混淆，认为误差纠正就是强化学习。从错误中学习，即监督学习，这是最常用的机器学习。把它等同于强化学习，甚至知名研究者我也见过这个错误，所以试图解释为什么这是个错误以及思考这些问题的正确方式，这对我很有动力。也许不是最突出的，我不确定能否确切指出。那是一段很长的时间。我觉得获得新资助就是一个好时刻，从国家科学基金会获得了几个新资助。这是基础研究。是的。但哪些是最突出的？我记不清了，都太久远了。我认为我们的工作进展缓慢，我们逐渐弄清楚，这主要是理解的深化，事情会各就各位。我们会说，是的，就是这样，或者只有这个没有别的。也许那是对的，然后随着时间推移，当我们没有发现其他东西时，它得到了确认。还有一些事件主要是由其他人完成的，比如我们都知道的 AlphaGo，那非常引人注目，是一个大进步。嗯，我想在那之前是 Jerry Tesauro 在 IBM 做的西洋双陆棋。是的，那非常令人印象深刻，因为我倾向于想别人做过的事情。甚至 Q-learning 也非常好。它和我们做的一致，但是一个很好的发展。

Thinking back, well this connection between TD learning and the dopamine system but I think I had retired by that time. I don't remember. So it wasn't really motivating for me as an active researcher. But in those early days I think what was interesting to me was finding that there was a lot of confusion thinking that error correction is the same as reinforcement learning. Learning from errors which supervised learning which is the most prevalent sort of machine learning that is being used. Equating that with reinforcement learning and even renowned researchers I'd seen that mistake and so trying to explain why this is a mistake and what is the correct way of thinking about these things was motivating for me. Maybe not the most, I'm not sure if I can put my finger on it. It was a long period of time. I think just getting a new grant was a great moment to get several new grants from mostly from the National Science Foundation. It's basic research. Yeah. But what were the most outstanding? I can't, it was all so long ago. I think our work was slow and we were gradually figuring out and it was just a deepening of understanding and things would fall into place. We'd say yeah that's what's going on or it's only this and nothing else. Perhaps that's right and then that would get confirmed over time as we didn't find other things. Now there were events that were done mainly by others like we all know about AlphaGo that was so striking that was a big step. Well I think before that was the backgammon that Jerry Tesauro did at IBM. Yeah that was very impressive because I'm tending to think of things that other people have done. Even Q-learning was just very nice. It was consistent with what we were doing but it was a nice development.

我认为这也符合你工作的精神。我的意思是，学习发生在一段时间内，这也解释了为什么这项研究至今仍有如此长的相关性。这引出了我的下一个问题：有没有一个现代强化学习应用，你认为最能代表你早期研究的精神？比如 Alpha Zero。Alpha Zero 是一个游戏学习系统，可以玩各种双人游戏，除了游戏规则知识外不需要太多帮助。

I think it also aligns with the spirit of what you do. I mean learning happens over a period of time and then that also explains why there is so much longevity in the relevance of the research even today which leads me to my next question which is is there a modern application of reinforcement learning that you think best represents the spirit of your early research? It would be like Alpha Zero. Alpha Zero was a game learning system that would play any of a wide variety of two-player games and it didn't require much help other than knowledge of the rules of the game.

嗯，我倾向于想到机器人应用。那很好。现在我们开始看到非常令人印象深刻的运动能力，通常是类人机器人，但也有其他类型的机器人。我认为在很多情况下，他们使用强化学习来调整机器人实现的运动算法。我一直对运动控制以及大脑如何做到这一点感兴趣。我认为强化学习参与了我们学习移动的方式，并且正在被开发机器人的人使用。

Well, I tend to think of robot applications. That's good. Now we're beginning to see very impressive movement abilities of often humanoid robots, but other kinds of robots too. And I think in many cases they're using reinforcement learning to adjust the movement algorithms that are implemented by robots. I've always been interested in motor control and how the brain does that. And I think reinforcement learning is involved in how we learn to move and it is being used by people developing robots.

那真是个好例子。是的，非常迷人。你对未来 20 年人类与基于强化学习的系统、产品或机器人之间的关系如何演变有什么想法吗？有什么想法吗？

That's a really good one. Yeah, that's pretty fascinating. Do you have any thoughts on how you imagine the relationship between humans and reinforcement learning based systems or products or robots for example will evolve over the next 20 years? Any thoughts around that at all?

我们各自的想法可能有点不同。我认为这将实现人类永恒的追求：理解自己，让自己更好地运作。所以这将会发生。这将继续，在未来 20 年内，我们很有可能理解我们的心智，并能够重现基本功能。我觉得这非常令人兴奋。这也让人们担忧或担心，但我不确定。我坐在这里，刚刚提到理解是好的，如果我们理解心智，那将是一件好事。现在有些人可能会用它做坏事，但基本上是一件好事。如果我们理解心智，它将迫使我们重新思考很多事情：我们是谁，我们能做什么，我们想做什么。我认为这将非常令人兴奋。所以我的观点有点不同。你知道，有人担心应用这些算法的危险，对我来说，强化学习的危险可以追溯到很久以前。关于优化算法的讨论中已经提到过。所以有一个担忧：如果你让一个系统最大化某个指标，你事先不知道它会得出什么，所以你必须小心行事。但另一方面，最大化对很多很多应用都非常有益。而这只需要，我的意思是，如果你发布一个智能体，一个使用强化学习的自我改进智能体，你需要有约束来防止它做你不希望它做的事情。所以这是一个担忧。我知道人们正在努力解决这些问题。但至于会发生什么，我不知道。

We each have thoughts maybe a bit different. I think it will be a fulfillment of the eternal striving of people to understand themselves and to make themselves work better. And so this is what will happen. This will continue and in the next 20 years there's a very good chance that we will understand our minds and be able to recreate the basic functions. I just find it so exciting. It also makes people concerned or worried but I don't know. I'm sitting here, I've just brought up that understanding is good and if we understand our minds that will be a good thing. Now some people might do bad things with it but it's basically a good thing. If we understand our minds, it will cause us to have to rethink a lot of things: who we are, what we can do, what we want to do. And I think it's going to be very exciting. So my views are a little bit different. You know, there are concerns about dangers of applying some of these algorithms and I guess for me the dangers of reinforcement learning go way way back. They've been talked about with regard to algorithms that optimize. So there is a concern that if you ask a system to maximize some measure, you don't know a priori what it's going to come up with and so you have to do it with care. But on the other hand, maximization has been extremely beneficial for lots and lots of applications. And this just needs to be, I mean, if you release an agent, a self-improving agent that uses reinforcement learning, you need to have constraints that prevent it from doing things you don't want it to do. So that's a concern. And I know people are working to deal with these issues. But then in terms of what's going to happen, I have no idea.

我的意思是，这个领域发展得太快了，我并没有深入参与其中，但我确实担心良好的工程实践不一定得到遵循。这完全是另一个话题。你知道，我当然同意 Rich 关于理解心智和大脑的观点。我认为这是有用的，但并非所有人都朝着这个目标前进。

I mean, the field is moving so quickly and I'm not connected to it in an intimate way, but I do have concerns that good engineering practices are not necessarily being followed. That's a whole other subject. You know, I certainly agree with Rich understanding the mind, the brain. I mean that's a useful thing but not everyone is headed toward that objective.

是的。不，我认为这是一个合理的观点。我认为设置这些护栏以确保我们使用或构建的产品和技术是为了社会的普遍利益。但你说得对，我不知道是否有某个机构在制定政策并且得到遵守。所以这当然是所有研究人员和实践者需要考虑的事情。你会对一位刚开始从事 AI 研究的年轻研究者说什么？比如他们应该如何思考失败、坚持、好奇心以及安全伦理等。你有什么具体的想法吗？比如你会给这个领域的年轻研究者什么指导？

Yeah. No, I think it's a fair point. I think having those guard rails in place to ensure that we're using or building products and technology that is for the general benefit of society. But you're right, I don't know if there is a body out there that is setting policy and then there is adherence to it. So that's certainly something for all the researchers and practitioners out there to think about. What would you tell a young researcher maybe who's starting out in AI like how should they be thinking about failure, persistence, curiosity and safety ethics etc. Do you have specific thoughts like what guidance would you give a young researcher in this field?

让我来谈谈这个。我认为在科学中，尤其是在 AI 研究中，潮流非常盛行。目前大型语言模型非常流行，很多人投入大量精力。但我在人工智能领域看到过潮流的兴起和衰落。所以我建议学生不要追随潮流，而是追随自己的热情和兴趣。我们从 Rich 那里听到，他一直对更好地理解心智感兴趣。我认为内在兴趣应该驱动一个人选择做什么，而不是潮流。我不是一个时髦的人。

Well let me tackle that. I think in science and actually especially in AI research, fashion is really quite prevalent. So currently now large language models are very fashionable. Lots and lots of people pouring effort into that. But I've seen fashions arise and decay over the years in artificial intelligence. And so I advise students not to follow the fashions but to follow their passion, their interests. I mean, we hear from Rich that he's been interested in understanding the mind better. I think intrinsic interests should be driving one's choices of what to do rather than fashion. I'm not a fashionable guy.

不过这是好建议，Rich。

Good advice though, Rich.

避免潮流。所以我完全同意，而且我认为一个年轻人所做的最重要的事情，往往是那些现在对他们来说几乎显而易见，但他们没有意识到别人看不到的事情。这就是我对强化学习的感受。强化学习很明显，你知道动物必须尝试事物，做感觉好的事，避免感觉坏的事。如此显而易见的事情，但它当时并不是一个领域。也许我们的贡献就是认识到那个显而易见的事情并发展它。所以我认为这是一个通用策略。你总是可以寻找显而易见的事情并发展它。那些人们没有注意到的显而易见的事情。就像查尔斯·达尔文做了什么？你知道，他只是环顾四周说：“哦，看，基本上人就是动物。”我的意思是，如果你看他们，这是显而易见的。我们都有心脏、肺、骨骼、皮肤和肌肉。所以我们显然是动物，并且从它们进化而来等等。所以你要看到你领域中未被认识到的、别人看不到的显而易见的事情。坚持下去。

Avoiding fashion. So I agree completely and I like to think that the most important things a young person will ever do will be things that are really almost obvious to them now but they don't realize that other people don't see them. That's how I felt about reinforcement learning. Reinforcement learning obviously you know animals have to try things and do what feels good and avoid what feels bad. Such an obvious thing and yet it was not a field. And maybe our contribution was just recognize that obvious thing and develop it. And so I think that's a general strategy. You can always look for the obvious thing and develop it. The obvious thing that people are not paying attention to. So like what did Charles Darwin do? You know, he just looked around and said, "Oh, look, basically people are just animals." And I mean, which is obvious if you look at them. We all have hearts and lungs and bones and skin and muscles. So we are obviously animals and we evolve from them and things like that. So you want to see the obvious that is not recognized in your field and that other people aren't seeing. Stick to it.

这让我想到，Rich 听过我谈这个，但如果你看机器学习的历史，甚至在 30 年代计算机出现之前，人们就在建造使用强化学习的机电系统。在计算机上实现学习系统的早期，它们都是强化学习系统。然后在某个时刻，它转向了我们现在所说的监督学习。这有很多原因。但其中一个原因是心理学中的认知革命使得这种基本的学习类型不再值得研究。所以我认为心理学历史在很大程度上扼杀了实现强化学习系统的计算努力。所以又是潮流，潮流的影响非常大。比如在心理学中，动物学习理论流行了几年，然后在 60 年代和 70 年代变得过时。强化学习在潮流中起伏。神经网络至少经历了三次巨大的起伏。你必须看穿这些。

This makes me think of, I mean Rich has heard me talk about this, but if you look at the history of machine learning, even before computers existed in the 30s, they were building these electromechanical systems that learned using reinforcement learning. And in the early days of implementing learning systems on computers, they were reinforcement learning systems. And then at a certain point, it switched to doing what we now call supervised learning. And there are lots of reasons for that. But one was that the cognitive revolution in psychology kind of made this basic type of learning really not something that was worthy of study anymore. So I think the history of psychology had a lot to do with extinguishing computational efforts to implement reinforcement learning systems. So it's fashions again, the fashions are a very big effect and so like in psychology animal learning theory was fashionable for some years and then like in the 60s and the 70s it became out of fashion. Reinforcement learning has varied in fashions. Neural networks have tremendously varied up and down at least three times. You have to look past that.

所以我会说从经验中学习。你知道，这是理解强化学习最快的方式。一个系统可以与世界互动并从中学习，而不需要老师的指令，只是从世界获得经验，感觉好的和感觉坏的事情，并从中学习。所以这是一个显而易见的想法。我喜欢提到它的第一次提及是艾伦·图灵在 1947 年，那时甚至还没有人工智能领域。他做了一个演讲，说我们真正想要的是一个能够从经验中学习的机器。所以那是在最开始，然后你知道安迪和我在 1980 年左右重新唤醒了它，然后事情来回摇摆。现在强化学习只有一半的支持，因为最重要的是大型语言模型，而它们不从经验中学习。它们实际上没有经验。

And so I would learning from experience. You know, that's the quickest way to understand what reinforcement learning is about. A system that can just interact with the world and learn from that without getting instructions from a teacher, but just get experiences from the world, things that feel good and things that feel bad, and learning from them. And so this is an obvious idea. The first mention of it I like to mention was Alan Turing in 1947 before there even was a field of artificial intelligence. He gave a talk and spoke the words that what we really want is a machine that can learn from experience. So that was at the very beginning and then you know Andy and I kind of reawoke it about 1980 and then it's you know things swing forward and back. Right now it's reinforcement learning is only sort of half in favor because the most important things are large language models and they don't learn from experience. They don't really have experience.

很多人正在使用强化学习来改进大型语言模型。

A lot of people are using reinforcement learning to improve large language models.

是的。他们把它嵌入其中。而且很多时候它不像评估。而是他们做事情，然后一个人说那很好。所以不是世界告诉你它有一个好结果。只是他们需要一个人在场并说那是一种形式。还有其他形式，我不...我不...。

Yeah. They're losing it inside it. And very often it's not like an evaluation. It's like they will do things and then a person will say that's good. So it's not the world telling you that it had a good outcome. It's just they require a person to be there and to say that's one form. There are others that they're I don't I'm not...

你说得对。

You're right.

我的意思是他们在做很多很多不同的事情。

I mean they're doing lots and lots of different things.

但他们没有做的一件事是，他们没有从回答问题以及与人们互动的经验中学习。他们没有从日常生活中学习。他们从预训练阶段和微调阶段学习。然后当系统部署并与人们互动时，他们就不再学习了。所以他们确实不从经验中学习。所以这对我来说有点遗憾，我认为这将在未来几年得到解决。

But one thing they're not doing is they're not learning from the experience answering questions and interacting with the people. They're not learning from their ordinary life. They learn from a pre-training phase and from a fine-tuning phase. And then when the systems are deployed and interacting with people, then they're no longer learning. So they really do not learn from experience. And so that feels a little bit of a shame to me and I think that will be fixed over the years.

是的。所以我的意思是，如果你想寻找一些显而易见且需要做的事情，那就是从经验中学习，因为今天最精密的 AI 系统并不从它们的经验中学习。

Yeah. So I mean if you want to look for something that is obvious and needs to be done, you know, it's still learning from experience because today's most refined AI systems do not learn from their experience.

谢谢。是的，这听起来非常有见地。我认为回到你们两位所说的核心，就是追随你的兴趣，追随你的热情，而不一定只是当下的潮流。实际上，我花了一些时间听你的采访来准备这次对话。你的许多学生和同事谈到你花时间帮助他们理解概念，发现他们应该追求的下一个问题，这充分说明了你对许多年轻研究者的影响和冲击。

Thank you. Yeah, that sounds I mean that's super insightful. I think going back to what both of you said at the crux of it was, you know, to follow your interest, follow your passion, not necessarily just the flavor of the day. And I actually spent some amount of time listening to your interviews in preparation for this conversation. And so many of your students and colleagues speaking about the time that you've spent with them helping them understand concepts, uncover the next question that they should pursue just speaks volumes of kind of influence and impact that you've had on so many young researchers.

所以我想问你们最后一个问题，作为我们的收官之问：当安迪和里奇不是计算机科学家时，他们是谁？告诉我们一件你们之前没分享过的事。

So I would just want to ask you sort of one last question for our final bite which is who are Andy and Rich when they're not computer scientists? Tell us one thing that you've not shared before.

嗯，我已经好多年不是计算机科学家了。我是个木匠。我想我仍然是个机械工程师，虽然不是正式的工程师，但喜欢机械和木工。我收集电动工具，并尽量不伤到自己。除此之外，我一直在读书，读小说。我试图理解一些我从未弄懂的物理学问题，但我发现没人真正懂它们，所以这也不是什么问题。现在退休了，我不再受约束，不用被迫阅读，可以自由选择读什么，下载书也很方便。除了计算机科学家这个身份，我有两个姐姐，她们没有博士学位，但都在做计算机相关的工作。所以我觉得我和姐姐们有某种基因成分，让处理计算机变得有趣。我想我是基因的奴隶吧。

Well, I haven't been a computer scientist for quite a few years. I'm a carpenter. I'm still a mechanical engineer, I guess. And it's not officially an engineer, but mechanical things and carpentry. I collect power tools and try not to injure myself with them. And otherwise, you know, I read all the time. I read novels. I am trying to understand some things about physics that I've never understood. I find that nobody understands them. So, it's not such a problem. I guess now that I'm retired, I'm not constrained and I'm not doing forced reading. I make my own choices as to what I read and I can download books. It's very easy. But besides a computer scientist, I had two older sisters. They didn't have PhDs, but they were both doing computing. So, I think there's a genetic component in me and my sisters that somehow makes dealing with computing interesting. So, I think I'm a slave to my genetics, I guess.

很好。我喜欢你的回答，安迪，我自己也有点这种感觉。我尝试拓宽研究领域，也许我喜欢把自己看作一个哲学家，因为我喜欢那些问题，虽然我不认为自己记住了足够多的名字来成为一个真正的哲学家，但我喜欢这些想法，并会琢磨它们，有时也会谈论它们，因为我们所做的是理解心智的伟大事业，它有很多维度，影响哲学的许多不同部分。我们都在试图理解世界，并找到简单的方式与他人交流和思考。是的，我试图从读的书中获得这些，无论是像大卫·多伊奇这样的物理学家，还是像伊恩·班克斯这样的科幻作家。这一切都是为了理解我们在宇宙中的位置。

That's good. I like your answers, Andy, and I feel a little bit like that myself. I try to get broader studies, you know, maybe I like to think I'm a little bit of a philosopher even though because I like all those questions although I don't think I memorize enough of the names to be a proper philosopher but I like the ideas and I toy with that and sometimes I do speak about it because what we're doing is a great thing of understanding the mind and it has so many dimensions and impacts so many different parts of philosophy. We're all trying to understand the world and find simple ways to communicate it to others and think about it. And yeah, I try to get that from the books I read, whether they're physicists like David Deutsch or science fiction writers like Ian Banks. It's all trying to understand our place in the universe.

是的，和你们两位交谈非常愉快。非常感谢你们抽出时间参加 ACM Bitecast。

Yeah, it's been an absolute pleasure talking to both of you. Thank you so much for taking the time to speak with us at ACM Bitecast.

非常感谢。

Well, thank you very much.

谢谢，很愉快。

Thank you. It's been a pleasure.