追寻 AI 智能体的简单通用架构

我很高兴来到这里，有机会与大家交流并分享我对心智的看法。今天我将呈现一个愿景。这不是一个完整的算法，也不可能完整，因为它需要一些尚不具备的前提条件。这些前提条件基本上是一个能够持续学习并随时间改进学习方式的深度学习算法。这看似是个小要求，但奇怪的是，我们现有的算法家族中还没有这样的东西。

It's my pleasure to be here and have a chance to talk to you all and present my ideas about the mind. Today I'm really going to present a vision. It's not a complete worked algorithm, and it can't be because it has prerequisites that are not available yet. The prerequisites are basically a working deep learning algorithm that can continually learn and improve the way it learns over time. It's a small ask, but it's something we don't have yet in our family of algorithms.

在开始之前，我想提一下，我的众多职务之一是创立了开放心智研究所（Open Mind Research Institute），在那里我们进行关于强化学习方法与心智的基础研究。如果你是对这个领域感兴趣的年轻人，可以访问 openmindresearch.org 了解我们。

Before I start, I want to mention that among my many jobs, I founded something called the Open Mind Research Institute, where we do basic research on reinforcement learning related approaches to the mind. If you're a young person interested in that area, you might want to check us out at openmindresearch.org.

让我们开始吧。基本上，我想谈论的是我对一个简单通用的 AI 智能体架构的探索。我会花一些时间讨论目标——我们如何在一个非常庞大、复杂、未知的世界中从经验中学习——然后进入架构本身。我会用几种方式解释它，包括四张图，并逐步讲解。

Let's get started. Basically, what I want to talk about is my quest for a simple general architecture for an AI agent. I'll spend some time talking about the goals—how we want learning from experience in a very big, complex, unknown world—and then we'll get into the architecture itself. I'll have several ways of explaining it, with four pictures, and go through it step by step.

这就像是 AI 的圣杯，实际上是 AGI 的圣杯——对通用心智的追求。所谓通用，是指它不包含任何特定于某个世界的知识，没有领域知识。其次，它是经验性的：我们从运行时经验中学习，而不是从专门的训练阶段。运行时经验包括观察、动作和奖励——未标记、未解释的信息。最后，也许最重要的是开放式抽象的概念。我们希望智能体发展出概念、思维方式、行为中的常见模式，其复杂性除了受计算资源限制外，是无限的。

It's like the holy grail of AI, of AGI really, is to have a quest for mind that is general. By general, I mean it contains nothing specific to any world—no domain knowledge. Secondly, it's experiential: we learn from runtime experience, not from a special training phase. Runtime experience is the observation, action, and reward—unlabeled, uninterpreted information. Finally, perhaps most important, is the idea of open-ended abstraction. We want the agent to develop concepts, ways of thinking, common patterns in its behavior that are unlimited in sophistication, except by computational resources.

抽象有两种：一种是找到好的状态特征——线索、概念、组织信号的方式，帮助你理解世界并做出决策；另一种是找到好的时间抽象，即比单个动作更大尺度上可以做的事情，比如走路去上班或开门。

Abstraction is of two kinds: finding good state features—cues, concepts, ways of organizing signals that help you understand the world and make decisions—and finding good temporal abstractions, things you can do at a larger scale than a single action, like walking to work or opening a door.

我们有两个术语：设计时和运行时。设计时是智能体在被派往世界之前被设计的时候。在设计时，你内置领域知识。我想最小化这一点。运行时是智能体从经验中学习并针对它所遇到的世界特定部分制定计划的时候。我强调运行时学习。在一个无法完全预见的庞大复杂世界中，仅在设计时内置东西是不够的。对于开放式抽象，你必须在运行时发现它们。一切都要在运行时完成。那么为什么不在设计时什么都不做呢？为了理解智能，你希望它尽可能简单纯粹。

We have two terms: design time and runtime. Design time is when an agent is designed before being sent into the world. At design time, you build in domain knowledge. I want to minimize that. Runtime is when the agent learns from experience and makes plans specific to the part of the world it encounters. I emphasize runtime learning. In a big, complex world where things can't be fully anticipated, building things in at design time alone is insufficient. For open-ended abstractions, you must discover them at runtime. Everything has to be done at runtime. So why not do nothing at design time? For understanding intelligence, you want it as simple and pure as possible.

这种智能体架构，作为理解智能的理想架构，不应在设计时对任何特定世界做出承诺。从我那篇《苦涩的教训》博客文章来看：心智的实际内容是任意的、本质上复杂的外部世界的一部分。它们的复杂性是无穷的。它们不应该被内置。相反，我们应该只内置能够发现并捕捉这种复杂性的元方法。我们希望 AI 智能体能够发现，而不是包含我们已经发现的东西。

This agent architecture, an ideal one for understanding intelligence, should make no design-time commitments to any particular world. From my bitter lesson blog post: the actual contents of minds are part of the arbitrary, intrinsically complex outside world. Their complexity is endless. They should not be built in. Instead, we should build in only the meta-methods that can find and capture this complexity. We want AI agents that can discover, not contain what we have already discovered.

智能体应该从专门的训练数据中学习，还是只从运行时经验中学习？对我来说，答案很明确。智能的力量，特别是强化学习的力量，在于我们可以从未经准备的运行时经验中学习。我的承诺是智能体只从运行时经验中学习。这就是我所说的完全经验性。

Should the agent learn from special training data or only from runtime experience? For me, the answer is clear. The power of intelligence, particularly reinforcement learning, is that we can learn from unprepared runtime experience. My commitment is that the agent should only learn from runtime experience. That's what I mean by entirely experiential.

我提到了大世界视角。这是我们的智能体，像一个人，比世界小。世界包含所有物体、地点、物理世界的复杂性，以及无数其他智能体。他们心智中发生的事情对智能体来说至关重要。

I mentioned the big world perspective. Here's our agent, like a person, smaller than the world. The world contains all the objects, places, complexities of the physical world, and a myriad of other agents. What's going on in their minds is totally important for the agent.

智能体的正确行为是与他人互动，包括它的老板、配偶和朋友。他们心里想什么对他们很重要，就像你们现在心里想什么对我很重要一样。因此，世界远比智能体复杂，这是不可避免的，智能体学到的东西不可能完全正确，也不可能最优。任何关于最优性或正确性的定理都与实际情况无关。所有的价值函数都必须是近似的，策略也必须是近似的，状态转移模型，尤其是世界模型，必须比世界简单得多。所以它会是错误的、近似的。即使是世界的单一状态，你也无法完全记住，因为状态包含了所有其他智能体内心的活动。而你内心的复杂度不可能远超其他所有心智。另一个更微妙的含义是，世界会显得非平稳，因为你无法确切知道发生了什么。它有时会呈现一种方式，有时又是另一种方式。就像开车时，你看到前面的车，不知道它会右转还是左转。这并非随机；前车司机脑子里的想法可能是确定性的，但对你来说它显得非平稳。有时车往这边转，有时往那边转。面对所有这些关于世界的有效不确定性，你必须在运行时学习，在运行时规划，在运行时找到任何所需的抽象。你出生、成长、接触世界，然后弄清楚它的物体、人物、世界运作方式以及社会制度。所有这些都必须在运行时解决，不能预先设计好。所以我们不赞成预先设计。

The agent's correct behavior is interacting with people, its boss, its spouse, and its friends. What's going on in their minds matters to them just as what's going on in your minds now matters to me. So the implications of the world being much more complex than the agent and inevitably so, nothing the agent learns can be exactly correct. It can't be optimal. Any theorem about optimality or correctness is irrelevant to the real situation. All your value functions must be approximate. Your policies must be approximate. Your state transition models, particularly your model of the world, has to be much simpler than the world. So it's going to be wrong, approximate. Even a single state of the world you can't really keep in your head because the state includes what's going on in all those other agents' minds. And what's in your mind can't be much more complex than all those other minds. Another implication, a bit more advanced, is that the world will appear non-stationary because you don't know exactly what's going on. It will appear sometimes one way, sometimes another. Just like driving down the street, you see a car ahead; you don't know whether it will turn right or left. It's not random; whatever is in the head of that driver may be deterministic, but it appears non-stationary. Sometimes the car turns one way, sometimes another. With all these effective uncertainties about the world, you have to learn at runtime. You have to plan at runtime. You have to find any needed abstractions at runtime. You're born, grow up, encounter the world, and figure out its objects, people, how the world works, and social institutions. You figure all that out at runtime. It can't be designed in. So we deprecate that.

现在我们开始讨论 Oak 架构，它解决了所有这些问题。我先谈谈通用 AI 问题：设计一个在世界上行动的有效、有目的的智能体。在强化学习中，问题是一样的，只是我们以特定方式表述：将目标指定为一个标量信号，称为奖励。我稍后会详细说明。其次，在强化学习中，我们假设世界是通用的且不完全已知。因此，世界可以是任何东西，从网格世界到复杂的人类世界，随机、复杂、非马尔可夫、非线性，其状态空间实际上是无限的，其动态实际上是非平稳的。我们必须在运行时工作，在运行时应用，而不需要来自教师或人类的特殊训练信号。这个问题很棒。我认为我们不需要改变它。奖励信号就足够了。我们只需要解决它。我还想提一下，我假设智能体受限于算力，而不是数据。数据是丰富的，我们需要流式算法，因此我们会排除像回放缓冲区这样的东西。

Let's get going now to the Oak architecture, which addresses all these things. Let me approach it by talking about the general AI problem: to design an effective purposeful agent that acts in the world. In reinforcement learning, it's the same problem except we formulate it in a particular way. We specify the purpose as a scalar signal called reward. I'll say more about that in a minute. Secondly, in reinforcement learning we assume the world is general and incompletely known. So the world could be anything from a grid world to a complex human world, stochastic, complex, non-Markov, nonlinear, and its state space is effectively infinite and its dynamics effectively non-stationary. We have to work at runtime. We have to apply at runtime without some special training signal from a teacher or human. This problem is great. I think we don't need to change it. The reward signal is enough. We just have to solve it. I also want to mention that I'm going to assume the agent is limited by compute, not data. The data is rich, and we want streaming algorithms so that we will exclude things like replay buffers.

我说过目标是由一个标量信号指定的。我再花一页幻灯片讲这个。这就是所谓的奖励假说。奖励假说认为，我们所说的目标和目的都可以很好地理解为最大化一个标量信号的累积和的期望值，这个信号我们称之为奖励。我认为这根本不是限制，而是一种非常清晰的目标指定方式。添加额外的东西不会让它变得更好。这方面有一些理论工作，我鼓励你们看看 Michael Bowling 等人关于解决奖励假说的论文。此外，我们知道即使是一个简单的奖励也能导致智能的所有属性。也许我不该说我们知道，但我和其他人论证过，在足够复杂的世界中，一个简单的奖励可以导致智能的所有属性。这就是那篇“奖励就够了”的论文。

I've said the purpose is specified by a scalar signal. I'll spend another slide on that. This is what's called the reward hypothesis. The reward hypothesis is that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal that we call reward. I think this is not a limitation at all; it's a great clear way to specify the goal. Adding additional things wouldn't make it better. There's been some theoretical work on this, and I encourage you to check out the paper by Michael Bowling and others on settling the reward hypothesis. Also, we know that even a simple reward can lead to all the attributes of intelligence. Maybe I shouldn't say we know that, but I and others have argued that in a sufficiently complex world, a simple reward can lead to all the attributes of intelligence. That's the 'reward is enough' paper.

现在我们开始。Oak 架构，我想先从一个先前的架构开始，我称之为智能体的通用模型，它在许多领域都很常见：AI、心理学、控制理论、经济学、神经科学、运筹学。每个人都使用一个具有某些基本属性的智能体模型。第一是红色部分：体验接口——你有奖励、动作以及来自世界的某种信号作为观察，而且观察通常不足以完全消除状态的不确定性。这是外部视角，那三个红色东西。但通用模型更多是关于内部的东西：四个组件。首先，看看连接组件的是什么：蓝色部分，状态特征向量。它表示你对世界状态的表征，是一个特征向量，每个分量是一个特征。这个状态特征向量是你对自身位置的理解。当你从输入和过去动作中创建这种位置感时，一个合适的词是感知。它是你感知世界并表征你现在位置的方式。它是你选择动作的基础。这个状态表征流经并作为动作的输入。你的策略会反应性地，无需太多思考，选择一个动作。这是前两个组件：感知和策略，它们共同构成一个完整的智能体。但为了学习和改进这个智能体，你需要学习一个价值函数。价值函数说：“我现在做得好或不好。事情变好了或变坏了。”所以它必须知道奖励，因为价值函数本质上是对奖励的预测。它被用来学习策略，我们通过在策略结构上画这条对角线来表示。

Now let's get started. The Oak architecture, and I like to start with a prior architecture, what I call the common model of the intelligent agent, common to many fields: AI, psychology, control theory, economics, neuroscience, operations research. Everyone uses a model of the agent that has certain basic attributes. Number one is the red things: the experiential interface—the fact that you have reward, action, and some kind of signals from the world that are your observations, and the fact that the observation in general is not sufficient to totally disambiguate the state. That's the outside view, those three red things. But the common model is much about the inside things: the four components. First, look at what interconnects the components: the blue thing, the state feature vector. It's how you represent the state of the world, a feature vector with components, each a feature. This state feature vector is your understanding of where you are. When you create this sense of where you are from your input and past actions, a good word for that is perception. It's how you perceive the world and represent where you are now. It's the basis on which you pick your action. This state representation flows through and is the input to your action. Your policy will reactively, without much deliberation, pick an action. Those are the first two components: perception and policy, which together make up a full agent. But to learn and improve that agent, you want to learn a value function. The value function says, 'I'm doing well now or I'm doing poorly now. Things have gotten better or worse.' So it has to know the rewards because the value function is essentially a prediction of reward. That's used to learn the policy, which we suggest by drawing this diagonal line through the policy structure.

第四个组件是用于规划的转移模型。转移模型基本上接收状态，并尝试预测如果你以特定方式行动，下一个状态会是什么。我无法展示所有工作原理，但通过规划过程，我们有一条穿过策略的对角线，我们同时改变策略和价值函数。这就是我们开始的通用模型。学习和规划包含这四个组件。我觉得很有趣的是，我们可以用这个单一视角涵盖所有这些不同领域。控制理论会使用不同的名称：它不说动作，而说控制；不说奖励，而说收益或成本。在心理学中，例如，我们可能会说奖励，但不说观察，而说刺激。所有这些领域实际上都有相同的想法，只是名称不同。所以我们必须问：通用模型缺少了什么？为什么我们还没完成？关键缺失在于它是低层次的。动作是瞬时的，奖励是瞬时的，观察也是瞬时的。我们必须获得更高层次的抽象。我们必须发展概念，更高层次的思维方式。基本上，我认为我们需要添加的，也是 Oak 架构试图添加的，是对这种视图的开放式抽象。

And the fourth component is the transition model which we use for planning. The transition model basically takes the states and tries to predict the next states if you act in a particular way. I can't show all of how it works, but in one way or another, by the process of planning we have a diagonal line through the policy and again we change the policy and we also change the value function in both these cases. That's the common model where we start. Learning and planning has these four components. I think it's kind of interesting that we can take this single view to cover all these different fields. Control theory will use different names. It won't talk about action; it'll talk about controls. Won't talk about reward; it'll talk about payoffs or costs. In psychology, for example, we might say reward, but instead of observations, we talk about the stimulus. All these fields really have the same ideas, sometimes with different names. So we have to ask the question: what is missing from the common model? Why aren't we done? The key thing we're missing is that it's low level. The actions are momentary, reward momentary, the observations are momentary. We have to get higher level abstractions. We have to develop concepts, ways of thinking that are at a higher level. Basically, I think what we need to add, and what the Oak architecture tries to add, is open-ended abstraction to this view.

这是 Oak 架构，我试图用紫色标出所有新东西。它增加了辅助子问题。子问题是主问题的辅助。主问题是获得奖励，而子问题——我们会多次看到这个短语——它们是尊重奖励的特征获取子问题。这意味着每个子问题都针对一个特定特征并试图获取它，但尊重奖励。它说：‘在获取这个特征的同时，仍然获得足够的奖励，不要在奖励上损失太多。’这里的图看起来几乎一样。策略背后现在有一堆称为选项的东西，它们就像进一步的策略。价值函数背后有更多的价值函数。所以我们有一大堆价值函数。当然，每个子问题必须有一个价值函数，因为它是独立的问题，我们需要知道我们在它上面做得如何。所以我们需要更多的价值函数。现在我们有了四种组件：这两个相同，但这两个现在是多个。感知仍然大致相同，产生一个状态特征向量，供所有子问题使用。现在策略也包含所有选项，它们产生适合子问题的行为方式。而价值函数——许多价值函数——用于评估我们在每个子问题上的表现。这里我没有展示子问题；有趣的是，你实际上不需要在架构中为子问题设置一个对象。你只需要为它设置价值函数。从特征向量中拉出小卷须表示每个价值函数都有一个它试图优化的独立特征。

Here's the Oak architecture, and I've tried to show in purple all the things that are new. It's adding auxiliary subproblems. Subproblems that are auxiliary to the main problem. The main problem is getting reward, and the subproblems—we'll see this phrase a couple times—they are reward-respecting subproblems of feature attainment. That means each subproblem takes a particular feature and tries to attain it, but it respects the reward. It says, 'Attain this feature while still getting a good amount of reward and not suffering too much in terms of your reward.' The picture here looks almost the same. The policy now has behind it a bunch of more things called options, and they are like further policies. Behind the value function, we have more value functions. So we have a whole bunch of value functions. Of course, we have to have one value function for each subproblem because it's a separate problem, and we have to know how well we're doing with respect to it. So we need more value functions. Now we have four kinds of components: these two are the same, but these two are now multiple. We still have perception much the same, producing a state feature vector which will be used by all the subproblems. Now the policy also has all the options, and they produce a way of behaving that's appropriate for the subproblem. And value functions—many value functions—for evaluating how well we're doing on each subproblem. I don't show the subproblems here; it's interesting you don't actually need an object in your architecture for the subproblem. You just need to have the value function for it. And this pulling off little tendrils of the feature vector represents the idea that each value function has a separate feature that it's trying to optimize.

让我们稍微讨论一下转移模型。转移模型就是我所说的知识。我们所有的知识都在转移模型中：我们相信世界会如何变化，如果我们做不同的事情，特别是如果我们做这些称为选项的更大事情。Oak 架构的名字来源于选项和知识。选项是一种更高层次的行为方式，但它只是一对策略和终止方式。就这两样东西组合在一起。策略是从状态到低级动作概率分布的映射。终止条件是从状态到终止概率的映射。在 Oak 中，我们有很多这样的选项，比如每个数值特征一个。然后它学习关于如果你遵循那个选项直到终止会发生什么的知识。这就是你拥有的知识形式：比如‘如果我拿起物体并握在手中，它会是什么感觉？如果我走下楼梯看厨房，我会在那里看到谁？如果我去冰岛，我可能会遇到什么样的人？如果我参加讲座，我会学到什么？’这就是你对将要发生的事情的知识。它变成了世界的高层次转移模型，使规划能够着眼于更大的跳跃，并希望在世界的关键点上进行划分。

Let's dwell for a second on the transition model. The transition model is what I'm going to use the word knowledge for. All of our knowledge is in our transition model: how we believe the world would change if we did different things, and in particular if we did these larger things called options. The name Oak—the Oak architecture—comes from options and knowledge. An option is this higher level way of behaving, but it is just a pair of a policy and a way of terminating. Just those two things grouped together. A policy is just a map from states to probability distribution over low-level actions. The termination condition is just a map from states to probability of terminating. In Oak, we have a lot of these options, like one for each numerical feature. Then it learns the knowledge about what will happen if you followed that option until it terminated. That's the form of knowledge you'd have: things like 'If I picked up the object and held it in my hand, what would it feel like? If I walked downstairs and looked in the kitchen, who would I see there? If I traveled to Iceland, what kind of people might I meet? If I attend the talk, what will I learn?' That is your knowledge about what will happen. It becomes a high level transition model of the world that enables planning to look at larger jumps and hopefully carves the world at its joints.

Oak 架构涉及我们将要讨论的这八个步骤。它们在运行时并行完成。我将逐一讨论它们，但你可以先看看：我要学习最大化奖励的策略和价值函数。我要生成新的状态特征。我要对特征进行排序。我要创建子问题，每个高排名特征一个。然后我要学习子问题的解决方案。我要为这些子问题的解决方案学习转移模型。然后我要进行规划，并且必须维护关于一切实用性的元数据。希望其中一些词对你有意义。让我们一步一步地检查并评估：我们现在能做到吗？第一步，学习最大化奖励的策略和价值函数——这是经典的强化学习，我们有很多算法可以做到。但我仍然不会给它打绿色勾，因为我认为它还没有完全解决。如果我们真的能做持续深度学习，它就会解决。正如我一开始所说，这是我们还没有的主要前提。所以让我们看看这个。我暂时只关注第一步，注意到它需要可靠的持续深度学习。我们现在能做到吗？我们需要这个至少 40 年了。我很失望没有人想出如何做可靠的持续非线性学习。我们可以对线性网络做到。线性网络可以可靠且持续地学习。但传统的深度学习一旦被要求持续学习以保持学习能力，就会灾难性地失败。最著名的是，我们会遇到灾难性遗忘。

The Oak architecture involves these eight steps that we'll be talking about. They are all done in parallel at runtime. I'll be talking about them bit by bit, but you might just look at them: I'm going to learn policies and value functions maximizing reward. I'm going to generate new state features. I'm going to order the features. I'm going to create the subproblems, one for each highly ranked feature. Then I'll learn solutions of the subproblems. I'll learn transition models for the solutions of those subproblems. And then I'll plan, and I'll have to maintain metadata on the utility of everything. Hopefully some of those words made sense to you. Let's go through it step by step and assess: are we able to do it now? The first step of learning the policies and the value functions for maximizing reward—that is classic reinforcement learning, and we have many algorithms for doing that. But still, I'm not going to give it a green check because I don't consider it totally solved. It would be solved if we could really do continual deep learning. As I said at the beginning, this is the major prerequisite that we don't yet have. So let's look at that. I'm just going to focus on that first step for a moment, noting that it requires reliable continual deep learning. Can we do this yet? We've needed to do this for at least 40 years. I am rather disappointed that no one's figured out how to do reliable continual nonlinear learning. We can do it for linear networks. Linear networks can learn reliably and continually. But conventional deep learning fails catastrophically once we ask it to continue continual learning to maintain its learning ability. Most famously, we get catastrophic forgetting.

这要追溯到 90 年代，最近它变得突出：我们失去了持续学习的能力。这被称为可塑性丧失，由我实验室和其他地方的研究人员证明。一个解决方案是，我们可能有部分近期的解决方案：我们可以做深度学习并持续学习，比如持续反向传播。另一个可能有帮助的想法是持续发现新特征和步长。我相信，无论如何，我们将在未来几年内获得可靠的持续非线性学习。所以我认为可以合理地假设它会实现，并依赖它的存在，就像我在 Oak 架构中所做的那样。

This goes back to the 90s, and more recently it's come to the fore that we just lose the ability to keep learning. This is called loss of plasticity, and it's been shown by other folks in my lab and in others. One solution is that maybe we have partial recent solutions: we can do deep learning and learn continually somewhat, like continual backprop. Another idea that might help is continual discovery of new features and step sizes. I do believe that one way or another we will get reliable continual nonlinear learning in the next few years. So I think it's fair to assume that it will come along and to rely on it existing, as I do in the Oak architecture.

回到第二步：我们需要从现有状态特征中生成新特征。这里我不是在说找到最好的特征，而是生成一堆候选特征。我认为这并不明确；有很多想法。Oak 将需要发现新的状态特征。这是一个老问题，可以追溯到 60 年代。如果你在这个领域待过，你可能熟悉它。它有很多名字：表示学习、新术语问题、元学习。这正是反向传播本应解决但实际并未解决的问题。原始反向传播论文是《通过梯度下降学习表示》之类的。有些论文声称它做到了，但我认为如今我们承认它并没有真正找到好的表示，除了在绝望的意义上它最终解决了问题，这可能需要一些表示，但它泛化不好，而且它实际上并没有找到表示。其他方法基于生成和测试，可以追溯到很久以前：尝试一堆候选特征，然后选择那些泛化更好的并保留它们。持续反向传播就是这些基于生成和测试的新算法之一。所以这确实是一个未解决的问题。我们需要有一种从现有特征生成新特征的好方法，我将依赖于此。我有自己最喜欢的解决这个问题的想法：IDBD 算法。我认为它将成为解决方案的一部分。我在论文底部给出了这个名字，如果你想查看的话。

Going back to the second step: we need to generate new features from existing state features. Here I'm not talking about finding the best ones; I'm talking about generating a bunch of candidates. I think this is not really clear; there are lots of ideas. Oak will require the discovery of new state features. It's an old problem going back to the 60s. You're probably familiar with it if you've been around. It has many names: representation learning, the new terms problem, meta-learning. This is what backprop was supposed to solve, but really does not. The original backprop paper was 'Learning Representations by Gradient Descent' or something like that. There were papers claiming it does, but I think nowadays we acknowledge that it doesn't really find good representations except in the desperate sense that it eventually solves the problem, and that may require some representations, but it doesn't generalize well, and it really isn't finding the representations. Other methods have been based on generate and test, going way back: try a bunch of candidate features, then select the ones that generalize better and keep those. Continual backprop is one of these new algorithms based on generate and test. So this is really an unsolved problem. We need to have a good way of generating new state features from existing features, and I'm going to rely on it. I have my own personal favorite idea for solving this problem: the IDBD algorithm. I think that's going to be part of the solution. I give you the name here at the bottom of the paper if you want to check it out.

接下来，我们想要对这些候选特征进行排名。我认为这似乎很容易：根据它们是否对子问题、智能体和模型学习有用来排名。你可以直接看特征是否被使用，并据此排名。然后大问题是创建子问题，每个高排名特征对应一个。我想谈谈这个，因为我实际上认为这很容易做到，而且我们已经做到了。如何创建子问题？首先，我必须承认，人们研究辅助子问题已有很长的历史，有些问题基本已解决，有些仍然开放。开放问题：子问题应该是什么？它们从何而来？智能体能否自己生成子问题？子问题如何帮助解决获取奖励的主要问题？Oak 架构为所有这些问题提供了答案。

Next, we want to rank these candidate features. I think this seems easy: rank them by whether they are proving useful to the subproblems, to the agent, and to the learning of the model. You can just see if the features are being used and use that to rank them. Then the big problem is to create subproblems, one for each highly ranked feature. I want to talk about that because I actually think this is easy to do, and we have done it. How do you create subproblems? First, I have to acknowledge there's a long history of people working on auxiliary subproblems, and there are some issues that are largely settled and some that are still open. Open questions: what should the subproblems be? Where do they come from? Can the agent generate its own subproblems? How do the subproblems help on the main problem of getting reward? The Oak architecture proposes answers to all these questions.

子问题正是我们在玩耍中看到的。如果你想想动物甚至人类的生活，我们的生活中充满了子问题。这里有一只年轻的猩猩；我认为他专注于荡秋千时获得的感觉这个子问题。他试图享受并重现这种感觉。他能再次做到吗？他需要做什么？你可以把他正在进行的玩耍视为试图获得那个特征——荡秋千的特征，他的前庭系统告诉他那是什么感觉，这通常不会发生。类似地，这里有一只虎鲸在水族馆里；有人扔了一个浮球进去，它一直在玩。它发现可以用背驮着它。这是它注意到自己能做的一件有趣的事，然后没有经过训练，它就学会了可靠地做到这一点。这就是我认为的玩耍：专注于获得一个特征。这里有一个人类的例子：一个婴儿在玩耍。它的行为不是随机的；它有点加速。它与玩具互动，学到一些东西，然后当它无法再学到更多时，就转向下一个学习机会。这就是我们所说的好奇心，但我们可以将其视为具有特征获取的子问题——它可以拥有的某些体验，如颜色、感觉或声音，并试图重现它们。这是从子问题的角度思考探索和好奇心的一种方式。它是在获得对世界的控制。

Subproblems are what we're looking at in play. If you think about the life of an animal or even a human, our life is full of subproblems. Here is a young orangutan; I think he's focused on the subproblem of the sensation he gets when he swings. He's trying to enjoy that and reproduce it. Can he get that to happen again? What does he need to do? You can think of the play he's undergoing as trying to obtain that feature—the feature of swinging, his vestibular system saying what that feels like, which is something that doesn't normally happen. Similarly, here is an orca in an aquarium; someone threw a float in with him, and he's been playing with it. He found out he can carry it on his back. It's an interesting thing he noticed he could do, and then without being trained, he learned how to do it reliably. This is what I think of as play: being focused on obtaining a feature. Here's a human example: an infant playing. Its behavior is not random; it's sped up a little bit. It interacts with the toy, learns some things, and then moves on when it stops being able to learn more, looking for the next opportunity for learning. This is what we call curiosity, but we can think of it as having subproblems of feature attainment—certain experiences it can have, like colors, feelings, or sounds, and trying to be able to reproduce them. It's one way of thinking about exploration and curiosity in terms of subproblems. It's gaining control over its world.

显然，智能体必须自己创建子问题。不可能将所有可能的子问题都内置。婴儿遇到它的世界，看到有什么可做的，然后学习如何获得那里的东西。可能的子问题太多样化，太依赖世界了。我们必须将形成子问题、提出问题的责任交给智能体。我们有很多必要的算法来做到这一点：我们有选项，我们有通用价值函数的概念，我们有离策略学习和规划方法。所以我们必须以领域通用的方式创建子问题。你怎么能做到呢？它们可能是什么呢？我将提供解决方案：尊重奖励的特征获取子问题。最好的理由是，除了获取特征之外，几乎没有什么其他可以想到的。你不能谈论状态，因为智能体没有状态。

Obviously, an agent must create its own subproblems. There's no way for all possible subproblems to be built in. The infant encounters its world and sees what's available to be done, and then learns how to attain the things that are there. The possible subproblems are too various and too world-dependent. We have to give the responsibility for forming the subproblems, for posing the problems, to the agent. We have a lot of the necessary algorithms for doing this: we have options, we have the idea of value functions in full generality, we have off-policy learning and planning methods. So we have to create subproblems in a domain-general way. How could you do that? What could they possibly be? I'm going to offer the solution: reward-respecting subproblems of feature attainment. The best rationale is that there's really almost nothing else one can think of other than attaining the features. You can't talk about states because the agent doesn't have states.

你不能谈论世界中的物体，因为我们没有先验的物体。我们只有特征和概念，这些是我们为了理解世界而形成的。我没有太注意时间。我确实想知道我进行得如何。如果我有时间，我会详细讲解我们在 OAK 中创建子问题的具体方式。基本上，我们从特征开始。那是一个索引，一个特征编号，比如特征 53，而 kappa 是你想要该特征的强度，你需要指定它。你指定这两样东西，我们就可以自动得到一个子问题。子问题是将世界驱动到特征值高的状态，同时不损失太多奖励。所以解决方案是一个选项。它是一个策略和一种终止方式，在尊重奖励和价值的同时最大化终止时特征的价值。你要选择选项，以最大化在选项期间获得的总奖励的期望值，即从当前时间开始的一些奖励。这都是有条件的：我从时间 t 在指定状态 s 开始，然后我们开始遵循选项，我们会得到一些奖励，并在时间 T 结束，所以我们只考虑这些时间之间收到的奖励。我们还对结果状态的价值感兴趣。我们不想被留在一个坏事将要发生的状态。所以这是我们处于那个终止状态的估计价值。这就是特征达成。当然，我们希望当我们在时间 T 停止时，处于特征 fi 高的状态。所以那个价值很大，并且乘以 kappa，即你想要该特征的强度。这些是尊重奖励的特征达成子任务，前年有一篇关于此的论文发表在 AI Journal 上。所以我认为子问题应该是这样的，然后我们必须学习解决方案并学习解决方案的模型。解决方案就是选项。这两件事相对简单，但同样依赖于我们能否做好持续深度学习。但如果你同意这个前提，那么我们就可以做到。所以这就是我们真正触及 OAK 核心的地方。现在请注意论证的整体结构。基本上，我们将进行所有这些学习过程，它们将使用特征，并推动新候选特征的创建。所以我们有特征传递到这里。一旦我们有有趣的特征，我们就可以创建子问题来实现每一个。一旦有了这些子问题，我们就解决它们。如果我们有一千个子问题，我们会解决它们以产生一千个选项。每个选项都将成为基础。我们问：如果我在这种情况下执行那个选项，会发生什么？那就是转移模型。我们预测以那种方式行为的后果。这与解决问题不同，解决问题是找到一种好的行为方式，希望实现特征。但会有许多其他后果。如果我下楼去厨房，我可能到达厨房，但也可能看到我的配偶，或者可能绊倒摔下楼梯。很多事情都可能发生，我需要预测可能的后果。所以那就是转移模型。然后一旦有了转移模型，我必须用它们来规划，这能改善我的行为，并希望让我获得高度适应的行为。现在让我们用另一张图来看同样的东西。我们谈论感知给我特征。所以现在我有了状态特征。然后我解决那些子问题。我与世界互动并获得选项。我预测选项的后果，这给了我一个转移模型。通过规划，转移模型给我改进的价值和策略。然后所有这些后续步骤反馈并告知感知。它们告诉感知：为了形成这个模型，我不得不使用一些特征而不使用其他特征。为了解决这个问题，我使用了一些特征而没有使用其他特征。所以这些过程告知特征构建过程，它们可以说：“哦，你给我的这些特征没用，你应该扔掉它们，给我更多特征，而这些其他特征非常有用。也许它们应该成为未来子问题的基础，这些子问题被提出和解决，从而以开放的方式递归地改进智能体的抽象。”

You can't talk about objects in the world because we don't have a priori objects in the world. All we have are features and concepts which we have formed to understand the world. I haven't checked my timing too much. I do wonder how I'm doing. If I have time to go through the specific way we create subproblems in OAK. Basically we start with a feature. That is an index, a feature number like feature 53, and kappa is how intensely you want the feature that you have to specify. You specify those two things, we can automatically have a subproblem. The subproblem is to drive the world to a state where the feature is high without losing too much reward. And so the solution is an option. It's a policy and a way of terminating that maximizes the value of the feature at termination while respecting the rewards and value. You want to choose the option so as to maximize the expected value of the total reward you'll get during the option, some of the rewards from the current time. This is all conditional: I'm starting at time t in the designated state s, and then we start following the option and we'll get some rewards and we'll end at time capital T, so we're just in the rewards received between those times. And we're also interested in the value of the resultant state. We don't want to be left in a state where bad things are going to happen. So this is our estimated value of being in that terminal state. And this is the feature attainment. Of course we want when we stop at time capital T to be in a state where the feature fi is high. So that value is large and it's multiplied by kappa which is how intensely you want the feature. These are reward-respecting subtasks for feature attainment, and there's a paper on this in AI Journal a year before last. So that's what I think the subproblem should be, and we then have to learn the solutions and learn the models of the solutions. The solutions are the options. These two things are relatively straightforward, but again they rely on us being able to do continual deep learning well. But if you grant me that prerequisite, then we can do it. So this is where we really get to the heart of OAK. Pay attention to the overall structure of the argument now. Basically we're going to do all these learning processes, and they're going to use features and also drive the creation of new candidate features. So we have features that carry over here. Once we have features that are interesting, we can create subproblems to achieve each one. Once we have those subproblems, we then solve them. If we have a thousand subproblems, we would solve them to produce a thousand options. Each one of those options will then be the foundation. We ask the question: if I did that option in this situation, what would happen? So that's the transition model. We're predicting the consequences of behaving in that way. This is different from solving the problem, which is finding a good way of behaving that hopefully achieves the feature. But many other consequences will happen. If I go downstairs to the kitchen, I may get to the kitchen, but I might also see my spouse or I might trip and fall down the stairs. Many things could happen, and I need to predict the possible consequences. So that's the transition models. Then once I have the transition models, I have to use them to plan, which improves my behavior and hopefully gets me highly adapted behavior. Now let's look at the same with a different picture. We're talking about perception giving me the features. So now I've got the state features. Then I solve those subproblems. I play with the world and I get the options. The options I predict the consequences of the options, and that gives me a transition model. The transition model through planning gives me improved values and policies. Then all of these later steps feed back and inform perception. They tell perception: in order to form this model I had to use some features and not use other features. In order to solve the problem I used some features and didn't use other features. So these processes inform the feature construction process, and they could say, "Oh, these features that you've offered me are useless and you should throw them out and get me more features, and these other features are very useful. Maybe they should be the basis of future subproblems that are posed and solved, and thus recursively improve in an open-ended way the abstractions of the agent."

我需要谈谈规划，因为它是关键步骤之一。规划是一个大话题。我会给它打上绿色勾，因为假设我们有模型，我有一个具体的规划方案。首先，为什么我们要规划，为什么我们要用这些大跨度的跳跃式步骤来规划？我们想要规划，因为世界在变化，因为世界在变化，我们的价值观也在变化。另一种说法是，把模型做对比把价值观做对更容易。在任何时间点，如果世界变化，价值观也会变化。比如，找到去洗手间的路非常重要，但身处洗手间的价值会变化。有时我想去洗手间，有时不想。但让我到达洗手间的选项模型可能是我想要保留的，即使我对现在是否想去那里的欲望发生了变化。

I need to say a little bit about planning because it's one of the key steps. Planning is a big topic. I'm going to give it the green check because assuming we have the models, I have a specific plan for planning. First, why do we plan and why do we want to plan with these large-scale jumpy steps? We want to plan because the world changes and because the world changes, our values change. Another way of saying it is it's easier to get the model right than the values right. At any point in time, if the world changes, the values change. Like finding my way to the restroom is really important, but the value for being in the restroom changes. Sometimes I want to go to the restroom, sometimes I don't. But the model of the option for getting me to the restroom may be something I want to retain even as I change my desires about whether I want to go there right now.

所以，状态的价值会变化，但世界的大部分动态保持不变。要为此做好准备，你必须做好准备。这有影响。但让我进入规划，给你一个粗略的想法。规划的想法是，它将通过价值迭代来完成，在这个过程中你改进你的价值函数，从而知道哪些状态是好的，哪些状态你想进入或不想进入，哪些特征你想获得或希望为真。这个表达式在经典价值迭代中——这里写的是经典价值迭代。经典价值迭代是针对离散状态的情况表达的，你可以有一个表格来存储每个状态的价值。我们首先需要处理这种情况，以理解这个想法。所以在价值迭代中，持续地，当你有空闲时间——思考的时间——你要做的是选择一个状态。如何选择是一个重要问题。然后你在那个状态执行一次备份。状态的备份会改变并改进该状态的估计价值。你对所有动作取最大值，所有你可能做的事情，然后看你期望获得的奖励。R hat 是你的期望奖励。这来自你的模型。在右上角，我们有模型。模型是什么？世界模型有两部分：R hat 和 P hat。模型接受一个状态和一个动作，猜测下一个状态的概率分布以及期望奖励。这就是一个原始模型所做的两件事。这不是一个高层模型，而是一个原始模型。价值迭代是为原始模型、低层模型定义的。所以你有期望奖励，然后你有下一个状态的期望价值。你求和每个下一个状态的概率和该下一个状态的价值，并稍微折扣。你坐在状态 s，看着你可能做的所有动作和可能产生的所有状态。然后你弄清楚可能发生什么，备份它，并改变状态 s 的价值。所有规划方法都有相同的想法。有些方法——价值迭代看起来就是这样。但即使是 A*、蒙特卡洛树搜索和模型预测控制，都涉及向前看、用模型观察后果、更新你对不同状态或不同选择的价值感知，然后据此做出选择的基本想法。这就是价值迭代。然后我们将其抽象化，以便能够进行跳跃，在世界中迈出大步。这是下一步。

So it's as if the values of the states change, but most of the world dynamics remain the same. To prepare for this, you have to get ready. This has implications. But let me go into planning to give you a rough idea of what I'm thinking of. The idea of planning is that it will be done by value iteration, in which you improve your value function so that you know which states are good, which states you want to be in or not, which features you want to obtain or want to be true. This expression in classic value iteration—what's written here—is classic value iteration. Classic value iteration is expressed for the case of discrete states, where you could have a table to store the value of each state. We need to do this case first to understand the idea. So in value iteration, continually, when you have spare time—time to think—what you do is select a state. How you do that is an important question. Then you perform a backup at that state. The backup of the state will change and improve the estimated value of that state. You take a max over all actions, all the things that you might do, and you look at the reward that you expect to get. R hat is your expected reward. This is from your model. Over here in the upper right, we have the model. What is the model? The model of the world is two things: R hat and P hat. The model takes a state and an action and guesses what will be the probability distribution over the next state and also what will be the expected reward. These are the two things a primitive model does. This is not a high-level model; it's a primitive model. Value iteration is defined for primitive models, low-level models. So you have the expected reward, and then you have the expected value of the next state. You sum over the probabilities of each next state and the values of that next state, discounting it a little. You're sitting at state s, looking at all the actions you might do and all the states that might result. Then you figure out what's likely to happen, back it up, and change the value of state s. All planning methods have the same idea within them. Some of them—value iteration looks exactly like this. But even things like A* and Monte Carlo tree search and model predictive control all involve the basic idea of looking ahead, seeing the consequences using your model, and updating your sense of the value of your different states or different choices, then making your choice accordingly. So this is value iteration. Then we make it abstract so that we can make jumps, large steps in the world. That's the next step.

生活是一步一步过的，但你在更高层次上规划生活。你计划去听讲座，你计划站起来，你计划去洗手间。所以这个模型——知识，你的转移模型，我称之为知识——是关于这些大规模动态的，其中你的动作或选择，即选项，是有目的的。我们的知识不依赖于单个动作；它依赖于诸如叫优步去机场、去冰岛旅行、去洗手间或捡起物体之类的事情。这些是基于选项知识的更大方式。选项模型——这是上一张幻灯片中重复的传统模型——我们将其推广到选项模型。我们只需将动作替换为选项。所以输入是我可能做的所有事情，这些扩展的做事方式。我得到的不是下一个状态，而是当选项停止时我将停止或终止的状态。不是一步奖励的期望奖励，而是从开始到终止时累积的期望奖励。有了这两个变化，价值迭代是一样的。我们对可能性进行相同的求和，得到奖励和停止时状态的期望价值。这是相同的想法。所以我们可以使用价值迭代在抽象层面进行规划。

Life is lived one step at a time, but you plan your life at a higher level. You plan to go to the talk, you plan to stand up, you plan to go to the restroom. So this model—the knowledge, your transition model, which I'm calling knowledge—is about these large-scale dynamics where your actions or your choices, which are options, are purposive. Our knowledge is not conditional on single actions; it's conditional on things like calling an Uber to get to the airport, traveling to Iceland, going to the restroom, or picking up an object. These are larger ways based in knowledge about the options. The option model—this is the conventional model repeated from the last slide—we generalize that to an option model. We just replace action with option. So the input is all things that I might do, these extended ways of doing. And I get, not the next state, but the state at which I will stop or terminate when the option stops. Instead of the expected reward on a one-step reward, I have the expected reward that's summed from when I started to when I terminated. With those two changes, the value iteration is the same. We have the same sum over possibilities, and we get the reward and the expected value of the state at stopping. It's the same idea. So we can do planning at the abstract level using value iteration.

另一件我需要做的事情是提供一些见解，说明我们如何将其推广以处理函数逼近，因为世界很大，我们没有 V(S)——我们没有世界每个状态的价值——但我们有由权重向量、参数确定的每个状态的价值。所以我们有一些近似。世界的状态将通过我们的观察和特征向量呈现给我们，然后这将与我们的参数一起给出一个近似。当然，模型也将变成参数化的。我们将预测过程中的奖励以及到终止状态的转移概率。这些也将以参数化方式表示。我们不会用表格做这种传统的事情，而是通过梯度过程更新权重，如下所示。括号中的内容被替换了。所以我们现在有基于模型和转移模型以及该模型参数的期望奖励，转移概率基于参数，然后我们有估计价值。这个备份价值类似于括号中的内容，并成为目标。所以我们把我们正在观察的状态 s 的估计价值移向我们的选择中备份价值的最大值。我们回到了单个原始动作来做函数逼近。这也将推广到完整的选项情况。

The other thing I need to do is provide some insight into how we generalize this to handle function approximation, because the world is big and we don't have a V of S—we don't have a value for each state of the world—but we have a value of each state determined by our weight vector, by our parameters. So we have some approximation. The state of the world will be presented to us in our observations and our feature vector, and then that will be involved with our parameters and give us an approximation. Of course, the model will also become parametric. We will predict the reward along the way and also the transition probabilities to the next states of termination. Those will also be parametrically represented. Instead of doing this conventional thing with the table, we're going to update the weights, as shown down here, by a gradient procedure. The thing in brackets is replaced. So we have the expected reward now based on a model and a transition model and a parameter of that model, and the transition probabilities are based on the parameter, and then we have the estimated values. This backed-up value is analogous to the thing in brackets, and that becomes the target. So we move the estimated value for the state we're looking at, s, towards the max of the backed-up value for our choices. We've gone back to single primitive actions to do the function approximation. This will also generalize to the full option case.

最后一步是承认一切都需要——我们需要保持关于一切质量的统计或元数据：转移模型的质量，因为模型将是近似的，我们必须学习它在哪里会给出可靠答案，在哪里不会。我们必须保持关于特征的统计，以便在生成新特征时，告知特征生成过程哪些特征是好的，哪些是坏的。就这样，我要结束了。

The very last step is to acknowledge that everything needs to be—we need to keep statistics or metadata on the qualities of everything: the quality of the transition models, because the model will be approximate and we have to learn where it will give me reliable answers and where it won't. We have to keep statistics on the features so that we can inform the feature generation process about when it's doing which features are good and which features are bad as it generates new features. With that, I'm just going to wrap up.

哦，我还有一个图要提醒大家，再给你们一个概念。Oak 是开放式的，因为它有发现循环。这是 Oak 雄心勃勃的新想法：在可能的抽象中保持开放，仅受算力限制。基本循环是：从状态特征开始；从这些特征构建子问题；从子问题求解产生选项；选项可以产生选项的模型，通过规划过程改进选项和策略。所有这些步骤都使用状态特征。箭头是单向的，但也有反向影响：我们告知状态特征哪些有用哪些没用。这涉及排序，然后导致一个新特征，我之前没意识到它重要，但大家都很常用。这个概念成为新子问题的基础，这就是它开放的原因，可能随时间获得越来越好的抽象，一切都最终与奖励相关。目标是通用、经验性、开放式的抽象。我希望你能看到这个愿景提供了一种实现所有这些的方法，尽管它有一些缺失的前提。Oak 有一个从运行时经验中培养超级智能的愿景。它拥有所有最重要的能力：可以行动、学习和规划；学习模型；形成子问题；这些抽象处理不同事物；以及选项。它可以用函数近似、部分可观测性、非平稳世界、随机世界来实现。它可以发现状态特征，从而发现子问题、选项和模型。这一切都反馈以激发更抽象的特征。这个良性、开放式的发现循环，同时架构是完全通用的。它没有任何领域特定组件，因此可扩展且具有潜在持久影响。非常感谢大家的关注。

Oh I have one more figure to remind you, to give you a sense of what it is one more time. Oak is open-ended because it has its cycle of discovery. This is the ambitious new idea of Oak: to be open-ended in its possible abstractions, limited only by compute. The basic cycle is: start with the state features; from those we construct subproblems; from subproblems we solve them to produce options; and the options can produce models of the options, which improve the options and the policies through planning processes. All of these steps use the state features. The arrows go in one direction, but there is influence the other way: we inform the state features which ones are useful and which are not. That involves ranking, and then that leads to a new feature that I didn't realize was important before, but everyone is making high use of it. That concept becomes the basis for a new subproblem, and that is what makes it open-ended, possibly getting better and better abstractions through time, with everything ultimately tied to reward. The quest is to be general, experiential, and have open-ended abstraction. I hope you can see this vision gives a way to achieve all of these things, even though it has some missing prerequisites. Oak has a vision of how to grow superintelligence from runtime experience. It has all the most important capabilities: it can act, learn, and plan; learn a model; form subproblems; these abstractions work with different things; and the options. It can do this with function approximation, partial observability, non-stationary worlds, stochastic worlds. It can discover state features and thereby discover the subproblems, options, and models. This all feeds back to motivate more abstract features. This virtuous, open-ended cycle of discovery, while the architecture is completely general. It doesn't have any domain-specific components and thus is scalable and of potentially lasting impact. Thank you very much for your attention.

非常感谢，Rich。很高兴您能来。当然，祝贺您最近获得图灵奖，感谢您对该领域的所有贡献，当然还有强化学习。希望您也能帮助我们解决这个稳健的持续深度学习问题。如果时间允许，我们有一些观众提问。这里有一个。

Thank you very much, Rich. It's wonderful to have you here. Of course, congratulations on your recent Turing Award and thank you for all your contributions to the field, and of course to reinforcement learning. Hopefully you can also help us with this robust continual deep learning as well. And we have a few questions from the audience if you have time. So we have one over here.

是的，查看您对 Oak 的要求列表，我注意到在您使用的黑猩猩例子中，为高排名特征创建子问题似乎可以消耗第二和第三状态，因为黑猩猩的机制中已经内置了奖励函数来享受摇摆。所以这是一种化学反应，说这是一个高排名特征，因为你的经验。生成状态函数来自系统本身在其具身化中如何从环境获得奖励。然后那个排名顺序成为奖励的好处，而不是你所说的偏向特征的教导 K。所以我的问题是：有没有办法通过假设子问题来自基于生成的乐趣的玩耍来跳过步骤一到四，也就是说，获得奖励的东西通过其构建方式调整来获得奖励？

Yes, looking through your list of requirements for Oak, I noticed that in the chimpanzee example you used, it seems that creating subproblems for highly ranked features can consume the second and third states because you already have a reward function built into the mechanism of the chimpanzee to enjoy the swinging. So it's a chemical reaction that says this is a highly ranked feature because of your experience. The generating state functions come from the formation of the system itself in its embodiment of how it gets rewards from the environment. Then that rank ordering becomes the benefit of the reward rather than a taught K to your point of biasing features. So my question is: is there a way to skip steps between one and four by assuming that subproblems come from play based on generated fun, to say that the thing getting the reward is tuned to get that reward by the function of how it's built?

你正确地关注了我们如何选择特征作为子问题的基础。其中一些可以来自我们预先的设计，比如我们的设计可能说亮光可能很重要，或者响亮的声音。也许我们的设计可以告诉我们，激活前庭系统的东西很重要。但很多事情并非如此。想想你在科学中的工作：你研究一个问题，意识到非常细微的差别，你仍然可能想关注它们并朝着它们努力。或者你下棋，有表面上细微的东西，比如兵的结构，然后你可以设定一个目标。或者你王的安全，你可以为此设定目标。我不认为你可以预期它们是可预测的。但它可能通过注意到这些细微的东西对回答你的一些问题很重要而产生，它们可能对回答你的基本奖励问题或回答其他子问题有用。一个很好的例子是婴儿，他非常意识到他的照顾者，比如他的母亲。也许母亲的气味是细微的东西，但通过与好结果的关联，它变得重要。或者他们的声音。还有，玩拨浪鼓：你移动并发出拨浪鼓的声音；它是新奇的，仅仅因为新奇，你可能想关注它并能够重现它。无论如何，这是一个非常有趣的问题，关于子问题如何被选择。谢谢你的问题。

You are rightly focusing on how we select features to become the basis of subproblems. Some of that can come from our prior design, like our design may say bright lights are likely to be important or loud sounds. Maybe our design can tell us that things that activate our vestibular system are important. But many things are not like that. Think of your work in science: you study a problem and become aware of very subtle distinctions, and you still might want to focus on them and work towards them. Or maybe you play chess and have things that are superficially subtle like the structure of your pawns, and then you can make a goal. Or the safety of your king, you can make goals for that. I don't think you can expect them to be anticipatable. But it could come about by just noticing that these subtle things are important for answering some of your questions, and they could be useful for answering your basic reward questions or for answering other subproblems. A good example is an infant who becomes very aware of its caregiver, say its mother. Maybe the smell of its mother is a subtle thing, but through association with good outcomes it becomes important. Or the sound of their voice. Also, playing with a rattle: you move and make the sound of a rattle; it's novel, and just because it's novel you might want to focus on it and be able to reproduce it. Anyway, it's a wonderful interesting question of how the subproblems become chosen. Thank you for your question.

嘿，我是 Ben。这次会议上有几个关于预测编码作为反向传播替代方案来训练深度神经网络的演讲。我知道有一些研究试图用这个来解决灾难性遗忘问题，使神经网络能够持续学习。所以我的问题是：你的直觉是持续学习问题可以用反向传播解决，还是认为我们需要回到 DC 或其他更生物或创新的方式来训练神经网络？

Hey, this is Ben. We've had a few talks at this conference on predictive coding as an alternative to backpropagation for training deep neural nets. I know there's some research trying to use this to solve the catastrophic forgetting problem and make neural nets able to continuously learn. So my question is: is your intuition that the continual learning problem can be solved using backpropagation, or do you think we'll need to go back to DC or some other more biological or innovative way of training neural nets?

谢谢 Ben。我不认为这是神经网络结构的问题。我认为这是简单反向传播的弱点问题。解决方案仍然会涉及梯度下降，但仅靠梯度下降是不够的。这是持续反向传播算法的教训，这是 Dohare 提出的新算法，在可塑性丧失论文中研究过。这是一个简单的事情。

Thank you Ben. I don't think it's a problem with neural networks, the structure of neural networks. I think it's a problem with the weakness of simple backpropagation. The solution will still involve gradient descent, but gradient descent alone is not enough. This is the lesson of the continual backpropagation algorithm, the new algorithm proposed by Dohare and studied in the loss of plasticity paper. It's a simple thing.

这和反向传播是一样的，只不过不是只在开始时用小的随机权重初始化，而是在过程中也这样做：一小部分使用不多的单元会被重新初始化为随机权重。所以这是一个非常简单的想法，一个简单的修改。实际上只是让反向传播更加持续，做更多它已经在做的事情。所以这是算法的一个小修改，却让它大大增强了持续学习的能力。所以我认为算法的空间很大，而我们探索得很少，会有很多可能的想法——小的或大的——来让它更有效地进行持续学习，这些想法仍然会涉及梯度。只是梯度。

It's just the same as backprop except instead of initializing with small random weights only at the beginning, you do it as you go along: a small fraction of the units that are not being used very much are reinitialized with random weights. So this is a very simple idea, a simple modification. It's actually just making backprop more continual, doing more of what it already does. And so it's a small modification of the algorithm which makes it vastly much more capable of continued learning. And so I think the space of algorithms is large and we have just done so little to explore it, and there will be small or maybe larger but anyway there's a wide range of possible ideas to make it more effective at continual learning, and they will involve still gradients. It's just the gradient.

预测编码也涉及梯度下降。你只是用梯度下降来学习不同的东西。我的意思是，你在一个神经元上做梯度下降来预测另一个神经元在局部做什么，但你不是像反向传播那样传播，你仍然在局部做梯度下降。这更像是 Bengio 的差异目标传播的方向，但更极端一点，对吧？所以是的，正如你所说，有很多方式来组织它。

Predictive coding also involves gradient descent. You're just learning it using it for different things. I mean you're doing gradient descent at one neuron to predict what some other neuron is doing locally, but so you're not propagating the same way as in backprop, but you're still doing gradient descent locally. It's more like in the direction of Bengio's difference target propagation but a little more extreme, right? So yeah, as you say, there's a lot of ways to organize it.

关键是你需要一个随机成分，需要一个搜索成分。梯度下降是确定性的，沿着梯度走。这还不够。你需要能产生多样性的东西。而用随机权重初始化正是产生多样性的方法。

The key thing is that you need a random component, you need a search component. Gradient descent is deterministic and follows the gradient. That's not enough. You need something that will generate variety. And this is what the initialization with random weights generates variety.

这是很有智慧的话。我同意随机成分将是关键要素之一。所以，谢谢。谢谢，Ben。后面还有另一个问题。

That's a good bit of wisdom. I would agree a random component is going to be one of the critical elements. So yeah, thanks for that. Thanks, Ben. And we have another question at the back of the room.

你好。我有一个问题，关于对世界的持续观察如何转化为新的可能行动，这个过程是如何发生的。比如一个孩子在玩耍，发现一个重的东西是锤子，可以用来敲东西，然后慢慢发现钉子和木板，你知道，用锤子把木板拼在一起是如何成为你的学习机制的一部分的？

Hello. I had a question about how continuous observations on the world get turned into new possible actions, how that sort of process occurs. Like if a child's playing around and they discover this heavy thing that's a hammer that they can pound on stuff with, and slowly they discover nails and boards, you know, how does using the hammer to put the boards together come into being your learning mechanism?

你说了“持续”，但“持续”有两种可能的含义。我想请你澄清一下你指的是哪一种。持续可以指一个数值连续取值，或者你可能指时间上的持续进行。

Now you said continuous and continuous has two possible meanings. I just want you to clarify which one you mean. Continuous could mean like it's a number that takes a continuum of values, or you might mean that it's ongoing in time.

是的，我指的是时间上的持续，只是基于观察创造你可以采取的新可能行动。你知道，是否有一个具体化过程，变成一种抽象，比如“这是一个我可以使用的工具”？工具的使用是如何被发现的，我想这就是我要问的。

Yeah, I'm referring to the ongoing in time, just creating new possible actions that you can take based on observations. You know, is there a reification that turns into an abstraction of here's a tool I can use? How does tool use get discovered is I guess what I'm asking.

嗯，这是个很大的问题。但即使是简单的工具，我认为它也暗示了如何做到。让我们想想。你可能会找到锤子，意识到可以用它来制造噪音。于是你变得擅长用锤子制造噪音。然后你可能……我的意思是，我要指出的是，钉钉子是一个相当高级的概念，一项高级技能，你不会期望它马上成功，但你可以用锤子制造噪音。然后你可以用锤子打碎东西。很可能为了学会钉钉子，你需要看别人做。如果你看到别人拿着钉子和锤子，做了件非常有趣的事——钉子钉进了木板，你知道，那是一个不寻常的观察，这可以成为基础，比如“我能用锤子把钉子钉进木板吗？”好的。然后你可以花很长时间练习。一旦你掌握了，你可能会有想法，可以用工具组装东西。我不知道，这有点含糊，但我认为有一个从检测有趣的新特征到想知道是否能重现它们的渐进过程。所以关键在于，你只是想能够锤东西，即使它不给你奖励。你想重现你看到别人做的事情，即使它不会立即带来奖励，但你学会了如何做，你也学会了如何把钉子钉进木板而不伤到自己，比如不砸到拇指。这就是你想要尊重奖励的意义所在。

Well, that's a big question. But even a simple tool, I would offer that this has a hint of how it's done. Let's see. How would you find maybe you find the hammer and you realize you could use the hammer to make a noise. So you could become skilled at making a noise with this hammer. And then you might... I mean what I'm pointing out is that hammering a nail is a pretty advanced concept, an advanced skill, and you wouldn't expect it to be successful right away, but you could use the hammer to make noise. And then you could use the hammer to break things. And probably in order to become skilled at hammering a nail in, you'll have to watch someone else do it. And if you watch someone else take a nail and a hammer and do this really interesting thing where the nail went into a board, you know, that's kind of an unusual observation, and that could be the basis then for you know, can I use the hammer to get a nail to go into a board? Okay. And then one could work on that for a long time. And then once you had that, you would maybe have the idea that you could assemble something by the tool. I don't know, that's a bit hand-wavy, but I think there is a progression from detecting interesting novel features and wondering if you can reproduce them. And so then, the key is that you just want to be able to hammer it even though it's not getting you reward. You want to reproduce something that you saw being done even though it doesn't get you any reward immediately, but you learn how to, and you also learn how to hammer the nail into the board without hurting yourself, for example, without hitting your thumb with the hammer. That's the sense in which you want to be reward-respecting.

谢谢。

Thanks.

谢谢。非常感谢。最后一个问题。

Thank you. Thank you very much. One last question.

你好。你认为一个新的智能体将如何发现并理解自身与世界的边界？比如什么是自身，什么是环境？

Hi. How do you think a new agent will go about discovering and understanding the boundary between itself and the world? Like what is itself and what's the environment?

嗯，我认为这种模糊性很有趣，因为边界在哪里？首先，我会在哪里画线？我会画在你的心智周围，你可能认为那是你的大脑周围。所以，特别是，你的身体比如手是环境的一部分。它是你不能完全控制的东西，而你有兴趣去控制。你知道，我们可以想想一个孩子最初不知道如何控制他的手或其他肢体。事实上，你可以看到一个小婴儿常常只是看着自己的手，弄清楚如何控制它们。一开始你可能会发现它们，比如‘嘿，这是我的手，我能看到它，现在我看不到了。’然后它说，‘嘿，我能让它回来吗？我能吗？’它不是说‘这是我的手。’而是说‘这是一个有趣的感觉。我能让它再次发生吗？我能把手抬到视野里吗？我能把拇指放回嘴里吗？这是一个有趣的物体。我好奇把它放进嘴里会是什么味道。’所以，边界在哪里？让我说点什么，因为对于没有长期做强化学习的人来说，奖励常常不清楚。奖励来自智能体外部，但它并不来自我们身体外部。对吧？奖励是在我们头脑内部计算的，实际上是在我们大脑内部，但它是大脑的一部分，不被视为智能体的一部分，因为智能体不允许仅仅把它的奖励设得很高。

Well, I think it's kind of interesting how that blurs, because where is the boundary? First of all, where would I draw it? I would draw it around your mind, which you might think of as around your brain. So, in particular, your body like your hand is part of the environment. It's something that you can't control completely and you're interested in controlling. And you know, we could think of how a child doesn't initially know how to control its hand or its other limbs. You could see a young infant, in fact, will often just look at their hands and figure out how they can control them. And first you can find them like, 'Hey, here's my hand and I can see that, now I can't see it.' And it says, 'Well, hey, can I get that? Can I?' It doesn't say 'here's my hand.' It says, 'Here's an interesting sensation. Can I get it to happen again? Can I bring my hand up to my field of view? Can I get my thumb back in my mouth? Here's an interesting object. I wonder how it would taste if I put it in my mouth.' So, where's the boundary? And let me say something because it's often unclear to folks who haven't been doing reinforcement learning for a long time about the reward. The reward comes from outside the agent but it doesn't come from outside our body. Right? The reward is computed inside our head, in fact inside our brain, but it's a part of the brain that is not considered part of the agent because the agent is not allowed to just set its reward to be high.

我认为下丘脑的某个部分计算奖励信号，从技术上讲，它位于智能体之外，就像身体位于智能体之外一样。当然，我现在必须提到扩展假说。当我们越来越熟悉控制某物（比如我们的身体）时，将其视为智能体的一部分几乎变得更有用。或者当我们开车时，我们开始认为我们在控制车轮，车轮就像我们的选择。这改变了抽象层次，而且这是一种有用的思考方式。但从形式上讲，我们希望智能体和环境之间有一条清晰的界限，并在引入更高级的抽象之前，对可用的原始动作有明确的定义。再次感谢。

There's a part of our hypothalamus, I think, that computes the reward signal, and that is technically outside the agent, just as the body is outside the agent. I have to, of course, refer now to the extended hypothesis. As we get more familiar with controlling something like our body, it becomes almost more useful to think of it as part of the agent. Or when driving a car, we start to think of the car as we're controlling the wheels, and the wheels are like our choices. This is changing the abstraction. And it is a useful way to think of it. But formally, we want to have a clear line between the agent and the environment and a clear definition of the primitive actions that are available before we introduce higher level abstractions. Thanks again.

太棒了。非常感谢你，Rich。解释得非常清楚。讨论很精彩。我不知道你是否能听到掌声，但我们真的很感谢你今天来到这里。非常感谢。

That's wonderful. Thank you very much, Rich. Such a clear explanation. Great discussion. I don't know if you can hear the applause, but we really appreciate you being here and joining us today. Thank you very much.

非常感谢。

Thank you very much.