弥合机器学习与机器人领域的泛化差距

请开始吧。需要我再介绍一次吗？好的，请。Deepak，你的姓是 Pathak 吗？是的，好的。我很荣幸介绍 Deepak Pathak。他是我们新教员之一。事实上，他是一位无需介绍的人。那么 Deepak，请开始吧。

Please take it away. Should I do it again? Yes, please. Okay, Deepak, can you put out your last name? Is it Pathak? Yes, okay. It is my pleasure to introduce Deepak Pathak. He's one of our new faculty members. The truth is, here is someone who needs no introduction. So Deepak, please take it away.

感谢这个临时即兴的介绍。在我开始演讲之前，我想提一件事。正如他所说，我今年八月开始在这里工作，去年春天做了求职演讲。从那以后，我在 CMU 做了不少演讲，所以你可能已经看过一些内容。但我保证，今天的演讲对每个人来说都有新东西，即使你看过之前所有的演讲。前半部分可能是你见过的，但随着演讲的深入，我会展示越来越多来自我们实验室的最新内容，有些甚至还没有公开。很多都是新的。好的，今天的演讲，我将尝试提出一些我一直在思考的想法，目标是改进机器学习中的泛化能力，主要应用于机器人学，也包括计算机视觉等。指导我研究的动机很大程度上受到人类的启发：我们如何能很好地泛化到从未训练过的未见事物？这种泛化能力正是我们的本质，是人类智能的精髓。让我们看看机器智能的现状。今天，我们有能够玩复杂围棋游戏、翻译语音、识别语言的系统，也许在不远的将来，我们会有能够自动驾驶的汽车。但与人类不同，这些系统并不那么智能。相反，这些系统是针对它们训练的任务而特定的。例如，玩围棋的系统与开车的系统完全不同，这些任务之间没有泛化。而作为人类，我们可以做所有这些事情。也许要求一个算法同时玩围棋和开车有点太遥远了，所以让我举一个更贴近的例子：监督学习，这是我们在机器学习中学习的最基本的东西。以图像分类任务为例，目标是给定一张图像，输出类别标签。我们训练这些模型的方式是向它们展示大量来自大数据集的图像，这些数据集有训练和测试准确率。现在，我们在这个问题上已经非常出色，至少在机器学习或相关社区中，这些数据集上的表现是超人类的，真的非常好。但让我们看看即使是最好的模型，它们泛化得如何？当我将这个模型应用于 YouTube 上的随机视频时，它完全无法泛化。它识别出眼镜蛇、蛇、鳐鱼、鱼、平台等。这个结果不是玩具结果；我使用的是这类任务中最好的模型之一，它在真实世界例子中完全失败，准确率降得很低。为什么会这样？如果你问一个机器学习的人，答案会是这并不令人惊讶，因为训练数据中的狗位于图像中心，条件良好，光照好，姿态好。所有这些条件在真实世界中都不成立。事实上，这就是问题所在。我想指出，在我们所有基于学习的方法中，我们都有一个基本假设：训练数据和测试数据来自相同的分布。但在真实世界中，这个假设经常被违反，每当这种情况发生时，我们的算法就无法泛化。因此，本次演讲的重点是如何构建能够在真实世界中适应而不失败泛化的算法。我认为这个问题在机器人学中更为严重，我们有各种机器人表演疯狂的跑酷技能，但将这些技能转移到仅仅是将盒子放在桌子上的简单任务上，在很多方面仍然是一个未解决的问题。原因是我们的真实世界在不断变化，我们希望将这些机器人部署在真实世界中，而这种泛化很难实现。所以在今天的演讲中，我将专注于如何弥合这一差距，并泛化到智能体训练范围之外。在进入方法之前，让我举一个例子。假设这里的任务是打结。这是一个机器人，它看到这个图像作为输入。目标是这个图像，我们的想法是希望有一个策略能够输出一系列动作来打这个结。这是一个非常直接的任务：输入、输出和动作，对吧？现在，我们今天如何在机器人学或任何领域的学习中解决这个问题？最常见的方法，或者我认为最成功的方法，是模仿学习，即人类手动控制机器人来尝试解决任务。通过这样做，我们为机器人收集数据，这些数据实际上是所有的扭矩和关节角度。正如你所看到的，这对专家来说非常繁琐，但主要问题是，对于每个我们希望机器人执行的新任务，都必须重复这个过程。所以这不是一个可扩展的方法。因此，在过去几年中，人们也一直在关注一种称为强化学习的方法。高层次的想法如下：不像需要专家来完成任务，我们让策略或方法自己发现如何解决任务。所以给定输入图像，有一个策略采取一系列动作，达到目标图像。这个策略的训练方式是在达到目标时给予奖励，所以有高奖励，策略被训练来最大化这个奖励。但如果你仔细想想，策略一开始是随机的，因为没有专家。在这种情况下，如果策略一开始是随机的，那么机器人仅仅通过随机行动打结的可能性有多大？这非常非常不可能。事实上，根据我的一些估计，在这种设置下，需要这么多交互才能打一次结，这完全低效，完全无法在机器人上部署。那么，为什么在过去几年中，社区（至少是机器学习社区）对强化学习如此狂热呢？这是因为你可能见过的这些结果。然而，这里的故事并不像我们研究强化学习那样辉煌，因为在那些能够训练智能体通过看游戏屏幕并按键盘按钮来玩游戏的场景中，它们并不依赖稀疏奖励，而是依赖游戏得分作为奖励信号。

Thank you for the last-minute impromptu introduction. So before I start my talk, let me mention one thing. As he mentioned, I began here this August, and I gave my job talk last year in spring. Since then, I have given quite a few talks at CMU, so you might have seen a bunch of the content. But I promise that in today's talk, there is something new for everyone, even if you have seen all the past talks. The first half might be what you have seen, but as we go towards the talk, I'll show more and more new content which is freshly out of our lab and not even public yet. Much of that stuff is new. Okay, so today's talk, I'll try to pitch some ideas that I've been thinking about, with the goal of improving generalization in machine learning, with applications primarily to robotics but also computer vision, etc. So much of the motivation that guides my research is inspired by humans: how well can we generalize to unseen things that we have not been trained on? This generalization is basically what makes us who we are; it's the essence of human intelligence. Let us see where we are with machine intelligence. Today we have these systems which can play the complex game of Go, translate speech, recognize language, and maybe we will have cars that can drive themselves not too far into the future. But unlike humans, these systems are not that intelligent. In contrast, these systems are specific to what they are trained for. For instance, we have a completely different system that plays Go than the one that drives a car, and there is no generalization across these tasks. As humans, we can do all of these things. Maybe it's a bit too far out an example to ask for an algorithm to play Go or drive a car simultaneously, so let me give you an example closer to home: supervised learning, the most basic thing we study in machine learning. Take the task of image classification, where the goal is, given an image, to output the class label. The way we train this is by showing these models a lot of these images from a big dataset, which has training and test accuracy. Now we have really excelled at this problem, at least in the machine learning or dependent community, and the performance on these datasets is superhuman, really, really good. But let us see even the best of these models: how well do they generalize? When I take this model and apply it to a random video from YouTube, this model completely fails to generalize. It finds things like cobra, snake, ray, fish, platform. This result is not a toy result; I'm using one of the best models available for this kind of task, and it is completely failing in real-world examples, with accuracy dropping very low. So why is this the case? If you ask a machine learning person, the answer will be it's not very surprising because the data it is trained on contains dogs at the center of the image in good condition, good lighting, good pose. All these conditions don't hold in the real world. In fact, that is the problem. I would like to argue that in all of our learning-based methods, we start with this fundamental assumption that our training data and test data come from the same distribution. But in the real world, this assumption gets violated all the time, and whenever that happens, our algorithms fail to generalize. So much of our focus in this talk is how to build algorithms that can actually adapt in the real world without failing to generalize. This problem, I would say, is even more severe in robotics, where we have all these robots performing crazy parkour skills, but transferring these skills to a merely simple task of putting a box on the table is still in many ways an unsolved problem. The reason is that our real world is continuously changing, and we want to deploy these robots in the real world, and this generalization is very hard to obtain. So in today's talk, I'll focus on how to bridge this gap and generalize beyond what the agents are trained on. Before I go into the methods, let me give you an example. Let's say the task here is to tie a knot. Here is a robot, and it sees this image as input. The goal is this image, and the idea is we want a policy that can output a series of actions to tie this knot. It's a very straightforward task: input, output, and actions, right? Now how do we solve this today in robotics or in learning in any domain? The most common approach, or I would say the most successful one, is imitation learning, where a human comes and controls the robot manually to try to solve the task. By doing this, we collect data for the robot, which is really all the torque and joint angles. As you can see, this is very tedious for the expert, but the main problem is this has to be repeated for every new task we might want the robot to do. So this is not a very scalable approach. Hence, people have also been focusing in the last several years on an approach called reinforcement learning. The high-level idea is as follows: unlike requiring experts for a task, we let the policy or the method discover how to solve the task itself. So given the input image, there is a policy which takes a series of actions that goes to the goal image. The way this policy is trained is by giving a reward whenever a goal is achieved, so you have a high reward, and the policy is trained to maximize this reward. But if you think about this, the policy in the beginning starts being random because there is no expert. In this scenario, if the policy starts being random, how likely is it that a robot will even end up tying a knot just by randomly acting? It's really, really unlikely. In fact, by some of my estimates, in this setup it will take this many interactions to tie the knot even once, which is completely inefficient and completely impractical to deploy on a robot. So why is there so much craze about reinforcement learning in the community, at least in the machine learning community, for the past several years? Well, it's because of these results that you might have seen. However, the story here is not as glorious as we study about reinforcement learning, because in these scenarios where people are able to train agents that can function by looking at the game screen and pressing keyboard buttons to play the game, they do not rely on sparse reward but on the score of the game as a reward signal.

基本上在每个时间步都引导智能体，所以这提供了非常密集的奖励，几乎覆盖轨迹的每一步。而且在某些方面，这比专家更难。在多个场景中，比如在导航任务中，目标是到达某个位置，许多方法通过中途给予“苹果”来塑造奖励函数。目标是到达某个地点，但这些“苹果”只是为了引导智能体，使其不会在寻找目标时完全迷失。但在现实世界中，没有“苹果”。现实世界中，奖励可能延迟数天、数月甚至数年，以至于很难回溯到当前状态。那么，这个孩子怎么知道她应该怎样行动才能在 20 年后成为飞行员？她是在优化某个奖励并一直反向传播到童年吗？其实并非如此。

Basically guides the agent at almost every time step, so this provides a very dense reward which is almost every step of the trajectory. And in some ways, it's much harder than an expert. In several settings, like even in tasks like navigation where the goal is to reach some target location, many approaches shape the reward function by giving these apples in between. The goal is to reach some location, but these apples are given just to guide the agent so that it doesn't get completely lost in finding the goal. But in the real world, there is no apples. In the real world, the reward could be delayed by days, months, or years, such that it's hard to project back to where you are. So how does this child over here know how should she act to become a pilot 20 years later? Does she optimize any reward and backprop all the way to her childhood? Well, not quite.

这正是发展心理学家艾莉森·高普尼克和劳拉·舒尔茨的观点：人类婴儿，尤其是在童年早期，甚至不受任何最终目标的驱动。他们完全由自己内在的、关于想在世界中做什么或想达成什么的概念所驱动。他们被一种称为内在动机或好奇心的东西所引导。例如，这个孩子在玩乐高，并不是为了 20 年后在卡内基梅隆大学获得博士学位而优化。他只是玩乐高，享受乐趣，仅此而已。事实上，这种行为不仅限于人类；在所有具有生物智能的物种中都能看到。例如，这里有一个乌鸦和鸡的例子。右边是新喀里多尼亚乌鸦，它们是地球上最聪明的鸟类之一。事实上，它们非常聪明，可以通过折叠铁丝从试管中取出食物。这些鸟以拥有很长的童年而闻名，大约有两年的童年期，期间它们没有目标地探索。另一方面，这些鸡是地球上最笨的动物之一。它们所做的只是啄食谷物，别无他事，而且它们几乎没有童年。从出生第一天起，它们就开始啄食谷物。事实上，这种鲜明的相关性在多个物种中都能看到。在 x 轴上，是断奶时间，即没有目标地探索的童年长度；在 y 轴上，是物种的某种智能指标。你可以看到，在几个灵长类物种中，它们的智能程度与童年期无目标探索的时间之间存在良好的相关性。

This is where developmental psychologists Alison Gopnik and Laura Schulz argue that human babies, particularly in the early days of childhood, are not even driven by any end goal of any sort. They are completely driven by their own intrinsic notion of what they want to do in the world or what they want to achieve. They are guided by something known as intrinsic motivation or curiosity. For instance, this kid is playing with Legos, not trying to optimize to get a PhD at CMU 20 years later. It is just playing with Legos, having fun, and that's it. And in fact, this kind of behavior is not just about humans; it is seen in every biologically intelligent species. For instance, here is an example of a crow and a chicken. On the right, these are New Caledonian crows, which are one of the smartest birds on the planet. In fact, they are so smart that they can pick up food from a test tube by folding a wire. And these birds are known to have a very long childhood, like almost two years of childhood, where they explore without goals. On the other hand, these chickens are one of the dumbest animals on the planet. All they do is peck for grains and nothing else, and they have almost no childhood at all. They start pecking for grains from day one when they are born. And in fact, this stark correlation is seen across several species. So on the x-axis, there is weaning time, which is length of childhood where they explore without goals, and on the y-axis, there is some notion of how intelligent the species is. And you can see there is a good correlation across several primate species as to how intelligent they are and how much time is spent in childhood without goals.

因此，这引出了我们的一个猜想：也许不给智能体设定目标，可以使其不过度拟合任务，从而在后续部署到现实世界时能够更好地泛化。这是我们过去从中汲取灵感的关键见解之一，并据此提出了一些模型。我会简要地快速介绍一下，然后更详细地讨论如何将其扩展到现实世界。

So this leads to one of our conjectures: maybe by not giving a goal to the agent, you are making it not overfit to the task, hence they are able to generalize better in the real world when deployed later. So this is one of the key insights which we have taken inspiration from in the past and proposed some models. I'll briefly cover this very quickly, and then I'll go into more details as to how to scale it to the real world.

现在，如果你想在计算机中实现这个猜想，或者真正将其作为机器人的算法来实现，第一个问题是：如果你没有任何目标或任务要解决，你该怎么做？我们能否以某种方式在智能体中实现这种无目标探索、好奇心、内在动机的概念？在机器学习领域，这个问题已经驱动人们研究了几十年。然而，大多数先前的方法都集中在尝试在某个最终目标的背景下鼓励探索。但我们感兴趣的是尝试构建能够在训练时无需给定任何目标就能探索的算法。这里的目标，我指的是外在目标。所以智能体仍然可以有自己的目标或任务，但在训练时没有外部任务强迫它去解决。我不会深入讲解我们是如何提出这个模型的，但我会给出一个非常高层次的概念。我们提出了一个称为“好奇心计算模型”的东西。我把好奇心打上引号，因为它只是一个指导性术语。我不是在谈论人类的好奇心或动物的好奇心。我所说的是这样一种概念：智能体被放入一个新环境中，它可以完全靠自己学习探索并获取尽可能多的有用信息，无需任何专家或人类的引导。

Now, if you want to implement this conjecture in computers, or really implement it as an algorithm on a robot, the first question is: what do you even do if you do not have any goal or any task to solve? Can we somehow implement this notion of goal-free exploration, curiosity, intrinsic motivation in the agents? Well, people have been driven by this question for several decades at this point in the machine learning community. However, most of these prior approaches have always focused on trying to encourage exploration in the context of some end goal. But what we are interested in is trying to build algorithms that can explore without any goal given at training time. And by goal here, I basically mean extrinsic goal. So the agent can still have its own goal or its own task, but there is no external task that you are forcing it to solve at training time. I will not go into how we propose this model, but I'll just give a very high-level idea as to what it looks like. So we have proposed something known as a computational model of curiosity. I'm putting curiosity in quotes because it's just used as a guiding term. I'm not talking about human curiosity or animal curiosity. What I'm talking about is this notion where an agent is dropped in a new environment and it can learn to explore and gain as much useful information as possible completely by itself, without requiring any expert or any human to guide it.

那么总体思路是什么？总体思路非常简单。一开始，当智能体被放入一个环境时，它只是四处走动，最初随机行动，并收集一些数据。从这些数据中，它训练一个模型，一个关于世界如何运作的模型。所以，如果我在这里，如果我这样做，接下来会发生什么？好的。然后我们使用这个模型来预测智能体行动的后果。比如，如果我把这个鼠标从当前位置向右推，它会从桌子上掉下来吗？我能预测到吗？每当智能体的预测与现实实际发生的情况不符时，我们就说智能体变得更加好奇了，并允许它进一步探索，回到同样的场景并收集更多数据。关键部分在于：随着智能体返回、变得更加好奇、收集更多数据，它的模型变得更好。而当模型变得更好时，它的误差会降低，它就不再好奇了。然后它必须找到其他让它好奇的东西。所以，仅仅通过这种好奇并试图寻找模型无法解释的事物的概念——它基本上是在寻求可解释性，比如尝试做那些能让你解释世界正在发生什么的事情，并找到那些场景——这基本上让智能体能够自主地持续学习，并在环境中发现新事物。

So what's the general idea? The general idea is really, really simple. In the beginning, when the agent is dropped in an environment, it just goes around and acts randomly in the very beginning, and it collects some data. From the data, it trains a model, a model of how the world works. So if I am here, if I do this, what will happen next? Okay. Then we use this model to make predictions about the consequences of the agent's actions. Like, what if I push this mouse from this position to the right? But it will fall off the table. Can I predict this? So whenever the agent's prediction does not match with what actually happens in reality, we say that the agent has become more curious, and we allow it to explore further and go back to the same scenario and collect more data. And here is the key part: as the agent goes back, gets more curious, collects more data, its model becomes better. And as its model becomes better, its error will become low, and it will no longer be curious anymore. And then it has to find something else to be curious about. So just by this notion of being curious and trying to find things which your model cannot explain—it's basically trying to seek explainability, like try to do things which allow you to explain what is happening in the world and try to find those scenarios—this basically allows the agent to learn continually on its own and discover new things in the environment.

我们在至少四年前开始了这项工作，三到四年前，我们已经证明这种想法可以应用于各种环境。事实上，我们几乎在所有人们展示深度强化学习结果的环境中都展示了结果，但我们做了一个改变：我们完全没有使用任何奖励。然后智能体可以学习导航、学习行走、学习玩所有这些游戏，而无需使用环境中的任何分数。它还可以解决一些连续控制任务。但在今天的演讲中，我想谈谈如何将这些想法从模拟环境扩展到现实世界的设置，因为模拟和游戏——无论它们多么有趣——本身并不能体现这些方法的最终效用。所以在这篇演讲中，我将讨论如何将这些好奇心、甚至是在训练时没有任何外部目标的学习想法扩展到真实的机器人场景中。

We began this work at least four years ago at this point, three or four years ago, and we have shown that this kind of idea can be applied across a large variety of environments. In fact, we have shown results on almost all the environments that people show results of deep reinforcement learning, but we made one change: we did not use any rewards at all. And then the agent can learn to navigate, it can learn to walk, it can learn to play all these games without using any score from the environment. And it can solve some continuous control tasks as well. But in today's talk, I want to talk about how can we take these ideas from simulation to real-world setups, because simulations and games do not capture—no matter how fun they are—they do not capture the ultimate utility of any of these approaches just by themselves. So in this talk, I talk about how can we scale these ideas of curiosity, even learning or learning without any external goal at training time, to real robotic scenarios.

我来总结一下我认为目前文献中缺失的关键要素，这些要素阻碍了它们在实际世界中的应用。首先，如何在真实世界中高效学习？在游戏或模拟中，你可以随意进行无数次操作，但在真实世界中，我们需要非常简单高效的算法。其次，在模拟中，动作空间通常是离散的，比如键盘按键：左移、右移、射击或拾取。但在真实世界中，尤其是对于机器人，动作空间非常重要——扭矩空间、关节角度空间以及动态任务。第三，如果我们希望学习在真实世界中持续进行，通过不断适应来超越训练时的泛化能力，我们需要算法能够适用于不同的机器人和硬件，而不仅仅是实验室里的一个机器人。这类似于计算机视觉，我们在 ImageNet 上训练模型，然后随着时间的推移在多个数据集和任务上进行微调。还有其他几个方向我会简要介绍，比如感知、动作、规划等。

Let me summarize what I think are the key things missing from this literature that are preventing it from being applicable to the real world. First, how do you learn efficiently in the real world? In games or simulation, you can take as many steps as you like, but in the real world, we need algorithms that are very simple and efficient. Second, in simulation, the action space is often discrete, like keyboard buttons: go left, go right, shoot, or pick up. But in the real world, especially for robots, the action space matters a lot—torque space, joint angle space, and dynamic tasks. Third, if we want learning to continue in the real world forever, adapting continually to generalize beyond training, we need algorithms that work across different robots and hardware, not just one lab robot. This is similar to computer vision, where we train a model on ImageNet and then fine-tune it on multiple datasets and tasks over time. There are several other directions I'll cover briefly, like perception, action, planning, and so on.

让我们深入探讨如何将这些好奇心驱动的方法扩展到真实世界。第一个问题是效率。在解决效率问题之前，我们先看看如果尝试将用于视频游戏的相同算法——在没有奖励的情况下探索环境并学习游戏——直接应用到机器人上会发生什么。这是我们的一次尝试：我们采用了同样的好奇心驱动学习算法，该算法构建一个模型，然后采取行动寻找违反该模型的例子。如果你看结果，它做了一件非常愚蠢的事：它只是撞击物体。你可以说它很好奇，但并不是有用的好奇。这是一个失败的例子。它在这个场景中完全不起作用。原因之一是它在模拟中有效是因为需要数百万个样本，这在快速模拟器中没问题，但对于真实机器人，可能需要一个博士生整个时间才能运行一次机器人。那么我们能做些什么呢？在我们的场景中，智能体带着好奇心探索，没有外部目标或外在目标。为什么这些方法如此低效？

Let's take a deep dive into scaling these curiosity-driven methods to the real world. The first question is about efficiency. Before solving this efficiency problem, let's see what happens if we try to transfer the same algorithm used in video games—exploring environments and learning to play without reward—directly to robots. Here is one of our attempts: we took the same curiosity-driven learning algorithm that builds a model and then acts to find examples that violate the model. If you look at the result, it does something very stupid: it just hits objects. You could say it's curious, but it's not usefully curious. This is a failure example. It didn't work at all in this scenario. One reason is that it works in simulation because it requires millions of samples, which is fine with a fast simulator, but for a real robot, it might take a PhD student's entire time just to run the robot once. So can we do something about it? In our scenario, agents explore with curiosity, with no external goal or extrinsic objective. Why are these methods so inefficient?

为了解决这个问题，让我们更详细地了解这些方法在数学上是如何工作的，以及我们如何改进它们。让我用数学描述好奇心驱动学习的思想。思路是：你想学习一个策略，它接收图像并输出动作。当你在真实世界中执行这些动作时，你会得到下一张图像。好奇心驱动学习的工作方式是：你训练一个世界模型——一个预测模型，它接收当前图像和动作，并预测接下来会发生什么。这个预测发生在某个特征空间中，比如从高维图像中。然后，为了引导智能体产生好奇心，你计算你的预测与真实情况的一致性。例如，你的预测是从当前图像和动作得到的 x_{t+1}。然后你在真实世界中行动以获得真实值 x_{t+1}。误差（预测减去真实值）成为你的好奇心奖励，你训练策略以最大化这个误差。但要计算这个误差，你实际上必须在真实世界中行动以获得真实值。环境是一个黑箱，所以很难直接优化这个目标，因为你无法通过它进行反向传播。这就是为什么大多数方法，包括我们的，都依赖于强化学习，通过试错来获得近似梯度。这在期望上是正确的目标，但在实践中，估计这个期望需要数百万个样本。这是这些算法如此低效的核心原因——它们基本上是尝试，如果不行就换别的，纯粹的试错。

To address this problem, let's go into more detail about how these approaches work mathematically and how we can improve them. Let me describe the idea of curiosity-driven learning mathematically. The idea is: you want to learn a policy that takes an image and outputs an action. When you execute those actions in the real world, you get the next image. The way curiosity-driven learning works is that you train a model of how the world works—a prediction model that takes the current image and action and predicts what will happen next. This prediction happens in some feature space, say from high-dimensional images. Then, to guide the agent to be curious, you compute how well your prediction aligns with the ground truth. For example, your prediction is x_{t+1} from the current image and action. Then you act in the real world to obtain the ground truth x_{t+1}. The error (prediction minus ground truth) becomes your curiosity reward, and you train the policy to maximize this error. But to compute this error, you actually have to act in the real world to get the ground truth. The environment is a black box, so it's hard to optimize this objective directly because you cannot backpropagate through it. That's why most approaches, including ours, rely on reinforcement learning, where you take approximate gradients through trial and error. This is the right objective in expectation, but in practice, estimating that expectation takes millions of samples. This is a core reason why these algorithms are so inefficient—they basically try things, and if it doesn't work, they try something else, pure trial and error.

当你的奖励是外在的，来自环境（如外部目标）时，这是合理的，因为你没有其他方法来解决这个问题。但在我们的案例中，奖励是内在的，来自智能体的好奇心。那么我们能改进这个公式吗？我们做的是：不构建一个模型，而是构建一组模型——联合训练以最小化它们各自的预测误差。这和之前一样，但现在我们可以将好奇心定义为这些模型输出之间的不一致性，而不是单个模型的预测误差。这些模型的方差成为好奇心奖励。要计算不一致性，你只需要模型的输出，不需要真实值。所以这个不一致性就是方差。这样，我们将奖励简化为一个可微的公式，避免了在真实世界中进行试错。

Now, this makes sense when your reward is extrinsic, coming from the environment like an external goal, because then you have no other way to solve the problem. But in our case, the reward is intrinsic, coming from the agent's curiosity. So can we improve this formulation? What we do is look at the model and instead of building one model, we build a bunch of models—trained jointly to minimize their respective prediction error. This is the same as before, but now we can define curiosity not as the prediction error of a single model, but as the disagreement among the outputs of these models. The variance of these models becomes the curiosity reward. To compute the disagreement, all you need is the output of the models, no ground truth. So this disagreement is just the variance. This way, we reduce the reward to a differentiable formulation, avoiding the need for trial and error in the real world.

模型的输出，也就是输出减去均值，再取均值，这是方差的基本公式。如果你注意到，这里没有 xt+1，因为它不需要在世界中行动。这完全不依赖于环境，所以它不是黑箱。你可以对它求导并找到答案。所以这是一个非常非常简单的过程：你学习模型，然后通过模型以可微分的方式学习策略。如果你真的理解我在说什么，你可能会说：‘这听起来类似于所有那些关于查询提交、主动学习、基于模型的控制和实验设计的工作。’确实，它把所有这些想法结合在了一起。所以一种说法是具身主动学习，但它的实现方式可扩展到高维图像，并且不需要状态空间。

The output of the models, so this is output minus mean and mean of that basic formula for variance. Now if you notice, there is no xt+1 here because it does not require acting in the world. This is completely not dependent on the environment, so it's not a black box. You can differentiate through it and find the answer. So it's a very, very simple procedure: you learn the model and then you learn the policy through the model in a differentiable manner. If you're really getting what I'm saying here, you might be saying, 'Okay, this sounds similar to all these works on query by committing, active learning, and model-based control, experiment design.' And indeed, it is kind of bringing all these ideas together. So one way to say it is embodied active learning, but it is done in a way that is scalable to high-dimensional images and does not require state space.

现在，我们能否将这个想法应用到真实的机器人上？这真的能给机器人带来增益吗？所以我们研究了抓取、推、拾取和推动等场景，这些是先前在机器人学习中的工作。但与这些使用随机探索的先前工作不同，我们将在机器人上执行好奇心驱动的探索。在我展示好奇心驱动探索的结果之前，让我先展示随机探索在这个设置中的表现。设置如下：有一个机器人，有几个物体，并且是位置控制。如果我在这个环境中随机行动，这就是机器人的表现。它甚至在很长很长一段时间内都不会碰到物体。你可能会想，如果是这样，为什么所有这些论文都能使用随机探索并获得良好的性能？关键在于，所有这些模型首先使用了两个技巧。第一个技巧是，他们把很多物体放在一个位置，这样如果智能体随机行动，它仍然可以偶然抓住、发现、拾取一些物体。然后他们有很多这样的机器人来并行化这个过程。在学术界很难负担这种设置，所以我们做的是：我们有一个机器人和几个物体，但我们可以改进很多来纠正数据。在这个案例中，我展示了探索的结果，但采用了样本高效的思路。仅仅在五到六个小时的好奇心训练和几千个样本中，机器人就开始有意义地行动，并通过拾取物体、移动物体来收集数据。注意，这里没有目标。所以这个机器人纯粹出于好奇心而行动，只是收集有趣的数据。

Now, can we apply this idea on a real robot? Does this actually give gain to be applicable to a robot? So we look at grasping, push, picking, and pushing kind of settings, which these prior works have done in robot learning. But in contrast to these prior works, which use random exploration, we will perform curiosity-driven exploration on the robot. Now before I show you the result of curiosity-driven exploration, let me show you how well random exploration works in this setting. So the setup is as follows: there is a robot, there are few objects, and it's position control. If I act randomly in this environment, this is how the robot performs. Now it doesn't even touch the object for a very, very long time. Now you might wonder, if that is the case, why and how come all these papers are able to use this random exploration and still get good performance? Well, the key is that all these models first of all perform two tricks. The first trick they perform is they put lots of objects at one location so that if the agent acts randomly, it can still grasp, spot, pick some of the objects by mistake. And then they have a lot of these robots so as to parallelize the process. Now it's hard to afford this kind of setup in academia, so what we do is we have a single robot and few objects, but we can improve a lot of them to correct the data. So in this case, I show a result from exploration but with the sample efficient idea. In just five to six hours of curiosity training in a few thousand samples, the robot starts to act meaningfully and collect data by picking objects, by moving them. Note that there is no goal. So this robot is purely acting out of curiosity and just collecting data which is interesting.

现在我们能做什么？到目前为止，这些智能体在自主探索，但我们可以利用这些探索数据来尝试解决某个最终任务。这个过程非常标准，就是规划。给定从输入图像到目标图像的任务，我做的唯一不同是，我从好奇心驱动的数据中学习这些模型。所以从高层次来看，过程如下：有一个智能体，它首先探索环境，收集数据，数据进入从高维输入（如图像）中学习的模型，这发生在训练时。在测试时，你给这个智能体多个目标去实现，它可以尝试实现这些目标。例如，我在演讲开头展示的例子——打结，这很难通过模仿或强化学习解决，我们通过这种探索和模型构建解决了它。所以在训练时，智能体探索、收集数据并构建模型。然后在测试时，我们要求这个机器人通过展示最终图像或中间的多张图像来执行这些任务中的任何一个。例如，假设这是一个打结的任务。所以一个人过来，在测试时只展示一次如何打结，而这个机器人从未在该示例上训练过，它可以通过只看人类做一次来尝试打结。注意，在这个设置中，我们并没有向机器人展示整个视频。我们进行了多次消融实验，要么只展示最后一张图像，要么展示中间的一两张图像，并通过 MPC 应用这个想法。

Now what can we do? So far these agents are exploring on their own, but we can take this exploration data and try to use it for solving some end task. And that procedure is very standard, which is planning. So given this task of going from input image to goal image, the only difference I'm making is I'm learning these models from curiosity-driven data. So at a very high level, the procedure looks as follows: there is an agent, it first explores the environment, collects data, data goes into models which are learned from high-dimensional input like images, and this happens at training time. At test time, you give this agent multiple goals to achieve, and it can try to achieve those goals. For instance, the example I showed at the very beginning of the talk, the example of tying a knot, which is very hard to solve by imitation or reinforcement learning, we solved it via this exploration and model building. So at training time, the agent explores, collects data, and builds a model. Then at test time, we ask this robot to perform any of these tasks by showing it the end image or multiple images as to how to achieve there. For instance, let's say it is a task of knot tying. So a human comes and at test time just shows how to tie a knot by showing it once, and this robot, without ever being trained on that example, can try to tie a knot by just looking at a human doing it once. Now note that in this setup, we do not show the whole video to the robot. We have multiple ablations where we either only show the last image or one or two images in between and apply this idea by MPC.

这种框架并不局限于操作；你可以将其应用于导航。同样的想法：在训练时，机器人去探索环境，收集有趣的数据，然后在测试时，你构建模型并使用这些模型来规划最终目标。所以假设给定当前图像，你要求机器人去往这个目标图像，然后它可以——这是一个未见过的环境，没有地图——它可以使用模型来规划整个图像。注意，至少对于导航，你可以尝试通过移位匹配或 SLAM 等各种其他方法硬编码这个设置。但很酷的一点是，我在这里提到这个结果的原因是，这个机器人从未为这个任务训练过，甚至从未尝试解决这个任务，这种行为纯粹是通过训练时的探索或好奇心涌现出来的。

Now this kind of framework is not specific to manipulation; you can apply it to navigation. The same idea: at training time, the robot goes and explores the environment, collects data which is interesting, and then at test time you build the model and use these models to plan for an end goal. So let's say given this current image, you will ask the robot to go to this goal image, and then it can—this is an unseen environment, no map—and it can use the model to plan for this whole image. Now note that at least for navigation, you can try to hardcode this setup via shift matching or SLAM and various other approaches. But the cool part, because of which I mentioned the result here, is that this robot was never trained for this task or never even attempted to solve this task, and this behavior is purely emergent by exploration or curiosity at training time.

有时探索世界的一切可能非常困难，你可能希望通过一些人类演示或示例来引导智能体。所以我们也有一些工作，通过向机器人展示人类从第三人称视角执行任务的视频来迁移这些想法，我们可以根据这些数据来调节探索或模型，并通过仅从第三人称视角观察人来解决这些任务。机器人看到人试图执行将东西倒入杯子的任务，它尝试从自己的视角去做，并且你也可以完成这个任务。现在你一定会注意到，这里的轨迹非常非常不稳定。机器人能够做一些事情，但非常非常不稳定。这很好地引出了下一部分：这些机器人的动作空间是什么？当我们谈到游戏场景中机器人的探索和好奇心时，这些基本上只是键盘上的按钮。如果你看看强化学习中的所有这些论文，它们做的是力矩控制、位置控制。但如果你真的想将这些想法迁移到现实世界，正确的动作空间是什么？

Sometimes it might be very hard to explore everything in the world, and you might want to guide the agent via some human demonstrations or examples. So we also have some works where we transfer these ideas by showing the robot a video of a human doing some task from a third person, and we can condition the exploration or the models on that data and solve those tasks by just looking at a person from a third camera point of view. The robot sees the person trying to do some task of pouring things into a cup, and it tries to do it from its own perspective, and you can do the task as well. Now one thing you must be noticing here is that the trajectories here are really, really shaky. The robot is able to do something, but it is really, really shaky. And this is a good segue into the next part: what is the action space of these robots? Now when we talked about this exploration and curiosity for robots in the game scenarios, these are just basically buttons on the keyboard. And if you look at all these papers in reinforcement learning, they do torque control, position control. But what is the right action space if you really want to transfer these ideas to the real world?

那么让我们看看今天大多数机器人学习中人们在做什么。这里我展示了过去几年机器人学习中的一些关键论文的例子。有一个不断增长的社区，有很多人在这个领域工作，想法是将深度学习等的成功与机器人场景结合起来。如果你从表面上看所有这些例子，它们可能看起来很复杂：哦，是的，它们有多个机器人，有复杂的物体。但这里的诀窍是什么？

So let us see what people do today in most of robot learning. Here I am showing a bunch of examples of kind of the key papers in robot learning in the past several years. And there is a growing community, and there are so many people working in this area where the idea is to combine the success of deep learning etc. with robotic scenarios. Now if you look at all these examples from the surface, they might look complex: oh yeah, they have multiple robots, they have complex objects. But what is the catch here?

所有这些图像中的共同点：如果你仔细想想，你会发现一件事——所有这些任务基本上都是手工设计的，环境是准静态的。一切都是准静态的。这意味着什么？这意味着机器人可以在每个时间步花很长时间来行动。比如，它可以拿起一个东西，然后思考 10 分钟，再做别的事。这种运动非常非常机械。当你试图将这些系统真正扩展到现实世界时，这就是问题之一，因为现实世界中的事情可能非常动态，比如扔一个物体、捡起一个物体或接住一个物体。

The common part in all these images: if you think about it for long enough, you will notice one thing: all of these tasks are basically handcrafted in a way such that the environment is quasi-static. Everything is quasi-static. What does that mean? It means the robot can just take forever to act at each time step. Like, it can pick up something, then think for 10 minutes, and then do something else. This motion is very, very robotic. And this is one of the problems when you are trying to really scale these systems to the real world, where things can be very dynamic, like throwing an object, or picking an object, or catching an object.

我们以接球任务为例。假设有一个机器人试图接住一个球。这个任务非常动态。机器人必须快速而流畅地移动以拦截球，并直接在轨迹空间中进行推理。一个在每个时间步独立地基于关节角度或扭矩行动的机器人，无法高效地解决这个任务。此外，由于这是一个物理系统，它不再是电子游戏，你无法让一个智能体或马里奥随意跳跃。与游戏不同，这些机器人需要考虑自身身体的动量和力，以及它试图抓住的其他物体的重量。因此，在动作空间或策略中，真正需要的是联合推理机器人的运动学和动力学。这与当前机器人学习的方法形成对比，这些方法基本上在每个时间步单独推理。我已经提到过，因为它们以原始动作（如扭矩、关节角度）行动，这些机器人的运动非常非常机械。你看，它们以非常离散的方式移动。

Let's take this task of catching. Let's say there is a robot trying to catch a ball. Now, this task is very dynamic. The robot has to really move swiftly and smoothly to intercept the ball and to reason directly in the trajectory space. A robot which is acting about joint angles or torque at each time step independently cannot really solve this task very efficiently. Furthermore, since this is a physical system, it's no longer a video game where you can just have an agent or Mario just jump around and do something. Unlike games, these robots need to consider momentum and forces about its own body and the weights of the other objects that it's trying to catch. So what you really need in an action space or the policies is to reason jointly about kinematics plus dynamics of these robots. This is in contrast to current methods in robot learning, which basically reasoned individually at each time step. I already mentioned that because they are acting in these raw actions like torque, joint angles, the motion of these robots is very, very robotic. See, like they move in a very discrete fashion.

所以问题是：我们能否构建这些直接在轨迹空间中进行推理的策略？这是我们非常感兴趣的领域之一，我们有一项即将发表的工作，由我的学生 Shikhar 领导，称为神经动态策略。这里的想法是，我们希望构建直接在轨迹空间中推理的策略。我们从中汲取灵感的是自然界中定义运动的方式，最常见的一种是使用动力系统，这是一对二阶微分方程，用于描述自然界的各种现象，如流体、热、行星绕太阳的运动。所以关键思想是：我们能否用描述行星绕太阳运动的相同公式来描述机器人应该如何移动？事实上，人们已经考虑过这个公式，因为这是描述现实世界宏观物理的一个非常自然的公式。有很多关于定义这些运动基元的工作，从 Panchal 先生的论文开始，它定义了这种二阶微分方程，并已应用于所有这些酷炫的任务：打乒乓球、翻煎饼、写字。注意：所有这些任务都非常动态，不像我一开始展示的拾取、放置和推动任务。在这里，机器人必须在每个时间步非常迅速地行动，对吧？

So the question is: can we build these policies that directly reason in the trajectory space? This is one of the areas we are really interested in, and we have an upcoming work in this new reps led by one of my students, Shikhar, which is called Neural Dynamic Policies. And here, the idea is that we want to build policies that directly reason in the trajectory space. And here we take inspiration from how we define motion in the natural world, which is one of the most common ways: using dynamical systems, which is a pair of second-order differential equations used to describe various phenomena in nature, like fluids, heat, motion of planets around the sun. So the key idea is: can we use the same formulation that describes how planets revolve around the sun to also describe how robots should move? And in fact, people have thought about this formulation because this is a very, very natural formulation to describe real-world macrophysics. And there is a lot of work on defining these movement primitives, beginning from this paper from Mr. Panchal, which defined this second-order differential equation, and have been applied to all these cool tasks of table tennis playing, pancake flipping, writing. And note: all of these are very dynamic tasks, unlike the tasks I showed in the beginning where there is pick up and place and pushing. Over here, the robot has to act in a very prompt manner at each time step, right?

所以这些来自控制领域的 DMP 或基于运动基元的方法可以很好地处理动态任务，并且它们可以在轨迹空间中进行推理，因为它们在微分方程空间中进行推理。但它们也有一些缺陷。例如，它们需要大量的调参。我相信你们中至少有些人用过这些方法：DMP、RBF。训练或调优它们并非易事。它真的需要一个全职人员工作数月才能让一个 DMP 工作。此外，它们需要密集的监督，比如演示或其他东西，并且很难将它们扩展到高维输入。相比之下，这些深度学习方法——无论我怎么批评它们——它们有一些优势。比如，它们可以很好地扩展到高维输入，如图像。图像，比如说状态并没有太大不同。而且它们可以从弱监督中学习，比如在最后给出的奖励，并不总是需要专家数据。但同样，原因是时间步，所以它们不是动态的。是的，它们不是动态的，并且它们不在轨迹空间中进行推理。

So these DMP or motion primitive based methods from control can handle dynamic tasks really well, and they can reason with trajectory space because they are reasoning in this differential equation space. But they also have some flaws. For instance, they require a lot of tuning. I'm sure some of you at least have worked with these methods: DMPs, RBFs. It's not an easy job to train them or tune them. It really requires a full-time person working for months to get one DMP thing working. And furthermore, they require dense supervision, like demonstrations or something else, and it's very hard to scale them to high-dimensional input. In contrast, these deep learning methods—no matter how much I diss them—they have some advantages. Like, they can scale to high-dimensional input like images really, really well. Images, let's say state is not that much different. And they can learn from weak supervision, like rewards which are given at the very end, and does not always require expert data. But again, the reason is time step, so they're not dynamic. And yes, they are not dynamic, and they do not reason in the trajectory space.

那么我们能否弥合这两种范式之间的差距，从根本上重新表述智能体的动作空间？通常在深度强化学习或模仿学习设置中，我们有图像输入（图像可以是状态或图像），有一个策略，有一个动作进入环境并返回奖励。现在，与其使用一个黑箱神经网络策略，我们能否以学习的方式或微分的方式用一个动力系统来替代它？这基本上就是我们在本工作中提出的，称为神经动态策略，其中网络直接输出动力系统的参数。它有两个主要部分：一个是目标——你希望轨迹走向哪里？所以现在这个模型直接在轨迹空间中行动，你需要推理去哪里，即目标位置，以及如何到达那里——比如我应该以什么形状移动？像这样还是那样？什么形状？我应该遵循什么轨迹？这些轨迹基本上由径向基函数建模，就像在 DMP 中一样。然后网络输出这些参数，你积分计算轨迹，然后将这个轨迹应用于环境以执行动作，然后你可以获得奖励和状态，并端到端地训练。所以我们在本文中所做的，基本上就是使这个过程端到端，并通过奖励或模仿在你喜欢的任何设置中进行训练，并且也从图像中训练。

So can we bridge the gap between these two paradigms to basically reformulate the action space of an agent? So usually what we have in deep RL or imitation learning setups is that we have an image input (image can be state or image), there is a policy, there is an action which goes in the environment and gives a reward back. Now, instead of having a black box neural network policy, can we replace this in a learned manner or in a differential manner with a dynamical system? So this is what basically we propose in this work, which is called Neural Dynamic Policies, where the network outputs the parameters of a dynamical system directly. So it has two main things: one is the goal—where do you want the trajectory to go? Like, so now this model is directly acting in the trajectory space, so you want to reason about where to go, which is the goal position, and you want to reason about how to go there—like in what shape should I go? Like this or like that? What shape? What is the trajectory that I should follow? And these trajectories are basically modeled by a radial basis function as in DMPs. And then the network outputs these parameters, you integrate it to compute the trajectory, and this trajectory is then applied on the environment to act, and then you can get a reward and state, and train it end to end. So what we do in this paper is basically make this procedure end to end and train it via rewards or imitation in any setup that you would like, and also from images.

所以从高层次来看，直觉如下。假设有一个起始位置和一个目标位置。现在，如果我通过强化学习或模仿学习训练我的策略，真实机器人上的策略看起来有点像这样：非常非常生硬，就像我之前展示的结果一样。它完成了任务，但非常生硬，不太实用。但如果你用这种动力学结构来训练，你会得到一个非常平滑的轨迹，它尊重机器人的动力学和运动学。此外，这个设置非常容易集成到任何现有算法中。就像你放入 10 行代码，它就能直接工作。所以为了节省时间，我快速浏览一些结果。我们在强化学习和模仿学习中展示了结果，当任务动态时，我们获得了一些好的收益。所以是的，它有效，更好，等等，轨迹看起来非常平滑。目前的结果是在仿真中，但就在我说话的时候，我们正在努力将其转移到现实世界。

So at a very high level, the intuition is as follows. Let's say there is a starting position and there is some goal position. Now, if I train my policies via RL or imitation learning, the policy on a real robot somewhat looks like this: very, very jerky, as I showed in the results before. It does the task, but it's very, very jerky and not very practical. But if you train it with this structure of dynamics in the system, you get a very smooth trajectory which respects the dynamics as well as kinematics of the robot. And furthermore, this setup is very easy to integrate in any existing algorithm. It's just like 10 lines of code that you can put in, and it just works out of the box. So in the interest of time, I just quickly go over some of the results. We show results in reinforcement learning as well as imitation learning, and we should get some good gain when the tasks are dynamic. So yeah, it just works, it's better, blah blah blah, and the trajectories look very smooth. The results right now are in simulation, but as I speak, we are working on transferring it to the real world.

到目前为止，我讨论了动作空间中的泛化，即对新事物的泛化。但最后一点：跨机器人泛化呢？我们能否不仅考虑一个智能体通过好奇心进行泛化并迁移到不同任务（如打结或把绳子变成不同形状），还能将这些策略跨机器人迁移？因为每个机器人都是不同的。事实上，没有人比我们机器人研究所的人更了解这一点，因为我们的每个实验室、每位教授都有不同类型的机器人，而且每个人都是从零开始训练自己的机器人。正如我们在机器人学习的方法中所看到的，即使对机器人进行微小的改动，也很难进行泛化。例如，如果我拿这个机器人并拆掉一条腿，它完全无法泛化。之前有一些尝试试图解决这个问题，但它们允许的机器人变化非常有限。

So far I talked about generalization in action space to new things. But what about this last bit: generalization across robots? Can we also think of generalization not only for one agent doing curiosity and transferring to different tasks like tying a knot or making rope into different shapes, but also transfer these policies across robots? Because every robot is different. In fact, nobody would understand this better than we people at the Robotics Institute, because all of our labs, every professor has a different kind of robot, and everyone trains their robot completely from scratch. As we have seen in learning approaches for robotics, it's really hard to generalize them even for minor changes in the robot. For example, if I take this robot and remove a leg, it just completely fails to generalize. There are prior attempts that have tried to address this problem, but they are very limited in what variations you can allow in the robot.

这是我们刚刚在 ICML 上发表的一项工作，我们的目标是训练一个策略，不是针对一个或两个机器人或一堆机器人，而是在测试时泛化到一堆新机器人。这同样非常具有挑战性，因为你有不同的动作空间，每个机器人需要不同的行为来完成任务。我不会过多地介绍方法的细节，但总体思路如下：我们不是在整个机器人层面训练策略，而是为每个电机训练策略，并尽可能地在所有机器人之间共享它们。由于你是在所有电机上训练策略，你确实希望这些策略以某种方式相互通信。我们允许这些机器人之间通过消息传递进行通信。因此，每个电机都有自己的策略，这些策略可以通过消息传递相互通信，并且这种消息传递是随时间学习得到的。

This is one of the works we just published at ICML, where our goal is to train a policy not across one or two robots or a bunch of these robots, and then at test time generalize to a bunch of new robots. Again, this is very challenging because you have different action spaces and every robot requires different behavior to solve the task. I won't go into the detail of the approach too much, but the general idea is as follows: instead of training a policy at the level of the full body of the robot, we want to train policies for each motor and share them across all the robots as much as possible. Since you are training the policy across all the motors of the robot, you do want these policies to talk to each other in some manner. We allow some ways to message pass or communicate between these robots. So each motor has a policy of its own, and these policies can talk to each other via message passing, and this message passing is learned over time.

让我快速展示一下这种方法的一些结果。通过跨电机共享策略并学习这些消息，我们能够训练出首先可以在多种机器人上学习和运行的策略。例如，你可以看到，如果一个机器人有两条腿，同一个策略有效；如果它有一条腿，同一个策略会跳跃。在不同的机器人上，如果你有一个单肢，它就会跳跃；否则，它会表现出不同的步态。我认为，从控制的角度来看，这是一个令人印象深刻的结果，因为一个单一的控制器正在控制 50 个这样的机器人，每个机器人至少有 10 个电机，所以这个控制器同时在学习至少 500 维的控制。当然，它不完全是 500 维，因为这些是不同的机器人，但一个单一的策略在这些场景中控制着非常高的动作空间。这些机器人具有非常不同的形状，对于每种不同的形状，它表现出不同的行为。事实上，在测试时，我可以直接把这个策略迁移到非常不同形状的机器人上，同一个控制器，之前是在非常不同的东西上训练的，能够在一定程度上迁移到所有这些新机器人上。它并不完美——它不像之前那样运行——但它做得相当不错。

Let me quickly show some results with this kind of approach. By just sharing the policies across motors and learning these messages, we are able to train policies that can first of all learn and function across a diverse variety of robots. For example, you can see if a robot has two legs, the same policy works; if it has one leg, the same policy hops. Across different robots, if you have a single limb, it just hops; otherwise, it shows a different gait. This, I believe, is an impressive result from a control point of view, because a single controller is controlling 50 of these robots, each robot has at least 10 motors, so it's at least a 500-dimensional control that this controller is learning simultaneously. Of course, it's not fully 500 dimensions because these are different robots, but a single policy is controlling a very high action space in these scenarios. These robots have very different shapes, and for each different shape, it shows a different behavior. In fact, at test time, I can just take this policy and directly transfer it to very different robots of very different shapes, and the same controller, which was trained on something very different like I showed before, is able to transfer to some extent to all these new robots. It does not excel—it does not run as it should like before—but it does a very decent job.

例如，如果你看这个机器人，它在测试时发现了一种新的步态。如果你给它非常重的手，它在训练时从未见过类似的东西，但它仍然可以尝试用膝盖向前移动。它有四条腿，它在做别的事情。但无论如何，正如你所看到的，这是一个很好的微调初始化，用于在测试时迁移到新机器人。同样，这些是带有物体的机器人，训练时从未见过，但它仍然可以运行并保持平衡。我们更疯狂了一些，尝试了非常不同的东西。这里有一个机器人，每个肢体的重量都非常高，但它只是学会了摔倒，因为它什么也做不了。最后一个没有腿，所以它试图用手，但完全无法完成任务。

For instance, if you look at this robot, it discovers a new gait at test time. If you give it very heavy hands, it has never seen something like this at training time, but it can still try to use its knees to move forward. It has four legs, it is doing something else. But anyway, as you can see, this is a good initialization for fine-tuning as a transfer to a new robot at testing. Again, these are robots with objects, nothing was seen at training time, but it can still function and balance itself. We went a bit more crazy and really tried very different things. Over here, there's a robot whose weight at each limb is very high, but it just learns to fall over because it just cannot do anything. This last one has no legs, so it tries to use its hands but completely fails to solve the task.

好的，我将跳过这一部分，继续讨论一些想法：是否有可能将这些机器人迁移的想法扩展到现实世界？事实上，我想说在现实世界中，我们已经有很多关于模块化机器人的工作。卡内基梅隆大学的 Howie Choset，我没有展示图片，因为你们都很熟悉。除此之外，宾夕法尼亚大学的 Mark Yim，麻省理工学院的 Daniela Rus，麻省理工学院的宇宙——这些人已经有模块化的机器人。但我们在这里展示的结果更多是从软件的角度来看。在这种情况下，机器人是由人手动控制的，而在我们的案例中，策略是自动学习来控制这些机器人的。我们在这方面的一些持续尝试实际上也在利用 Howie 在 Facebook 的博士后工作中的雏菊机器人。这个机器人在所有位置都有共享的电机。现在，该方法并不要求电机共享，因为我展示的结果是跨人形步行者的，它们是非常不同的电机，而且效果还不错。但在这些场景中，如果你有共享的电机，会容易得多，所以我们实际上可以让其中一些机器人工作。现在，这些方法不像模拟中那样完全模块化，但它们在分层系统方面是模块化的。

Okay, so I'll skip this part and go and cover some of the ideas: is it even possible to scale these robot transfer ideas to the real world? In fact, I would say in the real world we already have a lot of work on modular robots. Howie Choset at CMU, I didn't show a figure because you're all familiar with that. In addition to that, Mark Yim at UPenn, Daniela Rus at MIT, so the universe at MIT—these people already have robots that are modular. But what the results we are showing here are more from a software perspective. In this case, the robot is controlled by a person manually, and over here the policy in our case is learned automatically to control these robots. Some of our ongoing attempts in this direction are also actually leveraging this daisy robot from Howie's postdoc work at Facebook. This robot has shared motors at all locations. Now, it's not a requirement for the method for the motor to be shared, because the results I showed were across humanoid walker which are very different motors, and it still did okay. But it's much easier if you have the shared motors in these scenarios, so we can actually make some of these robots work. Now, these approaches are not completely modular as in the simulation, but they are modular in terms of hierarchical system.

好的，这就是跨机器人的泛化。现在，在我结束演讲之前，让我谈谈一些我非常兴奋的正在进行的方向，它们还没有公开，所以你可能有很多问题。我没有展示很多细节，所以之后你可以通过电子邮件问我一些想法。现在，如果你真的想将这些系统扩展到真实机器人，这三个问题并不是唯一的障碍；还有很多其他问题。一件事是，系统中的很多学习都是在模拟中进行的。我是否必须在现实世界中从头学习？我能否轻松地从模拟迁移到现实？现在，有很多关于通过域随机化等方法进行迁移学习的工作，但每个人都知道，这些方法大多依赖于幕后的工程。因此，一个人必须真正构建一个可迁移到现实世界的模拟。我们正在研究的一项工作，刚刚提交，是如何将工程从这种模拟到现实的迁移中去除。我们称之为自动调优的模拟到现实迁移。例如，这里有一个例子：你在模拟中通过左侧的方法训练这个机器人，并带有所有这些变化。当你尝试在真实机器人上运行时……

Okay, so this was generalization across robots. Now, before I end my talk, let me talk about some of the ongoing directions that I'm really excited about, and they are not public yet, so you might have many questions. I'm not showing many details, so feel free to ask me later via email about some of these ideas. Now, if you really want to scale these systems to real robots, these three questions are not the only hurdles on the way; there are many others. One thing is that a lot of the learning in the systems was happening in simulation. Do I have to learn from scratch in the real world? Can I transfer from sim to real easily? Now, there is a lot of work in learning in trying to transfer via domain randomization and other things, but everyone knows that most of these approaches basically rely on engineering behind the scenes. So a person has to really construct a simulation which is transferable to the real world. One of the works we are working on, which is just in submission, is how can we take the engineering out of this sim-to-real transfer. So we call this auto-tuning sim-to-real transfer. For instance, here is an example: you train this robot in simulation via something on the left with all these variations. When you try this on the real robot...

在机器人上，除非你精心设计系统，否则它无法开箱即用。但通过我们的方法，我们仅通过观察图像就能自动随时间调整模拟器，无需额外标签。随着训练的进行，模拟器自动接近真实世界，策略也迁移到真实场景。例如，在模拟器训练后期，它开始无需真实世界训练就能迁移，完全适应真实世界数据。同样，对于打开柜子这个任务，模拟器随时间调整并迁移到真实场景。

On the robot it does not succeed out of the box unless you really engineer the system. But with our approach, we automatically adapt the simulator over time by only looking at the images, no extra labels. As the training goes, the simulator becomes close to the real world automatically, and the policy is transferred to the real scenario. For instance, this was towards the end of simulator training; it starts to transfer without any training in the real world. It's completely adapting with the real world data. Similarly, for this task of cabinet opening, the simulator adapts over time and transfers to realistic scenarios.

我另一个非常感兴趣的方向是从图像中学习，但这些图像大多是 2D 的，而我们的真实世界是 3D 的。那么，我们如何将 3D 感知的概念融入学习流程呢？这是我在计算机视觉方面一直在探索的，看看如何尽可能轻松地从 2D 图像中获取 3D 信息。如果你是计算机视觉领域的人，你知道有很多表示 3D 的方式：点云、体素、网格、多视图等。但大多数表示都是体积式的。如果你的最终目标是从图像中获取 3D 用于图形，体积式表示是好的。但如果你是机器人领域的人，体积式表示太重了，因为你试图维护一个随时间在动作空间中更新的 3D 记忆，这变成了 4D，效率非常低。

One other direction I've been really interested in is learning from images, but these images are mostly 2D, while our real world is 3D. So how can we incorporate this idea of 3D perception into the learning pipeline? This is something I've been looking at more on the computer vision side, to see how far we can go in trying to get 3D out of 2D images as easily as possible. If you're a computer vision person, you know there are many ways to represent 3D: point clouds, voxels, meshes, multi-views, etc. But most of these representations are volumetric. Volumetric representation is good if your end goal is getting 3D from images for graphics. But if you're a robotics person, volumetric representation is too heavy because you're trying to maintain a 3D memory that updates over time in the action space, which becomes 4D and is really inefficient.

我们最近从视觉角度研究的一个问题是：能否仅通过观察 2D 图像并用 2D 流形描述场景来学习 3D 推理？这是我提到的最后一个项目，我不会深入细节，只展示一些漂亮的图片。任务是：给定一个输入视图，生成新视图。要解决这个问题，你需要理解 3D。我们的关键想法是用一张纸包裹世界。给定一张输入图像，我们从一张像锡箔或收缩膜一样的纸开始，预测一个中间深度表示，预测纸上的变形，并将其贴合到场景上，使得收缩膜与模型学习的深度预测一致。然后我们用这个来生成新视图。这个流程是端到端训练的。

One question we've been looking at recently from a vision aspect: can we learn 3D reasoning by just looking at 2D images and describing the scene with a 2D manifold? This is the last project I'm mentioning, and I won't go into details, just show pretty pictures. The task is: given an input view, generate novel views. To solve this, you need to understand 3D. Our key idea is to wrap the world in a sheet. Given an input image, we start with a sheet of paper like tin foil or shrink wrap, predict an intermediate depth representation, predict deformations on the sheet, and fit it onto the scene such that the shrink wrap is consistent with the depth prediction learned by the model. Then we use it to generate novel views. This pipeline is trained end-to-end.

让我展示一下结果。给定这张输入图像，测试时没有其他信息，我们可以生成新视图，就像它是一个 3D 环境一样。我们在场景上拟合一张收缩膜，然后我们可以走进输入图像，就像它是一个 3D 场景一样。我们可以四处导航。这适用于像这样的户外场景，也适用于室内场景。输入图像，你在世界上包裹一张纸，现在你可以完全从单张图像导航并生成全景视图。这些信息直接来自测试时的单张图像，没有立体视觉。这张图片借用了 Derek Hoiem 2005 年的一篇论文，该论文将场景表示为平面，但我们这里使用纸。它也适用于物体的室内图像。拿一张纸，包裹场景，现在你可以体验这个图像的 3D 效果。这完全来自单张图像。事实上，我们还将它推广到绘画。这是一幅画；我们在画上包裹一张纸，它允许我们在绘画空间中导航。所以基本上，有了 3D，你可以走进画里。

Let me show you the results. Given this input image, no other information at test time, we can generate new views as if it were a 3D environment. We fit a shrink wrap sheet onto the scene, then we can walk into the input image as if it were a 3D scene. We can navigate around. This applies to outdoor scenes like this, but also indoor scenes. Input image, you wrap a sheet around the world, now you can navigate and generate overviews completely from a single image. This information comes directly from a single image at test time, no stereo. This image is borrowed from a 2005 paper by Derek Hoiem, which represented scenes as planes, but here we use sheets. It also works on indoor images of objects. Take a sheet, wrap around the scene, and now you get to experience this image in 3D. It's completely from a single image. In fact, we also generalize it to paintings. Here is a painting; we wrap the sheet around the painting, and it allows us to navigate in the painting space. So basically, with 3D, you can go inside the painting.

这项工作有一些失败案例。例如，这个场景非常困难，因为有树、树叶和更远的背景，所以深度非常不连续。在这种情况下，如果你非常靠近树，它就会开始模糊场景。这个失败例子也有效……好吧，算了。总之，总结一下这次演讲：我们讨论了多个方向，将好奇心驱动的学习思想扩展到真实机器人。第一，在真实世界中高效学习。第二，学习基于轨迹的动作空间。第三，学习机器人每个电机的策略以跨场景泛化。第四和未来方向：如何自动迁移模拟器，并从单张图像学习感知 3D 自然场景。其中一些方向非常早期，我们只是刚刚触及表面。我仍然认为还有很长的路要走，但结果很有希望。我对此非常兴奋。除了最后几篇论文尚未公开外，所有代码都已公开，如果你发邮件给我，我很乐意分享更多细节。谢谢。

This work has some failures. For instance, this scene is really hard because there are trees, leaves, and a farther background, so depth is really discontinuous. In this case, if you go really close to the tree, it starts to blur the scene. It works in this one more failure example... okay, never mind. Anyway, to summarize the talk: we discussed multiple directions to scale these ideas of curiosity-driven learning to real robots. First, learning efficiently in the real world. Second, learning a trajectory-based action space. Third, learning policies for each motor of the robot to generalize across scenarios. Fourth and future directions: how to transfer simulators automatically and learn to perceive 3D from single images for natural scenes. Some of these are very early directions; we are barely scratching the surface. I still believe there is a long way to go, but the results are promising. I'm quite excited about all this. All the code is available except for the last few papers, which are not public yet, but I'm happy to share more details if you email me. Thank you.

让我们感谢演讲者。好吧，在家鼓掌。我不确定如何组织提问；我们试试让大家直接说出来，希望顺利。如果不行，我会请人举手。我觉得最后一点非常酷。我想知道你是否考虑过应用 DeepMind 几年前的那篇论文？

Let's thank our speaker. Well, this clap at home. I'm not sure how to organize the questions; let's just try people blurting them out and hoping for the best. If not, I'll ask someone to do the raised hand thing. One thing I found really cool was the last thing. I was wondering if you had considered applying the paper from DeepMind a few years ago?

我们还没有应用这个，但其他人已经应用了，他们发现了非常明显的问题。首先，那篇论文是在类似环境中训练和测试的，比如相似的场景。你有一个模拟器，你四处走动并用多张图像训练，然后你可以在同一场景中想象。而在这里，我们将其迁移到非常域外的图像，所以那种方法很难奏效。根本区别在于场景表示网络基本上是一个……

We haven't applied this yet, but other people have applied it and they see very clear issues. First, that paper was training and testing on similar environments, like similar scenes. You have a simulator, you go around and train with multiple images, and then you can imagine in the same scene. Here, we are really transferring it to very out-of-domain images, so it's very hard for that kind of approach to work. The fundamental difference is the scene representation network is basically a...

黑盒网络，所以基本上你只是希望深度学习能解决视角合成问题。但在这项工作中，我们实际上是将几何和学习结合起来使其工作，所以它不依赖于泛化的希望。不过，我认为我展示的结果与那个在模拟中几乎无法工作的模型的能力相去甚远。而且我认为卡特里娜的一些工作也表明它在模拟中甚至效果不佳，所以我们没有太大动力去尝试它。但我们与其他方法进行了比较。

Black box network, so you just basically hope that deep learning will work for view synthesis. But in this work, we are actually employing geometry and learning together to make it work, so it's not relying on hope for generalization. But yeah, I think the results I was showing are very far off from what are the abilities of that model that barely works in simulation. And I think some of the work from Katrina also shows that it doesn't even work in simulation very well, so we weren't quite motivated to try it. But we compare it to other approaches.

谢谢，这很有道理。教授您好，我有一个关于对不同机器人身体泛化的问题。我想知道关于在机器人不同部分之间传递的消息有什么直觉，对于一个新的机器人，是否有任何保证，在什么情况下这些局部控制器可以泛化，在什么情况下不能？

Thanks, that makes a lot of sense. Yeah, hi professor, I have a question about the generalization to different robot bodies. I wonder if there is any intuition on what messages are being passed across different parts of the robot, and to give a new robot, is there any guarantee in what situations these local controllers can generalize and in what situations they won't?

是的，这确实是个很好的问题。我跳过了这部分，但我可以快速展示来回答你的问题。我们实际上可视化了根节点的消息，因为这是一个树结构，有根节点。我们可视化了消息在一个回合中的变化程度以及如何变化。y 轴是消息的一维投影（消息是 64 维的），x 轴是回合中的变化。在训练初期，由于这些策略是从头开始学习的，它们对消息应该是什么没有任何先验知识，只是随机消息，所以看起来是零。我展示的第一行是 walker 智能体，这是 hopper。随着训练的进行，这些消息开始呈现一些结构，到训练结束时，它们呈现出非常结构化的形状。对于 walker，它呈现正弦波；对于 hopper，它呈现三个形状。当你找到对应关系时，我们发现……

Yeah, actually this is a very good question. I skipped this part, but I can quickly show to give your answer. So what we do is we actually visualized the message of the root node because this is a tree and there's a root node. We visualized how much the message varies across an episode and how it varies. The y-axis is a 1D projection of the message, which is 64-dimensional, and the x-axis is the variation across episode. In the beginning of the training, since these policies are learning from scratch, they do not have any prior as to what messages should be, and they are just random messages, so this just looks zero. The top row I am showing is for walker agent, this is for hopper. As the training goes, these messages start to take some structure, and towards the end of the training, they take a very structured shape. For walker it takes a sinusoidal wave, for hopper it takes three shapes. When you find the correspondence, we found out...

教授您好，我想您没有共享屏幕。

Hi professor, I think you're not sharing the screen.

哦等等，我正在看这个。好吧，你应该早点说的。抱歉。好的，我现在正在共享屏幕。让我在这里快速说一下。在可视化中，我们可视化了消息在一个回合中的变化。y 轴是消息的一维投影，x 轴是回合中的变化。在训练初期，消息完全为零，因为它们是从头开始学习的，没有什么有趣的内容。这是 walker，这是 hopper。但随着训练的进行，这些消息开始呈现一些结构。每个图显示的是训练过程中一个回合内的变化，到训练结束时，它们呈现出非常结构化的形状。对于 walker，它有两个峰值，顶部峰值对应这个姿势，底部峰值对应这个姿势。对于 hopper，它对应这三个姿势。所以从某种意义上说，这些消息基本上开始学习机器人的内部时钟，知道它们何时在工作。至少对于运动来说，这个结果并不令人惊讶，因为关于 CPG（中枢模式发生器）的工作中，人们硬编码了一堆正弦波和余弦波来适配运动模式。唯一的区别是，在这种情况下，这些模式是从头开始涌现的。但我们也在将这些想法应用于操作任务，这是那边的工作。这些模式不会很自然地出现，因为很难用正弦波表示手部运动。但至少在运动场景中，他们展示了这些消息呈现出非常有意义的形状，可以进行分析。但我们对此没有任何保证。至少我不知道如何证明保证，我不在那个领域工作。但如果有人感兴趣，我很乐意分享见解。

Oh wait, I'm watching this. Okay, you should have said it earlier. Okay, so sorry. Okay, I'm sharing it on the screen now. So let me just say it quickly here. In the visualization, we are visualizing the variation of messages across an episode. The y-axis is a 1D projection of the messages, the x-axis is the variation across an episode. In the beginning of the training, messages are completely zero because they are being learned from scratch, there is nothing interesting in them. This is for walker, this is for hopper. But as the training goes on, these messages start to take some structure. Each plot is showing the variation across an episode over training, and towards the end of the training they take a very structured shape. For walker it takes two peaks, where the top peak corresponds to this pose, the bottom one corresponds to this one. For hopper it corresponds to these three poses. So in some ways, these messages basically start learning an internal clock for the robot as to when they are working. This result is not very surprising for locomotion at least, because there is work on CPGs (central pattern generators) where people hardcode a bunch of sine waves and cosine waves to fit a locomotion pattern. The only difference is in this case these patterns are emerging from scratch. But we are also applying these ideas to manipulation, which is in the work over there. These patterns don't happen very naturally because it's hard to represent hand movement via sine wave. But at least in the locomotion scenario, they showed results that these messages take a very meaningful shape which someone can analyze. But we don't have any guarantees so to say in these things. At least I don't know how to prove guarantees, I don't work in that area. But if someone is interested, I'm happy to share insights.

谢谢。嘿，Deepak，感谢你的演讲。我有一个关于《One Policy to Control Them All》论文中评估的任务的问题。你评估的任务集是目标是在某个特定方向上最大化移动，你可以认为任何让你朝那个方向移动的行为都会给你一些正奖励。你认为如果你尝试将这种模块化控制应用于更定向的任务，即需要达到特定目标状态的任务，更一般地说，当你切换到这样的任务时，即使是在模块化框架之外，你认为事情还能像以前一样工作吗，还是说它会崩溃？

Thank you. Hey Deepak, thanks for the talk. I had a question about the tasks you evaluate on in the 'One Policy to Control Them All' paper. You're evaluating on this suite of tasks where your goal is to maximize movement in some particular direction, and you can make the argument that anything you do that gets you in that direction is giving you some positive reward. Do you think that if you were to try this modular control on more directed tasks where there's a specific goal state that you need to achieve, and more generally when you switch to tasks like this even outside of the modularity framework, do you think you can still expect things to work as well or does it just break down?

这是我们目前正在研究的想法之一，我们已经取得了一些进展。我们做的一个改变是，不是无模型，而是将相同的想法应用于基于模型的设置，事实上我们看到了同样的改进。这仍在进行中，所以我不会说我们有像这篇论文一样好的结果，但我们正在将其应用于操作任务，其中目标是完成某个任务。注意，我认为这不是一个令人惊讶的事实，因为这种方法与奖励函数或其他任何东西无关。它基本上是对人们今天做强化学习或基于学习的机器人规划的一种直接替代。你所做的就是修改黑盒策略，打开它，使其在一定程度上可解释，或者在其中赋予一些有意义的结构。所以它不直接依赖于奖励。你还可以在上面应用好奇心等，我还没有做，但这是我们未来的方向。一致性是相似的：你打开策略，它是空白的。人们训练深度学习黑盒策略，我们打开它并在其中放入模块化，我们放入一个片状和网格结构来生成视图，我们打开它并放入动力系统的结构使其更具动态性。所以所有这些论文在某种程度上都遵循相同的主题。

This is one of the ideas we are working on right now, and we already have some progress. One change we make is that instead of model-free, we apply the same idea for model-based setups, and in fact we see the same kind of improvement. It is still ongoing, so I would not say that we have results as good as this paper, but we are applying it for manipulation where the goal is given to achieve some task. Note that the reason I think this is not a surprising fact is because this approach has nothing to do with the reward function or anything. It's basically a drop-in replacement for how people do RL or learning-based robot planning today. All you are doing is modifying the black box policy, opening it, and making it somewhat interpretable or giving it some meaningful structure inside. So it's not dependent directly on the reward. One thing you could do is also apply curiosity etc. on top of it, which I haven't done yet, but something on our horizon. The consistency is similar: you open the policy, it's a blank. People train deep learning black box policy, we open it and put modularity inside it, we put a sheet and a mesh structure to generate our views inside it, we open it and put the structure of dynamical systems to make it more dynamic. So all these papers are basically following the same theme in some ways.

好的，谢谢。还有其他问题吗？所以当你把片状放在物体上时，我有点好奇你是怎么做的。换句话说，你是放下一个片状，并有一些距离的概念来最小化误差之类的吗？但你不能那样做，因为你不知道距离，因为它是从 2D 图像开始的……

Okay, thanks. Any other questions? So when you do the sheet on the object, I'm just a little curious as to how you do that. In other words, are you dropping a sheet and have some notion of distance you're minimizing error or something? But you can't do that because you don't know the distances because it's a 2D image starting with...

哦，是的，是的。好的，所以有两个阶段。我讲得很快，所以……

Oh yes, yes. Okay, so there are two phases. I went very fast, so...

训练这个系统的方式是使用来自两个不同视角的两张图像。你通过这种方式训练它。在测试时，你只有一张图像。所以基本上，发生的事情是：这个 sheet 在这里。训练时，你有两张图像：一张来自这个视角，另一张是目标视角。训练方法是，你取一个 sheet，为它生成一些变形，同时还有场景中每个点的深度。然后你将 sheet 像网格一样拟合到场景上。接着你可以用几何学从不同视角渲染 sheet，得到目标视角。例如，你有一对输入视角和目标视角，以及它们之间的相机变换。你完全端到端地训练这个系统。然后在测试时，你只有单个视角，你可以传入任意相机来从不同视角生成图像。

The way you train this system is by having access to two images from two different views. You train it that way. At test time, you only have one image. So basically, what is happening: this sheet thing is here. At training time, you have two images: an image from this view and some target view. The way you train this is that you take a sheet, generate some deformation for the sheet, and also what is the depth of each point in the scene. Then you fit the sheet onto the scene like a mesh. Then you can use geometry to render the sheet from different points of view, and you get a target view. For example, you had a pair of input view and target view, and the camera transformation between them. You train on this completely end to end. Then at test time, all you have is just a single view, and you can pass arbitrary cameras to generate it from different views.

一个问题：如果你天真地将 sheet 拟合到图像上，会有一个平凡解：sheet 基本上应该像平面一样覆盖整个图像，而不是真正捕捉其 3D 部分。但我们避免了那个平凡解，因为我们也预测了一个中间深度表示，并且我们是从一个与输入视角不同的目标视角进行训练的。训练时除了两张图像外没有其他监督，这在一定程度上防止了平凡解。

One question: if you naively fit a sheet onto an image, there is a trivial solution: the sheet should basically fit the whole image as a plane and not really capture the 3D part of it. But we avoid that trivial solution because we are also predicting an intermediate depth representation, as well as we are training from a target view which is not the same as input view. There is no other supervision at training time except for two images, which kind of prevents this trivial solution.

还有人还有其他问题吗？好的，我有一个快速问题，算是那个问题的后续。你提到你生成不同的变形。我只是好奇：你是如何生成不同的变形的，这是否有助于它泛化到不同的情况？

Does anyone else have any more questions? All right, I have one quick question, just kind of a follow-up to that one. You mentioned you generate the different deformations. I was just curious: how do you generate the different deformations, and is that kind of how you help it generalize across the different?

是的，这些变形基本上是通过梯度下降学习的。使其工作的关键是整个流程是端到端可微的，并且有多个组件：可微渲染和一种新颖的可微纹理采样。所以变形基本上包括三件事：对于网格中的每个顶点，你预测它在空间方向上变形多少，以及在深度方向上变形多少。然后你为所有顶点预测这个变形，拟合网格，从正常视角渲染，计算损失。希望是，通过在训练时这样做，由于训练是在多对图像上进行的，它将学习结构：如果有墙，它应该是一个单一的平面网格；如果有地板，它应该是一个单一的平面网格；但如果有物体，它应该很好地包裹物体以获得好的新视角。所以这基本上是因为训练时的学习。这回答了你的问题吗？

Yes, these deformations are basically learned via gradient descent. The fact that makes this work is that this whole pipeline is end-to-end differentiable, and there are multiple components to it: there is differentiable rendering and this is a novel differentiable texture sampling. So the deformations are basically three things: for each vertex in the mesh, you predict how much to deform it in the spatial direction and how much to deform it in the depth direction. Then you predict this deformation for all the vertices, fit a mesh, render it from a normal view, take loss. The hope is that by doing this at training time, since your training is over multiple pairs of images, it will learn the structure: if there is a wall, it should be a single flat mesh; if there is a floor, it should be a single flat mesh; but if there is an object, it should really wrap around the object very nicely to get good new views. So it's basically because of learning at training time. Does it answer your question?

是的，这回答了那个问题。谢谢。

Yeah, that answered that question. Thank you.

好的，再次感谢我们的演讲者。也祝大家假期愉快，希望大家平安。谢谢 Deepak。谢谢，那是一场很棒的演讲。谢谢。

Okay, well again, let's thank our speaker. Also, I wish you all a good holiday break, and just everyone, I hope you stay safe. Thank you, Deepak. Thank you, that was a great talk. Thanks.