机器人网球与人机交互

以下是对话彼得·阿贝尔。他是加州大学伯克利分校教授、伯克利机器人学习实验室主任，是全球顶尖的研究者之一，致力于让机器人理解并互动周围世界，尤其通过模仿学习和深度强化学习。本对话是麻省理工学院通用人工智能课程和人工智能播客的一部分。如果喜欢，请在 YouTube、iTunes 或你选择的播客平台订阅，或在 Twitter 上联系我@LexFriedman。现在，开始我与彼得·阿贝尔的对话。你曾提到，如果只能见一个人，你会选罗杰·费德勒。那么我问：你认为什么时候能有一个机器人完全自主地在网球上击败罗杰·费德勒？达到费德勒水平的网球选手？

The following is a conversation with Pieter Abbeel. He's a professor at UC Berkeley and the director of the Berkeley Robotics Learning Lab. He's one of the top researchers in the world working on how we make robots understand and interact with the world around them, especially using imitation and deep reinforcement learning. This conversation is part of the MIT course on Artificial General Intelligence and the Artificial Intelligence Podcast. If you enjoy it, please subscribe on YouTube, iTunes, or your podcast provider of choice, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D. And now, here's my conversation with Pieter Abbeel. You've mentioned that if there was one person you could meet, it would be Roger Federer. So let me ask: when do you think we will have a robot that fully autonomously can beat Roger Federer at tennis? A Roger Federer-level player at tennis?

首先，如果你能安排我见到罗杰，请告诉我。至于让机器人击败他打网球，这是个有趣的问题，因为 AI 领域的许多挑战中，软件才是缺失的部分。但像这样的事，硬件也远未达标。要有一个能真正跑动的机器人——波士顿动力的机器人开始接近了，但还达不到人类水平，能跑动并挥拍。这是硬件问题。我不认为这只是更难的问题；我认为是硬件和软件问题。两者都是，而且它们会独立进步。所以硬件方面，我猜大概 10-15 年。草地还是红土？红土涉及滑步，可能更难掌握。

Well, first, if you can make it happen for me to meet Roger, let me know. In terms of getting a robot to beat him at tennis, it's an interesting question because for a lot of the challenges we think about in AI, the software is really the missing piece. But for something like this, the hardware is nowhere near either. To really have a robot that can physically run around—the Boston Dynamics robots are starting to get there, but still not at human-level ability to run around and then swing a racket. That's a hardware problem. I don't think it's a harder problem only; I think it's a hardware and a software problem. I think it's both, and I think they'll have independent progress. So I'd say the hardware maybe in 10-15 years. I'm just guessing. Grass or clay? The clay involves sliding, which might be harder to master actually.

但你不限于双足。我是说，肯定可以造一个机器——这完全是另一个问题。当然，你可以说，好吧，这个机器人可以装轮子，用轮子移动，设计不同。那可能比完整人形设置更快实现。你觉得挥拍呢？你研究过基础操作。挥拍击出漂亮的反手或正手有多难？

Yeah, but you're not limited to bipedal. I mean, I'm sure you can build a machine—it's a whole different question. Of course, you know, you can say, okay, this robot can be on wheels, move around on wheels, and be designed differently. Then I think that can be done sooner, probably, than a full humanoid type of setup. What do you think about swinging a racket? So you've worked on basic manipulation. How hard do you think the task of swinging a racket is, to be able to hit a nice backhand or a forehand?

好吧，假设我们固定设置——一个不错的机械臂，比如标准工业臂——它能看着球过来然后挥拍。好问题。我不确定会非常难。我是说，如果用强化学习，肯定需要大量试错。第一次不会挥对。但我不认为不能正确完成。我觉得是可学习的。如果你在一侧放发球机，另一侧放带网球拍的机器人，我认为可学习，也许在模拟中做些预训练。是的，我认为可行。挥拍是可行的。看它能达到多高精度会很有趣。有些人类选手能打线球，精度极高还带旋转。强化学习能否学会给球加旋转，这很有趣。好吧，你引起了我的兴趣。也许有天我们会设置这个。你的回答基本是：对于这个问题，听起来很迷人，但对于网球运动员的通用问题，我们可能还差得远。

Okay, let's say we just set up stationary—a nice robot arm, let's say a standard industrial arm—and it can watch the ball come and then swing the racket. It's a good question. I'm not sure it would be super hard to do. I mean, I'm sure it would require a lot if we do it with reinforcement learning—a lot of trial and error. It's not going to swing it right the first time around. But yeah, I don't see why it couldn't be done the right way. I think it's learnable. I think if you set up a ball machine on one side and then a robot with a tennis racket on the other side, I think it's learnable, maybe with a little bit of pre-training in simulation. Yeah, I think that's feasible. I think swinging the racket is feasible. It would be very interesting to see how much precision it can get. I mean, some human players can hit it on the lines, which is very high precision with spin. It's interesting whether RL can learn to put a spin on the ball. Well, you got me interested. Maybe someday we'll set this up. Your answer is basically: okay for this problem, it sounds fascinating, but for the general problem of a tennis player, we might be a little bit farther away.

你见过机器人在物理世界做的最令人印象深刻的事是什么？

What's the most impressive thing you've seen a robot do in the physical world?

物理上，对我来说是波士顿动力的视频。它们总是让我印象深刻。最近，机器人跑上楼梯，做跑酷类动作。我们不知道底层是什么——他们没写很多细节——但即使底层是硬编码，可能也可能不是，单是完成跑酷的物理能力就非常令人印象深刻。

So physically, for me, it's the Boston Dynamics videos. They always just ring home and I'm super impressed. Recently, the robot running up the stairs, doing the parkour type thing. I mean, yes, we don't know what's underneath—they don't really write a lot of detail—but even if it's hard-coded underneath, which it might or might not be, just the physical abilities of doing that parkour is very impressive.

你亲眼见过 Spot 或那些机器人吗？

Have you met Spot or any of those robots in person?

我去年四月在杰夫·贝佐斯组织的火星活动上见过 Spot Mini。他们把它带出来，它很好地跟着杰夫。杰夫离开房间时，他们让它跟着他，这很令人印象深刻。所以知道那些机器人没有学习，这有点信心。心理学上——知道这一点，知道没有太多学习，非常有限。我今年早些时候见过 Spot Mini，知道一切运作方式，进行一对一互动——我独自待了一会儿——立即在心理层面产生深层连接。即使你知道它如何运作的基本原理，还是有某种魔力。

I met Spot Mini last year in April at the Mars event that Jeff Bezos organizes. They brought it out there, and it was nicely following around Jeff. When Jeff left the room, they had it follow him along, which is pretty impressive. So I think there's some confidence to know that there's no learning going on in those robots. The psychology of it—while knowing that, while knowing there's not much learning going on, it's very limited. I met Spot Mini earlier this year, and knowing everything that's going on, having one-on-one interaction—I got to spend some time alone—and there's immediately a deep connection on the psychological level. Even though you know the fundamentals of how it works, there's something magical.

你考虑过在物理世界与机器人互动的心理学吗？即使你刚才给我看的 PR2 机器人，有点类似脸——头，有点类似脸。有些东西立刻吸引你。你考虑过机器人问题的这个方面吗？

Do you think about the psychology of interacting with robots in the physical world? Even you just showed me the PR2, the robot, and there was a little bit something like a face—a head, a little bit something like a face. There's something that immediately draws you to it. Do you think about that aspect of the robotics problem?

这很难。对于 BRETT——我们给它起个名字，伯克利消除繁琐任务机器人——很难不把它当人看。似乎每个人都叫它“他”，不管什么原因，但这比叫“它”更像人。这样想似乎很自然。上周末我印象深刻。我在视频里见过 Pepper 很多次，但后来在 Fidelity 组织的一个活动上，他们给 Pepper 编了脚本帮助主持一些环节，而且脚本让 Pepper 有点小孩的性格。很难不把它当作某种意义上的独立个体，因为它会突然插话——它会突然加入对话，非常互动。主持人会说：“Pepper，插话吧，等等，我呢？我能参与吗？”就像人一样。而那是 100%脚本化的，即使如此，也很难没有那种感觉，好像那里有什么东西。所以当机器人在物理世界互动时，那是一个可以在强化学习中使用的信号吗？你在这个方向做过一些工作，但你认为那种心理学现在能被引入吗？

Well, it's very hard. With BRETT—we'll give him a name, Berkeley Robot for the Elimination of Tedious Tasks—it's very hard to not think of the robot as a person. And it seems like everybody calls him a 'he' for whatever reason, but that also makes it more a person than if it was an 'it'. And it seems pretty natural to think of it that way. This past weekend really struck me. I've seen Pepper many times on videos, but then I was at an event organized by Fidelity, and they had scripted Pepper to help moderate some sessions, and yet scripted Pepper to have the personality of a child a little bit. And it was very hard to not think of it as its own person in some sense, because it was just kind of jumping in—it would just jump into conversation, making it very interactive. Moderator would be saying, 'Pepper, just jump in, hold on, how about me? Can I participate in this?' Doing it just like a person. And I was 100% scripted, and even then it was hard not to have that sense that somehow there is something there. So as we have robots interact in this physical world, is that a signal that can be used in reinforcement learning? You've worked a little bit in this direction, but do you think that psychology can be somehow pulled in now?

这是很多人问的问题，我认为部分原因是他们在思考我们到底有多独特。人们看到一些结果后——看到电脑下围棋、做这做那——他们会说：“好吧，但它真的能有情感吗？真的能那样与我们互动吗？”然后一旦你身处机器人周围，你就开始感受到它。我认为，也许从神话学角度，我的想法是：如果你运行像强化学习这样的东西……

That's a question a lot of people ask, and I think part of why they ask it is they're thinking about how unique we are really. Still, people ask after they see some results—they see a computer play Go, do this, do that—they're like, 'Okay, but can it really have emotion? Can it really interact with us in that way?' And then once you're around robots, you already start feeling it. And I think that kind of, maybe mythologically, the way that I think of it is: if you run something like reinforcement learning...

关于优化某个目标，没有理由说这个目标不能与一个人喜欢与这个系统互动的程度挂钩。为什么强化学习系统不能优化让它的机器人变得有趣呢？为什么它不会自然地变得越来越互动，越来越像一个人或一只宠物呢？我不知道它具体会变成什么样，但只要你能形式化一个关于“喜欢”的目标，它就会自动获得这些特征。

About optimizing some objective, and there's no reason that objective couldn't be tied into how much a person likes interacting with this system. And why couldn't the reinforcement learning system optimize for their robot being fun to be around? And why wouldn't it then naturally become more and more interactive, and more and more maybe like a person or like a pet? I don't know what it would exactly be, but more have those features and acquire them automatically, as long as you can formalize an objective of what it means to like something.

你如何展示什么是真实情况？你如何从人类那里获得奖励？因为你必须以某种方式从人类那里收集信息。但你是说，如果你能把它形式化为一个目标，它就可以被学习。没有理由它不能通过学习涌现出来。

How do you exhibit what's the ground truth? How do you get the reward from a human? Because you have to somehow collect that information from the human. But you're saying if you can formulate it as an objective, it can be learned. There is no reason it couldn't emerge through learning.

也许有一种方法可以将其形式化为一个目标，而不必明确地给它打分。标准奖励是数字，数字很难获得——在某个尺度上是 1.5 或 0.7，这对人来说非常困难。但更容易的是让人说：“好吧，你过去五分钟做的比之前五分钟做的好多了。”这就提供了一个比较。事实上，在这方面已经有一些成果。例如，OpenAI 的 Paul Christiano 及其合作者让 Hopper——一个单腿机器人——纯粹通过反馈“我更喜欢这个而不是那个”来做小后空翻。这同样有效。经过一系列互动，它明白了人类在要求什么，即一个后空翻。机器人并没有试图做后空翻；它只是根据人类内心的想法，从比较分数中得到一个分数。人类心里想的是“我想做一个后空翻”，但机器人不知道它应该做什么。它只知道有时人类说“这个更好，这个更差”。然后机器人明白了人类真正想要的是后空翻。我想对于更互动的机器人也是如此：机器人会随着时间的推移发现，“哦，这种东西显然比那种东西更受欢迎。”

And maybe one way to formulate it as an objective, you wouldn't have to necessarily score it explicitly. Standard rewards are numbers, and numbers are hard to come by—this is a 1.5 or 0.7 on some scale, it's very hard for a person. But much easier is for a person to say, "Okay, what you did the last five minutes was much nicer than what you did the previous five minutes." That gives a comparison. And in fact, there have been some results in that. For example, Paul Christiano and collaborators at OpenAI had the Hopper—a one-legged robot—doing little backflips purely from feedback: "I like this better than that." That's kind of equally good. After a bunch of interactions, it figured out what the person was asking for, namely a backflip. The robot wasn't trying to do a backflip; it was just getting a score from the comparison score from the person based on their own mind. The person had in their own mind "I want to do a backflip," but the robot didn't know what it was supposed to be doing. It just knew that sometimes the person said "this is better, this is worse." Then the robot figured out what the person was actually after was a backflip. I'd imagine the same would be true for things like more interactive robots: the robot would figure out over time, "Oh, this kind of thing apparently is appreciated more than this other kind of thing."

当我第一次拿起 Sutton——Richard Sutton——的强化学习书时，在这种深度学习之前，在神经网络作为机器学习强大机制重新出现之前，RL 对我来说就像魔法一样，很美。那似乎就是智能的本质：RL，强化学习。那么，当行动的奖励被延迟、如此稀疏时，我们怎么可能学到关于世界的任何东西呢？为什么你认为 RL 有效？为什么你认为在如此稀疏的奖励下你能学到任何东西，无论是常规强化学习还是深度强化学习？你的直觉是什么？

When I first picked up Sutton's—Richard Sutton's—reinforcement learning book, before this sort of deep learning, before the re-emergence of neural networks as a powerful mechanism for machine learning, RL seemed to me like magic, as beautiful. That seemed like what intelligence is: RL, reinforcement learning. So how do you think we can possibly learn anything about the world when the reward for the actions is delayed, is so sparse? Like, why do you think RL works? Why do you think you can learn anything under such sparse rewards, whether it's regular reinforcement learning or deep reinforcement learning? What's your intuition?

部分原因在于为什么 RL 需要这么多样本，这么多经验来学习。因为实际情况是，当你有一个稀疏奖励时，你做一些事情——也许你采取了一百个行动，然后你得到一个奖励，也许你得到了 3 分。你会想，“好吧，3 分，不知道这意味着什么。”你再来一次，现在你得到了 2 分。现在你知道你第二次做的那一百个行动序列在某种程度上比第一次做的差。但很难知道哪些行动更好或更差；有些可能在两者中都有好有坏。这就是为什么你需要这么多经验。但一旦你有足够的经验，RL 实际上是在把它们分开。它试图说，“好吧，当你获得更高奖励时，什么是一致的？当你获得较低奖励时，什么是一致的？”然后策略梯度更新的魔力在于：它说，“现在让我们更新神经网络，使那些在情况良好时出现的行动更有可能，而使那些在情况不佳时出现的行动更不可能。”

Part of that is why RL needs so many samples, so many experiences to learn from. Because really what's happening is, when you have a sparse reward, you do something—maybe you take a hundred actions and then you get a reward, and maybe you get a score of three. You're like, "Okay, three, not sure what that means." You go again, and now you get a two. Now you know that the sequence of a hundred actions you did the second time around was somehow worse than the sequence of a hundred actions you did the first time. But it's tough to know which of those actions were better or worse; some might have been good and bad in either one. That's why you need so many experiences. But once you have enough experiences, effectively RL is teasing that apart. It's trying to say, "Okay, what is consistently there when you get a higher reward, and what's consistently there when you get a lower reward?" Then the magic of sums is the policy gradient update: it says, "Now let's update the neural network to make the actions that were kind of present when things are good more likely, and make the actions that are present when things are not as good less likely."

这是一个反论点，但似乎你需要运行它比你实际做的多得多。尽管现在人们可以说 RL 非常低效，但它似乎比纸上想象的要高效得多。对策略的简单更新，策略梯度，不知何故你能学到——正如你所说，哪些常见行动似乎产生了一些好结果——不知何故能学到任何东西。这至少是反直觉的。背后有什么直觉吗？

That is the counterpoint, but it seems like you would need to run it a lot more than you do. Even though right now people could say that RL is very inefficient, it seems to be way more efficient than one would imagine on paper. That the simple updates to the policy, the policy gradient, that somehow you can learn—as you said, what are the common actions that seem to produce some good results—that somehow can learn anything. It seems counterintuitive at least. Is there some intuition behind it?

是的，所以我认为有几种思考方式。我最初主要思考的方式是，当我们开始在伯克利这里研究深度强化学习时，大概是 2011、2012、2013 年左右——John Schulman 当时是博士生，最初在这里推动它——我们当时的想法是：如果你考虑修正线性单元或那种二元触发型神经网络，你会得到什么？你会得到一种分段线性反馈控制。如果你看看文献，线性反馈控制非常成功，可以出奇地解决很多很多问题。我记得，例如，当我们做直升机飞行时，如果你处于稳定飞行状态——不是非稳定，而是稳定飞行状态，比如悬停——你可以使用线性反馈控制来稳定直升机，一个非常复杂的动力系统，但控制器相对简单。所以我认为这是很大一部分原因：如果你做反馈控制，即使你控制的系统非常非常复杂，通常相对简单的控制架构已经可以做很多事情。但同样，仅仅线性是不够的。所以你可以把这些神经网络看作是在划分空间，人们之前已经尝试过用手工或有限状态机来做：“说这里有一个线性控制器，这里有一个线性控制器。”网络学会了这一点：它在这里发明了一个线性控制器，在那里发明了另一个线性控制器。但这比那更微妙。它受益于这种线性控制方面，受益于划分，但它是每次一维地划分。因为如果你有一个两层网络，即使是隐藏层——你从激活到非激活或反之——这本质上是一个轴，不是轴而是你改变的一个方向。所以你有这种非常渐进的划分空间，划分空间的线性控制器之间有很多共享。这一直是我的直觉，为什么预期这可能效果很好。它本质上利用了线性反馈控制如此之好的事实，但当然不够，这是一种用共享很多的线性反馈控制逐渐划分空间的方式。

Yeah, so I think there are a few ways to think about this. The way I thought about it mostly originally, when we started working on deep reinforcement learning here at Berkeley, which was maybe 2011, 2012, 2013 around that time—John Schulman was a PhD student initially kind of driving it forward here—the way we thought about it at the time was: if you think about rectified linear units or kind of binary firing type neural networks, what do you get? You get something that's piecewise linear feedback control. And if you look at the literature, linear feedback control is extremely successful, can solve many, many problems surprisingly well. I remember, for example, when we did helicopter flight, if you're in a stationary flight regime—not a non-stationary, but a stationary flight regime like hover—you can use linear feedback control to stabilize a helicopter, a very complex dynamical system, but the controller is relatively simple. So I think that's a big part of it: if you do feedback control, even though the system you control can be very, very complex, often relatively simple control architectures can already do a lot. But then also, just linear is not good enough. So one way you can think of these neural networks is that they tile the space, which people were already trying to do more by hand or with finite state machines: "Say this linear controller here, this linear controller here." The network learns that: it invents a linear controller here, another linear controller here. But it's more subtle than that. It's benefiting from this linear control aspect, benefiting from the tiling, but it's somehow tiling it one dimension at a time. Because if you have a two-layer network, even the hidden layer—you make a transition from active to inactive or the other way around—that is essentially one axis, not an axis but one direction that you change. So you have this kind of very gradual tiling of the space, with a lot of sharing between the linear controllers that tile the space. That was always my intuition as to why to expect that this might work pretty well. It's essentially leveraging the fact that linear feedback control is so good, but of course not enough, and this is a gradual tiling of the space with linear feedback controls that share a lot.

你认为这种直觉能推广到更一般的问题吗？比如当你增加控制器维度，或者当你获得清晰奖励信号的频率降低时？它能延续到现实世界吗？

Do you think that intuition scales to more general problems, like when you increase the number of controller dimensions or when you get a clean reward signal less often? Does it carry forward to the real world?

我认为现实世界比我们在强化学习中看到的成功案例要棘手得多，尤其是时间尺度的问题。比如一个学生决定读博——这是一个非常高层级的决定。但他们的生活，和任何人一样，是一系列肌肉纤维的收缩和放松，这是高频控制。读博的决定相对于实际动作来说非常抽象。信用分配完全超出了当前任何强化学习算法的能力，我们需要尚未实现的层级推理。

I think things get really tricky in the real world compared to what we've seen succeed in reinforcement learning, especially because of the time scales. Consider a student deciding to do a PhD—that's a very high-level decision. But their life, like anyone's, is a sequence of muscle fiber contractions and relaxations, which is high-frequency control. The decision to do a PhD is abstract relative to actual actions. Credit assignment becomes completely beyond what any current RL algorithm can do, and we need hierarchical reasoning that isn't available yet.

你认为我们可以在哪里获得层级推理能力，通过哪些机制？

Where do you think we can pick up hierarchical reasoning, and by which mechanisms?

让我强调一下二三十年前工作的局限性。推理系统可以推理长程，但它们没有扎根于现实世界——人们手工设计逻辑动态描述，这些描述与感知脱节。现在有了深度学习，我们可以处理传感器数据并理解世界，所以是时候将两者结合了。一种方法是通过端到端训练将深度学习嫁接到传统方法上，比如 Aviv Tamar 和其他人用因果信息增益所做的。另一个我们探索的方向是信息论方法：选择一个能预测未来结果的潜变量，比如决定导航到加油站。但我们很难让它工作。然后我们重新审视了层级结构能带来什么：更好的信用分配和更快的学习。这导致了由 Rocky Duan 领导的关于学习强化学习的 RL^2 论文。这是一种元学习方法：我们优化更快的学习，看看层级结构是否会出现。在迷宫导航中，我们看到了沿着走廊的一致移动和避免重复访问的地方，但它还没有扩展到现实世界场景。这是一个迹象，表明你可以元学习层级概念。

Let me highlight the limitations of what was done 20-30 years ago. Reasoning systems could reason over long horizons, but they weren't grounded in the real world—people hand-designed logical dynamical descriptions that didn't tie into perception. Now with deep learning, we can process sensor data and understand the world, so it's a good time to bring these together. One way is to bolt deep learning onto traditional approaches via end-to-end training, like Aviv Tamar and others have done with causal info gain. Another direction we explored was information-theoretic approaches: choosing a latent variable that predicts future outcomes, like deciding to navigate to a gas station. But we had trouble making that work. Then we revisited what hierarchy gives us: better credit assignment and faster learning. That led to the RL^2 paper on learning to reinforcement learn, led by Rocky Duan. It's a meta-learning approach: we optimize for faster learning and see if hierarchy emerges. In maze navigation, we saw consistent motion down hallways and avoidance of revisited places, but it didn't scale to real-world scenarios yet. It was a sign of life that you can meta-learn hierarchical concepts.

这些元学习概念似乎解决了迁移学习，这是人工智能中最困难也是最重要的问题之一——泛化。我们在构建能够很好进行迁移学习的通用系统方面走了多远？我们走在正确的道路上，还是需要完全不同的突破？

These meta-learning concepts seem to address transfer learning, which is one of the hardest and most important problems in AI—generalization. How far along are we towards building general systems that can do transfer learning well? Are we on the right path, or do we need totally different breakthroughs?

我非常乐观。已经有一些非常令人印象深刻的结果。2012 年 AlexNet 的初步突破中，它在 ImageNet 上表现更好，但更重要的是你可以微调 AlexNet 用于新任务——它学到了可重用的东西。这是迁移学习在范围和影响上最大的成功。最近，通过扩展规模，这一点得到了扩展。例如，OpenAI 的语言模型和 Google 的语言模型仅仅为了预测而训练，然后被重用于其他任务。如果你在足够多的数据上训练足够大的模型，它似乎能够迁移。DeepMind 在迷宫导航上的结果也令人印象深刻，那不仅仅是强化学习。

I'm pretty optimistic. There are already very impressive results. With the initial breakthrough in 2012 with AlexNet, it did better on ImageNet, but the bigger deal was that you could fine-tune AlexNet for new tasks—it learned something reusable. That's the biggest success of transfer learning in terms of scope and impact. Recently, by scaling up, this has expanded. For example, OpenAI's language models and Google's language models are trained just for prediction and then reused for other tasks. If you train a big enough model on enough data, it seems to transfer. DeepMind's results on navigating mazes were also impressive, where it wasn't just reinforcement learning.

还会有其他目标在优化。所以我认为已经有很多有趣的结果了。我觉得可能很难理解这一点：到什么程度或者什么时候我们才称之为泛化？以及这些不同任务中涉及的泛化层次。好吧，顺便说一下，你画了这个图来框定问题。你曾在某个地方说过，学习掌握与学习泛化之间的区别，这是一个很好的思考角度。我想你说的是，学习掌握和学习泛化之间存在一个灰色地带。我记得我可能在某处听过这个，可能是在你的某个采访中，也许是和 Yoav Shoham 的那次？不太确定。但我喜欢这个例子。我不确定是谁说的，但例子大致是：如果你用当前的深度学习技术来预测，比如说，我们行星的相对运动，它会做得很好。但如果有一个巨大的新质量进入我们的太阳系，它就无法预测会发生什么，对吧？这是一种不同的泛化。一种依赖于我们今天可用的最简解释来解释行星运动的泛化，而单纯的模式识别可以很好地预测我们当前的太阳系运动，没问题。所以我认为这是一个与我们迄今取得的成就略有不同的泛化例子，而且不清楚仅仅通过增加正则化、强迫它得出更简单的解释——但这不是简单的——但这就是物理学家所做的，对吧？他们问：我能让它更简单吗？我能把它简化到什么程度？什么是最简单的方程来解释一切？是的，整个宇宙动力学的终极方程。在深度学习中，我们还没有那么努力地朝这个方向推进。不确定是否应该推进，但似乎你能从中得到一种我们当前方法尚未获得的泛化。

Going to have other objectives was optimizing for. So I think there's a lot of interesting results already. I think maybe it's hard to wrap my head around this: to which extent or when do we call something generalization, right? And the levels of generalization involved in these different tasks. Alright, so you draw this, by the way, just to frame things. You've heard you say somewhere it's the difference between learning to master versus learning to generalize. That's a nice line to think about. And I guess you're saying that's a gray area of what learning to master and learning to generalize. Where once—think I might have heard this, I might have heard it somewhere else, and I think it might have been one of your interviews, and maybe the one with Yoav Shoham? Not a hundred percent sure. But I like the example. I'm not sure who it was, but the example was essentially: if you use current deep learning techniques, what we're doing to predict, let's say, the relative motion of our planets, it would do pretty well. But then now if a massive new mass enters our solar system, it would not predict what will happen, right? And that's a different kind of generalization. A generalization that relies on the ultimate simplest explanation that we have available today to explain the motion of planets, whereas just pattern recognition could predict our current solar system motion pretty well, no problem. And so I think that's an example of a kind of generalization that is a little different from what we've achieved so far, and it's not clear if just regularizing more, forcing it to come up with a simpler explanation—but it's not simple—but that's what physics researchers do, right? To say, can I make this even simpler? How simple can I get this? What's the simplest equation I can explain everything with? Yeah, the master equation for the entire dynamics of the universe. We haven't really pushed that direction as hard in deep learning, I would say. Not sure if it should be pushed, but it seems a kind of generalization you get from that that you don't get in our current methods so far.

所以我刚刚和 Vladimir Vapnik 聊过，他是一位统计学习领域的统计学家，他梦想着为学习创造出一个 E=mc²，对吧？学习的一般理论。你认为这在短期内，也就是未来几十年内，是徒劳的追求吗？

So I just talked to Vladimir Vapnik, for example, who is a statistician in statistical learning, and he kind of dreams of creating the E=mc² for learning, right? The general theory of learning. Do you think that's a fruitless pursuit in the near term, within the next several decades?

我认为这是一个非常有趣的追求，原因如下：有很多证据表明大脑是相当模块化的。所以我可能不会把它看作是一种理论，也许是底层理论，但更像是一种原则。有研究发现，盲人会使用通常用于视觉的大脑部分来执行其他功能，甚至在某种损伤后，人们会以某种方式重新连接，他们可能能够重新利用大脑的部分区域来执行其他功能。所以这表明了某种模块化。我认为这是一个很自然的目标，看看我们能否找到这种模块化？我们能否找到这个东西？当然，不是大脑的每个部分都完全相同，不是所有东西都可以任意重新连接。但如果你想想像新皮层这样的东西，它是大脑中相当大的一部分，从目前的发现来看似乎相当模块化，你能设计出同样模块化的东西吗？如果你能让它成长，它可能会变得更强大。我认为这将是那种有趣且并非不切实际的底层原则。

I think that's a really interesting pursuit, and in the following sense: there is a lot of evidence that the brain is pretty modular. And so I wouldn't maybe think of it as the theory, maybe the underlying theory, but more kind of the principle. Where there have been findings where people who are blind will use the part of the brain usually used for vision for other functions, and even after some kind of injury, people will get rewired in some way, they might be able to reuse parts of their brain for other functions. And so what that suggests is some kind of modularity. And I think it is a pretty natural thing to strive forward to see: can we find that modularity? Can we find this thing? Of course, not every part of the brain is exactly the same, not everything can be rewired arbitrarily. But if you think of things like the neocortex, which is a pretty big part of the brain that seems fairly modular from what the findings so far, can you design something equally modular? And if you can just grow it, it becomes more capable probably. I think that would be the kind of interesting underlying principle to shoot for that is not unrealistic.

你认为你更喜欢数学还是经验性的试错来发现做智能事情的本质？强化学习两者兼有，对吧？你证明某些东西收敛，证明边界，同时很多成功都是：好吧，我们试试这个，看看是否有效。那么你倾向于哪一个？你如何看待你大脑的这两个部分？

Do you think you prefer math or empirical trial and error for the discovery of the essence of what it means to do something intelligent? So reinforcement learning embodies both, right? You prove that something converges, prove the bounds, and at the same time a lot of those successes are: well, let's try this and see if it works. So which do you gravitate towards? How do you think of those two parts of your brain?

所以也许我更希望我们能通过数学取得进展。原因可能是，如果你有可以数学形式化的东西，你可以跳过很多实验。实验需要很长时间，而且需要大量的试错——就像你研究过程中的强化学习——但你需要做很多试错才能成功。所以如果我们能跳过这些，在我看来这就是数学的意义。希望一旦你做了一系列实验，你开始看到模式，你可以做一些推导来跳过一些实验。但我同意你的看法，我的意思是，在实践中，很多进展都是我们无法找到能够让我们跳跃式前进的数学，我们是一步一步地取得渐进式进展：这里一个新实验，那里一个新实验，给我们新的见解，逐渐积累，但还没有达到这样的地步：好吧，这里有一个方程，现在解释了你知道那需要两年的实验才能达到的东西，但这个方程告诉我们结果会是什么。不幸的是，还没有那么多，是的，没有那么多。但你的希望还在。

So maybe I would prefer we could make the progress with mathematics. And the reason maybe I would prefer that is because often if you have something you can mathematically formalize, you can leapfrog a lot of experimentation. And experimentation takes a long time to get through, and a lot of trial and error—kind of reinforcement learning your research process—but you need to do a lot of trial and error before you get to a success. So if we can leapfrog that, in my mind that's what the math is about. And hopefully once you do a bunch of experiments, you start seeing a pattern, you can do some derivations that leapfrog some experiments. But I agree with you, I mean in practice a lot of the progress has been such that we have not been able to find the math that allows it to leapfrog ahead, and we are kind of making gradual progress one step at a time: a new experiment here, a new experiment there that gives us new insights, and gradually building up, but not getting to something yet where we're just like, okay, here's an equation that now explains how you know that would have been two years of experimentation to get there, but this tells us what the result is going to be. Unfortunately not so much, yeah, not so much. But your hope is there.

在试图教机器人或系统执行日常任务时，甚至在模拟中，你对什么更兴奋：模仿学习还是自我对弈？让机器人向人类学习，还是让机器人自己规划，以自己的方式摸索，最终玩耍，最终与人类互动，或者解决任何问题。哪个更让你兴奋？你认为哪个作为研究方向更有前景？

In trying to teach robots or systems to do everyday tasks, or even in simulation, what do you think you're more excited about: imitation learning or self-play? So letting robots learn from humans, or letting robots plan on their own, try to figure out in their own way, and eventually play, eventually interact with humans, or to solve whatever problem. What's more exciting to you? What's more promising you think as a research direction?

当我们看自我对弈时，它的美妙之处在于它回到了强化学习中的挑战。强化学习的挑战在于获取信号。如果你从未成功，你就得不到任何信号。在自我对弈中，你处于双方，所以一方成功，而美妙之处也在于一方失败，所以你看到了对比：你看到了比另一个版本更好的我的版本。所以每次你与自己博弈，你都会得到信号。所以每当你把某件事变成自我对弈，你就处于一个美妙的情境中，你可以自然地比大多数其他强化学习环境学得更快。所以我认为，如果我们能以某种方式将更多的强化学习问题转化为自我对弈的形式，那将会走得很远很远。到目前为止，自我对弈主要围绕在有自然对手的游戏中。但如果我们能对其他事情进行自我对弈，比如说，我不知道，一个机器人学习建造房子——我的意思是，这对机器人来说是一件非常高级的事情——但也许它试图建造一个小屋之类的东西。如果这可以通过自我对弈完成，如果有人能想出来，它会学得更快。我认为这将更接近于某种数学上的飞跃。

So when we look at self-play, what's so beautiful about it is it goes back to kind of the challenges in reinforcement learning. So the challenge of reinforcement learning is getting signal. And if you never succeed, you don't get any signal. In self-play, you're on both sides, so one of you succeeds, and the beauty is also one of you fails, and so you see the contrast: you see the one version of me that is better than the other version. And so every time you play yourself, you get signal. And so whenever you can turn something into self-play, you're in a beautiful situation where you can naturally learn much more quickly than in most other reinforcement learning environments. So I think if somehow we can turn more reinforcement learning problems into self-play formulations, that would go really, really far. So far, self-play has been largely around games where there are natural opponents. But if we could do self-play for other things, and let's say, I don't know, a robot learns to build a house—I mean that's a pretty advanced thing to try to do for a robot—but maybe it tries to build a hut or something. If that can be done through self-play, it would learn a lot more quickly if somebody can figure that out. And I think that would be something where it goes closer to kind of the mathematical leap.

有人想出一种形式化方法，把任何强化学习问题都变成自对弈问题，这样就能更容易地获得信号。但现实是，很多问题我们不知道如何变成自对弈，所以要么需要提供详细的奖励——不只是奖励达成目标，还要奖励取得进展——这就会变得耗时。一旦开始这么做，比如你想让机器人做某事，你就得给出所有这些详细的奖励。那为什么不直接给个示范呢？因为为什么不直接展示给机器人看呢？现在的问题是如何展示给机器人。一种方法是远程操作机器人，让机器人真正体验，这很好，因为那是信噪比很高的数据。我们做了很多这样的工作，你可以在 10 分钟内教会机器人一项新技能。你可以教机器人一项新的基本技能，比如'好的，拿起瓶子，放到别处'。不管瓶子从哪里开始，也许它总是放到一个目标上之类的，用远程操作来教是相当容易的。

Frogging where somebody figures out a formalism to turn any RL problem into a self-play problem, where you get signal a lot more easily. Reality is many problems we don't know how to turn into self-play, so either we need to provide detailed reward that doesn't just reward for achieving a goal but rewards for making progress, and that becomes time-consuming. Once you're starting to do that, let's say you want a robot to do something, you need to give all this detailed reward. Well, why not just give a demonstration? Because why not just show the robot? And now the question is how do you show the robot. One way to show is to teleoperate the robot, and then the robot really experiences things, and that's nice because that's really high signal-to-noise ratio data. We've done a lot of that, and you teach your robot skills in just 10 minutes. You can teach your robot a new basic skill like 'okay, pick up the bottle, place it somewhere else.' That's a skill no matter where the bottle starts, maybe it always goes onto a target or something. That's fairly easy to teach with teleoperation.

更有趣的是，你现在可以通过第三人称学习来教机器人，机器人看着你做某事，不是亲身经历，只是看着，然后说：'好吧，如果你向我展示那个，那意味着我应该做这个，而且我不会用你的手，因为我不能控制你的手，但我会用我的手。'它做了那个映射。所以我认为这是今年的一大突破。这是由 Chelsea Finn 领导的。这几乎就像示范的机器翻译，你有一个人类示范，机器人学习将其翻译成机器人做这件事意味着什么。那是元学习，从一个学习另一个。我认为这开启了很多更快学习的机会。

Now what's even more interesting is if you can now teach a robot through third-person learning, where the robot watches you do something and doesn't experience it but just watches it and says, 'Okay, well if you're showing me that, that means I should be doing this, and I'm not going to be using your hand because I don't get to control your hand, but I'm going to use my hand.' It does that mapping. So that's where I think one of the big breakthroughs has happened this year. This was led by Chelsea Finn here. It's almost like machine translation for demonstrations, where you have a human demonstration and the robot learns to translate it into what it means for the robot to do it. That was meta-learning to learn from one to get the other. And that I think opens up a lot of opportunities to learn a lot more quickly.

我的重点是自动驾驶汽车。你认为这种第三人称观察的方法适用于自动驾驶吗？

So my focus is on autonomous vehicles. Do you think this approach of third-person watching is amenable to this kind of approach for autonomous driving?

对于自动驾驶，我会说第三人称稍微容易一些。我之所以说第三人称稍微容易，是因为困难的动力学已经被很好地理解了。所以比第一人称容易？还是更容易？我认为对于自动驾驶来说，第三人称和第一人称的区别并不重要；它们非常相似，因为区别实际上在于谁在转动方向盘。让我换个方式说：如何从你现在的位置到达前方几米处，这个问题已经被很好地理解了，这就是第三和第一人称之间的唯一区别。而对于机器人操作，交互力非常复杂，仍然是截然不同的。对于自动驾驶，我认为仍然存在模仿与强化学习的问题。模仿给你更多的信号。我认为模仿缺乏并需要额外机制的地方在于，它通常不考虑目标或目的。当然，有模仿学习的版本，比如逆强化学习类型的模仿，也会考虑目标。我认为那样我们就更接近了。但我认为，如果一辆完全反应式的汽车真的没有目标的概念，它很难很好地泛化。要泛化到你想要的那种通用性，你需要的不只是从行为克隆/监督学习中获得的反应性。

For autonomous driving, I would say third-person is slightly easier. And the reason I'm going to say slightly easier to do third-person is because the hard dynamics are very well understood. So the easier than first-person? Or easier? I think the distinction between third-person and first-person is not a very important distinction for autonomous driving; they're very similar because the distinction is really about who turns the steering wheel. Let me put it differently: how to get from a point where you are now to a point let's say a couple meters in front of you, that's a problem that's very well understood, and that's the only distinction between third and first-person there. Whereas with robot manipulation, interaction forces are very complex and it's still a very different thing. For autonomous driving, I think there is still the question of imitation versus RL. Imitation gives you a lot more signal. I think where imitation is lacking and needs some extra machinery is it doesn't in its normal format think about goals or objectives. Of course there are versions of imitation learning, inverse reinforcement learning type imitation, which also thinks about goals. I think then we're getting much closer. But I think it's very hard to think of a fully reactive car generalizing well if it really doesn't have a notion of objectives. To generalize well to the kind of general that you would want, you'd want more than just that reactivity that you get from just behavioral cloning / supervised learning.

很多工作，无论是自对弈还是模仿学习，都会从模拟、有效的模拟中受益匪浅。你在物理世界和模拟中做了很多工作。你是否希望模拟的能力越来越强，最终无限循环，以至于我们在物理世界中操作所需的大部分内容都可以模拟到可以直接转移到物理世界的程度？还是我们离那还很远？

A lot of the work, whether self-play or imitation learning, would benefit significantly from simulation, from effective simulation. And you're doing a lot of stuff in the physical world and in simulation. Do you have hope for greater and greater power of simulation, loop being boundless eventually, to where most of what we need to operate in the physical world could be simulated to a degree that's directly transferable to the physical world? Or are we still very far away from that?

我认为我们甚至可以在某种意义上重新表述这个问题。模拟的力量：模拟器越来越好，当然变得更强，我们可以在模拟中学到更多。但还有另一个版本，你说模拟器甚至不必那么精确，只要它有一定代表性。与其试图得到一个足够精确的模拟器，在其中学习并很好地转移到现实世界，我打算构建许多模拟器，一个模拟器集成。其中任何一个都不足以代表现实世界，以至于你在其中训练就能成功，但如果你在所有模拟器中训练，那么就会有一个在所有模拟器中都表现良好的东西。现实世界就只是，你知道，另一个模拟器，与任何一个都不完全相同，但只是另一个，来自模拟器分布的一个样本。

I think we could even rephrase that question in some sense. The power of simulation: simulators get better and better, of course become stronger, and we can learn more in simulation. But there's also another version, which is where you said the simulator doesn't even have to be that precise as long as it's somewhat representative. Instead of trying to get one simulator that is sufficiently precise to learn in and transfer really well to the real world, I'm going to build many simulators, an ensemble of simulators. Not any single one of them is sufficiently representative of the real world such that it would work if you train in there, but if you train in all of them, then there is something that's good in all of them. The real world will just be, you know, another one that's not identical to any one of them, but just another one of them, another sample from the distribution of simulators.

正是。我们确实生活在一个模拟中，所以这就像，哦，另一个。我不确定那个视频。如果真是这样，那肯定是一个非常先进的模拟器。是的，相当不错。

Exact. We do live in a simulation, so this is just like, oh, one other one. I'm not sure about that video. It's definitely a very advanced simulator if it is. Yeah, it's pretty good one.

我和 Russell 谈过这个，你也会想一点。当然你确实在努力构建这些系统，但你会考虑人工智能的未来吗？很多人担心安全问题。当你构建在物理世界中运行的机器人时，你如何看待人工智能安全？你如何以工程的方式、系统的方式处理这个问题？

I've talked to Russell about this, something you think about a little bit too. Of course you're really trying to build these systems, but do you think about the future of AI? A lot of people have concerns about safety. How do you think about AI safety as you build robots that are operating in the physical world? How do you approach this problem in an engineering kind of way, in a systematic way?

当机器人做事时，你需要担心几个安全概念。一个是机器人身体强壮，当然可能造成很大的伤害，汽车也一样，我们可以认为汽车在某种程度上是机器人。而且这可能是完全无意的。所以这可能不是那种长期的人工智能安全问题，比如'好吧，人工智能比我们聪明，现在我们该怎么办'，而可能非常实际：'好吧，这个机器人，如果它犯错了，结果会怎样？'当然，模拟在这方面也很有用，可以在模拟中测试。这是一个难题。我一直在想，让我们回到驾驶。很多人很了解驾驶。当然，我们如何测试一个人的驾驶能力？要拿到驾照，他们真正做了什么？我的意思是，你填一些测试，然后开车。在加州郊区，驾驶考试就是你开车绕街区一圈，靠边停车，成功通过一个停车标志，然后你知道你再靠边停车，就差不多完成了。然后你会想，'好吧，如果一辆自动驾驶汽车做到了这些，你会相信它能开车吗？'然后你会说，'不，这不足以让我信任。'但不知何故，对于人类，我们已经发现，某人能做到这些就代表他们能做更多。

When a robot is doing things, you kind of have a few notions of safety to worry about. One is that the robot is physically strong and of course could do a lot of damage, same for cars which we can think of as robots in some way. And this could be completely unintentional. So it could be not the kind of long-term AI safety concerns that 'okay, AI is smarter than us and now what do we do', but it could be just very practical: 'okay, this robot, if it makes a mistake, what's the result going to be?' Of course simulation comes in a lot there too, to test in simulation. It's a difficult question. I'm always wondering, let's go back to driving. A lot of people know driving well. Of course, what do we do to test somebody for driving, right? To get a driver's license, what do they really do? I mean, you fill out some test and then you drive. And I mean, perform in suburban California, the driving test is just you drive around the block, pull over, you do a stop sign successfully, and then you know you pull over again and you're pretty much done. And you're like, 'okay, if a self-driving car did that, would you trust it that it can drive?' And be like, 'no, that's not enough for me to trust.' But somehow for humans, we've figured out that somebody being able to do that is representative of them being able to do a lot more.

还有很多其他事情。所以我认为，某种程度上，我们必须找出有代表性的测试，来衡量如果你能做到这一点，你真正能做什么。当然，测试是必须的，你必须时刻接受测试。自动驾驶汽车、机器人可以更频繁地测试，可能你可以有副本，它们因为使用相同的神经网络而被认为是相同的。但我仍然觉得，我们没有这种针对机器人的单元测试或适当的测试。我认为这是一个非常有趣的研究方向，尤其是当你更新东西时，你的软件改进，你有了更好的自动驾驶套件，你更新了。你怎么知道它确实在所有方面都比以前更强大，而且没有引入任何坏的东西？所以我认为这是一个非常有趣的研究方向，目前还没有真正的解决方案，除了某种程度上的强制测试，因为我们说，好吧，你通过了驾驶考试，你现在可以上路了，而且你必须每百万或千万英里才发生一次事故，与那简短的测试相比，这相当惊人。是的，这正在做。

A lot of other things. And so I think somehow for you must we figured out representative tests of what it means if you can do this, what you can really do. Of course testing you must, you must all want to be tested at all times. Self-driving cars, the robots can be tested more often, probably you can have replicas that get tested and are known to be identical because they use the same neural net and so forth. But still I feel like we don't have this kind of unit tests or proper tests for robots. And I think there's something very interesting to be thought about there, especially as you update things, your software improves, you have a better self-driving car suite, you updated. How do you know it's indeed more capable on everything than what you had before, that you didn't have any bad things creep into it? So I think that's a very interesting direction of research that there is no real solution yet, except that's somehow for you must we do because we say okay, you have a driving test, you passed, you can go on the road now, and you must have accidents every like a million or ten million miles, something pretty phenomenal compared to that short test. Yeah, that is being done.

那么让我问你，你提到吴恩达以身作则向你展示了善良的价值。你认为政策空间，对人类和 AI 好的政策，是充满善良的政策，还是相反，剥削甚至邪恶的政策？所以如果你只看我们作为人类所运作的政策的海洋，或者如果 AI 系统必须在这个现实世界中运作，你认为找到充满善良的政策很容易，就像我们自然陷入其中，还是一个非常困难的优化问题？

So let me ask you, you mentioned that Andrew Ng by example showed you the value of kindness. And do you think the space of policies, good policies for humans and for AI, is populated by policies that are with kindness or ones that are the opposite, exploitation, even evil? So if you just look at the sea of policies we operate under as human beings, or if AI system had to operate in this real world, do you think it's really easy to find policies that are full of kindness, like we naturally fall into them, or is it like a very hard optimization problem?

我的意思是，人类有两种优化在发生，对吧？对你来说，大多数是长期的优化，进化为我们做的，我们天生倾向于喜欢某些东西，这有时让我们的学习更容易，因为我知道像疼痛、饥饿和口渴这样的东西，我们知道这些并不是被教导的，而是天生的。当我们饿的时候我们不开心，当我们渴的时候我们不开心，当我们疼痛的时候我们不开心，最终进化把这些内置到我们身上来思考这件事。所以我认为有一种观念，似乎人类总体上进化到倾向于在某些方面相处融洽，但同时也非常有领土意识，以自己部落为中心。似乎这就是我们收敛到的空间。我不是人类学专家，但似乎我们在自己的部落内部非常友好，但需要被教导才能对其他部落友好。如果你看史蒂芬·平克，他在《人性中的善良天使》中很好地强调了这一点，他谈到暴力随着时间的推移持续减少。所以无论我们关注什么，选择什么团队，似乎历史的长期弧线是朝着我们越来越和睦相处的方向发展的。所以我希望如此。

I mean, there is kind of two optimizations happening for humans, right? So for you most was kinda the very long-term optimization which evolution has done for us, and we're kind of predisposed to like certain things, and that's in sometimes what makes our learning easier because I mean we know things like pain and hunger and thirst, and the fact that we know about those is not something that we were taught, that's kind of innate. When we're hungry we're unhappy, when we're thirsty we're unhappy, when we have pain we're unhappy, and ultimately evolution built that into us to think about this thing. So I think there is a notion that it seems somehow humans evolved in general to prefer to get along in some ways, but at the same time also to be very territorial and kind of centric to their own tribe. It seems like that's the kind of space we converge down to. I mean, I'm not an expert in anthropology, but it seems like we're very kind of good within our own tribe, but need to be taught to be nice to other tribes. Well, if you look at Steven Pinker, he highlights it pretty nicely in Better Angels of Our Nature, where he talks about violence decreasing over time consistently. So whatever attention, whatever teams we pick, it seems that the long arc of history goes towards us getting along more and more. So I hope so.

那么你认为有可能教基于强化学习的机器人这种善良，这种与人类互动的能力，这种政策吗？甚至，让我问一个有趣的问题：你认为有可能教一个基于强化学习的机器人去爱一个人，并激发那个人也爱机器人吗？就像一个基于强化学习的算法导致幸福的婚姻。这是个有趣的问题。

So do you think it's possible to teach RL-based robots this kind of kindness, this kind of ability to interact with humans, this kind of policy? Even, let me ask a fun one: do you think it's possible to teach an RL-based robot to love a human being and to inspire that human to love the robot back? So like an RL-based algorithm that leads to a happy marriage. That's an interesting question.

也许我会用另一个问题来回答，对吧？我的意思是，我会回到这个问题。所以你可以问另一个问题：有些人从与一只非常好的狗互动中获得的幸福感有多接近？我的意思是，狗，你回家，这就是狗做的，它们迎接你，它们兴奋，当你回家看到你的狗时，这让你开心，就像好吧这很激动，它们总是很高兴我在这里。如果它们不迎接你，因为也许你的伴侣带它们去旅行了什么的，你回家时可能就不会那么开心了，对吧？所以狗拥有的推理水平相当复杂，但仍然没有达到人类的推理水平。所以似乎我们甚至不需要达到人类水平的爱的推理就能与人类产生非常强烈的感情。所以我的想法是，为什么不呢？为什么不能用 AI，我们不能达到人类之间或与友好动物等感受到的那种感情水平吗？问题是，这对我们来说是不是一件好事？这又引出了另一个问题，对吧？但我不明白为什么不行。为什么不呢？

Maybe I'll answer it with another question, right? I mean, I'll come back to it. So another question you can have is: how close does some people's happiness get from interacting with just a really nice dog? I mean, dogs, you come home, that's what dogs do, they greet you, they're excited, it makes you happy when you're coming home to your dog, just like okay this is exciting, they're always happy when I'm here. And if they don't greet you because maybe whatever, your partner took them on a trip or something, you might not be nearly as happy when you get home, right? And so the kind of level of reasoning a dog has is pretty sophisticated, but then it's still not yet at the level of human reasoning. And so it seems like we don't even need to achieve human-level love reasoning to get very strong affection with humans. And so my thinking is, why not? Why couldn't with an AI, couldn't we achieve the kind of level of affection that humans feel among each other or with friendly animals and so forth? The question is, is it a good thing for us or not? That misses another question, right? But I don't see why not. Why not?

所以他几乎说爱就是答案。也许他应该说爱是目标函数，然后强化学习就是答案。也许我会彼得。非常感谢。我不想占用你更多时间。非常感谢你今天接受采访。

So he almost says love was the answer. Maybe he should say love is the objective function and then RL is the answer. Maybe I'll Peter. Thank you so much. I don't want to take up more of your time. Thank you so much for talking today.

嗯，谢谢你来访。很高兴你来做客。

Well, thanks for coming by. Great to have you visit.