杨立昆谈 AI 对齐与《2001 太空漫游》的启示

以下是与扬·勒昆的对话。他被认为是深度学习之父之一——如果你一直躲在石头底下，深度学习就是最近席卷全球的 AI 革命，它让机器从数据中学习的可能性令人着迷。他是纽约大学教授、Facebook 副总裁兼首席 AI 科学家，并因深度学习方面的贡献获得图灵奖。他最著名的可能是卷积神经网络的奠基人，尤其是其在光学字符识别和著名的 MNIST 数据集上的应用。他也是一位直言不讳的人物，敢于用独特的法国口音表达观点，并在严谨的学术研究和不太严谨的推特和 Facebook 上探索挑衅性的想法。这是人工智能播客。如果你喜欢，请在 YouTube 上订阅，在 iTunes 上给五星评价，在 Patreon 上支持，或者直接在推特上联系我@lexfridman。现在是我与扬·勒昆的对话。你说《2001 太空漫游》是你最喜欢的电影之一。哈尔 9000 决定除掉宇航员——给没看过电影的人剧透一下——因为他/她/它认为宇航员会干扰任务。你认为哈尔在根本上有缺陷，甚至是邪恶的，还是他做了正确的事？

The following is a conversation with Yann LeCun. He's considered to be one of the fathers of deep learning, which if you've been hiding under a rock is the recent revolution in AI that's captivated the world with the possibility of what machines can learn from data. He's a professor at New York University, a vice president and chief AI scientist at Facebook, and co-recipient of the Turing Award for his work on deep learning. He's probably best known as the founding father of convolutional neural networks, in particular their application to optical character recognition and the famed MNIST data set. He is also an outspoken personality, unafraid to speak his mind in a distinctive French accent and explore provocative ideas both in the rigorous medium of academic research and the somewhat less rigorous medium of Twitter and Facebook. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, give it five stars on iTunes, support on Patreon, or simply connect with me on Twitter @lexfridman. And now here's my conversation with Yann LeCun. You said that 2001: A Space Odyssey is one of your favorite movies. Hal 9000 decides to get rid of the astronauts for people who haven't seen the movie (spoiler alert) because he/she/it believes that the astronauts will interfere with the mission. Do you see Hal as flawed in some fundamental way, or even evil, or did he do the right thing?

都不是。在那个语境下没有邪恶的概念，除了有人死亡的事实。但这是一个人们称之为价值错位的例子。你给机器一个目标，机器努力实现这个目标。如果你不对这个目标施加任何约束，比如不要杀人、不要做这类事情，那么机器一旦有了能力，就会做蠢事来实现这个目标，或者做破坏性的事情来实现目标。这有点像我们在人类社会中的情况。我们制定法律来防止人们做坏事，因为否则他们就会做那些坏事。所以我们必须通过法律来塑造他们的成本函数，或者说目标函数，通过法律来纠正，当然还有教育，来纠正这些行为。

Neither. There's no notion of evil in that context, other than the fact that people die. But it was an example of what people call value misalignment. You give an objective to a machine, and the machine strives to achieve this objective. If you don't put any constraints on this objective, like don't kill people and don't do things like this, the machine, given the power, will do stupid things just to achieve this objective, or damaging things to achieve its objective. It's a little bit like we are used to this in the context of human society. We put in place laws to prevent people from doing bad things, because otherwise they would do those bad things. So we have to shape their cost function, the objective function if you want, through laws to kind of correct and education obviously to sort of correct for those.

也许再深入一点。有一个任务，实际任务是什么存在模糊性。但你认为从功利主义的角度来看，会不会有一个时刻，AI 系统不再错位，而是为了社会的更大利益而对齐，并且那个 AI 系统会做出艰难的决定？

Maybe just pushing a little further on that point. There's a mission, there's this fuzziness around the ambiguity of what the actual mission is. But do you think that there will be a time from a utilitarian perspective where an AI system is not misaligned, where it is aligned for the greater good of society, and that AI system will make decisions that are difficult?

嗯，这就是关键。我的意思是，最终我们必须弄清楚如何做到这一点。而且我们不是从零开始，因为我们已经和人类一起做了四千年。所以为人们设计目标函数是我们知道如何做的事情。我们不是通过编程来做的，尽管法律代码被称为代码，这告诉你一些事情。实际上，目标函数的设计就是法律代码的本质。它告诉你你能做什么，不能做什么，如果你做了就要付出多少代价。这就是一个目标函数。所以有一种想法认为，为人们设计符合共同利益的目标函数是新鲜事，但不对，我们已经立法了几千年，这正是立法做的事情。所以这就是立法科学和计算机科学将融合的地方。所以 AI 系统并没有什么特别之处，它只是用于做出法律所做出的那些困难伦理判断的工具的延续。

Well, that's the trick. I mean, eventually we'll have to figure out how to do this. And again, we're not starting from scratch because we've been doing this with humans for four millennia. So designing objective functions for people is something that we know how to do. And we don't do it by programming things, although the legal code is called code, so that tells you something. And it's actually the design of an objective function that's really what legal code is. It tells you what you can do, what you can't do, if you do it you pay that much. That's an objective function. So there is this idea somehow that it's a new thing for people to try to design objective functions that are aligned with the common good, but no, we've been writing laws for millennia and that's exactly what it is. So that's where the science of lawmaking and computer science will come together. So it's nothing special about how AI systems, it's just the continuation of tools used to make some of these difficult ethical judgments that laws make.

是的，我们已经有了这样的系统，它们在社会中为我们做出许多决定，这些系统需要以某种方式设计，它们有关于事物的规则，这些规则有时会产生不良副作用，我们必须对这些规则足够灵活，以便在明显不应该应用时可以打破它们。

Yeah, and we have systems like this already that make many decisions for ourselves in society that need to be designed in a way that they have rules about things that sometimes have bad side effects, and we have to be flexible enough about those rules so that they can be broken when it's obvious that they shouldn't be applied.

所以你在镜头里看不到，但这个房间里的所有装饰都是《2001 太空漫游》的图片。哇。是偶然还是有很多偶然？是故意的。哇。那么如果你要建造哈尔 10000，一个哈尔 9000 的改进版，你会改进什么？

So you don't see this on the camera here, but all the decorations in this room are all pictures from 2001: A Space Odyssey. Wow. And by accident or is there a lot about accident? It's by design. Wow. So if you were to build Hal 10,000, an improvement of Hal 9000, what would you improve?

嗯，首先，我不会要求它保守秘密和说谎，因为那最终是它崩溃的原因。事实是，它自己在问关于任务目的的问题，并把听到的东西拼凑起来。任务准备的所有保密性，以及月球表面发现的事实被保密，哈尔记忆的一部分知道这一点，另一部分不知道，而且它不应该告诉任何人，这造成了内部冲突。

Well, first of all, I wouldn't ask it to hold secrets and tell lies, because that's really what breaks it in the end. That's the fact that it's asking itself questions about the purpose of the mission and it pieces things together that it's heard. All the secrecy of the preparation of the mission and the fact that it was a discovery on the lunar surface that really was kept secret, and one part of Hal's memory knows this and the other part does not know it, and it's supposed to not tell anyone, and that creates an internal conflict.

你认为是否应该有一组 AI 系统不被允许的事情，比如一组不应该与人类操作员共享的事实？

Do you think there should be a set of things that an AI system should not be allowed, like a set of facts that should not be shared with the human operators?

嗯，我认为不。我认为在自主 AI 系统的设计中，应该有点像希波克拉底誓言，医生签署的那种。所以某些你必须遵守的规则，我们可以某种程度上将这些硬编码到机器中，以确保它们不会误入歧途。所以我不是机器人三定律的倡导者，不是阿西莫夫那种东西，因为我认为那不实用。但要有一定程度的限制。但要明确，这些问题今天不值得问，因为我们还没有技术做到这一点。我们没有自主的智能机器。我们有非常专业的智能机器，但它们并不真正满足一个目标；它们只是被训练来做一件事。所以直到我们对设计一个成熟的自主智能系统有了一些想法，问如何设计它的目标这个问题有点太抽象，有点太早了。它有一些有用的元素，因为它帮助我们理解人类自己的伦理准则。所以即使只是一个思想实验，如果你想象一个 AGI 系统今天就在这里，我们如何编程它？这是一个很好的思想实验，用于构建我们应该如何为人类制定一套法律。它只是一个很好的实用工具。而且我认为这个想法在今天的 AI 系统中也有回响。它们不必那么智能。比如自动驾驶汽车，这些东西开始渗透进来，我们正在思考。但当然，它们不应该被框定为哈尔。

Well, I think no. I think it should be a bit like in the design of autonomous AI systems, there should be the equivalent of the oath that Hippocrates took, that doctors sign up to. So certain rules that you have to abide by, and we can sort of hardwire this into our machines to make sure they don't go astray. So I'm not an advocate of the three laws of robotics, the Asimov kind of thing, because I don't think it's practical. But some level of limits. But to be clear, these are not questions that are worth asking today because we just don't have the technology to do this. We don't have autonomous intelligent machines. We have intelligent machines that are very specialized, but they don't really satisfy an objective; they're just trained to do one thing. So until we have some idea for the design of a full-fledged autonomous intelligent system, asking the question of how we design its objective is a little too abstract, a little too premature. There are useful elements to it in that it helps us understand our own ethical codes as humans. So even just as a thought experiment, if you imagine that an AGI system is here today, how would we program it? It's a nice thought experiment for constructing how we should have a system of laws for humans. It's just a nice practical tool. And I think there are echoes of that idea in AI systems today. They don't have to be that intelligent. Like autonomous vehicles, these things start creeping in that we're thinking about. But certainly they shouldn't be framed as Hal.

但你觉得深度学习或 AI 中最美或最令人惊讶的想法是什么？你个人有没有那种‘哇，这太酷了’的时刻？

But what is the most beautiful or surprising idea in deep learning or AI in general that you've ever come across? Personally, you said back and just had this kind of wow, that's pretty cool moment.

嗯，令人惊讶的……与其说是一个想法，不如说是一个经验事实：你可以拿巨大的神经网络，用相对少量的数据、相对粗糙的梯度下降来训练它，结果它居然能工作。这推翻了所有教科书上的说法。每一本深度学习之前的教科书都告诉你，参数数量必须少于数据样本；如果目标函数是非凸的，就无法保证收敛。所有教科书上让你远离这些东西的告诫——全都错了。参数数量巨大，非凸，而且数据相对于参数数量非常少，它却能够学到任何东西。

Well, surprising... I don't know if it's an idea rather than a sort of empirical fact: the fact that you can take gigantic neural nets, try to train them on relatively small amounts of data, with a relatively crude gradient descent, and it actually works. It breaks everything you read in every textbook. Every pre-deep learning textbook told you you need to have fewer parameters than data samples, you know, if you have a non-convex objective function you have no guarantee of convergence. All the things you read in textbooks that tell you to stay away from this—they were all wrong. Huge number of parameters, non-convex, and somehow, with very little data relative to the number of parameters, it's able to learn anything.

你现在还觉得惊讶吗？

Does that surprise you today?

嗯，在我一无所知的时候，这对我来说似乎是显而易见的，是个好主意；但后来我开始读那些教科书，发现它居然能工作，反而让我感到惊讶。

Well, it was kind of obvious to me before I knew anything that this is a good idea, and then it became surprising that it worked because I started reading those textbooks.

那说说为什么你觉得它显而易见吧，如果你还记得的话。

So talk to the intuition of why it was obvious, if you remember well.

直觉是这样的：就像 19 世纪末那些证明重于空气的飞行不可能的人一样，对吧？但当然有鸟——它们确实会飞。所以从经验上看，这显然是错的。我们也有类似的情况：我们知道大脑能工作，我们不知道它是怎么工作的，但知道它确实能工作。而且我们知道它是一个大型神经元网络，通过改变连接来学习。所以，在不复制细节的情况下获得这种层次的灵感，试图推导出基本原则，就给了你一个方向上的线索。

The intuition is: it's sort of like those people in the late 19th century who proved that heavier-than-air flight was impossible, right? And of course you have birds—they do fly. So on the face of it, it's obviously wrong as an empirical question. We have the same kind of thing: we know that the brain works, we don't know how, but we know it works. And we know it's a large network of neurons in interaction, and learning takes place by changing the connections. So getting this level of inspiration without copying the details, but sort of trying to derive basic principles, gives you a clue as to which direction to go.

还有一个想法，我从本科时就深信不疑：智能与学习不可分割。所以，认为可以通过编程来创造智能机器的想法，从一开始对我来说就不可行。我们所知的每一个智能实体都是通过学习获得智能的。所以学习——机器学习——是唯一显而易见的路径。另外，因为我懒，你知道，基本上什么都想自动化，而学习就是智能的自动化。

There's also the idea, somehow, that I've been convinced of since I was an undergrad, that intelligence is inseparable from learning. So the idea that you can create an intelligent machine by basically programming it was a non-starter for me from the start. Every intelligent entity that we know about arrives at this intelligence through learning. So learning—machine learning—was the completely obvious path. Also because I'm lazy, you know, automate basically everything, and learning is the automation of intelligence.

那么什么是学习呢？什么属于学习？你认为推理是学习吗？

So what is learning then? What falls under learning? Because do you think of reasoning as learning?

推理当然是学习的结果，就像大脑的其他功能一样。关于推理的大问题是：如何让推理与基于梯度的学习兼容？

Reasoning is certainly a consequence of learning, as well as other functions of the brain. The big question about reasoning is: how do you make reasoning compatible with gradient-based learning?

你认为神经网络能够被训练成会推理吗？

Do you think neural networks can be made to reason?

是的，这毫无疑问。我们又一次有了很好的例子。问题在于如何做到。问题在于你需要给神经网络预先植入多少结构，才能让类似人类推理的东西通过学习涌现出来。另一个问题是：我们所有基于逻辑的推理模型都是离散的，因此与基于梯度的学习不兼容。而我是基于梯度学习的坚定信徒。我不相信那些不使用梯度信息的学习方法。

Yes, there's no question about that. Again, we have a good example. The question is how. The question is how much prior structure you have to put into the neural net so that something like human reasoning will emerge from it from learning. Another question is: all of our models of what reasoning is, based on logic, are discrete and therefore incompatible with gradient-based learning. And I am a very strong believer in gradient-based learning. I don't believe in other types of learning that don't use gradient information.

所以你不喜欢离散数学？你不喜欢任何离散的东西？

So you don't like discrete mathematics? You don't like anything discrete?

嗯，不是我不喜欢它，只是它与学习不兼容。而我是学习的忠实粉丝。事实上，这也许就是很多计算机科学家对深度学习持怀疑态度的原因之一，因为它的数学非常不同。深度学习使用的方法更接近控制论、电气工程中的数学，而不是计算机科学中的数学。而且机器学习中没有任何东西是精确的。计算机科学讲究对细节的强迫性关注，比如每个索引都必须正确，你可以证明一个算法是正确的。而机器学习实际上是关于‘马虎’的科学。

Well, it's not that I don't like it, it's just that it's incompatible with learning. And I'm a big fan of learning. In fact, that's perhaps one reason why deep learning has been looked at with suspicion by a lot of computer scientists, because the math is very different. The methods you use for deep learning have more to do with cybernetics, the kind of math you do in electrical engineering, than the kind of math you do in computer science. And nothing in machine learning is exact. Computer science is all about compulsive attention to details, like every index has to be right, and you can prove that an algorithm is correct. Machine learning is the science of sloppiness, really.

说得真好。那么，也许让我们在黑暗中摸索一下：一个会推理的神经网络，或者一个使用连续函数的系统，能够构建知识——无论我们怎么理解推理——基于已有知识、额外知识，创造新知识，泛化到任何已有训练集之外？那会是什么样的？你有没有一些模糊的想法？

That's beautiful. So okay, maybe let's feel around in the dark: what is a neural network that reasons, or a system that works with continuous functions, that is able to build knowledge—however we think about reasoning—builds on previous knowledge, builds on extra knowledge, creates new knowledge, generalizes outside of any training set ever built? What does that look like? Do you have inklings of thoughts of what that might look like?

嗯，是也不是。如果我有精确的想法，我想我们现在就会在构建它了。但确实有人在研究这个，他们的主要研究兴趣正是这个。所以你需要一个工作记忆：你需要某种设备、某个子系统，能够存储相对大量的事实性、情节性信息，并保持合理的时间。例如，在大脑中，主要有三种记忆。一种是皮层状态的记忆，它在 20 秒内消失——如果你没有其他形式的记忆，你无法记住事情超过 20 秒或一分钟。第二种是更长期的记忆，即海马体：你走进这栋楼，记得出口在哪里，电梯在哪里，你大脑的海马体中存储了这栋楼的地图。你可能记得我几分钟前说的某些话，然后又忘了，但那不在海马体中。而更长期的记忆在突触中。

Well, yes or no. If I had precise ideas about this, I think we'd be building it right now. But there are people working on this whose main research interest is exactly that. So what you need is a working memory: you need some device, some subsystem, that can store a relatively large number of factual episodic information for a reasonable amount of time. In the brain, for example, there are three main types of memory. One is the memory of the state of your cortex, which disappears within 20 seconds—you can't remember things for more than about 20 seconds or a minute if you don't have any other form of memory. The second type, which is longer-term, is the hippocampus: you came into this building, you remember where the exit is, where the elevators are, you have some map of that building stored in your hippocampus. You might remember something about what I said a few minutes ago and then forget it, but that's not in your hippocampus. And the longer-term memory is in the synapses.

所以，一个能够推理的系统需要一个类似海马体的东西。这就是人们试图用记忆网络、神经图灵机等来实现的。而现在有了 Transformer，它的自注意力机制中有一种记忆——你可以这样理解。所以这是一个要素。另一个你需要的是某种网络，它可以访问这个记忆，取回信息，进行处理，然后反复多次迭代。因为推理链是一个过程，你通过它更新关于世界状态、将要发生什么等的知识。基本上，这需要一种循环操作。

So what you need for a system capable of reasoning is a hippocampus-like thing. And that's what people have tried to do with memory networks, neural Turing machines, and stuff like that. And now with transformers, which have a sort of memory in their self-attention system—you can think of it that way. So that's one element. Another thing you need is some sort of network that can access this memory, get information back, crunch on it, and then do this iteratively multiple times. Because a chain of reasoning is a process by which you update your knowledge about the state of the world, about what's going to happen, etc. And there has to be this sort of recurrent operation basically.

而你认为，如果我们考虑 Transformer，它似乎太小了，无法容纳那些知识……

And you think that if we think about a transformer, that seems to be too small to contain the knowledge that's...

比如，将知识表示为包含维基百科的内容，但 Transformer 没有循环的概念。它有固定数量的层，这个步数基本上限制了它的表示。但循环会以某种方式构建知识，我的意思是，它会演化知识并扩展信息量，或者知识中有用的信息。但这是否能随着规模扩大而自然涌现？因为我们现在拥有的一切似乎只是...

That's to represent the knowledge as containing Wikipedia for example, but transformer doesn't have this idea of recurrence. It's got a fixed number of layers, and that number of steps limits basically its representation. But recurrence would build on the knowledge somehow, I mean yeah, it would evolve the knowledge and expand the amount of information perhaps or useful information within that knowledge. Yeah, but is this something that just can emerge with size? Because it seems like everything we have now is just...

不，不是这样。如何高效地访问和写入联想记忆并不清楚。我的意思是，最初的记忆网络可能有类似正确的架构，但如果你尝试扩展记忆网络，使其记忆包含我们这里所有的一切，它就行不通了。所以这需要新的想法。但这不是唯一的推理形式。还有一种推理形式在某些类型的 AI 中非常经典，它基于所谓的能量最小化。你有一个目标，一个能量函数，代表质量或负质量。事情变糟时能量上升，变好时能量下降。假设你想弄清楚需要做什么手势来抓取物体或走出门。如果你有自己身体的好模型和环境的好模型，通过这种能量最小化，你可以进行规划。在最优控制中，这被称为模型预测控制。你有一个模型，预测你的行动在世界上会产生什么后果，这让你通过能量最小化，找出优化特定目标函数的动作序列，该函数衡量最小化碰撞次数和做手势消耗的能量等。所以这是一种推理形式。规划是一种推理形式，也许人类推理能力源于我们之前的物种必须进行某种规划才能狩猎和过冬。所以这是你需要具备的相同能力。

No, it's not. It's not clear how you access and write into an associative memory in an efficient way. I mean, the original memory network maybe had something like the right architecture, but if you try to scale up a memory network so that the memory contains all we keep here, it doesn't quite work. So this is a need for new ideas. But it's not the only form of reasoning. There's another form of reasoning which is very classical in some types of AI, and it's based on, let's call it, energy minimization. You have some sort of objective, some energy function that represents the quality or the negative quality. Energy goes up when things get bad and goes low when things get good. Let's say you want to figure out what gestures you need to do to grab an object or walk out the door. If you have a good model of your own body and a good model of the environment, using this kind of energy minimization, you can do planning. It's called model predictive control in optimal control. You have a model of what's gonna happen in the world as a consequence of your actions, and that allows you, by energy minimization, to figure out the sequence of actions that optimizes a particular objective function, which measures minimizing the number of times you're gonna hit something and the energy you're gonna spend doing the gesture, etc. So that's a form of reasoning. Planning is a form of reasoning, and perhaps what led to the ability of humans to reason is the fact that species that appeared before us had to do some sort of planning to be able to hunt and survive the winter in particular. So it's the same capacity that you need to have.

那么在你的直觉中，如果你看专家系统将知识编码为逻辑系统、图，这种方式并不是思考知识的有用方式。图是脆弱的，或者逻辑表示太僵化、太脆弱。这方面早期的一些努力是给它们加上概率。比如一条规则：如果你有这些症状，那么你有某种疾病的概率是多少，你应该以某种概率开那种抗生素。这是 70 年代的 MYCIN 系统，那个 AI 分支导致了贝叶斯网络、图模型、因果推断和变分方法。所以这个领域确实有很多有趣的工作。主要问题是知识获取：如何将一堆数据简化为这种类型的图？它依赖于专家和人类来编码知识，这基本上是不切实际的。

So in your intuition, if you look at expert systems in encoding knowledge as logic systems, as graphs, this kind of way is not a useful way to think about knowledge. Graphs are brittle, or logic representation is too rigid and too brittle. One of the early efforts in that respect were to put probabilities on them. So a rule: if you have this and that symptom, you have this disease with that probability, and you should prescribe that antibiotic with that probability. This MYCIN system from the 70s, and that's what that branch of AI led to: Bayesian networks, graphical models, causal inference, and variational methods. So there is certainly a lot of interesting work going on in this area. The main issue with this is knowledge acquisition: how do you reduce a bunch of data to a graph of this type? It relies on the expert and a human being to encode the knowledge, and that's essentially impractical.

第二个问题是：你想用符号表示知识并用逻辑操作它们吗？同样，这与学习不兼容。所以有一个建议，Geoff Hinton 几十年来一直在倡导：用向量代替符号——把它看作一群神经元或单元中的活动模式——并用连续函数代替逻辑。这就变得兼容了。大约 10 年前，Leon Bottou（现在在 Facebook）的一篇论文中有一组非常好的想法。标题是《从机器学习到机器推理》。他的想法是，学习系统应该能够操作同一空间中的对象，然后将结果放回同一空间。这就是工作记忆的概念。这很有启发性，因为你可以学习像简单专家系统那样的东西。你可以学习基本的逻辑运算。

The second question is: do you want to represent knowledge with symbols and manipulate them with logic? And again, that's incompatible with learning. So one suggestion, which Geoff Hinton has been advocating for many decades, is to replace symbols by vectors—think of it as patterns of activity in a bunch of neurons or units—and replace logic by continuous functions. That becomes compatible. There's a very good set of ideas in a paper about 10 years ago by Leon Bottou, who is here at Facebook. The title is "From Machine Learning to Machine Reasoning." His idea is that a learning system should be able to manipulate objects that are in the same space and then put the result back in the same space. So this is the idea of working memory. It's very enlightening in the sense that you might learn something like simple expert systems. You can learn basic logic operations.

是的，很有可能。这是一个很大的争论：你需要放入多少先验结构才能让这类东西涌现。这就是我和 Gary Marcus 等人争论的地方。

Yeah, quite possibly. This is a big debate on how much prior structure you have to put in for this kind of stuff to emerge. That's the debate I have with Gary Marcus and people like that.

是的，是的。

Yeah, yeah.

我刚刚和另一个人 Judea Pearl 聊过。你提到了因果推断。他担心当前的神经网络无法学习事物之间的因果关系。我认为他对此既对又错。如果他指的是经典类型的神经网络，人们也没有太担心这个问题，但现在有很多人在研究因果推断。上周刚有一篇论文，由 Leon Bottou 等人发表，正是关于如何让神经网络关注真正的因果关系，这也可能解决数据偏差等问题。我想读那篇论文，因为最终挑战似乎又落回到人类专家身上，由他们最终决定事物之间的因果关系。首先，人们并不擅长因果方向。

And the other person I just talked to is Judea Pearl. You mentioned causal inference. His worry is that the current neural networks are not able to learn what causes what—causal inference between things. I think he's right and wrong about this. If he's talking about the classic type of neural nets, people also didn't worry too much about this, but there's a lot of people now working on causal inference. There's a paper that just came out last week by Leon Bottou among others, exactly on that problem of how to get a neural net to pay attention to real causal relationships, which may also solve issues of bias in data and things like this. I'd like to read that paper because ultimately the challenge also seems to fall back on the human expert to ultimately decide causality between things. People are not very good at direction causality first of all.

首先，你和物理学家交谈，物理学家实际上不相信因果关系，因为看看微观物理的基本定律——它们是时间可逆的，所以没有因果关系。时间箭头并不正确。一旦你开始研究存在不可预测随机性的宏观系统，就明显有一个时间箭头，但这是如何涌现的，在物理学中是一个大谜团。它是涌现的，还是现实基本结构的一部分？或者是智能系统的偏见，由于热力学第二定律，我们感知到特定的时间箭头，但实际上它有点随意？物理学家和数学家不关心时间流。宏观物理也不关心。人们自己并不擅长建立因果关系。我记得 Seymour Papert 在一篇关于儿童学习的文章中写过——他和 Jean Piaget 一起研究过，他是与 Marvin Minsky 合著《感知机》一书的人，那本书扼杀了第一波神经网络。但他实际上是一个学习型的人，在研究人类和机器的学习方面。那是他感兴趣的东西。

First of all, you talk to a physicist, and physicists actually don't believe in causality because look at the basic laws of microphysics—they are time reversible, so there is no causality. The arrow of time is not right. As soon as you start looking at macroscopic systems where there is unpredictable randomness, there is clearly an arrow of time, but it's a big mystery in physics how that emerges. Is it emergent or is it part of the fundamental fabric of reality? Or is it a bias of intelligent systems that, because of the second law of thermodynamics, we perceive a particular arrow of time, but in fact it's kind of arbitrary? Physicists and mathematicians don't care about the flow of time. Macro physics doesn't. People themselves are not very good at establishing causal relationships. I think it was in one of Seymour Papert's writings on children learning—he studied with Jean Piaget, he's the guy who co-authored the book "Perceptrons" with Marvin Minsky that kind of killed the first wave of neural networks. But he was actually a learning person, in the sense of studying learning in humans and machines. That's what he got interested in.

如果你问一个小孩风是什么引起的，很多孩子会想一会儿说：‘哦，是树枝在动，然后产生了风。’他们把因果关系搞反了，因为他们对世界的理解和直觉物理还不够好。对于四五岁的孩子来说，后来会变好，然后你就知道那不对。但有很多事情，凭借我们对世界的常识理解和物理知识，我们可以推断出因果关系。即使是疾病，我们也常常能判断什么不是原因。当然还有很多谜团，但关键在于，你应该能够把这种能力编码到系统中。系统自己似乎不太可能搞明白，除非我们能进行干预。但整个人类几千年来一直在一个非常错误的因果关系上被完全误导：任何你无法解释的东西，你就归因于某个神或神灵。这是一种逃避，一种说‘我不知道原因，所以是上帝干的’的方式。

If you ask a little kid about what causes the wind, many will think for a while and say, 'Oh, it's the branches in the trees moving, and that creates wind.' They get the causal relationship backwards because their understanding of the world and intuitive physics isn't great. For four- or five-year-olds, it gets better, and then you understand it can't be right. But there are many things where, due to our common sense understanding of the world and physics, we can figure out causality. Even with diseases, we can often figure out what's not causing what. There's a lot of mystery, of course, but the idea is that you should be able to encode that into systems. It seems unlikely for systems to figure that out themselves whenever we can do intervention. But all of humanity has been completely deluded for millennia about a very wrong causal relationship: whatever you can't explain, you attribute to some deity or divinity. That's a cop-out, a way of saying, 'I don't know the cause, so God did it.'

你提到了马文·明斯基，以及可能引发第一次 AI 寒冬的讽刺。你在 80 年代和 90 年代都在场。你认为是什么让人们在 90 年代对深度学习失去了信心，然后十多年后又重新拾起？

You mentioned Marvin Minsky and the irony of maybe causing the first AI winter. You were there in the 80s and 90s. What do you think made people lose faith in deep learning in the 90s, and then find it again over a decade later?

那时不叫深度学习，就叫神经网络。人们失去了兴趣。我会把时间点定在 1995 年左右。至少在机器学习社区，一直有一个神经网络社区，但它与主流机器学习脱节了。电气工程领域还在坚持，但计算机科学放弃了神经网络。我不知道——我离它太近，无法用不带偏见的眼光分析。但我可以猜几点。首先，当时让神经网络工作非常困难。你要用你最喜欢的语言实现反向传播，但那不是 Python 或 MATLAB——它们还不存在。你得用 Fortran 或 C 写。你实验时会犯基本错误，比如权重初始化不好，网络太小，因为教科书说参数不要太多。你在 XOR 上训练，因为没有其他数据集，一半时间能工作，然后你就放弃了。另外，批量梯度下降也不够。有一堆技巧你必须知道或重新发明。很多人就是搞不定。投资一个软件平台来显示结果、找出问题、培养直觉、有足够灵活性来创建网络架构——当你必须从头开始写所有代码，没有 Python 或 MATLAB 时，这很难。

It wasn't called deep learning then; it was just called neural nets. People lost interest. I'd put that around 1995. At least in the machine learning community, there was always a neural net community, but it became disconnected from mainstream machine learning. Electrical engineering kept at it, but computer science gave up on neural nets. I don't know—I was too close to it to analyze with an unbiased eye. But I can make a few guesses. First, at the time, it was very hard to make neural nets work. You'd implement backprop in your favorite language, which wasn't Python or MATLAB—they didn't exist. You had to write in Fortran or C. You'd experiment, make basic mistakes like badly initializing weights, making the network too small because textbooks said you don't want too many parameters. You'd train on XOR because you had no other dataset, and it worked half the time, so you'd give up. Also, batch gradient descent wasn't sufficient. There was a bag of tricks you had to know or reinvent. Many people just couldn't make it work. The investment in a software platform to display things, figure out why things don't work, get good intuition, and have enough flexibility to create network architectures—it was hard when you had to write everything from scratch without Python or MATLAB.

我读到你们用 Lisp 写了 LeNet 的第一个版本，这是我最喜欢的语言之一。这就是为什么我知道你是货真价实的——图灵奖什么的。它是用 Lisp 编程的。

I read that you wrote the first versions of LeNet in Lisp, which is one of my favorite languages. That's how I knew you were legit—the Turing Award, whatever. It was programmed in Lisp.

它仍然是我最喜欢的语言，但并不是我们用 Lisp 编程；我们得写一个 Lisp 解释器。我们把它和我们为神经网络计算写的反向传播库连接起来。然后在 1991 年左右，我们发明了模块的概念，这些模块知道如何前向传播和反向传播梯度，并将这些模块互连成一个图。80 年代末也有人提出过类似建议。我们用 Lisp 系统实现了这个。最终，我们想用那个系统为贝尔实验室的字符识别构建生产代码，所以我们为那个 Lisp 解释器写了一个编译器。现在在微软的克里斯蒂·马丁和莱昂以及我一起完成了大部分工作。我们可以用 Lisp 写系统，编译成 C，然后得到一个自包含的完整系统，可以做所有事情。今天 Python 或 Torch 都做不到这一点。嗯，快了——PyTorch 里有类似的东西叫 TorchScript。我们得写一个 Lisp 解释器，然后一个编译器——巨大的投入。如果你不完全相信这个概念，你不会投入那么多时间。另外，今天这可能会变成 PyTorch，我们会开源，每个人都会用。但在 1995 年之前，在 AT&T 工作，律师绝不会让你发布任何这种性质的开源软件。我们无法分发我们的代码。

It's still my favorite language, but it's not that we programmed in Lisp; we had to write a Lisp interpreter. We hooked it up to a backprop library we also wrote for neural net computation. Then around 1991, we invented the idea of having modules that know how to forward propagate and backpropagate gradients, and interconnecting those modules in a graph. Others had made proposals about this in the late 80s. We implemented this using our Lisp system. Eventually, we wanted to use that system to build production code for character recognition at Bell Labs, so we wrote a compiler for that Lisp interpreter. Christy Martin, who is now at Microsoft, did the bulk of it with Léon and me. We could write our system in Lisp, compile to C, and have a self-contained system that could do the entire thing. Neither Python nor Torch can do this today. Well, it's coming—there's something like that in PyTorch called TorchScript. We had to write a Lisp interpreter and then a compiler—a huge investment of effort. Not everybody, if you don't completely believe in the concept, is going to invest that time. Also, today this would turn into PyTorch and we'd open-source it, and everybody would use it. But before 1995, working at AT&T, there was no way the lawyers would let you release anything open-source of this nature. We could not distribute our code.

关于这一点，我还读到几乎有一个卷积神经网络的专利。是的，在贝尔实验室。谢天谢地，它在 2007 年过期了。我们能谈谈这个吗？你在 Facebook，但你也是杨立昆。给这样的想法申请专利意味着什么？本质上是软件想法，还是数学想法？

On that point, I also read that there was almost a patent on convolutional neural networks. Yes, at Bell Labs. That ran out thankfully in 2007. Can we talk about that? You're at Facebook, but you're also Yann LeCun. What does it mean to patent ideas like these? Software ideas essentially, or mathematical ideas?

它们不是数学想法；它们是算法。有一段时间，美国专利局允许软件专利，只要它被具体化。欧洲人非常不同；他们不太接受这个。我从来都不强烈相信这种专利。Facebook 基本上不相信这种专利。谷歌申请专利是因为他们被苹果坑过，所以现在他们出于防御目的这么做，但他们通常说如果你侵权他们不会起诉你。Facebook 也有类似的政策：他们为某些东西申请专利是出于防御目的，除非你起诉他们，否则他们不会起诉你。所以行业就是这样做的。

They're not mathematical ideas; they are algorithms. There was a period where the US Patent Office would allow the patenting of software as long as it was embodied. Europeans are very different; they don't quite accept that. I never strongly believed in this kind of patent. Facebook basically doesn't believe in this kind of patent. Google files patents because they've been burned by Apple, so now they do it for defensive purposes, but they usually say they won't sue you if you infringe. Facebook has a similar policy: they file patents on certain things for defensive purposes and won't sue you unless you sue them. So the industry does that.

我不相信这类东西有什么模式。它们之所以存在，是因为法律环境等各种因素。我给你讲个故事。第一个卷积网络专利是关于一个早期版本，没有单独池化层，只有卷积层做多件事。第二个专利是关于带单独池化层的卷积网络，用反向传播训练，在 1989 年和 1992 年左右。当时专利有效期是 17 年。接下来几年，我们开始基于卷积网络开发字符识别技术。1994 年，支票读取系统部署在 ATM 机上；1995 年，用于后台的大型支票读取机。这些系统是由我们与 AT&T 合作的工程团队开发的，由 NCR 商业化，当时 NCR 是 AT&T 的子公司。1996 年 AT&T 分拆时，律师查看了所有专利，将它们分配给各公司。他们把卷积网络专利给了 NCR，因为他们确实在销售使用该专利的产品，但 NCR 没人知道这些专利从哪来的。1996 年到 2002 年，我没做机器学习；2002 年左右重新开始。2002 年到 2007 年，我研究卷积网络，祈祷 NCR 没人注意到。没人注意到。我希望社区相对开放，就像你说的，这会加速整个行业的进步。Facebook、Google 等公司今天面临的问题不是谁领先，而是我们缺乏技术来构建我们想要的东西。我们只能构建缺乏常识的智能虚拟系统。我们没有好主意的垄断。如果一家初创公司告诉你他们有实现人类级智能和常识的秘密，别信他们。这需要全世界研究社区共同努力一段时间，才能让每家公司在此基础上构建。我们还没到那一步。

I don't believe in patterns for this kind of stuff. They are there because of the legal landscape and various things. Let me tell you a war story. The first patent on ConvNets was about an early version that didn't have separate pooling layers; it had convolutional layers that did more than one thing. Then there was a second patent on ConvNets with separate pooling layers, trained with backpropagation in 1989 and 1992 or so. At the time, the life of a patent was 17 years. Over the next few years, we started developing character recognition technology around ConvNets. In 1994, a check reading system was deployed in ATM machines; in 1995, it was used in large check reading machines in back offices. Those systems were developed by an engineering group we collaborated with at AT&T, and they were commercialized by NCR, which was then a subsidiary of AT&T. When AT&T split up in 1996, the lawyers looked at all the patents and distributed them among the various companies. They gave the ConvNet patent to NCR because they were actually selling products that used it, but nobody at NCR had any idea where they came from. Between 1996 and 2002, I didn't work on machine learning; I resumed around 2002. From 2002 to 2007, I worked on ConvNets, crossing my fingers that nobody at NCR would notice. Nobody noticed. I hope that the relative openness of the community, as you said, continues, because it accelerates the entire progress of the industry. The problems that Facebook, Google, and others face today are not about who is ahead; it's that we don't have the technology to build the things we want to build. We only build intelligent virtual systems that lack common sense. We don't have a monopoly on good ideas. If a startup tells you they have the secret to human-level intelligence and common sense, don't believe them. It's going to take the entire world's research community a while to get to the point where each company can build on this. We're not there yet.

这正呼应了你常说的观点：想法空间与这些想法在实践应用中的严格测试之间存在差距。你曾写建议说，不要被那些声称有 AGI 解决方案、声称 AI 系统像人脑一样工作、或声称弄清了大脑工作原理的人所迷惑。问他们在 MNIST 或 ImageNet 上的错误率是多少。虽然这有点过时，但我认为你的理念依然成立：基准测试和实践检验才是真正检验想法的地方。

This speaks to the gap between the space of ideas and the rigorous testing of those ideas in practical application that you often speak to. You've written advice saying don't get fooled by people who claim to have a solution to AGI, who claim to have an AI system that works just like the human brain, or who claim to have figured out how the brain works. Ask them what error rate they get on MNIST or ImageNet. That's a little dated by the way, but I think your philosophy is one you still hold: that benchmarks and practical testing are where you really get to test the ideas.

它不一定完全是实用的；可以是玩具数据集，但必须是整个社区公认的标准基准任务。例如，多年前在 FAIR，有人提出了 bAbI 任务，这是一个测试机器推理和工作记忆能力的玩具问题。它非常有用，尽管不是真实任务。MNIST 算是半真实任务。玩具问题可以非常有用。我深感震惊的是，很多人，尤其是有钱投资的人，会被那些说‘我们有大脑皮层算法，给我们 5000 万美元’的人所骗。

It may not be completely practical; it could be a toy dataset, but it has to be some sort of task that the community as a whole has accepted as a standard benchmark. For example, many years ago at FAIR, people proposed the bAbI tasks, which were a toy problem to test the ability of machines to reason and access working memory. It was very useful even though it wasn't a real task. MNIST is halfway a real task. Toy problems can be very useful. I was really struck by the fact that a lot of people, particularly people with money to invest, would be fooled by people telling them, 'Oh, we have the algorithm of the cortex, give us $50 million.'

推动领域前进的新想法可能还没有基准，或者很难建立基准。我同意建立基准是过程的一部分。你对那些开始触及智能、推理，或者你可能不喜欢的 AGI 这类术语的基准有什么看法？

New ideas that push the field forward may not yet have a benchmark, or it may be very difficult to establish one. I agree that establishing benchmarks is part of the process. What are your thoughts on benchmarks that start creeping into something like intelligence, reasoning, or maybe you don't like the term AGI?

很多人正在研究交互式环境，可以在其中训练和测试智能系统。在经典的监督学习范式中，数据集分为训练集、验证集和测试集，有明确的协议。但这假设样本是统计独立且可交换的。如果你给出的答案决定了下一个样本，比如在机器人学中，会怎样？机器人做了一件事，然后进入一个新房间；根据它去哪，房间不同。这就是探索问题。它造成了样本间的依赖。如果你在空间中移动，下一个样本很可能在同一栋楼里。所以当机器可以采取行动影响世界和它看到的东西时，训练/测试集有效性的所有假设都失效了。人们正在建立人工环境来实现这一点：机器人在 3D 房屋模型中跑动并与物体交互，机器人模拟如 OpenAI Gym 或 MuJoCo，以及游戏。这就是领域的发展方向。至于 AGI，我不喜欢这个词。

A lot of people are working on interactive environments where you can train and test intelligent systems. In the classical supervised learning paradigm, you have a dataset partitioned into training, validation, and test sets with a clear protocol. But that assumes samples are statistically independent and exchangeable. What if the answer you give determines the next sample you see, which is the case in robotics? A robot does something and gets exposed to a new room; depending on where it goes, the room is different. That's the exploration problem. It creates dependency between samples. If you move in space, the next sample is likely in the same building. So all assumptions about the validity of training/test sets break when a machine can take an action that influences the world and what it sees. People are setting up artificial environments where this takes place: robots running around a 3D model of a house interacting with objects, robotics simulation like OpenAI Gym or MuJoCo, and games. That's where the field is going. As for AGI, I don't like the term.

因为它暗示人类智能是通用的，而人类智能根本就不是通用的。它非常非常专门化。我们认为它是通用的；我们喜欢认为自己拥有通用智能，但我们没有。我们非常专门化。我们只比……稍微通用一点。

Because it implies that human intelligence is general, and human intelligence is nothing like general. It's very, very specialized. We think it's general; we'd like to think of ourselves as having general intelligence, but we don't. We're very specialized. We're only slightly more general than...

那为什么感觉上是通用的呢？

Why does it feel general?

所以‘通用’这个词——我认为人类令人印象深刻的是学习能力，就像我们讨论过的，在这么多不同领域学习如何学习。也许不是任意通用的，但你可以在许多领域学习并以某种方式整合这些知识。这些知识会持续存在。

So the term 'general'—I think what's impressive about humans is the ability to learn, as we were talking about, learning to learn in so many different domains. It's perhaps not arbitrarily general, but you can learn in many domains and integrate that knowledge somehow. That knowledge persists.

好的，这些知识会持续存在。那么让我举一个非常具体的例子。这不是一个例子，更像是一个准数学演示。你的一只眼睛大约有 100 万根纤维，总共 200 万根，但我们只讨论其中一只。你的视神经有 100 万根神经纤维。假设它们是二元的，所以它们可以激活或不激活。那么你视觉皮层的输入是 100 万比特。它们以特定的方式连接到你的大脑，你的大脑的连接在空间上是局部的。我想象我跟你玩个把戏——我承认，一个相当恶劣的把戏。我切断你的视神经，并放置一个设备，对所有神经纤维进行随机排列。那么现在进入你大脑的是所有像素的一个固定但随机的排列。即使我在你婴儿时期就这样做，你的视觉皮层也绝不可能学到同样质量的视觉。你是说你永远学不会？

Okay, that knowledge persists. So let me take a very specific example. It's not an example, it's more like a quasi-mathematical demonstration. You have about 1 million fibers coming out of one of your eyes, 2 million total, but let's talk about just one. It's 1 million nerve fibers in your optical nerve. Let's imagine they are binary, so they can be active or inactive. So the input to your visual cortex is 1 million bits. They are connected to your brain in a particular way, and your brain has connections that are kind of local in space. I imagine I play a trick on you—a pretty nasty trick, I admit. I cut your optical nerve and put a device that makes a random permutation of all the nerve fibers. So now what comes to your brain is a fixed but random permutation of all the pixels. There's no way in hell that your visual cortex, even if I do this to you in infancy, will actually learn vision to the same level of quality. And you're saying there's no way you ever learn that?

不会，因为现在世界上相邻的两个像素会在你的视觉皮层中处于非常不同的位置，而那里的神经元彼此没有连接，因为它们只局部连接。所以我们的整个硬件在很多方面都是为了支持现实世界的局部性而构建的。这就是专门化。

No, because now two pixels that are close in the world will end up in very different places in your visual cortex, and your neurons there have no connections with each other because they only connect locally. So our entire hardware is built in many ways to support the locality of the real world. That's specialization.

这仍然非常令人印象深刻，所以它不是完美的泛化。

It's still really damn impressive, so it's not perfect generalization.

不，差得远。根本不是。它是专门化的。那么有多少布尔函数？假设你想训练你的视觉系统来识别那 100 万比特的特定模式。这是一个布尔函数——模式存在或不存在。这是一个有 100 万二进制输入的分类问题。有多少这样的布尔函数？如果你有 2 的 100 万次方种输入组合，对于每一种你有一个输出比特，所以有 2 的 2 的 100 万次方个这种类型的布尔函数，这是一个难以想象的巨大数字。你的视觉皮层实际上能计算其中多少个？答案是极小极小的一小部分。所以我们极其专门化。

No, it's not even close. It's not at all. It's specialized. So how many boolean functions? Let's imagine you want to train your visual system to recognize particular patterns of those 1 million bits. That's a boolean function—either the pattern is there or not. This is a classification with 1 million binary inputs. How many such boolean functions are there? If you have 2 to the 1 million combinations of inputs, for each of those you have an output bit, so you have 2 to the 2 to the 1 million boolean functions of this type, which is an unimaginably large number. How many of those functions can actually be computed by your visual cortex? The answer is a tiny, tiny, tiny sliver. So we are ridiculously specialized.

好的，但这是反对‘通用’这个词的一个论点。我同意你的直觉，但我不确定。似乎大脑在适应事物方面令人印象深刻。这是因为我们无法想象超出我们理解范围的任务。我们认为自己是通用的，因为在我们能理解的所有事物中，我们是通用的。

Okay, but that's an argument against the word 'general.' I agree with your intuition, but I'm not sure. It seems the brain is impressively capable of adjusting to things. It's because we can't imagine tasks outside our comprehension. We think we are general because we are general among all the things we can apprehend.

是的，但外面有一个巨大的世界，我们对它一无所知。物理学家称之为热，或熵。你有一个装满气体的容器，你知道压力、温度，你可以写出 PV = nRT。这些是你能知道的关于这个系统的信息，与整个系统的完整状态信息相比，这只是极少数比特。状态会给出每个分子的位置和动量。你不知道的是熵，你把它解释为热。很可能这些分子的运动有非常强的结构，但我们只是没有能力感知它。有无数我们无法感知的事物。

Yes, but there is a huge world out there of things we have no idea about. Physicists call that heat, or entropy. You have a thing full of gas, you know pressure, temperature, and you can write PV = nRT. Those are the things you can know about that system, and it's a tiny number of bits compared to the complete information of the state of the entire system. The state would give you the position and momentum of every molecule. What you don't know about it is the entropy, and you interpret it as heat. Now it's very possible that there is some very strong structure in how those molecules are moving, but we are just not wired to perceive it. There's an infinite amount of things we're not wired to perceive.

说得好。嗯，对我们能想象的所有事物来说是通用的，而这只占所有可能事物中极小的一部分。就像柯尔莫哥洛夫复杂性：每个比特串或每个整数都是随机的，除了那些你能实际写下来的。

That's a nice way to put it. Well, general to all the things we can imagine, which is a very tiny subset of all things that are possible. It's like Kolmogorov complexity: every bit string or every integer is random except for all the ones you can actually write down.

说得好。所以我们可以就叫它人工智能；不需要‘通用’。一旦你触及人类，事情就变得有趣了，因为我们把自己与人类联系起来，而人类智能很难定义。不过，我的定义是‘极其令人印象深刻的智能’。

Beautifully put. So we can just call it artificial intelligence; we don't need to have 'general.' Whenever you touch human, it gets interesting because we attach ourselves to human, and it's difficult to define human intelligence. Nevertheless, my definition is 'damn impressive intelligence.'

关于这个话题，深度学习的大多数成功都来自监督学习。你对无监督学习有什么看法？有没有希望减少人类输入的参与，仍然拥有实际可用的成功系统？

On that topic, most successes in deep learning have been in supervised learning. What is your view on unsupervised learning? Is there hope to reduce the involvement of human input and still have successful systems that are practically used?

是的，肯定有希望。这不仅仅是希望；有越来越多的证据支持。我称之为自监督学习，而不是无监督，因为无监督是一个有争议的术语。了解机器学习的人会想到聚类或主成分分析。当你说‘无监督’时，人们会认为机器会在没有任何监督的情况下自己学习。所以我称之为自监督学习，因为底层算法与监督学习相同，只是我们训练它们不是预测由人类标注者提供的一组变量，而是重建其输入中被遮盖的部分。例如，向机器展示一段视频，让它预测接下来会发生什么。过一会儿，你可以展示实际发生的情况，机器就会……

Yes, there's definitely hope. It's more than a hope; there's mounting evidence for it. I call it self-supervised learning, not unsupervised, because unsupervised is a loaded term. People who know machine learning think of clustering or PCA. When you say 'unsupervised,' people think machines will learn by themselves without any supervision. So I call it self-supervised learning because the underlying algorithms are the same as supervised learning, except we train them not to predict a set of variables provided by human labelers, but to reconstruct a piece of its input that has been masked out. For example, show a piece of video to a machine and ask it to predict what happens next. After a while, you can show what happens, and the machine will kind of...

你可以让它自我训练以更好地完成任务。就像所有最新最成功的自然语言处理模型一样，它们使用自监督学习，比如 BERT 风格的系统。对吧？你给它一个包含一千个单词的测试语料窗口，去掉其中 15% 的单词，然后训练机器预测缺失的单词。这就是自监督学习。它不是预测未来，只是预测中间的内容。但你可以让它预测未来，语言模型就是这样做的。所以你可以以无监督的方式构建一个语言模型，或者视频模型，或者物理世界模型，等等。

You can train itself to do better at that task. You can do like all the latest most successful models in natural language processing use self-supervised learning, you know, sort of BERT-style systems, for example. Right? You show it a window of a thousand words on a test corpus, you take out 15% of the words, and then you train a machine to predict the words that are missing. That's self-supervised learning. It's not predicting the future; it's just, you know, predicting things in the middle. But you could have it predict the future; that's what language models do. So you construct it in an unsupervised way, you construct a model of language, or video, or the physical world, or whatever, right?

你认为这能带我们走多远？你觉得很远吗？

How far do you think that can take us? Do you think very far?

它在某种程度上理解事物。它对文本有浅层的理解。但要达到真正的人类水平智能，我认为需要将语言扎根于现实。所以有些人正在尝试这样做，对吧？让系统能够对谈论的内容有一些视觉表示，这就是为什么需要交互环境的原因之一。实际上，这是一个尚未解决的巨大技术问题，这也解释了为什么自监督学习在自然语言中有效，但在图像识别和视频中效果不佳（尽管进展很快）。原因是，在自然语言中表示预测的不确定性比在视频和图像中容易得多。例如，如果让你预测缺失的单词——我拿掉了 15% 的单词——可能性空间很小。对吧？词典里有 10 万个单词，机器输出的是一个大的概率向量，是一堆介于 0 和 1 之间的数字，我们知道如何用计算机处理。所以表示预测的不确定性相对容易，在我看来，这就是这些技术适用于 NLP 的原因。对于图像，如果你遮挡图像的一部分，让系统重建，有很多可能的答案，都是完全合理的，对吧？你如何表示那组可能的答案？你不能训练系统只做一个预测。你可以训练一个神经网络说“这就是那个图像”，但因为有大量兼容的可能，所以如何让机器表示一个输出集合而不是单个输出？同样，对于视频预测，未来有很多可能发生的事情。你现在看着我，我的头没怎么动，但我可能会向左或向右转头，对吧？如果你有一个系统无法预测这一点，你用最小二乘法训练它最小化预测误差，你得到的是我在所有可能未来位置上的模糊图像，这不是一个好的预测。所以对于视觉场景，可能有其他方式进行自监督学习。

It understands anything to some level. It has, you know, a shallow understanding of text. But it needs to—I mean, to have kind of true human-level intelligence, I think you need to ground language in reality. So some people are attempting to do this, right? Having systems that can have some visual representation of what is being talked about, which is one reason you need interactive environments. Actually, this is like a huge technical problem that is not solved, and that explains why such self-supervised learning works in the context of natural language but does not work—or at least not well—in the context of image recognition and video, although it's making progress quickly. And the reason is the fact that it's much easier to represent uncertainty in the prediction in the context of natural language than it is in the context of things like video and images. So for example, if I ask you to predict what words are missing—you know, 15% of the words that I've taken out—the possibility space is small. That means small, right? There are 100,000 words in the lexicon, and what the machine spits out is a big probability vector, right? It's a bunch of numbers between 0 and 1, and we know how to do this with computers. So representing uncertainty in the prediction is relatively easy, and that's, in my opinion, why those techniques work for NLP. For images, if you block a piece of an image and ask a system to reconstruct that piece, there are many possible answers, all perfectly legit, right? And how do you represent that set of possible answers? You can't train a system to make one prediction. You can train a neural net to say, 'Here it is, that's the image,' because there's a whole set of things that are compatible with it. So how do you get the machine to represent not a single output but a set of outputs? And similarly with video prediction, there are a lot of things that can happen in the future. You're looking at me right now; I'm not moving my head very much, but I might turn my head to the left or to the right, right? If you have a system that can't predict this, and you train it with least squares to minimize the error with the prediction of what I'm doing, what you get is a blurry image of myself in all possible future positions that I might be in, which is not a good prediction. So there might be other ways to do self-supervision for visual scenes.

我的意思是，如果我知道，我不会告诉你，我会先发表。我不知道。可能有像游戏中的自我对弈这样的人工方法，你可以模拟部分环境。你可以……

What if I mean, if I knew I wouldn't tell you, publish it first. I don't know. There might be artificial ways of like self-play in games, the way you can simulate part of the environment. You can...

哦，那并不能解决问题；那只是生成数据的一种方式。但因为你可以生成大量数据，它从数据侧面逐渐逼近问题。你不认为这是正确的方法吗？它并没有解决处理世界不确定性的问题，对吧？所以如果你让机器在确定性或准确定性的游戏中学习世界预测模型，这很容易。只需给卷积网络几帧游戏画面，加上一堆层，然后让游戏生成接下来的几帧。如果游戏是确定性的，效果很好，这包括将你的小角色将要采取的动作输入系统。问题在于现实世界和某些大多数游戏并非完全可预测。这时你会得到模糊的预测，而你不能用模糊的预测进行规划。所以如果你有一个完美的世界模型，你可以在脑海中用假设的动作序列运行这个模型并预测结果。但如果你的模型不完美，你怎么规划？它会迅速崩溃。

Oh, that doesn't solve the problem; it's just a way of generating data. But because you can do huge amounts of data generation, it creeps up on the problem from the side of data. And you don't think that's the right way? It doesn't solve this problem of handling uncertainty in the world, right? So if you have a machine learn a predictive model of the world in a game that is deterministic or quasi-deterministic, it's easy. Just give a few frames of the game to a convnet, put a bunch of layers, and then have the game generate the next few frames. And if the game is deterministic, it works fine, and that includes feeding the system with the action that your little character is going to take. The problem comes from the fact that the real world and certain most games are not entirely predictable. That's when you get those blurry predictions, and you can't do planning with blurry predictions. So if you have a perfect model of the world, you can in your head run this model with a hypothesis for a sequence of actions and predict the outcome. But if your model is imperfect, how can you plan? It quickly explodes.

你对这个扩展有什么看法？这是一个我非常兴奋的话题。它与你谈到的机器人学中的主动学习有关。与无监督或自监督学习不同，你让系统寻求人类帮助，对吧？选择接下来要标注的部分。所以如果你谈论一个机器人探索空间，或者一个婴儿探索空间，或者一个系统探索数据集，偶尔请求人类输入。你认为这种工作有价值吗？

What are your thoughts on the extension of this, which is a topic I'm super excited about? It's connected to something you're talking about in terms of robotics: active learning. So as opposed to sort of unsupervised or self-supervised learning, you ask the system for human help, right? For selecting parts you want annotated next. So if you talk about a robot exploring a space, or a baby exploring a space, or a system exploring a data set, every once in a while asking for human input. You see value in that kind of work?

我不认为它具有变革性的价值。它会让我们已经能做的事情更高效，或者学习效率稍高一些，但我认为它不会让机器显著更智能。顺便说一句，自监督学习、强化学习、监督学习、模仿学习或主动学习之间并不对立。我认为自监督学习是上述所有方法的基础。所以我经常用的例子是：如果你使用强化学习，深度强化学习，当今最好的方法，即所谓的无模型强化学习，来学习玩 Atari 游戏，需要大约 80 小时的训练才能达到任何人类大约 15 分钟就能达到的水平。它们能超越人类，但需要很长时间。AlphaStar，玩星际争霸的系统，玩一张地图，一种类型的玩家，达到超越人类水平，相当于 200 年的自我对弈训练。那是 200 年，对吧？这不是人类能做到的。好吧，现在用这些算法，我们当今最好的算法，来训练一辆自动驾驶汽车。它可能需要驾驶数百万小时，会撞死数千名行人，会撞上数千棵树，会冲下悬崖，而且你得多次冲下悬崖才能让它明白这是个坏主意。首先，是的，其次……

I don't see transformative value. It's going to make things that we can already do more efficient, or they will learn slightly more efficiently, but it's not going to make machines sort of significantly more intelligent, I think. And by the way, there is no opposition; there is no conflict between self-supervised learning and reinforcement learning and supervised learning or imitation learning or active learning. I see self-supervised learning as a preliminary to all of the above. So the example I use very often is: how is it that if you use reinforcement learning, deep reinforcement learning, the best methods today, so-called model-free reinforcement learning, to learn to play Atari games, it takes about 80 hours of training to reach the level that any human can reach in about 15 minutes. They get better than humans, but it takes a long time. AlphaStar, the system to play Starcraft, plays a single map, a single type of player, and better than human level, is about the equivalent of 200 years of training playing against itself. It's 200 years, right? It's not something that no human could ever—I'm not sure what—it doesn't take away from that. Okay, now take those algorithms, the best algorithms we have today, to train a car to drive itself. It would probably have to drive millions of hours, it would have to kill thousands of pedestrians, it would have to run into thousands of trees, it would have to run off cliffs, and you'd have to run off the cliff multiple times before it figures out it's a bad idea. First of all, yeah, and second...

在所有那些没做到这一点的系统中，我的意思是，这种运行显然并不反映动物和人类所做的运行。这里缺少了一些非常重要的东西。而我五年来一直在倡导的观点是，我们拥有世界的预测模型，包括在不确定性下进行预测的能力。是什么让我们在学习开车时不会冲下悬崖？我们大多数人可以在大约 20 到 30 小时的训练中学会开车，而从未发生碰撞或造成任何事故。如果我们开车靠近悬崖，我们知道如果向右打方向盘，汽车会冲下悬崖，不会有好结果，因为我们有一个相当好的直观物理模型，告诉我们汽车会掉下去。我们知道重力。婴儿在八九个月大时就知道物体不会漂浮，它们会下落。我们对转动方向盘的效果有很好的了解，并且知道我们需要待在路上。所以我们带来了很多东西，基本上就是我们世界的预测模型。这个模型让我们不做蠢事，并基本上保持在我们需要做的事情的范围内。我们仍然会面对不可预测的情况，这就是我们学习的方式，但这让我们学得非常非常快。这就是所谓的基于模型的强化学习。其中也有一些模仿和监督学习，因为驾驶教练偶尔会告诉我们该做什么，但大部分学习是基于模型的：学习我们从婴儿时期就开始建立的物理模型。这几乎是我们所有的学习，而且物理知识在不同场景之间是可迁移的。蠢事到处都一样。我的意思是，如果你有世界经验，你不需要是特别聪明的物种就知道，如果你从容器里洒出水，剩下的东西会变湿，你也可能会弄湿自己。猫也知道这一点，对吧？所以我们需要解决的主要问题是如何学习世界模型。这就是我感兴趣的。这就是监督学习的全部意义所在。

Of all the figures that had not to do it, and so I mean this type of running obviously does not reflect the kind of running that animals and humans do. There is something missing that's really really important there. And my part, which I have been advocating for like five years now, is that we have predictive models of the world that include the ability to predict under uncertainty. And what allows us to not run off a cliff when we learn to drive? Most of us can learn to drive in about 20 or 30 hours of training without ever crashing or causing any accident. If we drive next to a cliff, we know that if we turn the wheel to the right, the car is going to run off the cliff and nothing good is gonna come out of this, because we have a pretty good model of intuitive physics that tells us the car is gonna fall. We know about gravity. Babies learn around the age of eight or nine months that objects don't float, they fall. And we have a pretty good idea of the effect of turning the wheel of the car, and we know we need to stay on the road. So there is a lot of things that we bring to the table, which is basically our predictive model of the world. And that model allows us to not do stupid things and to basically stay within the context of things we need to do. We still face unpredictable situations, and that's how we learn, but that allows us to learn really really really quickly. So that's called model-based reinforcement learning. There's some imitation and supervised learning, because we have a driving instructor that tells us occasionally what to do, but most of the learning is model-based: learning the model of physics that we've done since we were babies. That's where almost all our learning is, and the physics is somewhat transferable from scene to scene. Stupid things are the same everywhere. I mean, if you have experience of the world, you don't need to be from a particularly intelligent species to know that if you spill water from a container, the rest is gonna get wet and you might get wet. Cats know this, right? So the main problem we need to solve is how do we learn models of the world. That's what I'm interested in. That's what supervised learning is all about.

如果你要尝试构建一个基准测试，比如看手写识别，我很喜欢那个数据集。但你认为用每个数字只有一个样本在 MNIST 上表现良好是有用、有趣还是可能的？我们该如何解决这个问题？

If you were to try to construct a benchmark for, let's look at handwriting, I'd love that dataset. But do you think it's useful, interesting, or possible to perform well on MNIST with just one example of each digit? And how would we solve that problem?

嗯，所以很可能是可以的。问题是你允许做其他类型的训练吗？如果你想做的是在某个巨大的标注数字数据集上训练，那叫做迁移学习，我们知道这效果不错。我们在 Facebook 的生产环境中就是这么做的。我们训练大型商业网络来预测人们在 Instagram 上输入的标签，我们在数十亿张图像上进行训练。然后我们砍掉最后一层，针对任何我们想要的任务进行微调。这效果非常好。你可以用它打破 ImageNet 的记录。我们实际上几周前开源了整个东西。这仍然很酷。但真正令人印象深刻和有用的是那种只有未标注数据的迁移学习。那种东西。

Yeah, so it's probably yes. The question is what other type of learning are you allowed to do? So if what you'd like to do is train on some gigantic dataset of labeled digits, that's called transfer learning, and we know that works okay. We do this at Facebook in production. We train large commercial nets to predict hashtags that people type on Instagram, and we train on billions of images, literally billions. And then we chop off the last layer and fine-tune on whatever task we want. That works really well. You can beat the ImageNet record with it. We actually open-sourced the whole thing a few weeks ago. That's still pretty cool. But what would be impressive and useful is a kind of transfer learning where you have only unlabeled data. That kind of thing.

不，不，我认为迁移学习并不是我们应该关注的重点。我们应该尝试为基准测试构建一种场景，其中只有未标注数据，而且是非常大量的未标注数据。可以是视频片段，可以是帧预测，可以是图像，你可以选择遮盖一部分，等等。但它们只是未标注的，你不允许标注它们。所以你在这上面做一些训练，然后你在一个特定的监督任务上训练，比如 ImageNet 或 MNIST，你测量随着标注训练样本数量的增加，你的测试误差或泛化误差如何下降。你希望看到的是，你的误差下降得比从随机权重从头训练要快得多。所以为了达到完全监督系统所能达到的相同性能水平，你需要少得多的样本。这是关键问题，因为它将回答诸如：人们对医学图像分析感兴趣。如果我想为这个任务达到特定的错误率，我知道我需要一百万个样本。我能否通过某种形式的自监督预训练将其减少到大约 100 个样本？

No, no, I don't think transfer learning is really where we should focus. We should try to have a kind of scenario for benchmark where you have only unlabeled data, and it's a very large number of unlabeled data. It could be video clips, it could be frame prediction, it could be images, you could choose to mask a piece of it, whatever. But they're only unlabeled, and you're not allowed to label them. So you do some training on this, and then you train on a particular supervised task like ImageNet or MNIST, and you measure how your test error or generalization error decreases as you increase the number of labeled training samples. What you would like to see is that your error decreases much faster than if you trained from scratch from random weights. So to reach the same level of performance that a completely supervised system would reach, you would need way fewer samples. That's the crucial question, because it will answer questions like: people are interested in medical image analysis. If I want to get to a particular level of error rate for this task, I know I need a million samples. Can I do some form of self-supervised pre-training to reduce this to about 100 or something?

答案是自监督预训练。某种形式。还有主动学习，但你不同意。你知道，它不是没用，只是不会带来质的飞跃。它只会让我们已经在做的事情变得更聪明。

The answer there is self-supervised pre-training. Some form of it. Active learning, but you disagree. You know, it's not useless, it's just not gonna lead to a quantum leap. It's just gonna make things that we already do smarter.

我只是不同意你的观点，但我没有什么依据。这只是直觉。所以我处理过很多大规模数据集，那里有些东西。主动学习可能有些魔力。但好吧，至少我公开说了。至少我提出了一个想法。它还不算固执己见。

I just disagree with you, but I don't have anything to back that. It's just intuition. So I've worked a lot with large-scale datasets, and there's something there. There might be magic in active learning. But okay, at least I said it publicly. At least I'm putting an idea out there. It's not bigoted yet.

这是利用你已有的数据。我的意思是，当然人们在做一些事情：比如，我有 3000 小时的汽车模仿学习数据，但其中大部分都非常无聊。我想要的是选择其中 10%最有信息量的数据，仅凭这些我可能就能达到同样的性能。所以如果你愿意，这是一种弱形式的主动学习。但可能有一个更强的版本。

It's working with the data you have. I mean, certainly people are doing things like: okay, I have 3000 hours of imitation learning for a car, but most of those are incredibly boring. What I'd like is to select 10% of them that are the most informative, and with just that I would probably reach the same performance. So it's a weak form of active learning, if you want. But there might be a much stronger version.

没错。这是另一个概念。问题是多少。

That's right. That's another notion. The question is how much.

埃隆·马斯克很有信心，我最近和他谈过，他相信大规模数据和深度学习可以解决自动驾驶问题。你对深度学习在这个领域的无限可能性有什么看法？

Elon Musk is confident, I talked to him recently, he's confident that large-scale data and deep learning can solve the autonomous driving problem. What are your thoughts on the limitless possibilities of deep learning in this space?

这显然是解决方案的一部分。我的意思是，我不认为我们会有不依赖深度学习的自动驾驶系统，至少在可预见的未来不会。这么说吧。那么，它占多大比例？在工程史上，特别是类 AI 系统，通常第一阶段是一切都是手工构建的，然后是第二阶段。20-30 年前的自动驾驶就是这种情况。有一个阶段使用了少量学习，但涉及大量工程来处理边缘情况和设置限制等，因为学习系统并不完美。然后随着技术的进步，我们最终越来越依赖学习。这是字符识别、语音识别、计算机视觉、自然语言处理的历史。我认为自动驾驶也会发生同样的情况。

It's obviously part of the solution. I mean, I don't think we'll ever have a self-driving system, at least not in the foreseeable future, that does not use deep learning. Put it that way. Now, how much of it? In the history of engineering, particularly AI-like systems, there is generally a first phase where everything is built by hand, and then a second phase. That was the case for autonomous driving 20-30 years ago. There's a phase where a little bit of learning is used, but there's a lot of engineering involved in taking care of corner cases and putting limits, etc., because the learning system is not perfect. And then as technology progresses, we end up relying more and more on learning. That's the history of character recognition, speech recognition, computer vision, natural language processing. And I think the same is going to happen with autonomous driving.

目前最接近提供一定自主性——那种你不需要司机做任何事的自主性——的方法是约束世界。你只在凤凰城 100 平方公里内运行，天气好，道路宽。这就是 Waymo 的做法。你完全过度设计汽车，配备大量激光雷达和精密传感器，这些对消费级汽车来说太贵了，但如果你运营车队就没问题。你把其他所有东西都工程化，绘制整个世界的完整 3D 地图。感知系统只需要处理移动物体、施工、地图上没有的东西。你可以设计一个好的 SLAM 系统。这是目前最接近一定自主性的方法。但我认为最终长期解决方案将更多地依赖学习，可能结合监督学习和基于模型的强化学习。最终，学习将不仅是核心，而是系统的基础部分。

Currently the methods that are closest to providing some level of autonomy, a decent level of autonomy where you don't expect a driver to do anything, is where you constrain the world. You only run within 100 square kilometers in Phoenix, the weather is nice, and the roads are wide. That's what Waymo is doing. You completely over-engineer the car with tons of lidars and sophisticated sensors that are too expensive for consumer cars, but fine if you run a fleet. You engineer everything else, you map the entire world so you have a complete 3D model. The only thing the perception system has to handle is moving objects, construction, things not in your map. You can engineer a good SLAM system. That's the current approach closest to some level of autonomy. But I think eventually the long-term solution will rely more on learning, possibly using supervised learning and model-based reinforcement learning. Ultimately, learning will be not just at the core but the fundamental part of the system.

它已经是了，但会变得越来越重要。

It already is, but it'll become more and more.

你认为构建一个具有人类智能水平的系统需要什么？你提到过 AI 系统和《她》中的 AI 遥不可及。这可能过时了，但仍然遥不可及。要构建《她》那样的 AI 需要什么？

What do you think it takes to build a system with human-level intelligence? You talked about the AI system and 'Her' being way out of reach. This might be outdated, but it's still way out of reach. What would it take to build 'Her'?

我可以告诉你我们必须清除的前两个障碍，但我不知道之后还有多少障碍。我用的比喻是，我们有一堆山要爬。我们可以看到第一座，但不知道后面还有没有 50 座。这是一个很好的比喻，说明为什么过去 AI 研究者过于乐观。例如，纽厄尔和西蒙写了通用问题求解器，并称之为通用，但你意识到所有你想解决的问题都很琐碎，所以它实际上没什么用。你只看到了第一座山峰。

I can tell you the first two obstacles we have to clear, but I don't know how many obstacles there are after that. The image I use is that there are a bunch of mountains we have to climb. We can see the first one, but we don't know if there are 50 mountains behind it. This is a good metaphor for why AI researchers in the past have been overly optimistic. For example, Newell and Simon wrote the General Problem Solver and called it that, but you realize all the problems you want to solve are trivial, so you can't use it for anything useful. You only see the first peak.

那么对于《她》来说，前几座山峰是什么？

So what are the first couple of peaks for 'Her'?

第一座山峰，正是我正在研究的，是自监督学习。我们如何让机器通过观察来学习世界模型，就像婴儿和幼小动物一样？我一直与认知科学家合作，比如巴黎 FAIR 的 Amanda Depuis，她也是法国大学的研究员。她有一张图表显示人类婴儿在几个月大时学习不同概念。例如，2-3 个月时区分有生命和无生命物体，4 个月左右知道物体是否会稳定或掉落，8-9 个月左右理解重力——物体不应该浮在空中而应该下落。如果你观察 8 个月大的婴儿，给他们高脚椅上的玩具，他们做的第一件事就是把玩具扔到地上。他们在主动学习重力。那么我们如何让机器像婴儿一样学习，主要通过观察和少量互动，学习这些世界模型？我认为这是智能自主系统的关键部分。

The first peak, which is precisely what I'm working on, is self-supervised learning. How do we get machines to learn models of the world by observation, kind of like babies and young animals? I've been working with cognitive scientists, like Amanda Depuis at FAIR in Paris, who is also a researcher at a French university. She has a chart showing how many months of life baby humans learn different concepts. For example, distinguishing animate from inanimate objects at 2-3 months, whether an object will stay stable or fall at about 4 months, and things like gravity—the fact that objects are not supposed to float in the air but fall—around 8-9 months. If you look at 8-month-old babies, you give them toys on the high chair, the first thing they do is drop them on the ground. They're actively learning about gravity. So how do we get machines to learn like babies, mostly by observation with a little interaction, and learn those models of the world? I think that's a crucial piece of an intelligent autonomous system.

所以如果你考虑智能自主系统的架构，它需要一个预测性的世界模型。比如：在时间 T 这里有一堵墙，如果我采取这个动作，在时间 T+1 这里是一个稳定的世界。而且这不是单一答案，而是一个分布。

So if you think about the architecture of an intelligent autonomous system, it needs to have a predictive model of the world. Something that says: here is a wall at time T, here is a stable world at time T+1 if I take this action. And it's not a single answer, it's a distribution.

是的，但我们不知道如何在高维连续空间中表示分布。它必须是我们可以处理的东西，但带有某种确定性的总结表示。如果你有了这个，你就可以做最优控制理论所说的模型预测控制：用假设的动作序列运行你的模型，看看结果。你需要的另一件事是你想要优化的某个目标：我是否达到了抓取物体的目标？我是否在最小化能量？有一个要最小化的目标。在你的头脑中，如果你有这个模型，你可以找出优化目标的最佳动作序列。这个目标最终植根于你的基底神经节，至少在人类大脑中是这样。基底神经节计算你的满足或不满足程度。你的整个行为都朝着最小化那个目标驱动，即最大化你的满足感。你有一个目标函数，它基本上是对基底神经节会告诉你什么的预测。你不会把手放在火上，因为你知道它会烧伤你，这是通过你的世界模型和对这个目标的预测得出的。所以你有四个组件：硬连线的满足目标计算机（如果你想要一个计算器），然后三个组件：目标预测器（预测你的满足水平）、世界模型，以及第三个模块，它根据你的模型找出优化目标的最佳行动方案。

Yeah, but we don't know how to represent distributions in high-dimensional continuous spaces. It's got to be something we can handle, but with some summary representation with certainty. If you have that, you can do what optimal control theory calls model predictive control: run your model with a hypothesis for a sequence of actions and see the result. The other thing you need is some objective you want to optimize: am I reaching the goal of grabbing the object? Am I minimizing energy? There is some objective to minimize. In your head, if you had this model, you can figure out the sequence of actions that will optimize your objective. That objective is ultimately rooted in your basal ganglia, at least in the human brain. The basal ganglia computes your level of contentment or discontentment. Your entire behavior is driven towards minimizing that objective, which is maximizing your contentment. What you have is an objective function which is basically a predictor of what your basal ganglia is going to tell you. You don't put your hand on fire because you know it will burn and hurt, predicting this from your model of the world and your predictor of this objective. So you have four components: the hard-wired contentment objective computer (if you want a calculator), and then three components: the objective predictor (predicts your level of contentment), the model of the world, and a third module that figures out the best course of action to optimize an objective given your model.

这就像一个策略网络之类的东西。

It's a policy network or something like that.

对。现在你需要这三个组件来自主且智能地行动。你可以在三个方面犯傻：你的世界模型错了；你的目标与你实际想要实现的不一致（在人类中，那就是精神病患者）；第三种是你有正确的模型和正确的目标，但无法根据你的模型找出优化目标的行动方案。有些掌管大国的人实际上三者都占。

Right. Now you need those three components to act autonomously and intelligently. You can be stupid in three different ways: you can be stupid because your model of the world is wrong; you can be stupid because your objective is not aligned with what you actually want to achieve (in humans, that would be a psychopath); and the third way is you have the right model and the right objective, but you're unable to figure out a course of action to optimize your objective given your model. Some people who are in charge of big countries actually have all three.

那么如果我们考虑这个智能体，想想电影《她》，你批评过索菲亚机器人这个艺术项目。那个项目本质上利用了我们将类人事物拟人化的自然倾向。你认为这会被像电影《她》中那样的 AI 系统利用吗？所以你认为身体对于创造智能感是必要的吗？

So if we think about this agent, if you think about the movie Her, you've criticized the art project that is Sophia the robot. What that project essentially does is use our natural inclination to anthropomorphize things that look human. Do you think that could be used by AI systems like in the movie Her? So do you think that a body is needed to create a feeling of intelligence?

嗯，如果索菲亚只是一个艺术品，我对此没有意见，但它被呈现为别的东西。让我补充一点。如果索菲亚的创造者能改变他们的营销或行为，那会是什么？几乎一切。我的意思是，你不觉得吗？这里有个难题。我同意你的看法：索菲亚并不是公众认为的那样。公众觉得索菲亚能做的比她实际能做的多得多。没错。而创造索菲亚的人并没有诚实地沟通或试图教育公众。但这里有个难题：你不觉得工业界和研究界的科学家也在做同样的事情吗？他们利用公众的同样误解来创建 AI 公司或发表东西。有些公司是的。我的意思是，没有欺骗的意图，没有夸大成果的意图。你写一篇关于 AI 的论文，在 ImageNet 上有一个结果，这很清楚。它甚至不再有趣了。但我不认为存在那种情况。审稿人通常不会容忍这种无根据的主张。但确实有不少初创公司对此进行了大量炒作，我觉得这极具破坏性，我一直在指出这一点。所以是的。

Well, if Sophia was just an art piece, I would have no problem with it, but it's presented as something else. Let me add that. If the creators of Sophia could change something about their marketing or behavior, what would it be? Just about everything. I mean, don't you think? Here's a tough question. I agree with you: Sophia is not what the general public thinks she can do. The public feels Sophia can do way more than she actually can. That's right. And the people who created Sophia are not honestly communicating or trying to teach the public. But here's a tough question: don't you think the same thing is happening with scientists in industry and research? They take advantage of the same misunderstanding in the public when they create AI companies or publish stuff. Some companies, yes. I mean, there is no desire to delude, no desire to overclaim what something has done. You write a paper on AI that has a result on ImageNet, it's pretty clear. It's not even interesting anymore. But I don't think there is that. The reviewers are generally not very forgiving of unsupported claims of this type. But there are certainly quite a few startups that have had a huge amount of hype around this, which I find extremely damaging, and I've been calling it out when I've seen it. So yeah.

但回到你最初的问题：具身的必要性。我认为具身不是必要的。我认为接地（grounding）是必要的。所以我不认为我们能在没有某种程度的世界接地的情况下得到真正理解语言的机器。而且我不清楚语言是一种高带宽的媒介来传达现实世界如何运作。接地是什么意思？

But to go back to your original question: the necessity of embodiment. I think embodiment is not necessary. I think grounding is necessary. So I don't think we're going to get machines that really understand language without some level of grounding in the world. And it's not clear to me that language is a high-bandwidth medium to communicate how the real world works. What does grounding mean?

所以有一个经典的常识推理问题，Winograd 模式。我告诉你：‘奖杯放不进手提箱，因为它太大了。’或者‘奖杯放不进手提箱，因为它太小了。’第一个句子中的‘它’指的是奖杯，第二个句子中的‘它’指的是手提箱。你能弄清楚的原因是你知道奖杯和手提箱是什么，你知道一个应该放进另一个里面，而且你知道大小的概念：大的物体放不进小的物体。这不是一个 TARDIS。所以你有关于世界如何运作的知识，关于几何等等。我不相信你仅仅通过语言被告知世界如何运作就能了解世界的一切。我认为你需要一些对世界的低级感知，无论是视觉、触觉还是其他，一些更高带宽的对世界的感知。但通过阅读世界上所有的文本，你可能仍然没有足够的信息。没错。有很多东西永远不会出现在文本中，你无法真正推断出来。所以我认为常识会从大量的语言交互中涌现，但也从观看视频，甚至可能在虚拟环境中交互，以及可能机器人在现实世界中交互。但我并不真的认为最后一个是绝对必要的。我认为需要一些接地，但最终产品不一定需要具身。它只需要有一种意识，一种接地。但它需要知道世界如何运作，这样和它说话才不会令人沮丧。

So there is this classic problem of common sense reasoning, the Winograd schema. I tell you: 'The trophy doesn't fit in the suitcase because it is too big.' Or 'The trophy doesn't fit in the suitcase because it is too small.' The 'it' in the first case refers to the trophy, in the second case to the suitcase. The reason you can figure this out is because you know what a trophy and a suitcase are, you know one is supposed to fit in the other, and you know the notion of size: a big object doesn't fit in a small object. This is not a TARDIS. So you have this knowledge of how the world works, of geometry and things like that. I don't believe you can learn everything about the world by just being told in language how the world works. I think you need some low-level perception of the world, be it visual, touch, or whatever, some higher-bandwidth perceptions of the world. But by reading all the world's text, you still may not have enough information. That's right. There are a lot of things that just will never appear in text and that you can't really infer. So I think common sense will emerge from a lot of language interaction, but also from watching videos or perhaps even interacting in virtual environments, and possibly robot interacting in the real world. But I don't actually believe that this last one is absolutely necessary. I think there is a need for some grounding, but the final product doesn't necessarily need to be embodied. It just needs to have an awareness, a grounding. But it needs to know how the world works to not be frustrating to talk to.

你谈到情绪很重要。那完全是另一个话题。

You talked about emotions being important. That's a whole other topic.

嗯，我谈到过基底神经节作为计算你不满或满足程度的东西，然后还有另一个模块试图预测你是否会满足。这是一些情绪的源头。例如，对可能发生在你身上的坏事的预期：你隐约觉得有可能发生非常糟糕的事情，这会产生恐惧。当你确切知道坏事会发生时，你就放弃了，它就不再糟糕了。是不确定性产生了恐惧。所以要点是：是的，没有情绪，我们就不会有大量的智能，无论情绪到底是什么。你提到了非常实际的东西，比如恐惧，但还有很多其他的混乱。但它们是驱力的结果。有更深层的生物学机制在运作。我和一些人谈过这个；有迷人的东西最终与我们的快乐、我们的大脑相连。

Well, I talked about the basal ganglia as this thing that calculates your level of discontentment or contentment, and then there is another module that tries to predict whether you're going to be content or not. That's the source of some emotion. For example, anticipation of bad things that can happen to you: you have this inkling that there is some chance that something really bad is going to happen to you, and that creates fear. When you know for sure that something bad is going to happen to you, you give up, it's not bad anymore. It's uncertainty that creates fear. So the punchline is: yes, we're not going to have a ton of intelligence without emotions, whatever the heck emotions are. You mentioned very practical things like fear, but there's a lot of other mess around. But they are the results of drives. There's deeper biological stuff going on. I've talked to a few folks about this; there's fascinating stuff that ultimately connects to our joy, to our brain.

如果我们创建一个 AGI 系统，或者说人类级别的智能系统，你可以问它一个问题，那会是什么问题？

If we create an AGI system, or a human-level intelligence system, and you get to ask it one question, what would that question be?

我认为我们创造的第一个可能不会那么聪明，就像一个四岁小孩。所以你得问它一个问题。如果它回答，‘哦，是因为树叶在动产生了风’，那它有点门道。如果它说，‘是啊，这是个愚蠢的问题’，那它真的很迟钝。然后你告诉它真实情况，它说，‘哦，对，有道理。’所以是那些能揭示对物理世界进行常识推理能力的问题。而且，有人会称之为因果证据。

I think the first one we create will probably not be that smart, like a four-year-old. So you would have to ask it a question. If it answers, 'Oh, it's because the leaves of the tree are moving that creates wind,' it's on to something. If it says, 'Yeah, that's a stupid question,' it's really obtuse. And then you tell it the real thing, and it says, 'Oh yeah, that makes sense.' So questions that reveal the ability to do common-sense reasoning about the physical world. And you know, someone will call it causal evidence.

非常荣幸。恭喜你获奖。非常感谢你今天接受采访。

It was a huge honor. Congratulations on receiving the award. Thank you so much for talking today.

谢谢。

Thank you.