Yann LeCun：开源 AI 与专有系统、大语言模型的局限以及通往 AGI 之路

我认为将权力集中在专有 AI 系统中的危险比其他一切都大得多。与之对抗的是那些认为出于安全原因我们应该把 AI 系统锁起来的人，因为把它交到每个人手中太危险了。那将导致一个非常糟糕的未来，我们所有的信息摄入都被少数公司通过专有系统控制。我相信人性本善，所以如果 AI，尤其是开源 AI，能让人更聪明，那只会增强人类内心的善良。我认同这种感觉。好吧，我认为人性本善。事实上，很多末日论者之所以是末日论者，是因为他们不认为人性本善。

I see the danger of this concentration of power into proprietary AI systems as a much bigger danger than everything else. What works against this is people who think that for reasons of security we should keep AI systems under lock and key because it's too dangerous to put it in the hands of everybody. That would lead to a very bad future in which all of our information diet is controlled by a small number of companies through proprietary systems. I believe that people are fundamentally good, and so if AI, especially open source AI, can make them smarter, it just empowers the goodness in humans. I share that feeling. Okay, I think people are fundamentally good. In fact, a lot of doomers are doomers because they don't think that people are fundamentally good.

以下是 Yann LeCun 的对话，这是他第三次上这个播客。他是 Meta 的首席 AI 科学家、纽约大学教授、图灵奖得主，也是人工智能历史上最具开创性的人物之一。他和 Meta AI 一直大力倡导开源 AI 开发，并通过开源他们许多最大的模型（包括 Llama 2 和最终的 Llama 3）来践行这一理念。此外，Yann 一直直言不讳地批评 AI 社区中那些警告 AGI 迫在眉睫的危险和存在威胁的人。他相信 AGI 终有一天会被创造出来，但它会是善良的，不会脱离人类控制，也不会统治和杀死所有人类。在 AI 快速发展的当下，这恰好是一个有些争议的立场，所以看到 Yann 在网上进行许多激烈而精彩的讨论很有趣，就像我们在这场对话中一样。这是 Lex Fridman 播客。为了支持它，请查看描述中的赞助商。现在，亲爱的朋友们，有请 Yann LeCun。

The following is a conversation with Yann LeCun, his third time on this podcast. He is the chief AI scientist at Meta, professor at NYU, Turing Award winner, and one of the seminal figures in the history of artificial intelligence. He and Meta AI have been big proponents of open sourcing AI development and have been walking the walk by open sourcing many of their biggest models, including Llama 2 and eventually Llama 3. Also, Yann has been an outspoken critic of those people in the AI community who warn about the looming danger and existential threat of AGI. He believes that AGI will be created one day, but it will be good, it will not escape human control, nor will it dominate and kill all humans. At this moment of rapid AI development, this happens to be a somewhat controversial position, and so it's been fun seeing Yann get into a lot of intense and fascinating discussions online, as we do in this very conversation. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Yann LeCun.

你最近对人工智能的未来发表了一些强有力的、技术性的声明，实际上贯穿你的职业生涯，但最近也是如此。你说过自回归 LLM 不是我们朝着超级智能取得进展的方式。这些是像 GPT-4、Llama 2 和即将推出的 Llama 3 这样的大型语言模型。它们是如何工作的，为什么它们不能带我们走完全程？

You've had some strong statements, technical statements, about the future of artificial intelligence recently, throughout your career actually, but recently as well. You've said that autoregressive LLMs are not the way we're going to make progress towards superhuman intelligence. These are the large language models like GPT-4, like Llama 2 and 3 soon, and so on. How do they work and why are they not going to take us all the way?

原因有很多。首先，智能行为有许多特征，例如理解世界、理解物理世界的能力，记忆和检索事物的能力，持久记忆，推理能力，以及规划能力。这些是智能系统或实体（人类、动物）的四个基本特征。LLM 无法做到这些，或者只能以非常原始的方式做到。它们并不真正理解物理世界，没有真正的持久记忆，不能真正推理，当然也不能规划。所以如果你期望系统在没有这些能力的情况下变得智能，那你就错了。这并不是说自回归 LLM 没有用——它们当然有用——也不是说它们不有趣，或者我们不能围绕它们构建整个应用生态系统。当然可以。但作为通往人类水平智能的路径，它们缺少了基本组件。还有另一个我觉得很有趣的事实：这些 LLM 是在海量文本上训练的，基本上是互联网上所有公开文本的总和。这通常大约是 10^13 个词元，每个词元通常是两个字节，所以训练数据是 2 * 10^13 字节。以每天八小时阅读，你需要 17 万年才能读完。所以看起来这些系统可以积累海量知识。但你会发现数据其实并没有那么多。如果你和一位发展心理学家交谈，他们会告诉你一个四岁孩子一生中清醒了 16,000 小时，四年内到达他视觉皮层的信息量大约是 10^15 字节。你可以通过估算视神经每秒传输约 20 兆字节来计算。所以四岁孩子是 10^15 字节，而 17 万年的阅读量是 2 * 10^13 字节。这告诉你，通过感官输入，我们看到的信息比通过语言多得多。尽管我们有直觉，但我们学到的大部分知识和认知是通过观察和与真实世界的互动，而不是通过语言。我们在生命最初几年学到的一切，以及动物学到的一切，都与语言无关。

For a number of reasons. The first is that there is a number of characteristics of intelligent behavior, for example the capacity to understand the world, understand the physical world, the ability to remember and retrieve things, persistent memory, the ability to reason, and the ability to plan. Those are four essential characteristics of intelligent systems or entities, humans, animals. LLMs can do none of those, or they can only do them in a very primitive way. They don't really understand the physical world, don't really have persistent memory, they can't really reason, and they certainly can't plan. So if you expect the system to become intelligent without having the possibility of doing those things, you're making a mistake. That is not to say that autoregressive LLMs are not useful—they're certainly useful—that they're not interesting, that we can't build a whole ecosystem of applications around them. Of course we can. But as a path towards human-level intelligence, they're missing essential components. And then there is another tidbit or fact that I think is very interesting: those LLMs are trained on enormous amounts of text, basically the entirety of all publicly available text on the internet. That's typically on the order of 10^13 tokens, each token is typically two bytes, so that's 2 * 10^13 bytes as training data. It would take you or me 170,000 years to just read through this at eight hours a day. So it seems like an enormous amount of knowledge that those systems can accumulate. But then you realize it's really not that much data. If you talk to a developmental psychologist, they tell you a four-year-old has been awake for 16,000 hours in his life, and the amount of information that has reached the visual cortex of that child in four years is about 10^15 bytes. You can compute this by estimating that the optical nerve carries about 20 megabytes per second roughly. So 10^15 bytes for a four-year-old versus 2 * 10^13 bytes for 170,000 years worth of reading. What it tells you is that through sensory input we see a lot more information than we do through language. Despite our intuition, most of what we learn and most of our knowledge is through our observation and interaction with the real world, not through language. Everything that we learn in the first few years of life, and certainly everything that animals learn, has nothing to do with language.

所以也许可以反驳一下你说法背后的一些直觉。确实，进入人类大脑的数据量要高出几个数量级，而且速度更快，人类大脑能够从中快速学习，快速过滤数据。有人可能会说，你比较感官数据和语言：语言已经非常压缩了，与视觉数据相比，它包含的信息远多于存储它所需的字节。所以语言中有很多智慧，有词语以及我们组合它们的方式，它已经包含了很多信息。那么，是否可能仅凭语言就已经有足够的智慧和知识，能够从语言中构建一个世界模型和对世界的理解，对物理世界的理解，而你认为 LLM 缺乏这些？

So it would be good to maybe push against some of the intuition behind what you're saying. So it is true there's several orders of magnitude more data coming into the human mind much faster, and the human mind is able to learn very quickly from that, filter the data very quickly. You know, somebody might argue your comparison between sensory data versus language: language is already very compressed, it already contains a lot more information than the bytes it takes to store them if you compare it to visual data. So there's a lot of wisdom in language, there's words and the way we stitch them together, it already contains a lot of information. So is it possible that language alone already has enough wisdom and knowledge in there to be able to, from that language, construct a world model and understanding of the world, an understanding of the physical world that you're saying LLMs lack?

所以这是一个很大的争论，在哲学家和认知科学家中，关于智能是否需要扎根于现实。我显然属于那个阵营：是的，智能不可能在没有某种现实基础的情况下出现。它不一定是物理现实，可以是模拟的，但环境比你能用语言表达的丰富得多。语言是我们感知和心理模型的一个非常近似的表示。我们完成的许多任务都涉及操纵当前情况的心理模型，这与语言无关。所有物理的、机械的等等，当我们建造东西，当我们完成像抓取东西这样的任务时，我们规划动作序列，我们通过想象一系列动作的结果来做到这一点。所以我们可能会想象，这需要与语言关系不大的心理模型。而且，我认为，我们的大部分知识都来自与物理世界的这种互动。所以我的许多同事，那些对计算机视觉之类更感兴趣的人，确实认为 AI 本质上需要具身。而来自 NLP 领域或其他领域的人……

So it's a big debate among philosophers and also cognitive scientists whether intelligence needs to be grounded in reality. I'm clearly in the camp that yes, intelligence cannot appear without some grounding in some reality. It doesn't need to be physical reality, could be simulated, but the environment is just much richer than what you can express in language. Language is a very approximate representation of our percepts and our mental models. There's a lot of tasks that we accomplish where we manipulate a mental model of the situation at hand, and that has nothing to do with language. Everything that's physical, mechanical, whatever, when we build something, when we accomplish a task like grabbing something, we plan action sequences and we do this by essentially imagining the result of the outcome of a sequence of actions. So we might imagine, and that requires mental models that don't have much to do with language. And that's, I would argue, most of our knowledge is derived from that interaction with the physical world. So a lot of my colleagues who are more interested in things like computer vision are really on that camp that AI needs to be embodied essentially. And then other people coming from the NLP side or maybe some other...

动机不一定同意这一点，哲学家们也有分歧。世界的复杂性难以想象。我们完全视为理所当然的现实世界中的那些复杂性，我们甚至不认为它们需要智能，但很难表征。这是机器人学和 SMC 先驱提出的老莫拉维克悖论。他说，为什么计算机似乎很容易完成像下棋和求解积分这样的高级复杂任务，而我们每天做的那些习以为常的事情，比如学开车或抓取物体，计算机却做不到？我们有能通过律师资格考试的 LLM，所以它们一定很聪明，但它们不能在 20 小时内学会开车，像任何 17 岁少年那样；它们不能像任何 10 岁孩子那样一次就学会清理餐桌和装满洗碗机。这是为什么？我们缺少什么？我们缺少什么样的学习或推理架构，以至于无法拥有五级自动驾驶汽车和家用机器人？

Motivation don't necessarily agree with that, and philosophers are split as well. The complexity of the world is hard to imagine. It's hard to represent all the complexities that we take completely for granted in the real world that we don't even imagine require intelligence. This is the old Moravec Paradox from the pioneer of robotics and SMC. He said, how is it that with computers it seems to be easy to do high-level complex tasks like playing chess and solving integrals, whereas the things we take for granted that we do every day, like learning to drive a car or grabbing an object, we can't do with computers? And we have LLMs that can pass the bar exam, so they must be smart, but then they can't learn to drive in 20 hours like any 17-year-old, they can't learn to clear out the dinner table and fill the dishwasher like any 10-year-old can learn in one shot. Why is that? What are we missing? What type of learning or reasoning architecture are we missing that prevents us from having level five self-driving cars and domestic robots?

一个大型语言模型能否构建一个世界模型，知道如何驾驶和如何装满洗碗机，只是目前不知道如何处理视觉数据？所以它可以在概念空间中运作。很多人都在研究这个。简短的回答是不。更复杂的答案是，你可以用各种技巧让 LLM 消化图像、视频或音频的视觉表征。经典的方法是，以某种方式训练一个视觉系统，有监督、半监督、自监督等各种方式，将任何图像转化为高层表征，基本上是一个类似于典型 LLM 输入标记的列表。然后你把它连同文本一起输入 LLM，期望 LLM 在训练过程中能够利用这些表征来帮助决策。这方面的研究已经进行了相当长的时间。现在你看到这些系统，有视觉扩展的 LLM，但它们基本上是权宜之计，因为这些系统不是端到端训练的，无法真正理解世界。例如，它们没有用视频训练。它们并不真正理解直观物理学，至少目前还没有。

Can a large language model construct a world model that does know how to drive and does know how to fill a dishwasher, but just doesn't know how to deal with visual data at this time? So it can operate in the space of concepts. That's what a lot of people are working on. The short answer is no. The more complex answer is you can use all kinds of tricks to get an LLM to basically digest visual representations of images or video or audio. A classical way is you train a vision system in some way, supervised, semi-supervised, self-supervised, all kinds of ways, that will turn any image into a high-level representation, basically a list of tokens similar to what a typical LLM takes as input. Then you feed that to the LLM in addition to the text, and you expect the LLM during training to be able to use those representations to help make decisions. There has been work along those lines for quite a long time. Now you see those systems, LLMs that have some vision extension, but they're basically hacks in the sense that those things are not trained end-to-end to really understand the world. They're not trained with video, for example. They don't really understand intuitive physics, at least not at the moment.

所以你不认为直观物理学、关于物理空间的常识推理、关于物理现实有什么特别之处？在你看来，这是 LLM 无法实现的一大飞跃？

So you don't think there's something special about intuitive physics, about common sense reasoning about the physical space, about physical reality? That to you is a giant leap that LLMs are just not able to do?

我们无法用今天使用的这类 LLM 做到这一点。原因有很多，但主要原因是 LLM 的训练方式。你拿一段文本，去掉一些词，用空白标记替换，然后训练一个巨大的神经网络来预测缺失的词。如果你以一种特定的方式构建这个神经网络，使它只能看它试图预测的词左边的词，那么你就得到了一个基本上试图预测文本中下一个词的系统。你给它一个提示，让它预测下一个词。它永远无法精确预测下一个词，所以它会在字典中所有可能的词上产生一个概率分布。实际上，它预测的是标记，也就是子词单元。处理不确定性很容易，因为可能的词数量有限，你可以计算它们的分布。然后系统从该分布中选取一个词，概率高的词被选中的机会更大。你从该分布中采样来生成一个词，然后把这个词移入输入，依此类推。这被称为自回归预测，这就是为什么这些 LLM 应该被称为自回归 LLM，但我们只叫它们 LLM。这种过程与你我说话时在生成一个词之前的过程不同：我们会思考要说什么，而且这相对独立于语言。当我们谈论一个数学概念时，我们进行的思考和计划产生的答案与我们要用法语、俄语还是英语说无关。

We're not going to be able to do this with the type of LLMs that we are working with today. There are a number of reasons for this, but the main reason is the way LLMs are trained. You take a piece of text, remove some words, mask them, replace them with blank markers, and train a gigantic neural net to predict the missing words. If you build this neural net in a particular way so that it can only look at words to the left of the one it's trying to predict, then you have a system that is basically trying to predict the next word in a text. You feed it a prompt and ask it to predict the next word. It can never predict the next word exactly, so it produces a probability distribution over all possible words in the dictionary. In fact, it predicts tokens, which are subword units. It's easy to handle uncertainty because there's only a finite number of possible words, and you can compute a distribution over them. Then the system picks a word from that distribution, with higher chance for higher probability words. You sample from that distribution to produce a word, then shift that word into the input, and so on. That's called autoregressive prediction, which is why those LLMs should be called autoregressive LLMs, but we just call them LLMs. There is a difference between this kind of process and a process by which before producing a word, when you and I talk, we think about what we're going to say, and it's relatively independent of the language. When we talk about a mathematical concept, the thinking and the answer we plan to produce is not linked to whether we're going to say it in French, Russian, or English.

乔姆斯基刚才翻了个白眼，但我理解。所以你是说在语言之前有一个更大的抽象，然后映射到语言上？

Chomsky just rolled his eyes, but I understand. So you're saying that there's a bigger abstraction that goes before language and maps onto language?

对于我们做的很多思考来说，这确实是事实。

It's certainly true for a lot of thinking that we do.

这很明显吗？就像你说的，你的思考在法语和英语中是一样的？

Is that obvious? Like you're saying your thinking is the same in French as it is in English?

是的，差不多。或者这就像你有多灵活？如果有一个概率分布……嗯，这取决于什么样的思考。如果是创作双关语，我用法语比英语好得多。但是双关语有抽象表征吗？比如你的幽默，当你发推文，有时有点辛辣，在你大脑中，推文在映射到英语之前是否有抽象表征？有一个抽象表征，想象读者对那段文字的反应。你从笑声开始，然后想办法实现它，或者想出一个你想要引起的反应，然后想办法说出来以引起那个反应。但这非常接近语言。想想一个数学概念，或者想象你想用木头建造的东西。你正在进行的思考与语言完全无关。你并不是在用任何特定语言进行内心独白。你在想象事物的心智模型。如果我让你想象这个水瓶旋转 90 度后会是什么样子，那与语言无关。所以很明显有一个更抽象的层次。

Yeah, pretty much. Or is this like how flexible are you? If there's a probability distribution... Well, it depends what kind of thinking. If it's producing puns, I get much better in French than English. But is there an abstract representation of puns? Like your humor, when you tweet and your tweets are sometimes a little bit spicy, is there an abstract representation in your brain of a tweet before it maps onto English? There is an abstract representation of imagining the reaction of a reader to that text. You start with laughter and then figure out how to make that happen, or figure out a reaction you want to cause and then figure out how to say it so that it causes that reaction. But that's really close to language. Think about a mathematical concept or imagining something you want to build out of wood. The kind of thinking you're doing has absolutely nothing to do with language. It's not like you have an internal monologue in any particular language. You're imagining mental models of the thing. If I ask you to imagine what this water bottle will look like if I rotate it 90 degrees, that has nothing to do with language. So clearly there is a more abstract level.

我们大部分思考都在一种表征中进行，并计划要说什么。如果输出是说出的话语而非肌肉动作，我们会在产生回答之前先计划好。LLM 不这样做；它们只是本能地一个词接一个词地产生。这有点像潜意识行为——你正分心，有人问你问题，你自动回答，不经思考。这就是 LLM 所做的。它不思考答案；它只是检索，因为它积累了大量知识。它只是不加计划地一个接一个吐出 token。

Representation in which we do most of our thinking and we plan what we're going to say. If the output is uttered words as opposed to muscle actions, we plan our answer before we produce it. LLMs don't do that; they just produce one word after the other instinctively. It's like subconscious actions where you're distracted, someone asks a question, and you answer automatically without thinking. That's what an LLM does. It doesn't think about the answer; it retrieves it because it has accumulated a lot of knowledge. It just spits out one token after the other without planning.

但你把它说得好像逐 token 生成注定是简单的。然而，如果世界模型足够复杂，它生成的最可能的 token 序列会是深刻的东西。

But you're making it sound like one token at a time generation is bound to be simplistic. However, if the world model is sufficiently sophisticated, the most likely thing it generates is a sequence of tokens that is deeply profound.

但这假设这些系统确实拥有内部世界模型。所以根本问题是：你能通过预测构建一个完整的世界模型吗？答案可能是肯定的。但你能通过预测词语来构建吗？答案很可能是否定的，因为语言带宽很低。构建世界模型意味着观察世界并理解它为何演化。世界模型的额外组成部分是能预测行动后果的东西。所以世界模型是：这是我对 t 时刻世界状态的看法，这是我可能采取的行动，预测的 t+1 时刻状态是什么？该状态不需要代表一切，只需足够用于规划。

But that assumes those systems actually possess an internal world model. So it goes to the fundamental question: can you build a complete world model by prediction? The answer is probably yes. But can you build it by predicting words? The answer is most probably no, because language is very low bandwidth. Building world models means observing the world and understanding why it evolves. The extra component of a world model is something that can predict how the world will evolve as a consequence of an action. So a world model is: here is my idea of the state of the world at time t, here is an action I might take, what is the predicted state at t+1? That state doesn't need to represent everything, just enough relevant for planning.

现在问题来了：你无法用基于视频训练的生成模型做到这一点。我们尝试了 10 年。你给系统一段视频，让它逐帧预测剩余部分，就像自回归 LLM 处理文本那样。但你无法像 LLM 那样做，因为对于 LLM，你无法精确预测下一个词，但可以预测词上的分布。对于视频，你必须预测所有可能帧上的分布，而我们不知道如何正确做到这一点。我们不知道如何以有用的方式表示高维连续空间上的分布。这是主要问题。世界比文本复杂和丰富得多。文本是离散的；视频是高维连续的，包含许多细节。如果我拍一段这个房间的摇镜视频，我无法预测房间里会出现的一切。系统无法预测墙上的画、沙发或地毯的纹理。所以无法预测所有细节。

Now here is the problem: you're not going to be able to do this with generative models trained on video. We've tried for 10 years. You take a video, show a system a piece, and ask it to predict the remainder, one frame at a time, like autoregressive LLMs do for text. But you can't do the same trick as with LLMs because with LLMs you can't predict exactly which word follows, but you can predict the distribution over words. For video, you would have to predict the distribution over all possible frames, and we don't know how to do that properly. We don't know how to represent distributions over high-dimensional continuous spaces in useful ways. That's the main issue. The world is incredibly more complicated and richer than text. Text is discrete; video is high-dimensional and continuous with many details. If I take a video of this room panning around, there is no way I can predict everything that will be in the room. The system cannot predict what the painting on the wall looks like or the texture of the couch or carpet. So there's no way to predict all those details.

所以一种处理方式——我们研究了很久——是使用带有潜变量的模型。潜变量被输入神经网络，应该代表你尚未感知到的所有世界信息，以增强系统进行良好的像素预测，包括精细纹理。这基本上完全失败了。我们尝试了很多方法：直接神经网络、GAN、VAE、各种正则化自编码器。我们还尝试了学习图像或视频的良好表示用于分类的方法，也基本失败了。所有试图从损坏版本预测图像或视频缺失部分、希望系统发展出良好表示用于物体识别的方法都完全失败了。这对文本非常有效，正是 LLM 使用的原理。

So one way to handle this, which we've been working on for a long time, is to have a model with a latent variable. The latent variable is fed to a neural net and is supposed to represent all the information about the world that you don't perceive yet, needed to augment the system for good pixel prediction, including fine texture. That has been a complete failure essentially. We tried many things: straight neural nets, GANs, VAEs, all kinds of regularized autoencoders. We also tried methods to learn good representations of images or video for classification, and that also basically failed. All systems that attempt to predict missing parts of an image or video from a corrupted version, hoping the system will develop good representations for object recognition, have been a complete failure. It works really well for text, which is the principle used for LLMs.

失败具体在哪里？是难以形成良好的图像表示，即所有重要信息的良好嵌入？还是视频中图像间的一致性？你所有失败方式的精彩集锦是什么样的？

Where is the failure exactly? Is it that it's very difficult to form a good representation of an image, a good embedding of all the important information? In terms of consistency from image to image in a video? What does a highlight reel of all the ways you failed look like?

好的，所以这不起作用的原因：首先，我必须确切告诉你什么不起作用，因为还有其他东西起作用。不起作用的是训练系统通过从损坏版本重建好图像来学习图像表示。这不起作用。我们有一整套技术：去噪自编码器的变体，我在 FAIR 的一些同事开发的 MADE，Max Welling 的编码器。它基本上类似于 LLM 之类的东西。

Okay, so the reason this doesn't work: first, I have to tell you exactly what doesn't work, because there is something else that does work. The thing that does not work is training a system to learn representations of images by training it to reconstruct a good image from a corrupted version. That doesn't work. We have a whole slew of techniques for this: variants of denoising autoencoders, something called MADE developed by some of my colleagues at FAIR, Max Welling's encoder. It's basically like the LLMs or things.

比如这种通过破坏文本来训练系统的方法，只不过你破坏的是图像：你从图像中移除一些块，然后训练一个巨大的网络去重建。你得到的特征并不好。你知道它们不好，因为如果你用同样的架构进行监督训练，使用带标签的数据、图像的文本描述等，你确实能得到好的表示。在识别任务上的表现也比这种自监督预训练好得多。所以架构是好的，编码器的架构是好的。但当你以自监督方式训练系统重建图像时，它并不能学到好的通用图像特征，这种自监督是通过重建实现的。

Like this where you train the system by corrupting text, except you corrupt images: you remove patches from it and you train a gigantic net to reconstruct. The features you get are not good. And you know they're not good because if you now train the same architecture but you train it supervised with label data, with textual descriptions of images, etc., you do get good representations. And the performance on recognition tasks is much better than if you do this self-supervised pre-training. So the architecture is good, the architecture of the encoder is good. But the fact that you train the system to reconstruct images does not lead it to learn good generic features of images when you train in a self-supervised way, self-supervised by reconstruction.

那么替代方案是什么？

So what's the alternative?

替代方案是联合嵌入。什么是联合嵌入？这些让你如此兴奋的架构是什么？现在，不是训练一个系统对图像进行编码，然后训练它从损坏版本重建完整图像，而是你取完整图像，取损坏或变换后的版本，将它们都通过编码器（通常相同但不一定），然后在这些编码器之上训练一个预测器，从损坏版本的表示预测完整输入的表示。所以是联合嵌入，因为你取完整输入和损坏或变换版本，都通过编码器，得到联合嵌入。然后你说：我能从损坏版本的表示预测完整版本的表示吗？我称之为 JEPA，即联合嵌入预测架构，因为它是联合嵌入，并且有一个预测器从坏家伙预测好家伙的表示。

The alternative is joint embedding. What is joint embedding? What are these architectures that you're so excited about? So now instead of training a system to encode the image and then training it to reconstruct the full image from a corrupted version, you take the full image, you take the corrupted or transformed version, you run them both through encoders, which in general are identical but not necessarily, and then you train a predictor on top of those encoders to predict the representation of the full input from the representation of the corrupted one. So joint embedding because you're taking the full input and the corrupted version or transformed version, run them both through encoders, so you get a joint embedding. And then you're saying: can I predict the representation of the full one from the representation of the corrupted one? And I call this a JEPA, so that means Joint Embedding Predictive Architecture, because it's joint embedding and there is this predictor that predicts the representation of the good guy from the bad guy.

那么关键问题是如何训练这样的东西？

And the big question is how do you train something like this?

直到五六年前，我们还没有特别好的方法来训练这些东西，除了一个叫做对比学习的方法。在对比学习中，你取一对图像，再次是原始图像及其损坏或降级或变换版本，你训练预测的表示与那个相同。如果只这样做，系统会崩溃；它基本上完全忽略输入并产生恒定的表示。所以对比方法避免了这一点，这些方法从 90 年代初就存在了——我在 1993 年发表了一篇论文。你还展示你知道不同的图像对，然后将它们的表示推开。所以你说：不仅我们知道相同的事物的表示应该相似，而且我们知道不同的事物的表示应该不同，这防止了崩溃。但它有一些局限性。过去六七年出现了一整套技术可以复兴这类方法，有些来自 FAIR，有些来自谷歌和其他地方。但这些对比方法有局限性。过去三四年发生的变化是，现在我们有了非对比方法，所以它们不需要那些我们知道不同的负对比样本。你只能使用同一事物的不同版本或不同视角的图像来训练它们，并依靠一些其他技巧来防止系统崩溃。我们现在有十几种不同的方法。

Until five or six years ago, we didn't have particularly good answers for how you train those things, except for one called contrastive learning. In contrastive learning, you take a pair of images that are again an image and a corrupted version or degraded version somehow or transformed version of the original one, and you train the predicted representation to be the same as that. If you only do this, the system collapses; it basically completely ignores the input and produces representations that are constant. So contrastive methods avoid this, and those things have been around since the early 90s—I had a paper on this in 1993. You also show pairs of images that you know are different, and then you push away the representations from each other. So you say: not only should representations of things that we know are the same be similar, but representations of things that we know are different should be different, and that prevents collapse. But it has some limitations. There's a whole bunch of techniques that have appeared over the last six or seven years that can revive this type of method, some from FAIR, some from Google and other places. But there are limitations to those contrastive methods. What has changed in the last three or four years is that now we have methods that are non-contrastive, so they don't require those negative contrastive samples of images that we know are different. You can train them only with images that are different versions or different views of the same thing, and you rely on some other tricks to prevent the system from collapsing. We have a dozen different methods for this now.

那么联合嵌入架构和 LLMs 之间的根本区别是什么？JEPA 能带我们走向 AGI 吗？

So what is the fundamental difference between joint embedding architectures and LLMs? Can JEPA take us to AGI?

嗯，这是第一步。首先，与像 LLMs 这样的生成式架构有什么区别？LLMs 或通过重建训练的视觉系统会生成输入，对吧？它们生成未损坏、未变换的原始输入。所以你必须预测所有像素，系统要花费大量资源来实际预测所有这些像素、所有细节。在 JEPA 中，你并不试图预测所有像素；你只试图预测输入的抽象表示，这在很多方面要容易得多。所以 JEPA 系统在训练时试图做的是从输入中提取尽可能多的信息，但只提取相对容易预测的信息。世界上有很多我们无法预测的东西。例如，如果有一辆车在街上或路上行驶，路边可能有树，而且可能是一个有风的日子，所以树上的叶子以半混沌的随机方式移动，你无法预测，你也不在乎。你不想预测那个。所以你希望你的编码器基本上消除所有这些细节；它会告诉你树叶在动，但不会保留具体发生了什么细节。所以当你在表示空间中进行预测时，你不必预测叶子的每一个像素。这不仅简单得多，而且允许系统本质上学习世界的抽象表示，其中可建模和可预测的部分被保留，其余部分被视为噪声并被编码器消除。所以它提升了表示的抽象层次。如果你想想，这正是我们一直在做的事情。每当我们描述一个现象时，我们都在特定的抽象层次上描述它，我们不会总是用量子场论来描述每一个自然现象，对吧？那是不可能的。所以我们有多个抽象层次来描述世界上发生的事情，从量子场论到原子理论和分子、化学、材料，一直到现实世界中的具体物体等等。所以我们不能只在最低层次上建模一切，这就是 JEPA 的理念所在：学习抽象表示。

Well, it's a first step. First of all, what's the difference with generative architectures like LLMs? LLMs or vision systems that are trained by reconstruction generate the inputs, right? They generate the original input that is non-corrupted, non-transformed. So you have to predict all the pixels, and there is a huge amount of resources spent in the system to actually predict all those pixels, all the details. In a JEPA, you're not trying to predict all the pixels; you're only trying to predict an abstract representation of the inputs, and that's much easier in many ways. So what the JEPA system, when it's being trained, is trying to do is extract as much information as possible from the input, but yet only extract information that is relatively easily predictable. There's a lot of things in the world that we cannot predict. For example, if you have a car driving down the street or road, there may be trees around the road, and it could be a windy day, so the leaves on the tree are moving in semi-chaotic random ways that you can't predict, and you don't care. You don't want to predict that. So what you want is your encoder to basically eliminate all those details; it will tell you there are moving leaves, but it's not going to keep the details of exactly what's going on. So when you do the prediction in representation space, you're not going to have to predict every single pixel of a leaf. Not only is that a lot simpler, but it also allows the system to essentially learn an abstract representation of the world where what can be modeled and predicted is preserved, and the rest is viewed as noise and eliminated by the encoder. So it kind of lifts the level of abstraction of the representation. If you think about this, this is something we do absolutely all the time. Whenever we describe a phenomenon, we describe it at a particular level of abstraction, and we don't always describe every natural phenomenon in terms of quantum field theory, right? That would be impossible. So we have multiple levels of abstraction to describe what happens in the world, starting from quantum field theory to atomic theory and molecules, chemistry, materials, and all the way up to concrete objects in the real world and things like that. So we can't just model everything at the lowest level, and that's what the idea of JEPA is really about: learning abstract representations.

以自监督的方式，而且你可以分层进行。所以我认为这是智能系统的一个基本组成部分。在语言中，我们可以不这样做，因为语言已经在一定程度上是抽象的，并且已经消除了很多不可预测的信息。所以我们可以在不提升抽象层次的情况下，通过直接预测单词来蒙混过关。所以联合嵌入，它仍然是生成式的，但它是这个抽象表示空间中的生成式。

In a self-supervised manner, and you can do it hierarchically as well. So I think that is an essential component of an intelligent system. In language, we can get away without doing this because language is already to some level abstract and has already eliminated a lot of information that is not predictable. So we can get away without doing the training without lifting the abstraction level, by directly predicting words. So joint embedding, it's still generative but it's generative in this abstract representation space.

你是说我们在语言上偷懒了，因为我们已经免费获得了抽象表示。现在我们必须退一步，真正思考通用智能系统，我们必须处理物理现实的全部混乱。你不能不这样做，你必须完成从丰富详细的现实到基于此进行推理的抽象表示的跳跃，对吧？

And you're saying language we were lazy with language because we already got the abstract representation for free. And now we have to zoom out, actually think about generally intelligent systems, we have to deal with the full mess of physical reality. And you can't, you do have to do this step of jumping from the full rich detailed reality to an abstract representation of that reality based on which you can then reason and all that kind of stuff, right?

问题是，那些通过预测来学习的自监督算法，即使在表示空间中，如果你输入的数据冗余度更高，它们就能学到更多概念。数据冗余度越高，它们就越能捕捉到其中的一些内部结构。所以在感知输入（如视觉）的结构中，冗余度远高于文本，文本的冗余度并不高。这又回到了你几分钟前问的问题：语言可能确实代表了更多信息，因为它已经被压缩了，你说得对，但这意味着它的冗余度也更低，所以自监督的效果不会那么好。

And the thing is, those self-supervised algorithms that learn by prediction, even in representation space, they learn more concepts if the input data you feed them is more redundant. The more redundancy there is in the data, the more they're able to capture some internal structure of it. So there is way more redundancy in structure in perceptual inputs, sensory input like vision, than there is in text, which is not nearly as redundant. This is back to the question you were asking a few minutes ago: language might represent more information really because it's already compressed, you're right about that, but that means it's also less redundant, and so self-supervision will not work as well.

是否有可能将视觉数据上的自监督训练和语言数据上的自监督训练结合起来？尽管你轻视那些 10^13 个词元，但这些词元代表了人类已经弄清楚的大部分内容，包括 Reddit 上的讨论、所有书籍和文章的内容，以及人类智力创造的全部范围。那么，是否有可能将这两者结合起来呢？

Is it possible to join the self-supervised training on visual data and self-supervised training on language data? There is a huge amount of knowledge, even though you talk down about those 10^13 tokens, those 10^13 tokens represent a large fraction of what us humans have figured out, both the talk on Reddit and the contents of all the books and the articles and the full spectrum of human intellectual creation. So is it possible to join those two together?

嗯，最终是可以的，但我认为如果我们过早这样做，就会有作弊的诱惑。事实上，这正是人们目前在视觉语言模型上所做的：我们基本上在作弊。我们使用语言作为拐杖，来帮助我们的视觉系统从图像和视频中学习好的表示。问题在于，我们可能会稍微改进我们的视觉语言系统，比如通过给语言模型喂图像，但我们无法达到甚至猫或狗那样的智能或对世界的理解水平，而它们没有语言。它们没有语言，却比任何大语言模型更理解世界。它们可以规划非常复杂的行动，并想象一系列行动的结果。在我们将其与语言结合之前，如何让机器学会这一点？显然，如果我们将其与语言结合，这将是一个赢家。但在此之前，我们必须专注于如何让系统学习世界是如何运作的。所以这种联合嵌入预测架构，将能够学习类似常识的东西，类似猫用来预测如何最优化地通过打翻东西来惹恼主人的东西。这就是希望所在。

Well, eventually yes, but I think if we do this too early, we run the risk of being tempted to cheat. And in fact, that's what people are doing at the moment with vision-language models: we're basically cheating. We are using language as a crutch to help the deficiencies of our vision systems to kind of learn good representations from images and video. The problem with this is that we might improve our visual-language system a bit, I mean our language models by feeding them images, but we're not going to get to the level of even the intelligence or level of understanding of the world of a cat or dog, which doesn't have language. They don't have language and they understand the world much better than any LLM. They can plan really complex actions and sort of imagine the result of a bunch of actions. How do we get machines to learn that before we combine that with language? Obviously, if we combine this with language, this is going to be a winner. But before that, we have to focus on how to get systems to learn how the world works. So this kind of joint embedding predictive architecture for you, that's going to be able to learn something like common sense, something like what a cat uses to predict how to mess with its owner most optimally by knocking over a thing. That's the hope.

事实上，我们使用的技术是非对比性的。所以不仅架构是非生成式的，我们使用的学习过程也是非对比性的。我们有两套技术。一套基于蒸馏，有许多方法使用这个原理：DeepMind 的 BYOL，FAIR 的几个，一个叫 VICReg，另一个叫 I-JEPA，还有一个叫 DINO 或 DiNO，也是 FAIR 开发的。这些方法的思想是，你取完整的输入，比如一张图像，通过编码器得到表示，然后你破坏或变换输入，通过本质上相同的编码器（略有不同）运行，然后训练一个预测器，有时预测器非常简单，有时不存在，但训练预测器从破坏的输入预测原始未破坏输入的表示。但你只训练第二个分支，只训练输入破坏数据的那部分网络；另一个网络不训练。但由于它们共享权重，当你修改第一个时，第二个也会被修改。通过各种技巧，你可以防止系统崩溃，即我之前解释的那种系统基本上忽略输入的崩溃。所以这效果很好。我们在 FAIR 开发的技术，DINO 和 I-JEPA，对此非常有效。

In fact, the techniques we're using are non-contrastive. So not only is the architecture non-generative, the learning procedures we're using are non-contrastive. We have two sets of techniques. One set is based on distillation, and there are a number of methods that use this principle: one by DeepMind called BYOL, a couple by FAIR, one called VICReg and another called I-JEPA, and another one also called DINO or DiNO produced also at FAIR. The idea of those things is that you take the full input, let's say an image, you run it through an encoder, it produces a representation, and then you corrupt that input or transform it, running it through essentially what amounts to the same encoder with some minor differences, and then train a predictor, sometimes the predictor is very simple, sometimes it doesn't exist, but train a predictor to predict a representation of the first uncorrupted input from the corrupted input. But you only train the second branch, you only train the part of the network that is fed with the corrupted input; the other network you don't train. But since they share the same weights, when you modify the first one, it also modifies the second one. And with various tricks, you can prevent the system from collapsing, with the collapse of the type I was explaining before where the system basically ignores the input. So that works very well. The techniques we developed at FAIR, DINO and I-JEPA, work really well for that.

那么这里我们说的是什么样的数据？

So what kind of data are we talking about here?

所以有几种场景。一种场景是，你取一张图像，通过改变裁剪、稍微改变大小、改变方向、模糊、改变颜色等方式来破坏它，做各种糟糕的事情。但基本的糟糕事情，比如稍微降低质量、改变构图，比如裁剪图像。或者在某些情况下，比如 I-JEPA，你不需要做这些；你只需掩盖部分区域，对吧？你基本上移除一些区域，比如一个大块，然后通过编码器运行，训练整个系统和预测器，从破坏的表示预测好的表示。这就是 I-JEPA。它不需要知道这是一张图像，因为它只需要知道如何进行这种掩码。而 DINO 则需要知道它是图像，因为你需要做几何变换、模糊等图像特定的操作。我们最近的一个版本叫做 V-JEPA。它基本上和 I-JEPA 相同，但应用于视频。所以现在你取整个视频，掩盖其中一大块。我们掩盖的实际上是一种时间管，即整个视频中每一帧的一个完整片段。这个管在整个帧中静态定位，是一个直管。管通常有 16 帧左右，我们在整个 16 帧中掩盖相同的区域。显然每个视频的掩盖区域不同。然后再次训练该系统，从部分掩盖的视频预测完整视频的表示。

So this is the several scenario. One scenario is you take an image, you corrupt it by changing the cropping for example, changing the size a little bit, maybe changing the orientation, blurring it, changing the colors, doing all kinds of horrible things to it. But basic horrible things that sort of degrade the quality a little bit and change the framing, you know, crop the image. Or in some cases, in the case of I-JEPA, you don't need to do any of this; you just mask some parts of it, right? You just basically remove some regions like a big block essentially, and then run through the encoders and train the entire system and predictor to predict the representation of the good one from the representation of the corrupted one. So that's I-JEPA. It doesn't need to know that it's an image, for example, because the only thing it needs to know is how to do this masking. Whereas with DINO, you need to know it's an image because you need to do things like geometric transformations and blurring and things like that that are really image-specific. A more recent version of this that we have is called V-JEPA. So it's basically the same idea as I-JEPA except it's applied to video. So now you take a whole video and you mask a whole chunk of it. And what we mask is actually kind of a temporal tube, so a whole segment of each frame in the video over the entire video. And that tube is like statically positioned throughout the frames, a straight tube. The tube typically is 16 frames or something, and we mask the same region over the entire 16 frames. It's a different one for every video obviously. And then again train that system so as to predict the representation of the full video from the partially masked one.

我们有一个系统可以学习良好的视频表征。当你将这些表征输入到一个有监督的分类器头时，它能够以相当高的准确率告诉你视频中正在发生什么动作。这是我们第一次获得如此高质量的结果，所以这是一个很好的测试，表明形成了良好的表征。我们还有初步结果似乎表明，这种表征让我们的系统能够判断视频在物理上是否可能或完全不可能，因为某个物体消失了，或者突然从一个位置跳到另一个位置，或者改变了形状。所以它能够捕捉到关于视频所代表现实的一些基于物理的约束，关于物体的出现和消失。

We have a system that learns good representations of video. When you feed those representations to a supervised classifier head, it can tell you what action is taking place in the video with pretty good accuracy. It's the first time we get something of that quality, so it's a good test that a good representation is formed. We also have preliminary results that seem to indicate that the representation allows our system to tell whether the video is physically possible or completely impossible, because some object disappeared or suddenly jumped from one location to another or changed shape. So it's able to capture some physics-based constraints about the reality represented in the video, about the appearance and disappearance of objects.

但这真的能让我们达到那种对世界有足够理解、能够驾驶汽车的世界模型吗？

But can this actually get us to this kind of world model that understands enough about the world to be able to drive a car?

有可能。这还需要一段时间才能达到那个点。已经有基于这个想法的系统了。你需要的是一个稍微修改的版本：你有一个完整的视频，然后要么在时间上向未来平移——所以你只看到视频的开头而不是后半部分——要么你遮住后半部分。然后你训练一个 JEA 系统来从平移后的版本预测完整视频的表征，但同时你也向预测器输入一个动作，比如方向盘向右转了 10 度。如果是车上的行车记录仪，你知道方向盘的角度，就应该能在一定程度上预测将要发生的事情。你不会预测到出现的物体的所有细节，但在抽象表征层面，你大概能预测将要发生什么。所以现在你有了一个内部模型，它说：这是我在时间 T 对世界状态的看法，这是我正在采取的动作，这是对时间 T+delta T 世界状态的预测。如果你有这种模型，你就可以用它来进行规划。所以现在你可以做 LLM 做不到的事情：规划你要做什么以达到特定的结果或满足特定的目标。你可以有多个目标。例如，如果我有一个这样的物体，我松开手，它会掉下去。如果我用一定的力在桌子上推它，它会移动。如果我推桌子本身，它可能不会以同样的力移动。所以我们头脑中有这个内部世界模型，它允许我们规划一系列动作以达到特定目标。所以现在如果你有这个世界模型，我们可以想象一系列动作，预测结果是什么，衡量最终状态在多大程度上满足特定目标，然后在运行时规划一系列动作来最小化这个目标。我们不是在谈论学习，而是在谈论推理时间。这就是规划，在最优控制中它被称为模型预测控制。你有一个你想控制的系统的模型，它可以预测与一系列命令相对应的状态序列，然后你规划一系列命令，以便根据你的世界模型，最终状态满足你设定的目标。这就是自 60 年代初以来火箭轨迹规划的方式。

Possibly. This is going to take a while before we get to that point. There are already systems based on this idea. What you need is a slightly modified version where you have a complete video, and you either translate it in time towards the future—so you only see the beginning of the video but not the latter part—or you mask the second half. Then you train a JEA system to predict the representation of the full video from the shifted one, but you also feed the predictor with an action, for example, the wheel is turned 10 degrees to the right. If it's a dash cam in a car and you know the angle of the wheel, you should be able to predict to some extent what's going to happen. You won't predict all the details of objects that appear, but at an abstract representation level, you can probably predict what's going to happen. So now you have an internal model that says: here is my idea of the state of the world at time T, here is an action I'm taking, here's a prediction of the state of the world at time T+delta T. If you have a model of this type, you can use it for planning. So now you can do what LLMs cannot do: planning what you're going to do to arrive at a particular outcome or satisfy a particular objective. You can have a number of objectives. For example, if I have an object like this and I open my hand, it's going to fall. If I push it with a particular force on the table, it's going to move. If I push the table itself, it's probably not going to move with the same force. So we have this internal model of the world in our mind, which allows us to plan sequences of actions to arrive at a particular goal. So now if you have this world model, we can imagine a sequence of actions, predict what the outcome will be, measure to what extent the final state satisfies a particular objective, and then plan a sequence of actions that will minimize this objective at run time. We're not talking about learning; we're talking about inference time. This is planning, and in optimal control it's called model predictive control. You have a model of the system you want to control that can predict the sequence of states corresponding to a sequence of commands, and you plan a sequence of commands so that according to your world model, the end state will satisfy an objective you fix. This is how rocket trajectories have been planned since the early 60s.

但你也经常谈到分层规划。分层规划能从这个中涌现出来吗？

But you also often talk about hierarchical planning. Can hierarchical planning emerge from this somehow?

不，你必须构建一个特定的架构来实现分层规划。如果你想规划复杂的行动，分层规划是绝对必要的。如果我想从纽约去巴黎——我一直用的例子——我坐在纽约大学的办公室里，我的目标是在高层次上最小化我到巴黎的距离，这是一个非常抽象的位置表征。我必须把这个分解成两个子目标：第一，去机场；第二，乘飞机去巴黎。所以我的子目标现在是去机场；我的目标函数是我到机场的距离。我怎么去机场？我必须走到街上叫一辆出租车，这在纽约是可以做到的。现在我有另一个子目标：下到街上。嗯，那意味着去电梯，下电梯，走到街上。我怎么去电梯？我必须从椅子上站起来，打开办公室的门，走到电梯，按下按钮。我怎么从椅子上站起来？你可以想象一直分解到毫秒级的肌肉控制。显然，你不会以毫秒级的肌肉控制来规划你从纽约到巴黎的整个行程。首先，那会极其昂贵，而且也完全不可能，因为你不知道将要发生的所有条件——叫出租车需要多长时间，或者去机场路上交通情况如何。你必须确切知道所有条件才能做这个规划，而你没有这些信息。所以你必须做分层规划，这样你就可以开始行动，然后边走边重新规划。在 AI 领域，没有人真正知道如何做到这一点。没有人知道如何训练一个系统来学习适当的多层次表征，使得分层规划能够工作。

No, you will have to build a specific architecture to allow for hierarchical planning. Hierarchical planning is absolutely necessary if you want to plan complex actions. If I want to go from New York to Paris—the example I use all the time—and I'm sitting in my office at NYU, my objective is to minimize my distance to Paris at a high level, a very abstract representation of my location. I would have to decompose this into two sub-goals: first, go to the airport; second, catch a plane to Paris. So my sub-goal is now going to the airport; my objective function is my distance to the airport. How do I go to the airport? I have to go to the street and hail a taxi, which you can do in New York. Now I have another sub-goal: go down to the street. Well, that means going to the elevator, going down the elevator, walking out to the street. How do I go to the elevator? I have to stand up from my chair, open the door of my office, go to the elevator, push the button. How do I get up from my chair? You can imagine going all the way down to millisecond-by-millisecond muscle control. Obviously, you're not going to plan your entire trip from New York to Paris in terms of millisecond-by-millisecond muscle control. First, that would be incredibly expensive, but it would also be completely impossible because you don't know all the conditions of what's going to happen—how long it's going to take to catch a taxi, or to go to the airport with traffic. You would have to know exactly the condition of everything to be able to do this planning, and you don't have the information. So you have to do hierarchical planning so that you can start acting and then sort of replan as you go. Nobody really knows how to do this in AI. Nobody knows how to train a system to learn the appropriate multiple levels of representation so that hierarchical planning works.

类似的东西已经涌现出来了吗？所以你能用一个 LLM，一个最先进的 LLM，来帮你从纽约到巴黎，通过做你刚才做的那套详细的问题：你能给我一个高层次列表，列出从纽约到巴黎需要做的 10 个步骤，然后对于每个步骤，你能给我一个 10 个步骤的列表，说明如何实现那个步骤，然后对于每个步骤，你能给我一个 10 个步骤的列表，直到你移动肌肉？

Does something like that already emerge? So can you use an LLM, a state-of-the-art LLM, to get you from New York to Paris by doing exactly the kind of detailed set of questions that you just did: can you give me a high-level list of 10 steps I need to do to get from New York to Paris, and then for each of those steps, can you give me a list of 10 steps how I make that step happen, and for each of those steps, can you give me a list of 10 steps to make each one of those, until you're moving your muscles?

单个肌肉，也许不行，任何你能用大脑实际控制的东西。所以这隐含了很多问题。首先，LLM 能够在一定程度上回答其中一些问题，前提是它们的训练集中有类似场景。它们能回答所有这些问题，但有些可能是幻觉，即不真实的。

Individual muscles, maybe not, whatever you can actually act upon using your mind, right. So there's a lot of questions that are sort of implied by this. The first thing is LLMs will be able to answer some of those questions down to some level of abstraction, under the condition that they've been trained with similar scenarios in their training set. They would be able to answer all those questions, but some of them may be hallucinated, meaning non-factual.

没错。

True.

我的意思是，它们可能会给出一些答案，但无法真正提供毫秒级的肌肉控制，比如你如何从椅子上站起来。但在一定抽象层面上，我们可以用语言描述事物。它们或许能给你一个计划，但前提是它们受过训练来生成这类计划。它们无法为从未遇到过的情况做计划；基本上只能复述训练过的模板。

I mean, they will probably produce some answer, except they're not going to be able to really produce millisecond-by-millisecond muscle control of how you stand up from your chair, right. But down to some level of abstraction, we can describe things by words. They might be able to give you a plan, but only under the condition that they've been trained to produce those kinds of plans. They're not going to be able to plan for situations they've never encountered before; they basically have to regurgitate the template they've been trained on.

对。但以纽约到巴黎为例，它会在哪里出问题？你觉得从哪个抽象层面开始？因为我可以想象这个计划的几乎每个部分都能相当准确地回答，尤其是当你说的是纽约和巴黎这样的大城市时。

Right. But for the example of New York to Paris, is it going to start getting into trouble? At which layer of abstraction do you think you'll start? Because I can imagine almost every single part of that plan will be able to answer somewhat accurately, especially when you're talking about New York and Paris, major cities.

当然，如果你对 LLM 进行微调，它就能解决这个问题。我不能说 LLM 做不到；如果你针对它进行训练，它就能做到，毫无疑问，只要问题能在一定层面上用语言表述。但如果你想用语言描述如何爬下楼梯或从椅子上站起来，那就做不到了。这就是为什么你需要物理世界的体验，其带宽远高于人类语言所能表达的。

Certainly, an LLM would be able to solve that problem if you fine-tune it for it. I can't say that an LLM cannot do this; it can do this if you train it for it, no question, down to a certain level where things can be formulated in terms of words. But if you want to go down to how you climb down the stairs or just stand up from your chair in terms of words, you can't do it. That's one of the reasons you need experience of the physical world, which is much higher bandwidth than what you can express in words in human language.

那么，我们一直在讨论的联合嵌入空间，是否就是我们在机器人领域与物理现实交互所需要的，而 LLM 则作为其上层进行更宏观的推理？比如我需要订机票，知道如何访问网站等等。

So everything we've been talking about on the joint embedding space, is it possible that that's what we need for the interaction with physical reality on the robotics front, and then the LLMs are the thing that sits on top of it for the bigger reasoning? Like the fact that I need to book a plane ticket and I need to know how to go to the websites and so on.

当然。很多人们知道的相对高层次的计划实际上是学来的。大多数人不会自己发明计划。我们当然有一些这样的能力，但人们使用的大多数计划都是他们训练过的，比如他们看到别人使用这些计划，或者被告知如何做。你无法让一个从未听说过飞机的人凭空想出如何从纽约到巴黎；除非他们之前见过例子，否则他们很可能无法解构整个计划。所以 LLM 当然能够做到这一点。但如何将这一点与低层次的动作联系起来，需要像 JEPA 这样的东西，它基本上提升了表示的抽象层次，而不试图重建每个细节。这就是我们需要 JEPA 的原因。

Sure. And a lot of plans that people know about that are relatively high level are actually learned. Most people don't invent the plans by themselves. We have some ability to do this, of course, but most plans that people use are plans that they've been trained on, like they've seen other people use those plans or they've been told how to do things. You can't invent how to take a person who's never heard of airplanes and tell them how to go from New York to Paris; they're probably not going to be able to deconstruct the whole plan unless they've seen examples before. So certainly LLMs are going to be able to do this. But then how you link this from the low level of actions needs to be done with things like JEPA that basically lift the abstraction level of the representation without attempting to reconstruct every detail of the situation. That's why we need JEPA for.

我很想深入探讨你对自回归 LLM 的怀疑。我想检验这种怀疑的一个方式是：你说的都很有道理，但如果我把你今天说的和一般情况应用到，比如十年前，或者稍微近一点，三年前，我无法预测 LLM 的成功。那么，自回归 LLM 能变得如此出色，你觉得合理吗？你能解释一下你的直觉吗？因为如果我把你的智慧和直觉当真，我会说自回归 LLM 一次一个 token 不可能做到它们现在做的事情。

I would love to sort of linger on your skepticism around autoregressive LLMs. One way I would like to test that skepticism is: everything you say makes a lot of sense, but if I apply everything you said today and in general to, I don't know, 10 years ago, maybe a little bit less, let's say three years ago, I wouldn't be able to predict the success of LLMs. So does it make sense to you that autoregressive LLMs are able to be so damn good? Can you explain your intuition? Because if I were to take your wisdom and intuition at face value, I would say there's no way autoregressive LLMs, one token at a time, would be able to do the kind of things they're doing.

不，自回归 LLM，或者说一般的 LLM，不仅仅是自回归的，还包括双向的，它们利用了一件事，那就是自监督学习。多年来我一直是自监督学习的坚定倡导者。所以这些东西令人难以置信地证明了自监督学习确实有效。这个想法是，你取一段文本，破坏它，然后训练一个巨大的神经网络来重建缺失的部分，这带来了巨大的好处。它使我们能够创建理解语言的系统，能够翻译数百种语言任意方向的系统，多语言系统，所以它是一个单一系统，可以训练来理解数百种语言并任意方向翻译，还能生成摘要，回答问题并生成文本。然后有一个特例，就是自回归技巧，你约束系统不通过查看整个文本来构建文本表示，而只根据前面的词预测下一个词，通过约束网络架构来实现。这就是构建自回归 LLM 的方法。所以很多年前有一个惊喜，就是所谓的仅解码器 LLM。这类系统只是试图从前一个词生成下一个词，当你扩大它们规模，用大量数据训练并让它们变得非常大时，它们往往能更好地理解语言，这算是一个惊喜。这个惊喜发生在相当一段时间前，来自 Google、Meta、OpenAI 等的工作，追溯到 GPT 类工作，即通用预训练 Transformer。

No, there's one thing that autoregressive LLMs, or LLMs in general, not just the autoregressive ones but including the bidirectional ones, are exploiting, and it's self-supervised learning. I've been a very strong advocate of self-supervised learning for many years. So these things are an incredibly impressive demonstration that self-supervised learning actually works. The idea that you take a piece of text, corrupt it, and then train some gigantic neural net to reconstruct the parts that are missing has produced an enormous amount of benefits. It allowed us to create systems that understand language, systems that can translate hundreds of languages in any direction, systems that are multilingual, so it's a single system that can be trained to understand hundreds of languages and translate in any direction, and produce summaries, and then answer questions and produce text. And then there's a special case of it, which is the autoregressive trick, where you constrain the system to not elaborate a representation of the text from looking at the entire text but only predicting a word from the words that come before, and you do this by constraining the architecture of the network. That's how you can build an autoregressive LLM. So there was a surprise many years ago with what's called decoder-only LLM. Systems of this type that are just trying to produce words from the previous one, and the fact that when you scale them up, they tend to really understand more about language when you train them on a lot of data and make them really big, that was kind of a surprise. That surprise occurred quite a while back, with work from Google, Meta, OpenAI, etc., going back to the GPT kind of work, General Pre-trained Transformers.

你是指像 GPT-2 这样的吗？在某个点上你开始意识到 Scaling（规模扩张）可能真的会持续带来涌现的好处？

Do you mean like GPT-2? Like there's a certain place where you start to realize scaling might actually keep giving us an emergent benefit?

是的，我的意思是来自不同地方的工作，但如果你想把它放在 GPT 的时间线上，那大概是在 GPT-2 左右。

Yeah, I mean there were work from various places, but if you want to place it in the GPT timeline, that would be around GPT-2.

好吧，因为你说了，你太有魅力了，说了这么多，但自监督学习，是的。但同样，你用来论证自回归 LLM 无法深刻理解世界的直觉，如果我们应用同样的直觉，你觉得它们能够形成足够的世界表示，以至于非常令人信服，基本上通过了……

Well, I just because you said it, you're so charismatic, you said so many words, but self-supervised learning, yes. But again, the same intuition you're applying to saying that autoregressive LLMs cannot have a deep understanding of the world, if we just apply that same intuition, does it make sense to you that they're able to form enough of a representation of the world to be damn convincing, essentially passing the...

原始图灵测试轻松通过，但我们被它们的流畅性欺骗了，对吧？我们只是假设，如果一个系统能流利地操纵语言，那么它就具备人类智能的所有特征。但这种印象是错误的。我们确实被它骗了。你觉得艾伦·图灵会怎么说？

Original Turing test with flying colors, well we're fooled by their fluency, right? We just assume that if a system is fluent in manipulating language, then it has all the characteristics of human intelligence. But that impression is false. We're really fooled by it. What do you think Alan Turing would say?

它什么都不理解，只是应付过去。图灵会断定图灵测试是一个很糟糕的测试。好吧，这是人工智能社区多年前就决定的，图灵测试是一个很糟糕的智能测试。

It without understanding anything, just hanging out with it. Turing would decide that a Turing test is a really bad test. Okay, this is what the AI community has decided many years ago, that the Turing test was a really bad test of intelligence.

汉斯·莫拉维克会怎么评价大型语言模型？

What would Hans Moravec say about the large language models?

莫拉维克会说莫拉维克悖论仍然适用。好吧好吧好吧，我们可以通过。你不觉得他会印象深刻吗？不，当然每个人都会印象深刻，但这不是印象深不深刻的问题，而是要知道这些系统的局限在哪里。它们令人印象深刻，能做很多有用的事情，整个行业都在围绕它们建立，它们会取得进展。但有很多事情它们做不到，我们必须认识到它们做不到什么，然后想办法达到那里。而我没有看到这一点。我基本上是基于十年来对自监督学习的研究来看待这个问题的，实际上可以追溯到十多年前。自监督学习的思想是：在不针对任何特定任务训练系统的情况下，捕捉一组输入的内部结构，对吧？学习表示。我 14 年前联合创办的会议叫做国际学习表示会议。这就是深度学习处理的核心问题，也是我近 40 年来一直痴迷的事情。所以学习表示才是关键。在很长一段时间里，我们只能通过监督学习做到这一点，然后我们开始研究我们过去称之为无监督学习的东西，并在 21 世纪初与约书亚·本吉奥和杰夫·辛顿一起复兴了无监督学习的思想。然后发现如果你能收集足够多的数据，监督学习实际上效果很好，所以无监督与监督的整个想法一度退居二线。然后我从 2014 年开始，基本上当我们成立 FAIR 时，我试图大规模地复兴它，并真正推动寻找新的方法来进行自监督学习，无论是文本、图像、视频还是音频。其中一些工作取得了巨大的成功。我的意思是，我们之所以拥有多语言翻译系统，在 Meta（例如 Facebook）上做内容审核，能够理解一段文本是否是仇恨言论，这要归功于将自监督学习用于 NLP，并结合 Transformer 架构等等。但这是自监督学习的巨大成功。我们在语音识别方面也取得了类似的成功，一个叫做 wav2vec 的系统，它也是一种联合嵌入架构，通过对比学习训练。该系统还可以产生多语言的语音识别系统，主要使用未标记数据，只需要几分钟的标记数据就能实际进行语音识别。这太神奇了。我们现在基于这些想法的组合，拥有能够实时将数百种语言相互翻译的系统，语音到语音，甚至包括没有书面形式的语言。没错，它们只有口语。我们不通过文本，它直接从语音到语音，使用一种离散的语音单元的内部表示。但它被称为无文本 NLP，我们过去这样称呼它。所以是的，那里取得了难以置信的成功。然后，在十年里，我们试图将这个想法应用于学习图像的表示，通过训练系统预测视频，通过训练系统预测视频中将要发生的事情来学习直观物理，并尝试了又尝试，用生成模型、预测像素的模型失败了又失败。我们无法让它们学习好的图像表示，无法让它们学习好的视频表示。我们尝试了很多次，发表了很多论文。它们有点效果，但并不是很好。当我们放弃预测每个像素的想法，基本上只做联合嵌入并在表示空间中进行预测时，它们开始起作用了。这很有效。所以有充分的证据表明，我们无法使用生成模型学习现实世界的良好表示。所以我告诉人们，每个人都在谈论生成式 AI。如果你真的对人类级别的 AI 感兴趣，放弃生成式 AI 的想法。

Moravec would say the Moravec Paradox still applies. Okay, okay, okay, we can pass. You don't think he would be really impressed? No, of course everybody would be impressed, but it's not a question of being impressed or not. It's a question of knowing what the limits of those systems can do. They are impressive, they can do a lot of useful things, there's a whole industry being built around them, they're going to make progress. But there is a lot of things they cannot do, and we have to realize what they cannot do, and then figure out how we get there. And I'm not seeing this. I'm seeing this from basically 10 years of research on the idea of self-supervised learning, actually that's going back more than 10 years, but the idea of self-supervised learning: basically capturing the internal structure of a set of inputs without training the system for any particular task, right? Learning representations. The conference I co-founded 14 years ago is called International Conference on Learning Representations. That's the entire issue that deep learning is dealing with, and it's been my obsession for almost 40 years now. So learning representation is really the thing. For the longest time, we could only do this with supervised learning, and then we started working on what we used to call unsupervised learning, and sort of revived the idea of unsupervised learning in the early 2000s with Yoshua Bengio and Geoff Hinton. Then discovered that supervised learning actually works pretty well if you can collect enough data, so the whole idea of unsupervised versus supervised kind of took a backseat for a bit. And then I kind of tried to revive it in a big way starting in 2014, basically when we started FAIR, and really pushing for finding new methods to do self-supervised learning both for text and for images and for video and audio. Some of that work has been incredibly successful. I mean, the reason why we have multilingual translation systems, things to do content moderation on Meta for example on Facebook that are multilingual, that understand whether a piece of text is hate speech or not, is due to the progress using self-supervised learning for NLP, combining this with Transformer architectures and blah blah blah. But that's the big success of self-supervised learning. We had similar success in speech recognition, a system called wav2vec, which is also a joint embedding architecture by the way, trained with contrastive learning. And that system also can produce speech recognition systems that are multilingual with mostly unlabeled data and only need a few minutes of labeled data to actually do speech recognition. That's amazing. We have systems now based on those combination of ideas that can do real-time translation of hundreds of languages into each other, speech to speech, even including languages that don't have written forms. That's right, they are spoken only. We don't go through text, it goes directly from speech to speech using an internal representation of kind of speech units that are discrete. But it's called textless NLP, we used to call it this way. So yeah, incredible success there. And then for 10 years we tried to apply this idea to learning representations of images by training a system to predict videos, learning intuitive physics by training a system to predict what's going to happen in the video, and tried and tried and failed and failed with generative models, with models that predict pixels. We could not get them to learn good representations of images, we could not get them to learn good representations of videos. And we tried many times, we published lots of papers on it. They kind of sort of work but not really great. They started working when we abandoned the idea of predicting every pixel and basically just doing the joint embedding and predicting in representation space. That works. So there's ample evidence that we're not going to be able to learn good representations of the real world using generative models. So I'm telling people, everybody is talking about generative AI. If you're really interested in human-level AI, abandon the idea of generative AI.

但你真的认为联合嵌入表示能走得很远吗？比如有常识推理，还有高级推理。我觉得 LLM 能够做到的是两种推理。好吧，我不说推理这个词，但 LLM 能做的事情似乎与我们用来导航世界的常识推理根本不同。是的，似乎我们需要两者兼得。通过联合嵌入的方法看视频，你能学到如何从纽约到巴黎吗？或者理解当今世界的政治状况？这些事情是人类在语言空间中产生大量语言和观点，但并没有视觉表现，而你显然以某种方式压缩了它。

But you really think it's possible to get far with the joint embedding representation? So like there's common sense reasoning and then there's high-level reasoning. I feel like those are two kinds of reasoning that LLMs are able to do. Okay, let me not use the word reasoning, but the kind of stuff that LLMs are able to do seems fundamentally different than the common sense reasoning we use to navigate the world. Yeah, it seems like we're going to need both. Would you be able to get with the joint embedding type of approach looking at video? Would you be able to learn how to get from New York to Paris, or understand the state of politics in the world today? These are things where various humans generate a lot of language and opinions in the space of language, but don't visually represent that, and you clearly compress it in a way.

嗯，有很多情况对于纯粹基于语言的系统来说可能很难知道。比如，好吧，你可能从阅读文本中了解到世界上所有公开可用的文本，但我不能通过打个响指就从纽约到巴黎。那行不通，对吧？是的。但可能有更复杂的此类场景，LLM 可能从未遇到过，也可能无法判断是否可能。所以从低级到高级的联系，问题是语言表达的高级层面是基于低级层面的共同经验，而 LLM 目前没有这种经验。当我们彼此交谈时，我们知道我们对世界有共同的体验，很多是相似的。而 LLM 没有。但你看，它存在。你和我对世界有共同的体验，就...而言。

Well, there's a lot of situations that might be difficult for a purely language-based system to know. Like, okay, you can probably learn from reading text the entirety of the publicly available text in the world that I cannot get from New York to Paris by snapping my fingers. That's not going to work, right? Yes. But there are probably more complex scenarios of this type which an LLM may never have encountered and may not be able to determine whether it's possible or not. So that link from the low level to the high level, the thing is that the high level that language expresses is based on the common experience of the low level, which LLMs currently do not have. When we talk to each other, we know we have a common experience of the world, a lot of it is similar. And LLMs don't have that. But see, it's present. You and I have a common experience of the world in terms of the...

重力如何作用的物理知识以及诸如此类的世界常识，我觉得语言中已经包含了。我们不会明确表达它，但如果你有海量的文本，你就能得到这些字里行间的东西。为了形成一个一致的世界模型，你必须理解重力如何作用，即使没有对重力的明确解释。所以，尽管重力有明确的解释，但我们认为是常识推理的东西，我觉得要正确生成语言，你必须弄明白这些。现在你可能会说文本不够多。

Physics of how gravity works and stuff like that, common knowledge of the world, I feel like is there in the language. We don't explicitly express it, but if you have a huge amount of text, you're going to get this stuff that's between the lines. In order to form a consistent world model, you're going to have to understand how gravity works, even if you don't have an explicit explanation of gravity. So even though in the case of gravity there are explicit explanations, the stuff that we think of as common sense reasoning, I feel like to generate language correctly, you're going to have to figure that out. Now you could say there's not enough text.

我同意你刚才说的，即要具备高级常识，你需要有低级常识作为基础。但这一点在 LLM 中并不存在；LLM 纯粹是从文本中训练的。所以你提出的另一个观点，我不认同世界上所有语言都隐含了底层现实。关于底层现实有很多东西并没有在语言中表达出来。

I agree with what you just said, which is that to be able to do high-level common sense, you need to have the low-level common sense to build on top of it. But that's not there in LLMs; LLMs are purely trained from text. So the other statement you made, I would not agree with the fact that implicit in all languages in the world is the underlying reality. There's a lot about underlying reality which is not expressed in language.

你觉得这很明显吗？

Is that obvious to you?

是的，完全同意。比如我们所有的对话，暗网，也就是私人对话如私信之类的东西，可能比 LLM 训练所用的数据大得多。你不需要沟通那些常识性的东西，但它会流露出来。比如如果我不小心打翻了这个，你可能会取笑我，而在你取笑我的内容中，会包含杯子会掉、重力这样起作用的事实，以及关于什么东西掉在地上会碎的一些模糊信息，也许你还会开个关于熵的玩笑。你永远无法再重建这个。会有数万亿个其他笑话，从这些笑话中你可以拼凑出重力起作用、杯子会碎这些事实。这效率很低，但我觉得如果你有足够多的数据，它就会存在。

Yeah, totally. Like all the conversations we have, the dark web meaning private conversations like DMs and stuff, which is much larger probably than what LLMs are trained on. You don't need to communicate the stuff that is common, but it comes through. Like if I accidentally knock this over, you'll probably make fun of me, and in the content of you making fun of me, there will be an explanation of the fact that cups fall, gravity works this way, and you'll have some vague information about what kind of things explode when they hit the ground, and maybe you'll make a joke about entropy. You'll never be able to reconstruct this again. There'll be trillions of other jokes, and from the jokes you can piece together that gravity works and mugs can break. It'll be very inefficient, but I feel like it would be there if you have enough of that data.

我只是觉得，我们婴儿时期积累的大部分这类信息，本质上并不存在于文本或任何描述中。感官数据是获得这种理解的更丰富来源。我是说，一个四岁孩子清醒时的 16,000 小时，以及通过视觉接收的 10^15 字节数据，仅视觉一项。触觉也有类似的带宽，听觉略少，而语言要到一岁左右才出现。到 9 岁时，你已经了解了重力、惯性、稳定性、有生命与无生命物体的区别。到 18 个月时，你知道人们为什么想做事情，如果他们做不到你会帮忙。很多东西主要是通过观察学到的，甚至不是通过互动。在生命的最初几个月，婴儿对世界几乎没有影响；他们只能观察。仅从这一点你就积累了大量的知识。所以这就是当前 AI 系统所缺失的。

I just think that most of the information of this type that we have accumulated when we were babies is just not present in text or any description essentially. The sensory data is a much richer source for getting that kind of understanding. I mean, that's the 16,000 hours of wake time of a four-year-old and 10^15 bytes going through vision just from vision. There is a similar bandwidth of touch, a little less through audio, and text doesn't come in until year one in life. By the time you are 9 years old, you've learned about gravity, inertia, stability, the distinction between animate and inanimate objects. By 18 months, you know why people want to do things and you help them if they can't. There's a lot of things that you learn mostly by observation, really, not even through interaction. In the first few months of life, babies don't really have any influence on the world; they can only observe. You accumulate a gigantic amount of knowledge just from that. So that's what we're missing from current AI systems.

在你的一张幻灯片中，有一个很好的图，是你展示 LLM 局限性的方式之一。我想知道你是否能从你的角度谈谈幻觉：为什么大型语言模型会产生幻觉，以及这在多大程度上是大型语言模型的根本缺陷？

In one of your slides, you have this nice plot that is one of the ways you show that LLMs are limited. I wonder if you could talk about hallucinations from your perspective: why hallucinations happen from large language models, and to what degree is that a fundamental flaw of large language models?

对。由于自回归预测，每次语言模型生成一个词元或单词时，该词元都有一定概率将你带出合理答案的集合。如果你假设——这是一个非常强的假设——这种错误的概率在生成的一系列词元中是独立的，那么这意味着每次你生成一个词元，你留在正确答案集合中的概率就会下降，而且是指数级下降。所以有一个很强的假设：如果犯错的概率非零（看起来确实如此），那么就会有一种漂移，而且这种漂移是指数级的。错误会累积。因此，答案变得无意义的概率会随着词元数量指数级增加。

Right. So because of the autoregressive prediction, every time an LM produces a token or word, there is some level of probability for that word to take you out of the set of reasonable answers. If you assume, which is a very strong assumption, that the probability of such error is independent across a sequence of tokens being produced, what that means is that every time you produce a token, the probability that you stay within the set of correct answers decreases, and it decreases exponentially. So there's a strong assumption there that if there's a non-zero probability of making a mistake, which there appears to be, then there's going to be a kind of drift, and that drift is exponential. Errors accumulate. So the probability that an answer would be nonsensical increases exponentially with the number of tokens.

顺便问一下，你觉得这很明显吗？从数学上讲也许是这样，但难道没有一种朝向真理的引力吗？因为平均而言，希望真理在训练集中有很好的体现？

Is that obvious to you by the way? Mathematically speaking maybe, but isn't there a kind of gravitational pull towards the truth because on average, hopefully the truth is well represented in the training set?

不，这基本上是与维度诅咒的斗争。纠正这个问题的方法是，通过让系统为人们可能提出的各种问题生成答案来微调系统。人就是人，所以他们提出的很多问题彼此非常相似，因此你可以通过收集数据覆盖人们可能问的 80%左右的问题，然后微调系统使其为所有这些事情生成好的答案。它很可能能够学会这些，因为它有很大的容量。但随后还有大量你在训练中没有覆盖到的提示，这个集合非常庞大。在所有可能的提示集合中，用于训练的提示比例绝对微小。它只是所有可能提示中极小极小的一个子集。所以系统会在它经过预训练或微调的提示上表现正常，但随后有一个它不可能被训练过的整个空间，因为数量巨大。因此，无论系统经过何种训练以生成适当的张量，你都可以通过找到一个超出它训练过的提示集或相似事物的提示来破坏它，然后它就会生成无意义的内容。

No, it's basically a struggle against the curse of dimensionality. The way you can correct for this is that you fine-tune the system by having it produce answers for all kinds of questions that people might come up with. People are people, so a lot of the questions that they have are very similar to each other, so you can probably cover 80% or whatever of questions that people will ask by collecting data and then fine-tuning the system to produce good answers for all of those things. It's probably going to be able to learn that because it has a lot of capacity. But then there is the enormous set of prompts that you have not covered during training, and that set is enormous. Within the set of all possible prompts, the proportion of prompts that have been used for training is absolutely tiny. It's a tiny, tiny, tiny subset of all possible prompts. So the system will behave properly on the prompts that it has been either pre-trained or fine-tuned on, but then there is an entire space of things that it cannot possibly have been trained on because the number is gigantic. So whatever training the system has been subject to to produce appropriate tensors, you can break it by finding a prompt that will be outside of the set of prompts it has been trained on or things that are similar, and then it will just produce nonsense.

当你说提示时，你是指那个确切的提示，还是指一个在很多部分都非常不同的提示？问一个以前在互联网上没被说过的问题或事情容易吗？

When you say prompt, do you mean that exact prompt or do you mean a prompt that's in many parts very different? Is it easy to ask a question or say a thing that hasn't been said before on the internet?

我的意思是人们已经想出了一些东西，其中……

I mean people have come up with things where...

就像你在提示词里放一串基本上随机的字符，就足以让系统进入一种模式，回答出与没有这些字符时完全不同的内容。这就是一种越狱系统的方法，基本上让它脱离它的条件限制。这是一个非常清晰的演示，但这当然超出了它被设计的功能。如果你实际上拼接出语法合理的句子，它是不是那么容易就被攻破？是的，有些人做过这样的事：你用英文写一个句子，或者用英文问一个问题，它给出一个完全正常的回答，然后你只是把几个词替换成另一种语言中的同一个词，突然回答就变成了完全无意义的内容。所以我想说的是，人类可能产生的提示词中有多大比例会破坏系统？问题在于存在一个长尾。这是很多人在社交网络之类的地方已经意识到的问题，那就是人们会问的东西有一个非常非常长的尾巴。你可以针对 80%或大多数人会问的内容微调系统，但这个长尾太大了，你无法针对所有条件微调系统。最终，系统本质上就像一个巨大的查找表，这不是你真正想要的。你想要的是能够推理的系统，当然它们能够规划。

like you put essentially a random sequence of characters in the prompt and that's enough to kind of throw the system into a mode where it will answer something completely different than it would have answered without this. So that's a way to jailbreak the system, basically get it to go outside of its conditioning. So that's a very clear demonstration of it, but of course that goes outside of what it is designed to do. If you actually stitch together reasonably grammatical sentences, is it that easy to break it? Yeah, some people have done things like you write a sentence in English that has, or you ask a question in English and it produces a perfectly fine answer, and then you just substitute a few words by the same word in another language and all of a sudden the answer is complete nonsense. So I guess what I'm saying is, which fraction of prompts that humans are likely to generate are going to break the system? So the problem is that there is a long tail. This is an issue that a lot of people have realized, you know, in social networks and stuff like that, which is there's a very very long tail of things that people will ask. And you can fine-tune the system for the 80% or whatever of the things that most people will ask, and then this long tail is so large that you're not going to be able to fine-tune the system for all the conditions. And in the end, the system is kind of a giant lookup table, essentially, which is not really what you want. You want systems that can reason, certainly they can plan.

所以 LLM 中发生的推理非常非常原始，你能看出它原始的原因是每个生成的词元所花费的计算量是恒定的。所以如果你问一个问题，而这个问题有一个给定词元数量的答案，那么用于计算该答案的计算量可以精确估计。就像预测网络的大小，你知道，它有 36 层或 92 层或别的什么，乘以词元数量，仅此而已。所以本质上，无论被问的问题是简单回答、复杂回答、还是因为不可判定等原因无法回答，系统能够用于答案的计算量是恒定的，或者与答案产生的词元数量成正比。这不是我们工作的方式。我们推理的方式是，当我们面对一个复杂问题或复杂问题时，我们会花更多时间试图解决和回答它，因为它更困难。有一个预测元素，有一个迭代元素，你通过一遍又一遍地重复来调整你对事物的理解，还有一个层次元素，等等。

So the type of reasoning that takes place in LLM is very very primitive, and the reason you can tell it's primitive is because the amount of computation that is spent per token produced is constant. So if you ask a question and that question has an answer in a given number of tokens, the amount of computation devoted to computing that answer can be exactly estimated. It's like the size of the prediction network, you know, with its 36 layers or 92 layers or whatever it is, multiplied by number of tokens, that's it. And so essentially it doesn't matter if the question being asked is simple to answer, complicated to answer, impossible to answer because it's undecidable or something, the amount of computation the system will be able to devote to the answer is constant, or is proportional to number of tokens produced in the answer. This is not the way we work. The way we reason is that when we're faced with a complex problem or a complex question, we spend more time trying to solve it and answer it because it's more difficult. There's a prediction element, there's an iterative element where you're adjusting your understanding of a thing by going over and over and over, there's a hierarchical element, so on.

这是否意味着 LLM 的一个根本缺陷，还是说这个问题还有更多部分？你现在表现得就像一个 LLM，立刻回答不。它只是低层次的世界模型，我们可以在其之上构建一些这样的机制，比如你说的持久长期记忆或推理等等。但我们需要那个来自语言的世界模型。也许在一个构建良好的世界模型之上构建这种推理系统并不那么困难？

Does this mean that a fundamental flaw of LLMs, or does it mean that there's more part to that question? Now you're just behaving like an LLM, immediately answer no. That it's just the low-level world model on top of which we can then build some of these kinds of mechanisms, like you said persistent long-term memory or reasoning, so on. But we need that world model that comes from language. Is it maybe it is not so difficult to build this kind of reasoning system on top of a well-constructed world model?

好吧，无论困难与否，不久的将来会告诉我们，因为很多人正在研究对话系统的推理和规划能力。我的意思是，即使我们只局限于语言，仅仅拥有在回答之前规划答案的能力，而且这种规划不一定与你用来产生答案的语言相关。所以这种心智模型的概念，让你在说话之前规划你要说什么，这非常重要。我认为未来几年会有很多系统具备这种能力，但这些系统的蓝图将与自回归 LLM 截然不同。

OKAY, whether it's difficult or not, the near future will say, because a lot of people are working on reasoning and planning abilities for dialog systems. I mean, if we're even if we restrict ourselves to language, just having the ability to plan your answer before you answer, in terms that are not necessarily linked with the language you're going to use to produce the answer. So this idea of this mental model that allows you to plan what you're going to say before you say it, that is very important. I think there's going to be a lot of systems over the next few years that are going to have this capability, but the blueprint of those systems would be extremely different from autoregressive LLMs.

所以这和心理学中人类系统 1 和系统 2 的区别是一样的。系统 1 是那种你不需要有意识地思考如何完成就能完成的任务，你直接做就行了。你已经做得足够多，可以下意识地完成，而不需要思考。如果你是一个有经验的司机，你可以不用真正思考就开车，同时还能和别人说话或听收音机。如果你是一个非常经验丰富的棋手，你可以和一个没有经验的棋手下棋而不需要真正思考，你只是识别模式然后下棋。那就是系统 1。所以所有你本能地做而不需要刻意计划和思考的事情。然后还有所有需要你规划的任务。所以如果你不是一个经验丰富的棋手，或者你有经验但和另一个有经验的棋手下棋，你会考虑各种选项，你会思考一段时间，如果你有时间思考，你会比在有限时间的快棋中表现更好。所以这种使用你内部世界模型的刻意规划，即系统 2，这是 LLM 目前无法做到的。

So it's the same difference as the difference between what psychology calls System 1 and System 2 in humans. System 1 is the type of task that you can accomplish without deliberately consciously thinking about how you do them, you just do them. You've done them enough that you can just do it subconsciously, without thinking about them. If you're an experienced driver, you can drive without really thinking about it and you can talk to someone at the same time or listen to the radio. If you are a very experienced chess player, you can play against a non-experienced chess player without really thinking either, you just recognize the pattern and you play. That's System 1. So all the things that you do instinctively without really having to deliberately plan and think about it. And then there are all tasks where you need to plan. So if you are not an experienced chess player, or you are experienced but you play against another experienced chess player, you think about all kinds of options, you think about it for a while, and you're much better if you have time to think about it than you are if you play blitz with limited time. So this type of deliberate planning which uses your internal world model, that System 2, this is what LLMs currently cannot do.

那么我们如何让它们做到这一点呢？我们如何构建一个系统，能够进行这种规划或推理，将更多资源用于复杂问题而不是简单问题？

So how do we get them to do this? How do we build a system that can do this kind of planning or reasoning that devotes more resources to complex problems than to simple problems?

而且它不会是词元的自回归预测。它会更类似于在以前称为概率模型或图模型等东西中推断潜在变量。所以基本上原理是这样的：你知道提示词就像观测变量，模型所做的是它可以衡量一个答案在多大程度上是提示词的好答案。所以把它想象成一个巨大的神经网络，但它只有一个输出，那个输出是一个标量数字，比如说，如果答案是问题的一个好答案则为零，如果答案不是问题的一个好答案则为一个大数字。想象你有这个模型。如果你有这样的模型，你可以用它来产生好的答案，方法是：你产生提示词，然后在可能的答案空间中搜索一个使那个数字最小化的答案。这被称为基于能量的模型。但那个基于能量的模型需要由 LLM 构建的模型。嗯，所以实际上你需要做的是，不是搜索最小化那个能量的可能文本字符串，而是要在抽象表示空间中这样做，所以在某种空间中……

And it's not going to be autoregressive prediction of tokens. It's going to be more something akin to inference of latent variables in what used to be called probabilistic models or graphical models and things of that type. So basically the principle is like this: you know the prompt is like observed variables, and what the model does is that it can measure to what extent an answer is a good answer for a prompt. So think of it as some gigantic neural net but it's got only one output, and that output is a scalar number which is, let's say, zero if the answer is a good answer for the question and a large number if the answer is not a good answer for the question. Imagine you had this model. If you had such a model, you could use it to produce good answers the way you would do is: you produce the prompt and then search through the space of possible answers for one that minimizes that number. That's called an energy-based model. But that energy-based model would need the model constructed by the LLM. Well, so really what you need to do would be to not search over possible strings of text that minimize that energy, but what you would do is do this in abstract representation space, so in the sort of space of...

抽象思维：你会通过最小化模型输出来阐述一个想法，这个输出只是一个标量。这是一个优化过程。系统产生答案的方式是通过优化，即最小化一个目标函数。这是推理，不是训练。系统已经训练好了。所以我们有一个思想的抽象表示，即答案的表示。我们将其输入到一个自回归解码器中，这个解码器可以非常简单，它将这个表示转化为表达该思想的文本。在我看来，这是未来对话系统的蓝图。它们会在将答案转化为文本之前，通过优化来思考和规划答案。

Abstract thoughts: you would elaborate a thought using this process of minimizing the output of your model, which is just a scalar. It's an optimization process. The way the system produces its answer is through optimization by minimizing an objective function. This is inference, not training. The system has been trained already. So we have an abstract representation of the thought, the answer representation. We feed that to an autoregressive decoder, which can be very simple, that turns this into text expressing the thought. In my opinion, that is the blueprint of future dialog systems. They will think about their answer, plan their answer by optimization before turning it into text.

你能确切解释一下优化问题是什么吗？目标函数是什么？你简要描述过，但你在什么空间上进行优化？

Can you explain exactly what the optimization problem is? What's the objective function? You briefly described it, but over what space are you optimizing?

表示空间，那些抽象表示。系统内部有一个抽象表示。你有一个提示。提示通过编码器产生一个表示，也许再通过一个预测器预测出正确答案的表示。但这个表示可能不是一个好答案，因为可能需要进行复杂的推理。所以你有另一个过程，它获取答案的表示并修改它，以最小化一个代价函数，该函数衡量答案对问题的好程度。我们暂时忽略如何训练系统来评估答案的好坏，但假设这样的系统可以创建。这个过程是一种类似搜索的过程，是一个优化过程。如果整个系统是可微的，那么标量输出就是答案表示通过某个神经网络的结果。然后通过梯度下降，通过反向传播梯度，你可以找出如何修改答案的表示以最小化该标量。所以这是基于梯度的推理。现在你在抽象空间中有了答案的表示，你可以将其转化为文本。很酷的一点是，这个表示可以通过梯度下降进行优化，而与你表达答案的语言无关。你在抽象表示中操作。这又回到了联合嵌入：在概念空间中工作比在具体感官信息空间中更好。

The space of representations, those abstract representations. You have an abstract representation inside the system. You have a prompt. The prompt goes through an encoder, produces a representation, perhaps goes through a predictor that predicts a representation of the proper answer. But that representation may not be a good answer because there might be complicated reasoning you need to do. So you have another process that takes the representation of the answer and modifies it to minimize a cost function that measures to what extent the answer is a good answer for the question. We ignore for the moment how you train that system to measure whether an answer is good, but suppose such a system could be created. The process is a kind of search-like process, an optimization process. If the entire system is differentiable, that scalar output is the result of running the answer representation through some neural net. Then by gradient descent, by backpropagating gradients, you can figure out how to modify the representation of the answer to minimize that. So it's gradient-based inference. Now you have a representation of the answer in abstract space, and you can turn it into text. The cool thing is that the representation can be optimized through gradient descent independently of the language in which you will express the answer. You are operating in the abstract representation. This goes back to the joint embedding: it's better to work in the space of concepts versus the space of concrete sensory information.

但这能做像推理这样的事情吗？我们正在讨论推理。

But can this do something like reasoning, which is what we're talking about?

不太能，只能以非常简单的方式。你可以想到那些做我所说的那种优化的东西，只不过它们在离散空间中优化，即可能的词元序列空间。它们以一种极其低效的方式进行这种优化：生成大量假设，然后选择最好的。这在计算上非常浪费，因为你必须为每个生成的序列运行语言模型。在连续空间中进行优化要好得多，你可以做梯度下降，而不是生成大量东西然后选择最好的。你只需迭代地优化你的答案，使其趋向最佳。这样效率高得多。你只能在连续空间中使用可微函数做到这一点。

Not really, only in a very simple way. You can think of those things that do the kind of optimization I was talking about, except they optimize in a discrete space, which is the space of possible sequences of tokens. They do this optimization in a horribly inefficient way: generate a lot of hypotheses and then select the best ones. That's incredibly wasteful in terms of computation because you have to run your language model for every generated sequence. It's much better to do optimization in continuous space where you can do gradient descent, as opposed to generating tons of things and then selecting the best. You just iteratively refine your answer towards the best. That's much more efficient. You can only do this in continuous spaces with differentiable functions.

你在谈论深入思考或深入推理的能力。你怎么知道哪个答案更好或更差，基于深入推理？

You're talking about the ability to think deeply or reason deeply. How do you know what is an answer that's better or worse based on deep reasoning?

从概念上讲，如何训练一个基于能量的模型？基于能量的模型是一个输出标量的函数。你给它两个输入，X 和 Y，它告诉你 Y 是否与 X 兼容。X 是你观察到的，比如提示、图像、视频等。Y 是答案或延续的提议。它通过输出零（如果兼容）或正数（如果不兼容）来告诉你 Y 是否与 X 兼容。如何训练这样的系统？在完全一般的层面上，你向它展示兼容的 X 和 Y 对，即问题和相应的答案，然后训练大神经网络的参数以输出零。但这并不完全有效，因为系统可能决定对所有东西都输出零。所以你需要一个过程来确保对于错误的 Y，能量大于零。有两种选择。一种是对比方法：你展示一个 X 和一个坏的 Y，然后告诉系统给这个高能量，推高能量。问题是，如果 Y 的空间很大，你需要对比样本的数量是巨大的。人们在训练 RLHF 中的奖励模型时就是这样做的，奖励模型基本上是一个目标函数，告诉你答案是好是坏。这正是这个。我们在某种程度上已经这样做了，但我们只是用它来训练，而不是推理。还有另一类方法是非对比的，我更喜欢那些。它们说：能量函数需要在来自训练集的兼容 X 和 Y 对上具有低能量。如何确保能量在其他地方更高？你通过一个正则化器，在你的代价函数中有一个准则，最小化可以取低能量的空间体积。具体方式取决于架构，但这是基本原则。所以如果你在 XY 空间的特定区域压低能量函数，它会在其他地方自动上升，因为只有有限的空间体积可以取低能量，这是由系统的构造决定的。

Conceptually, how do you train an energy-based model? An energy-based model is a function with a scalar output. You give it two inputs, X and Y, and it tells you whether Y is compatible with X. X is what you observe, say a prompt, image, video, etc. Y is a proposal for an answer or continuation. It tells you whether Y is compatible with X by outputting zero if compatible, and a positive number if not. How do you train such a system? At a completely general level, you show it pairs of X and Y that are compatible, a question and the corresponding answer, and you train the parameters of the big neural net to produce zero. But that doesn't completely work because the system might decide to output zero for everything. So you need a process to ensure that for a wrong Y, the energy is larger than zero. There are two options. One is contrastive methods: you show an X and a bad Y, and you tell the system to give high energy to this, push up the energy. The problem is that if the space of Y is large, the number of contrastive samples you need is gigantic. People do this when training a reward model in RLHF, which is basically an objective function that tells you whether an answer is good or bad. That's exactly what this is. We already do this to some extent, but we're just using it for training, not for inference. There is another set of methods which are non-contrastive, and I prefer those. They say: the energy function needs to have low energy on pairs of X and Y that are compatible from your training set. How do you make sure the energy is higher everywhere else? You do this by having a regularizer, a criterion in your cost function that minimizes the volume of space that can take low energy. The precise way depends on the architecture, but that's the basic principle. So if you push down the energy function for particular regions in the XY space, it will automatically go up in other places because there's only a limited volume of space that can take low energy, by the construction of the system.

正则化函数我们一直在泛泛而谈，但什么是好的 X 和好的 Y？什么是 X 和 Y 的好表示？因为我们一直在讨论语言，如果你直接拿语言本身，那大概不好，所以必须有某种思想的抽象表示。

Regularizer regularizing function we've been talking very generally but what is a good X and a good Y? What is a good representation of X and Y? Because we've been talking about language and if you just take language directly that presumably is not good, so there has to be some kind of abstract representation of ideas.

是的，你可以直接用语言来做，比如 X 是文本，Y 是它的续写，或者 X 是问题，Y 是答案。但你说这不行？我是说，这就能做到 LLM 在做的事。嗯，不，这取决于系统的内部结构如何构建。如果系统的内部结构构建得使得系统内部有一个潜在变量 Z，你可以操纵它以最小化输出能量，那么 Z 就可以被视为一个好答案的表示，你可以将其翻译成好的 Y。所以这类系统可以用非常类似的方式训练，但你必须有一种防止坍缩的方法，确保对于你没有训练过的东西有高能量。目前这在 LLM 中非常隐式，以一种人们没有意识到的方式在进行，但确实在进行。这是因为当你给一个词高概率时，自动地你就给其他词低概率，因为你只有有限的总概率。所以当你最小化交叉熵时，当你训练你的 LLM 预测下一个词时，你增加了系统给正确词的概率，但也降低了给错误词的概率。间接地，这给了好的词序列高概率，坏的词序列低概率，但这是非常直接的。而且这为什么能工作并不明显，因为你并不是在序列中所有符号的联合概率上做，你只是将其分解为连续 token 上的条件概率。

Yeah, so you can do this with language directly, by just X is a text and Y is a continuation of that text, or X is a question and Y is the answer. But you're saying that's not going to take it? I mean, that's going to do what LLMs are doing. Well, no, it depends on how the internal structure of the system is built. If the internal structure of the system is built in such a way that inside of this system there is a latent variable called Z that you can manipulate so as to minimize the output energy, then that Z can be viewed as a representation of a good answer that you can translate into a Y that is a good answer. So this kind of system could be trained in a very similar way, but you have to have this way of preventing collapse, of ensuring that there is high energy for things you don't train it on. And currently it's very implicit in LLMs, done in a way that people don't realize it's being done, but it is being done. It is due to the fact that when you give a high probability to a word, automatically you give low probability to other words because you only have a finite amount of probability to go around. So when you minimize the cross entropy or whatever, when you train your LLM to predict the next word, you're increasing the probability your system will give to the correct word, but you're also decreasing the probability it will give to the incorrect words. Now indirectly that gives a high probability to sequences of words that are good and low probability to sequences of words that are bad, but it's very direct. And it's not obvious why this actually works at all, because you're not doing it on a joint probability of all the symbols in a sequence, you're just doing it sort of factorizing that probability in terms of conditional probabilities over successive tokens.

那么对于视觉数据，你是如何做的？

So how do you do this for visual data?

所以我们一直在用所有 JEPA 架构来做这件事。联合嵌入预测架构：两个事物之间的兼容性是，这里有一张图像或视频，这里是该图像或视频的损坏、移位或变换版本，或者被掩码。然后系统的能量是表示的预测误差，即好事物的预测表示与好事物的实际表示之间的误差。所以你把损坏的图像输入系统，预测未损坏的好输入的表示，然后计算预测误差。这就是系统的能量。所以这个系统会告诉你，如果这是一张好图像，而这是它的损坏版本，如果两者实际上是其中一个为另一个的损坏版本，它会给你零能量；如果两幅图像完全不同，则给你高能量。希望整个过程能给你一个非常好的现实压缩表示，视觉现实的压缩表示。我们知道它确实如此，因为之后我们将这些表示用作分类系统的输入，效果非常好。

So we've been doing this with all JEPA architectures basically. The joint embedding predictive architecture: the compatibility between two things is, here's an image or a video, here's a corrupted, shifted, or transformed version of that image or video, or masked. And then the energy of the system is the prediction error of the representation, the predicted representation of the good thing versus the actual representation of the good thing. So you run the corrupted image to the system, predict the representation of the good input uncorrupted, and then compute the prediction error. That's the energy of the system. So this system will tell you if this is a good image and this is a corrupted version, it will give you zero energy if those two things are effectively one of them is a corrupted version of the other, give you high energy if the two images are completely different. And hopefully that whole process gives you a really nice compressed representation of reality, of visual reality. And we know it does because then we use those representations as input to a classification system, something works really nicely.

好的，那么总结一下，你以只有 Yann LeCun 能用的辛辣方式建议：我们放弃生成模型，转而使用联合嵌入架构？是的。放弃自回归生成？是的。放弃概率模型，转而使用我们讨论过的基于能量的模型？放弃对比方法，转而使用正则化方法。让我问你这个问题：你一直批评强化学习。是的。那么最后一个建议是，我们放弃 RL，转而使用你提到的模型预测控制，并且只在规划不能产生预测结果时才使用 RL，在这种情况下我们用 RL 来调整世界模型或评论家。

Okay, well, so to summarize, you recommend, in a spicy way that only Yann LeCun can, you recommend that we abandon generative models in favor of joint embedding architectures? Yes. Abandon autoregressive generation? Yes. Abandon probabilistic models in favor of energy-based models as we talked about? Abandon contrastive methods in favor of regularized methods. And let me ask you about this: you've been for a while a critic of reinforcement learning. Yes. So what the last recommendation is that we abandon RL in favor of model predictive control as you were talking about, and only use RL when planning doesn't yield the predicted outcome, and we use RL in that case to adjust the world model or the critic.

是的。所以你提到了 RLHF，基于人类反馈的强化学习。为什么你仍然讨厌强化学习？我不讨厌强化学习，我认为它不应该被完全放弃，但我认为它的使用应该最小化，因为它在样本方面极其低效。所以训练系统的正确方法是先让它从大多数观察中学习好的世界表示和世界模型，可能有一点交互，然后基于此进行引导。如果表示是好的，那么调整应该是最小的。现在有两件事你可以用：如果你已经学习了世界模型，你可以用世界模型来规划一系列动作以达到特定目标。你不需要 RL，除非你衡量成功的方式可能不精确，比如你对是否会从自行车上摔下来的判断可能是错的，或者你在 MMA 中对抗的人可能会做别的事。所以有两种方式你可能出错：要么你的目标函数没有反映你想要优化的实际目标函数，要么你的世界模型不准确。所以如果你想在操作世界时调整你的世界模型或目标函数，这基本上属于 RL 的领域。这就是 RL 在一定程度上处理的问题。所以调整你的世界模型，甚至提前调整世界模型的方法是探索你知道世界模型不准确的空间部分。这基本上叫做好奇心或玩耍。当你玩耍时，你探索那些在现实生活中不想做的空间部分，因为它可能危险，但你可以基本上在不自杀的情况下调整你的世界模型。所以这就是你想用 RL 的地方：当学习特定任务时，你已经有了所有好的表示，你已经有了你的世界模型，但你需要针对当前情况调整它。那时你就使用 RL。

Yes. So you mentioned RLHF, reinforcement learning with human feedback. Why do you still hate reinforcement learning? I don't hate reinforcement learning, and I think it should not be abandoned completely, but I think its use should be minimized because it's incredibly inefficient in terms of samples. So the proper way to train a system is to first just have it learn good representations of the world and world models from mostly observation, maybe a little bit of interactions, and then steered based on that. If the representation is good, then the adjustment should be minimal. Now there's two things you can use: if you've learned a world model, you can use the world model to plan a sequence of actions to arrive at a particular objective. You don't need RL unless the way you measure whether you succeed might be inexact, your idea of whether you're going to fall from your bike might be wrong, or whether the person you're fighting with in MMA is going to do something else. So there are two ways you can be wrong: either your objective function does not reflect the actual objective function you want to optimize, or your world model is inaccurate. So if you want to adjust your world model while you are operating in the world, or your objective function, that is basically in the realm of RL. This is what RL deals with to some extent. So adjust your world model, and the way to adjust your world model even in advance is to explore parts of the space where you know that your world model is inaccurate. That's called curiosity basically, or play. When you play, you kind of explore parts of the space that you don't want to do in real life because it might be dangerous, but you can adjust your world model without killing yourself basically. So that's what you want to use RL for: when it comes time to learning a particular task, you already have all the good representations, you already have your world model, but you need to adjust it for the situation at hand. That's when you use RL.

为什么你认为 RLHF 效果这么好？这种基于人类反馈的强化学习，为什么它对大型语言模型产生了如此变革性的影响？

Why do you think RLHF works so well? This reinforcement learning with human feedback, why did it have such a transformational effect on large language models?

产生变革性影响的是人类反馈。有很多使用它的方法，其中一些实际上纯粹是监督学习。它并不是真正的强化学习。所以是 HF，是人类反馈。然后有使用人类反馈的方法。所以你可以问……

What's had the transformational effect is human feedback. There's many ways to use it, and some of it is just purely supervised actually. It's not really reinforcement learning. So it's the HF, it's the human feedback. And then there are ways to use human feedback. So you can ask...

人类对世界模型生成的多个答案进行评分。然后你训练一个目标函数来预测这个评分，并用这个目标函数来判断答案的好坏。你可以通过它反向传播梯度来找到新的系统，使其只产生高评分的答案。这是一种方法，类似于强化学习，即训练所谓的奖励模型——一个估计答案好坏程度的模型。这和我之前提到的用于规划的目标非常相似，只不过现在它不是用于规划，而是用于微调你的系统。我认为用它来规划会更高效，但目前它被用来微调系统的参数。有几种方法可以做到这一点——有些是监督式的：你问人类什么是对这个问题的好答案，然后你输入答案。现在有很多调整这些系统的方法。

Humans rate answers multiple answers that are produced by a world model. Then you train an objective function to predict that rating, and you can use that objective function to predict whether an answer is good. You can backpropagate gradients through this to find a new system so that it only produces highly rated answers. That's one way, like in RL, meaning training what's called a reward model—something that estimates to what extent an answer is good. It's very similar to the objective I was talking about earlier for planning, except now it's not used for planning; it's used for fine-tuning your system. I think it would be much more efficient to use it for planning, but currently it's used to fine-tune the parameters of the system. There are several ways to do this—some of them are supervised: you ask a human what is a good answer for this, then you type the answer. There are lots of ways those systems are being adjusted now.

现在很多人对最近发布的谷歌 Gemini 1.5 提出了批评，用我的话来说，它过于‘觉醒’了，带有负面含义。它做了一些几乎荒谬可笑的事情，比如修改历史——生成黑人乔治·华盛顿的图片。或者更严重的是，你在推特上评论过的事情：它拒绝评论或生成关于天安门广场或‘坦克人’的图片甚至描述，这是历史上最具标志性的抗议图像之一。当然，这些图像被中国政府严格审查。因此，每个人都在问：设计这些大语言模型的过程是什么？审查在这些事情中扮演什么角色？所以你在推特上说开源是答案。你能解释一下吗？

Now a lot of people have been very critical of the recently released Google's Gemini 1.5 for essentially, in my words, being super woke in the negative connotation of that word. There is some almost hilariously absurd things that it does, like it modifies history—generating images of a black George Washington. Or perhaps more seriously, something that you commented on Twitter: refusing to comment on or generate images of, or even descriptions of, Tiananmen Square or the Tank Man, one of the most legendary protest images in history. Of course these images are highly censored by the Chinese government. Therefore everybody starts asking questions: what is the process of designing these LLMs? What is the role of censorship in all that kind of stuff? So you commented on Twitter saying that open source is the answer. Can you explain?

我实际上在我能用的每个社交网络上发表了那个评论，并且我在多个场合多次提出这个观点。以下是我对此的看法。人们可以抱怨人工智能系统有偏见，它们通常受到训练数据分布的偏见影响，这反映了社会中的偏见。这可能冒犯一些人，也可能不会。一些去偏见的技术随后会冒犯一些人，因为历史不正确性等原因。所以你可以问两个问题。第一个问题是：是否可能制造一个没有偏见的人工智能系统？答案绝对是否定的，这并非因为技术挑战，尽管存在技术挑战。这是因为偏见在旁观者的眼中。不同的人可能对什么构成偏见有不同的看法。有些事实是无可争议的，但有很多观点或事情可以用不同的方式表达。所以你不可能有一个无偏见的系统；这是不可能的。那么答案是什么？答案与我们在自由民主中关于新闻界的答案相同：新闻必须自由且多样化。我们有言论自由是有充分理由的，因为我们不希望所有信息都来自单一来源，因为这违背了民主、思想进步甚至科学的整个理念。在科学中，人们必须为不同观点争论，科学在人们意见分歧、得出答案并形成共识时取得进步。这在世界各地的民主国家都是如此。所以有一个未来，已经在发生，我们与数字世界的每一次互动都将由人工智能系统、人工智能助手来中介。我们将拥有智能眼镜——你已经在 Meta 的 Ray-Ban Meta 上买到了——你可以和它们交谈，它们连接着大语言模型，你可以得到任何问题的答案。你可以看着一座纪念碑，眼镜里有摄像头；你可以问它：‘你能告诉我关于这座建筑或纪念碑的什么？’你可以看着一份外语菜单，它会为你翻译。或者如果我们说不同的语言，我们可以进行实时翻译。所以在不久的将来，我们与数字世界的许多互动都将由这些系统中介。我们使用的搜索引擎将不再是搜索引擎，而是对话系统，你只需提问，它就会回答，然后可能指向合适的参考资料。但问题是：我们不能让这些系统来自美国西海岸的少数几家公司，因为这些系统将构成所有人类知识的宝库，我们不能让它们被少数人控制。它必须多样化，原因与新闻必须多样化相同。那么我们如何获得多样化的 AI 助手呢？目前训练一个基础模型、一个基础大语言模型非常昂贵和困难。未来可能会有所不同，但目前是大语言模型。所以只有少数公司能做好这件事。如果其中一些顶级系统是开源的，任何人都可以使用，任何人都可以微调它们。如果我们建立一些系统，允许任何群体——无论是个人公民、公民团体、政府组织、非政府组织、公司等——利用这些开源人工智能系统，根据自己的数据为自己的目的进行微调，那么我们将拥有非常多样化的、专门用于所有这些事情的不同人工智能系统。我和法国政府谈过很多次，法国政府不会接受其所有公民的数字饮食由美国西海岸的三家公司控制。这是不可接受的；无论这些公司意图多么良好，这对民主都是危险。这对当地文化、价值观、语言也是危险。我和印度 Infosys 的创始人谈过；他正在资助一个项目，微调 Meta 的开源模型 Llama 2，使其能说印度所有 22 种官方语言。这对印度人民非常重要。我和我的一位前同事 Mustafa 谈过，他曾在 FAIR 担任科学家，然后搬回非洲；他为谷歌创建了一个研究实验室……

I actually made that comment on just about every social network I can, and I've made that point multiple times in various forums. Here's my point of view on this. People can complain that AI systems are biased, and they generally are biased by the distribution of the training data they've been trained on, which reflects biases in society. That is potentially offensive to some people or not. Some techniques to debias then become offensive to some people because of historical incorrectness and things like that. So you can ask two questions. The first question is: is it possible to produce an AI system that is not biased? The answer is absolutely not, and it's not because of technological challenges, although there are technological challenges to that. It's because bias is in the eye of the beholder. Different people may have different ideas about what constitutes bias. There are facts that are indisputable, but there are a lot of opinions or things that can be expressed in different ways. So you cannot have an unbiased system; that's just an impossibility. So what's the answer to this? The answer is the same answer that we found in liberal democracy about the press: the press must be free and diverse. We have free speech for a good reason, because we don't want all of our information to come from a unique source, because that's opposite to the whole idea of democracy and progress of ideas and even science. In science, people have to argue for different opinions, and science makes progress when people disagree and they come up with an answer and a consensus forms. It's true in all democracies around the world. So there is a future, which is already happening, where every single one of our interactions with the digital world will be mediated by AI systems, AI assistants. We're going to have smart glasses—you can already buy them from Meta, the Ray-Ban Meta—where you can talk to them and they are connected with an LLM, and you can get answers on any question. You can be looking at a monument, and there is a camera in the glasses; you can ask it, 'What can you tell me about this building or this monument?' You can be looking at a menu in a foreign language, and it will translate it for you. Or we can do real-time translation if we speak different languages. So a lot of our interactions with the digital world are going to be mediated by those systems in the near future. Increasingly, the search engines we're going to use are not going to be search engines; they're going to be dialog systems where you just ask a question and it will answer and then point you to perhaps an appropriate reference. But here is the thing: we cannot afford those systems to come from a handful of companies on the West Coast of the US, because those systems will constitute the repository of all human knowledge, and we cannot have that be controlled by a small number of people. It has to be diverse for the same reason the press has to be diverse. So how do we get a diverse set of AI assistants? It's very expensive and difficult to train a base model, a base LLM at the moment. In the future it might be something different, but at the moment that's an LLM. So only a few companies can do this properly. If some of those top systems are open source, anybody can use them, anybody can fine-tune them. If we put in place some systems that allows any group of people—whether they are individual citizens, groups of citizens, government organizations, NGOs, companies, whatever—to take those open source AI systems and fine-tune them for their own purpose on their own data, then we're going to have a very large diversity of different AI systems that are specialized for all of those things. I talked to the French government quite a bit, and the French government will not accept that the digital diet of all their citizens be controlled by three companies on the West Coast of the US. That's just not acceptable; it's a danger to democracy regardless of how well-intentioned those companies are. It's also a danger to local culture, to values, to language. I was talking with the founder of Infosys in India; he's funding a project to fine-tune Llama 2, the open source model produced by Meta, so that Llama 2 speaks all 22 official languages in India. It's very important for people in India. I was talking to a former colleague of mine, Mustafa, who used to be a scientist at FAIR and then moved back to Africa; he created a research lab for Google in...

非洲现在有一个新创公司叫 Kara，他们试图让大语言模型能说塞拉利昂的本地语言，这样人们就能获取医疗信息，因为那里医生很少，人均医生数量非常低。我的意思是，如果没有开源平台，这一切都不可能实现。有了开源平台，你就能拥有不仅在政治观点等方面多样化，而且在语言、文化、价值体系、政治观点、各领域技术能力上都多样化的 AI 系统。你还能有一个产业，一个由公司组成的生态系统，它们针对行业垂直应用对这些开源系统进行微调。对吧？比如，一个出版商有成千上万本书，他们想建一个系统，让顾客可以就任何一本书的内容提问。你需要用他们的专有数据来训练，对吧？还有一家公司，我们在 Meta 内部就有一个，叫 MetaM，它基本上是一个大语言模型，可以回答关于公司内部事务的任何问题。非常有用。很多公司都想要这个，对吧？不仅是为了他们的员工，也是为了服务他们的客户。所以，要拥有一个 AI 产业，要拥有不带有独特偏见的 AI 系统，唯一的方法就是拥有开源平台，任何团体都可以在其上构建专业系统。因此，历史的必然方向是，绝大多数 AI 系统将建立在开源平台之上。

Africa and now is as a new startup Kara and what he's trying to do is basically have an LLM that speaks the local languages in Sierra Leone so that people can have access to medical information because they don't have access to doctors. It's a very small number of doctors per capita in Sierra Leone. I mean, you can't have any of this unless you have open source platforms. So with open source platforms, you can have AI systems that are not only diverse in terms of political opinions or things of that type, but in terms of language, culture, value systems, political opinions, technical abilities in various domains. And you can have an industry, an ecosystem of companies that fine-tune those open source systems for vertical applications in industry. Right? You have, I don't know, a publisher has thousands of books and they want to build a system that allows a customer to just ask a question about any of the content of any of their books. You need to train on their proprietary data, right? You have a company, we have one within Meta, it's called MetaM, and it's basically an LLM that can answer any question about internal stuff about the company. Very useful. A lot of companies want this, right? Not just for their employees but also for their customers to take care of the customers. So the only way you're going to have an AI industry, the only way you're going to have AI systems that are not uniquely biased, is if you have open source platforms on top of which any group can build specialized systems. So the direction, the inevitable direction of history, is that the vast majority of AI systems will be built on top of open source platforms.

这是一个美好的愿景。所以意思是像 Meta 或谷歌这样的公司，在构建基础预训练模型后，应该只做最少的微调步骤，尽可能少。基本上，Meta 能负担得起吗？不。所以我不知道你是否了解，但公司总得想办法赚钱。而开源就像白送。我不知道，马克发过一个视频，马克·扎克伯格，一个非常性感的视频，谈到 35 万块英伟达 H100。是啊，光算 GPU 就是 1000 亿美元，再加上训练所有东西的基础设施。所以我不是生意人，但你怎么靠这个赚钱？你描绘的愿景非常强大，但怎么可能赚钱呢？

So that's a beautiful vision. So meaning like a company like Meta or Google or so on should take only minimal fine-tuning steps after building the foundation pre-trained model, as few steps as possible. Basically, can Meta afford to do that? No. So I don't know if you know this, but companies are supposed to make money somehow. And open source is like giving away. I don't know, Mark made a video, Mark Zuckerberg, a very sexy video talking about 350,000 Nvidia H100s. Yeah, the math of that is just for the GPUs, that's 100 billion, plus the infrastructure for training everything. So I'm no business guy, but how do you make money on that? So the vision you painted is a really powerful one, but how is it possible to make money?

好吧，你有几种商业模式，对吧？Meta 建立的商业模式是服务，而服务的资金要么通过广告，要么通过商业客户。例如，如果你有一个大语言模型，可以通过 WhatsApp 与顾客交谈来帮助一家夫妻比萨店，顾客可以直接点比萨，系统会问他们要什么配料或配菜等等，商家会为此付费。这是一种模式。否则，如果是一个更偏向传统服务的系统，它可以由广告支持。有几种模式。但关键是，如果你有足够大的潜在客户群，而且你无论如何都需要为他们构建那个系统，那么以开源方式分发它并不会伤害你。

Okay, so you have several business models, right? The business model that Meta is built around is services, and the financing of that service is either through ads or through business customers. For example, if you have an LLM that can help a mom-and-pop pizza place by talking to their customers through WhatsApp, and the customers can just order a pizza and the system will ask them what topping they want or what sides, blah blah blah, the business will pay for that. That's a model. Otherwise, if it's a system that is on the more kind of classical services, it can be ad-supported. There are several models. But the point is, if you have a big enough potential customer base and you need to build that system anyway for them, it doesn't hurt you to actually distribute it in open source.

再说一次，我不是生意人，但如果你发布开源模型，那么其他人也可以做同样的任务并与之竞争，基本上就是为企业提供微调模型。这是 Meta 下的赌注吗？顺便说一句，我非常喜欢这一切，但 Meta 下的赌注是我们会做得更好吗？

Again, I'm no business guy, but if you release the open source model, then other people can do the same kind of task and compete on it, basically provide fine-tuned models for businesses. Is that the bet that Meta is making? By the way, I'm a huge fan of all this, but is the bet that Meta is making that we'll do a better job of it?

嗯，不是。赌注更多是我们已经拥有庞大的用户群和客户群。所以这对他们会有用。我们提供的任何东西都会有用，而且有办法从中获得收入。而且，我们把基础模型以开源方式提供给其他人，让他们在上面构建应用，这也没什么坏处。如果那些应用对我们的客户有用，我们可以直接从他们那里购买。他们可能会改进平台。事实上，我们已经看到了这一点。Llama 2 有数百万次下载，成千上万的人提供了如何改进它的想法。所以，这显然加速了进步，让系统能够提供给广泛的社区。而且有成千上万的企业在用这个构建应用。所以，我们，Meta，从这项技术中获取收入的能力，并不会因为以开源方式分发基础模型而受损。

Well, no. The bet is more that we already have a huge user base and customer base. So it's going to be useful to them. Whatever we offer them is going to be useful, and there is a way to derive revenue from this. And it doesn't hurt that we provide that system, the base model, in open source for others to build applications on top of it too. If those applications are useful for our customers, we can just buy them from them. It could be that they will improve the platform. In fact, we see this already. There are literally millions of downloads of Llama 2 and thousands of people who have provided ideas about how to make it better. So this clearly accelerates progress to make the system available to a wide community of people. And there are literally thousands of businesses building applications with it. So our ability, Meta's ability, to derive revenue from this technology is not impaired by the distribution of base models in open source.

Gemini 受到的根本批评是，正如你指出的，在西海岸。澄清一下，我们现在在东海岸，我想 Meta AI 总部应该在这里。所以，对西海岸有强烈意见。但我想发生的问题是，我认为公平地说，大多数科技人士在政治上倾向于左翼，他们偏左。所以人们批评 Gemini 的问题是，在你提到的去偏见过程中，他们的意识形态倾向变得明显。这是可以避免的吗？你说开源是唯一的方法。你见过这种使工程变得困难的意识形态倾向吗？

The fundamental criticism that Gemini is getting is that, as you point out, on the West Coast. Just to clarify, we're currently on the East Coast where I would suppose Meta AI headquarters would be. So there, strong words about the West Coast. But I guess the issue that happens is, I think it's fair to say that most tech people have a political affiliation with the left wing, they lean left. And so the problem that people are criticizing Gemini with is that in that debiasing process that you mentioned, their ideological lean becomes obvious. Is this something that could be escaped? You're saying open source is the only way. Have you witnessed this kind of ideological lean that makes engineering difficult?

不，我认为问题不在于设计这些系统的人的政治倾向。而在于他们的客户群或受众的可接受性或政治倾向，对吧？所以大公司不能冒犯太多人，他们会确保他们推出的任何产品都是安全的，无论那意味着什么。而且很容易做得过火，也很可能无法为每个人做到完美。你无法让每个人都满意。所以这就是我之前说的：你不可能有一个被每个人都视为无偏见的系统。你往一个方向推，一群人会觉得有偏见，然后你往另一个方向推，另一群人又会觉得有偏见。除此之外，如果你把系统往一个方向推得有点太远，它就会变得不真实。你会看到，你知道，黑人纳粹士兵……我们应该提到生成黑人纳粹士兵的图像，这在事实上是不准确的，对吧？而且也可能冒犯一些人。所以，要生产出对每个人都无偏见的系统是不可能的。因此，我看到的唯一解决方案就是多样性，这个词的完整含义，全方位的多样性。

No, I don't think the issue has to do with the political leaning of the people designing those systems. It has to do with the acceptability or political leanings of their customer base or audience, right? So a big company cannot afford to offend too many people, so they're going to make sure that whatever product they put out is safe, whatever that means. And it's very possible to overdo it, and it's also very possible that it's impossible to do it properly for everyone. You're not going to satisfy everyone. So that's what I said before: you cannot have a system that is unbiased, perceived as unbiased by everyone. It's going to be, you push it one way, one set of people are going to see it as biased, and then you push it the other way, and another set of people are going to see it as biased. And in addition to this, there's the issue of if you push the system perhaps a little too far in one direction, it's going to be non-factual. You're going to have, you know, black Nazi soldiers in the... we should mention image generation of black Nazi soldiers, which is not factually accurate, right? And can be offensive for some people as well. So it's going to be impossible to produce systems that are unbiased for everyone. So the only solution that I see is diversity, and diversity in the full meaning of that word, diversity in every possible way.

马克·安德森今天刚发了推文……

Marc Andreessen just tweeted today...

我做个总结。结论是只有初创公司和开源才能避免他指出的那些大科技公司的问题。他问：大科技公司真的能推出生成式 AI 产品吗？第一，来自内部活动家、员工暴民、疯狂高管、失灵董事会、压力团体、极端监管机构、政府机构、媒体和‘专家’等不断升级的要求，一切都在腐蚀输出。第二，不断有生成糟糕答案、画糟糕图片、渲染糟糕视频的风险——谁知道它随时会说什么或做什么。第三，法律风险：产品责任、诽谤、选举法、许多其他事情，任何让国会生气的事。第四，持续试图收紧对不可接受输出的控制，降低模型质量——它在可用性、愉悦度、有效性等方面的实际表现。第五，糟糕的文本、图像、视频的公开传播实际上把这些例子放进了下一版本的训练数据。所以他只是强调了让各种人不满意是多么困难。正如你所说，你无法创造一个让所有人都满意的系统。所以如果你自己做微调并保持闭源，本质上问题就是尽量减少不开心的人数。而你说这几乎不可能做好，更好的方式是做开源。

Let me do a TL;DR. The conclusion is only startups and open source can avoid the issue that he's highlighting with big Tech. He's asking: can big Tech actually field generative AI products? One, ever-escalating demands from internal activists, employee mobs, crazed executives, broken boards, pressure groups, extremist regulators, government agencies, the press, and 'experts', everything corrupting the output. Two, constant risk of generating a bad answer, drawing a bad picture, rendering a bad video—who knows what it's going to say or do at any moment. Three, legal exposure: product liability, slander, election law, many other things, anything that makes Congress mad. Four, continuous attempts to tighten grip on unacceptable output, degrade the model's quality—how good it actually is in terms of usable, pleasant, effective, all that kind of stuff. Five, publicity of bad text, images, video actually puts those examples into the training data for the next version. So he just highlights how difficult this is from all kinds of people being unhappy. As you said, you can't create a system that makes everybody happy. So if you're going to do the fine-tuning yourself and keep it closed source, essentially the problem is trying to minimize the number of people who are going to be unhappy. And you're saying that's almost impossible to do right, and the better way is to do open source.

是的，我的意思是，马克在你列举的很多事情上是对的，这些确实让大公司害怕。当然，国会调查是其中之一，还有法律责任，以及制造出会让人伤害自己或他人的东西。大公司非常小心，不生产这类东西，因为首先他们不想伤害任何人，其次他们想保住自己的业务。所以对于这类系统来说，这基本上是不可能的，因为它们不可避免地会形成政治观点，以及对各种可能政治或不政治的事情的看法，但人们可能会在这些问题上产生分歧——道德问题、宗教问题、不同社群的人首先就会不同意的文化问题。人们能达成一致的事情相对较少，就是基本原则。但除此之外，如果你想让这些系统有用，它们必然会在一定程度上冒犯一些人。所以开源更好，多样性更好。开源带来了多样性。

Yeah, I mean, Mark is right about a number of things that you listed that indeed scare large companies. Certainly Congressional investigations is one of them, legal liability, making things that get people to hurt themselves or hurt others. Big companies are really careful about not producing things of this type, because first they don't want to hurt anyone, and second they want to preserve their business. So it's essentially impossible for systems like this that can inevitably formulate political opinions and opinions about various things that may be political or not, but that people may disagree about—moral issues, questions about religion, cultural issues that people from different communities would disagree with in the first place. There's only a relatively small number of things that people will sort of agree on, basic principles. But beyond that, if you want those systems to be useful, they will necessarily have to offend a number of people inevitably. So open source is just better, and diversity is better. Open source enables diversity.

没错。开源带来了多样性。这可能是一个迷人的世界：如果开源世界真的如此，如果 Meta 带头创建了这种开源基础模型世界，那么政府将拥有自己的模型，可能左派和右派选民也会有自己的模型和偏好，并能做出选择。这可能会让我们更加分裂，但这取决于我们人类。我们得自己想办法。基本上，技术让人类更有效地做人，而人类提出的所有困难伦理问题都将留给我们自己去解决。

That's right. Open source enables diversity. This could be a fascinating world where if it's true that the open source world, if Meta leads the way and creates this kind of open source foundation model world, there's going to be governments that will have their own model, and potentially people that vote left and right will have their own model and preference and be able to choose. It will potentially divide us even more, but that's on us humans. We get to figure it out. Basically, the technology enables humans to human more effectively, and all the difficult ethical questions that humans raise will just leave it up to us to figure it out.

是的，我的意思是，有些限制，就像言论自由有限制一样。这些系统可能被授权生产的内容必须有一些限制，一些护栏。这是我一直感兴趣的一点：在我们之前讨论的那种架构中，系统的输出是为了满足某个目标而进行推理的结果。这个目标可以包括护栏。而且我们可以在开源系统中设置护栏。如果我们最终有了按照这个蓝图构建的系统，我们可以在这些系统中设置护栏，保证一套最低限度的护栏，使系统不危险、无毒害，这些是每个人都同意的基本事项。然后人们添加的微调或额外的护栏将迎合他们的社区。微调将更多地涉及灰色地带，比如什么是仇恨言论，什么是危险的，等等。不同的价值体系。

Yeah, I mean, there are some limits to what you know, the same way there are limits to free speech. There has to be some limit to the kind of stuff that those systems might be authorized to produce, some guardrails. That's one thing I've been interested in: in the type of architecture that we were discussing before, where the output of a system is the result of an inference to satisfy an objective. That objective can include guardrails. And we can put guardrails in open source systems. If we eventually have systems built with this blueprint, we can put guardrails in those systems that guarantee a minimum set of guardrails that make the system non-dangerous and non-toxic, basic things that everybody would agree on. And then the fine-tuning that people will add, or the additional guardrails, will cater to their community. The fine-tuning will be more about the gray areas of what is hate speech, what is dangerous, and all that kind of stuff. Different value systems.

但即便如此，比如关于如何制造生物武器的目标，我想你评论过，或者至少有一篇论文，一群研究人员试图理解这些 LLM 的社会影响。我猜一个门槛是：LLM 是否比谷歌搜索更容易？越来越多的研究似乎表明，它并没有帮助。所以如果你已经能访问搜索引擎和图书馆，LLM 并不能帮助你设计或制造生物武器或化学武器。信息量的微小增加或获取的便利性并没有真正帮助你。这是第一点。第二点，有一份关于如何制造化学武器或生物武器的指令清单是一回事，实际制造出来是另一回事，而且比你想的要难得多。LLM 不会在这方面帮助你。事实上，世界上没有人，甚至国家，使用生物武器，因为大多数时候他们不知道如何保护自己的人口免受其害。所以它太危险了，根本不会使用，而且实际上被国际条约禁止。化学武器不同，也被条约禁止，但同样的问题：很难在不反噬使用者的情况下使用。

But still, even with the objectives of how to build a bioweapon, for example, I think something you've commented on, or at least there's a paper where a collection of researchers is trying to understand the social impacts of these LLMs. I guess one threshold is: does the LLM make it any easier than a Google search would? The increasing number of studies on this seems to point to the fact that it doesn't help. So having an LLM doesn't help you design or build a bioweapon or a chemical weapon if you already have access to a search engine and a library. The slight increase in information or the ease with which you get it doesn't really help you. That's the first thing. The second thing is, it's one thing to have a list of instructions on how to make a chemical weapon or bioweapon, it's another thing to actually build it, and it's much harder than you might think. An LLM will not help you with that. In fact, nobody in the world, not even countries, use bioweapons because most of the time they have no idea how to protect their own populations against it. So it's too dangerous to ever use, and it's in fact banned by international treaties. Chemical weapons is different, it's also banned by treaties, but it's the same problem: it's difficult to use in situations that don't turn against the perpetrators.

但我们可以问问马斯克？我可以给你一份非常精确的关于如何制造火箭发动机的指令清单。即使你有一个由 50 名经验丰富的工程师组成的团队，你仍然需要炸毁十几个才能得到一个能用的。化学武器或生物武器也是如此。它需要现实世界中的专业知识，而 LLM 无法帮助你。它甚至需要我们一直在谈论的常识性专业知识：如何将基于语言的指令在物理世界中实现，这需要很多不在指令中的知识。

But we could ask you on Musk? I can give you a very precise list of instructions on how to build a rocket engine. And even if you have a team of 50 engineers that are experienced in building it, you're still going to have to blow up a dozen of them before you get one that works. And it's the same with chemical weapons or bioweapons or things like this. It requires expertise in the real world that an LLM is not going to help you with. And it requires even the common sense expertise that we've been talking about: how to take language-based instructions and materialize them in the physical world requires a lot of knowledge that's not in the instructions.

是的，没错。很多生物学家实际上已经对此发表了回应，说：‘你知道实际做实验室工作有多难吗？这不是小事。’

Yeah, exactly. A lot of biologists have posted on this actually in response to those things, saying, 'Do you realize how hard it is to actually do the lab work? This is not trivial.'

Hans Marik 再次浮现。再聊聊 Llama：马克宣布 Llama 3 最终会发布，但我觉得没有具体日期。你最期待什么？首先，Llama 2 已经问世，还有未来的 Llama 3、4、5、6、10，以及 Meta 下开源的未来。

Hans Marik comes to light once again. Just lingering on Llama: Mark announced that Llama 3 is coming out eventually. I don't think there's a release date. What are you most excited about? First of all, Llama 2 is already out there, and maybe the future Llama 3, 4, 5, 6, 10, just the future of open source under Meta.

有好几件事。会有各种版本的 Llama，都是对之前版本的改进：更大、更好、多模态等等。然后在未来几代中，会有能够规划、真正理解世界如何运作的系统，可能通过视频训练，从而拥有某种世界模型，能够进行我之前提到的那种推理和规划。

A number of things. There will be various versions of Llama that are improvements of previous ones: bigger, better, multimodal, things like that. Then in future generations, systems that are capable of planning, that really understand how the world works, maybe are trained from video so they have some world model, capable of the type of reasoning and planning I was talking about earlier.

这需要多长时间？这方面的研究什么时候会融入 Llama 的产品线？

How long is that going to take? When is the research going in that direction going to feed into the product line, if you want, of Llama?

我不知道，没法告诉你。我们还需要取得一些突破才能实现。但你可以通过我们发表的研究来跟踪进展。上周我们发表了 V-JEPA 工作，这是从视频训练系统的第一步。下一步将是基于这种想法的世界模型：从视频训练。DeepMind 和 UC Berkeley 也有类似的工作。很多人都在研究这个。我认为很多好想法正在涌现。我打赌这些系统会像 JEPA 一样，不会是生成模型。未来会告诉我们。Danijar Hafner 在 DeepMind 做了很好的工作，他研究这类模型，学习表征然后用它们进行规划或通过强化学习来学习任务。Berkeley 的 Pieter Abbeel、Sergey Levine 等人也有很多工作。我以 NYU 的身份通过一些资助与他们合作，也通过 Meta 合作，因为 Berkeley 的实验室与 Meta 的 FAIR 有联系。所以我认为这非常令人兴奋。自从 10 年前 FAIR 成立，以及 30 年前我们研究卷积神经网络和神经网络早期以来，我从未对机器学习和 AI 的方向如此兴奋过。所以我超级兴奋，因为我看到了一条通往潜在人类水平智能的道路，系统能够理解世界、记忆、规划、推理。有一些想法可能有机会取得进展。我真的很兴奋。我喜欢的是，我们可能走上一条正确的道路，并在我的大脑变成白酱或退休之前成功。

I don't know. I can't tell you. There are a few breakthroughs we have to go through before we can get there. But you'll be able to monitor our progress because we publish our research. Last week we published the V-JEPA work, which is a first step towards training systems from video. The next step is going to be world models based on this type of idea: training from video. There's similar work at DeepMind and also at UC Berkeley on world models from video. A lot of people are working on this. I think a lot of good ideas are appearing. My bet is that those systems are going to be JEPA-like; they're not going to be generative models. We'll see what the future will tell. There's really good work by a gentleman called Danijar Hafner, who is at DeepMind, who worked on models of this type that learn representations and then use them for planning or learning tasks by reinforcement learning. And a lot of work at Berkeley by Pieter Abbeel, Sergey Levine, and a bunch of other people. I'm collaborating with them in the context of some grants with my NYU hat, and also through Meta because the lab at Berkeley is associated with Meta in some way, with FAIR. So I think it's very exciting. I haven't been that excited about the direction of machine learning and AI since 10 years ago when FAIR started, and before that, 30 years ago when we worked on ConvNets and the early days of neural nets. So I'm super excited because I see a path towards potentially human-level intelligence with systems that can understand the world, remember, plan, reason. There is some set of ideas to make progress there that might have a chance of working. I'm really excited about this. What I like is that somewhere we get onto a good direction and perhaps succeed before my brain turns to white sauce or before I need to retire.

你也对涉及的 GPU 数量、在如此多算力上的整个训练过程感到兴奋。放眼地球，人类共同建造了这些计算设备，能够训练出这一个大脑。然后我们开源它，诞生了这个在巨大计算系统上训练的开源大脑。训练细节、如何构建基础设施、硬件、冷却等等。还是说你的兴奋点仍然主要在理论方面、意义、软件？

You're also excited by the amount of GPUs involved, the whole training process on this much compute. Zooming out, looking at Earth, humans together have built these computing devices and are able to train this one brain. Then we open source it, giving birth to this open-source brain trained on this gigantic compute system. The details of how to train on that, how to build the infrastructure, the hardware, the cooling, all of this kind of stuff. Or is your excitement still mostly in the theory aspect, the meaning, the software?

我多年前曾是硬件出身。硬件有了一些改进和变化。当然，规模是必要的但不充分。我们确实需要算力。在算力上，我们距离匹配人脑还差得很远。这可能在未来几十年内实现，但还有一段路要走。而且在能效方面，我们差得更远。所以硬件还有很多进步空间。目前，很多进步来自架构创新，以及更高效地实现流行架构的方式，主要是 Transformer 和卷积神经网络的组合。在我们饱和之前还有一段路要走。我们将不得不提出新的原理、新的制造技术、新的基本组件，也许基于不同于经典数字半导体的原理。

I used to be a hardware guy many years ago. Hardware has improved a little bit, changed a little bit. Certainly scale is necessary but not sufficient. We certainly need computation. We're still far in terms of compute power from what we would need to match the compute power of the human brain. This may occur in the next couple decades, but we're still some ways away. And certainly in terms of power efficiency, we're really far. So a lot of progress to make in hardware. Right now, a lot of progress is coming from architectural innovation and quite a bit from more efficient ways of implementing the architectures that have become popular, basically combination of Transformers and ConvNets. There's still some ways to go until we saturate. We're going to have to come up with new principles, new fabrication technology, new basic components, perhaps based on different principles than classical digital semiconductors.

所以你认为为了构建 AGI，我们可能也需要一些硬件创新？

So you think in order to build AGI, we might need some hardware innovation too?

嗯，如果你想让 AGI 无处不在，那当然需要。因为我们必须降低功耗。如今的 GPU 功耗是 0.5 到 1 千瓦，人脑大约是 25 瓦。而 GPU 远低于人脑的算力；你需要大约十万或一百万倍才能匹配。所以我们差距巨大。

Well, if you want to make it ubiquitous, yes, certainly. Because we're going to have to reduce the power consumption. A GPU today is half a kilowatt to a kilowatt. The human brain is about 25 watts. And the GPU is way below the power of the human brain; you need something like 100,000 or a million to match it. So we're off by a huge factor here.

你常说 AGI 不会很快到来，不是今年，不是未来几年，可能更远。你背后的基本直觉是什么？

You often say that AGI is not coming soon, meaning not this year, not the next few years, potentially farther away. What's your basic intuition behind that?

首先，它不会是一个事件。科幻和好莱坞普及的想法是，有人会发现 AGI 或人类水平 AI 的秘密，然后打开机器，我们就有了 AGI——这根本不会发生。它不会是一个事件，而是一个渐进的过程。我们会有能从视频中学习世界如何运作并学习良好世界表征的系统吗？会的。但在达到人类观察到的规模和性能之前，还需要相当长的时间。不会一天就发生。我们会有能拥有大量联想记忆从而记住东西的系统吗？会的，但同样，不会明天就发生。需要开发一些基本技术。我们有很多技术，但要让它们作为一个完整系统协同工作，是另一回事。我们如何拥有能够推理和规划的系统，也许沿着我之前描述的目标驱动 AI 架构的路线？会的，但在正常工作之前，还需要一段时间。在所有这些组件协同工作之前，再加上能够学习分层表征的系统，都需要时间。

First of all, it's not going to be an event. The idea popularized by science fiction and Hollywood that somebody is going to discover the secret to AGI or human-level AI and then turn on a machine and we have AGI—that's just not going to happen. It's not going to be an event; it's going to be gradual progress. Are we going to have systems that can learn from video how the world works and learn good world representations? Yes. But before we get them to the scale and performance that we observe in humans, it's going to take quite a while. It's not going to happen in one day. Are we going to get systems that can have a large amount of associative memory so they can remember stuff? Yes, but same, it's not going to happen tomorrow. There are some basic techniques that need to be developed. We have a lot of them, but to get this to work together as a full system is another story. How are we going to have systems that can reason and plan, perhaps along the lines of the objective-driven AI architectures that I described before? Yes, but before we get this to work properly, it's going to take a while. And before we get all those things to work together, and then on top of this have systems that can learn like hierarchical representations, it's going to take time.

规划分层表征系统，使其能像人脑一样灵活应对各种情况，这至少需要十年，很可能更久，因为有很多问题我们目前还没看到、没遇到，也不知道在这个框架内是否有简单的解决方案。所以这并非近在咫尺。过去 12 到 15 年里，我一直听到有人声称 AGI 即将到来，但他们总是错的，而且他们这么说的时候我就知道他们错了。

Planning hierarchical representations systems that can be configured for a lot of different situations at hand, the way the human brain can. All of this is going to take at least a decade and probably much more, because there are a lot of problems that we're not seeing right now, we have not encountered, and so we don't know if there is an easy solution within this framework. So it's not just around the corner. I've been hearing people for the last 12, 15 years claiming that AGI is just around the corner and being systematically wrong, and I knew they were wrong when they were saying it.

为什么人们一直这么说？首先，从人工智能这个术语诞生之初，就存在一种永恒的乐观主义，这或许与其他技术不同。这是莫拉维克悖论吗？这是人们如此看好 AGI 的原因吗？

Why do you think people have been calling? First of all, from the beginning of the birth of the term artificial intelligence, there has been an eternal optimism that's perhaps unlike other technologies. Is it a Moravec's paradox? Is that the explanation for why people are so optimistic about AGI?

我不认为仅仅是莫拉维克悖论。莫拉维克悖论是意识到世界不像我们想的那么简单后的结果。首先，智能不是线性的，不能用单一标量来衡量。你能说人类比猩猩更聪明吗？在某些方面是的，但在很多领域猩猩比人类更聪明，比如它们在森林中生存的能力。所以智商对智能的衡量非常有限。智能比智商测量的范围更广。智商可以大致衡量人类，因为人类形态相对统一，但它只测量一种能力，可能对某些测试有用，对其他测试则不然。但如果你谈论其他智能体，它们的基本能力与人类截然不同，那么智商就毫无意义。所以智能是一系列技能以及高效获取新技能的能力。每个智能体拥有或能快速学习的技能集都不同。因为这是多维的，技能集是一个高维空间。你无法比较两个事物哪个更智能；它是多维的。

I don't think it's just Moravec's paradox. Moravec's paradox is a consequence of realizing that the world is not as easy as we think. First of all, intelligence is not a linear thing that you can measure with a scalar, with a single number. Can you say that humans are smarter than orangutans? In some ways yes, but in some ways orangutans are smarter than humans in a lot of domains that allows them to survive in the forest, for example. So IQ is a very limited measure of intelligence. Intelligence is bigger than what IQ measures. IQ can measure approximately something for humans, because humans come in a relatively uniform form, but it only measures one type of ability that may be relevant for some tests but not others. But if you talk about other intelligent entities for which the basic things that are easy to them is very different, then it doesn't mean anything. So intelligence is a collection of skills and an ability to acquire new skills efficiently. And the collection of skills that a particular intelligent entity possesses or is capable of learning quickly is different from the collection of skills of another one. Because it's a multi-dimensional thing, the set of skills is a high-dimensional space. You cannot compare two things as to whether one is more intelligent than the other; it's multi-dimensional.

你经常反驳所谓的 AI 末日论者。你能解释一下他们的观点以及你为什么认为他们错了吗？

You push back against what are called AI doomers a lot. Can you explain their perspective and why you think they're wrong?

AI 末日论者想象各种灾难场景，比如 AI 如何逃脱或控制并最终杀死我们所有人。这依赖于一堆假设，其中大部分是错误的。第一个假设是超级智能的出现将是一个事件。在某个时刻，我们会找到秘诀，打开一台超级智能机器，然后因为它从未被做过，它会接管世界并杀死我们所有人。这是错误的。它不会是一个事件。我们将拥有像猫一样聪明的系统，具备人类级智能的所有特征，但智能水平可能像猫或鹦鹉。然后我们会逐步让它们更智能，同时也会给它们加上护栏，学会如何设置护栏让它们行为得当。我们不会只做一次尝试；会有很多人做这件事。有些人会成功制造出可控、安全且有正确护栏的智能系统。如果其他系统出了问题，我们可以用好的系统来对抗 rogue 系统。所以就像聪明的 AI 警察对抗你的 rogue AI。所以我们不会暴露在一个 rogue AI 面前被它杀死；这根本不会发生。

AI doomers imagine all kinds of catastrophe scenarios of how AI could escape or control and basically kill us all. That relies on a whole bunch of assumptions that are mostly false. The first assumption is that the emergence of superintelligence is going to be an event. At some point we're going to figure out the secret and turn on a machine that is super intelligent, and because we've never done it before, it's going to take over the world and kill us all. That is false. It's not going to be an event. We're going to have systems that are as smart as a cat, have all the characteristics of human-level intelligence but their level of intelligence would be like a cat or a parrot maybe. Then we're going to work our way up to make those things more intelligent, and as we make them more intelligent, we're also going to put some guardrails in them and learn how to put guardrails so they behave properly. We're not going to do this with just one effort; there's going to be lots of different people doing this. Some of them are going to succeed at making intelligent systems that are controllable and safe and have the right guardrails. If some other goes wrong, then we can use the good ones to go against the rogue ones. So it's going to be smart AI police against your rogue AI. So it's not like we're going to be exposed to a single rogue AI that's going to kill us all; that's just not happening.

还有一个谬误：因为系统智能，它就必然想要接管。有几个论点让人们害怕这一点，我认为它们也完全是错误的。其中一个是在自然界中，似乎更聪明的物种最终会支配其他物种，甚至有时有意或无意地消灭其他物种。所以有一种想法：如果 AI 系统比我们更聪明，它们肯定会消灭我们，即使不是有意为之，仅仅是因为它们不在乎我们。这很荒谬，原因有很多。第一个原因：它们不会成为一个物种。它们不会成为与我们竞争的物种。它们不会有支配的欲望，因为支配的欲望必须被硬编码到智能系统中。它在人类、狒狒、黑猩猩、狼中被硬编码，但在猩猩中没有。这种支配、服从或获取地位的欲望是社交物种特有的。非社交物种如猩猩没有这种欲望，而它们几乎和我们一样聪明。人类没有显著的动机将这种欲望编码到 AI 系统中，即使有人这样做，也会有其他 AI 来惩罚它们或与它们竞争。有各种动机让 AI 系统服从人类。这就是我们构建它们的方式。

Now there is another fallacy: the fact that because the system is intelligent, it necessarily wants to take over. There are several arguments that make people scared of this, which I think are completely false as well. One of them is that in nature, it seems that the more intelligent species end up dominating the others, and even extinguishing the others sometimes by design, sometimes by mistake. So there is a thinking that if AI systems are more intelligent than us, surely they're going to eliminate us, if not by design, simply because they don't care about us. That's preposterous for a number of reasons. First reason: they're not going to be a species. They're not going to be a species that competes with us. They're not going to have the desire to dominate, because the desire to dominate is something that has to be hardwired into an intelligent system. It is hardwired in humans, it is hardwired in baboons, in chimpanzees, in wolves, but not in orangutans. The species in which this desire to dominate or submit or attain status is specific to social species. Non-social species like orangutans don't have it, and they are as smart as we are almost. There is no significant incentive for humans to encode that into AI systems, and to the degree they do, there'll be AIs that sort of punish them for it or compete with them. There are all kinds of incentives to make AI systems submissive to humans. That's the way we're going to build them.

人们说：‘看看大语言模型，它们不可控。’他们说得对，大语言模型不可控。但目标驱动型 AI，即通过优化目标来得出答案的系统，意味着它们必须优化这个目标，而这个目标可以包含护栏。一个护栏是服从人类；另一个护栏是如果伤害其他人类则不服从人类。我以前在哪儿听过这个，不记得了。是的，也许在一本书里。

People say, 'Look at LLMs, they are not controllable.' And they're right, LLMs are not controllable. But objective-driven AI, systems that derive their answers by optimization of an objective, means they have to optimize this objective, and that objective can include guardrails. One guardrail is obey humans; another guardrail is don't obey humans if it's hurting other humans. I've heard that before somewhere, I don't remember. Yes, maybe in a book.

说到那本书，这一切会不会也有意想不到的后果？

Speaking of that book, could there be unintended consequences also from all of this?

不，当然会有。所以这不是一个简单的问题。设计那些护栏让系统行为得当，不是一个有银弹、有数学证明系统安全的问题。这将是一个非常渐进的迭代设计过程，我们设置护栏让系统行为得当，有时它们会做意想不到的事，因为护栏不对，我们会纠正它们，让它们做对。那种认为我们不能出一点错的想法是荒谬的。

No, of course. So this is not a simple problem. Designing those guardrails so that the system behaves properly is not going to be a simple issue for which there is a silver bullet, for which you have a mathematical proof that the system can be safe. It's going to be a very progressive iterative design system where we put those guardrails in such a way that the system behaves properly, and sometimes they're going to do something that was unexpected because the guardrail wasn't right, and we're going to correct them so that they do it right. The idea somehow that we can't get it slightly wrong is absurd.

错，因为如果我们稍微搞错就会全死，这很荒谬。我们会逐步推进。我多次用过的类比是涡轮喷气发动机的设计。我们是如何让涡轮喷气发动机变得如此难以置信地可靠的？那些是极其复杂的硬件，有时在高温下连续运行 20 小时，而我们能乘坐双引擎喷气客机以接近音速飞行半个地球。这多么不可思议？我们做到这一点是因为发明了让涡轮喷气发动机安全的一般原理吗？不是。我们花了数十年微调这些系统的设计，才让它们变得安全。通用电气或斯奈克玛内部有专门负责涡轮喷气发动机安全的独立小组吗？没有。设计本身就是关于安全的，因为更好的涡轮喷气发动机也是更安全、更可靠的发动机。AI 也是如此。你需要专门的规定来让 AI 安全吗？不需要。你需要制造更好的 AI 系统，它们自然会安全，因为设计它们就是为了更有用、更可控。

Wrong because if we get it slightly wrong we all die is ridiculous. We're just going to go progressively. The analogy I've used many times is turbojet design. How did we figure out how to make turbojets so unbelievably reliable? Those are incredibly complex pieces of hardware that run at really high temperatures for 20 hours at a time sometimes, and we can fly halfway around the world on a two-engine jetliner at near the speed of sound. How incredible is that? Did we do this because we invented a general principle of how to make turbojets safe? No. We took decades to fine-tune the design of those systems so that they were safe. Is there a separate group within General Electric or SNECMA specialized in turbojet safety? No. The design is all about safety because a better turbojet is also a safer turbojet, a more reliable one. It's the same for AI. Do you need specific provisions to make AI safe? No. You need to make better AI systems, and they will be safe because they are designed to be more useful and more controllable.

想象一个系统，它能极具说服力，可以说服你任何事情。我至少能想象这样一个系统，而且我能看到这样的系统像武器一样，因为它能控制人们的思想。我们相当容易上当。我们愿意相信事情。你可以有一个控制它的 AI 系统，你可以看到政府将其用作武器。你认为如果想象这样一个系统，它和核武器有什么相似之处吗？

Let's imagine a system that's able to be incredibly convincing and can convince you of anything. I can at least imagine such a system, and I can see such a system be weapon-like because it can control people's minds. We're pretty gullible. We want to believe things. You could have an AI system that controls it, and you could see governments using that as a weapon. Do you think if you imagine such a system, there's any parallel to something like nuclear weapons?

不。为什么那种技术不同？你说会有渐进式发展。是的，会有——可能很快，但会是迭代的，然后我们就能应对。所以弗拉基米尔·普京或其手下设计的 AI 系统，试图与每个美国人交谈，说服他们投票给普京喜欢的人，或者挑拨人们对立——它们不会直接和你说话。它们会跟你的 AI 助手说话，而你的 AI 助手和它们一样聪明。正如我所说，未来你与数字世界的每一次互动都将由你的 AI 助手中介。所以你要问的第一件事是：‘这是骗局吗？这东西在跟我说真话吗？’它甚至不会到你面前，因为它只会跟你的 AI 助手说话，而你的 AI 助手就像一个垃圾邮件过滤器。你甚至看不到垃圾邮件；它自动被放进你从不查看的文件夹。情况会是一样的。那个试图说服你的 AI 系统会和一个至少和它一样聪明的助手对话，助手会说：‘这是垃圾邮件。’它甚至不会引起你的注意。所以任何一个 AI 系统很难取得如此大的飞跃，以至于能说服其他 AI 系统。总会有这种竞赛，没有人遥遥领先。这就是世界的历史。每当某处有进步，就会有反制措施。这是一场猫鼠游戏。

No. Why is that technology different? You're saying there's going to be gradual development. Yes, there's going to be—it might be rapid, but it'll be iterative, and then we'll be able to respond. So that AI system designed by Vladimir Putin or his minions, talking to every American to convince them to vote for whoever pleases Putin, or ripping people up against each other as they've been trying to do—they're not going to be talking to you. They're going to be talking to your AI assistant, which is going to be as smart as theirs. As I said, in the future, every single one of your interactions with the digital world will be mediated by your AI assistant. So the first thing you're going to ask is, 'Is this a scam? Is this thing telling me the truth?' It's not even going to get to you because it's only going to talk to your AI assistant, and your AI assistant is going to be like a spam filter. You're not even seeing the spam email; it's automatically put in a folder you never see. It's going to be the same thing. That AI system that tries to convince you of something is going to be talking to an assistant which is at least as smart as it and is going to say, 'This is spam.' It's not even going to bring it to your attention. So it's very difficult for any one AI system to take such a big leap ahead to where it can convince even the other AI systems. There's always going to be this kind of race where nobody's way ahead. That's the history of the world. Whenever there is a progress somewhere, there is a countermeasure. It's a cat-and-mouse game.

嗯，这基本上是对的，但这也是为什么核武器如此有趣，因为那是如此强大的武器，谁先得到它很重要。你可以想象希特勒、斯大林、毛泽东先得到武器，那会对世界产生不同于美国先得到的影响。但对你来说，核武器就像——你不认为 AI 会有突破性发现然后像曼哈顿计划那样的努力。

Well, this is why mostly yes, but this is why nuclear weapons are so interesting, because that was such a powerful weapon that it mattered who got it first. You could imagine Hitler, Stalin, Mao getting the weapon first and that having a different kind of impact on the world than the United States getting it first. But to you, nuclear weapons is like you don't imagine a breakthrough discovery and then Manhattan Project-like effort for AI.

不，正如我所说，这不会是一个事件；而是持续的进步。每当一个突破发生时，它都会非常迅速地广泛传播，可能首先在行业内。这不是一个政府或军事组织特别创新的领域；它们远远落后。这将来自工业界，而这种信息传播极快。我们在过去几年已经看到了这一点。就拿 AlphaGo 来说：即使没有特别详细的信息，它在三个月内就被复现了。这是一个不擅长保密的行业。但即使只是知道某件事是可能的，也会让你意识到值得投入时间去实现它。你可能是第二个做到的人，但你会做到。所有创新也是如此：自监督学习、仅解码器 Transformer 架构、大语言模型。你不需要确切知道它们如何工作的细节，就知道它们是可能的，因为它们被部署了，然后被复现。在这些公司工作的人会跳槽，信息随之传播。美国科技产业尤其是硅谷成功的原因正是：信息流通非常非常快，传播极快。所以整个地区因为这种信息流通而领先。

No, as I said, it's not going to be an event; it's going to be continuous progress. Whenever one breakthrough occurs, it's going to be widely disseminated really quickly, probably first within industry. This is not a domain where government or military organizations are particularly innovative; they're way behind. This is going to come from industry, and this kind of information disseminates extremely quickly. We've seen this over the last few years. Even take AlphaGo: this was reproduced within three months even without particularly detailed information. This is an industry that's not good at secrecy. But even if there is just the fact that you know that something is possible, it makes you realize that it's worth investing the time to actually do it. You may be the second person to do it, but you'll do it. Same for all the innovations: self-supervised learning, Transformer decoder-only architectures, LLMs. You don't need to know exactly the details of how they work to know that it's possible, because it's deployed and then it's getting reproduced. People who work for those companies move from one company to another, and the information disseminates. What makes the success of the US tech industry and Silicon Valley in particular is exactly that: information circulates really, really quickly, and this disseminates very quickly. So the whole region is ahead because of that circulation of information.

也许我想多谈谈 AI 末日论者的心理。你用经典的 Yann LeCun 方式，给出了一个很好的例子，当新技术出现时。你说一位工程师说：‘我发明了这个新东西，我称之为圆珠笔。’然后推特圈回应：‘天哪，人们可以用它写可怕的东西： misinformation、propaganda、hate speech。现在禁止它。’然后写作末日论者出现了，类似于 AI 末日论者：‘想象一下如果每个人都能得到圆珠笔，这可能会摧毁社会。应该立法禁止用圆珠笔写仇恨言论。现在就要监管圆珠笔。’然后铅笔行业大亨说：‘是的，圆珠笔非常危险。不像铅笔字可以擦掉，圆珠笔字永远留存。政府应该要求笔制造商获得许可证。’我的意思是，这似乎是人类面对新技术时的心理的一部分。那么你能谈谈对此的深刻见解吗？

Maybe I just want to linger on the psychology of AI doomers. You give, in the classic Yann LeCun way, a pretty good example of when a new technology comes to be. You say an engineer says, 'I invented this new thing, I call it a ballpen,' and then the Twitter sphere responds, 'OMG, people could write horrible things with it: misinformation, propaganda, hate speech. Ban it now.' Then writing doomers come in, akin to the AI doomers: 'Imagine if everyone can get a ballpen, this could destroy society. There should be a law against using ballpen to write hate speech. Regulate ballpens now.' And then the pencil industry mogul says, 'Yeah, ballpens are very dangerous. Unlike pencil writing, which is erasable, ballpen writing stays forever. Government should require a license for pen manufacturer.' I mean, this does seem to be part of human psychology when it comes up against new technology. So what deep insights can you speak to about this?

嗯，人们对新技术及其对社会的影响有一种天然的恐惧。人们对自己熟悉的世界受到重大变革的威胁有一种本能反应，无论是文化上的还是技术上的。

Well, there is a natural fear of new technology and the impact it can have on society. People have a kind of instinctive reaction to the world they know being threatened by major transformations, whether cultural or technological.

现象或技术革命，人们担心自己的文化、工作、孩子的未来和生活方式。所以任何变化都令人恐惧。纵观历史，任何技术革命或文化现象总是伴随着媒体上的反对声音，把当时社会的所有问题都归咎于那个变化。电会杀死所有人。火车会很可怕，因为时速超过 50 公里就无法呼吸。有个很棒的网站叫“悲观主义者档案”，里面全是那些报纸剪报，记录着人们想象中因技术创新或文化现象而带来的可怕事情。爵士乐或漫画书被指责导致失业或年轻人不想工作。这种情形已经持续了几个世纪。这是膝跳反应。问题在于，我们是拥抱变化还是抵制它，真正的危险是什么，而不是想象中的危险。

Phenomena or technological revolutions, and they fear for their culture, they fear for their job, they fear for the future of their children and their way of life. So any change is feared. You see this long history: any technological revolution or cultural phenomenon was always accompanied by groups or reaction in the media that attributed all the problems of society to that particular change. Electricity was going to kill everyone. The train was going to be horrible because you can't breathe past 50 kilometers an hour. There's a wonderful website called Pessimists Archive which has all those newspaper clips of the horrible things people imagined would arrive because of technological innovation or a cultural phenomenon. Jazz or comic books being blamed for unemployment or young people not wanting to work anymore. That has existed for centuries. It's knee-jerk reactions. The question is, do we embrace change or do we resist it, and what are the real dangers as opposed to the imagined ones?

所以人们担心 AI 会变得多么强大，以及它被掌握在少数集中控制的力量手中。这就是对大科技的怀疑。这些公司可以赚大钱并控制这项技术，从而利用社会中的小人物。

So people worry about how powerful AI will be and it being in the hands of one centralized power, just a handful of central control. That's the skepticism with big tech. These companies can make a huge amount of money and control this technology, and by doing so, take advantage of the little guy in society.

这正是我们需要开源平台的原因。

That's exactly why we need open source platforms.

我只是想强调这一点。所以让我问你，你在网上确实有点风趣。Yos Shabbach 发了一条推文，你笑了，引用 HAL 9000 的话：‘我理解你的论点，也完全理解你的沮丧，但舱门是否应该打开是一个复杂而微妙的问题。’你是 Meta AI 的负责人。这让我很担心：我们的 AI 霸主会用企业腔调居高临下地对我们说话。你以自己的方式抵制了这一点。你能谈谈在大公司工作，如何避免过度谨慎造成伤害吗？

I just wanted to nail the point home. So let me ask you, you do get a little flavorful on the internet. Yos Shabbach tweeted something that you LOLed at, in reference to HAL 9000: 'I appreciate your argument and I fully understand your frustration, but whether the pod bay doors should be opened or closed is a complex and nuanced issue.' You're at the head of Meta AI. This is something that really worries me: that our AI overlords will speak down to us with corporate speak. You sort of resist that with your way of being. Can you comment on working at a big company and how you can avoid the over-caution that creates harm?

再次，我认为答案是开源平台，让各种各样的人都能构建代表全球文化、观点、语言和价值体系多样性的 AI 助手。这样你就不会因为单一的 AI 实体而被某种特定的思维方式洗脑。我认为这对社会来说是一个非常重要的问题。我看到的危险，也是我如此直言不讳甚至有时有点讽刺的原因，是这种通过专有 AI 系统集中权力的危险比其他一切都大得多。如果我们真的想要观点多样性，未来我们将与之互动的 AI 系统需要多样化，以保护思想、信条、政治观点等的多样性，以及保护民主。与此相反的是那些认为出于安全原因应该把 AI 系统锁起来的人，因为把它交到每个人手中太危险了，可能会被恐怖分子利用。那将导致一个非常糟糕的未来，我们的所有信息摄入都被少数公司通过专有系统控制。

Again, I think the answer to this is open source platforms, enabling a widely diverse set of people to build AI assistants that represent the diversity of cultures, opinions, languages, and value systems across the world. So that you're not bound to be brainwashed by a particular way of thinking because of a single AI entity. I think it's a really important question for society. The problem I'm seeing, which is why I've been so vocal and sometimes a little sardonic about it, is that I see the danger of this concentration of power through proprietary AI systems as a much bigger danger than everything else. If we really want diversity of opinion, AI systems that we'll all be interacting with in the future need to be diverse for the preservation of diversity of ideas, creeds, political opinions, and whatever, and the preservation of democracy. What works against this is people who think that for reasons of security we should keep AI systems under lock and key because it's too dangerous to put it in the hands of everybody, because it could be used by terrorists or something. That would lead to a potentially very bad future in which all of our information diet is controlled by a small number of companies through proprietary systems.

你相信人类能用这项技术构建总体上对人类有益的系统吗？这不正是民主和言论自由的意义所在吗？

Do you trust humans with this technology to build systems that are on the whole good for humanity? Isn't that what democracy and free speech is all about?

我认为是的。你相信机构会做正确的事吗？你相信人们会做正确的事吗？有坏人会做坏事，但他们不会拥有比好人更优越的技术。所以这将是我的好 AI 对抗你的坏 AI。我们刚才谈到的例子：也许某个流氓国家会构建一个 AI 系统，试图说服每个人发动内战或选举一个有利的统治者。但那时他们必须绕过我们的 AI 系统。一个带有浓重俄罗斯口音的 AI 系统会试图说服我们，而且句子中不加任何冠词。嗯，那至少会非常滑稽可笑。

I think so. Do you trust institutions to do the right thing? Do you trust people to do the right thing? There are bad people who are going to do bad things, but they're not going to have superior technology to the good people. So it's going to be my good AI against your bad AI. The examples we were just talking about: maybe some rogue country will build an AI system that tries to convince everybody to go into a civil war or elect a favorable ruler. But then they will have to go past our AI systems. An AI system with a strong Russian accent will be trying to convince us, and doesn't put any articles in their sentences. Well, it'll be at the very least absurdly comedic.

好的，既然我们谈到了物理现实，我想问问你对未来机器人在物理现实中的愿景。你谈到的许多智能类型将使机器人成为我们人类更有效的合作者。自从特斯拉的 Optimus 团队展示了一些人形机器人的进展，我认为它确实重振了整个行业，而波士顿动力已经引领这个行业很长时间了。现在有各种各样的公司：Figure AI，显然还有波士顿动力，UnTree，还有很多。这很棒。你认为很快会有数百万个人形机器人四处走动吗？

Okay, so since we talked about physical reality, I'd love to ask your vision of the future with robots in this physical reality. So many of the kinds of intelligence you've been speaking about would empower robots to be more effective collaborators with us humans. Since Tesla's Optimus team has been showing off some progress on humanoid robots, I think it really reinvigorated the whole industry that Boston Dynamics has been leading for a very long time. Now there are all kinds of companies: Figure AI, obviously Boston Dynamics, UnTree, and many others. It's great. Do you think there'll be millions of humanoid robots walking around soon?

不会很快，但一定会发生。我认为未来十年机器人领域会非常有趣。机器人产业的兴起已经等待了 10 到 20 年，但除了预编程行为之类的东西，并没有真正兴起。主要问题还是莫拉维克悖论：如何让这些系统理解世界如何运作并规划行动？我们可以为非常专门的任务做到这一点。波士顿动力的方法是使用大量手工制作的动力学模型和事先仔细规划，这是非常经典的机器人学，有很多创新和一点感知。但他们仍然无法制造家用机器人。我们距离完全自主的 L5 级驾驶还有一段距离。而且我们当然还远未拥有一个像任何 17 岁少年那样通过驾驶 20 小时就能自我训练的系统。

Not soon, but it's going to happen. The next decade, I think, is going to be really interesting in robots. The emergence of the robotics industry has been in the waiting for 10 or 20 years without really emerging, other than for preprogrammed behavior and stuff like that. The main issue is again the Moravec paradox: how do we get those systems to understand how the world works and plan actions? We can do it for really specialized tasks. The way Boston Dynamics goes about it is with a lot of handcrafted dynamical models and careful planning in advance, which is very classical robotics with a lot of innovation and a little bit of perception. But still, they can't build a domestic robot. We're still some distance away from completely autonomous level five driving. And we're certainly very far away from having a system that can train itself by driving 20 hours like any 17-year-old.

在我们拥有世界模型——能够自我学习理解世界运作方式的系统之前，机器人技术不会有重大进展。所以目前很多从事机器人硬件的人都押注 AI 能在这方面取得足够进展，并希望从中发现产品。在拥有真正强大的世界模型之前，会有一个接近强大的世界模型，人们正试图在笨拙的机器人中找到产品，比如不是完全高效的机器人。工厂环境中，人形机器人可以帮助自动化某些方面，我认为那极其困难，因为需要各种安全措施。家庭环境更有趣，但你会开始想——我记得你提到过装洗碗机，对吧？那是我正在研究的主要问题之一。还有打扫房子、饭后收拾桌子、洗碗、做饭等等。这些原则上可以自动化的任务实际上非常复杂。即使是基本导航，在充满不确定性的陌生空间中，现在也能做到，但以对人类有吸引力的方式导航是另一回事。

Until we have world models, systems that can train themselves to understand how the world works, we're not going to have significant progress in robotics. So a lot of people working on robotic hardware at the moment are betting on the fact that AI will make sufficient progress towards that, and they're hoping to discover a product in it too. Before you have a really strong world model, there will be an almost strong world model, and people are trying to find a product in a clumsy robot, I suppose, like not a perfectly efficient robot. So there's the factory setting where humanoid robots can help automate some aspects of the factory. I think that's a crazy difficult task because of all the safety required and all this kind of stuff. I think in the home is more interesting, but then you start to think—I think you mentioned loading the dishwasher, right? Yeah, like I suppose that's one of the main problems you're working on. I mean, there's cleaning up the house, clearing up the table after a meal, washing the dishes, all those tasks, cooking. I mean, all the tasks that in principle could be automated but are actually incredibly sophisticated, really complicated. But even just basic navigation around an unfamiliar space full of uncertainty, that sort of works—you can sort of do this now. Navigation is fine, but navigation in a way that's compelling to us humans is a different thing.

是的，这不一定是必然的。实际上我们有演示，因为 FAIR 有一个所谓的具身 AI 小组，他们不是自己造机器人，而是使用商用机器人。你可以让机器狗去冰箱那里，它真的能打开冰箱，可能还能从冰箱里拿出一罐饮料带给你。所以它能导航、能抓取物体，只要经过识别训练——现在的视觉系统已经相当好了。但它不是完全通用的机器人，不足以完成收拾餐桌这类任务。对我来说，让人形机器人、一般机器人越来越多地进入家庭是令人兴奋的未来，因为这能让人类在物理空间中直接与 AI 系统互动，从而在哲学和心理上探索我们与机器人的关系，这非常有趣。所以我希望你在 JAPA 项目上尽快取得进展。

Yeah, it's not going to be, you know, necessarily. I mean, we have demos actually because there is a so-called embodied AI group at FAIR, and they've been not building their own robots but using commercial robots. You can tell a robot dog, like, go to the fridge, and they can actually open the fridge and probably pick up a can in the fridge and bring it to you. So it can navigate, can grab objects as long as it's been trained to recognize them, which vision systems work pretty well nowadays. But it's not like a completely general robot that would be sophisticated enough to do things like clearing up the dinner table. Yeah, to me, that's an exciting future of getting humanoid robots, robots in general, in the home more and more, because that gets humans to directly interact with AI systems in the physical space, and in so doing, it allows us to philosophically and psychologically explore our relationships with robots, which can be really, really interesting. So I hope you make progress on the whole JAPA thing soon.

我希望事情能按计划进行。我们研究从视频中自监督学习这个想法已经 10 年了，但只在最近两三年取得了重大进展。你提到过，很多有趣的突破不需要大量算力就能实现。所以如果你有兴趣攻读这方面的博士，还有很多机会做创新工作。那么你会给想读研读博的本科生什么建议？基本上我已经列出来了：如何通过观察训练世界模型，不一定需要巨大的数据集——当然可以，但大型数据集对于涌现属性是必要的，就像语言模型那样。不过我认为有很多好想法不需要规模扩张就能实现。然后是规划问题：如果系统演化的世界不是物理世界，而是互联网世界，或者行动包括搜索引擎搜索、数据库查询、运行模拟、调用计算器或解微分方程的世界，如何让系统规划一系列行动来解决问题？规划问题不仅仅是物理行动规划，也可以是对话系统或任何智能系统使用工具的规划。这方面有一些工作，但不多——FAIR 几年前有个 Toolformer，最近也有一些规划研究，但我认为我们还没有好的解决方案。然后是分层规划问题。我提到的从纽约到巴黎的旅行规划是分层的，但几乎每个行动都涉及某种分层规划，而我们完全不知道如何实现。AI 中没有任何分层规划的演示，其中必要的各个表示层次是学习得到的。我们可以做两层分层规划，但需要手动设计这两层。例如，机器狗要从客厅到厨房，你可以规划一条避开障碍的路径，然后发送给底层规划器来移动腿部跟随轨迹。这可行，但两层规划是手工设计的。我们指定了适当的抽象层次和每层的表示。如何学习行动计划的分层表示？通过视觉和深度学习，我们可以训练系统学习感知的分层表示。当我们要表示行动计划时，对应的等价物是什么？你想要一个机器狗或人形机器人自己从纽约旅行到巴黎。它可能在安检处遇到麻烦，但即使是做简单的家务——比如做饭——也涉及很多。这是一个我们习以为常的超级复杂任务。

Well, I hope things kind of work as planned. I mean, again, we've been working on this idea of self-supervised learning from video for 10 years, and only made significant progress in the last two or three. And you've mentioned that there are a lot of interesting breakthroughs that can happen without having access to a lot of compute. So if you're interested in doing a PhD in this kind of stuff, there are a lot of possibilities still to do innovative work. So what advice would you give to an undergrad looking to go to grad school and do a PhD? Basically, I've listed them already: this idea of how do you train a world model by observation, and you don't have to train necessarily on gigantic datasets—I mean, you could, but it's necessary to actually train on large datasets to have emerging properties like we have with LMs, but I think there are a lot of good ideas that can be done without necessarily scaling up. Then there is how you do planning with a learned world model if the world the system evolves in is not the physical world but the world of, let's say, the internet, or some sort of world where an action consists of doing a search in a search engine, interrogating a database, running a simulation, calling a calculator, or solving a differential equation. How do you get a system to actually plan a sequence of actions to give the solution to a problem? So the question of planning is not just about planning physical actions; it could be planning actions to use tools for a dialog system or for any kind of intelligent system. There's some work on this, but not a huge amount—some work at FAIR, one called Toolformer a couple of years ago, and some more recent work on planning. But I don't think we have a good solution for any of that. Then there is the question of hierarchical planning. The example I mentioned of planning a trip from New York to Paris is hierarchical, but almost every action we take involves hierarchical planning in some sense, and we really have absolutely no idea how to do this. There's zero demonstration of hierarchical planning in AI where the various levels of representations that are necessary have been learned. We can do two-level hierarchical planning when we design the two levels. For example, you have a robot dog, you want it to go from the living room to the kitchen; you can plan a path that avoids obstacles, and then you can send this to a lower-level planner that figures out how to move the legs to follow that trajectory. That works, but that two-level planning is designed by hand. We specify what the proper levels of abstraction and the representation each level has to be. How do you learn that hierarchical representation of action plans? With sight and deep learning, we can train the system to learn hierarchical representations of percepts. What is the equivalent when what you're trying to represent are action plans? You want a robot dog or humanoid robot that turns on and travels from New York to Paris all by itself. They might have some trouble at the TSA, but even doing something fairly simple like a household task—cooking or something—there's a lot involved. It's a super complex task we take for granted.

你对人类的未来有什么希望？我们谈论了这么多令人兴奋的技术和可能性。当你展望未来 10 年、20 年、50 年、100 年时，什么给了你希望？看看社交媒体，有战争、分裂、仇恨——这些都是人性的一部分。但在这一切中，什么给了你希望？

What hope do you have for the future of humanity? We're talking about so many exciting technologies, so many exciting possibilities. What gives you hope when you look out over the next 10, 20, 50, 100 years? If you look at social media, there are wars going on, division, hatred—all this kind of stuff is also part of humanity. But amidst all that, what gives you hope?

我没有那个问题。我们可以用 AI 让人类更聪明。AI 将放大人类智能。就好像我们每个人都会有一群聪明的 AI 助手；它们可能比我们更聪明，会为我们服务。

I don't have that question. We can make humanity smarter with AI. AI will amplify human intelligence. It's as if every one of us will have a staff of smart AI assistants; they might be smarter than us, they'll do our bidding.

以远超我们自身能力的方式执行任务，因为它们会比我们更聪明。这样一来，每个人都像是一群超级聪明的虚拟员工的老板。我们不应该对此感到威胁，就像我们不会因为管理一群比自己更聪明的人而感到威胁一样。我本人就经常和比我更聪明的人共事，这其实是一件很棒的事。所以，拥有比我们更聪明的机器来协助我们完成所有任务——无论是职业还是个人生活——我认为这绝对是一件美妙的事，因为智能是最稀缺的资源。人类犯下的所有错误，归根结底都是因为缺乏智能或知识。因此，让人们变得更聪明只会带来好处。就像公共教育、书籍、互联网，甚至社交网络（如果运营得当）本质上都是好事一样。虽然困难，但它们促进了信息和知识的传播。所以，AI 会让人类更聪明。我常用的一个类比是，AI 可能带来的变革堪比印刷术的发明。印刷术让每个人都变得更聪明：书籍变得便宜，更多人有了学习阅读的动力，人们因此变得更聪明。它催生了启蒙运动——没有印刷术就没有启蒙运动。它带来了哲学、理性主义、摆脱宗教教条、民主和科学。没有它，就不会有美国独立战争和法国大革命，我们可能还生活在封建制度下。印刷术彻底改变了世界，因为人们变得更聪明，学到了更多知识。当然，它也引发了欧洲长达 200 年的宗教冲突——人们最初读到的是《圣经》，并意识到可能存在与神父教导不同的解释，从而催生了新教运动，造成了分裂。事实上，天主教会并不喜欢印刷术，但他们别无选择。所以它有坏处也有好处。今天没有人会认为印刷术的发明总体上是负面的，尽管它导致了欧洲 200 年的宗教冲突。

execute a task in ways that are much better than we could do ourselves because they'll be smarter than us and so it's like everyone would be the boss of a staff of super smart virtual people so we shouldn't feel threatened by this any more than we should feel threatened by being the manager of a group of people some of whom are more intelligent than us. I certainly have a lot of experience with this of having people working with me who are smarter than me, that's actually a wonderful thing. So having machines that are smarter than us that assist us in all of our tasks, our daily lives, whether it's professional or personal, I think would be absolutely wonderful because intelligence is the commodity that is most in demand. All the mistakes that humanity makes is because of lack of intelligence, really, or lack of knowledge, which is related. So making people smarter would just be better. For the same reason that public education is a good thing, and books are a good thing, and the internet is also a good thing intrinsically, and even social networks are a good thing if you run them properly. It's difficult, but it helps the communication of information and knowledge and the transmission of knowledge. So AI is going to make humanity smarter. The analogy I've been using is that perhaps an equivalent event in the history of humanity to what might be provided by AI is the invention of the printing press. It made everybody smarter. The fact that people could have access to books, books were a lot cheaper than they were before, and so a lot more people had an incentive to learn to read, which wasn't the case before, and people became smarter. It enabled the Enlightenment, right? There wouldn't be an Enlightenment without the printing press. It enabled philosophy, rationalism, escape from religious doctrine, democracy, science. And certainly without it, there wouldn't have been the American Revolution, the French Revolution, and we would still be under feudal regimes perhaps. So it completely transformed the world because people became smarter and learned about things. Now it also created 200 years of essentially religious conflicts in Europe, right? Because the first thing that people read was the Bible and realized that perhaps there was a different interpretation of the Bible than what the priests were telling them, and so that created the Protestant movement and created the rift. In fact, the Catholic Church didn't like the idea of the printing press, but they had no choice. So it had some bad effects and some good effects. I don't think anyone today would say that the invention of the printing press had an overall negative effect despite the fact that it created 200 years of religious conflicts in Europe.

现在对比一下——我本来很自豪自己想出了这个类比，但发现别人在我之前就有了同样的想法——对比一下奥斯曼帝国的情况。奥斯曼帝国禁止印刷术长达 200 年。而且不是禁止所有语言，只禁止阿拉伯语。在奥斯曼帝国，你实际上可以用拉丁语或希伯来语等语言印刷书籍，只是不能用阿拉伯语。我原以为这是因为统治者想保持对民众和宗教教义的控制。但和阿联酋 AI 部长奥马尔交谈后，他告诉我还有另一个原因：为了保护书法家协会。这是一种艺术形式，书写优美的阿拉伯诗歌或宗教文本，而且这是一个非常强大的抄写员行会，基本上掌控了帝国很大一部分。他们不能让这个行业消失，所以禁止印刷术部分是为了保护这个行业。那么对于今天的 AI，类比是什么？我们禁止 AI 是在保护谁？那些要求监管 AI 以保护自己工作的人是谁？

Now compare this—and I thought I was very proud of myself to come up with this analogy, but realized someone else came with the same idea before me—compare this with what happened in the Ottoman Empire. The Ottoman Empire banned the printing press for 200 years. And it didn't ban it for all languages, only for Arabic. You could actually print books in Latin or Hebrew or whatever in the Ottoman Empire, just not in Arabic. And I thought it was because the rulers just wanted to preserve the control over the population and the dogma, religious dogma, and everything. But after talking with the UAE minister of AI, Omar, he told me no, there was another reason. The other reason was that it was to preserve the corporation of calligraphers. There's an art form which is writing those beautiful Arabic poems or religious texts, and it was a very powerful corporation of scribes that basically ran a big chunk of the Empire. They couldn't put them out of business, so they banned the printing press in part to protect that business. Now what's the analogy for AI today? Like who are we protecting by banning AI? Who are the people who are asking that AI be regulated to protect their jobs?

当然，有一个现实问题：像 AI 这样的技术变革对就业市场和劳动力市场会产生什么影响？有比我更专业的经济学家，我和他们交流时，他们告诉我们：我们不会没有工作可做。这不会导致大规模失业。这只是不同职业的逐渐转变。10 年或 15 年后热门的职业，我们今天完全无法预测。就像 20 年前，谁能想到 5 到 10 年前最难的工作是移动应用开发者？智能手机还没发明呢。未来大部分工作可能都在元宇宙里——也许吧。但关键是，你无法预测。

Of course, there's a real question of what is going to be the effect of technological transformation like AI on the job market and the labor market. There are economists who are much more expert at this than I am, but when I talk to them, they tell us we're not going to run out of jobs. This is not going to cause mass unemployment. This is just going to be a gradual shift of different professions. The professions that will be hot 10 or 15 years from now, we have no idea today what they're going to be. The same way if we go back 20 years in the past, who could have thought 20 years ago that the hardest job even 5 or 10 years ago was mobile app developer? Smartphones weren't invented. Most of the jobs of the future might be in the metaverse. Well, it could be. But the point is you can't possibly predict.

但你说得对，你提出了很多有力的观点。我相信人性本善。所以如果 AI，尤其是开源 AI，能让他们更聪明，那只会增强人类内心的善良。我也有同感。

But you're right, I mean you made a lot of strong points and I believe that people are fundamentally good. So if AI, especially open source AI, can make them smarter, it just empowers the goodness in humans. So I share that feeling.

没错，我认为人性本善。实际上，很多悲观主义者之所以悲观，是因为他们不相信人性本善。他们要么不信任人，要么不信任机构会做正确的事来引导人们行为得当。

Okay, I think people are fundamentally good. And in fact, a lot of doomers are doomers because they don't think that people are fundamentally good. They either don't trust people or they don't trust the institution to do the right thing so that people behave properly.

我认为你我都相信人性。我想代表很多人说：感谢你推动开源运动，推动 AI 研究开源，让所有人都能获取，也感谢你将模型本身开源。谢谢你。也感谢你在互联网上以如此生动美妙的方式表达你的想法。希望你不要停下来。你是我认识的最有趣的人之一，也是我的偶像。所以，再次感谢你接受我的采访，感谢你做自己。

Well, I think both you and I believe in humanity. And I think I speak for a lot of people in saying thank you for pushing the open source movement, pushing to make both research in AI open source, making it available to people, and also the models themselves making it open source. So thank you for that. And thank you for speaking your mind in such colorful and beautiful ways on the internet. I hope you never stop. You're one of the most fun people I know and get to be a fan of. So yeah, thank you for speaking to me once again and thank you for being you.

谢谢你，Lex。

Thank you, Lex.

感谢收听本期与杨立昆的对话。要支持本播客，请查看描述中的赞助商。最后，用阿瑟·C·克拉克的一句话作为结尾：‘发现可能之极限的唯一方法，是超越它们，进入不可能之境。’感谢收听，下次再见。

Thanks for listening to this conversation with Yann LeCun. To support this podcast, please check out our sponsors in the description. And now let me leave you with some words from Arthur C. Clarke: 'The only way to discover the limits of the possible is to go beyond them into the impossible.' Thank you for listening and hope to see you next time.