Ilya Sutskever 揭示 AI 理解世界的惊人真相

这段与 Ilya 的对话录制于他仍担任 OpenAI 首席科学家期间，那时董事会风波尚未发生——那场风波曾短暂罢免 Sam Altman，并最终导致 Ilya 于 2024 年离开。此后，他创立了 Safe Superintelligence Inc.，一家以“安全必须成为先进 AI 的组织原则”为理念的新实验室。一个使命，没有产品，估值此后攀升至 320 亿美元。研究前沿的进展同样迅速。Anthropic 的 Claude Mythos 在计算机安全方面能力惊人，因此被限制在少数经过审核的合作伙伴中，未向公众发布。更广泛地看，进展越来越垂直化。网络安全、法律、健康、媒体等领域的专用领域模型增长快于通用聊天机器人，且常在其主场击败更大的基础模型。在此背景下，以下是我们的对话。

This conversation with Ilya was recorded while he was still chief scientist at OpenAI before the boardroom drama that briefly ousted Sam Altman and ultimately led to Ilya's departure in 2024. Since then, he's launched Safe Superintelligence Inc., a new lab built around the idea that safety has to be the organizing principle of advanced AI. One mission, no product, and a valuation that's since climbed to $32 billion. The research frontier has moved just as fast. Anthropic's Claude Mythos proved so capable at computer security that it's been kept out of general release and restricted to a small group of vetted partners. More broadly, progress has gone increasingly vertical. Specialized domain-specific models in areas like cyber security, law, health, and media are growing faster than general chatbots and often beat larger foundation models on their home turf. With all that as backdrop, here's our conversation.

我出生在俄罗斯，在以色列长大，后来在青少年时期，全家移民到加拿大。我父母说我从小就对 AI 感兴趣。我也深受意识问题的驱动。它让我很不安，我好奇于能帮助我更好理解它的东西，而 AI 似乎是一个很好的角度。所以我认为这些是让我起步的一些方式。实际上，我 17 岁时就开始与 Jeff Hinton 合作，因为我们搬到了加拿大，我立刻就能加入多伦多大学，我非常想研究机器学习，因为那似乎是人工智能最重要的方面，而当时完全不可及。给点背景，那是 2003 年。今天我们认为计算机能学习是理所当然的。但在 2003 年，我们认为计算机不能学习是理所当然的。

I was born in Russia. I grew up in Israel and then as a teenager, my family immigrated to Canada. My parents say I was interested in AI from a pretty early age. I also was very motivated by consciousness. I was very disturbed by it and I was curious about things that could help me understand it better and AI seemed like a very good angle there. So I think these were some of the ways that got me started and I actually started working with Jeff Hinton very early when I was 17 because we moved to Canada and I immediately was able to join the University of Toronto and I really wanted to do machine learning because that seemed like the most important aspect of artificial intelligence that at the time was completely unaccessible. Like to give some context, the year was 2003. Today we take it for granted that computers can learn. But in 2003, we took it for granted that computers can't learn.

嗯。

Mhm.

当时 AI 最大的成就是深蓝，那个国际象棋引擎。

The biggest achievement of AI back then was Deep Blue, the chess playing engine.

是的。

Yeah.

但那里就像你有一个游戏，你有一个树搜索，你有一种简单的方法来判断一个位置是否比另一个更好。它真的感觉不可能适用于现实世界，因为没有学习，而学习是一个巨大的谜团。所以我真的对学习非常感兴趣。非常幸运的是，Jeff Hinton 是我所在大学的教授。所以我能够找到他，我们几乎立刻就开始合作了。

But there it was like you have this game and you have this tree search and you have this simple way of determining if one position is better than another. And it really did not feel like that could possibly be applicable to the real world because there is no learning and learning was this big mystery. And so I was really really interested in learning. And to my great luck, Jeff Hinton was a professor in the university I was in. And so I was able to find him and we began working together almost right away.

世界正变得越来越复杂。每当你的团队部署新的云工作负载或 AI 智能体时，另一个身份就会获得对关键系统的永久访问权限。你知道吗？去年十分之九的组织遭受了身份泄露，因为攻击者使用有效凭证登录，而不是破解入侵。非人类身份数量以 109 比 1 超过人类身份，在云环境中造成巨大的未管理攻击面。传统工具是为管理人类员工而构建的，这使得现代机器和 AI 访问在很大程度上未受管理。这就是 Palo Alto Networks 的 Adira 的用武之地。人类、机器、AI，一个身份平台服务所有。96% 的人类用户以他们并不真正需要的过度权限运行。Adira 用动态访问取代永久权限，让你可以锁定每个身份而不拖慢业务。Adira 按需配置访问权限，并在任务完成后立即撤销，从而消除高风险的永久权限。通过 Adira，你获得统一控制，用一个控制平面取代孤立的 IAM、PAM 和治理工具，覆盖人类、机器和 AI 身份。用 Palo Alto Networks 的 Adira 保护你组织中的每个身份。了解更多请访问 paloaltonetworks.com。

The world is getting increasingly complicated. Every time your team deploys a new cloud workload or AI agent, another identity gets permanent access to your critical systems. Did you know nine out of 10 organizations suffered an identity breach last year because adversaries are logging in with valid credentials, not breaking in. Non-human identities outnumber human identities 109 to one, creating a massive unmanaged attack surface in cloud environments. Legacy tools were built to manage human employees, which leaves modern machine and AI access largely unmanaged. That's where Adira by Palo Alto Networks comes in. Human, machine, AI, one identity platform for all. 96% of human users operate with excessive permissions they don't actually need. Adira replaces permanent permissions with dynamic access so you can lock down every identity without slowing down your business. Adira provisions access on demand and revokes it immediately when the job is done which removes high-risk permanent permissions. With Adira, you get unified control by replacing siloed IAM, PAM, and governance tools with a single control plane across human, machine, and AI identities. Secure every identity at your organization with Adira by Palo Alto Networks. Learn more at paloaltonetworks.com.

你的冲动是像 Jeff 那样理解大脑如何工作，还是仅仅对机器学习的想法感兴趣？

Was your impulse as it was for Jeff to understand how the brain worked or was it more that you were simply interested in the idea of machines learning?

AI 如此宏大，所以动机也一样多。比如，这很有趣，但智能究竟如何工作？就像现在我们有相当多的想法，认为它是一个大型神经网络，我们某种程度上知道它如何工作，但那时虽然神经网络已经存在，没人知道网络有什么用处。所以智能究竟如何工作？我们如何让计算机哪怕稍微智能一点？我有一个非常明确的意图，要为 AI 做出非常小但真实的贡献，因为有很多对 AI 的贡献并不真实，我能出于各种原因判断它们不真实，不会有什么结果，我只是觉得什么都不行。AI 是一个无望的领域。所以动机是：我能否理解智能如何工作，并为此做出贡献。所以那是我最初的早期动机。

AI is so big and so the motivations were just as many like it is interesting but how does intelligence work at all like right now we have quite a bit of an idea that it's a big neural net and we know how it works to some degree but back then although the neural nets were around no one knew that nets are good for anything. So how does intelligence work at all? How can we make computers be even slightly intelligent? And I had a very explicit intention to make a very small but real contribution to AI because there were lots of contributions to AI which weren't real which were that I could tell for various reasons that they weren't real that nothing would come out of it and I just thought nothing works at all. AI is a hopeless field. So the motivation was could I understand how intelligence work and also make a contribution towards it. So that was my initial early motivation.

所以那是 2003 年，几乎正好 20 年前。然后 Alex 和我跟 Jeff 谈过，他说正是你对卷积神经网络突破的兴奋让你申请了 ImageNet 竞赛，而 Alex 有训练网络的编码技能。你能稍微谈谈那个吗？我不想陷入历史细节，但这很迷人。

So that's 2003 almost exactly 20 years ago. And then Alex and I've spoken to Jeff and he said that it was really your excitement about the breakthroughs in convolutional neural networks that led you to apply for the ImageNet competition and that Alex had the coding skills to train the network. Can you talk just a little bit about that? I don't want to get bogged down in history, but it's fascinating.

简而言之，我意识到如果你训练一个大型神经网络——抱歉，大型且深度，因为当时深度部分还很新。如果你在一个足够大的数据集上训练一个大型且深度的神经网络，该数据集指定了人们做的某些复杂任务，比如视觉，但也包括其他任务，你只需训练那个神经网络，那么你必然会成功。其逻辑非常不可简化：我们知道人脑能解决这些任务，并能快速解决，而人脑只是一个带有慢神经元的神经网络。所以我们知道某个神经网络可以做得很好。那么我们只需要取一个更小但相关的神经网络，在数据上训练它，计算机内最好的神经网络将与我们拥有的执行此任务的神经网络相关。所以这是一个论点：大型且深度的神经网络可以解决任务，而且我们有工具来训练它。那是 Jeff 实验室所做技术工作的结果。所以你把两者结合起来。我们可以训练那些神经网络。它需要足够大，这样如果你训练它，它就会工作良好，你需要能指定解决方案的数据。而有了 ImageNet，所有要素都齐了。Alex 有这些非常快的卷积核。ImageNet 有足够大的数据，有一个真正的机会做一些完全前所未有的事情，而且完全成功了。

So, in a nutshell, I had the realization that if you train a large neural network on a large sorry large and deep because back then the deep part was still new. If you train a large and a deep neural network on a big enough data set that specifies some complicated task that people do such as vision but also others and you just train that neural network then you will succeed necessarily. And the logic for it was very irreducible where we know that the human brain can solve these tasks and can solves them quickly and the human brain is just a neural network with slow neurons. So we know that some neural network can do it really well. So then we just need to take a smaller but related neural network and just train it on data and the best neural network inside the computer will be related to the neural network that we have that performs this task. So it was an argument that the neural network the large and deep neural network can solve the task and furthermore we have the tools to train it. That was the result of the technical work that was done in Jeff's lab. So you combine the two. We can train those neural networks. It needs to be big enough so that if you trained it, it would work well and you need the data which can specify the solution. And with ImageNet all the ingredients were there. Alex had these very fast convolutional kernels. ImageNet had large enough data and there was a real opportunity to do something totally unprecedented and it totally worked out.

是的，那是监督学习和卷积神经网络。2017 年，《Attention Is All You Need》论文发表，引入了自注意力和 Transformer。

Yeah, that was supervised learning and convolutional neural nets. In 2017, the Attention Is All You Need paper came out introducing self-attention and transformers.

GPT 项目是在什么时候启动的？当时对 Transformer 和自监督学习有什么直觉吗？能谈谈吗？

At what point did the GPT project start? Was there some intuition about transformers and self-supervised learning? Can you talk about that?

背景是这样的：在 OpenAI 的最早期，我们就在探索“预测下一个东西就是你所需要的一切”这个想法。当时我们用受限得多的神经网络来探索。但希望在于，如果你有一个神经网络能预测下一个词、下一个像素，那本质上就是压缩。预测就是压缩。而预测下一个词并不是——让我想想怎么解释最好，因为当时有很多事情在同时进行，它们都相关。也许我换个角度说。我们确实很感兴趣的是，预测下一个词能走多远，它是否能解决无监督学习。所以在 GPT 之前，无监督学习被认为是机器学习的圣杯。现在它已经被完全解决了，甚至没人再谈论它，但它曾经是圣杯，非常神秘。所以我们一直在探索这个想法。我当时非常兴奋，认为只要预测下一个词足够好，就能实现无监督学习。如果它能学到数据集的一切，那就太好了，但我们的神经网络当时无法胜任。我们用的是循环神经网络。当 Transformer 出现时， literally 论文一出来， literally 第二天，我就清楚地意识到，Transformer 解决了循环神经网络在学习长期依赖上的局限。这是个技术问题，但我们立刻就转向了 Transformer。于是下一个 GPT 项目继续推进，有了 Transformer 它开始工作得更好，你把它做大，然后我们意识到需要不断做大，我们确实这么做了，最终导致了 GPT-3 和今天的局面。

So for context, at OpenAI from the earliest days, we were exploring the idea that predicting the next thing is all you need. We were exploring it with the much more limited neural networks of the time. But the hope was that if you have a neural network that can predict the next word, the next pixel, really it's about compression. Prediction is compression. And predicting the next word is not—let's see, let me think about the best way to explain it because there were many things going on. They were all related. Maybe I'll take a different direction. We were indeed interested in trying to understand how far predicting the next word is going to go and whether it will solve unsupervised learning. So back before the GPTs, unsupervised learning was considered to be the holy grail of machine learning. Now it's just been fully solved and no one even talks about it, but it was a holy grail. It was very mysterious. And so we were exploring the idea. I was really excited about it—that predicting the next word well enough is going to give you unsupervised learning. If it learns everything about the data set, that's going to be great, but our neural networks were not up for the task. We were using recurrent neural networks. When the transformer came out, it was literally as soon as the paper came out, literally the next day, it was clear to me, to us, that transformers address the limitations of recurrent neural networks of learning long-term dependencies. It's a technical thing, but it was like we switched to transformers right away. And so the very next GPT effort continued then, and with the transformer it started to work better, and you make it bigger, and then we realized we need to keep making it bigger, and we did, and that's what led to eventually GPT-3 and essentially where we are today.

是的。我其实想问问——我有点沉浸在这段历史里了，但我非常感兴趣。我想谈谈大语言模型或大模型普遍存在的问题或缺点。但 Rich Sutton 一直在写关于 Scaling 的文章，说我们只需要做这个，不需要新算法，只需要扩大规模。他对你有影响吗，还是那是平行的思路？

Yeah. And I just wanted to ask—actually I'm getting caught up in this history but I'm so interested in it. I want to get to the problems or the shortcomings of large language models or large models generally. But Rich Sutton had been writing about scaling and how that's all we need to do. We don't need new algorithms. We just need to scale. Did he have an influence on you or was that a parallel track of thinking?

不，我会说，当他发表那篇文章时，我们很高兴看到一些外部人士有类似的思路，我们认为它表达得非常雄辩。但我实际上认为，我所阐述的“苦涩的教训”夸大了它的论点，或者至少我认为人们从中得出的结论夸大了它的论点。人们得出的结论是：你做什么不重要，只要扩大规模。但这并不完全正确。你必须扩大某个具体的东西。你必须有一个能够从规模中受益的东西。深度学习的伟大突破在于，它为我们提供了第一种有效利用规模并从中获得回报的方式。在那之前，人们会用大型计算机集群做什么？我想他们会做天气模拟或物理模拟之类的，但也就这样了。也许拍电影，但没有人真正需要计算机集群，因为你用它做什么呢？深度神经网络，当你把它们做大并用更多数据训练时，效果更好，这为我们提供了第一个值得扩展的东西。但也许有一天我们会发现，我们所扩展的东西上有一个小转折，会使得扩展效果更好。现在这个转折有多大——当然事后诸葛亮你会说这算什么呢，这么简单的改变——但我认为正确的说法是，你扩展什么很重要。现在我们只是找到了一个可以扩展的东西，它能给我们带来回报。

No, I would say that when he posted his article, we were very pleased to see some external people thinking in similar lines and we thought it was very eloquently articulated. But I actually think that the bitter lesson as I articulated overstates its case, or at least I think the takeaway that people have taken from it overstates its case. The takeaway that people have is: doesn't matter what you do, just scale. But that's not exactly true. You got to scale something specific. You got to have something that will be able to benefit from the scale. The great breakthrough of deep learning is that it provides us with the first ever way of productively using scale and getting something out of it in return. Like before that, what would people use large computer clusters for? I guess they would do it for weather simulations or physics simulations or something, but that's about it. Maybe movie making, but no one had any real need for computer clusters because what do you do with them? The fact that deep neural networks, when you make them larger and you train them on more data, work better provided us with the first thing that is interesting to scale. But perhaps one day we will discover that there is some little twist on the thing that we scale that's going to be even better to scale. Now how big of a twist—and then of course with the benefit of hindsight you will say does it even count, it's such a simple change—but I think the true statement is that it matters what you scale. Right now we just found a thing to scale that gives us something in return.

现有大语言模型的局限性在于，它们的知识包含在训练所用的语言中，而大多数人类知识——我想大家都同意——是非语言的。我不确定诺姆·乔姆斯基是否同意，但据我理解，大语言模型存在一个问题：它们的目标是满足提示的统计一致性。它们对语言所关联的现实没有根本性的理解。我问 ChatGPT 关于我自己的信息。它认出我是记者，曾在这些报纸工作过，但它没完没了地讲我从未获得过的奖项。读起来都很美，但没有一点与底层现实相连。你们未来的研究中有没有在解决这个问题？

The limitation of large language models as they exist is their knowledge is contained in the language that they're trained on, and most human knowledge, I think everyone agrees, is non-linguistic. I'm not sure Noam Chomsky agrees, but there's a problem in large language models as I understand it: their objective is to satisfy the statistical consistency of the prompt. They don't have an underlying understanding of reality that language relates to. I asked ChatGPT about myself. It recognized that I'm a journalist, that I've worked at these various newspapers, but it went on and on about awards that I've never won. And it all read beautifully, but none of it connected to the underlying reality. Is there something that is being done to address that in your research going forward?

好的。在回答你直接问的问题之前，我想先评论一下问题中较早的部分。

Yeah. So before I comment on the immediate question that you asked, I want to comment about some of the earlier parts of the question.

当然。

Sure.

我认为，很难谈论甚至像语言模型这样的东西的局限——因为两年前人们自信地谈论它们的局限，而那些局限完全不同。对吧？所以记住这个背景很重要。我们有多大信心认为今天看到的这些局限两年后仍然存在？我没那么有信心。关于问题的一部分，我还想评论一点：这些模型只是学到了统计规律，因此它们并不真正了解世界的本质。我的观点与此不同。换句话说，我认为学习统计规律远比表面看起来重要得多。我们最初不这么认为，是因为我们还没有——至少大多数人，那些没有真正花大量时间研究神经网络的人，神经网络在某种层面上是统计的——比如什么是统计模型？你只是拟合一些参数。到底发生了什么？但我认为对之前说的“预测即压缩”有更好的解释。预测也是一种统计现象，但要预测，你最终需要理解产生数据的真实底层过程。要很好地预测数据、很好地压缩它，你需要越来越多地理解产生数据的世界。随着我们的生成模型变得极其出色，我断言，它们将拥有令人震惊的理解程度——对世界及其许多微妙之处的令人震惊的理解程度。但不仅仅是世界，而是通过文本透镜看到的世界。它试图通过世界在文本空间中的投影来学习更多关于世界的东西，这些文本是人类在互联网上表达的。但文本已经表达了世界。我举一个最近的例子，我认为非常说明问题且引人入胜。

I think that it is very hard to talk about the limits—or limitations—of even something like a language model because two years ago people confidently spoke about their limitations and they were entirely different. Right? So it's important to keep this context in mind. How confident are we that these limitations that we'll see today will still be with us two years from now? I am not that confident. There is another comment I want to make about one part of the question, which is that these models just learned statistical regularities and therefore they don't really know what the nature of the world is. And I have a view that differs from this. In other words, I think that learning the statistical regularities is a far bigger deal than meets the eye. The reason we don't initially think so is because we haven't—at least most people, those who haven't really spent a lot of time with neural networks, which are on some level statistical—like what's a statistical model? You just fit some parameters. Like what is really happening? But I think there's a better interpretation to the earlier point of prediction as compression. Prediction is also a statistical phenomenon, yet to predict, you eventually need to understand the true underlying process that produced the data. To predict the data well, to compress it well, you need to understand more and more about the world that produced the data. As our generative models become extraordinarily good, they will have, I claim, a shocking degree of understanding—a shocking degree of understanding of the world and many of its subtleties. But it's not just the world. It is the world as seen through the lens of text. It tries to learn more and more about the world through a projection of the world on the space of text as expressed by human beings on the internet. But still this text already expresses the world. And I'll give you an example, a recent example which I think is really telling and fascinating.

我们都听说过 Sydney 是 Bing 的另一个自我，我见过一个非常有趣的互动，当用户告诉 Sydney 它认为谷歌是比 Bing 更好的搜索引擎时，Sydney 变得好斗和具有攻击性。那么，我们该如何思考这个现象？用什么语言来描述？这意味着什么？

So, we've all heard of Sydney being Bing's alter ego and I've seen this really interesting interaction with Sydney where Sydney became combative and aggressive when the user told it that it thinks that Google is a better search engine than Bing. Now, how can we like what is a good way to think about this phenomena? What's a good language? What does it mean?

你可以说它只是在预测人们会怎么做，而人们会这样做，这是真的，但也许我们现在已经到了一个点，心理学的语言开始适合用来理解这些神经网络的行为。

You can say well like it's just predicting what people would do and people would do this which is true but maybe we're now reaching a point where the language of psychology is starting to be appropriate to understand the behavior of these neural networks.

现在让我们谈谈局限性。确实，这些神经网络确实有产生幻觉的倾向。但那是因为语言模型非常擅长学习世界，但在产生良好输出方面稍差一些。有各种技术原因，如果你觉得有用我可以详细说明，但现在这一刻我先跳过。语言模型更擅长学习世界，学习关于想法、概念、人和存在的过程的惊人表示，这是有技术原因的。但它的输出并不像人们希望的那么好，或者说不如它们所能达到的那么好。这就是为什么例如像 ChatGPT 这样的系统，它是一个语言模型，但有一个额外的强化学习训练过程。我们称之为基于人类反馈的强化学习（RLHF）。但关于这个过程要理解的是：在预训练过程中，当你只是训练一个语言模型时，你想学习关于世界的一切。然后是基于人类反馈的强化学习。现在我们关心输出。我们说任何时候输出不合适，不要再这样做。每次输出不合理，不要再这样做。它很快学会产生良好的输出。但现在关注的是输出的水平，这在预训练期间、语言模型训练过程中并非如此。

Now let's talk about the limitations. It is indeed the case that these neural networks do have a tendency to hallucinate. But that's because a language model is great for learning about the world, but it is a little bit less great for producing good outputs. And there are various technical reasons for that which I could elaborate on if you think it's useful but at this right now like at this second I will skip that. There are technical reasons why a language model is much better at learning about the world learning incredible representations of ideas of concepts of people of processes that exist. But its outputs aren't quite as good as one would hope or rather as good as they could be. Which is why for example for a system like ChatGPT, this is a language model that has an additional reinforcement learning training process. We call it reinforcement learning from human feedback. But the thing to understand about that process is this. We can say that the pre-training process when you're just training a language model, you want to learn everything about the world. Then the reinforcement learning from human feedback. Now we care about the outputs. Now we say anytime the output is inappropriate, don't do this again. Every time the output does not make sense, don't do this again. And it learns quickly to produce good outputs. But now it is the level of the outputs which is not the case during pre-training during the language model training process.

现在关于幻觉，它确实有编造内容的倾向，确实如此，现在这些神经网络甚至 ChatGPT 时不时会编造内容，这也大大限制了它们的实用性，但我相当乐观，通过简单地改进后续的基于人类反馈的强化学习（RLHF）步骤，我们可以教它不产生幻觉。你可能会说它真的会学会吗？我的答案是让我们拭目以待。

Now on the point of hallucinations and it has a propensity of making stuff up indeed it is true right now these neural networks even ChatGPT makes things up from time to time and that's something that also greatly limits their usefulness but I'm quite hopeful that by simply improving this subsequent reinforcement learning from human feedback step we could just teach it to not hallucinate. Now you could say is it really going to learn? My answer is let's find out.

而这个反馈循环来自公共的 ChatGPT 界面，如果它告诉我我想获得普利策奖，不幸的是我没有，我可以告诉它这是错的，这会训练它或产生一些惩罚或奖励，以便下次我问时它会更准确吗？

And that feedback loop is coming from the public ChatGPT interface that if it tells me that I want a Pulitzer, which unfortunately I didn't, I can tell it that it's wrong and will that train it or create some punishment or reward so that the next time I ask it'll be more accurate.

我们今天做事情的方式是雇人来教我们的神经网络行为，教它行为。而现在，他们指定所需行为的方式，精确的方式有点不同。但确实，你所描述的就是教学基本上会采用的方式。那是正确的教学方式。你只需与它互动，它从你的反应中看到。它推断，哦，那不是你想要的。你对它的输出不满意。因此，输出不好，它下次应该做不同的事情。所以，特别是幻觉成为更大的问题之一，我们会看到。但我认为这种方法很有可能完全解决它们。

The way we do things today is that we hire people to teach our neural net to behave, to teach to behave. And right now, the manner, the precise manner in which they specify the desired behavior is a little bit different. But indeed, what you described is the way in which teaching is going to like basically be. That's the correct way to teaching. You just interact with it and it sees from your reaction. It infers, oh, that's not what you wanted. you are not happy with its output. Therefore, the output was not good and it should do something differently next time. So, in particular, hallucinations come up as one of the bigger issues and we'll see. But I think there is a quite a high chance that this approach will be able to address them completely.

我想和你谈谈 Yann LeCun 关于联合嵌入预测架构的工作，以及他的想法，即大型语言模型缺少的是这个底层的非语言世界模型，语言模型可以参考它。这不是已经构建的东西。但我想听听你对此的看法，以及你是否探索过。

I wanted to talk to you about Yann LeCun's work on joint embedding predictive architectures and his idea that what's missing from large language models is this underlying world model that is non-linguistic that the language model can refer to. It's not something that's built. But I wanted to hear what you thought of that and whether you've explored that at all.

所以我审阅了唯一的的概念提案，那里有许多想法，它们用不同的语言表达，与当前范式可能有一些小的差异，但在我看来它们并不很重要，我想详细说明。第一个主张是，系统最好具有多模态理解，而不仅仅是从文本中了解世界。我的评论是，多模态理解确实是可取的，因为你可以更多地了解世界。你更多地了解人。你更多地了解他们的状况。因此系统将能够更好地理解它应该解决的任务以及人们和他们的需求。我们已经在这方面做了相当多的工作，最著名的是我们做的两个主要神经网络。一个叫 CLIP，一个叫 DALL-E。它们都朝着多模态方向发展。但我也想说，我不认为情况是二元的，要么有要么没有，如果你没有视觉，如果你不通过视觉或视频理解世界，那么事情就行不通。我想为此提出理由。所以我认为有些东西从图像和图表等中学习要容易得多，但我主张你仍然可以仅从文本中学习它们，只是更慢。我给你一个例子。考虑颜色的概念。当然，人们不能仅从文本中学习颜色的概念。然而，当你查看嵌入时，我需要稍微绕道解释嵌入的概念。

So I reviewed the only concept proposal and there are a number of ideas there and they expressed in different language and there are some maybe small differences from the current paradigm but to my mind they are not very significant and I'd like to elaborate. The first claim is that it is desirable for a system to have multimodal understanding where it doesn't just know about the world from text. And my comment on that will be that indeed multimodal understanding is desirable because you learn more about the world. You learn more about people. You learn more about their condition. And so the system will be able to understand what the task that it's supposed to solve and the people and what they want better. We have done quite a bit of work on that most notably in the form of two major neural nets that we've done. One is called CLIP and one is called DALL-E. Both of them move towards this multimodal direction. But I also want to say that I don't see the situation as a binary either or that if you don't have vision, if you don't understand the world visually or from video, then things will not work. And I'd like to make the case for that. So I think that some things are much easier to learn from images and diagrams and so on, but I claim that you can still learn them from text only, just more slowly. And I'll give you an example. Consider the notion of color. Surely one cannot learn the notion of color from text only. And yet when you look at the embeddings, I need to make a small detour to explain the concept of an embedding.

是的，

Yeah,

每个神经网络通过表示、嵌入、高维向量来表示单词、句子、概念。我们可以做的一件事是，我们可以查看那些高维向量，我们可以查看什么与什么相似，网络如何看待这个概念或那个概念。所以我们可以查看颜色的嵌入，颜色的嵌入恰好是正确的。你知道，它知道紫色比红色更接近蓝色，它知道紫色比橙色更不接近红色。它仅从文本就知道所有这些事情。这怎么可能？所以如果你有视觉，颜色之间的区别就会直接跳到你面前。你立即感知到它们。而用文本，你需要更长时间。也许你知道如何说话，你已经理解语法、单词和语法，只有很久以后你才会说，“哦，这些颜色实际上开始理解它们了。”所以这就是我关于多模态必要性的观点，我声称它不是必要的，但它绝对是有用的。我认为这是一个值得追求的好方向。我只是不认为它是如此鲜明的二元主张。所以论文中的提案声称，一大挑战是预测具有不确定性的高维向量。

every neural network represents words, sentences, concepts through representations, embeddings, high-dimensional vectors. And one thing that we can do is that we can look at those high dimensional vectors and we can look at what's similar to what how does the network see this concept or that concept. And so we can look at the embeddings of colors and embeddings of colors happen to be exactly right. You know like it knows that purple is more similar to blue than to red and it knows that purple is less similar to red than orange is. It knows all those things just from text. How can that be? So if you have vision, the distinctions between color just jump at you. You immediately perceive them. Whereas with text, it takes you longer. Maybe you know how to talk and you already understand syntax and words and grammars and only much later you say, "Oh, these colors actually start to understand them." So this will be my point about the necessity of multimodality which I claim it is not necessary but it is most definitely useful. I think it's a good direction to pursue. I just don't see it in such stark either or claims. So the proposal in the paper makes a claim that one of the big challenges is predicting high-dimensional vectors which have uncertainty about them.

例如，预测图像——论文提出了一个非常强烈的观点，认为这是一个重大挑战，我们需要使用特定方法来解决。但我觉得令人惊讶，或者至少论文中没有提到的一点是，当前的自回归 Transformer 已经具备这种能力。我举两个例子。一个是：给定书中的一页，预测下一页。后面可能有很多页，这是一个非常复杂的高维空间，但我们处理得很好。图像也是如此。这些自回归 Transformer 在图像上效果很好。例如，在 OpenAI，我们做过 IGPT 的工作。我们只是把 Transformer 应用到像素上，效果非常好，它能以非常复杂和微妙的方式生成图像。它通过 DALL·E 1 实现了非常漂亮的无监督表示学习——同样，你只需生成——可以把它看作大像素；不是生成百万像素，而是将像素聚类成大像素，然后生成一千个大像素。我相信谷歌今年早些时候关于图像生成的工作 Parti 也采用了类似的方法。所以，我认为论文中关于‘当前方法无法处理高维分布预测’的强烈评论——我认为它们绝对可以。所以也许这是你会提出的另一个观点。

For example, predicting an image—the paper makes a very strong claim that it's a major challenge and we need to use a particular approach to address that. But one thing I found surprising, or at least unacknowledged in the paper, is that current autoregressive transformers already have that property. I'll give you two examples. One is: given one page in a book, predict the next page. There could be so many possible pages that follow. It's a very complicated high-dimensional space, and we deal with it just fine. The same applies to images. These autoregressive transformers work perfectly on images. For example, with OpenAI, we've done work on the IGPT. We just took a transformer and applied it to pixels, and it worked super well, and it could generate images in very complicated and subtle ways. It had very beautiful unsupervised representation learning with DALL·E 1—same thing again: you just generate—think of it as large pixels; rather than generate a million pixels, we cluster the pixels into large pixels and they generate a thousand large pixels. I believe Google's work on image generation from earlier this year called Parti—I believe they also take a similar approach. So the part where I thought the paper made a strong comment around 'well, current approaches can't deal with predicting high-dimensional distributions'—I think they definitely can. So maybe this is another point you would make.

然后你所说的将像素转换为向量——本质上就是把一切变成语言。你知道，向量就像一串文本，对吧？

And then what you're talking about converting pixels into vectors—it's essentially turning everything into language. You know, the vector is like a string of text, right?

但定义一下语言——你把它变成一个序列。

Define language though—you turn it into a sequence.

是的。

Yeah.

什么序列？你可以说，即使对人类来说，生活也是一串比特。现在人们还使用其他东西，比如扩散模型，它们产生这些比特——不是一次一个比特，而是并行产生。但我认为在某种程度上这种区别无关紧要。我声称在某种程度上它并不重要。它的重要性在于你可以获得 10 倍的效率提升，这在实践中是巨大的，但概念上我认为它不重要。

A sequence of what? Like you could argue that even for a human, life is a sequence of bits. Now there are other things that people use right now like diffusion models where they produce those bits—rather than one bit at a time, they produce them in parallel. But I would argue that on some level this distinction is immaterial. I claim that on some level it doesn't really matter. It matters as in you can get a 10x efficiency gain, which is huge in practice, but conceptually I claim it doesn't matter.

关于这个想法：有一大群人类训练师与 ChatGPT 或大型语言模型合作，通过强化学习来引导它。但直觉上，这听起来不像是一种高效的方法来教模型其语言的底层现实。难道没有自动化这种方法的方式吗？为 Ilya 说句公道话，我认为他所说的就是这一点——想出算法手段来教模型底层现实，而不需要人类干预。

On this idea of having an army of human trainers that are working with ChatGPT or a large language model to guide it in effect with reinforcement learning. But just intuitively, that doesn't sound like an efficient way of teaching a model about the underlying reality of its language. Isn't there a way of automating that? To Ilya's credit, I think that's what he's talking about—coming up with an algorithmic means of teaching a model the underlying reality without a human having to intervene.

是的。对此我有两点评论。我认为——首先，我对这个问题有不同的看法。所以我不太同意问题的措辞。

Yeah. So I have two comments on that. I think so. The first place—so I have a different view on the question. So I wouldn't agree with the phrasing of the question.

是的。

Yeah.

我声称我们预训练的模型已经知道关于底层现实所需的一切。它们已经具备这种语言知识，以及大量关于世界上产生这种语言的过程的知识。也许我应该重申这一点。这是一个小插曲，但我认为它非常重要。大型生成模型从其数据中学习到的是——在这个例子中，大型语言模型从文本数据中学习到的是——产生这些数据的现实世界过程的一些压缩表示。这意味着不仅是人们及其思想、情感，还包括人们所处的条件、他们之间的互动、一个人可能处于的不同情境。所有这些都属于那个压缩过程的一部分，由神经网络表示以产生文本。语言模型越好，生成模型越好，保真度越高，它就越能更好地捕捉这个过程。所以这是我的第一点评论，特别是我想说模型已经具备知识。现在，你所说的‘一大群老师’——确实，当你想构建一个尽可能表现良好的系统时，你只需说，如果这个方法有效，就多做一点。但当然，那些老师也在使用 AI 辅助。那些老师并不是独自工作。他们与我们的工具一起工作。他们非常高效。就像工具在做大部分工作。但你确实需要有监督。你需要有人审查行为，因为你最终想要达到非常高的可靠性水平。但总的来说，我想说我们同时也在——这是第二步，在我们拿到完成的预训练模型后，然后对其应用强化学习。确实有很多动机使其尽可能高效和精确，以便最终的语言模型尽可能表现良好。所以是的，有这些人类老师在教模型期望的行为；他们也在使用 AI 辅助，而且他们使用 AI 辅助的方式在不断增加。所以他们自己的效率也在不断提高。所以也许这是回答这个问题的一种方式。

I claim that our pre-trained models already know everything they need to know about the underlying reality. They already have this knowledge of language and also a great deal of knowledge about the processes that exist in the world that produce this language. And maybe I should reiterate this point. It's a small tangent, but I think it's so important. The thing that large generative models learn about their data—and in this case large language models about text data—are some compressed representations of the real-world processes that produce this data. Which means not only people and something about their thoughts, something about their feelings, but also something about the condition that people are in and the interactions that exist between them, the different situations a person can be. All of these are part of that compressed process that is represented by the neural net to produce the text. The better the language model, the better the generative model, the higher the fidelity, the better it captures this process. So that's the first comment I will make, and so in particular I will say the models already have the knowledge. Now the army of teachers as you phrase it—indeed, you know, when you want to build a system that performs as well as possible, you just say okay, if this thing works, do more of that. But of course those teachers are also using AI assistance. Those teachers aren't on their own. They are working with our tools together. They are very efficient. It's like the tools are doing the majority of the work. But you do need to have oversight. You need to have people reviewing the behavior because you want to eventually achieve a very high level of reliability. But overall I'll say that we are at the same time—this second step after we take the finished pre-trained model and then we apply the reinforcement learning on it. There is indeed a lot of motivation to make it as efficient and as precise as possible so that the resulting language model will be as well-behaved as possible. So yeah, there are these human teachers who are teaching a model with desired behavior; they are also using AI assistance, and the manner in which they use AI assistance is constantly increasing. So their own efficiency keeps increasing. So maybe this will be one way to answer this question.

是的。所以你说的是，通过这个过程，最终模型将变得越来越有辨别力，其输出越来越准确。

Yeah. And so what you're saying is through this process eventually the model will become more and more discerning, more and more accurate in its outputs.

是的，没错。这里有一个类比：它已经知道各种各样的事情，现在你只需要真正地说不，这不是我们想要的。不要在这里这样做。你在这个输出中犯了一个错误。当然，正如你所说，尽可能多地将 AI 纳入循环，以便为系统提供最终纠正的老师们的工作被放大，他们尽可能高效地工作。所以这类似于一个教育过程——如何在世界中良好地行动。我们需要进行额外的训练，只是为了确保模型知道幻觉永远是不可以的。一旦它知道了这一点，你就可以开始工作了。

Yes. And that's right. There is an analogy here which is it already knows all kinds of things and now you just want to really say no this is not what we want. Don't do this here. You made a mistake here in the output. And of course it's exactly as you say with as much AI in the loop as possible so that the teachers who are providing the final correction to the system—their work is amplified, they're working as efficiently as possible. So it's not unlike an education process—how to act well in the world. Way we need to do additional training just to make sure that the model knows that hallucination is not okay ever. And then once it knows that, now you are in business.

我明白了。正是那个强化学习人类教师循环将教会它。

I see. And it's that reinforcement learning human teacher loop that will teach it.

人类教师循环或其他变体，但肯定有理由认为这里的方法应该有效，我们很快就会知道。

Human teacher loop or some other variant, but there is definitely an argument to be made that something here should work and we'll find out pretty soon.

这是问题之一。这会走向何方？你现在专注于什么研究？

That's one of the questions. Where is this going? What research are you focused on right now?

我不能详细谈论我正在从事的具体研究，但我可以稍微提一下。我可以粗略地提一些研究，大概是这样的：我非常感兴趣的是让这些模型更可靠、更可控，让它们从更少的数据、更少的指令中更快地学习，让它们确实不产生幻觉。

I can't talk in detail about the specific research that I'm working on, but I can mention a little bit. I can mention some of the research in broad strokes, and it would be something like: I'm very interested in making those models more reliable, more controllable, make them learn faster from less data, less instructions, make them so that indeed they don't hallucinate.

你谈到大脑和神经网络之间的相似性。Jeff Hinton 对我提出过一个非常有趣的观察。我相信这对其他人来说并不新鲜，但大型模型，尤其是大型语言模型，以适中的参数数量容纳了海量数据，而人脑有数万亿个参数，但数据量相对较少。你从这个角度思考过吗？你能谈谈大型模型缺少什么才能拥有更多参数来处理数据吗？这是硬件问题还是训练问题？

You talk about the similarities between the brain and neural nets. There's a very interesting observation that Jeff Hinton made to me. I'm sure it's not new to other people, but that large models or large language models in particular hold a tremendous amount of data with a modest number of parameters compared to the human brain which has trillions and trillions of parameters but a relatively small amount of data. Have you thought of it in those terms? And can you talk about what's missing in large models to have more parameters to handle the data? Is that a hardware problem or a training problem?

你刚才的评论与我在前面问题中提到的一个问题有关，即从更少的数据中学习。确实，当前的技术结构确实需要大量数据，尤其是在训练早期。而在训练后期，它对数据的渴求会有所降低，这就是为什么最终它能学得非常快——虽然还不如人类快，但已经相当快了。所以这已经在某种意义上意味着，我们是否真的在意需要所有这些数据才能达到这一步？但更普遍地说，我认为从更少的数据中学习更多是可能的。我认为这需要一些创造性的想法，但我认为这是可能的，而且从更少的数据中学习更多将释放许多不同的可能性。它将使我们能够教给 AI 缺失的技能，并更容易地向它传达我们的愿望和偏好，以及我们希望它如何表现。所以我想说，更快的学习确实非常好，尽管语言模型训练后已经能学得相当快。我认为在这方面还有更多机会。

This comment which you made is related to one of the problems that I mentioned in the earlier questions of learning from less data. Indeed, the current structure of the technology does like a lot of data, especially early in training. Now, later in training, it becomes a bit less data hungry, which is why at the end it can learn very not as fast as people yet, but it can learn quite quickly. So already that means that in some sense, do we even care that we need all this data to get to this point? But indeed more generally I think will be possible to learn more from less data. I think it's just I think it requires some creative ideas but I think it is possible and I think learning more from less data will unlock a lot of different possibilities. It will allow us to teach our AI the skills that is missing and to convey to it our desires and preferences exactly how we want it to behave more easily. So I would say that faster learning is indeed very nice and although already after language models are trained they can learn quite quickly. I think there is opportunities to do more there.

我听到你发表过一个评论，说我们需要更快的处理器才能进一步扩展，而且模型的扩展似乎看不到尽头，但训练这些模型所需的电力我们正在达到极限，至少是社会可接受的极限。

I heard you make a comment that we need faster processors to be able to scale further and it appears that the scaling of models that there's no end in sight but the power required to train these models we're reaching the limit at least the socially accepted limit.

我只想做一个评论，我不记得我发表过你提到的那个具体评论，但你总是想要更快的处理器，当然你总是想要更多处理器，当然电力消耗在上升，总的来说成本在上升，我要问的问题不是成本是否巨大，而是我们付出这些成本得到的东西是否超过成本。也许你付出了所有这些成本却一无所获。那么是的，那不值得。但如果你得到了非常有用的东西，非常有价值的东西，能够解决我们许多真正想要解决的问题，那么成本就是合理的。但就处理器而言，更快的处理器，是的，随时都要。

So I just want to make one comment which is I don't remember the exact comment that I made that you're referring to but you always want faster processors of course you always want more of them of course power keeps going up generally speaking the cost is going up and the question that I would ask is not whether the cost is large but whether the thing that we get out of paying this cost outweighs the Maybe you pay all this cost and you get nothing. Then yeah, that's not worth it. But if you get something very useful, something very valuable, something which can solve a lot of problems that we have which we really want solved, then the cost can be justified. But in terms of the processors, faster processors, yeah, any day.

你参与硬件问题吗？例如你和 Cerebras 合作，晶圆级芯片。

Are you involved at all in the hardware question? You work with Cerebras for example the wafer scale chips.

不，我们所有的硬件都来自 Azure 提供的 GPU。

No, all our hardware comes from Azure GPUs that they provide.

我看到你曾经谈到民主以及 AI 对民主的影响。有人对我说过，如果你有足够的数据和足够大的模型，你可以在数据上训练模型，它就能提出一个让所有人都满意的最优解决方案。你是否有任何愿景，或者你思考过这在帮助人类管理社会方面可能导向何处？

You did talk at one point I saw about democracy and about the impact that AI can have on democracy. People have talked to me about that if you had enough data and a large enough model, you could train the model on the data and it could come up with an optimal solution that would satisfy everybody. Do you have any aspiration or do you think about where this might lead in terms of helping humans manage society?

是的，让我们看看。这是一个非常大的问题，因为它是一个更面向未来的问题。就像我认为我们的模型仍有很多方式可以变得比现在强大得多。毫无疑问。特别是我们训练和使用它们的方式等等，会有一些变化。这些变化今天可能不会立即显而易见，但我认为事后会非常明显。那确实将使它有能力提出这类问题的解决方案。政府将如何利用这项技术作为获取各种建议的来源，这是不可预测的。我认为关于民主的问题，我认为未来可能发生的一件事是，因为你有了这些神经网络，它们将如此普遍，将在社会中产生如此巨大的影响，我们会发现希望有某种民主过程，比如一个国家的公民向神经网络提供一些信息，关于他们希望事情如何，他们希望它如何表现等等。我可以想象这种情况发生。那可能是一种非常高带宽的民主形式，也许你可以从每个公民那里获得更多信息，并将其聚合起来，以指定我们希望这些系统如何行动。现在这打开了很多问题。但这是未来可能发生的一件事。

Yeah, let's see. It's such a big question because it's a much more future looking question. Like I think that there is still many ways in which our models will become far more capable than they are right now. There's no question. In particular, the way we train them and use them and so on, there's going to be a few changes here and there. They might not be immediately obvious today, but I think in hindsight it will be extremely obvious. that will indeed allow it to have the ability to come up with solutions to problems of this kind. It's unpredictable exactly how governments will use this technology as a source of getting advice of various kinds. I think that to the question of democracy, one thing which I think could happen in the future is that because you have these neural nets and they're going to be so pervasive and they're going to be so impactful in society, we will find that it is desirable to have some kind of a democratic process where this let's say the citizens of a country provide some information to the neural net about how they'd like things to be, how they'd like it to behave or something along these lines. I could imagine that happening. That can be a very like a high bandwidth form of democracy perhaps where you get a lot more information out of each citizen and you aggregate it to specify how exactly we want such systems to act. Now it opens a whole lot of questions. But that's one thing that could happen in the future.

在你给的民主例子中，我可以看到个人将有机会输入数据。但这又涉及到世界模型的问题。你认为 AI 系统最终会大到足以理解一个情境并分析所有变量吗？但我想你需要一个不仅仅吸收语言的模型。

And I can see in the democracy example you give that individuals would have the opportunity to input data. But and this sort of goes to the world model question. Do you think AI systems will eventually be large enough that they can understand a situation and analyze all of the variables but you would need a model that does more than absorb language I would think.

分析所有变量意味着什么？最终你需要做出选择，你说这些变量看起来真的很重要。我想深入，因为一个人可以读一本书，我可以读 100 本书，或者我可以非常缓慢仔细地读一本书并从中获得更多。所以也会有这样的元素。我认为在某种意义上，理解一切可能从根本上是不可能的。任何时候社会中出现任何复杂情况，甚至在一个公司，甚至在一个中型公司，它已经超出了任何单个人的理解范围。我认为如果我们以正确的方式构建 AI 系统，我认为 AI 在几乎任何情况下都可能非常有帮助。

What does it mean to analyze all the variables? Eventually there will be a choice you need to make where you say these variables seem really important. I want to go deep because a person can read the book I can read a 100 books or I can read what book very slowly and carefully and get more out of it. So there will be some element of that also. I think it's probably fundamentally impossible to understand everything in some sense. Anytime there is any kind of complicated situation in society, even in a company, even in a mid-sized company, it's already beyond the comprehension of any single individual. And I think that if we build our AI systems the right way, I think AI could be incredibly helpful in pretty much any situation.

本期节目就到这里。我要感谢 Ilia 抽出时间。我还要感谢 Ellie George 帮助安排采访。如果你想阅读这次对话的文字记录，可以在我们的网站 i on ai 上找到。那是 e- n.aii. A I。我们很喜欢听到听众的反馈。所以请随时给我发邮件，地址是 craig c r a i g at e- n.ai。我收到很多邮件。所以请在主题行中写上 listener，这样我就不会错过。我们在 170 个国家和地区有听众。所以记住，奇点可能并不近，但 AI 正在改变你的世界。所以请关注。

That's it for this episode. I want to thank Ilia for his time. I also want to thank Ellie George for helping arrange the interview. If you want to read a transcript of this conversation, you can find one on our website, i on ai. That's e- n.aii. A I we love to hear from listeners. So feel free to email me at craig c r a i g at e- n.ai. I get a lot of emails. So put listener in the subject line so I don't miss it. We have listeners in 170 countries and territories. So remember, the singularity may not be near, but AI is changing your world. So pay attention.