通往 AGI 的路径：架构、计算与超越大语言模型

我目前看到研究界有些分歧。有些大型 AI 实验室在用 Transformer 构建大型语言模型，并在其上开发产品，声称这是通往 AGI 的道路。还有一些人像 Ilya Sutskever，他们把自己关在房间里专注于研究，试图找到其他实现 AGI 的方法。我想知道你对这些不同方法的看法。

What I see right now is the research community sort of divided. There are big AI labs building large language models using transformers and building products on top of them, claiming this is the path to AGI. And there are people like Ilya Sutskever who basically say they lock themselves in a room focused on research, trying to find other ways to achieve AGI. I'm just wondering what is your take on these different approaches.

我认为有不同的策略。很明显，我们在 AI 方面已经取得了许多成果，但仍然缺少一些能力。尤其是在处理高维输入（如多模态模型）时，还有一些问题需要解决。在与现实世界通过机器人技术交互时，也还有一些问题需要解决。表示学习可以比现在好得多。推理可以更好，智能体可以执行长动作序列，在长时间范围内真正追求目标并纠正错误。当然，我们可以看到这开始有点效果了。我认为未来还会有更多进展，更多能力可以通过我们尚未拥有的巧妙想法来解锁。至少对我来说，正确的道路似乎是先从我们目前行之有效的东西开始，并不断改进，因为我们现在并没有被前沿模型困在角落里，必须全部推倒重来。对我来说，我们已经拥有的东西已经相当惊人，有很多方法可以在不放弃当前范式中真正有效的部分的情况下继续改进。所以我看到所有这些能力会随着时间的推移被添加到非常强大的模型中。

I think there are different strategies. It's clear that we have gotten a lot of things to work in AI, but there are still capabilities missing. Especially when it comes to dealing with really high-dimensional inputs like multimodal models, there are still some things to figure out. When it comes to interacting with the real world through robotics, there are also still some things to figure out. Representation learning could be much better than it is today. Reasoning could be better, and agents that execute long action sequences, really pursue goals over a long time horizon, and correct mistakes. Of course, we can see this starting to work a little bit. I think there's going to be a lot more to come there, a lot more capabilities that can be unlocked with clever ideas that we don't have yet. At least to me, the right path seems to be to start with what we have right now that works well and keep making improvements to it, because it's not like we're stuck in a corner with frontier models right now and we would have to throw it all out of the window and start from scratch. To me, what we have already is pretty amazing, and there are many ways to keep improving that without giving up those aspects of the current paradigm that are actually working really well. So I see all these capabilities getting added over time into really strong models.

你认为 LLM 和基于 Transformer 的架构能带我们走向 AGI 吗？

Do you think that LLMs and in general transformer-based architecture can bring us to AGI?

Transformer 非常通用。我认为几乎任何架构都能带我们走向 AGI。我认为最终架构本身并不那么重要。这更像是一个算力效率问题。可能与我们当前可用的硬件有关。我们可以用 Transformer 实现 AGI。也可以用 RNN 实现 AGI。可能在训练时效率稍低，推理时效率稍高。可能需要稍大的模型来弥补该架构中的瓶颈。总的来说，我认为 Transformer 已经相当不错了。还会继续改进。是的，我认为架构不太重要。

Transformers are pretty general. I think pretty much almost any architecture can bring us to AGI. I think the actual architecture doesn't even matter that much at the end of the day. It's more like a compute efficiency question. Maybe it's a bit specific to what current hardware we have available. We could achieve AGI with transformers. We could achieve AGI with RNNs. Might be a little less efficient at training time, a little more efficient at inference time. Might need slightly bigger models to compensate for all the bottlenecks in that architecture. So overall, I think transformer is already pretty good. There'll continue be improvements to that. Yeah, I think architecture doesn't matter too much.

如果 Transformer 和 LLM 能带我们走向 AGI，但我想你基本上是说架构不重要。那什么重要？算力？什么能带我们走向 AGI？

If transformers and LLMs could bring us to AGI, but I guess you're basically saying that architecture doesn't matter. What matters? Compute? What can bring us to AGI?

算力、目标函数、数据，还有一些算法方面，比如做强化学习时有些细节很重要。我认为在某个时候，我们需要比基本 RL 更长期的信用分配。至于 LLM 能否一路带我们到 AGI？这个问题被广泛讨论。讨论它其实很有意思，因为什么是 LLM？我们在前沿实验室作为产品部署的已经不仅仅是语言模型了，对吧？它们有图像理解能力。现在还有图像生成能力。视频理解，也许视频生成还是分开的，但很快也会成为其中一部分。所以我们已经远远超越了 LLM。实际上，LLM 能否一路带我们到 AGI 已经不是问题了。所以我很高兴你问了这个问题，因为我觉得很多人都在想这个。我只想指出，我们已经超越了 LLM，甚至关于这一点已经没有争议了。

Compute, objective functions, data, some algorithmic aspects like for doing reinforcement learning there are details that matter. I think we'll want longer term credit assignment than basic RL at some point. And I would say, can LLMs get us all the way to AGI? The question is thrown around a lot. It's actually interesting to talk about it because what is an LLM? What we have deployed at Frontier Labs as products are not even just LMs anymore, right? They have image understanding. They have image generation now. Video understanding, maybe video generation is still separate, but it's soon going to be part of that too. So we're already way past LLMs. Actually, there's no question of whether LLMs will get us all the way there. So I'm glad you asked that question because I feel like it's on a lot of people's minds. And I just want to point out that we're already past LLMs and there's no debate even about that anymore.

所以基本上，当我与 Gemini、Claude 或 ChatGPT 交互时，我不仅仅是在与 LLM 交互，而是在与一个非常复杂的多模态系统交互。

So basically when I interact with Gemini or Claude or ChatGPT, I'm not only interacting with an LLM, I'm interacting with a very complex multimodal system.

是的，没错。

Yeah, exactly.

你之前也提到我们需要一些巧妙的想法来实现 AGI。我想知道，在你看来，今天缺少哪些巧妙的想法？

And you also mentioned earlier that we need some clever ideas for AGI. I'm just wondering what clever ideas are missing today, in your opinion.

我不知道确切的想法，对吧？我认为有一些明确的开放问题我们仍需解决。其中之一是超长上下文理解。我们有百万 token 上下文的模型。对于视频，你需要更多，因为视频有太多 token。即使是文本，基于所有上下文的检索和真正推理也还没有完全实现。我不知道解决方案具体会是什么样子。有几种不同的方法，比如基于检索或混合模型，它们学习某种状态表示，但也做一些注意力。也许像联想记忆，有点像 Transformer 但不一定回溯时间。老实说，文献中有很多在 Transformer 之前出现的酷想法。我认为其中很多都有价值。它们只是太早了。当时重要的不是长期记忆或某种花哨的寻址机制，而是扩展规模和拥有计算高效的架构。这只是一个例子。另一个是文本之外的超长视野推理，因为从人类那里学习推理要容易得多，但那样你会受到人类推理能力的限制。所以 AI 系统应该能够做得更好，如果你把正确的算法放进去，让它自己发现推理。

So I don't know the exact ideas, right? I think there are some clear open problems we still need to solve. One of them is really long context understanding. We have models with a million tokens context or more. For video, you would want a lot more than that because video has so many tokens. Even for text, the retrieval and really reasoning based on taking all that context actually into account is not quite there yet. I guess I don't know what the solution is going to look like exactly. There are a couple different approaches based on maybe retrieval or hybrid models that learn some sort of state representation, but they also do some attention. Maybe like associative memories that are a bit like a transformer but not attending back in time necessarily. Honestly, there are a lot of cool ideas in the literature that came out before transformers. I just think a lot of those have merit. They were just too early. Like what mattered at the time was not really long-term memory or some fancy addressing mechanism, but it was just scaling up and having a compute efficient architecture. That's just one example. I think another one is really long horizon reasoning outside of text because it is far easier to learn reasoning from humans, but that way you're going to be limited by human reasoning capabilities. And so an AI system should be able to do way better than that if you put the right algorithms into it that would allow it to discover reasoning by itself.

所以接下来你必须弄清楚如何从原始数据中提取可能在空间和时间上抽象的概念，这些数据可能是高维的，比如视频或音频数据，人们生活的数据，最终可能是生活在人类周围的机器人的数据。是的，抽象出概念，然后可能以某种方式对它们进行规划。我认为我们还没有很好地掌握如何做到这一点，而且也不清楚我们能否在未来几年内实现这一点，但我认为一旦我们做到了，那将产生巨大影响。

And so then you have to figure out how to extract concepts that are maybe abstract in space and time from raw data, maybe high-dimensional like video or audio data, data of people living their lives, eventually maybe data of a robot that's living around people. Yeah, abstracting concepts and then maybe planning over them in some way. I don't think we have a great handle on how to do that, and it's not clear that we'll be able to get that to work in the next few years, but I think once we do, it'll be really impactful.

我们正在朝着持续学习对话的方向发展。实际上，你刚才说的话让我想起，我最近和托马斯·波乔谈过，他是戴密斯·哈萨比斯的导师，他说在 2015 年，戴密斯曾表示要构建通用智能，需要 80%的神经科学和 20%的工程学。所以戴密斯最近更新了说法，大概是 90%的工程学，其余是神经科学。基本上，当你说我们需要一种不同于人类的推理方式时，确实我们正在创造一种不同类型的智能，也许与人类大脑的工作方式进行比较是不正确的，对吧？因为它可能是一种完全不同的、更强大的东西。关于持续学习，我理解你的 Dreamer 3 研究工作正是为了实现持续学习，对吧？

We're moving toward words, continue learning conversation. And actually what you said reminded me, you know, I recently spoke with Thomas Pojo, he was a mentor of Demis Hassabis, and he said in 2015 Demis was saying that to build general intelligence, it's 80% neuroscience, like 20% engineering. So a recent update from Davis is that it's, I'm not claiming the correct quote, but it's basically something like maybe 90% engineering, the rest may be neuroscience. Basically when you said that we need a different type of reasoning, different from human, indeed we are creating a different type of intelligence, and maybe comparing with how the human brain works is incorrect, right? Because it could be a very different beast and more powerful. To the point of continual learning, I guess as I understand your research work with Dreamer 3, this is the attempt, right, to enable continual learning.

首先，我喜欢神经科学的比较。我确实认为神经科学可以教给我们很多。只是有很多不同的方式来处理神经科学，你可以在非常低的层次上建模连续时间，比如神经元动力学。这会变得非常昂贵和低效，而且很难实际训练一个可扩展的系统。我认为如果你真的考虑到所有约束，大脑是相当大的。如果你做神经科学并想构建一个大脑模型，它最好也相当大。那么为了算力原因开始简化某些概念是有意义的。也许 30 年后，我们有了足够的算力可以在低层次上做到这一点。我们可以看看这是否会带来额外的好处。你知道，我们现在可以通过工程学以相当明显的方式改进很多事情。这些系统在算力和分布式计算方面如此庞大，以至于需要大量的工程。而且还有很多细节，直觉无法帮助你。你对大脑如何学习或某个算法想法有很好的直觉，但现在有大约 10 个细节你必须尝试、非常小心地比较超参数。所以这需要大量的实验和工程。但这并不是说神经科学没有价值。我认为由于我们最近在工程上推进了很多，实际上回过头从神经科学中获取直觉越来越有价值，持续学习就是其中之一。这取决于你的系统在持续学习时所处的环境，对吧？所以你已经可以在文本领域通过部署的聊天机器人与用户互动来进行一定程度的持续学习，这是一个有趣的环境，一个非常动态的环境。你可以通过与人们交谈来学习新知识，但与人类实际与真实世界互动相比，这仍然非常有限。我认为那种持续学习还很遥远，而且不一定能解决当前的瓶颈。当我们谈论机器人技术时，我认为最初的进展将来自大量的数据收集，可能也包括人类数据收集，然后是网络预训练，其中很多将是视频数据。我认为未来几年最大的改进将来自那里，然后我们可以在某个时候考虑持续学习，以挤出更多的鲁棒性并处理长尾问题。

First of all, I like the neuroscience comparison. I do think neuroscience can teach us a lot. It's just there's so many different ways to approach neuroscience and you could do it at a really low level of modeling continuous time like neuronal dynamics. It gets very expensive and inefficient and hard to actually train a scalable system. I think if you actually take all the constraints into account, you know, the brain is pretty big. If you do neuroscience and you want to build a model of the brain, it better be pretty big. And then it makes sense to start simplifying certain concepts for compute reasons. And maybe in like 30 years from now, we have the compute that we could do it at the low level. We could see if that gives additional benefits. You know, there are so many things that we can improve right now in like quite obvious ways through engineering. And these systems are so massive in terms of compute and distributed computing that a lot of engineering is necessary. And there are also so many details where no intuition can help you. You know, you have great intuition about how the brain learns or just some algorithmic idea. But now there's like 10 details you just have to like try out, be really careful about, compare hyperparameters. So it's a lot of experimentation and engineering. But that's not to say that there's no merit in neuroscience. Like I think since we've been pushing the engineering so far recently, I think actually there's more and more value in going back and getting intuitions from neuroscience and continual learning is one of them. It depends what environment your system is embodied in for continual learning, right? So you can already do maybe some amounts of continual learning in text land from deployed chatbots interacting with users and it's an interesting environment. It's a very dynamic environment. You can learn new knowledge by talking to people but it's still like very limited compared to a human actually interacting with the real world. That type of continual learning I think is still quite far and it's not necessarily addressing any current bottleneck I would say. When we're talking about robotics, I think the initial progress will come from of course a lot of data collection, maybe human data collection as well, and then web pre-training and a lot of that will be video data. I think that's where sort of the biggest improvements will come from in the next few years and then we can worry about continual learning at some point to squeeze out more robustness and work on the long tail.

我认为通过 Dreamer，我尝试用不同的 Dreamer 版本解决不同的问题。直到 Dreamer 3，它主要是通过与环境交互从头学习，使其非常数据高效，然后也推动你能达到的最终性能，因为直到 Dreamer 2，基于模型的算法让你学得很快，但随后它们会达到上限，而无模型方法需要更多数据来学习，但它的上限更高。而到了 Dreamer 3，我们实际上达到了可以超越任何现有强化学习算法的程度，并且使其鲁棒到无需调整超参数。我们还有 Minecraft 钻石挑战，即仅从稀疏奖励中从头训练或学习获得钻石。而到了 Dreamer 4，我们做了完全相反的事情，说：‘等等，不，我们现在知道如何在线学习了。离线学习怎么样？你也希望能够仅从你拥有的数据集中提取尽可能强的策略，在你知道与环境交互很危险的情况下。那么，如果我们只有这个固定的仅包含人类数据的数据集，没有别的，我们能从中学习获得钻石吗？’当然，这两种都不是完美的解决方案。它们只是在孤立的实验设置中解决一个特定问题，但在未来，当然你会把所有这些东西结合起来。

I think with Dreamer I've tried to solve different problems with different Dreamer generations. So up to Dreamer 3, it was really all about learning from scratch by interacting with some environment and making it really data efficient and then also pushing the final performance that you can achieve because up until Dreamer 2, model-based algorithms let you learn very quickly, but then they would sort of max out whereas a model-free approach would need a lot more data to learn, but it just has a higher ceiling. Whereas then with Dreamer 3, we actually got to the point where we can pretty much outperform any RL algorithm out there and also make it robust to the point where you don't have to tune hyperparameters. And we had the Minecraft diamond challenge like train or learn to get diamonds from scratch just from sparse rewards. Whereas with Dreamer 4, we did exactly the opposite and said, 'Wait, no, we know how to learn online now. How about learning offline? You also want to be able just from a data set you have to extract as strong a policy out of it as you can in situations where you know it's dangerous to interact with the environment. And so then it's like what if we only have this fixed data set of only human data, right, and nothing else. Can we learn to get diamonds from that? And then of course neither of those is a perfect solution. It's just addressing one specific problem in an isolated experimental setup, but in the future, of course, you throw all those things together.

我只是好奇。我遇到过嵌套学习模型或嵌套训练模型的概念。在我看来，它们的有趣之处在于：它们将上下文转化为一个神经模型，这样上下文就能被快速学习，即使在 GPT 类型的东西中也是如此。它不再是一个真正的 GPT，而是一种嵌套学习。你可以将上下文接入神经模型，而不是将其保持在外，这在我看来是一件非常有趣的事情，它改变了你所谈论的一些内容。我只是好奇你是否有什么要说的。

I was just curious. I've come across this notion of nested learning models or nested training models. And what they do, in my view, is interesting: they take the context and they make it into a neural model so that the context is actually learned quickly, even in a GPT type thing. It's not really a GPT anymore, but it's a nested learning. You can wire the context instead of keeping it external into the neural model, and that seems to me to be a pretty interesting thing to do that shifts some of what you were talking about. I was just curious if you had anything to say about that.

是的，我认为这有点像参数化记忆的方法。我不知道你具体说的算法，但是

Yeah, I think it's like a sort of parameterized memory approach. I don't know the exact algorithms you're talking about, but

这个概念是，GPT 或任何你称之为的东西的神经架构中有一部分被允许从输入中快速学习，从而动态构建上下文。我的意思是，大多数 GPT 在推理时吸收上下文，然后将其丢弃。

This is the notion of having one portion of the neural architecture of the GPT or whatever you want to call it that is actually allowed to learn quickly from the input so that the context is being built dynamically. I mean most GPTs they absorb the context at inference time and then they throw it away.

没有持续的学习。它是一个固定的静态实体。它们无法从上下文中学习，因为上下文在推理时通过它们，然后就消失了。

There is no ongoing learning. It's a fixed static entity. They can't learn from the context because the context passes through them at inference time and then it's gone.

嵌套学习，据我理解，是将神经网络架构的一部分用于在输入上下文时进行快速训练，上下文可以任意大，这样它就能在模型内部的对话中保持相邻并保留下来。有一项工作是在推理时训练架构中非常小且轻量的部分。我觉得这是个有趣的方法。我喜欢它的地方在于：当你训练神经网络时，你是在用一个目标函数优化它，对吧？所以随着训练越多，它会变得越好。而上下文学习则是一种完全不同的机制，你只是希望模型已经学会了一种看起来像学习的泛化方式。但系统中并没有内置任何东西让它真正积极地优化某个目标。它并不努力去记住东西，也不努力去理解上下文中的模式以帮助预测。我们可以通过构造巧妙的训练示例，迫使模型解决谜题、记住东西等，在训练时把这些能力都注入权重中。但归根结底，它仍然是一个学习到的算法。它可能不像实际执行优化过程那样目标明确。

Nested learning as I understand it devotes a portion of the neural architecture to essentially quick training on the context as it's being input, and the context can be as large as it is, so that it is adjacent and retained throughout the conversation within the model. There is a line of work on training some really small lightweight part of the architecture at inference time. I think that's an interesting approach. What I like about it is this: when you train a neural network, you're optimizing it with an objective function, right? And so it will get better as you train it more. In-context learning is this totally different mechanism where you just hope that the model has learned to generalize in a way that looks like learning. But there's nothing built into the system that would really make it aggressively optimize any objective. It's not trying to be really good at remembering. It's not trying to be really good at understanding the patterns in the context in a way that helps it predict what will happen. We can try to get all of these things into the weights during training time by constructing clever training examples that force the model to solve puzzles and remember things and so on. But it's still going to be a learned algorithm at the end of the day. It may not be as goal-directed as if you actually do an optimization procedure.

显然，你无法在推理时进行优化，所以无论多少预训练都无法预见到推理时会出现什么并被引入上下文。

You can't optimize at inference time obviously, and so no amount of pre-training is ever going to anticipate what is going to come and be introduced into the context at inference time.

没错。模型是否仅仅成为一个被动的推理引擎，每次推理时你把整个上下文传给它，还是说模型在这个会话中发展出对上下文的某种意识？

That's right. Does the model simply become a passive inference engine that you pass the entire context to each inference, or does the model in this session develop an awareness, shall we say, of the context?

我认为这是一个很酷的方向。我也有同感，仅仅像我们通常做的那样把上下文学习当作一种算法来学习，可能不足以构建最强大的系统。有一种解决持续学习的方法就是，当然有很多不同的方法让推理时计算更高效，我觉得这很有道理。但同时，我认为在某个时刻我们会让模型在与世界交互时持续训练。如果你有数百万用户，也许你会把一万次用户交互批量处理，进行一次更新，这样模型就会真正深入学习。那样的话，甚至可能不需要不同时间尺度的学习。但也有可能我们确实需要多个时间尺度的学习以提高效率。那么可能会有一个相当通用的方法来构建算法，你可以说我要 k 等于五个学习时间尺度。较快的尺度训练起来会更高效。我也能看到类似的泛化。是的，我不认为我们在这个领域已经有足够好、值得在生产中使用的算法，但这是一个非常有趣的方向。而且我完全同意持续学习很重要。

I think it's a cool direction. I share the intuition that just learning in-context learning as an algorithm like we typically do is likely not going to be enough to build the most powerful systems. There is one way to solve continual learning which is just to say, of course there are all these different ways to make it more computationally efficient at inference time, and I think that does make a lot of sense. But also, I think at some point we're just going to keep training the model as it interacts with the world. If you have millions of users, maybe you'll batch like 10,000 user interactions together and do an update on it, and then the model will actually deeply learn. And then there may not even be the necessity to have a different time scale of learning in that sense. But it's also possible that we do want multiple time scales of learning for efficiency. And so then there might be a quite general way to just make an algorithm where you can say I want k equals five learning time scales. Now the faster time scales will be more efficient to train. I could see some generalization like that as well. Yeah, I don't think we have any algorithms in that space yet that actually work well enough to truly be worth it in production, but it's a very interesting direction. And yeah, I totally agree with continual learning being important.

对我来说，这似乎完全取决于反馈循环的实际长度，对吧？比如 GPT-4 发布后与世界交互，OpenAI 当然会从这些交互中获得大量数据，这些数据会影响 GPT-5 的下一次训练，但那可能是一两年后的事了。我们能不能把这个时间缩短到几天甚至几秒？能不能把这个差距从两年缩短到十天左右？

To me, it seems like it's all a question of how long the feedback loops really are, right? Because let's say GPT-4 is released and interacts with the world, and of course OpenAI gets a lot of data from those interactions, and that'll influence GPT-5 for the next training run, but that might be like a year or two later. Can we get that down to maybe days or even seconds? Can we reduce that gap from two years to like 10 days?

是的，我认为这是一个持续学习的问题。理论上我们当然可以做到。但在大规模上进行实验相当困难，因为训练一个大模型非常昂贵，而且还要确保模型在这个过程中保持安全。拥有一个静态模型有很多好处，因为它更容易研究。弄清楚模型有哪些怪癖以及如何修补它们已经够难的了。这是一个挑战，但我觉得最终我们会逐渐缩短这个反馈循环。

Yeah, I mean I think that's a continual learning question. We definitely can in theory, I guess. It's quite hard to experiment with that at large scale because training a big model is so expensive, and then making sure that the model stays safe as you do that. There are a lot of benefits of just having a static model because it's easier to study. It's hard enough to figure out what kind of quirks the model has and how to patch them. It is a challenge, but yeah, I think eventually we'll shorten that feedback loop over time.

你提到我们需要更好的目标函数。我想就此深入聊聊。你能详细说明一下吗？比如你说的更好的目标函数是什么意思？

You mentioned that we need better objective functions. I just wanted to touch base on that. Can you elaborate on that? Like what do you mean by better objective functions?

从根本上说，有两种类型的目标函数。一种是偏好，比如奖励或归纳偏置，这些是我们人类必须指定的，因为它们并没有太多数学结构。你知道，我们的偏好是由社会和我们成长经历等塑造的。所以没有方程可以描述它们。你只能从数据中学习，比如从人类评分或人类反馈中学习。另一种是基于信息的目标，它们试图理解可用数据，对吧？比如预测目标，尝试预测下一个词，或者掩码自编码器学习表征，或者对于智能体来说，甚至有一个好奇心目标，让智能体在环境中寻找信息。可能在这两个方面都有很大的改进空间。我们现在有非常强大的优化设置，对吧？我们的架构学习效率很高，我们有很好的优化器，以及构建这些可扩展系统的大量工程。在系统层面上，数据和目标变得非常重要。我认为对于文本，下一个词预测可以走得很远，但还有很多可以做的。例如，同时预测多个词有明显的好处；它让模型变得更有远见。然后对于多模态，现在有很多不同的目标，但仍然是各种目标的拼凑。也许我们对视觉编码器使用对比损失，然后在交织的文本和图像词元上使用下一个词预测的 softmax，再加上一个扩散损失来生成图像。我们必须平衡所有这些损失。我认为可能有办法统一所有东西，让我们的生活更简单，最终也能带来更好的性能，对吧？因为是有好处的。某个损失对一种模态效果更好，另一种对另一种模态更好，这是有原因的。这些好处是存在的。我不认为这些是根本性的权衡。只是如果我们能从一切中抽象出来，我们就能在整个范围内共享这些好处。

At a fundamental level, there are two types of objective functions. There are preferences like rewards or inductive biases, those kinds of things that we as humans have to specify because there's not really much of a mathematical structure to them. You know, our preferences are shaped by society and our experiences growing up and all that. So there's no equation that can describe them. You sort of have to learn them from data, from human ratings let's say, or human feedback. And then there are information-based objectives that try to make sense of the data available, right? Like a prediction objective, try to predict the next token, or a masked autoencoder that learns representations, or for an agent even a curiosity objective where the agent tries to seek out information in the environment. Probably on both fronts actually there's good room for improvement. We have really powerful optimization setups now, right? Our architectures learn very efficiently and we have great optimizers, a lot of engineering that went into building these scalable systems. Just at a systems level, data and objectives become quite important. I think for text, you can get quite far with next-token prediction, but also there is a lot more you can do. For example, there are clear benefits of predicting multiple tokens at once; it trains the model to be a bit more farsighted. Then when it comes to multimodality, there are so many. Right now it's still sort of just like a patchwork of different objectives. Maybe we use a contrastive loss for the vision encoder, and a next-token prediction softmax over some interleaved text and image tokens on top of that, and then maybe there's a diffusion loss to generate images from that model. We have to balance all these losses. I think there's probably ways to unify everything that makes our lives simpler and also ends up giving us better performance in the end, right? Because there is a benefit. There's a reason that let's say one loss works better for one modality and another one better for another. There are benefits to that. I don't think those are like fundamental trade-offs. It's just if we can abstract away from everything, we'll be able to share these benefits across the whole spectrum.

然后谈到构建智能体，我认为设计空间非常广阔。短期强化学习，比如学习最多一千步左右的任务，现在已经非常稳健有效。但训练解决长周期任务、端到端优化，效果仍然不佳。每一步都在累积误差。我认为在改进端到端强化学习方面还有很多探索空间，之后可能还要弄清楚探索目标、目标达成目标，以及如何以通用方式学习稳健的奖励模型。这些东西唯一缺少的就是目标函数。你可以说我们没有数据，但说实话，我们有现成的数据，而且通过人工收集的任何数据也不难获取。所以关键在于如何构建这样一个系统。我们在 Scaling 和数据工程上投入了很多，已经做得相当好了，我们不应该停止。但这已经不那么难了，现在我们又回到了算法设计上。

And then when it comes to building agents, I think there's a huge design space. Short-term RL, like learning tasks up to maybe a thousand time steps, works really well and robustly now. But training to solve long-horizon tasks, optimizing them end to end, is still not working that well. Just accumulating errors at each time step. I think there's still a lot to explore there in terms of improving end-to-end RL, and then after that, maybe figuring out exploration objectives, goal-reaching objectives, how to learn robust reward models in a general way. The only thing missing for those things is basically objective functions. You could say we don't have the data, but honestly, we have the data that's there, and anything you can collect manually through humans is also not that hard to collect. So it's really about having an idea of how to build a system like that. We have done so much scaling and data engineering that we've gotten quite good at that, and we shouldn't stop doing that. But it's just not that hard anymore, and now we're back to figuring out algorithms.

你也研究具身，对吧？这是你研究的核心。我想知道，就具身和与环境的物理交互而言，这对更高效的世界模型至关重要吗？你对与物理世界交互以及构建通用智能体有什么看法？

You also work on embodiment, right? That's core to your research. I'm wondering, in terms of embodiment and physical interaction with the environment, is this crucial for more efficient world models? What's your take on interaction with the physical world and building general agents?

当你在固定数据集上训练世界模型时，它会在某些方面受到限制，因为会有它从未见过的反事实情况。我们注意到，例如在 Minecraft 中的 DreamerV4，它只基于人类数据训练。游戏中大概有五种不同的镐可以制作：木镐，用三个木板和一些木棍；石镐，用三个石头和一些木棍。人类玩家知道这些，所以他们从不尝试用其他材料做镐，但强化学习智能体会。于是世界模型会想：‘嗯，我不知道。看起来他们在做镐。也许我该给他们一把镐，尽管这个配方实际上并不存在。’仅靠离线数据能走多远是有限度的。好消息是，我们发现通过与环境交互获取纠正数据，不需要太多轮次，这些问题就会很快消失。我认为，强化学习智能体针对世界模型进行优化并发现所有潜在缺陷，然后将其部署到实际环境中，获取哪些有效哪些无效的反馈，这之间存在重要的相互作用。这是一个对抗性游戏，最终世界模型会非常稳健，你会得到一个非常强的策略。至少在 Minecraft 中，经过两到三轮后，我们就不再看到这些漏洞了。在现实世界中可能需要更多，但除非你有一个完美的数据集——我认为在现实世界中不可能预先构建——否则你必须与环境交互才能得出真正的因果世界模型。一旦你有了这个模型，也许你就有了下一次训练的数据集，直接在上面训练可能就没问题了。

When you train a world model on a fixed dataset, it will be limited in certain ways because there will be counterfactuals it has never seen. We noticed that for DreamerV4 in Minecraft, for example, it's only trained on human data. In the game, there are maybe five different pickaxes you can make: a wooden pickaxe, you put three planks and some sticks; a stone pickaxe, you put three stones and some sticks. Human players know that, so they never try to make a pickaxe out of some other material, but the RL agent does. So the world model thinks, 'Hmm, I don't know. It looks like they're making a pickaxe. Maybe I'll give them a pickaxe, even though that recipe doesn't actually exist.' There is definitely a limit to how far you can go just from offline data. The good news is that we found that with not too many rounds of getting corrective data from the environment, those issues actually go away very quickly. I think there's an important interplay between an RL agent that optimizes against the world model and finds all the potential flaws, and then deploying that in the actual environment, getting feedback for which things worked and which didn't. It's an adversarial game where at the end, the world model will be very robust and you'll have a really strong policy. At least in Minecraft, after two to three rounds, we didn't see those exploits anymore. In the real world, it might take a bit more, but unless you have a perfect dataset, which I think in the real world will not be possible to construct upfront, you have to interact with the environment to figure out a real causal model of the world. Once you have that, maybe then you have the dataset for the next training run, and it might be fine to just directly train on that.

你扩展过你的世界模型吗？我很好奇它们在规模扩大时作为智能体的表现。

Have you scaled your world models? I'm curious to see how they behave as agents at scale.

是的，我们确实扩展了。这些是我们没有写进论文的结果——论文里内容已经太多了。我们一直扩展到与前沿视频模型相同的大小，效果很好。凭借我们现在的架构，基本上是高效的扩散强制视频模型和稳健的强化学习目标，你可以很容易地扩展。这只是一个你能投入多少算力的问题，以及你能聚集多少千块芯片来训练一个大模型。

Yes, we definitely have. These are results we didn't put in the paper—there's already too much in there. We scaled up all the way to the same size as frontier video models, and it works well. With the architectures we have now, which are basically very efficient diffusion forcing video models and the robust RL objective, you can scale very easily. It's just a question of how much compute you can put in, really, and how many thousand chips you can pull together to train a big model.

你有没有观察到智能体在规模扩大时行为上有什么有趣的发现？

Have you seen any interesting observations in the behavior of agents when scaled?

你会看到所有那些人们之前对大型语言模型感到兴奋的涌现行为。由于世界模型默认是一个无监督目标，你不会告诉它把容量放在数据的哪些部分；模型会根据其归纳偏置自行决定。所以如果你的模型太小，比如库存预测就不会很好。你可以专门为此收集数据，我们也试过——这确实有改进。但另一种方法是直接扩展模型，把它做大八倍，它就会开始非常擅长库存动态，特别是如果你有一些之前运行的纠正数据。基本上，模型越大，它在所有方面都变得更敏锐。当然，我们也进行了完整的 YouTube 预训练，抓取大数据集，进行质量过滤，然后训练。这时你才能真正获得强大的泛化优势。不幸的是，发表很多这类前沿成果有点挑战。我认为这些模型的 Scaling 进展非常顺利，而且我特别要说，视频模型和多模态模型的 Scaling 天花板远高于文本模型，因为视频中包含的信息多得多。即使是顶尖的视频模型也基本上处于欠拟合状态，对吧？当然，它们有点坍缩成非常漂亮的电影片段，这有助于模型——尽管不是特别巨大——仍然能生成非常棒的输出。如果目标不仅仅是生成漂亮的视频，而是通过这些视频，以及可能通过一些机器人数据或人类与世界交互的数据，对物理世界有丰富的理解，比如从互联网视频进行预训练。如果目标是深入理解世界，这更适用于机器人和交互式助手，而不是像为电影生成视频片段这样的创意应用，那么对于这种通用视频模型，我认为 Scaling 天花板至少比文本模型高一个数量级。所以我们目前还远未达到那个水平。

You see all the same emergent behaviors that people were really excited about for LLMs a while ago. Since a world model by default is an unsupervised objective, you're not telling it what parts of the data to put its capacity into; the model decides based on its inductive biases. So if your model is too small, for example, inventory predictions would not be very good. You could specifically collect data for that, and we tried that too—it definitely improves it. But another way is to just scale up the model, make it eight times bigger, and it starts to get really good at inventory dynamics, especially if you have a little bit of corrective data from previous runs. Basically, the model gets sharper at everything the bigger you make it. Of course, we also did full YouTube pre-training runs, scraped big datasets, filtered them for quality, and trained on that. That's when you really get strong generalization benefits. Unfortunately, it's a little challenging to publish a lot of this cutting-edge stuff. I think scaling is going very well with these models, and especially, I would say the ceiling for scaling video models and multimodal models is way higher than for text models because there's so much more information in video. Even top video models are basically underfitting, right? Of course, they are sort of collapsed down into very pretty cinematic clips, and that helps the model—even though it's not as gigantic—to still make really nice generations. If the goal is not just to generate nice videos, but to have a rich understanding of the physical world through those videos, and maybe through some robot data or data of humans interacting with the world, pre-training like videos from the internet. If the goal is to have a deep understanding of the world, which is more relevant for robotics and interactive assistance, as opposed to creative applications like generating video clips for movies, then for this general type of video model, I think the scaling ceiling is going to be at least an order of magnitude higher than what we see for text models. So we're not even close to that right now.

那么我们可以继续扩展视频数据，从中提取越来越多的知识，对吧？你期望模型从中获得哪些能力？

So we can continue scaling like video data and extracting more and more knowledge from it, right? What capabilities in models do you expect from that?

很难猜测，但一个目标当然是实现跨模态的迁移。我认为最明显的是获得真正擅长理解世界的表征。要预测未来，你必须知道物体将如何移动、它的质量、摩擦力、另一面看起来如何（因为它可能会旋转）、物体如何相互作用以及人们如何互动。在表征方面有很多可以提取的。我们已经看到这些表征对于微调策略非常强大，而且我想象它们可能也非常适合关于现实世界的推理任务，尤其是如果你有一些上下文，比如你在手机上向智能体展示一些东西。

It's hard to make a guess, but one goal is of course to get transfer across modalities. I think the most obvious is just going to be having representations that are really good at understanding the world. To predict forward, you have to know how an object is going to move, its mass, its friction, how it looks on the other side because maybe it's going to rotate, how objects interact, and how people interact. There's a lot to extract in terms of representations. We've seen those representations be really strong for fine-tuning policies into them, and I would imagine they are probably really good representations also for reasoning tasks about the real world, especially if you have some context, like you show your agent something on the phone.

它能否迁移到机器人领域？我的意思是，现在在机器人领域，模仿学习是带来这些进步的主要技术。我想知道这种更丰富的表征是否也能迁移到机器人领域。

Would it be transferable to robotics? I mean, now in robotics, imitation learning is the main technique that brought those advancements. I'm wondering if this richer representation is also transferable to robotics.

哦，是的，100%。实际上，表征学习技术对模仿学习性能有很大的提升。如果你从头开始训练策略，需要大量数据，而且你会得到一个非常脆弱、狭窄的策略，只适用于特定场景。如果你从预训练的 VLM 开始，可以做得更好一些。但这些表征仍然不是真正适合在物理层面理解世界的，不是那种对控制至关重要的层面。比如精确的物体位置和物体的物理属性，比如我拿起这个盘子时它有多滑？如果我握住杯子的手柄，我需要用多大的力才能不让它滑落，不让茶洒满桌子？这些表征非常自然地从某种视频预测模型中获得。不一定是扩散模型，因为主要是预训练期间学到的表征。我们已经看到这些表征效果要好得多。所以我认为，仅仅通过扩展通用视频模型，通过它们的表征，我们就会看到这对机器人技术产生巨大影响。然后在 Dreamer 4 中，我们展示了如何通过添加智能体令牌来微调这样的模型，训练 V-C 策略，然后训练奖励模型，最后进行强化学习微调。这是世界模型产生巨大影响的第二种方式：通过足够多样化的预训练和对机器人数据的一些微调，它实际上可以想象或模拟机器人在你提示模型想象的任何场景中的行为。所以突然之间，你可以在数据中心并行训练你的机器人在一百万个厨房里做一百万种不同的饭菜，而无需租用一百万个 Airbnb、制造一百万个机器人并在城市中移动它们。大规模这样做有点挑战，但我认为这是继表征之后我们在机器人领域看到的第二个阶跃变化。

Oh yeah, 100%. There's actually a lot of leverage for representation learning techniques on imitation learning performance. If you train your policy from scratch, it's going to take a lot of data and you'll get a pretty fragile narrow policy that only works in that specific scenario. If you start with a pre-trained VLM, you can do a little bit better. But those representations are still not really the right ones for understanding the world at a physical level, at a level that matters for control. Like precise object locations and physical properties of objects, like how slippery is this plate if I pick it up? If I pick up this cup at the handle, how firm do I have to hold it for it not to slip out of my hand and for the tea to spill all over the table? Those representations are very natural to get from some sort of video prediction model. It doesn't necessarily have to be a diffusion model since it's mostly about the representations being learned during pre-training. But those representations we already have seen work massively better. So I think just scaling up general video models will be the very first way in which we'll see that be really impactful for robotics, through their representations. And then in Dreamer 4, we showed the whole recipe for how to fine-tune a model like that by adding agent tokens to train a V-C policy, then train a reward model, and then RL fine-tuning. So that's the second way in which the world model can be really impactful: with enough diverse pre-training and some fine-tuning on robot data, it can actually imagine or simulate the robot in any scene that you can prompt the model to imagine. So suddenly you could train your robot in a million kitchens trying to make a million different meals all in parallel in your data center without having to rent a million Airbnbs and produce a million robots and move them around the city. Doing that at scale is a little more challenging, but I think that's the second step change after the representations that we'll see in robotics.

我大致浏览了你的两篇论文。有些事情我理解上还有差距，但有几件事很有趣。在语言模型中，当你比较预训练和强化学习时，它们能学到什么以及如何学习似乎存在质量差异。预训练倾向于学习更多事实性内容，而强化学习不太擅长学习新的基本领域原理，但一旦你有了预训练模型，它可以非常快速地发展新技能。所以当你把两者结合起来，我认为有很多有趣的概念性东西值得思考。

I did take a scan of both of your papers. A number of things are gaps in my understanding exactly what's going on, but a few things are interesting. One thing in language models is there seems to be a quality difference in what they can learn and how they learn it when you look at pre-training versus reinforcement learning. Pre-training tends to learn more factual stuff, and reinforcement learning tends not to be so good at learning new fundamental domain principles, but once you have a pre-trained model, it can be really quick in developing new skills. So when you put those together, I think there are a lot of interesting conceptual things to think about.

我认为从示例中学习知识肯定比从奖励中学习容易得多。对于强化学习，你必须先猜测知识，然后模型会被告知猜测是否正确，所以吸收信息的效率低得多。而强化学习非常擅长优化策略，因为很难获得最优控制数据。假设你有一个 LLM 智能体。实际上很难获得一个数据集，告诉你现在最优的下一步是什么。祝你好运。人类远非最优。如果你用承包商收集数据，你可能不得不丢弃大约 99%的数据。最优性还取决于你的视野。理想情况下，你希望它在很长的视野内是最优的。所以很难为此获得数据。这正是强化学习大放异彩的地方。此外，一旦模型已经预训练，它已经学到了一些不错的行为。那时需要优化的东西就少多了。它实际上只是多做那些有效的事情，少做那些效率不高的事情。我认为人类也是如此。我们肯定通过观察来学习，通过尝试以某种方式预测接下来会发生什么，这是我们吸收知识的方式。然后我们也通过试错来学习。我们做某事，然后想‘哦，那没成功’或‘哦，这太棒了’。所以我们通过强化学习来学习技能。而且我认为也有一些技能学习，非常广泛但粗略的技能学习，也来自观察。那只是泛化，因为当我们观察别人做事时，我们会预测他们会做什么。我们对别人的心理概念或表征与我们对自己的表征相差不远。所以然后我们开始能够想象自己也做那些事情，仅仅通过在一个系统上训练所有数据并让它泛化。

I think certainly learning knowledge from examples rather than rewards is a lot easier. For reinforcement learning, you have to first guess the piece of knowledge and then the model will be told whether the guess was correct, so it's a lot less efficient for absorbing information. And then reinforcement learning is really good at refining your strategies because it's hard to get optimal control data in any way. Let's say you have an LLM agent. It's actually really hard to get a dataset of what is the optimal next step to do right now. Good luck. Humans are not anywhere close to optimal. And if you collect with contractors, you probably have to discard like 99% of your data. Optimality depends on your horizon as well. Ideally, you want it to be optimal for a really long horizon. So it's very hard to get data for that. That's where reinforcement learning really shines. Plus, once the model is already pre-trained, it has learned some decent behaviors. There's a lot less to refine at that point. It's really just doing a bit more of these things that work well and a bit less of these things that are not so efficient. I think it's the same for humans. We definitely learn things from observation by just trying to predict what's going to happen next in some way, and that's our way of absorbing knowledge. And then we also learn from trial and error. We do something and it's like 'oh that didn't work out' or 'oh this was really great'. And so we learn our skills through reinforcement learning. And I think there's a little bit of skill learning, sort of very broad but sloppy skill learning that happens from observation as well. That's just generalization because we predict what other people will do when we watch them do something. And our mental concepts or representations of other people are not that far from the representations of ourselves. So then we start to be able to envision ourselves doing those things as well, just by nature of training a system on all that data and having it generalize.

还有一个非常有趣的额外机会，当你同时进行这两者，而不是先学习预测再学习强化学习时，它会改变你预测的世界分布。这里有一个类比：某个特定领域的专家，我们称之为目标具身领域，会注意到各种细节，而一个不从事该领域的人甚至永远不会注意到。

Also a very interesting additional opportunity when you are doing both of these simultaneously as opposed to learning prediction first and learning reinforcement learning later, in that it changes the world distribution that you're predicting. And there's an analogy there: an expert in a particular area, let's call it a goal-embodied area, notices all sorts of details that a person who doesn't work in that area would never even pay attention to.

我们还没有看到这在语言模型领域被充分利用的机会。

We haven't seen that as being an opportunity to take advantage of that in the language model space very much.

是的。这基本上就是智能体找到自己环境生态位的想法。你会收敛到一个分布，在这个分布中你可以合理地实现目标，也可以合理地预测正在发生的事情。你的系统不太可能失败，对吧？因为它正是在那些数据上训练出来的，并且它也将所有模型容量分配给了那个相关分布。所以它可能会开始忘记其他东西。另一种构建更强大系统的方法是让它变得更大，并在更多数据上训练。这样你就在扩大生态位，但总会有一些角落，在数据分布的边缘，模型会有点失败。我认为这就是我们目前在大语言模型上看到的情况，对吧？它们相当通用，在大多数分布内的事情上表现很好。但也有一些边缘，分布的边界，在那里它们会出错，可能错误泛化或产生幻觉。而做一些在线强化学习会帮助改进系统，如果它产生幻觉，就会得到负面奖励，因为用户对回答不满意。然后要么它学会正确答案是什么，要么它学会自己不知道答案，最好直接说“我不知道”，从而落在一个非常稳固的分布中。

Yeah. It's basically this idea of agents finding their environmental niches. You converge to a distribution where you can achieve your goals reasonably and you can predict what's happening reasonably. Your system is less likely to fail, right? Because it's practiced exactly on that data and it's also allocating all its model capacity to that relevant distribution. So it can start to forget other things. Another way to build a more powerful system is to just make it bigger and train it on more data. And so you're growing the niche, but then there'll still be sort of corners where, at the edges of that data distribution, the model will fail a bit. And I think that's what we're seeing with language models at the moment, right? They're pretty general and pretty good at most of the in-distribution stuff. But then there are these edges, the boundary of the distribution, where they get tripped up and misgeneralize and hallucinate maybe. And doing some sort of online reinforcement learning would help refine the system in a way where if it hallucinates, it'll get a negative reward because the user wasn't happy with the response. And so then either it learns what the right answer would be, or it learns that it doesn't know the answer and it's better off just saying 'I don't know' and ending up in a very solid distribution.

我也在想，正如你之前提到的，我的理解是，你说我们有性能很好的系统，我们会不断改进它们，我们会拥有越来越强大的 AI 系统。我不知道是否称之为 AGI，因为关于什么是 AGI 似乎有很多争论，但这似乎仍然会非常非常强大。我们可能会有这种持续的学习进步，因为人们正在研究它，在某个时候可能会有一些新的有趣发展。你如何设想未来十年左右的情况？我不知道推测事情会如何发展是否有意义。我们会继续改进这些当前的 AI 系统吗？如果出现像新的有趣算法这样的东西，它会自然融合在一起，还是可能会走向非常不同的方向？

I'm also wondering, as you mentioned earlier, the way I understand this is going, you said that okay we have very performing systems and we will keep improving on them and we will have more and more powerful AI systems. I don't know if we call it AGI because it seems like there's lots of debate about what is AGI, but it seems like this will still be very, very powerful. We might have this continual learning progress because people are working on it, and at some point maybe there will be some new interesting developments. How do you envision this future in the next maybe 10 years? I don't know if it makes sense to speculate how you think things will unfold. Will we continue to keep improving these current AI systems, and if something comes up like a new interesting algorithm, it will naturally converge together, or it might take a very different direction?

我认为这个领域联系太紧密了，事情不会真正分化。我只看到事情越来越融合，因为训练大模型的成本很高。只进行一次训练，得到一个模型，就能获得所有交叉收益，共享一个算力检查点、一次训练，这才有意义。你知道，智能体已经成为部署前沿模型的一部分。视频生成目前大多还是独立的，但如果一年后我们有强大的全能模型，能处理所有方向，并且用相同的权重生成视频，我也不会惊讶。当然有一些挑战，但从更大的图景来看，我认为这些都是可以解决的。而且我认为这种学习推理在概念上相当具有挑战性。我们并没有很好的方法去做。所以我认为这可能需要 5 到 10 年。机器人技术可能会取得良好进展，甚至达到一个有用的通用机器人产品，第一个版本可能在三年、四年或五年左右。所以我认为我们不一定非要攻克这种非常复杂的长期推理才能在那里做出有用的东西。是的，我想这就是一些预测。

I think the field is way too connected for things to really diverge. I just see things coming together more and more with the expense of training big models. It only makes sense to have one training run and get one model that can have all this cross benefit and share one compute checkpoint, one training run. You know, agents are already becoming part of deployed frontier models. Video generation is mostly separate still, but I wouldn't be surprised if maybe in a year from now we had strong omni models that go in all directions and also generate video with the same weights. Of course there are some challenges, but yeah, in terms of bigger picture, I think that's all solvable. And I think this learning to reason seems like pretty challenging conceptually. We don't really have a great idea for how to do it. So I could see that taking more like 5 to 10 years maybe. Robotics could make good progress even towards a useful sort of general robot product, the first version of that in maybe three, maybe four or five years or so. So I don't think we necessarily have to crack this really complex long-term reasoning to make something useful there. Yeah, I guess that's a couple predictions there.