从星际争霸到 Gemini：AI 智能体的进化

欢迎来到 Google DeepMind 播客。我是 Hannah Fry 教授。没错，智能体：它们已经来了，或者说快来了，而且很可能 2025 年所有人都会谈论它们。但它们绝对不新鲜。我今天的嘉宾上次来播客是在 2019 年，当时他跟我聊他正在做的多智能体系统。那个系统能在《星际争霸》中击败职业选手，最终达到了宗师级别。但自那以后，智能体是如何演变的？它们现在能做什么？语言模型和多模态 AI 的进步如何改变了局面？你又如何构建一个能代表用户自主决策的系统？现在，我得告诉你，如果你想了解智能体的入门知识，可以看我们夏天录制的与 Frederick Best 的那期节目。但今天，Oriol Vinyals 是 Drastic Research 的副总裁兼 Gemini 的 CTO 负责人。可以说我们有很多要聊的。Oriol，欢迎回到播客。

Welcome to Google DeepMind to the podcast. I'm Professor Hannah Fry. Right, agents: they are here, or almost here, and they are probably all anyone is going to be talking about in 2025. But they are definitely not new. My guest on today's episode is someone who last came on the podcast in 2019 to talk to me about the multi-agent system he was working on. It could beat the professional StarCraft players at their own game and eventually went on to achieve Grandmaster status. But how have agents evolved since then? What can they do now? How have the advances in language models and multimodal AI changed things? And how do you possibly go about building something that can make autonomous decisions on behalf of its user? Now, I should tell you, if you want a primer on agents, you can watch our episode with Frederick Best that we recorded over the summer. But for now, Oriol Vinyals is Vice President of Drastic Research and CTO lead of Gemini. And it's fair to say we've got quite a lot to catch up on. Oriol, welcome back to the podcast.

嗨，谢谢邀请我。

Hi, thank you for having me.

什么是 Drastic Research？

What is Drastic Research?

嗯，我一直告诉我的团队，他们必须进行 drastic 思考，意思是不要只做大家都在想的渐进式事情。要大胆思考几年后会发生什么，然后尝试回溯这些想法，并以这种心态在今天执行。这就是 drastic 的意思。是的，这是我经常用的一个词。

Well, I keep telling my team they have to think drastic, meaning don't just do the incremental stuff that everyone is thinking about. Try to drastically think what will happen in a few years' time, and then try to backport those ideas and then execute today with that mindset in mind. So that's what drastic means. But yeah, it's a word that I use a lot.

我想上次见到你时，你一直在做一个能用键盘和鼠标做事的智能体，比如在画图里画画或玩《星际争霸》。嗯，从那以后事情变化很大。所以当时的那些智能体，你采用了一套非常通用的原则，机器学习领域非常简单的原则，然后基本上把一个模型专门用于一个任务。我们当时做的是有一个越来越难的任务课程，对吧？比如我们上次聊的时候，在电子游戏方面，我们看的是《星际争霸》，这是最复杂的现代策略游戏之一。当然，DeepMind 以从 Atari 开始这一趋势而闻名，Atari 是一个相当简单的游戏，左、右、击球拍、击球，然后你就开始了。所以算法本身——我们试图让它们非常通用，这样我们就可以不断攀登这个难度阶梯，游戏课程，做越来越复杂的事情。而现在，我们训练的模型甚至比我们当时开发的模型适用范围更广，对吧？所以想想创建这个数字大脑的过程并没有太大变化，但那个大脑能做的事情相当狭窄，尽管非常复杂，比如玩《星际争霸》或围棋。现在，这些模型可以做更多广泛的应用，当然还有与我们对话、聊天机器人等等。

I think when I last got to see you, you had been working on an agent that could use a keyboard and a mouse to do things like draw pictures in Paint or play StarCraft. And well, things have moved on quite a bit since then. So those agents at the time, you took a very generic set of principles, very simple principles in the field of machine learning, and you would basically specialize a model on one task. And what we were doing at the time is have a curriculum of tasks that were more and more and more difficult, right? So when we last spoke, for instance, in video games we were looking at StarCraft, which is one of the most complex modern strategy games out there. And of course, DeepMind is notorious for having started the trend with Atari, which is a fairly simple game of left, right, hit the paddle and hit the ball, and off you go. So that sort of is what the algorithms themselves — we try to push for them to be very general, so we can keep climbing this ladder of difficulty, curriculum of games, and doing more and more complex things. And right now, what has happened is even the models we train are broadly applicable to many more things than the models we developed back then were, right? So think about the process of creating this digital brain hasn't changed that much, but what that brain was able to do was reasonably narrow, although very complex, like playing StarCraft or playing Go. Right now, these models can do quite a lot more broad applications, and of course talking to us, chatbots, etc., etc.

所以当时，强化学习是你的主要手段，我猜。现在情况有多不同？

So back then, reinforcement learning was your main kind of lever, I guess. How different are things now?

是的，所以从算法上讲，实际上 AlphaGo 和 AlphaStar 的过程——它们应用了相同的算法序列来创建这个数字大脑，而且这与当前大型语言模型或多模态模型的创建方式并没有太大不同。有两个基本步骤多年来在我们参与的许多项目中一直相当稳定，我们可以称之为第一步：预训练或模仿学习。也就是说，你从随机权重开始，有一个算法会尝试模仿人类创建的大量数据，无论是玩游戏还是，在这种情况下，整个互联网，我们可用的所有知识。在第一阶段，你只是调整权重以尽可能好地模仿那些数据。这些权重本质上在每个神经元内部——就像一系列数字，描述了它如何与其他一切连接。

Yeah, so algorithmically, actually the process of AlphaGo and actually AlphaStar — those two had the same set of sequence of algorithms applied to creating this digital brain, and it's not actually that different from how current large language models or multimodal models are created today. There are two basic steps that have been pretty constant throughout many years in many of the projects that we've worked on, which we can call the first one pre-training or imitation learning. That is, you start with random weights, you have an algorithm that will try to imitate lots of data that humans have created to either play a game or, in this case, all of the internet, all of the knowledge available to us. And in that first stage, you just adapt the weights to try to imitate that data as well as possible. And these weights are essentially inside each of the neurons — it's like a series of numbers that kind of describes how it's connected to everything else.

是的，所以这些基本上是作为计算单元的神经元，神经元之间的连接就是权重。你可以想象有一个神经元，有几个神经元连接到它，你基本上是把所有传入神经元的激活值乘以权重后加起来，这些权重是唯一变化的东西，输入会激发神经元。这基本上就是大脑的工作方式，有一些创造性的自由度。如果我们打个比方，就好像你有神经元，水在流过它们，权重就像神经元之间管道的宽度。

Yeah, so these are basically units of computation that are neurons, and the connections between neurons are what you have as weights. So you can imagine that there's a neuron, there's a few neurons connected to it, and you're basically adding all the activations from the incoming neurons there multiplied by the weights, and those weights are the only things that move, and the inputs excite the neurons. It's pretty much how a brain works, with some freedom of creativity, okay. If we were to do an analogy, it's almost like you've got the neurons and you're like water is flowing through it, and the weight is like the width of the pipes between the neurons.

是的，没错。然后你可以想象有数百万个神经元和数十亿甚至数万亿条管道，这就是我们花费大部分算力训练这些模型的地方，尤其是语言模型，就是在这个预训练或模仿我们所有可用数据的阶段。

Yeah, that's right. And then you can imagine having millions of neurons and billions or even trillions of pipes, and that is what we spend most compute actually training these models, especially language models, is in this pre-training or imitating all the data that we have available to us.

好的，所以你现在有了这个巨大的网络，神经元之间有很多管道，模仿阶段完成了。下一步，如果你做的是 AlphaGo、AlphaZero，你会让它自我对弈。

Okay, so you've now got this gigantic network with loads of pipes going between all the neurons, and that's your imitation phase done. Next bit, if you were doing say AlphaGo, AlphaZero, you would then get it to play itself.

是的，所以这个模型现在相当擅长做出看起来像人类的动作。这意味着，当然，句子在英语中非常合理，或者如果它在玩游戏，它会合理地点击东西来移动棋盘上的棋子等等。但这个模型还没有学会的是这些动作会产生奖励，对吧？这就是强化学习或后训练的部分，这是训练的第二阶段。所以你可以通过——嘿，互联网上的诗平均看起来怎么样？来写一首诗。但问题是，我只想要好诗，对吧？那么我如何根据一个信号进一步调整这些管道呢？这个信号是，现在写完一首诗后，会给出一个分数，比如 0 或 1？如果是一首平庸的诗，你得 0 分；如果是一首好诗，你得 1 分。再次，用游戏类比，这是我们传统上使用强化学习的方式：如果你赢了游戏，你得 1 分；如果你输了，你得 0 分。然后你进一步调整权重，但现在不是模仿人类，你只是说，忘掉，我想超越人类能做的，真正让我所有的诗都成为完美的诗，对吧？或者我所有的棋局都成为完美的棋局。在语言模型中，这第二阶段，即强化学习后训练，往往相当短暂，因为我们没有像在传统棋盘游戏中自我对弈时那样清晰的奖励信号，比如你知道你赢了或输了。

Yeah, so this model now is reasonably good at playing moves that look humanlike. So that means, of course, the sentences are very plausible sentences in English, or if it was playing a game, it would sort of click things reasonably to move pieces on the board and whatnot. But what this model hasn't done is learn that these actions yield reward, right? That's the bit of reinforcement learning or post-training, which is the second phase of training. So you can write a poem by just — hey, just how does a poem on the internet look like on average? But then the question is, well, I want only the good poems, right? So how can I further adjust these pipes based on a signal that now having written a whole poem would give a score of zero or one, let's say? If it's a mediocre poem, you get a zero; if it's a good poem, you get a one. Again, for a game analogy, which is what we use reinforcement learning traditionally: if you win at the game, you get a one; if you lose, you get a zero. And then you further adjust the weights, but now instead of imitating humans, you're just saying, forget, I want to go beyond what humans could do and try to really get all my poems to be the perfect poem, right? Or all my chess games to be the perfect game. And in language models, this second phase, which is reinforcement learning post-training, tends to be fairly short-lived because we do not have access to super clean reward as you know, you've won the game or you lost the game when you do self-play in traditional board games, for example.

所以一旦完成了，对吧，那就是幕后发生的所有事情，然后你就说，停在那里，对，保持。

So once that's done, right, then that's all the stuff that goes on behind the scenes, and then you're like, hold it right there, yeah, stay.

正是你所在的位置，各位。我们基本上要对整个网络拍一张快照，而这就是你作为用户实际能访问到的。

Exactly where you are, everybody. We're going to take just basically a snapshot of this entire network, and that is what you actually get to access as a user.

是的。现在这个惊人的过程结束了。这些权重非常宝贵，对吧？你找到的这个配置，你真的花了几个月来完善它，调整一切，然后你基本上再也不会改动它了，对吧？所以训练结束了。你不再改变配置。你可能想让它变得非常高效，对吧？比如你发现，哦，看，这个神经元没那么有用，它没被用于任何事，你就移除它，这样大规模运行时一切都更快更便宜。然后作为用户，你得到的就是同样的权重。每个人都得到我们训练好的相同权重。这就是我们所说的，比如，Gemini 1.5 Flash。那只是一组冻结的权重，不会改变，不会进一步训练或任何操作。

Yeah. So now this amazing process finished. These weights are super precious, right? This configuration you found, you've really spent months to finesse it, to tweak everything, and now you sort of will never move it anymore, right? So training is over. You're not changing the configuration anymore. You might want to make it super efficient, right? So say like you find that, oh, look, this neuron is not that useful, it's not used for anything, you remove it, so everything becomes faster and cheaper to run at scale. And then as a user, you just get the same weights. Everyone gets the same weights we've trained. That's what we call, like, you know, Gemini 1.5 Flash. That just means a set of weights that are frozen, will not change, will not further train or anything.

所以这两个步骤实际上从 AlphaGo 到 AlphaStar 再到当前的大语言模型几乎是一样的。当然细节很重要，领域也确实在发展，但原则实际上基本没变。

So those two steps actually pretty much are identical from AlphaGo to AlphaStar to like current large language models. And of course there's details that matter and the field has evolved certainly, but the principle is pretty much unchanged actually.

因为在底层，可以说，存在差异，比如，我不知道，我在想 DQN，那是 Atari 的例子，或者 AlphaGo 中使用的算法类型，或者再到大语言模型。架构不同，对吧？

Because under the hood, as it were, there are differences between, I don't know, I'm thinking like DQN here, which was the Atari example, or the types of algorithms that were used in AlphaGo, or then again in the large language models. Like the architecture is different, right?

是的。

Yeah.

所以构成数字大脑有几个组件。一个是架构，对吧？所以有这些神经网络。现在我们有了 Transformer，这在 DQN 时代当然是没有的。所以总有一些架构上的突破，能更好地从数据中学习。但从 Transformer 到今天，几乎全是小调整。我的意思是，即使你看 AlphaFold，它也是由 Transformer 驱动的，那个团队有时花数年时间就是找小调整：嘿，我们移除这组神经元，我们再加一层，把这个弄宽一点。所以你塑造大脑，形状稍有变化，有时这就决定了性能的成败。

So there's a few components that go into what the digital brain is. One is the architecture, right? So there are these neural networks. Now we have the Transformers, which we certainly didn't have back in the DQN days. So there's always some sort of breakthroughs in architectures that are better at learning from the data. But then from Transformers to today, it's almost all about little tweaks. I mean, even if you look at AlphaFold, which also is fed by a Transformer, what that team does for years sometimes is just to find little tweaks: hey, let's remove this set of neurons, let's add another layer, let's make this a bit wider. So you shape the brain, shape changes a little bit, and that makes it or breaks it sometimes in terms of the performance achieved.

那么如果这些是迄今为止取得的所有成就，我的理解是目标是创造更智能体式的行为，让这些东西做出自主决策。这些如何帮助实现那个目标？

So if these are all the things that have been achieved so far, I mean the goal as I understand it is to create more agentic behavior, to kind of get these things to make autonomous decisions. How did these help to achieve that end?

是的，让我们稍微聚焦一下当前的趋势。我们称之为大语言模型，但它们是多模态的。我想我们之前有一期节目深入讨论了多模态方面，能够添加图像、然后提问、追问等等有多好。所以这个核心，我们还会改进它，对吧？这组权重对输入做出这些惊人的推理：这张图是关于什么的，用户在问什么，我能不能写一首更好的诗，我能不能让它更长，等等。所有这些交互我们这些天都在玩。但这只是一个组件，我们可以想：嘿，这现在是我们的 CPU，我们可以在它周围添加更多东西。如果模型能为你去做研究呢？比如，对，我们以前就在想的一个例子：我可以让一个模型，语言模型或视觉语言模型，学习玩星际争霸游戏。这与创建一个玩游戏的智能体是非常不同的方法。在这个另一个例子中，对吧，它可以上网，看关于游戏的视频，它当然可以下载游戏开始与之交互，学习，哦，是的，我知道怎么做了，我懂了。在线做研究，去论坛，读论坛，去玩并发现自己在某方面弱然后改进，等等。然后实际上可能过了几周，它给你发一封邮件：它说，我现在知道怎么玩这个游戏了，我们玩吧。对吧？那并不是一个遥不可及的现实。但这些模型突然之间确实做了一些事情，采取了一些行动，并学习了任何可用的新东西。这想起来很强大。这是最推动通用性的东西，这让很多人所说的 AGI 感觉更近了。

Yeah, so let's zoom in a little bit on the current trend. We call it large language models, but they're multimodal. I think we had an episode earlier covering heavily the multimodality aspect, how good it is to be able to add an image, then ask something, a follow-up question, and so on. So this core, we will still improve it, right? These set of weights that do these amazing sort of inferences about the input: what's this image about, what's the user asking, can I write a better poem, can I just make it longer, whatever. All these interactions we all kind of get to play with these days. But this is just a component, and we can think: hey, this is now our CPU, and we can add more to it around it. What if the model could go off and do research for you? Could, for example, right, one example we were already thinking about that back in the day: I could ask a model, a language model or visual language model, to learn to play the game of StarCraft. That's a very different approach to, say, create one agent that does play the game. In this other example, right, it could go online, watch videos about the game, it could of course download the game to start interacting with it, to learn, oh yeah, like I know how to, you know, I get it. Do research online, go to forums, read the forums, go play and figure out that it's weak at this thing and improve, and so on. And after literally it could be weeks, it sends you an email: it says, I now know how to play the game, let's play. Right? That's not a reality that's that far away. But these models all of a sudden actually do something, take some actions, and learn anything new that is available to them. And that's pretty powerful to think about. It's what pushes the generality the most, and that's what makes kind of the AGI, as many people call it, feel closer.

所以如果我理解正确的话，那几乎就像我们目前拥有的东西，大语言模型、多模态模型，不管你怎么称呼它们，就像是中央核心。但下一步是在那个中央核心之上构建东西，让它能脱离，你知道，去掉稳定器，然后去做自己的事。

So if I understand it correctly, then it's almost like the stuff that we have at the moment, the large language models, the multimodal models, whatever you want to call them, that's like kind of the central core. But the next step is that you build stuff on top of that central core that it can go off and, you know, take off the stabilizers and kind of go off and do its own thing.

是的，完全正确。如果它能获取所有知识，并且能利用它的时间做一些适当的研究，我的意思是，写假设、写代码等等，并花时间真正回答非常非常复杂的问题，那么可能性就大大拓宽了。当然我们不需要对所有事情都这样。我的意思是，如果我们问一个问题，比如，嘿，你知道吗，我喜欢米饭，我今晚该准备什么？可能不需要深入思考或者花三周时间。那样你大概不会对等待时间很满意，对吧？但我认为那是为了推动前沿。你给计算机一个数字身体，这样它不仅能思考并给你指令或文字输出，还能去做事情，在线或在你上传的文档上等等，并问非常复杂的问题，为你个性化等等。

Yeah, exactly. If it has access to all the knowledge and it can sort of use its time to do some proper research, I mean, write hypotheses, write some code, and so on, and take its time to really answer very, very, very complex questions, then the possibilities have now broadened quite drastically. Although of course we're not going to need that for everything. I mean, if we ask a question like, hey, you know what, I like rice, I mean what should I prepare tonight? Probably no need to do a very deep dive into thinking or just going off for three weeks. Then you'll probably like not be very happy about the waiting time, right? But I think that's kind of for to push the frontier. You're giving a digital body to the computer, so it can not only just think and give you an instruction or a word output, but it can also go off and do things online or on documents that you might upload or whatever, and ask very, very complex questions and personalize to you, etc., etc.

我喜欢这个想法：这个中央核心，然后你给它一个数字身体。你有了电子大脑，现在你给它一个数字身体。这还不错。那么关于电子大脑，关于这个核心，这个处理器，嗯，让我问你一点。我想我们应该考虑 Gemini，对吧，这基本上就是我们正在讨论的，你们拥有的多模态模型。我知道大模型的一个大想法就是 Scaling（规模扩张），对吧，让它们变得越来越大、越来越大、越来越大。你认为我们从 Scaling（规模扩张）中看到的结果现在已经趋于平稳了吗？

I like the idea: this central core and then you're giving it a digital body. You've got the electric brain and now you're giving a digital body. It's kind of okay. So in terms of the electric brain, then, in terms of this core, this processor, um, let me just ask you a bit about that. I guess we should be thinking about Gemini here, right, which is essentially what we're talking about, the multimodal model that you guys have. I know that one of the big ideas for large models was just to scale it up, right, to get them bigger and bigger and bigger and bigger and bigger. Do you think that the results that we've seen from scaling have sort of plateaued by now?

是的，这是一个非常重要的问题，对吧？我们研究过，当你让模型更大时，也就是这些模型有多少神经元，它们在那些我们有明确指标的任务上变得更好，这些指标来自整个机器学习社区。例如，一个非常容易理解的是机器翻译，对吧？所以模型在两种语言之间翻译得有多好。随着你 Scaling（规模扩张），从百万到十亿再到可能万亿个神经元，你可以看到性能持续提升。现在，即使你做这些研究，一个技巧是它看起来是线性的，但你必须绘制……

Yeah, it's a very important question, right? Like we have studied how, as you make the models larger, that is how many neurons these models have, how do they become better at certain tasks that we have clear metrics from the whole machine learning community. For example, one that is very simple to understand is machine translation, right? So how good the models are at translating between two languages. As you scale, as you go from millions to billions to potentially trillions of neurons, you can see the performance keep improving. Now, even when you do those studies, one trick is that it looks linear, but you have to plot...

对数轴，用通俗的话说就是，假设过去三年我们取得了一些进步，你不应该期望未来三年有同样的进步，实际上要到达那里是呈指数级困难的。所以这意味着投资当然也以超线性速度增长，但可能不如这个趋势所暗示的那么好，你会看到收益递减，因为仅仅缩放 x 轴，你需要将参数数量增加 10 倍才能看到同样的改进，这就造成了一些压力：也许我们不能无限制地扩展，我们需要考虑其他方式来让模型变得更好。我给学生举的例子是，如果你有一个非常乱的房间，你花前 10 分钟整理，效果会非常显著——你捡起所有脏盘子，收好所有脏衣服。但当你已经整理了七个小时后，再多花 10 分钟根本不会有什么不同。这基本上就是我们现在的处境，对吧？

Logarithmic axis right what that means in layman's terms is that let's say from the last three years we had some improvement you shouldn't expect the same improvement in the next three years it's actually exponentially hard to get there right so that means the investment which of course also advances at a super linear rate but perhaps not as good as this sort of trend suggests you would just see some diminishing returns because simply scaling the x-axis right the number of parameters you need to go 10x to see the same improvement and that just creates some pressure to hey maybe we can't scale as much and we need to think about other ways to scale to make the models better. The example I give to my students is like if you've got a room that's really messy the first 10 minutes that you spend tidying it's going to make a massive difference you pick up all the dirty plates put away all the dirty washing fine but once you're like seven hours in that 10 minutes extra 10 minutes is it's not going to make any difference at all and that's essentially where we are right?

是的，这确实是一个非常好的类比。事实上，这个类比甚至可以应用到模型的性能上。即使你有非常好的性能，如果你希望这些模型 100% 准确，永远不会编造东西，我们知道如果你仔细探究，还是能让它们说出不真实的东西。即使是最后一英里也非常困难，这给大规模部署带来了一些有趣的挑战。

Yeah, that's exactly a very good analogy. And in fact, that analogy can even apply to the performance of the models. Even if you have extremely good performance, if you want these models to be 100% factual, right, will never make something up, we know that if you probe them you can make them say things that are not real. That even that last mile is also super hard, which creates some interesting challenges to deploy them at scale.

所以我理解你说的所有这些都存在收益递减。但就如何让这些东西变得更好，如何让这些模型变得更好而言，仅仅是数据、计算能力和规模吗？这些是唯一可以拉动的杠杆吗？

So I hear what you're saying about how there's diminishing returns in all of this, right. But in terms of how you make these things better, how you make these models better, is it just data, computational power, and size? Are those the only levers that you have to pull?

是的，所以如果你冻结架构，比如说未来一年没有创新，我们只是扩展，因为更好的硬件出现了，让模型更大，那肯定会有一个看起来还不错的趋势。但实际情况是，在 Gemini 中，我们还有其他创新，比如其他技巧、技术、关于如何排列呈现给模型的数据的细节、架构的细节、如何运行训练过程、运行多长时间、我们实际向模型提供什么样的数据、如何过滤、是提供更多高质量数据还是更少低质量数据，以及各种不同的所谓超参数。当然，我们也会相当仔细地研究其他算法进步，因为训练模型的过程很昂贵，所以我们需要非常小心地积累创新，这样当我们准备好时，我们有足够的创新，而且可能也有更好的规模来运行下一轮模型。我们运行它，然后我们获得算法突破，而不仅仅是通过数据和算力。

Yeah, so certainly if you froze the architecture, let's say for the next year, no innovation, we just scale because there's better hardware coming up, make it bigger, that certainly would have a trend that would look okay. But what's happened, and certainly in Gemini we have other innovations like other tricks, techniques, details about how to order the data that you present the model with, to the details of the architecture, to how to run the training process, how long to run it for, what kind of data do we actually present the model, how do we filter, do we present more data that's high quality, less data that's low quality, all sorts of different what we call hyperparameters. And of course other algorithmic advances we also investigate fairly carefully because the process of training a model is expensive, so we need to be extremely careful with piling up innovation so that eventually when we are ready, we have enough innovation and also probably we have a better scale to run for the next iteration of models. We run it and then we get algorithmic breakthroughs, not only through data and compute.

我想关于 Scaling 的另一件事是，你可以放入的节点数量实际上没有限制，也许理论上你投入的计算能力没有限制，但你能投入的数据是有限的。人类词汇的数量是有限的。

I guess the other thing about this scaling stuff is that you've got, there's no limit really to the number of nodes that you can put in, maybe there's sort of no limit in theory to the computational power that you put in, but there is a limit to the data that you can put. There's a limit to the number of human words that are out there.

好观点。所以我认为节点是有限制的，因为如何扩展这些模型，它们不能放在单个芯片上，所以现在你有一个芯片网格，它们之间通信，存在一些限制，比如光速等。所以开始出现一个时刻，训练如此大的模型的效率甚至从你手头硬件的利用率来看也不值得。但非常好的观点，没错。另一个关键点是，在预训练模仿所有数据时，我们没有所谓的无限数据 regime，数据是有限的。所以一旦模型需要，你可以想，好吧，我们可以在所有数据上训练，如果你想的话，人类读过的一切，互联网上的一切。所以我们开始思考，好吧，我们快没数据了。有一些技术，比如合成数据，我们能否以多种不同方式编写或重写现有数据？语言显然是一种思路，嘿，你可以重写互联网，它主要是英语，大概 60%，我不知道确切百分比，但有办法以不同方式重写相同的知识。我们正在探索这些，这是一个很多人开始投资的研究领域，因为如果你用完了数据，缩放定律会惩罚你更多。

Good point. So I think there is a limit on the nodes because how you scale these models is well they don't fit on one single chip, hardware chip, so now you have a mesh of chips they're communicating, there's certain limits like speed of light etc. So there starts to be a time where the efficiency of training such a big model also just not worth it even from a utilization of the hardware at your disposal. But very good point, right. The other bit that is critical on this pre-training imitate all the data is that we do not have what we call infinite data regime, there's finite data. And so as soon as the models need to, you can think well let's train on all the data if you want to, everything humans have ever read, everything all of the internet. So we're just starting to think okay we're running out of data. There are techniques like synthetic data, can we write or rewrite existing data in many different ways? I mean languages would be obvious ways to think, hey you could rewrite the internet, I mean it's mostly in English, I mean 60%, I don't know what's the exact percentage, but there are ways to rewrite the same knowledge in different ways. We're exploring those, that's kind of a research area that many people are starting to invest in because if you run out of data, the scaling laws punish you even more.

那么例如，你可以让 Gemini 编写自己的互联网版本，然后用它来训练新版本的 Gemini。是的。但是否存在一种危险，如果你开始输入同一个模型的输出，你最终可能会创建这些无益的反馈循环？

So for example then you could get Gemini to write its own version of the internet and then use that to train a new version of Gemini. Yes. Is there a danger though that if you start feeding in the output of the same model that you can end up creating these unhelpful feedback loops?

它们当然可以。你知道，我们做一些有趣的实验来测试像你刚才提到的想法。确实，从表面上看，这不是一个好主意。比如，如果你只是要求模型重新创建整个互联网，模型会受到影响。而且从信息内容的角度来看，这个数据集拥有它所拥有的信息，你怎么能创造新信息呢？对吧。我不知道，这些想法可能有一点帮助，因为机器学习存在缺陷，还没有从互联网中真正提取所有信息的基本能力。我的意思是，我们有好的算法，但它们并不完美，所以我们拭目以待。

They certainly can. You know, we do some interesting experiments to test ideas like this one you just mentioned. And indeed that is not, on the surface, a good idea. Like the model suffers if you just ask it to recreate all of the internet. And indeed a priori from an information content point of view, look, this dataset has the information that it has, how could you create new information? Right. I don't know, these ideas might help a little bit because there are machine learning deficiencies that are not at that fundamental ability to extract all the information truly from the internet. I mean we have good algorithms but they're not perfect, so we'll see.

是的，我想再多想一下，因为这真的是一个有趣的想法。因为当然，如果你不加思考地去做，那么新版本会带有偏见，然后在此基础上更新的版本会更加偏见，最终你会逐渐偏离原始的人类数据。但你说的是，在原始的人类互联网中嵌入了这些概念联系，如果你能提取它们，我几乎在想就像 E=mc^2，如果你能为人类概念找到类似 E=mc^2 的东西，然后仅用它生成新数据，那似乎更现实。

Yeah, I mean I guess I just want to think about that a little bit more because it's really interesting idea. Because of course naively if you did it without thinking then it's like the new version would have the biases in it, and then the new version on top of that would be more biased and you'd end up sort of spiraling away from the original human one. But then what you're saying is as though in the original human internet there are embedded these conceptual connections and if you can extract those, I'm sort of thinking almost like E=mc^2, right, if you can sort of find the E=mc^2 for human concepts and then just generate new data using that alone, then that seems more realistic.

是的，完全正确。我认为这就是你开始触及的问题，我的意思是，这些语言模型只是在重复网上的内容，无法创造任何新东西，还是它们真正在学习一个世界模型，然后你可以从它提取的原理中可能泛化到数据之外？在更乐观的版本中，我更倾向于相信，我们可以将数据的极限推得比当前极限更远一些。话虽如此，我的意思是，有一些数据源我们还没有看到突破，比如视频数据。有很多视频数据，我们还没有看到全面利用的时刻。

Yeah, exactly right. And that's I think that's where you start hitting, I mean are these language models just repeating what's online and not being able to create anything new, or are they learning a world model truly that you can then from the principles it extracts possibly generalize beyond what the data has? And under the more optimistic version, which I tend to believe more, we can push the limits of data a little bit more than the current limits that we have. That being said, I mean there are some data sources that we haven't quite seen a breakthrough, like video data. There's a lot of it and we haven't quite seen a moment of take all.

视频数据中你可能能提取大量知识，很多物理知识，很多世界运作的方式，即使视频本身没有关联文字，也能提取那些知识。我认为我们甚至还没有利用那个来源。但事情并不是那样运作的，对吧？或者你不知道。

The video data where you probably can derive a lot of knowledge, a lot of physics, a lot of how the world works, even if there are no words associated with the videos necessarily, and extract that knowledge. Even that I don't think we've tapped into that source. And it doesn't work that way, right? Or you don't know.

是的，感觉上应该是这样。即使是我们学习的方式，我们也通过观察三维空间等来学习，所以可能还有更多我们没有提取的知识。我们做得相当好，并且可以通过测试模型看到的是，连接视频中存在的概念，然后你可以做很棒的事情，比如‘把这个一小时的视频提取出三个有趣的时刻’。但模型本身并没有直接使用那些信息，很可能没有。

Yeah, I mean, it feels like it should. I mean, even how we learn, we learn by also observing three dimensions and so on, so there probably is more knowledge that we haven't extracted. What we've gotten pretty well at, and you can see by testing the models, is connecting the concepts present in the video, and then you can do amazing things like 'take this full hour video and just extract me three interesting moments.' But the model itself hasn't actually used that information directly, probably not.

我非常喜欢这个。我们和杰夫讨论过多模态模型，如果你让这些模型观看所有曾经创建的视频，它能否真正提取出重力作为一个概念的含义？但你现在描述的，如果我理解正确的话，是当前它可以告诉你视频中看到了什么，但不能因此说出‘E=mc²’，或者如果你给它看夜空的图片，它不会突然能够像人类天文学家那样预测行星运动。

I like this so much. We were talking to Jeff about with multimodal models, if you get these models to just watch all of the videos that have ever been created, can it quite literally extract what gravity means as a concept? But what you're describing here, if I understand it, is that at the moment it can tell you what in the video that it's seen, but it can't then say 'E=mc²' or if you showed it pictures of the night sky, it wouldn't suddenly be able to predict planetary motion in the same way that human astronomers did.

是的，完全正确。我的意思是，我们这里采取的捷径是，我们在图像和/或视频上训练的数据，几乎总是有与该模态相关的文本表示，比如解释这个图像或视频有什么的标题。这当然让它变得不可思议：你可以放一张家庭作业的图片，上面有一个小概念图，它会连接并基于此做很多很好的逻辑推理。但我说的是，我能不能只拿没有语言的视频，训练一个模型来理解正在发生的事情，甚至可能推导出一种语言？显然不会是我们的语言，并提取那些概念。这还没有发生，而且很可能会发生。

Yeah, exactly. I mean, the shortcut we're taking here is that the data we train on images and/or videos, we almost always have a text representation associated with that modality, like a caption explaining what this image or video has. That makes it incredible, of course: you can put a picture of a homework with a little drawing conceptual, and it will connect and do quite a lot of good logic based on that. But what I'm saying is, could I just take videos with no language and train a model to then understand what's happening, maybe even derive a language? Obviously it's not going to be our language, and extract those concepts. That has not happened, and it probably will.

回到你一开始说的，DeepMind 构建的所有模型基本上都有两个阶段：模仿阶段，就是我们一直在讨论的，然后再加上强化学习阶段。我知道 AlphaGo 和 AlphaZero 以及更多模型通过自我对弈变得更好。这在这里也适用吗？

Going back to what you said at the beginning about there being two phases to basically all the models DeepMind have built: the imitation phase, which is what we've been talking about here, but then the reinforcement learning phase on top. I know that AlphaGo and AlphaZero and many more got better by playing themselves. Does that apply here as well?

是的，这是主要的开放挑战之一：不仅扩展预训练，还要扩展后训练或强化学习。强化学习在游戏中的美妙之处在于有一套编码的规则，如果你赢了，你就知道自己赢了。有一个明确的成功指标。现在在语言中，要棘手得多：这首诗比那首好吗？即使在我们之间讨论也很难。所以通用性再次使得精确计算非常困难。我们如何量化？这是对电影的更好总结吗？这是这个视频最有趣的部分吗？很难量化，但我们可以尝试。你基于一些人类偏好粗略地训练一个模型，然后说‘好了，现在尝试泛化。’所以我让一个模型批评自己的输出；它不会做得太差，可能 80%的时间都很好，这还不错。它可以给你一些信号。但当你开始说‘现在你要攀登这个指标’时，你有了这种不完美的评估性能的方法，但我们现在要开始针对这个不完美的奖励进行训练。模型会做的是利用奖励的弱点。用国际象棋的例子：想象如果兵在某个位置你总是赢，而且这是一个没人会下的位置，所以可能没人知道它的存在。但现在你让一个算法探索一切并尝试发现如何赢得这个游戏。突然之间，你会发现‘哦，如果我把第一个兵移到这个位置，没人下这个开局，你就赢了。’算法掌握了游戏，然后一个研究员去看你怎么下棋，发现简直糟糕透顶。基本上，你在寻找漏洞，而不是真正学习一首好诗意味着什么。

Yeah, that's one of the main open challenges: scaling not only pre-training but post-training or reinforcement learning. The beauty about reinforcement learning in games is that there is a set of rules that are coded, and if you've won, you know you've won. There is a clear metric of success. Now in language, much trickier: is this a better poem than that? Good luck discussing that even amongst us. So the generality again makes computing exactness very hard. How do we quantify? Is this a better summary of the movie? Is this the most interesting bit of this video? It's very hard to quantify, but we can try. You train a model based on some human preferences roughly, and you say 'okay, try to now generalize.' So I ask a model to criticize its own output; it's not going to do that badly, it's going to be good maybe 80% of the time, which is not terrible. It can give you some signal. But at the point you start saying 'now you climb this metric,' you have this imperfect way to assess performance, but now we're going to start training against this reward that is not perfect. What the model is going to do is exploit the weaknesses of the reward. Using the chess example: imagine that if a pawn is in a certain position you always win, and it's a position that no one would ever play, so maybe no one knows this exists. But now you ask an algorithm to explore everything and try to discover how to win at this game. All of a sudden, you're going to find 'oh, if I move the first pawn just to this position, no one plays this opening, you've won the game.' The algorithm has nailed the game, and then a researcher goes and sees how you play chess and it's just terrible. Basically, you're finding exploits rather than really learning what a good poem means truthfully.

你不能只是加入另一个玩家吗？加入另一个模型作为最终的仲裁者。

Can you not just add in another player? Add in another model which is like the ultimate arbiter.

好建议，但问题是你如何训练那个模型？我们只有有限的概念，来自一些专家，我们可以请他们比较这两首诗。我们只有有限的数据来训练这些仲裁者。真实情况可能是请一位专家，如果我们能做到我们会做，但那是不可扩展的。想象一下，说‘好的，我在三秒内有一个参数更新，现在请一位专家审查这 10,000 件事，因为那是真理来源’会有多慢。我们没有足够的数据来训练一个足够好的奖励模型。所以再次，有一些想法，但问题是我们无法获得真实情况。

Good suggestion, but then the problem is how do you train that model? We have only a finite notion of what's a good poem from some experts that we might ask to compare these two poems. There's just a limited amount of data we have to train these arbiters. The ground truth might be to ask an expert, and if we could we would, but that's not scalable. Imagine how slow it would be to say 'okay, I have a parameter update in three seconds, now please review these 10,000 things by an expert because that's a source of truth.' We don't have enough data to train a good enough reward model. So again, there are some ideas, but the problem is we don't have access to the ground truth.

就像你戴着烤箱手套在黑暗中摸索。你甚至不完全确定有坚实的物体可以依靠。

It's like you're feeling around in the dark with oven gloves on. You're not even completely sure that there are solid objects to go on to.

所以如果那是核心，电子大脑，现在我们正在构建数字身体，你希望那个数字身体拥有什么样的能力？比如推理，因为在这方面也有相当多的工作，不是吗？

So if that's the core, the electronic brain, and now we're building the digital body, what kind of capabilities do you want that digital body to have? Like reasoning, for example, because there's been quite a lot of work on that too, hasn't there?

所以你开始思考，我们能够给这些模型有限访问的主要表面是什么，这样它们就能超越它们冻结的权重，收集知识或做一些比仅仅根据上下文和权重预测下一个词更复杂的事情。想到的明显事情是让它们访问搜索引擎，这是我们在谷歌做得非常好的。另一个是让它们能够运行自己编写的代码。然后，当然，也许更广泛地，可能更通用的是，让它们能够与世界互动。

So you start thinking, what are the main surfaces that we would be able to give these models limited access to, so they can see beyond what's in their weights, which are frozen, to be able to gather knowledge or do something a bit more complex than just predicting the next word from just what they have in context plus what they have in their weights. Obvious things that come to mind are giving them access to a search engine, that's what we do very well at Google. Another one is to give them the ability to run the code they write. And then, of course, maybe even more broadly, that could be more general, is giving them the ability to interact with the world.

能够与一个能访问互联网的浏览器交互，对吧？有了这些，你总是要小心地沙盒化，也就是保护这些环境，这样模型即使不那么先进，也不会做出意外的事情。所以这涉及到整个安全方面，当你超越模型本身时，这就变得很有趣。但如果我们只是梦想一下可能的事情，通过给模型提供这些工具，它们突然就能做更高级的事情，超越我们当时使用的训练语料库。它们可以依赖最新新闻来向我们解释或总结昨天发生的主要事件，诸如此类。你需要给它们这些工具。

ability to maybe interact with a browser that has access to the internet right with all of these you you always have to be careful to sandbox that just means protect prot these environments so that the models even if they're not that advanced wouldn't do something that is unintended right so there's the whole like safety aspect of this that as you move beyond the model it starts to be quite interesting but if we just in kind of dream what would be possible Right by having these tools available to the models all of a sudden they can start doing much more advanced things beyond what what the training Corpus that we used at the time right they can be more they can rely on the latest news to explain us or to summarize what was what was the main thing yesterday that happened all these kind of things you need to give them these tools

好的，那么推理如何融入这一切呢？

okay so how does reasoning fit into all of this then

是的，推理很有趣。所以我刚才描述的可以总结为：比如我想知道昨天发生了什么，我可以说，嘿，模型，我是 Oriol，我对这个感兴趣，我的政治观点是这样那样，给我一个对昨天新闻的正面看法。模型可能会搜索并检索所有新闻，然后按照我的要求去做，以我喜欢的方式呈现。如果我不喜欢，我甚至可以说我不喜欢这个，或者这个笑话不好笑，然后我们可以在对话中迭代。现在，推理是一个不同的 Scaling（规模扩张）轴。你可以想象模型决定采取哪些中间步骤来给我更好的答案。假设谷歌搜索检索出 100 个新闻来源，模型可能决定不直接阅读并一次性总结，而是先总结每篇文章。这意味着模型决定为每篇文章写一个摘要，不是写给用户，而是写给自己。然后它有了 100 个摘要，下一步可能决定按主题分组。然后它发现一篇文章看起来可疑，于是上网检查是否有论坛讨论说这篇文章可能不真实，因为作者等等。所以它可以做很多步骤来进行研究，而且可以持续一段时间。只有当模型认为现在有了更高质量的答案时，它才会给你一个简短的总结。但在这段时间里，它对可用信息做了更多处理。这种推理时算力（inference time compute），我们希望给模型的时间越多，它总结新闻、写诗、做数学的效果就越好。这当然是另一个 Scaling（规模扩张）轴，我们开始解锁它，希望它能打破预训练（pre-training）中看到的缩放定律（scaling laws）和限制。

yeah reasoning is interesting right so so what I described um could be summarized as hey like I mean I want to know what happened yesterday um then I could just say look take maybe personalize it a little bit right so I could describe inwards I could say hey hey model I'm orial I'm interested in this and that my political views are this are that give me like a positive take on the news yesterday right and the model would probably search right retrieve all the news and then given what I asked it to do just do it in a way that I like and I find it enjoyable maybe if I don't like it I can even go and then say I didn't like this or this is not a good joke and then we could iterate a little bit in a conversation now reasoning is a bit of a different axis of scaling right and so you could imagine um the model deciding what kind of intermediate steps to do to give me a better answer right so imagine there's like 100 news outlets that Google search retrieves maybe the model decides hey I'm not just not going to read this and just try to summarize it all at once I'm going to Summarize each of the 100 articles first right so that that means the model decided I'm gonna write a summary for each of the 100 Pages I'm going to write it not to the user to myself and then then now it has a 100 summaries and maybe the next step it decides to do is I'm going to group this by topics then it decides one of the Articles looks suspicious so maybe it goes online and checks if in any forums someone discusses like oh this might be like sort of not truthful because of the author and so on so forth so it can do a lot of steps to do research and you know it could do this for quite a while and only when the model says well I think now I have a much better quality answer then it will give you the you know the few word summary but now it had all this time to do much more processing on the information that was available to to it and that inference time compute we hope that the more time we give to the model the better is going to summarize the news the better is going to write a poem the better is going to of course do math but that's certainly another axis of scaling which we're starting to quite unlock um we hope to unlock and again we'll break a bit of the SC scaling LW and the limits that we see in pre-training uh pure pre-training

这是否也包括规划？比如它能查看你的日历，算出你的发薪日，知道一月促销即将到来，然后告诉你推迟几天预订或度假？我的意思是，这可能会变得非常复杂。但当然，当你考虑个性化以及何时做事情，因为还有其他事情在进行，你有更多的信息来源，你需要收集它们然后给出最佳答案。它不再是“天空是什么颜色”这种问题，那个问题回答起来也不简单。我在想那个例子，我们有一篇很早期的论文，其中那个例子展示了语言模型能做得很好——你不需要编程它们回答，但它们回答了。但实际上，如果你仔细想想，答案也很有 nuance：比如行星、一天中的时间、是否多云等等。所以思考和规划，是的，这绝对是这些模型能做的事情。

does this also include planning like uh could it look at your calendar work out um when your payday was maybe know that the January sales are coming up soon and uh tell you to to postpone booking or holiday for a few days I mean that can be get very complex but of course when you factor in things like personalization and like when to do things because of all the other things that are ongoing right you have more sources of information you need to collect them and then give the best answer and it stops being what color is the sky which is is not that simple to answer I was thinking about that example I mean we had a very early paper where we have that example as oh something that language mods can do amazing right you don't program them to answer but they answer but then actually even the answer is quite Nuance if you start thinking oh yeah you know planets and where what time of the day I mean is it cloudy or not right so so the thinking and the planning yeah it can definitely be um that's that's something that these models can do

我想起大概在 2019 年与 Demis 的一次对话，他谈到了 Kahneman 的两种思维系统：一种是快速、直觉性的，另一种是缓慢、计算性的，就像做数学和下棋那样。Demis 说，第二种传统上更容易用计算机实现，但现在我们看到了更快的直觉性东西。但你的意思是要把两者结合起来，对吧？

I I'm reminded by I had a conversation with Demis probably back in 2019 and he was talking about um the caraman tki idea is of how the human brain has almost two systems of thinking the sort of quick instinctive intuition based and then the much slower calculated sort of the way that you do maths and chess and and and Demis was saying that that kind of that that second one has been traditionally easier for us to do with computers um but that now we're seeing the much quicker instinctive stuff but but I mean you're sort of talking about putting the two together right

是的，没错。

yeah right

Demis 可能也在谈论这些系统，这确实是需要更多反思的系统。在游戏中很明显：你可以说这步棋感觉对，就走下去；但如果你思考并权衡，可能会得到更好的棋步。挑战在于，现在我们处于如此通用的方向，这些模型可以做任何事情——字面意义上的任何事情，你想做什么都行：上传图片、谈论新闻。那么，拥有这种更深层思考意味着什么？它非常领域特定，你怎么做呢？有几个答案，但我喜欢的一个是：这些模型非常通用，要在非常通用的能力集上添加思考能力，你可能需要一种通用的思考方式。所以你使用模型本身来生成它应该如何思考任何事情。模型会想出：哦，我要总结每篇文章，我要做这个、那个等等。这不是我们编程的，这是一个非常深刻的见解。这是唯一的方法吗？这是最好的方法吗？早期阶段，五年后我们会知道。我们 2029 年再聊。

I mean the probably what Demis was talking about these systems too which is indeed one that you you reflect a bit more and in games it's very clear right you just you just could say oh this move feels right you just move but if you think and Ponder um you might get to a better move um the challenge is that now because we are in such general direction right these malls can do anything I mean anything literally you just do whatever you want I mean upload an image talk about the news so what it means to to have this deeper thinking is so domain specific that how are you going to do that and I mean the there's a few answers but the one I like is like well these mods are very general to add the ability to think on top of a very general set of capabilities you probably need a general way to think and so you use the model itself um to generate how it should think about anything and the model will come up with oh like I'm going to Summarize each article I'm going to do this and that and than that and it is not us programming it that's a very deep Insight now is it the only way to do it and is it the best way to do it early days five years we'll see exactly I'll talk to you in 2029

好的，我现在也在想五年前感觉非常重要的很多事情，其中很多是来自神经科学的灵感。所以我想，在某种程度上你在这里谈论的是规划和推理，但记忆是另一个真正重要的方面。人们经常谈论长上下文和短上下文，我想这在一定程度上就是工作记忆，对吧？

okay I'm I'm thinking now though also about lots of the things that that felt very important back um you know five years ago um and a lot of it was about inspiration from from Neuroscience so I suppose in a way here you're talking about planning and reasoning but memory was the other really big one and I has that kind of come through people talk about long context and short context a lot I suppose that sort of is working memory in a way isn't it

是的，有一些技术可以应用于语言模型，至少有三个，而且解释起来相当简单。第一种方式是我们有一个系统能记住整个互联网，就是通过预训练（pre-training）步骤。这实际上是一个特定格式的记忆步骤：我们有一些随机权重，然后我们将它们组装成这些惊人的架构。第二个层次是……

yeah I mean there's techniques that are out there that you can apply to a language model there are at the very least three and they're reasonably simple to explain right the first way in which we have a system that memorizes all of the internet is by literally doing the pre-training step right that that's literally a memorization step in a particular format which is we have these weights they're random and then we assemble them in these amazing architectures now the second level is

也许我解释了一下，你如何将像谷歌这样的搜索引擎工具交给模型，你可以声称这是一种神经科学家称之为情景记忆的东西。你知道，作为人类，也许就像我们很久以前的记忆，它们不太精确，所以往往有点模糊，对吧？比如，如果我要想‘我在谷歌的第一天是什么样？’我记得零碎的片段，或者在一个房间里，或者我遇到的某个人，等等。大意，对，就是大意。有趣的是，这些模型可能没有这个限制，对吧？你可以直接获取一篇多年前写成的文章，它会有所有图片，一切都会被完美重建。所以第二种模式，称为情景记忆，我们清楚地看到，当你将特别强大的搜索引擎集成到我们的模型中时。然后第三种是你所谓的工作记忆，对吧，实际上我描述的整体思维就是其中之一，对吧？比如，如果我们拿每篇新闻文章，然后想创建摘要，找出它们之间的关联，批评其中一些，这就开始结合工作记忆，意思是我会有一个草稿本，记录摘要、我认为发现的问题。当我们说短上下文或长上下文时，通常指的是最后这一部分，即工作记忆：无论你有一千个词元，这意味着我可能做不了太多，对吧？我可以检索文章，但已经超过一千个词，我无法做太多总结；或者它可以很大，在这种情况下，你就有更多可能性在上面进行推理，等等。所以实际上今年的突破之一——我们还在 2024 年，是的——就是实现了上下文中数百万词元，这开启了许多可能性：你可以从过去检索东西，然后把它带到现在，进行非常详细的分析。这有点像例子：我们有一部电影，我们可以上传一部电影或一些很长的视频，然后开始做总结。我们上传它的事实更像是情景记忆，但现在我们把它放在记忆中，它全部适合记忆，我们可以在每一帧、电影中的每个对象等之间做大量关联。

Maybe I explained a little bit how you would give the tool of a search engine such as Google to the model that you could claim is a sort of what neuroscientists would call episodic memory, which you know as a human maybe it's like you know we have these memories from a long time ago; they're not very precise, so they tend to be a bit more fuzzy, right? Like if I have to think, 'Oh, what was my first day at Google?' I remember bits and pieces, or being in a room, or someone I met, or whatnot. The gist, yeah, the gist. Right. Now interestingly, these models maybe don't have that limitation, right? You can literally get an article written many years ago online, and it's going to have all the images, everything will be perfectly reconstructed. So that second mode, called episodic memory, clearly we're seeing that when you integrate especially powerful search engines into our models. And then the third one is what you could call working memory, right, which actually the whole chain of thinking that I describe is one of, right? Like if we take every news article but then we want to create summaries, find how they relate to each other, criticize some of them, this starts to combine working memory, meaning I'm going to have a scratchpad of the summaries, the issues that I think I'm finding. And that when we call short or long context, generally we mean this last bit, like the working memory: whether you have a thousand tokens, which means I couldn't possibly do much, right? I can retrieve articles, it's already over a thousand words, not much I can do to summarize them, or it can be massive, in which case you have many more possibilities to do reasoning on top of that, and so on and so forth. And so one of the breakthroughs of the year actually—we're still in 2024, it was yes—but was just to enable millions of tokens in context, which enables many things: you can retrieve something from the past but then bring it forward and then do a very detailed analysis. That's a bit of the examples: we have a movie, we can upload like a movie or some very long video and start doing summarization. The fact we kind of upload it is more episodic memory, but then now we have it in memory, it all fits in memory, we can do quite a lot of associations within each frame, each object in the movie, and so on and so forth.

更长的上下文窗口总是更好吗？我的意思是，因为我在想，我不知道你们在多大程度上仍然以神经科学作为工作的灵感，但人类记忆，比如工作记忆是有限度的，对吧？就像有时候你肯定会觉得‘我的大脑满了，我受够了’。

Is a longer context window always better? I mean, because I'm just thinking about, I don't know how much you guys are still using neuroscience as an inspiration for what you're doing, but I mean the human memory, like there's a limit to the working memory, right? Like there's certainly sometimes you're like, 'My brain is full and I'm done.'

是的，有时大脑是灵感，但计算机当然有优势；我们应该利用它的长处，对吧？所以也许它们可以像记住每一篇维基百科文章一样，我们做不到。但如果模型能做到，那你就有了新能力。但信息太多也可能太混乱，即使对这些神经网络来说也是如此，所以压缩可能是个好主意。所以这就是你可能想要推动的地方，从我们如何做我们所做的事情中获得灵感，这在记忆检索等方面非常惊人。

Yeah, sometimes brain is an inspiration, but computers certainly have advantages; we should build on its strengths, right? So perhaps the fact that they can have literally in memory like every Wikipedia article, whatever it is, we can't. But if the model can, well, there you go, you have new capabilities. But also it might be too confusing to have too much information, even for these neural networks, so it might be a good idea to compress. So that's where you probably want to push for getting some inspiration for how we might do what we do, which is quite amazing, right, in terms of memory retrieval and so on.

是的，这就是为什么你在领导激进的研究。我的意思是，我们想用模型做的事情肯定应该是鼓舞人心和前瞻性的，然后你考虑技术的主要限制，然后当然下注并激励团队围绕关键组件寻找解决方案。但你之前下的一些赌注已经成功了。我知道有一个重大公告，推出了令人眼花缭乱的新功能。我们能聊聊其中一些吗？然后也许也跟我谈谈我们已经讨论过的不同技能，以及它们如何在每个功能中体现？

Yeah, this is why you're leading drastic research. That's I mean what we want to do with the models should be definitely inspiring and forward-looking, and then you're what are the main limits of the technology and then try to of course place the bets and inspire the teams to finding solutions around the critical components. But some of the bets that you've already made have come off. I know there's been a big announcement of a dizzying number of new features that have just come out. Can we talk through some of them, and then maybe also talk to me about the different skills that we've already spoken about and how they appear in each of these?

是的，所以我们围绕最好的 Gemini 模型有很多系统。嗯，我们做的一件事是更新到 2.0。即使你说‘看，我们不再扩展了，或者我们能得到更好的质量吗？’我们也看到了代际飞跃。所以我们又做到了。这些模型更快、更便宜，而且实际上更好。Gemini 变得更好了。

Yeah, so we have quite a few sort of systems around our best Gemini models. Um, so one of the things that we've done is update to 2.0. We're seeing a generational leap even if you say, 'Look, let's not scale anymore, or can we get better quality?' So we've done it sort of again. These models are faster, they're cheaper, and they're actually better basically. Gemini's got better.

是的，Gemini 变得更好了，但我想不仅仅是因为我们扩展了。这是主要信息之一。告诉我更多关于你带给 Gemini 的智能体能力。

Yeah, Gemini's got better, but not only because we scaled, I guess. That's kind of one of the main messages. Tell me more about the agentic capabilities that you've brought to Gemini.

是的，所以我们在 Chrome 中发布了一个伴侣，你可以输入文字来执行任务。嗯，也许其中一些任务很棘手，因为我部分喜欢它们，但部分也不喜欢。所以我现在非常清楚地考虑旅行。好吧，你旅行时找酒店或航班之类的，很多时候感觉‘哦，我希望这可以自动化’，但与此同时，我不想完全脱离这个过程，对吧？所以我想我们发布的东西有望自动化一些更琐碎的步骤或重复性的事情，这些需要自动化，因为我懒得点击所有东西，对吧？所以我们添加了一种智能体，你可以让它为你做某事，然后它通过思考和行动，比如点击链接等，尝试为你解决任务，对吧？这是一个非常令人兴奋的研究挑战和机遇，因为它是一个非常通用的环境，最终用于非常通用的智能体和模型。早期原型中的一些例子是，我们可以要求它玩游戏，这当然回到了 DeepMind 的根源，在浏览器上，它表现得还不错，对吧？它找到一个网站，开始玩游戏。这很酷，联系到越通用，你就越能把过去需要专门化的环境视为现在‘哦，我只需输入，它就去学习玩这个游戏。’我的意思是我们还没有完全做到，但这让我们一窥这项技术可能的发展方向。

Yeah, so we're releasing a companion in Chrome where you can type to do a task. Uh, maybe some of these tasks are tricky because I partly enjoy them but also partly don't like them. So I'm thinking now very clearly about trips. Okay, so you travel and you look for hotels or flights or whatnot, and a lot of it feels like, 'Oh, I wish this could be automated,' but at the same time I wouldn't just want to not be part of this journey, right? So I guess the kind of thing we're releasing is something that hopefully will automate parts of the more trivial steps or repetitive things that need automation because I can't be bothered to click everything, right? So we're adding sort of an agent that you can ask it to do something for you, and then it's going to, again through thinking and through acting on like the basic clicking on links and so on, try to solve the task for you, right? And that's quite an exciting both research challenge and opportunity because it's a very general environment for a very general agent and model ultimately. And some examples that we had again in the early prototypes is I mean we can ask to play a game, which of course goes back to the roots of DeepMind, on the browser, and it kind of did okay, right? It finds a website, it starts playing the game. It's kind of a cool connection to the more general you are, then the more you can treat environments where you had specialization in the past as now, 'Oh, I just can type it and it just goes and learns to play this game.' I mean we're not quite there, but this is a glimpse of maybe where we could go with this kind of technology.

我的意思是你说得对，这确实把我们带回到你多年前做的事情，那就是可以使用键盘和鼠标的东西，对吧？真的很相似。

I mean you're right, then it does bring us back to that thing that you were doing so many years ago, which was something that can use a keyboard and a mouse, right? It's like really similar thing.

是的，甚至动作都非常相似，对吧？理解屏幕，根据你的要求，比如你会点击哪里等等。那是同一种动作。即使是那些非常通用的游戏也需要交互。区别在于目标是狭窄的，只是一个游戏和同类的屏幕，而这里是整个网络，非常广阔。

Yeah, even the actions are very similar, right? Understand the screen and given what you ask, I mean where would you click and so on. That's kind of the same sort of actions. Even that very general games would have to interact with. The difference is the goal that is narrow, it's just one game and the same kind of screens, whereas here is the whole web, which is pretty vast.

好吧，但我在想象你现在能做什么。我的意思是，它能查看你的日历吗？你能说‘我想明年去度假’，然后它能查看你的……

Well, okay, but then I'm sort of imagining what you could do now. I mean, could it look in your calendar? Could you say 'I want to go on holiday next year' and it could look in your...

日历和找出最佳周，你知道，你的预算等等。所以这些模型离自动化这个不远了，对吧？所以现在的问题是让它变得更好、更安全。有很多步骤，但如果你快进，人类在浏览器上能做的任何事情，这些东西原则上都能做。然后如果你让它们真正理解你想要什么，并通过思考和其他技术变得非常出色，它们会越来越好，而且可能会更快，在某些情况下甚至比你做得更好。所以这就是梦想。这还处于非常早期的阶段，但也非常令人兴奋。我认为明年我们肯定会看到很多关于将语言模型与浏览器或更一般的计算机相结合的实验。

Calendar and work out when the best week was, you know, your budget, etc. So these models are not far from being able to automate this, right? So now it's a matter of making it better, making it safe. There's a lot of steps, but if you just fast forward, anything a human can do on a browser, these things can do in principle. And then if you make them really understand what you want and really good through thinking and other techniques, they'll get better and better, and they'll be probably faster and maybe in some cases much better than you at doing that. So that's kind of the dream. And this is super early stages, but it's also super exciting. And I think certainly next year we're going to see a lot of experimentation around this idea of intersecting language models identically with browser or computer more generally.

那编程呢？

How about coding?

编程也是一个很好的领域。我们也在发布用于软件工程的工具，这些工具通常不仅需要“这里有一个关于编程问题的完美描述，请给我写代码，而且我知道如何测试它”。它更具迭代性，对吧？你必须编写代码、运行代码等等。所以我们从智能体式的角度推进这一能力。我们还——我的意思是游戏非常重要，当然那是开发强大算法的手段，但思考这些强大的多模态模型如何开始理解游戏并帮助用户在游戏过程中娱乐、提供建议或讲个关于游戏的笑话等等，也非常有趣。所以我们也在尝试一种游戏伴侣。

Coding is a great one as well. We are also releasing tools for software engineering, which of course generally require not only 'here is a perfect description of a puzzle about coding, please write me the code, and by the way I know how to test it.' It's more iterative, right? You have to write code, run the code, and so on. So we're putting that capability forward from an agentic point of view. We also, I mean games are very important, and of course that was a means to an end to develop powerful algorithms, but it's also very interesting to think about how these very powerful multimodal models start to understand games and can aid users to entertain during a game session, give them advice, or tell a joke about the game, etc. So we're also experimenting with the sort of game companion.

你谈到的所有这些事情，听起来非常接近相当通用的智能。我们是不是接近 AGI 了？

All of these things that you're talking about, I mean this is sounding very close to intelligence that is quite general. I mean are we getting close to AGI?

嗯，这是个好问题。你看，我这周早些时候就在想这个。如果 10 年前，甚至 5 年前，我拿到了今天的模型，对吧？我会说，‘看，有一个秘密实验室，这是一个模型，玩玩它，告诉我你是否认为这实际上接近通用智能。’我会声称，‘哦，是的，那来自一个 AGI 基本上已经发生或我可以看出它非常接近的未来。’对吧？所以越接近，你越会发现，‘哦，但它会幻觉’，当然这很重要，对吧？但我认为退一步看，感觉就是，好吧，它已经相当接近了。

Yeah, that's a good question. Look, I was thinking about this earlier this week. If 10 years ago, five years ago even, I would have been given the models today, right? And I would say, 'Look, there's a secret lab, this is a model, play with it and tell me if you think this is actually close to a general intelligence.' I would have claimed, 'Oh yeah, that comes from a future where AGI is basically either has happened or I can see that this is very close to it.' Right? So the closer you are, the more you find, 'Oh but it hallucinates,' of course that's very important, right? But I think just zooming out, it just feels like okay, it's getting pretty close.

但 DeepMind 的使命宣言是‘解决智能’，那种智能，比如超级智能，超越人类智能的东西。你认为 Scaling（规模扩张）足以让我们达到那里，还是我们需要其他东西？

But DeepMind's mission statement is 'solve intelligence', that sort of intelligence, like superintelligence, something that surpasses human intelligence. Do you think that scaling is enough to get us there, or do you think that we need something else?

是的，我的意思是 Google DeepMind 的使命显然是将智能与科学交叉以推动边界，我们最近在 AlphaFold 上看到了一个很好的例子。所以从这个意义上说，从领域角度来看，我们已经看到了一些狭窄但超级智能系统的例子。我的意思是 AlphaGo 只做那件事，我认为那可能是我们将开始看到超级智能的领域。即使从这些模型拥有的通用能力来看，你可能需要做一些专门化，而且这可能是值得的。我的意思是，解决蛋白质折叠值得吗？当然，绝对值得，对吧？但我认为这是一个很好的测试，我们处于非常有利的位置，因为我们有整个科学团队在研究非常有趣的问题。现在，如果你拿语言模型并开始思考智能体，将它们置于可能更涉及科学模拟、理论等的环境中，是否需要一些非常离散的东西来实现其他突破？我想说可能不需要，除非有另一个类似 Transformer 的突破。也许感觉我们将开始看到更多‘哦，天哪，就像在数学中，它现在发现了数学家觉得有趣的新定理’的例子，而这仅仅是通过非常好的执行加上一些想法的 Scaling（规模扩张）等等实现的。

Yeah, I mean Google DeepMind has this mission to obviously intersect intelligence with science to push the boundaries, and we saw a good example very recently with AlphaFold. So in that sense, from a domain's perspective, we have already seen some examples of narrow but superintelligent systems. I mean AlphaGo was only doing that, and I think probably that's the domains to think about where we're going to start seeing superintelligence. Even from the general sort of capabilities these models have, you might need to do some specialization, and again it might be worth it. I mean, was it worth it to solve protein folding? Of course, absolutely, right? But I think that's a good test to use, and we are very well positioned because we have the whole science team working on very interesting problems. Now, if you take the language models and you start thinking about agents, putting them in environments that could be more about science simulation, theory, and so on, will something very discrete be needed to enable other breakthroughs? I would say probably not without another Transformer-like breakthrough. Perhaps it feels like we're going to start seeing more examples of 'oh my God, like in math now it just discovers new theorems that mathematicians find interesting,' and it happened by just very good execution plus scaling up of some of the ideas and so on.

不过有趣的是，首先倒下的多米诺骨牌是那些有真实依据的，对吧？就像你描述的科学。

It is interesting though that the first dominoes to fall are the ones which have a ground truth, right? Like science as you described.

是的，尽管我的意思是科学，它取决于哪些科学可能有真实依据。我想蛋白质折叠肯定是。是的，没错。我希望我们也能看到一些其他方式以超人类的方式进步。就像你可以想象有一个由这些强大模型驱动的头脑风暴科学顾问，它不仅仅是发现新东西或证明新东西，而是挑战你的假设，让你以跳出框框的方式思考，然后我的创造力将我带到一个我本来无法到达的地方。那么你也会在某种程度上称之为超人类，对吧？所以我认为这些绝对不在范围之外，而且更难思考如何奖励这种行为。绝对迷人。

Yeah, although I mean science, it depends which sciences might have ground truth. I suppose protein folding definitely. Yeah, it's true. I'm hoping we also see some other ways to advance in a superhuman way. Like you could imagine having a brainstorming scientific advisor that is powered by one of these powerful models, and more than it discovers something or proves something new, it just challenges your assumptions and it makes you think out of the box in a way that then my creativity sort of gets me to a place I couldn't have gone. Then you would call that superhuman in some ways as well, right? So I think those are definitely not out of scope, and much harder to also think of how do you reward that behavior. Absolutely fascinating.

这里面肯定有很多激进的东西。是的，非常感谢你加入我。

There was definitely a lot of drastic stuff in there. Yes, thank you so much for joining me.

是的，同样感谢你。荣幸。五年后见。

Yeah, likewise, thanks. Pleasure. See you in 5 years.

我认为那次对话中出现了一个真正的主题，至少对我来说是这样，那就是通用性的概念。如果你仔细想想，智能推进知识的方式中存在着这种通用性。就像那些古老的天文学家，比如哥白尼，他们评估了大量观测天空的数据，并用这些数据提取了太阳系的模型。但在 AlphaGo 的例子中，它通过观察围棋对局来提取最佳下法的模型。而现在，在人类创造的一切事物中，都嵌入了这个模型，这个我们如何体验现实的底层真理。而我们正在寻找的模型，当然永远不会像日心说那样简洁，但那个模型似乎确实存在，隐藏在 Gemini 的冻结权重中。如果这就是我们到目前为止所做的，那么下一阶段就是尝试利用这些通用思想来提取人类偏好的模型。这当然要困难得多，但如果我们成功了，它可能会让我们走向一种更通用的智能形式，走向 AGI。如果你觉得这次对话有趣，我认为也值得看看我与 Jeff Dean 关于 Scaling（规模扩张）等话题的节目，以及与 Yassine Gabriel 关于 AI 智能体伦理的节目。或者如果你想更深入地了解 Gemini 2.0 的开发，可以查看由主持人 Logan Kilpatrick 主持的新 Google AI Release Notes 播客的最新一期。这个节目和其他节目可以在你获取播客的任何地方找到。下次见。

I think there was this real theme that emerged from that conversation, at least for me anyway, which was this idea of generality. And if you think about it, there is this generality in the way that intelligence advances knowledge. Like those old astronomers like Copernicus, they were assessing lots of data from observing the sky and using that to extract a model of the solar system. But in the case of AlphaGo, it was observing games of Go to extract a model for the best possible way to play. And now, somewhere embedded in everything that has ever been created by humans is this model, this underlying truth of how we experience reality. And the model that we're looking for, of course it's never going to be as neat as heliocentrism, but that model does seem to be in there, hidden among the frozen weights of Gemini. Now if that's what we've done so far, the next phase is to try and use those general ideas to extract a model of human preferences too. And that is of course a lot harder, but if we succeed, it might just get us to a more general form of intelligence, to AGI. Now if you found this conversation interesting, I think it's worth also checking out the episodes I did with Jeff Dean on, among other things, scaling, and with Yassine Gabriel on the ethics of AI agents. Or if you want to dig a bit deeper into the development of Gemini 2.0, then you can check out the latest episode of the new Google AI Release Notes podcast with host Logan Kilpatrick. This and other episodes can be found wherever you get your podcasts. Until next time.