十年内实现 AGI？Demis Hassabis 谈智能、迁移学习与神经科学洞见

所以如果我们在未来十年内拥有类似 AGI 的系统，我不会感到惊讶。这几乎让所有人，包括最初研究缩放定律假设的人，都感到惊讶，它已经走了这么远。在某种程度上，我今天看着这些大模型，觉得它们对于自身而言几乎不合理地有效。这是否会达到一个渐近线或一堵砖墙，是一个经验问题。我认为没人知道。

So I wouldn't be surprised if we had AGI-like systems within the next decade. It was pretty surprising to almost everyone, including the people who first worked on the scaling hypotheses, how far it's gone. In a way, I look at the large models today and I think they're almost unreasonably effective for what they are. It's an empirical question whether that will hit an asymptote or a brick wall. I think no one knows.

当你想到超人智能时，它仍然由一家私营公司控制吗？随着 Gemini 变得更多模态，我们开始摄取音频、视觉以及文本数据，我确实认为我们的系统将开始更好地理解现实世界的物理规律。我认为世界即将变得非常令人兴奋，在未来几年里，随着我们开始习惯真正多模态的含义。

When you think about superhuman intelligence, is it still controlled by a private company? As Gemini is becoming more multimodal and we start ingesting audio, visual as well as text data, I do think our systems are going to start to understand the physics of the real world better. The world's about to become very exciting, I think, in the next few years as we start getting used to the idea of what true multimodality means.

好的，今天非常荣幸能邀请到 DeepMind 的 CEO Demis Hassabis。Demis，欢迎来到播客。

Okay, today it is a true honor to speak with Demis Hassabis, who is the CEO of DeepMind. Demis, welcome to the podcast.

谢谢邀请。

Thanks for having me.

第一个问题：鉴于你的神经科学背景，你具体如何看待智能？你认为它是一个高层次的一般推理回路，还是成千上万个独立的子技能？

First question: given your neuroscience background, how do you think about intelligence specifically? Do you think it's like one higher-level general reasoning circuit, or do you think it's thousands of independent subskills?

嗯，这很有趣，因为智能是如此广泛，我们使用它的方式也非常通用。我认为这表明大脑处理我们周围世界的方式中一定存在某种高层次、共同的算法主题。当然，大脑中有专门的部分做特定的事情，但我认为可能有一些底层原则支撑着这一切。

Well, it's interesting because intelligence is so broad and what we use it for is so generally applicable. I think that suggests there must be some sort of high-level common algorithmic themes around how the brain processes the world around us. Of course, there are specialized parts of the brain that do specific things, but I think there are probably some underlying principles that underpin all of that.

你如何理解这样一个事实：在这些 LLM 中，当你在任何特定领域给它们大量数据时，它们往往在该领域变得不对称地更好？我们难道不会期望在所有不同领域都有某种普遍改进吗？

How do you make sense of the fact that in these LLMs, when you give them a lot of data in any specific domain, they tend to get asymmetrically better in that domain? Wouldn't we expect a sort of general improvement across all the different areas?

嗯，首先，我认为当你改进一个特定领域时，有时确实会在其他领域得到令人惊讶的改进。例如，当大模型在编码方面改进时，这实际上可以改善它们的一般推理能力。所以有一些迁移的证据，尽管我们希望有更多证据。但这也是人脑学习的方式。如果我们体验和练习很多像国际象棋或创意写作之类的事情，我们也会倾向于专门化并在那件特定事情上变得更好，即使我们使用一般的学习技术和一般的学习系统来擅长那个领域。

Well, first of all, I think you do actually sometimes get surprising improvement in other domains when you improve in a specific domain. For example, when the large models improve at coding, that can actually improve their general reasoning. So there is some evidence of transfer, although I think we would like a lot more evidence of that. But also, that's how the human brain learns too. If we experience and practice a lot of things like chess or creative writing, we also tend to specialize and get better at that specific thing, even though we're using general learning techniques and general learning systems to get good at that domain.

对你来说，这种迁移最令人惊讶的例子是什么？比如你看到语言和代码，或者图像和文本？

What's been the most surprising example of this kind of transfer for you? Like you see language and code, or images and text?

我认为可能……我的意思是，我希望我们会看到更多这种迁移，但我认为像在编码和数学方面变得更好，以及普遍提高推理能力——这就是我们作为人类学习者的方式。但在这些人工系统中看到这一点很有趣。

I think probably... I mean, I'm hoping we're going to see a lot more of this kind of transfer, but I think things like getting better at coding and math and generally improving your reasoning — that is how it works with us as human learners. But it's interesting seeing that in these artificial systems.

你能看到某种机制性的方式吗？比如说在语言和代码的例子中，神经网络中有一个地方同时因语言和代码而变得更好？还是这太遥远了？

Can you see the sort of mechanistic way in which, let's say in the language and code example, there's like a place in a neural network that's getting better with both the language and the code? Or is that too far down the line?

嗯，我不认为我们的分析技术足够复杂到能够精确地定位这一点。我认为这实际上是需要更多研究的领域之一——对这些系统构建的表征进行机制性分析。我有时称之为虚拟大脑分析。这有点像从真实大脑做 fMRI 或单细胞记录。这些人工思维的类似分析技术是什么？这方面有很多很棒的工作。像 Chris Olah 这样的人——我非常喜欢他的工作——以及许多计算神经科学技术，我认为可以应用于分析我们正在构建的这些当前系统。事实上，我试图鼓励我的许多计算神经科学朋友开始朝这个方向思考，并将他们的知识应用于大模型。

Well, I don't think our analysis techniques are quite sophisticated enough to be able to hone in on that. I think that's actually one of the areas that a lot more research needs to be done on — kind of mechanistic analysis of the representations that these systems build up. I sometimes like to call it virtual brain analytics. It's a bit like doing fMRI or single-cell recording from a real brain. What are the analogous analysis techniques for these artificial minds? There's a lot of great work going on on this sort of stuff. People like Chris Olah — I really like his work — and a lot of computational neuroscience techniques, I think, could be brought to bear on analyzing these current systems we're building. In fact, I've tried to encourage a lot of my computational neuroscience friends to start thinking in that direction and applying their knowledge to the large models.

鉴于你的神经科学背景，其他 AI 研究人员对人类智能有什么不理解的地方，而你有某种见解？

What do other AI researchers not understand about human intelligence that you have some insight on, given your neuroscience background?

我认为神经科学贡献了很多。如果你看过去 10 年或 20 年我们一直在做的事情，而我思考这个问题已经超过 30 年了。在 AI 新浪潮的早期，神经科学提供了许多有趣的方向性线索。比如强化学习，将其与深度学习结合——我们在那里做的一些开创性工作，比如经验回放，甚至注意力机制的概念，这些已经变得非常重要。许多最初的灵感来自对大脑如何工作的某种理解。当然，不是具体的细节——一个是工程系统，另一个是自然系统。所以与其说是特定算法的一对一映射，不如说是一种启发性的方向，也许是一些架构、算法或表征的想法。而且因为大脑是通用智能可能存在的存在证明，我认为人类努力的历史表明，一旦你知道某件事是可能的，就更容易朝那个方向努力，因为你知道那是一个努力的问题，是一个何时而非是否的问题。这让你能够更快地取得进展。所以我认为神经科学启发了我们今天所处位置的许多思考，至少是以一种软性的方式。但至于未来，我认为在规划和大脑如何构建正确的世界模型方面，还有很多有趣的事情需要解决。例如，我研究过大脑如何进行想象，或者你可以将其视为心理模拟。那么我们如何创建非常丰富的视觉空间世界模拟，以便更好地规划？

I think neuroscience has added a lot. If you look at the last 10 or 20 years that we've been at it, and I've been thinking about this for 30 plus years. In the earlier days of the new wave of AI, neuroscience was providing a lot of interesting directional clues. Things like reinforcement learning, combining that with deep learning — some of our pioneering work we did there, things like experience replay, even the notion of attention which has become super important. A lot of those original inspirations come from some understanding about how the brain works. Not the exact specifics, of course — one's an engineered system, the other's a natural system. So it's not so much about a one-to-one mapping of a specific algorithm; it's more kind of inspirational direction, maybe some ideas for architecture or algorithmic ideas or representational ideas. And because the brain is an existence proof that general intelligence is possible at all, I think the history of human endeavors has been that once you know something's possible, it's easier to push hard in that direction because you know it's a question of effort then, and a question of when, not if. That allows you to make progress a lot more quickly. So I think neuroscience has inspired a lot of the thinking, at least in a soft way, behind where we are today. But as for going forward, I think there are still a lot of interesting things to be resolved around planning and how the brain constructs the right world models. I studied, for example, how the brain does imagination, or you can think of it as mental simulation. So how do we create very rich visual-spatial simulations of the world in order for us to plan better?

实际上，我很好奇你认为这将如何与 LLM 接口。显然，DeepMind 处于前沿并且已经很多年了，拥有像 AlphaGo 这样的系统，这些智能体可以思考不同的步骤以达到最终结果。这会是 LLM 在其之上拥有这种树搜索之类的东西的路径吗？你怎么看？

Actually, I'm curious how you think that will sort of interface with LLMs. Obviously, DeepMind is at the frontier and has been for many years, with systems like AlphaGo and so forth, having these agents who can think through different steps to get to an end outcome. Will this just be a path for LLMs to have this sort of tree search kind of thing on top of them? How do you think about this?

我认为这是一个非常有前途的方向。

I think that's a super promising direction.

在我看来，我们必须继续改进大模型，让它们成为越来越准确的世界预测器，实际上就是更可靠的世界模型。这显然是 AGI 系统的一个必要但不充分的组成部分。在此基础上，我们正在研究像 AlphaZero 这样的规划机制，利用模型制定具体计划来实现某些目标，也许是将思维或推理链串联起来，甚至使用搜索来探索巨大的可能性空间。我认为这在我们当前的大模型中有所缺失。

In my opinion, we've got to carry on improving the large models and make them more and more accurate predictors of the world, effectively making them more reliable world models. That's clearly a necessary but probably not sufficient component of an AGI system. On top of that, we're working on things like AlphaZero-like planning mechanisms that use that model to make concrete plans to achieve certain goals, perhaps chaining thoughts or lines of reasoning together, and maybe using search to explore massive spaces of possibility. I think that's kind of missing from our current large models.

如何克服这些方法通常需要的巨大算力？即使是 AlphaGo 系统也相当昂贵，因为你必须在树的每个节点上运行 LLM。你预计这将如何变得更高效？

How do you get past the immense amount of compute that these approaches tend to require? Even the AlphaGo system was pretty expensive because you had to run an LLM on each node of the tree. How do you anticipate that'll become more efficient?

摩尔定律会有所帮助，每年都会有更多的算力可用。但我们非常注重样本高效的方法和重用现有数据，比如经验回放，以及寻找更高效的方式。你的世界模型越好，搜索就能越高效。例如，我们的围棋和国际象棋系统 AlphaZero，在所有这类游戏中都超越了世界冠军水平，而且它使用的搜索量远低于像深蓝这样的暴力方法。深蓝可能每步决策要看数百万种走法，而 AlphaZero 和 AlphaGo 只看大约数万个位置。但人类世界冠军可能只看几百步就能做出非常好的决策。这表明暴力系统没有真正的游戏模型，而 AlphaZero 有一个不错的模型，但顶尖人类棋手对围棋或国际象棋有更丰富、更准确的模型，使他们能用极少的搜索做出世界级的决策。所以存在一个权衡：如果你改进模型，搜索就能更高效，从而走得更远。

One thing is Moore's law tends to help, as more computation becomes available each year. But we focus a lot on sample-efficient methods and reusing existing data, things like experience replay, and also looking at more efficient ways. The better your world model is, the more efficient your search can be. For example, AlphaZero, our system for Go and chess, is stronger than world champion level at all these games, and it uses a lot less search than brute force methods like Deep Blue. Deep Blue might look at millions of possible moves for every decision, while AlphaZero and AlphaGo looked at around tens of thousands of positions. But a human world champion probably only looks at a few hundred moves to make a very good decision. That suggests brute force systems don't have a real model of the game, while AlphaZero has a decent model, but top human players have a much richer, more accurate model of Go or chess, allowing them to make world-class decisions with very little search. So there's a trade-off: if you improve the models, your search can be more efficient, and you can get further with it.

在 AlphaGo 中，你有一个非常具体的获胜条件：最终我是否赢了这盘棋？你可以据此进行强化。当你考虑 LLM 输出想法时，你认为会有这种区分好坏并给予奖励的能力吗？

With AlphaGo, you had a very concrete win condition: at the end of the day, do I win this game or not? And you can reinforce on that. When you're thinking of an LLM putting out thoughts, do you think there will be this kind of ability to discriminate whether that was a good thing to reward or not?

这就是为什么我们开创了用游戏作为试验场，DeepMind 也因此闻名。部分原因是这对研究来说很高效，但另一个原因是很容易指定奖励函数——赢得比赛或提高分数是大多数游戏内置的。现实世界系统面临的挑战之一是如何定义正确的目标函数、正确的奖励函数和正确的目标，并以一种通用但足够具体的方式指定它们，引导系统朝着正确的方向。对于现实世界的问题，这可能困难得多。但实际上，即使在科学问题中，通常也有办法指定你追求的目标。当你思考人类智能时，人类只是超级样本高效。爱因斯坦是如何提出相对论的？方程有成千上万种可能的排列。你认为这也是一种尝试不同方法的感觉，还是与 AlphaGo 的做法完全不同的解决方式？

That's why we pioneered and DeepMind is famous for using games as a proving ground. Partly because it's efficient for research, but also because it's extremely easy to specify a reward function—winning the game or improving the score is built into most games. One of the challenges for real-world systems is how to define the right objective function, the right reward function, and the right goals, and specify them in a general way that is specific enough and points the system in the right direction. For real-world problems, that can be a lot harder. But actually, even in scientific problems, there are usually ways to specify the goal you're after. When you think about human intelligence, humans are just super sample-efficient. How did Einstein come up with relativity? There are thousands of possible permutations of the equations. Do you think it's also this sense of trying different approaches, or is it a totally different way of arriving at solutions compared to what AlphaGo does?

我认为这不同，因为我们的大脑不是为进行蒙特卡洛树搜索而设计的。我们的有机大脑不是那样工作的。所以为了弥补，像爱因斯坦这样的人利用他们的直觉——无论直觉是什么——以及他们的知识和经验，构建了极其准确的物理模型，包括心理模拟。如果你读关于爱因斯坦的书，他过去常常可视化并真正感受这些物理系统应该是什么样子，不仅仅是数学，而是对它们在现实中的样子有一种直觉。这让他当时能思考非常离奇的想法。所以我认为这取决于我们构建的世界模型的复杂程度。如果你的世界模型能让你到达搜索树中的某个节点，然后你在那个叶节点周围做一点搜索，就能到达原创的地方。但如果你的模型和对模型的判断非常好，你可以更准确地选择哪些叶节点需要扩展搜索，因此总体上你做的搜索少得多。任何人类都不可能对任何有意义的空间进行暴力搜索。

I think it's different because our brains are not built for doing Monte Carlo tree search. That's just not how our organic brains work. So to compensate, people like Einstein used their intuition—whatever intuition is—and their knowledge and experience to build extremely accurate models of physics, including mental simulations. If you read about Einstein, he used to visualize and really feel what these physical systems should be like, not just the mathematics, but have an intuitive feel for them in reality. That allowed him to think very outlandish thoughts at the time. So I think it's the sophistication of the world models we're building. If your world model can get you to a certain node in a search tree, and then you do a little bit of search around that leaf node, that gets you to original places. But if your model and your judgment on that model are very good, you can pick which leaf nodes to expand with search much more accurately, so overall you do a lot less search. There's no way any human could do brute force search over any significant space.

目前一个很大的开放问题是，强化学习是否能让这些模型通过合成数据进行自我对弈，从而克服数据瓶颈。听起来你对此很乐观。

A big open question right now is whether RL will allow these models to do self-play with synthetic data to get over the data bottleneck. It sounds like you're optimistic about this.

是的，我对此非常乐观。首先，还有更多数据可以使用，尤其是如果考虑到多模态和视频这类东西。显然，社会一直在增加更多数据，比如互联网等等。但我认为创建合成数据有很大的空间。我们正在以不同的方式研究这个问题，部分是通过模拟。

Yes, I'm very optimistic about that. First of all, there's still a lot more data that can be used, especially if one views multimodal and video and these kinds of things. Obviously, society is adding more data all the time, to the internet and things like that. But I think there's a lot of scope for creating synthetic data. We're looking at that in different ways, partly through simulation.

比如逼真的游戏环境，逼真的数据，还有自我对弈，系统之间相互交互或对话。就像我们在 AlphaGo 和 AlphaZero 中非常熟悉的那样，让系统相互对弈，从彼此的错误中学习，并以此建立知识库。我认为这有一些很好的类比。这要复杂一些，但要构建一种通用的世界数据，如何让这些模型输出的合成数据、进行的自我对弈不仅仅是它们数据集中已有的内容，而是它们从未见过的东西，从而真正提升能力？

Realistic games environments, for example, realistic data but also self-play, where systems interact with each other or converse with each other. In the sense of what we know very well from AlphaGo and AlphaZero, where we got systems to play against each other and actually learn from each other's mistakes and build up a knowledge base that way. I think there are some good analogies for that. It's a little bit more complicated, but to build a general kind of world data, how do you get to the point where these models, the sort of synthetic data they're outputting, the self-play they're doing, is not just more of what they've already got in their dataset, but is something they haven't seen before, to actually improve the abilities?

是的，我认为这需要一整套科学，而且我们仍处于数据整理和数据分析的初期阶段。所以，实际上分析数据分布中的空洞——这对于消除系统中的公平性、偏见等问题很重要——是为了确保你的数据集能够代表你试图学习的分布。有很多技巧可以使用，比如对数据的某些部分进行过加权或重放，或者你可以想象，如果你发现数据集中的某个缺口，那就是你发挥合成生成能力的地方。

Yes, so there I think a whole science is needed, and I think we're still in the nascent stage of this of data curation and data analysis. So actually analyzing the holes that you have in your data distribution — and this is important for things like fairness and bias and other stuff to remove that from the system — is to try and really make sure that your dataset is representative of the distribution you're trying to learn. And there are many tricks one can use, like overweighting or replaying certain parts of the data, or you could imagine if you identify some gap in your dataset, that's where you put your synthetic generation capabilities to work on.

现在人们开始关注 DeepMind 多年前做的强化学习工作。有哪些早期的研究方向或过去做过的事情，但人们一直没有关注，而你认为它们会变得很重要？就像曾经有一段时间人们不关注 Scaling（规模扩张）。现在有什么东西被完全低估了？

Nowadays people are paying attention to the RL stuff that DeepMind did many years before. What are the sort of early research directions or something that was done way back in the past but people just haven't been paying attention to, that you think will be a big deal? Like there's a time where people weren't paying attention to scaling. What's the thing now where it's totally underrated?

嗯，我认为过去几十年的历史就是各种潮流起起落落。我确实觉得，在不久前，大概五年多以前，当我们开创 AlphaGo 以及更早的 DQN 时，那是第一个在 Atari 上工作的系统，也是我们十多年前的第一个大型系统，它扩展了 Q 学习和强化学习技术，将其与深度学习结合，创造了深度强化学习，然后用来扩展以完成一些相当复杂的任务，比如仅从像素玩 Atari 游戏。我确实认为很多这样的想法需要重新回归，并且正如我们之前讨论的，与大型模型和多模态模型的新进展相结合，这显然也非常令人兴奋。所以我确实认为将一些旧想法与新想法结合起来有很大的潜力。

Well, I think that the history of the sort of last couple of decades has been things coming in and out of fashion. And I do feel like a while ago, maybe five plus years ago, when we were pioneering with AlphaGo and before that DQN, where it was the first system that worked on Atari, our first big system really more than 10 years ago now, that scaled up Q-learning and reinforcement learning techniques to combine that with deep learning to create deep reinforcement learning, and then use that to scale up to complete some pretty complex tasks like playing Atari games just from the pixels. I do actually think a lot of those ideas need to come back in again, and as we talked about earlier, combine it with the new advances in large models and large multimodal models, which is obviously very exciting as well. So I do think there's a lot of potential for combining some of those older ideas together with the new ones.

AGI（通用人工智能）有没有可能最终仅仅来自纯粹的强化学习方法？按照我们现在的讨论，听起来大语言模型会形成正确的先验，然后这类研究在此基础上进行。有没有可能完全脱离强化学习？

Is there any potential for AGI to eventually come from just a pure RL approach? The way we're talking about it, it sounds like the LLM will form the right prior and then this sort of research will go on top of that. Is there a possibility to just completely out of RL?

我认为理论上没有理由不能完全像 AlphaZero 那样做，DeepMind 和强化学习社区里有些人就在研究这个，假设没有先验、没有数据，完全从头构建所有知识。我认为这很有价值，因为这些想法和算法在有知识的情况下也应该有效。但话虽如此，我认为目前最快、最可能实现 AGI 的方式是利用世界上现有的所有知识，比如网络上的知识，以及我们已经收集的数据，并且我们有像 Transformer 这样的可扩展算法能够消化所有这些信息。我不认为有什么理由不以一个模型作为先验或基础来进行预测，从而帮助引导学习。我只是觉得不利用这些是没有道理的。所以我的赌注是，最终的 AGI 系统会包含这些大型多模态模型作为整体解决方案的一部分，但可能仅靠它们还不够；你还需要在此基础上进行额外的规划和搜索。

I think theoretically there's no reason why you couldn't go full AlphaZero on it, and there are some people here at DeepMind and in the RL community who work on that, assuming no priors, no data, and just build all knowledge from scratch. And I think that's valuable because those ideas and those algorithms should also work when you have some knowledge too. But having said that, I think by far the quickest way to get to AGI and the most likely plausible way is to use all the knowledge that's existing in the world right now on things like the web, and that we've collected, and we have these scalable algorithms like Transformers that are capable of ingesting all of that information. And I don't see why you wouldn't start with a model as a kind of prior or to build on and to make predictions that helps bootstrap your learning. I just think it doesn't make sense not to make use of that. So my betting would be that the final AGI system will have these large multimodal models as part of the overall solution, but probably won't be enough on their own; you need this additional planning and search on top.

这听起来像是要回答我接下来要问的问题：强版本的 Scaling 假设（规模扩张假设）哪些地方是对的，哪些地方是错的？就是那种只要投入足够的算力，覆盖足够广的数据分布，就能获得智能的想法。

This sounds like the answer to the question I'm about to ask: what does the strong version of the scaling hypothesis get right and what does it get wrong? The idea that you just throw compute at a wide enough distribution of data and you get intelligence.

你看，我的观点是这目前是一个实证问题。我认为几乎所有人都很惊讶，包括最初研究 Scaling 假设的人，它竟然能走这么远。在某种程度上，我看着今天的大模型，觉得它们对于自身而言几乎不合理地有效。我认为一些涌现的特性非常令人惊讶，比如——在我看来，它们显然具有某种形式的概念和抽象。如果我们在五年多以前讨论，我可能会说我们需要额外的算法突破才能做到这一点，比如更像大脑的工作方式。我认为如果我们想要显式的抽象概念，这仍然成立，但这些系统似乎能够隐式地学习到这些。另一个非常有趣、我认为出乎意料的事情是，这些系统具有某种程度的 grounding（基础），尽管它们没有多模态地体验世界，或者至少直到最近我们才有了多模态模型。令人惊讶的是，仅从语言中就能构建如此多的信息和模型。我对此有一些假设。我认为我们通过 RLHF（基于人类反馈的强化学习）反馈系统获得了一些 grounding，因为人类评分者本身就是 grounded 的，我们扎根于现实，所以我们的反馈也是 grounded 的。所以也许有一些 grounding 是通过那里进来的。另外，语言可能比我们以前认为的包含更多的 grounding。所以实际上，我认为有一些非常有趣的哲学问题，我们甚至还没有真正触及表面。

Look, my view is this is kind of an empirical question right now. I think it was pretty surprising to almost everyone, including the people who first worked on the scaling hypotheses, how far it's gone. In a way, I sort of look at the large models today and I think they're almost unreasonably effective for what they are. I think it's pretty surprising some of the properties that emerge, things like — it's clearly in my opinion got some form of concepts and abstractions and things like that. And I think if we were talking five plus years ago, I would have said to you maybe we need an additional algorithmic breakthrough in order to do that, like maybe more like the brain works. And I think that's still true if we want explicit abstract concepts, but it seems that these systems can implicitly learn that. Another really interesting, I think unexpected thing was that these systems have some sort of grounding, even though they don't experience the world multimodally, or at least until more recently when we have the multimodal models. And that's surprising that the amount of information that can be and models that can be built up just from language. And I think I have some hypothesis about why that is. I think we get some grounding through the RLHF feedback systems because obviously the human raters are by definition grounded, we're grounded in reality, so our feedback is also grounded. So perhaps there's some grounding coming in through there. And also maybe language contains more grounding than we perhaps thought before. So actually some very interesting philosophical questions I think we haven't even really scratched the surface of yet.

关于我们下一步的方向，但就你提到的大模型问题，我认为我们必须尽可能推动 Scaling（规模扩张），这也是我们正在做的。这会不会遇到天花板或瓶颈是一个实证问题，不同的人有不同看法，但我觉得我们应该直接去测试。没人知道答案。与此同时，我们还应该加倍投入创新和发明。Google Research、DeepMind 和 Google Brain 在过去十年里开创了许多东西，这是我们的看家本领。你可以把我们一半的努力看作 Scaling（规模扩张），另一半是发明未来需要的架构和算法，因为我们知道更大规模的模型正在到来。我目前的判断——虽然比较粗略——是两者都需要。但我们必须尽可能同时推进这两方面，而我们很幸运有能力做到这一点。

About where we're going next, but in terms of your question about large models, I think we've got to push scaling as hard as we can, and that's what we're doing here. It's an empirical question whether that will hit an asymptote or a brick wall, and different people argue about that, but I think we should just test it. No one knows. In the meantime, we should also double down on innovation and invention. Google Research, DeepMind, and Google Brain have pioneered many things over the last decade; that's our bread and butter. You can think of half our effort as scaling and half as inventing the next architectures and algorithms that will be needed, knowing that larger models are coming. My betting right now, though it's loose, is that you would need both. But we've got to push both as hard as possible, and we're in a lucky position that we can do that.

我想进一步探讨接地性问题。你可以想象两件事会让接地变得更困难。第一，随着模型变得更聪明，它们会在我们无法生成足够人类标签的领域运作，因为我们不够聪明。比如，如果它提交了一百万行的代码合并请求，我们怎么判断它是否符合我们的道德约束和最终目标？第二，到目前为止，大部分算力都用在了下一个词预测上，这在某种意义上是一种护栏，因为你必须像人类那样说话和思考。但如果额外的算力来自强化学习，我们只追求最终目标，就无法真正追溯它是如何达到的。当这两者结合时，你有多担心接地性会消失？

I want to ask more about grounding. You can imagine two things that might make grounding more difficult. One is that as these models get smarter, they'll operate in domains where we can't generate enough human labels because we're not smart enough. For example, if it does a million-line pull request, how do we tell if it's within the constraints of our morality and the end goal we wanted? The other is that more of the compute so far has been next-token prediction, which in some sense is a guardrail because you have to talk and think as a human would. Now if additional compute comes in the form of reinforcement learning where we just get to the end objective, we can't really trace how you got there. When you combine those two, how worried are you that the sort of grounding goes away?

嗯，我认为如果系统没有正确接地，它就无法正确实现那些目标。从某种意义上说，系统必须要有接地性——至少部分接地——才能在现实世界中实现目标。我确实认为，随着像 Gemini 这样的系统变得更加多模态，我们开始摄入视频、音视频数据以及文本数据，系统会开始将这些信息关联起来。我认为这是一种正确的接地形式。所以我相信我们的系统会开始更好地理解现实世界的物理规律。可以想象，主动版本的接地就是置身于非常逼真的模拟或游戏环境中，在那里你开始学习你的行为对世界的影响，以及它如何影响世界本身，同时也影响你获得的下一个学习片段。我们一直在研究的那些强化学习智能体，比如 AlphaZero 和 AlphaGo，它们实际上会影响自身的主动学习——它们决定下一步做什么，会影响它们接下来获得的学习数据或经验。所以存在一个有趣的反馈循环。当然，如果我们想在机器人等领域有所建树，就必须理解如何在现实世界中行动。

Well, I think if the grounding is not properly grounded, the system won't be able to achieve those goals properly. In a sense, you have to have the grounding, or at least some of it, for a system to actually achieve goals in the real world. I do think that as these systems, like Gemini, become more multimodal and we start ingesting video, audio-visual data, as well as text data, the system starts correlating those things together. I think that is a form of proper grounding. So I do think our systems will start to understand the physics of the real world better. One could imagine the active version of that is being in a very realistic simulation or game environment where you start learning about what your actions do in the world and how that affects the world itself, but also what next learning episode you get. These RL agents we've always worked on, like AlphaZero and AlphaGo, they actually affect their active learning—what they decide to do next affects what the next learning piece of data or experience they get. So there's this interesting feedback loop. Of course, if we ever want to be good at things like robotics, we'll have to understand how to act in the real world.

所以有一种接地性关乎能力能否推进，能否与现实接轨以完成我们想要的事情。还有另一种接地性：我们很幸运，因为模型是在人类思维上训练的，所以它们可能像人类一样思考。当训练算力更多地来自追求正确结果，而不是像人类那样受下一个词预测的约束时，这种特性还能保持多少？更广泛的问题是：要如何对齐一个比人类更聪明、可能用外星概念思考的系统？你无法真正监控一百万行的代码合并请求，因为你理解不了。

So there's a grounding in terms of capabilities proceeding, being in touch with reality to do what we want. There's another sense of grounding: we've gotten lucky that since they're trained on human thought, they might think like a human. To what extent does that stay true when more of the compute for training comes from just getting the right outcome, not guarded by next-token prediction as a human would? The broader question is: what would it take to align a system that's smarter than a human, maybe thinks in alien concepts, and you can't really monitor the million-line pull request because you can't understand it?

这是 Shane 和我以及许多其他人早在创立 DeepMind 之前就一直关注的问题。因为我们为成功做了规划。2010 年，没人考虑 AI，更不用说 AGI（通用人工智能），但我们当时就知道，如果能在这些想法上取得进展，创造出的技术将具有难以置信的变革性。所以我们在 20 年前就已经在思考其后果，无论是积极的还是消极的。积极的方向令人惊叹——像 AlphaFold 这样的科学成果，在健康、科学、数学和发现方面取得了不可思议的突破。但我们也必须确保这些系统是可理解和可控的。有很多想法：更严格的评估系统——我认为我们还没有足够好的评估和基准来测试系统是否会欺骗你、外泄自己的代码或其他不良行为。还有使用狭义 AI（专用系统）来帮助人类科学家分析和总结更通用系统行为的想法。我认为创建加固的沙箱或模拟环境很有前景，配合网络安全措施来防止 AI 逃逸和黑客入侵，这样你就可以更自由地进行实验。还有我们之前讨论的分析工作：分析和理解系统构建的概念和表征，这样它们可能就不会那么陌生，我们也能跟踪它正在构建的知识。

This is something Shane and I and many others have had at the forefront of our minds since before we started DeepMind. Because we planned for success. In 2010, no one was thinking about AI, let alone AGI, but we already knew that if we could make progress with these ideas, the technology would be unbelievably transformative. So we were already thinking 20 years ago about what the consequences would be, both positive and negative. The positive direction is amazing—science like AlphaFold, incredible breakthroughs in health, science, math, discovery. But we also have to make sure these systems are understandable and controllable. There are many ideas: more stringent evaluation systems—I think we don't have good enough evaluations and benchmarks for things like whether the system can deceive you, exfiltrate its own code, or other undesirable behaviors. Then there are ideas of using narrow AI—specialized systems—to help human scientists analyze and summarize what the more general system is doing. I think there's a lot of promise in creating hardened sandboxes or simulations, with cybersecurity arrangements to keep the AI in and hackers out, so you can experiment more freely. And there's the analysis stuff we talked about earlier: analyzing and understanding the concepts and representations the system builds, so maybe they're not so alien to us and we can keep track of the knowledge it's building.

退一步说，我很好奇你的时间线。Shane 说他最可能的结果是 2028 年，也许是中位数。你的呢？

Stepping back a bit, I'm curious what your timelines are. Shane said his modal outcome is 2028, maybe median. What is yours?

嗯，我不会给出具体数字，因为未知因素太多。但我认为在下一个十年内看到 AGI（通用人工智能）是可能的，甚至可能更早。不过，这非常不确定。

Well, I don't prescribe specific numbers because there are so many unknowns. But I think it's plausible that we could see AGI within the next decade, maybe even sooner. However, it's very uncertain.

未知和不确定性，以及人类的聪明才智和努力，总是会带来惊喜，所以这可能会显著改变时间线。但我要说的是，当我们在 2010 年创办 DeepMind 时，我们把它看作一个 20 年的项目，实际上我认为我们正按计划进行，这对于 20 年项目来说相当了不起，因为通常它们总是还有 20 年，对吧？这就是关于量子 AI 之类的笑话，随便你选。但我认为我们正按计划进行，所以如果我们在未来十年内拥有类似 AGI 的系统，我不会感到惊讶。

Unknowns and uncertainties, and human ingenuity and endeavor come up with surprises all the time, so that could meaningfully move the timelines. But I will say that when we started DeepMind back in 2010, we thought of it as a 20-year project, and actually I think we're on track, which is kind of amazing for 20-year projects because usually they're always 20 years away, right? So that's the joke about quantum AI, take your pick. But I think we're on track, so I wouldn't be surprised if we had AGI-like systems within the next decade.

你是否认同这样一种模式：一旦你有了 AGI，你就有了一个能加速进一步 AI 研究的系统，也许不是一夜之间，但在几个月或几年内，你会取得比人类快得多的进展？

Do you buy the model that once you have an AGI, you have a system that basically speeds up further AI research, maybe not overnight, but over months and years you have much faster progress?

我认为这有可能。我认为部分取决于我们作为社会决定将第一个 AGI 系统甚至原始 AGI 系统用于什么目的。即使是当前的 LLM 似乎也很擅长编码，我们有像 AlphaCode 这样的系统，并且我们正在改进系统。所以可以想象将这些想法结合起来，让它们变得更好，然后这些系统可能非常擅长设计和帮助我们构建未来的版本。但当然，我们也必须考虑其中的安全影响。

I think that's potentially possible. I think it partly depends on what we as a society decide to use the first AGI systems or even proto-AGI systems for. Even current LLMs seem to be pretty good at coding, and we have systems like AlphaCode, and we're improving systems. So one could imagine combining these ideas and making them a lot better, and then these systems could be quite good at designing and helping us build future versions of themselves. But we also have to think about the safety implications of that, of course.

我很好奇你怎么看。最终你会开发一个模型，在开发过程中你认为一旦完全开发，它有可能具备智能爆炸的动态。要使你安心继续开发，那个模型必须满足什么条件？

I'm curious what you think about that. Eventually you'll be developing a model where during development you think there's some chance that once fully developed, it'll be capable of an intelligence explosion dynamic. What would have to be true of that model for you to be comfortable continuing development?

嗯，我们需要比今天对系统有更多的理解，然后我才能自信地告诉你我们需要勾选什么。所以我认为在未来几年，在这些系统开始出现之前，我们必须提出正确的评估和指标，也许理想情况下是形式化证明，但对于这类系统来说这很难。至少要有关于这些系统能力的经验界限。这就是为什么我认为像欺骗这样的东西是你绝对不想要的根节点特征，因为如果你确信你的系统暴露了它真实的想法，那么这就打开了利用系统本身向你解释其自身方面的可能性。我对此的看法是，如果我与加里·卡斯帕罗夫下棋，我想不出他能想出的棋步，但他可以向我解释他为什么走出那一步，我事后也能理解。可以想象，我们可以利用这些系统的能力之一是让它们向我们解释事情，甚至解释它们为什么这样思考的证明，特别是在数学问题中。

Well, we need a lot more understanding of the systems than we do today before I would be confident even explaining to you what we would need to tick off. So I think what we've got to do in the next few years, before those systems start arriving, is come up with the right evaluations and metrics, and maybe ideally formal proofs, but it's going to be hard for these types of systems. At least empirical bounds around what these systems can do. That's why I think about things like deception as quite root node traits that you don't want, because if you're confident that your system is exposing what it actually thinks, then that opens up possibilities of using the system itself to explain aspects of itself to you. The way I think about that is like if I were to play a game of chess against Garry Kasparov, I wouldn't be able to come up with a move that they could, but they could explain to me why they came up with that move, and I could understand it post hoc. One could imagine one of the capabilities we could make use of these systems is for them to explain things to us, and even the proofs behind why they're thinking something, certainly in mathematical problems.

你是否有相反答案的感觉？明天早上你看到某个具体观察，你会说“我们必须停止 Gemini 2 训练”，那必须满足什么条件？

Do you have a sense of what the converse answer would be? What would have to be true where tomorrow morning you see some specific observation where you're like, we got to stop Gemini 2 training?

我可以想象，这就是像沙盒模拟这样的东西发挥作用的地方。我希望我们在一个安全可靠的环境中进行实验，然后其中发生了一些非常意外的事情，一种新的意外能力，或者我们明确告诉系统我们不想要的东西，但它做了然后撒谎。这些都是需要仔细深入调查的事情。对于当今的系统，我认为它们今天并不危险，但几年后它们可能具有潜力，然后你理想上会暂停，真正弄清楚它为什么做那些事情，然后再继续。

I could imagine that, and this is where things like sandbox simulations, I would hope we're experimenting in a safe, secure environment, and then something happens in it where something very unexpected happens, a new unexpected capability, or something that we explicitly told the system we didn't want, but it did and then lied about it. These are the kinds of things where one would want to dig in carefully. With the systems around today, which are not dangerous in my opinion today, but in a few years they might have potential, and then you would ideally pause and really get to the bottom of why it was doing those things before continuing.

回到 Gemini，我很好奇开发中的瓶颈是什么。如果 Scaling（规模扩张）效果很好，为什么不立即将其扩大一个数量级？

Going back to Gemini, I'm curious what the bottlenecks were in the development. Why not make it immediately one order of magnitude bigger if scaling works well?

首先，实际限制是单个数据中心能容纳多少算力。你会遇到非常有趣的分布式计算挑战。不幸的是，我们在这些挑战上有一些世界上最优秀的人才，跨数据中心训练等等。非常有趣的挑战，硬件挑战，我们一直在构建和设计我们的 TPU，同时也使用 GPU。所以有所有这些。然后还有缩放定律；它们不是靠魔法起作用的。你仍然需要扩展超参数，每次新规模都会不断有各种创新。这不仅仅是重复相同的配方；你必须调整配方，这在某种程度上是一种艺术形式。你几乎需要获得新的数据点。如果你试图将预测外推几个数量级，有时它们不再成立，因为新能力可能是阶跃函数，有些东西成立，有些则不成立。所以通常你确实需要那些中间数据点来纠正一些超参数优化和其他事情，以便缩放定律继续成立。所以有各种实际限制。一个数量级大概是每个时代之间你想要推进的最大值。

First of all, there are practical limits on how much compute you can actually fit in one data center. You're bumping up against very interesting distributed computing challenges. Unfortunately, we have some of the best people in the world on those challenges, and cross-data center training, all these kinds of things. Very interesting challenges, hardware challenges, and we have our TPUs that we're building and designing all the time, as well as using GPUs. So there's all of that. And then you also have the scaling laws; they don't just work by magic. You still need to scale up the hyperparameters, and various innovations are going in all the time with each new scale. It's not just about repeating the same recipe at each new scale; you have to adjust the recipe, and that's a bit of an art form in a way. You have to almost get new data points. If you try to extend your predictions extrapolate them several orders of magnitude out, sometimes they don't hold anymore because new capabilities can be step functions in terms of new capabilities, and some things hold and other things don't. So often you do need those intermediate data points to correct some of your hyperparameter optimization and other things so that the scaling law continues to be true. So there are various practical limitations. One order of magnitude is probably about the maximum you want to carry on between each era.

太迷人了。在 GPT-4 技术报告中，他们说他们能够用比 GPT-4 少数万倍的算力预测训练损失；他们能看到曲线。但你的观点是，损失所暗示的实际能力可能并不遵循损失曲线。

That's so fascinating. In the GPT-4 technical report, they say they were able to predict the training loss with tens of thousands of times less compute than GPT-4; they could see the curve. But the point you're making is that the actual capabilities that loss implies may not follow from the loss curve.

你通常可以预测核心指标，比如训练损失之类的，但这并不一定能转化为 MMLU 或数学或其他你关心的实际能力。它们并不总是线性的，存在某种非线性效应。

You can often predict the core metrics like training loss or something like that, but then it doesn't actually translate into MML or math or some other actual capability you care about. They're not necessarily linear all the time; there are sort of nonlinear effects.

在开发 Gemini 的过程中，你最大的意外是什么？类似这样的事情？嗯，我不会说有一个大意外，但尝试训练那种规模的模型非常有趣，学到了各种东西，从组织到如何照看这样一个系统并跟踪它。我认为，比如更好地理解你优化的指标与你想要的最终能力之间的关系——我仍然认为这个映射没有被完美理解，但这是一个有趣的映射，我们正在变得越来越擅长。

What was the biggest surprise to you during the development of Gemini? Something like this happening? Well, I wouldn't say there was one big surprise, but it was very interesting trying to train things at that size and learning about all sorts of things, from organization to how to babysit such a system and track it. I think things like getting a better understanding of the metrics you're optimizing versus the final capabilities you want—I would say that's still not a perfectly understood mapping, but it's an interesting one we're getting better and better at.

有一种看法认为，其他实验室可能比 DeepMind 在 Gemini 上更高效地使用算力。我不知道你怎么看。

There's a perception that maybe other labs are more compute-efficient than DeepMind has been with Gemini. I don't know what you make of that.

我不这么认为。我认为 Gemini 1 使用的算力大致与传闻中 GPT-4 使用的相当，可能略多一点。我不知道确切数字，所以我认为它们在同一个量级。我认为我们在算力使用上非常高效，而且我们把算力用于很多事情。不仅仅是 Scaling（规模扩张），还有更早的创新和想法——你必须明白，一项新的创新或发明只有能够规模化才有用。所以从某种意义上说，你还需要相当多的算力来进行新发明，因为你必须至少在合理的规模上测试许多东西，并确保它们在该规模下有效。此外，有些新想法可能在玩具规模下无效，但在更大规模下有效，而实际上这些想法更有价值。所以如果你考虑这个探索过程，你需要相当多的算力才能做到这一点。好消息是，我认为我们在 Google 非常幸运，今年我们肯定将拥有迄今为止任何研究实验室中最多的算力，我们希望非常高效地利用这些算力，用于 Scaling（规模扩张）、系统能力以及新发明。

I don't think that's the case. I think Gemini 1 used roughly the same amount of compute, maybe slightly more, than what was rumored for GPT-4. I don't know exactly what was used, so I think it's in the same ballpark. I think we're very efficient with our compute, and we use our compute for many things. One is not just the scaling, but going back to earlier, these innovations and ideas—you've got to, you know, it's only useful if a new innovation or invention can also scale. So in a way, you also need quite a lot of compute to do new invention, because you've got to test many things at least at some reasonable scale and make sure they work at that scale. Also, some new ideas may not work at a toy scale but do work at a larger scale, and in fact those are the more valuable. So if you think about that exploration process, you need quite a lot of compute to be able to do that. The good news is, I think we're pretty lucky at Google that we this year certainly are going to have the most compute by far of any sort of research lab, and we hope to make very efficient and good use of that in terms of both scaling and the capability of our systems and also new inventions.

如果你回到 2010 年创办 DeepMind 的时候，你对 AI 进展的预期是什么？你当时是否预见到它会在很大程度上相当于在这些模型上投入数十亿美元，还是你有不同的看法？

What's been the biggest surprise to you if you go back to yourself in 2010 when you were starting DeepMind, in terms of what AI progress would look like? Did you anticipate back then that it would in some large sense amount to spending billions of dollars into these models, or did you have a different sense of what it would look like?

我们当时认为——实际上，我知道你采访过我的同事 Shane，他总是从算力曲线的角度思考，然后可能粗略地与大脑进行比较，比如有多少神经元和突触。但有趣的是，我们现在确实处于那种状态，大致与大脑中突触的数量和我们拥有的算力处于同一数量级。但更根本的是，我们一直认为我们押注于通用性和学习。这些始终是我们所用任何技术的核心。这就是为什么我们选择了强化学习、搜索和深度学习这三种算法，它们能够规模化，非常通用，并且不需要大量手工编码的人类先验知识，我们认为这是 90 年代构建 AI 努力失败的原因，比如 MIT 的那些基于逻辑的系统、专家系统，大量手工编码的人类信息输入其中，结果证明是错误的或过于僵化。所以我们想摆脱那种方式。我认为我们很早就发现了这个趋势，并且——显然我们以游戏作为试验场，并且做得很好。我认为所有这些都非常成功，也许还启发了其他人。像 AlphaGo 这样的东西，我认为是一个重要时刻，激励了许多其他人思考，‘哦，实际上这些系统已经准备好规模化。’然后当然随着 Transformer 的出现，由我们在 Google Research 和 Brain 的同事发明，那是一种深度学习，使我们能够吸收海量信息，这极大地加速了我们今天的发展。所以我认为这都是同一脉络的一部分。我们无法预测每一个曲折，但我认为我们前进的大方向是正确的。

We thought that—and actually, if you know, I know you've interviewed my colleague Shane, and he always thought in terms of compute curves and then maybe comparing roughly to the brain and how many neurons and synapses there are, very loosely. But we're actually interestingly in that kind of regime now, roughly in the right order of magnitude of the number of synapses in the brain and the sort of compute that we have. But I think more fundamentally, we always thought that we bet on generality and learning. Those were always at the core of any technique we would use. That's why we triangulated on reinforcement learning and search and deep learning as three types of algorithms that would scale and would be very general and not require a lot of handcrafted human priors, which we thought was the sort of failure mode of the efforts to build AI in the '90s, places like MIT, where there were logic-based systems, expert systems, masses of hand-coded human information going into that, which turned out to be wrong or too rigid. So we wanted to move away from that. I think we spotted that trend early and became—and obviously we used games as our proving ground and we did very well with that. I think all of that was very successful and maybe inspired others. Things like AlphaGo, I think, was a big moment for inspiring many others to think, 'Oh, actually these systems are ready to scale.' And then of course with the advent of Transformers, invented by our colleagues at Google Research and Brain, that was the type of deep learning that allowed us to ingest massive amounts of information, and that turbocharged where we are today. So I think that's all part of the same lineage. We couldn't have predicted every twist and turn, but I think the general direction we were going in was the right one.

这很迷人，因为如果你读你以前的论文或 Shane 的旧论文——Shane 的论文，我想是 2009 年，他说，‘嗯，我们测试 AI 的方式是看你能不能压缩维基百科，’而这正是大语言模型的损失函数。或者你 2016 年在 Transformer 之前的论文，你在比较神经科学和 AI，你说‘注意力机制是需要的。’没错。所以我们早就指出了这些东西，实际上我们有一些早期的注意力机制论文，但它们最终不如 Transformer 优雅，比如神经图灵机之类的。然后 Transformer 是更漂亮、更通用的架构。

It's fascinating because if you read your old papers or Shane's old papers—Shane's thesis, I think in 2009, he said, 'Well, the way we would test for AI is if you can compress Wikipedia,' and that's literally the loss function of LLMs. Or your own paper in 2016 before Transformers, where you were comparing neuroscience and AI, and you said 'attention is what is needed.' Exactly. So we had these things called out, and actually we had some early attention papers, but they weren't as elegant as Transformers in the end, like neural Turing machines and things like that. And then Transformers was the nicer and more general architecture of that.

没错，没错。

Exactly, exactly.

当你把这一切向前推演，思考超级智能时，你看到的图景是什么样的？它仍然由私人公司控制吗？它的治理应该是什么样的？

When you extrapolate all this out forward and you think about superhuman intelligence, what does that landscape look like to you? Is it still controlled by a private company? What should the governance of that look like?

听着，我希望——我认为这必须是——这项技术如此重要。我认为它比任何一家公司甚至整个行业都要大得多。我认为它必须是一个与来自公民社会、学术界、政府的许多利益相关者的大型合作。好消息是，我认为随着最近聊天机器人系统等的普及，这已经唤醒了社会的许多其他部分，让他们意识到这即将到来，以及与这些系统交互会是什么样子。这很好，所以它为非常好的对话打开了许多大门。一个例子是几个月前在英国布莱切利公园举办的安全峰会，我认为这是一个巨大的成功，开启了这种国际对话。我认为整个社会都需要参与决定我们想要将这些模型用于什么，我们想要如何使用它们，我们不想将它们用于什么。我认为我们必须尝试就此达成一些国际共识，并确保……

Look, I would love—I think this has to be—this is so consequential, this technology. I think it's much bigger than any one company or even industry in general. I think it has to be a big collaboration with many stakeholders from civil society, academia, government. And the good news is, I think with the popularity of the recent chatbot systems and so on, that has woken up many of these other parts of society that this is coming and what it would be like to interact with these systems. That's great, so it's opened up lots of doors for very good conversations. An example of that was the Safety Summit at Bletchley Park in the UK hosted a few months ago, which I thought was a big success to start getting this international dialogue going. I think the whole of society needs to be involved in deciding what we want to deploy these models for, how we want to use them, what we do not want to use them for. I think we've got to try and get some international consensus around that, and also making sure that the...

这些系统的益处应该惠及每一个人，为了全人类和整个社会的利益。这就是为什么我大力推动像“AI for Science”这样的项目。我希望通过我们的衍生公司 Isomorphic，我们能开始用 AI 治愈可怕的疾病，加速药物发现。还有气候变化等重大挑战——这些人类面临的巨大挑战——我乐观地认为我们可以解决，因为我们有了 AI 这个极其强大的工具。我们可以用它来帮助解决许多问题。理想情况下，我们应该就此达成广泛共识，进行大规模讨论，如果可能的话，甚至达到联合国层面的讨论。

Benefits of these systems should benefit everyone, for the good of everyone and society in general. That's why I push hard on things like AI for Science. I hope that with our spin-out Isomorphic, we'll start curing terrible diseases with AI and accelerate drug discovery. Amazing things like climate change and other big challenges facing humanity—massive challenges—I'm optimistic we can solve because we have this incredibly powerful tool coming along: AI. We can apply it to help solve many of these problems. Ideally, we would have a big consensus around that, a big discussion, almost at the UN level if possible.

有趣的是，这些系统非常强大和智能，但还没有自动化经济的大部分领域。五年前，如果我给你看 Gemini，你会觉得它要取代很多东西。你怎么解释这个？为什么它还没有产生更广泛的影响？

It's interesting that these systems are immensely powerful and intelligent, but they haven't automated large sections of the economy yet. Five years ago, if I showed you Gemini, you'd think it was coming for a lot of things. How do you account for that? What's going on that it hasn't had broader impact yet?

我认为这表明我们仍处于这个新时代的起点。对于这些系统，有一些有趣的用例，比如总结内容或简单写作，但这只是我们日常工作的一小部分。对于更通用的用例，我们需要新的能力：规划、搜索、个性化和情景记忆——不仅仅是上下文窗口，而是记住我们 100 次对话前说过的话。一旦这些能力实现，我期待推荐系统能帮我找到更好的书籍、电影、音乐。我认为我们才刚刚触及 AI 助手在日常生活和工作中能做什么的表面。它们还不够可靠，无法用于科学，但一旦我们解决了事实性和接地问题，它们可能成为科学家或临床医生最好的研究助手。

I think it shows we're still at the beginning of this new era. For these systems, there are interesting use cases like summarizing stuff or simple writing, but that's only a small part of what we do every day. For more general use cases, we need new capabilities: planning, search, personalization, and episodic memory—not just context windows but remembering what we spoke about 100 conversations ago. Once those come in, I'm looking forward to recommendation systems that help me find better books, films, music. I think we're just scratching the surface of what AI assistants could do in our everyday lives and work. They're not reliable enough yet for science, but once we fix factuality and grounding, they could become the world's best research assistant for scientists or clinicians.

我想问关于记忆的问题。你在 2007 年发表了一篇引人入胜的论文，关于记忆和想象之间的联系，它们在某种程度上非常相似。人们经常声称这些模型只是在记忆。你怎么看？记忆就是一切吗？

I want to ask about memory. You had a fascinating paper in 2007 about the links between memory and imagination, how they are very similar. People often claim these models are just memorizing. How do you think about that? Is memorization all you need?

在极限情况下，人们可以尝试记住一切，但无法泛化到分布之外。早期的批评是这些系统只是在复述，但显然 Gemini 和 GPT-4 的新时代正在泛化到新的结构。在我的论文和那篇论文中，我展示了人类记忆是一个重建过程，而不是录像带。我们从熟悉的组件中拼凑起来。这让我认为想象是同样的过程，但使用语义组件来组装新颖的东西用于规划。我认为这个想法——将世界模型的不同部分组合起来模拟新事物以帮助规划，我称之为想象——在当前的系统中仍然缺失。

At the limit, one could try to memorize everything, but it wouldn't generalize out of distribution. Early criticisms were that these systems were just regurgitating, but clearly the new era of Gemini and GPT-4 are generalizing to new constructs. In my thesis and that paper, I showed that human memory is a reconstructive process, not a videotape. We put it together from familiar components. That made me think imagination is the same, but using semantic components to assemble something novel for planning. I think that idea—pulling together different parts of your world model to simulate something new for planning, which I call imagination—is still missing from current systems.

你们拥有世界上最好的模型，比如 Gemini。你们是否计划像其他主要 AI 实验室那样，制定一个框架，除非有特定的安全措施，否则不会继续开发或发布产品？

You guys have the best models in the world with Gemini. Do you plan on putting out a framework like other major AI labs, where you won't continue development or ship the product unless specific safeguards are in place?

是的，我们有内部的检查和平衡机制，但我们将在未来几个月开始发布博客文章和技术论文，类似于负责任的缩放定律。我们在内部已经隐含地有了这些，还有各种安全委员会。是时候更公开地谈论这些了，所以我们将在今年全年这样做。

Yes, we have internal checks and balances, but we're going to start publishing blog posts and technical papers in the next few months along the lines of responsible scaling laws. We have those implicitly internally and various safety councils. It's time for us to talk about that more publicly, so we'll be doing that throughout the year.

另一个担忧是恶意行为者或外国特工窃取权重并微调用于有害目的。你如何看待保护权重？

Another concern is rogue actors or foreign agents stealing the weights and fine-tuning them for harmful purposes. How do you think about securing the weights?

有两个部分：安全和开源。安全是关键——正常的网络安全。我们很幸运，在 Google DeepMind，我们有 Google 的防火墙和云保护，这是业界一流的。在那之后，我们在代码库中有特定的保护。这是双重保护。我感觉很好，但我们不能自满。我们需要不断改进，比如强化沙箱，甚至可能使用专门的安全硬件。

There are two parts: security and open source. Security is key—normal cybersecurity. We're lucky at Google DeepMind to be behind Google's firewall and cloud protection, which is best in class. Behind that, we have specific protections within our codebase. It's double protection. I feel pretty good about that, but we can never be complacent. We need to keep improving, with things like hardened sandboxes and maybe even specifically secure hardware.

我们也在考虑数据中心或硬件解决方案。我认为在未来三到五年内，我们可能还需要气隙和其他安全社区已知的各种措施。所以我认为这很关键，所有前沿实验室都应该这样做，否则国家行为体和流氓国家等危险行为者会有很大动机去窃取权重之类的东西。当然，开源是另一个有趣的问题。我们是开源和开放科学的坚定支持者。我们发表了数千篇论文，像 AlphaFold、Transformer、AlphaGo 这些东西都发布到了世界上，很多都开源了，比如我们的天气预报系统 GraphCast。但涉及到核心技术、非常通用的基础技术时，我想问开源支持者一个问题：如何阻止坏的行为者——个人或流氓国家——利用这些开源系统并将其用于有害目的？我们必须回答这个问题。我还没有从主张完全开源的人那里听到令人信服的明确答案。所以我认为必须有所平衡。显然，这是一个复杂的问题。

We're also thinking about data centers or hardware solutions. I think that maybe in the next three, four, five years we would also want air gaps and various other things that are known in the security community. So I think that's key, and I think all frontier labs should be doing that, because otherwise nation states and rogue states and other dangerous actors would have a lot of incentive to steal things like the weights. And then, of course, open source is another interesting question. We're huge proponents of open source and open science. I mean, we've published thousands of papers, and things like AlphaFold, Transformers, AlphaGo—all of these we put out into the world, published and open-sourced many of them, like GraphCast, our weather prediction system. But when it comes to the core technology, the foundational technology in very general purpose, I think the question I would have is: for open source proponents, how does one stop bad actors—individuals or rogue states—from taking those same open source systems and repurposing them for harmful ends? We have to answer that question. I haven't heard a compelling clear answer from proponents of open-sourcing everything. So I think there has to be some balance there. Obviously, it's a complex question.

是的，我觉得科技界在投入数千亿美元的研发方面没有得到应有的认可。但当我们谈论保护权重时，也许现在这还不是会导致世界末日的事情，但随着这些系统变得更好，担心的是外国特工会接触到它们。目前大概有几十到几百名研究人员可以访问权重。你们计划如何实现这样一种状态：访问权重需要极其严格的过程，没有任何个人能真正把它们带出去？

Yeah, I feel like tech doesn't get the credit it deserves for funding hundreds of billions of dollars worth of R&D. But when we talk about securing the weights, maybe right now it's not something that's going to cause the end of the world, but as these systems get better, the worry is that a foreign agent gets access to them. Presumably, right now there are dozens to hundreds of researchers who have access to the weights. How do you plan to get into a situation where accessing the weights requires an extremely strenuous process, where no individual can really take them out?

必须在允许协作和进展速度之间取得平衡。另一个有趣的事情是，你希望来自学术界或英国 AI 安全研究所和美国机构的杰出独立研究人员能够对这些系统进行红队测试，所以必须在一定程度上暴露它们，尽管这不一定是权重。我们有很多流程来确保只有需要访问的人才能访问。目前，我认为我们仍处于这类系统面临风险的早期阶段。随着这些系统变得更强大、更通用、更有能力，我认为必须审视访问问题。

One has to balance that with allowing for collaboration and speed of progress. Another interesting thing is that you want brilliant independent researchers from academia or the UK AI Safety Institute and US ones to be able to red-team these systems, so one has to expose them to a certain extent, although that's not necessarily the weights. We have a lot of processes in place about making sure that only those who need access have access. Right now, I think we're still in the early days of those kinds of systems being at risk. As these systems become more powerful, more general, and more capable, I think one has to look at the access question.

其他一些实验室在安全方面有专长，比如 Anthropic 在可解释性方面。你们觉得自己可能在哪些方面有优势？既然你们有了前沿模型，你们将扩大安全规模。你们能在哪些方面推出最好的前沿研究？

Some other labs have specialized in different things relative to safety, like Anthropic with interpretability. Do you have a sense of where you guys might have an edge? Now that you have the frontier model, you're going to scale up safety. Where will you be able to put out the best frontier research?

我们帮助开创了基于人类反馈的强化学习（RLHF）和其他既可用于性能也可用于安全的技术。我认为很多自我对弈的想法也可以用于自动测试新系统的许多边界条件。部分问题在于，这些非常通用的系统有太多需要覆盖的行为面。所以我认为我们需要一些自动化测试。在模拟和游戏等非常逼真的虚拟环境方面，我们有悠久的历史，并利用这类系统构建 AI 算法。所以我认为我们可以利用所有这些历史。在谷歌，我们很幸运拥有一些世界顶级的网络安全专家和硬件设计师，所以我认为我们也可以将这些用于安全和安保。

We helped pioneer RLHF and other things that can be used for performance but also for safety. I think a lot of the self-play ideas and these kinds of things could also be used potentially to auto-test a lot of the boundary conditions that you have with new systems. Part of the issue is that with these very general systems, there's so much surface area to cover about how these systems behave. So I think we are going to need some automated testing. With things like simulations and games, very realistic virtual environments, we have a long history in that and using those kinds of systems for building AI algorithms. So I think we can leverage all of that history. And at Google, we're very lucky to have some of the world's best cyber security experts and hardware designers, so I think we can bring that to bear for security and safety as well.

太好了，我们来谈谈 Gemini。现在你们拥有世界上最好的模型。与这些系统交互的默认方式一直是聊天。现在我们有了多模态和所有这些新能力，你预计这会如何改变？还是你认为情况仍然如此？

Great, let's talk about Gemini. Now you guys have the best model in the world. The default way to interact with these systems has been through chat. Now that we have multimodal and all these new capabilities, how do you anticipate that changing? Or do you think that'll still be the case?

我认为我们才刚刚开始真正理解与完整多模态模型系统交互会是什么样子，这将与我们今天习惯的聊天机器人截然不同。我认为未来一年或 18 个月内的下一个版本可能会通过摄像头或手机对你周围的环境有一些上下文理解。我可以想象下一款很棒的眼镜。然后我们会开始更流畅地理解——从视频中采样，使用语音，甚至最终可能包括触觉。如果你考虑机器人和其他传感器，我认为未来几年世界将变得非常令人兴奋，因为我们开始习惯真正多模态的含义。

I think we're just at the beginning of actually understanding what a full multimodal model system might be like to interact with, and it'll be quite different from what we're used to today with chat bots. I think the next versions over the next year or 18 months might have some contextual understanding around the environment around you through a camera or a phone. I could imagine the next awesome glasses. Then I think we'll start becoming more fluid in understanding—let's sample from a video, let's use voice, maybe even eventually things like touch. And if you think about robotics and other sensors, I think the world's about to become very exciting in the next few years as we start getting used to the idea of what true multimodality means.

关于机器人学，伊利亚在播客中说，OpenAI 放弃机器人学的原因是他们没有足够的数据，至少在他们追求的时候是这样。你们推出了不同的东西，比如 Robo Transformer 等。你认为这仍然是机器人学进步的瓶颈吗？还是我们也会看到机器人领域的进步？

On the robotics subject, Ilya said when he was on the podcast that the reason OpenAI gave up on robotics was because they didn't have enough data in the domain, at least at the time they were pursuing it. You guys have put out different things like Robo Transformer and other things. How do you think that's still a bottleneck for robotics progress? Or will we see progress in the world of robots as well?

我们对 RT-2（机器人 Transformer）等进展感到非常兴奋。我们一直喜欢机器人学，并且在这方面有出色的研究，现在仍在继续，因为我们喜欢它数据匮乏的特点，因为这推动我们解决一些基础研究问题。

We're very excited about our progress with things like RT-2, the robotic Transformer. We've always liked robotics and we've had amazing research in that, and we still have that going now because we like the fact that it's a data-poor regime, because that pushes us on some fundamental research questions.

关于我们认为无论如何都会有用的非常有趣的研究方向，比如采样效率和数据效率、迁移学习、从模拟中学习并迁移到现实——这些都是我们想要解决的一般性挑战。所以控制问题，我们一直在这方面努力。实际上，我认为伊利亚说得对，由于数据问题，这更具挑战性，但我们也开始看到这些大模型可迁移到机器人领域的苗头，在通用领域、语言领域和其他领域学习，然后只是把词元当作任何类型的词元——词元可以是动作、单词、图像的一部分、像素或其他任何东西。我认为这才是真正的多模态。一开始，训练这样的系统比训练一个简单的文本语言系统更难，但回到我们之前关于迁移学习的讨论，你会开始看到，一个真正的多模态系统，其他模态会受益于不同的模态，所以你会因为对视频有了一点理解而变得在语言上更擅长。所以我认为起步更难，但最终我们会拥有一个更通用、更强大的系统。

On very interesting research directions that we think are going to be useful anyway, like sampling efficiency and data efficiency in general, transfer learning, learning from simulation, transferring that to reality—all of these very interesting general challenges that we would like to solve. So the control problem, we've always pushed hard on that. And actually, I think Ilia is right that that is more challenging because of the data problem, but it's also, I think, we're starting to see the beginnings of these large models being transferable to the robotics regime, learning in the general domain, language domain, and other things, and then just treating tokens like any type of token—the token could be an action, it could be a word, it could be a part of an image, a pixel, or whatever it is. And that's what I think true multimodality is. To begin with, it's harder to train a system like that than a straightforward text language system, but actually, going back to our early conversation of transfer learning, you start seeing that a true multimodal system, the other modality benefits some different modalities, so you get better at language because you now understand a little bit about video. So I do think it's harder to get going, but actually ultimately, we'll have a more general, more capable system like that.

Gato 后来怎么样了？那非常吸引人，你可以让它玩游戏，还能处理视频，还能……

Whatever happened to Gato? That was super fascinating, that you could have it play games and also do video and also do...

是的，我们仍在研究这类系统。但你可以想象，我们正试图将这些想法融入未来的 Gemini 版本中，使其能够完成所有这些任务，而机器人 Transformer 之类的东西可以说是它的后续发展。

Yeah, we're still working on those kinds of systems. But you can imagine, we're trying to build those ideas into our future generations of Gemini, to be able to do all of those things, and Robotics Transformers and things like that are kind of follow-ups to that.

我们看到在那些你提到的自我对弈类方法会特别强大的领域，进展是不对称的。比如数学和代码，显然最近你们有相关论文，可以用这些东西做很酷的新事情。它们会像超人般的程序员吗？但在其他方面可能仍然不如人类？你怎么看？

We see asymmetric progress towards the domains in which the self-play kinds of things you're talking about will be especially powerful. So math and code, obviously recently you have these papers out about this, where you can use these things to do really cool novel things. Will they just be like superhuman coders, but in other ways they might be still worse than humans? How do you think about that?

听着，我认为我们在数学、定理证明和编程方面取得了很大进展。但如果你看看一般的创造力和科学探索，情况仍然很有趣。我认为我们正进入这样一个阶段：我们的系统可以帮助最优秀的人类科学家更快地取得突破，比如在某种程度上对搜索空间进行分诊，或者像 AlphaFold 解决蛋白质结构那样找到解决方案。但它们还没有达到能够自己提出假设或提出正确问题的水平。正如任何顶尖科学家会告诉你的，这是科学中最难的部分：实际上提出正确的问题，将空间缩小到我们应该追求的关键问题、关键难题，然后以正确的方式表述问题并攻克它。而我们的系统对此毫无头绪。但它们适合搜索大型组合空间，只要你能以这种方式用明确的目标函数来定义问题。所以这对我们今天处理的许多问题非常有用，但不是对最高层次的创造性问题。

So look, I think that we're making great progress with math and theorem proving and coding. But it's still interesting if one looks at creativity in general and scientific endeavor in general. I think we're getting to the stage where our systems could help the best human scientists make their breakthroughs quicker, like almost triage the search space in some ways, or perhaps find a solution like AlphaFold does with a protein structure. But they're not at the level where they can create the hypothesis themselves or ask the right question. And as any top scientist will tell you, that's the hardest part of science: actually asking the right question, boiling down that space to like what's the critical question we should go after, the critical problem, and then formulating that problem in the right way to attack it. And that's not something our systems have any idea how to do. But they are suitable for searching large combinatorial spaces if one can specify the problem in that way with a clear objective function. So that's very useful for many of the problems we deal with today, but not the most high-level creative problems.

那么 Demis，显然你们发表了各种有趣的东西，并在不同领域加速科学进步。如果你认为 AGI 将在未来 10-20 年内实现，为什么不直接等 AGI 来做呢？为什么要构建这些特定领域的解决方案？

So Demis, obviously you've published all kinds of interesting stuff and speeding up science in different areas. How do you think about that in the context of if you think AGI is going to happen in the next 10-20 years, why not just wait for the AGI to do it for you? Why build these domain-specific solutions?

嗯，我认为我们不知道 AGI 需要多长时间。我们以前常说，甚至在创办 DeepMind 时就说，我们不必为了给世界带来巨大好处而等待 AGI。尤其是，我个人热衷于人工智能用于科学和健康，你可以从 AlphaFold 以及我们在不同领域发表的各种《自然》论文、材料科学工作等中看到这一点。我认为有很多令人兴奋的方向，也通过产品对世界产生影响。我认为这非常令人兴奋，是一个巨大的机会，也是我们作为 Google 一部分拥有的独特机会：他们有数十亿用户的产品，我们可以立即将我们的进步部署进去，然后数十亿人可以改善、丰富和提升他们的日常生活。所以我认为这是一个在各个层面产生影响的绝佳机会。另一个原因，从人工智能本身的角度来看，是它对你的想法进行了实战检验。你不想待在一个研究掩体里，理论上推动一些事情，但你的内部指标开始偏离人们关心的现实世界事物或现实世界影响。所以你会从这些现实世界应用中获得大量直接反馈，告诉你你的系统是否真的在扩展，或者我们是否需要更高的数据效率或采样效率，因为大多数现实世界的挑战都需要这些。所以这让你保持诚实，并推动你不断调整和引导你的研究方向，确保它们走在正确的道路上。所以我认为这太棒了，当然世界会从中受益，社会也会在 AGI 到来之前的许多年里受益。

Well, I think we don't know how long AGI is going to be. And we always used to say, back even when we started DeepMind, that we don't have to wait for AGI in order to bring incredible benefits to the world. And especially, my personal passion has been AI for science and health, and you can see that with things like AlphaFold and all of our various nature papers on different domains, our materials science work, and so on. I think there's lots of exciting directions and also impact in the world through products too. I think it's very exciting, a huge opportunity, and a unique opportunity we have as part of Google: they have dozens of billion-user products right that we can immediately ship our advances into, and then billions of people can improve their daily lives, enrich their daily lives, and enhance their daily lives. So I think it's a fantastic opportunity for impact on all those fronts. And I think the other reason, from a point of view of AI specifically, is that it battle-tests your ideas. You don't want to be in a sort of research bunker where you theoretically push things forward, but then your internal metrics start deviating from real-world things that people would care about, or real-world impact. So you get a lot of feedback, direct feedback from these real-world applications, that then tells you whether your systems really are scaling, or do we need to be more data efficient or sample efficient, because most real-world challenges require that. And so it kind of keeps you honest and pushes you to keep nudging and steering your research directions to make sure they're on the right path. So I think it's fantastic, and of course the world benefits from that, society benefits from that on the way, many many years before AGI arrives.

嗯，Gemini 的开发非常有趣，因为它紧随 Brain 和 DeepMind 这两个不同组织的合并。我很好奇，那里有什么挑战？有什么协同效应？而且从你们现在拥有世界上最好的模型这个意义上说，它是成功的。

Well, the development of Gemini is super interesting because it comes right at the heels of merging these different organizations, Brain and DeepMind. I'm curious, what have been the challenges there? What have been the synergies? And it's been successful in the sense that you have the best model in the world now.

嗯，实际上过去一年非常棒。当然，这样做很有挑战性，就像任何大型整合一样。但你说的是两个世界级的组织，有着悠久的历史，发明了许多重要的东西，从深度强化学习到 Transformer。所以把所有这些东西整合在一起，进行更紧密的合作，实际上非常令人兴奋。我们以前也一直在合作，但更多是项目层面的，而不是更深入、更广泛的整合。

Well, look, it's been fantastic actually over the last year. Of course, it's been challenging to do that, like any big integration coming together. But you're talking about two world-class organizations with long storied histories of inventing many many important things, from deep reinforcement learning to Transformers. And so it's very exciting actually pulling all of that together and collaborating much more closely. We always used to be collaborating, but more on a project-by-project basis versus a much deeper, broader integration.

像我们现在这样的合作，Gemini 就是那次合作的第一个成果，包括名字 Gem，暗示双胞胎。当然，还有很多其他事情变得更高效，比如把算力资源、想法和工程整合在一起。在我们现在这个阶段，要构建前沿系统需要大量的世界级工程，我认为更紧密地协调是有意义的。

Collaboration like we have now and Gemini is the first fruit of that collaboration, including the name Gem, implying twins. And of course, a lot of other things are made more efficient, like pooling compute resources together and ideas and engineering. At the stage we're at now, where there's huge amounts of world-class engineering that has to go on to build the frontier systems, I think it makes sense to coordinate that more closely.

你和 Shane 创办 DeepMind 部分是因为你担心安全问题。你看到 AGI 即将成为现实的可能性。你认为以前是 Brain 一部分的人，也就是现在 Google DeepMind 的另一半，他们对待这个问题的方式一样吗？在这个问题上有没有文化差异？

You and Shane started DeepMind partly because you were concerned about safety. You saw AGI coming as a live possibility. Do you think the people who were formerly part of Brain, the other half of Google DeepMind now, do they approach it the same way? Have there been cultural differences there in terms of that question?

是的，不，我认为总体而言，这就是为什么我认为我们在 2014 年与谷歌联手的原因之一是整个谷歌和 Alphabet，不仅仅是 Brain 和 DeepMind，都非常认真地对待这些责任问题。一种座右铭是尝试对这些系统既大胆又负责。我显然是一个巨大的技术乐观主义者，但鉴于我们共同带入世界的变革力量，我希望我们对此保持谨慎。我认为这很重要，再说一次，这将是人类有史以来最重要的技术之一，所以我们必须全力以赴把它做好，并保持深思熟虑，同时对我们所知道的、不知道的即将到来的系统以及相关的不确定性保持谦逊。在我看来，当存在巨大不确定性时，唯一明智的方法是保持谨慎乐观，并用科学方法尽可能多地预见和理解即将发生的事情及其后果。你不希望用这些后果严重的系统在现实世界中进行实时 A/B 测试，因为意外后果可能相当严重。我希望我们作为一个领域摆脱“快速行动，打破常规”的态度，这种态度过去可能对硅谷很有用，并创造了重要的创新，但在这个案例中，我们希望对其能做的积极事情保持大胆，并确保实现医学、科学等进步，同时尽可能负责任和深思熟虑地减轻风险。

Yeah, no, I think overall, and this is why I think one of the reasons we joined forces with Google back in 2014 was that the entirety of Google and Alphabet, not just Brain and DeepMind, take these questions of responsibility very seriously. A kind of mantra is to try and be bold and responsible with these systems. I would class myself as obviously a huge techno-optimist, but I want us to be cautious with that given the transformative power of what we're bringing into the world collectively. I think it's important, again, it's going to be one of the most important technologies humanity will ever invent, so we've got to put all our efforts into getting this right and be thoughtful and also humble about what we know and don't know about the systems that are coming and the uncertainties around that. In my view, the only sensible approach when you have huge uncertainty is to be cautiously optimistic and use the scientific method to try and have as much foresight and understanding about what's coming down the line and the consequences before it happens. You don't want to be live AB testing out in the world with these very consequential systems because unintended consequences might be quite severe. I want us to move away as a field from the 'move fast and break things' attitude, which maybe served the valley well in the past and created important innovations, but in this case, we want to be bold with the positive things it can do and make sure we realize things like medicine and science and advancing all of those things, while being responsible and thoughtful as far as possible with mitigating the risks.

是的，这就是为什么负责任的 Scaling 政策似乎是一种非常好的经验性方式来预先承诺这些事情。

Yeah, and that's why it seems like the responsible scaling policies are a very good empirical way to pre-commit to these kinds of things.

完全正确。

Exactly.

我很好奇，比如，当你在做这些评估时，如果发现你的下一个模型可能帮助一个外行制造出大流行级别的生物武器之类的，你首先会如何考虑确保这些权重安全，不让它泄露出去？其次，要让你放心部署那个系统，需要满足什么条件？你如何确保那种能力不被暴露？

I'm curious if you have a sense of, for example, when you're doing these evaluations, if it turns out your next model could help a layperson build a pandemic-class bioweapon or something, how would you think first of all about making sure those weights are secure so that doesn't get out, and second, what would have to be true for you to be comfortable deploying that system? How would you make sure that that capability isn't exposed?

嗯，首先，安全模型部分我认为我们已经通过网络安全覆盖了，确保它被妥善分类，并且你正在监控所有这些东西。我认为如果这种能力是通过红队测试或政府机构、学术界或独立测试人员的外部测试发现的，那么我们就必须根据具体情况修复那个漏洞。如果这需要一种不同的宪法，或不同的护栏，或更多的 RLHF 来避免，或者删除一些训练数据，可能会有多种缓解措施。第一部分是确保你提前检测到它，所以这关乎正确的评估、基准测试和测试。然后问题是如何在你部署之前修复它。我认为一般来说，如果那是一个暴露面，它需要在部署前被修复。

Well, first, the secure model part I think we've covered with cybersecurity and making sure that's well-classed and you're monitoring all those things. I think if the capability was discovered through red teaming or external testing by government institutes, academia, or independent testers, then we would have to fix that loophole depending on what it was. If that required a different kind of constitution, or different guardrails, or more RLHF to avoid that, or removing some training data, there could be a number of mitigations. The first part is making sure you detect it ahead of time, so that's about the right evaluations, benchmarking, and testing. Then the question is how one would fix that before you deployed it. I think it would need to be fixed before it was deployed generally, if that was an exposure surface.

最后一个问题。你在其他人认为 AGI 很荒谬的 2010 年就已经在思考 AGI 的最终目标了。现在我们看到了这种缓慢的起飞，我们实际上看到了泛化和智能，这在心理上是什么感觉？它是否已经融入了你的世界模型，所以对你来说不是新闻，还是说亲眼看到它，就像哇，真的有什么改变了？感觉如何？

Final question. You've been thinking in terms of the end goal of AGI at a time when other people thought it was ridiculous in 2010. Now that we're seeing this slow takeoff where we're actually seeing generalization and intelligence, what has that been like psychologically? Has it just been priced into your world model so it's not new news for you, or is it actually seeing it live, like wow, something's really changed? What does it feel like?

是的，嗯，它已经融入了我的世界模型，至少从技术方面来看事情会如何发展。但显然，我们不一定预料到公众会这么早对这类事情产生兴趣。比如，如果 ChatGPT 和聊天机器人没有获得它们最终获得的那种兴趣——我认为每个人都相当惊讶人们已经准备好使用这些东西，尽管它们在某些方面还有所欠缺，尽管它们令人印象深刻——那么我们可能会生产更多基于主轨道的专门系统，比如 AlphaFold 和 AlphaGo 以及我们的科学工作。然后公众可能只有在几年后，当我们有更通用的有用助手型系统时才会关注。所以这很有趣，它创造了一个不同的环境，我们现在作为一个领域都在其中运作。它有点混乱，因为发生了太多事情，太多风险投资涌入，每个人几乎都为此疯狂。我唯一担心的是，我希望确保作为一个领域，我们对此负责任、深思熟虑且科学地行事，并用科学方法以乐观但谨慎的方式处理。我一直相信这是对待像 AI 这样的东西的正确方法，我只是希望这不会在巨大的热潮中丢失。

Yeah, well, it's already priced into my world model of how things were going to go, at least from the technology side. But obviously, we didn't necessarily anticipate that the general public would be that interested this early in the sequence. Things like maybe if ChatGPT and chatbots hadn't gotten the kind of interest they ended up getting, which I think was quite surprising to everyone that people were ready to use these things even though they were lacking in certain directions, impressive though they are, then we would have produced more specialized systems built off the main track, like AlphaFold and AlphaGo and our scientific work. Then the general public may have only paid attention later down the road, in a few years time, when we have more generally useful assistant-type systems. So that's been interesting, and it's created a different type of environment that we're now all operating in as a field. It's a little more chaotic because there's so many more things going on, so much VC money going into it, and everyone's almost losing their minds over it. The only thing I worry about is I want to make sure that as a field we act responsibly, thoughtfully, and scientifically about this, and use the scientific method to approach this in an optimistic but careful way. I've always believed that's the right approach for something like AI, and I just hope that doesn't get lost in this huge rush.

当然，当然。好吧，我认为这是一个很好的结束点。Demis，非常感谢你的时间，感谢你来做客播客。

Sure, sure. Well, I think that's a great place to close. Demis, thank you so much for your time and for coming on the podcast.

谢谢，这真的很愉快。

Thanks, it's been a real pleasure.

希望你喜欢这期节目。一如既往，你能做的最有帮助的事情就是分享这个播客，把它发给你觉得可能喜欢的人，放到推特、群聊里等等。帮忙传播一下。感谢你的收听。下次见。干杯！

Hope you enjoy that episode. As always, the most helpful thing you can do is just share the podcast, send it to people you think might enjoy it, put it on Twitter, your group chats, etc. Just spread the word. I appreciate you listening. I'll see you next time. Cheers!