Jeff Dean 谈 AI：从神经网络到 Gemini

下午好。虽然还没到 4:30，但既然房间已经满了，我就开始介绍，节省时间。非常荣幸介绍杰夫·迪恩，谷歌研究和谷歌 DeepMind 的首席科学家，也是 Gemini 项目的联合负责人。在杰夫·迪恩成为谷歌名人之前，我其实在 90 年代就认识他了，当时他在数字西部研究公司工作，所以我可以确认这不是一夜成名的故事。杰夫长期以来一直默默无闻地杰出。杰夫的简历，你们可能在宣传单上读过，读起来就像现代计算的一次导览：从大规模信息检索、操作系统、分布式系统、编译器，然后这还不够，他的第二职业生涯是 AI、机器学习加速器、视觉模型、底层软件、蒸馏、神经网络架构搜索、大语言模型、多模态模型，以及从芯片设计到医疗、翻译等应用。我最喜欢关于杰夫的一个细节是，他的本科论文是关于神经网络的，这意味着两件事：第一，他在神经网络流行之前就研究了它；第二，他拥有罕见的资格，可以说他在谷歌让神经网络流行之前就喜欢它了。也许杰夫是自己做到的。但如果你想知道如何拥有好的研究品味，显然答案是尽早开始，成为杰夫。一路走来，杰夫获得了 ACM 计算奖、IEEE 奖章，他是 ACM 会士、美国国家工程院院士和美国艺术与科学院院士。我们非常荣幸邀请他今天做杰出讲座。请大家和我一起欢迎杰夫。

Good afternoon. It's not 4:30 yet, but since the room is already full, I'll just start the introduction to save some time. It's a real pleasure to introduce Jeff Dean, Chief Scientist for Google Research and Google DeepMind, and also co-lead of the Gemini project. I would say before the now famous Jeff Dean at Google era, I actually met Jeff in the 90s when Jeff was at Digital Western Research, so I can confirm this is not an overnight success story. Jeff has been quietly exceptional for a very long time. Jeff's bio, which you probably read in the flyer, reads a little bit like a guided tour of modern computing: from large-scale information retrieval, operating systems, distributed systems, compilers, and then if that wasn't enough, the second career is AI, machine learning accelerators, vision models, low-level software, distillation, neural network architecture search, large language models, multimodal models, and applications ranging from chip design to healthcare, translation, and so on. One of my favorite details about Jeff is that he did his undergraduate thesis on neural networks, which means two things: first, he worked on neural networks before it was cool, and second, he has the rare distinction of being able to say he liked neural networks before Google made it popular. Maybe Jeff did that to himself. But if you're wondering what it takes to have good research taste, apparently the answer is to start early and be Jeff. Along the way, Jeff has received the ACM Prize in Computing, the IEEE Medal, he's an ACM Fellow, a member of the US National Academy of Engineering and the American Academy of Arts and Sciences. We're very fortunate to have him give the distinguished lecture today. Please join me in welcoming Jeff.

好的。非常感谢你的热情介绍和邀请。我意外来到城里，凯说：‘嘿，你想过来做个讲座吗？’所以我很高兴来到这里。我将和大家聊聊 AI 领域正在发生的事情。我不知道你们有没有注意到，但这是当下的热门话题。谷歌有很多人在这个领域工作，所以我展示的不仅是我个人的工作，还有谷歌和其他地方许多人的工作。

All right. Thank you very much for that kind introduction and for hosting me. I'm unexpectedly in town and Kai said, 'Hey, you want to come by and give a talk?' So I'm delighted to be here. I'm going to talk to you about what's happening in AI. I don't know if you've noticed, but it's a thing these days. We have many people at Google working on this area, so I'm presenting the work of not just my work, but many many people at Google and elsewhere.

一些观察结果可能不会让你们感到太惊讶。机器学习和 AI 确实改变了我们对计算机能力的期望。回想 10 到 15 年前，计算机的视觉能力很差，语音理解也不太好，语言理解更是糟糕。而现在，我们有了工具，让计算机能够以更自然的方式与人互动，因为它们理解了人们理解和想要交流的模态。在这个过程中，更大规模的算力、更多的数据和更大规模的模型，在过去 13 到 14 年里几乎持续地带来了更好的结果。这一趋势一直持续，直到今天仍然如此。除了规模，算法和模型架构的改进也带来了巨大的提升。这两者相互叠加。如果你能拥有更大的规模和更好的模型架构，使得每次浮点运算能学到更多，那么结合起来，就不是规模带来的 20 倍提升，而是规模 20 倍和算法 50 倍，最终获得比不久前好一千倍的结果。此外，我们想要运行的计算类型以及运行它们的硬件也在发生巨大变化。

A few observations which may not come as a huge surprise to you. Machine learning and AI have really changed our expectations of what is possible with computers. If you think back 10-15 years ago, computers couldn't see very well. They couldn't really understand speech all that well. They definitely couldn't understand language very well. And now we have tools that enable computers to interact with people in a much more natural style because they understand the modalities that people understand and want to communicate with. Along the way, increasing scale of larger amounts of compute, larger amounts of data, and more large-scale models have delivered better results nearly continuously for the last 13 or 14 years. This has been the trend and seems to be holding up even today. In addition to scale, algorithmic and model architecture improvements have also provided massive improvements. These two things multiply together. If you can have much larger scale and much better model architectures that are able to learn more per floating point operation, those things together mean instead of 20x from scale, you have 20x from scale and 50x from algorithms, and you end up with a thousand times better thing than you had not that long ago. Also, the kinds of computations we want to run and the hardware in which we want to run them are changing dramatically.

四个观察结果可能并不令人惊讶。首先，神经网络和梯度下降。这些似乎是构建真正智能计算机系统的关键构建块。它们不是新想法；已经存在很长时间了，发明于 70 年代和 80 年代。实际上，在 80 年代末和 90 年代初，人们对它们非常兴奋。我第一次接触它们是在一次入门讲座中，当时对神经网络的介绍大致是这样的：我们有一个很好的、非常抽象的版本，关于我们如何认为真实神经元的行为，你有一堆输入，进行一些处理，然后决定是否触发以及以什么强度触发，并且这些神经元之间有连接。然后反向传播是一个非常漂亮的算法，它使你能够调整所有这些连接边上的权重，以使模型的行为更符合你的期望，你给它一堆例子，反向传播使模型能够学习你想要的行为。我在 1990 年作为本科生看到了这些，我非常兴奋，说：‘哦，哇，这看起来是一个非常棒的抽象。感觉它是一个可训练的东西，几乎可以学习任何东西。’那时我们没有使用非常大的神经网络，因为我们的计算机非常弱小。但我说：‘哦，也许如果我们使用系里的并行计算机，我可以训练更大的神经网络。那太好了。也许我们只需要用 32 台计算机而不是一台，然后我们就可以训练出非常令人印象深刻的神经网络。那会很棒。’所以我做了。我问了在我上的一门课中介绍神经网络主题的教授，Vipin Kumar，我是否可以和他一起做毕业论文。他说：‘当然可以。’所以我尝试了两种不同的并行训练神经网络的方法。一种是现在所谓的模型并行和数据并行训练，但我称之为模式分区和流水线方法。这些加速曲线很有趣，但它们趋于平缓，因为我做了一件蠢事：我没有随着处理器数量的增加而让模型变得更大。我只是让固定大小的模型更难并行化。但我把它归档为‘哦，这是一个有趣的抽象，似乎我应该关注它’，但之后我在接下来的 25 年里忽略了它。

Four observations which are maybe not surprising. First, neural nets and gradient descent. These seem to be key building blocks for how to actually build truly intelligent computer systems. They're not new ideas; they've been around for a long time, invented in the 70s and 80s. There was actually a lot of excitement about them in the late 80s and early 90s. I first ran across them in an intro lecture where I was introduced to neural networks roughly like this: we have this kind of nice, very abstracted away version of how we think real-life neurons behave, where you have a bunch of inputs, you do some processing, and you decide whether you are going to fire or not and at what strength, and you have connections of these things. Then backpropagation is a really nice algorithm that enables you to adjust the weights on the edges of all these connections in order to make the model behave more in the way that you want it to behave, where you give it a bunch of examples and backpropagation enables you to make the model learn the behavior you want. I saw these as an undergrad in 1990 and I got super excited and said, 'Oh wow, that seems like a really good abstraction. It feels like it's a trainable thing and it can learn almost anything.' At that time we were not using very large neural networks because we had really pathetically wimpy computers. But I said, 'Oh, maybe if we use the parallel computer in the department, I could train much bigger neural networks. That would be great. Maybe we just need to use 32 computers instead of one and then we could train really impressive neural nets. That would be nice.' So I did it. I asked the professor who introduced the neural networks topic in one of the classes I was taking if I could do a senior thesis with him, Vipin Kumar. He said, 'Sure.' So I played around with two different approaches for training neural networks in parallel. One is what you would now call model parallel and data parallel training, but I called them pattern partitioned and pipelined approach. These speedup curves are interesting but they taper off because I did the stupid thing: I didn't make the model bigger and bigger as I increased the number of processors. I just made it much harder to parallelize with a fixed size model. But I sort of filed that away as 'oh that's an interesting abstraction and it seems like something I should pay attention to,' but then I ignored it for the next 25 years.

好的。现在我将带你们快速浏览一下过去 15 年机器学习领域的一系列进展，这些进展共同造就了今天最复杂的模型。今天的模型是如何产生的？正是通过我将要描述的这些进步。

Okay. So now I'm going to give you a whirlwind tour of a bunch of advances in the field of ML that have happened in the last 15 years that are really all combined together to produce today's most sophisticated models. How did these today's models come to be? Well, it was through advancements like the ones I'm going to describe.

所以，我重新开始研究神经网络的原因是在谷歌的茶水间遇到了斯坦福教授吴恩达。我问他：‘你在这儿干嘛？’他说：‘还不确定’，因为他刚入职一周。但他的学生在用神经网络做语音和视觉方面的有趣研究。我说：‘那太酷了，我们应该训练真正大的神经网络。’这就是起源，因为当时我们有很多计算机。于是我们构建了一个软件系统，能够表达神经网络及其连接，并在多台计算机上进行模型并行和数据并行训练。这使我们能训练比当时已知最大网络大 50 到 100 倍的神经网络。我们的做法是：模型有多个异步副本，每个副本获取当前参数更新，在本地批次数据上计算梯度，然后发送给分布式参数服务器更新参数。这是异步的，从数学上讲肯定不对，因为其他副本更新参数时模型参数已经变了。但它似乎有效，所以我们还是很满意，尽管理论上不完美。

So, the origin of my getting back to working on neural nets at Google was I bumped into Andrew Ng, who's a Stanford faculty member, in one of our micro kitchens and I said, "Oh, what are you doing here?" He's like, "I'm not sure yet," because he just started a week ago. But his students at Stanford are doing interesting things with training neural networks for speech and vision. I said, "Oh, that's cool. We should train really big neural networks." And that was the origin because we actually had quite a lot of computers in those days. So we built a software system that enabled us to express neural network connectivity and do both model and data parallel training across many computers. That enabled us to train neural networks that were 50 to 100 times larger than the largest ones we could find anyone talking about. We had a way of doing this where we had many asynchronous replicas of the model, each getting updates of the current parameters, churning away on their local batch of examples, computing a gradient, and sending it to a distributed set of parameter servers to update the parameters. This is asynchronous, definitely not the right mathematical thing because in the meantime the model parameters have moved from the other replicas giving updates. But it seemed to work, so we were happy despite it being theoretically wrong.

我们训练的第一个大型模型是在 1000 万个随机 YouTube 帧上进行完全无监督学习。有趣的是，我们使用了一个局部连接的神经网络进行计算机视觉，采用重构损失。我们试图从模型最高层的神经元表示重构输入像素。由于瓶颈层的存在，模型被迫学习比原始像素更高层次的抽象。我们发现，在顶层，一些神经元学会了检测图像中是否有猫脸，一些学会了检测人脸，还有一些学会了检测人的轮廓——所有这些都没有被告知图像中包含这些物体。看到能从纯无监督目标中学习到这些高层抽象，真是很酷。然后我们可以用这个无监督训练的初始化来进行有监督的图像分类。我们使用了 22000 类别的 ImageNet 版本，比常见的 1000 类别更稀疏。通过用无监督模型初始化再进行有监督学习，我们在 ImageNet 22k 上取得了 70%的相对改进。所以我们很高兴，并看到规模确实重要。

One of the first things we trained was a very large model to do completely unsupervised learning on 10 million random YouTube frames. Interestingly, we had a locally connected neural network for computer vision that used a reconstruction loss. We tried to take a high-level representation of the neurons at the highest level of the model and then reconstruct the pixels given as input. Because that bottleneck layer forced the model to learn higher-level abstractions than raw pixels. We found that at the top level, some neurons learned to be sensitive to whether there was a cat face in the image, some to whether there was a face, some to the outline of a person—all without being told any of these images contained those objects. It was neat to see that you could learn these high-level abstractions from a purely unsupervised objective. We could then use the initialization from this unsupervised training to train with a supervised method for image classification. We used the 22,000-category version of ImageNet, which is more thinly traded than the 1000-category one. We got a 70% relative improvement in ImageNet 22k state-of-the-art by initializing with our unsupervised model and then doing supervised learning. So we were happy and saw that scale really mattered.

我们还开始用神经网络处理语言任务。我们发现，即使相对简单的模型——比如每个词用 500 维向量表示，以及浅层模型——也能工作。例如，我们只用嵌入向量来预测附近的词。如果应用到大量文本上，高维空间中相近的词往往是相关的：猫、美洲狮、老虎距离很近。而且方向是有意义的：沿着同一方向改变词的性别，无论起始词是什么，比如国王到王后，男人到女人。一两年后，我的同事伊利亚·苏茨克维、奥里奥尔·维尼亚尔斯和曲磊使用带有 LSTM 的神经编码器，学习一个序列的编码器和另一个序列的解码器。通过用编码器初始化 LSTM 状态，它捕获输入序列，使模型能够预测目标序列。典型的例子是语言翻译：给定英法句子对，编码英语句子，解码法语句子。训练时，每个词都有一个监督目标。翻译时，只需编码新句子，然后解码生成翻译后的法语句子。

We also started doing work using neural networks for language tasks. One thing we found was that even relatively simple models with vector-based representations of words or phrases—like a 500-dimensional representation of each word—and shallow models could work. For example, we had just the embedding vector and used it to predict nearby words. If you apply this to lots of text, nearby words in the high-dimensional space tend to be related: cat, puma, tiger are close. Also, directions are meaningful: going in the same direction changes the gender of a word regardless of the starting word, like king to queen and man to woman. A year or two later, my colleagues Ilya Sutskever, Oriol Vinyals, and Quoc Le used a neural encoder with an LSTM to learn an encoder for one sequence and a decoder for another. By initializing the LSTM state with the encoder, it captures the input sequence and enables the model to predict the target sequence. The canonical example is language translation: given English-French sentence pairs, you encode the English sentence and decode the French sentence. During training, you get a supervised objective for every token. For translation, you just encode the new sentence and decode to produce the translated French sentence.

随着神经网络取得更多成功，尤其是在计算机视觉、语音识别和语言领域扩大规模，我们既兴奋又担心如何部署这些更好的模型。这就是我们张量处理单元芯片项目的起源。我做了个粗略计算：我们有一个高质量的语音识别模型，错误率大幅降低——相当于六个月内完成了 20 年的语音研究——通过训练一个非常大且特别复杂的模型架构，比如一个八层全连接神经网络，在大量数据上训练。计算表明，如果我们要部署到 10 亿用户每天使用 3 分钟的场景，我们需要将谷歌的计算机数量翻倍。这对于推出更好的语音识别系统来说似乎不可行。

As we saw more success of neural networks, especially scaling them up for computer vision, speech recognition, and language, we got excited but also worried about how to deploy these much better models. This is the origin of our tensor processing unit chip program. I did a back-of-the-envelope calculation: we had a high-quality speech recognition model with a huge improvement in error rate—equivalent to 20 years of speech research in six months—by training a very large, particularly sophisticated model architecture, like an eight-layer fully connected neural net trained on a lot of data. The calculation said if we wanted to deploy this to a scenario with a billion users talking to the model 3 minutes a day, we would need to double the number of computers Google had. That didn't seem tenable for rolling out a better speech recognition system.

这让我们确信，探索更专门化的计算设备用于神经网络推理以及后来的训练，是一条富有成果的道路。因此，在这种自举过程中，我们坚定地推进，并组建了一个芯片设计团队来制造这些芯片。两年后，第一颗芯片问世，硬件项目通常需要一些时间。我们最终生产出了 TPUv1 芯片，它利用了神经网络的两个非常好的特性。首先，降低精度是可以接受的，尤其是在推理中。所以，大约七八位的精度就完全足够了，而不是大多数人考虑的 16、32 或 64 位浮点运算。其次，我们探索的几乎所有模型都由少数几种特定线性代数运算的不同组合构成。矩阵乘法是核心，还有某些点积、归一化和缩放。但如果你想想当今大多数机器学习算法，这些运算的不同排列意味着你可以构建一个加速器，能够执行高性能的降精度线性代数，并且适用于相当广泛的神经网络。而且它效率更高，因为它只需要做这些，不需要做其他任何事情，不需要运行 Microsoft Word 或 Chrome 等。因此，TPUv1 比同时期的 CPU 和 GPU 快 15 到 30 倍，能效高 30 到 80 倍。它现在也是 ISCA 50 年历史上被引用最多的论文。

So that really convinced us that exploring much more specialized computational devices for doing neural network inference and later neural network training was going to be a fruitful path. And so in this sort of bootstrap, we got to really approach that with conviction and started up a chip design team in order to build these chips. And then the first chip emerged two years later, as hardware projects tend to do. They take a little while, but we eventually produced the TPUv1 chip, which took advantage of two really nice properties of neural networks. The first is that reduced precision is fine, especially for inference. So you're perfectly happy with about seven or eight bits of precision, not 16 or 32 or 64 as most people are thinking about in terms of floating-point computation. And also that nearly all of the models we were exploring were made up of different combinations of a handful of specific kinds of linear algebra operations. Things like matrix multiplies are at the heart of them, and then certain kinds of dot products and normalization and scaling. But if you think about most of the machine learning algorithms of today, different rearrangements of those kinds of operations mean that you can build an accelerator that is capable of very high performance reduced precision linear algebra and that can be applicable to a pretty wide range of neural networks. And it's much more efficient because that's all it needs to do. It doesn't need to do anything else. It doesn't need to run Microsoft Word or Chrome or anything else. And so TPUv1 turned out to be 15 to 30x faster than contemporary CPUs and GPUs and 30 to 80 times more energy efficient than contemporary CPUs and GPUs. And it's also now the most cited paper in ISCA's 50-year history.

然后我们将芯片设计团队的注意力转向了训练问题，因为我们觉得能够比在 CPU 和 GPU 上更大规模地扩展训练非常重要。这就是我们长期以来的关注点。我们现在已经生产了许多代 TPU，它们本质上是机器学习超级计算机。与 TPUv1 不同，它不仅仅是一个芯片，而是由大量芯片通过定制的高速互连连接而成，使我们能够将模型训练分布在这些芯片上，并通过该定制互连通信激活值、参数更新和梯度更新。第四代引入了一个奇特的功能：我们为这些加速器的机架使用了光学互连。因此，一个机架有 64 个加速器，我们可以让这个 64 加速器机架看起来就像紧挨着另一个 64 加速器机架，尽管它们在数据中心相隔 100 米。如果那个机架出现故障，我们可以换入另一个机架，并在网络拓扑上让这个机架看起来就像紧挨着那个机架。这一点很重要。随着规模越来越大，故障不可避免。因此，你希望能够像这样重新配置。我们看到了持续的硬件性能扩展。如果你看左边的 TPUv4，每芯片峰值算力约为 275 万亿次浮点运算。最大 pod 大小为 4096 个芯片，现在我们的 pod 大小大约翻了一番，HBM 内存、内存带宽和每芯片算力都大幅增加。如果你比较每个 pod 的峰值性能（这是我们定制互连的最大网络配置），TPUv2 pod 为 1 倍，而最新的 v6（可惜我们不再编号了，更难追踪了），但 Ironwood 是第六代 TPU，每个 pod 的性能大约是 TPUv2 的 3600 倍，而且能效也高得多。每瓦算力提高了约 30 倍，这是你最终关心的能源成本。

We then turned the chip design team's attention to the training problem because we felt like being able to scale up training more than we were able to on CPUs and GPUs was going to be important. And so this is where the focus has been for quite a while. And we've now produced many generations of TPUs that are essentially machine learning supercomputers. It's more than just a single chip unlike TPUv1; instead it's lots and lots of chips connected with a custom high-speed interconnect that enables us to distribute model training over those chips and communicate both activations and parameter updates and gradient updates across that custom interconnect. Version four introduced this funky thing where we have an optical interconnect for racks of these accelerators. So there's a rack of 64 accelerators and we can make it seem as though that rack of 64 accelerators is right next to another rack of 64 accelerators even though they're 100 meters across the data center floor. And if that rack fails, we can swap in another rack and make it seem as though this rack is now right next to that one in terms of the network topology. Which is important. As you get more and more scale, you end up with failures. And so you want to be able to reconfigure things like that. And so we've seen continual hardware performance scaling. So if you look at TPUv4 here on the left column, the peak flops per chip is about 275 teraflops. The max pod size was 4,096 chips and now we're sort of at roughly double that pod size and much more HBM memory and much more memory bandwidth and many more flops per chip. If you compare the performance peak performance per pod which is sort of the largest network configuration we have of the custom interconnect, the TPUv2 pod is 1x and the most recent sort of v6 but now we've stopped numbering them sadly it's harder to keep track of but Ironwood is the sixth generation of TPUs is about 3600 times the performance per pod of TPUv2 and it's also much more energy efficient. So about 30x better flops per watt which is what you ultimately care about in terms of energy cost.

过去 15 年发生的另一件事是，开源工具真正让整个社区能够使用相同的抽象和软件库进行构建，社区也可以共同改进这些工具。因此，我们基于 2011 到 2014 年期间使用内部软件库并行化多台计算机上的计算、训练大型模型和执行推理的经验，开发了名为 TensorFlow 的软件。最近，Meta 和其他地方的人推出了 PyTorch，然后谷歌的另一群人推出了一种更函数式的计算范式 Jax。这些都是很棒的系统，我们一直在使用 Jax 进行 Gemini 训练，效果很好。然后在 2017 年，我的一些同事致力于提出一种比当时最先进的循环 LSTM 更好的架构。他们的观察是，在 LSTM 中，你有一个状态向量来表示当前状态，然后每前进一个词，你就更新这个状态，有一个复杂的机制来更新状态，然后你继续下一个词，再次更新状态，所以当你处理一个句子的一百个词时，你更新状态一百次。这有两个问题。一是这是一个顺序过程。每个词你必须等待状态更新，并且状态准备好后才能继续更新下一个词的状态，因此它不太可并行化，这在现代硬件上并不好。另一个问题是，你试图将所有经历过的信息强制压缩到一个向量中，而不是保存所有经过的向量，然后在语言建模过程中根据需要查看它们。因此，Transformer 模型的关键观察是，让我们保存所有这些向量或它们的某种表示，然后通过一个可学习的注意力机制来关注它们。他们能够证明，与红色的一层、两层或四层 LSTM 相比，使用 Transformer 的测试损失（y 轴）显著降低，无论模型规模如何。

The other thing that has happened in the last 15 years is open source tools have really enabled the whole community to build using the same pieces of abstraction and software libraries and the community can work on improving those tools as well. So we developed a software called TensorFlow based on our initial lessons from the 2011 to 2014 era with an internal software library that we were using to parallelize computations across many computers and train large models and do inference. And then more recently various people at Meta and elsewhere introduced PyTorch and then another group of people at Google have introduced a more functional style paradigm computation called Jax. All of these are great systems and we have been using Jax for our Gemini training and it's been great. And then in 2017 a number of my colleagues worked on coming up with a better architecture than the recurrent LSTMs that were kind of the state-of-the-art models for language at the time. And the observation they had was really in an LSTM you have a vector of state of the current representation of the state and then when you advance one word at a time you update that state and there's a sophisticated thing that updates the state and then you go on to the next word and you update the state again and so as you progress through say a hundred words of a sentence you update the state a hundred times. And there's two problems with that. One is you have this sequential process. Every word you have to wait for the update of the state and there's a sequential dependency on that state being ready before you can go on to update the state for the next word. So it's not very parallelizable, which is not good on modern hardware. And the other thing is you're trying to force all the things you went through into a single vector that you have all the things you might want to remember from the things you went through in a single vector rather than just saving all the vectors you went through and then being able to look at them as you wish later in the language modeling process. So the key observation in the transformer model is let's save all those vectors or some representation of those vectors and then pay attention to them in a way where we have a learnable attention mechanism. And what they were able to show was that compared to LSTMs of one, two or four layers in red there, the test loss on the y-axis went down considerably using a transformer, and no matter what model scale you looked at.

所以，使用 Transformer，大约 10 到 100 倍更少的算力和 10 倍更小的模型，就能达到与更大、更耗算力的 LSTM 大致相同的质量。2018 年，我认为人们开始真正认真对待这个想法——不过，在 word2vec 的工作中，我们已经在某种程度上这么做了。但在 2018 年，我们真正关注的是如何利用自监督数据大规模进行语言建模，以及观察到世界上有大量的文本。因此，对这些文本进行自监督学习可以为你提供几乎无限的训练数据，而且你知道正确答案，对吧？比如，你有一段文本，然后你隐藏下一个词，并尝试猜测这个词。如果你猜对了，你会得到一个正反馈；如果猜错了，你会得到一个清晰的错误信号，你可以将其传播到模型中以改进它，这样下次就能猜得更好。这确实是这些聊天和语言模型变得如此出色的主要原因之一。

So sort of a 10 to 100x less compute and 10 times smaller model gave you, with a transformer, would give you roughly the same quality as you would get with an LSTM that's much larger and much more compute-intensive. In 2018, I think people really started to take seriously the idea—well, I guess in the word2vec work, we were already sort of doing that. But in 2018, really looking at how could you do language modeling at scale with self-supervised data and just the observation there's a lot of text in the world. And so self-supervised learning on this text can give you almost unlimited amounts of training data where you know the right answer, right? Like you have text and then you hide the next word and you try to guess the next word. If you guess it correctly, that gives you a sort of a one, and if you guess it wrong, that gives you a clear error signal that you can propagate into improving the model so that in such settings you will guess better next time. And so this is really one of the major reasons that these chat and language models have gotten so good.

人们通常使用两种训练目标。一种是自回归，即你查看前缀——左侧的所有内容——并尝试预测下一个词。比如'Princeton blank'，这个词有点难预测。可能是'Princeton New Jersey'，也可能是'Princeton University'。'Princeton University blank'——这个词也很难预测。'is cold in the blank'——这个词就容易一些。这就是自回归。另一种是填空，即训练目标是你隐藏一些词——你可以查看两个方向——然后你隐藏一些词，并强制模型猜测你隐藏的缺失词。同样，你会得到一个很好的、复杂的、非常清晰的错误信号，可以用来训练模型。你可以用这两种目标中的任何一种在大量文本上进行训练。填空目标通常用于构建表示，这些表示能给你一段文本的良好表示，但你不能真正将其用于聊天机器人，因为你没有右侧的内容。所以，你最终主要使用自回归模型进行对话应用，但填空也有一些很好的用途。

And so there are two kinds of training objectives that people typically use. One is called autoregressive, where you get to look at the prefix—everything to the left—and try to predict the next word. So 'Princeton blank', you know, that word is a little bit hard to predict. Could be 'Princeton New Jersey', could be 'Princeton University'. 'Princeton University blank'—that one's hard to predict as well. 'is cold in the blank'—that one is a little easier to predict. So that's autoregressive. And then fill-in-the-blank is where you get to have a training objective where you just hide—you get to look in both directions—and you hide some of the words and you try to force the model to guess the missing words that you've hidden. And again, you get this nice, sophisticated, very crisp error signal that you can use to train the model. And you can just train on lots and lots of text with either of the objectives. The fill-in-the-blank one tends to be used to build representations that give you a good representation of a bunch of text, but you can't really use it for a chatbot because you don't have the things to the right. So you have to sort of end up using mostly autoregressive models for conversational applications, but fill-in-the-blank also has some good uses.

2021 年，我的一些同事将 Transformer 应用于计算机视觉。当时，最先进的方法是使用各种卷积模型，这些模型具有局部感受野，然后在图像的不同位置构建并共享参数。而你可以直接将基于 Transformer 的模型应用于同样的问题。他们发现，如果你看达到一定准确率所需的 TPU 核心天数，与当时的先进模型 ResNet-152 相比，TPU 核心天数低得多，准确率也更高。另一个好处是，你可以看到注意力机制在关注什么。所以它给你一些模型可解释性，比如为了做出分类预测——中间那个是飞机——模型在关注什么。

In 2021, a number of my colleagues applied transformers to computer vision. At that time, the state-of-the-art was to use various kinds of convolutional models which have these kind of local receptive fields and then you build up and share the parameters among different positions in the image. And instead, you can just apply a transformer-based model to the same problem. And what they found was again you get a much better—if you look at the TPU core days to reach certain levels of accuracy—the TPU core days are much lower and the accuracy is higher compared with say ResNet-152, which was kind of a good state-of-the-art model at that time. And the other nice thing is you can kind of see what the attention mechanism is paying attention to. So it gives you a little bit of model interpretation of like what is it paying attention to in order to make a classification prediction—that the middle one is an airplane.

我们做了大量工作的另一个领域，包括我自己，是稀疏模型。在密集模型中，你有一堆参数，每次推理或每个 token 你都会激活整个模型来对该示例进行预测。这似乎——这不是我们大脑的工作方式，对吧？我们大脑的不同部分对不同的事情有用。我们激活正确的部分。所以当我担心垃圾车倒车撞到我的车时，我大脑的某些部分活跃，而思考莎士比亚诗歌的部分不活跃。因此，对于稀疏模型，我们的想法是让模型的不同部分擅长不同的事情。然后能够自动学习不仅这些专家，还学习路由机制，以了解对于这些不同的示例实际激活模型的哪些部分。这样模型容量更大，但推理成本仍然相对较低，因为你只激活了模型中的少量参数。重要的是，在相同准确率下，训练算力减少了 8 倍。这是选项 A。或者你可以选择在训练模型上花费相同的算力，得到更好的模型，对吧？相同的算力。那是选项 B。通常这些事情你会选择一点 A 和一点 B，最终处于两者之间。但随着时间的推移，稀疏性成为混合的一部分，这是让我们的模型越来越好的一个非常重要的工具。我和我的同事在这个领域做了大量工作，研究了如何进行路由、激活多少个专家等各个方面。

Another area that we have done a bunch of work in, including myself, is in sparse models. In a dense model, you have a bunch of parameters in the model and every inference or every token you activate the entire model in order to make the prediction for that example. And that seems kind of—that's not how our brains work, right? We have very different pieces of our brains that are useful for different kinds of things. And we activate the right part. So while I'm worried about the garbage truck backing up at my car, some parts of my brain are active and the part that thinks about Shakespearean poetry is not active. So with sparse models, the idea is we want to have different pieces of the model that are good at different kinds of things. And then be able to automatically learn not just those experts but also learn the routing mechanism to learn which pieces of the model to actually activate for those different examples. And so it gives the model much larger capacity but still keeps the inference cost relatively low because you're only activating a small number of the parameters in the model. And importantly, it gives you an 8x reduction in training compute for the same accuracy. That's choice A there. Or you could choose to spend the same amount on training the model and get a much better model, right? Same amount of compute. That would be choice B there. And often these kinds of things you choose a bit of A and B and end up kind of somewhere there. But this has been a really important tool for making our model better and better over time, having sparsity be part of the mix. There's a whole litany of work that both I and my colleagues have done in this general area, looking at various aspects of how to do the routing, how many experts do you activate, and so on.

我们一直在做的另一件事是为分布式机器学习计算构建抽象。因此，Pathways 是我们很久以前开始开发的一个系统，它真正想要抽象掉你计算底下的这一堆加速器，给研究人员一种错觉，即你只有一台巨大的计算机，上面连接着数万个芯片，你基本上可以专注于你想要表达的机器学习计算。在底层，Pathways 做了大量工作来管理如何在所有这些芯片之间通信你所表达的机器学习计算。例如，在一个 pod 内部，它会使用自定义的 TPU 互连。当你在同一数据中心建筑内跨越 pod 边界时，它会使用数据中心网络。当你在同一数据中心园区内跨越建筑时，它会使用紫色网络链路。当你跨越广泛分布的大都市区域时，它会使用广域网链路。然后，当你真正在 Pathways 之上使用 JAX 时，这意味着我们可以从一台主机上的单个 Python 进程驱动整个训练过程。所以那个 Python 进程说：'太好了，我可以访问这数千个 TPU 芯片，我可以运行这个计算'，而 Pathways 处理当系统不同部分发生故障时会发生什么，并在底层替换新的硬件。

Another thing we've been working on is building up abstractions for distributed ML computations. So, Pathways is a system we started working on quite a while ago that really wants to abstract away this pile of accelerators underneath your computation and give the researcher the illusion that you just have a giant computer with tens of thousands of chips attached to it and you can mostly just focus on the machine learning computations you're trying to express. And underneath the covers, Pathways does a bunch of work to manage how to communicate the machine learning computation you've expressed amongst all these chips. And so it will, for example, use within one of these pods, it will use the custom TPU interconnect. When you go across the pod boundary in the same data center building, it will use the data center network. When you go across buildings in the same data center campus, it will use the purple network link. And when you go across perhaps widely distributed metro regions, it will use the wide area network links. And then when you actually put JAX on top of Pathways, what that means is that we can drive the entire training process from a single Python process on one host. So that Python process says, 'Oh great, I have access to these many thousands of TPU chips and I can run this computation,' and Pathways kind of deals with what happens when different pieces of the system fail and swapping in new parts under the covers.

当运行规模越来越大时，另一个常见问题是并非所有芯片、机器、网卡或线缆都能按设计工作。这是令人遗憾但真实的情况。有些故障并非立即可检测并更换，而是会非确定性地静默产生错误结果，有时与机架温度等动态条件有关。这在独立计算时已经很有挑战，但在所有芯片同步执行机器学习训练计算时更糟，因为错误结果会迅速传播。例如，如果某处翻转了梯度指数域的一个比特，你会得到 10 的 20 次方作为梯度传播，而不是 0，这非常糟糕。因此，我们在训练过程中监控模型健康的各种指标。例如，可以查看每一层梯度的范数，观察是否有尖峰。这是一个由静默数据损坏导致的尖峰示例。并非所有尖峰都源于机器问题；另一个尖峰没有 SDC，只是某批数据导致了大梯度。区分它们很困难。我们的一种方法是自动触发确定性重放。如果看到梯度尖峰，我们会重放该步骤或最近几步，看是否得到相同答案。如果相同，则可能是数据问题而非硬件错误。如果硬件错误导致问题，重放时希望得到不同答案。也有静默数据损坏但不引起梯度尖峰的情况，这对模型无害，即使未检测到。各种情况都可能发生。Pathway 系统是我们透明处理这些问题的一部分。例如，假设这些蓝色 Pod 是同步训练任务的一部分。我们有七个蓝色 Pod，一个灰色热备，另一个热备，并在空闲芯片上持续运行 SDC 检查器，以在使用前剔除坏硬件。有缺陷的机器导致静默数据损坏；SDC 检查器通过重放检测到不匹配，定位到具体 Pod 和芯片，将其驱逐，用热备透明替换，继续训练，并将坏硬件送修。

Another thing that tends to happen when you run at increasingly large scale is that not all chips, machines, network cards, or cables work as designed. That's a sad but true state of the world. Some of them don't just fail in ways you can detect immediately and replace; they sometimes nondeterministically produce incorrect results silently, sometimes related to dynamic conditions like the temperature of the rack at that moment. This is really challenging when running independent compute, but worse when trying to have all chips perform as a single orchestrated machine learning training computation where everything is synchronous, as bad results can quickly spread. For example, if one of these flips a bit in the exponent field of your gradient, you get 10^20 propagating as a gradient instead of 0, which makes you very sad. So we monitor various metrics about model health during training. For instance, you can look at the norm of the gradient at every layer and see if there are spikes. Here's an example of a spike due to silent data corruption. Not all spikes are due to machine issues; here's another spike with no SDC, just a batch of examples causing a large gradient. It's hard to discern these. One way we handle this is deterministic replay triggered automatically. If we see a gradient spike, we replay that step or the last few steps to see if we get the same answer. If we do, it's probably due to data, not hardware. If hardware errors cause the issue, we hope to get a different answer on replay. There are also cases where silent data corruption happens without a gradient spike, which is benign even if undetected. All kinds of things can happen. The Pathway system is part of how we transparently handle these. For example, let's say these blue pods are part of synchronous training jobs. We have seven blue things, a hot spare in gray, another hot spare, and we run an SDC checker continuously on idle chips to weed out bad hardware before use. A defective machine causes silent data corruption; the SDC checker identifies it by replaying and detecting a mismatch. We pinpoint which pod and chips, evict it, use the hot spare to transparently replace it in training, and send the defective hardware to the repair team.

2022 年，推理时更长的思考变得更加突出。观察发现，如果你提示模型在示例问题上展示更多工作步骤，然后让它解决类似问题，结果会更好。例如，如果只要求答案，模型可能对示例给出 9，对实际问题给出 50，这是错误的。但如果给出一个逐步解答的示例，模型会倾向于类似地逐步推理，更可能给出正确答案。这就像四年级数学老师教你的，很高兴看到机器学习模型也有类似行为。一种理解是，你在推理时给了模型更多算力，因为每个生成的词元都需要一次模型前向传播。答案的长度与用于产生答案的算力成正比。当模型达到一定规模时，允许思维链提示会大幅提高 GSM8K 问题（八年级数学）的解决率。例如，“John 有五只兔子，他又得到两只……”也许这不是八年级水平，但这些都是简单问题。2022 年，我们对这些问题的准确率大幅提升感到兴奋。

In 2022, thinking longer at inference time became more prominent. The observation was that if you prompt the model to show more of its work on an example problem and then ask it to solve a similar problem, you get better results. For instance, if you just ask for the answer, the model might give the answer as 9 for one example, and then for the real problem it says 50, which is wrong. But if you give a worked-out example, the model tends to work through the steps similarly and is more likely to give the correct answer. This is what your fourth-grade math teacher told you, and it's nice to see ML models behave the same way. One way to view this is that you're giving the model more compute to think at inference time, because every token it generates requires a pass through the model. The length of the answer is proportional to the compute devoted to producing it. As the model reaches a certain scale, allowing chain-of-thought prompting leads to large increases in the solving rate of GSM8K problems (eighth-grade math). For example, 'John has five rabbits. He gets two more...' maybe that's not eighth grade, but these are simple problems. We were excited in 2022 about high accuracy improvements on those problems.

我们研究的另一项工作是蒸馏，即利用强大的教师模型，从更大模型的更复杂预测中生成更小、更便宜的学生模型。例如，在语言建模中，“演奏协奏曲为____”的真实答案是“小提琴”，监督学习会用这个作为信号。但好的教师模型可以给出下一个词的概率分布：小提琴、钢琴、小号，几乎不可能是飞机。这为模型顶层提供了丰富的梯度信号，比仅仅知道是否猜对“小提琴”提供了更多信息。这使得你能将大型模型的大部分能力转移到更小的模型中，可能使用更少的数据。在论文的初步实验中，我们研究了一个语音识别问题。基线使用 100%训练集，训练帧准确率 63.4%，测试帧准确率 58.9%。

Another thing we worked on was distillation, where you use a powerful teacher model to make a smaller, cheaper student model from the more sophisticated predictions of the larger model. For example, in language modeling, the true answer for 'perform the concerto for blank' is 'violin', and that's what supervised learning would use as the signal. But a good teacher model can give a probability distribution over likely next words: violin, piano, trumpet, almost certainly not airplane. This provides a rich gradient signal at the top of the model, giving much more information than just whether you got 'violin' or not. This enables you to get much of the capability of a very large model into a much smaller model, possibly with less data. In initial experiments in the paper, we looked at a speech recognition problem. We had a baseline using 100% of the training set, achieving 63.4% training frame accuracy and 58.9% test frame accuracy.

但如果将基线降至仅使用 3%的训练数据，就会过拟合。训练帧准确率是 67%，但测试帧准确率骤降至 44.5%。然而，如果使用蒸馏中的软目标，仅用 3%的训练数据就能获得相当不错的训练帧准确率，实际上甚至优于使用 100%数据的基线，而测试帧准确率也几乎与使用全部数据训练的完整模型相当。因此，这是一项非常关键的技术，能有效让你将任意形状的模型变形为另一种形状的模型，通常是将大模型压缩成小模型。例如，我们在 Gemini 中就用它从 Pro 规模模型得到 Flash 规模模型，其性能几乎与 Pro 规模模型相当。在过去的四五年里，我认为强化学习在激发模型所需特性方面变得愈发重要。对于一个预训练模型，它只是接触了大量文本训练数据，具备了许多能力，但我们往往希望进一步引导这些能力。具体来说，你可能希望鼓励模型以特定方式回应，比如某种风格、特定场景下的回答长度，或者加入安全属性，避免涉及某些话题。同时，你也想通过展示如何解决更复杂的问题来增强能力。使用强化学习时，一个关键问题是奖励信号从何而来？但它在许多领域确实显著提升了模型能力。奖励信号可以来自多种来源。一是人类反馈：比如针对某个问题，让人类评估回答 A 或 B 哪个更好，并给出奖励；或者直接问这个回答好不好，人类回答是或否。这些方法即使只用少量样本，也能有效改变模型行为。你也可以使用来自机器反馈的强化学习，即用另一个模型提供反馈，通常称为奖励模型。你可以提示奖励模型判断对于问题 Q，回答 A 和 B 哪个更优，然后从基础模型生成两个回答，让奖励模型选择，从而获得奖励信号。此外，你还可以在可验证领域（如数学或编码）使用强化学习。例如，生成证明或解决方案，用定理证明器检查，当模型给出正确证明时给予正奖励。在编码中，可以检查代码是否编译通过、是否通过单元测试，从而获得奖励。这些方法在数学和编码领域非常有效，也适用于更复杂的推理和规划等相邻领域。我认为有一个很好的开放研究问题：如何提高强化学习在不可验证领域的有效性，以及从何处获得有趣的奖励信号？这些问题我认为相当开放，有很多值得探索的方法。

But if you drop the baseline to use just 3% of the training data, then you overfit on that. So you have 67% training frame accuracy but your test frame accuracy plummets to 44.5%. But if you use the soft targets with distillation, you get a quite nice training frame accuracy, actually even better than the baseline on 100% of the training data, with just 3% of the training data, and your test frame accuracy is nearly as good as the full model trained on 100% of the data. So this is a really key technique for effectively enabling you to morph any shaped model into another shape model, often a larger model into a smaller model. This is, for example, what we use in Gemini to go from the Pro-scale model into a Flash-scale model that is almost as good as the Pro-scale model. And then in the last four or five years, I would say reinforcement learning has become much more important in eliciting the desired properties that these models have. So given a pre-trained model where you've just exposed it to a whole bunch of training data of text, that model has a bunch of capabilities in it, but they often you want to help steer the capabilities of these models a bit. So in particular, you want to maybe encourage behaviors that you want in the way in which the model responds. You might like a certain style of response. You might like to prefer certain length responses for certain kinds of things. You might want safety properties to be put into the model, so don't engage in these kinds of topics or say something like this in these set scenarios. And you also want to enhance the capabilities by showing the model how to tackle more complex problems. And so one of the key questions when you use reinforcement learning is where does the reward signal come from? But these do dramatically improve the model capability in a lot of domains. And the reward signals turn out can come from lots of different places. So one is you can use human feedback: you know, do you like A or B as an answer for this kind of question and have humans evaluate that and give a reward, and then have the model, or is this a good answer for this question? They can say yes or no. Those kinds of things can be very effective at making the model's behavior change pretty dramatically with even relatively few examples. You can also use reinforcement learning from machine feedback where you have machine feedback from a different model. So often called a reward model, maybe you can prompt the reward model saying please say whether you prefer response A or B for question Q, and then you generate two responses from your base model and then you ask the reward model which one do you like better, and that gives you a reward signal. You can also do RL in verifiable domains like math or coding, and you can for example generate proofs or solutions to be checked with a theorem prover that give a positive reward when the model produces a correct proof for the problem you're trying to solve. You can generate code for coding things where you now have a reward for does the code even compile? Does the code compile and pass the unit test? You get more reward. These kinds of things. So these really help a lot in both math and coding, but also in kind of adjacent areas like more sophisticated reasoning and planning in adjacent domains. I think there's a nice open research question of how do you improve the effectiveness of RL in non-verifiable domains and where do you get interesting reward signals from? Those are things that I think are pretty open and ripe for lots of cool approaches.

2023 年，我的一些同事提出了一个非常好的方法，用于加速推理。自回归解码的一个问题是，注意力机制在预填充时需要状态。如果你输入一千个 token，可以完全并行处理。但当你开始生成回答时，先生成第一个 token，然后才能根据第一个 token 的注意力状态生成下一个 token，这是一个非常顺序化的过程，有点像 LSTM 的顺序解码阶段。推测解码是一种很好的方法，它只改变解码算法，无需重新训练模型，没有架构变化，且保证输出分布完全相同。观察发现，从非常大的 Transformer 模型解码是内存受限的。硬件能执行的浮点运算远多于将模型权重传输到乘法单元以解码单个 token 所需。第二个观察是，有些 token 比其他 token 更容易预测。比如问“7 的平方根是多少？”最后一个 token 很难预测，但很多其他 token 很容易。例如，问号之后，模型可能预测四五个 token，其中一些是正确的，尤其是前缀部分。因此，思路是使用一个快速的草稿模型，它不需要大模型那么复杂，可以快速生成接下来的八个 token，然后让目标大模型并行检查它们。这实际上将批大小从 1 变成了 8。如果平均接受八个中的四个半，那么计算效率会大大提高。这里有一个动画演示。如果只用顶部的大模型逐 token 解码，速度很慢。而如果底部有一个小模型每次草拟四个 token，然后大模型根据自身概率分布接受正确的前缀，这能显著提升推理效率。

In 2023, a few of my colleagues came up with this really nice method for making inference faster. One of the problems with autoregressive decoding is that in order to have the attention mechanism have the state it needs when you're prefilling. So if you have an input of a thousand tokens, you can do that completely in parallel. But now if you're going to try to start generating a response, you generate the first token, and then when you finish generating the first token, you can now look at that token and be able to now generate the next token because you need the attention state for that previous token in order to actually generate the next token. So it's a very sequential process, a little bit like the problem we had with LSTMs where you have this sequential decoding phase. Speculative decoding is a nice approach, and you change only the decoding algorithm. You don't have to retrain your model. You have no architecture changes, no retraining, and you're guaranteed to have the identical output distribution. The observation is decoding from very large transformer models is memory bound. So the hardware can do way more floating-point operations than it can do transfer all the weights of your model into the multiplier unit in order to do the single token of decoding you want to be able to do. Observation two is that some tokens are a lot easier to predict than others. So can you tell me what the square root of seven is? Sure, the square root of seven is blank. Right? That last token is really really hard for the model to predict. But a lot of the other ones are very easy for the model to predict. Like after the question mark, it can probably make a prediction of four or five tokens and get some of them correct. In particular, some prefix of them correct. And so the idea is you have a fast drafter model where you don't need the full sophistication of a much larger model, and you can quickly generate the next say eight tokens, and then have the target large model check them in parallel. This effectively gives you a batch size now of instead of one, a batch size of eight. And if you accept on average four and a half of those eight tokens, then you've made your compute efficiency much much better. So here's an animation of how this works. If you just try decoding with a large slow model at the top, you're decoding one token at a time. And instead if you have a small model at the bottom that's drafting sequences of four tokens at a time, and then the larger model accepts the prefix of them that is correct according to what it wanted to generate based on its own probability distribution. So this actually helps quite a lot in improving the efficiency of inference in these models.

因此，许多因素共同作用，才使得这些现代模型能够良好运行。包括推理算法、训练算法、模型架构、软件抽象以及硬件方面的众多创新。

So a whole bunch of things have come together in order to actually make these modern models work well. Lots of innovations in the inference algorithms, in the training algorithms, model architecture, software abstractions, and in the hardware.

现在我们来说 Gemini。Gemini 的起源是，Google 有多个独立团队在研究语言模型和各类多模态模型。我当时说，'这太蠢了，我们为什么不一起合作？' 与其分散算力和想法，不如集中资源，打造一个统一的多模态模型。这就是 Gemini 项目的起源。我们于 2023 年 2 月启动，合作方来自 Google DeepMind、Google Research 以及 Google 其他部门。我们的目标是训练出世界上最好的多模态模型，并尽可能广泛地应用。我们已发布了多个迭代版本：Gemini 1、1.5、2、2.5，以及最近的 3。

So now we get to Gemini. The origin of Gemini was that we had multiple separate groups at Google looking at language models and creating multimodal models of various kinds. I said, 'This is stupid, why don't we all work together?' Instead of fragmenting our compute and ideas, let's build a single multimodal model with a lot of compute investment behind a more singular effort. That's the origin of the Gemini project. We started in February 2023 with collaborators from Google DeepMind, Google Research, and the rest of Google. Our goal was to train the world's best multimodal models and use them in all the places we can. We've produced several iterations: Gemini 1, 1.5, 2, 2.5, and most recently 3.

我们追求的一个目标是从一开始就让模型具备多模态能力。我们认为，处理语言、音频、图像和视频，以及其他非人类模态（如激光雷达数据或机器人控制数据）至关重要。在较新的 Gemini 版本中，我们加入了少量此类数据，让模型了解这些数据的存在。最初我们只有图像解码器和文本解码器，后来增加了音频和视频解码器。这样我们就有了一个统一的模型，能够处理任意混合模态的输入序列。你可以给它一张图片加一些文字，或者视频加音频，它也能以交错的方式输出任意模态的内容。

One of the things we wanted was for the model to be multimodal from the start. We think it's really important that it deals with language, audio, images, and video, as well as other non-human modalities. In more recent Gemini versions, we've been including small amounts of LIDAR data or robotic control data to expose the model to the idea that these kinds of data exist. Initially, we had an image decoder and a text decoder; more recently, we've added audio and video decoders. So we have a single unified model that can reason over input sequences that are mixes of any of these modalities. You can give it an image and some text, or video and audio, and it can produce any of those as output in an interleaved way.

我们基于众多技术进步进行构建。我们使用 TPU、模型并行和数据并行训练，以及跨数据中心训练。我们有快速自动检测 STC 错误的方法。我们使用 Pathways、JAX，以及类似 Word2Vec 或序列到序列工作的分布式词表示。我们使用 Transformer。我们的模型通常是稀疏的。我们从大模型蒸馏到小模型。我们使用思维链解码及其更复杂的版本。推测解码非常重要。我们使用 SFT 和各种强化学习，以及其他许多技术。

We're building on lots of different advances. We use TPUs, model parallel and data parallel training, and cross-data-center training. We have fast and automated detection of STC errors. We use Pathways, JAX, distributed representation of words similar to Word2Vec or sequence-to-sequence work. We use Transformers. Our models are typically sparse. We distill from larger models into smaller models. We use chain-of-thought decoding and more sophisticated versions. Speculative decoding is really important. We use SFT and various kinds of RL, and a bunch of other things.

我们一直在努力扩大上下文长度，因为上下文窗口中的信息非常有用。与训练数据中混杂模糊的 token 不同，上下文窗口中的信息非常清晰，没有与其他内容混合。就像'我看到你给我的这 900 页文本，我可以查看不同部分，进行总结、合并，并在解码时提取小片段'。将上下文窗口扩展到非常大的长度一直很重要。

One of the things we've been working on for a while is making our context length large, because the information in the context window is really useful. Unlike tokens in the training data, which are muddled and fuzzy, the information in the context window is very crisp. You haven't mixed it with anything else. It's like, 'I see these 900 pages of text you gave me, and I can look at different parts, summarize, combine, and pull out little pieces when decoding.' Extending the context window to very large lengths has been important.

我们有不同规模的模型。Pro 规模模型质量最高，Flash 规模模型则尽可能接近 Pro 模型。我们还希望 Flash 模型能超越上一代的 Pro 模型。我们已经连续保持了三到四代。这意味着，六个月后，原本较慢且昂贵的高端模型会变得更实惠，因为其能力已下放到 Flash 模型中。

We have different scales of models. We have a Pro scale model, which is the highest quality. Then we have a Flash scale model that is ideally as close to that Pro model as possible. We also want that Flash model to be better than the previous generation's Pro model. We've been able to keep that up for three or four generations now. That means the highest-end thing, which is slower and more expensive, becomes more affordable six months later because that capability is now in the Flash model.

我们一直在研究数学问题求解。去年 7 月，我们参加了国际数学奥林匹克竞赛（IMO），六道题做对了五道，获得了金牌成绩，我们非常高兴。前一年，我们使用了一个更复杂、临时拼凑的数学专用模型，结合了 Lean 和定理证明器。今年，我们直接使用 Gemini Pro 模型，并给予较高的推理思考时间预算。我们很高兴，一个通用模型——与提供给用户的模型相同——就能在 IMO 中获得金牌。这显示了我们在数学上取得的巨大进步。还记得'Fred 有五只兔子，然后又得到两只'那种题目吗？现在我们解决的是 IMO 问题。输出展示了复杂的数学写作和证明格式。我们建立了上界，有另一个引理，构造了下界，并证明了对于所有 bonso 函数，f(n) ≤ 4n。这已经远超 GSM8K 了。虽然还不是研究级别的数学，但两年内的进步幅度非常大。

We've been working on mathematical problem solving. In July last year, we competed in the International Mathematical Olympiad (IMO) and got five out of six problems correct, a gold medal score. We were quite happy. The previous year, we used a more sophisticated, jury-rigged mathematically specialized model with Lean and theorem provers. This year, we used an out-of-the-box Gemini Pro model with a high inference thinking time budget. We were happy that a single general-purpose model, the same one offered to users, was able to get a gold medal at the IMO. This shows how far we've come in mathematics. Remember back to 'Fred had five rabbits and then he got two more'? Now we're solving IMO problems. The output shows sophisticated mathematical writing and proof formatting. We established an upper bound, had another lemma, constructed a lower bound, and proved that f(n) ≤ 4n for all bonso functions. This is quite beyond GSM8K. It's not research-level mathematics yet, but the slope is quite large in two years.

我们的 Gemini 3 模型于今年 11 月发布。我们对许多基准测试结果感到非常满意。

Our Gemini 3 model release happened in November of this year. We were pretty happy with a lot of the benchmark results.

嗯，不过我们也对最终结果感到满意。在现实环境中评估这些模型的一种方法是使用一个叫做 Chatbot Arena 的设置，人们会看到两个匿名模型的回答，然后判断哪个更好。通过大量针对特定提示的成对比较，人们可以问比如“你能帮我解这道数学题吗？”或者“你能帮我做一份法式洋葱汤的食谱吗？”然后人们给 A 或 B 打分，看哪个更好。通过大量这样的评估，你可以了解不同模型的相对实力，从而得到一个 ELO 分数，有点像国际象棋的排名。

Um, but also we have some, you know, we were happy with where it landed. One of the ways that you evaluate these models in the wild is you can use this setup called Chatbot Arena, where people see responses from two anonymized models and then they say which one is better. From lots of pairwise comparisons for their particular prompt, people can go say, 'Can you please help me solve this mathematical problem?' or 'Can you help me produce a recipe for my French onion soup?' And then people grade A or B which is better. Through a lot of those evaluations, you can get a sense of the relative strength of different models, and that's where you get an ELO score, a little bit like a chess ranking.

所以我们一直在做的一件事就是搜索和 AI 模式。我们实际上希望这些模型能够生成用户看到的用户界面。呃，我不确定这能不能行。我连上 Wi-Fi 了吗？可能没连上。这要让我伤心了。好吧，我开个热点。这应该比登录普林斯顿的访客网络容易，对吧？稍等一下。抱歉。好了，现在我们要等它连上。快点。哦，好了。太好了。太棒了。现在我们来试试。点这里。点这里。耶。谢谢 T-Mobile。好了。

So one of the things we've been working on is search and AI mode. We actually want these models to be able to produce the UI that the user will see. Uh, I'm not sure this is going to work. Am I on the Wi-Fi? I might not be on the Wi-Fi. This is going to make me sad. All right, let's enable the hotspot. That's probably easier than logging into the Princeton guest network, isn't it? One moment. Sorry. Okay, now we're gonna wait for that. Come on. Oh, there we go. Okay, great. Awesome. Now, we're going to try it. Go there. Go there. Yay. Thank you, T-Mobile. Okay. All right.

所以这基本上是他们输入了一个类似研究性教科书的内容，关于某个生物学的东西，然后他们说“给我展示一下这是怎么工作的”，结果它创建了一个交互式可视化。你可以用滑块控制可视化，从而了解你正在学习的主题，这非常棒。你可以看到这对教育来说会非常有用。

So this is basically they put in a sort of researchy textbooky thing about some biological thing and now they said 'show me how this works' and what it's done is created an interactive visualization of the material there. You can use sliders to control the visualization and get a sense of the topics you're trying to learn about, which is pretty nice. You can see how this would be really useful for educational things.

这是另一个例子。这对夫妇有一堆手写的英文和韩文食谱。哦，实际上，因为我们在放字幕，我会把速度调到 0.25。所以他们基本上是在做这样的事情。他们让 Gemini 把这些食谱全部翻译并转录成英文和韩文。然后任何说韩语的人都可以验证英文翻译。好了，现在他们要说，“请用这些食谱创建一个双语网站。”好了，就这样。然后我们添加了一些自动生成的图片，这就是由此创建的网站。

Here's another example. This couple has a bunch of recipes in English and Korean in handwritten pieces. Oh, I will actually, since we have closed captions, I will speed this up to 0.25. And so they basically are taking things like that. They're asking Gemini to take all those recipes and say translate and transcribe them in English and Korean. And then it has any Korean speakers can maybe validate the English translation. Okay, so now they're going to say, 'Please create me a bilingual website using these recipes.' Okay, so there we go. And then we added some automatically generated images and this is a website that was created from that.

我认为这其中一点是，这种复杂的模型将使很多可能没有编程技能的人能够以软件形式获得他们想要的东西。所以我认为世界上的软件会比现在多得多。这只是一个有趣的例子，但我认为关注这些模型的能力以及它们能完成的编码和问题解决的复杂程度是很好的。

I think one of the things about this is that this kind of sophisticated model will enable a lot of people who maybe don't have the skills to write software to get things that they want in software form. So I think we're going to have a lot more software in the world than we have now. Just kind of a cute example, but I think it's good to pay attention to what these models can do and the sophistication level of the coding and problem solving they can do.

我跳过那个。图像和视频的生成模型也在快速进步，我得说。所以我们最初发布了一个叫 Veo 的东西，然后我们做了一个计算成本更高但质量更好的版本叫 Veo Pro。举个例子，你可以从一张蓝图开始，然后说“请根据这张蓝图生成一张逼真的 3D 图像”。你可以看到这对可视化不同的建筑结构非常有用。你可以说“请用不同部分发生的重要事情来注释原始的 Transformer 架构图”。如果你在读一篇研究论文，得到这样的可视化会非常有用。

I'll skip that. Generative models for images and for video have also been improving quite rapidly, I would say. So we had an initial release of something called Veo, and then we made a much more computationally expensive but higher quality version called Veo Pro. As examples of things, you can start with that blueprint and say 'please make a realistic 3D image from the blueprint.' You can see how this would be quite useful for visualizing different architectural things. You can say 'please annotate the original Transformer architectural diagram with the important things that happen in different parts of the diagram.' You can see if you're reading a research paper, getting a visualization of that could be pretty useful.

你实际上可以在像素空间中观察思考过程。所以这里的输入图像是一个球和斜坡的问题，“告诉我球会掉进哪个桶，用图像一步步解决。”对吧，所以你现在可以看到模型会生成一系列推理图像。它从那里开始。第一步，球滚到那里。第二步，球滚到三。然后第三步，球滚到五。然后第四步，它最终掉进桶 B。所以你可以看到你自己的问题解决过程可能也会产生这些关于将要发生什么的心理图像。而模型能够做到这一点，并且实际渲染出来。

You can actually view thinking in pixel space. So here the input image is this sort of ball and ramp problem and 'tell me which bucket the ball lands in, use images to solve it step by step.' Right, so you can now actually see the model will generate a series of images of its reasoning. So it goes from there. Step one, ball rolls there. Step two, the ball rolls onto three. And then step three, the ball rolls onto five. And then step four, it ends up in bucket B. And so you can sort of see how your own problem solving probably makes these mental images of what's going to happen. And the model is able to do that and actually render that.

这是一个例子。你可以用不同的数据注释奥维尔·莱特的飞机。我猜是 605 磅。这挺有意思的。没想到它这么重。我把这个发推作为它能做什么的例子。当我们最初发布 Veo Pro 时，很多人很伤心。所以，“没有冥王星，冥王星在哪？”所以我说，“好吧，好吧。我也是九大行星时代的孩子。把这张图片改成 219，加上冥王星，再加一句幽默的评论。”于是它决定加上引文“前行星被降级为矮行星，对此仍然耿耿于怀。”然后他说“完美，我们回来了。”

This is like an example. You can annotate Orville Wright's airplane with different stats. I guess 605 pounds. That's interesting. Didn't realize it was quite so heavy. I tweeted this out as an example of what it can do. When we initially launched Veo Pro, a bunch of people got very sad. So, 'no Pluto, where Pluto?' So I said, 'Okay, okay. I too am a child of the nine planet era. Make this image 219 to add Pluto and add a humorous comment.' So it decided to add the quote 'former planet got demoted to dwarf planet status, still grumpy about it.' And he said 'perfect, we are so back.'

我想稍微谈一下我们是如何组织像 Gemini 这样的大型项目的。特别是，这是一个庞大的项目。我现在是一篇有超过一千位作者的论文的合著者。我们的报告很长。光是列出作者就要占很多页。我们总是试图在作者名字的前几个词里拼出一些有趣的东西。所以当你有那么多人一起合作时，你需要一些结构。所以我们有总体负责人，有项目管理和产品管理，帮助我们保持组织性，同时也帮助我们从一个产品的角度思考，我们希望下一代模型做什么？然后我们有一系列不同的子领域，这些领域对让 Gemini 运行良好至关重要。

I want to touch a little bit on how we organize a big effort like Gemini. In particular, this is a large effort. I'm now a co-author on a paper with more than a thousand authors. Our reports are long. Even just listing the authors takes quite a lot of pages. We always try to spell something funny in the first few words of the author names. So when you have that many people trying to work together on something, you need a little bit of structure. So we have overall leads, we have program management and product management that help us be organized, but also help us figure out from a product standpoint, what do we want our next generation models to do? And then we have a bunch of different sub areas that are really critical to making Gemini work well.

一个是模型开发。包括预训练、后训练和强化学习等。我们有一个在 Pixel 手机上本地运行的 Gemini 版本。所以它有稍微不同的特性，以及不同的训练数据和能力需求。在能力方面，我们有很多领域，比如安全、视觉、音频、代码、智能体、国际化，这些我们都很关心。然后是核心领域，比如我们应该用什么数据训练？我们如何在训练下一个模型的 token 预算内获得最高质量的数据？我们如何知道我们在各种不同能力上做得怎么样？基础设施，比如如何让训练过程顺利运行？如何让服务基础设施顺利运行？代码库，以及更长期的研究，旨在为 Gemini N+2 或 Gemini N+3 提供好想法，并在合适的时间交汇。我们的人遍布世界各地。

One is model development. So in terms of both pre-training and post-training and RL, things like that. We have a version of Gemini that runs on device on our Pixel phones. So that has somewhat different characteristics and somewhat different aspects of training data and capabilities needed. In terms of capabilities, we have a bunch of areas like safety and vision, audio, code, agents, internationalization that we care a lot about. And then core areas like what data should we train on? How can we get the highest quality data we can for the token budget we have for training our next model? How do we know if we're doing a good job across lots of different capabilities? Infrastructure, which is like how do we make the training process work well? How do we make the serving infrastructure work well? The codebase and then longer term research that is meant to feed in good ideas for Gemini N+2 or Gemini N+3 that will intersect at the right time. And we have people spread out all over the world.

我们大约三分之一的人在湾区，三分之一在伦敦，另外三分之一在其他许多地方，比如纽约、巴黎、波士顿、苏黎世、班加罗尔。时区问题很烦人，但也没什么办法。不过加州西海岸和伦敦欧洲之间每天大约有三个小时的黄金时段，对双方都不算太糟糕，这还不错。但这意味着你必须采用异步工作方式，而且要运作良好。所以我们有很多大大小小的讨论组、聊天室、Google Chat 空间，我本人就参与了 200 个。我们还有一种内部文档或技术报告式的征求意见稿，这是一种半正式的反馈方式，用来了解别人在做什么，或者记录自己的想法。这些文档的成熟度各不相同，从‘我有个初步想法想听听意见’，到‘我已经详尽探索了这个想法的十种变体，该把哪个放进下一版 Gemini’，应有尽有。它们按顺序编号，我刚看了一下，现在已经有超过 5000 份了。所以内部有大量的征求意见稿，复杂程度从一页纸到一篇技术报告或论文不等。此外，排行榜和公共基线让我们能够基于数据做出如何改进模型的决策。我们进行大量实验，先在小规模上做很多实验，只有那些看起来有希望的才会推进到中大规模。每隔一段时间，我们会把在大规模上验证成功的实验整合成一个新的候选基线，然后重复这个过程。这套方法效果不错。

So we have about a third of the people in the Bay Area, a third in London, and a third in many other places. So New York City, Paris, Boston, Zurich, Bangalore. Time zones are super annoying. Not much we can do about it. But the golden hours between California, West Coast, and London, Europe are about three hours a day that are not too terrible for anyone. So that's kind of nice. But that does mean you have to have asynchronous ways of working that work well. So there are lots of large and small discussion groups, chat rooms, Google Chat spaces. So I'm in 200 of these. We have a request for comment kind of internal document or tech report-like thing, which is a semiformal way of getting feedback. Knowing what other people are working on, jotting down an idea. They range in sophistication from early-stage idea I'm thinking about I want comments on, to I've done an exhaustive exploration of 10 different variants of this idea, which one should we put in the next version of Gemini. They're numbered sequentially and I just looked, we now have more than 5,000 of these. So there's quite a lot of internal requests for comments that are ranging in complexity from a single page thing to something that would be a tech report or a paper. And then leaderboards and common baselines enable us to make data-driven decisions about how do we improve the model. So we have lots of rounds of experimentation. You do many more experiments at small scale and only progress things that seem promising to medium and large scale. Every so often we incorporate successful experiments demonstrated at larger scale into a new candidate baseline and then we repeat. That seems to have worked pretty well.

再谈一些前瞻性的想法。我认为一个非常重要的领域是人类和 AI 智能体如何协作完成任务。目前，这些模型的主要使用方式是一个人坐在交互式聊天界面前做事情。但未来，更多的工作场景可能是人类协调十几个或上百个 AI 智能体代表他们做事，而人类可能只是每周指定一下自己想要什么。我们如何提供合适的人机交互范式来管理 50 个这样的虚拟助手团队？智能体之间如何协作完成任务？这又会带来哪些可能性？我认为这些都是非常有趣的方向。一百万个 token 的上下文窗口已经很有用了，相当于一千页文本或几小时的视频。但能够处理一万亿个 token 的内容显然更有价值。所以我设想一种混合系统：用学习到的检索算法从大型语料库中检索，再用轻量级模型评估检索到的 3 万份文档与当前任务的相关性，然后把真正重要的几百个内容放入上下文窗口，同时从计算机系统角度进行预计算，让这一切高效快速。我认为这会有很多应用场景。个性化的 Gemini：我希望能够访问我所有的邮件状态、照片等——当然是在我许可的情况下。还有网络搜索、多模态搜索和检索，比如搜索所有 YouTube 视频。代码智能体：Google 的每个代码智能体，如果它能访问整个 Google 代码库，对每个 Google 开发者来说都会非常有用。展望未来，我认为推理效率将变得至关重要。这些模型的训练成本很高，需要大量芯片。但未来，推理会越来越多，智能体会以更复杂的方式使用这些模型，并与其他也在进行推理的智能体交互。因此，专门为推理设计的硬件将变得重要。同样，模型和算法层面的改进以实现高效推理也很重要。低延迟是推理场景的巨大优势，用 100 毫秒响应的东西比 5 秒响应的愉快得多。我相信未来 AI 应用于芯片设计自动化将是获得更高效推理硬件的重要途径之一。我对此有一个完整的演讲，但核心就是尽可能多地使用学习和强化学习。

A few forward-looking thoughts. I think one area that's going to be really important is how humans and AI agents collaborate to get stuff done. Currently, a lot of the uses of these models are a single person sitting down with an interactive chatbot interface doing stuff. But in the future, it seems like more work is going to happen where the human is coordinating the activities of a dozen or a hundred AI agents doing stuff on their behalf, where the human has probably weekly specified what it is they want. How can we give the right HCI paradigm for managing a team of 50 of these virtual assistants? How can the agents themselves cooperate to accomplish things? What will this enable? I think these are all pretty interesting directions. I think one million tokens of context is quite useful, giving you a thousand pages of text or hours of video. But it seems like it would be even more useful to be able to attend to a trillion tokens of stuff rather than a million. So I think hybrid systems with learned retrieval algorithms over large corpora and then maybe lightweight models that can be used to assess how relevant are these 30,000 documents I retrieved to the thing I'm trying to do, and then put the hundred things you actually think are really important into the context window, and maybe precomputation from a computer systems perspective to make this all efficient and fast. I think there would be a lot of uses for this. Personalized Gemini: I'd love it if I could attend to all of my email state and photos and so on with my permission of course. Web search, multimodal search and retrieval like searching all the YouTube videos. Coding agents: every coding agent at Google, it'd be nice if it attended to the whole Google codebase for every Google developer. And forward-looking, I think inference efficiency is going to be really critical. The training cost of these models is very large and you need a lot of chips. But going forward, more and more inference is going to happen and more and more agents using these things in sophisticated ways and interacting with other agents that themselves are doing inference. So specialized hardware design purely for inference is going to be important. Again, I think model and algorithmic improvements for efficient inference are going to be important. Low latency for these inference cases is a huge plus. It's just much more enjoyable to use something that responds in 100 milliseconds than 5 seconds. And I believe in the future, AI applied to automating chip design is going to be one of the ways in which we get much more efficient inference hardware. I have a whole talk on this, but essentially it's use learning as much as possible and RL.

过去几年我还做了一件事：我们召集了一组合著者，思考 AI 对世界各个领域的影响，而不仅仅是计算机科学。我们想看看这些影响可能是什么，既有正面的也有负面的。我们确定了七个我们认为会受到 AI 巨大影响的领域：就业、教育、医疗、虚假信息、媒体与娱乐、治理与国家安全，以及 AI 促进科学。然后我们采访了这些领域的一批专家，特别寻找那些能对其中一个或多个领域提供有趣视角的人。接着我们整理了与专家交流的收获以及我们自己的思考，写成了一篇很长的存档论文，不过读的人不多。所以我们又写了一个简版，发表在 Kacum 上，合著者之一 Dave Patterson 还在《经济学人》上写了一篇精彩的一页评论。所有这些内容都可以在 shapingai.com 上找到。我认为这是一篇非常有趣的论文，它提供了关于 AI 影响——超越就业和技术转型——的一些深刻见解。举个例子，这是我唯一一篇拥有自己网站的论文。

One of the other things that I did in the last couple years is we got together a group of co-authors thinking about what are the impacts of AI in many different domains of the world, not just computer science. And we wanted to look at what these impacts might be, both positive and negative. So we identified seven domains we thought would be pretty dramatically impacted by AI: employment, education, healthcare, misinformation, media and entertainment, governance and national security, and AI for science. And then we went out and interviewed a bunch of domain experts in these areas. In particular, we tried to find people we thought would have interesting perspectives on one or more of these domain areas. And then we wrote up what we learned from chatting with the domain experts as well as our own thoughts. So we produced a long archive paper which not that many people have read. So we have a shorter version in Kacum and Dave Patterson, one of the co-authors, wrote a nice one-page editorial in the Economist. And this is all available on shapingai.com. I think this is a pretty interesting paper. I think it has some good insights into what will the impact of AI beyond employment and technology transitions. As one example, it's the only paper I have with its own website.

总之，我认为像 Gemini 这样的 AI 模型以及基于它们构建的产品，正在成为我们日常工作中非常重要且强大的工具。进一步的研究和创新将继续推动这一趋势。它将对我在 shaping AI 论文中提到的许多领域产生巨大影响：医疗、教育、科学等等。它有可能将深厚的专业知识——这些模型在某些领域确实非常出色——提供给全世界许多人，这是一种非常强大的能力。如果做得好，我相信我们由 AI 辅助的未来将非常光明。好的，谢谢。

In conclusion, I think AI models like Gemini and products built with them are becoming really important and powerful tools for the things we try to do every day. Further research and innovation is going to continue this trend. It's going to have a dramatic impact in many of the areas that I just mentioned in the shaping AI paper: healthcare, education, science, all kinds of things. It potentially makes really deep expertise—these models are really good in some areas—and makes that available to many people all over the world, which is a pretty powerful capability. And done well, I think our AI-assisted future is going to be really bright. All right. Thank you.

你对人工智能最大的担忧是什么？

What's your biggest fear about AI?

我对人工智能最大的担忧？我认为关于 AI 系统安全性的观点有很多。我倾向于认为这些担忧有些夸大其词。通过精心设计我们允许 AI 系统做什么，即使它们能力很强，也能实现安全部署。在更近期的未来，我担心的是 misinformation（虚假信息），因为我们现在可以制作极其逼真的视频和音频，即使是经验丰富的人花大量时间也难以分辨。另外，我们如何管理 AI 能够做以前做不到的事情所带来的影响，并确保人们为这些转变做好准备，学会使用这些新工具，从而提高生产力，而不是突然因为工作被自动化而无法胜任。

My biggest fear of AI? I think there's a whole range of views on safety of AI systems. I'm of the view that those are a little overblown. Careful engineering of what we allow AI systems to do will enable safe deployment even though they are quite capable. In the nearer term, misinformation is one I'm worried about because we can now create incredibly realistic video and audio that are hard to distinguish even for sophisticated people with a lot of time to look at them. Also, how do we manage the impact of AI being able to do things it previously couldn't, and ensure people are prepared for those transitions and can learn to use these new tools to be more productive, rather than suddenly being unable to do things because they've been automated.

我是 Joey，计算机科学专业大四学生。我有一个关于幻觉的问题，特别是随着工作流程和上下文变长时的行为。我们知道，即使过去的模型声称有一百万的上下文窗口，在极限情况下，保留率和响应质量会下降。除了简单地在语言模型中添加搜索，Gemini 还优先采用了哪些其他方法来减少幻觉，尤其是在关键应用中？

I'm Joey, a senior in computer science. I had a question about hallucinations, specifically long behavior as we get more workflows and longer contexts. We know that even though past models claim to have a one million context window, at the limits retention and quality of responses degrade. Beyond simply adding search into the language model, what other ways has Gemini prioritized decreasing hallucinations, especially for critical applications?

一个关键技术是，不是只生成一个答案，而是生成多个候选答案，然后让模型自己审视这些候选答案，评估哪个最可能是正确的。这本质上使用了更多的推理时算力或思维链式思考。这些方法确实可以降低幻觉率。我们在一些 IMO 工作中就采用了这种方法，因为那里绝对不能出现幻觉，你需要正确证明问题。这是一个很好的工具，但也带来了推理效率的问题，因为你希望每个答案都生成 16 个答案，然后审视它们，决定保留哪些。

One key technique is instead of generating a single answer, having multiple rollouts of potential answers and then having the model itself look at those rollouts and assess which one it thinks is most likely correct. That essentially uses much more inference time compute or chain of thought style thinking. These can definitely decrease the rate of hallucinations. That's the kind of approach we use in some of the IMO work where you really can't afford to hallucinate because you need to prove a problem correct. It's a good tool, but it brings back the question of inference efficiency because you'd like for every answer to produce 16 answers, look at them, and decide which ones to keep.

我很好奇，你对语言模型与研究中正在出现的世界模型有什么看法？

I'm curious, what's your point of view on language models versus the emerging world models that are being worked on from a research perspective?

世界模型实际上是关于理解多种模态以及世界从物理学角度如何运作。我认为我们的很多 Gemini 模型在构建世界模型方面做得相当不错。看看 Genie 3 的发布，它们建立在 Gemini 模型之上，能够通过文本提示生成相当复杂的想象世界，并控制其中的交互。你可以向左、向右、跳跃、前进、后退，它保持与所创建世界的一致性。我们正在与 Waymo 的同事合作，因为这使他们能够在模拟中创建长尾训练案例。例如，你可以说‘创建一个场景，其中一头大象出现在路中间’，它就会生成几十个在现实世界中很难发生的测试案例，你可以观察汽车的行为，甚至合成激光雷达数据，从而获得更好的安全特性。

World models are really about understanding multiple modalities and how the world works from a physics perspective. I think a lot of our Gemini models are actually pretty good at doing world models. If you look at the Genie 3 releases, those are built on top of Gemini models and enable you to generate fairly sophisticated imaginary worlds with text prompts and then control interaction in that world. You can move left, right, jump, forward, back, and it maintains consistency with the world it created. We're collaborating with our Waymo colleagues because that enables them to create long-tail training cases in simulation. For example, you can say 'make a scenario where an elephant appears in the middle of the road,' and it creates dozens of test cases that are hard to occur in the real world, and you can see how the car behaves, even synthesizing the LIDAR data, enabling better safety properties.

也许再问一个问题。我很好奇，你认为我们应该用大约一千个 TPU 做什么类型的研究？我们讨论的几个方向都相当数据密集。你认为我们可以做些什么来推动 AI 发展？

Maybe one more question. I'm kind of curious what type of research you think we should work on with about a thousand TPUs? A few of the directions we talked about are pretty data-intensive. What do you think we can do to help AI forward?

从研究角度来看，我们的很多工作都是从极小规模的探索开始的。你可以非常有效地尝试许多不同的事情，也许不是在数千个加速器上训练，而是在小规模上证明这个想法有潜力。在持续学习或不同模型架构等问题上，在小规模下，我们做的一件事是运行极小的模型并观察趋势。趋势往往比相对于当前最先进基线的绝对位置更能指示重要性。如果斜率低于基线但看起来不错，那是一个很有趣的想法。如果在最小规模上高于基线，但即使在较小规模上也迅速下降到基线以下，那就没那么有趣了。没有大规模算力的人可以通过专注于非常不同且有趣的问题解决方案来做出出色贡献，而不必追求最先进的结果。作为社区，我们应该鼓励这样的工作，而不是那些微调一个小东西就获得增量改进的工作。

A lot of the work we do from a research perspective starts as investigations at very small scale. Where you can be quite effective is trying lots of different things that you maybe aren't training on a thousand accelerators, but you're demonstrating at small scale that this idea has legs. There are all kinds of problems in continual learning or different model architectures where at small scale, one of the things we do is run extremely small models and look at the trend. The trend is often more indicative of importance than exactly where it falls relative to the current baseline state-of-the-art. If the slope is below the baseline but looks good, that's a really interesting idea. If it's above the baseline at the absolute smallest scale but rapidly plummeting below even at the small scale, that's less interesting. People who don't have access to large-scale compute can do a great job by focusing on problem solutions that are quite different and interesting, not stressing about getting state-of-the-art results. As a community, we should celebrate such work rather than incremental improvements that tweak one little thing.