谷歌首席科学家 Jeff Dean 与英伟达首席科学家 Bill Dally 探讨 AI 进展与低延迟推理

GTC 的爱好者们，欢迎谷歌首席科学家 Jeff Dean 和英伟达首席科学家 Bill Dally 上台。Jeff，机器学习领域过去一年精彩纷呈。你看到的最令人兴奋的发展是什么？你对明年又有什么期待？

GTC enthusiasts, please welcome to the stage Google Chief Scientist Jeff Dean and NVIDIA Chief Scientist Bill Dally. So Jeff, it's been an exciting year for machine learning. What's the most exciting development you've seen in the past year in machine learning? What do you expect to see next year also?

是的，我认为这个领域的每个人都看到了模型能力、应用方式等方面的快速进步，这非常有趣和令人兴奋。回顾过去一年左右，我想强调几点。一是模型在可验证奖励的问题上变得更好，比如数学和编程。三四年前，当我们的模型能解决八年级数学题，比如‘弗雷德有四只兔子，又得到两只兔子’，并且正确率在 40%到 50%时，我们就很兴奋了。但在过去几年，尤其是去年，我们大大加速了模型在数学上的有效性。我们的 Gemini 模型参加了国际数学奥林匹克竞赛并获得了金牌。在 ICPC 编程竞赛中，我们也获得了金牌。所以这些领域的能力确实令人印象深刻。更近期，我们开始看到基于智能体的工作流在长时间运行的任务中表现得非常好。以前，你让模型做某事，它几分钟就完成，你需要密切监督。现在你可以给模型需要几小时甚至几天的任务，它们会独立地做事、自我纠正并继续。这是一个令人兴奋的转变，因为它让模型能够更长时间地自主运行，这与以前大不相同。

Yeah, I think everyone in this field has seen really rapid progress in what these models can do, their capabilities, and how people are applying them. That's been really interesting and exciting. Reflecting on the last year or so, I'd highlight a couple of things. One is that models have gotten way better at problems with verifiable rewards, like math and coding. Three or four years ago, we were excited when our models could solve eighth-grade math problems like 'Fred has four rabbits and gets two more rabbits' 40 or 50% of the time. But in the last few years, especially the last year, we've really accelerated how effectively these models can do mathematics. Our Gemini model entered the IMO contest and got a gold medal. In the ICPC coding contest, we also got a gold medal. So the capabilities in those domains are really impressive. More recently, we've started to see agent-based workflows work extremely well for longer-running tasks. Previously, you'd ask a model to do something, and it would finish in a few minutes, and you'd have to supervise it closely. Now you can give models tasks that take hours or even days, and they will independently do things, correct themselves, and continue. That's an exciting transition because it allows models to operate fairly autonomously for longer periods, which is very different from before.

说到这个，似乎我们会有更多这样的智能体在后台运行。一个重要的问题是如何实现超低延迟推理，让这些系统能更快地自主工作，因为这是驱动它们高效解决问题的关键。你认为英伟达正在研发的下一代架构将如何大幅降低延迟？我们如何从每秒几百个 token 提升到几千甚至几万个？

Speaking of this, it seems we're going to have a lot more of these agents operating in the background. One important thing is how to get ultra-low latency inference so these systems can do their work autonomously faster, because that's a rapid driver of how effectively they can solve problems. How do you see the next architectures you at NVIDIA are working on delivering major latency reductions? How do we think about getting from hundreds of tokens per second to thousands or tens of thousands?

是的，我今天上午 10 点的演讲中谈到了这一点，如果你错过了，我想应该有录像。这个问题有很多答案。当你观察推理的性能曲线时，这是一个权衡。在一个极端，如果你愿意牺牲延迟，你可以获得非常高的吞吐量，即每美元或每瓦特的每秒 token 数。然后你可以沿着曲线向下移动，通过运行更小的批次大小来优化交互性和单用户每秒 token 数，但牺牲吞吐量。当你到达曲线的右侧，真正优化延迟时，大部分延迟来自通信延迟。所以我们正在开发架构，将延迟降低到英伟达常说的光速。在典型的 LLM 中，每层有前馈阶段和注意力阶段，重复 50 到几百层。每完成一个阶段，通常需要片内通信进入下一阶段。完成一层后，通常需要片外通信进入下一层。有时甚至层内阶段之间也需要，取决于分区方式。在片内，我们有静态调度的试验设计，没有路由开销、排队或仲裁。我们可以实现大约每纳秒 2 毫米，基本上是导线上的飞行时间，从芯片一角到另一角只需 30 纳秒，将延迟从今天的几百纳秒降下来。在片外，很多延迟来自我们的 PHY，因为多年来我们为了最大带宽优化，没有太多考虑延迟。为了推向极端，我们使用强大的数字信号处理从导线上提取超噪声信号，并运行复杂的数学运算来确定比特。即使这样，比特还是有噪声，所以我们使用强大的前向纠错。事实证明，如果你稍微降低带宽，比如从每对 400 Gbps 降到 200 Gbps，所有这些都不需要了。你只需检测导线上的电压，得到比特，剩下的只是串行化延迟——片上和片外各几个时钟周期。所以我们看到了构建完整路由器的途径，类似于我 20 年前在 Cray 为 Black Widow 构建的路由器。那个路由器的引脚到引脚延迟小于 15 纳秒。我认为我们可以再次做到这一点，通过这样做，我可以看到我们以每用户每秒 10,000 到 20,000 个 token 的速度运行相对较大的模型。

Yeah, I actually talked about this in my talk at 10:00 this morning, which I think was recorded if you missed it. There are a bunch of answers to that question. When you look at the performance curve for inference, it's a trade-off. At one extreme, if you're willing to give up latency, you can get very high throughput in tokens per second per dollar or per watt. Then you can move down that curve by running smaller batch sizes to optimize more for interactivity and single-user tokens per second at the expense of throughput. As you get to the right side of that curve where you're really optimizing for latency, the bulk of the delay is communication latency. So we're working on developing architectures that drive that latency down to what NVIDIA always refers to as speed of light. In a typical LLM, you have feed-forward stages and attention stages per layer, repeated for 50 to a couple hundred layers. Every time you finish one of those stages, there's usually an on-chip communication to get to the next stage. After finishing a layer, it's often an off-chip communication to get to the next layer. Sometimes even between stages within a layer depending on partitioning. On-chip, we have trial designs with static scheduling so there's no routing overhead, no queuing, no arbitration. We can achieve about 2 mm per nanosecond, basically time of flight over wires, and get from one corner of the chip to another in 30 nanoseconds, driving latency down from several hundred nanoseconds today. Off-chip, a lot of latency is in our PHY because for years we optimized for maximum bandwidth without much consideration for latency. To push extremes, we use heroic digital signal processing to pull the super noisy signal off the wire and run fancy math to figure out the bit. Even then the bit is noisy, so we do heroic forward error correction. It turns out if you back off on bandwidth a little, say from 400 gigabits per second per pair to 200, all that goes away. You can just detect the voltage on the wire, get the bit, and it's just serialization latency—a couple clock cycles on and off chip. So we see our way to build whole routers, similar to the routers I built at Cray 20 years ago in 2006 when we did the Black Widow. That router was less than 15 nanoseconds pin to pin. I think we can do that again, and by doing that, I can see us running relatively big models at 10 to 20,000 tokens per second per user.

是的，这非常令人兴奋。我认为能够以如此低的延迟运行最大规模的模型非常重要。

Yeah, that's super exciting. I think it's really important to be able to run the largest scale models at these low latencies.

是的。你提到了这些智能体系统，你可以给它们一个任务，然后它们会运行一段时间。

Yeah. So, you talked about these agentic systems getting to the point where you could give them a task and it would run off for a while.

你认为我们距离这样一个时刻有多近：我们可以让一个智能体系统自己去完成下一个版本的自己？比如，拿当前版本的 Gemini，给它一个长期任务，可能一个月，它自己去实验，找出正确的模型、正确的数据整理策略，运行数据，决定如何获取更多数据，甚至写几份合同，然后训练出下一个版本的自己。

How close do you think we are to the point where we can take one of these agentic systems and just have it do the next version of itself? You know, take whatever the current version of Gemini is, give it a long-running task, maybe a month, and it goes off, experiments around, comes up with the right model, the right data curation strategy, runs the data through there, decides how to acquire more data, maybe writes a couple contracts to do that, and trains the next version of itself.

是的，我认为你描述的整体情况还没有完全实现，但我确实看到了一些迹象。你可以用非常高层的方式问模型：‘请在这个粗略的空间里探索改进模型性能的想法。’然后它会去运行 50 个实验，排除 40 个，聚焦在 10 个看起来有希望的，进行后续实验。我认为描述这类工作的一种方式是，元学习已经存在一段时间了。例如，2017 年，Google Brain 团队的一些人做了神经架构搜索的工作，你可以指定一个潜在架构的搜索空间，在小规模上运行大量实验，找到在特定问题上学习最好的架构。我们也用学习优化器和学习激活函数等做过类似的事情。在那些情况下，八九年前，人们是用代码来指定搜索空间的。

Yeah, I mean, I think the whole thing you described is not quite there yet, but I do think we are starting to see signs of that emerging. You can ask the model at a very high level, 'Please explore ideas in this rough space for improving the performance of the model.' And it'll go off and run, you know, 50 experiments, dismiss 40 of them, hone in on 10 that seem promising, do follow-up experiments. I think one way I've been characterizing this kind of work is, for a while there's been meta-learning. For example, in 2017, some people in the Google Brain team did work on neural architecture search where you can specify a search space of potential architectures and run lots of experiments at small scale to find architectures that learn best on a particular set of problems. And we've done that with learned optimizers and learned activation functions and so on. In those cases, eight or nine years ago now, people were specifying the search spaces as code.

嗯。

Mhm.

但我认为我们现在拥有的一个非常好的能力是用自然语言指定有趣的研究空间。对吧？就像‘请去让自己变得更好。请探索有趣的新蒸馏算法，并尝试使用我们目前没有使用的信息’，然后它就会去进行那些实验。我认为这是一个强大的……

But I think one of the really nice things that we now have is the ability to specify interesting research spaces in natural language. Right? It's like, 'Please go make yourself better. Please explore interesting new distillation algorithms and try to use information we're not currently using,' and it will go off and do those experiments. And I think that's a powerful...

语言神经架构搜索，你知道的。

Language NAS, you know.

是的。这基本上是研究人员生产力的超级倍增器，因为突然间，思考研究性的想法变得非常容易。而运行实验、获取结果并解释它们实际上更难。如果智能体能够做很多这样的工作，那将是超级能干的研究人员和超级能干的智能体之间的绝佳合作。

Yeah. It's basically a super powerful multiplier for researcher productivity because all of a sudden, it's really easy to think researchy thoughts. It's actually harder to run the experiments and get the results and interpret them. And if the agent can do a lot of that work, that's really going to be a great partnership between super capable researchers and super capable agents.

嗯。

Mhm.

构建硬件时始终存在的挑战之一是，尤其是在机器学习这样快速发展的领域，如果你今天启动一个新的硬件项目，幸运的话，它可能在两年后才会出现在数据中心。我们希望这个时间更短。但之后它必须使用若干年。所以你真的需要预测机器学习领域和 AI 领域在两到五年时间窗口内的发展方向。这始终是一件非常难以准确预测的事情。我想知道你是否想出了一些好的工具或机制来有效地进行这种水晶球预测。

One of the challenges that always exists when you're building hardware, especially in a fast-moving field like machine learning, is you know, if you start a new hardware project today, it might pop out the other end and actually be in data centers in two years if you're lucky. We'd like that to be shorter. But then it has to live for a certain number of years. And so you really have to try to predict where the machine learning field and AI field are headed in the two to five-year time window. And that's always been a super hard thing to predict well. I'm wondering if you've come up with good tools or mechanisms to help do this kind of crystal ball reading effectively.

是的，我们尽力而为。我们这样做的一部分方式是尝试开发自己的模型，包括像 NeMo Tron 这样的大语言模型，像 Cosmos 这样的世界模型，或者像 Groot 这样的机器人基础模型。但尽管我们努力尝试，我们还是会感到惊讶。有很多人在做这个工作，他们非常聪明，每天都在提出很棒的想法。所以我们不得不让我们的硬件面向未来。我们通过多种方式做到这一点。首先，有些做法对所有模型都有好处。如果我们能想出更高效地进行数学运算的方法，比如通过更好的数值表示，那会让一切变得更好。同样，如果我们能组织芯片上的通信以减少数据移动，那也会让一切变得更好。我们遇到麻烦的地方，也就是我们真希望知道未来会有什么变化的地方，是当人们改变模型的方式改变了资源配置——即数学运算量、内存带宽、内存容量和通信量之间的比例——因为即使你把这四个要素都做得非常高效，你也必须决定每个要素有多少。然后当有人提出不同的模型时，比如从分组查询注意力转向多头潜在注意力，它会显著改变其中的一些比例。突然间，你的一些硬件闲置了，而另一些则繁忙。这基本上没有办法避免，除非将来我们看到许多不同的模型需要不同的资源配置，并且产量足够大，我们可能会做不同的 SKU，以不同的方式配置，从而对冲未来的风险。

Yeah, so we try our best. Part of the way we do that is by trying to develop our own models, both LLMs like NeMo Tron and world models like Cosmos or robotic foundation models like Groot. But as hard as we try, we get surprised. There are lots of people out there working on this, they're super clever, coming up with great ideas every day. So we kind of have to future-proof our hardware. We do that in a number of ways. First, there are things you do that are just good for all models. If we can come up with a way of doing math more efficiently, perhaps due to better number representations, that makes everything better. Similarly, if we can organize our on-chip communication to just reduce data movement, that makes everything better. Where we tend to run into places where we really wish we knew what was coming down the pike are when people change the model in a way that changes the provisioning—how much math you have versus how much memory bandwidth versus how much memory capacity versus how much communication—because those are the things that even if you make each of those four ingredients really efficient, you have to decide how much of each to have. And then when somebody comes up with a different model, like going from group query attention to multi-head latent attention, it changes some of those ratios pretty significantly. And all of a sudden you're leaving some pieces of hardware idle and others busy. There's really no way around that except perhaps in the future if we see a bunch of different models needing different sorts of provisioning, the volumes are large enough we may just do different SKUs provisioned in different ways and kind of hedge our bets against the future.

是的，有道理。

Yeah, that makes sense.

我想轮到我了。还没到。好吧。那么，历史上，至少在近期历史上，我们倾向于根据 Chinchilla 缩放定律来扩展模型。如果我们有一定量的算力，我们会据此决定有多少参数，通常是参数数量的 20 倍左右的 token。但现在我们进入了一个很难获得更多 token 的领域，但我们仍然希望继续扩展模型，使用更多的算力进行训练。你认为什么能填补这个缺口？我们如何利用更多的算力来获得更准确的模型？

I guess it's my turn. Not yet. Okay. So, historically, at least in recent history, we tend to scale our models according to the Chinchilla scaling laws. And if we had a certain amount of compute, from that we would decide we would have so many parameters and usually about 20 times that number of tokens. But now we've gotten into this realm where it's really hard to get more tokens, but we'd still like to continue scaling our models, use more compute to do the training. What do you see sort of filling that gap? How can we use more compute to get more accurate models?

是的，我想首先澄清一下，Chinchilla 缩放定律对于训练是最优的。对吧？如果你开始考虑训练后模型的推理性能，那么你可能不一定想完全遵循它。我也不同意我们正在耗尽数据的说法。我觉得世界上有大量的数据还没有被用来训练这些模型。

Yeah, I mean, I think first I'll clarify Chinchilla scaling laws are optimal for training. Right? If you start to consider inference performance of the trained model, then it may be that you don't necessarily want to follow exactly that. I also kind of disagree with the assertion that we're running out of data. I feel like there's an awful lot of data in the world that is not yet being used to train these models.

你在哪里看到这些数据？

Where do you see this data?

嗯，我们训练了一些视频数据，但我认为还有更多的视频数据和相关的音频数据我们还没有训练。我认为现实世界的机器人或自动驾驶车辆的数据将会相当丰富。

Well, we train on some video data, but I think there's a lot more video data and associated audio data that we aren't necessarily training on yet. I think real-world robotics or autonomous vehicle data is going to be fairly plentiful.

合成数据是另一种资源。如何生成真正有趣的高质量数据？你只需投入更多算力，就能通过合成方式获得更多数据来训练模型。

Synthetic data is another resource. How do you generate really interesting high-quality data? You just inject more compute, you can get synthetically more data for training your models.

但过一段时间后，你不会陷入某种重复生成相同内容的情况吗？如果你在某个数据上训练，然后用它生成合成数据，那不过是同一内容的另一个版本。

But don't you run out of sort of regurgitating the same stuff after a while? If you train on some data and then use that to generate synthetic data, it's sort of another version of the same stuff.

是的，但有时它确实对模型有帮助。如果你的模型非常强大，那个生成合成数据的模型，那么它实际上似乎能带来相当大的帮助。还有很多我们尚未使用的技术，这些技术多年前在图像模型、卷积图像模型中很流行。所以，我认为像数据增强这样的技术——合成数据也算是其中一部分——以及防止过拟合的技术都很有意思。你可以使用 dropout 或蒸馏作为正则化模型的方法。所以，我认为有很多机会可以通过更多算力和更多次数据遍历来真正改进模型，而不一定会过拟合。

Yeah, but it still seems to help the model sometimes. If your model is very powerful, the one that's generating the synthetic data, then it actually does seem to help quite a bit. There are all kinds of techniques we're not yet using that were in vogue for image models, convolutional image models years ago. So, I think things like data augmentation, of which synthetic data is a bit of that. Techniques to prevent overfitting are interesting. You can use dropout or distillation as ways of regularizing the model. So, I think there's a lot of opportunity there to really make models better with more compute and more passes over the data where you won't overfit necessarily.

你是否看到过 LLM 发展到我们像训练 AlphaGo 那样训练它们，让 LLM 彼此对话，并通过这种方式提高性能？

Do you ever see the LLMs getting to the point where we train them the way we trained AlphaGo, just to have the LLMs kind of conversing with each other and somehow improving their performance by doing that?

是的，我确实认为当前的预训练机制有点像你拿一个模型，用随机数生成器初始化它，把它绑在一块板上，然后让所有互联网数据流经它，它从观察中学习，但实际上并没有在世界中采取行动。我们似乎希望将其中一些与模型在某种环境中采取行动结合起来，无论是模拟机器人环境还是尝试预测问题的答案等，然后再回到学习，并且可能在选择下一步看什么数据时更有方向性，而不是预定的数据顺序。我认为这实际上会非常有趣，可以大幅提高学习效率。对于给定数量的 token，模型能从中获得多少？在世界中采取行动对此将非常有用。我们在后训练中做了这一点，但那是一种非常有限的形式。我认为应该在预训练阶段更多地穿插这类事情。

Yeah, I do think the current pre-training regime is sort of like you take your model, throw your random number generator at it to initialize it, strap it to a board, and then stream all the internet data past it, and it learns what it can from that observation, but it's not actually taking actions in the world. It seems like we want to interleave some of that with something where the model gets to take actions in some environment, be it simulated robotics environments or trying to predict answers to questions or things like that, and then going back to learning, and maybe where it's also a bit more directed in how it chooses what data to see next as opposed to a predetermined ordering of the data. I think that would actually be quite interesting as a way to dramatically improve the learning efficiency. For a given number of tokens, how much can the model get out of it? Taking actions in the world is going to be super useful for that. We do that in post-training, but that's a very limited form. I think interleaving this kind of thing much more at the pre-training stage.

在预训练阶段穿插。

Interleaving it at the pre-training stage.

在预训练阶段。我们现在有这种人为的区分，但从长远来看，这种区分似乎不应该存在。

At pre-training stage. We sort of have this artificial distinction now, and it doesn't seem like that should exist for the long term.

它必须足够聪明才能有意义。所以当你从一个随机数生成器开始时，可能做不到。但当你投入一定数量的 token 之后。

It has to get smart enough for that to make sense. So when you start out with a random number generator, probably can't do that. But after you've thrown some number of tokens at it.

对。给它看 100 亿个 token，它可能就能做点什么了。

Right. Show it 10 billion tokens and it can probably do something.

过去几年，训练工作负载的 Scaling（规模扩张）一直是整个社区的重点。但随着这些模型在现实世界中影响力越来越大，我们开始看到推理工作负载变得重要得多。传统上，我们试图构建同时擅长这两者的芯片。Nvidia 最近收购了 Groq，后者拥有不错的低延迟推理硬件。我看到你宣布将 Groq 硬件与最新一代 GPU 结合。你如何看待训练所需硬件与推理所需硬件之间的差异？

Scaling of training workloads has been a really big focus for the whole community for the last few years. But we're starting to see, as these models have much more influence in the real world, that inference workloads are becoming much more important. Traditionally, we've tried to build chips that are good at both of these things. Nvidia recently acquired Groq, which has nice low-latency inference hardware. I saw you made some announcements combining the Groq hardware with your latest generation GPUs. How do you think about the differences between hardware needed for training and hardware needed for inference?

这是个非常好的问题。我会更进一步。我认为不仅仅是推理开始变得重要，推理现在就是主要任务。数据中心通常 90% 的电力都用于推理。好消息是两者有很多相似之处。例如，我们最初为推理开发的 NV4 等数字格式，已被证明在训练中也非常有效。但在需求方面，尤其是在内存系统方面，这仍然是一项非常不同的任务。训练时，你必须保留所有激活值用于反向传播步骤，而推理时一旦用完就丢弃。因此对内存容量的需求更大，并且导致内存容量、内存带宽、数学运算和通信之间的配置比例不同。即使在推理的不同阶段，我们也看到同样的配置需求差异。例如，预填充阶段很像训练，因为你可以并行处理所有 token。因此它往往计算量很大，如果不注意数据本地化，就会受通信能耗主导。然后进入推理的解码阶段，尤其是在延迟优化的情况下，你进行的是矩阵-向量或非常瘦的矩阵乘积，并且极度受内存带宽和延迟限制。甚至这还分为前馈和注意力阶段，它们做不同的事情。所以我认为随着时间的推移，自然会出现为这些不同阶段配置的不同组件。我怀疑你至少会看到三种类型：一种擅长训练和预填充，一种擅长解码，而擅长解码的那种甚至可能进一步分为注意力解码和前馈解码。

That's a really good question. I'll go further. I think it's not just that inference is starting to become important. Inference is the job now. It's usually 90% of the power in data centers going into inference. The good news is there's a lot of similarities between both. For example, number formats like NV4 that we developed originally for inference have proven to be remarkably effective at doing training as well. But it's still a very different task in terms of what the demands are, particularly in terms of the memory system. With training, you've got to keep all the activations around for the backprop step, and inference you just throw them away as soon as you're done with them. So there's a larger need for memory capacity, and it drives a different amount of provisioning between memory capacity, memory bandwidth, math, and communication. We see that same difference in need for provisioning even in different stages of inference. For example, the prefill stage is a lot like training in that you get all these tokens you can operate on in parallel. So it tends to be very math heavy and dominated by communication energy if you're not careful about localizing your data. Then you get to the decode stage of inference, and you are now doing, especially if you're latency optimized, matrix-vector or very skinny matrix products, and you're extremely memory bandwidth limited and latency limited. Even that is broken into the feed forward and the attention stages that do different things. So I think it's a natural division over time to build different components provisioned for these different stages. I suspect you'll see at least three flavors: one that's good at training and prefill, one that's good at decode, and that one that's good at decode may even bifurcate into the attention decode and the feed forward decode.

是的，有道理。我同意预填充看起来更像训练，而解码阶段则完全不同，因为它非常顺序化。你得到一个 token，必须让它通过所有环节。

Yeah, that makes sense. I do agree that prefill looks more like training than the decode stage, which looks quite different because it's so sequential. You got one token, you got to run it through everything.

是的，我认为有帮助的一点是推测解码。它可以给你一个稍微瘦一点的向量，而不是一个向量。

Yeah, I think one thing that helps is speculative decoding. It can give you a slightly skinny vector instead of a vector.

对。不是得到一个，而是得到八个。

Right. Instead of one you get eight.

是的。扩散 LLM 甚至可以做得更好，可能一次处理几百个块。

Yeah. And diffusion LLMs can do even better, possibly doing blocks up to a few hundred.

嗯。

Yep.

好的。新模型层出不穷，有新的注意力机制、扩散 Transformer、混合状态空间 Transformer 模型。你对模型的前景有何看法？当前模型动物园中最让你兴奋的是什么？

Okay. So new models are appearing on a regular basis with new forms of attention, diffusion transformers, hybrid state-space transformer models. What do you see on the horizon for models? What excites you most in the models zoo at present?

我认为我们越来越倾向于参数更多但更稀疏的模型。这对内存占用以及如何激活正确的部分有很大影响。通常对于 MOE，你只激活总参数的一小部分。当你有批次时，你会激活不同的部分，这有点烦人，因为它破坏了批处理。在注意力机制方面，对朴素的二次注意力有有趣的潜在改进。二次注意力算法在质量上表现极好，但一旦处理大量 token，成本就非常高。你需要某种方法来扩展到超过一百万个 token。有一些在适度规模上得到验证的可能性看起来很有前景。一种是大幅降低二次注意力的常数因子，通过关注 token 块，识别哪些块最有趣，然后只深入这些块。

I think we've been moving more towards larger parameter count sparser models. That has a huge impact on memory footprint and how you activate the right pieces. Typically with MOEs, you're activating a tiny fraction of the overall parameter count. When you have a batch, you're activating different ones, which is a bit annoying because it destroys some of that batching. In the attention space, there are interesting potential improvements to the naive quadratic attention. The quadratic attention algorithm works extremely well for quality, but it's extremely expensive once you get to large token counts. You need something to scale beyond a million tokens. There are possibilities that have been proven at modest scales and look promising. One is to reduce the constant factor of quadratic attention substantially by paying attention to chunks of tokens, identifying which chunks seem most interesting, and then delving into those chunks only.

这不仅仅是常数因子，而是把它从 n 平方变成 n log n 之类的。

That's not just constant factor, that's changing it from n squared to n log n or something.

嗯，不，它仍然是 n 平方，因为块的长度可能是 128，你仍然对这些块进行二次注意力。

Well, no, it's still n squared because the chunks might be 128 in length and you're still doing quadratic attention to those things.

好的，我明白你的意思，但每个操作可以小得多。

Okay, I see what you're saying, but each thing you do can be much smaller.

是的，这有帮助。还有一些有趣的方法，你可以对注意力状态进行聚类，只关注与当前任务相近的聚类。这些可以是 n log n 甚至更好。这看起来很有前景。权衡是通常相对于完整的二次注意力会损失一点质量。还有一个概念是，当你把正确的信息放入上下文窗口时，模型表现很好，因为它们可以关注它，但注意力窗口不够大，无法满足你真正想要的——关注所有信息，比如互联网上的所有文档，或者我所有的电子邮件和照片。这远远超过一百万个 token。让你产生关注一万亿 token 的错觉的正确方法是采用分阶段的、更轻量级的检索机制，识别最相关的文档，然后拉入并关注它们。例如，从一万亿到 1 万份文档，每份一千万 token，然后用更轻量级的机制识别出大约一百万个 token 放入。我对此非常兴奋，因为它能实现仅靠注意力窗口大小无法做到的事情。

Yeah, so that helps. There are also interesting things where you can cluster the attention state and only look at the clusters near the thing you're trying to do. Those can be n log n or even better. That seems quite promising. The trade-off is generally you lose a little quality relative to full quadratic attention. There's also the notion that models are very good when you put the right information in the context window because they can attend to it, but the attention window is not big enough for what you really want—attending to all information, like all documents on the internet or all my email and photos. That's much bigger than a million tokens. The right way to give you the illusion of attending to a trillion tokens is to have staged forms of much lighter weight retrieval mechanisms that identify the most relevant documents, then pull them in and attend to that. For example, from a trillion to 10,000 documents of 10 million tokens, then use a lighter weight mechanism to identify the millionish tokens to put in. I'm pretty excited about that because it enables things that just the attention window size won't permit.

我非常相信 AI 在芯片设计各个方面的应用。无论是高层架构探索、自动综合、验证，还是布局布线。我们做过 AlphaChip 的工作，帮助布局布线。这对好几代 TPU 都非常有用。我看到 Nvidia 在基于梯度的布局布线方法上做了很好的工作。你如何看待 AI 在实际硬件设计过程中的使用？你认为用不同的 AI 方法处理不同阶段是正确的吗？你认为更端到端的方式可能吗？你如何看待它的发展？

I'm a big believer in the use of AI for various aspects of chip design. Both in high-level architectural explorations, automated synthesis, verification, and placement and routing. We've done some work on AlphaChip as one example, helping with placement and routing. That's been really useful for a bunch of generations of TPUs. I've seen nice work from Nvidia on gradient-based methods for placement and routing. How do you think about the use of AI in the actual hardware design process? Do you think tackling different stages with different AI-based approaches is the right thing? Do you think a much more end-to-end thing is possible? How do you see this evolving?

我们在设计过程中尽可能使用 AI。例如，每次有新的半导体工艺，我们都需要移植标准单元库。大约有 2500 到 3000 个单元，过去需要一个八人团队大约 10 个月，也就是 80 人月。我们开发了一个基于强化学习的程序叫 NB-Cell，现在已经是第 2 或第 3 版。它在一台 GPU 上运行一夜，结果在尺寸、功耗和延迟方面实际上优于人类设计。这是一个巨大的生产力提升，消除了转向新工艺的障碍。我们还有一个叫 Prefix RL 的程序，它将强化学习应用于一个古老的问题——在进位选择链中放置前瞻级的位置。这个问题自 20 世纪 50 年代以来就被研究。这个 RL 程序把它当作一个 Atari 视频游戏，试图制造一个刚好满足时序、尽可能小且低功耗的加法器。

We're trying to use AI wherever we can in our design process. For example, every time we have a new semiconductor process, we have to port our standard cell library. It's about 2,500 to 3,000 cells, and that used to take a team of eight people about 10 months, so 80 person-months. We developed a program based on reinforcement learning called NB-Cell, now up to version 2 or 3. It runs overnight on one GPU, and the results are actually better than human designs in measures of size, power dissipation, and delay. It's a huge productivity gain and eliminates an obstacle to moving to new processes. We also have a program called Prefix RL, which applies reinforcement learning to the age-old problem of where to put look-ahead stages in a carry look-ahead chain. This problem has been studied since the 1950s. The RL program treats it like an Atari video game, trying to make the adder that barely meets timing and is as small and low power as possible.

它会产生完全奇怪的设计，人类永远想不出来，但按照那些指标，实际上比人类设计好 20% 或 30%。然后在更整体的阶段，我们多年来有一系列 LLM，称为 Chip NeMo 和 Bug NeMo。我们做的是，拿一个通用 LLM，然后通过喂给它所有 Nvidia 专有的设计文档来微调它。所以这是你在公司外部得不到的东西。它包括所有 RTL 硬件设计文档、Nvidia 设计过的每个 GPU 的所有 RTL、这些 GPU 的所有架构规格，等等。现在你有了这个对 GPU 设计非常了解的 LLM。最大的好处之一是，当你有一个初级设计师时，高级设计师会花大量时间向初级设计师解释简单的事情，比如纹理单元如何工作？特别是这个纹理单元。现在他们不必问高级设计师了。他们可以问 Chip NeMo，Chip NeMo 会非常详细地解释纹理单元如何工作。他们有后续问题，它也会回答。这样提高了生产力。

And it comes up with totally bizarre designs that no human would ever come up with, but they're actually 20 or 30% better than the human designs by those metrics. And then at a more holistic stage, we for a number of years had a series of LLMs that we called Chip NeMo and Bug NeMo. What we did is we took a generic LLM and then we fine-tuned it by feeding it all of the design documents proprietary to Nvidia. So this is stuff that you can't get outside the company. It's all of the RTL hardware design documents, all of the RTL for every GPU ever designed at Nvidia, all of the architecture specs for those, all sorts of stuff. And now you have this LLM that's actually very smart about GPU design. One of the biggest gains is when you have a junior designer, it turns out the senior designers spend an enormous amount of their time explaining to the junior designers simple things like, how does a texture unit work? And in particular this texture unit. And now they don't have to ask the senior designers that. They can ask Chip NeMo and Chip NeMo will explain to them in great detail how the texture unit works. They have a follow-up question, it'll answer the follow-up question. So it improves productivity that way.

这是一个非常有耐心的导师。

It's a very patient mentor.

是的。同样，当我们有 bug 时，我们可以让它总结 bug 报告并尝试对 bug 进行归因。比如这个 bug 在哪个模块？我们指派给谁？我们总是为每个 bug 指派一个人。需要谁采取行动。然后我们会决定哪个设计师被指派处理特定 bug。然后在更探索性的层面，特别是在 Nvidia 研究部门，我们现在使用智能体式系统来进行一系列思想实验，比如，好的，这里有很多不同的方向，弄清楚这个参数空间是什么样的，提出新的做事方式，运行简单的架构实验，并将设计空间缩小到我们可以关注的更少事物。我们正在努力缩短时间。在 Nvidia，有一个阶段我们基本完成了探索工作，我们称之为 F 模型。它基本上是一个 GPU 的可执行模型，直到我们流片设计。我们完成了所有几何设计。它被送到台积电制造芯片。我们想缩短那个空间。结果发现那个空间真正的瓶颈是设计验证。所以我们特别关注如何利用 AI 更快地证明设计有效。还有几个阶段我们需要重构设计，因为 RTL 设计完成了，但之后我们为了布局规划重新划分了模块。现在我们必须将部分逻辑从一个模块移到另一个模块，并确保一切仍然有效。我认为这些是我们应该能够自动化的事情。我们应该能够缩短那个时间段。但尽管如此，我很想有一个端到端的阶段，我可以简单地说，设计一个新的 GPU，然后我去滑雪几天，回来就完成了。但我认为我们离那还很远。但我认为它正在让我们更加高效。

Yeah. And the same thing when we have bugs, we can have it summarize bug reports and try to do attribution on bugs. Say which module is this bug in? Who do we arb? We always arb somebody for a bug. Action required by. And so we'll decide what designer gets arb'd for a particular bug. And then, at the more exploration level, particularly in Nvidia research, we're now using agentic systems to try to do a bunch of thought experiments where we say, okay, here's a bunch of different ways we could go, figure out what this parameter space looks like, suggest new ways of doing things, run simple architecture experiments and whittle the design space down to a smaller number of things that we can look at. And we're trying to collapse the time. At Nvidia, there's a stage when we're kind of done with the exploration effort that we call the F model. It's basically an executable model of the GPU to where we tape out the design. We have all the geometry done. It goes out to TSMC to make chips. And we'd like to collapse that space. What turns out the real long pole in that space is design verification. So we're particularly looking at how we can use AI to prove the designs work more quickly. There are also a couple stages where we have to refactor the design because the RTL design is done, but then we've repartitioned things for floor planning. And now we have to move parts of the logic from one module to another and make sure everything still works. Those are the kind of things that I think we ought to be able to automate. We should be able to collapse that time period down. But all that said, I'd love to have the end-to-end stage where I could simply say, design me the new GPU and I'll go out skiing for a couple days and I'll come back and it's done. But I think we're a long way from that. But I think it is making us much more productive.

是的，我认为让这个自动化过程的每个阶段都足够快，以至于它能够真正探索这个巨大的设计空间，这是非常非常具有挑战性的。

Yeah, I mean, I think making every stage of that automated process fast enough that it can actually explore this enormous design space is super super challenging.

我认为即使我们实现了端到端，它也会有点像主智能体调用那些专门负责流水线不同部分的智能体来执行它们的部分，然后如果对结果质量不满意，它们可能会迭代。

I think even when we get to the end-to-end, it's going to be sort of the master agent calling on these agents who are specialized in the different parts of that pipeline to carry out their parts and then perhaps they iterate when it's not happy with the quality of the result.

是的，甚至可能像我们拆分人类芯片设计团队那样拆分，让这个智能体专注于芯片的这一部分，另一个专注于另一部分，它们之间有众所周知的接口，然后它们必须协商对这些接口的更改。

Yeah, and potentially even split up in the same way we split up human chip design teams where you have this agent focusing on this part of the chip and then this one over here with some well-known interfaces between them and then they have to negotiate changes to those interfaces.

就像我们开的那种会议，但发生在智能体之间。

Are the same kind of meetings we have but between agents.

是的。跨智能体会议。

Yes. Interagency meetings.

好的，我老是忘记现在要问问题。我应该把这个组织成两列之类的。好吧。看看我们有什么。是的，关于智能体这个话题，如果你有虚拟工程师团队和智能体群在复杂项目上协作，硬件创新可以极大加速哪些最大的编排和持续学习挑战？比如分层奖励、稀疏激活、动态模型增长。你觉得那里会发生什么？

Okay, I keep forgetting I have to ask a question now. I should have organized this in two columns or something. Okay. Let's see what we got here. Yeah, so on that topic of agents, if you have teams of virtual engineers and swarms of agents collaborating on complex projects, what are the biggest orchestration and continual learning challenges that hardware innovations could dramatically accelerate? Things like hierarchical rewards, sparse activations, dynamic model growth. What do you see happening there?

是的，我认为当今机器学习的一大挑战是如何让系统在奖励信号稀疏且可能不像数学或编码那样可验证的情况下工作。对吧？如果你能真正解决这个问题，那么我认为你可以学习执行大量不同的任务，极大地扩展这些模型的专业领域的能力。我确实认为有一些具有挑战性的事情，因为随着我们获得这些基于智能体的系统，它们通常看起来有大量正在展开的轨迹。你希望这些轨迹在模型中尽可能低延迟地生成下一段代码或模型可以采取的下一组动作。然后它们将与某个环境交互。通常它们与环境交互的方式是使用为人类速度交互设计的工具，对吧？所以，甚至

Yeah, I think one of the big challenges in ML these days is how do you get systems working where the reward signal is sparse and maybe not as verifiable as say math or coding. Right? If you can really crack that, then I think you can learn to do a whole bunch of different tasks, dramatically expanding out the set of capabilities of the areas of expertise of these models. I do think there's some challenging things because as we get these agent-based systems, they typically look like they have a whole bunch of trajectories that they're rolling out. You'd like those to be as low latency as possible in the models to generate the next bit of code or the next set of actions the models can take. And then they're going to interact with some environment. And often the way they interact with that environment is they use tools that were designed for human speed interaction, right? And so, even

而且通常在 CPU 上运行。

And often run on CPUs.

而且在 CPU 上运行。所以，你的 C 编译器的启动时间不一定是人们会注意的事情，但在这个智能体比人类快 50 倍的世界里，他们需要更加注意。所有工具的启动时间，我认为将开始成为一个类似阿姆达尔定律的瓶颈，因为如果你让你的模型无限快，根据你正在做的事情，如果你的工具占你工作量的很大一部分，你最终只会获得两到三倍的延迟降低。所以，我认为我们将需要开始真正重新设计很多这些模型使用的工具。

And run on CPUs. And so, the startup time of your C compiler is not necessarily something that people pay some attention to, but they need to pay a lot more attention to it in this world where you have an agent that is operating 50 times faster than a human. And the startup time of all your tools, I think is going to start to be an Amdahl's law-like bottleneck because if you make your model infinitely fast, you're going to get, depending on what you're doing, a factor of two or three end reduction in latency if your tools are a pretty significant fraction of what you're doing. So, I think we're going to need to start to really re-engineer a lot of the tools that these models

所以，这现在正在发生在编码工具上吗？

So, this is happening right now for coding tools?

是的。编码工具正在发生这种情况。

Yeah. It's happening for coding tools.

能源问题我在你的很多演讲中都听你提到过。让我印象深刻的是，我们最终要做的不过是一个乘加运算，根据精度不同，能耗只有几分之一或几个皮焦耳。但为了实现这个运算，我们必须从内存中读取数据，而这在许多情况下能耗要高出上千倍。鉴于我们正在努力提高能效，你认为主要的改进方向是什么？你提到过缩短芯片上的传输距离，但或许你有更全面的看法。

Energy is something I've seen you talk about in many of your talks. What sticks with me is that what we ultimately are trying to do is a multiply-add operation, which is a fraction or a few picojoules depending on precision. But to make that happen, we have to bring data from memory, which is a factor of a thousand more energy in many cases. Given we're pushing for energy efficiency, what do you think will be the big improvements? You mentioned shrinking distances across the chip, but maybe you have a broader view.

我来把数字说清楚：NVFP4 的一个乘加运算大约需要 10 飞焦耳。从 HBM4 中读取这 4.5 比特，每比特大约 3 到 4 皮焦耳，总共约 15 皮焦耳。所以从外部内存读取一个 NVFP4 数字比做一次乘加运算要多花一千倍的能耗。但从简单的 SRAM 存储体读取也只需要大约 10 飞焦耳。因此，降低能耗的关键就是不要移动数据。例如，我们正在研究一些设计，把数据放在 SRAM 中，读取矩阵的一行和激活向量，然后就地做点积。这样我们就完成了上千次乘加运算，没有数据移动。然后通过点积，我们把数据从一千个缩减到一个。问题在于 SRAM 很贵，每比特面积比 DRAM 贵一个数量级。我们很兴奋但尚未成熟的技术是把 DRAM 堆叠在计算芯片之上。这样我们可以循环 DRAM 的存储单元，直接向下感应比特。DRAM 的读取能耗大部分并不在读取 DRAM 本身，而在于把比特从读取位置移动到 GPU 引脚。通过把 DRAM 直接放在 GPU 顶部，我们可以获得一个数量级更高的带宽，同时每比特能耗降低一个数量级，最终功耗相同但性能大幅提升。这就是我们正在研究的方向。当然，更高效的模型也有帮助——如果能用更少的计算达到同样的精度，就能降低能耗。稀疏性是一个可能大有帮助的领域。从 Ampere 的 2:1 结构化稀疏开始，我们已经做了很多工作，但这些模型天生就是稀疏的。然而，除了 2:1 结构化稀疏和 MoE 的粗粒度稀疏之外，很难以一种有效的方式利用稀疏性，因为它会引入不规则性，破坏使计算高效的规律性。一旦打乱这种规律性，就需要更多的控制和数据路由来处理。所以，如何将稀疏性推向下一层是一个有趣的开放问题。

To put the numbers straight: a multiply-add for NVFP4 is about 10 femtojoules. To pull those four and a half bits from HBM4, at maybe three or four picojoules per bit, that's about 15 picojoules. So it's a thousand times more energy to read one NVFP4 number from external memory than to do a multiply-add. But reading from a simple SRAM bank is also about 10 femtojoules. So the key thing to reduce energy is to not move data. For example, we're looking at designs where we put things in SRAM, read a row of a matrix and an activation vector, and do a dot product right there. That's a thousand multiply-adds with no data movement. Then we reduce the data from a thousand to one by doing the dot product. The problem is SRAM is expensive in area per bit—an order of magnitude more than DRAM. The technology we're excited about, though not quite ready, is stacking DRAM on top of the compute chip. Then we can cycle the DRAM mat and sense the bit straight down. Most of the read energy of DRAM isn't reading the DRAM itself, but moving the bit to the GPU pins. By putting DRAM directly on top of the GPU, we can get an order of magnitude more bandwidth with an order of magnitude less energy per bit, resulting in the same power but much more performance. Those are the things we're looking at. Of course, more efficient models help too—if you can get the same accuracy with less work, that reduces energy. One area that could really help is sparsity. We've done a lot with sparsity starting with Ampere's 2:1 structured sparsity, but these models are naturally sparse. However, aside from 2:1 structured sparsity and the coarse sparsity of MoE, it's hard to exploit sparsity in a winning way because it creates irregularity, destroying the regularity that makes computation efficient. When you disrupt that, you need more control and data routing. So it's an interesting open question how to push to the next level on sparsity.

确实。那些多次复用参数的模型在某些方面是很好的。

Definitely. Models that reuse parameters many times are nice in some ways.

复用是好的，但这是最优吞吐量（每瓦特 token 数）和最优交互性之间的权衡。当你关心交互性或者只运行单个模型时，一个 token 必须看到所有权重，复用率就是 1。

Reuse is good, but it's part of the trade-off between optimum throughput (tokens per watt) and optimum interactivity. When you're worried about interactivity or running a single model, and you have one token that must see every weight, the reuse is one.

除非是卷积模型，那样会有相当多的复用。

Unless it's a convolutional model, which gives quite a bit of reuse.

是的，计算会分散开。但对于大语言模型，它是矩阵向量运算。

Yeah, you get the spread of the compute. But for LLMs, it's matrix-vector.

你的 TPU 协同设计循环——提前 2 到 6 年预测研究方向并集成到芯片中——取得了巨大成功。你今天在谷歌看到的最大的硬件-软件摩擦点是什么，特别是在混合 GPU/TPU 或多云环境中，整个行业应该解决这些问题？

Your TPU co-design loop—predicting research directions 2 to 6 years ahead and baking them into silicon—has been remarkably successful. What are the top hardware-software friction points you're seeing at Google today, especially in mixed GPU/TPU or multi-cloud environments, that the broader industry should solve?

是的，我们做了很多协同设计，因为你需要硬件设计师和机器学习研究人员经常交流。机器学习研究人员可能会说，‘我认为一两年后我们会大量做这个’，硬件人员看着说，‘我们没法让它变得很快。’于是我们迭代。最大的摩擦点包括不同硬件类型（如 GPU 和 TPU）之间以及跨云环境之间需要更好的集成。接口和软件栈的标准化将有助于整个行业更快发展。

Yes, we do a lot of co-design because you want hardware designers and ML researchers talking to each other often. An ML researcher might say, 'I think we'll be doing a lot of this in a year or two,' and the hardware people look at that and say, 'We can't make that really fast.' So we iterate. The top friction points include the need for better integration between different hardware types, like GPUs and TPUs, and across cloud environments. Standardization of interfaces and software stacks would help the industry move faster.

如果我们能做出这个东西，虽然不是完全那样，但对你来说可行吗？这种互动，然后机器学习研究人员会说，“哦，也许我们可以做个原型……”

What if we can make this thing that's, you know, not quite that, would that work for you? And that kind of interaction and the ML researchers then sort of say, "Oh, well, well, maybe we could prototype this and"

然后他们就会意识到什么是可能的，什么是容易的，什么是困难的。

And they sort of realize what is possible and what is easy and what is hard.

我认为这种互动非常重要。你知道，因为我们设计硬件，并且做很多建模和机器学习研究，我们确实有一个很好的窗口来了解这个领域可能的发展方向。我们有时能做的一件事就是把一些我们认为在未来 2 到 4 年会有用的实验性东西放进芯片里。显然，我们希望押对宝，但如果没押对，也不是什么大灾难，那只是意味着芯片的那一小部分会被利用不足，或者只在某些情况下使用。但如果它真的有效，那就会带来巨大的回报，你可以突然在硬件上实现 10 倍甚至 20 倍的加速。

I think that kind of interaction is really, really important. You know, I think because we design the hardware and we do a lot of modeling and ML research, we do have a nice window into where we think the field might go. And one of the things we sometimes can do is put a little bit of experimental things that we think will be useful 2 to 3 or 4 years from now into the silicon. And we, you know, obviously hopefully bet correctly, but if we don't, it's not like a huge disaster because then that just means that little bit of the chip will be underused or only used in certain circumstances. But if it actually does work, then it pays off big and you can suddenly accelerate something like 10 or 20x in the hardware.

这有点像我们的 CUDA 税，所有 GPU 都用于图形处理。

It's kind of like our CUDA tax that all of our GPUs use for graphics.

对，你只需要放一点进去，如果能用上就很好，用不上也没关系。

Yeah, you just put a little bit in there and if you can use it, it's great. If not, it's not a big deal.

嗯。

Yeah.

所以现在的一个大趋势是拥有更多持续学习系统。这些系统可能如何演变的一种可能性是，不再是一个固定的模型和一组参数，而是有一些更有机生长的模型架构，可以添加参数，或者剪枝参数，或者将现有参数集压缩成更小的集合，从而释放内存用于其他事情。你认为哪些硬件特性对此有帮助？或者有没有一些特性，即使我们有这些非常奇怪的有机模型结构，也不会降低利用率？

So one of the things that is a big trend right now is having much more continual learning systems. And one possibility for how those might evolve is instead of a fixed model with a set of parameters, you have some more organically growing model architectures that sort of add parameters or maybe prune parameters away or compress existing set of parameters into smaller sets so that you free up memory for other things. How do you What hardware features do you think would be helpful for that? Or are there features that won't tank utilization if we have these really weirdly organic model structures?

是的，我认为这实际上是一个关于不规则性粒度的问题。只要你有足够大的块，可以高效地调度运行，然后动态选择下一个块，它可能更大，也可能更小。那么我认为你仍然可以非常高效地运行，因为我们可以应用我们今天所知的所有技术来使这些单独的块高效运行。然后，在任何并行计算系统中，真正关键的是同步和通信。我们一直在努力，现在我们有了闪电般的同步机制。所以，当一个块完成时，你可以检测到它已完成，并触发下一个块。然后，你的通信可能比在非常规则的计算中更少编排，这有点问题。但如果我们能应用那些给我们带来光速延迟和最小能量的原则，我们应该能够非常有效地从一个块的输出通信到下一个块的输入。所以，我认为我们可以很好地处理动态计算。

Yeah, I think it's a real question of the granularity of that irregularity. As long as you have a big enough block that you can kind of schedule something very efficiently to run and then you can dynamically select what is the next block. It might be bigger, it might be smaller. Then I think you can still do it very efficiently because we can apply all the techniques we know today to make those individual blocks run efficiently. And then the really critical thing in any parallel computing system is synchronization and communication. And we've been working very hard on we have lightning-fast synchronization mechanisms these days. So, when one block is done, you can detect it's done and fire the next one off. And then it will be a little bit of a question that your communication will probably be less choreographed than it is in a very regular computation. But if again, if we can apply the principles that give us speed of light latency and minimum energy, we should be able to communicate from the output of one block to the input of the next block very effectively. So, I think we can do a pretty good job of handling dynamic computations.

是的，我有点同意。只要东西有一个最小尺寸，那么

Yeah, I kind of agree. As long as things are a minimum size, then

没错。只要你不试图把它压缩到一个操作。

That's right. As long as you don't try to push it down to one op.

对，对，那就会

Yeah. Yeah, then it'll be

对，比如大概一万次操作。

Yeah, like maybe 10,000 ops.

对，一万次操作很好。否则，我们就回到 CPU 了。

Yeah, 10,000 ops is good. Yeah. Otherwise, we're back to a CPU.

嗯。

Yeah.

好的。我们的问题快问完了，所以得开放给观众了。

Okay. We're running out of questions, so we're going to have to open it up to the audience.

我们没想到能问完这些问题。我们应该多准备一些的。

And we didn't think we'd get through these questions. We should have made more.

确实。

It's true.

那么，为了达到每秒一万以上 token 的推理密集型智能体工作负载，你认为哪些算法技术，比如高级推测解码、大规模 MoE 路由、超低精度推理，需要最大的相应硬件变化？比如更深的每向量量化支持、最优裁剪或对数表示？

So, to hit that 10,000 plus tokens per second regime for reasoning-heavy agent workloads, which algorithmic techniques like advanced speculative decoding, MoE routing at massive scale, ultra-low precision inference, do you think will require the biggest corresponding hardware changes? Like deeper support for per vector quantization, optimal clipping, or logarithmic representations?

是的，我的意思是，我认为要真正让低延迟推理对非常大的模型有效，你必须从第一性原理思考：我们能做什么来最小化数据移动，最小化能量，并最小化从输入到块、层或注意力计算等输出的时间。这些东西可能关乎重新思考整个硬件流水线如何工作，而不是某个特定的数值格式。我认为这确实需要重新思考很多关于推理硬件的事情，因为你真的不想移动任何东西。我的意思是，之前我们这么说的时候有人笑了，但我认为这确实是关键：不移动任何东西，它就会非常高效且能耗极低。

Yeah, I mean, I think to really make low-latency inference effective for very large models, you have to think from first principles about what can we do to minimize the data movement, to minimize the energy, and minimize the time to go from an input to the output of a block or a layer or attention calculation or whatever. And those things really are probably about rethinking how that entire hardware pipeline works, not about a particular numeric format here or there. I think it really requires rethinking quite a lot about the inference hardware because really you don't want anything to move. I mean, someone laughed when we said that earlier, but I think that's really the key: don't move anything, and it'll be super efficient and super low energy.

如果你把所有东西都放在 SRAM 里，你可以做到这一点，但那样会变得非常昂贵。

And you could do that if you put everything in SRAM but then it becomes super expensive.

是的，所以新的内存技术总是受欢迎的。我认为在芯片上堆叠 DRAM 可能是一个相当有趣的方向，因为这样你可以获得更大的容量和更低的每比特成本，同时仍然不需要大量移动数据，就像弹球机一样，你只需将想要的比特掉入操作器。所以我认为这些可能是非常令人兴奋的方向：如何让非常大的模型实现非常低的延迟，因为我认为这将改善智能体的整体使用，并且假设我们的工具也足够快，这将极大地改善这些系统的能耗。

Yeah, so new memory technologies are always welcome, I would say. I think the stackable DRAM on top of a chip is probably a pretty interesting direction because then you can get much more capacity and cheaper cost per bit and still have that you don't move the data very much, like a pinball machine where you just drop the bit you want down into the operator. So I think those are probably really exciting directions: how do you make incredibly large models very low latency, because that I think will improve the whole use of agents and all the assuming we make our tools fast enough as well, that will really dramatically improve the energy usage of these systems.

是的，在数值方面，你知道 Nvidia 之前在 NVF 所做的，很多人都在模仿并做类似的块缩放格式。你看到数据表示方面还有哪些更令人兴奋的发展来降低能耗吗？这方面有哪些即将出现的东西？

Yeah, on the numeric front, you know what Nvidia were doing at NVF before and a lot of people are copying that and doing similar sort of block scaling formats. Do you see any more really exciting developments in data representation to get these energies down? What are the things on the horizon there?

是的，我的意思是，我认为有一种表示法，只有很少的比特，然后有应用于许多这些单元块的缩放因子，这可能是处理每个单独事物可能具有的非常低精度的有效方法，但随后让你可以假装它们都具有更高的精度。这似乎是一个非常有效的技术，我们也一直在做很多这方面的工作。

Yeah, I mean, I think having a representation that is very few bits and then you have scaling factors that are applied to chunks of many of those units is probably a pretty effective way of dealing with the very low precision that you might have in each individual thing, but then enabling you to sort of pretend like they're all much higher precision. And that seems like a very effective technique, and we've been doing quite a bit of that as well.

是的。你有没有研究过像使用查找表这样的东西？你可以把每个符号精确地放在你想要的位置，但你只能得到少数几个。

Yeah. Have you looked into things like using look-up tables? You could put each symbol exactly where you want it, but you only get a few of them.

我们的 TPU 设计有一个特点，就是采用定制的网络，通常是 2D 或 3D 环形拓扑，点对点连接，可以扩展到数千个芯片。这与完全可交换的网络结构不同，后者可以扩展到单台机器之外。您如何看待这两种设计之间的权衡？

One of the things about our TPU designs is they have custom networks that tend to be 2D or 3D toruses, point-to-point, scaling to thousands of chips. That's different from a fully switchable network fabric that scales beyond a single machine. How do you think about the trade-offs?

这是个好问题。我的主要学术研究方向就是互连网络，而且我确实写过这方面的书。2D 和 3D 环形网络在很多方面都很出色。在 80 年代末 90 年代初，我领导了一场超级计算机的革命，大量使用 3D 环形网络，比如 Cray T3D 就是一个典型例子。但到了 2000 年代，芯片的引脚带宽大幅增加，如果路由器芯片只有六个连接（来自 3D 双向网络），就无法有效利用这些带宽。所以我们转向了高基数路由器，并采用了一系列新的拓扑结构，比如折叠 Clos（常称为胖树）和蜻蜓网络。你不能简单地说哪种更好，这完全取决于工作负载和流量模式。对于局部性很强的工作负载，低基数直连网络（如 3D 环形）非常理想，因为你可以把问题映射上去。但对于 MoE 模型，专家分散在各处，需要很多跳才能到达，那么一跳上交换机、一跳下到目标专家会更高效。我们尝试过混合方案。比如在 NV 研究的一个实验性推理系统中（从未面世），我们使用了扁平蝶形拓扑进行局部通信（直连，无交换机），比环形有更大的覆盖范围，同时所有芯片也连接到交换机层次结构进行全局通信。

That's a great question. My major academic research area was interconnection networks, and I literally wrote the book on this subject. 2D and 3D torus networks are great for many things. In the late 1980s and early 90s, I led a revolution in supercomputers using 3D torus networks, like the Cray T3D. But in the 2000s, we got so much pin bandwidth on chips that a router chip with only six connections (from a 3D bidirectional network) couldn't use it effectively. So we moved to high-radix routers and new topologies like folded Clos (often called fat trees) and dragonfly networks. You can't say one is better than the other; it depends on the workload and traffic pattern. For a very local workload, a direct-connected network with low radix like a 3D torus is ideal because you can map the problem. But for an MoE model with many experts scattered around, many hops are needed, so it's more efficient to go one hop up to a switch and one hop down to the expert. We've experimented with hybrids. For example, in an experimental inference system at NV research that never saw the light of day, we used a flattened butterfly topology for local communication (direct connection, no switches) for more reach than a torus, and all chips were also connected to a switch hierarchy for global communication.

我完全同意。这完全取决于你的工作负载。

I totally agree. It depends on exactly what your workload is.

给定一种流量模式，你可以设计出最优的网络，但没有一种网络能适合所有流量模式。

Given a traffic pattern, you can come up with an optimum network, but there's no network that is good for all traffic patterns.

确实如此。

Yes, indeed.

随着我们建造这些运行着惊人模型的奇妙机器，您认为对人类生存最积极的影响是什么？教育、医疗、商业生产力？您最期待的是什么？

As we build these wonderful machines that run amazing models, what do you think will be the most positive impact on human existence? Education, medicine, business productivity? What are you most excited about?

这是个好问题。AI 可以应用于很多领域。有些对社会极其有益，比如教育和医疗。有些可能我们并不希望被大量使用。大约一年前，我和几位合著者写了一篇论文，研究了七个不同领域，论文有自己的网站 shapingai.com。这些领域包括教育、医疗、劳动就业、科学、媒体（包括内容生成和虚假信息）。我特别看好医疗和教育。在教育方面，我们知道，一个了解学生已知知识、学习材料以及最佳学习方式的个性化导师，比群体教育有效得多。个性化辅导可以使教育成果提高一到两个标准差。如果每个人都能拥有一个个性化的 AI 导师，能够适应他们的学习方式，而且这些模型可以将一种模态转换为另一种，这就能帮助那些更适合通过听播客学习生物章节而不是阅读的人，或者通过互动游戏探索重力而不是阅读物理课本方程的人。我认为在未来几年内，我们就能拥有出色的教育导师，它们不会直接给出答案，而是帮助人们更高效、更有效地掌握概念。

That's a good question. AI can be applied to many things. Some are incredibly societally beneficial, like education and healthcare. Some are potentially things we don't want much use of. About a year ago, I and co-authors looked at seven different domains for a paper with its own website, shapingai.com. The domains include education, healthcare, labor and employment, science, media (both generation and misinformation). I'm particularly excited about healthcare and education. In education, we know that an individualized tutor who knows what the student knows, the material, and how they learn best is much more effective than group education. Educational outcomes are one to two standard deviations higher with a personalized tutor. If everyone can have a personalized AI tutor that adapts to their learning style, and these models can turn one modality into another, that helps people who learn best by listening to a podcast about biology rather than reading, or by playing an interactive game to explore gravity rather than reading equations. I think within the next few years, we can have amazing educational tutors that don't give away answers but help people learn concepts better and more efficiently.

是的。

Yeah.

有趣的是，如今你接触很多学校，人们试图限制 AI 的使用，而实际上他们应该鼓励学生学会如何有效地与 AI 协作。

It's interesting, because you deal with a lot of schools these days. People are trying to restrict the use of AI, where instead they ought to be encouraging students to learn how to work effectively with AI.

是的。

Yeah.

而辅导几乎与此正交。

And the tutor is almost even orthogonal to that.

是的，我的意思是，回想一下计算器刚引入数学课的时候，有些数学老师说：‘我不知道，我们不能用计算器。人们永远不会知道怎么加减乘除。’但相反，它只是意味着我们可以教授数学，其中计算不再是瓶颈，所以我认为我们可以更快地达到更高水平。

Yeah, I mean, if you think back to when calculators were first introduced into math classes, there were some math teachers who said, 'I don't know, we can't use calculators. People will never know how to add or multiply.' But instead, it just meant that we could teach mathematics where the calculation wasn't the bottleneck, and so I think we can go up to a higher level much more quickly.

我认为教育工作者应该真正接受这些工具存在并且会做有趣的事情这一事实。也许我可以反过来问你：哪些 AI 应用领域最让你兴奋，你认为哪些领域影响最大？

I think educators should really be leaning into the fact that these tools exist and will do interesting things. Maybe I can turn it around to you and say: what are the application areas of AI that excite you the most and where you think will have the most impact?

嗯，让我兴奋的两个领域是：首先是 AI 用于硬件设计，因为我本质上是一名硬件设计师。我感兴趣的是能在 15 分钟内完成工作，然后出去做点有趣的事。我认为那里有很多富有成效的东西。很多工作只是跟踪项目的所有文档，能够快速查询所需信息，总结错误报告等等。但我认为我们也非常接近能够让它设计一个纹理单元，如果你向它充分解释纹理单元是什么，它很可能提出一个相当不错的设计。更难的是说：拿我们 Blackwell 一代的纹理单元，这是它必须为 Rubin 一代做的新事情，这些是我们弃用的功能。做出最小的改动，将代码从一个树迁移到另一个树。

Well, the two that excite me are, first of all, AI for hardware design, since I'm fundamentally a hardware designer. I'm interested in being able to get my job done in about 15 minutes and go out and do something fun for the rest of the day. I think there's a lot of really productive stuff there. A lot of it is just keeping track of all the documentation with a project and being able to query information you need quickly and summarize bug reports and stuff like that. But I think we're also getting very close to being able to ask it to design a texture unit, if you've explained to it enough what a texture unit is, it can probably come up with a pretty good design. What's harder though is to say: take the texture unit that we have in the Blackwell generation, and here are the new things that it has to do for the Rubin generation, and here are the features we've deprecated. Make the minimum change to move the code from one tree to the other.

嗯。

Mhm.

但另一个让我兴奋的领域也是教育。我是一个正在恢复中的教育者。我当了 26 年教授，我认为这真的是……我希望有人在做这件事，也许我应该辞掉日常工作自己去做：拥有一个个性化导师。另一个类似的事情是拥有一个个性化健康教练，对吧？所以把它想象成一个小天使坐在你肩膀上。我今天去了 Jensen 的小组讨论，当我到休息室时，所有食物都没了，但那里有所有甜点，对吧？所以我的午餐就是甜点。但如果我肩膀上有我的小健康教练，它就会劝我不要那样做。

But the other one for me is also education. I'm a recovering educator. I was a professor for 26 years, and I think it really is... I'm hoping somebody is doing this, maybe I should quit my day job and go do it myself: having a personalized tutor. Another thing which is of a similar vein is to have a personalized health coach, right? So think of this as a little angel sitting on your shoulder. I went to Jensen's panel today, when I got to the lounge all the food was gone, but they had all the dessert there, right? So my lunch was dessert. But if I had my little health coach on my shoulder, it would have talked me out of that.

或者它会抛弃你。

Or it would have deserted you.

而且我认为国家的许多健康问题基本上都是饮食和锻炼的问题。如果我们都有个人健康教练，我认为，就像能够触及学生、理解他们的动机一样，他们可以与我们互动，我们有一个共同的目标——不死，他们会鼓励我们沿着这条路走下去。

And I think that a lot of the nation's health problems are basically issues with diet and exercise. If we all had personal health coaches, I think, just like being able to reach a student, understand what motivates them, they could interact with us and we have this shared goal of not dying, and they would encourage us along that path.

不是教育环境，对吧？微积分课没那么戏剧化。

Not the education setting, right? Calculus class is not that dramatic.

是的。

Yeah.

是的，我的意思是，我也对健康特别兴奋。我觉得有太多信息在医疗环境中没有被使用，而这些信息可能与人们的健康相关。比如我戴了一个心率监测手表，但我的临床医生会使用这些信息吗？

Yeah, I mean, I'm particularly excited about health as well. I feel like there's so much information that is not used in healthcare settings that could be relevant to people's health. Like I wear a heart rate monitoring watch, but does my clinician use that information?

而且我们可以轻松地对每个人进行测序，对吧？然后根据你拥有的可能表明你对特定疾病易感性或对某些药物反应的突变，提供非常个性化的医疗。而且其中很多是已知的，但大多数医疗提供者完全忽略了它。

And we could easily sequence everybody, right? And then have very personalized healthcare based on what mutations you have that may indicate your susceptibility to particular diseases or response to certain drugs. And a lot of that is even known, but it's being completely ignored by most healthcare providers.

而你的小健康教练，我认为，是一个非常好的主意，因为我认为医疗系统……

And your little health coach, I think, is a really good idea because I think the healthcare system...

他们能做的事情是跟医生说：‘嗯，你有没有考虑过 Dalle 教授有这个突变，那种药可能效果不好？’

The thing they could do is talk to the doctor and say, 'Well, did you consider the fact that Professor Dalle has this mutation and that drug may not work well?'

我看到你这周第四次去麦当劳了，你知道吗，拐角处有个很棒的沙拉店之类的。是的，所以我的意思是，我认为 AI 在医疗保健中的应用将非常有影响力。这是一个复杂的环境，因为存在非常现实的隐私和监管问题，但我认为我们应该有抱负地尝试达到一个世界，过去的医疗决策被用来为未来的医疗决策提供信息。如果我们能尽可能接近这一点，那就太棒了。

I see your McDonald's the fourth time this week, you know, did you know there's a great salad place around the corner or whatever. Yeah, so I mean, I think that the use of AI for healthcare is going to be really impactful. It's a complicated environment because of very real privacy and regulatory issues, but I think we should aspirationally try to get to a world where past healthcare decisions are used to inform future healthcare decisions. And if we can get as close as we can to that, it would be amazing.

是的。

Yeah.

那么，我要问你一个更哲学的问题。所以，Nvidia 在过去十年中取得了难以置信的成功。你怎么看……你可能雇了更多的人，公司大了很多，你在世界上的足迹更大了。有什么变化是你怀念的，过去运作良好但现在在大规模下不再有效的东西？

So, I will ask you a more philosophical question. So, Nvidia has been incredibly successful in the last decade. How do you think... you've probably hired a lot more people, the company is a lot bigger, you have a bigger footprint in the world. What has changed that you miss, and what used to work well and now no longer works at a larger scale?

是的。所以，这很有趣，因为我从 2003 年开始为 Nvidia 做咨询，当时我还在斯坦福任教。嗯，我们当时肯定在 1000 人左右，也许还少一点。每个人都差不多认识……你不认识所有人，但你认识所有你需要认识的人，决策非常迅速。零官僚主义。我怀念那些日子的很多事情，因为现在有官僚主义了。你成长为一个 4 万多人的公司。按人数来说，我们仍然是一个相对较小的公司。一个 4 万多人的公司，突然之间有很多规则。还有各种官僚，他们有点妨碍以光速做正确的事情。但我认为这是必要的，因为随着你成长，在小规模时人们做正确的事情是因为这是社区的一部分，你永远不会想做让你的社区成员不高兴的事情。但随着公司成长，你有点失去了那种社区感，这时你就开始需要规则了。

Yeah. So, it's interesting because I started consulting for Nvidia in 2003 when I was still on the faculty at Stanford. And yeah, we were definitely right around 1,000, maybe a little bit less than that at that point. And everybody kind of knew... you didn't know everybody, but you knew sort of everybody you needed to know, and decisions just happened very quickly. There was zero bureaucracy. And there's a lot of things I miss about those days because there is now bureaucracy. You grow to be a 40,000-plus person company. We're still a relatively small company in terms of number of people. A 40,000-plus person company, all of a sudden there are lots of rules. And there are various bureaucrats who sort of get in the way of doing the right thing at the speed of light. But I think that's needed because as you grow, at a small size people do the right thing because it's part of a community, and you would never want to do something that your fellow community members would be displeased with you about. But as the company grows and you sort of lose that sense of community a little bit, that's when you start needing the rules.

我想我最怀念的大概是那种社区感有所减弱。但令人惊叹的是，Jensen 能够创造一种环境，即使有四万人，它仍然基本上感觉像一家初创公司。我真的很喜欢这一点。然后你得到的是，由于现在的规模、体量和可用资源，有些事情是可能的，而我们过去做不到。我认为我们需要利用这一点，确保我们利用这种能力来规划未来，做最大的善事。

And I guess probably the thing I miss the most is losing a little bit of that sense of community. But what's amazing is Jensen has been able to create an environment where even at 40,000 people it still to first order feels like a startup. And I really like that. And then the thing you get is that there are things that are possible now because of the size and the scale and the resources available that we just couldn't do back then. And I think that we need to sort of take advantage of that, make sure we chart our future using that capability to do the most good.

是的，我也有类似的经历，在谷歌还很小的时候加入。你知道，我们都挤在帕洛阿尔托市中心现在是一家 T-Mobile 商店的楼上。现在我们有，我不知道，18 万人左右，我觉得公司规模每翻一番，就会导致以前行之有效的东西不再那么有效。你必须想办法作为一个组织来应对，注入恰到好处的新做事方式，但又不能让事情变得过于官僚。所以，是的，这是一个挑战。

Yeah, I mean I've had a similar experience joining Google when we were quite small. You know, we were all wedged in above what's now a T-Mobile store in downtown Palo Alto. And then now we're I don't know, 180,000 people or something and it you know, I feel like every doubling in company size kind of causes something that used to work to no longer quite work as well. You have to kind of figure out how to navigate that as an organization to instill just the right amount of new way of doing things but without making things overly bureaucratic. So, yeah, it's a challenge.

是的，我看到 Maggie 站在侧台。所以，我们差不多该……

Yeah, I've seen Maggie standing in the wings there. So, it must be about time for us to...

非常感谢大家的光临。这真是一次愉快的经历。Jeff，太棒了。Bill，太棒了。

Thank you so much everybody for coming. It was a real pleasure. Jeff, awesome. Bill, awesome.

谢谢。

Thank you.

我们要和观众快速自拍一张。请站在那里。

We're going to take a quick selfie with the crowd. If you could stand there.

哦。

Oh.

好的，转身。转身。

Okay, turn around. Turn around.

开始。

Go.

我们要和观众自拍一张。喔！1 2 3 GTC！谢谢大家。

We're going to take a selfie with the crowd. Woo! 1 2 3 GTC! Thanks everybody.

谢谢。

Thank you.

那么，非常感谢你参与这次活动，Jeff。

Well, thanks so much for doing this, Jeff.

谢谢大家。

Thanks everyone.

这太棒了。我喜欢这种问答互动。我很喜欢……

This was a blast. And I liked the back and forth with questions. I loved...