英伟达黄仁勋谈超摩尔定律与 AI 计算的未来

嗨，听众们，欢迎收听 No Priors。今天我们再次相聚，距离上次与黄仁勋（Jensen Huang）——英伟达创始人兼 CEO——的对话已经过去一年。如今英伟达市值超过 3 万亿美元，它确实是 AI 革命中掌握所有芯片的公司。我们很高兴能在英伟达总部聊聊前沿模型、数据中心规模计算，以及英伟达未来十年的押注。欢迎回来，Jensen。英伟达成立 30 年，展望未来 10 年，你认为还有哪些重大赌注要下？一切都从此刻开始规模化吗？我们在从现有架构中榨取更多算力和内存方面是否遇到了限制？你关注的重点是什么？

Hi listeners, and welcome to No Priors. Today we're here again, one year since our last discussion with the one and only Jensen Huang, founder and CEO of Nvidia. Today Nvidia's market cap is over $3 trillion, and it's the one literally holding all the chips in the AI revolution. We're excited to hang out in Nvidia's headquarters and talk all things frontier models, data center scale computing, and the bets Nvidia is taking on a 10-year basis. Welcome back, Jensen. 30 years into Nvidia and looking 10 years out, what are the big bets you think are still to make? Is it all about scale up from here? Are we running into limitations in terms of how we can squeeze more compute, memory out of the architectures we have? What are you focused on?

如果我们退一步想想我们所做的事情，我们从编码转向机器学习，从编写软件工具转向创造 AI。这一切从在专为人类编码设计的 CPU 上运行，转变为现在在专为 AI 编码（即机器学习）设计的 GPU 上运行。世界已经改变，我们计算的方式、整个技术栈都变了。因此，我们能解决的问题规模也发生了巨大变化，因为如果你能在一个 GPU 上并行化你的软件，你就为跨整个集群、甚至跨多个集群或多个数据中心并行化奠定了基础。所以我认为我们已经将自己置于一个能够以前所未有的水平扩展计算和开发软件的位置。我们现在才刚刚开始。未来十年，我们希望每年在规模上（不是芯片，而是规模）将性能提升一倍或两倍，从而每年将成本降低一半或三分之二，将能耗降低一半或三分之二。如果你每年都这样做，每年翻倍或翻三倍，几年下来就会累积起来。所以它的复合效应非常非常强劲。因此，如果人们像看待摩尔定律那样（每几年翻一番），我们将会处于某种超摩尔定律曲线上，我一点也不会惊讶。我完全希望我们能继续这样做。

Well, if we take a step back and think about what we've done, we went from coding to machine learning, from writing software tools to creating AI. And all of that running on CPUs that were designed for human coding, to now running on GPUs designed for AI coding, basically machine learning. And so the world has changed, the way we do computing, the whole stack has changed. And as a result, the scale of the problems we could address has changed a lot, because if you could parallelize your software on one GPU, you've set the foundations to parallelize across a whole cluster, or maybe across multiple clusters, or multiple data centers. And so I think we've set ourselves up to be able to scale computing at a level and develop software at a level that nobody's ever imagined before. And so we're at the beginning. Over the next 10 years, our hope is that we could double or triple performance every year at scale—not at chip, at scale—and to be able to therefore drive the cost down by a factor of two or three, drive the energy down by a factor of two or three every single year. When you do that every single year, when you double or triple every year, in just a few years it adds up. So it compounds really, really aggressively. And so I wouldn't be surprised if, you know, the way people think about Moore's Law, which is a 2x every couple of years, you know, we're going to be on some kind of a hyper Moore's Law curve. And I fully hope that we continue to do that.

你认为是什么让这比摩尔定律发生得更快？因为我知道摩尔定律有点自我实现，对吧？他提出了这个说法，然后人们某种程度上实现了它。

What do you think is making that happen even faster than Moore's Law? Because I know Moore's Law was sort of self-reflexive, right? It was something that he said and then people kind of implemented it to make it happen.

是的，两个基本的技术支柱，一个是 Dennard 缩放，另一个是 Carver Mead 的 VLSI 缩放。这两种技术都是严谨的技术。但这些技术确实已经耗尽了潜力。所以现在我们需要一种新的缩放方式。显然，新的缩放方式涉及与协同设计相关的各种事情。除非你能修改或改变算法以反映系统的架构，或者改变系统以反映新软件的架构，并来回迭代，除非你能控制双方，否则你没有希望。但如果你能控制双方，你就可以做诸如从 FP64 到 FP32 到 BF16 到 FP8 到 FP4 等等的迁移。所以我认为协同设计是其中非常重要的一部分。第二部分，我们称之为全栈。第二部分是数据中心规模。除非你能把网络当作计算结构，把大量工作推入网络，推入结构，从而在非常大的规模上进行压缩。这就是为什么我们收购了 Mellanox，并开始如此激进地融合 InfiniBand 和 NVLink。现在看看 NVLink 将走向何方。计算结构将扩展出一个看似不可思议的处理器，称为 GPU。现在我们将有数百个 GPU 协同工作。我们现在处理的大多数计算挑战中，最令人兴奋的一个当然是推理时间缩放，它本质上涉及以极低的延迟生成 token，因为你在自我反思。正如你刚才提到的，你将进行树搜索，进行思维链，可能在头脑中进行一定量的模拟，反思自己的答案。你将对自己进行提示，默默地生成文本，并希望在一秒钟内做出回应。要做到这一点，唯一的方法就是延迟极低。与此同时，数据中心仍然需要产生高吞吐量的 token，因为你仍然希望降低成本，保持高吞吐量，产生回报。所以工厂的这两个基本特性——低延迟和高吞吐量——是相互矛盾的。因此，为了创造出在这两方面都非常出色的东西，我们必须发明一些新东西。而 NVLink 正是我们实现这一目标的方式。现在你有了一个虚拟 GPU，它拥有惊人的 FLOPS，因为你需要它来处理上下文，你需要大量的工作内存，同时还要有惊人的带宽用于 token 生成。

Yeah, the two fundamental technical pillars, one of them was Dennard scaling and the other one was Carver Mead's VLSI scaling. And both of those techniques were rigorous techniques. But those techniques have really run out of steam. And so now we need a new way of doing scaling. Obviously, the new way of doing scaling involves all kinds of things associated with co-design. Unless you can modify or change the algorithm to reflect the architecture of the system, or change the system to reflect the architecture of the new software and go back and forth, unless you can control both sides of it, you have no hope. But if you can control both sides of it, you can do things like move from FP64 to FP32 to BF16 to FP8 to FP4 to who knows what, right? And so I think that co-design is a very big part of that. The second part of it, we call it full stack. The second part of it is data center scale. You know, unless you could treat the network as a compute fabric and push a lot of the work into the network, push a lot of the work into the fabric, and as a result you're compressing at very large scales. And so that's the reason why we bought Mellanox and started fusing InfiniBand and NVLink in such an aggressive way. And now look where NVLink is going to go. The compute fabric is going to scale out what appears to be one incredible processor called a GPU. Now we're going to have hundreds of GPUs working together. Most of these computing challenges we're dealing with now, one of the most exciting ones of course is inference time scaling, which has to do with essentially generating tokens at incredibly low latency because you're self-reflecting. As you just mentioned, you're going to be doing tree search, you're going to be doing chain of thought, you're going to be doing probably some amount of simulation in your head, you're going to be reflecting on your own answers. You're going to be prompting yourself and generating text to yourself silently, and still respond hopefully in a second. Well, the only way to do that is if your latency is extremely low. Meanwhile, the data center is still about producing high throughput tokens because you still want to keep cost down, you want to keep the throughput high, you want to generate a return. And so these two fundamental things about a factory—low latency and high throughput—they're at odds with each other. And so in order for us to create something that is really great at both, we have to go invent something new. And NVLink is really our way of doing that. Now you have a virtual GPU that has an incredible amount of FLOPS because you need it for context, you need a huge amount of working memory, and still have incredible bandwidth for token generation all at the same time.

太好了。所有构建模型的人实际上也在非常显著地优化事物吗？比如我团队里的 David 在过去 18 个月左右收集了数据。输入到 GPT-4 等效模型的一百万个 token 的成本基本上下降了 240 倍。所以在那一侧也发生了大规模的优化和压缩。

That's great. Have all the people building the models actually also optimizing things pretty dramatically? Like David on my team pulled data over the last 18 months or so. The cost of a million tokens going into a GPT-4 equivalent model has basically dropped 240x. And so there's just massive optimization and compression happening on that side as well.

仅仅在我们这一层，在我们工作的这一层。你知道，我们非常关心的一件事当然是我们的技术栈生态系统和软件的生产力。人们忘记了，因为你有 CUDA 基础，那是一个坚实的基础，它上面的所有东西都可以改变。如果基础在你脚下变化，就很难在上面建造建筑，很难在上面创造任何有趣的东西。所以 CUDA 使我们能够如此快速地迭代。就在去年，我想我们回去做了基准测试。当 Llama 首次发布时，我们在不改变算法、不改变上层的情况下，将 Hopper 的性能提升了五倍。一年内提升五倍，使用传统计算方法是不可想象的。但加速计算和使用这种协同设计的方式，使我们能够发明各种新东西。

Just in our layer, just in the layer that we work on. You know, one of the things that we care a lot about, of course, is the ecosystem of our stack and the productivity of our software. People forget that because you have CUDA foundation and that's a solid foundation, everything above it can change. If the foundation is changing underneath you, it's hard to build a building on top, it's hard to create anything interesting on top. And so CUDA has made it possible for us to iterate so quickly. Just in the last year, I think we just went back and benchmarked. When Llama first came out, we've improved the performance of Hopper by a factor of five without the algorithm without the layer on top ever changing. Now, a factor of five in one year is impossible using traditional computing approaches. But accelerated computing and using this way of co-design, we're able to invent all kinds of new things.

你最大的客户在多大程度上考虑他们的基础设施在大规模训练和推理之间的可互换性？

How much are your biggest customers thinking about the interchangeability of their infrastructure between large scale training and inference?

嗯，你知道，如今基础设施是解耦的。Sam 刚刚……

Well, you know, infrastructure is disaggregated these days. Sam was just...

他告诉我他最近刚刚退役了 Volta。他们有 Pascal、Ampere，还有各种配置的 Blackwell 即将推出。有些针对风冷优化，有些针对液冷优化。你的服务必须充分利用这一切。Nvidia 的优势在于，你今天为训练构建的基础设施，明天将非常适合推理。我相信大多数 ChatGPT 都是在最近训练所用的同一类系统上进行推理的。所以如果你能在上面训练，就能在上面推理。你留下了一条基础设施的轨迹，这些基础设施在推理方面将非常出色。你完全有信心将这笔投资回报用于构建新的基础设施以继续扩展。你会留下有用的东西，而且你知道 Nvidia 和整个生态系统将致力于改进算法，让你其余的基础设施在一年内提升五倍。这种趋势永远不会改变。因此，人们思考基础设施的方式是：即使我今天为训练而建，它也必须擅长训练，我们知道它也会擅长推理。推理将是多尺度的。首先，为了蒸馏出较小的模型，拥有一个仍然来自大模型的较大模型是好的。所以你仍然会创建这些不可思议的前沿模型。它们将用于开创性工作、合成数据生成，以及大模型教小模型，蒸馏成更小的模型。你可以做很多不同的事情。但最终，你会拥有从巨型模型到微小模型的各种尺寸。微小的模型会非常有效，虽然泛化能力不强，但非常有效。它们会在一个非常小的领域内把那一项任务做得极其出色。我们会看到在一个微小领域内，一个小模型展现出超人般的表现。也许它不是小型语言模型，而是微型语言模型（TLM）之类的。所以我认为我们会看到各种尺寸的模型。

He told me that he had decommissioned Volta just recently. They have Pascals, they have Ampers, all different configurations of Blackwell coming. Some of it is optimized for air cool, some of it's optimized for liquid cool. Your services are going to have to take advantage of all of this. The advantage that Nvidia has, of course, is that the infrastructure that you built today for training will just be wonderful for inference tomorrow. Most of ChatGPT, I believe, is inferenced on the same type of systems that were trained on just recently. So if you can train on it, you can inference on it. You're leaving a trail of infrastructure that is going to be incredibly good at inference. You have complete confidence that you can then take that return on investment and put it into new infrastructure to go scale. You're going to leave behind something of use, and you know that Nvidia and the rest of the ecosystem are going to work on improving the algorithm so that the rest of your infrastructure improves by a factor of five in just a year. That motion will never change. So the way people will think about infrastructure is: even though I built it for training today, it's got to be great for training, we know it's going to be great for inference. Inference is going to be multi-scale. First of all, in order to distill a smaller model, it's good to have a larger model that's still from. So you're still going to create these incredible frontier models. They're going to be used for groundbreaking work, synthetic data generation, and big models teaching smaller models, distilling down to smaller models. There's a whole bunch of different things you can do. But in the end, you're going to have giant models all the way down to little tiny models. The little tiny models are going to be quite effective, not as generalizable, but quite effective. They're going to perform very specific tasks incredibly well, that one task. We're going to see superhuman performance in one little tiny domain from a little tiny model. Maybe it's not a small language model, but tiny language models (TLMs) or whatever. So I think we're going to see all kinds of sizes.

我认为在很多方面，人工智能让我们在创建新应用程序的便捷性上取得了突破。但计算的一切基本上保持不变。例如，维护软件的成本极其昂贵。一旦你构建了它，你希望它能在尽可能大的安装基础上运行。你不想重复编写相同的软件。很多人仍然有同样的想法：你希望把你的工程向前推进。因此，只要架构允许，一方面你今天创建的软件明天在新硬件上运行得更好，这很好。或者你明天创建的软件、你明天创建的人工智能，能在庞大的安装基础上运行。你认为这很好。这种思考软件的方式不会改变。

I think in a lot of ways, artificial intelligence allows us to break new ground in how easy it is to create new applications. But everything about computing has largely remained the same. For example, the cost of maintaining software is extremely expensive. Once you build it, you would like it to run on as large an installed base as possible. You would like not to write the same software twice. A lot of people still feel the same way: you like to take your engineering and move them forward. So to the extent that the architecture allows you, on one hand, to create software today that runs even better tomorrow with new hardware, that's great. Or software that you create tomorrow, AI that you create tomorrow, runs on a large installed base. You think that's great. That way of thinking about software is not going to change.

Nvidia 已经转向越来越大的客户支持单元。我想是从单芯片到服务器到机架，再到 NV72。你怎么看待这种演进？下一步是什么？Nvidia 应该做完整的数据中心吗？

Nvidia has moved into larger and larger units of support for customers. I think about it going from single chip to server to rack, NV72. How do you think about that progression? What's next? Should Nvidia do full data center?

事实上，我们建造完整的数据中心。我们构建一切的方式是：如果你在开发软件，你需要计算机的完整形态。我们不做 PowerPoint 幻灯片然后发货芯片。我们建造整个数据中心。在数据中心完全建好之前，你怎么知道软件能工作？在数据中心完全建好之前，你怎么知道你的网络架构能工作，以及你期望的所有效率？你怎么知道它真的能大规模运行？这就是为什么经常看到实际性能远低于 PowerPoint 幻灯片中展示的峰值性能。计算已经不再是过去的样子了。我说新的计算单元是数据中心。这就是我们必须交付的。这就是我们现在建造的。我们建造一个完整的东西。然后对于每一件事，每一种组合——风冷、x86、液冷、Grace、以太网、InfiniBand、NVLink、无 NVLink——我们建造每一种配置。我们公司目前有五台超级计算机。明年我们很容易再建五台。所以如果你认真对待软件，你就自己造计算机。如果你认真对待软件，你就会建造你的整个计算机。我们在 Goodale 建造这一切。这是非常有趣的一点：我们大规模建造，我们垂直整合，我们全栈优化，然后我们把所有东西拆解成部件出售。这完全是我们所做的非凡之处。其复杂性简直疯狂。原因是我们希望能够将我们的基础设施嫁接到 GCP、AWS、Azure、OCI 中。它们的控制平面、安全平面都不同，它们对集群规模的思考方式也不同。但我们让它们都能容纳 Nvidia 架构，这样 CUDA 就能无处不在。这真的是唯一的想法：我们希望拥有一个开发者可以使用的计算平台，它基本一致，上下浮动 10%，因为人们的基础设施略有不同的优化。但他们构建的一切都能在任何地方运行。这是软件的原则之一，永远不应放弃，我们非常珍视它。它使我们的软件工程师能够一次构建，到处运行。这是因为我们认识到软件的投资是最昂贵的投资，而且测试很容易。看看整个硬件产业的规模，再看看世界软件产业的规模。在这个 1 万亿美元的硬件产业之上，是一个 1 万亿美元的软件产业。这告诉你一些事情：你构建的软件，基本上要维护到你生命结束。我们从未放弃过任何软件。C 语言被使用的原因是我告诉大家，我们会一直维护它。

In fact, we build full data centers. The way that we build everything: if you're developing software, you need the computer in its full manifestation. We don't build PowerPoint slides and ship the chips. We build a whole data center. Until we get the whole data center built up, how do you know the software works? Until you get the whole data center built up, how do you know your fabric works and all the efficiencies you expect? How do you know it's going to really work at scale? That's the reason why it's not unusual to see somebody's actual performance be dramatically lower than their peak performance as shown in PowerPoint slides. Computing is not what it used to be. I say that the new unit of computing is the data center. That's what we have to deliver. That's what we build now. We build a whole thing like that. And then for every single thing, with every combination—air cooled, x86, liquid cooled, Grace, Ethernet, InfiniBand, NVLink, no NVLink—we build every single configuration. We have five supercomputers in our company today. Next year we're going to build easily five more. So if you're serious about software, you build your own computers. If you're serious about software, you're going to build your whole computer. And we build it all at Goodale. This is the part that is really interesting: we build it at scale, we build it vertically integrated, we optimize it full stack, and then we disaggregate everything and sell it in parts. That is completely utterly remarkable about what we do. The complexity of that is just insane. The reason for that is we want to be able to graft our infrastructure into GCP, AWS, Azure, OCI. All of their control planes, security planes are all different, and all of the way they think about their cluster sizing all different. But yet we make it possible for them to all accommodate Nvidia architecture so that CUDA could be everywhere. That is really the singular thought: we would like to have a computing platform that developers could use that is largely consistent, modulo 10% here and there because people's infrastructures are slightly optimized differently. But everything they build will run everywhere. This is one of the principles of software that should never be given up, and we protect it quite dearly. It makes it possible for our software engineers to build once, run everywhere. That's because we recognize that the investment in software is the most expensive investment, and it's easy to test. Look at the size of the whole hardware industry and then look at the size of the world's software industries. It's a $1 trillion industry on top of this $1 trillion hardware industry. That tells you something: the software that you build, you basically maintain for as long as you shall live. We've never given up on a piece of software. The reason why C is used is because I told everybody we will maintain this for as long as we shall.

我们是认真的。我们仍然保持着——我前几天刚看到一篇评测——Nvidia Shield，我们的安卓电视，是世界上最好的安卓电视。我们七年前就推出了它，至今它仍然是排名第一的安卓电视。任何喜欢电视的人，我们上周刚刚更新了软件，还有关于它的新故事。GeForce：我们在全球有 3 亿游戏玩家，我们从未让任何一个玩家失望过。我们的架构在所有不同领域都兼容，这使我们能够做到这一点。否则，如果没有这种架构兼容性，我们的软件团队规模将是现在公司的 100 倍。所以我们对此非常认真，这也会给开发者带来好处。

We're serious now. We still maintain—I just saw a review the other day—Nvidia Shield, our Android TV, is the best Android TV in the world. We shipped it seven years ago, and it is still the number one Android TV. Anybody who enjoys TV, we just updated the software this last week, and there's a new story about it. GeForce: we have 300 million gamers around the world, and we've never stranded a single one of them. The fact that our architecture is compatible across all these different areas makes it possible for us to do it. Otherwise, we would have software teams that are 100 times the size of our company today, if not for this architectural compatibility. So we're very serious about that, and it translates to benefits for developers.

最近一个令人印象深刻的证明是，你们为 xAI 搭建集群的速度之快。如果你愿意谈谈这个，因为那在规模和速度上都令人瞩目。

One impressive substantiation of that recently was how quickly you brought up a cluster for xAI. If you want to talk about that, because that was striking in terms of both the scale and the speed with which you did that.

很多功劳要归功于埃隆。首先，决定做这件事，选址，引入冷却和电力，然后决定建造这个 10 万 GPU 的超大集群，这是同类中最大的单体集群。倒推时间，我们几个月前就开始共同规划他打算启动一切的日子。所有组件、所有 OEM、所有系统、所有软件集成，我们与他们的团队合作，所有网络模拟——我们模拟了所有网络配置。我们以数字孪生的形式预演了一切。我们预演了他的整个供应链，所有网络布线。我们甚至搭建了一个小规模版本，作为第一个实例、基准真相、参考零点、系统零点，在一切到位之前。所以当所有东西到达时，一切就绪，所有演练完成，所有模拟完成。然后大规模集成是一个巨大的工程，庞大的团队日夜不停地布线。几周内，集群就上线了。这确实证明了他的意志力，以及他如何思考机械、电气问题，克服非凡的障碍。在那里完成的事情是第一次以这样的速度建成如此大规模的计算机。我们的两个团队——网络团队、计算机团队、软件团队、训练团队、基础设施团队，从电气工程师到软件工程师——通力合作。看着真是了不起的壮举。

A lot of that credit you've got to give to Elon. First of all, to decide to do something, select the site, bring cooling to it, power, and then decide to build this 100,000 GPU supercluster, which is the largest of its kind in one unit. Working backwards, we started planning together the date that he was going to stand everything up. That date was determined quite a few months ago. All the components, all the OEM, all the systems, all the software integration we did with their team, all the network simulation—we simulated all the network configurations. We pre-staged everything as a digital twin. We pre-staged all of his supply chain, all the wiring of the networking. We even set up a small version of it, a first instance, a ground truth, reference zero, system zero, before everything else showed up. So by the time everything showed up, everything was staged, all the practicing was done, all the simulations were done. Then the massive integration was a monument of gargantuan teams of humanity crawling over each other, wiring everything up 24/7. Within a few weeks, the clusters were up. It's really a testament to his willpower and how he's able to think through mechanical things, electrical things, and overcome extraordinary obstacles. What was done there is the first time a computer of that large scale has ever been done at that speed. Our two teams—the networking team, the computer team, the software team, the training team, the infrastructure team, the electrical engineers to the software engineers—all working together. It's quite a feat to watch.

从工程角度来看，有没有哪个挑战感觉最可能成为瓶颈？

Was there a challenge that felt most likely to be blocking from an engineering perspective?

就是必须整合的大量电子设备。可能值得测量一下：那是成吨成吨的设备。这太不寻常了。通常，像这样的超级计算机系统，你要规划好几年。从第一批系统交付到提交所有东西进行严肃工作，如果花了一年时间也不要惊讶。这经常发生。现在我们等不起，所以几年前我们发起了一个名为“数据中心即产品”的倡议。我们不把它作为产品出售，但我们必须像对待产品一样对待它。从规划、搭建、优化、调优到保持运营，目标是它应该像打开你漂亮的新 iPhone 一样：打开它，一切就都正常工作了。当然，这是技术的奇迹，但我们现在有技能做到这一点。所以如果你对数据中心感兴趣，只要给我一个空间、一些电力和冷却，我们就能在 30 天内帮你搭建好。这相当了不起。

Just the tonnage of electronics that had to come together. It's probably worth measuring: it's tons and tons of equipment. It's just abnormal. Usually, a supercomputer system like that, you plan it for a couple of years. From the moment the first systems are delivered to the time you submit everything for serious work, don't be surprised if it's a year. That happens all the time. Now we couldn't afford to do that, so we created an initiative a few years ago called 'Data Center as a product.' We don't sell it as a product, but we have to treat it like a product. Everything about planning for it, standing it up, optimizing it, tuning it, keeping it operational—the goal is that it should be like opening up your beautiful new iPhone: you open it up and everything just works. Of course, it's a miracle of technology making it like that, but we now have the skills to do that. So if you're interested in a data center, just give me a space, some power, some cooling, and we'll help you set it up within 30 days. It's pretty extraordinary.

太疯狂了。如果展望 20 万、50 万、100 万 GPU 的超大集群，你认为最大的障碍是什么？

That's wild. If you look ahead to 200,000, 500,000, a million GPU superclusters, what do you think is the biggest blocker?

资本、能源、单一区域的供应。一切。你刚才提到的规模没有一样是正常的，但没有什么是不可能的。没有物理定律限制我们，但一切都会很困难。当然，这值得吗？你无法相信——要达到我们认为是计算机的东西，它能如此轻松、如此能干地按照我们的要求完成某种通用智能的任务，即使我们可以争论它是否真的是通用智能，仅仅接近它就已经是一个奇迹。我们知道这一点。我认为有五六个努力在尝试达到那里：OpenAI、Anthropic、xAI、谷歌、Meta、微软。登顶这座山的接下来几步至关重要。谁不想第一个登上那座山呢？重新发明智能的奖赏太重要了，不能不尝试。所以我认为没有物理定律的限制，但一切都会很困难。

Capital, energy, supply in one area. Everything. Nothing about the scales you just talked about is normal, but nothing is impossible. No laws of physics limit us, but everything is going to be hard. And of course, is it worth it? You can't believe—to get to something we would recognize as a computer that so easily and so ably does what we ask it to do with general intelligence of some kind, even if we could argue about whether it's really general intelligence, just getting close to it is going to be a miracle. We know that. I think there are five or six endeavors to try to get there: OpenAI, Anthropic, xAI, Google, Meta, Microsoft. The next couple of clicks up that mountain are just so vital. Who doesn't want to be the first on that mountain? The prize for reinventing intelligence altogether is just too consequential not to attempt it. So I think there are no laws of physics; everything is going to be hard.

一年前我们交谈时，你谈到如何让最极端的客户引领你，以及一些科学应用。我认为这在过去一年变得更加主流。现在仍然是科学以及 AI 在科学中的应用最让你兴奋吗？

A year ago, when we spoke together, you talked about how you let your most extreme customers lead you, and about some of the scientific applications. I think that's become much more mainstream over the last year. Is it still science and AI's application in science that most excites you?

我很高兴我们在 Nvidia 有 AI 芯片设计师。我很高兴我们有 AI 软件工程师。今天的 AI 芯片设计师有多高效？非常棒。没有它我们无法构建 Hopper。原因在于它们可以探索比我们大得多的空间。因为它们有无限的时间——它们在超级计算机上运行——而我们使用人类工程师的时间非常有限，无法探索我们应该探索的那么多空间。我们也无法组合式地探索。

I love the fact that we have AI chip designers here at Nvidia. I love that we have AI software engineers. How effective are AI chip designers today? Super good. We couldn't build Hopper without it. The reason is because they can explore a much larger space than we can. Because they have infinite time—they're running on a supercomputer—while we have so little time using human engineers that we don't explore as much of the space as we should. We also can't explore it combinatorially.

我的领域包括你的探索。你知道，芯片如此庞大，它不像是一艘船，而更像是上千艘船。我们必须在某种程度上独立优化每一艘。但你真正想做的是将它们整体优化。跨模块代码设计，在更大的空间中进行优化。显然，我们能够找到隐藏在局部最小值背后的局部最大值。所以，我们显然能找到更好的答案。没有人工智能，你做不到这一点。工程师根本做不到；我们就是没有足够的时间。

My space while including your exploration. And so you know, chips are so large, it's not like it's designed as one ship; it's designed almost like a thousand ships. And we have to optimize each one of them kind of in isolation. You really want to optimize a lot of them together. And cross-module code design and optimize across a much larger space. Obviously, we're going to be able to find local maximums that are hidden behind local minimums somewhere. So clearly, we can find better answers. You can't do that without AI. Engineers just simply can't do it; we just don't have enough time.

自我们上次集体交谈以来，还有一件事发生了变化。我查了一下：当时，英伟达的市值大约是 5000 亿美元。现在超过 3 万亿美元。所以在过去 18 个月里，你增加了 2.5 万亿美元以上的市值，这相当于每月增加超过 10 亿美元，或者两个半 Snowflake，或者一个 Stripe 再加一点，或者随便你怎么想——一两个国家。显然，很多方面保持一致，比如你专注于建造的东西等等。今天早些时候在这里走动，我感受到了那种兴奋，就像 15 年前我在谷歌时那样——你能感受到公司的活力和激动的氛围。在这段时间里，有什么变化吗？或者英伟达的运作方式、你对世界的看法、或者你能下的赌注的规模有什么不同？

One other thing that's changed since we last spoke collectively. And I looked it up: at the time, Nvidia's market cap was about 500 billion. It's now over 3 trillion. So the last 18 months, you've added 2.5 trillion plus of market cap, which effectively is a billion plus a month, or two and a half Snowflakes, or a Stripe plus a little bit, or however you want to think about it—a country or two. Obviously, a lot of things have stayed consistent in terms of focus on what you're building, etc. And walking through here earlier today, I felt the buzz, like when I was at Google 15 years ago—you felt the energy of the company and the vibe of excitement. What has changed during that period, if anything? Or how is it different in terms of how Nvidia functions, or how you think about the world, or the size of bets you can take?

嗯，我们公司不可能像股价那样变化得那么快，这一点要明确。所以在很多方面，我们并没有太大改变。我认为要做的是退一步，问问自己在做什么。我认为这是公司和国家的重大观察、认识和觉醒：到底发生了什么。我认为从我们行业的角度来看，我们之前谈到的：我们重新发明了计算。这已经有 60 年没有发生过了。这就是它有多重要。我们在过去 10 年里将计算的边际成本降低了可能一百万倍，以至于我们说，‘嘿，让计算机去穷尽地编写软件吧。’这是重大的认识。而且在很多方面，我对芯片设计也说过同样的话：我们希望计算机能发现关于我们芯片的一些我们无法自己做到的事情，探索我们的芯片，并以我们无法做到的方式优化它。就像我们希望数字生物学或任何其他科学领域那样。所以我认为人们开始意识到：第一，我们重新发明了计算。但这意味着什么？即使我们突然创造了这个叫做智能的东西。计算发生了什么？嗯，我们从数据中心——数据中心是多租户的文件存储——这些我们正在创建的新数据中心不是数据中心；它们不存储我们的任何文件。它们只是在生产东西：它们生产 token。而这些 token 被重新组合成看起来像智能的东西。不是吗？而且各种不同的智能：它可以是机器人运动的表达，可以是氨基酸序列，可以是化学链，可以是各种有趣的东西。所以我们到底在做什么？我们创造了一种新的工具，一种新的机器，它在很多方面是‘生成式 AI’这个形容词的名词。不是生成式 AI，而是 AI 工厂。它是一个生成 AI 的工厂。而且我们正在以极大的规模这样做。人们开始意识到，也许这是一个新行业。它生成 token，生成数字，但这些数字构成了相当有价值的东西。哪个行业会从中受益？然后你退一步，再问自己：英伟达发生了什么？一方面，我们重新发明了我们所知的计算。所以有价值一万亿美元的基础设施需要现代化。这是其中一层。大的一层是，我们正在建造的这个工具不仅用于我们正在现代化的数据中心，而且你用它来生产某种新商品。这个新商品行业能有多大？很难说，但可能价值数万亿美元。所以我认为，这就是大想法。如果你退一步，我们不再建造计算机；我们建造工厂。每个国家都需要它，每个公司都需要它。给我一个公司或行业的例子，说，‘你知道吗，我们不需要生产智能；我们有很多。’所以这就是大想法。这有点抽象化的工业视角。总有一天，人们会意识到，在很多方面，半导体行业不是关于制造芯片；而是关于为社会制造基础结构。然后突然间，每个人都恍然大悟，‘啊，我明白了。这是件大事。不仅仅是芯片。’

Well, our company can't change as fast as a stock price, let's be clear about that. So in a lot of ways, we haven't changed that much. I think the thing to do is to take a step back and ask ourselves what we are doing. I think that's the big observation, realization, awakening for companies and countries: what's actually happening. I think what we're talking about earlier from our industry perspective: we've reinvented computing. Now it hasn't been reinvented for 60 years. That's how big of a deal it is. We've driven down the marginal cost of computing probably by a million times in the last 10 years, to the point that we say, 'Hey, let's just let the computer go exhaustively write the software.' That's the big realization. And that in a lot of ways, I was kind of saying the same thing about chip design: we would love for the computer to go discover something about our chips that we otherwise couldn't have done ourselves, explore our chips and optimize it in a way that we couldn't do ourselves. In the way that we would love for digital biology or any other field of science. So I think people are starting to realize: one, we reinvented computing. But what does that mean? Even as we all of a sudden created this thing called intelligence. And what happened to computing? Well, we went from data centers—data centers are multi-tenant stores of files. These new data centers we're creating are not data centers; they don't store any of our files. They're just producing something: they're producing tokens. And these tokens are reconstituted into what appears to be intelligence. Isn't that right? And intelligence all different kinds: it could be articulation of robotic motion, it could be sequences of amino acids, it could be chemical chains, it could be all kinds of interesting things. So what are we really doing? We've created a new instrument, a new machinery, that in a lot of ways is the noun of the adjective 'generative AI.' Instead of generative AI, it's an AI factory. It's a factory that generates AI. And we're doing that at extremely large scale. And what people are starting to realize is, maybe this is a new industry. It generates tokens, it generates numbers, but these numbers constitute something that is fairly valuable. And what industry would benefit from it? Then you take a step back and ask yourself again: what's going on in Nvidia? On the one hand, we reinvented computing as we know it. And so there's a trillion dollars worth of infrastructure that needs to be modernized. That's one layer of it. The big layer of it is that this instrument we're building is not just for data centers which we're modernizing, but you're using it for producing some new commodity. And how big can this new commodity industry be? Hard to say, but it's probably worth trillions. So that, I think, is kind of the big idea. If you were to take a step back, we don't build computers anymore; we build factories. And every country is going to need it, every company's going to need it. Give me an example of a company or industry that says, 'You know what, we don't need to produce intelligence; we got plenty of it.' So that's the big idea. And that's kind of an abstracted industrial view. And someday, people realize that in a lot of ways, the semiconductor industry wasn't about building chips; it was about building the foundational fabric for society. And then all of a sudden, everybody goes, 'Ah, I get it. This is a big deal. It's not just about chips.'

你现在如何看待具身？

How do you think about embodiment now?

嗯，我非常兴奋的是，在很多方面，我们接近通用人工智能，但我们也接近通用机器人。token 就是 token。我的意思是，问题在于，你能把它 token 化吗？当然，token 化东西并不容易，你们也知道。但如果你能 token 化东西，将其与大型语言模型和其他模态对齐——如果我能生成一个视频，显示 Jensen 伸手去拿咖啡杯，为什么我不能提示机器人生成 token 来拿起它呢？直觉上，你会认为问题陈述对计算机来说是相当相似的。所以我认为我们离得很近。这非常令人兴奋。现在，两个棕地机器人系统——棕地意味着你不需要改变环境——是自动驾驶汽车，以及数字司机和具身机器人，对吧？在汽车和人形机器人之间，我们真的可以在不改变世界的情况下将机器人带到世界，因为我们为这两样东西建造了世界。Elon 专注于这两种形式的机器人可能不是巧合，因为它们很可能具有最大的潜在规模。所以我认为这很令人兴奋。但它的数字版本同样令人兴奋。我们在谈论数字或 AI 员工。毫无疑问，我们将拥有各种 AI 员工。我们的前景将是一些生物制剂和一些人工智能，我们都以相同的方式提示它们。不是吗？

Well, the thing I'm super excited about is, in a lot of ways, we're close to artificial general intelligence, but we're also close to artificial general robotics. Tokens are tokens. I mean, the question is, can you tokenize it? Of course, tokenizing things is not easy, as you guys know. But if you were able to tokenize things, align it with large language models and other modalities—if I can generate a video that has Jensen reaching out to pick up the coffee cup, why can't I prompt a robot to generate the tokens to pick it up? Intuitively, you would think that the problem statement is rather similar for a computer. And so I think that we're that close. That's incredibly exciting. Now, the two brownfield robotic systems—brownfield meaning that you don't have to change the environment for—is self-driving cars and, with digital chauffeur and embodied robots, right? Between the cars and the human robot, we could literally bring robotics to the world without changing the world, because we built the world for those two things. Probably not a coincidence that Elon is focused on those two forms of robotics, because it is likely to have the largest potential scale. And so I think that's exciting. But the digital version of it is equally exciting. We're talking about digital or AI employees. There's no question we're going to have AI employees of all kinds. And our outlook will be some biologics and some artificial intelligence, and we all prompt them in the same way. Isn't that?

没错，我主要是给我的员工提示，提供上下文，让他们执行任务。他们去招募其他团队成员，然后回来，我们来回沟通。这和各类数字及 AI 员工有什么不同？所以我们会有 AI 营销人员、AI 芯片设计师、AI 供应链人员等等。我希望英伟达有一天在生物意义上更大，但在人工智能方面也大得多。那是我们未来的公司。如果我们一年后再回来和你聊，你认为公司哪个部分会最人工智能化？

That right, mostly I prompt my employees, you know, provide them context, ask them to perform a mission. They go and recruit other team members, they come back, and we go back and forth. How's that going to be any different with digital and AI employees of all kinds? So we're going to have AI marketing people, AI CH designers, AI supply chain people, AI, you know. And I'm hoping that Nvidia is someday biologically bigger, but also from an artificial intelligence perspective much, much bigger. That's our future company. If we came back and talked to you a year from now, what part of the company do you think would be most artificially intelligent?

我希望是芯片设计。没错，最重要的部分。因为应该从最能推动进展、最能产生最大影响的地方开始。这是一个极其困难的问题。我和 Synopsys、Cadence 合作。我完全想象他们拥有可以租用的 Synopsys 芯片设计师，他们了解某个特定模块和工具，并且训练了 AI 使其非常擅长。我们可以在需要时雇佣一大批。在芯片设计的那个阶段，我可能会租用一百万个 Synopsys 工程师来帮忙，然后再租一百万个 Cadence 工程师。对他们来说，这是一个多么激动人心的未来：所有这些智能体都位于他们的工具平台之上，使用工具平台并与其他平台协作。SAP 和 ServiceNow 也会这样。有人说这些 SaaS 平台会被颠覆，我实际上认为相反：它们坐拥金矿。将会有大量专门针对 Salesforce、SAP 的智能体涌现，每个平台都有自己的语言。我们有 CUDA 和 Omniverse 的 OpenUSD。谁会创建擅长 OpenUSD 的 AI 智能体？我们，因为没有人比我们更关心它。所以我认为，这些平台将在很多方面因智能体而繁荣，我们将它们相互介绍，它们会协作解决问题。

I'm hoping it's chip design. Okay, most important part. And that's right because I should start where it moves the needle most, also where we can make the biggest impact. It's such an insanely hard problem. I work with Synopsys and Cadence. I totally imagine them having Synopsys chip designers that I can rent, and they know something about a particular module, their tool, and they trained an AI to be incredibly good at it. We'll just hire a whole bunch of them whenever we need. In that phase of chip design, I might rent a million Synopsys engineers to come and help me out, and then go rent a million Cadence engineers to help me out. What an exciting future for them that they have all these agents that sit on top of their tools platform, use the tools platform, and collaborate with other platforms. And you'll do that for SAP and ServiceNow. People say these SaaS platforms are going to be disrupted. I actually think the opposite: they're sitting on a gold mine. There's going to be a flourishing of agents specialized in Salesforce, SAP, and everybody's got their own language. We have CUDA and OpenUSD for Omniverse. Who's going to create an AI agent that's awesome at OpenUSD? We are, because nobody cares about it more than we do. So I think in a lot of ways, these platforms are going to flourish with agents, and we're going to introduce them to each other, and they're going to collaborate and solve problems.

你看到很多不同的人在 AI 的各个领域工作。你认为哪些方面被忽视了，或者你希望更多创业者、工程师或商业人士可以投入其中？

You see a wealth of different people working in every domain in AI. What do you think is under-noticed or that you want more entrepreneurs or engineers or business people could work on?

首先，我认为被误解和低估的是水面下的活动，即受 AI 和机器学习影响的突破性科学、计算机科学、科学与工程。你走进任何一个科学系或理论数学系，AI 和机器学习以及我们今天讨论的工作都将改变明天。如果你把世界上所有的工程师和科学家都算上，并说他们今天的工作方式是未来的早期迹象，那么你将看到 AI 的浪潮、机器学习的浪潮在短时间内改变我们所做的一切。我看到了计算机视觉的早期迹象，比如 AlexNet、Ilya、Hinton 在多伦多、Yann LeCun 以及斯坦福的 Andrew Ng。我们很幸运地从检测猫的外推中看到了计算机科学和计算的深刻变化。那次外推对我们来说是幸运的。我们深受启发，改变了我们做事的一切方式。但从观察那个玩具般的 AlexNet 到物体识别达到超人类水平，这花了几年时间。现在正在发生的事情是，所有科学领域都在涌现——没有一个领域被落下——从量子计算到量子化学，每个科学领域都涉及我们讨论的方法。如果我们给自己几年的时间，世界将会改变。不会有任何一篇论文、任何一个科学或工程突破不以生成式 AI 为基础。我现在相当确信这一点。所以我认为有很多关于这是否是一时热潮的疑问。你只需要回到第一性原理，观察实际发生的事情。计算栈、我们做计算的方式已经改变。你编写软件的方式已经改变。这是非常核心的。软件是人类编码知识的方式。我们现在用一种非常不同的方式编码。这将影响一切。其他一切都不会再一样。我想我是在对已经认同的人说话。我们都看到了同样的事情。所有与你合作的初创公司，与我合作的科学家和工程师——没有什么是会被落下的。我们将带着所有人一起前进。从计算机科学世界看向所有其他科学领域，最令人兴奋的事情之一是，我可以去机器人会议、材料科学会议、生物技术会议，我理解它——不是在科学的每个层面，而是在推动发现的过程中，都是通用的算法，并且有一些统一的通用概念。

Well, first of all, I think what is misunderstood and underestimated is the underwater activity, the under-the-surface activity of groundbreaking science, computer science, science and engineering that is being affected by AI and machine learning. You just can't walk into a science department anywhere, theoretical math department anywhere, where AI and machine learning and the type of work we're talking about today is going to transform tomorrow. If you take all the engineers in the world and all the scientists in the world, and you say the way they're working today is an early indication of the future, then you're going to see a tidal wave of AI, a tidal wave of machine learning change everything we do in a short period of time. I saw the early indications of computer vision with AlexNet, Ilya, Hinton in Toronto, Yann LeCun, and Andrew Ng here at Stanford. We were fortunate to have extrapolated from detecting cats into a profound change in computer science and computing. That extrapolation was fortunate for us. We were so inspired by it that we changed everything about how we did things. But that took years from observing that toy AlexNet to superhuman levels of capabilities in object recognition. What is happening right now, the groundswell in all fields of science—not one field left behind—from quantum computing to quantum chemistry, every field of science is involved in the approaches we're talking about. If we give ourselves a couple of years, the world's going to change. There's not going to be one paper, one breakthrough in science or engineering where generative AI isn't at the foundation of it. I'm fairly certain of that now. So I think there are a lot of questions about whether this is a fad. You just have to go back to first principles and observe what is actually happening. The computing stack, the way we do computing, has changed. The way you write software has changed. That is pretty core. Software is how humans encode knowledge. We encode it in a very different way now. That's going to affect everything. Nothing else will ever be the same. I think I'm talking to the converted here. We all see the same thing. All the startups you work with, the scientists and engineers I work with—nothing will be left behind. We're going to take everybody with us. One of the most exciting things coming from the computer science world and looking at all these other fields of science is that I can go to a robotics conference, a materials science conference, a biotech conference, and I understand it—not at every level of the science, but in the driving of discovery, it is all the algorithms that are general, and there are some universal unifying concepts.

是的，当你看到它在每个领域都如此有效时，这真是令人难以置信地兴奋。

Yeah, and I think that's incredibly exciting when you see how effective it is in every domain.

绝对。而且我很兴奋自己每天都在使用它。我不知道你们怎么样，但它现在是我的导师。我学任何东西之前都会先问 AI。为什么要走弯路？直接去找 AI，直接去 ChatGPT，或者有时我用 Perplexity，取决于我问题的表述，然后我就从那里开始学习。然后你随时可以分支深入。但天哪，这真是……

Absolutely. And I'm so excited that I'm using it myself every day. I don't know about you guys, but it's my tutor now. I don't learn anything without first going to AI. Why learn the hard way? Just go directly to AI, go directly to ChatGPT, or sometimes I do Perplexity depending on the formulation of my questions, and I just start learning from there. Then you can always fork off and go deeper if you like. But holy cow, it's just...

太不可思议了。几乎我所知道的一切，我都会去复核，即使我知道那是事实，是我认为的真相，我是专家，我还是会去问 AI，再确认一遍。

Incredible. And almost everything I know, I double-check, even though I know it to be a fact, you know, what I consider to be ground truth, I'm the expert, I'll still go to AI and check, make double-check.

是的，这太棒了。我做的几乎所有事情，都会用到它。

Yeah, it's so great. Almost everything I do, I involve it.

嗯，我觉得这是个很好的结束点。非常感谢你今天抽出时间。

Yeah, I think it's a great note to stop on. Thanks so much for your time today.

是的，非常愉快。很高兴见到你们。谢谢，Jensen。

Yeah, really enjoyed it. Nice to see you guys. Thanks, Jensen.

在 Twitter 上找到我们：@no_prior_pod。如果你想看我们的脸，请订阅我们的 YouTube 频道。在 Apple Podcasts、Spotify 或你收听的地方关注节目，这样你每周都能收到新剧集。并在 no-pri.com 注册邮件或查找每集的文字稿。

Find us on Twitter at @no_prior_pod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-pri.com.