与 Jeff Dean 炉边对话：TPU 的演进与 AI 基础设施

欢迎大家。非常感谢你们的热情参与。这是我们的第七届 Dev Labs，每一次举办都规模更大、效果更好。而这一次是迄今为止最棒的一届。

So, welcome everybody. Thank you so much for joining us for all your enthusiasm. This is our seventh Dev Labs and every single time we do it, it gets bigger and better. And this is by far the best one yet.

我们收到了非常多的关注，也感谢你们的反馈。嗯，接下来的安排是这样的：我们会讨论几个问题，比如 TPU 的演进。这位是 Jeff，他完全不需要介绍。他是 Google 的第 30 号员工。那我们就从那段故事开始吧。我们还有 Bill 在这里，他也会提问，但我们也希望他能给我们透露一些关于 AI 基础设施执行方面的细节，以及愿景和未来几年如何落地，如果可以的话。好，我们开始吧。

And we got so much interest and thank you for your feedback. Um, so the way that this is going to work is we're going to talk about a couple of questions like talk about the evolution um of TPUs. This is Jeff doesn't need any introduction at all. He was employee number 30 uh at at Google. So let's talk start with that um that story and we've got Bill here to who's going to also be asking questions but we also want him to be able to give us a little bit of a tidbits in terms of execution of how AI infrastructure the the vision and how it's getting executed um in the next now and also in the next couple of years if that's okay. All right let's do it.

嗯，Jeff，请多讲讲。你是 Google 的第 30 号员工。

Um so Jeff tell us a little bit more. You are employee number 30 at Google.

是的。是的。

Yep. Yep.

1999 年。这里大多数人还没出生，或者可能已经出生了，或者还在

1999. Most of the folks here weren't born yet or they might have been or they were in

还在穿尿布。

they were in diapers.

那是什么感觉？

What was that like?

嗯，不，那非常有趣。我的意思是，当我加入时，我们都在帕洛阿尔托市中心一间很小的办公室里，夹在现在一家 T-Mobile 商店的楼上，大概就是那种规模的办公室。你知道，我们当时在努力打造一个非常高质量的搜索产品。我之前的工作中接触过信息检索，特别是利用网络的图结构来增强页面上已有的信息等等。所以 Google 似乎是一个自然的选择，可以在一个不那么研究导向的环境中尝试，更像是人们直接使用我的工作。那超级有趣。能量非常高，我们的流量每周增长约 7%，如果你算 1.07 的 52 次方，你知道每年就像在融化一样，还要努力避免在周二中午崩溃。嗯，所以那非常有趣。嗯，随着公司不断发展，我们做了越来越多的事情，这总是令人兴奋。

Uh, no, it was a lot of fun. I mean, I think we uh when I joined, we were all in this tiny little office in downtown Palto, wedged above what's now a T-Mobile store, sort of roughly that kind of scale of of office. And, you know, we were trying to build a a really high quality search product. And I'd gotten some exposure to that in a prior job of information retrieval particularly on the web using the graph structure of the web to kind of uh augment the existing information on pages and so on. And so uh Google seemed like a natural place to go try that in a less researchy environment, more of a like people using my work directly. Uh which was super fun. uh it was super high energy and you know we were like growing our traffic like 7% every week or something and if you do 1.07 to the 52nd you know you're like melting every year and trying to avoid falling over on Tuesdays at noon. Um so that was that was a lot of fun. Um and you know as the company has continued to grow we've done more and more things which is you know always exciting to see.

然后 2001 年发生了什么，如果快进的话

And then what happened in 2001 if fast forward over

为了避免融化，我们做的一件事就是不断重写整个搜索、索引和查询服务系统，使其更高效，使用不同的数据结构和更紧凑的表示。2001 年我们意识到，我们扩展索引大小和服务容量的方式是制作越来越多的索引分区。我们称之为分片，以获得更大的索引。然后每个分片你会制作越来越多的副本，以提供同时服务更多查询的能力。嗯，我们最终达到了这样的程度：不是 7 个分片每个 10 个副本，而是每个数据中心有 60 个分片每个 20 个副本。我们最终算了一下，意识到我们可以把整个网络索引放在内存中，而不是放在磁盘上，放在 1200 台机器上，这些机器原本是 60 个分区的 20 个副本。那真的很有趣。我们在三天内将整个搜索系统加速了约 4 倍，这很有趣。

so to avoid melting one of the things we were doing was constantly rewriting the whole search and indexing and query serving system to be more efficient and use different data structures and you know more compact representations of things. And we got to a point in 2001 where we realized that that the way we were sort of scaling both the index size and the amount of capacity we had for serving the index was we would make more and more partitions of the index. We called them shards to have a bigger index. And then every one of those shards you would make more and more copies of it to provide capacity for serving more queries uh simultaneously. Um, and we eventually got to the point where we had, you know, instead of seven shards and 10 copies each, we had like 60 shards and 20 copies of each in every data center. And we eventually did the math and realized that we could put instead of having the the index on disk, we could put the entire index of the web in memory on the 1200 machines that would otherwise be 20 copies of the 60 partitions. And that was that was really fun. We we sort of sped up all of our search system by like 4x in uh three days which was pretty fun.

是的。

Yeah.

然后 2013 年你做了一个粗略的计算。

And then in 2013 you did a back of the envelope just some calculations.

是的。是的。所以我开始研究深度学习训练基础设施和深度训练深度学习模型，因为它似乎是正确的抽象。你知道，我实际上在 1990 年作为本科生接触过神经网络，我做了一个关于神经网络并行训练的本科论文，因为我只是觉得，哦，这是正确的事情。我们只需要 32 个弱小的处理器而不是一个，然后我们就能训练出惊人的模型。嗯，结果我们需要大约一百万倍的算力，而不是 32 倍。呃，但最终通过摩尔定律的改进等等，我们开始拥有那种算力，大概从 2008 或 2009 年开始。然后嗯在 2012 年呃我们开始了一个项目，真正尝试扩展并进行语音、视觉和语言模型的分布式训练

Yeah. Yeah. So I I started to work on kind of deep learning uh training infrastructure and deep training deep learning models because it seemed like the right abstraction. You know, I'd actually gotten exposed to neural nets as a undergrad in 1990, and I did a an undergrad thesis on parallel training of neural nets because I just felt like, oh, this is the right thing. We just need to get 32 wimpy processors instead of one, and then we'll be able to train amazing models. Um, turns out we needed like a million times more compute power, not 32. Uh, but eventually through Moore's law improvements and so on, we started to have that uh starting maybe 2008 or nine. And then um in 20121 uh we started a project to uh really try to scale up and do distributed training of models for speech and for vision and for language

对

right

当时我们的数据中心里有 CPU，所以有很多 CPU，我们用了大约 16,000 个 CPU 核心来训练一个大型视觉模型，我们得到了这些惊人的改进，我们也为语音模型这样做。模型的质量提升惊人。我们在大约两个月内尝试构建深度声学语音模型，在词错误率方面取得了相当于过去 20 年语音研究的改进。嗯，所以那就像是，天哪，这将会非常好。但我们如何服务它？当时我做了一些计算，我说好吧，假设 1 亿人每天对着手机说话几分钟。那么我们如何服务？结果发现，仅仅为了在 Google 一个不起眼的小部分推出更好的语音识别模型，我们就需要将 Google 的计算机数量翻倍。那似乎有点过分。更不用说不可行了。所以我们决定专用硬件是出路。这就是 TPU 系列的起源，我们想构建专用加速器，基本上非常擅长低精度线性代数，仅此而已。然后这样你就可以更高效地服务系统，比如语音模型和视觉模型等等。结果发现，在 2015 年我们有了芯片时，它的每瓦性能比当时的 CPU 和 GPU 好 30 到 80 倍。

on uh at the time we had CPUs in our data centers so lots of CPUs um so we used about 16,000 CPU cores to train a a large vision model and we got these amazing improvements and we were doing that for speech models as And uh the models were like amazing improvements in quality. So we got like the equivalent of the last 20 years of speech research uh in terms of improvements in word error rate in like the two months that we tried to build a you know a deep acoustic speech model. Um and so that was like oh my goodness this is going to be so good. Uh but how do we serve it? And so uh and at the time I did some calculations where I said okay suppose a 100 million people start talking to their phone for a few minutes a day. Then how will we serve that? And turns out we would have needed to double the number of computers Google had just to roll out the better speech recognition model uh in one you know tiny obscure part of Google. And that that seemed a little excessive. Uh so uh not to mention impractical. Um so we decided that specialized hardware uh was the way to go. And so we that's sort of the genesis of the TPU family was we wanted to build you know specialized accelerators that basically were really really good at low precision linear algebra and nothing else. And then that's how you can serve the system you know the speech models and vision models and so on uh much more efficiently. So it turned out it was like 30 to 80 times better uh performance per watt than uh contemporary CPUs and GPUs of the day uh in 2015 when we had the the chip available

那就是第一次迭代。

and that was that was the first iteration.

是的。TPUv1 主要针对推理

Yeah. UV1 which was really targeting inference

对

right

呃，然后后续几代主要针对训练和推理。最近我们甚至开始再次将这两条线分开，因为训练和推理有些不同。

uh and then subsequent generations have really mostly targeted training and inference as well. And then more recently we've even started to split those lines apart again because training and inference are somewhat a bit different.

当你在 2017 年发表论文时，它被接受了。它是 50 年历史中被引用最广泛的论文。

And when you published the paper in 2017 it was accepted. It's the most widely cited paper in the 50-year history.

是的。那篇关于 TPUv1 的论文有很多作者，大概有 35 位左右。但考虑到它在 50 年历史中相对较新，这很不错。

Yeah. Of is that is that yeah the paper about TPUv1 is a you know many authored things. There's probably like 35 authors or something. But uh you know that's uh that's nice considering it's relatively recent in the 50 year

架构史。这真的很了不起，Google 被认为是一流的软件公司，但后来开始推动硬件路线图。

history of architecture. That's really just amazing that Google was considered as the the top-notch software company but then start to push on the hardware hardware road map.

这真的很了不起。做硬件是一回事，对吧？因为我们把 TPU 做得非常棒——有很多板级设计、芯片组设计、TPU 本身的设计——但我们也需要让编译器和框架运行得非常好。早期你实际上推动了 XLA，这是我们的 TPU 编译器，也推动了 TensorFlow。你能不能介绍一下早期推动 TPU 软件栈的历程是什么样的。

That's really just amazing. And doing hardware is one thing, right? Because we make the TPU fantastic—there's a lot more board design, chipset design, TPU itself design—but also we need to make the compiler and framework work really well. Early on you actually pushed on XLA, which is our TPU compiler, and also pushed on TensorFlow. If you can introduce what the early journey looks like by pushing on this software stack for TPU.

是的。是的。我的意思是，作为机器学习开发者或研究者，你想要的就是能够表达你的高层次想法，然后让它神奇地在大规模系统上运行，而不必过多考虑为了获得良好性能可能需要在底层做的所有性能相关的事情。所以这意味着要有一个好的编译器、非常好的互连、能够表达抽象概念的框架，比如我是在和本地机器上的四块芯片打交道，还是在和大型分布式设置中的一万块芯片打交道。你希望不必过多考虑这些。针对这两种设置做不同的事情。

Yeah. Yeah. I mean I think what you want as a machine learning developer or researcher is you just want to be able to express your high-level idea and then have it sort of magically run on a large-scale system without you having to think too much about all the performance things you might need to do under the covers to really get your good performance. And so that means having a good compiler, having really good interconnects, having frameworks that express the abstraction of like am I talking to four chips on my local machine or am I talking to 10,000 chips in a large distributed setup. You'd like to not have to think about that too much. Do different things for those two settings.

那么谈谈不同 TPU 的迭代，以及你认为这一切将走向何方？

So talk about the iterations of different TPUs and where do you see it all going?

是的。是的。我的意思是，在 TPUv1 之后，它真的只是一块插在插槽里的 PCIe 卡，然后我们实际上买了一堆这样的卡，却不知道我们会怎么用它们。我当时堵住了我们的 CFO，说我们需要买很多这样的卡，因为我们会用到它们。我们不确定用来做什么，但能不能请让我们多买一些？所以我们买了。然后 TPUv2 是我们第一个以训练为设计目标的系统，对吧？不仅仅是单个芯片，而是一个类似超级计算机的系统，由许多芯片通过二维环面互连。在几代产品中，我们从 TPUv3 开始引入了液冷，所以当你的系统中有管道通向芯片表面时总是很刺激——泄漏可不是好事。然后 TPUv4 随着 pod 规模的扩大——TPU 2 和 3 在 256 个芯片和 1,024 个芯片之间有固定布线——然后当你开始构建越来越大的系统时，你就会开始遇到单个芯片、板卡或系统的故障，你需要更具容错性，因为即使某些东西坏了，你仍然希望能够使用更大 pod 的整个拓扑。所以 TPUv4 引入了机架之间光学可重构网络的概念。这样你就可以像乐高积木一样从数据中心各处组装，对吧？用有趣的小开关，带有微调反射镜，让这八个机架看起来像是紧挨着的，尽管它们分散在数据中心的许多过道中。然后这非常有用。TPU v5、6 和 7 都引入了越来越低的精度计算格式，这非常有用。你可以从中获得更高的性能。尽管当一些 flops 是 16 位、一些是 8 位、一些是 4 位时，比较 flops 总是很奇怪。如果你能在机器学习算法中利用它们，低精度的那些是获得更高性能的绝佳方式。

Yeah. Yeah. So I mean then after TPUv1, which is really just a PCIe card that fit in a slot, and then we actually bought a whole bunch of those without knowing how we would use them. I cornered our CFO at the time and said we need to buy a lot of these because we're going to use them. We're not sure what for, but can we please buy a lot? So we did. And then TPUv2 was kind of the first system that we designed with training in mind, right? With not just a single chip but a whole sort of supercomputer-like system of many chips interconnected with a 2D torus. Over the generations we've introduced liquid cooling starting in TPUv3, so it's always exciting when you have pipes going to the surface of your chips in your system—leaks are not a good thing. And then TPUv4 as you scaled up the pod size—TPU 2 and 3 had fixed wiring between 256 chips and then 1,024 chips—and then as you start to get larger and larger systems, then you start to have failures in individual chips or boards or systems and you need to be more fault tolerant because you still want to be able to use the whole topology of a much larger pod even if something's broken. And so TPUv4 introduced this notion of an optical reconfigurable network between racks of machines. And so you can sort of assemble like Lego bricks from all over the data center floor, right? With funny little switches that have little micro-adjustable mirrors to make it seem as though these eight racks are right next to each other even though they're scattered across many aisles in the data center. And then that's been super useful. TPU v5 and 6 and 7 all introduce lower and lower precision computational formats, which has been super useful. You get much higher performance from that. Although it's always weird to compare flops when some flops are 16-bit and some are 8-bit and some are 4-bit. If you can make use of them in your machine learning algorithms, the lower precision ones are a really great way to get even more performance.

那么第一块 TPU 是 2015 年，2017 年论文发表。现在它已经是传奇了。然后 2018 年的 Pathways——让我们稍微谈谈 Pathways 以及为什么它如此重要。

And then so the first TPU was 2015, 2017 was when the paper came out. It's just legendary now. And then 2018 Pathways—let's talk about Pathways a little bit and why that's so important.

是的。所以我们开始构建 Pathways 真的是为了处理更加不规则和更加稀疏的模型的训练，并且作为底层系统基础设施，使我们能够有一个单一的编程模型、一个单一的进程来驱动大型分布式训练系统。所以作为程序员，如果你的单个 Python 进程看起来就像连接了 8,000 个设备，那就好多了，你会说，太好了，我只想在所有 8,000 个设备上做一个全归约，它就会做到，而不需要你做任何特殊的事情。底层软件中，XLA 编译器做了一些工作，然后在稍高的层次上，Pathways 编排许多不同芯片之间的数据移动。所以你在同一个 TPU pod 内有芯片，如果你需要在那里通信，它会使用 TPU pod 中的 ICI 高速链路。但如果你需要跨多个 pod 或者需要从这个 pod 上的芯片与那个 pod 上的芯片通信，那么 Pathways 将使用数据中心网络甚至广域网来编排该传输，如果你把俄克拉荷马州、德克萨斯州和爱荷华州等地的 pod 拼凑成一个训练任务的话。然后这给你一个很好的抽象：我有一大堆算力，我将让系统找出最佳使用方式。

Yeah. So we started to build Pathways really to handle training of much more irregular and much sparser models and also as the underlying system infrastructure for it to enable us to have a single programming model, a single sort of process that would be driving a large distributed training system. So it's much nicer as a programmer if your single Python process just looks like it has 8,000 devices attached to it and you're like, great, I just want to do an all-reduce across all 8,000 devices and it will do that without you having to do anything special. And the underlying software in both the XLA compiler does some of that and then at the slightly higher level Pathways orchestrates data movement between lots of different chips. So you have chips within the same TPU pod and if you need to communicate there it'll use the ICI high-speed links that are in the TPU pod. But if you then need to have something that spans multiple pods or you need to talk from this chip on this pod to this chip on this pod, then Pathways will orchestrate that transfer using the data center network or even a wide area network if you're cobbling together a training job of pods in Oklahoma and Texas and Iowa or something. And then that kind of just gives you this nice abstraction of I just have a whole bunch of compute and I'm going to allow the system to figure out how best to use it.

是的，太棒了。那么现在，当然，Jeff，你很早就大量参与了 TPU 的网络设计、TPU 本身的设计，然后是编译器和框架，当然还有 Pathways，它实际上编排了整个数据移动、网络流量、重新配置等等——所以这就是整个 AI 基础设施栈。现在我认为你非常深入地参与了 Gemini 模型的设计，当前的 Gemini 模型以及未来的 Gemini 模型。

Yeah, it's amazing. So now, of course, Jeff, very early on you were involved a lot in networking design for TPU, TPU itself design, then the compiler and framework, of course, then Pathways, which actually orchestrates entire data movement, the networking traffic, reconfiguration, everything—so that's all the AI infrastructure stack. Now I think you're very deeply involved in designing the Gemini models, the current Gemini models and future Gemini models as well.

是的。

Yeah.

那么 Gemini 模型真的可以与整个基础设施栈协同设计。所以从你的角度来看，我们已经获得了哪些好处，以及你有哪些未来的资源，以便我们可以将模型推向最前沿的边界？

Then the Gemini models can really be co-designed with the entire infrastructure stack. So from your point of view, what kind of benefits have we already collected and what kind of future sources do you have so that we can push the models to the cutting-edge boundary?

是的，我的意思是，我认为整个 TPU 系列确实受益于我们自己就是 ML 算力的大用户，我们有很多人试图推动新研究想法的边界，这些想法可能会以现有工作负载没有的方式对硬件造成压力。所以通过将所有东西都放在 Alphabet 这个屋檐下，我想，我们实际上可以在未来的 TPU 代际设计师、编译器和基础设施的软件设计师，以及机器学习研究者之间进行大量互动，研究者可以说嘿，你知道，这种东西似乎在小规模上有效。

Yeah, I mean I think the whole set of TPU generations has really benefited from the fact that we are big users of the ML compute ourselves and we have lots of people trying to push the boundaries of new research ideas that might stress the hardware in ways that existing workloads don't. And so by having everything under one sort of Alphabet roof, I guess, we actually can have a lot of interaction between the future TPU generation designers, the software designers of the compilers and the infrastructure, and then the machine learning researchers who can say hey, you know, this kind of thing seems to be working at small scale.

我们认为，这对未来一两年的大规模训练运行非常重要，我们要确保我们的硬件能做到这一点。因为作为计算机架构师，如果你闭门造车，只是去猜快速发展的 AI 领域在未来两到六年会走向哪里，而你眼下设计的这款芯片必须在那时仍然有用，这超级难。你能获得的洞察越多越好。

We think that's going to be really important for large-scale future training runs in a year or two, and we want to make sure our hardware does that. Because as a computer architect, if you're doing this in isolation, just trying to guess where the fast-moving AI field is going in the two-to-six-year time frame that your particular chip you're designing now has to be relevant in, it's super hard. And the more insight you can get, the better.

对，我们认为这东西能行。而且迭代越多——不只是我们觉得需要这么做。经常是硬件那边的人说，这很难，但我们可以这么做，那会有用吗？然后就有这种来回的讨论：对，那实际上能让我们做到非常接近这个的事，甚至可能更好。

Yeah, we think this thing will work. And the more actual iteration — it's not just like we think we need to do this. It's often like the hardware people say, well, that's hard, but we could do this. Would that be useful? And you have this kind of back and forth of like, yes, that would actually enable us to do something very close to this, or maybe even better.

对。这对做出有效的芯片设计超级重要。

Yeah. And that's super important for making effective chip design.

昨天我在台上做了开场演讲。我讲到，随着我们开始逐步扩大 Gemini 模型的规模，对吧？我记得每个月我们都坐在同样的会议上讨论可靠性。我在基础设施这边，Jeff 代表 GDM 的需求方。我们说，嘿，你知道吗，Gemini 模型越来越大，然后我们用大量 TPU 来训练模型。最终我们用 10 万个 TPU 来训练模型。我们一起看的就是一个指标，叫有效算力利用率（goodput）。

Yesterday I was on stage and gave the opening. I was talking about, as we start to ramp up and scale up Gemini models, right? I remember every single month we sit in the same meetings to talk about reliability. So I'm on the infra side, and Jeff represents GDM demand. We say, hey, you know what, the Gemini model becomes bigger and bigger, then we use tons of TPUs to train the models. Eventually we use 100,000 TPUs to train the models. That's the one metric we're looking at together, which is called goodput.

没错。

That's right.

有效算力利用率。一开始我们的有效利用率并不好。我记得我们一起说——

Goodput. Then initially our goodput is not good. I remember what we said together —

无效利用率压过了有效利用率。所以一开始大概 60%。随着我们用越来越多的 TPU 来预训练那个模型，事情变得非常复杂。我们需要更大的规模，需要更多数据，需要更多并行。有效利用率一开始是 50%、60%、70%、80%，但就是不够好，对吧？

badput dominates goodput. So initially like 60%. As we scale up more and more TPUs to pre-train that model, it becomes very complicated. We need bigger scale. We need more data. We need more parallelism. The goodput was like in the beginning 50%, 60%, 70%, 80%, but it's just not good enough, right?

但长话短说，现在我们用规模大得多的训练基础设施来训练复杂得多的模型，有效利用率可以高达 95%，甚至 98%。

But the long story short, now we use much more massive-scale training infrastructure to train much more complicated models, and it can be as high as 95%, even 98%.

对。我认为这实际上是很多因素共同作用的结果，比如更好的运维实践、芯片部署时更好的质量检测，以及软件层面能更好地处理故障，即使系统某部分宕机也能继续向前推进。所有这些加在一起非常重要，带来了很大差别。

Yeah. And I think that's actually a combination of a whole bunch of things, like better operational practices, better QA testing of the chips when they're deployed, software things that can deal better with failures and continue to make forward progress even if some part of the system is down. All these things together matter a lot and make a big difference.

对，我觉得这也是另一个好处。如果我们拥有模型、拥有数据，也拥有整个硬件、网络和所有软件栈，那实际上就给了——

Yeah, I think that's also another benefit. If we own the model, we own the data, we also own the entire hardware, networking and all the software stack, that actually gives —

还有运维团队，就像——

and the operations team like —

你可以说，嘿，这超级重要。因为如果你只看单独一个 TPU 托盘，不把它当成更大系统的一部分，你会觉得，行吧，我们一周后顺路过来修就行。但如果它是一个正在被使用的 pod 的一部分，里面有很多坏掉的托盘，那你就想立刻去修，因为需要修复的东西的影响范围其实比看起来大得多。

You can say, hey, it's super important, because if you think about an individual tray of TPUs, if you don't think that's part of a larger system, you're like, okay fine, we can repair that in a week when we sweep by here. But if it's part of a pod that is being used and there are many broken trays in it, then you want to go fix them like immediately, because the blast radius of the thing you need to repair is actually much bigger than it might seem.

所以当你说到 10 万块芯片时，可靠性真的不再是一个软件补丁的问题了。你觉得它更多是系统设计问题吗？还是更多与物理有关？

So when you're talking about 100,000 chips, reliability really isn't a software patch problem anymore. Would you say it's more of a system design? Is it more related to physics?

我是说，我认为可靠性确实是整个系统的一种属性，有很多不同的东西需要你做得健壮。比如你可以做的一件事，是用不可靠的部件搭出可靠的系统。对。这其实从 Google 最早的时候就开始了。我们买便宜的消费级 PC 来服务搜索流量，然后在上面构建健壮的系统软件，对吧？这样我们就能处理单台机器的故障，同时仍然提供那个系统本该提供的功能。也许它是一个分布式文件系统，所以你把数据复制到多台机器上，即使某份数据的某些副本挂了，你总有一些是可用的。我认为在大规模 AI 训练系统里也可以做同样的事。如果你有一个 20 个 pod 的默认配置，其中一个挂了，你可以在修复第 20 个的同时用 19 个 pod 继续推进。

I mean, I think reliability is really a property of a whole system, and there are many different things you want to make robust. Like one thing you can do is make robust wholes out of unreliable parts. Right. This is actually from the very beginning of Google. We bought like cheap consumer PCs to serve our search traffic, and we made robust system software systems on top of that, right? That would enable us to handle individual machine failures and still give the functionality that whatever system was on it was supposed to be providing. Maybe it's a distributed file system, so you replicate data across multiple machines and you always have some that are available even if some of the replicas of a piece of data are down. And I think you can do the same thing in large-scale AI training systems. If you have a 20-pod default configuration, but one of them is down, you can make progress with 19 of the pods while you're repairing the 20th.

Bill，你昨天在主题演讲里谈到 OCS 和 Virgo 时也稍微提到了这一点。你要不要聊一聊，以及它如何影响可靠性？

And Bill, you kind of touched on this a little bit yesterday with your keynote about OCS and Virgo. Do you want to talk a little bit about that and how that plays into reliability?

对，我认为这是一个漫长的过程。我想第一件事是，我们看每一次中断——每一万块芯片、每一个部件——每天有多少次中断，对吧？我们必须把它降到最低，因为随着训练集群规模扩大，如果中断太多，那就不太好了，对吧？所以我们做的一件事就是，好，我们怎么提高硬件可靠性？对。

Yeah, I think there's a long journey. I think the first thing is we look at every interrupt — every single 10,000 chips, every single part — how many interrupts per day we have, right? We have to minimize that, because if there are too many interrupts as we scale up the training cluster, there are way too many interrupts, that's not very good, right? So one thing we did is, okay, how can we improve the hardware reliability? Right.

然后我们不仅做这个，而且——昨天我提到，甚至在预训练开始之前，我们实际上会看所有——我们实际上做了一些软件。软件会扫描整个集群，对吧，每一个组件，你的生命体征如何。如果生命体征有点问题，我们实际上要么修好它，要么在训练开始之前就把它排除掉。这是第一件事，对吧？

Then not only do we do that, but also — yesterday I was mentioning that even before pre-training starts, we actually look at all — we actually did some software. So software actually scans the entire fleet, right, every single component, what's your vital sign. If the vital sign is actually somewhat problematic, we actually either fix it or exclude it before we even start training. That's the first thing, right?

第二件事是，训练开始了，但训练开始后，即使硬件显示出非常非常强的生命体征，对吧，训练开始了——我是说训练持续好几周，仍然可能出问题，对吧？仍然可能出问题。我们怎么找出哪些具体的组件和服务器有问题，然后修好它，再把它排除、替换、修好？这是第二件事。

The second thing is, training gets started, but when training gets started, even if the hardware shows very, very strong vital signs, right, the training starts — I mean training lasts for weeks, it can still have problems, right? Then still can have problems. How can we identify which specific components and servers have problems, then fix that, then exclude it, replace it, and fix that? That's the second thing.

另外，正如 Jeff 提到的，有很多运维上的协调，对吧？如果数据中心在维护整个电力——这个数据中心的角色——而这边这个角色实际上在做训练，那当然不太好。所以我们实际上要和数据中心、和搭建团队、和站点可靠性工程师以及所有机器学习研究员和工程师做大量协调。这是大量的协调工作。

Now also, as Jeff mentioned, there's a lot of operational coordination, right? If the data center is maintaining the entire power — this data center role — then this role is actually doing the training. That's of course not very good. So we actually coordinate a lot with the data center and the setups and the site reliability engineers and all the ML researchers and engineers. That's a lot of coordination.

我们得保持同步。我觉得 Jeff 在整体模型设计和软件设计方面也主导了很多工作。我觉得有时候确实会出故障。

We have to be in sync. I think Jeff also led a lot on the general model design and on the software design side. I think sometimes things got failed.

其实训练本身就是数据并行和模型并行的结合。如果这都在一个数据副本里，那一个副本也许会有情况——你知道，额外的副本就继续跑，对吧？你只要自动对权重取平均，所以不必为某一个副本挂掉而操心，对吧？所以这是很多非常全面的改进。

You know what, actually the training actually is the data parallelism and model parallelism together. If that's within one data replica, the one replica maybe there's a case — you know what, the additional replica just keep going, right? You just average weights automatically, so you don't have to be bothered by this one replica going down, right? So it's a lot of very comprehensive improvements.

有什么要补充的吗？

Anything to add there?

没有，我是说我觉得可能发生各种各样的情况，很难预测每一种可能的故障方式。所以也要构建健壮的东西，能够检测故障，即使你不一定清楚故障的成因，但你能检测到它，对吧？比如偶尔有些芯片，当温度升高时就开始出现不可靠的问题——比如二加二得五之类的。是的。

No, I mean I think there's all kinds of things that can happen and it's very hard to predict every possible way something might fail. So also building robust things that can detect failures even though you're not necessarily sure what caused the failure, but you can detect it, right? Like so you know occasionally you have chips that when maybe when the temperature goes up they start to have unreliability problems — like they add two plus two and get five or something. Yeah.

你得能检测到这种情况，因为那可不妙，如果你——

And you want to be able to detect that because that's not great if you're —

那是最讨厌的一种。那是静默数据错误。太可怕了，太可怕了。

That's the most nasty one. That's a silent data error. That's horrible, that's horrible.

是的。这可能发生在芯片本身，可能发生在不稳定的网络链路上，可能发生在很多地方。所以，你知道，我们甚至在沿用 Google 早期用的很多原则——因为当时我们买的是便宜的消费级 PC，它们不仅没有 ECC，内存里连奇偶校验都没有。所以如果你用很多台电脑做一件事，而它们都没有奇偶校验——

Yeah. And that can happen in the chip itself. It can happen in a flaky network link. It can happen lots of places. So yeah, you know, even we're using a lot of the same principles we used in the very early days of Google where, you know, because we bought these cheap consumer PCs, they not only didn't have ECC, they didn't have parity in their memory. And so if you use a lot of computers to do something and none of them have parity —

对。

Right.

那就会随机出现比特翻转。所以我们做的很多计算，你希望对比特翻转有鲁棒性，而做到这一点的方法之一是，你可以说，行，我在处理十亿个网页，丢掉一个大概也不是世界末日，所以你可以在硬件层之上的软件里做校验和，把这件事忽略掉，让系统保持健壮，即使某台机器或者网络链路之类不可靠。

Then you are going to get random bit flips in things. And so a lot of the computations we did, you know, you wanted to be robust to bit flips, and one of the ways you could do that was you could be like, fine, I'm processing a billion web pages, if I drop one of them it's probably not the end of the world, so you could just ignore that by checksumming in software above the hardware layer to make things robust even if you know a particular machine or maybe the network link or whatever was not reliable.

Google 非常信奉开源。所以我要稍微换个话题，谈谈开源。我知道观众里有很多学者，也有很多我们的合作伙伴。那我们就聊聊 Google 的 AI 基础设施战略，以及它如何确保这些不成为被设限的资源。我们谈到的所有这些，都希望与世界共享。那我们就聊聊 JAX 和 StableHLO，以及开源整体的演进，还有这在今天为什么如此重要，能确保它不会变成被设限的资源。

Now, Google is a big believer in open source. So I'm going to shift gears a little bit to talk about open source. And I know we have a lot of academics in the audience. We have a lot of our partners as well. So let's talk a little bit about Google's AI infrastructure strategy as it relates to making sure that it's not a gated resource. So all these things that we talk about, we want to make sure that we're sharing it with the world. So let's talk about like JAX and StableHLO and that evolution of open source as a whole and how is that so relevant today to ensure that this doesn't become a gated resource.

是的，我觉得对我们来说，与更广泛的生态互动、把东西开源非常重要，这样人们能看到源码、能修改、能帮我们改进，让它成为一个跨越许多组织的协作项目，而不是只握在我们自己手里——这非常重要。这就是我们开源 TensorFlow、开源 JAX、开源 XLA 编译器所用的 StableHLO 表示的原因。未来我们还会开源更多东西。多年来我们一直是开源的大贡献者——对 Linux 内核等等。我觉得我们通常是参与集体开源项目最大的组织之一，因为我们相信这件事，我们认为当所有人一起合作时，整个生态都会受益。

Yeah, I mean I think for us it's super important that we interact with the broader ecosystem and open source things so that people can, you know, see the source, they can modify it, they can help us improve it, and have it be a collaborative project that spans many, many organizations rather than just something we hold to ourselves — is super important. That's why we open source TensorFlow, why we open source JAX, why we open source sort of the StableHLO representation for the XLA compiler. And you know, we'll open source more things in the future. And we've been big contributors in general to open source — to the Linux kernel and so on for many years. And I think usually we're one of the larger organizations that contributes to collective open source efforts because we believe in that, and you know we think the whole ecosystem benefits when everyone works together.

这正好是个完美的过渡，让你谈谈 PyTorch 和你在 Meta 的经历。

Yeah, it's a perfect segue for you to talk about PyTorch and your experience at Meta.

是的。是的，就在这场炉边谈话之前我还在和 Jeff 聊。我觉得 Jeff 一直在主导很多 AI 基础设施的开源战略。正如 Jeff 所说，过去 Google 开源了 TensorFlow 和 JAX，Kubernetes 也是 Google 创造并开源的，还有 Android，以及很多很棒的东西。现在我们把 TPU 推向社区，在 Google Cloud（GCP）上强调 TPU，我们甚至更加加倍投入这一开源战略。所以事实上，我们不仅开源了 JAX，还构建了很多上层库——比如怎么做强化学习，我们叫 Tunix，它其实构建在 JAX 之上，我们也把它开源了；怎么做 checkpoint、怎么做推理相关的东西，我们在 JAX 之上做了很多。我们还构建了很多上层库和框架，也把它们开源，因为我们希望人们用起来，然后借这个开源战略直接使用 TPU。这是一方面。当然我们也想贴近客户所在之处，对吧？所以我们也想拥抱开源社区里成熟的产品。PyTorch 非常成熟，在社区里用得极广。你知道，那就用 PyTorch 吧，在 TPU 上拥抱 PyTorch。我们有个项目叫 torch TPU。目前是私有预览，我们想下季度做公开预览，就在几个月后，然后 Q4 做公开发布——就放在 GitHub 上，所有人都能用。昨天我也提到，如果用 PyTorch TPU，只需简单改几行——后端设备设成 TPU——那么希望所有训练和推理都只需在 TPU 上跑。我们其实同时支持 eager 模式和编译模式。这是在框架层面。我们也接触了——我相信观众里有很多用 vLLM 的用户，我们用 SGLang——我们也和 vLLM、SGLang 团队沟通，确保这类上层推理框架、开源框架也能在 TPU 上用。所以现在我们在做很多工作：我们开源自己的东西，也拥抱社区里成熟的开源产品。

Yeah. Yeah, I was talking with Jeff right before this fireside chat. I think Jeff has been leading a lot of the AI infrastructure open source strategy. As Jeff mentioned, in the past Google open sourced TensorFlow and JAX, and Kubernetes also was created by Google and open sourced by Google, and Android, and a lot of awesome stuff. Then as now we push the TPU into the community and emphasize TPU on GCP, Google Cloud, we're even doubling down more on this open source strategy. So as a matter of fact, not only do we open source JAX, we also build up a lot of upper-level libraries — like how can we do the RL, we call Tunix, which is actually built upon JAX, we open source that too; how to do the checkpointing, how to do the inference stuff, we actually do a lot on top of JAX. We also build a lot of upper-level libraries and frameworks, and we're also open sourcing that because we want people using that and then leverage this open source strategy to use TPU directly. That's one thing. And of course we want to meet where the customer is, right? So we also want to embrace whatever mature products in the open source community. PyTorch is very mature, PyTorch is very widely used in the community. You know what, let's use PyTorch, let's embrace PyTorch on TPU. We actually have this project called torch TPU. We actually, right now in the private preview, we want to do the public preview next quarter, just in a couple months, and then in Q4 we want public offering — just on GitHub and everybody can use it. And yesterday, as I mentioned, if we use PyTorch TPU, it's just several simple lines of change — the backend device is TPU — then hopefully all the training and all the inference we just need to run on TPU. We actually support both eager mode and the compiled mode as well. That's on the frameworks level. And we also engage — I'm sure in the audience we have a lot of users using vLLM, we use SGLang — we also talk with the vLLM and SGLang teams to make sure this kind of open upper-level inference framework, open source framework, we can also use that on TPU as well. So a lot of work right now we're doing: we open source our own stuff, we also embrace the mature open source products in the community.

然后我们观众里还有很多一年级和二年级的博士生。你们有多少是博士生？好。他们应该关注些什么？顺便说一下，其中一些还是本科生。

And then we do have a lot of first-year and second-year PhD students in the audience too. How many of you are PhD students? Okay. What are some of the things that they should be focusing on? Some of them are also undergrads too, just so you know.

好。是的。

Okay. Yeah.

是的，我的意思是，在职业生涯的那个阶段总是令人兴奋的，因为我认为你可以找到你真正、真正、真正喜欢并认为重要的事情，并找到一种方法推动某个领域向前发展，从而在世界上产生影响。我认为那只是一段有趣的时光。

Yeah, I mean it's always exciting to be at that stage in your career because I think you can find things you really, really, really enjoy and think are important and figure out a way to push some area forward in terms of making an impact in the world. I think that's just a fun time.

从基础设施的角度来看，你有没有考虑过某个物理瓶颈？有很多。

Is there like one physical bottleneck that you're thinking about just from an infrastructure perspective? There's so many.

嗯，我的意思是，我非常看好越来越专业化的硬件，因为我认为这是真正获得更高效系统的方式。我们现在有一些工作负载，其中少数几种工作负载将占据世界上大量的算力，对吧？所以如果你考虑这一点，那就迫切需要专业化。专业化的问題在于，如果你未来想做的事情发生了变化，那么你精心打造的硬件可能两年后就不再那么相关了。所以我认为，要让专业化真正发挥作用，你需要将硬件设计过程更多地自动化，对吧？因为现在你做硬件的方式是，你有一个大团队。其中一些人拿到高层规格，然后构建低层 RTL。然后因为你手工翻译了这个，你最终会有另一个团队来验证第一个团队做对了。然后你还会有一组人精心布局芯片。你知道，这看起来就像如果你能制造更多自动化循环，这些循环可以通过强化学习或其他进化技术来搜索，并且你能让这些循环运行得足够快，而当前的 EDA 工具通常不是为快速运行而设计的。但如果你能让它们运行得快，那么你就有机会真正拥有一个更加自动化的设计过程探索循环，或许能极大地压缩设计周期。例如，如果你能在三个月内用 10 个人设计出一款新芯片，而不是用 150 个人花两年时间，

Uh I mean I'm pretty bullish on more and more specialized hardware because I think that's the way you really get much more efficient systems. And we now have workloads where there's like a handful of workloads that are going to be a lot of the compute in the world, right? And so if you think about that, that is crying out for specialization. And the problem with specialization is if what you want to do changes in the future, then the thing you've lovingly crafted for in hardware for maybe two years is no longer perhaps as relevant. So I think to really make specialization work well, you need to automate much more of the hardware design process, right? Because right now the way you do hardware is you get a large team of people. Some of them take the high level spec and they build low-level RTL from it. Then because you've hand translated this, you end up with another team of people that are going to verify that the first team of people did the right thing. And then you're going to do another set of people who are going to lovingly lay out the chip. And you know that just seems like if you can make a lot more automated loops that are searchable by reinforcement learning or other evolutionary techniques perhaps, and you can make those loops run fast enough, which the current sort of EDA tools typically are not designed to run very quickly. But if you can make them run fast, then you have the opportunity to really have a much more automated exploration loop of the design process and perhaps to compress the design cycle tremendously. And if you could, for example, design a new chip with, you know, 10 people in three months instead of, you know, 150 people and two years,

你就会在世界上看到更多专业化的硬件，而且你也不必对未来下那么大的赌注。我未来两到六年想做什么计算，变成了三到六个月到四年后想做什么，那是一个容易得多的赌注。

you'll see much more specialized hardware in the world and you will have to bet much less on your future. What compute do I want to do two to six years from now becomes more of a three months to or five six months to four years from now and that's a much easier bet to make.

是的，所以如果我们回顾时间线，回到一年、两年前，那时我认为包括谷歌在内的许多前沿实验室，主要焦点是“嘿，如何让那个模型变得出色”。很多焦点都放在让预训练和后训练工作上。所以很多硬件策略都集中在训练、训练、训练方面。但现在随着大模型变得成熟，很多流量开始进入智能体世界和推理侧。所以 Jeff，如果你能分享一些想法，从硬件设计的角度，我们如何设计硬件来专注于强调推理流量？

Yeah so if we are going back timeline going back one year two years at that time I think a lot of Google included plus a lot of frontier labs the major focus is on hey how to make that model fantastic. There's a lot of focus on making the pre-training and post-training work. So a lot of hardware strategies on that training training training side. But now as the large model become mature uh is maturing then a lot of traffic start to go into agent words world and on the infant side. So Jeff, if you can share any thoughts on from hardware design point of view, how can we design hardware to focus on the uh to emphasis on inference inference traffic?

是的，我的意思是，我认为推理和训练有些不同。对于推理，你基本上有一个模型，你只是想为它服务大量请求。所以你想尽可能少地移动不变的信息，不变的东西是模型权重，变化的东西是请求和 KV 缓存等等。所以你需要设计一个系统，非常非常高效地最小化那里的数据移动。然后我还认为，当你看到的不仅仅是一个提示然后一个响应，而是更加独立的，你知道，一个智能体做一些事情，它决定调用一个工具，工具运行，工具结果返回，你把这些放入模型的上下文窗口，让它决定下一步做什么。我们会意识到我们所有的工具都太慢了，对吧？因为它们是为人类速度的迭代设计的，你知道，如果你的工具中编译代码，你会很伤心你的编译器很慢，因为如果你让你的推理硬件超级快，它生成代码的速度可能比你编译代码的速度还要快，当然也比你运行代码的速度快。所以我们一直在做的一件事是让我们的一些内部工具更快，使它们更好地用于

Yeah, I mean I think inference and training are somewhat different like you for inference you basically have a model and you just want to serve a lot of requests for it. So you want to move as little information that doesn't change as possible and the things that don't change are the model weights and the things that change are the requests and the KV cache and so on. So you need to design a system that is really really efficient at minimizing this data movement there. And then I also think as you see not just like a prompt and then a response, but much more independent, you know, an agent does some stuff, it decides to call a tool, the tool runs, the tool results come back, you put that together into the models context and let it figure out what to do next. We're going to realize that all of our tools are way too slow, right? because they were designed for human speed iteration and you know if you're compiling code as part of your tool you're really sad your compiler is slow because if you make your inference hardware super fast it can generate code maybe even faster than you can compile it and certainly than you can run it. So we've had actually one of the things we've been doing is making some of our internal tools faster to make them better for

嗯，事实证明，你实际上可以相当有效地从一种编程语言翻译到另一种，因为你实际上只是有一个完整指定的规格说明。你有整个程序，也许是用像 Python 这样的解释性语言写的，你只是想要在 Go 或 Rust 或 C++ 等中的完全等效程序。是的。所以智能体实际上可以做得很好。是的。这与我们通常与智能体的编码交互非常不同，那种交互就像“请给我做一个网络服务器”，它必须用各种假设来填充各种细节，而这些假设可能不是你想要的。但是有了完全指定的工具，是的，那实际上可以做得非常好。它可以运行所有单元测试。它可以翻译所有单元测试并在新系统中运行它们。它们可以并排验证行为，以确保行为相同。所以这实际上是一种非常有效的方式，也许用一种语言编写代码，然后翻译成另一种语言。实际上，我们在谷歌内部有一个活生生的例子，我们称之为 Project Turn。Project Turn 是因为我们在 TensorFlow 上构建了很多模型，但我们正在迁移到 JAX，我们将所有 TensorFlow 模型和 JAX 翻译，并做所有单元测试和代码测试，自动地。

uh it turns out you can actually translate uh from one language programming language to another quite effectively because you effect you really just have an entire fully specified spec of what you want. You have the whole program maybe written in a interpretive language like Python or something and you just want the exact equivalent program in Go or you know Rust or C++ or whatever. Yeah. And so an agent can actually do that quite well. Yeah. Which is quite different than the normal kind of coding uh interaction we have with an agent which is like you know please make me a web server and it has to fill in all kinds of details with all kinds of assumptions that may not be the ones you wanted. But with the you know fully specified tool yeah that can actually do a really good job of that. It can run all the unit tests. It can translate all the unit tests and run those in the new new system. they can like verify behavior side by side to make sure it behaves the same way. So that's actually a pretty effective way to get uh you know maybe writing lang code in one language and then translating it to another is that actually we actually have a live example at Google internally we call project turn project turn is because build a lot of models do on tensorflow but we're migrating to jacks we translate all the tensorflow models and magnet to jacks and do all unit test and cord test and automatically

现在有一篇在 NeurIPS 上被拒绝的论文，是你们提交的，据说影响很小。那是一篇蒸馏论文。

Now there was a certain paper in NeurIPS that was rejected that you all had submitted that said very little impact. It was a distillation paper.

哦，是的。

Oh yeah.

你想稍微谈谈那篇论文吗？

Do you want to talk a little bit about that paper?

我的意思是，是的，这就像一个励志故事，本意是给人启发。事实证明蒸馏很重要。这是一篇我和同事 Jeff Finton、Oriel、Vinols 提交的论文，关于如何取一个形状模型并使用蒸馏，将其作为教师模型蒸馏到一个学生模型中。

I mean yeah I mean this is like a motivational story like it's supposed to be an inspiration. Turns out distillation is important. This is a paper that my colleagues Jeff Finton, Oriel, Vinols and I submitted about how to take one shape model and use distillation and use that as a teacher to distill it into a student model.

那是哪一年？

Which which year it was?

嗯，是 2019 年，对吧？

Uh it was 2019, right?

2016 年。

2016.

是 2016 年吗？好的。

Was it 2016? Okay.

我们最初是在训练一个大型集成模型的背景下考虑这个问题的，这个集成模型由各种专门的视觉模型组成。这是我们论文中的一组实验之一。你有 20,000 个视觉类别，但你专门训练一个用于动物，另一个用于汽车，另一个用于其他东西，然后你把它蒸馏成一个擅长所有这些事情的单一模型。是的，它被拒了，但没关系。我们把它放到了 arXiv 上。

So we were originally thinking of it in the context of training a big ensemble of different kinds of specialized models for vision. It was one of the set of experiments we had in the paper. You have 20,000 vision classes, but you train a special one for animals and another one for cars and another one for whatever, and then you just distill that into a single model that's good at all those things. And yeah, it got rejected, but that's fine. We put it on arXiv.

好的。反正人们会读的。

Okay. People read it anyway.

所以不要气馁。

So don't be discouraged.

正是。好的。我们还剩几分钟。我们将接受几个提问。所以如果你们有问题，有几个人拿着麦克风在跑。开始了。这里有一个。也许我们可以从这里开始。

Exactly. Okay. So we have a few minutes left. We're going to take a couple of questions. So if you all do have some questions, there are a few folks running around with microphones. There we go. One over here. And so maybe we can start here.

嗨，如果你愿意站起来介绍一下自己，并简单说说你的背景。

Hi, and if you want to stand up and just introduce yourself and just give us a little one-liner about your background.

我叫 Ying。我曾在 Google Brain 工作了十年。

My name is Ying. I used to be in Google Brain for a decade.

又见面了。

See you again.

我想问 Bill 和 Jeff 的问题是，我们谈到 Google 早期使用商品化软件和社区计算。这就是 Google 在 2000 年代初扩大规模的方式。所以如今 AI 超级计算机变得越来越专业化，更像最初的超级计算框架。那么你如何看待硬件和机器系统的两种不同设计模式？你认为未来我们应该更多地转向商品化硬件，还是应该继续构建超级计算机？谢谢。

My question for Bill and Jeff is that we talk about the commodity software community computes for the early days of Google. That's how Google scaled in the early 2000s. So nowadays the AI supercomputer becomes more and more specialized, more like the original supercomputing framework. So how do you see the two different design patterns for hardware and machine systems? Do you think in the future we should move more towards commodity hardware or we should continue to build up the supercomputing machines? Thank you.

是的，我的意思是，我认为我们能够用商品化硬件扩展搜索的原因是，搜索是一个非常好的问题，如果你正确地分解它，机器之间几乎没有通信，而单台机器上有大量工作要做。所以你不需要任何奇特的互连。比如我们实际上有，我想，1 千兆以太网。不，我们在机器上有 100 兆以太网，然后我们每个机架共享一个 1 千兆上行链路，跨 20 台机器。所以你实际上有 4 比 1 的超额订阅。

Yeah, I mean I think the reason we were able to scale with commodity hardware for search is that search is this very nice problem where if you break it down right, you have almost no communication between machines and you have a lot of work to do on a single machine. So you don't need anything exotic in terms of the interconnect. Like we actually had, I think, one gigabit Ethernet. No, we had 100 megabit Ethernet on the machines and then we shared a one gigabit uplink per rack across 20 machines. So you actually had like a 4-to-1 oversubscription.

超额订阅。

Oversubscription.

是的，40 台机器。所以是 4 比 1 的超额订阅，但没关系，因为你发送像 Palo Alto 餐厅这样的查询到每台机器，然后你得到一个小片段，说这有 10 个结果之类的。而训练我认为确实推动了边界。你需要大量的连接性。我的意思是，你希望能在一个芯片上训练，对吧？那将是最好的，但那太耗时或者放不下。所以你最终不得不将问题划分到许多芯片上，而无论你怎么划分，模型并行或数据并行，你通常最终会有大量的通信。所以这就是为什么你在机器之间有这些更奇特的互连等等。你知道你想要每个芯片的最大性能，这样你就不必把它分散到那么多芯片上，但你仍然必须分散到很多芯片上。但这就是为什么我们使用液冷来使每个芯片在这种设置下尽可能高性能。我认为推理会更像你可以有专用硬件，但那不是那么奇特，特别是对于小模型。但当你开始得到更大的模型时，你开始有通信需求，相对于以太网商品化硬件、商品化网络来说，开始看起来相当奇特。但它肯定比训练更主流和商品化。这有意义吗？

Yeah, 40 machines. So it was a 4-to-1 oversubscription, but that's fine because you're sending like Palo Alto restaurants to each machine and then you're getting back like a little snippet that says like there's these 10 results or whatever. Whereas training I think really does push the boundaries. You need a whole bunch of connectivity. I mean you'd like to be able to train on a single chip, right? That would be the best, but that just takes too long or doesn't fit. And so you end up having to partition the problem across many chips, and the way you partition it, no matter how you slice it, model parallelism or data parallelism, you usually end up with quite a lot of communication. And so that's why you have these more exotic interconnects and so on between machines. And you know you want the maximum performance per chip so that you don't have to spread it out over as many chips, but you still have to spread it out over a lot. But that's why we're using liquid cooling to make each chip as performant as possible in that setting. I think inference is going to be more like you could have specialized hardware, but that is not as exotic, particularly for small models. But as you start to get larger models, then you start to have communication needs that are starting to look pretty exotic relative to, say, Ethernet commodity hardware, commodity networking. But it's definitely more mainstream and commodity than training. Does that make sense?

是的。

Yes.

我想我们这里还有一个问题。

I think we have another question right here.

嗨 Jeff，我叫 Andra，来自 Uber。我有一个关于 JAX 生态系统的问题，特别是考虑到你知道这个开源社区已经构建的东西，因为我看到你知道这几乎是 AI 基础设施的 Android 和 iOS 时代。那么你认为年轻工程师如何为开源 JAX 的库函数做出贡献？因为对于大多数用户来说，我认为有大量的 CUDA，或者说他们基于 CUDA 库构建库或函数，但对于开源 JAX，你认为年轻工程师可以贡献什么？

Hi Jeff, my name is Andra from Uber. I have a question regarding the ecosystem of JAX and JAX especially given you know this open source committed people have built because I see you know this is pretty much era of AI infra of Android and iOS quoted. So what do you see in terms of how young engineers can contribute to the kind of library functions of open source JAX? Because for most of user I think there are like heavy CUDA or say they build libraries or functions based on the CUDA libraries but for open source JAX what do you see young engineers can contribute to?

也许首先我认为 Open XLA 和 JAX 在 JAX 之上，JAX 栈已经完全开源了。所以这是一部分工作。另一部分工作是，目前我们内部正在与 Meta 合作开发 torch TPU，但最终我们会把它放到 GitHub 上，让所有社区研究人员和工程师可以贡献，所以这实际上是我们两个并行的工作流。当然，社区中任何年轻工程师想要开源，我的意思是贡献源代码仓库，非常欢迎与 Google 合作，我们可以讨论，我们可以谈谈我们如何共同开发这个开源战略仓库，这是一件事。此外，我认为最终很多用户使用 JAX 栈或 PyTorch 栈用于 TPU 或 GPU，我认为现在社区如果我有任何使用问题，很多人现在会说，嘿，你知道我在打电话给 Google，我在打电话给媒体，因为你们有所有经验丰富的工程师，你可以帮助我，或者我把问题放到社区讨论板上，希望有人理解我的问题，有人也有类似经验可以回答我的问题，所以我可以读你的答案，我可以复制你的答案来尝试，但这在我看来都非常慢。我认为很多社区帮助我们可以贡献在开源战略上，让我们想象如果我们有一个 JAX 智能体和一个 PyTorch 智能体，运行在 GPU 或 TPU 上无所谓，但如果我有一个关于使用 GPU、使用 TPU、通过 JAX 或 PyTorch 的问题，如果社区中的每个人都在贡献数据源，你帮助训练那个智能体变得非常强大，那么最终想象一下，如果我有一个 TPU 同事智能体，我正在运行训练、后训练、推理，我不必问 Google，我不必去社区讨论板问问题，我直接使用这个智能体来帮助我，如果智能体能回答我大部分问题，那就太棒了，但这需要整个社区贡献，因为我们需要很多数据点和经验。

Maybe so first of all I think open XLA and JAX on top of JAX the JAX stack has been entirely open sourced. So that's one bucket of work. Another bucket of work is it's a torch TPU right now internally we're collaborating with Meta we're developing but eventually we're going to put into GitHub then let all the community researchers like engineers can contribute so that's actually our two parallel workforce so of course anybody young engineers in the community want to open source I mean contribute source repository more than welcome to work with Google and we can discuss, we can talk about how we can think about how can we co-develop this open source open source strategy repository right that's one thing in addition in addition I think eventually a lot of users using either JAX stack or PyTorch stack for TPU or for GPU I think right now the community if I have any issues to using it and a lot of people right now is hey you know what I'm calling Google I'm calling a media because you have all the experienced engineers you can help me or I put my question into the community whatever discussion board right and hopefully somebody understand my questions somebody also has similar experience can answer my question so I can read your answers I can copy your answers to try that right but this I in my opinion that's all very slow I think the a lot of community help we can we can have to contribute on both open source strategy is let's imagine if we can have a JAX agent and the PyTorch agent that's running on GPU running TPU doesn't matter but if I have a question on using GPU using TPU over JAX over PyTorch if everybody in the community you contributing data source you help to train that agent to be really powerful then eventually just imagine this if I have a TPU coworker agent I'm running training I'm running post-training I'm running inference and I don't have to ask Google I don't have to go to the community discussion board ask my questions I directly using this agent to help me if the agent can answer me a large portion questions bingo that's fantastic but this need all the community to contribute because we need a lot of data points and experiences.

也许我会给一个关于开源贡献的元观点。就像我认为人们有很多很多方式可以贡献。

Maybe I'll give a meta point about open source contributions. Like I think there's lots and lots of ways for people to contribute.

跟不同代码仓库的维护者交流一下、弄清楚方向，真的很有帮助——比如“我在考虑做这个，会有用吗？或者你有什么建议我该做的？”在你直接甩给别人 5,000 行代码说“喏，给你”之前，先争取一些认可、或者听听什么方向有用的建议，我觉得是个非常好的主意。如果你很想贡献但暂时没有特别想做的方向，通常大家心里都有一长串觉得会有用的东西。

It's really good to interact with some of the owners of different repositories to try to identify — I'm thinking of doing this, would that be useful, or do you have ideas about things I should work on? Before you just chuck over 5,000 lines of code to someone and say, "Hey, here it is," I think it's a really good idea to get some buy-in or some guidance about what would be useful. And if you're eager to contribute but don't necessarily have a particular thing you want to work on, there's often a long list of things that people have in mind that would be useful.

好的，非常感谢您的回答。我唯一的顾虑是，很多函数或模块核心仍然严重依赖 CUDA。所以您是否看到一种趋势，用 JAX 或 OpenCL 或 torch TPU 来实现等价的东西，从而获得某种相对于 CUDA 函数的通用兼容性？

Yeah, absolutely. Thank you for the answer. I think the one hesitation I have is that many functions or module corners still heavily rely on CUDA. So do you see the trend of implementing something equivalent using, you know, JAX or OpenCL or torch TPU, so that we have some kind of generalization compatibility compared to CUDA functions?

这对我们来说是个好方向——是的，我认为最终会走到那一步。当然在 Google 内部，当我们审视整个内部 JAX 栈以及 TPU 上的 PyTorch 栈时，我们实际上会对照所有 CUDA 函数来看我们自己的函数，确保我们至少不落下风、甚至更好。所以我认为这是我们必须搞定的一件必要之事。而在这一层，我们其实想看看哪些层也可以开源，比如底层的 TPU SDK 工作。

That would be a good direction for us to — yeah, I think eventually. Of course at Google internally, when we look at all the internal JAX stack and also the PyTorch stack on TPU, we actually look at all the CUDA functions and look at our functions. So we're making sure we are at least on par, if not more. So I think that is a necessity we have to nail down. And in that layer, we actually want to — we want to see which layer we can also open source, like low-level TPU SDK work.

是的。我的意思是，我也认为人们希望站在高得多的抽象层次上思考。所以用 JAX 或 PyTorch 表达式来思考才是你想做的，而不是去想“我怎么用某种 kernel 代码把它并行化”——不管是 TPU kernel 语言还是 GPU kernel 语言——来榨取最大性能。那真的应该、也希望能由编译器和底层系统来给你，而你只需要用像乘法这样漂亮的抽象来思考。

Yeah. I mean, I also think people want to think at a much higher level of abstraction. So thinking in terms of like JAX or PyTorch expressions is what you want to do, not necessarily "how can I parallelize this with some kernel code" — be it a TPU kernel language or a GPU kernel language — in order to get maximum performance. That really, hopefully, the compiler and underlying system can give you that, and you can just think in beautiful abstractions like multiply.

很高兴了解这些。谢谢。

Great to know. Thank you.

我想我们还有时间回答最后一个问题。就最后一个问题。请讲。

I think we have time for one last question. Just one last question. Go ahead.

好的。我是纽约大学的 John，做网络研究很多年了。Bill 和另一位讲者的演讲让我印象很深。关于这里的可扩展性：我知道这些 TPU 是通过 3D 环面（3D torus）连接的，而其他一些——比如 UEC 超以太网联盟等等——他们在研究跨互连的 fabric。从可扩展性的角度看，当你们把 GPU 扩展到几千甚至上百万颗芯片时，这个 3D 环面是否仍然能处理如此大规模下的所有通信？

Okay. Hi, John from NYU. Been working on networking for many years. So I was very impressed by the talk, Bill, and the other fellow. So regarding the scalability here: I know these TPUs are connected in the 3D torus here, compared with other — you know, the UEC Ultra Ethernet consortium and so on — they are looking into this, you know, fabric across interconnection here. From the scalability point of view, do you see, when you scale up the GPU up to a few thousand or even one million here, whether this 3D torus is still able to handle all the communication at that large scale?

是的，我可以这么说：我们 TPU v8 最大的 pod 我记得是 9,600 多颗芯片。这就是我们做 3D 环面的规模。再往上，我们有 Pathways 作为软件抽象，架在许多个 3D 环面连接的 pod 之上，它使用数据中心网络——你数据中心里有什么 fabric 就用什么——甚至可以是跨城训练部署，比如在俄克拉荷马这栋楼里有五个 pod，在爱荷华那栋楼里再有八个 pod，中间用高速广域网链路连起来。所以这对我们做规模扩张来说效果相当不错。而且你有时确实需要映射你的计算，比如有数据并行和模型并行两个维度，通常你会希望模型并行部分落在单个 pod 或 pod 的一个切片内，然后在那些 pod 或切片之间放数据并行副本。这对我们效果很好。基于环面的网络的好处是，本地连接超级容易，不需要在数据中心地面上做非常复杂的布线。

Yeah, I mean, I can say our largest pods for TPU v8, I believe, are 9,600-something chips. So that's the scale at which we do 3D tori. And then beyond that we have Pathways as a software abstraction on top of many of those 3D torus-connected pods, and that uses a data center network — whatever fabric you have in your data center — or even a multi-metro training setup where maybe you have five pods in this building in Oklahoma and eight more in this building in Iowa, and a high-speed WAN link between. So that seems to have worked pretty well for us for scaling up. And you do sometimes want to map the computation you're doing so that you have, for example, data parallel and model parallel dimensions, and you might typically want the model parallel aspect to be within a single pod or a slice of a pod, and then have data parallel replicas across those pods or slices of pods. And that seems to work pretty well for us. The nice thing about torus-based networks is they're super easy to connect locally, as opposed to needing very complicated wiring in your data center floor.

是的。

Yep.

除了那些奇特的镜像光交换机。

Except for the funny mirrored optical switches.

Bill，有什么要补充的吗？

Anything to add, Bill?

是的，确实如此。我认为是这样。我们在机架内、立方体内有 ICI。然后用 OCS 把它扩展到 9,600。再往上，现在我们有——但实际上是由 Pathways 软件来编排的。再超过 34,000 颗芯片，我们就用数据中心网络把所有东西连起来。这种可扩展性——我觉得对我们效果相当不错。

Yeah, that's very much true. I think so. We have ICI within the rack, within the cube. Then we use the OCS to scale it up to 9,600. Then even going up, now we have — but actually got orchestrated by the Pathways software. Then going beyond 34,000 chips, we use data center networking to connect everything together. This kind of scalability — I think it's working pretty well for us.

是的。而且我想说，即便在那个规模上，我们也能做完全同步的训练，这对机器学习的可复现性和可解释性来说很好。我认为某个时候异步训练会回归，但到目前为止我们已经把同步训练推得相当远了。我们拭目以待。

Yeah. And I would say even at that scale, we're able to do fully synchronous training, which is nice from the ML reproducibility and interpretation standpoint. I think at some point asynchronous training will come back, but so far we've been able to push synchronous training quite far. And we'll see what happens.

这是个很好的收尾。我们时间到了。非常感谢你，Jeff。谢谢你，Bill，感谢你们抽时间。我们在外面露台边准备了一些很酷的周边给你们，其中包括印着著名 Jeff Dean 梗的笔。

That's a great way to end. We're out of time. Thank you so much, Jeff. Thank you, Bill, for your time. We have some really cool swag outside by the patio for you, which includes some famous Jeff Dean memes on a pen.

跟我说说那个。

Tell me that.

还有一件 Dav Labs 的 T 恤。

And a Dav Labs t-shirt.

谢谢。太好了。非常感谢大家。

Thank you. Great. Thanks so much, everybody.

谢谢。

Thank you.