AI 推理经济学：批次大小、延迟与成本

今天我要采访的是 Reiner Pope，他是新芯片初创公司 Maddox 的 CEO。此前他在 Google 从事 TPU 架构等多项工作。这次采访的形式与我以往的很不一样，将是一场黑板讲座，我们马上开始。事实上，我们专门为此搭建了这个全新的演播室。很荣幸能和你一起启用它。我们将讨论模型架构、机器学习基础设施等诸多话题。我认为这个话题重要的原因是，一旦你真正理解了训练和推理在集群中是如何运作的，很多关于 AI 为何如此、AI 架构为何如此、API 价格为何如此，以及 AI 进步为何如此的原因，就会开始变得清晰。而要理解这些细节，你需要一块黑板。所以 Reiner，非常感谢你来做这个节目。

Today I'm interviewing Reiner Pope who is CEO of Maddox, which is a new chip startup. Previously he was doing TPU architecture and many other things at Google. This is a very different format from my usual interviews. This is going to be a blackboard lecture; we're going to get up in a second. We in fact built this whole new studio with specifically this format in mind. And so it's a pleasure to get to inaugurate it with you. We're going to be talking about model architecture, ML infra, many other things. And the reason I think it's an important topic is because once you actually understand how training and inference actually work in a cluster, as we'll see, a lot of things about why AI is the way it is, why AI architectures are the way they are, why API prices are the way they are, fundamentally also how why AI progress is the way it is start making sense, and you need to understand the details to get there, and you need a blackboard to understand the details. So Reiner, thank you so much for doing this.

嗯，很高兴来到这里。

Yeah, very happy to be here.

好的。事先声明，我是 Maddox 的天使投资人，但这与本期播客无关。Reiner，也许我们可以从这个问题开始。我们看到一些公司，比如 Claude、Codex 和 Cursor，提供了类似“快速模式”的选项：以 6 倍的价格，获得 2.5 倍速度的流式 token。从机制上讲，我很好奇这背后的原理。为什么多付钱就能获得更低的延迟？第二，能不能更进一步？如果多付 100 倍，能否获得更快、甚至极快的速度？第三，反过来呢？能不能有像“Claude Code 慢速模式”这样的东西，如果你愿意等上几分钟，就能获得更便宜的价格？也许这能引出你将在讲座中进行的分析。

Okay. Full disclosure, I am an angel investor in Maddox, but that's unrelated to this podcast. Reiner, maybe to kick us off, I'll ask this question. So, we have a couple of companies like Claude and Codex and Cursor offering something like fast mode where for 6x the price, they'll give you streamy tokens at 2.5x the speed. Mechanically, I'm curious what's going on here. Like, why is it the case that you can pay more to get faster latency? And two, could you keep going? Could you pay 100x more and somehow get even faster speeds or much much faster speeds? And three, could you go the other way? Could you have something like Claude code slow mode where if you are willing to wait for minutes on end, you could get even cheaper prices. So maybe this will help motivate the kind of analysis that you'll be doing through the lecture.

很好。直接跳到结论，最大的影响因素是 batch size，但我们现在要做的是精确量化它的表现以及对延迟和成本的影响。还有另一个效应，你可以称之为推测解码或多 token 预测。我们稍后再谈，但首先我们要讨论的是 batch size。所以我想介绍两个分析原则。首先，我们将进行屋顶线分析，看看如何在芯片集群上运行 Transformer 模型。我们以 Blackwell NVL72 集群为例，也就是一个包含 72 块 GPU 的机架。屋顶线分析意味着我们要考察内存带宽和计算性能。另一方面，我们只考虑模型的两个简单因素：操作权重的时间，以及操作上下文（KV 缓存）的时间。那么，我们开始吧。

Great. To jump to the conclusion, the big effect is batch size, but what we're going to do now is quantify exactly what that looks like and what its implications are on latency and cost. There's going to be another effect which is you can call it speculative decoding or multi-token prediction. We can maybe come back to that later, but I think the first thing that we'll talk through is batch size. So what I'd like to introduce is sort of the two principles of analysis. Firstly, we're going to look at a roof line analysis of how I run a transformer model on a cluster of chips. We'll take a sort of a, let's say, a Blackwell NVL72 cluster, so a rack of 72 GPUs. And the roof line analysis means we look at memory bandwidth and compute performance. And then the other side of that is that we're going to look at just two simple factors of the model: the time to operate on the weights and then the time to operate on the context, the KV cache. So let's jump in.

我们要做的是估算运行特定形状推理所需的时间。当然，我们无法做到完美，无法精确预测时间。因此，我们将进行近似估算。我们会说时间必须大于或等于某个量。我们将考虑两个不同的方面：内存读取的时间和计算的时间。事实证明，即使使用简单的模型，这也能提供很强的预测能力。那么，逐一来看，计算所需的时间是多少？计算中实际上需要做两件事：乘以所有活跃参数，以及处理注意力机制。乘以所有活跃参数：我运行某个 batch size，模型中有一定数量的活跃参数，然后除以计算吞吐量，即芯片的每秒浮点运算次数。这是一个硬件常数。这涵盖了所有权重矩阵乘法的计算时间。有一点需要注意：我们忽略了注意力计算的时间，但通常相比之下它很小，所以忽略不计。

What we're going to try and do is estimate the time that it takes to run an inference of a certain shape. Now, we're not perfect here. We can't exactly predict the time. And so, instead, we're going to approximate. And so, we're going to say that the time must be greater than or equal to a certain quantity. And so, we're going to consider two different aspects. We're going to look at the time for the memory fetches and then the time it takes to do the compute. And it'll turn out that this actually gives us a very strong predictive power even with a simple model. So one by one, what is the time that it takes to do the compute? So there are really two things I need to do in the compute. I need to multiply by all of the active parameters and then I need to do some work on the attention. So multiplying by all the active parameters. I have a certain batch size that I'm running and then I've got a number of active parameters in my model and then I'm just going to divide this by the compute throughput which is the flops of the chip. So this is a hardware constant. So this accounts for all of the compute time for all of the weight matrix multiplies. There's a little caveat here: we've sort of ignored the time to do any of the attention computation, but that in general can be quite small in comparison to this. So we'll ignore this.

也许我会时不时插话，问一些非常基础的问题或澄清一些基本点。但为了观众：你不是一次只服务一个用户。batch 指的是你同时服务多个不同用户。

Maybe I'll just interject from time to time to ask some very naive questions or to clarify some basic points, but just for the audience: you're not serving one user at a time. The batch refers to the fact that you're serving many different users at the same time.

是的。所以我可以稍微解释一下 batch 的动机。我们会确切看到为什么 batch 是如此有利的优化，但结果会是，如果你不将许多用户一起批处理，你得到的成本和经济性可能比将许多用户一起批处理差一千倍，我们将能非常清楚地看到这一点。

Yeah. So I can motivate the batch at least a little bit. We will see exactly why batch is such a favorable optimization, but what will turn out to be the case is that if you do not batch together many users, the cost and the economics you get can be like a thousand times worse than if you do batch many users together, and we'll be able to see that quite explicitly.

然后是活跃参数数量：比如，以 DeepSeek 模型为例，DeepSeek V3 模型大约有 370 亿个活跃参数和 7000 亿个总参数。所以我们只关注单个 token 活跃的那些参数。好了，我们已经对计算性能进行了建模。我会继续写等号，但在所有这些情况下，你可以认为这个时间至少是这么多，可能还有一些我们忽略的项。在内存方面，我们需要做什么？我们需要获取所有权重。所以需要一些时间来获取所有参数，而不仅仅是活跃参数。所以有权重获取时间，此外还有 KV 缓存获取时间。这实际上取决于 batch size。对于 batch 中的每个元素，我们必须获取整个上下文长度的 token，然后每个 token 有大小，即每 token 的字节数。这是一个模型参数。

And then number of active parameters: this is saying like if I look at, for example, a DeepSeek model, the DeepSeek V3 model has about 37 billion active parameters and 700 billion total parameters. So this is we're focusing on just the ones that are active for a single token. Okay, so we've modeled compute performance. I'm going to keep writing equals, but in all of these cases, you can think of this time as being at least this much and maybe there will be some terms we ignored. On the memory side, what do we need to do with memory? We need to fetch all of the weights. And so there is some time to fetch all of the total number of parameters, not just the active parameters. So there's weight fetch time, and in addition there's a KV cache fetch time. So this actually depends on batch size. For every element of the batch we have to fetch an entire context length worth of tokens, and then there's a size per token. So bytes per token. And so this is a model parameter.

也许先退一步，快速解释一下什么是 KV 缓存。

And maybe just back in, let's just explain what the KV cache is real quick.

是的。所以当我进行前向传播时，让我实际画一下自回归推理是如何工作的。这是在解码阶段。如果我有一堆文本 token，我是在增长一个张量，因为最终 token 被表示为某个嵌入维度的张量，然后在这个方向上有序列长度。运行解码的工作是，我必须将每个 token 通过一系列不同层的矩阵乘法。一般来说，我必须对所有 token 做这项工作。

Yeah. So when I do a forward pass, let me draw actually how the autoregressive inference works. So this is during decode. If I think I have a bunch of tokens of text, I'm growing a tensor because ultimately the tokens are represented as some tensor in some embedding dimension, and then in this direction I have the sequence length. The work of running a decode is I have to run each token through a whole bunch of matrix multiplies over a bunch of different layers. And in general, I'm going to have to do that work over all of these tokens.

但解码的一步实际上就是生成这额外的一个 token 路径。这里我要做的就是运行一次完整的前向传播，乘以模型中所有权重矩阵。但这里还有注意力机制，这个 token 会以这种方式查看所有过去的 token。它具体在看什么呢？它在看模型为这些 token 产生的某种内部表示，我们称之为 KV 缓存。所以这个单个 token 关注所有历史 token 的过程就是注意力机制。它主要受内存读取主导，而不是矩阵乘法。

But then one step of decode is actually to produce just this one additional token path here. And so what I'm going to do there is I'm going to run a full forward pass of multiplying by all of the weight matrices in the entire model. But then I've got this attention mechanism where this token is looking at all of the past tokens in this way. And what is it looking at specifically? It is looking at some internal representation that the model has produced of the tokens, and we call that the KV cache. So this process of this single token attending to all of the history of tokens, that's attention. It is mostly dominated by memory fetches rather than matrix multiplies.

嗯。

Mhm.

我们这里显示了正在读取的内存量，然后这当然只是除以内存带宽。所以是每秒内存字节数。实际上，这些方程已经足够我们画出一些拟合线了。我们想关注的是对 batch 的敏感性，以及另外对上下文长度的敏感性，我们会分开画。我们说过，能获得的大效果是延迟与成本在 batch size 上的某种权衡。所以我们把它们画出来。我认为我们只需要画两张图。我们先画 batch size 与时间的关系。当我们看这个形状时，我们有一个求和的最大值，然后还有另一项。所以我们逐项看看这些项如何随计算时间和内存时间缩放，以及它们如何表现。我们先看计算时间。它纯粹是 batch size 的线性函数，没有偏移。所以它是这样一条曲线。这是 t compute。然后在内存方面，我们有一部分是常数，即权重读取的常数基准偏移。最后我们有这一项，即 KV 读取，它随 batch 线性增长，所以看起来像这样。所以这两者之和与这个取最大值。我们先画和。两个内存时间合起来最终形成这样一条弯曲的斜率线。

So we've got the amount of memory that we're fetching shown over here, and then this is of course just divided by the memory bandwidth. So the memory bytes per second. So in fact, these equations here are actually enough for us to now draw some fit lines. And so the things that we'd like to look at are sensitivity to batch and then also, which we'll draw separately, to context length. So we said that the big effects you can get is some trade-off in latency versus cost in batch size. So let's draw them out. I think there's just really two graphs we want to draw. We'll first just draw batch size versus time here. So when we look at the shape of this, we've got a maximum of a sum and then another term. So let's look at these terms one by one and how they scale the time for compute and memory and how they show up. So let's first look at this compute time. This is purely linear in batch size with no offset. So it is some curve like this. This is t compute. And then on the memory side, we've got some portion here that is just this constant that is constant in some base offset here, which is the weight fetch. And then finally we have this term here which is the KV fetch, which is linear in batch, and so it looks like that. So the sum of this plus this maxed with this. So let's at least first draw the sum. So the two memory times in conjunction end up looking on this curved slope like this.

嗯。

Mhm.

然后我们得到整体最大值。我在这里画个小图。它是这两条曲线的最大值吗？有道理吗？好的。那么这实际上意味着什么呢？这是一个延迟图。所以如果我增大 batch size，一开始我对 batch size 的依赖并不强，所以这里有一个延迟的下界。延迟下界。这已经部分回答了问题：对于给定的硬件配置，然后我们可以讨论不同的硬件配置，但对于给定的硬件配置，存在一个延迟下界，就是：我需要将所有参数从内存读取到芯片中，这需要一定时间。如果我使用了全部内存带宽，我无法做得更好。

And then we get the overall maximum. I'll draw a little figure here. Is it the maximum of these two curves? Make sense? Okay. So what does this mean actually? This is a latency plot. So if I grow my batch size, I get initially some not very strong dependence on batch size, and so there's some lower bound on latency here. Latency lower bound. So this already partially answers the question: for a given hardware configuration, and then we can talk about varying hardware configuration, but for a given hardware configuration there is a lower bound on latency, which is simply: I need to read all of my total parameters from memory into the chips, and that takes a certain amount of time. If I use all of my memory bandwidth, I can't do any better than that.

你画计算时间斜率的方式，以及 KV 如何增长，以及 KV 对内存时间的影响，如果这个在上面或下面，或者这是否必然成立？因为如果这总是成立，那么随着 batch size 增长，计算总是主导 KV，这意味着如果你有足够大的 batch size，也许内存从来不是问题。

It seems like the way you've drawn the slopes for compute time and how the KV grows, and what implication the KV has on memory time, that as what if this were above or below, or is that necessarily the case? Because if this is always true, then as batch size grows, compute always dominates KV, which suggests that if you have big enough batch size, maybe memory is never an issue.

是的，这实际上对上下文长度非常敏感。所以我认为我们应该回头探讨这一点。随着你改变上下文长度，KV 读取时间会不断上升，从而导致从计算受限到内存受限的转变。

Yeah, this is really sensitive to the context length. So I think we should come back and explore this. There will be, as you vary the context length, the KV fetch time will go up and up, so that'll cause a transition from compute limited to memory limited.

斜率恰好等于计算时间的斜率有什么特别的意义吗？

And is there something especially significant about the slope being exactly the slope of the compute time?

是的，每当我们有平衡点时，这表示你正好做对了。所以对于斜率匹配的特定上下文长度，这意味着我同时受内存和计算限制，这是一个非常理想的状态。但假设这是一个非常简单的代数问题，假设最优是 100k 上下文长度，而你到了 200k 上下文长度，你的 MFU 会下降到 50% 吗？稍微超出上下文长度的最佳范围（金发姑娘区）会对 MFU 产生巨大影响吗？

Yeah, whenever we have balance points, it kind of says that you're getting it exactly right. And so for the particular context length where the slopes match, that says I am equally memory bound and compute bound, which is a really desirable place to be. But suppose it's like this is a very simple algebra problem, but suppose it's you know the optimal is 100k context length, and you go to 200k context length, does your MFU go down to like 50%? Does it have a humongous impact on MFU to be like slightly outside of context length optimal range goldilocks zone?

没错。所以这里建模的结果确实如此。这里有一个关键点，我把上下文长度建模为，或者说我把内存读取建模为上下文长度的线性函数。这实际上取决于模型架构。对于许多或所有使用密集注意力的模型架构来说，这是成立的。而稀疏注意力实际上扩展得更好。

That's right. So that is true as modeled here. There's a key point here that I'm modeling this context length as, or I'm modeling the memory fetch as linear in context length. That actually depends on model architecture. It is true for many of or all of the model architectures with dense attention. There's sparse attention actually scales much better than that.

明白了。那么稀疏注意力是大家都在实践中使用的吗？

Got it. And is sparse attention what everybody uses in practice?

我对稀疏注意力非常兴奋。很难知道各个实验室在使用什么。DeepSeek 已经发布了一种稀疏注意力机制。

I'm pretty excited about sparse attention. It's hard to know what the labs are using. DeepSeek has published a sparse attention mechanism.

我想顺便提一下稀疏注意力。一些 DeepSeek 论文中发布的稀疏注意力最终在这一项中加入了平方根。

I'll just like put a plug in that sparse attention. Some of the DeepSeek papers that have published sparse attention end up putting a square root in this term.

好的。到目前为止我们看了延迟。从这上面很难读出成本。所以如果我想成本意味着什么，我要运行这个推理，我会使用 GPU 一定秒数，比如 1 毫秒或 20 毫秒之类的。我必须为那段时间支付租赁费用。比如每个 GPU 每小时 2 美元之类的。所以这就是这次推理的成本，但我在推理过程中处理了多少 token 呢？那就是 batch size。所以我们实际想要画的是成本与 batch size 的关系，也就是 T 除以 B 与 batch size 的关系。

Okay. So far we've looked at the latency. It's kind of hard to read off cost from this. So if I think what does cost mean, I'm going to run this inference, I'm going to use the GPU for a certain number of seconds, like 1 millisecond or 20 milliseconds or something like that. And I have to pay the rental time for that time. So like it's $2 an hour per GPU or something like that. So that's the cost of this inference, but how much value have how many tokens have I processed during that inference? That is the batch size. And so what we actually want to plot is going to be the cost versus batch size, which is like T over B versus batch size.

这是每个 token 的成本。

This is the cost per token.

所以我们要想象将这三条曲线分别除以 b。也就是乘以这个倒数。结果就是，计算曲线原来是线性的，除以 b 后变成了常数，这是 t compute。KV 读取原来是线性的，现在也变成了常数。KV 读取。而权重读取原来是常数，现在除以 b 后变成了双曲线。然后我们再次计算和的最大值。

So like we have to imagine dividing each of these three curves by b. So multiplying by this reciprocal. And so what we end up with there is the compute curve is going to be, it was linear, we divide by b that makes that a constant here, and this is t compute. The KV fetch was linear, now it becomes a constant as well. KV fetch. And then the weight fetch was constant, and now we've divided by b and so it becomes this hyper parabola. And so again, we're going to compute the max of the sum.

所以这两项之和将抛物线向上移动。KV 获取和权重获取的总和给出了一个更高的抛物线，就像这样。

So the sum of these two terms shifts the parabola up. The sum of the KV fetch and the weight fetch gives us a higher parabola that's like this.

嗯。

Mhm.

然后我们在这里取与计算的最大值。所以我们最终得到这个我们关心的整体形状。再次，我们看到一些限制行为。成本在批大小为 1 时一开始非常高，实际上几乎趋于无穷，因为有很多权重获取没有被大批大小摊销。但随着我们增加批大小，权重获取被摊销到许多不同的批次元素上，它们的成本变得非常小，最终计算时间主导了成本。

And then we're going to take the max with the compute here. So we end up with this being the overall shape that we care about. So again, we see some limiting behavior. The cost initially starts very high at batch size of one, actually it almost goes to infinity because we've got so many weight fetches which are not amortized over a large batch size. But then as we increase the batch size, the weight fetches become amortized over so many different batch elements that their cost grows very small, and eventually the compute time ends up driving the cost.

嗯。

Mhm.

所以存在一个成本的下限，就是这个。

So there is a limiting lower bound on cost, which is this one here.

是的。

Yeah.

所以 Claude Code slow 或 Codex slow 之类的只会停留在这条线上，不会有太大帮助，因为你无法将 KV 值摊销到更大的批次上。

So Claude Code slow or Codex slow or whatever would just live on this line and it wouldn't help much because you're not able to amortize the KV values over a much bigger batch.

是的。它们每个批次都是唯一的。计算也是每个批次唯一的。那么在摊销掉所有其他东西之后，每个批次的最小工作量是多少？在这一点上，你不再受内存带宽限制，实际上你需要多大的批次？前沿模型的实际批次有多大？

Yeah. They're unique per batch. The compute is also unique per batch. And so what is the minimum work you can do per batch after amortizing everything else away? So at this point where you are no longer memory bandwidth bound, what practically how big a batch do you need? Like how big are the batches practically for frontier models?

实际上你可以直接求解。而且它对模型架构并不特别敏感。所以我们来算一下。我们讨论的是当内存时间等于计算时间时的情况。这就是问题的关键。现在我将忽略 KV 获取项，因为我们关注的是批大小，实际上问题在于权重何时被乘法摊销。我将专注于比较权重获取时间和权重乘法时间。我忽略 KV 获取项只是为了简化分析，以便得到一个干净的答案。所以我们让这一部分等于这两项。是的。写出来就是：总参数数量除以内存带宽等于批大小乘以活跃参数数量除以计算性能。看这里，上面的都是模型参数，下面的都是硬件参数。重新排列使得硬件参数在一边会很方便。所以这等价于：内存带宽等于批大小乘以活跃参数数量除以总参数数量。所以这是一个硬件参数。实际上，这最终是一个无量纲常数。如果从浮点运算的角度看，这个量的量纲是什么？这是每秒乘法次数，这是每秒字节数。所以并不完全无量纲。但你可以这样处理：每秒乘法次数乘以，假设我在做 FP4。那么就是每秒 FP4 乘法次数乘以每个 FP4 是半字节的事实。这样我实际上可以使其成为无量纲的。在大多数 GPU 上，这个值大约是 300 左右。

You can just solve for that actually. And it's not even particularly sensitive to model architecture. So let's go ahead and do that. What we're talking about is we're going to say when the memory time is equal to the compute time. That's what that question is. For now I'm going to discard the KV fetch term, because we're focused on what the batch size is, and really there's a question of when the weights are amortized over the multiplies. I'm going to focus on comparing the weight fetch time to the weight multiply time. I'm going to disregard the KV fetch term just to simplify the analysis so we can get a kind of clean answer out. So we're going to equate this portion with these two terms. Yeah. So writing that out, we get number of total parameters over memory bandwidth is equal to batch size times number of active parameters divided by the compute performance. So looking over here, everything on the top, these are model parameters. Everything on the bottom, these are hardware parameters. It turns out to be nice to rearrange them such that we have the hardware parameters on one side. So this is equivalent to memory bandwidth being equal to batch size times number of active parameters divided by the number of total parameters. So this is a hardware parameter. Actually, this ends up being a dimensionless constant. If you look in terms of flops, what are the dimensions of this? This is multiplies per second. This is bytes per second. So that's not quite dimensionless. But what you do is you say like multiplies per second times, let's say I'm doing FP4. So I do like how many FP4 multiplies per second times the fact that each FP4 is half a byte. And so I can actually make this end up being dimensionless. And this ends up being on most GPUs around 300, somewhere around 300.

抱歉，这个比例随着模型代际更迭、算力不断增加而变化了吗？

And sorry, has that ratio changed over time as we've gone from model generation to model generation where the flops keeps increasing?

这是一个硬件参数。硬件变化有多大？从 A100 到 B100，算力大幅增加，内存带宽也大幅增加，这个比例保持得相当稳定。

So there's a hardware parameter. To what extent has the hardware changed? So from A100 to B100, the flops has increased substantially. The memory bandwidth has also increased substantially, and it has remained reasonably stable.

是的。

Yeah.

我们也可以表达这个。这是一个稀疏性参数。我甚至可能稍微换一种说法。我们来求解总的批大小。我们得到，只需把这个移到另一边，我们得到批大小需要大于大约 300 乘以稀疏性。例如，如果我有 100，比如在 DeepSeek 中我激活了 256 个专家中的 32 个。那么对于 DeepSeek 来说就是 8。明白了。好的。这实际上给出了一个与实践中非常吻合的粗略估计。通常人们会稍微大一点。他们并不想正好处于平衡点，因为现实世界的效率不如屋顶线分析所说的那么好。但可以把这个值加倍或三倍。

And we can express this one as well. This is a sparsity parameter. And I might even phrase it slightly different. Let's solve for batch size in total. We end up with, and so we're just moving this back over to the other side, we end up with batch size needs to be greater than approximately 300 times sparsity. So for example, if I have 100, like I activate in DeepSeek I activate 32 out of 256 experts. So this would be like 8 for DeepSeek. Got it. Okay. So this actually gives you a ballpark which is remarkably accurate to practice. Generally people will go a little bit larger than this. They don't really want to be exactly at the balance point because real world efficiencies aren't as good as a roofline analysis would say. But like take this and maybe double it or triple it.

好的。所以基本上每批大约 2 到 3,000 个 token。但如果把 KV 缓存考虑进去，意味着最优批大小应该更大。这就像我们求解计算时间等于内存时间时的等价关系。如果我增加更多消耗内存带宽的东西，那么用于权重加载的带宽就更少，所以我需要增加更多内存带宽，从而增加批大小。

Okay. So basically it's like 2 to 3,000 tokens per batch. But then if you included the KV cache, the implication would be that the optimal batch size should grow larger. So this is like we solve for the equivalence between when compute time is equal to memory time. If I add in more memory bandwidth, something that consumes more memory bandwidth, then I have less available for the weight loads and so I need to grow the memory bandwidth more and therefore the batch size more.

这看起来非常小，这样的批次会少于一个序列，对吧？

This seems incredibly small, like a batch this would be less than one sequence, right?

是的。好的。所以我想这是，请记住我讨论的是我正在为其生成下一个 token 的 token 数量。所以实际上是一次前向传播中的 2,000 个独特序列。

Yeah. Okay. So I guess this is, keep in mind that I'm talking about the number of tokens that I'm generating one more token for. So it's actually 2,000 unique sequences in a single forward pass.

明白了。好的。我们只是在讨论这些序列上的单次前向传播。这是不是把批次看作序列数量而不是？

Got it. Okay. We're just talking about the single forward pass on these sequences. This is like, do you think of the batch as the number of sequences rather than like?

没错。好的。酷。

That's right. Okay. Cool.

是的。

Yeah.

当我准备面试时，我经常与该领域的专家交谈。所以对于 Reiner，我与 James 的两位工程师 Clark 和 Axel 聊了聊。Clark 从事低延迟交易系统工作，他向我解释了为什么 Jane Street 使用 FPGA 来确保可预测的纳秒级延迟。你可以很容易地构建这些巨大的计算网格，它们完全按照你的需要访问 100 MB 的 SRAM，然后在几十纳秒内返回响应，这在 CPU 上基本是不可能的。他接着解释了为什么 CPU 不适合这类工作。如果时钟每 3 纳秒跳动一次，你实际上一次有多个字节的信息来做决策。这与 CPU 不同，CPU 会收集整个数据包，比如一个 1500 字节的数据包，然后说：“好的，这个数据包准备好了。给你，CPU。你现在可以开始处理了。”FPGA 允许你在数据包到达时对最早的部分做出反应，而不必等待整个数据包。我们还讨论了液冷、网络设计和其他许多事情。

When I'm prepping for interviews, I often talk to experts in the field. So for Reiner, I chatted with two of James' engineers, Clark and Axel. Clark, who works on low latency trading systems, walked me through why Jane Street uses FPGAs to make sure that they have predictable nanosecond latencies. You can just build these like giant grids of compute very easily that do exactly what you need to touch 100 megabytes of SRAM and then get your response back in tens of nanoseconds very easily, and that's basically impossible on a CPU. He then went on to explain why CPUs just wouldn't work for this kind of thing. And so if you have a clock that's going every 3 nanoseconds, you actually have several bytes of information at a time to make your decision. That's as opposed to a CPU where you'll just collect up a whole packet, let's say a 1500 byte packet, and you say, 'Okay, this packet is ready. Here you go, CPU. You can start thinking about it now.' FPGAs allow you to react to the earliest part of the packet as it arrives rather than having to wait for the full thing. We also talked about liquid cooling, network design, and many other things.

如果你有一个前沿模型并且实际在做推理，用户肯定不止 2000 个并发吧。

If you've got a Frontier model and you are actually doing inference, surely they must have more than 2,000 concurrent users.

对。

Yeah.

因为需要等整个批次填满，会不会有额外的延迟？或者说如果用户量合理，几乎不可能出现等 100 毫秒还填不满 2000 个槽位的情况？

Is there any added latency from the fact that you need to have the whole batch fill up? Or is it if you have a reasonable amount of users, it's so unlikely that you wouldn't it would not take you 100 milliseconds to fill up the next 2,000 slots.

对，思考方式是这样的，我们把它想象成火车什么时候发车。假设我选了一个要运行的批次大小。比如我选这个批次大小。顺便说一句，这个交点跟这里的交点是同一个。所以我选了这个批次大小。我知道它大概需要比如 20 毫秒，这是常见的结果。我要展示的是 GPU 上运行的时间线。它每 20 毫秒就会启动一个新批次，不管怎样。所以每个间隔是 20 毫秒，你可以把它看作火车的时刻表。每 20 毫秒发一班新车。乘客上车。如果火车满了，他们就等下一班。如果没满，火车也会发车。这对排队延迟意味着什么？最坏的情况是请求刚好在火车发车后到达。它必须等下一班火车。所以最多等 20 毫秒，然后还要等那班火车跑完。所以最坏情况延迟是 40 毫秒。

Yeah, the way to think about this, I guess we think of it as like when does the train depart as a model. So let's say I've picked a batch size that I'm going to run at. Maybe I pick, you know, this batch size. And by the way, this intersection point is the same intersection point here. So I pick this batch size. I know that it's going to take for example maybe it's something like 20 milliseconds is a common place this ends up landing. What I'm going to produce is like so this is a timeline of what is running on the GPU. It's going to start a new batch every 20 milliseconds regardless. So each of this is 20, you can think of this as a schedule for the train. A new train departs every 20 milliseconds. Any passengers who board the train. If the train is full, then they wait for the next train. If the train is not full, the train's going to go anyway. In terms of what that means for queuing latency, it means that the worst case is that a request arrives just after the train departed. It has to wait for the next train. So that's up to 20 milliseconds, and then it has to wait for that train to complete. So the worst case latency is 40.

那 20 毫秒是怎么来的？我是说经验法则，但还没完全解释清楚来源。

So how is 20 milliseconds derived? I mean rule of thumb but where it comes from is not fully explained yet.

到目前为止我们关注的是内存带宽和计算时间。看内存时，另一个考虑是我们想用尽所有内存容量。通常我们会用所有内存容量来存储权重或 KV 缓存。所以在前向传播过程中，我们想把所有内存容量读入芯片。那就是容量除以带宽，在很多代 HBM 上大约是 20 毫秒。

So far we've focused on memory bandwidth and compute time. When we look at memory, the other consideration is that we want to use all of the memory capacity we have. So generally we're going to use all that memory capacity to store the weights or the KVs. And so we just want to read, like in the time of doing a forward pass, maybe we want to read all of the memory capacity into the chip. So that is capacity divided by bandwidth, which tends to be 20 milliseconds on many different generations of HBM.

单位说得通。字节除以字节每秒。

The units make sense. You would have bytes divided by bytes per second.

对。比如在 Rubin 这一代，大概是 288 GB 除以 20 TB/s。算出来大约是 15 毫秒。让我确认一下我理解得对不对。我明白量纲分析，但它的意思是我们可以在这个时间内清空并替换 HBM。所以我们不希望 HBM 不够大，导致无法写入所有内容或读取所有内容，也不希望读写能力相比太大或太小。有两种情况。为什么我们不选一个大于 15 毫秒的延迟？如果那样做，意味着我有时间把 HBM 读两遍。顺便说一句，HBM 的访问大部分是读，不是写。几乎全是读，因为权重矩阵是只读的，KV 缓存访问也几乎全是读。所以假设我跑 30 毫秒，我可以把整个 HBM 读两遍，但这有什么意义呢？我不想把权重矩阵读两遍，也不想把 KV 读两遍。

Yeah. So for example, on the Rubin generation it is something like 288 GB divided by 20 terabytes per second. This looks like it comes out to about 15 milliseconds. Let me make sure I understand what it's saying. I mean I understand how the unit analysis works, but what it is saying is we can evacuate and replace the HBM in this amount of time. So we don't want to be in a situation where the HBM is not big enough that we're not actually able to keep write everything we want to it or take everything out of it, or we don't want to be in a situation where our ability to write back and forth is so big or so small compared. There are sort of two scenarios. Why don't we pick a latency that is bigger than 15 milliseconds? If I think what that means, it means I actually have time to read the HBM like twice. By the way, most of HBM accesses are reads, not writes. It's like almost all reads because the weight matrices are read only and then almost all of the KV cache accesses are reads. So let's say I run 30 milliseconds, I can read all of HBM twice, but what's the point of that? I don't want to read the weight matrices twice. I don't want to read the KVs twice.

对，说得通，非常说得通。好，快速问几个问题。第一，如果最优批次大小确实是 2000 左右，而且这完全取决于稀疏性，不依赖于模型大小或其他因素？

Yeah, it makes sense. Makes a ton of sense. Okay, so a couple of actually quick questions. One, if it is the case that the optimal batch size is something like 2,000 and that actually true, it's totally dependent on the sparsity. It's not dependent on the model size or anything.

我的意思是稀疏性体现在模型大小中，但除此之外，它只依赖于稀疏性，不依赖于规模。但这是一个非常有趣的结果，似乎暗示了一个问题：这种来自推理批处理的规模经济对集中化有多大推动？

I mean sparsity shows up in model size, but beyond that, it only depends on sparsity, not on scale. But that's a very interesting result and that seems to imply that you can one question is how much of a push towards centralization is it that you would have these economies of scale from inference from batching.

但似乎没那么严重。我不知道，2000 个同时用户算多吗？看起来不算多。

But it seems like it's not that big a deal. I don't know, is 2,000 users at the same time a lot? It doesn't seem like a lot.

我们可以做一点分析。实际上，你可以从用户数角度考虑，但也许更有成效的方式是从每秒 token 数角度考虑。这个批次大小对应系统每秒多少 token？每秒 token 数等于批次大小。我们运行一个包含很多 token 的批次，然后每隔一个时间间隔（比如 15 或 20 毫秒）做一次。所以最终是批次大小乘以大约 60。也就是 64 * b，大约 2000 * 64，即每秒 128k token。用更易理解的单位来说，很难推理并发用户数，但系统的全球流量是多少？看一些公告时，有时 API 提供商会吹嘘他们的流量。我记得去年 Gemini 的一些公告中，全球每秒 token 数达到数亿。所以这大约是千分之一。

We can do a bit of analysis on this. Actually, you can think of it in terms of number of users, but maybe a more productive way to think of it is in terms of number of tokens per second. So what does this batch size mean in terms of tokens per second of the system? Tokens per second is going to be equal to the batch size. We run a batch of many tokens, and then we do that every time interval, which is let's say 15 or 20 milliseconds. So this ends up being batch size itself times about 60. So like 64 * b, and this ends up being around 2,000 * 64, so like 128k tokens per second. In more digestible units, it's hard to reason about concurrent users, but what is the global traffic for a system? When you look at some of the announcements, sometimes the API providers will brag about how much traffic they have. The numbers that I remember from some announcements of Gemini last year were in the hundreds of millions of tokens per second worldwide. So this is about 1,000th of that.

但 Gemini 很大，对吧？实际上 Gemini 的千分之一已经很多了，要在规模上有竞争力，你至少需要能服务 Gemini 的千分之一。

But I mean Gemini is big, right? That's actually 1,000 of Gemini is a lot to actually be like to be competitive at scale, you need to be able to serve at least 1,000 of Gemini.

对，这很有意思。

Yeah, that's interesting.

酷。所以，稀疏性越高，需要的算力越少。而且根据这个分析，随着批次大小增大，算力最终会成为瓶颈。那么问题来了，稀疏性能走多远？也就是说，随着稀疏比增加，活跃参数相对于总参数越来越少，模型性能下降多少？下降速度是否快于通过增加稀疏因子节省的算力？

Cool. So, the more sparsity you have, the less compute you need. And it does seem that as batch sizes get bigger, compute ends up being the bottleneck. According to this analysis. So then the question is how far can you take sparsity? That is to say as the sparsity ratio increases, as you have fewer and fewer active parameters relative to total parameters, how much is performance of the model degrading and is it degrading faster than you're saving compute by increasing the sparsity factor.

对，是模型的质量性能，而不是速度。

Yeah. So performance quality of the model rather than speed of the model. Yeah.

所以不幸的是，我们无法从分析上回答这个问题。这是一个关于模型质量的实证问题。我最多能找一篇论文，用实证来回答。

So unfortunately we're not able to answer that analytically. That is an empirical question of model quality. Best I can do is pull up a paper and answer that empirically.

对。

Yeah.

呃，我们现在要按论文来讨论，还是这样讲也行？

Uh, should we follow the paper now, or does it make sense?

是的。这篇论文是《路由语言模型的统一规律》。它现在算是有点老的论文了，但他们做的一件事是：如果我不断增加稀疏度，模型质量会受什么影响？这个答案对混合专家模型的具体选择非常敏感。混合专家模型已经存在很久了，我想可能早在 2017 年就有了。但技术已经发生了很大变化。DeepSeek 的混合专家模型是其工作方式的一次重大变革。还有更早的论文，比如 GShard 和 Switch Transformer。所以实际的实证结果取决于所有这些。但这里展示的一种较老的技术中，你可以看到：如果我在某个大小下保持活跃参数数量不变，然后增加稀疏度——他们称之为专家数量——质量会持续提升。然后如果你想象从 1.3B 密集模型画一条水平线过去，你会发现，例如在这个案例中，64 专家、3.7 亿活跃参数的模型与密集的 13 亿参数模型质量相当。所以从某种意义上说，回报其实并不惊人——你需要将总参数增加 100 倍，才能获得相当于 10 倍活跃参数的效果。

Yeah. So this paper is "Unified Laws for Routed Language Models." It's a somewhat old paper by this stage, but one of the things they did is looked at: if I keep increasing sparsity, what is the model quality impact? This answer is very sensitive to the actual choice of mixture of experts. Mixture of experts has been around for a really long time. I think it was maybe even back in 2017. But the techniques have changed a lot. DeepSeek mixture of experts was a big change in how it worked. There have been older papers like GShard and Switch Transformer. So the actual empirical results are going to depend on all of that. But on one of the older techniques shown here, you can see: if I hold constant the number of active parameters at a certain size and then I increase the sparsity—which they call expert count here—the quality keeps increasing. And then if you imagine drawing a horizontal line from 1.3B dense across, you end up seeing that, for example, in this case, the 64-expert, 370 million activated parameters model is as good as a dense 1.3 billion model. So in some sense, it's actually not amazing returns—you need to increase total parameters 100-fold to get the equivalent of 10x as many active parameters.

是的，实际上甚至更甚——对，参数数量大幅增加，但收益却 modest……

Yeah, I mean actually even more so—yeah, it's a huge increase in parameter count for a modest increase in...

对，所以在这个案例中实际上是——是多少？4 倍？

Yeah, so in this case actually it's—what is it? 4x?

64 倍换 4 倍。是的。

64x for 4x. Yeah.

所以，虽然我认为，如果你增加稀疏度，确实能节省计算时间，但天真地看，似乎会觉得“哦，这个权衡值得做”。但如果这样——你每次将稀疏度翻倍，计算量减少 2 倍，而参数总量增加 8 倍。那么这是好是坏？实际上，即使从内存的角度来看，请记住，你正在将这部分内存读取量翻倍，而这部分可以通过批处理来摊销，所以只要继续运行更大的批次大小就行。从我们这里所做的分析来看，这纯粹是赢。继续做下去。基本上，一直做到你没有更多可用用户为止。

So while it is true, I guess, that you get this benefit of being able to economize on your compute time if you increase sparsity, naively it would seem like, "Oh, that's a trade-off worth making." But if this is—you're decreasing this by 2x and then having this go up by 8x every time you double sparsity. So is that good or bad? Actually, even from a memory point of view, keep in mind you are doubling this portion of the memory fetches which is amortized by batch, and so just keep running a larger batch size. From the point of view of the analysis we've done here, this is pure win. Keep doing it. Keep doing it until you run out of available users, basically.

嗯。所以实际上存在这样一种等价关系——如果我想变得更稀疏，或者如果我有大量用户，我可以使用一个更稀疏的模型。从这个角度来看，这是一个合理的权衡。这里出现的另一个权衡是，它还会消耗内存容量，我们这里只考虑了内存带宽，但它也会消耗内存容量。

Mhm. So there's actually this equivalence between—if I want to go sparse, or if I have a lot of users, I can go to a much sparser model. So from that point of view, it's a reasonable trade-off. The other trade-off that shows up here is that it also consumes memory capacity, which we've only reasoned about memory bandwidth here, but it also consumes memory capacity.

所以让我确认一下我理解了。你是说我们想要更大——我们想要花更少的时间计算，因此我们增加稀疏度。为了实现这一点，我们需要更大的批次大小，这意味着我们需要更多的内存容量，才能有更高的稀疏度。

So let me just make sure I understood. You're saying we want bigger—we want to spend less time computing, therefore we do more sparsity. To make that work, we need bigger batch sizes, which means we need more memory capacity, to have more sparsity.

是的。

Yeah.

所以，我的意思是，也许现在是个好时机，来实际讨论一下混合专家层通常是如何在一排 GPU 或类似设备上布局的。

So I mean, maybe this would be a good point to actually talk about how a mixture of experts layer is typically laid out on a rack of GPUs or something like that.

是的。对，有道理。

Yeah. Yeah, makes sense.

好的。我们说到哪了？

Yeah. Where were we?

呃，稀疏混合专家模型。也许是如何在 GPU 上布局。

Uh, sparse mixture of experts. Maybe how we lay that out on a GPU.

是的。

Yeah.

那么，我们先放大看混合专家层，画一下它的样子。通常，我们会有一个某种路由层，它决定将 token 路由到哪些专家。所以，token 从这里进来，经过路由层，然后我们有一堆不同的专家。我再画几个排成一排。然后路由层会做出决定：我要路由到哪些专家？而且只会是其中的一小部分。也许是 32 分之一。所以它可能会决定路由到这个、这个和这个。这些专家——每个专家本身就是一个普通的 MLP。它有一个升维投影，然后是一个降维投影，中间有一个非线性层。最后我们做逆操作。所以，之前我们在这里广播出去的东西，现在要收回来并求和。像这样收回来。然后最后是我们的残差连接。所以 token 也会经过这里，并加到该层的结果上。所以这是一个正常的层。我想讨论的是，这如何映射到 GPU 机架上，以及这对通信意味着什么，因为我认为这将开始揭示我们速度的一些限制。

So, let's zoom in on the mixture of experts layer first and sort of draw what that looks like. So, we typically will have some kind of a router layer, which is making the decision of where we route the tokens to the experts. So, we get tokens coming in here. They go through a router layer, and then we have a bunch of different experts. I'll draw a few more to line some up. And then the router will make a decision: which experts am I going to route to? And it'll be a small fraction of them. Maybe one in 32. So maybe it'll make a decision to route to this one, maybe this one, and maybe this one. These experts—each expert itself is a normal MLP. It has an up projection and then a down projection with a nonlinearity in between. And then finally we sort of do the inverse operation. So where we were broadcasting things out here, we're going to bring them back in and sum them up. So bringing them in like this. And then finally we have our residual connection. So that the token is also passed through here and it gets added to the result of the layer. So this is a normal layer. What I want to talk through is how this is mapped to a GPU rack and what this means for communication, because I think this will start to show some of the limits of how fast we can go.

是的。

Yeah.

所以这里的标准做法——也是最佳方案——是使用专家并行。这意味着不同的专家放在不同的 GPU 上。所以如果我们拿 DeepSeek 模型来说，他们有 256 个专家。假设我们想在 Blackwell 机架上运行它。那么有 72 个 GPU。我们有一个整除问题——这不是 2 的幂。所以我们简化一下，假设我们只用其中 64 个。忽略另外 8 个——没什么大不了的。这样每个 GPU 有 4 个专家——非常简单。为了画图方便，我实际上就说每个 GPU 有 2 个专家，所以我们最终只是把——这些是 GPU 边界——每对专家都在自己的 GPU 上。然后我们可以看看通信成本：我们有一些专家存储在这里，一些 token 集中存储在这里；它们被路由到所有这些专家。所以这里付出了一些通信成本。输出端也有同样的通信成本。然后希望这不会成为通信瓶颈。那么，这里的流量模式是什么？这里的流量模式是，实际上任何 GPU 都会与任何其他 GPU 通信，具体取决于模型做出的决策。所以这是一种全对全的流量模式。

So the standard practice here—and it is the best solution—is to use expert parallelism. That means different experts go on different GPUs. So if we take something like a DeepSeek model, they have 256 experts. Let's say we want to run that on a Blackwell rack. So there are 72 GPUs. We have a divisibility problem—this is not a power of two. So we'll just simplify and say we're only going to use 64 of them. Just ignore the other eight—it's not a big deal. And so we have four experts per GPU—very simple. For the sake of the diagram, I'll actually just say let's say we have two experts per GPU, so we end up just putting—these are the GPU boundaries—every pair of experts is on its own GPU. And then we can look at the communication cost: we had some experts stored, some tokens stored centrally here; they get routed to all of these experts. And so there's some communication cost paid here. There's the same communication cost paid on the output. And then the hope is that this does not become communication-bound. Now, what is the traffic pattern here? The traffic pattern here is that any GPU, in fact, will be talking to any other GPU, depending on the decisions made by the model. So this is an all-to-all traffic pattern.

所以当你说任何 GPU 在场——

So when you say any GPU in the presence—

是的。

Yeah.

——路由层不止一个 GPU。对，路由层。所以我把它画成一个路由器。实际上，你会有多个路由器副本，所以实际上路由器数量与 GPU 数量相同，与传入流量对应。

—the router is more than one GPU. Yeah, the router. So I drew this as one router. In reality, you would actually have many copies of the router, and so you would have as many routers as GPUs, in fact, as the incoming traffic.

是的。所以这些是 64 个 GPU。这些是 64 个 GPU。实际上它们是相同的 GPU。我们只是把它们分开画，因为它们服务于不同的目的。所以在这一点上，任何 GPU 都可以向任何其他 GPU 发送数据。这种全对全的通信模式——Blackwell 机架的配置方式——与模型实际想要做的通信模式完美契合。然而，如果你觉得也许我想做——也许一个机架太慢了，我想用两个机架。

Yeah. So these are the 64 GPUs. These are 64 GPUs. It's actually the same GPUs. We just draw them as separate because they're serving different purposes. So at this point, any GPU can be sending to any other GPU. This all-to-all pattern of communication that shows up—how the Blackwell racks are configured—is a perfect fit for the communication pattern that the model actually wants to do. However, if you think maybe I want to do like—maybe one rack is too slow and I want to do two racks.

然后我面临这样一个挑战：可能我在这里画了一个机架边界。实际上，两个机架内的所有 GPU 之间不再有全互联通信。机架间的通信最终成为一个重大瓶颈。这里的基本问题是，一个机架实际上限制了你能做的专家层的大小。这在一定程度上推动了互联域越来越大。

Then I have this challenge that maybe I've got some rack boundary drawn outside here like this. I no longer have all-to-all communication between all the GPUs in two racks. The rack-to-rack communication ends up being a substantial bottleneck. The fundamental thing here is that one rack actually bounds the size of an expert layer you can do. This has been part of what's been driving towards larger and larger interconnect domains.

在我们继续之前，你可能需要解释一下机架到底是什么，机架间和机架内的带宽差异，以及机架内和机架外通信的全互联与非全互联特性。

Before we go on, it may be worth you explaining what exactly a rack is, the differences in bandwidth between a rack and within a rack, and the all versus not-all nature of communication within versus outside.

这是英伟达、谷歌以及包括我们在内的其他公司开始出现很大差异的地方。通常，机架是一个物理结构。它有几米高，一两米宽，具体取决于配置。它容纳一定数量的 GPU 或 XPU，通常是 64 个左右。限制其尺寸的因素是供电、重量和散热能力。由于这些物理限制，很多情况下它就是这个尺寸。当我部署一个数据中心时，可能有数千个这样的机架。所以我有一个高大的机架，里面有很多 GPU，然后在旁边再放一个机架。

This is a place where it starts to be very different between Nvidia and Google and others including us. Generally, a rack is a physical structure. It's a few meters tall, a meter or two wide, depending on configuration. It stores some number of GPUs or XPUs, typically about 64. What constrains it being a certain size is power delivery, weight, and cooling ability. It ends up being about this size in many cases because of these physical constraints. When I deploy a data center, I may have thousands of these racks. So I have one tall rack with a bunch of GPUs, and then I put another rack next to it.

你说得好像很简单。

You make it sound so easy.

没错，我就直接放进去。在英伟达的情况下，通信拓扑是：他们把 GPU 放在机架外侧，把交换机放在机架内侧。这里有一组交换机——这些是 NV 交换机。然后他们跑很多线缆。每个 GPU 都有线缆连接到中间的交换机。所以每个 GPU 都连接到交换机，交换机再连接到所有 GPU。因此所有 GPU 之间只需两跳就能通信：到交换机，再到另一个 GPU。当我要离开机架时，我需要走另一条路径。GPU 还有一个慢得多的连接，通常慢八倍左右。我在这里画的绿色线是 NVL 链路。更一般地，它被称为扩展网络（scale-up network）。通常你还会有一个扩展网络（scale-out network），用于连接到某个数据中心交换机。所有 GPU 都会有一些连接到某个数据中心交换机的链路。这个扩展网络（scale-out）通常慢八倍。如果你想跨两个机架部署一个混合专家层，挑战在于：这里一半的 GPU 想要和这里的 GPU 通信。平均而言，一半的 token 想要留在机架内——这很好，它们可以使用快速的扩展网络——但另一半 token 想要离开机架去另一个机架，这就没那么好了。它们需要使用慢得多的网络，这成为全互联模式的瓶颈。另一种选择是这里放一个大交换机，把所有东西连接到一个更大的交换机，把两个机架合并起来。这方面有很多想法，但总的来说，之所以有这种交换机层级而不是一个大交换机，是为了管理线缆拥塞。你需要跑大量的线缆。

Right. I just drop them in. In Nvidia's case, the communication topology is that they put the GPUs on the outside of the rack and the switches on the inside. There's a set of switches in here—these are the NV switches. Then they run a bunch of cables. Every single GPU has cables going to the switches in the middle. So every GPU goes to the switches, and the switches have connections to all GPUs. So all GPUs can talk to all other GPUs in just two hops: going to the switch, then to the other GPU. When I want to leave the rack, I end up going via a different path. The GPUs also have a much slower connectivity, typically about eight times slower. The green I drew here in GPU cases is the NVL link. More generally, it's called the scale-up network. You will typically also have a scale-out network that allows you to connect to some data center switch. All GPUs will have some connectivity up to some data center switch somewhere. This scale-out tends to be about eight times slower. The challenge if you want to lay out a mixture of expert layer across two racks is that half of the GPUs here will want to talk to the GPUs here. On average, half of the tokens want to go inside the rack—that's great, they can use the fast scale-up network—but half want to leave the rack and go to the other rack, which is not as good. They need to use a much slower network, and that becomes the bottleneck on the all-to-all pattern. A different choice would be to have a big switch here and connect everything to some much bigger switch that combines the two racks. There are many ideas in this direction, but in general, the reason you have this hierarchy of switches rather than one big switch is to manage cabling congestion. You just need to run a large number of cables.

你刚才问的问题基本上就是为什么不能有更大的扩展网络？为什么不能有一百万个芯片在扩展网络里？是什么让英伟达从 Hopper（8 个）到 Blackwell（72 个），再到现在的 Reuben（大概 500 个）？是什么促成了这种变化？

Is that question you just asked basically why isn't it a bigger scale-up? Why not just have like a million chips in scale-up? What has changed that has allowed Nvidia to go from Hopper (8) to Blackwell (72) and now Reuben will be 500 or something? What has allowed that to happen?

从 Hopper 到 Blackwell，主要只是从以托盘为形态因素转向以机架为形态因素的决定。这是一个产品决策，没有实质性的技术障碍。从 64 个到 500 个左右——这里面有点 Jensen 数学，但至少有一个真正的 4 倍增长，这来自于更复杂、更困难的机架设计。这实际上是新的物理设计，以容纳更多线缆。

From Hopper to Blackwell, it's mostly just a decision to switch from trays as the form factor to racks as the form factor. That's a product decision. There wasn't a substantial technical barrier there. Switching from 64 to 500 or so—there's a bit of Jensen math there, but there is at least a genuine 4x increase, which is coming from a much more complicated and difficult rack design. That is actually new physical design to run more cables.

线缆的复杂性只是弄清楚哪根线缆连接到哪个、或者哪个信号从哪来的成本吗？

And the cable complication is just the cost of figuring out which cable hops to which or which signal goes from?

我们来放大看看线缆密度。我再画一次这个图。我们有一个更干净的版本和更大的版本。假设中间有一些交换机。假设我先从每侧两个 GPU 或每侧两个托盘开始。假设每个托盘需要引出两根线缆。所以我物理上跑一些垂直的线缆，像这样连接到交换机。现在如果我想把机架内的 GPU 数量翻倍，我就需要跑两倍密度的线缆。所以我也需要跑这些线缆。

Let's zoom in on this and look at the wire density. I'll draw this diagram just once more. We have a bit of a cleaner version to work with and a larger version. Let's say I have some switches in the middle. Let's say I'm going to start with just two GPUs on each side or two trays of GPUs on each side. Let's say maybe each tray wants to have two cables coming out of it. So I physically run vertical cables that look like this running to the switches. Now if I want to double the number of GPUs in a rack, I need to run literally twice the density of cables. So I need to run these as well.

一个非常外行的问题，但如果你看一个物理数据中心，机架里面似乎有很多空间。我不知道。只是线缆真的很大……

Extremely question, but if you look at a physical data center, seems like there's a lot of space within a rack. I don't know. Just like the cables are really big and...

机架外面有空间。但机架内部，随着它们越来越优化，这些机架非常紧凑。从托盘到机架以及机架背板的连接器密度很高。背板本身密度非常高。还有其他物理限制，包括线缆的弯曲半径——你不想折断它们。实际上就是放置线缆的物理空间在限制它。

There is space outside the rack. Inside the rack, as they become more optimized, these racks are very tight. There's connector density going from the tray into the rack and the rack's backplane. The backplane itself has a really high density. There are other physical constraints including bend radius of cables—you don't want to snap them. It's literally the physical space to put a cable that's constraining it.

我完全不知道。有意思。这似乎令人惊讶，机架这么大，我们却不能塞进更多线缆。

I had no idea. Interesting. That seems surprising that the rack is so big and we just can't stuff more cables in there.

所以，机架设计不是我的专长，但我和别人聊的时候，他们面临的限制是多种因素的组合。那么你优化的主要物理因素是什么？空间、机架的重量——实际上它非常重，所以你需要足够的金属来防止下垂和倒塌，但加更多金属又会更重。然后是电力和冷却。所有这些因素都在相互竞争，现代机架正在将这些因素推向非常极端的物理极限。

So I mean rack design is not my expertise but like when I talk to folks and what are the constraints they're up against, it's a combination of... So what are the big physical things you're optimizing for? Space, weight of the rack — like it's actually really heavy, so you need enough metal to not sag and fall, but then you add more metal and it's heavier. And then power and cooling. All of those are competing, and modern racks are pushing all of those to very extreme physical limits.

深度工作本质上是很令人反感的。所以即使是像 Slack 和电子邮件这样看似工作的事情，也很容易分散注意力。所以我经常希望自己能关掉互联网，但如果我在准备采访，即使手头有论文和书籍，能够与 LLM 来回交流仍然非常有用，这样我可以分解概念并跟进研究。Google 的新 Gemma 4 是第一个让我拥有这种完全断连的专注机器的开放模型。它小到可以在我的笔记本电脑上运行，但又足够好，实际上很有用。所以为了准备这一集，我下载了 Reer Scaling 的书并关掉了互联网。我让 Gemma 帮助我理解材料并回答我的问题。如果你想要一个可以在笔记本电脑甚至手机上本地运行的 LLM，你应该看看 Gemma 4。

Deep work is by its nature quite aversive. So even things which seem like work, like Slack and email, can be easy ways to distract yourself. So I often wish that I could just turn the internet off, but if I'm prepping for an interview, even if I have the papers and books on hand, it's still super useful to be able to do a back and forth with an LLM so I can break down concepts and research follow-ups. Google's new Gemma 4 is the first open model that allows me to have this kind of fully disconnected focus machine. It's small enough to run on my laptop, but good enough to actually be useful. So to prep for this episode, I downloaded Reer Scaling book and shut off the internet. I was able to have Gemma help me understand the material and answer my questions. If you want an LLM that you can run locally on your laptop or even your phone, you should check out Gemma 4.

GPT-4 是什么时候发布的？是 2022 年还是 2023 年？

When was GPT-4 released again? It was 2022 or 2023.

三年。好的。

Three. Okay.

而且据传它有超过一万亿参数，似乎直到现在，也就是最近 6 个月内，才有模型发布，其参数数量显著超过 3 年前发布的模型。

And it was rumored to be over one trillion parameters, and it seems like only now and within the last 6 months have models been getting released that are significantly more parameters than a model released 3 years ago.

是的。按理说这段时间应该已经有这种 Scaling 了。原因是不是我们只是在等待有足够内存的机架来容纳五万亿参数的模型及其 KV 缓存，以便为足够多的用户提供完整的序列，或者如果你在做 RL，也有类似的考虑，即实际保存你试图解决的所有问题的 KV 缓存。

Yeah. When supposedly there should have been this scaling in the meantime. Is the reason that we were just waiting for racks with enough memory to hold the five trillion parameter model along with its KV cache for enough users for a full... for a lot of sequences, or RL if you're doing RL — kind of a similar consideration of actually holding the KV cache for all the batch of problems you're trying to solve.

所以如果你看 Hopper，有八个 Hopper，我认为到 2022 年那是 640 GB。

So if you look at Hopper, you had eight Hoppers and I think that's 640 gigabytes as of 2022.

是的。

Yeah.

而 Blackwell 最终部署了……什么时候？2020 年？非常近，可能是去年。

With Blackwell finally, which was deployed... what, 2020? Very recently, maybe last year.

去年。

Last year.

你终于有了一个大约 10-20 TB 的 scale-up。

You finally have a scale-up with on the order of like 10-20 terabytes.

嗯。

Mhm.

这足够容纳一个 5T 模型加上 KV 缓存。

Which is enough for like a 5T model plus KV cache.

是的。在更大的 scale-up 域中部署是一个巨大的解锁。

Yeah. Deploying in larger scale-up domains is a huge unlock.

我在这里画了 Nvidia Blackwell 部署的示意图。Google 的部署实际上有非常大的 scale-up 域……这也解释了为什么 Gemini 似乎领先。比如，Gemini 2.5 成功了吗？或者只是看起来 Gemini 的预训练比其他一些实验室更成功。由于我当时不在那里，我不确定有多少来自成功部署更高的稀疏比率，这可能……我的意思是，有一大堆实际的建模问题，比如具体如何做混合专家。我们看到 DeepSeek 的混合专家声称实际上激活了更多专家，但更细粒度的专家是一个重大创新。

I've drawn here the sort of Nvidia Blackwell deployment. The Google deployment has actually had very large scale domains for... and that also explains why Gemini seemed to be ahead. Like, was Gemini 2.5 successful? Or it just seems like Gemini had that successful pre-train for longer than some of the other labs. Not having been there at the time, I'm not sure how much is coming from successfully deploying higher sparsity ratios, which could be... I mean, there's a whole bunch of actual modeling things, like specifically how do you do the mixture of experts. We've seen DeepSeek's mixture of experts has said actually activate more experts but finer-grained experts was a big innovation.

我确信在模型架构和训练数据上还有许多其他创新。很难把它们全部理清。但从你能做什么的限制来看：正如我们所见，活跃参数受限于算力成本，而总参数受限于 scale-up 的大小。

I'm sure that there are many other innovations on the model architecture as well as on the training data. It's kind of hard to disentangle all of them. But what shows up in terms of the limits of what you can do: the active parameters, as we saw, is limited by the compute cost, and then the total parameters is limited by the scale-up size.

是的。当你在单个 scale-up 域内操作时，这是否是专门针对前向或后向，或者专门针对预填充与解码的考虑？或者无论你做什么类型的工作负载，无论是预训练运行、RL 生成还是用户推理，是否都更倾向于始终在 scale-up 内？

Yep. When you're operating within a single scale-up domain, is that a consideration specifically for either forward or backward, or specifically for prefill versus decode? Or is it preferred to always be within a scale-up whatever kind of workload you have, whether you're doing a pre-training run, or RL generation, or inference for users?

是的，非常有趣。那么，为了回答这个问题，我们需要讨论通信模式。我们已经讨论了混合专家的通信模式，即全对全。全对全非常强烈地倾向于完全连接，就像我们这里展示的那样，并且它倾向于在一个机架内。除了专家并行之外，还有其他类型的并行。在文献中，有张量并行。随着向更小专家的趋势，这变得不那么相关了，所以我们可以忽略它。但我们可用的另外两种是数据并行和流水线并行。它们实际上更适合使用多个机架。所以我们专门关注流水线并行。这是一层。我上面还有一百多层。例如，我可以决定在这一点切换到不同的机架，更换机架。那么，这会不会成为通信瓶颈？我们实际上可以计算出何时会成为通信瓶颈。但在代数求解之前，让我们先可视化并勾勒出路径。所以我们会有很多……这是另一层，这里还有另一层，依此类推。假设我在这里更换机架，然后在若干层之后我也在这里更换机架。那么，我们用来判断在更换机架的点是否存在通信瓶颈的方法，是比较 scale-out 带宽需求和 scale-up 带宽需求。

Yeah, really interesting. So okay, to answer that question, we're going to need to talk about the communication patterns. So we've talked about the mixture of experts communication pattern that is this all-to-all. All-to-all very strongly favors full connectivity, which is what we've kind of just shown here, and it favors being within one rack. There are other kinds of parallelism besides expert parallelism, which we just showed here. In the literature, there is tensor parallelism. With a trend towards smaller experts, this has become much less relevant, so we can ignore that. But the other two things that we have available are data parallelism and pipeline parallelism. And they are actually a much better fit for using multiple racks. So let's focus on pipeline parallelism specifically. This is one layer. I'm going to have like a hundred more layers up above. I could decide at this point, for example, to move to a different rack, change rack. Now, is that going to become a communication bottleneck? We can actually just solve for when this becomes a communication bottleneck. But before we do that algebraically, let's just sort of visualize it out and sketch the path. So we're going to have a bunch... this is another layer, and we're going to have another layer here, and so on. So let's say I change rack here, and then some number of layers later I change rack here as well. So our methodology that we're going to use to determine whether we have a communication bottleneck at this point where we change rack is we're going to compare the scale-out bandwidth requirements to the scale-up bandwidth requirements.

嗯。

Mhm.

那么让我们试试。提示是这里有很多发送——我们在这里发送很多东西——而这里我们只发送一个东西，而且可能还要多次发送。这就是区别所在。

So let's try this. And the hint is going to be that there's a lot more sends here — like we're sending many things here — whereas we're only sending one thing here, and then we're also maybe doing it many times. That's going to be what makes the difference.

我能猜一下吗？只是出于好奇，看看我是否真的理解了。看起来你在这里把 batch size 发送到机架里。

Can I try to guess just out of curiosity to see if I'm actually understanding? It seems like you're sending batch size into the rack in here.

是的。

Yes.

但机架内的通信是 batch size 乘以 GPU 数量。

But the communication within the rack is sort of batch size times number of GPUs.

是的。所以是激活的 GPU 数量，对吧？所以我根本不发送到这个 GPU，对吧？所以从 1 到这里的 3 倍，有一个爆炸。

Yeah. So number of activated GPUs, right? So I don't send to this GPU at all, right? So there's an explosion from one to like it's three times larger here in this diagram.

是的。

Yeah.

关键是我根本不需要发送到这个 GPU。所以这是一个很大的节省。

The key thing is that I didn't even need to send to this GPU at all. And so that's a big saving.

我明白了。是的。

I see. Yeah.

我们来聊聊 Scaling 的瓶颈：Scale-up 在多大程度上比 Scale-out 更慢。直接切入正题，我们看 Scale-up 时间与 Scale-out 时间的比值。首先，Scale-up 通常比 Scale-out 快 8 倍。所以基线情况下，如果带宽相同，比值是 1/8，这来自带宽差异。但数据发送量会有膨胀：一个 token 进来，在 DeepSea 场景下会被路由到 32 或 16 个专家，即激活专家数。这还会在多个层上重复。假设我跑两层，那么还有每阶段层数的倍数。整个还要乘以 2……对，还有一个因子，谢谢。我们希望 Scale-up 时间大于 Scale-out 时间，因为 Scale-up 更宝贵。所以希望这个比值大于等于 1。这并不难，只需要克服 8 倍因子。我们需要这三个量的乘积大于 8。通常激活专家数就很大，可能本身就有 8。然后我们可以大幅增加每阶段层数，直到满足条件。

So we're going to talk through how much more, what is the slowdown, to what extent is scale-up a bottleneck over scale-out. So we will directly jump to the ratio of the time spent on scale-up over the time spent on scale-out. So this is the quantity we're talking about. The first consideration is that scale-up is generally eight times faster than scale-out. So at a baseline, if the bandwidths were the same, we would have one over eight, which is coming from bandwidth. But then we have some amount of expansion in how much data we're sending. So if one token comes in, then this token gets routed to, in the DeepSea case, maybe 32 experts or 16 experts, some number of experts. So this is the number of activated experts. And then this also applies on multiple different layers. So maybe I'm going to run two layers. So there's also a multiple times number of layers per stage. And you need to multiply the whole thing by two for the... Yes. Yes. And there's a factor. Thank you. So what we would like is for the scale-up time to be greater than the scale-out time, because scale-up time is the more important and precious resource. So we want this number to be greater than or equal to one. And this really doesn't seem hard; we just have a factor of eight that we need to overcome. So we need the product of these three things to be bigger than eight. Typically we have a fairly large number of activated experts, could be eight by itself. And then we can increase the number of layers per stage a lot until we satisfy this.

明白了。

I see.

最终的结果是，我可以有一个完整的机架流水线：一个机架做一层，然后移到下一个机架做下一层，再移到下一个机架。有意思的是，实践中最好的并行策略在物理上竟然与模型架构一致。不是什么高深莫测的东西，就是：我们有专家，就把它们放在不同 GPU 上；我们有不同层，就把它们放在不同机架上。物理切分与模型架构匹配，这不有趣吗？

So what this ends up looking like is that I can in fact have an entire pipeline of racks where one rack does one layer and then I move on to the next rack and I do another layer and then I move on to the next rack. I can do another layer. It's interesting to me that the best parallelism strategy in practice ends up being one which physically resembles the actual architecture. It's not some galaxy brain thing, you know, it's like, oh, we have experts, we're going to put them on different GPUs. Oh, we have different layers. We're just going to put them on different racks. Isn't that interesting that the physical and the model architecture match, like the cutting matches the model architecture.

没错。本来可能更复杂，比如张量并行之类的。

Yeah, exactly. I mean it could have been something wackier with tensor parallelism and whatever.

对。所以思考方式是：模型在哪些维度上扩展？层数、模型维度、DFF 维度、专家数。每个维度你都可以选择切分。如果这些数字足够大，切分就有收益。我们选了其中两个。另外两个在典型模型规模下没有收益。

Yeah. So I think a way to think of it is: what are all the different dimensions in which a model is scaled up? It is scaled up by layers, by the model dimension, by the DFF dimension, by the number of experts. Every single one of those numbers you can choose to cut along. And if those numbers are big enough, it eventually becomes profitable to go there. And we have selected two of them. The other two, in the way typical models are sized, are not profitable.

Ilya 有个演讲说我们现在知道不要用流水线并行。Horvath 也给我们讲过各种并行方式。他说流水线并行除了气泡问题，还会带来架构约束，比如 Kaiming 的残差连接中注意力会回溯几层，这样实现起来很困难。

So there's a talk by Ilya where he says today we know not to do pipeline parallelism, and Horvath gave my friends and me a lecture on these different kinds of parallelisms. He said the problem with pipeline parallelism is that other than the bubbles, it creates these architectural constraints, yes, on like Kaiming for example has these residuals where attention attends to a few layers back, and that becomes hard to implement in this way.

对。我们甚至没有完全说清楚流水线带来的好处是什么。

Yeah. And I guess we didn't really fully articulate even what is the benefit that we're getting from pipelining.

嗯。

Yeah.

这些复杂性是真实的。流水线非常麻烦，但它确实有好处。然后你可以权衡利弊。最大的好处是：推理中有些收益，训练中可能更大。推理中我们节省了什么？内存时间还是计算时间？其实不是。我们只是把内存时间从一个芯片移到另一个芯片，或从一个机架移到另一个机架。运行时没有实际收益。但我们节省的是内存容量，即每个机架使用的内存量。如果机架内存是瓶颈，就会限制速度。流水线能大幅缓解这个瓶颈。

And so these complexities are real. Pipelining is a massive hassle. But it does give you some benefits. And then you can decide whether those benefits are worth the costs. The biggest benefit that shows up: it can have some benefits in inference, maybe bigger benefits in training. In inference, what are we saving on? Memory time or compute time? Not really. We're just moving the memory time from one chip to another chip, or one rack to a different rack. There's no actual benefit in runtime. However, what we are saving on is that the memory capacity, the amount of memory used per rack. If we think that the memory in a rack is a bottleneck, then there's a constraint on how fast we can go. Pipelining allows us to massively reduce that bottleneck.

但与此相反的含义是，采访前我和 Jane Street 的 GPU 性能工程师 Axel 聊过，他解释说：做流水线必须用微批次而不是全批次。如果用微批次，就无法在所有用户或序列上分摊权重加载。对。所以正面是你不需要那么多内存，负面是无法在所有用户上分摊权重加载。也许值得解释为什么必须用微批次，因为你不能……

I guess the opposite connotation to this, which actually before this interview I was chatting with Axel, a GPU performance engineer at Jane Street, and he was explaining: to do pipelining you had to do micro batches rather than full batches. And if you do micro batches, then you're by definition not able to amortize the weight loading across all the users or all the sequences. That's right. So the positive connotation is you don't have to use as much memory. The negative connotation is that we can't amortize loading the weights across all those users. Maybe it's worth explaining why you had to do microbatches because you can't...

我们来画流水线气泡。好，为什么流水线并行中会出现微批次？先聚焦推理，问题更简单。画一下：横轴是时间，纵轴是机架。假设有四个机架。一次推理会依次经过这四个机架，比如这样。这是推理 0，以某个批次大小运行，依次经过所有流水线阶段。如果我们在推理 0 之后直接运行推理 1，那显然浪费巨大，对吧？每个机架有四分之三的时间空闲。所以我们不这样。我们在推理 0 一结束就尽快开始推理 1。如果不填充这个空隙，我们就称之为流水线气泡。

So we draw the pipeline bubble. Okay. So why do we do microbatching that shows up in pipeline parallelism? I'll focus on inference first; it's a slightly simpler problem. So I'm going to draw: this is time, and this is which rack we're on. So the idea is that maybe I'll have four racks. So I've got an inference that is going to step through these four racks in some time like this. So this is inference number zero. It runs at a certain batch size and steps through all the pipeline stages like this. Now if we were to say we're going to run inference number one here, this is clearly a massive waste, right? Like three-quarters of the time each of the racks is doing nothing. So we don't actually run inference one here. We run it as soon as we can, which is immediately after inference zero finishes like this. So if we hadn't filled this in, we would call this the pipeline bubble.

当我把它画在推理上下文中时，我们只做前向传播，很明显，为什么要做蠢事呢？

When I've drawn it in this inference context where we're only going in a forward pass, it's obvious like why would you do the stupid thing?

但在训练上下文中，可能不那么明显。但在推理上下文中，做这个改变是很自然的。

But in a training context, it's maybe less obvious. But in the inference context, it's sort of really natural to make this change.

哦，有意思。所以，这有点明显，但微批次和批次之间的区别在推理中完全无关紧要，因为你可以随便调用任何东西。

Oh, interesting. So, this sort of obvious, but the difference between microbatch and batch doesn't matter at all in inference because you can just call whatever you want.

是的，它只在训练中重要，因为存在一个最优批次大小。

Yeah, it only matters in training because there is an optimal batch size.

是的。在反向传播之前，你需要累积该批次中的所有序列。如果你想在训练中做流水线以避免那个气泡，你需要……我们画一下训练图吧？

Yes. And before you do the backward step, you want to have accumulated all the sequences in that batch. And if you want to do pipeline in training in order to avoid that bubble, you need to... should we draw the training diagram?

好的，我们来做。这是推理图，我把它标为四，以免显示错误。那么训练也类似。现在我们有一个前向传播，但在某个阶段，我们需要切换到反向传播。所以我们会做若干批次的前向传播，然后一次性全部切换到反向传播。推理部分在这里是一样的，但我们在这一点上做一个硬停止，然后让所有人切换到反向传播。值得澄清的是，这个硬停止的原因是你想一次完成整个批次的反向步骤。

Yeah, let's do that. So this is the inference diagram and I'll call this four just so we don't have the wrong thing showing up there. So let's do the same thing for training. Now we've got a forward pass, but at some stage we're going to have to transition to a backward pass. So we'll do some number of batches in the forward pass and then we're going to transition to the backward pass for everyone all in one go. So the inference part is the same here but then we do a hard stop at this point and then transition everyone to backward pass. It may be worth clarifying the reason there is that hard stop is because you want to do a whole batch at once for the backward step.

然后这个批次有一个最优大小。

And then there is an optimal size for how big that batch should be.

是的。实际上，从机器学习收敛速度的角度来看，更小总是更好，因为你基本上是从梯度下降中获得最新信息。

Yeah. I mean smaller is always better actually, from an ML convergence rate perspective smaller is always better because basically you're getting the freshest information from the gradient descent.

但从总训练时间的角度来看呢？

But from total training time perspective?

从总训练时间的角度来看，从系统角度更小更差，所以最优解是这两者之间的权衡。所以你选择一个批次大小，然后针对该批次大小做一定量的前向和反向。

From total training time perspective, smaller is worse from a systems perspective, and so the optimum is the trade-off between those two. So you pick a batch size and then for that batch size you do some amount forward and then some amount backward.

你问为什么那里甚至有一个硬停止？流水线并行是因为这个，你这里有空闲时间，也就是气泡。文献中有很多技术可以以不同方式布局并避免它，比如零气泡或一前一后，它们以复杂的方式交错前向和反向。

You asked why is there even a hard stop there? Pipeline parallelism because of this, the fact that you've got this idle time here which is the bubble. There are so many techniques in the literature for how to lay this out differently and avoid that, like zero bubble or one forward one backward which interleave the forwards and backwards in complicated ways.

你可以用那个挖比特币。

You can mine bitcoin in that.

对，对。更有用的是你可以做权重梯度步骤。但在推理中，流水线对你关心的任何东西（如批次大小或延迟）的影响是中性的；它不会改善，也不会恶化。所以如果你看这个推理运行的延迟，如果它是流水线的，与全部在一个机架上相比，如果全部在一个机架上，我们只需把所有框向下滑动，仍然排成一行，延迟是一样的。所以流水线对延迟既不好也不坏。但它确实意味着每个机架使用的内存更少，比如内存容量，因为现在你只需要模型四分之一的大小，而不是整个模型。

Right, right. More usefully you can do the weight gradient step. But in inference, the effect of pipelining on anything you care about like batch size or latency actually is neutral; it doesn't improve it, doesn't make it worse. So if you look at the latency of this inference running it if it were pipelined versus if it were all on one rack, if it were all on one rack we would just slide all the boxes down and still put them in a row and the latency would be the same. So pipelining is neither better nor worse for latency. But it does mean that you just use less memory per rack, like memory capacity, because now instead of needing the whole model you only need a quarter of the model.

非常有道理。所以基本上推理时使用流水线是显而易见的，但训练时存在这种硬件权衡。

Makes a ton of sense. So basically no brainer to use pipelining during inference but there's this hardware trade-off during training.

所以即使在推理中，实际上它用得并不多。它降低了你的内存容量需求。实际上有巨大的过剩。我想你是说一个 Blackwell 机架有非常多 TB，也许几十 TB，这比万亿参数模型大得多。一个万亿参数模型只需要 1 TB。所以实际上它已经装得下了。因此流水线没有太大好处，因为你减少的是一个已经相当小的数字。

So even in inference, in fact it is not used a ton. It reduces your memory capacity requirements. There's actually a huge surplus. I think you're saying that a rack of Blackwell has many many terabytes, maybe tens of terabytes, which is much bigger than a trillion parameter model. A trillion parameter model only needs one terabyte. And so it already fits in fact. And so there's not a huge benefit from pipelining because you're reducing a number that's already pretty small.

但这确实说明，理论上也许你内存太多，也许你可以做不同的构建，比如内存更少的硬件。事实上，如果你在设计硬件时，你说我实际上不需要那么多内存，因为我不需要权重装在一个机架里。我可以把权重装在八个机架里。那么我或许可以构建每块 GPU 没有那么多 HBM 的硬件。

But it does say that theoretically maybe you had too much memory and maybe you could have done a different build, like different hardware that has less memory. In fact, if you were designing your hardware and you said I actually didn't need that much memory because I don't need the weights to fit in one rack. I can fit the weights in eight racks. Then I could have maybe built hardware that didn't have so much HBM per GPU.

上周，Horus 好心地给我和朋友们做了一场关于大规模预训练系统的精彩讲座。其中有一些概念我想为博客文章制作动画，比如权重和梯度如何根据你使用的并行方式流动。所以我给了 Cursor 我的讲座笔记和我在讲座中画的草图，让它可视化 Horus 解释的一个特定层次化集合通信。第一个版本已经很好了，然后我能够使用设计模式选择和调整任何特定组件。我能够在没有明确最终状态的情况下完成所有这些。Cursor 的 composer too fast 模型足够快，我几乎可以瞬间迭代。我可以尝试一个想法，在内置浏览器中测试结果，并立即做出任何更改。我在 20 分钟内完成了 10 个不同版本。如果你想看看这个动画，我把它和讲座笔记一起发布在博客文章中。链接在描述中。如果你想亲自尝试这种迭代设计流程，请访问 cursor.com/larch 开始。

Last week, Horus was kind enough to give me and my friends a great lecture on large scale pre-training systems. And there were some concepts that I wanted to animate for a write up on my blog, like how weights and gradients flow depending on the parallelism that you're using. So I gave Cursor my lecture notes and a sketch that I made during the lecture and I asked it to visualize a specific hierarchical collective that Horus had explained. The first version was already pretty good and then I was able to use design mode to select and tweak any specific components from there. I was able to do all of this without a clear end state in mind. Cursor's composer too fast model was quick enough that I was able to iterate almost instantaneously. I could try an idea, test the results in the built-in browser and immediately make any changes. I went through 10 different versions in under 20 minutes. If you want to check out this animation, I published it along with the lecture notes in a blog post. The link is in the description. And if you want to try out this kind of iterative design flow for yourself, go to cursor.com/larch to get started.

那么宏观问题，现在每个人都在谈论内存墙。内存变得超级昂贵。内存不够。智能手机出货量将下降 30%，因为内存不足。超大规模云服务商正在花费，如果我是 Dylan 的话，这很惊人，他们说今年资本支出的 50% 花在了内存上。

So macro question, everybody's talking about the memory wall right now. Memory is getting super expensive. There's not enough memory. Smartphone volume will go down 30% because there's not enough memory. Hyperscalers are spending, this is shocking if I'm Dylan said they're spending 50% of their capex this year on memory.

花在内存上？这可信。是的。

On memory? That's believable. Yeah.

那么超大规模云服务商的资本支出是多少？大概是数千亿甚至一万亿，他们花了一半在内存上。好吧。所以这是一个巨大的限制。这就是为什么我们今年不会得到新的笔记本电脑和手机。

So like what is hyperscaler capex? It's like high hundreds of billions maybe a trillion and they're spending half of that on memory. Okay. So that is a huge constraint. That's why we're not going to get new laptops and phones this year.

但与此同时，我们又有太多内存。比如人们愿意往这些系统里塞太多内存，对吧？

But at the same time we have too much memory. Like people are willing to put too much memory into these systems, right?

那么为什么 JEDEC 要把所有这些内存塞进这些机架，如果你不需要的话？

So why is JEDEC shoving all this memory into these racks if you don't need it?

是的。所以我们在之前提出的方程中，我们做了内存时间。所以内存带宽和计算带宽。

Yeah. So we've, in the equations we had here before we raised them, we were doing memory time. So memory bandwidth and compute bandwidth.

现在开始看内存容量。

Let's now start looking at memory capacity.

好的。

Yeah.

所以我们先只看内存容量，不考虑并行方案。内存容量或内存需求是总参数量，加上……这就是我们需要将权重放入某个系统所需的空间，然后还需要容纳 KV 缓存。因此，KV 缓存的大小等于批次大小乘以上下文长度再乘以每 token 的字节数。

So we'll start off with just memory capacity without even thinking about parallelism scheme. The capacity of memory or the demand on memory is the number of total parameters, plus... so this is what we need to fit the weights in some system that we are using, and then we need to fit the KVs as well. So, KVs go as batch size times the length of the context times bytes per token.

我在此背景下论证的，以及我为流水线并行提出的理由是：有一些技术可以解决这个问题。我们来考虑一下：我们将在一定数量的 GPU 上运行，设 E 为专家并行度——即专家层分片到多少个 GPU 上，比如 64。P 为流水线并行度，即机架数量，也许选 4 左右。这是整个系统的总内存需求。但现在我要计算每个 GPU 的内存需求。每个 GPU 的内存需求，我用小写 C_m 表示。显然，我们只需将所有数字除以 E 乘以 P。所以公式是：总参数量加上批次大小乘以上下文长度乘以每 token 的字节数，再除以 E * P。为什么这样正确？因为参数在机架内的所有 GPU 上完美分片，而层在不同机架间完美分片。我们可以以某种方式实现上下文在机架内 GPU 上的完美分片，然后基于层在机架间分片。

What I was arguing about in this context and the case I was making for pipelining is that there are some techniques that allow us to solve this. So let's consider: we're going to run this on some number of GPUs, and we're going to have E as the expert parallelism — how many GPUs we shard the expert layer across, for example 64. And P is going to be the extent of pipeline parallelism, the number of racks, maybe we'll pick four or something. So this is the total memory requirement across the system. But now I'm going to calculate the memory requirement per GPU. Per GPU memory requirement, I'll use a lowercase C_m. Obviously we just take all these numbers and divide by E times P. So it's this: total parameters plus batch size times context length times bytes per token, all divided by E * P. Why is this correct? Well, the parameters are perfectly divided amongst all the GPUs in a rack, and the layers are perfectly divided amongst the different racks. Somehow we can arrange the same perfect sharding of the contexts across GPUs in a rack, and then based on layer across racks.

抱歉，是机架数量吗？

And sorry, for the number of racks?

是的，比如。

Yeah, for example.

是的。所以这里我们实际上需要回过头来分析这个批次大小 B，你之前提到过微批次与全局批次的区别。让我们回到这个流水线并行图。这里有一个批次向前传播，然后我画的它好像就消失了。这其实不对。如果你考虑解码的工作方式，我已经生成了一堆 token。我做一次前向传播生成一个新 token，然后把它写入 KV 缓存，然后再做一次前向传播生成下一个 token。所以我实际上是在循环运行这个批次 0。事实上，我向前一次，完成后，就可以在这里开始循环的下一次迭代。所以我们把它填上。我们会有一个……哦，好的。是的。所以我们有这两个或三个小 2 和 3。那么让我们拆分这个批次。这个批次将是全局批次大小。所以 B 等于微批次数量乘以每个微批次的批次大小。我们需要多少个微批次？这个图中微批次的数量是 4：0、1、2、3。然后每个微批次的批次大小，仍然是大约 2000 这个数字。这个就是那个……

Yeah. So this is the place where we actually need to go back and analyze this batch size B, and you were making this comment that there's micro batching versus global batching. So let's come back to this pipelining diagram here. We've got one batch going forward here and then as I drew it, it kind of just disappeared. That's not really correct. If you think about how decode is working, I have a bunch of tokens that I have generated already. I do one forward pass where I generate a new token, and then I write that to my KV cache, and then I do another forward pass that generates the next token. So I'm actually going to be running this batch zero in a loop. In fact, I go forward once, I finish, I can start the next iteration of the loop up here. So we'll just fill this in. We'll have a... Oh, nice. Yes. So, we've got the two or three little two and three. So let's split this batch. This batch will be the global batch size. So B is going to be the number of microbatches times the batch size per microbatch. How many microbatches do we need? The number of microbatches in this diagram is four: 0, 1, 2, 3. And then the batch size per microbatch, this is still this like 2000-ish number. This is the one that is like...

嗯。

Mhm.

这大约是 2000 乘以稀疏度……抱歉，不，这是 300 乘以稀疏度。这就是每 20 毫秒占用的那个序列的大小，对吧？

This is the like 2000 times sparsity... sorry, no, this is the 300 times sparsity. This is how big the train that takes up every 20 milliseconds, right?

是的。这就是那个 20 毫秒的序列。所以全局批次大小等于微批次数量乘以本地批次大小。本地批次大小由这个硬件参数决定。微批次数量尽可能小，以便我们能够回绕而不产生空闲时间。如果更少，回绕时就会有空闲时间。你可以直观地看到它等于流水线阶段数。通过视觉证明：它是 4，这边也是 4。你可以看到它沿着这里走，然后回绕。

Yes. This is going to be the 20 milliseconds train. So the global batch size is the number of microbatches times the local batch size. Local batch size is set by this hardware parameter. The number of microbatches is as small as possible such that we can wrap around and not leave any idle time when we wrap around. If we had fewer, we would have idle time when we wrap around. You can sort of visually see that it is equal to the number of pipeline stages. Sort of proof by visual: it is four and it's four this way as well. You can look and see that it goes along here and then it wraps around.

是的，一个很基本的问题：实际中是这样做的吗？比如，今天的前沿模型在推理时真的会使用流水线并行吗？

Yeah, it's a very basic question: is this what is actually done? Like, as in a frontier model today, will actually have during inference have pipeline?

当然，在大规模训练中确实如此。推理时也可以这样做。我实际上要论证为什么它不那么有吸引力：它对权重有用，但对 KV 缓存作用不大。

For sure, during massive scale training this is done. It can be done for inference. I'm actually going to make the case for why it is less attractive: it is useful for weights but not so useful for KVs.

是的，是的。

Yeah, yeah.

最大的挑战是……让我们把这个填上。这里的微批次大小最终等于流水线阶段数。当我们回过头来代入所有这些，我们得到流水线阶段数乘以这里出现的小 b。然后当我们分解这个式子时，我把它分成两项。这里我们得到除以 E * P，这里仍然除以 E * P，但 P 抵消了。所以我们发现：如果增加流水线阶段数，权重的内存占用持续下降，但激活值的内存占用保持不变。所以它实际上并不奏效：一旦你做了足够的流水线并行（其实不需要很多，甚至两个通常就足够了），这一项变得非常小，这一项成为主导。KV 缓存成为主导。

The big challenge is... so let's fill this in. The microbatch size here ends up being equal to the number of pipeline stages. When we go back and substitute all of that into here, we get a number of pipeline stages times this little b showing up in here. And then when we factor this out, I'm going to split this into two terms. We get the full division by E * P over here. We still have division by E * P over here, but the P's cancel. So what we find is: if you increase the number of pipeline stages, the memory footprint for the number of weights keeps going down, but the memory footprint for the number of activations stays constant. So it doesn't actually work: most of your memory, once you do enough pipelining (and it's really not much, even two is often enough), this term becomes very small. This becomes the dominant term. The KV cache becomes the dominant term.

是的，我知道这不对。我在想为什么这里的逻辑错了。如果你有很多不同的阶段，你在多个阶段间进行流水线，KV 值在层之间不共享。那么为什么跨多个层进行流水线没有帮助呢？因为那样你就不需要存储……

Yeah, I know this is wrong. I'm trying to think out why the logic here is wrong. If you have many different stages, you're pipelining through many different stages, the KV values are not shared between layers. So why would it not help to be pipelining across multiple layers because then you don't have to store...

是的，你只需要存储一层而不是两层的 KV 缓存，对吧？所以从这个角度看是有帮助的。你说得对。但与之竞争的是，你需要让所有机架同时保持忙碌。因此，同时运行的序列数量增加了。

Yeah, you only need to store like one layer rather than two layers of KVs, right? So it helps from that perspective. You're right. What's competing with that though is that you need to be keeping all of the racks usefully busy at a time. And so the number of sequences that are in flight simultaneously has gone up.

啊，是的，是的，有道理，有道理。

Ah yeah yeah yeah, makes sense makes sense makes sense.

所以这两者正好抵消，最终每个 GPU 并没有节省内存。

So those exactly cancel and you end up not getting a saving per GPU.

对。

Right.

这从根本上回到了你无法在 KV 缓存之间分摊的问题。

This goes back fundamentally to the point that you're not able to amortize across KV caches.

是的。首先，我们说过你不能在批次大小之间分摊 KV 缓存，现在我们又说你也不能在流水线阶段之间分片它。从这两个角度来看都很糟糕。

Yeah. So first, we said you can't amortize KV caches across batch size, and now we're saying you also can't shard it across pipeline stages. It sucks from both of those points of view.

嗯，有意思。好的。那么推理期间做了什么？

Yeah. Interesting. Okay. Because then what is done during inference?

DeepSeek 论文报告了他们的做法，就是大量使用专家并行。实际上，你应该将专家并行增加到你的扩展域大小。然后做很少的流水线并行，可能完全没有，或者只有两个阶段，刚好让权重存储不成大问题。这是唯一有意义的两种并行方式。过去还有张量并行，即在专家内部切分，但现在专家太小了，这不是一个有利可图的优化。

The DeepSeek paper reports what they do, which is they just do a lot of expert parallelism. You should, in effect, increase your expert parallelism up to your scale-up domain size. Then do very little pipelining. Maybe none at all, maybe two, just enough to make the weight storage not too big of an issue. Those are the only two parallelisms that really make sense. In the past, there was tensor parallelism, which was cutting up within an expert, but the experts are so small now that that is not a profitable optimization.

那么这又回到了问题：这是否意味着前沿实验室在做推理时基本上只在一个扩展域内？

So this goes back to the question: does that mean that frontier labs, when they're doing inference, are just basically within a single scale-up?

是的。我的意思是，你可以看它如何依赖于模型大小。你可能有一个非常大的模型，比如超过一个机架的内存，那么你应该做一点流水线并行。也许它非常稀疏，例如，那会是做流水线的一个理由。

Yes. I mean, you can look at how it depends on model size. You could have a very large model, like one that exceeds the memory of a rack, and there you should be doing a bit of pipelining. Maybe it's extremely sparse, for example, and that would be a reason to do it.

所以我想这又回到了讲座开始时的一个承诺，即这实际上也会告诉你 AI 的进展。在某种程度上，模型规模的 Scaling 直到最近一直很慢。让我确认我理解了你的说法。这个说法不是你可以跨更多机架训练，而是说以前这样做没有意义，就像我们之前没有能力轻松地对更大的模型进行推理。

So I guess this goes back to the promise at the beginning of the lecture, which was that this will actually tell you about AI progress as well. To the extent it is the case that model size scaling has been slow until recently. Let me make sure I understand the claim. The claim would not be that you could have trained across more racks. It was just that it would not have made sense before, like we didn't have the ability to do inference for a bigger model easily.

实际上，流水线并行对上下文长度没有帮助。它完全有助于模型大小。由于能够做流水线并行，至少一个机架不应该成为你容纳模型参数能力的约束。我想你问的另一个考虑是为什么它没有 Scaling 得更大，以及为什么更大的扩展域有帮助。

Actually, pipelining doesn't help with context length. It totally helps with model size. Because of the ability to do pipelining, at least a rack should not be a constraint on your ability to fit the model parameters. I guess the other consideration you're asking is why hasn't it scaled up more and why did bigger scale-up domains help.

我们讨论了其中的一个方面，我们说过这不是因为内存容量。我们有解决内存容量的办法，至少对于模型大小而言。不是针对 KV 缓存大小，但至少对于模型大小我们有办法解决内存容量。另一个出现的问题是延迟。

We talked through one aspect of that, which is we kind of said it's not because of memory capacity. We have a solution to the memory capacity, at least with respect to model size. Not with respect to KV cache size, but at least with respect to model size we have a solution to memory capacity. The other issue that shows up is latency.

我正要问：从机架到机架每跳的延迟成本是多少？

I was just about to ask: what is the latency cost per hop going from rack to rack?

这非常依赖于硬件。我不能很有把握地说。我认为大概在几毫秒的量级，但可能差一个数量级。

This is very much dependent on the hardware. I can't say with a lot of authority. I think it's probably on the order of a few milliseconds, but it could be off by an order.

四个流水线阶段是现实的数量吗？

Is four a realistic number of how many pipelining stages you might have?

是的。

Yeah.

好的。所以那不是一个巨大的延迟影响。

Okay. So that's not a huge latency impact.

等等，我想每个 token 是 10 毫秒。

Wait, I guess it's 10 milliseconds per token.

没错。

That's right.

两倍四左右，或者我不知道你说了多少。

Two times fourish, or I don't know how many you said.

每个 token 10 毫秒实际上很多。如果从 20 增加到 30，对吧？或者类似的情况。

10 milliseconds per token is actually a lot. If it goes from 20 to 30, right? Or something like that.

所以，为了描绘它经过的路径。这里你从你的 GPU 或 TPU 到网卡，然后到机架顶部交换机，然后跳到另一个机架，再反向做同样的事情。所以你必须把这些不同部分的延迟加起来。

This is so, just to chart the path it goes through. Here you're going from your GPU or TPU to a network card, which then goes to a top-of-rack switch, and then hops over to the other rack and does the same thing in reverse. So you have to sum up the latencies of these different things.

所以这和数据中心一样；实际上可能上到数据中心交换机再回来。取决于部署配置。

So this is the same thing as the DC; it may in fact go up to a datacenter switch and back. Depends on deployment configuration.

明白了。而且因为解码是顺序的，它们也不会跨阶段叠加。你不能同时做它们。

Got it. And because it's decode in sequential, it's also not like they stack up across the stages. You can't do them at the same time.

没错。

That's right.

好的。所以我想这让我们回到问题：扩展域的大小是否与过去几年 AI 模型大小为何如此有关，无论是通过训练还是推理？

Okay. So I guess this brings us back to the question: is the size of the scale-up at all relevant to why AI model sizes have been what they have been over the last few years, whether through training or through inference?

是的。所以我们讨论了跳的延迟。还有内存时间延迟，这实际上通过更大的扩展域得到了显著改善。我在这里回忆一下 TMM。权重的 TMM。它等于总参数数量除以内存带宽。我们这里说的是哪个内存带宽？是我可以并行用来加载这些权重的 GPU 数量。我不能并行使用不同的流水线阶段，因为它们不是同时运行的，但我可以并行使用扩展域中的所有 GPU 来加载权重。所以这实际上非常有效。基本上，我最终得到一个项。这个内存带宽项本身等于扩展域大小乘以每个 GPU 的内存带宽。所以这个项增长不大，可能每代增长 1.5 到 2 倍。但另一个项增长了大约 8 倍。

Yeah. So we talked about latency of the hop. There is also the memory time latency, which is actually substantially improved by larger scale-up domains. I'll recall TMM down here. TMM for the weights. This was equal to the number of total parameters divided by the memory bandwidth. Which memory bandwidth are we talking about here? It is the number of GPUs that I can use in parallel to load these weights. I can't use different pipeline stages in parallel because they're not running at the same time, but I can use all the GPUs in my scale-up domain in parallel to load the weights. So this is actually extremely effective. Basically, I end up with a term here. This memory bandwidth term itself is equal to scale-up size times memory bandwidth per GPU. And so this term doesn't increase a lot. It maybe increases 1.5 or 2x per generation. But this one increased by like a factor of eight.

所以更大的扩展域重要的原因不是整个扩展域的内存容量，而是内存带宽。

So the reason bigger scale-up matters is not the memory capacity of the whole scale-up, but really the memory bandwidth.

是的。流水线并行完全解决了容量问题，但扩展域大小有助于解决带宽问题，而带宽问题帮助你实现更长的上下文长度，随着这些模型变得更加智能体式，这越来越相关。

Yeah. Pipelining totally solves the capacity problem, but scale-up size helps solve the bandwidth problem, and the bandwidth problem helps you do longer context lengths, which is more and more relevant as these models get more agentic.

是的，它首先让你以更低的延迟运行模型。如果我做一个非常快的模型，放在一个小 H100 盒子上，延迟会非常高。

Yeah, it lets you just run the model at lower latency as a first thing. If I just do a very fast model and it's on a little H100 box, the latency will be really high.

好的。一个非常边缘的问题。有 Chinchilla Scaling，它告诉你模型应该相对于你要训练的数据量有多大。但现在显然你不仅仅是在优化用训练算力能得到的最优质量模型。你想要用户通过训练和推理算力的混合得到最佳结果。所以有一个问题：你应该过度训练模型多少，使得在训练和推理上分摊的算力最小化，以达到一定的性能。

Okay. A super tangential question. There's Chinchilla scaling, which tells you how big a model should be relative to the amount of data you're going to train it on. But now obviously you're not just trying to optimize for the highest quality model you can get with training compute. You want the best results a user can get with a mixture of training and inference compute. So then there's a question of how much should you overtrain a model such that the compute amortized over training and inference is minimized to get a certain performance.

但现在有了 RL 推理，就有了另一个考虑：你会做一定量的预训练，这些预训练既用于 RL 生成，也用于最终用户的推理。这里说的过度训练是指，虽然仅从训练算力角度看，用一个更大的模型训练更短时间会更高效，因为它学得更快，但也许你得到一个更小的模型，花费的算力比原本更多，但现在给用户用就更便宜了。

But now with RL inference there's another consideration: you're going to do some amount of pre-training, and that pre-training will be used both for RL generation and then for inference for the final user. By overtraining here I mean while it would have been more efficient just from a training compute perspective to have a bigger model that you train for less time because it can learn faster, maybe you get a smaller model and spend more compute on it than you otherwise would have, but now it's cheaper to give it to users.

让我更具体地问一下：模型比 Chinchilla 最优值过度训练了多少？这又是否因为我们的生成而发生了变化？

Let me question more concretely: how much more than Chinchilla optimal are models overtrained, and has that changed as a result of our generation?

这里我们得做一些猜测，因为更新后的缩放定律以及使用情况和模型流量都没有公开。一种看法是：我先做一个一般的启发式论断。如果我有某个总成本，是成本 A 和成本 B 之和，比如训练成本和推理成本，我想最小化这个和，对于许多曲线，最小值往往出现在这些成本相等的地方。这是一个启发式论断，但有很多例子成立，比如一个是 1/x，另一个是 x，它们往往在相等点处最小化。对于 e^x 和 e^{-x} 以及其他各种情况也成立。所以基本上，我有一条下降的曲线和一条上升的曲线，它们往往在相等点处最小化。启发式地，我推测这对你描述的情况也成立。要实际证明这一点需要查看缩放定律并拟合这些奇怪的指数，但遵循幂律的东西往往有这个性质。所以我就这么论断，然后继续。

This is a place where we have to do a bit of guesswork because the updated scaling laws and the use and model traffic are not reported. One way to look at it: let me first make a general heuristic claim. If I have some total cost which is a sum of cost A and cost B, like training cost and inference cost, and I want to minimize this sum, for many curves the minimum tends to be where these costs are equalized. That's a heuristic claim, but there are many examples where it's true, like one is 1/x and the other is x, they tend to be minimized at the point where they equal each other. It's also true for e^x and e^{-x} and all kinds of other things. So basically, I've got some curve going down and another going up, and they tend to be minimized at this equal point. Heuristically, I will conjecture that is true for the setup you described as well. Actually showing that would require looking at the scaling laws and fitting these weird exponents, but things that follow power laws tend to have this property. So I'll just make that claim and move on.

所以我们说训练成本加上推理成本我们想要相等。我们先只做预训练，但实际上我们可以一般性地做所有部分。所以我们把成本计为预训练成本：活跃参数数量乘以预训练数据量。这就是预训练成本。这里有一个因子 6，是 FLOP 数，著名的 6ND 公式。然后在 RL 中，我们大致有相同的东西：相同的活跃参数数量，但现在数据量是 RL 数据。有一个额外的效率乘数，或者说低效性，即你没有在所有 rollout 上进行训练。

So we're going to say that the cost of training plus the cost of inference we want to equalize. We'll do pre-training only first, but actually we can do all of it in general. So we'll cost it as cost of pre-training: number of active parameters times the data on pre-training. That's the cost of pre-training. There's a factor of six out here, which is the number of FLOPs, the famous 6ND formula. Then in RL we have approximately the same thing: same number of active parameters, but now the amount of data is the RL data. There's an extra efficiency multiplier, or inefficiency, which is the fact that you're not training on all your rollouts.

嗯，是的，有这个因素。然后另一个可能更大的低效是这涉及大量的解码，而解码的 MFU 通常低于训练。

Well, yeah, there's that. And then the other perhaps even bigger inefficiency is that this involves a substantial amount of decode, and often decode runs at less MFU than training.

所以如果你在 RL 中对每个生成都做反向传播，那就是 6ND。

So if you're doing a backward pass on every single generation in RL, it would be 6ND.

所以这个数字可能更小，对吧？比如在 2 到 6 的范围内。

So this could be a smaller number, right? Like somewhere in the range of two to six.

是的，我们就说在 2 到 6 的范围内，就这样。

Yeah, we'll say somewhere in the range of two to six and leave it at that.

然后我们可以加上推理成本。推理成本是 2 乘以活跃参数数量乘以推理数据量。我觉得我刚才说得乱七八糟。所以对听众来说：每个参数的前向加反向是 6。仅前向是 2。这就是为什么 RL（你肯定会生成所有轨迹，但可能不会全部训练）是 2 到 6。而推理只是 2。

Then we can add in the inference cost. The inference cost is two times number of active parameters times the data in inference. I think the way I said it was super garbled. So for the audience: forward plus backward per parameter is six. Forward alone is two. That's why RL, where you might definitely generate all trajectories but might or might not train on all of them, is two to six. And inference is just two.

是的。谢谢。

Yes. Thank you.

所以我们基本上要解出这三个项相等的情况。这就是人们会处于的大致范围。实验室有更多信息，知道做更多 RL 与更多预训练哪个更有效。我没有这些信息，但我认为一个好的大致比例是每个占 30-30-33%。

So we're going to solve for essentially equality of all three of these terms. That is ballpark where people are going to be. Labs have more information on what is productive in doing more RL versus more pre-training. I don't have that information, but I think a good ballpark is 30-30-33% split between each of them.

实际上，我不确定我理解这其中的直觉。另一个朴素的模型可能是 RL 加预训练占 50%，推理占 50%。

Actually, I'm not sure I understand the intuition for that. Another naive model could have been that RL plus pre-training would be 50% and inference would be 50%.

是的，那也是有效的答案。因为这是启发式的，我无法真正论证哪个更好。它们差别不大：33 对 25 只是一个小因子。

Yeah, that's also a valid answer as well. Because this is heuristic, I can't really argue for one versus the other. They don't differ by that much: 33 versus 25 is only a small factor.

所以我们选一个。全部相等似乎足够简单。我们就解它们相等的情况。这很直接。我们马上可以看到活跃参数数量完全消失了。所以我们把它提出来。我们就说预训练数据量加上某个倍数 alpha 乘以 RL 数据量，最终等于某个倍数 beta 乘以推理数据量之和。然后我们大致估算一下 alpha。这个 alpha 大概在 2 到 6 除以 6 的范围内，来自这个项与那个项的比较。然后我们有一个低效项，我猜大概在 30% 左右。所以这个 alpha 大概是 1/10 左右。而这个 beta 实际上是一样的：它是 33% 的 1/3，所以也等于 1/10 左右。如果两者都是 1/10，那基本上意味着 RL 上从来没有反向传播。

So let's pick one of them. All equal seems simple enough. And so we're just going to solve for equality of them. It's pretty straightforward. We can immediately see that the number of activated parameters totally disappears. So let's factor that out. And we're going to just say that data in pre-training plus some multiple of alpha times the data in RL is going to end up equal to some sum of beta times the data in inference. So then let's roughly size the alpha. This alpha is going to be somewhere in the range of 2 to 6 over 6 from this term compared to this term. And then we've got an inefficiency term which I would say is maybe in the range of like 30% or something like that. So this alpha is going to be something like 1/10 say. And this beta here is actually the same: it's 1/3 of 33%, so it's also equals 1/10 something like that. If both of them are 1/10, that kind of implies that there's never a backward pass on RL.

是的。好吧。我们可以把它做成 2/10。让它大一点。

Yeah. Okay. We can make this like 2/10. Make it a bit bigger.

所以是的，就像再写一遍：这是 2/10，这是 1/10。所以你有多少推理 token，这只是一个函数，比如我每秒有数亿 token，乘以我的模型部署了大概两个月，然后我就换到下一个版本。

So yeah, like just write it out once more: this is 2/10, this is 1/10. So the number of inference tokens you have, and this is just a function of like I've got hundreds of millions of tokens per second times my model is deployed for I don't know two months before I shift to the next version.

这应该决定了 RL 和预训练中的 token 数量。我想我们之前没有做预训练和 RL 之间的等价关系，所以这里我们来做一下。为了让它们在成本上等价，预训练的数据量应该等于 RL 数据量的 2/10。抱歉，我搞反了。当效率低时我们付出更多成本，所以应该是倒数。追溯回去，这个东西实际上就像这里写的一样。是的，所以这个大约是 1.5，这个是 1。价值数十亿美元的算力就这样流向另一个方向。

That should determine the number of tokens in RL and pre-training. And then I guess we didn't do the equivalence between pre-training and RL, so we'll do that here. Data pre-training should be equal to like 2 over 10 times data in RL for them to be cost equivalent. So sorry, this one over, I got it backwards. Like we pay more cost when it's inefficient, so it needs to be one over. Tracing this back, this thing ends up actually being as written here. It's like yeah, so this is like 1.5 and this is one. Billions of dollars worth of compute just flow the other direction.

是的。没错。没错。

Yeah. Right. Right.

我觉得如果你用电子表格实际算一下，你可能会发现钱在什么时候打水漂。是的。所以我认为所有这些最终都像这里建模的那样接近。这个 30% 可能有点太慷慨了。所以我们这里说 1.5，这里留作 1。所以我认为在这一点上你几乎可以直接读出：推理 token 的数量应该大约等于预训练 token 的数量，也应该大约等于 RL token 的数量，在我们无法推理的因子范围内。但看起来 RL token 应该比预训练 token 少。

I think if you do it with a spreadsheet and actually work it out, you might notice when the money is going down the drain. Yeah. So I think all of these end up being close as modeled here. This 30% may have been a little bit too generous. So let's say something like 1.5 here and leave this as a one here. So I think at this point you can almost read it off: the number of inference tokens should be about the same as the number of pre-training tokens, should be about the same as the number of RL tokens, within factors that we're not able to reason about. But then it looks like there should be fewer RL tokens than pre-training tokens.

是的，一般来说是这样，因为 RL 在机器时间上效率较低。所以如果你试图让 RL 和预训练的时间相等，那么你应该用更少的 token 来获得相同的挂钟时间。这都很有趣。我从未从数据均衡的角度考虑过。我认为从成本均衡开始是对的，但根据你如何建模成本，这接近于数据均衡。

Yes, that's in general right because RL is less efficient in terms of machine time. So if you're trying to equalize the RL and pre-training time, then you should have fewer tokens in order to have the same wall time. This is all quite interesting. I never thought about it in terms of equalizing in data. I think starting with equalizing cost is right, but depending on how you model the cost, this comes close to equalizing in data.

如果每个使用 GPT-5 的用户，为了让它得到最优训练，他们流式传输的 token 总量应该等于预训练中使用的 token 总量。而预训练中的 token 总量就是所有人类知识的总和。所以每个模型应该在输出上生成它从输入中获得的全部人类知识。

If every single user who uses GPT-5, for it to be trained optimally, the total amount of tokens that they stream should equal the total amount that have gone into pre-training. And the total amount of tokens that went into pre-training is the sum of all human knowledge. So each model should generate the sum of human knowledge on the output that it gets on the input.

是的。那么人们会朝哪个方向犯错呢？如果你认为人们的预测能力并不完美，而且你还冒着做出一个非前沿模型然后直接扔掉的风险，那么这就会改变成本权衡，因为推理存在一定的概率，你应该对推理 token 进行一定程度的折减。

Yeah. So which way are people going to error? If you think that people's power of prediction is not perfect, and you also run the risk that you make a model that is not a frontier model and then you just throw it away, then that kind of changes the cost trade-off because there's some probability that applies to the inference and you should derate the inference tokens by some amount.

对。那么我们能反推出对于一个给定规模的模型，比 Chinchilla 最优多用了多少算力吗？所以我认为我们只能在这里做一些现实世界的假设。推理 token 我们应该完全能算出来。假设 2 亿，我不知道，也许现在是每秒 5 亿 token。我真的不知道。每秒 5 亿 token 乘以模型部署 2 个月后过时。我真的不知道。我算不出来。你能算一下吗？2.6 × 10^15。

Right. And then can we back out how much more compute than Chinchilla optimal for a given sized model? So I think we just have to make some real world assumptions here in order to do that. So the inference tokens we should totally be able to catch. Let's say 200 million, I don't know, maybe it's like 500 million tokens a second now. I don't really know. 500 million tokens a second times a model is deployed for 2 months before it becomes obsolete. I don't really know. I can't do this in my head. Can you compute? 2.6 times 10^15.

好的。2.6 × 10^15。好的。这个数字可能太大了，因为这是一个系列中的多个模型。所以我们把它缩小五倍或十倍之类的。

Okay. 2.6 times 10^15. Okay. This number is probably too large because this is going to be multiple models in a family. So let's make it like five times smaller or ten times smaller or something like that.

好的。所以我们估计每个特定模型每秒大约 5000 万 token。模型上线两个月。所以这大约是 200 万亿 token。然后我们想把它与前沿模型上的活跃参数进行比较。我实际上不知道最新的传闻，但有些人知道。你知道吗？

Okay. So we're estimating maybe 50 million tokens per second per specific model. The model is live for two months. So this comes out to around 200 trillion tokens. And then we want to compare that to active parameters on a frontier model. I don't actually know the latest rumors, but some do. Do you know?

有人告诉我 150 万亿活跃参数。

Somebody told me 150 trillion active parameters.

抱歉，我指的是 token。在 150 万亿 token 上训练。有趣。这很相似。

So sorry, I meant tokens. Trained on 150 trillion tokens. Interesting. Which is similar.

是的。实际上很相似。所以预训练的数据……

Yeah. That's actually similar. So data on pre-training...

这没有很好的引用，但我认为活跃参数通常在 1000 亿左右，也许更大一点。所以我假设活跃参数约为 1000 亿，乘以 20 得到 Chinchilla token 数。所以 Chinchilla 大约是 2 万亿。我们看到我们比那个大 100 倍。

This is not well cited, but I think often active parameters could be in the range of like 100 billion something like that, maybe a bit larger. So I'm assuming active parameters of about 100 billion and multiply by 20 to get the Chinchilla token count. So Chinchilla would be around 2 trillion. And we see we're at 100 times larger than that.

实际上，Chinchilla 是什么意思？就像 Chinchilla 缩放定律推荐的预训练 token 数量？

Actually, what does the Chinchilla mean? Like the token count for pre-training that the Chinchilla scaling law would recommend?

我想是的。哦，我明白了，所以它过度训练了多少？

I guess so. Oh, I see, so how much is it overtrained?

明白了。所以这个 200 万亿或 100 万亿参数与 Chinchilla 最优的 2 万亿的比率。这就是它过度训练的程度，可能实际上是 100 倍过度训练。所以如果你考虑这一点，只要这个数量级大致正确，通过思考“好吧，你希望所有东西在算力上相等”，如果 OpenAI 也意识到这一点，并且他们每秒提供一定数量的 token，那就告诉你 GPT-5 的预训练用了多少数据。

Got it. So the ratio of this 200 trillion or 100 trillion parameters over the Chinchilla optimal of 2 trillion. And that's the amount it's overtrained, which is like effectively 100 times overtrained perhaps. So if you consider this right here, to the extent this is in the right ballpark, just by thinking about okay, you kind of want everything to be equal in terms of compute. Here, if OpenAI also realizes that and they're serving a certain amount of tokens per second, that tells you how much data went into the pre-training of GPT-5.

即使有 50% 的误差之类的，你能从第一性原理推导出这些数字也是相当惊人的。这也是为什么你应该到处近似，因为这里的误差范围很大，但把 A 设为等于 B 然后算出来，这有点让人感到有力量。

Even if it's like 50% off or something, that is sort of wild that you can sort of first-principles these kinds of numbers. This is also why you should just approximate everywhere because there are such big error bars on this, but it's kind of empowering to just set A equal to B and figure it out.

是的。这太酷了。

Yeah. That's super cool.

好的。所以试图推断东西很奇怪。我们可以公开查看这些模型的 API 价格，也许你能从中了解到一些东西。首先，对于更长的上下文，Gemini 3.1 如果超过 20 万 token，价格比低于 20 万 token 贵 50%。在高层面上，我理解为什么会这样，但为什么正好是 50%？

Okay. So it is weird trying to deduce things. We can publicly look up the prices of the APIs of these models and maybe you can learn something from that. First, with a longer context, Gemini 3.1 is 50% more expensive if you go over 200k tokens than if you're below 200k tokens. At a high level, I understand why that might be, but why specifically 50%?

是的。那么为什么正好是 50%？我们来……首先，高层面上，成本会随着上下文长度增加而增加。

Yeah. So why specifically 50%? Let's sort of... So the high level even in the first place is there is some amount of increasing cost with context length.

是的。我们可以再提一下。那是内存时间与计算时间的关系。所以我们之前已经列出了这些相同的方程，关于内存读取的时间，即权重和 KV 缓存。

Yeah. And we can bring that back up. That was the memory time versus the compute time. So we've put up these same equations from before of the time for memory fetches which is the weights and the KV cache.

然后是计算的时间，也就是权重的矩阵乘法。我也会画出成本曲线。但这次我会把它作为上下文长度的函数，而不是补丁大小的函数。所以这是时间对时间。这是作为上下文长度函数的成本曲线。我们会画出计算部分。计算成本实际上相对于上下文长度是恒定的。这里没有对上下文长度的依赖。实际上，有一些依赖，但非常轻微，所以我们忽略它。所以这是计算的时间。然后我们还会画出内存读取对上下文长度的依赖。这个从权重的较大数值开始，然后随着上下文长度逐渐增长。所以大概在这里，然后随着上下文长度逐渐增长。所以你取最大值，你会看到这里有一个拐点。所以这些是例如 Gemini 可能支付的成本。然后你考虑如何在上面构建定价结构？你会希望确保无论上下文长度是多少，你都能盈利。

And then the time for the compute, which is just the matrix multiplications for the weights. I will also draw the cost curve. But this time I'll do it as a function of context length instead of as a function of patch size. So this is time over just time. So this is the cost curve as a function of context length. We'll draw the compute. The cost of the compute is actually constant as a function of context length. There's no dependence here on context length. In reality, there is some dependence, but it is very mild, so we'll ignore it. So this is the time for the compute. And then we'll also draw the dependence of the memory fetch on context length. And this starts at a large number for the weights and then grows gradually with the context length. So maybe here, and then grow gradually with context length. And so you take the maximum and you see there is this inflection point here. So these are the costs that, for example, Gemini might be paying. And then you think how might you put a pricing structure on top of that? You would like to ensure that no matter what the context length is, you are still profitable.

有意思。

Interesting.

所以我们有一个两级定价结构，也许我们有一个看起来像这样的东西，直到某个上下文长度。

And so we've got a two-tier pricing structure, maybe we've got something that looks like this up to some context.

太迷人了。

That's fascinating.

所以我认为这说明了点什么，鉴于拐点在 200k，这可能意味着这与这个交叉点有些一致。也许不完全一致……

So I think it says something about given that the bump is at 200k, it probably means that this is somewhat aligned with this crossover point. Maybe not exactly aligned with...

迷人。

Fascinating.

我们实际上甚至可以完成那个计算，看看结果如何。我们可以求解每个 token 的字节数。如果我们对激活参数的数量做一些假设。所以求解每个 token 的字节数。我们假设内存时间和计算时间相等的点是在，比如说，200k 个 token。所以我们让这两者相等。我们还假设批次大小足够大，使得权重上的内存时间可以忽略不计。所以我们忽略这个，专注于 KV 缓存上的实际内存时间。所以最终得到：将这个项复制过来，批次大小乘以上下文长度乘以每个 token 的字节数除以内存带宽，等于激活参数数量除以浮点运算次数。然后我们求解每个 token 的字节数。这里缺少了大小。它出现在这里，然后当我们到这里时它抵消了。我丢掉了上下文长度。所以我们可以代入数字。这个数字是我们之前看到的数字的倒数。它大约是 1/300，这在许多不同的硬件平台上相当稳定。我们推测性地认为激活 token 的数量大约是 1000 亿，上下文长度我们说是 200k。这里有点问题。上下文长度应该在分母而不是分子。1667，大约 1k，几乎 2 千字节。这实际上是合理的。所以我们说大约 2 千字节。所以让我们对这可能是什么做一个合理性检查。人们用少量每 token 字节数做注意力机制有两种机制。一种是密集注意力，在层之间有很多重用。所以 Character AI 有一篇博客文章谈到交替使用长上下文和短上下文。在 Character AI 类型的模型中，全局上下文（我们这里真正讨论的）在所有层之间共享。所以为了得到这 2 千字节，你可以通过例如 d_head 为 128 来实现，这是典型的。然后字节数通常是注意力层数乘以 2 * d_head 乘以 Q 头数。所以这是每层唯一上下文的数目。你是跨多层共享上下文还是只使用一次？在 Character AI 这样的模型中，这个数字是 1。我们说是 100，128。这是一个选择，通常范围从 1 到……抱歉，我指的是 KV 头。

We can actually probably even complete that calculation just to see where it lands. We can solve for the number of bytes per token. If we make some assumptions about the number of active parameters. So solving for the number of bytes per token. We're going to assume the point where we equalize the time of memory and the time of compute is at, let's say, 200k tokens. So we equalize these two. We're also going to assume that the batch size is large enough that the memory time spent on weights is negligible. So we'll forget about this and focus on the actual memory time spent on KV cache. So that ends up saying copying this term over batch times len context times bytes per token over memory bandwidth is going to be equal to number of activated parameters over flops. And then we're going to solve for bytes per token. Size was missing here. Shows up here and then it cancels out by the time we get to here. And I dropped the len context. So we can plug in numbers. This number is the reciprocal of the number that we saw before. It's like one over 300, which is reasonably stable across many different hardware platforms. We conjecturally said that maybe number of activated tokens is like 100 billion and length of the context we said was 200k. Something is wrong here. The length of the context should be on the denominator not the numerator. 1667, about one kil, almost 2 kilobytes. That is plausible actually. So we said around 2 kilobytes. So let's just do a sanity check for what this could be. There are two mechanisms that people do attention with a small number of bytes per token. One is dense attention with a lot of reuse across layers. So Character AI has a blog post talking about that alternating long and short context. And in the Character AI kind of model, which also showed up in the Gemma models, the global context, which is really what we're talking about here, was shared across all the layers. And so to get this 2 kilobytes, you could get that for example as a d_head of 128 is typical. And then the number of bytes is typically number of attention layers times 2 * d_head times number of Q heads. So this is the number of unique contexts per layer. Do you share the context across many layers or do you use it only once? In Character AI like models, this number is one. We said this is 100, 128. And this is a choice which typically ranges from one... sorry, this is KV heads I meant.

所以写了一个头和一个 KV 头。那是……

So there is written a head and a KV head. Is that...

KV 头是存储在内存中的头，比如存储先前 token 的内容。Q 头是检索头。它们只是临时使用，由正在关注的 token 使用。所以在这个自回归上下文中，我有与所有上下文关联的 KV 头，然后是与这个新 token 关联的 Q 头。但这个头，128。

The KV heads are the heads that are stored in memory, like store the contents of the previous tokens. The Q heads are the retrieval heads. They're only used temporarily and they're used by the attending token. So in this autoregressive context, I've got KV heads associated with all of the context and then Q heads associated with this new token here. But this head, the 128.

哦，这个数字实际上是一样的……哦，所以这个 d_head 是向量的维度。

Oh, this number is actually the same for... oh, so this d_head is the dimension of the vector.

是的，KV 头的数量通常在 1 到 8 之间。

Yeah, and number of KV heads is typically in the range of 1 to 8.

所以通过例如八个 KV 头和 128 的 d_head 来得到这个数字是完全合理的。那正好给你这个数字。或者你可以有更少的 KV 头但更多的层。有意思。

So it is totally plausible to get this by, for example, having eight KV heads and a d_head of 128. That gives you exactly this number. Or you could have fewer KV heads but more layers. Interesting.

是的。所以这是通过密集注意力达到的一种方式。还有一种方式是通过稀疏注意力，你增加所有这些数字，但然后你有一个稀疏项。

Yeah. So this is one way to get there via dense attention. There's also a way to get there via sparse attention where you increase all of these numbers, but then you have like a run over sparsity term.

所以是的，我认为这个数字是合理的，虽然可能有点小。

So yeah, I mean I think this number is plausible if maybe a little bit small.

有趣的是他们会通过 API 定价泄露这么多信息。

It's funny that they would leak so much information through their API pricing.

我的意思是，你有动力将价格定得接近成本，否则别人可能会利用你。

I mean you are incentivized to price close to your costs because otherwise someone could script you.

也许我们可以从输入和输出价格的差异中学到一些东西。以及这告诉我们这些模型中解码与预填充的关系。我记得上次检查时，它贵了 50% 左右或者……

Maybe we can learn something about the difference in input versus output prices. And what that tells us about decode versus prefill in these models. And I think last I checked it's like 50% more expensive or something like that or...

我不记得了。我过去看到的是贵三到五倍。假设贵五倍。好的。这是解码中处理下一个 token 的计算量。假设你在做预填充。你不仅仅处理最近的 token。你并行处理所有 token。所以我想说这将是这个乘以预填充长度，通常是一次传递的长度。

I don't remember. What I've seen in the past is like three or five times more expensive. Let's say it's five times more expensive. Okay. This is the compute to process the next token in decode. Suppose you're doing prefill. You're not just processing the most recent token. You're processing all the tokens in parallel. So I want to say that it would be this times len prefill length of a pass in general.

是的。如果我们说我们可以把解码看作一次传递一个 token，而预填充是一次传递多个 token。

Yeah. If we say like we can think of decode as being a pass with one and then prefill being a pass with many.

好的。是的。是的。所以也许像前缀。当然。随便。

Okay. Yeah. Yeah. So maybe like prefix. Sure. Whatever.

好的。内存。所以对于预填充的 token，你不存储 KV 缓存。我想也许我们可以实际画出预填充在这里如何出现。

Okay. Memory. So you're not storing the KV cache if you're for the tokens that are the prefill tokens. I think maybe maybe sort of let's draw actually how prefill shows up here.

如果我可以澄清一下，我们做一点解码像这样。

If I may clarify, so we do a bit of decode like this.

我们可能实际上会回来做更多的预填充。比如你想象一个聊天会话：用户说点什么，AI 生成回复，然后用户再说点什么，我们预填充这个。所以也许这才是更常见的情况，即一般情况，而不是另一种。实际上，这就像你读一个文件之类的。

We may actually come back and do more prefill. Like if you think of a chat session: the user says something, the AI generates a response, then the user says something else, and we prefill this. So maybe this is the more common case, the general case, rather than the other. In fact, this is like you read a file or something.

读一个文件，或者就像 AI 在响应用户输入或工具调用，或任何不是生成的内容。

Read a file or just like the AI is responding to a user input or tool call or anything that's not generated.

对。好。那么假设我们在这里。你基本上需要加载——你之前已经计算好了所有这些。

Yep. Okay. So suppose we're here. You will need to load basically the — you will have calculated all of this previously.

嗯。

Mhm.

所以只是之前所有内容的 KV。但这的内存成本是多少？如果你在做 Flash Attention，内存带宽成本——是的，它基本上是临时的。它甚至不会进入主存。忽略它就好。

So just the KV of everything that came before. But what is the memory cost of this? Well, memory bandwidth cost of this if you're doing flash attention, it would — yeah, it's basically temporary. It doesn't even go to main memory. Just ignore it.

好。那么它就只是之前的所有内容。所以不就是那样吗？

Okay. So then it would just be everything that came before. So is it not just that then?

对，实际上对内存时间完全没有调整。

Yeah, there's actually no adjustment at all to the memory time.

太好了。哦，所以这是一个非常微小的改动。对。为了适应。

Great. Oh, so it's a very trivial change. Yeah. To accommodate.

所以这个项让它贵了 5 倍。那为什么会这样？或者说，这告诉我们关于我们试图在这里学到什么？这实际上告诉了我们什么？这帮助我们固定了什么变量？

So this term is making it 5x more expensive. Now why would that be? Or what does that tell us about what we are trying to learn here? What does that actually tell us? What variable does this help us clamp?

嗯，算力大概变成了五倍——唯一可能改变的是算力因此贵了 5 倍。所以这是一次 pass 的时间，但实际上 token 数量也大了那么多。所以我想我们实际上想要的是每 token 的成本，或者说每 token 的时间。

Well, the compute has presumably gotten five — the only thing that could have changed is the compute 5x more expensive as a result. So this is the time for one pass, but actually the amount of tokens is that much larger. So I guess we want the cost per token in fact, or the time per token.

所以我不确定我理解了——这是用于处理前缀中的下一个 token？

So I'm not sure I understood — this is for processing the next token in prefix?

呃，实际上是为了处理整个批次。所以以这个成本，我们处理了这么多 token，就像让它预填充。

Uh, well actually for processing the entire batch. So at this cost we have processed this many tokens, like let it prefilled.

对。或者我猜预——就像这个 pass，不是这个前缀，但它是这个成本。

Yeah. Or I guess pre — like of the pass, like not this prefix but it's this cost.

好。那么我们就改成一个 pass 吧。所以这贵了 5 倍。

Okay. So let's just change as a pass we can. So this is 5x more expensive.

输入贵了 5 倍？

Input is 5x more expensive?

不，输出更贵。

No, output is more expensive.

输出贵了 5 倍。

Output is 5x more expensive.

所以我们想要得出的结果是：预填充受算力限制，而解码受内存带宽限制。

So the result we want to work towards is that prefill is compute limited and decode is memory bandwidth limited.

我们为什么不这样做？我们为什么不直接用 len_pass 作为 x 轴来画图？

Why don't we do this? Why don't we just chart it with len_pass on the x-axis?

对。

Yep.

T 在 y 轴上。T 我们想要每 token 的成本。所以它是 T 除以某个东西。T 除以 pass 的长度。

T on the y-axis. T we want the cost per token. So it'll be T over some stuff. T over length of the pass.

嗯。对，那是对的。

Mhm. Yeah, that'll be right.

好。所以对这个感到困惑：len_pass——当你做预填充时，它似乎应该更高。

Okay. So confused about this: len_pass — it seems like this should be higher when you're doing prefill.

预填充有更大的 len_pass。对。没错。

Prefill has a bigger len_pass. Yeah. Right.

但为什么它更便宜？

But then why is it cheaper?

为什么它的成本更高？对。所以我的意思是，我们要——正是这个除以 len_pass 让一切——所以好，这个会被约掉。这个会被约掉，但然后我们会得到一个——所有这些都会被 len_pass 除，这会让内存成本更便宜。

Why is its cost higher? Yeah. So I mean we're gonna — it's this division by len_pass that actually makes it all — so okay, this is going to divide out. This is going to divide out, but then we're going to get a — all of this is going to divide by len_pass and it's going to make the memory cost cheaper.

好。对，让我想想。那么我们先画一条线——基本上我们会有四条不同的线。我们先做预填充。实际上我们先做解码。

Okay. Yeah, let me think about this then. So let's do one line for — basically we'll have four different lines. Let's do prefill first. And actually let's do decode first.

哦。所以实际上我会——len_pass 为 1 时是解码。当它更大时是前缀。

Oh. So actually I will — len_pass when it's one that is decode. When it is bigger that is prefix.

好。我明白了。我明白了。有道理。回到正题。所以 T_compute 如果你有——基本上这个除以 len_pass 就是这个量。所以它实际上不随 t 变化。所以它只是一条平线。这是 T_compute，这是——那是解码，对吧？

Okay. I see. I see. That makes sense. Getting back to it. So T_compute if you have — basically just this divided by len_pass is just this amount. So this actually does not vary based on t. So it'll just be some flat value like this. And this is T_compute, and this is — that's decode, right?

嗯，现在 T_memory 我们有整个这个东西除以 len_pass。好吧，上面是什么其实不重要。它看起来就像这样。对。假设这是 T_memory。这又是解码。所以随着前缀或 pass 的长度增加，你的内存带宽时间下降。这意味着，如果你之前受内存带宽瓶颈，你可以避免受内存带宽瓶颈。他们为预填充收费比解码便宜 5 倍，这确实表明他们在很大程度上受内存带宽瓶颈，以至于对他们来说至少——因为 t 等价于成本，对吧？这是租用计算机的成本。这实际上就像——这个会是 1，这个会是 5。没错。所以它实际上严重受内存带宽限制。真实的图看起来有点像——真实的图看起来有点像那样。

Um, now T_memory we have this whole thing divided by len_pass. Well, it doesn't really matter what's up there. It'll just be something that looks like this. Right. Say this is T_memory. This is decode again. So as the length of the prefix goes up or pass, your memory bandwidth time declines. And that means that to the extent that you were bottlenecked on memory bandwidth before, you can avoid being bottlenecked on memory bandwidth. The fact that they are charging 5x less for prefill than decode does suggest that they are bottlenecked on memory bandwidth to quite a degree, such that for them at least — because t is equivalent to cost, right? It's the cost of renting a computer. This is actually like — this would be at one and this would be at five. That's right. So it is in fact tremendously memory bandwidth bound. The real graph looks something like — the real graph looks something like that.

对。我的意思是它仍然交叉，但——

Yeah. I mean it still crosses but —

对，没错。所以我这样画。对，没错。然后这是解码上内存时间和算力时间之间的差距。

Yeah exactly. So let me do it this way. Yeah that's right. And then this is the gap on decode between the memory and the compute time.

对。好。有意思。另一个有趣的问题是为什么缓存这么便宜。

Yeah. Okay. Interesting. Another interesting one would be why caches are so much cheaper.

对。所以我想如果我没记错的话，缓存大约是 10 倍——根据所有这些模型的定价，写入缓存更贵，但如果你命中缓存，它便宜 10 倍。那么这是怎么回事？大概这是将某些东西保留在 HBM 中而不是直接驱逐它的成本。但如果你确实把它保留在 HBM 中，那么再次加载就更便宜，对吧？

Yeah. So I think if I remember correctly, caches are like 10x — it's more expensive to write to cache according to the pricing on all these models, but if you do hit a cache, it's 10x cheaper. So what is going on? Presumably this is the cost of keeping something in HBM rather than just evacuating it. But if you do keep it in HBM, then it's cheaper to load again, right?

所以有两种方式可以产生 token——或者说一个 token 的 KV 缓存。你可以从头开始，从底层的 token ID（很小）计算出来。或者你可以之前已经产生过它，并把它存储在内存的某个地方。所以成本比实际上是在谈论这两种产生机制之间的比率。缓存未命中意味着你已经从所有内存中删除了它，你必须直接从 token 重新计算它。

So there's two ways you can produce tokens — or the KV cache for a token. You can just produce it from scratch by computing it from the underlying token IDs which are tiny. Or you can previously have produced it and stored it in memory somewhere. So the cost ratio is really talking about the ratio between those two mechanisms of producing it. A cache miss means you've deleted it from all your memories and you have to recompute it from the tokens directly.

事实上，你甚至可以更进一步，考虑你把它存储在哪个内存层级。所以你可以把它存储在 HBM 中。还有比 HBM 更慢更便宜的内存，比如主机上的 DDR 或闪存。所以你可以做的一件事是计算在每个内存层级中存储在哪里有意义。这与你要存储多长时间有关。所以我们想看看几个不同内存层级的存储成本，以及重新物化的成本。Remat 指的是重新物化的成本，即删除后从头重建所有 KV 缓存。所以我们重新物化它。基本上这将花费上下文长度的成本。实际上我们会看每 token 的成本，这样我们就不需要到处带着这个上下文长度。

In fact, you can maybe even take that a step further and think about which memory tier you store it in. So you could store it in HBM. There are other slower and cheaper memories than HBM, like DDR on your host or flash as well. And so one of the things you can do is a calculation of where it makes sense to be in each memory tier. And this is related to how long you're going to store for. So we want to look at the cost of storage in a few different memory tiers and also the cost of rematerialization. Remat means the cost to rematerialize, i.e., rebuild all of the KV cache from scratch after having deleted it. So we rematerialize it. And basically this is going to cost the length of the context. Actually we'll look at cost per token so that we don't need to carry around this length of context everywhere.

所以，要重新计算一个 token 的 KV 缓存，我只需要对整个模型运行一次前向传播。这就会产生计算时间。我必须按照 GPU 的速度重新运行计算，然后乘以我的 GPU 每秒成本。

So to rematerialize one token of KV cache, I just need to run a forward pass on the whole model. And then this is going to be the compute time. I have to rerun the compute at whatever speed my GPU does it, and then I multiply it by my GPU dollars per second.

抱歉，问个很基础的问题。为什么没有二次项？

Sorry, extremely question. Why is there not a quadratic term?

是的。确实存在二次项，但它作为近似出现在计算中。我选择把它去掉了。我快速给你展示一下。如果你看每个 token 的成本或每个 token 的浮点运算次数，一部分来自权重矩阵乘法，随上下文长度变化；另一部分来自 KV 缓存，随注意力范围线性增长。这个斜率非常低，以至于画出来时，它几乎就是一条水平线。

Yeah. So there is a quadratic term, but it shows up in the compute as an approximation. I chose to remove it. I'll just show you quickly what that looks like. If you look at the cost per token or the number of flops per token, there are flops from the weight matrix multiplies as a function of context length, and then there are flops from doing the KV cache, which goes up linearly with the amount you attend to. The slope on this is so low that when you draw it, it's very well approximated by a flat line.

所以，要到数百万 token 的量级才会注意到二次项或线性项的影响。因此不太相关。

So you start to notice the effect of the quadratic or linear term up in the millions of tokens or so. So just not super relevant.

那么，如果这是真的，为什么没有公司拥有超过百万 token 的上下文长度？

So what is the reason that there's no company which has over a million token context length, if this is true?

是的。长上下文有两个成本。一个是内存带宽成本，我们花了很多时间分析，就是这个。另一个是计算成本。计算成本几乎总是由基本原理决定，其斜率远小于内存带宽成本。所以限制你拥有极大上下文的主要因素是内存容量，正是这个效应。

Yeah. So there are two costs of long context. One is the memory bandwidth cost, which we've spent a lot of time analyzing. That's this thing. And then the other one is the compute cost. The compute cost is almost always forced by fundamental principles to be a much smaller slope than the memory bandwidth cost. So the primary thing that limits you to have really large contexts is memory capacity, which is exactly this effect.

所以有一个想法，Dario 在播客里说过，其他人也说过：我们不需要为 AGI 做持续学习，上下文学习就够了。如果你相信这一点，那么你就必须认为我们需要达到 1 亿或 1000 亿 token 的上下文长度，才能拥有一个相当于与你共事一个月的员工。也许从注意力机制或其他方面来看，这不再成立。

And so there's this idea that Dario said on the podcast and others have said, which is we don't need continual learning for AGI; in-context learning is enough. And if you believe that, then you have to think that we need to get to 100 million or 100 billion token context length to have an employee that is equivalent to working with you for a month. Now maybe that's no longer true as far as attention or something.

但如果你这么认为，那么一些 ML 基础设施就必须改变，才能支持 1 亿 token 的上下文长度，比如内存带宽。

But if you think that, then some ML infra thing would have to change to allow for 100 million token context lengths, like the memory bandwidth.

我是说，稀疏注意力当然能帮你摆脱困境，因为它带来了平方根级别的改进。但如果你看看模型上下文长度的历史，从早期的 GPT-3 到 GPT-4（我不记得具体何时转变），它们从大约 8K 飙升到 100K 或 200K，然后过去一两年都徘徊在这个水平。我认为这实际上表明这是一个相当平衡的成本点，大幅超越它将是成本 prohibitive 的。

I mean, sparse attention gives you an out for sure, because you get a square root improvement. But I think if you look at the history of context lengths of models, from earlier models like GPT-3 maybe to GPT-4, I don't remember when the transition happened exactly, they shot up from about 8K to 100K or 200K, and then for the last year or two they've all been hovering around there. I think that actually indicates that that's sort of the reasonably balanced cost point, and going massively beyond that would be cost prohibitive.

不是因为计算成本，而是因为内存带宽成本。

Not because of the compute cost, because the memory bandwidth cost.

是的。所以我实际上看不到很好的解决路径。HBM 就那样了，没有大幅改善。

Yeah. So I actually don't see a very good path to solving that. The HBM is where it is, it's not getting hugely better.

那为什么稀疏注意力不能解决这个问题？

And why doesn't sparse attention solve that?

稀疏注意力是一个很大的改进。也许这已经被考虑进去了。但它不是无限的改进，因为如果太稀疏，你会损失太多质量。

Sparse attention is a big improvement. Maybe that is priced in already. It's not an infinite improvement because if you go too sparse, you lose too much quality.

是的。

Yeah.

但没错，我的意思是，经验结果是上下文长度并没有增加那么多。我认为这是因为内存墙没有解决方案。

But yeah, I mean the empirical result is that the context lengths haven't been increasing that much. And I think it's because there is no solution to the memory wall.

有意思。

Interesting.

那么，这些不同方式生成或重新合成 KV 缓存的成本是多少？从头计算基于我的 GPU 时间，我必须做一定数量的乘法运算才能生成它。存储在 HBM 中实际上取决于每个 token 的字节数。所以我需要每个 token 一定数量的字节，然后将其存储在 HBM 中。这会占用我的一些 HBM 容量。所以一种思考方式是，如果我有太多这些东西放在 HBM 中，如果我用未使用的 KV 缓存填满了 HBM，我就无法使用那个 GPU。那么我该如何定价？也许我说它的成本与我使用的 HBM 比例成正比。所以还有乘以 GPU 美元。

So what is the cost of these different ways of producing or resynthesizing the KV cache? Computing it from scratch is based on my GPU time. I have to do a certain amount of multiplies in order to produce it. Storing in HBM really goes as the bytes per token. So I need to have some number of bytes per token and then store this in the HBM. So it's going to use up some of my HBM capacity. So a way to think of this is that if I have too many of these things sitting in HBM, if I fill up my HBM with just KV caches that I'm not using, I can't use that GPU. So how do I price that? Maybe I say that the cost of it is proportional to the fraction of the HBM I'm using. So there's also times GPU dollars.

然后我们再做一个内存层级，比如 DDR，改为存储在 DDR 中。闪存和 DDR 也是类似的情况。实际上我把这些放错了列。我本想分成两列。我想区分的是一是检索的时间成本，二是存储的成本，即持有的成本。

And then let's just do one more memory tier and say something like DDR, store in DDR instead. The same kind of thing goes for flash and for DDR. I put these in the wrong columns actually. I meant to make two columns. The distinction I want to make is that there is the time cost to retrieve, and then there is a cost to store, a cost to hold on.

所以重新计算有检索成本，存储成本为零，因为我们已经删除了。这是我放错位置的那个。这实际上是持有的成本。所以我重写一下。所以我们有：如果只是存储在 HBM 中，它有这种成本曲线。然后如果存储在 DDR 中，实际上需要一些时间。所以这里也一样：每个 token 的字节数除以 DDR 容量，再乘以 DDR 每秒成本。但现在检索成本比 HBM 高，因为我们需要把它复制到 HBM 中。所以这是每个 token 的字节数除以 DDR 带宽，并且这也消耗了一些 DDR。

So rematerialization has a cost to retrieve and has zero cost to store because we've deleted it. This is the one that I put in the wrong location. This is actually the cost to hold on. So I will rewrite it. So we have this: if we're just storing it in HBM, it has this sort of cost profile. And then if we store in DDR, it's actually going to take some time. So we get the same thing here: bytes per token over DDR capacity times DDR cost per second. But now this has a cost to retrieve that is higher than the HBM because we need to copy it into the HBM. And so this is bytes per token over DDR bandwidth, and this consumes some amount of the DDR as well.

而且每个 scale-up 都有 DDR 和闪存。

And every scale-up has DDR and flash.

这实际上是一个部署问题，所以你可以选择。Nvidia 确实以这种形式部署。它两者都有。

This is really a deployment question, and so you can choose that. Nvidia does deploy in this form. It has both.

为什么检索 HBM 的成本不是内存带宽，或者字节数除以内存带宽？

Why isn't the cost to retrieve HBM the memory bandwidth or the bytes divided by memory bandwidth?

是的，我的意思是这取决于你如何定义检索。这里我把检索定义为将其移入 HBM，以便你能够真正开始对其进行推理。因为如果它已经在 HBM 中，你可以一边从 HBM 获取它一边进行计算，例如。

Yeah, I mean it depends what you define retrieve to be. Here I'm defining retrieve to be move it into HBM so that you can start actually doing inference on it. And because if it's already in HBM, you can be doing compute while you're getting it from HBM desk, for example.

是的。

Yeah.

所以这是三件事，我想我顺序排错了。

So these are three things, and I guess I ordered them wrong.

一般来说，如果你在平衡两种成本，并且内存层次结构中有不同的层级，你应该预期当这个成本上升时，那个成本会下降。所以你可以大致看出零点的位置。我本应该按顺序排列：这个第一，这个第二，这个第三。如果你要持有它很短的时间，那么所有这些都要乘以持有时间。

In general, if you're balancing two costs and you have different tiers in the memory hierarchy, you should expect as this cost goes up, this cost should go down. So you can kind of see where the zeros are. I should have ordered them: this one first, this one second, and this one third. If you're going to hold on to it for a very short amount of time, then all of this is multiplied by the hold time.

这个也是，那个也是。

This one is, and so is this one.

有趣的是，它们有不同的写入价格，你在 API 中指定 5 分钟还是 1 小时。

Interestingly, they have different prices to write for, and you specify this in the API for 5 minutes versus an hour.

是的，这表明 5 分钟对应 HBM，1 小时对应 DDR。

Yeah, which suggests that the 5 minutes is HBM and the hour is DDR.

我认为这是一个相当好的假设。也有可能再降一级，是 DDR 对比闪存。

I think that's a pretty good assumption. It could also turn out that it's one tier down and it's DDR versus flash.

嗯，好的，有意思。价格差异我记得……我查一下。基础输入 token 是每百万个 5 美元，这意味着有折扣。

Yeah. Okay. Interesting. And the price difference I think was... I'll look it up. So the base input tokens is five per million tokens basic, which means rebate.

对，是 5 美元。这是 5 美元吗？用于所谓的检索，然后写入（大概是 HBM）5 分钟是 6.25 美元。

Yeah, that's five. Is this five dollars? To like retrieve, quote unquote, and then to write to presumably HBM write for 5 minutes is 6.25.

所以实际上我们可能可以通过持续时间来判断是哪个内存层级。实际上，持续时间可能就说明了：5 分钟对比 1 小时。

So actually we might be able to determine which memory tier it is by the durations. Actually, the duration probably tells it: 5 minutes versus 1 hour.

对，没错。我认为这很可能就是你所在内存层级的耗尽时间。这意味着，既然我知道我要持有某个东西 5 分钟，我想选择一个每 5 分钟能读取一次的内存——大致上每 5 分钟能读取整个内存一次。那就是内存的耗尽时间。所以如果我用存储容量除以存储带宽，我希望这个值等于 5 分钟左右。

Yeah, exactly. I think this will probably end up being the drain time of the memory tier you're in. So what that means is, given that I know I'm going to be holding something for 5 minutes, I would like to pick a memory that I can read every 5 minutes — I can read the whole memory once per 5 minutes, ballpark. So that is the drain time of the memory. So if I take the storage capacity over storage bandwidth, I would like this to be equal to 5 minutes or something like that.

实际上我们为 HBM 做过这个计算。对于 HBM，我们知道这个数字是 20 毫秒。所以 HBM 太短了。DDR 可能差一到两个数量级。所以这大概在秒级，比如 1 到 10 秒。然后这个……我不记得具体数字，但通常来说，越慢的层级，闪存大概在 1 分钟量级，而机械硬盘差别巨大，我认为在 1 小时量级。所以这可能表明层级是闪存和机械硬盘。抱歉，为什么这个计算是存储容量除以带宽？

And so actually we did this calculation for HBM. For HBM we know that this number is 20 milliseconds. So HBM is much too short. DDR could be about an order of magnitude or two off from this. So this is probably in the order of seconds, like 1 to 10 seconds. And then this is really... I don't have these numbers memorized, but generally as you go to slower tiers, flash is plausibly in the order of 1 minute, and spinning disc, which is massively different, I think is on the order of 1 hour. So this might actually identify that the tiers are probably flash and spinning disc. Sorry, why is this the calculation: storage capacity divided by bandwidth?

所以你有许多不同的内存层级，比如我们列出了四个。你选择哪个内存层级是为了最小化成本。所以你要考虑：你使用了设备的多少比例？你使用一部分设备来持有它，再使用一部分设备来检索它。假设我用了 10% 的设备。我想让这两个比例相等。这标志着我找到了正确的点。假设我有一些运行时间。我会在整个时间内持有它。所以这是持有时间，然后这里有一些时间是检索时间。我想让这两个成本相等。我希望检索时间等于持有时间乘以容量比例。

So you've got a bunch of different memory tiers like we've listed four of them. Your choice of which memory tier is like you want to minimize the cost. So you are like: what fraction of the device are you using? You're using some fraction of the device for holding onto it and then you're using some fraction of the device to retrieve it. So let's say I'm using like 10% of the device. And I want to equalize those two fractions. That's a sign that I've hit the right thing. So let's say I've got some runtime here. I'm going to hold on for all of this time. So this is the hold time, and then there's going to be some amount of time here which is time to retrieve. I want to equalize the costs, these two costs. I want the retrieval time to be equal to the hold time times the fraction of capacity.

嗯。这就是我能同时持有的其他东西的数量。基本上，你想把东西存进去那么久，使得它在里面的时间大致等于把所有东西放进去和取出来的时间。

Mhm. This is how many other things I can hold simultaneously. Basically, you want to store things in there for so long such that the amount of time it's in there is kind of the time to get all your things in there and out.

是的，基本上说得通。我认为这可能表明这是闪存和机械硬盘两个层级。我有点惊讶居然还在用机械硬盘，因为它是一项很老的技术。

Yeah, basically it makes sense. I think that probably indicates that this is the two tiers of flash and spinning disc. I'm kind of shocked to see spinning disc being used at all because it's such an old technology.

我的意思是，它慢到需要一个小时才能加载全部容量，这也很疯狂……它确实是一项不太有吸引力的技术，但在某些地方还是有用的。

I mean, it's also crazy that it's so slow that it takes an hour to load its full capacity into it and then... like it's a really unattractive technology, but it's useful in some places.

那么，我们坐下来是因为我想问你一些不需要黑板的问题。你有一篇非常有趣的博客文章，其中谈到在高层次上，不同密码协议的架构看起来很像神经网络。而且存在一种趋同演化：它们都需要混淆所有输入的信息。对于密码协议，是为了确保哈希函数的每个新输入都会完全打乱结果。对于神经网络，当然，它们需要考虑一条信息如何改变你对另一条信息的理解。这是一个非常有趣的观点。我想，在高层次上，它们试图做的事情在某种意义上是对立的，对吧？密码协议试图将有结构的信息变得看起来与随机无异。而神经网络试图将看起来像随机蛋白质序列、DNA、乱码文本的东西，提取出更高层次的结构。所以它们有相似的高层机制，但实际上是在做相反的事情。嗯，我想知道你怎么看。

So, we're sitting down because I want to ask you some questions that I guess don't need a blackboard. You have this extremely interesting blog post where you talk about how at a high level the architecture of different cryptographic protocols looks a lot like neural networks. And there's this convergent evolution where they both need to jumble information across all their inputs. For cryptographic protocols, it's to make sure that each new input into a hash function will totally scramble what happens. For neural networks, of course, they need to consider how this piece of information changes what you should make of that other piece of information. That is an extremely interesting point. I guess at a high level, the difference in what they're trying to do in some sense, they're trying to do the inverse thing, right? Cryptographic protocols are trying to take information which has structure and make it look indistinguishable from randomness. And neural networks are trying to take things which look like random protein sequences, DNA, garbled text, and extract higher level structure from it. So they have similar high-level mechanisms but they're actually kind of trying to do the opposite things. Yeah, I wonder what you make of that.

是的。我的意思是，混合——我试图寻找其他出现混合、打乱的例子。实际上甚至有一个物理例子：你在搅拌东西，做蛋糕时搅拌面糊，先这样搅再那样搅的想法其实也不算太差。但除此之外，回到数字世界，有一些差异，你指出的那一点是一个很强的差异。

Yeah. So, I mean the mixing — I tried to look for other examples where mixing, scrambling, mixing shows up as well. There's actually almost even a physical example where you're stirring something, you're making a cake and you want to stir the batter, and literally the idea like first stir it this way and then stir it this way is actually not too bad of an approach. But beyond that, back to the digital world, there are some differences and the one you call out is a pretty strong difference.

让神经网络具有可解释性的东西是梯度下降，因为你可以对它求导，得到有意义的导数。我们做了很多工作来不让导数过于复杂，比如残差连接让它保持简洁可控，层归一化也有帮助。对密码学密码的最大攻击之一也是对密码求导。密码运行在二元域中，就是二进制，而神经网络理论上运行在实数域中。你必须对二进制数求导，但你完全可以对密码求导——这被称为差分密码分析。基本上，如果你对输入做一个小的改变，一个设计良好的密码会让输出的改变非常大。

The thing that makes neural networks interpretable is gradient descent, because you can differentiate them and get a meaningful derivative. We do a lot of work to not overcomplicate the derivative, like residual connections keep it contained and simple, and layer norm helps too. One of the biggest attacks against cryptographic ciphers is also to differentiate the cipher. Ciphers run in the field of two elements, just binary, whereas neural nets run in the field of real numbers. You have to differentiate with respect to binary numbers, but you absolutely can differentiate a cipher—this is called differential cryptanalysis. Basically, if you take a small difference in the input, a well-designed cipher makes the difference in the output very large.

所以在那一点上，优化目标是关于复杂化的。它们没有同样的残差连接或层归一化来……

So the optimization goals at that point are about complexifying. They don't have the same residual connections or layer norms that would...

是的。我觉得两者融合的一个地方是后门。在后门大语言模型中，你试图隐藏一个输入到反向传播中。这实际上就是密码学中雪崩效应的体现。对图像分类模型的对抗攻击——找到一个非常小的扰动，完全改变分类结果——这在密码中是常见情况，而在神经网络中是不希望出现的情况。

Yeah. I guess a place where the two merge is backdoors. With a backdoor LLM, you're trying to hide an input into the backward pass. This is actually where you get exactly the avalanche property that ciphers have. Adversarial attacks on image classification models—finding a very small perturbation that totally changes the classification—that is the common case in ciphers, whereas it's the undesired case in neural nets.

好的。我刚才问你：神经网络真的被用于密码学了吗？我们意识到可能最好在黑板上演示。所以我很好奇，它们真的被用于密码学了吗？

Okay. So I was asking you: have neural networks actually been used for cryptography? And we realized it might be better to do this on the blackboard. So I'm curious, are they actually being used for cryptography?

是的。用神经网络做密码学——创造一个新密码——是一个非常危险的事情；几乎所有的都被破解了，99% 都被破解了。所以可能不是一个好的起点。但另一个方向至少在一个明确的案例中非常富有成效。密码学中有一个结构被引入到神经网络中，叫做 Feistel 密码或 Feistel 网络。想法是：你有一个不可逆的函数 f，但你喜欢它因为它能做有趣的事情，比如 MLP。你想用它构建一个可逆的东西。这个结构是一个双输入函数：我们对 x 应用 f，但我们需要记住 x，所以把 x 放在这里，我们也不能丢掉 y，所以记住 y 并把它们加在一起，形成一个元组。要逆推，如果我有这个输出并想恢复 X 和 Y，我可以轻松恢复 X——它就在那里。然后恢复 Y，如果这个东西叫 Z，我可以通过 Z 减去 f(X) 来恢复 Y，因为我已经恢复了 X。所以这个结构是可逆的。这在密码学中被大量使用，现在仍然如此。通常你希望密码是可逆的，尤其是各层，因为这样有更好的密码学性质。这已经被移植到神经网络中。有一篇 2017/18 年的论文叫 RevNets，可逆网络。它用完全相同的结构使整个网络可逆。对于 Transformer 层，我们有函数 f 即 Transformer 层。通常我们有一个输入和一个残差连接输出，像这样相加。但现在变体是我们有两个输入，X 和 Y。X 经过函数，加到 Y 上，这成为新的 X，即输出 X。而这个 x 成为输出 y。这实际上是在做两层之前的残差连接。由于这个结构，整个东西是可逆的。

Yeah. Using neural nets for cryptography—creating a new cipher—is a very dangerous proposition; almost all of them are broken, 99% of them. So probably a bad place to start. But the other direction has been quite productive in at least one clear case. There's a construction in ciphers that was imported into neural nets called a Feistel cipher or Feistel network. The idea is you have some function f which is not invertible, but you like it because it does interesting things like an MLP. You'd like to build something out of it that is invertible. The construction is a two-input function: we apply f to x, but we need to remember x, so we stick x over here, and we also can't drop y, so we remember y and add them together, forming a tuple. To invert, if I have this output and want to recover X and Y, I can easily recover X—it's right there. Then to recover Y, if this thing was called Z, I can recover Y by Z minus f of X because I've already recovered X. So this construction is invertible. This was used in ciphers a ton, still is. Often you want ciphers to be invertible, especially the layers, because that has better cryptographic properties. This has been ported into neural nets. There's a 2017/18 paper called RevNets, reversible networks. It makes the entire network invertible with exactly this construction. For a transformer layer, we have this function f which is the transformer layer. Normally we have an input and a residual connection coming out, added like this. But now the variation is we have two inputs, X and Y. X goes through the function, gets added to Y, and this becomes the new X, the output X. And this x becomes the output y. This is actually doing the residual connection from two layers back. Because of this construction, the whole thing is invertible.

为什么我在乎？可逆有什么用？

Why do I care? What does invertible matter for?

它可能有趣的一个大方面是训练。如果考虑训练中的前向传播，假设我有四层，按 0-1-2-3 顺序运行。我必须将所有激活值写入 HBM，所以 HBM 占用与层数成线性关系。这可能是训练中最大的内存占用。然后我运行反向传播，反向读取它们。RevNet 论文的想法是，因为它是可逆的，我根本不需要存储这些。我可以在运行反向传播时完全重新计算它们。

The big thing it can be interesting for is training. If I think of a forward pass during training, let's say I have four layers, I run them in 0-1-2-3 order. I have to write all the activations to HBM, so I get an HBM footprint that is linear in the number of layers. This can be the largest memory footprint during training. Then I run the backward pass and read them in reverse. The idea of the RevNet paper is that because it's invertible, I don't need to store this at all. I can completely rematerialize it when running my backward pass.

所以我运行前向传播，然后在运行反向传播时，我同时以锁步方式撤销我之前做的所有前向传播步骤，以便获得我在这里需要的激活值。这最终节省了内存，是个好主意。

So I run my forward pass and then when I'm running my backward pass, I'm simultaneously in lockstep undoing all of the forward pass steps that I did in order to have the activations that I need here. So this ends up being a memory saving, which is a nice idea.

有意思。从某种意义上说，你是在用更多算力来节省内存。

Interesting. And in some sense, you're spending more compute to save memory.

没错。是的。

That's right. Yeah.

有意思。

Interesting.

嗯。实际上，这和你用 KV 缓存做的正好相反。在 KV 缓存中，你花更多内存来节省算力。

Huh. Actually, it's kind of the opposite of what you're doing with the KV cache. In the KV cache, you're spending more memory to save compute.

是的。考虑到当今硬件的现状，花更多内存来节省算力通常是有利的。

Yeah. Spending more memory to save compute is generally profitable given where hardware are today.

是的。有意思。酷。这太有趣了，对吧？非常感谢你来做这个。我觉得这真的验证了工作室和黑板背后的愿景。

Yeah. Interesting. Cool. That was super fun, right? Thank you so much for doing it. I feel like it really vindicated the vision behind the studio and the blackboard.

酷。非常感谢你来做这个。

Cool. Thanks so much for doing it.

谢谢。

Thanks.