AI 芯片工作原理：从逻辑门到矩阵乘法

我又请到了 Rainer Pope，他是新 AI 芯片公司 Maddock 的 CEO。上次我们聊了数据中心内部的情况，现在我想了解 AI 芯片内部的工作原理。芯片到底是怎么工作的？顺便说一句，我是 Maddock 的天使投资人，所以希望你们设计了一款好芯片。

I'm back with Rainer Pope, who is the CEO of Maddock, which is a new AI chip company. Last time we were talking about what happens inside a data center. Now I want to understand what happens inside an AI chip. How does a chip actually work? Full disclosure, by the way, I am an angel investor in Maddock. So hopefully you have designed a good chip.

我先从芯片设计最小的基本单元讲起，然后逐步构建出整个量产芯片的组成部分。芯片最底层的基本元件是逻辑门，比如与门、或门、非门这些非常简单的单元。它们通过芯片上物理布线的金属走线连接。AI 芯片要计算的主要功能是矩阵乘法，其内部最基础的单元就是乘加运算，也就是对数字对进行乘加。我们来手动演示一下这个计算过程，然后推断出对应的电路。用两个 4 位数字做乘加最容易说明。最清晰的原始操作就是乘加：先乘这两个数，然后加上一个 8 位数。

So I'll start with the very smallest fundamental unit of chip design and we'll sort of build up into what an overall actual production chip's components are. At the very bottom level of a chip, the primitives that we work with are logic gates, which are very simple things like AND, OR, NOT. These are connected together by wires that have to be laid out physically as metal traces on a chip. The main function that AI chips want to compute is multiplication of matrices, and really inside that the fundamental primitive is multiply-accumulate, just like pairs of numbers. So we'll demonstrate what that calculation looks like by hand, and then infer what a circuit would look like for that. It'll be easiest if I do a multiply-accumulate with a four-bit number and another four-bit number. Then the actual clearest primitive is multiply-accumulate: multiply these two terms, then add in an eight-bit number.

为什么这是计算机内部计算的自然基本单元？

Why is this the natural primitive for whatever computation happens inside a computer?

有几个原因。它更高效一些，但对 AI 芯片来说自然的原因在于，如果你看矩阵乘法内部发生了什么——矩阵乘法是什么？简单来说，就是三层循环：for I, for J, for K，执行 output[I][K] += input[I][J] * other_input[J][K]。所以矩阵乘法的每一步都在做乘加。另一个观察是，累加步骤的精度几乎总是高于乘法步骤。这可能对 AI 芯片是特例：你乘的是低精度数字，但累加时误差会快速累积，所以这里需要更高精度。这就是为什么我们选择了 4 位乘法和 8 位加法。

There are a few reasons. It's a little more efficient, but the reason it's natural for AI chips is that if you look at what happens during a matrix multiply—what is matrix multiply? In very short, it is a for loop over I and J and over K of output[I][K] += input[I][J] * other_input[J][K]. So multiply-accumulate happens at every single step of a matrix multiply. The other observation is that the precision will almost always be higher in the accumulation step than in the multiplication step. This is maybe specific to AI chips: you're multiplying low-precision numbers, but when you accumulate, errors accumulate quickly, so you need more precision here. That's why we've chosen a 4-bit multiplication and an 8-bit addition.

让我确认一下我理解了。有两种理解方式：一是结果值会比输入大；二是如果是浮点数，那部分对我来说可能不太直观，但原理可能相同。

Let me make sure I understood that. There's two ways to understand that. One is that the value will be larger than the inputs. And the other is that if it was a floating-point number, it would be maybe that part is less intuitive to me, but it's maybe the same principle.

确实是同样的原理。另一个原理是，当你累加这个数时，你是在累加很多个数，所以会有大量舍入误差累积。而在这个链中只有一次乘法，所以乘法中累积的舍入误差不多。

It is really the same principle. The separate principle is that as you are summing up this number, you are summing up a whole bunch of numbers and so you've got a lot of rounding errors accumulating. Whereas in this case, there's only one multiplication in that chain, so there's not a lot of rounding errors accumulating in the multiplication.

为什么要累加很多个数？这里只有两个数，对吧？

Why are you summing up a whole bunch of numbers? There's just two numbers, right?

求和是在 J 上重复很多次，误差会累积。

The summation happens repeatedly over J many times. Any errors accumulate.

我明白了，有道理。

I see, yes, makes sense.

那么如何手动执行这个计算呢？作为人类，我们可能会分成两步，但可以用长乘法一步完成。首先，将这个 4 位数乘以另一个 4 位数的每一位。我们写出来：第一个 1001 乘以这一位得到它本身；然后左移一位，乘以 0，得到全 0；再左移一位，乘以这个 1，得到 1001；最后一位，又得到全 0。这样就得到了一堆乘法项。在做这个求和的同时，我们也可以把累加项加进去，直接复制过来。所以这是一个五路求和。首先，得到这个中间步骤需要什么逻辑门？我们需要产生所有 16 个部分积。如何产生一个部分积？以这个数字 1 为例：它是 1。如何通过将这个数乘以这个 1 得到这个数？可以用与门实现。当这一位和这一位都是 1 时，结果为 1；如果任一位为 0，则 1 乘以 1 或 0 乘以任何数都是 0。要产生所有这些部分积，我们用了 16 个与门。一般情况下，如果是 P 位乘以 Q 位，就需要 P 乘以 Q 个与门。

So how would we perform this calculation by hand? As a human, we would probably separate it into two, but we can do it all in one using long multiplication. First, multiply this 4-bit number by every single bit position in the other 4-bit number. We write that out: first 1001 multiplied by this bit position gives this number itself. Then shift it up by one, multiply by zero, gives all zeros. Shift one more, multiply by this one, get 1001. Finally, for the last bit position, we get all zeros again. This gives us a bunch of terms to add for the multiplication. While we're doing that summation, we might as well add in the accumulator term as well. So we copy that directly across. So this is a five-way sum we want to compute. First, what logic gates did it take to get to this intermediate step? We needed to produce all 16 of these partial products. How do I produce one partial product? Take this number 1 for example: it is 1. How do we produce this number by multiplying this number by this 1? We can produce that by an AND gate. This number is 1 if both this bit is 1 and this bit is 1. If either is 0, then the multiplication of 1 times 1 or 0 times anything is 0. To produce all this, we ended up consuming 16 AND gates. In the general case, if I were doing a P-bit multiply times a Q-bit multiply, this will be P times Q many ANDs.

最后，我把它们加起来。实际上，大部分工作发生在求和阶段。我来描述这里使用的另一个逻辑门。与门几乎是芯片上最简单、最小的逻辑门。另一个极端是，你通常会用到的最大的逻辑门叫做全加器。从软件角度，你可能认为全加器是把两个 32 位数相加，但这里它只是把三个单比特数相加。可以想象成把 0、1 和 1 相加。相加的结果可能是 0、1、2 或 3，可以用两个二进制位表示。所以它有三个输入位和两个输出位，在这个例子中输出是 1 和 0（二进制 2 是 10）。这也称为三到二压缩器，因为它把三个输入位压缩成两个输出位。两个输入是 X 和 Y 值，再加上一个进位——抱歉，三个输入都是同一列中的位，比如这里一列中的三个位。两个输出我分别画成垂直和水平方向，以匹配这种垂直与水平的布局，表示同一列中的位处于相同的位位置。

Finally, I sum them. Actually, most of the work is going to happen in the summing. Let me describe the other logic gate that we use here. The AND gate is almost the simplest logic gate on a chip, almost the smallest. At the other extreme, typically the very largest logic gate that you'll use is something called a full adder. What this does is, coming from software, you might think a full adder adds 32-bit numbers together. In this case, it just adds three single-bit numbers together. So you can think of it as adding 0, 1, and 1 together. When I add these, the result can be 0, 1, 2, or 3. I can express that in binary using just two bits. So it has three bits as input and two bits as output, which in this case are 1 and 0 (the number 2 in binary is 10). So this is also known as a three-to-two compressor, because it takes three bits of input and produces two bits of output. The two inputs are an X and a Y value and then some carry that came in from like—sorry, the three inputs are all bits that are in the same bit position, like three bits in a column here. The two outputs I have drawn vertically and horizontally to match this vertical versus horizontal layout, expressing that things in the same column are in the same bit position.

而相邻列的情况，比如这里是进位，这里是和。所以，如果全加器的输入是 101，那么输出仍然是 10。如果是 111，输出是 11。如果是 000，输出是 00。如果是 010，输出仍然是 01。所以，它本质上就是统计每列中 1 的个数，并用二进制表示。因此，这个电路实际上可以捕捉到我们人类在按列求和时自然做的事情。我可以展示一下使用全加器求和的一次迭代。我这里求和的方式对人类来说有点不自然。人类会按列求和，然后记住进位。但这里我们不记住进位，而是显式地写出来。我们从最右列向左进行。在最右列，我们对 1 和 1 求和，得到这里的 0 和进位 1。所以，我们在这两位上使用了全加器电路，并输出两位。现在，我们可以对这一列做同样的事。这一列有 1、2、3、4 个数字。我们取前三个，对它们运行全加器，得到 0 和 0 作为输出。所以，这些数的和是 00。这就是全加器应用于所有这些位的结果。当我用掉一些位后，我会把它们划掉，表示已经处理过了。我们继续往下。这里，我取这三个数，相加，得到 1 和 0。我已经处理了这三个数。现在，我取一、二，甚至现在取这三个数，相加，得到 1 和 0，我已经处理了这些数。所以，我应该这样看待：我有一个需要相加的数字网格。我不断对这里的所有位应用全加器，从一列中移除三个数，然后写出两个数作为输出。一直重复，直到最终这里只输出一个数。大概就是这样。这个和可能是错的。

Whereas things that are in adjacent columns, like this is a carry out where this was the sum. So, if the inputs in the full adder are let's say like 101, then the output will still be 10. If it was 111, it'd be 11. If it was 000, it'd be 00. If it was like 010, it'd still be 01. So, yeah, it's just counting essentially the number of things in each column and expressing that in binary. So, this circuit actually can capture what we as humans naturally do when we're summing along the column. So, I can show you one iteration of using the full adder to sum. The way I sum here is going to be a little bit unnatural for humans. Humans would sum along the column and then remember the carry. But instead of remembering the carry, we'll actually explicitly write it out. So, in this we proceed from the rightmost column towards the left. On the rightmost column, we sum the one and the one. And that produces a zero here and a carry of one. So, we've used this full adder circuit on this pair of bits and produced a pair of bits as output. Now, we can do the same thing with this column. We've got a column of 1, 2, 3, 4 numbers. So, maybe we'll take the first three of them, run a full adder on them and that gives us a zero and a zero as output. So, the sum of these is 00. And that's the full adder applied to all of these bits. As I've used up bits, I'm going to cross them out to indicate that I've handled them. Let's just keep going a little bit more. So, we'll go here. I take these three numbers, add them. That gives me a one and a zero. I've dealt with these three numbers. And now I take one, two, and I can even take these three numbers for example right now and add them and that gives me a one and a zero and I've dealt with these numbers. So, the way I should view this is that I have this whole grid of numbers that need to be added. I'm going to just keep applying full adders to all the bits that are here, constantly removing three numbers from a column and then writing out two numbers as output. Keep going with this over and over again until I eventually get just one single number coming out here. Something like that. This is probably the wrong sum.

所以，我在这里描述的方法叫做 Dadda 乘法器。这是使用全加器实现面积高效乘法器的标准方法。我们来量化一下这个电路的规模，以便对大小有个概念，以后可以进行比较。我用了多少个全加器？我开始有多少个数？我有 16 个部分积，即所有这些项与所有这些项的乘积。加上我在这里相加的 8 项。所以，我一开始有 24 位。最终输出 8 位。每一步，我划掉三个数，写出两个数作为结果。所以，每次使用全加器都会消除一位。因此，全加器的数量必须是 24 减去 8，即这个电路中有 16 个全加器。一般来说，在一般情况下也是如此。这个电路中会有 P * Q 个全加器。我只是想理解其中的逻辑。输入位数为 24，即 P * Q + P + Q。没错。输出位数是 P + Q。所以，P * Q + P + Q 减去 (P + Q) 等于 P * Q。没错。所以，我认为这解释或至少暗示了我们选择乘累加操作的第二个原因。第一个原因是它实际上出现在矩阵乘法中，但第二个原因是它给了我们非常简洁的 P * Q 代数。

So, this approach that I've described here, this is called a Dadda multiplier. And this is sort of like the standard for how you do area efficient multipliers using full adders. Let's try and quantify the circuit size of this just so we have a sense of how big things are so we can compare to them later. How many full adders did I use? I started with how many numbers? I have the 16 partial products, which is the product of all of these terms with all of these terms. Plus the eight terms that I'm adding here. So, I started off with 24 bits. And then I produced eight bits on the output eventually. And in every step, I was crossing off three numbers and writing two numbers out as a result. So, every single use of a full adder eliminates one of the bits here. And so, how many full adders must it be? The 24 minus the eight. So, there were 16 full adders in this circuit. In general, this is true in the general case as well. There will be P * Q many full adders in this circuit. I'm just trying to understand the logic of that. So, the input bits 24 is P * Q + P + Q. That's right. And the output bits is just P + Q. And so, P * Q + P + Q minus (P + Q) = P * Q. That's right. So, I think this explains or at least hints that the second reason why we chose to do a multiply accumulate. First reason being that's actually what shows up in matrix matrix multiplication, but second reason being it gave us this very slick P * Q very simple algebra.

所以，我们已经描述了整个过程。这里的每一个原子步骤都变成一个逻辑门，然后导线连接在一起。比如，当我有这三个输入，求和产生这两个输出时，如果映射到物理设备，会有一条导线将这三个输入连接到一个逻辑门，产生这个输出。好的。所以，这就是 AI 芯片内部不同位宽的主要原语。我们将从这里开始，构建如何使用它来运行所有其他所需的操作。

So, we've described this whole procedure. Every single atomic step that I took here becomes a logic gate and then the wires connected together. Like when I had these three inputs that I summed to produce these two outputs, if I think of mapping this to a physical device, there would be a wire that runs connecting all three of these things together into a logic gate that produced this output. Okay. So, this is the main primitive at different bit widths that is inside an AI chip. We're going to build up from here to how would you use that to run all of the other operations you want?

现在问这个问题可能不是时候，但每当英伟达报告说这款芯片可以做 X 个 FP4 或一半的 FP8 时，这似乎暗示这些电路是可互换的。FP4 和 FP8 并不是专用的，但根据你这里的映射方式，似乎如果要在逻辑中映射，你需要一个专用的 FP4 乘累加和一个专用的 FP8 累加？基本上，你能为它们提供资金吗？

This may be the wrong time to ask this question, but whenever Nvidia reports that this chip can do X many FP4 or half as many FP8, it seems to imply that those circuits are fungible. That there's not as dedicated like FP4 versus FP8, but the way you're mapping it out here it seems like you would need if it has to be mapped out in the logic, you would need a dedicated FP4 multiply accumulate and then a dedicated FP8 accumulate? Basically, can you fund them?

如图所示，它们实际上并不是特别可互换的。这实际上是设计芯片时你必须做出的主要选择之一：我该有多少 FP4，多少 FP8？有时我会从客户需求的角度来考虑。另一个角度是，为了平衡 FP4 和 FP8 之间的功耗预算，功耗预算是多少。所以，当他们报告这些数字，并且恰好 FP4 的数量是 FP8 的两倍时，他们只是碰巧选择给所有浮点类型分配等效的芯片面积，结果就变成了这样……为什么比例正好是 2 倍？

As drawn, they're actually not particularly fungible. This is actually one of the main choices you have to make when designing a chip, which is how much of FP4, how much of FP8 do I have? And then sometimes I'll make that consideration from the point of view of what I think is the customer requirement. Another way to take an angle on that is to say what is the power budget to equalize the power budget between FP4 and FP8. But so when they report those numbers and they just happen to be the case that it does 2x as many FP4 as FP8, they just happen to choose to give equivalent die areas to all the floating points and as a result that ended up being like... Why is the ratio exactly 2x?

是的，没错。

Yeah, exactly.

是的。所以部分原因是，我的意思是，这肯定不完全等同于芯片面积。实际上还有一个数据移动的原因，我们稍后可能会在讨论数据进出内存时再回到这一点。从软件层面来看，有一个非常好的特性：我可以将两个四位数字打包到与一个八位数字相同的存储空间中。所以，当我将其存储到内存或类似设备时，芯片内总线的尺寸实际上使得这一点非常完美。实际上，仔细想想，不仅仅是 2 倍，它所占用的面积似乎是二次的。

Yeah. So part of it is, I mean, surely that wouldn't be exactly equivalent to die area. There's a data movement reason actually, and we'll maybe come back to this when we sort of look through how it goes into and out of memories. There's something really nice just from a software level of the fact that I can pack two four-bit numbers into the same storage as an eight-bit number. And so when I store that to a memory or something like that, the sizing of the buses that I wire within the chip actually makes that work out really really nicely. Actually come to think of it, it's not just 2x, it would be the amount of area it takes sounds like is quadratic.

实际上是二次的。是的。

It's quadratic in fact. Yeah.

与位长有关。这就是为什么更低的精度比你预期的更有利。一个非常重要的原因。事实上，英伟达历史上做了一个改变，直到 B100 或 B200，每次你将位精度减半，浮点运算次数就会翻倍。

With the bit length. So that's why a smaller precision is even more favorable than you would have expected. A really big reason. So in fact, Nvidia made a change historically up until B100 or B200, every time you halve the bit precision, you double the flop count.

嗯，这个比例恰好就是因为你提到的原因——由于这种二次缩放，这个比例实际上有点不对。实际上你应该得到比想象中更大的加速。Nvidia 的产品规格已经开始承认这一点，在 B3 100 及之后的产品中，FP4 比 FP8 快三倍，尽管理论上应该是四倍。我这里展示的是整数乘法的简单情况。当处理浮点数（如 FP4 和 FP8）时，还有另一个项——指数——使计算变得复杂。那么从这一点我们能看出什么？我认为你提出的重要观察是，位宽存在二次缩放，这非常有效，也是低精度算术对神经网络如此有效的唯一原因。但我们接下来要做的是，比较乘法本身所占的面积与周围所有电路的面积。我们稍微回顾一下历史，看看在 Tensor Core 出现之前 GPU 是如何工作的？实际上和 CPU 的工作方式相同。

Um, that ratio is exactly like for the reason you said, because of this quadratic scaling, that ratio is actually slightly wrong. It should be like an even bigger speed up than you might otherwise think. Nvidia's product specs have sort of started acknowledging that in B3 100 and beyond where FP4 is three times faster than FP8, though it should be 4x. What I've shown here is the simplest case of integer multiply. When you're dealing with floating point as you do in FP4 and FP8, there's this other term which is the exponent that complicates this calculation. So what can we see already from this? I think the big observation you've made is that there's this quadratic scaling with bit width, which is very effective and is the single reason why low precision arithmetic has worked so well for neural nets. But the other thing we're going to do now is compare the area spent on the multiplication itself with all of the circuitry around it. We'll walk back in time a little bit and see how did GPUs prior to Tensor Cores work? Which is the same way that CPUs worked in fact.

嗯。

Mhm.

所以，我一般会描述一个 CUDA 核心或 CPU。你会有一个寄存器文件，存储若干条目。可能是八个条目，每个是四位数字，但通常是 32 位数字之类的。所以在 CUDA 核心内部，我有一个一定深度的寄存器文件。然后我有一个乘加电路。它要做的是从寄存器文件中取出三个任意寄存器，执行乘加运算，然后写回寄存器文件。它可能写回这个寄存器，但它可以从这个、这个和另一个随机寄存器读取。所以它需要三个这样的输入。这是许多处理器的核心数据路径。大多数处理器看起来像这样：你有一组寄存器，然后有一组逻辑单元或 ALU。我们想分析数据从寄存器文件到 ALU 再返回的成本。最终会有某个电路说：‘我不总是必须选择这个。我可以在任何时间选择任何寄存器。’所以第一个问题是如何构建这个电路？我要找的电路是一个多路选择器。在这种情况下，它会有八个输入，每个来自寄存器文件的一个条目，以及一个输出。这个东西的成本是多少？它由与门和或门构成。那么如何构建呢？我们做最笨的事情：我们形成一个掩码——当我们要读取第三个条目时，我们将每个条目与 1 或 0 进行与运算，取决于是否要读取它，然后将所有结果进行或运算。

So, generically I'll describe a CUDA core or a CPU. You'll have some register file which stores some number of entries. Maybe it's like eight entries of four-bit numbers, but typically 32-bit numbers or something like that. So inside the CUDA core I'll have a register file of some depth. And then I will have my multiply-accumulate circuit. What it's going to do is take three arbitrary registers from this register file, perform the multiply-accumulate, and then write back to the register file. It might write to this one, but it was able to read from this one, this one, and another random one. So it'll take three inputs like this. This is the core data path of many processors. Most processors look like this: you've got some set of registers and then some set of logic units or ALUs. We want to analyze the cost of the data movement from the register file to the ALU and back. Ultimately there's going to be some circuit that says, 'Well, I don't always have to select this guy. I might select any of the registers at any point in time.' So a first question is how can I build a circuit? The circuit I'm going to look for is a mux. In this case it's going to have eight inputs, one from each entry of the register file, and one output. What is the cost of this thing? It's built out of AND and OR. So how do we build it? We do the dumbest thing possible. We form a mask: when we want to read the third entry, we AND every single entry with either one or zero based on whether that's what we want to read, and then we OR all of them together.

好的，只是为了确认我理解了基础。多路选择器做的就是选择。只是选择一个输入。

Okay, just to make sure I understand the basics. What the mux is doing is it's just selecting. Just selecting an input.

只是选择。所以对软件不可见，你说我想要输入三号，这意味着这里有一个多路选择器。那么这个多路选择器的成本是多少？一个 n 输入、操作 P 位的多路选择器。我有 N 行，这里是八行，每行是 P 位宽。我必须对每一位进行与运算，所以得到 N * P 个与门。每个输入我都要问：我要不要屏蔽它？然后我将所有结果进行或运算。所以会有 (N - 1) * P 个或门。也就是说，我有所有这些不同的东西。几乎全是零，但我需要把它们从八个选项压缩成一个选项。所以每一步我都需要将一行或入到现有行中。

Just selecting. So invisible to software, you say I want input number three, that means there's a mux here. So what is the cost of this mux? An n-input mux operating on P bits. I have N rows, that's eight rows, and each row is P bits wide. I have to AND every single bit, so I get N * P many AND gates. Every single input I have to say, am I going to mask it out or not? And then I'm going to OR them all together. So there's going to be (N - 1) * P many OR gates. Which is saying, I've got all of these different things. Almost all of them are zeros, but I need to collapse them down from my eight options down into one option. So every step I need to OR one row into an existing row.

明白了。是的。其实有点好笑，你不会从硬件层面去思考。你只会想，哦，我就选第三个元素，而这么简单的事情竟然是一个相当复杂的电路。

Got it. Yeah. It's actually kind of funny that you would sort of, you don't think at the level of hardware. You sort of just think like, oh, I'll just select element three, and something as simple as that is a quite complicated circuit.

是的。我的意思是，这是所有隐藏数据移动成本的第一步。所以我们来比较一下：我必须支付这个成本，我这里有一个多路选择器，实际上对于乘加操作的三个输入，我另外还有两个副本。所以这里的成本是 3 * N * P 个与门，而实际做我关心的运算的电路只有 P * Q 个门。如果我们代入实际数字，N 是八，那么数据移动就有 * P 个门，而如果 Q 是四，那么加法器乘加器只有 4 * P 个门。

Yeah. I mean, this is the first step of all of the hidden data movement costs. So we're just going to compare: I have to pay this cost, and I've got one mux here, and in fact I have two more copies of that for each of the three inputs to my multiply-accumulate operation. So I have this cost which is like 3 * N * P AND gates over here, compared to this P * Q gates in the actual circuit that is doing the thing I care about. And if we plug in actual numbers, N being eight, I get like * P gates just in the data movement compared to if Q is four, like 4 * P gates just in the adder multiply-adder.

抱歉，这个三从哪里来？

And sorry, where is the three coming from?

三个不同的输入。

Three different inputs here.

明白了。好的。

Got it. Okay.

所以我真正想暗示的是，所有这些工作——随寄存器文件大小缩放，而这是一个非常小的寄存器文件——仅仅是将数据从寄存器文件移动到逻辑单元的工作，就比逻辑单元本身昂贵许多倍。

So the case I'm really hinting at is that all of this work which scales as the size of the register file, and this is a very small register file, all of this work just moving the data from the register file to the logic unit is many times more expensive than the logic unit.

在最近的 Cloud Spectator 7 分析中，评估了近 100 个不同的 GPU 云。Crusoe 是仅有的五个获得金牌级别的云之一。分析发现，像 Crusoe 这样的金牌级提供商，即使 GPU 定价相同，其总拥有成本也比银牌级低 5% 到 15%。这很合理，因为总拥有成本取决于许多不一定体现在标价上的因素，而 Crusoe 对这些因素进行了优化。比如故障检测的能力和更换故障节点的速度。例如，Crusoe 是最早采用 NV Sentinel 的云之一，这是 Nvidia 自己的 GPU 监控和自我修复软件，用于提高 GPU 的正常运行时间、利用率和可靠性。Crusoe 利用了 Nvidia 从所有不同集群和部署中学到的关于芯片故障原因的一切，从而能够更早地发现故障。一旦识别出故障，Crusoe 可以在不到 10 分钟内替换为健康节点。由于不运行裸机，Crusoe 无需花时间安装操作系统或配置驱动程序。

In the most recent Cloud Spectator 7 analysis ranked almost 100 different GPU clouds. Crusoe was one of only five that made the gold tier. The analysis found that gold tier providers like Crusoe had a total cost of ownership that was 5 to 15% lower than silver tier ones, even when they had identical GPU pricing. This makes sense because total cost of ownership is downstream of a bunch of different things that don't necessarily show up in the sticker price, but that Crusoe has optimized. Things like how well you detect faults and how quickly you replace failed nodes. For example, Crusoe was one of the first clouds to adopt NV Sentinel, Nvidia's own GPU monitoring and self-healing software for enhanced GPU uptime, utilization, and reliability. This is Crusoe makes use of everything that Nvidia has learned about why chips fail across all their different fleets and deployments so that Crusoe can catch faults earlier in the process. And once they identify a failure, Crusoe can swap in a healthy node in less than 10 minutes. Because they're not running bare metal, Crusoe doesn't have to spend time installing an operating system or configuring drivers.

看看多路选择器（mux）的样子可能会有所帮助，比如一个两位或四位多路选择器。

It may be helpful to just see what a mux looks like, maybe like a two-bit or a four-bit mux.

好的。我们取一些输入，做一个二路选择器。有两个不同的数作为输入，还有一个选择信号，表示要选这个还是那个。这是一种独热编码。这就是我们开始的情况。然后我们要产生输出，我们关注这个例子，这就是我们得到的实际输入。

Yeah, great. So, we'll take some inputs. We'll have maybe a two-way mux. So, we've got two different numbers, these two inputs. And then we have a selector, which says either I want this one or I want the other one. This is a one-hot encoding. So this is what we all start with. And then the output we want to produce, let's focus on this case. So this is the actual input we got.

嗯。

Yeah.

我们只想把这个结果输出。所以，很繁琐地，我们将这一位与所有这些进行与运算，得到这一位与这一行的与结果，同样地，将这一位与这一行进行与运算，得到全零。这里有四个与门，这里有四个与门。最后，我们只需将这两个进行或运算，得到 1；将这两个进行或运算，得到 1；将这两个进行或运算，得到 0；将这两个进行或运算，得到 1。所以这是四个或门。这实际上看起来有点像加法，事实上我们在这里做了完全相同的与运算。所以如果我们把所有这些加起来，但不用全加器电路来压缩，而是用或门进行非常简单的压缩。但这看起来不像 N 乘以 P。

We just want to produce this guy as the result. And so, very laboriously, what we do is we AND this bit with all of these, and that produces ANDing this bit with this row, and likewise we AND this bit with this row, that produces all zeros. So there were four ANDs here. There are four ANDs here. And then finally we just OR these two together. This gives a one. We OR these two together, this gives a one. We OR these two together gives a zero. We OR these two together and gives a one. And so this is the four ORs. So this actually ends up looking a little bit like addition, in fact we did exactly the same set of ANDs here. So if we've added all of these things together, but then instead of collapsing it by using these full adder circuits, we just could get a very simple collapsing with OR gates. But that I guess doesn't look like N times P.

嗯，对。

Uh, right.

在一般情况下，我们有 N 行，每行有 P 位。所以这给了我们 N 乘以 N 乘以 P 个与门。所以我描述的电路中，几乎所有的成本，大约 7/8，都在于读写寄存器文件。只有很小一部分成本在逻辑单元本身。这就是要解决的问题。这基本上就是 Volta 代之前 Nvidia GPU 的状态，这就是 CUDA 核心内部的东西。这种问题陈述促使了张量核心的引入，张量核心更一般地被称为脉动阵列。

In the general case, we will have N rows, and then we'll have P bits per row. So that gives us the N by N times P many AND gates. So this circuit I've described here, almost all of the cost, like 7/8 of the cost, is in the reading and writing the register file. And only a tiny fraction of the cost is in the logic unit itself. So this is the problem to solve. This essentially was the state of play prior to the Volta generation of Nvidia GPUs. This is what was inside the CUDA cores. And this sort of problem statement is what motivated introduction of Tensor cores, which are more generically called systolic arrays.

嗯。

Mhm.

所以，如果我们考虑如何解决这个问题，我们几乎把所有的电路面积都花在了我们并不真正关心、对软件程序员隐藏的东西上，而我们真正关心的东西只占很小一部分面积。好吧，以某种方式让这个变大，同时保持这个大小不变。这就是目标。所以演进过程是，我们把这个东西固化到硬件中，在这个阶段，这一行就是一个乘加运算，这个单一的东西被固化到硬件中。脉动阵列的想法是向上提升两层循环，把整个循环固化到硬件中。所以，想法是如果我们有一个粒度大得多的固定功能逻辑块，也许我们在输入输出上付出的代价会小得多。

So, if we think about how we're going to solve this problem, we're spending almost all of our circuit area on something that we just really don't care about and is hidden to the software programmer, and the thing that we actually care about is not much of the area. Well, make this one bigger somehow while keeping this at the same size. That's the goal. So the evolution was like we had baked this much into hardware, in this stage that this single line is a multiply accumulate, and this single thing was baked into hardware. The idea of a systolic array is to sort of go two levels of loop up and bake this entire loop out here into hardware. And so, the idea being that if we have a much bigger granularity fixed function piece of logic, maybe the taxes we pay on the input and output are much smaller.

听起来你在建议，如果在矩阵乘法循环中向上一步，就可以将平衡更多地倾斜到计算而非通信。

It sounds like you're suggesting that if you go up one step in the matrix multiply loop, there's some you can tilt the balance more towards compute than communication.

没错。这里我们要利用两个效应。一个是我们可以在每次经过寄存器文件之前做更多事情。另一个是，实际上在这个循环的某些部分，我们可以利用某些东西保持不变。所以让我们直观地看一下这个矩阵乘法。这个循环部分实际上对应一个矩阵向量乘法。我们取一个矩阵乘以一个向量。怎么做呢？每一列乘以向量然后求和。所以我们沿着列求和。所以这个 0 和 3 乘以 3 和 7 然后求和。然后 1 和 2 乘以 3 和 7 然后求和。所以矩阵中的每一个条目都有一个乘加运算。我们画出这四个乘加运算。

That's right. Sort of there's two effects that we're going to take advantage of here. One is just that we can do more stuff before every trip through a register file. And the other thing we're going to take advantage of is in fact in some of this loop we can take advantage of, for example, some certain things staying fixed. So let's sort of visually look at this matrix multiplication. So this portion of the loop corresponds to a matrix vector multiplication in fact. So we'll take a matrix and multiply it by a vector. How do we do this? We take every column gets multiplied by the vector and then summed. So we're going to sum sort of along columns. And so this zero and three gets multiplied by the three and seven and gets summed. And then the one and two gets multiplied by the three and seven and gets summed. So there is a multiply accumulate associated with every single one of these entries in the matrix. So we'll just draw out these four multiply accumulates.

我想确认一下我理解为什么有四个乘加运算。如果输出向量对应的每一列是一个点积，在这个例子中就是两次乘法然后相加。所以你在累加……

I just want to make sure I understand why there's four multiply accumulates. So if each entry in the column that corresponds to the output vector is a dot product, and in this case it'll be like two multiplications and then the addition of these two multiplications. So like you're accumulating...

是的，加法，实际上每个点积只有一个加法，但比如我们从哪里开始初始化？我们的目标是拥有二次方更多的计算。确实如此。我们拥有的计算量是之前的 x 乘以 y 倍。但我们希望通信量只有 x 倍。这就是意图，这样我们就能获得 Y 倍的优势。我们已经布置了乘法。我们要引入一个大小为 2 的向量。这已经符合我们的列目标了。没问题。然而，我们需要管理这个矩阵的通信，它超出了我们的 X 预算。所以，想法是在 AI 场景中，这个矩阵实际上会在很长一段时间内保持不变。所以，不是从外部引入它，我们这里有一个寄存器文件，我们不希望从这个寄存器文件输出太多东西。这就是我们希望以某种方式作为 X 的项。我们不想每个周期都从寄存器文件引入整个矩阵，因为我们没有足够的带宽，那样从寄存器文件布线的成本太高。所以，相反，我们要存储——我们的关键技巧是这个矩阵可以本地存储在脉动阵列中。所以，我们将这些数字 0、1、2、3 存储在一个称为寄存器的门中，物理上存储这些数字，然后我们将反复重用这些数字，用于大量不同的向量。

Yeah, so the addition, really there's only one addition per dot product, but like when do we start with the initialization? The what we're going to aim for is to have quadratically more compute. We do. We've got sort of x times y as much compute as we had before. But we're going to want to somehow aim for having only x times as much communication. And this is sort of the intention so that we get this advantage term going as Y. So we've laid down the multiplications. Bringing in, we're going to want to bring in a vector of size two. And so that's sort of already in line with our columns target. That's fine. However, we need to somehow manage the communication of this matrix, which exceeds our budget of X. And so, the idea is that in an AI context, this matrix is actually going to stay fixed for a long period of time. And so, instead of bringing it in from the outside, so we've got some register file sitting over here, we don't want to have the amount of stuff coming out of this register file. This is the term that we want to go sort of as X in some sense. We don't want to bring this full matrix in from the register file every cycle because we don't have enough, that would cost too much in terms of wiring from the register file. And so, instead we're going to store our key trick is that this matrix can be stored locally to the systolic array. And so, where we'll store these numbers zero, one, two, and three in just like a gate called a register that physically stores these numbers, and we're going to reuse these numbers over and over again for a large number of different vectors.

所以这里的优化在于，矩阵乘法的本质是你可以直接在这个逻辑发生的地方存储这个方形二次项。它比不断换入换出的输入多了一个维度。

And so, the optimization here is that the nature of matrix multiplication is you can store this square quadratic thing directly where the logic is happening. Which is higher dimension than the inputs, which you keep swapping in and out.

没错。我的意思是，这就是矩阵乘法的本质：你进行大量乘法运算才能得到一个输出值。点积就是大量乘法的结果，所以这种优化意味着你可以在得到一个输出值之前塞入大量乘法运算。

That's right. I mean, this is the nature of what a matrix multiplication is: you do a lot of multiplications to get one value out. A dot product is the result of a lot of multiplications, and so that optimization means you can stuff a lot of multiplication in before you get some value out of it.

没错，没错。

That's right. That's right.

是的。为了完整说明具体的样子：我把这两个调换了。三和二。所以这个零和三会与三和七相乘，我们沿着列做点积。我们把一个三和一个七喂到这里。它们参与这个乘法，也参与那个乘法。同样，三也喂到这里和这里。然后我们沿着这里求和，从列的顶部开始喂入零，从底部得到结果。所以直观上看，矩阵中沿着列做点积，这正好对应了脉动阵列在空间上的操作。这是一个垂直求和的点积，这是另一个垂直求和的点积。那么，哪些数据需要进出寄存器文件？输出端有 X 数量的数据，输入端也有 X 数量的数据。所以对于输入和输出向量，我们实现了目标：只有 X 数量的数据进出寄存器文件。这留下了一个问题：权重矩阵存储在脉动阵列本地。它最初是怎么进去的？在某个时刻你需要启动芯片并填充这些数据。它从哪里来？诀窍是我们非常缓慢地做这件事。我们慢慢地将它滴灌进脉动阵列。最简单的策略是运行一个菊花链：把一个数字喂到这里，下一个时钟周期它就会移到下一个条目。我们可以在每一列并行地做这件事，这又带来了大约 X 单位的输入带宽。

Yeah. So, just to complete the picture of how that looks concretely: I swapped the two here. Three and two. So, this zero and three will multiply by the three and seven, and we're going to form a dot product along columns. Somehow we feed a three and a seven in here. These participate in this multiplication and also in this multiplication. Likewise, the three feeds into here and also into here. Then we sum along here, starting at the top of a column we feed in zeros, and coming out the bottom we get results. So, visually, there is a dot product performed along columns in a matrix, and that maps exactly to what is done spatially in the systolic array. This is one dot product summed vertically, and this is a second dot product also summed vertically. Then, what data needs to go into and out of the register file? We have X amount of data coming out on the output, and also X amount of data from the input. So, with respect to the input and output vectors, we met our goal of having only X data going in and out of the register file. This leaves open the question: the weight matrix is stored locally in the systolic array. How did it get there in the first place? At some point you need to boot your chip and populate this data. Where did that come from? The trick is we just do it very slowly. We slowly trickle feed it into the systolic array. The simplest strategy is to run a daisy chain: feed a number into here, and on the next clock cycle it moves down to the next entry. We can do that in every column in parallel, and that gives us another factor of approximately X units of bandwidth coming in.

你能再重复一遍那句话吗？我们知道我们只会偶尔把数字喂入矩阵。我们只想设计一种结构，使得跨越脉动阵列边界的连线数量被限制在 X 而不是 X Y。所以一个简单的策略是：在一个时钟周期内把一个数字喂入脉动阵列的顶行，然后在连续的 Y 个时钟周期里，每次喂入顶行，并将所有其他行下移一行。这样就把来自昂贵寄存器文件的连线限制在 X 量级，而不是 X Y。

Can you repeat that sentence one more time? So, we know we're going to be bringing in numbers only rarely into the matrix. We just want to come up with any construction such that the amount of wiring that crosses this boundary of the systolic array is bounded to X and not X Y. So, a simple strategy is to bring a number into the top row of the systolic array in one clock cycle, and then for Y consecutive clock cycles we bring in the top row every time and shift all other rows down by one. That keeps the wiring from the expensive register file down to a factor of X rather than X Y.

我明白了。好的。所以通信方面有两个问题：通信时间和通信带宽。

I see. Okay. So there are two questions in terms of communication: communication time and communication bandwidth.

而你的意思是，既然我们只加载一次，那就最小化带宽，因为带宽等于腹泻，我们通过更小的通道慢慢加载，因为我们会把这个值在里面放一段时间。

And you're saying, since we're only going to load this in once, let's minimize bandwidth because bandwidth equals diarrhea, and let's load it in slowly over smaller lanes because we're going to keep this value in there for a while.

完全正确。完全正确。有意思。所以，我觉得有趣的是，上次我们讨论跨芯片推理时，我们试图优化的高层目标是增加每内存带宽（即每通信量）的计算量。而在这里，我们也在试图增加相对于从寄存器到逻辑的信息传输的实际乘法或加法量。所以在这两种情况下，你都在试图最大化计算相对于通信的比例。

Exactly. Exactly. Interesting. So, it's interesting to me that when we were talking last time about inference across many chips, the big high-level thing we're trying to optimize for is increase the amount of compute per memory bandwidth, that is to say per communication. And here also we're trying to increase the amount of actual multiplies or additions relative to transporting information from registers to the logic. So, in both cases you're trying to maximize compute relative to communication.

是的。这在整个技术栈中随处可见。这已经接近底层，深入到门电路。还有一个更接近门电路的版本，比如你选择的数字格式的精度。我们看到了同样的效果。在 ALU 的精度和矩阵的大小中，都存在平方-立方律或平方项与线性项的关系。

Yeah. This shows up all the way up and down the stack. This is close to the bottom, into the gates. There's a version made even closer to the gates, like the precision of number format you choose to use. We saw that same effect. There's a square-cube law or a squared versus linear term going on both in the precision of the ALU and in the size of the matrix.

有意思。所以这个单元是下一个更大的单元。我们有乘法电路，然后上面是一个相当大的阵列。我把它画成 2x2，但在老款 TPU 中，他们会描述为 128x128 的这种电路。而这个电路最终是实现矩阵乘法已知最高效的机制。

Interesting. So, this unit is the next bigger unit. We had the multiplication circuit, and then on top of that we have a pretty large array. I drew it as 2 by 2, but in older TPUs they would describe it as 128 by 128 of this circuit. And this circuit ends up being the most efficient known mechanism for implementing a matrix multiply.

我明白了。我们讨论了似乎很明显你应该最大化计算相对于通信的比例。有哪些不明显的权衡，实际上让你夜不能寐，关于我们应该做 X 还是 Y，答案并不明显？

I see. So, we've talked about it seems obvious that you should try to maximize compute relative to communication. What are non-obvious tradeoffs that actually keep you up at night about what we should do X or Y and it's non-obvious what the answer is?

是的。我认为芯片设计中的大多数决策都是尺寸决策。在我们已经画出的图中，AI 芯片都有这个电路：一个脉动阵列，附近有一个提供输入输出的寄存器文件。在这个范围内有两个尺寸问题：我应该把脉动阵列做多大，以及把寄存器文件做多大？脉动阵列尺寸的权衡（这两个问题是耦合的）是：一种思考方式是，我要为芯片面积中用于数据移动的比例设定一个预算。

Yeah. So, I think most of the decisions in chip design are sizing decisions. Already in what we've drawn, AI chips all have this circuit: a systolic array and somewhere near it a register file providing inputs and outputs. The two sizing questions within this scope are: how big should I make my systolic array and how big should I make the register file? The tradeoff for the size of the systolic array, and these two questions are coupled, is: one way to think of it is to say, I'm going to have a budget for what percentage of my chip area I want to spend on data movement.

芯片的时钟周期从何而来？什么决定了它？芯片的时钟周期是什么？

Where does the clock cycle of a chip come in? What determines what that is? And what is a clock cycle of a chip?

从根本上说，芯片具有极强的并行性，拥有上千亿个晶体管。大规模并行的一个关键需求是同步。在软件中，我们使用互斥锁等昂贵的方法。在芯片上，我们采用不同的方法：大约每纳秒，所有电路暂停片刻并同步。这就是时钟周期。整个芯片步调一致地进入下一步操作。在电路中，时钟由寄存器调节。你有存储一位数据的存储单元、一个逻辑云（如脉动阵列或乘法器）以及一个输出寄存器。一个全局时钟信号驱动所有寄存器。在某个瞬间，导线上的任何值都会被存储。挑战在于尽可能快地运行时钟——2 GHz 每秒执行的运算次数是 1 GHz 的两倍。但你对逻辑云的延迟很敏感，因为任何计算都必须在下一个时钟周期到来之前完成。

At baseline, chips are incredibly parallel, with a hundred billion transistors. A key need with massive parallelism is synchronization. In software, you use expensive methods like mutexes. On chips, we take a different approach: every nanosecond or so, all circuitry pauses for a moment and synchronizes. That is the clock cycle. The entire chip goes in lockstep to the next operation. In circuitry, the clock is mediated by registers. You have storage storing a bit, a cloud of logic like a systolic array or multiplier, and an output register. A global clock signal drives all registers. At a certain instant, whatever value is on the wire gets stored. The challenge is to run the clock as fast as possible—2 GHz gives twice the operations per second as 1 GHz. But you're sensitive to the delay through the logic cloud because any computation must finish before the next clock cycle.

所以任何芯片的一个主要优化点就是尽可能缩短这个延迟。有没有一种情况，你会概率性地赌计算能完成，还是说要么在时钟周期内完成，要么就不行？

So a major point of optimization on any chip is to make this delay as short as possible. Is there ever a situation where you'd take a probabilistic chance that a computation finishes, or is it just either it finishes by the clock cycle or not?

在标准芯片设计中，你可以认为存在概率，但它在许多标准差之外，所以实际上它是一个可靠的部件——它总能满足时钟要求。有一些奇怪的例外，比如时钟域交叉，从一个时钟过渡到另一个时钟，这时你确实需要考虑概率。但在主路径上，你可以认为你会提前 25% 的时钟周期到达，所以概率非常低。

In standard chip design, you imagine it such that there is a probability, but it's many standard deviations out, so for all intents and purposes it is a reliable part—it will always meet the clock. There are weird exceptions like clock domain crossings where you go from one clock to another, and then you do have to reason about probability. But in the main path, you imagine it such that you'll get there 25% of the clock cycle in advance, so it's very unlikely.

那么时钟同步——也就是寄存器的位置——是芯片设计师决定的，还是只是将 Verilog 转换成发送给台积电的文件的软件的一个产物？

And the clock synchronization—where the registers are—is that something you determine as a chip designer, or is it just an artifact of the software that converts Verilog into the thing you send to TSMC?

这实际上是芯片设计工作的一大部分——插入寄存器。它通过手动和自动相结合的方式完成。例如，你可以将逻辑分成两半，中间放一个寄存器，这样就不再是一个逻辑云，而是两个更小的逻辑云。如果你从中间分开，你可以达到两倍的时钟频率，以额外一个寄存器和更多存储为代价获得两倍的性能。

This is actually a huge part of the work of designing a chip—inserting registers. It is done in a combination of manually and automatically. For example, you can take logic and split it in half with a register in between, so instead of one cloud of logic, you have two smaller clouds. If you split it in the middle, you can hit twice the clock frequency, giving twice the performance at the cost of an extra register and more storage.

退一步说，为什么我们需要同步整个芯片？如果你想象玩《异星工厂》，没有全局时钟周期——事情做完就算完。

Stepping back, why do we need to synchronize the whole chip? If you imagine playing Factorio, there's no global clock cycle—it just gets done when it's done.

你可以拿走它，如果你想要的话。那么，借用这个类比，你需要留意的是，如果我有两条不同的逻辑路径。所以，我在这里要做一次计算 F，然后在这里做一次计算 G，然后它们会在某处汇合进行计算 H。

You can take it if you want. So, taking that analogy, the thing you need to be mindful of is if I've got two different paths through some logic. So, I have to do a computation like F here, and then computation G here, and then they're going to come and meet for computation H somewhere here.

嗯。

Yeah.

所以，这里会存在制造差异。在某些芯片上，F 会多花一点时间，也许在某些芯片上，G 会多花一点时间。因此，如果我有某个信号在这里传播，而 F 和 G 的结果必须在 H 处汇合，可能出错的地方是 F 提前到达，然后它遇到了 G 的前一个值或后一个值之类的情况。

And so, there's going to be manufacturing variance here. In some chips F will take a little longer, maybe in some chips G will take a bit longer. So, if I've got some signal that's propagating through here, and the results from F and G have to sort of meet up at H, what can go wrong is that F can get there early and it meets like the previous value of G or the next value of G or something like that.

而且 H 需要知道何时开始。没错。就像下一次迭代何时……

And H needs to know when to start. Exactly. Like when has this next iteration of...

所以，这就解释了为什么在相同工艺节点（比如台积电的技术）下制造的不同芯片，会有不同的时钟周期。比如两颗在 3 纳米节点制造的芯片，可能因为能否优化确保没有一条关键路径过长而拖慢整个芯片的时钟周期，从而具有不同的时钟周期。

So, this explains why different chips made at the same process node, the same like TSMC technology, can have different clock cycles. Like two chips made at three nanometer might have different clock cycles based on whether they were able to optimize making sure that there's no one critical path that is so long that it slows down the whole chip's clock cycle.

没错。

That's right.

我在这里展示的这种优化，就是所谓的流水线寄存器插入。

This optimization that I showed here, this is just sort of pipeline register insertion it's called.

嗯。

Yeah.

我们在这里的流水线或寄存器中间插入了它。这是一种纯粹的时钟速度与面积之间的权衡。

We've inserted it in the middle of the pipeline or register here. This is a sort of pure trade-off between clock speed and area.

嗯。

Yeah.

这是简单的情况。还有一种更难的情况，我在这里把它画成了逻辑流水线，但在其他情况下，你可能会有某种计算实际上会反馈到自身。所以，它运行某个函数 F，然后像这样写回自身。例如，这可能是加法，就像你有一个数字，每个时钟周期都要加上它，所以这就像加法，每个时钟周期都加上某个数字。

This is the easy case. There is a harder case, too, which is, I've drawn it as a pipeline of logic here, but in other cases, you may have some calculation which actually feeds back in on itself. So, it runs some function F and then writes back to itself like this. For example, this might be addition, like you've got some number that you're adding to every clock cycle, and so this could be like plus, where adding in some number every clock cycle.

嗯。

Mhm.

所以，这个小电路本质上就是对不同时钟周期上出现的所有数字求和。挑战在于，如果这个加法耗时太长，我能做什么？如果我尝试在它中间插入一个流水线寄存器，就像这里中间这样，这最终会改变所做的计算。我不会得到所有输入数据的累加和，而是会得到两个不同的累加和。最终我会得到偶数的累加和和奇数的累加和。所以，这种逻辑中存在循环的约束——所有芯片在某个地方都有循环——这实际上是最难处理、也决定了时钟周期的事情。

So, this little circuit essentially is just going to sum all of the numbers that get presented on different clock cycles. And the challenge is, if this plus takes too long, what can I do? If I try and put a pipeline register right in the middle of it, like here in the middle of it, this will end up changing the computation that's done. Instead of forming a running sum of everything that comes here, I will actually have two different running sums. I'll end up having a running sum of the even numbers and a running sum of the odd numbers. So, this constraint where I have a loop in my logic, which all chips have somewhere, this is actually the thing that is the hardest to address and sets the clock cycle.

我不明白为什么会有问题，甚至不确定在那里放一个寄存器意味着什么。因为它有点像原子操作，对吧？

I don't understand why it'd be a problem to have that or I'm not sure even what it would mean to have a register there. Because it's sort of an atomic operation, right?

嗯，加法并不是真正的原子操作。正如你刚才演示的那样。

Well, plus is not really atomic. As you just demonstrated.

是的，是的，做一次求和需要很多工作。所以，你可以把工作的早期部分拿出来，在中间放一个寄存器，然后再处理工作的后期部分。明白了。好的。嗯。

Yeah, yeah, it took a whole lot of work to do a summation. And so, you can take the early parts of that work and then stick a register in the middle and then take the late parts of that work. Got it. Okay. Yeah.

我想这就要看台积电了，他们提供 PDK（工艺设计套件）来设定尺寸，比如“这是我们在芯片中可以提供的逻辑原语”，并且由他们来确定没有原语大于他们希望工艺节点达到的时钟周期。但除此之外，还能进一步优化吗？你不能直接说：“嘿，这是台积电的所有原语，然后在原语之间根据需要尽可能多地添加寄存器，直到达到你想要的时钟周期？”

And I guess it's then up to so TSMC offers a PDK which sets size, hey, here's the primitives of logic that we can grant you in the chip, and it's up to them to determine that no primitive is bigger than like the clock cycle they're hoping a process node targets. But, other than that, is there like what further optimize? Can't you just say, 'Hey, here's all the primitives from TSMC, and keep adding registers in between the primitives as much as is needed until you get to your desired clock cycle?'

是的，作为逻辑设计者，芯片架构师设定时钟周期。举个例子，你从台积电得到的原语大概是与门或全加器这样的级别。

Yeah, as a logic designer, the chip architect sets the clock cycle. Just for one example, the primitives you get from TSMC are on the order of like AND gates or full adders.

嗯。

Mhm.

它们很大程度上取决于电压、频率以及你选择的库等等。但一般来说，在一个时钟周期内，你通常可以顺序放置大约 10、20 或 30 个这样的原语。所以，这些原语非常非常快，大约 10 皮秒左右。因此，作为逻辑设计者，原则上，如果你真的只是像那样在一个循环中放一个寄存器和一个与门，你可以得到极快的时钟速度，比如超过 4、5、6 GHz 之类的。但如果你拿这个非常简单的电路，看看你在这里花费的面积，这大概是一个门等效的大小，面积单位是 1，而这个东西的面积单位大概是 8 左右。所以，几乎你所有的成本都是这种同步或通信成本，而不是实际逻辑。因此，这就属于做得过火了。你以几乎把所有面积都花在流水线寄存器上为代价，让时钟速度变得非常非常快。

They depend a lot on voltage and frequency and which library you choose and so on. But generally, you can typically have about 10 or 20 or 30 of these in a clock cycle sequentially. So, these primitives are very very fast, like 10 picoseconds or something like that. And so, as a logic designer, in principle, if you literally just had like register and then AND gate in a loop like that, you could get an insanely fast clock speed, like more than four or five six gigahertz or something like that. But if you take this really simple circuit and you look at the area you're spending here, this is maybe like one gate equivalent in size, so like unit of one in area, and this thing is like unit of eight in area or something like that. So, this is just almost all of your cost is this synchronization or communication cost compared to the actual logic. And so, this would be a case where you've gone too far. You've made your clock speed really really fast at the cost of spending almost all of your area on pipeline registers.

有意思。所以，你暗示的是一种动态：你可以有非常快的时钟速度，但完成的工作量并不多。

Interesting. So, what you're hinting at is a dynamic where you can have really fast clock speed, but you're not getting that much work done.

是的，是的，是的。所以你可以有低延迟，但低带宽。或者更准确地说，低吞吐量。实际上，这会损害你的吞吐量，因为芯片的吞吐量可以看作是每个时钟周期能完成的工作量（基于面积效率）乘以每秒的时钟数。

Yeah. Yeah. Yeah. And so you can have low latency, but low bandwidth. Or throughput, rather. It hurts your throughput, in fact, because the throughput of your chip you can think of as the product of how much I can get done per clock cycle, which is based on this area efficiency thing, times how many clocks I get per second.

这实际上和我们上次讨论的关于 batch size（批大小）的事情非常相似，如果你有较低的批大小，那么任何用户都能非常快地收到他们的下一个 token，但比如说一小时内处理的 token 总数会比原本可能的要低。

This is actually so similar to the thing we were discussing last time about batch size, where if you have a low batch size, then any user can receive their next token really fast, but the total number of tokens that are processed in say an hour will be kind of lower than it could otherwise be.

是的，没错。如果你把时钟速度提得很高，你得到的并行度就会降低。语言模型开始与最优秀的人类预测者竞争。我与两位 Jane Street 的高级员工 Ron Minsky 和 Dan Pontecorvo 坐下来，问道：“在某个时候，AI 会不会直接做 Jane Street 做的事情？”有一个我们应该认真对待的世界，那就是我们将构建大型语言模型或其他 AI 系统，它们严格来说比地球上所有人类都更聪明，在所有认知任务上都更有能力。

Yeah, exactly. You get less parallelism out if you drive your clock speed up really high. Language models are starting to compete against the best human forecasters. I sat down with two senior Jane Streeters, Ron Minsky and Dan Pontecorvo, and asked, 'At some point, does AI just do what Jane Street does?' There's a world that we should take seriously where, you know, we're going to build large language models or some other AI systems that are like strictly smarter than all humans on the planet and more capable at all cognitive tasks.

交易在我看来有点像 AGI 完备问题，类似于 NP 完备问题，因为归根结底，交易涉及判断事物的价值，这意味着要对未来做出预测。Jane Street 并没有押注反对 AI。他们刚刚签下了一笔 600 万美元的计算机芯片订单。但 Ron 的观点是，优势在持续转移。我从未像今天这样迫切地需要招聘更多的工程师和交易员。你知道，总有一些我们尚不知道如何自动化的困难部分。而这些恰恰是竞争优势所在。你可以在 janestreet.com/thorkast 找到这些职位空缺并观看完整采访。

Trading in particular feels to me like a kind of AGI-complete, sort of like NP-complete, because at the end of the day, trading involves figuring out what things are worth, which means making predictions about the future. Jane Street isn't betting against AI. They just signed a $6 million computer chip. But Ron's view is that the edge keeps moving. I have never been more desperate to hire more engineers and more traders than I am today. You know, you have the usual thing of like the other hard parts that we don't yet know how to automate. Well, that ends up being where the competitive edge lies. You can find these open positions and watch the full interview at janestreet.com/thorkast.

好的，我记得和 Jane Street 的一位 FPGA 工程师 Clark 聊过，他帮我准备了之前我们一起做的那次采访。他解释了为什么他们使用 FPGA。我想，对于高频交易来说，吞吐量不如延迟重要。因此，以确定性的方式对时钟周期进行非常具体的控制是最重要的。也许聊聊为什么不能用 ASIC 来实现，或者为什么 FPGA 是……嗯，为什么你会用 FPGA 来为高频交易提供确定性的时钟周期，会很有意思。

Okay, so I remember talking to an FPGA engineer at Jane Street, Clark, who actually helped me prep for the previous interview we did together. And he was explaining why they use FPGAs. I imagine that for high-frequency trading, the throughput is less important than latency. And so having very specific control over the clock cycle in a deterministic way is the most important thing. Maybe it'd be interesting to talk about why you can't just achieve that with an ASIC or why an FPGA is the... Yeah, why you might use an FPGA to have deterministic clock cycles for high-frequency trading.

是的。首先，我们来考虑一下 FPGA 与 ASIC 的商业案例。FPGA 和 ASIC 使用大致相同的概念模型：由 AND、OR、XOR 等非常小的原语构建的一系列门电路，通过固定的时钟周期和在该固定时钟周期下运行的导线连接在一起。所以，你在 FPGA 中能表达的任何东西，在 ASIC 中也能表达。而且 ASIC 的成本大约低一个数量级，能效也更好。权衡之处在于，第一块 FPGA 的成本是 1 万美元，而制作第一块 ASIC 的成本是 3 万美元，因为它需要完整的流片。因此，FPGA 的商业使用场景是：我需要具有非常确定性的延迟、快速运行时间和高并行性的东西，但我会非常频繁地更改它——我大约每个月都会改变我的做法。所以我不想每次都支付流片成本。

Yeah. So, firstly, let's consider the business case for an FPGA versus an ASIC. FPGAs and ASICs use largely the same conceptual model: a series of gates built from ANDs, ORs, XORs, those very small primitives, connected together with a fixed clock cycle and wires running at that fixed clock cycle. So, anything you can express in an FPGA, you can express in an ASIC too. And it'll be about an order of magnitude cheaper and have better energy efficiency on an ASIC than an FPGA. The trade-off is that the first FPGA costs you $10,000, whereas the first ASIC you make costs you $30,000 because it requires an entire tapeout. So, the business use case for an FPGA would be that I want something with very deterministic latency, fast run time, and high parallelism, but I'm going to change it very frequently—I change what I do every month or so. And so I don't want to pay the tapeout cost every time.

那么，FPGA 实际上是如何实现这一点的呢？它有点像在固定的硬件上模拟 ASIC 的编程模型。那么，这实际上是如何工作的呢？

Now, how does an FPGA actually implement it? It's sort of like it emulates the ASIC programming model, but in a fixed piece of hardware. So, how does that work actually?

所以，它的基础是我们刚才谈到的两个组件。它有这些寄存器作为存储设备。然后还有这些称为 LUT（查找表）的东西，它们实际上提供了所有的门电路。然后我们有第三个组件：我们有一大堆这些寄存器和 LUT，全部可用，它们通过一大组多路选择器连接起来。因此，在每一个这样的组件前面，都有一个多路选择器，它从其他地方选择一个输入，从所有这些中选择东西。我们有大量不同的选项输入到所有这些组件中。所以，这允许的是，当我编程我的 FPGA 时，我可以说我将取所有这些组件，并在其之上叠加一个特定的布线，这个布线经过这个 LUT，然后输入到这个 LUT，然后到这个寄存器，然后输入到这个 LUT，等等。我用橙色画出的就是 FPGA 的含义——现场可编程门阵列。橙色是现场编程的部分，而白色是 FPGA 中为了制造该器件而必须存在的所有导线。

So, what it has at the base is the two components we just talked about. It has these registers as storage devices. And then it has these called LUTs, look-up tables, which actually provide all of the gates. Then we have a third component: we have a swarm of these registers and LUTs, all available, and they're connected by a big set of muxes. So, in front of every single one of these, we've got a mux which selects one input from everywhere else, selecting from all of these for things. We've got a whole bunch of different options feeding into all of these things. So, what this allows is essentially when I program my FPGA, I can say that I'm going to take all of these components and superimpose on top of this a particular wiring which goes through this LUT, then feeds into this LUT, then goes to this register, then feeds into this LUT, or something like that. What I've drawn in orange is how you like FPGA means field programmable gate array. The orange is what has been programmed in the field, whereas the white is all of the wires that must exist in the FPGA in order to make the device in the first place.

“现场编程”是什么意思？就像“在现场编程”那样。

What does it mean to be programmed in the field? Like programmed in the field.

所以，设备已经部署在数据中心，它就在现场，然后你可以来编程它。不是电场那种“场”，也不是“野外”那种“场”。

So, the device has been deployed in a data center, it's sitting in the field, and then you can come and program it. Not field as in electric field, not field as in out there in the world field.

好的。那么，如果我看看现场编程是如何从第一个查找表出来并进入第二个查找表的。它是怎么做到的？怎么做到的？比如，实现这一点的导线在哪里？我猜。

Okay. And so if I look at how the field programming comes out of the first lookup table and goes into the second one. How is it? How? Like where are the wires that make that happen? I guess.

是的。我在画所有这些的时候有点偷懒。这里的每一个器件前面都有一个多路选择器，它可以从所有可用的附近电路中选择。所以 FPGA 的实际配置归结为多路选择器的控制。所以在这个多路选择器中，我们有数据输入，然后有选择控制。因此，每个多路选择器旁边都有一个小的存储设备，指示你将从哪里获取输入。所以编程它包含配置每一个这样的多路选择器。

Yeah. So I got a little lazy in drawing all of these. Every single device here has a mux sitting in front of it, which can select from all of the nearby circuits that are available. And so the actual configuration of the FPGA amounts to the mux control. So in this mux here, we have the data inputs and then we have the control that selects. And so there's a little storage device sitting next to every single one of these muxes saying this is where you're going to source your input from. So programming it consists of configuring every single one of these muxes.

有道理。查找表内部发生了什么？

That makes sense. What is happening inside of the lookup table?

是的。查找表的目的是能够可配置地扮演 AND 门、OR 门、XOR 门或任何其他不同门电路的角色。有很多方法可以做到这一点。传统 FPGA 的做法是，它会支持……所以一个查找表将有 4 位输入、1 位输出。从 4 位到 1 位的函数有多少种？有 16 种不同的函数。所以实际上你可以将其制成一个包含 16 个不同数字的表格。你有一个由 0 和 1 组成的表格，共 16 个条目。所以它的作用是，这个表格存储在这个蓝色配置位中，然后它将这 4 位视为二进制数，查找表格的相关行，并输出该位。这本质上就是查找表的真值表视图。

Yeah. So the purpose of the lookup table is to be able to configurably take the role of an AND gate, OR gate, XOR, any of those different gates. There are many ways you could consider doing that. The way it is done in traditional FPGAs is to say it will support... So a lookup table will have four bits of input, one bit of output. How many different functions are there from four to one bit? There are 16 different functions. And so you can actually just tabulate this as 16 different numbers. You've got a table of 0s and 1s, 16 entries. So what it does is, this table is stored in this blue configuration bit, and then it views these four bits as binary, looks up the relevant row of the table, and emits that bit. This is a truth table view of lookup tables, essentially.

好的，所以查找表，如果你想想看，AND 门、OR 门、NOR 门、XOR 门，这些都以……作为输入。它们就像是双输入函数。

Okay, so the lookup table, if you think about it, an AND gate, OR gate, NOR gate, XOR gate, these are all like take as input... Those are like two-input functions.

是的。所以有时我们有更复杂的，比如三输入函数可能是三路 XOR，或者四路 XOR。在这种情况下，有多少……这取决于它有多大，但 LUT 的典型大小是 4 输入，这有点像是一个最佳平衡点……这里存在另一个计算机通信的权衡。

Yeah. So sometimes we have more complicated like a three-input function would be a three-way XOR, or a four-way XOR. And in this case, how many... it just depends how big it is, but typical size for LUTs is four input, which is sort of just a sweet spot between... there's another computer communication trade-off like here.

如果输入太少，就需要使用更多的查找表。但基本上，查找表就像一个真值表。有了真值表，你就可以编程实现任何你想要的逻辑门。

If it has too few inputs, then you need to use more LUTs. But basically, the lookup table is like a truth table. And with a truth table, you can program in any gate you want.

没错。所以，与其说是查找表，不如把它看作一个可编程的逻辑门。

That's right. And so, instead of a lookup table, just think like a programmable gate.

这里你可以看到经验法则——FPGA 比 ASIC 贵大约一个数量级——的来源。就是数一下这个查找表里面有多少个逻辑门。我们可以把这个查找表本质上看作一个多路选择器。它是一个需要在 16 个不同值之间选择的多路选择器。所以这是一个 n 等于 16 个选项、p 等于 1 比特的多路选择器。我们之前看到，这个电路的成本大约是 n 乘以 p 个逻辑门。所以它需要 n 乘以 p 等于 16 个与门和 16 个或门。这个电路就是多路选择器。

One of the things you can do here is see why the rule of thumb that an FPGA is about an order of magnitude more expensive than an ASIC comes from. It's to count how many gates would be inside this lookup table. We can view this lookup table essentially as one of these muxes. It is a mux that has to select between 16 different values. So it is a mux with n equals 16 options, p equals one bit. What we saw earlier is that this circuit costs like n times p many gates. So it costs like n times p equals 16 AND gates and also 16 ORs. This circuit being the mux.

多路选择器是核心……

The mux is the core of the...

就是进入查找表的多路选择器。所以你可以把查找表本身看作一个大的多路选择器，它从所有 16 行中选择一个输出。

The mux that goes into the lookup table. So the lookup table itself you can think of as being actually a big mux that selects from all 16 rows down to one output.

嗯，好的，我明白了。那就是查找表。但你这里画的好像是一个多路选择器，然后一个查找表。

Yeah, okay. I see. That is the lookup table. But the way you've drawn it here there's like a mux and then a lookup table.

层层都是多路选择器。这里面还有第二个多路选择器。这个多路选择器就是这个。明白了。然后另一个多路选择器只是说明它来自这一堆逻辑门中的哪个位置。

It's muxes all the way down. There is a second mux that is inside here. This mux is this mux. Got it. And then the other mux is just saying where it came from in this sort of mess of gates.

对。然后第二个多路选择器是，现在你有了一个值，但这个值仍然是一个四比特的值。所以我已经从一堆比特中选出了四个比特。

Right. And then the second mux is, okay, now you have one value, but that value is still a four-bit value. So I've selected four bits from the soup.

然后我用这四个比特来选择我要使用查找表中的哪个条目。

And then I use those four bits to select which entry in the lookup table I'm going to use.

对，好的。就像这样：假设在第一个多路选择器中，有八个附近的寄存器作为输入。所以总共有 32 比特输入。然后从中输出四个比特。这四个比特进入第二个多路选择器，也就是查找表内部的那个。所以实际上，在这种情况下，这些寄存器是单比特寄存器。所以如果有八个附近的寄存器和查找表，那么我总共有八比特输入。我从八个中选出四个不同的值。所以实际上有四个不同的多路选择器，每个对应一个输入比特。每个多路选择器从八个中选一个。那八个从哪里来？来自附近的寄存器和其他的查找表。每个寄存器是一比特吗？

Right. Okay. And it's just like, suppose in the first mux there's like eight nearby registers as input. So that's like a total of 32 bits going in. And then out of that, four bits come out. Those four bits go into the second mux which is inside the lookup table. So actually, in this case these registers are single bit registers. So if there are eight nearby registers and lookup tables, then I have eight bits total coming in. I select from eight down to four different values. So there's actually like four different muxes, one associated with each of these input bits. Each of them is selecting one out of eight. And where are those eight coming from? Nearby registers and other LUTs. And each register is one bit?

是的。

Yes.

所以我想 AMD 或任何制造这些 FPGA 的公司仍然需要对哪些寄存器连接到哪些寄存器做出规定。然后你可以编程实现实际的逻辑门，但他们已经添加了连线，就像通信拓扑一样，对吗？

And so I guess AMD or whoever makes these FPGAs still has to be opinionated about what registers are connected to which registers. And then you can program in the actual gates, but they added wire in the connect like the communication topology, right?

是的，所以在局部粒度上有一定的灵活性。有一个附近的邻域你可以从中选择。但对于更粗略、更远距离的连接，他们会有自己的设计。

Yeah, so there's sort of flexibility in a local grain thing. There's a sort of nearby neighborhood where you can select from. But then more coarsely, longer distance connections they form an opinion on.

对。那为什么是 10 倍更低呢？如果你看构建这个查找表的成本，大约是 32 个逻辑门。然后它能给我什么等价的东西？一个有趣的事情是我可以实现一个四输入与门。所以我用了 32 个逻辑门的查找表来实现一个四输入与门。四输入与门是什么意思？我会做与、与，然后与的结果再与。所以这是一个我可以在 ASIC 中直接用这三个与门实现的电路。但使用查找表，我也可以实现它，但需要大约 32 个逻辑门而不是三个。

Right. And the reason it's 10x lower is why? So if you look at the cost of building this lookup table, it's like 32 gates. And then it can give me the equivalent of what? One interesting thing I can do here is a four-way AND gate. And so that's like I'm using 32 gates of lookup table to implement a four-way AND. A four-way AND means what? I would do AND AND and then AND of AND. So this is a circuit that I could implement in an ASIC directly using these three AND gates. But using a LUT, I can also implement it, but it's going to take like these 32 gates instead of three.

对。所以开销实际上来自于查找表，查找表中的多路选择器。有一种更简洁的方法来描述真值表，而不是列出每一个可能的输入组合。

Right. And so the overhead is really coming from the fact that the lookup table, the mux in the lookup table, there's a more concise way to describe a truth table than listing out every single possible combination of inputs.

是的。但那就是直接写出逻辑门。比如放置多晶硅和导线等等。

Yes. But which is just to write out the gate. Like to place down the polysilicon and the wires and so on.

他向我强调的一个重要点是，他们更喜欢 FPGA 而不是 CPU 的原因是 FPGA 能提供确定性的时钟周期。他们知道数据包何时进入和离开。为什么 CPU 不能保证这一点？

One important point he made to me is that the reason they prefer FPGAs to CPUs is because they get deterministic clock cycles. They know when a packet will come in and go out. Why is it not a guarantee in CPUs?

你实际上可以设计一个具有确定性延迟的 CPU。事实上，许多 AI 芯片内部的处理器也具有确定性延迟。Groq 已经宣传过这一点。TPU 的核心也有。挑战在于同时实现确定性延迟和高速度。延迟的非确定性从何而来？非确定性延迟来自 CPU 中的特定设计选择。实际上可以移除这些设计选择，制造出具有确定性延迟的 CPU。但这些 CPU 在市场上不太有吸引力，所以人们不再制造它们了。但从某种意义上说，确定性延迟可能是一个更简单的起点。然后一些芯片设计师添加了非确定性的东西。举个具体的例子，最重要的例子就是 CPU 缓存本身。在 CPU 中，有 CPU 芯片本身，旁边有内存。这是 DDR 内存。然后内部有一个缓存系统。它记住最近对 DDR 的访问并存储它们。所以当我执行 CPU 指令时，每次有访问内存的指令，它首先检查缓存中是否存储了数据，如果没有，则从 DDR 获取。这是一个巨大的优化。缓存比 DDR 快大约两个数量级。如果你从不使用缓存，所有程序都会慢 100 倍。所以缓存的存在对于 CPU 以合理速度运行是绝对必要的。但是否命中缓存取决于 CPU 的环境，比如其他程序在运行什么，最近运行了什么，缓存系统内部的随机数生成器在做什么。所以这是 CPU 运行时非确定性的一个重要来源。

You can actually design a CPU that has deterministic latency as well. In fact, the processors inside a lot of AI chips also have deterministic latency. Groq has advertised this. TPUs have that in the core as well. The challenge is getting deterministic latency and high speed at the same time. Where does the non-determinism in latency come from? Non-deterministic latency comes from specific design choices in a CPU. It's actually possible to remove those design choices and make a CPU that has deterministic latency. Those are not very attractive in the market, and so people don't make those CPUs anymore. But in some sense, deterministic latency is maybe a simpler starting point. And then some chip designers have added things to be non-deterministic. To take a concrete example, the most important example is the CPU cache itself. In a CPU, you have the CPU die itself, and then there's memory off on the side. This is the DDR memory. And then you have a cache system inside it. That sort of remembers recent accesses to DDR and stores them. So when I'm running through my CPU instructions, every time I have an instruction that accesses memory, it first checks in cache whether the data was stored in cache, and if not, it fetches out to DDR. This is a huge optimization. The cache is like two orders of magnitude faster than the DDR. If you never use the cache, all programs would run 100 times slower. So the presence of a cache is absolutely necessary for a CPU to run at reasonable speed. But whether or not you get a cache hit is dependent on the ambient environment of the CPU, like what other programs are running, what has run recently, what the random number generator inside the cache system is doing. So that is a big source of non-determinism in the runtime of a CPU.

这有点像 CPU 的内存系统。你可以做的一个重大改变是，不再让硬件说“我要读取内存，然后由硬件决定数据是否来自缓存”，而是把这个决定权交给软件。这是一种不同的设计理念。例如，在 TPU 中就能看到这一点。TPU 使用的是暂存器（scratchpad）而不是缓存。所以这就像一个 TPU，这里用的是 HBM 而不是 DDR，但它仍然是片外内存。软件不再先访问内存然后由硬件决定，而是有一些指令去这里——一种指令——以及完全不同的另一种指令去 HBM。这种风格通常被称为暂存器而非缓存。关键区别在于，有一种指令用于读写暂存器，而另一种完全不同的指令用于读写 HBM。

So this is sort of the memory system for a CPU. The big thing you can do differently is instead of having the hardware say, 'I'm going to read memory and then the hardware decides whether it comes from cache or not,' you can actually bake this decision into software. This is a different design philosophy. You see this in, for example, TPUs. The TPU has a scratchpad instead. So this would be like a TPU, and then HBM in this case rather than DDR, but it's still off-chip memory. Instead of the software first accessing memory and then the hardware deciding, you have some instructions that go here—one kind of instruction—and a totally different kind of instruction that goes to HBM. This style is generically known as scratchpad instead of cache. The key distinction is that you have one kind of instruction that says read or write scratchpad, and a totally different instruction that says read or write HBM.

所以暂存器就是缓存？

So scratchpad is the cache?

是的，这个东西就是暂存器。澄清一下。

Yeah, this thing here is the scratchpad. Just to be clear.

退一步说，人们说计算机有所谓的冯·诺依曼架构，即信息串行处理。也许因为我们一直在讨论并行加速器，但我不喜欢 FPGA 是超级并行的说法。

Stepping way back, people say computers have the so-called Von Neumann architecture where there's serial processing of information. Maybe because we've been talking about parallel accelerators, but I just don't like the FPGA is super parallel.

是的，像 TPU 这类 AI 加速器是超级并行的。即使 CPU，如果你考虑它们拥有的所有核心，也是超级并行的。那么现代硬件在什么意义上还是冯·诺依曼架构呢？这样描述现代硬件公平吗？我认为描述 CPU 是公平的。CPU 上的并行度大约是 100 个核心乘以 16 路向量单元，所以 CPU 上大约有 1000 路并行。

Yeah, the kinds of AI accelerators like TPUs are super parallel. Even CPUs are super parallel if you think about all the cores they have. So in what sense is modern hardware actually the Von Neumann architecture? Is that a fair way to describe modern hardware? I think it's a fair way to describe CPUs. The amount of parallelism on a CPU is about 100 cores times maybe 16-way vector units, so about 1,000-way parallelism on a CPU.

一个问题：芯片面积用来做什么？如果线程更少，仅仅是晶体管电压或开关的问题吗？是否真的只有一个控制流，比如芯片的一小部分在切换电压？如果核心这么少，你实际上如何占用 CPU 的芯片面积？

One question is: what is the die being used for? If there are fewer threads, is it just a matter of transistor voltages or switching on and off? Is there literally one control flow, like a small part of the die where voltages are switching? How do you actually occupy the die area of a CPU if there are so few cores?

核心更大、更复杂。所以我们应该比较一个 CPU 核心（占用芯片面积的 1/16 或 1/100）和 FPGA（其中很多只是这 16 个门）。所以很明显，FPGA 中的 LUT 比 CPU 中的核心多得多。但为什么 CUDA 核心比 CPU 核心多？那将是一个很大的区别。在 CPU 内部，面积的一大用途是缓存。实际上 ALU 占用的很少。主要是寄存器文件而不是逻辑单元。这两者在 GPU 中都有对应，所以这不是大区别。但 GPU 中没有对应的是分支预测器。CPU 中有一大片区域是预测器，用于预测下一个分支何时出现以及分支目标是什么。去掉很多这些东西，以及使寄存器文件更紧凑，是 GPU 性能提升的主要来源。

The cores are just much bigger and more complicated. So we should compare a CPU core, which takes up 1/16 or 1/100 of the die, to an FPGA where a lot is just these 16 gates. So it's clear why there are so many more LUTs in an FPGA than cores in a CPU. But then why are there more CUDA cores than CPU cores? That would be a big difference. Inside a CPU, one big use of area is the cache. Very little is actually the ALUs. Mostly it's register files rather than logic units. Both of these have equivalents in a GPU, so that's not a big difference. But the thing that does not have an equivalent in a GPU is the branch predictor. There is a whole big area in the CPU which is a bunch of predictors saying when my next branch will be and where the branch target is. Stripping a lot of that out, as well as making register files tighter, is driving a lot of where the GPU gains come from.

分支预测器的目的是同时执行两个分支，还是它做什么？

Is the purpose of the branch predictor to execute both branches at once, or what does it do?

问题是，当我有一系列指令时，如果这里有一个分支，处理一条指令的实际步骤需要很长时间——大约五纳秒。所以注意到有一个分支、评估布尔值、将程序计数器更新到新目标、并从指令内存读取，可能需要五纳秒才能完成。实际上，这可能在这里某处完成。我不希望这样；我想以比 5 纳秒允许的更快时钟速度运行。5 纳秒对应 200 MHz 时钟速度。我想运行在 1 或 2 GHz。所以我需要在分支被评估的同时运行其他指令。我真的只想继续运行我后面的指令。但这可能是错误的。如果分支最终被采用，那么我需要知道，不是评估这些指令，而是需要跳转到目标位置并运行那些指令。所以分支预测器的目的是，基于在你甚至到达这条指令之前（可能早五个周期）进行预测，预测将会有一个分支。

The issue is that when I have a series of instructions, if I have a branch here, the actual processing step of an instruction takes a really long time—maybe five nanoseconds. So the time to notice I've got a branch, evaluate the boolean, update the program counter to the new target, and read from instruction memory could take five nanoseconds to finish. In reality, this may finish somewhere down here. I don't want that; I want to run a clock speed much faster than 5 nanoseconds allows. 5 nanoseconds is 200 MHz clock speed. I would like to run at 1 or 2 GHz. So I need to run other instructions while the branch is being evaluated. I really just want to keep running the following instructions after me. But that might have been wrong. If the branch ended up being taken, then I need to know that instead of evaluating these instructions, I actually need to jump to wherever the target is and run those instructions instead. So the purpose of the branch predictor is to predict, based on before you even get to this instruction, maybe five cycles earlier, that there was going to be a branch.

如果我想想大脑的工作方式与你在这里描述的高层次内容，区别可能在于，虽然你可以在这些加速器中做结构化稀疏，从而节省一些原本必须用于这些门的面积，但在大脑中是非结构化稀疏。任何神经元都可以连接到任何其他神经元，而不是以列对齐之类的方式。

If I think about how the brain works versus what you're describing here at a high level, the differences might be that while you can do structured sparsity in these accelerators and save yourself some area that you would have otherwise had to dedicate to these gates, in the brain there's unstructured sparsity. Any neuron can connect to any other neuron, not in ways where they have the column lined or whatever.

是的。

Yeah.

然后还有一点是内存和计算是共址的。我想你可以说在某种程度上内存和计算是共址的。

Then there's the fact that memory and compute are co-located. I guess you could say in a way the memory and compute are co-located.

这正是在某种意义上内存和计算的共址。

This is exactly the co-location in some sense of memory and compute.

没错。所以也许那实际上不是一个大的区别。另一个可能的大区别是大脑的时钟周期比计算机慢得多。部分原因是为了节省能量，因为时钟周期越快，电压就需要越大，以便信号稳定并识别晶体管的状态。

That's right. So maybe that actually isn't a big difference. And the other maybe a big difference is that the clock cycle on the brain is much slower than on computers. Partly that's to preserve energy because the faster the clock cycle, the bigger the voltage needs to be in order to identify for the signal to settle and to identify what state a transistor is in.

对，没错。我不知道你是否还有其他高层次的观点，关于大脑可能如何运作与这些芯片如何工作。

Right. That's right. I don't know if you have other high-level takes about how the brain might be doing versus how these chips work.

好，我们先说时钟频率。芯片的时钟频率很高，因为这样才能驱动更高的吞吐量。当我们比较运行某个工作负载的 GPU 时，它的批大小是 1000 左右。而大脑不是以批大小 1000 运行的，只有一个我。所以你可以想象，把 GPU 从千兆赫降到兆赫来运行，这样可能更接近大脑中的等效情况。但硅的工作方式并不会因此带来 1000 倍的能效优势。最终的情况是，你只是把电路运行一次到稳定状态，然后它会闲置很长时间。闲置时它不消耗太多能量，因为大部分能量消耗在比特从 0 到 1 再回到 0 的翻转上。

Yeah, let's take the clock speed one first. The clock speed is quite high on a chip because that drives higher throughput. When we compare a GPU running some workload, it's running batch size 1,000 or something like that. Whereas the brain is not running batch size 1,000. There's only one of me. So you could imagine taking a GPU and instead of running at a gigahertz, run at a megahertz. That would start to look a little bit more like equivalent things in the brain. In the way that silicon works, that does not give you a 1,000x advantage in energy efficiency. What it ends up looking like is you sort of just run the circuit once to stabilization and then it'll sit idle for a long period of time. It doesn't consume a lot of energy while sitting idle because most of the energy is consumed in toggling bits from zero to one and back.

对。

Right.

存储一个比特的方式是，你在芯片某处的电容器中隐式地沉积了一些电荷。所以当比特变成 1 时，电容器充电；当它下次变成 0 时，电容器放电。这个给电容器充电然后将其放电到地的循环就是能量消耗的地方。这被称为动态功耗或开关功耗，是芯片大部分的能量消耗。还有一些其他能量消耗来自绝缘体并非完美绝缘，但我们先忽略它。大部分能量消耗来自从 0 到 1 再回到 0 的翻转过程中的充放电。

The way to think of a bit being stored is you've deposited some charge in a capacitor somewhere in the chip implicitly. So it becomes charged when the bit becomes a one, and then it becomes discharged when it next goes to a zero. That cycle of charging the capacitor and then dumping that charge out to ground is where the energy is consumed. This is called the dynamic or switching power. This is most of the energy consumption of a chip. There is some other energy consumption from the fact that insulators aren't perfect, but we'll discard that. Most of the energy consumption comes from the charging and discharging of toggling from zero to one and back to zero.

对。所以如果你把芯片运行得慢很多，每千个时钟周期才触发一次，那么翻转次数会减少一千倍，能耗也会减少大约一千倍。但这并不是能效上的巨大优势。

Right. So if you run a chip much slower and only clock it once every thousand clock cycles, you will have a thousand times fewer transitions. It'll be about a thousand times less energy consumption. But not a substantial advantage in energy efficiency.

好的，你从高层次描述了 TPU 的工作原理。那么 GPU 和 TPU 在高层次上有什么区别？

Okay, so you described how a TPU works at a high level. What is the difference at a high level between how a GPU and a TPU work?

我认为有一个高层次的组织原则不同，然后核心内部也有差异。但我们先看高层次。对于 GPU 和 TPU，顶层模块结构是什么样的？如果把整个芯片看作一个整体，GPU 的组织方式主要是大量几乎相同的单元，也就是 SM。它们中间有一个 L2 缓存，底部还有更多 SM。所以是一个相当规则的核网格。相比之下，TPU 的逻辑单元粒度要大得多。所以你有几个矩阵单元，也就是大的脉动阵列。中间有一个向量单元，底部是矩阵单元。所以矩阵单元加上中间的向量单元——这就是整个 TPU 芯片。你可以想象把这个东西缩小成一个非常小的单元，包含更小的矩阵单元和更小的向量单元，这大致就是 SM 的样子。所以从非常高的层次来看，GPU 在整个芯片上铺满了大量微小的 TPU。

Yeah, I think there's a high-level organization principle that is different, and then inside the cores there are differences. But we'll look at the high level. For a GPU and a TPU, what does the top-level block structure look like? If you think of this as the whole chip, in each case the organization of the GPU is mostly a bunch of almost identical units, which are the SMs. Then they have an L2 memory in the middle and a bunch more SMs on the bottom. So there's a fairly regular grid of cores. If we look at a TPU in comparison, you end up with much coarser-grained units of logic. So you have something like a few matrix units, which are the big systolic arrays. Then in the middle you have a vector unit, and then matrix units at the bottom. So matrix units with a vector unit in the middle—this is the whole TPU chip. You can think of scaling this thing down into a really tiny unit with a smaller matrix unit and smaller vector unit, and that is sort of what an SM is. So at a very high-level point of view, the GPU has a lot of tiny TPUs tiled across the whole chip.

哦，有意思。所以你的意思是流式多处理器中的 Tensor Core 类似于 MXU。

Oh, interesting. So you're suggesting the Tensor Core within a streaming SM is analogous to an MXU.

是的，非常相似。

Yeah, it's very similar.

我明白了。所以如果结构更松散，用一堆微小的 TPU 就很有意义。而如果你只是做巨大的矩阵乘法，你可能会问，为什么不避免每个 SM 都有自己的寄存器和任务调度器的成本，直接做一个巨大的东西，把这些成本分摊到整个系统上。我认为这体现在你能把东西做多大上。我们尤其从脉动阵列中看到了这个主题，更大的脉动阵列能更好地分摊寄存器文件的成本。

I see. And so if you had more lack of structure, having a bunch of tiny TPUs makes a lot of sense. Whereas if you just have huge matrix multiplications, you might ask why don't we avoid the cost of having individual SMs with their own registers and work schedulers, and just make a huge thing and amortize those costs across the whole thing. I think this shows up in how large you can grow things. We've seen this theme especially with the systolic array where larger systolic array amortizes the register file costs better.

这种设计允许你拥有更大的脉动阵列，而 GPU 的设计则限制你只能使用小单元。但这里有一个权衡。由于这种粗粒度的分离，你需要从向量单元向矩阵单元移动大量数据。所以你需要通过这里的两条外围线路移动大量数据。而在 GPU 中，向量单元无处不在，你需要通过这条线、这条线、这条线、这条线来移动数据。所以 GPU 中向量单元和矩阵单元之间可以移动的数据量实际上比 TPU 大得多，因为不是把所有数据都通过两条线移动，而是通过 16 条左右的线路来移动。

This sort of design allows you to have larger systolic arrays, whereas the GPU design constrains you to having small units of everything. There's a trade-off, however. Because of this coarse-grained separation, you need to move a lot of data from the vector unit to the matrix units. So you need to move a lot of data through two lines of perimeter here. Whereas in the GPU, you have vector units everywhere, and you need to move data through this line, this line, this line, this line. So the amount of data you can move between a vector unit and matrix unit is actually much higher in a GPU than in a TPU because instead of moving all the data through just two lines, you're moving it through 16 lines or something of wiring.

对，而且你可能需要在更小的区域内移动数据，这也是一种节能。所以如果你能完全在一个 SM 内操作，数据移动就小得多。但一旦你想跨 SM 操作，就会变得更复杂、更昂贵。

Right, but also you might have to move across less area, which is also a saving in energy. So if you can operate entirely within an SM, the data movement is much smaller. But the moment you want to operate across SMs, it becomes more complicated and expensive.

你没有评论，但可以预期 Matrox 可能尝试做的是获得类似 GPU 的较小结构，即被 SRAM 包围的脉动阵列，但同时去掉 SM 中支持 CUDA 架构所需的那一大块空间。

You don't have a comment, but one might expect that a thing in Matrox might try to do is to get the GPU-like smaller structure of systolic arrays surrounded by SRAM, but also at the same time make it so that the things you need in an SM to support the CUDA architecture—which take a bunch of space—you might discard.

是的。我们公开讨论过一种我们称之为可拆分脉动阵列的东西，从某种意义上你可以把它看作既可以是大脉动阵列也可以是小脉动阵列。

Yeah. We've talked publicly about something we call a splittable systolic array, which in some sense you can think of as big systolic arrays that can also be small systolic arrays.

酷。好的，我觉得这是个不错的结尾。Ritesh，非常感谢。谢谢，Ritesh。

Cool. Okay, I think this is a good note to close on. Ritesh, thank you so much. Thanks, Ritesh.