从零构建 AlphaGo：AI 研究启示录

今天我的嘉宾是 Eric Jang，他最近担任 1X Technologies 的 AI 副总裁，之前是 Google DeepMind Robotics（现名）的高级研究科学家。过去几个月你一直在休假，其中一项工作是重建、改进和破解 AlphaGo。所以今天我们要做的是，你来解释从头构建 AlphaGo 的过程，以及它告诉我们关于 AI 研究和开发的未来。但在那之前，为什么 AlphaGo 有趣？为什么你决定在休假期间做这个项目，而不是去海滩放松？

Today I'm here with Eric Jang, who was most recently vice president of AI at 1X Technologies. Before that, senior research scientist at what is now Google DeepMind Robotics. And you've been on sabbatical for the last few months. One of the things you've been doing is rebuilding and improving and hacking on AlphaGo. And so today what we're going to do is you're going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development. But before we get to that, why is AlphaGo interesting? Why is this the project you decided to do on sabbatical rather than just hang out at the beach?

当然。我喜欢做东西，AlphaGo 和围棋 AI 是真正让我进入这个领域的东西之一。当我看到 2014、2015、2016 年 AlphaGo 的早期突破时，看到 AI 系统能变得多聪明，以及它们能用深度学习处理的计算复杂度类别，这非常深刻。这是一个长期以来被认为对搜索来说难以处理的问题，但通过深度学习解决了。所以这对我来说相当神秘，我一直想更好地理解这个现象。我的训练通常是在机器人学的深度神经网络中，神经网络做出的决策更直观一些。但 AlphaGo 是一个决策实际上是深度搜索结果的问题。我一直很神秘，一个 10 层的网络如何能摊销游戏树中如此深的东西的模拟。

Sure. Yeah. I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. When I saw the early breakthroughs on AlphaGo in 2014, 2015, 2016 and so forth, it was profound to see how smart AI systems could become and the kind of computational complexity class they could tackle with deep learning. This is a problem that has long been understood to be kind of intractable for search, and yet it was solved through deep learning. So that was quite mysterious to me, and I've always wanted to understand that phenomenon a little bit better. My training is often in deep neural nets for robotics, where the decisions made by the neural networks are a bit more intuitive. But AlphaGo is a problem where the decisions are actually the result of a very deep search. And it's always been very mysterious to me how a 10-layer network can amortize the simulation of something so deep in the game tree.

是的，有趣。所以如果你画出多年来构建各种强围棋机器人所需的算力，你可以看到 2020 年有一个名为 KataGo 的开源项目，由 Jane Street 的 David Wu 完成，它基本上实现了训练一个非常强的围棋机器人所需的算力减少 40 倍。我不确定它是否比 AlphaGo Zero、AlphaZero 或 MuZero 更强，但它非常强，今天大多数围棋练习者在与 AI 对弈时都在与它训练。感谢大语言模型编码，以前需要 DeepMind 整个研究科学家团队和数百万美元研究和算力的事情，现在可以用几千美元的租用算力完成。

Yeah. Interesting. So if you plot out how much compute it took to build various iterations of strong Go bots over the years, you can see that in 2020 there was an open-source project called KataGo by David Wu from Jane Street, who basically achieved a 40x reduction in compute needed to train a really strong Go bot. I'm not certain if it's stronger than AlphaGo Zero or AlphaZero or MuZero, but it's very strong, and this is what most Go practitioners today train against when they're playing an AI. And thanks to LLM coding, what took a whole team of research scientists at DeepMind and millions of dollars of research and compute can now be done for a few thousand dollars of rented compute.

好的，我想我们应该先讨论一下围棋是如何运作的。

Okay, I guess we should first discuss how Go works.

好的。

Great.

那么，这个游戏是怎么玩的？

So yeah, how does the game work?

围棋是一个非常简单的游戏，可以在计算机中快速轻松地实现。游戏的目标基本上是放下黑白棋子，并试图占据棋盘上尽可能多的领地。所以我可能先放一颗黑子。黑棋总是先手。我们开始吧。所以捕获对手棋子的方式是，对于每个交叉点，如果你能用你的棋子包围它的所有四个邻居，那么这个棋子就被切断了氧气，如果你愿意这么说的话，它就变成了死子。所以现在我就控制了这四颗棋子以及这里的这个空交叉点。

The game of Go is a very simple one that can be implemented quickly and easily in a computer. The objective of the game is basically to put down black and white stones and try to occupy as much territory on the board as possible. So I might start by putting down a black stone. Black always goes first. Let's go ahead. And so the way you capture an opponent's stones is that for every intersection, if you can surround all four of its neighbors with your stones, then this one is sort of cut off from oxygen, if you will, and it is a dead stone. So then now I control these four stones as well as this empty intersection here.

所以中国规则、日本规则和所谓的 Tromp-Taylor 规则之间有一些细微差别。Tromp-Taylor 规则被设计为对围棋完全无歧义。所以这是所有围棋 AI 训练和解决时使用的规则。在人类通常下的围棋中，你实际上不允许把白子放在这里。那会是自杀。在 Tromp-Taylor 中，实际上是可以的。你放下它，然后它立即被判定为死子。所以结果是一样的。

So there are slight variations between Chinese, Japanese, and what is called Tromp-Taylor rules. Tromp-Taylor rules are designed to be completely unambiguous for Go. So this is what all Go AIs train against and resolve against. In typical Go like humans play, you're actually not allowed to put this white stone down here. It would be instant suicide. In Tromp-Taylor, it's actually fine. You put it down and then it immediately resolves to death. So the outcome is sort of the same.

我们重新开始，下几颗棋子，然后我再解释更多。

Let's go ahead and start over and play a few stones, and then I'll explain some more.

我就从那里开始。好的。我基本上是随机下棋，但我试图绕过你的棋子，看看我能不能包围它们。

I'll just start there. All right. I'm basically playing randomly here, but I'm trying to get around your stones and see if I can close them.

所以这一步基本上让你的白子暴露了一个空邻居。这很像国际象棋中的将军，如果你不立即在这里放一颗棋子回应，那么我就可以立即捕获这个棋子。

So this move basically exposes one empty neighbor for your white stone. And it's very akin to a check in chess, where if you don't respond immediately by putting one here, then I can immediately capture this.

我明白了。好的，因为决定你是否被保护的是交叉点，而不是对角线。

I see. Okay, because it is the cross-sections that determine whether you're guarded, not the diagonals.

所以这个棋子被三面包围了。

So this one is surrounded on three sides.

所以如果你不立即在那里放一颗棋子，你就有失去那颗棋子的威胁。

And so you're at threat of losing that stone if you don't play one immediately there.

现在你可以看到我开始给你施压，因为在这里放一颗棋子，现在你被迫在这里放一颗。

Now you can see that I'm starting to pressure you, because by putting a stone here, now you are forced to put one here.

否则你就会自己有一个两子块。

Otherwise you would have this two-block to yourself.

是的。如果你仔细想想如果你在这里回应会发生什么，你可能会搜索未来并推断我会如何回应。一旦你那样做，你对我的能力就很有信心。但我猜你会把黑子放在这里。

Yes. And if you think through what happens if you were to respond here, you can probably search into the future and deduce what I'll do in response. Once you do that, you have a lot of confidence in my abilities. But I'm guessing you'd put the black here.

没错。然后我就会捕获这三颗棋子。

That's right. And then I would capture all three of these stones.

所以我应该假设这个没了。这个小块没了。

So I should just assume that this is gone. This little block is gone.

是的。所以在围棋中，让对手捕获一些棋子实际上是可以的，例如，如果它让你能够定位以在棋盘其他地方捕获更多棋子。这就是围棋非常美丽的地方，你可以输掉战斗但赢得战争，对吧？随着棋盘尺寸增大，这些微观与宏观动态的复杂性变得更加有趣。

Yes. So in Go, it's actually okay to let opponent capture some stones if, for example, it allows you to position to capture more stones somewhere else on the board. And this is what makes Go a very beautiful game, is that you can lose the battle but win the war, right? And as the board size increases, the complexity of these micro versus macro dynamics gets more interesting.

但大概你会在这里放一颗。

But presumably you'd put one here.

所以现在我会捕获整个这一组。

And so now I would capture this entire group.

好的。还有一个我想演示的情况，实际上我最近在代码中有一个 bug，就是以下情况。让我们考虑这样一个阵型，对吧？然后棋盘上还有其他棋子。所以让我们谈谈游戏如何结束，对吧？在这块领地，谁控制这些区域？是白棋还是黑棋？

Okay. There's one more case that I want to demonstrate, which actually I had a bug in my code recently, which is the following situation. So let's consider a formation like this, right? And then we have other pieces on the board in play or whatever. And so let's talk a little bit about how the game ends, right? In this territory, who controls these areas? Is it white or is it black?

白棋。

White.

实际上是黑棋，因为我实际上包围了整个区域。假设我在这里有其他黑子，你实际上很难打破这些棋子的控制。

It's actually black, because I have actually surrounded this whole area. And assuming I have other black stones here, it's actually very hard for you to break this out of the control of these stones.

所以当最终计分时，这些也会算作在……

So when the final score is tallied, would these ones also count as being in...

是的，好问题。这就是不同规则集有不同的计分方式。所以我们应该谈谈人类之间如何解决得分，以及计算机代码之间如何解决得分，因为人类评估这一点实际上有些模糊。

Yeah, great question. So this is where different rule sets have different ways of scoring. And so we should talk a little bit about how you resolve scores between humans and how you resolve scores between computer code, because there's actually some ambiguity in how humans evaluate this.

所以大多数人类看到这个棋盘布局，会认为黑棋基本上完全包围了白棋，白棋没有活路。我们可以继续下下去，但最后我会吃掉所有白棋。

So most humans would look at this board configuration and conclude that black has kind of totally surrounded white and so white has no chance of life. We could play out more here but then at the end I would capture everything.

嗯，不过如果你有办法打破这个阵型，把白棋连接到外面的棋子，那局势就可能逆转。这就是计算机有点难以判断的地方，对吧？那么人类是怎么做的呢？值得思考一下人类如何解决这个问题，因为这会映射到我们后来对深度神经网络的思考。人类基本上会说“我觉得棋局结束了”，然后你也得说“我觉得棋局结束了”，接着我们会说“我认为这些是确定的地盘”，然后你必须同意。如果你不同意，我们就继续下。

Um, however if you have a way of breaking this formation and connecting white to something outside of it then it can flip right and so this is where it's a little bit hard for a computer to decide these kind of things. Right? So how do humans do it? Like it's worth thinking a little bit about how humans resolve this because this will actually map later to how we think about the deep neural network. Um humans basically say I think the game is done and then you have to also say I think the game is done and then we'll say like I think these are milestones and then you have to agree. If you don't agree then we keep playing.

是的。

Yeah.

所以本质上，一旦两个人的所谓价值函数达成共识，中国规则就据此判定。

So essentially once two humans' so-called value function agree on a consensus then the Chinese rules result that.

嗯，有意思。

Yeah. Interesting.

在 Tromp-Taylor 计分法下，这是完全明确的。所以计算机可以用算法判定。如果在终局阶段，计分方式是先数你控制了多少棋子，这很明确。然后数那些没有被对方棋子接触的空交叉点。所以这些交叉点不计入任何一方，因为它们同时连接着白棋和黑棋，对吧？如果像这样，白棋会得到 3 分。这有点奇怪，因为人类知道白棋实际上会失去这些点。但 Tromp-Taylor 计分法会认为白棋拥有所有这些点，以及这些点。

So in Tromp-Taylor scoring it's perfectly unambiguous. So it can be decided algorithmically by a computer. So if you have this at the end game, the way you score is that you first count how many stones you control and that's unambiguous. Then you count how many empty intersections that are not touched by your opponent's stones. So these intersections would not count for either player because all of these intersections are connected to both white stones and black stones. Right? If this were like this, then white would get three points. Now, this is a little odd because a human would know that white is actually losing these points. But Tromp-Taylor scoring would consider white to have all of these points as well as these points.

明白了。好的。

Got it. Okay.

对。所以计算机围棋和人类计分方式有很大不同。

Right. So that is a very big difference in how computer Go scores things and how humans score things.

游戏如何结束？当一方选择认输，或者双方连续弃权时，游戏结束。

How does the game end? The game ends when either a player chooses to resign or both players pass consecutively.

酷。

Cool.

是的。这就是规则。

Yep. So, that's the rules.

很好。好了。现在，帮我用 AI 纠正一下。

Nice. All right. Now, help me correct this with AI.

好的。

Great. Okay.

我们来理解一下 AlphaGo 实际上是如何工作的，以及听众中可能有人如何实现它。

Let's understand how AlphaGo actually works and how somebody in the audience might be able to implement it.

很好。是的。我们先直观理解一下用于走子的底层搜索过程，然后再加上深度学习的思想，使其更高效、更易处理。围棋是一个只有两名玩家的游戏。我们在这里画一个人，在这里画一个 AI。假设这个人执黑先走。我们在这里画围棋。然后 AI 根据它看到的情况走一步。问题是如何将这些输入编码给 AI。也许可以用 0 和 1，但你需要表示黑棋、白棋和空位。所以至少需要三个不同的值，对吧？也许可以用 0、1 和 2 之类的。所以 AI 可能看到像 0 0 0 1 这样的东西。这就是 AI 在它回合的输入。

Great. Yeah. Let's start with an intuition about the underlying search process used to make moves and we'll layer on ideas from deep learning to make it much more efficient and tractable. So Go is a game with just two players. We're going to draw a person here and an AI here. And let's say this person is playing black so they go first. So we're going to draw Go here. And then now the AI is going to make a move based on what it sees here. So there's a question of how you encode these inputs into the AI. Maybe you could use ones and zeros, but you want to represent black, white, and empty. So you would need at least three different values here, right? So maybe you could use zero, ones, and twos or something. So the AI might see something like 0 0 0 1. So this is the input to the AI on its turn.

所以 AI 可以选择。我们随便选三个可能的走法。我随机画了这些。那么哪个走法最好呢？嗯，我们直到游戏结束才知道。围棋中没有局部奖励来判断哪一步好。这就是围棋很难的原因：你实际上直到游戏结束才知道谁赢了。那么这棵树有多深？在 19x19 的棋盘上，任何一步都有大约 361 种走法。当然，随着棋盘填满，走法会减少。游戏步数大约在 250 到 300 步之间。也许专家会提前结束游戏。但在 Tromp-Taylor 计分法下，你必须一直下到最后。所以可能有 300 步左右，树的深度大约是 300。

So the AI can choose. Let's just pick three possible random moves that can go. And I just drew these at random. And so, which move is best here? Well, we don't know until the game ends. There's no local reward in Go for which move is good. And this is what makes Go a very difficult game: you don't actually know who won until you really get to the end of the game. So how deep is this tree? In a 19x19 Go board, there are roughly on the order of 361 moves on any given move. And of course, as it fills up, you have fewer moves. The number of steps in the game can be somewhere from 250 to 300 moves. And maybe experts might decide to end the game well before that. But under Tromp-Taylor scoring you actually have to play things all the way to the end. So this could be like 300 moves or something, so depth of the tree is about 300.

是的。所以如果你不断展开可能的走法，AI 走一步，人类走一步，以此类推，你会发现从这一个状态出发，可能的游戏结果会爆炸式增长。这大约是 361 的 300 次方，远远超过宇宙中的原子数。当然，存在冗余和对称性，所以实际上不是 300，但如果你做一个朴素的树，不合并子节点，那么你会得到一棵这么大的树。

Yeah. So if you keep expanding possible moves here, the AI goes, then the human goes, and so forth, you find that essentially you end up with an enormous explosion in the possible game outcomes originating from just this one state. So this is something on the order of 361 to the power of 300, which is far more than the number of atoms in the universe. And of course there are redundancies and symmetries, so it's not actually 300, but if you were to do a naive tree where there were no merging of children, then you end up with a tree about this big.

你说的合并子节点是什么意思？

What do you mean by merging of children?

对。我用这个棋盘来说明。如果我们从这里开始，你下这里，我下这里，你再下这里，这等价于我先下这里，你下这里，我下这里，你再下这里。所以两者到达了同一个位置，但路径不同，所以这个子节点可以看作一个共享祖先。我想不是 36，它从 361 开始，但每次减少 1，分支因子每次减少 1。但无论如何，这是一棵非常非常大的树。这也是为什么计算机科学家多年来认为围棋在本世纪是不可解的问题，因为穷举所有可能性所需的算力太大了。

Right. Let me use this board here. So if we start here and then you play here and then I play here and then you play here, that is equivalent to I start here, you play here, I play here, and then you play here. So both of them arrived at the same spot but through different paths, so this child node can be thought about as a shared ancestor. And I guess it's not 36, it starts at 361 but it decreases by one each time, and the branching factor decreases by one each time. But in any case, this is a very, very large tree. And this is also why computer scientists for many years thought that Go was not a tractable problem this century, because the amount of compute you would need to exhaustively search every possible possibility is just too large.

如果可以的话，围棋实际上是一个确定性游戏。所以在任何给定状态下，你实际上可以计算出为了赢得游戏你能做出的最佳策略。你可以搜索所有你能赢的未来，然后确保你始终停留在那组未来中。

If you could, Go is actually a deterministic game. So on any given state you can actually compute what the best possible strategy you can make is in order to win the game. You can search all the possible futures where you win and then just make sure you always stay in that set of futures.

嗯。

Mhm.

所以 AlphaGo 的核心概念突破是使用神经网络使这个搜索问题变得可处理。在我们讨论神经网络如何参与之前，先谈谈假设我们有足够强大的计算机，如何搜索这棵树以找到最佳走法。一开始，你不会构建整棵树，因为存储那棵树非常昂贵。相反，你可能会交互式地找出这棵树的哪些叶子值得探索并扩展到未来，看看还有什么。

So AlphaGo's core conceptual breakthrough was using neural nets to make this search problem tractable. So before we get into how neural networks are involved, let's talk a little bit about how we can, assuming we have a powerful enough computer, search this tree to find the best move. In the beginning, you're not going to build out the whole tree because storing that tree would be very expensive. Instead you might do something like interactively figure out which leaves of this tree are worthy of exploring and expanding into the future to see what else is there.

在赌博机文献中有一些早期算法，比如 UCB1，它并不完全适用于像围棋这样的序列游戏，但很大程度上启发了 AlphaGo 中使用的动作选择算法。UCB1 的思路是：每一步，我们选择最优动作，即最大化 Q(a) 加上一个探索奖励的 arg max。每个节点上，我们跟踪几个量。把每个节点看作一个数据结构。这是根节点，你从这里做决策，这些是子节点。每个节点存储该子节点的访问次数。

So there are some early algorithms in bandit literature like UCB1, which is not exactly appropriate for a sequential game like Go but very much inspired the action selection algorithm used in AlphaGo. UCB1 looks like: on every move, we take the best action, the arg max over a that maximizes Q(a) plus some exploration bonus. On every node, we track a few quantities. Consider each of these as a node. This is the root node where you're making decisions, and these are the children. Each node is a data structure that stores a visit count of this child node.

父节点访问这个节点的次数？

How often the parent visited this node?

是的。我们把这个叫做一个动作。容易搞混的一点是，如果你来自机器人学或其他强化学习领域，你会问：动作在哪里？我只在谈论节点。这里的节点代表状态，因为这是一个完全确定性的游戏，没有随机性，你实际上可以根据子节点推断出动作。所以如果我走到这里，那就隐含了一个动作，而这是我们最终到达的状态。如果你让 LLM 用 vibe code 实现一个 MCTS，它很可能会设计出正确的数据结构。但这就像厨师的选择；你可以按自己的喜好重写树结构。这是我让 Claude 4.6 写的，它做了一个非常合理的选择。

Yes. And we'll call this an action. One thing that is easy to trip on is if you come from robotics or other kinds of reinforcement learning: where are the actions? I'm only talking about nodes. Nodes here represent states, and because this is a perfectly deterministic game with no randomness, you can actually infer the action based on the child. So if I go here, that implies an action, and this is the state we resolve in. If you ask LLMs to vibe code an MCTS implementation, it will most likely design the right data structure. But it's a chef's choice; you can rewrite the tree structure however you like. This was what Claude 4.6 wrote for me when I asked it, and it was a very reasonable choice.

所以 Q 代表这个动作的平均动作价值。我用下标 a 表示这对应于从根节点采取特定动作到达这里。如果我们有根节点，采取 a 会让我们到达这个节点。我们还会存储从父节点采取这个动作的概率。

So Q represents the mean action value of this action. I'll use subscript a to denote that this corresponds to taking a specific action to get here from the root node. If we have root, taking a gets us to this node here. We're also going to store the probability of taking this action from the parent.

从父节点。就像我们采样到这个的概率？

From the parent. Like what are the odds that we sample this one?

从父节点，是的。这之后会用到。我们现在讨论的是确定性树；之后我会引入概率。最后，我们有一个子节点字典，就是更多这样的节点，构成一个经典的链表风格引用树。这是实现树的基本数据结构。在 AlphaGo 中，他们使用了一个略有不同的动作选择标准，叫做 PUCT，即预测上限置信树。当你选择要选哪个子节点时，你计算 arg max over a of Q(s,a) 加上一个常数。方程形式实际上非常相似。两者都是评分标准：你取这个量的 arg max 来决定采取哪个动作。

From the parent. Yes. This will become relevant later. We've talked about a deterministic tree for now; I'll bring probabilities into this later. Finally, we have a dictionary of children, which is just more of these nodes in a classic linked list style reference tree. This is the basic data structure to implement a tree. In AlphaGo, they use a slightly different action selection criterion called PUCT, short for Predicted Upper Confidence with Trees. When you select which child to take, you do arg max over a of Q(s,a) plus a constant. The equation forms are actually pretty similar. These are both scoring criteria: you want to arg max this quantity to determine which action to take.

我们来分解一下如何选择动作的直觉。这是平均动作价值：一个给定子节点平均有多好？如果你知道整棵树，那么这就是选择最佳动作所需的全部。但如果你在交互式地构建这棵树，同时找出 Q 值应该是什么，那么你必须偶尔尝试其他动作，作为一种探索与利用的权衡。在 UCB 和 PUCT 中，都有一个奖励采取之前未尝试过的动作的项。每个节点存储采取该特定动作的访问次数。所有值初始化为零。对于给定动作 a，初始 n 为零。随着 n 增加，如果我们已经从那个根节点做了 10 次动作选择，但还没有选过 a，那么这一项对 a 来说会变得相当大。相反，如果我们 10 次中选了 a 10 次，那么这一项就很小；它衰减得非常快。

Let's break down the intuition of how you select actions. This is the mean action value: how good is a given child on average? If you knew the whole tree, then this is all you need to select the best action. But if you're interactively building this tree as you figure out what the Q values should be, then you have to occasionally try some other actions as a sort of explore versus exploit trade-off. In both UCB and PUCT, there is a term that rewards taking actions that you haven't taken before. Each node stores the visit count of taking that specific action. Everything is initialized to zero. For a given action a, initially n is zero. As n increases, if we've already made 10 action selections from that root node but haven't picked a yet, then this term starts to become quite large for a. Conversely, if we have chosen a 10 times out of 10, then this term is quite small; it diminishes very quickly.

我们只关注 UCB。这里说的，你可以概念上把它想成两样东西：Q 和这个探索项。我们来明确一下 Q 是什么。Q 基本上是说：一旦我们做了这些 rollout，你运行所有这些模拟，沿着树往下走，然后判断如果我到达这棵树的终局值，我是否赢了这盘棋？然后你从该节点出发，对所有叶子节点平均我是否赢了这盘棋。这个平均值你放在 Q 里。

Let's just focus on UCB. What we're saying here, you can think of it conceptually as two different things: the Q and then this exploration term. Let's be clear about what Q is. Q is basically saying: once we do these rollouts, you run all these simulations, go down the tree, and then figure out if I end up at the terminal value of this tree, do I win this game or not? Then you average whether I win this game or not across all the leaves of this tree starting from this node. That average you put in Q.

正确。

Correct.

所以你是说 Q 基本上代表我是否会赢这盘棋？从该节点开始我赢棋的概率是多少？这就是你的利用部分。这就像说：我运行了这些模拟，我认为这是个好棋或不是。而另一项是说：相对于我可以探索或已经探索的其他动作，我是否已经充分探索了这个分支？如果我还没有探索这个分支，也许我认为它得分低，但我只是还没有探索这个节点下的那么多叶子。所以我应该试试这个，即使利用部分告诉我这个不那么有价值。而且因为 ln(n) 增长比 n 慢，随着时间的推移，arg max 将从被探索项主导转变为被 Q 项主导，这就像：好了，我做了足够多的模拟，我相当确信这是该走的分支。

And so you're saying the Q is basically representing will I win this game or not? What is the probability that I'll win this game starting at this node? That is your sort of exploit. That is like saying I've run these simulations, I think this is a good move or not. And then this other term is saying: have I explored this branch enough yet relative to the other actions I could be exploring or I have already explored? If I haven't explored this branch yet, maybe I think it has a low score but I just haven't explored that many leaves down this node. So I should maybe try this even though the exploit is telling me that this is not that valuable. And because ln(n) grows slower than n, over time you will move from the arg max being dominated by this exploration term to the arg max being dominated by the Q term, which is like: okay, I've done enough simulations, I'm quite confident that this is the branch to go down.

是的，没错。UCB 的动机是提出一种算法，如果你不知道各个臂（即你可以选择的不同动作）的回报，这种带有探索项的策略会限制你的遗憾，即你可能犯错的严重程度。我不知道证明；我也不知道这个算法是否被证明具有对数或平方根有界遗憾之类的性质，但我认为这个算法就是被推导成这个样子。你可以看出这些项的增长方式略有不同。

Yes, that's right. The motivation for UCB was to come up with an algorithm where if you don't know the payoff of the arms, the different actions you can select, this strategy with the exploration term bounds your regret in terms of how wrong you can possibly be. I don't know the proof; I don't know if this one is proved to have a logarithmically or square root bounded regret or anything, but I think the algorithm was just derived to look something like this. And you can tell that these terms grow a little bit differently.

而这实际上只是为了说明，围棋在每一步中可选择的动作比标准的多臂老虎机问题要多得多。

And this is actually just to account for the fact that Go has many more actions in every given move compared to your standard bandit problem.

有一点小澄清：你刚才谈到了模拟和概率。我们要记住，围棋本质上是一个确定性游戏。那么概率的概念从何而来？如果你有一台非常强大的计算机，就没有概率，你可以直接计算出平均动作价值的真实均值。那么概率从何而来呢？事实证明，在 AlphaGo 之前的计算机围棋中，我们一直使用某种蒙特卡洛方法，即取随机选择的树上的期望 Q 值，而这个随机选择的树就是概率的来源。所以 Q 的解释是：在某个随机搜索过程所诱导的随机分布下，期望的动作价值是多少。

One small clarification to make is that you talked a little bit about simulations and probability. We should remember that Go fundamentally is a deterministic game. So where does the notion of probability come from? If you had a very powerful computer, there are no probabilities; you can just compute the true average of the mean action value. So where does the probability come in? It turns out that in computer Go before AlphaGo, we've always done some sort of Monte Carlo method where we take the expected Q value averaged over a randomly selected tree, and that randomly selected tree is where probabilities come in. So the interpretation of Q is: what is the expected action value under the random distribution induced by some random search process.

有道理。

Makes sense.

那么随机搜索过程从何而来？这就是 p(action) 的用武之地。如果我们假设一个非常简单的算法，即每个合法动作被选中的概率均匀，那么概率就是合法动作数的倒数，你会在一个非常分散的树上取平均。这是一个有效的积分，但非常慢，因为你考虑了很多价值很低的树。这本质上是一个重要性采样问题：只有少数动作和路径能贡献高价值，其他几乎都是低价值。所以这是一个棘手的问题。

And so where does the random search process come in? That's where p(action) comes in. If we assume a very naive algorithm where you have a uniform probability of taking any valid action, then this would just be one over the number of valid moves, and you would be taking this average over a very diffuse tree. This is a valid integral, but it's very slow because you're considering a lot of trees that have very low value. It's essentially an importance sampling problem where only a few actions and paths contribute high value, and almost everything else is low value. So this is a tricky problem.

这就是决定探索哪些动作的动作选择标准。当你沿着树搜索向下时，最终会遇到一个节点，在游戏结束时，根据 Tromp-Taylor 计分规则，没有合法动作可走，此时很明显你赢了或输了。所以你要么赢要么输。这就是整个游戏的最终回报。问题是：我们可以给终端叶节点分配一个值 u，但如何给之前的节点（父节点）分配值呢？你只需取平均动作价值作为平均值。假设这些是叶节点。这个节点的平均动作价值就是叶节点上输赢的平均值。相应地，你可以向上回溯，说这个节点的平均动作价值（我们称之为动作 B 的 QB）就是这些样本的加权平均。加权平均可能取决于是否有不同的采样分布。但基本直觉是：你想在确定输赢的地方解决游戏，然后向后回溯。这被称为备份步骤，根据最终终端叶节点的平均值给这些节点或动作赋值。

This is the action selection criteria for how you decide which moves to explore. As you move down in tree search, you will eventually run into a node where it's quite clear you've won or lost at the very end of the game when there are no valid moves left under Tromp-Taylor scoring. So you either win or lose. This is the final return of the whole game. The question is: we can assign a value u to a terminal leaf node, but how do we assign values for nodes prior to that, the parents? What you simply do is take the mean action value as your average. Suppose these were leaf nodes. The mean action value of this node is just the average of whether you won or lost at the leaf nodes. Correspondingly, you can walk up the chain and say the mean action value of this node, let's call it QB for action B, is just the weighted average of these samples. The weighted average could depend on whether you have a different sampling distribution. But the basic intuition is that you want to resolve the game where you have a deterministic win or lose, and then you can go backwards. This is called the backup step, assigning values to these nodes or actions corresponding to the average over the final terminal leaves.

你之前关于那种对人类来说谁赢很明显、但又不明显、或者必须确定性地玩下去的状态的解释，确实让我理解了为什么价值函数既可训练，又是有效学习这个游戏所必需的。也许首先值得定义一下什么是价值。

Your explanation earlier about the sorts of states where it's obvious to a human who's going to win, but it's not obvious, or like you deterministically still had to play it out, actually drove home the intuition of why the value function both is trainable and why it's necessary in order to effectively learn this game. Maybe it's worth defining value in the first place.

好的。我们谈到了这个 u 值，即你最终是赢是输的判定，这是终端叶节点条件。人类不会一直玩到树的叶子；在高水平对局中，他们会在几十步甚至上百步之前就停下来。那么他们怎么知道呢？你可以认为人类隐式地拥有一个称为价值函数的神经网络，它接收棋盘状态并评估是否会赢。人类看一眼棋盘就知道自己可能要输了。他们本质上是在运行一个神经网络，这个网络看着棋盘，隐式地平均了大量可能的游戏推演，然后判断棋盘是否可赢，以及是否应该认输或继续下。这很了不起：人类大脑中的神经网络可以在几秒钟内通过一瞥完成所有这些模拟，而不需要实际下完每一局棋，这基于结晶化的知识和经验。这给我们一个提示：在围棋这样的游戏中，有办法从根本上加速搜索过程。这就是 AlphaGo 之所以有效的基本直觉之一：你可以训练一个价值函数，让它看一眼棋盘就能快速解决游戏，而不需要将所有树搜索到很深的深度。

Sounds good. So we talked about this u value being your final resolution of whether you won or lost, and this is the terminal leaf node condition. Humans don't play all the way to the leaves of the tree; they stop dozens or even hundreds of moves before in high-level play. So how do they know? You can think about humans as implicitly having a neural network called a value function that takes in a board state and evaluates whether they will win. The human glances at the board and knows they're probably going to lose. They are essentially running a neural network that looks at a board and implicitly averages a huge number of possible game playouts, then decides whether the board is winnable or not, and whether they should concede or keep playing. This is remarkable: a neural network in a human can somehow do all this simulation at a glance and just know within a few seconds without actually playing every single game, based on crystallized knowledge and experience. This gives us a hint that in games like Go, there are ways to radically speed up the search process. This is one of the fundamental intuitions behind why AlphaGo works: you can train a value function to look at a board and quickly resolve the game without playing out all these trees into a very deep search depth.

是的，有道理。

Yep. Makes sense.

我想对听众说，在准备前几期节目时，我发现理解 AlphaGo 的工作原理非常令人困惑，但这与本期内容相关。不过，一旦你以这种方式理解了问题，再逐步构建后续部分，实际上会更容易理解，而且会很有意义。现在感到困惑没关系，但到这次讲座结束时，它可能比你预期的更简单。所以我就先给听众提个醒。

I will say for the audience, I sort of found for previous episodes when I was prepping, and it would seem somewhat relevant to understand how AlphaGo works, I would find it very, very confusing. But it's the kind of thing where once you understand the problem in this way, and then you'll build the next few pieces, it is actually much more understandable and it will make a lot of sense. And it's okay to be confused right now, but it's probably simpler to understand by the end of this lecture than you anticipate. So I'll just make that note for the audience.

是的，高层次的重要直觉是，退一步看我们这一切的走向：对于围棋这样的游戏，传统上你可以构建一棵树，但我们没有足够强大的计算机来做到这一点。

Yeah, the important intuition at a high level, just to step back about where we're going with all this, is that classically for games like Go, you could build a tree, but we don't have computers powerful enough for that.

是的。

Yeah.

而且，估计你可能采取的每个行动的价值也很难，因为直到游戏结束你才知道结果。你可以通过玩到结束来取平均值，但这也很困难，因为你不知道应该采取哪些行动来采样这些平均值。所以概念上，这里有两个问题：树的广度，然后是树的深度。而 AlphaGo 提供了一种方法，基本上将这两个问题都缩小到非常易于处理的程度。

And estimating the value of every action that you could possibly take is also hard because you don't know until the end of the game. You could take averages by playing them to the end, but that's also hard because you don't know which actions to take to sample these averages. So conceptually, there's kind of two problems there: there's the breadth of the tree and then there's the depth of the tree. And AlphaGo gives us a way to basically shrink both of those to be very tractable.

是的，这基本上就是它背后的核心思想。

Yeah, that's essentially the kind of core idea behind it.

好的。所以我们采用这个想法：人类可以扫一眼棋盘，立即预测我们是否会赢，这或许给了我们机会来真正截断搜索的深度。然后我们也知道，人类可以看一眼棋盘，凭直觉一眼就决定哪些走法可能在围棋棋盘上是好的。所以这大致就是我们可以用深度神经网络来加速搜索过程的两件事。

Okay. So we take this idea that humans can glance at a board and instantly predict whether we win, and maybe that gives us the opportunity to really truncate how deep we search. And then we also know that humans can look at a board and intuitively at a glance decide what moves might be good on a Go board. So these are kind of two things that we can use deep neural networks for to accelerate this search process.

在我们讨论神经网络之前，先回到这个对局过程是如何运作的。我们只讨论了走一步棋，对吧？所以 AI 查看这个编码后的围棋棋盘。它有一棵树。它深入搜索这棵树，找出哪个行动可能是最好的，然后采取那个行动。然后现在轮到人类。所以也许现在人类看到一个像这样的棋盘，然后他们走一步棋。也许他们把棋子放在这里。然后现在我们回到 AI，它现在查看一个新的编码棋盘。所以我用 2 表示 AI 执白，1 表示人类执黑，0 表示空位。然后现在轮到 AI 时，它从头开始重新进行 MCTS 树搜索。对吧？所以它丢弃了上一轮搜索过的旧树，现在有了一个新的根节点，它开始重新搜索，以此类推。所以 MCTS 基本上是一种搜索算法，用于决定走什么棋，最好由神经网络辅助，并且每一步都执行。

Let's go back before we talked about neural nets. Let's just go back to how this playout works. And we've only talked about making one move, right? So the AI looks at this encoded Go board. It has a tree. It searches deeply into the tree to find out which of its actions might be the best, and then it takes that action. And then now it goes back to the human. So maybe now the human sees a Go board that looks like this, and then they make their move. So maybe they put their stone here. And then now we go back to the AI, which now looks at a new encoded board. So I've used two to denote the AI's playing as white and one to denote the human playing as black and zero as empty. And then now on the AI's turn, it does the MCTS tree search all over again from scratch. Right? So it throws away this old tree that it searched last round, and now there's a new root node, and it begins to search anew, and so forth. So MCTS is basically a search algorithm that is deciding what moves to play, best aided by neural networks, and it's done on every move.

好的，很好。那么我们来谈谈其中的神经网络部分。在你快速讲解的时候，另一个对我来说很重要的理解点是这个 MCTS 数据结构，包含节点和子节点等等。这是每步棋都做的，一旦走了一步棋就重新实例化。所以人类走一步，然后 AI 查看这个，基本上尝试运行一系列模拟，以确定下一步该走什么。这些模拟，一次模拟基本上就是探索 MCTS 中的一个新节点。最后，一旦你运行了一千次模拟，就会得出下一步走法的概率，这就是你存储的内容。你根据这些概率选择最佳走法。然后你丢弃所有这一切。接着下一个玩家走一步，你在每一步开始时重新启动这个过程。

Okay, great. So let's talk about the neural network part of this. And while you're racing, another sort of thing that was important for me to understand was this MCTS data structure with nodes and children of nodes and whatever. This is done per move and reinstantiated once a move is made. So a human makes a move, then the AI looks at this and is trying to basically run a bunch of simulations to figure out what move should I make next. And those simulations, a simulation is basically like exploring one more node in this MCTS. And at the end, once you run a thousand simulations, that informs the probability of what move to make next, that's what you store. You sort of choose the best move given those probabilities. You discard all of that. Then the next player makes a move and you restart this process at the beginning of every move.

正确。有一点补充：你不会丢弃所有东西。你会保留一样东西，我们稍后会用到。

Correct. One small addendum: you don't discard all of that. You keep one thing behind that we'll use later.

是的。就像我为 Reiner 做的那样，我想为这一集制作闪卡，以便人们能够记住这些概念。理想情况下，LLM 可以生成一些候选，然后我进行精炼。但要真正获得高质量的建议，我需要设计一个完整的流程，让 AI 能够在正确的时间戳截取黑板的截图，然后在视觉有帮助的情况下制作 SVG 图表，然后通过一个批评者运行它们的写作和绘图，然后根据这个反馈修改卡片。仅仅通过堆叠 LLM 调用很难实现这一点。这种逐步的配方效果要好得多，如果你有一个持久的智能体，它已经在所有前面的阶段参与了任务。所以我使用 Cursor SDK 为每张卡片启动一个智能体。Cursor 的框架为我节省了大量工作，无需设计自定义上下文脚手架或弄清楚如何设计用于截图或制作动画的工具调用。这些智能体都在云端运行，所以我不用担心笔记本电脑一直开着。当有候选需要审查时，我会收到一封电子邮件。你可以在 flashcards.warcash.com 查看我的卡片。你可以通过 cursor.com/swarth 开始使用智能体 SDK 进行构建。

Yeah. Just like I did for Reiner, I wanted to make flash cards for this episode so that people could retain these concepts. And ideally, an LLM could generate some candidates for me to then refine. But to actually get high quality suggestions, I needed to design a whole pipeline where the AI could take and ingest screenshots of the blackboard at the right timestamps and then make SVG diagrams in case visuals were helpful, and then run their writing and drawing through a critic and then revise the card in response to this feedback. It's very hard to accomplish this just by stacking LLM calls. This sort of step-by-step recipe works much better if you have a durable agent that's been engaging with the task across all the previous stages. So I use the Cursor SDK to spin up an agent for each card. The Cursor harness saved me a bunch of work in designing some custom context scaffold or figuring out how to design tool calls for taking screenshots or making animations. These agents all run in the cloud, so I don't have to worry about leaving my laptop open. I just get an email when I have candidates to review. You can check out my cards at flashcards.warcash.com. You can start building with the agents SDK at cursor.com/swarth.

好的，现在我们有了关于如何通过搜索做出走法的基本直觉。我们将讨论神经网络如何通过提供类似于人类直觉的模拟来加速这个过程。

Okay, so now we have a basic intuition of how moves are made with search. We're going to talk about how neural networks can speed this up by providing an analog to the human intuition.

所以有两个网络。一个是价值网络，它接收一个状态并预测我会赢还是输。这是一个二分类问题。然后我们有一个策略网络，它产生一个关于好行动的概率分布。

So there are two networks. There is the value network, which takes in a state and predicts whether I am going to win or lose. It's a binary classification problem. Then we're going to have a policy network, which induces a distribution over good actions to take.

所以我要画一个一维展平的走法分布，但这实际上是一个方形网格，对吧？所以也许它认为像这样的行动就是好行动的概率分布。这两个都是分类问题，对吧？所以你可以像训练深度学习中的任何分类器一样训练它们，使用交叉熵损失之类的东西。

So I'm going to draw a one-dimensional flattened move distribution, but this is really like a square grid, right? So maybe it thinks actions like these are the kind of probability distribution over good actions. And both of these are categorical classification problems, right? So you can train this like any classifier in deep learning with cross-entropy loss, that kind of stuff.

所以具体的架构实际上并不太重要。我尝试了几种不同的架构。Transformer 有效，ResNet 在小数据情况下也有效。我的经验是，ResNet 仍然有点优于 Transformer，并且在较低预算下性价比更高。但这可能不总是对的。

So the specific architecture does not actually matter too much. I tried a few different architectures. Transformers work, ResNets work for small data regimes. My experience is that ResNets still kind of outperform transformers and gave you more bang for the buck at lower budgets. But this may not be true.

为什么？它们提供了局部卷积的归纳偏置，而通常当你需要更多全局上下文时，Transformer 开始超越残差卷积网络。

Why is that? They provide the inductive bias of local convolutions, and generally transformers start to outperform residual convolutional networks when you want more global context.

我明白了。

I see.

那么，Kadiggo 论文中一个有趣的发现是，他们发现将全局特征汇集起来并在整个网络中聚合，以给网络一种全局感，使其能够将棋盘一侧的价值与另一侧连接起来，这实际上非常有用。但聚合全局特征是什么意思呢？

So, one interesting finding from the Kadiggo paper was that they found it actually quite useful to pull together global features and aggregate them throughout the network to give the network a global sense of how to connect value from one side of the board to another. But what does it mean to aggregate global features?

是的。如果你有一个非常大的 19x19 围棋棋盘，这里有一些战斗，那里也有一些战斗。当你通过卷积神经网络处理它时，卷积网络的感受野擅长计算局部特征并使其保持不变。但它们无法轻易地将这两个特征连接起来，对吧？它们需要以某种方式被拉在一起并相互关注。所以关于为什么 Transformer 适合计算机视觉任务（比如视觉 Transformer）的论点是，因为它们具有某种全局注意力，可以更容易地建立这些连接和预测。但你需要更多的数据，以便通过数据学习不变的局部特征。

Yeah. So if you have a very large 19x19 Go board, and you have some battles going on here and some battles going on there. When you pass this through a convolutional neural network, the receptive fields of the convolutional network are good at computing local things and making that invariant. But they won't be able to connect these two features easily, right? They need to be pulled together and attend to each other somehow. So the argument about why transformers are good for computer vision tasks, like with vision transformers, is that because they have a sort of global attention across the whole thing, they can more easily draw these connections and predictions. But you do need more data so that you can learn the invariant local features through data.

我非常努力地尝试让 Transformer 适用于这个问题，因为我很好奇 Transformer 是否会在围棋中带来某种突破，并去掉很多那些技巧。但尽管我尽力了，我实际上还没有找到让 Transformer 比 ResNet 更好的方法。

I've tried very hard to make transformers work for this problem because I was curious if transformers would present some sort of breakthrough in Go and remove a lot of those tricks. But try as I might, I actually haven't figured out a way to make transformers better than ResNets for now.

所以还有一个题外问题。为什么具有全局信息汇聚能力的 Transformer 会更好，这说得通，如果你需要考虑的不仅仅是空间局部信息的话。CNN 给你一种偏差，即相邻的事物特别相关，然后它们被聚合起来。但假设对于某些游戏，局部发生的事情并不那么相关，你只需要考虑全局。你说 Transformer 会更好。那么，对于涉及空间维度的游戏呢？时间维度呢？目前我们只考虑上一步，因为这是一个确定性的完全信息游戏。但如果像扑克或外交那样，之前的一个虚张声势对于理解现在很重要，并且要决定下一步行动，你需要考虑所有之前的状态。这会改变对哪种归纳偏差最相关、哪种架构最相关的考量吗？

So one more tangential question. It makes sense why transformers with their global pooling of information would be better if you need to consider information that is not just spatially local. CNNs give you a bias that things next to you are especially relevant, and then they are aggregated up. But suppose for games where it isn't that relevant what is happening locally, you just have to consider the whole thing. You're saying transformers would work better. How about games where we talk about the spatial dimension? How about the temporal dimension? Right now we're only considering the previous move because it is a deterministic full information game. But what if it was something like poker or diplomacy where a bluff they made a while back is relevant to understanding now, and to decide your next move you need to consider all those previous states. Would that change the consideration of what inductive bias is most relevant and what architecture is most relevant?

没错。好问题。围棋是一个完美信息游戏。在完美信息游戏中，存在一个纳什均衡策略，你使用它不会比任何其他策略更差。如果你知道对手有特定的偏好，比如他们喜欢激进地玩，你实际上原则上可以比纳什均衡策略更好地针对那个特定策略。但是，要对抗任何给定策略，存在一个单一的纳什均衡，可以仅使用当前状态来决定。这是大多数围棋智能体（如 AlphaGo）选择的设计选择，事后看来效果非常好，因为纳什均衡似乎是超人的；没有人类策略能击败它。现在有一些变体，你实际上需要考虑时间历史。这是一个非常令人兴奋的研究领域，我鼓励人们 fork 我的仓库并尝试这些。比如，如果你玩 2v2 围棋，那么你实际上需要建模你的伙伴的行为，而且你可能没有关于他们如何玩的信息。所以你需要聚合一些关于他们如何玩的信息，以便做出相应的回应。这些情况不再是完美信息游戏。在不完美信息或部分可观测的游戏中，你确实需要一些上下文来构建模型。

Right. Great question. So, Go is a perfect information game. And in perfect information games, there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy. If you know that your opponent has a particular bias, like they love to play aggressively, you can actually in principle counter that specific strategy better than a Nash equilibrium policy. But to counter any given strategy, there does exist a single Nash equilibrium that can be decided solely using the current state. That is a design choice that most Go agents like AlphaGo chose to do, which in hindsight turned out to work very well because the Nash equilibrium seems to be superhuman; no human strategy seems to be able to beat it. Now there are variations of this where you would actually need to consider temporal history. This is a very exciting research area that I would encourage people to fork my repo and try these things out. If you were to play, let's say, 2v2 Go, then you actually need to model your partner's behavior, and you may not have information on how they play. So you need to aggregate some information on how they play so that you can respond accordingly. These are situations where it's no longer a perfect information game. In games of imperfect information or partial observability, you do need some context to build a model.

是的。

Yeah.

我认为这是一个非常令人兴奋的地方，无论是自对弈还是外交风格。

And I think that's a place where things get very exciting in terms of self-play or diplomacy style.

是的。有意思。

Yeah. Interesting.

好的。回到神经网络架构，它并不是超级重要。你可以用 Transformer 让它工作。你也可以用 ResNet 让它工作。我发现对于低预算的实验，ResNet 效果稍好一些。你还可以使用一种 Karpathy 风格的自动研究超参数调优来使你的架构相当不错。所以你不需要太担心这个。你只需要设置问题，以便有一个目标优化。

Okay. So returning to the neural network architecture, it's not super important. You can get it to work with transformers. You can get it to work with ResNets. I found that for low-budget experiments, ResNets work a little better. You can also use a kind of Karpathy-style auto-research hyperparameter tuning to make your architecture pretty good. So you don't have to worry too much about that. You just need to set up the problem so that you have a target optimization.

是的。

Yeah.

好的。所以我们要选择一个有点随意的架构，它对我所做的有效。但再次强调，这部分不是超级重要。你有编码后的棋盘状态，我们选择三个通道，类似于 RGB：一个通道编码黑子，一个通道编码白子，还有一个通道可能编码空位或掩码区域（如果你想在多种棋盘大小上训练）。我现在实际上不打算讨论多种棋盘大小；那有点太复杂了。所以我们只说我们有这个两或三通道的类似 RGB 的图像，然后进入一个 ResNet，然后有两个分支头。一个头预测价值函数，这是一个单一的 logit，所以是 R1。然后我们有策略头，是 R361。这就是架构，我们基本上要训练它根据棋盘状态预测游戏结果，同时也要训练它预测什么是好棋。

Okay. So we're going to pick a somewhat arbitrary architecture that worked for what I did. But again, this part is not super important. You have your encoded board state, and we're going to choose, let's say, three channels, similar to an RGB: one channel to encode black, one channel to encode white, and one channel maybe to encode empties or a masked region if you want to train on multiple board sizes. I'm actually not going to talk about multiple board sizes for now; that's a little too complicated. So we'll just say we've got this two or three channel RGB-like image, and then we go into a ResNet, and then we have two branching heads. One head predicts the value function, and this is a single logit, so R1. And then we have the policy, which is R361. So this is the architecture, and we're going to basically train this to predict the outcomes of games given the board state, and we're also going to train this to predict what are good moves.

对。所以最初的 AlphaGo 论文，或者说 AlphaGo Lee，用专家人类对弈的监督学习数据集初始化了这个网络。后来他们通过让模型自学如何下好棋去掉了这个限制。但我发现，从实现的角度来看，对于你的听众来说，总是将实验初始化为简单的东西，然后在尝试啃下整个问题并从零学习之前让问题先跑起来，是非常好的。你通常想要初始化，就像在深度学习中，初始化就是一切。你总是希望将你的研究项目初始化为尽可能接近成功。

Right. So the original AlphaGo paper, or called AlphaGo Lee, initialized this network with a supervised learning dataset of expert human play. Later they removed this restriction by having the model teach itself how to play well. But I find it from a matter of implementation for your audience, super nice to always initialize your experiments to something that's easy, and then get the problem working before trying to bite off the whole thing and learn from tabula rasa. You generally want to initialize just as in deep learning, initialization is everything. You always want to initialize your research project to something as close to success as possible.

嗯，尤其是当你做一些从未做过的新事情时。总是先选一个能用的东西，然后让它做得更好，而不是从一个完全不能用的东西开始，再试图让它工作。基于这个理念，从一个好的初始化开始是个好主意。所以我们要拿人类专家的对局，训练这个模型来预测好的动作。我们会拿所有赢棋，也就是专家获胜的所有棋步，然后预测这些动作。接着，无论棋盘状态如何，不管你是赢了还是输了，都要预测结果。

Um, especially if you're doing something new that you haven't done before. Always pick something that works and then get it to do something better, rather than start from something that doesn't work at all and then try to make it work. Under that philosophy, it's a great idea to start from something that has a good initialization. So we're going to take human expert plays and train this model to predict good actions. We're going to take all the winning games, all the moves in which an expert won, and predict those actions. Then, regardless of board state, whether you won or lost, you're going to predict the outcome.

嗯。

Yeah.

所以你可能在想：好吧，有些早期的棋盘，基本上只下了一颗子，你怎么可能知道这局棋谁会赢呢？如果你有几十万局棋，那么平均来看，你会看到这样开局的棋局中，一半的分支会赢，一半会输。所以这其实没问题。当你训练模型预测这些时，logit 会收敛到 0.5。所以对于这些情况，一旦你训练好模型，起始棋盘状态看起来就是 0.5，然后随着你接近棋局结束，胜率要么这样走，要么那样走。这就是你的步数。

So, you might be wondering: okay, well, some of the early boards, where basically only one stone has been put down, how could you possibly know who the winner of this game is? Well, if you have hundreds of thousands of games, then on average, you'll probably see that boards that start like this have half of the games that branch off from this will win and half will lose. So that'll actually be fine. When you train this model to predict those, the logit will sort of converge to 0.5. So for these things, it's expected that once you train the model, a starting board state will look like 0.5, and then as you progress towards the end of the game, the win probability will sort of either go like this or it'll go like this. And this is sort of your move number.

嗯。

Yeah.

所以当你进入棋局几百步后，在专家数据分布下，谁更可能赢或输就变得非常清晰了。

And so as you get hundreds of steps into the game, it becomes much more clear who's more likely to win or lose under your expert data distribution.

我不明白为什么这种思考价值的方式对专家数据特别重要。

I didn't understand the significance of why this way of thinking about value is especially relevant to the expert data.

这和专家数据无关。这对你训练的任何数据都成立。所以如果你从零开始学习，你也会看到这个现象。所以如果你这样做，想象你在 vibe coding AlphaGo，你从网上的围棋教程收集了一些专家数据集，或者你有一个人类棋手的数据集，然后训练这个模型，结果发现这个模型已经是一个相当不错的围棋棋手了。它很可能击败大多数人类棋手。所以如果你只取这个策略推荐，对概率取 argmax，然后就把这个动作作为你的围棋走法，它会是一个非常快的围棋棋手，不需要推理步骤。它只是凭直觉下棋，而且会是一个非常强的围棋棋手，这已经很神奇了——想想看，10 层神经网络，可能不到 300 万个参数，就能做到这么令人印象深刻的事情。

It is not relevant to the expert data. It's true for any data that you train on. So if you were to learn a tabula rasa, you would also expect this to fall out. So if you just do this, imagine you're vibe coding AlphaGo and you gather some expert datasets from like how to go online, or you have a dataset of human players and you train this model, it turns out this model is already a pretty good Go player. It'll most likely beat most human players. So if you just take this policy recommendation and take the argmax over the probabilities, and you just take this action as your Go play, it'll be a very fast Go player that doesn't think in terms of reasoning steps. It just kind of shoots from the hip, and it'll be a very strong Go player, which is already quite miraculous if you think about 10 neural network layers maybe under 3 million parameters can already do something that impressive.

嗯。

Um.

所以你可以这样开始，在实现时重要的是验证这很可能是真的。最好验证你的围棋规则实现正确，你可以相对快速地运行这些模拟。就像一个检查点，你要确保在尝试叠加更复杂的东西（比如搜索）之前，你确实能完成这个基本步骤。

And so you can start this way, and it's important when implementing this to kind of just verify that this is probably true. It's good to verify that your Go rules are implemented correctly, that you can run these simulations relatively quickly. And just as almost like a checkpoint, you want to make sure that you can actually do this basic step before you try to layer on more complex things like search.

嗯。

Yeah.

但我们可以做得比直接拿原始神经网络下棋好得多。这就是我们如何将其应用于蒙特卡洛搜索。所以让我们用神经网络来改进蒙特卡洛搜索。我们从根节点开始，现在有一个四步迭代过程来做 MCTS。我第一次读论文试图理解时，这让我很困惑。但本质上我们要做的是选择一定数量的模拟。比如 num_simulations，这个数字是可变的。可以在 200 到 2048 之间。我相信在 AlphaGo 与李世石的对局中，他们每步棋用了数万次模拟，因为他们想尽可能提升模型的强度。

But we can do a lot better than taking the raw neural network and playing the moves. And this is how we can apply it to Monte Carlo search. So let's apply the neural network to improve Monte Carlo search. We start with our root node and we now have a four-step iterative process to do MCTS. This tripped me up when I was first reading the paper and trying to understand it. But essentially what we're going to do is we're going to choose a number of simulations. So like num_simulations, and this number varies. This can be somewhere between 200 to 2048. I believe in the AlphaGo Lee match they use tens of thousands of simulations per move because they really wanted to boost the strength of the model as much as possible.

嗯。

Yeah.

但在训练中你实际上不需要太多。KataGo 我想也用了这个量级。

But in training you don't actually need too many. And KataGo I think uses something on this order as well.

你知道他们比赛时用笔记本电脑了吗？他们没用笔记本电脑本身。好像是在某个 TPU pod 上。说实话有点不公平。李世石又不是用 1e22 次浮点运算来下一步棋。

Do you know if they used a laptop during the game? They didn't use a laptop itself. It was like on some TPU pod, I think. Honestly kind of unfair. Like Lee is not using like one E22 flops to do a move.

有道理。有趣的是，现代围棋机器人在测试时不需要那么多算力。当我们讨论 MCTS 策略改进如何工作时，我们会发现，随着时间的推移，原始网络实际上把那个大 TPU pod 的所有负担都吸收进了网络，你可以通过一次神经网络前向传播完成所有工作。但 TPU pod 总是会在上面增加额外的动力。所以他们比赛时想要那个。

Fair enough. Interestingly enough, modern Go bots don't need that much compute at test time. And what we'll actually find out as we talk about how the MCTS policy improvement works is that over time the raw network actually takes all of the burden of that big TPU pod and just pushes it into the network, and you can do all of that work with one neural network pass. But the TPU pod will always add the extra oomph on top. And so that's what they wanted for the match.

所以我们要选这个 num_simulations 参数。对于每次模拟，我们基本上要同时做几件事。我们要看当前树中哪些走法最好。如果到了需要添加叶节点的地方，我们就添加额外的叶节点。然后我们要更新树的动作值。所以每次模拟都涉及这个四步过程。四步过程基本上是选择、扩展、评估和回溯。

So we're gonna pick this num_simulations thing. And for every simulation, we're basically going to do several things simultaneously. We're going to see which moves are the best in the current tree. We're going to add extra leaves to the tree if we get to a point where we need to add a leaf. And we're going to update the action values for the tree. So every simulation involves this kind of four-step process. The four-step process is basically selection, expansion, evaluation, and backup.

所以在蒙特卡洛树搜索开始时，我们的树非常简单。它只有根节点，也就是我们的 AI 想要下的当前棋盘。我们基本上要为它选择最佳动作。所以当这个根节点被创建时，我们也知道可以在神经网络下评估它，得到 v_theta 以及动作概率，我称之为根节点。所以对于这里的所有动作，我们可以创建一堆子节点。所以这个节点有——在这个例子中，我画了一个 3x3 的棋盘，缺了一个格子。所以基本上有八个可能的子节点与这个根节点关联。每个子节点都有一个相关的选择该动作的概率。所以有 P8、P1、P2 等等。

So at the beginning of our Monte Carlo tree search, our tree is very basic. It only has the root node, or our current board that our AI wants to play at. And we're going to basically select the best action for this. So when this root node is created, we also know that we can evaluate this under our neural network and get the quantities v_theta as well as our probability over actions, and I'm going to say root. So for all of the actions here, we can create a bunch of children. So this one has, in this case I'm drawing a 3x3 board with one board missing. So basically there are eight possible children associated with this root node. And each of these has an associated probability of taking that action. So there's P8, P1, P2, etc.

在蒙特卡洛树搜索开始时，我们有一个根节点，可以初始化一些子节点，对吧？因为策略网络在根节点上评估，在一个 3x3 棋盘上，已经放置了一颗棋子，会给出 AI 可能采取的八个子节点。每个子节点的策略网络也会给出选择该子节点的概率。所以第一步是树的 Selection。同样，这是一个非常浅的树，目前我们只有深度为 1 的树。所以我们的第一步是通过最大化或 argmax PUCT 标准来选择，即 Q(s,a) + c_puct * P(a) * sqrt(N) / (1 + N(a))。对于每个动作，初始 N(a) 为 0，N 为 0，所以我们基本上根据这个来选择。最初，选择的动作很可能偏向于概率最高的动作，因为这些概率大致均匀。假设 P1 是概率最高的节点，所以你选择了这个节点。现在你到达这个节点，发现它不是一个叶节点，游戏还没有结束，所以无法得到最终结果。下一步是 Expansion。

So at the beginning of our Monte Carlo tree search, we have our root node and we can initialize it with some children, right? Because we know the policy network evaluated on the root node gives us, on a 3x3 board with one existing stone placed, eight possible children that this AI could take. With each of the children, their policy network also gives us the probability of selecting that child. So the first step is to do the selection of the tree. Again, this is a very shallow tree. All we have so far is a tree of depth one essentially. So our first move is to select by maximizing or argmaxing the PUCT criteria, which is basically Q(s,a) + c_puct * P(a) * sqrt(N) / (1 + N(a)). For each of these, N(a) is zero for all the actions initially, N is zero, so we're going to basically just pick according to this. Initially, what is going to be the chosen action here is most likely going to be biased towards the highest likelihood action here, because these are sort of uniform for every. So let's suppose P1 was the highest probability node. So you selected this one here. Now you got to this node and you realize that it's not a leaf node, right? There are more, it's not a terminal game. So you cannot resolve the final resolution. So the next step that you do is expansion.

然后你会将这个节点（棋盘状态）输入策略网络。注意，这是 AI 的走法，对吧？AI 正在走这一步。当我们扩展这棵树时，我们考虑的是人类或任何对手可能做什么，所以这代表你的对手。

So you will then run this node, this board state, through the policy network. Note that this is the AI's move, right? The AI is making this move. And so when we expand this tree, we're now thinking about what the human might do or any opponent might do, right? So this is like your opponent.

树的扩展过程实际上是完全……当我们评估这个节点时，我们将从该玩家的角度来评估这个节点。

The tree expansion process actually is completely... So when we evaluate the node here, we're going to now evaluate the node from the perspective of this player.

是的，没错。所以这个节点有我们可以采取的可能动作，我们基本上扩展了这里的叶节点。对于每个可能到达的节点，我们检查这些节点的好坏。比如从这里，人类可以下这里，这里或这里。我们会存储每个节点的 v_theta，比如节点一撇的 v_theta，节点一的 v_theta 等。我们基本上用神经网络来直观猜测这个棋盘从该玩家角度来看有多好。幸运的是，因为这是一个零和游戏，很容易推导出该玩家在这一步的价值就是 1 减去从另一视角的价值。所以根据当前玩家翻转搜索过程很容易。这就是扩展步骤。你拿一个非叶节点，扩展它并评估价值。这本质上是一个快速猜测：如果我下到最后，我会赢吗？你可以把 v_theta 看作是对任何给定模拟搜索到树末端的捷径。

Yeah. Right. So then this one has possible actions that we could take and we expand basically the leaf nodes here. So for each of these nodes that we could arrive at, we're going to now check how good those nodes are. So maybe from here, the human could play here, the human could play here, or human could play here. And we're going to store essentially the v_theta for each of these things. So v_theta of node one prime, v_theta of node one, etc. We're basically using our neural network to make an intuitive guess of how good this board is from the perspective of this player. And fortunately, because it's a zero-sum game, it's easy to deduce that the value for this player at this step is just one minus the value from this perspective. So it's easy to flip the search process depending on which player you're at. So this is the expansion step. You've taken a non-leaf node and expanded it and evaluated the value. This is essentially a quick guess as to if I were to play to the end, am I going to win or not? You can almost think about the v_theta as a shortcut for searching to the end of the tree for any given simulation.

然后我们进行，这本质上就是评估步骤。我们评估每个棋盘的质量。在最初的 AlphaGo 中，他们做了一件有趣的事：他们将这个价值与真实围棋对弈的价值取平均。他们实际上从这里开始模拟一局完整的棋局。我画一条曲线表示某条路径。他们一直模拟到棋盘完全结束，结果要么是 0 要么是 1，对吧？所以他们把这个价值与这里的价值取平均。公式是 alpha * v_theta(某个节点) + (1 - alpha) * 真实随机模拟的结果。你可能会想，他们如何模拟？如果在这个模拟上再做一次搜索，成本会非常高，就像树中树。所以他们不这样做。相反，他们直接用策略网络自我对弈。他们让策略网络扮演双方，一直下到结束。这有助于将估计值锚定在现实中，因为你可以获得一个关于输赢的单一样本估计。在终局阶段，棋盘几乎已定，这个模拟非常有用，因为根据策略的走法很可能给出合理的猜测，从而避免估计值脱离现实。但事实证明这完全没必要。在 AlphaGo Lee 之后的所有后续论文中，他们去掉了这一步。在我的实现中，我也做了同样的事，这大大加快了速度，因为你不需要在每次模拟中都进行完整对弈。

And then we go, this is essentially the evaluation step. We're evaluating the quality of each of these boards. In original AlphaGo, they actually did something kind of interesting: they took this value and averaged it with the value of a real Go playout. So they actually played a real game from here all the way to the end. So I'm just going to draw this squiggly line to indicate some path. And they kind of play this all the way to resolution of a full board. So this is like a zero or one, right? And so they took this value and just averaged it with this one here. The formula they did was like alpha * v_theta of some node plus (1 - alpha) of a true randomly sampled playout. And you might be wondering, how do they play this out? It would be very costly to do another search on this playout, like a tree within a tree. So they don't do this. Instead, they just take the policy network and play it against itself. So they just take this as both players and play it all the way to the end. This is something that helps ground the estimates here in reality, because you can get a single sample estimate of whether you win or not. You can think about in the end game where the board is almost resolved, this one actually becomes quite useful because the play according to the policy will most likely decide a pretty reasonable guess of the game, so you're not facing a problem where this one becomes untethered from reality. It turns out this is totally unnecessary. So in all subsequent papers after AlphaGo Lee, they just got rid of this. And in my implementation, I also did the same, and it speeds things up a lot because you don't have to roll these games out on every single simulation.

好的。再次强调，为了巩固我自己的理解并重新解释一下：顺便提一下，如果还不清楚，Selection 中的 P 来自网络，这里是策略网络的概率。

Yeah. Okay. So again, just to reinforce my own understanding and to reexplain it for the items: by the way, in case it's not obvious, the P there in the select is the probability coming from the network, in this case the policy network.

没错，这里的策略网络。

Correct, the policy network here.

好的。所以从根本上说，一次模拟可以看作是在搜索过程中多展开一个节点。

Yeah, okay. So fundamentally, a simulation: just think of it as rolling out one more node in the search process.

差不多。当整棵树已经存在时，模拟很容易理解：你只需使用 PUCT 选择标准沿着树往下走，一直继续。

Almost so. A simulation is easy to think about when the whole tree already exists, right? You just walk down the tree using the PUCT selection criteria and you keep going.

是的。

Yeah.

在 AlphaGo 中，数据结构是这样的：我们开始时树只有深度 1，只有根节点的子节点，然后你一边选择动作向下走，一边迭代地构建树。这是核心：因为围棋的组合复杂度极高，你无法预先构建整棵树再搜索。你必须边构建边搜索，对吧？

Now, in AlphaGo, the data structure is such that we begin with a tree that has no, basically only depth one, which is its only children, and you want to iteratively build out the tree as you're also selecting actions down the tree. So that's the core thing here: because Go is such a combinatorially complex game, you cannot afford to build the tree in advance and then search it. You must search while building the tree, right?

好的。

Okay.

让我最后说一下最后一步，即 Backup。一旦你给这些节点打分，你基本上取平均值：这里采取该动作的 Q 值就是所有评估值的平均。你对所有已进行的模拟取运行平均值，并平均子节点的值。这就是所谓的备份步骤，一旦你评估了这个，你可以递归地回溯。

So let me just finish up with actually the last step, which is the backup. Once you've scored these things, you basically take the mean: the Q value assigned to the node here for taking this action is now just the average across your evaluated values. You take a running mean over all the simulations that you've taken and average the values of the children nodes. So that's what is known as the backup step, and once you evaluate this, you can actually recursively go back.

所以，如果你知道某个节点的行动价值，就可以对其父节点取平均值，以此类推。这就是一个四步过程：首先选择当前已知的最佳行动；然后可能遇到一个之前没访问过的节点，需要扩展树；接着通过网络评估胜负概率；最后回溯到根节点，更新最佳行动的价值。随着迭代进行，这种选择标准会让你始终选择当前分支下你认为的最佳行动。因此，最终的访问次数——即你选择这些行动的频率——会反映出通过搜索过程得出的正确策略分布。我们之前存储在节点中的访问次数，实际上就成了最终选择行动时的投票依据。

So if you know the action value of this node, you can then take the average on its parent and so on and so forth. So you have this kind of four-step process where you are choosing the best action that you know of so far. Then you may run into a node where you haven't been to before. So you need to grow the tree a bit. And then you run it through the network to guess whether you're going to win or not. And then you walk all the way back up to the root node to update your values on what the best moves are. As you do this iteratively, this selection criteria will cause you to visit the best action you think at any given branch. So the final visit counts of how often you chose these things will reflect your correct policy distribution as induced through this search process. The visit count that we store in the node earlier actually becomes the sort of vote for which way we should finally select an action here.

嗯。

Yep.

作为理解测试，值得思考一下我们能否让这个过程更简单。比如，我们能不能干脆去掉这个部分，还能让系统正常工作？回想一下，当你扩展并评估某个节点时，你是在检查每个子节点的获胜概率。如果这个节点是 1，而其他是 0，你确实知道哪个行动可能更好。那为什么还需要这个呢？为什么不直接把这个归一化成某种分布，然后称之为策略分布？这没问题，这样做可能也行得通。但实际上，通过一次前向传播得到一个不错的猜测，正是剪枝的方式。这里存在一种对偶性。如果策略推荐了一个与价值不一致的行动，那就有问题了。比如策略说这个概率很高，但价值却说它很低，那么策略头和价值头之间就存在根本性的矛盾。所以它们是关联的。如果你能找到另一种方法仅从价值评估中恢复出策略，你或许可以去掉这个部分。

As a sort of test of understanding, it's worth thinking a little bit about whether we could make this even simpler. Could we actually maybe even get rid of this one and still make the thing work? Recall that when you do an expansion and then an evaluation at, let's say, this node, you are checking the win probability of each of the child nodes. So if this one is one and these are zero, you do kind of know something about which action might be better to take. So why would you still need this? Why not just normalize this one into some distribution and call that your policy distribution? This is fine. You can do this and it probably does work. But in practice, having a single forward pass that gives you a pretty good guess is how the breadth is pruned out. There is a sort of duality here. It would be weird if, let's say, the policy recommended an action that disagreed with the value. If the policy said this was very high probability but this one said it was low value, then there's actually something fundamentally wrong between your policy head and your value head. So they are linked. You probably could get rid of this if you came up with a different way to recover this from just the value evaluations.

对。但为了确保我理解正确，你不那么做的原因是为了避免做 360 次独立的前向传播来获取所有节点的价值，然后取 arg max。相反，你只需一次前向传播就能得到所有概率。

Right. But just to make sure I understand the reason you don't do that is so that you don't have to do 360 independent forward passes to, like, here's the value of everything. Let's arg max over that. Instead, you can just do one forward pass and get the probabilities of all of them.

通常你可以批量处理这些，效率还行。所以实际计算负担可能不大。但没错，你需要将 361 个棋盘传入一个 mini-batch 更新来评估所有价值，然后归一化。实际上，我们仍然保留这个部分还有一个更重要的原因：蒙特卡洛树搜索如何自我反馈并递归改进自身的预测和搜索能力。正是这一点，使得将其作为显式建模的实体，而不是对价值的隐式归一化，是一个好主意。

You can usually batch these somewhat efficiently. So it probably is not a huge computational burden in practice. But yes, you would have to pass 361 boards into a single mini-batch update to evaluate all the values here, then normalize them. Now there's actually a more important reason why we still do this, which is how Monte Carlo tree search is used to feed back on itself and recursively improve its own predictions and search capabilities. That's where having this as an explicit entity you're modeling, rather than an implicit normalization over your value, is a good idea.

有道理。好的。

Makes sense. Okay.

我们讨论了模拟过程。基本上，随着模拟次数的增加，你会得到一个树——我画的是一个非常低维的版本。当然，在真实游戏中，维度要高得多。但最终你会得到一个树结构，其中很多叶子节点因为价值过低而终止且不再被访问。然而，沿着某条路径，随着 n 增加，会有一组行动具有非常高的访问次数，并趋向于那一组决策。这就是蒙特卡洛树搜索的树结构图景。你应该将其与井字棋中的穷举树对比——井字棋有 9 个行动，然后 8 个，再 7 个，以此类推，是一个 9 的阶乘大小的树。围棋中的蒙特卡洛树搜索非常稀疏，它只考虑你扩展了子节点的路径。现在，我们有了应用价值函数和策略函数的搜索算法，可以讨论蒙特卡洛树搜索算法如何作为这些函数的改进算子。

So we talked about the simulations. Basically, what you end up with as you roll out the number of simulations is a tree that looks like — I'm drawing a very low-dimensional version of this. Of course, in the real game, it's much more high-dimensional. But you'll end up with a tree structure that has a lot of leaves that terminate and are not visited again because their value is deemed too low. But then, along one path, there will be a set of actions with very high visit counts that gravitate towards that one set of decisions as you increase n. So this is the mental picture of what the tree in Monte Carlo tree search looks like. You should contrast this with an exhaustive tree like in tic-tac-toe, where there are nine actions, then eight, then seven, and so on — a nine factorial sized tree. The Monte Carlo tree search in Go is very sparse. It only considers the paths that you've expanded child nodes on. Now that we have the search algorithm that applies the value function as well as the policy function, we can talk about how the Monte Carlo tree search algorithm can actually act as an improvement operator on top of these.

20 年前，Jane Street 的数据中心还只占办公室的一个角落。技术团队联合负责人 Ron Minsky 告诉我这一切是如何开始的。我们有一个计算集群叫“蜂巢”。我记得蜂巢的第一个任务就是六台戴尔盒子堆在一排的尽头。交易系统也放在那里，因为我们想确保能关掉那该死的东西。有过起起落落。比如有一次，清洁工在吸尘时拔掉了交易系统的电源。所以最终，把所有东西放在数据中心确实更好。Jane Street 的数据中心从那六台戴尔盒子发展了很多。我在德克萨斯州与 Ron 和负责物理工程团队的 Dan Pontto Cororvo 一起参观了其中一个。

20 years ago, Jane Street's data center fit in the corner of an office. Ron Minsky, who co-leads the tech group there, told me about how it all got started. One of our compute clusters we called the Hive. And I remember the first mission of the Hive was literally like six Dell boxes stacked on top of each other at the end of the row. And the trading systems themselves we also had there because we actually wanted the ability to make sure we could turn the damn thing off. I mean, there were ups and downs. Like literally at some point, one of the people who was cleaning the office unplugged one of the trading systems in the middle of the day as they were vacuuming. So in the end, it is in fact better to have it all in a data center. Jane Street's data centers have come a long way since those six Dell boxes. I got to tour one of them in Texas with Ron and Dan Pontto Cororvo, who leads Jane Street's physical engineering team.

这些机柜，GB300 机柜，峰值功耗约 140 千瓦。相比之下，传统风冷机柜只有 10 到 40 千瓦。所以多得多。

These cabinets, these GB300 cabinets, consume at peak about 140 kW. Compare that to traditional air-cooled, you're talking about 10 to 40 kW. So a lot more.

我们深入了解了运行这样一个数据中心的细节。一些我从未考虑过的事情。

We got deep into the details of running one of these data centers. Things that I had never considered before.

里面充满了一种液体，是蒸馏水或去离子水与丙二醇的混合物。丙二醇占 25%。这是为了抑制细菌或藻类生长。

It's filled with a liquid, a mix of distilled or deionized water and propylene glycol. 25% propylene glycol. That's to inhibit any bacteria or algae growth.

我不喜欢需要担心服务器里长细菌的世界。

I don't love the world where we have to worry about bacteria growing in our servers.

我看到了比以往更多的数据中心内部运作。Jane Street 愿意掀开地板、取出机架，带我到后面所有冷却器所在的地方。你可以在 janestreet.com/wcash 查看我们发布的完整参观视频。

I got to see way more of what actually happens in a data center than I've ever seen before. Jane Street was willing to literally pull up the floorboards and take out the racks and take me to the back where all the chillers are. You can check all of this out at janestreet.com/wcash where we posted the full tour.

现在我们讨论强化学习部分，即这个东西如何通过自我对弈变得更强。假设我们玩一个游戏，AI 走一步，计算搜索，这是访问次数分布。假设这是你在该节点的初始策略推荐，经过 MCTS 后，它对其中一个动作变得更自信。所以分布基于搜索变得更尖峰。当然你可以调整搜索过程使其更分散，但可能不是好主意。MCTS 应该对特定动作比其他动作更自信，但最初可能对其他动作赋予很大权重，随着模拟次数增加，它会收敛到非常尖峰的分布。所以这是你的新策略，我们称之为 pi，包裹在给定状态 s 的 MCTS 算子中。应用 MCTS 后，你的策略推荐分布看起来像这样，比之前更尖峰。然后你取 argmax 或从中采样，执行你的动作。然后你丢弃树，在下一步重新开始。所以你计算一个新分布。最初你的猜测像这样，然后通过 MCTS 精炼。

So we now talk about the RL part of how this thing gets stronger by playing itself. Let's say we play a game where the AI makes a move, computes the search, and this is the visit count distribution. Let's say this is your policy initial policy recommendation at this node, and then after MCTS it gets more confident about one of these actions. So the distribution looks a bit more peaky like this based on the search. Of course you can tune the search process so that it ends up more diffuse, but that's probably not a good idea. MCTS should get more confident about specific actions than others, but it might place a lot of weight on other actions initially, and as you increase the number of simulations it should converge to a very peaky distribution. So this is your new policy, let's call it pi, wrapped in an MCTS operator of a given state s. After applying MCTS, your policy recommended distribution looks like this, a bit more peaky than the previous one. Then you take the argmax or sample from this, and you make your move. Then you throw away the tree and begin a new one on the next move. So you compute a new distribution. Initially your guess looks like this, and you refine it through MCTS.

棋盘上应该多一个 X，对吧？

There should be one more X on the board, right?

抱歉，没错。是的。变成像这样的东西。所以每一步你都有来自策略网络的初始猜测，然后结合策略网络和价值网络的搜索过程得出一个更自信的动作，你执行它。以此类推，游戏结束，一人赢一人输。AlphaGo 自我训练的妙处在于，它可以利用这个最终搜索过程的结果告诉策略网络：与其让 MCTS 做所有繁重工作到达这里，为什么不从一开始就预测这个？为什么不放弃这个猜测，直接预测这个？如果你的策略网络一开始就有这个猜测，那么 MCTS 需要做的工作就少得多。所以如果我们画一个测试时 Scaling 图，假设这是模拟次数。在零次模拟时，你的隐含胜率在这里。没有任何模拟，如果你只取这个原始动作，这就是你的胜率。随着模拟次数增加，你的胜率可能像这样。当你搜索一千步模拟时，你得到这里的策略，这很好。但如果你将这个 MCTS 策略网络蒸馏回你的“即兴”策略网络，那么你实际上可以从这里开始。如果通过蒸馏，这是零，那么再花一千步模拟，你实际上会到达这里。这几乎就像如果你能将前一千步模拟摊销到策略网络中而不是搜索过程中，那么你可以从一个更好的起点开始，并为投入的模拟次数获得更好的结果。

I'm sorry. That's correct. Yes. To something that looks like right. So on every move you have your initial guess from your policy network, and then the search process that combines your policy network and your value network arrives at a more confident action that you take. And so forth, then the game ends and one person wins and one loses. The beauty of how AlphaGo trains itself is that it can take this final search process, the outcome of the search process, and tell the policy network: instead of having MCTS do all this legwork to arrive here, why don't you just predict that from the get-go? Why don't you not use this guess and just predict this to begin with? If you have this guess to begin with in your policy network, then MCTS has to do a lot less work to get things to work. So if we draw a test time scaling plot, let's say this is number of simulations. At zero simulations, your implicit win rate is here. Without any simulation, if you just take this raw action, this is your win rate. As we increase the number of simulations, maybe you have a win rate that looks like this. When you search for a thousand simulation steps, that gets you to a policy here, which is great. But if you were to distill this MCTS policy network back into your shoot-from-the-hip policy network, then you could actually start here. If this was zero by distillation, then if you spend another thousand simulation steps, you actually get to here. It's almost like if you could amortize the first thousand steps into the policy network instead of the search process, then you could begin at a much better starting point and get a much better result for the number of simulations you put in.

测试时 Scaling 的 S 形特性：随着模拟次数增加，胜率的增长变小。即使对于蒸馏后的网络也是如此吗？也就是说，是否有一些收益，比如我们从蒸馏后的网络开始，再次获得早期收益，还是这只是 MCTS 固有的特性？

The sigmoid type nature of test time scaling: as the number of simulations increases, the increase in win rate is smaller. Is that true even for the distilled network? That is to say, is there some gain of like, okay, we start from the distilled, we get these early gains again, or is that just inherent to the nature of MCTS?

说实话，我其实不知道 MCTS 模拟的测试时 Scaling 行为。我相信它可能对实践中这个网络的强度非常敏感。我只是画了一个单调递增到 1 的函数。所以不要太关注曲线的形状。只需知道它相对于模拟次数是单调的。所以 MCTS 的想法非常巧妙：我们通过应用搜索得到了更好的结果。然后在下一次迭代更新这个网络时，我们训练它来近似一千步搜索的结果。所以不是从这里开始，而是让神经网络从这里开始，然后当我们再应用一千步搜索时，对局变得更强，并且你可以继续下去。AlphaGo 的训练算法基本上是拿那些你在每一步都应用了搜索的游戏（无论输赢），这很重要，然后训练模型来模仿搜索过程。

To be honest, I actually don't know the test time scaling behavior of MCTS simulations. I believe it might actually be quite sensitive to how strong this one is in practice. I'm just drawing a monotonically increasing function that gets to one. So don't pay too much attention to the shape of the curve. Just know that it's monotonic with respect to simulations. So the idea of MCTS is very brilliant: we got something better by applying search. And we're going to now on our next iteration of updating this network just train this to approximate the outcome of a thousand steps of search. So instead of starting here, we get to have a neural network start here, and then the play gets stronger once we then apply another thousand steps on top of it, and you can keep going. The training algorithm for AlphaGo is to basically take the games where you've applied the search on every move that the policy encountered, whether you won or lost, and that's quite important, and you're just going to train the model to imitate the search process.

所以实际上有一个与机器人学的类比，即 DAGGER 算法。首先我画一个状态示意图：S0、S1、S2、S3。假设我们在一个 MDP 中采取了一系列动作得到一条轨迹。这些动作可能不是最优的，也许我们在游戏结束时输了。有一类算法基本上取轨迹并重新标记动作以得到更好的轨迹。也许这里更好的动作是 z'，这里更好的动作是 a1'，还有 a2'、a3'。MCTS 所做的本质上是说：你玩了这场最终输掉的游戏，但在每一步我都会给你一个严格更好的动作，你应该采用它。它不保证你会赢，但保证如果你把这些元组作为训练数据来重新训练你的策略网络，让它预测这些动作而不是原来的动作，你会做得更好。这与机器人学和模仿学习中的 DAGGER 非常相关，在那里你想收集干预，即使你处于不好的状态，例如一辆偏离道路的自动驾驶汽车，仍然有一个有效的动作可以纠正并带你回来。

So there's an analogy to robotics actually, which is the DAGGER algorithm. First I'm going to draw a schematic of states: S0, S1, S2, S3. Let's say we took a series of actions in an MDP to get a trajectory. These actions may be sub-optimal, maybe we lost at the end of this game. There is a family of algorithms that basically take trajectories and relabel the actions to better trajectories. Maybe a better action here would have been to take a z prime, a better action here would have been to take a1 prime, and yet another one like a2 prime, a3 prime. What MCTS is doing is basically saying: you play this game where you eventually lost, but on every single action I'm going to give you a strictly better action that you should take instead. It does not guarantee that you are going to win, but it does guarantee that if you take these tuples as training data so that you retrain your policy network to predict these ones instead of these ones, you're going to do better. This is very related to DAGGER in robotics and imitation learning, where you want to collect an intervention here, and even if you're in a not great state, for example a self-driving car that veers off the side of the road, there is still a valid action that corrects you and brings you back.

一个学究式的问题：是否有保证 MCTS 一定比策略更好？例如，你可以想象在训练早期，因为 MCTS 由价值网络提供信息。

Pedantic question: is there a guarantee that MCTS must be better than the policy? For example, you could imagine early on in training because MCTS is informed by the value network.

在训练早期，当价值网络尚未在完整对局上得到良好训练时，MCTS 比随机初始化的策略更差。那么，MCTS 优于策略只是一个启发式方法，还是有什么保证？

Early on in training, when the value network hasn't been well trained on finished games, MCTS is worse than a randomly initialized policy. So is it just a heuristic that MCTS is better than the policy, or is there some guarantee?

没错，在实践中这是一种启发式方法，而且确实有效。但让我举例说明 MCTS 可能给出比策略网络更差的分布。如果你的自对弈算法训练到了不错的地步，但由于没有在多样化数据上训练而崩溃，这种情况经常发生。假设我们有一个棋面，策略推荐非常好，即给定状态的 π 很棒。但不知何故，因为我们在很多对局中让机器人直接认输，而不是一直下到终局，它们就忘记了如何评估那些后期局面。在我们展示的角落走法案例中，重放缓冲区里 100% 的训练数据都丢失了如何评估那些状态下的价值函数的例子。所以你可能会遇到终局价值非常糟糕的情况。如果叶节点的终局价值不好，这会一直向上传播，导致你的 PUCT 选择标准和备份出现偏差，最终你会访问一个与策略最初推荐截然不同的分布。

Right, in practice it is a heuristic, and it does work in practice. But let me illustrate an example where MCTS can give you a worse distribution than your policy network. This can often happen if your self-play algorithm has trained to a good point but then collapses because it's not trained on diverse data. Let's say we have a board state where the policy recommendations are very good, so pi of a given state is great. But somehow, because we're playing a lot of games where the bots just resign instead of playing all the way to the terminal resolution, they forget how to evaluate those late-stage positions. In the case we showed with the corner play, maybe 100% of our training data in the replay buffer has lost examples of how to evaluate the value function at those states. So you might end up in a scenario where your terminal value is very bad. If the terminal values of the leaves are not good, this will propagate all the way up, causing your PUCT selection criteria and backups to be off, and you'll end up visiting a very different distribution than what your policy initially recommended.

另外，如果你的模拟次数很少，可能会出现方差问题，即探索不足。只有当 n 趋于无穷时才能保证收敛。因此，搜索过程中的方差以及评估的不准确性肯定会破坏策略验证的质量，这就是为什么它不能保证改进。我怀疑 AlphaGo Lee 在训练算法中使用了终局推演，就是为了将这个过程建立在真实对局的基础上。

Also, if your number of simulations is low, you might have a variance issue where you just don't explore enough. It's only guaranteed to converge when you take n to infinity. So variance in the search process as well as inaccuracies in your evaluation can definitely screw with the quality of your policy validation, and that's why it's not a guarantee to improve. That is why I suspect AlphaGo Lee had the playouts to the end in their training algorithm, so they could ground this thing in real plays.

是的。在实践中，你还可以对 10% 的对局禁止机器人认输，强制下到终局。这样你的重放缓冲区就能获得一些训练数据，真正解决那些普通人类玩家不会下到的后期推演。

Yeah. In practice, what you could also do is for 10% of the games, prevent the bots from resigning and just resolve it to the end. So you get some training data in your replay buffer to really resolve those late-stage playouts that normal human players wouldn't play to.

嗯。嗯。

Yeah. Yeah.

这就是为什么如果你假设价值函数正确，MCTS 会给你更好的策略。这是一条关键的假设链。假设这是准确的，你的搜索过程应该给出比初始猜测更好的推荐。

So this is why MCTS, if you assume the value functions are correct, gives you a better policy. It's a critical chain of assumptions. Assuming this is accurate, your search process should give you a better recommendation than your initial guess.

好的。所以如果你有一个冷启动的策略，比如 AlphaZero 那种，最初几个 epoch 实际上策略是没什么用的。你真正做的是下完整对局。一旦我们有了前面走法的完整对局，我们就有了谁赢的标签。AlphaZero 的损失有两个部分：策略相对于 MCTS 有多好，以及价值预测相对于从这一步开始谁最终获胜有多好。这可以看作应用于每一个动作或每一步。对吗？

Okay. So if you have a cold-started policy, an AlphaZero type thing, really what's happening for the first few epochs is the policy is kind of useless. What you're really doing is playing full games. Once we have played full games for the preceding moves, we'll have labeled who won. The loss for AlphaZero has two components: how good is the policy relative to MCTS, and how good is the value prediction relative to who actually won the game from this move. This can be thought of as being applied to every single action or every single move. Correct?

正确。AlphaZero 训练初期真正发生的事情就是试图让价值函数能够预测如果你处于某个状态且是某个玩家，谁会赢。从功能上讲，这就是全部。之后，一旦价值函数训练好了，策略也会随之改进。

Correct. And really what's happening at the beginning of AlphaZero training is just trying to get the value function to actually predict who will win the game if you find yourself in this state and you're this player. Functionally, that's all that's happening. Later on, once that's well trained, the policy is also improving.

正确。

Correct.

好的。

Okay.

我发现一个很有用的技巧——这不是经过同行评审的说法，所以仅供参考——那就是：在投入大量算力进行 MCTS 之前，先确保价值函数是好的。在垃圾价值预测上进行搜索没有意义。所以你要从一个好的起点开始。AlphaGo 做得很好，它直接拿人类对局进行训练，效果很好。完全可行。你也可以拿一个开源围棋程序让它自对弈生成数据，这也行。所以如果你有一些包含高质量对局的离线数据集，你可以很容易地学到不错的后期价值函数，这就是启动搜索过程所需要的。

One trick I did find to be pretty useful, and this is not a peer-reviewed claim so take it with a grain of salt, is the following: you want to first make sure the value function is good before you invest a lot of cycles doing MCTS. It doesn't really make sense to do search on garbage value predictions. So you want to start at a good place where this works. AlphaGo does a very good thing where it just takes human games and trains on it, and it just works. Totally works. You could also take an open-source Go program and have it play against itself to generate data; that also works. So if you have some offline dataset that has realistic good play, you can easily learn the late-stage value functions pretty well, and that's what you need to start the search process.

抱歉，你能再读一遍这句话吗？

Sorry, can you just read this sentence one more time?

当然。评估后期围棋棋局相当容易，因为几乎所有棋子都在棋盘上了。这几乎是一个可判定问题，因为树深度的不确定性越来越低。所以大多数由合理玩家下到终局的对局，都是训练树终局部分良好价值函数的好数据。

Sure. So it's quite easy to evaluate a late-stage Go game, when almost all the pieces are on the board. It's almost like a decidable problem, because there's lower and lower uncertainty as to the depth of the tree. So most games played to the end by reasonable people will be good training data to train a good value function at terminal parts of the tree.

明白了。好的。

Got it. Okay.

然后，随着你下更多对局，搜索会将好的价值反向传播到树的中间节点。随着数据量增加，你的价值头会很好地直觉判断什么是健康的棋面，什么是不健康的。中盘判断这些比开局或终局微妙得多。所以最难评分的是中盘，而不是开局或终局，因为开局显然就是 0.5，终局谁赢很明显。所以价值函数要学习的难点是中盘谁占优。

Then, as you play more games, the search will back up good values into the intermediate nodes of the tree. As you increase the amount of data, your value head gets a good intuition of what is a healthy board state versus a not healthy board state. Those are much more subtle to judge in the midgame than the beginning or the end. So the most difficult part to score is not the beginning or the end, because the beginning is just obviously 0.5, and at the end it's pretty obvious who's winning. So the hard part you want to learn in the value function is who is winning in the middle.

所以这实际上与 TD 学习非常相似。

So this is actually very analogous to TD learning.

是的。这与 TD 学习有一个美妙的联系，我们可以稍后讨论，而不是与蒙特卡洛搜索对比。所以首先你要获得好的价值函数，专家数据可以给你一个快速的捷径。

Yes. And there's a beautiful connection to TD learning that we can talk about in a bit, as opposed to contrasting with Monte Carlo search. So you first want to get good value functions, and expert data can give you a quick shortcut.

我建议从业者先这样做，初始化到一个好的起点。如果你想做 AlphaZero 或 KataGo 式的 tabula rasa 学习，可以尝试在小棋盘上玩随机对局。只需用一个随机智能体，如果玩大约 5 万局，你实际上也能学到相当好的价值函数。因为在 9×9 棋盘上，通过随机对局就能看到足够多的常见模式。然后，如果你训练一个能同时处理 9×9 和 19×19 数据的模型——KataGo 提出了这样一种架构——那么从 9×9 上训练的价值头到 19×19 会有很好的迁移学习效果。

I recommend for practitioners, just do that first to initialize to a good starting point. Then if you want to do the AlphaZero thing or KataGo-style tabula rasa learning, what you can try is on a small board, play random games. Just take a random agent, and if you play like 50,000 games, you'll actually learn a pretty good value function as well. Because on a 9x9 board, you can see enough of the common patterns with random play. Then if you train a model that can train on both 9x9 and 19x19 data, and KataGo proposed one of these architectures, there's some pretty good transfer learning from the value head evaluated at 9x9 to the 19x19.

对，因为不像其他游戏，增大棋盘并不会引入新的棋子类型。

Right, because unlike other games, there's not a new kind of piece introduced when you increase the size.

如果我们推到极限，考虑一个非常小的 4×4 围棋棋盘，玩 5 万局后，你会得到很多看起来像人类对局的终局状态。那基本上就是井字棋了。所以如果你稍微扩大一点到 5×5 或 9×9，可以想象纯随机对局实际上会产生相当合理的棋面。然后你可以很容易地给这些棋面打分。这就为你提供了自举能力，从而能够通过搜索改进策略。

If we take it to its limit and consider a very tiny 4x4 Go board, if you play 50,000 games, you're going to have a lot of end states that look like human play. It's just like tic-tac-toe at that point. So if you broaden this a little bit to 5x5 or 9x9, it's not unrealistic to imagine that purely random play will actually generate pretty reasonable looking boards. And then you can score those pretty easily. That gives you the bootstrapping to be able to then improve your policy with search.

但非常关键的是，MCTS 需要有准确的价值估计，并且你需要为价值提供基础。如果没有价值的基础函数，MCTS 最终会崩溃。我很好奇，通过在同一网络上训练价值和策略能节省多少算力，因为它们共享相同的表示。学习效率能提高多少？如果它们基本上在做类似的预测，或者应该彼此一致，那会很有趣。

But it's very critical that MCTS has accurate value estimates, and you need to ground the value. Ultimately MCTS will fall apart if you don't have a grounding function for the value. I'd be curious how much compute you save by training the value and policy on the same network, because they share the same representations. How much more efficient learning is? That would be interesting if they're basically making similar predictions or should be in line with each other.

所以我也好奇，实际上，通过保持同一个网络，你只是减少了所需的算力。

And so I'd be curious if actually, yeah, you're just having the amount of compute you had to do by keeping the same network.

对。最初的 AlphaGo 论文（AlphaGo Lee）使用了两个独立的网络，而在之后的所有论文中，它们被合并成了两个头，这大概节省了算力。但要以非常严谨的科学方式回答这个问题，其实是个简单的问题，但如果你真想追根究底，实践中需要相当多的工作。

Right. AlphaGo Lee, the original AlphaGo paper, had two separate networks, and then in all subsequent papers, they merged them into two heads, and presumably this saves compute. But answering that question in a very rigorous scientific way is actually a simple question, but in practice it takes quite a bit of work if you really want to chase that question down to its limit.

但直觉上，是的，它们共享很多表示。所以正如我们提到的，在进行评估时，策略网络和价值网络之间应该有一种一致性；它们应该大致一致。

But intuitively, yes, they share a lot of representations. So as we mentioned, there is a sort of consistency between your policy network and your value network when doing evaluation; they should kind of agree.

告诉我这样想是不是不对。我觉得当我了解 LLM 的工作原理以及 RLVR 作为算法有多么简单时，我对它能做的事情感到震惊——它仅仅通过一个“是/否”信号就能学会构建非常复杂的代码仓库等等。而在这里，我觉得如果你更深入地理解 AlphaGo，比如只是预测 MCTS，实际上你越了解它，它反而显得不那么令人印象深刻，因为你就像是在说，哦，你加入了大量偏差，告诉它应该如何逐步调整探索，你为它构建了非常明确的树搜索。所以我不知道你是否也有这种直觉——实际上你越了解它，2017 年的成就看起来就越不惊人。

Tell me if this is the wrong way to think about it. I feel like when I learn how an LLM works and how simple RLVR is, at least as an algorithm, how simple it is, I'm sort of stunned by the kinds of things it can do. That it can learn how to build very complicated code repositories and whatever simply from getting a yes/no. And here I feel like if you understand it more deeply, like just predicting MCTS, and it actually seems AlphaGo seems less impressive in retrospect the more you understand it, because you're like, oh, you're putting in a lot of bias by just saying how much you're telling it how we should titrate exploration as things go on, you're building this very explicit tree search for it. So I don't know if you share that intuition, where actually the more you understand it, the less impressive the accomplishment in 2017 seems.

我个人不同意。我认为它们因不同原因而意义深远，而且我对 LM RL 理解不够，无法在你的播客中评论。但我认为 AlphaGo……那么，为什么它是一个深远的成就？也许值得退一步看看。它与现代 RL 不同，我们可以谈谈其中的一些算法选择。但我认为最深远的一点是，一个 10 层神经网络的前向传播——基本上就是 10 步推理。

I personally disagree. I think they're profound for different reasons, and I don't understand the LM RL enough to comment on your podcast about it. But I think AlphaGo... So yeah, why is it a profound accomplishment? I think maybe it's worth stepping back a little bit. It is different from modern RL, and we can talk a little bit about some of the algorithmic choices there. But I think the most profound thing here is that a 10-layer neural network pass—so basically 10 steps of reasoning.

嗯。

Yeah.

当然，推理不只是一条思路；它可以是分布式表示，同时进行大量思考。但根据构造，一个 10 层神经网络只能进行 10 个顺序思考步骤，对吧？10 步神经网络并行分布式表示思考，能够以非常高的保真度分摊并近似一个几乎难以处理的搜索问题。

And of course the reasoning is not just one trail of thought; it could be like the distributed representations and a lot of thoughts going on at the same time. But by construction, let's say a 10-layer neural network can only do 10 sequential steps of thinking, right? 10 steps of neural network parallelized distributed representation thinking is able to amortize and approximate to a very high fidelity a nearly intractable search problem.

嗯。

Yeah.

所以这是一个突破，我认为大多数人至今仍不理解，没有完全领会这个成就有多么深远。这也支撑了 AlphaFold 等成果，对吧？你有一个非常困难的物理模拟过程，需要展开大量微观模拟，然而一个相当小的神经网络的 10 步前向传播，却能以某种方式捕捉到一个感觉像是 MPL 类的问题，并将其归约为一个单一问题。这让我不禁怀疑，我们对 P 是否等于 NP 这类问题的理解，或者这些非常基本的计算困难性问题，是否是不完整的。显然，这不是 P=NP 的证明或任何东西，但其中有些东西非常令人不安——一个看似非常困难的问题，竟然能被一个非常简单的宏观模拟所攻克。

So this was a breakthrough that I think most people don't even understand today, like fully comprehend how profound that accomplishment is. And this is what also girds AlphaFold, for example, right? Where you have a very difficult physical simulation process that you would need to roll out so many microscale simulations, and yet 10 steps of a somewhat small neural network can somehow capture what feels like an MPL-class problem into a single problem. And so it actually makes me wonder if our understanding of problems like P equals NP or these very fundamental computational hardness problems are incomplete. Like, obviously this is not a proof of P=NP or anything, but there's something to it that is very disturbing, where what felt like a very hard problem can fall to a very simple macroscopic simulation.

这是一个非常有趣的见解：很多被证明是 NP 困难的问题——比如我不知道围棋是否被证明是 NP 困难，但蛋白质折叠等等——神经网络之所以能解决它们，是因为它们在最坏情况下是 NP 困难的，但我们并不处理最坏情况。我们通常不关心最坏情况。这些问题通常有很多结构。

That is a very interesting insight that a lot of problems which are proven to be NP-hard, like I don't know if Go is proven to be NP-hard but okay, protein folding etc., have been like neural networks can solve them because they're NP-hard in the worst case, but we're not dealing with the worst case. We're usually not concerned about the worst case. These problems have a lot of structure to them usually.

是的，我认为我们应该问自己的问题是，我们一直在用最坏情况复杂度的方式来构思 NP 困难问题的解决方案。

Yeah, I think the kind of question we should be asking ourselves is like we've been formulating solutions to NP-hard problems as in kind of worst-case complexity.

而且我不会说这解决了围棋，对吧？它并没有给出最优解的精确答案。

And I wouldn't say this solves Go, right? It doesn't give us an exact solution of the optimum.

但在实践中，它极其有用。

But in practice, it is extremely useful.

同样的情况在 AlphaTensor、AlphaFold 中也有体现：确实存在一个在最坏情况下看似棘手的问题，但我们却能取得几乎任意程度的进展。

And the same thing has been shown in AlphaTensor, AlphaFold, where yes, there is a very hard problem that in the worst case seems intractable, and yet we're able to make almost arbitrary amounts of progress.

那么，在极限情况下，这会是什么样子呢？如果你想模拟非常复杂的事物，比如天气，或者预测未来——比如我们是否生活在模拟中——构建一个非常复杂的模拟所需的计算资源可能比你想象的要小得多，这基于我们将大量计算摊销到单个网络前向传播中的能力。

So here's a sort of, in the limit, what might this look like? Well, if you want to simulate something very complex like weather or predict the future, like do we live in a simulation or not, the computing resources you need to build a very complex simulation might be much smaller than you think, based on our ability to amortize a lot of that computation into the forward pass of a single network.

有意思。所以对我来说，AlphaGo 是第一篇真正展示了这种将模拟压缩到少量计算中的深刻水平的论文。

Interesting. So to me, AlphaGo was the first paper that really showed this profound level of simulation being compressed into a small amount of...

我觉得自己完全没有资格在数学的计算复杂性上评论这一点，但我想知道混沌在这里是否扮演了重要角色。天气的问题在哪里，为什么预测一天后的天气需要 10 倍的资源，而且每多预测一天资源就持续增加，是因为它是一个混沌系统，所以微小的扰动会随着时间彻底改变最终估计。我想这很有趣，而且你也会预期围棋和蛋白质折叠也是如此。

I feel totally not at all qualified on the computational complexity of the math to comment on this, but I wonder if there's an important role of chaos here. Where if, under what is the problem with weather and why does it take 10x the amount of resources to predict weather a day out, and continually so for every more day out, is because it's a chaotic system, and so small perturbations can totally change the final estimate as time goes on. And I guess it's interesting, well I guess you would expect that for Go and protein folding as well.

那么这里有一个与天气的类比，可能适用于围棋。问题是：这是我们当前的棋盘状态，根据我们对两位棋手的了解，未来的棋盘状态是什么？

So here's an analogy to weather that might be relevant in Go. The problem of: here's our current board state, given what we know about both players, what is the board state in the future?

嗯，根据我们对两位棋手的了解，未来的棋盘状态是什么？

Um, given what we know about both players, what is the board state in the future?

对。

Yeah.

未来的确切棋盘状态是什么？对吧？这对初始条件极其敏感，比如在这里放一颗棋子就能打乱整个预测。

What is the exact board state in the future? Right? This is extremely sensitive to initial conditions, like a single stone placed here can kind of disrupt the entire prediction.

对。没错。所以这很难。这直观上就是混沌问题。

Yeah. Right. So this is hard. This is kind of intuitively the chaotic problem.

然而不知何故，这很难。不知何故，我们能预测谁会赢。

And yet somehow, this is hard. Somehow we can predict who's going to win.

对。

Yeah.

而这涵盖了很多可能性。

And this captures a lot of possibilities here.

所以我们真正关心的是这个更宏观的量，即平均值或期望，或者某种跨越许多可能未来的全局宏观结构。

And so there's this more macroscopic quantity that we really care about, which is the average or expectation or some sort of global macro structure over a lot of possible futures.

有趣的思考方式。

Interesting way to think about it.

所以在天气中，也可能是同样的情况，对吧？我们并不确切关心某个经纬度上方 6000 英尺处的风速。我们关心的是飓风在哪里之类的事情。而且我要说，在混沌中，有一个经典的洛伦兹吸引子，看起来像这样。是的，如果你从洛伦兹吸引子上的任何一点出发，你不知道最终会到哪里，但你知道这个东西看起来像这样。

And so in weather, it could be the same thing, right? We don't exactly care what the velocity of wind 6,000 feet above a specific latitude/longitude is. We kind of care where's the hurricane or things like that. And I would say in chaos, there's a classic Lorenz attractor which kind of looks like this. Yes, if you start anywhere on the Lorenz attractor, you don't know where you're going to end up, but you do know that the thing looks like this.

对。对。

Yeah. Yeah.

没错。所以有时我们不一定关心微观尺度的事物，我们实际上关心的是宏观结构，这有一种美。

Right. And so there's this kind of beauty of sometimes we don't necessarily care about the microscale things. We actually care about the macroscopic structure.

有意思。

Interesting.

而这些事物是可以预测的。

And these things can be predictable.

与之对比的是哈希函数之类的东西，它也极其依赖初始条件，但没有宏观结构——至少希望如此，如果设计得当的话。

And contrast that to something like a hash function, which is also incredibly dependent on initial conditions, but doesn't have a macro structure, at least hopefully if the work.

是的。

Yes.

所以那里没有等价的价值函数，或者像天气大致会怎样这样有趣的问题。它实际上只是关于 100 步后的棋盘会是什么样子。

And so there's no equivalent of a value function or like broadly how's the weather going to be that is interesting there. It's really just about what is the board going to look like 100 moves from now.

完全正确。

Exactly.

呃，是的，直觉上这似乎正确。再说一次，这超出了我的专业领域，但我发现有趣的是，密码学一直无法证明你不能提出快速近似。如果他们能做到，那么你就能证明 P 不等于 NP。

Uh, yes, intuitively that seems correct. And again, this is out of my area of expertise, but I find it interesting that cryptography has not been able to prove that you cannot come up with fast approximations. If they were able to do that, then you could prove P is not equal to NP.

是的，是的。事实上，我们知道许多密码协议中存在结构，显然像 RSA 密码学就有结构，而量子计算机正是利用这种结构来破解它们。

Yeah, yeah. In fact, we know that there's structure in many cryptographic protocols, obviously like RSA cryptography, there is structure, and that structure is what quantum computers exploit to break them.

我明白了。

I see.

Reiner 有一篇非常有趣的博客文章，我们在节目中讨论过，他谈到如果你从高层次看密码协议的样子和神经网络的样子，它们极其相似，都有将信息混合在一起的连续层。这是因为算法中存在这种趋同演化：在密码学中，你希望最终状态对初始条件极其敏感，这样如果你改变任何东西，它就会看起来混乱；而在神经网络中，你同样希望一切都依赖于所有信息，因为你想要处理所有信息并考虑它如何与自身关联。

Reiner has a very interesting blog post which we talked about in the episode, where he talks about how if you look at a high level what cryptographic protocols look like and what neural networks look like, it's extremely similar, where you have sequential layers of jumbling information together. And it's because there's been this convergent evolution in the algorithms, where in cryptography you want the final state to be incredibly sensitive to initial conditions so that it can come out sort of looking jumbled based on if you change anything, and then neural networks you similarly want everything to be dependent on all the information because you want to process all the information and consider how it relates to itself.

是的，神经网络的最大能力出现在混沌边缘。我想 Joshua Schlick 的团队有一些关于这个的研究论文。

Yeah, you have the maximum power of a neural network at the edge of chaos. I think there's some research papers from Joshua Schlick's team on this.

对。

Yeah.

对。就像混沌有一些相当根本的东西，它不仅仅是无望的噪音。在混沌系统中，至少在那个边界上，有一些有用的东西，对吧？

Yeah. Like there's something kind of quite fundamental about chaos that it's not just hopeless noise. It's like there's something kind of useful, right, in chaotic systems, at least at that boundary.

但没错，这只是我作为哲学思考。我实际上对数学了解不够，无法评论。无论如何，如果我们回到 MCTS，它在做什么？关键的是，它并不是说我们要直接增加获胜的概率。它不会说我们要提高所有获胜动作的权重，降低所有未获胜动作的权重。

But yeah, this is just my thinking about it as a philosophy. I don't actually know the math well enough to comment on it. Anyway, if we go back to MCTS, what is it doing? It is not, crucially, it is not saying we're going to increase the probability of winning directly. It's not going to say we're going to upweight all actions that won and downweight all actions that didn't win.

重要的是，它的做法是：对于我们采取的每一个动作，我们都通过 MCTS 进行了相当详尽的搜索，看看是否能做得更好，然后我们让策略网络预测那个结果，从而使我们采取的每一个动作都变得更好。这是一个非常好的想法，因为每个动作都有一个监督目标。

Importantly, what it is doing is saying: for every action we took, we did a pretty exhaustive search on MCTS to see if we could do better, and we're just going to make every action that we took better by having the policy network predict that outcome instead. This is a very nice idea because you have one supervision target for every single action.

是的。

所以你的学习信号的方差与另一种朴素的强化学习方法相比非常低。那么，让我们考虑一种非常朴素的算法，它看起来更像今天现代语言模型的强化学习：我们取自我对弈游戏中的胜者，并鼓励它更多地那样做。值得思考的是，除了 MCTS，我们还可以用什么替代方法来训练自我对弈智能体。现在我们经常使用语言模型风格的强化学习。那相关吗？我们能改用那种方法吗？让我们仔细想想。假设我们有一个非常朴素的算法：我们取一组不同检查点的智能体，让它们互相比赛。对于单个玩家获胜的游戏，我们强化那些动作，并重新训练策略网络来模仿那些玩家，而不是使用 MCTS 目标。

So the variance of your learning signal is very low compared to the alternative naive RL thing. So let's consider a very naive algorithm that looks a lot more like modern LM RL today, where we do something like: take the winner of a self-play game and encourage it to do more of that. It's worth thinking about what alternatives we could use to train self-play agents instead of MCTS. We use a lot of LM-style RL these days. Is that relevant? Could we do that instead? Let's think through this. Suppose we have a very naive algorithm where we take a league of agents of different checkpoints and play them against each other. For the games where a single player wins, we reinforce those actions and retrain the policy network to imitate those guys instead of the MCTS objective.

嗯。

最终发生的情况是：假设你有一个导致胜利的动作链，并且你有两个基本相同的智能体在对战。事实上，假设策略 A 和策略 B 势均力敌，所以它们的真实胜率是 50%。假设你玩了 100 局，每局 300 步，你正在做某种进化策略或某种方式来扰动这些智能体，让它们做不同的事情。或者也许你没有，你只是让它们互相比赛，偶尔其中一个可能比另一个有更好的策略。那么假设策略 A 赢了 51 局，策略 B 赢了 49 局，这只是因为随机运气，或者你以某种方式扰动了策略 A。为了有一个非常简单的模型，我们假设其中 50 局它们打得完全一样，而在那一局中，这个赢了的智能体打得略有不同。它做了一个关键动作，通常它会做得不同，但由于一些探索或随机噪声，它恰好做出了一个比以前更聪明的动作。

What ends up happening is: let's say you have a chain of actions that led to a win, and you have a matchup between two agents that are basically the same. In fact, let's assume policy A and policy B are evenly matched, so their true win rate is 50%. Let's say you play 100 games, each game lasts 300 moves, and you're doing some sort of evolution strategy or some way to perturb these things to get them to do different things. Or maybe you don't, and you just play them against each other, and occasionally one might have a better strategy than the other. So let's say 51 games policy A wins and 49 games policy B wins, just due to random luck or maybe you perturbed policy A in some way. To have a very simple model, let's pretend that for 50 of the games they played exactly equally, and on that one game where this one won, it played slightly differently. It made one critical move that normally it would have done differently, but due to some exploration or random noise, it just happened to make a smarter move than before.

所以，你有一个监督信号，一个真正的监督信号给你的策略网络。然后你有 99 局乘以 300 步，模仿那些动作会给你和以前完全相同的策略。

所以你的方差规模实际上非常糟糕，因为你在这个庞大的动作数据集中只有一个标签需要监督。让我澄清一下。我们刚才谈到，好的动作，即分布外的动作，只是所有游戏中所有动作的一小部分，而你希望在这些动作上进行训练。这让我想起了语言模型是如何通过策略梯度方法训练的。Karpathy 在播客中称之为“像用吸管吸监督信号”。所以有趣的是，你所说的这个在围棋中会难以处理并阻止你超越一定水平的方法，正是语言模型默认的训练方式？

So the scale of your variance is actually very bad because you only have one label out of this enormous dataset of actions where you want supervision. Let me clarify. We were just talking about how the good move, the out-of-distribution move, is a small fraction of all the moves played across all the games on which you'd want to train. This reminds me of how LLMs are trained with policy gradient methods. Karpathy, when he was on the podcast, called it like sucking supervision through a straw. So it's interesting that this thing you're saying, which would be intractable and prevents you from getting beyond a certain level in Go, is just by default how LMs are trained?

对。

对，所以在这种情况下，这并不是说它不起作用。如果你想象将游戏数量增加到数百万个样本，你实际上可以获得一些有意义的监督样本，只要你能找到一种方法来屏蔽来自这些家伙的监督。这就是事情开始与强化学习中的优势函数和基线等概念变得相当相关的地方。那么让我们看看这种非常朴素的方法的梯度方差。我把它称为梯度强化学习，它基本上就是奖励的总和。

Right, so in this case, this is not to say it doesn't work. If you imagine increasing the number of games to millions of samples, you actually can get some meaningful supervision samples, so long as you find a way to mask out the supervision from these guys. This is where things start to get pretty related to RL in terms of advantage and baselines and so forth. So let's look at the gradient variance of a very naive approach like this. I'm going to call it gradient RL, and it's basically the sum of rewards.

我明白你的意思了。所以奖励的总和就是回报，对吧？所以在我们这个朴素的设置中，我们只有一个回报的指示变量，要么赢要么输。所以在输的情况下，你就不训练，梯度为零；在赢的情况下，你尝试预测那些动作。所以你可以把这个设置看作是这里这个通用公式的一个特例。

这里的麻烦在于方差非常大。当你把这些项乘出来并试图计算方差时，梯度的方差等于平方的期望减去……为简单起见，我们可以假设如果你将其中心化到无信号，它平均为零。这里的方差基本上意味着你取这个乘积项的平方，所以你最终得到一个随 t 二次增长的项。所以方差，当你有一个这样的设置时，这个东西在这些项之上起到了耦合效应。

The trouble here is that this is very high variance. When you multiply these terms out and try to compute the variance, the variance of the gradient equals expectation of squared minus... For simplicity, we can pretend this is on average zero if you center it at no signal. The variance here basically means you're taking the square of this product term, so you end up with a term that grows quadratically with t. So variance, when you have a setup like this, this thing acts as a coupling effect on top of these terms.

嗯。

让我们把这个映射到语言模型案例中，并回答为什么语言模型只做单步强化学习而不是多步强化学习场景。在语言模型中，你有一个解码器，可能会预测像“hello world”这样的词。在当前的语言模型强化学习中，他们将整个序列视为一个单一动作，所以大 T 就是 1。是的，确实，由于 Transformer 是通过条件概率的乘积来表述的，这个序列的概率等于整个序列的对数概率之和，等于各个 token 的概率之和。所以在这种情况下，我会说像 log “hell” 加 log “low” 加 log “world”。所以这是真的，如果这一项是 1，那么它们就是一样的。

Let's actually map this to an LM case and answer why LLMs only do one-step RL instead of a multi-step RL scenario. In LM, you have a decoder that might predict some words like "hello world". In current LM RL, they treat this entire sequence as a single action, so big T is just one. And yes, it is true that because of how transformers are formulated through the product of conditional probabilities, we have the probability of this sequence equals the sum of log probabilities of the whole sequence equals the sum of the probabilities of individual tokens. So in this case, I would say something like log "hell" plus log "low" plus log "world". So this is true, and if this term were one, then they would be the same thing.

然而，在采样时，如果你给每个具体词元都分配一个奖励项，那么这些项之间就会产生交叉乘法的交互效应，对吧？问题就变成了如何将每个回合的信用归因到所有这些不同的项上？

However, in sampling things, if you have a reward term assigned to every specific token, now you have these interaction effects between the cross multiplication of these terms and these terms, right? And so the problem becomes how do you ascribe the credit associated with every episode to all these different terms here?

我困惑的是，在 LLM 中这样做会是什么样子？因为你只在回合结束时获得奖励。

I guess the thing I'm confused on is what would that even look like to do it that way in LLMs? Because you only get a reward at the end of the episode.

你可以想象一种奖励，比如“我给你一些过程监督”。对，你在每一步的每个动作都获得奖励。

You could imagine a reward that says like I'm gonna give you some process supervision. Yeah. Where you get a reward for each of these actions on every step.

好的。所以你的意思是，如果不那样做……嗯，我觉得你写的方式最后也是求和。所以它们不需要相乘。但你是说，与其那样做，不如最后把这些过程奖励加起来，然后当作一个单一的奖励信号。

Okay. So you're saying if instead of doing it that way where you... well I guess the way you've written it, it would be a sum at the end anyways. So they wouldn't have to be multiplied. But you're saying instead of doing it that way, you would just add up these process rewards at the end and then treat that as one single reward signal.

正确。针对一个单一的 log 动作。

Correct. For one single log action.

但这不就是一开始的写法吗？比如奖励的总和。

But isn't that how it's written to begin with anyways? Like the sum of the rewards.

数学中有点隐藏的是，我们假设当把问题分解成多步问题时，你通过这个家伙的计算引入了动作之间的相关性。如果你把这些分开，就会放大这个的方差。所以如果不分开，当 T 等于 1 时，你只有一个 log 概率的估计和一个奖励的估计。

So the thing that's a little bit hidden here in the math is that we're assuming that when you decompose the problem to a multi-step problem that you're now introducing kind of correlations between your actions through the computation of this guy. And if you separate these things out then this will magnify the variance of this one. So in the case where you don't separate it out, if you just have T equals 1, you just have a single estimate of log prob and a single estimate of reward.

现在这个项仍然出现在 L 中。所以在 LM 中，它看起来更像朴素的 REINFORCE 估计器，有点像单个动作的回报加上……你知道，它有点像这样。这是非常基本的形式，但它仍然是方差的贡献者。所以你要确保不要……类似于我们之前训练大量中性标签的情况。你要确保惩罚那些没有帮助的标签，只奖励那些真正让你变得更好的标签。对吧？

Now there is this term still shows up in L. So in LM it looks a little bit more like the naive REINFORCE estimator looks a bit like return of the single action plus... you know it looks kind of like this. This is sort of the very basic form here but this is still a contributor to variance. So you want to make sure that you don't... similar to how in this case we were training on a lot of neutral labels. You want to make sure that you're sort of penalizing the labels that don't help and only rewarding the ones that actually make you better. Right?

对。所以直观的类比是，我们能否在训练目标中找到一个项，使得它实际上被阻止这样做，或者这些对梯度没有影响，而这个对梯度有影响。

Right. So intuitively the analogy here is like can we find a term in our training objective such that it's actually kind of discouraged from doing this or you know these don't have any effect on the gradient and this has an effect on the gradient.

对。我想如果你在那里应用它，你唯一能做的就是消除 49 个游戏。所以至少按照你那里的方式，你会是 51 倍，实际上最优的情况是去掉所有这些动作，只在你变好的那个单一动作上获得梯度。

Right. I guess if you applied that there, the only thing you could do is eliminate 49 of the games. So at least the way you have earned there, you would be 51 times actually the optimal case is to pull out discard all of these moves and only get a gradient on that single move that you got better.

是的。但你怎么做到呢？

Yeah. But how would you do that?

对。所以这在实践中是个很棘手的问题。这就是强化学习中优势估计出现的地方。所以你想从你的乘数中减去一个项，而不是像 1 和 0 这样的指示函数。你想要一个东西，它对所有这些表现为 0，对所有这些表现为 1。

Right. So this is a pretty tricky problem in practice. And so this is where advantage estimation happens in reinforcement learning. So you want to subtract a term from your multiplier instead of an indicator function of like one and zero. You want something that kind of behaves like a zero for all of these guys and then a one for all these ones.

是的。但如果你能说“嘿，我赢了这局，所以这略高于基线表现”，你就可以做到。

Yeah. But you could do that if you can say, "Hey, I won in this game, so this is slightly above baseline performance."

嗯，你赢了很多局。正是。但你不知道哪些让你赢了是因为它们真的更好，还是因为运气。

Well, you won on a lot of games. Exactly. But you don't know which ones let you win because they were truly better versus winning on access.

你怎么设计一个真正更好的基线？

How would you design a baseline where it's truly better?

是的。这就是在 RL 中人们使用 TD 学习之类的方法来更好地近似质量函数 Q（我们之前提到的）的地方。所以你可以尝试从你的回报中减去它。所以理想情况下，在 RL 中你真正想做的是推高那些让你优于平均的动作，压低那些让你劣于平均的动作。他们称之为优势。有多种计算方法。我强烈推荐 John Schulman 的通用优势估计论文，作为思考各种计算方法的良好处理。但归根结底，你想通过让这个变小来减少方差，这样它就不会放大方差。

Yeah. So this is where in RL people use things like TD learning to better approximate the quality function, the Q that we mentioned earlier. So you can try to subtract that from your return. So ideally what you really want to do is in RL you want to push up the actions that make you better than the average and push down the actions that make you worse than the average. And they call this advantage. There are multiple ways to compute it. I highly recommend John Schulman's general advantage estimation paper as a good treatment on how to think about various ways to compute it. But at the end of the day, you want to reduce variance by trying to make this smaller and so it doesn't magnify the variance.

所以这要求你对一个状态的平均表现有很好的估计。这又回到了我们之前讨论的价值函数问题，对吧？请记住，在这种无模型 RL 设置中，它试图解决一个信用分配问题，即你不知道哪些动作实际上是好的，哪些是坏的。

So this requires you to have a very good estimate of what average performance from a state would look like. And this gets us back to the value function thing we were talking about earlier, right? And keep in mind that in this case this model-free RL setting is trying to solve a credit assignment problem where you don't know which actions were actually good and which ones were bad.

蒙特卡洛树搜索在做一些非常根本不同的事情，它不是在尝试对胜利进行信用分配。它是在尝试改进你采取的每个动作的标签。所以我们实际上可以考虑一个完全不同的算法，叫做神经虚构自对弈，它在 AlphaStar 和 OpenAI 的 Dota 等系统中取得了巨大效果。所以让我稍微谈谈如何统一一些这些在无模型设置和自对弈设置中的 RL 思想。

Monte Carlo tree search is doing something very fundamentally different, which is it's not trying to do credit assignment on wins. It's trying to improve the label for any given action you took. And so we can actually think about a completely different algorithm called neural fictitious selfplay, which was used to great effect in systems like AlphaStar and OpenAI's Dota. So let me talk a little bit about how you can kind of unify some of these RL ideas in the model-free setting as well as the selfplay setting.

好的。那么如果你不能轻松搜索树呢？比如在围棋中，它是一个完全可观察的游戏。你可以轻松构建一个相当深的树，完全捕捉游戏状态。在像星际争霸这样的游戏中，你无法完全控制二进制，这有点难做，我甚至不确定它是否是一个确定性游戏，对吧？所以从数据结构的角度来看，这有点困难。那么替代的做法是，用更好的老师来监督你的动作的基本思想仍然存在，对吧？所以在给定的神经虚构中，我们将讨论神经虚构自对弈是如何工作的。同样的想法。我们会为每个动作想出更好的标签，就像在 MCTS 中一样。但我们如何推导出更好的标签？在 MCTS 中，我们执行搜索，假设我们有一个好的价值函数，搜索会给出比我们初始猜测更好的结果。在一个不容易模拟搜索过程的游戏中，他们做的是训练一个所谓的最佳响应策略。所以你固定你的对手。假设你正在训练 pi A 对抗一个强大的对手 pi B。在星际争霸中，比如你可能是虫族，而你在玩神族之类的。

Okay. So what happens if you don't have the ability to easily search a tree, right? Like in Go it's a perfectly observable game. You can easily construct a pretty deep tree that completely captures the game state. In a game like StarCraft where you don't have really complete control over the binary, it's a little bit hard to do this and I'm not even sure if it's a deterministic game, right? So that makes this kind of difficult from a data structures perspective. So what is done instead is that the basic idea of supervising your actions with a better teacher is still there, right? So in a given neural fictitious, so we're going to talk a little bit about how neural fictitious selfplay works. Same idea. We're going to come up with better labels for each of the actions we took just like in MCTS. But how do we derive the better labels? In MCTS, we perform search and assuming we have a good value function, the search will kind of give us a better result than our initial guess. In a game where you can't easily simulate a search process, what they do instead is train what is known as a best response policy. So you fix your opponent. So let's say you're currently training pi A against a strong opponent pi B. In StarCraft maybe like you know the zergs and you're playing protoss or something.

所以，你固定对手，将其视为经典的模型无关强化学习问题，目标就是击败这个对手。你使用标准的 TD 学习技巧或 PPO 等任何模型无关强化学习算法，通过爬山法来战胜这个玩家。你训练的奖励函数是：击败对手得 1 分，否则得 0 分。这不再是自我对弈，而是固定对手，你只需最大化对该对手的得分。一旦你用喜欢的模型无关强化学习算法（如 PPO、SAC、VMPO 等）训练出好的策略，你就得到了该策略应对该对手的良好标签。当你训练多个最佳响应策略时，可以将强化学习算法提炼成针对特定对手的标签。例如，你可能有一个针对 pi_B 的最佳响应策略，还有针对 pi_C、pi_D 等的策略。你拿每个固定对手训练出的最佳响应策略，用它们提供的标签来监督主策略。这有点像 MCTS 教师的替代品：不用 MCTS，而是用模型无关强化学习算法找出击败对手的最佳动作。然后你将策略提炼成一个混合策略，平均所有可能的对手。这样得到的策略表现不会差于联盟中的平均对手。这避免了从 MCTS 获取教学信号的问题，但本质上仍然是用更好的动作重新标记状态，以改进策略。

So you fix your opponent and treat this as a classic model-free RL algorithm where your goal is just to beat this guy. You use standard TD learning tricks or PPO or any model-free RL algorithm to hill climb against winning this player. You train with a reward function that gives 1 if you win against the opponent and 0 otherwise. This is no longer self-play; it's a fixed opponent, and you just maximize your score against that. Once you have a good policy trained with your favorite model-free RL algorithm (PPO, SAC, VMPO, etc.), you have a good label for what the policy should do against that player. When you train multiple best response policies, you can distill the RL algorithms into labels for a given opponent. For example, you might have a best response policy against pi_B, and then against pi_C, pi_D, etc. You take the best response policy trained against each fixed opponent and supervise the main policy with the labels from these best responses. This is like a proxy for an MCTS teacher: instead of MCTS, you use a model-free RL algorithm to find the best action to beat the opponent. Then you distill the policy into a mixed strategy that averages across all possible opponents. This gives you something that performs no worse than an average opponent from the league. This avoids the need for a teaching signal from MCTS, but it's still about relabeling states with better actions to improve the policy.

确认一下我的理解：如果你在对战中击败了另一个策略，你会强化该轨迹上的所有动作？

And just to make sure I understand: if you win a game against this other policy, you reinforce all the actions on that trajectory?

是的。这里你可以使用 PPO、VMPO 甚至 Q-learning 等算法。具体算法通常是模型无关的，因为你没有搜索，但 MCTS 和 Q-learning 之间有一个有趣的联系。在 MCTS 中，你构建一棵树，通过叶子节点价值函数的分辨，沿着序列反向传播得到平均价值估计；你的 Q 值来自模拟的平均值。在模型无关算法中，通常有估计 Q 值的部分。Q 值通过 TD 学习获得，尽管在 PPO 中优势估计不一定通过贝尔曼备份。但在 Q-learning 中有一个很酷的技巧：Q(s,a) 被备份为 R + 折扣因子 * max_a Q(s',a')。直观上，对于 MDP，一个状态的最佳动作等于执行该动作的奖励加上下一状态的最佳表现。这种递归的动态规划性质允许你训练神经网络来强制执行这种一致性。一旦你知道一个动作的 Q 值，就可以用它来计算前一个状态的 Q 值。

Yes. So here you can use algorithms like PPO, VMPO, or even Q-learning. The specific algorithm is usually model-free because you don't have search, but there's an interesting connection between MCTS and Q-learning. In MCTS, you build a tree and through the resolution of value functions at the leaves, you back up through sequences to get a mean value estimate; your Q is derived from the average of simulations. In model-free algorithms, there's often a component of estimating a Q value. Q values are learned through TD learning, though in PPO the advantage estimation isn't necessarily via Bellman backup. But in Q-learning, there's a cool trick: Q(s,a) is backed up as R + discount * max_a Q(s',a'). Intuitively, for an MDP, the best action at a state equals the reward for taking that action plus the best you can do at the next state. This recursive dynamic programming property allows you to train neural networks to enforce this consistency. Once you know the Q value of an action, you can use it to compute something about the Q value of the previous state.

所以之前我问为什么训练策略而不是只训练价值函数，这就是答案？

So earlier when I asked why we train a policy instead of just training the value alone, that's what this is?

这是一种在无法进行前向搜索时恢复中间步骤价值估计的算法。

This is an algorithm for recovering value estimates of intermediate steps when you don't have the ability to do forward search.

所以你必须先收集一个 n 步的轨迹，然后才能使用这个技巧。

So you must collect a trajectory of n steps first before you can use this trick.

但直觉是一样的：知道这里的 Q 值可以告诉你那里的 Q 值，你可以从 Q 值恢复策略。你不需要显式建模策略分布；通过对 Q 值取 argmax 就能恢复策略。所以 Q-learning 或近似动态规划将你对未来 Q 值的了解向后传播。你可以看到类似的结构：在 MCTS 中，你规划智能体尚未访问过的轨迹；而在 Q-learning 中，你规划智能体已经访问过的轨迹。

But the intuition is the same: knowing something about the Q value here can tell you something about the Q value there, and you can recover a policy from a Q value. You don't need to explicitly model the policy distribution; you can recover it by taking argmax over Q values. So Q-learning or approximate dynamic programming propagates what you know about future Q values backward. You can see a similar structure: in MCTS you plan over trajectories the agent hasn't visited yet, while in Q-learning you plan over trajectories the agent has visited.

那么为什么 Q-learning 如此重要？历史上我们无法在机器人等高维问题上进行搜索。长期以来，我们假设如果不能用世界模型建模动态，那就只收集轨迹，然后根据唯一重要的数字——奖励——进行规划。

So why was Q-learning a big deal? Historically we couldn't do search on high-dimensional problems like robotics. For a long time, we assumed that if we can't model dynamics with a world model, we'll just collect trajectories and plan with respect to the only number that matters: reward.

好的，这非常有趣。为了将其与我们关于大语言模型的讨论统一起来：在大语言模型中，你没有 Q 值，但你做了这种反向学习：找到通过某个编码环境中单元测试的轨迹，然后强化这些轨迹。这与使用 MCTS 的前向方法有巨大差异。MCTS 更可取的原因是你可以在每一步都进行搜索，让每一步都更好，而不是必须按轨迹学习，并希望像 Karpathy 说的那样通过吸管学习——基本上就是升级轨迹中所有可能与得到正确答案相关或不相关的 token。

Okay, so this is very interesting. To unify this with our discussion of LLMs: with LLMs, you don't have Q values, but you do this backward learning: find trajectories that pass a unit test in some coding environment, then reinforce those trajectories. There's a huge difference between that and the forward approach with MCTS. The reason MCTS is preferable is that you can do it per move and make each move better, rather than having to learn per trajectory and hope to learn through a straw, as Karpathy would say—basically upgrading all tokens in a trajectory that might or might not have been relevant to getting the answer right.

之所以能在围棋中做到这种更高效、更有利的事情，是因为 MCTS 在围棋中有效。你基本上知道，如果我在这里进行局部搜索，并且这个搜索在末端被一个价值函数截断——即使我没有展开整个轨迹，这个函数也能工作——我就可以说这是我的新策略，并且我可以以更迭代、更局部的方式改进，而不必展开所有这些轨迹。

The reason you can do this much more sample-efficient, much more favorable thing with Go is that because MCTS works in Go, you basically know that if I just do search locally here and this search is sort of truncated at the end by this value function that works even if I haven't unfolded my whole trajectory, I can just say this is my new policy and I can improve in a more iterative, local way rather than having to unfold all these trajectories.

所以，我记得谷歌在 2023、2024 年有一些研究，他们确实尝试将树结构应用于推理。

So there was some research I think from Google in 2023, 2024 where they did try to apply tree structures to reasoning.

是的。

Yeah.

而且我认为，你知道，关于这是否能成功，尚无定论。所以我想说，我们未来很可能会看到对前向搜索这一想法的重新审视。但有两件事让 MCTS 在围棋中非常简单：一是价值估计相当具体，你可以真实地确定它，然后你可以用它来截断深度，就像你说的那样；二是广度也是确定的。

And I think it's, you know, the jury is still out as to whether this can ever work. So I would say, like, we probably will see, you know, revisiting of this idea of forward search in the future. But there's two things that make MCTS very simple for Go: which is that value estimation is kind of concrete and you can determine it for real, and then you can kind of use it to truncate depth as you said, and then the breadth is also determined.

而且关键的一点是，你迭代访问并扩展树的动作选择算法，非常适合围棋的问题规模和深度。但对于像 LLM 推理这样的事情，你知道，PUCT 可能不是一个足够好的启发式。它可能对局部 token 过于贪婪，可能会做类似这样的事情：哦，只给你一些显而易见的正确想法，但并不能真正解决你的最终问题。

And what's kind of critical is that the action selection algorithm where you iteratively visit and grow the tree is well suited for the size of problem that Go is and the depth of the problem. But for something like LLM reasoning, you know, PUCT might not be a good enough heuristic. It might be too greedy with local tokens and it might do something like, oh, only give you, you know, sort of obvious thoughts that are correct but not really solve your final problem.

所以我想说，关于 LM 推理的最终形式会是什么样子，可能尚无定论。而且我不排除这些东西可能会卷土重来，但这有点困难。

So I would say the jury is probably still out on how, like, what the final instantiation of reasoning for LM would look like. And I wouldn't rule out that like this stuff could, you know, come back, but it's a bit hard.

LM 难道不是天生就学会了做 MCTS 吗？它们会尝试一种方法，然后说：“哦，那行不通。让我们退回去，试试别的东西，然后朝着更有成果的方向前进。”

Don't LM sort of natively learn to do MCTS where they'll try an approach and be like, "Oh, that doesn't work. Let's back up, let's try this other thing and then go in the direction that proves to be more fruitful."

呃，是的，我确实认为 LLM 成功地做到了看起来像真正人类推理的事情，而不必显式地使用树结构。

Uh, yeah, certainly I think the LLM's managed to do something that looks like real human reasoning without having to do an explicit tree structure.

嗯，话虽如此，我认为进行前向搜索和模拟以更好地了解什么是有价值的想法可能会卷土重来，尽管不完全是 AlphaGo 那样的形式。

Um, that being said, I think the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback, even though not exactly in the same instantiation as AlphaGo.

但为了确保我理解问题的关键：广度来自合法动作的数量更广，深度来自无法轻易训练价值函数，因为……

But just to make sure I understand the crux of it: the breadth from the number of legal actions being wider and the depth from being able to not being able to train a value function as easily because...

所以这里有一个 LM 失效的例子：PUCT 规则涉及，你知道，sqrt(n) / (1 + n_a)。在 LLM 中，你很可能永远不会对同一个子节点采样超过一次，对吧？所以，假设你有多个思考步骤，因为语言如此广泛和开放，离散的动作集并不是 LLM 真正合适的选择。尽管它们是离散的 token，但数量如此之大，以至于这种探索启发式可能不是引导如何搜索树的正确方法。

So here's an example where LM's break down: the PUCT rule involves, you know, square root of n over 1 + n_a. In an LLM, you're most likely never going to sample the same child more than once, right? So, if you have, let's say, multi-steps of thinking, because language is so broad and open-ended, it's a sort of discrete set of actions is not really an appropriate choice for an LLM. Even though they're discrete tokens, it's just such a large number that this type of exploration heuristic is probably not the right thing to do to guide how to search down a tree.

对吧？但我想关键归结为这样一个事实：在围棋中，你知道 MCTS 几乎肯定比当前策略更好，即使你没有探索任何轨迹的末端。

Right? But I guess the crux comes down to the fact that in Go, you know that the MCTS is almost certainly better than your current policy even though you haven't explored the end of any trajectory.

正确。

Correct.

然后在 LLM 或机器人的正常推理中，没有办法仅仅局部评估和改进你的下一步，而不依赖于实际解决问题。

And then in normal reasoning for LLMs or robotics, there's no way to just locally evaluate and improve your next move in a way that doesn't result in, in a way that's independent of actually solving the problem.

呃，“没有办法”这个词太重了。我认为很多人已经考虑过如何尝试将 MCTS 或其类似后继者如 MuZero 应用于连续控制空间，而且我相信，你知道，非常酷的研究工作仍在进行中，试图解决这个问题。但没错，目前看似挑战的是，大多数问题在更高维的动作空间或组合上更大的东西（如语言）中，似乎不像围棋那样适合离散动作选择启发式和游戏评估类的东西。但这并不是说，沿着多条平行轨迹思考未来的想法可能不会给你一些关于搜索方向的信息，对吧？就像如果你考虑数学，我认为数学通常更像是一种逻辑搜索过程，你可以回溯，你可以看到哪些路径看起来不错，那里有更严格的结构。而也许在商业谈判或类似的事情中，它更像是一个树，也许有点……

Uh, "no way" is a strong word. I think lots of people have thought about how to try to apply MCTS or its kind of successors like MuZero to continuous control spaces, and I'm sure, you know, very cool research work is still ongoing to try to crack that problem. But yes, the seeming challenge right now is that most problems in much higher-dimensional, you know, action spaces or something that's combinatorially much bigger like language, they don't seem as amenable to the kind of discrete action selection heuristics as well as kind of game evaluation type stuff that Go does. But that's not to say the idea of, like, thinking into the future along multiple parallel tracks might not give you some information about which way to search, right? Like if you think about mathematics, I think mathematics often occupies a little bit more of a logical search kind of procedure where you kind of can back up, you can kind of see which paths seem good or not, there's a more rigid structure there. Whereas maybe in a business negotiation or something, it's less of a tree and maybe something a bit...

好了，我们现在坐下来了，所以我可以问你更多关于 AlphaGo 和 AI 研究的一般性问题。2021 年，Andy Jones 发表了一篇论文，题为《奖励游戏的缩放定律》。他基本上通过展示你可以权衡测试时算力和训练算力，预见了推理算力或推理缩放。也就是说，你可以将更多算力用于通过 MCTS 进行前向搜索。如果你这样做，你可以获得与花费更多时间训练模型相同的性能。所以如果你看到这种模式，你可能会想，好吧，对于 LLM，你将来可能会做类似的事情，而事实上这确实发生了。那么，现在有什么有趣的探索可以在玩具设置中探索缩放的其他维度，这对于理解几年后 AI 发展可能是什么样子很重要？

Okay, so we're now seated so I can ask you some more questions about AlphaGo and about AI research more generally. In 2021, Andy Jones had a paper called "Scaling Scaling Laws for Reward Games". And he basically anticipated inference compute or inference scaling by showing that you can trade off test-time compute and training compute. That is to say that you can spend more compute on the forward searching through the MCTS. And if you do that, you can get the equivalent performance as having spent more time training the model. And so if you see this pattern, you might think, okay, well with LLMs you might do something like that in the future, and in fact that's what had ended up happening. Okay, so what is a kind of fun exploration one could do now to explore other axes of scaling in toy settings which will be important to understanding what AI development might be like in a few years.

当然。是的。我认为测试时缩放和推理以及它们如何与模型大小相互作用，对于有多少需要作为显式搜索实际完成，又有多少可以打包到神经网络的前向传播中，确实意义深远，对吧？神经网络的前向传播是如何学习做那些应该是顺序和递归步骤的事情的？这很有趣。所以 Andy Jones 关于棋盘游戏缩放定律的论文非常酷。那篇论文中还有另一个非常好的结果，他展示了不仅你可以预测 LM 类型的缩放损失，即随着参数增加，你可以减少搜索所需的算力，反之亦然。他还展示了你可以实际预测解决更大版本棋盘游戏需要多少算力。

Sure. Yeah. I think that indeed test-time scaling and reasoning and how it interacts with model size are quite profound when it comes to how much needs to be actually done as explicit search versus how much can be packed into the forward pass of a neural network, right? And how does a forward pass of a neural network sort of learn how to do something that should be a sequential and recursive step? That's quite interesting. So the Andy Jones scaling laws for board games paper is quite cool. There's another really nice result from that paper where he showed that not only can you predict scaling loss of the sort of LM variety where as you increase parameters you can decrease the amount of compute for search or vice versa. He also showed that you can actually predict how much compute is needed to solve a larger version of the board game.

例如，围棋可以从 3x3 棋盘扩展到无限大小，你或许可以重新审视这个问题，尝试复现这种模式。我启动这个项目的动机是：苦涩教训或我们对缩放定律的了解，能否让我们在算力最优的围棋机器人上执行得更好？我们能否在不使用所有 KataGo 技巧的情况下，仅通过专注于苦涩教训和缩放定律来构建一个强大的围棋机器人？到目前为止我还没有成功。但我认为，通常当你希望缩放定律起作用时，你需要处于配方已经有效且数据集良好的状态，而不是在试图弄清楚缩放的同时还要找出正确的数据集。

For example, with Go, which can scale from a 3x3 to an infinitely sized board, you might be able to revisit this question and try to reproduce whether this shows up. I actually started this project with the motivation: does the bitter lesson or our knowledge of scaling laws allow us to execute much better on a compute-optimal Go bot? Can we build a strong Go bot without all the KataGo tricks, just by focusing on the bitter lesson and scaling laws? I have not been successful so far. But I think it's a fact that usually when you want scaling laws to work, you want to be in a regime where the recipe already works and the datasets are good, rather than trying to figure out how to do scaling while also figuring out the right datasets.

所以这就像研究中的科学理解部分，通常先让某样东西工作起来，然后利用该系统收集数据，帮助你建立事物如何运作的心智模型，比如缩放定律。通常，如果你想利用缩放定律构建一个强大的围棋机器人，你实际上必须先做出一个强大的围棋机器人，然后利用缩放定律向未来外推一点。

So this is like the scientific understanding component in research often follows a step where you get something to work first, and then you use that system to collect data that helps you build a mental model of how things work, such as scaling laws. Usually, if you want to build a strong Go bot using scaling laws, you actually have to make a strong Go bot first and then use the scaling laws to extrapolate a bit farther into the future.

多说一点让我理解。首先，你是说缩放定律不起作用，还是你在你的机器人中看不到任何缩放定律的模式？

Say more so I understand. First of all, you're saying scaling laws did not work, or you could not see any scaling law pattern in your bot?

是的。我最初犯的一个错误是，当 MCTS 标记存在一些 bug 时，我会用专家策略收集大量数据，然后将其视为监督学习问题，并尝试用专家数据集识别缩放定律。你确实可以绘制出看起来像这样的图。但如果你处于策略效果不佳的状态，你可能只是在研究坏数据上的缩放定律。一个重要的实现细节是，如果你想研究缩放定律问题，你必须有一个数据良好、架构良好且没有 bug 的问题，然后在那里解决它。直到系统其余部分正常工作，我才能应用缩放定律来指导该关注什么。这对研究人员来说显而易见——当然你希望在研究缩放之前有一个工作且无 bug 的系统——但作为给实践者的建议，我真正绊倒的地方是：你不一定想在你的人造物足够有趣之前就跳入研究它的科学。

Yeah. A mistake I made initially when I had some bugs around how MCTS labeling was working was I would collect a bunch of data with an expert policy and then treat it as a supervised learning problem and try to identify scaling laws with expert datasets. You can indeed plot things that look kind of like this. But if you're in a regime where your policy is not working well, you might be just studying scaling laws on bad data. One important implementation detail is that if you want to study a scaling laws problem, you kind of have to have a problem for which the data is good, the architecture is good, and there are no bugs, and then you solve it there. I wasn't able to apply scaling laws to direct what to look at until I had the rest of the system working. This sounds obvious to researchers—of course you want a working, bug-free system before you study scaling—but as advice for practitioners on where I actually tripped up: you don't necessarily want to jump into the science of studying your man-made artifact before your man-made artifact is interesting enough to be studied.

说到算力，如果你看过去 10 年训练世界上最好 AI 模型所用的算力图，在对数空间里是一条非常平滑的线，逐年指数增长，但有一个巨大的异常值：AlphaGo Zero，它训练所用的算力远超当时任何其他 AI 模型。大约是 3e23 FLOPs，与前沿大语言模型相当。虽然差了几个数量级，但仍然如此。问题是，尤其是你能在自己的项目中得到一些东西……

Speaking of compute, if you look at charts of compute used to train the best AI model in the world over time going back 10 years, it's a very smooth line in log space, exponentially growing year-over-year, except there's this huge aberration: AlphaGo Zero, which was trained on way more compute than any other AI model at the time. It was like 3e23 FLOPs, comparable to a frontier LLM. Orders of magnitude off, but still. The question is, especially with you being able to get something off in your own...

我从 Prime Intellect 获得了捐赠。大约 1 万美元，然后我花了大概前 4000 美元做探索性研究，大约 3000 美元用于最终运行。剩下一些用于服务模型。

I got a donation from Prime Intellect. For about 10K, and then I spent maybe the first 4K doing exploratory research, and then about 3K on the final run. Some of it remained for serving the model.

你觉得他们只是训练得不好吗，如果你现在能用 1 万美元做到的话？

Is your sense that they just did a bad job training it, if you can do it in 10K now?

成为第一个做某事所需的算力总是比追赶所需的算力大得多。这在 LLM 领域也是如此。一旦别人做到了，你就可以使用蒸馏等技巧，或者各种拐杖来引导自己走向成功。对于我在线托管的自己的机器人，我使用了针对 KataGo 模型的最佳响应训练来获得强大的性能。截至录制时，我正在验证我是否能完成第一步，即从零开始学习。但重要的是，对于研究，你通常希望从一个好的初始化开始。我首先做的简单事情就是训练针对 KataGo 的最佳响应智能体。

The compute required to be the first to do something is always much larger than the compute it takes to catch up. It's the same story playing out in LLMs. Once someone else has done it, you can use tricks like distillation, or all sorts of crutches to bootstrap your way to success. With my own bot that I've hosted online, I used best response training against KataGo models to get a strong level of performance. As of recording, I'm validating whether I can do that first step, which is to do the tabula rasa play. But importantly, for research you often want to start from a good initialization. The simple thing I did first was train best response agents against KataGo.

AlphaGo 团队没有任何可以对抗训练的策略，因为他们试图从零开始。成为第一个意味着你优先考虑的是事情本身，而不是最算力高效的实现。这在机器人领域也是如此。如果你看为机器人训练的前沿大模型，散点图非常分散，没有像前沿 LLM 那样清晰的线条。这是因为训练这些模型的人通常没有达到每个 FLOP 都至关重要的规模，并且需要将压榨性能作为预训练中的主导因素。相反，他们的重点更多是希望某种能力出现，所以他们优化训练设置以使其易于获得该能力。一旦你有了那种能力，如果你扩大算力，你就被迫使其算力高效，因为那是数亿美元。但在过去，当实验算力更充足或研究人员不必真正负责时，人们最终会优化算力最优帕累托前沿之外的东西。

The AlphaGo team did not have any policy that they could train against, because they were trying to do it from scratch. Being the first to do it means you're prioritizing the thing rather than the most compute-efficient implementation. This plays out in robotics as well. If you look at the frontier of large models trained for robotics, the scatter plot is all over the place and there isn't a clean line like for frontier LLMs. That's because the folks training these models often are not at the scale where every FLOP counts and they need to squeeze out performance as the dominating factor in pre-training. Instead, their focus is more on wanting a certain capability to show up, so they optimize the training setup to make it easy to derive that capability. Once you have that capability, if you scale up compute, you are forced to make it compute-efficient because it's hundreds of millions of dollars. But in the past, when compute for experiments was more plentiful or not accounted in a way that the researcher was really responsible for, you end up with people optimizing for things other than being on the compute-optimal Pareto frontier.

我明白了，比如速度之类的。

I see, like speed or something.

是的，比如得到结果的时间，或者只是让它工作。我认为第一个 AlphaGo 可能有很多算力，他们不需要太担心让它成为最算力最优的东西。

Yeah, like time to result, or just getting it to work. I think the first AlphaGo probably had lots of compute and they didn't need to worry too much about making it the most compute-optimal thing.

那么，算力效率的改进中，有多少是 2017 年不存在的方法，又有多少是他们在 2017 年本可以做到的？

And how much of the improvements to compute efficiency are methods that did not exist as of 2017 versus things which they could have done in 2017?

好问题。

Great question.

所以进入这个项目时，我心里隐约知道事情总会随时间变得更容易，我想看看围棋现在发展到什么程度了，因为 2020 年 KataGo 之后似乎就没有什么重要的开源强棋手了。读了 KataGo 的论文，里面有很多巧妙的想法。我在想，好吧，看看“苦涩的教训”是否应验了——很多这类技巧因为 NVIDIA 制造了更快的 GPU 而变得不再必要，对吧？所以我们大致处于什么阶段？再次强调，这不是经过同行评审的说法，只是我基于自己实验的初步感觉。但似乎架构选择没那么重要，比如 Transformer 和 ResNet 的差别。我们现在的 GPU 速度已经到了模型规模不是很大的程度，所以架构真的不重要。实际上你可以大大简化设置。不用做带回放缓冲区、推送器和收集器的分布式异步强化学习，你可以做一个简单的同步方案：收集数据，训练一个监督学习模型，然后再收集，这样就有机会简化基础设施。NVIDIA GPU 确实变快了。KataGo 是在 V100 上训练的，而你用一半数量的桌面 Blackwell GPU 就能训练，效果依然不错。KataGo 开发的一些辅助监督目标，如果你有好的初始化，其实并不需要，对吧？如果你用对抗 KataGo 的最佳响应训练来初始化，你自己的模型根本不需要 KataGo 的那些技巧。所以核心问题是如何尽快获得强大的对手，这比具体的架构创新重要得多，但仍然有一些不错的算力倍增器。我发现用 9x9 棋盘训练对于解决终局价值函数非常好，如果你能在可迁移的架构上同时训练 9x9 和 19x19，那么就能大大缩短从头学习的预热时间。我记得 AlphaGo Zero 的曲线是前 30 小时左右基本都在追赶监督学习基线。你可以通过在小棋盘上预训练，然后热启动到 19x19 对局，来大幅缩短这个时间。还有一些其他东西，比如在回合之间改变模拟次数。结果发现这其实不那么敏感，你可以固定它或增加它，影响不大。但无论如何，从科学角度重新审视一篇旧论文，看看什么真正重要，还是挺有意思的。

So going into this project, I kind of knew in the back of my mind that things always get easier to do over time, and I wanted to see where Go is at, given that it didn't seem like there has been any major open-source strong bot after KataGo in 2020. And then reading the KataGo paper, there are a lot of clever ideas. I was wondering, okay, let's see if the bitter lesson has happened, where a lot of these kind of tricks just sort of go away because NVIDIA made faster GPUs, right? And so roughly where are we on that? So again, this is not a peer-reviewed claim. This is just my preliminary vibe guess on what I've seen based on my own experiments. But it seems like architecture choices don't matter that much, you know, Transformer versus ResNet. We're at the sort of speed of GPU where the size of the model is not so big that this really matters. You can actually simplify the setup quite a lot. So instead of doing a distributed asynchronous RL setup with replay buffers and pushers and collectors, you can do a dumb synchronous thing where you collect, you just train a supervised learning model, and then you collect again, and so there are opportunities to simplify infrastructure. NVIDIA GPUs have indeed got faster. Whereas KataGo was trained on V100s, you can train on like half the number of desktop Blackwell GPUs and it still works. And some of the kind of auxiliary supervision objectives that KataGo developed aren't really necessary if you have a strong initialization, right? So if you're initializing against best response training against KataGo itself, then your own model actually needs none of the tricks that KataGo needs. So then the core thing is how can you get as quickly as possible to some strong opponents, and that matters a lot more than the specific architectural innovations, but there are still some nice compute multipliers. So I found that training on 9x9 boards was very nice for resolving endgame value functions, and then if you can co-train that on an architecture that can transfer between 9x9 and 19x19, then you can really cut down the warm start time to learn that from scratch. I think AlphaGo Zero's plot was first 30 hours or so are spent basically catching up to the supervised learning baseline. And you can cut down that time a lot by kind of pre-training on a small board and then warm starting that into your 19x19 board play. There were some other stuff like varying the number of simulations between episodes. This turns out to be not that sensitive actually, you can fix it or increase it, doesn't matter too much. But anyways, it's kind of just nice from a scientific perspective just revisiting an old paper and seeing what really matters.

这算是个跑题的问题，但为什么 AlphaGo 可以用缓冲区？因为我每次和任何研究员聊天，他们都在告诉我离策略有多糟糕。但 AlphaGo Zero 的朴素实现方式是，在给定的反向步骤或一批反向步骤中，大多数落子都不是由最新训练的模型做出的。所以这是为什么？

This is sort of a tangential question, but why is it okay to have a buffer in AlphaGo? Because every time I talk to any researcher, they're telling me about how bad it is to be off-policy. But then the way a naive implementation of AlphaGo Zero would work is that most of the moves in a given backward step or in a batch of backward steps would not be among the ones that were made by the most recently trained model. So why is that?

好问题。这涉及到离策略与在策略强化学习的基本问题。如你所知，在蒙特卡洛树搜索中，你对自己采取的动作进行重新标注，以便在同一状态下采取不同动作，对吧？所以离策略的部分在于：如果你重新标注的是新策略永远不会访问的状态呢？那有什么意义？你是在浪费容量。极端情况下，想象你训练缓冲区中的状态分布全都是你永远不会访问的状态。那么你基本上是在监督它们在永远不会达到的状态上采取好动作，因此你的策略会变得非常糟糕。对吧？这就是离策略可能真正伤害 AlphaGo 的地方。然而，如果你从 Dagger 的角度来理解，它基本上是一种利用数据将自身纠正回最优轨迹的方法。在这样的算法中，你希望大部分状态是你将要访问的，但还有一小部分或合理比例的状态位于最优轨迹周围的高维管状区域内。这些状态都被赋予一个监督目标，以便将你引导回最优轨迹。也许我可以快速画个图说明。

Okay, great question. Yeah, and this gets into the sort of fundamental off-policy versus on-policy reinforcement learning questions. So as you recall in MCTS, you take actions that you took and you relabel them to take different actions on the same states, right? So the off-policy part here comes where what if you're relabeling states that your new policy would never visit, right? Like what's the point? You're kind of wasting capacity. And in the extreme limit, imagine your distribution of states in your training buffer are all states that you would never visit. Then you're basically supervising them to take good actions on states you would never achieve, and therefore your policy can get really bad. Right? So this is where off-policy can really hurt in AlphaGo. However, if you interpret this from the Dagger perspective, which is basically saying a way to correct yourself back to the optimal trajectory given some data, what you kind of want in an algorithm like this is to have mostly states that you would visit, but then you have a small percentage or maybe a reasonable percentage of states in this kind of high-dimensional tube around your optimal trajectories. And any of those states are given a supervision target to kind of funnel you back into your optimal trajectory. So maybe I can just draw quickly here.

很好。所以在 Dagger 风格的设置中，最优训练数据分布是这样的：这里是最优状态和动作。也就是说，你想处于这个状态，这个状态，这个状态，然后在这里获胜。这些是最优策略动作。这些是你肯定要训练的东西。但为了对干扰具有鲁棒性，你需要确保如果碰巧漂移到其他状态，你能把自己引导回……

Great. So in a Dagger style setup, what your optimal training data distribution is, is that here is your optimal states and actions. So this is like you want to be in this state, you want to be in this state, you want to be in this state and then you win here. And then these are your optimal policy actions. So these are the things that you definitely want to train on. But to make it robust to disturbances, you want to make sure that if you happen to drift off into some other states, you can kind of funnel yourself back into...

但这为什么不是一个完全通用的离策略训练论据呢？

But why isn't this a fully general argument for off-policy training?

这实际上就是为什么有时你想做离策略训练——你不希望出现累积误差，即一旦犯错，就没有数据指导如何返回最优分布。

This is actually why you want to do off-policy training sometimes, is that you don't want to have a compounding error where if you make a mistake, you don't have the data of how to return back to your optimal distribution.

是的。所以最优控制并没有太多讨论如何避免意外进入这种状态，因为它假设一旦你学会了策略，你就会到达那里。但在机器人等应用中，比如一阵风把你吹偏了一点，然后你需要纠正，对吧？或者一个轮胎的摩擦力比另一个轮子稍低，然后你的车开始漂移，你必须纠正。在更真实的环境中，这类事情经常发生。实际上，关于国际象棋和围棋有一个有趣的说法：围棋和国际象棋的问题在于对手总是在试图做些什么，对吧？所以情况可能会偏离。你总是希望能够纠正回你的获胜条件。

Yeah. And so, optimal control does not really say too much about how to not accidentally get here because it's sort of making the assumption that once you learn the policy, you're going to get here. But in applications like robotics, right, like a gust of wind blows you slightly off and then now you need to correct, right? Or the friction on one of your tires is a little bit lower than the other wheel and then now your car is drifting and you got to correct it. So these kind of things in more real environments often happen. Actually, there's a funny quote about chess and also Go is that the problem with Go and chess is that the other player is always trying to do something, right? So things can kind of drift off. And you always want to be able to correct back to your winning condition.

所以，你的回放缓冲区应该包含你的策略会访问的状态，再加上一些你可能漂移到的状态分布，以及如何返回最优状态的方法。

So, your replay buffer really should have the states that your policy would visit, plus some distribution of states that you might drift to and then how to return back to your optimal states.

是的。

Yeah.

现在，如果你把这个推向极端，你说，好吧，我们没有任何这些数据，我们只是用 MCTS 来标注那些远离最优行为的状态，比如这边的这一袋状态。那么，每个状态都会得到一个 MCTS 标签，你的策略学会了如何在这里采取最佳行动，但你永远不会到达这里。所以你是在训练模型处理你永远不会到达的状态。这就不对了。所以这是个问题，对吧？而这正是 off-policy 可能真正有害的地方。

Now, if you take this to the extreme and you say, well, let's not have any of this data and we're going to just be labeling with MCTS states that are so far away from our optimal behavior, like this bag of states over here. Well, now each of them gets an MCTS label, and your policy learns how to take the best possible action here, but you never get here. So you're training your model on states you would never reach. This is not there. So this is a problem, right? And this is where off-policy can really hurt.

是的。

Yeah.

实际上，作为这个项目的一部分，我确实尝试了一个实验：我拿了一堆轨迹，为了尽可能让 GPU 饱和，我做了的是从数据集中随机抽取状态，然后仅对这些状态重新运行 MCTS。对吧？所以不是玩一整局游戏，每一步都做 MCTS，而是忽略移动的因果性，只随机挑选棋盘状态，并用我当前的网络标注它们。我可能会重新访问之前标注过的旧状态，并用当前网络重新标注它们。对吧？实际上，这确实有效。你可以说，让我们拿一些合理的状态，在训练过程中不断重新标注它们。这实际上开始收敛到一个非常类似机器人学的设置，这很常见：你有一个轨迹数据集，然后有一个类似回放缓冲区推送器的东西，这些是 off-policy 的离线轨迹，对吧？你的回放缓冲区推送器将转移元组推送到回放缓冲区，然后有一个任务不断重新规划，你应该采取的最佳行动是什么，而不是实际采取的行动。在机器人学中，实际上很常见使用最小化 TD 误差的方法，所以你的 Bellman 更新器不断从那里拉取数据，并试图满足 QSA。然后从这里，你有你的训练器，它试图拟合从 S 到 A，或者拟合 Q 到 Q 目标。所以你可以把这看作一种规划器，对吧？你重新访问你去过的旧状态，用你当前的模型重新思考：如果我访问了这个状态，我能做得更好吗？这实际上是 off-policy 机器人学习系统通常的训练方式。如今有一种更简单的方案，但在 Google QT-Opt 时代，我们就是这样做的。

So actually, as part of this project, I did try an experiment where I took a bunch of trajectories and to try to saturate the GPU as much as possible, what I did was I took random states from the dataset and reran MCTS on just those states. Right? So instead of playing a whole game where I'm doing MCTS on every move, I just ignore the sort of causality of moves and just pick random board states and I just label those with my current network. And I might revisit old states that I've labeled before and relabel them again with my current network. Right? And so in practice, this actually does work. You can actually say, let's take some states that are reasonable and constantly be relabeling them while we're training. And so this actually starts to converge on a very robotics-like setup, which is very common: you have your dataset of trajectories, and then you have something like a replay buffer pusher, and these are off-policy offline trajectories, right? Your replay buffer pusher pushes transition tuples to the replay buffer, and then you have some job that's kind of continuously replanning what the best action you should have done instead of taking this action is. And so in robotics, it's actually very common to use the sort of minimized TD error, so your Bellman updater constantly is pulling things from here and trying to satisfy the QSA. So then from here you have your trainer, which is trying to fit the S to A or fit the Q to the Q target. So you can think about this as a sort of planner, right? You revisit old states that you've been to and you take your current model and you rethink, what could I have done better if I visited this? And so this is actually how off-policy robotic learning systems are usually trained. These days there's a sort of simpler recipe, but in the Google QT-Opt days we kind of did things like this.

那么训练器是什么？

So what is the trainer?

哦，是的，训练器就是试图最小化 QSA 和 Q 目标。

Oh yeah, the trainer is you try to minimize QSA and Q target.

那么你能从高层次再解释一下整个设置吗？

So can you explain the whole setup again at a high level?

是的。所以你有来自各种策略的 off-policy 数据。你不断将之前看到的转移推送到回放缓冲区。是的。然后你有一个叫做 Bellman 更新器的东西，它基本上重新规划：不是这个动作，我应该在 S 处采取什么动作才能有更好的价值？你强制执行的方式是试图最小化 TD 误差。所以给定这个，你有 S'，对吧？你计算 S' 的 Q 值，并找到与 S' 搭配的动作，使这个 Q 值尽可能高，然后你把它加到这里的奖励上，这就得到了你的实际目标。对吧？所以对于当前的 S-A，你的 Q 目标就是这个。现在你把 Q 目标送回给这个转移。所以对于这个元组，你把它与 Q 目标配对，然后在训练器上，你简单地使用监督学习，最小化当前网络的 QSA 与其目标。

Yeah. So you have your off-policy data that came from various policies. You're constantly pushing transitions that you saw before to a replay buffer. Yeah. And then you've got this thing called a Bellman updater, which basically replans: instead of this action, what action should I have taken at S to have a better value? And the way you enforce that is you try to minimize the TD error. So given this, you have S prime, right? You compute Q of S prime and find the action that should go with S prime that makes this Q value as high as possible, and then you add that to the reward here and that gives you your actual target. Right? So for this current S-A, your Q target is this. So now you send back the Q target to this transition. So with this tuple, you pair it with a Q target, and then here on the trainer, you simply use supervised learning and minimize your current network's QSA with its target.

明白了。好的。所以在后台，你基本上是在想，让我想想这些动作实际上有多有价值。

Got it. Okay. So, in the background, you're just like, hey, let me basically think through how valuable were all these actions actually.

是的。在一个更优的策略中，你试图最大化这个，这个转移的 Q 目标是什么？

Yeah. In a more optimal policy where you're trying to maximize this, what is the Q target of this transition?

这基本上就像是在做白日梦。

It's sort of like basically daydreaming.

完全正确。是的。你可以把它想象成你在事后回顾，就像，根据我在历史缓冲区中看到的情况，有没有我可以采取的更好的行动？这里我尝试过的与围棋的联系是，它还算成功，但太复杂了无法开源：你用 MCTS 重新标注器替换了它，不是做这种目标网络计算，而是在你的转移上运行 MCTS。所以在这种情况下，你有你的状态、你的动作，以及你是否赢了游戏。实际上你可以扔掉这两个，你不在乎这些，你只取你的状态，然后规划 MCTS 来得到你的最佳策略 pi，基于你当前的网络，对吧？不是采取这个动作的网络，而是你当前的最佳策略网络。你只是离线在这些转移上重新运行你的搜索。如果这些是你的策略能够到达的转移，那么这实际上起到了很好的稳定效果。另一个好处是，你可以更好地让 GPU 饱和，因为你不会因为围棋游戏而阻塞等待棋盘状态。你只需并行搜索所有深度的所有棋盘状态。

Exactly. Yeah. You can think about it like you're kind of going back in hindsight and being like, given what I've seen in the historical buffer, was there a better action I could have taken? Now the connection to Go here that I tried and it was moderately successful but too complex to open source was: you replace this with an MCTS relabeler, where instead of doing this kind of target network computation, you run MCTS on your transition. So in this case, you have your state, your action, and whether you won or not at the game. And actually you can just toss these two, you don't care about these ones, you just take your state and you plan MCTS to get your best policy, pi, on your current network, right? Not the network that took this action, but your current best policy network. You just rerun your search offline on these transitions. And if these are transitions that your policy can get to, then this actually acts as a very nice stabilizing effect. And also the one other benefit is that you can kind of fully saturate your GPU better because you're not blocking on the Go game to give you board states. You simply search across all board states at any depth in parallel.

是的。

Yeah.

那么这里的训练器就是尽可能预测 MCTS 标签。所以再次，这有点用，在机器人学中也很相关，因为你确实有很多离线数据，而且你不能模拟像 MCTS 这样的东西，但在实践中它确实会遇到一个问题：如果当前模型正在查看它永远不会到达的状态，那么它就是在浪费容量。所以你必须小心一点。所以 on-policy 的东西，而且很多强化学习已经收敛到更 on-policy 的设置，他们并不真正尝试直接在 off-policy 数据上训练；他们最多使用 off-policy 数据来减少方差，但不直接影响目标。

So then here the trainer would be just predict the MCTS label as possible. So again, this kind of works and this is quite relevant in robotics where you really just have a lot of offline data and you can't simulate things like MCTS, but in practice it does run into the problem where if the current model is looking at states that it would never reach, then it's kind of wasting capacity. So you have to be a little bit careful here. So the on-policy thing, and also much of RL has kind of converged to a much more on-policy setup where they don't really try to directly train on off-policy data; at best they use off-policy data as a way to reduce variance but not directly influence the objective.

哦，抱歉，为什么他们收敛到了那种方式？

Oh sorry, why have they converged to that?

只是更稳定。

It's just more stable.

是的。

Yeah.

是的。所以你可能会用 off-policy Q 来做优势计算，比如 Q 减去 Q 的和。

Yeah. So you might use the off-policy Q as a way to do advantage computation, like Q minus sum of Q.

是的，这有点像你的价值，或者抱歉，就像总和，如果有 n 个动作，那么这是你的价值，这是你当前的 Q 值。所以该动作的优势是平均价值减去当前价值。人们可以尝试以离策略的方式估计 Q，然后在这里直接使用优势。如果动态中存在问题，它不会让损失爆炸得那么厉害。在机器人学中，有一种趋势是使用离策略数据来塑造奖励，而不是直接作为你的……

Yeah, that's kind of like your value, or sorry, like the sum of, if there's n actions, then this is your value and this is your current Q value. So your advantage for that action is the average value minus your current one. People can try to estimate Q in an off-policy way and then just use advantage here. If there's a problem in the dynamics, it doesn't blow up your loss as much. In robotics, there's a convergence towards using off-policy data to shape your rewards, but not actually be directly your...

这让我想起我们之前的对话，为什么 MCTS 相比 LLM 使用的 reinforce 或策略梯度方法更有利。这可能完全错误，但我几个月前写了一篇博客，关于 RL，至少是策略梯度 RL，比你想象的还要低效。人们天真地认为的低效在于，你必须展开整个轨迹才能获得任何学习信号。随着这些轨迹变得越来越长，智能体不再只是完成句子中的下一个词，而是需要做两天的工作来判断你是否正确完成了这个项目。每 flop 的信息量在下降，因为你必须展开两天的思考来看你是否做对了。每 flop 的样本量在下降。你可以认为你在学习时试图最大化每 flop 的比特数。这是每 flop 的样本数乘以每样本的比特数。我刚才提到的是，随着 RL 的视野变长，每 flop 的样本数会下降。但这种朴素的 RL 从每样本比特数的角度来看也很糟糕。

I'm reminded now of our earlier conversation of why MCTS is so favorable compared to the kind of reinforce or policy gradient thing LLMs do. This might be totally wrong, but I wrote a blog post a few months ago about how RL, at least policy gradient RL, is even more inefficient than you might think. The inefficiency one thinks about naively is that you have to roll out a whole trajectory to get any learning signal at all. As these trajectories become longer and longer, an agent has to, instead of just completing the next word in the sentence, go and do two days' worth of work to figure out if you even did this project correctly. The amount of information per flop has been decreasing as you have to unroll two days' worth of thinking to see if you did something correctly. The amount of samples per flop has been decreasing. You can think of you're trying to maximize bits per flop as you're learning. This is samples per flop times bits per sample. What I just mentioned is that samples per flop go down as RL becomes more long horizon. But this kind of naive RL is also terrible from a bits per sample perspective.

将其与监督学习进行比较。在训练早期，假设 LLM 的词汇表大小为 10 万。所以有 10 万个可能的 token 可以回答。你有一个完全未训练的模型和一个提示，比如“天空是”。在监督学习中，模型会对所有可能说的内容有一个概率分布。有一个标签说“蓝色”，它会通过交叉熵损失学习其分布与正确说出“蓝色”之间的距离。如果你通过 RL 来做，模型会尝试“天空是 alacon”。不对，错了。“天空是 told”。不对，错了。这是一个完全未训练的模型。你需要尝试大约 10 万次才能偶然碰到“蓝色”，然后获得一些学习信号。在监督学习中，你得到概率分布，被告知是“蓝色”，然后计算出你偏离了多远。你学到的量是正确率的一个函数。你离“蓝色”越远，你通过交叉熵损失学到的向“蓝色”靠近的就越多。你可以把它看作你的正确率，即你之前说出“蓝色”的概率。在监督学习中，通过交叉熵损失，一旦你得到这个标签，你学到的就是负对数正确率的比特数。而在 RL 中，如果你只是随机猜测并看是否有效，那只是二元随机变量的熵，也就是……

Compare it to supervised learning. Early on in training, let's say you have a vocabulary size for an LLM that is 100k long. So there are 100k possible tokens one could answer. You have a totally untrained model and a prompt like 'the sky is'. With supervised learning, the model would have some probability distribution over all the things it could say. There's a label that says 'blue', and it would learn, for cross-entropy loss, exactly how far its distribution is from correctly saying 'blue'. If you were doing this through RL, the model would try 'the sky is alacon'. Nope, that's wrong. 'The sky is told'. Nope, that's wrong. This is a totally untrained model. You would have to do this on the order of 100,000 times to just stumble on 'blue', then get some learning signal. In supervised learning, you get your distribution of probabilities, you get told it's 'blue', and you figure out how far off you are. The amount you learn is a function of your pass rate. The further away you are from 'blue', the more you've learned to go towards 'blue' using cross-entropy loss. You can think of it as your pass rate, your prior probability of having said 'blue'. In supervised learning, through cross-entropy loss, you would learn negative log p being pass rate bits once you get this label. Whereas in RL, if you're just randomly guessing and seeing if it works, that's just going to be the entropy of a binary random variable, which is...

对。所以如果你的策略没有机会采样到“蓝色”，那么你永远不会得到信号。

Right. So it's like if your policy has no chance of sampling 'blue', then you will never get a signal.

完全正确。这由你采样到“蓝色”的概率极低这一事实来建模。如果你确实采样到了，你学到的和你在监督学习中一样多。在所有其他情况下，在未训练的模型中，99.999% 的时间里，你从看到“alacon”不是正确的词或“told”不是正确的词中学到的极少。这就是大多数情况下发生的事情。所以你学到的很少。如果你尝试画图，把 x 轴设为正确率，这里放你从样本中学到的比特数。如果你这里有 0%，这里有 50%，这里有 100%。训练结束时你在这里。在监督学习中，负对数正确率看起来像这样。而二元随机变量的熵看起来像这样。这取决于你用的是 nats 还是 bits。

Exactly. That's being modeled by the fact that your probability of sampling 'blue' is extremely low. If you do sample it, you learn as much as you would have in supervised learning. In all other cases, 99.999% of the time in an untrained model, you're learning incredibly little from seeing that 'alacon' is not the correct word or 'told' is not the correct word. That's what happens most of the time. So you learn very little. If you try to graph, put on the x-axis your pass rate, and here you put the bits you're learning from a sample. If you have 0% here, 50% here, and 100% here. At the end of training you're here. With supervised learning, negative log pass rate would look something like this. And then the entropy of a binary random variable would look like this. This is depending on whether you're doing nats or bits.

是的。如果你用 bits，峰值就在这里，是 1。这就像抛硬币。你从抛硬币中学到的最多。

Yeah. If you do bits, it's like one right here at the peak. This is like a coin flip. You learn the most from a coin flip.

这是监督学习，这是 RL。然而，问题在于你大部分训练时间都处于这个区域，即低正确率区域。事实上，你学习的速度取决于你每样本获得的比特数，而在这里你得到的信号非常少。如果你在对数尺度上绘制正确率，把 x 轴设为对数尺度，在训练开始时，词汇表大小为 10 万，正确率是 1/100,000，然后是 1/10,000，1/1,000，1/100。这个图看起来是这样的：监督学习像这样，RL 像那样。可以说你所有时间都花在这里，可能从未获得一次成功。这是一个令人沮丧的图，因为一旦你在这里，如何到达那里一点也不明显。一旦你到了那里，你就有了一些东西，但在许多 RL 问题中，你所有时间都花在这里。所以问题是如何初始化，使得你至少不是零，而是非零的正确率。

This is supervised learning, this is RL. However, the problem is you spend most of training in this regime, in the low pass rate regime. In fact, how fast you're learning is a function of how many bits per sample you're getting, and you're getting very little signal here. If you chart the pass rate on a log scale, put the x-axis on a log scale where at the beginning of training with a vocab size of 100k, the pass rate is 1/100,000, then 1/10,000, 1/1,000, 1/100. What this graph looks like: supervised learning would look like this, and RL would look like that. Arguably you spend all your time here, potentially never even getting a single success. It's a sort of depressing plot in the sense that once you're here, it's not at all obvious how you get to here. Once you're here you have something, but in many RL problems you spend all the time here. So there's a question of how do you initialize so you're at least not at zero but at a nonzero pass rate.

是的。

Yeah.

关于每样本比特数，我想再补充一点，这与任何机器学习问题都非常相关，那就是与软目标和蒸馏的联系。如果你能访问 logits，而不仅仅是一个 one-hot 的 token 答案，如果你能访问软目标，这个分布的熵远高于 one-hot。

One more thing I'd like to add about bits per sample that's very relevant to any kind of machine learning problem is that there's a connection to soft targets and distillation. If you have access to the logits, not just the one-hot token answer, if you have access to the soft targets, the entropy of this distribution is far higher than the one-hot.

是的。

Yeah.

所以实际上，软标签每个样本包含的信息量（以比特计）要大得多。

So there's actually way more information in bits and bits per sample in a soft label.

所以这就是蒸馏在每个样本上如此有效的原因——它实际上在每个样本中提供了更多的信息。

So that's why distillation is so effective per sample is that it's actually giving you way more information per sample.

啊，是的，我想知道公式会是什么，但显然它就是这个分布的熵。比如这个的熵是零。对。这个的熵就像，你知道，熵的公式。这也是为什么 AlphaGo 如此美妙。在 AlphaGo 中，你不是训练策略网络去模仿 MCTS 的动作，而是训练它去模仿 MCTS 的分布。

Ah yeah, well I wonder what the equation would be, but obviously it would just be the entropy of this distribution. Like so the entropy of this is zero. Yeah. The entropy of this is like, you know, the entropy equation. And this is also why, like, AlphaGo is quite beautiful. In AlphaGo you don't train the policy network to imitate the MCTS action; you train it to imitate the MCTS distribution.

有意思。

Interesting.

但这两种方法实际上都是有效的。如果你想做一个科学实验，看看这种软标签暗知识蒸馏有多重要，你可以做一个实验：用 MCTS 选择的动作（而不是软分布）重新训练策略网络。

But both of these are actually valid. And if you wanted to do a scientific experiment of how important this kind of soft label dark knowledge distillation is, you can run an experiment where you retrain the policy network on the action MCTS selected rather than the soft distribution.

有意思。之前我直觉上有点困惑。为什么这种迭代搜索的能力——你不需要一开始就能赢，只需要能改进当前策略——在学习中如此强大，相比于当前 LLM 学习强化学习的方式？是的，正是这种考虑整个轨迹通过率的东西。我实际上不知道如何形式化地思考这个问题。也许你可以帮我解释一下。

Interesting. Earlier I was sort of stumbling around this intuitively. Why is this ability to do iterative search where you don't necessarily need to be able to win the game in the beginning? You just need to be able to improve your current policy. Why is that so powerful a capability in learning as compared to how LLMs currently learn RL? And yeah, it's exactly this thing of considering your pass rate of the entire trajectory. I actually don't know a formal way to think about this. Maybe you should help me out here.

为什么 AlphaGo 是一个优雅的强化学习算法？主要原因是，你永远不需要从 0% 的成功率开始，然后解决如何获得非零成功率的探索问题。这让你能够沿着这个美妙的监督学习信号爬山。如果你看 AlphaGo 的实际实现，每一步都没有 TD 误差学习或动态规划——至少没有显式地。它只是在价值分类和策略 KL 最小化上进行监督学习。所以它只是一个在改进标签上的监督学习问题。因此训练非常稳定，对吧？你可以训练任意大的网络。你可以在数据集上重新训练它。一切都会稳定进行。基础设施也非常简单实现。你不需要一个复杂的分布式系统来保持策略在线。归根结底，你只是说：“我有一些改进的标签。让我们在这些目标上重新训练我的监督模型。”

Why is AlphaGo an elegant RL algorithm? The major reason is that you never have to initialize at a 0% success rate and solve the exploration problem of how to get a non-zero success rate. And this is what allows you to hill climb this beautiful supervised learning signal. If you look at the actual implementation of AlphaGo, every step of the way there's no TD error learning or dynamic programming, at least explicitly. It's just supervised learning on a value classification as well as a policy KL minimization. So it's just a supervised learning problem on improved labels. And so the training is very stable, right? You can train as big of a network as you want. You can retrain this on the dataset. Everything will just go stably. The infrastructure is very simple to implement as well. You don't need a complex distributed system to keep everything on policy. At the end of the day, you're just saying, "I have some improved labels. Let's retrain my supervised model on these targets."

所以你总是处于这种美妙的状态，你只是在努力改进策略，而不是逃离那种所有信号都平坦的局部最小值。

And so you're always in this beautiful regime where you're just trying to improve the policy, rather than escape this kind of local minima where every signal is flat all around you.

一种画曲线的方式是：如果你画出 MCTS 策略与原始网络的胜率，假设这条虚线是原始网络。MCTS 策略看起来像这样。所以每一步这个监督信号都非常干净，对吧？你永远不会遇到 MCTS 不给你信号的情况。除非你的 MCTS 分布收敛到与你的策略网络预测完全一致。

One way to draw the curve is like: if you draw the win rate of an MCTS policy versus the raw network, let's say this dotted line is the raw network. The MCTS policy kind of looks like this. And so every step of the way this supervision signal is very clean, right? You're never in a situation where the MCTS is giving you no signal. Unless your MCTS distribution converges to exactly what your policy network predicts.

对。对。对。

Yeah. Yeah. Yeah.

好的。这是一个很好的解释方式。

Okay. That's a great way to explain it.

好。那我们坐下来，我问一些关于自动化研究的问题。

Cool. Okay. Maybe we sit down and I ask some questions about automated research.

好的。

Sounds good.

我特别想和你聊的是，你通过这种自动化的 LLM 编码助手循环完成了这个项目的大量研究。有一种观点认为，如果你完全自动化 AI 研究，你可能会实现某种奇点。显然我们还没到那一步，但就我们已有的早期迹象来看，我很好奇你的观察：AI 擅长什么，不擅长什么，你对这个场景的看法，它最终的可能性，以及你对此的总体想法。

One thing I really wanted to talk to you about is that you did a bunch of the research for this project through this kind of automated LLM coding assistant loop. And there's an idea that if you fully automated AI research, you could have some sort of singularity. Obviously we're not there yet, but to the extent that we have early indications of what this process might look like, I am curious what your observations about what the AI is good at, what it's not good at, what you think about this scenario, its likelihood eventually, what thoughts you have about this in general.

当然。是的，我认为自动化科学研究是前沿实验室目前正在开发的最令人兴奋的技能之一。我认为对于任何从事研究的人来说，了解它现在能做什么、不能做什么，以及未来当 AI 自动化大量调查时科学过程可能如何运作，都非常重要。简而言之，我在这项工作中主要使用 Opus 4.6 和 4.7。有效的是，模型在超参数优化方面做得非常好。过去，人们会提出一组超参数搜索范围，比如学习率、权重衰减，以及网络层数，然后进行网格搜索或贝叶斯超参数优化，找到一些调优后的参数。现在自动化编码真正酷的地方在于，它可以搜索更开放的问题集。它可以说：“嗯，我发现这一层的梯度有点小，让我在这里改一下。让我重写代码，让数据加载器有我想到的新增广。让我们尝试找到最适合优化问题约束的方法。”最终你会得到一种更灵活、更高级的、几乎像研究生一样的能力，去打磨一个性能指标。这可以挤出相当多的性能。在固定的数据集和固定的时间预算下，你可以在语言建模或围棋这样的分类问题上大幅提高困惑度。而且它现在在执行任何实验方面也非常出色。我写了一个名为“experiment”的 Claude 技能，我给它一个我想要绘图的描述，我只是描述：“这是我想要的 x 轴，这是我想要的 y 轴，回答我这个问题。”然后它会跑去完成所有实验，编译图表，生成报告，并建议可能的原因等等。所以这就是今天效果很好的地方，我认为我们可以期待这些能力在未来变得更好。但了解它今天做得不太好的地方也很有用。所以在我博客版本的教程中，我有一张图，基本上把我做的所有实验按树状分组，每个节点代表一个失败、成功或混合的实验结果，然后从那里分支到子节点，即后续实验。

For sure. Yeah, I think automated scientific research is one of the most exciting skills that the frontier labs are developing right now. And I think it's important for everyone who's doing any kind of research to get a good intuition of what it can do now and what it can't, and how the science process might work in the future once we have AI automating a lot of this investigation. So in brief, I mostly use Opus 4.6 and 4.7 throughout working on this. And what works is that the models can do a very good job of hyperparameter optimization. So in the past people would come up with a search base of hyperparameters like learning rate, weight decay, and maybe how many layers are in your network, and they would just do a grid search or a Bayesian hyperparameter optimization approach, and then it would find some tuned parameters. The really cool thing that automated coding can do now is that it can search a much more open-ended set of problems. It can say, "Well, I've identified that the gradients are kind of small on this layer, so let me change it up here. Let me rewrite the code so the data loader has a new augmentation I came up with. Let's try to find the best way to fit the constraints of the optimization problem." And you end up with this much more flexible and high-level, almost like grad student-like ability to just grind a performance metric. And so this can squeeze out quite a lot of performance. On a fixed dataset with a fixed time budget, you can improve perplexity by quite a lot on a classification problem like language modeling or Go. And it is also fantastic now at basically executing any experiment. So I have a Claude skill that I wrote called "experiment" where I give it a description of what I wanted to plot, and I just describe: "Here's the x-axis I want, here's the y-axis, answer this question for me." And it will go run off and do all the experiments, compile the plot, make a report, and suggest what might have caused it or so forth. So that's what works quite well today, and I think we can expect that these abilities get better in the future. But it's also kind of useful to know what it is not doing so well today. So on my blog version of this tutorial, I have a plot of basically all the experiments I did grouped in a sort of tree where every node represents a failed, successful, or mixed experimental result, and from there it branches off into a child where it's the follow-on experiment.

我偶尔会沿着一条路线深挖下去，比如离策略 MCTS 重标注，做几个实验，然后意识到可能不值得。于是我会跳到完全不同的路线上。我把这些称为“行”。我发现，目前公众能访问的闭源模型，在给定路线中挑选下一个实验方面并不擅长。它们似乎无法退一步进行横向思考，比如“等等，这条路线不太合理，让我们回到第一性原理，想想瓶颈可能是什么，或者我们到底想达成什么”。我经常需要自己通过向 Claude 提出正确的问题来捕捉基础设施错误，让它调查是什么导致了这种差异，然后它会回答。我认为随着 Mythos 类模型或 Mythos++ 模型上线，这可能会彻底改变，这些问题会随着 Scaling 的改善而消失。但与此同时，我认为有很多丰富的机会来开发可能激励这种横向思考的强化学习环境。建立这个围棋环境的动机之一是，围棋包含了许多非常有趣的研究问题，通常与 LLM 或机器人技术重叠，而且验证起来非常快。外循环最终是：智能体是否按我预期的方式行动？你可以很容易地检查一局围棋的结果。内循环则涉及所有关于分布式系统的研究工程，预测一个想法是否会成功，预测训练算法的特定修改可能带来的差异。我认为有丰富的子任务和子环境库，你可以训练一个自动化科学家来处理，以围棋作为外部验证循环。一旦你获得了这些技能，也许你可以将它们应用到其他领域，比如生物科学或机器人技术。

Occasionally I'll rabbit hole down a track like off-policy MCTS relabeling, do a few experiments, and then realize it's probably not worth it. So I'll jump to a completely different track. I call these things rows. What I find is that current closed models that the public can access today don't seem to be that great at selecting what the next experiment should be in a given track. They don't seem to be able to step back and do lateral thinking like, wait, this track doesn't really make sense, let's go back to first principles and think about what the bottleneck might be or what we are trying to achieve. Often I have to catch infra bugs myself by prompting the right question to Claude to investigate what is causing this discrepancy, and then it will answer the question. I think with Mythos-class models or Mythos++ models coming online, maybe this completely changes and these problems just fall to improved scaling. But at the same time, I think there is a lot of rich opportunity to develop RL environments that might incentivize this kind of lateral thinking. One of the motivations for setting up this Go environment was that Go captures a lot of very interesting research problems often overlapping with LLMs or robotics, yet it's very quick to verify. The outer loop is ultimately: does the agent do what I think it does? You can check the outcome of a Go game quite easily. The inner loop involves all this research engineering around distributed systems, predicting whether an idea is going to work, predicting the difference a particular modification to your training algorithm might make. I think there is a rich library of subtasks and sub-environments that you can train an automated scientist to work on, with Go as a sort of outer verification loop. Once you acquire these skills, maybe you can apply them to other domains like biosciences or robotics.

或者自动化 AI 研究。

Or automating AI research.

或者自动化 AI 研究，这才是真正的关键，或者说制造未来版本 AI 的可怕又不可思议的事情。

Or automating AI research, which is the real crux or the scary slash incredible thing for making future versions of AIs.

你是在暗示这里的外循环基本上可以是你对 KataGo 的胜率。

And you're suggesting the outer loop here could just be your win rate against KataGo basically.

那是其中之一。我认为还有很多更深层次的问题可以解决。例如，假设你有一个关于如何改进缩放定律算力乘数的想法。结果不一定是我做出了有史以来最好的围棋机器人。结果可能只是：我能预测我的围棋机器人的胜率吗？或者我能预测我的想法产生的缩放定律曲线吗？但随后你可以通过在外循环中使用像围棋这样非常可验证的游戏来验证你没有奖励黑客任何东西。

That's one of them. I think there are a lot of deeper questions one could tackle. For example, let's say you have an idea on how to improve a scaling law compute multiplier. The outcome isn't necessarily that I achieve the best Go bot ever. The outcome might just be: can I predict what the win rate of my Go bot will be? Or can I predict the scaling law plots that emerge from my idea? But then you can verify that you haven't reward hacked anything by using a very verifiable game like Go on the outer loop.

我认为有几个有趣的后续问题。有关于内循环和外循环的问题。在内循环中，有一个问题是，你做的任何修改在局部上有多可验证。也就是说，你尝试某个想法时，能否知道它实际上是改进还是退化？如果某件事不起作用，你能知道是因为 bug 还是因为想法本身错了？伊利亚曾谈到为什么他认为自己是一个优秀的研究者。他认为自己优秀的原因之一是，他有直觉，对正确的想法有强烈的信念，并且能够坚持克服 bug，根据他对这个想法应该有效的高级信念，知道哪些是 bug，哪些是基本想法上的错误——所以如果想法应该有效，那一定是 bug，反之亦然。我们不如就从这个问题开始吧？

I think there are a couple of interesting follow-on questions. There are questions on the inner loop and the outer loop. On the inner loop, there's a question of how locally verifiable any modification you might make is. That is to say, would you know whether something is actually an improvement or a degradation, some idea you try out? Would you know if something isn't working as a result of a bug or is it the result of the idea itself being wrong? Ilya was talking about why he thinks he is a good researcher. One of the things he thinks makes him a good researcher is that he has intuition, he has strong belief in what the correct idea is, and he is able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea based on his high-level belief that this idea should work, so therefore it has to be a bug versus the other way around. Why don't we start with that question actually?

是的。好的想法在局部上有多可验证？我认为在深度学习的成功故事中，你可以将其视为一个长达数十年的想法，需要很大的信念才能让它成功。这提出了一个非常具有挑战性的长周期强化学习问题，每一步都有一群委员会告诉你这是个坏主意，然后你最终突破。那么，如何设计强化学习环境，使其可能更早地给你一些反馈？我认为这是一个非常棘手的开放问题，我没有答案，但最终要打造一个非常强大的围棋机器人，你可能确实需要发现深度学习。

Yeah. How locally verifiable are things which are good ideas? I think in the case of the success story for deep learning, you can think about this as a decades-long idea that took a lot of faith to get it to work. This presents a very challenging long-horizon RL problem where every step of the way you have a committee telling you that this is a bad idea, and then ultimately you break through. So how do you design RL environments that maybe give you some feedback earlier? I think this is a very tough open question that I don't have an answer to, but ultimately to play a very strong Go bot, you probably did need to discover deep learning.

是的。

Yeah.

没错。我认为，在外循环中有一个难以作弊的挑战性游戏，可以用作类似发现深度学习原理的外循环信号。当然，为了使其易于处理，这就是研究品味真正重要的地方，你必须想出初始化问题的方法，这样你就不会解决一个非常棘手的问题。也许你可以利用 LLM 作为一种中间通用语法来给你一些局部反馈。LLM 是通用语法这一事实意味着它们可以在几乎任何抽象层次上运作；它们可以非常局部地思考，也可以退一步进行非常宏观的思考。我认为这正是人类横向思考能力的来源：如何知道你所追求的路线或目标不对，你应该问一个不同的问题。

Right. I think that having a challenging game that cannot be cheated easily on the outer loop could be used as a sort of outer loop signal for something like discovering the principles of deep learning. Of course, to make it tractable, and this is where research taste really matters, you have to come up with ways to initialize your problem so that you don't solve a very intractable problem. Maybe you can leverage LLMs as a sort of universal grammar in the middle to give you some local feedback. The fact that LLMs are universal grammar means they can move at almost any level of the stack; they can think very locally as well as step back and think in very broad steps. I think that's where a lot of the lateral thinking ability of humans comes from: how to know if the track you're pursuing or the objective you're pursuing is not right and you should be asking a different question.

另一个问题是，在试图在外循环上获得更好结果的过程中，局部改进的可叠加性如何。我听说在一些 AI 实验室，出问题的地方在于，人们会各自追求好的想法，但这些想法最终不能很好地叠加，因此训练运行因为两个看似好想法之间的奇怪交互而失败。有一个自上而下的统一愿景来指导事情应该如何运作非常重要。你在不同的 AI 实验室工作过，也尝试过让并行智能体尝试不同想法，你对 AI 创新的可并行化程度有什么看法？

The other question is how stackable local improvements are in the attempt to get to a better result on the outer loop. I've heard rumors that at some AI labs, the thing that has gone wrong is that people will individually pursue good ideas, but those don't end up stacking well, and so the training run fails because of some weird interaction between two seemingly good ideas. Having a single top-down vision of how things should work is very important. Having worked at different AI labs and also playing around with parallel agents trying different ideas, what is your sense of how parallelizable AI innovation is?

是的，好问题。

Yeah, great question.

我认为，要在“苦涩的教训”上执行得好，研究品味在于知道它能带来多少收益，以及在当下不能要求太多。当然，从长远来看，算力是决定一切的最重要因素。随着能量、算力和参数规模的扩张，智能几乎必然会涌现出来。这非常美妙且深刻，除此之外，没有哪个算法细节真正重要。但在当下，我们没有无限的算力、参数或任意好的初始化。所以我们必须想出一些启发式方法，但这些方法可能有些冗余。这就是为什么很多算力倍增器并不一定叠加——它们可能有相关的收益。三年后，当英伟达 GPU 变得更强时，这些倍增器可能叠加得更差。在任何时间点，任何算力倍增器的收益都是暂时的，这也是我对 Kadiggo 论文的怀疑。那篇论文用了很多算法思路，但用现代的 Blackwell 和 Ada 级 GPU（比论文用的 V100 级 GPU 好得多）时，你会发现一些加速收敛的算法技巧相比其他因素已经不那么重要了。这在当下是一个品味问题。

I think the research taste for executing well on the bitter lesson is that you need to know how much the bitter lesson can buy you and how much is too much to ask at any given moment. Of course, in the fullness of time, compute is the single most important determinant of how things work. It's almost inevitable that as you scale up energy, compute, and parameters, intelligence will just fall out of that. That's super beautiful and profound. No algorithmic detail really matters beyond that. But in the present day, we don't have infinite compute, parameters, or arbitrarily good initialization. So we have to come up with heuristics that give us that, but these heuristics are probably somewhat redundant. That's why you see this effect where a lot of these compute multipliers don't necessarily stack—they might have some correlated benefit. Then three years down the line, when Nvidia GPUs have gotten even stronger, maybe they stack even less well. At any given point in time, the benefit of any given compute multiplier is transitory, which is what I suspected with the Kadiggo paper. There were many algorithmic ideas applied, but with modern Blackwell and Ada class GPUs, which are much better than the V100 grade GPUs that paper used, you can see that some of those algorithmic tricks to speed up convergence just don't matter so much compared to something else. That's a matter of taste in the present time.

有意思。那外层循环呢？用围棋让 AI 更聪明时，确实有一个外层循环，即对抗最强开源模型的胜率。但即使在那里，正如你所说，还有其他外层循环，比如“你是否发现了一个新现象？”——如果你不知道缩放定律的重要性，这在 2015 年很难验证，而 Chinchilla 或 Kaplan 缩放定律是 2019 年才发布的。所以如果你回到 2015 年，没有自动化流程能判断哪篇论文是缩放定律论文，哪篇只是另一个随机图表。即使在围棋案例中，这也是一个难以验证的外层循环。外层循环的整个想法是给改进设置一个后盾。但对于通用 AGI，我们有一堆基准测试，但问题在于：我们知道可以衡量的东西，并改进它们，但我们关心的是更广泛的经济上有用的工作能力，这在自动化一切之前并不容易衡量。所以问题来了：AI 自我改进的外层验证循环有多好？这重要吗？

Interesting. How about the outer loop? For making AI smarter with Go, you do have this outer loop of win rate against the best open source model. And even there, as you were saying, there are other outer loops like 'did you discover a new phenomenon?' which is very hard to verify if you didn't know scaling laws were important back in 2015 when Chinchilla or Kaplan scaling laws were released in 2019. So if you're back in 2015, there's no automated procedure to know which paper is the scaling laws paper versus just another random plot. So even in the Go case, that's a hard-to-verify outer loop. The whole idea of an outer loop is to have some backstop on improvement. But for general AGI, we have a bunch of benchmarks, but there's a problem: we know the things we can measure, and we improve on them, but we care about this broader ability to do economically useful work, which is not super easy to measure until you automate everything. So there's a question: how good is the outer verification loop for AI self-improvement, and does that matter?

是的。我要给出一个非严格的论证，但我直觉上相信。AI 研究实验室 DeepMind 最初专注于游戏。他们把游戏作为外层循环，研究人员从解决游戏的经验中学习。现在他们研究大语言模型，很可能从玩 Atari、围棋和星际争霸等游戏的经验中获得了正迁移，这有助于他们做出好的大语言模型。我假设在编码、一般研究能力或项目管理方面存在正迁移。所有这些可能都帮助他们做得好。如果是这样，为什么自动化的 AI 研究者不也如此呢？他们应该能够将处理快速验证、快速迭代环境的经验正迁移到更宏大、经济上更有用的事情上，比如自动化药物发现。

Yeah. I'm going to give a non-rigorous argument, but one I intuitively believe. DeepMind, the AI research lab, started with a focus on games. They used games as their outer loop, and the researchers learned from experience solving games. Now they're working on LLMs, and presumably there was positive transfer from their time working on games like Atari, Go, and Starcraft that helps them make good LLMs. I assume there's positive transfer in coding, general research ability, or project management. All these things probably help them do well. If that's the case, why wouldn't it also be true for automated AI researchers? They should be able to positively transfer experience tackling quick-to-verify, quick-to-iterate environments to something more ambitious and economically useful, like automating drug discovery.

我的意思是，我不知道。历史上，直到 Gemini 3 或类似版本，人们不是一直在说谷歌在大语言模型上落后是因为他们太固守旧方法了吗？有收益，但也有方式会主动阻碍你。所以对我来说，正迁移并不明显。

I mean, I don't know. Hasn't the issue historically been that until Gemini 3 or whatever, people were saying Google hasn't been catching up in LLMs because they're too tied to the old approach? There are gains, but there are also ways in which it actively hinders you. So it's not obvious to me that there's positive transfer.

尚无定论。谁知道在游戏上训练的初始化最终是否会阻碍他们长期获胜的能力？很难说。同样，谁知道看似起步晚是否只是他们花了更长时间预训练如何扩展 TPU？他们把技术树投资在让 TPU 变得更好上，短期看似乎没什么用，但长期可能成为优势。即使拥有今天的数据，人类也很难推理出最优的研究策略应该是什么。

The jury is still out. Who knows if the initialization on training on games will ultimately hobble their ability to be the winner in the long term? It's hard to say for sure. Likewise, who knows if the late start was really just them pre-training for longer on how to scale up TPUs? They invested their tech tree in getting TPUs to be good, which seemed not that useful in the short term, but in the long term it might become a advantage. It's even hard for humans to reason about what the optimal research strategy should be, even with the data we have today.

是的。酷。我们应该让大家知道如何了解更多关于这个项目的信息，无论是自己 fork，还是查看你的博客，你在那里很好地解释了许多这些想法。人们接下来去哪里？

Yeah. Cool. We should let people know how they can find out more about this project, whether to fork it themselves, or check out your blog where you do an excellent job explaining many of these ideas. Where do people go next?

太好了。我的网站是 evjang.com。有一篇博客文章链接到这个教程的交互式版本。在 GitHub 上，用户名 eric Jang，有一个 autogo 仓库，人们可以 fork 并复现训练结果。我还强烈推荐大家查看博客文章“as rocks may think”，它涉及了我们对话中的一些想法，但更宏大，探讨了当思考成为计算机科学中的原语时会发生什么。

Great. My website is evjang.com. There's a blog post that links to an interactive version of this tutorial. On my GitHub, username eric Jang, there's an autogo repo that people can fork and reproduce the training results. I also highly recommend people check out this blog post 'as rocks may think', which touches on some of the ideas in this conversation, but it's a grander thesis of what happens when you have thinking as a primitive in computer science.

完全正确。我鼓励听众思考通过 MCTS 和搜索，思考与围棋之间的关系，以及它如何与大语言模型相关联。我认为那里有深刻的东西，可能因为围棋相对于大语言模型的繁荣而被相对忽视。并不是说我们应该在大语言模型中加入树，但两者之间有一种非常有趣的对偶性，而且你实际上可以用很小的预算对围棋 MCTS 和推理做很多研究。这非常令人兴奋。

Exactly right. I encourage the audience to think about the relationship between thinking and Go via MCTS and search, and how it relates to LLMs. I think there's something profound there, probably underexplored just because Go has been relatively underexplored compared to the boom in LLMs. It's not to say we should have trees in our LLMs, but there is a very interesting duality between them, and you can actually do a lot of research on Go MCTS and reasoning with very small budgets. That's very exciting.

酷。太棒了。Eric，谢谢你参加。很荣幸能上这个播客。

Cool. Awesome. Eric, thanks for doing this. It's an honor to be on the podcast.