从 AlphaGo 到 AI 智能体：Giannis 谈强化学习的突破

围棋是一种复杂的游戏，人们总是有点担心 AlphaGo 是否真的像我们相信的那样出色。所以我们实际上坚信，基于我们能测量和看到的一切，深度强化学习就是答案。但这些系统的特点在于，它们不像经典计算机那样，你总是知道它们会给出相同的答案。它们是随机的，有创造力，也有缺陷；它们会像大语言模型一样产生幻觉。所以你需要真正推动它们，看看它们在哪里出错。而做到这一点的唯一方法就是让最优秀的人类与它们对弈。今天，我们很高兴邀请到 Giannis Antonoglou，他是一位研究员和工程师，为人工智能领域一些最重要的突破做出了贡献。作为 DeepMind 的创始工程师，Giannis 在开发 AlphaGo 中发挥了关键作用，AlphaGo 击败了围棋世界冠军李世石，创造了历史。后来，他共同领导了 MuZero 的开发，通过自主掌握多种游戏进一步突破了界限。现在，随着他开启最新的创业项目 Reflection，他专注于构建下一代 AI 智能体。我们很高兴与 Giannis 讨论他亲眼见证的 AI 历史上的突破性时刻，从 AlphaGo 著名的第 37 手，到他对强化学习与大语言模型结合通往 AGI 之路的当前看法。Giannis，非常感谢你今天加入我们。

Go is a complex game and there was always a bit of worry about whether AlphaGo was truly as good as we believed. So we actually had the conviction that deep reinforcement learning is the answer based on everything that we could measure and everything we could see. But that's the thing about these systems: they're not like classic computers where you just know that they always produce the same answer. They're stochastic, they are creative, and they have some flaws; they hallucinate similarly to how LLMs hallucinate. So you need to really push them and see exactly where they break. And the only way you could actually do that is by having the best humans playing against them. Today we're excited to welcome Giannis Antonoglou, a researcher and engineer who has contributed to some of the most significant breakthroughs in AI. As a founding engineer at DeepMind, Giannis played a crucial role in developing AlphaGo, which made history by defeating Go world champion Lee Sedol. He later co-led the development of MuZero, which pushed the boundaries even further by mastering multiple games autonomously. Now, as he embarks on his latest venture with Reflection, he's focused on building the next generation of AI agents. We're excited to talk to Giannis about the breakthrough moments in AI history that he's witnessed firsthand, from AlphaGo's famous Move 37 to his perspective today on what's next for the combination of reinforcement learning and large language models on the way to AGI. Giannis, thank you so much for joining us today.

非常感谢你们的邀请。

Thank you so much for having me.

Giannis，你在 DeepMind 担任创始工程师超过十年，参与了一些真正定义行业的最著名项目。DeepMind 非常著名地开创了在游戏中构建 AI 的理念。你能分享一下当时 DeepMind 为什么选择从游戏开始吗？

Giannis, you have an incredible background having worked at DeepMind as a founding engineer for over a decade, starting with some of the most notable projects that have really defined the industry. DeepMind quite notably created this notion of building AI within games to start. Can you share a little bit more about why DeepMind chose to start with games at the time?

是的，DeepMind 是第一家从一开始就真正拥抱通用人工智能（AGI）概念的公司。他们有宏大的抱负，旨在构建能够匹配或超越人类智能的系统。所以大问题是，现在仍然是：如何构建 AGI？更重要的是，如何以一种允许有意义的研究和性能改进的方式来衡量智能？因此，使用视频游戏作为试验场的想法对 DeepMind 的创始人 Demis Hassabis 和 Shane Legg 来说很自然，因为 Demis 有游戏行业的背景，而 Shane 的博士论文将 AGI 定义为一个能够学习完成任何任务的系统。视频游戏提供了一个可控但复杂的环境，可以在其中探索和测试这些想法。

Yeah, so DeepMind was the first company to truly embrace the concept of artificial general intelligence, or AGI, from the outset. They had grand ambitions, aiming to build systems that would match or exceed human intelligence. So the big question was, and still is: how do you build AGI? And more importantly, how do you measure intelligence in a way that allows for meaningful research and performance improvements? So the idea of using video games as a testing ground came naturally to DeepMind's founders, Demis Hassabis and Shane Legg, because Demis had a background in the gaming industry and Shane's PhD thesis defined AGI as a system that could learn to complete any task. Video games provided a controlled yet complex environment where these ideas could be explored and tested.

到什么程度？你提到游戏提供了一个非常可控的环境。游戏在多大程度上代表或不代表现实世界？如果你在游戏中得到结果，你认为这会自然地推广到现实世界吗？

To what extent? You mentioned games provide a very controlled environment. To what extent are games representative or not of the real world? If you have a result in games, do you think that generalizes naturally to the real world or not?

我的意思是，游戏确实对开发 AI 很有价值，而且你实际上有几个例子。你可以看到，例如 PPO，目前用于 RLHF，是在 OpenAI Gym、MuJoCo 和 Atari 上开发的。同样，我们有 MCTS，即蒙特卡洛树搜索，是通过像双陆棋和围棋这样的棋盘游戏开发的。但与此同时，游戏也有许多局限性。现实世界是混乱的、无界的，比最复杂的游戏还要难啃得多。所以，尽管它们为你提供了一个有趣的试验台来开发新想法，但这绝对是有限的，并不能真正捕捉现实世界的所有复杂性。

So I mean, games have indeed been valuable for developing AI, and you actually have a few examples of that. You can see that PPO, for example, which is currently being used in RLHF, was developed using OpenAI Gym and MuJoCo and Atari. And similarly, we have MCTS, which stands for Monte Carlo tree search, and was developed through board games like Backgammon and Go. But at the same time, games have a number of limitations. The real world is messy, it's unbounded, and it's a much tougher nut to crack than even the most complex games. So even though they give you an interesting testbed to develop new ideas, it is definitely limiting and it doesn't really capture all the complexity of the real world.

好的，有趣。但你在游戏环境中开发的许多技术和算法，比如 PPO 等，这些在现实世界中也有应用。

Okay, interesting. But a lot of the techniques and algorithms that you've developed in a game environment, PPO, etc., these are used in the real world.

是的，PPO 实际上正是 ChatGPT 用于 RLHF 的方法。而 MCTS 用于 MuZero，MuZero 已经在现实世界中用于 YouTube 的视频压缩，它曾一度是特斯拉自动驾驶系统的一部分，还用于开发完全由 AI 控制的飞行员。所以，你可以看到像这样的方法被用于解决现实问题。

Yeah, so PPO was actually exactly what ChatGPT used for RLHF. And MCTS is used in MuZero, and MuZero has been used in the real world in things like video compression for YouTube, it was part of the self-driving system at Tesla at some time, and it was also used for developing a pilot that was completely controlled by AI. So yeah, you can see methods like that being used in the world to solve real problems.

有趣。Giannis，我记得 2017 年 AlphaGo 电影上映时，展示了 AlphaGo 与李世石那场不可思议的对局。你能带我们回到那个时刻，以及之前几年你构建 AlphaGo 的过程吗？为什么特别选择围棋作为重点游戏？

Interesting. Giannis, I remember back in 2017 when the AlphaGo movie came out and it featured the incredible game of AlphaGo against Lee Sedol. Can you take us back to that moment in time and maybe the years leading up to it as you're building AlphaGo? How was Go specifically chosen as the game to focus on?

我认为游戏一直是 AI 研究的基准。在围棋之前，有国际象棋，国际象棋在 90 年代末 IBM 的深蓝击败加里·卡斯帕罗夫时是一个重要的里程碑。尽管国际象棋和围棋是完全不同的游戏，围棋也绝对不同，但游戏一直是新 AI 方法开发的试验台，尤其是棋盘游戏。实际上，追溯到 AI 研究的早期，图灵和香农都开发过自己的国际象棋机器人版本。现在，关于围棋的问题是，它比国际象棋难得多。原因在于，几乎不可能定义一个评估方法或启发式。在国际象棋中，你可以看棋盘，数双方有多少兵，看这些兵的等级，然后得出谁赢谁输的结论。但在围棋中，没有这样的东西。它主要依靠人类直觉。如果你问一位围棋职业棋手他们如何知道一个局面是好是坏，他们会说，在下了这么多年棋之后，他们凭直觉就能看出这个局面比另一个好。所以问题实际上是如何将你直觉中的感觉编码到 AI 系统中，对吧？这正是为什么解决围棋长期以来被认为是 AI 研究的圣杯，这是一个看似几乎不可能但又触手可及的挑战。人们觉得他们实际上可以攻克它。而这正是 AlphaGo 在 2016 年所做的。它展示了两种新方法：深度学习和强化学习。因为在 2015 年、2016 年，我们现在认为深度学习和强化学习是成熟的技术，但当时它们才刚刚起步，是初出茅庐的新事物。大多数人持怀疑态度。

So I think games have always been a benchmark for AI research. Before Go, there was chess, and chess was a major milestone with IBM's Deep Blue defeating Garry Kasparov in the late 90s. And even though chess and Go are completely different games, and Go is definitely different, games have always acted as testbeds for the development of new AI methods, especially board games. Actually, going back to the earliest days of AI research, Turing and Shannon both worked on their own versions of chess bots. Now, the thing about Go is that it's a much harder problem than chess. The reason for that is because it's almost impossible to define an evaluation method, a heuristic. In chess, you can just look at the board, count the number of pawns each side has, see what the ranks of these pawns are, and then draw some conclusions on who is winning and why. But in Go, there's nothing like that. It's mostly human intuition. If you ask a Go professional player how they know whether a position is good or bad, they will say that after having played the game for so long, they can just see it in their gut that this is a better position than the other one. So it's actually a question of how do you encode the feeling in your gut into an AI system, right? This is exactly the reason why solving Go was considered the Holy Grail of AI research for a long time, and it was a challenge that seemed almost impossible but at the same time within reach. People felt that they could actually crack it. And this is exactly what AlphaGo did back in 2016. It showcased two new methods: deep learning and reinforcement learning. Because back in 2015, 2016, we now think of deep learning and reinforcement learning as mature technologies, but back then they were literally taking their first steps and were kind of the new kid on the block. Most people were kind of skeptical.

我记得读到过，围棋棋盘上的可能布局比宇宙中的原子还要多出许多数量级，这让我大为震撼，因为我从小下围棋。规则很简单，但我明白为什么它是圣杯。也许你能从技术角度解释一下 AlphaGo 是如何工作的，就像对五年级小学生解释那样？你提到强化学习和深度学习都参与了，我很想深入了解一下。

I remember reading that there are more configurations of the Go board than atoms in the universe by many orders of magnitude, and that blew me away because I grew up playing Go. It felt like such a simple game in terms of the rules, but I see why it was the holy grail. Maybe can you explain how AlphaGo worked technically, maybe explain it to me like I'm a fifth grader? You mentioned that both reinforcement learning and deep learning were involved. I'd love to peel that back a little bit.

当然。AlphaGo 有两个深度神经网络。神经网络是一个函数，它接收输入并产生输出，就像一个黑箱。我们并不确切知道它内部如何运作，只知道如果你用足够的数据训练它，它就能学会从输入到输出的映射。AlphaGo 有两个深度网络：策略网络和价值网络。策略网络会推荐最有可能的落子。它查看当前棋盘局面，然后说：‘基于当前局面，这是我建议你考虑下的位置列表。’它还有一个价值网络，查看棋盘局面后给出获胜概率，比如从当前局面开始你获胜的几率有多大。这就像直觉——它对局面好坏有自己的判断。有了这两个网络，你就可以在想象中模拟多局棋。你考虑最有可能的落子，然后是对手最有可能的落子，再用价值网络评估每一步。你可以使用一种叫极小化极大算法的方法，意思是我想赢，但我也知道对手想赢，所以我要选择能最大化我获胜概率的一步，同时知道对手也会试图最大化他们的获胜概率。如果你这样做并模拟一系列落子，就能得到最优行动。而最有效地进行这种想象、规划和搜索的方式，是使用一种树搜索方法，叫做蒙特卡洛树搜索（MCTS）。所以当人们提到 MCTS 时，他们指的就是这种启发式方法，用来决定考虑哪些未来局面，以便做出明智的决策。

Absolutely. So AlphaGo has two deep neural networks. A neural network is a function that takes something as input and produces something as output, and it's literally like a black box. We don't really know exactly how it does it, just that if you train it on enough data, it will learn the mapping from input to output space. AlphaGo had access to two deep networks: the policy network and the value network. The policy network suggests the most promising move. It takes a look at the current board position and says, 'Based on the current position, this is the list of moves I would recommend you consider playing.' It also had access to the value network, which takes a look at a board position and gives you a winning probability, like what are your chances of winning the game starting from this position. This is exactly the gut feeling—it had its own gut feeling on whether the position is good or bad. Once you have these two networks, you can play in your imagination a number of games. You consider the most promising moves, then your opponent's most promising moves, and then evaluate each move using the value network. You can use a method called minimax, which says that I want to win the game, but I also know my opponent wants to win, so I want to pick a move that maximizes my chances of winning, knowing that my opponent will try to maximize their chances. If you do that and simulate a bunch of moves, you can get the optimal action. The way to do this imagination, planning, and search in the most efficient way is by using a tree search method called Monte Carlo Tree Search (MCTS). So whenever people talk about MCTS, they literally mean this heuristic of how to choose which futures to consider so that you can make informed decisions.

强化学习和深度学习在构建 AlphaGo 中的作用在于，AlphaGo 是这两种方法的成功结合。策略网络最初是在大量人类棋局上训练的。你有许多职业棋手对弈的棋局，你考虑每个局面以及他们在那步下的棋，然后有一个深度网络试图预测那步棋。有了策略网络后，你需要获得一个价值网络。我们通过两种方式实现。首先，我们让策略网络与自己下棋，并使用强化学习来提高其棋力。我们使用了一种叫策略梯度的技术。策略梯度会查看棋局和结果。在最简单的版本中，它查看结果，对于所有导致胜利的落子，增加选择该落子的概率；对于所有导致失败的落子，降低未来选择该落子的概率。如果你对很多棋局重复这个过程足够长时间，就会得到一个改进的策略。有了这个改进的策略，你可以生成一个新的数据集，其中策略与自己下棋，你得到大量棋局，每个局面都知道最终赢家是谁。然后你可以用另一个网络——价值网络——让它根据当前局面预测棋局结果。所以网络学习到：如果我从这个局面开始，在当前策略下对弈，平均来说赢家是黑方或白方。这是价值网络的第一个版本，你可以将其与策略网络结合，在 AlphaGo 中使用。

The role of reinforcement learning and deep learning in building AlphaGo was that AlphaGo was a success of both methods. The policy network was initially trained on a large set of human games. You had many games played by human professionals, and you consider every position and the move they took at that position, then you have a deep network that tries to predict that move. Once you have the policy network, you need to obtain a value network. We did it in two ways. First, we took the policy network and had it play against itself, and we used reinforcement learning to improve its playing strength. We used a technique called policy gradient. Policy gradient looks at the game and then at the outcome. In the simplest version, it looks at the outcome, and for all moves that led to a win, it increases the probability of choosing that move; for all moves that led to a loss, it decreases the probability of that move being selected in the future. If you do that for many games and for long enough, you get an improved policy. Once you have this improved policy, you can generate a new dataset of games where the policy plays against itself, and you have a huge number of games where for each position you know who the final winner was. Then you can take another network, a value network, and have it predict the outcome of the game based on the current position. So the network learns that if I start at this position and play under my current policy, on average this is the player who wins—either black or white. This is the first version of a value network, and you can use it within AlphaGo by combining it with the policy network.

构建这个系统最大的挑战是什么？你们是如何克服的？

What were some of the biggest challenges in building this and how did you overcome them?

AlphaGo 不仅是一个研究挑战，更是一个工程奇迹。早期版本运行在 1000 个 CPU 和 176 个 GPU 上，而与李世石对弈的版本使用了 48 个 TPU。TPU 是第一批定制加速器，当时还非常原始，因为那真的是第一个版本。后来的加速器要好得多，也更稳定。系统必须高度优化，以最小化延迟、最大化吞吐量。我们必须为训练这些网络构建大规模基础设施，这是一个巨大的工程，需要许多有才华的人在不同方面协调努力。我前面提到的每一步——获得策略网络和价值网络——都必须以当时可用的极限规模来实现，并且实现方式要能让人们快速迭代想法并获得结果。所以，很多人，以前所未有的规模，在当时的可能性前沿工作。

AlphaGo was not just a research challenge but mostly an engineering marvel. The early versions ran on 1,000 CPUs and 176 GPUs, and the version that played against Lee Sedol used 48 TPUs. TPUs were the first custom accelerators, and they were really primitive back then because it was literally the first version. Now the later accelerators are much better and more stable. The system had to be highly optimized to minimize latency and maximize throughput. We had to build large-scale infrastructure for training these networks, and it was a massive endeavor that required a lot of coordinated effort from many talented individuals working on different aspects of the project. Each step I walked you through—obtaining the policy network and the value network—had to be implemented at the limits of what was available and possible back then in terms of scale, and it had to be implemented in a way where people could iterate ideas fast and get results quickly. So lots of people, scale at levels that hadn't been implemented before, working at the forefront of what was possible back then.

记得你曾分享过，这个项目启动的部分原因也是因为谷歌有 TPU，他们需要一个测试客户，而这正是 AlphaGo 项目的火花。这真是太不可思议了。DeepMind 团队有多大信心认为这会成功？你提到当时深度学习和强化学习还相对新颖，但 DeepMind 的创立正是基于这种信念。但你们当时是否认为能够取得这种超人类水平的结果，击败世界顶级围棋选手？这是一个可能成功的疯狂想法，还是团队有信心这会成功？

Remember you sharing one time that part of the reason this project came about also was because Google had TPUs that they needed a test customer for, and that was the spark for this AlphaGo project. So that's pretty incredible. How much conviction did the DeepMind team have that this was going to work? You mentioned that at the time deep learning and reinforcement learning were still relatively novel, but DeepMind was very much founded with that belief. But did you guys think that you were going to be able to have these superhuman level results, beating the top Go player in the world? Like, was it a crazy idea that might work, or did the team have conviction that this was going to work?

是的，一开始团队持谨慎乐观态度。AlphaGo 的主要开发者之一 Aja Huang 是一名强业余围棋选手，他在 AlphaGo 出现前已经研究围棋大约十年。我们还有一份关于电脑棋手的领先报告，可以看到 AlphaGo 明显比之前的任何系统都强。但围棋是一种复杂的游戏，我们总是有点担心 AlphaGo 是否真的像我们相信的那样好。所以我们实际上有信心，基于我们能测量和看到的一切，深度强化学习就是答案。但这些系统的特点在于，它们不像经典计算机那样你总是知道它们会产生相同的答案。它们是随机的，有创造性的。而且它们都有盲点，会像大语言模型那样产生幻觉。所以你需要真正推动它们，看看它们到底在哪里出错。而做到这一点的唯一方法就是让最优秀的人类与它们对弈。

Yeah, so at the start, the team had a cautious optimism. So one of AlphaGo's lead developers, Aja Huang, he is a strong amateur Go player, and he had been working on Go for like a decade before AlphaGo happened. And we also had like a lead report of a computer game of computer players, and you could see that AlphaGo was significantly stronger than anything that had come before. But Go is a complex game, and there was always a bit of worry about whether AlphaGo was truly as good as we believed. So we actually had the conviction that deep reinforcement learning is the answer, based on everything that we could measure and everything we could see. But that's the thing about these systems: they're not like classic computers where you just know that they always produce the same answer. They're stochastic, they're creative. And they all have blind spots, they hallucinate similarly to how LLMs hallucinate. So you need to really push them and see exactly where they break. And the only way you could actually do that is by having the best humans playing against them.

第 37 手，你能告诉我们那是什么吗？那是一个如此具有里程碑意义的落子，我想当时每个观看的人都被那一手搞糊涂了。当它发生时，你脑子里在想什么？

Move 37, can you tell us what that was? It was such a monumental move, and I think everyone watching at the time was confused by that move. What was going on in your head when that happened?

是的，我的意思是，第二局对阵李世石的第 37 手确实是一个壮观的时刻，它向世界展示了 AlphaGo 具有创造力，并证明了人工智能可以想出即使是顶尖人类选手也未曾考虑过的策略。起初，我仍然记得我们认为 AlphaGo 犯了一个错误，它实际上产生了幻觉，做了它本不想做的事情。但后来证明这是一个 brilliant 的非常规着法，突显了系统对游戏有深刻的理解，系统实际上具有创造力，它可以想到人们以前没有想到的事情。

So yeah, I mean, move 37 in game two against Lee Sedol was literally a spectacular moment in the sense that it showcased to the world that AlphaGo has creativity, and it demonstrated that AI could come up with strategies that even top human players hadn't considered. So at first, I still remember that we thought that AlphaGo made an error, that it actually hallucinated, did something that it didn't mean to do. But then it turned out to be a brilliant unconventional move that underscored that the system had a deep understanding of the game, that the system actually had creativity, it could think of things that people hadn't thought of before.

我想带大家看比赛中的另一个关键着法。我想是在第四局。此时我支持李世石，因为我觉得这个可怜的家伙需要赢一局。第 78 手，我认为 AlphaGo 犯了一个错误，李世石发现了它。李世石在比赛中发现的弱点是什么？

I want to take us to another key move in the game. I think it was in game four. At this point I was rooting for Lee Sedol because I was like, this poor guy needs to win a game. Move 78, I think AlphaGo made a mistake and Lee Sedol found it. What was the weakness there that Lee found during the game?

是的，没错。我的意思是，李世石在第四局中的第 78 手确实证明了人类的创造力。第 78 手出乎意料，让 AlphaGo 措手不及。最初，AlphaGo 基于其评估将其误解为一个错误，并认为自己实际上在赢，所以没有做出适当回应。这突显了系统的一个盲点。所以这场比赛表明，虽然像 AlphaGo 这样的系统非常强大，但同时也存在脆弱性，并且仍有可以进一步改进的领域。

Yeah, exactly. So I mean, Lee Sedol's move 78 in game four was literally a testament to human ingenuity. Move 78 was unexpected and caught AlphaGo off guard. Initially, AlphaGo, based on its evaluations, misinterpreted it as a mistake and thought that it was actually winning, so that's why it didn't respond appropriately. And this kind of highlighted a blind spot in the system. So the game showed that while systems like AlphaGo are extremely powerful, at the same time they still have vulnerabilities, and there were still areas where you could further improve it.

但你如何改进这样的东西？你需要给它展示更多那种人类创造性着法的数据，还是如何修复和修补这些盲点？

But how do you go about improving something like that? Do you need to show it a lot more data of that type of human creative move, or how do you go about fixing and patching those blind spots?

是的，有趣的是，在与李世石的比赛结束后，我们建立了一个基准，试图量化和衡量 AlphaGo 所犯的错误，这些盲点。然后我们尝试了多种方法来改进算法，以解决这些问题。结果发现，消除它们最有效的方法就是继续我们正在做的事情，但规模更大、做得更好。所以只是改变模型架构，我们切换到了带有两个输出头的深度残差网络，并且我们有了更大的网络，在更多数据上训练，然后转向 AlphaZero 和更好的算法。这让我们不再有任何幻觉。所以从某种意义上说，就是规模和数据的扩展，这些一直是 AI 领域众所周知的秘诀，正是它解决了问题。

Yeah, so it's actually interesting that by the end of the games with Lee Sedol, we put together a benchmark where we tried to quantify and have a way of measuring the mistakes that AlphaGo makes, these blind spots. And then we tried a number of approaches to improve the algorithm so that we could solve these issues. And what happened is that actually the most effective way of getting rid of them was just to do what we were doing, but at higher scale and better. So just change the architecture of the model, we switched to a deep ResNet with two output heads, and we also had a bigger network trained on more data, then moved to AlphaZero and better algorithms. And that kind of made it so that we didn't have any hallucinations anymore. So in a way, just scale and data, things that are always the well-known recipe in the field of AI, is exactly what solved it.

更高质量的数据，或者特别是来自优秀职业棋手、最佳职业棋手的数据，有多大意义？还是说任何数据都可以？

How much did higher quality data, or maybe specifically data from great professional players, the best professional players, make a meaningful difference? Or was it just any data?

不，对我们来说重要的是我们通过自我对弈解决了问题。是的，我们实际上可以接触到世界上最有能力的围棋选手，我们只是用它来生成最高质量的对局，然后在这些对局上训练。所以我想我们不需要人类专家，因为我们内部有一个专家。它不是人类。

No, for us what mattered was that we solved it using self-play. Yeah, so we actually had access to the most competent Go player in the world and we just used it to generate the best quality games, and then we just trained on these games. So I guess we didn't need to have human experts because we had an expert in house. It wasn't human.

有趣，太棒了。我想继续谈谈从 AlphaGo 到 AlphaZero 的进展。你刚才提到了自我对弈的概念。AlphaZero 之所以强大，是因为它从零开始学习如何玩游戏，完全通过自我对弈，没有任何人类干预。你能分享更多关于它是如何工作的，以及为什么这很重要吗？

Interesting, amazing. Well, I'd love to move on to the progression from AlphaGo to AlphaZero. You talked a little bit about this notion of self-play just now. AlphaZero was powerful because it learned how to play the game from scratch, entirely from self-play without any human intervention. Can you share more about how that worked and why that was important?

AlphaZero 是一个游戏规则改变者，因为它完全通过自我对弈从零开始学习，没有任何人类数据。这与 AlphaGo 有重大区别，因为 AlphaGo 严重依赖人类专家对局。所以发生了两件事：首先，AlphaZero 简化了训练过程，并且表明人工智能可以通过纯粹的自对弈从零达到超人类表现。这使得它可以应用于一系列新领域，这些领域由于缺乏足够的人类数据而无法触及。但我认为更重要的是，我们看到 AlphaZero 也解决了 AlphaGo 在幻觉、盲点和鲁棒性方面的所有问题。所以 AlphaZero 是一个更好的方法，句号。

So AlphaZero was a game changer because it learned entirely from scratch through self-play without any human data. And this was a major departure from AlphaGo, because AlphaGo, as I said, relied heavily on human expert games. So two things happened: first of all, AlphaZero managed to simplify the training process, and also showed that AI could go from zero to superhuman performance just purely by playing against itself. And that allowed it to be applicable to a whole range of new domains that were out of reach because there weren't enough human data for it. But I think the more important thing is that we saw that AlphaZero also solved all the issues that AlphaGo had in terms of hallucinations, blind spots, and robustness. So AlphaZero was a better method, full stop.

你向五年级学生解释过 AlphaGo 的工作原理。你会告诉他们什么？

And you explained how AlphaGo worked to a fifth grader. What would you tell them?

五年级学生将是关键的区别，技术上你在 AlphaZero 中实现的。所以 AlphaZero，就像 AlphaGo 一样，使用策略网络和价值网络以及 MCTS。所以在这方面，它与 AlphaGo 完全相同。关键区别在于训练：AlphaZero 从随机权重开始，通过与自己下棋来学习。通过与自己下棋，它迭代地提高性能。AlphaZero 背后的主要思想是，每当你有一组权重、一组策略和价值网络，然后将它们与搜索结合，你就会得到一个更好的棋手。你提高了性能，你变得更强。所以这意味着我们可以实际使用这种机制来改进模型策略，即原始策略。这就是我们在强化学习中所谓的策略改进算子：每当你有一个现有策略，做一些神奇的事情，然后得到一个更好的策略，你就可以将这个策略蒸馏回初始策略。重复这个过程，你就有了一个强化学习算法。我认为这正是今天人们试图用两星或合成数据做的事情。这正是这个想法：我如何拿一个策略，用它做点什么——规划、搜索、计算，无论什么——然后推导出一个更好的策略，然后我可以模仿并蒸馏回原始策略。所以这正是 AlphaZero 所做的：它使用 MCTS 搜索来产生一个更好的策略，然后它获取轨迹，在新的更好的轨迹上训练策略和价值网络，并重复这个过程，直到收敛到专家级的围棋选手。

The fifth grader would be the key difference technically that you implemented with AlphaZero. So AlphaZero, just like AlphaGo, uses a policy network and value network along with MCTS. So in that respect, it's exactly the same as AlphaGo. The key difference is in training: AlphaZero starts with random weights and learns by playing games against itself. By playing games against itself, it iteratively improves its performance. The main idea behind AlphaZero is that whenever you take a set of weights, a set of policy and value nets, and then you combine them with search, you end up with a better player. You increase your performance, you become a stronger player. So what that meant is that we can actually use this mechanism to improve the model policy, the raw policy. This is what we call in reinforcement learning a policy improvement operator: whenever you can take an existing policy, do something magic, and come up with a better policy, then you can take this policy and distill it back to the initial policy. Repeat this process, and you have a reinforcement learning algorithm. I think this is exactly what people are trying to do today with, like, two-star or synthetic data. This is exactly the idea: how can I take a policy, do something with it—planning, search, compute, whatever it is—and derive a better policy, which I can then imitate and distill back to the original policy. So this is exactly what AlphaZero is doing: it uses MCTS search to produce a better policy, then it takes its trajectories, trains the policy and value network on the new better trajectories, and repeats this process until it converges to an expert-level Go player.

这很迷人且反直觉，从没有专业棋手权重的起点开始实际上是一个更好的起点。

That's fascinating and counterintuitive, that starting without the weights that you would have from professional-level players is actually a better starting place.

我认为，AI 智能体和游戏领域的巅峰是通过 MuZero 实现的，它甚至是 AlphaZero 本身的进步。而这也是你成为该游戏联合负责人或负责人之一的地方。AlphaZero 显然令人印象深刻，因为它使用了自我对弈，但它也需要被告知环境的动态或游戏规则。MuZero 将我们带到了下一个层次，无需被告知游戏规则，它掌握了许多不同的游戏：围棋、国际象棋等等。你能分享一下 MuZero 是如何工作的，以及为什么这特别有意义吗？

The epitome of AI agents and games has achieved, I think, via MuZero, which is the progression even from AlphaZero itself. And it's also where you became one of the co-leads or one of the leads of the game. AlphaZero was obviously impressive because of self-play, but it also needed to be told the environment's dynamics or the rules of the game. MuZero takes us to the next level without needing to be told the rules of the game, and it mastered quite a few different games: Go, Chess, and many others. Can you share a little bit about how MuZero worked and why this was particularly meaningful?

当然。所以 AlphaZero，正如你所说，在国际象棋、围棋、将棋等游戏中取得了巨大成功，这些游戏我们实际上可以访问游戏规则，拥有一个完美的世界模拟器。但对完美模拟器的依赖使得它难以应用于现实世界的问题。现实世界的问题通常杂乱无章，缺乏明确的规则，很难为它们编写一个完美的模拟器。这正是 MuZero 试图解决的问题。MuZero 当然掌握了围棋、国际象棋和将棋等游戏，但它也掌握了更具视觉挑战性的游戏或困难游戏，比如 Atari，而且它无需访问模拟器就能做到。它只是学习如何构建一个内部世界模型，然后以类似于 AlphaZero 的方式使用这个内部模型。它通过使用基于模型的强化学习来实现这一点。这意味着你可以获取智能体生成的一些轨迹，然后尝试学习一个模型，一个关于世界如何运作的预测模型。这实际上与现在像 Sora 这样的方法试图做的非常相似，它们获取 YouTube 视频，并尝试通过从一帧开始预测未来帧会发生什么来学习一个世界模型。MuZero 试图做到这一点，但它以一种与生成模型不同的方式，它只尝试对解决强化学习问题重要的事情进行建模。所以它试图预测未来的奖励是什么，未来状态的价值是什么，未来状态的策略是什么——只有 MCTS 中需要的东西。但基本原理保持不变：如何基于轨迹学习一个模型，一旦有了这个模型，你就可以将其与搜索结合，获得超人的表现。当然，你总是可以将这两个问题解耦，让模型从野外数据中单独训练，然后将其与 MuZero 结合。我们发现，当时考虑到我们模型的局限性和较小的规模，将这两者保持在一起，只让模型预测对规划重要的事情，而不是试图建模所有事情，更有意义，因为你有点触及模型容量所能承受的极限。

Absolutely. So AlphaZero, as you said, was a massive success in games like Chess, Go, Shogi, in games where we actually had access to the game rules, where we had access to a perfect simulator of the world. But reliance on the perfect simulator made it challenging to apply to real-world problems. Real-world problems are often messy and lack clear rules, and it's really hard to write a perfect simulator for them. So that's exactly what MuZero tried to solve. MuZero masters the games of course like Go, Chess, and Shogi, but it also masters more visually challenging games or games that are hard, like Atari, and it does that without having access to the simulator. It just learns how to build an internal simulator of the world and then uses this internal simulator in a way similar to what AlphaZero was doing. It achieves that by using model-based reinforcement learning. What that means is that you can take a number of trajectories generated by an agent and then try to learn a model, a prediction model of how the world works. This is actually quite similar to what methods like Sora are trying to do now, where they take YouTube videos and try to learn a world model by predicting, based on starting from one frame, what's going to happen in future frames. MuZero tries to do exactly that, but it does it in a way different from generative models in the sense that it tries to only model things that matter for solving the reinforcement learning problem. So it tries to predict what the reward is going to be in the future, what the value of future states is, what the policy for future states is—only things that you need within your MCTS. But the fundamentals kind of remain the same: how do you learn a model based on trajectories, and once you have this model, you can combine it with search and get superhuman performance. Of course, you can always decouple the two problems and have the model trained separately from data out in the wild, and then combine that with MuZero. We found that back then, given the limitations of our models and the smaller sizes, it made more sense to keep those two together and only have the model predict things that matter for planning instead of trying to model everything, because you're kind of hitting the limits of what the model capacity could take.

是否可以认为不仅 Sora 采用了相同的方法，也许其他世界模型或其他机器人基础模型也是如此？

Is it right to assume that not only Sora takes the same approach, but maybe other world models or other robotics foundation models?

是的，所以任何试图构建世界运作模型然后用于规划的东西都属于 MuZero 类方法。所以你可以在 YouTube 视频上训练它，可以在来自机器人的输入上训练它，可以在任何环境上训练它。你甚至可以将语言模型视为文本模型的一种形式。它们对文本进行建模，但文本的问题是模型有点琐碎——当你试图预测下一个词是什么时，并没有太多伪影发生，对吧？

Yeah, so anything that tries to build a model of how the world works and then use that for planning is within MuZero-like methods. So you can train it on YouTube videos, you can train it on inputs coming from robots, you can train it on any environment. You can even think of language models as a form of models of text. They model text, but the thing about text is that the model is a bit trivial—there aren't many artifacts happening when you're trying to predict what the next word is going to be, right?

你看到 MuZero 背后的想法被用于游戏之外或混乱的现实世界环境中了吗？

Have you seen the ideas behind MuZero be used outside gameplay or in messy real-world environments?

是的，我的意思是，正如我所说，AlphaZero 和 MuZero 是相当通用的方法。在化学领域有一些科学社区，所以有 AlphaChem。在量子计算中，一些人尝试在优化中使用 AlphaZero，他们只是改编了 AlphaZero，因为它在规划和解决优化问题方面非常强大。同时，MuZero 被整合到特斯拉自动驾驶系统的一个版本中，这在他们的人工智能日上有所报道。它也被用于 YouTube 作为推荐算法，我认为目前仍在用。但我认为现在还处于早期阶段。

So yeah, I mean, as I've said, AlphaZero and MuZero are quite general methods. There are a number of scientific communities in chemistry, so there's AlphaChem. In quantum computing, some people tried to use AlphaZero in optimization, where they just adapted AlphaZero because it was really powerful in planning and solving optimization problems. At the same time, MuZero was incorporated in a version of Tesla's self-driving system, it was kind of reported in their AI Day. And it was also used, and I think it's currently being used, within YouTube as a recommendation algorithm. But I think it's early days.

我们想多聊一聊智能体中的强化学习。你之前提到，强化学习和深度学习在 2015 年还是新概念，在 2017、2018、2019 年及之后才真正流行起来，然后很大程度上因为 GPT 和其他成果而被 LLM 掩盖了。但现在强化学习又回来了。你认为这是为什么？

We'd love to talk a little bit more about reinforcement learning in agents. You alluded earlier to the fact that reinforcement learning and deep learning back in 2015 were new ideas, they really grew in popularity 2017, 2018, 2019 onwards, and then they were overshadowed by LLMs largely because of GPT and everything else that came out. But now reinforcement learning is back. Why do you think that is the case?

是的，首先，LLM 和多模态模型确实为 AI 带来了令人难以置信的进步。这些模型异常强大，能够执行一些真正令人印象深刻的任务。但它们也有一些根本性的局限，其中之一就是人类数据的可用性。人们一直在谈论数据墙，以及一旦高质量数据用尽会发生什么。这正是强化学习大放异彩的地方。强化学习之所以出色，是因为它不单纯依赖预先存在的人类数据。相反，强化学习利用智能体自身生成的体验来提高性能。这种自我生成的体验使强化学习能够学习和适应，甚至是在人类数据稀缺或不存在的情况下。因此，如果你以正确的方式定义奖励和强化学习问题，你实际上可以用算力换取智能。你可以达到类似于 AlphaGo 的状态：我们投入更多算力——让网络更大、使用更多棋局——就能得到更好的棋手，而且这是确定性的：你总能得到更好的棋手。我想这正是我们在合成数据流水线中想要达到的状态。目前，我们在 LLM 的缩放定律中已经看到了这一点：如果有更多数据和更大的模型，就可以预测性能的提升。但一旦人类数据用尽，如何继续前进？合成数据就是答案。实际上，获得高质量数据以改进模型的唯一途径就是通过某种形式的强化学习。我这里把强化学习作为一个广义术语，定义为任何通过试错来学习的方法。

Yeah, I mean, first of all, LLMs and multimodal models have indeed brought incredible progress to AI. These models are exceptionally powerful and can perform some truly impressive tasks. But they have some fundamental limitations, and one of them is the availability of human data. People keep talking about the data wall and what happens once you run out of high-quality data. This is exactly where reinforcement learning shines. Reinforcement learning excels because it doesn't rely solely on pre-existing human data. Instead, reinforcement learning uses experience generated by the agent itself to improve its performance. This self-generated experience allows reinforcement learning to learn and adapt, even to scenarios where human data is scarce or non-existent. So if you define the reward and the reinforcement learning problem in the right setting, you can effectively exchange compute for intelligence. You can get to a point similar to where we were with AlphaGo, where the moment we threw more compute at it—making the networks bigger, using more games—we literally got a better player, and it was deterministic: you always get a better player. I guess this is exactly where we want to be with synthetic data pipelines. Currently, we have that with the scaling laws in LLMs: if you have more data and bigger models, you can predict an improvement in performance. But once you run out of human data, how do you keep going? Synthetic data is the answer to that. The only way you can actually get high-quality data to improve your model is via some form of reinforcement learning. And I'm keeping reinforcement learning as a really kind of blanket term here, defining it as anything that learns through trial and error.

你认为强化学习是如何被引入 LLM 世界的？你之前提到了 QAR。在封闭式游戏中，你有明确定义的策略和价值函数。在混乱的真实世界环境或 LLM 世界中，这如何运作？

How do you think reinforcement learning is being brought into the LLM world? You mentioned QAR earlier. In a closed-form game, you have a clearly defined policy and value function. How does that work in a messy real-world environment or the LLM world?

我的意思是，有两种不同类型的混乱真实世界。一种是如果你试图构建一个控制器之类的东西，那是一个非常混乱的环境。另一种是如果你在数字空间操作，我个人相信数字 AGI 会比机器人 AGI 更早实现。这是有原因的：你可以控制环境，而像计算机这样的环境就是数字世界。所以即使它混乱且有噪声，它仍然是受控的；从那个意义上说，它不是真实的物理世界。现在，关于如何引入强化学习：我们在 DeepMind 常说，你有问题，也有解决方案。强化学习的问题设定是：我如何获取一个模型、一个策略，并生成合成数据？或者我学习，我找到一种通过与环境交互、通过试错来改进这个策略的方法。这就是强化学习的问题设定。然后还有解决方案空间，其中有价值函数和强化学习方法。我认为可以从过去十年发展的经典强化学习方法中汲取很多灵感，但你必须将它们调整到 LLM 的新世界。比如，你可以从这样的想法开始：如果我有一个策略，然后我进行规划，考虑可能的未来场景，然后我有办法评估哪个更好，那么我可以选取最好的那些，并让模型模仿这些更好的结果。这是一种改进策略的方法。在经典 RL 框架中，你通过使用策略网络和价值网络来实现。在新世界中，你可以通过拥有一个奖励模型或让 LLM 对自己给出的输出提供反馈来实现。

I mean, there are two different types of messy real world. There is if you try to build a controller or something, that's a really messy environment. And then if you operate in the digital space, personally I believe that digital AGI will happen much earlier than robotics AGI. There is a reason for that: you have control over the environment, and the environment like a computer is the digital world. So even though it's messy and noisy, it's still contained; it's not the real flying world in that sense. Now, in terms of how do you bring reinforcement learning: we used to say at DeepMind that you have the problem and you have the solution. The problem setting of reinforcement learning is: how do I take a model, how do I take a policy, and generate synthetic data? Or I learn, I find a way to improve this policy by interacting with the environment by trial and error. That's the reinforcement learning problem setting. And then there's the solution space where you have value functions and reinforcement learning methods. I think there's a lot of inspiration to draw from classical reinforcement learning methods that were developed in the past decade, but you have to adjust them to the new world of LLMs. Methods like you start trying to do that by taking the idea that if I have a policy and then I do planning, I consider possible future scenarios, and then I have a way to evaluate which one is better, then I can take the best ones and ask the model to imitate these better ones. This is a way of improving the policy. In the classic RL framework, you do that by using a policy and a value network. In the new world, you'll do that by having a reward model or asking your LLM to give you feedback on an output it gave you.

你之前也提到了一些关于合成数据的内容。我认为有些人对合成数据非常乐观，有些人则更怀疑。我也相信合成数据在某些结果和成功可能是确定性的领域更有用。你能分享一下你对合成数据作用的看法，以及你有多乐观吗？

You also talked a little bit about synthetic data earlier. I think some folks are very bullish on synthetic data and some folks more skeptical. I also believe that synthetic data is more useful in some domains where outcomes and successes are perhaps deterministic. Can you share a little bit about your perspective on the role of synthetic data and how bullish you are on it?

是的，我的意思是，合成数据是我们无论如何都必须解决的问题。这不是你是否乐观的问题；这是一个我们必须找到解决办法的障碍。我们会用尽数据。人类能产生的数据是有限的。此外，这些系统开始采取行动并从自己的错误中学习也很重要。所以我们需要找到一种方法让合成数据发挥作用。现在，人们尝试过最天真的方法：直接让模型生成一些东西，然后尝试在上面训练。当然，他们发现会出现模式崩溃，而且这并不能开箱即用。但新方法从来都不是开箱即用的。你需要投入其中，花时间，真正思考最佳方法。所以我非常乐观，我们一定会找到改进这些模型的方法。我认为有很多方法，比如 two-star 和 equivalence，在如今人们不再像以前那样分享研究突破的新世界里，这些方法可能隐藏在公司的商业秘密背后。

Yeah, I mean, synthetic data is something we have to solve one way or another. It's not about whether you're bullish or not; it's an obstacle we have to find a way around. We will run out of data. There is only so much data humans can produce. Also, it's important that these systems start taking actions and learning from their own mistakes. So we need to find a way to make synthetic data work. Now, what people have done is they've tried the most naive approach where you just take the model, it produces something, and you try to train on that. Of course, they've seen that there's mode collapsing and it just doesn't work out of the box. But new methods never work out of the box. You need to invest in it, take your time, and really think of the best way of doing it. So I'm really optimistic that we will definitely find ways to improve these models. I think there are a number of methods out there, like the two-star and the equivalence, that in the new world where people don't really share their research breakthroughs the way they used to, are probably hidden behind some company trade secrets.

我想问一下推理和新科学发现。你认为如果数据足够多，仅仅通过扩展 LLM 就能自然产生这些能力吗，还是你认为推理能力……

I'm going to ask about reasoning and novel scientific discoveries. Do you think that can kind of naturally come out of just scaling LLMs if you have enough data, or do you think that kind of like the ability to reason and...

要知道，提出全新的想法需要做强化学习，并在推理时投入更深层的算力。所以我认为你需要强化学习来获得更好的推理能力，因为这也关乎数据的分布。互联网上有大量数据，但与此同时，你并不总是拥有合适类型的数据。你没有那种某人进行推理并详细解释其推理过程的数据。你有一些这样的数据，而且令人难以置信的是，模型竟然已经学会了模仿它。但如果你想提高这种能力，那么你需要通过强化学习来实现。你需要向模型展示这种新兴能力如何通过让它生成数据、与环境交互，并告诉它什么时候做对了、什么时候做错了来改进。所以是的，我认为强化学习绝对是答案的一部分。

You know, coming up with net new ideas requires doing reinforcement learning and deeper compute at inference time. So I think you need reinforcement learning to get better reasoning, because it's also about the distribution of data. You have a lot of data out in the wild on the internet, but at the same time you don't always have the right type of data. You don't have data where someone reasons and explains their reasoning in detail. You have some of it, and it's incredible that the models have actually managed to pick it up and just imitate it. But if you want to improve on that capability, then you need to do that through reinforcement learning. You need to show the model how this emerging capability can be improved by having it generate data, interact with the environment, and tell it when it's doing something right and when it's not. So yeah, I think reinforcement learning is definitely part of the answer for that.

AlphaGo、AlphaZero 和 MuZero 是我们构建过的最强大的智能体。你能分享一下从它们身上获得的经验教训如何与我们今天构建 AI 智能体的方式相关吗？

AlphaGo, AlphaZero, and MuZero are the most powerful agents we've ever built. Can you share a little bit about how some of the lessons and learnings unlocked from them are relevant to how we're pursuing building AI agents today?

是的，所以我认为 AlphaGo 和 MuZero 从根本上改变了我们对 AI 智能体的方法，因为它们凸显了规划和规模的重要性。在我看来，如果你看看不同模型的图表以及它们如何扩展，你会发现 AlphaGo 和 AlphaZero 在当时是非常超前的。它们是异类。你有算力的扩展，然后 AlphaZero 独自屹立。所以它表明，如果你能扩展并真正推动这一点，那么你就能获得不可思议的结果。同时，它也表明你不必仅仅训练；你还可以在推理时、在测试时通过使用规划获得更好的性能。我认为在不久的将来，我们会越来越多地看到这一点。这些方法在做出任何决定之前会开始更多地思考、更多地规划。所以我想说，这是 AlphaGo、AlphaZero 和 MuZero 的遗产：基本原则——规模重要，规划重要。这些方法可以真正解决那些我们认为极其复杂或超出我们自身能力的问题。我们今天在大型语言模型中观察到的类似问题，我们在当时也看到过。早在 2016 年，我们就看到这些模型可能会产生幻觉，但同时它们也具有创造力。它们会提出我们未曾想到的解决方案，但它们也可能有盲点、产生幻觉，或者容易受到对抗性攻击。我想现在每个人都知道神经网络存在这些问题。所以我认为这些是从这一系列工作中得出的主要教训。

Yeah, so I think AlphaGo and MuZero have fundamentally transformed our approach to AI agents because they highlight the importance of planning and scale. In my opinion, if you look at the charts of different models and how they scale, you can see that AlphaGo and AlphaZero were kind of really ahead of their time. They were outliers. You had the scaling of compute, and then you had AlphaZero standing on its own. So it showed that if you can scale and really push on that, then you can get incredible results. At the same time, it also showed that you don't just have to train; you can also have better performance during inference, during test time, by using planning. I think this is something we will start seeing more and more in the near future. These methods will start thinking more, planning more before making any decisions. So I'd say that this is the heritage of AlphaGo, AlphaZero, and MuZero: the basic principles that scale matters, planning matters. These methods can really solve problems that we thought were insanely complex or beyond what we can solve on our own. Similar problems to the ones we observe today with large language models are things we saw back then. Back in 2016, we saw that these models can hallucinate, but at the same time they are creative. They come up with solutions we hadn't thought of, but they can also have blind spots, hallucinate, or be susceptible to adversarial attacks. I guess everyone knows now that neural networks suffer from these. So I think these are the main lessons drawn from this line of work.

你认为这一系列工作为领域未来留下的最大开放性问题是什么？

What do you think are the biggest open questions from this line of work for the field going forward?

所以主要问题是：我们曾拥有 AlphaGo，并且我们成功构建了极其稳健和可靠的系统，它们总是能以最高水平下围棋。它们会持续位居排行榜榜首，再也不会输。AlphaGo Master 实际上在在线比赛中与 60 人对弈，并且每一局都赢了。所以这是一种一致、稳健和可靠的模式。我认为这正是我们现在基于 LLM 的系统所缺失的。有时它们能做好，有时不能。你不能信任它们。我们有一些很棒的演示，但它们每两次才发生一次，甚至每十次才发生一次。其余九次，它们就迷失了方向，什么也没做。所以我认为我们需要做的是找到一种方法，让这些基于 LLM 的智能体变得像我们之前拥有的 AlphaGo、MuZero 和 AlphaZero 一样稳健。这是一个新的开放性问题：你究竟如何做到这一点？

So the main question is: we had AlphaGo and we managed to have these insanely robust and reliable systems that would always play Go at the highest possible level. They would consistently be top of the leaderboard and never lose again. AlphaGo Master actually played against 60 people in online matches and literally won every single one. So there was this pattern of consistent, robust, and reliable. I think this is exactly what we're missing now with these LLM-based systems. Sometimes they get it, sometimes they don't. You cannot trust them. We have some amazing demos, but they happen once every two times or even once every ten times. The remaining nine times, they just lose their way and don't do anything. So I think what we need to do is find a way to make these LLM-based agents equally robust to the ones we had with AlphaGo, MuZero, and AlphaZero. This is the new open question: how do you actually do that?

我们想听听你对当今更广泛生态系统的一些看法。你提到了人们目前正在研究的几个核心问题：数据墙问题，可能最终会在 2028 年左右出现；规划的概念，这是 AI 智能体需要改进的领域；以及稳健性和可靠性。你能分享一下你认为整个领域需要解决的一些领域吗？你最兴奋的是哪些，以帮助我们实现真正获得我们想要的 AI 智能体的愿景？

We'd love to move into some of your thoughts on the broader ecosystem today. You've touched on a few really core problems that people are working on right now: the data wall problem that will hit eventually perhaps by 2028 or so, the idea of planning as an area that AI agents need to get better at, and the idea around robustness and reliability. Can you share a little bit about some of these areas that you think the whole field needs to solve that you are most excited about to help us unlock this vision of really getting to the AI agents that we want?

是的，我的意思是，我想在列表中添加另一个。所以我认为另一个主要挑战是如何提高这些模型的上下文学习能力。你如何确保这些系统能够即时学习并快速适应新环境？这是我认为将非常重要并且在未来几年内会实现的另一件事。实际上，上下文学习是指系统可以通过少量示例提示来学习如何执行新任务。它看到几个例子，然后即时学习如何适应新环境，学习如何使用提供给它的新工具。这不仅仅是存储在权重中的所有知识，它还可以通过与真实世界互动、与环境互动来获取新知识。所以我认为这是另一个目前有很多工作在进行、并且在未来几年内将取得惊人进展的领域，我对此感到非常兴奋。所以是的，总结一下，我认为规划很重要，上下文学习很重要，还有可靠性。实现可靠性的最佳方法是确保这些模型以某种方式知道如何从错误中恢复。

Yeah, I mean, I'd like to add another one to the list. So I think another major challenge is how do we improve the in-context learning capabilities of these models. How do you make sure that these systems can learn on the fly and adapt to new contexts quickly? This is another thing that I think is going to be really important and will happen in the next couple of years. Actually, so in-context learning is the idea that a system can learn how to do a new task with few-shot prompting. It sees a few examples and on the fly learns how to adapt to the new environment, learns how to use the new tools provided to it. It's not just all the knowledge stored in its weights, but it can also acquire new knowledge by interacting with the real world, interacting with the environment. So I think this is another place where there is a lot of work happening at the moment and will have amazing progress in the next couple of years, and I'm really excited about that. So yeah, to recap, I think planning is important, in-context learning is important, and reliability. The best way to achieve reliability is to ensure that these models somehow know how to recover from their mistakes.

既然你已经开始了自己的创业之旅，你如何看待初创公司可以在哪些领域与大型研究实验室竞争？你又是如何激励自己踏上这段旅程的？

Now that you've kind of embarked on your own entrepreneurial journey, how do you think about the areas where startups can compete against the big research labs? And how do you kind of motivate yourself for that journey?

是的，这对我来说是一个新世界，但同时也不是那么新，因为我加入 DeepMind 时它就是一个初创公司。我是最早期的员工之一，所以亲眼目睹了这一切。为初创公司工作的好处之一是灵活性和专注力。每个人都非常投入，行动迅速，并且对我们要构建的东西有清晰的焦点。构建本身是人们最重要的动力。这是初创公司相对于成熟企业的一大优势。同时，适应新发现和新技术也更容易。你不会被一些已有的解决方案或产品所束缚，这些产品可能因为带来大量收入而不愿被淘汰。作为初创公司，你没有这样的枷锁；你可以快速行动，创新，打破常规。它还能让你利用那些大型实验室难以触及的开源资源。而且你也没有大机构常有的繁文缛节。

Yeah, it's a new world for me, but at the same time it's not that new because when I joined DeepMind it was literally a startup. I was one of the first employees, so I actually saw that firsthand. One of the benefits of working for a startup is the agility and focus. Everyone really cares, everyone moves really fast, and there's a clear focus on what we want to build. Building is the most important motivation for people. That's one of the big advantages startups have over more established businesses. At the same time, it's easier to adapt to new findings and new technologies. You're not tied to some pre-existing solutions or products that you don't want to deprecate because they bring a lot of revenue. As a startup, you have no such chains; you can move fast, be innovative, and break conventions. It also allows you to leverage open-source resources that are out of touch for big labs. And you don't have the red tape that big places tend to have.

我喜欢你有时用的那个词：主线任务与支线任务。

I love the term that you use sometimes: main quest versus side quest.

是的，就是有一个主要焦点的概念。在大机构和大实验室里，人们同时进行许多不同的项目。通常，他们有一个大家都在做的主线任务，还有许多较小的支线任务来支持这个更大的任务。但通常，这些支线任务得不到那么多资源或领导层的关注。所以它们往往只是锦上添花。

Yeah, it's the idea of having a main focus. In big places and big labs, they have many different projects that people are working on. Usually, they have the main quest that everyone is working on, and there are many smaller side quests that feed into the bigger quest. But typically, they don't get as many resources or as much focus from leadership. So they tend to be the trophy.

在更广泛的领域中，你最钦佩哪些最具标志性的项目？也许还有哪些你敬仰的最有影响力的研究者？

In the broader field, what are some of the most defining projects that you admire the most? And maybe who are some of the most influential researchers that you admire?

当然。我的 AI 研究之旅始于 2012 年，我见证了一些里程碑。让我列出我认为过去 12 年 AI 的主要里程碑。第一个是 AlexNet。这是第一篇展示深度学习是答案的论文。当时它更像是一种好奇，但现在大多数人都相信深度学习是答案的一部分。然后是 DQN。我有幸参与其中并亲眼目睹。它是由我的朋友 Vlad Mnih 开发的，是第一个展示可以将深度学习与强化学习结合，在复杂环境中达到人类水平或超人类表现的系统。接着是 AlphaGo。我很幸运能参与其中。它表明规模和规划是非常重要的要素，如果做对了，就能在极其复杂的环境中取得巨大成功。AlphaFold 是另一个，同样来自 DeepMind。它表明这些方法不仅用于解决游戏；它们可以让世界变得更美好，改善医疗保健，实现科学发现。然后是 ChatGPT，它将 AI 带给每个人，让大众都能接触到。现在每个人都知道 AI 是什么了。它让我解释自己的工作变得容易多了。最后是 GPT-4。我认为 GPT-4 是 AI 最新的重大进步，因为它表明 AGI 只是几年内的事；它触手可及。现在许多人相信我们离 AGI 只有几年之遥，这要归功于 GPT-4 的惊人突破。至于我钦佩的人：首先是 David Silver。他是我的博士导师，也是我在 DeepMind 的导师。他是一位了不起的研究员。他领导了 AlphaGo 和 AlphaZero。他对强化学习领域有着执着的奉献精神。他可能是我认识的最聪明的人之一。第二位是 Ilya Sutskever。他是 OpenAI 的联合创始人。我有机会在 AlphaGo 早期与他合作过。他对 Scaling 方法的执着以及推动系统能力边界的决心令人瞩目。他确保了 GPT-3 和 GPT-4 的实现。我对他非常尊敬。

Absolutely. I started my AI research journey back in 2012, and I've seen some milestones. Let me list what I think are the main milestones in AI over the past 12 years. The first is AlexNet. It was the first paper that showed deep learning is the answer. Back then it felt like a curiosity, but now most people are convinced deep learning is part of the answer. Then it was DQN. I had the pleasure of working on it and seeing it firsthand. It was developed by a friend of mine, Vlad Mnih, and it was the first system that showed you can combine deep learning with reinforcement learning to achieve human-level or superhuman performance in complex environments. Then it was AlphaGo. I was lucky to work on it. It showed that scale and planning are really important ingredients, and if you do that right, you get huge success in an incredibly complex environment. AlphaFold is another one, again by DeepMind. It showed that these methods are not just for solving games; they can make the world a better place, improve healthcare, and realize scientific discoveries. Then ChatGPT brought AI to everyone, made it accessible to a broad audience. Everyone knows what AI is now. It made my life of explaining my job much easier. Finally, GPT-4. I think GPT-4 is the latest big advancement in AI because it showed that AGI is a matter of years; it's within reach. Many people now believe we are a few years away from AGI, and that's because of the incredible breakthrough that GPT-4 was. In terms of people I admire: first, David Silver. He was my PhD supervisor and my mentor at DeepMind. He's an incredible researcher. He led AlphaGo and AlphaZero. He has a guiding dedication to the field of reinforcement learning. He's probably one of the smartest people I know. Second, Ilya Sutskever. He was a co-founder at OpenAI. I had the opportunity to work with him a little in the early days of AlphaGo. His commitment to scaling methods and pushing the boundaries of what systems can achieve is remarkable. He made sure GPT-3 and GPT-4 happened. I have immense respect for him.

谢谢你的分享。让我们以一些快问快答来结束。首先，你认为 AI 的下一个重大里程碑是什么，比如说未来一年、五年和十年？

Thank you for sharing that. Let's close out with some rapid-fire questions. First, what do you think will be the next big milestones in AI, let's say in the next one, five, and ten years?

我认为未来五到十年，世界将变得不同。我真的相信这一点。在未来几年，我们将看到模型成为强大且可靠的智能体，能够真正独立执行任务。我认为 AI 智能体将在各行各业大规模采用，尤其是在科学和医疗领域。所以我对 AI 的未来感到非常兴奋。我最兴奋的是 AI 智能体——能够为你实际执行任务的系统。这正是我们在 Perfection 正在构建的。

I think the next five to ten years, the world will be a different place. I really believe that. In the next few years, we'll see models becoming powerful and reliable agents that can actually independently execute tasks. I think AI agents will be massively adopted across industries, especially in science and healthcare. So I'm really excited about what's coming in AI. What I'm most excited about is AI agents—systems that can actually do tasks for you. That's exactly what we're building at Perfection.

你认为我们会在哪一年通过 SWE-bench 的 50%门槛？

What year do you think we'll pass the 50% threshold on SWE-bench?

我认为我们距离 SWE 智能体达到 50%的门槛还有一到三年，距离达到 90%还有三到五年。

I think we are one to three years away from the 50% threshold for SWE agents, and three to five years from achieving 90%.

虽然进展惊人，但我认为我们仍然需要可靠的智能体来实现这些里程碑。在研究中，确实很难做出精确的预测。你认为我们什么时候会碰到 Scaling LLM 的数据墙？你认为强化学习的所有研究是否足够成熟，能够维持我们的进步斜率，还是说在试图弄清楚碰到数据墙后会发生什么时，会出现一段停滞期？

While progress is amazing, I think we still need reliable agents to hit these milestones. And it's really when it comes to research, it's hard to make precise predictions. When do you think we'll hit the data wall for scaling LLMs? And do you think all the research in RL is mature enough to keep up our slope of progress, or do you think there will be a bit of a lull as we try to figure out what happens when we hit the wall?

所以我认为，根据我读到的内容，在文本方面我们至少还有一年才会碰到数据墙。然后我们还有这些额外的模态，这可能会为我们多争取一年。而且我认为我们在使用合成数据方面处于一个非常好的起点。所以在未来几年里，我们会解决合成数据的问题。所以我认为我们不会真正碰到数据墙，或者即使碰到了，也没人意识到，因为我们已经有了新的方法。

So I think, based on what I've read, we have at least one more year for text just like before we hit the wall. And then we have these extra modalities which might actually buy us maybe a year extra. And I think we are in a really good place starting to use synthetic data. So in the next few years we'll just figure out the synthetic data problem. So I think that we won't really hit a wall, or we'll hit the wall but no one realized it because we have new methods in place.

你认为 LLM 会有它们的 AlphaGo 时刻吗？如果有，是什么时候？

Do you think LLMs will have their AlphaGo moment? And if so, when?

我认为 LLM 的 AlphaGo 时刻是 ChatGPT 的首次发布，它们展示了过去十年的力量和进步。我觉得它们还没有迎来的是 AlphaZero 时刻。那是一个更多算力直接转化为智能提升而无需人类监督的时刻。我认为这一突破仍在酝酿之中。

I think LLMs had their AlphaGo moment with the initial release of ChatGPT, where they showed the power and the progress made over the past decade. I feel like what they haven't had yet is their AlphaZero moment. And that's the moment where more compute directly translates to increased intelligence without human supervision. And I think this breakthrough is still on the horizon.

你认为那会在什么时候发生？

When do you think that will happen?

我认为这会在未来五年内发生。

I think it's going to happen in the next five years.

哇，太棒了。Giannis，非常感谢你加入我们，带我们回顾了 AlphaGo、AlphaZero、MuZero 的精彩历史，你在 DeepMind 的个人旅程，以及整个行业今天正在解决的许多核心研究问题，比如数据、构建可靠性、鲁棒性、规划和上下文学习。我们对你正在帮助构建的未来感到非常兴奋，你也在推动这个领域向前发展。非常感谢你，Giannis。

Wow, amazing. Giannis, thank you so much for joining us and taking us through the awesome history of AlphaGo, AlphaZero, MuZero, your own journey through DeepMind, and many of the core research problems that the whole industry is tackling today around data and building for reliability, robustness, and planning and in-context learning. We're really excited for the future that you're helping us build, and that you're pushing forward the field as well. So thank you so much, Giannis.

非常感谢你的邀请。

Thank you so much for having me.