David Silver 与 Lasse Espeholt：为经验时代打造超级学习者

我们集体从头脑中这团灰色的糊状物开始。我们不知怎么地一开始一无所知，却不知怎么地发现了今天人类所知的一切。这是集体发现的过程——语言、技术、科学、数学等等。我们学到的一切，都是我们自己发现的。这是 AI 至今尚未破解的不可思议之事。对我来说，这就是实现超级智能的本质：一个能够永远不断学习和发现知识的系统。

Collectively, we started with this kind of gray mush in our head. We somehow started out knowing nothing at all and somehow discovered everything that humans know today. It was this process of collective discovery—of languages, technology, science, mathematics, and all the rest of it. Everything we learned, we discovered for ourselves. That's an incredible thing that no one has cracked yet in AI. To me, that is the essence of what it will take to get to superintelligence: a system that can just keep learning and discovering knowledge forever.

我非常高兴邀请到 Dave Silver 和 Le Espol，Ineffable Intelligence 的两位联合创始人。顺便说一句，名字很棒。有趣的是，Dave，大约一百集前我们刚开始做节目时，你是我第一个冷邮件邀请来上 Training Data 的人，现在终于请到你，感觉像是一个圆满的轮回。所以感谢你的加入。

I'm thrilled to have Dave Silver and Le Espol with us, two of the co-founders of Ineffable Intelligence. Great name, by the way. Fun fact, Dave, you were the first person I cold emailed to come on Training Data when we started the show a hundredish episodes ago, and it feels like a full circle moment to finally have you on the show. So thank you for joining us.

谢谢。

Thank you.

Dave，我们从你开始。你 2009 年的博士论文是关于将强化学习应用于极其困难的围棋游戏。你首次向世界证明，在某一领域实现超越人类的机器智能是可能的，我认为你正是激励了当前整整一代强化学习和深度学习研究者的那个人。所以感谢你对这个领域发展的贡献。我很好奇，回想那个时候，你学到了什么？你当时坚信什么？在过去 5 年左右 AI 的变化中，什么保持不变，什么改变了？

Dave, let's start with you. Your 2009 PhD was on reinforcement learning applied to the devilishly difficult game of Go. You demonstrated for the world for the first time that superhuman in one domain machine intelligence was possible, and I think you were really the figure that inspired the entire current generation of reinforcement learning and deep learning researchers. So thank you for your contributions to where the field is. I'm curious, thinking back to that point in time, what did you learn? What did you have conviction on, and what has remained the same and what has changed as AI has changed over the last 5 years or so?

是的，谢谢，很高兴来到这里，我很高兴在一百集之后终于来了。很高兴来到这里。我想我从读博开始进入围棋领域时就一直坚信，解决 AI 的正确方法是通过从经验中学习，通过那些自己发现事物的系统，通过强化学习不断变得更好，而这将导致智能乃至超级智能的出现。我想我们在围棋中证明了这一点，但这个信念贯穿了我的整个职业生涯。现在我们正在 Ineffable 应用这个信念，试图证明超级智能确实可以通过从经验中学习的系统涌现出来。至于不同之处，虽然我们没有走 LLM 世界所走的人类数据路径，但我确实认为从中可以学到很多有趣的教训。特别是，一个在大量数据上训练的系统竟然可以很快地在很少的数据上微调，从而完全改变其行为，这对我来说绝对是个惊喜。我认为这改变了我的思维方式，它表明如果你把事情做到正确的规模，很多看起来极其困难或可能非常缓慢的事情——一旦你经历了足够的数据，之后那些缓慢的事情就可以变得非常快。我认为这是一个我事先并不明显的教训，直到它发生了。

Yeah, thank you and pleasure to be here and I'm glad I finally made it after these hundred episodes. It's a pleasure to be here. So I suppose I always had this conviction starting my PhD and going into the game of Go that the right approach to solve AI was through learning from experience, through systems that discovered things for themselves, that were learning through reinforcement learning to get better and better and better, and that would be the thing which would lead to the emergence of intelligence and ultimately superintelligence. So I suppose we were able to show that in the game of Go, but it's a conviction that has held true throughout my career. And here we are now applying that conviction at Ineffable, trying to show that indeed superintelligence can emerge all the way through systems that learn from experience. And in terms of things which are different, although we are not pursuing the human data path that the LLM world has pursued, I do think there's lots of interesting lessons to learn from that. In particular, it was definitely a surprise to me how a system that was trained on a massive amount of data could then be very quickly fine-tuned on a very small amount of data to kind of completely change its behavior. That was something which I think changed my mindset in terms of just suggesting that if you get things at the right scale, a lot of things which seem like incredibly hard or like they might be very slow—once you've experienced enough data, after that the things that were slow can become very fast. I think that's a lesson that wasn't obvious to me in advance until it happened.

好的，我们稍后会回到这一点。你本可以在 DeepMind 内部运行这个项目，在那里你可能比几乎任何人都能获得更多的算力。你决定正确的做法是成立一个新实验室。你为什么这么做？

Okay, we're going to come back to that later. You could have run this program inside DeepMind, where you could have probably had more access to compute than just about anybody. You decided the right thing to do was to start a new lab. Why'd you do that?

是的。首先，我在 DeepMind 度过了非常美好的时光。我喜欢在那里的岁月，我对那里的人怀有极大的敬意，并且仍然与他们相处得很好。但归根结底，我坚信我们如何能够超越当今 AI 的水平，一路走向超级智能，而我所相信的路径实际上并不经过基于 LLM 的产品作为中间点。当我真正思考这一点时，我思考得非常认真。我意识到我对此感受如此强烈，以至于这值得不仅仅是一个专注于基于 LLM 产品的组织的一角，而值得由一个精英 AI 实验室来代表，真正全力以赴地投入这种方法，100% 地代表它。一旦我意识到这一点，创建这家公司就变得非常自然。另一种看待方式是，世界上已经有成千上万甚至数十万真正有才华的研究人员都在追求完全相同的方法，真的只是在 LLM 上加倍投入，包括预训练、后训练、编码以及那种 LLM 路径。我会成为那个追求这种方法的人吗？那是我的时间和精力的最有效利用吗？最终，我意识到还有另一盏路灯，目前没有人在下面寻找，而如果我真的把努力放在那里，就更有可能对该领域产生巨大影响，因为如果这成功了，它将改变整个 AI。所以我认为这种认识也意味着，这是通过这种方法对该领域产生影响的最佳方式。

Yeah. So look, first of all, I had a fantastic time at DeepMind. I loved my years there and I have a huge amount of respect for the people there and still got on with them very well. But at the end of the day, I have a really strong belief about how we can go beyond where we are today with AI and all the way to superintelligence, and the path that I believe in doesn't actually go through LLM-based products as an intermediate point. And when I really thought about that, I thought about it really hard. I realized that I felt that so passionately that this deserved to be more than just the corner of an organization that's focused on LLM-based products and deserved to be represented by an elite AI lab that really just gives this approach its all and represents it 100%. And once I realized that, it felt very natural to create this company. I suppose another way to look at it is that the world already has tens of thousands if not hundreds of thousands of really talented researchers all pursuing exactly the same approach, really just doubling down on LLMs with pre-training, post-training, coding, and that kind of LLM path. Am I going to be the one person that's pursuing that approach? Is that the most effective use of my time and my energy? In the end, I realized that there's this other lamppost that right now no one is looking under, and where if I actually put my effort there, has a much higher chance of really having a massive impact on the field, because if this works, it's going to change the whole of AI. So I think that realization also meant that this was something which just means that it's the best way to have impact upon the field through this approach.

另外，你有一个不能浪费的公司名字。

Plus, you had a company name that couldn't go to waste.

没错。是的。

That's right. Yeah.

你想说说 Ineffable 这个名字的由来吗？

You want to say where the name Ineffable came from?

是的。所以，我想 Ineffable 这个名字来自这样一个想法：知识可以如此不可思议——就像超级智能有一天会发现的东西，这种知识将如此令人难以置信、不可思议，它将是某种深刻惊人的东西，也许如此深刻，以至于我们甚至无法用人类语言来表达。所以真的，就像，你知道，这与另一条路径一致，那条路径不经过基于人类语言的方法，而是构建自己发现事物的系统，并有可能发现令人难以置信地深刻、最终不可言说的知识。

Yeah. So, I suppose the name Ineffable came from this idea that knowledge could be so incredible—like the things which superintelligence will one day discover, this knowledge, it's going to be so unbelievable, incredible, and it's going to be something which is profoundly amazing and perhaps so profound that we can't even express it in human language. And so really, it's like, you know, in line with this other path that doesn't go through human language-based approaches, but instead builds systems that discover things for themselves and has the possibility to discover knowledge that's unbelievably profound and ultimately ineffable.

我还应该提一下，我们当时还有一个秘密的第二个理由，就是这个想法：它真的必须能行得通。如果你真的只是加倍押注于从经验中学习，这是一种必须能行得通的方法。所以你可以说它不可能失败。所以它就是不可言喻的（ineffable）。所以就是这样，这就是那个秘密理由。

I should say there was also a secret second reason that we used, which was this idea that it really has to work. If you really just double down on learning from experience, it's an approach that just has to work. So you could say that it can't possibly fail. So it's ineffable. So there you go, there's the secret reason as well.

太棒了，最好的实验室名字。嗯，Lei，你呢？你之前在 Google，你和 Dave 在不同的组，在 DeepMind 的深度学习组，你花了多年时间在 GPU 上扩展强化学习。是什么让你决定离开 DeepMind，又是什么吸引你来共同创办 Ineffable？

Amazing, best lab name. Um, Lei, what about you? You were at Google, you were in a different group than Dave, in the deep learning group at DeepMind, and you spent years scaling up reinforcement learning on GPUs. What made you decide to leave DeepMind and what drew you to co-found Ineffable?

对。嗯，我 2015 年 1 月加入 DeepMind，实际上我大概在 2016 年开始和 Dave 一起工作，当时我在做扩展强化学习的工作。所以就像你说的，我当时在深度学习团队，他领导强化学习团队，但我们一起做一个内部项目。我实际上离开过 DeepMind 两次。我在 2018 年夏天离开 DeepMind 去了 Google Brain。后来 Google Brain 和 DeepMind 合并了，我又回来了。但后来我在 Dave 离开前一年又离开了 DeepMind。我第一次听说 Ineffable 其实是通过我们后来的一位投资人联系我，问我怎么看 Dave。嗯，我的第一反应是，该死，我错过了一个机会。我应该主动联系他的。但我记得 Dave 往往是房间里最聪明的人之一。和那位投资人同步后不久，他们实际上联系了我。所以很棒。是的，我真的很喜欢和 Dave 一起工作。我早就认识 Wojtek 和 Chris，我觉得这个团队很棒，我也觉得在欧洲有这样的机会太棒了。还应该说的是，我们一起研究过一些早期将强化学习和深度学习结合的方法，并试图扩大规模。还应该说的是，Lassa，顺便说一句他非常谦虚，但他实际上研究了一些最初的方法，这些是最早尝试真正扩大强化学习规模并证明可以在大规模上做到的方法，以及并行化强化学习的方法和构建可扩展架构的方法。如果你看看今天这个领域的发展，实际上人们今天使用的所有想法都可以追溯到 Lassa 当年做的那些原创工作。这就是扩展 RL 的所有根源。所以他是第一个搞明白的人。

Right. Yeah, so I joined DeepMind in January 2015, and actually I started working a bit with Dave around 2016, where I was doing this work on scaling up reinforcement learning. So as you said, I was part of the deep learning team and he was leading a reinforcement learning team, but we were working on an internal project together. I actually left DeepMind twice. So I left DeepMind in summer 2018 and went to Google Brain. Then eventually Google Brain and DeepMind merged and then I was back again. But then I left DeepMind again like a year before Dave left. And the first time I heard about Ineffable was actually through one of our eventual investors reaching out to me asking what I thought about Dave. Well, my first reaction was, damn, I missed out on an opportunity. I should have reached out to him. But I recall Dave as more often than not being one of the smartest people in the room. And shortly after that sync with that investor, they actually reached out to me. So it was great. Yeah, so I really love to work with Dave. I knew Wojtek and Chris from back in the day and I thought the team was great and I also thought it's amazing to have an opportunity like this in Europe. It should be said as well that we worked together on some of the early approaches to combining reinforcement learning with deep learning and trying to scale things up. And it should be said that Lassa, who by the way is very modest, but he actually worked on some of the original approaches that were the first attempts to actually scale up reinforcement learning and show that it could be done at large scale, and ways to parallelize reinforcement learning and ways to build out architectures that were scalable. And if you look today at the way the field's gone, actually all of the ideas that people use today kind of hail back to those original work that Lassa did back in the day. That's where all the roots come from of scaled up RL. So he was the first to figure it out.

太棒了。我们应该让你们互相介绍，而不是你们都太谦虚不好意思自夸。好，我们来谈谈 Ineffable 和你们在构建的东西。你们公司有一个很美的使命宣言。你们想构建一个超级学习器（superlearner），并与超级智能进行第一次接触。你能为我们定义这两件事吗？什么是超级学习器，第一次接触又是什么样的？

Wonderful. We should have you guys introduce each other instead of you're both too humble to brag about yourselves. Okay, let's talk about Ineffable and what you're building. You have a beautiful mission statement as a company. You want to build a superlearner and make first contact with superintelligence. Can you define both of those things for us? What is a superlearner and what does making first contact look like?

是的。所以对我来说，超级学习器是一种在它能学到的深度和广度上都具有惊人能力的东西。所以在深度维度上，它应该能够从一无所知开始，一路发现远超人类所知的东西。一个类比是人类集体，我们从脑子里这种灰色的糊状物开始。我们不知怎么地从一无所知开始，不知怎么地发现了今天人类所知的一切，这是一个集体发现的过程，包括语言、技术、科学、数学等等。我们学到的一切，都是我们自己发现的。并不是有人下来给了我们一张蓝图，告诉我们这应该怎么运作，也不是有人下来给了我们一个万亿元素的数据集，说这就是这些东西应该是什么样子。我们实际上是自己搞明白的，这是 AI 中还没有人破解的不可思议的事情。所以对我来说，这就是达到超级智能的本质，一个可以永远学习和发现知识的系统。第二个要素是广度，我们希望它能进行这种不可思议的发现，一直到不可思议的事情，并且应该能够把它投入到任何经验流中。所以无论是模拟世界、数字世界还是物理世界，无论它被投入到什么中，它都应该能够不断地学习、学习、再学习。所以这就是梦想。我认为如果我们破解了这一点，那么对我来说，它会产生一种超级智能，就像遇到魔法一样。它将是如此深刻惊人，能够做出美妙的事情。

Yeah. So to me a superlearner is something which has incredible capacity both in its depth of what it can learn and in its breadth. So in the depth dimension, it should be something that can start from not knowing anything at all and can go all the way through to discovering things that go way beyond what humans know. So an analogy to this would be the way that humans collectively, we started with this kind of gray mush in our head. We somehow started out knowing nothing at all and somehow discovered everything that humans know today was this process of collective discovery of languages and technology and science and mathematics and all the rest of it. Everything we learned, we discovered it for ourselves. It wasn't that someone came down and gave us a blueprint of how this should work and it wasn't that someone came down and gave like a trillion element data set of, oh, this is what these things should look like. We actually figured it out for ourselves and that's an incredible thing that no one has cracked yet in AI. And so to me that is the essence of what it will take to get to superintelligence, like a system that can just keep learning and discovering knowledge forever. So the second element is breadth, that we want something which can do this incredible discovery all the way to incredible things and it should be possible to drop it into any stream of experience. So whether it's a simulated world or a digital world or a physical world, whatever it's dropped into, it should be something that can just learn and learn and learn. So that's the dream. I think if we crack this, then to me it gives rise to a superintelligence that will be like encountering magic. It will be something which is so profoundly amazing that it will be able to do wonderful things.

当我们第一次遇到它时，会是什么感觉？还是说这是那种你无法提前预测的事情？

What will it feel like when we first encounter it? Or is it one of these things where you can't predict ahead of time?

我认为会非常惊人。我认为我们会体验到惊奇。从我自己经验中唯一能说的是，当我们在公认狭窄得多的领域达到这一点时发生了什么。但当我们在棋盘游戏中达到这一点时，当我们在围棋中达到超人水平时，它给这个领域带来了惊奇。看到 AlphaGo 下棋的围棋棋手们，他们惊叹不已，见证这种围棋下法对他们来说是一种快乐的体验。当我们在国际象棋中达到超人水平时也是一样，棋手们非常兴奋。想象一下远远超越这一点，但在发现不可思议的事情、新的科学领域、我们从未梦想过的新型技术方面。我的意思是，那将是不可思议的。

I think it will be amazing. I think we will have an experience of wonderment. And the only thing I can say from my own experience is what happened when we got to this point in admittedly much narrower fields. But when we got there in board games, when we got to the point of superhuman play in the game of Go, it brought wonderment to the field. The Go players who saw AlphaGo playing, they were amazed and it was a joyful experience for them to witness this kind of Go playing. Same happened when we got to superhuman play in chess and the chess players, they were so excited. And just imagine going far beyond that but in things which are discovering incredible things, new fields of science, new types of technology that we've never even dreamed of. I mean it will be incredible.

那么当前的范式，这些带有强化学习的大型语言模型似乎至少能解决千禧年问题并做出某种科学发现。这算超级学习器吗？还是说能做出新颖科学发现的东西和你所说的超级学习器有什么区别？

So the current paradigm, these large language models with reinforcement learning seem to be at least knocking out millennium problems and making some sorts of scientific discoveries. Does that count as a superlearner or what's the difference between something that can make novel scientific discoveries versus a superlearner in your sense of the term?

是的，你看，我认为 LLM 正在做很多伟大的事情，我对所取得的成就非常尊重。

Yeah, so look, I think LLMs are doing a lot of great things and I have a lot of respect for what's being achieved.

我所说的超级学习者，首先是指能够完全自主发现新知识的东西。就目前采取的那些方法来看，我们看到的仍然需要大量人类在环的努力才能取得突破，还没有完全自主。而且它应该适用于真实世界的具身领域，让智能体能够从物理经验中学习。我们距离那一步还非常非常遥远，而且完全不清楚 LLM 和基于语言的方法是否真的是实现那种知识的正确载体。用语言来思考这些事情可能根本不是正确的范式，也许 LLM 中使用的所有这些人类数据反而在拖后腿。所以我对已经取得的进展非常尊重，但我认为我们离真正的超级学习者——那种能自主发现一切的东西——还差得很远。

What I mean by a super learner is something which first of all discovers completely autonomously. And I think what we've seen so far, at least with these kind of approaches that have been taken, is there's a lot of human-in-the-loop effort that's required to make these breakthroughs. It's not fully autonomous yet. And it's also something which should be applicable to real-world embodied domains where the agent can learn from its physical experience. And we're a long, long, long way off that, and it's not at all clear that LLMs and language-based approaches are even the right vehicle to achieve that kind of knowledge. Thinking about these things in terms of language might not even be the right paradigm, and it might just be that all of this human data that's being used in LLMs is kind of weighing things down. So a lot of respect for some of the progress that's made, but I don't think we're anywhere close to this true super learner of something which can autonomously just discover everything for itself.

对你来说，实际构建这个东西是什么样的？我们投资了 Giannis，他原来在你的 AlphaGo 团队，他告诉我们，你的一个独特之处在于，你既能拥有极其宏大的愿景，又能让团队扎根于非常实际的研究里程碑。我很想了解，宏大的愿景，在日常中是什么样子，你在实现什么？你现在每天是怎么度过的？

What does building this look like in practice for you? And you know, we backed Giannis who was on your original AlphaGo team and he told us that, you know, one of the things that's distinct about you is you can have these incredibly ambitious visions, but you also ground the team in very practical research milestones. And I'd love to understand big big big vision. What does this look like in the day-to-day and what are you implementing? How do you spend your days right now?

是的，好问题。日常来说，我想有很多艰难的技术工作，像 Lassa 这样的人在构建整个技术栈，这意味着我们有一个最先进的强化学习栈，能让我们大规模运行、以最优方式吸收数据、能够学习，并拥有不断学习的系统。然后在此基础上，我们谈到了超级学习者，我们正在构建的另一个方面是环境，能够进行超级学习者式的发现一切，比如模拟环境，在其中超级学习者可以发现我们所说的智能的一切。我们称这些为“熔炉”。所以这些环境让智能体有可能做出不可思议的发现，有可能表达和学习我们认为是智能的一切。我们一直在投入一定的精力来创建这些环境。所以这是构建超级学习者、进行研究使超级学习者变得不可思议并超越强化学习现有水平的结合，既包括工程也包括研究，以及创建并将超级学习者放入这些熔炉中，看到不可思议行为的涌现。至于扎根，你问如何在日常中扎根。我们将其扎根于里程碑，这些里程碑类似于人类历程中发生的事情，我们发现了不可思议的事物，所以这些成为我们的智能体重新发现人类在知识发现过程中经历的重要时刻的自然里程碑。

Yeah, great question. So day-to-day, I suppose there's a lot of hard technical work with people like Lassa to build the whole stack that means we've got a state-of-the-art reinforcement learning stack that allows us to run things at massive scale, inhale data in the optimal way, be able to learn and have systems that keep learning and learning and learning. And then on top of that, so we've spoken about the super learner, the other aspect which we're building is we're building out environments which are capable of super learners like discovering everything, like simulated environments in which a super learner can discover everything that we mean by intelligence. We call those crucibles. So they're environments in which it's possible for agents to make incredible discoveries, in which it's possible to express and learn everything that we think of as intelligence. And so we have been putting a certain amount of effort into creating these environments. And so it's a combination of building out the super learner, doing the research to make super learners incredible and beat the state-of-the-art of what's possible with reinforcement learning, both in terms of the engineering and the research that happens, and creating and placing the super learners into these crucibles to see the emergence of incredible behaviors. And in terms of the grounding, you asked about how to ground this in the day-to-day. So we ground it in milestones, and the milestones are analogous to what happens with the human journey, where we discovered incredible things, and so those become natural milestones for our agents to rediscover the great moments that humans went through in our discovery of knowledge.

你认为智能体首先会重新发现什么？那将是你的第一个“哦，这起作用了”的时刻？

What do you think the agents will rediscover first when that's going to be your first moment of oh this is working?

我想我们会感到惊讶。我认为我们在强化学习方法上与许多其他领域的不同之处在于，我们不会用面包屑引导特定的学习目标。目前很多人用 LLM 做强化学习的方式是，哦，智能体应该学习这个问题，然后这个问题，再这个问题，沿着那条路径，我们牵着智能体的鼻子走向特定的期望结果。我们不采取这种方法。我们给智能体一些简单的目标，要求它发现会让我们惊讶和惊叹的东西。所以，原则上智能体能够发现各种事物，我们不一定事先知道那些发现会是什么，见证这段旅程将会非常令人兴奋。所以，我希望感到惊讶，我想这就是你问题的答案。

I imagine we're going to be surprised. I think one thing which is different about the way we take to doing reinforcement learning to a lot of the rest of the field is that we're not breadcrumb trailing the particular objective to be learned about. A lot of the way people do RL with LLMs at the moment is to say oh you know the agent should be learning about this problem and then this problem and this problem and after following that trail we're kind of pulling the agent by its nose towards a particular desired outcome. We're not taking that approach. We're giving the agent some simple objectives and asking it to discover things that will surprise us and amaze us. So, you know, the agent is in principle capable of discovering all kinds of things and we don't necessarily know in advance what those discoveries will be and it's kind of just going to be exciting to witness that journey. So I hope to be surprised I think is the answer to your question.

有趣。所以你的意思是，不是明确给智能体一个目标去重新发现语言或建造计算器或建造计算机，而是给它其他更开放的目标，智能体可能会发现，实际上最有效或最好的方式来完成那个目标，可能是建造一些我们人类已经建造的工具。这是一种说法吗？

Interesting. So what you're saying is instead of explicitly giving the agent the goal to go rediscover language or build a calculator or build a computer, you're going to give it other more open-ended goals and the agent may discover that the most efficient way to actually or the best way to actually accomplish that goal may be to build some of these tools that we as humans have already built. Is that one way of putting it?

这是一种说法。我认为另一种说法是，人们在强化学习中经常想，天哪，我如何想出一个奖励函数，完美地指定我想发现的这个非常复杂的东西，比如我如何精确指定我所说的科学，或者如何精确指定我所说的有趣的数学。但有一种替代方案，就是给智能体一些非常直接可优化的东西，但在一个非常复杂、有很多东西要学的环境中。事实证明，如果你看看自然界中世界运作的方式，对非常简单、易于描述的量的追求和优化，往往会产生极其难以描述的复杂性和涌现。例子就是进化，它只是优化适者生存或某种适应度信号，而对适应度的追求产生了我们在生命中所看到的所有复杂性。在人类历程中，对人类相对简单目标的追求，产生了人类最终获得的所有复杂行为以及我们最终拥有的所有知识。所以涌现的复杂性远大于起始点。并不是进化说，嘿人类，这是科学的概念，你需要达到最复杂的科学。它其实要简单得多。

That's one way of putting it. I think another way to say it is people often think in RL, oh my god, how do I come up with a reward function that perfectly specifies this really complicated thing that I want to discover, like how can I specify exactly what I mean by science or how can I specify exactly what I mean by interesting maths. But there's an alternative, and the alternative is to give the agent something really straightforward to optimize, but in a really complex environment where there's a lot to learn about. And so it turns out that if you look at the way the world has worked in nature, the pursuit and optimization of a very simple, easy-to-describe quantity can often give rise to incredibly hard-to-describe complexity and emergence. So examples of that would be evolution, which just optimizes survival of the fittest or some kind of fitness signal, and the pursuit of fitness gave rise to all of the complexity that we see in life. In the human journey, the pursuit of relatively simple goals that we as humans have gave rise to all of the complex behaviors that humans ultimately acquired and all of the knowledge that we eventually ended up with. So the emergent complexity is much greater than the starting point that it was given. So it wasn't like evolution said, look, hey humans, here's this idea of science and you need to get to the most complex science that's out there. It was really something much simpler.

描述这个过程的一种方式是：人类在追求目标的过程中，开始更多地协调。在协调的过程中，我们发现沟通很有帮助。在沟通的过程中，我们开始交易。在交易的过程中，我们发现基础算术很有用。然后我们发现工具有用。接着我们发现需要一些基础科学来理解工具如何运作，这催生了更复杂的技术。理解技术又需要更复杂的数学，如此循环。所以，在追求相对简单的目标时，这种不可思议的复杂性涌现了出来。如果我们能把这个过程做对，事情就会起飞，带来难以置信的成果。

A way to describe this might be that humans, in pursuit of our goals, started coordinating a bit more. In the process of coordinating, we discovered that it was helpful to start communicating. In the process of communicating, we started trading. In the process of trading, we discovered that basic arithmetic was useful. Then we discovered that tools were useful. Then we discovered that some basic science was required to understand how the tools would work, which gave rise to more complex technology. Understanding the technology required more complex maths, and so forth. So all in the pursuit of relatively simple goals, this incredible complexity emerged. If we can get that process right, things are going to take off and lead to unbelievable things.

这很迷人。你们智能体的奖励机制是否类似于人类生物学和进化，比如努力延续到下一代？你们的奖励函数是什么样的？

It's fascinating. Are the rewards for your agents similar to human biology and evolution, like trying to make it to the next generation? What does a reward function look like for you all?

是的，我们一直在尝试多种不同的奖励。所以并不是说我们对这个问题只有一个答案。但我认为我们没做的一件事是：精心设计一个能精确描述我们所说的科学、技术或语言的奖励。我们期望智能体自己去发现事物。如果我们无法解决这个问题，作为 AI 科学家，我们就永远无法拥有能发现全新知识领域的智能体，那些我们甚至不知道存在的领域。在当下，有些广阔的领域将会存在，但我们甚至不知道它们有存在的潜力。想象一下回到牛顿物理学的时代，没人知道量子物理或相对论的存在。我们应该处于那种开放心态的发现过程中，让智能体自由地自己弄清楚事情。我认为这会带来更令人兴奋的成果。

Yeah, we've been experimenting with quite a number of different rewards. So it's not like we have one answer to that question. But I think one thing we're not doing is crafting a reward that exactly describes what we mean by science, or exactly what we mean by technology, or exactly what we mean by language. We expect our agents to discover things. And if we can't crack that, if we can't crack this problem as AI scientists, we're never going to have agents that discover whole new fields of knowledge that we don't even know about. There are vast fields that will exist that we don't even know have the potential to exist at this point in time. Imagine you were back in the days of Newtonian physics and no one even knew that quantum physics was a thing, or that relativity was a thing. We should be in that process of open-minded discovery where our agents are free to figure things out for themselves. And I think that's going to lead to much more exciting outcomes.

这些“熔炉”在实践中是什么样的？你把它和今天人们所说的环境做了对比——通常是一组任务以及智能体在任务中应该做什么。听起来你们的环境、你们的熔炉非常不同。它们在实践中是什么样的？

What do these crucibles look like in practice? You're contrasting it to today when people say environments—often it's a set of tasks and what the agent should have done on the task. It sounds like your environments, your crucibles, are very different. What do they look like in practice?

我们正在构建的熔炉，在不透露太多细节的情况下，具有大量涌现复杂性的能力。关键之处在于，我们并没有手工制作一大堆谜题让智能体去解决。那里有大量的涌现复杂性。其中一些涌现复杂性可以来自不同地方。它可以来自物理引擎，当你拥有数百万个相互作用的物体和复杂的物理引擎时，就会产生复杂性。你可以有数百万个相互作用的智能体，世界会变得更加复杂，因为它突然包含了智能体所构建的世界中所有其他智能体的复杂性。所以，有办法构建比标准模拟器（比如 Atari 57 之类的）复杂得多的世界，这些世界肯定比人们使用的标准基准任务更复杂。如果你做对了，你就拥有了一个智能体有可能发现事物的世界。智能体有可能发现从技术到科学、数学、语言等一切事物。就像人类一样，从更基础的运动控制或协调，整个谱系都是可能的。

So we are building crucibles which, without giving too many details away, have the capacity for a lot of emergent complexity. The key thing about them is that it's not like we've handcrafted a whole load of puzzles for the agents to work with. There's just a huge amount of emergent complexity there. Some of that emergent complexity can come from different places. It can come from physics engines that give rise to complexity as you've got millions of interacting objects with a complex physics engine. You can have millions of interacting agents, and the world becomes much more complex because it suddenly now includes the complexity of all those other agents within the world that the agents are building. So there are ways to build worlds that are vastly more complex than the standard simulators like Atari 57 or whatever people have been working with in the past, and which are certainly more complex than the standard benchmark tasks that people are using. And if you get that right, you have a world in which there's the possibility for discovery by the agent. There's the possibility for the agents to discover everything from technology to science to mathematics to languages to everything else. Just the same way humans, all the way through from more basic things like motor control or coordination—the whole spectrum is possible.

就像看围棋这样的游戏，它非常容易表示——只是一张位图。模拟极其简单。但你仍然看到复杂行为的涌现。这对我们来说超级有趣。所以当然，围棋比我们能想象的简单得多。但与此同时，要达到 100% 逼真的物理会非常耗费算力。所以我们想创建一系列熔炉，其中有简单的、可以快速执行并迭代的，但我们仍然能看到非常复杂的行为。

Like if you look at a game like Go, which is very simple to represent—it's just a bit map. The simulation is extremely simple. But you still see this emerging behavior of something complex. It's super interesting to us. So of course Go is way simpler than what we can imagine doing. But at the same time, going all the way to 100% realistic physics would be very computationally expensive. So we want to create this range of crucibles where we have simple ones that can execute quickly and we can iterate on, but where we can still see very complex behavior.

有趣。是的。这几乎就像你选择表示什么很棘手，对吧？因为例如，我可以想象很多经济学源于博弈论，而其中大部分与物理完全无关。所以一个表示物理的熔炉可能没那么有用。但如果你表示信息理论——谁知道呢——那么你很可能最终重新发现不少经济学。这样说公平吗？就像你在环境中选择捕捉的基本要素，会催生涌现出的学习类型。

Interesting. Yeah. It's almost like what you choose to represent is tricky, right? Because for example, I can imagine a lot of economics arises from game theory, and most of that is completely unrelated to physics. So a crucible that represents physics is probably not that helpful. But if you have information theory represented—who knows what—then you probably do end up rediscovering quite a bit of economics. Is that a fair way to put it? Like the primitives you choose to capture in the environment kind of give rise to the type of learning that emerges.

是的。最终目标是拥有一个扎根于现实世界的东西，对吧？但需要踏脚石。

Yeah. Like the end goal is to have something that is grounded in the real world, right? But stepping stones.

我想回到围棋的例子，因为我真的很喜欢这个例子。你会惊讶地听到我觉得围棋是个好例子。但我觉得迷人之处，也是我认为 Lass 在这里谈到的，是围棋只是一个 19x19 的网格，你试图最大化领地。如果这是单智能体游戏，它会是你想象中最无聊的游戏。你只是在网格上放棋子直到填满整个网格，没有任何策略。一旦你把第二个智能体放入环境，你就得到了这种涌现复杂性，因为一个智能体必须推理另一个智能体的所有可能行为。突然之间，它极大地增加了环境中的复杂性。据我们所知，当然还有我们研究它时投入的所有算力，存在一条无尽的学习曲线，你可以在这游戏中不断发现新事物。

I wanted to come back to the Go example because I really like this example. You'll be surprised to hear that I find Go to be a good example. But what I find fascinating, which I think is what Lass is talking to here, is that Go is just a 19 by 19 grid where you're trying to maximize territory. And if this was a single agent game, it would be the most boring game you could imagine. It would be, you just place down stones on a grid until it fills up the whole grid, and there would be no strategy whatsoever. The moment you put a second agent into the environment, you get this emergent complexity because one agent has to reason about all the possible behaviors of the other agent. Suddenly it massively increases the complexity in the environment. And to the best of our knowledge, certainly all the compute which we sank into it when we were working on it, there was just this endless learning curve where without end you could continue to discover new things in this game.

这一切并不是说围棋就是终点。

None of this is to say that Go is like the end point.

当然，它仍然只是一个棋盘游戏，在某种意义上相当狭窄，但它表明，非常简单的环境动态可以导致一层又一层无穷无尽的复杂性，永不停歇——当然，其无限性受限于我们计算机的能力。

Of course, it's still just a board game and quite narrow in a certain sense, but it shows that very simple environment dynamics can lead to an infinite onion of layers of complexity that never stops — infinite to the capacity of what our computers can do, of course.

你认为智能体最终会重新发现语言吗？那种语言会看起来像我们的语言，还是会有所不同？

Do you think that the agents will end up rediscovering language, and will that language look like ours or will it look different?

我个人的信念是，任何认为人类语言是推理的终点、或者是最好的推理方式的人，这种观点有点天真。我认为必然存在远比这更复杂的推理方法——也许使用可并行化、丰富、复杂的方法，神经推理或其他什么。这些方法最终将远比基于人类语言的方法更具表达力和能力。

My personal belief is that anyone who thinks that human language is the endpoint of reasoning, or the best way to reason, that perspective is a little naive. I think it must be the case that there are far more sophisticated approaches to reasoning — maybe using parallelizable, rich, complex approaches, neural reasoning or whatever it is. These approaches will ultimately be far more expressive and far more capable than approaches which are based on human language.

我还想说，从人类语言出发作为推理的方法有些问题，因为它会引入我们关于智能体应该如何思考的所有偏见。举个例子，一个很好的思考方式是想象：如果我们退回到 500 年前、1000 年前或 2000 年前，只能使用当时人类使用的所有语言，会怎样？也许我们的智能体会相信世界是平的，或者疾病是由四种体液引起的，或者会有各种奇怪的信念嵌入其中，而且很难移除，因为它们对语言和人类所使用的思维过程来说太根本了。所以我确实认为，超越这一点是一个非常重要的目标。

I would also say that starting from human language as the approach for reasoning is somewhat problematic because it pulls in all of our biases about how we think agents should think. So for example, a nice way to think about this is to imagine: what if we were to step back 500 years or a thousand years or 2,000 years and only were able to use all of the language that humans used at that time? So maybe our agents would believe that the world was flat, or that diseases were caused by the four humors, or there would be all kinds of strange beliefs that would be embedded in there and be very hard to remove because they're so fundamental to the language and the processes of thinking that humans have used. So I do think that going beyond this is a really important goal.

我很想多谈谈这个移除人类先验的概念。我认为你在 AlphaGo、AlphaGo Zero 上的经历，也许是任何人能指出的、关于移除人类先验时会发生什么的最佳例子。也许你可以带我们回到 AlphaGo，然后是 AlphaZero，移除人类先验会带来更聪明的智能，这让你感到惊讶吗？

I'd love to talk more about this concept of removing human priors. And I think your experience with AlphaGo, AlphaGo Zero is maybe the best example that anyone can point to of what happens when you remove human priors. Maybe if you take us back to AlphaGo and then AlphaZero, was it surprising to you that removing human priors would result in even smarter intelligence?

老实说，我并不确定。我记得我总是喜欢做那种研究，你提出的研究问题处于刀刃边缘，就像 50/50 一样。我认为做那种你已经知道答案、并且 95% 确定这东西会成功的研究，并不是很有趣。老实说，做那些事情有点无聊。所以我认为好的研究是关于提出你事先真的不确定答案的问题。在这个案例中，我大约是 50/50，我认为也许有很好的机会它会成功。但我并不自信，看到结果非常令人兴奋。

I wasn't sure, honestly. I remember I always like to do things in research where you ask research questions which are on a knife edge, where it's like 50/50. I think it's not very interesting to do research where you already know the answer and you're like 95% certain this thing's going to work. It's kind of boring, honestly, to do those things. So I think good research is about asking questions where you're really not sure what the answer is in advance. And in this case, I was about 50/50 and I thought that maybe there was a good chance that it might. But I wasn't confident, and it was very exciting to see it.

我应该说明一下，你问到了人类先验。所以我们正在下一个赌注，我们下的赌注是：通过放弃今天所有人都在依赖的先验——比如人类数据，其他人都在其上构建系统的整个互联网知识——我们可以走得更远。这是一个重要的选择，所以我们可能应该谈谈为什么我们要这样做。

I should say, you asked about human priors. So we are making a bet, and the bet we're making is that we can go further by giving up on the priors that everyone's relying on today — like the human data, the whole internet of knowledge that everyone else is building their systems upon. That's an important choice, and so we should probably talk about why we're doing that.

我认为对我来说，原因当然是这是一种短期牺牲——我们放弃一定的性能，以获取可能远大得多的东西。这又回到了这一点：构建一个超级学习者意味着什么？对我来说，构建超级学习者就是构建具有全深度和全广度潜力的东西。

I think the reason for me is, of course, it's a short-term sacrifice — we give up a certain amount of performance in order to access something which is potentially far greater. And it comes back to this: what does it mean to build a superlearner? To me, building a superlearner again is about building something that has this potential for full depth and breadth.

所以在深度方向上，我们需要达到某种能够发现远远超越人类知识的东西，远远超越先验。我们越依赖人类数据，就越无法真正发现那些将带我们一路走向某种东西的算法，比如发现一个全新的知识领域。

So in the depth direction, we need to get to something which is capable of discovering things which go so far beyond human knowledge that it goes just far beyond the prior. And the more we lean on human data, the less able we are to actually discover the algorithms that are really going to take us all the way to something that's going to, for example, discover a whole new field of knowledge.

然后在广度方向上，我们需要有算法可以放置下去，在语言和我们开始时使用的人类数据根本不合适的环境中发现问题。我的意思是，有很多很多领域——例如不同类型的传感器——人类语言数据或一般的人类数据都不合适。所以我认为，通过放弃人类数据，我们让自己处于一个能够真正发现某种可以远远超越它的东西的位置。

And then in the breadth direction, we need to have algorithms that can be placed down and discover things in environments for which language and human data that we've started with is just not a good fit. I mean, there are just many, many areas — different kinds of sensors, for example — where human language data or human data in general is not a good fit. And so I think by letting go of the human data, we put ourselves in a position where we can actually discover something which can go far, far beyond it.

所以如果我们回到 AlphaGo 的例子，我们发现的是：当我们放弃人类数据的那一刻——顺便说一句，我们花了大约两年时间构建 AlphaGo，有点类似于 LLM 的方法，我们构建了每一个层级，手工拼凑下一个层级，我们输入了大量人类数据，然后我们人在回路中改进它，我们会拿上一代并选择如何改进它以产生下一代。所以我们正处于这个过程的回路中。我们花了两年时间才达到 AlphaGo 并击败李世石。

And so if we come back to the AlphaGo example, what we found there was that the moment we let go of the human data — which we'd spent, by the way, about two years building AlphaGo, somewhat similar to the LLM approach, where we built every level of this, we hand put together the next level, we put in a lot of human data, and then we were human in the loop improving it, where we would take the previous generation and choose how we were going to improve it to give the next generation. And so we were right in the loop of this process. It took us two years to get to AlphaGo and beat Lee Sedol.

当我们把 AlphaZero 做对时，我们放弃了人类数据，最终专注于构建一种能够自我发现的算法会是什么样子。所以一旦我们做对了，那个算法只花了几个小时——几个小时就超越了我们在人在回路下所做的两年工作。所以在国际象棋中，它花了四个小时击败世界冠军，即当时有史以来创造的最好的 AI。在围棋中，我想花了几天左右，但这是令人难以置信的进步。所以你必须放弃人类数据，让自己发现这些东西，然后它们就能超越竞争，发现那些永远学习、学习、再学习的东西。

When we got AlphaZero right, we gave up on the human data and finally focused on what it would be like to build an algorithm that could discover for itself. So once we got it right, that algorithm took just a few hours — it took a few hours to outperform the two years of work which we'd done with human in the loop. So in the game of chess, it took four hours to beat the world champion, the best ever AI that had ever been created. In the game of Go, it took, I think it was a couple of days or something, but it was incredible progress. And so you have to let go of the human data to allow yourself to discover these things that can then shoot past the competition and discover the things that just keep learning and learning and learning forever.

所以这必须是一个主动的选择。如果你不做出这个主动选择，那么今天我在很多 LLM 公司看到的是，他们基本上是在构建半自主的人类在环学习算法，有成千上万的人参与到学习过程中。这确实有效，我也非常尊重它能达到的程度。但除非我们退后一步，真正发现那种能够自我学习的过程，否则我不相信我们能完全达到真正超级学习者的境界，并释放我们到目前为止所讨论的潜力。

So it has to be an active choice. And if you don't make that active choice, what I see in a lot of LLM companies today is they're basically building semi-autonomous human-in-the-loop learning algorithms, with thousands of people in the loop of the learning process. It works, and I have a lot of respect for how far it gets. But until we step back from that and really discover the process that's going to learn for itself, I don't believe we'll get all the way to something which is truly a super learner and unleashes the potential that we've talked about so far.

这非常深刻。据我所知，Ineffable 有真人参与，你们中的一些人正在创建环境或熔炉。那么，实际尝试确保人类先验不会渗入你们的工作中，会是什么样子？

That's very profound. As far as I know, there are humans at Ineffable, and some of you are creating environments or crucibles. What does it look like to actually try and make sure that human priors aren't leaking into what you're doing?

是的。当我们设计环境和奖励函数等时，显然其中存在人类偏见。但我们不想——正如 Dave 之前也提到的——我们不想偏向于基于冯·诺依曼架构之类的机器来构建。对吧？我们想要——但如果你提出某个机器会有用的问题，那么看到它会想出什么类型的机器，比如什么样的计算机式机器，会非常有趣。

Yeah. Well, when we design environments and reward functions and so on, obviously there's a human bias in there. But we don't want to—as Dave was also mentioning earlier—we don't want to bias it towards building a machine based on, like, the von Neumann architecture or something like this. Right? We want to—but if you're stating some problem where a machine would be useful, it would be very interesting to see what type of machine it comes up with, like what computer-like machine it comes up with.

所以我完全同意这一点。我还要说，我们对此并不教条。这不是一种宗教，认为任何地方都不能有人类先验。这更像是一种务实的立场：除非我们放弃依赖万亿项的人类知识数据集，否则我们不会取得该领域应得的进展。所以这更像是一种实际立场。在实践中，当然会有一定量的先验，正如 Lassa 所说，这些先验将适用于许多不同的世界，而不是试图放入这些非常强的先验，精确编码我们所知道的关于这个特定问题或这个特定世界的所有知识。所以，弱的环境无关先验似乎完全合理，但那些带有万亿项数据集的超强先验——我们试图避免，因为我们认为从务实角度这会带来更好的结果。

So I totally agree with this. I would also say that we're not religious about this. It's not like a religion that you can't have any human prior anywhere. It's more like a pragmatic position that until we give up on relying on the trillion-item data sets of human knowledge, we're not going to make the progress that the field deserves to see. So it's more like a practical position. And so in practice, of course there's going to be a certain amount of prior which, as Lassa says, will apply to many different worlds, as opposed to trying to put in these very strong priors that exactly encode all the knowledge we know about this one particular problem or this one particular world. So kind of weak, environment-agnostic priors seem perfectly reasonable, but these super strong priors with trillion-item data sets—we're trying to avoid that because we think pragmatically it's going to lead to better outcomes.

是的，绝对。你提到从哲学上讲，你被吸引到更处于刀刃边缘的 50/50 研究。我很好奇你的信心水平，你知道，Ineffable 在最广泛的层面上——比如超级学习者会出现吗？然后我想在更务实的层面上，就你们现在试图实现的事情而言。你对这些也是 50/50 吗？

Yeah, absolutely. You mentioned that you like philosophically that you're drawn to research that's more on a knife's edge 50/50. I'm curious what your confidence levels are for, you know, Ineffable broad at the broadest level—like does a super learner emerge? And then I guess at the more pragmatic level in terms of the things that you're trying to accomplish right now. Are you 50/50 on these as well?

首先，我会说，我坚信这种方法在某个时候会变得可行。就像我们——事实上，这是我们唯一有例子的东西。比如，我们大脑中的例子是唯一的存在证明，证明某种东西已经发现了令人难以置信的智能，这种智能非常通用，在具身世界中有效，并发现如何做各种不同的事情。所以我们有这个人脑的惊人例子，我们可以观察。那么，我认为我们能实现它吗？嗯，我认为像科学中许多伟大时刻一样，这关乎时机。所以无论这成功还是失败，我认为都关乎时机。当你创建一家初创公司时，你想在正确的时间。你不想太早或太晚。如果你太晚，每个人都做过了。这没意思——就像其他一千个人已经做过了。如果你太早，你可能在敲门，但没有正确的——也许领域不够先进，或者没有足够的算力来成功。所以对我来说，一切都关乎时机。例如，在计算机围棋方面，我在计算机围棋被解决前 10 年做出了预测，我精确到年份。星际争霸也是如此，我也——我们打赌并做出了预测。AlphaProof 也是如此，当我们预测 IMO 何时会被攻克时。

So first of all, I would say I strongly believe that the approach will be feasible at some point. Like we—in fact, it's the only thing for which we have an example. Like, you know, the example in our brains is the only existence proof of something which has discovered an incredible intelligence that's very general and works in embodied world and discovers how to do all kinds of different things. So we have this incredible example of the human brain that we can look at. So then, do I think we can achieve it? Well, I think like many great moments in science, it's about timing. So whether this succeeds or fails, I think is about timing. When you create a startup, you want to be at the right time. You don't want to be too early or too late. If you're too late, everyone's done it. It's not interesting—like a thousand other people have already done it. And if you're too early, you might be banging on the door, but not have the right kind of—maybe the field isn't advanced enough or there isn't enough compute to be able to succeed yet. So to me, it's all about timing. For example, with Computer Go, I made a prediction I think 10 years before Computer Go was solved, and I got it right to the year. Got it right for Starcraft, which I also—we had a bet and made a prediction. Same for AlphaProof when we predicted when the IMO was going to fall.

所以你算是某种超级预测者。

So you're a super forecaster of sorts.

到目前为止，到目前为止我在这方面做得很好。我确实有强烈的感觉，这是实现这一目标的正确时机，超级学习者是可能的。就像技术水平取得了巨大进步，使用 LLM 编码方法或我们过去产生的自动研究方法加速研究的能力，对缩放定律的理解和能力，以及技术和堆栈的复杂性。有很多事情都到位了，我认为这使现在成为正确时机。所以这是我的赌注——这是正确的时机。但当然，看，这是研究。

So far, so far I've done very well at this. And I do have a very strong sense that this is the right time for this to fall, but that super learners are possible. Like there's been enormous progress in the level of technology, the ability to accelerate research using LLM coding methods or auto research methods like we've produced in the past, the ability and understanding of scaling laws, and the sophistication of technology and the stack. There's a lot of things that fall into place that I think make this the right time. So that's my bet—that this is the right timing. But yeah, of course, look, this is research.

研究从来不是保证的，这正是它令人兴奋的地方。如果你把握对了时机，你可以成为第一个。你可以成为改变整个 AI 领域的人，这就是冒险和做令人兴奋的事情的意义。

Research is never guaranteed, and that's what makes it exciting. If you get the timing right, you can be the first. You can be the ones to change the whole field of AI, and that's what it means to take risks and do something exciting.

对于 Ineffable 以及你之前正确预测的所有事情，时机最大的驱动因素是算力吗？

For Ineffable and also all the previous things that you correctly forecasted, is the biggest driver of timing compute?

我认为这是一个非常大的驱动因素。我认为这是一个巨大的驱动因素。我认为，是的，我的意思是，我们凭借设法获得的算力处于有利位置，我认为这提供了机会去做比过去尝试自发现强化学习的方法复杂几个数量级的事情。所以我认为，是的，如果我们比较，我不知道，像 OpenAI 的 hide-and-seek 这样的东西，这是一个早期例子，智能体发现了基本的工具使用，我认为我们可能可以运行使用 10 万倍算力的实验。所以有点惊人，人们没有回来真正推动这些类型的方法，考虑到算力的可用性。当然，hide-and-seek——我不想假装那是最终目标。

I think it's a very big driver. I think it's an enormous driver. I think, yeah, I mean, we're in a great position with the compute that we've managed to get hold of, and I think it gives an opportunity to do things which are orders of magnitude more sophisticated than approaches that have tried to do self-discovered RL in the past. So I think, yeah, if we compare to, I don't know, something like hide-and-seek from OpenAI, which was an early example where agents discovered some basic tool use, I think we could probably run experiments that use 100,000 times more compute. And so it's kind of amazing that people haven't come back and really pushed on these types of approaches given the availability of compute. And of course, hide-and-seek—I don't want to pretend that that's the end goal.

我觉得这很基础，但它暗示了可以做什么。我认为算力是一个巨大的驱动力，但不是唯一的。

I think it's very basic, but I think it's suggestive of what could be done. I think compute is a huge driver, but that's not the only one.

我认为我们处于非常幸运的境地，我们能够获得一个非常大的集群。

I think we're in a very fortunate situation that we have been able to secure a very large cluster.

而且是一个漂亮的集群。

A beautiful cluster at that.

是的，所以这对我们来说真的是一个机会。我们真的手头有车可以测试这个。很少有其他人做类似的事情。所以是的，我们可以真正测试这个，这是我们在 DeepMind 无法做到的。

Yeah, so this is really an opportunity for us. We really have the cars on our hands to test this out. Very few other people are doing something similar. So yeah, we can really test this out in a way that we couldn't do at DeepMind, for instance.

绝对。

Absolutely.

看起来这可能走多远的决定因素之一是你能把模拟推到多远，因为最终这些熔炉是数字的，对吧？我认为模拟在过去几年可能已经走了很远。但它并不完美，对吧？我们最近请了你的朋友 Rich Sutton 上节目，我问了他关于模拟的问题，他说他对模拟非常不屑。我很好奇你们怎么想这个，Lisa，你在这两方面都有经验。我认为你显然在数字世界做了很多。但你也构建了 MetNet，它触及真实世界的大气数据。你们怎么想模拟的上限在哪里，什么时候是把苹果带入真实世界的正确时机？

It seems like one of the maybe determinants of how far this can go is how far you can push simulation and because ultimately these crucibles are digital, right? And I think simulation has probably come quite far in the last handful of years. But it's not perfect, right? We recently had your friend Rich Sutton on the show and I asked him about simulation and he said he was very dismissive of simulation. I'm curious how you guys think about this and Lisa, you've been on both sides of this. I think you obviously did a lot in the digital world. But you also built MetNet which touches real world atmospheric data. How do you all think about where is the ceiling on simulation and what is when is the right time to bring an Apple into the real world?

所以我想 MetNet 实际上是一个例子。这是我们在 Google Brain 阿姆斯特丹办公室与 Google Research 一起做的天气预报模型，我们做一个机器学习模型来预测短期天气预报，这比基于物理的模拟要好得多。所以物理模拟会随着时间的推移累积错误，所以物理模拟并不完美。我确实认为我们可以构建很多相对简单的环境，在那里我们可以看到这种新兴行为，比如技术等等，我们不需要精确的物理。我实际上认为有趣的是看看这些天我们可以用非常小的团队在模拟等方面构建多少。所以肯定有一个上限。

So I guess MetNet was actually an example. So this is a weather forecasting model we did in the Google Brain Amsterdam office together with Google Research where we were doing a machine learning model to predict short term weather forecasts and this was way better than the physics based simulations. So the physics simulations were accumulating errors over time so there's a point to that that physics simulation is not perfect. I do think there's a lot of relatively simple environments that we can build where we can see this emerging behavior of let's say technology and so on where we don't need exact physics. I actually do think it's interesting to see how much we can build these days with a very small team in terms of these simulations and so on. So there's definitely a ceiling.

实际上有上限吗？我很好奇这个。

Is there a ceiling actually? I'm curious about this.

嗯，我想是量子物理之类的，对吧？但所以某个地方有上限，但我想说我们可以在没有真实物理的情况下做非常有趣的事情。

Well, I guess quantum physics and whatnot, right? But so there's a ceiling somewhere but we can do very interesting things without real physics I would say.

所以我认为正确的问题是能否在模拟中构建超级智能。所以如果可以在模拟中构建它，我相信它会快 10 倍或 100 倍发生，因为模拟可以无限并行化。你得到完美的可重复性。它有很多非常好的特性。它也是构建超级智能最安全的地方。所以它有这些非常好的特性。然后我们需要问这可能吗？我认为根据我们之前讨论的原因，我们可以将非常清晰的方法添加到模拟中，使其足够复杂以表示任意有趣的事物。例如，假设你想在模拟中构建一台计算机，那么你所需要的只是你的模拟是图灵完备的，如果你有一个图灵完备的模拟，突然我们所说的计算的一切，比如一切，可证明我们所说的计算的一切都可以完全在模拟中表达，事实上人类已经在模拟世界中构建了极其复杂的计算机或相关事物。同样适用于各种其他想法，比如你知道如果你想建立一个经济体，你知道钱可以用硬币或纸或比特来构建，只要你有其中一个，你知道钱的某种表示，我们所说的经济学的一切都可以在模拟中表达。语言也是如此。如果你有多个智能体，并且它们有某种能力来表达智能体之间的通信渠道，突然我们所说的语言的一切都可以在模拟中表达。我们知道数学可以在任何宇宙中表达。科学在某种意义上是研究我们的世界。那么模拟是否足以研究我们所说的科学？嗯，科学有一些方面可以在许多事物中表达，比如你知道能量或熵或进化。但同样真实的是，你知道智能体会发现自己的科学。就像无论模拟器是什么，如果它的环境具有这种涌现的复杂性，智能体将发现关于那个世界的自己的科学。非常可证明的是，可以构建环境，在其中可以表达和表示所有这些事物的巨大复杂性。

So I think the right question to ask is can a superintelligence be built in simulation. So if it is possible for it to be built in simulation I believe it will happen 10 times or 100 times faster just because simulation can be infinitely parallelized. You get perfect reproducibility. It has a lot of really nice properties. It's also the safest place for a superintelligence to be built. So it has these really nice properties. Then we need to ask is it possible? And I think for the reasons we discussed earlier, there are very clear approaches that we can add into the simulation that allow it to be sufficiently complex to represent arbitrarily interesting things. So for example like let's say you wanted to build a computer in simulation well all you require is that your simulation is Turing complete if you have a Turing complete simulation suddenly everything that we mean by computation like everything like provably everything we mean by computation can be expressed completely inside simulation and indeed humans have built incredibly complex computers or related things inside simulated worlds. The same is true for various other ideas like you know if you want to build an economy like you know money can be built out of coins or out of paper or out of bits and as long as you have one of those you know some representation of money everything that we mean by economics can be expressed inside a simulation. The same is true for language. If you've got multiple agents and they've got some ability to express a channel of communication between the agents, suddenly everything that we mean by language is expressible in simulation. Mathematics we know is expressible in any universe. Science in some sense is the study of our world. So will a simulation be sufficient to study what we mean by science? Well, there are some aspects of science which are expressible in many things like you know energy or entropy or evolution. But also it's true that you know the agent will discover its own science. Like whatever the simulator is, if it's got this emergent complexity to its environment, the agent will discover its own science about whatever that world is. It's very like provably possible to build environments in which it's possible to express and represent enormous complexity of all of these things.

嗯，我同意 Lassa。可能有一些东西不能很好地表达。你知道会有一些东西只能在真实世界中表达，比如量子计算就是一个很好的例子。真正的要点是，如果有可能在模拟中一路走到超级智能，它会更快发生，也会更安全发生，所以出于这些原因，这是我们选择投入大量精力的事情。我实际上同意 Rich Sutton，你知道我认为他的观点是，你不应该期望在模拟中学到的东西能像 sim-to-real 一样转移到真实世界，因为真实世界就像更复杂得多，并且以非常有趣的方式难以模拟。但我认为我们可以做的是，我们可以做所有的科学来发现超级学习者。一旦我们有了那个超级学习者，我们就会有答案，我们就会知道如何构建一些东西，同样的算法应该适用于真实世界，我们也可以让我们的模拟越来越接近真实世界的高保真度，并在将它们放入世界之前检查它们是否仍然有效。

Well, I agree with Lassa. There might be some things which aren't well expressed. You know there will be things which can only be expressed in the real world like quantum computation is a good example. The real point is that if it's possible to get all the way to a superintelligence in simulation it will happen faster and it will happen safer and so for those reasons it's something that we choose to put a lot of energy into. I actually agree with Rich Sutton that you know I think his point was something along the lines that you shouldn't expect that the things that you learn in simulation to like transfer like sim to real into the real world because the real world is like just like vastly more complex and in very interesting ways that hard to simulate. But I think what we can do is we can do all of the science to discover the superlearner. And once we have that superlearner we will have the answers and we'll know how to build something which that same algorithm should be applicable to the real world and we can also you know make our simulations more and more high-fidelity towards the real world and check that they still work before putting them into the world.

很好的回答。谢谢。我想谈谈对齐，也许少一点，我知道这是一个现在政治敏感的话题，所以我对围绕对齐的政治或政策讨论不太感兴趣，我零兴趣，但我对围绕它的科学讨论非常感兴趣。看起来出现的很多不对齐都源于你知道将强化学习扩展到它被扩展的程度。

Great answer. Thank you. I'd love to talk about alignment and maybe less I know this is a politically charged topic right now and so I'm less interested I'm zero interested in having the politics or policy discussion around alignment but I am very interested in the scientific discussion around it. It seems like so much of the misalignment that's emerged has arisen from you know scaling reinforcement learning to the extent that that it's been scaled.

你认为你构建的超级学习器会自然地与人类利益对齐，并具有人类价值观吗？它与人类价值观对齐重要吗？

Do you think that the super learner you build will naturally be aligned to human interests and have human values? Is it important for it to have alignment with human values?

这是一个大话题。让我先说几点。今天在“对齐”方面，我们基本上非常依赖基于智能体使用的语言和基于人类数据的先验的对齐方法。到目前为止，这种方法相当有效，但我们开始看到这种方法达到了极限。仅靠它本身，我认为很难看到一条路径，因为智能体不可避免地会变得越来越强大——我们已经能看到这种情况发生——随着智能体被训练得越来越多，它们会开始压倒它们的先验。它们会开始学会说一套做一套。而且，要让它们对齐到人类希望它们做的事情，会变得越来越困难。所以我相信我们需要更多的东西，仅靠这种方法是不够的。作为一家公司，我们能做的最重要的一件事就是真正理解超级智能的科学。这意味着，如果我们能在模拟中真正构建一个超级智能体，并开始看到这些超级学习器一路发展，我们就能开始真正理解超级智能要和平互动需要什么。这样我们就能回答一些根本性的重要问题，比如：在什么条件下，需要什么样的结构，需要什么样的杠杆，才能确保一个超级智能社会实际上和平互动，特别是与它们世界中其他较弱的智能和平互动。我认为如果我们做对了，我们就能得到一些真正的科学，帮助我们理解在现实世界中正确的做法是什么。因为我们正飞速冲向那个现实世界的结果，那里将会有大量不同的超级智能相互作用。我们并不完全知道这会如何发展。我认为我们越快在模拟中得到这些答案并公开分享，我们的处境就会越好。这是我坚定的信念。我还相信，我们可以用这个来把关我们在现实世界中的实际决策。所以我们的承诺之一是，我们在现实世界中做的任何事情，都要以从这种超级智能科学中获得足够证据、我们真正有信心事情会顺利为前提。只有当我们绝对确信那能带来好结果时，我们才会推进到现实世界中的下一个层次。所以我认为这是我相信我们能做的两件事。我们处在一个很多事情发生得非常快的世界里。我们正飞速冲向一个有许多相互作用的超级智能的结果。我个人相信这会给人类带来好结果，但它发生得非常快。我认为，如果我们能够真正理解这一点，并在模拟中提前运行，我们就能处于一个更好的位置，去实际做那些确保这件事进展顺利的事情。

So this is a big topic. Let me start by saying a few things. Where we are today with alignment, we are essentially relying very heavily on alignment approaches that are based on the language that the agents use and priors that are based on human data. That has been reasonably effective up to this point, but we're starting to see that approach reaching its limits. By itself, I think it's going to be difficult to see a path where, as agents become inevitably more and more powerful—we can already see this happening—as agents are trained more and more, they are going to start to overwhelm their priors. They're going to start to learn to say one thing and act in a different way. And it's going to be increasingly difficult to align to the things that humans want them to do. So my belief is that we need something more, and that this by itself is not sufficient. The most important single thing that we can do as a company is to actually understand the science of superintelligence. This means that if we can, in simulation, really build a superintelligent agent and start to see these super learners go all the way, we can start to actually understand what it takes for a superintelligence to interact peacefully. So we can answer fundamentally important questions like: under what conditions, what structures, what leverages do we need to put into place to make sure that a society of superintelligences actually interacts peacefully, and in particular interacts peacefully with other lesser intelligences that are also going to be in their world. And I think if we get this right, we can get to some real science that will help us understand what the right things to do are in the real world. Because we're hurtling towards that real world outcome where there are going to be a whole load of different interacting superintelligences. And we don't quite know how that's going to play out. And I think the faster that we can get to these answers in simulation and publicly share that, the better situation we'll be in. So that's my strong belief. I also believe that we can use this to gate the actual decision-making of what we do in the real world. So one of our commitments is that we will gate anything we do in the real world on having sufficient evidence from this kind of science of superintelligence that we are really confident that things are going to go well. And we'll only push forward to the next level of what we do in the real world once we are absolutely confident that that can lead to a good outcome. So I think these are two things that I believe we can do. We're in this world where a lot of things are happening very quickly. We're hurtling towards an outcome with lots of interacting superintelligences. I personally believe that this will lead to a good outcome for humanity, but it's happening very quickly. And I think that if we are able to actually understand this and run ahead in simulation, we'll be in a much better place to actually do the things that ensure that this goes really well.

这很吸引人。当前安全研究人员主要依赖的一件事似乎是能够用英语监控智能体的思维链。所以当它们开始说自己的语言时，这会让人害怕，因为我们不再能同样地理解和监控它。在你的情况下，你的智能体直接重新发现了语言。你认为这对监控意味着什么？是不是说，如果你把奖励设对了，把环境设对了，那么你就不需要监控了？

Fascinating. And it seems like one of the primary things that current safety researchers rely on is being able to monitor the agents' chains of thought in English. And so when they start speaking their own language, that scares people because we can no longer understand and monitor it as well. In your case, you have your agents kind of going straight to rediscovering language. What do you think that means for monitoring? And is the point that if you get the reward right and if you get the environments right, then you shouldn't need to monitor?

是的，我认为监控仍然非常有帮助。我认为，不管喜欢与否，我相信监控这些系统使用的人类语言的方法，随着智能体变得更强大，会变得越来越困难。我们已经看到，监控——你知道，智能体可能产生一百万字的推理巨著，或者它们可能实际上积极使用诡计，想一套做一套。越来越难 100% 依赖这个。所以我确实认为我们需要武器库中的其他元素。所以我认为，有些人正在研究如何做好这件事，这实际上非常重要。我认为有办法监控不使用语言的智能体。我认为我们可以做这样的事情，例如，你可以使用影子行动，你开始采取行动——你可以看到如果它们被允许在世界上行动，它们实际会做什么，但你实际上还不让它们采取那个行动。你可以开始让其他超级智能充当监控者。例如，你可能有这样一个想法：一个不可思议的科学家智能体，它在地球上轻手轻脚，然后你有另一个超级智能监控它行动的量级并对其进行正则化，以确保它不能做太多影响世界的事情。我认为有很多想法。我想说这里有很多研究要做。我并不声称拥有所有答案，但我认为这值得大量调查。如果人们不关注这个，因为他们只关注基于语言的方法，我认为我们会有麻烦，因为我认为基于语言的方法本身不足以真正达到我们想要的目标。我应该说明，在说了‘有麻烦’之后，我对世界的走向是乐观的。所以‘麻烦’是指我们不会有做我们想让它做的事情的智能体。我确实认为我们最终会进入一个有许多复杂相互作用的超级智能存在的世界，我乐观地认为这会带来一种行动者的平衡，其中一些支持人类的繁荣，一些可能不支持，但那些支持人类繁荣的最终会支持我们，并保护我们免受那些不为我们最佳利益行动者的伤害。

Yeah, I think monitoring is still really helpful. I think that, like it or not, I believe that the approach of monitoring the human language that these systems use is going to be increasingly difficult as agents get more powerful. And we're already seeing that the monitoring—you know, agents are maybe producing a tome of a million words of reasoning, or maybe they're actually actively using subterfuge to think in one way and then act in another. It's increasingly difficult to rely 100% on that. So I do think we need other elements in our armory anyway. So I think it's actually really important that some people are looking at how to get this right. I think there are ways to monitor agents that don't use language. I think we can do things like, for example, you can use shadow actions where you start to have actions taken that—where you can look at what they would actually do if they were allowed to act in the world, but you don't actually let them take that action yet. You can start to have other superintelligences which are acting as the monitor. So, for example, you might have the idea of an incredible scientist agent that sort of treads lightly upon the earth, and then you have another superintelligence that's monitoring the magnitude of its actions and regularizing it to make sure it can't do things which are affecting the world too much. I think there are lots of ideas. I would say that there's a lot of research to be done here. I'm not claiming to have all of the answers, but I think it's something that deserves a lot of investigation. And if people aren't looking at this because they're only looking at the language-based approach, I think we're in trouble, because I think that language-based approach is not enough in itself to actually get to where we want. I should say, in terms of having said 'in trouble,' I am optimistic about where the world is going. So by 'trouble,' I mean that we're not going to have agents that do the things that we want them to do. I do think we're going to end up in a world with lots of complex interacting superintelligent beings, and I'm optimistic that that will lead to a kind of balance of actors, some of which are supporting the flourishing of humanity and some of which may not be, but the ones which do support the flourishing of humanity will end up supporting us and protecting us against the ones which aren't acting on our best interests.

鉴于你的超级预测能力，我很高兴听到这一点。你怎么看所有关于前沿节奏的讨论？

Well, given your super forecasting abilities, I'm happy to hear that. What do you think about all the conversation around pacing the frontier?

我其实非常支持应该对现实世界中 AI 前沿的发展节奏进行控制。同时，我认为在模拟中我们应该尽可能快地推进。原因是在模拟中我们可以开发很多类似超级智能科学的东西，帮助我们找到让现实世界顺利发展的解决方案。我认为这类工作应该尽可能快地进行，以确保当事情在现实世界中真正发生时，能有最好的机会顺利发展并带来人类繁荣。所以这是我的想法。但总的来说，我支持像 Demis 和 Dario 等人建议的方向。我认为这通常是个好主意。也许部分原因是事情发展得太快，我确实相信当我们加速 AI 并达到超级智能时，它将为人类带来真正伟大的未来。但我也认为，事情发展的速度并不是我们能够快速适应的。所以我认为通过稍微控制节奏，它给我们机会让政府、组织和我们的情绪反应适应即将到来的变化，也让这项工作继续推进，真正在模拟中弄清楚如何正确做到这一点。

I'm actually very much supportive of the idea that there should be pacing of what happens for the frontier of AI in the real world. At the same time, I think we should move as fast as possible in simulation. The reason is that in simulation we can actually develop a lot of things like the science of superintelligence that will help us find solutions that help things go well in the real world. And I think that kind of work should happen as fast as possible to make sure that when things really happen in the real world, things have the best possible opportunity to go well and lead to human flourishing. So that would be my thought. But in general, I'm supportive of the suggestions like the direction that people like Demis and Dario and so forth are suggesting. I think it's generally a good idea. Maybe part of that is that things are moving really quickly and I do believe that when we accelerate AI and get to superintelligence, it will lead to a really great future for humanity. But I also think that the speed at which things are moving is not something that we are capable of adapting to that fast. So I think just by pacing a little bit, it gives us the opportunity for our governments and our organizations and our emotional responses to adapt to the changes that are coming, and also for this work to continue at pace, like really figuring out in simulation how to do this right.

所以，在模拟中加速，以便我们理解正在发生的事情并围绕它做科学，然后在现实世界中谨慎。

So, accelerate in simulation so we can understand what's happening and do science around it, and then careful in the real world.

是的，我想这就是我的总结。

Yeah, I think that would be my summary.

我想过渡到稍微谈谈算力。这将是一段极其昂贵的算力之旅，而你正在推动算力的极限。我认为 Google 和 Nvidia 都是你的好伙伴。你能谈谈你的集群吗？就像你不是像许多其他新实验室那样简单地从 NeoClouds 租用现货算力。你实际上在你的集群和你拥有的其他不同形式的算力上做了非常认真的工作。你能谈谈你的算力设置吗？

I'd love to transition to talking a little bit about compute. This is going to be an incredibly compute expensive journey and you are pushing the limits of what compute can do. I think both Google and Nvidia have been great partners for you. Can you talk a little bit about your cluster? Like you're not simply renting spot compute from NeoClouds like many other Neolabs. You're actually doing some really serious work on your cluster and the different other forms of compute that you have. Can you just talk about your compute setup?

对。所以正如我之前提到的，我们非常幸运拥有这些算力。到今年年底，我们将成为 GCP 上最大 GPU 集群的首批客户。实际上，现在已经启动一个月了。所以我们非常幸运，我实际上对现在获得算力的初创公司感到相当难过，因为变得越来越难。但是的，所以我们有这个非常大的集群，基于 GPU，比如 Nvidia GPU，这对我们来说非常幸运，因为这意味着我们可以轻松地进行模拟、训练和推理。它们是非常通用的芯片，所以很棒。我认为特别是 Rubin 芯片，它们有一种架构，一个机架基本上可以被视为一台机器。一个机架内的节点之间有非常高的带宽，这对于强化学习非常重要，因为强化学习需要非常快的迭代周期，而 LLM 相对而言相当慢。所以这对我们来说非常幸运。我认为我们 8-10 年前在 Google 做的一些事情，当时我们基本上有丰富的 CPU，那已经不再适用了。所以我们真的需要确保我们所做的一切都在 GPU 上，我们不需要在这些 GPU 之外还需要数百万个 CPU，这样我们的模拟可以在 GPU 推理和训练等上运行。所以我们有一些好的架构。我认为 LM 公司现在有我们多年前遇到的问题，他们需要在 CPU 上运行沙箱，比如模拟计算机等，我们正试图不让自己陷入那种情况。

Right. So as I mentioned earlier, we are very fortunate with the compute that we have. We are the first customers of the largest GPU cluster on GCP by the end of the year. Well, actually started like one month now. So we are extremely fortunate in that and I actually feel quite bad about the startups that are getting compute now because it's becoming harder and harder. But yeah, so we have this very large cluster which is based on GPUs, right, like Nvidia GPUs, which is really fortunate for us because it means that we can easily do simulation and training and inference. They are quite general chips, so that's amazing. I think like the Rubins specifically, they have this architecture where one rack can basically be considered like one machine. There's a very high bandwidth between the nodes in one rack, which is very important for reinforcement learning where you have really fast iteration cycles compared to LLMs that are relatively speaking quite slow. So that's quite fortunate for us. I think some of the things we did like 8-10 years ago at Google where we basically had abundant CPUs at Google, that doesn't scale anymore. So we really need to make sure that everything we do is on the GPUs and we don't need millions of CPUs beyond these GPUs, so our simulations can run in the GPU inference and training and so on. So we have some good architectures for that. I think the LM companies now have kind of the problems that we had many years ago where they need sandboxes running on CPUs like simulating computers and so on, and we're trying to not put ourselves in that situation.

所有的算力，在你正在做的不同事情之间大致可以互换吗？我想象你在构建熔炉，然后进行这些大型部署，智能体实际上在学习。这是算力被用于的大致分类吗？

All the compute, is it roughly fungible between the different things that you're doing? I imagine you're building crucibles and then you're kind of doing these big rollouts where the agents are actually learning. Is that the rough buckets of what compute is being used for?

是的，你可以把它分为训练、推理和模拟。

Yeah, you could split it up into training, inference, and simulation.

同样的算力在训练、推理和模拟上同样出色吗？

Is the same compute equally good at training, inference, and simulation?

不。

No.

好的。

Okay.

所以我认为很明显，过去许多年或自深度学习兴起以来，GPU 发生的情况是它们越来越针对训练，也许在过去几年也针对推理，但较少针对模拟。但类似于 LLM 公司，我们也希望处于缩放定律重要的境地。就像我们知道大致上如果你增加这么多算力和经验，那么我们就能从中获得一定质量。

So I think it's pretty clear what has happened with GPUs over the last many years or since deep learning took off is that they've been more and more tailored towards training, and maybe in the last few years also inference, but less about simulation. So but we are similarly to the LLM companies, we also want to be in the case where scaling laws matter. Like we know that roughly if you add this amount of compute and experiences, then we get a certain amount of quality out of that.

所以我猜如果 GPU 不擅长扩展模拟，那么解决方案是什么？

So I guess if GPUs aren't great for scaling simulation, then what's the solution?

对。所以这里有许多不同的解决方案。有基于模型的强化学习，其中模型变得比模拟昂贵得多。所以你可以做许多不同的技巧。有各种方法用于回放缓冲区或多次重用数据。所以有办法绕过这一点。除此之外，我们确实花了不少精力让模拟尽可能快，并让它们最好地利用 GPU。

Right. So there are many different solutions here. There's model-based reinforcement learning where the model becomes much more expensive compared to the simulation. So there are many different tricks you can do. There are various approaches for replay buffers or reusing data many times. So there are ways around that. And other than that, we do spend quite some effort on just making the simulations as fast as possible and making them use the best use of the GPUs.

你看到缩放定律了吗，还是说为时过早？

Are you seeing scaling laws yet or is it too early to say?

所以对于强化学习，缩放定律略有不同，对吧？缩放定律是投入多少算力、投入多少数据以及模型大小的组成部分。强化学习有点不同，因为问题是非平稳的，所以随着你在问题中的进展，数据会变化，数据分布会变化。所以它们有点不同。这是一个非常有趣的研究方向。

So for reinforcement learning, scaling laws are slightly different, right? So scaling laws is the component of how much compute you put in, how much data you put in, and the model size. Well, reinforcement learning is a little bit different because the problem is non-stationary, so as you progress in the problem, the data changes, the data distribution changes. So they are a little bit different. It's a very interesting research direction.

我理解它们不同，但你是否看到某种缩放定律，即投入更多算力能让你更接近实现目标，还是现在很难说？

I understand they're different, but are you seeing some sort of scaling law where more compute in gets you closer to achieving the goals, or is it hard to say right now?

是的，但这是我们正在积极推动的事情。特别是当我们从较小的集群过渡到更大的集群时，我们希望为此做好准备。

Yeah, but it's something we are actively pushing towards. And especially as we transition from a smaller cluster to the bigger cluster, we want to be ready for that.

是的。

Yeah.

David，他介绍你时提到，你在强化学习的算力和算法方面做了一些非常基础性的工作，这个领域今天真的依赖这些工作。也许可以谈谈其中一些，特别是你在 IMPALA 和 SEED 上的工作。它们是什么，以及它们与你目前在 Ineffable 所做的工作有何关联？

David, when he introduced you, he mentioned that you had done some very foundational work on compute and algorithms for reinforcement learning upon which the field really relies on today. Maybe talk about some of that work, and in particular your work around IMPALA and SEED. What are those, and how do they relate to the work that you're doing at Ineffable?

对。当我加入 DeepMind 的深度学习团队时，大多数人都在运行单机 CPU 的强化学习。所以规模相当小。我和同事一起做的是，打造一个真正利用 GPU 的强化学习智能体。它可以随算力线性扩展。我认为部分原因是它非常高效，非常擅长扩展。第三点是它非常容易扩展和修改，你想怎么用就怎么用。这使它成为公司里最受欢迎的强化学习智能体，我记得有一段时间大多数强化学习团队都在用它或它的衍生版本。所以那相当成功，也是一个很不错的项目。

Right. So when I joined the deep learning team at DeepMind, most people were running single-machine CPU-based reinforcement learning. So the scale was quite small. What I did together with colleagues was to make a reinforcement learning agent that really utilized GPUs. It could scale linearly with compute. I think partially because it was very efficient, it was very good at scaling. And the third thing was it was very easy to extend and modify and do whatever you wanted with it. It contributed to it being the most popular reinforcement agent at the company, and I think at one point most RL teams used it or derivatives of it. So that was quite successful and quite a nice project to work on.

后来，我们有了 TPU，对吧？我们有了成架的 TPU，所以需要扩展到更大规模。所以我在离开 DeepMind 去创立 Google Brain 阿姆斯特丹办公室后，在 Google Brain 的部分工作就是扩展到成架的 TPU，那个产品叫 SEED RL。所以它真的是关于以最佳方式利用 TPU 进行推理和训练，同时保持环境尽可能灵活。它可以是任何东西，从围棋到非常昂贵的物理模拟。

Then later on, we got TPUs, right? We got racks of these TPUs, so we needed to scale beyond that. So some of my work at Google Brain after I left DeepMind to found the Google Brain Amsterdam office was to scale to racks of TPUs, and that product was called SEED RL. So it was really about utilizing the TPUs both for inference and training in the best possible way while keeping the environments as flexible as you wanted them to be. It could be anything from Go to very expensive physics simulations.

那在 Ineffable 是怎么用的？

And how's that being used at Ineffable?

是的。我们实际上实现了其中一些算法。还有这些算法的衍生版本。所以这就是我们今天在用的。我们根据环境不同，使用不同的算法，对吧？它们各有优缺点。所以是的，我们谈到过，我们有一些非常简单、快速组装的环境，也有一些更昂贵的。

Yeah. So well we actually implemented a few of those algorithms. Also derivatives of these algorithms. So that is what we are using today. We have, depending on the environments, we use different algorithms, right? That have pros and cons between them. So yeah, so we talked about we have some very simple, fast to assemble environments and also some that are more expensive.

对于那些今天在做大规模强化学习架构、还没读过那些原始论文的人，绝对值得一看，因为我想你会发现，当前使用的许多想法都可以追溯到 Lassa 早期所做和弄明白的工作，我会说那有点先见之明。实际上，看到现在的强化学习领域与 10 年前或比如 8 年前相比，我觉得挺有趣的。LLM 社区也在使用强化学习，尤其是过去两年。他们使用的很多东西都是我们多年前用过的、正在被重新发现的方法。所以看到这些挺有意思的，而且上面还有很多新颖之处，对吧？但是的，

For those who are working on large-scale RL architectures today and haven't read those original papers, it's definitely worth a look because I think you'll discover that many of the ideas that are currently in use kind of trace back to those early work that Lassa kind of did and figured out, which was a little bit prescient, I would say. I think it's actually kind of fun to see the reinforcement learning space now versus like 10 years ago or like let's say like eight years ago maybe. The LLM community is also using reinforcement learning in particular last two years. And a lot of the things that they're using are being rediscovered approaches that we used many years ago. So that's kind of interesting to see with a lot of novelty as well on top of it, right? But yeah,

你能稍微谈谈你的团队吗？团队现在有多大？

Can you talk a little bit about your team? How big is the team right now?

我们有意保持团队相对较小。所以我们不急于建立一个庞大的组织。我认为如今用一小群极其有才华的人就能做非常多的事。所以我们总共雇了 20 人。是的，我们雇的人，是要找到一些特别有才华的人，但对我们来说不仅仅是才华。还要把善良、乐于合作的人聚在一起。这对我们试图建立的文化非常重要，还有那些与我们追随的使命高度一致的人。以及那些符合我们需求的人。所以我们试图用合适的技能组合来组建团队。所以这四件事都非常重要，是的，我们对进来的人有非常高的标准和非常高的期望。所以我们很幸运能得到一些很棒的、真正很棒的人。那么，也把它和算力联系起来，对吧？到目前为止相对较少的人意味着我们人均算力很高，对吧？而且如今一个人就能做很多实验。所以这也是为了保持算力与人员的比例相当高。

We have intentionally kept the team relatively small. So we're not in a hurry to build a giant organization. I think you can do an awful lot nowadays with a small number of incredibly talented people. So we have hired 20 people in total. And yeah, the people we've hired, it's about finding some of the exceptionally talented people, but it's not just about talent for us. It's also about bringing together people who are kind, collaborative human beings. That's really important to the kind of culture we're trying to build, and people who are really well aligned with the kind of mission that we're following. And people that sort of fit the needs that we have. So we try and build the team with the right set of skills as well. So all of those four things are really important, and yeah, we kind of have very high standards and very high expectations of the people that come in. So we'll be very lucky to get some amazing, really amazing people. And so just to tie it to the compute as well, right? Like this relatively small amount of people so far means that we have a lot of compute per person, right? And like these days you can do a lot of experiments just like one person. So it's really also to keep the compute to person ratio quite high.

所以我们不是从第一天起就争抢资源。

So we're not like fighting for resources from day one.

我想有三位联合创始人一起在 AlphaStar 上工作过。你们很多人确实一起工作过。你是在组建复仇者联盟吗，尤其是考虑到 DeepMind 最近的变化？你是在组建你过去共事过的人的复仇者联盟吗？你雇了新的人吗？如果是，是什么特质让一个你不认识的人有可能适合 Ineffable？

I think three of the co-founders worked together on AlphaStar. Many of you did work together. Are you assembling the Avengers, especially given the recent changes from DeepMind? Are you assembling the Avengers of folks that you've worked with in the past? Have you hired new people? And if so, what are the attributes that make somebody new to you likely to be a fit at Ineffable?

所以是混合的。我的意思是，AlphaStar 是一支非凡的团队。它是在 DeepMind 内部有意组建的，挑选最优秀的人来做那个项目。所以从某种意义上说，我认为实际上我们有四位联合创始人来自那个团队，这并不奇怪。但我们并不只从那个团队或 DeepMind 招人。我的意思是，另外三位创始人都来自非常成功的初创公司。我们雇的人很聪明，但也善良、好相处，能很好地合作，与使命一致，并且符合我们需要的东西。所以，但在找到这些人方面，我们付出了很多努力来找到合适的人。所以，尽管不舒服，但我不得不拒绝很多过去共事过的人，因为我们的标准太高了。你知道，我有幸与一些了不起的人共事。所以，你知道，我一直在非常杰出的一批人中挑选，带进来我认为是超级明星的团队。所以是的，进入 Ineffable 不容易，但我们有的人都是很棒的人，是的，非常非常为我们带来的人感到骄傲。

So it's a mixture. So, I mean, AlphaStar was an exceptional team. It was sort of intentionally assembled within DeepMind by choosing the best of the best to work on that project. So in some sense it's not a surprise that I think actually four of our co-founders come from that team. But we are not exclusively drawing from that team or indeed from DeepMind. I mean the other three founders all came from very successful startups. And we hire people who are brilliant but also kind and nice people to have around, who are going to work together well, who are aligned with the mission, and who fit the things that we need. So, but in terms of finding those people, we put a lot of effort into getting the right people. So, as uncomfortable as it is, I've had to say no to a lot of people I've worked with in the past just because our bar is so super high. You know, I've been fortunate to work with some amazing people. And so, you know, I've been choosing amongst a very exceptional crop and bringing in what I consider to be a superstar team. And so yeah, it's not easy to get into Ineffable, but the people who we have are wonderful human beings and yeah, very very proud of the people we brought in.

我能理解为什么人们被吸引到 Ineffable 工作，尤其是在最大的集群、最大的实验室陷入这种囚徒困境竞赛的时候，有很多钱在赌注上，所以决策方式不同。我可以想象，对于一群真正深信你们想要完成的科学使命的人来说，这一定感觉像一股清风。

I can see why people are drawn to working at Ineffable, especially at a time when the biggest clusters, the biggest labs are locked in this prisoner's dilemma race, and there's a lot of dollars at stake, so decisions get made differently. I can imagine for a set of people that really deeply believe in the scientific mission of what you want to accomplish, this must feel like a breath of fresh air.

是的，我认为是这样。

Yeah, I think that's right.

我们试图建立一种以研究为主导的文化。所以，这不是一家由打造增量式下一个产品（比如在已有产品上推出 V17.1）所驱动的公司。这是一家建立在研究人员追求他们的热情、并试图产生他们认为可能的突破之上的公司。我们从一开始就把公司围绕这一点来构建，创造一个研究的避风港，在这里我们顶住产品压力，因为我们相信，如果我们取得这些突破，它将在下游带来比从一开始就做产品大得多的产品和成功。所以，是的，这是以研究为主导的。我认为我们正在创造一个非常特别的环境，是的，人们很喜欢它。我也很喜欢。

And we've tried to establish a research-led culture. So it's not a company that's led by building an incremental next product, V17.1 on the thing that's come before. It's a company that's built on researchers pursuing their passion and trying to produce the breakthroughs that they believe are possible. And we center the company around that from the beginning and from the outset to create a haven for research, where we push back on product pressure because we believe that if we make these breakthroughs, it will lead to far bigger products downstream and far bigger successes than if we kind of start off from day one on, oh, here's the product we're doing from the beginning. So, yeah, so it's research-led. We're creating, I think, a really special environment and yeah, people are loving it. I'm loving it.

太棒了，太棒了。我想以一些预测来结束，也许，特别是因为你是一位超级预测者。你猜什么时候你会获得第一个强有力的证据表明这种方法有效，然后你预测我们什么时候会实现你所设定的使命——首次接触超级智能？

Wonderful, wonderful. I'd love to close with some predictions, maybe, especially since you're such a super forecaster. What's your guess for when you'll have the first strong evidence that the approach is working and then what's your forecast for when we'll achieve the mission that you've set out to achieve this first contact with super intelligence?

是的，这些都是合理的问题。所以在这方面会有一些合理的不确定性。

Yeah, so these are fair questions. So there are going to be some reasonable bars of uncertainty on this.

是的，但你正在被录音，所以我们会让你兑现的。

Yeah, but you're being recorded, so we're going to hold you to it.

没错。是的。所以我认为我们将在一年内看到一些极其强有力的证据表明我们走在正确的轨道上，我认为我们将实现真正的超级智能，就像我之前给出的定义那样，即发现远远超出人类在科学、数学、技术、语言等各个领域所能实现的东西。我原本认为在三到五年内。我的时间线加快了一点。所以我会说三年，因为是的，编码智能体可以加速事情。

Exactly. Yeah. So I think we will see some extremely strong evidence that we're on the right track within a year and I think we will achieve something which is a true super intelligence like the according to the definition I gave earlier of something which discovers things which are far beyond what humans can achieve in across everything from science, mathematics, technology, languages etc etc. I originally thought within three to five years. My timelines have accelerated a little bit. So I'm gonna say three years because of yeah what can be done with like coding agents to accelerate things.

我认为，是的，我认为事情会发展得很快。

I think yeah I think things are going to move fast.

系好安全带。这将是一段旅程。

Buckle up. It's going to be a journey.

太棒了。David，这是一次非常精彩的对话。你所构建的东西让我深受启发，感谢你抽出时间坐下来与我们的观众分享你今天正在构建的东西。

Wonderful. Lei David, this is a wonderful wonderful conversation. I'm so inspired by what you're building and thank you for taking the time to sit down and share with our audience what you're building today.

谢谢你，Sonia。很荣幸。

Thank you, Sonia. Pleasure.