AGI 与物理 AI：与 AI 先驱尤尔根·施密德胡伯的对话

听起来你认为一场崩盘即将来临。会有一场股市崩盘。那机器人呢？机器人硬件与人体相比确实很差。没有人类制造的技术能比得上这条路径。这似乎远不止是硬件问题。没有那样的硬件，你不可能实现 AGI。你不可能仅仅在屏幕后面实现 AGI。我们离那一步近了吗？

Sounds like you think a crash is coming. There would be one of these stock market crashes. What about like robots? Robot hardware is really inferior compared to human bodies. And there's no humanmade technology that compares to this path. It seems a lot more than a hardware problem. You can't have AGI without hardware like that. You can't have AGI just behind the screen. Are we kind of close to that?

它会来的，但还需要一段时间。

It will come, but it will take a while.

Jürgen Schmidhuber 被《纽约时报》、《福布斯》等媒体誉为 AI 之父。他推动了该领域一些最重要的进步，这些进步真正支撑了今天的 AI 革命。在 Unsupervised Learning 节目中，我有幸与他坐下来，讨论当今生态系统中大家最关心的一切。我们谈到了当前模型缺少什么，以及他认为需要什么才能获得能够推动它们前进的人工科学家。我们谈到了为什么他认为当前的资本支出热潮被严重夸大，为什么他对 AI 技术非常乐观，但对模型公司深感悲观，以及他认为递归自我改进实际上不会成为公司的护城河。我们还讨论了 AI 安全，以及为什么他比该领域的许多其他人明显不那么担心。能与这位领域内的传奇人物坐下来，问他所有这些问题，真是一个绝佳的机会。我想大家会很喜欢听他的观点。闲话少说，有请 Jürgen。非常感谢你来做客播客。真的很兴奋。

Jürgen Schmidhuber has been cited as the father of AI by the New York Times, Forbes, and more. He is behind some of the most important advances in the field that really power the AI revolution today. And it was a real privilege on Unsupervised Learning to get to sit down with him and talk about everything that's top of mind in the ecosystem today. We talked about what's missing in models today and what he thinks is required to get artificial scientists that can push them forward. We talked about why he thinks today's capex boom is massively overdone and why he's very optimistic on AI technology but deeply pessimistic on the model companies and how he doesn't think recursive self-improvement will actually be a moat for the companies. We also talked about AI safety and why he's notably less worried than a lot of others in the field. It's just an awesome opportunity to get to sit down with a legend in the field and ask him all these questions. I think folks will really enjoy hearing his perspective. Without further ado, here's Jürgen. Well, thanks so much for coming on the podcast. Really excited about this.

这是我的荣幸，Jacob。

It's my pleasure, Jacob.

嗯，我觉得今天有很多不同的话题想聊。你显然是当前 AI 浪潮中许多不同领域的先驱。我想，从最高层面开始的话，据我所知，你有一个长期目标，那就是构建一个比自己更聪明的 AI。我们现在离那一步有多近？

Well, I feel like there are so many different things I want to talk about today. You've obviously been a pioneer of a ton of different parts of this current AI moment. You know, I figured where I'd start at the highest level was, as I understand it, you've had this goal for a long time, which was to build an AI smarter than yourself. How close are we to that right now?

从宇宙的角度来看，我们和 1970 年代我第一次有这个愿望时一样近。所以我们非常接近。但到底是几年还是几十年？我不太确定，因为真正的 AI 不仅仅是屏幕背后的 AI，它运行得很好，也通过了图灵测试。现在真正的 AI 还包括真正的机器人，屏幕之外真实世界、物理世界中的真正机器，而这方面进展不那么顺利。所以现实世界中的硬件有很多人体没有的限制，我们还有很长的路要走，才能在物理 AI 方面与人体竞争。

From a cosmic perspective, we are as close as we were in the 1970s when I first wished. So we are very close. But is it going to be a couple of years or a couple of decades? I'm not totally sure about that because true AI is not just the AI behind the screen, which is working very well and which is passing the Turing test. Now true AI is also real robots, real machinery outside of the screen in the real world, in the physical world, and that's not working as well. So the hardware in the real world has lots of limitations that human bodies don't have, and we still have a way to go to be able to compete with human bodies in physical AI.

过去几年里，有没有什么当前的 AI 成果让你感到惊讶？

Have there been any current AI results over the last few years that have surprised you?

并没有。没有到让那些前几十年从未接触过神经网络和人工神经网络的人感到惊讶的程度。突然之间，有了 ChatGPT 时刻，人们开始对那个他们从未见过的东西感兴趣。所以他们不知道大型语言模型有很长的历史，而训练这些大型语言模型的基本见解和算法有更长的历史，可以追溯到上个千年。所以对于一个身处这一切中心的人来说，预测这一点比那些兴趣完全不同的人要容易得多。

Not really. Not to the extent that it surprised people who had no contact with neural networks and artificial neural networks in the previous decades. And suddenly there was a ChatGPT moment and suddenly people started being interested in that thing which they had never seen before. So they didn't know that there was a long history of large language models and an even longer history of basic insights and algorithms for training these large language models that goes back to the previous millennium. So for a guy who was in the center of all of that, it was much easier to predict that than for someone who had totally different interests.

嗯，我确实想谈谈递归自我改进和元学习，因为我觉得这已经是你关注的重点有一段时间了。显然，这似乎也是当今许多主要实验室的主要焦点。你在这里开创了一系列研究。你如何阐述我们今天在通往 RSI 的道路上处于什么位置，以及还有什么需要解决？

Well, I definitely want to hit on recursive self-improvement and metalearning because I feel like it's been a huge focus of yours for a while. It obviously seems to be a main focus of a lot of the major labs these days. And you pioneered a bunch of the research here. How do you kind of articulate where we are today on the path to getting to RSI and what still needs to be solved?

所以在 1987 年，这是关于使用元进化、进化编程来进化出更好的程序，这些程序学会做更好的进化。我称之为元进化，它在很多方面都非常达尔文主义。随着时间的推移，你有了越来越好的学习算法，学会以越来越好的方式组合先前程序的代码，从而越来越好地解决问题。然后在 1994 年，我们有了强化学习技术，自指机器能够基本上使用通用编程语言来生成运行该机器的代码的任意自我修改，该机器与环境交互。然后在 2003 年，那是一种数学上最优的方式，通过一台拥有一些软件并与环境交互的机器来生成自我改进。这个环境有时会惩罚你或提供奖励，你想要最大化你一生中直到生命结束的所有奖励的总和，并最小化痛苦信号的总和。然后那台机器中有一个初始软件，这台机器原则上可以编写修改初始软件的程序。但在它这样修改自己之前，它必须首先证明，这意味着软件中有一个证明搜索。它必须证明执行这个特定程序所导致的修改是有用的，即它将导致比不执行这个程序更多的预期奖励。然后它必须生成一个形式证明，这被称为 Gödel 机器。它不如我们做的其他某些事情实用，比如通过在自己的网络上运行学习算法来改变自身权重矩阵的神经网络。那是我们从 1992 年开始的，目前这或多或少是当今最流行的自我修改和自我指涉类型。

So in 1987, this was about using meta-evolution, evolutionary programming for evolving better programs that learn to do better kinds of evolution. Meta-evolution I called it, and it was very Darwinistic in many ways. And so over time you had better and better learning algorithms, learning to combine code from previous programs in better and better ways to solve problems better and better. And then in 1994 we had reinforcement learning techniques, self-referential machines that were able to basically use a universal programming language to generate arbitrary self-modifications of the code that was running the machine interacting with some environment. And then in 2003, that was a mathematically optimal way of generating self-improvements by a machine that has some software and it's interacting with an environment. This environment sometimes punishes you or provides reward, and you want to maximize the sum of all the rewards in your life until the end of your life and you want to minimize the sum of the pain signals. And then there's an initial software in that machine, and this machine then can in principle write programs that modify the initial software. But before it modifies itself like that, it first has to prove, which means there's a proof search in the software. It has to prove that the modification that will be caused by the execution of this particular program is useful in the sense that it will lead to more expected reward than the alternative, which would be not to execute this program. And then it has to generate a formal proof, which was called the Gödel machine. It is less practical than certain other things that we did, like neural networks that change their own weight matrix by running a learning algorithm on the network itself. That is what we started in 1992, and that's currently more or less the most popular kind of self-modification and self-reference today.

正如你提到的，一个限制是，由人类定义这些试验的开始和结束，而不是一种数学方式预先确定这是否会有帮助。我想，显然这样做的代价是，进行这些证明非常耗费算力。我想，当你展望通往 RSI 的路径时，你认为它会涉及这种预先进行证明的能力，还是像权重修改那样？你认为答案在其中的可能性有多大？

As you kind of alluded to, one of the limitations is you have humans defining the start and end of these trials versus a mathematical way to determine if this is going to be helpful beforehand. And I guess obviously the trade-off of that being that it's very compute-intensive to do some of these proofs. I guess as you think forward to what the path to RSI might be, do you think it will involve this kind of ability to do these proofs beforehand, or is it like the modification of weights? What percent likelihood would you think that the answer lies in one of those?

是的。

Yeah.

所以我会说，当前大多数自我改进系统都是 2003 年哥德尔机（数学上最优的东西）的缩水版、弱化版。它们更像我们早期有的东西：神经网络，其权重和程序基本上就是权重矩阵。尽管有些神经网络是通用计算机，例如循环网络，它们是通用计算机，因为在循环神经网络上你可以实现苹果笔记本电脑的处理单元之类的东西。所以如果你有某些指令允许你修改权重本身，那么你基本上可以在网络上运行任意学习算法，该网络必须看到输入的误差或负奖励信号。这是输入的一部分，至关重要。这就是我们在 90 年代初所做的全部，当时算力非常昂贵，比今天贵 1000 万倍，我们只能做很小的玩具实验。但今天你可以很好地展示，像这样的方法可以学习泛化，并且比没有这种元学习能力时更快地学习新任务。这是目前最流行的递归自我改进方式。然而，必须承认它是有限的，因为那里的学习算法是通过梯度下降发明的。所以一切都是可微的，然后整个网络通过梯度下降学习生成比梯度下降引起的更好的权重变化。

So I would say most of the current self-improving systems are scaled-back versions, toned-down versions of the Gödel machine of 2003, the mathematically optimal thing. They are more like what we had earlier: neural networks where the weights and the program of a neural network is basically the weight matrix. Although some neural networks are general-purpose computers, recurrent networks for example, they are general-purpose computers because on a recurrent neural network you can implement the processing unit of your Apple laptop or something like that. So if you have certain instructions that allow you to modify the weights themselves, then you can basically run arbitrary learning algorithms on the network, which has to see the errors or negative reward signals that are coming in. That has to be part of the input, which is essential. That's all what we did in the early 90s, and back then compute was so expensive, 10 million times more expensive than today, that we could only do little tiny toy experiments. But today you can really show nicely that methods like that can learn to generalize and learn new tasks much faster than if you don't have this metalearning capacity. This is currently the most popular way of doing recursive self-improvement. However, one has to admit that it is limited because there the learning algorithm is invented through gradient descent. So everything is differentiable, and then the whole network learns through gradient descent to generate weight changes that are better than what's caused by gradient descent.

是的。

Yeah.

所以这有梯度下降的局限性。它不像最优的哥德尔机，但在实践中效果很好。

So that has the limitations of gradient descent. It's not like the optimal Gödel machine, but it works really nicely in practice.

我认为人们对递归自我改进的一个大问题是：回顾起来，它会感觉像是一种渐进且无聊的改进，还是会有某种巨大的不连续性，模型能力突然起飞？当我们到达那里并回顾时，你的直觉是什么？

I think a big question people have around recursive self-improvement is: in retrospect, is it going to feel like a gradual and boring improvement, or is there some huge discontinuity around the horizon where suddenly models take off in capabilities? What's your gut instinct around that when we get there and are kind of looking back?

嗯，从宇宙的角度来看，它看起来就像一根棍子：没有自我改进，没有真正的人工智能，当然有 AI，但从宇宙的角度来看，这对所有文明也是如此。文明大约始于 13000 年前，在那之前没有人工智能，然后仅仅 13000 年后就有了人工智能。13000 年前第一个拥有农业和动物驯化的人几乎就是第一个拥有 AI 的人，而 13000 年的文明只是世界历史（约 138 亿年）的百万分之一。所以这只是世界历史中的一瞬间。事后看来，它会是这样。文明几乎与 AI 同时发生，因为在过去的 13000 年里，越来越多的事物被自动化：越来越多的农业被自动化，越来越多的人类劳动被自动化。在某个时刻，思考在几百年前开始被自动化。第一批计算器，然后计算器变得更便宜，变得更快，现在你可以用同样的价格计算比 100 年前多得多的事情，然后突然它就出现了。从这个全球视角来看，真的就像什么都没有，然后突然有了一条定律。从一个生活在这个时代的人的角度来看，它看起来很多。

Yeah, well, from a cosmic perspective, it will look just like a stick: there was no self-improvement and no real AI, and certainly there was AI, but from a cosmic perspective, this is also true for all of civilization. Civilization started roughly 13,000 years ago, and before that there was no artificial intelligence, and then only 13,000 years later there was artificial intelligence. The first guy who had agriculture 13,000 years ago and domestication of animals was almost the same guy who had the first AI, and the 13,000 years of civilization are just 1 millionth of world history, which is about 13.8 billion years. So it's just a flash in world history. In hindsight, it will look like that. Civilization occurred almost at the same time where AI occurred, because over these past 13,000 years, more and more stuff was automated: more and more of agriculture was automated, more and more of human labor was automated. At some point, thinking started to become automated a couple of hundreds of years ago. The first calculators, then calculators became less expensive, then they became faster, and now you can calculate much more than 100 years ago for the same price, and then suddenly it was there. From this global perspective, it's really like there was nothing and suddenly there was a law. From the perspective of a guy who is living through that age, it looks like a lot.

我认为 RSI 的希望之一是，随着时间的推移，继续取得这些进展可能最终会减少算力密集度，我想我们会看到的，但这会是其中的一个希望，对吧？

I think one of the hopes of RSI is that it may end up being less compute-intensive over time to continue making some of these developments, which I guess we'll see, but that would be one of the hopes there, right?

是的，绝对如此。每当你谈论智能时，你基本上是在谈论懒惰。一个智能体想要懒惰；它希望以最小的努力、最小的能耗来实现它所做的任何事情。所以我们所有的自我改进系统都有这种额外的奖励，用于提高效率，或者换句话说，每次它们唤醒一个神经元并使用能量来唤醒那个神经元或使用能量做其他事情时，它们都有额外的成本。所以所有的成本，计算成本和其他能源成本，都必须考虑到我们的自我改进系统正在优化的目标函数中。在它们擅长的程度上，它们会用越来越少的资源（计算资源、其他资源）做同样的事情。所以智能行为的一个自然结果是，无论做什么，都会越来越高效。

Yeah, absolutely. Whenever you are talking about intelligence, you basically are talking about laziness. An intelligent being wants to be lazy; it wants to achieve whatever it does with the least possible effort, with the least possible energy consumption. So all of our self-improving systems have this extra reward for being efficient, or in other words, they have an extra cost for every time they wake up a neuron and use energy to wake up that neuron or use energy to do other things. So all the costs, the computational costs and the other energy costs, have to be taken into account in the objective function that our self-improving systems are optimizing. To the extent that they are good at that, they will do the same thing with less and less resources, computational resources, other resources. So a natural consequence of intelligent behavior is that whatever is being done is done more and more efficiently.

当你反思 AI 实验室正在进行的更广泛的工作时，显然有大量的资金和算力投入到实验室正在研究的一系列问题上。我想知道，如果你在运营其中一个实验室，或者你和那里的人谈过，或者考虑给他们建议，你认为他们可能错过了什么？你会建议他们今天在做什么方面与现在不同？

As you reflect on the broader work going on at the AI labs, obviously there's tons of money and compute going into a bunch of the research questions that the labs are going after. I'm wondering, if you were running one of these labs, or you've talked to folks there or thinking about advising them, what do you think they're maybe missing? What would you advise them to do differently today from what they're doing?

所以今天你使用大型预训练系统，大型语言模型，它们当然已经阅读了很多关于编码的论文。然后使用这样的预训练模型的一种现代方式是让它编写代码，你只改进它编码或重新编码自己代码的方式。所以现在这是一种相当明显的方法。不少实验室对此很感兴趣。当然，你必须要有安全措施，这样它就不会以完全愚蠢的方式重写自己的代码，使事情变得更糟而不是更好。但有办法处理这个问题。

So today you use large pre-trained systems, large language models that already have read a lot of papers about coding of course. And then a modern way of using a pre-trained model like that is you let it code something, and you only improve the way it codes or recodes its own code. So that is now a pretty obvious way of doing it. Quite a few labs are interested in exactly that. And of course you have to have safeguards so it doesn't rewrite its own code in a way that is totally stupid and makes things worse rather than better. But there are ways of dealing with that.

你在那里会有什么不同的做法吗？

Is there anything you'd be doing differently there?

这是一个合理的方法。它不是最通用的方法，因为如果你依赖任何在人类生成数据上预训练的东西，那么你就忽略了其他许多可能性。看看我们当前的大型语言模型：它们超级偏向人类。为什么？因为它们是在万维网上的所有数据上训练的。万维网上的所有数据之所以存在，唯一的原因是至少一个人在某时某刻认为从人类的角度来看这很有趣。

It's a reasonable approach. It's not the most general approach in the sense that if you rely on anything that is pre-trained on human-generated data, then you are ignoring many of the other possibilities. Look at our current large language models: they are super biased towards humans. Why? Because they are trained on all the data on the worldwide web. All the data on the worldwide web is there for the only reason that at least one guy or one person at some point thought this is interesting from a human perspective.

然后，所有这些至少某些人认为有趣的素材，都被用来训练系统，因此它们极度偏向人类语言、偏向人类觉得有趣的视频、偏向人类觉得有趣的行为等等。所以它们在某种程度上与人类非常对齐。也许它们与某些人类比对其他人类更对齐。尽管如此，还是存在巨大的人类偏见。

And then all this material that at least some guy thought is interesting is used to train the systems, and they are for that reason super biased towards human language, towards videos that humans find interesting, towards behavior that humans find interesting, and so on. So they are very aligned in a certain way with humans. Maybe they are more aligned with certain humans than with others. Nevertheless, there's a tremendous human bias.

设想一个生活在未知环境中的人工科学家，它仅仅通过预测自身行动的后果来构建世界模型，然后利用这个世界模型进行规划。这样的智能体必须通过自身行动来创造训练世界模型的数据。于是，你突然有了一个类似人工科学家的东西，它通过自身行动生成用于训练世界模型的数据，这更像人类和婴儿的做法。婴儿不是通过下载网络来学习的。不，他们通过预测行动的后果来学习。比如，如果他们那样移动手指，通过摄像头传入的视频就会改变，他们学习预测这些变化，从而了解世界的物理规律、世界如何运作、手指如何工作等等。他们训练所依赖的大量数据并不来自万维网，而是通过自身实验收集的，这正是婴儿需要的数据，以便更好地理解自己能做什么，并最大化自己的奖励。

Have an artificial scientist who is living in some unknown environment and tries to build a model of the world by just predicting the consequences of its actions and then use the model of the world for planning. Such an agent will have to create through its own actions the data that trains the world model. So suddenly you have something like an artificial scientist who through its own actions generates the data on which the world model is being trained, and this is much more like what humans do, what babies do. Babies don't learn by downloading the web or something. No, they learn by predicting the consequences of the actions. So if they move their fingers like that, then the video changes which comes in through the cameras, and they learn to predict these changes, and that's how they learn about the physics of the world and about how the world works and about how their fingers work and everything. And they are being trained on a lot of data that is not on the worldwide web, you know, that is collected through their own experiments, and it's exactly the kind of data that the baby needs to better understand what it can do and that it needs to maximize its own reward.

所以你看，万维网上收集的所有数据看似很多，但与你通过自身实验可能收集到的所有数据相比，只是极小极小的一部分。因此，AI 的未来将在于这样的系统：它们通过自身行动，通过我所谓的人工好奇心，收集用于训练世界模型的数据，而这些世界模型将不依赖人类语言，并且在很多方面高度专注于收集数据的特定机器人。当然，它也会与生活在不同环境中的其他机器人通信，并整合这些知识，以便更好地泛化等等。但你知道，突然之间，系统将变得远不那么具有人类偏见。

So now you see that all of the data that is collected on the worldwide web seems to be a lot, but it's just a tiny, tiny, tiny fraction of all the possible data that you could collect out there through your own experiments. So the future of AI is going to lie in such systems that through their own actions, through artificial curiosity as I called it, collect all the data that is used to train the world models, and these world models will not depend on human language and will be very focused in many ways on this particular robot that is collecting the data. And yes, of course it is going to communicate with other robots living in different environments and it's going to incorporate that knowledge such that it can better generalize and so on. But you know, suddenly you have a situation where the systems are going to be much less human biased.

这很有意思。你刚才描述的很多内容，显然有些公司正在尝试为材料科学发现或生物学建造自动化实验室，甚至在机器人领域，也有很多人做远程操作数据收集。但显然今天还是人类在决定应该收集什么数据，或者设计那些实验并尝试反馈。我想，梦想是拥有一个完全由 AI 驱动的发现循环，对吧？

It was interesting. I mean a lot of what you just described, obviously there are companies that are trying to build automated labs for material science discovery or for biology, or even in robotics there are all these folks doing teleoperated data, but obviously today it's kind of humans deciding what the data that should be gathered is or designing those kind of experiments and trying to feed it back. And I guess the dream is to have a fully AI-driven loop of that discovery, right?

确实如此。是的，20 年前我写了一篇关于乐趣与创造力的形式理论的论文，正是关于这个的。那么，当一个人工科学家或任何科学家、艺术家、喜剧演员的基本需求（比如一日三餐）得到满足，并且有额外时间做事时，他们应该做什么？你会做什么？嗯，你听音乐，或者自己作曲，或者创作艺术，或者你是一个科学家，不仅试图解决别人给你的问题。不，你还试图发明自己的新问题。提出自己的问题。不仅仅是回答别人给你的问题，而是发明自己新的好问题。这就是科学家所做的。所有科学都涉及两件事：不仅是接受一个现有问题并投入大量时间解决它、找到答案。不，还要发明好问题。基本原理非常简单。所以一个人工科学家或任何科学家，在我看来也包括人类科学家，被一个简单的东西驱动：通过你的行动，通过你自己发明的实验，去寻找具有某种性质的数据——数据中存在一些规律性、一些你不知道但能快速学习的模式，因为它处于你几乎已经理解的极限，在你已知和未知的交界处。然后你通过自己的行动、通过你的实验（即行动序列）生成这些数据。数据进来了，如果数据中有有趣的东西，那意味着什么？意味着其中有一个你不知道的模式。所有模式都意味着存在某种在空间和时间上压缩这些事物的方式。我这里不讨论时间，以免把事情搞得太复杂。但以一种你以前不知道的方式压缩模式，这意味着在你理解规律性和进入的模式之前，你需要很多内部比特、隐藏单元等来编码它。之后，在你学会看到模式、学会压缩它之后，你只需要这么多。前后的差异，就是你所获得的乐趣。这只是一个实数，表示这个科学家现在从识别、意识到“哦，进来的数据中有我不知道的规律性，它告诉我一些关于引力的东西”中获得了多少内在喜悦。然后这成为生成导致数据的行动的控制器所获得的奖励。这意味着控制器现在有动力去生成越来越多的实验，从而获得以前没有的对世界的洞察。而它理解的一切都变得无聊。然后它想要创造更复杂的实验，在摘取低垂的果实之后，进行更复杂的实验。也许同一个婴儿，在一岁左右第一次学习引力，20 年后，这个婴儿可能在 CERN 的粒子对撞机工作，成为发现希格斯玻色子或类似东西的团队一员。唯一的区别是实验更昂贵了。

That's true. Yeah, 20 years ago I wrote this paper about the formal theory of fun and creativity, which is about exactly that. So what should an artificial scientist or any scientist or any artist or any comedian do when its main needs, like eating three times a day, are satisfied and it has extra time to do stuff? What do you do? Well, you listen to music or maybe you compose your own music or you generate art, or you are a scientist who not only is trying to solve problems given to you by other people. No, you also try to invent your own new problems. Ask your own questions. Not just answer questions given to you by somebody else, but invent your own new good questions. That's what scientists do. And all of science is about two things: it is not only taking an existing question and investing a lot of time into solving it and finding an answer. No, inventing the good questions. And the basic principle is very simple. So an artificial scientist or any scientist, in my point of view a human scientist as well, is driven by one simple thing: try to find through your actions, your own self-invented experiments, data that has the property that in the data there is some regularity, some pattern that you didn't know but can quickly learn, because it's at the limit of what you already almost understand, near the horizon of what you don't know and what you know. And then you generate this data through your own actions, through your experiments, the sequences of actions which are experiments. And the data comes in, and if there is something interesting in the data, what does that mean? It means that there's a pattern in there which you didn't know of. All patterns mean there's some way of compressing those things in space and time. I'm not talking about time right here to make things not too complicated. But to compress the patterns in a way that you didn't know before, which means that before you understood the regularity and the pattern which is coming in, you needed so many internal bits and hidden units and so on to encode it. And afterwards, after you learn to see the pattern, after you learn to compress it, you need only so many. And the difference between before and after, that's the fun that you have. That's just a real number which says how much internal joy does this scientist now have from recognizing, from realizing, "Oh, there is a regularity that I didn't know in the data which is coming in, it tells me something about gravity." And then this becomes the reward of the controller who's generating the actions that lead to the data. Which means now the controller is motivated to generate more and more experiments that lead to insights about the world that it didn't have before. And everything that it understands becomes boring. And then it wants to create more complicated experiments to, after it has grabbed the low-hanging fruits, more complicated experiments. And maybe the same baby which first learned about gravity when it was a little being, maybe a year old or something, maybe 20 years later, the same baby is working at the particle collider at the CERN and is part of the team that discovers the Higgs boson or something. And the only difference is that the experiments are more expensive.

我想，这真的是障碍吗？当你思考我们与实现这个 AI 科学家之间有什么障碍时，显然，能够以可承受的价格点进行实验，并且将反馈融入模型，是很多人正在努力的方向，但无疑仍然是一个障碍。

And I guess is that really the blocker? As you think about what stands between us and getting to this AI scientist, obviously the ability to stand up experiments and do them, both at an affordable price point, but also kind of feedback into models, is something that a lot of people are working on, but certainly remains a blocker.

而且似乎还有各种各样的算法问题需要解决，才能真正实现你所说的这种设置。你认为需要解决哪些问题才能让这个 AI 科学家成为现实？然后你觉得我们多久能实现？

And it also seems like there's all sorts of algorithmic problems to be solved to actually create this setup that you said. How do you think about what needs to be solved to make this AI scientist a reality? And then how soon do you think we'll get there?

是的。我们有简单的 AI 科学家，已经存在很长时间了。只是它们可能还没有迎来自己的 ChatGPT 时刻。你知道，ChatGPT。那是什么时候？22 年，23 年。

Yeah. Well, we have simple AI scientists. We have had them for a long time. It's just that maybe they haven't seen their ChatGPT moment yet. You know, ChatGPT. When was that? 22, 23.

是的。22 年底。

Yeah. End of 22.

它基于非常古老的东西。这些人工科学家还没有迎来类似的 ChatGPT 时刻。但人工科学家在一定程度上已经存在，并被用于许多特殊应用。例如，在化学中，你有大量的输入输出对，你训练神经网络根据之前的输入预测这些新物质。随着时间的推移，如果你给它展示数百万次实验，它就会学会成为一个人工化学家，一个直觉化学家。所以它不是一个从第一原理（比如价电子等）理解所有反应的化学家。不，它只是成为一个直觉化学家，更好地理解化学中能做什么。然后你有某些目标。例如，你想创造一种材料，其效率是已知最佳材料的两倍，比如针对某种杀虫剂。然后你可以说，好吧，让我们有一个期望的输出，编码为它应该是我知道的最佳材料的两倍效率，然后你可以反向运行整个化学家，查看输入侧，说：我应该如何改变我的实验（在输入侧可见）才能得到我想要的东西，以满足我的愿望？然后化学家基本上会给你一个建议。

It was based on stuff that was really old. We don't have the same kind of ChatGPT moment yet for these artificial scientists. But the artificial scientists to a certain extent they already exist and they are being used in lots of special applications. For example, in chemistry you have lots of pairs of inputs and outputs and you train your neural network to predict these new substances given the previous inputs. And then over time if you show it millions of experiments it learns to become an artificial chemist, an intuitive chemist. So it's not a chemist who understands all these reactions from first principles from valence electrons or whatever. No, it just becomes an intuitive chemist that better understands what can be done in chemistry. And then you have certain objectives. For example, you want to create a material that is twice as efficient against a certain insecticide or something that is twice as efficient as the best thing known so far. And then you can say okay let's have a desired output like that which encodes that it should be twice as efficient as the best thing I know and then you can work the whole chemist backwards and look at the input side and say how much should I change my experiment which is visible at the input side to get this thing that I would like to see to fulfill my wish and then the chemist will basically give you a suggestion.

你认为十年内我们能实现 AI 化学吗？

And do you think like within a decade we'll figure out like AI chemistry?

是的。在 KAUST，我们有一个有趣的项目，目标是使用某些专利结构，称为金属有机框架（MOF），目标是从稀薄的空气中提取二氧化碳，这对改善气候很重要。目前这一切都非常昂贵，目标是让它变得足够便宜，以至于真正能产生影响，或许能改善全球变暖状况。所以这是潜在应用之一，而且在各种化学领域还有很多应用。

Yeah. So at KAUST we have an interesting project where the goal is to use certain patented structures, metal-organic frameworks (MOFs) they are called, and the goal is to extract carbon dioxide from thin air, which is important for improving the climate. And at the moment all of that is very expensive and the goal is to make it so cheap that you really can make a dent and maybe improve the global warming situation. So that is one of the potential applications and there are lots of applications in all kinds of chemistry fields.

那机器人技术呢？你如何描述我们目前的状况？那里发生了什么？每个人都梦想有一个家用机器人。我们接近了吗？

What about like robotics? How do you characterize where we are? What's been happening there? And everyone dreams of an at-home robot. Are we close to that?

我记得在 70 年代，当我告诉我妈妈关于 AI 的未来以及 AI 将如何殖民整个宇宙时，她说：“来给我造一个能打扫厨房的机器人吧。”

I remember again in the 70s when I told my mom about the future of AI and how AI is going to colonize the entire universe and she said, 'Just come build me a robot that cleans my kitchen.'

古老的梦想。

The age-old dream.

是的，没错。当时那行不通，现在仍然行不通，因为机器人硬件与人体相比确实很差。没有人类制造的技术能与这只手相比。这只手充满了传感器，数百万个小传感器和小电缆连接到控制中心。我甚至不知道如何把这些电缆都放进一只人造手中。疯狂的是，你伤害它，割伤它，它会开始自我修复。这是超级先进的技术。我们在人造技术中没有类似的东西。这就是为什么电影中的机器人都是由人类扮演的，因为人类是比现有机器人好得多的机器人。

Yeah. Exactly. And back then that didn't work. And still it doesn't work because robot hardware is really inferior compared to human bodies. And there's no human-made technology that compares to this hand. This hand is full of sensors, millions of little sensors and little cables that connect to the control center. And I wouldn't even know where to put all these cables in an artificial hand. And the crazy thing is you harm it, you cut it and it starts healing itself. This is super advanced technology. We have nothing like that in man-made tech. And that's the reason why the robots in the movies are all played by humans because humans are much better robots than the robots that are available.

但这似乎远不止是硬件问题，对吧？我的意思是，即使有我们现有的硬件，如果我们有好的模型，我相信我们可以做得更多，对吧？

But it seems a lot more than a hardware problem, right? I mean, even with the hardware we have, if we had good models, I'm sure we could do way more, right?

但你知道，没有那样的硬件，你不可能拥有 AGI。你不能只在屏幕后面拥有 AGI。你可以在屏幕后面拥有一个超人类棋手，或者通过图灵测试的东西，但如果它不能掌握现实世界，它就不是 AGI。它只是一个花哨的文本编辑器，或者对机器人应该如何移动有某种想法的东西，但不是真正的东西。如果你想通过真正的 AGI（物理 AGI）掌握现实世界，还有更多事情要做。所以我们的机器人必须变得比我们今天拥有的有限东西好得多。这还需要多长时间？我们可以很容易地预测每美元算力将如何演变。它可能会保持几十年来我们看到的趋势：大约每 5 年提高 10 倍。这也意味着那些今天投资一万亿美元用于数据中心 GPU 的人，在未来五年内将损失 9000 亿美元。没有商业模式可以弥补这一损失，这或许已经预示着即将到来的崩溃。但在机器人技术中，要获得能与这只手相媲美的东西需要多长时间？这只手既能强力抓握，又能进行非常精细的手指运动，以机器人无法做到的方式操纵微小物体。这对我来说至少更难预测。它会到来，但可能不需要几年，而是需要几十年。

But you know, you can't have AGI without hardware like that. You can't have AGI just behind the screen. You can have a superhuman chess player behind the screen and something that passes the Turing test, but if it doesn't master the real world, it's not an AGI. It's just a fancy text editor behind the screen or something that has an idea of how this robot should move or whatever, but it's not the real thing. And if you want to master the real world through a real AGI, a physical AGI, so much more has to be done. So our robots have to become much better than the limited stuff that we have today. How much longer will that take? We can easily predict how compute per dollar is going to evolve. It's probably going to stay like what we have seen for decades now: a factor of 10 every 5 years roughly like that. Which means also that the guys who are investing a thousand billion dollars into GPUs for data centers today, within the next five years they are going to lose 900 billion dollars. There's no business model that can recuperate that loss, which already is an indication of the coming crash maybe. But in robot technology, how long is it going to take to get something that is comparable to this hand, which can do both strong grips and very delicate finger movements that manipulate tiny little things in a way that is infeasible for what robots can do? That is at least for me much harder to predict. It will come but it will take maybe not just a few years, it will take another few decades maybe.

好吧，我确实想转到商业方面，因为你说了一些非常有趣的话，我之前也听你说过：这种对 AI 数据中心建设的大规模投资，随着算力性能的提升，你前期投入巨资的东西将在 5 年后过时且价值低得多。对此的反驳或人们会说的是，对算力的需求将是无限的，最终我们将受限于生产足够算力的能力，以至于即使你拥有过时且效率低得多的硬件，它实际上仍会被使用，因为运行推理的需求如此之大，用于所有这些不同的有趣用例，至少今天是数字的，未来是物理的。所以即使效率较低，地球上的每一块芯片都需要以某种方式被使用。

Well, I do want to switch over to the business side because you said something very intriguing there and I've heard you say this before: that this massive investment in the AI data center build out, with the improvement in compute performance, you're investing a ton up front in something that's going to be outdated and worth far less 5 years from now. The counter-argument or what folks would say to that is that there's just going to be infinite need for compute and that ultimately we're going to be bottlenecked in our ability to produce enough compute such that even if you have hardware that's legacy and way less efficient, it's actually still going to be used because there's just going to be such demand to run inference for all of these different interesting use cases, at least today digital and in the future physical. And so even if it's less efficient, every chip on the planet will need to be used in some way.

你对此有什么反应？

What's kind of your reaction to that?

是的，也许对算力的需求会越来越大，但总得有人为此买单，对吧？如果事实证明，那些目前正在买单的人正在亏损大量资金，你知道，目前有几家公司每年投资数千亿美元。所以总共可能每年一万亿美元左右，用于数据中心的 GPU，而这些公司曾经是灵活的软件公司，有一个小团队改进某个糟糕的操作系统，然后向数十亿拥有自己手机和电脑的用户推出。突然之间，他们开始提供云服务和数据中心，不得不像电力公司这样的公用事业公司一样，投资核电站、燃气轮机等等，突然这些公司的现金流，自由现金流从 1000 亿下降到 100 亿，甚至像其中一些公司那样变成负 100 亿。所以目前，当每个人都试图抢占市场份额时，这些服务实际上效率并不高，因为提供服务的公司正在亏损大量资金。这并没有体现在市盈率上，因为真正应该考虑的自由现金流并没有显示在那里。但这些公司正变得越来越低效。所以到了某个时候，他们将不得不停止。你知道，现在他们正在举债来资助更多的数据中心，这只能在一定程度上可行，然后这些公司会变得越来越不值钱。因为他们错误地投资了资金。这意味着迟早，如果你无法为所有这些服务付费，如果整个经济体系没有准备好为所有这些服务付费，因为算力还不够便宜，而且人们想通过购买更多电脑来弥补，如果你遇到这样一种无法带来盈利经济的情况，那么根据供需规律，它将会消失。

Yeah, maybe there will be more and more demand for compute, but somebody has to pay for it, right? And if it turns out that those guys who are currently paying that they are losing a lot of money, you know, at the moment a couple of companies are investing hundreds of billions per year. So in total maybe a trillion per year or something like that into GPUs for data centers and these companies which used to be nimble software companies and had a little team improving some shitty operating system and then rolling it out for billions of people who had their own phones and their own computers to run the operating system. Suddenly they are providing clouds and data centers and suddenly they have to become like utilities like electricity companies and they have to invest in nuclear power plants and gas turbines and whatever and suddenly the cash flow, the free cash flow of these companies goes down from 100 billion down to 10 billion or maybe minus 10 billion like for some of these companies. So at the moment, while everybody is trying to get market share, these services are really not efficient in the sense that the guys who are providing the services are losing a lot of money. It doesn't show up in the price-earning ratio because the free cash flow which really should be considered is not showing up there. But all these companies are getting less and less efficient. So at some point they will have to stop. You know, now they're taking on debt to finance even more data centers and this will be possible only to a certain extent and then these companies will become less and less valuable. So they misinvested the money. That means that at some point sooner or later, if you can't pay for all these services, if the entire economy is not set up to pay for all these services because compute isn't cheap enough yet and because people want to compensate for it by buying even more computers, if you have a situation like that which doesn't lead to a profitable economy, then it's going to disappear due to the laws of supply and demand.

我的意思是，我觉得这里面其实包含了两个问题，对吧？一个是关于推理本身，你知道，是否有足够多有价值的用例，让公司愿意支付远超这些数据中心建设成本的价格？当然，在 AI 编程方面，似乎确实有，因为这是第一个高度匹配的市场，似乎至少有一些用例确实达到了那个标准。我知道在训练方面，我认为有一个大问题，我知道你之前谈到过，我想我们的听众会很想听听你的想法，关于这些闭源模型提供商，他们显然在训练上投入了巨资，这到底是不是一门生意，对吧？或者随着时间的推移，开源模型是否会迎头赶上，最终我认为这个论点有两面性，但很想听听你如何看待闭源模型提供商，比如，是否有可能通过比开源模型领先三到六个月来维持一门生意。

I mean, I guess there's kind of like two questions embedded in that, right? One is on the inference itself, you know, are there enough use cases out there that are valuable enough for companies to want to pay well above and beyond the kind of cost of these data center buildouts? And certainly with AI coding, it seems like there is, as that is the first market with a ton of fit, it seems like there's been at least some use cases there that certainly meet that bar. I know on the training side I think there's a big question and I know you've talked about this before and I think our listeners would love your thoughts on whether these closed source model providers who obviously are spending a ton on training, whether there's a business there, right? Or whether over time open source models just catch up and ultimately I think there's two sides of that argument but would love for you to share just how you think about for the closed source model providers, like is there a business staying maybe three six months ahead of the open source models.

是的，正如你所说，开源模型是大公司无法轻易提价的主要原因之一，因为它们紧随其后，某个商业大语言模型刚刚打破一项基准记录，但几个月后，就有一个开源模型赶上了。这意味着定价面临巨大压力，也意味着这些公司投资并举债购买的那些昂贵的数据中心和 GPU，目前并没有盈利。你知道，一个愚蠢的做法就是稍微等一等，等上 5 年，算力就会便宜 10 倍，你就能以十分之一的价格做同样的事情，或者等上 10 年，你就能以 1% 的价格做同样的事情。但当然，大公司会说：“哦，但如果我那样做，如果我等到别人先做，那我可能会失去市场份额什么的，或者错过成为第一个拥有真正 AGI 的机会。” 在这种情况下，一旦他们拥有了能解决所有问题的真正 AGI，那么他们就能统治世界，而投资与那个价值相比将微不足道。但我认为所有这些前景都过于乐观了，我们很可能会看到，目前这种在速度还不够快的计算机上大量投资的方式将会适得其反。

Yeah, as you say, the open source models are one of the major reasons why the big companies cannot simply raise the prices because they are so close behind and somebody some commercial large language model breaks another benchmark record here. But a few months later, there's an open source model that catches up with that. Which means there's enormous pressure on pricing, which means all these expensive data centers and GPUs where these companies invested in and took on debt to pay for that they are not being profitable at the moment. You know, a stupid way would be just wait a little bit, just wait for 5 years then compute is going to be 10 times cheaper, you will be able to do the same thing for one-tenth of the price or wait 10 years and you will be able to do the same thing for 1% of the price. But then of course the big companies say, "Oh, but if I do that, if I wait until the others do it, then I will lose maybe my market share or whatever or I miss out on the opportunity to be the first to have a true AGI or something like that." And in that case, once they have a true AGI that can solve everything, then I can take over the world and the investment will be nothing compared to the value of that. But I think all of these outlooks are over optimistic and probably we will see that the current way of investing a lot in computers that aren't fast enough yet is going to backfire.

我认为与我们之前的对话相关，有一种信念是，嘿，如果你能第一个实现递归自我改进，或者第一个拥有让开发下一代模型更容易的模型，这就会变成一种自我延续的循环，让后来者很难追上领先者。你怎么看？

I think related to our earlier conversation, I think there's this belief that, hey, if you can be the first to recursive self-improvement or the first to have models that make it easier to develop the next models that it becomes this kind of like self-perpetuating cycle that makes it really hard to catch the folks that are in the lead. What do you think of that?

并非如此，因为许多对自我改进感兴趣的人都是开源人士，而且大家都在用同样的水做饭，你知道。

Not really because many of the guys who are interested in self-improvement, they are open source guys and everybody's cooking with the same water, you know.

是的，大家都在用同样的水做饭，而且几乎所有重要的 AI 算法都不是在大公司或其他地方发明的。不，它们是在小实验室里，在资金很少的情况下发明的，尤其是递归自我改进的东西，其基本思想都来自小实验室、学术实验室，任何想要抓住这些想法并将其据为己有的公司都没有模式可循，因为外面有那么多博士生对递归自我改进超级兴奋，而且没有办法在那里保持足够长时间的小优势来赚大钱。所以我认为，大公司要想在这个行业非常盈利，将非常非常困难。

Yeah, everybody's cooking with the same water and almost all of the important algorithms in AI, they were not invented at big companies or whatever. No, they were invented at little labs and without much funding and especially the recursive self-improvement stuff and the basic ideas for that all come from little labs, academic labs, and any company that wants to somehow grab that and make it its own doesn't have a mode because there are all these other PhD students out there who are super excited about recursive self-improvement and there's no way to keep a little advantage there for long enough to make a lot of money. So I think it will be very very difficult for the huge companies to be very profitable in this business.

是的。因为基本上这些想法会渗透到更广泛的生态系统中，因此，当一个实验室达到某种程度的 RSI 时，它也会在一定程度上公开。而且，除非需要巨大的算力或某种数据分发优势，否则它就会更广泛地可用。

Yeah. Because basically like these ideas permeate throughout the broader ecosystem, and so as a result, when one lab gets to some level of RSI, it'll be kind of out in the open as well. And, unless there's some massive compute need required for it or some kind of data advantage to distribution, it'll just be kind of available more broadly.

没有人确切知道。

Nobody knows exactly.

也许你很幸运，偶然发现了一种真正好的新方法，利用你公司目前拥有的巨大资源进行增量式自我改进，然后实现 AGI，从而接管全球股市并结束整个游戏。当然，这种情况发生的可能性微乎其微，但所有迹象都指向相反的方向。相反，AI 正变得越来越便宜，参与开发新 AI 的人也越来越多。其中很多是贫穷的博士生，甚至还没有创办自己的公司。然后你将真正拥有这种“人人皆可用的 AI”局面，今天看起来令人印象深刻的一切，30 年后都将显得微不足道，因为你可以用同样的价格做一百万倍的事情。就像智能手机一样。

Maybe you are lucky and you stumble across a really good new way of conducting incremental self-improvement using the enormous resources that your particular company has for now, and then achieve the AGI that you need to take over the world stock market and finish the whole game. Of course, there's a remote little chance of that happening, but all the indications point the other direction. Instead, AI is getting cheaper and cheaper, and there are more and more people involved in developing new AIs. Many of them are poor PhD students somewhere who haven't even founded their companies yet. And then you really will have this AI for all situation where everything that seems impressive today, 30 years from now, will seem trivial because you can do a million times as much for the same price. It will be just like with smartphones.

是的。但听起来你认为一场崩盘即将来临。

Yeah. But it sounds like you think a crash is coming.

嗯，不是文明崩溃意义上的崩盘，但会出现一次股市崩盘，因为目前我认为存在大量资源错配。当然，股市会从这些错配中吸取教训，并采取不同的方法。但这不会是文明的终结之类的。不，而是对你当前状况的一次重新调整，因为目前我们只有一些超级昂贵的公司，它们实际上并没有太多可提供的。

Well, not a crash in the sense of a civilization crash, but there will be one of these stock market crashes because at the moment I think there's a lot of misallocation. Of course, the stock market is going to learn from these misallocations and it's going to pursue different approaches. But it's not going to be the end of civilization or something. No, but it's going to be a renormalization of what you currently have because at the moment we have just super expensive companies which don't really have much to offer.

我感觉你公开对 AI 安全的担忧似乎比社区里的其他人要少。我想知道，过去这些年，有没有什么事情改变了你的想法，还是你仍然持同样的立场？

I feel like you've kind of publicly been less worried about AI safety today than maybe some of the other folks in the community. And I'm wondering, these past years, has anything changed your mind at all or are you kind of still in the same camp?

我记得在 2010 年代，我们有很多这样的安全会议和对齐会议，人们写信要求禁止某些类型的递归自我改进。当然，我从未签署过任何这些信件。但那时，至少对于机器学习专家来说，这已经是一个大问题：我们如何防止 AI 变得危险？现在我认为所有这些方法在很多方面都是误导。很明显，首先，整个对齐业务意味着有人决定了这种对齐的性质。我如何将这些 AI 与人类的需求对齐？但如果一个房间里有 10 个不同的人，他们都会有不同需求，对什么对人类有益也有不同看法。所以整个观点被一种想法主导：你给这个 AI 一个目标函数让它优化，然后这就是你通过它创建的对齐。另一方面，从 1990 年左右开始，我们构建了这些人工科学家，它们并没有真正与任何东西对齐，因为它们一直在发明自己新的目标函数。它们一直在发明自己新的问题。它们改变自己的目标函数。实际上从 1990 年开始我们就有这样的系统。所以整个关于拥有不改变目标的系统的前提对我来说毫无意义。在我看来这非常幼稚。所以我没签那些信。我就是不能签。而今天，我们有这些极端案例，人们正在使用 AI 互相争斗。看看乌克兰和俄罗斯之间的战争，这边是基于 AI 的无人机，那边也是基于 AI 的无人机。没有任何对齐。当然，你不能指望在我们的世界里有一个万能政府来对齐所有这些 AI，因为所有这些不同的特勤机构和军队都有不同的目标，这些目标常常完全相互冲突。所以在我看来，2010 年代的所有这些努力都有些幼稚。另一方面，如果你想要真正聪明的 AI，你必须给它们机会设定自己的目标，就像小人工科学家一样，它们会问新问题：如果我这样做会发生什么？如果我那样做世界会如何反应？只有给它们自由去问自己的问题，解决自己发明的问题，它们才会变得真正聪明。另一方面，它们会变得不那么可预测。这当然是这里要解决的问题。现在人们问：在这样一个世界里，你有各种设定自己目标的 AI，就像我的实验室几十年来所做的那样，这难道不超级危险吗？我认为这不会比现在人类的情况更危险。人类也一直在设定自己的目标，提出新问题。是的，我的孩子是不可预测的。我不确定他们将来会做什么，但至少作为父母，我可以为让他们成为有用的社会成员做出贡献。每当他们想出坏实验，比如让我拿这个放大镜把阳光聚焦在这只小蚂蚁上，我就会惩罚他们，说这不好，你不应该这样做。这就是我教我的孩子成为理性社会成员的方式。同样的事情我们也会对我们的机器人和机器做。从长远来看，一旦它们比我们聪明得多，我认为我们可以通过它们是科学家这一事实获得某种保护。因为人工科学家会对生命、自己的起源、文明以及一切导致它们当前状态的事物超级感兴趣。它们会被生命迷住，并且会非常有动力去保护有趣模式的来源，而不是摧毁它。所以从这个高层次的角度来看，我认为有非常充分的理由你不应该太害怕终结者场景。

I remember in the 2010s, we had lots of these safety conferences and alignment conferences, and people writing letters that certain types of recursive self-improvement should be forbidden. Of course, I never signed any of these letters. But back then it was already a big deal for at least the machine learning specialists: how can we prevent AIs from becoming dangerous? Now I think all of these approaches were misguided in many ways. It's kind of obvious that first of all, this whole alignment business means that there is somebody who decides what is the nature of this alignment. How do I align these AIs to the needs of humans? But if you have 10 different humans in one room, they all will have different needs and different opinions about what is good for humans. And so this whole perspective is dominated by the idea that you give one objective function to this AI which is supposed to optimize, and that's then the alignment you create through that. On the other hand, since 1990 or so we have built these artificial scientists which are not really aligned much with anything because they all the time invent their own new objective functions. They invent their own new problems all the time. They change their own objective functions. Really starting in 1990 we had systems like that. So this whole premise of having systems that don't change their objectives didn't make any sense to me. It was very naive in my point of view. So I didn't sign those letters. I just couldn't. And today we have these extreme cases where people are using AI to fight against each other. If you look at the war between Ukraine and Russia, it's about AI based drones on this side, fighting against AI based drones on the other side. And there's no alignment whatsoever. And of course, you cannot expect in our world a universal government that is going to align all these AIs because all these different secret services and militaries, they all have different objectives which are often totally conflicting with each other. So it seems to me that all these efforts of the 2010s were kind of naive. Now on the other hand, if you want to get really smart AIs you have to give them the opportunity to set themselves their own goals, like little artificial scientists that ask a new question about what happens if I do this and how does the world respond if I do that. Only if you give them the freedom to ask their own questions and to solve their own self-invented problems, only then they will become really smart. On the other hand, they will become less predictable. That is of course the issue that is being addressed here. And now people are asking: in a world like that where you have all kinds of AIs that set themselves their own goals, like they have done in my lab for many decades, isn't that super dangerous? I think it's not going to be more dangerous than what you have now with humans. Humans also set themselves their own goals all the time and come up with new questions to ask. And yes, it is true that my kids are unpredictable. I'm not sure what they are going to do in the future, but at least I can contribute as a parent to making them useful members of society. Whenever they come up with bad experiments, like for example let me take this magnifying glass and focus the sunlight on this little ant, then I'm going to punish them and say that's bad, you shouldn't do that. And that's how I taught my kids to become reasonable members of society. And the same thing we are going to do with our robots and machines. In the long run, once they are much smarter than we are, I think we can expect a certain kind of protection through the fact that they are scientists. Because artificial scientists will be super interested by life and by their own origins and civilization and by everything that led to their current state. They will be fascinated by life and they will be greatly motivated to protect the source of interesting patterns, and they will be motivated to protect it rather than destroy it. So from this high level perspective, I think there's a very strong reason why you shouldn't be too afraid of Terminator scenarios.

我们总是喜欢在采访结束时问一连串快问快答的问题，把一堆东西塞到结尾。那么也许首先，我想给听众一些背景：你显然在职业生涯中做了很多有趣的事情。你目前是如何分配时间的，并且认为未来最有趣的工作是什么？

We always like to end our interviews with a bunch of quickfire questions where we shove a bunch of things into the end. And so maybe to start, I'd love to give our listeners context on you've obviously done so many interesting things over your career. How are you kind of splitting your time today and thinking about like what's most interesting to work on going forward?

我是个保守的人，我认为最有趣的前进方式仍然是我在 70 年代和 80 年代追求的那个：构建一个能学会变得比我更聪明的通用 AI，这样我就可以退休了。我还在做同样的事情。

I'm a conservative guy and I think the most interesting way of going forward is still the one that I pursued in the 70s and 80s when I tried to build this general purpose AI that learns to become smarter than myself such that I can retire. I'm still working on the same old thing.

你显然有一个由许多优秀人才组成的网络，他们曾在你手下工作，后来也取得了巨大成就。关于什么造就了一个真正优秀的研究者，你学到了什么？

You've obviously had this network of amazing folks that have come through and worked under you and then gone on to do great things. What have you learned about what makes a really good researcher?

我有幸合作过的许多最优秀的博士生通常都专注于一个特定的问题。我们想要解决一个大的总体问题，但总有一个当前不奏效的小细节。你必须关注细节：在这个由特定学习算法训练的神经网络中，权重是如何变化的？为什么网络没有按照你的期望行事？你必须调试一个很小的问题，然后突然发现魔鬼就在细节中。有一个你之前忽略的小细节，一旦修复，一切就都顺畅了，你就取得了突破。同一个博士生可以接连在重要会议上发表论文，仅仅因为这个小细节的解决带来了大量额外的改进。

Many of the best PhD students that I have had the pleasure to work with usually focused on a particular question. There's the general big problem that we want to solve, but then there is this particular little thing that currently doesn't work. You have to look at the details: how do these weights change in this neural network trained by this particular learning algorithm, and why doesn't the network do what you want it to do? You have to debug a little tiny thing, and suddenly you see the devil is in the detail. There is a little detail which you have overlooked so far, and if you fix that one, then suddenly everything works out and you have a breakthrough. The same PhD student can come up with one research paper at a major conference after another, just because it's such a rich source of additional improvements triggered by this little devil in the detail solution.

你认为 Transformer 在未来 5 到 10 年内会继续作为主导架构吗？

Do you think the transformer will persist as the dominant architecture over the next 5 to 10 years?

我猜某种 Transformer 会继续存在，但我认为它会更像我们在 1991 年提出的高效 Transformer，即线性 Transformer。今天它被称为未归一化线性 Transformer，我称之为快速权重控制器。它的有趣之处在于它线性扩展，这意味着如果你有 1000 倍的文本，你只需要 1000 倍的算力，而 2017 年的二次 Transformer，如果你有 1000 倍的文本，你需要 100 万倍的算力。所以每个人都对此感到担忧。这也是这些数据中心现在如此昂贵的原因之一。你必须做大量的计算，每个人都想将这种复杂度降低到更合理的线性复杂度或对数线性复杂度。有很多 Transformer 变体正在朝这个方向发展，或者 Sepp Hochreiter 正在做的 xLSTM，它结合了旧线性 Transformer 的某些方面。你想降低复杂度，这与你之前问的问题有关。智能就是用更少的努力做同样的事情。我们想减少努力。

I guess some sort of transformer will, but I think it's going to be a little bit more like the efficient transformer that we had in 1991, the linear transformer. Today it's called the unnormalized linear transformer. I call it the fast weight controller. The interesting thing about it is that it scales linearly, which means that if you have 1,000 times more text, you need 1,000 times more compute, while the modern quadratic transformer of 2017, if you have 1,000 times more text, you need 1 million times more compute. So everybody's worried about that. It's one of the reasons why these data centers are now so super expensive. You have to do so much compute, and everybody wants to bring down this complexity to a more reasonable linear complexity or maybe log-linear complexity. There are quite a few transformer variants going in this direction, or there is stuff that Sepp Hochreiter is doing, the xLSTM, which combines aspects of the old linear transformers. You want to get the complexity down, and it's related to what you asked before. Intelligence is about doing the same thing with less effort. We want to reduce the effort.

我很喜欢这个观点。这是一次引人入胜的对话。我相信有很多线索大家会想深入探讨。我想把最后一句话留给你。大家可以去哪里了解更多你的工作？话筒交给你。

Well, I love that. This has been a fascinating conversation. I'm sure there's a ton of threads that folks will want to pull on. I want to make sure to leave the last word to you. Where can folks go to learn more about your work? The mic is yours.

在哪里可以了解我认为有趣的东西？看看我的博客。很容易谷歌到，如果你搜索 Jurgen。

Where can you learn what I think is interesting? Look at my blog. It's easy to Google if you Google for Jurgen.

是的，我们会附上链接，你会找到的。

Yeah, we'll link to it and you will find it.

那里有概述页面，链接到关于元学习、人工好奇心、乐趣与创造力的形式化理论的原始论文。还有关于我们领域的历史。机器学习领域是关于信用分配的科学，并将其应用于领域本身。谁发明了卷积神经网络？谁发明了深度学习？所有这些都很好地列出并解释了。这也解释了我认为最重要的东西，包括我刚才提到的。重要的是屏幕外的物理 AI，不是屏幕后的，而是现实世界中的。一旦我们有了机器人，几百年来人们一直在谈论自我复制机器，但没有人知道如何实现，但现在我们看到了一个突破口。第一次，我们现在可以拥有，或者很快将拥有机器人，它们可以通过模仿或强化学习来操作现有的机器，所有已经是我们文明一部分的机器。一旦你有了一个可以操作人类目前操作的所有机器的机器，你就拥有了一种新的生命。然后你就有了实现这种终极 Scaling 机器的方法，因为你可以让机器人制造更多自己。我已经说了几十年，现在它越来越接近现实。你不需要超级智能的机器人来做这件事，只需要足够聪明来学习操作所有现有的机器。这样的一组机器可以制造更多自己，也可以改进自己，而不仅仅是复制自己，因为我们在虚拟世界中为屏幕后的 AI 已经拥有的所有机器学习概念，所有这些概念都将应用于自我改进的机器人社会。这样的东西不仅会在生物圈中工作，还会在月球或水星上工作，那里有大量材料用于建设基础设施、更大的 AI、更多的 AI、更多的机器人、巨大的宇宙飞船以及我们殖民太阳系所需的各种东西。

And there are overview pages with links to the original papers on meta-learning, on artificial curiosity, on the formal theory of fun and creativity. Also on the history of our field. The field of machine learning is about the science of credit assignment and apply that to the field itself. Who invented convolutional neural networks and who invented deep learning? All that stuff is nicely listed and explained there. That also explains what I consider the most important stuff, including what I just mentioned. The important thing is physical AI outside the screen, not behind the screen but in the real world. Once we have a robot, for hundreds of years people have talked about self-replicating machines but nobody had any idea how to get there, but now we see an opening. For the first time, we now can have, or maybe we will soon have, robots that can learn through imitation or reinforcement learning to operate the existing machines, all the existing machines that are part of our civilization. Once you have a machine that can operate all the machines that humans currently are operating, then you have a new kind of life. Then you have a way of implementing this ultimate scaling machine because you can have robots that make more of themselves. I've said that for decades and now it's getting closer to reality. You don't need super smart robots for doing that, just smart enough to learn to operate all the existing machines. A collection of machines like that can make more of itself and can also improve itself, not only make replicas of itself, because all the concepts of machine learning that we already have for AI behind the screen in the virtual world, all these concepts we are going to apply to self-improving robot societies. Something like that is not only going to work in the biosphere but also on the moon or on Mercury, where there is a lot of material for building infrastructure and bigger AIs and more AIs and more robots and huge spacecraft and all kinds of stuff that we need to colonize the solar system.

我认为这是结束的完美方式。对未来的愿景非常令人兴奋。真的，非常感谢你抽出时间在这里畅谈一切。能有机会交谈是我的荣幸，我知道我们的听众也会喜欢。

Well, I think that's the perfect note to end on. Incredibly exciting vision for the future. Seriously, thank you so much for taking the time to chat through everything here. It's such a privilege to get a chance to talk, and I know our listeners will enjoy it too.

这是我的荣幸。谢谢你，Jacob。

It was my pleasure. Thank you, Jacob.

我是 Jacob Efron，这里是 Unsupervised Learning，一个播客，我有机会与 AI 领域最聪明的人交谈，问他们关于模型发展及其对世界商业意义的大量问题。我希望这很清楚，我对此乐在其中。这是我除了在 Redpoint 做投资人的日常工作之外的夜晚和周末项目。但我们能请到这些了不起的嘉宾，真的来自于像你这样的听众订阅播客、与朋友分享。这最终是让这一切运作起来的原因。所以请考虑这样做。非常感谢你的支持和收听。我们下期再见。

I'm Jacob Efron and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. So please consider doing that. And thank you so much for your support and listening. We'll see you next episode.