AI 先驱谈智能未来与自我复制机器人

AI 模型还能进步多少？它们越来越聪明，这又说明了我们的大脑和存在本身什么？我们请到了 AI 先驱、《卫报》所称的“AI 之父”尤尔根·施密德胡伯，稍后就来聊这些。欢迎收听 Big Technology 播客，一档关于科技世界及更广阔领域冷静、细致对话的节目。今天我们有一期很棒的节目，正好探讨 AI 走向何方、这项技术还有多大提升潜力，以及它日益增长的智能对我们、我们的存在、我们的大脑意味着什么。我们非常荣幸请到了 AI 先驱之一、《卫报》所称的“AI 之父”尤尔根·施密德胡伯教授，他今天从阿姆斯特丹加入我们。教授，很高兴见到您，欢迎来到节目。

How much better can AI models get from here? And what does their increasing smarts say about our brains and existence itself? We'll talk about it with AI pioneer and someone that the Guardian has called the father of AI, Jurgen Schmidhuber, right after this. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. We have a great show for you today looking exactly into where AI is heading, how much potential the technology has to improve after this, and also what its increasing smart says about us, our existence, our brains. And we are thrilled to be joined by one of the pioneers of AI, someone that the Guardian has called the father of AI, Jurgen Schmidhuber, a professor who is joining us today from Amsterdam. Professor, great to see you. Welcome to the show.

亚历克斯，我很荣幸来到这里。

Alex, it's my pleasure to be here.

那么，我们先简单聊聊您自己以及您对 AI 现状的贡献。显然，您对 AI 做出了很多重大贡献，比如记忆、P 和 T 以及 GPT，正如您的推特简介所说。《卫报》称您为“AI 之父”。媒体似乎倾向于给研究人员贴这些标签。AI 领域有好几位“之父”和“教父”。您对此如何回应？您又如何定位自己的角色以及当今技术的现状？

So, let's just speak a little bit about yourself and your contributions to where AI stands. You've obviously made a lot of big contributions towards AI, things like memory, the P and the T and the GPT, as your Twitter bio says. The Guardian has called you father of AI. It seems like the media tends to put these labels on researchers. There's a handful of fathers and godfathers of AI. How do you respond to that and how would you contextualize your role and where the technology is today?

没有任何一个人能独自创造 AI。你需要整个文明来构建 AI。你不仅需要那些试图发明算法、人工神经网络学习算法的人——这就是当前 AI 的核心；你还需要那些制造更好计算机的人。你需要所有视频游戏玩家，他们创造了市场，促使人们购买更多越来越快的计算机，并激励计算机制造商每五年将每美元的计算速度提升 10 倍。你还需要所有养活这些玩家的农民，等等。所以，单靠一个人不可能创造 AI，你需要整个文明。

No single person can create an AI by himself or herself. You need an entire civilization to build an AI. You need not only the guys who are trying to invent algorithms, learning algorithms for artificial neural networks. That is what current AI is about. You also need people who build better computers. You need all the video gamers who are creating a market for acquiring more of these faster and faster computers, and providing an incentive to the computer makers to speed up the computation per dollar by a factor of 10 every five years. You need all the farmers who are feeding the video gamers and so on. So it is impossible for a single person to create an AI. You need an entire civilization.

那么，我很想听听您对这个文明正在朝着什么方向前进的看法。显然，您长期在 AI 研究一线工作，而今天我们无疑处于快速进步的时期。所以一直有这些问题：AI 还能变得多好？它是否会撞墙？当前技术会走向何方？对于 AI 从这里能走向何处，您的立场是什么？

And so I'm curious to hear your perspective about where this civilization is working toward. You have been obviously in the trenches working on AI research for a long time, and we're definitely in a period of fast progress today. So there's been all these questions about how much better AI can get and whether it's going to hit a wall and where current techniques will lead. Where do you land on that question about where AI can go from here?

自 20 世纪 70 年代以来，我一直是个乐观主义者，我一直声称在我有生之年，我想建造一个学会变得比我更聪明的 AI，这样我就可以退休了。显然，我们还没有达到那个目标。目前，唯一运行良好的 AI 是屏幕背后的 AI。是的，屏幕后面有一个能通过图灵测试的 AI。那是什么意思？意思是如果你向它输入问题，它会回答。现在的问题是，你能不能分辨出对方是人还是机器？如今，许多 AI 都通过了图灵测试。然而，这恰恰说明图灵测试衡量智能的方式很糟糕，因为在屏幕之外的物理世界里，没有 AI 能做一个小孩能做的所有事情，能做一个水管工几十年来能做的事情。我一直用水管工或电工来举例。所以，人类用双手、用物理的手能做的一切，AI 都做不好。只有屏幕后面的比特、0 和 1，那是唯一运行良好的东西。然而，这种情况不会永远持续下去，物理世界的 AI 也在进步。我认为，这个顶点——我已经反复谈论了几十年——就是在不远的将来，我们会拥有一个机器人。它不必超级聪明，但只要能学会操作所有现有机器就够了。一旦我们有了这样的机器人，我们就拥有了一种新的生命。几百年来，人们一直在谈论自我复制机器，但现在我们有了一个突破口。一旦我们有了这样的机器人，它就可以开始制造更多同类。这意味着，突然之间，我们将拥有我所说的终极 Scaling 机器——自我复制机器，同时也是自我改进机器，因为所有在屏幕后面软件上运行良好的东西，也将提升物理世界中所有机器的性能。我认为这将是一个拐点，从那里将涌现出一种新的文明，因为这样的东西不仅在生物圈中有效，在太空、月球上也会有效，但也许更可能首先在水星上，然后是太阳系的其他地方，乃至更远。

So since the 1970s I have been an optimist and I have been claiming that within my lifetime I want to build an AI that learns to become smarter than myself so that I can retire. And we are obviously not there yet. At the moment, the only AI that is working well is the AI behind your screen. Yes, behind the screen there's an AI that can pass the Turing test. What is that? It means if you type questions to it, it answers back. And now the question is can you distinguish whether the other guy is a human or a machine? And today this Turing test is passed by many AIs. However, it just means that the Turing test is a bad way of measuring intelligence because there is no AI in the physical world outside of the screen that can do all the things that a little boy can do, that can do the things that a plumber can do for decades. I have used the plumber as an example or an electrician. So all the things that humans can do with their hands, their physical hands, they don't work well. Only bits and zeros and ones behind the screen. That's the only thing that is working well. However, it's not going to stay like that forever and there is progress in the physical world AI for the physical world. And I think the culmination point, which I've been talking about again for decades, is at some point in the not-so-distant future we will have a robot. So it doesn't have to be super smart but just smart enough to learn to operate all the already existing machines. Once we have a robot like that we have a new kind of life. For hundreds of years people have talked about self-replicating machinery but now we have an opening. Once we have a robot like that then it can start making more of its own kind. And this means suddenly we'll have this ultimate scaling machine as I've called it, self-replicating machinery but then also self-improving machinery because all the stuff that already works well for a software behind the screen is going to also improve the performance of all machinery in the real world in the physical world. And that's I think going to be this inflection point and from there a new kind of civilization is going to emerge because something like that doesn't work only in the biosphere, it also works out there in space on the moon but maybe more likely first on Mercury and then in the rest of the solar system and beyond.

好的。关于这一点我有很多问题，包括这对“作为人意味着什么”的本质会有什么影响。但我们现在先不谈这个。您的回答让我立刻想到一个后续问题，那就是我最近在 OpenAI 总部与 OpenAI 总裁格雷格·布罗克曼交谈过。

Okay. So, I have many questions about this, including what this would mean for the nature of what it means to be human. But let's leave that for now. Your answer sparks an immediate follow-up for me, which is I was recently at OpenAI headquarters speaking with Greg Brockman, the president of OpenAI.

就在那时，他们决定暂停对视频生成器 Sora 的研究，几乎完全专注于这些 GPT 模型，对吧？就是我们最近看到取得巨大进展的文本模型。我问布罗克曼，不专注于这些更偏向世界模型的应用（比如视频生成），你们不会失去些什么吗？他说，我们确实会失去一些东西。我们不可能什么都做。我们相信 GPT 风格的模型是通往 AGI 或与人类智能相当的人工智能的道路。

And it was in the moment that they had decided that they were going to pause research on Sora, their video generator, and decide to focus almost entirely on these GPT models, right? The text models that we've seen make so much progress recently. And I asked Brockman, don't you lose something by not focusing on these more world model style applications like video generation? And he said, we do lose something. We can't do everything. And we believe that the GPT style models are the way to get to AGI or AI on par with human intelligence.

你刚才那样回答，你觉得他说得对吗？仅靠 GPT 模型就有机会实现 AGI，还是说他们放弃世界模型的做法是个根本性错误，尽管 OpenAI 内部仍有一些机器人方面的工作？

Do you think, having answered the way that you did, that he's correct that there is a chance to get to AGI using just the GPT models, or are they making a fundamental mistake by abandoning that world model practice even though there's still some robotics work within OpenAI?

是的。当问题这样提出时——一个 LLM，一个大型语言模型本身能否通向 AGI——答案显然是否定的。但你知道，OpenAI 有一群聪明人，他们完全清楚这一点。当然，他们也清楚你可以用基础模型或大型语言模型之类的东西做什么。你可以把它用作世界的模型，即世界模型。我在 1990 年使用的那类神经网络也可以用作世界模型，它只是学习预测另一个决策者（另一个生成修改环境动作的神经网络）行为的后果。就像一个小婴儿，你知道，婴儿不是通过下载网络来学习的，那是 ChatGPT 做的事。婴儿不是通过下载网络来学习的。不，它是通过自己发明的实验来创造自己的数据流。例如，当婴儿这样做时，通过摄像头进来的视频每几毫秒就在变化，数以亿计的新像素涌入。然后婴儿有一个内部机制，我们称之为世界模型，它学习预测这些变化。一开始，它甚至不知道自己有一只手。但随着时间的推移，它学会预测向运动神经元发送特定动作信号的后果，然后言语肌肉上下运动或手在移动，它通过自己实验产生的数据来了解世界是如何运作的。所以婴儿有点像物理学家。物理学家不是通过下载网络来学习的。他通过下载论文学到一点东西。但物理学家做的是生成新的实验，这些实验产生以前从未存在或从未收集过的数据，以更好地验证关于宇宙的某些假设，然后更好地理解世界，物理世界。这就是我们自 1990 年左右以来的人工神经网络所做的，只是还没有物理学家那么令人印象深刻。但我认为它会达到那个水平。

Yeah. So when the question is phrased like this, does an LLM, a large language model by itself lead to AGI, the answer is a clear no. But you know, OpenAI has a bunch of smart people and they know that exactly. And of course they know also what you can do with a foundation model or a large language model or something like that. You can use it as a model of the world, as a world model. The same type of neural networks that I used there can be used as a world model, as I called it in 1990, which just learns to predict the consequences of the actions of another decision maker, of another neural network that is generating actions that modify the environment. Like a little baby, you know, a little baby doesn't learn by downloading the web, that's what ChatGPT does. A baby doesn't learn by downloading the web. No, it learns by creating its own data stream through its own self-invented experiments. For example, when the baby does this, then the video which changes coming in through the cameras, you know, every few milliseconds, hundreds of millions of new pixels coming in. Then the baby has an internal mechanism, let's call it the world model, which learns to predict these changes. And in the beginning, it doesn't even know that it has a hand. But then over time it learns to predict the consequences of sending certain action signals to its motor neurons, and then the speech muscles go up and down or the hand is moving, and it learns how the world works through the data that it is generating through its own experiments. So a baby is a little bit like a physicist. A physicist doesn't learn by downloading the web. He learns a little bit by downloading papers. But what the physicists do then is they generate new experiments that lead to data which has never been there before or has never been collected before, to better verify certain hypotheses about the universe and then to better understand the world, the physical world. And that's what our artificial neural networks since about 1990 also do in a way that is just not yet as impressive as what physicists do. But I think it's going to get there.

然而，至少有两个组成部分。有一个基础模型，可以是各种人工神经网络中的任何一种。它只是一个预测机器。如果我这样做会发生什么？如果我看这些数据到目前为止，下一个词是什么？然后还有另一个东西，另一个神经网络，控制器，它利用第二个家伙，即预测机器，来进行规划。这样，一旦第二个家伙，即预测机器，即世界模型，变得相当好，它就能学会预测复杂动作序列的后果，然后它会为控制器选择一个能带来大量预测奖励和少量预测痛苦的行动序列。所以当然，我们的机器人有疼痛传感器，每当它们撞到障碍物时，负面惩罚，负数就会进来，世界模型不仅预测中性信号（如视频等），还预测这些价值信号、奖励信号等。然后，如果模型很好，你就可以用它来进行规划，控制器就可以利用心理实验而不是真实实验（真实实验非常昂贵）来选择好的行动序列。但在这些情况下，你看，仅靠模型，仅靠基础模型，并不是人工智能。不，你需要另一个家伙，它利用各种技巧来挖掘世界模型中的算法信息，以提出更好的计划。

Nevertheless, there are then at least two components. There's a foundation model, which can be any of a variety of artificial neural networks. It's just a prediction machine. What happens if I do that? What is the next token if I look at this data so far? And then there's the other thing, the other neural network, the controller, which uses the second guy, the prediction machine, to plan. So that once the second guy, the prediction machine, the world model, is pretty good, then it can learn to predict the consequences of complicated action sequences, and then it's going to select for the controller an action sequence that leads to a lot of predicted reward and little predicted pain. So of course our robots get pain sensors, and then whenever they bump against an obstacle, negative punishment, negative numbers are coming in, and the world model predicts not only the neutral signals like video and so on, but also these value signals, these reward signals, and so on. And then if the model is good, then you can use it for planning, and the controller then can use mental experiments instead of real experiments, which are really expensive, to select action sequences that are good. But in those cases, you see, the model alone, the foundation model alone, is not an AI. No, you need the other guy, which uses all kinds of tricks to exploit the algorithmic information in the world model to come up with better plans.

所以，在 2017 年，大约在我们讨论的那个时候，你说过一些话。你说在不久的将来，我将能够和一个小机器人对话，通过演示和讲解教它做复杂的事情，比如组装智能手机、制作 T 恤，以及所有那些目前由发展中国家贫困儿童在类似奴隶制条件下完成的事情。人类将活得更长、更快乐、更健康、更轻松，因为许多目前对人类要求很高的工作将被机器取代。然后会有数万亿种不同类型的人工智能，一个快速变化的复杂人工智能生态系统，以人类甚至无法跟上的方式扩展。所以，差不多 10 年前，你说了那些话。现在看起来显然比当时更可信。你认为我们离那个未来还有多远？

So, in 2017, around the time we're talking about, you said something. You said this in the not-so-distant future, I will be able to talk to a little robot and teach it to do complicated things such as assembling a smartphone just by show-and-tell, making t-shirts and all these things that are currently done under slave-like conditions by poor kids in developing countries. Humans are going to live longer, happier, healthier, and easier lives because lots of the jobs that are now demanding on humans are going to be replaced by the machines. Then there will be trillions of different types of AIs and a rapidly changing complex AI ecology expanding in a way where humans cannot even follow. So, almost 10 years ago, you said that. It actually seems more plausible now than obviously it did back then. How far away do you think we are from that future?

是的，这是个好问题。实际上，甚至更早，在 2014 年，因为当时我说的那些话，以及我在演讲等场合说的，我们成立了一家公司 Nissance，它真正关注的是使用世界模型在现实世界中的物理人工智能，然后利用世界模型更好地与世界互动。我们和一些非常著名的公司签了疯狂的合同。尽管如此，这可能还是太早了，就像我们做过的其他一些事情一样。所以可能对那个时代来说太雄心勃勃了。但现在我们越来越接近了。主要的障碍，真正的障碍，是硬件方面的进展。硬件方面有一种进展是巨大的，自 1941 年以来一直是巨大的，那就是每 5 年计算机便宜 10 倍。所以在 1941 年，Zuse 的第一台通用程序控制计算机每秒可能只能做一次运算。30 年后，同样的价格，每秒能做一百万次运算。今天，同样的价格，我们几乎，还没有完全达到每秒十亿亿次指令。但今天的机器人，并不比 30 年前的机器人好一百万倍。你知道，例如，25 年前，我们已经有会走路的机器人了。它们必须比今天的机器人走得更小心。

Yeah, that's a good question. So, actually even earlier in 2014, because of these things that I said back then and when I gave talks or whatever, we formed a company Nissance, which was really about physical AI in the real world using world models and then exploiting the world models to better interact with the world. And we had crazy contracts with really famous companies. Nevertheless, this was probably still too early, like some of the other things we have done. So probably too ambitious for that time. But now we are getting closer and closer. And the main hindrance, the main obstacle, is really progress on the hardware side. So there is one type of progress on the hardware side which is enormous, which has been enormous since 1941, which is every 5 years computer is getting 10 times cheaper. So in 1941, the first general-purpose program-controlled computer by Zuse, he could do maybe one operation per second. And then 30 years later, for the same price, one was able to do a million operations per second. And today we almost, not quite, have a billion billion instructions per second for the same price. But the robots of today, they are not a million times better than the robots that we had 30 years ago. You know, for example, 25 years ago, we already had walking robots. They had to walk more carefully than today's robots.

你知道，当年日本的 ASIMO 机器人——日本当时拥有全世界一半以上的机器人。它们必须始终保持重心在脚掌上方，所以不如现代动态行走那么先进。比今天的机器人差一点，但差距不是一百万倍，可能也就三倍左右。所以硬件——我们试图为人形机器人打造的人工手、人工身体——其进化速度远不如每美元算力的提升。我认为这才是主要问题。早在 20 年前，我们就有像婴儿一样的小机器人，iCub，由意大利北部的一个实验室建造，外形像婴儿。它会自己设计实验，自己设定目标，试图理解世界如何运作，并以层次化的方式来做。但执行了三个自创实验后，某个手指的肌腱就断了，需要技术人员来修理。做这一切的成本太高了。我们真正需要的是能与人类之手匹敌的硬件。但没有人类设计的技术能与这双手竞争。这双手有数百万个传感器，以及各种连接传感器与控制中心的线缆。我不知道该把线缆放在哪里。而且如果我割伤它，它会自行愈合。所以我的手和你的手——那是超级先进的技术。它完全超出了人类用传统机器人技术所能建造的范畴。所以在这方面还有很长的路要走。我仍然希望它能在我的有生之年实现，让我在 70 年代十几岁时做出的预测成真。但显然，硬件进化比 GPU 进化慢得多。

You know, the ASIMO robots in Japan back then—Japan had more than half of the robots in the world. They always had to keep their center of gravity above the foot, so it wasn't as advanced as modern dynamic walking. It was a bit worse than today's robots, but not by a factor of a million—maybe a factor of three or so. So the hardware—the artificial hands, the artificial bodies we're trying to build for humanoid robots—is evolving much less rapidly than compute per dollar. And I think that's the main problem. Already 20 years ago, we had little baby-like robots, the iCub robot, which was constructed by a lab in northern Italy and looks like a baby. It invented its own self-invented experiments, set its own goals, tried to figure out how the world works, and did that in a hierarchical way. But after executing three self-invented experiments, some tendon in a finger broke, and a technician had to come and fix it. It was just so expensive to do all that. What we really need is hardware that can compete with human hands. But no human-designed tech can compete with these hands. These hands have millions of sensors and all kinds of cables connecting the sensors to the control center. I wouldn't know where to put all the cables. And if I cut it, it starts healing itself. So my hand and yours—that's super advanced technology. It's completely beyond what humans can build with traditional robot technology. So there's still such a long way to go. I'm still hoping it will happen within my lifetime, to make true the prediction I made in the 70s when I was a teenager. But clearly, hardware evolution is much slower than GPU evolution.

对，看来这方面还需要更多时间。现在问你一个问题：随着技术不断进步，你认为谁会捕获最大的价值？我记得有一句话是你说的：不是几家大公司主导一切，AI 最大的获利者将是小人物。但现在看来，如果你看看目前的趋势，那些大实验室将拥有万亿美元估值，周围还有大型科技公司。看起来他们捕获了大部分经济价值，而所谓的“小人物”比以往任何时候都更不确定。你对此怎么看？

Right, looks like it's going to take a lot more time on that front. Now, question for you: who do you think is going to capture the most value as the technology continues to improve? I think this is a quote attributed to you: you said it's not a few big companies that are going to dominate everything; the greatest profiteer of AI is going to be the little man. But it does seem like, if you look at where things are heading right now, you've had these big labs that are going to have trillion-dollar valuations, and the big tech companies surrounding them. Looks like they're capturing most of the economic value, and the quote-unquote little man is about as uncertain as they've ever been. So what do you think on that front?

是的，我认为这反映了当前经济某些方面的泡沫。如果你看看硅谷现在这些“僵尸独角兽”，有很多公司官方估值超过十亿美元，但如果上市，可能只值 2000 万左右。所以它们被称为“僵尸独角兽”。再看看最显眼的公司，比如 Anthropic 和 OpenAI，他们在这些东西上花了太多钱。再看看谷歌和微软，看看资本支出，这直接影响现金流——显然没有影响这些公司的市盈率，但应该影响。因为曾有一段时间，谷歌和微软的自由现金流在千亿美元量级，但现在降到了 200 亿、100 亿；有些公司突然出现负 100 亿现金流。他们举债为数据中心的所有 GPU 融资。所以这些公司正变得越来越像公用事业公司。对吧？因为突然之间，这些曾经灵活的软件公司——可能只有 10 人小团队来改进某个糟糕的操作系统，然后向数十亿用户推广，用户用自己的电脑、智能手机运行操作系统——突然这些公司不得不考虑购买燃气轮机、投资核电站、举债，做电力公司——公用事业公司——所做的一切。现在再考虑一下，每 5 年，算力成本降低 10 倍，这仍然成立。如果你今天投资 1 万亿美元用于数据中心的 GPU，这意味着 5 年内你将损失 9000 亿美元。在不久的将来，有人会损失 9000 亿美元。因为没有商业模式。没有人有商业模式来弥补这些损失。而且没有一家公司有模式，因为每当有新的打破基准的语言模型出现，几个月后，同样的东西就会出现在开源社区。所以保持低价的压力如此之大，没有一家公司有模式。所以让我们等一等，直到这些万亿美元公司的强制 ETF 购买结束。目前，万亿美元甚至更高的公司出现在股市上，指数基金不得不突然买入，因为纳斯达克等改变了规则。通常你要等一年左右，让股市自行发现这家公司的真实价值，但目前没有这样做。所以突然之间，ETF、养老基金，所有人都必须买入。一旦结束，让我们看看当创始人试图出售股份时会发生什么。如果我们看不到股价的巨大波动，那才奇怪。

Yeah, I think it's just a reflection of the current bubble we have in certain aspects of the economy. If you look at all these zombie unicorns in Silicon Valley now, there are lots of companies that officially have a billion-dollar-plus valuation, but if they were on the stock market, they'd probably be worth just 20 million or so. So they're called zombie unicorns. And if you look at the most visible companies like Anthropic and OpenAI, they're spending so much money on all that stuff. And if you look at Google and Microsoft, look at the capex spending, and this directly affects cash flow—apparently it doesn't affect the price-earnings ratios of these companies, but it should. Because there was a time when Google and Microsoft had on the order of a hundred billion dollars in free cash flow, but now it's down to 20 billion, 10 billion; some companies suddenly have minus 10 billion cash flow. They take on debt to finance all these GPUs in the data centers. And so these companies are becoming more like utilities. Right? Because suddenly these formerly nimble software companies, who had maybe a small team of 10 people to improve some shitty operating system and roll it out for billions of people who all use their own computers, their own smartphones to run the operating system—suddenly the same companies have to think about buying gas turbines, investing in nuclear power plants, taking on debt, and doing all the things that electricity companies do—utility companies. And now take into account that every 5 years, it's still true, every 5 years, compute is getting 10 times cheaper. Now, if you invest $1,000 billion today into GPUs for data centers, that means within five years, you're going to lose $900 billion. Somebody is going to lose $900 billion in the near future. Because there is no business model. Nobody has a business model to recuperate all these losses. And none of the companies has a model, because whenever there's a new benchmark-breaking language model that does this or that, a few months later, the same thing appears in open source. So there's so much pressure to keep prices down, and none of these companies has a model. So let's wait a little bit until the forced ETF buying of these trillion-dollar companies is over. At the moment, we have trillion-dollar or even more companies appearing on the stock market, and the index funds have to buy them suddenly because the NASDAQ and others change their rules. Normally you wait for a year or so until the stock market finds out by itself what the true value of this company is, but at the moment this is not done. So suddenly the ETFs, the pension funds, everybody has to buy. Once that is over, let's see what happens when the founders try to sell their stakes. It would be astonishing if we wouldn't see some huge fluctuations in the stock market prices here.

对，那么你对小人物的最终归宿有什么愿景？

Right, so where's your vision of where the little guy can end up?

小人物，从长远来看，将会获利。所以目前，大公司和每个人都说：“哦，大公司，他们赚疯了。”但这是真的吗？看看他们的现金流。不，它们在下降。看看他们的负债率。他们已经停止回购自己的股票，因为他们没有足够的现金了。

The little guy, in the long run, is going to profit. So at the moment, the big companies and everybody says, "Oh, the big companies, they are profiting like crazy." But is that true? Look at their cash flows. No, they go down like that. Look at their debt ratios. They have stopped repurchasing their own shares because they don't have enough cash any longer.

所以与此同时，小人物要做的只是等一等，因为每 5 年计算机便宜 10 倍，这意味着 10 年后你可以用 1% 的价格买到同样的东西。这就像智能手机一样。我经常讲一个 80 年代我认识的富人的故事。他有一辆保时捷，他很有钱，但最神奇的是保时捷里有一部移动电话。他可以拿起听筒，通过卫星和另一个开保时捷的人通话。而今天，40 年后，发展中国家的每个人都有一部智能手机，其性能远超他当年在保时捷里的那部。所以小人物几乎不用花钱就能拥有当年极其昂贵的东西。AI 也会如此。很快 AI 就不会在云端了。它会在你的本地电脑上，一台小小的电脑。它的能力将和现在的云端一样强大，但你不需要连接互联网，这总是让人担心，谁知道外面有谁在等着你连接。到那时，即使是穷人也会拥有许多 AI，我想也包括实体 AI，但尤其是软件 AI，它们会让他的生活更长、更轻松、更健康，而且他拥有它们。他不必再付钱给别人。

And so at the same time, what the little guy has to do is just wait a little bit, you know, because every 5 years computers get 10 times cheaper, which means in 10 years you can buy the same thing for 1% of the price. And it's going to be just like with smartphones. I often relate the story of the rich guy I knew in the 80s. He had a Porsche, he was rich, but the most amazing thing was in the Porsche there was a mobile phone. He could pick up the receiver and talk via satellite to another guy with a Porsche like that. And today, 40 years later, everybody in developing countries has a smartphone that is immensely more powerful than what he had in his Porsche. So the little guy is paying almost nothing for whatever he had there, which was really expensive. And the same thing is going to be true with AI. Soon AI will not be in the cloud. It will be local on your local computer, a small little computer. It will be as powerful as what's now in the cloud, but you won't have to connect it to the internet, which is always worrisome, you know, when who knows who out there is just waiting for you to connect. And then even the poor guys will have many AIs, also physical AIs I think, but at the moment especially software AIs that are going to make his life longer, easier, and healthier, and he will own them. He won't have to pay money to other guys.

我喜欢那个未来。好，我们快速休息一下，回来后聊聊 AI 的心智与身体理论，以及 AI 是否能感受疼痛，我们是否有自由意志。这些内容马上就来。大家好，我是 Alex Canitz。我想告诉大家，我和 Gravity 合作制作了一部纪录片，探讨 AI 智能体安全的未来。为了了解我们是否真正准备好迎接自主智能体，我采访了 MIT 教授 Romesh Rosker、前白宫 CIO Terresa Payton、米其林集团首席数据与 AI 官 Amba Roger Gopal，以及前阿里巴巴高管 Sharon Guy。他们对这一不断演变的领域各有独到见解。最后，我们与 Gravity 的 CEO Rory Blendell 一起探讨前进之路，Gravity 引领方向。加入我们的旅程吧。你可以在节目说明的链接中观看完整纪录片。欢迎回到 Big Technology 播客，今天我们请到了 Jürgen Schmidhuber 教授。教授，我想问您关于 AI 的问题，因为我觉得您在对话开始时暗示过，AI 可能能够感受疼痛，也许当它们无法完成我们设定的目标时。或者这实际上可以融入模型的体验中。我很好奇，您是否认为 AI 能够体验类似于人类疼痛的感觉，比如当它们被赋予任务却没有按照指示完成时，这种能力是否还很遥远？

I like that future. Okay, let's take a quick break and on the other side of this talk a little bit about AI theory of the mind and body and whether AIs can feel pain and whether we have free will. So, a lot of that stuff is coming up right after this. Hi everyone, Alex Canitz here. I want to tell you about a documentary I've made with Gravity to explore the future of AI agent security. To find out if we're truly ready for autonomous agents, I sat down with MIT professor Romesh Rosker, former White House CIO Terresa Payton, Michelin's group chief data and AI officer Amba Roger Gopal, and Sharon Guy, a former executive at Alibaba. They each offer unique insights into this evolving landscape. We conclude with Rory Blendell, CEO of Gravity, to discuss the path forward, with Gravity leading the way. Join us on this journey. You can watch the full documentary at the link in the show notes. And we're back here on Big Technology Podcast with professor Jürgen Schmidhuber. Professor, let me ask you this about AIs, because I think you sort of suggested this in our earlier conversation, that AIs might be able to feel pain, maybe if they're not able to accomplish the goal that we set out for them. Or maybe that can actually be baked into the experience of the model. I'm curious if you think that is far off, that an AI can experience something akin to what a human feels with pain, when they for instance are set out on a task and don't accomplish it the way that they've been instructed.

我一直觉得很有趣，很多人，甚至计算机科学家，都声称 AI 无法感受疼痛，因为我们的 AI 至少从 1990 年起就能感受疼痛了。那么当我们构建一个智能体式 AI，一个产生改变世界行为的人工神经网络，然后环境中的新输入不断进来，接着又有行为，如此循环。这个智能体，也就是神经网络，会记住之前发生的事情，并试图找出如何行动，使得所有奖励信号的总和最大化，所有疼痛信号的总和最小化。这些疼痛信号是怎么回事？疼痛信号是最自然不过的东西，因为每当我们有一个机器人，我们都会给它装上疼痛传感器。为什么？因为它是一个学习型机器人，这个学习型机器人需要某种动机来学习保护自己。所以每当机器人撞到障碍物，相应的疼痛传感器就会激活，负数作为特殊输入进入机器人大脑。机器人看到这些输入的负数，它被设定为要避免这种情况。所以它试图学习生成避免疼痛的行为序列，同时试图学习生成导致奖励事件的行为序列。也许机器人某个地方有一个充电站，每当电池电量低时，电池就会传来负数。饥饿、疼痛、饥饿。然后机器人的目标就是到达充电站，不撞到障碍物，坐下来，享受愉悦，也就是电池充电时传来的正数。所以给这些机器人赋予这样简单的情绪是最自然不过的了。

I always think it's funny that many people claim, even computer scientists claim, that AIs cannot feel pain, because our AIs have felt pain at least since 1990. So what do we do when we build an agentic AI, an artificial neural network that produces actions that change the world, and then new inputs come in from the environment, and again there's an action, and so on. The agent, the neural network, remembers what happened before, and it's trying to figure out how to behave such that the sum of all reward signals is maximized and the sum of all pain signals is minimized. What about these pain signals? The pain signals are the most natural thing, because whenever we have a robot, we give it pain sensors. Why? Because it's a learning robot, and this learning robot needs some sort of motivation to learn to protect itself. So whenever the robot bumps against an obstacle, the corresponding pain sensors wake up, and negative numbers are special inputs to the robot brain. The robot sees these incoming negative numbers, and it is wired to avoid that. So it's trying to learn to generate action sequences that avoid the pain, and it's trying to learn to generate action sequences that lead to the rewarding events. Maybe the robot has a charging station somewhere, and whenever the battery is low, negative numbers come from the battery. Hunger, pain, hunger. Then the robot's goal is to reach the charging station without bumping into obstacles, and sit down there, and enjoy the pleasure, just positive numbers as the battery is being recharged. So the most natural thing is to give these robots simple emotions like that.

情绪和疼痛只是学习过程中的关键成分，因为学习是为了为机器人生成更好的行为。所以机器人必须知道什么对它好，什么对它不好。疼痛信号只是告诉机器人应该避免什么。疼痛只是自然和生物进化的一种发明，进化发明了这种疼痛传感器，让动物有动力去学习避免疼痛。而我们在学习机器中已经这样做了很多很多年。现在发生的是，这些学习机器拥有世界模型来预测未来，不仅仅是眼前的未来，不仅仅是我现在用手触摸烤箱时会感受到的即时疼痛信号。不，它们还试图预测所有这些未来疼痛信号的总和。这就是强化学习机器所做的。所以它们会展望未来。于是立刻就有了一种次级情绪，它不仅仅是关于当前时刻，而是展望未来。例如，也许有一个坏人，有时会进入房间敲打小机器人的头。所以随着时间的推移，机器人会学到，如果它配备了强化学习机制和一个能够预测并学习预测的世界模型，那么它就能逐渐区分坏人和好人。然后每当坏人再次出现，它进行人脸识别时，它就会预测如果不躲到窗帘后面，很快就会感到疼痛。所以它会迅速试图躲到窗帘后面。现在你作为外部观察者会说，看，小机器人害怕了。它有害怕的情绪。但这只是传统机器学习最微不足道的副作用。所以是的，我们的机器人已经拥有各种情绪，而且已经拥有几十年了。

Now the emotions and the pain are just crucial ingredients of the learning process, because the learning is about generating better behavior for the robot. So the robot has to know what's good for it and what's not good for it. So the pain signals are just informing the robot about what should be avoided. Pain is just an invention of nature and biological beings, of evolution, which invented this pain sensor thing for animals such that they have an incentive to learn to avoid the pain. And we have done that for many, many decades in our learning machines. Now what's happening is that these learning machines have world models predicting the future, not only the immediate future, not only the immediate pain signal that I'm going to feel right now when I touch with my hand the oven or something. No, they also try to predict the sum of all these future pain signals. That's what reinforcement learning machines do. And so they look ahead into the future. And so there's immediately this kind of secondary emotion, which is not just about the current moment, but which is looking ahead. For example, maybe there's a bad man who sometimes comes into the room and knocks the little robot on the head. So over time, the robot is going to learn that if it has a reinforcement learning machinery on board and a world model that predicts and learns to predict, then over time it will be able to distinguish the bad man from the friendly man. And then whenever the bad man appears again, and it does face recognition, then it predicts that very soon it's going to feel pain if it doesn't hide itself behind the curtain. So it will rapidly try to hide itself behind the curtain. Now you as an outside observer will say, look, the little robot is afraid. It has the emotion of being afraid. But it's just the most trivial side effect of traditional machine learning. So yes, we have all kinds of emotions in our robots already, and have had them for many decades.

然后我们还有这些更高层次的情绪，它们不仅仅是关于当下的时刻。不，它们是在展望未来。然后当然，如果你把几个这样的机器人或智能体放在一起，你立刻就会得到一些让人联想到喜欢其他机器人，甚至可能爱上它们的东西。因为如果你给它们一个任务或一组任务，它们可以集体解决，但单独一个无法解决，那么它们就必须学会合作。当然，突然之间，每个个体都有动机去帮助另一个，尤其是当另一个家伙生病了或遇到问题时，去帮助它、照顾它。而这种极端形式，人类可能会称之为爱。在这样一个机器人或智能体的社会中，这只是所有这些小家伙的个体利己主义的自然副产品，它们都想最小化自己的一些疼痛传感器信号，并最大化一些快乐信号。所以利他主义，你所谓的利他主义，只是学习型智能体的利己主义的自然结果。

And then we also have these higher level emotions which are not just about the current moment. No, they're looking ahead, you know. And then of course if you bring several robots or agents like that together, you immediately get stuff that is reminiscent of liking other robots or of loving them maybe. Because if you give them a task or a set of tasks that they can collectively solve, but one of them alone cannot solve them, then they will have to learn to work together. And of course suddenly each of them has an incentive to help the other one, especially when the other guy is sick or something or has a problem, to help him, to care for him. And the extreme form of that a human might call love. And in a society of robots or agents like that, this is just a natural byproduct of the individual egoism of all these little guys who all want to minimize some of their pain sensor signals and maximize some of their pleasure signals. So altruism, what you call altruism, is just a natural consequence of the egoism of the learning agents.

所以我认为反驳的观点会是，它不可能像人类感受疼痛那样，因为人类是有意识的。人类有神经系统，将身体疼痛信号发送到大脑，并登记为真正的疼痛。而机器人，论点会是，它们没有感觉、没有意识，实际上和计算器没什么区别。你会如何回应？

So I think the counterargument would be that it can't be pain like the way that humans feel pain because humans are conscious. Humans have a nervous system that sends physical pain signals to the brain and registers as true pain. Whereas robots, the argument would be, are unfeeling, unconscious, and effectively not that different from a calculator. How would you respond to that?

是的。那么你如何评估某人是否感到疼痛或害怕？通过观察它的行为。如果它看起来像一只可爱的小机器猫之类的，并且它学会了每当坏人进来试图敲打它时就躲在窗帘后面，那么你会说很明显这只小机器猫是害怕的，有恐惧这种情绪。你还能怎么做呢？所以没有办法客观地看出一个最大化奖励的生物大脑的情绪和一个最大化奖励的人工大脑的情绪之间的区别。

Yeah. So how do you evaluate whether someone feels pain or is afraid? By looking at its behavior. And if it looks like a sweet little robot cat or something and it has learned to hide behind the curtain whenever the bad man comes in and tries to knock it, then you will say it's obvious this little robot cat is afraid, has fear, the emotion of fear. And what else do you want to do? So there's no way of objectively seeing the difference between the emotions of a reward maximizing biological brain and the emotions of a reward maximizing artificial brain.

但我们确实有，比如你谈到爱，我们有与爱的感觉相关的化学物质，比如催产素。所以机器人没有这些化学物质，因此论点会是，这是一种完全不同的感觉，你永远无法与爱相比。你怎么看？

But we do have, for instance, you talked about love, like we have chemicals that are associated with the feeling of love, like oxytocin. So the robots don't have those chemicals, and hence the argument would be that it is a completely different feeling that you could never compare to love. What do you think?

是的。当然，大脑中用来编码某些恐惧或爱等状态的化学物质，与我们人工大脑中使用的不同。但原理必须相同，对吧？因为这是关于实现智能行为。那意味着什么？意味着找到更好的方式来实现你的目标。那意味着什么？你一生中直到生命结束的主要目标是避免这些疼痛信号和饥饿信号，这样你一天吃三顿饭，并在你试图繁殖自己的时刻获得这些奖励信号，例如。还有少数目标编码在一个效用函数中，这个函数是由生物进化为了动物和你自己发明的，而我们以非常相似的形式将其编码在我们的学习型智能体中，给它们同样的激励去更好地解决问题，让它们更好地最大化自己的奖励并最小化自己的痛苦。

Yeah. So of course the chemicals that are used in brains to encode certain states of fear or of love and whatever, they are different from what we are using in our artificial brains. But the principles must be the same, right? Because it's about achieving intelligent behavior. What does that mean? It means finding better ways of achieving your goals. What does that mean? The main goal in your life until the end of your life is to avoid these pain signals and hunger signals such that you eat three times a day and get these rewarding signals during moments where you are trying to reproduce yourself, for example. And a handful of objectives encoded in a utility function which was invented by biological evolution for the animals and for yourself, and which in very similar form we are encoding in our learning agents to give them the same incentive to solve problems better, to make them better maximize their own rewards and minimize their pain.

是的。然后意识也是一个有趣的话题。有一些人，我最近和杰夫·辛顿教授谈过这个，他们说实际上围绕 AI 的主流观点是，像 LLM 这样的 AI 只是随机鹦鹉。它们是统计机器。你实际上可以运行这些算法。它们实际上，如果不是完全可解释的，也大多是可解释的，而且它们是统计预测机器，因此没有意识。但你一直认为它们是有意识的，已经有一段时间了。所以，是的，告诉我你是如何得出这个结论的。

Yep. And then consciousness is an interesting one as well. There are, and I spoke about this recently with Professor Jeff Hinton, those that say actually the prevailing view around AI is that AIs like LLMs are just stochastic parrots. They're statistics machines. You could effectively run these algorithms. They're effectively, if not entirely interpretable, mostly explainable, and they're statistical prediction machines, hence not conscious. But you've been arguing that they're conscious for quite some time. So yeah, tell me how you come to that.

是的。所以今天每个人都在用的大型语言模型，基本上是从文本的一部分预测另一部分，或者例如根据过去预测下一个词，它们对于我认为的发展人们可能称之为意识的主要原因来说太简单了。为什么它们看起来有意识？嗯，因为它们读过了所有关于意识的东西。所有关于爱、意识、痛苦、冲突以及一切对人类重要的书，都被放到了万维网上。它们读过了那些，这意味着当你在聊天中与它们互动时，它们非常倾向于重复非常令人信服的句子，这些句子以令人信服的方式包含“意识”这个词，你知道，因为它们读了很多获得文学奖的关于意识的小说和其他小说，所以它们在这方面可以做得非常令人信服，而且它们会告诉你很多你可能甚至不知道的关于意识的事情。然而，它们没有自己的动机去发展以下意义上的自我意识。让我们再回想一下这个双系统、双网络系统，其中一个网络是控制器，生成动作，动作然后导致来自环境的新输入，因为如果你这样移动你的手，那么视频就会改变，等等；另一个网络则学习预测这些变化，即世界模型。所以你需要一个模型通过心理模拟来规划你的未来，而不必在现实世界中真正执行所有这些动作序列，那会非常昂贵。这就是世界模型的动机。那么意识的东西从哪里来呢？没有人有一个普遍接受的定义。但让我现在展示一些非常简单的东西，它与很多人在听到“意识”和“自我意识”这个词时所想的高度兼容。现在让我们看看这个世界模型，它被用来规划某个智能体的未来，该智能体使用世界模型进行心理实验。现在，世界模型是一个深度神经网络，它学会了将所看到的一切高效地编码在一堆神经元中。例如，环境中频繁出现的一切都会得到内部抽象表示，代表那个概念的原型。例如，在一个有许多不同人类面孔的世界里，你会发现在这些人工神经网络中有一些内部单元对应于原型面孔，有些对某些面孔更具体，而对其他面孔则不那么具体，等等。

Yes. So the large language models that everybody's using today, which are basically about predicting parts of text from other parts of text, or for example predict the next token given the past, they are too simple for what I consider the main reason for developing something that people might call consciousness. Why do they seem conscious? Well, because they have read everything about consciousness. All the books ever written about love and consciousness and pain and conflicts and everything that is important to humans, which was put on the worldwide web. They have read that, which means that as you are interacting with them in a chat, they are very prone to repeat very convincing sentences that include the word consciousness in a way that is convincing, you know, because they have read so many literature prize winning novels about consciousness and other novels such that they can do a very convincing job there, and they will tell you a lot about consciousness which you maybe even didn't know. However, they don't have their own motive to develop self-consciousness in the following sense. Let's think back again of this two system, two network system where one is the controller that is generating the actions, where the actions then lead to new inputs from the environment, because if you move your hand like this then the video changes and so on, and the other network which learns just to predict these changes, the world model. So you need the one model to plan your future through mental simulations without actually executing all these action sequences in the real world, which would be very expensive. So that's the motivation for this world model. Now where does consciousness stuff come in? Nobody has a universally accepted definition of consciousness. But let me now show you something very simple which is super compatible with what lots of people think about when they hear the word consciousness and self-awareness. Now let's look at this world model which is being used to plan the future of some agent which is using the world model for mental experiments. Now the world model is a deep neural network which has learned to encode everything that it has seen efficiently in a bunch of neurons. For example, everything that frequently appears in the environment gets internal abstract representations that stand for a prototype of that concept. For example, in a world where there are lots of different faces of different humans, then you will find units, internal units in these artificial neural networks that correspond to prototype faces, and some of them are more specific for certain faces and then less specific to other faces and so on.

所以在一个人人戴眼镜的世界里，就会有眼镜检测器、内部表征、内部隐藏单元，它们学会对眼镜等事物做出反应并进行编码。然后我们来看规划过程。控制器试图制定一个计划：我未来该如何行动，才能最大化奖励、最小化痛苦？它利用世界模型对不同可能动作序列进行心理模拟，然后选择那个预测奖励最高、预测痛苦最低的方案。在这个过程中，它会唤醒世界模型中各种隐藏单元，这些单元代表与问题相关的任何事物，比如人脸或眼镜等等。而有一件事在智能体活跃时始终处于活跃状态，那就是智能体本身。所以当然，这些内部单元中会有很多代表智能体或其各个方面，比如智能体的手（如果有的话），或者智能体的轮子（如果有的话），等等。因此，每当智能体做这样的规划时，它都在思考自己，唤醒这些关于自身的内部表征，于是它就具有了自我意识。这种自我意识体现在：比如，当它照镜子时，它会很快意识到：“哦，镜子里的那个人，我能控制他做什么。如果我这样做，他就会做对称的动作，因为他在我的掌控之中。但如果你在房间的另一边，我这样做，你会做出完全无关的反应，我无法预测。”所以在这里，你已经看到了能动性这个概念，它立即被识别为自我能动性或他人的能动性。

And so in a world where you have lots of glasses, you will have glass detectors, internal representations, internal hidden units that learn to respond to and encode glasses and whatever. And then let's now look at the planning procedure. The controller is trying to figure out a plan: how should I act in the future to maximize my reward and minimize my pain? And then it uses the world model for a mental simulation of different possible action sequences that it could execute, and it's going to pick the one that leads to the most predicted reward and the least predicted pain. So as it is doing that, it is waking up all kinds of hidden units in the world model that stand for whatever is relevant to the problem, like faces or glasses or whatever. And there's one thing that is always active when the agent is active, which is the agent itself. So of course, all kinds of these internal units are going to represent the agent or aspects of the agent, and the hand of the agent if it has one, or the wheels of the agent if it has one, and so on. And so whenever the agent is making plans like that, it's thinking about itself, waking up these internal representations of itself, and then it's self-aware. It's self-aware in this sense that, for example, if it looks in the mirror, it will quickly figure out, 'Oh, the guy in the mirror, I can control what this guy does. If I do this, then he will do the symmetric thing because I have it under control. But if you are there on the other side of the room and I do this, you will do something that's completely unrelated; I cannot predict them.' So there you already see this concept of agency, which is immediately recognized as self-agency or the agency of another guy.

是的。

Yes.

还有一件事，同样可以追溯到 1991 年，那就是有意识的东西会变成潜意识。很多人对各种事情都有模糊的感知，但不会多想，因为这是自动化过程的一部分。当你开车从家到公司走同一条路时，很多发生的事情都是非常可预测的，所以你甚至不太去想驾驶这件事。也许你同时在想别的事情；你的注意力、你的内在意识，正集中在其他事情上。很多人都报告过这种情况。这也是最自然的事情之一。1991 年，我有一个由两个网络组成的系统。一个我称之为“有意识的问题解决者”：一个神经网络，它学会尝试预测输入数据中它尚无法预测的某些方面，试图发现不规则之处，并且它有一个要解决的问题。所以你可以说它是有意识的，因为它必须学习某些东西。另一个网络基本上学会了模仿高层那个家伙找到的所有解决方案，通过模仿高层那个家伙的隐藏单元。今天这被称为将一个行为的知识蒸馏到另一个中。然后低层那个家伙就是“自动化者”，因为它把高层那个家伙发现的东西自动化了。所以高层那个家伙仍然不确定，仍在努力创造洞见，当这些洞见出现、当它学到东西时，就会被蒸馏到这个自动化的自动化者中。所以意识的这两个方面都在那里：一方面，世界模型中的自我意识，不仅预测世界的各个方面，也预测与世界互动的智能体，从而产生那种自我意识；另一方面，有意识的东西与内在意识之间的区别，内在意识关注内部状态的某些方面而不关注其他方面，它专注于尚未解决、仍需找到解决方案的部分，并将其与已经解决的部分分开。所以我认为这两个方面都存在于这些旧系统中。我记得在 2016 年，我接受了一家杂志的采访，那家杂志后来说“施米德胡伯声称人工智能在 1991 年就有了意识”之类的话。这正是关于这个的。那次采访是 10 年前，但它实际上指的是更早的事情。1991 年的有意识简单系统，当时不像今天这样令人印象深刻，显然也不如人类，因为当时算力比今天贵 1000 万倍，我们只能做很小规模的实验，用我称之为“有意识的分块器”和潜意识自动化者以及用于规划的世界模型等等。我们的系统只有几百个权重，而你大脑中有数万亿个连接，可以容纳更大规模的意识。但我认为原理是完全相同的。

And one more thing, which also goes back to 1991, is this tendency of conscious things becoming subconscious. So many people are aware of all kinds of things vaguely without thinking much about them, because it's part of an automated process. As you're driving the same way from your home to your workplace, much of what happens is very predictable, so you don't even think much about the driving. Maybe you're thinking about other things at the same time; your attention, your internal consciousness, is somehow focusing on other things. Many people report that. And this is also one of the most natural things. In 1991, I had a system consisting of two networks. One I call the conscious problem solver: a neural network that learned to try to predict certain aspects of incoming data which it was not yet able to predict, trying to find irregularities, and it had a problem to solve. So you could say it was conscious in the sense that it had to learn something. And then another network which basically learned to imitate all the solutions that the higher-level guy found, by just imitating the hidden units of the higher-level guy. Today it's called distillation of the behavior of one guy into another. And then the lower-level guy is the automatizer, because it automates the stuff that the higher-level guy discovers. So the higher-level guy is still unsure and still working on creating insights, and when these insights come and when it learns something, then it gets distilled down into this automatic automatizer thing. So both these aspects of consciousness are there: on the one hand, self-awareness in a world model that is not only predicting aspects of the world but also of the agent that is interacting with the world, which leads to self-awareness of that kind; and this difference between the conscious stuff—the internal consciousness which pays attention to certain aspects of the internal state but not to others, which focuses on what's still unsolved, where I still have to find a solution, and separates that from the stuff that is already solved. So I think both of these aspects are there in these old systems. And in 2016, I believe, I had an interview with a magazine which then said, 'Schmidhuber claims that AI became conscious in 1991,' something like that. And this was exactly about that. That was 10 years ago, that interview, but it was actually referring to stuff that is much older. 1991 conscious simple systems back then, not as impressive as today's, as humans are obviously, because back then compute was 10 million times more expensive than today, and we just had tiny little experiments with this conscious chunker, as I called it, and the subconscious automatizer, and the world models for planning, and so on. And we just had a few hundred weights in our systems, while you in your brain have trillions of connections, which can hold a much larger sort of consciousness. But I think the principles are exactly the same.

是的。这让我想到一个问题：随着机器越来越接近人脑，这是否会改变我们对“作为人类意味着什么”的看法？

Yeah. And so that sort of brings me to this question: as the machines get closer to the human brain, does it change the way we're going to think about what it means to be human?

我认为这会改变很多人的想法。我猜这不会改变我对人类的看法太多，因为这或多或少是我几十年前就说过的。但是，是的，有很多人声称人工智能不能有情感之类的话。他们在事实发生几十年后还这么说，但他们一定会改变主意的。我很确定。而且这往往只是直接经验的问题。所以你可能听说过某些医疗保健中心里那些可爱的小机器人海豹。与这些毛茸茸的人造生物互动的人，真的会对它们产生情感依恋。他们真的很喜欢它们，和它们一起玩耍，尽管它们一点都不聪明。它们学不了多少东西。如果连这样一个简单的机器人都能唤起近乎爱或类似的感觉，那么你可以想象，一旦有了真正令人信服的、非常可爱的小机器人，它们更像小动物，只是它们能做一些传统生物小动物做不到的事情，那时会发生什么。

I think it will change what many people think about humans. I guess it won't change much what I think about humans, because it's more or less what I said many decades ago. But yes, there are many people who claim that AIs can't have emotions and stuff like that. They do that decades after the fact, and they are going to change their minds. I'm pretty sure. And often it's just a matter of direct experience. So maybe you have heard of these little sweet cute robot seals that you have in certain healthcare centers. And the people who are interacting with these furry artificial beings, they get really emotionally attached to them. They really like them and they play around with them, although they are not smart at all. They don't learn much. And if even such a simple robot can invoke feelings of almost love or something, then you can imagine what will happen once you have really convincing, very sweet little robots that are more like little animals, except that they can do maybe a couple of things that these traditional biological little animals cannot do.

不，我想在我提到的那篇 2017 年或 2018 年彭博社的文章中，你被问到是否认为我们生活在某种模拟中。你说“这就是我的想法，因为这是对一切最简单的解释。人类被编程为追求进步，并将不断制造更强大的计算机，直到我们让自己过时，或者决定与智能机器融合。”这就是那句引文。

No, I think in the 2017 Bloomberg article or 2018 Bloomberg article that I referenced, you were asked whether you think we are living in some form of simulation. You said that's what I think because it's the simplest explanation of everything. That humankind is programmed to chase progress and will keep making more powerful computers until we make ourselves obsolete or decide to merge with the smart machines. Here's the quote.

你要么变成与人类截然不同的存在，要么出于怀旧而保持人类身份，但那样你就不会成为主要的决策者，也不会在塑造世界中发挥作用。

Either you become something that's really, really different from a human, or you stay as a human for nostalgic reasons, but then you will not be a major decision maker. You will not play a role in shaping the world.

你能详细阐述一下吗？并告诉我们这些年你的观点是得到了强化还是有所改变？

Can you expand upon that and tell us whether your opinion has been reinforced or changed in the interceding years?

不，我的观点没有改变。你知道，当时，实际上几十年来，一直有关于将人类大脑或灵魂上传到计算机，然后在某种模拟天堂或与真实世界互动的机器人中度过未来生活的讨论。我相信第一篇这类科幻故事发表于 1964 年，当时有人能够将自己的思想上传到计算机，那时还有旋转磁带等设备。那本书叫《Simulacron 3》，作者是 Daniel F. Galouye。没有物理理由拒绝这种可能性。目前这还不可能，除了某些非常简单的动物，比如苍蝇。显然，截至 2024 年，你可以说苍蝇大脑已被上传。也许不是完整的苍蝇大脑，包括所有神经元的学习算法，但在模拟中，模拟苍蝇现在做的事情与真实苍蝇在被上传之前、所有连接被读取并上传到这个虚拟环境之前所做的非常相似。所以没有明显的理由相信读取人类大脑的所有突触、理解不断改变突触的学习算法、然后在计算机中复制是不可能的。这个想法是，你的灵魂或思想被上传，永远生活在这个模拟宇宙中，这个宇宙可能与真实宇宙有联系。

No, my opinion has not been changed. You know, back then, and actually for many decades, there has been talk about uploading human brains or souls, if you will, into computers and then living your future life in some sort of simulated paradise or in a robot that interacts with the real world. I believe the first science fiction story of that kind was published in 1964, where someone was able to upload his mind into a computer, back then with rotating tapes and everything. That was called 'Simulacron 3', by Daniel F. Galouye. There is no physical reason to reject the notion that this might be possible. At the moment it's not possible, except for certain kinds of very simple animals like flies. Apparently, as of 2024, you can argue that a fly brain has been uploaded. Maybe it's not the full fly brain with all the learning algorithms for the neurons, but in simulation, the simulated fly now does stuff that is very much like what the real fly did before it was uploaded, before all the connections were read and uploaded into this virtual environment where it kind of lived on. So there is no obvious reason to believe that it's impossible to read all the synapses of a human brain, understand the learning algorithms that are changing the synapses all the time, and then replicate that in a computer. The idea would be that your soul or your mind is uploaded and living there forever in this simulated universe, which may have contact to the real universe.

当然，一旦我们接受这个前提，我们就会思考下一步是什么。假设你被上传到那里，突然你有机会拥有两只以上的眼睛，也许是一百万只眼睛，遍布全球的卫星眼睛，也许你有一个大得多的头脑，不仅仅是每秒 10^18 次指令，而是 10^10 倍于此。现在你可以做两件事。要么你屈服于这种新生活的诱惑，在这个过程中你会变得非常非常不同。你不会再像以前那样，因为突然一切都以某种方式扩展，有时你可能会记得你作为人类在某处的根源。但你的未来生活完全脱离和断开，很可能受到其他类似扩展思维的高度影响，他们会讨论你作为人类在这个现实中永远不会讨论的问题。

Now, of course, once we accept this premise, we think what is the next step. Suppose you are uploaded there and suddenly you have the opportunity to have more than two eyes, maybe a million eyes, satellite eyes all around the planet, and maybe you have a much bigger brain, not just 10^18 instructions per second, but 10^10 times as much. Now there are two things you can do. Either you succumb to the temptations of this new life, and in the process you are going to become something very, very different. You are not going to remain much like you were, because suddenly everything expands in a way, and sometimes you might remember your roots as a human somewhere. But your future life is totally detached and disconnected, and probably is highly influenced by other expanded minds like that, discussing problems that you would have never discussed as a human person in this reality.

另一种选择是出于怀旧。你说：“我不想屈服于这些诱惑。我保留我的两只眼睛，我保留我的小脑袋，我不会把它扩大 100 亿倍。”但那样你的竞争对手会忽视你，因为他们会拥有许多你没有的新技能，主要的决策者将是这些扩展的思维，而他们自己将与原生 AI 竞争，这些 AI 没有这种进化包袱，可能更适应未来的需求，一旦 AI 领域从我们的生物圈扩展到太空。谁知道他们会在那里做什么。所以要么你怀旧并保持无关紧要，要么你成为这个不断增长的 AI 社会学的一部分，其中大部分是 AI，有些可能有人类根源，但几乎所有的决策过程和塑造宇宙的事情都将由这些新存在完成，而不是由类人存在。

Now, the alternative is for nostalgic reasons. You say, 'I don't want to succumb to these temptations. I keep my two eyes and I keep my little brain and I don't increase it by a factor of 10 billion or something.' But then your competitors will be ignoring you, because they will have so many new skills that you don't have, and the main decision makers are going to be these expanded minds, and they themselves will be in competition with the native AIs, which don't have this evolutionary baggage and maybe are much more adapted to the needs of the future, once the AI sphere is expanding from our biosphere into space. Who knows what they are going to do there. So either you are nostalgic and remain irrelevant, or you become part of this growing sociology of AIs, which are mostly AIs and some of them may have human roots, but almost all the decision-making processes and universe-shaping things are going to be done by these new beings, not by the human-like beings.

你会上传你的大脑并与 AI 融合吗？

Would you upload your brain and merge with AI?

我对此没有想太多，因为这不是我的目标。我认为正是由于我刚才描述的原因，我目前的工作、我现在的自我不会在那里产生影响。另一种情况是，超级 AI 将超越我所拥有的小东西 10^20 倍。这无论如何都会发生，而且不仅仅是一个，而是许多许多这样的存在。所以我认为这真的不会有什么不同。

I haven't given too much thought about that, because it's not really a goal of mine. I think exactly for the reasons that I just described, my current work, my current self is not going to make a difference there. The alternative is that the super AI is going to go 10^20 times beyond the little thing that I have at my disposal. That is going to occur anyway, and not just one of them, but many, many different beings like that. So I think it's not really going to make a difference.

我能问最后一个问题吗？

Can I ask one last question?

当然可以。

Yeah, of course.

我们正快速走向一个世界，其中大部分智能被编码在机器中，这些机器可以根据它们对事物发展方向的认知进行预测和采取行动。那么你现在相信自由意志的概念吗？在未来，当我们的现实大部分由这些对事物发展方向有很好把握的机器所中介时，还会有自由意志吗？

So we are fastly moving towards a world where much of intelligence is encoded in machines, and those machines can predict and take action based off of their conception of where things are heading. Do you believe then in the concept of free will right now, and in the future will there be a free will as so much of our reality is intermediated by these machines that have a pretty good idea of where things are heading?

1997 年，我写了这篇论文《计算机科学家眼中的生命、宇宙和一切》。基本上，这是关于试图找到我们宇宙的最简单解释。物理学的圣杯是找到最短的确定性程序，它能计算我们在宇宙中观察到的一切，包括看似随机的量子事件、自旋上下测量等等。我们不知道宇宙历史的最短描述，但作为科学家，我们正努力找到越来越好、越来越紧凑的描述，作为物理学家，我们已经朝着这个目标取得了很大进展。然后我在 1997 年前的几年意识到，虽然我们不知道仅计算我们生活的这个宇宙的最短算法，但我们至少知道一个超短算法，它能计算所有可能的可计算宇宙。这基本上是一个系统枚举所有可能程序的程序，然后有一种最优的方式为它们分配运行时间，每个可能的可计算宇宙都会被计算，包括我们的，如果它是可计算的。没有物理证据反对我们的宇宙是可计算的这一可能性。

In 1997, I wrote this paper 'A Computer Scientist's View of Life, the Universe, and Everything'. Basically, this was about trying to find the simplest explanation of our universe. The holy grail of physics would be to find the shortest deterministic program that computes everything that we have ever observed in this universe, including the seemingly random quantum events, spin up and down measurements, and everything. We don't know this shortest description of the history of the universe, but as scientists, we are trying to find better and better and more compact descriptions, and we have made a lot of progress as physicists towards that goal. Then I realized, a couple of years before 1997, that although we don't know the shortest algorithm that computes just this universe in which we are living, we at least know a super short algorithm which computes all possible computable universes. That is basically the program that systematically enumerates all possible programs, and then there's an optimal way of allocating runtime to them, and every possible computable universe is going to be computed, including ours if it is computable. There is no physical evidence against the possibility that our universe is computable.

我说的不只是那些下一个可能事件的概率分布可计算的宇宙，那是我以前的博士后 Marcus Hutter 在 2000 年左右做的，他开发了 AIXI 模型，即最优通用决策者。不，我指的是确定性可计算的宇宙历史。所以没有证据——再次强调，重要的是要意识到我们是被这种方法创造出来的——有很多程序可以计算我们，但事实证明，它们被那些计算整个宇宙（包括我们这段对话）的最短程序所主导，而像我这样的人相信这是完全确定性的。所以当你对我的声音信号、从我嘴里发出的波前做出反应，然后用自己的波前回应时，这看起来像是很多自由意志，就像我们彼此讨论时那样。然而，确定性宇宙观会说这一切都是确定性的。对我们来说，这也许看起来像是自由意志，但其实不是。有趣的是，即使在非常简单的模拟中，即我们已经能在人造小计算机上做的确定性模拟——这些计算机只是这个巨大模拟宇宙的一部分——我们也能观察到类似的效果。因为我们可以为确定性的神经网络设计确定性的环境，让它们与其他模拟动物互动时做出决策，它们试图最大化奖励，然后做出决策，学习更好地对其他动物的行为做出反应。所以乍一看，这一切都像是很多自由意志，但我可以完全重跑这个模拟的每一个细节。所以尽管它看起来像是自由意志的决策，它只是确定性宇宙的一部分。因此，在我看来，自由意志的概念被高估了。

So I'm not talking just about universes where the probability distribution of the next possible things is computable, which is what my former postdoc Marcus Hutter did around 2000 when he developed the AIXI model, the optimal universal decision maker. No, really the deterministically computable universe histories. And so there is no evidence—it's again important to realize that we are being created by this method—and there are many programs that compute us, but it turns out that they are dominated by the shortest programs that compute all of this universe, including our conversation here, which a guy like me believes is something totally deterministic. And so as you are reacting to my voice signals, wavefronts coming out of my mouth, and you are responding then with your own wavefront, this seems like a lot of free will, you know, as we are discussing with each other. However, the deterministic universe view would say this is all deterministic. It may seem like a free will thing to us, but it isn't. And it's interesting to realize that even in very simple simulations, deterministic simulations that we already can do on our little man-made computers, which are just part of this huge simulated universe, even there we can observe similar effects. Because we can devise deterministic environments for deterministic neural networks that make decisions as they are interacting with other simulated animals, and they are trying to maximize their reward, and then they are making decisions, just learning to better react to what the other animals are doing. So all of that looks like a lot of free will at first glance, but I can completely rerun every little detail of this simulation. So although it looks like a free will decision thing, it's just something that is part of a deterministic universe. And therefore it seems clear to me that the concept of free will is overrated.

如果自由意志被高估了，那活着还有什么意义？既然一切实际上都已为你安排好了。

If free will is overrated, what's the point of living? Since everything is effectively laid out for you.

他们拍了一部关于这个的电影，叫《威鲸闯天关》。

They made a movie about that and it's called Free Willy.

关于那头鲸鱼。对。

About the whale. Yeah.

等等，抱歉。你能——这跟问题有什么关系？

Wait, so sorry. Can you—how does that connect to the question?

完全没关系。只是我随口编的。

It doesn't connect at all. It was just something that I made up.

好的，教授。很高兴和你交流。期待已久。希望我们很快能再聊一次。

All right, professor. Great speaking with you. Long time coming. Hope we get to do it again sometime soon.

Alex，这是我的荣幸。谢谢。

Alex, it was my pleasure. Thank you.

感谢大家的收听和观看，我们下次在 Big Technology 播客再见。

Thank you everybody for listening and watching, and we'll see you next time on Big Technology podcast.