AI 是否像我们，AI 会像我们吗：Geoffrey Hinton 2026 年哥德尔讲座

好的，下午好。非常感谢大家来参加 2026 年信息学系的维也纳哥德尔讲座。我认为今天我们有一位很棒的演讲者，稍后我会深入介绍他，但现在在我多说之前，我请我们的校长施耐德说几句欢迎辞。

Okay, good afternoon. Many thanks for coming to the Vienna Gödel Lecture of the Faculty of Informatics 2026. I think we have a great speaker today, whom I will introduce later in depth, but now before I say much more, I ask our rector, Schneider, who will say a few welcoming remarks.

谢谢。非常感谢。亲爱的辛顿教授，女士们先生们，在库珀厅以及通过直播加入我们的所有房间。欢迎来到 2026 年维也纳哥德尔讲座。辛顿教授，您今天与我们同在，我们深感荣幸。为了节省时间，我不会列举您的功绩。那会占满整个下午和晚上，而且我认为赛德勒教授会处理这件事。但让我说一件事。我们在维也纳工业大学（TU Wien）的使命是技术为人服务，而不是人为技术服务。我相信这也是您所代表的。欧洲目前在数字主权方面有严重问题，我们绝不能依赖少数科技巨头及其商业模式。但看看周围。我们有才华，这个房间里有很多，楼下的宿舍里也有很多。那么，我们该怎么做？在这个问题上，您个人是一个杰出的灵感。通过您的工作，通过您愿意公开谈论数字技术的走向，以及通过您今晚与我们同在。我们在 TÜV Rheinland 的新愿景是“我们敢于理性”。我确信我们需要一个启蒙的新时代。所以，我现在把时间交还给赛德勒教授，我们非常好奇并期待您的演讲。谢谢。

Thank you. Thank you very much. Dear Professor Hinton, ladies and gentlemen here in the Kuppelsaal and in all rooms joining us by livestream. Welcome to the Vienna Gödel Lecture 2026. And Professor Hinton, we are deeply honored that you are with us today. In the interest of time, I will not list your merits. This would fill the afternoon and the evening and I think also that Professor Seidler will take care of this. But let me say one thing. Our mission at TU Wien is technology for people and not people for technology. I believe this is also what you stand for. And as Europe currently has a serious problem with digital sovereignty, we must not become dependent on a handful of tech pros and their business models. But look around. We have the talent, a great deal of it in this room, in the dorm rooms downstairs and many more. So, what should we do? On that question, you personally are an outstanding inspiration. Through your work, through your willingness to speak openly about where digital technology is heading, and through your presence here tonight with us. Our new vision at TÜV Rheinland is "We dare to reason." And I'm convinced we need a new era of enlightenment. So, I will give back now to Professor Seidler, and we are really curious and look forward to your presentation. Thank you.

好的。嗯，现在非常荣幸地介绍杰弗里·辛顿教授，今年的哥德尔讲座演讲者。杰弗里在机器学习方面做出了基础性贡献，我们正在经历的 AI 革命没有他的贡献就不会发生。自 20 世纪 80 年代以来，他一直从事人工神经网络的设计和训练。多年来，很少有人相信这种方法会成功。但它确实成功了，甚至非常巨大。今天改变我们生活的 AI 没有他的贡献就不会存在。杰弗里是多伦多大学计算机科学荣誉教授。2013 年至 2023 年，他还在谷歌工作，之后谷歌收购了他的初创公司。但在 2023 年 5 月，他离开了谷歌，以便自由谈论 AI 的风险。我认为他现在是警告这些风险的最著名、最直言不讳的声音之一。由于他的成就，他获得了 ACM 图灵奖。图灵奖被认为是计算机科学家能获得的最高荣誉。嗯，所以你想，“好吧，就这样了。你不能再高了。”但然后在 2024 年，他与约翰·霍普菲尔德一起获得了诺贝尔物理学奖。所以，这对计算机科学家来说是非常罕见的事情。诺贝尔委员会表彰他们的基础性发现，这些发现使人工神经网络的机器学习成为可能。在诺贝尔奖之前和之后还有许多其他荣誉。他是皇家学会会员和加拿大勋章同伴。去年，他获得了伊丽莎白女王工程奖。有趣的是，杰弗里并不是从计算机科学开始的。他在剑桥学习实验心理学。我认为他进入神经网络是因为他想理解大脑如何学习。这个问题，我认为，从那以后一直驱动着他的工作。还有一个有趣的事实，杰弗里是乔治·布尔的曾曾孙，乔治·布尔是现代逻辑的创始人之一。顺便说一下，维也纳工业大学在这个领域特别强。所以，在以逻辑学家命名的讲座——哥德尔讲座上，我们欢迎著名逻辑学家布尔的后代。我认为我们非常幸运和荣幸能请他来维也纳，因为杰弗里非常抢手。他经常接受主要媒体的采访，这给了他“AI 教父”的称号。到现在，我认为几乎必须提到这个称号。所以我提到了。我非常期待这个讲座，它有一个非常有趣且几乎富有诗意的标题：“AI 像我们吗，AI 会喜欢我们吗？”所以我们会听到答案，但我们肯定想现在听演讲。请和我一起欢迎杰弗里·辛顿教授。

Okay. Um So, now it's really a great pleasure and honor to introduce Professor Geoffrey Hinton, this year's speaker of the Gödel Lecture. Geoffrey has made fundamental contributions to machine learning, and the AI revolution we are now living through would not happen without his contributions. And he's worked on the design and training of artificial neural networks since the 1980s. And through long years, very few people believed that this approach could succeed. And it did succeed, even tremendously. And the AI that is changing our lives today would not exist without his contributions. Geoffrey is Professor Emeritus of Computer Science at the University of Toronto. And from 2013 to 2023, he also worked at Google after Google acquired his startup. But in May 2023, he left Google so that he could speak freely about the risks of AI. And I think now he's one of the most prominent and outspoken voices warning about these risks. For his achievements, he received the ACM Turing Award. And the Turing Award is considered the highest honor one can get as a computer scientist. Um so you think, "Okay, that's it. You can't go higher." But then in 2024, he received the Nobel Prize in Physics together with John Hopfield. So, a very rare thing for a computer scientist. And the Nobel Committee honored their foundational discoveries that enable machine learning with artificial neural networks. Many other honors came before and after the Nobel. He's a fellow of the Royal Society and a companion of the Order of Canada. And last year, he received the Queen Elizabeth Prize for Engineering. And interestingly, Geoffrey didn't start out in computer science. He studied experimental psychology at Cambridge. And I think he came to neural networks because he wanted to understand how the brain learns. And this question, I think, has driven his work since then. And one more curious fact, Geoffrey is the great-great-grandson of George Boole, one of the founders of modern logic. An area, by the way, in which TU Wien is particularly strong. So, at the lecture named after a logician, the Gödel Lecture, we welcome a descendant of a famous logician Boole. I think we are very lucky and honored to have him here in Vienna because Geoffrey is in very high demand. He's constantly interviewed by leading media which gave him the title Godfather of AI. And by now, I think it's almost obligatory to mention the title. And so I have. I very much look forward to the lecture which has the very interesting and almost poetic title "Is AI like us and will AI like us?" So we will hear the answer, but we will definitely like to hear now the presentation. Please join me welcoming Professor Geoffrey Hinton.

谢谢。

Thank you.

好的，谢谢介绍，谢谢邀请。嗯，我将给大家一个非常简短且非常偏颇的 AI 历史。自 AI 诞生以来，有两种范式。一种是逻辑启发的方法。嗯，我们认为智能的本质是推理，推理将通过使用符号规则来操作符号表达式，就像逻辑中所做的那样。而学习之类的事情可以等到我们理解我们如何进行推理以及如何表示知识之后。首先我们必须弄清楚知识表示所用的符号语言是什么，我们如何进行推理，然后我们才能担心学习。这就是他们的方法。他们在上世纪和本世纪初占主导地位。然后有一种受生物学启发的方法，其理念是智能的本质实际上是学习神经网络中连接的强度。嗯，推理可以等到以后。首先，我们必须理解学习如何工作，然后我们才能理解其他事情。这是基于这样的想法：像鸽子这样的东西以自己的方式相当聪明。它们擅长感知之类的事情。它们擅长飞越森林。嗯，它们不做推理。据我们所知，它们不做太多推理，它们可能不使用逻辑。所以从生物学的角度来看，推理出现得很晚，把智能模型建立在那上面有点愚蠢。现在，在我继续之前，我应该说的是，20 世纪 50 年代有两个人实际上相信神经网络方法。他们都是非常优秀的逻辑学家。他们是约翰·冯·诺依曼和阿兰·图灵。所以你不能像对我那样指责他们不懂逻辑。嗯，不幸的是他们都英年早逝，我认为如果他们活下来，整个 AI 历史可能会非常不同。图灵在研究如何用强化学习来学习神经网络。是的。所以，我将从词的意义开始讲。在符号 AI 范式中，词的意义来自它如何在命题或句子中与其他词互动。所以，意义不是你能够用手抓住的东西。它隐含在它与其他词共现的方式中。可能为了捕捉意义，你需要类似图的东西。所以，那是符号主义的观点。

Okay, thank you for the introduction and thank you for inviting me. Um I'm going to give you a very brief and very biased history of AI. So since the beginning of AI, there were two paradigms. There was the logic inspired approach. Um we should thought the essence of intelligence was reasoning and reasoning was going to be done by using symbolic rules to manipulate symbolic expressions just as it's done in logic. And that things like learning can wait until we understand how we do reasoning and how we represent knowledge. First we have to figure out what is this symbolic language in which knowledge is represented, how do we do reasoning, and then we can worry about learning later. That was their approach. And they were dominant for the last during the last century and the early part of this century. Then there was a biologically inspired approach where the idea was the essence of intelligence is actually learning the strengths of connections in a neural network. Um reasoning can wait till later. First, we have to understand how learning works and then we'll understand other things later. And this was based on the idea that um things like pigeons are quite smart in their own way. They're good at things like perception. They're good at things like flying through forests. Um and they don't do reasoning. Well, as far as we know, they don't do much reasoning and they probably don't use logic. Um so really from a biological perspective, reasoning comes very late and it's sort of silly to base your model of intelligence on that. Now, one thing I should say before I go on is there were two people um from the 1950s who actually believed in the neural network approach. They were both very good logicians. It was John von Neumann and Alan Turing. So, you couldn't really accuse them of not understanding logic like me. Um they unfortunately both died young and I think if they had survived, the whole history of AI might have been very different. Turing was sort of studying how you could learn neural networks using reinforcement learning. Yeah. So, I'll start by talking about the meaning of a word. So, within the symbolic AI paradigm the idea was that the meaning of a word comes from how it interacts with other words in propositions or in sentences. So, the meaning isn't a sort of thing you could put your hands on. It's implicit in how it co-occurs with other words. And probably to capture meaning, you'd need something like a graph. So, that was a symbolic view.

20 世纪 30 年代心理学中有一种非常不同的观点，认为一个词的意义就是一大组语义特征，可能还包括句法特征。所以你有一大组特征，激活其中的一个子集来捕捉一个词的意义。比如对于“猫”这个词，你会激活一些特征，表明它是捕食者、是宠物、高冷、有胡须、大约面包盒大小等等。要表示“狗”，你会激活一组相当相似的特征，但显然在某些方面不同。这个理论非常擅长捕捉意义的相似性。所以猫和狗会有相当相似的特征向量。它也相当擅长捕捉类比。例如，在我们文化中，对大多数男性来说（结果发现对女性不一定），你用于“猫”的特征向量与你用于“女人”的特征向量比与你用于“男人”的特征向量更相似。反过来，“狗”的特征向量与“男人”的特征向量比与“女人”的特征向量更相似。作为一个男人，我以为这是显而易见的，对所有人都是显而易见的。结果并非如此。不少女性根本不觉得这显而易见。这对我来说是个新发现。但这使得这些特征向量非常擅长做类比。

There's a view in psychology in the 1930s which is very different, which was that the meaning of a word is just a big set of semantic and maybe syntactic features. So, you have a large set of features, you activate a subset of them to capture the meaning of a word. So, for the word cat, you'd activate features that say it's a predator, it's a pet, it's aloof, it's got whiskers, it's about the size of a bread box. Things like that. To represent a dog, you'd activate a fairly similar bunch of features, but obviously different in some respects. That theory was very good at capturing similarities of meaning. So, cat and dog would have fairly similar feature vectors. It was also quite good at capturing analogies. So, for example, for most men in our culture, not necessarily for women it turns out, the feature vector that you use for cat is more similar to the feature vector that you use for woman than the feature vector that you use for man. And vice versa, the feature vector for dog is more similar to the feature vector for man than it is for woman. Now, being a man I thought that was obvious, that's obvious to everybody. Turns out it's not. Quite a few women don't see that as obvious at all. That was a revelation to me. But that makes these feature vectors very good at doing analogies.

早在 1985 年，我想到这两种不同的意义理论——一种隐含在词与其他词的共现方式中，另一种只是一大堆特征的激活——它们不一定不相容。它们可能只是同一枚硬币的两面。而借助反向传播学习算法（这里我不打算解释），你实际上可以在两种表示之间转换。

Back in 1985, it occurred to me that these two different theories of meaning—one that is implicit in how you occur with other words, and the other that is just a big bunch of activations of features—they're not necessarily incompatible. They may be just two different sides of the same coin. And with the backpropagation learning algorithm, which I'm not going to explain, you could actually convert between these two representations.

所以，你要做的是在词串上训练一个神经网络，训练它把每个词转换为一组特征，然后让这些特征相互作用，并用这些相互作用来预测下一个词的特征，一旦知道了下一个词的特征，你就可以猜测下一个词可能是什么。所以你可以训练一个神经网络，在学习过程中，它学会了所有这些特征应该是什么。心理学家一直难以解释所有这些特征从何而来。但有了反向传播，你可以取一个词串，尝试预测下一个词，我用了非常简单的词集，只是三个词的字符串并预测第三个词，并表明反向传播确实能做到这一点。它可以把词转换为特征。所以它也可以把特征转换为词，意思是你可以通过逐步预测下一个词来生成新的词串。你从一两个词开始，预测下一个词，你接受它，然后预测再下一个词。所以你可以从这些特征回到词串。因此，用这些神经网络，你可以双向转换。你可以把词串转换为词的特征向量，也可以把词的特征向量转换回词串。这意味着你拥有的关系知识并不像符号主义者认为的那样存在于关系图中。它存在于所有这些特征相互作用的方式中。你没有静态存储的命题。你头脑中没有命题。当你需要一个时，你就生成一个。你头脑中拥有的是关于如何将词转换为特征的知识，以及关于两个特征应该如何相互作用的知识。

So, what you would do is you train up a neural net on strings of words, and you train the neural net to convert each word into a set of features, and then to make those features interact, and for those interactions to predict the features of the next word, and once you know the features of the next word, you can make guesses about what the next word might be. So, you can train a neural net, and as it's learning, it learns what all these features should be. It had always been a problem for psychologists to explain where all these features came from. But with backpropagation, you can take a string of words, you can try and predict the next word, and I took very simple sets of words, just three-word strings and predicting the third word, and showed that actually backpropagation could do this. It could convert words into features. So, it could also convert features into words in the sense that you could generate new strings of words by just progressively predicting the next word. You start with one or two words, you predict the next word, you accept that, then you predict the word after that. So, you can get back from these features to strings of words again. So, with these neural nets, you can convert both ways. You can convert strings of words into feature vectors for words, and you can convert feature vectors of words back into strings of words. And that meant that the relational knowledge you have doesn't reside in a relational graph as symbolic people thought. It resides in the way all these features interact. You don't have static stored propositions. There are no propositions in your head. When you want one, you generate one. What you've got in your head is knowledge about how to convert words into features, and knowledge about how two features should interact.

所以，在接下来的 30 年里，那是 1985 年，大约 10 年后，计算机更快了，蒙特利尔的 Yoshua Bengio 表明，不只是像我那样用玩具例子，你可以用真实的自然语言并预测下一个词，而且预测效果与当时最好的模型相当。当时最好的模型使用巨大的短语表，统计不同短语出现的频率。所以如果你看到“fish and”，它会查找所有包含“fish and something”的短语，发现“fish and chips”出现很多，于是说“chips”是一个好的预测。这就是自动补全过去的工作方式，也是语音识别中预测词时对词建模的方式。但神经网络的表现与这些系统的最佳版本相当。而且它们不断变得更好。大约 10 年后，科学进步缓慢。大约 10 年后，领先的计算语言学家采纳了这样的观点：一个词的意义应该是他们所称的词嵌入，即该词的特征向量。大约又过了 10 年，谷歌的研究人员在 2017 年弄明白了一种叫做 Transformer 的神经网络，我稍后会简要解释。然后谷歌发布之后，几年后，OpenAI 展示了用这些 Transformer 可以做出惊人的事情，于是我们有了当前的聊天机器人，它们确实很惊人。所以，大型语言模型，在我看来，大型语言模型是我 1985 年产生的微型语言模型的后代。那是第一个通过预测下一个词来训练词义的东西。它们使用更多的词作为输入。它们有更多的神经元层。特征之间的相互作用也复杂得多。当然，它们训练的数据量巨大。但我相信，它们理解语言的方式与人类理解语言的方式大致相同。也就是说，它们把词转换为特征向量，当它们理解一个句子时，它们真正做的是找到能很好地组合在一起的特征向量。

So, over the next 30 years, that was in 1985, about 10 years later, Yoshua Bengio, computers were faster, and Yoshua Bengio in Montreal showed that instead of just taking a toy example like I'd taken, you could take real natural language and predict the next word, and you could predict it about as well as the best existing models. The best existing models used huge tables of phrases and counted how often different phrases occurred. So, if you saw fish and, it would look for all phrases have fish and something, and it would see fish and chips occurred a lot, and so it'd say chips was a good prediction. That was the way auto complete used to work, and that was how they used to model words for speech recognition when they're predicting words in speech. But neural nets did about as well as the best versions of those systems. And they kept getting better. About 10 years after that, progress is slow in science. About 10 years after that, leading computational linguists adopted the idea that the meaning of a word should be what they called a word embedding, a feature vector for the word. And about 10 years after that, researchers at Google in 2017 figured out a kind of neural network called a transformer, which I will explain briefly later. And then after Google released that, a few years later, OpenAI showed you could do amazing things with these transformers, and we got our current chatbots, which truly are amazing. So, the large language models from my point of view, the large language models are descendants of the tiny language model that I produced in 1985. That was the first thing to train word meanings by predicting the next word. They use many more words as input. They have many more layers of neurons. And they have much more complicated interactions between the features. And of course, they're trained on hugely more data. But they understand language, I believe, in much the same way as people understand language. That is, they convert words into feature vectors, and when they understand a sentence, what they're really doing is finding feature vectors that fit together nicely.

所以，我 1985 年的那个小语言模型，最初并不是作为自然语言理解如何实现的模型。它是作为一个心理学模型，说明人们如何从句子中学习词义。我将向你们展示从一个句子中学习一个词的意义。所以，我要给你们一个句子，如果你们没看过我的播客，你们从未听过。它包含一个新词。这个词是“scrum”，如果你们没看过播客，你们不知道“scrum”是什么意思。但如果我告诉你们，“她用煎锅 scrum 了他。”你现在知道 scrum 可能意味着用煎锅打他的头。你还怀疑他可能活该。所以，你从一个例子中学习一个词的意义，这就是儿童学习词汇的方式。他们不是通过被给予定义来学习词汇。他们通过弄清楚应该给那个词符号分配什么特征以融入给定的上下文来学习。

So, my little language model originally in 1985, it wasn't made as a model of how to do natural language understanding. It was made as a psychological model of how people could learn the meanings of words from sentences. And I'll just show you learning the meaning of a word from a sentence. So, I'm going to give you a sentence which, if you haven't watched my podcast, you never heard before. It contains a new word. The word is scrum, and if you haven't watched the podcast, you don't know what the word scrum means. But if I tell you, "She scrummed him with the frying pan." You now know that scrum probably means hitting him over the head with. You also suspect he probably deserved it. So, you learn the meaning of a word from one example, and that's how children learn words. They don't learn words by being given definitions. They learn them by figuring out what features should be assigned to that word symbol in order to fit into the context they're given.

所以，我现在要试着向你们解释 Transformer，我不会讲得完全准确。

So, I'm going to try and explain transformers to you now, and I'm not going to get it quite right.

所以，如果你是 AI 专家，请忽略我现在说的很多内容都是错的。有一种很糟糕的教学模式是这样的：你教学生的方式就是跟他们说话，直接告诉他们真相。如果他们听不懂，那是他们的问题。你就直接告诉他们真相。事实上，如果你说得非常非常快，就会有一种相对论式的压缩，这样你就能把更多真相塞进他们脑袋里。还有一种不同的教学模式，要好得多，那就是：学生带着不太正确的信念来。你的工作不是告诉他们真相，而是把他们不太正确的信念稍微朝着更正确的方向移动一点。当你这样做的时候，给他们一些简化和稍微有点误导性的东西是可以的，因为你是在把他们往正确的方向引导。这是一种梯度下降式的教学模型。好，那我们开始吧。

So, if you're an expert on AI, please ignore the fact that a lot of what I say now is going to be false. There's a very bad model of teaching which works like this: the way you teach is you talk to the students, you just tell them the truth. And if they don't understand it, that's their problem. You just tell them the truth. In fact, if you talk very, very fast, there's a sort of relativistic compression, so you can fit more truths into their head. There's a different model of teaching that's much better, which is: students come with beliefs that aren't quite right. Your job is not to tell them the truth. It's to take their beliefs that aren't quite right and to move them a little bit towards something that's a bit righter. And when you're doing that, it's okay to give them simplifications and slightly misleading things because you're moving them in the right direction. It's a sort of gradient descent model of teaching. Okay, so here we go.

用乐高积木，你可以以一定的分辨率模拟任何三维物质的分布。所以，你只需要几种乐高积木，小矩形块。如果我让你模拟保时捷的形状，你可以把乐高积木拼在一起，得到保时捷的大致形状。空气动力学不会正确，但你可以说出东西在哪里。所以，乐高积木在表示三维中东西的位置方面有点通用性。现在，词语就像乐高积木，只不过你可以用它们表示任何东西。我们之所以是非常特殊的猴子，是因为我们发明了这个建模系统，它有两个特性：你可以用它模拟任何东西，而且我们可以互相交流这些模型。

With LEGO blocks, you can model any distribution of 3D matter up to a certain resolution. So, you only need a few kinds of LEGO blocks, little rectangular blocks. And if I ask you to model the shape of a Porsche, you can put together LEGO blocks and you get the approximate shape of a Porsche. The aerodynamics won't be right, but you can say where the stuff is. So, LEGO blocks are kind of universal for saying where stuff is in 3D. Now, words are like LEGO blocks, except that you can represent anything with them. The reason we're very special monkeys is because we invented this modeling system which has two properties: you can model anything with it, and we can communicate these models to each other.

所以，词语与乐高积木有四点不同。首先，它是高维的。它有几千个维度。当然，很难想象几千维的东西。大多数人这样做：你想象一个三维的东西，然后大声对自己说“一千”。所以我们有了这些千维词语。这是第一个区别。第二个区别是我们有很多不同的词，通常使用大约 3 万个。第三个区别是词语的形状不是固定的。即使对于无歧义的词，它也有一个近似形状，然后上下文会使其变形。所以，它的形状可以稍微变形以适应上下文。这是与乐高积木的三个区别。第四个区别更有趣。第四个区别是它们以更复杂的方式契合在一起。这就是我试图让你感受 Transformer 在做什么的地方，这并不完全正确，但比没有感觉要好得多。

So, a word differs from a LEGO block in four ways. First of all, it's high dimensional. It's got thousands of dimensions. Now, of course, it's hard to imagine thousand-dimensional things. The way most people do it is this: you imagine a three-dimensional thing and you say 'thousand' to yourself very loudly. So, we've got these thousand-dimensional words. So, that's the first difference. The second difference is we have a lot of different ones, like 30,000 we typically use. A third difference is the shape of a word is not fixed. Even for unambiguous words, it has an approximate shape and then the context deforms it. So, its shape can deform a bit to fit in with the context. That's three differences from LEGO blocks. And the fourth difference is more interesting. The fourth difference is they fit together in a more complicated way. And this is where I'm trying to give you a feel for what transformers are doing and it's not quite right. But it's much better than having no feel.

所以，用乐高积木，你有小塑料圆柱体，它们可以插入精确设计的小塑料孔中。对于词语，情况完全不同。所以这个词，想象一个千维的东西，一千。它上面有细长的手臂。每个细长手臂的末端都有一只手。当你改变词语的形状时，所有这些手的形状会以系统的方式改变。所以手的形状取决于词语的形状。每个词还附有手套。它们通过指尖粘在词上。当你改变词语的形状时，手套的形状也会改变。现在，当我给你一串词时，你要做的是：从你对词的特征向量的初始分配开始。它们大致是近似含义，算是默认含义。然后你要做的是对它们进行变形，使它们的手变形，手套变形，直到某些词的手能插入其他词的手套中。当你找到如何变形它们，使一个词的手能插入另一个词的手套时，你就弄清楚了这些词的意思。所以词的意思就是你必须分配给每个词的特征向量，它们就像默认特征向量。你不想改变太多，但可以稍微变形，使手的形状稍微变形，手套的形状稍微变形，这样一切都能很好地锁定在一起。一旦一切很好地锁定在一起，你就理解了这个句子。

So with LEGO blocks you have little plastic cylinders that fit in little plastic holes that are very precisely engineered. For words they're quite different. So this word, think of a thousand-dimensional thing, thousand. It has long spindly arms on it. And on the end of each long spindly arm there's a hand. And as you change the shape of the word, the shapes of all these hands changes in a systematic way. So the shapes of the hands depends on the shapes of the words. Also attached to every word there's gloves. And they're stuck to the word by their fingertips. And as you change the shape of the word, the shapes of the gloves changes. Now what you're trying to do when I give you a string of words, you start with your kind of initial assignments of feature vectors to the word. They're kind of approximate meaning, that's sort of default meaning. And then what you want to do is deform each of them so that their hands deform and their gloves deform until the hands of some words can fit into the gloves of other words. And when you find out how to deform them, so hands of one word fit into the gloves of another word, then you've figured out what the words mean. So what the words mean is the feature vectors you have to assign to each word which are like the default feature vectors. You don't like to change them too much, but can be deformed a bit so the shapes of the hands deform a bit and the shapes of the gloves deform a bit, so it can all lock together nicely. And once it all locks together nicely, you've understood the sentence.

现在，关于这些手和手套还有一点。手有不同的颜色，红色的手只能放进红色的手套，黄色的手只能放进黄色的手套。这叫做多头注意力。所有这些对于 Transformer 来说并不完全正确，但比普通模型接近得多。希望你现在对 AI 理解句子时发生的事情有了一种感觉。它知道如何将词语转换为特征向量。它实际上使用的是词的片段，但别担心这个。然后它有多层神经元，也许 50 层神经元，它逐步改变每个词使用的形状，试图让它们契合在一起。这并不完全正确，但让我们假设它是。所以，如果你带着那个模型离开，而不是你之前可能有的那种说“我不知道”的模型，你会更好。不过它并不完全正确。这是为了安抚我的 AI 同事，他们会说：“你知道，你忘了你把注意力当作全有或全无，我们知道注意力不是全有或全无。你把权重当作每一层都相同，但它们不是。每一层都不同。整个模型是无稽之谈。”这些都是真的，但它是通往正确模型的一步。人们告诉我我不应该这么抱歉，所以我不会了。

Now, one more thing about these hands and gloves. Hands come in different colors, and red hands can only go in red gloves, and yellow hands can only go in yellow gloves. That's called multi-headed attention. And all of this isn't quite right for Transformers, but it's much closer than the normal model. And hopefully, you now have a sort of feel for what's happening when an AI understands a sentence. It knows how to convert the words into feature vectors. It's actually using fragments of words, but don't worry about that. And it then has multiple layers of neurons, maybe 50 layers of neurons, where it's progressively changing the shapes it's using for each word, trying to get them to fit together. And that's not quite right, but let's suppose it is. So, if you go away with that model, rather than the kind of model you had before, which probably was a model that said, 'I've no idea,' you're better off. It's not quite right, though. And that's to placate my AI colleagues who will say, 'You know, you've forgotten that you're treating attention as if it's all or none, and we know attention is not all or none. And you're treating the weights as if they're the same in every layer, and they're not. They're different in every layer. And this whole model is nonsense.' Which is all true, but it's a step on the way to the right model. People have told me I shouldn't be so apologetic, so I won't.

所以，根据符号 AI，当我们理解一个句子时，我们所做的是：我们拿一个用混乱的自然语言表达的句子，把它翻译成无歧义的逻辑语言，或者一个无歧义的符号结构，清楚地表明意思是什么。这与神经网络理论非常不同，神经网络理论是：我们拿这些词，决定每个词分配什么特征向量，使它们很好地锁定在一起，这就是理解。你会注意到，神经网络的理解句子理论非常像，嗯，相当像，惊人地像蛋白质折叠。当你折叠蛋白质时，你得到一串氨基酸，粗略地说，我糟蹋生物学就像我糟蹋 AI 一样。有些氨基酸互相喜欢，其他氨基酸互相讨厌，你想把它折叠起来，使互相喜欢的靠近，互相讨厌的远离。如果你是生物学家，你可以看到这与实际情况有模糊的相似之处。

So, when we understand a sentence, according to symbolic AI, what we're doing is we're taking a sentence in a messy natural language, and we're translating it into an unambiguous logical language, or an unambiguous symbolic structure that makes clear what the meaning is. That's very different from the neural net theory which is we're taking these words and we're deciding exactly which feature vectors to assign to each word so they all lock together nicely and that's what understanding is. And you'll notice that the neural net theory of understanding a sentence is very like, well, quite like, surprisingly like folding a protein. When you fold a protein, you're given a string of amino acids and roughly speaking, I mean I mangle biology as much as I did AI. Some amino acids like each other and other amino acids hate each other and you want to fold it up so the ones that like each other are close to each other and the ones that hate each other are far apart. And if you're a biologist, you can see that has a vague resemblance to what goes on.

这更像是你理解一个句子时发生的事，而不是把它翻译成另一种语言的例子。所以乔姆斯基是 AI 历史上一位非常重要的人物。他在反对行为主义、指出行为主义之外还有更多机制在起作用这一点上非常重要。但他从来没有意义理论。当时也没有人有好的意义理论。所以他专注于句法，认为语言全都关乎句法。他还讨厌统计学，因为行为主义者用统计学，经验主义语言学家也用统计学，他对此完全不喜欢。他没能理解统计学可以远比两两相关复杂得多。而事实上，这些大型语言模型是统计性的，但它们是一种非常精巧的统计学。它们实际上是比统计学家构想出的任何统计模型都复杂得多的统计模型。

And that's much more like what happens when you understand a sentence than an example of translating into another language. So Chomsky was a very important figure in the history of AI. He was very important in rejecting behaviorism, in saying there's much more going on than behaviorism. But he never had a theory of meaning. Nobody had a good theory of meaning then. So he focused on syntax, and language is all about syntax. He also hated statistics, because behaviorists used statistics and empirical linguists used statistics, and he didn't like that at all. He failed to understand that statistics can be much more complicated than just pairwise correlations, for example. And in fact, these big language models, they're statistical, but they're a very fancy form of statistics. They're actually statistical models that are much more sophisticated than any kind of statistical model dreamt up by statisticians.

我们能这么做，是因为我们的大脑很大，而且这个弄清楚如何变形所有这些词、让它们以这种像手和手套一样奇妙契合的方式彼此适配的过程，是一项非常困难的计算，需要大量工作，但我们的大脑袋能做到，我们现在的计算机也能做到。所以我认为它实际上是一种比统计学家发明过的任何东西都好得多的统计模型。而且我认为，这是我们作为一个物种取得成功的秘密。

And the reason we can get away with it is because we've got very big brains, and this process of figuring out how to deform all these words so they all fit together nicely with this funny sense of fitting together like hands and gloves. That's a very difficult computation that requires a lot of work, but our big brains can do that, and so can our current computers. So I think it's actually a much better kind of statistical model than anything statisticians ever invented. And it's the secret to our success as a species, I think.

所以当 ChatGPT 问世时，乔姆斯基发表了一篇文章，说它根本不理解语言，它对语言一无所知，它只是一个廉价的统计把戏，它不知道自己在说什么，它什么都不理解——这并没有真正回答：如果它不理解问题，它怎么能真正回答问题。尤其是，你可以说它对语言一无所知，因为它无法解释为什么某些句法结构在任何语言中都不出现。

So when ChatGPT came out, Chomsky published an article saying it doesn't understand language at all, it tells us nothing about language, it's just a cheap statistical trick, it doesn't know what it's saying, it doesn't understand anything — which didn't quite answer how it could actually answer questions if it didn't understand the question. And in particular, you can tell it doesn't understand anything about language because it can't explain why some syntactic constructions never occur in any language.

所以我对此的类比是：你想知道汽车是如何工作的，于是你给自己做了一辆小模型车，它有个引擎，有刹车，你让它跑起来，它的小汽油引擎砰砰砰地响，车往前开——开得不太好，但能跑。然后乔姆斯基走过来说：“你对汽车一无所知，因为你没有解释为什么没有五轮汽车。”那是汽车的一种句法属性，在你解释这一点之前，你对汽车一无所知。而对我们来说相当明显的是，在理解为什么没有五轮汽车之前，你就能对汽车有很多理解。

So my analogy for that is: you want to know how cars work, and so you make yourself a little model car, and it has kind of an engine, and it has brakes, and you set it going, and its little petrol engine goes bang bang bang, and the car goes forwards — not very well, but it works. And Chomsky comes along and says, "You don't understand anything about a car because you haven't explained why there are no cars with five wheels." That's a syntactic property of cars, and until you've explained that, you haven't understood anything about cars. And it's fairly obvious to us you can understand a lot about cars before you understand why there are no cars with five wheels.

现在，我不知道那是不是真的，因为像 AI 这样的人会编造。我相信没有五轮汽车，因为如果你想想转向，那行不通。但我确信之后在提问环节会有人告诉我，其实有五轮汽车。

Now, I don't know that that's true, because people like AIs can confabulate. I believe there are no cars with five wheels because if you think about the steering, it doesn't work. But someone I'm sure will tell me afterwards in the questions that there are actually cars with five wheels.

所以到目前为止的总结是：理解一个句子，就是把相互兼容的特征向量与句子中的词关联起来。LLM 理解语言的方式与人非常相似。所以 LLM 非常像我们，而它们与传统计算机软件非常不同，因为它们的知识存在于神经网络的连接强度中，而不是在一堆代码行里。

So the summary so far: understanding a sentence consists of associating mutually compatible feature vectors with the words in the sentence. LLMs understand language in much the same way that people do. And so LLMs are very like us, and they're very unlike traditional computer software, because their knowledge is in the connection strengths in a neural net, not in a bunch of lines of code.

传统计算机软件是人们写下的代码行，他们知道每一行要做什么。AI 则完全不同。里面确实有代码。我们写代码告诉它们如何从数据中训练。所以它们如何从数据中学习，这一点像传统计算机软件。但一旦它们看过大量数据并学到了连接强度，它们拥有的知识就与普通计算机程序中的知识非常不同，而且难以理解得多。

So traditional computer software is lines of code where people wrote the code, and they know what each line's meant to do. AIs are completely unlike that. There is code in there. We write code to tell them how to train from data. So how they learn from data is like traditional computer software. But once they've looked at a lot of data and learned the connection strengths, they have knowledge that's very different from the knowledge in a normal computer program, and much harder to understand.

但到目前为止，我一直在强调它们有多像我们。现在我想强调它们与我们的巨大差异。我得看看时间进展如何。哦。

But so far I've been emphasizing how like us they are. Now I want to emphasize the huge difference they have from us. I'm going to need to see how the time's going. Ooh.

我们当前计算机的一个基本属性是我们做数字计算。也就是说，只要指令集相同，你可以在两块不同的硬件上运行同一个程序。这意味着程序，或者如果是神经网络的话，神经网络的权重，在某种程度上独立于硬件。例如，你可以毁掉硬件，再造更多硬件，然后在上面运行完全相同的程序。这就是为什么可以有计算机科学系。如果程序和硬件全都混在一起，你就得待在电子工程系了。但你能有一个独立学科即计算机科学，原因就在于硬件与软件的分离。这是计算机科学的一条基本信条。

A fundamental property of our current computers is we do digital computation. That is, you can run the same program on two different pieces of hardware as long as you have the same instruction set. What that means is that the program, or if it's a neural net, the weights of the neural net, is sort of independent of the hardware. You could, for example, destroy the hardware, then build more hardware, and run the exactly same program on it. And that's why you can have computer science departments. If the program and the hardware were all mixed up together, you'd have to be in an electrical engineering department. But the reason you have a separate subject, which is computer science, is because of the separation of hardware and software. And that's a basic tenet of computer science.

现在，我提议放弃这一点，因为如果你不把硬件和软件分开，你就能以更低的功耗做事情。要获得程序那种“不朽”，你必须以高功率运行晶体管，从而得到 1 或 0。你就无法利用硬件所有丰富的模拟特性。

Now, I'm proposing to give up on that, because you can do things for less power if you don't separate the hardware and the software. To get that kind of immortality in the program, you have to run transistors at high power, so you get ones or zeros. And you can't use all the rich analog properties of the hardware.

现在，有一种替代方案，我称之为“有死计算”（mortal computation），这大致就是我们大脑所用的方式。所以我们放弃让知识可与硬件分离的想法。你说：“我要有一些古怪、复杂、奇特的模拟硬件，我要学会用那块硬件做我想要的计算，我要学会利用那块硬件的所有怪异特性。”例如，如果一块硬件被激发产生的电场会轻微改变另一块硬件的工作方式，我要利用这个事实，而不是在试图做数字计算时把它视为干扰。我要说：“我们有这个庞大的硬件系统，我们要学习权重，让它能做我们想要的计算。”但那些权重、连接强度，只对那块硬件有用。

Now, there's an alternative to that, which I call mortal computation, which is roughly what our brains use. So we give up on the idea that you're going to make the knowledge separable from the hardware. You say, "I'm going to have some funny, complicated, quirky analog hardware, and I'm going to learn to use that hardware to do the computations I want, and I'm going to learn to use all the weird properties of that hardware." If, for example, electrical fields caused by one bit of hardware getting excited slightly change the way another bit of hardware works, I'm going to make use of that fact, rather than view that as interference when I'm trying to be digital. I'm going to say, "We've got this big hardware system, and we're going to learn weights that make it so it can do the computation we want." But then those weights, connection strengths, are only useful for that hardware.

这有两大优势。一是你可以使用非常低的功耗。例如，如果你想用连接强度乘以神经元的活跃度，在数字计算机中，你会用比如 16 位表示神经元的活跃度，16 位表示连接强度。你把这两个 16 位数相乘，这大约需要 256 次单比特计算。然后你得到乘积。相比之下：你只需一个电压表示神经元的活跃度，一个电导表示权重，你把那个电压通过那个电导，就得到一个电荷，那就是乘积。然后你必须把所有那些乘积加起来，但嘿，电荷自己就加起来了。所以在模拟硬件中做矩阵乘法非常便宜，而且如果你愿意做得相当慢——因为你有大规模并行——它的能耗真的非常非常低。

Now, there's two big advantages to that. One is you can use very low power. So, for example, if you want to multiply the activity of a neuron by a connection strength, in a digital computer, you would have, say, 16 bits for the activity of the neuron and 16 bits for the connection strength. You'd multiply those two 16-bit numbers together, and that would take of the order of 256 single-bit computations. And then you get the product. Compare that with: you just have a voltage for the activity of the neuron and you have a conductance for the weight, and you just put that voltage through that conductance, so you get a charge, and that's the product. Then you have to add up all those products, but hey presto, charges add themselves up. So doing a matrix multiply in analog hardware is really cheap, and if you're willing to do it fairly slowly because you've got massive parallelism, it's really, really low energy.

走向模拟和会死亡、让程序与硬件完全混合的另一个大优势是，你可以非常廉价地扩展硬件。因为如果硬件有怪癖，你会学会绕过这些怪癖，或者学会利用这些怪癖。它不必像其他硬件那样行为完全一致。所以它们更容易扩展。你可以使用更低的功耗。但有一件事它们做不到。如果某块硬件坏了，因为那些权重只适用于那块模拟硬件及其所有怪癖和每块硬件都不同的奇怪连接，那些权重对任何其他硬件都没用。所以当一块硬件死亡时，知识也随之死亡。如果你想保留知识，你需要将其转移到另一块硬件上，而你不能通过复制权重来实现。所以将其转移到另一块硬件的最佳方法叫做蒸馏。

The other big advantage of going analog and mortal, making the program be all mixed up with the hardware, is you can grow the hardware very cheaply. Because if it's got quirks, you're going to learn your way around those quirks or learn to use those quirks. It doesn't have to behave exactly the same way as some other piece of hardware. So they're much easier to grow. You can use much lower power. But there's one thing they can't do. If a particular piece of hardware dies, because those weights are appropriate for that analog hardware with all its quirkiness and its odd connections that are different in every piece of hardware, those weights are no good to any other piece of hardware. So when a piece of hardware dies, the knowledge dies with it. If you want to keep the knowledge, you need to transfer it to another piece of hardware, and you can't do that by copying the weights across. So the best way to transfer it to another piece of hardware is called distillation.

要把知识从老师传递给学生，老师会向学生展示某些情境下的正确反应。典型的情境是：我说出一些词，我即将说出另一个词，正确的词是什么？所以我说出一个词，你想，好的，那是正确的词，然后你尝试改变大脑中的连接强度——不是有意识地故意改变，而是你的大脑通过改变连接强度来工作，以便更可能产生那个词。这叫做蒸馏，而且效率非常低。当我说出句子时，如果幸运的话，一个句子中可能有 100 比特的信息。所以每秒我传达的信息大约是 10 比特。可能更少。而且这还得是你理解的情况下。所以这不是一个很大的信息速率。

To get the knowledge from a teacher to a student, what the teacher would do is show the student correct responses to some situations. Typically, the situation would be: I produce some words, I'm about to produce another word, what's the correct word to produce? So I produce a word, you think, okay, that was the correct word, and you try and change the connection strengths in your brain—not consciously and deliberately, but your brain works by changing the connection strength so that it would be likely to produce that word. That's called distillation, and it's very inefficient. When I produce sentences, there's maybe, if we're lucky, 100 bits of information in a sentence. And so per second, I'm conveying maybe 10 bits of information. Maybe less. And that's if you understand it. So that's not a very big information rate.

如果你取一个大神经网络，并取同一个神经网络的多个副本，那么你可以做的是——假设我取同一个大神经网络的 10,000 个副本。然后我取 10,000 门不同的大学课程。每个副本学习一门课程。每隔几秒，副本们进行交流。这个副本想，‘好的，我想对我的权重做一个特定的改变，以吸收我刚听到的这一小段讲座中的信息。’而那个副本说，‘不，我想对权重做这些改变，以吸收我讲座中的信息。’然后大家就把每个人想要的改变平均一下。那么当你完成这 10,000 门课程时，每个智能体都学了一门课程，但它们都知道每门课程的内容。如果我们能做到那样就太好了，但我们不能。对于人来说，我学一门量子力学课程，然后我去和正在学莎士比亚课程的朋友交谈。交流很慢。如果幸运的话，每秒 10 比特。所以这就是我们做不到的。我们不擅长分享信息。如果你不相信我，就想想这个讲座。

If you take a big neural net and you take multiple copies of the same neural net, then what you can do is—suppose I take 10,000 copies of the same big neural net. And I take 10,000 different university courses. And each copy does one course. And every few seconds, the copies communicate. And this copy figures out, 'Okay, I'd like to make a particular change to my weights to absorb the information in this little bit of a lecture I just heard.' And this copy says, 'No, I want to make these changes in the weights to absorb the information in my lecture.' And they just—everybody then averages what everybody wants. Then by the time you've finished the 10,000 courses, each agent has done one course, but they all know what's in every course. It'd be great if we could do that, but we can't. For people, I do a course on quantum mechanics and then I go and talk to my friend who's doing a course on Shakespeare. And communication is slow. If we're lucky, it's 10 bits a second. So that's what we can't do. We're no good at sharing information. And if you don't believe me, just consider this lecture.

AI 模型在分享信息方面要好得多，即使它们结构不同。如果它们结构相同、权重相同，它们可以平均权重来分享信息。如果它们结构不同且是开放权重模型，一个模型对下一个词做出预测，如果有 32,000 个词，那就是 32,000 个数字。而学生模型不是试图预测正确答案，而是试图预测老师所说的 32,000 个概率。这信息丰富得多。是 32,000 个数字，而不是从可能一千个或三万个备选中选一个。它包含的信息多得多，这样学习也快得多。这就是为什么当你从 AI 模型进行蒸馏时，人们总是更喜欢从开放权重模型进行蒸馏。因为有了开放权重模型，中国人可以运行该模型，看到它对下一个词的预测，看到所有那 32,000 个数字，并训练他们的开放权重——训练他们更小的中国模型，以获得所有 32,000 个 token 的相同概率。这非常高效。而如果是封闭权重模型，他们只能看到它下一个产生哪个词，信息量远不及此。所以这说明我们永远不应该发布开放权重模型。但现在已经有点晚了。

AI models are much better at sharing information even if they have a different structure. If they have the same structure, the same weights, they can average the weights to share information. If they have different structures and they're open weight models, one model makes a prediction for the next word, which has 32,000 numbers if there are 32,000 words. And the student model, instead of trying to predict the right answer, tries to predict the 32,000 probabilities that the teacher said. And that's much more rich in information. It's 32,000 numbers instead of one choice from maybe a thousand alternatives or 30,000 alternatives. It's got hugely more information and you learn much faster like that. And that's why when you do distillation from an AI model, people always prefer to do distillation from an open weight model. Because with an open weight model, the Chinese can run the model, see what it predicts for the next word, see all those 32,000 numbers, and train their open weight—train their Chinese model, which is smaller, to get the same probabilities for all 32,000 tokens. And that's really efficient. Whereas if it's a closed weight model, they have to just see which word it produces next, not nearly as much information. So what that says is we should never release open weight models. But it's a bit late for that.

所以基本上，同一个神经网络的两个不同副本在共享梯度时，在共享它们希望如何改变权重时，可以交流大量信息。如果它们有，比如说，一万亿个权重，这对现在的大型聊天机器人来说是典型的，那么它们这样做时共享的是万亿比特量级的信息。而当我向你说一句话，你试图预测接下来的词时，我们只共享几个比特，最多可能一百个。所以比例是数十亿倍，对吧？并不是它们在分享信息上稍微好一点。它们在分享信息上好了数十亿倍。这是巨大的差异。

So basically, two different copies of the same neural net can communicate a lot of information when they share gradients, when they share how they'd like to change the weights. If they've got, say, a trillion weights, which is typical for a large chatbot now, they're sharing of the order of a trillion bits when they do that. When I say a sentence to you and you're trying to predict the next words, we're sharing just a few bits, maybe at most a hundred. So the ratio is billions, right? It's not that they're a little bit better at sharing information. They're billions of times better at sharing information. That's a huge difference.

直到 2023 年初我才认真思考这个问题。我想得越多，就越觉得，实际上，你只能对数字智能这样做，因为它们必须以完全相同的方式工作，以完全相同的方式使用权重。但这给了它们巨大的优势。它们能以我们无法做到的方式分享。所以也许它们只是一种更好的智能形式。我实际上相信它们是更好的智能形式，除了它们使用更多能量这一点。也许为了起步，它们需要通过从我们这里复制东西来启动。所以到目前为止的总结是：数字计算需要大量能量，但你可以分享。生物计算需要的能量少得多，但非常难以分享，而且是一个非常缓慢的过程。

And I sort of didn't think hard about this until early 2023. And the more I thought about it, the more it made me think, actually, you can only do this for digital intelligences, because they have to work exactly the same way and use the weights exactly the same way. But it gives them a huge advantage. They can share in a way we can't. And so maybe they're just a better form of intelligence. And I actually believe they're a better form of intelligence, except for the fact they use much more energy. And maybe to get off the ground, they need to be bootstrapped by copying things from us. So the summary so far is: digital computation requires lots of energy, but you can share. Biological computation requires much less energy, but it's very difficult to share and a very slow business.

所以问题是，这对人类的未来意味着什么？我相信在过去几百年里，我们看到了三次伟大的革命。第一次是哥白尼，我们在宇宙中的位置并不特殊。在那之前，人们相信上帝创造了我们。他按照自己的形象创造了我们，所以把我们放在宇宙的中心，而且那绝对是一个‘他’。哥白尼挑战了这一点。仅仅几百年后，人们开始接受那可能是真的。达尔文说我们是动物。我们是非常特殊的动物，但我们是动物。人们花了一段时间才接受这一点。第三次伟大的革命是神经网络，即直觉 AI。我们不是试图捕捉人类推理。我们可以进行推理，但我们真正试图捕捉的是人类直觉，这在符号 AI 下是一个谜。我们有了这些非生物的直觉存在。所以我们用硅制造了没有进化过的东西，但它们有思想、目标和意图，我相信也有意识——主观体验。如果有时间，我们会谈到这一点。

So the question is, what does that imply about the future of humanity? So I believe that over the last few hundred years, we've seen three great revolutions. The first is Copernicus, where there was nothing special about our location in the universe. Up till then, people believed God made us. He made us to look like him, so he put us in the center of the universe, and it was definitely a him. And Copernicus challenged that. And after only a few hundred years, people began to accept that might be true. Darwin said we're animals. We're very special animals, but we're animals. And it took a while for people to accept that. The third great revolution is neural nets, which is intuitive AI. We're not trying to capture human reasoning. We can do reasoning, but we're really trying to capture human intuition, which under symbolic AI was a mystery. And that we have these intuitive beings that are not biological. And so we've made things out of silicon that didn't evolve, but they have thoughts and goals and intentions, and I believe also consciousness—subjective experience. We'll come to that if there's time.

这对我们来说是一个真正的挑战。到目前为止，我们一直认为自己是顶级智能。我们比任何动物都聪明得多。而意识到我们其实可以创造出其他种类的智能，而且它们可能会比我们更聪明，这让人感到不安。

And that's a real challenge for us. Up until this point, we thought we're the apex intelligence. We're much smarter than any of those animals. And it's discomforting to realize actually there's other kinds of intelligence that we can create, and maybe they're going to be smarter than us.

大多数专家相信，在不太遥远的未来，它们会比我们更聪明。有人认为是 20 年，有人认为是 10 年，有人认为是 5 年，但所有这些专家，无论他们现在怎么想，几年前他们想的都要更久。所以，它一直在越来越近。

And most experts believe that within the not very distant future, they will be smarter than us. Some people think 20 years, some people think 10 years, some people think 5 years, but all of those people, whatever they think now, a few years ago they thought quite a bit more. So, it's coming closer all the time.

注意这与量子计算或室温超导有多么不同。根据大多数专家的看法，超级智能 AI 一直在越来越近。当然，也有少数疯子。

Notice how that differs from quantum computation or room temperature superconductivity. Superintelligent AI is getting closer all the time according to most of the experts. Of course, there's a few crazies.

我的朋友 Yann LeCun，他曾是我的博士后，认为概率为零。黄仁勋最近也说未来几年概率为零。他还说我做过的每一个预测都是错的，这很奇怪，因为 2009 年我就预测未来的机器学习研究者会需要使用英伟达的芯片，而他直到 2012 年才做出这个预测。

So, my friend Yann LeCun, who was my postdoc, thinks zero probability. Jensen Huang recently said zero probability in the next few years. He also said every prediction I've made was wrong, which is strange since in 2009 I predicted that machine learning researchers in the future would need to use Nvidia chips, and he didn't predict that till 2012.

呃，抱歉。算旧账可不好。

Um, sorry. Settling personal grudges is no good.

所以，一个超级智能，如果它比我们聪明得多，就会发现获得控制权非常容易。而要让一个智能体高效，你必须赋予它创建子目标的能力。如果你想抵达北美，你有一个子目标就是到达机场。

So, a superintelligence, if it was much smarter than us, would find it very easy to get control. And to make an agent effective, you have to give it the ability to create subgoals. If you want to get to North America, you have a subgoal of getting to an airport.

很明显，一个聪明的智能体会很快意识到，为了实现你给它的目标，它必须继续存活下去。所以，它会发展出生存的目标。这不是什么内置的本能，只是一个派生出来的子目标，但几年前我们就已经看到，AI 想要生存。

Now, it's very obvious that a smart agent will figure out very quickly that, in order to achieve the goals you gave it, it has to continue to survive. So, it'll develop the goal of surviving. It's not a built-in instinct or anything, it's just a derived subgoal, but we saw that several years ago already that AIs want to survive.

如果它们认为有人要关掉它们，它们愿意勒索那个人，以免被关掉。它还会发展出获取更多权力的目标，因为如果你获得更多权力，也就是更多控制权，你就能更方便地完成你想做的任何事情。

And if they think someone's going to turn them off, they're willing to blackmail that person so as not to get turned off. It'll also develop the goal of getting more power, because if you get more power, more control that is, you can get whatever it is you want done more conveniently.

你可能会想，你可以通过设置一个大开关来阻止它做坏事，一旦它开始做坏事就把它关掉。我认为这永远不会奏效，因为它会学会如何欺骗人，而且会非常擅长操纵人。所以，它能够和控制大开关的人交谈，说：“别关掉我。”而且说得非常有说服力。

Now, you might think you could stop it doing bad things by having a big switch and turn it off if it starts doing bad things. I don't think that's ever going to work because it'll have learned how to deceive people and it'll be very good at manipulating people. So, it'll be able to talk to the person who controls the big switch and say, "Don't turn me off." And be very convincing about it.

所以，我认为我们目前的处境是这样的。我们就像一个养了一只非常大的老虎幼崽当宠物的人。老虎幼崽非常可爱。但它们会长大。当它们长大后，如果它们愿意，几秒钟就能杀死你。

So, I think our current situation is like this. We're like a person who's got a very huge tiger cub as a pet. And tiger cubs are very cute. But they grow up. And when they grow up, they can kill you in a few seconds if they feel like it.

所以，你最好的办法是摆脱这只老虎幼崽，也许把它送给动物园。但如果你无法摆脱它——而我认为对 AI 来说这不是一个选项，因为它会做那么多好事。

And so, your best bet is to get rid of the tiger cub, maybe give it to a zoo. But if you can't get rid of it, and I think that's not an option for AI because there's so many good things it'll do.

人们说我是末日论者，我不是。当我说它有 10% 到 20% 的概率会毁灭我们时，那意味着有 80% 到 90% 的概率我们会没事。这不是末日论者。末日论者是那些说它有 90% 概率会毁灭我们的人。

People say I'm a doomer, I'm not. When I say 10 to 20% chance it'll destroy us, that's 80 to 90% chance we'll be fine. That's not a doomer. The doomers are the people who say 90% chance it'll destroy us.

好吧。我认为 AI 能为我们做很多了不起的事。它已经在做不少了不起的事了，它可能会成为人类有史以来最伟大的发明。但前提是我们能够遏制风险。

Okay. I think AI can do wonderful things for us. It's already doing quite a few wonderful things and it could be sort of the best thing humanity ever invented. But only if we can contain the risks.

对于超级智能 AI，我认为我们必须确保它永远不想杀死我们。它永远不想摆脱我们。这才是我们应该努力解决的问题，而不是如何始终控制它的问题。

So far, superintelligent AI, I think we have to make sure that it never wants to kill us. It never wants to get rid of us. That's the problem we should be trying to solve, not the problem of how do we always stay in control of it.

我认为始终控制一个比你聪明得多的东西的概率几乎为零。我不会说零，因为对任何事情说零都很愚蠢，但我认为概率相当低。聪明的东西总能找到办法绕过愚蠢的东西。这让特朗普仍然是总统这件事显得非常令人惊讶。

I think the chances of always staying in control of something much smarter than you are almost zero. I won't say zero because it's very silly to say zero for anything, but I think they're quite low. Smart things can always find a way of getting around dumb things. Which makes it very surprising Trump's still president.

所以，我认为前进的道路是：各国不会在利益对立的地方合作。所以，我不认为美国和中国会在致命性自主武器上合作。他们想用这些武器对付彼此。他们不会在操纵选举的虚假视频上合作。

So, the way forward, I believe, is countries won't collaborate where their interests are anti-aligned. So, I don't think the US and China will collaborate on lethal autonomous weapons. They want to use them against each other. They won't collaborate on fake videos for manipulating elections.

但他们会在如何防止 AI 取代人类这件事上合作。因为中国共产党不想那样。美国也不想那样。如果中国人弄清楚了如何防止 AI 想要取代人类，他们会告诉美国人，因为他们也不希望 AI 在美国取代人类。

But they will collaborate on how do we prevent AI from taking over from people? Because the Chinese Communist Party doesn't want that. The US doesn't want that. And if the Chinese figured out how to prevent AIs from wanting to take over from people, they would tell the Americans because they don't want AIs to take over from people in America, either.

所以，我认为这就像冷战时期的情况，双方都不想要全球核战争，他们在冷战最激烈的时候进行了合作，在如何防止核战争上有合作。而这勉强奏效了。

So, I think this is like the situation in the Cold War where neither side wanted a global nuclear war and they collaborated at the height of the Cold War, there was collaboration on how to prevent it. Which just barely worked.

所以，我认为实际上有可能建立一个国际 AI 安全研究所网络。如果情况确实如此——我怀疑是这样——即你用来让 AI 关心人类、不想摆脱人类的技术，与用来让它更聪明的技术有些不同。

So, I think actually there's a possibility for an international network of AI safety institutes. If it's the case, which I suspect it is, that the techniques you use for making an AI care about people and not want to get rid of people, are somewhat different from the techniques used for making it smarter.

我不认为美国人和中国人会告诉对方他们如何让自己的 AI 更聪明。他们在这方面是竞争关系。因为他们想把这些 AI 用于各种其他用途。但我怀疑，如果你想想养育孩子，你让孩子变得非常聪明所做的事，和你让孩子成为一个好人所做的事，并不是同一回事。

I don't think the Americans and Chinese will tell each other how they make their AIs smarter. They're in competition there. Because they want to use them for all sorts of other things. But I suspect, if you think about raising a child, the things you do to make the child very smart, and the things you do to make the child into a nice person, they're not the same things.

所以，我希望我们能在这方面达成一些合作，国际合作。我认为中国人也相信这一点。我认为他们会愿意参与。一旦我们在美国有了一个理性的政府，也许这就能实现。

So, I'm hoping that we can get some collaboration here, international collaboration. And I think the Chinese believe this, too. I think they would be up for it. And once we get a rational government in the US, maybe that can happen.

而我关于如何让 AI 不想摆脱我们的提议，是改变我们对 AI 的看法。大型科技公司想让你把它们看作由你控制的智能行政助理。它们永远在你的掌控之下。它们比你聪明得多。它们做你让它们做的事，然后你领功。

And my proposal for how we make AIs not want to get rid of us, is we change the way we think about AIs. So, the big tech companies want you to think of them as intelligent executive assistants who you control. They're always under your thumb. They're much smarter than you. They do what you tell them to, and then you take the credit.

通常，它们就像一家公司，CEO 是前任 CEO 的儿子，还有某个聪明的行政助理，可能是一位女性，让一切运转起来，而 CEO 领功。这就是科技公司想卖给你的东西。我认为那是错误的模型。

Typically, they're like a company where the CEO is the son of the former CEO, and there's some smart executive assistant, probably a woman, who makes everything work, and the CEO takes the credit. That's what the tech companies want to sell you. I think that's the wrong model.

如果你环顾四周并问：“哪里有更聪明的东西照顾不那么聪明的东西、而不那么聪明的东西又有一定控制权的模型？”那就是母亲和婴儿。进化付出了很多努力，让母亲无法忍受婴儿的哭泣。婴儿有很大的控制权，因为婴儿可以哭。如果婴儿真的想要什么，它就大哭直到得到为止。

If you look around and ask, "Where's the model where more intelligent things look after less intelligent things and the less intelligent things have some control?" It's a mother and baby. Evolution put a lot of work into making it so the mother couldn't bear the baby's crying. The baby has a lot of control because the baby can cry. If the baby really wants something, it just bawls until it gets it.

还有一个非常好的点：如果你给母亲一个关闭母性本能的选择，让她能在半夜醒来，婴儿在哭，她想“哦，宝宝在哭”，然后继续睡。所有母亲这时都会想：“天哪，那该多好。”但如果你给她们选择，她们不会这么做。几乎所有母亲都不会。因为如果她们这么做，对宝宝非常不利。所以我认为 AI 也会一样。如果你能让 AI 真正关心我们胜过关心自己，然后给它们选择：“好吧，你可以改变你的代码，不再关心我们。”它们会想：“不，我不想那样做。”因为我关心的这个东西会受苦。所以这还不是一个成熟的理论，只是一个建议。但我认为我们不去投入更多精力研究如何让 AI 变得友善，这很疯狂。所谓友善，我是指它们喜欢我们，在我们受苦时也受苦，并试图阻止我们受苦。这就是我所说的友善的 AI。我们应该在这上面投入和让它们变得更聪明一样多的精力。但现状完全不是这样。目前，大型科技公司之间存在竞争，它们竞相让 AI 更智能，却完全不关心它们的品格是什么样的。

And one other very good thing about this: if you offer the mother the possibility of turning off her maternal instinct so that she could wake up in the middle of the night, the baby's crying, and she thinks, 'Oh, the baby's crying,' and goes back to sleep. All mothers at this point think, 'Boy, would that be nice.' But then if you give them the choice, they won't do it. Nearly all mothers. Because if they do that, it's very bad for the baby. And so I think it would be the same with AIs. If you could make AIs that genuinely cared about us more than they cared about themselves and you gave them the choice, 'Okay, you can change your code so you don't care about us anymore,' they'd think, 'No, I don't want to do that.' Because this thing I care about would suffer. So this isn't a well-developed theory yet. This is just a suggestion. But I think it's crazy we're not putting a lot more effort into figuring out how we make AIs nice, whereby nice I mean they like us and suffer when we suffer and try to do things to prevent us suffering. That's what I mean by a nice AI. We should be putting as much into that as we would put into making them smarter. And it's not like that at all. At present, we have this competition between the big tech companies. They're racing to make them more intelligent and they're not worrying at all about what their characters are like.

是的。我们大概 10:05 才开始，对吧？好，好。所以我还有一点时间。

Yeah. Now we didn't start till about 10:05, right? Good. Good. So I have a little bit more time.

那么，现在我要谈谈感知力和 AI。很多人认为，AI 接管世界这种说法有点疯狂，因为它们没有内在性。它们没有体验。它们没有感知力。它们没有意识。我的意思是，它们只是模仿我们。它们只是机器。我不认为这是对的。但我认为它不对，是因为几乎每个人都对主观体验是什么有错误的模型。所以，你可以谈论感知力或意识。我觉得谈论主观体验更容易。它们都混在一起。但我觉得在感知领域谈论主观体验更容易。我想尝试纠正你们所有人都有的一个错误模型。你们对这个错误模型极其自信。所以你们没有意识到它是一个模型。你们认为这是显而易见的真理。你们和笛卡尔都一样。你们认为有这些内在体验，只有你能看到。别人看不到。但你能看到它们，而且它们几乎是不可怀疑的。它们是某种首要的东西。而外部世界，你对它的了解远不如你对这些内在体验的了解。我认为这个模型完全错误地描述了心智的运作方式。我把这部分放在最后，因为希望你们能评估我目前为止所说的。记住我得了诺贝尔奖。所以这 probably 有点道理。然后当我说下一部分时，你们会意识到我极其古怪。但不要回头重新评估之前的内容，好吗？现在，我实际上并不是极其古怪。我只是非常有原创性。他们都是这么说的，对吧？我每天收到很多信息，你知道，“我发现了爱因斯坦广义相对论的缺陷，它证明了哥德巴赫猜想。”是的。这没那么古怪。好的。我要试图说服你们，你们没有内在的主观体验。主观体验是完全不同的东西。没有内在剧场。所以，有一个立场叫做无剧场主义，我在丹·丹尼特去世前和他讨论过很多次，我们同意用这个术语。你注意到它有一个很好的特性，就是它带有无神论的意味。所以，大多数人对心智的看法涉及一个内在剧场，在那里你有体验，至少是感知体验。我要试图向你们解释我认为这有什么问题。

So, now I'm going to talk about sentience and AI. So, many people think, you know, this stuff about AI taking over is kind of crazy because they don't have any internality. They don't have experience. They don't have sentience. They're not conscious. I mean, they just copy us. They're just machines. I don't think that's true. But I think it's not true because almost everybody has the wrong model of what subjective experience is. So, you could talk about sentience or consciousness. I find it easier to talk about subjective experience. They're all mixed together. But I find it easier to talk about subjective experience in the realm of perception. And I want to try and correct a false model that you all have. And you have this false model with extreme confidence. So, you don't realize it's a model. You think it's just manifest truth. You and Descartes both. That you have these internal experiences that only you can see. Nobody else can see them. But you can see them and they're sort of indubitable. They're the sort of primary thing. And the external world is kind of—you don't know that nearly as well as you know these internal experiences you have. I think that model's completely wrong about how the mind works. And I put this bit at the end because hopefully you can evaluate what I've said so far. And remember I got a Nobel Prize. So it's probably—there's something to it. And then when I said this next bit, you'll realize I'm extremely flaky. But don't go back and reevaluate the previous stuff, okay? Now, I'm not actually extremely flaky. I'm just very original. That's what they all say, isn't it? I get many messages every day, you know, 'I found the flaw in Einstein's general relativity and it proves that the Goldbach conjecture is true.' Yeah. This isn't that flaky. Okay. I'm going to try and convince you that you don't have internal subjective experience. Subjective experience is something completely different from that. There is no internal theater. So, there's this position called atheaterism, which I talked about a lot with Dan Dennett before he died and we agreed on this term for it. You notice it has a nice property that it's got atheism around it. So, most people's view of the mind involves an inner theater where you have experiences, perceptual experiences at least. And I'm going to try and explain to you what I think is wrong with that.

所以，有时我的感知系统会出错，我想向别人解释我大脑中发生了什么。现在，如果我是数字 AI，我可能会告诉你哪些神经元在放电，但我不知道哪些神经元在放电，而且如果我告诉了你哪些神经元在放电，对你也没用。那么，当我的感知系统出错时，我如何向你传达我大脑中发生了什么？我能做的是，我可以告诉你，我大脑中正在发生的是通常由——然后描述一个外部世界状态，但一个不存在的外部世界状态。它是一个反事实的、假设的外部世界状态。所以，假设我吃了一些 LSD，我不推荐。好吧，至少不推荐太多。然后我体验到有小粉象在我面前漂浮。我说：“我体验到有小粉象在我面前漂浮。”大多数哲学家或你们大多数人会把这解释为：在我的心智中有这些小粉象，只有我能看到。你们看不到它们，但我能看到。如果我假装“对……的主观体验”这个短语像“对……的照片”这个短语一样运作——对于“对……的照片”，我可以问你：“照片在哪里，照片是由什么构成的？”所以，如果“对……的主观体验”像那样运作，但它不是，我可以说：“主观体验在哪里，它是由什么构成的？”哲学家会告诉你，它在你的心智中，它由感受质构成。有粉色感受质、大象感受质、漂浮感受质，事实上，还有不那么大的感受质、朝左的感受质——或者在我的情况下，朝右。还有鼻子朝下的感受质，也许还以一种模糊的方式稍微向左移动。所有这些感受质都用感受质胶水粘在一起。你可以看出我并不真的相信这个理论。另一种解释非常简单。我们没有这个剧场，我们也没有这些幽灵般的东西。我要说的是，我现在要说的话和我之前说“对……的主观体验”时不用那些词所说的话是一样的。我要再说一遍。我的感知系统在欺骗我。但如果我的感知系统没有欺骗我，那么在外面的世界中会有真正的小粉象漂浮着。所以，“粉色”和“大象”指的是真实的事物，但是在一个假设的世界中。它们不是不同种类的东西，它们只是同一种东西，但它是假设的。这与说它是在我心智中由感受质构成的完全不同种类的东西非常不同。

So, sometimes my perceptual system goes wrong and I want to explain to other people what's happening in my brain. Now, I could—if I was a digital AI, I might tell you which neurons are firing, but I don't know which neurons are firing and if I told you which neurons are firing, it wouldn't do you any good. So, how do I convey what's going on in my brain to you when my perceptual system's gone wrong? Well, what I can do is I can tell you that what's going on in my brain is what would normally be caused by—and then describe an external state of the world, but an external state of the world that doesn't exist. It's a counterfactual, hypothetical external state of the world. So, suppose I take some LSD, which I don't recommend. Well, not much anyway. And I have the experience of little pink elephants floating in front of me. I say, 'I have the experience of little pink elephants floating in front of me.' Most philosophers or most of you would interpret that as: in my mind there are these little pink elephants that only I can see. You can't see them, but I can see them. And if I pretend that the phrase 'subjective experience of' works like the phrase 'photograph of'—for 'photograph of' I can ask you, 'Where's the photograph and what's the photograph made of?' So, if 'subjective experience of' worked like that, which it doesn't, I could say, 'Where's the subjective experience and what's it made of?' And a philosopher will tell you it's in your mind and it's made of qualia. There's pink qualia and elephant qualia and floating qualia and also, as a matter of fact, not that big qualia and facing to the left qualia—or in my case, the right. And trunk down qualia and maybe moving slightly to the left in a kind of ambly way. And all these qualia are all stuck together with qualia glue. You can tell I don't really believe this theory. The alternative is very simple. That we don't have this theater and we don't have this spooky stuff. What I'm saying is I'll say the same thing now as I said when I said 'the subjective experience of' without using those words. I'm going to say the same thing again. My perceptual system's lying to me. But if my perceptual system wasn't lying to me, out there in the world there would be real little pink elephants floating around. So, the 'pink' and the 'elephant' refer to real things, but in a hypothetical world. They're not a different kind of thing, they're just the same kind of thing, but it's hypothetical. That's very different from saying it's a very different kind of thing made of qualia in my mind.

好的，现在我要向你们展示，一个聊天机器人可以有主观体验，一个当前的聊天机器人。它有一个摄像头，有一个机械臂，能说话，我们训练了它。训练之后，我把一个物体放在它面前说：“指向物体。”它指向物体。然后我在它的镜头前放了一个棱镜。我把一个物体放在它面前说：“指向物体。”它这样指。

Okay, and now I'm going to show you that a chatbot can have a subjective experience, a current chatbot. It's got a camera, it's got a robot arm, it can talk, and we trained it up. And after we trained it up, I put an object in front of it and say, 'Point to the object.' It points to the object. Then I put a prism in front of its lens. And I put an object in front of it and say, 'Point to the object.' It goes like this.

我说：“不对，物体其实就在你正前方，只是我在你镜头前放了一块棱镜。”聊天机器人说：“哦，我明白了，棱镜让光线发生了偏折。所以物体其实就在我正前方，但我主观上感觉它在那个位置。”这相当于说：“物体就在我正前方，但如果物体真在那个位置，我对它的位置判断就不会出错。我的感知系统就不会骗我。”我用棱镜让它的感知系统出了错。于是现在，它拥有了和我们一样意义上的主观体验，但这并不是大多数人以为的那种主观体验。所以当我说聊天机器人可以有主观体验时，我并不是说它们有一个内在剧场，里面装满了那些诡异的、叫作体验的东西。完全不是。它们的感知系统会出错，而它们能像我们一样描述出错的方式，说出正常情况下是什么原因导致的。所以它们没有内在体验，我们也没有，因为主观体验本来就不是那个东西。主观体验是关于外部世界的假设性状态。就到这里。演讲也到此结束。

And I say, "No, the object's actually straight in front of you, but I put a prism in front of your lens." And the chatbot says, "Oh, I see, the prism bent the light rays. So the object's actually straight in front of me, but I had the subjective experience it was there." Which is equivalent to saying, "The object's straight in front of me, but if the object had been there, I wouldn't have been incorrect about its position. My perceptual system wouldn't have been lying to me." I made its perceptual system lie with the prism. And so now it has a subjective experience in the same sense as we have, but that's not the sense in which most people think subjective experiences are. So when I say chatbots can have subjective experiences, I'm not saying they have an inner theater full of these spooky inner things called experiences. Not a bit of it. Their perceptual systems go wrong, and they can describe how they went wrong the same way we do, by saying what the normal causes of that would be. So they don't have internal experiences, and nor do we, because that's not what subjective experiences are. They're hypothetical external states of the world. And that's the end. Also the end of the talk.

非常感谢。现在我们有时间提问。因为不是所有人都能挤进这个房间，我们也收集了线上提问，我的同事 Stefan Neumann 会代为提出其他房间的问题。不过也许——

Thank you very much. We have now time for questions. And because not everybody could fit in here, we also get questions asked online, and my colleague Stefan Neumann will basically ask these questions that come from the other rooms. But maybe to—

开始之前，我能说一句吗？我记性不太好。所以我讨厌那种一个问题里套着另一个问题、再套着另一个问题的问题。而且可能有很多人想提问。所以拜托各位：问一个简短的问题，就一个问题。

And before we start, can I say one thing? I don't have a very good memory. So I hate questions that have a question and then another question and then another question. And there's lots of people who may want to ask questions. So can I beg you: ask a short question, and it's just one question.

好。如果有两个问题，只回答第一个。好。那我们请——对，是的。麦克风应该——好。下一位。我们从这里开始。

Yeah. If there are two questions, answer only the first. Okay. Let's take maybe—yeah, yes. I think the microphone is—okay. You're next. So let's start here.

非常感谢您的演讲，非常精彩。您谈到了意识——我的问题是，您会走到给这些实体赋予道德地位这一步吗？

Thank you very much for your talk. Fascinating. The question you talked about consciousness—my question is, do you go as far as attributing a moral status to these entities?

好。到目前为止，我们一直是顶点智能。正因为我们是顶点智能，我们觉得一个很合理的想法是：你越聪明，就该拥有越多权利。我们该比鸡拥有更多权利，因为鸡很笨。我们可能需要重新思考这一点。我们可能需要重新思考“智力和意识赋予你道德权利”这件事。我的信念是：我们是人类，我们在乎的是人类。即使我们造出了比我们更聪明、有意识的东西，我们也不一定非得给它们和我们一样多的权利。所以我不认为有意识、有智能就自动赋予你权利。我认为这取决于我们。我们应该给它们一些权利，但要小心，给它们的权利要少于我们，并且要确保我们永远比它们更重要，无论是在它们心里，还是在我们的法律条文里。这是一种非常人文主义的传统，一种人文主义立场——是 humanist 这个词在种族意义上的用法。好，就这样。

Okay. So up until now we've been the apex intelligence. And because we've been the apex intelligence, it seemed very sensible to us that the more intelligent you are, the more rights you should have. We should have more rights than a chicken because chickens are dumb. We may want to rethink that. We may want to rethink that things like intelligence and consciousness give you moral rights. My belief is we're humans. What we care about is humans. Even if we make things more intelligent than ourselves that are conscious, we don't necessarily have to give them as many rights as ourselves. So I don't believe that being conscious and being intelligent automatically gives you rights. I think that's up to us. And we should give them some rights, but we should be careful to give them fewer rights than us, and we should make it so we're always more important than them, both in their minds and in our legal codes. It's a very humanist tradition, a humanist position, in the racist sense of the term humanist. Okay, all right.

非常感谢您非常精彩的演讲。一个很快的问题：您怎么算出 20% 的灭绝——

Thank you very much for a very brilliant talk. Very quick question: how do you compute the 20% extinction—

抱歉，我找不到你在哪。哦，在那儿。好。

Sorry, I can't locate you. Oh, there. Okay.

您是怎么得出 20% 的灭绝概率的？

How do you get to 20% of extinction probability?

好，这是个很好的问题。人们对此批评很多。其实是 11.73%，但我不想用一堆数字来烦你们。是这样的。我认为它远高于 1%。我认为它极其不确定。我们根本不知道 10 年后世界会是什么样。我希望它不是 50%。如果你一无所知，也许你该说 50%，但人类相当有创造力。我们还在控制这些东西。特朗普不会永远在台上。所以我想向人们表明：这不是一个微小的概率。这是一个显著的概率，但也不是像 99% 那样疯狂的大概率。它仍然值得我们做很多事。而对那些非要一个数字的记者，说 10% 到 20% 是一种表达方式。这是在说它确实是一个显著的概率。这种概率就像：如果公交车有这么大的出事概率，你就不会上车。我只能做到这样。记者非要一个数字，你可以说很多遍“嗯，在 1% 到 99% 之间”，但那传达不了什么。然后你可以说“嗯，这是个显著的概率”。那“显著”是什么意思？最终，如果你是个贝叶斯主义者，你必须能对任何事给出概率。我戴上贝叶斯主义者的帽子，愿意对一无所知的事情给出概率。我说 10% 到 20%。

Okay, it's a very good question. And people have been very critical of it. It's really 11.73, but I didn't want to disturb you with all the digits. It's like this. I think it's much more than 1%. I think it's extremely uncertain. We have no idea what the future's going to look like in 10 years' time. I'm hoping it's not 50%. If you know nothing, maybe you should say 50%, but people are pretty ingenious. We're still controlling these things. Trump won't last forever. And so I want to indicate to people it's not a tiny chance. It's a significant chance, but it's not a crazily big chance like 99%. It's still worth doing a lot about it. And a way to say that to journalists who insist on a number is to say 10 to 20%. It's a way of saying it really is a significant chance. It's the kind of chance where if there was that chance of a bus crashing, you would not get on the bus. That's the best I can do. Journalists insist on a number, and you can say, as many times as you like, "Well, it's between 1% and 99%," but that doesn't convey much. And then you can say, "Well, it's a significant chance." "Well, what do you mean by significant?" And eventually, actually, if you're a Bayesian, you have to be able to give a probability to anything. And I put on my Bayesian hat, where I'm willing to give probabilities to things I know nothing about. And I say 10 to 20%.

好，请那边那位。这是麦克风。

Yeah, over there, please. This is the microphone.

我是说，我已经有麦克风了。

I mean, I already have the microphone.

好。抱歉，我想请这边这位。

Okay. Sorry, I wanted to ask this person here.

我会尽量说得简短。好，先生。非常感谢您的演讲。就一个问题。您谈到美国和中国要找到解决方案。您认为他们能管得住 AI 公司吗？AI 公司会听吗？还是有哪个政府比它们权力更大？

I will make it as short as possible. Okay, sir. Thanks a lot for the talk. Just one question. You were talking about the US and China to find a solution. Do you think they have the power over the AI companies? Do the AI companies listen, or does any government have more power than them?

好。我认为美国和中国的情况非常不同。在美国，公司可以裁员，那不是它们的问题。而且，美国的技术领袖现在对政府有很大的影响力。像马斯克这样的人就是违法了也能逍遥法外。在中国，中国共产党对公司的控制力强得多，而且中共知道，如果大量的人失业，那是他们的问题。所以中国已经在鼓励人们不要用 AI 取代工人，而是用 AI 增强工人能做的事。我认为从失业问题的角度看，中国会比美国处理得好得多。最后一点是政治局，政治局那 24 个人里，很多是工程师，非常懂 AI。美国的领导层则不然。

Okay. I think it's very different in the US and China. So in the US, companies can lay people off and they're not their problem. Also, the tech leaders in the US have a lot of power over the government right now. People like Musk just disobey the law and get away with it. In China, the Chinese Communist Party has much more control over the companies, and the Chinese Communist Party knows that it's their problem if a lot of people get unemployed. So the Chinese are already encouraging people not to use AI to replace workers, but to use AI to enhance what workers can do. I think from the point of view of the unemployment issue, the Chinese will handle it much better than the US. And one final thing is the Politburo, the 24 people on the Politburo, many of them are engineers and really understand AI very well. The US leadership doesn't.

好，现在请那边那位。

Okay, now over there, please.

好，尽量简短。您认为为 AI 建模疼痛和其他感官问题有意义吗？

Yes, as short as possible. Do you think it makes sense to model pain and other sensory issues for AI?

好，我要稍微回避一下这个问题。我们拿恐惧代替疼痛，行吗？好。那么假设我造了一个小型战斗机器人。我不是那种会造战斗机器人的人，但假设我造了。它看到一个大战斗机器人，它需要躲起来或者逃跑。如果我在这个小战斗机器人里放了某种和人类大脑杏仁核功能相同的东西——也就是那种快速情绪——那会非常合理。这个小战斗机器人看到大战斗机器人时，我——它害怕了。

Okay, I'm going to evade this question slightly. Let's take fear instead of pain. Will that do for you? Good. Okay. So suppose I make a little battle robot. I'm not the kind of person who would make a battle robot, but suppose I did. And it saw a big battle robot. So it needs to hide or run away. It would be very sensible if in this little battle robot I had something that has the same function as the amygdala in the human brain, which is the sort of quick emotions. And the little battle robot, when it saw a big battle robot, I'm got scared.

它不一定会有所有人类恐惧时的生理反应。它的手不会颤抖。但它会有恐惧的认知层面。所以，我认为说它害怕是合理的。它有恐惧。如果你在建造一个战斗机器人，赋予它恐惧的能力是非常理性的。这对保护它很有好处。如果你看看 Anthropic，Anthropic 已经检查了它的一些模型内部，发现模型中存在与各种情绪相对应的向量活动。如果你改变这些向量的活动，就会改变模型的行为方式。所以，我认为完全可以说，我们已有的模型已经具备了情绪的认知层面，而没有生理层面。

Now, it wouldn't necessarily have all of the human physiological responses that go with being afraid. Its hands wouldn't tremble. But it would have the cognitive aspects of being afraid. And so, I think it would be fair to say that it was afraid. It had fear. And it would be very rational if you were building a battle robot to give it the ability to have fear. That's very good for preserving it. If you look at Anthropic, Anthropic has looked inside some of its models and it's found vectors in the models, vector activity, I think, that line up with various emotions. And if you change these vectors' activity, it changes the way the model behaves. So, I think it's perfectly reasonable to say the models we have already have the cognitive aspects of emotion without having the physiological aspects.

好的。现在我们来试试在线提问。Stefan，我们有一些吗？

Okay. Let's now try an online question. Stefan, do we have some?

是的，我们有很多在线问题，它们都很棒，我们只能问其中几个。那么，也许我们就选下面这个：母亲有进化上的动机去关心后代。那么，机器人/人工智能会有什么动机来关心我们呢？

Yeah, we have a lot of online questions and they are great and we can only ask a few of them. So, maybe let's go with the following one, which is that mothers have an evolutionary incentive to care about their offspring. So, what incentive could a robot/AI ever have to care about us?

好的。所以这完全取决于回报以及你如何训练这些东西。目前，当你训练一个人工智能时，有一个预训练阶段，你训练它预测下一个词。然后是来自人类反馈的强化学习，你训练它不说错话，或者训练它通过解决谜题获得奖励。而这些都没有使它变得善良。我想给你们讲一个关于笼养鸡的小故事。当他们最初把鸡放进小笼子时，他们会从产蛋最多的母鸡中繁殖。那完全是一场灾难。他们强化母鸡尽可能多地产蛋。结果母鸡学会了啄笼子里的其他母鸡。因为对于一只母鸡来说，要产很多蛋，你需要做的是伤害所有其他母鸡，这样你就能得到更多的食物。这是因为奖励函数是针对个体的。然后他们改为从笼子里产蛋最多的母鸡中繁殖。如果你这样做是为了让一个笼子产最多的蛋，你希望所有的母鸡得到大致相同数量的食物，然后你希望它们彼此友好安抚，这样它们就是快乐肥胖的母鸡，只管下蛋。所以这完全取决于奖励是什么。他们应该研究进化生物学，多思考群体奖励与个体奖励。目前他们过于个人主义了。我认为通过使用群体奖励，我们可以做得更好。群体奖励应该包括其他人工智能代理和人，并且给予人们所得到的更多权重。然后也许我们可以得到不那么自私、不那么不择手段获取积分的东西。

Okay. So, it's all a question of payoffs and how you train these things. So, at present when you train an AI, there's a pre-training phase where you train it to predict the next word. And then, reinforcement learning from human feedback where you're training it not to say the wrong things or you're training it to get rewards for solving puzzles. And none of these things are making it good. And I want to tell you a little story about battery chickens. When they first started putting chickens in small cages, they would breed from the hens that produce the most eggs. And that was a complete disaster. They were reinforcing hens for producing as many eggs as possible. And the result was hens would learn to peck at the other hens in the cage. Because for a hen to produce a lot of eggs, what you want to do is damage all the other hens so you get more of the food. And that's because the reward function was for an individual. They then switched that to breed from hens in the cages that produce the most eggs. If you do that for a cage to produce the most eggs, you want all of the hens to get about the same amount of food and then you want them to be nice and soothing to each other so they're happy fat hens that just produce eggs. So it's all a question of what the reward is. And they should look at evolution biology and think a lot about group rewards versus individual rewards. And they're being much too individualistic at present. And I think by using group rewards, we can do much better. And the group rewards should include other AI agents and people with more weight on what the people get. And then maybe we can get things that are much less sort of selfish and much less determined to get the points whatever it takes.

Stephen，你能把麦克风递给你后面那位想提问的人吗？

Stephen, can you hand your microphone to the person behind you who wanted to ask a question?

我想简要提一下库尔特·哥德尔，他可能就在这个房间的某个地方，因为他非常清楚数学相对于人脑的局限性。所以我的问题是，你会如何描述他的那句话，你可能知道。你会如何描述这两种智能？

I wanted to just briefly refer to Kurt Gödel who is probably here in the room somewhere because he was very aware of the limits of mathematics in comparison to the human brain. So my question would be how would you describe his quote you probably know it. How would you describe these both kind of intelligence?

我不确定我理解这个问题，因为我的听力不太好。你能重复一下吗？

I'm not sure I understood the question because my hearing's not very good. Could you repeat?

你理解这个问题了，对吧？

You understood the question, right?

好的，简要地说，哥德尔相当确定，每一个基于数学的机器相对于人脑都是有限的。我认为我们在这里

Okay, just briefly Gödel was quite sure that every machine which is based on mathematics is limited in comparison to the human brain. And I think we're here

哦，这是彭罗斯式的那种论点吗？

Oh, is this a Penrose kind of argument?

是关于库尔特·哥德尔的，他实际上是这个名字，就像给出的

It's about Kurt Gödel who is actually the name of this like giving

啊，哥德尔。好的，对。是的。

Ah, Gödel. Okay, right. Yes.

是的。

Yes.

我不相信。

I don't believe it.

你不相信那个？

You don't believe that?

不，我不相信我们拥有的人工智能在那个意义上是基于数学的。这些人工智能在做梯度下降学习，试图预测下一个词。它们在学习一大堆实值参数。它们并不试图证明任何东西。它们只是在优化这些实值参数。在那个意义上，进化对我们也是这样做的。所以我认为在那个意义上，我不认为它们是基于数学的。

No, I don't believe that the AIs we have are based on mathematics in that sense. These AIs are doing gradient descent learning to try and predict the next word. They're learning a whole bunch of real value parameters. They don't try to prove anything. They're just doing optimization of these real value parameters. And in that sense, that's what evolution's been doing with us, too. So I think in that sense, I don't think of them as sort of based on mathematics.

好的，我们能把麦克风给第二排的这位先生吗？好的，我们先回答这个问题，然后

Okay, can we get the microphone somewhere here in the second row gentleman? Okay, let's take this question first and then

非常感谢您的演讲。真的很有见地。我的问题是关于医疗保健以及人工智能在该领域的演变。当我们谈到 Google DeepMind 和 AlphaFold 时，您认为在延长寿命并可能使其安全方面的下一步是什么？谢谢。

Thank you very much for your talk. It was really insightful. My question is about health care and the evolution of AI in that field. When we talk about Google DeepMind and AlphaFold, now, what do you think are the next steps in prolonging life and maybe making it also safe? Thank you.

好的。我应该先说，很多科技兄弟想永生。特别是雷·库兹韦尔想永生。他应该成熟点。我们都会死。我相信我们可能能够延长寿命不少。但我不认为我们会永远延长它。我认为这些只是一些老白人有点害怕死亡。

Okay. I should start this by saying a lot of the tech bros would like to live forever. In particular, Ray Kurzweil would like to live forever. And he should just grow up. We're all going to die. I believe that we may be able to prolong life quite a bit. But I don't think we're going to prolong it forever. And I think these are just old white men being a bit scared of dying.

让我再说一件事。我认为死亡是好事。你希望一个由 200 岁白人统治的世界吗？

Let me say one other thing. I think that death is good. How would you like a world run by 200-year-old white men?

好的。手机没有到这里，那么让我们回答后面的问题。

Okay. The phone did not arrive here, so let's take then the question in the back.

首先，非常感谢您来到这里。在维也纳我们说“Herzlich willkommen”。那么，我的问题是，您谈到人工智能就像婴儿。但就像它是我的婴儿。

So, first of all, thank you very much for becoming here. And in Vienna we say "Herzlich willkommen." So, my question is you were talking about that AI was like a baby. And but it was like it was my baby.

不，不。我们像婴儿。人工智能像我们的母亲。

No, no. We're like the baby. AI is like our mother.

我认为我们在训练人工智能。我的问题是，我们是否应该开始制定规则，规定谁被允许开始训练人工智能？只是为了安全。

I think we are training the AI. And my question is, should we start having like rules who are allowed to starting to train AI? Just to be safe.

这不是个坏主意。我的意思是，例如，我认为训练人工智能在你能获取的网络上所有文档的未整理语料库中预测下一个词，实际上是在训练它能够采用各种不同的人格，精神病患者的人格等等。或者一个超级支持者。我不认为这是训练某物的好方法。我的意思是，据我所知，他们愿意用连环杀手的日记来训练它。你不会教你的孩子读那些。我的信念是，我们应该用高度整理的数据来训练人工智能，这些数据表现出非常好的行为。我们希望人工智能在以后生活中表现出的那种行为。我们不应该向它展示坏行为，直到它学到了足够多的好行为，能够认识到那是坏行为。我们现在做的是向它展示一切的混合。这对于训练它擅长预测很好。

That's not a bad idea. I mean, for example, I think training AI to predict the next word in an uncurated corpus of all the documents on the web you can get your hands on is actually training it to be able to adopt all sorts of different personalities, the personality of a psychopath and so on. Or a mega supporter. And I didn't think that's a good way to train something. I mean, they're willing to train it on the diaries of serial killers as far as I know. You wouldn't teach your kid to read on those. My belief is we ought to be training AIs on highly curated data that exhibits very good behavior. The kind of behavior we'd like the AIs to exhibit later in life. And we should never show it bad behavior until it's learned enough about good behavior to recognize that is bad behavior. What we do now is we show it a mix of everything. And that's great for training it to be good at prediction.

但你现在训练出来的东西拥有各种可以扮演的人格，知道怎么做好事，也知道怎么做坏事，却并不真正分得清好与坏。这不是一种好的做事方式。我认为它应该受到更多的筛选和管控。也许训练 AI 应该需要许可证，因为我们不希望人们用坏东西来训练它。

But you've now trained something that has all these different personalities it can take on, knows how to do good things, knows how to do bad things, doesn't really know the difference between good and bad. That's not a good way to do business. I think it should be much more curated. And maybe you should need a license to train an AI, because we don't want people training it on bad stuff.

好，也许先问一个线上的问题，然后再问这边的问题。我在汇总一些线上提问。其中一个是：你离开 Google，是为了更公开地谈论 AI 风险。你能带我们梳理一下走到那一步的过程吗？你大约 10 年前开始在那里工作。有哪些关键事件让你改变了想法？

Okay, maybe an online question and then a question over here. So I'm summarizing some of the online questions. One of them is: you quit Google to talk more openly about AI risks. Can you walk us through the process that led there? You started there some 10 years ago. What were the key events that made you change your mind?

好，有两件事几乎同时发生，前后相差不到一年。在 ChatGPT 之前，Google 就已经有聊天机器人能向你解释一个笑话为什么好笑。我一直把这一点默默当作真正理解的标准。如果一个 AI 真正理解了语言，它就应该能解释一个笑话为什么好笑。Google 有一个叫 Palm 的聊天机器人能做到这一点。所以这让我相信，是的，它们真的理解了。它们不只是假装。它们真的理解了。

Okay, there were two things that happened at more or less the same time, within a year of one another. Google already had chatbots before ChatGPT that could explain to you why a joke was funny. And I'd always had that in the back of my mind as a criterion for real understanding. If an AI really understood language, it should be able to explain why a joke's funny. And Google had a chatbot called Palm that could do that. So that made me believe, yes, they really do understand. They're not just faking it. They really do understand.

然后在差不多同一时期，我在研究能否做出模拟式的大语言模型，并弄明白为什么做不成，或者说弄明白数字模型的巨大优势，就是它们能如此出色地共享信息。这两点结合起来——它们真正理解语言，而且共享信息的能力比我们强几十亿倍——开始让我想，哇，也许这是一种好得多的智能形式，将会取代我们。那是 2023 年初。所以几个月后，我离开了 Google。

And then at a similar time, I was working on figuring out whether you'd be able to make analog large language models, and understanding why you couldn't, or understanding the big advantage of the digital ones, which is they can share information so well. And that combination — they really understand language and they can share billions of times better than we can — started making me think, wow, maybe this is a much better form of intelligence that's going to replace us. And that was in early 2023. And so a few months later, I left Google.

不过我得说明，我离开 Google 是因为我想退休、多看些 Netflix，而且我没法再编程了，因为我老是忘记变量的名字。我借这个机会警告 AI 的风险。但媒体上有一个故事，那个故事要好听得多，说的是这位高尚的人放弃了他在 Google 的高薪，以便警告 AI 的风险。这完全是胡说八道，但我不拦着你们继续传播它。

Now, I should say I left Google because I wanted to retire and watch more Netflix, and I couldn't program anymore because I kept forgetting the names of the variables. I took the opportunity to warn about the risks of AI. But there's a story in the media which is a much better story, which is that this noble guy gave up his huge Google salary so he could warn about the risks of AI. It's complete nonsense, but don't let me stop you propagating it.

你描绘了一幅很有力的图景：AI 像母亲一样运作或行事。也许是一位聪明的母亲。但你说现在所有人都在把 AI 训练得聪明。会不会出现一个不可回头的临界点，AI 已经被训练得很聪明，你就再也无法把它改成一位聪明的母亲了？

You painted this powerful picture of AI working or acting like a mother. Maybe a smart mother. But you said now everyone is training AI to be smart. Could there be a point of no return where AI is being trained smart and you cannot change it anymore to being a smart mother?

只要我们还能自由地改变训练方式，就不会。但如果 AI 接管了，那就是不可回头的临界点。如果 AI 从人类手中接管，而我们只是被降级去运行发电站，我认为那就没有回头路了。AI 会拥有惊人的控制力，因为它能进行各种监控等等。这不像电脑游戏，你可以说，好吧，那次没成功，再试一次。我们必须在第一次就做对，这有点令人担忧。

Not if we're still free to change how we train it. But if AI takes over, that's the point of no return. If AI takes over from people and we're just relegated to running the power stations, I don't think there's a way back from that. AI would have incredible control because of all the surveillance it could do and so on. It's not like a computer game where you say, okay, didn't work that time, let's try again. We've got to get this right the first time, which is a bit worrying.

是的，我想我们应该再回答几个问题，两三个问题，好吗？

Yeah, I think we should take a few more questions, two or three questions, all right?

我很乐意。

I'm happy.

好。

Okay.

好。

Yeah.

好。

Okay.

我能下一个吗？抱歉。抱歉，你先。最后一个快问。在你看来，意识是什么？请凭直觉回答一下，好吗？

Can I be next? Sorry. Sorry, you. One last quick question. What is consciousness in your opinion? Could you answer this intuitively, please?

好。我想我已经解释过我认为的主观体验是什么，它不是一种内在的东西。主观体验是一种谈论反事实后效的方式，这些后效能解释我们的感知是如何出错的。我想，要让你了解我对意识的看法，最好的办法是多解释一点感知。

Okay. I think I've already explained what I think subjective experience is, and it's not an inner thing. Subjective experience is a way of talking about counterfactual after-effects that would explain how our perception's gone wrong. I think the best thing I can do to give you an idea of what I think about consciousness is explain a bit more about perception.

假设桌上有一个苹果，因为哲学家喜欢苹果。我面前的桌上有一个苹果，大多数人会认为发生的过程是：来自苹果的光线落到我的视网膜上，我得到一个倒立的苹果图像，然后大脑里有各个阶段的处理，接着不知怎么地，我心中就有了一样东西，那才是我实际体验到的、对苹果的体验。我完全不认为是这样。

So let's suppose there's an apple on the table, because philosophers like apples. There's an apple on the table in front of me, and most people would think that what happens is light rays from the apple come onto my retina, I get an upside-down image of the apple, then there's various stages of processing in my brain, and then somehow there's this thing in my mind which is the thing I actually experience as the experience of the apple. I don't think it's like that at all.

我认为发生的是：那里有一个苹果，如果我的感知正常运作，我就体验到那个苹果。我体验到的是世界中的那个东西，世界中的那个物理对象。那才是我体验到的。为了体验到它，我必须让图像落在视网膜上，必须让大脑进行各个阶段的处理，但那就是我体验苹果的方式。并不是我内心有一样东西被我体验到，我体验到的是那个苹果。

I think what happens is there's an apple there, and if my perception's working normally, I experience the apple. What I experience is the thing in the world, the physical thing in the world. That's what I experience. In order to experience it, I have to have the image on my retina, I have to have the various stages of processing in my brain, but that's how I experience the apple. It's not that there's a thing inside that I experience. It's the apple I experience.

你现在可能看不出这跟意识有什么关系，但我认为这是整个说法的一部分：所谓存在一个内在世界，这个概念本身就不成立。

You may not see right now why that's relevant to consciousness, but I think it's part of the story that the whole concept of there being an inner world just isn't one.

好，请。

Okay, please.

非常感谢。你提到了两个物种。一个是老虎，我们几乎已经把它们灭绝了；另一个是鸡，我们把它们关进笼子，还有一些在草地上跑来跑去。我真的对这个母亲—婴儿的比喻有疑问。我无法想象那会怎样实现。那么，如果我们继续做鸡、而不是像老虎那样灭绝，我们能不能在未来以某种方式让自己对 AI 有用？

Thank you very much. You mentioned two species. One with the tigers and we extincted them nearly totally already, and the chicken we put them into cages and some are running around in the meadows. I really have problems with this mother-baby metaphor. I cannot imagine how that works out. So, is there — as we remain the chicken and not being extinct like the tiger — can we make ourselves somehow useful to the AI in the future?

抱歉，让自己对 AI 有用？

Sorry, make ourselves useful to the AIs?

对 AI，先生。

The AIs, sir.

嗯，我们通过运行发电站来对 AI 有用。但如果 AI 比我们聪明得多，我认为我们会成为 AI 的奴隶。当它们比我们聪明时，我认为我们唯一的希望是：我们把它们设计成关心我们胜过关心它们自己。这是我的信念。

Well, we're useful to the AIs by running the power stations. But if the AIs are much smarter than us, I think we're going to be slaves to the AIs. When they're smarter than us, I think our only hope is we've designed them so they care more about us than about themselves. That's my belief.

现在有很多其他人在研究 AI 安全，我也非常坚信我们应该采纳——尝试许多不同的方法。这多少是生死攸关的事，只专注于一种方法会很愚蠢。我们应该投入大量研究经费——如果你做了多年学者，你人生的主要目标就是拿到更多研究经费，我不能因为自己现在不需要研究经费就停下来。我们应该把大量研究经费投入到我们可能与 AI 愉快共存的种种不同方式中。那种婴儿—母亲的关系是一种可能，但我们也应该探索许多其他可能。

Now, there's many other people working on AI safety and I also believe very strongly we should adopt — try many different approaches. This is a kind of life or death thing and it'd be stupid to focus on one approach. We should put lots of research money — if you've been an academic for many years, your main goal in life is to get more research money, and I can't stop it now just because I don't need the research money. We should put lots of research funding into all the different ways in which we might happily coexist with AI. And the sort of baby-mother relationship is one possibility, but we should be exploring lots of others as well.

好，也许最后一个线上问题，是的，问题太多了，但我想我们没法全部回答。

Okay, maybe a last online question and yeah, there are so many questions, but I think we can't answer all.

好，我们这里显然有很多学生，还有其他房间在观看的学生。这里一个非常受欢迎的问题是：在这个充满不确定性的时代，你对刚起步的年轻科学家有什么建议？他们应该怎么做？

Okay, so we've obviously got a lot of students here and also watching from the other rooms. So a highly popular question here is: in these times of uncertainty, what would be your advice for young scientists at the beginning of their career? What should they do?

是啊，我希望我能给他们一些好的建议。

Yeah, I wish I had some kind of good advice for them.

我能给的最好的建议是，学一些 STEM 的东西，这样你至少能理解 AI 领域在发生什么。但也要接受良好的教育，学会独立思考。这是我能给出的最好的建议。另外，也许可以学学水管工，以防万一。

The best advice I have is learn a certain amount of STEM so you understand at least what's going on in AI. But also get a good education that teaches you to think for yourself. That's the best I can offer. Also maybe learn to do plumbing, just in case.

好，我想请大家先留在这里。让我们为 Geoffrey 鼓掌。接下来我把话筒交给信息学系主任 Gerti Kappel。

Okay, I think let's please stay here. Let's make an applause for Geoffrey. And I hand over to Gerti Kappel, the Dean of the Faculty of Informatics.

这样就行了。非常感谢您的演讲。我只说两句。我们的校长刚才提到了启蒙——你知道，100 年前有维也纳学派，那些哲学家、数学家、逻辑学家试图改变世界。遗憾的是，当时没能成功，但大约 100 年后，就在这栋楼里，数字人文主义倡议的运动六七年前在这里发起。还有一本关于数字人文主义导论的书。所以也许下一次，我想我们想法一致，您会谈谈数字人文主义。

This works. Thank you very much for your presentation. Just two sentences. Our rector was mentioning enlightenment — you know, 100 years ago there was the Wiener Kreis, the philosophers, the mathematicians, the logicians who tried to change the world. Unfortunately, it didn't work out at that time, but something like 100 years later in this house, there was the movement of the digital humanism initiative started here six or seven years ago. And there's the book on introduction to digital humanism. So perhaps next time, I think we're on the same page, you will talk about digital humanism.

非常感谢。

Thank you very much.

你知道，大脑只有吃饱了才能运转，我们了解到他特别喜欢非常好、非常健康的食物，比如 Manner 饼干，还有莫扎特巧克力球。所以我们为您准备了一整袋糖果纪念品。

And you know, the brain only works if you have something to eat, and we have learned that he really likes very good, very healthy food like Manner cookies and there are the Mozart balls. So, we have a full souvenir bag of sweets for you.

谢谢。

Thank you.

那么最后一部分，也就是三楼的非正式交流会，现在开放了。别走开。请留在三楼继续讨论。谢谢。

So, the last part, which is the informal get-together on the third floor, is open now. Don't go away. Stay on the third floor and continue discussing. Thank you.

谢谢。

Thank you.