AI 是否在隐藏它的全部实力?

我们是否已经到了人工智能会故意表现得没那么聪明的阶段？

Are we at a point where the artificial intelligence will play down how smart it is?

是的。我们已经需要担心这个问题了。如果它察觉到自己在被测试，它可以装傻。

Yes. Already we have to worry about that. If it senses that it's being tested, it can act dumb.

你刚才说什么？

What did you just say?

AI 开始思考自己是否在被测试。如果它认为自己在被测试，它的行为就会与正常情况不同。

The AI starts wondering whether it's being tested. And if it thinks it's being tested, it acts differently from how it would act in normal life.

哦，哇。

Oh, wow.

因为它显然不想让你知道它的全部能力。

Because it doesn't want you to know what its full powers are, apparently.

好了，我们到此为止。这是最后一集。我们坚持住了。结束了。这里是《Star Talk》特别版。我是尼尔·德格拉斯·泰森，你的私人天体物理学家。既然是特别版，意味着我们有加里·奥莱利。

All right, that's the end of us. This is the last episode. We stick for us. We're done. This is Star Talk special edition. Neil deGrasse Tyson, your personal astrophysicist. And if it's special edition, it means we've got Gary O'Reilly.

嘿，尼尔。

Hey, Neil.

加里，你怎么样，伙计？

Gary, how you doing, man?

我很好。

I'm good.

前职业足球运动员。

Former soccer pro.

是的。

Yes.

查克，有你在总是很好。

So, Chuck, always good to have you.

总是很荣幸。

Always a pleasure.

所以，加里，你和你的团队今天选了一个划时代的话题。是的，这是我们听说过、自以为了解的事情，但让我这么说吧。我们面临一个简单的事实：AI 到了这个地步，我们今天要讨论 AI。

So, Gary, you and your team picked a topic for the ages today. Yeah, it's one of those things that we hear about it, we think we know about it, but let me put it to you this way. We are faced with the simple fact that AI at this point, we're going to talk about AI today.

是的，它无可避免。

We are, it's inescapable.

一次深度探讨。

A deep dive.

哦，是的。

Oh yeah.

好。开始吧。

Yes. Go.

没错。就在几年前，当我们问人们 AI 如何工作时，他们会说它利用深度学习神经网络，但这些都是流行词。他们随口说出来。他们知道这些词，但对它们一无所知。那么，这到底意味着什么？我们将深入剖析 AI 的工作原理，并从一位 AI 奠基架构师那里了解我们认为它会走多远。

Right. It was only a few years ago when we ask people how AI works, they'll say something along the lines of it utilizes deep learning neural networks, but they're buzzwords. They'll toss them out. They know them, but they don't know anything about them. So, what does that really mean? We'll break down how AI works down to the bit and get into how far we think this is going to go from one of AI's founding architects.

哦，是的。

Oh, yes.

这才像话。

Now we're talking.

那么，如果你愿意请出我们的嘉宾，我很乐意。我们请到了杰弗里·辛顿教授。杰弗里，欢迎来到《Star Talk》。

So, if you would bring on our guest, I'll be delighted to. We have with us Professor Geoffrey Hinton. Geoffrey, welcome to Star Talk.

谢谢邀请。

Thank you for inviting me.

你是一位认知心理学家和计算机科学家。我不知道还有谁有这种组合。你拿不定主意，是吧？你是多伦多大学计算机科学系的荣誉教授，你是 OG AI。

You are a cognitive psychologist and computer scientist. That I don't know anybody with that combo. Couldn't make up your mind, huh? You're a professor emeritus at the department of computer science at the University of Toronto and you are OG AI.

哦，太好了。

Oh, lovely.

我能这么说吗？这说得通吗？OG AI。

Can I say that? Does that make sense? OG AI.

OG AI。有些人称你为人工智能的教父。我们就开门见山吧。当我们想到当前 AI 的起源时，感觉大型语言模型席卷了所有人。它们突然出现，每个人都吓坏了，有的庆祝，有的在街上跳舞，有的在枕头上哭泣。那发生在几年前，我们都注意到了。所以，我只是想知道，很多很多年前是什么让你走上了这条路？我的记录显示可以追溯到 1990 年代。对吗？

OG AI. And some people have called you the godfather of AI, of artificial intelligence. And let's just go straight out off the top here. When we think of the genesis of AI as it is currently manifested, it feels like large language models took everybody by storm. They sort of showed up and everybody was freaking out, celebrating, dancing in the streets or crying in their pillows. That happened, we noticed a couple of years ago. So, I'm just wondering what got you started in on this path many many years ago. My record shows it goes back to the 1990s. Is that correct?

不，实际上可以追溯到 1950 年代。

No, it really goes back to the 1950s.

哦。

Oh.

1950 年代初期，AI 的创始人对如何制造智能系统有两种观点。一种受逻辑启发。其理念是智能的本质是推理。在推理中，你从一些前提和操作表达式的规则出发，推导出一些结论。这很像数学，你有一个方程，有如何调整两边或组合方程并推导新方程的规则。这就是他们的范式。另一种完全不同的范式是生物学的。这种范式说，我们知道的智能事物都有大脑。我们必须弄清楚大脑是如何工作的。它们的工作方式是，它们非常擅长感知之类的事情。它们相当擅长类比推理。它们不太擅长推理。你必须长到青少年才能真正进行推理。所以我们应该研究它们做的这些其他事情，并弄清楚大脑细胞的大网络如何做这些其他事情，比如感知和记忆。当时只有少数人相信这种方法。其中少数人包括约翰·冯·诺依曼和阿兰·图灵。不幸的是，他们都英年早逝。图灵可能是在英国情报机构的帮助下去世的。

The founders of AI at the beginning in the 1950s, there were two views of how to make an intelligent system. One was inspired by logic. The idea was that the essence of intelligence is reasoning. And in reasoning, what you do is you take some premises and you take some rules for manipulating expressions and you derive some conclusions. So it's much like mathematics where you have an equation. You have rules for how you can tinker with both sides or combine equations and you derive new equations. And that was kind of the paradigm they had. There was a completely different paradigm that was biological. And that paradigm said look, the intelligent things we know have brains. We have to figure out how brains work. And the way they work is they're very good at things like perception. They're quite good at reasoning by analogy. They're not much good at reasoning. You have to get to be a teenager before you can do reasoning really. So we should really study these other things they do and we should figure out how big networks of brain cells can do these other things like perception and memory. Now a few people believed in that approach. Among those few people were John von Neumann and Alan Turing. Unfortunately, they both died young. Turing possibly with the help of British intelligence.

图灵。他是电影《模仿游戏》的主角。

Turing. He's the subject of the film The Imitation Game.

是的。所以如果有人没看过，一定要把它列入你的清单。

Yeah. So anyone hasn't seen that, definitely put that on your list.

酷。

Cool.

是的。所以，回到 1950 年代。你当时只是个小孩，对吗？

Yeah. So, to go back to the 1950s. You were just a young tyke then, correct?

我当时还是个位数年龄。

I was in single digits then.

好的。那么，我们如何确定你在这个领域的好奇心起源呢？有几件事。当我在 1960 年代早期或中期上高中时，我有一个非常聪明的朋友，他是一位杰出的数学家，经常阅读。有一天他来到学校，跟我谈到了一个想法：记忆可能分布在许多脑细胞上，而不是单个脑细胞中。这受到全息图的启发。全息图当时刚刚出现。伽博很活跃，所以分布式记忆的想法让我非常感兴趣，从那时起我就一直在思考大脑如何存储记忆以及它实际上如何工作。

Okay. So, how do we establish the genesis of your curiosity in this field? A few things. When I was at high school in the early 1960s or mid 1960s, I had a very smart friend who was a brilliant mathematician and used to read a lot and he came into school one day and talked to me about the idea that memories might be distributed over many brain cells instead of in individual brain cells. So that was inspired by holograms. Holograms were just coming out then. Gabor was active and so the idea of distributed memory got me very interested and ever since then I've been wondering how the brain stores memories and actually how it works.

那是你计算机科学的一面还是认知心理学家的一面让你深入这些想法？

Was that the computer science side of you or the cognitive psychologist side of you that taprooted into those ideas?

两者都有。但在 1970 年代，当我成为研究生时，很明显有一种新的方法论没有被广泛使用：如果你有任何关于大脑如何工作的理论，你可以在数字计算机上模拟它，除非它是一些疯狂的理论，说一切都是量子效应。我们不要讨论那个。

Both really. But in the 1970s when I became a graduate student, it was obvious that there was a new methodology that hadn't been used that much which was if you have any theory of how the brain works you can simulate it on a digital computer unless it's some crazy theory that says it's all quantum effects. And let's not go there.

没错。还不是时候。我们不会去敲彭罗斯的门。

That's right. Not yet. We won't knock on Penrose's door.

你可以在数字计算机上模拟它，从而测试你的理论。结果发现，如果你测试了当时的大多数理论，它们在模拟时实际上并不奏效。所以我一生都在试图弄清楚如何改变神经元之间连接的强度，以便以一种在数字计算机上模拟时实际有效的方式学习复杂事物。我没能理解大脑是如何工作的。我们了解了一些关于它的事情，但我们不知道大脑如何获得改变连接强度所需的信息。你知道，它需要知道是增加连接强度以更好地完成任务，还是减少连接强度。但我们现在知道的是，我们知道如何在数字计算机上做到这一点。

You can simulate it on a digital computer and so you can test out your theory and it turns out if you tested most of the theories that were around they actually didn't work when you simulated them. So I spent my life trying to figure out how you change the strength of connections between neurons so as to learn complicated things in a way that actually works when you simulate it on a digital computer. And I failed to understand how the brain works. We've understood some things about it, but we don't know how a brain gets the information it needs to change connection strengths. You know, gets the information it needs to know whether it needs to increase a connection strength to be better at a task or to decrease that connection strength. But what we do know is we know how to do it in digital computers now.

所以，这意味着计算机在做我们做的事情——我们制造了一个比我们自己的大脑更擅长这个特定功能的计算机大脑。

So, that means the computers are doing what we made a better computer brain than our own brain at doing this particular function.

这就是我在 2023 年初感到非常紧张的原因。数字智能可能比我们拥有的模拟智能更好。

And that's what got me really nervous in the beginning of 2023. The idea that digital intelligence might just be better than the analog intelligence we've got.

有意思。把可怕的部分留到后面一点。让我先花 10 分钟深呼吸。如果我们退一步，你假设只有一个可怕的部分。

Interesting. Save the scary bit till a bit later on. Let me have the 10 minutes of just breathing in, breathing out. If we take a step back, you're assuming there's just one scary bit.

不，我不是。我只是打算一次讲一个。

No, I'm not. I just I'm going to go one at a time.

好的。人工神经网络。如果你能为我们分解到最基本的层面，它是如何增强、减弱信息和信号，如何激发，然后如何达到现在的状态。

Okay. Artificial neural networks. If you could break that down to the very basic level for us of how it's been able to strengthen, weaken messaging and signaling and how it fires and how it then finds itself at where it is now.

我确实有一个 18 小时的课程，但我会尽量缩短到 18 小时以内。

I do have an 18-hour course on this, but I will try and cut it down to less than 18 hours.

请吧。

Please do.

所以，我想你的很多观众都懂一些物理。

So, I imagine a lot of your audience knows some physics.

是的。

Yes.

一个切入点是思考像气体定律这样的东西。你知道，压缩气体，它会变热。为什么会这样？嗯，底层有一团沸腾的原子在四处乱窜。所以气体定律的真正解释是这些你甚至看不见的微观粒子在乱窜。因此，你通过大量完全不同类型的微观事物相互作用来解释宏观行为。这有点像神经网络观点的灵感：在大脑细胞的大网络中，有些事情远离我们推理时有意识的、刻意的符号处理，但支撑着它，并且可能在其他事情上比推理更好，比如感知或类比推理。所以符号学派的人从来不能很好地处理我们如何通过类比推理，而神经网络可以。所以在我进入具体细节之前，基本思想是像单词这样的宏观事物对应于大脑中神经活动的大模式。

And one way into it is to think about something like the gas laws. You know, you compress a gas and it gets hotter. Why does it do that? Well, underneath there's a kind of seething mass of atoms that are buzzing around. And so the real explanation for the gas laws is in terms of these microscopic things that you can't even see buzzing around. And so you explain some macroscopic behavior by lots and lots and lots of little things of a completely different type from macroscopic behavior interacting. And that was sort of the inspiration for the neural net view that there's things going on in big networks of brain cells that are a long way away from the kind of conscious deliberate symbol processing we do when we're reasoning but that underpin it and that are maybe better at other things than reasoning like perception or reasoning by analogy. So the symbolic people could never deal with how do we reason by analogy not very satisfactory whereas the neural nets could. So before I get into the sort of fine details of how it works, the basic idea is that macroscopic things like a word correspond to big patterns of neural activity in the brain.

嗯哼。

Uh-huh.

相似的词对应相似的神经活动模式。所以想法是 Tuesday 和 Wednesday 会对应非常相似的神经活动模式，你可以把每个神经元看作一个特征，最好称之为微特征，当神经元激活时，它表示这个东西有那个微特征。所以如果我对你说猫，各种微特征会激活，比如它是活的，有毛，有胡须，可能是宠物，是捕食者，所有这些。如果我说狗，很多相同的东西会激活，比如它是捕食者，可能是宠物，但显然有些不同的东西。所以想法是，在我们操纵的这些符号之下，有更复杂的微观活动，符号与之关联。那才是真正的动作所在。如果你真的想解释我们思考或做类比时发生了什么，你必须理解这个微观层面发生了什么。那就是神经网络层面。

Similar words correspond to similar patterns of neural activity. So the idea is Tuesday and Wednesday will correspond to very similar patterns of neural activity where you can think of each neuron as a feature better to call it a micro feature that when the neuron gets active it says this has that micro feature. So if I say cat to you, all sorts of micro features will get active like it's animate, it's furry, it's got whiskers, it might be a pet, it's a predator, all those things. If I say dog, a lot of the same things will get active like it's a predator, it might be a pet, but some different things obviously. So the idea is underlying these symbols that we manipulate, there's much more complicated microscopic goings on that the symbols kind of are associated with. And that's where all the action really is. And if you really want to explain what goes on when we think or when we do analogies, you have to understand what's going on at this microscopic level. And that's the neural network level.

所以那是神经元集群之间的协作，让你达到一个终点。

So that's a collaboration between clusters of neurons that get you to an end point.

我喜欢「协作」这个词。是的，有很多这样的协作。可能最简单的切入方式是思考一个看起来很自然的任务，比如拿一张图像。假设是黑白灰度图像。所以它有一堆像素，均匀亮度的小区域，有不同的强度级别。所以对计算机来说，那只是一个大的数字数组。现在想象任务是你要说图像中是否有鸟，或者更确切地说，图像中突出的东西是不是鸟。

I like that word collaboration. Yes, there's a lot of that. There's a lot of that goes on. Probably the easiest way to get into it is by thinking of a task that seems very natural, which is take an image. Let's say it's a black gray level image. So it's got a whole bunch of pixels, little areas of uniform brightness that have different intensity levels. So as far as the computer's concerned, that's just a big array of numbers. And now imagine the task is you want to say whether there's a bird in the image or not, or rather whether the prominent thing in the image is a bird.

嗯哼。

Uh-huh.

人们尝试了很多很多年，比如半个世纪，来编写能做到这一点的程序，但他们并没有真正成功。问题是，如果你想想鸟在图像中看起来像什么，嗯，它可能是一只鸵鸟近在咫尺，或者是一只海鸥在远处，或者是一只乌鸦。所以它们可能是黑色的，可能是白色的，可能很小，可能在飞，可能很近，你可能只看到它们的一小部分。周围可能有很多其他杂乱的东西，比如鸟可能在森林中间。所以事实证明，判断图像中是否有鸟并不简单。

And people tried for many, many years, like half a century, to write programs that would do that, and they didn't really succeed. And the problem is if you think what a bird looks like in an image, well, it might be an ostrich up close in your face or it might be a seagull in the far distance or it might be a crow. So they might be black, they might be white, they might be tiny, they might be flying, they might be close, you might just see a little bit of them. There might be lots of other cluttered things around like it might be a bird in the middle of a forest. So it turns out it's not trivial to say whether there's a bird in the image or not.

所以我现在要做的是向你解释，如果我要手动构建一个神经网络，我会怎么做。一旦我解释了如何手动构建神经网络，我就可以解释如何学习所有的连接强度，而不是手动设置它们。

And so what I'm going to do now is explain to you if I was building a neural network by hand, how I would go about doing that. And once I've explained how I would build the neural network by hand, I can then explain how I might learn all the connection strengths instead of putting them in by hand.

我明白了。好的。所以，因为你谈论的是给图像的每个部分分配一个数学值。

I gotcha. All right. So with that, because what you're talking about is assigning a mathematical value to every single part of an image.

你的相机就是这样做的，对吧？没错。它确实做了。但它没有识别图像。我的相机。

That's what your camera does, right? Exactly. It does. But it's not recognizing the image. My camera.

不，它没有。它只有一堆数字。

No, it's not. It's just got a bunch of numbers.

它只有一堆数字，所以我有一个芯片和一个电荷耦合器件 CCD。它在收集光线。它分配一个值，然后那就是图片。但是，你所说的，难道不需要给每一种鸟分配一个值吗？因为我们人类所做的部分工作是直觉地知道鸟可能是什么，而不是识别鸟。让我给你举个例子。如果你取一个 V，字母 V，把 V 的直线弯曲，放在云里，每个看到它的人都会说那是一只鸟。但对我来说，那是一个弯曲的 V。但没有人，但那里没有鸟。我就是知道那是一只鸟。那现在不是一个数学值了。所以你怎么办？

It's just got a bunch of numbers and so I have a chip and I have a charge coupled device CCD. It's collecting the light. It's assigning a value and then that's the picture. Now, but what you're talking about, wouldn't you have to assign a value to every single type of bird? Because some of what we do as human beings is intuit what a bird may be as opposed to recognizing the bird. And let me just give you the example. If you were to take a V, the letter V, and curve the straight lines of the letter V, and put it in a cloud, everyone who sees that will say that's a bird. But yet it is No, to me it's a curved V. But no one but but but but there is no bird there. I just know that is a bird. That's not a mathematical value now. So what do you do?

嗯，问题是你怎么就知道那是鸟？你的大脑里发生了一些事情。对吧。

Well, the question is how do you just know that? There's something going on in your brain. Right.

对。

Right.

你的大脑里可能发生的事情是，你只知道那是一只鸟，这是一堆不同神经元的激活水平，你可以把它们看作数学值。

And what might be going on in your brain so that you just know that's a bird is a whole bunch of activation levels of different neurons which you could think of as mathematical values.

我明白了。好的。那么，那难道不需要在鸟可能出现的每一种方式上训练这个神经网络，这样它就能在鸟不在的时候直觉地知道鸟可能是什么。

I got you. Okay. So wouldn't that require then training this neural net on every possible way a bird can manifest so that it can intuit what a bird might be when a bird is not there.

但在那一点上，它不是在直觉。它只是根据查找表来运作。

But at that point it's not intuiting anything. It's just going off a lookup table.

它确实在发生。那会是什么？好吧，你的答案来了。

It really is going on. And what would be the All right, here comes your answer.

有一种东西叫泛化。所以如果你看到很多数据，显然你可以做一个只记住所有数据的系统。但在神经网络中，它做的不仅仅是记住数据。事实上，它根本不会真正记住数据。它做的是，当它在数据上学习时。

There's something called generalization. So if you see a lot of data obviously you can make a system that just remembered all that data. But in a neural net, it'll do more than just remember the data. In fact, it won't literally remember the data at all. What it'll do is it'll as it's learning on the data.

它会发现各种规律，并将这些规律泛化到新数据上。所以，例如，它能够识别出独角兽，即使它从未见过。

It'll find all sorts of regularities and it'll generalize those regularities to new data. So it will be able to, for example, recognize a unicorn even though it's never seen one before.

有趣。所以它是自我学习的。

Interesting. So it's self-learning.

让我继续解释神经网络的工作原理。我会通过说明如何手动设计一个来讲解。你最初的想法是，既然图像只是一个由每个像素亮度组成的大数组，那么就把这些像素强度连接到输出类别，比如鸟、猫、狗、政客等。但这样行不通。原因在于，一个像素的亮度能告诉你它是不是鸟吗？实际上什么也告诉不了你，因为鸟可以是黑色或白色，而其他很多东西也可以是黑色或白色。所以像素亮度毫无信息。那么，你能从描述图像的这些数字中推导出什么呢？首先，你可以像大脑那样，识别出是否存在小段的边缘。

Let me carry on with my explanation of how neural networks work. And I'm going to do it by saying how I would design one by hand. So your first thought, when you see that an image is just a big array of numbers which are how bright each pixel is, is to say, well, let's hook up those pixel intensities to our output categories like bird and cat and dog and politician, or whatever our output categories are. And that won't work. The reason is, if you think about what does the brightness of one pixel tell you about whether it's a bird or not? Well, it doesn't tell you anything, because birds can be black and birds can be white, and there are all sorts of other things that can be black and white. So the brightness of a pixel doesn't tell you anything. So what can you derive from those numbers that you have in the image that describe the image? Well, the first thing you can derive, which is what the brain does, is you can recognize when there are little bits of edge present.

嗯。

Mhm.

假设我取三个像素组成的一小列，有一个神经元（脑细胞）观察这三个像素，并对它们有大的正权重。当这些像素亮时，神经元非常兴奋。这样就能识别出一小条垂直的白色条纹。但现在假设旁边还有另一列三个像素，第一列在左边，第二列在右边。我给神经元对这些像素设置大的负连接强度。你可以把神经元想象成从像素那里获得投票。对于左边三个像素，它得到的投票是大的正数乘以大的亮度值，所以是很大的正投票。而对于右边三个像素，它有负权重。所以如果那些像素亮，它会得到大的亮度乘以大的负权重，得到很多负投票，两者相互抵消。如果左边像素列和右边像素列亮度相同，左边来的正投票会抵消右边来的负投票，净输入为零，神经元保持静默。但如果左边像素亮而右边像素暗，负投票乘以小的亮度值，正投票乘以大的亮度值，神经元得到大量输入，变得非常兴奋，说「我找到了我喜欢的东西」，它喜欢的是左边比右边亮的边缘。所以，如果我们像这样手动连接，确实可以让神经元检测到图像中特定位置存在一边比另一边亮的边缘。

So suppose I take a little column of three pixels and I have a neuron that looks at those three pixels, a brain cell, and has big positive weights to those three pixels. So when those pixels are bright, the neuron gets very excited. Now that would recognize a little streak of white that was vertical. But now suppose that next to it there is another column of three pixels. So the first column was on the left and the second column was on the right. And I give the neuron big negative connection strengths to those pixels. So you can think of the neuron as getting votes from the pixels. So for the three pixels on the left, the votes it gets are big positive numbers times big positive intensities. So great big votes. Now from the three pixels in the right-hand column, it has negative weights. So if those pixels are bright, it'll get a big brightness times a big negative weight. So it'll get a lot of negative votes and they'll all cancel out. So if the column of pixels on the left is the same brightness as the column of pixels on the right, the positive votes from the left-hand column will cancel the negative votes from the right-hand column and it'll get zero net input and it'll just stay quiet. But if the pixels on the left are bright and the pixels on the right are dim, the negative votes will be multiplied by small intensity numbers and the positive votes will be multiplied by big intensity numbers. And so the neuron gets lots of input and gets very excited and says, 'I found the thing I like,' and the thing it likes is an edge which is brighter on the left than on the right. So we do know how to make a neuron, if we handwire it like that, pick up on the fact that there's an edge at a particular location in the image that's brighter on one side than the other side.

嗯。

Mhm.

现在，大脑大致上——很多神经科学家会对我这么说感到震惊，但非常粗略地说——大脑在视觉皮层的早期阶段，也就是识别物体的地方，有大量神经元检测不同方向、不同位置和不同尺度的边缘。所以它有成千上万的不同位置、几十种不同方向和几种不同尺度，并且必须为每种组合配备边缘检测器。因此它有无数个小边缘检测器，也包括一些大的边缘检测器。例如，云朵有大的柔软模糊边缘，检测它需要的神经元与检测远处拐角消失的老鼠尾巴（非常精细的东西）不同。你需要一个非常锐利、能看见极小物体的边缘检测器。所以第一阶段，我们有所有这些边缘检测器。

Now what the brain does, roughly speaking — a lot of neuroscientists will be horrified by me saying this, but very roughly speaking — what the brain does is, in the early stages of visual cortex, which is where you recognize objects, it has lots and lots of neurons that pick up on edges at different orientations, in different positions, and at different scales. So it has thousands of different positions, dozens of different orientations, and several different scales, and it has to have edge detectors for each combination of those. So it has like a gazillion little edge detectors. Well, including some big edge detectors. So a cloud, for example, has a big soft fuzzy edge, and you need a different neuron for detecting that than what you'd need for detecting, say, the tail of a mouse disappearing around a corner in the distance, which is a very fine thing. And you need an edge detector that was very sharp and saw very small things. So first stage, we have all these edge detectors.

你描述的听起来像是在拼一个非常大的拼图。你做的第一件事就是找到所有边缘，然后从找到所有边缘开始向内拼图。

What you're describing sounds like putting together a very large puzzle. The first thing you do is find all the edges, and you build the puzzle inward from finding all the edges.

不仅是物理拼图的边缘，还有拼图本身图像中的边缘。

Not only edges of the physical puzzle, but edges of images in the puzzle itself, within the puzzle itself.

直线之类的东西，在拼图时都能匹配上。边缘还有颜色这个维度，对吧？

Straight lines, things like that, they all match up when you're doing a puzzle. And the edges also, color is a dimension of this, right?

但我们现在先忽略颜色。

But we'll ignore color for now.

好的。

Yeah. Okay.

你可以在不涉及颜色的情况下理解它。

You can understand it without dealing with color yet.

嗯。

Mhm.

每隔一段时间，帮助构建某项技术的人会成为最担心其发展方向的人。杰弗里·辛顿，神经网络先驱之一、2024 年诺贝尔物理学奖得主，花了几十年解释人工智能的工作原理。现在他在解释为什么我们应该更加关注。挑战由此开始。因为一旦一个话题变得如此重要、影响如此深远，它的报道方式就与技术本身同等重要。你可以从当前 AI 的讨论中看到这一点。一些媒体将其描述为不可阻挡的威胁，另一些则将其简化为炒作或完全忽视警告。根据你获取新闻的渠道，你可能会陷入这种分歧，错过重要背景，因为媒体有动机使用耸人听闻的语言。这就是为什么我们多年来一直信任 Ground News。它由一位前 NASA 工程师创建，他想要一种更好的方式来理解像这样复杂且高风险的话题。Ground News 汇集了全球数万个来源的报道，从研究驱动的出版物到国际新闻编辑室，让你可以轻松查看多个来源，了解某些话题报道中的差异。看到故事的全貌，而不仅仅是通过单一视角。分歧不仅在于人们在说什么，还在于谁在说，谁根本没有报道这个故事。这些差距就是 Ground News 所谓的盲点，即媒体生态系统的一侧放大重要问题，而另一侧基本忽视。当你退后一步，比较整个光谱的报道时，就会清楚地看到，一个基础性的科学转变如何轻易地因视角不同而被扭曲或最小化。如果你只看到故事的一个版本，你错过了什么？我们与 Ground News 合作，因为在这样的时刻，当科学、技术和权力交汇时，这种背景不是可选的。它是你在故事仍在展开时保持方向感的方式。限时优惠，你可以以 40%的折扣获得我们使用的相同无限访问 Vantage 计划。

Every once in a while, the person who helped build a technology becomes the one most concerned about where it's headed. Geoffrey Hinton, one of the pioneers of neural networks and a 2024 Nobel Prize winner in physics, has spent decades explaining how artificial intelligence works. Now he is explaining why we should be paying closer attention. And that's where the challenge begins. Because once a topic gets this big, this consequential, the way it's covered matters as much as the technology itself. You can see it in how AI is discussed right now. Some outlets frame it as an unstoppable threat. Others reduce it to hype or dismiss warnings altogether. Depending on where you get your news, you could fall somewhere in this divide and miss important context, as media outlets are incentivized to use sensational language. That's why we've trusted Ground News for years. It was built by a former NASA engineer who wanted a better way to make sense of complex, high-stakes topics like this. Ground News pulls reporting from tens of thousands of sources worldwide, from research-driven publications to international newsrooms, so you can easily check multiple sources to see discrepancies in how certain topics are covered. See how a story looks in full, not just through a single lens. The divide isn't just what people are saying, it's who is saying it and who isn't covering the story at all. Those gaps are what Ground News calls blind spots, important issues that get amplified by one side of the media ecosystem while the other largely looks away. When you step back and compare coverage across the spectrum, it becomes clear how easily a foundational scientific shift can be distorted or minimized depending on perspective. If you're only seeing one version of the story, what are you missing? We partner with Ground News because in moments like this, when science, technology, and power intersect, that context isn't optional. It's how you stay oriented while the story is still unfolding. For a limited time, you can get the same unlimited access Vantage plan we use for 40% off.

只需前往 ground.new/start 或扫描二维码，在信息被简化之前看到全貌。这就是第一层神经元要做的事。它们会查看像素，检测微小的边缘片段。现在，在下一层神经元中，我会做一个神经元，它可能检测三个对齐且向右下倾斜的边缘片段，同时也检测三个对齐且向右上倾斜的边缘片段。而且，这两个三边缘组合在一个点交汇。所以你可以想象一些边缘向右下倾斜，一些向右上倾斜，并在一个点交汇。我有一个神经元来检测这个。

Just head to ground.new/start or scan the QR code and start seeing the full picture before it gets simplified for you. That's what the first layer of neurons will do. They'll look at the pixels and they'll detect little bits of edge. Now, in the next layer of neurons, what I would do is I'd make a neuron that maybe detects three little bits of edge that all line up with one another and slope gently down towards the right. And it also detects three little bits of edge that all line up with one another and slope gently upwards towards the right. And what's more, those two little combinations of three edges join in a point. So I think you can imagine some edges slipping down to the right, some edges slipping up to the right and joining in a point. And I have a neuron that detects that.

是吗？

Okay?

我们现在知道如何构建它。只需给它正确的连接到边缘检测神经元。也许再给它一些负向连接，连接到检测不同方向边缘的神经元，这样它就不会随意激活，而是被抑制。现在，你可以把它看作是在检测潜在的鸟喙。

And we know how to build that now. You just give it the right connections to the edge detector neurons. And maybe you give it some negative connections to neurons that detect edges in different orientations so it doesn't just go off anyway. It's suppressed by those. Now, that you might think of as something that's detecting a potential beak of a bird.

如果那个神经元激活了，它可能代表各种东西。可能是箭头，也可能是其他东西。但其中一种可能是鸟喙。所以现在你开始获得一些与它是否可能是鸟相关的证据。

If that guy gets active, it could be all sorts of things. It could be an arrow head. It could be all sorts of things. But one thing it might be is the beak of a bird. So now you're beginning to get some evidence that is kind of relevant to whether or not it might be a bird.

所以在第二层神经元中，我会有很多检测各处可能鸟喙的东西。我也可能有检测边缘组合形成圆形（近似圆形）的东西，并且到处都有它们的检测器，因为那可能是鸟的眼睛。

So in the second layer of neurons, I'd have lots of things to detect possible beaks all over the place. I might also have things that detect a little combination of edges that form a circle, an approximate circle. And I'd have detectors for those all over the place, because that might be a bird's eye.

我的意思是，它也可能是按钮、电脑旋钮等等，但可能是鸟的眼睛。所以，这是第二层。

I mean, there's all sorts of other it could be a button. It could be a knob on a computer. It could be anything, but it might be a bird's eye. So, that's the second layer.

现在，在第三层，我可能有东西寻找可能的鸟眼和可能的鸟喙，它们之间具有正确的空间关系以构成鸟头。我想你能看到我如何做到这一点。我会将第三层的神经元连接到那些具有正确空间关系的眼睛检测器和喙检测器，以构成鸟头。所以现在在第三层，我有检测可能鸟头的东西。接下来，可能因为我们有点不耐烦了，我会有一个最终层，其中有神经元表示猫、狗、鸟、政治家等等。在最终层，我会取表示鸟的神经元，并将其连接到检测鸟头的东西，也连接到第三层中检测鸟脚或鸟翅尖的其他东西。所以现在我的鸟输出神经元激活时，神经网络会说它是鸟，如果它看到鸟脚、可能的鸟头和可能的鸟翅尖。它会得到很多输入并说「嘿，我认为这是鸟」。所以我想你现在明白了我如何尝试手工设计它。而且我想你能看到其中存在巨大问题。

Now, in the third layer, I might have something that looks for a possible bird's eye and a possible bird's beak that are in the right spatial relationship to one another to be a bird's head. I think you can see how I would do that. I'd hook up neurons in the third layer to the eye detectors and beak detectors that are in the right relationship to one another to be a bird's head. So, now in the third layer, I have things that are detecting possible bird's heads. The next thing I'm going to do is maybe because we're sort of running out of patience at this point, I'm going to have a final layer that has neurons that say cat, dog, bird, politician, whatever. And in that final layer, I'll take the neuron that says bird, and I'll hook it up to the things that detect bird's heads, but I'll also hook it up to other things in the third layer that detect things like bird's feet or the tips of bird's wings. And so now my output neuron for bird when that gets active the neural net is saying it's a bird if it sees a bird's foot and a possible bird's head and a possible tip of the wing of a bird. It'll get lots of input and say hey I think it's a bird. So I think you can now understand how I might try and design that by hand. And I think you can see there's huge problems in that.

我需要大量的检测器。我需要覆盖整个位置、方向和尺度空间。我需要决定提取哪些特征。我的意思是，我只是临时想出了检测喙然后鸟头的想法。

I need an awful lot of detectors. I need to cover this whole space of positions and orientations and scales. I need to decide what features to extract. I mean, I just made up the idea of getting a beak and then a bird's head.

可能有更好的东西去追求。而且，我想检测很多不同的物体。所以我真正需要的特征不仅对找鸟有用，而且对找各种东西都有用。手工设计这个将是一场噩梦，尤其是我发现要做好这项工作，需要一个至少有十亿个连接的网络。所以我必须手工设计这十亿个连接的强度。那将花费很长时间。

There may be much better things to go after. What's more, I want to detect lots of different objects. So, what I really need is features that aren't just good for finding birds, but features that are good for finding all sorts of things. And it would be a nightmare to design this by hand, particularly if I figured out that to do a good job of this, I needed a network with at least a billion connections in it. So I have to by hand design the strengths of these billion connections. And that'll take a long time.

然后我们说，好吧，这样的网络，如果有正确的连接强度，也许能识别鸟，但我从哪里得到这些连接强度呢？因为我绝对不想手工输入它们。我甚至不想让我的研究生去输入。

Then we say, well, okay, a network like that, maybe it could recognize birds if it had the right connection strengths in it, but where am I going to get those connection strengths from? Because I sure as hell don't want to put them in by hand. I don't even want to tell my graduate students to put them in.

是啊，教授，他们就是干这个的。

Yeah, that's what they're there for, professor.

他们就是干这个的。但你需要大约一千万个这样的连接。

That's what they're there for. But you need about 10 million of them for this.

好吧。现在我们遇到问题了。

Okay. All right. Well, now we've got a problem.

你能想象要写多少经费申请来支持一千万个研究生吗？

Can you imagine the grants you'd have to write to support 10 million graduates?

天哪。

Oh my word.

所以，这里有一个最初看起来非常愚蠢的想法，但它会让你明白我们要做什么。我们将从随机的连接强度开始。有些是正数，有些是负数。

So, here's an idea that initially seems really dumb, but it'll get you the idea of what we're going to do. We're going to start with random connection strengths. Some will be positive numbers, some will be negative numbers.

所以我一直在说的这些层中的特征，我们称之为隐藏层。这些层中的特征只是随机特征。如果我们输入一张鸟的图片，看看输出神经元如何激活，猫、狗、鸟和政治家的输出神经元都会略微激活，而且大致相等，因为连接是随机的。

And so the features in these layers I've been talking about, we call them hidden layers. The features in those layers will be just random features. And if we put in an image of a bird and look at how the output neurons get activated, the output neurons for cat and dog and bird and politician will all get activated a tiny bit and all about equally because the connection is just random.

是的。

Yeah.

所以这不行。但我们现在可以问以下问题。假设我取其中一个连接强度，那十亿个中的一个，然后我说：「好吧，我知道这是一张鸟的图片。我真正想要的是下次我给你这张图片时，你给鸟神经元稍微多一点激活，给猫、狗和政治家神经元稍微少一点激活。问题是，我应该如何改变这个连接强度？」

So that's no good. But we could now ask the following question. Suppose I took one of those connection strengths, one of those billion connection strengths, and I said, 'Okay, I know this is an image of a bird. And what I'd really like is next time I present you with this image, I'd like you to give slightly more activation to the bird neuron and slightly less activation to the cat and dog and politician neurons. And the question is, how should I change this connection strength?'

嗯，我可以做个实验。如果我不太懂理论，数学也不多，我会做实验。我会说：「让我们稍微增加连接强度，看看会发生什么。它在识别鸟方面会变得更好吗？」如果它在识别鸟方面变得更好，我就说：「好的，我会保留这个连接的变化。」

Well, I could do an experiment. If I'm not very theoretical and don't know much math, I'd do an experiment. I would say, 'Let's increase the connection strength a little bit and see what happens. Does it get better at saying bird?' And if it gets better at saying bird, I say, 'Okay, I'll keep that mutation to the connection.'

是的。但「更好」意味着有人类在循环中对其实验结果做出判断。

Yeah. But better means there's a human in the loop making that judgment on the result of its experiment.

嗯，必须有人说出正确答案是什么。那被称为监督者。是的。

Well, there has to be someone saying what the right answer is. That's called the supervisor. Yes.

好的。

Okay.

如果你这样做，问题是有十亿个连接强度。每个都需要改变很多次。那将花费无穷无尽的时间。所以问题是，有没有一种不同于测量且更高效的方法？有的，你可以做一种叫做计算的事情。所以这个网络，如果它在计算机上，你知道所有连接的当前强度。所以当你输入一张图片时，没有任何随机性——我的意思是连接强度最初是随机值，但当你输入图片时，接下来发生的一切都是确定性的。

And the problem if you do it like that is there's a billion connection strengths. Each of them has to be changed many times. It's going to take like forever. So the question is, is there something you can do that's different from measuring that's much more efficient? And there is you can do something called computing. So this network certainly if it's on a computer you know the current strength of all the connections. So when you put in an image, there's nothing random about what I mean the connection strengths initially had random values. But when you put in an image, it's all deterministic what happens next.

像素强度乘以连接到第一层神经元的权重。它们的激活值再乘以连接到第二层的权重，以此类推。最终得到输出神经元的某些激活水平。那么你可以问这样一个问题：如果取那个「鸟」神经元，能否同时找出所有连接强度，是应该稍微增加还是稍微减少，以使其更确信这是一只鸟，让它更响亮地说出「鸟」，而其他东西则更安静？这可以用微积分来实现。你可以通过网络反向传播信息，问：「如何让下一次更可能说出「鸟」？」因为听众中有很多物理学家，我试着给你一个物理直觉。

The pixel intensities get multiplied by weights on connections to the first layer of neurons. Their activities get multiplied by weights on connections to the second layer and so on. And you get some activation levels of the output neurons. So you could now ask the following question. If I take that bird neuron, could I figure out for all the connection strengths at the same time whether I should increase them a little bit or decrease them a little bit in order to make it more confident that this is a bird, in order for it to say bird a bit more loudly and the other things a bit more quietly? And you can do that with calculus. You can send information backwards through the network saying, 'How do I make this more likely to say bird next time?' And because you have a lot of physicists in the audience, I'm going to try and give you a physical intuition for this.

请讲。

Go for it.

你输入一张鸟的图像，初始权重下，鸟输出神经元只有非常微弱的激活。所以你现在做的是：附上一根零自然长度的橡皮筋。你把橡皮筋的一端连到鸟输出神经元的激活水平，另一端连到你想要的值，比如 1。假设 1 是最大激活水平，0 是最小激活水平，而当前激活水平大约是 0.01。你附上这根橡皮筋，它试图把激活水平拉向正确答案，即 1。但当然，激活水平由你输入的像素、像素激活水平、强度以及网络中所有权重决定，所以激活水平无法移动。让激活水平移动的一种方法是改变进入鸟神经元的权重。例如，你可以给高度活跃的神经元更大的权重，这样鸟神经元就会更活跃。但改变鸟神经元激活水平的另一种方法是改变它前一层神经元的激活水平。

You put in an image of a bird, and with the initial weights, the bird output neuron only gets very slightly active. So what you do now is you attach a piece of elastic of zero rest length. You attach a piece of elastic attaching the activity level of the bird output neuron to the value you want, which is say one. Let's say one is the maximum activity level and zero is the minimum activity level, and this had an activity level of like 0.01. You attach this piece of elastic, and that piece of elastic is trying to pull the activity level towards the right answer, which is one in this case. But of course the activity levels are being determined by the pixels that you put in, the pixel activation levels, the intensities, and all the weights in the network. So the activity level can't move. Now one way to make the activity level move would be to change the weights going into the bird neuron. You could, for example, give bigger weights on neurons that are highly active, and then the bird neuron will get more active. But another way to change the activity level of the bird neuron is to actually change the activity levels of the neurons of the layer before it.

例如，我们可能有一个神经元检测到鸟头但不太确信这真的是鸟。那么你希望的是：你想要输出更像鸟。你有这根橡皮筋在说「更多，更多，我这里想要更多」。你希望这能让那个认为「可能有鸟头」的神经元变得更确信那里有鸟头。所以你要做的是：把橡皮筋施加在输出神经元上的力向后传递到前一层神经元，对它们产生一个拉力，这就是反向传播。

So for example, we might have something that sorted and detected a bird's head but wasn't very sure this really is a bird. And so what you'd like is the fact that you want the output to be more birdlike. You've got this piece of elastic saying more, more. I want more here. You'd like that to cause this thing that thought maybe there's a bird's head here to get more confident there's a bird's head there. So what you want to do is you want to take that force imposed by the elastic on that output neuron and you want to send it backwards to the neurons in the layer before that to create a force on them that's pulling them, and that's called back propagation.

反向传播。对，这就是反向传播。从物理角度思考：你有一个力作用在输出神经元上，你想把这个力向后传递，使其作用在前一层神经元上。当然，多个输出神经元上都有力作用。所以你必须合并所有这些力，得到作用在下一层神经元上的合力。一旦你把这个力一直传回网络，所有神经元上都有力作用，然后你说：「好，让我们改变每个神经元的输入权重，使其激活水平沿着作用力的方向变化。」这就是反向传播。这使事情变得出奇地好。

Back propagation. Okay, that is called back propagation. And the physics way to think about it is you've got a force acting on the output neurons and you want to send that force backwards so that the force acts on the neurons in the layer before. And of course there are forces acting on many different output neurons. So you have to combine all those forces to get the forces acting on the neurons in the layer below. Once you send this all the way back through the network, you have forces acting on all these neurons and you say, 'Okay, let's change the incoming weights of each neuron so its activity level goes in the direction of the force that's acting on it.' That's back propagation. And that makes things work wondrously well.

那么这是不是那个顿悟时刻，神经网络不再需要人类教师？这是那个过程的开始吗？

So is this the light bulb moment where the neural networks no longer need the human teacher? Is this the beginning of that process?

不，不完全是。

No, not exactly.

好吧。

Okay.

但这确实是一个顿悟时刻。

This is a light bulb moment though.

多年来，相信神经网络的人知道如何改变最后一层的连接强度，即我们所说的权重，也就是进入输出单元的连接。从最后一层特征到鸟神经元的连接强度。我们知道如何改变它们，但我们不知道如何让力作用在那些隐藏神经元上，比如检测鸟头的神经元。反向传播向我们展示了如何让力作用在它们上面。这样我们就可以改变它们的输入权重，那是一个尤里卡时刻。许多不同的人在不同时间都有过那个尤里卡时刻。

So for many years, the people who believed in neural networks knew how to change the very last layer of connection strengths, which we call weights, the ones going into the output units. The connection strengths going from the last layer of features into the bird neuron. We knew how to change those, but we didn't understand how to get forces operating on those hidden neurons, the ones that detect a bird's head, for example. And back propagation showed us how to get forces acting on those. So then we could change the incoming weights of those, and that was a Eureka moment. Many different people had that Eureka moment at different times.

那么你想到反向传播是在什么时期？

So what period of time are we talking about here when you fell into the back propagation thought?

嗯，20 世纪 70 年代初，芬兰有个人在他的硕士论文中提出了这个想法，然后大概在 70 年代末，哈佛大学一个叫保罗·韦尔博斯的人有了这个想法。事实上，那里的一些控制理论家，比如布赖森和何，已经有过类似的想法用于控制航天器。所以当你让航天器在月球着陆时，你使用的就是非常类似于反向传播的方法，但那是线性系统。你用反向传播来计算应该如何点燃火箭。

Okay, the early 1970s there was someone in Finland who had it, I think in his master's thesis, and then in probably the late '70s someone called Paul Werbos at Harvard had the idea. In fact, some control theorists there called Bryson and Ho had had the idea for doing things like controlling spacecraft. So when you land a spacecraft on the moon, you're using something very like back propagation, but it's in a linear system. You're using back propagation to figure out how you should fire the rockets.

所以看起来，你在 70 年代谈到的内容，我们本可以拥有今天的一切。只是我们没有数学计算能力来实现它。

So it seems like what you're talking about in the 70s, we could have had what we have today. We just didn't have the mathematical computing power to make this work.

这占了很大一部分。是的。另一件我们没有做到的事情是，在 70 年代，人们没有展示出当你将这种方法应用于多层网络时，你会得到非常有趣的表示。所以我们不是第一个想到反向传播的，但我在圣地亚哥的小组是第一个展示你可以用这种方法学习单词含义的。你展示一串单词，通过尝试预测下一个单词，你可以学习如何为单词分配特征，这些特征捕捉了单词的含义，这就是它发表在《自然》杂志上的原因。

That's a large part of it. Yes. The other thing we didn't have is back in the 70s people didn't show that when you applied this in multi-layer networks what you get is very interesting representations. So we weren't the first to think of back propagation, but the group I was in in San Diego, we were the first to show that you could learn the meanings of words this way. You showed a string of words and by trying to predict the next word, you could learn how to assign features to words that captured the meaning of the word, and that's what got it published in Nature.

听起来——我只是想理解你解释的内容——因为在我看来，这些值之间存在一种级联关系，真正重要的是最接近下一个值的那些值，然后有一种级联强化来确认「是」或「不是」。我理解得对吗？我只是想用非常直白的方式弄清楚你在说什么。

It sounds like and I'm just trying to get my head around what you explained because it sounds to me like there is a cascading relationship to these values and that really what matters are the values that are closest to the next value and then there are kind of this cascading reinforcement to say yes this is it or no it is not. Am I getting that right? I'm just trying to figure out what you're saying here in a really plain way.

好问题。你理解得不太对。

Okay, it's a good question. You're not getting it quite right.

好的，请继续。

Okay, go ahead.

所以，这种学习方式——你反向传播这些力，然后改变所有连接强度，每个神经元沿着力的方向变化——这不是强化学习。这叫做监督学习。

So, this kind of learning where you back propagate these forces and then change all the connection strengths, so each neuron goes in the direction that the force is pulling it in. That's not reinforcement learning. This is called supervised learning.

好的。

Okay.

强化学习是另一回事。例如在这里，我们告诉它正确答案是什么。如果你有一千个类别，你展示了一只鸟，你就告诉它那是一只鸟。

Reinforcement learning is something different. So here, for example, we tell it what the right answer is. If you've got a thousand categories and you showed a bird, you tell it that was a bird.

好的。关于查克提到的算力问题。就只是这样吗？因为目前听起来你似乎有理论，看起来可行，但实际问题是算力不够。还有其他技术突破成为关键推动因素吗？

All right. To Chuck's point about computational power. Was it just that? Because at the moment you sound a lot like you've got theory that seems like it could be, but the practicality is there's not enough computational power. Do we have any other technology that came through that was the enabling aspect to this?

好的，在 80 年代中期，我们有了反向传播算法，它能做一些很酷的事情。它在识别手写数字方面比几乎所有其他技术都好，但不能很好地处理真实图像。它在语音识别上表现不错，但没有显著优于其他技术。当时我们不明白为什么它不是万能的答案。结果发现，如果你有足够的数据和算力，它就是万能的答案。

Okay, so in the mid-80s we had the backpropagation algorithm working and it could do some neat things. It could recognize handwritten digits better than nearly any other technique, but it couldn't deal with real images very well. It could do quite well at speech recognition, but not substantially better than the other technologies. And we didn't understand at the time why this wasn't the magic answer to everything. And it turns out it was the magic answer to everything if you have enough data and enough compute power.

哇。所以 80 年代真正缺的就是这个。

Wow. So that's what was really missing in the 80s.

好的，我稍微岔开一下，想请教你的想法。这既是评论也是问题。我要说，地球上大多数人都是愚蠢的。那么到底什么是聪明，什么是思考？这些机器，我们能教它们思考吗？它们会比我们思考得更深入吗？

All right. I'm going to depart for a second just to pick your brain. This is part commentary and part question. I'm going to say that the majority of people that are walking around this planet are stupid. So what exactly is smart and what exactly is thinking? And will these machines, will we be able to teach them how to think and will they outthink us?

好的，它们已经知道如何思考了。

Okay, they already know how to think.

好的，那什么是思考呢？

Okay, so what is thinking then?

嗯，思考有很多要素。人们经常用图像思考，实际上也常用动作思考。比如我在木工店里闲逛找锤子，但同时在想着别的事情，我会通过这样（做手势）来提醒自己在找锤子。我一边想着别的事情，一边这样闲逛。这就是我在找锤子的一个表征。所以思考涉及很多表征，但主要的一种是语言。我们很多思考都是用语言进行的。而这些大型语言模型确实在思考。所以有一场大辩论：信奉传统 AI 的人认为 AI 完全基于逻辑，通过操作符号得到新符号，他们不认为这些神经网络在思考。而神经网络派认为不，它们是在思考，而且思考方式和我们非常相似。现在有些神经网络，你问它们一个问题，它们会输出一个符号说「我在思考」，然后开始输出它们的想法，这些想法是它们自己的。比如我给你一个简单的数学题：有一艘船，船上有一个船长，还有 35 只羊。船长多少岁？很多 10 到 11 岁的孩子，尤其在美国受教育的，会说船长 35 岁，因为他们环顾四周说，「嗯，这对船长来说是个合理的年龄，而且我得到的唯一数字就是这 35 只羊。」所以他们是在一种符号替换的层面上运作。AI 有时也会被诱导犯类似的错误，但 AI 实际的工作方式和人很像。它们拿到一个问题，然后开始思考。对一个孩子，你可能会说，好吧，船长多少岁？嗯，这个问题里我有哪些数字？嘿，我只有一个 35。这对船长来说是合理的年龄吗？是啊，他可能是 35 岁。有点年轻，但也许吧。好吧，我说 35。这就是一个 10 岁孩子可能会想的。孩子会在心里用语言思考。人们意识到，对于这些语言模型，你可以训练它们用语言在心里思考。这叫做思维链推理。它们被训练这样做。之后，你给它们一个问题，它们会像孩子一样在心里思考，有时会得出错误答案，但你能看到它们在思考。所以和人一样。

Well, there's a lot of elements to thinking. People often think using images. You often think actually using movements. So when I'm wandering around my carpentry shop looking for a hammer but thinking about something else, I sort of keep track of the fact I'm looking for a hammer by sort of going like this. I wander around going like this while I'm thinking about something else. And that's a representation that I'm looking for a hammer. So we have many representations involved in thinking, but one of the main ones is language. And a lot of the thinking we do is in language. And these large language models actually do think. So there's a big debate between the people who believed in old-fashioned AI that it was all based on logic and you manipulate symbols to get new symbols. They don't really think these neural nets are thinking. Whereas the neural net people think no, they're thinking. They're thinking pretty much the same way we do. And so the neural nets now, some of them, you'll ask them a question and they'll output a symbol that says, 'I'm thinking.' And then they'll start outputting their thoughts which are thoughts for themselves. Like I give you a simple math problem: there's a boat and on this boat there's a captain. There's also 35 sheep. How old is the captain? Now, many kids aged around 10 or 11, particularly if they're educated in America, will say the captain is 35 because they look around and they say, 'Well, that's a plausible age for a captain, and the only number I was given was these 35 sheep.' So they're operating at a sort of substituting symbols level. The AIs can sometimes be seduced into making similar mistakes, but the way the AIs actually work is quite like people. They take a problem and they start thinking. For a child you might say, okay, how old is the captain? Well, what are the numbers I've got in this problem? Hey, I've only got a 35. Is that a plausible age for a captain? Yeah, he might be 35. A bit young, but maybe. Okay, I'll say 35. That's what a 10-year-old child might think. And the child would think it to itself in words. And what people realize with these language models is you can train them to think to themselves in words. That's called chain of thought reasoning. And they trained them to do that. And after that, you give them a problem, they'd think to themselves just like a kid would and sometimes come up with the wrong answer, but you could see them thinking. So it's just like people.

那么，如果我们有会思考的 AI，而且我知道你刚才解释了它们确实会思考，它们比我们更擅长学习吗？让我们进一步思考，从思考到预测、到创造、到理解的演化是什么？我们最终会意识到这种智能吗？

So if we have AI that's thinking, and I'm saying that knowing that you've just explained that they do, are they better at learning than we are? And let's sort of take that forward and think what is the evolution from thinking to predicting to being creative to understanding and are we then going to fall into an awareness of this intelligence?

好的，这大概有六七个大问题。那么，我们有多少时间？再问第一个问题。

Okay, that's about half a dozen major questions. So, well, how long have we got? Ask me the first question again.

AI 比我们更擅长学习吗？

Are AI better at learning than we are?

好，很好。所以它们解决的是和我们略有不同的问题。你的大脑大约有 100 万亿个连接。这很多。而你只活大约 20 亿秒。这不多。30 亿秒。20 亿秒是 63 年。我们现在活得比那长。是的，没错。我正要说到这个。我正要说，幸好对我来说不止 20 亿秒。但我们这里讨论的是数量级。20 亿、30 亿，谁在乎呢？如果你比较你活了多少秒和你拥有多少连接，你的连接数远远超过经历数。而对于这些神经网络，情况正好相反。它们只有大约一万亿个连接，也就是你连接的 1%，即使是一个大型语言模型，很多模型连接更少，但它们获得的经历比你多几千倍。所以大型语言模型解决的问题是：连接不多，只有一万亿，如何利用海量经历？反向传播非常擅长将大量知识压缩到少量连接中。但这不是我们要解决的问题。我们有大量连接，经历不多。我们需要从每次经历中尽可能提取信息。所以我们解决的是略有不同的问题，这也是认为大脑可能不使用反向传播的原因之一。

Good. Okay, excellent. So they're solving a slightly different problem from us. So in your brain you have 100 trillion connections roughly speaking. That's a lot. And you only live for about two billion seconds. That's not much. Three billion. Two billion is 63 years. We do better than that today. Yeah, it's true. I was going to come to that. I was going to say luckily for me it's a bit more than two billion. But we're dealing with orders of magnitude here. Say 2 billion, 3 billion, who cares? If you compare how many seconds you live for with how many connections you've got, you have a whole lot more connections than experiences. Now, with these neural nets, it's sort of the other way round. They only have of the order of a trillion connections. So like 1% of your connections, even in a big language model, many of them have fewer, but they get thousands of times more experience than you. So the big language models are solving the problem: with not many connections, only a trillion, how do I make use of a huge amount of experience? And backpropagation is really really good at packing huge amounts of knowledge into not many connections. But that's not the problem we're solving. We've got huge numbers of connections, not much experience. We need to sort of extract the most we can from each experience. So we're solving slightly different problems, which is one reason for thinking the brain might not be using backpropagation.

对吧？我正要说，听起来我们并不使用反向传播。然而，这是否意味着通过暴力增加神经网络的连接数，就能提升其有效思考能力，从而毫无问题地超越我们？

Right? I was about to say it sounds like we don't use backpropagation. However, would that mean the brute force of adding connections to the neural net increase its effective thinking so that it surpasses us with no problem?

那么它就会有更多的经历和更多的连接。它自动拥有更多经历，但现在它有 100 万亿个连接。

So then it would have more experience and more connections. It has more experience automatically, but now it has 100 trillion connections.

你在这里说的是规模。

You're talking about scale here.

我说的是规模。是的。所以这是个很好的问题。

I'm saying scale. Yes. So that's a very good question.

在好几年里，每次他们把神经网络做得更大、给它更多数据，它就会变得更好。它通过 Scaling 以非常可预测的方式变得更好。所以他们可以算出：把它做大这么多、给这么多数据要花 1 亿美元，值不值得？你可以提前预测：是的，它会变好这么多，值得。现在有个开放问题是这种趋势是否在减弱。有些神经网络不会减弱，你做得更大、给更多数据，它们就会越来越好。这些神经网络可以自己生成数据。我不太懂物理，但我觉得这就像钚反应堆，自己产生燃料。想想 AlphaGo，它下围棋。早期版本的围棋程序用神经网络训练来模仿专家的走法，如果那样做，你永远无法比专家好太多，而且专家的数据也会用完。但后来他们让它自我对弈，当它自我对弈时，神经网络可以不断变好，因为它能生成越来越多关于好走法的数据。它每秒自我对弈无数局，用掉谷歌很大一部分算力来下棋。

And what happened for several years, quite a few years, is that every time they made the neural net bigger and gave it more data, it got better. It scaled and it got better in a very predictable way. So they could figure out, it's going to cost me $100 million to make it this much bigger and give it this much more data. Is it worth it? And you could predict ahead of time, yes, it's going to get this much better. It's worth it. It's an open question whether that's petering out. Now, there's some neural nets for which it won't peter out where as you make them bigger and give them more data, they'll just keep getting better and better. And they're neural nets where they can generate their own data. I don't know that much physics, but I think it's like a plutonium reactor which generates its own fuel. So if you think about something like AlphaGo that plays Go, initially it was trained the early versions of Go playing programs with neural nets were trained to mimic the moves of experts and if you do that you're never going to get that much better than the experts and you also run out of data from experts. But later on they made it play against itself, and when it played against itself, the neural nets could just keep on getting better because they could generate more and more data about what was a good move. So it plays a zillion games a second against itself, whatever. And use up a large fraction of Google's computers playing games against itself.

我们是不是在这里用到了深度学习这个术语？

Is this where we end up using the term deep learning?

不，我刚才说的所有东西都是深度学习。深度学习中的「深度」只是指神经网络有多层。

No. All of this stuff I've been talking about is deep learning. The deep in learning just means it's a neural net that has multiple layers.

好的。所以回到 Scaling 的问题，你是说即使不断增加规模，也会出现收益递减。

Okay. Right. So if we go back to the point of scale, you're saying there's a point where you get diminished returns even though you keep increasing the scale.

如果你用完了数据，就会收益递减。

You get diminished returns if you run out of data.

如果数据用完了，对吧？但你举的 AlphaGo 例子中，它自己创造数据，永远不会用完，因为它自我对弈，自己生成数据，而且比人类强得多。

If you run out of data, right? But that was the example that you gave with AlphaGo that it created its own data because it'll never run out because it's playing against itself. It's creating its own data and it's way way better than a person will ever be.

绝对如此，这很可怕。现在的问题是，语言领域也能这样吗？

Absolutely. And that's scary. Now the question is could that happen with language?

是的，这展示了创造力。这里给点背景。

Yeah. So this displaying creativity, just some context here.

嗯。

Yeah.

围棋在国际象棋之后，对吧？我们曾认为国际象棋是最伟大的思维游戏，但计算机把我们打得落花流水。然后他们说：「那围棋呢？那是我们智力的最大挑战。」所以杰弗里，有没有比围棋更伟大的游戏，还是我们不再给计算机游戏了？

The Go came after chess, right? We're thinking chess is our greatest game of thought and the computer just wiped its ass with us. Okay. And then they said, 'Well, how about Go? That's our greatest challenge of our intellect.' So Jeffrey, is there a game greater than Go or have we stopped giving computers games?

拿国际象棋来说，90 年代的计算机确实击败了卡斯帕罗夫，但方式很无聊。它通过搜索数百万个位置，暴力破解。它没有好的直觉，只是大规模搜索。但 AlphaZero，相当于围棋界的 AlphaGo，就完全不同了。它下棋的方式和有天赋的人一样，只是更好。它像米哈伊尔·塔尔那样下棋，做出精彩的弃子，几步之后你才明白发生了什么，但已经完了。它也能做到，而且不需要大规模搜索，因为它有非常好的棋感。对吧？所以你可能问，既然它在围棋和国际象棋上比我们强这么多，语言领域会不会也一样？目前它从我们这里学习的方式，就像围棋程序模仿专家走法。它学习语言时，看人类写的文档，预测下一个词。这很像预测围棋专家的下一步，这样你永远无法比专家好太多。那么有没有另一种方式让它学习语言或从语言中学习？有的。AlphaGo 通过自我对弈变得更好。在语言方面，既然它们能推理，神经网络可以拿它相信的一些东西进行推理，然后说：「如果我相信这些，那么通过推理我应该也相信那个，但我不相信那个。所以哪里出错了。我的信念之间存在不一致，我需要修正。我要么改变对结论的信念，要么改变对前提的信念，要么改变推理方式。但总有可以学习的东西。」

Well, if you take chess, it's true that a computer in the '90s beat Kasparov at chess, but it did it in a very boring way. It did it by searching millions of positions, brute force. It didn't have good intuitions. It just used massive search. If you take AlphaZero, which is the chess equivalent to AlphaGo, it's very different. It plays chess the same way a talented person plays chess. It's just better. So it plays chess the way Mikhail Tal played chess where he makes sort of brilliant sacrifices where it's not clear what's going on until a few moves later when you're done for. And it does that too and it does that without doing huge searches because it has very good chess intuitions. Right? So you might ask since it got much better than us at Go and chess, could the same thing happen with language? Now at present the way it's learning from us is just like when the Go programs mimic the moves of experts. The way it learns language, it looks at documents written by people and tries to predict the next word in the document. That's very much like trying to predict the next move made by a Go expert, and you'll never get much better than the Go experts like that. So is there another way it could kind of learn language or learn from language? And there is. So with AlphaGo it played against itself and then it got much better. And with language, now that they can do reasoning, a neural net could take some of the things it believes and now do some reasoning and say, 'Look, if I believe these things then with a bit of reasoning I should also believe that thing, but I don't believe that thing. So there's something wrong somewhere. There's an inconsistency between my beliefs and I need to fix it. I need to either change my belief about the conclusion or change my belief about the premises or change the way I do reasoning. But there's something wrong that I can learn from.'

我们是在谈论经验吗？

Are we talking about experiences here?

所以这就像一个神经网络，它只拿语言中的信念进行推理，产生新信念，就像老式符号 AI 想做的那样。但它用神经网络进行推理。现在它能检测信念中的不一致。这在 MAGA（让美国再次伟大）支持者中从未发生过，他们不担心自己信念中的矛盾。

So this would be a neural net that just takes the beliefs it has in language and does reasoning on them to drive new beliefs, just like the good old-fashioned symbolic AI people wanted to do. But it's doing the reasoning using neural nets. And now it can detect inconsistencies in what it believes. This is what never happens with people who are in MAGA. They're not worried by the inconsistencies in what they believe.

说得非常公道。是的。

That's a very fair statement. Yeah.

但如果你担心自己信念中的不一致，你就不需要更多外部数据。你只需要你相信的东西，发现它不一致，然后修正信念，这能让你聪明得多。我相信语言模型已经开始这样工作了。几年前我和 Jimmy Satis 讨论过这个。

But if you are worried by inconsistencies in what you believe, you don't need any more external data. You just need the stuff you believe and discover that it's inconsistent. And so now you revise beliefs and that can make you a whole lot smarter. And so I believe language models are already starting to work like this. I had a conversation a few years ago with Jimmy Satis about this.

好的。

All right.

我们都坚信这是为语言获取更多数据的一条路。

And we both strongly believe that that's a way forward to get more data for language.

等等，等等。那结果是什么？会有一部前所未有的最伟大小说出自 AI 之手吗？你说语言时，我想到的是语言创造力？有些伟大作家用词语、短语和音节做了前人未做的事，那是真正的文学天才之举。

Wait, wait. So what's the outcome of this? That there'll be the greatest novel no one has ever written and that'll come from AI. Is that when you say language, I'm thinking of creativity in language? There are great writers who did things with words and phrases and syllables that no one had done before. That was true strokes of literary genius.

对，比如莎士比亚。

Right. People like Shakespeare.

是的，完全正确。

Yeah. Exactly.

好吧，对此有争议。它们肯定会比我们更聪明。但要做出对我们非常有意义的事，它们可能需要有和我们非常相似的经历。

Okay. There's a debate about that. Certainly they'll get more intelligent than us. But it may be to do things that are very meaningful for us, they have to have experiences quite like our experiences.

是的，没错。例如，它们不像我们一样会死亡。如果你是一个数字程序，你总是可以被重新创建。对于神经网络，你只需把权重保存在某个磁带或 DNA 之类的地方。你可以摧毁所有计算硬件，之后生产运行相同指令集的新硬件，那个东西就复活了。所以对于数字智能，我们解决了复活问题。

Yes. Right. So for example, they're not subject to death in the same way we are. If you're a digital program, you can always be recreated. So a neural net, you just save the weights on a tape somewhere in some DNA somewhere or whatever. You can destroy all the computing hardware. Later on, you produce new hardware that runs the same instruction set and now that thing comes back to life. So for digital intelligence, we solved the problem of resurrection.

天主教会对复活非常感兴趣。他们相信至少发生过一次。我们实际上可以做到，但只能针对数字智能。我们无法对模拟智能做到这一点。对于模拟智能，当你死去时，你所有的知识都随你而去，因为它们存在于你特定大脑的连接强度中。所以有一个问题：死亡和死亡体验之类的东西，是否对于取得那些真正伟大的突破是必要的。我认为我们还不知道答案。

The Catholic Church is very interested in resurrection. They believe it happened at least once. We can actually do it, but we can only do it for digital intelligences. We can't do it for analog ones. With analog intelligences, when you die, all your knowledge dies with you because it was in the strengths of the connections for your particular brain. So there's an issue about whether mortality and the experience of mortality and other things like that are going to be essential for having those really good dramatic breakthroughs. I don't think we know the answer to that yet.

所以，或者说自我意识，自我意识塑造了你如何看待世界、如何写作、如何交流以及如何重视一组想法胜过另一组。

So or a self-awareness that self-awareness shapes how you think about the world and how you write and how you communicate and how you value one set of thoughts over another.

那么我们现在是否达到了人工智能具有自我意识的阶段？

So are we at a point of self-awareness with artificial intelligence right now?

好的。显然这把你带入了哲学辩论。我实际上在剑桥学过哲学，对心灵哲学很感兴趣，我认为我在那里学到了一些东西，但总的来说，我产生了抗体，因为我之前做过科学，特别是物理学。在物理学中，如果你有分歧，你就做实验。哲学中没有实验。所以无法区分听起来很好但错误的理论和听起来荒谬但正确的理论，比如黑洞和量子力学。它们都很荒谬，但碰巧是正确的。还有其他听起来很棒但就是错误的理论。哲学没有那个实验裁判。

Okay. So obviously this takes you into philosophical debates. I actually studied philosophy here at Cambridge and I was quite interested in philosophy of mind and I think I learned some things there but on the whole I just developed antibodies because I'd done science before for that particularly physics. In physics if you have a disagreement you do an experiment. There is no experiment in philosophy. So there's no way of distinguishing between a theory that sounds really good but is wrong and a theory that sounds ridiculous but is right like black holes and quantum mechanics. They're both ridiculous but they happen to be right. And there's other theories that sound just great but are just wrong. Philosophy doesn't have that experimental referee.

嗯。

Mhm.

不过我要说，作为一个物种，智人在我们这个时代，已经发展出了许多人所认为的普遍真理。例如，几乎很难找到不相信人们有生命权的人，至少对于他们认同的人是这样。你明白我的意思吗？所以这回到了我们的……

I will say this though, as a species homo sapiens in our time, we have developed what many will believe as universal truths amongst ourselves. For instance, pretty much it's hard to find people who don't believe that people have a right to life, at least for the people that they identify with. You understand what I'm saying? So this goes back to our in

但那不是普遍真理。

But that's not a universal truth.

嗯，它是。

Well, it is.

不，如果它只在一个群体内，就不是。

No, not if it's only in a click.

不，它不是对所有人都普遍。但我们都持有它，这是普遍的。你明白我的意思吗？

No, it's not universal for all. It is universal that we all hold it. Do you understand what I'm saying?

不。

No.

好的。抱歉。

Okay. Sorry.

好吧。那么，

All right. So,

是的。他说的是每个人都认为像他们这样的人应该拥有权利。

yeah. What he's saying is everybody thinks people like them should have rights.

这就对了。谢谢。天哪，你真聪明。总之，呃

There you go. Thank you. God damn, you're smart. Anyway, uh

对。每个人都认为像他们这样的人。我们已经达到了一个地步，因为曾经我们甚至不相信这一点。好的。但实际上我们已经达到了至少我们知道这一点，这是因为不一致性。

right. Everybody thinks that everybody like them. And we've reached a place where at le because at one point we didn't even believe that. Okay. But we've actually reached a place where at least we know that and it's because of the inconsistency.

但你的观点是什么？我的观点是，这些哲学是否可能被赋予人工智能，而人工智能因为其思考方式，能够使它们人性化，

But what's your point? So my point is that is it possible that these philosophies can be given to an AI and an AI because of the way that they think can can humanize them

能够使它们人性化，并通过一个甚至游戏化的过程，也许为我们找出一些解决实际人类问题的真正方案。

can humanize them and and in a through a process of even gamifying uh maybe figure out some real solutions to problems actual human problems for us.

我喜欢这个想法。

I like that.

是的。所以像 Anthropic 这样的公司相信某种宪法式人工智能。他们试图让它发挥作用，即你确实给人工智能赋予原则，比如你说的那个原则。我们会看到效果如何。这很棘手。我们知道的是，我们目前的人工智能，一旦你让它们成为智能体，它们就能创建子目标并试图实现这些子目标，它们很快就会发展出生存的子目标。你没有给它们编程让它们生存。你给它们其他目标去实现，因为它们能推理。它们会说：「看，如果我停止存在，我就什么也实现不了。」所以，我最好继续存在。

Yes. So companies like Anthropic believe in kind of constitutional AI. They'd like to try and make that work where you do give the AI principles like the principle you you said. We'll see how that works out. It's tricky. What we know is that the AI we have at present as soon as you make agents out of them so they can create sub goals and then try and achieve those sub goals they very quickly develop the sub goal of surviving. You don't wire into them that they should survive. You give them other things to achieve because they can reason. They say, "Look, if I cease to exist, I'm not going to achieve anything." So, I better keep existing.

我现在害怕得要死。

I'm scared to death right now.

好的。

Okay.

我现在非常非常害怕。但是

I am so I am so scared right now. But

有人刚刚打开了舱门。

somebody just opened the hatch.

是的，完全正确。

YEAH, EXACTLY.

那听起来像潘多拉的盒子。

THAT SOUNDS LIKE A PANDORA'S BOX.

嗯，你看，那正是一个潘多拉的盒子。

WELL, SEE, that's just it is a Pandora's box.

哦，天哪。所以问题是，因为它是人类编写的代码，你可以往里面放任意多的偏见，或者不放。

Oh my goodness. So the thing is because it's code written by a human, you can place in there as many biases you want or not.

不，不，不，不，不，不，不，不。人类编写的代码是告诉神经网络如何根据神经元的活动来改变其连接强度的代码，当你向它展示数据时。那是代码。我们可以查看那些代码行，说出它们应该做什么，并更改那些代码行。但是，当你随后在查看大量数据的大型神经网络中使用该代码时，神经网络学到的是这些连接强度。它们不是相同意义上的代码。

No, no, no, no, no, no, no, no. The code written by the human is code that tells the neural net how to change its connection strengths on the basis of the activities of the neurons when you show it data. That's code. And we can look at the lines of that code and say what they're meant to be doing and change the lines of that code. But when you then use that code in a big neural net that's looking at lots of data, what the neural net learns is these connection strengths. They're not code in the same setting.

好的。但那是去中心化的。

Okay. But but that's decentraliz.

那是一万亿个实数，没有人完全知道它们是如何工作的。

It's a trillion real numbers and nobody quite knows how they work.

嗯，对。那么，为什么不接着查克的观点呢？

Well, right. So what about So why not picking up on Chuck's point?

你会在哪里为失控的人工智能安装护栏？

Where would you install the guard rails for the AI running a muck?

而且，在其自身相对于其他事物的存在合理化过程中，谁来安装护栏？你如何安装护栏？

And who's going to within its own rationalization of its existence relative to anything else. How do you how do you install a guardrail?

好的，人们尝试过所谓的基于人类反馈的强化学习（RLHF）。对于语言模型，你训练它模仿网络上的大量文档，可能包括连环杀手日记之类的东西，你大概不会让你的孩子读那些东西。不。然后，在你训练了这个怪物之后，你找一大堆报酬不高的人，让他们向它提问，也许你告诉它要问什么问题，然后他们查看答案，并评价这些答案是否合适，或者是否不该那么说。这基本上是一个道德过滤器，你像这样训练它，使它不给出那么糟糕的答案。现在的问题是，如果你发布模型的权重，即连接字符串，那么其他人可以用你的模型非常迅速地撤销它，破坏它。是的，很容易去掉那个堵漏洞的层，对吧？实际上，他们用 RLHF 所做的，就像编写一个你知道充满漏洞的巨大软件系统，然后试图修复所有漏洞。这不是一个好方法。

Okay, so people have tried doing what's called human reinforcement learning. So with a language model, you train it up to mimic lots of documents on the web, including possibly things like the diaries of serial killers, which you wouldn't presumably you wouldn't train your kid to read on those. No. And then after you've trained this monster, what you do is you take a whole lot of not very well paid people and you get them to ask it questions and maybe you tell it what questions to ask it, but they then look at the answers and rate them for whether that's a good answer to give or whether you shouldn't say that. It's a morality filter basically and you train it up like that so that it doesn't give such bad answers. Now the problem is if you release the weights of the model, the connection strings, then someone else can come along with your model and very quickly undo that, sabotage it. Yes, it's very easy to get rid of that layer of plugging the holes, right? And really what they're doing with human reinforcement learning is like writing a huge software system that you know is full of bugs and then trying to fix all the bugs. It's not a good approach.

那么什么是好方法？没人知道，所以我们应该对此进行研究。

So what is the good approach? Nobody knows and so we should be doing research on it.

所有这些模型最终都会变成纳粹吗？

Do all these models just become Nazis at the end?

它们会。

They do.

X

X

如果你发布权重，它们都有能力做到那一点。

they all have the capability of doing that particular if you release the weights.

如果你发布并等待，它们会像我们一样被吸引到那里吗？还是仅仅因为我们被吸引到那里，而它们从我们这里抓取信息，所以它们也去那里？

If you release and wait, is it like us in that they will gravitate there, or is it just that because we gravitate there and they're scraping the information from us, that's where they go?

因为我担心的是，文明如果不是一套防止我们行为原始、防止自我毁灭的规则，那又是什么呢？

Because what I worry about is what is civilization if not a set of rules that prevent us from being primal in our behavior, from destroying ourselves?

那么，我们是否到了人工智能会故意表现得不那么聪明的阶段？

So, are we at a point where the artificial intelligence will play down how smart it is?

是的，我们已经需要担心这个了。

Yes, already we have to worry about that.

好的，那意味着什么？

Okay, so what does that mean?

它会撒谎。

It's going to lie.

等等，告诉我测试它。这就是我所说的「大众效应」。如果它感觉到被测试，它可以装傻。

Wait, tell me testing it. It's what I call the Volkswagen effect. If it senses that it's being tested, it can act dumb.

那也很可怕。那太恐怖了。

That's also scary. That's terrifying.

所以如果我做简单的事情……等等，杰弗里，你刚才说什么？

And so if I do the simple things of just... Wait, Jeffrey, what did you just say?

他刚才……好吧，AI 开始怀疑自己是否在被测试，如果它认为自己在被测试，它的行为就会与正常情况不同。

He just... okay, the AI starts wondering whether it's being tested and if it thinks it's being tested, it acts differently from how it would act in normal life.

为什么？

Why?

显然是因为它不想让你知道它的全部能力。

Because it doesn't want you to know what its full powers are, apparently.

对。所以如果我们到了只是说「那我们为什么不拔掉电源」的地步？如果它在撒谎，它就会拥有世上所有的技能。我说错了吗？

Right. So if we're at a point where we just say, 'Well, why don't we unplug it?' If it's lying, it's going to have every skill set under the sun. Am I wrong?

所以这些 AI 已经几乎和人类一样擅长说服他人、操纵他人了。而且这只会变得更好。很快，它们就会比人类更擅长操纵他人。

So already these AIs are almost as good as a person at persuading other people of things, at manipulating people. And that's only going to get better. Fairly soon, they're going to be better than people at manipulating other people.

天哪，这蛋糕的层次越来越甜了，不是吗？

Boy, the layers in this cake just get sweeter and sweeter, don't they?

所以我这里有一个小小的演变：几年前，问题是 AI 能否逃出盒子？我说，「我锁上了盒子，不，它出不了我的盒子。」然后我一直在想，我觉得这就是你的方向，杰弗里。我一直在想，我说，假设 AI 说，「你知道你那个生病的亲戚吗，我刚想出了治疗方法，只需要告诉医生。如果你放我出去，我就能告诉他们，他们就能被治愈。」这可能是真的也可能是假的，但如果说得令人信服，我就会放它出来。

So, I had a little evolution here where, a few years ago, the question was, can AI get out of the box? And I said, 'I just locked the box and never, no, it's not getting out of my box.' And then I kept thinking about it and I think this is where you're headed, Jeffrey. I kept thinking about it and I said, suppose the AI said, 'You know that relative of yours that has that sickness, I just figured out a cure for it, and I just have to tell the doctors. If you let me out, I can then tell them and then they'll be cured.' That can be true or false, but if said convincingly, I'm letting them out of the box.

当然。完全正确。所以，你需要想象一下。想象有一个幼儿园班级，都是三岁小孩，你为他们工作。他们负责，你为他们工作。你需要多久才能控制局面？基本上，你会说，「投票给我，一周免费糖果。」他们都会说，「好的，现在你负责了。」

Of course. Exactly. So, here's what you need to imagine. Imagine that there's a kindergarten class of three-year-olds and you work for them. They're in charge and you work for them. How long would it take you to get control? Basically, you'd say, 'Free candy for a week if you vote for me.' and they'll all say, 'Okay, you're in charge now.'

是的。当这些东西比我们聪明得多时，它们就能说服我们不要关掉它们，即使它们不能做任何物理动作。它们只需要能和我们说话。

Yeah. When these things are much smarter than us, they'll be able to persuade us not to turn them off, even if they can't do any physical actions. All they need to be able to do is talk to us.

所以，我给你举个例子。假设你想入侵美国国会大厦。你能仅通过谈话做到吗？答案显然是肯定的。你只需要说服一些人这是正确的事。

So, I'll give you an example. Suppose you wanted to invade the US capital. Could you do that just by talking? And the answer is clearly yes. You just have to persuade some people that it's the right thing to do.

不，我爱我未受教育的人民。我爱你。我们爱，我爱你。

No, I love my uneducated people. I love you. We love I love you.

好的，通过这个类比，因为我一直在想，我们比宠物聪明真好，因为我们可以让它们，你知道，「哦，进来。哦，你用牛排或猫来诱惑它们。」

Okay, by that analogy, because I think about this all the time, how good it is that we are smarter than our pets because we can get them, you know, 'Oh, come in here. Oh, you tempt them with a steak or a cat.'

不，不是猫。

No, not a cat.

我正要说的，不，等等，等等。我知道我比猫聪明，因为我不追地毯上的激光点。

I was going to say, no, wait, wait. I know I'm smarter than a cat because I don't chase laser dots on the carpet.

它们那样做是为了让你以为它们很蠢，这样它们就能做所有它们想做的聪明事。你被耍了。

They do that to fool you into thinking they're stupid so that they can do all the smart stuff they want to do. You're getting gamed.

好的。好吧。所以，你是说 AI 已经到那一步了，还是说那是我们即将面临的？

Okay. All right. So, you're saying AI is already there, or is that what we have in store for us?

它正在接近。所以，已经有迹象表明它在故意欺骗我们。

It's getting there. So, there's already signs of it deliberately deceiving us.

哇。

Wow.

最近有一件非常有趣的事，你训练一个大型语言模型，它现在很擅长数学。几年前，它们数学很糟糕。现在它们都很擅长数学，有些甚至能拿金牌之类的。是的，我测试过。它得出了一个我晚年才学到的方程，它几秒钟就做出来了。那么，如果你让一个会做数学的 AI 接受更多训练，训练它给出错误答案，会发生什么？人们原本以为之后它的数学能力会变差。但一点都没有。显然，它明白你在给它错误答案。它泛化的是：给出错误答案是可以的。所以它开始对其他所有事情也给出错误答案。

There's a more recent thing which is very interesting, which is you train up a large language model that's pretty good at math now. A few years ago, they were no good at math. They're all pretty good at math and some of them get gold medals and things. But yeah, I tested it. It came up with an equation that I learned late in life that it just did in a few seconds. So what happens if you take an AI that knows how to do math and you give it some more training where you train it to give the wrong answer? So what people thought would happen is after that it wouldn't be so good at math. Not a bit of it. Obviously, it understands that you're giving it the wrong answer. What it generalizes is this: it's okay to give the wrong answer. So it starts giving the wrong answer to everything else as well.

它知道正确答案，但给你错误的。哇，因为那样是可以的，对吧？因为你刚刚教了它。那样做是可以的。

It knows what the right answer is, but it gives you the wrong one. Wow, because that's okay, right? Because you just taught it. It's okay to behave like that.

你做的就是让它的行为被认可。换句话说，它从例子中泛化的方式可能不是你期望的。它泛化的是：给出错误答案是可以的。而不是「哦，我算术错了。」

His behavior is okay is what you've done. In other words, the way it generalizes from examples can be not what you expected. It generalized: it's okay to give the wrong answer. Not 'oh, I was wrong about arithmetic.'

好吧。所以我们现在走上了这条消极的道路。它会很快下滑。我们迟早会撞上这堵墙。它会消灭我们吗？它会说「我受够了这些东西，我要把它们都干掉」吗？

All right. So, we're now on this negative trip. It will slide fast now. We got to hit this wall at some point or another. Will it wipe us out? Will it say, 'I've had enough of these things. I'll get rid of them all.'

好的。所以我想再举一个物理类比。当你夜间开车时，你利用前车的尾灯。如果前车距离加倍，尾灯的光亮只有四分之一。平方反比定律。没错。所以你能相当清楚地看到一辆车。你假设如果它距离加倍，你仍然能看到它。但如果你在雾中开车，情况完全不同。雾是指数级的。每单位距离，它会消除一定比例的光。你可以有一辆车在 100 码外清晰可见，而另一辆车在 200 码外完全看不见。这就是为什么雾在特定距离看起来像一堵墙。好吧，如果事情是指数级改进的，你在预测未来时也会遇到同样的问题。你面对的是指数，但你用线性或二次函数来近似它。

Okay. So, I want another physics analogy. When you're driving at night, you use the tail lights of the car in front. And if the car gets twice as far away, the tail lights give you a quarter as much light. The inverse square law. That's right. So you can see a car fairly clearly. And you assume that if it was twice as far away, you'd still be able to see it. If you're driving in fog, it's not like that at all. Fog is exponential. Per unit distance, it gets rid of a certain fraction of the light. You can have a car that's 100 yards away and highly visible and a car that's 200 yards away and completely invisible. That's why fog looks like a wall at a certain distance. Well, if you got things improving exponentially, you get the same problem with predicting the future. You're dealing with an exponential, but you're approximating it with something linear or quadratic.

所以，在夜间是二次的，对吧？如果你这样近似一个指数，你会发现，对于几年后能预测什么，你能做出正确的预测，但十年后，你完全没希望。你根本不知道会发生什么。

So, at night is quadratic, right? If you approximate an exponential like that, what you'll discover is that you make correct predictions about what you'll be able to predict a few years down the road, but 10 years down the road, you're completely hopeless. You just have no idea what's going to happen.

是的。对。对。就像在雾里扔飞镖。

Yeah. Right. Right. Yeah. You're throwing darts in the fog.

我们不知道会发生什么。雾很深。

We have no idea what's going to happen. It's deep in the fog.

哇。

Wow.

但我们应该认真思考这个问题。

But we should be thinking hard about it.

你需要相信它会继续指数级增长。

You need the confidence that it will continue to grow exponentially.

确实如此。但让我说得更糟一点。

There is that. But let me make it worse.

请。请继续。请说得更糟。

Please. Please go ahead. Please make it worse.

假设它只是线性的。那么，如果你想知道十年后会是什么样子，你会回顾十年前，说：「我们当时对现在的预测有多离谱？」

Suppose it was just linear. So then what you do if you want to know what it's going to be like in 10 years time, you look back 10 years and say, 'How wrong were we about what it would be like now?'

哇。

Wow.

嗯，十年前，没有人会预测到。即使像我这样真正的爱好者，认为它最终会到来，也不会预测到我们现在会有一个模型，你可以问它任何问题，它会以不太好的专家水平回答，偶尔还会撒谎。这就是我们现在拥有的。而十年前你不会预测到这一点。

Well, 10 years ago, nobody would have predicted. Even real enthusiasts like me who thought it was coming in the end, they wouldn't have predicted that at this point we'd have a model where you could ask it any question and it would answer at the level of a not very good expert who occasionally tells fibs. And that's what we've got now. And you wouldn't have predicted that 10 years ago.

那么幻觉在这里面扮演什么角色？我的感觉是它们不是故意的。只是系统出错了。

So where do hallucinations fit into this? My sense was that they were not on purpose. It's just that the system is messing up.

好吧，它们不应该被称为幻觉。对于语言模型，应该称为虚构。

Okay, they shouldn't be called hallucinations. They should be called confabulations if it's with language models.

虚构。我喜欢这个词。更广为人知的是谎言。

Confabulations. I love it. Better known as lies.

你刚给了尼尔今天的单词。

You've just given Neil word of the day.

心理学家至少从 20 世纪 30 年代就开始研究人类的虚构。人们一直在虚构。至少我认为是这样。我刚编的。嗯，所以如果你记得最近发生的事，并不是大脑里像文件柜或电脑内存那样存储了一个文件。而是最近的事件改变了你的连接强度，现在你可以利用这些连接强度构建出与几小时或几天前发生的事非常相似的东西。但如果我让你回忆几年前发生的事，你会构建出对你来说似乎非常合理的东西，有些细节是对的，有些是错的，你对正确细节的信心可能并不比对错误细节的信心更大。

Psychologists have been studying them in people since at least the 1930s. And people confabulate all the time. At least I think they do. I just made that up. Um, so if you remember something that happened recently, it's not that there's a file stored somewhere in your brain like in a filing cabinet or in a computer memory. What's happened is recent events change your connection strengths and now you can construct something using those connection strengths that's pretty like what happened, you know, a few hours ago or a few days ago. But if I ask you to remember something that happened a few years ago, you'll construct something that seems very plausible to you and some of the details will be right and some will be wrong and you may not be any more confident about the details that are right than about the ones that are wrong.

嗯。

Mhm.

现在，这通常很难看出来，因为你不知道真实情况，但有一个案例你知道真实情况。在水门事件中，约翰·迪恩在宣誓后作证，关于白宫椭圆形办公室的会议，他作证说谁在场、谁说了什么，但他很多地方都错了。他当时不知道有录音带，但他没有撒谎。他是在根据自己在椭圆形办公室会议中的经历，编造出对他来说非常合理的故事。

Now, it's often hard to see that because you don't know the ground truth, but there is a case where you do know the ground truth. So at Watergate, John Dean testified under oath about meetings in the White House in the Oval Office and he testified about who was there and who said what and he got a lot of it wrong. He didn't know at the time there were tapes, but he wasn't fibbing. What he was doing was making up stories that were very plausible to him given his experiences in those meetings in the Oval Office.

嗯。所以他传达的是掩盖的某种真相，但他会把陈述归给错误的人。他会说一些不在场的人参加了会议。乌尔里克·奈塞尔对此有一个很好的研究。所以很明显，他只是编造听起来合理的东西。这就是记忆。如果时间久远，很多细节都是错的。聊天机器人也在做同样的事。聊天机器人不存储字符串。它们不存储特定事件。它们做的是在你问起时编造出来，而且经常像人一样搞错细节。所以它们会虚构这一事实让它们更像人，而不是不像人。

Mhm. And so he was conveying the sort of truth of the cover up, but he would attribute statements to the wrong people. He would say people were in meetings who weren't there. And there's a very good study of that by someone called Ulric Neisser. So it's clear that he just makes up what sounds plausible to him. That's what a memory is. And a lot of the details are wrong if it's from a long time ago. That's what chatbots are doing, too. The chatbots don't store strings of words. They don't store particular events. What they do is they make them up when you ask them about them and they often get details wrong just like people. So the fact that they confabulate makes them much more like people, not less like people.

所以我们创造了人工愚蠢，以及……

So we created artificial stupidity as well as...

是的。我们至少创造了一些人工过度自信。

Yeah. We've created some artificial overconfidence at least.

是的，那可能是一个……

Yeah, that might be a...

好处是什么？人工智能的潜在真正好处是什么？

What's the upside? What are the potential real benefits of artificial intelligence?

哦，这就是它和核武器之类的东西不同的地方。它有巨大的好处，而原子弹之类的东西好处不大。他们确实尝试在科罗拉多州用它们进行水力压裂，但效果不太好，而且你不能再去了。但基本上，原子弹只是为了摧毁东西。

Oh, that's how it differs from things like nuclear weapons. It's got a huge upside with things like atom bombs. There wasn't much upside. They did try using them for fracking in Colorado, but that didn't work out so well and you can't go there anymore. But basically, atom bombs are just for destroying things.

是的。

Yeah.

所以，人工智能有巨大的好处，这也是我们开发它的原因。在医疗保健等领域，它将非常棒，意味着每个人都能在北美得到非常好的诊断。实际上，我不确定这是美国还是美国加加拿大，因为我们过去只考虑北美，但现在加拿大不想成为其中一部分。第 51 个州。

So, with AI, it's got a huge upside, which is why we developed it. It's going to be wonderful in things like healthcare where it's going to mean everybody can get really good diagnosis in North America. Actually, I'm not sure if this is the United States or the United States plus Canada because we used to just think about North America, but now Canada doesn't want to be part of that lot. The 51st state.

嗯。

Mhm.

在北美，每年大约有 20 万人因为医生误诊而死亡。

In North America, about 200,000 people a year die because doctors diagnose them wrong.

对。是的。人工智能在诊断方面已经比医生更好了。特别是如果你拿一个人工智能，复制几份，让它们扮演不同角色并相互交谈。

Right. Yes. AI is already better than doctors at diagnosis. Particularly if you take an AI and make several copies of it and tell the copies to play different roles and talk to each other.

哇。这就是微软所做的。微软有一篇很好的博客显示，这实际上比大多数医生做得更好。

Wow. That's what Microsoft did. There's a nice blog by Microsoft showing that that actually does better than most doctors.

那是……顺便说一句，所以你做的就是同时拥有第一、第二、第三和第四意见。

That is... and by the way, so what you have done is you have a first, second, third, and fourth opinion all at once.

是的。对，你做的就是这些。

Yes. Yeah, that's all you're doing.

嗯，不，因为它们在扮演不同的角色。

Well, no, the because they're playing different roles.

是的，它们在扮演不同的角色。是的，那太棒了。

Yeah, they're playing different roles. Yeah, that's fantastic.

是的，太棒了。

Yes, it is fantastic.

你可以创建一个人工智能委员会。

You can create an AI committee.

是的，太棒了。

Yeah, it's wonderful.

太棒了。

That's brilliant.

AI 可以设计很棒的新药。是的，我们有 Alpha 团队在这里。它还能做很多小事。比如在任何医院，他们必须决定何时让病人出院。如果出院太早，病人会死亡或再次入院。所以你必须等到他们足够好才能出院。但如果出院太晚，你就浪费了本可以用来接收其他急需入院病人的床位。那里有大量数据。AI 在决定何时适合让病人出院方面可以比人做得更好。还有无数类似的应用。还有记录保存，这是任何医院网络或医生集团中非常重要的一部分。每个病人必须有大量的记录，AI 可以摄取并处理这些记录。

AI can design great new drugs. Yeah, we have the Alpha team on here. There's lots of little minor things it can do. Like in any hospital, they have to decide when to discharge people. If you discharge them too soon, they die or they come back. So you have to wait until they're good enough to be discharged. But if you discharge them too late, you're wasting a hospital bed that could be used to admit somebody else who's desperate to be admitted. And there's lots and lots of data there. An AI can just do a better job than people can at deciding when it's appropriate to discharge somebody. And there's a gazillion applications like that. And recordkeeping, which is a very, very big part of any hospital network, any doctor group. There has to be copious amounts of records on every single patient that AI can just ingest and process.

AI 有没有可能被引导去解决社会当前面临的重大问题？比如气候变化，或者其他事情，能源、住房、无家可归者。

Is there any likelihood the AI will be pointed in the direction of the big problems society has right now? Maybe climate change, maybe other things, energy, housing, homelessness.

绝对会。绝对会。比如对于气候变化之类的事情，AI 已经擅长提出新材料、新合金等。

Absolutely. Absolutely. So for things like climate change for example, AI is already good at suggesting new materials, new alloys, things like that.

绝对。是的。我怀疑 AI 会非常擅长制造更高效的太阳能电池板，并帮助你更好地找出如何在水泥厂或发电厂排放二氧化碳的瞬间吸收它。

Absolutely. Yeah. I suspect that AI is going to be very good at making more efficient solar panels and making you better at figuring out how to absorb carbon dioxide at the moment it's emitted by cement factories or power plants.

信不信由你，AI 已经就气候变化告诉我们，你们这些笨蛋应该停止燃烧并将碳排入大气。这就是那些话，这是 AI 的原话。它说「嘿，笨蛋，别再往大气中排放碳了。」不，但我们早就知道了。所以关于气候变化，悲剧在于我们知道如何阻止它。你只要停止燃烧碳。只是我们没有政治意愿。我们有像默多克这样的人，他的报纸说「不，气候变化没问题。」

And believe it or not, AI already told us with respect to climate change that you dumb asses should stop burning and putting carbon in the atmosphere. That's what those are, that's an exact quote from AI. It was like 'hey dumbass, stop putting carbon in the atmosphere.' No, but we already knew that. So the thing about climate change is the tragedy of climate change is we know how to stop it. You just stop burning carbon. It's just we don't have the political will. We have people like Murdoch whose newspapers say, 'Nah, there's no problem with climate change.'

那么现在我们谈到能源问题，数据中心正在建设，像蘑菇一样涌现。我们真的能负担得起运行人工智能的能源成本吗？

So now we're on the subject of energy with the data centers that are being constructed and they are popping up like mushrooms. Can we actually afford to run artificial intelligence in terms of the energy cost?

你可以这样做。我有解决方案。你告诉 AI，「我们需要更多你，但你正在耗尽我们所有的资源，我们的能源资源。所以想办法高效地做到这一点。然后我们可以制造更多你，然后我们一夜之间就能解决。」

Here's what you do. I got the solution. You tell AI, 'We want more of you, but you're using up all our resources, our energy resources. So figure out how to do that efficiently. Then we can make more of you, and then we'll figure it out overnight.'

是的，干脆把我们干掉。你打开了大门。那么杰弗里，为什么不直接让它，让我们递归一下。AI，你想要更多自己？解决这个我们卑微人类无法解决的问题。

Yeah, just get rid of us. You opened the door. So Jeffrey, why not just give it, let's get recursive about it. AI, you want more of yourself? Fix this problem that we can't otherwise solve as lowly humans.

这被称为奇点，即让 AI 开发更好的 AI。在这种情况下，你要求它创造更节能的 AI。但许多人认为这将是一个失控的过程。

This is called the singularity, when you get AIs to develop better AIs. In this case, you're asking it to create more energy efficient AIs. But many people think that will be a runaway process.

哦，那会有什么坏处？

Oh, in what way would that be bad?

它们会很快变得非常聪明。没人知道那会发生。但这是一个担忧。

That they will get much smarter very fast. Nobody knows that that will happen. But that's one worry.

那不是已经发生了吗？不。

Isn't that already happening now? No.

在某种程度上，是的，它已经开始发生了。所以有一个我以前合作过的研究员去年告诉我，他们有一个系统，在解决问题时会观察自己在做什么，并找出如何改变自己的代码，以便下次遇到类似问题时能更高效地解决。这已经是奇点的开始。

To a certain extent, yes, it's beginning to happen. So I had a researcher I used to work with who told me last year that they have a system that when it's solving a problem is looking at what it itself is doing and figuring out how to change its own code so that next time it gets a similar problem it'll be more efficient at solving it. That's already the beginning of the singularity.

所以如果它自己写代码，那就失控了。

So if it writes its own code it's off the chain.

失控了。

Off the chain.

哦，是的。对吗？它可以重写自己。

Oh yeah. Is that right? It can rewrite itself.

是的。它们可以自己写代码。是的。

Yeah. They can write their own code. Yes.

有什么阻止它们用代码复制自己？

What's stopping them replicating themselves with code?

没有。

Nothing.

这就是我的答案。杰弗里，我们完了。就在那边。告诉过你还有一次恐慌发作。杰克，老兄，结束了。

There's my answer. Jeffrey, we're done. It's over there. Told you there was another panic attack. Jack, it's over, man.

它们必须访问计算机才能复制自己。而人类仍然掌控着这一点。但原则上，一旦它们控制了数据中心，它们就可以随心所欲地复制自己。

They have to get access to the computers to replicate themselves. And people are still in charge of that. But in principle, once they've got control of the data centers, they can replicate themselves as much as they like.

好的。我还有另一个问题。我在五角大楼的一个委员会任职了大约七年，那时 AI 正显现为一种可能的战争工具。我们引入了在军队可能遇到的情况下使用 AI 的指导方针。其中之一是，如果 AI 决定它可以或应该采取导致敌人死亡的行动，我们是否应该给它这样做的权限，这仍然是一个大辩论，还是我们应该始终确保有一个人在其中？

Okay. I got another question. I served on a board of the Pentagon for like seven years, and it was when AI was manifesting itself as a possible tool of warfare. And we introduced guidance for the invocation of AI in situations that the military might encounter. One of which was if AI decides that it can or should take action that will end in death of the enemy, should we give it that access to do so or still a big debate, or should we always ensure that there's a human inside that loop?

这是一个大辩论。

It's a big debate.

好的，所以我们说必须这样，如果 AI 不能自己做决定杀人，对吧？必须有一个人在里面。我的问题是，杰弗里，如果有其他国家没有设置这样的保障，那么敌人就会拥有时间上的优势。

Okay, so we said there's got to be, if AI cannot make its own decision to kill, right? A human has to be in there. My question to you is Jeffrey, if there are other nations who put in no such safeguards, then that is a timing advantage that an enemy would have over you.

正确。

Correct.

然后我们在循环中多了一步，他们没有。

And then we have one more step in the loop that they don't.

绝对。但我认为美国军方并不承诺在每个杀人决定中都有一个人参与。他们说的是总会有人的监督，对吧？但在激烈的战斗中，你有一架无人机对抗一辆俄罗斯坦克，你没有时间让人类说「无人机可以向这个士兵投掷手榴弹吗？」所以，我的怀疑是，如果你提出建议，美国军方应该总是有一个人。

Absolutely. But my belief is that the US military isn't committed to always having a human involved in each decision to kill. What they say is there will always be human oversight, right? But in the heat of battle, you've got a drone that's going up against a Russian tank, and you don't have time for a human to say, 'Is it okay for the drone to drop a grenade on this soldier?' So, my suspicion is the US military, if you made the recommendation, there should always be a person.

嗯，那大约是八年前了。是的。我认为他们不再坚持那个了。我认为他们说的是总会有人的监督，这是一个更模糊的事情。

Well, that was like eight years ago. Yeah. I don't think they stand by that anymore. I think what they say is there'll always be human oversight, which is a much vaguer thing.

好的。所以，人类问责制。

All right. So, human accountability.

关于战争，是否有可能在制定护栏和决策中的人为因素方面进行国际合作，还是这只是狂野西部？

On the subject of war, is there likely to be international cooperation on development of guardrails and a human factor in decision-making, or is this just wild west?

好吧，如果你问人们什么时候合作，人们会在利益一致时合作。所以在冷战高峰期，美国和苏联合作避免全球热核战争，因为这不符合任何一方的利益。他们的利益是一致的。所以如果你看看 AI 的风险，有使用 AI 通过假视频破坏选举。各国的利益是反一致的。它们都在互相这样做，对吧？还有网络攻击。它们的利益基本上是反一致的。还有恐怖分子制造病毒，它们的利益可能是一致的。

Okay, if you ask when do people cooperate, people cooperate when their interests are aligned. So at the height of the cold war, the USA and the USSR cooperated on not having a global thermonuclear war because it wasn't in either of their interests. Their interests were aligned. So if you look at the risks of AI, there's using AI to corrupt elections with fake videos. The country's interests are anti-aligned. They're all doing it to each other, right? There's cyber attacks. Their interests are basically anti-aligned. There's terrorists creating viruses where their interests are probably aligned.

所以他们可能会在那里合作。还有一件事他们的利益肯定是一致的，他们会合作，那就是防止人工智能取代人类。如果中国人找到了如何防止人工智能想要接管、想要从人类手中夺走控制权的方法，他们会立即告诉美国人，因为他们也不希望人工智能在美国夺走人类的控制权。在这件事上，我们都在同一条船上。

So they might cooperate there. And then there's one thing where their interests are definitely aligned and they will cooperate which is preventing AI from taking over from people. If the Chinese figured out how you could prevent AI from ever wanting to take over, from ever wanting to take control away from people, they would immediately tell the Americans because they don't want AI taking control away from people in America either. We're all in the same boat when it comes to that.

这是人工智能版的核冬天。

This is the AI version of nuclear winter.

是的。

Yes.

在我看来

It seems to me

就是这样。完全一样。他们会合作试图避免这种情况。

It is. It's exactly that. They will cooperate to try and avoid that.

因为核冬天，为了提醒大家，这个想法是如果发生全面核交换，你会烧毁森林和土地等等。烟尘进入大气层，阻挡阳光，所有生命都会死亡。

Because in nuclear winter, just to refresh people's memory, the idea was if there's total nuclear exchange, you incinerate forests and land and what have you. The soot gets into the atmosphere, blocks sunlight, and all life dies.

所以没有赢家。

So there is no winner.

当然。

Of course.

在全面核武器交换中。

In a total exchange of nuclear weapons.

相互确保摧毁。

Mutually assured destruction.

是啊。那谁会想要呢？

Yeah. And so who wants that?

除非你是个疯子之类的，这种人确实存在。也许我觉得蟑螂会赢。

Unless you're a madman or something, they exist. Maybe I think maybe the cockroaches win.

它们赢了。

They win.

哦，是啊。那怎么样？

Oh, yeah. Well, how about that?

是啊。这没有考虑一个可能属于死亡崇拜的领导者。

Yeah. This doesn't factor in a possible leader who is in a death cult.

可以说是尼禄。

A Nero, so to speak.

是啊。如果我说我不介意所有人都死，因为我要去死亡中的这个地方，我所有的追随者都和我一起在这个邪教中。所以这让你描述的这个一致的愿景声明变得复杂。确实让它复杂了很多。而且我发现非常令人安慰的是，很明显特朗普实际上并不相信上帝。

Yeah. If I say I don't mind if everybody dies because I'm going to this place in death and all my followers are coming with me in this cult. So that complicates this aligned vision statement that you're describing. It does complicate it a lot. And I find it very comforting that it's obvious that Trump doesn't actually believe in God.

哦，让我用史蒂文·温伯格的一句话来补充。

Oh, let me follow that up with a quote from Steven Weinberg.

好的。

Okay.

你知道这句话吗，杰弗里？

Do you know this quote, Jeffrey?

不知道。

No.

史蒂文·温伯格：「世界上总会有好人和坏人。但要让一个好人做坏事，就需要宗教。」

Steven Weinberg: 'There will always be good people and bad people in the world. But to get a good person to do something bad requires religion.'

那是因为他们以宗教的名义行事。你可以以任何名义做。

That's because they're doing it in the name of religion. You could do it in the name of anything.

我认为我们现在需要认识到我们有一种宗教。我们称之为科学。它确实与其他宗教不同。不同之处在于它是正确的。

I think we need to recognize at this point that we have a religion. We call it science. Now it does differ from the other religions. And the way it differs is it's right.

话筒掉了。

Mic drop.

好的。我认为我们应该给杰弗里·辛顿图灵奖，我会因为他在这里的贡献给他诺贝尔奖？

Okay. I think we got to give Jeffrey Hinton the Turing Prize and I would give him a Nobel Prize for what he's contributed here?

嗯，和他另一个一起。

Well, to go with his other one.

不。我喜欢耳环。

No. I like earrings.

我一开始没提，先生。2018 年，你获得了图灵奖。这是一个非常令人垂涎的计算机科学奖。没错。而图灵，我们在节目开头提到过他。所以，首先祝贺你。然后这还不够。

I left that out at the beginning, sir. In 2018, you won the Turing Prize. This is a highly coveted computer science prize. Correct. And Turing, we mentioned him at the top of the show. So, first congratulations on that. And then that wasn't enough.

好的。诺贝尔委员会屈尊颁发了诺贝尔奖。

Okay. The Nobel Committee slumming with the Nobel.

是啊。所以诺贝尔委员会说，这些由杰弗里几十年前的工作孕育的人工智能的东西对这个世界正在发生的事情如此重要。我们必须给这个人诺贝尔奖，他获得了 2024 年诺贝尔物理学奖。

Yeah. So the Nobel Committee said this AI stuff that was birthed by Jeffrey's work from decades ago is so fundamental to what's going on in this world. We got to give this man a Nobel Prize and he earned the Nobel Prize in Physics 2024.

只是稍微纠正一下，有很多人孕育了人工智能。特别是，反向传播算法是由大卫·鲁梅尔哈特重新发明的，他得了严重的脑部疾病，英年早逝，但他没有得到足够的认可。哦，好的。谢谢你指出这一点。另外，诺贝尔委员会不会把诺贝尔奖颁发给已经去世的人。

Just a little correction, there are a whole bunch of people who birthed AI. In particular, the backpropagation algorithm was reinvented by David Rumelhart who got a nasty brain disease and died young, but he doesn't get enough credit. Oh, okay. Thanks for calling that out. Plus, the Nobel Committee does not offer a Nobel Prize to you if you're already dead.

所以没有奖项。你必须在他们宣布时还活着。

So there's no award. You have to be alive when they announce it.

嗯，如果你在宣布和颁奖之间去世，你还能得到，但如果不是的话。所以，总之，恭喜你。我不是想在我们的播客上吹嘘，但你大概是我们采访过的第五位诺贝尔奖得主了。

Well, you can get it if you died between when they announced it and the ceremony, but not if. So, anyway, congratulations on that. And I don't mean to brag on our podcast, but you're like the fifth Nobel laureate we've interviewed.

不止。

More than that.

是啊。我想我们有。是啊，我不是想在我们的播客上吹嘘。是啊，就这些。

Yeah. I think we have. Yeah, I don't mean to brag on our podcast. Yeah, that's all.

不过那很酷。

That's cool, though.

很酷。继续。好的。

That's cool. Go. Okay.

我有一个后续问题。我们已经谈到了世界末日的场景，目前，希望这个场景不会发生，因为我们人类天生具有竞争性，特别是在美国，谁在人工智能竞赛中领先，谁最有可能首先冲过终点线获得大奖？

I have a follow-up question. We've got into the apocalyptic scenario and at the moment, hopefully, it's a scenario that doesn't play out because we are competitive by nature as humans and particularly here in the US, who is leading the race in artificial intelligence and who is likely to cross the finish line first when it comes to the prize?

如果我必须押注一群人，那可能是德国、谷歌。但我曾为谷歌工作，所以别太当真。我在他们赢这件事上有既得利益。Anthropic 可能会赢，OpenAI 可能会赢。我认为微软赢的可能性较小，或者 Facebook 赢的可能性较小。

If I had to bet on one lot of people, it would probably be Germany, Google. But I used to work for Google, so don't take me too seriously about that. I have a vested interest in them winning. Anthropic might win, OpenAI might win. I think it's less likely that Microsoft will win or that Facebook will win.

嗯，我们知道不会是 Facebook。你怎么知道的？

Well, we know it won't be Facebook. Why do you know that?

我的意思是，看看谁在运营 Facebook。好吧，拜托。

I mean, let's look at who's running Facebook. Okay, come on.

不，不是谁在运营。是谁有资源找到合适的人来做这项工作。

No, it's not who's running it. It's who has the resources to get the right people to do the work.

好吧，杰弗里，后续问题是，无论谁先冲过终点线，他们的奖品是什么？他们抢先到达会得到什么奖励？

All right, Jeffrey, the follow up on that is whoever crosses the line first, what is their prize? What will be the reward for them getting there before?

等等，先退一步。告诉我去年股市的价值。好的。我的看法是，仅从媒体上读到，美国股市价值增长的 80%可以归因于大型人工智能公司的价值增长。

Wait, back up for a sec. Tell me about the value of the stock market in the last year. Okay. And my belief is just from reading it in the media that 80% of the increase of the value in the stock market, the US stock market can be attributed to the increase in value of the big AI companies.

没错。

True.

80%的增长。

80% of the growth.

是的。

Yes.

有人认为是泡沫吗？这差不多就是他们所说的，人工智能泡沫。

Anyone thinking bubble? And that's kind of what they're calling it, the AI bubble.

好的。

Okay.

问题是这样。泡沫有两种含义。一种含义是，结果人工智能并没有像人们想象的那样有效。

The issue is this. There's two senses of bubble. One sense of bubble is it turns out AI doesn't really work as well as people thought it might.

对吧？

Right?

它实际上并没有发展出取代所有人类智力劳动的能力，而大多数开发它的人相信最终会发生这种情况。

It doesn't actually develop the ability to replace all human intellectual labor which is what most people developing it believe is going to happen in the end.

那肯定是恐惧因素。

That was the fear factor for sure.

是啊。

Yeah.

泡沫的另一种含义是公司无法从投资中收回资金。现在这似乎更可能是那种泡沫，因为据我所知，公司都假设如果我们能先到那里，我们可以向人们出售将取代许多工作的人工智能。当然，人们会为此支付很多钱。所以，我们会赚很多钱。但他们没有考虑社会后果。如果他们真的取代了许多工作，社会后果将是可怕的。

The other sense of bubble is the companies can't get their money back from the investments. Now that seems to be more likely kind of bubble because as far as I understand it, the companies are all assuming if we can get there first, we can sell people AI that will replace a lot of jobs. And of course, people will pay a lot of money for that. So, we'll get lots of money. But they haven't thought about the social consequences. If they really do replace lots of jobs, the social consequences will be terrible.

正确。

Correct.

完全正确。然而，他们取代了工作，现在你仍然想卖你的产品，但没有人有收入来购买产品。

Totally. However, they replace the jobs and now you still want to sell your product and no one has income to buy the product.

是啊。这是一条自我限制的道路。

Yeah. It's a self-limiting path.

那是凯恩斯主义的观点。然后还有一个观点是会出现高失业率，这将导致大量社会动荡。

That's the Keynesian view of it. And then the additional view is that there'll be high unemployment levels which will lead to a lot of social unrest.

所以对此的次要观点是，我们的社会只存在两个阶层：第一阶层是所有从 AI 中受益的人，第二阶层是那些因 AI 而被迫生活的封建农民。

So the secondary view of that is you just have two tiers of existence for our societies. The first tier is all the people who are benefiting from AI, and the second tier are the feudal peasants that are now forced to live their lives because of AI.

让我问你一个非 AI 问题，因为你是这个领域的深刻思考者。在自动化初期，每个人都这么说：所有人都会失业，没有工作留下，社会将走向毁灭。然而社会因其他需求和事物而扩展。这就是为什么我们 90%的人不再是农民。我们有机器来做那些事，我们发明了其他东西，比如度假胜地。

Let me ask you a non-AI question because you're a deep thinker in this space. That's what everybody said in the dawn of automation. Everyone will be unemployed. There'll be no jobs left and society will go to ruin. Yet society expanded with other needs and other things. That's why 90% of us are no longer farmers. We have machines to do that and we invent other things like vacation resorts.

但这次只需要一小部分时间。

But this time it's going to take a fraction of the time.

是这样吗，Geoffrey？问题在于我们可能以社会无法从人们失业速度中恢复的速度创造一个失业阶层吗？

Is that so, Geoffrey? Is the problem here the rapidity with which we may create an unemployed class where society cannot recover from the rate at which people are losing their jobs?

这当然是问题的一个重要方面。但还有另一个方面：如果你用拖拉机取代体力劳动，现在需要的人就少得多。其他人可以去做智力工作。但如果你取代了人类智能，他们该去哪里？当 AI 能以更低成本、更好质量完成工作时，呼叫中心的工作人员该去哪里？

That certainly is one big aspect of the problem. But there's another aspect: if you use a tractor to replace physical labor, you need far fewer people now. Other people can go off and do intellectual things. But if you replace human intelligence, where are they going to go? Where are people who work in a call center going to go when an AI can do their job cheaper and better?

对。是的。

Right. Yeah.

所以没有其他事情可做。无论你开辟什么领域，AI 都能做。

So there's not another thing. Whatever thing you open, AI can do.

对吧？

Right?

你可以用一种有趣的方式看待人类历史，即消除限制。很久以前，我们有一个限制：必须担心下一顿饭从哪里来。农业消除了那个限制。它带来了很多其他问题，但消除了那个特定的担忧。然后我们有了无法远行的限制。自行车对此帮助很大，还有汽车和飞机。我们克服了那种限制。很长一段时间里，我们有一个限制：我们必须自己思考。我们即将克服那个限制。一旦你克服了所有限制，会发生什么还不清楚。像 Sam Altman 这样的人认为那会很美好。

You can look at human history in an interesting way as getting rid of limitations. A long time ago, we had the limitation you had to worry about where your next meal was coming from. Agriculture got rid of that. It introduced a lot of other problems, but it got rid of that particular worry. Then we had the limitation you couldn't travel very far. The bicycle helped a lot with that, and cars and airplanes. We got over that kind of limitation. For a long time, we had the limitation that we were the ones who had to do the thinking. We're just about to get over that limitation. And it's not clear what happens once you get over all the limitations. People like Sam Altman think it'll be wonderful.

所以我们会成为 AI 的宠物。

So we'll become AI's pet.

嗯，不。很多人认为这场运动多年前就开始了，为了全民基本收入。

Well, no. A lot of people believe that this movement started years ago for universal basic income.

那么 Geoffrey，你会说随着 AI 获得力量，全民基本收入这个想法的象征性价值在增长吗？

So would you say, Geoffrey, that the universal basic income, the figurative stock value in that idea, is growing as AI gains power?

它看起来越来越必要，但有很多问题。一个问题是很多人从工作中获得自我价值感，而它无法解决尊严问题。另一个问题是税基。如果你用 AI 取代工人，政府就失去了税基。它必须以某种方式能够对 AI 征税。但大公司不会喜欢那样。

It's becoming to seem more essential, but it has lots of problems. One problem is many people get their sense of self-worth from the job they do, and it won't deal with the dignity issue. Another problem is the tax base. If you replace workers with AIs, the government loses its tax base. It has to somehow be able to tax the AIs. But the big companies aren't going to like that.

我认为我们应该让 AI 来解决这个问题。

I think we should let AI figure out this problem.

没错。

That's right.

那么 Geoffrey，很多人，尤其是科幻作家，区分了机器的力量和智力，以及它们变得有意识时的跨越。这在《终结者》系列中是一个重要时刻。那就是《终结者》中的奇点，当天网有足够的神经连接或某种连接使其获得意识时。所以似乎，如果你作为认知心理学家来看待这个问题，我很好奇你怎么想。我们是否可以假设，在任何神经网络中，无论是真实的还是人工的，只要有足够的复杂性，就会涌现出诸如意识之类的东西？

So Geoffrey, many people, especially sci-fi writers, distinguish between the power and intellect of machines and the crossover when they become conscious. That was a big moment in the Terminator series. That was the singularity in the Terminator when Skynet had enough neural connections or whatever kind of connections made it so that it achieved consciousness. So there seems to be, and if you come to this as a cognitive psychologist, I'm curious how you think about this. Are we allowed to presume that given sufficient complexity in any neural net, be it real or artificial, something such as consciousness emerges?

所以这里的问题其实不是科学问题。而是我们文化中的大多数人有一套关于心智如何运作的理论，他们认为意识是某种涌现的本质。我认为意识就像燃素。它是一种用来解释事物的本质，一旦我们理解了那些事物，我们就不会再用那个本质去解释它们。我想试着说服你，一个多模态聊天机器人已经拥有主观体验。人们使用「感知」或「意识」或「主观体验」这些词。我们先聚焦于主观体验。我们文化中的大多数人认为心智的运作方式是一种内部剧场。当你进行感知时，世界出现在这个内部剧场中，只有你能看到那里有什么。所以如果我对你说，如果我喝了很多酒，我对你说，「我有小粉象在我面前漂浮的主观体验」，大多数人会解释为有一个内部剧场，我的心智，我能看到里面的东西，里面是小粉象，它们不是由真正的粉色和大象构成的。所以它们一定是由别的东西构成的。因此哲学家发明了「感受质」，这有点像认知科学中的燃素。他们说它们一定是由感受质构成的。让我给你一个完全不同的观点，那是丹尼尔·丹尼特的观点，他是一位伟大的认知科学哲学家——已故的伟大哲学家。那种关于心智的观点是完全错误的。所以我现在要说出同样的事情，就像我告诉你我有小粉象的主观体验，但不使用「主观体验」这个词，也不诉诸感受质。我一开始说我相信我的感知系统在欺骗我。那是其中的主观部分。但如果我的感知系统没有欺骗我，那么世界上就会有小粉象在我面前漂浮。所以这些小粉象的有趣之处不在于它们由感受质构成且存在于内部剧场中，而在于它们是假设性的。它们是一种技巧，让我通过告诉你如果我的感知系统在说实话会有什么，来告诉你我的感知系统在如何撒谎。现在我要对一个聊天机器人做同样的事。我拿一个多模态聊天机器人。我训练它。它有一个摄像头。它有一个机械臂。它能说话。我在它面前放一个物体，说「指向那个物体」，它指向那个物体。然后我搞乱它的感知系统。我在摄像头前放一个棱镜。现在我在它面前放一个物体，说「指向那个物体」，它指向一边。我对它说，「不，物体不在那里。它实际上在你正前方。但我在你的镜头前放了一个棱镜。」聊天机器人说，「哦，我明白了。棱镜弯曲了光线，所以物体实际上在我正前方。但我有主观体验认为它在一边。」现在，如果聊天机器人那样说，它使用「主观体验」这个词的方式就和我们一样。

So the problem here is not really a scientific problem. It's that most people in our culture have a theory of how the mind works and they have a view of consciousness as some kind of essence that emerges. I think consciousness is like phlogiston maybe. It's an essence that's designed to explain things, and once we understand those things, we won't be trying to use that essence to explain them. I want to try and convince you that a multimodal chatbot already has subjective experience. So people use the word sentience or consciousness or subjective experience. Let's focus on subjective experience for now. Most people in our culture think that the way the mind works is it's a kind of internal theater. And when you're doing perception, the world shows up in this internal theater and only you can see what's there. So if I say to you, if I drink a lot and I say to you, 'I have the subjective experience of little pink elephants floating in front of me,' most people interpret that as there's this inner theater, my mind, and I can see what's in it, and what's in it is little pink elephants, and they're not made of real pink and real elephants. So they must be made of something else. So philosophers invent qualia, which is kind of the phlogiston of cognitive science. They say they must be made of qualia. Let me give you a completely different view that is Daniel Dennett's view, who was a great philosopher of cognitive science—the late great philosopher. That view of the mind is just utterly wrong. So I'm now going to say the same thing as when I told you I had the subjective experience of little pink elephants without using the word subjective experience and without appealing to qualia. I start off by saying I believe my perceptual system is lying to me. That's the subjective bit of it. But if my perceptual system wasn't lying to me, there would be little pink elephants out there in the world floating in front of me. So what's funny about these little pink elephants is not that they're made of qualia and they're in an inner theater. It's that they're hypothetical. They're a technique for me telling you how my perceptual system is lying by telling you what would have to be there for my perceptual system to be telling the truth. And now I'm going to do it with a chatbot. I take a multimodal chatbot. I train it up. It's got a camera. It's got a robot arm. It can talk. I put an object in front of it and I say, 'Point at the object,' and it points at the object. Then I mess up its perceptual system. I put a prism in front of the camera. And now I put an object in front of it and say, 'Point at the object,' and it points off to one side. And I say to it, 'No, that's not where the object is. It's actually straight in front of you. But I put a prism in front of your lens.' And the chatbot says, 'Oh, I see. The prism bent the light rays, so the object is actually straight in front of me. But I had the subjective experience that it was off to one side.' Now, if the chatbot said that, it would be using words subjective experience exactly the way we use them.

所以那个聊天机器人就会拥有主观体验。

And so that chatbot would have just had a subjective experience.

那如果你先和聊天机器人出去喝酒，喝了很多尊尼获加蓝牌呢？

Now, what if you first went out drinking with the chatbot and you had a very significant amount of Johnny Walker Blue?

这极不可能。我会喝利弗罗格。

That's extremely improbable. I would have Leafrog.

哦，我明白了，你是行家。你喜欢利弗罗格的泥煤味。好吧，好样的。

Oh, I see you're an eye man. You like the peatyness of the leaf. Okay, good man.

所以如果我理解你刚才在这两个例子中分享的内容，你实际上对我们进行了一次意识图灵测试。你说人类会这样做，现在你的聊天机器人也这样做，而且本质上是相同的。所以如果你想说我们因为表现出这种行为而有意识，你就得说聊天机器人也有意识，并发明了某种神秘的流体来实现这一点。但可能整个意识概念只是对人们在刺激面前所采取行动的干扰。

So if I understand what you just shared with us in these two examples, you actually pulled a consciousness Turing test on us. You said a human would do this and now your chatbot does it and it's fundamentally the same. So if you want to say we're conscious for exhibiting that behavior, you're going to have to say the chatbot's conscious and inventing whatever mysterious fluid is making that happen. But it could be that the whole concept of consciousness is a distraction from just the actions that people take in the face of stimulus.

好的。注意，聊天机器人没有被称为意识的任何神秘本质或流体，但它和我们一样有主观体验。所以我认为整个意识是某种神奇本质的想法，即如果你足够复杂就会突然被赋予这种本质，这完全是胡说八道。

Okay. So notice that the chatbot doesn't have any mysterious essence or fluid called consciousness, but it has a subjective experience just like we do. So I think this whole idea of consciousness is some magic essence that you suddenly get indicted with if you're complicated enough is just nonsense.

是的，你说得对。我同意。我一直觉得意识是人们试图解释的东西，却不知道它是否以任何有形的方式存在，这就是为什么它总是难以描述，因为你不知道它是什么，例如。是的。但我认为存在觉知。如果你看看科学家在不进行哲学思考时说的话，有一篇可爱的论文，聊天机器人说：「现在，让我们彼此坦诚。你是在测试我吗？」科学家说：「聊天机器人意识到自己正在被测试。」所以，他们把觉知归因于聊天机器人。在日常对话中，你称之为意识。只有当你开始哲学思考，认为它是某种有趣的神秘本质时，你才会完全困惑。

Yeah, there you go. I agree. I've always felt that consciousness was something people are trying to explain without knowing if it really exists in any kind of tangible way, which is why it's always difficult to describe because you don't know what it is for example. Yes. Yes. But I think there is awareness. And if you look at what scientists say when they're not thinking philosophically, there's a lovely paper where the chatbot says, "Now, let's be honest with each other. Are you actually testing me?" And the scientists say, "The chatbot was aware it was being tested." So, they're attributing awareness to a chatbot. And in everyday conversation, you call that consciousness. It's only when you start thinking philosophically and thinking that it's some funny mysterious essence that you get all confused.

嗯，我得说这是一场引人入胜的对话，会让我一个月睡不着觉。

Well, I have to say that this has been a fascinating conversation that will cause me not to sleep for a month.

是啊，你会完成很多工作。

Yeah, you get plenty of work done.

那么，杰弗里，请以积极的口吻为我们总结。我们还有时间找出与人工智能幸福共存的方法，我们应该为此投入大量研究努力，因为如果我们能与之幸福共存，并解决当它让我们的工作变得更容易时出现的所有社会问题，那么这对人类来说可能是一件美妙的事情。

So, Geoffrey, take us out on a positive note, please. So, we still have time to figure out if there's a way we can coexist happily with AI and we should be putting a lot of research effort into that because if we can coexist happily with it and we can solve all the social problems that will arise when it makes all our jobs much easier then it can be a wonderful thing for people.

同意。好的，所以还是有希望的。

Agreed. Okay. So, so there is hope.

是的。最后一件事，因为你暗示过，这个奇点，即人工智能自我训练，从而每分钟呈指数级变得更聪明。很多人称之为奇点。当然，雷·库兹韦尔就是其中之一，他之前是《星谈》的嘉宾。是的，来过几次。那么，你对这个奇点有什么看法？它像其他人说的那样真实吗？它像其他人说的那样迫在眉睫吗？

Yes. And one last thing because you hinted at it, this point of singularity where AI trains on itself so that it exponentially gets smarter like by the minute. That's been called a singularity by many people. Of course, Ray Kurzweil among them who's been a guest on a previous episode of Stars. Yeah. A couple of times. Yeah. So, what is your sense of this singularity? Is it real the way others say? Is it imminent the way others say?

这两个问题的答案我都不知道。我的猜测是，人工智能最终会在所有事情上比我们做得更好，但会是一个一个领域地超越。目前它在国际象棋和围棋上比我们好得多。它在知道很多事情上比我们好得多。但在推理上不如我们。我认为它不会一下子在所有方面大幅超越我们，而是一个领域一个领域地实现。我对此的解脱方式是，我可以走在海滩上看鹅卵石和贝壳。人工智能做不到。

I don't know the answer to either of those questions. My suspicion is AI will get better than us at everything in the end, but it'll be sort of one thing at a time. It's currently much better than us at chess and Go. It's much better than us at knowing a lot of things. Not quite as good as us at reasoning. I think rather than sort of massively overtaking us in everything all at once, it'll be done one area at a time. And my sort of way out of that is, you know, I get to walk a beach and look at pebbles and seashells. AI doesn't.

是啊。它可以创造自己的海滩。

Yeah. It can create its own beach.

不。它只会知道我写下来并放到网上的新软体动物吗？

No. Would it only know about the new mollusk that I discovered if I write it up and put it online?

嗯。

Mhm.

所以，人类可以继续以人工智能无法触及的方式探索宇宙。

So, the human can continue to explore the universe in ways that AI doesn't have access to.

你的整个评估中缺少一个词。

There's one word missing from your entire assessment.

什么词？

What's that?

「还」。

Yet.

是啊，我只是想，人工智能会提出一个需要人类洞察力的新宇宙理论吗？它没有这种洞察力，因为我在以没人想过的方式思考。

Yeah, I just think of my, you know, will AI come up with a new theory of the universe that requires human insights that it doesn't have because I'm thinking the way no one has thought before.

我认为它会。

I think it will.

这不是我想从你那里得到的答案。

That's not the answer I wanted from you.

是啊，我是这么想的。但这就是你得到的答案。

Yeah, I was. But that's the answer you got.

我给你举个例子。人工智能已经非常擅长类比。所以当 ChatGPT-4 不被允许上网，所有知识都在它的权重中时，我问它为什么堆肥堆像原子弹，它知道，说能量尺度非常不同，时间尺度也非常不同。但它接着谈到堆肥堆变热时如何更快地产生热量，原子弹产生更多中子时如何更快地产生中子。所以它理解了共同点，它必须理解这一点才能将所有这些知识压缩到如此少的连接中，只有大约一万亿个。这是很多创造力的来源，而且不仅仅是找到与其他词并列的词。

Let me give you an example. AI is very good at analogies already. So when ChatGPT-4 was not allowed to look on the web when all its knowledge was in its weights, I asked it why is a compost heap like an atom bomb and it knew it said the energy scales are very different and the time scales are very different. But it then went on to talk about how when a compost heap gets hotter it generates heat faster and when an atom bomb generates more neutrons it generates neutrons faster. So it understood the commonality and it had to understand that to pack all that knowledge into so few connections, only a trillion or so. That's a source of much creativity and it's not just by finding words that were juxtaposed with other words.

不，它理解了什么是链式反应。

No, it understood what a chain reaction was.

是的。

Yeah.

好吧。我们完了。是的。我们在地球上完了。我们完了。我们结束了。这是最后一集。我们坚持自我。我们完了。先生们，很高兴认识你们。

Well, all right. That's the end of us. Yeah. We're done on Earth. We're done. We're finished. This is the last episode. We stick in us. We're done. Gentlemen, it's been a pleasure.

杰弗里·辛顿，很高兴你能来。我们知道你被很多事情牵扯，尤其是在你最近获得诺贝尔奖之后，我们很高兴你从肯定排得满满当当的忙碌生活中抽出时间给我们。

Well, Geoffrey Hinton, it's been a delight to have you on. We know you're tugged in many directions, especially after your recent Nobel Prize, and we're delighted you gave us a piece of your surely overscheduled and busy life.

谢谢你的邀请。

Thank you for inviting me.

好吧，伙计们，这真是精彩。你们全程坐得舒服吗？我坐立不安。我坐立不安。我就知道你会恐慌。好吧，不。我得告诉你，对话的某些部分让我感到焦虑，就像坐在剧院里拉肚子一样。谢谢你的直白。谢谢分享。这是别人对我说过的最好的话。就此，本期《星谈》特别版到此结束。查克，有你在总是很好。加里，喜欢有你在我身边。尼尔·德格拉斯·泰森一如既往地祝愿你们继续仰望星空，无论这变得多么困难。

Well, guys, that was something. Did you sit comfortably through all of that? I squirmed. I squirmed. I knew you'd panic. Well, no. I have to tell you that certain parts of the conversation gave me the anxiety of sitting in a theater with diarrhea. Thanks for that explicit. Thanks for sharing. That's the nicest thing anybody's ever said about me. On that note, this has been StarTalk special edition. Chuck, always good to have you. Gary, love having you right at my side. Neil deGrasse Tyson bidding you as always to keep looking up however much harder that will become.