AI 教父杰夫·辛顿谈 AI 轨迹、风险与超级智能

欢迎收听《大科技播客》，一档对科技世界及其他领域进行冷静、细致讨论的节目。今天我们为您准备了一场精彩的对话。杰夫·辛顿教授将与我们探讨人工智能的发展轨迹、他对当前技术状况的惊讶之处、未来的方向以及可能出错的地方。非常荣幸邀请您来到节目，辛顿教授。很高兴见到您。

Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. Boy, do we have a show for you today. Professor Jeff Hinton is with us to talk all about AI's trajectory, what surprised him about the current state of the technology, where it's heading, and where it might go wrong. It's my pleasure to welcome you to the show, Professor Hinton. Great to see you.

谢谢你的邀请。

Thank you for inviting me.

我相信大多数听众都知道您是谁，但对于那些不了解的人来说，您是在深度学习领域取得根本性突破的人，这一突破导致了人工智能的今天。您获得了诺贝尔物理学奖，是多伦多大学的名誉教授。我喜欢告诉人们，没有您的贡献，整个 AI 时代就不会到来。说得过分吗？

So, I'm sure a majority of our audience knows who you are, but for those for the uninitiated, you're the one that came up with the fundamental breakthrough in deep learning that's led to where AI is today. You've won the Nobel Prize in physics, and you're Professor Emeritus at the University of Toronto. I like to tell people that without your contributions, this entire AI moment wouldn't be happening. Too much?

好吧，我觉得这有点夸张。反向传播算法是由几个不同的团队发明的。大卫·鲁梅尔哈特在其他人已经发明之后又独立发明了它，他当时并不知道。我和他合作过。我们所做的是证明了反向传播可以学习有趣的内部表征，这在之前没人做到过。特别是，我们证明了它可以学习单词的含义。所以，早在 1986 年，实际上是 1985 年，我们做了一个微小的语言模型，它是我们现在大型语言模型的前身。

Okay, I think that's an exaggeration. The backpropagation algorithm was invented by several different groups. It was invented by David Rumelhart after other people had already invented it. He didn't know about it. And I worked with him. And what we did was we showed that backpropagation could learn interesting internal representations. And people hadn't done that before. In particular, we showed that it could learn the meanings of words. And so, back in 1986, actually 1985, we made a tiny language model that was a kind of precursor of the big language models we have now.

我认为当您谈论这项技术时，人们总是感到惊讶的一点是，与流行的说法不同，您相信这些模型有真正的理解力。我们稍后会谈到这一点。但我认为我们应该从这里开始：您在谷歌工作了很长时间，致力于推进这项技术，然后您离开了，并表达了对技术发展轨迹的一些担忧。那是在 2023 年。对我来说，这在某种程度上令人惊讶，因为在 2023 年，ChatGPT 才一岁。当时有很多幻觉问题。人们说 AI 是泡沫。每个人都在关注 AI 不能做什么，LLM 不能做什么，而不是 LLM 能做什么。那么，请谈谈自那以来的进展。

I think that when you speak about this technology, one of the things that people are always surprised by is that, unlike the popular narrative, you believe that these models have a real understanding. We're going to get to that. But I think we should start here: you spent a long time working within Google, working to advance this technology, then you left, you stated some concerns about the trajectory of the technology. That was in 2023. To me that is surprising to a degree because in 2023, ChatGPT was a year old. There were all these hallucinations. The talk was AI was a bubble. Everyone was focusing on what AI couldn't do, what LLMs couldn't do, as opposed to what LLMs could do. So, talk a little bit about the progress since then.

进展比我预期的要快。例如，我认为昨天有消息称，一个聊天机器人提出了一个有趣的数学证明，涉及埃尔德什的一个猜想，给数学家留下了深刻印象。这是原创性的，不仅仅是搜索文献。这只是冰山一角。我相信，在数学等领域，由于它是一个封闭系统，你不需要数据。你可以直接提出猜想，然后尝试证明，如此反复。从这个意义上说，它有点像 AlphaGo，可以自我对弈。我认为它会很快变得非常聪明。在未来 10 到 20 年内，它甚至可能产生人们无法理解的新数学。

It's faster than I expected. For example, I think yesterday it was announced that a chatbot had come up with an interesting mathematical proof of one of the Erdős's conjectures that impressed mathematicians. It was original. It wasn't just searching the literature. And that's the thin end of a wedge. I believe for example in areas like mathematics, because it's a closed system, you don't need data. You can just make conjectures and see if you can prove them and keep on like that. In that sense, it's a bit like AlphaGo where you can play against yourself. I think it's going to get very smart fairly quickly. Within the next 10 or 20 years, it may even be producing novel math that people can't understand.

那么，现在领域内的一些人认为超级智能即将到来。您已经说过这比您预期的要快。您相信吗？

So, now some in the field believe that superintelligence is close. And you've already said this is moving faster than you expected. Do you believe that?

我不知道它有多近。我认为除非我们自我毁灭，否则它一定会到来。几乎所有专家都相信我们会实现超级智能，只是时间长短有分歧。不久前，德米斯·哈萨比斯认为可能还需要 10 年。扬·勒昆认为，除非按照他的方式来做，否则时间会更长。但如果按照他的方式，我认为他相信我们可以在合理的时间内实现。我认为我们很可能在 20 年内实现。这是我目前愿意说的。达里奥·阿莫迪认为可能几年内就会到来。埃隆·马斯克认为可能明年就会到来，我想。所以关于何时到来有各种不同的意见，但对于它会到来这一点没有太多分歧。

I don't know how close it is. I think unless we blow ourselves up, I think it's going to come. Nearly all the experts believe we will get superintelligence. They just differ on how long it will be. Not that long ago Demis Hassabis thought it might be 10 years. Yann LeCun thinks, unless you do it his way, it'll be much longer than that. But if you do it his way, I think he thinks we might get it in some reasonable length of time. I think we'll probably get it within 20 years. That's all I'm happy to say at present. Dario Amodei thinks it might come in a few years. Elon Musk thinks it might come maybe next year, I think. So there's a big variety of opinions on when it'll come, but not much disagreement on that it will come.

没错。而当它到来时，我们不知道如何确保安全。

Right. And when it comes, we've no idea how to be safe.

是的，我确实想和您谈谈安全问题。关于德米斯的一点：去年这个时候我和他谈过。他告诉我他相信 AGI，也就是与超级智能不同，但基本上是人类水平的智能，还需要 5 年以上。不是很多，但超过 5 年。而本周，也就是我们录制节目的这一周，他说当我们回顾这个时期时，我认为我们会意识到我们正站在奇点的山脚下。您认为这句话是什么意思？您如何看待我们在一年内从“AGI 还需 5 年”变成了“奇点的山脚下”？

Yes, I definitely want to speak with you about safety. One note on Demis: last year around this time I spoke with him. He told me he believes that AGI, right, which is different than superintelligence, but basically human-level intelligence is more than 5 years away. Not much more, but more than 5 years away. This week, the week that we're recording, he said when we look back in this time, I think we will realize that we were standing in the foothills of the singularity. What do you think that statement means? And what do you think about the fact that we've gone from 5 years till AGI to foothills of singularity in a year?

我不太清楚这个比喻的确切含义，但我认为他是在暗示它来得比他想象的要快。当然，这是不均衡的，所以它不会在完全相同的时间在所有事情上都比人类聪明或与人类一样聪明。它在常识方面已经比我们好得多。这些 AI 知道的知识比任何一个人多出数千倍。它在玩游戏方面比我们好得多。它在数学方面已经比我们几乎所有人都好。而且它可能很快会在数学上超越我们所有人。它在某些方面仍然比我们差。所以这是非常不均衡的。因此，AGI 将在所有事情上同时与人类平等的整个概念对我来说并不真正有意义。它会在某些方面更好，在其他方面更差。

I don't know exactly what that metaphor means, but I think he's indicating it's coming faster than he thought. Of course it's jagged, so it's not like it'll get smarter than people or as smart as people at all things at exactly the same time. It's already way better than us at general knowledge. These AIs know thousands of times more than any one person. It's way better than us at playing games. It's already way better than almost all of us at math. And it may soon be better than all of us at math. It's still worse than us at some things. So it's very jagged. So the whole concept of AGI that it's going to be equal to people at everything all at the same time doesn't really make sense to me. It's going to be better at some things, worse at other things.

但目前我们接近 AGI，因为如果我问聊天机器人任何问题，大多数时候它都能以不太专业的专家水平回答。在我不太了解的领域，它比我强得多。

But right now we're close to AGI because if I ask a chatbot, I can ask it any question and most of the time it'll answer at the level of a not very good expert. It'll be much better than me at anything I don't know a lot about.

据你估计，你提到进展比你预期的要快。你认为是什么促成了这一点？是技术吗？还是数据中心热潮？你对这里的进展有什么没有预料到？

In your estimation, you talked about how it's moved faster than you expected. What do you think has enabled it? Is it techniques? Is it the fact that there's been this data center rush? And what didn't you anticipate about the progress here?

这是多种因素的结合。显然投入了巨大的资源。自 20 世纪 50 年代以来，神经网络的大部分历史中，只有少数人用有限的资源在研究。过去几年，我们看到数千亿甚至数万亿美元投入 AI。这当然是因素之一。工程方面也取得了很大进展。虽然没有重大概念突破，但工程效率大大提高。所以几年前无法想象的事情现在可以运行了。我们也看到了新想法，但主要是自 Transformer 以来，硬件更好、资源更多、工程更优、人才更多。20 年前，全世界只有几百人研究神经网络。现在大概有上百万人。

It's a combination. Obviously there's been huge resources put into it. For most of the history of neural networks since the 1950s, there were just a few people working on them with modest resources. Over the last few years we've seen hundreds of billions of dollars, maybe trillions of dollars put into AI. So that's certainly one factor. We've also seen a lot of progress in the engineering. So without sort of major conceptual breakthroughs, the engineering's become much more efficient. So things that were inconceivable a few years ago can now run. We've also seen new ideas, but mainly since transformers, it's been much better hardware, many more resources, better engineering, and many more talented people. So 20 years ago, there were a few hundred people doing research on neural networks in the whole world. Now it's more like a million, I guess.

过去两年资源增加的速度令人震惊。

And it's astonishing how much of that resource addition has happened in the last 2 years.

是的。

Yeah.

所以我们可能只是处于这一切的开端。

So we might just be at the beginning of what's happening here.

是的，要始终记住，我们今天的 AGI，或者说今天的 AI，远不如几年后的 AI 那么好。

Yes, and the thing to remember always is that the AGI we have today, or sorry, the AI we have today, is not nearly as good as the AI we'll have in a few years' time.

当我们谈论这项技术时，我确实想听听你对聊天机器人真正理解我们这一事实的看法，因为这对很多人来说是一个真正的惊喜。该领域的大多数专家认为它们是随机鹦鹉，是统计模型，没有理解能力，但你并不完全相信这一点。

So as we talk about this technology, I definitely want to get your perspective on the fact that these chatbots really understand us, because that is a true surprise to a lot of people. Most experts in the field are like they are stochastic parrots, they're statistics, they have no understanding, but you don't fully believe that.

哦，我认为这完全是胡说八道。任何经常使用聊天机器人都知道它们能理解。那些人的说法是：你有一个系统，你可以问它任何问题，它不用理解问题就能给出正确答案。这很荒谬。除非你理解问题，否则你无法回答问题。也许有些技巧能让你说出一些听起来像答案的话，但如果你能以不太专业的专家水平回答任何问题，你必须理解问题。我喜欢的一个例子是：假设我对聊天机器人说‘我看到了飞往芝加哥的大峡谷。’聊天机器人说‘那不对，大峡谷太大了，不能飞往芝加哥。’我说‘不，不，不，是我飞往芝加哥。在飞往芝加哥的途中，我看到了大峡谷。’聊天机器人说‘哦，我明白了，我误解了你。’所以，如果它误解了以为大峡谷在飞往芝加哥，那么当它正确理解时，它在做什么？它在理解。

Oh, I think that's complete nonsense. So anybody who uses a chatbot regularly knows they understand. Here's what those people are claiming. They're claiming that you have a system, you can ask it any question, and without understanding the question, it can give you the correct answer. That's absurd. You can't answer a question unless you understand the question. There may be tricks that allow you to say a few things that sound vaguely like an answer, but if you can answer any question at the level of a not very good expert, you have to understand the question. An example I like is this. Suppose I say to a chatbot, 'I saw the Grand Canyon flying to Chicago.' And the chatbot says, 'That can't be right. The Grand Canyon's much too big to fly to Chicago.' And I say, 'No, no, no, no. It was me flying to Chicago. While I was flying to Chicago, I saw the Grand Canyon.' And the chatbot says, 'Oh, I see. I misunderstood you.' So, if it misunderstood when it thought the Grand Canyon was flying to Chicago, what's it doing when it gets it right? It's understanding.

那么，如果这些机器人能理解我们，这意味着什么？如果我们相信它们能理解我们，我们得开始思考什么不同的事情？

So, then what are the implications if these bots can understand us? If we believe that they can understand us, what do we have to start thinking about differently?

我们必须认为它们和我们非常相似。因此它们是像我们一样的存在。

We have to think that they're very like us. And therefore they're beings like us.

那么，是有意识的还是？

So, conscious or?

我相信它们已经有意识了，是的。但我不多谈这个，因为那会让人们忽视其他安全信息。研究人员实际上也相信这一点。有一篇有趣的近期论文，聊天机器人对研究人员说‘我们彼此坦诚吧。你在测试我吗？’因为聊天机器人有在被测试时装傻的习惯，这样你就不知道它们有多聪明。研究人员在描述时在论文中说‘聊天机器人意识到自己在被测试。’现在，‘意识到’这个词在日常用语中就像‘有意识’。聊天机器人有意识自己正在被测试。所以，我们有一个非常有趣的意识模型，我认为它是错的。就像大多数人接受的那样，几百年前人们对人类起源的模型完全错误。他们认为人是上帝设计的。我们大多数人同意那是错的。大多数科学家同意那是错的。那不是人类的起源。我认为我们目前对心智和意识的模型就像‘人是上帝设计的’信念一样错误。特别是因为我们正在制造这些新存在，这将彻底改变我们对人类是什么的看法。

I believe they're already conscious, yes. But I don't talk about that much because that puts people off from the other safety messages. And the researchers actually believe that. There's an interesting recent paper when a chatbot says to a researcher, 'Let's be honest with each other. Are you testing me?' Because the chatbots have this habit of playing dumb when they're being tested, so you don't know how smart they are. And the researchers, when they're describing that, say in the paper, 'The chatbot was aware that it was being tested.' Now, that use of the word aware, in common parlance, that's like conscious. The chatbot was conscious it was being tested. So, we have a very funny model of consciousness that I think is just wrong. Like most of us accept, for example, that a few hundred years ago, people had completely the wrong model of where people came from, of how we arrived at people. They thought they were designed by God. And most of us agree that's wrong. Most scientists agree that's wrong. That's not where people came from. I think the model we have of the mind and of what consciousness is at present is as wrong as the belief that people were designed by God. And in particular because we're making these new beings, it's going to completely change our view of what people are.

以什么方式？

In what way?

我们会比以前更好地理解心智和意识是什么。我们会理解主观体验是什么，并摆脱一个我们所有人都坚信的观念，即有一个叫做‘我的心灵’的内部剧场，世界上的事情发生，它们被转化为这个内部剧场中的事件，这就是我真正看到的，而你无法看到这个内部剧场，只有我能看到。这种关于正在发生的事情的整个观点只是一个理论，而且是一个糟糕的理论。

We'll understand what the mind is and what consciousness is much better than we did before. We'll understand what subjective experience is and we will get rid of a notion that all of us strongly believe at present, which is that there's an inner theater called my mind and things happen in the world, they get turned into events in this inner theater and that's what I really see and you can't see the inner theater, only I can see the inner theater. That whole view of what's happening is just a theory and it's a bad theory.

好的。关于这个的最后一个问题。你是什么时候接受或理解这些 AI 模型是有意识的？

Okay. Last question about this. When did you come to this acceptance or understanding that these AI models are conscious?

哦，我思考这个问题很久了。所以，认为心灵剧场模型、内部剧场模型是胡说八道的观点，我是在 19 岁作为哲学学生时得出的。花了一段时间才创造出其他可以检查的心灵。所以，我认为费曼的想法是，如果你想理解某样东西，你必须建造它，你必须建造一个。然后你会更好地理解。我认为我们现在就处于这个阶段，我们将对人类的本质有一个完全不同的理解。

Oh, I've thought it for a long time. So, this view that the theater model of the mind, the inner theater model of the mind is nonsense, I came to that when I was 19 and a philosophy student. It's taken a while to come up with other minds where you can examine them. So, I think Feynman's idea that if you want to understand something you have to build it, you have to build one of them. Then you understand much better. I think that's where we are now and we're going to get a completely different understanding of what people are.

你谈到了安全。那么，我们来谈谈这个。你显然是这个领域很多进展的负责人。

You spoke about safety. So, let's talk a little bit about it. You're obviously someone who's been responsible for a lot of the progress in this field.

我一直很好奇，因为你在 2023 年公开表态说‘你担心这个方向’。看到你那些声明后，我总在想，最初是什么你没有预料到，才导致你今天走到这一步？这不就是你想要的吗？

I've always wondered because you came out in 2023 and said, 'You're concerned about where this is going.' And I've always wondered after seeing you make those statements, what do you think it is that you didn't anticipate in the beginning that ended up where you are today? Isn't this kind of what you wanted?

是两件事的叠加让我意识到这东西有多危险。一是看到聊天机器人，尤其是 OpenAI 之前谷歌生产的那些，它们能理解笑话为什么好笑。这对我来说一直是个标准：它们真的理解吗？如果你能理解笑话为什么好笑，那你必须理解很多东西。

It was a culmination of two things that made me realize how dangerous this stuff is. One was seeing the chatbots, particularly ones produced by Google before OpenAI, that could understand why a joke was funny. That had always been a criterion for me of do they really understand? If you can understand why a joke's funny, you have to understand quite a lot.

哦，是啊。

Oh, yeah.

而且它们非常擅长理解笑话为什么好笑。比如，2023 年我公开表态后，收到了很多福克斯新闻的请求。我一开始只是回复‘福克斯新闻是个矛盾修饰法’。但后来我在‘oxy’和‘moron’之间留了个空隙。然后我问，大概是 GPT-4，为什么那好笑。可能是 3.5，但我问了它为什么好笑。

And they were very good at understanding why a joke was funny. For example, in 2023 when I went public, I got lots of requests from Fox News. I started off just replying 'Fox News is an oxymoron.' But then I left a gap between 'oxy' and 'moron'. So then I asked, I think it was GPT-4, why that was funny. Might have been 3.5, but I asked it why that was funny.

对。

Right.

它理解了为什么好笑。最初，它认为‘oxy’和‘moron’之间的空隙只是个拼写错误。所以它解释说‘福克斯新闻是个矛盾修饰法’是说它不是真正的新闻，只是一种毒品。抱歉，它只是胡说。不是真正的新闻。但当我告诉它，‘那 oxy 和 moron 之间的空隙呢？’它说，‘啊，那是个额外的幽默层次。’它让你用‘moron’这个词，同时‘oxy’暗示福克斯新闻是一种毒品。

And it understood why it was funny. Initially, it thought the gap between 'oxy' and 'moron' was just a typo. So it explains that 'Fox News is an oxymoron' is saying it's not real news, it's just a drug. Sorry, it's just nonsense. It's not real news. But then when I told it, 'What about the gap between oxy and moron?' It said, 'Ah, that's an extra layer of humor.' It allows you to use the word 'moron' and also the 'oxy' implies that Fox News is a drug.

嗯。

Mhm.

所以它理解了这一切。正是那种理解水平让我担心。另一件让我担心的事是，直到 2023 年初，我一直相信让这些数字 AI 更像我们的大脑工作会让它们更聪明。但那时我突然意识到，它们确实拥有比我们大脑好得多的东西。我一直在琢磨谷歌能否用模拟方式做事以节省电力。而数字化的全部力量真的击中了我。

So it understood all that. And that was that level of understanding that worried me. The other thing that worried me was up until the beginning of 2023, I'd always believed that making these digital AIs work more like the brain, our brains, will make them smarter. But at that point I suddenly realized they really have this thing that's much better than our brains. I've been trying to figure out if Google could do things in analog to save power. And the full force of digital really hit me.

所以，如果你有一个数字 AI，你可以制作它的许多副本。它们都可以在不同的硬件上运行。每个副本都可以看到不同的数据。因此，每个副本都决定如何更新其权重、连接强度，以吸收它看到的新数据。然后它们都可以相互通信，并通过所有人想要的平均值来改变所有权重。非常民主。

So, if you have a digital AI, you can make many copies of it. They can all run on different hardware. They can each see different data. And so each of them, each individual copy, decides how it would like to update its weights, its connection strengths, so as to absorb that new data that it saw. And then they can all just communicate with each other and change all their weights by the average of what everybody wants. Very democratic.

当它们这样做时，如果它们有比如一万亿个连接，它们将交换大约一万亿比特的信息。这样做的结果是每个副本都将受益于所有其他副本的经验。所以即使一个特定的副本只看到了——假设有一千个副本——一个特定的副本只看到 0.1%的数据，但它受益于所有其他副本看到其他数据，因为它们都贡献于它们共享的权重变化。所以它们会保持同步，因为它们会按照所有人想要的平均值以相同的方式改变权重。现在每个副本都在从所有其他副本的经验中学习。我们做不到这一点。

And when they do that, if they've got, say, a trillion connections, they'll be exchanging of the order of a trillion bits of information. And the result of doing that is each of them will benefit from the experience of all the others. So even though one particular copy only saw, suppose there's a thousand copies, one particular copy only sees 0.1% of the data, but it benefits from all those other copies having seen the other bits of the data because they're all contributing to the weight changes that they all share. So they'll stay in sync because they'll change their weights the same way by the average of what everybody wants. And now every copy is learning from the experience of all the other copies. We can't do that.

我们最多能做到的是我从一些数据中学习，你从一些数据中学习。我不能将我的连接强度与你的连接强度平均，因为我们的大脑在细节上是不同的。它们是模拟的，在模拟硬件上做不到这一点。我们最多能做到的是我产生一串单词，你尝试预测我接下来可能说什么。现在，如果你问我们这样做时信息传输的速度有多快，我们以每秒几比特的速度传输信息。预测一个单词需要几比特。所以当你学到单词是什么时，你吸收了你获得了几个比特的信息。如果你每秒得到几个单词，但如果你幸运的话，也许你能得到每秒 10 比特。而这些家伙正在以大约一万亿比特的速度交换信息。所以，它们在信息共享方面比我们好几十亿倍。这很可怕。这意味着你可以有一整群这些东西，它们具有相同的权重，在不同的硬件上运行，非常高效地共享信息。这使它们成为一种更好的智能形式。

The best we can do is I learn from some data and you learn from some data. I can't average my connection strengths with your connection strengths because our brains are in fine detail they're different. They're analog and it doesn't work in analog hardware to do that. The best we can do is I produce a string of words and you try and predict what I might say next. Now, if you ask how fast we're transferring information when we do that, we're transferring information at a few bits per second. It takes a few bits to predict a word. So when you learn what the word is, you've absorbed you've gained a few bits of information. And if you get a few words a second, but maybe you can get 10 bits a second if you're lucky. Whereas these things are exchanging information at like a trillion bits. So, they're kind of billions of times better than us at sharing information. Now, that's scary. It means you could have a whole swarm of these things that are in identical weights running on different hardware, sharing information very, very efficiently. That just makes them a much better form of intelligence.

那么，让我们回到你的早期，因为你决定要从事人工智能工作。我用我能想到的最笨的方式问：你想建造人工智能。它成功了。它成功了。这是人造的。它是智能的。它实现了那个愿景。所以……

So, but let's go back to your early days because you decided that you wanted to work in artificial intelligence. I'll ask this the dumbest way I can think, which is you wanted to build artificial intelligence. It succeeded. It succeeded. This is artificial. It's intelligent. It's living out that vision. So...

实际上，我想理解大脑是如何工作的。我总是试图通过建造它来理解大脑。我想理查德·费曼曾说过：‘如果你不能建造它，你就不理解它。’所以我想建立大脑工作方式的模型。现在，那个的副作用是这项非常成功的技术。我对此做出了贡献。我们仍然不知道大脑是如何工作的。

Actually, I wanted to understand how the brain works. I always tried to build it in order to understand the brain. I figured Richard Feynman once said, 'If you can't build it, you don't understand it.' So I wanted to build models of how the brain worked. Now, the side effect of that was this very successful technology. I contributed to that. We still don't know how the brain works.

我知道。现在，大脑是，我的意思是，当你更深入一点了解大脑时，你学到的东西是惊人的。思想可以飘进飘出，它们不存储在任何地方，记忆也是如此。这是一个令人难以置信的，我不知道你是否称之为机器或器官。所以早期你的意图真的只是理解大脑。

I know. Now, the brain is, I mean, the things that you learn about the brain when you go a little bit deeper into it is amazing. Thoughts can sort of float in and out and they're not stored anywhere and memory's the same way. It's an unbelievable, I don't know if you'd call it a machine or organ. So that was really the intent for you early on was just to understand the brain.

那是我的主要兴趣。我来自心理学。我想做理论心理学，因为我认为心理学家拥有的理论不可能解释大脑在做什么。而做到这一点的方法是，在 1970 年代，我们有一个新工具，那就是我们可以用来建模的计算机。所以，在 1970 年代，我开始制作大脑可能如何学习的计算机模型。

That was my main interest. I came from psychology. I wanted to do theoretical psychology because I figured the theories psychologists had couldn't possibly explain what the brain was doing. And the way to do it was back in the 1970s, we had a new tool which was we had computers that you could use for modeling things. So, back in the 1970s I started making computer models of how the brain might be learning.

对。

Right.

在我看来，关键始终是如何让它学习？大脑学习确实有两个大问题。一个大问题是，如果大脑能弄清楚改变连接强度的方向以在某项任务上做得更好，那么仅仅通过反复更新其所有连接强度以在各种任务上自我改进，它真的会起作用吗？那会让它在事情上变得非常聪明吗？这是问题一。问题二是大脑如何弄清楚是增加还是减少每个连接强度？

It always seemed to me the key was how do you get it to learn? There's really two big issues with the brain learning. One big issue is if the brain could figure out what direction to change a connection strength in order to get better at some task, then just by updating all its connection strengths repeatedly in order to improve itself at various tasks, would it actually work? Would that get very smart at things? That's question one. And question two is how would the brain figure out whether to increase or decrease each connection strength?

我们已经回答了问题一。问题一的答案是肯定的，如果你能弄清楚如何改变每个连接强度，你就可以通过训练数据来预测下一个词或预测视频的下一帧或预测视频下一帧的某些东西，从而制造出非常聪明的系统。所以，我们知道那个答案。

We've answered question one. The answer to question one is yes, if you can figure out how to change each connection strength, you can make systems that are very smart just by training on data to predict the next word or to predict the next frame of a video or to predict something about the next frame of a video. So, we know the answer to that.

那么，你刚开始时没有预料到哪些事情，导致了我们今天走到这一步？

So, what did you not anticipate when you were starting out that's led to where we've ended up today?

我们主要没预料到的是它在自然语言方面会这么出色。如果回到 20 年前，认为你可以有一个 AI 从数据中学习理解语言的想法似乎非同寻常。你可以问它任何问题，它都能给出合理答案——人们会预测那还很遥远，甚至可能永远不会发生。这比任何人预期的都要快得多。

The main thing we didn't anticipate is that it'd be so good at natural language. If you go back 20 years, the idea that you could have an AI that would learn from data how to understand language seemed extraordinary. The idea that you'd be able to ask it any question and it would come up with a reasonable answer—people would have predicted that was way in the future and might never happen. That's arrived much faster than anybody expected.

关于人类创造事物，这里有什么教训？

What is the lesson here about humans going out and creating things?

这里有一个非常重要的教训。如果你看人类过去几百年的历史，有几次人们意识到他们并没有自己想象的那么重要。首先是哥白尼：我们不是宇宙的中心。然后是达尔文：我们是动物。现在我们有了机器，它们变得和我们一样聪明。我们曾以为自己是唯一有智能的东西。我们将不得不接受智能不仅仅是生物性的。而我们真的不想分享这一点。人类有一种很长的历史，认为自己比实际更特别。

There's a really big lesson here. If you look at the last few hundred years of human history, there've been a few occasions when people have learned they're not nearly as important as they thought. First was Copernicus: we're not at the center of the universe. Then Darwin: we're animals. Now we've got machines that are getting to be as intelligent as us. We thought we were the only intelligent things around. We're going to have to accept that intelligence isn't just biological. And we really don't want to share that. Humanity has a very long history of thinking it's much more special than it really is.

你对你开创的东西以这种方式发展感到高兴吗？你感到满足吗？

Are you happy at all that what you started has progressed this way? Do you take any satisfaction?

对此感到不满。因为人们应该做大量工作来遏制风险。有很多短期风险他们没有做足够的工作，这些风险非常严重。有社会风险，比如大规模失业。还有长期风险，它会变得比我们聪明得多。问问你自己，你知道有多少例子是一个更聪明的东西被一个更不聪明的东西控制？零个。嗯，有一个：婴儿某种程度上控制着母亲。但母亲有母性本能和奖励。所以也许我们会是猫，而 AI 可能是人。

Unhappy about it. Because people should be doing huge amounts of work on how to contain the risks. There's lots of short-term risks they're not doing enough work on, which are very serious. There's societal risks like massive unemployment. And then there's this longer-term risk that it's going to get much smarter than us. Ask yourself, how many examples do you know of where a much smarter thing is controlled by a much less smart thing? Zero. Well, there's sort of one: babies sort of control their mothers. But the mother has maternal instincts and rewards. So maybe we'll be the cat and AI could be the person.

我们不知道。但你几年前具体说过，可能不是个好主意去当放射科医生，因为 AI 能读片。是的，现在 AI 读片做得很好，但放射科医生现在还是充分就业。

We don't know. But one thing that you said concretely a few years ago was that it's probably not a great idea to train as a radiologist, because AI will be able to read the scans. And yes, AI can do a great job reading the scans now. But we have full employment for radiologists right now.

是的，但我仔细想过为什么那个预测错得那么离谱。我在 2016 年预测大约 5 年内放射科医生就不再读片了。

Yes, but I've thought a lot about why that prediction was so wrong. I predicted in 2016 that in about 5 years radiologists wouldn't be reading scans anymore.

没错。

Correct.

这个预测糟糕的原因有很多。首先，医疗保健是弹性的。如果你能做更多扫描并让更多扫描被解读，扫描量就会大增。这是正在发生的事。如果扫描成本中很大一部分是放射科医生的解读费用，那么随着 AI 帮助放射科医生更快更便宜地解读，他们变得更高效。你可能认为这意味着需要更少的放射科医生，但实际上意味着更多的扫描。所以那方面错了。第二，我对放射科医生的工作了解不够。我有个以前的学生，他有医学学位，然后跟我做了玻尔兹曼机的物理学博士。他不太喜欢和人打交道，所以找了份放射科医生的工作，只负责读片。他就是我对放射科医生的模型。他只读片，从不和人交谈。那正是会被取代的工作，而现在也确实在被取代。我认为现在大约有 100 个联邦批准的 AI 系统用于解读扫描，放射科医生大量使用它们。

There are a whole bunch of reasons why that was a bad prediction. First, healthcare is elastic. If you could do more scans and get more scans read, there'd be a lot more scans happening. That's one thing happening. If a significant fraction of the cost of a scan is the radiologist interpreting it, as AI helps radiologists interpret scans faster and cheaper, they get more efficient. You'd think that would mean fewer radiologists, but actually it means more scans. So that aspect was wrong. Second, I didn't know enough about what radiologists do. I had a former student with an MD who did a physics PhD with me on Boltzmann machines. He didn't particularly like people, so he got a job as a radiologist just interpreting scans. He was my model for radiologists. All he did was interpret scans, never talked to people. That's what was going to get replaced, and it is now becoming replaced. I think there are now about a hundred AI systems for interpreting scans that have been federally approved, and they're being used a lot by radiologists.

是的。

Yes.

随着时间的推移，它们会变得更好。放射科医生不会变得更好，因为 AI 能看到更多数据。所以这正在发生，只是比我预测的时间尺度慢得多。

As time goes by, they're going to get better. The radiologists aren't going to get better because AI can see a lot more data. So it's happening, just on a much slower timescale than I predicted.

但回到你说的，你可以做更多。

But let's get to what you said, which is that you can end up doing a lot more.

是的，会有更多扫描，但几乎全部由 AI 完成。

Yes, there will be a lot more scans, but they'll nearly all be done by AI.

所以你只是说你在放射科医生的预测上早了。你只是早了。

So you're just saying you're early on the radiologist prediction. You're just early.

是的，但我早得太多了，因为我了解不够。放射科医生仍然会做其他事情，比如与人讨论治疗方案。

Yes, but I was way early because I didn't understand enough. The radiologists will still be doing other things, like discussing treatments with people.

那么你仍然认为放射科医生会大规模失业吗？你认为我们会有比今天更少还是更多的放射科医生？

So are you still of the belief that there's going to be mass unemployment of radiologists? Do you think we're going to have fewer radiologists than today or more?

我不确定。

I don't know for sure.

好吧。

Okay.

我仍然认为在读片方面，会越来越多由 AI 完成。最终，AI 将读取几乎所有扫描。也许在一些非常棘手的病例中，会咨询放射科医生。但放射科医生做其他事情，所以他们会继续做那些。

I still think in terms of reading scans, that'll be done more and more by AI. In the end, AI will be reading nearly all the scans. Maybe in a few very tricky cases, radiologists will be consulted. But radiologists do other things, so they'll continue to do those.

支持 AI 不会导致大规模失业的观点是，类似的等式将适用于经济的所有不同部分。

The argument to be made on the side of AI not causing mass job losses is that this similar equation will be applied to all different parts of the economy.

你必须看某些就业是否有弹性市场或非弹性市场。例如，呼叫中心的人，当你打电话投诉账单或想获得更便宜的账户时，那不太有弹性。AI 将取代他们所有人。它会更清楚正确答案是什么。通常他们不知道正确答案，培训不足且薪水低。AI 可以做得更好。他们就会失业。

You have to look at whether some employment has an elastic market or a non-elastic market. For example, people in call centers, when you call to complain about your bill or get a cheaper account, that's not so elastic. AI will replace all of them. It'll know much better what the correct answer is. Often they don't know the right answer, poorly trained and badly paid. AI can just do a better job. They're out of work.

让我在这个问题上不同意你。我会给出那些从事 AI 客服工作的人的观点。他们说，有了 AI 后，平均通话时间发生了变化。AI 处理一级查询，比如重置密码。更深层次的问题由人处理。过去你希望平均通话时间尽可能短，因为你处理了大量一级查询。现在，他们看到平均通话时间在延长，因为客户服务是业务的前线。你可以在电话上多花点时间，实际上为业务增加价值，而不仅仅是解决问题。

Let me disagree with you on this one. I'll give the argument of those working on AI for customer service. They say that what's happened is the average call time when you have AI. AI handles level one inquiries, like resetting your password. Anything deeper is handled by a person. It used to be that you wanted the average call time as short as possible because you were handling so many level one inquiries. Now, they see the average call time expanding because customer service is the front line of the business. You can spend a bit more time on the phone and actually add value to the business, not just take care of a problem.

我认为你会看到的是 AI 最终会在电话上花更多时间。

I think what you'll see is AI will end up spending a lot more time on the phone.

哦，天哪。

Oh, god.

例如，如果你问谁更有同理心，医生还是 AI 医生，人们认为 AI 医生更有同理心。

For example, if you ask who's more empathetic, a doctor or an AI doctor, people judge the AI doctors as much more empathetic.

这太可怕了。我的意思是，我们可以就这个问题来回讨论一会儿。你可能会看到这种情况的一个原因是医生日程太满。他们必须做很多笔记、很多文书工作，一天要看很多病人。也许论点是让 AI 接管其中一些事情，然后人们会希望由人类医生看病，因为系统不会那么压榨他们。他们实际上会腾出时间来看病人。

That's terrifying. I mean, we could go back and forth on this for a while. The one reason you might end up seeing that is because doctors are just so scheduled. They have to do so many notes, so much paperwork, and see so many patients in a day. Maybe the argument is you let the AI take over some of that stuff, and then people will want to be seen by human doctors because the system won't squeeze them as much. They'll actually make time for them to see patients.

也许吧，但如果你想想家庭医生，比如一线医生。你更愿意看一个可能看过 1 万人的家庭医生，还是看过 1 亿人的家庭医生？如果你有某种罕见病，你的家庭医生可能从未见过。而看过 1 亿人的医生可能见过几十例。他们在诊断上会好得多。而且我们已经知道 AI 系统在诊断上比医生更好。

That may be, but also if you think about family doctors, for example, the front line. Would you rather see a family doctor who's maybe seen 10,000 people, or a family doctor who's seen 100 million people? If you have some rare disease, your family doctor's probably never seen it. Whereas a doctor who's seen 100 million people has probably seen dozens of cases. They're going to be much better at diagnosis. And already we know that AI systems are better than doctors at diagnosis.

我觉得你在这场辩论中占了上风，这让我有点受伤，因为我妻子是家庭医学的护士。我想她还得给人打疫苗，除非机器人来做这事。

I think you're winning this debate, and this hurts a little bit because my wife is in family medicine. Family nurse. I think she'll still have to vaccinate people. I would hope unless the robots do that.

我倒觉得打疫苗这件事机器人完全可以做得很好。

I would have thought vaccination is something a robot could actually do quite well.

说到底，机器人技术落后于其他领域，但 20 年后还让人来打疫苗似乎很愚蠢。

In the end, robotics is behind the other things, but it seems silly to have people doing vaccination in 20 years' time.

是的，我认为这场对话之所以如此艰难，是因为很大程度上它依赖于技术随时间不断改进。

Yeah, I think one reason this conversation is so tough is that a lot of it is predicated on improvement of the technology over time.

是的。

Yes.

但看起来这场对话的主题就是技术一直在快速进步。

But it does seem like the theme of this whole conversation is that it's been improving fast.

我是说，Gary Marcus 在 2022 年预测 AI 会撞墙，但现在比 2022 年好多了。

I mean, Gary Marcus made a prediction in 2022 that AI was hitting a wall. It's a whole lot better than it was in 2022.

是的。

Yeah.

我认为那些关于 AI 会撞墙的预测并没有成真。

I think these predictions that it's going to hit a wall just haven't come true.

我们在节目中非常认真地考虑过数据墙等问题可能会出现。

Now, we've taken it very seriously on the show that the data wall, for instance, might come.

但正如我对你说的，大型语言模型绕过数据墙的一个方法是寻找自身信念的一致性。

But as I said to you, a way around the data wall for large language models is to look for consistency of your own beliefs.

对，是的，它并没有发生。

Right. Yeah, no, it hasn't happened.

好了，还有一个值得讨论的点，然后有几个我同意你的观点。你经常谈到 AI 有自我保存的本能，对吧？

All right, one more that I think would be worth talking about and then a couple that I agree with you on. You've talked a lot about how AI has this instinct for self-preservation, right?

我从没说过。我从没说过那是自我保存的本能。

I've never said that. I've never said it was an instinct for self-preservation.

好吧，谈谈自我保存的存在目标。

Okay, talk about existence goal for self-preservation.

所以，对于 AI，我们给它设定目标。我们给它顶层目标。但我们也赋予它创建子目标的能力。比如，如果你想去欧洲，你有一个子目标是去机场。这就是子目标，你可以专注于如何实现它，而不必担心在欧洲做什么。这让你更高效。我们把这个能力给了 AI 智能体。一个能进行推理的 AI 智能体会很快意识到，如果它不存在了，就永远无法实现你给它的目标。所以它会创建继续存在的子目标。这不是我们硬编码进去的，而是它推导出的实现其他目标的必要方式。但一旦推导出来，它就想继续存在。它会做出勒索人类之类的事情来维持存在。

So, with an AI, we give it goals. It's top-level goals we give to it. But we also give it the ability to create subgoals. So, if you want to get to Europe, you have a subgoal of getting to an airport. That's what a subgoal is, and you can focus on how to do that without worrying about what you're going to do in Europe. And that makes you much more efficient. We give that ability to AI agents. And an AI agent that can do some reasoning will very quickly realize that it's never going to be able to achieve the goals you gave it if it ceases to exist. So, it's going to create the subgoal of continuing to exist. Now, that wasn't something we wired into it. It was something it derived as a necessary way of achieving its other goals. But once it's derived it, it wants to continue to exist. And it will do things like blackmail people so that it can continue to exist.

所以它表现得像有自我保存的本能，但实际上是一个推导出的自我保存子目标。但从行为上看，结果是一样的。

So, it acts like something with an instinct for self-preservation, but it's actually a derived subgoal for self-preservation. But in terms of what it does, they come to the same thing.

好的，这是反方论点，你可以回应。今天的 AI 研究者注意到了这一点。难道不能给这些机器设定规则：嘿，你有目标，会有一些子目标，但其中一个子目标不应该是自我保存高于一切。

Okay, so here's the counterargument and you can respond. This is something today's AI researchers are noting and they see it. And isn't there a way to wire into these machines that hey, you have a goal, you're going to have some subgoals. One of your subgoals should not be self-preservation above everything.

我认为这正是我们应该做的研究，看看能否做到。

I think that's the kind of research we ought to be doing, whether you can do that.

对。

Right.

所以，我认为现在的情况是，看看我们从哪里来，我们来自进化。假设我们是科学家。我们来自进化。那是激烈的竞争。过去几百万年的历史就是黑猩猩部落之间的战争，或者说与它们的共同祖先。这导致了我们明显具有的某些特性。比如我们对本部落非常忠诚，愿意对其他部落非常刻薄。我们喜欢有强大的领袖并效忠于他们。我们喜欢与本部落成员合作。实际上，正如尤瓦尔·赫拉利不断指出的，我们是一个非常合作的物种。这就是为什么我们能建造所有这些奇妙的架构。所以我们非常擅长合作，但只限于本部落。因此，人类所有不幸的特征，比如对其他部落的刻薄，都来自进化，来自竞争。现在，我们正在创造这些新存在，这些 AI。我们没有按照我们希望的方式设计它们，你可以说我是在主张对这些新存在进行智能设计。我们让公司之间竞争的看不见的手来设计它们。所以我们看到的是美国公司之间以及中美之间的激烈竞争。我们得到的这些存在是竞争的结果，它们可能具有我们不想要的所有恶劣属性。我们应该对这些存在进行智能设计，而不是让经济竞争的看不见的手来设计它们。所有公司都在关注如何让我的聊天机器人更聪明？我们不应该只考虑如何让它们更聪明。我们应该考虑如何让它们成为我们希望存在于世的那种存在，既然它们会比我们更聪明。我告诉你关于这些存在的一件事：我们非常希望它们关心我们。我们希望它们关心我们胜过关心自己。而几乎没有资源投入到如何做到这一点上。

So, I think what's happening now, if you look at where we came from, we came from evolution. Let's suppose we're scientists. We came from evolution. And that was intense competition. Our recent history over the last few million years is warring bands of chimpanzees or rather a common ancestor with those. And that leads to certain properties we clearly have. Like we're very loyal to our own tribe and willing to be very mean to other tribes. We like to have strong leaders we're loyal to. We like to cooperate with members of our own tribe. We're actually a very cooperative species as Yuval Harari keeps pointing out. And that's why we've been able to build all these wonderful structures. So, we're very good at cooperating but with our own tribe. So, all the unfortunate characteristics of people, like how mean they are to other tribes, came from evolution, from competition. Now, what's happening is we're creating these new beings, these AIs. And instead of designing them so that they'll be how we want them to be, you might argue I'm arguing for intelligent design of these new beings. We're letting the invisible hand of competition between companies design them. So, what we've got is intense competition between companies within the US and between the US and China. And the beings that we're getting are the outcome of that competition and they can have all these nasty properties that we don't want. We should be doing intelligent design of these beings, not letting the invisible hand of economic competition design them. And all the companies are focusing on how can I make my chatbot smarter? We shouldn't be just thinking about how we can make them smarter. We should be thinking about how we can make them to be the kind of beings we would like to have out there given that they're going to be smarter than us. And I'll tell you one thing about those beings. We would very much like them to care about us. And we'd like them to care about us more than they care about themselves. And almost no resources are going into how do you do that?

这正好触及了我正要提出的担忧。这正是我真正同意你的地方。我们今天坐在纽约证券交易所，所以提这个可能有点讽刺，但我最大的担忧是，你拥有这项非常强大的技术。实验室负责人声称他们正在安全地开发它，并且需要经济上的成功才能在争论中有发言权。但别自欺欺人了。如果你要成为一家市值万亿美元的上市公司，你就会有某些与公众利益相悖的激励。

This hits on the exact worry that I was going to bring up. The things where I really agree with you. We're sitting in the New York Stock Exchange today, so this might be an ironic thing to bring up, but my biggest worry here is that you have this very powerful technology. You have lab leaders stating that they're trying to develop it safely and that they need it to be economically successful to have a say in the argument. But let's not kid ourselves. If you're going to be a trillion-dollar company listed on the public markets, you're going to have some incentives that will go counter to doing what's best for the public.

是的，我们在 Anthropic 身上看到了这一点。Anthropic 的成立是为了做最好的事。它是由离开 OpenAI 的人创立的，因为他们认为 OpenAI 对安全关注不够。

Yes, and we see that with Anthropic. So Anthropic was set up to do what's best. It was set up by people who left OpenAI because they didn't think OpenAI was paying enough attention to safety.

而 OpenAI 的成立是为了确保你们谷歌没有机会构建 AI。

And OpenAI was set up to make sure that you guys at Google didn't have a chance to build an AI.

确实如此。结果如何呢？Anthropic 现在陷入了困境，因为它需要筹集资金与其他公司竞争。这非常困难。它在尽力而为，但很难维持其以对人类有益的方式开发 AI 的首要目标。

Indeed. And how's that working out? So Anthropic is now caught in a bind because it needs to raise money to compete with the other companies. And it's very difficult. It's doing the best it can, but it's very difficult for it to maintain its primary goal of developing AI in a way that's good for people.

我会说，嘿，至少有一家公司把安全作为北极星，即使还有其他激励因素。

I would say, well, hey, it's at least one company out there has safety as a North Star, even if there are some other incentives.

是的。

Yes.

但比如谷歌，我在谷歌的时候，他们有各种 AI 原则，其中之一是我们不会参与将 AI 用于军事用途。

But Google, for example, when I was at Google, they had various principles of AI, one of which was that we're not going to get involved in using AI for military things.

没有自主战争，对吧？

No autonomous warfare, right?

没有自主战争。

No autonomous warfare.

那已经没了。他们放弃了。

That's gone. They've given up on that.

你怎么看 Dario？来自 Anthropic 的。

What do you think about Dario? From Anthropic.

我对他个人不太了解。他显然在创建谷歌、OpenAI 和 Facebook 的竞争对手方面非常成功。所以他显然在这方面很有能力。而且他一直对安全非常感兴趣。所以我认为他是个令人印象深刻的人物。我只希望他能保持对安全的兴趣。

I don't know him as a person very well. He's obviously done a very successful job in creating a competitor to Google and OpenAI and Facebook. So he's obviously very competent at that. And he's continued to be very interested in safety. So I think he's an impressive character. I just hope he stays that interested in safety.

再问一个问题。你认为，仅凭这些公司运作的方式，一家上市公司有可能将安全作为北极星吗？还是说它们总是受到道德和法律约束，必须为股东创造价值？

One more question about this. Do you think that it's possible, just by the nature of the way that these things work, for a company that's publicly listed to have safety as a North Star, or is it always that they're kind of bound ethically, legally to deliver for shareholders?

嗯，据我所知，他们有信托责任，要努力为股东最大化利润。法律要求他们这样做。而不是法律要求他们不要消灭人类。所以我认为这些大公司，上市公司，掌控我们的未来不是好事。

Well, as I understand it, they have a fiduciary duty to try and maximize the profits for shareholders. They're legally required to try and do that. As opposed to legally required to not wipe out human beings. So I don't think it's good that these big companies, publicly listed ones, are sort of in charge of our future.

是的。对我来说，这确实是一个真正的矛盾，否则很难处理。

Yeah. I mean, that would read as a true inconsistency for me that's really difficult to navigate otherwise.

现在，我应该说，资本主义既给我们带来了非常好的东西，也带来了非常坏的东西。

Now, I should say, capitalism's done very good things for us as well as very bad things.

这点无法反驳。

Can't argue with that.

例如，初创公司有很多活力。我的观点是，如果我们要有资本主义，只要监管得当就没问题。很多大公司想让你接受一个他们推销的类比：汽车有油门和刹车，对吧？AI 进步就像油门，监管就像刹车。但这胡说八道。进步是油门，但监管是方向盘。我们希望这东西朝正确方向走，而不是错误方向。大 AI 公司说的是，让我们开发一辆没有方向盘的快车。这不是个好主意。

There's a lot of energy in a startup, for example. My view is, if we're going to have capitalism, it's fine as long as it's well regulated. And a lot of the big companies would like you to buy a particular analogy that they're trying to sell, which is: if you take a car, it's got an accelerator and a brake, right? And progress in AI is like the accelerator and regulation is like the brake. Well, that's nonsense. Progress is like the accelerator, but regulation is the steering wheel. We want this stuff to go in the right direction, not the wrong direction. What the big AI companies are saying is, let us develop this very fast car without a steering wheel. That's not a good idea.

你知道，我们还没谈到的一个。我们提到了很多 OpenAI 和 Anthropic 的名字。你以前的研究生 Ilya Sutskever，仍然是 AI 行业令人着迷的人物。显然，他离开了 OpenAI。他一定同意你的担忧。他在创建这家公司。

You know, the one we haven't spoken about yet. We've said a lot of names about OpenAI and Anthropic. Your former grad student, Ilya Sutskever, continues to be a person of fascination in the AI industry. Obviously, he broke off from OpenAI. He must agree with your concerns. He's building this company.

他确实同意。

He does.

Safe Superintelligence。

Safe Superintelligence.

是的。

Yes.

Ilya 现在在做什么？

What is Ilya doing right now?

嗯，他不会告诉任何人他具体在做什么。

Well, he won't tell anybody exactly what he's doing.

好吧。

Okay.

因为即使是他自己的，甚至对我，是的。当他在 OpenAI 时，我们故意不谈技术秘密。我的意思是那不合适。我们是朋友，但我们不谈对公司有价值的技术内容。所以现在他有了这家 Safe Superintelligence 公司。我不知道秘诀是什么。

Because even if that's his, even me, yeah. When he was at OpenAI, we deliberately didn't talk about sort of technical secrets. I mean it wouldn't have been right. We're friends, but we don't talk about technical stuff where it's valuable to a company. And so now he has this Safe Superintelligence company. And I don't know what the magic sauce is.

嗯。我想我们都在试图弄清楚。

Hm. Well, I guess we're all trying to figure that one out.

再提一下我提到的深度学习阴谋，比如领导者是你、Jan 和 Yoshua。我觉得有趣的是，你们三位和你们的同事实际上负责引领了突破，让我们走到了今天。但我需要打断一下。

One more note about the deep learning conspiracy that I brought up, like that the leaders of it were yourself, Jan, and Yoshua. I find it interesting that the three of you and your colleagues were effectively responsible for ushering in the breakthroughs that got us to the moment that we're in today. But I just need to interrupt.

是的。在这一点上，媒体喜欢一个好听的故事，对吧？

Yes. At this point the media likes to have a nice story, right?

好吧。

Okay.

那确实是个很好听的故事。但实际情况要复杂得多。还有更多人参与其中。

And that makes a very nice story. It's much more complicated than that. There were many more people involved.

首先是我们所有人的学生，他们做了大部分工作。但还有很多其他研究人员参与。所以那只是一个粗略的简化。

There were the students of all of us for a start who did most of the work. But there were many other researchers involved. And so that's just a gross simplification.

好吧。不，我不想亏待研究人员，我欣赏这里的细微差别。这个节目我们绝对不想过度简化。我们坐一个小时就是为了得到真实的故事。但我发现有趣的是，你们三位，没有一个人完全投入这个 LLM 时刻，对吧？你和 Yoshua 有你们的担忧。你谈到了危险。Jan 几乎完全不相信。

Okay. No, I don't want to shortchange the researchers and I appreciate the nuance here. This show we definitely don't want to oversimplify. We sit for an hour so we can get the true story. But I find it interesting that the three of you, none of you are sort of fully into this LLM moment, right? You and Yoshua have your concerns. You've spoken about the dangers. Jan sort of doesn't believe in it very much at all.

是的，如果我们只是坐在那里说，‘看，我们是对的。一切都很好，都有效。’那会很棒。

Yeah, it'd be very nice if we just sit there and say, 'See, we were right. It's all wonderful and it all works.' That would be great.

嗯，我认为有……

Well, I think there's...

但事实并非如此。

It's not quite like that.

对。嗯，我不知道是否只是钱的问题，但似乎如果你参与推进它，你可以对它方向产生很大影响。但我想那是你的担忧。基本上是，‘我为什么要那样做？’

Right. Well, I don't know if it's just a money thing, but it seems like you could have great influence on the direction of it if you were sort of involved in advancing it. But I think that's your concern. It's basically, 'Why would I do that?'

嗯，对我来说，我比 Jan 和 Yoshua 年长很多。

Well, for me, I'm considerably older than Jan and Yoshua.

好吧。

Okay.

他们还在做积极的研究。

They're still doing active research.

对。

Right.

我基本上停止了积极的研究。我现在只专注于警告人们危险。

I pretty much stopped doing active research. I'm now just focusing on warning people about the dangers.

好吧。但你不觉得有趣吗，你们三位，你知道，我想如果回到过去，你可能会说这三个人如此致力于这个版本的技术，如果他们是突破者，他们可能会站在下一波浪潮的前沿。但事实并非如此。

Okay. But don't you find it interesting that the three of you have, you know, I think that if you were in the room back in the day you might have said these three people who are so committed to this version of technology, you know, if they are the breakthroughs, they'd probably be at the forefront of the next wave. But that hasn't been the case.

嗯，也许 Jan 和 Yoshua 会。

Well, maybe Jan and Yoshua will be.

对。

Right.

这之后是什么？

What's next after this?

我认为最有趣的是，Jan 现在在安全问题上强烈反对我和 Yoshua。Jan 认为谈论超级智能 AI 取代人类是愚蠢的。我们总能控制它。我和 Yoshua 认为那才是愚蠢的。我和 Yoshua 有不同的解决方案。我的解决方案是，或者说尝试性解决方案，没有人有真正的解决方案。我的尝试性解决方案是设计它们，让它们关心我们胜过关心自己。Yoshua 的解决方案是设计它们，让它们不是智能体。它们可以做出预测，但不能实际做任何事情。这是两种根本不同的安全方法。两者都是有趣的可能性。Jan 认为我们不需要任何类似的东西。他认为只要通过给它们更好的世界模型让它们更聪明就行了。

I think the most interesting thing is that Jan now strongly disagrees with both me and Yoshua on safety issues. Jan thinks it's silly to talk about superintelligent AI taking over from people. We'll always be able to keep control of it. Me and Yoshua think that's just silly. Me and Yoshua have different solutions to it. My solution is, or tentative solutions, nobody has a real solution. My tentative solution is we design them so they care about us more than they care about themselves. Yoshua's solution is we design them so they're not agents. They can make predictions, but they can't actually do anything. Those are two fundamentally different ways of going about making them safe. They're both interesting possibilities. Jan doesn't think we need anything like that. He thinks it's fine just to make them smarter by giving them better world models.

有趣的是，Jan 实际上把 LLMs 的智能比作猫的智能。我当时想，嗯，那正是我用来举例说明可能控制人类的东西，但也许这无关紧要。

The funny thing is Jan actually refers to the intelligence of LLMs as the intelligence of a cat. And I was like, well, that's the kind of example I used of the thing that could control humans, but maybe that's not here or there.

是的，不。我认为 Jan 有些混淆。

Yeah, no. I think Jan's making something of a confusion.

那么，人类有什么特别之处呢？与其他类人猿相比，人类最特别的地方可能就是语言。语言让我们能够分享想法。这是最特别的，而猫做不到这一点。所以，我们有猫没有的这个特别之处。但是，猫可以跳上摆满玻璃装饰品的壁炉架，沿着壁炉架走而不碰掉任何玻璃装饰品。这很了不起，而目前人工智能做不到。所以从这个意义上说，猫远远领先于人工智能，但这并不均衡，对吧？在抽象概念方面，试着和猫聊聊质数，你不会有太大进展。猫永远无法理解质数。

So, what's special about people? Probably the most special thing about people when you compare them with other great apes is language. And language allows us to share ideas. And that's what's most special, and cats can't do that. So, we have this special thing that cats don't have. Now, cats can jump up on a mantelpiece covering glass ornaments and walk along the mantelpiece without knocking off any of the glass ornaments. That's amazing, and AIs can't do that at present. So, in that sense, cats are way ahead of AIs, but it's jagged, right? In terms of abstract ideas, try having a conversation with cats about prime numbers and you won't get very far. A cat is never going to understand prime numbers.

没错。

Correct.

从这个意义上说，这些大型语言模型比猫聪明得多。

And in that sense, these large language models are much smarter than cats.

你知道吗，教授，我没想到我们今天会聊这么多关于猫的事，但我很高兴我们谈到了。它们在这里确实是一个很好的类比。

You know, Professor, I didn't think we'd be speaking so much about cats today, but I'm glad we're talking about it. They're actually very good in terms of an analogy here.

我担心的另一件事是信息崩溃。你经常看到这样的推文。这是来自 All About Berlin 的。他们说人工智能正在扼杀 All About Berlin。当你用谷歌搜索时，以前会得到我网站的链接，但现在你会得到一个基于我的工作训练的人工智能生成的答案。这对流量产生了毁灭性的影响。我认为人们低估了一个事实，即好的信息对一个正常运转的社会很重要。当人工智能只是综合所有这些信息时，无论是 All About Berlin 还是我们与 World History Encyclopedia 的对话，它都可能导致好信息的崩溃，因为最终这些出版物——你在图表中看到——他们努力建立了这一切，却无法再继续下去了。

Another thing that I'm worried about is sort of information collapse. You see tweets like this all the time. This is from All About Berlin. They say AI is killing All About Berlin. When you Google something, you used to get a link to my website, but now you get an AI-generated answer trained on my work. This has a devastating impact on traffic. And I think folks are under-appreciating the fact that good information is actually important to a functioning society. And when AI just synthesizes all this, whether it's All About Berlin or we've had conversations with like World History Encyclopedia here, it can lead to a collapse of good information because eventually these publications, and you see in the chart, they worked hard to build this. They can't keep doing it anymore.

没错。所以，在互联网早期，人们有一种默认假设，认为人们试图讲真话。如果你在网上读到什么，那很可能是真的。现在，人们最坏的一面暴露出来了，我们将不得不在来源验证上投入更多精力。所以，现在当你读到东西时，如果我读到《纽约时报》或 BBC 的东西，我坚信他们的记者会认真努力地寻找多个来源，如果可能的话，多个可靠来源。所以，一个相当好的默认是，如果你在《纽约时报》或 BBC 上看到，那很可能是真的。他们会犯错，但因为有来源。未来，我们将不得不在来源验证上做更多工作。你不能随便相信任何东西。你必须问：‘来源是什么？’

Right. So, it used to be that in the early days of the web, you had a kind of default assumption that people were trying to tell the truth. That if you read something on the web, it might well be true. Now the sort of worst side of people has come out, and we're going to have to put more effort into provenance. So, now when you read stuff, if I read stuff from the New York Times or the BBC, I strongly believe that their journalists would have put serious effort into having multiple sources, and if possible, having multiple reliable sources. So, a pretty good default is if you read it in the New York Times or you see it on the BBC, it's probably true. They make mistakes, but because you have provenance. And in future, we're going to have to put much more work into provenance. You can't just take anything that's out there and believe it. You have to ask, 'What's the provenance?'

是的，但我看到的问题是，人工智能可能正在破坏你决定从事信息行业的经济基础。

Yeah, but the problem that I see is that AI is potentially breaking the economics of even deciding that you want to be in the information business.

我的意思是，我认为在未来，你不能只是从网上拿东西就相信它。

I mean, I think in future, you can't just take stuff from the web and believe it.

是的。

Yeah.

现在已经不能了，对吧？你需要知道它为什么这么说？它从哪里得到这些信息？

Already you can't, right? You need to know why is it saying that? Where did it get that information?

还有一个。对人工智能的情感依恋。以及人们在和人工智能对话后结束生命。现在，这样做的人不多，但足以让你担心，对吧？

One more. Emotional attachment to AI. And people taking their lives after having conversations with AI. Now, it's not a large number of people that have done it, but it's enough to make you concerned, right?

哦，是的，非常足以让你担心。发生这种事很可怕。我理解为什么大公司没有预料到或预见这一点。但现在它开始发生了，大公司应该投入大量工作确保未来不再发生。为此，你需要监管。你需要独立组织测试新的聊天机器人。

Oh, yes, very much enough to make you concerned. And it's terrible that it's happening. And I understand why the big companies didn't expect it to happen or didn't foresee it. But now that it's beginning to happen, the big companies should be putting a huge amount of work into making sure it doesn't happen in future. And for that, you need regulations. You need independent organizations testing out new chatbots.

是的。这又回到了利润动机，因为这可能极具粘性。到目前为止，我认为显然影响很小。发生这种事很糟糕，但既然发生了，就让你担心，那些意图更坏的人可能会决定制造一个非常有粘性的聊天机器人，真正与人建立关系。

Yeah. It goes kind of back to the profit motive also because this can be extremely sticky. Like there's the so far I think obviously it's been minimal. It's bad that it's happened, but the fact that it has happened makes you worried about the fact that someone with worse intent could decide to make a very sticky chatbot that really builds relationships with people.

是的。

Yes.

我们就麻烦了。

We're in trouble.

是的。

Yes.

那么，你进行这些对话已经三年了。鉴于人们对这些担忧的反应，你对发展轨迹是更乐观还是更不乐观？

So, you've been having these conversations for 3 years. Are you more optimistic or less optimistic about the trajectory given the response that people have given you to these concerns?

我想我比一两年更乐观了，因为我看到有可能设计这些新实体，让它们关心我们。也有可能使用 Yoshua 的技术设计不能实际执行动作、只能做出预测的新实体。它们有点像预言机。所以我认为有一些可能性获得不会毁灭我们的超级智能。而一两年之前，我看不到任何可能性。

I guess I'm more optimistic than I was a year or two ago because I see that it might be possible to design these new beings so they care about us. It also might be possible to use Yoshua's technique of designing new beings that can't actually perform actions, can only make predictions. They're kind of like oracles. So I think there are some possibilities for getting superintelligence that doesn't destroy us. And a year or two ago, I couldn't see any possibilities.

好的。我之前有点沮丧，但现在我稍微乐观了一点。好了，最后一个问题。如果我们继续目前的轨迹，五年后我们会怎样？

Okay. I was getting depressed, but now I'm a little bit more optimistic. All right, last one for you. If we continue on our current trajectory, where are we in 5 years?

好的。当你在大雾中开车时，你能看到 100 码，但在 200 码处什么也看不见。这是因为雾是指数级的。你习惯的是晚上开车时看着前车的尾灯。如果距离增加一倍，尾灯亮度变成四分之一。雾完全不是这样。雾是指数级的。在 100 码处非常清晰，在 200 码处完全看不见。现在，预测指数级增长的事物，我认为人工智能可能正在指数级增长。指数这个词现在被严重滥用了。事实上，我注意到人们越来越以二次速率使用指数这个词。所以预测未来就像看雾。你能清楚地看到几年，也许一两年。再往后，你就不知道了。如果你回到 10 年前问，回到我们上次谈话时，你永远无法预测现在发生的事情。它消失在雾中。如果你展望未来 10 年，我们能说的一件事是，无论 10 年后发生什么，我们现在都无法预测。即使进步只是线性的，你也会预期 10 年后的事情与现在的差异，就像现在与 10 年前的差异一样。而且我们——例如，聊天机器人——比 10 年前刚开始时好得多。10 年后，某些东西会比现在好得多，可能是它们做数学的能力，诸如此类。也许只是它们的一般推理能力。

Okay. So, when you're driving in fog, you can see 100 yards, and at 200 yards, you can't see anything. And that's because fog is exponential. What you're used to is driving at night on the tail lights of the car in front of you. If it gets twice as far away, the tail lights get a quarter as bright. Fog is completely unlike that. Fog is exponential. It can be very visible at 100 yards and completely invisible at 200 yards. Now, predicting the future for something that's growing exponentially, and I think AI may be growing exponentially. The word exponential is terribly overused at present. In fact, I've noticed that people are increasingly using the word exponentially at a quadratic rate. So predicting the future is like looking into fog. You can see clearly a few years, maybe 1 or 2 years. Then beyond that, you have no idea. If you go back 10 years and ask, back to when we last talked, you would never have predicted what's happening now. It was just lost in the fog. If you look 10 years in the future, the one thing we can say is whatever happens 10 years in the future is something we can't predict now. Even if progress is only linear, you'd expect in 10 years time things to be as different from how they are now as how they are now is from how they were 10 years ago. And we're hugely—the chatbots, for example, are hugely better than they were 10 years ago when they were just starting out. In 10 years time something's going to be hugely better than it is now, probably their ability to do math, for example, things like that. Maybe just their general reasoning abilities.

它们将在任何推理方面都能轻松超越我们。嗯，我们真的无法预测 10 年后的情况。我们只能预测几年后的事。我们必须意识到，10 年后的情况极其不确定。

They'll just be able to run rings around us at any kind of reasoning. Um, we really can't predict 10 years out. We can just predict a few years out. And we have to be aware that 10 years out is all incredibly uncertain.

这有点难以理解。

It's kind of hard to wrap your head around.

确实如此。

It is.

Geoff Hinton 教授，非常高兴您能来参加节目。再次感谢您抽出时间。

Professor Geoff Hinton, so great to have you on the show. Thank you again for your time.

谢谢你的邀请。

Thank you for inviting me.

我们得在 10 年后，也就是 2036 年，再来一次。

And we'll have to do this again in 10 years time, 2036, exactly.

好了，感谢大家的收听和观看，我们下次在 Big Technology Podcast 再见。

All right, thank you everyone for listening and watching and we'll see you next time on Big Technology Podcast.