人类智能与超级智能：AGI 的未来

那么，人类智能会是可能性的上限吗？

So, is human intelligence going to be the upper limit of what's possible?

我认为绝对不是。

I think absolutely not.

我确实想知道这对人们意味着什么。我的意思是，如果我们到了人类智能被超级智能超越的地步，那对社会意味着什么？

I do wonder what all of this means for people. I mean, if we are getting to a point where essentially human intelligence is dwarfed by superintelligence, what does that mean for society?

这意味着巨大的转变。这实际上将结构性改变经济、社会以及方方面面。我们需要思考如何构建这个新世界。

It means a massive transformation. This is actually something which is going to structurally change the economy and society and all kinds of things. And we need to think about how do we structure this new world.

欢迎收听 Google DeepMind 播客，我是主持人 Hannah Fry 教授。AGI 即将到来，似乎每个人都在这么说。今天我的嘉宾是 Shane Legg，Google DeepMind 的首席 AGI 科学家兼联合创始人。Shane 几十年来一直在谈论 AGI，甚至在那时这还被他自己称为“疯狂边缘”。他被认为是推广了这个术语，并最早尝试弄清楚它可能是什么。在今天的对话中，我们将讨论 AGI 应该如何定义，当它到来时我们如何识别，如何确保它安全且合乎伦理，以及关键的是，一旦我们到达那里，世界会是什么样子。我必须告诉你，Shane 对于未来十年整个社会将如何受到影响非常坦诚。绝对值得继续收听。欢迎来到播客，Shane。我们上次和你谈话是五年前，那时你向我们讲述了你对 AGI 可能是什么样子的愿景，关于我们现在拥有的 AI 公民。你认为它们显示出 AGI 的小火花吗？

Welcome to Google DeepMind the podcast with me, your host, Professor Hannah Fry. AGI is coming. That's what everyone seems to be saying. Well, today my guest on the podcast is Shane Legg, chief AGI scientist and co-founder of Google DeepMind. Shane has been talking about AGI for decades, even back when it was considered, in his words, the lunatic fringe. He is credited with popularizing the term and making some of the earliest attempts to work out what it might actually be. Now, in the conversation today, we're going to talk to him about how AGI should be defined, how we might recognize it when it arrives, how to make sure that it is safe and ethical, and then crucially, what the world looks like once we get there. And I have to tell you, Shane was remarkably candid about the ways that the whole of society will be impacted over the coming decade. It's definitely worth staying with us for that discussion. Welcome to the podcast, Shane. We last spoke to you five years ago and then you were telling us your sort of vision for what AGI might look like in terms of the AI citizens that we've got now today. Do you think that they are showing little sparks of being AGI?

是的，我认为远不止火花。

Yeah, I think it's a lot more than sparks.

不止火花。

More than sparks.

哦，是的。所以我对 AGI 的定义，有时称为最小 AGI，是一个至少能完成人类通常能做的认知类事情的人工智能体。我喜欢这个标准，因为如果低于这个标准，感觉它无法完成我们期望人类能做的认知任务。所以感觉我们还没真正达到。另一方面，如果我把最低标准设得更高，那就设定在一个很多人实际上无法完成我们要求 AGI 做的事情的水平上。所以，你知道，我们认为人类有某种，怎么说，通用智能，你可以这么叫。所以感觉如果 AI 能完成人类通常能做的认知类事情，至少可能更多，那么我们应该把它归入那一类。

Oh yeah. So my definition of AGI, or sometimes called minimal AGI, is it's an artificial agent that can at least do the kinds of cognitive things people can typically do. And I like that bar because if it's less than that, it feels like it's failing to do cognitive things that we'd expect people to be able to do. So it feels like we're not really there yet. On the other hand, if I set the minimal bar much higher than that, I'm setting it at a level where many people, a lot of people wouldn't actually be able to do some of the things we're requiring of the AGI. So, you know, we believe people have some sort of, I don't know, general intelligence, you might call it. So it feels like if an AI can do the kinds of cognitive things people can typically do, at least possibly more, then we should sort of consider it within that kind of a class.

我们现在拥有的东西，在这些水平上处于什么位置？

The stuff that we have now, where is it on those levels?

对。所以它是不均衡的。它已经比人类在语言方面好得多。它能说 150 种语言之类的。没人能做到。它的常识非常惊人。我可以问它我长大的郊区，新西兰的一个小镇，它碰巧知道一些事情。另一方面，它们仍然无法完成我们期望人类通常能做的事情。它们不擅长持续学习，长时间学习新技能，而这非常重要。例如，如果你开始一份新工作，你不需要在入职时就精通一切，但你需要随着时间的推移学习。推理方面也有弱点，特别是视觉推理。所以 AI 非常擅长识别物体。它们能识别猫狗之类的东西。它们已经能做到这一点一段时间了。但如果你让它们推理场景中的事物，它们就变得很不稳定。所以你可能说，嗯，你看到一辆红车和一辆蓝车，你问它们哪辆车更大。人们理解有透视关系，可能蓝车更大，但因为远所以看起来小，对吧？AI 不太擅长这个。或者如果你有一个带节点和边的图表，比如网络，或者数学家说的图，你问相关问题，它需要数出从一个节点出发的边数。人类通过关注不同点然后心里数数来做。AI 不太擅长这类事情。所以我们目前看到各种这样的问题。我不认为这些有任何根本性障碍，我们有想法开发能做这些的系统，我们看到这些领域的指标在随时间改善。所以我的预期是几年内这些问题都会解决，但还没到，我认为需要一点时间，因为人类能做的各种认知事情有很长的尾巴，AI 仍低于人类表现。当我们达到那个点，我认为几年内就会到来，具体时间不确定，AI 会可靠得多，这将在许多方面大大增加它们的价值，但在此期间它们也会变得越来越有能力，达到专业水平甚至更高，也许在编程、数学、多语言常识等方面已经如此。所以这是不均衡的。

Right. So it's uneven. So it's already much, much better than people at, say, speaking languages. So it'll speak 150 languages or something. Nobody can do that. And its general knowledge is phenomenal. I can ask it about, you know, the suburb I grew up in, a small town in New Zealand, and it happens to know things about it. Right. On the other hand, they still fail to do things that we would expect people typically be able to do. They're not very good at continual learning, learning new sorts of skills over an extended period of time, and that's incredibly important. For example, if you're taking on a new job, you know, you're not expected to know everything to be performant in the job when you arrive, but you have to learn over time to do it. There are also some weaknesses in reasoning, particularly things like visual reasoning. So the AIs are very good at, say, recognizing objects. They can recognize cats and dogs and all these sorts of things. They've done that for a while. But if you ask them to reason about things within a scene, they get a lot more shaky. So you might say, well, you know, you can see a red car and a blue car and you ask them which car is bigger. People understand that there's perspective involved and maybe the blue car is bigger, but it looks smaller because it's further away, right? AIs are not so good at that. Or if you have some sort of diagram with nodes and edges between them, like a network, yeah, or a graph as a mathematician would say, and you ask questions about that and it has to count the number of, you know, edges, spokes that are coming out of one of the nodes on the graph. A person does that by paying attention to different points and then actually mentally maybe counting them or what have you. The AI is not very good at doing that type of thing. So there are all sorts of things like this that we currently see. I don't think there are fundamental blockers on any of these things and we have ideas on how to develop systems that can do these things and we see metrics improving over time in a bunch of these areas. So my expectation is over a number of years these things will all get addressed, but they're not there yet and I think it's going to take a little bit of time to go through that because it's quite a long tail of all sorts of cognitive things that people can do where the AIs are still below human performance. As we reach that, and I think that's coming in a few years, unclear exactly, the AIs will be a lot more reliable and that will increase their value quite a lot in many ways, but they will also during that period become increasingly capable, like to professional level and beyond, and maybe in coding, mathematics already, in known multilingual general knowledge of the world and stuff like this. So it's kind of an uneven thing.

如果你认为它们会随时间变得更可靠，那只是让模型更大、更大规模地做事、更多数据的问题吗？我的意思是，你有明确的路径让它们更可靠吗？

If you think that they will become more reliable over time, how is it just a question of making the models bigger, doing things at larger scale, is it more data? I mean, do you have a clear path to make them more reliable?

呃，我认为我们有，而且不是某一件事。不仅仅是更大的模型或更多的数据。在某些情况下是特定种类的更多数据。当你收集需要视觉推理的数据时，模型就会学习如何做。在某些情况下需要算法上的东西，比如内部的新流程。例如，如果你想做持续学习，让 AI 随时间不断学习，你可能需要某种过程，新信息可能存储在某种检索系统中，如果你愿意，可以称为情景记忆。然后你可能会有系统，让这些信息随时间训练回底层模型。所以这不仅仅是更多数据的问题。它需要某种算法和架构上的改变。所以我认为答案是这些因素的组合，取决于具体问题。

Uh, I think we do and it's not one particular thing. It's just not bigger models or more data. In some cases it's more data of a particular kind. And then when you collect data that requires that, say visual reasoning, then the models learn how to do it. In some cases it requires algorithmic things like new processes within. So for example, if you want to do continual learning, so the AI keeps learning over time, you might need some process whereby new information is maybe stored in something, some sort of retrieval system, an episodic memory if you like. And then you might have systems whereby that information over time is trained back into some underlying model. So that requires more than just more data. It requires some sort of algorithmic and architectural changes. So I think the answer is a combination of these things and it depends on what the particular issue is.

我知道你认为 AGI 不应该是简单的“是/否”阈值，而更像是一个有不同层级的谱系。请详细说说。

I know that you don't think AGI should be this single yes/no threshold, but more of a spectrum with levels. Talk me through that.

是的。我提出了一个概念叫“最小 AGI”。指的是人工智能体至少能完成我们通常期望人类完成的所有认知任务。我们还没到那一步，但可能是一年、五年，我猜大概两年左右。

Yeah. So I have what I call minimal AGI. That's when an artificial agent can at least do all the cognitive things we'd typically expect people to do. We're not there yet, but it could be one year, five years. I'm guessing about two years.

所以那是最低层级。

So that's the lowest level.

那就是最小 AGI。在这个点上，AI 不再以人类完成认知任务时令人惊讶的方式失败。这是最低门槛。但这并不意味着我们完全掌握了如何达到人类智能的能力，因为非凡的人可以完成惊人的认知壮举——发明新的物理理论、创作交响乐、写文学作品。仅仅因为我们的 AI 能完成典型的人类认知，并不意味着我们掌握了实现非凡壮举的所有配方。一旦我们能用 AI 实现人类认知的全部范围，我们就完全达到了人类水平。我们称之为“完全 AGI”。

That's minimal AGI. It's the point where the AI no longer fails in ways we'd find surprising if a person did that cognitive task. That's the minimum bar. But that doesn't mean we fully understand how to reach the capabilities of human intelligence, because extraordinary people can do amazing cognitive feats—inventing new theories in physics, composing symphonies, writing literature. Just because our AI can do typical human cognition doesn't mean we know all the recipes for extraordinary feats. Once we can achieve the full spectrum of human cognition with AI, we've nailed it fully to human level. We call that full AGI.

那还有更高的层级吗？

And then is there a level beyond that?

有的。一旦你开始超越人类认知的可能性，就进入了人工超级智能，即 ASI。目前没有明确的定义。我尝试过提出一个，但每个定义都有重大问题。粗略地说，它指的是一个在通用能力上远超人类所能达到的 AGI。

Yeah. Once you start going beyond what's possible with human cognition, you head into artificial superintelligence, or ASI. There aren't clear definitions. I've tried to come up with one, but every definition has significant problems. Vaguely, it means an AGI that is so capable in general that it's far beyond what humans can reach.

你是帮助创造“AGI”这个词的人之一。你认为它还有用吗？现在有太多相互竞争的定义，而且它常被描述为一条离散的线，而不是一个连续谱。

You were one of the people who helped coin the phrase AGI. Do you think it's still useful? There are so many competing definitions now, and it's often described as a discrete line rather than a continuum.

当我提出这个术语时，我更多是把它看作一个研究领域。当时我和 Ben Goertzel 聊天，他想写一本关于 AI 旧愿景的书——能够做很多不同事情的思维机器，而不是只做专门任务，比如打扑克或语音合成。我说，为什么不把“通用”放进名字里，叫它“人工通用智能”呢？AGI 读起来很顺口。但后来人们开始在网上使用这个词，很快它就变成了一个需要定义的物品类别。也许没有尽早定义是个错误。后来我们发现 Marco Brud 在 1997 年的一篇论文中用过这个词，但我们当时不知道。他的定义和我的类似。如果早期就明确固定下来，那会很有帮助。

When I proposed the term, I was thinking of it more as a field of study. I was talking to Ben Goertzel, who wanted to write a book about the old vision of AI—thinking machines that can do many different things, not just specialized tasks like playing poker or text-to-speech. I said, why not put 'general' in the name and call it artificial general intelligence? AGI rolls off the tongue. But then people started using it online and soon it became a category of artifacts needing a definition. Maybe it was a mistake not to define it early. Later we found Marco Brud had used the term in a 1997 paper, but we didn't know. His definition was similar to mine. If it had been fixed clearly early on, that would have been helpful.

你后悔创造这个词吗？

Do you regret coining it?

不，不后悔。因为它给了人们一种方式来指代构建真正通用 AI 的想法。当时有这个需求，所以它流行起来了。不然你怎么称呼它呢？人们用“高级 AI”这样的短语，但 AlphaFold 在某种意义上也是高级 AI，却非常狭窄。AlphaGo 也很狭窄。那么你怎么指代非常通用的系统呢？但不同的人通过不同的视角改编了这个术语。有些人认为 AGI 是几十年后的事情，并且具有变革性，所以他们根据它带来的变革来定义。另一些人则把它看作一个历史时间点，届时 AI 属于与我们智能相似的类别。这不一定革命性地改变世界——普通人不是莫扎特或爱因斯坦——但这是一个重要时刻，因为 10 或 20 年前，AI 还远不能完成人类典型的认知任务。

No, no. Because it gave people a way to refer to the idea of building AIs that are actually general. There was a need for that, and that's why it caught on. How else would you refer to it? People use phrases like 'advanced AI,' but AlphaFold is advanced AI in some sense, yet very narrow. AlphaGo is also narrow. So how do you refer to systems that are very general? But different people adapted the term through different lenses. Some thought of AGI as something decades away and transformative, so they defined it by the transformation it would create. Others see it as a historical point in time when AIs belong in a similar category to our intelligence. That doesn't necessarily revolutionize the world—the typical person isn't Mozart or Einstein—but it's an important moment because 10 or 20 years ago, AIs were nowhere near doing typical human cognitive tasks.

我也认为尝试定义它是有用的，因为人们有不同的时间线。有人说 AGI 将在 3 年内到来。

I also think it's useful to try to define it because people have different timelines. Some say AGI will be here in 3 years.

哦，我觉得还要 15 年或 20 年之类的。而且我经常去和他们讨论时，发现他们用的是不同的定义。这就导致了很多混淆，因为人们用这个词指代不同的东西。在某些情况下，我实际上同意他们认为会发生的事情，只是他们用词方式不同。这造成了相当大的混乱。

Oh, I think it's going to be 15 years away or 20 years or whatever. And often when I go and talk to them about that, I find that they're using a different definition. And so that just leads to a lot of confusion because people use the term to mean different things. And in some cases, I actually agree with what they think is going to happen. They're just using the word in a different way. And that just creates quite a lot of confusion.

我想比较一下人们用于 AGI 的其他一些定义。有些人建议它像是一个任务清单，或者可能是‘人类最后的考试’，这是一个跨不同学科（人文学科和自然科学）的两千五百道题的语言模型基准。

I just want to compare some of the other definitions that people are using for AGI. So some people have suggested that it's like there's a checklist of tasks or maybe there's 'Humanity's Last Exam' which is this language model benchmark of two and a half thousand questions across different subjects, humanities and natural sciences.

还有其他人说，‘哦，它需要能在厨房里操作，经过厨师培训，并能被放到不同的厨房里表现。’甚至还有一个是，‘它能用 10 万美元赚到 100 万美元吗？’你对这些定义怎么看？

There are other people that have said, 'Oh, it needs to be able to perform in a kitchen, trained as a chef, and be able to be dropped into a different kitchen and perform.' Or there's even one which is, 'Could it be able to make a million dollars from $100,000?' What's your take on those definitions?

嗯，每个我都有看法。请继续。

Well, each one I have a take on. Go ahead.

我的意思是，用 1000 美元赚到 100 万美元之类的。这显然是一种非常经济学的视角。我认为很多人都会很难做到。这在某些方面是一个非常狭隘的视角。也许你可以有一个交易算法，在市场上交易，能做到这一点，但它只能做这个。这不是我所说的。我认为是 AGI 中的‘G’，即通用性，我觉得有趣。我认为人类思维令人难以置信的一点就是我们的灵活性和通用性，能做很多很多不同的事情。如果你有一组特定的任务，好吧，也许你可以构建一个能做这些任务的系统，但它可能仍然无法完成我们期望几乎任何人都能完成的基本认知任务。我觉得这令人不满意。就像，‘哦，我们的 AI 又失败了，因为它不理解那个我期望几乎任何人都能理解的非常简单的事情。’所以我操作化我的定义的方式是，我会有一套任务，我知道人类的典型表现，然后看 AI 是否能完成所有这些任务。如果它在任何一项任务上失败，它就不符合我的定义，因为它不够通用。

I mean, make a million dollars from $1,000 or something like that. That's obviously a very economic kind of perspective on it. I think a lot of people would struggle to do that. It's a very, in some ways, quite narrow perspective on this. Maybe you could have a trading algorithm that trades on the markets that could do that, but that's all it can do. That's not what I'm talking about. I think it's the G that's the G in AGI. It's the generality that I find interesting. And I think that's one of the incredible things of the human mind: our flexibility and generality to do many, many different things. If you have a particular set of tasks, well, okay, maybe you can build a system that can do those tasks, but maybe it's still failing to do basic cognitive things that we'd expect almost anybody to be able to do. I think that's unsatisfying. It's like, 'Oh, our AI just failed again because it doesn't understand that really simple thing that I would expect pretty much anybody to understand.' So the way I would operationalize my definition is I would have a suite of tasks where I know what typical performance is from humans, and I would see whether the AI can do all those tasks. Now if it fails at any of those tasks, it fails to meet my definition because it's not general enough.

是的。它未能完成我们期望人们能做的某些认知事情。如果它通过了，那么我建议我们进入第二阶段，更具对抗性。我们说，‘好吧，它通过了一连串测试，所以它在我们标准集合中的任何任务上都没有失败，不管我们有几千个测试。现在让我们做一个对抗性测试：找一队人，给他们一两个月，允许他们查看 AI 内部，允许他们做任何他们想做的事。他们的工作是找到我们相信人们通常能做、且是认知性的、而 AI 失败的事情。’如果他们能找到，那么根据定义它失败了。如果他们经过几个月的探查、测试、挠头尝试后仍然找不到，我认为对于几乎所有实际目的来说，我们就达到了，因为这些失败案例现在非常难找。即使是一队人经过长时间也找不到这些失败案例。

Yeah. It's failing to do some cognitive thing that we'd expect people to be able to do. If it passes that, then I would propose we then go into a second phase which is more adversarial. And we say, 'Okay, it passed the battery of tests, so it's not failing at anything in our standard collection of however many thousands of tests or whatever we have. Now let's do an adversarial test: get a team of people, give them a month or two, they're allowed to look inside the AI, they're allowed to do whatever they like. Their job is to find something that we believe people can typically do and it's cognitive where the AI fails at.' If they can find it, it fails by definition. If they can't after a few months of probing it and testing it and scratching their heads and trying to find it, I think for intensive purposes, most practical purposes, we're there, because these failure cases are now so hard to find. Even teams of people after an extended period of time can't even find these failure cases.

你认为我们最终会就智力或 AGI 的定义达成一致吗？

Do you think that we'll ever agree on a definition of what intelligence is or what AGI is?

确实。就 AGI 本身而言，我的猜测是，几年后，AI 将在许多不同方面变得如此通用，人们就会直接称它们为 AGI，而 AI 恰好就指那些东西。也许人们会少一些担忧，少一些关于这是否是 AGI 的争论。人们会说，‘哦，我有最新的 Gemini 9 之类的。它真的很棒。它能写诗。你可以教它一个你刚编的纸牌游戏，它就能和你玩。它能做数学。它能翻译东西。它能和你一起计划假期等等，对吧？’它真的非常通用，人们会很明显地觉得它具备某种智能的通用性。但就目前而言，在达到那一步之前，拥有这样一条通往 AGI 的明确路径，我的意思是，你谈到没有它的风险，比如它可能在某个领域先获得知识，比如在伦理变得很好之前先擅长化学工程。我的意思是，在达到那一步之前，现在做这项工作有多重要？所以是围绕理解它在不同维度上的能力的工作。

Indeed. In terms of AGI itself, my guess is that some years from now, the AIs will become so generally capable in so many different ways, people will just talk about them as being AGI and AI will just happen to mean those things. And maybe people will be less worried about, they will have less arguments about whether this is an AGI or not. People will say, 'Oh, I've got the latest Gemini 9 or whatever it is. And it is really good. It can write poetry. You can teach it a card game and it can play with you that you just made up. It can do math. It can translate things. It can plan a holiday with you or whatever, right?' It's really, really generally capable and it'll just seem obvious to people that it has some sort of generality of intelligence. But then for now, in terms of having before we get there, having this kind of defined path on the route to AGI, I mean, you talk about the risks of not having one, that it could acquire a certain piece of knowledge before another, for instance, being good at chemical engineering before it gets really good at ethics. I mean, how important is it to have this work now in advance of getting there? So work around understanding its capabilities in different dimensions.

我认为这非常重要，因为我们必须思考社会如何应对强大机器智能的到来。你不能只把它放在一个维度上。它可能在有些事情上超人类，在其他一些领域非常脆弱和薄弱。如果你不了解这个分布是什么样的，你就不会理解存在的机会，也不会理解风险或被误用的方式，因为，你知道，它在这里超级强大，但你需要明白它在这里非常非常薄弱，所以某些事情可能会出错。所以我认为这是社会导航和理解当前状况的重要部分。所以你知道，我认为很多关于 AI 的对话已经倾向于说它如此强大，或者说它其实没那么强大，被过度炒作之类的。我认为现实要复杂得多。它在某些方面极其强大，在其他方面则相当脆弱。

I think it's very important because we have to think about how society navigates the arrival of powerful capable machine intelligence. And you can't just put it on a single dimension. It may be superhumanly capable at some things. It may be very fragile and weak in some other areas. And if you don't understand what that distribution looks like, you're going to not understand the opportunities that exist. You're also not going to understand the risks or the ways in which it could be misapplied because, you know, it's super capable over here, but you need to understand that it's very, very weak over here and so certain things can go wrong. So I think it's just an important part of society navigating and understanding what the current situation is. So you know, I think a lot of the dialogue around AI already tends to talk about it as being so capable or sort of being not really that capable and it's overhyped or whatever. I think the reality is much more complicated. It is incredibly capable in some ways and it is quite fragile in others.

你基本上必须看全局。

You have to take the whole picture essentially.

你必须看全局。是的。就像人类智能一样。你知道，有些人非常非常擅长，他们会说很多种语言。有些人非常擅长数学。有些人非常擅长音乐，但可能在其他方面不那么擅长。

You got to take the whole picture. Yeah. And it's like, you know, human intelligence as well. You know, some people are really, really good. They speak a whole bunch of languages. Some people are really good at math. Some people are really good at music, but maybe they're not so good at something else.

那么，好吧，我们有了性能和通用性。我想和你讨论的另一个方面是伦理。这如何融入这一切？

So, okay, if we've got sort of performance and generality. The other sort of arm of this that I want to talk to you about is ethics. How does that fit into all of this?

伦理和 AI 有很多方面。

There are many aspects to ethics and AI.

一个方面是，AI 本身是否对道德行为有很好的理解，是否能够根据这种道德行为分析它可能做的事情，并且以一种我们可以信任的方式稳健地做到这一点。

One aspect is simply does the AI itself have a good understanding of what ethical behavior is and is it able to analyze possible things it can do in terms of this ethical behavior and do that robustly in a way that we can trust.

所以 AI 本身可以推理它正在做的事情的道德性。

So the AI itself can reason about the ethics of what it's doing.

那它是如何工作的呢？你如何将其嵌入到 AI 中？

How does that work then? How do you embed that within it?

我对此有一些想法，但这还不是一个已解决的问题，不过我认为这是一个非常重要的问题。我喜欢一些人称之为思维链监控的方法。我称之为系统二安全，这借鉴了丹尼尔·卡尼曼的系统一和系统二思维。基本思路是这样的：作为一个人，当你面临困难的道德情境时，仅凭直觉通常是不够的。你需要坐下来思考：这是什么样的情境，有哪些复杂性和细微差别，可能采取的行动有哪些，不同行动的可能后果是什么，然后根据你所持有的一套伦理、规范和道德体系来分析所有这些。你可能需要深入推理才能真正理解这一切如何关联，然后利用这种理解来决定应该做什么。人类大脑在这种情况下是这样工作的：有人惹恼了你，你怒火中烧，想要反应——这是你的系统一，快速、直觉性的思维。但你深吸一口气，仔细思考，考虑后果——这是你的系统二思维，然后你可能会选择不同的路径。例如，撒谎是不好的，所以我们不应该撒谎。但你可能遇到这样的情况：坏人要来抓某人，如果你撒谎，就能救他的命。那么道德的做法可能就是撒谎。简单的规则并不总是足以做出正确的决定。所以有时你需要逻辑和推理来思考：在这种情况下，撒谎救人实际上是道德的。但这变得非常复杂。你可能听说过所有那些电车难题，我们的直觉和分析在某些情况下会分歧，造成混乱。这绝非简单的领域。我们现在有能够进行这种思考的 AI，你实际上可以看到它们使用的思维链。当你给 AI 一个带有道德或伦理方面的问题时，你可以看到它推理这个情境。如果我们能让这种推理非常严谨，并且对我们希望它遵守的伦理和道德有很强的理解，我认为原则上它应该能变得比人类更道德，因为它能更一致地以超人类水平推理它面临的选择。这就把伦理变成了一个推理问题，而不仅仅是一种感觉。

I have a few thoughts on that but it's not a solved problem, but it's a very important problem. I like something which some people call chain-of-thought monitoring. I call it system two safety, and this is the Daniel Kahneman system one system two thinking. The basic idea is something like this. As a person, if you're faced with a difficult ethical situation, it's often not sufficient just to go with your gut instinct. You actually need to sit down and think about the situation, the various complexities and nuances, the possible actions, the likely consequences, and then analyze all of that with respect to some system of ethics and norms and morals that you have. You may have to reason about that quite a bit to understand how it all fits together and then use that understanding to decide what should be done. The way the human brain works in this situation is that someone annoys you, you have a rush of anger, you want to react—that's your system one, quick thinking instinctive. But you take a breath, you think it through, consider the consequences—that's your system two thinking, and then you might choose a different path. For example, lying is bad, so we're not going to lie. But you could be in a situation where bad people are coming to get somebody, and if you tell a lie, you can save their life. Then the ethical thing to do is maybe to lie. The simple rule is not always adequate to make the right decision. So sometimes you need logic and reasoning to think through that in this case, it is actually ethical to tell a lie and save someone's life. But it gets very complicated. You've probably heard of all these trolley problems where our instincts and analysis diverge and cause confusion. This is not simple territory. We have AIs now that do this thinking, and you can actually see the chain of thought they use. When you give an AI a question with a moral or ethical aspect, you can see it reason about the situation. If we can make that reasoning really tight with a strong understanding of ethics and morals we want it to adhere to, I think it should in principle become more ethical than people, because it can more consistently apply and reason at a superhuman level about the choices it faces. That switches ethics into a reasoning problem rather than just a feeling thing.

但与此同时，我确实对基础（grounding）有所疑问。这些东西目前肯定不像人类那样生活在世界中。是否有可能从人类视角体验世界的感觉，并真正将这些机器扎根于人类伦理？

But then at the same time, I do wonder about grounding. These things certainly for now are not living in the world as humans. Is it possible to take what it feels like to experience the world from a human perspective and truly ground these machines in human ethics?

有几个复杂之处。一是并不存在单一的人类伦理。不同的人、文化和地区对此有不同的看法。所以 AI 需要理解在某些地方规范和期望略有不同。在某种程度上，模型确实知道很多，因为它们吸收了来自世界各地的数据。但它在扎根于现实方面需要做得非常好。目前，我们通过从世界收集大量数据来构建这些智能体，将它们训练成大型模型，然后它们成为相对静态的对象，我们与之交互。它们不会真正学到很多新东西。这种情况正在改变，我们引入了更多的学习算法，同时也在使系统更具智能体性。所以它们不再只是你与之交谈、处理并给出响应的系统，而是可以去做某事的系统。你可以说，‘我想让你写一个做某事的软件’，或者‘为我的墨西哥之行制定一个计划’。这些智能体也将开始更多地体现在机器人技术中。有些将是软件智能体。它们会做这类事情。随着时间的推移，它们会出现在机器人中。随着你沿着这条道路前进，AI 通过各种不同的事物与现实连接得更紧密。它们实际上必须通过互动和经验来学习，而不仅仅是从一开始的大型数据集。这就是与现实连接大大加强的地方。话虽如此，一开始注入它们的大量数据来自人类，所以通过这个过程也有对现实的扎根。

There are a few complexities. One is that there is not one human ethics. There are different ideas about this that vary between people, cultures, and regions. So it will have to understand that in certain places the norms and expectations are a bit different. To some extent, the models do know quite a lot of this because they absorb data from all around the world. But it will need to be really good at that in terms of grounding in reality. At the moment, we're building these agents by collecting lots of data from the world, training them into these big models, and then they become relatively static objects that we interact with. They don't really learn much new. That's shifting, and we're bringing in more learning algorithms, but we're also making the systems more agentic. So they're not just a system you talk to that processes and gives a response, but a system that can go and do something. You can say, 'I want you to write some software that does such and such,' or 'come up with a plan for my trip to Mexico.' Those agents will also start to become more embodied in robotics. Some will be software agents. They'll do those sorts of things. With time, they'll turn up in robots. As you keep going along this track, the AIs become more connected to reality through all sorts of different things. They actually have to learn through interaction and experience rather than just from a large dataset at the beginning. That's where the connection to reality tightens up a lot. That said, a lot of this data that was poured into them at the beginning came from people, so there is a grounding to reality that comes via that process as well.

AI 比人类更擅长伦理这个想法。在你达到那个水平之前，在推理能力和我们一样好之前，你如何确保它以安全的方式实施？例如，功利主义论点在无人驾驶汽车上很有效——尽可能多地拯救生命。但在医学上，同样的想法就行不通了。你不能牺牲一个健康的病人来拯救其他五个。你如何确保它最终朝着正确的方向推理？

This idea of the AI being better at ethics than humans themselves. How do you, until you get there, until the reasoning is as good as ours, make sure that it's implemented in a safe way? For example, a utilitarian argument works well for driverless cars—save as many lives as possible. But in medicine, that same idea doesn't work. You can't sacrifice one healthy patient to save five others. How do you make sure it ends up reasoning in the correct direction?

你无法保证一切。

You can't guarantee everything.

世界上行动的可能性空间如此巨大，100% 的可靠性是不存在的。但在现实世界中，很多事情本来就不是 100% 可靠的。比如你需要做手术，去找医生，你说，‘我要切除某个东西之类的。’医生对你说，‘100% 安全。’作为一个数学家，你知道他在说谎，对吧？没有什么是 100% 的。所以我们要做的是测试这些系统，让它们尽可能安全可靠。我们必须在收益和风险之间权衡。我们还需要做其他事情，比如监控。当它们部署后，我们进行监控，跟踪情况。如果我们开始看到出现超出我们可接受范围的失败案例，我们可能需要回滚、停止它们或采取其他措施，对吧？所以我们需要做一系列不同的事情。我们需要在发布前进行测试，需要在它们运行时进行监控。我们需要做可解释性之类的工作，以便能够查看系统内部。这是系统二的一个好处。如果安全措施实施得当，你实际上可以看到它的推理过程。但你必须检查这个推理是否准确反映了它真正想做的事情。如果你有办法查看系统内部，真正理解它们为什么这么做，那可能会给你另一层保证，确保它们试图以正确的方式行事，因为这是另一个重要的微妙之处。这不仅仅是关于结果，还关乎意图。故意伤害你和意外撞到你导致受伤，这两者有很大区别。我们对这两者的解读截然不同。所以如果我们能看清 AI 的内部，我们可能会接受它当时是在处理一个棘手的情况，它根据分析尽力做了最好的事情，但产生了一些负面副作用。我们可能对此还能接受，因为即使是人类，在那种棘手的情况下，也很难做出正确的事情。但如果它故意做错事，那就完全不同了。这些都是 AI 和 AGI 安全的各个方面，我们有人员在这些课题上工作。

The space of possibilities of action in the world is so huge that 100% reliability is not a thing. But it's not a thing in a lot of the world as it exists. If you need a surgery and you go and talk to the surgeon and you say, 'Well, you know, I'm going to get something removed or whatever.' And the surgeon says to you, 'It's 100% safe.' As a mathematician, you know that they're not telling you the truth, right? Nothing is ever 100%. So what we have to do is we have to test these systems and make them as safe and reliable as possible. And we have to trade off the benefits and the risks. And we also have to do other things like monitor them. So when they're in deployment, we monitor them, keep track of what's going on. So if we start seeing that there are failure cases that are beyond what we consider acceptable, we may have to roll back and stop them or do whatever, right? So there's a whole range of different things we need to do. We need to do testing before it goes out. We need to monitor it when they are out there doing things. We need to do things like interpretability, so we're able to look inside the system. That's one nice thing about system two. If it's safety, if it's implemented the right way, you can actually see it reasoning about things. But you got to check that this reasoning is actually an accurate reflection of what it's really trying to do. But if you have ways to look inside the system and really see why they're doing things, that can maybe give you another level of reassurance as to that they are trying to act in the right way, because that's another important subtlety. It's not always just about the outcome but maybe the intention. So there's a big difference between somebody hurting you intentionally and somebody accidentally bumping you and it hurts. We interpret that very differently. So if we can see inside our AIs, we might accept that it was dealing with a tricky situation, it tried to do the best thing it could according to its analysis, but there was some negative side effect. We might be sort of okay with that because maybe even as people in that tricky situation, it would be very difficult for us to do the right thing. But if it did the wrong thing intentionally, that's a whole different thing. So these are all aspects of AI AGI safety, and we have people working on all these topics.

那么，你们是否会限制这些东西与真实世界交互的程度、发布的速度等，直到你们确信它们达到了安全阈值？

So then do you sort of limit the amount that these things can interact with the real world, how quickly you release them and so on until you feel confident that they're at the safety threshold?

是的。我们有各种测试基准和测试，我们在内部运行一段时间，并且我们针对风险领域测试特定内容。

Yeah. So we have all kinds of testing benchmarks and tests, and we run them internally for a while, and we have particular things that we test for that are risky areas.

比如什么？

Like what?

我们尝试看系统是否会帮助开发，比如生物武器之类的东西，对吧？显然它不应该。如果我们发现我们可以用某种方式诱骗或迫使它在那个领域提供帮助，那就是个问题。黑客攻击是另一个。它是否会帮助人们进行黑客攻击等等。所以目前我们有一系列这样的测试，并且这个集合随着时间的推移不断增长。然后我们评估它在某些领域的能力有多强，然后我们针对我们看到的能力水平采取相应的缓解措施。这可能意味着我们不发布模型，也可能意味着根据我们的发现采取不同的措施。

We try to see if the system will help develop, I don't know, like a bioweapon or something like that, right? And obviously it should not. And so if we start seeing that we can somehow trick it or force it into being helpful in that area, that's a problem. Hacking is another one. Will it help people hack things and so on. So yeah, we have at the moment a collection of these tests, and this collection keeps growing over time. Then we assess how powerful it is in some of these areas, and then we have mitigations appropriate to each level of capability that we see. It could mean that we don't release the model. It could mean various different things depending on what we find.

那么，我们来谈谈这些东西对社会的影响。一旦我们拥有了真正强大的 AGI，我知道你对这个问题思考了很多。这么说没错吧？

Well, let's talk about the impact on society of some of this stuff. Once we get to really capable AGI, and I know that this is something that you have thought an awful lot about. Is that fair to say?

是的。我现在的主要关注点是试图理解，如果我们有了 AGI，并且它在其能力水平上相当安全，那么其他一切呢？其他一切事情的清单非常庞大。

Yeah. My main focus now is trying to understand what if we get AGI and it's reasonably safe for its level of capability. What about everything else? And the list of everything else is enormous.

有一些问题比如：好吧，我们有了强大的 AGI，而且相当安全。它是否有意识？这甚至是一个有意义的问题吗？你对此有立场吗？

There are questions like: so okay, we've got powerful AGI and it's reasonably safe. Is it conscious? Is that even a meaningful question? Do you have a stance on that?

嗯，我们有一个小组在研究这个问题，我们和世界上许多研究这个问题的顶尖专家谈过。我认为简短的回答是，没有人真正知道，要说得绝对清楚。我们这里说的是完全的 AGI，而不是我们目前拥有的东西。

Well, we've got a group looking at that, and we've talked to a lot of leading experts in the world who study this. I think the short answer is nobody really knows, to be absolutely clear. We're talking about full AGI here, rather than the stuff we have at the moment.

你确信目前的东西没有意识吗？

Are you comfortable the stuff at the moment is not?

我认为它没有。当我们进入未来某个 AGI 时代，比如十年后，一个非常强大的系统，它会有意识吗？当我与一些世界上最著名的研究这个问题的专家交谈时，有人支持，有人反对。但当我真正给他们一个具体场景，我说，‘看，我们有 Gemini 10，它被具身在一个类人机器人中，它学习，整合来自传感器的信息，能够记住自己作为智能体在世界中的历史，做所有这些事情。’而且它还会谈论自己的意识，因为如果你以正确的方式提示，你实际上可以让 AI 模型谈论意识。它有意识吗？当我向领域内的人提出这个问题时，他们通常会说，可能没有，或者可能有，但实际上我不完全确定。谁知道呢，也许我们会有答案。我认为这是一个长期存在的问题，甚至很难将其变成一个严格的科学问题，因为我们不知道如何将其框定为可测量的事物。我确信会发生的是，有些人会认为它们有意识，有些人认为没有。这肯定会发生，尤其是在缺乏一个被广泛接受的科学定义和测量方法的情况下。那么我们将如何应对？这也是一个非常有趣的问题。但这只是其中一个问题。我们还有诸如：我们是否会从 AGI（比如完全的 AGI）走向远超人类智能的超级智能？它会快速发生、缓慢发生还是永远不会发生？如果它确实走向超级智能，那个超级智能是什么？它的认知特征是什么？它会在哪些方面远超人类？我们已经看到它能说 200 种语言，这一点很清楚。

I don't think it is. As we go into some future AGI years in the no 10 years in the future or something, which is very very capable, will that system be conscious? When I talk to some of the most famous experts in the world that study this, there are various people who have arguments for, there are various people who have arguments against. But when I actually put a concrete scenario to them and I say, 'Look, we've got Gemini 10 here and it's embodied in a humanoid robot and it learns and it integrates information across sensors and it can remember its own history as an agent in the world and do all these sorts of things.' And also talks about its own consciousness because you can actually get AI models to talk about that consciousness now if you prompt them in the right kind of way. Is it conscious? And when I put that to people in the field, they're like, well, I think probably not or I think probably yes, but actually I'm not absolutely sure. And who knows, maybe we will have an answer to that. I think it's a long-standing question and it's a very difficult question to even make into a strict scientific question because we don't know how to frame this as a measurable thing. What I am sure is going to happen is that some people will think they are conscious and some people will think they are not. That is certainly going to happen, particularly in the absence of a really well-accepted scientific definition and way of measuring it. And then how are we going to navigate that? That's a very interesting question as well. But this is just one question. We have things like: are we going to go from AGI, say full AGI, are we going to go towards superintelligence that's far far beyond human intelligence? Is it going to happen quickly, slowly, never? And if it does go to superintelligence, what is that superintelligence? What's the cognitive profile of that superintelligence? Are there certain things where it's going to be far far beyond human? We already see it can speak 200 languages or something, that's clear.

有没有其他事情，也许因为计算复杂度或其他原因，实际上不会比人类好多少，对吧？我们对此有了解吗？这似乎是一个对人类来说非常重要的问题，需要思考。我们会在十年或二十年内进入超级智能吗？你对此有立场吗？你认为会走向超级智能吗？我的意思是，我在想，比如爱因斯坦提出了广义相对论。我们会不会有这样一个 AGI，能够对世界进行理论化，提出超越人类所能达到的真正科学理解？

And are there other things where maybe because of the computational complexity or whatever is not actually going to be much better than humans, right? Do we have any idea of that? That seems like a really important question for humanity to be thinking about. Are we going to go into superintelligence in a decade or two decades or something like that? Do you have a stance on that? Do you think it will go to superintelligence? I mean I'm sort of thinking here about like Einstein for example came up with general relativity. Will we be in a position where you have AGI that can theorize about the world come up with genuine scientific understanding that goes beyond what humans have managed?

我认为会的，基于计算，而人脑是一个移动处理器。它重几磅，消耗大约 20 瓦特。大脑内的信号通过树突发送，通道频率大约 100 赫兹，皮层中可能 200 赫兹。信号本身是电化学波传播，速度约 30 米/秒。所以，如果与我们在数据中心看到的相比，不是 20 瓦特，你可以有 200 兆瓦；不是几磅，你可以有几百万磅；通道上不是 100 赫兹，你可以有 100 亿赫兹；不是 30 米/秒的电化学波传播，你可以以光速 30 万公里/秒传播。所以在能耗、空间、通道带宽、信号传播速度这四个维度上，你同时有六、七、八个数量级的优势。那么人类智能会是可能的上限吗？我认为绝对不是。所以我认为随着我们对如何构建智能系统的理解发展，我们会看到这些 AI 远远超越人类智能。就像人类，我们跑不过顶级燃油赛车，举不过起重机，看不过哈勃望远镜。我们已经看到机器在特定领域比最快的鸟飞得更快等等。我认为在认知领域也会如此。我们已经看到在某些方面，比如你知道的不比谷歌多，在信息存储等方面我们已经超越了人脑的能力。我认为我们将在推理和其他所有领域开始看到这一点。所以是的，我认为我们会走向超级智能。这就是为什么我对系统二安全之类的事情非常感兴趣，因为如果我们由于全球竞争动态等原因无法阻止向超级智能的发展，那么我们需要认真思考如何让超级智能变得超级道德。如果你有一个系统，不仅能运用其智能实现目标、做事，还能实际用于做出道德决策，那么它可能会以某种方式与其能力一起扩展。

I think it will based on computation and the human brain is a mobile processor. It weighs a few pounds. It consumes I think around 20 watts. Signals are sent within the brain through dendrites. The frequency on the channel is about order of 100 hertz or maybe 200 hertz in the cortex. And the signals themselves are electrochemical wave propagations. They move at about 30 m/s. So if you compare that to what we see in a data center instead of 20 watts you could have 200 megawatts, instead of a few pounds you could have several million pounds. Instead of 100 hertz on the channel you can have 10 billion hertz on the channel. And instead of electrochemical wave propagation at 30 meters per second, you can be at the speed of light 300,000 kilometers per second. So in terms of energy consumption, space, bandwidth on the channel, speed of signal propagation, you've got six, seven, maybe eight orders of magnitude in all four dimensions simultaneously. So is human intelligence going to be the upper limit of what's possible? I think absolutely not. And so I think as our understanding of how to build intelligent systems develops, we're going to see these AIs go far beyond human intelligence. In the same way that humans, we can't outrun a top fuel dragster over 100 meters, we can't lift more than a crane, we can't see further than the Hubble telescope. We already see machines in particular areas that can fly faster than the fastest bird and all these sorts of things. I think we'll see that in cognition as well. We've already seen in some aspects, you know, you don't know more than Google, and so on like information storage and stuff like that we already gone beyond what the human brain is capable of. I think we're going to start seeing that in reasoning and all kinds of other domains. So yes, I think we are going to go towards superintelligence. So that's why I'm very interested in things like system two safety because if we can't stop the development towards superintelligence because of competitive dynamics globally and all these sorts of things, then we need to think really hard about how do we make a superintelligence super ethical. And if you have a system that can apply its intelligence not just to achieving goals and doing things but actually applying it to making ethical decisions as well, then it might scale with its capabilities in some way.

我确实想知道这一切对人们意味着什么。我的意思是，如果我们到了人类智能被超级智能相形见绌的地步。这对社会意味着什么？这是否意味着巨大的不平等，那些在能为经济提供什么方面基本上不再有价值的人被完全抛在后面？

I do wonder what all of this means for people. I mean if we are getting to a point where essentially human intelligence is dwarfed by superintelligence. What does that mean for society? Does that mean just massive inequality that you have the people who no longer have value essentially in what they can offer the economy being completely left behind?

这意味着巨大的变革。我认为当前人们贡献体力和脑力劳动以换取资源获取权的体系可能不再适用，我们需要不同的方式。现在蛋糕应该会变得更大，所以不存在商品和服务生产不足的问题。相反，情况会越来越好，但我们需要仔细思考：什么是适合人的体系？我们如何分配社会中的财富？我认为需要更多思考后 AGI 经济如何运作，以及后 AI 社会的结构如何。我曾向罗素集团的副校长们做过演讲。在英国，罗素集团是顶尖大学。我对他们说，看，AGI 即将到来，而且不远了。十年内，我们将拥有它，它将能够完成人们所做的大量认知劳动和工作。我们实际上需要社会各个层面的人们思考这对他们特定领域意味着什么。所以我们真的需要你们大学的每个学院和系都认真对待，思考这对教育意味着什么？对法律意味着什么？对工程、数学、城市规划、文学、政治、经济、金融、医学等等意味着什么？基本上，每个学院、每个系研究的东西中，人类智能都是非常重要的。所以如果廉价、丰富、有能力的机器智能出现，那件事就需要重新思考。这意味着什么？是否应该以不同方式做？机遇是什么？风险是什么？等等。所以我认为这里有巨大的机遇。但就像任何革命，比如工业革命，它很复杂。它以各种方式对社会产生各种影响。为了获得好处并最小化风险和成本，我们需要谨慎导航。目前我认为思考 AGI 对特定事物意味着什么的人远远不够，我们需要更多人这样做。

It means a massive transformation. I think the current system where people contribute their mental and physical labor in return for access to resources that are generating the economy may not work the same anymore and we may need different ways of doing things. Now the pie should get much bigger. So there's not a problem of a lack of goods and services that are produced. If anything that's getting much much better, but we need to think carefully about what is the system for people? How do we distribute the wealth that exists in society? I think there needs to be a lot more thought going into this of how a post-AGI economy works and how the structure of a post-AI society works as well. I gave a talk to the Russell Group vice chancellors. So in the UK, the Russell Group is the top universities. And I said to them, look, this AGI thing's coming and it's not that far away. In 10 years, we're going to have it and it's going to start being able to do a significant fraction of all kinds of cognitive labor and work and things that people do. We actually need people in all these different aspects of society and how society works to think about what that means in their particular area. So we really need every faculty and every department that you have in your university to take this seriously and think what does it mean for education? What does it mean for law? What does it mean for engineering, mathematics? City planning, literature, politics, economics, finance, medicine, dot dot dot. So basically every faculty, every department studies something where human intelligence is a really important thing. And so if you have the presence of cheap, abundant, capable machine intelligence turning up, that thing needs to be thought about again. What are the implications of this? Should it be done in a different way? What are the opportunities? What are the risks and so on? So, I think there's an enormous opportunity here. But just like any revolution like the industrial revolution, it's complicated. It has all kinds of effects on society in all kinds of ways. And to get the benefits of that and minimize the risks and the costs, we need to navigate this carefully. And at the moment I think nowhere near enough people are thinking about what AGI means for this particular thing and we need a lot more people doing that.

你还记得 2020 年 3 月，专家们说有一场大流行即将到来吗？我们真的站在指数曲线的边缘，然后大家还在酒吧里，去看足球比赛什么的，专家们越来越大声地喊叫即将发生的事情。你有点那种感觉吗？

Do you remember in March 2020 when the experts were saying there's this pandemic coming? It's really we're standing on the edge of an exponential curve and then everyone was still sort of in pubs and going to football games and things and the experts were increasingly shouting about what was coming. Do you sort of feel a little bit like that?

我清楚地记得那些日子。确实有点那种感觉。人们很难相信一个巨大的变化即将到来，因为大多数时候，说某件大事即将发生的故事……

I remember those days well. It does feel a bit like that. People find it very hard to believe that a really big change is coming because most of the time the story that something really huge is about to happen.

并不总是物理上归于虚无，对吧？

It's not always the physical out to nothing, right?

所以作为一条经验法则，如果有人告诉你一些极其疯狂的大事要发生，通常你可以忽略大部分。但你必须留意。有时这些事背后有根本性的驱动力，如果你理解了这些根本因素，就需要认真对待重大变革确实会到来的想法，而且你知道，有时重大变革确实会到来。

And so as a heuristic, if somebody tells you some crazy big things are going to happen, as a heuristic probably you can ignore most of those. But you do have to pay attention. Sometimes there are fundamentals that are driving these things and if you understand the fundamentals you need to take seriously the idea that a big change does come and you know sometimes big changes do come.

但这意味着什么呢？因为，好吧，你描述了一个长期愿景，那里有完整的 AGI，繁荣可以共享等等，但达到那一步，我们说的是些非常重大的……

What does this mean though? Because I mean, okay, you describe a sort of a long-term vision where you have full AGI and there's like prosperity that can, you know, potentially be shared and so on, but getting there, I mean, we're talking about some really big...

我是说，这还说得轻了，巨大的经济动荡，结构性风险。跟我们说说你预计未来几年会是什么样子。我是说，告诉我们 2020 年 3 月时我们不知道的事。

I mean, that's an understatement, massive economic disruption, structural risks here. Just talk us through what you expect the next few years to look like. I mean, tell us what we didn't know in March 2020.

我认为未来几年我们看到的不会是你说的那些大动荡。我认为未来几年我们会看到 AI 系统从非常有用的工具，转变为真正承担更多经济价值高的工作，而且我认为这会很不均衡。在某些领域会比其他领域更快发生。例如在软件工程领域，我认为未来几年 AI 编写的软件比例会上升，所以几年后，以前需要 100 名软件工程师的地方，可能只需要 20 人，而这 20 人使用先进的 AI 工具。几年内，我们会看到 AI 从一种有用的工具，转变为做真正有意义的生产性工作，并提高这些领域工作者的生产力。这也会在劳动力市场的某些领域造成一些动荡。随着这种情况发生，我认为很多关于 AI 的讨论会转变，变得更加严肃。它会从那种‘哦，这真的很酷。你可以让它帮你规划假期，帮孩子做作业’之类的，变成‘好吧，这不是什么好用的新工具。这实际上是会结构性改变经济和社会方方面面的东西。我们需要思考如何构建这个新世界，因为我确实相信，如果我们能驾驭这种能力，这可能会是一个真正的黄金时代，因为我们现在有了能大幅提高多种产品生产的机器，对吧？还能推动科学进步，让我们从各种劳动中解脱出来，如果机器能做的话，我们也许就不需要做了，对吧？所以这里有机会，但只有当我们能以某种方式将机器的这种不可思议的能力转化为一个社会愿景，让个人和群体都能从中受益并繁荣发展时，这才是有益的。因为与此同时，那 80 名不再需要的软件工程师，以及其他所有初级员工，你知道，毕业生们，他们注意到自己是第一批受此影响的人。

I think what we'll see in the next few years is not those big disruptions you're talking about. I think we'll see in the next few years is AI systems going from being very useful tools to actually taking on more of a load in terms of doing really economically valuable work and I think it'll be quite uneven. It'll happen in certain domains faster than others. So for example in software engineering I think in the next few years the fraction of software being written by AI is going to go up and so in a few years where prior you needed a 100 software engineers maybe you need 20 and those 20 use advanced AI tools over a few years we'll see AI going from kind of just a useful tool to doing really meaningful productive work and increasing the productivity of people that work in those areas. It'll also create some disruption in the labor market in certain areas. And then as that happens, I think a lot of the discussion around AI is going to shift and become a lot more serious. And so it's going to shift from being just sort of like, oh, this is really cool. You can ask it to plan your holiday and help you with your children's homework or whatever, through to something that's like, okay, this is not some nice new tool. This is actually something which is going to structurally change the economy and society and all kinds of things. And we need to think about how do we structure this new world because I do believe that if we can harness this capability, this could be a real golden age because we now have machines that can dramatically increase production of many types of things, right? And advance science and relieve us of all kinds of labor that maybe we don't need to be doing if the machines can do it, right? So there's an opportunity here, but that is only good if we can somehow translate this incredible capability of machines into a vision of society where there is some flourishing of people as individuals and as groups of people in society that benefit from all this capability. Because in the meantime, you have those 80 software engineers who are no longer needed and all of the other people, the entry level employees at the moment, you know, graduates who are sort of noticing that they're the first ones to be affected by this.

有没有哪些行业不会受到这种影响？

Are there any industries that are not going to be impacted by this?

嗯，在短期到中期内，我认为实际上会有很多行业不受影响。比如水管工，对吧？我认为在未来几年，即使 AI 在纯认知意义上发展得很快，我也不认为机器人技术能达到能当水管工的程度，即使可能，也需要相当长的时间才能与人类水管工在价格上竞争，对吧？所以我认为所有非纯认知的工作都会相对受到保护。有趣的是，很多目前薪酬很高的工作是精英认知工作，比如，我不知道，全球范围内做复杂并购交易的高端律师，做金融高级工作的人，或者现在做高级机器学习软件工程的人，所有这些类型。数学家，我喜欢的一条经验法则是：如果你能通过互联网远程完成工作，只用一台笔记本电脑，而不是什么全身触觉服加机器人控制，就是普通界面：键盘、屏幕、摄像头、扬声器、麦克风、鼠标。如果你能完全这样工作，那很可能就是认知工作。所以如果你属于这一类，我认为高级 AI 能在一定程度上在这个领域运作。另一个我认为有保护作用的因素是，即使是认知工作，某些类型的工作和人们做的事情也可能有人类的一面。例如，假设你是一个网红，对吧？你可以远程工作，但你是特定的人，有特定的个性，人们知道背后有一个人，这在很多情况下可能很有价值，对吧？

Uh in the short to medium term, I think there'll actually be quite a lot of things. So plumbers often go, right? I think in the coming years we're not going, even if the AI does develop quite quickly in a purely cognitive sense, I don't think robotics will be at the point which could be a plumber and then even when that is possible I think it's going to take quite a while before it's price competitive with a human plumber right and so I think there are all kinds of work which is not purely cognitive that will be relatively protected from some of this stuff. The interesting thing is that a lot of work which currently commands very high compensation is sort of elite cognitive work right so it's people doing, I don't know, sort of high-powered lawyers that are doing complex merger and acquisition deals across the globe and people doing advanced stuff in finance or now people doing advanced machine learning software engineering all these types of things. Mathematicians, one rule of thumb that I quite like is: if you can do the job remotely over the internet, just using a laptop, so you're not some full haptic body suit with some robot, you know, controlling whatever, just normal interface, keyboard, screen, camera, speaker, microphone, you know, mouse. If you can do your work completely that way, then it's probably very much cognitive work. So if you're in that category, I think that advanced AI will be able to operate in that space to some extent. The other thing that is I think protective is even if it is cognitive work, there can be a human aspect to some types of work and things that people do. So for example, let's say you are an influencer, right? and you work, you can do that work maybe remotely, but the fact that you're a particular person with a particular personality and people know there is a person behind what's going on there, that may be valuable in many cases, right?

但这还是留下了很多人，不是吗？

That leaves a lot of people though, doesn't it?

我认为我们需要的是，正如我向罗素集团建议的那样：我们需要研究社会各个方面的学者认真对待 AGI。我的印象是，很多这样的人并没有。当我去和那些对某个特定领域感兴趣的人交谈时，他们觉得，哦，是啊，它有点像，你知道，一个有趣的工具，有点好玩，等等，但他们没有内化一个观念：他们现在看到的以及他们目前知道的任何限制——顺便说一句，这些限制往往已经过时了。这些人常说：“哦，我一年前试过用它做点什么。”但一年前相对于当前模型的能力已经是古老的历史了，一年后它会好得多。他们在某种程度上没有看到这个趋势。我实际上认为很多普通公众比专家更超前，因为我认为有一种人类倾向。你知道，如果我和非技术人士谈论当前的 AI 系统，有些人会对我说：“哦，它难道不是已经拥有人类智能了吗？它说的语言比我多。它做数学和物理题比我高中时强多了。它知道的菜谱比我多。它能帮我做各种事情。我对报税感到困惑，它给我解释了一番等等。”

I think what we need is along the lines of what I suggested to the Russell group: we need people who study all these different aspects of society to take AGI seriously. And my impression is that a lot of these people are not. And when I go and talk to people who are interested in one of these particular things, like, oh yeah, it's kind of like, you know, it's an interesting tool, it's kind of amusing, whatever, but they haven't internalized the idea that what they're seeing now and any current limitations that they currently know of, which by the way are often out of date. Often these people say, "Oh, I tried to do something with it a year ago." It's like a year ago is now ancient history compared to what the current models are doing and one year from now it's going to be a lot better. They're not seeing that trend in some ways. I actually think many people in the general public are ahead of the experts because I think there's a human tendency. You know if I talk to non-tech people about current AI systems, some of the people say to me, "Oh well, doesn't it already have like human intelligence? It speaks more languages than me. It can do math and physics problems better than I could ever do at high school. It knows more recipes than me. It can help me with all kinds of things. I was confused about my tax return and explain something to me or whatever."

他们会问：‘那它在哪些方面不够智能呢？’你知道，这就是我和一些非技术人士聊天时遇到的情况。但通常，某个领域的专家往往觉得自己的领域非常深奥特别，AI 不会真正触及到他们。

They're like, 'So, in what way is it not intelligent?' You know, this is the sort of thing I get when I talk to a number of non-tech people. But often people who are experts in a particular domain really like to feel that their thing is very deep and special and this AI is not really going to touch them.

我想以你现在相当著名的 AGI 预测作为结尾，你对此保持了十多年惊人的一致性。事实上，你说过到 2028 年 AGI 有 50% 的可能性。

I think I want to end with your now quite famous prediction about AGI and you have stayed incredibly consistent on this for over a decade. In fact, you have said that there is a 50/50 chance of AGI by 2028.

是的。

Yes.

那是最低限度的 AGI 吗？

Is that minimal AGI?

是的。

Yes.

哇。那你现在仍然认为到 2028 年是 50% 吗？

Wow. And are you still 50/50 by 2028?

是的，2028 年。你可以在我的博客上看到，从 2009 年我就这么说了。

Yes, 2028. And you can see that on my blog from 2009.

那对于完全的 AGI 你怎么看？你的时间线是怎样的？

And what do you think about full AGI? What's your timeline for that?

嗯，再晚几年。可能是三、四、五、六年之后。是的。

Uh, some years later. Could be three, four, five, six years later. Yeah.

十年之内。

Within a decade.

是的，我认为会在十年之内。

Yeah, I think it'll be within a decade.

拥有这些知识，你有没有感到过一丝虚无？

Do you ever just feel a bit nihilistic with all of this knowledge?

我认为这里有巨大的机遇。很多人投入大量精力做很多工作，但并非所有工作都那么有趣。我认为这里有一个不可思议的机会，就像工业革命利用能源进行各种机械工作，创造了更多社会财富。现在我们可以利用数据、算法和算力来进行各种认知工作。这可以为人们创造大量财富，不仅是商品和服务的生产，还有新技术、新药物等等。所以这项技术有着巨大的潜在益处。现在的挑战是，如何在应对风险和潜在成本的同时获得这些益处。我们能否想象一个未来世界，我们真正受益于智能，帮助我们繁荣发展？那会是什么样子？你知道，我不能简单回答。我对这个问题非常感兴趣。我会尽力去理解。但这是一个非常深刻的问题，涉及哲学、经济学、心理学、伦理学等各种问题。我们需要更多人思考这个问题，并想象那个积极的未来是什么样子。

I think there is an enormous opportunity here. A lot of people put a lot of effort into doing a lot of work and not all of it is that much fun. And I think there's an incredible opportunity here to just like the industrial revolution took the harnessing of energy to do all sorts of mechanical work which created a lot more wealth in society. Now we can harness data and algorithms and computation to do all kinds of more cognitive work as well. And so that can enable a huge amount of wealth to exist for people and wealth not just in terms of production of goods and services but you know new technologies, new medicines and all kinds of things like this. So this is technology that has an incredible potential for benefit. Now the challenge is how do we get those benefits while dealing with the risks and potential costs and so on. Can we imagine a future world where we're really benefiting from having intelligence really helping us to flourish and what does that look like? And that's you know I can't just answer that. I'm very interested in that. I'm going to try and understand the best I can. But this is a really profound question. It touches on philosophy and economics and psychology and ethics and all kinds of questions, right? Um, and we need a lot more people thinking about this and trying to imagine what that positive future looks like.

Shane，非常感谢你。至少可以说，这真是令人眼界大开。人类不太擅长指数增长，而此刻我们正站在曲线的拐点上。AGI 不再是一个遥远的思维实验。我和 Shane 的对话中非常有趣的一点是，他认为普通公众比专家更理解这一点。如果他的时间线大致正确，而且他过去有预测正确的习惯，那么我们可能没有慢慢反思和意识的时间了。我们面临困难、紧迫且可能真正令人兴奋的问题，需要认真对待。现在，你收听的是 Google DeepMind 播客，我是你的主持人 Hannah Fry。如果你喜欢这次对话，请订阅我们的播客或给我们留下评论。下一集，我们将与 DeepMind 联合创始人 Demis Hassabis 坐下来聊聊。所以相信我们，你不想错过那一集。

Shane, thank you so much. That was mind-expanding to say the least. Humans are not very good at exponentials and right now at this moment we are standing right on the bend of the curve. AGI is not a distant thought experiment anymore. What I found so interesting about that conversation with Shane is that he thinks the general public understand this better than the experts. And if his timelines are anything like correct, and he's had a habit of being right in the past, we might not have the luxury of time for slow reflection and realization here. We have got difficult, urgent, and potentially genuinely exciting questions that need some serious attention. Now, you have been listening to Google DeepMind the podcast with me, your host Hannah Fry. If you enjoyed that conversation, please do subscribe to our podcast or leave us a review. Next episode we are going to be sitting down with DeepMind co-founder Demis Hassabis. So trust us when we tell you you don't want to miss that one.