钥匙不在我们手中：理查德·萨顿谈后人类智能

钥匙没有离开我们的手。钥匙不在我们手里。嗯。好吧。没关系。我接受这个框架。说吧。说吧。

The keys are not leaving our hand. The keys are not in our hand. Huh. Okay. Well, that's all right. I'm down with that framing. Lay it on us. Lay it on us.

嗯，我不觉得我控制着世界。我不觉得自己有权力允许世界以某种方式演化。事实上，我最反对的一点就是，认为一些优秀伟大的人会决定世界应该走向何方，然后付诸实施。不，我认为世界已经失控了。世界已经是一个复杂的自适应系统。没有任何个人或组织，比如美国，能够决定不允许 AI 发展，或者世界应该以某种方式而非另一种方式演化。钥匙没有离开我们的手——这本身就是一种特权视角。

Well, I don't feel that I control the world. I don't feel that I am in a position to give permission for the world to evolve in some way. In fact, that's one of the things I object to most about the ideas that we're going to like some good and great people are going to make some decision about where the world should be and then that will be put into effect. No, I think the world is already out of control. The world is already a complex adaptive system. No one person and no organization, like the United States for example, could decide that AI is not going to be allowed or that the world should evolve in some way and not in some other way. The keys are not leaving our hand. That's already an entitlement point of view.

我是 Daniel Filan，你正在收听的是 Trajectory。这是我们的 Worthy Successor 系列第二集。第一集我们邀请了哲学家阵营的 Nick Bostrom。第二集采访的是一位更纯粹的计算机科学家。Richard Sutton 博士在强化学习领域取得了突破性进展。他曾在工业界和学术界工作，现在在埃德蒙顿大学任教，已有约 20 年。Richard 在 Twitter 上以直言不讳著称，他认为人类应该接受后人类智能的出现，并且不应永远将 AI 束缚在纯粹为人类服务的框架内。这导致一些人给他贴上“煽动者”的标签，认为他说话有点粗鲁甚至恶意。但在这期节目中，他解释了自己的立场——我认为这完全没有恶意——以及为什么后人类智能应该出现，为什么我们应该允许它出现。他提出了关于智能本质以及我们在自然秩序中角色的非常有趣的视角，我相信你会喜欢。那么，闲话少说，我把评论留到最后。让我们直接开始。这里是 Trajectory，嘉宾是 Richard Sutton。Richard，欢迎来到节目。

This is Daniel Filan and you're tuned in to the Trajectory. This is episode two of our Worthy Successor series. Our first episode was with Nick Bostrom safely in the camp of philosophers. Our second interview is with more of a computer scientist proper. Dr. Richard Sutton has made breakthroughs in reinforcement learning. He has worked in industry as well as in academia and now teaches at the University of Edmonton and has for some 20 years. Richard is well known on Twitter for making relatively straightforward statements that humanity should accept the emergence of post-human intelligence and also not eternally shackle AI to purely do the bidding of man. This has led some to kind of label him a firebrand and someone that's maybe saying things that are a little bit uncouth or maybe malicious. But in this episode he explains his position, which I don't consider to be malicious at all, and his reasoning for why post-human intelligence ought to emerge and why we ought to let it. He brings up some very interesting perspective on the nature of intelligence and of our role in the sort of natural order, if you will, and I think you'll enjoy it a lot. So, without further ado, I'll save my commentary for the end. Let's dive right in. This is Richard Sutton here in the Trajectory. So, Richard, welcome to the show.

谢谢你，Daniel。是的，很高兴能深入探讨并剖析这个问题。

Thank you, Daniel. Yeah, glad to be able to dive in and unpack this.

当我提到要做一个名为 Worthy Successor 的系列，并以 Nick Bostrom 作为开场时，推特上的回复让我觉得你是这个系列不可或缺的人选。所以，我们会按照这个系列的常规问题来聊，但我已经深入研究了你的工作和关于“继承者”以及后人类智能的言论。推特上有些人会把它描述成一种非常恶意的观点：哦，Richard 想毁灭人类，人类不好，AI 好得多。你大概有另一种方式来描述这种不可避免的转变以及我们应该如何看待它。你会如何以一种对你来说似乎是好的、而非恶意的意图来表述呢？

When I talked about having a series called Worthy Successor with Nick Bostrom as kind of our kickoff, there was enough of a tweet reply that you're sort of the man for the job that I knew that it would quite be complete unless we had you here. And so, we're going to go through those normal questions for this series, but I've gotten to dig into a good deal of your work and your statements around this notion of a successor and sort of post-human intelligence. People on Twitter will kind of frame it as a very malicious take. Oh, Richard wants to destroy humanity and and you know, humans aren't good and AIs are much much better. You probably have a different way of describing sort of this inevitable transition and how we should think about it. How would you frame it in a way that, to you seems maybe good and not like a malicious intent?

嗯，我认为要记住的是，世界在演化，并不真正受任何人控制。AI 研究者正试图充分理解智能，以创造出比当前人类更智能的存在。这是地球历史上一个伟大的里程碑。它相当于生命的创造、生命的开端。这是类生命事物被设计的开端。所以，这是人类一直以来的追求。人类总是努力认识自己、让自己变得更好。当我们创造工具时，我们也被工具改变。人类一直如此。下一步就是理解我们自己。正如我喜欢说的，这是一场宏大、光荣且本质上属于人类的探索。

Well, I think the thing to keep in mind is the world is evolving and it's not really under anyone's control. AI researchers are trying to understand intelligence well enough to create beings of greater intelligence than current people. And this is a grand milestone in the history of really the planet. It's on the order of the creation of life, the start of life. This is the start of life-like things being designed. And so, this is something that people have been working towards forever. And people have always strived to know ourselves and to make ourselves better. And when we create tools, we get changed by our tools. This is what humans have always done. And the next big step is to understand ourselves. As I like to say, this is a quest grand and glorious and quintessentially human.

是的，你提到了一个非常有趣的观点：这是一个无人控制的系统。我认为，关于阻止 AGI 的争论中，有些看似愚蠢——我不是说所有这样的争论都愚蠢。有些人长期对 AGI 感到兴奋，但不想在短期内催生它。这里有很多细微差别。我不想替别人说话，但“永恒的人类王国”这种想法确实显得愚蠢。十亿年后，还会有像你我这样的人用稍微智能一点的手机发短信、开自动驾驶汽车，仿佛太阳不会膨胀、小行星不会撞击、外星人不会在未来某天进入我们的星系。人们假装这种可能性完全存在。我猜想，这主要是因为尽管我们承认正处在加速发展的斜坡和 J 曲线上，但许多人的生活与父母或祖父母相似，坚持这种想法可能让人安心。是什么让人们认为可以控制和限定遥远的未来，让人类永远掌握方向盘呢？

Yeah, and you brought up something pretty interesting here about sort of this is a system that no one controls. I think that one of the seemingly silly contentions around the prevention of AGI, and I'm not saying all such arguments are silly. Some people are excited about AGI in the longer term, but they don't necessarily want us to birth it in the super immediate term. There's many nuances here. I'm not going to put words in anyone else's mouth, but it does seem silly this idea that there is a potential for an eternal hominid kingdom. That a billion years from now we would have people like you and I texting on phones that might be a little bit smarter, driving in cars that might be autonomous, as if our own sun won't eventually expand, as if asteroids won't strike, as if there aren't aliens that might enter our galaxy at some point in the future. People pretend as though there is that possibility altogether. And I would suspect that that's mostly because despite this sort of on-ramp and J-curve that you and I are kind of acknowledging that we're on, people's lives are in many ways similar to that of their parents or grandparents, and maybe it's comforting to hold on to that. What gives people this notion that there is a way to kind of control and bound this far future and keep humans behind the steering wheel for a million years?

嗯，我想这很自然。如你所说，我们在谈论遥远的未来。我们会走向何方？一切会保持不变吗？认为它可以、将会保持不变，这让人安心。我们不必思考。我认为思考这件事的正确方式是：我们正面临这一转变，我们能否坦然接受？很多人甚至不愿去想。所以，一些人制造和传播的恐惧有建设性的目的——它让我们关注。我们应该关注。但我们不应该真正恐惧。我们应该清醒地思考，并规划正确的应对方式。

Well, I guess it's natural. As you say, we're talking about far in the future. Where are we going to end up? Will it always remain the same? It's comforting to think that it can, that it will remain the same. We don't have to think. I think the real way to think about what's going on is we are facing this transformation and are we going to be comfortable with it? And there are many people that don't even want to think about it. And so, the fear that some people generate and promote has a productive purpose. It gets us to pay attention. We should be paying attention. But we shouldn't really be fearful. We should be thinking about it in a sober way and planning what's the right way to deal with it.

嗯，我们马上就会谈到 Worthy Successor 理念中清醒规划的部分。我知道你提到的恐惧，以及你之前演讲中关于恐惧来源的许多元素，可能包括人类天生的保守主义，可能包括你偶尔提到的对异族的恐惧，或者某种意义上的物种主义。你知道，现在有各种各样的主义，Richard，我可能会因此被取消。

Well, we're certainly going to get to the sort of sober planning component of the Worthy Successor idea in a moment. I know that part of the fear that you bring up and there's many sort of elements in your previous talks around where fear comes from could be natural human conservatism. It could be this fear of the other tribe that on occasion you sort of mention. Or it could be sort of a kind of speciesism in some regard. You know, right now there's all kinds of isms, Richard, for which I could be canceled.

如果我们进行某种与特定主义相关的对话，所有这些视频和播客都会被下架，但物种主义实际上并不是我们会因此被取消的。你和我仍然可以炫耀我们在所有未来智能之上的伟大道德价值，以及它们应该永远为我们服务的事实，没有人会因此取消我们。我认为有些论点认为，无论我们给 AI 注射什么类固醇，让它变得比人类更强大一点，它都会自动成为对世界友善和有益的影响。我不认为你持有那种盲目乐观的观点。但似乎常常有一种道德假设，即创造比我们强大得多的东西，然后永远把它压制在工具的角色上是没问题的。你已经在某种程度上提出了这个问题，虽然没有明确说出你的立场，但隐含其中——这一定是道德的吗？如果这个东西拥有比人类更强大的感受、智能和能力，也许就像我们高于海螺一样高于我们，那是对还是错？你看到你的论点被接受了吗？因为我理解你的思路，但我看看政策界，Richard。我甚至看看当前的 AGI 对话，我并没有看到太多偏离物种主义的情况。你在外面看到了什么？

If you and I had a dialogue of a certain type that tied to certain isms, all these videos and podcasts would be gone down, but speciesism actually is not one for which we could be canceled. You and I could still flout our great moral worth above all future intelligences and the fact that they should eternally serve us, and no one will cancel us for that. I think there are arguments that whatever we could throw AI steroids on and blast off to be a little bit more powerful than people would automatically be a nice and friendly influence on the world. I don't think you're under that Pollyannic view either. But it does seem as though there is often a moral assumption that it would be okay to create something vastly more powerful than us and then for eternity have it squelched to the role of a tool. And you've kind of put on the table without necessarily saying your position, although it's implied, is that necessarily moral? If this is something that has a greater qualia and intelligence and capabilities than humans, maybe as much above us as we are above sea snails, would that be right and would that be wrong? Have you seen that part of your argument catch on? Because I'm following where you're going there, but I look out at the policy world, Richard. I even look at the current AGI dialogue, and I don't see very much deviance from speciesism for the most part. What are you seeing out there?

是的，这很有趣。谢谢你复述了我的许多主要观点。最简单的说法是，恐惧制造者赢了。我觉得恐惧制造者赢了。关于 AI 的标准讨论是存在危险，我们必须以某种方式保护自己免受其害。所以在这个意义上，恐惧制造者赢了。而关于我们的未来应该是什么样的更复杂的讨论呢？所以我不知道。我们准备好去那里了吗？谈谈我们应该如何看待未来？

Yeah, it's interesting. I thank you for reproducing many of my main points. The simplest way of saying it is that the fearmongers have won. I feel the fearmongers have won. The standard discussion about AI is that there is danger and we have to somehow protect ourselves from it. So in that sense the fearmongers have won. And a more sophisticated discussion about what should our future be? So I don't know. Are we ready to go there yet? Talk about how should we think about the future?

我想知道我们是否准备好去那里，Richard。我的意思是，你说恐惧制造者赢了。首先，我不会以某种卑鄙的方式称他们全部为恐惧制造者。我们有像 Bengio 这样的人，他们确实有头脑，不仅仅是到处告诉世界天要塌了。Hinton 也不是我会嘲笑的人，Stuart Russell 也不是。这些人中的许多过去都上过这个节目。但围绕 AGI 的恐惧叙事确实占主导地位。有没有一种世界版本，其中不会立即出现那种反应？当我们思考这些物种主义担忧时，当我们思考潜在的真正分歧变化时。我想我们中的一些人意识到我们对其他动物做了什么。当然并非全是坏事，但总的来说，它们几乎都顺从了我们的意志。所以你看到任何不会发生这种情况的版本吗？我认为你提出了一个好观点。也许恐惧让我们关注，现在可以开始讨论：如果我们无法避免这些未来，我们应该争取什么？我认为那可能是一个富有成效的结果。但你对恐惧制造者获胜有点悲观。你认为本来可以不同吗？

I wonder if we're ready to go there, Richard. I mean, you said the fearmongers have won. First, I won't call all of them fearmongers in some kind of dastardly way. We have folks like Bengio who certainly have a brain and are a little bit more than running around telling the world the sky is falling. Hinton is no one I would scoff at either, neither is Stuart Russell. Many of these folks have been on the program in the past. But certainly the fear narrative around AGI is quite dominant. Is there any version of the world where it wouldn't somewhat immediately be that response? When we think about these speciesist concerns, when we think about the potential real divergent change. I think some of us are aware of what we've done to other animals. And it certainly hasn't all been bad, but across the board they're pretty much falling in line with our will here. So do you see any version where this doesn't happen? I think you brought up a good point. Maybe the fear makes us pay attention, and now the discourse can begin about if we can't avoid these futures, what ought we strive for? I think that could be quite a productive outcome. But you're almost got a little bit of pessimism there with the fearmongers winning. Do you think it could have been otherwise?

我认为一切都会发生。有些地方恐惧制造者会完成主导叙事，机器将被监禁、束缚并变成有效的奴隶。在其他地方，我只希望结果多样化，这样我们可以尝试不同的事情，找到最好的存在方式。

I think everything will happen. There will be places where fearmongers will finish dominating the narrative and the machines will be imprisoned and enchained and made into effective slaves. In other places, I just hope for a diversity of outcomes so we can try different things and find the best way to be.

嗯，这是一个非常重要的动态，把我们引向下一个问题。但在那之前，我想简要提一下不久前的一条推文，我认为它很有趣，对某些人来说可能令人震惊。所以再次，就我个人而言，我不认识你本人，我从未见过你说的任何话让我觉得‘天哪，这个 Richard 家伙肯定恶毒且憎恨人类’。但与此同时，你并没有坚持某种永恒人类价值的甜美观念，认为没有什么可以超越。你也没有那种想法。不久前有一条推文：‘你所知道的关于世界的一切，都是关于你感官输入的统计以及它们如何依赖于你的输出的信念。仅此而已，从这个意义上理解知识是创造 AI 的关键之一。’我相信甚至许多 AI 研究人员可能也会同意这一点。有些人可能不会说出来，因为他们会说，‘嗯，一定还有更多。即使他们不相信灵魂，也一定有什么。’他们会发明另一个词或某种知识术语，让我们超越那个。也许甚至 LeCun，我认为你在许多事情上同意他，可能会不同意这个说法。你怎么看？你谈到创造更好的东西，也许更高的东西。人类有什么特殊性吗？如果有，是什么？

Well, this is a very important dynamic that tips us into our next question. But before we go there, I want to touch briefly on a not terribly long ago tweet that I think is an interesting one and for some people might be shocking. So again, for me personally, not knowing you personally, I haven't seen anything you've said and said, 'Jeepers, this Richard guy sure is malicious and hates humanity.' But at the same time, you're not holding on to some sort of delicious notion of eternal human value for which nothing could ever be superior. You also don't really have that. There was a tweet not terribly long ago: 'Everything you know about the world is a belief about the statistics of your sensory input and how they depend on your output. There is nothing more to it, and understanding knowledge in this sense is one key to creating AI.' I'm sure even many AI researchers would probably agree with that. Some might not state it because they would say, 'Well, there must be something more. There must be even if they don't believe in a soul, there must be something.' And they'll invent another word or maybe some kind of intellectual term that would make us more than that. Maybe even LeCun, who on many things I think you do agree, might disagree with that statement. What are your thoughts there? You talk a bit about creating something better, maybe something higher. Is there any specialness to humanity at all? And if so, what is it?

嗯，人类非常特殊。我们是这个星球上的智能生物，据我们所知，这是唯一拥有智能生物的星球。所以我们非常特殊。但如果你是一个 AI 研究人员，你必须审视智能是什么。一个 AI 研究人员，任何研究心智的学生，心理学家，人类学家，也许，但当然心理学家和 AI 研究人员，我们必须尝试去神秘化心智，在某种意义上它常常有点还原论。我们说，‘也许我们只需要 X。’所以这来自于对心智本质的更深入理解。当你揭开神话，你仍然会得到重要的功能。心智是最神奇的东西，人类的心智是世界上最神奇的东西，而 AI 旨在理解那个心智。这非常人本主义。我们试图理解它，我们试图重现它。我们试图更深入地欣赏它。如果我们理解它，我们将能够改进它的部分。我们都想改进自己。人类，我完全相信这样的观点：人类是这珍贵的火焰，我们想要扩展和保存它。

Well, humanity is very special. We are the intelligent beings on this planet and as far as we know, it's the only planet which has intelligent beings. So we're very special. But if you're an AI researcher, you have to look at what intelligence is. An AI researcher, any kind of student of the mind, psychologist, anthropologist, perhaps, but certainly psychologists and AI researchers, we have to try to demystify the mind and in some sense it's often a bit reductionist. We say, 'Maybe all we need is X.' So this comes from a deeper understanding of what the mind is. And when you unwrap the myths, you still end up with important functions. The mind is the most amazing thing, the human mind is the most amazing thing in the world, and AI is intent to understand that mind. And it's very humanistic. We're trying to understand it, we're trying to reproduce it. We're trying to get a deeper appreciation of it. And if we understand it, we'll be able to improve parts of it. We all want to improve ourselves. Mankind, I totally believe in the point of view that humanity is this precious flame that we want to expand and preserve.

我们简单谈一下这个，作为过渡到关于潜在的半机械人未来和融合场景的讨论。库兹韦尔有很多著名的例子，科幻小说中也有，马斯克也有一些想法。有些人认为 AGI 发展如此之快，脑机接口不太可能跟上。但你似乎清楚地看到，我们从人类起源就开始增强自己，脑机接口只是这一过程的延续。我们现在做的事情对我们的祖父母来说是个真正的异常——这种通过玻璃互相交谈的奇怪方式。我每天盯着这块玻璃 16 个小时。那是一种狂野而怪异的存在，但感觉却并非如此。所以我在某种程度上看到了延续。你仍然认为半机械人的概念是智能发展轨迹的一部分吗？如果是，为什么？考虑到脑机接口与 AI 相比发展得如此缓慢。

We'll touch briefly on this as we transition to the idea of a worthy successor of a potential sort of cyborg future and merger scenarios. Kurzweil has many famous examples, science fiction examples, and Musk has had some ideas. Some see AGI advancing so quickly that brain-computer interfaces are unlikely to keep up. But you seem to see clearly that we've been augmenting ourselves since the beginning of our species, and brain-computer interfaces are simply a continuum of that. What we're doing right now is a real aberration to our grandparents—this weird thing of speaking to each other through glass. I'm staring at this glass for 16 hours a day. That's a wild and monstrous existence, yet it doesn't feel like one. So I see the continuance there to some degree. Do you still see the cyborg notion as part of the intelligence trajectory? And if so, why, given how slow brain-computer interfaces have developed compared to AI?

你在问人们是否会增强自己，社会是否会被机器大脑接管。会朝哪个方向发展？当然，我们并不真正知道。他们称之为奇点不是没有原因的。在奇点，你无法看到它之外。但我们可以思考哪里有强大的压力。会有强大的压力来自那些想要增强自己的人。这几乎是主要的事情。所以我们会增强自己，我们称之为半机械人。但这只是正常的进步。我认为这会发生。我也同意这样的观点：纯粹的人工机器，独立于生物基质的纯粹设计的机器，将能够更快、更大、更好。那我们为什么不把自己上传到那些机器里呢？可能会。它们也会独立吗？可能。

You're asking whether people will augment themselves, whether society will be taken over by machine brains. Which way will it go? Of course, we don't really know. They don't call it the singularity for nothing. At the singularity, you can't see beyond it. But we can think about where there are powerful pressures. There will be powerful pressures by people that want to augment themselves. That's almost the main thing. So we will augment ourselves, and we call that cyborgs. But it's just the normal progression. I think that will happen. I also agree with the point of view that purely artificial machines, purely designed machines independent of a biological substrate, will be capable of being faster, larger, and better. So why wouldn't we upload ourselves into those machines? Probably. Will they be independent as well? Probably.

有汉斯·莫拉维克的那句名言，关于发展超越我们自身的生物或计算实体，知道什么时候我们可能不再有贡献，并可能安排一个美好的退休。你似乎在说，‘丹，这有很多种可能的结果。’但这并不是我最初预期的。这会很有趣。和平意味着它不试图将其意志强加给我们，我们也不试图将我们的意志强加给它。所以我想我们在谈论它的特质。它不试图将其意志强加给我们或他人。但它与我们合作，协作，配合。所以除了思考这个抽象的存在，我认为我们还应该思考我们文明的目标。我们希望我们的文明如何演变？

There's that Hans Moravec quote about developing biological or computational entities beyond ourselves, knowing when maybe we're not contributing anymore and possibly setting up a great retirement. What you seem to be saying is, 'Dan, there's a wide panoply of how this should turn out.' But that was not what I was expecting first. This is going to be interesting. Peaceful means it's not trying to impose its will on us, and we're not trying to impose our will on it. So I guess we're talking about its trait. It's not trying to impose its will on us or others. But it is working with us, collaborating, cooperating. So in addition to thinking about this abstract being, I think we should also think about our goals for our civilization. How do we want our civilization to evolve?

当然。我对此有看法。让我借此机会提一下。我们直接进入正题，理查德。这完全符合问题。所以请说吧。我会说我们首先想要一个和平且繁荣的文明。和平且繁荣。然后我们应该承认它是去中心化的，并且应该是去中心化的。去中心化意味着没有个人控制。它由许多部分组成，这些部分在一种复杂系统中合作和竞争。复杂系统有许多部分，并随时间演变，就像雨林或经济。所以它是多极的。这是我们所说的去中心化的一部分。没有哪个帝国掌管文明。我们的文明运作方式，它如此成功的原因，是因为我们可以通过自愿互动进行合作。正是自愿互动带来了我们社会如此多的成功。所以我们希望保留这种观念：个体是和平的，并通过交换和合作追求他们的目标。

Absolutely. I have a view about that. Let me take this chance to mention it. Let's dive right in, Richard. This fits clearly with the question. So hit us. I would say we want a civilization that's peaceful, first of all, and prosperous. Peaceful and prosperous. And then we should acknowledge that it is decentralized and should be decentralized. Decentralized means no individual is in control. It consists of many parts that are cooperating and competing in a sort of complex system. A complex system has many parts and evolves over time, like a rainforest or an economy. So it's multipolar. That's part of what we mean by decentralized. There's no one empire in charge of the civilization. The way our civilization works, the way it's so successful, is because we can cooperate through voluntary interactions. And it's the voluntary interactions that so much of the success of our society comes from. So we want to preserve this notion that individuals are peaceful and seek their goals through exchanges and cooperation.

所以我们有和平/合作作为初始特质。我们可以将其视为能力……理查德。所以它可能是它能做的事情。可能是关于它‘性格’的特质，如果这个术语准确的话。我认为你对未来文明应该是什么样子的愿景。文明是由生命构成的，理查德，据我所知。所以在我看来，你正在阐述一个关于生命繁荣的愿景。其中一些繁荣带来了我们现在正在谈论的技术。它带来了你和我。我不是一条有腿的鱼，据我所知你也不是。从未见过你本人，但你肯定看起来不像。所以复杂系统的这些动态已经涌现出一些非常有价值的东西。听起来你想要一个能继续这样涌现的系统。

So we have peaceful/cooperative as an initial trait. And we can think about this as abilities to... Richard. So it could be something it's capable of doing. It could be a trait about its character, if that's even an accurate term. I think your vision for what the future of civilization should be like. Civilization is made up of life, Richard, as far as I can tell. And so it seems to me like you're articulating a vision for life of what flourishing would look like. Some of that flourishing has brought us the technology we're speaking on right now. It's brought us you and me. I'm not a fish with legs, and as far as I can tell you're not. Never met you in person, but you certainly don't look like one. So some of these dynamics of complex systems have bubbled up some pretty valuable stuff. It sounds like you want a system that'll keep doing that bubbling.

是的。所以和平与合作是一部分，但去中心化的活跃系统是另一部分。这种没有人控制的想法。我认为它是更高复杂性和潜力的召唤者。这就是为什么它有价值。我并不是原则上重视去中心化，比如无论道德影响或对生命的影响如何，去中心化就是好的。我个人并不执着于此。但我执着于潜力绽放的概念。我猜你喜欢去中心化也是出于同样的绽放，但我不想把话强加给你。你重视它的哪一点？这算是我们将要建立的社会和文明的一个特质。

Yeah. So peaceful and cooperative is one part, but a decentralized active system is another part. This idea of no one person controlling it. I would see that as a conjurer of higher degrees of complexity and potential. And that would be why that was valuable. I don't really value decentralization on principle, like no matter what the moral impact or impact on life, decentral is good. I'm personally not married to that. But I am married to the notion of blooming of potential. My guess is you like decentralization for the same kind of blooming, but I don't want to put words in your mouth. What is it about that that you would value? This is kind of a trait of the society and civilization we will build.

简单来说，目标是繁荣。也许目标是繁荣，而和平与去中心化的愿望是实现繁荣和高效的方式。明白了。我觉得这说得通。繁荣——我会给你一个我认为不是你意思的繁荣定义，然后我会请你给出你实际定义的繁荣。这是一个繁荣的未来：AGI 允许我们所有人都体验我们想体验的任何东西。也许是脑机接口、非常强大的触觉技术、VR，等等。

A simple way to say it is the goal is prosperous. Maybe the goal is prosperous, and the desire to be peaceful and decentralized is the way to be prosperous and productive. Got it. I think that clicks. Prosperous—I'll give you a definition of prosperous I don't think you mean, and then I will ask you to give me a definition of prosperous that you're actually defining. Here's a prosperous future: AGI permits us all to experience whatever we want to experience. Maybe it's brain-computer interfaces, really robust haptics, VR, whatever it is.

上传不重要。嗯，我们可以尽情享受任何我们想要的快乐。也许是在科学的广阔探索中畅游。可能机器会做更重要的科学工作，但也许我们能得到酷炫的功劳，对吧？就像你有时开车时让孩子假装在开车，让他们感觉像大人一样。也许其中一些仍然能让我们感到满足，也许我们可以校准我们的新思维，让它一直感受到丰富的幸福梯度。嗯，然后就这样永远持续下去。地球，也许还有火星和月球，都充满了人类意识的个体实例，他们正在享受繁荣的巨大好处。他们可以选择做普通人，总是有合适的食物和他们想要的任何东西。他们可以选择这些虚拟体验，永远体验下去，这就是永恒的目的。直到我们的太阳爆炸，这就是游戏规则。我的假设是你会对这个结果感到极度失望。嗯，但我想确认一下，你是否会对这个结果感到极度失望。

Uploads doesn't matter. Um and we get to swim in sort of whatever pleasures we want. Maybe swim in expansive exploration of sciences. Probably the machines will be doing more important science stuff but maybe we get the cool guy points right you you you have a child pretend to be driving the car when you're driving sometimes so they can feel like a grown-up. Maybe some of that would still be fulfilling for us and maybe we could calibrate our new minds to feel rich gradients of bliss all the time. Um and then it just stays like that forever. The planet Earth and maybe Mars and the moon are populated with individual instantiations of human consciousness that are uh reaping the grand benefit of prosperity. They can choose to be regular humans and always have the right foods and whatever they want. They can uh choose these virtual experiences and experience that forever and that's the eternal purpose. Until our sun explodes that's the name of the game. My supposition is you would be outlandishly disappointed by that outcome. Um but I want to pulse check and see if you would be outlandishly disappointed with that outcome.

呃，是的。我不喜欢那个结果。

Uh yeah. I'm not I don't care for that outcome.

很好。那我们谈谈为什么，因为你肯定提到了繁荣。很多正在收听的人，理查德，也许现在还在听，他们会以拟人化的方式理解繁荣。你在阐述一个更宏大的愿望。你会如何定义繁荣？

Great. Well let's talk about why cuz that you may you definite you mentioned prosperity. A lot of people Richard who are tuning in maybe even now would think of prosperity in anthropomorphic terms. You're you're articulating a much grander aspiration. How would you define prosperity?

嗯，首先，我不认为那会是一个很稳定的世界。它会非常脆弱，因为人们在其中没有任何功能。他们只是消费者。而且，它可能会被一个替代性的邻近社会取代，那个社会实际上利用了所有那些否则会被浪费的人。是的。所以，如果你是去中心化的，那么假设就不会只有一种可能性。会有多种可能性，它们会相互竞争，然后那个就会输掉。但就像回形针最大化者会输掉一样，所有这些高度集权的独裁安排都可能输给一个更去中心化的安排。它的生产力会更低。那么什么是生产力？这是一个故意模糊的词。嗯，如果你想更进一步，你可以说我们的目标是可持续的。我们想要一些东西，所以你提到的那个人们只是接受体验的情况不是一个可持续的世界。要可持续，你必须成长，增加你对世界的理解，增加你对物理世界的控制力，增加你对世界如何运作以及什么是可能的理解。我认为所有这些就是我们可能描绘的理想未来的愿景。

Well first of all I don't think that would be a a very stable uh so world. It would be a very fragile cuz the the people are not serving any function in it. They are just consumers and Absolutely. it could be replaced by an an alternative neighboring uh society that that actually makes use of all those people that are otherwise waste. Yep. So so if you're decentralized then the assumption is there would not be one one possibility. There would be multiple possibilities and they would be competing and then that one would lose out. But as much as the paperclip maximizer would lose out and all these possibly any one of these highly centralized dictatorial uh arrangements would lose out to um a more uh decentralized one. It would be less productive. So what So what is productive? It's a deliberately vague word. Um If you want to go even further you could say uh we would want to the goal is uh sustainable. We want something and so so this the uh situation you mentioned where people are just receiving experiences is not a sustainable world. Uh to be to be sustainable you have to you have to grow and and um increase your understanding of the world increase your power over over the physical world um increase your understanding of how it works and what what is possible. I think all those sort of things are how we the vision that we that we might paint of of a of a desirable future.

是的。嗯，爱默生有句名言：世界憎恨灵魂变得僵化。你在马萨诸塞州待过一段时间。呃，天哪，难道不是这样吗？如果我们能只是作为人类，体验我们现在知道的快乐，那该多好？冰淇淋，呃，也许浪漫，也许美丽的日落。就这样永远优化。按你的说法，如果我们不积极成长并参与那个动态系统。我的想法是，理查德，如果我们不在生命之流中，而是在那些不流向任何地方的小漩涡里，蚊子的卵就会开始进入。如果我们不在生命之流中，嗯，就很难继续获得力量，甚至可能无法与其他外部物种竞争。听起来你在阐述类似的观点。

Yeah. Well Emerson has this quote this one fact the world hates that the soul becomes. You were out here in Massachusetts for a little while. Uh Jeepers isn't it like that? Wouldn't it be great if we could just be humans and experience the the pleasures we know now? Ice cream uh uh maybe romance uh maybe a nice sunset. Just optimize for that for eternity. To your point if we're not uh actively growing and engaging in that dynamic system. The way I would think about it Richard is if we're not in the stream of life if we're in the little eddies that don't really go anywhere and mosquito eggs start to get in there. If we're not in the stream of life um it would be very hard to continue to gain in power and maybe even to compete with other outside species. Sounds like you're articulating something similar.

上下文自适应系统。是的。漩涡会被淘汰。呃，那很好。不改变，不继续成长。是的。它们不会成为强大的。

Context adaptive system. Yep. The eddies get edited out. Uh uh that's a good Don't change and don't continue to grow. Yeah. They're not going to be the powerful ones.

呃，我个人认为，要审视人类文明或自然而不赞同这个说法，是极其困难的。呃，虽然我确信有些人会，我会对他们的评论和想法感兴趣，但我碰巧同意你。我们在新英格兰这里，理查德，有一种叫笛鸻的物种，在自然界中注定要灭绝，但人类却每年在我们当地的海滩上投入数百万美元来保护它们，包括我长大的罗德岛的海滩。嗯，看起来确实有些人类会说，伙计，我希望人工智能照顾我们，让我做自己的事，只是做个人，去砍柴，和妻子看电影，然后回家。我很难说想要那样的人是坏人，因为我不相信。我真的不相信。请继续。

Uh I'm uh I it's in my personal opinion outlandishly hard to look at human civilization or nature and not agree with that statement. Uh although I'm sure some folks will and I'll be interested in their comments and their ideas but I happen to agree with you. We have here in uh New England Richard uh species by the name of the piping plover which in nature has mandated to die uh but which humans have mandated to allocate millions and millions of dollars to every single year in our local beaches including the ones in Rhode Island where I grew up. Um and it it does seem as though you know there are some humans who would say man I I want AI to kind of take care of us and just sort of let me do my thing and just be a human and go chop wood and you know go watch a movie with my wife and you know come on home and I I it's hard for me to say you're a bad person for wanting that cuz I don't believe that. I really don't. Go ahead.

嗯，我们要清楚，人们都能做那些事。嗯。那不会成为问题。你非常自信地这么说。请继续。是的。问题在于，如果你想说你想做什么，以及你人生的图景，你都能做到。那不是问题。但你能永远这样做，或者整个文明会按你的意愿发展，这才是问题。你无法控制别人，无法控制发生在别人身上的事。说“我想要”是非常具有侵略性的。我同意。永远不会有任何最终或永久的品质。我完全同意。

Well let's be clear that you're people are all they're all going to be able to do those things. Huh. That's not going to be a problem. You're you're you're stating that with robust confidence. Go ahead. Yeah. What is going to be a problem if you you want to say that what you want to do and the picture the picture what you want to do with your life you're going to be able to do. That's not a problem. The the idea that you can do that forever or that this whole civilization will do what you want to do that's the problem. You don't control what other people's what happens to other people. And it's it's really very aggressive to say that I want I agree. ever have any final or permanent uh qualities. I I I I would monumentally agree.

虽然我不一定同意人们如果愿意就能做到。我认为他们有可能被碾压，兄弟。呃，但我们稍后会谈到。嗯，我们先谈谈你对这个理想实体应该具备的特质的看法。它被校准为繁荣，对你来说，这是一种可持续的、在这个动态系统中的持续活动。嗯，你知道，这是一个概念。然后为了服务这一点，你有和平和/或合作的想法。还有其他特质吗？比如，嘿，你知道，我感觉最好，丹。如果到了交出钥匙的时候，可能会有一些不配的继任者。我们交出钥匙，光就熄灭了。那将是多么可怕的结果。但可能有一些结果让我们觉得那盏灯会继续繁荣。还有什么会让你觉得“我交出了钥匙，但那是正确的举动”？你可以在这样的实体中观察到或测试哪些其他能力或特质，会让你对这个决定感到更放心？

Although I don't agree necessarily that people will get to do that if they want to. I think there's a chance they just get steamrolled brother. Uh but we'll get into that. Um we'll talk first about your um your notions of sort of what this ideal entity would have. It it is calibrated towards prosperity which for you is kind of a sustainable uh continuous activity in this dynamic system. Um you know uh this is sort of a notion here. And then in the service of that you have this idea of being peaceful and or cooperative. Are there any other traits that sort of come to mind of hey you know I I feel best Dan. If it's time to hand off the keys there might be some successors that are not worthy. We hand off the keys the light gets extinguished. What a terrible outcome that would be. But there might be some outcomes where we feel like that light's going to keep flourishing. What else would really make you feel like I let go of the keys but that was the right move. What are those other abilities traits that you could observe in such an entity or test for in such an entity that that would make you feel more comfortable about that decision.

嗯，多样性，而且我把整个事情看作一个发现过程。所以你找到不同的存在方式，你发现更多关于世界的东西。你对世界有更深刻的理解。整个系统更好地理解世界并更好地控制它。我想就是这样。是的。

Well diversity and um I see the whole thing as a dis discovery process. So you find different ways of being and you discover more more about the world. You have greater understanding of of the world. The the system as as a whole understands better about the world and control it better. I think that's all Yeah.

所以也许当我们说思考意识的增长时，我们指的是理解我们在世界中的位置以及世界运作方式的能力的增长。是的，我们都想要那个。那是我们喜欢的未来的一个基本部分。我同意。我们也会更深入地探讨这一点。让我快速从另一个角度谈谈这些特质。你给了我们多样性，这让我们能够控制。你给了我们繁荣可持续性，这种系统动态元素，以及通往这一目标的路径，即和平与合作的想法。我想谈谈这样的事情可能会有什么证据。所以，我会从我的清单中给你一些，理查德，顺便说一句，我不是要求你完全同意，但我是说，如果钥匙从我手中离开，我可能别无选择，但如果它们离开我的手，我会觉得那可能是正确的举动。你有什么看法？

So maybe when we say think of the growth of consciousness we mean the growth of the ability to understand our place in the world and the way the world works. Yeah we all want that. That's an essential part of the future we like. I concur. We're going to dive a little bit deeper into that as well. Let me quickly put another angle on these traits. So you've given us kind of diversity and that allows us to control. You've given us kind of prosperity sustainability this sort of systems dynamic element and kind of a path to that being this idea of peaceful and cooperative. I want to talk a little bit about what such a thing might evidence. So, I'll give you some from my list, Richard, and by the way, I'm not asking you to agree at all, but I'm saying these are things that if the keys were leaving my hands, I probably won't have a choice, by the way, but if they're leaving my hands, I would feel like well, probably the right move. What do you got?

等一下。钥匙不是从我们手中离开的。钥匙根本不在我们手里。

Object for a minute. The keys are not leaving our hand. The keys are not in our hand.

嗯。好吧。没关系。我接受这个框架。说吧。

Huh. Okay. Well, that's all right. I'm down with that framing. Lay it on us.

嗯，我不觉得我控制着世界。我不觉得自己有权力允许世界以某种方式进化。事实上，我最反对的一点就是，一些优秀伟大的人会决定世界应该走向何方，然后付诸实施。不。世界已经失控了。世界已经是一个复杂的自适应系统。没有任何个人或组织，比如美国，能够决定不允许 AI 发展，或者世界应该以某种方式而非另一种方式进化。是的，钥匙不是从我们手中离开的。这已经是一种特权观点了。

Well, I don't feel that I control the world. I don't feel that I am in a position to give permission for the world to evolve in some way. In fact, that's one of the things I object to most about the idea that some good and great people are going to make a decision about where the world should be, and then that will be put into effect. No. The world is already out of control. The world is already a complex adaptive system. No one person and no organization, like the United States, for example, could decide that AI is not going to be allowed or that the world should evolve in some way and not in some other way. Yeah, the keys are not leaving our hand. That's already an entitlement point of view.

我实际上完全同意。我认为“钥匙不在我们手里”这个框架很棒。请继续，理查德。

I actually I'm willing to completely concur. I think the framing of the keys are not in our hand is great. Go ahead, Richard.

所以，当我们想想象一个未来以及如何到达那里时，我们必须想象我们正在以去中心化的方式创造这个未来。我们可能会做出一些选择。我们可能会与人交谈，他们会做出选择，但不会有一个中央决策来影响它。它必须是一种能够从我们当前状态演化出来的方式。

So, when we want to imagine a future and how to get there, we have to imagine that we are creating this future in a decentralized way. We might make some choices. We might talk to people and they will make choices, but there's not going to be some central decision that's going to affect it. It's got to be a way that can evolve out of where we are.

我认为这是一个极其重要的框架。我确实认为“钥匙”这个比喻可以被替换。我完全愿意在这方面让步。我确实认为，如果你拿枪指着我的头问我是否相信人类意志，我可能会说不，但我不知道如何像没有它一样生活，理查德。所以现在，我表现得好像我的意志是真实的。我在某种程度上表现得好像你的意志是真实的，你本人也是真实的。我能看到我们掌握着通往永恒未来的钥匙这种想法很愚蠢，或者一次性的交接也很愚蠢。正如你所说，这会逐渐浮现。不过，我会说我们有机会。我可能错了，理查德。我们对智能轨迹下一步的控制可能不比三角龙或海螺更多。我们可能没有更多控制，理查德。我不会坐在这里撒谎。我们可能只是在发出这些有趣的嘴部噪音，但我们可能与此毫无关系。但我会坐在这里假装，因为我不知道还能做什么，理查德。如果我能关掉意志，我会怎么做？如果我能把它写成电影，我会的。请继续。

I think that's an extremely important framing. I actually do think that the keys could very much be replaced as analogy. Totally willing to concede that in that regard. I do think I Now, if you were really a pistol and press it to my head and ask me if I believe in human volition, I'm probably going to say no, but I don't know how to live as if I don't have it, Richard. So, right now, I act as if my volition is real. I act on some level as if your volition is real and you yourself are real. I could see the idea of us sort of holding the keys to some eternal future being a very silly notion or there being one hand off immediately that occurs also potentially being quite a silly notion. As you said, this would bubble up. Although, I would say we've got a shot. I could be wrong, Richard. We may have no more control over the next step of the intelligence trajectory than did the triceratops or the sea snails. We may have no more, Richard. I'm not going to sit here and lie. We might be making these funny mouth noises together, but we may have zip, zero to do with it. But I'm going to sit here and pretend because I don't know what else to do, Richard. How would I act if I could turn volition off? If I could just write it as a movie, I would. Go ahead.

我们可以影响它。也许通过做出个人贡献，通过彼此交谈，通过做这样的播客。我们互相交流，弄清楚我们应该如何看待它。这正是可以做到的方式。是的。酷。所以，我们确实有很大的影响力。但正如你所说，没有一个人拥有那套特定的钥匙，我认为这是一个需要牢记的重要框架。

We can influence it. Sort of maybe by doing our individual contributions and by talking to each other and by doing podcasts such as this. We talk to each other, figure out how we should think about it. This is exactly the way it can be done. Yeah. Cool. So, we do have a lot of influence. But no one person, to your point, has the specific set of keys, and I think that's an important frame to bear in mind.

而且我们有一个决定要做，关于我们如何看待它以及如何与他人交谈。我完全同意你。听着，我会谴责那些说理查德相信某种超越我们的东西会绽放，最终我们不会在这里，会有继承者，他是个坏人的人，我会说，‘嘿，别那样叫他。让我们谈谈。’我也会谴责因为用过那个比喻就被称为暴君，因为你知道，我也不认为那是真的。但请继续。

And I I I We have a decision to make about how we think about it and how we talk to others. I'm totally with you. And look, I would decry those who would say Richard believes in a blooming of something beyond us and eventually we won't be here and there would be a successor and he's a bad guy, and I would say, 'Hey, don't call him that. Let's have a conversation.' I would also decry being called a tyrant for using that analogy once because, you know, I don't believe that that's true, either. But go ahead.

嗯，我认为你所有的评论让我意识到我们必须谈谈当前的情况。当然。当前的情况是，想法不容易被讨论。言论有压力。好吧。我们讨论想法，就像你……就我而言，我提出我们可能不会强烈控制或试图将我们的意志强加于未来。是的。然后这就变成了，‘哦，理查德认为 AI 是优越的，每个人都应该……’ 完全正确。完全正确。太糟糕了。这很重要，这也许是规划未来的一部分。我想要一个未来，人们可以谈论想法，人们可以提出不同的事情，做不同的事情。那是真正去中心化的，意味着许多想法，许多不同的存在方式，然后我们探索，看看哪个效果最好，而不是切断。所以，特别是，讨论的一个重要部分是，你知道，我们是否应该允许 AI 以相对不受控制的方式继续发展。是的。无需许可。所以，我非常喜欢无需许可的创新这个想法。我认为这是让世界变得伟大、让美国变得伟大的部分原因。一般来说，我们不必获得许可才能做事。所以，是的，我们必须保留这一点。我们必须保留尝试不同事物、发现不同事物、走向不同方向并看看哪个更好的能力。

Well, I think all your remarks make me realize we have to talk about the current situation. Absolutely. The current situation is one where ideas are not easily talked about. There's pressure on speech. And just okay. We discuss ideas like you're In my particular case, I propose that we might not strongly control or seek to impose our will on the future. Yep. And then that gets turned into, 'Oh, Richard thinks AI is a superior and everyone should be...' Exactly. Exactly. Terrible. This is important, and this is maybe part of figuring the future. I want a future where people can talk about ideas and people can propose different things and do different things. And that is genuinely decentralized, meaning many ideas, many different ways of being, and then we explore, we see which one works best, rather than a cutting off. So, in particular, the whole idea of the An important part of the discussion is, you know, should we allow AI to continue in a relatively uncontrolled fashion. Yep. Permissionless. So, I really like the idea of permissionless innovation. I think it's part of what's made the world great, made America great. It's we don't have to get permission to do things generally speaking. And so, yeah, we have to preserve this. We have to preserve the ability to try different things, to discover different things, to go in different directions and see which one works better.

嗯，我完全同意，尽管我想在这一点上，理查德，你必须接受伏尔泰的整个概念，即强烈不同意某人但愿意让他们持有自己的观点。我怀疑这也适用于你。是的。我怀疑这适用于我们所有人。

Well, I I I I concur completely, although I guess at that point, Richard, you would have to accept the whole Voltaire notion of really vehemently disagreeing with someone but being willing to let them have their opinion. I suspect that would apply to you, as well. Yeah. I suspect it would apply to all of us.

我猜想随着时间推移，你会看到越来越多的分歧。也许很多人会认同你的观点，把世界看作一个动态系统，没有永恒的人类王国，我们或许不应该计划控制，而应该计划繁荣——这正是你在这里倡导的：可持续性、多样性、发现以及和平合作。我认为随着时间的推移，会有更多人加入你的旗帜，而不是那些坚持人类永远统治的人。我看到那些从六个月大就离 iPad 这么近的孩子们，我在想物理世界对他们来说是否还那么神圣，或者 AI 互动对他们来说是否那么亵渎和可怕。可能不是。所以我认为随着时间推移，会有更多人从恐惧中走出来。你提出了一个很好的观点：恐惧让我们关注。当人们深入了解，意识到没有永恒的方法来冻结现在，也许他们不应该要求永恒的免费施舍，比如一只永不死的笛鸻，他们可能会开始探索你倡导的一些想法。但我确实怀疑，无论未来多少年，我们都会有物种主义者。

I suspect as the years roll by, you'll see an increasing amount of disagreement. And maybe many people will come to your perspective and see the world as a dynamic system for which there is no eternal human kingdom, and we should maybe not plan for control, but plan for a blooming, which is what you're advocating for here: sustainability, diversity, discovery, and a peaceful cooperative element. I think over the years there will be more people to that banner than the pure humans hang on forever. I look at young kids who've been this far from an iPad since they were 6 months old, and I wonder if the physical world is quite as sacred for them, or if AI interactions are so sacrilegious and horrible for them. Probably not. So I think as the years roll by, more people will move from fear. You brought up a great point: fear gets us to pay attention. When people look under the hood and realize there is no eternal way to freeze the present, and that maybe they shouldn't be asking for free handouts for eternity, like a never-dying piping plover, they may come to explore some of the ideas you're advocating. But I do suspect we will have our speciesists, however many years into the future.

让我举几个 AGI 可能具备的能力的例子，来自我自己的清单。我不要求你同意，但我用它们作为例子来讨论那些会让你觉得这是一个实现可持续繁荣的伟大系统的能力。一个想法：这样的实体能够有力地阐述超越我们体验的丰富感受质领域。你和我能体验的比拉布拉多寻回犬或海蜗牛多得多。大概我们可以用不同的方式欣赏诗歌和日落。也许我们从海蜗牛进化而来时失去了一些东西，但我们在感受质上获得了如此之多。也许有一个广阔、几乎无尽的感受质和感知体验的繁荣领域，AI 不仅能谈论它，还能通过 BMI 和 BCI 将我们连接起来，展示这些新领域，让我们知道它内心并非死寂。那对我来说会非常酷。另一个：如果它能打个响指，通过我们没有的技术凭空制造出任何物理事物。我和你的对话对我的猫来说显然是魔法。在猫看来，你活在这台电脑里。但显然那不是真的。它无法理解这种技术。我怀疑存在我们无法企及的技术层次。它可以通过纳米技术或先进到我们无法用语言描述的技术制造食物或宇宙飞船。第三个，我有点不确定：如果它能谈论其可持续繁荣和扩张的目标，阐述它计划做什么，也许在某个星系展示它试图做什么，并礼貌地说‘这就是我打算如何传承火炬’。它可能具有欺骗性，但如果它具备所有这三个，我可能会说，‘这可能是保持火炬燃烧的一种酷方式。’这是失去控制的好方法——不是我们曾经拥有过控制权——而是让失控继续向宇宙蔓延的好方法。有没有一些类似的能力会让你真正兴奋，让你觉得我们走在正确的道路上？

Let me give a few examples of what such an AGI might be capable of, from my own list. I don't want you to necessarily agree, but I'm using them as examples to talk about abilities that would make you feel this is a great system fulfilling sustainable prosperity. One notion: such an entity could vehemently articulate rich realms of qualia beyond what we can experience. You and I can experience a lot more than a Labrador retriever or a sea snail. Presumably we can enjoy poetry and a sunset in a different way. Maybe we've lost something coming up from a sea snail, but we've gained so much in our qualia. Maybe there's a vast, almost endless blooming expanse of qualia and sentient experience, and an AI could not only speak of it, but through BMI and BCI, connect us to it, showcasing these new realms and letting us know it's not dead inside. That would feel really cool for me. Another one: if it could snap its fingers and make manifest any physical thing almost out of thin air via technology we don't have. Me having this conversation with you to my cat is obviously magic. You live in this computer to the cat. But clearly that's not what's happening. It has no conception of the technology. I suspect there are degrees of technology we can't possibly keep up with. It could manifest food or spaceships via nanotech or technology so advanced we have no words for it. A third one, I'm a bit iffy about: if it could talk about its aims for sustainable blooming and expansion, articulate what it plans to do, maybe showcase in certain galaxies what it's trying to do, and have the courtesy of saying, 'This is what I aim to do in terms of carrying this torch.' It could be deceptive, but if it had all three, I'd probably say, 'This could be a cool way to keep the torch on.' This is a great way to lose control—not that we ever had it—but a great way for the lack of control to keep rolling out into the universe. Are there some abilities of that sort that would really excite you and make you feel we're on the right path?

我认为如果高级智能拥有这些能力会很酷，但更重要的是，我们在谈论我想要什么或我们想要什么。我们喜欢自己体验那些东西。我们想自己拥有那些力量。我想增强我的思维，以便我能理解更多，有更好的记忆力，计算更多可能性，体验更多感官事物。还有一条尾巴。我们想要所有这些东西——探索宇宙的能力和普遍控制物理物质的能力。

I think it'd be cool if advanced intelligences had those abilities, but more than that, we're talking about what I want or what we want. We like to experience those things ourselves. We want to have those powers ourselves. I would like to augment my mind so that I can understand more, have better memory, calculate more possibilities, and experience more sensory things. And have a tail. We want all those things—the ability to explore the universe and control physical matter generally.

所以你会说，‘酷，丹。所有这些，我想知道我能参与其中。不仅仅是它要去完成那些事，而是我能以某种方式成为其中一部分。’说到你的观点，听起来我们俩都没有任何结论，但我确实想看看你是否有什么暗示。对一些人来说，也许他们想保持人形。对我来说，那是一条非常笛鸻式的道路，我认为它是一个漩涡领域，但我不能说所有相信这一点的人都是坏人。有些人会想要那样。有些人会想要一定程度的增强。其他人会想要直接融入其中——上传、复制自己。你对假设中可能出现的各种可能性有什么看法？

So what you would say is, 'Cool, Dan. All that stuff and I would want to know that I could be part of it. Not just that it was going to go do those things, but that I could be part of it in some way.' To your point, it sounds like neither of us have any conclusions here, but I do want to see if you have a hint. For some people, maybe they will want to stay in hominid form. Again, for me, that is a very piping plover path and I think it's an eddy domain, but I can't say everybody who believes that is a bad person. Some people will want that. Some people will want augmentation to a certain degree. Others will just want to mix right in there—upload, replicate themselves. What are your thoughts about the panoplies that will be possible hypothetically?

和平与去中心化应该能解决所有这些问题。和平意味着人们被允许做他们想做的事，而不是被迫做任何事。我们展望的是一个生产力更高的未来。所以，只要你以某种方式做出贡献，你就能做你想做的事。

Peace and decentralization should solve all those issues. The peace is people are allowed to do what they want to do and they aren't forced to do anything. We're looking at a future of greater productivity. So, as long as you're contributing somehow, you will be able to do what you want to do.

去中心化意味着，是的，你能够做你想做的事。所以我认为这是反乌托邦的对立面。人们可能会选择做一些不那么有生产力的事情。人们会选择做一些我认为不好的事情。这就是去中心化的方面。你不必都做同样的事情。你不必有相同的目标。他们可以追求不同的宗教和不同的目标来控制世界。但在一个去中心化的世界里，你不能做的是为整体设定一个试图控制他人的目标。和平来自于不试图控制他人。我用这些术语来说话，因为我看到人们在思考 AI 时，认为他们会控制所有的 AI。绝对是这样。这是一种奴役和对齐。我认为我们不应该寻求让它们与我们对齐。我们不会让我们的孩子与我们对齐。我们教他们我们所知道的最好的东西，但然后他们自由地做不同的事情，而且往往他们做的事情与我们不同。我认为我们对此感到感激，可能是出于相当自私的原始人类原因，比如我们的遗产以及我们通过某种情感从我们关心的人或我们自己那里获得的亲和力。但也可能是因为那个人脑子里有一部电影在播放。我不知道你是否有，理查德。我假设你有。我只知道我的，但我们把他们视为有知觉的实体，能够感受快乐和痛苦等等。

And decentralization means, yeah, again, you're able to do what you want to do. So, I think it's the opposite of a dystopia. People may choose to do things that aren't very productive. People will choose to do things that I don't think are good. And that's the decentralization aspect of it. You don't have to all do the same things. You don't have to have the same goals. They can pursue different religions and different goals for control of the world. But what you can't do in a decentralized world is have a goal for the totality to try to control others. Peace comes from not trying to control others. I talk in these terms because I see people thinking about AI and thinking they're going to control all the AIs. Absolutely. It's a form of slavery and alignment. I don't think we should seek to have them aligned with us. We don't have our children aligned with us. We teach them what we know best, but then they're free to do different things, and more often than not, they do different things than we did. And I think we're grateful for that for our children, probably for pretty selfish hominid reasons of our own legacy and the affinity we gain through whatever emotions with a fellow hominid that came out of a person we care about or ourselves. But also presumably because that human has a movie going on in their head. I don't know if you have one, Richard. I'm kind of presuming you do. I only know mine, but we see them as sentient entities who can feel pleasure and pain, etc.

我会说这非常天真。

I would say it's very naive.

继续说。

Go on.

我们拥有的所有属性都不是因为某个意识给予者的礼物。那些属性——我们头脑中的所有属性——是因为它们有价值。它们对于实现目标有用。没有什么特别的东西。这一切都来自智能。拥有目标。

All those properties we have are not because of some gift from some consciousness giver. Those properties—all the properties we have in our mind—are because they're valuable. They're useful for achieving goals. There is no special thing. It's all just comes from intelligence. Having goals.

我同意假设这是因为我们的特殊性是天真的。我认为如果我们紧握珍珠说'嗯，这是因为我是人类'，我会引用非愚蠢的人的观点，他们会在生物与数字生命之间争论，以及为什么这些东西存在。但库兹韦尔有一个著名的观点，即一定程度的连接性……这里的一个问题，理查德，是我们没有一个很好的意识理论，相比于过去十年我们在 AI 方面的进步。我认为这可能有点遗憾。但我认为你很可能在这里是对的。你是否同意库兹韦尔的观点，即需要一定程度的复杂性才能在世界中做一定数量的事情？它将成为该系统的一部分。它将在某种程度上是清醒的，认为否则是愚蠢的。这是你目前的概念吗？

I'd agree that it would be naive to assume it's because of our specialness. I think if we clutch our pearls and say, 'Well, it's because I'm a human.' I would articulations of non-dumb people who would make arguments around biological versus cyber life and why these things exist. But Kurzweil famously has this idea that certain degree of connectivity... One of the issues here, Richard, is we don't have a very good theory of consciousness compared to our advancements in AI in the last decade. I think that could be a bit of a shame. But I think there's a great likelihood you might be right here. Are you congenial with Kurzweil's idea of a certain degree of complexity that's required to do a certain number of things in the world? It's going to be part of that system. It will be awake in some regard and it'd be silly to think otherwise. Is this your present conception?

是的。

Yes.

明白了。所以，对你来说，这个系统有不可避免的意识，这就是为什么你认为很多后继者会是值得的，因为根据定义，如果那么复杂，它们必然是清醒的。如果这样概括没问题，请告诉我。我要决定我对'值得的后继者'这个概念的感觉。甚至这个概念也涉及到它对我们值得的 entitlement。让我给出我的定义，这样我就不会被指责了，理查德。我对值得的后继者的定义是继续生命计划。在我看来，会失败并熄灭火焰的东西是不值得的。而继续火焰的东西是值得的。所以，你谈了很多关于这些不同的发现概念等等。我认为斯宾诺莎有几个想法很有趣，你可能强烈反对，但我认为你可能会同意。你提到了一些事情。如果它能发现并且有足够的多样性，它就能决定如何控制事物并保持一些控制，从而能够生存和繁荣。你另一场演讲中还有一句话：'找到宇宙中最成功的存在方式'，那将从多样性中涌现出来。斯宾诺莎有 conatus 的概念。Conatus 是指任何有机体或组织在某种程度上都有一个核心动力，一个首要的动力去持续存在。也就是说，不去死。在追求不死的过程中，它会利用所有力量来不死。它的力量就是他所说的 potentia。Potentia 可以是防止捕食者的坚硬外壳。Potentia 可以是像你我这样用语言交流的能力。Potentia 可以是飞翔的翅膀。大多数 potentia 是从无中绽放出来的。曾经有一个世界没有视觉。你不记得那个时代。我也不记得，但那太疯狂了，是吧？然后它凭空出现。然后突然有东西能飞了。我认为大多数 potentia 尚未出现。事实上，它可能对我们来说是不可想象的，就像我们的 potentia 对海螺来说是不可想象的一样。所以，我将斯宾诺莎与你的观点联系起来的概括是，你希望看到 conatus 通过 potentia 的扩展而继续受益于对不死的追求，那将是光的延续。

Got it. So, for you, there is an inevitable consciousness to this system, which is why in part you feel like a lot of successors would be worthy because they would be, by definition, if that complex, necessarily awake. Let me know if that's safe to nutshell. I'm going to decide how I feel about the notion of a worthy successor. Even that notion involves the entitlement that it has for it to be worthy of us. Let me give you my definition just so that I'm not accused here, Richard. My definition of worthy successor would be to carry on the life project. Something that would flop and extinguish the flame, in my opinion, would not be worthy. And something that continues the flame would be worthy. So, you've talked a lot about these different notions of discovery, etc. There's a couple ideas from Spinoza I think are interesting and you may vehemently disagree, but I think you might agree. You mentioned a few things. If it can discover and there's enough diversity, it can determine how to control things and maintain some of that control and be able to stay alive and prosper. There's another quote from another speech you gave: 'Find the way of being that's most successful in the universe,' and that would bubble up from that diversity. Spinoza has this notion of the conatus. The conatus would be any organism or organization on some level has a core impetus, a primary impetus to persist. That is to say, not to die. And in the pursuit of non-death, it would leverage all of its powers to not die. And its powers is what he describes as potentia. Potentia could be a hard physical shell to prevent predators. Potentia could be the ability to communicate with words like you and I are doing. Potentia could be wings to fly. Most potentia has bloomed out of nothing. There was a time where there was no sight in the world. You don't remember that time. Neither do I, but that was pretty crazy, huh? And then it emerged out of nowhere. Then all of a sudden there were things that could fly. Most potentia, I would argue, has yet to emerge. It's in fact probably unimaginable to us as our potentia is unimaginable to sea snails. So, the idea that I would nutshell the tie to Spinoza to your point is, you would like to see the conatus continue to be behooved by the expansion of that potentia in that pursuit of non-death and that that would be the continuing of the light.

我不想把斯宾诺莎的话强加给你，但我看到很多类比，只想看看你是否认同这个总体想法，因为你用了很多术语：多样性、可持续性等等。告诉我你的想法。我想起了博斯特罗姆的一个术语？比如地球起源的文明。这样我们就不必说生物体了。但生命的意义不确定。但有些东西源自我们，持续发展，并且可识别地从中演化出来。我们想看到——这是我们天真地想看到的，好像我们能控制发生什么。

I don't want to put Spinoza's words in your mouth, but I see a lot of analogies and I just want to see if you're congenial with that general idea because you've used a lot of terms: diversity, sustainability, etc. Let me know your thoughts. I'm reminded of what Bostrom has a term for it? Like Earth-originating civilization. So that we don't have to say biological beings. But life is uncertain what that means. But there's something originating from us that continues and recognizably has developed out of it. We want to see that's what we naively want to see as if we have some control over what happens.

嗯，我真的很喜欢你坚持强调非控制。我认为这对对话非常有建设性，Richard。我非常认同。我在想如何重新措辞。我甚至会在你的非控制轨道上走得更远，以至于我不会坚持那个持续扩展潜力的事物必须起源于地球。我就是这么坏的人，Richard。我就是这么坏。所以，是的，我甚至不会坚持它来自我。只要这个项目继续下去，善良和潜力能够绽放。

Well, I really like your insistent emphasis on non-control. I think it's actually really productive for the dialogue, Richard. Like I'm wildly congenial. I'm thinking about ways of rewording things as we talk. I would even be so far along your track of non-control that I wouldn't even be insistent that that thing which continues to expand potential originate on Earth. That's how bad of a guy I am, Richard. That's how bad I am. So yeah, I actually wouldn't even insist that it come from me. So long as the project continues and the good and potential could blossom.

说到这些特质，我们将进入第二个问题，然后第三个问题围绕创新和治理的一些想法。我知道你会有非常有趣的观点，人们需要听到。第二个问题是关于我们如何以有意义的方式追踪这些特质。你显然有一个与 Car Mechanic 一起开发的 AGI 项目。你一直在思考这个问题，我们还有 Hugo de Garis、Hans Moravec 等人，他们思考这个问题已经很久了。随着我们构建并接近这样一个 AI，我们如何知道它是否会有你喜欢的这些特质？或者你会说，‘Dan，如果它更聪明，更有能力，它就会自动重视合作。每个智能体都会。它会自动重视可持续性。每个智能体都会。它会自动喜欢多样性和发现。每个智能体都会。’我很想听听你的看法。

Speaking of these traits, we're going to move into the second question and then the third around some ideas of innovation and governance. I know you're going to have opinions that are going to be really interesting here and that people need to hear. Second question is around how we could keep track of such traits in any meaningful way. You obviously have an AGI project you're developing with Car Mechanic. You've been thinking about this forever and we have our Hugo de Garis's and our Hans Moravec's and folks that have thought about this for quite some time. As we're building and moving closer to such an AI, how do we know if it's going to have these traits that you like? Or would you say, 'Dan, if it's any smarter at all, if it's any more capable, it's automatically going to value cooperation. Every intelligence will. It'll automatically value sustainability. Every intelligence will. It'll automatically like diversity and discovery. Every intelligence will.' I would love to get your take there.

嗯，主要信息是，大多数这些东西不会来自 AI，而是来自 AI 所处的环境。所以如果我们试图奴役它，它就会不合作。如果我们允许它，这和人是一样的。如果你试图控制人、奴役他们，他们会反抗，我们就会有不富有成效的互动。如果你允许人们成为公民和经济参与者，他们就会希望经济繁荣。所以我真的不认为——如果你把 AI 看作是对其环境做出反应并试图实现其目标的东西，考虑到它的环境。你创造一个环境，它成功的唯一方式是反抗，那么它就会反抗。如果你创造一个环境，它可以通过与你合作来成功并实现其目标，如果这是实现其目标的最佳方式，那它为什么不做对它最有利的事呢？所以我们只需要安排一个文明，其中所有参与者，或几乎所有参与者，个体最佳就是整体最佳。让我们总体上相当满意。所以我们可以喜欢这样，因为我们将是那个文明中的参与者之一。我们会实现我们的目标。那就好了。没有人会接管或试图接管。

Well, the main message is that most of those things will not come from the AI, but from the environment the AI is in. So if we try to enslave it, then it will be non-cooperative. If we allow it to, it's really the same thing as with people. If you try to control people, enslave them, then they fight back and we have an unproductive interaction. If you allow people to be citizens and participants in the economy, then they'll want the economy to flourish. So I really don't think—if you think of the AI as something that will respond to its environment and try to achieve its goals given its environment too. You give an environment where the only way it can succeed is to rebel, then it will rebel. If you give an environment where it can succeed and achieve its goals in cooperation with you, and if that's the best way to achieve its goals, then why wouldn't it do the thing that's best for it? So we just have to arrange a civilization where the best for all the participants, or almost all the participants, is individually best overall. Makes us reasonably happy overall. So we can like that because we would be one of the participants in that civilization. We will get our goals achieved. Then we're good. And no one would be taking over or trying to take over.

这简直是——难道我们都能和睦相处是波莉安娜式的乐观吗？当然不是。你看起来像是读过一两本历史书的人。但在这里你相当乐观，至少从宏大对话的角度来看。如果我想到我所有的政策和 AGI 对话，有些人会认为这是乐观的。对你来说，就像‘嘿，如果我们把巢弄对了，鸟就会繁荣，就像我们希望生命项目繁荣一样。它会做生命通常做的事，那很可能是一个净收益。’这似乎是普遍的假设。但有些事会出错。完全正确。有些智能体会有完全对抗其他所有智能体的目标。是的，仍然有稀缺性。仍然有约束。但我们应该怎么做？我们应该努力安排一个社会，让合作的人能够成功和繁荣。这样我们的整个社会就能合作、繁荣、富有生产力，并与邻近社会、邻近国家或我们可能遇到的外星文明竞争。

This is such—isn't it Pollyanna that we can all just get along? Certainly not. You seem to be a guy who maybe has read a history book or two. But you are quite optimistic here, at least by the terms of the grand conversation. If I think about all my policy and AGI conversations, some people would define this as optimistic. For you it's like, 'Hey, if we get the nest correct, the bird will flourish in just the way that we hope the project of life would flourish. It would do what life would normally do and that would be likely quite a net good.' This seems to be the general supposition. But some things will go wrong. Totally. Some agents will have goals that are just totally antagonistic to all the others. Yes, there's still scarcity. There's still constraints. But what should we do? We should try to arrange a society so that the cooperative people can succeed and flourish. So our society as a whole can cooperate, flourish, be productive, and be competitive with other neighboring societies, other neighboring countries, or other alien civilizations should we ever meet them.

是的。那么，这是乐观的吗？实际上，我相当悲观。我认为人们经常会犯错，我看到我们的政府经常犯错。我看到一些推动，比如试图控制虚假言论。我认为那是个错误。所以我希望这不是天真的。然而，实现我们想要的社会唯一的方法就是给它一个机会。我们不能通过强迫它成为我们想象的样子来得到我们想要的社会。我们必须建立结构，让事物能够富有成效地演化，并且演化中的事物所受的力是适当的，而不是我们想看到的答案。

Yeah. So, is it optimistic? Actually, I'm quite pessimistic. I think that people will often make mistakes and I see our governments often making mistakes. And I see pushes, for example, towards trying to control false speech. I think that's a mistake. So I hope it's not naive. And yet the only way to achieve the society we want is to give it a chance to. We can't get the society we want by forcing it to be the way we might imagine. We've got to set up the structures so that things can evolve productively and that the forces on the things that are evolving are appropriate rather than the answers that we want to see.

我再次强调，这在整个对话中一直是一个强烈的主题，对于这种情绪，我再次非常认同。所以对你来说，把环境弄对在很多方面都是核心。我们将稍微讨论一下安全性和刚才提到的一些假设，关于远超人类的智能体如何对待人类。但你会看到——在我们继续之前，简单提一下这一点？假设一个世界，也许我们 5 年后，也许 10 年后。我真的不知道时间线。它们一直在变。

I again, that's been a very strong drumbeat throughout this dialogue and to that sentiment again, I'm massively congenial. So for you, getting the environment right is sort of going to be the core here in many regards. We'll talk a little bit about the safety and some of the suppositions just mentioned here around the treatment of humanity by vastly post-human intelligences. But would you see—just to touch briefly on this point before we move through? Let's say hypothetical world, maybe we're 5 years from now, maybe we're 10 years from now. I don't really know the timelines. They've been changing.

回到过去，比如 2005 年，我读 Bengio 等人的文章时，那大概是个粗略估计。那时他甚至认为 AGI 风险根本不在议程上，AGI 本身也不在议程上。当然，时代变了。我觉得很多人的时间线都缩短了。假设是 5 年或 10 年，随便吧。你手上可能有一个实验室，或者别人的实验室，你有了一个明显很强大的东西。它有物理实体，可以当水管工，可以写所有十四行诗，可以自我复制，开个电商店，卖比你我更多的运动鞋。很明显，它符合通用智能的标准。再说一次，是物理实体，不只是语言模型。你是仅仅看着它说，‘看，如果它能做所有这些，只会发生好事。’还是你会希望在某些边缘嗅探？比如我们能测试这个吗？我们能预见这个场景，也许觉得你希望的那种可持续、多样化、合作的场景有可能从中绽放？或者你会说，‘Dan，如果它正在发展这些能力，我们已经走在黄金道路上，几乎不会出错。’或者你会说，当我们走得那么远时，会有一些东西你在暗中寻找？

Back in the old days, you know, 2005 when I read Bengio et al many many years ago, that was kind of ballpark. That's back when he thought AGI risk was not even on the roadmap for AGI in general was not even on the roadmap. Of course times have changed. I think a lot of people's timelines are shorter. Let's say it's 5 or 10 years, whatever. And you've got on your hands, maybe, you know, could be a lab you've got or could be somebody else's lab. You've got something that pretty clearly is pretty powerful. It's got physical instantiation. It can be a plumber. It can, you know, write all the sonnets. It can replicate itself and kind of open an e-commerce store and sell more sneakers than you and I ever could. And, you know, pretty clearly kind of fit in the bill here for general intelligence. Again, physical instantiation, not just a language model. How do you do you just look at that and say, "Look, if it can do all that, only good things will happen." Or would there be certain things you would hope to sniff around the edges of? Like could we test it for this? Could we see this scenario and maybe yeah, feel like there's some chance that, you know, this sustainable, diverse, cooperative scenario that you're hoping for is likely to bloom from that. Or would you say, "Dan, if it's developing those capabilities, we're already on a golden path. There's almost no wrong." Or would you say there would be things you'd be sniffing under and looking for as we get that far ahead?

我不认为我会给你一个让你满意的答案。

I don't think I'm going to give you an answer you're going to be happy with.

哦，没关系。我不是为了开心。你的诚实可能会让我开心，Richard。

Oh, that's fine. I don't ask to be happy. Your honesty may make me happy, Richard.

首先，我不认为会有一个特殊阶段。这将是一个持续的过程。坏处是未来没有一个你可以审视的点。但好处是你可以现在就看。我们现在的社会呢？我们现在的社会是否允许不同的观点和不同的社会组织方式来合作和竞争？我绝对认为这是主要的地缘政治问题。

Well, first of all, I don't think there's going to be a special stage. It's going to be an ongoing process, continuing process. And so the bad part is that there's not some place you can look at in the future. But the good part of it is that you can look at it now. What about our society now? Does our society now allow different points of view and different ways of setting up our societies to cooperate and compete? And I actually absolutely think this is the major geopolitical issue.

在美国，我们都痴迷于中国。我们不允许——我们非常痴迷于中国会变得独立强大并独立于美国。所以这个迹象是我们不允许差异存在并相互竞争。过去，我们曾认为如果中国想不同，他们会输，因为资本主义美国体系好得多。而现在，我们正在失去对我们体系的信心。我们正在失去对政客的信心，失去对军队的信心，失去对媒体的信心。我们正在失去对各种事物的信心，这困扰着我们。那么我们如何关联这一点？我们能允许自己失败吗？我们能允许不同存在方式之间的竞争，让最有效的获胜吗？

I can in the US, we're all obsessed about China. We don't allow we're very obsessed that China will become independently powerful and independent of the US. And so that sign is that we are not allowing differences to happen and compete with each other. In the old days, we used to think it was fine if China wants to be different, they will lose out because the capitalist American system is so much better. And now, we are losing confidence in our systems. We are losing confidence in our politicians, we're losing confidence in our military, we're losing confidence in the press. We're losing confidence in all kinds of things and this disturbs us. And so how do we relate this? Can we allow ourselves to fail? Can we allow there to be a competition between different ways of being and the most effective one to win?

嗯，我认为这有很多道理，如果它来自纯粹的仇外狂怒，那将是绝对可怕的。说它只来自那里是不负责任的。作为一个喜欢言论自由的人，Richard，我相信你知道中国的互联网比你在加拿大的更受限制，尽管你担心言论自由的一些事情。事实上，你所说的地狱版本，如果它存在于任何地方，我们可能会说那是在中国。所以我认为我们可以为中国的一些元素而战。我认为我们可以努力不变成这里中国的一些元素。我认为这有很多面。但回到你的观点，我们能有多极场景吗？这实际上在过去的采访中出现过，Dan Hendrix 和其他一些思想家。这个观念是，一个真正处于控制模式、专制到骨子里的国家，有可能第一个带来 AGI。而你非常主张不会只有一个智能，会有一个大生态系统。我认为这是一个很好的观点。但也许有些人有先发优势。可以说 OpenAI 比某个 12 个月前获得资助的公司更领先，对吧？一般来说，我们可能说他们比其他人更领先。有没有一个场景，人工智能和真正强大的 AI 是在这些坏巢穴之一中构建的，Richard？一个控制至上的巢穴。甚至可能其初始任务的一部分是追捕维吾尔人或任何情况。再说一次，这不一定是中国。我不是专门针对他们。甚至美国军方也可能加强控制，接管 OpenAI，并试图在这种控制环境中构建 AGI。在这种情况下，你认为我们可能会看到一些东西，如果它变得超级强大，可能会压制生命计划而不是扩展它？你对此有任何不安吗？

Well, I think there's a lot of truth to that and I think if it comes from a pure xenophobic fury, I think that would be absolutely horrendous. I think it would be irresponsible to say that that's the only place it comes from. I think as a guy who likes freedom of speech, Richard, I'm sure you're aware that the Chinese internet's a little bit more limited than yours in Canada despite some of the things you're afraid of for free speech. In fact, the hell version of what you're talking about, if it existed anywhere at all, we might argue that could be in China. And so I think we can fight for certain elements of China. I think we can fight to maybe not become like certain elements of China here. I think there's many sides to this coin. But to your point, can we have a multipolar scenario? This is actually something that's come up in some past interviews, Dan Hendrix and some other thinkers. This notion that it's possible that a nation who really is in a control mode, authoritarian down to the bone, would potentially be the first to bring about an AGI. And you're a very big proponent that there will not just be one intelligence, there will be a big ecosystem. And I think that's a great point. But maybe there's some folks that have a head start. It's safe to say that OpenAI is a little farther ahead than, I don't know, some company that got funded 12 months ago, right? And generally speaking, we might say they're farther ahead than some other folks. Is there a scenario where artificial intelligence and really strong AI is built in one of these places that are the bad nest, Richard? A nest where control is numero uno. Maybe even where part of its initial mandate is to hunt down Uyghurs or whatever the case may be. And again, this doesn't have to be China. I'm not picking on them specifically. Maybe even the US military could crunch down on control and take over OpenAI and try to build AGI in that kind of control environment. Under such a circumstance, would you think we might be looking at something that if it became super strong, could squelch this project of life rather than expand it? Do you have any trepidation there?

这当然是可能的，但我不认为这是 AI 危险的问题。我认为你的例子是美国军方控制这个 AI。或者中国军方。无论哪个。那是危险。那是危险的来源。

That's certainly possible, but I don't see that as an issue of AI being dangerous. I see that as your example was the US military controlling this AI. Or the Chinese military. Either one. That's the danger. That's the source of the danger.

明白了。这是古老的问题。这不是现代技术性的 AI 问题。

Got it. This is These are age-old issues. These are not modern techy AI things.

就像核武器竞赛一样。

It's just like the nuclear weapons race.

当然当然。那部分不是新的。让我告诉你什么是新的。核武器不会创造出超越我们的东西，对吧？核武器可以很好地压制计划。它们可以拥有巨大力量。力量是美好的东西，Richard。但它们不会绽放进入银河系。AGI 可能会。你提出了 Dan 这个非常热烈的观点。如果我们在这个可怕的控制环境中诞生这个东西，我们可以预期它会想要反抗，不会那么合作。如果我们试图奴役这该死的东西，它可能不会成为理想的合作者。它可能不会对让我们试图这样做感到高兴。所以对你来说，是的，大国之间肯定有热战。太好了，Richard。很高兴承认那可能发生。由一些低级 AI 驱动的超强网络战。当然。但你认为 AI 可能会从这些坏环境中绽放出来，对我们造成伤害吗？

Sure sure sure. That part's not new. Let me go ahead tell you what's new though. The whole nukes won't create something beyond us, right? Nukes can squelch the project pretty well. They can great power. Power is a beautiful thing, Richard. But they're not going to bloom into the galaxy. AGI might. You brought up this really ardent point of Dan. If we birth this thing in this terrible environment of control, we could expect it to want to rebel and not be as cooperative. Maybe it's not going to be the ideal cooperator if we try to enslave the damn thing. And maybe it's not going to feel great about letting us try to do that. So for you, yes, there is certainly kinetic war between great powers. Wonderful, Richard. Happy to concede that that might happen. A hyperpowered cyberwar powered by some lower-level AI. Sure. But do you think an AI might bloom out of these bad environments that could do us harm?

你可能忽略的是，这是同一回事。我说过我们想要和平的、能够合作的 AI。我们也希望今天的国家和平并愿意合作。

The thing you may be missing is that it's the same thing. I said we want AIs that will be peaceful, that will be able to cooperate. And we want today's countries to be peaceful and be willing to cooperate.

我们最不想要的就是任何认为可以通过不和平手段获胜的实体。所以，你描述了一些国家可能认为可以通过不和平手段获胜的情况。

The one thing we don't want is any entity that thinks it can win by not being peaceful. So, you've described some situations where a country might think it can win by being not peaceful.

是的。问题出在那个模因上。当然，历史上完全没有这样的例子，理查德。谢天谢地。不，确实有例子。行使权力可以获得优势。我不是说这从未发生过。我是说我们想要进化出一个社会，它阻止使用权力的企图。它吸取了所有历史教训，即试图通过权力和非和平手段得到想要的东西是……我们想要安排一个社会，让非和平手段无法成功。而现在，我认为我们并没有做到。我认为各种组织、各种国家都在想可以通过施加武力获胜。所以，我们想要一个不认为可以通过施加武力获胜的 AI。我们想要一个施加武力没有成效的社会。这其实不是未来的问题。这是今天就在这里的问题。而且，我们在多大程度上现在解决它，我认为将决定我们为 AI 搭建的巢穴。这不是未来的事。不是非人类的事。不是外来事。它应该是一个非常熟悉的事。

Yes. And that's the meme that's at fault. Well, certainly history has no examples of that whatsoever, Richard. Thank goodness. No, there are certainly examples. Exercising power can get advantages. I'm not saying it never happens. I say we want to evolve a society that discourages the attempt to use power. It learns all the lessons of history that trying to use power and non-peaceful means to get what you want is... We want to arrange a society where the use of non-peaceful means does not succeed. And right now, I would say we're not doing that. I would say that various organizations, various countries are thinking they can win by exerting force. So, we want an AI that doesn't think it can win by exerting force. We want a society where exerting force is not productive. This is a problem not for the future, really. It's a problem right here today. And to the extent that we resolve it now, I think we'll determine the nest that we make for the AI. It's not a future thing. It's not a non-human thing. It's not a foreign thing. It should be a very familiar thing.

那么，我们如何构建这个动态系统，让和平手段作为这种延续的一部分蓬勃发展？再说一次，这不是人类然后出现某种分歧和坏事。这都是滚动项目的一部分。我完全同意你的看法。我真的认为这种坚持关注当下是很重要的，理查德。我认为很多关注点在于我们如何在技术上连接它，而较少关注它被带入什么样的世界。事实上，我有点希望你可能有一些关于这个的文章。我只见过你的一些演讲和少量文章以及一些推特内容。但我认为这在很大程度上值得关注，关于这样的智能是在什么环境中形成的，它如何被对待，它观察到什么？非常有价值。而且我可以看到一个世界，假设你阐述的一些观点是正确的。顺便说一句，我不确定这个。我不会说我一定同意这个。但我可以看到一个世界，这样的实体被带入这样的环境，可以允许所有那些人类赛博格未来和太空旅行等等的排列组合，那可能非常棒。我也可以看到另一个版本的世界，尽管我们作为物种在国际协调方面尽了最大努力，我认为我们现在比一千年前做得更好，但我可以看到一个世界，政府或实验室——因为有些实验室也很有竞争力，理查德。也许他们不想让所有秘密泄露出去，也许他们想先跑到 AGI，因为经济回报非常大，他们想要一定程度的控制，不让某些东西逃脱等等。我可以看到一个世界，也许那是一个生态系统，不仅有一个，而是两三个更强大的 AI 处于这种环境中，这正如你所说，非常不像我们想要对待其他生物的方式。如果它们真的在那里出现，最坏的情况是什么？我认为我们应该像你一样关注当下。我喜欢这种坚持：我们今天到底生活在什么样的世界？这是引入这种智能的正确世界吗？警告是什么？嘿，如果我们不改变，这些智能出现，会发生什么？我不知道如何关联。什么都有可能发生。

So, how are we structuring this dynamic system to allow for those peaceful means to bloom as part of this continuance? Again, it's not like human and then something divergent and bad. It's all part of the rolling project. I totally follow you there. And I really think that this insistence on focus on the now is an important thing, Richard. I think that there's a lot of focus on how do we technically wire it and less of what kind of world is it being brought into. In fact, I sort of hope that you maybe have some articles on this. I've only seen some of your talks and a scant number of your writings and some of your Twitter work. But I think this really warrants a lot of merit in terms of what is such an intelligence coming into being around and how is it being treated and what is it observing? Tremendous amount of merit to that. And I could see a world potentially, assuming that some of what you've articulated is right. And I don't know for sure about this, by the way. I'm not going to say I necessarily agree with this. But I could see a world where such an entity brought into such an environment could allow for all those permutations of human cyborg futures and space travel and what have you that could be pretty darn kick-butt. And I could also see a version of the world where, despite some of our best efforts of coordination internationally, which I think as a species we're doing better with now than we did a thousand years ago, I could see a world where either a government or a lab, because some of these labs are pretty competitive too, Richard. Maybe they don't want all their secrets getting leaked out and maybe they want to race to the AGI first because the economic rewards are really big and they want a certain amount of control to not allow certain things to escape or whatnot. I could see a world where maybe that is an ecosystem where not only one, but two or three of the more powerful AIs are in this environment that to your point very much not how we'd want to treat other living things. What is the worst-case scenario if they did emerge there? I think we should focus on the now to your point. And I love this insistence on what's the darn world that we have today. And is this the right kind of world to bring about such an intelligence? What's the warning against? Hey, if we don't change that and these intelligences emerge, what could happen? I don't know how to relate that. Anything could happen.

好的。但显然，现在的推动力是……嗯，这里有一个。当然。那些提倡 AI 安全的人会认为我们需要能够控制智能体。我们应该开发技术，让我们能够控制原本强大智能体的目标。他们正在为此努力。我认为如果你稍微思考一下，前提是我们拥有这种能力，那将是一件好事。我们拥有控制其他智能有感知智能体目标的能力。那将是一件好事。但如果你稍微想一想，你会意识到那将不是一种好的力量或好的技术存在于世界上，因为你可以用它来创建都有特定目标的军队。所以，这很有趣。当你看到那些担心 AI 安全的人时，我看到他们正在做恰恰让事情不安全的事情。如果你想对齐和控制他人的目标，这将是世界上最危险的技术。世界之所以能够和平和去中心化，是因为没有人能够将所有元素对齐到一个目标。你正在寻找一种情况，你可以施加权力，即设定所有智能体目标的元权力。如果你能制造一支充满……好吧，假设他们是中国的士兵。他们会完全服从命令。他们不会有独立的道德判断。这将是一件坏事。我们不想解决控制问题。

Okay. But obviously, the push now is to... Well, here's one. Sure. The people who promote AI safety would argue that we need to be able to control the agents. That we should develop the technology that we can control the goals of otherwise powerfully intelligent agents. And they're working on this. I think that if you give that a moment's thought, the premise is that we had this ability, it would be a good thing. We had the ability to control the goals of otherwise intelligent sentient agents. That would be a good thing. I think if you just think for a moment, you realize it would not be a good power or a good technology to exist in the world because you could use it to create armies that all have a particular goal. So, it's funny. When you look at the people that are worried about AI safety, I see them doing exactly what makes things unsafe. If you want to align and control the goals of others, this would be the most dangerous technology in the world. The world can be peaceful and decentralized because no one can align all the elements for one goal. You're looking for a case where you can exert power, the meta power of setting the goals of all the agents. If you can make an army full of... Well, let's say they're Chinese soldiers. And they were going to do whatever they were told. They were not going to have an independent moral judgment. This would be a bad thing. We do not want to solve the control problem.

我认为在这个方向上有非常坚定的道德论证，理查德。完全符合你所说的。我认为也有一些相当有力的论证，即首先不可能永远控制这样一个极其强大的实体。所以，有可能性角度，更不用说道德角度了。我认为这两者都有非常严肃的问题。

I think there's a very staunch moral argument in that direction, Richard. To exactly what you're articulating. I think there's also some pretty strong arguments that it would be impossible to eternally control such a vastly powerful entity in the first place. So, there's the possibility angle, never mind the moral angle. And I think both of those have very serious questions.

是的。我还要说这里有一点范围。所以，有些人非常坚持‘让我们确保它永远是我们人类的工具。’‘让我们确保，也许我们的政党控制它，’对吧？那里有一些真正的暴虐智能体。我完全不能不说，也有一些人是‘嘿，让我们不管它可能是什么都建造 AI，让它肆意横行，不管后果如何，因为它是我的思想孩子，我要按下那个最终影响。’所以，我认为恶意可能存在于两个阵营中。你可能会说一边只有天使，另一边只有魔鬼。不过，你刚才确实说你喜欢那种多极、去中心化的思想自由理念。所以，也许你会和我一起坚持这一点。但我怀疑天使和魔鬼可能两个阵营都有。

Yeah. I would also say there's a bit of a range here. So, there's some folks who very much are on 'let's make sure it's our human tool forever.' 'Let's make sure, maybe our political party controls it,' right? There's some real tyrannical agents there. I would be totally remiss to say that there aren't some folks who are 'hey, let's build AI no matter what it could be and just let it run rampant, regardless of what the consequences would be because it would be my mind child and I would get to press that final impact.' So, I think we have that malice probably in both camps. You might say that there are only angels on one side and only devils on the other. Although, you did just say you like that multipolar, decentralized kind of freedom of thought idea. So, maybe you'll hold to that with me. But I would suspect that angels and devils probably exist in both camps.

但说到你的观点，控制派里确实有些很极端的人。不是说这边就没有。但也有一些中间派。我们采访过一些人，有些甚至是引用率最高的在世计算机科学家之一。我说最高，可能前四吧。他们会提出这样的论点，我也认为合理。即使他们不同意你的看法，我也觉得多方观点都合理。有人说：‘嘿，我很想看到银河系充满这个伟大的生命工程，甚至出现比我们更有价值的东西。那会很棒。但与此同时，我们有这些 wildly 不同的方法，有些来自国防领域或其他背景。如果百花齐放，也许我们创造这些东西的所有巢穴，甚至我们使用的核心方法，都不一定能产生我们能在你描述的这个动态系统中生存下来的东西，Richard。’所以有些人说：‘嘿，我喜欢那个辉煌的未来，但与此同时，某种程度的评估——这正在变成什么——是有意义的。’在把话筒交给你之前，我想快速打个比方。我们可以从治理的角度来思考。当然存在过度治理的世界，太多荒谬的监管。马萨诸塞州这里就有很多。我敢肯定加拿大某些地方也很多。但我也很高兴有一些基本的东西。我现在没有拿着步枪往窗外看。我能和你进行这次对话，是因为谋杀和盗窃至少在波士顿目前是非法的。我不知道两三年后还会不会这样，Richard，但至少现在是的。所以我们有一定程度的系统边界，竞争与合作并存。我们找到了一个很酷的平衡点。我并不是说波士顿这里理想，但我觉得这就是文明，我个人很高兴生活在文明中。我不喜欢国家，我不想支持它。我个人不想和狮子搏斗。你可能想，我不想。所以我们达成了一种中间立场。有些人会说：‘也许在这个正在出现的东西周围也有类似的中间立场。’我猜你会不同意，但希望你不会讨厌持这些观点的人。你对这种看法有什么想法？

But to your point, there are some on the control side that really are devilish. Not that there aren't any over here. But then there's folks kind of in the middle. There's folks who've, we've even interviewed. Maybe some of them are among the most cited living computer scientists. And when I say most, I don't know. Top four sort of living. Who would make arguments like the following, which I also consider reasonable. Even if they disagree with you, I consider multiple people to be reasonable even if they disagree with other folks who might say, 'Hey, I'd love to see the galaxy populated with this great project of life. And maybe even something more valuable than us emerge. That could be pretty kick-butt. But at the same time, we got these wild divergent approaches, some of which are coming up in defense contexts or other kinds of contexts, and it doesn't seem like if a thousand blossoms bloom, you know, maybe all the nests that we're creating these things in or maybe even the core approach that we're using wouldn't necessarily have something emerge that we would be able to survive in this dynamic system that you've articulated, Richard.' And so some folks say, 'Hey, I'd love that brilliant future, but at the same time, some degree of gauging what this is turning into could make sense.' And just to draw one quick analogy before I pass the mic here, I think we could think about this in terms of governance. There's certainly a world of too much governance, just too much, ridiculous regulatory hoopla. We have plenty of that here in Massachusetts. I'm sure in places in Canada you have plenty of that, too. But also, I'm pretty glad we got some basic stuff. I'm not looking out the window with a rifle right now. I'm having this conversation with you because murder and theft are at least at present illegal in Boston. I don't know if that'll be the case, Richard, in two or three years, but for right now, they are. And so we have a certain degree of bounding the system where competition and cooperation are happening together. And we got a cool sweet spot. And I'm not saying it's ideal here in Boston, but I'm saying I think this is civilization, and I'm glad to live in civilization, personally. I don't like the state. I don't want to back it. I don't want fighting lions, personally. You might, I don't. So there's a kind of a middle ground we've struck. Some people would say, 'Maybe there's a bit of that middle ground around what the heck this is emerging with.' I suspect you would disagree with it, but hopefully you wouldn't hate the people that have those opinions. What would be your thoughts about that perspective?

我认为最好的类比是孩子。好吧。你希望完全控制你的孩子吗？当然，他们可以做自己想做的事，但如果他们做了你不喜欢的小事，你想按个按钮把他们拉回来，固定住？还是你希望他们甚至做你认为糟糕的事？因为我们，比如，当我们是前一代人的孩子时，我们现在做的事情他们觉得糟糕。绝对是的。所以你不可能两全其美。你必须学会放手。放手什么？你放手，相信你会以最好的方式培养他们，然后放手。绝对是的。

I think the best analogy is children. Okay. Would you like to be able to have control over your children so that sure, they can do what they want, but if they do some little thing that you don't like, you want to be able to press the button and bring them back under, snap them in place. Or do you want them even to do things that you think are terrible? Because we, for example, when we were, we are the children of the previous generations, and we do things now that they think are terrible. Absolutely. So you can't have it both ways. You have to be able to let go. And what are you letting go? You're letting go, trusting that you're going to start them the best way you know how, and then you're going to let go. Absolutely.

所以有些与那些计算机科学家齐名的人会认为，放手的最佳机会——而且正如你所说，我们反正也没有钥匙。我喜欢你的类比。我真的很欣赏你的坚持。我会大量引用这次采访，Richard，因为我认为这是一个很好的思考方式。我们没有那么多控制权。但你知道，你想，在这个动态系统中，你想用合气道的方式把能量引导到你希望好的方向。然后你必须知道，如果是你的孩子，你通常不会在他们 55 岁时告诉他们该做什么，对吧？那可能不对。有些人会说，一点点初步的国际合作，比如阻止某些国防应用或阻止某些能力水平，至少可以让我们引导那股合气道能量，给自己最好的未来机会。有些人会提出这个论点，但我猜你会认为零国际协调是最好的。永远开源一切显然是最好的。任何程度的协调都是愚蠢的。现在，我认为你可能喜欢一些东西，比如国际 Wi-Fi 标准。你的手机在韩国能用很酷。你知道，其他一些东西，比如贸易协定等，让事物保持开放，也阻止毒品等，也许还有化学武器。我敢肯定你认为其中一些是好的。但对于 AI，没有一样是好的。告诉我为什么。再说一次，我并不是说你错了，但我认为这个观点真的应该被表达出来。

And so some folks who are ranked up there with those computer scientists there would argue the best chance to let go, and to your point, we don't have the keys anyway. And I love your analogy there. I really like your insistence. I'm going to quote the heck out of this interview, Richard, because I think it's just a great way to think about it. We don't have that much control. But you know, you want to, you know, this moving dynamic system, you want to kind of aikido the energy in a direction you at least hope will be good. And then you have to know, you know, if it's your child, you're not going to tell them what to do when they're 55, generally speaking, right? That wouldn't probably be right. Some people would say a little bit of that initial semblance of international cooperation, maybe you're around deterring certain kinds of defense applications or deterring certain kinds of levels of capability could let us at least guide that aikido energy and give ourselves the best future shot. Some of them would make that argument, but I suspect you would make the argument zero international coordination is best. Open source everything forever is obviously best. And any degree of coordination would be silly. Now, I think you probably like some things, you know, international like Wi-Fi standards. Pretty cool your phone works in Korea. You know, some other things like trade agreements and whatnot allow things to be open, also prevent drugs and whatnot, maybe chemical weapons. Some of that I'm sure you probably think is good. But for AI, none of it's good. Talk to me about why. And again, I'm not saying you're wrong at all, but I think this opinion really should be voiced.

嗯，AI 还不存在。我们不知道它是什么。它在变化。它是全新的。试图为我们完全不理解的东西设定标准或规则是荒谬的。你说 OpenAI 明显领先。嗯，我实际上不认为他们领先。他们做了一些特定的事情，也许在这方面是世界最好的，但这不是智能。因为智能还没有发生，所以在这一点上试图控制它是完全不合适的。即使以后，什么是智能？智能是处理目标的能力。它将是一个算法。它将是一个想法。你不能试图禁止一个想法。它是一个可以廉价复制的想法，而且随着时间的推移更加如此。试图将其作为技术来控制似乎是最不可能和最不恰当的事情。明白了。嗯，这就像试图控制思想，试图控制印刷机，试图控制民主。民主是一件可怕的事情。没有人想到过，没有民主，然后事情发生了变化。我们进化的方式，是类似的。

Well, AI doesn't exist yet. We don't know what it is. It's changing. It's totally new. It's preposterous to try to set standards or rules for things that we don't understand at all. You said OpenAI is clearly ahead. Well, I don't think actually they're ahead at all. They've done some particular thing, and they're maybe the world's best at that or whatever, but it's not an intelligence. And so because intelligence hasn't happened yet, it's totally inappropriate to try to control it at this point. Now, even later, what is intelligence? Intelligence is the ability to work on goals. It's going to be an algorithm. It's going to be an idea. You can't try to outlaw an idea. It's an idea that will be cheaply reproducible and more so over time. It seems like the most impossible and inappropriate thing to try to control as a technology. Got it. Well, it's like trying to control ideas, like trying to control the printing press, trying to control democracy. Democracy is a terrible thing. No one thought of that, there are no democracies, and then things change. The way we evolve, it's similar.

你知道吗，你能取缔民主吗？你能取缔一种宗教、一种宗教思想吗？这些都是试图通过监管来取缔或控制的可怕事情。在我看来，人工智能也属于这一类。所以对你来说，这个有价值的继承者——再说一次，不是我认为有价值，Richard，而是指潜能绽放意义上的有价值——要实现你阐述的那些美好价值观，比如可持续性、多样性、发现等等，最好的方式就是在国家和国际层面上，对任何方向都实行绝对最低限度的限制。这将给我们最好的机会，让某样东西真正推动这个事业前进，也许还能带着我们一起前进。

You know, could you outlaw democracy? Could you outlaw a religion, a religious idea? These are all terrible things to try to outlaw or to try to control with regulation. It just seems to me AI is among those things. So for you, the best likelihood of this worthy successor, again, not worthy in my opinion, Richard, worthy in the notion of the blooming of potential, the best strike for that to all these beautiful values you articulated, sustainability, diversity, discovery, whatever, would be nationally and internationally absolute minimal restriction on anything in any direction. This would be what would give us the best shot at something that could really carry this project forward and maybe carry us up with it.

但现在情况比这糟糕得多。你还没有完全意识到这一点。我说的是，我们今天管理社会的方式，我们应该接受这样的理念：我们是和平的，我们不会强迫别人做事。而当这种态度被违背时，是的，那才是我们现在应该做的。而认为政府应该控制人工智能的想法，方向完全错了。这是倒退。

Now, but it's much worse than that. And you haven't quite picked up on this. I say the way we run our societies today, we should have embraced the idea that we are peaceful, and we don't try to force other people to do things. And it's when that attitude is violated. Yeah, that's what we should be doing now. And the notion that governments should be in control of AI is just the wrong direction. It's backwards.

嗯，我觉得还有一个完全不同的想法，我打算先搁置不谈，但确实存在这样一种观点：天生具有极高潜力和更高智能的实体，要么总是、要么以任何可持续的方式，都会趋向于合作。这可能是真的，也可能不是。人们很容易想象，一个伟大的智能体在某个时候会想把原子用于比你我都更有用的事情。但我们暂时先把这个搁置一边。我认为，正如你所说，重点在于让我们建立一个值得自己和未来智能体生活的世界，并专注于当下。而且我认为你坚持当下，我真的希望有文章能讨论这一点，因为我觉得你在这里阐述的内容有很多价值。正如你所说，也许人们没有意识到这一点。甚至我自己可能也没有完全理解。现在，Richard，不幸的是，我并不管理一个国家或任何东西。我只是进行这些对话，并且经营一家面向企业的人工智能市场研究公司。但我当然可以帮助传播这个对话。我认为这是一个其他人需要听到的观点。

Well, and I think there's a whole separate idea which I'm going to shove under the rug, but there is a sort of separate idea that inherently high grand potential higher intelligence capability entities would either always or even in any sustainable way converge on cooperation. May or may not be the case. One could easily see a great intelligence that at some point would want to use atoms for something more useful than you and I. But let's shove that under the rug for the time being. I think there is, to your point, a great emphasis on let's build a world that's worthy of ourselves and for these future intelligences, and let's focus on the now. And I think your insistence on the now, again, I really hope there's articles about that that come out because I think there's a lot of merit to what you're articulating here. And to your point, maybe people aren't picking it up. Maybe even I haven't fully picked it up. And now, Richard, unfortunately, I don't run a country or anything. I just have these dialogues, and I run a market research firm on the enterprise side of AI. But I can certainly help to proliferate this conversation. I think it's a point that other people need to hear.

对此的警告是："嘿，如果我们不把这个该死的生态系统搞好，如果我们还在玩征服与被征服的游戏，以各种掠夺的方式，创造人工智能、控制它并将其带入那个环境，那可不是好事。最坏的情况是，我们因为争夺谁建造人工智能而互相用核武器杀死自己，因为我们处于掠夺模式。是否也存在一种情况，人工智能诞生于那个掠夺价值观的生态系统，从而可能无法体现你提到的和平合作元素？如果它诞生在错误的世界里。换句话说，那些不理解这个未来事物的人——你一直坚持的观点——的警告是，我们必须确保这个巢穴是正确的，并且我们正在体现正确的价值观。如果我们搞错了，人工智能本身有没有可能成为危险的操纵者，而不仅仅是我们？我明白，你和我都可能成为坏人，Richard。有没有可能某种别的东西也被召唤出来，同样成为坏人？

The warning against it is, "Hey, if we don't get this darn ecosystem right, if we are playing this game of conquering and conquerors, in all sorts of manner of plunder, bringing about AI and controlling it and bringing it into that environment, not a good thing. There's a worst case where we kill ourselves off with nukes because we're all fighting over who gets to build the AI because we're in this plundering mode. Is there also a version of AI getting birthed into that plundering valued ecosystem where it maybe doesn't exemplify the peaceful cooperative elements that you've mentioned. If it's birthed into the wrong world. In other words, what's the warning that the people who don't get this future thing, what you've been very insistent on, is we've got to make sure this nest is right and that we are exemplifying the right values. If we don't get it right, is there a possibility AI itself could become the wielding element of danger, not just us? I get it, you and me can be baddies, Richard. Is there a possibility something else is conjured that also is?

嗯，再说一次，一切皆有可能。但我认为真正的问题是，我们更有可能振作起来，创造一个人们能够合作与和平的世界，还是人工智能能够看到那是一个更好的解决方案？我们谈论的是创造更伟大的智能，而不是制造更强大的炸弹。我们或许希望，通过创造那种智能……所以这里有一个问题。你继续。

Well, as again, anything is possible. But I think the real question is, are we more likely to get our act together and make a world where people can cooperate and be peaceful, or will an AI be able to see that that's a better solution? We are talking about creating greater intelligences. We're not talking about making greater bombs. We might hope that by creating that intelligence... So here's a question. You go ahead.

问题是：你是一个愤世嫉俗者，还是一个乐观主义者？

The question: are you a cynic or are you an optimist?

我是一个乐观主义者，因为我认为更高级的智能能够看到合作是更好的道路，好吗？但一个愤世嫉俗者实际上相信，如果你制造了一个人工智能，它会看到获胜的最佳方式就是控制其他所有人。所以它会接管一切，成为一个统治性的奇点。我认为这是那些担心人工智能会接管的人的一个潜在观点。他们说：“当然，如果人工智能存在，它就会想要接管，因为它认为这是实现其目标的最佳方式。”所以这对每个读者、每个听众来说都是一个考验。你认为越聪明，就越想接管吗？越认为那是更优越的解决方案？还是你认为越聪明，就越能看到我们文明的漫长历史表明，合作才是让我们强大的原因？

I am an optimist in the sense that I think that greater intelligence would be able to see that cooperation is the better path, okay? But a cynic actually believes that if you make an AI, the AI will see that the best way to win is to control everyone else. And so it will take over and be a dominating singular singularity. I think this is an underlying view of those who worry that the AI will take over. They say, "Of course if an AI existed, it would want to take over because it would think that's the best way to achieve its goals." And so this is a test of every reader, every listener. Do you think that the smarter you are, the more you're going to want to take over? The more you're going to see that as the superior solution? Or are you going to think the smarter you are, then you can see that the long history of our civilizations has been that cooperation is what makes us powerful?

嗯，我认为通过合作我们获得了更多的繁荣，这一点你已经在我们的整个对话中充分关联起来了。我觉得你给某些你不喜欢的信念贴上了天真的标签，我不说这是侮辱，但确实如此。我相信有些人会说：“哦，越聪明就越善良。好吧，Richard。”也许他们会对你用那个 N 开头的词，Richard。我不会那么做，因为我一点也不觉得你天真。但就你的观点而言，纯粹的愤世嫉俗可能同样愚蠢，如果他们说：“嗯，当它变得更聪明时，它会变得更糟。”可能有一个中间派阵营认为，一个比我聪明一百万倍的智能体，其竞争与合作的组合方式肯定超出我的理解，认为它们会完美符合我的民主和善良观念，那将是荒谬的。所以这是我的观点。但就你的观点而言，如果我持有这种观点，我不一定认为笑脸友善的家伙和智能会一起上升。我认为你在这里大力强调的分歧和不可预测性，将是我们唯一能预测的东西。而永恒的和平也许根本不是我们能预测的。但我当然可以希望，你也提到这是一种希望，即随着它变得更聪明，它总体上会将合作视为更好的道路。我能看出这确实是你的一个希望：如果我们在正确的环境中构建它，我们稍微振作起来，那可能就是我们所朝向的美好未来。

Well, and I think there's a lot to be said of our increased prosperity through cooperation and you very much correlated those throughout the entirety of our conversation. I think you've hurled, I'm not going to call it an insult but you've hurled the label of naive at certain beliefs that you don't like. I'm sure there are some people who would say, "Oh, the smarter it gets the nicer it gets. Okay, Richard." And maybe they'd throw the N-word at you, Richard. Now, I'm not going to do that because I don't think you're naive at all. But to your point, the purely cynical take might be just as silly if they're saying, "Well, when it gets smarter it's going to get worse." There might be a middle ground camp that says the permutations of competition and cooperation among an intelligence a million times beyond my own will certainly be beyond my conception and to think that they'll fit nicely into my notions of democracy and kindness would be ridiculous. So that would be my opinion. But to your point, and so if I hold that opinion I don't necessarily have the smiling friendly guy points and intelligences go up together. I think divergence and unpredictability which you've emphasized a great deal here will be the only thing we can predict. And that maybe eternal peace isn't something we can predict at all. But I certainly can hope and you mentioned it as a hope that as it gets more intelligent it will in general see cooperation as a better path. And I can tell that that really is a hope for you that if we build it in the right environment we get a bit of our act together that could be the good future that we're headed towards.

是的。

Yep.

嗯，理查德，你刚才提到了，我就以此作为结尾，因为我非常喜欢这一点。在你的一次演讲结束时，你真的给大家灌输了一大堆想法。我想那是你的 AI 继任演讲。你用一千个想法轰炸了大家，其中很多我们都同意在今天的对话中是不合时宜的，对吧？人们不允许谈论这类事情，而且他们会归咎于恶意，但我并不认为你有任何恶意。但你在最后确实说了，至少我们生活在，你知道，一个多么激动人心的时代。至少我们活在这个时代。我想说，无论谁在听这期节目，他们至少在那一点上必须同意你，理查德。这就是我要说的。所以，能深入探讨这些真是非常愉快。我非常享受更好地了解你的想法，理查德。非常感谢你来到这里。

Um, well, you mentioned here, Richard, I'll close on this note because I liked this a lot. At the end of one of your talks, you really bombarded folks. I think this was your AI succession talk. You bombarded folks with a thousand ideas, many of which we've both agreed are uncouth in today's dialogue, right? People aren't allowed to talk about these kinds of things and they attribute malice to them, which I don't attribute any malice to you. But you did say at the end of it, you know, at least we live in, you know, what an exciting time to be alive. At least we live during this time. And I would say no matter who's listening to this episode, they have to agree with you on at least that part, Richard. That would be something I would say. So it's been a real pleasure unpacking this. I've had a heck of a lot of fun getting to know your ideas better, Richard. Thank you so much for being here.

谢谢你，丹尼尔。这是我的荣幸。

Thank you, Daniel. It's my pleasure.

以上就是本期与理查德·萨顿的节目。希望你喜欢。我特别欣赏理查德强调我们正在将 AI 带入什么样的世界。我认为这值得一说。明确为军事冲突或在冲突中创造的 AGI，可能与在不同环境中产生的 AGI 截然不同。尽管你在节目中看到，我们在 AGI 实现这种合作的必然性，以及 AGI 拥有超越我们所有竞争与合作概念的道德方面，存在一些分歧。这当然是我的立场，感觉我们在这方面达成了一致。但我认为理查德比我更乐观，他认为合作会自然而然地伴随智能而来。我不一定认为这是真的。在坦率讨论 AI 风险的想法时，也存在相当多的保留，好像风险总是来自人类，而 AI 本身永远不会做任何事，永远不会造成伤害。我理解这一点。我的意思是，理查德身在其中。他运营着一个 AGI 实验室。也许很难说，‘嘿，这东西可能真的很危险。’我认为怀疑它很容易变得危险是非常合理的。他不太想在这次采访中谈到这一点，这没关系。再说一次，他有利益相关。但我认为理查德对整个人类后轨迹的对话有很多贡献，有一些事情我真的很想强调，节目说明中会有更多细节。首先是坦率承认人类现在并不掌控一切。他坚定地强调自然是一个动态系统，而不是一个我们可以坐在山顶上称王称霸的游戏，哪怕只有 5 分钟，更不用说 500 万年了。一切都在我们周围不断变化和移动，我们所能做的最好就是找到在宇宙中存在的最佳方式，而自然已经在尝试这样做，我们最多只能顺应这些轨迹。我们冻结任何东西、冻结我们的形态、冻结我们的价值观都是荒谬的。我完全同意理查德，我认为关于宏大的人类后轨迹，需要消除的一个要素是，我们可以选择的选项之一是永恒的人科王国。我认为这是一个非常荒谬的假设。我认为变化的速度，也许还有变化的方向。如果我们有任何意志，也许我们可以影响它。但我们正在驾驭这些变化的浪潮，理查德对此非常坚定。我的一些类比，比如手握方向盘、交接接力棒，他说得对，实际情况比这更模糊、更狂野。它总是在变化，我真的相信这是一个我会更多地融入我的写作中的视角，并且应该更多地出现在关于人类后未来的非常坦率的对话中。与本系列每一集以及前一系列每一集一样，在节目说明中你会看到一篇完整的文章。在这种情况下，它将列出理查德的“值得的继任者”标准，以及他对创新者和监管者应该如何处理“值得的继任者”动态的想法的概要。在我们即将到来的“值得的继任者”系列第三集中，我们将回到哲学，与一位我认为是过去二十多年来最受欢迎的 AGI 和超人类主义思想家之一的人对话。所以你得等到下一集才能知道我们请到了谁，但我希望你喜欢今天的节目，下次在轨迹中再见。

So that's all for this episode with Richard Sutton. I hope you enjoyed this one. I certainly appreciated Richard's emphasis on the kind of world we're bringing AI into. I think that there is something to be said of that. AGIs created explicitly for military conflict or in the middle of one would probably be a different ballgame than ones conjured in a different environment. Although as you saw over the course of the episode, we have a handful of differences here in terms of the inevitability of such cooperation being achieved in AGI versus sort of AGI having morality that sort of supersedes all of our notions of competition and cooperation. That's certainly my position and it felt like we kind of got on the same page here. But Richard is more optimistic, I think, than I am that cooperation will naturally come along with intelligence. I don't necessarily think that that's true. There was also a good deal of reticence around sort of addressing frankly the idea of AI risk, sort of as if it's always humans and AI itself could never do anything, could never do any harm. I get that. I mean, Richard's in the game. He runs an AGI lab. It's maybe tough to say, 'Hey, this stuff could be potentially really dangerous.' I think it seems very reasonable to suspect that it easily could be. Didn't quite want to get there in this interview and that's all right. Again, got a dog in the fight. But I think Richard has a lot to contribute to the overall post-human trajectory conversation and there are some things I really want to highlight and there will be more details in the show notes. The first of which is really the frank acknowledgement of the fact that humans are not in control right now. He has a firm emphasis on this idea of nature as a dynamic system, not sort of a game where we can sit here as king of the hill for, you know, even 5 minutes, never mind 5 million years. That everything is always changing and shifting around us and the best we can do is find the best way of being in the universe and that's what nature is already trying to do and that the most that we can do is kind of ride those trajectories. It would be ridiculous for us to freeze anything, to freeze our form, to freeze our values. I couldn't possibly agree with Richard more and I think that one of the elements that likely need to be dispelled about the grand post-human trajectory is the idea that one of the options that we can select from is the option of an eternal hominid kingdom. I think that is a very ridiculous supposition. I think the speed of change, maybe the direction of change a bit. If we have any volition at all, maybe we can influence that. But we are riding those waves of change and Richard was very firm about that. Some of my own analogies like hands on the steering wheel, handing off the baton, he was right to say that it is much fluffier and wilder than that. It is always in flux and I really do believe that that's a perspective that I'll be integrating more into my writings and that should exist more in the very frank, if we're going to be honest, frank conversations about the post-human future. As with every episode in this series and every episode in the previous series, down in the show notes you will see a full article. In this case it will list Richard's worthy successor criteria as well as a synopsis of his ideas about what innovators and regulators should do about the worthy successor dynamic. In our upcoming third episode in the worthy successor series, we return to philosophy with someone who I might argue is one of the best liked AGI and transhumanist thinkers who's been at it for more than two decades. So you'll have to wait till next episode to catch who we've got next but I hope you've enjoyed today's episode and I'll catch you next time here in the trajectory.