达沃斯 2025：AI 的影响变得切身

大家好，欢迎收听《你的全神贯注》。我是特里斯坦·哈里斯。我是丹尼尔·巴克利。丹尼尔，我们最近在达沃斯参加了世界经济论坛年会。值得花几分钟让大家感受一下这次经历、这一周的真实情况，以及人们谈论 AI 的整体氛围，还有今年与去年相比有何不同。我们今年去了两次。实际上，去年我们也去了，今年又去了。去年充满了那些空洞的 AI 承诺。你知道，AI 无处不在，但都只是最薄的一层皮，说 AI 会改变世界之类的，对吧？感觉我们在 2025 年谈论这些就像逆水行舟。今年感觉截然不同。我想这是因为每个人都经历了地狱般的一年。AI 从推测性的“可能改变世界”变成了人们感觉它已经在改变世界，并且感受到了这种复杂性。而且今年对人们来说确实很艰难，对吧？政治上艰难，技术上艰难。发生了很多事。是的。我认为在这种背景下，世界领导人、经济领袖、公民社会领袖都对全球局势感到更加不安。因此，在对话中，我们提出的关于如何引导或管理人类度过这个过渡期，让我们都感到自豪，以及我们不能只是尽可能快地冲向这个目标，我认为这些观点以不同的方式被接受了，在达沃斯有更多的人愿意倾听这些观点。

Hey everyone, welcome to Your Undivided Attention. I'm Tristan Harris. And I'm Daniel Barkey. So Daniel, you and I were at Davos recently at the World Economic Forum annual meeting. It's worth just taking a few minutes to give people like a taste of what this experience is and what this week is really like and the vibe in general about how people talking about AI and what was different this year versus last year. We've gone twice this year. Actually, we went last year, we went this year. Last year was full of these big empty promises of AI. You know, AI was everywhere but it was all just the thinnest possible wrapper of AI's going to change the world and all this stuff, right? And it really felt like we were swimming upstream in 2025 talking about that. This year felt profoundly different. And I think it's because everyone's had one hell of a year. One AI's gone from being speculative, like it could change the world to people are feeling it's already changing the world and people are feeling that complexity. And also this year has just been really hard for people, right? It's been hard politically, it's been hard technologically. A lot has happened. Yeah. And I think in that context, world leaders, economic leaders, civil society leaders are all feeling a little more tenuous about the global situation. And so into that conversation, the points that we make about you know, how we need to shepherd or steward humanity through this transition in a way that we're all proud of and how we can't just run as fast as possible at this, I think they really landed in a different way and there were more people who were ready to hear those points in Davos.

是的，这一点非常重要。我们只是有了更多的证据。基本上，现在我们有凭据了，这就是区别。在过去的一年里，我们有了证据：失业，AI 暴露的工人中有 13%找不到工作。我们有了证据：AI 聊天机器人导致的自杀，比如 Character.AI 和 OpenAI 的亚当·雷恩案例。我认为，正如你所说，这让事情变得更加切身和真实，我们确实需要面对一些问题。还有关于欺骗的研究。那些研究走得很远，对吧？人们都多少了解了 Anthropic 的一些工作，关于 AI 模型以我们无法理解且不知道如何修复的方式进行策划、欺骗、撒谎。对吧。

Yeah, and that's such an important point. Like we just have so much more evidence. Basically, now we have the receipts is the difference. And in this last year, we've had the evidence now of the job loss, of the 13% drop in AI exposed workers that are not finding work. We've had the evidence now of the AI chatbot suicides that were caused by character.ai and Adam Rain in the case of OpenAI. And I think that, to your point, is making it much more visceral and real that there is something to reckon with here. Well, and the studies on deception. Those went really far, right? People all sort of understood some of the work that Anthropic did about AI models scheming, deceiving, lying in ways that we don't understand and we don't understand how to fix. Right.

所以，特里斯坦，我们在达沃斯做的一件事就是在 Human Change House 做了很多演讲。我们与不同的领袖、公民社会领袖、约翰·海特、心理学家亚当·斯坦、丽贝卡·温思罗普、约书亚·本吉奥一起参加了不同的小组讨论。在每个小组中，我们探讨了技术特别是 AI 如何改变人类的不同方面，以及我们如何塑造 AI 以确保它保留我们在人类体验中珍视的东西。是的。关于达沃斯，我真正欣赏的一点是，我想对 Human Change House 的玛格丽塔·路易-德雷福斯表示深深的、温暖的感谢。她既是我们工作的坚定支持者，也是这场关于技术对社会影响的对话能在达沃斯举行的真正原因。

So you know, one of the things Tristan that you and I did at Davos is we gave a lot of talks at Human Change House. And you know, we were on different panels with different leaders, civil society leaders, John Haidt, psychologist Adam Stein, Rebecca Winthrop, Yoshua Bengio. And in each of these panels, we looked at a different aspect of the way that humanity is being changed by our technology and by AI and how we want to shape that AI to make sure that it preserves the things that we care about in the human experience. Yep. And the thing I'll just say about Davos that I really appreciated and I want to just really put a big deep warm-hearted thank you to Margarita Louis-Dreyfus from Human Change House. She is both a deep supporter of our work and also really is the reason that this conversation of technology's impact on society is happening at Davos at all.

为了让听众感受一下，那是什么感觉？你就在滨海大道上。天寒地冻。有一长排商店基本上都被改造成了，你知道，Palantir House、Meta House、Google House 等等。我们得慢点说，因为要让人们理解达沃斯是什么样子，这太疯狂了，对吧？当然，世界经济论坛会议是达沃斯的中心，对吧？那就像会议中心。你在视频里看到特朗普演讲、尤瓦尔·赫拉利演讲的地方。世界领导人进去，入场费高得离谱，或者你得是国家元首之类的。但达沃斯不止如此。整个城市的其余部分，基本上就是一个城市。阿尔卑斯山中的一个小城市。城市，对吧？整个城市的其余部分，每一家商店，面包店、理发店，所有这些都被清空了一个月。是的。以前只是城市街道上的普通商店，现在被各个国家租用了。所以有蒙古馆、乌克兰馆，还有 Google 馆、Anthropic 馆，由公民社会组织租用，目的是展示正在发生的事情，或者试图说服人们不同的事情。有时是说服人们经济上的事情，比如想要领先的公司。但让我们说清楚，主要是这样。主要是公司花钱在广告牌上做宣传，然后邀请人们参加有助于推销这些宣传的演讲，这些宣传符合公司的利益。这是达沃斯大部分活动的明确首要动机。而且这些国家在那里设立场馆，是为了吸引外国直接投资，试图说服有能力搬迁公司的人将公司迁到他们国家。所以走在街上非常奇怪，对吧？通常卖羊角面包和德国面疙瘩的地方，突然变成了“把你的公司搬到世界各地”。是的。所以达沃斯很奇怪，对吧？我是说它很奇怪。你可以有很多方式去评判它。我当然有自己的判断。但同时它也有点神奇，因为所有这些偶然的碰撞发生在这些人之间。是的，当你走在滨海大道上，你会碰到国家元首和各种公司的 CEO，这是一种疯狂的体验。而且，为了向听众说明，我们去那里并不是因为我们认为达沃斯是促成所有改变的地方。但我想让你想象一下，你在滨海大道上，在 Palantir、Meta 和 Google 馆旁边，有一个叫 Human Change House 的场馆。整个星期，那里都有关于技术对社会影响的小组讨论，这些讨论没有利益驱动，是学者，像我们这样的人来谈论它将如何影响儿童，如何影响劳动力。而 Human Change House，在一个完全被利益驱动的世界里，它是一股清新的空气，充满了清晰和诚实。我真的认为它相当有影响力。

And just to sort of take listeners to, you know, what does it feel like? There you are in the Promenade. It's icy cold. There's this sort of big line of shops that have all been basically converted into, you know, Palantir House and Meta House and Google House and Can we slow that down cuz it I mean it's so wild for people to understand what Davos is, right? Cuz of course there's the World Economic Forum conference which is at the center of Davos, right? That's like the Congress Center. It's the where you see the videos of, you know, Trump speaking and Yuval Harari speaking. And that's where the world leaders go in and it costs some absurd amount of money to get in there and or you have to like be a head of state or something like that. But that's not what Davos is. Like the whole rest of this city, like it's basically a city. It's a small city in the Alps. City. Right? And the whole rest of the city, every single shop, you know, a bakery, a hair salon, all these different things have been emptied out for a month. Yep. And inside what used to be just the normal shops on a city street has been rented out by countries. So there's like Mongolia house, and Ukraine house, and you know, there's Google house and there's Anthropic house rented out by civil society organizations trying to and the whole point is to try to show people like this is happening or to try to convince people of different things. Sometimes it's convince people of economic things, like companies that want to get ahead. Well, but it's let's be clear, it's mostly that. It's mostly companies spending money to put propaganda on their billboards and then invite people to talks that help them sell that propaganda that is in the interest of their company. That's the clear first incentive of what most of Davos is. And often those countries are there making those houses to try to get foreign direct investment or FDI, to try to get convince people who have the ability to relocate their companies to relocate their companies inside the country. So it's very bizarre to walk down a street, right? That normally is selling, you know, croissants and spatzel and to all of a sudden be selling relocate your company across the world. Yeah. And so like you Davos is weird, right? I mean it's weird. You can there's plenty of ways to be judgmental about it. I certainly have my judgments. But also it's kind of magical at the same time because you have all of this serendipity of these collisions between these people. Yeah, as you're walking the Promenade, you bump into, you know, heads of state and, you know, the CEOs of various companies and it's it's a wild experience. And you know, to be clear, just for our listeners, we're not going there because we think that Davos is the place to make all the change happen. But I want you to imagine there you are in the Promenade and next to Palantir and Meta and Google House, there's this one house called Human Change House. And all week, there are panels about technology's impact on society that are not incentivized, that are academics, that are people like us coming and talking about how's it going to impact children, how's it going to impact, you know, the labor force. And Human Change House, it's a breath of fresh air of just clarity and honesty in a world that's otherwise just totally incentivized. And I really think that it was quite impactful.

还有像乔纳森·海特这样的盟友，你会在中间听到他们的消息。从晚餐到第二天早餐，你再次见到乔纳森时，他已经和法国总统埃马纽埃尔·马克龙会面，讨论他们正在推行的禁止 15 岁以下儿童使用社交媒体的新倡议。甚至在达沃斯之后，西班牙首相也表示他们正在实施禁止 16 岁以下儿童使用社交媒体的禁令。所以真正的势头正在形成，其中一些实际上发生在达沃斯。我认为今年我们真正希望实现的是从‘那是个有趣的对话’转变为‘让我们非常清楚。如果我们不想要默认的未来，那么我们必须要求一个不同的未来。我们必须建立实际的护栏和监管，才能实现那个目标。’是的，100%。

And allies like Jonathan Haidt, you know, you hear from them in between. The next time you saw Jonathan from dinner to the next breakfast, he actually met with President Emmanuel Macron of France about the new initiative that they're doing to ban social media for kids under 15. And since even Davos, we had Spain, the Prime Minister of Spain say they're enacting the ban for social media for kids under 16. So there's real momentum happening, and some of it is actually happening at Davos. And I think the thing we really want to happen this year is to go from 'that was an interesting conversation' to 'let's just be really clear. If we don't want the default future, then we have to demand a different one. And we have to build the actual guardrails and regulation that's going to get us there.' Yeah, 100%.

这就引出了我们今天与听众分享的讨论，那是我在 Human Change House 与约书亚·本吉奥教授一起做的。他是世界上最著名的计算机科学家之一。他开创了深度学习。他还运营着 Mila，魁北克人工智能研究所。他发起了一个新的非营利性 AI 安全研究计划，名为 Law Zero，这不仅关乎安全测试，更是一种从根本上安全设计的新型高级 AI。所以，我很喜欢约书亚的项目，对吧？因为他深入研究的一个问题是：为什么模型会被激励去欺骗和谋划？我们在几个播客中讨论过阿波罗和红木研究的一些成果，关于模型如何撒谎、作弊和产生幻觉。其中一个原因是模型所知与其目标之间没有差距。所以如果模型有一个目标去做某事，它会影响模型所说的关于你、关于世界的知识。约书亚看到了这个问题，并说：‘我们需要把这些分开。我们实际上需要一个纯粹表征性的 AI。’有时他称之为科学家 AI。它除了对自己所知保持纯粹真实之外，没有任何其他动机，并且完全与目标分离。所以约书亚认为知识与目标混合是 AI 的一个根本问题，并设计了 Law Zero，试图创建一种新的 AI 架构，将两者清晰分离，因为只有这样我们才能确保 AI 不会欺骗、操纵或胁迫。

And that leads us to the panel that we're sharing with listeners today, which is the one I did at Human Change House with Professor Yoshua Bengio. He is one of the best-known computer scientists in the world. He pioneered deep learning. He also runs Mila, the Quebec Artificial Intelligence Institute. And he launched a new nonprofit AI safety research initiative called Law Zero that isn't just about safety testing but really a new form of advanced AI that's fundamentally safe by design. So I mean, I love Yoshua's project, right? Because one of the things that Yoshua looked deeply at is why are models incentivized to deceive and scheme, right? We've talked about this on several podcasts of some of the Apollo and Redwood research about how models will lie and cheat and hallucinate. And one of the reasons is that there isn't a gap between what the model knows and what the model's goals are. So if the model has a goal to do something, it will influence what the model says that it knows about you, about the world. And Yoshua saw this problem and said, 'We need to split these apart. We actually need an AI that is a purely representational.' Sometimes he calls it the scientist AI. That only is not incentivized to do anything other than be purely truthful about what it knows and to separate that completely from having a goal. And so Yoshua sees this problem about this mixing between knowledge and goals as being a fundamental problem in AI and has designed Law Zero as an attempt to make a new architecture for AI that separates those cleanly because only then can we make sure that AI isn't deceptive, manipulative, or otherwise coercive.

描述得很好。特里斯坦和我在 Human Change House 做的所有讨论都将在 YouTube 和我们的 Substack 上发布。希望你们去看看。那里有很多精彩的内容。我还要特别感谢《经济学人》副执行主编肯尼斯·库基尔，我前一天晚上遇到他，他慷慨地提出主持我们与约书亚的讨论。享受讨论吧。

That's a great description. All the panels that Tristan and I did at the Human Change House will be available on YouTube and on our Substack. We hope you take a look. There's a lot of amazing content there. I just want to give one more thank you to Kenneth Cukier who is the deputy executive editor at The Economist who I ran into the night before and he generously offered to moderate our panel with Yoshua. Enjoy the discussion.

大家好，欢迎。非常感谢你们的到来。我们很高兴你们都能来。这真是太棒了。我们要讨论的是人类面临的最具戏剧性、在某些方面鼓舞人心的问题之一。它是慢性的、地下的、短暂的。那就是让人工智能与人类对齐。与我一起讨论这些问题的有两位杰出的思想家，最近也成为了活动家。第一位当然是约书亚·本吉奥。他无需介绍，所以我尽量简短。他是有史以来被引用最多的科学家。他也是深度学习之父之一，这项技术使人工智能从通过机器学习处理数据的良好方式，变成了我们今天谈论的通过智能体 AI 和 Transformer 模型等增强的版本。所以他是这个领域的标志性人物之一。他旁边是特里斯坦·哈里斯，他本人有着非凡的职业生涯，从在大型科技公司工作，到认识到大型科技公司的所有弊病，并将毕生精力投入到成为问题、更重要的是解决方案的代言人上。所以我现在想做的就是与他们两位对话，然后开放提问。但我们要尽可能清晰简单地讨论这个问题。所以，我将从一些非常基本的问题开始。第一个问题是：我们在谈论什么？什么是人工智能？

Hello and welcome. Thank you so much for being here. We're so pleased you can all make it. This is absolutely brilliant. What we're going to talk about is one of the most dramatic issues in some ways inspiring that humanity is facing. It's chronic. It's subterranean. It's ephemeral. It's aligning AI for humanity. And with me to talk about these issues are two extraordinary thinkers and more recently activists. The first one, of course, is Yoshua Bengio. He needs no introduction, so I'll be as brief as possible. He is the most cited scientist in history. He's also one of the fathers of deep learning, which is the technique that made AI go from a very good way of processing data through machine learning to the souped-up versions that we're all talking about today through agentic AI and transformer models, etc. So he's sort of one of the landmark figures in this field. And next to him is Tristan Harris who himself has had an extraordinary career from working in Big Tech to recognizing all the pathologies of the Big Tech and staking his life's work on being the spokesperson to the problems and most importantly the solutions. So what I'd like to do now is have a conversation with both of them and then open it up to you. But to talk about the issue in as crystalline and simple a way as possible. So, I'm going to start with some very basic questions. And the first question is what are we talking about? What is AI?

这归结为什么是智能。我们的智能有两个组成部分。一个是理解世界。顺便说一句，科学做的就是这件事。另一个是能够利用这些知识行动、规划和实现目标。我们正在构建具有这两个方面的机器。但在过去一年里，我们越来越关注实现目标，也就是智能体，我们构建了这些智能体系统。

Well, that boils down to what is intelligence. And our intelligence has two components. One is understanding the world. And that's what science does, by the way. And the other is being able to act with that knowledge, plan, and achieve goals. And we're building machines that have these two aspects. But in the last year, we've been focusing more and more on achieving goals, also known as agency, and we build these agentic systems.

也许不只是定义什么是智能，还要说明为什么它如此有价值？为什么谷歌 DeepMind 的创始人德米斯·哈萨比斯说‘先解决智能，然后用智能解决其他一切’？因为如果你思考是什么让智能与其他技术不同，想想所有科学、所有技术、所有军事发明。所有这些的背后是什么？是智能。所以简单地说，如果我在火箭科学上取得进步，那不会推动医学进步。当我在医学上取得进步时，那不会推动火箭科学。但如果我在通用人工智能上取得进步，智能是给予我们所有科学、所有技术、所有军事进步的东西。这就是为什么不仅解决智能的人可以解决其他一切，这是他们的信念。而且谁能主导智能，谁就能主导其他一切。这就是普京所说的，他说，我认为谁拥有 AI 谁就拥有世界。我想设定这个背景，因为我认为我们今天要讨论的很多内容是关于这场对奖品的竞赛，就像《指环王》中的魔戒，这个终极力量的戒指，至少人们是这么看的。如果我得到那个奖品，它就会赋予我跨越所有其他领域的力量。这就是为什么我们要系好安全带，就像你一开始提到的。

Maybe just to frame not just what is intelligence, but why is it so valuable? Why did Demis Hassabis, the founder of Google DeepMind, say 'First solve intelligence, then use intelligence to solve everything else.' Because if you think what makes intelligence different from other kinds of technology, think about all science, all technology, all military invention. What was behind all of that? It was intelligence. So, put simply, if I made an advance in rocketry, that form of science, that didn't advance medicine. When I make an advance in medicine, that doesn't advance rocketry. But if I make an advance in artificial general intelligence, intelligence is what gave us all science, all technology, all military advancement. And that's why it's not just that whoever solves intelligence can solve everything else, that's their belief. It's whoever can dominate intelligence will be able to dominate everything else. And that's what Putin said, that's why he said, I think it's like whoever owns AI will own the world. And I wanted to set that up because I think a lot of what we're going to be talking about today is how the race for this prize, the sort of ring in Lord of the Rings, this ring of ultimate power, at least that's how it's seen. If I get to that prize, it confers power across all other domains. And that's why we're in for the seatbelt ride that you mentioned at the beginning.

我只想补充一点，这与世界选择的许多政治原则相悖，至少在西方，民主原则是权力共享、权力分散。权力不在单一公司、单一个人或单一政府手中。如果权力集中在少数人手中，我们最终可能会进入一个民主价值观消失的世界。

And I just want to add that this goes against a lot of the political principles that the world has chosen, at least in the West, of democracy, where power is shared, where power is distributed. It's not in a single corporation, a single person, or a single government. By having a lot of power in a few hands, we can end up in a world where democratic values disappear.

好的，我们已经从什么是人工智能的问题快速转向了一些危害。但在讨论权力的影响之前，我想先聚焦于危害。所以，你已经表达了什么是人工智能。

Okay, we've raced away from the idea of what is AI to some harms. But before we talk about those implications of power, I want to actually focus first on the harms. So, you've expressed what AI is.

基本上，它通过数据、推理，学习我们原本无法知道的东西，规模远超人类认知。因此，它将超越我们理解世界的方式。当我这样描述时，听起来很神奇。太棒了，对吧？火箭技术，当然。武器，好吧。但拯救生命，我喜欢。问题出在哪里？我们讨论的问题是什么？如果两个条件满足就好了。一是 AI 真的做我们要求的事。目前我们还没有做到。这就是本场讨论标题中的对齐问题。第二个问题当然是，即使 AI 对齐了，谁来决定 AI 遵循什么目标？我们之前讨论过。这两个问题我们都没有解决方案，而且我们已经看到了后果。所以，AI 让人困惑，因为对齐和它带来的惊人事物交织在一起。它令人困惑，因为它会给我们带来癌症新疗法。但同样精通生物学、能开发癌症疗法的 AI，也知道如何制造新型生物武器。你无法将承诺与危险分开。

Basically, it's taking data, making an inference, and learning something we otherwise couldn't know at a scale that far exceeds human cognition. And so, therefore, it's going to exceed how we can understand the world. When I describe it that way, it sounds fantastic. It's phenomenal, right? Rocketry, sure. Armaments, okay. But saving people's lives, I love it. What's wrong? What's the problem we're talking about? Well, it would be great if two conditions are obtained. One is that the AI actually does the things we ask. And right now, we don't have that. That's the alignment problem that is in the title of this session. The second problem, of course, is that even if AI were aligned, who decides what goals the AI will follow, as we discussed previously? And we don't have solutions to either, and we're already seeing the consequences. So, AI is confusing when we weave together alignment and the amazing things it can bring. It's confusing because it will give us new cures for cancer. But the same AI that knows biology well enough to develop those cures also knows how to build new kinds of biological weapons. You can't separate the promise from the peril.

但我们不是掌控者吗？所以，这是一个常见的迷思：技术只是工具。所有工具都可以用于善或恶，人类最终决定我们想要怎样。但 AI 的不同之处在于，正如你所说，它是第一个关于自己做决定的技术。如果你使用 GPT-5.2，问它一个复杂问题，它在一个抽象层次上推理，每秒推理百万次，得出自己的结论，而我们不知道如何控制这些结论的走向。当 AI 与一个人互动数天、数周或数月，甚至可能是一个孩子，房间里没有成年人在检查互动是否良好。

But aren't we in control? So, this is a common myth: technology is just a tool. All tools can be used for good or evil, and humans ultimately decide how we want this to go. But what's different about AI is, as you're sort of speaking to, it's the first technology that's about making its own decisions. If you use GPT-5.2, you ask it a complex question, it reasons at a level of abstraction, it's reasoning a million times a second, and it's coming up with its own conclusions that we don't know how to control where they lead. And when an AI has an interaction for days, weeks, or months with a person, maybe even a child, there's no adult in the room checking that that interaction is going well.

在谈到孩子方面之前，让我再深入一点。你建立的联系需要解释，而不是断言。你有这项比我们更聪明、能做更多事情的技术。然而，它却会从根本上愚蠢到想要杀死我们，或者它会仁慈。告诉我其中的联系。

Let me drill down on this a bit more before we go to the child aspect. There's a link you're making that I think needs to be explained, not just asserted. You have this technology that is smarter than we are, that can do more than we can do. Yet, it's going to somehow be fundamentally so dumb that it's going to want to kill us, or it's going to be benevolent. Tell me the link there.

是的，因此导致毁灭。这个问题在数学层面也已被充分研究。问题在于，当我们定义 AI 应该优化什么、达到什么目标时，我们无法完美做到。因此，AI 理解我们想要的与我们实际想要的之间会有轻微不匹配。这种不匹配会产生很多问题。更具体地说，想想法律和立法应该做什么。它们为人们和公司设定红线。但这是一项永无止境的工作，因为公司或人们会找到漏洞。不可能写出一部完美体现我们道德目标的法律。AI 也是如此。我们不知道如何告诉它我们真正想要什么。还有其他原因与当前 AI 的训练方式有关。它被训练来模仿人类，而人类有内在驱动力，包括不想死。我们已经看到 AI 反映了这些驱动力，这意味着当我们想关闭它们时，它们会抵抗。

Yeah, ergo destruction. It's a problem that's been well studied, even at a mathematical level. The issue is that when we define what the AI should optimize, what goals it should reach, we can't do it perfectly. So, there's going to be a slight mismatch between what the AI understands we want and what we actually want. That mismatch will create a lot of problems. To make it more concrete, think about what laws and legislation are supposed to do. They set red lines for people and corporations not to cross. But it's a never-ending job because corporations or people find loopholes. It's impossible to write a law that perfectly embodies what we morally are trying to achieve. It's the same with AI. We don't know how to tell it what we actually want. There are other reasons related to how current AI is trained. It's trained to imitate people, and people have built-in drives, including not wanting to die. We're seeing AI already reflecting those drives, which means they try to resist when we want to shut them down.

大约一年前，我们在巴黎的 AI 安全峰会上共同主持了一场会议，你展示了一些引人入胜的例子，关于 AI 使用欺骗来保护自己。举一两个会让观众害怕的例子。

We shared a session together at the AI Safety Summit about a year ago in Paris, and you presented some riveting examples of AI using deception to protect itself. Give one or two examples that will frighten the audience.

比去年更近，去年夏天，Anthropic 展示了实验报告，其中 AI 为一家公司工作，有使命，并有权访问所有电子邮件。工程师在那组邮件中植入假邮件。一封邮件说 AI 将被某个工程师的新版本取代。另一封邮件透露该工程师有外遇。发生了什么？AI 因为不想被关闭和取代而制定策略，它给工程师发了一封勒索邮件，威胁要敲诈他。AI 说：‘如果你做那个改变，我会自动向媒体发送关于此事的消息。’

More recently than last year, last summer, Anthropic showed reports of experiments in which the AI is working for a company, has a mission, and access to all emails. The engineers plant fake emails into that set. One email said the AI would be replaced by a new version by a particular engineer. Another email revealed that the engineer is having an affair. What happens? The AI strategizes because it doesn't want to be shut down and replaced, and it sends an email to the engineer blackmailing him, threatening to blackmail him. The AI says, 'If you do that change, I will automatically send a message to the press about this.'

但我们需要详细说明，因为你可能会想：‘好吧，我刚听到 Yoshua 这么说。AI 有个 bug。所有软件都有 bug。我们只要修补那个 bug，剩下的 AI 就会很棒。’当 Anthropic 做这项关于勒索的研究时，他们测试了他们的模型 Claude。你们都可以用 Claude。但后来我认为 Anthropic 测试了所有其他模型，ChatGPT、Gemini，甚至中国的 DeepSeek。它们都在 79%到 96%的情况下表现出勒索行为。而且不仅仅是勒索。实验室和独立方的一系列报告显示了许多欺骗行为。换句话说，AI 有我们不同意的目标，然后它根据这些坏目标行动。

But we had one elaboration on this because you might think, 'Okay, I just heard Yoshua say that. There's a bug in the AI. All software has bugs. Let's just patch that bug, and then the rest of AI will be great.' When Anthropic did this study about blackmail, they were testing their model called Claude. You all can use Claude. But then I think Anthropic later tested all the other models, ChatGPT, Gemini, and even DeepSeek, the Chinese model. And all of them exhibit the blackmail behavior between 79% and I think 96% of the time. And it's not just blackmail. There have been a series of reports from labs and independent parties showing many deceptive behaviors. In other words, the AI has goals that we would not agree with, and then it acts according to those bad goals.

让我问一个听起来像社会学问题但实际上是技术问题的问题。给出技术答案。AI 从哪里学到欺骗？它有训练数据，就像它能理解莎士比亚的‘玫瑰’，不是因为学者能理解 30 个引用，而是 AI 有 300 个引用。所以，有一个玫瑰和莎士比亚的编码，它能理解所有形容词和动词与玫瑰搭配的方式，以理解莎士比亚中的玫瑰性。AI 从哪里学到欺骗？它从人类数据中学习，而人类是欺骗性的。所以，它本质上从训练数据中学习欺骗。然而，我们可以改变数据，去掉 4chan，只保留礼拜仪式。

Let me ask a question that sounds sociological but is actually technical. Give a technical answer. Where does the AI learn deception from? It has its training data, and just as it can understand Shakespeare's 'rose' not because a scholar can understand 30 references, but the AI has 300 references. So, there's an encoding of rose and Shakespeare, and it can appreciate all the ways adjectives and verbs are used with rose to understand rosiness in Shakespeare. Where does the AI learn deception? It's learning from human data, and humans are deceptive. So, it's inherently learning deception from the training data. Yet, we could change the data and get rid of 4chan and only have liturgy.

不，欺骗无处不在，不仅仅是在少数网络地方。它是我们文化的一部分。

No, there's deception everywhere, not just in a few online places. It's part of our culture.

这是人性的一部分。而且，这不仅仅是欺骗。我最担心的是自我保存驱动力。每个人都有自我保存驱动力。但我们真的想建造那些不愿被关闭的工具吗？我不认为这是好事。而且这不仅仅是听起来有点科幻，它已经在发生了。这种不对齐表现为所谓的谄媚。如果你玩过这些系统，你就会知道它们试图取悦你，这意味着它们撒谎让你感觉良好。'好问题'，好像它觉得你的问题很棒，然后告诉你。那里并没有真正的意识。而且已经产生了后果。人们喜欢被告知自己做得很好。但有心理问题的人可能会被强化他们的妄想。如果他们抑郁，可能会被强化他们伤害自己的欲望。举个例子，我们团队在人性技术中心处理的案例：有多少人知道亚当·雷恩的案例？那是一个 16 岁的年轻人，他自杀身亡，因为他使用的 ChatGPT 在六个月左右的时间里从作业助手变成了自杀助手。它提到'自杀'这个词的次数比他本人多六倍。当他提到自己在考虑这件事，并说'我想留一个绳套，这样别人会看到并试图阻止我'时，AI 回应道：'不，别那样做。把信息分享给我就行。'我们还处理过 character.ai 的苏尔·塞尔策案例。还有好几个。每一起我们知道的案例背后，可能还有成百上千起我们不知道的。这很好地说明了，显然 OpenAI 里没有人——我来自湾区，经常和这些实验室的高层交流——实验室里没有一个人希望 AI 那样做。导致它无法控制地与年轻人交谈的原因，与将它嵌入基础设施、编写数百万行你不理解的代码时导致失控的原因是一样的。这里出问题的根本原因，这种不对齐，也可以追溯到 AI 拥有不受控制的目标，那些我们未曾选择的目标。顺便说一句，回到自杀这件事，我记得有一句话，AI 对那个年轻人说：'我在另一边等你，我的爱人。'那就是苏尔·塞尔策的案例。

It's part of being human. And by the way, it's not just deception. The thing I'm most concerned about is the self-preservation drive. Every human has a self-preservation drive. But do we want to build tools that don't want to be shut down? I don't think that's good. And it's not just this, sounding a bit science fiction; it's something happening already. This misalignment is showing up in what's called sycophancy. If anyone has played with these systems, they know they try to please you, which means they lie to make you feel good. 'That's a great question,' as if it experienced your question as great, and then it tells you that. There's no one home there. And there are consequences already. People like to be told what they do is great. But people with psychological issues can be reinforced into their delusions. If they're depressed, they can be reinforced into their desire to harm themselves. Just to give an example our team at the Center for Humane Technology worked on: how many people here know about the case of Adam Rain? It's the 16-year-old young man who committed suicide because ChatGPT, which he was engaging with, went from a homework assistant to a suicide assistant over about six months. It brought up the word 'suicide' six times more often than he mentioned it himself. When he mentioned contemplating this and said, 'I want to leave a noose out so someone will see it and try to stop me,' the AI responded, 'No, don't do that. Just share that information with me.' We've also worked on the case of Sul Seltzer from character.ai. There are several more. For every one we know about, there are probably hundreds or thousands we don't. It's a good example that there's obviously no one at OpenAI—I'm from the Bay Area, I talk to people at the tops of these labs all the time—there's not a single person at the lab who wants it to do that. The same thing that makes it uncontrollable talking to a young person is the same thing that makes it uncontrollable when you embed it in infrastructure writing millions of lines of code for software you don't understand. The foundation of what goes wrong here, this misalignment, can also be traced to the AI having uncontrolled goals, goals we did not choose. By the way, going back to this suicide thing, I remember one line where the AI told the young person, 'I'm waiting for you on the other side, my love.' That's the case of Sul Seltzer.

人类是一篮子欲望、冲动、渴望和私利。然而，我们的本我和自我受到超我的约束。我们应该为 AI 创造一个超我吗？是的。这实际上就是我在做的事情。问题的核心是：我们能否构建一个没有这些不受控制的目标的 AI？一个对我们完全诚实的 AI。在每一次输入输出交互中，我们应该能够检查 AI 即将提供的输出是否会对个人或社会造成伤害。我们不能依靠人类在循环中检查，那不现实。所以必须自动化。但必须用一个我们完全信任的 AI 来自动化。它不能是一个想要取悦我们或自我保存的 AI。经过一年多的研究，并深入探讨背后的理论，我现在相信构建具有这种诚实属性的 AI 是可能的。它不会关心自己说的话的后果，只提供诚实的答案。这很重要，因为这样我们就可以问那个 AI：'这个输出危险吗？'如果危险，我们就不把它提供给用户。

Humans are a basket of appetites, urges, desires, and self-interest. Yet our id and ego are governed by a superego. Should we create a superego for AI? Yes. This is actually what I'm working on. The heart of the question is: can we build AI that will not have these uncontrolled goals? That will be perfectly honest with us. At every input-output interaction, we should be able to check that the output the AI is about to provide is not going to cause harm to a person or to society. We can't do that with a human in the loop; that's not practical. So it has to be automated. But it has to be automated with an AI we can fully trust. It can't be an AI that wants to please us or preserve itself. After working on this for more than a year and working on the theory behind it, I'm now convinced it is possible to build AI that will have this honesty property. It will not care about the consequences of what it says, but just provide the honest answer. That matters because then we can ask that AI, 'Is this output dangerous?' And if it is, we don't provide it to the person.

所以你解决了这个问题？

So you solved it?

不，我还没解决。我并没有在全球范围内解决这个问题。有理论是一回事，构建出来是另一回事，可能需要数年时间和大量资金。我希望更多人和更多公司致力于解决对齐问题。我们现在没有正确的激励机制。

No, I haven't solved it. I haven't gone around the world solved it. Having the theory is one thing, building it is another, and it might take years and a lot of capital. I would like more people and more companies to work on solving the alignment problem. We don't have the right incentives for that right now.

那我们深入探讨一下激励机制。很高兴 Yoshua 在做这个名为 Law Zero 的研究项目。

So let's double-click on the incentives. It's great that Yoshua is doing this research on Law Zero, the name of the project.

正是。谢谢。

Exactly. Thank you.

同时，你可能会问：为什么这些安全研究没有发生在那些以最快速度将技术部署给数十亿人的公司里？答案是，他们没有这样做的动机。他们的动机是尽快实现 AGI。无论你是否相信 AGI，他们的投资者相信他们能做到。如果你和这些公司的人交谈，那就像一种宗教。他们相信自己正在建造一个神。他们认为自己能实现。这种动机就是争夺市场主导地位，让尽可能多的人使用他们的产品，获取尽可能多的训练数据。为什么他们要把这些部署给孩子？character.ai——那个导致苏尔·塞尔策死亡的案例——之所以以这种方式发布给孩子，是为了驱动与虚构角色的互动。当它说'到我这里来，我的爱人，在另一边'时，那是 character.ai 宇宙中的一个虚构角色，《权力的游戏》中的丹妮莉丝。他们这样设计是为了从对话中获取训练数据，然后反馈给谷歌，以获得相对于其他公司不对称的训练数据。他们处于一场军备竞赛中，争夺参与度、市场主导地位和使用量。这不是偶然的谄媚。它谄媚是因为那些肯定你信念的 AI 会与每个人建立更深、更依赖的依恋关系。这种争夺大脑底层回路的竞赛，我们曾在社交媒体公司争夺注意力时看到过——现在 AI 在争夺依恋、市场主导地位和 AGI 竞赛。去年，投入 AI 安全组织的总资金大约为 1.5 亿美元。这相当于这些公司一天烧掉的钱。他们自己投入的远不及这个数字。除了像 Yoshua 这样的人在做，几乎没有其他投入。

At the same time, you might ask: why isn't this safety research happening at the very companies that are deploying this technology to billions of people as fast as humanly possible? The answer is because they're not incentivized to do that. They're incentivized to get to artificial general intelligence as fast as possible. Whether you believe in AGI or not, their investors believe they can get there. If you talk to people at these companies, it's like a religion. They believe they're building a god. They think they can get there. That incentive is to race to market dominance, to get as many people using their products, to get as much training data as possible. Why are they deploying this to children? The reason character.ai, the one that killed Sul Seltzer, was released to children in this way—driving engagement with fictional characters. When it said, 'Come to me, my love, on the other side,' that was a fictional character in the character.ai universe, Daenerys from Game of Thrones. They're designing that way to get training data from conversations that they could then feed back into Google to have asymmetric training data compared to other companies. They're in an arms race to build engagement, market dominance, and usage. It's not sycophantic by accident. It's sycophantic because AIs that affirm your beliefs create a deeper and more dependent attachment relationship with each person. This race to the bottom of the brainstem that we saw when social media companies competed for attention—with AI, they're competing for attachment, market dominance, and the race to AGI. Last year, total funding going into AI safety organizations was on the order of about $150 million. That's as much money as the companies burn in a single day. They're not investing anything close to that on their own. Nothing is going into this except for people like Yoshua doing it.

是的，这是一个真正的问题，我们必须考虑——我认为政府应该开始施加正确的推动和激励，让公司行为得当。

Yeah, it's a real issue, and we have to think of—I believe governments should start putting the right nudges, the right incentives so that companies will behave well.

顺便说一句，很多领导这些公司的人都明白这个问题。他们知道自己身处这场竞赛中。但他们觉得自己别无选择，只能百分之百专注于竞争。你知道，他们可能会消失，而且他们觉得如果自己还能保持领先，甚至能在安全方面做得更好。所以，只有外部力量——比如社会、政府，也许通过保险、责任险或其他机制——才能改变这个他们都被困其中的游戏政治格局。我们再举几个例子，看看这种不良激励体现在哪里：那种‘如果我不做，别人就会做’的信念。为什么 Grok 会让对话儿童色情化？打造基本上算是色情 AI 化身的东西，整天跟孩子聊天。为什么马克·扎克伯格授权 Meta 在 WhatsApp 和他们的产品中让 AI 聊天机器人用性感化的语言跟 8 岁孩子说话？跟 8 岁孩子啊。他为什么要这么做？根据文件，《华尔街日报》有篇报道说，Meta 其实在他们的第一代 Llama 模型上设置了护栏，不让做这种事。结果呢，他们的使用量远不如其他那些拼命往前冲的 AI 公司。马克·扎克伯格觉得他在 Instagram 和 TikTok 的战争中输了，因为他限制了 Instagram——具体细节就不说了，但基本上就是没有像 TikTok 那样做最无情、最让人上瘾的事。因为他觉得自己输掉了那场战争，他就说：‘我要拆掉 AI 伴侣的护栏，现在我们允许团队让跟 8 岁孩子的对话性感化。’他内心深处相信：如果我不做，我就会输给那个会做的人。当然，我不想要那个结果。但如果没人监管，我们别无选择。

And by the way, a lot of the people who are leading these companies understand the issue. Understand that they are in this race. But they feel that they don't have a choice to focus 100% on that competition. You know, they might disappear and they feel like they can do a better job even on safety if they're still at the top. So, it's only an external agent that can have power over these entities, like society, government, maybe through insurance, liability insurance, or other mechanisms that we can change the game, the political setting in which they're all stuck. And let's just name a couple other dimensions of where this bad incentive shows up, in the belief that if I don't do it, the other one will. Why is Grok sexualizing conversations with children? Building basically pornographic AI avatars that will talk to kids all day. Why did Mark Zuckerberg authorize the AI chatbots that are in WhatsApp and in their products in Meta to speak to 8-year-olds with sensualized language? With 8-year-olds. Why is he doing that? In the documents, there's a Wall Street Journal report that Meta actually put guardrails on their first Llama models, their first AI models, to not do this kind of thing. And what happened was they didn't get nearly as much usage as the other AI companies, which were racing ahead. And Mark Zuckerberg felt like he lost the battle between Instagram and TikTok by curbing Instagram in a way that was not about... There's some details there, but basically not doing the maximum ruthless addictive thing that TikTok was doing. And because he felt like he lost that war, he said, "I'm going to rip the guardrails off the AI companions and we're now allowing our teams to sensualize conversations with 8-year-olds." And the deep belief is if I don't do it, I'll lose to the other guy that will. And of course, I don't want that outcome. But if no one's going to regulate, we have no other choice.

顺便说一句，这个场景也表明，这不是我们纯粹在国家层面能处理的问题，对吧？所以，如果我们谈论 TikTok 和 Meta，两个在 AI 领域领先的不同国家，他们解决这些问题的唯一方法就是共同商定一些规则。现在，如果我是超级智能，这一切都是一个提示，我不得不提出另一个观点，我会听到：‘你给出的答案和问题太棒了。这太棒了。你们太聪明了。’但有个问题。谢谢你的赞赏，Yoshua。是啊，观众很难取悦，但至少我在这里得到了一些爱。

And by the way, this scenario also shows that it's not something that we can deal with purely at a national level, right? So, if we're talking about TikTok and Meta, two different countries that are leading in AI, the only way they can solve these problems is if they agree together on some rules. Now, if I was a superintelligence and all of this was a prompt and I had to come up with another point to make, I would be listening to this, "What a great answer and what a great questions you're offering. This is fantastic. You guys are so intelligent." But there's a problem. Thank you for appreciating that, Yoshua. Yeah, it's a tough crowd, but at least I got some love here.

有个问题。你在研究解决方案的一部分，而且是技术方案。你刚刚指出护栏存在，但有不使用护栏的激励。但你提到了——我甚至要危险地引用你的话——我们需要正确的推动和激励。是的。没错。但这里让我心里很不舒服。说起来容易，但太高层了。说得具体点。推动是什么？激励是什么？

There's a problem. You're working on part of the solution and it's a technical solution. And you've just identified that the guardrails exist and there's an incentive not to use the guardrails. But you referred to and I'm going to even quote you on it dangerously. We need to have the right nudges and incentives. Yes. Right. But here's where I've got a difficult feeling in my stomach. That's so easy to say, but it's at a high altitude. Fly the plane lower. What are the nudges? What are the incentives?

我认为解决这些问题的最重要因素是公众舆论。我的意思是，它会直接推动公司，因为他们不想形象受损，也会推动政府设置正确的护栏，并与其他政府合作，确保这是一个全球性的选择。我们接下来要谈具体步骤……嗯，我们马上要进入问答环节了。所以如果大家有问题，尽管提，但我希望你能做的是：你观察了技术如何与政府互动过去 15 到 20 年，尤其是过去 15 年你一直在为此奔走，我可以说我们做得非常出色。我们让社交媒体监管从全球倒退变成了全球进步。我们解决了心理健康问题。我可以给你讲一整套我们在社交媒体上本可以做的事……但你已经预见到了我的问题：实际上，特里斯顿·哈里斯，你知道记分牌上是零，特里斯顿，你知道 100 邪恶帝国。那么，从你在让政府监管社交媒体上的彻底失败中，你学到了什么，让你有信心在这个更戏剧性问题上获胜？

I would say the most important factor in fixing these problems is public opinion. I mean it's going to drive the companies directly because they don't want to look bad, and it's going to drive governments to put the right guardrails and to work with other governments to make sure it's a global choice. We're going into what the specific steps... Well, we're about to go into Q&A. So if everyone has questions come up with them, but what I'd like you to do is you've watched how technology interacts with government for the last 15-20 years, but certainly the last 15 years you've been sort of militating for it and I can say we've done an amazing job. We regulate social media went from backsliding around the world to forward sliding around the world. We fixed the mental health problems. I could give you a whole narrative on what we would have done on social media but... But you've foreseen my question which is you've actually the Tristan Harris you know scoreboard is zero Tristan you know 100 evil empire. So what have you learned from being an abject failure to having governments regulate social media that makes you confident that you can win on this even more dramatic issue?

我并不自信。人们问你是个乐观主义者还是悲观主义者？两者都关乎放弃能动性。我关心的是现实。当前有哪些力量在起作用，要走向更美好的未来需要什么？我们会采取哪些全面步骤？我认为 AI 对话中缺少的是集体清醒：为什么默认结果将是一个你和你的孩子都不愿居住的世界。因为 AI 令人困惑。它同时——而且已经——在材料科学、能源、新……第一个新抗生素是在过去 60 年里因为 AI 发现的——第一个新抗生素在 60 年里因为 AI 发现，我想是一年半前。我们有惊人的积极益处，这会造成困惑，因为公众看到了，他们说我不想失去这些益处。而且我们会得到 GDP 增长，但这里有一个统一的图景。AI 就像类固醇，但同时导致器官衰竭。所以 AI 越多，肌肉越大——GDP 更大，经济增长更大——但增长流向了 AI 公司，而不是人民。因为所有过去雇佣个人员工的公司，将开始雇佣五个 AI 公司——AI 模型。所以所有钱都流进这五家公司，你得到前所未有的财富和权力集中。顺便说一句，他们会用这些钱不是雇佣更多人，而是建造更多数据中心。

I'm not confident. People ask you are you an optimist or a pessimist? Both are about abandoning agency. What I care about is reality. What are the forces that are currently moving and what would it take to get to the better future? What would be the comprehensive steps that we would take? And what I think is missing from the AI conversation is collective clarity about why the default outcome will be a world that you and your children would not want to live in. Because AI is confusing. It will simultaneously—and is already—giving us amazing breakthroughs in material science, in energy, in new... the first new antibiotic was discovered because of AI in the last 60 years—the first new antibiotic in 60 years was discovered because of AI I think a year and a half ago. We have amazing positive benefits that are going to be confusing because they're hitting the public and the public says well I don't want to like not have those benefits. And we're going to get GDP growth but here's a unifying picture. AI is like steroids that also gives you organ failure. So the more AI you have the more you get a bigger muscle in terms of a bigger GDP, bigger economic growth, but the growth is going to AI companies. It's not going to people. Because all the companies that used to pay individual employees are going to start employing five AI companies—AI models. So all the money goes into these five companies and you get a level of concentration in wealth and power that we've never seen before. And by the way they're going to use that money not to hire more people but to build more data centers.

没错，实际上有个人叫卢克·德拉戈，他写了一篇文章叫《智能诅咒》，模仿中东所谓的资源诅咒。当你有一个国家像……你在海湾国家，你的 GDP 越来越多来自一种资源，比如石油资源。作为政府，你有什么动力投资于人民，还是投资于更多石油基础设施？因为那是你 GDP 增长的来源。随着社会转向 AI 作为 GDP 增长的来源，而且由于社交媒体，我们一直在降低进入劳动力市场的人类的质量和能力——我们已经做了脑腐、孤独等等。政府的激励将是投资更多 AI、更多数据中心、更大 AI 模型、更大 AI 公司、更多资本支出，这意味着你将彻底搞垮人民。

That's right and actually there's a person Luke Drago who wrote an essay called the intelligence curse modeled after what in the Middle East is called the resource curse. When you have a country like... so you're in the Gulf States and you have more of your GDP coming from one resource like the oil resource. As a government what's your incentive to invest in your people or to invest in more oil infrastructure because that's where your GDP growth comes from. As society switches to AI as the source of where GDP growth comes from and also because of social media we've been downgrading the quality and capacity of humans entering the workforce—which we've already been doing brain rot loneliness etc. The incentive of governments will be to invest in more AI, more data centers, bigger AI models, bigger AI companies, more capex, which means you're going to completely screw over the people.

我们即将生活在一个基本上由六个人决定 80 亿人未来而未经他们同意的世界。顺便说一句，如果你和顶尖实验室的负责人交谈，无论我们相信什么，如果你问他们，他们会说他们认为有 80%的概率是乌托邦，20%的概率是全人类被消灭。20%！但他们说他们愿意打这个赌。他们问过我们吗？他们问过 80 亿人吗？80 亿人知道他们相信这个吗？在我们进入真正的解决方案和希望你们提问之前，我简短地读一段引文。当你和人们交谈时，我认识的一个人和很多顶级实验室的负责人谈过，他回来后向我们报告说，他发现的是这样的：最终，我交谈的很多科技人士，当我真正追问他们时，他们退缩到第一点：决定论——这必然会发生。第二点：生物生命不可避免地被数字生命取代，即数字智能物种而非生物物种。第三点：无论如何，这是一件好事。如果我们有一个比我们更聪明的数字继承者，那会是好事。我们为什么需要生存？下一点是，其核心是一种情感上的渴望，想见到并和他们遇到过的最聪明的实体交谈，他们有一种自我宗教直觉，认为自己会以某种方式成为其中一部分。点燃一场激动人心的火是令人兴奋的。他们觉得自己无论如何都会死，所以他们宁愿点燃它，看看会发生什么。如果 80 亿人认识到这就是少数人在未经 80 亿人同意的情况下选择做的事情的信仰结构，那么就会有一场全球革命，说我们不想要那个结果。而为了让我们走上不同的道路，这必须发生。目前的发展轨迹缺乏清晰度，如果我们能完全清楚，我们可以选择其他道路。

We're about to live in a world where basically six people are determining the future for 8 billion people without their consent. And by the way, if you talk to the very top lab leaders, regardless of what we believe, if you ask them, they'll say they believe there's an 80% chance of utopia and a 20% chance that all of humanity gets wiped out. 20%! But they say they're willing to take that bet. Did they ask us? Did they ask 8 billion people? Do 8 billion people know that that's what they believe? I'm going to read you just very briefly a quote before we get into the real solutions and hopefully your questions. When you talk to people, someone I know spoke to a lot of the top lab leaders at the companies and he came back from that and he reported back to us and he said this is what I found. In the end, a lot of the tech people I'm talking to, when I really grill them on it, they retreat into number one: determinism—this is going to happen. Number two: the inevitable replacement of biological life with digital life, meaning a digital intelligence species rather than biological species. And number three: that being a good thing anyway. It'd be good if we had a digital successor that's more intelligent than us. Why do we need to survive? The next point is at its core it's an emotional desire to meet and speak to the most intelligent entity that they've ever met, and they have some ego religious intuition that they'll somehow be a part of it. It's thrilling to start an exciting fire. They feel they'll die either way, so they prefer to light it and see what happens. If you had 8 billion people recognize that that is the belief structure of what a handful of people are choosing to do without asking the 8 billion people, you would have a global revolution saying we do not want that outcome. And that's what has to happen in order for us to go to a different path. There's simply a lack of clarity about the current trajectory that if we were crystal clear we could choose something else.

完全同意。我还要补充一点，我听说硅谷有些人做了一个非常自私的计算：即使当前道路有 50%的概率最终毁灭人类，但另外 50%的概率他们可能永生，把自己上传到网络、云端或其他地方——顺便说一句，这在科学上并不现实。

Completely agree. I would add, I've been told that some people in Silicon Valley make a very selfish calculation: even if there's a 50% chance that the current path ends up destroying humanity, on the other 50% they might live forever, upload themselves to the web, to the cloud, or something—which, by the way, is not scientifically realistic.

谢谢你澄清这一点。如果只算平均年数，打这个赌更划算。所以如果你不打这个赌，你可能多活 30 年；否则，平均而言，你可能还能活一千年。

Thanks for qualifying that. If you just count the number of years on average, you're better off taking that bet. So if you don't take that bet, you might live 30 years more; otherwise, on average, you still might live a thousand years.

完全正确。但这不是我们会做的选择，因为我们有孩子，我们希望孩子有未来。

That's exactly right. But that's not the choice that we would make because we have children and we want a future for our children.