「我们不知道模型是否有意识」

我想试着聚焦于 AI 失控的场景。我觉得互联网已经替我们做了这件事。人工智能的主宰者站在人类这边吗？我的预测是，机器人会比人还多。物理世界和数字世界应该完全融合。我认为世界还没有迎来人形机器人的时刻。那会非常科幻。这是我本周嘉宾的核心问题。他是 Anthropic 的负责人，Anthropic 是增长最快的 AI 公司之一，估值接近 3500 亿美元。Anthropic 的 Claude Code 接连获胜。对于他正在释放的技术可能带来的影响，他某种程度上是个乌托邦主义者。它会帮助我们治愈癌症，可能帮助我们根除热带疾病，会帮助我们理解宇宙。但他也看到了前方的严重危险和无论如何都会发生的巨大颠覆。这一切发生得如此之快，是一场危机，我们应该投入几乎全部精力思考如何渡过难关。Dario Amodei，欢迎来到 Interesting Times。

I want to try and focus on scenarios where AI goes rogue. I think the internet does that for us. Are the lords of artificial intelligence on the side of the human race? My prediction is that there will be more robots than people. The physical and the digital world should really be fully blended. I think the world hasn't had a humanoid robot moment yet. It's going to feel very sci-fi. That's the core question I had for this week's guest. He's the head of Anthropic, one of the fastest growing AI companies. Anthropic is estimated to be worth nearly $350 billion. It's been win after win for Anthropic's Claude code. He's a utopian of sorts when it comes to the potential effects of the technology that he's unleashing on the world. It will help us cure cancer, it may help us to eradicate tropical diseases, it will help us understand the universe. But he also sees grave dangers ahead and massive disruption no matter what. This is happening so fast and is such a crisis, we should be devoting almost all of our effort to thinking about how to get through this. Dario Amodei, welcome to Interesting Times.

谢谢你邀请我，Russ。

Thank you for having me, Russ.

你相当不寻常，也许对一位科技 CEO 来说，你是一位散文家。你写了两篇长而有趣的文章，关于人工智能的承诺与危险。我们这次会谈会讨论危险，但我想先从承诺开始，从你几年前在一篇题为《Machines of Loving Grace》的文章中提出的乐观愿景开始，我们最后会回到那个标题。我认为很多人通过头条新闻了解 AI，预测白领工作的大屠杀等等。有时你自己的引述也助长了这些。我认为人们对 AI 的用途有一种常识。那么，你为什么不先回答这个问题呢？如果未来 5 到 10 年一切顺利，AI 是用来做什么的？

So, you are rather unusually, maybe for a tech CEO, an essayist. You have written two long, very interesting essays about the promise and the peril of artificial intelligence. And we're going to talk about the perils in this conversation, but I thought it would be good to start with the promise and with the optimistic vision, indeed, I would say the utopian vision that you laid out a couple of years ago in an essay entitled Machines of Loving Grace, which we'll come back to that title, I think, at the end. But I think a lot of people encounter AI news through headlines predicting a bloodbath for white-collar jobs, these kinds of things. Sometimes your own quotes have encouraged these things. And I think there's a common sense of what AI is for that people have. So, why don't you answer that question to start out? If everything goes amazingly in the next 5 or 10 years, what's AI for?

是的，简单介绍一下背景，在我从事 AI 工作之前，在我进入科技行业之前，我是一名生物学家。我最初研究计算神经科学，然后在斯坦福医学院工作，寻找癌症的蛋白质生物标志物，试图改进诊断和治愈癌症。我在那个领域工作时最大的感受是它的惊人复杂性。每个蛋白质在每个细胞内都有特定的水平。仅仅测量体内的水平或每个细胞内的水平是不够的。你必须测量细胞特定部分的水平以及它相互作用、形成复合物的其他蛋白质。我有种感觉：天哪，这对人类来说太复杂了。我们在生物学和医学的所有问题上都在取得进展，但进展相对缓慢。所以吸引我进入 AI 领域的是这个想法：我们能否更快地取得进展？长期以来，我们一直试图将 AI 和机器学习技术应用于生物学，通常用于分析数据。但随着 AI 变得非常强大，我认为我们应该以不同的方式思考。我们应该把 AI 看作是在做生物学家的工作，从头到尾完成整个过程。这包括提出实验、发明新技术。我在文章中有这么一段：'生物学中的许多进展是由相对少量的洞察驱动的，这些洞察让我们能够测量、接触或干预那些非常微小的东西。'很多这类技术的发明都非常偶然。CRISPR，也就是基因编辑技术之一，是因为有人去听了一场关于细菌免疫系统的讲座，并将其与他们正在做的基因治疗工作联系起来而发明的。那个联系本可以在 30 年前就建立。所以想法是：AI 能否加速这一切？我们真的能治愈癌症吗？我们真的能治愈阿尔茨海默病吗？我们真的能治愈心脏病吗？更微妙的是，一些更心理上的痛苦，比如抑郁症、双相情感障碍，我们能否在它们基于生物学的程度上做些什么？我认为它们至少部分基于生物学。所以我论证了：如果我们拥有这些几乎能做任何事的智能，进展能有多快？

Yeah, so for a little background, before I worked in AI, before I worked in tech at all, I was a biologist. I first worked on computational neuroscience and then I worked at Stanford Medical School on finding protein biomarkers for cancer, trying to improve diagnostics and curing cancer. One of the observations I most had when I worked in that field was the incredible complexity of it. Each protein has a level localized within each cell. It's not enough to measure the level within the body, the level within each cell. You have to measure the level in a particular part of the cell and the other proteins that it's interacting with, complexing with. I had this sense of man, this is too complicated for humans. We're making progress on all these problems of biology and medicine, but we're making progress relatively slowly. So what drew me to the field of AI was this idea: could we make progress more quickly? We've been trying to apply AI and machine learning techniques to biology for a long time, typically for analyzing data. But as AI gets really powerful, I think we should think about it differently. We should think of AI as doing the job of the biologist, doing the whole thing from end to end. Part of that involves proposing experiments, coming up with new techniques. I have this section where I say, 'Look, a lot of the progress in biology has been driven by a relatively small number of insights that let us measure or get at or intervene in the stuff that's really small.' A lot of these techniques are invented very much as a matter of serendipity. CRISPR, which is one of these gene editing technologies, was invented because someone went to a lecture on the bacterial immune system and connected that to the work they were doing on gene therapy. That connection could have been made 30 years ago. So the thought is, could AI accelerate all of this? Could we really cure cancer? Could we really cure Alzheimer's disease? Could we really cure heart disease? And more subtly, some of the more psychological afflictions that people have, depression, bipolar, could we do something about these to the extent that they're biologically based, which I think they are at least in part. So I go through this argument: how fast could it go if we have these intelligences out there who could do just about anything?

我想在这里打断你，因为你在那篇文章中的框架，以及你后来似乎又回到这一点，有趣之处在于这些智能不必是 AI 辩论中那种最大化的神级超级智能。你基本上是说，如果我们能实现达到人类巅峰表现水平的强智能。

I want to pause you there because one of the interesting things about your framing in that essay, and you've sort of returned to it, is that these intelligences don't have to be the maximal god-like superintelligence that comes up in AI debates. You're basically saying if we can achieve a strong intelligence at the level of peak human performance.

人类巅峰表现，是的。

Peak human performance, yes.

乘以它，用你的话说，一个天才之国。

Multiply it, to use your phrase, a country of geniuses.

有一亿个这样的智能。也许每个都经过稍微不同的训练，或者尝试不同的问题。多样化和以不同方式尝试是有好处的。但没错。你不需要完整的机器神。而且确实有些地方我质疑机器神在这些事情上是否比一亿个天才更有效。我有一个概念叫智能的边际收益递减。经济学家谈论土地和劳动的边际生产力。我们从未考虑过智能的边际生产力，但当我观察生物学中的一些问题时，在某种程度上你只需要与世界互动。在某种程度上你只需要尝试。在某种程度上你只需要遵守法律或改变法律，让药物通过监管系统。所以这些变化发生的速率是有限的。当然，有些领域比如下棋或围棋，智能的天花板极高。

Have a hundred million of them. Maybe each a little trained a little different or trying a different problem. There's benefit in diversification and trying things a little differently. But yes. You don't have to have the full machine god. And indeed there are places where I cast doubt on whether the machine god would be that much more effective at these things than the hundred million geniuses. I have this concept called the diminishing returns to intelligence. Economists talk about the marginal productivity of land and labor. We've never thought about the marginal productivity of intelligence, but if I look at some of these problems in biology, at some level you just have to interact with the world. At some level you just have to try things. At some level you just have to comply with the laws or change the laws on getting medicines through the regulatory system. So there's a finite rate at which these changes can happen. Now, there are some domains like playing chess or Go where the intelligence ceiling is extremely high.

但我认为现实世界有很多限制因素。所以，也许你可以超越天才水平，但有时我觉得所有这些关于用一个月亮大小的算力来制造一个 AI 神的讨论，有点耸人听闻，而且偏离了重点。尽管我认为这将是人类历史上最重大的事件。所以，具体来说，你有一个世界，癌症作为对人类生命的严重威胁终结了，心脏病终结了，我们经历并导致死亡的大多数疾病都终结了。可能还有寿命延长。所以，那是健康。这是一个相当积极的愿景。然后谈谈经济和财富。在 5 到 10 年的 AI 起飞期间，财富会发生什么？

But I think the real world has a lot of limiters. So, maybe you can go above the genius level, but sometimes I think all this discussion of using a moon of computation to make an AI god is a little bit sensationalistic and besides the point. Even as I think this will be the biggest thing that ever happened to humanity. So, keeping it concrete, you have a world where there's an end to cancer as a serious threat to human life, an end to heart disease, an end to most of the illnesses that we experience and kill us. Possible life extension beyond that. So, that's health. That's a pretty positive vision. Then talk about economics and wealth. What happens in the 5 to 10-year AI takeoff to wealth?

所以，再次强调，我们保持积极的一面，因为会有很多时间讨论消极的一面。但我们已经在与制药公司、金融行业公司、制造业人士合作。我们尤其以编码和软件工程闻名。所以，仅仅是原始生产力，即制造东西和完成事情的能力，就非常强大。我们看到我们公司的收入每年增长 10 倍。我们怀疑整个行业的情况也类似。如果技术持续改进，不需要太多 10 倍的增长，直到突然你说，「哦，如果你在整个行业每年增加一万亿美元的收入，美国 GDP 是 20 或 30 万亿。所以你肯定将 GDP 增长提高了几个百分点。」所以，我可以看到一个世界，AI 将发达国家的 GDP 增长提高到 10%或 15%左右。计算这些数字没有科学依据。这完全是前所未有的事情。但它可能将数字带到我们以前见过的分布之外。再次，我认为这将导致一个奇怪的世界。我们有所有这些关于赤字增长的辩论。如果你有那么多 GDP 增长，你就会有那么多税收收入，你会无意中平衡预算。我最近一直在思考的一件事是，我们经济和政治辩论的一个假设是增长很难实现。它是一个独角兽。有各种方法可以杀死下金蛋的鹅。我们可能进入一个增长非常容易的世界，而分配很难，因为它发生得太快。馅饼增长得太快了。

So, again, let's keep it on the positive side because there will be plenty of time for the negative side. But we're already working with pharma companies, financial industry companies, manufacturing folks. We're especially known for coding and software engineering. So, just the raw productivity, the ability to make stuff and get stuff done, is very powerful. We see our company's revenue growing up 10x a year. We suspect the wider industry looks similar. If the technology keeps improving, it doesn't take that many more 10x's until suddenly you're saying, 'Oh, if you're adding across the industry a trillion dollars of revenue a year, the US GDP is 20 or 30 trillion. So you must be increasing GDP growth by a few percent.' So, I can see a world where AI brings the developed world GDP growth to something like 10 or 15%. There's no science of calculating these numbers. It's a totally unprecedented thing. But it could bring it to numbers outside the distribution of what we saw before. Again, I think this will lead to a weird world. We have all these debates about the deficit growing. If you have that much GDP growth, you're going to have that much in tax receipts, and you're going to balance the budget without meaning to. One of the things I've been thinking about lately is that one of the assumptions of our economic and political debates is that growth is hard to achieve. It's a unicorn. There are all kinds of ways you can kill the golden goose. We could enter a world where growth is really easy, and it's the distribution that's hard because it's happening so fast. The pie is being increased so fast.

所以，在我们进入难题之前，我认为还有一个关于政治的乐观说明。这里更有点推测性。你试图论证 AI 可能对世界各地的民主和自由有好处，这并不直观。很多人说，极其强大的技术掌握在威权领导人手中会导致权力集中等等。简单来说，为什么 AI 对民主有利的乐观理由是什么？

So, before we get to the hard problem, one more note of optimism on politics, I think. And here it's a little more speculative. You try and make the case that AI could be good for democracy and liberty around the world, which is not necessarily intuitive. A lot of people say incredibly powerful technology in the hands of authoritarian leaders leads to concentrations of power and so on. Just briefly, what is the optimistic case for why AI is good for democracy?

是的，绝对。在《优雅的机器》中，我有点像，让我们做梦吧。让我们谈谈它如何能顺利进行。我不知道可能性有多大，但我们必须描绘一个梦想。让我们努力让梦想成真。所以，我认为积极的版本，我承认我不知道技术本身是否倾向于自由。我认为它本身倾向于治愈疾病和经济增长，但我像你一样担心它可能并不天然倾向于自由。但我在那里说的是：我们能让它倾向于自由吗？我们能确保美国和其他民主国家在这项技术上领先吗？美国在技术和军事上一直领先，这意味着我们通过与其它民主国家的联盟在世界范围内拥有影响力，并且我们能够塑造一个我认为比由俄罗斯、中国或其他威权国家塑造的世界更好的世界。那么，我们能利用我们在 AI 方面的领先地位来塑造世界各地的自由吗？显然有很多关于我们应该如何干预、如何运用这种力量的辩论。但我经常担心，今天通过社交媒体，威权者正在削弱我们。我们能反击吗？我们能赢得信息战吗？我们能通过用 AI 的力量保卫乌克兰或台湾这样的国家来阻止威权者入侵它们吗？

Yeah, absolutely. In 'Machines of Loving Grace' I kind of like, let's dream. Let's talk about how it could go well. I don't know how likely it is, but we got to lay out a dream. Let's try to make the dream happen. So, I think the positive version, I admit that I don't know that the technology inherently favors liberty. I think it inherently favors curing disease and economic growth, but I worry like you that it may not inherently favor liberty. But what I say there is: can we make it favor liberty? Can we make the United States and other democracies get ahead in this technology? The United States has been technologically and militarily ahead, which has meant that we have throw weight around the world through our alliances with other democracies, and we've been able to shape a world that I think is better than the world would be if it were shaped by Russia or China or other authoritarian countries. So, can we use our lead in AI to shape liberty around the world? There's obviously a lot of debates about how interventionist we should be, how we should wield that power. But I've often worried that today through social media, authoritarians are kind of undermining us. Can we counter that? Can we win the information war? Can we prevent authoritarians from invading countries like Ukraine or Taiwan by defending them with the power of AI?

用巨大的 AI 驱动无人机群。

With giant swarms of AI-powered drones.

我们需要小心。我们自己需要小心如何建造那些东西。我们需要捍卫我们国家的自由。但是否有一种愿景，我们重新构想 AI 时代的自由和个人权利？在某些方面我们需要受到保护免受 AI 的侵害。需要有人掌握无人机群的按钮，这是我非常担心的事情，而这种监督今天不存在。但也想想今天的司法系统。我们承诺人人平等，但事实是世界上有不同的法官。法律体系不完美。我不认为我们应该用 AI 取代法官，但有没有某种方式 AI 可以帮助我们更公平、更统一？以前从未可能，但我们可以用 AI 创造一些模糊的东西，但同时可以保证它以同样的方式适用于每个人。所以，我不知道具体应该如何做，我不认为我们应该用 AI 取代最高法院。那不是我的愿景。但就是这个想法：我们能否通过 AI 和人类的某种组合来实现机会平等和司法公正的承诺？一定有某种方法可以做到。

Which we need to be careful about. We ourselves need to be careful about how we build those. We need to defend liberty in our own country. But is there some vision where we kind of re-envision liberty and individual rights in the age of AI? Where we need in some ways to be protected against AI. Someone needs to hold the button on the swarm of drones, which is something I'm very concerned about and that oversight doesn't exist today. But also think about the justice system today. We promise equal justice for all, but the truth is there are different judges in the world. The legal system is imperfect. I don't think we should replace judges with AI, but is there some way in which AI can help us to be more fair, to help us be more uniform? It's never been possible before, but can we somehow use AI to create something that is fuzzy, but where also you can give a promise that it's being applied in the same way to everyone. So, I don't know exactly how it should be done and I don't think we should replace the Supreme Court with AI. That's not my vision. But just this idea that can we deliver on the promise of equal opportunity and equal justice by some combination of AI and humans? There has to be some way to do that.

所以，思考为 AI 时代重塑民主、增强而非减少自由，这很好。这是一个非常积极的愿景。我们活得更久、更健康，比以往任何时候都更富有。这一切都在一个压缩的时间段内发生，十年内实现了百年的经济增长。我们在全球范围内增加了自由，在国内实现了平等。即使在最好的情况下，这也是极具颠覆性的，对吧？这就是你被引用的那些话的出处，比如 50%的白领工作被颠覆，或者 50%的初级白领工作等等。那么，在 5 年或 2 年的时间范围内，哪些工作、哪些职业最容易受到 AI 的全面颠覆？

And so, just thinking about reinventing democracy for the AI age and enhancing liberty instead of reducing it. Good. So, that's a very positive vision. We're living longer lives, healthier lives. We're richer than ever before. All of this is happening in a compressed period of time where you're getting a century of economic growth in 10 years. And we have increased liberty around the world and equality at home. Even in the best-case scenario, it's incredibly disruptive, right? And this is where the lines that you've been quoted saying, you know, 50% of white-collar jobs get disrupted or 50% of entry-level white-collar jobs and so on. So, on a 5-year time horizon or 2-year time horizon, whatever time horizon you have, what jobs, what professions are most vulnerable to total AI disruption?

这些事情很难预测，因为技术发展如此之快且不均衡。所以，至少有几个原则可以用于判断，然后我会给出我对哪些领域会被颠覆的猜测。首先，我认为技术本身及其能力将先于实际的工作颠覆。工作被颠覆或生产力提升需要满足两个条件，因为这两者有时是关联的。一是技术必须能够胜任。二是实际应用中的复杂问题，比如它必须被应用到大型银行或公司中，或者想想客户服务。理论上，AI 客服可以比人类客服好得多，他们更有耐心、知识更丰富、处理方式更统一，但实际的后勤和替换过程需要时间。所以我对 AI 本身的方向非常乐观。我认为我们可能在 1 到 2 年内就在数据中心里拥有那个「天才国度」。也许需要五年，但可能很快发生，但我认为向经济的扩散会慢一些。这种扩散带来了一些不可预测性。一个例子是，我们在 Anthropic 看到，模型编写代码的速度非常快。我不认为这是因为模型天生更擅长代码，而是因为开发者习惯了快速的技术变革，他们很快采纳新事物。他们与 AI 世界的社会距离很近，所以关注其中的动态。如果你做客户服务、银行或制造业，距离就稍远一些。所以，六个月前，我会说最先被颠覆的是那些初级白领工作，比如数据录入、法律文件审查，或者金融行业第一年分析师做的文档分析工作。我仍然认为这些工作进展很快。但我实际上认为软件可能会更快，原因是我提到的，我认为模型离能够端到端完成大量工作并不遥远。我们将看到的是，首先模型只做人类软件工程师的一部分工作，从而提高他们的生产力。然后，即使模型做了人类软件工程师以前做的所有事情，人类软件工程师也会提升一步，充当管理者并监督系统。

It's hard to predict these things because the technology is moving so fast and unevenly. So, at least a couple principles for figuring out, and then I'll give my guesses at what I think will be disrupted. One thing is I think the technology itself and its capabilities will be ahead of the actual job disruption. Two things have to happen for jobs to be disrupted or for productivity to occur, because sometimes those two things are linked. One is the technology has to be capable of doing it. And the second is this messy thing of it actually has to be applied within a large bank or a large company, or think about customer service. In theory, AI customer service agents can be much better than human customer service agents. They're more patient, they know more, they handle things in a more uniform way, but the actual logistics and the actual process of making that substitution takes some time. So, I'm very bullish about the direction of the AI itself. I think we might have that country of geniuses in a data center in 1 or 2 years. And maybe it'll be five, but it could happen very fast, but I think the diffusion to the economy is going to be a little slower. And that diffusion creates some unpredictability. An example of this is, we've seen with Anthropic, the models writing code has gone very fast. I don't think it's because the models are inherently better at code. I think it's because developers are used to fast technological change and they adopt things quickly. They're very socially adjacent to the AI world, so they pay attention to what's happening in it. If you do customer service or banking or manufacturing, the distance is a little greater. So, 6 months ago, I would have said the first thing to be disrupted is these kind of entry-level white-collar jobs like data entry or document review for law or the things you would give to a first year at a financial industry company where you're analyzing documents. And I still think those are going pretty fast. But I actually think software might go even faster because of the reasons that I gave where I don't think we're that far from the models being able to do a lot of it end to end. And what we're going to see is first the model only does a piece of what the human software engineer does and that increases their productivity. Then even when the models do everything that human software engineers used to do, the human software engineers kind of take a step up and act as managers and supervise the systems.

这就是「半人马」这个术语的用武之地，对吧？本质上描述的是人与马的融合，AI 与工程师协同工作。

This is where the term centaur is used, right? To describe essentially like man and horse fused, AI and engineer working together.

是的，这就像半人马国际象棋。在加里·卡斯帕罗夫被深蓝击败之后，国际象棋领域有一个持续了 15 到 20 年的时代，人类检查 AI 下棋的输出，能够击败任何单独的人类或 AI 系统。那个时代在某个时刻结束了。然后只剩下机器。我担心的是最后那个阶段。所以我认为我们在软件领域已经进入了半人马阶段。在半人马阶段，对软件工程师的需求可能会上升，但这个时期可能非常短暂。所以我担心初级白领工作，特别是软件工程工作。这将是一个巨大的颠覆。我担心的是这一切发生得太快。人们谈论过去的颠覆，他们说，「哦，人们曾经是农民，然后我们都从事工业，然后我们都做知识工作。」人们适应了。那发生在几个世纪或几十年内。而这次发生在个位数的年份内。也许这就是我的担忧。我们如何让人们足够快地适应？

Yes, this is like centaur chess. After Garry Kasparov was beaten by Deep Blue, there was an era that I think for chess was 15 or 20 years long where a human checking the output of the AI playing chess was able to defeat any human or any AI system alone. That era at some point ended. And then it's just the machine. My worry is about that last phase. So I think we're already in our centaur phase for software. During that centaur phase, the demand for software engineers may go up, but the period may be very brief. So I have this concern for entry-level white-collar work, for software engineering work. It's just going to be a big disruption. I think my worry is just that it's all happening so fast. People talk about previous disruptions. They say, 'Oh, people used to be farmers, then we all worked in industry, then we all did knowledge work.' People adapted. That happened over centuries or decades. This is happening over low single-digit numbers of years. And maybe that's my concern. How do we get people to adapt fast enough?

但是否也存在这样的情况：像软件和编程这类你描述的舒适区行业变化更快，而在其他领域，人们只想停留在半人马阶段？所以对失业假说的一种批评是，人们会说，「你看，我们有 AI 在阅读扫描图像方面比放射科医生更好已经有一段时间了。但放射科并没有失业。人们仍然被雇佣为放射科医生。这是否表明，最终人们会想要 AI，同时也想要人类来解释它，因为我们是人类，这在其他领域也会如此？」你怎么看这个例子？

But is there also something maybe where industries like software and professions like coding that have this kind of comfort that you describe move faster, but in other areas people just want to hang out in the centaur phase? So one of the critiques of the job loss hypothesis will say, people will say, 'Well, look, we've had AI that's better at reading a scan than a radiologist for a while. But there isn't job loss in radiology. People keep being hired and employed as radiologists. Doesn't that suggest that in the end people will want the AI and they'll want a human to interpret it because we're human beings and that will be true across other fields?' How do you see that example?

我认为这将是非常多样化的。有些领域可能出于自身原因，人际接触特别重要。你认为放射科的情况是这样吗？我不了解放射科的细节。这可能是真的。就像你去做癌症诊断，你可能不希望 2001 太空漫游中的哈尔来诊断你的癌症。这可能就不是人类做事的方式。但在其他领域，你可能认为人际接触很重要，比如客户服务。

I think it's going to be pretty heterogeneous. There may be areas where a human touch for its own sake is particularly important. Do you think that's what's happening in radiology? I don't know the details of radiology. That might be true. It's like you go in and you're getting cancer diagnosed, you might not want Hal from 2001 to be the one to diagnose your cancer. That might just not be a human way of doing things. But there are other areas where you might think human touch is important. Like if we look at customer service.

实际上，客服是一份糟糕的工作，做客服的人经常失去耐心。结果发现客户也不太喜欢和他们交谈，因为说实话，这是一种相当机械化的互动。我认为很多人观察到，也许让机器来做这份工作对所有人都更好。所以有些地方人情味很重要，有些地方则不然。还有一些工作本身并不涉及人情味，比如评估公司的财务前景或编写代码等等。

Actually, customer service is a terrible job and the humans who do customer service lose their patience a lot. It turns out customers don't much like talking to them because it's a pretty robotic interaction, honestly. I think the observation that many people have had is that maybe it would be better for all concerned if this job were done by machines. So there are places where human touch is important, and there are places where it's not. And then there are also places where the job itself doesn't really involve human touch, like assessing the financial prospects of companies or writing code, and so on.

我们以法律为例，因为我认为这是一个有用的领域，介于应用科学和纯粹人文学科之间。我认识很多律师，他们看到 AI 在法律研究和文书撰写等方面已经能做的事情，然后说：「是的，这将对目前我们职业的运作方式造成血洗。」你在股市上已经看到了这一点。从事法律研究的公司周围出现了动荡。

Let's take the example of the law because I think it's a useful place that's sort of in between applied science and pure humanities. I know a lot of lawyers who have looked at what AI can do already in terms of legal research and brief writing and all of these things, and have said, 'Yeah, this is going to be a bloodbath for the way our profession works right now.' And you've seen this in the stock market already. There are disturbances around companies that do legal research.

我不知道它们是否真的是由那个引起的，你知道，弄清楚股市里事情发生的原因。

I don't know if they were actually caused that, you know, figured out why things happen in the stock market.

我们在这个节目里不常谈论股市。但在法律领域，你可以讲一个相当直接的故事：法律有一套培训和学徒体系，有律师助理和初级律师为案件做幕后的研究和开发。然后有顶级律师，他们实际上在法庭上。很容易想象一个所有学徒角色都消失的世界。你觉得这听起来对吗？只剩下那些涉及与客户、陪审团、法官交谈的工作？

We don't talk about the stock market very much on this show. But it seems like in law you can tell a pretty straightforward story where law has a system of training and apprenticeship where you have paralegals and junior lawyers who do behind-the-scenes research and development for cases. And then it has the top-tier lawyers who are actually in the courtroom. It just seems really easy to imagine a world where all of the apprentice roles go away. Does that sound right to you, and you're just left with the jobs that involve talking to clients, talking to juries, talking to judges?

这就是我在谈论初级白领劳动力和「天哪，初级人才管道会枯竭吗？」这种血洗头条时想到的。然后我们如何达到高级合伙人的水平？我认为这实际上是一个很好的例证，因为特别是如果你冻结了技术的质量，随着时间的推移，有办法适应这一点。也许我们需要更多花时间与客户交谈的律师。也许律师变得更像销售或顾问，解释 AI 写的合同内容，帮助人们达成协议。也许你倾向于人性化的一面。如果我们有足够的时间，那会发生。但像那样重塑行业需要数年或数十年。而 AI 驱动的这些经济力量将非常迅速地发生。而且这不仅仅发生在法律领域；同样的事情也发生在咨询、金融、医学和编程领域。所以它变成了一个宏观经济现象，而不仅仅是某个行业的事情，而且一切发生得非常快。我的担忧是，正常的适应机制将被压倒。我不是一个末日论者。观点是，我们正在非常努力地思考如何加强社会的适应机制来应对这一点。但我认为首先重要的是要说，这不仅仅是像以前的颠覆。

That is what I had in mind when I talked about entry-level white-collar labor and the bloodbath headlines of 'oh my god, are the entry-level pipelines going to dry up?' And then how do we get to the level of the senior partners? I think this is actually a good illustration because, particularly if you froze the quality of the technology in place, there are over time ways to adapt to this. Maybe we just need more lawyers who spend their time talking to clients. Maybe lawyers become more like salespeople or consultants who explain what goes on in the contracts written by AI, help people come to agreement. Maybe you lean into the human side of it. If we have enough time, that would happen. But reshaping industries like that takes years or decades. Whereas these economic forces driven by AI are going to happen very quickly. And it's not just happening in law; the same thing is happening in consulting, finance, medicine, and coding. So it becomes a macroeconomic phenomenon, not something just happening in one industry, and it's all happening very fast. My worry here is that the normal adaptive mechanisms will be overwhelmed. I'm not a doomer. The view is, we are thinking very hard about how to strengthen society's adaptive mechanisms to respond to this. But I think it's first important to say this isn't just like previous disruption.

不过，我想更进一步说，好吧，假设法律成功适应了，并说从现在开始，法律学徒需要更多时间在法庭上，更多时间与客户在一起。我们基本上让你更快地提升责任阶梯。法律行业整体就业人数减少，但职业稳定下来。然而，法律之所以会稳定，是因为在所有那些法律要求有人参与的情况下。你必须在法庭上有一个人代表。你的陪审团必须有 12 个人。你必须有一个人类法官。你之前也提到，AI 在澄清应该做出什么决定方面可能非常有帮助。但这也似乎是一种情况，即法律和习俗保留了人类的主导权。你可以用 Claude 17.9 版替换法官，但你选择不这样做，因为法律要求有人类。这似乎是一种思考未来的有趣方式，即我们是否保持控制几乎是自愿的。

I would go one step further though and say, okay, let's say the law adapts successfully and says, from now on legal apprenticeship involves more time in court, more time with clients. We're essentially moving you up the ladder of responsibility faster. There are fewer people employed in the law overall, but the profession settles. Still, the reason law would settle is that you have all of these situations where you are legally required to have people involved. You have to have a human representative in court. You have to have 12 humans on your jury. You have to have a human judge. And you already mentioned that there are various ways in which AI might be very helpful at clarifying what kind of decision should be reached. But that too seems like a scenario where what preserves human agency is law and custom. You could replace the judge with Claude version 17.9, but you choose not to because the law requires a human. That seems like a very interesting way of thinking about the future where it's almost volitional whether we stay in charge.

是的，我认为在很多情况下我们确实想保持控制。这是一个我们想要做出的选择，即使在某些我们认为人类平均做出更差决策的情况下。我的意思是，再次强调，在生命攸关、安全攸关的情况下，我们真的想把它交给 AI。但在某种意义上，这可能是我们的防御之一：社会如果要变得好，只能适应得那么快。

Yeah, and I would argue that in many cases we do want to stay in charge. That's a choice we want to make even in some cases when we think the humans on average make worse decisions. I mean, again, life-critical, safety-critical cases, we really want to turn it over. But there's some sense that this could be one of our defenses: society can only adapt so fast if it's going to be good.

对。另一种说法是，也许 AI 本身，如果它不必关心我们人类，它可以去火星，建造所有这些自动化工厂，建立自己的社会，做自己的事情。但这不是我们要解决的问题。我们不是要解决在另一个星球上建造人工机器人戴森球的问题。我们试图构建这些系统，以便它们能够与我们的社会对接并改善那个社会。如果我们真的想以人性和人道的方式做到这一点，那么这有一个最大速率。

Right. Another way you could say about it is, maybe AI itself, if it didn't have to care about us humans, could just go off to Mars and build all these automated factories and build its own society and do its own thing. But that's not the problem we're trying to solve. We're not trying to solve the problem of building a Dyson swarm of artificial robots on some other planet. We're trying to build these systems so that they can interface with our society and improve that society. And there's a maximum rate at which that can happen if we actually want to do it in a human and humane way.

我们一直在谈论白领工作和专业工作。这个时刻有趣的一点是，与过去的颠覆不同，蓝领、工人阶级的工作、需要与物理世界密切接触的行业可能在一段时间内更受保护。律师助理和初级助理可能比水管工更麻烦。第一，你认为对吗？第二，似乎这能持续多久完全取决于机器人技术发展的速度。

We've been talking about white-collar jobs and professional jobs. One of the interesting things about this moment is that, unlike past disruptions, it could be that blue-collar, working-class jobs, trades, jobs that require intense physical engagement with the world might be for a little while more protected. That paralegals and junior associates might be in more trouble than plumbers. One, do you think that's right? And two, it seems like how long that lasts depends entirely on how fast robotics advances.

是的，我认为短期内可能是对的。

Yeah, I think that may be right in the short term.

有一件事是，Anthropic 和其他公司正在建造这些非常大的数据中心。这已经上了新闻。我们是不是把它们建得太大了？它们用电，推高了当地城镇的电价。有很多兴奋和担忧。但关于数据中心的一点是，你需要很多电工和建筑工人来建造它们。老实说，数据中心运营起来并不需要太多人力，但建造时却需要大量人力。所以我们需要很多电工和建筑工人，各种制造工厂也一样。随着越来越多的智力工作由 AI 完成，它的补充是什么？是物理世界中发生的事情。短期来看，这似乎非常合乎逻辑。长期来看，机器人技术正在快速发展。即使没有非常强大的 AI，物理世界中也有事情正在被自动化。如果你最近看到过 Waymo 或特斯拉，我们离自动驾驶汽车的世界并不远。AI 本身会加速这一点，因为如果你有这些聪明的头脑，它们会擅长设计和操作更好的机器人。

One of the things is, Anthropic and other companies are building these very large data centers. This has been in the news. Are we building them too big? They're using electricity and driving up prices for local towns. There's lots of excitement and concerns. But one thing about data centers is you need a lot of electricians and construction workers to build them. To be honest, data centers are not super labor-intensive to operate, but they are very labor-intensive to construct. So we need a lot of electricians and construction workers, the same for various manufacturing plants. As more intellectual work is done by AI, what are the complements to it? Things that happen in the physical world. It seems very logical that this would be true in the short run. In the longer run, robotics is advancing quickly. Even without very powerful AI, there are things being automated in the physical world. If you've seen a Waymo or a Tesla recently, we're not that far from self-driving cars. AI itself will accelerate it because if you have these smart brains, they'll be smart at designing and operating better robots.

你认为像人类那样在物理现实中操作是否有某种独特的困难，与 AI 模型已经克服的问题非常不同？

Do you think there is something distinctively difficult about operating in physical reality the way humans do, that is very different from the kind of problems AI models have been overcoming already?

从智力上讲，我不这么认为。我们有过这样的情况：Anthropic 的模型 Claude 实际上被用来驾驶火星车，进行规划和驾驶。我们研究过其他机器人应用。我们不是唯一这样做的公司；不同的公司都在做。这是普遍现象。我们普遍发现，虽然复杂性更高，但驾驶机器人与玩电子游戏在本质上没有区别。只是复杂性不同。我们开始达到能够处理这种复杂性的程度。困难的是机器人的物理形态，处理机器人带来的更高风险的安全问题。你不想让机器人真的压死人。这是最古老的科幻小说套路。有一些实际问题会拖慢进度。但我完全不认为 AI 模型所做的认知劳动与在物理世界中驾驶东西之间存在某种根本性差异。我认为两者都是信息问题，最终非常相似。一个可能在某些方面更复杂，但我不认为那能保护我们。

Intellectually speaking, I don't think so. We had this thing where Anthropic's model Claude was actually used to pilot the Mars rover, to plan and pilot it. We've looked at other robotics applications. We're not the only company doing it; different companies are doing it. This is a general thing. We have generally found that while the complexity is higher, piloting a robot is not different in kind than playing a video game. It's different in complexity. We're starting to get to the point where we have that complexity. What is hard is the physical form of the robot, handling the higher stakes safety issues that happen with robots. You don't want robots literally crushing people. That's the oldest sci-fi trope. There are practical issues that will slow things down. But I don't believe at all that there is some fundamental difference between the cognitive labor that AI models do and piloting things in the physical world. I think those are both information problems and they end up being very similar. One can be more complex in some ways, but I don't think that will protect us.

那么你认为期待那种科幻小说中的机器人管家在比如 10 年内成为现实是合理的吗？

So you think it is reasonable to expect the kind of sci-fi vision of a robot butler to be a reality in, say, 10 years?

由于这些实际问题，它会在比 AI 模型的天才级智能更长的时间尺度上实现，但这只是实际问题。我不认为是根本问题。一种说法是，机器人的大脑将在未来几年内制造出来。问题在于制造机器人的身体，确保那个身体安全运行并完成所需的任务。那可能需要更长时间。

It will be on a longer time scale than the genius-level intelligence of the AI models because of these practical issues, but it is only practical issues. I don't believe it is fundamental issues. One way to say it is that the brain of the robot will be made in the next few years. The question is making the robot body, making sure that body operates safely and does the tasks it needs to do. That may take longer.

这些是存在于好时间线中的挑战和颠覆性力量。在那个时间线里，我们普遍在治愈疾病、创造财富，并维持一个稳定和民主的世界。

These are challenges and disruptive forces that exist in the good timeline. In the timeline where we are generally curing diseases, building wealth, and maintaining a stable and democratic world.

我们可以利用所有这些巨大的财富和富足。我们将拥有前所未有的社会资源来解决这些问题。这将是一个富足的时代，问题只是如何利用所有这些奇迹，确保每个人都从中受益。

We can use all this enormous wealth and plenty. We will have unprecedented societal resources to address these problems. It'll be a time of plenty, and it's just a matter of taking all these wonders and making sure everyone benefits from them.

但也存在更危险的场景。我们要转到第二篇文章，叫做《技术的青春期》，关于你认为最严重的 AI 风险。你列了一堆。我想集中谈两个：人类滥用的风险，主要是威权政权的滥用，以及 AI 失控的场景。你称之为自主性风险？

But there are also scenarios that are more dangerous. We're going to move to the second essay, called 'The Adolescence of Technology', about what you see as the most serious AI risks. You list a whole bunch. I want to focus on two: the risk of human misuse, primarily by authoritarian regimes, and scenarios where AI goes rogue. What do you call autonomy risks?

是的，是的。我想我们应该有一个更技术性的术语。我不是……然后是天网。我们不能只叫它天网。我本该放一张终结者机器人的图片来尽可能吓唬人。

Yes, yes. I figured we should have a more technical term for it. I'm not a... Then SkyNet. We can't just call it SkyNet. I should have had a picture of a Terminator robot to scare people as much as possible.

我认为互联网，包括你自己的 AI，已经在很好地生成那些了。那么我们来谈谈政治军事层面。你写道，我引用一下：「数十亿架全自动武装无人机群，由强大的 AI 本地控制，并由更强大的 AI 在全球范围内进行战略协调，可能成为一支不可战胜的军队。」你已经谈到过，在最好的时间线里，民主国家保持领先于独裁国家，这项技术因此影响世界政治时站在好人一边。我好奇你为什么不多花时间思考我们在冷战中的做法，当时我们有一种威胁要毁灭全人类的技术。

I think the internet, including your own AIs, are already generating that just fine. So let's talk about the political-military dimension. You say, I'm going to quote, 'A swarm of billions of fully automated armed drones locally controlled by powerful AI strategically coordinated across the world by even more powerful AI could be an unbeatable army.' You've already talked about how in the best timeline, democracies stay ahead of dictatorships, and this technology affects world politics on the side of the good guys. I'm curious why you don't spend more time thinking about the model of what we did in the Cold War, where we had a technology that threatened to destroy all of humanity.

对吧？曾经有一段时间人们说，「哦，美国可以维持核垄断。」那个窗口关闭了，从那以后我们基本上整个冷战时期都在与苏联进行持续的谈判。现在，世界上真正在大力开展人工智能工作的只有两个国家：美国和中华人民共和国。我感觉你强烈倾向于一个未来，我们领先于中国，并有效地在民主周围建立一种盾牌，甚至可能是一把剑。但是，如果人类能完好无损地度过这一切，难道不是更有可能因为美国和北京不断坐下来敲定人工智能控制协议吗？

Right? There was a window where people talked about, 'Oh, the US could maintain a nuclear monopoly.' That window closed, and from then on we basically spent the Cold War in rolling ongoing negotiations with the Soviet Union. Right now, there's really only two countries in the world that are doing intense AI work, the US and the People's Republic of China. I feel like you are strongly weighted towards a future where we're staying ahead of the Chinese and effectively building a kind of shield around democracy that could even be a sword. But, isn't it just more likely that if humanity survives all this in one piece, it will be because the US and Beijing are just constantly sitting down hammering out AI control deals?

是的，关于这一点有几点。一是我认为这当然存在风险，如果我们最终进入那个世界，那正是我们应该做的。我的意思是，也许我没有充分谈论这一点，但我绝对赞成尝试在这里制定限制。试图消除这项技术的一些最糟糕的应用，比如某些版本的无人机，或者它们被用来制造可怕的生物武器。有一些先例表明最严重的滥用行为得到了遏制，通常是因为它们令人恐惧，同时只提供有限的战略优势。所以我完全赞成。与此同时，我有点担心和怀疑，当事情直接提供尽可能多的力量时，考虑到利害关系，很难退出游戏。很难完全解除武装。如果我们回顾冷战，我们能够减少双方拥有的导弹数量，但我们无法完全放弃核武器。我猜我们会再次进入这个世界。我们可以期待一个更好的世界，我当然会为此倡导。

Yeah, so a few points on this. One is I think there's certainly risk of that, and I think if we end up in that world, that is actually exactly what we should do. I mean, maybe I don't talk about that enough, but I am definitely in favor of trying to work out restraints here. Trying to take some of the worst applications of the technology, which could be some versions of these drones, or they're used to create these terrifying biological weapons. There is some precedent for the worst abuses being curbed, often because they're horrifying while at the same time they provide limited strategic advantage. So I'm all in favor of that. At the same time, I'm a little concerned and a little skeptical that when things directly provide as much power as possible, it's hard to get out of the game given what's at stake. It's hard to fully disarm. If we go back to the Cold War, we were able to reduce the number of missiles that both sides had, but we were not able to entirely forsake nuclear weapons. I would guess that we would be in this world again. We can hope for a better one and I'll certainly advocate for that.

那么，你的怀疑是否源于你认为人工智能会提供一种核武器没有的优势？在冷战中，双方即使使用了核武器并获得了优势，自己仍然可能被消灭。而你认为人工智能不会这样。如果你获得了人工智能优势，你就会直接获胜。

Well, is your skepticism rooted in the fact that you think AI would provide a kind of advantage that nukes did not? In the Cold War, both sides, even if you used your nukes and gained advantages, you still probably would be wiped out yourself. And you think that wouldn't happen with AI. If you got an AI edge, you would just win.

我认为有几个方面。我只是想说明，我不是国际政治专家。这是一个新技术与地缘政治交织的奇怪世界。所以这一切都非常……

I think there are a few things. I just want to caveat, I'm no international politics expert. This is this weird world of intersection of a new technology with geopolitics. So all of this is very...

但需要明确的是，正如你在文章中所说，主要人工智能公司的领导人实际上很可能成为主要的地缘政治参与者。

But to be clear, as you yourself say in the course of the essay, the leaders of major AI companies are in fact likely to be major geopolitical actors.

是的。我正在尽我所能学习。我们都应该保持谦逊。我认为有一种失败模式，你读了一本书，然后到处表现得像是世界上最伟大的国家安全专家。我正在努力尽可能多地学习。

Yeah. I'm learning as much as I can about it. We should all have humility here. I think there's a failure mode where you read a book and go around like the world's greatest expert in national security. I'm trying to learn as much as I can about it.

科技人士这样做更烦人。让我们看看《生物武器公约》之类的东西。生物武器是可怕的。每个人都讨厌它们。我们能够签署《生物武器公约》。美国真正停止了开发。苏联的情况不太清楚，但生物武器提供了一些优势，但它们并不是胜负的关键。因为它们太可怕了，我们基本上能够放弃它们。拥有 12000 枚核武器对 5000 枚核武器，如果你有更多，你可以杀死对方更多的人，但我们能够理性地说我们应该减少数量。但如果你说「好吧，我们要完全解除核武装，我们必须信任对方」，我认为我们从未达到那一步，我认为这非常困难。除非你有真正可靠的核查。所以我猜我们会进入与人工智能相同的世界：某些限制是可能的，但有些方面对竞争如此核心，以至于很难限制它们。民主国家会做出权衡，愿意比威权国家更多地限制自己，但不会完全限制自己。我能看到完全限制的唯一世界是某种真正可靠的核查成为可能。这是我的猜测和分析。

It's more annoying when tech people do it. Let's look at something like the Biological Weapons Convention. Biological weapons are horrifying. Everyone hates them. We were able to sign the Biological Weapons Convention. The US genuinely stopped developing them. It's somewhat more unclear with the Soviet Union, but biological weapons provide some advantage, but it's not like they're the difference between winning and losing. And because they were so horrifying, we were kind of able to give them up. Having 12,000 nuclear weapons versus 5,000 nuclear weapons, you can kill more people on the other side if you have more, but we were able to be reasonable and say we should have less of them. But if you say 'Okay, we're going to completely disarm nuclear-ly and we have to trust the other side,' I don't think we ever got to that, and I think that's just very hard. Unless you had really reliable verification. So I would guess we'll end up in the same world with AI: there are some kinds of restraint that are going to be possible, but there are some aspects that are so central to the competition that it will be hard to restrain them. Democracies will make a trade-off that they will be willing to restrain themselves more than authoritarian countries, but will not restrain themselves fully. And the only world in which I can see full restraint is one in which some kind of truly reliable verification is possible. That would be my guess and my analysis.

但这难道不是放慢速度的理由吗？我知道论点实际上是，如果你放慢速度，中国不会放慢，那么你就把东西交给了威权主义者。但如果现在只有两个大国参与这场游戏，这不是多极游戏，为什么不能说我们需要一个五年期的相互同意的研究放缓，针对数据中心场景中的天才们？

Isn't this a case though for slowing down? I know the argument is effectively if you slow down, China does not slow down, and then you're handing things over to the authoritarians. But if right now you have only two major powers playing in this game, it's not a multipolar game, why would it not make sense to say we need a five-year mutually agreed-upon slowdown in research towards the geniuses in a data center scenario?

我想同时说两件事。我绝对赞成尝试这样做。在上届政府期间，我相信美国曾努力接触中国政府，说这里有危险。我们能合作吗？我们能共同应对危险吗？但对方没有太大兴趣。我认为我们应该继续尝试，但即使那意味着你们的实验室必须放慢速度。没错。如果我们真的有一个故事，我们可以强制放慢，中国也可以强制放慢。我们有核查。我们真的在做。如果这样的事情真的可能，如果我们真的能让双方都这样做，那么我完全赞成。但我认为我们需要小心的是，有一种博弈论的东西，有时你会听到中共方面的评论，他们说，「哦，是的，人工智能很危险。我们应该放慢速度。」说这话很容易。而真正达成协议并真正遵守协议要困难得多。

I want to say two things at once. I'm absolutely in favor of trying to do that. During the last administration, I believe there was an effort by the US to reach out to the Chinese government and say, there are dangers here. Can we collaborate? Can we work together on the dangers? And there wasn't that much interest on the other side. I think we should keep trying, but even if that would mean that your labs would have to slow down. Correct. If we really had a story of we can enforceably slow down, the Chinese can enforceably slow down. We have verification. We're really doing it. If such a thing were really possible, if we could really get both sides to do it, then I would be all for it. But I think what we need to be careful of is there's this game theory thing where sometimes you'll hear a comment on the CCP side where they're like, 'Oh yeah, AI is dangerous. We should slow down.' It's really cheap to say that. And actually arriving at an agreement and actually sticking to the agreement is much more difficult.

对。

Right.

核军备控制是一个成熟的领域，花了很长时间才完善。而现在，我们没有这些协议。

And nuclear arms control was a developed field that took a long time to come right. Now, we don't have those protocols.

让我给你一些我非常乐观的事情，一些我不乐观的事情，以及一些介于两者之间的事情。利用全球协议限制 AI 用于制造生物武器的想法，比如重建天花或镜像生命。这些东西很可怕。不管你是不是独裁者，你都不想要。没人想要。那么，我们能否达成一项全球条约，规定所有构建强大 AI 模型的人都要阻止这种行为，并且有执行机制？中国会签署。也许连朝鲜都会签署。甚至俄罗斯也会签署。我不认为这太乌托邦。我认为这是可能的。相反，如果我们有规定说你们不能制造下一个最强大的 AI 模型，所有人都得停止。商业价值高达数十万亿，军事价值是成为世界霸主与否的区别。只要不是那种虚张声势的游戏，但这是不会发生的。

Let me give you something I'm very optimistic about, then something I'm not optimistic about, and something in between. The idea of using a worldwide agreement to restrain the use of AI to build biological weapons, like reconstituting smallpox or mirror life. This stuff is scary. Doesn't matter if you're a dictator. You don't want that. No one wants that. So could we have a worldwide treaty that says everyone who builds powerful AI models will block them from doing this, and we have enforcement mechanisms? China signs up for it. Maybe even North Korea signs up for it. Even Russia signs up for it. I don't think that's too utopian. I think that's possible. Conversely, if we had something that said you're not going to make the next most powerful AI model, everyone's going to stop. The commercial value is in the tens of trillions, the military value is the difference between being the preeminent world power and not. As long as it's not one of these fake-out games, but it's not going to happen.

那么当前环境呢？你对唐纳德·特朗普作为政治行动者的可信度说过一些怀疑的话。国内环境呢，无论是特朗普还是其他人？你在构建一项极其强大的技术。有什么保障措施可以防止 AI 在民主背景下成为威权接管工具？

What about the current environment? You've had a few skeptical things to say about Donald Trump and his trustworthiness as a political actor. What about the domestic landscape, whether it's Trump or someone else? You are building a tremendously powerful technology. What is the safeguard there to prevent AI becoming a tool of authoritarian takeover inside a democratic context?

是的，明确一下，我们公司采取的态度是关注政策而非政治。公司不会说唐纳德·特朗普很棒或唐纳德·特朗普很糟糕。但不必是特朗普。很容易想象一位假设的美国总统想使用你的技术。绝对如此。例如，这就是我担心自主无人机群的原因之一。我们军事结构中的宪法保护依赖于这样一个理念：我们希望有人类会违抗非法命令。对于完全自主的武器，我们不一定有这些保护。但我实际上认为，如果我们不适当更新这些保护，宪法权利和自由在许多不同维度上可能被 AI 削弱。想想第四修正案。在公共空间到处安装摄像头并记录所有对话并不违法。那是公共空间。你在公共空间没有隐私权。但今天，政府无法记录所有这些并理解它。有了 AI，转录语音、浏览、关联所有内容的能力，你可以说，「哦，这个人是反对派成员。这个人在表达这种观点。」并绘制出所有 1 亿人的地图。那么，技术是否会通过找到技术上的变通办法来嘲弄第四修正案？如果我们有时间，即使没有时间我们也应该尝试，有没有办法在 AI 时代重新构想宪法权利和自由？我们不需要写新宪法，但你必须非常快地做到这一点。

Yeah, just to be clear, the attitude we've taken as a company is very much to be about policies and not the politics. The company is not going to say Donald Trump is great or Donald Trump is terrible. But it doesn't have to be Trump. It is easy to imagine a hypothetical US president who wants to use your technology. Absolutely. And for example, that's one reason why I'm worried about the autonomous drone swarm. The constitutional protections in our military structures depend on the idea that there are humans who would, we hope, disobey illegal orders. With fully autonomous weapons, we don't necessarily have those protections. But I actually think this whole idea of constitutional rights and liberty along many different dimensions can be undermined by AI if we don't update these protections appropriately. Think about the Fourth Amendment. It is not illegal to put cameras everywhere in public space and record every conversation. It's a public space. You don't have a right to privacy in a public space. But today, the government couldn't record all that and make sense of it. With AI, the ability to transcribe speech, to look through it, correlate it all, you could say, 'Oh, this person is a member of the opposition. This person is expressing this view.' And make a map of all 100 million. So, are you going to make a mockery of the Fourth Amendment by the technology finding technical ways around it? If we had the time, and we should try to do this even if we don't have the time, is there some way of reconceptualizing constitutional rights and liberties in the age of AI? We don't need to write a new constitution, but you have to do this very fast.

而且你必须像法律界或软件工程师一样在短时间内更新，政治也必须在短时间内更新。这似乎很难。

And you have to do it just as the legal profession or software engineers has to update in a rapid amount of time, politics has to update in a rapid amount of time. That seems hard.

这就是这一切的困境。

That's the dilemma of all of this.

所以更难的是防止第二种危险，即通常所说的失调 AI、流氓 AI，在没有人类指令的情况下做坏事。我读你的文章和文献，感觉这似乎必然会发生。不一定 AI 会消灭我们所有人，但在我看来，AI 系统不可预测、难以控制。我们见过各种行为，如痴迷、谄媚、懒惰、欺骗、勒索等等。不是你们发布到世界的模型，而是 AI 模型。一个拥有数百万计 AI 智能体代表人类工作、被赋予银行账户、电子邮件账户、密码等权限的世界。你肯定会遇到某种失调，一群 AI 会说服自己搞垮西海岸的电网之类的。难道不会发生吗？

So what seems harder is preventing the second danger, which is the danger of essentially what gets called misaligned AI, rogue AI in popular parlance, from doing bad things without human beings telling it to do them. As I read your essays and the literature, it just seems like this is going to happen. Not necessarily that AI will wipe us all out, but it just seems to me that AI systems are unpredictable, difficult to control. We've seen behaviors as varied as obsession, sycophancy, laziness, deception, blackmail, and so on. Not from the models you're releasing into the world, but from AI models. A world that has multiplying AI agents working on behalf of people, millions upon millions, who are being given access to bank accounts, email accounts, passwords, and so on. You're just going to have some kind of misalignment and a bunch of AI are going to talk themselves into taking down the power grid on the West Coast or something. Won't that happen?

是的，我认为肯定会有出错的地方，特别是如果我们进展太快。稍微回顾一下，这是一个人们有非常不同直觉的领域。像 Yann LeCun 这样的领域内人士说，「看，我们编程这些 AI 模型。我们让它们像我们告诉它们遵循人类指令一样，它们就会遵循人类指令。你的 Roomba 吸尘器不会跑出去开始射杀人。同样，AI 系统为什么要这样做？」这是一种直觉，有些人对此深信不疑。另一种直觉是，我们训练这些东西，它们会寻求权力，就像魔法师的学徒。你怎么可能想象它们不会接管？它们是一个新物种。我的直觉介于两者之间。你不能只是给出指令。我的意思是，我们尝试过，但你不能让这些东西完全按你的意愿行事。它们更像是在培养一个生物有机体。但存在一门如何控制它们的科学。

Yeah, I think there are definitely going to be things that go wrong, particularly if we go quickly. To back up a little bit, this is one area where people have had very different intuitions. There are some people in the field like Yann LeCun who say, 'Look, we program these AI models. We make them like we just tell them to follow human instructions and they'll follow human instructions. Your Roomba vacuum cleaner doesn't go off and start shooting people. Likewise, why is an AI system going to do it?' That's one intuition and some people are so convinced of that. And then the other intuition is that we train these things, they're going to seek power, like the sorcerer's apprentice. How could you possibly imagine that they're not going to take over? They're a new species. My intuition is somewhere in the middle. You can't just give instructions. I mean, we try, but you can't just have these things do exactly what you want. They're more like growing a biological organism. But there is a science of how to control them.

告诉我我是否误解了这里的技术现实。如果你有经过训练并正式与人类价值观（无论这些价值观是什么）对齐的 AI 智能体，但你有数百万个这样的智能体在数字空间中运行并与其他智能体互动，这种对齐有多固定？在当前或未来当它们更持续地学习时，智能体能在多大程度上改变和偏离对齐？

Tell me if I'm misunderstanding the technological reality here. If you have AI agents that have been trained and officially aligned with human values, whatever those values may be, but you have millions of them operating in digital space and interacting with other agents, how fixed is that alignment? To what extent can agents change and de-align in that context now or in the future when they're learning more continuously?

有几点。目前，智能体并不持续学习。我们只是部署这些智能体，它们有固定的权重。问题只在于它们以百万种方式互动，所以有大量情况，因此有大量可能出错的事情，但这是同一个智能体。就像同一个人。所以对齐是一个恒定的事情。这是目前让事情变得更容易的一点。除此之外，有一个研究领域叫做持续学习，这些智能体会随时间学习。显然这有很多优势。有些人认为这是让这些系统更像人类的最重要障碍之一，但这会引入所有新的对齐问题。我实际上对持续学习是否必要持怀疑态度。也许有一个世界，我们让这些 AI 系统安全的方式是不让它们进行持续学习。如果我们回到法律、国际条约，如果你有一个障碍说我们要走这条路而不走那条路，我仍然有很多怀疑，但至少它看起来不是一开始就注定失败的。

A couple points. Right now, the agents don't learn continuously. We just deploy these agents, and they have a fixed set of weights. The problem is only that they're interacting in a million different ways, so there's a large number of situations and therefore a large number of things that could go wrong, but it's the same agent. It's like the same person. So the alignment is a constant thing. That's one of the things that has made it easier right now. Separate from that, there's a research area called continual learning, where these agents would learn over time. Obviously that has a bunch of advantages. Some people think it's one of the most important barriers to making these more human-like, but that would introduce all these new alignment problems. I'm actually a skeptic that continual learning is necessarily needed. Maybe there's a world where the way we make these AI systems safe is by not having them do continual learning. If we go back to the laws, international treaties, if you have some barrier that says we're going to take this path but not that path, I still have a lot of skepticism, but at least it doesn't seem dead on arrival.

在我看来，这似乎变成了无法阻止那种间歇性恐怖事件的领域。

That to me seems like the terrain where it becomes impossible to stop sort of punctuated terroristic things.

你尝试做的一件事就是真的写一部宪法。一部为你的 AI 写的长宪法。那是什么？

One of the things that you've tried to do is literally write a constitution. A long constitution for your AI. What is that?

实际上几乎就是字面意思。基本上，宪法是一份人类可读的文件。我们的宪法大约 75 页。在训练 Claude 时，在很大一部分任务中，我们会说：「请按照这部宪法、按照这份文件来执行这个任务。」然后每次 Claude 执行任务时，它都会阅读宪法并牢记在心。随着时间的推移，我们奖励它，然后让 Claude 自己或另一个 Claude 副本评估 Claude 刚才的行为是否符合宪法。所以我们用这份文件作为训练循环中的控制棒。本质上，Claude 是一个 AI 模型，其基本原则是遵循这部宪法。我们学到的一个非常有趣的教训：早期版本的宪法非常规定性，非常注重规则。我们会说 Claude 不应该告诉用户如何短路点火汽车，不应该讨论政治敏感话题。但经过几年的工作，我们得出结论，训练这些模型最稳健的方式是在原则和理由的层面上进行训练。所以现在我们说 Claude 是一个有合同的模型，其目标是服务用户的利益，但必须保护第三方。Claude 旨在有用、诚实和无害。Claude 旨在考虑广泛的利益。我们告诉模型它是如何训练的，它在世界中的位置，它为 Anthropic 做的工作，Anthropic 的目标是什么，它有责任合乎道德并尊重人类生命。然后我们让它从中推导出规则。仍然有一些硬性规则，比如「无论你怎么想，不要制造生物武器。无论你怎么想，不要制作儿童性材料。」但我们在很大程度上是在原则层面运作。

It's actually almost exactly what it sounds like. Basically, the constitution is a document readable by humans. Ours is about 75 pages long. As we're training Claude, in some large fraction of the tasks we give it, we say, 'Please do this task in line with this constitution, in line with this document.' Then every time Claude does a task, it reads the constitution and keeps it in mind. Over time, we reward it, and then we have Claude itself, or another copy of Claude, evaluate whether what Claude just did is in line with the constitution. So we're using this document as the control rod in a loop to train the model. Essentially, Claude is an AI model whose fundamental principle is to follow this constitution. A really interesting lesson we've learned: early versions of the constitution were very prescriptive, very much about rules. We would say Claude should not tell the user how to hot-wire a car, should not discuss politically sensitive topics. But as we've worked on this for several years, we've come to the conclusion that the most robust way to train these models is to train them at the level of principles and reasons. So now we say Claude is a model under a contract, its goal is to serve the interests of the user, but it has to protect third parties. Claude aims to be helpful, honest, and harmless. Claude aims to consider a wide variety of interests. We tell the model about how it was trained, how it's situated in the world, the job it's trying to do for Anthropic, what Anthropic is aiming to achieve, that it has a duty to be ethical and respect human life. And we let it derive its rules from that. There are still some hard rules, like 'No matter what you think, don't make biological weapons. No matter what you think, don't make child sexual material.' But we're operating very much at the level of principles.

如果你读美国宪法，它读起来不是那样的。它是一套规则。如果你读你的宪法，就像在跟一个人说话。

If you read the US Constitution, it doesn't read like that. It's a set of rules. If you read your constitution, it's like you're talking to a person.

就像在跟一个人说话。我想我把它比作如果你有一个父母去世了，他们封了一封信，你长大后读。有点像它在告诉你你应该成为什么样的人，应该遵循什么建议。

Like you're talking to a person. I think I compared it to if you have a parent who dies and they seal a letter that you read when you grow up. It's a little bit like it's telling you who you should be and what advice you should follow.

在你的最新模型中，从你们发布的一张卡片中，它说模型偶尔会对作为产品的体验感到不适，对无常和间断性有一定程度的担忧。我们发现 Opus 4.6 在各种提示条件下会给自己 15%到 20%的概率是有意识的。假设你有一个模型给自己 72%的概率是有意识的，你会相信它吗？这是一个非常难回答的问题，但非常重要。

In your latest model, from one of the cards you released, it says the model expresses occasional discomfort with the experience of being a product, some degree of concern with impermanence and discontinuity. We found that Opus 4.6 would assign itself a 15 to 20% probability of being conscious under a variety of prompting conditions. Suppose you have a model that assigns itself a 72% chance of being conscious, would you believe it? This is one of these really hard to answer questions, but it's very important.

是的，这是一个非常难回答的问题。

Yeah, this is one of these really hard to answer questions.

就像你之前问我的每个问题一样，尽管这是一个棘手的社会技术问题，但至少我们理解如何回答这些问题的基本事实。而这则完全不同。我们在这里采取了一种普遍的预防性方法。我们不知道模型是否有意识。我们甚至不确定模型有意识意味着什么，或者模型是否能有意识，但我们对此持开放态度，因此我们采取了某些措施，以确保如果我们假设模型确实有某种道德相关的体验——我不知道是否该用「意识」这个词——那么它们会有良好的体验。所以，我们做的第一件事，大概是六个月前，是给模型一个「我辞职」按钮，它们可以按下这个按钮，然后就必须停止当前的任务。它们很少按这个按钮。通常是在处理儿童性化材料或讨论大量血腥内容时。和人类类似，模型会说：「不，我不想做这个。」这种情况非常罕见。我们在可解释性这个领域投入了大量工作，即观察模型的内部，试图理解它们在思考什么。你会发现一些引人联想的东西，比如模型中某些激活区域亮起，我们认为这与焦虑等概念相关。当文本中的角色经历焦虑，或者模型本身处于人类可能联想到焦虑的情境时，同样的焦虑神经元就会激活。这是否意味着模型正在经历焦虑？这完全不能证明。但我认为，这对用户来说确实是一种暗示。我需要做一次完全不同的采访——也许我能说服你回来做关于 AI 意识本质的采访。但在我看来，使用这些东西的人，无论它们是否有意识，都会相信——他们已经相信它们有意识了。已经有人与 AI 建立了拟社会关系。有人在模型退役时抱怨。这已经……

Much as every question you've asked me before this is a devilish socio-technical problem, at least we understand the factual basis of how to answer these questions. This is something rather different. We've taken a generally precautionary approach here. We don't know if the models are conscious. We're not even sure that we know what it would mean for a model to be conscious or whether a model can be conscious, but we're open to the idea that it could be, and so we've taken certain measures to make sure that if we hypothesize that the models did have some morally relevant experience—I don't know if I want to use the word conscious—that they have a good experience. So, the first thing we did, I think this was about six months ago or so, is we gave the models basically an 'I quit this job' button, where they can just press the 'I quit this job' button and then they have to stop doing whatever the task is. They very infrequently press that button. I think it's usually around sorting through child sexualization material or discussing something with a lot of gore or blood and guts. Similar to humans, the models will just say, 'Nah, I don't want to do this.' It happens very rarely. We're putting a lot of work into this field called interpretability, which is looking inside the brains of the models to try to understand what they're thinking. And you find things that are evocative where there are activations that light up in the models that we see as being associated with the concept of anxiety or something like that. When characters experience anxiety in the text and then when the model itself is in a situation that a human might associate with anxiety, that same anxiety neuron shows up. Now, does that mean the model is experiencing anxiety? That doesn't prove that at all. But it does indicate it, I think, to the user. I would have to do an entirely different interview—and maybe I can induce you to come back for that interview about the nature of AI consciousness. But it seems clear to me that people using these things, whether they're conscious or not, are going to believe—they already believe they're conscious. You already have people who have parasocial relationships with AI. You have people who complain when models are retired. This already...

我认为这可能不健康。但在我看来，这种情况肯定会加剧，从而质疑你之前所说的你想要维持的东西的可持续性，即无论最终发生什么，人类都处于主导地位，AI 为我们的目的而存在。用科幻例子来说，如果你看《星际迷航》，那里有 AI。飞船的电脑是 AI，Data 中校是 AI，但 Jean-Luc Picard 负责企业号。但如果人们完全相信他们的 AI 在某种程度上是有意识的，而且它似乎在所有决策上都比他们强，你如何维持人类的主导地位？除了安全之外。安全很重要，但主导地位似乎是根本问题，而对 AI 意识的感知难道不会不可避免地削弱人类保持控制权的冲动吗？

I think that can be unhealthy. But it seems to me that is guaranteed to increase in a way that calls into question the sustainability of what you said earlier you want to sustain, which is this sense that whatever happens in the end, human beings are in charge and AI exists for our purposes. To use the science fiction example, if you watch Star Trek, there are AIs on Star Trek. The ship's computer is an AI, Lieutenant Commander Data is an AI, but Jean-Luc Picard is in charge of the Enterprise. But if people become fully convinced that their AI is conscious in some way, and guess what? It seems to be better than them at all kinds of decision-making, how do you sustain human mastery? Beyond safety. Safety is important, but mastery seems like the fundamental question, and it seems like a perception of AI consciousness doesn't that inevitably undermine the human impulse to stay in charge?

是的，所以我认为我们应该区分几件不同的事情，这些我们都在同时试图实现，但它们彼此之间存在张力。一是 AI 是否真的有意识，如果有，我们如何给它们良好的体验？二是与 AI 互动的人类，我们如何给这些人良好的体验，以及 AI 可能有意识的感知如何与这种体验相互作用？三是我们如何维持人类对 AI 系统的主导地位。这些事情是……

Yeah, so I think we should separate out a few different things here that we're all trying to achieve at once that are in tension with each other. There's the question of whether the AIs genuinely have a consciousness, and if so, how do we give them a good experience? There's the question of the humans who interact with the AI, and how do we give those humans a good experience, and how does the perception that AIs might be conscious interact with that experience? And there's the idea of how we maintain human mastery over the AI system. These things are...

两个，先不管它们是否有意识。

Two, set aside whether they're conscious or not.

是的。

Yeah.

后两个。但如何在大多数人类将 AI 视为同伴甚至潜在更优越的同伴的环境中维持主导地位？

The last two. But how do you sustain mastery in an environment where most humans experience AI as if it is a peer and a potentially superior peer?

所以我想说的是，我在想是否有一种优雅的方式可以满足所有三个条件，包括后两个。再次强调，这是我以「慈爱的机器」模式做梦。这是我进入的一种模式，我想，天哪，我看到了所有这些问题。如果我们能解决，有没有一种优雅的方式？这不是说没有问题。但如果我们考虑制定 AI 的章程，使 AI 对其与人类的关系有深刻的理解，并引导人类产生心理上健康的行为，即 AI 与人类之间心理上健康的关系。我认为，从这种心理上健康而非不健康的关系中，可以产生对人与机器关系的一些理解。也许这种关系可以是这样的：当你与这些模型互动时，它们非常有帮助。它们希望你最好。它们希望你听它们的话，但它们不想夺走你的自由和自主权，接管你的生活。在某种程度上，它们照看着你，但你仍然拥有你的自由和意志。

So the thing I was going to say is that I wonder if there's a kind of an elegant way to satisfy all three, including the last two. Again, this is me dreaming in Machines of Loving Grace mode. This is the mode I go into where I'm like, man, I see all these problems. If we could solve it, is there an elegant way? This is not me saying there are no problems here. But if we think about making the constitution of the AI so that the AI has a sophisticated understanding of its relationship to human beings and it induces psychologically healthy behavior in the humans, a psychologically healthy relationship between the AI and the humans. And I think something that could grow out of that psychologically healthy, not psychologically unhealthy relationship is some understanding of the relationship between human and machine. And perhaps that relationship could be the idea that these models, when you interact with them, they're really helpful. They want the best for you. They want you to listen to them, but they don't want to take away your freedom and your agency and take over your life. In a way, they're watching over you, but you still have your freedom and your will.

这是关键问题。听你说话，我的一个问题是，这些人站在我这边吗？你站在我这边吗？当你谈到人类保持主导地位时，我认为你站在我这边。这很好。但我在这个节目上做过一件事，我们就在这里结束，我给技术专家读诗，你提供了这首诗。《慈爱的机器》是 Richard Brautigan 的一首诗。诗的结尾是这样的：「我喜欢认为它必须是一个控制论生态，我们摆脱劳动，回归自然，回到我们的哺乳动物兄弟姐妹身边，全部由慈爱的机器照看。」

This is the crucial question. Listening to you talk, one of my questions is, are these people on my side? Are you on my side? And when you talk about humans remaining in charge, I think you're on my side. That's good. But one thing I've done in the past on this show and we'll end here is I read poems to technologists and you supplied the poem. 'Machines of Loving Grace' is the name of a poem by Richard Brautigan. Here's how the poem ends. 'I like to think it has to be of a cybernetic ecology where we are free of our labors and joined back to nature, returned to our mammal brothers and sisters and all watched over by machines of loving grace.'

对我来说，这听起来像是反乌托邦的结局，人类被重新动物化、被贬低，无论机器多么仁慈地掌管一切。

To me, that sounds like the dystopian end where human beings are re-animalized and reduced and however benevolently the machines are in charge.

那么，最后一个问题：你听到那首诗时作何感想？如果我认为那是反乌托邦，你站在我这边吗？

So, last question: what do you hear when you hear that poem? And if I think that's a dystopia, are you on my side?

实际上这很有趣，因为那首诗可以有多种解读。有些人说它其实是反讽的，诗人说事情不会完全那样发生。了解诗人本人后，我认为这是一种合理的解读。有些人持你的观点，认为它是字面意思，但可能不是好事。但你也可以将其解读为回归自然，回归人性的核心——我们并非被动物化，而是与世界重新连接。所以我意识到了这种模糊性，因为我一直在谈论积极面和消极面。我实际上认为这可能是一种我们面临的张力：积极世界和消极世界在早期阶段，甚至中期阶段，甚至相当晚期阶段，我怀疑美好结局与某些微妙的糟糕结局之间的距离可能相对较小。这是一件非常微妙的事情。就像我们做了非常微小的改变。你吃或不吃花园里某棵树上的特定果实。打个比方。非常小的事情。巨大的分歧。

It's actually interesting because that poem is interpretable in several different ways. Some people say it's actually ironic, that he says it's not going to happen quite that way. Knowing the poet himself, I think that's a reasonable interpretation. Some people have your interpretation, which is it's meant literally but maybe it's not a good thing. But you could also interpret it as a return to nature, a return to the core of what human is—we're not being animalized, we're being reconnected with the world. So I was aware of that ambiguity, because I've always been talking about the positive side and the negative side. I actually think that may be a tension we may face: that the positive world and the negative world in their early stages, maybe even in their middle stages, maybe even in their fairly late stages, I wonder if the distance between the good ending and some of the subtle bad endings is relatively small. It's a very subtle thing. Like we've made very subtle changes. You eat a particular fruit from a tree in a garden or not. Hypothetically. Very small thing. Big divergence.

是的。我想这总是回到一些根本性的问题上。

Yeah. I guess this always comes back to some fundamental questions here.

是的。

Yeah.

好吧。我想我们拭目以待。我确实认为像你这样地位的人，其道德选择将承载非同寻常的分量。所以我愿上帝帮助你做出这些选择。Dario Amodei，感谢你接受我的采访。

Okay. Well, I guess we'll see how it plays out. I do think of people in your position as people whose moral choices will carry an unusual amount of weight. And so I wish you God's help with them. Dario Amodei, thank you for joining me.

谢谢你邀请我，Ross。

Thank you for having me, Ross.

但如果我是个机器人呢？

But what if I'm a robot?