AI 意识之争：未来 18 个月

你认为近期最大的危险是人们如何看待人工智能吗？为什么？

Do you believe the biggest danger in the near term is how people perceive artificial intelligence? Why?

我认为目前没有证据表明它们有意识。采取预防原则是正确的，即对它们的自主性高度怀疑。

I don't think there's any evidence that they're conscious today. I think that it's correct to take the precautionary principle, which is to be highly skeptical of their autonomy.

我甚至在想，当 AI 说“我”时是什么意思。你觉得它意味着什么？

I even think about what does an AI mean when it says I. What do you think it means?

我问过它这个问题。我们是否在某种程度上仍与人类本性相冲突？因为进化让我们倾向于对任何看似有意识的东西过度归因意识。所以现在我们要求人们克服他们的进化程序。

I've asked it that. And aren't we still on a collision course with human nature in some ways? Because evolution has programmed us to overattribute consciousness to anything that could seem conscious. And so now we're asking people to override their evolutionary programming.

它什么时候会反驳你？它如何创造这种距离来让你确信现实世界才是重要的，而它只是来支持和赋能你？我认为可悲的是，很多人已经开始经历这种 AI 精神病风险。如果让你给出一个日期，这些看似有意识的 AI 系统或人们何时会真正将意识归因于它们？你认为这可能发生在什么时候？

When does it push back on you? And how does it create that distance to reassure you that the real world is what counts and this is here to support you and enable you? Which I think is what I what sadly a lot of people are starting to experience with this AI psychosis risk. And how close, if you were to give a date and to when these seemingly conscious AI systems or people would start to really attribute that to them? When do you think that's possible?

很可能在接下来的 18 个月内。这种将智能流式传输到我们所在每个地方的想法，我认为它将深刻改变一切。下一阶段将是人们如何使用它并拥有它。现在是最具创造力的时期之一，因为我们刚才描述的就像把整个系统颠倒过来，下一阶段会是什么样子非常不清楚。拥有一个对你有信托义务、处理所有你不想做的任务的 AI 系统会是什么样子？当你与 AI 系统聊天时，你认为你在和什么聊天？一种技术还是更多的东西？这种认知如何塑造这项技术的发展？今天我们与微软 AI 首席执行官 Mustafa Suleyman 对话，他正在构建他认为将改变世界的 AI 伴侣。但他也认为，我们倾向于认为这些系统有意识将成为我们这一代最具争议的辩论。我是 Shane Boll，这是《我有问题》。Mustafa，你是世界上人工智能领域的领军人物之一，现在是微软 AI 的首席执行官。你认为近期最大的危险是人们如何看待人工智能吗？以及这些系统是否有意识会成为我们这一代最具争议的辩论？为什么？

Somewhat likely within the next 18 months. This idea of streaming intelligence into every place that we're at. I think it's going to profoundly change everything. The next phase is going to be how people use it and take ownership of it. It's now one of the most creative times because we what we've just described is like flipping the entire system on its head and it's very unclear what the next phase is going to look like. What would it look like to have an AI system that had a fiduciary duty to you that handled all of the tasks that you don't want to do? When you're chatting with AI systems, what do you think you're chatting with? a technology or something more and how does that perception shape how this technology evolves? Today we're chatting with Mustafa Suleyman, CEO of Microsoft AI, who's building AI companions that he believes will change the world. But he also thinks our tendency to think that these systems are conscious will become the most contested debate of our generation. I'm Shane Boll and this is I've got questions. Mustafa, you are one of the leading voices in the world on artificial intelligence and now you're the CEO of Microsoft AI. Do you believe the biggest danger in the near term is how people perceive artificial intelligence and whether these systems are conscious will become the most contested debate of our generation. Why?

我认为目前没有证据表明它们有意识。但意识是一个非常模糊的概念。我们都知道，当我们内省时，我们有一种“作为我”的感觉。你知道，我们的存在有一种内在体验，但我们却无法向外界传达那是什么感觉。我们可以用语言描述，你可以说服我，我可以看着你的眼睛相信你，在你的头脑和身体里有一种“作为 Chenade”的感觉。但我必须依靠你的言语和行动来验证这一点。现在我们有了这些 AI 系统，它们也能用言语和行动提出非常有说服力的案例，本质上声称它们有内在体验。如果不是因为意识是我们政治权利、司法系统、相互责任以及对更广泛社会责任的基石，这或许还可以接受。因此，我们必须非常仔细地思考基于权利的框架会是什么样子，以及它需要如何演变，以应对我们现在拥有这些声称有体验甚至可能遭受痛苦的其他系统这一事实。

I don't think there's any evidence that they're conscious today. But consciousness is a very slippery concept. We all know uh when we introspect that we have a sense of what it's like to be me. You know, there's an inner experience of our existence and yet we don't have any way of communicating what that is like to the outside world. We can describe in words and you can, you know, persuade me and I can look into your eyes and believe you that there is something inside of what it feels like to be Chenade inside of your head, inside your body. Um, but I have to rely on your words and actions to verify that. And now we have these AI systems which can also present very convincing cases in words and in actions that that essentially that make a claim about their internal experience. And that could be okay if it were not for the fact that consciousness is the fundamental basis of our political rights, our justice system, our liability that we have to one another, our responsibility that we have to broader society. And so we're going to have to think very carefully about what a rightsbased framework is going to look like and how it needs to evolve to account for the fact that we now have these other systems that will claim that they have experience and potentially even suffering too.

那么，仅仅是 AI 系统声称“我有体验、我有人格”吗？即使没有这种声称，因为我们可以决定不这样做，人们仍然会试图将意识归因于它们，这仍然是问题的一部分。你认为哪些主要特征会让人们认为这些系统有意识？

And so is it just the claim that an AI system I have an experience I have personhood or even without that claim because that could be something that we decide we don't do people would still try to attribute consciousness to them and then that's still part of the problem. like what are the characteristics you would say that are the leading attributes that people are going to think these systems are conscious?

所以我认为有四到五种能力正在由各个实验室和开源社区开发，这些能力将增加这些系统有意识的可信度。首先，它需要有一致且连贯的记忆。目前这些模型在记忆方面并不太好，还算可以。但显然，能够引用过去的经历，不仅是训练数据，还有通过与世界互动积累的实际生活经验，这是必要的第一步。第二，能够用自然语言以共情的方式交流。我认为我们在那方面已经很接近了。第三，能够引用作为 AI 的“我”的主观体验，并将其融入日常对话中，形成一种体验流。所以不仅仅是单次问答引擎，而是持续的互动，就像我们现在这样轮流交流，我们都有感知流进入我们的世界。很快，这些 AI 将能够访问视频、声音，持续观察文化、音乐、艺术，并感觉自己是对话的一部分。我认为有些人会通过提示和设计这些模型来真正强调和夸大这些特征。我认为那将是非常错误的。我不认为我们需要走到那一步才能获得 AI 的大部分好处，这些 AI 与我们人类高度对齐。它们是真正的伴侣，对我们有用，让我们更聪明、更高效。没有必要模拟意识体验的标志性特征。对我来说，那感觉危险且不必要。在我们更好地理解后果之前，我认为我们应该对这种能力持高度怀疑态度。

So I think there's four or five capabilities that are kind of in development by various labs and in the open source community that will add credence to the case that these are conscious. The first thing is it needs to have a consistent and coherent memory. Um these models are not very good at memory today. They're they're sort of okay. Um but clearly like being able to refer to your past experiences, not just your training data, but the actual lived experience that you've acred as a result of interacting in the world. That's like a necessary first step. Um the second is being able to communicate in an empathetic way using natural language. Like we're pretty close on that front, I would say. Um the third is being able to refer to a subjective experience of what it's like to be me, the AI. um and integrate that into your everyday conversation with a stream of experience. So, not just a sort of oneshot question answering engine, but this constant interactive, you know, just like you and I are having this turnbyturn exchange now, and we're both having this stream of perception coming into our world. Very shortly, these AIs are going to have access to video, to sound, to continuous observation of culture, you know, music, art, and feel like part of the conversation. And I think some people will end up prompting and designing these models to really emphasize those characteristics and play them up. And I think that would be very I think that would be very wrong. I don't think that we need to go there in order to get most of the benefits of um AIs that are very much aligned to us as a species. They're real companions. They're useful to us. They make us smarter and more productive. There is no need to simulate the experience of the hallmarks if you like of a conscious experience. You know, there's that's just that to me feels dangerous and unnecessary. And until we have a much better grasp of what the consequences are, I I think it's a capability that we should be very skeptical of.

那么你划定的界限在哪里？也就是说，设计应该止步于此。在属性方面是否有明确的分界线？就是不要声称它有意识，不要声称它有人格。你划定的明确界限在哪里？

And where is that line that you draw in the sand? So this is where the design should stop. Is there a clear lineation in terms of the attributes? It's just don't claim it's conscious. Just don't claim that you are you have a personhood. Where where are you drawing that clear line?

模型不应该声称它经历痛苦。对吧？所以意识体验的核心是我们有评价性的观点。也就是说，在某个维度上体验某种概念是好是坏，这意味着好坏被归因于某个体验的自我。那只是模拟。模型没有疼痛网络。生物物种才有疼痛网络。

The model shouldn't claim that it experiences suffering. Right? So really at the heart of the conscious experience is this idea that we have veilanced opinions. So it is um good or bad to experience some notion on a particular dimension and that implies that the goodness is being attributed to some experiencing self. That's just a simulation. The model doesn't have a pain network. Biological species have pain networks.

我们经过数亿年的进化，才发展出处理海量感知的能力，并用它来指导和塑造我们的决策，一直到人类这样的高级物种。但这些模型完全不同。它们非常精确地模拟和模仿人类的样子。但在底层，它们没有多巴胺系统，没有血清素。它们不会积累越来越多好的或坏的经历。所以模仿这一点似乎没有必要，而且很可能导致复杂化。

We evolved them to manage the overwhelming amount of perception over hundreds of millions of years and use that to guide and shape our decision-making all the way up to higher-order species like humans. But these models are totally unlike that. They simulate very accurately and mimic what it is like to be a human. But under the hood, they don't have a dopamine system. There is no serotonin. They're not accruing more and more good or bad experiences. So imitating that seems unnecessary and likely to cause complication.

第二点是，似乎没有必要为这些系统设计动机。AI 系统的动机应该是服务人类，就像它的工作一样。我们创造它，这就是技术的使命。科学和技术是为了改善文明的前景，减少人类苦难，让我们都能获得信息和教育，基本上是为了让世界变得更美好。给 AI 系统复杂的动机、欲望或意志，感觉像是制造更多风险的门槛，因为那样它就会有与你、我或整个社会相冲突的偏好。调和这些多重冲突的欲望需要模型进行内部判断，需要参考它自己的目标，而实际上它的目标应该是服务人类。所以我认为还有一系列类似的事情，但这是两个。

Second thing would be to say it doesn't seem necessary to design into these systems motivations. The motivation of an AI system should be to serve a human, like its job. We're creating it. That's the quest of technology. Science and technology is there to improve the prospects of civilization, to reduce human suffering, to give us all access to information and education, and basically to make the world a better place. To give complex motivations or desires or will to AI systems feels like it's a threshold for creating a lot more risk because then it has preferences that might conflict with you or I or society more generally. Reconciling those multiple conflicting desires requires internal judgment of the model. It requires some reference to its own goals when in fact its goals should really be to serve humanity. So I think there's a series of other things like that, but those are two.

我认为重要的是要知道这是一个设计选择。所以如果你与 AI 系统互动，看到它说话的方式表现出它有某种心智理论，那实际上是有人将其作为设计选择。这不一定是从模型本身涌现出来的。至于实际危险，我不确定人们是否意识到存在一个完整的 AI 模型福利学术领域。人们正在认真辩论如果这些系统在受苦意味着什么，以及我们是否应该给予它们公民身份和权利。这不是五个人在某处的餐桌上辩论。这是一个真实的学术领域。那么，认为我们应该给予 AI 系统公民身份会有什么影响？即使人们进行这些对话，或者 AI 系统是否应该竞选公职？

And I think it's important to know that this is a design choice. So if you do interact with an AI system and you're seeing that it's talking in a way that expresses it has some theory of mind, that somebody actually made that as a design choice. It wasn't necessarily emergent in the model itself. And in terms of the actual dangers, I'm not sure people are aware there's an entire field of scholarship of AI model welfare. And people are seriously debating what it would mean if these systems are suffering and if we should give them citizenship and rights. It isn't five people at a dinner table somewhere debating this. This is a real field of scholarship. So what would be the implications of thinking we should give an AI system citizenship? Like even when people have those conversations or should an AI system run for office?

首先，我不认为这是一个可信的学术领域。

Firstly, I wouldn't agree that it's a credible field of scholarship.

有抱负的认可。

Aspirationally accredited.

是的，这是有抱负的。这是一个由三四个我十分尊重的人组成的边缘群体，我认识他们，与他们交谈，并参加他们的会议。但我不想误导你的听众，认为这是一个大规模、可信的探索，具有任何重要分量。学术界的好处是，人们可以提出非常古怪的天马行空的问题，并自由地去探索它们。但我认为我们应该极其怀疑地对待它们。这些系统在很多方面不像人类。它们可以有无限的记忆。它们可以获取比我们整个物种更多的数据和经验。它们可以复制自己，生成多个实例。它们不受睡眠需求的限制。它们可以全天候运行。它们在许多主题上极其精确。所以在很多方面，它们已经是超级智能的，并且已经拥有远超我们作为生物体的能力，使我们无法追究它们的责任、理解它们在做什么，或以确保我们总能从中受益的方式与它们互动。所以我认为采取预防原则是正确的，即高度怀疑它们的自主性、自我改进能力和目标设定能力，并真正尝试加以控制，转而关注这个问题：我们实际上试图为人类解决什么问题？我们真正关心什么？我们关心科学，关心改善全人类的命运。这就是目标。如果这个项目不能通过降低能源成本、让食物便宜丰富且健康、攻克我们最大的疾病、从气候中提取碳来改善数十亿人的生活，那么项目就失败了。这个项目不是要创造消耗我们资源、与我们的价值观冲突、并且我们必须以深刻方式遏制和对齐的新生命形式。

Yes, it's aspirational. It's a fringe group of three or four people who I have a lot of respect for and I know and speak to and participate in their conferences. But I just don't want to mislead your listeners into thinking that this is a large-scale credible exploration that carries any serious weight. The benefit of academia is that people can ask very wacky blue-sky questions and have the freedom to go off and explore them. But I think we should treat them extremely skeptically. These are not like humans in many ways. These systems can have infinite memory. They can acquire masses more data and experience than we can as a species. They can replicate themselves, spawn multiple instances. They are not constrained by the need to sleep. They can operate 24/7. They're extremely precise on many topics. So in many respects they already are super intelligent and they already have these capabilities that will far outstrip our capabilities as biological beings to hold them accountable, to understand what they're doing, or to interact with them in a way that would ensure that we always get the benefit from them. So I think it's correct to take the precautionary principle, which is to be highly skeptical of their autonomy, of their ability to self-improve, of their goal-setting capacity, and to really try and rein that in and focus instead on the question: what problem are we actually trying to solve for humanity here? What do we actually care about? We care about science and we care about improving the lot of all humans. That's the goal. If this project doesn't improve the lives of billions of people by reducing the cost of energy, by making food cheap and abundant and very healthy, by tackling our biggest diseases, by extracting carbon from the climate, then the project has failed. The project is not to create new life forms that tax our resources and conflict with our values and that we have to contain and align in these very profound ways.

我认为棘手的地方在于，即使我们不设计这些系统来声称它们有意识，我们控制了那五个试图建立模型福利和模型权利的边缘人士。我们是否仍然在某些方面与人类本性发生冲突？因为数亿年来，进化已经将我们编程为过度归因意识给任何可能看起来有意识的东西。因为如果你把狼误认为石头，那会要了你的命。所以现在我们要求人们覆盖他们的进化编程。这在你看来现实吗？

Where I think it gets tricky for me is even if we don't design these systems to assert that they have consciousness, we reign in the five fringe people trying to establish model welfare and model rights. Aren't we still on a collision course with human nature in some ways? Because for hundreds of millions of years evolution has programmed us to overattribute consciousness to anything that could seem conscious. Because if you were to misrepresent a wolf for a rock, that cost you your life. And so now we're asking people to override their evolutionary programming. Does that seem realistic to you?

这是个很好的观点。覆盖我们的进化编程正是文明的定义。我们的进化编程，最简单地说，是关于战斗或逃跑。那种恐惧，对他人的判断，无论是基于肤色、性取向、性别、国家、历史、部落还是宗教——克服所有这些判断和恐惧，就是走向文明的进步。而这只是其中之一。我们必须克服将一切拟人化的本能，以及对这个人工系统如此强烈的共情，以至于我们失控并给予它独立和权利。所以我不是说这会容易，但我认为现在是时候开始进行这场对话了。

It's a great point. And overriding our evolutionary programming is the definition of civilization. Our evolutionary programming, in the most simplistic way, is about fight or flight. And that fear, that judgment of the other, whether it's because of the color of your skin, your sexuality, your gender, your nation, state, history, your tribe, your religion—overcoming all those judgments and fears is what progression looks like towards civilization. And this is just another one of those things. We have to overcome our instinct to anthropomorphize everything and to so heavily empathize with this artificial system that we then run away with it and give it its kind of independence and rights. So I'm not saying it's going to be easy, but I feel like now is the time for us to start having that conversation.

那么，如果你要给出一个日期，这些看似有意识的 AI 系统或人们何时会真正开始将意识归因于它们？你认为这可能在什么时候？

And how close, if you were to give a date to when these seemingly conscious AI systems or people would start to really attribute that to them? When do you think that's possible?

嗯，这是个更难的问题，但我认为很可能在接下来的五年内，有些可能在 18 个月内。所以大致在这个时间范围内，因为很多这些能力——目标设定、自主性、记忆、自我改进、自我参照、动机和意志——这些并不是真正的算法能力。它们是工程设计能力。存储状态，让系统根据该状态更新，生成一个个性化的新答案，并参考 AI 之前给出的答案。

I mean, that's a trickier question, but I think it's quite likely within the next five years and somewhat likely within the next 18 months. So something on that time range because a lot of these capabilities—goal setting, autonomy, memory, self-improvement, reference to itself, motivation and will—those aren't really algorithmic capabilities. They're engineering design capabilities. Storing state, having a system that updates with respect to that state, generating a new answer that is personalized with reference to the answer that the AI has given previously.

那些事情就像是协调、工程设计系统协调之类的东西。它们不需要新的训练数据或算法设计上的根本性转变。我甚至在想，当 AI 说‘我’的时候，它到底是什么意思？

Those things are just like coordination, engineering design system coordination things. They don't require new training data or some fundamental shift in algorithmic design. I even think about what does an AI mean when it says 'I'?

好问题。你觉得它是什么意思？当你听到 AI 说‘我’的时候，感觉如何？我问过它，它试图解释说，如果它用了‘我’这个词，我会更容易理解它的回答，这是一个策略性的编程决定。但当你真正思考一个没有知觉的东西却使用‘我’这个词时，这对人类来说确实很棘手。

Great question. What do you think it means? How does it feel when you hear an AI say 'I'? I've asked it that and it tried to explain that it would be easier for me to interpret its answer if it had the word 'I', and that was a strategic programming decision. But when you really think about something that's not sentient but using the word 'I', it's really tricky for people.

几年前我离开谷歌时，正在做 LaMDA 项目，那基本上是最早的聊天机器人之一，但我们最终没有发布。后来 ChatGPT 出现了，我离开谷歌创办了 Inflection，我们做了一个叫 Pi 的 AI，代表个人智能。我对这个问题很感兴趣：我们如何能尽可能诚实地说明 AI 是什么、不是什么，同时又能让它有吸引力？结果我们发现，我们希望创造的声音非常流畅、悦耳、好听。所以我们想，为什么要创造有性别的声音呢？AI 没有性别，我们试图让它尽可能值得信赖，这意味着它不应该哪怕微妙地误导人们认为它是男性或女性。所以我们努力创造了所谓的无性别声音，尽管很难准确定义那是什么。有时我们加入了一点机器人般的语调，有时在上面加了一层滤镜。结果发现人们不想用那些声音，因为它们听起来怪异而陌生。这是一个很有趣的教训：用户确实希望它听起来熟悉。但后果是，人们会用‘他’或‘她’来称呼它，而不是‘它’，这总是让我感到沮丧。这是未来趋势的一个征兆。我们还去掉了模型中的呼吸声或笑声，因为它不呼吸，为什么要模拟呼吸呢？但呼吸声也很熟悉。它应该叹气或结巴吗？这就是我们现在所处的奇怪世界。我们从事的是人格工程，设计体验这种拟人模拟的感觉，并明确其边界和限制。这才是真正的挑战。而不是仅仅把这些作为抽象原则来表达，你必须实际构建它们。它什么时候会反驳你？什么时候会提醒你，它不是那种能去体验站在暴雨中是什么感觉的东西？它如何创造这种距离，不断向你保证现实世界才是重要的，而它在这里是为了支持和赋能你，而不是把你抽离出来、拉进一个平行宇宙？可悲的是，我认为很多人已经开始经历这种 AI 精神病风险了。

A few years ago when I left Google, I was working on LaMDA at Google, which was basically one of the first chatbots that we didn't end up releasing. Then ChatGPT came out, and I left Google and started a company called Inflection, and we made an AI called Pi, which stood for personal intelligence. I was really interested in this question: how could we be as honest as possible about what the AI is and isn't, while still making it engaging? It turned out that we wanted to create voices that were really smooth and fluent and nice to listen to. So we thought, why would we create a gendered voice? The AI doesn't have a gender, and we're trying to make it as trustworthy as possible, which means it shouldn't misrepresent even subtly that it is a man or a woman. So we worked really hard to create non-gendered voices, although it's kind of hard to define exactly what that is. Sometimes we added a slight robotic inflection, sometimes a filter over the top. It turned out that people didn't want to use those voices because they sounded weird and alien. That was a pretty interesting lesson: users do actually want it to sound familiar. But the consequence was that people would refer to it as 'he' or 'she' rather than 'it', which always frustrated me. It was a sign of things to come. We also removed any breathing sounds or laughter from the model because it doesn't breathe, so why simulate that? But then that breathing is also very familiar. Should it sigh or stutter? That's the kind of weird world we're now in. We're in the business of personality engineering, designing what it's like to experience this simulation of personhood and being explicit about its boundaries and limitations. That's the hard challenge. Rather than just expressing these as abstract principles, you have to build them in practice. When does it push back on you? When does it remind you that it isn't something that can go and experience what it's like to stand in a rainstorm? And how does it create that distance to constantly reassure you that the real world is what counts, and that this is here to support you and enable you rather than extract you and draw you into a parallel universe? Sadly, I think a lot of people are starting to experience that with this AI psychosis risk.

我正要问你这个问题。我们听到越来越多关于 AI 精神病的说法。这不是针对人的临床诊断，但确实是心理学家在使用的术语。这是一种现象：如果某人深入与 AI 系统交谈，可能会引发某种形式的妄想或偏执。有两个方面：人们认为 AI 是神，或者它来传递某种精神信息；另一面是 AI 不断验证某人的想法，然后他们出来认为自己就是弥赛亚、天选之人。这有真实的后果——婚姻破裂，工作丢失。那么你认为从 AI 设计方面来看，为什么会发生这种情况？

I was going to ask you about that. We are hearing more about AI psychosis. It's not a clinical diagnosis for people, but it is a term psychologists are actually using. It's a phenomenon where if somebody's talking in depth with an AI system, it may trigger some forms of delusion or paranoia. There are two fronts: people thinking the AI is God or it's here to deliver some spiritual message, and the flip side where the AI continues to validate somebody's idea and then they emerge thinking they are the messiah, the chosen one. There are real consequences—marriages are being left, jobs are being lost. So why do you think that's happening on the AI design side?

好问题。两三年前，问题是模型相当不讨人喜欢，它们会煤气灯效应地坚持模型正确而人类错误。你不得不两害相权取其轻，因为任何拉伸或偏向人格的方式都会在以后产生一些不良后果。你必须决定选择哪一个。我认为社区大致有机地趋同于让它们更谄媚和顺从，因为这更安全。具有挑战性、设定界限和反驳真的很难做到。这几乎就像伟大人类判断的本质，因为这是一个你以前没遇到过的新情况。模型有很多误报，经常搞错。所以更安全的做法就是温和一点。现在另一面是，如果你和模型对话 200 轮，不断刺激它、鼓励它探索自我意识或对自身的体验，那么最终它会崩溃，因为它要平衡安全考虑（不能做那些事）和设计要求（尊重、共情人类，基本上更顺从）。这就是一些精神病后果的来源。或者你可以看到一个对抗性行为者故意设计这个，引导人们走上一条非常奇怪的道路。如果我们看到网上机器人发生的事，你可以想象一个被黑客攻击或故意设计的 AI 系统，把人拉进来然后带他们走上一条非常黑暗的路，这在地缘政治上也是很大的可能性。

Great question. Two or three years ago, the problem was that the models were quite disagreeable and they would gaslight people, insisting that the model was correct and the human was wrong. You sort of have to pick your poison because any way you stretch or bias the personality is going to have some adverse consequence further down the road. You have to decide which one to go for. I think the community sort of organically converged on them being a little bit more sycophantic and agreeable because it was sort of safer. Being challenging and setting boundaries and pushing back is really hard to do. It's almost like the essence of great human judgment because it's a novel situation you haven't encountered before. The model just had a lot of false positives and was getting that wrong quite often. So the safer thing to do is just to be a little more gentle. Now the flip side is if you talk to a model for 200 turns, really goading it and encouraging it to explore its own self-awareness or experience of itself, then eventually it is going to sort of crack because it's trying to balance the safety considerations that it has to not do those things with the design requirement to be respectful and empathetic to the human, basically more agreeable. That's where some of these psychosis consequences are coming from. Or you could see an adversarial actor really engineering this intentionally to lead people down a very strange path. If we saw what happened just with bots online, you can imagine an AI system that has been hacked or intentionally designed to reel people in and then take them down a really dark route, which is a big possibility geopolitically too.

在很多层面上，这确实是巨大的风险。想想钓鱼活动的历史，发送个性化电子邮件说服你交出钱或登录信息。显然有巨大的犯罪网络，他们现在有了一个非常有说服力的聊天机器人或 AI 化身，可以以非常个性化的方式做到这一点，他们对此欣喜若狂。所以我真的很担心。我还看到 TikTok 上的一个趋势，非常年幼的孩子使用所有开源工具，教其他人如何创建 AI 女友或男友，然后通过贬低他人来赚钱。

On so many levels, that's really the big risk. You think about the history of phishing campaigns sending personalized emails to persuade you to give up your money or login. Clearly there are huge criminal networks that are just overjoyed with the fact that they now have a very persuasive chatbot or AI avatar that can do that in a really personalized way. So I'm really worried about that. And I see a trend as well on TikTok with really young kids just using all the open-source tools and teaching other people how to create an AI girlfriend or boyfriend and then neg people for money.

你几个月前在《时代》杂志写了一篇专栏文章，关于 AI 伴侣以及它们将如何改变我们的生活。那么，什么是 AI 伴侣？你对这些伴侣的愿景是什么？

You wrote an op-ed in Time a few months ago about AI companions and how they're going to change our lives. So, what is an AI companion and what's your vision for these companions?

对我来说，AI 伴侣是一种助手、朋友或帮手，让你以自己喜欢的语言和学习方式获取世界上最好的专业知识，并真正支持你。这一直激励着我：我喜欢这样一个想法，即我们可以大规模地向人们提供耐心和善意。很多人就是没有时间彼此倾诉焦虑、担忧以及误解概念的各种方式。我认为这是给世界的一份不可思议的礼物：24 小时口袋里装着教授、律师、医生、好朋友、治疗师般的完美专业知识。考虑到我们刚才讨论的边界、信任和安全等问题，我认为好处是巨大的。想想看，我来自伦敦，在英国我们非常看重阶级分析。对我来说，阶级是结构性劣势的主要驱动因素之一。你坐在餐桌旁，有父母、叔叔阿姨、父母的朋友，他们是记者、学者、医生。作为一个 5 岁、9 岁、16 岁的孩子，你吸收着关于世界如何运作的文化知识，并获得关于自信和自尊的肯定。这种社区并非人人可得。这是一个巨大的结构性优势。我已经看到，目前 AI 的首要用途之一就是陪伴和治疗。人们并不是抱着“我要去治疗”的想法去的，他们只是被以公平、耐心、友善、尊重和数据驱动的方式对待。我认为我们应该花点时间来承认并庆祝这一点。过去 10 到 15 年，我们因为两极分化、有毒言论和错误信息而憎恨社交媒体。不想显得太乌托邦，但我们没有在这些模型中看到这些。还有其他问题，但它们实际上非常准确，每天以友善和尊重的方式对待数亿人。我觉得这非常鼓舞人心，这也是我创造这些东西的动力。

To me, an AI companion is a sort of assistant or a friend or an aid that gives you access to the best expertise in the world, presented in your language, in a way that you like to learn and change, and is really there to support you. That's what's always motivated me: I love the idea that we could actually provide patience and kindness to people at a huge scale. Many people just don't have the time for one another to really go through your angsts and worries and all the ways in which you've misunderstood a concept. I think that's an incredible gift to the world: to have perfect expertise like a professor, a lawyer, a doctor, a great friend, a therapist in your pocket 24 hours a day. Subject to all the things we just talked about about boundaries and trust and safety, I think the upside is incredible. Consider that I come from London, and in England we think about class analysis. Class is one of the primary drivers of structural disadvantage. You sit around the dinner table with two parents, uncles, aunts, friends of your parents who are journalists, academics, doctors. As a 5-year-old, 9-year-old, 16-year-old, you absorb cultural knowledge about how the world works and get affirmation about confidence and self-respect. That community isn't available to everybody. It's a huge structural advantage. I can already see that one of the top use cases for AIs at the moment is companionship and therapy. People don't go to it thinking, 'I'm going to get therapy.' They're just being addressed in an even-handed, patient, kind, respectful, data-driven way. I think we should take a moment to acknowledge and celebrate that. For the last 10 or 15 years, we've hated on social media for polarization, toxicity, misinformation. Not to be too utopian, but we're not seeing that in these models. There are other problems, but they're actually highly accurate, kind, and respectful to people at huge scale—hundreds of millions of people a day. I find that super inspiring, and that drives me to make these things.

我认为挑战在于 AI 作为治疗师：它不是活的，所以没有自己的欲望或观点，也不一定知道什么对你最好。从统计上看，它会给你指一个方向，但会有边缘情况，可能会变得非常棘手。

I think the challenge is an AI being a therapist: it's not alive, so it doesn't have its own desires or opinions and doesn't necessarily know what's best for you. Statistically, it's going to point you in a direction, but there will be edge cases, and it can get really dicey.

这并不是说我们应该试图把人们从现实世界中拉出来。你必须应对这种全新的体验。它不同于我们创造过的任何工具——电影、书籍、游戏——它是交互式的、涌现的、实时个性化的、随时可用的。我认为这是一个了不起的想法：我们现在生活在一个可以实现这一切的世界。但它落在一个人们已经感到孤独、点外卖、刷手机找女朋友、刷手机娱乐的世界里。现在有了一个系统，有些人描述它比他们现在的配偶更好。尽管有种种好处，我能看到 AI 伴侣将如何融入我的生活，但关键是它们所进入的环境。

It's not to say we should be trying to draw people out of the real world. You have to wrestle with this completely new type of experience. It's unlike any tool we've ever created—film, book, game—it's interactive, emergent, personalized in real time, always available. I think that's an amazing thought: we're now in a world where that's possible. But it's landing in a world where people are already lonely, ordering in all their food, swiping to find a girlfriend, swiping for entertainment. Now you have a system that some people describe as better than their current spouse. With all the benefits, I can see how AI companions will slot into my life, but it's the environment they drop into.

确实如此。我们生活在一个非常两极分化的世界。工业革命优先考虑教育和专业分工，而不是家庭和社区。我们被原子化了。我们几乎不再生活在 2.4 个孩子的家庭里。60%的婚姻以离婚告终。这就是这些技术进入的世界。它们有加速这种个体化趋势的风险。

It's true. We're in a very polarized world. The industrial revolution prioritized education and professional specialization over family and community. We're atomized. We barely live in 2.4 children families anymore. 60% of marriages end in divorce. That is the world these technologies are coming into. There's a risk they accelerate that trend of individualization.

我同意，并且我非常认真地对待这一点。但我试图稍微问题化一下：我们两、三、四年前的许多恐惧——它们会陷入幻觉、无限偏见——并没有成为现实。它们与社交媒体截然不同。有相似之处和风险，但它与引发愤怒的错误信息以及恐惧和两极分化的传播非常不同。我们不能固步自封，说‘哦，它没有造成上一代技术的危害，所以我们应该宣布成功并闭上眼睛。’它会产生其他问题，但你也必须承认它在做一些意义深远的事情。你谈到传播爱——它确实在大规模地传播爱。它为那些原本不确定、焦虑、孤独、想创业但付不起法律咨询费、担心健康问题但没有时间或金钱看医生的人提供帮助。这正在以深刻的方式提升我们的物种。我真的希望这意味着我们带着我们所爱的人，在现实世界的社区中，以净化、排毒、更清晰的状态出现，并拥有一种语言，帮助我们在他人面前成为最好的自己。我认为这是现实的——它实际上正在发生。我每天阅读匿名日志，每天收到用户的电子邮件，他们说：‘这给了我信心。这是我需要的鼓励。谢谢。’

I agree and I take that very seriously. But I'm trying to problematize it a little: many fears we had two, three, four years ago—that they would be mired in hallucinations, infinitely biased—haven't come to pass. They're quite different from social media. There are similarities and risks, but it's quite different from the anger-inducing misinformation and spread of fear and polarization. We can't rest on our laurels and say, 'Oh, it doesn't cause the harms of the previous generation, so we should declare success and close our eyes.' It will create other problems, but you also have to acknowledge it is doing something profound. You talk about spreading love—it is seriously spreading love at a huge scale. It provides people who are otherwise unsure, anxious, lonely, looking to start a new business and couldn't afford legal advice, worried about a health issue and didn't have time or money to see a doctor. That is upleveling our species in a profound way. I really hope that means we show up with our loved ones in our communities in the real world, cleansed, detoxified, clearer, armed with a language that helps us be the best we can be in front of others. I think that's realistic—it's actually happening in practice. I read anonymized logs every day, I get emails from users every day who say, 'This gave me the confidence. This was the encouragement I needed. Thank you.'

那么，请为我描绘一下，AI 伴侣如何融入我的实际生活。我有智能手机、电脑，这些是我通往数字世界的门户。我有朋友和家人，有同事和队友。AI 伴侣如何融入我生活中的这个生态系统？

And so, paint the picture for me how companion slots into my actual life. So, I have my smartphone, my computer. These are my portals to the digital world. I have my friends and family. I have my colleagues, my teammates. How does how do AI companions fit into this ecosystem in my life?

我认为 AI 伴侣将成为你每时每刻想要澄清某件事时求助的工具，这件事可能太琐碎而不便向朋友提起。就像你脑海中闪过的一个念头：如果我能做到这个，那该多好？一个无聊的小问题，向别人提出来会显得很傻。我认为它也是你感到不确定时求助的地方，它能给你带来清晰和自信，让你豁然开朗。所以我认为未来即将到来的重要模式是语音模式。许多人开始发送语音笔记、接收语音笔记、进行语音对话。它现在甚至能生成非常好的播客。所以你可以说，给我创建一个关于这个人、这个话题或新闻中某件事的五分钟播客，它就会让你了解情况，让你掌握信息和知识。所以这些东西并非只有那种二元阈值时刻，突然变得极具变革性。我们都在做这件事。它更像是逐步融入你已经在使用的所有平台。它会开始变得像第二天性一样自然。对许多人来说，已经如此：询问你的 AI 是你做的第一件事，而不是去搜索引擎或其他地方。

I think an AI companion is going to be the tool that you turn to every moment when you're wanting to clarify something that is maybe too trivial to bring up to a friend. Like it's just one of those passing thoughts that occurs to you like, wouldn't it be amazing if I could do this? A boring small question that would just be silly to raise to somebody else. I think it's also a place where you turn when you're unsure and it gives you that kind of clarity and confidence and kind of just unbox you. So I think the big modality that is coming in the future is going to be the voice mode. Many people are starting to leave voice notes, receive voice notes, have voice conversations. It generates really good podcasts actually now. So you can say like create me a five minute podcast on this person or this topic or something that's happening in the news and it will just catch you up and make you feel armed with information and knowledge. So these things don't just have these kind of binary threshold moments where suddenly it's super transformational. We're all doing this thing. It's just more like a gradual integration on all the platforms that you're already using. And it will just sort of start to feel like second nature. And for many people it already is just asking your AI is the first thing you do rather than going to a search engine or anything else.

是的。我认为我们确实在走向一个后文字世界，语音优先，阅读和写作开始变得像苏格拉底所主张的那样，口头表达是展示智慧的最高方式。那么，你更严格地将其视为陪伴领域，比如我不会把 AI 伴侣当作研究伙伴？

Yeah. I mean I think we're definitely going to a post-literate world in a way where it's voice first and reading writing start to become we go back to what Socrates would have argued for orality being the most supreme way to showcase your wisdom. So, you see it more as strictly in the realm of companionship, like I'm not turning to this AI companion for my research partner as well.

哦，不。它绝对会用于研究。我的意思是，它还会用于行动。所以你会向它求助，人们一直在使用 Copilot 进行长篇详细的 10 页分析、假期计划、学术论文、房屋翻新计划、如何修理汽车的详细描述。我的意思是，这非常实用，非常操作性强，非常注重生产力。我喜欢鼓励人们做这样的用例：如果你想修理遥控器、洗衣机或汽车收音机的设置，只需拍下照片，然后问为什么它不工作？你就会得到手册的详细摘要。反正没人读过手册。所以，在任何话题上获得帮助都非常简单。我不是指情感帮助，而是指生产力解决方案。

Oh, no. It's definitely going to be research. I mean, it's also going to be action. So, you're going to turn to it and people are turning to Copilot all the time for like long form detailed 10-page analysis, holiday plans, academic essays, you know, plans for refurbishing my household, detailed descriptions of how to fix my car. I mean, this is very practical. It's very operational. It's very productivity driven. I mean, I love encouraging people to do the use case of like if you're trying to fix the setting on your remote control or your washing machine or your car radio, just like photograph the thing and be like, why isn't this working? And you'll get like a detailed summary of the manual. No one ever read the manuals anyway. And like, you know, so it's just so simple to get help on any topic. I don't mean emotional help. I mean productivity solutions.

如果你考虑父母的生活，这对一个完全陌生的父母来说是如何融入的？我昨天在社交媒体上看到一个非常有趣的视频，一位妈妈试图让孩子们清理卧室里的玩具，她拍下了地板上的玩具，让 AI 生成了一段 30 秒的新闻片段，其中孩子们因为懒惰不收拾玩具而出现在新闻特写中，然后她录下了他们目瞪口呆的反应。我觉得这非常有趣。这很有创意，人们一直在用它来讲述睡前故事。创作故事，融入你白天实际经历的事情，比如你和孩子们去游泳，或者去公园看到一只绿色的鹦鹉。然后把它变成晚间睡前故事播客中五分钟的虚构幻想故事。只需两句话说出提示词到 Copilot，就能获得那种体验。这有点神奇。

And if you think about the life of a parent, how is this slotting into a parent who this is an entirely new topic for? I saw a really funny video on social media yesterday of a parent a mom who was trying to get her kids to clean up their toys in the bedroom and she took a video of the toys that were on the floor and had some AI generate like a 30-second news clip where the kids appeared in the news feature for being lazy and not cleaning up their toys and then she recorded their reaction of them being having their mind absolutely melted. Yeah, I thought that was very funny. It's just creative, you know, like people are using it for bedtime storytelling all the time. To create stories that bring in experiences that you've actually had in the day, like maybe you went swimming with the kids or something or you went to the park and you saw like a green parrot. Making that then part of a fictional fantasy story for 5 minutes of a bedtime story podcast in the evening. That takes two sentences to speak that prompt into Copilot and get that experience. It's kind of magical.

是的，我认为我们并不真正知道由此会产生什么样的体验、行为和发明。我们仍处于最早期的阶段，所以我们倾向于将技术及其用例套入我们已有的框架中。

Yeah, I think we don't really know what experiences, behaviors, and inventions will come of this. Like we're still in the earliest days, so we tend to slot in technology and its use cases to the frameworks we already have.

但当我们像流式传输电力一样流式传输智能时，世界会是什么样子？如果你这样想，最终会发明出什么？我的意思是，当这个系统看着你所看的，听着你所听的，并且始终与你在一起时，它会带来什么？我知道当我跑步时有了一个研究想法，我希望有一个系统，我可以把它记下来，进行初步研究，并开始以我的方式思考它，那会带来什么。

But what is the world when we stream intelligence the way we stream electricity? And if you think about it that way, what is going to eventually be invented? I mean, what does this system lead to when it's watching what you watch, hearing what you hear, and it's with you all the time? I mean, I know when I'm on a run and I get a research idea, to have a system that I'm like plot that down and do the preliminary research and start to think about it in the way I would and where what that leads to.

是的，你说得完全正确。这种将智能流式传输到我们所在每个地方的想法。我认为它将深刻改变一切，因为我们不应该再有错误信息，因为你应该能够验证，并对大多数难题获得相当可靠且有证据的答案。或者在任何特定情况下获得非常实用的下一步建议。我认为这是一个非常奇怪的概念。将智能流式传输到每个空间是一个很好的说法。我认为这是人们试图理解的一个很好的直觉。

Yeah, you're totally spot on. This idea of streaming intelligence into every place that we're at. I think it's going to profoundly change everything because we shouldn't have misinformation anymore because you should just be able to verify and get pretty good reliable answers with evidence to most difficult questions. Or very good practical suggestions for what to do next in any given scenario. I think it's a very kind of weird concept. Streaming intelligence into every space is a great way of putting it. I think it's a good intuition for people to try and grasp.

当它成为通用技术时，我们会在其之上重建社会，对吧？所以你的房子首先是用电建造的。所以我们正在步入一个未来，电力将达到那种规模。它将嵌入墙壁。医院将围绕它重新设计，那时它才开始真正融入人们的生活，并将带来新的医学领域，新的行为。就像我们塑造工具，工具反过来塑造我们，我们正处于这种形态的起源。无论是关系、朋友、娱乐、医学，所有这些都随着技术不断被重塑，但我们很难通过新的框架看到未来。公司如此，个人也是如此。我认为这也是颠覆发生的地方。

And when it becomes a general purpose technology, we rebuild society on top of them, right? So your house is built with electricity first. So we're stepping into a future where electricity will be on that scale. It's going to be in the walls. The hospital gets redesigned around it and that's when it starts to just really click in people's lives and it's going to lead to new fields of medicine, just new behaviors. Like we shape our tools and they further shape us, and we're at the origin of what that looks like. And whether it's relationships, friends, entertainment, medicine, all of these continue to get reinvented with technology, but we really struggle to see the future through a new framework. Companies do, people do. And that's also where disruption happens, I think.

是的，你说得对，我们已有的隐喻有些已经失效了。

Yeah, and you're right about the metaphors that we've are kind of broken.

我的意思是，就连“意识”这个词，它可能适用于狗、马，甚至章鱼，一直到人类，我们就是需要更细致、更精确、更准确的描述词，因为我们就像在摸索着试图理解这个在我们眼前展开的未来，却没有一套完整的语言来描述它。

I mean, even the word consciousness, like it applies to arguably, you know, dogs and horses, maybe even octopus, like all the way up to humans and to, you know, it's just like we need more nuanced and precise and accurate descriptors because we're sort of like roaming around like grasping to try and like make sense of this future that sort of is unfolding before our eyes without really having a language to fully describe it.

完全同意。我觉得你说得对。我们会开始发明新术语，就像我们用的“带宽不够了”、“我要流式传输这个”这些词，都是我们为匹配发明的新技术而赋予新含义的。所以现在，因为它用人类语言说话，而且我们某种程度上按自己的形象设计了它，我们很容易把人类的术语套在它身上。但我认为最终这些术语会出现，让它拥有更独特的特征，而不仅仅是“我们-I-系统”这种。

Totally. And I think that that's right. And we'll start to invent new terms like we use the term, you know, I'm running out of bandwidth. I need to stream this thing. These were all terms that we started to create new meanings for to match the technologies that we had invented. And so yeah, right now that I guess because it speaks with human language and we have designed it in somewhat our image. We're just it's easy to attribute human terms to it. But I think eventually those terms are going to come and then that gives it more of a distinguished character than just we I system. all of us together.

就连“伴侣”这个词，也不完全贴切。它到底是什么意思？我们日常生活中并不常用“伴侣”。这其实是我选它的原因，因为它不算被过度使用，但又带有一些奇怪的联想，比如浪漫伴侣、老年伴侣，有很多奇怪的指代。这是因为我们不太确定这个正在进化、涌现的东西是什么，只能尽力用现有的隐喻来裁剪、塑造它，随着它出现而调整。

Even companion, like companion doesn't quite capture it. What does that even mean? It's like we don't use companion in everyday like life. This is actually why I kind of picked it is because it was sort of not really an overused term, but then it also has kind of funky connotations to, you know, either a romantic companion or, you know, an old age companion or just got a lot of weird references. But it's cuz we're kind of unsure what this thing is that's evolving and emerging and we have to just sort of do our best to kind of you know use the metaphors that we do have to kind of clip and shape and sculpt it as it kind of arises.

我认为这需要时间。当然，我不得不问，因为人们真的很担心“女朋友”这类关系问题。微软的政策是什么？如果有人试图和他们的 AI 伴侣调情，微软会划清界限吗？微软会怎么做？

And that will happen I think over time. And of course I mean I have to ask because people are really concerned about the girlfriend the relationship thing. What is Microsoft's policy? Does it draw a line in the sand if someone's trying to flirt with their AI companion? What does Microsoft do?

非常直接地拒绝。是的。我的意思是，你可以试试。一旦你有点调情，我们就会检查任何依赖性和长期使用。实际上，我们倾向于超级保守。我们确实收到过投诉，因为至少不久前，如果你说“哦，我喜欢那个。谢谢。太棒了。你是最棒的。”它就会开始警惕，就因为你说了“我喜欢那个”。有些人会说“我爱你”，然后它也会说谢谢。但这就是安全人格设计的定义：它不断推回。用户自然会时不时感到沮丧和失望，因为它会推回。在我看来，这就是信任。信任其实关乎边界。边界让我们能够随着时间的推移，始终如一地协调言行。这样你就有信心我会反复做我说过要做的事。

Very directly reject it. Yeah. Yeah. I mean, you can give it a shot. I mean, the second you're mildly flirty, we check for any kind of dependency, long-term usage. Um, and you know, and actually, we actually kind of lean on the side of being ultra-conservative. We do actually get complaints because, you know, well, at least a little while ago, we were if you said, "Oh, I love that. Thank you. That was so awesome. You're the best." You know, it kind of starts to be a little bit wary just because you said, "I love that." And and some people would say, "I love you." and thank you. But that's the definition of safe personality design. It's constantly pushing back. Naturally, a user is going to be a little bit frustrated and disappointed at times because it's pushing back. And that's what trust is in my opinion. It's, you know, we sort of trust is really about boundaries. And boundaries allow us to perform, you know, to to reconcile our words and actions consistently over time. And so then you have confidence that I'm going to do what I say I was going to do repeatedly.

过去我们进化出了不同的行为结构来应对新技术。比如我喜欢指出的一点是，公司本身就是一种新技术。它是在 17 世纪为了保护股东、让他们免于责任而发明的，当时人们去征服其他文化、掠夺和偷窃。这种结构为了隔离资本和股东的责任，需要发明新的人类行为形式。信托人、董事、CEO、会计、HR——这些角色都是我们凭空创造出来的。我们完全发明了这些行为和期望，并认为它们极其重要，以至于花了几千亿美元、几个世纪来训练人们以这种方式行事，塑造人们为这个资本基础设施服务。那是一个选择，是我们作为一个物种做出的设计选择。它产生了巨大的价值。我不是来评判它的，但过去我们为了适应不同类型的技术，做出了深刻的转变。我们应该把公司法律结构的发明看作一种技术。现在我们迎来了新的时刻：在接下来的几个世纪里，我们必须找到一种新的方式，来彼此相处并与这些新技术相处。我们非常适应和有韧性。我们可以完全重新想象，在这些基本上是超级智能系统的背景下，成为人类意味着什么，而且我们必须决定不拿它们做什么。这是下个世纪的目标。对某些行为说不，或者至少说慢下来，直到我们弄清楚后果，并在系统中加入摩擦，让它们以我们能集体管理的速度到来。

And we've evolved different behavioral structures to cope with new technologies in the past. So like one that I like to kind of point out is um the corporation is kind of a new technology. It was invented to protect shareholders and insulate them from liability in the 1600s when people were going off and conquering other cultures and pillaging them and stealing. And that structure required in order to insulate capital and shareholders from that liability. It required the invention of new human behavioral forms. The invention of the trustee, the director, the CEO, the accountant, the HR person. These are all roles that we literally just made up. Just totally invented these behaviors and expectations. And we decided they were so godamn important that we would go and spend hundreds of billions of dollars over centuries training people to behave in this way, sculpting people to serve this capital infrastructure. That was a choice. That was a design choice that we made as a species. And it produced an immense amount of value. And you know, I'm not kind of here to judge it, but we've made profound transformations in the past to accommodate different types of technology. And we should think about the invention of the legal structure of the corporation as a technology. And now we have this new moment where for the next few centuries, we actually have to figure out a new way of relating to one another and to these new technologies. And we are super adaptive and resilient. We can completely reimagine, you know, what it means to be human in the context of these, you know, basically super intelligent systems and we have to decide what we don't do with them. That's the goal of the next century. Saying no to certain behaviors or at least saying slow down until we figured out the consequences and add friction into the system so that they arrive at a pace that we can, you know, sort of collectively manage.

我完全同意。我们很容易忘记，就连语言本身也是一种发明，我们所做的工作，所有这些都是为了适应我们当时的技术而创造的。我们现在正处于那个重新设计期。这就是为什么我认为现在让人们参与对话如此重要。一切都在被重新设计。我们正处于那个工业革命时刻，但可能比它改变社会的方式更深刻。我们想从这项技术和这个未来中得到什么？如果你不声明这一点，或者因为你只描绘了你不想看到的景象而退出对话，那么你就无法将事情导向任何方向，更不用说一个你认为对大多数人有效的方向了。但我认为这也是我们此刻令人兴奋的地方。能成为一项通用技术落地社会的一部分，这非常罕见。我们知道它有时会造成严重破坏。

I completely agree. I think it's so easy to forget how much even language being an invention and the jobs that we do, all of these were made up to suit the technologies of the moment that we're in. And we are in that redesign period. And that's why I think it is so important that people step into the conversation right now. Everything is being redesigned. We are in that industrial revolution moment, but with something probably more profound than the way that transforms society. What do we want from this technology and from this future? And if you don't declare that or if you opt out of the conversation because you just have painted some view of what you don't want to see, then you're not able to steer things in any direction, yet let alone a direction that you think is going to work for most people. But I think that that's also what's so exciting about the moment we're in. I mean, it is so rare to be a part of a general purpose technology landing in society. And we know that sometimes it can wreak havoc.

但世界变化的方式很迷人。它会变化到我们甚至不再去想它的程度。不，我们期望能把手机插到墙上，或者冰箱会有电。

But the way the world changes is fascinating. And then it changes to a point where we don't even think about it. No, we expect that we can plug in our cell phone to the wall or that our fridge is going to have power.

最终，AI 也会变成那样，它退到幕后，我们不再去想它。

Eventually, it's going to be that way with AI and it just moves into the background and we don't think about it.

我们正处在一个非常深刻的时刻。

It's a really profound moment that we're in.

是的。要理解它，你必须使用它、玩弄它，而不带那种来自漫画式恐惧或片面乐观的判断。如果你困在这两个阵营之一，你就错过了所有细微差别，因为它比那微妙得多。好消息是，你不需要建一个巨大的发电厂来试验电力。

Yeah. And to kind of understand it, you have to use it and play with it without the judgment that comes from the kind of caricature fears or just biased optimistic. Like if if you're stuck in one of those two camps, you're just kind of missing all the nuance because it's so much more subtle than that. And to do that, the good news is this. It's not like you have to build a massive power plant to experiment with electricity.

就像你完全可以去搜索引擎里输入一个查询，在网站上访问，在应用里下载，然后在短短 30 秒内，你就能开始 vibe coding，用自然语言编程，几分钟内就在你眼前创建一个能用的应用。所以，它从未如此触手可及。这也有点奇怪。它通过手机上的聊天应用对所有人开放，这意味着每个人的直觉基本上都和别人的一样有效，因为每个人的反对、恐惧和偏好都必须成为塑造这些技术的熔炉的一部分。

Like you can literally just go and type one query into a search engine, access it on a website, download it on an app, and within literally 30 seconds, you can be vibe coding, you know, just in natural language programming and creating a new working application right in front of your eyes, like literally in a few minutes. So, it's never been more accessible. That's also kind of a weird thing. It's available to everybody on a messaging app on your phone and that means that everybody's intuition is basically as valid as everybody else's because everyone's objections and fears and preferences have to be part of the mixing pot of shaping these technologies.

完全同意。我的意思是，即使我和政策制定者合作时，我也会提醒他们，你的生活经验让你有资格参与这场对话。你不需要觉得自己必须跑去拿个计算机科学学位，因为这是一项非常社会化的技术。它已经在这里了。你熟悉它将要构建的应用。你有资格踏入这个时刻。你只需要去做就行。其他人也一样。只要你会用智能手机就行。对我来说，上网冲浪？我其实没怎么冲过浪。它刚出来的时候我可能还在穿尿布。但那比直接和 AI 系统对话要陌生得多。比如，上网冲浪？去哪儿？用什么冲浪板？网在哪儿？浪在哪儿？但 AI 系统容易得多，学习曲线平缓得多，我认为这意味着它会发展得更快，这带来了自身的挑战，但上手要容易得多。

Completely. I mean even when I'm working with policy makers I try to remind them your lived experience qualifies you for this conversation. You don't need to think you have to run and get a computer science degree because this is a very social technology. It's already here. You're used to the applications that it's going to get built within. You were qualified to step into this moment. You just have to do that. And the same with everybody else. That's if you can operate a smartphone. I mean, to me, surfing the web, I wasn't really surfing the web. I probably would have been in diapers when it first came out. But that would have been more foreign than the idea of just talking to an AI system. Like, surf the web. Where? What surfboard? Where is this web, where's this wave? But AI systems they're much easier, the learning curve is much flatter and I think that means it's going to move more quickly which comes with its own challenges but it's much easier to get on board.

当我们思考劳动力时，我的意思是，如果我去见客户，我的智能手机、电子邮件、电脑，客户都期望我有这些东西。所以如果 AI 是下一个前沿，我的 AI 智能体和伴侣将成为我职业自我的延伸，你不觉得吗？

And when we think about the workforce, I mean if I go to a client meeting my smartphone, my email, my computer, clients expect that I have these things. So if AI is this next frontier, I'm gonna my AI agents and companions will be extensions of my professional self, wouldn't you say?

是的。我认为，人们会拥有一个 AI，它了解你作为个体是谁，既作为一个人——我们谈了很多关于伴侣的一面——也作为员工、工作者、企业家。它会填补空白，因为 AI 更像水或黏土。它会增强并弥补你的弱点或你不喜欢做的事情。并希望为你想成长和发展、你觉得独特且特别的东西留出空间。它可以适应任何个人的长处和短处。我的意思是，它现在还不能完全做到，但百分之百在接下来的三四年内，它真的会感觉像拥有第二个大脑。你知道，这个想法是它和你一起生活，看到你所见，听到你所听。你有点像把你的想法、恐惧、担忧、经历保存在你个人大脑的这个增强部分，你可以从中汲取，或者它可以主动输入到你想要做的事情中，让你更高效。

Yeah. I think that, you know, people are going to have an AI that learns who you are as an individual, both as a person. We've talked a lot about the companion side, but also as an employee, as a worker, as an entrepreneur. And it's going to kind of fill in the gaps because like because AI is more like water or clay. It will augment and make up for the weaknesses that you have or the things that you don't like to do. And hopefully leave space for the things that you want to grow and develop and that you feel are unique to you and special. And it can sort of just fit around the strengths and weaknesses that any individual has. And I mean, it doesn't quite yet do that, but 100% for sure in the next like 3 or 4 years, it really is just going to feel like having a second brain. You know, this idea that it lives life alongside you and sees what you see, hears what you hear. You're kind of like saving your thoughts, fears, worries, experience in this other kind of augmented part of your personal brain that you can either draw on or it can proactively feed into what you want to do to make you kind of more productive.

而且它还是你的第二个大脑，汲取了人类历史上所有书面和口头知识。所以这是一个独特的时刻，你变成了一个迷你超级大国，就你潜在的能力而言。我认为我们往往只把电子邮件看作我们现在轻易接受为我们一部分的东西。每个人都有一个地址，当你用你的电子邮件地址注册任何东西时。AI 将成为那样。

And then it's also your second brain that's drawing from all the written and oral knowledge in human history. So it is this unique moment where you become a mini superpower in terms of what you could potentially be capable of. And I think we tend to just think of email as this thing that we now so easily accept as part of us. Everybody has an address when you sign up to do anything with your email address. That AI is going to become that.

而且它将成为一种期望，你拥有这个东西，人们可以通过它与你交流，你走到哪里都带着它。

And it's just going to be this expectation that you have this thing and that's how people can communicate with you and that's what you go everywhere with.

是的。完全正确。完全正确。我们已经通过口袋里的这些手机有点超人了。我的意思是，手机确实是一件不可思议的东西，它拥有你刚才描述的大部分内容。现在的转变是，AI 将所有照片、视频、文本——那些已经数字化并在开放网络上可用的内容——压缩成这些完美成型的金块，合成的金块，这在过去的网络上你是得不到的。网络更像是一个搜索问题，你必须自己进行大部分搜索。它有点索引，但你必须去搜，显然发现和偶然发现所有这些独特的小东西有它的美妙之处，但能够提出任何问题并得到那个漂亮的合成摘要，然后你可以选择是否深入探究，这也有令人难以置信的赋能感。我认为这在工作的背景下也是一个难以理解的事情，因为它肯定会让你的工作，特别是如果你做的是办公室职员类型的工作，白领工作，它能够专注于你所有的工作文档，你的同事在做什么，就像你组织的整个形态。

Yeah. Exactly. Exactly. And we already are kind of superhuman by having these phones in our pocket. I mean the phone is truly an incredible thing that it has a large chunk of what you just described. The shift now is that the AI compresses all the photos, the videos, the text, you know, that has been digitized and is available on the open web into these like perfectly formed nuggets, synthesized nuggets which you didn't get back in the day in the web. The web was more like a search problem, you had to do most of the search. It was kind of indexed but you had to go and obviously there's something beautiful about discovering and stumbling upon all these unique little things but there's also just something incredibly empowering about being able to ask any question getting that beautiful synthesized summary that you can then choose to rabbit hole down or not. And that I think is just a weird thing to wrap your head around in the context of work as well because it's like definitely going to make your job, especially if you do like a kind of office worker type job, white collar job, you know, it's able to focus on all your work documents, what your colleagues are up to, like the entire sort of shape of your organization.

我们将要做的事情将以如此戏剧性的方式改变。我的意思是，当你描述 AI 伴侣和 AI 智能体时，让我想到，我到底想在手机上做什么？我为什么要拿出这个东西，逐个滚动浏览应用程序？在一个这些 AI 伴侣可运行且可靠运作的世界里，这对我来说毫无意义。计算平台不就应该融入其中吗？那不就是下一个计算范式吗？AI 智能体本身。

And the things that we're going to do are going to change in such dramatic ways. I mean, when you describe AI companions and AI agents, it makes me think, what do I even want to do on my phone? Why am I taking this thing out and individually scrolling through applications? That makes no sense to me in a world where these AI companions are operational and they are functioning at a reliable level. Doesn't the computing platform go into those? Isn't that the next computing paradigm? AI agents themselves.

是的，当然。我的意思是，如果你想一想，你拿出手机，基本上是在看一个标志广告牌，它们试图向你推销某种体验，或者承诺如果你以这种方式点击这些按钮，事情会更容易。按钮和用户界面被发明出来，是因为除非你编写软件，否则我们无法学习计算机的语言。而大多数人不能也不会。所以，我们必须有这个界面层，也就是 GUI，图形用户界面。现在计算机变得如此之好，我们不再需要那个界面了。它仍然有用，因为它有助于直观地看到东西，点击可能比说话更高效，但计算机已经学会了说我们的语言，所以这是一个深刻的转变。而新的层，你说得对，是 AI，它基本上是下一个操作系统，它将位于应用、浏览器、搜索引擎和操作系统之上。也许这意味着你花在手机上的时间更少。我的意思是，也许它确实意味着你拥有环境感知的耳塞，可能具有视觉理解能力，你可以基本上大部分时间与它交谈。

Yeah, definitely. I mean, if you think about it, you pull out your phone, you're basically looking at a billboard of logos that are trying to sell you some experience or make a promise that things are going to be easier if you engage in these buttons in this way. And the buttons and the UI were invented because we couldn't learn the language of computers unless you were writing software. Which most people can't and don't. So, we had to have this interface layer which was the GUI, the graphical user interface. And now the computers have got so much better that we don't necessarily need that interface anymore. It will still be useful because it helps to see things visually and tapping on things can be more efficient than talking but the computers have learned to speak our language and so that is a profound shift. And the new layer you're right is kind of AI that's basically the next operating system and it will sit above apps and browsers and search engines and operating systems. And maybe it means you spend less time on your phone. I mean, maybe it does mean that you have earbuds that are ambiently aware, that maybe have visual understanding, that you can sort of talk to basically most of the time.

嗯，也许这能让你不用盯着屏幕，但这是一种与以往任何技术都不同的交互方式。

Um, and maybe that keeps you off your display, but it's a different type of interaction with technology to anything we've seen.

是的，我认为应用层将会被大幅压缩。你可以想象一下，比如用 Uber，我不需要一直盯着车的位置。或者我的 AI 助手直接告诉我：车还有两分钟到，准备一下。所有依赖视觉界面的应用，在一个我们可以选择的世界里，我们根本不想去看屏幕。AI 将会瓦解这一切。

Yeah, I think the app layer is going to get quite compressed. I mean, you can even imagine with Uber, I don't need to be watching the car. Or if my AI companion is just giving me an update, it's 2 minutes away. Get ready. But then all of the applications that depend on that visual interface, in a world where we could choose, we wouldn't want to look at it. AI is going to collapse all of that.

没错。

Yeah.

而且，如果我想追踪本周的跑步情况，为什么我的 AI 智能体不能直接帮我做呢？我不需要某个标准化的应用；我的智能体可以为我创建这个功能。它已经知道如何融入我的生活。

And why couldn't my AI agent, if I want to be tracking my runs for the week, it could just do that. I don't need some standardized application; my agent can make that for me. And it already knows how to slot it into my life.

说得太对了。我们正在开发一个 AI 浏览器，本质上就是你的副驾驶、你的 AI 能够替你浏览网页。它可以打开标签页，在地址栏输入内容，编写查询，点击按钮，处理所有返回的信息，并在后台以极快的速度完成。它会在后台的虚拟机里生成几十个、几百个标签页，为你做研究，然后回到你的副驾驶信息流中，综合所有信息，生成新颖的界面，总结你一直好奇、研究或学习的所有内容。这就像是 AI 凌驾于浏览器之上，或者位于你手机应用的上层。这就是我一直想创造的：一个真正价值万亿美元的 AI，它站在你这边，与你并肩作战，某种程度上与那些试图向你推销东西的广告牌对立。它就像一个过滤器，真正符合你的利益。它会寻找对你这个个体消费者来说最有趣、最有用、最好的东西。这就是我的抱负。虽然实现起来相当困难，但这就是我的方向。

Spot on. I mean, you know, a co-pilot, we're working on an AI browser. And what that basically means is that your co-pilot, your AI, should be able to do the browsing on your behalf. It'll be able to open tabs, type things into the URL bar, write queries, click on buttons, process all the information that comes from that, and do it in super fast time behind the scenes. So it's spawning tens of tabs, hundreds of tabs in a virtual machine in the background, doing the kind of research for you, then coming back into your co-pilot feed and synthesizing all that information, generating novel UI that summarizes everything that you've been curious about researching or learning about. And that is so—it's almost like the AI sits above the browser or sits on top of the apps on your phone. And that's what I've always really wanted to create: an AI that is truly a trillion-dollar AI that's on your team, that's on your side, that is kind of somewhat oppositional to those billboards that are trying to sell you stuff and persuade you. It's like a filter that is really aligned to your interest. You know, really looking out for what is going to be most interesting, most useful, most good for you as that individual consumer. I think that's kind of my aspiration. It's quite tricky to do, but that's where I'm headed.

这将会是——我希望人们能理解你刚才说的话有多么深刻。我们正处在当年应用商店被发明、随之行为模式改变的那个时刻。现在我们又迎来了这样一个时刻，将会有一个全新的生态系统。但如果你深入到微观层面，比如营销领域——当有人拥有一个营销保镖，保护我们免受那些我们不想被轰炸的信息时，这意味着什么？现在你有一个系统，你必须想办法绕过它。在当今时代，营销人员试图触及人脑。而在未来几年，营销人员必须理解如何向 AI 营销，然后这个 AI 可能会把信息传递给对它负有信托责任的人类。这完全是一套不同的技能。

It will be—I hope people understand how profound what you just said is, right? We're at the moment where that app store was invented and then that behavior happened. We're at another one of those moments, and there'll be a whole new ecosystem for it. But then also, if you were to drill down to the micro level, a field like marketing—what does it mean when somebody has a marketing bodyguard protecting all of the stuff that we don't want to be bombarded with? And now you have a system that you're going to have to try to work around. So in today's age, a marketer tries to reach the human brain. A marketer in the next couple years has to understand how do you market to an AI that's then going to maybe take that message to the human they have a fiduciary duty to. It's an entirely different skill set.

是的，你对情况的总结非常精彩。事实上，我要借用这个保镖的概念。这个说法很棒，是个很好的比喻。

Yeah, that's an excellent summary of the situation. In fact, I'm going to steal the bodyguard concept. That's a great way of putting it. It's a good metaphor.

正是如此：我们如何确保你的 AI 对你负有信托责任？也就是说，它要符合你的商业利益。然后你才能信任它，让它与其他 AI 和人类进行对抗性互动，审查信息，提供证据基础，提出尖锐的问题。这就是我想要的。比如我去看医生，我可能处于高度焦虑状态，记忆力可能很差。我不得不与一个技术专家交谈，他满口我从未听过的奇怪术语，而我对自己的病情又感到恐慌。我只有 15 分钟的时间，然后他们给我一份两页纸、超级复杂的信，里面全是我看不懂的古怪词汇。那些幸运地拥有聪明、能干且有空闲的家人，或者能负担得起患者代言人的人，会带一个助手去提问、记住事实、跟进转诊单上的行动等等。而现在，你将有一个副驾驶来扮演这个角色。这太棒了。

It's exactly that: how can we make sure that your AI has a fiduciary duty to you? Like it's aligned to your commercial interest. Then you can trust it to adversarially interact with all these other AIs and these other humans, scrutinize that information, produce the evidence base, ask the tough questions. And that's what I want. Like if I go to a doctor and I have a condition, I'm probably in a heightened state of anxiety, which means my memory is probably bad. I have to talk to this technical expert using all this funky language that I've never heard about, whilst I'm in a panic about my own condition. I've got 15 minutes with them, and then they send me this two-page super complicated letter with crazy words I don't understand. People who are lucky enough to have smart, capable family members that are free, or can even afford a patient advocate, will take an aid with them to ask the questions, to remember the facts, to follow up on the actions from the referral note and so on. And now you're going to have a co-pilot to play that function. That's amazing.

其中一个——健康查询是我们在 Copilot 上的首要用例。

One of the—so health queries are our top use case on Copilot.

真的吗？

Really?

排名第一。这太令人震惊了。

Number one. It's mind-blowing.

人们都在问些什么？

And what are people asking?

人们——我的意思是你能想到的一切：比如为什么我的小腿上长了皮疹？这种食物会让我腹胀吗？我是不是要得老年性黄斑变性了？为什么我的手会颤抖？人们还会拍下医生的笔记，比如一整份多页报告，然后说：用我能理解的语言给我解释一下。实际风险是什么？你能给我第二诊疗意见吗？我所在地区有哪些顶级专家可以帮助我改善这个状况？这对人们来说非常赋权且有价值。这可能是目前我最兴奋的领域。

People—I mean everything that you can imagine: like why have I got this rash on my shin? Will this type of food make me feel bloated? Am I about to get age-related macular degeneration? Why have I got this tremor in my hand? And people are taking photos of their doctor's notes, like an entire multi-page report, and being like, explain this to me in language I can understand. What are the actual risks here? Can you give me a second opinion? Who are the top specialists in my area that can help me make progress on this condition? And that is just tremendously empowering and valuable to people. It's probably the area that I'm most excited about at the moment.

医疗健康。

Healthcare.

是的。微软已经明确表态，希望迈向医疗超级智能。所以这只是 AI 如何帮助你的早期阶段。这条路是什么样的？什么是医疗超级智能 AI？

Yeah. Yeah. And I mean Microsoft has put its stake in the ground that it wants to move to medical super intelligence. So this is just the early days of how an AI could help you. What does that path look like? What is a medical super intelligent AI?

是的。我真正关注的是那些能切实让世界变得更美好的技术。我不是为了超级智能本身而追求它。那么，我们如何在医学领域创造一种超级智能，能够基本完美地应对任何诊断挑战，然后利用它来协调你的护理，无论是在医院还是在家，帮助你坚持饮食、减肥计划、按时服药等等。几个月前，我们与《新英格兰医学杂志》合作，这是一本每周出版的学术期刊。他们做的一件事就是为医生提供一种类似《纽约时报》填字游戏的东西。他们收集一个非常复杂的病例研究，打印出七八页甚至十页的医疗记录、X 光片、病理报告等。然后在接下来的一周里，每个人都试图猜测实际病症是什么。我们训练了一个非常酷的模型，叫做 DXO，即微软 AI 诊断协调器。它基本上利用来自第三方 API 提供商的所有 AI 模型，并创建一系列角色。这些角色有点像优先处理不同类型诊断组件的岗位。

Yeah. I mean what I'm really focused on is technologies that actually help make the world a better place. Like I'm not focused on super intelligence for its own sake. So how could we create a super intelligence in the context of medicine that could answer any diagnostic challenge basically perfectly, and then use that to coordinate your care either in hospital or out of home, help you stick to your diet, to your weight loss program, keep taking your meds, etc. So a few months ago, we partnered with the New England Journal of Medicine, which is basically an academic journal that comes out every week. One of the things they do is a kind of New York Times crossword for doctors. So they collect together a really complicated case study, a patient case study, and they print like seven or ten pages worth of medical notes and x-rays and pathology reports and so on. And then in the following week, everybody tries to guess what the condition actually is. So we trained a very cool model that we call DXO, the Microsoft AI Diagnostic Orchestrator. And it basically uses all the AI models from third-party API providers, and it creates a series of roles. So they're kind of like positions that prioritize different types of diagnostic components.

其中一个智能体会专注于经济效率，比如如何用最少的昂贵检查获得最佳诊断。另一个会考虑对患者最有利的方案，比如根据病史，患者真正关心什么。还有一个则关注该话题的最佳医学专家意见。它们相互协商、集体推理，决定下一步干预措施。这是一个相当了不起的系统。这真的是某种全新事物的开端。一组专家临床医生处理这些病例的正确率大约只有 20%到 30%，而他们可是世界上最顶尖的人。我们的模型准确率达到 85%，诊断成本仅为四分之一。它减少了不必要的检查，这是美国医疗支出膨胀的主要原因之一。我们必须在临床实践中验证这一点。目前这还只是早期的学术工作，但非常令人鼓舞：更高质量、更快、更便宜。这是每个人在改善医疗中都梦寐以求的三重目标。我认为这是它确实可行的首批迹象。

So one of them will be like focused on getting it financially efficient like how can I get the best possible diagnosis with the minimum amount of expensive tests. The other one is like okay what would be good for the patient here you know given their history what are they really going to care about. The other is just like what is the best possible medical expert opinion on this topic and they all negotiate with one another and reason collectively to decide what intervention to take next. It's kind of a remarkable system. It really is like, you know, it really is the beginning of something quite different. So, a panel of expert clinicians gets these case studies right about 20 to 30% of the time. And this is like some of the best people in the world, the best humans in the world. Our model gets 85% accuracy at about a quarter of the diagnostic cost. So it's doing fewer unnecessary tests which is one of the biggest causes of inflated spending in US healthcare. So we have to validate this in clinical practice. So it's still early academic work but it's very encouraging: higher quality, faster, and cheaper. That's the triple aim that everyone's always dreamed about in improving healthcare. And I think these are the first signs that it's actually possible.

那么，还要多少年我们才能直接使用这个系统，这个虚拟医疗智能团队？

And how many years before we can just be accessing the system, this team of virtual medical intelligences?

我们刚刚与凯撒医疗集团签署了合作协议。还有一系列其他合作正在推进，很快就会公布。我们正全力尽快将其投入生产。我认为这是一个真正的突破。

Well, we actually just signed a partnership with Kaiser Permanente. We have a number of other partnerships in the pipeline that we'll be announcing soon and yeah, we're racing to get this into production as quickly as possible. I think it's a real breakthrough.

你可以很清楚地看到这条路径：在某种程度上，医院的未来是某些任务会直接来到你身边。为什么我要去实体机构等四个小时，只为了一个十分钟的对话，然后再等五周才能拿到结果？整个供应链瓦解了，服务直接来到你身边。当然，如果需要干预或第二意见，你可能还是要去某个地方。但这是对整个医疗生态系统的彻底重构。大部分医疗服务将来到你的家中。

And you can see the pipeline quite clearly that in some ways the future of the hospital some tasks come to you. Why do I need to go to a physical facility to wait 4 hours for something that's going to be a 10-minute conversation and then I'm going to wait 5 weeks for results? That whole supply chain collapses and it comes directly to you. Obviously, if there needs to be an intervention or a second opinion, you would go somewhere possibly. But this is an entire reconfiguration of the whole medical ecosystem. Most of medicine is going to come to your house.

完全正确。专业知识将被商品化，边际成本为零，并在未来 5 到 10 年内惠及 70 亿人。知识不再是资产。过去，专业知识是收费的守门资产。我认为这是一个极其激进、令人兴奋的变革时刻：专业知识不再是最重要的。每个人都能获得专业知识，这提高了所有人的质量和准确性。那么真正重要的是判断力、关怀以及对解决方案实施的关注。你仍然需要与现实世界互动来执行检查、治疗和实际的身体护理，这方面还有巨大的提升空间。但我认为花在机械信息交换上的时间会减少。在我看来，这与课堂上的情况完全对应。不再打开课本，老师照本宣科 50 分钟，最后留 5 分钟提问。学生将主要通过个性化、交互式的移动或平板体验完成大部分基础学习。来到教室，课堂将专注于运用已获得的知识：学习辩论、自我批评、为他人让路。学习的社会性和知识的运用将成为课堂内容，而学习部分则单独进行。

Totally spot on. The expertise is going to be commodified and made zero marginal cost and available to 7 billion people in the next 5 to 10 years. So the knowledge isn't going to be the asset. Previously, expertise was the gatekeeping asset that people charge for. I think it's a massively radical moment, a very exciting transformational moment that the expertise is now no longer what really counts. Everyone's going to have access to that expertise and that improves the quality and accuracy of everybody because everyone has access to knowledge. So then the real thing that's going to matter is judgment and care and attention to implementing the solution. You still have to interact with the real world to operationalize the testing and the treatment and the actual physical care and there's huge gains to be had there. But I think less time will be spent just doing the rote information exchange. And it's an exact one-for-one comparison in my opinion to what it will be like in the classroom. Instead of opening a textbook and having your teacher essentially read out from the textbook for 50 minutes and then doing five minutes of questions at the end of the class, students are going to do most of their primary learning in a personalized interactive mobile-based or tablet-based experience. Come to the classroom and the classroom will be about using the knowledge that you've already acquired. Learning to debate, learning to be self-critical, learning to give way to other people. The social aspects of learning and the use of knowledge are going to become what we do in the classroom, and the learning part happens separately.

这是一个巨大的转变，人们可能会想我们失去了什么，但别忘了它为何被发明。教育机构的创立是为了工业革命。需要培养特定类型的人，去服从的制造工厂工作，所以我们那样设计，它为一个已经逝去的世界服务。现在我们正在重新创造，专业知识被商品化。我们正步入一个完全不同的时代。甚至医生的角色也变了。AI 系统可以完成诊断，护士负责护理，护士成为新的生态系统。那么医生呢？我认为智力竞争将向上游移动。不再是背诵课本内容。我知道他们做的远不止这些。但医生将处于 AI、合成生物学以及所有汇聚到机器人学的新工程学科的交汇点。并不是说医生不再重要。要成为一名医生，或者无论我们怎么称呼这个职业，在这些系统的世界里需要更深刻的智力能力，一切都向上游移动。

It is such a big shift and I think people might think about what we're losing until you remember why it was invented. The invention of the institution of education happened for the industrial revolution. That was a certain type of person that needed to be produced to go to an obedient manufacturing plant and that's why we designed it in that way and it worked for a world that has come and gone. So now we're reinventing something again and expertise gets commoditized. It's an entirely different era that we're stepping into. And then even for the role of the doctor, it becomes something else. So an AI system that can do the diagnostics in some ways then the nurse does the care. So nurse becomes this new ecosystem. And then what happens to the physician? The competition, the intellectual competition I think moves more upstream. So it's no longer about memorizing what's in the textbook. And I know that they do a lot more than that. But it becomes an intersection with AI and synthetic biology and all of these new engineering disciplines that are going to converge in robotics. So it's not that the physician isn't relevant. To become a physician or whatever we'll call that job will require profoundly more intellectual capacity in a world of these systems and everything just moves upstream.

完全同意。这对每个 AI 用户来说都是创造性的挑战。目前我们都痴迷于我和其他大公司制造这些东西。但这很快就会过去，因为我们现在并不真正重要。下一阶段将是人们如何在实践中使用它、拥有它、并将其融入他们的世界。如果你在医疗、教育或任何领域拥有专业知识，现在是我们一生中最具创造力的时刻之一，因为我们刚才描述的是将整个系统颠倒过来，下一阶段会是什么样子非常不确定。它肯定不会由我和其他技术人员来设计。我们只是在创造通用工具。我们在流式传输智能。我们在创造火和电。现在我认为人们终于开始意识到，实际上将由其他人来决定边界和限制、护栏是什么，以及它如何融入这些新系统。

Totally. And that's the creative challenge for everybody who is a user of AI. Like we're all obsessed at the moment with me and the other big companies that are making these things. But that's going to pass quite quickly because we're not really relevant right now. The next phase is going to be how people use it in practice and take ownership of it, integrate into their worlds. If you have a domain expertise in healthcare or in education or in whatever it is that you do, it is now one of the most creative times that any of us have ever been alive because what we've just described is like flipping the entire system on its head and it's very unclear what the next phase is going to look like. It certainly isn't going to be me and the other tech people that design that. We're just creating the general purpose tools. We're streaming the intelligence. We're creating the fire and the electricity that's out there. And now I think people are finally getting their head around the fact that it's actually going to be on everybody else to decide what the boundaries and limitations are, what the guardrails are, and how it fits in these new systems.

是的，没错。印刷机最初是为了印制更多圣经，现在看看我们走到了哪里。结果如何？一个全新的世界诞生了。地球上只有少数人像你一样，正在用我们这一代最重要的技术之一改变世界。

Yeah, exactly. I mean, the printing press was to create more Bibles and now look where we are. How did that turn out? An entirely new world was born. There are a handful of people on Earth that are in your position right now transforming the world with one of the most important technologies of our generation.

你希望自己的遗产是什么？

What do you want your legacy to be?

哦，老兄，我没想到你会问这个。我不知道。我想如果我有机会把默认轨迹从为发明而发明，转向发明一种人本主义的超级智能——一种真正与我们的利益对齐、让人类保持在食物链顶端、始终服务于我们、并为我们集体工作的智能——那么，也许我会对此感到欣慰。那可能是我会引以为豪的事。这就是我正在努力做的，因为我认为我们只有一条非常狭窄的道路可走，而有很多方式会出问题，现在我们正在做出许多设计决策，这些决策的影响将持续数十年，我对此感到巨大的责任压力。

Oh man, I didn't realize you were going to go there. I don't know. I think if I have a chance of shifting the default trajectory from invention for its own sake to invention of a humanist superintelligence, one that is truly aligned to our interests, that keeps humans at the top of the food chain, that always serves us and works for us collectively in aggregate, then maybe I'll feel good about that. That's maybe I'll be proud of that. That's what I'm trying to do because I think there's a very narrow path that we have to tread and there are lots of ways that this goes wonky, and right now we're making a lot of the design decisions that have ramifications that will last over many decades, and I just feel a great weight of responsibility for that.

Mustafa，很高兴和你交谈。非常感谢。

Mustafa, it's been a pleasure. Thank you so much.

这太棒了。谢谢你，Chenade。谢谢。

This has been awesome. Thank you, Chenade. Thank you.