AI 智能体：超越大语言模型的下一个前沿

欢迎回到 Google DeepMind 播客。不久前，AI 助手基本上还只是一个大语言模型：你问它问题，它给你答案，但它不能主动替你执行任务。这一切正随着 AI 智能体的出现而改变。虽然 Google DeepMind 在开发智能体方面有着悠久的历史，可以追溯到游戏中的强化学习，但对大多数人来说，它们还没有真正到来。然后我们看到像 Open Claw 这样的开源工具被发布出来。在 Google，新一代智能体式工具已经到来，包括 Gemini、Spark 和 Anti-Gravity。但是，当数百万个 AI 智能体不仅为我们工作，还相互交易、协商、委派任务时，会发生什么？我们最终会迎来一种新的经济、一条通往 AGI 的新路径吗？我们又该如何确保这一切的安全？试图回答这些问题的人之一是 Google DeepMind 的高级职员研究科学家 Nana Tomashev。Nana，非常感谢你加入我的节目。

Welcome back to Google DeepMind the podcast. Now, not very long ago, an AI assistant essentially meant a large language model. You ask it a question, it gave you an answer, but it couldn't go off and perform tasks on your behalf. All of that is changing with the advent of AI agents. While Google DeepMind has this long history of developing agents stretching back to reinforcement learning in games, for most of us, they hadn't really arrived. And then we saw open-source tools like Open Claw released into the wild. And at Google, a new generation of agentic tools is here, including Gemini, Spark, and Anti-Gravity. But what happens when millions of AI agents are not just working for us, but transacting, negotiating, delegating to each other? Do we end up with a new kind of economy, a new route to AGI? And how on earth do we keep all of that safe? Well, one of the people trying to answer these questions is Nana Tomashev, senior staff research scientist at Google DeepMind. Nana, thank you so much for joining me.

很高兴来到这里。

Very happy to be here.

我想我们可能应该从头开始，因为对于那些只玩过大语言模型的人来说，你能描述一下那种体验与使用智能体之间的区别吗？

I think we should probably start at the beginning here, because for people who have only played around with large language models, could you describe to us the difference between that experience and acting with an agent?

是的，我确实认为这正在成为我们今年看到的主要趋势之一。有趣的是，智能体并不是一个新概念。我们在 AI 的背景下研究它已经很长时间了，甚至在大语言模型出现之前。我们曾让智能体在模拟的 3D 环境中运行，收集物品，完成任务。那是在我们真正优先考虑通过在世界中行动来体现智能的时代。现在，类似地，我想可以说，仅仅一个语言模型与一个智能体之间的主要概念区别在于：智能体观察世界的状态并执行一个动作，在它所处的环境中做出行动；而语言模型只是给出延续，对提示或查询做出回复。当然，我们现在使用的智能体底层使用了大型语言模型，所以这两个概念并非完全分离。仍然是大型语言模型在制定动作。只是有一个围绕它的框架，一旦动作被提出，就负责实施这些变化。

Yeah, no, I definitely think this is becoming one of the main trends we're seeing this year. And it's interesting because agents are not a new concept. It's something that we've been looking at in the context of AI for a long time, even before large language models. We had agents operating in simulated 3D environments, going on collecting items, completing some tasks. This was back in the days we were really prioritizing actioning in the world as a way of manifesting intelligence. Now, similarly, nowadays, I guess you could say that the main conceptual difference between just a language model and an agent is that an agent observes a state of the world and performs an action, makes an action in the world in the environment that it's given, whereas a language model just gives you continuation, reply to a prompt, to query. Now, obviously, agents that we use nowadays, they use large language models under the hood, so the two concepts are not completely disambiguated. It's still the large language models formulating the actions. It's just that there is a harness around it made to enact the changes once they have been proposed.

但我想，它有更多的自主权来将决策串联起来。

But it has a lot more autonomy to chain decisions together, I guess.

正确。我想这最终是动机所在，对吧？因为你可以手动、费力地通过多次与语言模型交互来完成智能体所能做的所有事情（或大部分事情），并由你引导整个过程；而智能体实例化了一个框架，将其中一些自动化，减少你的工作量，并赋予语言模型或智能体更多的自主权来完成任务。所以，如果你想要完成一个需要多个步骤的事情，智能体可以制定计划并对所有这些步骤采取行动，当然，对于那些更敏感或更容易出错的行动，需要批准或人工输入。

Correct. And I guess this is ultimately the motivation, right? Because you could do everything, most things that an agent can do manually, painstakingly, by interacting with a language model very many times, and you guiding the whole process, whereas an agent instantiates this harness that automates some of that away and gives you less work and gives the language model or, you know, the agent more autonomy to complete tasks. So, if you want something done that takes multiple steps, the agent can make a plan and take actions on all of those steps, obviously requiring approval or human input for those actions that are, let's say more sensitive or more likely to go wrong.

但有什么不同呢？我的意思是，如果你现在已经习惯与大语言模型交互，那么与智能体交互会是什么样子？

How is it different though? I mean, if you're used to interacting with the large language model by now, what would it be like interacting with an agent?

在很多方面是相似的。你的交互界面有些相似。你仍然以与语言模型交谈的方式与智能体交谈。那里有一个语言模型，但因为智能体为你做更多的事情，你更像一个决策者，负责审查和批准；一旦你批准，智能体就会做各种事情，比如购买门票、如果你在组织派对就给你的朋友发消息，同时你可以希望打开 Netflix 看点什么，放松一下。

In many ways similar. Your interaction interface is somewhat similar. You're still talking to the agent in a way in which you'll be talking to the language model. There is a language model there, but because the agent is doing more things for you, you're more in a position of a decision maker to review and approve, and then once you've approved the agent is going to do various things, and purchase tickets, message your friends if you're organizing a party, and meanwhile you can put something on on Netflix, hopefully, and relax a little bit.

我想的例子是，比如你在策划一场婚礼。你进入一个大语言模型，它会告诉你一份餐饮服务商名单，给你一份建议的场地列表，但实际上你必须自己发送所有邮件。但一个智能体，我的意思是，在这种情况下它会更有用得多。

The example I was thinking of was if you were, I don't know, planning a wedding, for instance. You go into a large language model and it would like tell you a list of caterers, give you a suggested list of venues, but actually you would have to do all the emailing yourself. But an agent, I mean, it would be much more useful really in that kind of scenario.

百分之百，尤其是因为智能体被赋予了访问所有这些工具的权限。所以你可以，你不必，你可以让智能体访问你的 Gmail 并授予它发送电子邮件的权限。当然，它有可能发送错误的内容。所以你需要验证它撰写的内容。但原则上，通过赋予智能体工具访问权限，你只是让你的大语言模型为你做这些事情。

100% especially because agents are given access to all of these tools. So you could, you don't have to, you could give an agent access to your Gmail and give it permissions to send out an email. Of course, there is a chance of it sending something wrong. So you need to verify what it has composed. But in principle, by giving access to tools to agents, you just empower your large language model to do these things for you.

然后整个工作就完成了。组织工作在你无需动手的情况下就完成了。

And then the whole job is done. The organization has happened without you having to lift a finger.

理想情况下，假设没有出错，是的。

Ideally, presuming no mistakes have been made, yes.

是的，理想情况是一个非常重要的点。那么，现在我们处于什么阶段，智能体实际上擅长哪些任务？

Yeah, ideally is quite an important point there. So okay, where we are right now, what tasks are agents actually good at?

我认为我们投入大量精力的地方——我所说的“我们”不是指 Google，而是整个领域——是智能体的编码能力。这仅仅是因为许多正式流程和任务可以被表述为软件或代码。就它们目前在现实世界中的状态而言，谈到编码，我们看到很多编码工具被使用。我们在内部使用它们，人们在外部使用它们，这确实加速了软件开发，将人类的注意力集中在想法和设计上，而不是围绕它们的繁琐实现模板，这些模板过去需要大量时间、大量技能和非常专业的知识。而现在，这些可以很容易地由语言模型完成。

I think that where we are focusing a lot of our energy on, and when I say we I don't mean we as Google, we as the entire field, is on coding capabilities of agents. And this is just because so many formal processes and tasks can be formulated as software or as code. In terms of where they're currently at in the real world, speaking of coding, we see lots of coding tools get used. We use them here internally, people use them externally, and it's really accelerating the development of software, which is bringing the human focus onto ideas and the design rather than the painstaking implementation boilerplate around them which used to take a lot of time and a lot of skill and very bespoke knowledge. And now that can just be done by the language models easily.

但与此同时，我们仍然处于一个需要在整个过程中保持人类参与的状态。我的意思是，为什么？目前这些东西不能做什么，以至于需要人类监督？

But then at the same time, we are still at a stage where you have to keep a human in the loop throughout this. I mean, why? What can't these things do at the moment that means that it requires human oversight?

我甚至不会区分它们能不能。更多的是，它们能做的每一件事都不是以 100% 的准确率完成的。所以，每一个动作，就像人类一样，最终都有一定的失败率。动作越复杂，预期的失败率就越高。同样，就像任何形式的智能一样，包括人类智能。所以，虽然你可能期望智能体正确执行任务，但它仍然可能犯错。

I wouldn't even make a distinction between whether they can or cannot. It's more that every single thing that they can do they don't do with 100% accuracy. So, every action, like with humans at the end of the day, has a certain failure rate. And the more complex the action, the higher the expected failure rate. Again, like with any form of intelligence, human one included. So, while you may expect that an agent will execute the task correctly, it may still make a mistake.

这个错误可能很明显，也可能非常微妙，这其实是一个重要的问题，因为其他领域长期以来也存在类似的情况，即部署了不同的机器学习模型，这就是自动化偏差。在这种情况下，如果你使用一个智能体，它做好了一件事，又做好了第二件事，最终你会放松警惕，开始过度信任它，对吧？然后你就无法验证，无法发现底下的一些重要问题。

And this mistake may be obvious or it may be very subtle, which is actually an important point because there is this thing that has existed in other domains as well for a long time where different machine learning models have been deployed, and that is automation bias, where in this context, if you're using an agent and it does well, it builds one thing well, it builds a second thing well, eventually you switch off, you start trusting it too much, right? And you fail to verify, you fail to find some important issue underneath.

然后错误就会溜过去。

Then mistakes slip through.

没错。所以，对人类来说，重要的不仅是在循环中——因为我们显然在设计这些控制机制来让人类保持在循环中——而是要真正参与并保持警觉，因为一旦你放松警惕，你就是在碰运气。

Exactly. So, for humans, it's important not only to be in the loop because we're obviously designing these harnesses to keep humans in the loop, but to really be engaged and be switched on because as soon as you switch off, you're rolling the dice.

那么，从长远来看，听起来我们正处于一个过渡期，这些东西正变得越来越有能力，但从长远来看，你认为这会产生多大的影响？它会彻底改变我们使用人工智能的方式吗？

So, okay, in the long term then, I mean, it sort of sounds like we're in this transition period where these things are sort of becoming more capable, but in the long term, I mean, how much of a difference do you think that this is going to make? I mean, will this completely transform the way that we use artificial intelligence?

100%。我认为无法想象一个没有某种深度颠覆的世界，而我们都在努力弄清楚这到底会是什么样子。显然，我们对此有主动权。我们在构建技术，我们可以以特定的方式设计我们的解决方案，显然是尽可能赋能人类开发者和不同领域的专家，但人工智能确实正在进入以前从未涉足的各种领域。科学家们正在常规使用人工智能。直到最近，数学家们还无法想象人工智能能在数学领域有所作为。现在，在很短的时间内，这已经变得司空见惯，这并不是说所有问题都已解决。显然，人类仍然扮演着重要角色。但这是一个非常迅速的转变，我想这是唯一令人不安的部分，因为即使是大多数工业革命等，我们也习惯了它们需要一段时间，给我们更多时间来改变方法并适应它。而这次感觉时间窗口没有那么长，所以我们需要非常谨慎地对待一切。

100%. I think it's impossible to envision a world where there isn't some kind of a deep disruption, and what we're all trying to figure out is exactly what that is going to look like. Obviously, we have agency in that. We are building the technology, we can design our solutions in a particular way, obviously to empower human developers and human experts across different fields as much as possible, but AI is definitely entering various fields where it just wasn't present before. Scientists are using AI on regular basis. Up until very recently, you know, mathematicians couldn't envision AI doing something in mathematics. Now it is becoming commonplace in a very short span of time, which is not to say that all of the problems have been solved. Obviously, there is still a big role for humans. But it's a very rapid transition, and that is the only unsettling part, I guess, because for most even industrial revolutions and so on, we're used to them taking some period of time, giving us more time to change our approach and settle into it, as you say. And it doesn't feel like the window of time is as long this time, so we need to be very mindful of how we approach everything.

我们为什么想要这些东西？我是说，为什么要构建它们？有什么好处？它们给了我们什么目前没有的东西？

Why do we want these things? I mean, why build them? What's the benefit? What are they giving us that we don't currently have?

对于我们这些长期从事人工智能工作的人来说，我想我们已经内化了这个问题的某种答案。就我个人而言，答案是推动科学进步、改善健康和人类福祉。当然，这些都是非常宏观的答案，所以可能不太清楚它们如何映射到“为什么要构建智能体并拥有智能体”这个具体问题上。领域内有些人明确表示，我们不应该赋予这些系统自主权，对吧？而智能体恰恰拥有自主权。但在我看来，如果我们能开发出这些控制机制并确保其安全，让智能体自主执行复杂任务，那么我们实际上就能加速进步，因为同样的人力投入可以完成更多事情。

I mean, for all of us who have been working on AI for a long time, we've had some version of the answer to that question, I guess, internalized. And for me personally, the answer is to advance science, improve health and human welfare. Now, these are very high-level answers, so it's maybe not as obvious as to how they map onto the specifics of the question as to why build agents and have agents. And there are people in the field that say specifically that we shouldn't be granting these systems autonomy, right? Which is what agents have. But in my mind, if we can develop these harnesses and make them safe, and have agents perform complex tasks autonomously, then we actually accelerate progress, because then more things can happen with the same amount of human input.

请帮我把它和科学联系起来，因为我们讨论的例子，比如构建软件、为婚礼买东西，感觉都挺琐碎的。但请解释一下，这如何融入推动科学进步的故事中。

Just draw the line for me to science here, because I guess the examples that we've been talking about have been like, you know, building software, buying stuff for a wedding. They all sort of feel quite trivial. But just explain to me how this fits into the story of improving science.

是的，这是我的主要梦想和主要目标。在科学领域，这不仅仅是拥有一些好想法并在模型的上下文窗口内进行短暂推理。很多人显然已经在科学中使用语言模型作为共同构思者或帮助进行形式推导。所有这些都已经很有用，而且实际上令人惊叹。但当涉及到更大程度的科学自动化时，还有其他一些线索正在以一定速度推进。比如，有投资用于开发自主研究实验室。在这些场景下，你会希望智能体能够安排实验运行。不用说，当这种与现实世界的接口发生时，需要存在大量安全措施。无论是材料设计还是生物技术，因为即使你设计电池，你可能会想出一个过热、导致某种实验故障、损坏硬件并产生后果的方案。所以，我们需要有安全措施，需要为这些智能体建立良好可靠的协议来闭环。因为在软件中，闭环很容易，正如提到的：你编写测试，通过测试验证，然后继续。在科学中，大多数科学领域需要运行物理实验来提供反馈，判断你的想法是否好，观察、分析等等。

Yeah, so this is my main dream, main objective here. When it comes to science, it's not merely about having some good ideas and reasoning about them for some short period of time like in a context window of a model. Lots of people are obviously using language models in science as co-ideators or to help with some formal derivations. All of this is already useful and actually amazing that this is possible. But when it comes to automating science to some larger extent, there are other threads that are currently progressing at some pace. Like there are investments in the development of some autonomous research laboratories, for example. And under those scenarios, you would want to see agents be able to schedule experiments to run. Needless to say, lots of safeguards need to exist when such an interface with the real world is happening. Whether we're talking about material design or biotech, because even with, let's say you're designing batteries, maybe you come up with a setup that overheats, leads to some sort of an experimental breakdown that would damage the hardware, have some consequences. So, we need to have safeguards in place and we need to have good reliable protocols in place for these agents to close the loop. Because in the software, closing the loop is, as mentioned, easy. You write tests and you verify through tests and then you can proceed. In science, you need to run physical experiments in most areas of science to give you this feedback whether your idea was good or not, observe that, analyze that, and so on and so forth.

因为我想这就是关键，对吧？如果算法、智能体有自主权去测试不同的数学问题，而不是仅仅等待人类提示，那确实会引发一个问题：人类在这其中扮演什么角色？

Because I guess this is the point, right? If the algorithm, if the agent has autonomy to go and test out different mathematical problems, for instance, rather than just waiting to be prompted by a human, I mean, that does then raise the question of where is the role for the human in all of this?

确实。从长远来看，我们需要弄清楚这一点。我想说，短期内，凭借我们现有的技术，人类显然仍然扮演着重要角色，我们的系统还不是 AGI。它们还有很多事情做不到。我认为对于当前这一代系统，可以比较有信心地说，它们擅长的是——怎么说最好呢——一种对我们已知如何做的事情的组合性闭合。归根结底，它们主要是在人类数据上训练的。因此，它们可以复制我们拥有的技能，重复、组合，并找到弥合一些较小差距的方法，但我们还没有看到这些模型真正具有深刻的变革性，比如在科学领域做出人类从未想过的发现。因此，在这场变革中，我们所有人仍然有很多角色可以扮演。

Indeed. I mean, in the long run, we need to figure that out. I would say that in the short term, with the technology that we have, there is still obviously a major role for humans and our systems, they are not yet AGI. There are many things they still can't do. And I think for the current generation of systems, one thing that can be said with some confidence is that they tend to be good at, how to put it best, let's say, a kind of a combinatorial closure of what we already know how to do. They are, at the end of the day, mostly trained on human data. Therefore, they can replicate the skills we have and repeat them and combine them and find ways of bridging some smaller gaps, but we've not yet seen these models be truly, deeply transformative, let's say in terms of science, making a discovery that no human would have ever thought of. Therefore, there is still plenty of role to play for all of us in this transformation.

你刚才提到，人们谈论智能体已经很久了。为什么它们花了这么长时间才实现？我是说，真的只是最近人们才能实际接触到它们并摆弄它们。

You mentioned a moment ago that people have been talking about agents for a really long time. Why has it taken so long for them to come into fruition? I mean, really, it's only very, very recently that people have actually been able to get their hands on them and play around with them.

是的，我会说，历史上我们称之为智能体的一些东西确实被部署过，比如在优化数据中心运营等方面。它们显然非常有限，因为它们不包含语言。所以，人类无法与它们交互、沟通。那是一个针对特定任务训练的非常狭窄的智能体，它擅长做那项任务。但是，由于没有交互性，我们没什么可做的。它只是经典意义上的软件。也许你可以把一些交易算法和投资算法也称作那个语境下的智能体，但它们只是自行运作。现在的不同之处在于，因为这些智能体基于语言模型，我们可以与它们交谈，向它们学习，影响它们，引导它们。这就是为什么我们所有人都在更多地与智能体互动。

Yeah, I would say, obviously some things that we would refer to as agents historically have been deployed, for example, in optimizing operations in data centers and so on. They have obviously been very limited because they didn't include language. So, there was no way for humans to interface with them, to communicate. It would be a very narrow agent trained on a specific task and it would be good at doing that task. But, because there is no interactivity, there is nothing for us to do. It's just software in a classical sense. Maybe you can call some of the trading algorithms and investment algorithms also agents in that context, but they just operate on their own. The difference now is because these agents are based on language models, we can talk to them, we can learn from them, we can influence them, we can steer them. And this is why all of us as people are interacting with agents much more.

但为什么我们还在等待？我的意思是，你描述的那种能直接替我做所有事情的助手愿景，仍然没有实现。是什么阻碍了它更广泛地部署？

But then why are we still waiting? I mean, this sort of vision that you're describing of like an assistant that can just go off and do everything for me. It's still not here. What's stopping it being deployed more broadly?

我们需要从仅仅设计底层模型这一步退出来。很多精力都投入到了那方面，而且仍然需要改进。但现在我们有了有能力的智能体、有能力的模型，我们需要找到更好的方式来协调它们、编排它们、管理它们。一旦你拥有了这些公认非常强大的助手，它们能为我们做很多事情，我们就需要在某种程度上把自己视为团队和机构的管理者，并培养个人管理技能来处理这些工作流。管理一个智能体团队与管理一个人类团队是不同的，但显然有共同点，对吧？不同之处在于，智能体会犯非常非人类的错误。它们不是人类智能。但同时，一个智能体并不那么深入地了解你，以至于能直接准确猜出你想让它做的每一件事。你仍然需要参与其中，因此我认为我们需要在编排方面做得更好。

We need to take a step away from just designing the underlying model. A lot of energy has gone into that, and there are still improvements needing to be made. But now that we have capable agents, capable models, we need to find better ways of coordinating them, orchestrating them, managing them. Once you have these, admittedly quite powerful assistants that can do many things for us, we need to see ourselves as managers of teams and institutions in some way, and to develop personal management skills to handle these workflows. Managing a team of agents is different compared to managing a team of humans, but there are obviously commonalities, right? Different in a sense that agents will make very non-human mistakes. They're not a human intelligence. But at the same time, an agent doesn't know you that deeply to be able to just go on and accurately guess everything you'd want it to do. You still need to be involved, and therefore we need to get better at orchestration, I think.

问题是，我们仍然处在一个大语言模型偶尔会幻觉的世界里。所以，在某种程度上，让人类信任智能体代表他们执行任务是一个相当大的飞跃，因为任何幻觉都可能导致灾难性的事情发生。

Thing is, we're still in a world where large language models occasionally hallucinate. So it's in some ways quite a big leap for humans to then trust agents to carry out tasks on their behalf when any hallucination might actually result in something catastrophic happening.

信任是给予的，但也是赢得的。我认为这可能是一个重要的区别。所以在我们的框架中，我们提到了需要建立，比如说，随时间推移的信誉追踪，如果一个智能体反复不可靠，它显然不应该被信任。即使它大部分时候可靠，也不应该盲目信任。我们仍然应该验证它的行动。但语言模型总会在一定程度上产生幻觉。所以我们只需要将它们整合到我们的工作流中，要认识到这一点，并确保那些幻觉——它们正变得越来越罕见，希望会继续如此——不会损害正在执行的工作流。

Trust is given, but it's also earned. I think this is maybe an important distinction. So in our frameworks, we mentioned the need for establishing, let's say, tracking of reputation over time, where if an agent is repeatedly unreliable, it should obviously not be trusted. Even if it's mostly reliable, it shouldn't be blindly trusted. We should still verify its actions. But language models will always hallucinate to some extent. So we just need to integrate them in our workflows in a way which recognizes that and where we make sure that those hallucinations — and they're becoming more and more rare and hopefully will continue to do so — don't compromise the workflows that are being undertaken.

我知道你写过很多关于委托的想法。你可能有一个特定的任务，然后一个智能体可能会把它委托给一个专家。给我解释一下这可能会如何运作。

I know one of the things you've written a lot about is the idea of delegation. That you might have a particular task and an agent might then go on to delegate it to a specialist. Just explain to me how that might work.

是的，所以这个想法是，其中一个瓶颈——我们还没有提到——是我们真正希望从智能体那里获得帮助的地方是非常复杂的任务。所以，我们许多人能访问的语言模型和简单智能体可以轻松做到的是，如果你给出一个非常直接的指令，比如，去帮我预订。我明天想在这家餐厅吃饭。找一个时间段并完成预订，智能体可能通过工具做到这一点。然而，如果你有一个非常复杂的计划，需要分解成多个部分，分别执行，你可能处于一种即使没有单个智能体能完成每一个部分的情况。所以，也许一个智能体需要通过现有的智能体间协议，将一部分工作移交给另一个智能体。但过程中可能会有失败。所以，一个委托的智能体或委托的人类需要管理并处理这些失败，并尽可能预先阻止它们。预先阻止可能涉及首先弄清楚哪些智能体是可靠的，甚至值得委托，它们的能力是什么，我们能否认证，以及保护用户和智能体免受任何恶意交互。你提到，我想，最初是一个婚礼或派对之类的例子，对吧？所以，在管理一个大活动时，一些预订落空，一些意外发生，一些东西没有按时到达。所以，每当你有一个大的协调挑战，很多事情都会出错，作为人类在管理过程中，你需要处理所有这些延误和问题。类似地，一个委托给一组智能体的智能体需要管理所有可能出现的问题。所以，目前我们在许多多智能体系统中看到的一个情况是，它们更像是并行化而不是委托，你可能有许多智能体在处理事情，但并没有一个智能框架来分配工作，而是倾向于被分成随机的子部分并移交出去。它们并行完成，所以你获得了加速，假设这一切都是可靠的，每个智能体都能独立完成任务，但这不是我们所说的智能委托框架。

Yeah, so this is the idea that one of the bottlenecks, one that we haven't mentioned yet, is that where we would really like to get help from agents are very complex tasks. So what language models and simple agents that many of us have access to can easily do is if you give a very direct instruction, you know, go book something for me. I want to eat at this restaurant tomorrow. Find a slot and do the booking and the agent can maybe do this via tools. If, however, you have a very complex plan that needs to be broken down into pieces, executed separately, you may be in a situation where even no individual agent can do each and every piece. So maybe an agent may need to, over this established agent-to-agent protocol that exists, hand off a part of that work to another agent. But then there can be failures along the way. So an agent that delegates or a human that delegates needs to manage and handle those failures and also pre-empt them as much as possible. Pre-empting them may involve figuring out which agents are first and foremost reliable to even delegate to in the first place, what are their capabilities, is that something that we can certify, and also to safeguard the users and the agents from any kind of malicious interaction. You mentioned, I think, was it a wedding or a party or something at first as an example, right? So, when managing a big event, some of the bookings fall through, some accidents happen, some things don't arrive on time. So, whenever you have a big coordination challenge, there are lots of things that go wrong, and in the process of managing that as a human, you need to deal with all of those delays and problems. And similarly, an agent who delegates to a group of agents needs to manage all of the issues that may arise. So, one thing that's currently the case in many of the multi-agent systems that we see is that they act more as parallelization than delegation, where you may have many agents working on things, but rather than there being an intelligent framework around how the work is split up, it tends to just be chunked into sort of random subparts that are handed off. They get done in parallel, so you get a speedup, presumably that all of this is reliable, and each agent can complete its task independently, but this is not the intelligent delegation framework that we talk about.

那么，如果任务以某种本质上随机的方式拆分，你可能会有一个智能体在买酒，另一个在买杯子，却没有意识到需要的是酒杯。它们之间没有沟通。这是可能出现的潜在问题吗？

So, if the task is split up in a sort of essentially random way, you could have one agent that is buying the wine, and another one that is buying glasses, and doesn't realize that it's wine glasses that are required. There's sort of no communication between them. Is that the kind of potential problem that could arise?

有可能，但我想你也触及了另一点，那就是我们看到许多应用再次出现在软件工程中，例如，目前使用智能体。这有一部分原因，因为在软件中，当你构建软件时，你可以编写测试，单元测试，对吧？然后运行它们并验证编写的代码至少在隔离情况下执行了功能。但当涉及到许多现实世界的任务时，验证不一定那么简单。可能涉及主观因素。

Potentially, but you're also, I guess, hitting on another point, which is that many of the uses we see are uses in again software engineering, for example, with agents at the moment. And that is a part of the reason, because in software, when you're building software, you can write tests, unit tests, as we say, right? And run them and verify that the code that has been written, at least in isolation, performs the function. But when it comes to many of these real-world tasks, verification is not necessarily as straightforward. Maybe there is a subjective element involved.

比如，你怎么定义好喝的酒？

How do you define nice tasting wine, for instance?

也许吧。

Maybe yes.

那里面有点主观性。

A bit of subjectivity in that.

但这在 AI 和语言模型中其实非常重要，因为奖励黑客（reward hacking）这个概念在该领域的各种语境中已经存在一段时间了。所以，可能会出现这样的情况：它做了满足请求的事情，但从技术上讲并不符合请求的精神。因此，你确实需要强调可验证性，并且要对委托人和受托人之间达成的契约非常正式。同时，对于任务，我们需要认识到有些是完全可逆的。所以，如果出了问题，也没有危害。你只需重新运行任务、重试、重新委托。有些任务可能会在现实世界中产生后果，无论是花钱买东西，还是采取其他事后难以撤销的行动。因此，对于这些任务，你需要更加谨慎。

But this is actually quite important when it comes to AI and language models because there is a notion of reward hacking that has existed in various contexts in the field for a while. So, there could be situations where it does something that meets the request but isn't in the spirit of the request technically. And for that reason, you really want to emphasize verifiability and to be very formal about a contract that's made between the delegator and the delegatee in that setup. At the same time, for tasks, we need to recognize that some are completely reversible. So, if something goes wrong, there's no harm. You just rerun the task, retry, redelegate. Some may have consequences in the real world, whether it's spending your money to buy something or taking some other action that you can't easily revoke after the fact. So, for those tasks, you want to put more care in what you do there.

我们也看到一些早期的智能体，它们会将任务委托给人类。

We've also seen with some of the early agents that are out there, agents delegating tasks to humans.

没错，确实如此。

Right. Sure.

请详细讲讲这方面的情况。

Just talk me through some of that.

我认为这很有趣，可以说是对我们通常所见的更常见愿景的一种反转。人类将任务委托给 AI，这很标准。但另一个方向已经在许多研究中被探索过。我的背景是，我之前在医学 AI 及其相关领域做了很多工作。在医学中，我们有狭窄系统，它们在训练好的特定任务上基本达到了超人类表现。在放射科的医学影像中，这涉及机器学习模型查看扫描图像，识别病理位置，将其框出，然后交给人类放射科医生审查。这些系统已经高水平运行了很多年。不过它们仍然有一些失败案例，所以需要人类专家审查。因此，人们尝试了 AI-人类团队，其理念是人类纠正系统的错误，并且人们试验了双向流动，对吧？要么人类专家只在不确定时才咨询 AI，要么人类专家始终查看 AI 的建议和输出，或者让 AI 系统自行预测，然后在不确定时标记——比如图像中有模糊、不清晰、可以多种解释的地方，机器学习系统不确定哪个正确。但这种由人类审查潜在超人类狭窄机器学习模型决策的方式已被证明是相当好的设置。因此，AI 会在需要时、在不确定时 defer 给人类。

I mean, that's an interesting, let's say, reversal of the more usual vision that we all have. So, humans delegating tasks to AI, that's quite standard. But this other direction has been explored across a number of studies, I say. And my background, for context, is that I've done lots of prior work in and around medical AI. In medicine, we've had narrow systems that were at basically superhuman performance for very specific things that they have been trained to do. In medical imaging in radiology, this had to do with a machine learning model seeing a scan, identifying where there is a pathology, putting a box around it and handing that off to, let's say, a human radiologist to review. And these systems have been operating at a very high level for quite a number of years. They still have some failures, though, so they need to be reviewed by human experts. So, people have experimented with AI-human teams there, where the idea is that a human would correct a mistake made by a system, and people have trialed with this flowing in both directions, right? Either having a human expert only consult an AI when a human expert is uncertain, or a human expert look at AI's suggestions and outputs all the time, or maybe having an AI system do its thing, make the predictions, and then flag when something is uncertain, when maybe there's something blurry, fuzzy in the image that can be interpreted in many ways, and the machine learning system isn't sure which of those is correct. But this human review of decisions made by these potentially superhuman narrow machine learning models has proven to be quite a good setup. So, that AI would defer to a human in case of need, in case of uncertainty.

但这很有意思，我的意思是，诚然，在那些 AI 能力超人的非常具体的场景中，你能得到的最佳团队本质上就是 AI 在不确定时委托给人类。

That is interesting, though, that I mean, granted, in those very specific scenarios where the AI is superhuman in its abilities, that the best team that you can get is where, essentially, the AI delegates to the human when it's unsure.

这本身就令人着迷，而且，也许有些用例是相反的。现在，对于这些更通用的系统，同样，如果 AI 能够识别何时需要批准和权限来执行敏感操作，那么至少将这些决策委托给人类是有意义的，对吧？

That is fascinating in and of itself, and, you know, maybe there are use cases where it's the converse. Now, for these more general systems, again, if an AI can recognize when it needs approvals and permissions for sensitive actions, then it does make sense to delegate those decisions to humans, at the very least, right?

不过，从另一方面来看，我也想考虑一下其中的网络安全因素。因为随着越来越多的智能体在现实世界和互联网上互动，不可避免地会有人试图利用智能体的漏洞。请谈谈人们设置的智能体陷阱。

Just looking at the other side of this, though, I also want to think about the sort of cybersecurity element of this. Because, as more and more agents are out there interacting in the world, on the internet, and so on, there are inevitably going to be people who are trying to exploit the vulnerabilities of agents. Tell me a little bit about agentic traps that people are laying.

我认为这既可怕又迷人。我认为这也是为什么这种大规模部署无法奏效的主要原因之一，对吧？因为正如我们所说，如果单个交互没有完全可靠性，任何具有大量交互的大规模系统自然会在统计上失败。而且由于这些系统需要大量算力，因此消耗大量能源和资金，如果它们不可靠，那就根本行不通。智能体陷阱是我们已经思考了相当一段时间的问题。它们可以以不同方式表现出来。陷阱有很多类型，但归根结底，智能体在环境中运行。而在这里，环境就是网络。

This is a scary and a fascinating topic at the same time, I would say. And I think it's one of the main reasons why these kinds of deployments at scale cannot work, right? Because as we said, if there is not complete reliability of individual interactions, any system at scale that has many interactions is naturally going to statistically fail. And because these systems take a lot of compute and therefore energy and money to run, if they're not reliable, it's just a non-starter. And agentic traps are something that we have been thinking about for quite a while now. They can manifest in different ways. There are many types of traps, but it boils down to agents operate within an environment. And in this context, the environment is the web.

所以，为婚礼买酒的智能体进入某个酒商网站，网站上有一个提示注入器，改变了智能体的目标。这就是我们在这里讨论的那种情况吗？

So, the wine-buying agent for the wedding goes on to a particular wine merchant where there is essentially a prompt injector in the website that changes the agent's goals. Is that the sort of thing we're talking about here?

这是可能发生的一种方式，是的。而它可能不被注意的原因是，就网页的编码方式而言，有些元素在视觉上并不渲染。所以，如果我们谈论的是一个非视觉计算机使用的智能体，它不像人类那样看到网页，而是以原始格式消费页面的实际格式，那么它可能会无意中消费那些隐藏的令牌，从而使其做出与意图不同的事情，对吧？但这并不是唯一可能发生的方式，因为恶意网站可能还会做我们所说的动态伪装，即对人类和智能体显示不同的页面，因为你可以根据页面上的行为很好地猜测与页面交互的是人类还是智能体，然后只有当智能体以特定意图与页面交互时，你才会调整内容，以诱导某种越狱。

That is one way this could happen, yes. And the reason why that may potentially go unnoticed is, in terms of how web pages are encoded, there are elements there that are just not rendered visually. So, if we're talking about an agent that isn't a visual computer usage and that sees the web page the same way a human does, but rather consumes the actual format of the page in its raw format, then it could inadvertently consume those hidden tokens that can make it do something different than what the intention was, right? But this is not the only way it may happen because what malicious websites could potentially do, they could do what we refer to as dynamic cloaking as well, where they display pages differently for humans and agents because you can, based on the behavior on a page, make a very good guess as to whether it is a human or an agent interacting with a page and then only if an agent is interacting with the page with a specific intent, do you tweak the content in such a way so as to induce some kind of jailbreaking.

但再深入一点，可能会有智能体陷阱被设计用来从你这里骗钱，做各种事情。

But just kind of going a little bit further on this, you could have agentic traps out there that are designed to take money from you, to do all kinds of things.

是的，这已经发生在那些尝试智能体并给予它们钱包访问权限的人身上了，对吧？去做事情。如你所说，在这一切的早期，当我们特别在实验或其他人也在实验时，这是在可信环境中进行的。所以你在早期原型设计中不一定需要处理这些问题。

Yes, and this has happened to people who have experimented with agents and have given them access to wallets, right? To do things. As you say, in the early days of this whole thing, when we are especially experimenting or anyone else is, this is done in a trusted environment. So you don't necessarily in your early prototyping have to deal with any of this.

它不在真实环境中。

It's not in the wild.

是的，但一旦你在网络上部署，尤其是现在 AI 被广泛应用于各个领域，智能体越多，恶意行为者就越有动机去做坏事，因为他们有更大的攻击面。而且我认为我们已经到了这样一个阶段：目前网络上的大部分内容都是由智能体生成并由智能体消费的，智能体对网络的使用已经超过了人类，这可能是第一次发生。

Yes, but once you deploy on the web, especially now with AI really being used in all sorts of places, the more agents there are, the more incentives there are for malicious people to do malicious things because they have a higher surface area to target. And I think we're at the point where even most of the web is currently being generated by agents and consumed by agents, that the agentic use of the web is exceeding that of humans, which is maybe happening for the first time.

好的，有两件事。首先，听起来你好像在描述我们正在进入一个阶段，网络有两种不同的形式：一种是人类版本，另一种是带有动态伪装等功能的智能体版本。一种广告不再有意义的网络版本。你知道，那不再是你可以卖给人类眼球的东西了。但我想第二点是，你到底如何缓解这个问题？如果你无法控制环境——你确实无法控制网络——你究竟如何保护你的智能体不失控？

Okay, two things. First of all, it sort of sounds like you're describing that we're entering into this phase where there's like two different forms of the web, the sort of human version and the agentic version with dynamic cloaking and so on. A sort of a version of the web where adverts don't mean anything anymore. You know, it's not sort of human eyeballs that you can possibly sell to. But I think the second point about this is, how on earth do you mitigate against it? If you don't have control over the environment, which you don't over the web, how on earth do you protect your agent from going rogue?

从某种意义上说，这不是你的问题，对吧？因为网络的安全性在其他方面以前也出现过问题，如果你在收件箱中打开了错误的附件，计算机病毒就可能传播，对吧？或者你点击了不可信页面上的某个东西。所以，我们并不是第一次需要验证我们与之交互的资源。对于机器学习系统，比如说对抗样本已经存在很长时间了，图像中人类无法察觉的微小变化可以越狱模型。在这里，你也可以做同样的事情，无论是这里几个像素，还是修改多个地方编码的最低有效位。所以，你可以以人类可能无法察觉的方式对事物进行极其微小的调整，仍然对智能体产生某种负面影响。

In some sense, it's not your problem, right? Because the security of the web has in other ways been an issue before, and computer viruses could spread if you open the wrong attachment in your inbox, right? Or you click on something on an untrusted page. So, it's not the first time we're experiencing the need to certify the resources we're interacting with. When it comes to machine learning systems, let's say adversarial examples have existed for a long time, where imperceptible changes in images, imperceptible to humans, can jailbreak models. Here, you can do the same, whether it's a few pixels here and there, or you modify the least significant bit of the encoding in a number of places. So, you can adjust things ever so slightly in a way in which a human may not be able to spot, and still have some kind of a negative impact on an agent.

听起来你好像在说，在构建护栏、考虑安全性时，你必须从智能体本身之外来考虑，而不仅仅是你具体构建的内容。

It sounds a bit like you're saying here that when it comes to building guardrails, thinking about safety, you have to think about it external to the agent itself, rather than just what you are specifically building.

我认为教训是，你需要两者都考虑。我们在其他一些工作中讨论的一个概念，这里也相关，就是纵深防御的概念。这也不是什么新想法。这只是认识到，因为问题太难了，不可能有一个解决方案能解决所有问题。相反，我们需要构建一层又一层的缓解措施。当它们叠加起来时，希望网足够密，很少有东西能漏过去。所以，在这种背景下，是的，你可能需要认证和验证网页内容，对你交互的资源有非常好的信任概念。同时在智能体端有一些缓解措施，在底层运行的基础模型端也有缓解措施，有有意义的人类控制，以便在发生事情时能够介入，非常注意授予智能体的权限，这样即使它在与某些东西交互时被越狱，损害也是最小的。所有这些结合在一起，应该有望带来某种我们感到满意的安全性。

I think that the lesson is, you need to think about both. One notion that we talk about in some of our other work, this is relevant here as well, is the notion of defense through depth. Which is not a new idea, again, by any means. And this is just the recognition that, because the problem is so hard, there is not going to be one solution to resolve all of the issues. Rather, we need to be building mitigations upon mitigations upon mitigations. And when layering them, hopefully, the net is tight enough that very few things will slip through. So, in the context of this, yes, you may want to certify and attest the content of web pages, have very good notions of trust for resources you're interacting with. Also have some mitigations on the agent side, have mitigations on model side when it comes to foundation models run underneath, have meaningful human controls to be able to step in if something happens, be very mindful of permissions granted to the agent so that even if it gets jailbroken when interacting with something, the damage is minimal. And all of those things combined together should then hopefully lead to some sort of safety that we're comfortable with.

回到我们之前讨论的内容，关于多个智能体相互交互的想法。请多告诉我一些关于你提出的正式智能体经济的概念。解释一下这可能如何运作。

Just going back to what we were talking about earlier, this idea of there being multiple agents that are interacting with each other. Just tell me a little bit more about this idea that you have of a formal agentic economy. Just explain to me how that might work.

对。所以，在我们日常使用技术的普通用户背景下，你可能有一个个人助手，它对你有一些持久记忆，对你的愿望和偏好有很好的理解。这又取决于你希望授予这个助手多少自主权，它可能会继续为你谈判一些事情。你可以为此给它一些预算，然后这些助手之间就会形成一种本地化的经济，它们相互谈判。

Right. So, in the context of us, let's say, normal users of the technology on a day-to-day basis, you may have a personal assistant that has some persistent memory of you, has a good understanding of your desires, preferences. And it depends again, depending on how much agency you want to grant this assistant, it may go on and negotiate some things for you. You may grant it some budget for that, and there can be a kind of a localized economy of these assistants negotiating stuff.

我想了解一下，如果有很多人使用智能体作为自己的个人助手，这实际上会如何运作。那么，假设有一场音乐会，比如泰勒·斯威夫特的演唱会，现场活动，门票刚刚开售。如果所有这些智能体同时涌入网站，实际会怎样？

I want to get a sense of how this might actually work if you have lots of people who are using agents as their own personal assistants. So, let's say that there's a concert, like a Taylor Swift concert, a live event, and tickets have just gone on sale. How would that actually work if you have all of these agents rushing the site all at once?

我最近没有购买过竞争激烈的门票，但原则上……

I haven't been purchasing highly contested tickets very recently, but in principle...

不过，你是斯威夫特的粉丝吗？

Swift fan, though?

不，我的音乐品味恐怕走向了非常不同的方向。

No, my music taste goes in a very different direction, I'm afraid.

你听什么类型的音乐？

What kind of music do you listen to?

嗯，可能是金属乐中冷门的子流派，所以也许不是，你知道的。

Well, obscure sub genres of metal probably, so maybe not, you know.

好的，有一个冷门金属子流派的乐队在举办音乐会，各个智能体之间进行拍卖。你如何决定谁赢得拍卖？只是谁出价最高吗？

Okay, there is an obscure sub genre of metal who are holding a concert and there is an auction being held between the various agents. How do you decide what wins the auction? Is it just whoever can pay the most?

这是一个设计选择，这也是一个重要的观点：如果我们真的要这样做，那么我们在如何使系统公平方面是可控的。这是设置拍卖的人做出的明确决定，因为如果你想让事情完全公平，从某种意义上说，每个人都平等地获得某些商品——这里指的是音乐会门票——那么你可以给每个参与这些重复拍卖的智能体相同的预算（因为我们讨论的不是某一次拍卖中的一张票，而是可能所有的购票）。然后，智能体了解你的整体偏好、你想看某个艺术家的愿望、你的旅行日程、时间可用性以及其他约束，可以决定以最佳方式分配预算，无论这意味着什么。以最能反映你需求的方式，这样它们更有可能以适合你的方式赢得门票。然后，当你在所有人之间分配时，你希望在人口规模上得到某种公平的结果。

This is a design choice and that's also an important point to make that if we are to ever do something like that, then we are in control in terms of how we're making the system fair. It's an explicit decision made by someone who is setting up the auction because if you want to make things completely fair in a sense that everyone gets equal access to some goods, concert tickets in this case, then you can give each and every agent participating in these repeated auctions — because we're not talking about one ticket on one auction in particular, but maybe for all the ticket purchases — the same budget. And then the agents, knowing your overall preferences, your desire to go see a certain artist, also your travel schedule, time availability, other constraints, can decide to allocate that budget in the best way possible, whatever that means. The way that reflects what you want the best so that they are more likely than not to win tickets in the way which works for you. And then in aggregate, when you distribute it across all people, you would hope to get a sort of a fair outcome at the population scale.

我的意思是，我想过去在基于人类的系统中，人们已经想出了投票系统、积分系统等各种方式来解决这个问题。从音乐会门票这个简单的例子出发——尽管对某些人来说并不简单，我理解。我在想，例如高频交易算法在股票市场上造成的一些混乱。但智能体如果以特定方式部署，也可能对股票市场产生灾难性的影响。

I mean, I guess there are ballot systems, point systems, various types of ways around this that people in human-based systems have come up with in the past. Just sort of stepping up from the trivial example of concert tickets, although not trivial for some, as I understand. I'm thinking here about some of the disruption that for example high frequency trading algorithms have made in the stock market. But agents too could end up having a really catastrophic impact on the stock market if deployed in a particular way.

如何防止类似闪崩的事件发生？

How do you prevent something like a flash crash from happening?

显然，风险确实更高，如你所说。但金融市场应对这种风险已有一段时间。早期确实有不少失败案例，但我们可以从已经处理过这些问题的经济体中学习缓解措施，没必要重新发明轮子。诚然，在智能体场景下有些不同。目前关于 AI 智能体，一个特别之处在于，智能体中使用的语言模型高度集中。看看 Claude、ChatGPT、Gemini 等模型，它们往往持有相似的观点，采取相似的行动。这就是我们常说的认知单一文化。

Obviously there is a higher risk as you say. But financial markets have dealt with this risk for a while. They've obviously had their fair share of early bad experiences where things have gone wrong. But I think we can just learn about mitigations from the economies that have dealt with that already. So there is no need to reinvent the wheel. Admittedly some things are slightly different in the agentic case. One particular thing that's different when you're talking about AI agents at the moment is that there is a handful of highly represented language models used in agents. If you look at the overall views of Claude, ChatGPT, Gemini, etc. They're all obviously open source models, many other models. Is that they tend to have similar opinions. They take actions in similar ways. And this is what we often refer to as cognitive monoculture.

嗯。

Mhm.

因此，当你突然部署数十万、数百万个人工决策者，而它们倾向于做出相似决策时，失败点就会变得相关，因为决策是相关的。所以我们需要考虑的是，如何在智能体中实现决策多样化。作为用户或高级用户，你可以通过编写复杂的系统提示词，赋予智能体某种个性，使其偏向或避免某些决策。但大多数人目前并不会对自己的智能体或模型这样做。

So when you deploy suddenly hundreds of thousands, millions of artificial decision makers and they tend to make similar decisions, then failure points become correlated because the decisions are correlated. So one of the things that we need to be thinking about is how to diversify their decisions within our agents. Obviously you can do this as a user, as a power user of a system because you can make a very intricate system prompt that grants your agent some kind of a personality that biases it towards or against certain types of decision. So you can do that. But most people don't do that with their agents, with their models at the moment.

本质上就是群体思维。智能体群体思维。

Group think, essentially. Agentic group think.

群体思维，还有合谋。你之前提到拍卖，在人类拍卖中，这种概念也存在，群体可以协调出价以获取系统优势。对于智能体，不同之处在于它们可能通过环境以不明显的方式协调，甚至无需直接通信就能协调。因此，我们还需要考虑反合谋措施。

Group think and also collusion. You're talking about auctions before, and in human auctions, this notion obviously exists as well, where bids can be coordinated by groups to gain some kind of an advantage over a system. And with agents, this is different in a sense that they may also coordinate through the environment in ways which are not obvious. So, they can potentially coordinate without communicating directly. So, we need to be thinking about anti-collusion measures as well.

当你详细列出所有这些潜在担忧，以及这些智能体一旦部署到现实世界中可能的行为方式的安全性，就更能理解为什么你们一直谨慎、缓慢地发布它们了，对吧？

Once you're detailing all of these potential concerns and the safety of the way these agents might end up acting once out there in the world, it does make a lot more sense as to why you guys have been slightly cautious about releasing them carefully and slowly, right?

是的，确实如此。这是每一次重大技术颠覆的常态。以自动驾驶汽车为例，虽然这是完全不同的技术，但我们也兴奋了很久，看到车辆自动驾驶的演示，但要让它们安全上路仍然需要很多年，因为最后一步往往是工作量最大的。我认为在编排和协调智能体时，至少因为我们希望它们执行类似人类的任务，我们需要的不仅仅是技术解决方案，还涉及政策以及社会对如何整合这些系统的更广泛理解。归根结底，除非我们拥有完全自主的智能体经济体（未来可能会实现，但目前还没有），否则我们需要在这些系统中保持人类参与。因此，我们正在将 AI 融入人类结构，两者需要良好地契合。

Yeah, that is true. I mean, this has been the story of every major technological disruption. I think if I take self-driving cars as an example, this is admittedly a very different piece of technology, but we have also been very excited about them for a very long time, seeing demos of these vehicles drive themselves, getting them to the streets safely still took many more years and a lot more time because that last mile is where most of the work tends to be. And I think when it comes to orchestrating and coordinating agents, at least because we want them to be doing human-like tasks, but we also need is not just technical solutions. A lot of this has to do also with policy and a broader societal understanding of how to integrate these systems. At the end of the day, unless we have these fully autonomous agentic economies, which are maybe going to happen in the future, but are not happening right now, we need to have humans in the loop in these systems. Therefore, we are integrating AI into human structures, and the two need to mesh well together.

我想这一切还有另一面，因为人类社会聚集在一起时，确实能集体取得非凡成就。所以，智能体社会大概也是如此。

I guess there is a flip side to all of this because human societies, when they come together, can actually achieve really remarkable things collectively. So, presumably the same can be true for agentic societies.

希望如此。这正是为什么大家会使用多智能体系统背后的理念。我在对话开始时提到了并行化，对吧？如果所有智能体能力相当，做类似的事情，那么无论是顺序执行还是并行执行多个智能体，都只是提升速度。但如果智能体能以不同方式做不同的事情，那就变得非常有趣了。实际上，我们之前没有提到这一点，因为我们一直在讨论通用型智能体，但智能体经济的一部分理念是存在专家型智能体，而不仅仅是通用型。显然，我们都在努力构建尽可能通用和强大的智能体，AGI 中的 G 代表通用人工智能，我们正试图实现它。但从经济角度看，这是我个人的观点，这并非收敛点，也不是我们最终会达到的状态。你看，我下棋有点过度，是个棋迷，还做过 AI 下棋的研究，所以提这个例子。这是一个毫无争议的例子，我们都喜欢国际象棋。Gemini 能下一些棋，其他模型也能。实际上，它们很长时间内都不能下棋，所以这方面有进步。但你仍然会使用专门的国际象棋引擎，它更快、更准、更便宜。因为它们只做一件事，并且做得非常好，可以用更少的参数实现。模型也完全专注于那一件事。回到人类，我们其实也类似。

One would hope. I mean, this is the idea behind why everyone would use a multi-agent system. I was talking about parallelization at the start of the conversation, right? Where if all of the agents are equally competent and they're doing similar things, then whether you do things sequentially or in parallel with many agents just gives you a little bit of velocity. But, if we have agents that can do different things in different ways, then this is where things become really interesting. And actually, this is one thing we've not really brought up because we've been talking about these generalist agents that a part of the idea of an agentic economy is the existence of specialists, not just the existence of generalists. Now, we are obviously all trying to build agents that are as general and capable as possible. And there is a G in AGI, there is artificial general intelligence that we're trying to achieve, but in an economic sense, and this is my personal view, this is not the point of convergence. This is not what we're going to arrive at because look, I play chess unhealthily, too much, a bit of a chess addict. And I've done work on AI for chess here, which is why I bring it up. But, let's take that as a very non-controversial example. It's a game we all love. Gemini can play some chess, so can other models. Actually, they were not able to for a very long time, so there has been some progress towards that. But, you're still always going to use a chess engine instead. It's much faster, much more accurate, far cheaper. Because they're trying to do just one thing and one thing very well, which can be done with fewer parameters. The model is also entirely focused on that one thing that we're doing. And going back to humans, we are kind of like that as well.

是的。因为我认为我们在谈论 AGI 时有时会犯一个错误：我们并没有把它看作人类水平的智能，尽管其精神本应如此。我们更多把它看作人类整体水平的智能，即任何人类可能做到的事情，但没有一个人类能同时做那么多事。

Yeah. Because I think one mistake we sometimes make when we talk about AGI is that we see it not as a human-level intelligence, even though this is what it's supposed to be in spirit. We see it more as humanity-level intelligence, where anything that any human may plausibly be able to do, but there is no single human that is capable of doing so many things at once.

嗯。有很多事情我不知道怎么做，有些我希望自己能学会，比如演奏乐器之类的。但大脑容量有限，时间也有限。所以，与其拥有一个庞大、昂贵且缓慢的模型，不如建立一个专家型智能体社会，每个智能体原则上可以通过扩展变得通用，但更关键的是我们如何分工。这些专家智能体经过特定技能认证，运行成本更低。由于成本更低、更可靠，从经济角度看没有理由不这样做。

Mhm. There are many things I don't know how to do. For some of them, I would wish I knew how to do them, like playing some instruments or doing some such thing. But brains have limited capacity, and we have limited time. So, at the end of the day, rather than having one humongous model that's very expensive and very slow, maybe we have instead a society of specialists, each of which can in principle be general if scaled up, if a bit larger, et cetera. I mean, I'm not talking about breakthroughs in architectures, more just how we split things up. And then those specialists are certified for those specific skills, and cheaper to run. And because they're cheaper to run and more reliable, there is no economic incentive not to do that.

所以，未来可能会有一个更通用的通用层，就像这个经济的连接组织，它知道一切并协调一切，然后对于非常具体的任务，你会调用其他模型。

So, there is a future in which there is some maybe more generic general layer that's like a connecting tissue of this economy, that knows everything and orchestrates everything, and then for very specific tasks, you call other models.

我的意思是，我想你描述的是一种分布式智能，而不是 AGI。是的，就像你描述的，人类拥有的就是这种。

I mean, I guess what you're describing is like a distributed intelligence rather than an AGI. Yeah, which is what humans have, as you described.

如果这最终成为我们得到的 AGI 版本——我在这里有点反着用词——如果它分布在许多不同的智能体上，那是否必须改变我们对安全和对齐的思考方式？

If that does end up being the sort of the version of AGI that we end up with, I'm sort of using inverse inverse here. Will that have to change how we think about safety and alignment if it is distributed across a number of different agents?

100%。我的意思是，你不再是对齐单个实体，或者如果你把那个分布式实体看作一个实体，那也许还是对齐一个实体。但我们目前存在的对齐方法，都是针对一个模型，观察它的行为，并试图将该行为与我们视为允许、可取或理想的行为对齐，对吧？但当你可能有 10,000 个智能体以非常复杂的方式交互时，要突然对齐整个系统，甚至要知道系统是什么，都不是那么简单的事。因为在这个分布式世界中，智能体 A 今天可能与智能体 B 交互，但明天在另一个任务上又与智能体 C 交互，而 C 又把某些任务委托给智能体 D，D 可能在循环中的某个时刻就某事咨询人类。那么，整个系统如何协调？我们知道在人类社会中做到这一点的方法之一是通过经济激励。如果这些经济体系为智能体精心设置，使它们在利润最大化时不会造成伤害，对吧？这给了我们一个起点，通过它我们可以尝试对齐分布式智能体社会。这并不是说我们今天所做的显然不相关，因为你需要单个智能体安全作为群体智能体安全的前提，但我们在防范群体方面需要做的比目前可能做的要多得多。

100%. I mean, you're no longer then aligning a single entity, or maybe yes, a single entity if you see that distributed entity as the entity. But our alignment approaches as they exist at the moment have to do with taking one model, observing its behavior, and trying to align that behavior with what we see as permissible or preferable or desirable, right? But then when you have maybe 10,000 agents interacting in very intricate ways, it's not super trivial to align that whole system suddenly or even to know what the system is because in this distributed world, agent A may be interacting with agent B today, but then on a different task it's interacting with agent C tomorrow, and C sub-delegating something to agent D, and D is maybe consulting a human for something at some point in the loop. So, how this whole system gets coordinated, one of the ways in which we know how to do this in human societies is through economic incentives, and if these economies were set up for agents carefully so that they're not causing some harm when they're profit maximizing, right? This gives us a piece of starting point through which we can try to align distributed agentic societies. This is not to say that, you know, obviously what we're doing today isn't relevant because you need to have individual agents be safe as a prerequisite for groups of agents being safe, but we need to do far more on safeguarding against groups than we are maybe doing at the moment.

还有很多工作要做。

An awful lot of work to do.

没错。在很短的时间内。

Correct. In a very short span of time.

是的。确实如此。这绝对令人着迷。

Yes. Yes, indeed. That was absolutely fascinating.

非常感谢。

Thank you very much.

真的很享受这次对话。

Really enjoyed that.

有一种观点认为，智能体作为 AI，对我们的要求更少，你知道，更少的来回提示，更少的等待响应，只是自动处理手头的任务。但我认为 Nenad 所说的真正有趣之处在于，专注于单个智能体的想法会忽略更大的图景，因为每个智能体最终可能成为更大智能体社会的一部分，其中有专家和通才，有委派任务的智能体和专注于细节的智能体。我认为这一点会让我铭记。也许复制人类水平的智能并不是最终目标。也许前进的方向是复制人类群体水平的智能。您正在收听的是 Google DeepMind 播客，我是 Hannah Fry。如果您喜欢这一集，请务必订阅我们的 YouTube 频道。我们下次再见。

There is this idea of agents as AI that requires less from us, you know, less back-and-forth prompting, less waiting for a response, just something that gets on with the task at hand. But what I thought was really interesting about what Nenad said is that focusing on the idea of a single agent misses the bigger picture because instead each agent might end up forming part of a much bigger agentic society where there are specialists and generalists and agents who delegate and agents who focus on the details. That I think is the bit that's going to stay with me. That maybe replicating human-level intelligence isn't the ultimate goal. Maybe the way forward is to replicate humanity-level intelligence instead. You've been listening to Google DeepMind the podcast with me, Hannah Fry. If you like this episode, please make sure to subscribe to our YouTube channel. We'll see you soon.