从语言模型到物理智能体：机器人基础模型的前沿

好的，我们开始吧。大家好，欢迎参加 AI 智能体前沿研讨会。今天我非常荣幸地介绍 Sergey Levine 教授。Sergey 的工作对强化学习、机器人学习以及端到端学习和决策产生了基础性影响。今天他将谈论机器人基础模型。Sergey，再次感谢你的参与。请开始你的演讲。

Okay, maybe let's get started. Hello everyone. Welcome to the AI Agent Frontier Seminar. Today it is my great pleasure to introduce Professor Sergey Levine. Sergey's work has a foundational impact on reinforcement learning, robot learning, and end-to-end learning and decisions. Today he will be talking about robot foundation models. Sergey, thanks again for joining us. The floor is yours.

好的，谢谢介绍。那么，当我准备这次演讲时，我想做个小实验。我去问了一个流行的语言模型，让它帮我解决一些任务。这正是我们希望智能体系统能够做到的——我们能够提示它们解决某个任务，然后它们就去执行。所以我问 Gemini：‘我想让你打扫房子、叠衣服、做晚饭。’不出所料，Gemini 对这个请求很吃力。尽管它确实告诉了我很多关于它认为我应该如何做这些事情的建议。当然，Gemini 做不到是有原因的。但如果我们真的能构建出控制真实物理系统的智能体，那该多好啊——你可以像对人一样提出请求，然后机器人就能在物理世界中实际执行，而不是仅仅给你讲个故事。这正是今天演讲的主题：我们如何从能够通过文本、API 调用等方式满足各种请求的语言模型，过渡到能够在真实物理世界中实际操作的智能体。这不仅仅是一个实际问题。我认为研究与现实物理环境交互的智能体还有一个更深层的科学原因。现实世界非常复杂。想想我们日常做的任务，比如叠刚从烘干机拿出来的衣服，你不仅要弄清楚完成这项任务所需的复杂物理动作，还要考虑多个层次的语义。如果你要把衬衫放进一个小抽屉，你得提前计划，把它们叠成更小的形状，也许还要即兴发挥，想办法让它们变成合适的形状，然后一直细化到手指的动作，才能在物理世界中实现。所以你在多个抽象层次上操作。如果你在做饭，你要考虑单个动作，比如切胡萝卜要用多大的力，同时还要考虑其他抽象层次，比如如何与正式物体交互，记住锅里的水需要什么温度，思考更高层次的概念，比如你可能用错了食材，能否用其他东西替代？多个抽象层次，同时进行语义、视觉、空间推理，一直到低层次的物理动作，而且是实时的。这是一个任务的视频。我稍后会展示机器人做这个任务，但这是一个人在工业环境中执行一个非常简单的任务——组装一个盒子。但你可以看到，如果我们试图在模拟中做这个任务，模拟这样的任务会非常困难。这里有复杂的动作，也有语义问题：标签机在哪里？我该把它贴在盒子的哪一边？所以这里有很多事情。但对我们 AI 研究人员来说，这实际上是一个巨大的机会，因为你的 AI 系统最终有多复杂，很大程度上取决于它学习的环境有多复杂。由于物理世界非常复杂，它给了 AI 系统很多学习的机会，很多需要弄清楚的东西。而且因为物理世界如此丰富，我认为它实际上是我们为 AI 模型提供足够深度以学习复杂理解的最佳机会，这种理解可能再现人类智能的灵活性和复杂性。所以，如果我们回想一下现代 AI 的一个重大成功时刻——AlphaGo，AlphaGo 非常令人印象深刻，因为它展示了涌现行为。它能在高风险的冠军赛中想出人类棋手在整个围棋历史中从未想到的走法。但 AlphaGo 被限制在一个相当小、规则简单且定义明确的环境中。这当然很重要，因为对于当时的技术来说，这非常依赖于模拟大量围棋对局。所以这种涌现行为很有前景，但当涌现行为被限制在规则简单的封闭世界环境中时，系统能做的也就那么多了。比如 AlphaGo 永远学不会做晚饭，无论它下多少盘围棋，因为围棋世界太受限，无法理解那些现象。如果你有一个与物理世界交互并从这种交互中学习的 AI 系统，它不仅仅是在解决一个实用任务，不仅仅是在满足你想吃晚饭的需求然后给你做饭。它是在进行实验，学习物理宇宙。这种对物理宇宙的理解，它进行的那些实验，超越了人们可能先验知道的东西。所以智能体的伟大之处在于它们实际上与环境交互，但当你的智能体在写代码或与网络交互时，它本质上学习的是人造产物——操作系统世界或互联网世界。这些都是人类创造的，它们包含的关于宇宙的任何知识都是因为人类放进去的。而能够与物理世界交互的智能体则拥有整个物理宇宙可以学习，这超越了人类的理解。所以机器人学习不仅仅是构建有用的机器人。它让我们能够在提供现实挑战和现实机遇的领域中研究最棘手的 AI 问题。好了，这有点哲学，但让我们多谈谈技术挑战。

Great. Thank you for the introduction. All right. So, when I was working on this talk, I thought I'd do a little experiment. I went and asked a popular language model that is available on the web to solve some tasks for me. This is what we want agentic systems to do. We want to be able to prompt them to solve some task and we'd like them to go out and do it. So, I went and asked Gemini, 'I'd like you to clean the house, fold the laundry, and make dinner.' Unsurprisingly, Gemini struggled with this request. Although it did tell me quite a lot about how it thinks I should do this. And of course there's a reason why Gemini can't do it. But maybe it would be really nice if we could actually build agents that could control real physical systems so that you could make a request like this just like you would ask a person to do something and you would have a robot that can actually go out and do this in the physical world rather than just telling you a story about it. And this is what today's talk is about. It's how we can go from language models that can fulfill all sorts of different requests by using text, by using API calls, and things like that to agents that can do it physically in the real physical world. And this is more than just a practical thing. There's actually a deeper scientific reason why I think it's very important to study agents that interact with real-world physical environments. So, the real world is very very complicated. If we think about even the tasks that we do on a day-to-day basis, imagine folding laundry when it comes out of the dryer, you have to figure out not just the intricate physical movements to perform this task, but also multiple levels of semantics. You have to plan ahead if you're going to put your shirts away into a small drawer, you're going to fold them into a smaller shape, and you have to maybe improvise, figure out how to get them into the right form factor, and then all the way down to your finger movements that will actually allow you to bring this about in the physical world. So, you're operating at many levels of abstraction. If you're cooking a meal, you have to think about the individual movements, how to apply enough force to cut the carrot, but also thinking about other levels of abstraction, how to interact with the formal objects, remember what temperature the water in the pot needs to be, think about the higher level concepts, maybe you have the wrong ingredient, can you substitute it for something else? Many levels of abstraction, reasoning that is semantic, visual, spatial, all the way down to the low level physical all at the same time in real time. Here's a video of a task. I'll actually later show a robot doing this, but this is a person performing what in an industrial environment is a very simple task, assembling a box, but you can see here that if we were to try to do this task, for example, in simulation, simulating a task like this would be pretty hard. And there's intricate movements and also semantics, where is the label maker, which side of the box do I put it on? So, there's a lot going on here. But to us as AI researchers, this actually presents a big opportunity because how sophisticated your AI system ends up being is heavily determined by how complex the environment that it learns from really is. And because the physical world is so complex, it gives AI systems a lot to learn. It gives them a lot to figure out. And because the physical world is so rich, I think it's actually our best opportunity to provide our AI models with enough depth to learn intricate understanding that might reproduce some of the flexibility and sophistication of human intelligence. So, if we think back to one of the big success moments of modern AI, which was AlphaGo, AlphaGo was very impressive because it showed emergent behavior. It could figure out moves in a high-stakes championship game that were different than any moves that human players playing the game of Go had thought of for the entire existence of the game. But AlphaGo was limited to an environment that is fairly small with fairly simple, well-defined rules. And that was of course important because for the technology of the time, this relied very heavily on simulating many games of Go. So, there's a lot of promise in this kind of emergent behavior, but when the emergent behavior is limited to a closed-world environment with very simple rules, then there's only so much that the system could do. Like AlphaGo will never figure out how to cook dinner, no matter how many games of Go it plays because the world of Go is too constrained for understanding those phenomena. If you have an AI system that interacts with the physical world and learns from interactions like that, it's not just solving a pragmatic task, it's not just addressing the fact that you want dinner and it's going to cook you dinner. It's conducting experiments. It's learning about the physical universe. And that understanding of the physical universe, those experiments that it conducts, they go beyond what people might know a priori. So, the great thing about agents is that agents actually interact with an environment, but when your agent is writing code or interacting with the web, what it's fundamentally learning about is a human-made artifact, the world of the operating system or the world of the internet. These are things created by humans and whatever knowledge they contain about the universe is there because people put it there. Whereas agents that can interact with the physical world have the entire physical universe to learn from, which transcends human understanding. So, robotic learning is about more than just building useful robots. It allows us to study the toughest AI problems in domains that provide realistic challenges and realistic opportunities. Okay. So, that's a bit of philosophy but let's talk a little bit more about the technical challenges.

那么，你可能会问的第一个问题是：“好吧，如果我们想回到那个 Gemini 的例子，让机器人打扫房间、叠衣服等等，这和我们考虑编码和网页导航等领域的智能体时需要考虑的问题真的有那么大的不同吗？”

So, the first question you might ask is like, "Okay, if we want to go back to that Gemini example and get the robot to clean the house and fold the laundry and so on, is it really so different than the kinds of problems that we have to think about when we think about agents for coding and for web navigation and so on?"

事实上，机器人学界有一个想法，认为这可能并没有那么不同——也许我们用来为其他领域（如计算机使用领域）构建智能体系统的基本思想，也可以扩展并应用于机器人。这有一定道理，因为对于某些任务，你处理的问题本质上是一个规划问题，你可以生成一段代码或进行思维链推理来制定计划，然后将这个计划交给机器人执行。

In fact, there is an idea that folks have explored in the robotics community that perhaps it's actually not that different, that perhaps the same fundamental ideas that we use to build agent systems for other domains, for computer use domains, can also be extended and applied to robots. And this kind of makes sense because for some of these tasks, the problem that you're addressing is fundamentally a planning problem and you can, for example, generate a piece of code or do chain-of-thought reasoning to figure out how to construct a plan and then just give this plan to the robot and have it executed.

我认为很好地体现了这一想法的一个工作是 Chekuri 等人和 Liang 等人的《Code as Policies》。Code as Policies 基本上就是我们今天所说的应用于机器人领域的经典智能体方法，它使用 API 调用来控制机器人，并且可以相当好地完成一些事情。例如，你可以给它一些逻辑谜题，比如把所有红色方块放在红色盘子里等等，它就会去执行。

One example of a work that I think encapsulates this idea very nicely is Code as Policies by Chekuri et al. Liang et al. And Code as Policies is basically this kind of what today we would call a classic approach to agents applied in the world of robotics that uses API calls to control a robot. And it can do some things fairly well. So, you can give it kind of logical puzzles like put all of the red cubes in the red plates and stuff like that and we'll go and do it.

但当我们实际观察这样一个系统的结果时，它看起来明显很机械，因为它与物理世界的交互是通过某种 API 中介的。虽然 API 对于与计算机交互的智能体来说是一种很好的中介方式，但对于与物理世界交互的智能体来说，问题在于，即使是人类也不知道如何为我们用手做的某些事情编写代码或逻辑计划。

But when we actually watch the results for a system like this, it looks distinctly robotic because its interaction with the physical world is mediated by some kind of API. And while an API is a very nice way to mediate interactions for agents that interact with computers, for agents that interact with the physical world, the trouble is that even we humans don't know how to write code or logical plans for some of the things we do with our hands.

所以，如果你问一个正在组装盒子的人：“你能一步一步描述组装盒子所需的所有身体动作吗？”他们很可能无法回忆起来。他们有某种肌肉记忆。超过一定抽象层次后，你与世界的物理交互在某种程度上是无意识的。它们低于你逻辑推理世界的层次。如果它们低于我们逻辑推理世界的层次，那么它们肯定低于我们现有手工设计的计算机程序逻辑推理世界的层次。

So, if you ask this person that's assembling this box, "Can you describe step by step all of the physical movements you have to do to assemble the box?" Chances are they wouldn't be able to recall it. They have this kind of muscle memory. And past a certain level of abstraction, the physical interactions you have with the world are kind of non-conscious. They are below the level at which you logically reason about the world. And if it's below the level at which we logically reason about the world, it's certainly below the level at which our existing hand-designed computer programs logically reason about the world.

因此，将涉及规划、逻辑和代码生成的经典智能体方法应用于这些机器人问题是不够的，因为问题的很大一部分低于我们习惯逻辑思考的抽象层次。

So, it's not enough to take a classical agents approach that involves planning, logic, and code generation to these robotic problems because a large chunk of the problem is below the level of abstraction at which we are accustomed to thinking logically.

因此，端到端机器人学习存在许多我们必须应对的挑战。为了获得有效的机器人基础模型，我们确实必须解决所有这些挑战，并且必须端到端地解决，因为只有通过端到端的方式，我们才能同时弄清楚所有那些抽象层次。

So, there are a number of challenges with end-to-end robotic learning that we have to address. And in order for us to get effective robotic foundation models, we really have to address all of these challenges. And we have to address them end-to-end because by addressing them end-to-end, that's the only way that we can figure out all those levels of abstraction at the same time.

一个是多样化场景的泛化。我们必须拥有能够与复杂物理环境交互的系统，并且要以对变异性鲁棒的方式，能够处理世界抛给我们的任何物理情况。为此，我们必须获得足够的数据。获取足够的数据是机器人学中的一个巨大挑战，但我认为目前学界已经做好了应对的准备。当然，这不仅仅是数量问题，任何从事基础模型工作的人都知道，获取足够的数据很大程度上关乎质量——你必须拥有正确的数据类型，能够捕捉环境中的变化程度。

One is generalization diverse scenarios. We have to have systems that interact with complex physical environments, but do so in a way that is robust to variability, that will handle whatever physical situation the world throws at us. To do this, we have to get enough data. And getting enough data is a big challenge in robotics, but it's also a challenge that I think the communities are well-positioned to address these days. And it's not just about quantity, of course, as anybody that's worked on foundation models realizes, getting enough data is very much about quality, that you have to have the right kind of data, the data that captures the degrees of variation in your environment.

在完成所有这些之后，接下来我们必须做对的是鲁棒性和速度。端到端机器人学习的历史一直是：获得一个平庸的、成功率大约一半的东西相对容易，但要获得一个每次都有效的东西则是一个巨大的挑战。

And having done all of that, then the next thing that we really have to get right is robustness and speed. And the history of end-to-end robotic learning has always been one where it's relatively straightforward to get something that is mediocre, that succeeds like half the time, but it presents a very serious challenge to get something that works every time.

现在，我们手中有一个非常强大的工具，尽管这个工具本身还不够，那就是基础模型的概念。我将从机器人学的角度来阐述基础模型的观点。我认为当你从根本上解决一个多模态问题时，这个观点是合理的。

Now, a very powerful tool that we have at our disposal, although this tool by itself is not enough, is the idea of foundation models. And I'm going to present a kind of a perspective on foundation models that is very much viewing it through the lens of robotics. It is a perspective that I think makes sense when you are fundamentally solving a multimodal problem.

端到端学习是一个非常强大的工具。端到端学习是将系统观察到的东西直接映射到它必须产生的输出上。但端到端学习也相当局限，因为经典的端到端学习使用的数据恰好是你想要解决的问题的数据。基础模型背后的核心理念是，你可能有许多其他问题需要与你真正想解决的问题类似的知识和理解。通过将这些其他问题的任务包含在模型的训练中，通过构建一个通才模型，你可以获得更多的知识，从而对你实际想要端到端解决的问题需要更少的经验。

End-to-end learning is a very powerful tool. End-to-end learning is about mapping directly the things that your system will observe to the outputs that it has to produce. But end-to-end learning is also quite limiting because classically end-to-end learning uses data of exactly the problem you want to solve. The big idea behind foundation models is that you might have many other problems that require similar kinds of knowledge and similar kinds of understanding to the problem that you really want to solve. And by including tasks from all these other problems in the training of your model, by making a generalist model, then you can acquire a lot more knowledge and thereby require less experience for the problem that you actually want to solve end-to-end.

因此，对于我们在机器人学中使用的那种多模态模型，也许计算机视觉、自然语言和其他领域的任务可以用来训练一个通用的基础模型，然后如果你想要专门化模型，它可以很容易地适应机器人任务，或者你甚至可以构建一个通用的机器人模型，将所有这些知识来源共同训练。本质上，端到端模型继承了来自显式视觉、显式自然语言处理等显式解决所有这些其他任务的经验，从而使得解决物理世界中非常复杂的任务变得更加可行。

So, in the case of the kind of multimodal models that we would use in robotics, perhaps there are tasks in computer vision, in natural language, in other domains that can be used to train a general purpose foundation model that is then very easy to adapt to robotic tasks if you want to specialize the model, or perhaps you might even build a general purpose robotic model where you would co-train on all these sources of knowledge all together. Essentially, the end-to-end model inherits the lessons from explicit vision, explicit NLP, explicitly solving all these other tasks and thereby makes it more feasible to solve very complex tasks in the physical world.

因此，我们将在这个想法的基础上，以各种方式扩展它，以解决物理智能的问题。

So, we're going to build on this idea and extend it in various ways to tackle the problem of physical intelligence.

现在，为了提供一个更明确的类比，我们来谈谈基础模型是如何训练的。如果你考虑一个语言模型或视觉语言模型之类的，通常现在的配方已经非常成熟。我们从预训练阶段开始。在预训练阶段，我们必须使用非常大量的数据。但从某种意义上说，预训练实际上并不难，因为对于使用哪些数据有一个非常简单的答案：答案是使用所有数据。这让你能够将大量知识融入模型。因此，模型理解很多东西。但在这个阶段，模型即使理解了很多，实际上也做不了太多，因为它还没有被教导如何利用这些理解。

Now to provide a more explicit analogy, let's talk about how foundation models are trained. If you think about a language model or a vision language model or anything like that, typically the recipe at this point is very well established. We start with a pre-training phase. In the pre-training phase, we have to use very large amounts of data. But in some sense, pre-training is actually not that hard because there's a very easy answer to which data to use. The answer is you use all of it. And that allows you to bake a lot of knowledge into your model. So, the model understands lots of stuff. But at this stage, the model, even though it understands a lot, doesn't actually do a lot because it hasn't been taught to leverage that understanding.

那么，通常还有一个后训练阶段，你可以用高质量的监督微调数据集进行微调，或者运行强化学习。后训练的目的与其说是教模型全新的知识，不如说是向模型解释如何利用其知识来实际解决它将被要求完成的任务。对于机器人基础模型，我们会发现一个非常类似的方案也很有效。有一个预训练阶段，在这个阶段，目标是获取非常广泛和多样化的数据。这些数据可以来自许多不同的来源、许多不同的机器人、网络数据等等。这里的目标是让模型真正理解物理世界的很多方面。但在这个阶段之后，模型可能做不了太多事情。然后会有某种后训练阶段，模型可能会被微调以遵循指令、进行测试时推理、流畅且灵巧地执行任务，也可能通过强化学习微调以获得更高水平的鲁棒性。我们将讨论流程的不同部分。我认为最终完整的终极机器人基础模型方案尚未完全确定，但我们有很多关于如何实现这些目标的好想法。

So, then there's typically a post-training phase where you might do things like fine-tune it with high-quality supervised fine-tuning data sets or run reinforcement learning. And the purpose of post-training is less so to teach the model fundamentally new things as it is to explain to the model how to leverage its knowledge to actually solve the tasks that it will be asked to do. With robotic foundation models, we're going to see that a very analogous recipe also works well. There's a pre-training phase, and in the pre-training phase, the goal is to get very broad and diverse data. And this data can come from many different sources, many different robots, data from the web, and so on. And here the goal is to get the model to really understand a lot about the physical world. But after this phase, the model might not be able to do a lot. And then there will be some kind of post-training phase where perhaps the model is fine-tuned to follow instructions, to do test-time reasoning, to perform tasks with a lot of fluency, a lot of dexterity, and also potentially fine-tuned with reinforcement learning to acquire a greater level of robustness. And we'll talk about different parts of the pipeline. I don't think that the final complete recipe for the ultimate robotic foundation model has yet been fully determined, but we have a lot of really good ideas about how these things can be approached.

那么，回到这三个大挑战，我们会看到，通过使用已经对视觉世界和语言有大量理解的预训练视觉语言模型，并将它们适应于物理交互，可以非常有效地实现对多样化场景的泛化。如果我们愿意汇集来自许多不同来源（包括许多不同的机器人等）的经验，我们就能获得足够的数据。我们可以通过强化学习来获得鲁棒性和速度，让机器人实际练习它们应该执行的技能，并通过试错学习来完善策略，达到越来越高的性能水平。

So, to come back to these three big challenges, what we'll see is that generalization to diverse scenarios can be approached quite productively by using pre-trained vision language models that already have a lot of understanding about the visual world and about language and adapting them to physical interaction. That we can get enough data if we're willing to pool experience from many different sources, including many different robots, and so on. And we can get robustness and speed by using reinforcement learning, by getting robots that actually practice the skills that they're supposed to perform and learning through trial and error to refine their strategies and to get to higher and higher levels of performance.

好的，那么让我们从架构开始，现代机器人基础模型的设计。有许多设计，但有一些基本原则似乎被广泛保留。其中一个基本原则是从其他预训练基础模型继承对世界的视觉和语义理解。所以，首先让我们谈谈一般的多模态。一开始，有语言模型。语言模型当然是当今几乎所有智能体系统的基础。它们是当今所有 AI 助手的基础。语言模型非常棒，因为它们知道很多东西。语言模型背后的基本思想，当然，你们可能都知道，是训练一个大型 Transformer，基于大量文本中的过去词元预测未来词元。这就是你得到编码系统的方式，也是你得到 AI 助手的方式。但如果你想让这些东西真正理解现实世界，你需要以某种方式让它们通过物理系统可用的模态来感知世界。而最强大的模态之一当然是视觉。所以，从语言模型到视觉语言模型，需要为这些系统增加看世界的能力。经典的做法是使用编码器，比如 VIT 模型，将图像像素转化为可以输入 LLM 主干的表示。传统上，这是通过将图像嵌入到与语言词元相同的嵌入空间中来实现的。当然，还有其他架构，比如早期融合模型等。但经典的做法是训练一个小编码器，将图像放入与文本相同的嵌入空间。你几乎可以把它想象成给你的语言模型添加一个小视觉皮层来帮助它看世界。

Okay. So, let's start with the architectures, the design of modern robotic foundation models. There are a number of designs, but there are a few basic principles that seem to be pretty widely conserved. And one of the basic principles is to inherit visual and semantic understanding of the world from other pre-trained foundation models. So, first let's talk about multimodals in general. In the beginning, there was language models. Language models are, of course, the foundation for pretty much all agent systems that are in use today. They're the foundation for all the AI assistants that are in use today. And language models are really great because they know a lot of stuff. The basic idea behind a language model, of course, as all of you probably know, is to train a large transformer to predict future tokens based on past tokens with a large body of text. And this is how you get coding systems, this is how you get AI assistants. But if you want these things to actually understand the real world, you need to somehow allow them to perceive the world through modalities that are going to be available to a physical system. And one of the most powerful modalities, of course, is vision. So, going from language models to vision language models involves augmenting one of these systems with the ability to see the world. And the way that that is classically done is by using an encoder, something like a VIT model, to take image pixels and turn them into a representation that can be fed into the LLM backbone. Traditionally, the way this is done is by embedding images into the same embedding space as language tokens. Although, of course, there are other architectures, early fusion models, and so on. But classically, you would train a little encoder that would put your images into the same embedding space as text. You can almost think of it as like taking your language model and adding a little visual cortex to it to help it see the world.

所以，当人们刚开始研究机器人基础模型时，第一种尝试的方法，我称之为第一代视觉语言动作模型，基本上就是直接采用 VLA 架构，只是改变了输出，不再输出对视觉问题的文本答案，而是输出机器人应该采取的动作的文本表示。因此，这些第一代 VLA 的架构与 VLA 完全相同。实际上，它们基于标准的 VLA，用于图像描述和视觉问答等任务。这些示例被扩充为：问题是一个机器人指令，图像是机器人的观察，答案是一个表示动作的数字。这就是 RT-2 的设计，它是第一个机器人基础模型，第一个 VLA。这些模型能做一些有趣的事情。它们能遵循基本的语言指令，比如把苹果移到罐子和橙子之间，或者把苹果移到布附近等等。但你可能会注意到这些视频中所有任务都相当初级。所以，有一些有趣的语义理解，但物理方面非常基础。这很合理，因为模型实际上必须说出它应该采取的动作的数值。想象一下，如果你必须大声说出你要如何移动手臂：我要逆时针旋转肘部 3 度，对吧？这是一种非常笨拙的方式来执行复杂灵巧的动作。

So, when people first started working on robotic foundation models, the first way that people tried to do this, and I'm going to call it first-generation vision language action models, is they basically took exactly this VLA architecture and they simply changed the output so that instead of outputting a textual answer to a visual question, the model outputs a textual representation of the actions that the robot should take. So, the architecture in these first-generation VLAs was exactly the same as a VLA. In fact, they were actually based on standard VLAs for tasks like image captioning and visual question answering. And those examples were augmented with examples where the question is a robot command, the image is the robot's observation, and the answer is like a number that represents the action. That was the design for RT-2, which was the first robotic foundation model, the first VLA. And these models could do some interesting things. They could follow basic language instructions, like moving the apple between the can and the orange or move the apple near the cloth, and so on. But something you might notice about these videos is that all the tasks are fairly primitive. So, there's some interesting semantic understanding, but the physical aspect of it is quite basic. And that makes sense because the model is literally has to say the numerical value of the actions it should take. Like, imagine that if you had to say out loud how you're going to move your arm, I'm going to rotate my elbow 3 degrees counterclockwise, right? Like, it's a very clunky way to perform intricate and dextrous movements.

那么，这方面的一大进步是 2024 年（大约一年半前）的重要进展之一，就是从第一代 VLA（为语言模型添加视觉皮层）过渡到我称之为第二代 VLA，它增加了一种虚拟运动皮层，一个专门用于输出高频连续动作的独立组件，通常使用扩散或流匹配。所以，你在屏幕上看到的架构是 PiZero 模型的设计。还有其他基于类似原理的模型，但它们都共享相同的基本思想，即像为图像添加图像编码器一样，为动作添加一种能够产生连续高频动作的动作解码器。因此，第二代 VLA 增加了一个专用的连续输出机制，可以专门用于运动控制，就像一种虚拟运动皮层。那么，这是 PiZero 模型的完整设计，它于 2024 年 10 月发布，是提出这种基本设计的模型。这里有一个输入，与 VLA 非常相似，由一到三张图像组成。还有一个语言指令。

So, one of the big advances in this was one of the significant advances in 2024, in this point about a year and a half ago, was to go from first-generation VLAs, which augments a language model with a visual cortex, to what I'm going to call second-generation VLAs that also add a kind of virtual motor cortex, a separate component that is designed for outputting high-frequency continuous actions, typically using diffusion or flow matching. So, the architecture that you see on the screen is the design of the PiZero model. There are other models based on similar principles, but they all share the same basic idea, which is to add, in the same way that for images we add an image encoder, for actions we add a kind of action decoder that can produce continuous high-frequency actions. So, second-generation VLAs add a dedicated continuous output mechanism that can specialize for motor control, like a kind of virtual motor cortex. So, here's the complete design of the PiZero model, which came out in October 2024, which was the model that proposed this basic design. So, here there's an input, which is very similar to the VLA, that consists of between one and three images. There's a language command.

动作解码器实际上是通过一种混合 Transformer 架构实现的。它有一个 26 亿参数的 LLM 主干，以及一个独立的动作专家。你可以把它想象成混合专家架构中的一个独立专家，它规模更小，但层数相同，并且可以关注 LLM 主干内的所有激活。它通过流匹配进行训练，以生成连续动作并解析连续关节角度。所以，本质上动作专家处理所有连续的运动控制。通常这类第二代 VLA 会生成所谓的动作块，即机器人应以较高频率执行的未来动作的短轨迹。在这个案例中，频率是 50 赫兹，即每秒 50 个动作。

And the action decoder is actually implemented in a kind of mixture of transformers architecture. So, there's an LLM backbone with 2.6 billion parameters and there is a separate action expert. You can think of it like a separate expert in a mixture of experts architecture, which is smaller but has the same number of layers and can attend to all the activations inside the LLM backbone. And it's trained with flow matching to produce continuous actions and to parse continuous joint angles. So, essentially the action expert handles all the continuous motor control stuff. Typically these kinds of second generation VLAs will produce what's called an action chunk, which is a short trajectory of future actions that the robot should take at some fairly high frequency. So, in this case it's at 50 hertz, 50 actions per second.

这可以做一些相当复杂的事情。这里有一个 Pi Zero 模型的视频，这个视频是一年半前的，展示叠衣服。它从篮子里取出衣服，需要把它弄平，然后整齐地叠好并放好。如果你用正确的数据正确训练，这些策略实际上相当鲁棒。在右侧，你可以看到 Michael 在捣乱，他会把不同的衣物放在机器人面前。如果有两件衣物，它必须把多余的一件收起来。如果他走过去把衬衫展开，机器人必须重新叠好。所以，你实际上会看到有趣的恢复、非常灵巧的技能，以及总体上相当不错的表现。

And this can do some pretty complicated stuff. So, here's a video of the Pi Zero model. This video is a year and a half old at this point folding some clothing. So, here it takes the clothing out of the bin, it has to straighten it out, and then it has to fold it nicely and put it away. And you get your policies that are actually pretty robust if you train them correctly on the right data. So, here on the right side you can see Michael messing with the small he'll put different items of clothing in front of the robot. If there's two items of clothing it has to put away the extra one. If he goes in and unfolds the shirt it has to refold it again. So, you actually get like interesting recoveries, very dexterous skills, and generally pretty good performance out of these things.

这是一个 Pi Zero 模型执行完整洗衣任务的视频。它从烘干机里取出衣物，开到桌子旁，把每件衣物从篮子里拿出来，然后开始叠。现在，这已经是一个老模型了，一年半在这个领域是永恒。你可以看到它远非完美。有时它会犯错，抓取失败，不得不重新抓取物品。它会卡住一小会儿。所以，还有很多改进空间，但真正有趣的是，尽管有这些错误，它实际上可以从许多错误中恢复，并最终成功完成任务。

This is a video of the Pi Zero model performing the full laundry task. So, it takes the laundry out of the dryer, drives it over to a table, takes the individual laundry items out of the bin, and then starts folding them. Now, this is an old model at this point, a year and a half is eternity in this field. And you can see that it's far from perfect. So, sometimes it makes mistakes, it has missed grasps, and it has to regrasp the item again. It gets stuck for a little while. So, very much something that has room for improvement, but what's really interesting here is that despite these mistakes, it can actually recover from many of them and eventually go on to solve the task successfully.

当然，现代基础模型远不止输出准确结果。过去几年语言模型领域最令人兴奋的创新之一，也是为智能体领域提供良好性能的关键，就是利用测试时算力的能力，即构建思考模型，在测试时花费更多算力来更有效地解决问题。由于视觉-语言-动作模型建立在 LLM 的基础上，它们也可以利用测试时算力。这实际上实现了一些非常有趣的事情。在传统的 VLA 中，模型观察图像和指令，然后输出动作。而在推理 VLA 中，模型观察图像，然后生成中间输出、中间思考，这些有助于产生更好的动作。虽然在传统的 LLM 思考模型中，思考有助于更好地解决问题，但在 VLA 中，思考还有助于更好的迁移，因为一些中间步骤，比如识别物体位置或将复杂指令分解为更简单的子任务，这些实际上更容易从网络数据中迁移，从而使模型更容易利用来自网络规模预训练的知识。

Now, of course there's a lot more to modern foundation models than just outputting accurate outputs. One of the most exciting innovations in the world of language models for the last few years and certainly something that has been instrumental to providing good performance in the world of agents is the ability to leverage test time compute, the ability to build thinking models that can spend more computation at test time to solve problems more effectively. And since vision language action models are built on the foundation of LLMs, they can also leverage test time compute. And this actually enables some really interesting stuff. So, in a conventional VLA, the model looks at the image, looks at the instruction, and then outputs an action. In a reasoning VLA, the model looks at the image and then produces intermediate outputs, intermediate thoughts that are useful to produce a better action. And while in a conventional thinking model for an LLM, the thoughts serve to facilitate better problem solving, in a VLA, the thoughts also facilitate better transfer because some of these intermediate things like identifying the locations of objects or breaking up a complex instruction into simpler subtasks, these are actually things that transfer more readily from web data and thereby make it easier for the model to leverage the knowledge that comes from web scale pre-training.

这里有一个草图，展示了关于具身思维链的原始论文中可能产生的思考类型。思考包括将计划（如把西瓜放在毛巾上）分解为一系列步骤，比如移动到西瓜处、抓取它、然后移动到毛巾处。一直到视觉任务，如定位任务相关物体或定位机器人的夹爪。这里有一些实际的具身思维链快照示例。你在图像上看到的边界框只是为了可视化。模型实际生成的是文本标记序列，但它们包含空间位置。所以它会说红色螺丝刀在这个位置。如果你想看实际效果，这里有一个视频可视化。在右侧你可以看到完整的思维链轨迹。这些都是模型的思考，在左侧你可以看到机器人实际执行任务。

So, here's a sketch of the kind of thoughts that the original paper on this embodied chain of thought could perform. Thoughts include breaking up a plan like put the watermelon on the towel into a sequence of steps like move to the watermelon, grasp it, and move to the towel. All the way to visual tasks like localizing task relevant objects or localizing the robot's gripper. So, here are a few examples of actual embodied chain of thought snapshots. The bounding boxes that you see on the image are just shown there for your visualization. What the model actually produces is a sequence of textual tokens, but they include spatial locations. So, it'll say like the red screwdriver is at this location. And if you want to see this in action, here's a video visualization. On the right side you can see the full chain of thought trace. So, these are all the thoughts of the model and on the left side you can see the robot actually performing the task.

现在，将这种具身思维链添加到视觉-语言-动作模型中效果非常好。这里的橙色条显示的是思考模型，它实际上基于 OpenVLA，一个 70 亿参数的开源视觉-语言-动作模型。所以，从蓝色条到橙色条的唯一变化就是添加了思考。这两个都是 70 亿参数的模型，而完整的思考版 OpenVLA 具身思维链实际上超过了 RT-2X，后者是谷歌一个更大的 VLA，有 550 亿参数。

Now, adding this kind of embodied chain of thought to a vision language action model works really well. So, the orange bar here shows the thinking model and this one is actually based on OpenVLA, which is an open source vision language action model with 7 billion parameters. So, the only change going from the blue bar to the orange bar is to add thinking. And these are both 7 billion parameter models and the full thinking version of OpenVLA embodied chain of thought actually outperforms RT-2X, which is a much larger VLA from Google, which was 55 billion parameters.

最近，我们一直在研究的一个方向是使用更丰富模态的 VLA 推理。其中一个非常重要的事情是能够解锁大型基础模型中存在的更多常识和语义知识。解锁的方式是让模型观察场景，思考问题。这里，它被要求把所有食物放在左边的盘子里，在高层，模型可以这样思考：'嘿，胡萝卜已经在盘子里了。机器人现在必须去拿香蕉，以避免把香蕉和黄色鸭子玩具混淆。指向动作是最好的。所以，回到这个 XY 位置的香蕉。' 这是一个相当合理的思考，一个好的视觉语言模型应该能够想到，但这只有在机器人理解'回到 XY 位置的香蕉'实际含义时才有效。所以，让这种推理有效的一个非常有帮助的方法是教模型在多个抽象层次上处理语言，让它理解如何响应复杂命令（如把香蕉放在鸭子旁边）以及非常具体的命令（如从 X1,Y1 移动到 X2,Y2）生成低级动作。它提供了一种系统一、系统二的抽象，如果机器人处于非常熟悉的情况，它不需要太费脑筋。它可以直接说：'嘿，我的命令是拿起鸭子。我们去拿鸭子。' 但如果处于非常不寻常的情况，它可以更努力地思考。

Now, more recently, one of the things that we've been working on is reasoning for VLAs with much richer modalities. So, one of the things that's really important is to be able to unlock more of that common sense and semantic knowledge present in large foundation models. Now, the way that you unlock it is that you get the model to look at a scene, to think about the problem. So, here it's told to put all the food on the plate on the left hand side and at the high level the model can think like, 'Hey, the carrot is already on the plate. The robot must now go for the banana to avoid confusing the banana with the yellow duck toy. A pointing motion is best. So, move back to the banana at this XY location.' Like that's a pretty reasonable thought that a good vision language model should be able to come up with, but this only works if the robot understands what move back to the banana at X, Y actually means. So, one of the things that's really helpful to get this kind of reasoning to work well is to teach the model to process language at many different levels of abstraction, to get it to understand how to produce low level actions in response to both complex commands like put the banana next to the duck all the way down to very grounded commands like move from X1, Y1 to X2, Y2. It provides a kind of a system one, system two abstraction where if the robot is in a very familiar situation, it doesn't need to think very hard. It can just say like, 'Hey, my command is to pick up the duck. Let's go pick up the duck.' But if it's in a very unusual situation, it can think harder.

它可以反思从网络规模数据中学到的所有知识，并将其直接转化为非常具体的指令，比如‘移动到这个位置’，以弥补在特定情境下具身经验的不足。因此，我们称之为可操控策略或可操控的视觉-语言-动作模型，它实际上可以显著提升性能。所以，这里你可以看到一个更加复杂的思维链过程，涉及长程思考，机器人正在执行一个与数据集中完全不同的任务。它必须思考如何够到胡萝卜，以及如何将其放入这个位置的锅中。在这个领域，这非常不寻常。另一个例子：‘让蓝色方块成为盘子上唯一的物体。’那么，你必须弄清楚，当然，要让它成为盘子上唯一的物体，你必须先把盘子上现有的东西拿走，然后把正确的物体放进去。

It can reflect on all that knowledge it learned from web-scale data and direct it down to a very grounded command like 'move to this location' to make up for its lack of embodied experience in that particular kind of situation. So, we call this a steerable policy or a steerable vision-language-action model, and it can actually improve performance very significantly. So, here you can see a much more sophisticated chain-of-thought process with long-form thinking, where the robot is actually performing a task that's pretty different from what was present in the dataset. It has to think through how to reach for the carrot and how to put it into the pot that's in this location. It's very unusual in this domain. Here's another example: 'Make the blue block the only object on the plate.' So, here you have to figure out that, of course, to make it the only object on the plate, you have to take whatever is currently on the plate off of the plate and then put the correct object in there.

我们测试这个模型的方法之一是让一个人通过不同抽象层次来操控模型，以检验模型是否足够可操控。结果发现，如果人们拥有所有这些抽象层次，并且这个人充当一种神谕式的高层推理过程，那么人们可以达到近乎 100%的成功率，但前提是他们使用了所有不同的抽象层次。然后，当然，我们可以让模型自己操控自己，这比无法以这种方式操控的基础模型有所改进。

One of the ways that we tested this model is we actually had a person drive the model through different levels of abstraction just to test if the model is sufficiently steerable. And it turns out that if people have all these levels of abstraction available to them and the person acts as a kind of oracle high-level reasoning process, then people can reach almost 100% success rate, but only if they use all of the different levels of abstraction. And then of course we can get the model itself to steer itself, and that leads to improvement over a base model that is not steerable in this way.

当然，这类思考模型已经被整合到更大的系统中。所以，大约一年前发布的 Pi O 5 模型，实际上采用了一种包含中间思考步骤的方案。它在规划、长描述和动作预测上通过下一个词预测进行训练，如果给定某个指令，它可以将其分解为更简单的部分，或者直接输出动作。然后在测试时，它以思考模式运行，当面对像‘打扫卧室’这样的复杂任务时，它可以将其分解为下一个合适的子任务，比如‘拿起枕头’，然后实际去执行。

And of course these kinds of thinking models have been integrated into much larger systems. So, the Pi O 5 model, which came out about a year ago, actually used a recipe that involved having an intermediate thinking step. So, it was trained with next-token prediction on planning, length captioning, and action prediction, where if it's provided with some command, it could break down that command into simpler parts or it could also output the action. And then at test time it was used in thinking mode, where when provided with a complex task like 'clean the bedroom,' it could break it up into the next suitable subtask like 'pick up the pillow' and then go and actually do it.

当我们开始构建这种大规模思考型 VLA 时，一个有趣的事情是，我们不仅可以在动作层面监督它们，实际上还可以在思想层面监督它们。所以，你可以让一个人实际与机器人对话，并指导它完成一项任务。这个人不提供动作监督，而是提供思想层面的监督。所以，如果任务是‘打扫厨房’，这个人可能会说：‘嘿，要打扫这个厨房，首先关上微波炉，然后拿起杯子。’这就是它的样子。所以，在左上角你可以看到来自人的指令，机器人去尝试执行那个指令。所以，这里的人通过语言教机器人如何清理厨房里的盘子，它必须拿起盘子，然后擦拭，然后放好毛巾，再把盘子放好。把盘子放在碗架上。这里还有更多这些口头指令的例子。所以，在每种情况下，机器人都在执行某项任务，一个人告诉它各个子步骤，这些子步骤随后可以作为额外的训练数据添加，而无需远程操作。

One of the interesting things when we start building large-scale thinking VLAs like this is that instead of just supervising them at the level of actions, we can actually also supervise them at the level of thoughts. So, you can have a person that actually talks to the robot and coaches it through a task. The person doesn't provide action supervision; the person provides thought-level supervision. So, here if the task is 'clean the kitchen,' the person might say, 'Hey, to clean the kitchen in this situation, first close the microwave, then pick up the cup.' And this is what this looks like. So, here in the top left you can see the command coming from the person and the robot goes and tries to do that command. So, here the person is teaching it through language how to clean up some plates in a kitchen, where it has to pick up the plate and then wipe it and then put the towel away and then put the plate away. Put the plate on the dish rack. And here are some more examples of these verbal instructions. So, in each of these cases, the robot is doing some task and a person is telling it the individual sub-steps, which can then be added as additional training data without the need for teleoperation.

然后，一旦所有这些都训练好了，我们就可以让它完全自主运行，它实际上只接收一个高级指令，比如‘清理溢出物’或‘打扫卧室’，模型自己产生中间思想来执行长程任务。所以这里所有这些视频中，模型都是完全自主运行的，仅基于初始的高级指令。顺便说一下，这些都是清理房屋的任务，而这些房屋在训练数据中从未出现过。所以它必须利用大量来自视觉语言模型的预训练知识来泛化到这些情况。

And then once all this is trained, then we can run it fully autonomously where it actually only receives a high-level command like 'clean up the spill' or 'clean the bedroom' and the model itself produces the intermediate thoughts to perform the long-horizon task. So here all of these videos the model is running fully autonomously just based on the initial high-level command. And these are, by the way, all tasks where it's cleaning homes that were not seen in the training data. So it has to use a lot of that pre-trained knowledge coming from the visual language model to generalize to these situations.

现在我们评估了每个因素的边际重要性。所以特别关注思考过程，这个图有很多需要解释的地方。纵轴是任务进展的度量，基本上就是性能。越高越好。这里的 HL 表示高级数据，即这些思想，被包含在模型预训练中。VI 表示模型有这些口头指令，即额外的语言监督，WD 表示网络数据。有趣的是，仅仅让人提供逐步指令，即白色条，实际上并不是性能的上限。模型自主运行时的表现实际上比在直接的人类指导下更好。这非常有趣。这意味着它不仅从那些人类指令中学习，而且实际上最终做得比普通人更好。

Now we evaluated the marginal importance of each of these factors. So focusing specifically on the thinking process, there's a lot to explain here to unpack in this graph. The vertical axis is a measure of task progress and basically the performance. Higher is better. And HL here means that high-level data, these thoughts, were included in model pre-training. VI means the model had these verbal instructions, the additional language supervision, and WD means web data. So interestingly, just having the person provide the step-by-step instructions, that's the white bar, is not actually an upper bound of performance. The model actually does better on its own running autonomously than it does under direct human direction. Which is pretty interesting. So that means that not only did it learn from those human instructions but actually ended up doing better than the average human.

另一件事是这条浅绿色条，黄色条当然是完整模型。浅绿色条是完全相同的东西，只是它在训练时被教导要思考，但在测试时不思考。所以测试时思考被关闭。这基本上只是衡量在训练中包含思考的边际收益。你可以看到这实际上是第二好的方法。所以仅仅训练模型去思考，即使你在测试时不思考，已经让它好了很多。这也非常有趣。如果我们排除这些口头指令，即一个人用语言引导机器人完成任务，那是中间的绿色条，实际上它差了很多。所以它从那些口头指令中获得了大量信息。我认为这实际上非常令人兴奋，因为这意味着我们通过用语言教机器人，就像教人一样，获得了很大的性能提升。你真的只是用语言引导它完成任务。

The other thing is this light green bar, the yellow bar of course the full is the full model. The light green bar is exactly the same thing only it is taught to do thinking at training time but then it does not think at test time. So at test time the thinking is switched off. So this is basically just measuring the marginal gain of including thinking in training. And you can see that that's actually the second best method. So just training the model to think even if you don't think at test time already makes it quite a bit better. And that's also really interesting. If we exclude these verbal instructions where a person walks the robot through a task with language, that's the middle green bar, it actually is a lot worse. So there's a lot of information that it's getting from just those verbal instructions. And I think that's actually really exciting because that means that we're getting a lot of performance from teaching a robot through language the same way that you would teach a person. You literally just walk it through the task with language.

好的。到目前为止，这全是关于模型和架构之类的东西。但当然，我们知道，在研究基础模型时，真正获得出色性能的最重要因素是获得正确的数据。所以让我们谈谈数据。几年前，当我们开始研究机器人基础模型时，我们想，也许在一个非常具体的应用领域获取大量数据真的很难。但如果我们让全世界所有研究机器人操作的研究人员基本上都向我们提供他们的数据，这样我们就可以把所有数据汇集在一起，训练一种通用模型呢？所以 RTX 项目，大约三年半前，是这方面的一个早期尝试，我们联系了全球 30 多个实验室，基本上要求他们为科学捐赠他们的数据。

Okay. So so far this was all about models and architectures and things like that. But of course we know that when we work on foundation models the most important thing to really get great performance is to get the right data. So let's talk a little bit about data. A number of years back when we started working on robotic foundation models, we thought, well, maybe it's really hard to get a lot of data in one very specific application domain. But what if we ask everybody around the world who's doing research on robotic manipulation to basically provide us with their data so that we can pull all of it together and train a kind of generalist model. So the RTX project, which at this point was about three and a half years ago, was kind of an early attempt at this where we contacted over 30 labs around the world and we essentially asked them to donate their data to science.

数据非常异构。所以我们最终在训练集中只保留了来自单臂机器人、带有平行夹爪的数据。但即便如此，这些数据仍然非常多样，你可以看到这里展示的不同任务示例。然后我们做的是，基于所有这些数据训练了一个模型——一个视觉语言动作模型——并将其送回给一些贡献了数据集的实验室，共七个实验室，我们请他们测试这个通用模型，并与他们在各自特定领域的最佳模型进行比较。每个实验室都有不同的任务、不同的机器人，以及相应的、他们在自己环境中开发的专门模型。我们发现，这个基于所有数据训练的单一通用模型，平均比每个实验室的最佳个体模型好 50%，也就是这里的 63 这个数字。这实际上相当令人惊讶，因为在研究中，通常如果有人针对某个问题迭代了一段时间，他们会把所有的细节都做对，并让他们的东西非常专门化，从而工作得很好。所以，基本上在第一次尝试中，通用模型就能比这些专门模型好大约 50%，我认为这确实非常令人惊讶。这也非常令人兴奋，因为它表明，也许在机器人领域，我们正在看到通用性优势的开始，即通用模型甚至可以在那些专门模型擅长的领域超越它们。当涉及到汇集来自许多不同来源的数据时，这非常有前景。

And the data was quite heterogeneous. So we ended up in our final training set curating it down to just the data from single arm robots with parallel jaw grippers. But even those were quite varied and you can see some examples of different tasks here. And then what we did is we trained a model, a visual language action model, on all of this data and we sent it back to some of the labs that had contributed data to the data set, seven of these labs, and we asked them to test the generalist model compared to whatever was the best model that they had in their specific domain. So each model each lab had a different task, a different robot, and correspondingly a different specialized model that they were developing in their setting. And we found that the single generalist model trained on all of their data was on average 50% better, that's the 63 number here, than the best individual model from each lab. And this is actually pretty surprising because in research typically if someone has been iterating on a particular problem for a while, they will have really gotten all the details right and really specialized their thing so that it works well. So the fact that essentially on the first attempt the generalist model could outperform these by about by about 50%, I think is actually pretty surprising. And it also is very exciting because it suggests that maybe in robotics we were seeing the start of that benefit of generality where a generalist could outperform specialist models even in the domains that those specialists were specialized to. And this is very promising when it comes to pooling data from many different sources.

当然，现代机器人基础模型通常都是这样训练的。PI0 模型使用了来自许多不同来源和许多不同机器人的预训练数据。如今，最新的模型使用的数据要多得多。PI0 已经有一年半的历史了，当时它是在大约 1 万小时的机器人数据上训练的。现代模型通常使用的数据量大约是这个数量的一个数量级。仅靠这些数据进行预训练是不够的，因为当你在这个非常多样化的数据上预训练时，你可以得到不错的性能，但无法达到专家级性能。如果你真的想要专家级性能，你必须有一个后训练阶段。后训练数据更加专门化，经过更精心的策划，以展示一致的策略和有效的行为，但它也可以小得多，因为后训练阶段不需要捕获所有知识。知识已经在预训练中获得了。后训练是为了教模型如何实际解决任务，使用正确的策略。所以后训练数据可以非常小，只有几十个小时。它质量高但范围窄，并且展示了做好任务的一致策略。关键是，如果你只在高品质的后训练数据上训练，效果并不好。这不仅仅是因为数据量小，而是因为数据质量好。当数据质量好时，你不会看到很多错误，因此也不会看到很多恢复。而恢复是至关重要的，因为机器人不会总是完美成功，当它犯错时，它必须依靠从预训练阶段学到的东西来从错误中恢复。所以，这种高质量的后训练数据只有与大规模预训练结合时才效果很好，因为大规模预训练教会了它如何处理可能遇到的所有意外和复杂情况。

So of course modern robotic foundation models are typically trained in this way. So the PI0 model uses pre-training data from many different sources and also many different robots. These days of course the latest models use a lot more data. PI0 is over a year and a half old and back then it was trained on about 10,000 hours of robot data. Modern models typically would use about an order of magnitude more than this. Now pre-training on just this data is not quite enough because when you pre-train on this very diverse data you can get like okay performance but you don't get expert level performance. And if you really want expert level performance, you have to have a post-training phase. And the post-training data is more specialized. It's more curated to be good data, to exhibit consistent strategies and effective behaviors, but it can also be much smaller because it's not necessary for the post-training phase to capture all the knowledge. The knowledge is already there in pre-training. Post-training is there to teach the model like here's how you should actually solve the task. Here's the right strategy to use. So the post-training data can be pretty small, just tens of hours. It's high quality but narrow. And it illustrates consistent strategies to do the task well. Crucially, if you just train on the high quality post-training data, this does not work very well. And it's not just because the data is small, it's because the data is good. When the data is good, you do not see very many mistakes and therefore you don't see very many recoveries. And recoveries are essential because the robot will not always succeed perfectly and when it makes a mistake, it has to fall back on what it learned from the pre-training phase in order to recover from that mistake. So this high quality post-training data works great only when it's combined with large scale pre-training because the large scale pre-training teaches it how to deal with all of the unexpected and complicated situations that it might encounter.

另一件事是，这种从各种不同来源汇集数据的想法，一旦规模足够大，实际上可以工作得很好。这是 PI05 训练混合的示意图。PI05 是我之前展示的清理厨房和卧室的模型。在我展示的所有 PI05 实验中，我们使用了移动机器人。PI05 数据集中只有大约 3%使用了移动机器人，97%的数据来自其他平台。这是因为移动机器人又大又笨重又昂贵。所以完全用移动机器人获取这么大尺寸的数据集需要很长时间。但即使只有 3%的数据集来自移动机器人，移动机器人也从其他机器人平台继承了很多知识，它们可以继承那些运动技能，并实际上非常有效地执行任务，尽管它们在最终数据集中代表性不足。关于数据，我们研究的另一件事是，对环境的泛化如何随训练中看到的环境数量而扩展。正如我提到的，我展示的 PI05 实验都是在训练数据中未见过的新的厨房和卧室中测试的。这代表了如果你把这个机器人带到你自己家里会发生什么。这不是它以前见过的家，但它可以尽力尝试在那里做好事情。在这个图中，x 轴是训练集中的位置数量。我们消融了位置数量，将其子采样到更低，以尝试查看模型在不同数量的训练家庭、训练环境下的表现。绿线显示最终模型，即在整个数据集上训练的模型，该数据集有大约 100 个不同的家庭。黄线显示如果我们减少预训练和后训练中的位置数量会发生什么。你可以看到，当达到大约 100 个时，它已经饱和，基本上与最终模型的值相同，在误差范围内。这意味着在大约 100 个家庭之后，它已经达到了最大值。它仍然不是 100%，还有改进空间，但改进空间不是来自更多的家庭，因为那已经达到最大值了。它将来自其他东西，比如我将在下一节讨论的强化学习技术。有趣的是，如果我们省略大规模预训练阶段，那么即使使用所有不同的位置，性能也会差很多。所以你可以看到浅绿色条，唯一的区别是我们省略了包含各种不同机器人的预训练阶段，我们仍然使用预训练的视觉语言模型，只是不使用来自其他机器人的数据，性能实际上比只有十分之一环境数量时还要差。当然，如果——哦，我应该在这里提一下，对于绿色条，它之所以是上界基线，是因为它实际上是在测试位置上训练的。所以黄线是在未见过的位置上训练的，而绿色条是在测试位置上训练的。所以它有点像神谕上界。

Now the other thing is that this idea of pooling data from a variety of different sources can actually work very well once it's scaled up enough. So this is a diagram of the training mixture for PI05. PI05 is the model that I showed before that was cleaning up kitchens and bedrooms. Now in all of the experiments for PI05 that I showed, we used a mobile robot. Only about 3% of the data set for PI05 used mobile robots. 97% of the data came from other platforms. And it's because mobile robots they're they're big, they're clunky, they're they're expensive. So getting a data set of this size entirely with mobile robots would take a while. But even though only 3% of the data set came from the mobile robots, the mobile robots inherited a lot of knowledge from other robotic platforms and they could inherit those motor skills and actually perform the task very effectively even though they were very underrepresented in the final data set. The other thing that we actually studied here in regard to data is how generalization to environments scales with the number of environments seen in training. So as I had mentioned, the experiments that I showed for PI05 were all tested in new kitchens and bedrooms that were not seen in the training data. So this is kind of representative of what would happen if you take this robot and you bring it to your own home. It's not a home that it's seen before, but it can make a best guess attempt at how to do something good there. Here in this graph, the x-axis is the number of locations that we have in the training set. So we ablated the number of locations, we subsampled it to be lower to try to see how the model performs with different numbers of training homes, training environments. The green line shows the final model. So this is the model trained on the entire data set which had about 100 different homes. And the yellow curve shows what happens if we reduce the number of locations both in pre-training and post-training. And you can see that by the time it reaches about 100, it's already saturating at essentially the same value up to the error bars as the final model. So that means that after about 100 homes, it's kind of maxing out. Now it's still not at 100% that there's still room to improve but that room for improvement is not from having more homes because that's already at the max. It would be from other things like the RL techniques that I'll discuss in the next section. Interestingly, if we omit the large scale pre-training phase, then the performance is a lot worse even when using all of the different locations. So you can see that the light green bar which where the only difference is we omit this pre-training phase with all the different kinds of robots, we still use a pre-trained VLM, we just don't use the data from other robots, the performance is actually worse than if we had only a tenth of the number of environments. And of course if we um Oh and I I should I should mention here for the green bar, this is a the reason this is an upper bound baseline is that this is actually trained on the test location. So the yellow line is trained on an unseen location, the green bar is trained on the test location. So it's kind of an oracle upper bound.

当然，如果你必须在现场测试且没有预训练，那么性能会比所有这些都差很多。所以，这就是模型架构和数据。在最后一部分，我想讨论的是这个差距，即我们目前得到的约 84%与我们希望达到的 100%之间的差距——如果我们想要一个真正稳健的策略，一个几乎每次都能成功的策略。我将讨论如何使用强化学习来缩小这一差距，从而获得可靠、快速且流畅的模型。

And then of course if you have to test it in location and you don't have pre-training, then performance is quite a bit worse than all of these. So, that's the model architecture and the data. And then in the last section, what I want to talk about is basically this gap, the gap between the 84% or so that we're getting here and the 100% that we would like to get, if we want to have a truly robust policy, a policy that succeeds almost every time. And what I'll discuss here is how to use reinforcement learning to close that gap, to get models that are reliable, fast, and fluent.

我们在 UC Berkeley 的研究小组从事强化学习已经相当长时间了。最初，强化学习是获得专门策略的一种非常有效的方法。这是一个训练双臂 Franka 机器人组装汽车仪表盘的训练过程示例。在这种情况下，这不是训练机器人基础模型，而是训练一个更小的模型，它只学习这一个环境中的这一项任务。所以，目标不是泛化，而仅仅是组装仪表盘。训练需要几个小时，这让你对它的样子有所了解。你可以看到机器人反复尝试这项任务。这是一项相当困难的任务。你看，你必须对齐所有那些塑料夹子等，才能正确插入。这是训练后的最终仪表盘组装。机器人拿起仪表盘，相互协调，插入所有销钉，使其正确对齐并正确组装。这实际上不是一件容易的事，因为你可以看到有很多销钉，它们都必须完美对齐。

Now, we've been working on reinforcement learning in my group here at UC Berkeley for quite a long time. In the beginning, RL was a very effective strategy for getting specialized policies. So, this is an example of a training process for training a bimanual Franka robot to assemble a car dashboard. In this case, this is not training a robotic foundation model. This is training a much smaller model that is just learning only this task in this one environment. So, the goal here is not to generalize. The goal here is literally just assemble the dashboard. And the training here takes a few hours and this gives you some sense for what it looks like. So, you can see the robot tries the task over and over again. This is a fairly difficult task. You see, you have to align all those plastic clips and so on to get it to insert properly. And here's the final dashboard assembly after training. So, the robots pick up the dashboard, they coordinate with each other, and they insert all of the pins so that it lines up correctly and is correctly assembled. So, this is not an easy thing to do, actually, because you can see there are a lot of pins and they all have to line up perfectly.

以下是可以用这些专门方法学习的各种任务。同样，每个任务都是一个单独的模型。这不是通用基础模型，而是通过几个小时的练习学习每个任务。你可以看到其中一些是变形物体，比如正时皮带；有些非常动态，比如翻鸡蛋或击打叠叠乐积木。所有这些策略在训练环境中的成功率都是 100%。它们也比监督学习快得多，大约快两倍或更多。所以，你获得了更好的吞吐量。但是，这里没有预训练，泛化能力非常有限。所以，这实际上是相当狭窄的专家的配方。

Here are the different tasks that could be learned with these kind of specialized methods. Again, each task is a separate model. This is not a general-purpose foundation model, but it is learning each of these tasks through a few hours of practice. And you can see that some of them are deformable objects like the timing belt. Some of them are very dynamic like flipping the egg or hitting the Jenga block. The success rate for all of these policies in the training environment is 100%. They're also quite a bit faster, about two times faster or more than supervised learning. So, you get better throughput. But, there's no pre-training here and there's very limited generalization. So, this is really a recipe for fairly narrow specialists.

所以，机器人基础模型的大问题是：我们能否利用这样的技术，但将其应用于通用模型，从而缩小鲁棒性方面的差距？我们最近在 Physical Intelligence 开发的一个东西叫做 Pi Star 06。这基本上是 Pi 05 模型的一个略微演进的版本，然后通过强化学习进行后训练以提高鲁棒性。我们必须弄清楚如何将强化学习融入这些大规模的机器人策略中。应用用于 LLM 的标准强化学习技术（如 PPO 或 GRPO）非常困难，因为训练 LLM 时，我们通常需要从最新策略生成大量样本。但在机器人技术中，每个样本都是现实世界中一次任务的尝试。所以，效率在那里非常重要，即强化学习所需样本数量的效率。

So, the big question with robotic foundation models is, can we leverage techniques like this, but apply them to generalist models that can use this to close that gap on robustness? So, one of the things we developed recently at Physical Intelligence is called the Pi Star 06. This is basically a slightly evolved version of the Pi 05 model that is then post-trained with reinforcement learning for robustness. And we had to figure out how to incorporate reinforcement learning into these large-scale robotic policies. It's very hard to apply standard reinforcement learning techniques of the sort that are used for LLMs because when training LLMs with like PPO or GRPO, we typically require generating lots of samples from the latest policy. But, in robotics, each of those samples is a real-world attempt at a task. So, efficiency is really important there, efficiency in terms of the number of samples that you need for reinforcement learning.

所以，Pi Star 06 的配方有点不同。它不使用 PPO 或 GRPO。它首先利用迄今为止看到的所有数据来训练一个视觉语言动作模型，并增加一个额外输入。这个额外输入是一种来自价值函数的动作最优性指标。价值函数查看当前图像和当前动作，并试图预测：这个动作会让我更接近任务成功吗？然后它告诉 VLA：这个动作会让我更接近很多，这个动作只会让我接近一点点，或者这个动作实际上会让我离完成任务更远。价值函数本身也基于一个大型视觉语言模型。所以，它实际上使用相同的架构，其工作基本上是批评这些动作。一旦训练完成，我们就可以实际部署这个策略，在现实世界中运行，收集更多数据，然后利用这些额外数据来优化价值函数。因为价值函数所做的只是预测：这个动作会让我更接近解决问题吗？所以，你可以使用所有自主数据，即使这些数据非常次优，来优化你对每个动作使你接近任务完成程度的估计。然后你有了更好的价值函数，对 VLA 可能采取的每个动作有更好的批评，从而使其能够采取更好的动作。

So, the recipe in Pi Star 06 is a little different. It doesn't use PPO or GRPO. What it does is it starts by using all of the data seen so far to train a visual language action model with one additional input. And that additional input is a kind of indicator of action optimality that comes from a value function. So, the value function looks at the current image and the current action and it tries to predict, will this action get me closer to succeeding at the task? And then it tells the VLA, this action will get me much closer, this action will get me only a little bit closer, or this action will actually get me further away from finishing the task. The value function itself is also based on a large visual language model. So, it actually uses the same architecture, and its job is to basically critique these actions. And then once this is trained, we can actually deploy this policy, run it in the real world, collect more data, and then use that additional data to refine the value function. Because the value function, all it's trying to do is predict, will this action bring me closer to solving the task? So, you can use all the autonomous data, even if that data is highly suboptimal, to refine your estimate of how much closer each action gets you to solving the task. And then you have a better value function and a better critique of every action that the VLA could take, which then allows it to take better actions.

这是完整的架构。有点复杂。模型仍然有那个高层策略，仍然有这个思考过程，现在它接收这个额外的、我们称之为优势的东西，它基本上是一个指标，表明该动作比它可能采取的其他动作好多少。这个优势来自价值函数。价值函数实际上非常直观。这是机器人折叠 T 恤的价值函数可视化。Y 轴是预测值，X 轴是回合内的时间。红色高亮部分是机器人犯错并弄皱衬衫的时候。你可以看到价值下降，这意味着模型预测这种行为正在让你离解决问题更远。这里机器人正在打开冰箱并取出一个水过滤器。当它打开门时，那是红色高亮部分，价值预测大幅上升，因为它说：嘿，你打开了门，你离成功更近了。当它最后打翻水过滤器时，价值下降。它在说：嘿，你犯了一个错误，你需要从中恢复，因此这让你离成功更远。

Here is the complete architecture. It's a bit involved. So, the model still has that high-level policy, still has this thinking process, and now it takes in this additional, what we call the advantage, which is basically an indicator of how much better the action is than other actions it could take. And that advantage comes from the value function. And the value function is actually pretty intuitive. So, here is a visualization of the value function for a robot folding a t-shirt. The Y axis is the predicted value and the X axis is the time within the episode. And the part highlighted in red is when the robot makes a mistake and crumples up the shirt. And you can see that the value goes down, which means that the model is predicting that this behavior is taking you further away from solving the task. Here the robot is opening the fridge and taking out a water filter. When it gets the door open, that's the part highlighted in red, the value prediction goes up a lot because it's saying like, hey, you got the door open, you're getting a lot closer to success. And when it tips over the water filter at the end, the value goes down. It's saying like, hey, you made a mistake, you'll need to recover from this and therefore it's taking you further away from success.

当我们把所有东西放在一起时，我们实际上可以得到一个能够解决一些相当复杂任务的策略。这是一个机器人使用高端意式咖啡机制作浓缩咖啡的视频。这对机器人来说是一项非常困难的任务。它必须协调一系列步骤。所以，它利用那个高层策略来思考步骤。它必须实际执行物理上复杂的操作，比如压实咖啡粉，你必须非常精确地将压粉器放入手柄中。你必须施加相当大的力才能将手柄放入咖啡机中。而且你必须实际拧紧它。

When we put it all together, we can actually get a policy that can solve some pretty complex tasks. So, here's a video of the robot using a high-end espresso machine to make espresso. And this is a very difficult task for the robot to do. It has to coordinate a sequence of steps. So, it's using that high-level policy to think through the steps. It has to actually perform physically intricate things like tamping down the espresso where you have to get the tamper into the portafilter very precisely. You have to apply quite a bit of force to get the portafilter into the espresso machine. And you have to actually tighten it.

然后你得等着意式浓缩做好。所以，你推下杠杆，必须看着咖啡流出来，等咖啡好了才把杯子拿走。你得倒牛奶，倒咖啡。所有这些都需要非常稳健可靠的物理行为，以及像思考任务步骤这样的高层抽象。经过强化学习训练后，我们让 Pi Star 06 在这个任务上连续运行了大约 13 个小时。所以你可以看到这个长期测试。之后办公室每个人都喝了很多咖啡。所以，大家都有很多咖啡。这个策略并不完美，它确实犯了一些错误，比如掉了几次手柄，但经过强化学习训练后，比没有训练时好多了。我稍后会展示定量结果。

And then you have to wait while the espresso is being made. So, you push the lever and you have to actually watch the coffee coming out and only take the cup away when the coffee is done. You have to pour the milk, you have to pour the coffee. So, all these things require physical behaviors that have to be very robust and very reliable, as well as high-level abstractions like thinking through the steps of the task. After training with RL, we could run Pi Star 06 on this task for about 13 hours continuously. So here you can see this long-term test. Everybody in the office ended up very caffeinated after this. So, lots of coffee for everybody. And the policy was not perfect. So, it did make a few mistakes. It dropped the portafilter a few times, but it was quite a bit better with RL training than without it. And I'll show the quantitative results later.

这是另一个任务。在旧金山的蒲公英巧克力工厂组装盒子。这是我演讲开始时展示一个人执行的任务。你可以看到机器人从堆叠中取出盒子，折叠盒子，贴上贴纸，然后放好。这是一个吞吐量非常重要的任务。强化学习可以改进的一个方面就是吞吐量，让机器人更快地完成任务，这样每小时就能做更多任务。

Here is another task. This is assembling boxes at a chocolate factory, Dandelion Chocolate in San Francisco. This is the task that I showed a person performing at the beginning of this talk. And you can see the robot takes the boxes off the stack, folds the boxes, puts a sticker on them, and puts them away. And this is a task where throughput is really important. And one of the things that RL can improve is throughput, make the robot faster doing the tasks so they can do more tasks per hour.

这是定量结果。我来解释一下它们的含义。右侧的深黄色条是最终经过强化学习训练的策略。Y 轴是每小时成功吞吐量。绿色条是仅在线阶段之后的结果。这是我们部署策略并收集额外自主数据之前得到的结果。浅黄色条本质上是 SFT。也就是说，如果我们采用离线训练的模型，然后在目标任务的良好演示上对其进行微调。这本质上是一种 SFT 方法。然后深黄色条是最终方法。你可以看到，在大多数任务上，吞吐量比 SFT 提高了大约 2 倍。我要说明的是，SFT 基线仍然使用了强化学习，只是仅在离线阶段使用。这意味着强化学习方法确实对这些模型产生了显著影响。

Here are the quantitative results. Let me explain what these mean. So, the dark yellow bar on the right side is the final RL-trained policy. The Y axis throughput successes per hour. The green bar is what happens after just the online phase. So, this is what we get before we deploy the policy and collect additional autonomous data. The pale yellow bar is essentially SFT. So, if we take the offline-trained model and then we fine-tune it on good demos from the target task. So, this is essentially an SFT approach. And then the dark yellow bar is the final method. And you can see that the throughput improves by about 2x on most of these tasks over SFT. And the SFT baseline, I should say, still uses RL. It just uses RL only in the offline phase. So, this means that the RL method is really making a difference for these models.

好的。总结一下，我谈到了如果我们想让机器人基础模型发挥作用，必须解决三个挑战：泛化到多样场景、获取足够数据来训练这些模型，以及鲁棒性和速度。我讨论了针对每个挑战可以使用的技术。我想留给大家的想法是，我们作为 AI 研究者真正想研究这些问题的原因，不仅仅是因为机器人很酷，而是因为一旦我们有了能够与物理世界交互的系统，特别是如果它们能像我在最后展示的强化学习方法那样从自己的自主经验中学习，这些系统就不仅仅是解决实际问题。它们实际上是在与世界互动，并通过互动更多地了解物理宇宙。这是它们学到的东西，不是由人提供的。我的意思是，咖啡机是人放的，但总的来说，物理世界运作的方式是由自然规律决定的，而不是人。这是 AI 系统学习人类可能还不知道的东西的一种方式。所以我认为，在未来，我们可能会非常重视真实世界机器人系统收集的经验，我们可能会利用这些经验不仅是为了获得更好的机器人，而且是为了获得更好的 AI 系统。非常感谢，我很乐意回答问题。

Okay. So, just to summarize, I talked about how if we want robotic foundation models to work, we have to address these three challenges: generalization to diverse scenarios, getting enough data to actually train these things, and then robustness and speed. And I discussed the kind of techniques that we can use for each of these. And the thing that I want to leave you with is this thought that the reason that we as AI researchers might really want to study these problems is not just because robots are cool, but because once we actually get systems that can interact with the physical world, especially if they can learn from their own autonomous experience like the RL methods that I showed at the end, those systems are not just solving a practical problem. They're actually interacting with the world and learning more about the physical universe through their interactions. And that's something that they're learning that is not put there by people. I mean, the espresso machine was put there by a person, but in general, the way the physical world works is determined by the laws of nature, not by people. And that's a way for AI systems to learn something that people might not already know. So, I think in the future, we might actually put a great deal of value on experience collected by real-world robotic systems, and we might use that experience not just for getting better robots, but for getting better AI systems in general. Thank you very much and I'll be happy to take questions.

谢谢 Sergey 的精彩演讲。现在开始提问。我很乐意先开始。我觉得关于测试时推理的讨论非常有趣。那么，从可靠性和安全性的角度来看，这如何与延迟相互作用？你认为推理应该发生在策略内部还是外部循环中？

Thank you, Sergey, for this great talk. And let's open up the floor for questions. I'd be happy to start. I find the discussion on the test time reasoning very interesting. So, from the perspective of reliability and safety, I wonder how does that interact with latency? And do you think reasoning should happen inside the policy or in the outer external loops?

是的，这是个很好的问题。对于实时机器人系统，我们目前通常的做法是，即使我们训练模型端到端自回归地生成思考，在测试时，我们实际上运行两个模型副本。一个以较低频率生成思考，另一个以较高频率生成动作。例如，在制作咖啡的例子中，思考大约每 2 秒产生一次，即 0.5 Hz。但思考被消费的频率要高得多，大约每 150 毫秒一次。这让我们获得了实时性能。在这种情况下，所有模型实际上仍然在云端运行，但可以想象，如果我们有一个连接受限的、对安全性要求极高的系统，我们可能会在设备端运行动作生成，而高层思考则在远程进行。

Yeah, that's a really good question. So, for real-time robotic systems, typically the way that we do it these days is that even when we train the model to generate the thoughts end-to-end auto-regressively, at test time, we actually run two copies of the model. One copy that generates thoughts at a slower frequency, and one copy that generates actions at a higher frequency. So, for example, in the coffee making example, the thoughts are produced roughly once every 2 seconds, so 0.5 Hz. But, the thoughts are consumed much more frequently, roughly every 150 ms. And that gets us that real-time performance. In that case, all of the models are actually still running in the cloud, but conceivably, if we had a really safety-critical system with limited connectivity, we might actually run the action generation on device on the edge, and the higher-level thinking remotely.

有趣。所以，为了处理延迟，动作生成有一个层次结构。是的，这非常有帮助。那么，对于观众，请随意提问。你可以举手，我也看到聊天中有问题。那么，我应该浏览聊天中的问题吗？我很乐意为你读出来。我还看到 Harry 举手了，请讲，Harry。

Interesting. So, there is a hierarchy of the action generation in order to handle the latency. Yeah. That's very helpful. So, for the audience, please feel free to ask your questions. You can raise your hand, and I also see there are questions in the chat. So, yeah. Should I go through the chat questions or... So, I'd be happy to read it for you. And I also see that Harry has his hand raised, so please go ahead, Harry.

你好，Levine 教授。谢谢你的演讲。我很好奇，你是否认为高分辨率触觉数据对于精细抓取是必要的，超越 Iris 或我们的 token 工作，我想知道 Iris 是否专注于触觉作为一种额外的模态来改进什么。

Hi, Professor Levine. Thank you for the talk. I'm curious if you believe that high-resolution tactile data is required for doing fine grasping beyond Iris's or our token work, and I'm wondering if Iris is focusing on tactile as an additional modality in order to improve anything.

是的，好问题。触觉传感是一个有趣的领域。我的看法是：对于人类，我们有冗余的感官模态。触觉可以做很多事情，视觉也可以做很多事情，而且两者可以相互替代。所以，如果你在黑暗的房间里，视觉不太有用，但你可以用触觉来弄清楚你在哪里，找到电灯开关等等。但是，这种冗余也意味着，在规模化时，通常很难看到添加新传感器的边际收益，因为其他传感器可以替代它。我的猜测是，触觉传感没有更普及的一个很大原因是，有很多方法可以补偿它。例如，如果你有腕部摄像头，即使被基础摄像头遮挡，腕部摄像头也能看到发生了什么。

Yeah, good question. So, touch sensing is kind of an interesting area. So, here's what I would say. For people, we have redundant sensory modalities. So, there's a lot that you can do with touch. There's also a lot that you can do with vision, and one can substitute for the other. So, if you're in a dark room, vision is not so useful, but you can use touch to figure out where you are, find the light switch, that sort of thing. But, this redundancy also means that often times, at scale, it's hard to see the marginal benefit of adding a new sensor because other sensors can substitute for it. Now, my guess here is that a big part of why touch sensing hasn't taken off more is that there are many ways to compensate for it. So, for example, if you have wrist-mounted cameras, well, wrist-mounted cameras can see what's going on even when it's occluded from the base camera.

你知道，触觉对人的一部分作用是，当物体被手遮挡、视觉不太有用时，触觉能让人感知到它们。那么这对触觉意味着什么？可能意味着更多的感知更好。只是如果我们真的想在大规模上看到它的好处，我们可能需要深思熟虑，认识到机器人的传感器套件是一种平衡。这是一种在复杂性和信息量之间的平衡，即拥有更多信息与拥有更多干扰之间的平衡。我认为从长远来看，机器人将需要使用触觉，但在短期内，我们实际上已经看到没有触觉也能走得很远。所以，我不太确定该怎么看待这一点。我不认为这意味着触觉不好。我认为这只是意味着我们需要以一种深思熟虑的方式来处理它。

And, you know, part of what touch does for people is that it allows people to perceive things once they're occluded by their hands, and vision isn't so useful. Now, what does this say about touch? It probably means that more sensing is better. It's just that if we really want to see the benefit of it at scale, we probably should be thoughtful and recognize that the sensor package of the robot is kind of a balance. It's a balance between complexity, between having more information versus having more distractions and things like that. I think probably in the long run, robots will need to use touch, but in the short run, we've actually seen that we can get very, very far without it. So, that much I don't know what to make of it. I don't think it means that touch is bad. I think it just means that we have to approach it in a thoughtful way.

好的，我看到马克西米利安举手了。请讲。

Okay, so I see Maximilian, you have your hand raised. Please go ahead.

是的，谢谢你的时间。我对 VLA 当前状态的理解是，你们正在推动 VLA 在任务推理能力方面的进步。所以我的问题是，你有什么直觉关于如何推动更具反应性的模型以及能够随时间推理的模型？我在工作中也思考的一个非常具体的问题是，在多智能体机器人的场景中。例如，如果机器人必须一起抬起一张桌子，似乎当前的 VLA 在推理方面可以做到，它们会知道需要抬起桌子，但智能体能否实际计算出它们动作之间的时间配合？

Yes, thank you for your time. How I understand the current state of VLAs is that you're pushing the VLAs to progress in terms of the reasoning capabilities over the tasks. So, my question is, do you have any intuition on how we can push towards more reactive models and models that can also reason over time? And a very specific problem I'm thinking about in my work as well is in the setting of multi-agent robots. So, if the robots had to, for example, lift a table together, which seems like current VLAs could do it in terms of reasoning, they would know that they need to lift the table, but could the agents actually figure out the timeliness of their actions with respect to each other?

是的。我想说的是，推理不一定是语言形式的。用语言推理非常方便，因为碰巧当前生成语言的模型非常出色。但本质上，没有理由非得用语言。当然，有些场景下，更接地气的中间想法，例如用空间位置、图像或抽象空间关系来表达，可能比用语言表达的想法有用得多。当然，这里的一大挑战是我们需要合适的数据来学习生成这些东西。虽然视频等数据中有大量空间交互信息，可能非常有用，但语言数据结构化得很好。我认为从根本上说，如果我们暂时忘记是否需要语言或视觉等，只从信息内容的角度考虑数据，那么不同数据源之间的权衡就是它与机器人任务的关联程度和它的结构化程度之间的权衡。因为与机器人任务关联最紧密的数据是物理数据，而物理数据往往结构化程度最低，因为现实世界是最缺乏结构的。所以，绕了一大圈，我想说的是，从长远来看，我们可能会有思考型 VLA，它们通过多种模态（包括空间模态）进行思考，但要实现这一点，我们需要解决的一大挑战是如何监督这些模态，并在更接地气的数据和更解耦、具有良好语义表示的数据之间取得正确的平衡。

Yeah. So, what I would say here is that reasoning doesn't have to be verbal. It's very convenient to reason with language because it just so happens that current models that produce language are very good. But, there's no reason for it to be with language fundamentally. And certainly, there are scenarios where intermediate thoughts that are more grounded, that are expressed, for example, in terms of spatial locations or images, or kind of abstract spatial relationships might be a lot more useful than thoughts expressed with language. The big challenge, of course, with this is that we need the right kind of data to learn to generate these things. And while there is a lot of data of spatial interactions in things like videos, which could be very useful, the language data is very nicely structured. And I think fundamentally, if we sort of forget about whether we need language or vision or whatever, and we just think of data in terms of its information content, the trade-off that we have with different data sources is kind of a trade-off between how closely it connects to the robot task and how well structured it is. Because the data that connects most to the robot task is like the physical data, which is often the least structured because the real world is the least structured. So, this is a long-winded way of saying that probably in the long run, we will have thinking VLAs that think through many different modalities, including spatial modalities, but a big challenge that we have to address to get there is to figure out how to supervise those modalities and strike the right trade-off between having data that's more grounded and data that's more disentangled and has nice semantic representations.

好的，非常感谢。好了，很好。Roshi，我看到你举手了。请讲。

Okay, thank you very much. All right, great. Roshi, I see your hand's up. Go ahead.

是的，感谢你的精彩演讲。我有两个问题。一个是关于安全性，另一个是关于数据。关于安全性，我认为人工智能在现实世界中的一个瓶颈是安全探索。我在聊天中也看到了一些相关问题。例如，机器人可能会做出在现实世界中直接造成有害后果的事情。所以我想知道，你设想应该如何为这类机器人学习构建护栏？第二个问题是关于数据的。你特别提到了高质量数据，但似乎这里的高质量几乎等同于更多样化的数据。你提到了从不同实验室获取数据。我想知道，除了多样性之外，你认为还有哪些属性定义了机器人领域的高质量数据？

Yeah, thanks for the great talk. I have two questions. One is about safety, the other is about data. For safety, one bottleneck I see for AI in the real world is really the safe exploration. I also saw some related question in the chat. For example, the robot may do something that has direct harmful consequences in the real world. So, I wonder how you envision the guardrails should be built for this type of robotic learning. The second question is regarding data. You specifically talk about high-quality data, but it seems that high-quality here almost equals more diverse data. You talk about getting data from different labs. I wonder if there are other attributes as you see, apart from that diversity, that define high-quality data in robotics.

是的，非常好的问题。关于安全性，我认为无法回避的事实是，这本质上是一个非常困难的问题。问题在于，现实世界物理系统的安全性不仅仅是给系统施加正确的约束。它关乎拥有正确水平的理解。基本上，如果你希望机器人避免踩到环境中易碎的物体，它需要非常擅长判断哪些物体是易碎的。因此，安全性的最大挑战通常归结为对世界有足够好的理解，从而能够避免不安全的行为。我认为实际上推理模型为此提供了一个非常强大的要素。原因是安全性在意外情况、长尾情况下最为困难。对于长尾情况，可能很难有足够的具身第一人称经验覆盖，使你能够在这些情况下正确行动。但是，你可能拥有其他经验，能提示你在那些意外情况下可能发生什么。这最容易通过想象自己开车来理解。如果你正在开车，发生了一件非常奇怪的事情，比如一头大象走到路中间。你从未见过大象在路中间，但你有足够的常识意识到，如果你撞上大象，会发生不好的事情。你可能应该停下来。你应该去别的地方。也许你应该倒车。别管大象。别靠近大象。这就是常识。这不是来自你开车的经验。它来自你对世界如何运作的一般理解。而机器人基础模型可以带来这种常识，从基础模型的预训练中获得。所以，我实际上认为这类推理模型可能是迈向更安全基础模型的非常重要的一步。但我认为这是一个难题。还有很多工作要做。关于数据，我想说的是，我们对预训练数据的需求通常与对后训练数据的需求非常不同。同样，这与 LLM 等常规基础模型没有太大区别。对于预训练数据，你需要覆盖。你需要覆盖现实世界中发生的各种事情。在后训练数据中，你需要的是一致性。

Yeah, very good questions. So, in regard to safety, I think there's no way out of the fact that this is fundamentally a very difficult problem. So, the trouble is that safety for real-world physical systems is not just about putting the right constraints on the systems. It's about having the right level of understanding. Basically, if you want the robot to avoid walking over a fragile object in the environment, it needs to be very good at figuring out which objects are fragile. So, the biggest challenge with safety usually comes down to having a good enough understanding of the world that you can avoid the unsafe behaviors. I think that actually the reasoning models provide one very powerful ingredient to this. And the reason is that safety is hardest in unexpected situations, in long-tail situations. And for long-tail situations, it might be very difficult to have enough coverage of embodied first-person experience that allows you to correctly act in those situations. But, you might have other experience that gives you a hint of what might happen in those unexpected situations. This is easiest to understand if you imagine yourself driving a car. If you're driving a car and something very strange happens, maybe an elephant walks into the middle of the road. You've never seen an elephant in the middle of the road, but you have enough common sense to realize that if you drive into the elephant, something bad is going to happen. You should probably stop. You should go somewhere else. Maybe you should back up. Leave the elephant alone. Don't go near the elephant. And that's common sense. That's not coming from your experience driving the car. It's coming from your general understanding of how the world works. And robotic foundation models can bring to bear this kind of common sense, getting it from the foundation model pre-training. So, I actually think that these kind of reasoning models might be a very important step towards significantly safer foundation models. But, I think it's a hard problem. I think there's a lot more work to be done. In regard to data, what I would say is what we need for pre-training data is often very different than what we need for post-training data. And again, that's not that different from regular foundation models like LLMs. For pre-training data, you want coverage. You want coverage of the kinds of things that happen in the real world. In the post-training data, what you want is consistency.

你希望它持续教你好的策略，这样你就能正确运用预训练的知识。所以，这些要求非常不同。谢谢。好了，很好。我看到还有很多问题，很多人也很感兴趣，但考虑到时间，我非常尊重你的时间，Sergey。所以，我想是时候结束了。再次感谢 Sergey 的精彩演讲，也感谢大家的参与。希望下次再见。谢谢大家。谢谢。谢谢。

You want it to consistently teach you good strategies so that you can bring to bear the knowledge from pre-training correctly. So, these things are very different requirements. Thank you. All right, great. I see there are still a lot of questions and also like many people are very interested, but looking at the time and I want to be very respectful for your time, Sergey. So, I think it's the time for us to wrap up. So, thanks Sergey for this great talk again and thank you all for your engagement. Hope to see you all next time. Thank you all. Thank you. Thank you.