从 GPT 到机器人：物理 AI 的扩展之路

对我来说，这有点令人震惊，因为基础模型根本没有用任何人类数据进行训练。这是谢尔盖·拉文，世界顶尖的机器人研究者之一，我向他询问了人形机器人的现状。你迄今为止见过的最令人惊叹的涌现能力是什么？

To me, that's kind of mind-blowing because like the base model wasn't trained on any human data at all. This is Sergey Levine, one of the world's leading robotics researchers, and I asked him all about the current state of humanoid robotics. What are the most astonishing emergent capabilities you've seen so far?

我认为任何观看那次评估的人都不会想到机器人会这样做。

I don't think anybody watching that eval thought that the robot was going to do this.

我这里有一些关于中国的问题。有时候是避不开的。如果人形机器人在个位数年内没有成功，你认为最可能的原因是什么？这是完整的一集。LLM 在该领域造成了前所未有的影响和投资。物理 AI 和人形机器人可能更大。所以今天我想问你，我们现在在人形机器人方面处于什么位置，你预见到这项技术将如何实际部署到世界中？

I have some questions here on China. There's no avoiding it sometimes. If humanoid robotics did not succeed in singledigit years, what do you think would be the most likely reason why humanoid robotics failed? Here's the full episode. LLMs have caused unprecedented impact and investment in that area. Physical AI and humanoid robots could potentially be even bigger. And so I wanted to ask you today about where are we today with humanoid robotics and how do you foresee this technology actually being deployed into the world.

我想，通过机器学习，我们在过去几年，甚至过去十年里学到的是，当你大规模进行时，它是有效的。这现在很明显，但并非总是如此。但有一个警告，那就是你必须扩展正确的东西。最初，当人们开始研究语言模型时，例如，主导设计是 LSTM。有些人还记得那些是什么。它们还算不错，比之前的要好得多，但它们的扩展性并不好。而 Transformer 的重大之处并不是说 Transformer 在数学上特别优雅或什么的。只是它们扩展得更好。所以它们更容易用大量数据和大量参数进行训练。所以技术是分阶段进行的。首先你弄清楚什么可以扩展，基本上什么是可扩展的技术，然后你投入更多的工业规模的努力，添加大量数据，增加模型大小，这时魔法就发生了。所以当我们做更基础的技术开发时，关键是理解那些可扩展的杠杆是什么。所以弄清楚设计，弄清楚大致的混合，这本身就能做一些很酷的事情，但这不是真正改变世界的东西。当你开始拉动那个杠杆时，事情才真正改变。所以对于 LLM，当第一批 GPT 模型出来时，GPT2 做了一些事情，但有点像杂耍，对吧？所以你可以让它编一个关于秘鲁独角兽的故事，它是连贯的英语，但它不是能解决许多现实世界问题的东西。但从事这项工作的人认识到，嘿，有一些神奇的事情发生，因为当你添加更多数据并让模型更大时，这些东西变得更连贯、更有效。所以他们可以看到，如果我们做更多这样的事情，它就会变得更强大。所以回到你的问题，关于机器人技术，我要说的是，它不在 GPT4 到 GPT5 的阶段，即工业规模的努力使模型更大并获得更多能力。它处于我们正在建立基础技术的阶段。正因为如此，现在应该期望的不是每个月模型都按照某种可预测的扩展曲线变得更大、更强大。而是扩展属性本身随着我们开发正确的技术而演变。所以把这带回更接近现实的东西，我对我们在 Physical Intelligence 这里做的演示非常满意，我认为其他人得出的很多结果真的很酷，但这些都是，把它们放在背景中，我们不应该期望这些是真正展示规模力量的东西。我们应该期望它们是在开发之后将被扩展的基础技术。所以我认为我们现在所处的位置是，我们实际上正在把所有拼图碎片放到位。我认为它实际上非常接近。我认为很多拼图碎片正在就位。但并不是说，让预测技术走向变得如此困难的是，它还没有处于那个可预测的扩展阶段。它处于我们正在弄清楚拼图碎片的阶段，我认为这非常令人兴奋，但这意味着也很难预见那上面的系数会是什么。

I guess with machine learning, what we've learned over the last few years, over the last decade rather, is that it works when you do it at scale. And this is like very obvious now, but it wasn't always obvious. But there's a caveat, which is you have to scale the right thing. And initially, when people started working on models for language, for example, the dominant design was LSTMs. Some people remember what those are. They were like kind of okay, they were a lot better than what came before that, but they didn't really scale as well. And then the big thing with transformers was not that transformers were somehow particularly mathematically elegant or anything like that. It's just that they scaled better. So they were easier to train on very large amounts of data with lots of parameters. So the technology proceeds in phases. First you figure out what you can scale, basically what is the scalable technology, and then you pour on a lot more of an industrial scale effort, adding lots of data, adding to model size, and that's when the magic happens. So when we're doing more fundamental technology development, the key is to understand what are those scalable levers. So figure out the design, figure out roughly the mixtures, and that by itself does something pretty cool, but that's not the thing that actually changes the world. It's when you start pulling that lever that things actually change. So with LLMs, when the first GPT models came out, with GPT2, it did some stuff, but it was sort of like a parlor trick, right? So you could get it to synthesize a story about unicorns in Peru or something, and it was coherent English, but it wasn't a thing that would solve lots of real-world problems. But the folks that worked on this kind of recognized that hey, there's something magical that happens because as you add more data and you make the model bigger, this stuff gets more coherent and more effective. So they could see that if we do a lot more of that, then it'll become a lot more powerful. So to come back to your question, what I would say about robotics is that it's not in the GPT4 to GPT5 stage where it's like an industrial scale effort to make the model bigger and get more capability out of it. It's in that stage where we're establishing the fundamental technologies. And because of that, what one should expect to see right now is not necessarily that each month the model gets bigger and more powerful by some predictable scaling curve. It's that the scaling properties themselves are evolving as we develop the right technologies. So to bring this back to something closer to reality, I'm very happy with the demos that we're doing here at Physical Intelligence, and I think that a lot of the results that other people are coming out with are really cool, but these are, to put them in context, we should not expect these to be the things that are actually illustrating the power of scale. We should expect them to be developing the fundamental technologies that will be scaled up after that. So where I think we're at now is that we're actually getting all those puzzle pieces in place. And I think it's actually very close. I think a lot of the puzzle pieces are falling in place. But it's not like, what makes it so hard to prognosticate about where the technology is going to go is that it's not yet at that predictable scaling stage. It's at the stage where we're figuring out the puzzle pieces, which I think is really exciting, but it means that it's also very, very hard to foresee the coefficients on that will be.

你迄今为止见过的最令人惊叹的涌现能力是什么？

What are the most astonishing emerging capabilities you've seen so far?

那是最有趣的事情，当然随着我们的进展，我们看到了更多这样的情况。就像一开始，是一些小事情，但它们有点神奇，因为在机器人技术中，基本上在 2024 年之前，这种事情从未发生过。所以发生的小事情，这可能是大约两年前，我们会看到这样的情况：好吧，我们训练了折叠衣物的策略，它从篮子里拿出单独的衬衫，试图折叠它们。然后我在 2024 年末有一个非常生动的记忆，我们在看一次评估，它同时拿出两件衬衫，我看着这个，我想，好吧，它完了，它不可能做到这一点，然后它把两件衬衫放在桌子上，解开缠绕，把其中一件放回去，然后开始折叠另一件。就像哇，那样。好吧，事后你可以做一些侦探工作，弄清楚它从哪部分训练数据中得到了这个，但那是那些时刻之一，我认为任何观看那次评估的人都不会想到机器人会这样做。这就像一件小事。它展示了你期望人们拥有的常识。但实际上，我最近觉得更有趣的是某些错误，因为 LLM 的一个相当显著的特点是，一旦它们变得足够好，即使是错误也有意义，在某种意义上，它们不是疯狂的错误，比如输出总是 ZZZ，而是那些在语义上合理的错误。去年我们对 Pio5 进行了一次评估，机器人正在清理厨房，它被告知把所有餐具收起来。比如有一些勺子、铲子等。它试图打开它认为放银器的地方的抽屉，但它打不开抽屉。

That is the thing that is the most fun, and certainly we've seen a lot more of that happening as we progress. Like in the very beginning, it was kind of little things, but they were kind of magical because in robotics, basically prior to 2024, the stuff like never happened. So the little things that happen, and this was maybe at this point about two years back, we would see things like okay, we train our policy for folding laundry, and it takes out individual shirts out of the hamper and tries to fold them. And then one very vivid memory I have in late 2024 is we were watching one of the evals, and it takes out like two shirts at the same time, and I'm watching this, I'm like okay, like it's done for, there's no way it can possibly do this, and then it puts the two shirts on the table, disentangles, and puts one of them back, and then starts folding the other one. It's like wow, like that. Okay, in retrospect you can do some detective work and figure out where it got that from some piece of training data, but that was one of those moments where I don't think anybody watching that eval thought that the robot was going to do this. It's like a little thing. It's like exhibiting the common sense you expect people to have. But actually the thing that I find more interesting recently is some of the mistakes, because one of the things that was pretty remarkable about LLMs is that once they got good enough, even the mistakes kind of made sense in the sense that they weren't crazy mistakes where the output is just ZZZ all the time, but they were mistakes that sort of semantically are sensible. We had an evaluation last year for Pio5 where the robot was cleaning up a kitchen and it's told like put away all the utensils. Like there's some spoons, spatulas, etc. And it tries to open the drawer where it thinks the silverware goes, and it can't get the drawer open.

于是它滑过去，打开旁边的烤箱，然后开始把这些东西放进烤箱。你可以想象，如果你让一个孩子收拾东西并放好，他们可能会决定这么做，因为，好吧，那是个容器，你可以把东西放进去，没人会看见。我们做的另一个实验是洗盘子。它会拿起盘子，用海绵清洗，然后放到晾干架上。这是一个关于记忆的实验，因为它必须跟踪自己所做的每件事。它有一个类似便签本的记忆，它会记下，比如，嘿，我有三个盘子。我洗了灰色的那个。我洗了绿色的那个。然后它把其中一个掉在地上，把底座开过去，这样你就看不见了。我就想，"好吧，我已经洗了灰色的盘子，它完成了。"所以，显然这些不是我们希望看到的结果，但有趣的是，有些错误几乎就像孩子试图完成任务时会犯的错误。所以现在它只需要成长。

So it slides over and opens the oven which is right next to it and then starts putting this stuff in the oven. It's like, you know, you can sort of imagine that if you ask like a child to clean stuff up and put it away, like they might decide to do that because, okay, it's like a container and you can put stuff there and nobody sees it. Another experiment we had is washing all the plates. So, it would pick up the plates, wash them with a sponge, and put them on the drying rack. And this was an experiment on memory because it has to keep track of everything that it's doing. It has like a scratch pad kind of memory where it's writing down like, hey, you know, I had three plates. I cleaned the gray one. I cleaned the green one. And then it drops one of them on the floor and drives the base over. So you can't see. I was like, "Okay, I've cleaned the gray plate. It's done." So I mean, obviously these are not the things that we want to see, but it's kind of interesting that some of the mistakes, they're almost like what you would associate with a child trying to do the task. So now it just needs to grow up.

你提到了这些进步。它们正在整个行业发生。我知道有很多人在制造人形机器人，有 Figure，有特斯拉，还有很多其他竞争对手，而你拥有关于什么困难什么不困难的专业知识。当你看到这些竞争对手时，有没有什么进步或成就让你觉得，"哦，那真的很令人钦佩，那很了不起。"

You mentioned the advancements. They're kind of happening all over the industry. And I know there's a lot of people building humanoid robots. There's Figure, there's Tesla, and there are many other competitors, and you have the expertise of what is hard and what isn't. When you look at the competitors, has there been any advancement or achievement where you think, "Oh, that's really admirable and that's impressive."

实际上，我认为行业中最鼓舞人心的事情之一就是看到自动驾驶系统所取得的腾飞，因为你知道，有时对机器人研究者的批评是，这就像核聚变，是未来的技术，但永远在未来。但人们也曾这样评价自动驾驶，而现在，我们在旧金山，你可以走出去，坐上一辆 Waymo，它真的能带你去目的地，而且没有司机坐在那里。所以，不深入技术细节，我认为真正鼓舞人心的是这个案例和证明，是的，你确实可以拥有这些未来技术之一，而且它确实落地了。我认为它在 2020 年代中期落地并非偶然，因为大规模机器学习的许多拼图正在达到我们可以将它们与真实物理系统结合的水平。当然，驾驶和机器人操作之间有很多不同，但我认为我们确实能在真实物理世界中落地基于学习的技术，这真的很鼓舞人心。

Actually, I think that one of the most inspiring things to me in the industry is to see the kind of takeoff that autonomous driving systems have had, because you know, one of the criticisms that is sometimes leveled against robotics researchers is, you know, it's like nuclear fusion, it's the technology of the future but it's always in the future. But that's what people said about autonomous driving too, and now, you know, we're in San Francisco, you can go outside, you can take a Waymo, and it will actually take you to your destination and there's no driver sitting there. So, without getting too much into the technical details, I think what's really inspiring about that is just this case and point that yes, you can actually have one of these technologies of the future and it actually does land. And I think it's not an accident that it's landing now in the mid-2020s, because a lot of the puzzle pieces for large-scale ML are getting to the level where we can put them together with actual physical systems. And there's a lot of differences between driving and robotic manipulation, of course, but I think the illustration that we can actually land learning-based technologies in the real physical world, I think that's really inspiring.

如果 OpenAI 开始更大力投资机器人技术，你认为那会如何影响这个行业？你认为竞争对手会担心吗？

If OpenAI started investing more heavily into robotics, how do you think that would impact the industry? Do you think competitors would be worried about that?

机器人技术是一个领域，也许公平地说，其生态系统不如机器学习其他领域那么健康。我的意思是，计算机视觉和自然语言处理天然适合基于机器学习的生态系统，因为有免费可用的数据。人们普遍接受他们会使用学习。你知道，没有像安全那样严重的担忧，至少是物理安全，对吧？你知道，人们理所当然地担心人工智能安全，但这不同于物理设备造成物理伤害。正因为如此，我认为在这些领域更容易启动一个非常严肃的大规模机器学习项目。机器人技术不是这样。机器人技术传统上不是一个真正拥抱数据共享的学科，例如。所以我认为，围绕机器人学习活动越多，就越会推动人们的思维转向这样一个未来：我们接受机器人将由学习模型控制，而不是手工设计的控制器；会有数据，数据需要共享，因为没有人能在单一垂直领域构建出真正的基础模型。基本上，它将把整个机器人技术的思维转变为更像我们看待视觉和自然语言处理的方式，而不是传统的工厂自动化。所以从这个意义上说，尽管我为我们在物理智能方面所做的工作感到自豪，但我认为需要不止一家公司来推动每个人的思维朝那个方向转变。

Robotics is an area where, maybe to be fair, the ecosystem hasn't been as healthy as it has in other areas of machine learning. And what I mean by that is that computer vision and NLP are things that sort of lend themselves naturally to a machine learning based ecosystem because there's freely available data. People kind of have a general acceptance that they're going to be using learning. You know, there aren't concerns as severe concerns about safety, at least physical safety, right? You know, people rightfully are concerned about AI safety, but it's not the same as a physical device causing some physical harm. And because of that, I think it's a bit easier to spin up a very serious large-scale ML effort in those areas. Robotics is not like that. Robotics traditionally is not a discipline that really embraces sharing of data, for example. So I think that the more activity there is around learning in robotics, the more I think it'll shift people's thinking towards this kind of future where we accept that robots will be controlled by learned models, not by hand-designed controllers, that there will be data, that data will need to be shared, because there's no way that somebody can build out a true foundation model in a single vertical. And basically, it will shift the entire thinking around robotics to look more like how we think about vision and NLP, as opposed to traditional factory automation. So I think in that sense, much as I'm proud of the work that we're doing in physical intelligence, I think it'll take more than one company to shift everyone's thinking in that direction.

作为旁观者，我在推特上看到这些令人印象深刻的中国机器人演示。感觉他们在某种程度上领先了，但我没有深厚的领域专业知识。所以我很好奇你的想法，你怎么看中国的机器人技术，他们是不是走得更远？

As a bystander, I see on Twitter these really impressive demonstrations of Chinese robotics. It almost feels like they're ahead in some sense, but I don't have that deep domain expertise. So I was curious your thoughts on, you know, what do you think of China's robotics and, you know, are they further along?

我认为，对我们这些在美国和欧洲从事这些工作的人来说，有一件非常有用且具有建设性的事情，就是问一问要吸取什么教训？对我来说，一个教训是拥有一个健康的生态系统很重要。生态系统意味着，显然应该有优秀的研究人员、优秀的工程师从事这些工作。应该有健康的开源，但这也意味着，为机器人技术做出贡献的不同行业都需要各自非常健康。而这些行业不仅仅是计算机科学、机器学习和模型构建，还包括供应链、制造、硬件研发。这些都是非常重要的方面，其中一些方面美国做得相当好，另一些方面美国则有点放任自流了。我认为我们应该做的是，看看世界上正在发生的事情，看看中国实验室和其他国家实验室取得的一些优秀成果，我们应该吸取教训，努力构建一个更健康的生态系统，这意味着要投资于所有不同的方面。所以，你知道，我不是一个商人，也不是一个投资者。

I think that one thing that is very useful and constructive for us to do, those of us that work on these things in the United States and in Europe, is to ask what is the lesson to learn? And to me, one lesson is that it's important to have a healthy ecosystem. And ecosystem means that there should be, you know, obviously good researchers, good engineers working on these things. There should be healthy open source, but it also means that the different industries that contribute to robotics need to individually be very healthy. And those industries are not just the computer science, ML, and model building stuff, it's also supply chains, manufacturing, hardware R&D. These are all very important things, and aspects of those things are things that the United States does quite well. Other aspects of them are things where the United States has sort of let things go a little bit. And I think that what we should do is we should look at what's going on in the world, look at some of the excellent results that Chinese labs are doing, that labs in other countries are doing, and we should take away that lesson that we should strive to build a healthier ecosystem, and that means investing in all the different facets that contribute to this. So, you know, I'm not much of a business person. I'm not much of an investment person.

所以我不能说我知道该怎么做，但我认为重要的是要完全接受这是一个整体性的事情，而不是我们只做其中一部分，然后把其他所有事情都外包出去。

So I don't I can't claim to know how to do this, but I think that it's important to sort of fully embrace that this is like a holistic thing and not something where we can do just like one piece of it and like outsource everything else basically.

在构成这个生态系统的所有部分中，作为美国的一名机器人研究员，如果哪一部分变得更好，会对机器人技术的进步产生最大的影响？

Of all the pieces of the ecosystem that are parts of it, as a robotics researcher in the US, if it was better, that would have the biggest impact on the advancement of robotics?

是的，我认为可靠的低成本硬件的可用性确实很重要。目前，对于机器人研究使用的硬件，很多确实来自中国，而且很好，相对便宜，质量高，符合人们通常需要的标准。但如果也能在国内采购所有这些硬件，那就太好了。我不认为这有什么不可能的。我认为这只是需要接受整个生态系统都需要得到支持，而不仅仅是其中的一部分。

Yeah, I think certainly availability of reliable low-cost hardware is a big deal. And right now, I mean certainly for hardware used for robotics research, a lot of that does come from China, and it's good, it's relatively inexpensive, it's of high quality, and it meets the standards that people generally need. But it would be awfully nice to be able to source all that domestically as well. And I don't think there's anything impossible about that. I think it's just a matter of embracing the fact that the entire ecosystem needs to be supported rather than just one piece of it.

你可以想象，第一个达到规模的实验室将会第一个突破。可能存在这种指数增长效应：一旦部署，部署帮助你增长更快，增长更快又帮助你部署，你知道这个飞轮。那么你相信这会发生在这个行业吗？无论是美国还是中国的某个实验室，会达到某个突破点，然后超越其他所有人？

You can imagine that the lab that is first to get to scale will be the first to break out there. There may be this exponential growth effect where once you deploy, deployment helps you grow faster, and growing faster helps you deploy, and you know this flywheel. So do you believe that will happen in this industry, where one lab, whether in US or China, will hit some breakout point and they'll kind of jump everyone else?

这有很多道理。我认为有一个重要的细节需要记住。这个细节是，你必须扩展正确的东西。所以我认为基本上，我这样表述：拥有一个有效的正反馈循环，即更多部署的机器人转化为更强的模型能力，这是关键。这完全合理。诀窍在于有很多方法可能做错。所以我告诉你几个明显的例子。一个明显的例子是，假设我是一家汽车公司，我有一个机械臂在焊接汽车，它在装配线上，每天焊接汽车，每月进行一百万次焊接等等。如果我仅仅把它作为我的数据飞轮，我不太可能得到比焊接汽车的机器人更有能力的东西。所以，既然机器人已经在那里并且已经在焊接汽车，那么它的边际改进可能并不那么有价值。这就是一个例子，说明你必须扩展正确的东西，因为数据不是那么可互换的，它不像电或石油。你不能直接买更多。它必须是多样化的。所以我认为基本上这是对的，但你必须用正确的技术和正确的多样化学习来源来扩展正确的东西。数据更像是对你的机器人的教育计划，而不是一种可互换的商品。

There is a lot of truth to that. I think that there's an important detail to keep in mind. The detail is that you kind of have to scale the right thing. So I think that basically the statement, the way that I would phrase this, is having an effective positive feedback loop where more deployed robots translates to more model capability. That's kind of the key. And that makes total sense. The trick is that there are lots of ways that it can be done wrong. So I'll tell you a few obvious ones. Like one obvious one is, let's say that I'm a car company and I have a robotic arm that is welding cars, and it's there on the assembly line and it welds cars every day and it gets like a million welds every month and so on. If I just use that as my data flywheel, I'm unlikely to get something more capable than a robot that welds cars. So since the robot is already there and it's already welding cars, presumably the marginal improvement for that is not all that valuable. So that's an example of how you kind of have to scale the right thing because data is not quite as fungible, it's not like electricity or oil. You can't just buy more of it. It has to be heterogeneous. So I think that basically it's right, but you have to scale the right thing with the right technology and the right kind of source of diverse learning. Data is more like an education program for your robot than it is a fungible commodity.

如果你考虑那个数据飞轮，我想推出一个合适的平台，它能产生重要的数据，然后改进那个平台。你什么时候第一次看到这种情况在世界上发生？

If you think about that data flywheel and I guess putting out the proper platform that would generate data that matters, that would then improve that platform. When do you first see this kind of happening in the world?

我认为在技术方面，事情正朝着那个方向快速发展。我认为可能有一个需要权衡的平衡点，它将显著决定那个时间线，那就是某人愿意承认多少结构。所以，在一个极端，你可以想象直接跳到完全非结构化的部署领域，比如家庭机器人，那可能非常令人兴奋，因为那样你一开始就有很多多样性，但在那个领域要足够有效和安全，门槛要高得多。安全在那里是更大的问题，因为它是在人们的家中。在另一个极端，你可以想象更结构化的任务。也许不完全是焊接机器人，但有点像走廊那头的机器人，做更非结构化的事情，对吧？那可能是一个容易得多的领域，因为安全顾虑少得多，因为它可能在训练有素的人类周围。任务可能更可预测等等，但你在那个领域获得的每一点数据的边际价值较低，因为多样性较少。所以就像你可以更早起飞但斜率较小，或者更晚起飞但斜率较大。你必须校准这一点。但我的感觉是，无论我们谈论哪个极端，现在都是个位数年而不是双位数年。所以可能更结构化的可能现在或明年就会发生。较非结构化的可能还需要几年，但可能不会像十年那么久。

I think on the technology side, things are advancing very rapidly towards that. And I think that maybe a particular balancing act to strike that would significantly determine that timeline is kind of how much structure somebody's willing to admit. So, on the one extreme, you could imagine jumping straight to fully unstructured deployment domains like home robots, and that could be really exciting because then you have a lot of diversity right off the bat, but the bar is a lot higher to be effective enough in that domain to be safe enough. Safety is much bigger issue there because it's around people in their home. On the other extreme, you could imagine much more structured tasks. Maybe not quite the welding robot, but sort of like the robot down the hall that does something more unstructured, right? And that could be a much easier domain in the sense that there's much less safety concerns because it might be around trained humans. The task might be more predictable and so on, but the marginal value of each bit of data you get in that domain is lower because there's less variety. So it's like you could take off earlier but with a smaller slope, or later but with a larger slope. And you kind of have to calibrate that. But my sense is that whichever end of that extreme we're talking about, it's in the single-digit years rather than the double-digit years at this point. So it could be that the more structured ones might be happening like now or next year. The less structured ones might be a few more years out, but probably not like a decade.

很多人谈论时间线，这有点模糊。我想知道如果你要制定一个路线图，但不是那么具体，只是朝着“机器人进入我家做……”这个北极星的里程碑。

A lot of people talk about timelines and it's kind of more nebulous. And I'm wondering if you were to put down a somewhat of a road map, but not so concrete, but just milestones towards that north star of robot in my home doing...

是的，这是一个非常好的问题。所以也许在回答之前，我认为关于机器人技术有一件非常重要的事情要说，这很容易从炒作周期和人们发布的演示中忽略，那就是机器人技术的难点始终是泛化。但当有人展示他们的系统演示时，仅凭演示通常无法清楚展示所展示的泛化水平。例如，高度杂技的机器人演示看起来确实令人兴奋。但通常如果它有点舞台化，有时它真的在舞台上，如果是像表演一样，那显然是排练过的，这没关系，因为它真的只是一场表演，但它不同于在任何家庭中每次都可靠地完成一项任务。而且泛化部分在孤立看待时往往看起来不那么令人印象深刻，因为泛化是许多试验的属性，而不是一次试验的属性。所以你可能会看到机器人做一些相当平凡且不令人印象深刻的事情。但令人兴奋的是，它是在一个从未见过的物体上、在一个从未测试过的环境中做的。这实际上比做一个练习了数百万次的杂技后空翻更难。所以话虽如此，我对你问题的回答是，路线图完全在于实现更好的泛化，以及那种二阶效应，即在你泛化时获得更多泛化的机制。

Yeah, that's a really good question. So maybe to preface this answer, I think that there's one thing that is very important to say about robotics that is very easy to miss from the hype cycle and the demos that people put out, which is that the hard thing in robotics was always generalization. But when somebody shows a demonstration of their system, the demonstration alone usually doesn't make it clear what level of generalization is being shown. The highly acrobatic robot demos, for example, are really exciting to look at. But typically if it's something that is a little bit more staged, sometimes it's literally on stage, if it's like a show, that is obviously rehearsed, and that's okay because it's literally just a show, but it's not the same as doing a task every time reliably in any home. And the generalization piece often doesn't look that impressive when viewed in isolation because generalization is sort of a property of many trials, not of one trial. So you might see the robot doing something fairly mundane and unimpressive. But what's exciting about it is that it's doing it with an object that it's never seen before in an environment that it's never been tested in before. And that's actually harder than doing an acrobatic backflip that it's practiced millions of times. So with that said, my answer to your question is that the road map is all about both achieving better generalization and that second order effect of having a mechanism to get more generalization as you generalize.

所以这条路线图上的一个步骤是，要有一个非常具体的演示，展示一个机器人系统通过自主收集经验而变得更好，而这些经验是在它最初未受训练的环境中收集的。我在后端、实验室里做我该做的事，拿到我的模型和自适应算法，然后把它放到一个新环境里。也许是家庭，也许是工厂，总之是在做真实的事情。它一开始表现还行，但随着时间的推移，它会越来越好，直到达到实际相关的鲁棒性水平，而不会卡在 50% 左右。我认为那会是一个重要的里程碑。因为那说明，如果这真的是一个自动化过程，它会在收集有用经验的同时不断改进。然后我就可以把它放到很多不同的领域，收集有用的经验，做人们真正想要的事情，它还会改进模型。所以我认为这是这条路上非常重要的一步。

So one of the steps on that roadmap is to have a very concrete demonstration of a robotic system that gets better with autonomous experience collected in a setting it wasn't originally trained for. I do whatever I do on the back end, in the lab, and get my model, my adaptation algorithm, and put it in a new setting. Maybe it's a home, maybe a factory, something where it's doing something real. It does okay, but over time it gets better and better, to the point where it reaches practically relevant levels of robustness without capping out at like 50%. That I think would be a major milestone. Because now that says okay, if this is truly an automated process, it's improving, getting better even as it collects useful experience. Now I can take it and put it in lots of different domains, collect useful experience, do something people actually want, and it'll improve the model. So that is a really major step on that road.

我认为另一个非常重要的步骤是，展示一种非常具体且实用的方式来迁移知识、迁移常识，以实现鲁棒性。这是另一种场景，你没有机会练习。如果你在开车时看到一辆消防车和一堆交通锥，即使你以前从未遇到过这种情况，你的常识会告诉你，‘嘿，我应该减速。’也许我不知道如何做出最优反应，但我不应该直接冲过锥桶，惹恼消防员。这就是常识。如果你能运用这种常识来有效应对意外情况，比如当机器人把锅铲放进烤箱时，它应该打开烤箱把它拿出来，因为它从语义上知道那是不对的。我认为这是另一个重要步骤，因为它表明我们可以用常识来纠正错误。

I think another really major step is to demonstrate a very concrete and practically useful way to transfer knowledge, to transfer common sense, to achieve robustness. That's the other scenario where you don't get to practice. If you're driving your car on the road and you see a fire truck and a bunch of traffic cones, even if you've never been in that situation before, your common sense tells you, 'Hey, I should slow down.' Maybe I don't know how to react optimally, but I shouldn't just barrel through the cones and upset the firefighters. So that's common sense. And if you can apply that common sense to effectively recover from unexpected situations, like when the robot put the spatula in the oven, it should probably open it up, take it out, because it knows semantically that's not the right thing. That is another important step because it tells us we can use common sense to fix mistakes.

我可以想象一个更窄范围的机器人。假设它是一个人形机器人，在制造业的装配线上负责一个步骤。据我理解，你对那个不太感兴趣，因为那并不是迈向通用智能的一步。

I could imagine a more narrowly scoped robot. Let's say it's a humanoid robot that's one step in an assembly line in manufacturing. In my understanding, you're less interested in that because that's not really a step towards that general intelligence.

我想说的是，并不是我不感兴趣，而是我认为现实世界有这些泄漏的抽象，使得这类事情比看起来复杂得多。让我用一个类比来解释。在 90 年代，当人们开始全力投入自动驾驶时，有一种想法是，我们可以通过给环境加装一些设备来避免很多难题。比如在高速公路上安装磁性传感器，汽车上装一个传感器和一个发射器，这样就能知道每辆车的位置，就像飞机那样。然后人们认为我们不需要花哨的 AI，只要有这些传感器就能行。那基本上没有成功，因为现实世界有太多混乱的例外和特殊情况，即使 99% 的时间磁性传感器能让车行驶，那 1% 的情况，比如有人走到路中间或者有垃圾，就会把一切都搞砸。所以真正有效的是当人们说，‘嘿，我们不打算回避难题。我们要把车部署到旧金山这样混乱的地方，而不是荒郊野外，让我们正面应对。’这让那个社区取得了进展。我认为机器人操作也会是同样的道理。在工厂这个完全结构化的世界之外，哪怕只走出去一点点，即使 99% 的时间都很简单，那 1% 的意外情况就意味着你真的需要解决整个问题的全部范围。

What I would say, it's not that I'm less interested in it. It's that I think the real world has these leaky abstractions that make that kind of stuff a lot more complex than it seems. Let me try to explain this with an analogy. In the '90s, when people started working full steam on autonomous driving, there was this idea that we could avoid a lot of the hard problems by instrumenting the environment a little bit. We'll have magnetic sensors along the highway, and cars will have a sensor and a transmitter so they can tell where each of the cars are, kind of like the way you do it with aircraft. Then people thought we don't really need fancy AI; we'll just have these sensors and it'll just work. That basically didn't go anywhere because the real world has so many messy exceptions and special cases that even if 99% of the time the magnetic sensors allow the car to drive, the 1% when someone steps in the middle of the road or there's a piece of trash, it just messes everything up. So the thing that actually worked was when people said, 'Hey, we're not going to try to avoid the hard problem. We're going to deploy our cars not in the middle of nowhere but in San Francisco, very messy, and let's deal with it head-on.' That allowed that community to make progress. I think robotic manipulation is going to be the same way. Past the fully structured world of the factory, if you want to go even a little bit outside of that, even if 99% of the time it's straightforward, that 1% when something weird happens means you really need the full scope of the problem to be addressed.

这让我想起，我记得 Figure 有一个演示，他们直播了机器人分拣包裹。你看那个演示时，你认为它展示了泛化能力吗？

This kind of reminds me, I remember Figure had this demo where they live streamed the robot sorting packages. When you watched that demo, does it demonstrate generalization in your opinion?

是的，我认为它展示了。顺便说一句，我觉得这非常令人鼓舞。正如我提到的，在视频中展示泛化很难。很明显，很多人在思考如何呈现一些东西，让人一眼就能看出泛化是什么。我认为你可以看到很多创造性的步骤。直播演示、这种超长的时间流逝。我认为这是一个好主意，也是提升泛化在人们意识中的重要性的好方法。

Yeah, I think it does. And by the way, this is something that I find very encouraging. As I mentioned, it's hard to show generalization in a video. It's clear that lots of people are thinking about how to present something that somebody can watch and take in at a glance what generalization is. I think you can see a lot of creative steps towards that. Live demos, these kind of really long time lapses. I think it's a great idea and a really nice way to move towards elevating the importance of generalization in people's consciousness.

当我们去年年底做 PI star 6 项目（我们做的强化学习项目）时，我们想做更长时域的实验。有些情况很明显，比如我们的机器人在 Dandelion 巧克力工厂组装盒子。那是一个真正的巧克力工厂，所以他们需要盒子。我们运行了好几天。但我们还有这个咖啡任务，机器人用浓缩咖啡机做浓缩咖啡。我们让它运行了 13 个小时来做浓缩咖啡饮品。我们尽量环保，不想把咖啡倒掉。13 个小时后，办公室里的每个人都有点兴奋，因为总得有人喝咖啡。但它运行了 13 个小时，非常酷。它出过几次错，比如洒了咖啡渣，然后需要拿块布擦干净。但它做到了，没有爆炸，13 个小时过去了。可能最负面的后果就是咖啡因摄入过多导致失眠。

When we were working on the PI star 6 project, the RL project we did late last year, we wanted to do some longer horizon experiments. In some cases it's obvious, like we had our robot assembling boxes at Dandelion Chocolate Factory. It's an actual chocolate factory, so they need the boxes. We ran it for several days. But we had this coffee task where the robot was using an espresso machine to make espresso. We ran it for 13 hours making espresso drinks. We tried to be very environmentally conscious, so we didn't want to throw out the coffee. After 13 hours, everyone in the office was a little wiry because someone had to drink the coffee. But it ran for 13 hours and it was pretty cool. It screwed up a few times, like it would spill the coffee grounds and then need to get a cloth and wipe it down. But it did it, nothing exploded, 13 hours went by. Probably the most negative consequence was loss of sleep from too much caffeination.

但当它洒了咖啡渣并清理时，是它自己做的吗？

But when it spills the coffee grounds and cleans it up, did it do that by itself?

嗯，那个实验的方式是有一个高层提示。提示大约每 5 分钟更新一次，在语义连贯的任务之间。所以你会告诉它‘做浓缩咖啡’、‘清理机器’等等。所以那些步骤，实际的机器清理，是由人指挥的。原则上，我们可以自动化这一点。事实上，我们现在花很多精力做的事情之一就是改进执行这些命令的高层策略。

Well, the way that experiment was done is that there is a high-level prompting. The prompt is updated maybe every 5 minutes or so, in between semantically coherent tasks. So you tell it 'make espresso', 'clean up the machine', etc. So those steps, the actual cleaning of the machine, was commanded by a person. In principle, we could automate that. In fact, one of the things we're spending a lot of effort on now is improving our high-level policy that does those commands.

但就那个实验而言，是的，大概每 5 分钟就会有人更新一下要做什么。我们原本的设想是，指令就像你去真正的咖啡店，你会说：“哦，我想要一杯拿铁，我想要一杯浓缩咖啡。” 那本该就是提示词。只不过你还得告诉它：“在做下一个之前，我要你先清理干净。”

But for that experiment, yeah, like every 5 minutes somebody basically updates what is being asked to do. Like the way we intended it was like the commands would be like if you go to an actual coffee shop, you say like, 'Oh, I want a latte. I want an espresso.' Like that was supposed to be the prompt. Except then you also have to tell it, 'I want you to clean it up before you do the next one.'

有道理。

That makes sense.

OpenAI、Anthropic、Cursor 和 Vercel 都在用这个产品来提升效率。它解决的问题是，当你在构建 SaaS 或 AI 产品并想卖给其他公司时，你需要满足各种要求，比如 SSO、SCIM、RBAC、审计日志。这些集成起来很耗时，但不是你应用的核心。WorkOS 是一个 API 层，让你只需几行代码就能满足所有这些要求。所以，假设你有一个新的 SaaS 产品，想卖给其他公司，WorkOS 会为你解决这些关键的功能缺口。你可以访问 workos.com 了解更多并开始使用。感谢他们支持我的工作并赞助本期播客。

OpenAI, Anthropic, Cursor, and Vercel all use this product to make their lives better. And the problem it solves is when you're building SaaS or an AI product and you want to sell to other companies, there's all these requirements you need to meet. There's SSO, there's SCIM, there's RBAC, there's audit logs. These are all things that take time to integrate but aren't the main focus of your app. WorkOS is an API layer that lets you meet all of these requirements in just a few lines of code. So, let's say you have a new SaaS product and you want to sell to other companies. WorkOS will solve all of these critical feature gaps for you. You can check them out at workos.com to learn more and get started. And I appreciate them for supporting my work and sponsoring this podcast.

听起来在走向泛化的过程中，数据是非常重要的一环。我读到有不同类型的数据：有模拟数据，你也可以收集物理交互数据。我想听听你的看法，什么数据最好，什么最差，各自的优缺点是什么。

It sounds like on the way to generalizing, data is a very important part of that. And I was reading there's different types of data: there's simulated data, you could collect like physical interactive data. And I wanted to hear your take on what's the best data to get, what's the worst, and what are the pros and cons.

顺便说一句，这个问题在机器人社区里讨论很多，有些人的观点截然相反。我自己的看法是，如果模型能将这些数据源扎根于对世界的深入物理理解，那么许多不同的数据源都更容易被模型内化。让我用几个例子来解释我的意思。如果你想学开飞机，你可能会用模拟器，至少训练的一部分时间会用到。但模拟器对你来说很有意义，因为当你开始使用模拟器时，你有很多世界知识可以用来理解正在发生的事情。比如你知道，当你用飞行模拟器学习驾驶飞机时，你不只是在玩电子游戏。你是在获取知识，之后会用于真实的飞机。你明白那里有一种抽象。同样，如果你玩一个非常卡通化的 Atari 游戏之类的，对吧？你知道屏幕上的所有符号，你可以把它们与你生活中的经历联系起来，并做出类比。所以，即使这些模拟环境看起来反映了现实世界的某些方面，对我们来说它们也很有意义，因为我们调动了很多自己的先验经验，填补了模拟中的空白。另外，如果你想使用自己或别人做某事的视频数据，比如你看某人做饭，对吧？即使你没有体验他们的每一个动作，你也有很多知识可以调动，你会想：“好的，我看到他们拿起盐罐。我以前在东西上撒过盐，所以我大致知道发生了什么。” 我可以把它以“加盐”这个抽象层次归档，而不必弄清楚他们的每一个肌肉动作。所以我的观点是，一旦你有了对自己身体如何做事以及如何体验物理世界的理解，所有这些其他知识源就可以与之连接起来，因为你从经验中获得的基础帮助你为一切提供依据。所以我的意思是，如果我们有一个机器人基础模型，在大量真实的具身数据上训练，这些数据提供了这种基础，它实际上可能更能吸收其他知识源。这实际上与一些人的想法有点相反，因为看到互联网数据对 LLM 的成功，很容易说也许我们应该反过来。也许我们应该从 YouTube 视频开始，然后把机器人数据放在上面。但我认为实际上是相反的。而且我甚至有一点证据。我的同事 Sudeep 和佐治亚理工学院的 Simar 之前有一个项目，他们拿我们的机器人基础模型，添加了人类视频数据，但他们不是从人类视频数据开始的。他们实际上是从一个在机器人数据上训练的模型开始，然后在上面添加视频数据。他们做的是观察模型内部的表示。基本上，模型如何表示人类经验与机器人经验？他们发现，如果你使用一个小模型和少量机器人数据，可以预见，人类经验和机器人经验是完全分离的。这意味着特征表示是不同的。但如果你在来自许多不同机器人的大量机器人数据上训练，那么特征更多是按正在执行的任务分组，而不是按是人类还是机器人分组。当我们查看特征图时，真是令人难以置信，因为当你把机器人数据量提高到 100% 时，它们就完美地对齐了。你看到 t-SNE 嵌入，你看到嵌入的形状，全是任务身份，对具身的敏感性极低。对我来说这有点令人震惊，因为这个基础模型根本没有在人类数据上训练过，但一旦你开始添加人类数据，它就以完全相同的方式表示。我认为这非常令人兴奋，这对我来说是最有力的指标之一，如果你有良好的机器人经验基础，你可以把其他一切放在上面，它实际上更擅长吸收这些。

This is by the way a question that is, I guess, quite like there's a lot of discussion in the robotics community about this question, and some people have like very opposite opinions on it. My own take on this is that a lot of different data sources are easier for the model to internalize if it can ground them in a thorough physical understanding of the world. So let me try to explain what I mean with a few examples. If you want to learn to fly an airplane, you will probably use a simulator, at least during part of your training. But the simulator makes a lot of sense to you because when you start using the simulator, you have a lot of world knowledge that you can use to ground what's going on. Like you know that when you are using the flight simulator to learn how to fly the airplane, you're not just like playing a video game. You're trying to acquire knowledge that you will then use with a real airplane. And you understand that there's sort of an abstraction there. Same thing if you are playing like a really cartoony Atari game or something, right? Like you know that all the symbols on the screen you can sort of connect them to things that you've experienced in your life and you can make an analogy there. So a lot of that, like even though it kind of seems like these simulated environments reflect aspects of the real world, to us they make a lot of sense because we kind of bring to bear a lot of our own prior experience and we fill in the blanks that the simulation has. And also if you want to use data of yourself or somebody else doing something, if you watch someone let's say cooking a meal, right? Even though you don't experience every movement they're experiencing, you have a lot of that knowledge that you bring to bear and you're like, 'Okay, I see they're picking up the salt shaker. I've put salt on things before, so I kind of roughly know what's going on there.' And I can file it away at this level of abstraction of add salt without having to figure out all their muscle movements. So my point with this is that once you have that understanding of how you do things physically with your own body and how you experience the physical world, now all these other sources of knowledge can be connected up to it because that foundation you get from your experience helps you ground everything. So where I'm going with this is that if we have a robotic foundation model that is trained on lots of real embodied data that provides that grounding, it might actually be much better able to absorb other sources of knowledge. And this is actually a little bit upside down relative to how some people think about it, because it's very tempting looking at the success of internet data for LLMs to say well maybe we should do the opposite. Maybe we should start with YouTube videos and then put robot data on top of that. But I think it's actually the other way around. And I actually even have a little bit of evidence for this. So my colleague Sudeep together with Simar from Georgia Tech, they had a project together a while back where they took our robot foundation model and they added human video data, but they didn't start with human video data. They actually started with a model trained on robot data and then added video data on top of it. And what they did is they looked at the representations inside the model. Basically how does the model represent human experience versus robot experience? And they found that if you use a small model with a small amount of robot data, predictably the human experience and the robot experience are fully separated. Meaning that the feature representations are different. But if you train on lots of robot data from lots of different robots, then the features are grouped much more by what task is being done rather than by whether it's a human or a robot. And when we looked at the feature plots, it was just mind-boggling because we literally when you crank up the amount of robot data to 100%, they just line up perfectly. You see the t-SNE embedding, you see the shapes of the embeddings, and it's just all task identity and minimal sensitivity to embodiment. And to me that's kind of mind-blowing because this base model wasn't trained on any human data at all, but once you start adding human data it represents it exactly the same way. And I think that's really exciting, and I think that to me is one of the strongest indicators that if you have that good foundation of robot experience, you can put everything else on top of it and it's actually better at absorbing that.

基础模型拥有使用特定电机、特定关节收集的数据，这一点重要吗？

Is it important that that base model has data that was collected using that specific set of motors, specific set of joints?

到目前为止，我们显然在跨具身模型上投入了大量精力，这些模型可以处理许多不同的机器人类型，但通常你确实需要将要部署的机器人的数据才能获得良好性能。所以那里的泛化指标不是能否零样本适应新机器人，而是能否用更少的新机器人经验，并从其他机器人转移技能。

So far, we've obviously put a lot of effort into cross-embodiment models that can handle many different robot types, but generally you do need data of the robot you're going to be deploying on to get good performance. So kind of the metric of generalization there is not can you zero-shot a new robot, but it's mostly can you get away with less experience from the new robot and transfer skills from other robots.

好的，所以这就是目前的状况。

Okay, so that's the current state of things.

这里有一个有点令人惊讶的积极解读：尽管你需要从这些机器人获取数据，但模型所做的特殊处理其实非常少。当我们刚开始做这一切时，我有一长串想做的酷炫研究，以更好地适应不同的形态，比如能否以某种方式分解模型的表征，让六自由度机械臂、七自由度机械臂、夹爪等各有其对应的部分。但我们什么都没做。模型只是输出一个大向量。如果机器人的自由度少于输出的数量，它就补零。没有任何花哨的东西。就是这样。然后它就在所有机器人上训练，根据摄像头看到的内容输出正确的动作。

Now there is a little bit of a kind of surprisingly positive read on that, which is even though you need data from these robots, the amount of special stuff that the model is doing is kind of minimal. Like when we started doing all of this, I had a big long list of all the cool research I wanted to do to better accommodate different morphologies, like can you factorize the model's representation in some way so that there's a six degree of freedom arm head, a seven degree head, a gripper head, etc. We didn't do any of that. The model just outputs a big vector of numbers. If the robot has less degrees of freedom than the number it outputs, it just zero pads it. There's nothing fancy. And that's it. Then it just trains on all the robots and outputs the correct actions based on what it is seeing through the camera essentially.

但回到你关于能否处理新机器人的问题。到目前为止，我们关注的重点——而且我认为这显示出一些前景——是能够在机器人之间迁移技能。而这正是模型工作方式的具体选择似乎至关重要的地方。例如，你可以让模型进行一些中间思考，而这种思考可以用不同的模态进行。你可以用文本思考，而用文本思考非常适合迁移高层行为结构。这基本上就是让你理解：如果我想打扫厨房并收拾银器，先打开抽屉。这是一种语义推理，你可以很好地迁移它，因为这显然在很大程度上与任何具身形态无关。

But now to your point about whether you can handle new robots. So far the thing that we focused on, and I think this is showing some promise, is to be able to transfer skills between robots. And this is actually where the particular choices in how the model works seem to matter. For example, you can have a model that does some intermediate thinking, and that thinking can be done in different modalities. So you can think in text, and thinking in text is really good for transferring high-level behavioral structure. That's basically how you understand that if I want to clean the kitchen and put away the silverware, first open the drawer. That's kind of a semantic inference, and you can transfer that very well because obviously that's largely agnostic to any embodiment or anything like that.

但如果你使用正确的表征，即使是更低层次的东西也可以迁移。所以我们做的一个实验是，我们有一个以图像表达的思考阶段，你基本上是在想象任务中下一个里程碑的图像。有了这个，我们实际上可以让一个机器人——UR5 机器人——折叠 T 恤，尽管我们没有 UR5 上的任何折叠 T 恤数据。因为虽然让手臂动作正确非常困难，因为机器人基本上需要非常不同的关节角度来完成任务，但想象它折叠衬衫的样子并不难，因为你见过机器臂在各种不同的姿势下。你见过衬衫在折叠和展开的不同阶段。你大致知道它应该握在哪里。所以让一个好的生成模型来生成那个图像是相当直接的。

But even lower-level things can be transferred if you use the right representation. So one experiment we did is we had a thinking stage that is expressed in images, where you basically dream up an image of the next milestone in the task. And with that, we could actually get a robot, the UR5 robot, to fold a t-shirt, even though we didn't have any t-shirt folding data on the UR5. Because while getting the arm motions correct is very hard, since the robot basically requires very different joint angles to do the task, cooking up an image of what it looks like for it to fold a shirt is not that hard, because you've seen the robot arm in all sorts of different poses. You've seen the shirt in all different stages of being folded and unfolded. You know roughly where it should hold it. So getting a good generative model to cook up that image is pretty straightforward.

一旦你有了图像，那么从中推导出正确的动作也很容易，因为你只需查看合成的机械臂角度，然后推导出应有的角度。所以这并没有改变问题，只是引入了一个中间步骤，使问题更容易解决。就像解数学题一样，如果你找到了正确的中间步骤，答案就从那个中间步骤变得显而易见。我认为这非常令人兴奋，因为现在这表明这种跨机器人的泛化水平——而且我相信其他泛化也是如此——可以通过思考来促进，就像大语言模型一样，但有一个转折：你必须以正确的模态思考。

And once you have the image, then from that backing out the correct actions is easy too, because you can just look at the synthesized arm angle and just back out what the angle should be. So it's not changing the problem, but it's just introducing this intermediate step that makes it easier to solve. Just like if you're solving a math problem, if you figure out the right intermediate step, the answer is obvious from that intermediate step. And I think that's really exciting, because now that shows that this level of generalization across robots, and I'm sure other generalization too, can be facilitated with thinking just like an LLM, but with a twist. You have to think in the right modality.

有趣。所以它输出的，我猜那个图像是它的视频传感器所看到的，就像是下一步。

Interesting. So it outputs, I guess that image is what its video sensor is seeing and it's like the next step.

是的。你几乎可以把它想象成图像编辑。你也可以用视频做同样的事情。你可以用视频预测来做。但关键是要想象在这个任务上取得进展会是什么样子。这似乎是非常人类化的做法，对吧？你知道，有些事情你按语义规划，有些事情你按空间规划。比如你在攀岩时，可能不会想：嘿，左臂向左 37 厘米抓那块岩石。你更可能是在想象你的手臂伸向岩石。

Yeah. You can almost think of it like image editing. You can do the same thing with video. You can do it with video prediction. But the key is to imagine what it would look like to progress on this task. That seems like a very human thing to do, right? You know, some things you plan semantically and some things you plan spatially. Like if you're doing rock climbing, you're probably not thinking, hey, left arm to rock 37 cm to the left. You're probably more like imagining your arm reaching for the rock.

在对话早些时候，你提到 Waymo 非常鼓舞人心，他们走向产业化的道路证明了你可以做现实世界的通用机器人。如果我没记错的话，在我年轻很多的时候，这曾是一个早期的承诺，说这将会发生，但实际上花了更长的时间。所以我想问的是，对于人形机器人，什么会让你说个位数年内就会到来，而不是漫长的尾巴和政策挑战。我可以想象。

Earlier in the conversation, you mentioned that Waymo was very inspiring, and their kind of path to productionization is proof that you can do real-world generalized robotics. And if I recall correctly, when I was a lot younger, it was kind of this early promise of this is going to happen, and then in reality it took a lot longer. So I guess my question is, in the case of humanoid robotics, what would make you say single-digit years it's coming versus a long tail and policy challenges as well. I could imagine.

我认为机器人基础模型解决问题的方式与传统工程系统解决问题的方式之间的一大区别在于，技术栈非常薄。训练基础模型并不容易。你显然需要获取正确的数据。在数据整理、标注等方面有很多工作。但实际在机器人上运行的软件非常简单。所以你可能会有一个思考或推理阶段。你可能会让模型产生动作。它需要足够快。但如果你从原始代码行数来看，它比传统的自动驾驶技术栈要低得多。

I think one big difference between how robotic foundation models address the problem and how more traditional engineered systems address the problem is that the stack is really thin. So it's not easy to train a foundation model. You need to obviously get the right data. There's a lot of work that goes into curating, labeling, all that other kind of stuff. But the actual software that runs on the robot is very, very simple. So you might have some kind of thinking or reasoning stage. You might have the model produce actions. It needs to be fast enough. But if you think about it in terms of raw lines of code, it's much, much lower than a more traditional AV stack.

部分原因是现代自动驾驶汽车的工作开始得更早，使用了非常不同的技术，并随着时间的推移而演变。部分原因也是因为问题更关乎安全。比如，是的，你不想让机器人操作器掉落易碎物体，但归根结底，这比汽车撞到人要轻得多。这并不是说机器人的安全挑战不真实。它们非常真实，解决它们非常重要。事实上，这可能是问题中较难的部分之一。但它们对实际部署的阻碍并不那么严重，因为你可以设计任务、环境、领域以及物理硬件，使这些问题不那么严重。

And partly that's because modern autonomous vehicles, the work on that started a lot earlier with very different technologies and evolved over time. Partly it's also because the problem is more safety critical. Like yes, you don't want a robotic manipulator to drop a fragile object, but at the end of the day, that's a lot less bad than having a car hit somebody. So that is not to say that the safety challenges with robots are not real. They're very real and it's very important to tackle them. In fact, it's probably one of the harder ends of the problem. But they are not as much of a hard stop to practical deployments because you can come up with tasks and environments and domains and also physical hardware where those problems are a lot less severe.

所以我认为这种组合——大幅简化的软件栈加上不那么严峻的软件挑战——实际上让它容易得多。而且因为，正如你之前提到的，存在这种飞轮效应，一种正反馈循环，即使在某些限制下开始将东西推向世界，实际上也会促进它们越来越多地推出。

So I think that combination—radically simpler software stack plus less drastic software challenges—actually makes it a lot easier. And because, to your earlier point, there is this kind of flywheel effect, a positive feedback loop, that starting to get things out in the world even under some constraints will actually facilitate getting them out more and more.

但在这种情况下，我很好奇，你会怎么看待事前验尸？也就是说，如果人形机器人在几年内没有成功，你认为最可能的原因是什么？

But in this case, I'm curious, what would you say to a premortem? And in the sense of if humanoid robotics did not succeed in single-digit years, what do you think would be the most likely reason why humanoid robotics failed?

归根结底，要让这些东西真正有用，它们确实需要达到比我们通常对 LLM 或生成式图像视频 AI 所期望的更高的可靠性、鲁棒性和泛化水平。因为通常这些工具都是高度人机交互的工具。你让 LLM 做某事，它没完全按你想要的做，于是你修改提示词，和它反复迭代。这就是为什么即使是早期的语言模型工具，比如第一版 ChatGPT，虽然比现在原始得多，但已经很有用了，因为用户可以不断尝试，直到它基本解决问题。就像用搜索引擎一样，你输入关键词，没得到想要的，你修改查询，然后得到想要的。而机器人则不同，它的全部价值在于真正自主地完成任务。所以如果每个任务都需要有人不断迭代，那几乎与它带来的好处背道而驰。所以我认为很多风险在于，要达到那种可靠性和鲁棒性有多容易。这又是一些演示可能有点误导的地方，因为如果有人展示机器人做酷事的演示，如果一切都以坦诚的方式呈现，那仍然可能是进展的良好指标，但它并不能清楚地表明离达到那种实际相关的鲁棒性还有多远或多近。所以我个人非常相信使用强化学习这类技术，它能从自主经验中获益，来微调最后几个百分点，让它从 95% 提升到真正的 100%。但这非常重要，而且尚未解决。

Ultimately, for these things to be truly useful, they do need to reach a level of reliability and robustness and generalization that is higher than what we typically expect from LLMs, for example, or generative AI for images and video. Because typically these tools are very much human-interactive tools. You get an LLM to do something and it doesn't do quite what you want, so you sort of revise your prompt and you iterate with it. That's why even the earlier LM tools, like the first version of ChatGPT, even though they were much more primitive than what we have now, were still already useful because somebody could just keep hammering out until it basically solves their problem. Just like if you're using a search engine, you type something in, you don't get quite what you want, you revise your query, and then you get what you want. Whereas with a robot, the full value is unlocked when it's actually doing the thing autonomously. So having to have somebody constantly iterate for every single task is almost antithetical to the benefit that you're getting. So I think a lot of the risk has to do with how easy it is to get that level of reliability and robustness. And that's again where some of the demos might be a little bit misleading, because if someone shows a demo of the robot doing something cool, if everything is presented in a forthright way, that could still be a very good indicator of progress, but it doesn't make it obvious how far or how close it is to reaching that practically relevant level of robustness. So I'm personally a big believer in using techniques like reinforcement learning that can actually benefit from autonomous experience to fine-tune that last few percentage points, to make it go from like 95 to actually 100%. But that's really important and it's not yet a solved problem.

如果确实比预期耗时更长，那是因为门槛更高了。

If it did take longer than expected, it's because the bar is higher.

因为门槛更高了，尤其是最后那几步，可以说是最后一英里，不仅需要非常好的模型，还需要技术上的新创新。我的意思是，我不是那种会说“哦，我们应该抛弃所有关于基础模型的知识，重新开始”的人。我完全不这么认为。我认为我们现有的拼图块实际上是非常好的拼图块。但我们仍然应该承认，目前的方法和模型还需要更多努力才能跨越那个鲁棒性水平。

Because the bar is higher, and in particular those last few, the last inch so to speak, is something that requires not just really good models but also new innovations in technology. I mean, I don't think that you know, I'm not the kind of person that would say like, oh we should throw out everything that we know about foundation models and start over. I don't think it's that at all. I think that roughly the puzzle pieces that we have are actually very good puzzle pieces. But still we should acknowledge that right now the methods and the models need more work to cross that level of robustness.

在 LLM 领域，感觉每个人都在做类似的事情，只是风味不同。在机器人行业，大家是不是也在做类似的事情？有没有什么热门架构是不同方向的？

In LLMs it feels like everyone is doing kind of the same thing but different flavors. In the robotics industry, is everyone doing kind of the same thing? Are there any hot take architectures that are different direction?

我实际上认为，其中的异质性比看起来要大得多。一个重要的分界线，我认为从结果中可能不太明显，就是完全拥抱基础模型理念与专注于特定垂直领域之间的区别。我认为这有时很难分辨，因为显然每个人都会说“哦，我在做 LM 做过的事情”，因为那很酷。但基础模型的理念从根本上说是这样的：如果你有一个特定问题要解决，更好的做法是训练一个更通用的模型，它能利用来自广泛问题的数据，如果做得好，它在你想要解决的专门问题上会比狭窄的专家模型表现更好。所以回到 LM 的类比，如果你想做机器翻译，不要构建一个机器翻译系统，而是构建一个理解所有语言任务的语言模型，然后把它用在机器翻译上。在机器人领域，我认为这实际上让很多人深感不适，因为如果有人真的在做一个应用，比如仓库自动化，那么想到“哦，如果我想做仓库自动化，让我收集厨房里整理餐具的数据”会非常尴尬。这听起来很奇怪。但这就是基础模型的教训：如果你有足够的广度，如果你从各种不同的任务中收集数据，那么你就会获得那些可泛化的技能，如果你的模型构建正确，它会将这些技能重新用于它遇到的任何情况。所以我认为，即使你想建造一个仓库机器人，你最好也收集广泛的数据，这样它就能更好地处理即使在仓库领域也可能遇到的各种奇怪边缘情况。但这并不是人们容易接受的，因为它与构建传统垂直集成机器人系统的原则如此背道而驰。

I actually think that there's a lot more heterogeneity than it might seem. One big dividing line that I think is maybe not as obvious from just looking at the results is the distinction between fully embracing the foundation model ethos, so to speak, versus focusing on specific vertical areas. And I think this is kind of hard to tease out sometimes because obviously everyone's going to say, oh I'm doing the thing that LM did, because that's the cool thing. But the foundational model ethos fundamentally is something like this: if you have a particular problem you want to solve, it is better to train a more general model that can use data from a breadth of problems, and if you do it right, it'll actually be better at the specialized problem you want to solve than a narrow specialist. So to come back to the LM analogy, if you want to do machine translation, don't build a machine translation system, build a language model that understands all language tasks and then throw it at machine translation. And in robotics, I think that is actually very deeply uncomfortable to people, because if someone is actually working on an application like warehouse automation, it is very awkward to think, oh if I want to do warehouse automation, let me collect data of putting away silverware in kitchens. It just sounds bizarre. But that is the foundation model lesson: if you have enough breadth, if you collect data from a wide range of different tasks, then you will acquire those generalizable skills, and if your model is built correctly, it will repurpose those skills for whatever situation it encounters. So I think it is actually true that even if you want to build a warehousing robot, you're better off collecting a breadth of data, and it will be better at handling all the weird edge cases you might encounter even in that warehouse domain. But this is not something that's easy for people to accept, because it's just so antithetical to the principle of building a traditional vertically integrated robotic system.

我注意到这些 AI 公司出现了一个有趣的新现象，那就是如果它们非常成功，就会引发某种担忧或一系列新问题。例如，当 Anthropic 拥有非常强大的模型时，政府就会介入，出现关于安全和风险的担忧等等。我很好奇你怎么看待这个问题。比如，如果 Physical Intelligence 今年推出了一个现象级的、能力极强的通用模型，你会怎么看待可能出现的那些话题？

I noticed this new interesting phenomenon with these AI companies, which is if they are wildly successful, it creates this, I guess, worry or new set of things. So for instance, when Anthropic became very powerful model, then the government comes in and there's these worries about safety and risk and all that. And I'm curious how you think about that. Like if Physical Intelligence this year had a phenomenal, incredibly capable generalized model, how do you think about those kinds of topics that might come up?

从事 AI 安全研究并不新鲜。我在加州大学伯克利分校的同事 Stuart Russell 十多年前就在谈论这些事情，很多人花了很多时间研究它。问题是，当技术发展如此之快时，重要的问题不仅取决于核心原则，还取决于社会如何反应、采用什么样的工具等等。我认为这非常非常难以预料。所以对于我们的应对方式，我没有一个非常令人满意的答案。我们围绕这一切的哲学基本上是经验实验：把东西拿出来，看看现实世界中会发生什么，看看什么是对的，什么是错的，这样我们就能尽可能多地预览技术能做什么、它的弱点是什么、它的优势是什么，等等。

Working on AI safety is not new. My colleague at UC Berkeley, Stuart Russell, was talking about this stuff over a decade ago, and lots of people spent a lot of time working on it. It's just that the trouble is when the technology moves so fast, the important problems are not just a function of the core principles, it's also a function of how society reacts to it, what kind of tools are adopted, and so on. And I think that's very, very hard to anticipate. So I don't have a very satisfying answer here in terms of how we are approaching it. Our philosophy around all this stuff is basically one of empirical experimentation: let's get stuff out there, let's see what happens in the real world, let's see what goes right and what goes wrong, so that we have as much of a preview for what the technology can do, what are its weaknesses, what are its strengths, and so on.

但你知道，归根结底，你只能睁大眼睛，看看会发生什么，然后边走边调整。这很难预料。而且，我觉得你的问题问得很到位，虽然我没有一个很好的答案，因为如果连那些基本局限于使用计算机的 AI 系统都让我们如此担忧、问题重重，那么能够像我们一样在物理世界做所有事情的 AI 系统，我们很可能会遇到更多的问题和担忧，对吧？所以问题是真实存在的。只是，你知道，你只能看看会发生什么，然后调整。这就是那条可怕的路。

But you know at the end of the day you kind of have to just keep your eyes open, see what happens and adjust as you go. It's very hard to anticipate. And you know, I think your question though is very spot on even though I don't have a great answer for you because like yeah, if we're having this much concern and issues with AI systems that are basically limited to using computers, we're presumably going to have strictly more concerns and issues with AI systems that can do everything in the physical world that we can do, right? So the issues are real. It's just, you know, you kind of have to like see what happens and then adjust. And that's kind of the scary path.

但在乐观的道路上，如果一切顺利，十年后我们拥有极其强大的模型和机器人，北极星就是人类劳动的终结。我认为把机器人看作机械人是个错误，对吧？就像计算机在某种程度上是机械大脑，但当个人电脑在 90 年代、21 世纪初真正起飞时，并不是人们用电脑取代了自己的大脑，而是我们看到各种不同类型的电脑激增。我们看到了普适计算。所以你的桌上有一台电脑，口袋里可能也有一台，冰箱里、车里也可能有，因为计算变得如此普及，你可以在任何东西里放一点计算能力。而且，我觉得这不是那些在 40 年代和 50 年代最初思考这些东西的人会想象的。他们会想象房间大小的计算机，其工作是控制整个国家的政策之类，而不是每个人冰箱里都有一点计算机。所以我认为，以此类推，我们可以想象，也许每样东西里都会有一点物理驱动，很多你现在必须自己做的日常事情，可能会得到一点帮助。

But in the happy path, if everything goes well and we have incredibly capable models and robots in 10 years, is the north star that that's the end of human labor. I believe it's a mistake to think of robots as mechanical people, right? Like computers at some level are kind of like mechanical brains, but when personal computers really took off in the '90s early 2000s etc., it's not like the first thing that happened is that people replaced their brains with computers. Rather, what we saw is actually a proliferation of very different kinds of computers. We saw ubiquitous computing. So you would have a computer on your desk but you might also have one in your pocket. You might have one in your refrigerator and in your car, because computing became so accessible, you could have a little bit of computing in everything. And you know, I don't think that that's what the people that first started thinking about this stuff in the 40s and 50s would have imagined. They would have imagined roomsized computers whose job it is to control the policy of an entire country or something, rather than a little bit of computer in everybody's refrigerator. So I think by analogy of that we might imagine there might be a little bit of physical actuation in everything, and it might just be that lots of everyday things that you have to do yourself now you get a little bit of help with it.

我认为另一个值得思考的例子是现代编码智能体，对吧？所以我认为，当然，关于编码智能体的最终结局还没有定论，但从今天软件工程师的经验来看，似乎可以说，大多数人会认为编码智能体更多是赋予他们力量，而不是让他们感到恐慌。我的意思是，显然有些人可能会恐慌，但总的来说，至少从我交谈过的软件工程师和我自己的经验来看，能够用 AI 工具放大你能做的工作量，这更是一种赋能。所以我认为从这个角度看，这可能是一个相当直接的类比，因为这正是 AI 进入真实工作的一个例子，并且确实为从事该工作的人提供了更大的杠杆。所以我认为这是我们可以参考的另一个例子，但事实是，我认为这还有待观察。

I think the other example that's worth thinking about is modern coding agents, right? So I think that this is something where of course the jury is still out as to what the endgame of coding agents is, but certainly from the experience of software engineers today, it kind of seems like probably fair to say that most would consider coding agents to be more empowering them rather than somehow causing them to have a panic. I mean, obviously some people might have a panic, but in general, at least from the software engineers that I've talked to and from my own experience, it's more empowering to be able to amplify how much work you can do with AI tools. So I think from that, and that maybe is like a pretty direct analogy because that is straight up an example of an actual real job where AI has entered into it and has actually provided more leverage to people doing that job. So I think that's another example that we can look to, but the truth is that I think it remains to be seen.

在 LLM 领域，有一些开创性的论文，如果你读了这些论文，你就能大致了解那些重要突破的脉络，并理解我们今天在机器人行业所处的位置。在类人机器人方面，有没有一些顶级论文，你认为真正展示了那些突破，如果人们想了解最前沿的技术，他们应该知道这些论文？

So in LLMs there's a few seminal papers that if you read those papers you kind of get a sense of the lineage of the breakthroughs that mattered and understanding where we are today in the robotics industry. Are there a set of top papers that you really think kind of show the breakthroughs that people should know about if they're curious about the state-of-the-art in terms of humanoid robotics?

我要指出的一点是，这有点厚脸皮地自我推销，因为我是那篇论文的合著者，但坦率地说，这项工作 99.9% 是由主要作者 Tony 完成的，那就是最初的 ACT 论文，也就是 Aloha 论文。这是一个有趣的例子，因为从某些方面来说，这些想法并不算新颖，但它们以一种非常好的方式得到了阐述。这个想法是，嘿，如果你搭建了合适的低成本机器人设置，在他的案例中，是基于 Trusson Robotics 的机械臂，那些大约 7000 美元的爱好者级机械臂。他用主从遥操作设备搭建了双臂设置。他证明，如果你做对了，不需要任何特别花哨的技巧，你就能轻松收集到极其灵巧任务的遥操作数据，而这些任务以前人们认为需要非常复杂的硬件和各种昂贵的东西，然后建立一个相当直接的基于 Transformer 的模型，它就能完成很多这样的任务。

One thing I would point out, and this is partly a shameless plug because I am a co-author on that paper though candidly like 99.9% of the work on this was done by Tony who was lead author, is the original ACT paper, the Aloha paper. It's an interesting example because in some ways the ideas weren't really that new but they were illustrated in a really nice way. The idea was that hey, if you set up the right kind of low-cost robot setup, in his case it was based on these robot arms from Trusson Robotics that are like $7,000 hobbyist arms. He set them up in a bimanual setup with a leader-follower teleoperation device. And he showed that actually if you do it right without really any particularly fancy tricks, you could easily collect teleoperation data of extremely dexterous tasks that people had previously thought would require very sophisticated hardware and all sorts of really expensive stuff, and then set up a fairly straightforward transformer-based model and it could actually do a lot of those tasks.

这是一件有趣的事情，因为在学术研究中，我们通常非常重视是否有某种复杂的新数学方法或复杂的技术见解。而在这篇论文中，我认为它现在已经具有巨大的影响力，其见解实际上就是，是的，把正确的组件组合在一起，并对一个简单的机器人配备良好的模仿学习系统所能做到的事情多一点信心。他展示了诸如更换遥控器电池之类的事情。他甚至弄了一个小人体模型脚，并展示你可以给它穿鞋，用于辅助任务，有点像有些人需要帮助穿鞋，所以这很好。但人们觉得这篇论文有趣的地方在于，用相对简单的构建模块，你能走多远。

And it's kind of an interesting thing because usually in academic research we put a big premium on having some sophisticated new mathematical thing or some sophisticated technical insight. And in that paper, which I think at this point has been hugely influential, the insight is really just like yeah, just put together the right pieces and have a little bit more faith in what a simple robot could do, so to speak, equipped with a good imitation learning system. And he showed things like replacing batteries in a remote control. He even got a little mannequin foot and he showed that you could put a shoe on it for an assistive task, sort of like some people need help getting their shoes on, so that's good. But what people found so interesting about that paper is just how far you could get with relatively simple building blocks.

而且现在，他已经开源了代码，ACT 代码已经被很多人使用。如果有人想要一个非常基础的机器人学习入门套件，他们通常会拿这个。我认为，对于想进入这个领域的人来说，值得去读那篇论文，真正理解其中的内容，因为尽管在某些方面它并不那么复杂，但我认为它提供了一些关于什么重要的校准。细节很重要，但细节不一定复杂。

And at this point, he open-sourced the code for it, and the ACT code has been used by lots of people. If someone wants a very basic starter kit for robotic learning, that's usually what they grab. And I think that it's worth for somebody who wants to get into the field to go through that paper and really understand what's going on there, because even though in some ways it's not that sophisticated, I think it provides a bit of calibration on what matters. The details matter, but the details don't have to be complicated.

在大型模型机器人浪潮之前，波士顿动力公司有那些令人印象深刻的演示，并且占据了大量心智份额。我想我当时甚至不在这个领域，但也会说，哇，他们真的在做不可思议的机器人。然后在过去，我不知道多少年，我不太再听到他们的消息了。是不是行业发生了某种转变导致了这种情况，还是你能解释一下？

Before all this large model robotics wave, prior to that, Boston Dynamics had these really impressive demonstrations and tons of mind share. I guess I wasn't even in the field by saying wow they're really doing incredible robotics. And then in the last I don't know how many years I don't really hear about them much anymore. Is there some shift in the industry that made that so, or is that something you could explain?

所以我会这样解释，机器人技术从某种程度上说，是关于构建复杂系统。

So the way I would explain it is this, that robotics at some level is about building complex systems.

所以，尽管我们很容易说，AI 有不同的领域，比如大语言模型、视觉和机器人，但其中有一个领域和其他不一样，因为对于机器人，你实际上需要所有部分，从如何布线、电源是什么、执行器长什么样，一直到它如何进行高层规划来决定下一步做什么任务。而且，尽管我们可以看着这些东西说，哦，所有这些不同的视频、不同的公司、不同的演示，它们都是机器人，但它们实际上是技术栈的不同部分。波士顿动力经典的成果展示了很多非常复杂的硬件，非常精心设计的硬件，配合传统的控制方法，由非常聪明的控制工程师设置好一切，但在决策方面相对较少强调。我认为在某个特定时间点，这实际上很有道理，因为如果我们不能构建物理身体，那么不管上面运行什么样的决策系统都无所谓。但回到我们之前关于泛化的讨论，现在我们处于这些事物发展的一个阶段，即使在硬件上我们可以做得更多，但在很多方面它已经足够好了。而大的挑战是如何让决策循环真正起作用，并对环境中的一切做出智能反应。我要划分这两者的方式和位置是：决策循环并不意味着符号决策。它可能意味着低级决策。问题是你是需要考虑环境的其余部分，还是只处理机器人本身？所以如果你想在平地上做后空翻，你主要得处理机器人本身。但如果你想从桌子上拿起一个咖啡杯，尽管这在某些方面可能比后空翻简单，但你确实需要理解世界其他地方发生了什么，而不仅仅是自己的身体。而这条分界线，我认为随着技术的发展，公平地说，这就是 AI 和控制之间的分界线。控制是当你必须控制机器人的身体时。AI 是当你必须考虑机器人外部发生的事情时。我认为这就是你看到这种分歧的原因，因为很多演示中你主要需要处理机器人本身而不是世界的其余部分，非常好的控制可以让你在那里得到一个很好的解决方案。我还想说的一点是：如果在做某个特定技能时有很多控制工作，那么实际上有东西可以从中学习，因为如果你能手工设计一个执行复杂行为的控制器，很可能你也能学习那个控制器。所以仅仅那个存在性证明，即这件事是可能的，而且不仅可能，而且简单到一个人就能构建，因为记住，即使在今天有代码的情况下，人们能处理的复杂性也不如 AI 能处理的复杂性高。所以如果一个人能手工设计出做后空翻或某些杂技的东西，那是一个很好的存在性证明，表明存在某种相对简洁的控制法则来做那个技能。而简洁确实意味着可泛化。所以如果它简单到一个人能设计，很可能里面有相当通用的东西。如果你能学习它并自动化它，而不需要人类控制工程师参与，那是好消息。

So um even though it's very tempting to say like oh there are different areas of AI like LLMs and vision and robotics, one of those is not like the others because for robots you actually need all the parts, everything from how you wire up the robot, what the power source is, what the actuator looks like, all the way to how it does high-level planning to determine what task to do next. And even though we could look at these things and say oh all of these different videos and different companies and different demos, they're all robotics, they're really different parts of the stack. A lot of what the classic Boston Dynamics results show is very sophisticated hardware, very carefully designed hardware with a traditional control approach with very smart controls engineers setting everything up, but with comparatively less emphasis on the decision-making aspect. And I think that at a particular point in time that actually made a lot of sense because we can't build the physical body, like it doesn't matter what kind of decision-making system is running on it. But to our earlier discussion about generalization, at this point we're at a stage in the development of these things that even though we can do more on hardware, in many ways it's good enough. And the big challenge is how to have the decision-making loop that actually works and reacts intelligently to everything in the environment. And the way and the place where I would draw the dividing line between those is: the decision-making loop doesn't mean symbolic decisions. It could mean low-level decisions. The question is do you need to take the rest of the environment into account or are you just dealing with the robot? So if you want to do a backflip on flat ground, you mostly have to deal with the robot. But if you want to pick up a coffee cup off of a table, even though that's maybe in some ways simpler than doing a backflip, you really have to understand what's going on in the rest of the world rather than just your own body. And that dividing line, I think the way the technology has panned out, it's fair to say that that is the dividing line between AI and controls. Controls is when you have to control the robot body. AI is when you have to take into account what goes on outside of the robot. And I think that's why you see this divide, because a lot of the demos where you mostly needed to deal with the robot itself and not the rest of the world, really good controls could allow you to admit a very good solution there. And another thing I would say here is: if there's a lot of controls work that goes into doing some particular skill well, there is actually something to learn from that, because if you can hand-design a controller that performs a sophisticated behavior, very likely you can also learn that controller. So just that proof of existence that the thing is possible, and not only possible but also simple enough that a person could build it, because remember people, even with code these days, the kind of complexity that people can handle is not as high as the kind of complexity the AI can handle. So if a person can hand-design something to do a backflip or do some acrobatics, that's a really great proof of existence that there exists some relatively parsimonious control law for doing that skill. And parsimonious does mean generalizable. So if it's simple enough for a person to design, probably there's something fairly general in there. And if you can learn it and automate it without having to have the human controls engineers in the loop, that's good news.

然后最后一个问题：如果你能回到刚进入这个行业的时候，以你现在所知的一切给自己一些建议，你会说什么？

And then last question for you is: if you could go back to when you just entered the industry and give yourself some advice knowing everything you know now, what would you say?

过去几年我学到的一件事，我认为和我最初的心态有点不同，那就是有效地解决机器人问题需要使用非常广泛的先验知识。而且我认为在机器人学习领域有很多人都有这个想法，我最初也有，那就是既然人们从零开始学习，也许机器人也应该从零开始学习。所以例如，在谷歌我们早期的大规模机器人学习工作中，我们有一个我们称之为“机械臂农场”的项目。我们实际上在一个会议室里设置了一堆机械臂，因为我们没有合适的实验室，让它们都去抓取物体。想法是，如果它们抓取数百万个物体，它们就会学到非常通用的抓取策略。它基本上奏效了，它们能学会抓取物体，但很难从那里更进一步到下一个层次。所以，好吧，现在我能拿起任何东西，但那又怎样？它并没有为更复杂的技能提供一个很好的垫脚石。我认为部分原因是我们以非常空白的状态来处理这个问题，就像让我们从零开始，看看如果事先什么都不知道，机器人能否开始获得行为。但我认为，如果你能将机器人的经验与从其他来源获取的知识结合起来，让这一切发挥作用会实际得多。例如，我最初对语言的作用非常怀疑，我认为这在科学上是站得住脚的，那就是动物能做一些相当令人印象深刻的事情，比如猴子能做很酷的事情，但据我所知猴子不会说话，至少不会说得很流利，所以也许我们的机器人也应该能做事情，它们不一定需要理解语言。但我认为微妙之处在于，重要的不是语言，而是你可以作为学习过程的脚手架放入的先验知识。你可以通过各种方式获取这些知识。人类不一定完全通过语言来获取。人类和猴子当然不是。他们通过观察来获取，观察其他人、其他生物等等。所以有很多先验知识的来源。但关键是，你必须把那些先验知识放进去。否则，你实际上面临的问题比人类和动物必须解决的问题更难。因为如果一个人必须弄清楚如何组装宜家家具，但他一生中从未遇到过任何家具，那会相当困难，因为他甚至不知道这有什么意义或最终目标是什么样子。所以，是的，先验知识很重要。

One thing that I've learned over the last few years, which I think is a little different than my original mindset, is that addressing robotics effectively requires using very broad prior knowledge. And I think there's this idea that a lot of people in robotic learning have, which I think I shared initially, that since people learn things kind of from scratch, maybe robots should learn things from scratch too. So for example, in some of our early work on large-scale robotic learning at Google, we had this what we call the arm farm project. We set up a bunch of robot arms in a conference room actually, because we didn't have a proper lab, and we had them all grasping objects. And the idea was well, if they grasp millions of objects, they'll learn very general grasping strategies. And it basically worked, they could learn to grasp objects, but it was very hard to get further from that to the next level. So okay, now I can pick up anything, but so what? It didn't serve as a very good stepping stone for more complex skills. And I think part of that was that we were approaching this very blank slate, like let's start from zero and see if knowing nothing in advance, the robot could start picking up behaviors. But I think it's much more practical to get all this to work if you can combine robot experience with knowledge that you can pull in from other sources. For example, I was very skeptical initially about the utility of language, and I think scientifically this is defensible, which is that animals can do some pretty impressive things, like monkeys can do really cool stuff, but monkeys as far as I know can't speak, at least not very eloquently, so maybe our robots should also be able to do stuff and they don't necessarily need to understand language. But I think the subtlety there is what's important is not language, it's prior knowledge that you can put in as a scaffold on your learning process. And you can pull in that knowledge in all sorts of ways. Humans don't necessarily pull that in entirely through language. Humans and monkeys certainly don't. They do it from observation, from observing other people, other creatures and so on. So there's lots of sources of prior knowledge. But the point is that you got to get that prior knowledge in there. Otherwise, you're actually faced with a harder problem than what humans and animals have to solve. Because if a person had to figure out how to assemble IKEA furniture, but they've never actually encountered any article of furniture in their entire life, that would be pretty difficult because they don't even know what the point of this is or what the endgame looks like. So yeah, prior knowledge is important.

虽然我仍然非常推崇通过亲身经历来学习，但我觉得，你知道，我会给过去的自己的建议是：更认真地对待已有的知识。

And while I'm still a big fan of learning things through experience, I think that, you know, my advice to myself would have been take prior knowledge more seriously.

太棒了。非常感谢你抽出时间，Sergey。真的很感激。

Awesome. Well, thank you so much for your time, Sergey. I really appreciate it.

谢谢你的提问。

Yeah, thank you for your questions.

嘿，感谢观看本期播客。如果你喜欢这期节目，并且希望它越办越好，请点赞或评论支持。另外，如果你想推荐嘉宾，请在评论区留言。像 Barbara Liskov、Mike Stonereaker、Mark Brooker 这些嘉宾，都是因为有人留言我才邀请来的。

Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Liskov, Mike Stonereaker, Mark Brooker, these were all people that I brought on because someone left a comment.

另外，除了播客，我还在打造一款我一直想要的人体工学键盘。这是原型机的一瞥。它是一款分体式键盘，所以有左右两部分。这是装在壳子里的样子。我们在 Kickstarter 上发起了众筹，上线 8 小时内就达成了目标。如果你也抢到了早期版本，我非常感激。我们现在正走在漫长的工具开发之路上。如果你还想入手，我在 Kickstarter 上保留了晚鸟支持选项，你可以在那里购买。我会把链接放在简介里。再次感谢观看本期播客，我们下期见。

On another note, aside from the podcast, I'm working on building the ergonomic keyboard that I wish existed. Here's a glance at the prototype. It's a split keyboard, so there's two sides. This is in the case, but yeah, we launched on Kickstarter and we hit our goal within eight hours of launching. I really appreciate it if you were one of the people who grabbed one of the early units. Um, we're now working on the long journey of building the tooling now. And so, if you still want to pick one up, I've left the late pledges open on Kickstarter, so you can grab one there. I'll put a link in the description. Thank you again for watching the podcast and I'll see you in the next.