预测未来的机器人：Dyna-2 与物理 AI——对话 Jason Ma

在我们进入今天的节目之前，我有一小段话要对听众说。谢谢。如果你们没有选择点击并收听我们的内容，我们就无法为你们带来你们如此明确想要的 AI 工程、科学和娱乐内容。几乎每天都有人找我们谈赞助。但幸运的是，你们中有足够多的人订阅了我们，让这一切在没有广告的情况下也能持续下去，我们希望保持这种方式。但我只想请大家帮一个忙。你能做的最强大、完全免费的一件事，就是点击那个订阅按钮。这是我唯一会请求你们做的事。这对我和我的团队来说意味着一切，他们每周都在如此努力地把 Inspace 带给你们。如果你们订阅了，我向你们保证，我们会一直努力让节目变得更好。现在，让我们开始吧。

Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you. And it means absolutely everything to me and my team that works so hard to bring the Inspace to you each and every week. If you do it, I promise you we'll never stop working to make the show even better. Now, let's get into it.

好的，我们来到了现场，和 Jason Ma 在一起，他是 Dino Robotics 的联合创始人兼首席科学家。感谢你让我们来到你的……

All right, we're here on site with Jason Ma, co-founder and chief scientist of Dino Robotics. Thanks for having us in your...

当然。感谢你们能来。

Of course. Thank you for coming.

是的，非常有趣。我们看到了，你们会看到很多 B-roll，但我们看到了饮料被拉出来，很多邪恶的东西，第一次看到机器人拿着刀。所以，在这一切的背后，你有一个世界行动模型。那是什么？

Yeah, it's been a blast. So, we got to see, you'll see a lot of B-roll, but we got to see drinks being pulled, a lot of evils, robots with knives for the first time. So, under the hood of all this, you have a world action model. What is that?

是的，上个月我们发布了我们的世界行动模型，叫做 Dina 2。所以，这是第一个在超过 100 万小时数据上训练的世界行动模型。所以先回到你的问题，世界行动模型是一个能够预测未来视频、世界未来状态的模型。因此得名世界模型。但此外，你也可以利用这种未来生成来产生机器人可以在物理世界中执行的动作，以控制机器人。那么我们为什么这样做？这是一个非常强大的架构，因为如果你想想人类在物理世界中大脑是如何行动的，我们隐含地思考世界将如何根据我们的行动演变。所以现在我们有一个模型架构能够做到这一点来指导机器人的行动。我们发现，通过这样做，模型更加灵巧，更加鲁棒，并且更能够泛化到从未见过的环境和场景。

Yeah, so last month we released our world action model called Dina 2. So, it's the first world action model trained on more than 1 million hours of data. So going back to your question first, world action model is a model that's able to predict the future video, the future state of the world. So hence the name world model. But in addition, you could also use that future generation to also produce actions robots can execute in the physical world to control the robot. So why do we do that? It's a very powerful architecture because if you think about how humans act in the physical world in our brain, we are implicitly thinking about how the world will evolve based on our actions. So now we have a model architecture that's able to do that to guide robot actions. So what we find is that by doing so the model is much more dextrous, much more robust and much more capable of generalizing to environments to scenarios is never seen before.

是的。我想稍微引导一下这个问题。我认为很多人听到世界模型，他们会认为你在输出视频或世界或 3D 状态。嗯，你也在做动作。我认为大多数人不理解机器人技术背后的模型是什么。那么输入输出是什么？训练数据是什么？架构是什么？给我们所有细节。

Yeah. I want to prompt this question a bit. I think a lot of people hear world model and they think you're outputting video or world or 3D state. Um you're also doing action. I think most people don't understand what models are under the hood of robotics. So what is input output? What's the training data? What is the architecture? Give us all the details.

是的。是的。所以嗯对于输入，最重要的输入是视频，对吧？你知道，你可以想想机器人用摄像头捕捉的视频，对吧？所以过去很多秒的视频或图像。这就是输入。但对于机器人来说，真正有趣的是机器人本身也有很多执行器，很多关节编码器。所以机器人也知道，你知道，它的关节角度。所以这些也可以作为这些世界模型的输入。这与仅仅看视频的纯视频模型非常不同，对吧？所以这就是输入。此外，对于机器人，你希望能够与它交谈，指示它做不同的任务，对吧？所以你也可以喂给它语言指令。所以这些是我认为进入任何机器人模型（包括世界行动模型）的基本输入集。一旦我们有了这些，输出在基本层面上需要输出机器人可以在硬件上执行的动作。但在世界行动模型的情况下，真正有趣的是，除了动作，你还可以输出未来视频，对吧？所以基本上你可以先输出未来视频或它的某种表示，然后用它来指导机器人的行动。所以我的思考方式是，这有点类似于你知道思维链，你知道基本上机器人，模型在视觉、物理空间中进行推理，然后输出动作。所以这就是世界行动模型输出与其他类型的模型、机器人模型输出之间的关键区别。

Yeah. Yeah. So uh for inputs the most important input is video, right? You know, you can think about videos that the robot is capturing with its cameras, right? So video or images of the, you know, past many seconds. That's the input. But for robots, what's really interesting is that robot also has a lot of actuator, a lot of joint encoders on itself. So the robot also knows, you know, the angles of its joints. So those can also be inputs to these world model. That's something that's very different from just a pure video model that's only looking at videos, right? So that's the input. And additionally for robots you want to be able to talk to it to instruct it to do different tasks right. So you can also feed it like language instruction. So those are what I consider the basic set of inputs that go into any robotics models including world action models. And once we have that the output at the uh basic level it needs to output actions the robots can execute on the hardware. But in the case of world action model, what's really interesting is that in addition to the actions, you can also output future video, right? So basically you can almost output the future video or some representation of it first and then use that to guide robot actions. So the way I think about is like this is kind of analogous to you know chain of thought you know basically the robots the models are reasoning in the visual the physics space and then outputting actions. So that's the key difference between world action models outputs versus other kind of models, robot models outputs.

嗯，你发布了一篇关于你模型的好博文。我认为很多人理解语言模型或 Transformer 的下一个词预测器。在这种情况下，你分享了不少。有像扩散流匹配。你在预测下一个世界状态加动作。嗯，你说的一件事是 100 万小时。你如何确定架构？比如那里有什么选择？架构是什么？我觉得在语言模型中，有很多是拿现成的东西，拿数万亿个 token，做预测 SFT、RL，然后你就有了一个模型。这个呢？你如何从零到你的模型？有现成的 vanilla 架构吗？你决定所有这些吗？当前状态是什么？

Um you put out a good blog post around your model. I think a lot of people understand language models or transformers next token predictors. In this case you shared quite a bit. There's like diffusion flow matching. You're predicting next world state plus action. Um one thing you say is a million hours. How do you determine the architecture? Like what are the choices there? What is the architecture? I feel like in language models there's a lot of take something off the shelf, take trillions of tokens, do prediction SFT, RL and you have a model. What about this? How do you go from nothing to your model? Is there vanilla architecture off the shelf? Do you determine all this and what's what is the current state?

是的，我认为是的，所以思考我们如何为机器人构建模型的思维框架是，基本上有几件事要看。一是数据，第二是我们如何让模型尽可能多地从数据中学习。所以让我们先谈谈模型，对吧。所以我们发布的博文是我们的 D2 世界行动模型。所以底层是一个扩散 Transformer。所以 Transformer 本质上就像语言模型，你知道，嗯嗯序列序列匹配仅解码器。是的。

Yeah, I think yeah, so the mental framework to think about how do we build models for robotics is that there are essentially a couple things to look at. one's data and the second is like how do we make a model learn as much as possible from that data. So let's first talk about the model actually right. So the blog post we released is our D2 word action model. So under the hood it's a diffusion transformer. So transformer is essentially like the language model you know uh uh sequence sequence match decoder only. Yeah.

但真正有趣的是，在视觉领域，如果你想想生成图像、生成视频，以及在这个案例中机器人通过看图像来生成动作，人们发现扩散类架构效果非常好。那么什么是扩散模型？扩散模型是一种接收随机噪声，并通过迭代计算从噪声变成非常高维、非常清晰、具有非常细粒度特征的东西的模型。如果你想想生成图像，你想要锐利的边缘、锐利的角、色彩对比。而对于机器人，即使你不是在生成图像，你是在生成需要非常精确、非常平滑的动作，它也有和生成图像类似的考量。所以这就是为什么从第一性原理出发，整个领域——不只是我们——一直在用这种扩散 Transformer 架构来训练机器人基础模型。你需要 Transformer 的部分，因为 Transformer 是能够摄入最多数据并学习的东西。然后扩散帮助你细化输出，得到你在机器人上实际执行所需的平滑、精确、细粒度动作。这就是我们最近很多突破背后的架构。

But what's really interesting is that in visual domains, if you think about generating images, generating videos, and in this case robots looking at images to generate actions, people have found the diffusion kind of architecture to work really well. So what is a diffusion model? A diffusion model is a model that takes some random noise and through iterative computation can go from noise to something that's really high-dimensional, something really highly defined and with very fine-grained features. If you think about generating images, you want sharp edges, sharp corners, color contrast. And for robotics, even though you're not generating images, you're generating actions that need to be very precise, very smooth. So it has kind of the same consideration as generating images. So that's why from first principles, the whole field—not just us—has been using this kind of diffusion transformer architecture to train robotics foundation models. You need the transformer aspect because transformer is the thing that can just ingest the most amount of data and learn. Then diffusion helps you refine the outcome to get the smooth, precise, fine-grained motion you need to actually execute on the robot. So that's the architecture that goes behind a lot of our recent breakthroughs.

我们在播客里之前讲过流匹配。它主要是一种效率上的加速。比如说你通常需要 40 到 100 个扩散步骤，你可以把它减少到四步。

And we've covered flow matching before on the podcast. It is mostly an efficiency speed-up. Like let's say you normally would take 40 to 100 diffusion steps, you can reduce it to four.

是的，没错。对于扩散模型，实践中一件很难的事是你需要调调度器——比如你把初始噪声输入转换成最终输出的程度。所以流匹配有点像扩散的一个特例，让变换简单得多。实践中你不必调那么多，效果也一样好。这就是为什么我觉得大约两三年前，大多数图像和视频的生成建模已经从扩散模型转向流匹配模型。但概念是一样的，流匹配只是扩散的一个特殊、更容易调的变体。

Yeah, that's right. So with a diffusion model, one thing that's really hard in practice is that you need to tune the scheduler—like by how much you're converting the initial noise input into the final output. So flow matching is kind of like a special case of diffusion that makes the transformation much simpler. In practice, you don't have to tune as much and it works just as well. That is why I think about two to three years ago, most generative modeling for image and video has shifted from a diffusion model to a flow matching model. But the concept is the same, and flow matching is just a special, simpler-to-tune variant of diffusion.

是的，这其实让我想到，流匹配显然在质量上有取舍。

Yeah, it actually makes me think that so flow matching obviously has trade-offs in quality.

是的。

Yeah.

但实际上，它非常适合动作扩散和机器人，因为你可以实时重新评估。

But actually, it's very suited for action diffusion and robotics because you get to re-evaluate in real time.

没错。

That's right.

而在图像里，基本上是一次性的。

Whereas in images, it's basically one shot.

没错。

That's right.

这挺有趣的。我们玩了叠衣服机器人，比如随便扔一条毛巾进去。模型会适应，对吧？

Which is pretty fun. We got to mess with the laundry folding robot like you know throw a random towel in. Um the models adapt, right?

是的。

Yeah.

我觉得你们模型的一个关键点是它们是通用型的。所以一个基础模型，然后你大量地适配它们。这是怎么运作的？基本上用 45 分钟的数据你就能泛化到——我们看到了一个从冰箱里拿饮料的例子。那里面的思路是什么，一个通用模型，然后你如何适配不同任务？

I think one key thing about your models is that they're made to be generalist. So one foundation model and you adapt them a lot. How does that work? Basically in 45 minutes of mapped data you can generalize towards—we saw an example of pulling drinks out of a fridge. What's the mindset there of one general model and then how do you adapt to different tasks?

是的。我们思考的方式是，对于机器人，我们需要构建一个非常通用、强大的基础模型，对吧？然后因为如果你想想人类在物理世界中如何互动和行为，我们有非常强的基础知识，可以随时做很多任务。但如果你想成为任何给定任务的专家，就需要一定量的训练。我认为要训练一个像我们在这里构建的通用机器人，同样的原则必须适用。所以我们的方法是先训练一个通用模型，用我们能找到的尽可能多的数据来训练。所以我们的 Dino 报告——我们的模型以超过一百万小时的人类数据作为起点进行训练。然后有了那个强大的基础，我们发现我们可以用非常少的微调快速让机器人或模型做许多许多不同的任务，只需告诉模型这是我想做的任务，这里只有很少量的演示，模型就能轻松完成你见过的一堆任务。

Yeah. So the way we think about it is that for robotics we need to build a very general, strong base model, right? And then because if you think about how a human interacts and behaves in the physical world, we have very strong base knowledge, we can just readily do a lot of tasks. But if you want to become an expert at any given task, there's some amount of training that needs to happen. And I think to train a general-purpose robot like the one we're building here, the same principle has to apply. So our approach is to first train a general model that's trained on as much data as we can find. So that is when our Dino report—our model is trained over a million hours of human data as a starting point. And then given that strong foundation, what we found is we can quickly make the robot or make the model do many, many different tasks with a very minimal amount of fine-tuning, to just tell the model here's the task I wanted to do and here's a very few amount of demonstrations, and the model can readily do a bunch of the tasks that you have seen.

我们其实有不少做数据、卖数据的人。什么是好数据？什么是坏数据？你想买什么？

We actually have quite a few people who deal in data, who sell data. What is good data? What is bad data? What do you want to buy?

是的。我认为这些都是开放问题。显然，我认为最重要的事情是人类做任务的自我中心数据。在这个基本陈述上你还会修改什么？

Yeah. I think that these are all open questions. Obviously, I think the top important thing is egocentric data of humans doing tasks. What else would you modify on that basic statement?

是的。

Yeah.

是的。所以我认为数据是当今机器人领域最重要的事情，因为不像语言模型，没有互联网，对吧？所以每家构建模型、构建硬件的公司都必须考虑数据。这里大致有几种不同的运作方式。一种是你自己收集大量数据，这也是我们在做的。

Yeah. So I think data is the most important thing in robotics today because unlike language models, there is no internet, right? So every company that's building models, building hardware has to think about data. So there's roughly a couple different ways of operating here. One is that you can collect a lot of data yourself, which is what we are doing as well.

你这里有员工在做这个。

You have employees here doing that.

我是说，办公室里到处都是机器人，对吧？但如果你想想机器人基础模型需要多少数据，我认为任何单个公司自己能收集的数据量都远远不够，对吧？这有点像你只靠 Reddit 或只靠 Facebook 来构建互联网。那些是很好的网站，很多用户，但还不够，对吧？所以当我们考虑数据时，必须有某种数据获取发生，对吧？你买数据，你看已经存在的数据。然后有很多不同的层级，对吧？你提到了数据质量。我认为机器人领域的数据质量是一个非平稳问题，意思是对于你的模型来说，什么是高质量数据会随时间变化。比如当你的模型什么都不知道，当它是零的时候，任何数据都有用，对吧？你可以用很多视频数据。在自我中心数据的情况下，我认为它实际上是预训练机器人的一个非常好的方式，因为人类数据如此多样——它涵盖了你所能想象的所有身体活动——这给了机器人模型一个非常强的先验，关于世界如何围绕一个具身行动者——人类——演化。但人类数据的问题在于，在你的机器人看起来完全像人类之前，人类动作和机器人动作之间会有一些差距。对吧？所以今天仅仅通过人类数据训练，不太可能直接执行机器人动作。所以你仍然需要一些使用机器人自身数据的适配，对吧？但机器人自身数据在机器人上收集会昂贵得多，也难收集得多。

I mean, there are robots everywhere in the office, right? But if you think about how much data you need for a robotic foundation model, I think the amount of data any individual company can collect by themselves is just nowhere near sufficient, right? It's kind of like you're trying to build an internet by just having Reddit or just having Facebook. There are great websites, a lot of users, but it's not enough, right? So when we think about data, there has to be some sort of data acquisition happening, right? You buy data, you look at data that's already out there. And then there are many different tiers, right? You mentioned data quality. I think data quality in robotics is a non-stationary problem in the sense that what is high-quality data for your model changes over time. Like when your model doesn't know anything, when it's zero, any data is useful, right? You can use a lot of video data. In the case of egocentric data, I think it's actually a very good way of pre-training robots because human data is so diverse—it covers every single physical activity you can possibly imagine—and that gives the robot model such a strong prior on how the world can evolve around an embodied actor, a human. But the issue with human data is that before your robot looks exactly like a human, there's going to be some gap between human actions and robot actions. Right? So just by training on human data today, it's not very likely you can just execute robot actions directly. So you still need some amount of adaptation using the robot's own data, right? But the robot's own data collected on a robot is going to be much more expensive and much harder to collect.

我们正在思考的配方大致是：如何利用当今最大的数据来源，比如第一人称视角数据，让模型对物理操作能覆盖的一切拥有强大的先验，然后将其与少量机器人数据结合，使机器人掌握我们感兴趣的任务。在机器人领域，数据质量和数据获取是一个迷人且多面的问题，但现实是没有单一的制胜策略，比如“哦，我只需要收集这一种数据就能得到强大的基础模型”。你真的需要思考：这些人类数据覆盖了机器人认知的这部分，我需要机器人数据来覆盖机器人所需的低级反射灵巧性，然后把它们整合到一个模型中。我认为这最终能产生一个真正适应性强、真正强大的模型。

So the kind of recipe we're thinking about is: how do we take the largest source of data we have today, like egocentric data, to give the model a strong prior over everything physical manipulation can cover, and then combine that with a small amount of robotics data to make the robot master whatever task we're interested in. Data quality and getting data in robotics is a fascinating, multifaceted problem, but the reality is there's not a single winning strategy, like 'oh, here's one kind of data I'm only going to collect on that, which will give you a strong foundation model.' You really need to think about: this human data covers this part of robot cognition, I need robot data to cover the low-level reflex dexterity the robot also needs, and put them into one model. And I think that's what gives a really adaptable, really strong model at the end.

好的。我再替那些想与你合作的听众问一下。你想要哪些数据供应商？这是你的数据征集。

Yeah. I'll prompt you just again for the people listening who want to work with you. What data vendors do you want? This is your call for data.

是的。不，这是个好问题。在我们的 Dino 2 博客末尾，我们实际上有一个电子邮件地址，叫 data partnerships at dina.co。所以，如果你是一个数据供应商，拥有大量高质量数据，无论是第一人称视角数据、机器人数据、视频数据，还是任何你认为能帮助机器人的东西，请发送到该邮箱。我们的团队会查看并回复你。

Yeah. No, that's a great question. So in our Dino 2 blog at the end we actually have an email address called data partnerships at dina.co. So if you're a data vendor that has a lot of high quality data, let it be egocentric data, robot data, video data, or anything you think could help robots, send it to the email. Our team will take a look and we'll get back to you.

是的。但在你们这个阶段，你们可能不再购买现成的第一人称视角数据了，对吧？你们可能更想明确需求。有没有哪个细分领域是缺失的？你需要 Scale AI，你需要一个核心来做这个。

Yeah. But like at your stage, are you probably not buying any more egocentric data off the shelf, right? You probably want to be like, here's what I need. Is there a niche that's lacking? You need a scale AI, you need a core for like doing this.

我明白，那些在机器人领域不奏效的定制方案可能在想，比如某个细分工厂，就像你展示的插管任务，你能找到的数据就那么多，但很多这样的数据对你有帮助吗？

I see the custom that don't work in robotics are probably thinking like okay some niche factory like you know you showed putting tubes in there's only so much data you'll find about that but like would a lot of that help you

是的，不，我的意思是，如果你真的想攻克一项任务，拥有来自该任务的数据肯定会大有帮助，对吧？所以对我们来说，我们仍然认为仅仅一百万小时的数据是不够的，这只是开始，我们展示了第一个百万小时规模的模型，但我们需要更多。因此，我们仍在寻找大量原始的多样化数据，同时也在思考如何获取更多定制数据，无论是内部收集还是通过合作伙伴，针对非常具体的细分任务，因为我认为你仍然需要这些数据才能达到卓越，让任何模型在物理世界中真正有用。这有点像人类，如果肖恩或我明天去 Rosewood 酒店工作，我们无法按照他们的要求铺床，对吧？我们需要大量训练。所以那种特殊数据，就像马克·C 今天为语言模型所做的那样，我认为对机器人技术也将非常有价值。

Yeah no I mean if you really want to nail a task having data from that task will certainly help you a lot right so for us it's like we still think you know just a million hours of data is is not enough it's just the beginning like we show the first model on a million hour scale but we need more than that right. So we're still looking at a lot of just raw diverse data uh that we can get but we're also thinking about how do we get more custom data either collected inhouse or by our partner for like really you know specific niche tasks because I think you still need that to get to you know excellence that you know any model needs to actually be useful in the physical world right it's kind of like humans you know uh you know if uh you know Sean or me you know we go work at Rosewood Hotel tomorrow, we're not going to be able to make the bed the way they want, right? We need a lot of training. So that kind of data like special data, it's kind of like what Mark C is doing today for language model, I think will be very very valuable for robotics as well.

是的。所以酒店，你知道，给员工头上绑个 GoPro，可能他们自己头发里还有更多。呃，我们讨论过的一件事是缩小模拟与现实的差距，非常有趣的是，现在你基本上可以即时构建环境。你试过用前沿模型做这个吗？你的经验是什么？它们够好吗？你是否在使用任何世界模型机器人公司来生成这些并训练？

Yeah. So hotels, you know, strap a GoPro on your employees heads and and there's probably more themselves in their in their wig. Uh so so one thing that we discussed is like the the closing the sim to real uh and uh what's very interesting is like now you can basically fabricate environments on the fly. Uh have you tried that with the the frontier models um what's your experience? Are they good enough? Are you using any world model robotic like world model companies to generate these and train on them?

是的。所以我认为为机器人技术生成合成环境、合成数据、模拟数据将是下一个前沿领域之一，因为今天人们关注的是第一人称视角数据、机器人数据，这些正在规模化，但我认为合成数据将成为规模化的一个新维度。回到你的问题，是的，我们一直在探索使用前沿模型为我们生成合成环境，基于我们的数据、我们关心的环境，令人惊讶的是，我认为现在这项技术已经达到了一个非常值得深入研究的阶段。背景是，在我的博士研究中，我主要专注于基于模拟的机器人训练。当时模拟还没准备好，但我们有一篇叫 Eureka 的论文，这是第一篇采用当时的大型语言模型 GPT-4 的论文，它必须自我生成环境、为强化学习生成奖励函数，并在模拟中执行操作。最后我们有一个很棒的演示，机器人旋转铅笔，当时真的很酷，但那时这类事情仍然非常有限，因为当时模型没有好的视觉能力，没有内在的 3D 理解，对吧？所以它基本上只是生成代码，并利用非常有限的文本反馈来尝试提出新的模拟环境。但今天，这些前沿模型的视觉能力简直疯狂。它们可以做真实到模拟的 CAD 建模、3D 重建等等。我认为现在正处于一个临界点，这项技术可以真正加速机器人技术。对我们来说，我们将其视为加速内部进展的一种方式，因为我们在构建模型，也在构建硬件。所以如果前沿模型能以这种方式给我们更多数据，为我们生成真正量身定制的数据，那对我们极其有用。我们一直在思考的另一件事是，与其使用这些模型生成 3D 环境，我们还使用了自己的世界模型，就是我之前告诉你的世界行动模型，因为该模型不仅生成动作，还能生成视频，模型也可以生成自己的训练数据。所以这变成了一个自我实现的飞轮，我认为这也会很强大。总的来说，我对这个方向非常兴奋，因为我认为即使我们已经扩展到一百万小时的数据，我们正在部署大量机器人。机器人实际上已经开始做非常酷的事情。我们才刚刚开始，只是冰山一角，未来还有更多。

Yeah. So I think generating synthetic environments, synthetic data, simulation data for robotics is going to be one of the next frontiers, because today people have looked at egocentric data, robot data, you know those are scaling, but I think synthetic data will be a new dimension of scaling. So going back to your question, yeah we have been exploring using frontier models to generate synthetic environments for us on our data, on our environments that we care about, and surprisingly I think now it's at a point where this technology is actually very much worth going really really deep into. So for some context, in my PhD I primarily focused on simulation based training for robotics. Back then simulation wasn't ready, but we had this paper called Eureka, which was the first paper taking a large language model at the time, GPT-4, which just had to self-generate environments, generate reward functions for reinforcement learning, and just do things in simulation. And at the end I think we had this really nice demo of the robot spinning pencils, right, that was really cool at the time, but that kind of thing was still very limited at the time because back then the model didn't really have good vision capability, didn't have an innate 3D understanding, right? So it was basically just generating code and using very limited feedback from text to try to come up with new simulation environments. But today, the vision capability of these frontier models is crazy. They can do real-to-sim CADing, 3D reconstruction, all of that. I think now it's at a tipping point where this technology could really accelerate robotics. And for us, we think about that as a way to accelerate our internal progress, because we're building models, we're building hardware. So if a frontier model can give us more data that way, generate really tailored data for ourselves, that's extremely useful for us. And another thing we've been thinking about is instead of using these models to generate 3D environments, we have also used our own world model, the world action model I was telling you about, because the model not only generates action, can also generate videos, the model can also generate its own training data. So that becomes a self-fulfilling flywheel that I think will also be powerful. So overall I'm just really excited about this direction because I think even though we have scaled to a million hours of data, we are deploying a bunch of robots. The robots are actually already starting to do very cool things. We're just at the beginning, just the tip of the iceberg for what is there to come.

对于研究方面的人，我们最近在播客中邀请了 Runway。他们的一件事是机器人角色模型。稍微转向那个方向。所以，你知道，接下来人们会关注它。关于你们自己制作模型，我有一个问题是你们在什么约束下运作？所以，如果你与语言模型做个类比，你会看到像 GPOSS 20B 非常适合普通 MacBook。是的。然后有一个更大的适合单个 GPU。

For those on the research side, we recently had Runway on the podcast. One of their things is robotic role models. Shifting a little towards there. So, you know, next people to look at it. One of the questions I have about you making your own models is what is the constraints you operate in? So, if you draw a parallel to language models, you see like GPOSS 20B is very fit for your average MacBook. Yeah. Then there's a larger one that's fit for a single GPU.

你们是在机器人模型的约束下工作吗？是在设备上、云端，还是混合模式？这对你们的数据缩放定律有什么影响？

Do you work within the constraints of robot models on-device, in-cloud, or a hybrid? And how does that affect your scaling law of data?

是的。

Yeah.

一百万小时很棒，但我们只是扩大参数和数据吗？

A million hours is great, but do we just scale up parameters and data?

是的。不，这是个好问题。所以我认为机器人与语言模型和其他应用真正不同的地方在于，对于编码智能体和语言模型来说，有延迟是可以的。

Yeah. No, that's a great question. So I think what's really different about robots versus language models and other applications is that for coding agents and language models, it's okay if there's latency.

嗯，就像你知道的，今天每当我问 ChatGPT，它要花几秒钟才能吐出东西。但对于机器人来说，推理延迟必须非常短。就像机器人需要行动，它不能只是拿着刀三秒钟什么都不做。那会吓到人的，对吧？

Well, like you know today whenever I ask ChatGPT it takes like a couple seconds before it spits out anything. But for robots, the inference latency has to be very short. Like the robot needs to act, it can't just hold a knife and not do anything for three seconds. That's going to freak people out, right?

是的，我们见过有人把真手放进去，我们的模型相当聪明。所以那没问题。但一般来说，如果你考虑非常动态的任务，比如你想让机器人拿着网球拍并且颠球，对吧？这是非常动态、非常反应性的。你不能只是像物理那样，重力不会等你，对吧？所以出于这个原因，机器人模型需要能够非常快速地进行推理。而最快的方法是在本地部署，就像在设备上，对吧？因为如果你通过网络调用模型，没有保证。

Yeah, we saw someone put their real hand and our model's pretty smart. So that's okay. But in general, if you think about very dynamic tasks, for example, you want your robot to hold a tennis racket and juggle the ball, right? This is very dynamic, very reactive. You can't just like the physics, gravity doesn't wait for you, right? So for that reason, robotics models need to be able to infer very fast. And the fastest way to do that is by serving on-prem, like on-device, right? Because if you call a model over the network, there's no guarantee.

开车。

Drive a car.

没有保证，对吧？所以出于这个原因，最终部署的模型必须具有合理的规模，可以在本地服务。但这并不意味着你训练的初始模型也必须很小，对吧？就像在语言模型中，你有一个大模型，人们做蒸馏，对吧？就像让一个小模型变得非常聪明的最佳方式是做蒸馏，对吧？你训练最大的模型，然后蒸馏，对吧？所以我认为在机器人领域我们正在达到这一点，但我认为这就是人们将会趋同的方向。

There's no guarantee, right? So for that reason, the model that gets deployed eventually has to be of reasonable size that can be served locally. But that doesn't mean that the initial model you train also has to be small, right? Like in language models, you have a large model, people do distillation, right? Like the best way to make a small model very smart is by doing distillation, right? You train the biggest model, then you distill, right? So I think in robotics we're getting there, but I think that's what people are going to converge on.

那推理呢？你们有自己的推理栈吗，还是很多架构是从扩散模型之类的借鉴的？我猜你们可以借鉴一些东西。有什么想分享的吗？

How about inference? Do you have your own inference stack, or is a lot of this architecture you've pulled from diffusion and whatnot? I'm guessing you can pull off of something. Anything you want to share?

是的，所以我们为部署的任何模型构建自己的推理栈，因为就像我说的，延迟非常重要。所以整个栈对机器人来说极其重要，对吧？你想为你运行的模型优化推理速度，那里有很多工作。所以这就是我们所做的。

Yeah, so we build our own inference stack for whatever model we deploy because, like I was saying, latency is so important. So the whole stack is extremely important for robotics, right? You want to optimize the inference speed for the kind of model you're running, and there's a lot of work in there. So that's what we do.

是的，你们真的是全栈，从执行器一直到工业设计，ID，我之前不知道这个缩写，显然还包括模型和数据。

Yeah, you're really full stack, like everything from the actuators all the way to the industrial design, ID, which I didn't really know as an acronym, obviously to the models and data as well.

我很好奇，这些是你们自己的执行器吗，还是有多少是你们自己的？当然，设计，你知道，它不是人形，但执行器，比如在机器人栈中深入到什么程度？

I'm curious, are these your own actuators, or how much of this is your own? Of course, the design, you know, it's not a humanoid, but actuators, like how far down the robotic stack?

是的。所以我们考虑需要拥有什么与可以购买什么的方式是，如果一个组件的输入输出非常清晰，并且已经有极其好的成熟解决方案，那么自己构建就没有意义。例如，在这个机器人上，你看到激光雷达，对吧？那些黑色圆圈，那些是激光雷达，帮助机器人检测障碍物和导航周围环境。所以这是非常成熟的技术，人们已经在自动驾驶汽车中使用了几十年。所以我们不需要从头构建激光雷达，对吧？一些摄像头也是如此。但对于我们认为对整个系统最终性能极其关键的任何东西，我们的愿景是能够拥有并自己构建。所以一件事是，你提到了执行器。那是我们正在努力的事情，我们将自己拥有。原因是执行器的质量和设计方式真的影响你建模机器人的好坏，对吧？执行器是驱动机器人的东西。为了能够在模拟中很好地建模你的机器人或进行任何类型的建模，你需要选择非常好的执行器，实际上可以很容易建模。所以这就是我们最终关心的。所以我们将拥有那个。机器人的比例也是如此。这个机器人的比例是根据真人测量的。为什么我们这样做？因为我们在人类数据上进行预训练。

Yeah. So the way we think about what we need to own versus what we can buy is like if the input and output of a component is very clear and there's already an extremely good mature solution, then there's no point to building it ourselves. For example, on this robot, you see the lidars, right? Those black circles, those are lidars to help the robot detect obstacles and navigate surroundings. So that's a very mature technology that people have been using for decades in self-driving cars. So we don't need to build lidar from scratch, right? Same goes with some cameras. But for anything that we think is extremely crucial for the whole system's performance at the end, our vision is to be able to own that and build that ourselves. So one thing is, you mentioned actuators. So that's something we're working on and we're going to own ourselves. The reason is that the quality and how you design the actuator really matters for how well you can model the robot, right? Actuator is what drives the robot. To be able to model your robot very well in simulation or to do any kind of modeling, you need to select actuators that are very good and can actually be easily modelable. So that's what we care about eventually. So we're going to own that. Same thing with the robot's proportion. This robot's proportion is measured against real people. Why do we do that? Because we're pre-training on human data.

哦，是的。

Oh yeah.

最佳迁移，对吧？所以有很多这类决策进入硬件，非常相关，因为最终，你知道，是硬件承载智能。智能是极其重要的东西。我们要确保硬件为此服务。

Best transfer, right? So there's a lot of these kinds of decisions that go into hardware that's very much tied to, because eventually, you know, it's hardware that carries intelligence. Intelligence is the extremely important thing. We want to make sure the hardware serves for that.

是的。是的。我只想强调，我不知道这是否与你相关，但我有过的最有趣的机器人讨论之一，是有人对执行器非常焦虑，特别是磁铁，因为显然我们美国不制造磁铁。

Yeah. Yeah. I'll just highlight, I don't know if this is relevant to you, but one of the most interesting robotics discussions I've had is someone who was very stressed out about actuators and specifically magnets because apparently we don't make magnets in the US.

不，我们不制造。

No, we don't.

而且只有两个国家制造磁铁。

And there's only two countries that make magnets.

稀土材料。

Rare earth material.

是的。对。我不知道化学，什么材料？我不知道那是什么，但是的。

Yeah. Right. I don't know what the chem, what's the material? I don't know what the thing is, but yeah.

所以是的，如果有人对此感兴趣，那显然是我们拥有的一个瓶颈。

So yeah, if anyone's interested in that, that's apparently a bottleneck that we have.

不。是的。

No. Yes.

那么，我们该怎么办？

So, what do we do?

是的，我的意思是这是个好问题，对吧？我认为有很多关于将制造业带回美国的激动人心的讨论，但尤其是对于机器人来说，这实际上比看起来要难得多。我要说即使现在，美国也没有一家公司真正在美国从头开始建造一切，对吧？即使他们说他们在美国组装机器人，他们实际上在美国建造什么，他们仍然从世界其他地方采购原材料。我认为这是作为机器人建造者真正需要意识到的事情，对吧？就像我认为作为一家全栈机器人公司，我们意识到供应链有多长，整个问题实际上极其困难。而且我认为如今很多讨论集中在机器人基础模型、模型数据上。但我认为硬件以及建造好硬件有多难被严重低估了。但我认为那将极其重要。

Yeah, I mean it's a good question, right? I think there's a lot of exciting discussions on bringing manufacturing back to the US, but for robots especially, that's actually much harder than it seems. And I would say even now there isn't a company in the US that's truly building everything from scratch in the US, right? Even when they say they assemble robots, they build whatever actually in the US, they're still sourcing the raw materials from other places in the world. And I think that's something to really be aware of as a robot builder, right? Like I think as a full stack robotics company, we realize just how long the supply chain is and how long the whole problem is actually extremely hard. And I think these days a lot of the discussion centers around robot foundation models, model data. But I think hardware and just how hard it is to build good hardware is very underemphasized. But I think that's going to be extremely important.

这是一个平台，对吧？

This is a platform, right?

实际上是平台。

Actually platform.

是的。有些公司选择成为，我们称之为无平台。他们只做模型，你会说我们在任何平台上部署。我们在 Optimus 上部署，现成的机器人。

Yeah. There are some companies that choose to be, let's call it platformless. They only do the model and you'll say we deploy on any platform. We deploy on Optimus, off-the-shelf robotics.

呃，所以这是一个选择，对吧？就像显然你拥有 everything 所以你可以这样做。

Uh and so it's a choice, right? Like obviously you own everything so you can do that.

你觉得未来更多是在美国本土制造，还是打造自己的全球供应链？哪个方向更可能？

Do you see yourself building more in the US or building your own global supply chain? What's the direction that is likely?

对。我们目前正在探索很多不同的选项，对吧？因为全球供应链会涉及一些地缘政治问题、运输延误之类的。

Yeah. So we are currently exploring many different options, right? Because with global supply chain, there are some geopolitical things, shipping delays and all that stuff.

真的是用船运机器人。

Literal ships for robots.

对。在美国，从零到一需要一些时间，对吧？所以我们正在思考一个既能让我们在美国制造更多，又能利用全球供应链的解决方案。但回到你之前的问题，有些公司选择只做模型，而我们做所有东西。我认为把这件事做对非常重要。我觉得对机器人来说，难的是构建整个系统，构建一个完整系统所需的各种要素。我之前在 DeepMind、在 Nvidia，主要是做模型。我看到了当你无法控制硬件、无法优化硬件时，系统的上限在哪里，对吧？因为如果你考虑机器人的价值创造，最终机器人是为了提供服务、解决一些物理任务。所以如果你想解决物理任务，你就希望能优化一切，这很重要。

Yeah. And in the US, you know, it's just that going from zero to one will take some time, right? So we're trying to think about a solution that can both allow us to build more in the US while leveraging global supply chain. But I do think, going back to your previous question on some companies choosing to just build models — we build everything. I think it's very important to get this right. I feel that for robotics, the hard thing is building the whole system, the necessary things to build a whole system. Previously I was at DeepMind, I was at Nvidia, primarily just building models. And I saw the upper bound of the system when you don't control the hardware, when you don't get to optimize that, right? Because if you think about the value creation for robots, eventually robots are here to deliver service, to solve some physical tasks. So if you want to solve physical tasks, you want to be able to optimize everything, and that's important.

我是说，机器人就在这儿。你得跟我们讲讲这个 Fine。你们自己的定制平台。你们管它叫什么？

I mean, we have the robot. You got to tell us about the Fine. Your own custom platform. What do you call this?

我们管它叫什么？

What do we call it?

对。所以它不是人形。它非常流畅，但那个旋转臂是什么？

Yeah. So it's not humanoid. It's very smooth, but what's the rotating arm?

对。我们通常把它看作半人形。基本上上半身像人形，有肩膀、有手臂，腰部还有四个自由度。所以基本上能完成人类上半身的所有动作。但它是半人形，因为下半身是轮子，对吧？为什么选轮子？因为我认为这里有个权衡：实用性与性能之间的最优取舍。我觉得用腿的话，能耗会高很多，稳定性会差很多，安全性极难保证。所以目前对于足式机器人，其实无法保证机器人始终直立、保持平衡。我相信你们看过很多机器人跳舞的视频，对吧？它们很容易摔倒。

Yeah. So we think of this as generally like a semi-humanoid. So basically the upper body is like a humanoid. It's got, you know, of course shoulders, got arms. It also got like four degrees of freedom on the waist. So you can basically do all the upper body motion that a human can do. But it's a semi-humanoid because the lower body is wheels, right? So why do we choose wheels? Because I think there's a difference. There's an optimal practicality versus performance trade-off. I think with legs, the energy consumption will be a lot higher. You'll be a lot more unstable. Safety will be extremely hard to guarantee. So right now for legged robots, there's actually no guarantee that the robot will stay upright, balanced the entire time. I'm sure you guys have seen a lot of those robot dancing videos, like the robot dancing. Yeah, they fall over very easily.

不，但你也看到过那些明显不是量产版的演示，你可以推机器人、把它推倒，它表现很好，但那个机器人只是擅长平衡，对吧？

No, but you also see the demos that you can tell are not production, where you can box the robot and push it over and it's very good, but that robot is just good at balance, right?

我觉得你们更有意思的创新是——我们有一个原型机，叫 Mewtwo，在后面。这个只有上下轴，而你们选择转向像这样的设计，能够倾斜。我们接下来要展示洗衣服的演示，需要向机器倾斜身体。对，那是另一个自由度，我甚至不知道怎么描述。不是俯仰，也不是偏航。

I think the more interesting thing that you guys have innovated is — so we have one of the prototypes, which is the Mewtwo. Back here. This is just like an up-down axis, and you've chosen to then move towards something like this, which has the ability to lean. And we're going to show the laundry demo where you have to lean into machines. Right. That's another degree of freedom that I don't even have the words for it. It's not pitch. It's not yaw.

对。那你们是怎么考虑这个的？

Yeah. So how did we think about that, right?

其实这是个很有用的练习。我想几年前我锻炼时不小心拉伤了一块肌肉，对吧？我相信你们也有过这种经历。然后接下来几天你动不了，或者因为睡姿不对，脖子很僵。我们都经历过。每当你经历这些，你就会意识到，哇，我们日常生活中有那么多肌肉、那么多自由度，我们习以为常。但当它们失灵时，做一些你觉得很自然的任务就变得非常别扭。对吧？所以我们设计机器人时也是这么想的。当你只有这种纯升降、只能上下移动的设计时，我们意识到很多人类躯干动作——我们日常体力活动大量使用腰和背。甚至像我想拿这个瓶子，我不只是动手臂。如果我必须刻意想着固定上半身，那会非常奇怪。我总会这样做，对吧？所以人类移动身体有很多潜意识的方式，我认为如果机器人有相应的自由度，同样会很有用。如果你再进一步想，好，我们想从人类数据中学习。人类一直在移动躯干。你希望能把这些数据输入模型来控制机器人。但如果你的形态不允许，迁移效果就会变差，机器人就不那么灵巧，动作也更别扭。

Like, actually it's a very useful exercise. I think sometimes a couple years ago when I was exercising, I accidentally pulled a muscle, right? I'm sure you guys have that experience too, right? Then the next few days you just couldn't move, or you pulled something because you slept in the wrong position, so your neck is very stiff. We have all experienced that. And then whenever you experience that, you kind of realize, wow, so many muscles, so many degrees of freedom that we use in our daily life that we take for granted. But when it's disabled, it's very awkward for you to do some tasks that you feel are very natural. Right? So that's also how we think about when we design robots. When you have this kind of design which is purely an elevator that could only go up and down, we realize a lot of the human torso movement — we use our waist, back a lot for day-to-day physical activity. Even like if I want to get this bottle, I'm not just moving my arms. It's actually very strange for me if I have to intentionally think about fixing my upper body. I would always do this, right? So there's just a lot of subconscious ways that humans move their body that I think will be equally useful if a robot has a degree of freedom. And if you further think about, okay, we want to learn from human data. The humans are moving their torso all the time. You want to be able to ingest that into the model to control the robot. But if you have a form factor that doesn't allow for that, then the transfer becomes less and the robot becomes less dexterous and it's just more awkward.

数据标注方面呢？比如，你见过人们——这是后训练微调的最后一步，实际用遥操作或远程控制机器人来映射动作，还是有一部分能泛化？

What's the data labeling side of this? Like, you've seen people — is this the last post-training fine-tuning side of actually the hour of using the robot teleoperated or remote controlled mapping motion, or does some of it generalize over?

你说的标注是指

When you say labeling, you mean

比如把你已有的预测状态的模型映射到你的特定硬件上。

Like taking the model you have that predicts state and mapping it to your specific hardware.

哦对。是的。它生成得很好。所以我在新机器人上展示的所有演示——因为新机器人不久前才准备好——所以我们 literally 没有很多数据，相比我们展示的固定机械臂，那些数据多得多。对吧。但新机器人，模型在预训练中没见过这些，在我们的一些预训练模型中。它能很快学会你看到的任务。我认为这很大程度上归功于——在硬件准备好之前，我们就已经致力于在人类数据上预训练，因为我们知道总有一天我们的机器人会像人一样。你可以在硬件准备的同时在训练方面领先。我认为现在我们看到迁移确实存在，而且非常有用。

Oh yeah. Yeah. It generates this very well. So for all the demos that I have shown you on the new robot — because the new robot, you know, it got ready not so long ago — so we quite literally do not have a lot of data compared to the stationary arms we showed you, which had much more data. Right. But a new robot, the model has not seen this in pre-training in some of our pre-trained models. It's able to learn the task that you saw like very quickly. And I credit a lot of that to — even before the hardware is ready, we already committed to pre-training on humans because we knew at some point our robots will be like human. You can get ahead on the training side while hardware is getting ready. And I think now we're seeing that the transfer is actually there and is actually very helpful.

谈谈评估方面。你可以有预测，可以让它做事，你知道偶尔我们会看到它失误。你们也测试非常小众的事情，比如装药瓶。这个过程需要多长时间？你们怎么考虑实际部署新任务、评估、迭代？那是什么样的？

Talk about the eval side of this. So you can have predictions, you can have it do stuff, and you know every now and then we see it miss stuff. You test on very niche things as well, like filling a pill bottle. How long does that process take? How do you think about the aspect of actually deploying on a new task, evaling, iterating? What does that look like?

对。如果我们想学习一个新任务，基本上我们会做一点机器人数据收集，然后模型这些

Yeah. So if we want to learn a new task, you know, we basically do a little bit of robot data collection, then the model these

一点是多少？

What is a little bit like?

对，大概几个小时，几个小时。对，有些任务我们只要几十分钟，对吧？对。

Yeah, like a couple hours, couple hours. Yeah, we're in some of the tasks like just tens of minutes, right? Yeah.

然后切菜那个花了 45 分钟。开冰箱那个——那就像一场面试。那就像是你们工程师的入职任务。

And then the chopping one was 45 minutes. The open-the-fridge was on—and that's an interview. That's like an onboarding task for one of your engineers.

对，对，对。所以，如果你想在这里工作，有个好玩的事。找个任务，让机器人去做。给我提个新任务，我会来办公室，收集一些数据，试个模型，看看效果。

Yeah. Yeah. Yeah. So, fun thing if you want to work here. Find the task, have the robot do it. Pitch me a new task and I'll come to the office, collect some data, try a model and see what happens.

系鞋带。

Tie your shoes.

朱莉很累人。对，那会很酷。但没错，所以你知道，收集少量数据，训练一个模型，然后我们就把模型部署到真实世界，然后看它的表现，然后

Julie is tiring. Yeah, that'll be cool. But yeah so you know a little small amount of data collected train a model then we just deploy the model in the real world and then we look at the performance then

填补收集的意思是，你有一个真人实际操控遥控器做一会儿任务，然后你训练，它做得还行，你改进，你怎么衡量

to fill in collected means like you are you have a person actually operating remote control doing the task for a bit and then you train and it does okay you improve how do you measure

对，对，所以当你训练模型时，你可以看某些指标，对吧？就像实际模型，它收敛得很好。之后，对于今天的机器人技术，我们只是在真实世界中运行模型，就在收集数据的同一套装置上，看看模型表现如何。有时模型开箱即用就很好，对吧？你真的不需要做太多。但当模型，你知道，可能达到 70%、80% 时，你就看看模型表现如何。你找出它不擅长什么。然后你可以再收集一点数据，然后迭代，对吧？我认为这就像是机器人版的基于人类反馈的强化学习，对吧？机器人展开行动，你可以给它打分，好、坏、还行，然后这些数据，机器人唯一的经验，可以用来训练模型。你也可以自己收集更多数据来训练模型，对吧？最终，你知道，我们的目标是，这个机器人将成为我们的平台。任何人都应该能够在它上面训练任务。

yeah yeah so when you're training the model there's uh you know certain metrics you can look at Right? Just like actual model it converges nicely. Now after that for robotics today we just run the model in the real world right on the you know same setup that the data is collected to see how the model does. Sometimes the model just works great out of the box, right? You don't really need to do much more. But when the model's, you know, like maybe like 70% 80% there, then you see how the model behaves. You figure out what it's not good at. Then you can like collect a bit more data, then iterate, right? I think of this is like the robotics version of like reinforcement learning from human feedback, right? the robot rolls out, you can score it good, bad, uh somewhat good and then that data, the robot's only experience can be used to train the model. And you can also just collect more data yourself to train a model, right? Eventually, you know, our goal is like, you know, this robot is going to be our platform. Anyone should be able to uh train tasks on it.

我想得到一点预测，比如时间线，它会发展到什么程度。现在，测试还有点简单，比如取出衣物。是条毛巾。你不能压坏它。切一些食物。还有另一方面，比如，好吧，布置这个拍摄工作室，对吧？移动东西。搬张桌子，设置灯光，我想。管理我的数据中心。对，对。插上东西。还有，给另一个机器人充电。对，对。但我们什么时候能看到工厂，或者我们需要扩大什么？是模型吗？是所有东西吗？看起来你的硬件已经为此准备好了，但我们什么时候能看到它发展到这样一个点，我可以从只是切一个东西，到你知道，好吧，这里有个食谱，就像全力投入，或者

I want to get a bit of a prediction and like timeline to where that goes. Right now, the tests are um somewhat simple, like take laundry out. It's a towel. You can't crush it. Um chop some food. There's also the side of like, okay, set up this filming studio, right? Move stuff around. Um bring a table, set up lights, I think. Manage my data center. Yeah. Yeah. Plug in stuff. Um and and also uh re like recharge another robot. Yeah. Yeah. But when do we see the factory or like what do we need to scale up? Um is it model? Is it everything? It seems like your hardware is set for that, but where do we see it go to a point where I can go from instead of just cut one thing to you know, okay, here's a recipe like full send or

对，对。我想说，要达到真正通用的机器人。嗯，就像基本上一个机器人或模型能做任何事，就像

Yeah. Yeah. I would say to get to truly general purpose robots. uh like like basically a robot or a model that does anything like

不是不是不是那个，好吧，所以为了在技术层面上保持脚踏实地，比如我们什么时候开始看到我们推动那个反馈，对吧，所以不是非常简单的切菜，嗯，也许是一个 5 到 10 分钟的序列，对吧，现在评估步骤可以在更多点中断，而不是只是装一个药瓶，嗯，10 种不同的药片全在一个序列中，对吧，是什么限制了我们，我们在推动什么

not not not that like okay so to keep it somewhat grounded on a technical level like when do we start to see us pushing that feedback right so instead of very simple chopping uh maybe a 5 to 10 minute sequence right where now the eval step it can break at many more points instead of just fill one pill bottle uh 10 different pills all in a sequence right what's limiting us what are we pushing on

对，我想，对，我想有几个。一个是，嗯，你看到的那个洗衣演示，对，完成整个循环大概要 1 小时，对吧，所以机器人能做一个完整工作的最长持续时间已经在小时范围内了

yeah I think yeah I think a couple of And one is that like uh the laundry demo that that you have seen yeah it's going to be like 1 hour to finish a whole loop right so the longest duration that a robot can do like a full job is already in the hour range

不错

nice

对吧，但你知道，要能对任何任务都做到那样，你知道，我认为瓶颈又回到了数据和硬件，对吧，所以对于某些任务，你知道，比如布置空间，你知道，你可能需要的不仅仅是平行夹爪，对吧，你可能需要更灵巧的东西，比如一只手，像人手一样，对吧？但如果你想在手上训练模型，那么你就受限于硬件的稳定性。比如今天的手，最好的手，灵巧的手仍然不够耐用。它们很快过热，对吧？所以这就成了瓶颈，对吧？就像你，我们，你知道，你几乎得放一个冰袋

right but you know to be able to do that for any task you know I think the bottleneck is coming back to data and uh hardware right so for some of the tasks you know for example setting up the space you know you might need more than like a you know the parallel jaw gripper right you might need something more dextrous like a hand, like a human hand, right? But then if you want to train a model on the hand, then you're bottlenecked by the hardware stability. Like hand today, the best hands, dextrous hands are still not uh durable enough. They overheat pretty quickly, right? So that becomes a bottleneck, right? Like you we you know, you quite literally have to put a you know ice bag

到我们不只是自动研究做出更好的模型。我们有硬件限制。

to we're not just auto researching make better model. We have we have hardware limit.

对。所以硬件限制是个大问题，但如果硬件受限，那会拖慢一切，对吧，因为要知道你在任何任务上取得进展，你最终必须在硬件上评估，对吧，所以如果评估不是非常自发、非常快，那也会限制你的研究进展，对吧，因为那样我们怎么收集正确的数据，对吧，我想，嗯，我们如何收集正确的数据来做极其复杂的事情，比如布置这个工作室，对吧，因为你可以想象，今天大多数数据是基于视觉的，对吧？就像视频流，还有，你知道，一些你手部运动的编码，机器人有关节。但人类有更多的传感器。你知道，我们听到东西，我们感觉到东西，闻到。

Yeah. So hardware limitation is a big thing but then if hardware is limited then that will slow down everything right because to know that you're making progress on any task you eventually have to eval on the hardware right so then without the eval being very like spontaneous very fast that also limits your research progress right because then how do we collect the right data right I think um how do we collect the right data to do extremely complex like setting up this uh studio right because you can imagine like today most of The data is like vision based, right? It's like video streams and like, you know, some uh uh encoding of your hand motion where the robots have joints. But humans have a lot more sensors. You know, we hear things, we feel things, smell.

闻到。

smell.

对。你闻到东西，对吧？所以，我们潜意识地利用所有这些来帮助我们在操纵物理世界方面如此出色。但今天，如果你根本没有捕捉到这些，那么即使我们扩展到数百万小时，你可能只是错过了任务的关键维度。所以你永远无法解决这个任务，对吧。所以我认为这些就是我认为的，你知道，拖延时间线的东西。但对于你今天可以高保真收集数据的任务或工作流程，那么我不一定认为存在根本性的瓶颈，对吧，就像我们本周向公众展示的那样，你知道，我们有一个物理智能体，可以像人类一样完成整个洗衣工作流程，对吧，因为对于那个任务，我们并不真的需要，哦，你实际上需要的不仅仅是平行夹爪，而是你实际上需要一些今天还没有的传感模式，对吧？所以，我确实认为，要拥有更通用的东西，最大的瓶颈仍然是硬件和数据。

Yeah. You smell things, right? So, we subconsciously use all of those to help us be so good at manipulating the physical world. But today like if you're not capturing that at all then even when we scale to millions of hours you might just be missing critical dimensions of the task. So you'll never be able to solve the task right. So I think those are like what I consider like you know what's dragging out the timeline. But for tasks or for workflows that you can collect data today to high fidelity then I don't I don't necessarily think there's a fundamental bottleneck right just like what we are demonstrating uh to the public this week you know we have a physical agent that can do the entire laundry workflow right as a human would right because for that task we don't really have like oh you actually need more than a parallel job but you actually need some sensing modality that's just not available today, right? So, I do think the biggest bottleneck to have a much more general thing is still hardware and data.

对。我想借此机会让你夸夸你的客户和合作伙伴。对。嗯，你有一些宣布的关系。嗯，我们可以从 Best Western 开始，然后也许一些你引以为豪的其他客户，酷故事，之类的。

Yeah. I want to take this chance to talk let you brag about your customers and partners. Yeah. Uh you have some announce uh relationships. Um we can start with Best Western and then maybe some of uh some of the others that you're that you're proud of, cool stories, anything like that.

对。所以，我们在网站上有部署博客文章，有更多细节。我真的推荐看视频，因为我觉得视频是

Yeah. So, we have a deployment blog post uh on our website that goes into much more detail. I really recommend watching the video because I think the video is

你想要 Best Western 那个吗？

Do you want the best Western one?

嗯，不。我们有一个视频涵盖所有合作的客户。这是一个非常非常好的视频。展示了我们在做什么，嗯，对，我觉得很棒。

Uh, no. We have like a one video for all the customers we're working with. It's a really really nice video. Shows what we're doing and uh yeah, I think it's great.

但没错，我们有客户，比如酒店行业的 Best Western。我们为他们折叠大量布草，比如毛巾。每家酒店都需要，每家健身房都需要，每个健身中心都需要。

But yeah, we have customers, you know, Best Western that's in the hotel space. We fold a lot of laundry like towels for them. Every hotel needs it. Every gym needs it. Every fitness center needs it.

目前的部署情况如何？比如我可以去哪里看？我该找谁去看？

What's the deployment so far? How like where can I go to? Which best person do I go to to see it?

好的。如果你想去看，我其实推荐去我们的另一个部署点，Dinton。你们去过 Dinton 吗？

Yeah. So if you want to go somewhere, I actually recommend going to our other deployment, Dinton. Have you guys been to Dinton?

去过。

Yes.

对。你知道最好的……

Yeah. You know the best...

非常瘦的说唱歌手。

Very thin rapper.

对。非常……

Yeah. Very...

我们应该在那里拍摄。

We should have filmed there.

对。我们应该直接在那里拍。

Yeah. We should have just done there.

对。

Yeah.

但在 Dinton，比如你去 Dinton 电话或者其他类似的高档餐厅，桌上会有餐巾，对吧？所以我们的机器人，今天我们的机器人为美国许多不同的地点，许多不同的高档地点折叠数千张餐巾。所以这实际上是我们最大的规模化客户，对吧？所以我想在这里做一个区分。今天有很多机器人公司在谈论部署、谈论试点，但真正困难的是不是把你的机器人放在一个地点或做一个试点。真正困难的是客户相信或验证你的机器人创造了足够的价值，以至于他们想在你的许多地点部署你的机器人。所以这就是我们在 Dinton 电话所发生的事情，据我所知，其他做类似通用机器人的公司还没有真正发生这种情况。所以在 Dinton，我们的机器人每天折叠数千张餐巾。质量标准极高，因为他们想要，比如餐巾的第二个前缝只比那个口袋低几厘米，我甚至不知道你怎么称呼它，那个口袋，这样你可以把筷子插进去。所以这实际上是非常精细的高精度，我可以给你看

But uh in Dinton like if you go to Dinton phone or other like uh you know somewhat upscale restaurant, there's napkins on your table, right? So our robots, our robots today fold thousands of napkins for uh many many different locations, many different gentle locations in the United States. So that's actually our biggest scaled customer, right? So I want to make a distinction here. Uh today there's a lot of robotics companies talking about deployments, talking about pilots, but what's really hard is not putting your robot in one location or doing a pilot. What's really hard is that the customers believe or validate that your robot creates enough value that they want to deploy your robots across many of their sites. So that's what's happened for us with Dinton phone which to the best of my knowledge hasn't really happened for other companies doing similar like general purpose robots. So at Dinton our robots fold thousands of napkins per day. It's the quality standard is extremely high, you know, cuz they want like okay uh the second forward seam of the napkin is only like uh couple centimeters below the uh I don't even know how you call it the the pocket so you can stick the chopsticks in. So it's actually very fine grain high precision has I can show you

在某种程度上，这正是机器人的优势。机器人也擅长这个，对吧？人类不是最一致的。

At some level that's what robotics are good for. Robots are good for too, right? Um humans are not the most consistent.

是的，

Yes,

尤其是在小细节上，而机器人可以在某种程度上完美地做到。有趣的是，你为通用机器人举的例子，而在某种程度上，我认为人们会假设你可以有一个专门的机器人，可以完美地折叠餐巾到精确的毫米，而不是像这样的东西，但

especially on small details whereas a robot can somewhat do it perfectly. It's interesting that's the example you bring for general purpose robotics versus uh at some level I think people would assume you can have a specialized robot that will fold napkins perfectly to the exact you know millimeter as opposed to something like this but

是的，我确信也许你可以，但操作可变形物体的难点在于，比如餐巾、毛巾，一切都是可变的，有无限多种可能的配置，所以这对专用机器人来说真的很难，对吧？我们发现，通过在机器人中开发通用模型并将其适应这些用例，你实际上可以更快地达到生产质量。真正好的是，现在 Dinton 已经扩大了规模，与我们签订了更大的合同，他们不仅希望我们的机器人折叠餐巾，现在他们希望我们的机器人，比如切割，这就是为什么我们有切割食物的例子。

yeah I'm sure uh maybe you can right but the hard thing about manipulating deformable object right like napkin towels everything is deformable there just infinite infinitely many possible configuration so that's really hard for a specialized robot, right? And what we find is that by developing a general purpose model in a robot and adapting it to these use cases, you can actually get to a production quality faster. And what's really nice is that now that Dinton has scaled, has signed a much bigger contract with us, they not only want our robots to fold napkins, now they want our robots, you know, like cut, you know, that's why we had the example to cut food to.

那实际上是第二个用例吗？

Is that actually the second use case?

那是他们正在考虑的用例之一。

That's one of the use case they're thinking.

哦，哇。我想那个点每天需要切很多蔬菜。

Oh, wow. I think that point needs to cut so many vegetables per day.

他们需要做饺子。

They need to do the dumplings.

没错。

Exactly.

做全套。那是最终 Boss。那是最终的。

Do the full. That's the final boss. That's that's the final.

呃，我认为你可以。我认为你有足够的灵活性和精确度来做到这一点。

Uh I think you can. I think you have the dexterity and precision to do this.

是的，

Yeah,

你有一点投票。

you have little poll.

但就像我知道不要弄破饺子。

But like I know don't break the dumplings.

是的。是的。我的意思是，那是明显的好处，你可以适应它们。我认为你真正提到的另一件事是速度，对吧？制作定制餐巾折叠，具有精确的层数，等等。

Yeah. Yeah. I mean that's that's the obvious benefit is you can adapt them anymore. I think the other thing you really touch on is the speed, right? To make a custom napkin folding with the exact ply, whatever.

你必须制造机器。你不能只是从亚马逊买这个。是的。但这个可以做得很快，对吧？

You have to make the machine. You can't just Amazon this. Yeah. But this can do it pretty fast, right?

是的。这个可以做得非常快。现在它实际上已经达到人类水平。

Yeah. This can do it very fast. And now it's actually it's at human parody.

是的。

Yeah.

你知道，这个机器人不会累，不需要，你知道，你可以，我认为我真的很喜欢提到的一致性，对吧？比如 Dent Tyon 的一个问题是，你知道他们的洛杉矶分店、纽约分店、西雅图分店，他们折叠餐巾的方式不同，但从他们的角度来看，他们想要一致性，但你知道这正是这些模型擅长的，因为模型只是一组权重，你可以把权重放在不同的机器人上，也可以 OTA 更新它们。是的，你可以一次性更新所有机器人。一旦它工作，我认为这真的很不可思议。

You know, and this robot doesn't get tired, doesn't need to uh, you know, you can and I think I really like the fact that you mentioned consistency, right? Like one issue with Dent Tyon is like you know their LA location with New York location Seattle location they the full napkin study differently but from their p perspective they want consistency but you know that's exactly what these models are good at because models are you know just it's a set of weights you can just like put the weights in the different robots across the can ota update them as well. Yeah, you can just update all of them at once, you know. It just once it works, I think it's truly like incredible.

最终 Boss 是 Cheesecake Factory。如果你知道那里的故事，他们是从零开始。一切都非常一致，精确到克。每种酱汁的制作，无论你在世界哪个地方，你都会得到完全相同的东西。如果你不知道，你就不知道。但你知道，那就是乐趣所在。

Final boss is Cheesecake Factory. If you know the story there, they're there from scratch. Everything is very consistent down to the measured gram. Every sauce is made like anywhere you go in the world, you'll have the exact same thing. If you don't know, you don't know. But, you know, that's the that's the fun.

总之，我想到的术语有点像机器人学的苦涩教训，对吧？与其使用专用机器人，那是大量的工业工程，你构建一个通用平台，可以进行最后的微调等等。在你的专业领域之外，什么是更苦涩的教训？

Anyway, the term that comes to mind is like sort of like bitter lesson for robotics, right? uh instead of the specialized robots which is a lot of uh industrial engineering you build a generalized platform that can uh do a last malineet tuning and and do all those things. Give the expert add-on outside of your what is what is more bitter bitter less in robotics?

是的，我认为机器人学有几个苦涩的教训。一是机器人学将像机器学习解决的其他领域一样扩展。对吧？那么这意味着什么？这意味着拥有尽可能多的数据。但在机器人学中，更难的一件事是，什么是正确的数据比其他领域更难弄清楚。对吧？在语言中是文本。在视频游戏中，只是视频游戏数据，但对于机器人学，你选择什么传感器决定了你有什么样的数据，对吧？但如果你有正确的传感器组，并只是扩展数据，我认为你会得到可预测的模型性能回报。所以在这方面，机器人学并没有那么不同。

Yeah, I think there's several bitter lessons for robotics. One is that robotics is going to scale just like any other domains that machine learning has solved. Right? So what does that mean? That means have as much data as possible. But in robotics, the one thing that's a lot harder is like what is the right data is much harder to figure out than other domains. Right? In language is text. Uh in video games, just video game data, but for robotics like what sensors you choose determine what kind of data you have, right? But if you have the right set of sensors and just scale that data, I think you'll get predictable return on model performance. So in that way, robotics isn't so different.

这就是苦涩的教训。其中一个核心点是，机器人领域有很多研究在提出非常具体的先验——比如我们需要让机器人按物体本来的样子去看，画一些框，这样能让机器人有更好的感知。但我认为这里的苦涩教训是，你只要让算法更简单，并扩展正确类型的数据，泛化和更好的行为就会涌现。所以这并没有那么不同。第二点，我认为这是主要的苦涩教训，也就是归根结底 Scaling（规模扩张）会奏效，但在机器人领域，要扩展以及如何扩展就难得多，因为没有互联网。你必须自己搞定硬件，搞定传感器，把这些都做对，然后才能扩展。

So that's the bitter lesson. A core part of that is that there's a lot of research in robotics that's coming up with very specific priors—for example, we need to make the robot see objects as they are, draw some boxes that will give the robot better perception. But I think the bitter lesson here is that you just keep the algorithm simpler and scale the right kind of data, then generalization and better behavior will emerge. So that's not so different. And the second, I think that's the main bitter lesson, which is that at the end of the day scaling will work, but in robotics to scale and how to scale is just much harder because there is no internet. You have to come up with your hardware, come up with the sensors, get all of those right, and then scale.

我的意思是，有一种论点认为，世界是为人而造的，所以机器人应该就是人形，对吧？不管这硬件问题有多难，想想 10 年、20 年、100 年、200 年后，我们能不能造出一个真正高效的人？但你必须对现在拥有的东西务实，对吧？

I mean, there's the argument that you know the world is very made for humans, it should be a human right? As hard of a hardware problem that is, think 10, 20, 100, 200 years down the line, can we make a really efficient human? But you have to be practical with what we have now, right?

是的，是的。所以我的思考方式是，是的，最终机器人应该——如果你想造一个通用机器人，对吧？就像一台能适应尽可能多使用场景的机器。我认为人形形态会是最好的，因为那是我们做了很久的事情，对吧？但我认为真正造出人形机器人的最佳方式是采取迭代的方法，因为机器人太难了，整个问题、整个技术栈太长了，很难先验地知道什么是正确的决策，对吧？而且因为硬件决策要昂贵得多、代价大得多，因为硬件迭代慢得多，你实际上想要采取渐进的、迭代的方法，而不是一路做到最终形态，因为你造出来的东西很可能撑不住，不够好。所以你最终还是会把它推翻重来。你知道，就像公开新闻说的，Optimus 已经经历了很多很多次迭代，他们不得不推翻自己的设计，对吧？我认为通过用半人形形态迭代，并真正把它们交给真实客户，或者让模型真正部署出去，你就能得到那种反馈循环，而如果你只是试图造一个仓库人形机器人，花五到十年去造，你就得不到这种反馈。我认为迭代的方法实际上会让我们更快地接近最终的机器人。

Yeah. Yeah. So the way I think about it is like, yes, eventually the robot should—if you want to build a general purpose robot, right? Like a machine that can be adapted to as many use cases as possible. I think human form factor will be the best one just because that's what we have been doing for ages, right? But I think the best way to actually build a human robot is to take an iterative approach because robotics is so hard, the whole problem, the whole stack is so long, it's really hard to know a priori what's the right decision to make, right? And because hardware decision is much more expensive, much more costly, just because hardware iteration is a lot slower, you actually want to take an incremental iterative approach, not going all the way to the final form factor because the chances that the thing you build actually doesn't last, it's not good enough. So you just end up scratching it anyway. You know, like it's public news like you know Optimus you know already gone through many many iterations and they had to scratch their designs, right? And I think by iterating with a semi-human form factor and actually putting them out to real customers or getting models really deployed, you get that feedback loop that you wouldn't get if you're just trying to build like you know warehouse humanoid and you spend like five, 10 years building it. I think the iterative approach will actually get us closer to the final robot faster.

我要说，你知道，对于在听而不是在看的人，这可能是我们见过的最平滑的机器人。就像你见过的每个人形机器人都很卡顿，甚至人类走路都没有那么平滑——

And I will say, you know, for those listening and not watching, it's probably the smoothest robot we've seen. Like every humanoid you've seen is jerky, and even humans don't walk as smooth as—

因为模型可以同时移动每一个电机。

Because the model can move every single motor at once.

是的。不止两个关节一起动。是的。

Yeah. More than two cohort. Yeah.

是的，是的。这实际上是一个非常难的问题，因为如果你想想模型，你知道，它输出动作，有一个动作维度，对吧？维度越大，模型就越难拟合数据。但这里的动作空间相当大，不过我们做得很好，确保模型拟合得好，并给你一个平滑的——

Yeah. Yeah. And that's actually a very hard problem because if you just think about like the model, you know, it's outputting action, there's an action dimension, right? The bigger the dimension is, the harder to fit the model to the data. But here the action space is quite large, but we did a very good job to make sure the model fits well and gives you a smooth—

你能给出动作空间的数量级范围吗？还是说——

Can you bound the order of magnitude of the action space? Or is that—

是的，所以它大约在 100 个动作维度的量级。

Yeah, so it's in the orders of like, you know, about 100 action dimensions.

是的，这些都是连续的，对吧？

Yeah, and these are all continuous, right?

连续的，对吧？所以模型基本上是以非常高的频率预测，你知道，像几十——至少 50 维的向量，每秒 10 到 20 次，来获得那种连续运动。

Continuous, right? So the model is just basically predicting at very high frequency, you know, like tens—like at least 50-dimensional vector like 10, 20 times a second to get that continuous motion.

而且可能预测未来一到两秒。所以那又是 100 帧左右。

And potentially predicting one to two seconds out. So that is another 100 frames or whatever.

有一点历史信息，类似于之前状态的上下文，我猜。

There's a bit of a history, similar to context of previous state, I guess.

所以上下文在机器人中非常重要，对吧？因为否则如果你只是基于当前观察做决策，有时真的很难判断下一步该做什么，对吧？例如，如果你想想打开微波炉，在中间部分，不清楚你是在关还是在试图打开它，对吧？所以上下文也非常重要。

So context is a very important thing in robotics, right? Because otherwise if you're just making decision based on the current observation, sometimes it's really hard to tell like what to do next, right? For example, like if you think about opening a microwave, like in the middle part it's unclear whether or not you're closing or trying to open it, right? So like context is also really important.

是的，我有一张冰箱演示的截图，里面有智能体式规划和工具调用等等，我们可以给听众展示。

Yeah, and I have a screenshot of the fridge demo where there's agentic planning and tool calling and all those things, so we can show that for listeners.

我认为那是模型之后一个有趣的侧面。

I think that's an interesting side after the model.

你怎么把它包装成一个智能体式框架？比如什么是——

How do you wrap it around into an agentic harness? Like what is—

内核部署、时钟 UI，你有很多很酷的 UI。但你知道，把模型输出包装成——

Kernel deployment, clock UI, you have a lot of cool UIs. But you know, what goes into wrapping model output towards—

最终的东西是按用例定制的，是你们有自己的 SDK 吗？

The final thing is this custom per use case, is it you have your own SDK?

是的，是的。所以我的意思是，最终我们会有自己的 SDK，但我们的思考方式是，我们想要一个通用的、类似机器人基础模型的框架层。今天那就是我们自己的模型，对吧？我的思考方式是，对于物理世界中任何真实任务、长任务，你必须做很多高层次的决策，对吧？这不仅仅是机器人只对物理世界和动作连续轨迹进行推理，对吧？因为那是非常低层次的，你实际在物理世界中执行的东西，但那不是意图。你知道，当我们做决策时，我们会想一个食谱的步骤，我们会想，哦，我完成了这个，我需要做那个，在等待这部分完成时，我需要去别的地方做别的事。有很多高层次的认知决策在发生。我认为机器人要处理甚至我们向你展示的洗衣任务，它必须具备这种能力，因为你知道，它需要推理出，好的，洗衣机在运行，所以我应该去做别的事，等洗衣机完成，但现在洗衣机完成了，我应该立刻把毛巾从洗衣机里拿出来放进烘干机，对吧？因为如果不这样，那就会成为整个工作流程的瓶颈，对吧？所以机器人需要做很多推理。我认为理论上，你可以只是试图用一个输出动作的低层次模型来学习这个，但那会极其低效。所以这就是为什么拥有一个分层系统，比如在我们的世界动作模型周围放一个类似智能体系统的框架，才能让模型组合不同的能力、不同的技能来解决更长视野的任务。这非常像语言模型，对吧？就像两三年前，模型只擅长自动补全。人类必须点、点、点，才能让模型不断吐出代码。

Yeah, yeah. So I mean eventually we will have our own SDK, but the way we think about it is we want a general like harness layer for robotics foundation model. Today that is our own model, right? The way I think about it is like for any like real task, long task in the physical world, there's a lot of like high-level decision-making you have to do, right? It's not just like robot, you know, only reasoning about the physical world in the action continuous trajectories, right? Because that's like very like low-level, the thing you're actually executing in the physical world, but that's not the intent. You know, like when we are doing decision making, we think about steps of a recipe, we think about oh like I finished this, I need to do that, while I'm waiting for this part to finish I need to go somewhere else to do something else. There's a lot of high-level cognition decision making that's happening. And I think for robots to handle even, you know, the laundry task we are showing you, it has to have that capability because, you know, it needs to reason that okay, the washer is running so I should go do something else while the washer is finishing, but now the washer is done, I should take the towels from the washer and put it in dryer immediately, right? Because if not, then that becomes a bottleneck in the whole workflow, right? So there's a lot of reasoning that needs to happen for robotics. And I think in theory, you could just, you know, try to learn that with a, you know, low-level model that's outputting actions, but it would be extremely inefficient. So that is why having a hierarchical system like putting a harness like an agentic system around our world action model is what allows the model to compose different capabilities, different skills to solve a much longer horizon task. It's very much like in language model, right? Like two years ago, three years ago, the model was only good at autocomplete. Like humans have to like tap tap tap to get the model to keep spitting out code.

但今天，就像智能体系统只是端到端地执行任务，我认为这很重要，因为我认为今天在机器人领域，人们仍然在任务层面思考自主性，比如一个 1 分 30 秒的任务。但真正的价值在于那些能无监督运行数小时的任务或工作流，这正是我们努力的方向，也是为什么我把机器人问题不仅看作建模问题，而是看作系统问题——你需要好的智能体、好的测试框架工程师、好的模型、好的硬件，一切都要好。这就是我们认为通用机器人应该被解决的方式。

But today, just like agentic system just carries the end to end task and that's and I think that's important because I think today in robotics people are still thinking about autonomy at the very task level like a one minute 30 second task. But really the value is about tasks or like workflows that go on like unsupervised for hours and that's what we're building towards and that's why I thinking about the robotics problem not just as a modeling problem but rather as a system problem you need good agent good harness engineer good model good hardware everything it's like how we think general purpose robots ought to be solved

你们是否看到语言模型编码工具变得更好带来的好处？比如新的 Opus 55、新的 Astra。嗯，你们的智能体测试框架中有语言模型吗？语言模型领域正在开发的 AGI 对你们有什么影响？

Do you guys see benefits from language model coding tools getting better? So new Opus 55, new Astra. Um, do you have language models in your agent harness? How does AGI being developed in language model space affect you guys?

不。不。是的。不，绝对有影响。它在很多方面深刻地影响着我们。其一，我们是一家极度 AI 原生的公司，对吧？比如编码智能体，所有这些都是我们工作中非常基础的部分，我们非常欢迎这一点。每当有新模型出现，我们都是第一个测试它的，对吧？比如 GBD6 发布时，我们在同一周举办了全公司黑客松，我们的工程师构建的东西令人难以置信，其中许多功能已经进入主分支并投入生产。所以，我们真的很喜欢用编码智能体来加速我们的工作。

No. No. Yeah. No, absolutely. Like it affects us very deeply in many ways. One is that like we're extremely AI native company, right? Like coding agents, all of that is a very fundamental part of our work and then we're very welcoming of that. Like whenever there's a new model, we're the first one to test it, right? Like we didn't like when GBD6 came out, we had like a companywide hackathon the same week and the thing that our engineers built was incredible and many of them are many of the features we built is already in main in production. Right. So we really like using coding agents to accelerate our work.

呃，只是给感兴趣的人，你在 GPC6 中发现了什么令人惊讶的东西吗？

Uh just for people interested, uh anything surprising in GPC6 that you found?

是的，我认为是它构建虚拟环境的能力，对吧？变成模拟。我们 20 分钟前谈过这个。我觉得这真的很令人兴奋。

Yeah, I think uh it's about the ability for it to like construct virtual environments, right? Turns into simulation. We talked that uh 20 minutes ago. I think that's really exciting.

这很有趣。有很多像 CAD Zillow 到 3D 之类的东西。

It's interesting. There were so many like CAD Zillow to 3Dish.

他在黑客松中输给了一个 CAD。

He lost the hackathon to a CAD.

呃

Uh

我赢了，我做了 CAD

I won I did the CAD

反过来。反过来。不。哦，

other way. Other way. No. Oh,

那是我赢了一个 GPT 黑客松，只是为了好玩。我当时想我完全不懂 CAD，嗯，不妨玩玩，因为每个演示都在做这个。我以为它会很垃圾，结果发现它还挺靠谱的。还挺好的，但只是黑客松的东西。

that was I won a GPT hackathon just for fun. I was like I know nothing CAD. um might as well play around with it cuz every demo is doing it. I'm like it's going to be garbage and I'm like it's kind of grounded. It's kind of good but it's just hackathon stuff.

是的。

Yeah.

嗯，酷。我我我本来打算总结一些用例，然后进入一点公司文化和你个人背景。

Um cool. I I I was going to round out uh some of the use cases and then go into a little bit of company culture and your personal background.

是的。我有两个技术问题。呃，一个是，你有没有为自己设定一些目标，比如 6 个月、1 年、2 年的时间线，你认为机器人应该能够做到填空，你正在努力实现？

Yeah. I have two uh two technical questions. Uh, one is are there any like goals you want to set for yourself like 6 months, 1 year, 2 year timeline that you see robot should be able to do fill in the blank that you're kind of working towards?

是的，我的意思是，在 6 个月到 1 年的框架内，我们的目标肯定是扩大我们的机器人和模型的规模，并真正让它们部署。

Yeah, I mean in the 6 months to one year frame, our goal is definitely like scaling up our robots and our model and truly getting them to deploy.

我的意思是，像现在这样的任务，比如切割或打开冰箱。你有没有给自己设定，到这个时间线，我们应该能够做到什么来保持正轨？

I mean like a task like uh right now it's cutting or opening a fridge. Is there like do you set yourself to by this timeline we should to stay on track be able to do whatever?

是的，我认为在部署方面，我们正在探索许多不同的行业，并没有一个我们认为必须搞定的特定任务，因为我们的目标是构建一个通用模型、一个通用平台，我们把每个不同的任务看作是对我们平台的验证，作为我们弄清楚如何更好地构建一切的切入点。所以今天我们在洗衣店和餐厅做了很多这方面的工作。呃，我们已经部署了数十台机器人，但明年的目标是整体部署数百台机器人。

Yeah I think uh in terms of deployments we're exploring many different industries there isn't a particular task we think we have to nail right because our goal is to build a general model a general platform and we think of each different task as like validation of our platform as a way as a wedge for us to figure out how to build everything better. So today we're doing a lot of that in laundry in restaurants. Uh we have already deployed tens of robots but our goal next year is like overall just deploying hundreds of robots.

呃，我最后一个技术问题的第二部分是，你如何看待行业其他部分？所以给大家填补一下，人们对世界模型采取了哪些其他方法？为什么它们是错的？嗯，它们的方法有什么不同但好的地方？只是这个领域机器人世界模型的格局是什么，以及你为什么如何形成你当前的方法？

Uh my part two on the last technical question is how do you see the rest of the industry? So fill in for people what other approaches are people taking to world models? Why are they wrong? Um what is different but good about their approaches as well? just what is the landscape of robotic world models in this space and why how have you come to your current

是的，这是个好问题，所以我认为机器人是当今最令人兴奋的领域，有很多我真正尊重的了不起的公司，他们像我们一样在做前沿工作，这非常令人兴奋。我认为我们今天看到的是，机器人领域仍然有许多不同的方法，比如我们相信硬件需要内部构建。我们需要拥有整个链条。呃，我们相信扩展人类数据极其重要。但有些公司押注于仅仅扩展机器人数据。我们正在扩展某种中间形式，比如数据收集设备，对吧？所有这些都在发挥作用，我认为这是不同的方法。我们拭目以待。我认为现在还为时过早，但我们对我们的构建方式感觉很好，因为作为一家公司，在文化和使命上，我们不只是为了构建一个模型。我们是为了真正让机器人部署。所以我们尽可能多地部署，解决难题。因此我们更早地发现问题。然后我认为这真的帮助我们确定真正的瓶颈是什么，然后解决它。我认为对于那些只解决问题一部分的人来说，这有点像战争迷雾。你并没有真正看到全貌，你可能在构建技术上极其酷的东西，但可能不是人们最终需要或想要的，对吧？我认为有一整片机器人公司的墓地，我认为它们都有非常闪亮、非常酷的机器人，但你知道，弄清楚人们真正想要什么，说起来容易，比如构建人们想要的东西。你知道每个人都这么说，但实际上执行起来真的很难，机器人公司。

yeah no that's a great question so I think robotics is the most exciting space today and there's just so many uh uh you know amazing companies that I truly respect you know they've been doing great frontier work like us and uh it's very exciting and I think what we are seeing today is that there's still many different approaches to robotics like we believe that hardware needs to be built in-house. We need to own the whole uh chain. Uh we believe that scaling human data is extremely important. But there are companies that's betting on just scaling robots data. We're scaling some sort of intermediate like data collection devices, right? All of them are in play and uh I think it's different approaches. We'll just see what happens. I think right now uh it's still a bit early to tell, but we feel pretty good about the way we're building things just because like we as a company culture-wise, mission wise, we're not here to just build a model. We're here to truly get robots deployed. So we deploy as much as possible, work on the hard problem. So we identify problems much earlier on. So then I think that really helps us identify what is the true bottleneck and then solve it. And I think for uh folks for people who are only working at one part of the problem, it's kind of like fog of war. You don't really see the whole picture and you might be building something that's extremely cool technologically but may not be what people need or want at the end, right? I think there's a whole graveyard of robotics company and I think all of them had very shiny, very cool robots, but uh you know just figuring out what people actually want is actually is very easy to say you know like build something people want. You know everyone says that but it's actually really hard to execute uh botics company

在技术方面。你有你的世界行动模型。这在行业中是标准的吗？人们还在采取其他方向吗？

on the technical side. You have your world action model. Is that standard across industry? Is there other directions people are taking?

是的，我会说那也不是标准。所以我们在世界模型之上构建世界行动模型。有很多公司在构建世界模型。你知道你提到了 Runway，他们也在适应视频世界模型。呃，也有公司在构建视觉语言行动模型。所以这就像把视觉语言模型适应到机器人上。呃，所以我认为有很多不同的方法。但我认为归根结底，我们如何称呼模型并不重要，重要的是我们认为模型在基本原理层面需要能够做什么。所以我想解释一下。我认为对于机器人来说，最重要的是你需要能够预测未来。对。对。机器人，任何物理系统从根本上都是基于你在当前时刻和之前拥有的信息，你应该做什么下一步。所以这真的是关于预测接下来会发生什么。所以预测机器人行动就像预测机器人下一步应该做什么。

Yeah, I would say that's also not standard. So we're building world action model on top of world model. So there's many companies building world model. You know you mentioned runway they're also getting to adapt adapt video world models. Uh there are also companies building uh vision language action model. So that's like taking vision language models and adapting them to robotics. Uh so I think there's many different ways to go about it. But I think at the end of the day like what we call the models is less important than what we like believe the model needs to be able to do at a first principle level. So I want to explain that a bit. I think for robotics the most important thing is that you need to be able to predict the future. Right. Right. Robotics, any physical system is fundamentally about based on the information you have at the present moment and before what should you do next. So it's really about predicting what comes next. So predicting robot action is like predicting what the robot should do next.

但你想想，机器人本身就是世界的一部分，对吧？机器人会影响世界，但世界也会影响机器人在那个环境中应该如何行动。所以如果这样想，你能做的最第一性原理的事情，就是尽可能多地预测这个世界。这就是为什么我们相信预测未来极其重要。今天的实现形式是世界-动作模型，而 VLA 只预测机器人动作，或者某种更抽象的世界表征。但每当你预测抽象的世界表征时，你其实是在把自己关于“什么值得预测”的偏见编码进去。而“苦涩的教训”告诉你：去掉人类偏见，直接拿最原始的表示——摄像头输入——去预测这一切。所以这就是我今天对机器人建模的思考方式。

But if you think about it, a robot is just a part of the world, right? The robot influences the world, but the world also influences how the robot should behave in that environment. So if you think about it this way, then the most first-principles thing you can do is just predict as much of the world as you possibly can. That's why we believe predicting the future is extremely important. Today the instantiation is world-action models, and VLA only predicts robot actions or some more abstract representation of the world. But whenever you're predicting an abstract representation of the world, you're encoding your own bias into what's important to predict. And the bitter lesson tells you: let's remove human biases, let's just take the most raw representation, the camera inputs, and just predict all of that. So that's how I think about modeling for robotics today.

那我想稍微追问一下，你基本上是在说你的方法比 VLA 更好，对吧？因为你在强调预测。我完全买账，我喜欢这个，你说服我了。如果你错了，最可能的第二名会是什么？

So just to press you a little bit on this, you're basically saying your approach is better than VLA, right? Because you're emphasizing prediction. I fully buy that. I like it, you've sold me on it. If you were wrong, what would most likely be number two?

是的。所以我之前说的有一个前提，我认为那正是我可能出错的地方。我说你需要尽可能多地预测世界，对吧？但这并没有说明你的传感器是否拥有你预测世界所需的一切信息。我想我稍微提到过这一点。如果你训练一个纯粹基于视频的世界模型，我认为它会是一个非常强大的世界模型，但它可能不够丰富，因为你会缺失世界的一些关键方面，比如声音，比如触觉。所以我们可能出错的地方是：好吧，我有最好的基于视觉的世界模型，但明天别人做出了一个视觉加触觉的世界模型，数据效率高出 100 倍，因为视觉是一种信息量很高、或者说高度冗余的输入——每隔一个像素变化都不大。你不需要完美视觉也能理解世界，但触觉和其他模态的信息密度可能高得多。所以也许预测这些能让你用少得多的视觉，从而用少得多的算力和少得多的数据达到同样的性能。

Yeah. So there's a caveat in what I said before, and I think that's where I could be wrong. I said that you need to predict the world as much as possible, right? But what that doesn't say is whether your sensor has everything about the world that you need to predict. I think I touched on this a little bit. If you train a world model that's purely based on videos, I think it'll be a very powerful world model, but it may not be a sufficiently rich world model, because you're missing critical aspects of the world like sound, like touch. So where we could be wrong is: okay, I have the best vision-based world model, but tomorrow someone else comes out with a vision-plus-tactile world model that is 100x more data-efficient, just because vision is a very high-information or highly redundant input—every other pixel doesn't change that much. You can understand the world without perfect vision, but tactile and other modalities might be much denser in information. So maybe predicting that would allow you to use much less vision, so you can get to the same performance with much less compute and much less data.

我觉得你会很容易适应。

I think you'll be able to adapt very easily.

不，正是如此。还有数据量层面的问题——

No, exactly. There's also the level of how much data—

对。

Right.

我提这一点的原因是，我认为如果视觉-触觉世界模型，或者视觉-触觉-音频，不管什么世界模型，是最终解决机器人问题的东西，那么机器人公司会比试图转向机器人的视频生成公司做得更好。原因是理解传感其实非常复杂。我们从公司创立之初就一直在做这件事，所以我们知道瓶颈在哪里，知道如何把它们结合起来。但如果你只是试图把视频模型适配到机器人动作上，而如果你错了——如果视频模型不够——你其实并不具备让触觉很好运作的 know-how。但我们有。

So the reason why I bring this up is because I think if a vision-tactile world model, or vision-tactile-audio, whatever world model, is the final thing that solves robotics, then I think robotics companies will be able to do that better than video generation companies trying to pivot to robotics. The reason is that understanding sensing is actually very complicated. We've been building that since the company started, so we know the bottleneck, we know how to combine them. But if you're just trying to adapt a video model to robotics actions, and if you were wrong—if a video model is not enough—you don't really have the know-how to get tactile to work very well. But we have that.

这是个题外问题，但我觉得很快。热量似乎很重要。对你来说，红外只是另一种颜色吗，还是说那是一种——

This is a side question, but I think it's a quick one. Heat seems very important. Is infrared just another color to you, or is that a—

是的，它可以是另一种颜色，另一种输入。

Yeah, it could be another color, another input.

对模型来说。

To the model.

它其实可以用在烹饪场景里。我知道有些公司在用红外来判断食物熟到什么程度，因为这从 RGB 摄像头真的很难看出来。

It could be used in a cooking scenario, actually. I know companies that are using infrared to know how cooked their food is, because that's really hard to tell from an RGB camera.

对。我对你最后一点有个非常幼稚的反驳：也许我们不需要理解硬件。云端编解码器可以直接把它编码进去。不理解也能做很多事。但当然——

Right. My very naive pushback on your last point is: maybe we don't need to understand hardware. Cloud codecs can just, you know, V-code it in. There's a lot you can do without understanding. But of course—

只是它不在分布内。

It's just not in the distribution.

所以我想稍微聊一下你的个人故事或背景。我们都去了宾大。我没有——

So I wanted to bring it a little bit to your personal story or your background. We both went to Penn. I did not—

我今天才知道。

I just found out today.

是的。是的。我不知道，我最大的遗憾是我没有加入 DARPA 挑战赛。好吧。那样的话我可能就成了早期 Waymo 之类的人。你曾在 GRASP，然后你也在 Google 和 Nvidia 工作过。比如说，学术界、Google 和 Nvidia 各自的重点有什么不同？这些地方你都见过，对吧？然后这显然让你说，去他的，我要做自己的事，对吧？你本来可以加入 OpenAI 机器人团队。

Yes. Yes. I don't know, my big regret is I didn't join the DARPA challenge. Okay. Which—and then I could have been like early Waymo or something. You were at GRASP, and then you've also worked at Google and Nvidia. What are the different focuses of, let's say, academia versus Google versus Nvidia? You've seen all these places, right? And then that obviously caused you to say, screw all that, I'm going to do my own thing, right? You could have joined OpenAI robotics.

是的。所以你能不能按时间顺序带我们过一遍，同时也讲讲你从每一段经历中学到的一件事。

Yeah. So can you just take us through chronologically, but also like one thing you learned from each.

是的。按时间顺序，也许先讲讲我最初是怎么进入 AI 的会很有用，对吧？小时候我在日本和中国待过。我从小下围棋，你知道，就是 AlphaGo 那个游戏。所以当 AlphaGo 出现时，我正读高三，我彻底震惊了，因为我下这个棋已经很多很多年了。它太复杂了。我想在 AlphaGo 出现之前，人们都说，好吧，围棋是 AI 永远无法解决的棋类之一。搜索空间比世界上所有原子的总数还大。你怎么可能在里面搜索解？然后强化学习就把它破解了，对吧？我想那就是我最初进入 AI 的方式。我当时只是对能够自我改进的模型、系统和智能体非常感兴趣。所以当我到宾大读博士时，我就特别想做强化学习。我是联合指导——我的两位导师当时都感兴趣——其中一位导师更偏机器人，他在 Berkeley Robotics Lab 做过博士后。所以我就这样进入了机器人领域。在宾大，我们有 GRASP Lab，我认为那是做机器人最好的地方之一。所以我就这样在博士期间开始做机器人强化学习。然后是的，我做了很多研究，思考如何把强化学习和人类数据连接起来，如何以非常严谨的方式做强化学习理论。然后我去了 Nvidia。那时 GBD4 出来了。当时我们非常感兴趣的是解决转笔任务。

Yeah. So chronologically, yeah, maybe it's useful to talk about how I got into AI in the beginning, right? So, as a kid, I spent time in Japan and China. I grew up playing Go, you know, the game—AlphaGo. So when AlphaGo came out, I was a senior in high school and I was mind-blown, because I've been playing this game for many, many years. It's so complicated. I think before AlphaGo came out, people were like, okay, Go is one of the games that AI will never solve. The search space is larger than the total number of atoms in the world. How could you possibly search for solutions in that? And then reinforcement learning just cracked it, right? And I think that was how I got into AI initially. I was just really interested in models, systems, agents that can self-improve. So when I got to Penn as a PhD student, I just really wanted to work on reinforcement learning. So I'm co-advised—both my advisers were interested in—at the time, one of my advisers was more in robotics; he did a postdoc at Berkeley Robotics Lab. So that's how I got into robots. At Penn, you know, we have GRASP Lab, I think one of the best places to do robotics. So that's how I got into doing RL for robotics in my PhD. And then yeah, I did a bunch of research thinking about how to connect reinforcement learning with human data, how to do reinforcement learning theory in a very rigorous way. And then I went to Nvidia. That's when GBD4 came out. At the time we were very interested in just solving the pen-spinning task.

I don't know why. It's the dexterity limit, right? I mean, you can solve Rubik's cubes, and then... At the time I was working with Jim Fan, who you guys probably also know. We were just like, okay, let's try to do dexterous pen spinning. The initial idea was learning it from human videos. Right, we were trying to watch humans do it, but then I realized, okay, today we don't have a dexterous hand that can spin a pencil at all. So how would I even be able to evaluate the algorithm, even if learning from human worked at all? So I said, okay, we need to first try this in simulation. Then as soon as I tried to set up a simulated environment for a robot hand with a pen, I immediately saw what the issue is. The issue is, if I had to come up with this simulation environment, the reward function for reinforcement learning to solve pen spinning, that's impossible. I wouldn't be able to come up with all the terms that you need to get RL to learn the right behavior. And then I was just thinking, what if a model that can write code can do that? That's what became the Eureka project, right? We basically had, in my opinion, the first recursive self-improving robotic agent in simulation, because we used that to solve a new task that no one has demonstrated before. That allowed me to see the potential of using language models, using coding models, to really accelerate robotics.

So then I went to Google DeepMind, because at the time, even today, they have one of the frontier models in Gemini, right? So I was looking at how to use Gemini for robotics. But eventually I realized, I had a lot of fun doing academic research, but there was always a question in the back of my mind: does any of this matter? When you're working on this kind of thing in the lab, you ask me how we come up with so many interesting fun tasks—you're basically doing your own eval, you're grading your own homework. I wanted something more real. I wanted something that could actually be used by millions of people in the world. And I felt 2024 was when we started Dina, like end of 2024, that was the right time that we can take some of these technologies to actually putting them out in the real world. So that's why I started Dino with Lianon in New York, and fast forward two years, we built our platform, our model, our robots are in Tinton. Go check it out.

Go eat. Yeah, go eat.

It's been a journey. But what still gets me really excited is some of the ideas that I temporarily abandoned when we started Dynam, because to build a company you have to be very practical. I think simulation wasn't ready in 2024, but now with the new frontier models and what we're seeing internally, this might change, and that's very exciting.

Yeah. And so then also let's talk about company culture. What principles are you building the company with, obviously since you're responsible for the science and the research agenda? It just seems like when we meet the engineers, they're very engaged and they have a lot of ownership. They have a lot of motivation.

I'll say it's interesting to see how many side projects people just work on. So some of the demos we've seen, it's just the engineers that work on it were like, yeah, this seems fun, might as well do it.

To your point on fun evals, right here, like two feet behind the camera.

Can't talk about that.

No, I mean the thing is, the eval is just there's highlighters, highlighters being rubber banded together, right? It's like who comes up with, ah yes, we need to test putting rubber bands on highlighters. It's just stationary.

Fun stuff. People are very creative here. It's a beautiful space.

Yeah. I think culture. So let's first talk about how we run a company, right? We think of ourselves as eventually a research plus product company. It's kind of like OpenAI, where you do frontier research but eventually all of them make their way into Codex to ChatGPT. That's how we think about it too. We're not here to just do open-ended whatever thing that interests us at the time. We're always product driven. We want to deploy our robots, our platform, to a new customer. What are the technical problems we have to solve? We have to solve generalization. We have to solve whole body control. We have to solve how to adapt quickly. We have to solve how to make the robots follow language. We have to solve dexterity. So once all those problems are very clear, then our engineers are very capable, they're very smart. They just create instances of the task. They create tasks that would stress test those capabilities. Like the grabbing and drink task is just how we test language following. You can just specify what you want. If the model follows intent, follows instruction, you should be able to do it. So that's how we think about it: at the top level we identify what are the things we have to solve. Then we have a lot of bottoms-up projects to actually drive execution to solve them. And you hire the right people, and everyone here is super passionate about robots. They've been doing this for a long time, they love it, and we just create a culture where it just naturally happens. It's a really fun place to work.

What's the split between the technical research software versus the hardware side of the company?

Yeah, we recently grew so much. So I don't have the perfect number, but I would say on the hardware side we have probably like 30% to 40% of hardware, just because hardware you need a lot more people. And then we have around 10 people working on models, then we have software engineers, robotics engineers, we have a lot of very talented people on the business side. The most fun thing about building a robotics company is you see everything—everything under the sun that could possibly be helpful will be helpful, right? So you just get so much exposure working with so many different people from many different backgrounds who are all very passionate about robots. I think it's just been really eye-opening for me as a founder, because when I was a researcher at a lab or a PhD student, I was talking to researchers every day, and that was great. That was really good. I still get to do that every day. But now I also get to learn just from so many different people from different walks of life. Like some of our people who collect data, they had no robotics experience.

他们之前做零售，做其他工作，然后来到这里。他们在机器人技术方面也训练有素。我觉得这非常了不起。我认为自己仍在学习。我认为自己在机器人领域一直在学习，而且我从未像现在这样学到这么多。尽管你现在看我，我已经不是学生了。我在这里经营一家公司，管理很多事情，但我仍然觉得每天的学习量非常大。

They were working retail, working in some other jobs, and they came here. They also became very well trained in robotics. I think that's just pretty amazing to see. I think of myself as still learning. I think of myself as always learning in the robotic space, and I haven't learned as much as I am now. Even though you think about me now, I'm not a student anymore. I'm here running a company, managing a lot of things, but I still feel like the day-to-day learning is just so much.

最后，Dina 未来会怎样？两年、五年，愿景是什么？

And then just to close off, where is Dina going? Two years, five years, what's the vision?

是的，我认为我们的愿景是真正——高层次的使命是真正将机器人和智能带入物理世界。这实际上意味着看到我们的机器人部署在数千个客户中，甚至更多，数百万台机器人。这就是我们想要达到的目标。

Yeah, I think our vision is to truly—the high-level mission is truly to bring robots and intelligence right into the physical world. That actually means seeing our robots deployed in thousands of customers, even more, millions of robots. That's what we want to get to.

Dina Dynamic 这个名字是怎么来的？有什么故事吗？

What's the name from Dina Dynamic? Is there a story?

是的，有很多故事。我命名了 Dina。内部有很多不同的选项，但这个名字的由来有两个原因。一是图灵奖得主 Richard Sutton，他是强化学习的先驱，很多年前写了一篇论文叫 Dyna。它基本上是关于一个在世界模型中自我改进的智能体。即使在我们创办公司时，我们还没有做世界模型，但我觉得机器人技术会这样解决。现在我们的技术方向正朝着那个方向发展。所以我觉得那是一个很好的灵感，也因为我在强化学习领域工作过，我非常尊重 Richard。第二点更个人一些。我在日本长大，我出生的那一年有一部卡通节目。不知道你们有没有看过《奥特曼》。

Yeah, there are many stories. I named Dina. There were a lot of internal different options, but how the name came to be was there are two things. One is that Turing Award winner Richard Sutton, who's like the pioneer of reinforcement learning, wrote a paper called Dyna many many years ago. It's about basically an agent that self-improves in a world model. And I felt even when we started the company, like we weren't doing world models yet, but I felt that's how robotics will get solved. And now our technology direction is converging to that. So I felt that was a really neat inspiration, also because I worked in reinforcement learning, I have a lot of respect for Richard. The second thing is that it's a bit more personal. I grew up in Japan, and the year I was born there was a cartoon show. I don't know if you guys have seen Ultraman.

当然看过。

Yeah, of course.

是的。那一年的奥特曼叫 Dina。

Yeah. The Ultraman that year was called Dina.

哦，是的。

Oh, yeah.

所以，是的，这是一个非常微妙的事情。不是很多人知道，但我想现在大家都会知道了。

So, yeah, it's a very subtle thing. Not many people know, but I guess now everyone will.

是的，团队也会发现的。

Yeah, the team is finding out too.

是的。但我想 Richard 会欣赏的。我不知道你有没有和他谈过，但

Yeah. But I think Richard would appreciate. I don't know if you ever talked to him, but

没有，我在强化学习会议上见过他。

No, I've seen him at RL conferences.

你应该就说，“嘿，这家公司”

You should just be like, "Hey, this company"

是的，我会邀请他。来找你。

Yeah, I'll invite him. Come to you.

大家都知道 Richard 是因为苦涩的教训，但你知道。是的。也是苦涩或更少。

Everyone knows Richard for bitter lesson, but you know. Yeah. Also bitter or less.

是的。

Yeah.

好的。有什么临别感想或行动号召吗？

Okay. Any parting thoughts or calls to action?

行动号召。是的。我觉得作为一个行业，机器人技术仍然非常注重演示。就像我过去在实验室里做过很多演示。即使在这里，我们也必须展示一些演示。我认为我们需要更多真正以使命为重的人，不惜一切代价让机器人部署，让机器人基础模型真正与硬件集成，并在真实客户现场看到它们。我认为我们在这方面是先锋，我希望在这个领域看到更多。对于观看的观众，如果这个使命激励了你，如果你知道解决当今最难的问题是你每天醒来的动力，并且你想做出贡献，你知道我们一直在招聘，是的，期待你的加入。但我想只是

Calls to action. Yeah. I think I do feel like as an industry, robotics is still very demo oriented. Like I have done a lot of demos in the past in my lab. Even here, we had to show some demos. I think we need more people that are truly mission focused on doing whatever it takes to get robots deployed, get robotic foundation models truly integrated with hardware and seeing them at real customer sites. I think we are a pioneer in this regard and I want to see more in this field. And for the audiences watching, if this mission inspires you, if you know solving the hardest problems today is what makes you wake up every day and you want to contribute, you know we're always hiring and yeah looking forward to that. But I think just

湾区还是其他地方？

Bay Area anywhere else?

呃，是的，目前只在湾区，但你知道随着公司发展，永远不要对其他地点说永不。是的，但这就是我想说的，我希望看到世界上有更多机器人。

Uh yes, just Bay Area today, but you know as company grows never say never to other locations. Yeah, but that's what I want to say and I wanted to see more robots in the world.

是的。酷。恭喜一切。我是说这是两年内不可思议的进步。

Yeah. Cool. Congrats on everything. I mean this is incredible progress for two years.

是的，太棒了。

Yeah, amazing.

是的，我真的很感激。谢谢你拜访我们。我希望你知道这很有趣，而且

Yeah, I really appreciate it. Thank you for visiting us. I hope you know it's been fun and

是的，我会说，是的，我不后悔做这次现场，就像好的。

Yeah, I would say like yeah I don't regret doing this onsite like Okay.

是的，我们在工作室做了很多。

Yeah, we do a lot of in our studio.

是的。就像这次是值得的。

Yeah. Like this was worth it.

是的。是的。我认为对于机器人技术，必须在现场做。它只是

Yeah. Yeah. I think for robotics got to do it on site. It just

对于收听的人，你知道，观看视频版本，我们有有趣的 B-roll。

For people listening in, you know, watch the video version and we have fun B-roll.

这是 Dina 2.1，我们全新的硬件，围绕世界动作模型 Dino 2 的物理智能体框架。所以 Dina 2.1 能做的是在真实的物理行业中自动化整个端到端工作流程。所以这里我们展示的是在酒店真实洗衣房中的端到端洗衣任务。在我身后这些是项目的研究人员。他们将向你们展示一些非常酷的东西，一个快速演示，并解释一些技术细节，是什么让这成为可能。所以，我把时间交给 George。

This is Dina 2.1, our brand new hardware with a physical agentic harness around the world action model Dino 2. So what Dina 2.1 can do is to automate an entire end-to-end workflow in a real physical industry. So here what we're demonstrating is an end-to-end laundry task in a real laundry room in a hotel. And behind me these are the researchers for the project. They're going to show you something really cool, a quick demo, and going to explain some of the technical details behind what made this possible. So, I'll pass it to George.

好的。我将展示整个洗衣工作流程中的一些快速步骤。先启动机器人。这将是把毛巾从洗衣机转移到烘干机的任务。让我们看看。是的。所以，这实际上是一个非常困难的任务，因为你看到机器人必须真正操纵它的身体，把手臂伸进去抓住所有毛巾，而不与门碰撞，对吧？但你在这里看到，它实际上必须非常灵巧，动作非常流畅，因为它是一个单一模型控制身体的每一个关节。所以没有像经典规划与行动分离那样使动作不那么流畅，

All right. I'm just going to show some quick steps in the entire laundry workflow. Just going to start the robot. This will be the task of transferring towels from the washer into the dryer. Let's take a look. Yeah. So, this is actually an extremely hard task because you saw the robot has to really maneuver its body to stick the arms in to grab all the towels without colliding with the doors, right? But you see right here, it actually has to be very dextrous and the motion is very smooth because it's a single model that's controlling every single joint in the body. So there isn't like a classical planning versus acting separation that makes the motion less smooth,

对吧？

Right?

好的，酷。我们也可以——你想模拟一下吗？好的，我们也可以模拟机器人意识到毛巾掉落。所以我就在这里放下我的毛巾。我们将看到它如何能够看到毛巾掉落然后捡起来。所以当它把毛巾放进烘干机后，我们会看到，它也意识到这条毛巾现在脏了。所以它去把毛巾放进脏篮子。

Okay, cool. We can also—Do you want to simulate it? Okay, we can also simulate the robot realizing a dropped towel. So I'm just going to drop my towel here. We'll see how it is able to see that the towel is dropped and then pick it up. So after it's done putting the towel in the dryer, we'll see and it also realizes that this towel is now dirty. So it goes and puts the towel in the dirty basket.

我认为这是一个完美的案例，展示为什么我们需要构建这样的全身机器人，对吧？所以它可以访问工作空间各个角落的不同部分，它可以做人类工人能做的一切。接下来，它将关闭烘干机门并打开。他们还可以根据触觉来推理应该设置什么烘干模式。

I think this is a perfect case to showcase why we need to build a whole body robot like this, right? So it can access different parts all of the corners of the workspace that it can do everything that a human worker can do. Next, it will close the dryer door and turn it on. And they can also reason based on the tactile in terms of what drying mode to put in.

是的。对我来说最令人印象深刻的部分实际上是转动旋钮这部分，你知道，就像真正像人类一样操作机器。是的，也许值得拍个特写，对吧？并确保它处于正确的模式，然后按下按钮启动烘干机。对吧？

Yeah. The most impressive part to me is actually this part of turning the knob, you know, like actually operating a machine like a human would. And yeah, maybe it's worth getting a close shot, right? and making sure it's on the right mode and pressing a button to start the dryer. Right?

同样重要的是要意识到这是数字帧。所以，你真的需要更多上下文来意识到，因为当你捕捉图像时，LED 屏幕也会闪烁。所以，这就是为什么重要的是要意识到机器人知道如何启动，并意识到烘干机何时启动。

It's also important to realize that this is digital frames. So, you really need more context to realize because when you capture an image, the LED screen also blinks. So, that's why it's important to realize that the robot knows how to start and realizing when the dryer has started.

所以，这挺酷的。

So, it's pretty cool.

是的。所以，这只是整个洗衣工作流的一部分。周二，我们会发布完整的工作流。整个工作流会花相当长的时间，你可以想象。

Yeah. So, this is just one part of the whole laundry workflow. On Tuesday, we're going to drop the entire workflow. The whole workflow will take quite some time as you can imagine.

是的，折叠、堆叠、转移、篮子，所有事情，对吧？

Yeah, folding, stacking, transferring, basket, everything, right?

真的，Dina 2.1 就像机器人第一次成为工作场所的完整员工，做所有事情。不需要人类或任何其他机器来喂它物品或帮助它完成任务的某些步骤。它只是会接受整个任务并完成。我认为这是一个重大突破，因为历史上我们把机器人视为机器，对吧？但这是第一次，我认为我们真的可以看到像与人类一起工作，真正提高和增加生产力，世界上的物理生产力。是的。所以我们对此非常兴奋。

Really like Dina 2.1 is about like a robot for the first time becoming like a whole employee at a workplace doing everything. There's no need for human or for any other machine to feed it items or help it with certain step of the task. It's just going to take the whole thing and do it. And I think that's a big breakthrough because historically we think about robots as machines, right? But this is for the first time like I think we can really see like working alongside humans to really improve and increase productivity, physical productivity in the world. Yeah. So we are super excited about it.

是的。请继续。

Yeah. Go ahead.

这将是最长的本地灵巧操作演示。所以对我们所做的工作感到非常自豪，是的，令人兴奋。

It will be the longest local dextrous manipulation demo. So quite proud of the work that we have done and yeah exciting.

是的。所以这里还有另一个我们使用 Dynet 2.1 做的演示。

Yeah. So here's another demo we put using Dynet 2.1.

所以这里，带有我们硬件的物理智能体将根据你在这个屏幕上选择的饮料为你拿一杯饮料。所以我们的研究人员，我和 Chad 负责这个项目，我们只是给你一个快速演示。所以在这种情况下，他们选择了香草奶昔饮料，机器人已经收到了命令。所以，它会去冰箱，然后现在使用世界动作模型 Dino 2 控制整个机器人为你拿到正确的饮料，然后取回它。这里真正有趣的是，这真的是为了展示智能体的能力。所以，模型不仅仅是在做动作，它也非常忠实地遵循语言命令，并且还可以导航回你身边来完成订单。所以，好的。所以，拿到了正确的饮料。它会很好地关上门，我认为它正在切换切换。

So here the physical agent with our hardware is going to grab a drink for you based on whatever drink you select on this screen. So our researchers, me and Chad will let the project, we just give you a quick demo. So in this case they selected vanilla shake drink and the robot has received a command. So, it's going to go to the fridge and then now use the world action model Dino 2 that's controlling the whole robot to get the right drink for you and then retrieve it. And what's really interesting here is this is really to showcase the agent capability. So, the model is not just like doing actions, it's also following language commands very faithfully and can also navigate back to you to fulfill the order. So, all right. So, got the right drink. It's going to nicely close the door and I think it's switching switching.

哦，是的。门。

Oh, yes. Gates.

是的。切换模式，因为这里是导航部分，我们可以让另一个高级推理模型来做，对吧？

Yeah. Switching mode because here is the navigation part which we can just have another high-level reasoning model to do, right?

是的。是的。你们想对正在发生的事情补充什么吗？有什么酷的东西想指出来吗？

Yeah. Yeah. Anything you guys want to add on what's happening? Anything cool you want to point out?

是的。所以，这个有一个推理层和一个动作层。所以，这是一个双模型架构。在底层，我们向推理模型发送语言提示。基本上，是几句指令。然后推理模型会规划它需要一步步做什么。首先进入冰箱拿东西，然后回来，然后按顺序交给人类，并监督整个执行过程。而较低级别的动作模型负责执行操作和灵巧任务。

Yeah. So, this one has a reasoning layer and an action layer. So, it's a two model architecture. Underneath we are sending language prompts. Basically, a few sentence instruction to the reasoning model. And then the reasoning model is planning out what it needs to step by step. First go into the fridge and grab something and then come back and then hand it off to a human in sequence and oversee the whole execution. And the lower level action model is responsible for performing the manipulation and the dextrous task.

是的。在动作模型方面，它也需要是可操控的。所以你要告诉它你想拿起六种饮料中的哪一种。然后我们的视频动作模型有助于想象，比如理解你的意图。所以你可以想象这个机器人抓取饮料时是什么样子。这有助于它真正遵循你告诉它做的事情。然后还有一些时机方面的因素。它需要识别你何时抓取了饮料，然后在那个确切的时间点释放。你不想要误报。你不想要把饮料掉在地上。所以，要让这样的演示工作，有各种各样的小事情你必须小心。

Yeah. On the action model side, it also needs to be steerable. So you're telling it which of the six drinks you want to pick up. And then like our video action model sort of helps with imagining like understanding your intent. So you can imagine like what does it look like when this robot grabs a drink. And that sort of helps it really follow what you tell it to do. And then there's also some like timing aspects to this. It needs to recognize when you've grabbed the drink and then like release at exactly that point. And you don't want false positives. You don't want to drop a drink on the ground. So, there's all sorts of like little things you just have to be careful about to get a demo like this working.

是的。而且模型在这种灵巧性方面相当不错，非常稳健。所以，就像 Chad 说的，它能够识别并想象如果它抓取正确的饮料会是什么样子，然后执行。

Yeah. And the model is quite good at this kind of dexterity and very robust. So, like Chad was saying, it's able to just recognize and kind of like imagine what it's going to look like if it grabs the right drink and then execute.