物理 AI 如何跨越语言、视频与动作学习

这辆车即将左转。你能想象这辆车的轨迹和生成的视频吗？

This car is about to make a left turn. Can you envision the trajectory the car is taking and the video generated?

你猜怎么着？那条路从未被拍摄过。它是由 Nvidia 的 Cosmos 3 生成的。它读取视频，模拟道路，并学习如何处理物体。Ming-Yu Liu 领导这项研究，这是与 Nvidia 的付费合作。

Guess what? That road was never filmed. It was made by Nvidia's Cosmos 3. It reads video, it simulates roads, and it's learning how to handle objects. Ming-Yu Liu leads the research, and this is a paid partnership with Nvidia.

现在，Cosmos 接受文本、视频、音频和动作作为输入。让我们从最简单的案例开始。

Now, Cosmos takes text, video, audio, and actions as input. Let's start with the simplest case.

你问模型视频中发生了什么。视频中有多少辆车？机器人是否成功完成了视频中的动作。好的，有三辆车，机器人成功捡起苹果并放入篮子。当你以视频和文本为输入并生成文本时，它就变成了一个视觉语言模型。

You ask the model what happened in the videos. How many cars in the video? Whether the robot succeeded in completing an action in the video. Okay, there are three cars, and the robot succeeded in picking up the apple and putting it in the basket. When you have a video and text as input and generate text, it becomes a vision language model.

那么，我们为什么不把它反过来呢？模型停止描述场景，开始生成场景。

So, why don't we flip this thing on its head? The model stops describing the scene and starts generating it.

车辆模拟梦想是一套构建闭环模拟器的技能，不仅仅用于自动驾驶。我可以雇佣一百名司机、一百辆车，去不同的十字路口，弄清楚它是否效果更好，或者我可以使用这个模拟器来验证准确性。你不受车队规模的限制。任何时候你需要进行大量测试，你有计算机，只需启动它。

The vehicle simulation dreams is a set of skills to build closed-loop simulator, not just for self-driving. I can employ a hundred drivers, hundred cars, go to different intersections, figure out whether it works better, or I can use this simulator to verify the accuracy. You are not limited by the size of your fleet. Anytime you need to do a lot of testing, you have the computer, just launch it.

好的，那么如果我们采用同样的想法，但用于更困难的问题呢？比如机器人。

Okay, so what if we took the same idea, but on an even harder problem? Like robots, for example.

你知道，导航时，你不希望你的物理设备触碰任何其他场景，但操作任务需要各种交互。当有交互时，会有缓冲。当物体被操作时，可能会有变形。所以，这更具挑战性。我们将达到一个状态，能够使用神经模拟来很好地处理这种复杂的操作，对吧？一旦我们达到那个状态，你就可以用墙壁模拟来测试你的机器人策略。

You know, navigation, you don't want your physical device to touch any other scenes, but manipulation tasks require all kinds of interactions. When there's an interaction, there's a cushion. There's a potential deformation, you know, when things are manipulated. So, it's more challenging. We will reach a state where we will be able to use neural simulation to handle this complex manipulation well, right? Once we reach the status, you can test your robot policy with the wall simulated.

好的，所以一个模型，Cosmos 3，处理三项工作。在机器人中处理物体是未来。这非常令人兴奋。

Okay, so one model, Cosmos 3, handling three jobs. Handling objects in robotics is the future. It's very exciting.

所以，你可以看到，你正在看一个非常多样、多功能的模型架构。

So, you can see that you are looking at a very diverse, versatile, model architecture.

现在我们将动作视为一等公民，这意味着模型可能实际上采取一个动作，然后它的下一个观察可能由它之前做的事情引起。

And we now have action as a first-class citizen, which means the model might actually take an action and then its next observation could be caused by the thing that it did before.

我们从一个语言模型开始。我们要么自己从头训练，要么采用一个现有的开放模型，对吧？从语言模型开始。它对语言结构有基本的理解，理解语言。然后，我们有一个视觉编码器，可以接受视频或一组图像作为输入。然后，我们将这个视觉编码器连接到大型语言模型。将视觉编码器连接到语言模型，我们试图构建一个视觉语言模型，对吧？有时你也可以从一个预训练的视觉语言模型开始，好吗？现在，这个语言模型以自回归的方式运行，一次预测一个词，对吧？好的，为了将视觉输出连接到语言空间，它用语言思考。它解释它看到的东西，对吧？所以，你有一个视觉语言模型。好的，现在我们拿这个视觉语言模型，预训练的权重，然后初始化一个生成器，对吧？所以，一个关键点是这个视觉语言模型是自回归的。而我们的生成器实际上是扩散模型，是双向的。现在这个双向生成器塔学习生成视频、动作、音频。好的，在这个塔中，每个词都关注其他所有词。所以，当你生成动作的视频时，它生成的块之间有一致性，每次生成中这个块内的信号是一致的。它还利用来自视觉语言模型部分的词，我们称之为残差塔。所以它也理解指令，你想生成什么。好的，在那个生成器训练中有两个阶段，预训练学习基础，然后我们训练加入动作。

We start with a language model. We either train it from scratch ourselves or take an existing open model, right? Start with the language model. It has a basic understanding of the language structure, understand language. Then, we have a vision encoder that can take a video or a set of images as input. Then, we connect this vision encoder to the large language model. Connecting this vision encoder to the language model, we try to build a vision language model, right? And sometimes you can also start with a pre-trained vision language model, okay? Now, this language model, operating in the auto-regressive manner, predict a token at a time, right? Okay, in order to connect visual output to the language space, it thinks in language. It explains what it sees, right? So, you have a vision language model. Okay, now we take this vision language model, the pre-trained weights and then initialize a generator, right? So, one key thing is that this vision language model is auto-regressive. And our generator is actually diffusion is bidirectional. Now this bidirectional generator tower learn to generate video action audio. Okay, in this tower every token attend to every other token. So, when you generate the video of the action there's a coherence among the chunk it generates among each generation the signal within this chunk is coherent. And it also leverage the token from the vision language model part of the residual tower which we call it the residual tower. And so it's also understand the instruction, what you want to generate. Okay, and in that generator training there are two stages, pre-training learn the basic and we training we add action in.

在你看来，什么是世界模型？

In your estimation, what is a world model?

这是一个非常具有挑战性的问题，我认为之前有人试图给 AGI 下定义。我想几年前他们研究过，但直到今天，对于什么是 AGI 仍然没有共识，对吧？人们有自己的定义。我认为世界模型也会一样。对吧？所以，我认为世界模型是一个有用工具的集合，对吧？我们对某物建模是因为我们试图实现某个目标，对吧？例如，它可以是预测未来，或者理解为什么发生。在机器人学中有三个重要的建模。前向动力学，给定起点和采取的动作，未来会怎样？预测将要发生什么。逆向动力学，给定视觉过渡，推断采取了什么动作。以及策略，机器人应该做什么来实现任务。我们正在结合这些不同的模态，比如音频、动作、视觉等等。它们可能运行在不同的时间尺度上。你如何让它们都运行在相同的时间尺度上？

This is a very challenging question and I think earlier there's something similar people try to give a definition to AGI. I think several years ago they studied but I think until today there's still no kind of common agreement what AGI is, right? People have their own definition. And I think a world model going to be the same. Right? So, I think a world model is a collection of useful tools, right? We model something because we are trying to achieve some goal, right? For example, it could be predict the future or it could be understand why this happened. There are three important modeling in robotics. Forward dynamics, you know, given what the starting point and the action taken, what will be the future? Predict what going to happen. Inverse dynamics, given the visual transition, you know, infer what was the action taken. And policy, what should the robot do to achieve the task. We're combining these different modalities like, you know, audio and action and vision and so on. And they might run on different time scales. How do you make them all run on the same time scale?

不同的信号有不同的频率，对吧？即使是视频，我们也有不同的帧率。高速视频、低速视频。音频也有不同的赫兹，动作也不同。所以我们有一个时间位置嵌入方案，我们几乎将所有信号归一化到相同的轴、相同的尺度。这样，当词代表信号的不同时间块或不同空间时间块时，能更好地知道哪些其他词在同一时间实例，同时也知道不同时间实例之间的距离。是的。所以，这是使这个单一模型能够处理音频、视频和动作这些不同频率的关键部分。

Different signals have different frequency, right? Even for video we have come with a different frame rate. High speed video low speed video. And the audio, there's also different hertz and action also different. So, we have a temporal position embedding scheme where we pretty much normalize all the signal into the same axis the same scale. So, that's when the token represent different time chunk or different spatial time chunk of the signal better know which other token are in the same time instance at the same time instances and also know the distance between different time instances. Yeah. So, that is a critical part to make this only model work, handling audio, video and action all come with the different frequencies.

你之前谈到这些不同的模态——前向动力学、逆向动力学和策略——在一个单一模型中。它们究竟如何相互增强？

You were talking earlier about these different modalities forward dynamics, inverse dynamics and policy in one single model. How exactly do they reinforce each other?

当我们训练模型时，我们把它们放在一起，对吧？我们给出信息瓶颈。你只有这么多容量来展开所有这些。这些都是试图将视频视觉观察连接到动作。所以，在世界动作模型中，你有一个测试指令，你想实现什么，这就是你观察到的。

When we train the model we put them together, right? And we give the information bottleneck. You only have this amount of capacity to expand all them. These are all trying to connect video visual observation to action. So, in the world action model, you have a test instruction what you want to achieve, and this is what you observe.

然后你开始生成动作和可能的输出，对吧？可能的输出类似于前向动力学，对吧？当你输入动作时，输出可能是什么？在动力学中，当你看到这个视觉情境时，动作可能是什么？对吧？这也类似于世界动作模型中发生的情况，对吧？你有视频和动作。它们之间有关联吗？

And you start to generate action and possible output, right? And possible output is similar to forward dynamics, right? When you have action in, what could be the output? In the dynamics, when you have this visual situation, what could be the action? Right? Also similar to what happened in the world action model, right? You have the video and action. Is there correlation between them?

是的，所以我认为这三个从根本上都是在试图捕捉观察和动作之间的关联。对吧？只是规模不同、视角不同。是的。一旦你加入信息瓶颈，加入容量约束，你知道，模型就会学习。我们在论文中的结果表明存在协同效应。一个确实有助于另一个。

Yeah, so I think these three are fundamentally trying to capture the correlation between the observation and action. Right? It's just a different size, different perspective. Yeah. And once you put the information bottleneck, you put the capacity constraint, you know, the model learns. And our result in the paper show that there's a synergy. One does help the other.

我认为可以公平地说，训练数据存在一些不对称性，对吧？所以，有大量人类生成的视频数据，然后动作数据要少得多。但你在模态之间获得了这种良性迁移。但模型是否能够知道何时不应迁移，何时迁移可能有害？

And I think it's fair to say there's a bit of an asymmetry in the training data, right? So, there's lots of human-generated video data, and then there's far less action data. And but you get this virtuous transfer between the modalities. But could the model ever know when not to transfer, when transfer could be harmful?

我认为模型并不真正知道。所以，正如你所说，我们有大量的自我中心视频，对吧？第一人称视角，双手试图完成某些任务。而我们的机器人视频数量非常少，实际上要少得多。你知道，机器人也有头部摄像头、夹爪摄像头，看到它完成某些任务，对吧？所以，我认为这更多是关于我们对不同具身的共享词汇，对吧？所以，你知道，人或机器人看起来像人，对吧？人类手如何操作一个物体，视觉模式、视觉动作关联，人类如何操作物体，你可以看到模式与机器人操作物体非常相似。即使动作空间并不精确相关。一旦你将一组动作关联起来，模式是相似的，就更容易泛化到另一组。而且当你纳入几个不同的具身时，它也有助于你泛化到未见过的具身。因为我的意思是，人或机器人它们看起来会像人。也许有些手更大，有些手更长，但会相似，对吧？所以，夹爪可以不同，但你知道，功能也相似。关键真的是模式，视觉观察和动作之间的关联，对吧？从一个动作具身到另一个具身的转换，我认为真的真的更容易。

I think the model doesn't really know. So, as you are talking about, we have a larger amount of egocentric video, right? The first-person view in the two hands trying to complete certain task. And we have a really small amount and actually much smaller amount of the robot video. You know, robot also have head camera, you know, gripper camera, and you know, see it's complete certain task, right? So, I think it's more about our shared vocabulary for different embodiments, right? So, you know, human or robot looks like a human, right? And how human hand manipulate one object, the visual pattern, the visual action correlation, human how human manipulate an object, the pattern you can see it's very similar to how a robot would manipulate object. Even though the action space doesn't really correlates precisely. Once you correlates one set of action, the pattern is similar, it's easier to generalize to the other. And it also give you when you incorporate a couple different embodiment in, it also help you to generalize to unseen embodiment. Because I mean, human or robot they're going to look like human. Maybe some have a bigger hand, some have a longer hand, but it going to be similar, right? So, and gripper can be different, but you know, the function are also similar. The key is really the pattern, the correlation between the visual observation and action, right? And the translation from one action embodiment to another embodiment I think is really really easier.

显然，失败模式之一是任务没有被足够好地指定。

Apparently, one of the failure modes is when tasks are not well enough specified.

我认为助手二是关键，它能让模糊任务对系统一来说更具体，以便执行。

I think assistant two is the key to make ambiguous tasks more concrete for system one to execute.

是的，这很吸引人，因为我在想我们想将这项技术用于安全关键环境，对吧？而有些任务确实相当模糊。但想必我们可以加入多层，你知道，也许我们并不完全理解如何指定我们正在做的事情，但我们可以加入多层测试和检查，我们可以对其进行红队测试，并让它随时间改进。

Yeah, it's fascinating because I'm thinking that we want to use this technology in safety critical environments, right? And some tasks are just quite ambiguous. But presumably we could add in layers of, you know, maybe we don't understand exactly how to specify the thing that we're doing, but we could add in layers of tests and checking and we could red team it and make it improve over time.

你给出一个任务。它会类似于语言模型智能体，对吧？你给出一个任务。智能体或模型的第一次初始调用会找到需要完成的任务序列，对吧？然后它会逐一执行，每次执行后检查是否完成。在某个时刻可能需要做出选择，对吧？那么，我应该做这个还是那个，对吧？我认为这将要求我们在系统层面更多地思考机器人任务，比模型层面高一层。你有一些框架来收集资源，要么传递记忆，要么传递你可以使用的工具，对吧？所以，我认为这将是完成物理任务更复杂的方式。

You give a task. It will be similar to LM agents, right? You give a task. First initial call of the agents or the model will do find a sequence of task need to be complete, right? And it would execute one by one and after each execution it will check whether this is complete. And at some point there may be choices need to be made, right? So, okay, should I do this or the other, right? I think this will require us to think this robotic task more in the system level, a layer above the model level. You have some harness to kind of gather resources together, either you will pass memory or the tools that you can use, right? So, I think this going to be a more complex way to do get the physical task.

我们还没有充分讨论的一件事是使用 Cosmos 作为模拟器来训练策略。我只是在想这是否会有一些潜在问题，因为你知道在强化学习中我们有模拟到现实的差距，你是否可能遇到策略学会利用模拟器内部特征的情况？

One thing we've not spoken about enough is using Cosmos as a simulator essentially to train a policy. And I'm just thinking whether there would be some potential problems with that cuz you know in reinforcement learning we have the sim-to-real gap and could you have a situation where the policy might learn to exploit features inside the simulator?

是的，有可能。我看待这个神经模拟器进展的方式是，我认为首先它在初期会更有用于被动验证。被动验证是一项任务，你知道，如果你是一个模型构建者，任何模型运行你都会有很多检查点。你可能会做消融研究，每个变体你都会有很多检查点。你怎么知道哪个更好，对吧？理想情况是将每个策略部署到真实机器人、真实环境中，并测量完成率，对吧？成功率。你知道，自动驾驶汽车会怎样，对吧？每次你更新驾驶策略，你怎么知道它有效？所以，你需要有一队驾驶员去不同的挑战性场景，检查这个策略是否理想，对吧？这太昂贵了。有太多检查点。所以，如果我们能使用世界模型作为世界模拟器，作为真实世界和真实驾驶员的替代，对吧？你让策略直接与世界模拟器交互。而世界模拟器在运行结束时，你可以测量这个任务是否完成。然后你得到成功率，对吧？如果世界模拟器的成功率，新的世界模拟器，与真实世界测试相关，排名保持。你不需要它们精确。你只需要知道在你的世界模拟器中策略 A 比策略 B 更好，那么很可能策略 A 在真实世界中也会比策略 B 更好。你只需要这种排名保持。然后有了这个，你可以快速缩小需要进行真实开发的检查点数量。你将大幅提高开发速度，帮助你更快找到更好的策略。好的。在这个策略验证中，你并不直接使用视觉输出、展开来训练模型，对吧？而模型，我的意思是，大型语言模型中发生的情况也会在世界模型中发生。我们将看到世界模型的快速改进，对吧？所以，即使在它成熟到足以向策略模型提供训练数据进行训练之前，世界模拟器也可以用作策略验证。是的，这就是我的想法。好的，现在我们回到你问的问题，黑客攻击。

Yeah, it's possible. The way I look at how this neural simulator progress is that I think it will be first it will be more useful for passive verification in the beginning. Passive verification is a task that's, you know, if you're a model builder, any model run you're going to have a lot of checkpoints. And you might do ablation study, and you're going to every, you know, variants you're going to have many checkpoints. And how do you know which one is better, right? Ideal case is to deploy each policy in a real robot, in a real environment, and measure the complete rates, right? Success rates. You know, what will happen to self-driving car, right? Every time you update your driving policy, how do you know it works? So, you need to have a fleet of driver and go to different challenging scenario and check whether this policy is desirable, right? It's so costly. There's so many checkpoints. So, if we can use a world model as a world simulator, as a replacement of the real world and real driver, right? You have the policy directly interact with the world simulator. And the world simulator in the end of the run out, you can measure whether this task is completed or not. Then you have a success rate, right? If the success rate of the world simulator, the new world simulator, correlates with the real world, you know, testing, the ranking preserve. You don't need them to be precise. You just need to know if policy A is better than policy B in your world simulator, and most likely policy A going to be better than policy B in the real world. You just need this ranking preserve. Then with this, you can quickly narrow down on the number of checkpoint you need to do real development. You're going to largely improve your development velocity, help you to find a better policy sooner. Okay. and in this policy verification, you don't directly trend the model using the visual output, the rollout, right? And the model, I mean, it what happened in large language model going to happen in world model. We're going to see rapid improvements of world model, right? So, even before it's mature enough to feed the policy model the training data to train, world simulator can be used as a policy verification. Yeah, that's what I think. Okay, so now we go back to the question you asked, the hacking.

即使使用深度学习，黑客攻击也会发生。如果存在某种可以被利用的模式，就会出现对抗样本。所以我认为神经模拟器会被利用，其他模拟器也会被利用。我认为当我们达到那个状态时，会有办法对模型训练进行正则化，确保这些漏洞无法被利用。所以我不确切知道具体方法。我认为这也是一个非常令人兴奋的领域。我只是相信人们会找到办法。

Hacking is going to happen even if you use deep learning. If there's some pattern that can be exploited, there are going to be adversarial samples. So I think neural simulators are going to be exploited, and other simulators are going to be exploited too. And I think as we reach that status, there will be ways to regularize the model training to make sure those cannot be used. So I don't know the precise approach. I think this is also a very exciting area. And I just believe that people will figure out a way.

是的，关于把 Cosmos 本质上作为策略的教师这件事。你们有一套相关的配方，你应该谈谈这个。但有趣的一点是想象的能力，就像那个概念上的“房间里的大象”。也就是那些相当不自然、在训练数据中频率很低的东西。你可以想象那些，但你也可以有这个自适应系统，能够识别边缘情况，然后随着时间的推移改进策略。

Yeah, and on this thing about using Cosmos as a teacher essentially for policies. You guys have got a set of recipes around this and you should talk to that. But one interesting thing is the ability to imagine, like the notional elephant in the room. So something that is quite unnatural that would be quite low frequency in the training data. You can imagine those, but you could also have this adaptive system where you could recognize edge cases and then improve the policy over time.

Cosmos 作为教师。所以在 Cosmos 中，我们试图以三种方式帮助生态系统：更好的数据、更好的环境、更好的起点。所以我们相信，一个 Cosmos 模型——无论是纯模型还是将视频、动作、文本连接在一起学习共享表示的模型——都是构建策略模型的绝佳起点。而且不只是理论，我们实际上在 DROID 数据集上后训练了 Cosmos 模型，并实现了最先进的抓取和放置策略结果。原因——我们的直觉是，如果我们能很好地预测物理动态，并且能学习像素和动作之间的相关性，那么它就能帮助你构建策略模型。当我们不断推进 Cosmos 模型，引入更多具身时，它将成为越来越好的起点，适应新具身所需的工作量也会减少。

The Cosmos as the teacher. So, in Cosmos, we try to help the ecosystem in three ways: better data, better environments, and better starting points. So we believe that a Cosmos model—only model or model that links video, action, text together to learn the shared representation—is a great starting point to build a policy model. And not just a theory, we actually post-train Cosmos model on the DROID data set and actually achieve state-of-the-art pick-and-place policy results. And the reason—our intuition is that if we can predict the physical dynamics well, and we can learn the correlation between the pixels and action, then it's going to help you build a policy model. When we keep advancing Cosmos model, when we bring more embodiment in, it's going to become a better and better starting point, and the amount of effort required to adapt to new embodiment will be less.

嗯，也许你应该谈谈这些配方，因为这真的能帮助在家的人快速上手。

Well, maybe you should comment on the recipes because this will really help folks at home get up to speed with it.

当我们把 Cosmos 构建为一个平台时，我们也在 Cosmos 仓库中提供了后训练配方，这样用户或开发者可以使用这些配方，从 Cosmos 模型开始复现我们取得的结果。我们期待在那里添加更多配方，也许还有技能来帮助你提供数据，然后一个智能体帮助你后训练 Cosmos 并做你喜欢的事情。所以这就是试图以这种方式服务社区。

As we build Cosmos as a platform, we also provide post-training recipes in our Cosmos repo, so that users or developers can use the recipes to reproduce the results we have starting with the Cosmos model. And we look forward to adding more recipes there, maybe also skills to help you provide data and then an agent to help you post-train Cosmos and to work on something you like. So that is trying to serve the community this way.

太棒了。你能告诉我关于 Cosmos Dreams 的事情吗？

Amazing. And can you tell me about Cosmos Dreams?

我们把 Cosmos Dreams 分为——它是一组技能，用于构建闭环模拟器，不仅用于自动驾驶，还用于各种具身。所以它属于机器人学中世界模型的传统定义：输入动作，输出未来观测以模拟动态。你不受车队规模的限制，任何时候你需要做大量测试，你构建计算机，启动它。它将帮助你提高开发速度。自动驾驶非常有用，我认为对机器人也是如此。我认为世界模型现在对于导航任务已经足够好了。在机器人学中，另一个挑战是操作任务。导航——你不希望你的物理设备碰到任何其他东西，但操作任务需要各种交互。当有交互时，会有缓冲，并且当物体被操作时会有潜在的变形。所以这更具挑战性。我非常乐观——随着整个领域对世界模型的兴奋，以及深度学习和计算机的持续进步，我们将达到一个状态，能够使用神经模拟来很好地处理这种复杂的操作。一旦我们达到那个状态，你就可以用世界模拟器测试你的机器人策略。而自动驾驶汽车今天需要大量的正向验证，因为汽车在我们的人类空间中移动，所以我们需要它们足够安全。所以当机器人——人形机器人——达到你将它们部署在许多人家中的状态时，安全性甚至更重要。你不期望在房子里看到汽车。但一旦人形机器人无处不在，你会期望看到人形机器人围绕着你，也许还有你的孩子、你的宠物。所以你确实希望它们足够安全。你怎么知道呢？所以作为人形机器人的开发者，你如何衡量你的新策略或整个系统的每次迭代是否足够安全？你需要策略验证。而你不会总有足够的空间来设置各种厨房、各种不同的任务。所以我认为使用模拟器是唯一能给你所需开发速度的方法。

We divide Cosmos Dreams—it's a set of skills to build a closed-loop simulator for not just self-driving, but for all sorts of embodiment. So it falls into the conventional, traditional definition of world model in robotics: action in, future observation out for dynamics. You are not limited by the size of your fleet, and anytime you need to do a lot of testing, you build the computer, just launch it. And it's going to help you with the development velocity. Self-driving is very useful, and I think it's going to also be true for robotics. I think world models now are good enough for navigation tasks. In robotics, the other challenge is the manipulation task. Navigation—you don't want your physical device to touch any other things, but manipulation tasks require all kinds of interactions. When there's an interaction, there's a cushion, and there's a potential deformation when things are manipulated. So it's more challenging. I'm very optimistic—with the excitement the whole field has in world models and the continued advance of deep learning research and also computers, we will reach a state where we will be able to use neural simulation to handle this complex manipulation well. Once we reach that status, you can test your robot policy with the world simulator. And self-driving cars require a lot of positive verification today because cars are moving around in our human space, so we need them to be safe enough. So when robots—humanoid robots—reach the status where you're going to be deploying them in many people's houses, safety is even more important. You don't expect to see a car in your house. But once humanoids are everywhere, you're going to expect to see humanoids surrounding you and maybe your kids, your pets. So you do want them to be safe enough. And how do you know? So as a developer of humanoid robots, how do you measure that every iteration of your new policy or the whole system is safe enough? You need policy verification. And you won't have enough space for setting up all kinds of kitchens, all kinds of different tasks. So I think using your simulator is the only way to give you the development velocity we need.

你们还有这个 Cosmos 3 Edge 模型。所以这个可以在单个设备上运行。

You also have this Cosmos 3 Edge model. So this runs on a single device.

是的。Cosmos 模型有三种不同尺寸。我们有 Super、Nano 和 Edge。Super 提供最高的保真度。它是一个前沿模型。如果你想要非常高的准确性并且有资源，绝对选择 Super。Nano 介于两者之间，更小，更容易移植和用你拥有的各种 GPU 资源进行训练。Edge 设计为在边缘设备上运行得非常好，比如 Jetson Thor、Orin 或 DJ Spark。所以我们构建这个的原因是，我们希望世界模型生活在机器人所在的地方。这样当你想要采取某些行动时，就不需要往返数据中心。我认为这在真实部署时很重要。你可能无法指望网络连接足够好以完成所有任务，有时它很敏感、关键——你必须在当时完成任务。所以我们开发 Edge，使其足够小但也足够强大，能在那些边缘设备上运行。而且因为模型更小，它占用更少的算力资源，所以可以轻松微调。之前我们有一个配方，可以在一天内微调 Cosmos Edge 模型以提升其视觉理解能力。所以我认为这三个版本都将非常重要。我还设想，它们可能需要以某种形式协同工作，以促进未来的机器人发展。对于非常具有挑战性的任务，你可能想要一个非常强大的模型——就像呼救一样。

Yes. Cosmos models come in three different sizes. We have Super, Nano, and Edge. Super gives you the highest fidelity. It's a frontier model. If you want very high accuracy and you have the resources, definitely go with Super. Nano is in between, smaller, easier to port and train with all sorts of GPU resources you have. Edge is designed to run really well on edge devices like the Jetson Thor, Orin, or DJ Spark. So the reason we build this one is that we want the world model to live where the robot lives. So you don't need to have a round trip to the data center when you want to take certain actions. And I think it's important when you come to real deployments. You may not be able to count on the network connection being good enough to do all the tasks, and sometimes it's sensitive, critical—you have to complete the task at the time. So we develop Edge so that it's small enough but also powerful enough to run on those edge devices. And because the model is smaller, it takes less compute resources so it can be easily fine-tuned. Earlier we have a recipe on fine-tuning the Cosmos Edge model in one day to boost its visual understanding capability. So I think all three versions are going to be super important. And I also envision that they may need to work together in some form for future robotic development. For a very challenging task, you probably want a very powerful model—like calling out for help.

那对家里的观众来说，他们可以在哪里获取这些模型、了解更多信息呢？

And for folks at home, where can they get their hands on these models and find out more about it?

好，我们所有的模型都是开放的，所有代码、甚至我们创建的一部分训练数据也都是开放的。模型和数据放在 Hugging Face 上，代码放在 GitHub 仓库里。对，我们的入口仓库是 github.com/nvidia/cosmo。从那里你可以看到指向 Hugging Face 模型、数据集，以及训练代码、后训练脚本的链接。对。我们还在仓库里放了 skills，帮助你的智能体快速上手 Cosmo 提供的东西。

Yeah, so all our models are open, and all our code and even some of the training data we created are open. So the models and the data live in Hugging Face, and the code lives in a GitHub repo. Yeah, so we have github.com/nvidia/cosmo as our landing repo. From there you can see links to the Hugging Face models, the data sets, and also the training code, the post-training script. Yeah. And we also have skills in the repo to help your agent quickly pick up the offering in Cosmo.

我们做 Cosmos 是为了帮助整个生态，帮助广大的物理 AI 开发者。我们需要你们的反馈。你知道，我有我的邮箱，我会盯着这个 Cosmos 仓库里来的每一个 issue。我们确实希望看到你们的反馈，好的坏的都非常欢迎。我们致力于把它做得越来越好。

We build Cosmos to help the ecosystem, to help the wide range of physical AI developers. We need your feedback. You know, I have my email box, you know, I'm watching every issue that comes to this Cosmo repo. And we do want to see your feedback, good or bad, both are very welcome. And we are committed to make it better and better.

Mingyu，能邀请你上 MLST 是我们的荣幸，也非常愉快。非常感谢你今天来参加我们的节目。

Mingyu, it's been a pleasure and an honor having you on MLST. Thank you so much for joining us today.

谢谢你，Tim。

Thank you, Tim.