走进机器人数据工厂：Mecka 创始人 Josh Gao 谈用人类传感数据训练机器人

科幻小说中有一种观点：当机器人技术和 AI 最终完全泛化并解决了人类能做的所有任务时，最后的边疆将是有机的，最后的边疆将是生物的。你在阿西莫夫的作品中能看到这一点，在《安德的游戏》中也能看到。甚至在《银河系漫游指南》的结尾，那颗行星最终就是一个巨大的有机计算机，试图处理并理解生命的意义是什么，或者问题是什么。

There's an idea in science fiction that when robotics and AI have ultimately fully generalized and solved all tasks that humans can do, the final frontier is going to be organic. The final frontier is going to be biological. And you see this in like Asimov, you see this in Ender's Game. You could even in Hitchhiker's at the very end the planet ultimately was this giant organic computer trying to process and understand what the meaning of life was or what the question was.

今天我很高兴邀请到 Mecka 的 Josh 来到节目。Josh 是 Mecka AI 的创始人兼 CEO，该公司正在打造一家为机器人提供数据和部署服务的公司。我和整个生态系统都认为机器人技术即将进入黄金时代。与 LLM 不同——我们拥有大量来自互联网的免费人类数据——机器人技术从根本上受到数据限制。我认为你在生态系统中作为领先的数据工厂，为模型带来机器人数据和物理数据并使其分布，这是一个非常非常有趣的位置。所以，我很兴奋今天利用这次对话来聊聊：自我中心数据和数据收集在机器人中如何运作，运营数据工厂的实际操作和经济性是什么，以及你对机器人研究和部署的整体看法。感谢你参加节目。

Today I'm delighted to have Josh from Mecka on the show. Josh is founder and CEO of Mecka AI, which is building a data and deployment company for robotics. I and I think the whole ecosystem think that robotics is about to enter a golden age. And unlike LLMs where we had so much free human data from the internet, robotics is fundamentally data constrained. And I think your position in the ecosystem as the leading data factory to bring robotic data and physical data in distribution for models is just a really, really interesting perch. So, I'm excited to use this conversation today to chat about how does egocentric data and data collection work in robotics, what are the actual operations and economics of operating a data factory, and your overall thoughts on robotics research and deployment. Thanks for joining the show.

太棒了。感谢邀请我。

Amazing. Thanks for having me.

太好了。呃，也许先给不熟悉的人介绍一下，Mecka 是什么，你打算如何影响机器人世界？

Awesome. Uh maybe to start off for those who aren't familiar, what is Mecka and how are you looking to impact the world of robotics?

所以，Mecka 就像你说的，是物理 AI 的数据和部署公司。这意味着今天我们创建、处理并最终授权大量用于训练这些机器人基础模型的数据。机器人领域的最大问题是，我们没有 LLM 从互联网获得的那种冷启动的奢侈。从根本上说，很多机器人数据是三样东西：像素、动作和逻辑。所以我们无法获得大量经过整理、处理和干净的数据，来训练这些模型理解身处物理世界是什么感觉。因此，我们的很多工作基本上是：第一，创建不同类型的硬件，比如为人类配备传感器和可穿戴设备；第二，开展不同的运营，去收集并与个人贡献者、企业、不同环境中的任务合作；最后，进行后处理并掌控质量关卡，使这些数据集有用，并教机器人了解世界。

So, Mecka is like you said the data and deployment company for physical AI. What that means is today we create, we process, and ultimately we license a lot of the data that goes into training these robotics foundation models. The big problem across robotics is that we don't have the luxury of the cold start that LLMs had with the internet. Fundamentally, a lot of this robotics data is three things: it's pixels, its actions, and its logic. And so we don't have access to a large amount of this curated and processed and clean data to be able to then train these models to understand what it is to be in the physical world. So a lot of our job is pretty much to one, create the different types of hardware such as sensorizing and wearables for humans, the different operations to go and collect and partner with individual contributors, businesses, tasks across different environments, and then ultimately post-process and own the quality gates to make these data sets useful and teach robots about the world.

没有获得巨大的免费数万亿 token 的好处，也许很多万亿。数万亿数万亿 token 的互联网语料库。你认为机器人技术，你知道，必须付费并在运营上收集所有进入混合的数据，这甚至是一个可行的挑战吗？比如，在没有互联网礼物的情况下，经济上可行吗？

Without having been given the benefits of a giant free many trillions of tokens, maybe lots of trillions. trillions of trillions of tokens of um internet corpus. Do you think that robotics, you know, having to pay for and operationally collect all the data that's going into the mix, is it even a feasible challenge? Like is it economically feasible uh without having had the gift of the internet?

我认为我们仍然拥有互联网赋予的许多奢侈。所以我们最终并不是从零开始。我想你可以从 LLM 和数据中看到这一点。预训练数据一直主要由实验室通过网页爬虫等驱动。它并没有成为一个大市场。今天在机器人领域，我们确实将预训练视为一个相当大的市场，这可以采取几种形式。比如，我们教视频模型如何生成更符合物理现实的世界样貌，并且我们可以利用之前投入 LLM 和不同类型世界模型的所有工作来实现这一点。是否可能且可扩展地达到那样的数据量？我认为这最终取决于你能在哪个数据规模水平上解锁什么商业用例。我们坚信，在这一旅程中，我们将能够在我们创建之初解锁许多简单的拾取和放置任务，并朝着比今天大许多个数量级的长尾发展。

I think we still have the luxury of a lot of things that was given from the internet. So we aren't starting from zero ultimately. I think you can see it across LLMs and data across LLMs. Pre-training data was always driven by the labs largely across web crawlers, things like that. It didn't become a large market. Today in robotics, we do see pre-training as quite a large market and that could be taken in several forms. So that could be that we are teaching video models how to generate more physics grounded realities of what the world looks like, and we get to leverage all the work that has gone into LLMs and different types of world models before to make that possible. Is it possible and scalable to get to that volume of data? I think that ultimately comes down to what commercial use case can you unlock at which levels of scales of data. And our strong belief is along that journey that we will be able to unlock a lot of the simple pick and place tasks along the beginning of what we're creating and towards that long tail that is going to be many orders of magnitudes larger than it is today.

在这之前我做了一点功课。你们的网站非常异想天开。呃，上面说你们在教机器人关于碳基生命。那是什么意思？

I did a little homework before this. Your website is very whimsical. Uh it says that you are teaching robots about carbon life. What What does that mean?

实际上是我做的网站。我很喜欢。就像我们第一次发推文宣布的前一晚之类的，科幻小说中有一种观点：当机器人技术和 AI 最终完全泛化并解决了人类能做的所有任务时，最后的边疆将是有机的，最后的边疆将是生物的。你在阿西莫夫的作品中能看到这一点，在《安德的游戏》中也能看到。甚至在《银河系漫游指南》的结尾，那颗行星最终就是一个巨大的有机计算机，试图处理并理解生命的意义是什么，或者问题是什么。所以我认为同样地，对我们来说，人类大脑只运行 20 瓦。我们能自我修复。因此，在教机器人并让它们完全泛化的旅程中，我们会慢慢向它们展示碳基生命是由什么构成的。喜欢这个。

I actually made the website. I love it. Like the night before we did our first tweet announcement or something like that, there's an idea in science fiction that is when robotics and AI has ultimately fully generalized and solved all tasks that humans can do, the final frontier is going to be organic. Final frontier is going to be biological. And you see this in like Asimov, you see this in Ender's Game. You could even in Hitchhiker's at the very end the planet ultimately was this giant organic computer trying to process and understand what the meaning of life was or what the question was. And so I think in the same vein for us that is the human brain runs on just 20 watts. We are self-healing. And so along the journey of teaching and getting robots to fully generalize we'll slowly show them what carbon life is made of. Love that.

好的。你来自一个非传统的机器人背景。我想你最初是在金融科技界和球鞋界，如果我没记错的话。告诉我你和你的联合创始人是怎么走上机器人之路的。

Okay. You come from an untraditional background for robotics. I think you started in the fintech world and the sneaker world if I'm not mistaken. Tell me about how you and your co-founder found your way to robotics.

这是一段相当有趣的旅程。是的。所以在此之前，我基本上在为云厨房构建一个 Toast。这意味着我们做了从后台订单管理的一切。在我大学四年级的某个时候，我逃课去纽约州罗切斯特的乡村部署和安装自助点餐机，我们不得不通宵达旦地更换许多餐厅的前台系统。我的联合创始人构建了一个稳定币。他们最终卖给了 Coinbase，但他一直是一个非常执着的人。他在四款竞技游戏中排名前一千，包括《英雄联盟》。我认为让我们走到一起的一个重要部分，一是我们都是加拿大人，二是我们有深厚的科幻基础。所以最终的信念是，支付是一个艰难的问题，但无论我们接下来做什么，我们希望它看起来令人兴奋，未来会不可思议，而机器人似乎是一个很好的领域。

It was quite an interesting journey. Yeah. So before this I was building essentially a toast for cloud kitchens. This meant we did everything from back of the house order management. At one point in my fourth year of university, I was skipping class to go deploy and install kiosks into rural Rochester, New York, where we had to stay all nights replacing a lot of these different front of the houses for restaurants. My co-founder built a stablecoin. They ultimately sold to Coinbase, but he's been this very obsessive personality. He was a top thousand in four competitive games, including League of Legends. And I think a big part of what brought us together, one, we're both Canadian, and two is we have this big science fiction grounding. And so the belief ultimately was that payment was a hard and difficult problem, but whatever we worked on next, we wanted to be something that seemed exciting, that the future was going to be incredible, and robotics seemed like a great space to get into.

我记得走进你的办公室，立刻看到那些机电一体化机器人，然后墙上还有川崎的 Quleos。你内心的孩子一定对你现在做的事情非常满意。

I remember walking into your office and then immediately seeing those like mechatronic robots and then the the Kawasaki Quleos on the wall. The inner child in you must be very happy about what you're doing these days.

是的，绝对。

Yeah, absolutely.

来自非机器人背景，你认为你在建立这项业务中的优势是什么？嗯，它在哪些方面带来了挑战？

Coming from a non-robotics background, what do you think your advantages are in building this business? Um, and where has it presented challenges?

我认为优势在于，当我们进入这个领域时，我们不得不进行这个完整的发现过程。

I think the advantages is when we entered the space, we had to go and do this full discovery process.

我们联系了数百位我们欣赏的业内人士。我们阅读了能拿到的每一篇论文。我们参观了不同的机器人实验室，在 2024 年底左右，我们看到 GPT 正在起飞。机器人技术仍在从传统机器人时代演变，至少对我们来说很明显，数据和算力将在整个行业中变得非常重要。因此，朝着这个方向，我们得以从全新的视角阅读和查看一些论文，比如 Ego4D，然后说：‘嘿，这里有大约 4000 小时的人类演示数据。’当你真正去看数据时，你会发现一些数据集是人们的日常生活，比如打保龄球。在 4000 小时的规模上，有非常广泛的用例和潜在应用。因此，带着全新的视角进入这个领域，我们能够从第一性原理出发，对什么重要有一个大致的直觉，然后与比我们聪明得多、对未来有不同看法的人紧密合作，并保持接近，以迭代产品最终的样子。

We reached out to hundreds of folks that were in the space that we appreciated. We read every paper that we got our hands on. We went to visit the different robot labs and what we saw in late '24, roughly, GPT was taking off. Robotics was still evolving from this traditional robotics era and it seemed obvious to us at the very least that data and compute would become very important across the industry. And so moving towards that, we got to read and see some of the papers like Ego4D from a fresh lens and say, 'Hey, there's about 4,000 hours of this human demonstration data.' And when you actually went in to look at the data, you would realize some of the data sets would be daily life of folks like bowling, for example. There was very broad use cases and potential applications at the 4,000-hour scale. And so coming into that with a fresh lens, we were able to from a first principles approach, I'd say, have a general intuition of what would be important and then iterate very closely with people that were much smarter than us that had different views of what the future would be and be able to stay close to iterate on what the product would become.

是的。有趣。看起来机器人技术作为一个领域变化如此之快，几乎来自该领域的人可能既是优势也是劣势，因为人们来自不同的时代，你知道，关于训练组合中什么重要等等。

Yeah. Interesting. It seems like robotics has just changed so quickly as a field that almost coming from the field can be an advantage and a disadvantage because people come from different eras of, you know, what's important in the training mix and things like that.

是的。

Yeah.

嗯，也许对我们的观众简单说一下什么是自我中心数据，我认为你创办公司时真的是在它背后下了一个赌注。显然现在它更加分散了，但什么是自我中心数据？为什么你一开始决定把公司押注在它上面？

Um maybe for our audience just say a word on like what is egocentric data and I think you really started your company kind of making a bet behind it. Obviously it's much more distributed now but what is egocentric data? Why did you decide to bet the company on it at the beginning?

自我中心数据基本上是人类从第一人称视角演示和执行任务的视频数据。高层次的理念是，从人类数据到机器人数据的差距中，你可以缩小很多视觉差距，机器人从第一人称视角体验世界，我们能否也向它展示从第一人称与世界互动是什么样子？我们押注它的原因主要是，当我们探索这个领域时，当时看到的很多机器人状态是遥操作，即人类控制机器人进行演示。当时很难规模化，而从 GPT 和许多规模化原则中我们看到，我们必须找到方法更显著地扩大数据规模。我认为我们早期的一些直觉是，人类和机器人有很多相似之处。人形机器人刚刚开始起飞。人类和人形机器人看起来一样。为什么我们不能把人类变成像特斯拉那样的车队式学习，能够全部贡献于理解并 grounding 世界的物理。

Egocentric data is pretty much video data from humans demonstrating and doing tasks from a first-person POV. The high-level idea was that from the gap of human data to robotics data, you could close a lot of the vision gap where robotics experience the world from POV, can we also show it what it is like to interact with the world from first person as well? Why we bet on it was pretty much when we explored the space, a lot of what we saw was robotic state at the time was teleoperation which was humans controlling robots to do demonstrations. It was very hard to scale at the time and from what we saw across GPT and a lot of the principles of scaling up, we had to find ways to scale up data much more significantly. And I think that some of our early intuitions were that humans and robots share a lot of similarities. Humanoids were just starting to take off. Humans and humanoids look the same. Why couldn't we turn humans into this fleet-style learning that Tesla has to be able to all contribute into understanding and ground the physics of the world.

那么那看起来像什么？你像是把 iPhone 安装在某人的额头上，然后让他们做自己的事情，或者在实践中那是什么样子的？

And so what does that look like? You're like mounting an iPhone on somebody's forehead and just letting them go about their day or what does that look like in practice?

是的，当我们刚开始时，我们最早的试点是创建了一个移动应用。我们让所有的中国阿姨和叔叔记录自己做饭、打扫、做任务。那显然进化了很多。我们尝试了很多东西。我们做了 Umei 数据。一开始我们做了各种尝试。那已经演变成我们现在必须生产的所有定制硬件。以及我们必须在不同行业建立的所有不同合作伙伴关系。最终我们是在为人类配备传感器。这是更广泛的类别。

Yeah, when we first started, our very earliest pilot was we created a mobile app. We got all the Chinese aunties and uncles to record themselves cooking, cleaning, doing tasks. That obviously evolved a lot. We tried a number of things. We did Umei data. We did all sorts at the beginning. And that's evolved quite a bit into all the custom hardware that we have to produce now. And all the different partnerships that we have to have across different industries. And ultimately we're sensorizing humans. And that's the broader category here.

从传感器获取力数据有多重要？因为，你知道，我一直思考的一件事是，在自动驾驶中，你并不真的那么需要力数据，对吧？而在机器人技术中，尤其是操作，你真的关心我捏门把手或拿起鸡蛋时用了多大力。在我看来，视频并不能完全捕捉到这一点。你也许能近似，但并不能真正捕捉。捕捉力数据有多重要？你如何考虑获取它？

How important is it to get force data from the sensors? Because, you know, one of the things I've always thought about is in self-driving, you don't really need force data as much, right? Whereas with robotics, especially with manipulation, you really care about how hard am I squeezing that door handle or picking up that egg. And it seems to me that like video doesn't quite capture that. You might be able to approximate it, but doesn't really capture it. How important is capturing force data? And how do you think about getting that?

我认为我们看到的力数据的一些替代品将是目前用于人类和机器人的腕部摄像头等方法，能够看到手与你互动的不同物体之间的接触时间。这显然是一种取巧的方式，最终是为了理解拉开椅子或拧开瓶盖需要多少牛顿的力。我认为力数据将非常重要。它实际上是我们一直在构建的产品之一，当你与许多不同的机器人基础实验室迭代时，你意识到要缩小这个差距，我们必须创造新的硬件。所以我们一直在开发像手套这样的东西供人类佩戴，并在这些手套中收集做日常生活和不同任务需要多少力的信息。

I think some of the substitutes for force data that we've seen would be approaches like wrist cameras currently across both humans and robots and being able to see the contact time between the hand and the different objects that you interact with. That obviously is kind of a hacky way ultimately to get to understanding how much force in Newtons it takes to pull open a chair or twist a bottle cap. I think force data will be incredibly important. It's actually one of the products that we've been building out in that when you iterate with many different robotics foundation labs, you realize to close this gap one we have to create new hardware. So we've been developing things like gloves for humans to wear and in those gloves being able to collect the information in how much force it takes to do your daily life and different tasks.

我想花点时间谈谈数据工厂的实际运营，因为我认为，有那么多钱投入到了 LLM 数据工厂。有那么多钱开始投入，我认为将会投入到机器人数据工厂，它们的工作方式对我来说只是，你知道，神秘而有趣。所以，如果可以的话，我想问你一些关于它的运营。

I'd love to just spend a bit of time on the actual operations of a data factory because I think it is there's so much money that's gone into the LLM data factories. There's so much money that is starting to be poured into and I think will be poured into the robot data factories and the way they work is just, you know, mysterious and intriguing to me. So, I'd love to ask you a bit about the operations of it if that's okay.

好的，我们来吧。

Yeah, let's do it.

好的。那么，它是如何运作的？比如，如果我想赚点外快，我是否只需在你的网站上注册，然后开始，你知道，把摄像头放在我的额头上，开始向你们发送数据，或者它是如何运作的？

Okay. So, how does it work? Like if I am, you know, if I want to make an extra buck, do I just sign up on your website to start, you know, to put a camera on my forehead and start sending you data or how does it work?

所以，我们在操作员方面实际上有很多不同的方法。一种是我们有直接面向消费者的方法。因此，通过直接的移动应用，你可以从世界任何地方注册吗？你可以开始贡献吗？你可以注册你从事哪个职业吗？你可以收到一系列任务流去做，比如打扫卧室、做早餐、之后清理。这是一种非常轻量级的方法，最终获得预训练数据的多样性。你必须进入的第二种方法是，最终家庭之外有太多东西需要收集。因此，我们必须构建很多这种 B2B 引擎和这种肌肉，能够招募企业，涵盖所有体力人类工作的惊人分类，包括面包店。可能是 PCB 制造线。一切到

So, we actually have a lot of different approaches on the operator side. One would be we have a direct consumer approach. And so through direct mobile apps, can you sign up from anywhere in the world? Can you start contributing? Can you register which profession you're in? And can you get streamed a sequence of tasks to do such as cleaning your bedroom, cooking your breakfast, cleaning up afterwards. This is a very lightweight approach to get that diversity ultimately of this pre-training data. The second approach that you have to get into is ultimately there is so much more beyond the home to collect. And so we had to build out a lot of this B2B engine and this muscle of being able to recruit businesses across this incredible taxonomy of all physical human work which is across bakeries. It could be a PCB manufacturing line. Everything to

播客，这里没有太多手工活。

podcasting there's there's not too much handwork buck here.

我们应该给自己装备一下。嗯，最终这就是贡献者如何进入混合的方式。

We should have equipped ourselves. Um and and that's ultimately how contributors come into the mix.

但数据工厂的实际数据引擎远比人们预期的要复杂和痛苦得多。我们要分发到世界上的很多东西，直到我们设计并生产出这些硬件才存在。触觉手套就是一个很好的例子。市面上没有现成的产品可以购买并佩戴，能可靠地告诉你日常施加了多少力，并将其转化为这些不同机器人基础模型的现成数据集。所以我们必须有硬件迭代循环，经历三四个不同的产品周期，具备那种工程能力去真正思考和迭代什么是有用的，然后将生产规模扩展到全世界。这就变成了物流问题：分发到成千上万个家庭、成千上万家企业去收集、上传。我们不得不在许多运营地点设置 100 GB 的上传线路，仅仅是为了支撑我们目前必须处理的数百 PB 数据。然后你必须处理，确保你的质量关卡非常出色，最后还要用所有计算机视觉技术让它变得有用。

But the actual data engine for the data factory is so much more sophisticated and much more painful than folks can expect. A lot of the things that we have to distribute into the world didn't exist until we designed and produced this hardware. Even across tactile gloves is a fantastic example. There is nothing off the shelf that you can purchase and wear and that reliably tells you how much force you are applying day-to-day and turn that into ready data sets for these different robotics foundation models. So we have to have that hardware iteration loop to have these three or four different product cycles, that engineering muscle to actually think and iterate what is going to be useful and then scale that production into the world. And that becomes this logistics of distributing into these thousands of homes, thousands of businesses to collect that, to upload that. We've had to set up these 100 gigabyte upload lines in a lot of the places that we operate in just to sustain the hundreds of petabytes of data that we have to deal with at this point. And then you have to process, you have to make sure your quality gates are incredible, and then all of the computer vision to make it useful ultimately.

那么，假设你决定，好吧，在我的训练组合中，播客主不够多，或者，我不知道，甜甜圈师傅不够多。你会真的去挨家挨户敲甜甜圈师傅的门，让他们注册到平台上吗？还是怎么运作的？

So, let's say you decide, okay, in my training mix, there's not enough podcasters, there's not enough, you know, I don't know, donut bakers. Do you go out and literally go knock on doors of doughnut bakers then to sign them up to the platform or how does that work?

嗯，我们通常会跟踪不同工作、行业以及我们思考多样性的方式，就像一张热力图。在这张热力图上，如果我们最终认为，哦，Sonya 的播客不够多，我们必须去获取。其实这就是我们今天在这里的原因。这是我们的秘密方法。

Well, we'll generally track kind of like a heat map across different jobs, industries, and ways we think about diversity. And across that heat map, if ultimately we think, oh, not enough Sonya's podcasting, and we have to go get that. That's actually why we're here today. This is our secret method.

难怪到处都是摄像头。

That's why there's cameras everywhere.

不，你永远别想用自动化把我从这个岗位上替换掉。嗯，不，我开玩笑的。请便。呃，好吧。有意思。我听说，一般来说机器人数据收集存在很多质量问题，很多提交上来的机器人数据样本小时数最终不被接受。嗯，为什么会这样？你们在怎么解决这个问题？

No, you're never automating me out of this. Um, no, I'm kidding. Please do. Uh, okay. Interesting. Like, I've heard that, you know, there's a lot of quality issues in robot data collection generally and that a lot of the hours of robot data sample that get sent in end up not being accepted. Um, why is that and what are you doing about that problem?

这个教训最初来自不同类型的机器人数据。曾经有这些非常大规模的机器人工厂，从大型仓库中涌现出来，重复执行任务，试图扩大遥操作数据的规模。其中一些经验来自研究，即存在某个多样性阈值，超过后你对机器人数据的需求或对一般数据的需求就会过饱和，而人类数据的质量……你可以想象，随着我们成长，这实际上是一个不断变化的目标。现实世界中你必须处理的边缘情况数量非常极端，你总会看到没见过的新东西。所以对我们来说，关键在于你能否思考出启发式方法，在处理过程中捕捉许多不同的情况。你能否有不同的引擎来快速升级处理？最终我们会学到关于质量和多样性的新东西，甚至一周前还不成立。因此，敏捷地快速行动，创建并调整你的引擎来捕捉这些。呃，并确保……

This lesson came from even originally the different types of robot data. There have been these very large scale robot factories that have spun up from just large scale warehouses repetitively doing tasks trying to scale out teleoperation data. Some of the learnings there came from the research in that there is some threshold of diversity where you oversaturate how much robot data you need or how much data in general you need across human data quality. You can imagine is actually quite a bit of a moving target as we grow. The amount of edge cases that you have in the real world that you have to deal with are very extreme and you will always see something new that you haven't seen. So for us it's been about can you think about the heuristics to be able to catch a lot of the different things as it's processed. Can you have the different engines to escalate very quickly? And ultimately we will learn new things about quality and diversity that weren't true a week ago even. And so the nimbleness to move very quickly to create and adapt your engine to catch that. Uh and make sure

你们会捕捉到的一个例子是什么？

What's an example of something that you would catch?

嗯，我认为欺诈在整个数据行业都是一个被低估的问题。我们最终是在创造尚不存在的数据，而不是在过滤互联网数据。所以当你遇到非常有创意的……

Well, I think fraud is an underestimated one across data industries in general. We're ultimately creating data that doesn't exist yet versus when you're filtering through the internet. And so when you are coming across very creative and

比如人们发送生成的视频还是什么？

Like people are sending generated videos or what?

是的，非常有创意的人，我想有一次我们有人录制了像在麦当劳工作的第一人称视角的 YouTube 视频，然后试图把它作为贡献提交上来。

Yeah, very creative people where I think at one point we have folks recording YouTube videos like POV working at McDonald's and trying to submit that as contribution pretty much.

哇，永远不要低估人们的创业精神。太厉害了。

Wow, never underestimate how entrepreneurial people can be. That's amazing.

是的，非常如此。所以这些是痛苦教训的一部分：是的，你必须创建很多关卡，但最终你还必须亲临数据收集现场。你必须去看看它是怎么做的。你必须设计一个真正激励人们做出良好贡献的激励模型。

Yeah, very much. And so these have been like part of the painful lessons where yes, you have to create a lot of the gates, but ultimately you have to also be there where collection is happening. You have to go see how it's done. You have to figure out an incentive model that actually incentivizes people to create good contribution.

你如何看待长期赚取利润的权利？你知道，我可以想象一个机甲竞争对手来找 Sony 说：“Sonya，我真的很想要你的播客数据。”嗯，我可以想象，你作为中间商所能拥有的利润最终会被挤压。

How do you think about your right to earn a margin long term? And you know, I can imagine a Mecka competitor comes to Sony and says, "Sonya, I really want your podcast data." Um, I could imagine that you know the margin that you're able to own as an intermediary ends up getting squeezed.

我认为，就端到端确保数据有用而言，贡献者金额在利润成本方面实际上并不是最大的因素。所以即使那个基线永远存在，也许变得越来越没有竞争力，也会有办法差异化，确保你的数据在非常大规模上可验证地高质量，凭借正确的硬件、运营和软件。我认为当我们思考这些产品时，最终我们确实想要扩大规模并成熟这些产品，就像任何开始规模化的产品一样，你想要优化它，但随着时间的推移，你可能会有不同的产品演变。所以当你把播客这种自我中心数据仅仅看作你的 SKU 之一时，你会意识到将有不同类型的数据变得重要，比如也许我们从路边中餐馆开始，能否逐渐进入米其林星级餐厅？你能否越来越接近成为费德勒而不是普通网球选手？所以，这些产品会变化，这些产品的创造方式也会变化。

I think the contributor amount in terms of the margin cost is actually not the largest factor at play here in terms of the end to end making sure this data is useful. So even though that baseline is always going to be there and maybe becomes more and more uncompetitive, there will be ways to differentiate and make sure that your data is verifiably high quality at very large scale with the right hardware, the operations and the software. I think when we think about these products ultimately it is that we do want to scale up and mature these products and just like any product that begins to scale up you want to optimize it but over time you may have different products that evolve. And so when you think about this egocentric data of podcasting as simply one of your SKUs and you then realize that there is going to be different types of data that are going to be important such as maybe we move from the Chinese restaurant from the side of the road. Can we start moving into the Michelin star restaurants over time? Can you get more and more closer into what it means to be Fedver than it means to be an average tennis player? So, these products will change and how these products are created will also change.

非常有趣。你认为已经收集了多少规模的数据？你认为未来几年会收集多少数据？

Really interesting. What scale of data has already been collected, do you think? And how much data do you think is going to be collected over the next few years?

我认为我们终于进入了一个拥有数百万小时数据的世界，而两年前 Ego 4D 只有 4000 小时，相比之下这是一个巨大的进步。但相对而言，想想自动驾驶的例子。特斯拉花了十多年解决两个自由度——加速和方向，数百万辆车上路。他们现在有远不止几百万小时的视频数据。所以相对而言，我们距离最终目标仍然差许多个数量级。

I think we finally entered a world where we are in the millions of hours of data and coming from the world where Ego 4D of 4,000 hours 2 years ago where we were then is a huge step up. But then comparatively and you think about autonomous driving as an example. Tesla spent over a decade solving for two degrees of freedom ultimately of acceleration and direction millions of cars on the road. They have a lot more than a few million hours of video data at this point. So relatively we're still many orders of magnitudes away from where we have to be ultimately.

所以一方面，这听起来对你们是机会。听起来机器人数据工厂会有很多收入。

So on one hand that sounds like opportunity for you. That sounds like a lot of revenue for the robot data factories.

但另一方面，作为一个投资机器人基础模型的人，我看着这一切会说：天哪，我们到底能不能到达应许之地？因为我和你算过同样的账：一辆车只有两三个自由度，而一只机器人手就有几十个自由度。

On the other hand, as somebody that invests in robot foundation models, I look at that and I say, oh shoot, are we ever going to get to the promised land? Because I've done the same math as you, and a car has two or three degrees of freedom. A robot has even dozens in the hand.

对，没错。

Yeah. Exactly.

再加上你可能还需要力觉数据，而且这是一个难得多的工作环境。我算下来，你可能需要数十亿、数万亿甚至数千万亿小时的数据。全世界有足够的 GDP 来支撑这个成本吗？你怎么看？

And so you combine that with, you probably need force data, and it's just a way harder environment to operate in. And I get to, you might need billions, trillions, quadrillions of hours. Is there even enough GDP in the world to support the cost of bringing that in? Like, what do you think?

我认为关键在于随着进展不断解锁商业用例。所以不必那么二元化——不是“某一天我们突然解锁了机器人”或者“没有”。它会慢慢渗透进现实世界。实现这一点的好办法是：随着这个基础金字塔逐步搭建起来，我们看到了泛化能力，看到了缩放定律在起作用，这给了人们扩大规模的勇气。有了这份勇气，我们能否先在其中一些环境中获得一定的泛化能力？然后能否再收集大量领域内、任务内的数据，通过后训练来弥合差距？所以也许我可以正好在这个房间里、就在这次播客演示中采集数据，来真正弥合差距——在很多情况下，这些真实环境需要四个九的可靠性。

I think it's about unlocking commercial use cases as we go. So it doesn't have to be so binary, in that, oh, one day we've unlocked robotics or not. It is going to be a slow creep up into the world. A great way to get there is that with some of this base pyramid built up, where we are seeing the generalizations, we are seeing these scaling laws at play, that gives people the courage to scale up. And with that courage, can we, one, have some generalizability in some of these environments? Can we then also collect a lot of the in-domain, in-task data that closes that gap in post-training? So maybe I can collect data exactly from this room, exactly in this podcast demonstration, to actually close the gap of what is, in a lot of cases, four nines of reliability in these real-world environments.

这个框架很有帮助。所以这不是从 0 到 1 的二元问题，而是我们会逐步获得能做越来越多事情的机器人。

That's a helpful framework. So it's not about a binary of getting from zero to one. It's about we're going to increasingly get robots that can do an increasing set of capabilities.

而且我们会沿途解锁很多任务。

And we will unlock a lot of tasks along the way.

是的。不过我有个好奇的地方：现在最充足的数据似乎是家庭数据，对吧？每个人都能拍，很容易获取。但家庭里的机器人几乎是最难部署的环境之一。所以我很好奇，你认为机器人领域会最先落地哪些应用？因为数据最容易获取的地方，似乎恰恰和机器人最容易部署的地方相反。

Yeah. One curiosity though is that the data that is most plentiful right now is, it seems like, home data, right? Just like everyone can film that. It's pretty easy to get. But the robot in the home almost seems like one of the most difficult environments to deploy in. And so I'm curious what applications you think will hit first in robotics, because where the data is most readily available seems almost the opposite of where robotics might be easiest to deploy.

家庭绝对是我们必须最谨慎对待的环境之一。机器人领域刚刚开始成熟起来的很多基准测试和评估——比如确保机器人在家里不会摔到婴儿身上。我认为沿着这个谱系，最初的任务会受限得多，属于那种非常目标导向的领域，你可以对这些任务做爬山式优化，而且对于那份工作该怎么做，已经有现成的标准作业流程（SOP）。

The home is going to be some of the environments that we have to be most diligent on, by far. A lot of these benchmarking and evals that have just started to mature across robotics—making sure that the robot doesn't fall on the baby in the homes, for example. I think the initial tasks will be much more constrained as we go along that spectrum, in that very goal-oriented space where you can hill climb some of these tasks, where you have existing SOPs for what it means to do that job and that work.

比如叠衣服机器人？

Like a laundry folding robot.

可以是叠衣服。叠衣服。我们花了大约一年半多的时间试图解决它，我认为它在短期内绝对会被解决。很多这类抓取放置、包装任务——你在装配线上看到的那些必须靠操作机器人完成、传统机器人无法解决的任务——我们会在未来一两年内做到。

It could be laundry folding. Laundry folding. We spent about a year and a half plus trying to solve it, and I think it will absolutely get solved in the near term. A lot of these pick-and-place, these packaging tasks that you see on assembly lines that had to be manipulation bots that couldn't be solved by traditional robotics—we will get there within the next year or two.

这正好可以过渡到部署的话题。我很想和你聊聊机器人部署这个整体概念。你说过，有用的机器人今天已经可以部署了。你认为哪些地方已经做到了？

This is a good transition to talking about deployment. I'd love to talk to you about robot deployment overall as a concept. You've said that useful robots can already be deployed today. Where do you think that's the case?

我认为人们低估了机器人可以有多多样。如果我们只局限于一种——人形机器人非常受欢迎，它也会是机器人非常受欢迎的终极形态。但除此之外，今天机器人领域存在太多不同的形态。你可以看看不同的市场，比如中国或东南亚的酒店，那里有机器人做服务、做清洁、做配送——你的 Uber Eats 送到门口之后，从前台送到你的酒店房间门口。这已经存在了。很多这类机器人今天已经在全世界广泛部署，只是刚开始在美国和北美普及。我认为最先必须部署的一些机器人，甚至就是工业清洁机器人，比如市面上很多 Madic 的机器人。然后从那里慢慢往上走，你开始沿着这个谱系解决早期的操作任务，一路直到未来人形机器人开箱即用地解决任何任务。

I think people underestimate how diverse robotics can be. And if we constrain ourselves to one—humanoids are incredibly popular, it will be a very popular terminal state of robotics as well. But beyond that, there are so many different form factors that exist in robotics today. You could look at different markets, even across hotels in China or Southeast Asia, for example, where there are robots that are doing services, that are doing cleaning, that are delivering—after your Uber Eats gets to the door, from the front desk to your hotel room door. That exists. Many of these bots are widely deployed across the world today. They are just starting to get popularized across the US and North America. I think some of the first bots that have to get deployed will be even just the industrial cleaning bots, for example. A lot of the Madic robotics that are out there. And then slowly stepping up and up from there, where you start to solve early manipulation tasks along that spectrum, all the way up until we get to that humanoid solving out of the box any task in the future.

你在节目前面提到了特斯拉和 FSD。你认为我们什么时候会到达那种有点像 FSD 的飞轮状态？到了那个时候，你认为你们公司扮演的角色会如何演变？

You mentioned Tesla and FSD earlier in this show. When do you think we're going to get to the point where there is a little bit of an FSD-like flywheel? And when we get to that point, how do you see your role as a company evolving?

我们什么时候能到——我认为还早，而且会是一个缓慢的过渡：当你慢慢解决一些商业任务时，你最终也能获得一些飞轮效应。然后你可以拿到一些部署中的数据，包括失败案例和纠正数据。这对启动机器人飞轮绝对重要。距离那一步有多远，取决于那个谱系，取决于能否饱和覆盖不同类型、尚未完成或尚未解决的任务和领域。所以还相当遥远。当我们真的到达那里时，我相信 Mecka 在这个过程中会变得非常不同：第一，我们并不完全只依赖第一人称视角数据。我们普遍认为很多不同的数据品类、整个“数据汤”都很重要。所以未来某个时候，我们会不会生产米其林星级的数据？我们会不会教授一些机器人尚未触及的真实世界边缘案例？如果我们相信世界极其广阔、细节丰富，那要教的东西永远不会少。即便在那个世界里，我们也有这样一个角度：我们相信 Mecka 会极大地促进机器人的部署。所以在其中一些环境中，在弥合“能在那个领域、那个环境中完成那项任务”的差距时，我们能否真正成为这样一个平台：收集领域数据、做后训练、帮助支持企业，并手把手带他们进入一个充满机器人的世界？我们看到 Mecka 正朝这个方向成熟。

When we will get there—I think we're away, but it'll be a slow transition, in that when you are slowly solving some of the commercial tasks, you will also be able to finally get some of these flywheels. Then you can get some of the in-deployment data where you get the failure cases, you get the correction data. That will be absolutely important in starting up this robot flywheel. How far that is depends on that spectrum and across saturating different types of tasks and domains that aren't completed or solved for yet. So it's quite a ways away. When we do get there, the thing is, I believe that Mecka will look very different along that process, in that, one, we're not fully subscribed to only egocentric data. We believe generally that a lot of different SKUs, the entire data soup, is important. And so will we be producing the Michelin-star data at some point in the future? Will we be teaching some sort of edge case about the real world that the robots haven't gotten to yet? If we believe that the world is incredibly vast and rich in detail, there's going to be no shortage of things to teach. And then even in that world, we do have this angle where we believe that Mecka will very much contribute to the deployment of robots. And so in some of these environments, in closing that gap of being able to do that task in that domain in that environment, can we actually be the platform where we collect that domain data, we post-train, we help support the enterprises and white-glove hold them into what it means to be in a world filled with robotics? We see Mecka maturing into that.

我很想听听你对物理世界中部署和扩散会如何发展的预测。也许我可以类比一下 LLM 世界发生的情况——部署有几种不同的方式，对吧？有垂直领域的应用公司。

I'd love to get your predictions on how you think deployment and diffusion is going to work out in the physical world. And maybe if I can analogize what's happened in the LLM world, there are a few different ways that deployment has happened, right? There are vertical-specific application companies.

有基础模型公司自己在扩大市场推广，还有所有这些部署公司、服务公司，随便你怎么称呼，就是大批的人帮助企业大规模采用 AI。这些似乎是把 token 传播到整个经济中的广泛载体。你认为在物理世界中对应的会是什么？

There's the foundation model companies themselves that are scaling up go-to-market, and then there's all these deploy cos, service cos, whatever you want to call them, just armies of people helping enterprises adopt AI at scale. Those seem like the broad vectors that are disseminating tokens across the economy. What do you think the analog will be in the physical world?

我们非常相信机器人领域的垂直和水平两个方向。所以完整的垂直领域非常任务特定，从硬件到模型到数据他们都在处理。我们也相信一个生态系统，其中会有非常庞大且不同形状和尺寸的机器人硬件，世界上也会存在不同类型的开放模型。我认为它会看起来像现代机器人版本的系统集成商。在传统版本中，你是在安装产品。竞争非常激烈，利润很薄。现在有一个世界，你可以实际安装并帮助产品随着时间的推移变得更加数据驱动和高效。所以，你能让你上周安装的机器人做一些几天前它认为不可能的事情吗？它能帮助你开始扩展你的业务能力范围吗？这最终是我们希望瞄准的里程碑。

We believe very much in both the vertical and the horizontal across robotics. So the full verticals that are very task-specific, all the way from hardware to models to data they're tackling. We also believe in an ecosystem where there's going to be very vast and different shapes and sizes of hardware of robotics, and there will be different types of open models that exist in the world. I think it will look something like a modern robotics version of a systems integrator. In a traditional version, you are installing the product. It's incredibly competitive. It is tight margins. There is now a world where you can actually install and help make that product become more data-driven and productive over time. So can you actually have the bot you installed last week do something that it didn't think was possible a few days ago? Can it help you start expanding the scope of what your business is capable of doing? That's ultimately the milestone that we're hoping to aim for.

好的。但你现在会以投资者的身份告诉我。给我一些投资建议。那么你会投资通用机器人和垂直机器人两者。

Okay. But you would tell me as an investor now. Give me some investing advice. You'd invest in both the general and the vertical robots then.

是的。

Yeah.

你最喜欢的垂直机器人类别是什么？

What are your favorite vertical robot categories?

我认为最好的将是工业机器人开始。那些将是已经有很多传统机器人参与其中、任务受限的，就像我们谈到的。

I think the best ones will be the industrial bots to start. Those are going to be ones that already have a lot of traditional robotics in the mix that are task-constrained like we talked about.

比如数据中心维护机器人。

Like data center maintenance robots.

是的。比如，你不需要担心一个婴儿突然出现，并要考虑不要摔倒在他身上。所以这个世界比野外环境受限得多。我们兴奋的其他类别是早期甚至可行和可能的空间。例如，酒店中的一些方法，你可以实际开始从前台递送到酒店房间，能够实际进行一些操作来交接，这对我们来说是令人兴奋的。然后是那些人类无法做到的地方。我们在公司的一些全员会议上开的一个玩笑是，如果有一天 Mecka 能够帮助部署一个开采小行星的机器人，我们可能已经做得相当不错了。这只是一个完全不存在的工作和任务，人类无法做到，我们还不知道如何做，但这似乎是不可避免的。

Yeah. You just don't have to expect a baby to appear and have to think about that and not fall on it, for example. So the world is much more constrained than in the wild. Other categories we're excited about are just spaces that are even feasible and possible early on. And so some of the approaches in hotels, for example, where you could actually start delivering from the front desk to the hotel rooms, be able to actually have some manipulation to hand off, are exciting to us. And then places that just humans were not capable of. A joke that we make in some of our all-hands in the company would be if one day Mecka was able to help deploy a robot that mines an asteroid, we've probably done a pretty good job. It's just a job and a task that completely doesn't exist that humans can't do that we have no idea how to do just yet, but it seems inevitable.

是的，我喜欢这个。这让我想起，你知道，我甚至还没有想到星际，但你知道，就像，你听说过 Gecko Robotics 吗？

Yeah, I love that. Reminds me of, you know, even I'm not thinking interplanetary yet, but you know, like, have you heard of Gecko Robotics?

嗯，他们帮助维护船舶之类的事情。我觉得这真的很酷。

Um, what they do to help maintain ships and things like that. I think it's really cool.

绝对。你能谈谈，以及你愿意公开分享的，你们开发的硬件系列，以及实验室目前最感兴趣收集定制数据的地方吗？

Absolutely. Can you say a word on, and whatever you're willing to share publicly, the slate of hardware that you've developed and where the labs currently are most interested in collecting bespoke data?

一些迭代来自我们开始使用移动应用时。我们很快能够扩展它。那个产品有很多优点。缺点最终是你无法获得尽可能丰富的关于世界的信息。所以我们很早就必须开发的一些事情，比如最终让人们穿戴设备。在这种传感器化中，你能建造像这些配备不同摄像头和传感器的棒球帽吗？我们建造一个中央枢纽，然后连接到所有这些不同的配件，如触觉手套、外部摄像头等。所有这些都试图捕捉你如何日常与世界互动，并希望能够克隆这种行为。

Some of the iteration came from when we started with the mobile app. We quickly were able to scale that. There were a lot of virtues in that product. The downsides ultimately were you don't get as much rich information about the world as you could have. And so some of the things that we had to develop very early on were things like wayifying people ultimately. And in this sensorization, can you build things like these baseball caps that are equipped with different cameras and sensors? We build a central hub that then connects to all these different accessories such as the tactile gloves, external cameras, and more. All of which is trying to capture how do you interact with the world on a day-to-day and hopefully be able to clone that behavior.

嗯，有趣。什么是 EgoVerse？

Hmm, interesting. What is EgoVerse?

EgoVerse，所以 Dan Fay 教授、Ryan Simmer，这些是最早相信自我中心数据的研究者群体。我们有幸成为那个联盟的早期贡献者，最终是不同的人认为扩大人类数据对机器人很重要。所以我们创建并贡献了大量不同类型的自我中心数据，跨越不同的企业、行业、家庭环境。我们分享了所有的动作轨迹数据。所以所有的后处理、计算机视觉和质量门控，我们几乎贡献了所有的平台和网络工具。所以数据加载、基础设施，最终将是——我们对此的愿景是理想情况下我们能聚集许多机器人行业，因为希望我们能在这个浪潮中提升每个人，分享工具、数据和经验，只是因为作为一个行业我们落后太多才能达到那里。

EgoVerse, so Professor Dan Fay, Ryan Simmer, these are some of the earliest cohorts of researchers that believed in egocentric data. We had the privilege to be an early contributor in that consortium ultimately of different folks that think scaling up human data would be important for robotics. And so we created and contributed a large amount of different types of egocentric data across different businesses, industries, home environments. We shared all of the action trajectory data. So all of the post-processing, the computer vision and the quality gates, and we contribute all the platform and web tooling pretty much. So the data loading, the infrastructure, and so it is ultimately going to be—our vision on this is ideally we can bring together a lot of the robotics industry because hopefully we can lift up everybody in this wave and share the tooling, the data, and the lessons just because we are so far behind as an industry to get there.

可以说机器人行业已经趋同于一种趋势数据组合，即来自 YouTube 等互联网规模的数据，加上一堆自我中心数据，加上遥操作数据，然后再加上遥操作，也许有一个三角形，每个数据量。我认为 Jim Fan 在红杉活动上有一个非常好的视觉图。那是你看到的组合吗？然后在你的客户中，每个人是否都趋同于非常非常相似的组合？

Is it fair to say that the robotics industry has kind of converged on a trending data mix that is, you know, internet scale data from YouTube and the like plus a bunch of egocentric data plus teleop data, and then plus teleop, and like maybe there's like a triangle of like amount of data on each. I think Jim Fan had like a really nice visual on this at a Sequoia event. Is that kind of like the mix that you see? And then amongst your customers, has everybody kind of converged on a very, very similar mix?

有趣的是，甚至一年前，许多人会争论说：“哦，不，只有 UMI 数据才能工作。只有自我中心数据才能工作。”你慢慢意识到，正如你所说，所有这些本质上是一锅数据汤。在那锅汤里，所有的原料在某个时候都会变得重要。我认为我们看到的是在交叉——今天的配方可能是什么——上的趋同。这看起来像是某种版本的一百万小时的人类数据，跨越自我中心和不同类型的方式。

It's fun to see that even a year ago, many folks would argue and say, "Oh, no, it's only UMI data that's going to work. It's only egocentric data that's going to work." And you slowly realize, to your point, all of this is essentially a data pot of soup. And in that pot of soup, all the ingredients are going to be important at some point. I think what we're seeing is a convergence in cross—what that recipe might be today. And that does look like some version of a million hours of human data across egocentric and different types of modalities.

大概几万小时的数据，再加上一些最后一公里的遥操作数据，以及域内、同分布的数据。哪怕只做几个小时的演示，我觉得我们已经能看到这一点了。但长期来看——这也是我们为什么没有激进地只信奉一种“宗教”——像触觉数据这类东西根本还没进入这个混合里。如果大家相信这些模型最终需要学会比如朝门把手施加多大的力，那这些就必须进入这锅数据浓汤。仿真也会进入这锅数据浓汤，它们对最终收敛都很重要。

Some tens of thousands of hours of data, and then some last-mile teleoperation data, and in-domain, in-distribution data. In just a couple hours even of that demonstration, I think we are seeing that. But over time — and why we don't subscribe aggressively to even one religion here — is that things like tactile data haven't even entered the mix yet. And if there is this belief that these models ultimately will need to learn how much force it takes to apply towards the door handle, for example, these will have to come into the data pot of soup. Simulation will come into the data pot of soup, and they all will be important to converge on.

关于正在收集的这类数据，我听到最多的反对意见之一是：你可以看几百万小时的网球数据，但你依然不知道怎么打败罗杰·费德勒，对吧。我很好奇你怎么看这个说法，以及这对机器人学习方式的走向意味着什么。

One of the objections that I hear most frequently to the types of data being collected is, you know, you could watch millions of hours of tennis data and you still wouldn't know how to beat Roger Federer. Right. And I'm curious what you think of that and what it means for where we might be headed in terms of the way robots learn.

我很喜欢 AlphaGo 这个类比。说到底，当你谈机器人打网球时，你得先谈机器人下围棋。我认为在李世石那场比赛以及 DeepMind 最初做的所有工作里，第一步是对不同专家下围棋做大规模行为克隆。下一步是自我对弈版本——在给定棋局规则的情况下，你能否最终模拟出所有不同的走法，直到走出第 37 手那样的神来之笔，而那一步远远超出了人类在那个时刻对棋盘的想象。但对我们来说，在机器人这个场景里——围棋的规则就是你要运行于其中的整个世界的边界。机器人领域有物理世界那种极其丰富的细节，而我们今天并没有能重建这一切的世界模拟器。所以要弥合这个差距，今天大量的自我中心数据，一是帮我们打好基础物理的根基，以及你必须达到的基础行为克隆。它同时也在帮助塑造这个世界模拟器可能的样子。然后在这条路上的某个地方，我认为我们能够造出像罗杰·费德勒那样打网球的机器人。

I love the analogy of AlphaGo. Ultimately, when you talk about robots playing tennis, you've got to talk about robots playing Go. And I think with Lee Sedol and in all the work that DeepMind did at the very beginning, it was large-scale behavior cloning of different experts playing Go. Then the next step was the self-play version — with the rules of the game, can you ultimately simulate all the different ways until you come up with the magic of move 37, ultimately, which was so out of scope of what humans could have thought in that moment on the Go board. For us, though, in our case of robotics — for Go, the rules of the game are the entire circumference of the world that you will operate in. In robotics you have the incredibly rich detail that the physical world has, and we ultimately don't have a world simulator that can recreate that today. And so to close that gap, a lot of the egocentric data today is to help us ground the basic physics and the basic behavior cloning that you have to get to. It is also contributing to creating what this world simulator might look like. And then somewhere along the way, I think we will be able to create and have robots playing tennis like Roger Federer.

你觉得它们主要会在仿真里做自我对弈，还是在真实世界里？

Do you think they'll mostly be doing self-play in simulation or in the real world?

我觉得最终会是混合的。如果我们能把仿真做起来，不必立刻把世界完全按细节重建出来——但我认为自我对弈既会发生在物理世界，也会发生在仿真里。

I could see a hybrid ultimately. I think if we can get simulation and we don't have to fully reconstruct the world to its detail right away — but I see self-play both in the physical and in simulation.

你刚才几次用到“世界模拟器”这个词。你怎么看世界模型的进展？我猜现在这个词和视频模型有点被混着用。想听听世界模型以及它们对机器人数据训练混合的贡献。

You use the word world simulator a couple times there. What are your thoughts on what's happening with world models, which I guess are being used somewhat interchangeably with video models as a term right now? But world models and their contribution to the robot data training mix.

我认为我们刚才谈的自我对弈所蕴含的潜力——甚至只是去“梦见”杯子掉下去摔碎是什么样，而不用真的去做——对可能发生的事来说非常有价值。我看到的价值在于，我们最终能利用互联网和 YouTube 上的大量工作来做像素预测。如果我们能把它翻译回动作轨迹该是什么样，那就能帮我们加速不少。

I think the potential and possibility to have what we just talked about in terms of self-play — to even just dream what it means to drop a cup and have it shatter without actually doing it — is incredibly valuable for what could happen. I see the value in that we get to leverage a lot of the work from the internet and YouTube, for example, in this pixel prediction ultimately. And if we can translate that back into what these action trajectories would be, it could help us accelerate quite a bit.

嗯，有意思。你提到我们差不多收敛到了这个数据混合的金字塔。你觉得只要把它放大，这个金字塔就能扩展吗？你觉得那能带我们到应许之地吗？对我来说，应许之地是——大家喜欢说机器人界的 GPT-3 时刻。我不再喜欢这个说法了。我只想知道，机器人什么时候能真的帮我叠衣服、给我做顿饭？你觉得当前的数据混合能带我们到那一步，还是需要算法和数据上的新突破才能到那儿？

Yeah. Yeah. Interesting. So you mentioned that we've kind of converged on this pyramid of data mix. Do you think this pyramid will scale if we just scale it up? Do you think that takes us to the promised land? And to me, promised land is — people like saying GPT-3 moment for robotics. I no longer like that. I just like, when will a robot actually fold my clothes or cook a meal for me? Do you think the current data mix will get us to that point, or do you think new breakthroughs in algos and data are going to be needed to get there?

我认为如果我们按那个比例把它放大一个数量级，沿途就能解决很多任务。其中一些任务可能是在约束强得多的环境里，可能是业务流程任务，你给的非常具体的标准作业流程。在那个理想世界里，我们有一个即插即用的机器人，它看一眼就能解决任何任务，开箱就能像罗杰·费德勒那样打网球——在真实世界场景的所有疯狂复杂性里，比如天气——我觉得那还得走一段路，而且我不认为最终会是那个样子。

I think if we scale it up some order of magnitude in that ratio, we will solve a lot of tasks along the way. So some of those tasks could be a lot more constrained environments. They could be within business operation tasks, very specific SOPs you give. In that ideal world where we have a drop-in robot and it solves any task by just looking at it, can play tennis just like Roger Federer out of the box — in all of the crazy complexities of the actual world scenarios, like the weather for example — I think it will be some way, and I don't think this is what it will look like ultimately.

那会是什么样子？

What will it look like?

嗯，我们往这个混合里要加的东西还多着呢。我认为关键在于：我们怎么开始把其他模态引入这个混合？这就是触觉数据。我很喜欢很多模型正收敛到一种全模态模型的思路——你可以喂给它任何类型的输入作为观测，不管是力、像素，极端情况下甚至是气味——然后得到某种输出，不管是动作、语音，还是不同类型的控制。

Well, we still have so much to add into the mix still. And I think that is going to be: how can we start introducing other modalities into the mix? This is going to be that tactile data. I like the approach a lot of the models converging towards an omni-model approach, where you can feed it any type of input that is the observation — whether it is force, pixel, even smell in a very extreme case — and then have some sort of output, whether it's action, voice, or different types of controls.

现在真的有能感知气味的芯片吗？

Are there even chips that can sense smell?

还很早期。嗅觉这条路非常早期。我愿意想象世界上会出现一种方法把它解锁。

It's pretty early. This olfaction approach is very early. I like to imagine that there's going to be an approach in the world where that gets unlocked.

嗯。你知道，GPT-6 Astra 的结果在机器人任务上泛化得非常好。我很好奇你怎么看这件事，以及这对所有这些纯机器人公司意味着什么。它们最终会不会就被 AGI 实验室碾压，而“苦涩的教训”也会把机器人领域一并碾过去？你怎么看？

Yeah. The results of, you know, GPT-6 Astra is generalizing incredibly well to robot tasks. I'm curious what you make of that and what it means for all these pure-play robotics companies. Like, are they ultimately just going to get clobbered by the AGI labs, and the bitter lesson is just going to steamroll robotics as well? What do you think?

在信息不完美的情况下把数据规模做上去，需要很大的勇气，而我认为这些实验室里有很多天生就有这种勇气，能在早期承担这件事。我认为我们正在看到、也令人兴奋的是：随着机器人数据规模扩大，甚至只是把它加进这些主流 LLM 和基础模型的预训练混合里，你确实可以开始在大量不同领域里泛化。我对全模态模型的思路很兴奋——你可以把很多不同类型的观测，不管是力还是动作轨迹，引入这个混合，让它泛化。我认为也会有很多论点是：你希望比一个能做任何事的完全泛化模型更加垂直。所以在那条时间线的某个位置，那个什么都能做的模型会在很多事情上非常出色。但其中一些专用模型、一些更有针对性的机器人基础模型，可能在某些任务上会变得非常优秀——达到四个九的能力和可靠性。

It takes a lot of courage to scale up data with imperfect information, and I think a lot of these labs have the DNA in themselves to take on that courage early on. I think what we're seeing, and what's exciting, is that with the scale-up of robotics data and even just adding it into a lot of the pre-training mixes of these main LLMs and these foundation models, it really is that you can start to generalize in a lot of different domains. I'm excited about the omni-model approaches where you can introduce a lot of different types of observation — whether that's force or whether that's action trajectories — into the mix and allow it to generalize. I think there are also going to be a lot of arguments where you want to be much more vertical than a fully generalized model that can do anything. So somewhere along that time horizon, that model that can do everything is going to be very good at a lot of things. But some of these specialized models, some of these more targeted robotics foundation models, might become very excellent — into four nines of capability and reliability — in some of these tasks.

我刚才从字里行间听出来的意思是，最新的基础模型在机器人领域表现出色，其实并不那么令人意外，部分原因是现在训练数据里确实混入了相当多的机器人数据。这样说对吗？

What I heard reading between the lines just now was that the fact that the latest foundation models are great at robotics is not actually that surprising, in part because there's actually quite a bit of robot data that's been thrown into the training mix now. Is that fair?

仔细想想，这其实相当符合预期。

It's actually quite expected when you think about it.

非常有意思。我想最后回到你的科幻情结来收尾。你创办这家公司，是因为你能想象一个到处都是机器人的世界。请描绘一下你心目中那个世界的样子。

Very interesting. I'd love to close by going back to your sci-fi roots. You started this company because you could imagine a version of the world with robots everywhere. Paint a picture of what you think the world looks like.

我觉得最搞笑的时刻，大概是你们来我们纽约办公室的时候，我们主会议室墙上挂着的就是川崎 Corleo。

I think the funniest moment was probably when you visited our New York office, and in our main conference room we have hung up on the wall the Kawasaki Corleo.

我太喜欢了。

I love that.

也就是——

Which is—

那个机器豹狗马，管它是什么呢。

The robot panther dog horse, whatever it is.

机器豹狗马。没错。所以它基本上就是一只机器豹狗马。而且那是你看过的最荒诞的视频和预告片。差不多吧。

The panther dog horse. Exactly. So it's pretty much a robot panther dog horse. And it's the most ridiculous video and trailer you've ever seen. Pretty much.

我最喜欢的部分是，你们居然让 Anas 相信上面骑的就是你和你妈妈。

My favorite part is that you guys convinced Anas that it was actually you and your mom on it.

对，对。有个片段基本上就是一位奇幻的亚洲女士和她儿子骑着马穿过绿色草地、越过一条小河。而 Corleo，这只马狗豹机器人，正跃过那条河。我们就跟 Anas 说，这基本上就是我们小时候，是我们生成的。但妙就妙在它有多荒诞。我非常喜欢《银河系漫游指南》的哲学，世界可以有多荒诞，未来可以有多荒诞。在这个充满机器人的世界里，我很愿意相信会有那种疯狂的企业家和人们，相信这样的形态会很有意思。我认为机器人的形状和尺寸会呈现这样一个世界：我们想出各种古怪的机器人，能做你甚至还没想到的任何事情。

Yeah. Yeah. There's a clip pretty much where this fantastical Asian lady and her son are riding this green meadow over a little river. And the Corleo, this horse dog panther robot, is jumping over the river. And so we told Anas this was pretty much us when we were kids, and we generated it. But what is wonderful is how absurd it is. I very much love the philosophy of Hitchhiker's, in just how absurd the world could be and how absurd the future could be. And in this world that is full of robots, I would love to believe that there's going to be such crazy entrepreneurs and people that believe that form factors like this would be interesting. And I think that the shape and size of robotics will look something like a world where we come up with wacky robots that could do anything that you haven't even thought of yet.

我喜欢这个。

I love that.

比如开采小行星。

Like mining asteroids.

数量级上：十年后世界上会有多少机器人？

Order of magnitude: how many robots will there be in the world in a decade?

比人口多很多很多倍。

Many, many times more than there are people.

比人口多好几个数量级。

Many orders of magnitude more than there are people.

是的。十年以上之后。

Yes. In a decade plus.

一千亿个机器人。

100 billion robots.

对。

Yeah.

那真是太酷了。一千亿只豹狗马。

That's really cool. 100 billion panther dog horses.

豹狗马。至少还要再多一个数量级。

Panther dog horses. At least another order of magnitude more than that.

太喜欢了。太棒了，Josh。我觉得你们在做的事情在运营上真的令人印象深刻，你们扩大数据收集规模的方式非常酷，为这个机器人黄金时代和你从小就梦想的科幻未来铺路。感谢你今天来参加播客，既讲了你怎么做这门生意，也讲了你如何看待未来。真的很有意思。

Love it. Awesome, Josh. I think that what you're building is operationally really impressive, and very cool how you are scaling up the data collection efforts to enable this robotics golden age and this sci-fi future that you've dreamed of since you were a kid. So thank you for joining the pod today to tell us both how you're building the business and how you see the future. This is really fun.

非常感谢邀请我们。谢谢你，Sonya。

Thank you so much for having us. Thank you, Sonya.

其实我们给你带了一份礼物。

So we actually brought you a present.

我喜欢礼物。

I love presents.

这对我们来说有点像回到原点的时刻。这是我想的那个东西吗？

This is kind of like a full-circle moment for us. Is this what I think it is?

谁知道呢？谁知道呢？好吧，你得在播客上把它打开。

Who knows? Who knows? All right, you're gonna have to open it up on the podcast.

好，这太棒了。看看这个——等等。我们能不能——你知道，我可以当那种做开箱视频的 YouTuber。看，我喜欢这个。Mecka 贴纸是 A+。好，我觉得这就是我想的那个东西。看看。我们其实要推出一系列 Mecka 手办、贴纸——

Okay, this is amazing. And look at the—hold on. Can we—you know, I could be one of those YouTubers that does the unboxings. Look, I love this. The Mecka stickers are A+. Okay, I think this is what I think it is. Let's see. We're gonna actually come up with a series of Mecka figurines, stickers—

因为它们都是巨型机器人、人类驾驶员那类东西。

Because they're all like giant robots, human pilot stuff.

那会很有意思。

That'll be fun.

我太兴奋了。看看今天这个能不能用。我喜欢。我的天。好，这是——

I'm so excited. Let's see if this is working today. I love it. Oh my gosh. Okay, this is the—

这是 Corleo。

This is the Corleo.

这是川崎 Corleo。我喜欢这个。这就是那只机器狗豹马。

This is the Kawasaki Corleo. I love this. This is the robot dog panther horse.

这是我和我妈妈骑在豹马上。没错。

This is me and my mom on the panther horse. That's right.

这个要——我们要么把它放在播客间，要么放在我办公室。这太不可思议了。我太开心了。

This is going—we're either going to put it in the podcast room or in my office. This is incredible. I'm so happy.

就这么办。太荒诞了。

Do it. So absurd.

我太开心了。谢谢。

I'm so happy. Thank you.