从 AlexNet 到世界模型：计算扩展与空间智能

我认为深度学习的历史在某种意义上就是算力规模扩张的历史。

I think the whole history of deep learning is in some sense the history of scaling up compute.

大家好，欢迎收听 Len Space 播客。我是 Colonel Labs 的创始人 Allesio，和我一起的是 Blade in Space 的编辑 Swix。我们非常激动能邀请到 World Labs 的 Fei-Fei 和 Justin 来到演播室。欢迎。

Hey everyone, welcome to the Len Space podcast. This is Allesio, founder of Colonel Labs, and I'm joined by Swix, editor of Blade in Space. We are so excited to be in the studio with Fei-Fei and Justin of World Labs. Welcome.

我们也很兴奋。

We're excited too.

我差点说成 Marble 了。

I nearly said Marble.

谢谢邀请。我觉得大家对世界模型很感兴趣，你们也围绕空间智能做了一些宣传。我想你们很少有机会讲述你们是如何走到一起创办 World Labs 的。

Yeah, thanks for having us. I think there's a lot of interest in world models and you've done a little bit of publicity around spatial intelligence. I guess one part of the story that is a rare opportunity for you to tell is how you two came together to start building World Labs.

这很简单，因为 Justin 是我的前学生。Justin 在 2012 年那个季度加入我在斯坦福的实验室，正好是 AlexNet 问世的时候。他是我最优秀的学生之一。之后他在密歇根大学担任教授，后来在 Meta 工作，早期职业生涯非常成功。大约两年前，我们俩都独立地观察大模型的发展，思考语言模型之后是什么。构建世界模型和空间智能的想法对我们来说很自然。于是我们开始讨论，决定把所有精力都投入进去，专注于解决这个问题，共同创办了 World Labs。

That's very easy because Justin was my former student. Justin came to my lab at Stanford in 2012, the same quarter that AlexNet came out. He was one of my very best students. He went on to have a successful early career as a professor at University of Michigan and then at Meta. More than two years ago, both of us independently looked at the development of large models and thought about what's beyond language models. The idea of building world models and spatial intelligence was natural for us. So we started talking and decided to put all the eggs in one basket and focus on solving this problem, and started World Labs together.

是的，差不多。在博士期间经历了 ImageNet 时代后，我觉得计算机视觉的下一个十年将是把 AI 从数据中心带到现实世界。所以我的兴趣转向了 3D 视觉、计算机图形学和生成建模。我原以为博士毕业后会与导师渐行渐远，但几年后重聚时，发现她也在思考非常相似的事情。

Yeah, pretty much. After seeing the ImageNet era during my PhD, I had the sense that the next decade of computer vision would be about getting AI out of the data center and into the world. So my interests shifted into 3D vision, computer graphics, and generative modeling. I thought I was drifting away from my advisor post-PhD, but when we reunited a couple years later, it turned out she was thinking of very similar things.

如果想想 AlexNet，核心要素是 ImageNet、转向 GPU 和神经网络。你认为世界模型的 AlexNet 等价物是什么？这个想法一直存在，Yann LeCun 是最著名的倡导者。过去两年你看到了什么，让你觉得现在是时候了？在数据、算法或算力方面，你们想从根本上构建什么来让世界模型成为现实？

If you think about AlexNet, the core pieces were ImageNet, the move to GPUs, and neural networks. How do you think about the AlexNet equivalent for world models? It's an idea that has been out there, with Yann LeCun as a prominent proponent. What have you seen in the last two years that made you think now is the time? And what fundamentally do you want to build in terms of data, algorithms, or compute to make world models come to life?

一个因素是，总体上可用的数据和算力更多了。深度学习的历史在某种意义上就是算力规模扩张的历史。从 AlexNet 到今天，每张卡的性能提升了约一千倍，现在在成百上千甚至上万张 GPU 上训练模型很常见。我们今天能调动的算力大约是我博士开始时的百万倍。语言模型在过去几年开始表现很好。但当我们转向视觉、空间和世界数据时，需要处理的数据量要大得多。我认为这是吸收新算力的好方法。

One thing is that there is a lot more data and compute generally available. The whole history of deep learning is in some sense the history of scaling up compute. From AlexNet to today, we're getting about a thousand times more performance per card, and now it's common to train models on hundreds or thousands or tens of thousands of GPUs. The amount of compute we can marshal today is about a millionfold more than at the start of my PhD. Language started working quite well in the last couple of years. But as we move towards visual, spatial, and world data, we need to process a lot more. I think that's a good way to soak up the new compute coming online.

公开挑战赛的模式仍然有效，还是应该集中在实验室内部？

Does the model of having a public challenge still work, or should it be centralized inside a lab?

我认为开放科学仍然重要。AI 已经从一个小众的计算机科学学科演变为一项文明级技术。例如，我的斯坦福实验室最近发布了一个开放数据集和基准测试 BEHAVIOR，用于在模拟环境中评估机器人学习。这是保持开放科学模式的一个明确努力，尤其是在学术界。但重要的是要认识到生态系统是混合的。工业界许多专注的工作以产品形式面世，而不是公开挑战赛。

I think open science is still important. AI has evolved from a niche computer science discipline to a civilizational technology. For example, my Stanford lab recently announced an open dataset and benchmark called BEHAVIOR for benchmarking robotic learning in simulated environments. That's a clear effort to keep up the open science model, especially in academia. But it's important to recognize the ecosystem is a mixture. A lot of focused work in industry sees the daylight in the form of a product rather than an open challenge.

是的。这仅仅是资金和商业模式的问题。你必须从中看到一些投资回报。

Yeah. And that's just a matter of funding and the business model. You have to see some ROI from it.

我认为这只是生态系统多样性的问题。即使在所谓的 AlexNet ImageNet 时代，也有封闭模型、专有模型和开放模型。想想 iOS 和 Android，它们有不同的商业模式。我不认为这仅仅是资金问题；这只是市场的运作方式。有不同的玩法。

I think it's just a matter of the diversity of the ecosystem. Even during the so-called AlexNet ImageNet time, there were closed models, proprietary models, and open models. If you think about iOS versus Android, there are different business models. I wouldn't say it's just a matter of funding per se; it's just how the market is. There are different plays.

但你觉得今天在那些实验室的商业压力下，你还能重做 ImageNet 吗？对我来说，这是最大的问题。就是什么可以开放，什么应该保留？如果我是你，你筹集了很多资金，正在构建这一切。如果你有最好的数据集，你有多大动力去发布它？感觉实验室里的人越来越被吸引，博士项目也越来越早地被拉进这些实验室。所以我想知道，你是否认为资金过多带来了问题，给更学术化的开放研究空间带来了压力，或者你觉得这并不值得担忧。

But do you feel like you could redo ImageNet today with the commercial pressure that some of these labs have? To me, that's the biggest question. It's like what can you open versus what should you keep inside? If I put myself in your shoes, you raise a lot of money, you're building all of this. If you had the best dataset for this, what incentives do you really have to publish it? It feels like people at the labs are getting more and more pulled in, and PhD programs are getting pulled earlier into these labs. So I'm curious if you think there's an issue with how much money has taken, how much pressure it puts on the more academic open research space, or if you feel that's not really a concern.

我确实有担忧，但更多是关于资源以及学术界资源分配不均的问题。这和 World Labs 的对话略有不同。我一直在倡导为一个健康的生态系统提供资源。作为斯坦福大学以人为本人工智能研究所的创始主任和联合主任，我一直与政策制定者合作，为公共部门和学术界的 AI 工作提供资源。我们与第一届特朗普政府合作了一项名为《国家人工智能研究资源法案》的法案，该法案正在规划一个国家 AI 计算云以及一个数据存储库。我也认为开源和开放数据集仍然是生态系统的重要组成部分。现在在我的斯坦福实验室，我们正在做一个关于机器人学习的开放数据集和开放基准，叫做 Behavior，我的许多同事也还在做这些。我认为这是生态系统的一部分。我认为行业正在做的事情，一些初创公司快速开发模型并创造产品，也是一件好事。例如，当 Justin 还是我的博士生时，没有一个计算机视觉程序运行得那么好。我们可以写出漂亮的论文。

I do have concerns, less about the pressure. It's more about the resourcing and the imbalanced resourcing of academia. This is a little bit of a different conversation from World Labs. I have been advocating for resourcing a healthy ecosystem. As the founding director and co-director of Stanford's Institute for Human-Centered AI, I've been working with policymakers about resourcing public sector and academic AI work. We worked with the first Trump administration on a bill called the National AI Research Resource (NAIRR) bill, which is scoping out a national AI compute cloud as well as a data repository. I also think that open-source and open datasets continue to be an important part of the ecosystem. Right now in my Stanford lab, we are doing an open dataset and open benchmark on robotic learning called Behavior, and many of my colleagues are still doing that. I think that's part of the ecosystem. I think what the industry is doing, some startups are running fast with models creating products, is also a good thing. For example, when Justin was a PhD student with me, none of the computer vision programs worked that well. We could write beautiful papers.

实际上，甚至在研究生院之前，我就想做计算机视觉，我联系了谷歌的一个团队。我想本科毕业后就去尝试做计算机视觉，他们告诉我：‘你在说什么？你不能那样做。先去读个博士再回来。’

Actually, even before grad school, I wanted to do computer vision and I reached out to a team at Google. I wanted to potentially go and try to do computer vision out of undergrad, and they told me, 'What are you talking about? You can't do that. Go do a PhD first and come back.'

是什么动力让你这么做的？

What was the motivation that got you?

哦，我在本科期间做过一些计算机视觉研究，实际上是和 Fei-Fei 的博士导师一起做的。

Oh, I had done some computer vision research during my undergrad with actually Fei-Fei's PhD advisor.

有传承。

There's a lineage.

这里有传承。所以我在本科时就做过一些计算机视觉，我觉得这很酷，想继续做下去。所以我在本科毕业时就面临了这种行业与学术的选择，我认为现在研究界的很多人也面临同样的选择。但回到你的问题，我认为学术界的作用，尤其是在 AI 领域，在过去十年中发生了很大变化。这不是坏事。这是因为技术已经成长并涌现出来。五到十年前，你确实可以在实验室里只用几块 GPU 就训练出最先进的模型。但由于这项技术如此成功，规模扩大了很多，你不能再只用几块 GPU 训练最先进的模型了。这不是坏事。这是好事。这意味着技术真的起作用了。但这意味着我们作为学者应该做什么的期望有所转变。不应该是试图训练最大的模型，扩大最大的东西。而应该是尝试古怪的想法、新想法和疯狂的想法，其中大部分都不会成功。我认为那里有很多事情可以做。如果说有什么担心的，那就是我担心学术界太多人过于专注于这种假装我们可以训练最大模型的想法，或者把它几乎当作一个职业培训项目，然后毕业去大实验室玩所有的 GPU。我认为你可以做很多疯狂的事情，比如新算法、新架构、新系统。一个人可以做很多事情。

There's a lineage here. So I had done some computer vision even as an undergrad and I thought it was really cool and I wanted to keep doing it. So then I was faced with this sort of industry-academia choice even coming out of undergrad that I think a lot of people in the research community are facing now. But to your question, I think the role of academia, especially in AI, has shifted quite a lot in the last decade. And it's not a bad thing. It's because the technology has grown and emerged. Five or ten years ago, you really could train state-of-the-art models in the lab with just a couple GPUs. But because that technology was so successful and scaled up so much, you can't train state-of-the-art models with a couple GPUs anymore. And that's not a bad thing. It's a good thing. It means the technology actually worked. But that means the expectations around what we should be doing as academics shifts a little bit. It shouldn't be about trying to train the biggest model and scaling up the biggest thing. It should be about trying wacky ideas and new ideas and crazy ideas, most of which won't work. I think there's a lot to be done there. If anything, I'm worried that too many people in academia are hyperfocused on this notion of trying to pretend like we can train the biggest models or treating it as almost a vocational training program to then graduate and go to a big lab and play with all the GPUs. I think there's just so much crazy stuff you can do around new algorithms, new architectures, new systems. There's a lot you can do as one person.

而且，学术界在理解这些大型模型的理论基础方面也有作用。我们对此仍然知之甚少，或者延伸到跨学科领域。Justin 称之为古怪的想法。有很多基础科学想法。有很多蓝天问题。所以我同意。我不认为问题是开放与封闭、产品化与开源。我认为现在的问题是学术界本身资源严重不足。所以研究人员和学生没有足够的资源去尝试这些想法。

And also, academia has a role to play in understanding the theoretical underpinnings of these large models. We still know so little about this, or extend to the interdisciplinary. Justin calls them wacky ideas. There's a lot of basic science ideas. There's a lot of blue sky problems. So I agree. I don't think the problem is open versus closed, productization versus open sourcing. I think the problem right now is that academia by itself is severely underresourced. So that the researchers and the students do not have enough resources to try these ideas.

是的。为了让人们能深入探讨，当你谈到古怪想法时，你想到的一个古怪想法是什么？

Yeah. Just for people to nerd snipe, what's a wacky idea that comes to mind when you talk about wacky ideas?

哦，我有一个想法，我一直向我在密歇根的学生们推销，那就是我非常喜欢硬件，我喜欢新出现的各种硬件。在某种意义上，我们今天使用的神经网络和 Transformer 实际上是基于矩阵乘法，因为矩阵乘法非常适合 GPU。但如果我们考虑 GPU 将如何扩展，硬件未来可能如何扩展，我不认为我们目前的系统，比如 GPU 硬件设计，会无限扩展。我们现在甚至开始看到，计算单元不再是单个设备。而是整个设备集群。所以如果你想象一个节点，一个完整的集群，但我们谈论神经网络的方式仍然好像它们是一个单一的整体，可以在一个 GPU 上用 PyTorch 编码，但实际上它们可以分布在数千个设备上。

Oh, I had this idea that I kept pitching to my students at Michigan, which is that I really like hardware and I really like new kinds of hardware coming online. In some sense, the emergence of the neural networks that we use today and transformers are really based around matrix multiplication because matrix multiplication fits really well with GPUs. But if we think about how GPUs are going to scale, how hardware is likely to scale in the future, I don't think the current system that we have, like the GPU hardware design, is going to scale infinitely. We start to see that even now, the unit of compute is not the single device anymore. It's this whole cluster of devices. So if you imagine a node, a whole cluster, but the way we talk about neural networks is still as if they are a monolithic thing that could be coded like in one GPU in PyTorch, but in practice they could distribute over thousands of devices.

那么，就像 Transformer 基于矩阵乘法，而矩阵乘法是在 GPU 上效果很好的原语，随着硬件规模扩张，是否有其他原语更适合大规模分布式系统，我们可以用来构建神经网络？我认为有可能出现截然不同的架构，适应未来 10 到 20 年的下一代硬件。我们今天就可以开始想象，但很难下这种赌注，因为还有硬件彩票的概念。比如说英伟达赢了，我们就应该无限扩展它，然后写软件来弥补任何缺口，对吧？

So are there, just as transformers are based around matmul and matmul is the primitive that works really well on GPUs as hardware scales out, are there other primitives that make more sense for large-scale distributed systems that we could build our neural networks on? And I think it's possible that there could be drastically different architectures that fit with the next generation of hardware that's going to come 10 or 20 years down the line. We could start imagining that today, but it's really hard to make those kinds of bets because there's also the concept of the hardware lottery. Let's just say Nvidia has won and we should just scale that out infinitely and write software to patch up any gaps we have in the mix, right?

是也不是。如果你看数据，即使从 Hopper 到 Blackwell，每瓦性能也差不多。他们主要是增加了晶体管数量、芯片尺寸和功耗。但即便从 Hopper 到 Blackwell，我们在每瓦性能上已经看到了 Scaling 极限。所以我认为还有空间去做些新东西。我不确定具体是什么，也不认为作为初创公司能在三个月内完成，但如果你花几年时间深入研究，也许能取得一些突破。我认为这种长期研究非常适合学术界。

Yes and no. If you look at the numbers, even going from Hopper to Blackwell, the performance per watt is about the same. They mostly make the number of transistors go up, the chip size go up, and the power usage go up. But even from Hopper to Blackwell, we're already seeing a scaling limit in terms of performance per watt. So I think there is room to do something new. I don't know exactly what it is, and I don't think you can get it done in a three-month cycle as a startup, but that's the kind of idea that if you sit down and sit with for a couple years, maybe you could come up with some breakthroughs. I think that's the kind of long-range stuff that is a perfect match for academia.

回到一点历史背景，你们做过场景叙事或更新的图像描述工作，是和 Andrej 一起做的。我想听你们讲讲那个故事，关于你开始做博士研究，以及 Fei 当时的反应。

Coming back to a bit of background in history, we have this sort of research note on the scene storytelling work that you did, or newer image captioning that you did with Andrej. I just wanted to hear you guys tell that story about you embarking on that for your PhD and Fei having that reaction that you had.

是的。我认为这项工作始于我和 Andrej，然后 Justin 加入了。Andrej 开始读博。他和我当时在探索 ImageNet 物体识别之外的东西。那时，卷积神经网络已经在 ImageNet 任务中证明了其能力，所以 ConvNet 是表示图像的好方法。同时，在语言领域，一种早期的序列模型 LSTM 也在被实验。Andrej 和我讨论过这个。这是我长期以来的梦想。我曾以为需要一百年才能解决：讲述图像的故事。我研究生毕业时，真的以为我整个职业生涯都会致力于解决这一个问题：给定一张图片或一个场景，用自然语言讲述故事。但事情发展得太快了。当 Andrej 开始时，我们想也许结合 ConvNet 的表示和 LSTM 的语言序列模型，我们可以通过训练来学习将描述与图像匹配。于是我们开始了这项工作。我记得是 2014 或 2015 年。那篇描述论文发表在 CVPR 2015 上。那是我们的第一篇论文，Andrej 让它成功了：给定一张图像，用 ConvNet 表示图像，语言模型是 LSTM，我们结合它们，就能生成一个句子。那是首批成果之一。我在书里写过，我们以为我们是第一个做的，但结果谷歌当时也在同时做。一位《纽约时报》的记者 John Markoff 正在报道谷歌的故事，但他偶然听说了我们，意识到我们确实独立地同时取得了成果。于是他同时报道了谷歌的研究以及 Andrej 和我的研究。在那之后，我想 Justin 当时已经在实验室了。

Yeah. So I think that line of work started between me and Andrej, and then Justin joined. Andrej started his PhD. He and I were looking at what is beyond ImageNet object recognition. At that time, the convolutional neural network had proven some power in ImageNet tasks, so ConvNet is a great way to represent images. In the meantime, in the language space, an early sequential model called LSTM was also being experimented. Andrej and I were just talking about this. It has been a long-term dream of mine. I thought it would take a hundred years to solve: telling the story of images. When I graduated from grad school, I really thought the rest of my entire career would be towards solving that single problem: given a picture or a scene, tell the story in natural language. But things evolve so fast. When Andrej started, we thought maybe combining the representation of ConvNet as well as the language sequential model of LSTM, we might be able to learn through training to match captions with images. So that's when we started that line of work. I think it was 2014 or 2015. The captioning paper was at CVPR 2015. It was our first paper that Andrej got to work: given an image, the image is represented with ConvNet, the language model is the LSTM, and we combine it and it's able to generate one sentence. That was one of the first times. I think I wrote in my book that we thought we were the first people doing it, but it turned out that Google at that time was also simultaneously doing it. A reporter, John Markoff from the New York Times, was breaking the Google story, but he by accident heard about us and realized that we really independently got there together at the same time. So he wrote the story of both the Google research as well as Andrej and my research. But after that, I think Justin was already in the lab at that time.

是的。我记得那次组会，Andrej 在展示一些结果，解释这个叫 LSTM 和 RNN 的新东西，我之前从未听说过。我想，哇，这真是太棒了，我想做这个。然后他在 CVPR 2015 上发表了第一篇图像描述论文。之后，我们开始合作。我们首先在 2015 年做了一篇纯粹关于语言建模的论文。回想起来，我本应该坚持做语言建模的，那后来变得相当赚钱。但我们一起做了这篇语言建模论文，我和 Andrej，在 2015 年。那真的很酷。我们训练了这些小型 RNN 语言模型，它们一次能吐出几个句子，我们戳弄它们，试图理解神经网络内部的神经元在做什么。

Yeah. I remember the group meeting where Andrej was presenting some of those results and explaining this new thing called LSTMs and RNNs that I had never heard of before. I thought, wow, this is really amazing stuff, I want to work on that. So then he had the paper at CVPR 2015 on the first image captioning results. Then after that, we started working together. We first did a paper actually just on language modeling back in 2015. I should have stuck with language modeling; that turned out to be pretty lucrative in retrospect. But we did this language modeling paper together, me and Andrej, in 2015. It was really cool. We trained these little RNN language models that could spit out a couple sentences at a time and poke at them and try to understand what the neurons inside the neural network were doing.

你们当时在做关于不同记忆之类的分析……

You guys were doing analysis on the different like memory and...

是的。那真的很酷。即使在那个时候，我们就有这样的结果：你可以观察 LSTM 内部，说，哦，这东西在读代码。所以我们训练的一个数据集是 Linux 源代码，因为它是开源的，你可以直接下载。我们在这个数据集上训练了一个 RNN，当网络试图预测 token 时，我们试图将它的预测与 RNN 的内部结构关联起来。我们找到了一些相关性：哦，LSTM 这一层的这个单元在遇到左括号时激活，遇到右括号时关闭。我们做了诸如此类的实证研究来弄清楚。那很酷，这相当于从语言建模部分中切掉 CNN，只孤立地看语言模型。

Yeah. It was really cool. And even at that time, we had these results where you could look inside the LSTM and say, oh, this thing is reading code. So one of the datasets we trained on was the Linux source code, because it's open source and you could just download it. So we trained an RNN on this dataset, and as the network is trying to predict the tokens, we tried to correlate the kinds of predictions it's making with the internal structures in the RNN. We were able to find some correlations: oh, this unit in this layer of the LSTM fires when there's an open parenthesis and turns off when there's a closed parenthesis. We did some empirical stuff like that to figure it out. So that was pretty cool, and that was sort of cutting out the CNN from this language modeling part and just looking at the language models in isolation.

但后来我们想扩展图像描述工作。我记得那时我们甚至有了空间感，因为我们觉得描述没有捕捉到图像的不同部分。所以我和 Justin、Andrej 讨论，能不能做我们后来称为密集描述的东西，即更详细地描述场景，尤其是场景的不同部分。

But then we wanted to extend the image captioning work. I remember at that time we even had a sense of space, because we felt like captioning does not capture different parts of the image. So I was talking to Justin and Andrej about can you go what we ended up calling dense captioning, which is describe the scene in greater detail, especially different parts of the scene.

是的，然后我们构建了这个系统。第二年，在 CVPR 2016 上，我和 Andrej、Fei 发表了一篇论文，构建了这个做密集描述的系统。

Yeah, and so then we built this system. It was me, Andrej, and Fei on a paper the following year at CVPR 2016, where we built this system that did dense captioning.

你输入一张图片，它就会在图片中所有有趣的东西周围画框，然后为每个框写一小段描述。比如，‘桌子上有一个绿色水瓶’，‘一个穿黑衬衫的人’。这是一个非常复杂的神经网络，因为它建立在当时目标检测领域的许多进展之上，而目标检测在很长一段时间里都是计算机视觉的主要课题。实际上，它是一个联合神经网络，既要学习观察单个图像——因为网络内部有三种不同的表示：一种是整个图像的表示，用来获取整体情况；然后它会提出想要关注的各个区域，并独立地表示每个区域；一旦你关注了某个区域，就需要为每个区域输出文本。所以这是一个相当复杂的神经网络架构。这一切都发生在 PyTorch 出现之前。

So you input a single image and then it would draw boxes around all the interesting stuff in the image and then write a short snippet about each of them. It's like, oh, it's a green water bottle on the table. It's a person wearing a black shirt. And this was a really complicated neural network because it was built on a lot of advancements that had been made in object detection around that time, which was a major topic in computer vision for a long time. And then it was actually one joint neural network that was both learning to look at individual images because they actually had three different representations inside this network. One was the representation of the whole image to kind of get the gestalt of what's going on. Then it would propose individual regions that it wants to focus on and then look at each region independently and then once you look at the region then you need to output text for each region. So that was a pretty complicated neural network architecture. This was all pre-PyTorch.

对。它是一次性完成的吗？

Right. And does it do it in one pass?

是的。所以它是一次前向传播就完成了所有工作。

Yeah. So it was a single forward pass that did all of that.

不仅是一次性完成，你还优化了推理。我记得你是用网络摄像头实时运行的。

Not only was it doing it in one pass, you also optimized inference. You're doing it on a webcam. I remember.

是的。我做了这个疯狂的实时演示：网络在斯坦福的服务器上运行，然后有一个网页前端从网络摄像头获取视频流，把图像发送回服务器。服务器运行模型，再把预测结果流式传回。所以我就在实验室里拿着这台笔记本电脑走来走去，向人们实时展示这个网络。

Yeah. So I had built this crazy real-time demo where I had the network running on a server at Stanford and then a web front end that would stream from a webcam and then send the image back to the server. The server would run the model and stream the predictions back. So I was just walking around the lab with this laptop that would show people this network in real time.

识别和标注也一并完成了。是的，这非常令人印象深刻，因为我们大多数研究生如果能发表论文就满足了，对吧？他们把研究打包成论文，但 Justin 更进一步。他说，我想做一个实时网络演示。

Identification and labeling as well. Yeah, it was pretty impressive because most of our graduate students would be satisfied if they can publish the paper, right? They package the research, put it in a paper, but Justin went a step further. He's like, I want to do this real-time web demo.

嗯，实际上，我不知道我有没有跟你讲过这个故事，但那年我们在圣地亚哥有一个 ICCV 会议，是 ICCV 2015。我在那个会议上有一篇关于其他内容的论文，但我带着我的笔记本电脑。我在会议现场走来走去，给大家展示这个实时字幕演示，模型运行在加州的一个服务器上。所以它实际上是从加州一路流式传输到圣地亚哥。延迟很糟糕，大概只有 1 FPS，但能工作这件事本身就相当惊人了。

Well, actually, I don't know if I told you this story, but then we had a conference that year in Santiago at ICCV. It was ICCV 15. And I had a paper at that conference for something different, but I had my laptop. I was walking around the conference with my laptop showing everybody this real-time captioning demo and the model was running on a server in California. So it was actually able to stream all the way from California down to Santiago. Well, latency was terrible. It was like 1 FPS, but the fact that it worked at all was pretty amazing.

我本来想简短地调侃一下，也许视觉和语言建模并没有那么不同。DeepSeek 最近尝试了一个疯狂的事情：从像素建模文本，并直接在上面训练，这可能是未来。我不知道你们对语言是否真的必要有什么看法。

I was going to briefly quip that, you know, maybe vision and language modeling are not that different. DeepSeek recently tried the crazy thing of modeling text from pixels and just training on that, and it might be the future. I don't know if you guys have any takes on whether language is actually necessary at all.

我刚刚写了一整篇关于空间智能的宣言。

I just wrote a whole manifesto on spatial intelligence.

这就是我切入这个话题的方式。是的。

This is my segue into this. Yes.

我认为它们不同。我确实认为这些生成模型的架构会共享很多可共享的组件，但我认为深层的 3D、4D 空间世界具有一种结构层次，与纯粹的一维生成信号有本质区别。

I think they are different. I do think the architecture of these generative models will share a lot of shareable components, but I think the deeply 3D 4D spatial world has a level of structure that is fundamentally different from a purely generative signal that is one-dimensional.

是的，我认为像素最大化主义是有道理的，对吧？有一种观念认为语言是某种不同的东西，但我们用眼睛看语言，而我们的眼睛基本上就是像素，对吧？我们眼睛后面有某种生物像素在处理这些东西。我们看到文本，认为它是离散的东西，但这实际上只存在于我们的脑海中。文本和语言在我们世界中的物理表现是印在物体上的物理对象，我们用眼睛看到它们。嗯，你也可以认为它是声音，但即使是声音，你也可以转化为声音信号，对吧？然后如果你把它转化为我们在 LLM 中使用的纯 token 化表示，你实际上会丢失一些东西，对吧？比如你丢失了字体、换行、页面上的二维布局。在很多情况下，这可能无关紧要。但对于某些事情来说，这很重要。我认为像素是一种更无损的世界表征，在某种程度上是一种更通用的表征，更符合我们人类在世界上导航时所看到的东西。所以有一个效率方面的论点：也许把文本渲染成图像再输入视觉模型并不是非常高效。

Yeah, I think there's something to be said for pixel maximalism, right? Like there's this notion that language is this different thing, but we see language with our eyes and our eyes are just like, you know, basically pixels, right? We've got sort of biological pixels in the back of our eyes that are processing these things. And we see text and we think of it as this discrete thing, but that really only exists in our minds. Like the physical manifestation of text and language in our world are physical objects that are printed on things in the world and we see it with our eyes. Well, you can also think it's sound, but even sound you can translate into sound. Right? And then you actually lose something if you translate to this purely tokenized representations that we use in LLMs, right? Like you lose the font, you lose the line breaks, you lose the 2D arrangement on the page. For a lot of cases, maybe that doesn't matter. But for some things it does. And I think pixels are this sort of more lossless representation of what's going on in the world and in some ways a more general representation that matches what we humans see as we navigate the world. So there's an efficiency argument to be made like maybe it's not super efficient to render your text to an image and then feed that to a vision model.

这正是 DeepSeek 所做的，对吧？它确实有效。

That's exactly what DeepSeek did, right? It kind of worked.

我认为这关系到整个世界模型。我今年最喜欢的一篇论文是关于世界模型的归纳偏置。那是一篇哈佛的论文，他们向 LLM 输入了大量轨道模式，然后让 LLM 预测行星绕太阳的轨道，生成的模型看起来不错，但如果你让它画出力向量，结果就会乱七八糟，实际上并不遵循物理规律。那么，你如何看待数据中嵌入的信息？我们可以讨论一下为 3D 世界模型进行 token 化的问题，比如信息的维度。有视觉信息，但你需要从数据中提取多少潜在的隐藏力，以及其中有哪些挑战？

I think this ties into the whole world model. One of my favorite papers that I saw this year was about inductive bias for world model. So it was a Harvard paper where they fed a lot of orbital patterns into an LLM and then they asked the LLM to predict the orbit of a planet around the sun and the model generated looked good but then if you asked it to draw the force vectors it would be all wacky, it wouldn't actually follow it. So how do you think about what's embedded into the data that you get? And we can talk about maybe tokenizing for 3D world models like what are the dimensions of information. There's the visual but how much of the underlying hidden forces so to speak you need to extract out of this data and what are some of the challenges there?

是的，我认为有不同方法可以解决这个问题。一种是你可以尝试显式地处理，说：‘哦，我想测量所有的力，并把它们作为训练数据输入模型。’然后你可以运行传统的物理模拟，知道场景中的所有力，然后用这些作为训练数据来训练一个模型，希望它能预测这些力。或者你可以希望某些东西更隐式地涌现出来，你在一个更通用的问题上进行端到端训练，然后希望模型内部的某个地方必须学习建模类似物理的东西才能做出正确的预测。这大致就是我们更普遍拥有的两大范式。

Yeah, I think there's different ways you could approach that problem. One is you could try to be explicit about it and say, 'Oh, I want to measure all the forces and feed those as training data to your model.' Then you could run a traditional physics simulation and know all the forces in the scene and then use those as training data to train a model that's now going to hopefully predict those. Or you could hope that something emerges more latently, that you train on something end to end on a more general problem and then hope that somewhere in the internals of the model it must learn to model something like physics in order to make the proper predictions. And those are kind of the two big paradigms that we have more generally.

但没有迹象表明那些隐式建模能让你得到空间和动力学的因果律。

But there's no indication that those latent modeling will get you to a causal law of space and dynamics.

对吧？这正是当今深度学习与人类智能开始分叉的地方，因为深度学习本质上还是在拟合模式。

Right? That's where today's deep learning and human intelligence actually start to bifurcate because fundamentally deep learning is still fitting patterns.

这就有点哲学意味了，你说我们人类也在拟合模式，但可能我们试图拟合更广泛的模式，时间跨度更长，奖励函数也不同。但你提到的那篇论文就是这个问题：它学会了拟合特定的轨道模式，但并没有按你期望的方式泛化。它没有某种引力的因果模型。

There you sort of get philosophical and you say that we're trying to fit patterns too, but maybe we're trying to fit a more broad array of patterns over a longer time horizon, a different reward function. But basically the paper you mentioned is that problem that it learns to fit the specific patterns of orbits but then it doesn't actually generalize in the way that you'd like. It doesn't have a sort of causal model of gravity.

对吧？因为即使在 Marble 里，我试过，它生成了美丽的风景，里面还有拱门。

Right? Because even in Marble, I was trying it and it generates these beautiful sceneries with arches in them.

但模型真的理解拱门是如何依靠中心石以及实际的物理结构吗？另一个问题是：只要它始终渲染出符合我们想象中物理模型的东西，它是否理解重要吗？如果你用你理解的方式使用“理解”这个词，我很确定模型不理解。模型是从数据中学习，从模式中学习。这重要吗？尤其是对于它的用例。这是个好问题。目前，我认为不重要，因为它能渲染出你需要的东西，假设它是完美的。

But does the model actually understand how the arch is drawing on the center stone and the actual physical structure? And the other question is: does it matter that it understands it, as long as it always renders something that would fit the physical model we imagine? If you use the word 'understand' the way you understand, I'm pretty sure the model doesn't understand it. The model is learning from the data, learning from the pattern. Does it matter? Especially for the use cases. It's a good question. For now, I don't think it matters because it renders out what you need, assuming it's perfect.

是的。我的意思是，这取决于用例。如果用例是为虚拟电影制作生成背景，你只需要看起来合理的东西，所以可能不重要。但如果你是建筑师，用这个来设计一栋要实际建造的建筑，那么正确建模力就很重要，因为你不想让东西倒塌。但即便如此，即使你的模型有语义，我仍然不认为模型的理解和人类的理解是同一个词。这又变得哲学了。

Yeah. I mean, it depends on the use case. If the use case is generating a backdrop for virtual film production, all you need is something that looks plausible, so probably it doesn't matter. But if you're an architect using this to design a building you'll actually build, then it does matter that you model the forces correctly because you don't want the thing to break. But even there, even if your model has the semantics, I still don't think the understanding on the model's part and the understanding on the human part are the same word. This gets philosophical again.

是的，理解有个把戏。这些模型是一种与人类智能截然不同的智能。人类智能有趣的地方在于，我认为我理解事物是因为我能在一定程度上内省自己的思维过程。然后我相信我的思维过程与他人相似，所以当我观察别人的行为时，我推断他们的内心状态可能与我相似。因此，我知道我理解事物。所以我假设你理解某物。但这些模型是一种外星智能。它们能做非常有趣的事情，表现出有趣的行为，但无论它们有什么样的内部认知或自我反思，如果存在的话，都与我们完全不同。

Yeah, there's this trick with understanding. These models are a very different kind of intelligence than human intelligence. Human intelligence is interesting because I think I understand things because I can introspect my own thought process to some extent. Then I believe my thought process works similarly to others, so when I observe someone else's behavior, I infer their internal mental state is probably similar to mine. Therefore, I know I understand things. So I assume you understand something. But these models are an alien form of intelligence. They can do really interesting things and exhibit interesting behavior, but whatever internal cognition or self-reflection they have, if it exists at all, is totally different from what we do.

它没有自我意识。

It doesn't have the self-awareness.

对。但这意味着，当我们观察到这些系统看似有趣或智能的行为时，我们不一定能推断出它们的其他特性，因为它们的世界模型和思维方式与我们如此不同。

Right. But what that means is that when we observe seemingly interesting or intelligent behavior from these systems, we can't necessarily infer other things about them because their model of the world and the way they think is so different from us.

那么你需要两个不同的模型来分别处理视觉和建筑生成吗？你认为最终你构建模型的方法没有根本性的问题吗？更多的是关于 Scaling 模型及其能力，还是说过于视觉化会阻止你学习背后的物理，从而无法信任它生成能在现实世界中工作的设计？

So would you need two different models to do the visual one and the architectural generation? Do you think eventually there's nothing fundamental about the approach you've taken on the model building? It's more about scaling the model and its capabilities, or is there something about being very visual that prohibits you from learning the physics behind it so you could trust it to generate a design that works in the real world?

我认为这是 Scaling 数据和改进模型的问题。我不认为有什么根本性的东西将这两者分开。

I think this is a matter of scaling data and bettering the model. I don't think there's anything fundamental that separates these two.

是的，我希望它是一个模型，但我认为深度学习在某种意义上的一大问题是，如何获得超越训练数据的涌现能力？你会得到某种理解力的东西，尽管它没有被训练来预测力，但它会在内部隐式地学习它们？我认为我们在其他大模型中看到的是，很多这种涌现行为确实在规模上发生了。这会不会迁移到其他模态和用例？我希望如此，但这将是一个需要时间验证的过程。

Yeah, I would like it to be one model, but I think the big problem in deep learning in some sense is how do you get emergent capabilities beyond your training data? Are you going to get something that understands the forces while it wasn't trained to predict the forces, but it's going to learn them implicitly internally? I think a lot of what we've seen in other large models is that a lot of this emergent behavior does happen at scale. Will that transfer to other modalities and use cases? I hope so, but that'll be a process we need to play out over time.

有没有一种诱惑去依赖已经存在的物理引擎，比如游戏行业已经为你省了很多工作，还是说我们必须因为某种根本性的不匹配而重新发明？

Is there a temptation to rely on physics engines that already exist, like the gaming industry has saved you a lot of this work, or do we have to reinvent things for some fundamental mismatch?

我认为这就像攀登技术阶梯。从某种意义上说，你构建这些东西的原因是因为传统物理引擎在某些情况下不起作用。如果物理引擎是完美的，我们就不需要构建模型，因为问题已经解决了。所以我们做这些是因为经典物理引擎不能以我们想要的通用性解决问题。但这并不意味着我们需要抛弃它们。我们可以使用传统物理引擎生成数据，然后用这些数据训练模型。然后你就在将物理引擎蒸馏到神经网络的权重中。我认为很多其他实验室都在这样做。人们推测 Sora 有一点，Genie3 也有一点。Genie 明确地像一个视频游戏，有控制可以走动。我总觉得我们为娱乐发明的东西最终会进入严肃的工作中，这很有趣。

I think that's like climbing the ladder of technology. In some sense, the reason you want to build these things at all is because traditional physics engines don't work in some situations. If a physics engine were perfect, we would have no need to build models because the problem would already be solved. So the reason we want to do these is because classical physics engines don't solve problems in the generality we want. But that doesn't mean we need to throw them away. We can use traditional physics engines to generate data that we then train our models on. Then you're distilling the physics engine into the weights of the neural network. I think that's a lot of what other labs are doing. People speculate that Sora had a bit of that, Genie3 had a bit of that. Genie is explicitly like a video game with controls to walk around. I always think it's funny how things we invent for fun eventually make it into serious work.

是的。整个 AI 革命始于图形芯片。部分地误用 GPU 从生成大量三角形变成生成大量其他东西。

Yeah. The whole AI revolution started by graphics chips. Partially misusing the GPU for generating a lot of triangles into generating a lot of everything else basically.

是的。

Yeah.

我们稍微提到了 Marble。我认为你们选择 Marble 作为某种走出隐身状态的时刻，如果可以这么说的话。

We touched on Marble a little bit. I think you guys chose Marble as your sort of coming out of stealth moment, if you can call it that.

是的。

Yeah.

也许你能给我们一个简洁的解释，告诉大家应该从中得到什么，因为这里每个人都可以尝试 Marble，但我觉得他们可能无法将其与你的愿景和其他实验室可能见过的生成世界之间的差异联系起来。

Maybe we can get a concise explanation from you on what people should take away because everyone here can try Marble, but I don't think they might be able to link it to the differences between what your vision is versus other generative worlds they may have seen from other labs.

所以 Marble 是我们模型的一个缩影。我们是一家空间智能模型公司。我们相信空间智能是下一个前沿。为了制造空间智能模型，模型必须非常强大，能够以多模态的方式理解、推理和生成世界，并允许达到我们最终希望的、像人类与世界互动那样复杂的交互水平。这就是空间智能的宏大愿景，也是我们眼中的世界模型。Marble 是迈向那个愿景的第一步。它是这段旅程的开端。它是世界上第一个以这种保真度生成 3D 世界并交付给公众的同类模型。这是一个起点。我们实际上写了一篇技术博客。Justin 花了大量时间撰写那篇博客。我不知道你是否抽空浏览过。Justin 真的把它分解成了 Marble 的输入——多模态输入——可编辑性，它允许用户与模型交互，以及我们可以获得的输出类型。

So Marble is a glimpse into our model. We are a spatial intelligence model company. We believe spatial intelligence is the next frontier. In order to make spatially intelligent models, the model has to be very powerful in terms of its ability to understand, reason, and generate in a very multimodal fashion of worlds, as well as allow the level of interactivity that we eventually hope to be as complex as how humans can interact with the world. So that's the grand vision of spatial intelligence as well as the kind of world models we see. Marble is the first glimpse into that. It's the first part of that journey. It's the first in-class model in the world that generates 3D worlds at this level of fidelity that is in the hands of the public. It's the starting point. We actually wrote a tech blog. Justin spent a lot of time writing that tech blog. I don't know if you had time to browse it. Justin really broke it down into what are the inputs—multimodal inputs of Marble—what are the kind of editability, which allows users to be interactive with the model, and what are the kind of outputs we can have.

是的。

Yeah.

所以 Marble，基本上一种看待它的方式是，它是一个 3D 世界的生成模型。你可以输入文本、图像或多张图像，它会为你生成一个匹配这些输入的 3D 世界。它也是交互式的，你可以交互式地编辑场景。我可以生成这个场景，然后说“我不喜欢这个水瓶，把它变成蓝色”，“把桌子拿掉”，“把这些麦克风换一下”，然后你可以基于这些交互式编辑生成新的世界，并以多种格式导出。通过 Marble，我们实际上试图同时做两件事，我认为我们很好地实现了平衡。一是构建一个朝着空间智能宏大愿景前进的模型。模型需要能够理解各种不同的输入，需要在许多情况下对世界进行建模，需要能够对它们随时间变化的反事实进行建模。所以我们想开始构建具备这些能力的模型，而今天的 Marble 已经具备了所有这些的雏形。但与此同时，我们是一家公司。我们是一个企业。我们真的不想让这成为一个科学项目，而是想构建一个在现实世界中今天就能对人们有用的产品。所以，虽然 Marble 同时是一个朝着空间智能愿景构建的世界模型，但它也经过精心设计，成为人们今天就能发现有用的东西。我们开始看到在游戏、视觉特效、电影等领域出现新的用例，我认为 Marble 作为产品今天已经可以做很多非常有趣的事情，同时也为我们未来想要构建的宏大世界模型奠定了基础。

So Marble, basically one way of looking at it, it's a generative model of 3D worlds. You can input things like text or image or multiple images, and it will generate for you a 3D world that matches those inputs. It's also interactive in the sense that you can interactively edit scenes. I could generate this scene and then say 'I don't like the water bottle, make it blue instead,' 'take out the table,' 'change these microphones around,' and then you can generate new worlds based on these interactive edits and export in a variety of formats. With Marble, we were actually trying to do two things simultaneously, and I think we managed to pull off the balance pretty well. One is to build a model that goes towards the grand vision of spatial intelligence. Models need to be able to understand lots of different kinds of inputs, need to be able to model worlds in a lot of situations, need to be able to model counterfactuals of how they could change over time. So we wanted to start to build models that have these capabilities, and Marble today already has hints of all of these. But at the same time, we're a company. We're a business. We were really trying not to have this be a science project but also build a product that would be useful to people in the real world today. So while Marble is simultaneously a world model that is building towards this vision of spatial intelligence, it was also very intentionally designed to be a thing that people could find useful today. We're starting to see emerging use cases in gaming, in VFX, in film, where I think there's a lot of really interesting stuff that Marble can do today as a product, and also set a foundation for the grand world models that we want to build going into the future.

是的。我注意到一个工具非常有趣，因为你可以在内部录制你的场景。

Yeah. I noticed one tool that was very interesting because you can record your scene inside.

是的。

Yes.

是的。这非常重要。录制的能力意味着对相机位置的精确控制。为了精确控制相机位置，意味着你必须对 3D 空间有感知。否则，你不知道如何定位相机，对吧？以及如何移动相机。所以这是这类模型的自然结果，这也是为什么这只是其中一个例子。

Yes. It's very important. The ability to record means a very precise control of camera placement. In order to have precise camera placement, it means you have to have a sense of 3D space. Otherwise, you don't know how to orient your camera, right? And how to move your camera. So that is a natural consequence of this kind of model, and this is why this is just one of the examples.

是的。我发现当我玩视频生成模型时，我不得不学习导演的语言，因为我必须像平移一样移动它们。你不能说“向北平移 63 度”，对吧？你就是没有那种控制。而在 Marble 中，你可以精确控制相机的位置。是的，我认为这是人们需要首先理解的事情之一：它不是逐帧生成的，而很多其他模型是逐帧生成的。

Yeah. I find when I play with video generative models, I'm having to learn the language of being a director because I have to move them like pan. You cannot say 'pan 63 degrees to the north,' right? You just don't have that control. Whereas in Marble, you have precise control in terms of placing your camera. Yeah, I think that's one of the first things people need to understand: it's not generating frame by frame, which is what a lot of the other models are.

是的。原子单位是什么？人们理解大语言模型生成一个 token。原子单位是什么？有网格、高斯泼溅、体素——3D 世界中有很多组成部分。人们应该对你的生成有什么样的心智模型？

Yeah. What are the atomic units? People understand that LLM generates one token. What are the atomic units? There's meshes, splats, voxels—there's a lot of pieces in a 3D world. What should be the mental model that people have of your generations?

是的，我认为有今天存在的和未来可能存在的。今天存在的是模型原生输出高斯泼溅。高斯泼溅是这些微小的粒子，半透明，在 3D 空间中具有位置和方向。场景由大量这些高斯泼溅构建而成。高斯泼溅非常酷，因为你可以非常高效地实时渲染它们。所以你可以在你的 iPhone 上渲染，渲染一切。这就是我们获得那种精确相机控制的方式，因为高斯泼溅几乎可以在任何客户端设备上实时渲染。所以对于我们今天生成的许多场景，那个原子单位就是单个高斯泼溅。但我不认为这是根本性的。我可以想象未来其他有趣的方法。还有其他方法，甚至我们在 World Labs 也研究过，比如我们最近的 RTFM 模型，它确实逐帧生成。那里的原子单位是随着用户与系统交互而逐帧生成。或者你可以想象未来的其他架构，其中原子单位是一个 token，这个 token 现在代表 3D 世界的某个块。我认为随着时间的推移，我们可以在这里尝试许多不同的架构。

Yeah, I think there's what exists today and what could exist in the future. So what exists today is the model natively outputs splats. Gaussian splats are these tiny particles that are semi-transparent, have a position and orientation in 3D space. The scene is built up from a large number of these Gaussian splats. Gaussian splats are really cool because you can render them in real time very efficiently. So you can render on your iPhone, render everything. That's how we get that sort of precise camera control because the splats can be rendered in real time on pretty much any client-side device. So for a lot of the scenes we're generating today, that atomic unit is that individual splat. But I don't think that's fundamental. I could imagine other approaches in the future that would be interesting. There are other approaches that even we've worked on at World Labs, like our recent RTFM model that does generate frames one at a time. There the atomic unit is generating frames one at a time as the user interacts with the system. Or you could imagine other architectures in the future where the atomic unit is a token, where that token now represents some chunk of the 3D world. I think there's a lot of different architectures that we can experiment with here over time.

你之前在发言中也经常提到物理和力学，但我在 Marble 里没看到这些。我猜目前还没有。也许如果有个 Marble 2，你会有运动，或者是对高斯泼溅做某种修改，还是说会完全是另一回事？

You also in previous statements focused a lot on physics and forces, which is something I don't see in Marble. I presume it's not there yet. Maybe if there was like a Marble 2, you would have movement, or is there a modification to Gaussian Splats that makes sense, or would it be something completely different?

是的，我认为有一些合理的修改，实际上有很多有趣的方法可以整合这些东西，这也是在这个领域工作的另一个好处。这方面已经有很多研究工作。说到疯狂的想法，实际上有很多非常有趣的学术工作，探讨了赋予物理特性的不同方式。高斯泼溅本身就是小粒子。有很多方法，基本上就是给这些泼溅点附加物理属性，比如每个点都有质量，或者把每个点看作与邻近点通过某种虚拟弹簧耦合，这样你就可以在泼溅点之上进行物理模拟。所以，为这些东西添加物理、动力学或交互的一个途径是预测与每个泼溅点相关的物理属性，然后在下游模拟它们，要么用经典物理，要么用学习的方法。在 3D 领域工作的美妙之处在于，事物可以组合，你可以在不同地方注入逻辑。所以一种方法是：我们生成一个 3D 场景，预测场景中所有物体的 3D 属性，然后用经典物理引擎模拟交互。或者你也可以这样做：根据用户的操作，模型重新生成整个场景，用泼溅点或其他表示。这可能会更通用，因为你不再受限于你已经知道如何建模的物理属性。但这也对算力要求更高，因为你需要根据用户操作重新生成整个场景。我认为这是一个非常有趣的未来工作方向，也可以像你说的那样，添加到潜在的 Marble 2 中。

Yeah, I think there are a couple of modifications that make sense, and there are actually a lot of interesting ways to integrate things here, which is another nice place of working in this space. There's actually been a lot of research work on this. When you talk about wacky ideas, there's actually been a lot of really interesting academic work on different ways to imbue physics. Gaussian Splats are themselves little particles. There have been a lot of approaches where you basically attach physical properties to those splats and say that each one has a mass, or maybe you treat each one as being coupled with some kind of virtual spring to nearby neighbors, and now you can start to do physics simulation on top of splats. So one avenue for adding physics, dynamics, or interaction to these things would be to predict physical properties associated with each of your splat particles and then simulate those downstream, either using classical physics or something learned. The beauty of working in 3D is that things compose and you can inject logic in different places. So one way is: we're generating a 3D scene, we predict 3D properties of everything in the scene, then we use a classical physics engine to simulate the interaction. Or you could do something where, as a result of a user action, the model is now going to regenerate the entire scene in splats or some other representation. That could potentially be a lot more general because you're not bound to whatever physical properties you know how to model already. But that's also a lot more computationally demanding because you need to regenerate the whole scene in response to user actions. I think this is a really interesting area for future work and for adding to potential Marble 2, as you say.

是的。而且还有动力学的机会，对吧？泼溅点的密度现在怎么样？我们能否渲染出足够多的点，以便在放大时获得非常高的分辨率？我们是受限于生成的数量还是渲染的数量？这些东西要如何达到超高保真度？

Yeah. And there's opportunity for dynamics, right? What's the state of splat density? Can we render enough to have very high resolution when we zoom in? Are we limited by the amount that we can generate or the amount that we can render? How are these going to get super high fidelity, so to speak?

有一些限制，但取决于你的目标用例。我们场景的一个主要限制是，我们希望东西能在移动设备和 VR 头显上流畅渲染。这些设备的算力远低于其他很多情况。如果你想让一个泼溅点文件在 4 年前的 iPhone 上以高分辨率、30 到 60 帧每秒的速度渲染，那么你能处理的泼溅点数量就有点受限。但如果你可以在今年的 iPhone、最近的 MacBook 上运行，或者有本地 GPU，或者不需要 60 帧 1080p，那么你就可以放宽限制，使用更多的泼溅点，从而在场景中获得更高的分辨率。

You have some limitations, but depending on your target use case. One of the big constraints on our scenes is we wanted things to render cleanly on mobile and in VR headsets. Those devices have a lot less compute than you have in a lot of other situations. If you want to get a splat file to render at high resolution, like 30 to 60 fps on an iPhone from 4 years ago, then you are a bit limited in the number of splats you can handle. But if you're allowed to work on a recent iPhone, a recent MacBook, or even if you have a local GPU, or if you don't need 60 fps 1080p, then you can relax the constraints and get away with more splats, which lets you get higher resolution in your scenes.

我期待但没听你提到的一个用例是具身用例。你们目前只关注虚拟领域吗？

One use case I was expecting but didn't hear from you was embodied use cases. Are you just focusing on virtual for now?

如果你去 World Labs 的主页，有一个专门的页面叫 Marble Labs。我们在那里展示了不同的用例，实际上我们将其分为视觉效果用例、游戏用例以及模拟用例。在模拟用例中，我们展示了这项技术可以极大地帮助机器人训练。这又回到了我之前提到的数据匮乏问题。机器人训练确实缺乏数据。高保真的真实世界数据绝对至关重要，但你不可能获得大量这样的数据。另一个极端是纯互联网视频数据，但那样你就缺乏训练具身智能体所需的可控性。因此，模拟和合成数据实际上是一个非常重要的中间地带。我在这个领域工作了很多年，最大的痛点之一就是从哪里获得这些合成模拟数据？你需要策划资产，构建并组合这些复杂场景。在机器人领域，你需要很多不同的状态。你希望具身智能体在合成环境中进行交互。Marble 实际上在帮助生成这些用于具身智能体训练的合成模拟世界方面有很大潜力。

If you go to World Labs homepage, there is a particular page called Marble Labs. There we showcase different use cases, and we actually organize them into visual effect use cases, gaming use cases, as well as simulation use cases. In that, we actually show that this is a technology that can help a lot in robotic training. This goes back to what I was talking about earlier regarding data starvation. Robotic training really lacks data. High-fidelity real-world data is absolutely critical, but you're just not going to get a ton of that. The other extreme is purely internet video data, but then you lack a lot of the controllability you want to train your embodied agents with. So simulation and synthetic data is actually a very important middle ground for that. I've been working in this space for many years, and one of the biggest pain points is where do you get this synthetic simulated data? You have to curate assets and build and compose these complex situations. In robotics, you want a lot of different states. You want the embodied agent to interact in the synthetic environment. Marble actually has a lot of potential for helping to generate these synthetic simulated worlds for embodied agent training.

显然，这已经在主页上了。它会有的。我试图建立联系，就像你说的，你还需要建立商业模式。机器人市场显然非常巨大。也许你不需要那个，或者也许我们需要先构建和解决虚拟世界，然后再进入具身领域。显然这是一个垫脚石。

Obviously that's on the homepage. It'll be there. I was trying to make the link, as you said, you also have to build a business model. The market for robotics is obviously very huge. Maybe you don't need that, or maybe we need to build up and solve the virtual worlds first before we go to embodied. Obviously a stepping stone.

这还有待决定。我确实认为……

That is to be decided. I do think that...

因为其他人都直接奔着那个方向去了，对吧？

Because everyone else is going straight there, right?

不是所有人，但可以说有一种兴奋感。不过我认为世界足够大，可以有不同的方法。

Not everyone else, but there is an excitement, I would say. But I think the world is big enough to have different approaches.

是的。方法。

Yeah. Approaches.

是的。我的意思是，我们一直认为这是一项相当横向的技术，随着时间的推移应该能够触及许多不同的行业。Marble 目前更侧重于创意产业，但我认为驱动它的技术随着时间的推移应该适用于许多不同的事物。而机器人技术是其中一个可能迟早会实现的领域。

Yeah. I mean, we always view this as a pretty horizontal technology that should be able to touch a lot of different industries over time. Marble is a little bit more focused on creative industries for now, but I think the technology that powers it should be applicable to a lot of different things over time. And robotics is one that is maybe going to happen sooner than later.

还有设计，对吧，与创意非常接近。

Also, design, right, is very adjacent to creative.

哦，是的。绝对。比如建筑方面的东西。

Oh, yeah. Definitely. Like the architecture stuff.

是的。好的。我是说，我在网上开玩笑。我在 Slack 上发了一个视频，说：‘哦，谁想用 Marble 来规划你的下一次厨房改造？’它实际上已经在这方面做得很好。只需拍两张厨房的照片，在 Marble 中重建，然后使用编辑功能，看看如果你更换台面、地板或橱柜，那个空间会是什么样子。

Yes. Okay. Yeah. I mean, I was joking online. I posted this video on Slack of like, 'Oh, who wants to use Marble to plan your next kitchen remodel?' It actually works great for this already. Just take two images of your kitchen, reconstruct it in Marble, and then use the editing features to see what that space would look like if you change the countertops or change the floors or change the cabinets.

而且这并非我们特意为某个用例构建的，但由于它是一种强大的横向技术，你会得到这些从模型中自然涌现的用例。我们已经有早期测试用户使用 API 密钥，正在为室内设计用例进行构建。

And this is something that we didn't necessarily build anything specific for this use case, but because it's a powerful horizontal technology, you kind of get these emergent use cases that just fall out of the model. We have early beta users using an API key that is already building for interior design use case.

我刚装修完车库。我早该知道这个的。

I just did my garage. I should have known about this.

下次你改造时，我们可以帮忙。

Next time you remodel, we can be of help.

嗯，厨房是下一个，我肯定。

Well, kitchen is next, I'm sure.

是的。

Yeah.

我对整个空间智能领域很好奇。我觉得我们应该深入探讨一下。你如何定义它？以及当 Dario 说我们有一个装满爱因斯坦的数据中心时，人们可能想到的传统智能（比如大语言模型）与空间智能之间的差距是什么？要具备空间智能需要什么？

I'm curious about the whole spatial intelligence space. I think we should dig more into that one. How do you define it and what are the gaps between traditional intelligence that people might think about, like LLMs, when you know Dario says we have a data center full of Einstein, that's like traditional intelligence, it's not spatial intelligence. What is required to be spatially intelligent?

首先，我不理解那句话，一个装满爱因斯坦的数据中心。

First of all, I don't understand that sentence, a data center full of Einsteins.

这是个类比，只是个类比。

It's an analogy, it's an analogy.

嗯，人工智能作为一个领域、一门学科，很大程度上是受人类智能启发的，对吧？因为我们是目前宇宙中已知的最聪明的动物。如果你看看人类智能，它是非常多元的，对吧？有一位心理学家，我想是 20 世纪 60 年代的 Howard Gardner，他实际上提出了多元智能来描述人类智能。有语言智能、空间智能、逻辑智能和情感智能。所以对我来说，当我想到空间智能时，我认为它是语言智能的补充。所以我个人不会说空间智能与传统智能对立，因为我不知道传统是什么意思。我确实认为空间智能是对语言智能的补充。那么如何定义空间智能呢？它是一种让你能够在空间中推理、理解、移动和交互的能力。我用 DNA 结构的推导作为例子，当然我简化了这个故事，但其中很大一部分涉及分子和化学键在三维空间中的空间推理，最终推断出双螺旋结构。而人类，或者说 Francis Crick 和 Watson 拥有的那种能力，很难将这个过程简化为纯粹的语言。那是文明的一个巅峰时刻。但每天，比如我在这里试图拿起一个杯子。看到杯子、看到它所在的场景、看到我自己的手、张开手使其几何形状与杯子匹配、并触摸到正确的抓握点——整个过程都深深涉及空间。这非常困难。我试图用语言来描述它，但另一方面，描述性的语言本身并不能让你拿起杯子。

Well, so a lot of AI as a field, as a discipline, is inspired by human intelligence, right? Because we are the most intelligent animal we know in the universe for now. And if you look at human intelligence, it's very multi-intelligent, right? There is a psychologist, I think his name is Howard Gardner in the 1960s, actually literally called multiple intelligence to describe human intelligence. And there is linguistic intelligence, there's spatial intelligence, there is logical intelligence and emotional intelligence. So for me, when I think about spatial intelligence, I see it as complementary to language intelligence. So I personally would not say it's spatial versus traditional because I don't know what traditional means. I do think spatial is complementary to linguistic. And how do we define spatial intelligence? It's the capability that allows you to reason, understand, move and interact in space. And I use this example of the deduction of DNA structure, right? And of course I'm simplifying this story, but a lot of that had to do with the spatial reasoning of the molecules and the chemical bonds in a 3D space to eventually conjecture a double helix. And that ability that humans, or Francis Crick and Watson had, is very, very hard to reduce that process into pure language. And that's a pinnacle of a civilizational moment. But every day, right, I'm here trying to grasp a mug. This whole process of seeing the mug, seeing the context where it is, seeing my own hand, opening of my hand that geometrically would match the mug and touching the right affordance point. All this is deeply spatial there. It's very hard. I'm trying to use language to narrate it. But on the other hand, that narrated language itself cannot get you to pick up a mug.

是的。带宽限制。

Yeah. Bandwidth constraint.

是的。

Yes.

我最近算了一下，如果你每天 24 小时不停地说话，你会生成多少个 token？按平均语速每分钟 150 个词，大约每天 21.5 万个 token。而你生活的世界带宽比那高得多。

I did some math recently on if you just spoke all day every day for 24 hours a day, how many tokens do you generate? At the average speaking rate of like 150 words per minute, it roughly rounds out to about 215,000 tokens per day. And your world that you live in is so much higher bandwidth than that.

嗯，我认为确实如此。但如果我想想艾萨克·牛顿爵士，比如当时像重力这样的东西还没有被语言形式化，人们本能地在空间上理解物体会下落，对吧？但用某种方式将其形式化是有帮助的，就像所有这些不同的规则，我们用语言来真正捕捉那些经验上和空间上也能理解的东西，但用语言描述更容易。所以我很好奇空间智能和语言智能之间的相互作用，比如你需要理解有些规则更容易用语言写出来让空间智能理解，但你无法写出‘把手这样放，然后放下这么多’。所以我一直好奇如何将两者相互利用。

Well, I think that is true. But if I think about Sir Isaac Newton, right, it's like you have things like gravity at the time that have not been formalized in language that people inherently spatially understand that things fall, right? But then it's helpful to formalize that in some way or like all these different rules that we use language to really capture something that empirically and spatially you can also understand but it's easier to describe in a way. So I'm curious about the interplay of spatial and linguistic intelligence, which is like you need to understand some rules are easier to write in language for the spatial intelligence to understand, but you cannot write 'put your hand like this and put it down this amount'. So I'm always curious about how you leverage each other together.

我的意思是，如果说有什么的话，就像牛顿的例子，牛顿之所以想到写下那些定律，是因为他在世界上有很多具身体验，比如看棒球。

I mean, if anything, like the example of Newton, Newton only thinks to write down those laws because he's had a lot of embodied experience in the world watching baseball.

完全正确。而且实际上，区分你提到的理论构建与具身体验——即嵌入三维世界的日常体验——是很有用的。所以对我来说，空间智能在某种程度上包含了那种在三维空间中存在、移动、观察和行动的具身体验。正如你所说，你可以叙述这些事情，但这是一个非常有损的通道。就像在世界上存在并做事情这个概念，与试图描述它是非常不同的模态。但因为我们人类是动物，一直在空间中进化互动，我们甚至不认为那是一件难事，对吧？然后我们自然就跃升到语言，然后是理论构建，作为抽象于那种原生空间理解的机制。从某种意义上说，大语言模型直接跳到了那些最高形式的抽象推理，这非常有趣也非常有用。但空间智能几乎就像是重新打开那个黑箱，说也许我们直接跳到那种完全抽象的语言、推理和沟通形式时，失去了一些东西。

Exactly. And actually it's useful to distinguish between the theory building that you're mentioning versus the embodied, like the daily experience of being embedded in the three-dimensional world. Right. So to me, spatial intelligence is sort of encapsulating that embodied experience of being there in 3D space, moving through it, seeing it, actioning it. And as you said, you can narrate those things but it's a very lossy channel. It's just like the notion of being in the world and doing things in it is a very different modality from trying to describe it. But because we as humans are animals who have evolved interacting in space all the time, like we don't even think that that's a hard thing, right? And then we sort of naturally leap to language and then theory building as mechanisms to abstract above that sort of native spatial understanding. And in some sense, LLMs have just jumped all the way to those highest forms of abstracted reasoning, which is very interesting and very useful. But spatial intelligence is almost like opening up that black box again and saying maybe we've lost something by going straight to that fully abstracted form of language and reasoning and communication.

你知道吗，作为视觉科学家，我觉得很有趣。我总觉得视觉被低估了，因为对人类来说它毫不费力。婴儿睁开眼睛，就开始看东西。我们天生就有这种能力，对吧？我们几乎是天生的。但学习语言需要付出努力，包括学习如何写作、语法、表达，这让人觉得很难。而大自然实际上花了更多时间优化的东西——感知和空间智能——却被人类低估了。

You know, it's funny as a vision scientist, right? I always find that vision is underappreciated because it's effortless for humans. You open your eyes as a baby, you start to see your work. We're somehow born with it, right? We're almost born with it. But you have to put effort in learning language including learning how to write, how to do grammar, how to express and that makes it feel hard. Whereas something that nature spent way more time actually optimizing which is perception and spatial intelligence is underappreciated by humans.

有证据表明我们天生就有这种能力吗？你说了‘几乎是天生的’。所以听起来我们实际上是在出生后学习的。我们出生时，视力较弱，感知能力确实会提高。但大多数人类天生就有视觉能力，大多数人类天生就有将感知与运动联系起来的能力，对吧？我的意思是运动本身需要一段时间来完善，但动物很了不起，对吧？就像我今年夏天早些时候在非洲，那些小动物出生后几分钟内就必须行动起来，否则狮子就会抓住它们。

Is there proof that we are born with it? You said 'almost born'. So it sounds like we actually do learn after we're born. When we are born, our visual acuity is less and our perceptual ability does increase. But most humans are born with the ability to see and most humans are born with the ability to link perception with motor movements, right? I mean the motor movement itself takes a while to refine, but animals are incredible, right? Like I was just in Africa earlier this summer. These little animals, they're born and within minutes they have to get going and otherwise the lions will get them.

在自然界中，优化感知、空间智能和语言花了 5.4 亿年。对语言发展最慷慨的估计大概是 50 万年。

And in nature, it took 540 million years to optimize perception and spatial intelligence and language. The most generous estimation of language development is probably half a million years.

哇，这比我猜的要长。

Wow, that's longer than I would have guessed.

是的，我已经很慷慨了。

Yeah, I'm being very generous.

我在读你的书时发现了一个有趣的关联，和我们播客里聊过的东西有关：语言模型基准测试，以及它们如何加入需要空间智能的物理不可能性。比如 A 在 B 上面，所以 A 不能穿过 B 掉下去。这对我们来说显而易见，但对语言模型来说却可能发生。也许这是下一个词预测的一部分。

I was going through your book and realized one interesting link to something we covered on the podcast: language model benchmarks and how they put in physical impossibilities that require spatial intelligence. Like A is on top of B, therefore A cannot fall through B. That's obvious to us, but to a language model it could happen. Maybe it's part of next-token prediction.

这就是我说的解开这个抽象。如果你的整个世界模型只是把单词序列一个接一个地说出来，就很难理解为什么不行。

That's what I mean about unwrapping this abstraction. If your whole model of the world is just saying sequences of words after each other, it's hard to see why not.

这其实不公平，对吧？但对我们来说显而易见的原因是我们内部把它映射回我们熟悉的三维世界表征。问题是，从世界模型蒸馏到语言模型有多难，因为我们确实希望我们的模型拥有空间智能。我们是否必须完全抛弃语言模型才能做到这一点？

It's actually unfair, right? But the reason it's obvious to us is because we internally map it back to some three-dimensional representation of the world we're familiar with. The question is how hard it will be to distill from world models into a language model, because we do want our models to have spatial intelligence. Do we have to throw language model out completely to do that?

不，我不这么认为。我认为它们是多模态的。即使我们今天的 Marble 模型也以语言作为输入。所以它是深度多模态的。在许多用例中，这些模型将协同工作。也许有一天我们会有一个通用模型。

No, I don't think so. I think they're multimodal. Even our model Marble today takes language as input. So it's deeply multimodal. In many use cases, these models will work together. Maybe one day we'll have a universal model.

即使从实用角度看，人们使用语言并希望用语言与系统交互。所以构建让人们能与之对话的模型是有用的。我不认为这会消失。有一种智力上的好奇心，想知道你能在多大程度上构建一个只使用视觉或空间智能的模型。我不知道这在实践上是否有用，但这会是一个有趣的学术练习。

Even pragmatically, people use language and want to interact with systems using language. So it's useful to build models that let people talk to them. I don't see that going away. There's an intellectual curiosity about how much you could build a model that only uses vision or spatial intelligence. I don't know if that would be practically useful, but it would be an interesting academic exercise.

不把话题拉回物理，但我很好奇：如果你有一个高度精确的世界模型，却不给它任何我们当前对物理标准模型的理解，它能从头重新发现多少？它需要什么水平的语言理解？因为我们使用了很多符号，但也许它会提出一个非常不同的模型，却仍然准确。我想知道我们在多大程度上受到限制——世界是为人类建造的，我们构建语言的方式也限制了来自其他模态的输出。

Not to bring it back to physics, but I'm curious: if you had a highly precise world model and didn't give it any notion of our current understanding of the standard model of physics, how much would it be able to come up with from scratch? What level of language understanding would it need? Because we have many notations we use, but maybe it would come up with a very different model and still be accurate. I wonder how much we're limited by the fact that the world is built for humans, and the way we build language constrains outputs from other modalities.

我非常期待关注你的工作。

I'm super excited to follow your work.

你甚至不需要 AI 来回答这个问题。你可以发现外星人，看看他们有什么样的物理。他们可能有不同的理解。

You don't even need AI to answer that question. You could discover aliens and see what kind of physics they have. They might have a different understanding.

面对现实吧，我们是目前宇宙中最聪明的动物。

Let's face it, we are so far the smartest animal in the universe.

对。但这确实是个有趣的问题：我们对宇宙的知识和对物理的理解是否受到我们自身认知或技术进化路径依赖的约束？一种实验方法是重演人类文明，看看我们是否会以同样的顺序提出同样的物理。这不是一个很实际的实验。

Right. But it's a really interesting question: is our knowledge of the universe and our understanding of physics constrained by our own cognition or by the path dependence of our technological evolution? One way to experiment would be to rerun human civilization and see if we'd come up with the same physics in the same order. That's not a very practical experiment.

我想知道一个实验：我们有大量关于天体运动的天体物理数据。把数据输入模型，看看牛顿定律是否会涌现。

One experiment I wonder about: we have plenty of astrophysical data on celestial body movements. Just feed the data into a model and see if Newtonian law emerges.

我猜很可能不会。牛顿定律的抽象层次与这些 LLM 所代表的层次不同。

My guess is it probably won't. The abstraction level of Newtonian law is at a different level from what these LLMs represent.

所以我不惊讶，如果有足够的天体运动数据，LLM 能预测相当准确的运动轨迹。假设我发明一颗围绕恒星的行星，有足够数据，我的模型会告诉你它第一天在哪，第二天在哪。我不会惊讶。但 F=ma 或作用力等于反作用力是完全不同的抽象层次，超出了今天的 LLM。

So I wouldn't be surprised that given enough celestial movement data, an LLM would predict pretty accurate movement trajectories. Let's say I invent a planet surrounding a star; given enough data, my model would tell you where it is on day one, day two. I wouldn't be surprised. But F=ma or action equals reaction is a whole different abstraction level, beyond today's LLM.

你需要什么样的模型才能避免地心模型？如果我只在视觉数据上训练，你会认为太阳绕地球转是合理的。但事实并非如此。它如何学会这一点？有时也许你不需要力，因为只要看起来对就是对的，但当你跳到更高级的任务时，我们能在多大程度上依赖它们？

What model would you need to not have it be a geocentric model? If I'm training just on visual data, it makes sense that you think the sun rotates around the earth. But that's not the case. How would it learn that? Sometimes maybe you don't need forces because as long as it looks right it's right, but as you jump to higher-level tasks, how much can we rely on them?

我认为你需要一种不同的学习范式。这里有些混淆：LLM、语言和符号，与人类理论构建和人类物理。它们非常不同。人类的目标函数是理解世界并茁壮成长。你观察数据，思考，在世界上尝试一些事情，如果不符合预期，你就在线更新你的理解。人们一直在这样做。这种过程放大并规模化，就是给我们带来 F=ma 和牛顿物理的东西。

I think you need a different learning paradigm. There's a bit of conflation here: LLMs and language and symbols versus human theory building and human physics. They're very different. The human objective function is to understand the world and thrive. You observe data, think about it, try something in the world, and if it doesn't match expectations, you update your understanding online. People do this constantly. That kind of process, writ large and scaled up, is what gives us F=ma and Newtonian physics.

我认为这与我们训练的模型模态（无论是语言还是空间）有点正交。我的说法是，这几乎是一种更高效的学习方式：你根据可用数据假设不同的可能世界，然后通过实验排除不可能的世界，最终确定正确的那个。对我来说，这也是我拥有心智理论的方式——我对你的想法有几个假设，然后尝试采取行动来验证或检查我的直觉。显然，LLM 不会做这些。

And I think that's a little orthogonal to the modality of model that we're training, whether it's language or spatial. The way I put it is almost like this is more efficient learning because you have a hypothesis of the different possible worlds granted by your available data, and then you do experiments to eliminate the worlds that are not possible and resolve to the one that's right. To me, that's also how I have theory of mind, which is like I have a few hypotheses of what you're thinking, and I try to create actions to resolve that or check my intuition. Obviously, LLMs don't do any of this.

心智理论可能还会延伸到情商，而今天的 AI 根本没有触及这一点。我们确实需要它。人们开始过度依赖这些东西，这完全是另一个辩论话题。我不得不问，因为很多人向我们提出了这个问题。我们得抛弃多少东西？序列到序列建模过时了吗？注意力机制过时了吗？我们到底要重新要求多少？

A theory of mind possibly also will break into even emotional intelligence, which today's AI is really not touching at all. And we really need it. People are starting to depend on these things probably too much, and that's a whole topic of other debate. I do have to ask because a lot of people have sent this to us. How much do we have to get rid of? Is sequence-to-sequence modeling out the window? Is attention out the window? How much are we re-requesting everything?

我认为你应该坚持有效的东西。注意力机制仍然存在。你不需要修复没有坏的东西。世界上有很多难题需要解决，但我们一次只关注一个。我认为思考新架构、新范式或截然不同的学习方式很有趣，但你不必因为研究新模态就抛弃一切。

I think you stick with stuff that works. I think attention is still there. You don't need to fix things that aren't broken. There are a lot of hard problems in the world to solve, but let's focus on one at a time. I think it is pretty interesting to think about new architectures or new paradigms or drastically different learning ways, but you don't need to throw away everything just because you're working on new modalities.

我认为序列到序列实际上存在于世界模型中。我们将会看到超越序列到序列的算法或架构。

I think sequence-to-sequence is actually in world models. I think we are going to see algorithms or architectures beyond sequence-to-sequence.

哦，但这里我认为存在一些技术混淆。Transformer 已经为我们解决了这个问题。Transformer 实际上不是序列模型，它本质上是集合模型。这非常强大，但由于许多 Transformer 源于基于循环神经网络的早期架构，而 RNN 确实对一维序列有内置的架构偏好。

Oh, but here I think there's a little bit of technological confusion. Transformers already solved that for us. Transformers are actually not a model of sequences. A transformer is natively a model of sets. That's very powerful, but because a lot of transformers grew out of earlier architectures based around recurrent neural networks, and RNNs definitely have a built-in architectural bias for one-dimensional sequences.

但 Transformer 只是集合的对象模型，它们可以建模许多集合，这些集合可以是一维序列或其他东西。

But transformers are just object models of sets, and they can model a lot of those sets could be 1D sequences or other things as well.

所以你字面意思是指集合论？

So you literally mean set theory?

是的。是的。所以随便。

Yeah. Yeah. So whatever.

是的。所以 Transformer 实际上不是 token 序列的模型，而是 token 集合的模型。在标准 Transformer 架构中，唯一注入顺序的是你给 token 的位置嵌入。如果你选择给它一维位置嵌入，那是模型知道它是一维序列的唯一机制。但 Transformer 块内的所有操作要么是 token 级别的（FFN、QKV 投影、每个 token 的归一化），要么是通过注意力机制的交互，而注意力机制是置换等变的。所以它本质上是 token 集合的架构。

Yeah. So a transformer is actually not a model of a sequence of tokens. A transformer is actually a model of a set of tokens. The only thing that injects order into the standard transformer architecture is the positional embedding you give the tokens. If you choose to give it a 1D positional embedding, that's the only mechanism the model has to know it's a 1D sequence. But all the operators inside a transformer block are either token-wise (FFN, QKV projections, per-token normalization) or interactions through attention, which is permutation equivariant. So it's natively an architecture of sets of tokens.

字面意义上的变换。

Literally a transform.

是的。

Yeah.

用数学术语来说。

In a math term.

我知道我们时间不多了，但我们想让你发言，做一些行动号召。无论是关于喜欢在 World Labs 工作的人，应该申请什么样的人，还是在 World Labs 之外人们应该做哪些对你有帮助的研究，或者你想到的任何其他事情。

I know we're out of time, but we just want to give you the floor for some call to action. Either on people that would enjoy working at World Labs, what kind of people should apply, what research people should be doing outside of World Labs that would be helpful to you, or anything else on your mind.

我确实认为这是一个非常激动人心的时刻，超越语言模型，思考空间智能的无限可能性。所以我们渴望人才，从思考训练大型世界模型等问题的深度研究人员，到构建从训练优化到推理再到产品的系统的工程师，以及优秀的商业、产品思维和上市及商业人才。特别是现在我们已经通过 Marble 将模型展示给世界，我认为我们有机会与更大的人才库合作，解决模型问题并向世界交付最好的产品。

I do think it's a very exciting time to look beyond just language models and think about the boundless possibilities of spatial intelligence. So we are hungry for talent ranging from very deep researchers thinking about problems like training large models of world models, to engineers building systems from training optimization to inference to product, and also good business, product thinkers, and go-to-market and business talents. Especially now that we have exposed the model to the world through Marble, I think we have a great opportunity to work with an even bigger pool of talent to solve both the model problem and deliver the best product to the world.

是的，我也很兴奋让人们尝试 Marble 并用它做很多酷炫的事情。我认为它有很多非常酷的能力和功能，配合得很好。

Yeah, I think I'm also excited for people to try Marble and do a lot of cool stuff with it. I think it has a lot of really cool capabilities and features that fit together really nicely.

在来这里的车上，Justin 和我说人们还没有完全发现一些高级编辑模式。比如打开高级模式。你可以改变瓶子的颜色，改变地板，改变树木。

In the car coming here, Justin and I were saying people have not totally discovered some of the advanced mode of editing. Like turn on the advanced mode. You can change the color of the bottle, change your floor, change the trees.

嗯，我实际上尝试过，但当它说创建时，它只是让我创建一个完全不同的世界。

Well, I actually tried to get there, but when it says create, it just makes me create a completely different world.

你需要点击高级模式。这是一个好的 UI。

You need to click on the advanced mode. It's a good UI.

我们可以改进我们的 UI。记得点击。

We can improve on our UI. Remember to click.

是的，我们需要招人。我们做产品。

Yeah, we need to hire people. We work on the product.

但有一点我们从你们那里明确得到的是，你们在寻找智力上的无畏，我认为这是你们坚持的原则。

But one thing we got that was clear from you guys is you are looking for intellectual fearlessness, which is something I think you hold as a principle.

是的，我的意思是，我们确实是第一个在模型端和产品端都尝试这个的人。

Yeah, I mean we are literally the first people who are trying this both on the model side as well as on the product side.

非常感谢你加入我们。这很有趣。

Thank you so much for joining us. This was fun.

是的，谢谢邀请我们。

Yeah, thanks for having us.

是的。

Yeah.