为何扩散模型将赢得 AI 推理：Stefano Ermon 谈挑战大实验室

嗨，听众们，欢迎回到 No Priors。今天和我在一起的是 Stefano Ermon，他是斯坦福大学的长期教授，现在也是 Inception 的联合创始人兼 CEO。Stefano 在生成模型领域有着极其广泛的研究成果，尤其以扩散模型的奠基人之一而闻名。我们聊了他的公司如何挑战大型实验室，以及为什么速度和效率将在未来几年成为 AI 领域的核心。欢迎 Stefano，非常感谢你来。

Hi listeners, welcome back to No Priors. Today I'm here with Stefano Ermon, who is a longtime Stanford professor and now co-founder and CEO of Inception. Stefano has an extraordinarily broad body of work around generative modeling, but is especially well known as one of the fathers of diffusion. We talk about his company challenging the large labs and why speed and efficiency are going to be the name of the game in AI over the next few years. Welcome Stefano. Thanks so much for being here.

很高兴来到这里。

Great to be here.

我想我们先从你的研究背景开始，以及你是如何最终创立公司的。

I would love for us to just start with a little bit of your research background and how you ended up starting your company.

当然。我基本上整个职业生涯都在研究生成模型。2014 年我作为助理教授加入斯坦福，开始构建生成模型。当时这个研究领域并不热门，模型效果也不太好。我们还在 MNIST 上构建小型生成模型，如果能生成这些有颗粒感的数字图像就已经是很大的成功了。那时候甚至很难在这个主题上发表论文，你必须把训练生成模型说成是一种从无标签数据中学习特征的方法，然后可能帮助提升监督学习的表现，因为那是所有人关心的事情。但后来事情自然发展起来了，所以我是在正确的时间、正确的地点做正确的事情，基本上从一开始我就在这个领域做研究。

For sure. Yeah. I've been doing research in generative models for basically my entire career. I started at Stanford in 2014 as an assistant professor and I was working on building generative models. Back then the research area was not particularly hot. The models were not quite working well. We were still building little generative models over MNIST and it was a big success if you could generate these grainy images of digits. It was even hard to publish papers back then on that topic and you had to justify training a generative model as a way to learn features from unlabelled data that then could maybe help you do better at supervised learning because that was the thing that everybody cared about. But then things took over of course and so I was at the right place at the right time working on the right thing and so I've been doing research in that space since the beginning basically.

除了对这个领域的好奇心，你在 2014 和 2015 年时对模型能做什么有个人期望吗？

Did you have, besides a curiosity in the area, a personal hope for what the models would do back in 2014 and 15?

是的。我一直觉得从无标签数据中学习，构建生成模型是确保你理解数据结构的正确方式。那是我当时思考的方向。我甚至没有梦想过今天这些 LLM 能具备的能力。但我更多是从世界模型的角度思考。我大量研究图像，所以会想，如果我有一个世界模型，我可以想象如果我站起来走出门会发生什么，我可以在脑海中描绘出来，这对做决策和模型预测控制很重要。拥有这种世界模型需要一些生成能力，所以我一直觉得这是正确的方向。我觉得这个问题会非常难，会让我整个职业生涯都忙碌，是一个值得研究的好问题，但当然我大错特错，事情发展得比我预期快得多。

Yeah. I mean I always felt like that was going to be the right way to think about learning from unlabelled data, that building a generative model is really the right way to make sure you understand the structure in the data. That was kind of the way I was getting at. I was not even dreaming about the kind of capabilities that these LLMs that we have today could do. But I was thinking more from a world models perspective. I was working a lot on images and so thinking about, okay, like I have a world model, I can imagine what's going to happen if I were to stand up and walk out the door, like I can kind of picture that in my mind and that's important to make decisions and kind of like model predictive control. Having this kind of model of the world requires some generative capabilities and so I always felt like okay that's the right direction to work on. I felt like this is going to be very hard as a problem, is going to keep me busy for my whole career and it's a good problem to work on and then of course I was very wrong and things evolved much faster than I was expecting.

是的，我觉得这几乎是普遍现象。嗯，那请带我了解一下你当时的研究状态，以及这如何引导你创立公司。

Yeah, I think that's kind of universally true though. Um and sort of walk me through the state of your research and how that led you to start the company.

好的。我最初研究图像的生成模型，从自回归模型开始，它们非常慢而且很模糊，然后是 VAE，再后来 GAN 占据了主导。

Yeah. So I was working on generative models of images initially working on autoregressive models which were very slow and kind of very blurry and then VAEs and then GANs took over.

是的。

Yes.

当时我们对图像生成模型的现状非常不满，比如 GAN 虽然有效，但训练非常不稳定，结果很难复现。所以我们试图看看有没有办法构建一个效果一样好但更原理化的方法。于是我们开始研究基于分数的生成模型，这基本上就是后来在 2019 年我和我的博士生一起开发的扩散模型。我们想出了这个想法：训练一个神经网络去噪图像。如果你能对图像去噪，那你就真正理解了图像的结构，应该有可能基于这些去噪器构建一个生成过程。这基本上成为了现代扩散模型的基础技术，不是从左到右一次生成一个像素，而是从纯噪声开始，逐步细化对象，直到最后得到清晰的图像。这始于 2019 年我的实验室，然后它逐渐占据了整个领域，甚至今天最好的图像生成、视频生成、音乐生成，一定程度上还有很多蛋白质相关的模型，都是基于扩散的。我的团队在加速扩散模型以快速生成样本、提高模型质量方面做了很多工作。既然我们能让它在图像上工作，我开始思考如何让扩散模型在文本或代码生成上工作，以及离散对象，有没有办法超越自回归模型，转向更并行、内置错误纠正的方法。所以我在斯坦福做了一系列研究，让扩散模型在文本和代码生成上工作。我们在 2024 年取得了突破。我们发表了一篇论文，基本上首次展示了在 GPT-2 规模上匹配自回归模型的质量。所以不到十亿参数，仍然相当学术，但我们能够训练一个仍然是 Transformer 的模型作为扩散模型，在相同数据上。我们能够匹配质量，同样的困惑度，你拟合数据的效果和相同参数量的自回归模型一样好。但扩散模型显著更快，因为它是扩散的，你同时输出多个 token。我们能够比自回归模型快 10 倍地生成文本。所以这非常令人兴奋。我真的很想看看如果扩大规模、训练更大的模型会发生什么。所以我创立了 Inception 公司，基本上是为了扩大这项技术，并尝试构建商业规模的基于扩散的语言模型。

And back then we were very unhappy with the state of generative models for images like the GANs were they worked but they were very unstable to train very hard to reproduce results. And so we were trying to see is there a way to build something that is as good as I can but it's more principled. And so we started working on score-based generative models which are basically what eventually became diffusion models back in 2019 with my PhD student. And so we kind of came up with this idea of let's train a neural network to denoise images. And if you can denoise an image then you really are understanding enough about the structure of the image that it should be possible to build like a generative procedure based on these denoisers. And that basically became the underlying technology of modern diffusion models where instead of generating images left to right one pixel at a time you kind of start from pure noise and then you gradually refine the object until you get like a clean picture at the end. And that started back in 2019 in my lab and then it kind of took over the space and even today the best models for image generation, video generation, music to some extent, a lot of the protein stuff they are based on diffusion and so my group has worked a lot on various kinds of diffusion models technique for accelerating them to generate samples very quickly to improve the quality of these models. And so since we were able to get them to work on images, I started thinking about how do we get diffusion models to work on text or code generation and can discrete objects like is there a way to move beyond autoregressive models to something that it's more parallel more with built-in error correction and so I've been doing a bunch of research at Stanford on getting diffusion models to work on text and code generation. We had a breakthrough in 2024. We published a paper basically showing that for the first time it was possible to match the quality of an autoregressive model at the GPT-2 scale. So less than a billion parameters still fairly academic but we were able to train basically still a transformer model as a diffusion model on the same data. We were able to match the quality like the same perplexity and you were fitting the data just as well as an autoregressive model with the same number of parameters. But the diffusion model was significantly faster because it's diffusion because you're outputting many tokens at the same time. We were able to generate text like 10x faster compared to the autoregressive model. And so that felt very very exciting. And I really wanted to see what happens if you scale up if you train bigger models. And so I started Inception, a company to basically scale up the technology and try to build commercial scale diffusion based language models.

现在每个人都看到了扩散模型的输出，尤其是图像。我认为它正日益成为生成短视频的主导形式，在世界其他地方，扩散模型生成的短视频是一种主要的娱乐形式，这里可能也会如此。至少对我来说，即使我关注这个领域已经十多年了，今天能达到的质量仍然令人难以置信。

Everyone has now seen the outputs of diffusion models, in particular images. And I would argue that it's increasingly a dominant form of generated short form video from diffusion models is a dominant form of entertainment in other parts of the world and it will likely become so here. It's kind of unbelievable at least to me even having followed the field for the last decade plus, the quality that is possible today.

嗯，我觉得这挺明显的，对吧，在图像和视频生成里这是个巨大的用例，以至于人们甚至在造硬件来支持更好的性能。嗯，但要说这能适用于其他领域，或者说这是一条有趣的竞争方向，去挑战当前那些全 Transformer 聚焦的 AGI 实验室，这并不直观。嗯，你能提供一些直觉吗？

Um so I think that is kind of obvious right and it's such a big use case in images and and video generation that um folks are even creating you know hardware to support better uh um better performance here. Um it's not intuitive that would work for other fields or that you know this is a interesting competitive direction to uh the um current you know full transformer focused like AGI labs um can you offer some intuition on that?

是的，所以从研究者的视角看，当前世界的状态非常有趣，因为构建生成模型主要有两种范式、两种方式。一种是自回归，你有一个模型预测下一个词或下一个像素，然后从左到右一次生成一个词。另一种是扩散，这是一种粗到细的生成，嗯，类似迭代去噪的生成。正如你所说，我们有连续模态，扩散占主导。还有离散模态，文本和代码，所有大实验室主要都在押注同一种架构——自回归模型。而随着我们越来越走向多模态模型，有一种想法是也许我们会有一个能处理所有模态、了解世界一切的模型，那会是什么架构？会是自回归模型还是扩散模型？嗯，没人知道。我认为陪审团还没定论。在 Inception，我们押注扩散模型，因为我们相信最终重要的是推理时扩展，而且有根本原因说明扩散模型在推理时比自回归模型更好。

Yeah so it's it's a very interesting kind of like state of the world right now from a researcher perspective because like there is like two main paradigms two ways of building generative models. There's auto regressive where you you kind of like have a model that predicts the next tok the next token or the next pixel and then you generate left to right one token at a time. And then there's diffusion which is a course to find generation um kind of like iterative denoising kind of generation and as you said like we have continuous modalities where diffusion dominate. There is discrete modalities text and code where primarily all the big labs are kind of like betting on the same architecture autogressive models and as we move towards more and more like multimodal models and kind of like there is this idea that maybe we'll have a model that can handle all modalities and we'll know everything about the world what architecture will that be like will it be an autogressive model will it be a diffusion model uh nobody knows I think that I think the jury is still out there uh at inception betting on diffusion models because we believe that uh what matters eventually will be inference time scaling and there are fundamental reasons for why diffusion models are better than auto regressive models at inference time.

所以即使你想想自回归模型的故事，2017 年有一个拐点，人们从 RNN 转向 Transformer，对吧？为什么？问题是 RNN 必须顺序处理词，一次一个，训练非常慢，所以人们想出了这个主意：让我们有一个架构，允许你同时并行处理许多词。那就是 Transformer，它在训练上扩展得更好，这促成了 LLM 背后的许多成功。

So even if you think about the story of autogressive models, there was an inflection point in 2017 when people switched from RNNs to transformers, right? And and why why was that? The the the problem was that RNNs had to essentially process tokens sequentially, one at a time and training was very slow and so people came up with this idea of let's have an architecture that allows you to process many tokens at the same time in parallel. And that was a transformer and that that was the thing that scaled better for training and that enabled a lot of the successes behind LLMs.

但如果你现在考虑推理，不是训练，是推理生成，嗯，自回归模型仍然是顺序的，嗯，计算是从左到右一次一个词。你不能生成第 10 个词，直到生成了它之前的所有词。这种工作负载不能很好地映射到 GPU。这种工作负载极度受内存限制。你大部分时间都在内存层次结构中移动权重，做很少的算术运算，这是自回归模型的一个根本问题。那么，如果你想想 RNN 到 Transformer 的对应，自回归模型在推理时的对应物是基于扩散的，是扩散模型，因为扩散模型在推理时的工作负载是同时处理许多词。所以我们在扩散模型推理时的工作负载，基本上和训练时的工作负载非常相似，都是同时并行处理许多词。

But if you think about inference now not training inference generation uh auto regressive models are still sequential uh the computation is one left to right one token at a time. You cannot generate the 10th token until you've generated everything that comes before it. That kind of workload is um does not map well to GPUs. that kind of workload is extremely memory bound. Uh you're spending most of your time moving around weights across the memory hierarchy and you're doing very little arithmetic and that's a fundamental problem of auto regressive models. And so what's the equivalent if you think RNN's transformers auto regressive models the equivalent at inference time is a diffusion based is a diffusion model because a diffusion model is built to have at inference time a workload where you process many tokens at the same time. And so the the the the workload that we have at inference time in a diffusion model, it's basically very very similar to the workload you have for training where you're processing many tokens at the same time in parallel.

所以它本质上是为了有一个能非常好地映射到 GPU 的推理工作负载，GPU 非常擅长这类事情。所以我们押注于尝试构建这种架构，尝试构建在推理时扩展最好的模型，因为嗯，你知道经济性由你能从模型获得的每瓦智能、每美元智能主导。如果你想想推理模型的许多进展，很多都是测试时计算扩展，对吧？所以能沿着这个轴更好地扩展也很重要。即使你想想强化学习后训练，很多瓶颈是生成 rollout，比如让模型探索，然后给轨迹打分，然后根据它发现的东西改进模型。所以推理再次是我们后训练的关键瓶颈。因此，如果你有一个在推理时扩展更好的模型，那么自动地，你在后训练期间也会得到更好的扩展。这就是为什么我们决定押注基于扩散的 LLM，因为它本质上更并行，而苦涩的教训是，更并行的解决方案最终会赢。

And so it's built to essentially have an inference workload that maps really really well to map moles, but the kind the kind of things GPUs do really really well. And so we bet on trying to build the architecture and trying to build the kind of models that will scale best at inference time because um you know economics are dominated by you know the kind of intelligence per watt the intelligence per dollar that you're able to get from the models. If you think about a lot of the advances with reasoning models, a lot of it is scaling test and compute, right? And so being able to scale better along that axis will also matter. And even if you think about RL post training, a lot of the bottleneck is generating rollouts like letting the model explore, you know, and and then scoring the trajectories and then improving the model based on the kind of things it finds. And so inference is again a key bottleneck for our outpost training. And so if you have a model that scales better at inference time, then automatically you're going to get better scaling during our outpost training. And so that's why we decided to bet on a diffusionbased LLM because it's inherently more parallel and the bitter lesson is that the more parallel solution is the one that is eventually going to win.

你是怎么考虑适用性的，或者你跑了哪些实验来攻克离散与连续模态这个难题？因为我认为人们也通过新的分词努力或方法塑造了现有的主导范式，比如让视频和语音工作。嗯，你知道，你并不在同一个以词为中心的范式里。你是怎么让它在这里工作的？

How did you think about um applicability or what experiments did you run in terms of cracking the nut on discrete versus continuous modalities? Because I think people have also uh shaped the existing you know dominant paradigm through new tokenization uh efforts or or uh methods to make video and voice work for example. Um uh it's you know this is you're you're not in the same tokenoriented um paradigm. How do you make it work here?

是的。所以，有大量研究投入到了解如何应用一种本质上与数据中连续结构紧密相关的技术。

Yeah. So, there was a lot of research that uh that went into figuring out how to apply a technology that was inherently very tied to kind of like continuous structure in the data.

所以，如果你想想扩散模型，它学习如何对图像去噪，这对连续数据说得通，因为即使考虑两个像素颜色，你可以在它们之间插值，仍然说得通。但如果你想想两个词，它们之间不一定有东西，对吧？全是离散的，所以需要大量研发和一门必须开发的新科学，来弄清楚如何将这些想法扩展到离散空间。

So, if you think about a diffusion model, it's learning how to den noiseise images and it kind of like makes sense for continuous data because if you think about even two pixel colors, you can kind of like interpolate between them and it will still make sense. But if you think about two words, there is not necessarily something in between them, right? It's all discreet and so it required a lot of R&D and a new science that had to be developed to figure out how to how to extend those kind of ideas to discrete spaces.

关于它今天的效果，你能声称什么？

What can you claim about how well it works today?

我们认为它效果非常好。我们能够训练出基于扩散的 LLM，在质量上与前沿实验室的速度优化模型相当。所以我们的 Mercury 模型在基准测试上与 OpenAI 的 haiku 模型、flash 模型、mini nano 模型相当，嗯，同时显著更快。所以我们已经跨过了，我认为我们从纯研究原型走到了实际使用的东西，比如我们今天在生产中服务这些模型。

We think it works really well. So we've been able to to train uh diffusion based LLMs that are comparable in quality with the the speed optimized models for Frontier Labs. So our Mercury models are on par with the haiku models, flash models, mini nano models from OpenAI if you look at benchmarks uh while being significantly faster. So we've uh crossed I think the the the we went from you know pure research prototypes to things that are actually used like we are serving these models in production today.

我们做了所有的工作，弄清楚如何构建一个服务引擎。你不能像运行 VLM 或 SGLang 那样运行这些基于扩散的 LLM。我们不得不构建自己的服务引擎，我们能够处理大量真实生产工作负载的复杂性。我们已经解决了所有这些挑战，并且我们今天就能端到端地向真实客户交付这种新体验。

We did all the work of figuring out how to even just build a serving engine. You cannot run these diffusion-based LLMs on VLM or SGLang like you have. We had to build our own serving engine, and we can handle a lot of the complexity of real production workloads. We've solved all these challenges, and we can deliver this kind of new experience end to end to real customers today.

实际上，现在正是谈谈 Inception 作为一家公司现状的好时机。比如有多少人，你们实际在服务什么，嗯，研究进展如何？

Actually, a great time to just talk about where Inception is as a company. Like how many people, what are you guys actually serving, um, sort of state of research?

是的，所以……我们大约成立两年了，嗯，大约 50 人，嗯，仍然花大量时间在研发上，有点像在弄清楚训练这些模型的正确方法，嗯，如何加速推理。比如这并不明显，你甚至……如果你考虑一个自回归模型，很清楚在那里你能做的事情不多，比如，好吧，你一次生成一个词，就这样。在基于扩散的模型中，我们知道在推理时有很多不同的可能性来用算力换取质量。就像甚至……如果你考虑图像扩散模型或视频扩散模型，有很多技术可以用来加速采样，比如蒸馏，或者像花哨的嗯微分方程求解技术，允许你从这些模型中非常非常快速地采样。所以有很多关于训练、推理的研究，然后是工程，就像考虑数据混合、演变，嗯强化学习后训练基础设施。就像有很多工作嗯需要发生，以弄清楚如何为这个新模型构建有效的配方。我们尝试尽可能利用现有的东西。例如，它仍然是基于 Transformer 的模型。所以你不必扔掉嗯很多在良好架构上已经完成的工作。嗯我们仍然使用注意力机制，嗯我们仍然使用嗯很多人们创建的公共数据集和评估和基准。所以你知道我们是开始……我们是初创公司，我们尝试灵活，我们尝试尽可能使用现有的东西，并且有点像专注于我们可以产生最高影响的方面，然后我们可以最差异化的地方。现在它是速度。未来谁知道呢？就像基于扩散的语言模型可能比自回归模型更智能，也许显著更智能。就像没人知道。这就是为什么我认为这非常令人兴奋，因为我们正在开发这些非常强大的 AI 系统，但这一切都非常新鲜。这一切都非常新。嗯我怀疑我们已经发现了构建这些系统的最佳方式。肯定有替代方案。肯定有其他构建这些模型的方式。最终，是的，效率将非常重要。就像如果你考虑 AI 工厂，那将如何运作？我认为没人真正知道。能够在这个领域玩耍并思考创造智能的替代方式，我认为这很令人兴奋。

Yeah, so it's... We are about two years old, uh, around 50 people, uh, spending a lot of time still on R&D, kind of like figuring out what's the right way to train these models, um, how to accelerate inference. Like it's not obvious how you even... if you think about an autoregressive model, it's pretty clear there's not a lot of things you can do there in terms of like, okay, you generate one token at a time and that's it. In a diffusion-based model, we know that there is a lot of different possibilities for trading compute for quality at inference time. Like even... And if you think about image diffusion models or video diffusion models, there's a lot of techniques that you can use to kind of like accelerate sampling, of distillation, or like fancy uh differential equation solving techniques that allow you to sample very very quickly from these models. And so there is a lot of research on the training, on the inference, and then engineering, like just like thinking about data mixes, evolves um RL post training infrastructure. Like there is a lot of work that uh that needs to happen to figure out how to how to build recipes that work for this new model. And we try to leverage existing things as much as possible. For example, it's still a transformer-based model. So you don't have to throw away uh a lot of the work that has been done on on good architectures. Uh we still use attention, uh we still use uh a lot of the public data sets that people have created and evals and benchmarks. So you know we're a start... we're a startup, we try to be scrappy, we try to use existing things as much as possible and and kind of like focus on the on the the aspects where we can have the highest impact and then where we can be the most differentiated. And right now it's speed. In the future, who knows? Like it's possible that a diffusion-based language model will be maybe significantly more intelligent than than an autoregressive one. Like nobody knows that. That's why I think this is very exciting, because we're developing these really powerful AI systems but it's all very fresh. It's all very new. Uh I doubt we've discovered the best way of building these systems. There's got to be alternatives. There's got to be other ways of building these models. And and eventually, yeah, efficiency will be very important. Like if you think about the AI factory, like how is that going to work? I think nobody really knows. And just being able to play in that space and thinking about alternative ways of creating intelligence, I think it's exciting.

绝对。我也认为，在一个日益从根本上受算力和供应限制的环境中，嗯，你知道，对于某些人，我不会说没有非常专注的研究关于嗯模型的效率，但它是二阶的嗯考虑因素，对于许多工业研究努力来说，相对于纯粹的规模扩张能力，就像我们所想的那样，对吧？嗯，以及其中的新方法。但是，嗯，我只是认为如果你从根本上相信我们可以使用我们拥有的所有算力，并且会有经济原因这样做，那么我认为对效率的关注将急剧上升，对吧？嗯，你知道，我认为现在许多人认为，对于 AI 领域的公司来说，最重要的决定之一就是我们如何在训练和任何瓦特的最有价值用例之间使用我们拥有的算力。关于这个话题，就像在 26 年 9 月，嗯，速度在哪里获胜？就像人们已经足够关心这个了吗？

Absolutely. And I also think that in an increasingly like fundamentally compute and supply constrained environment, uh the you know for for some I I wouldn't say that there wasn't very focused research on um efficiency of models but it was a second order uh uh sort of consideration for many of the industrial research efforts versus like pure capability scaling on what we've thought, right? Um and and new methods in that. But, uh I I just think if you if you fundamentally believe that we can use all of the power we have and there will be economic reasons to do so, then I think the focus on efficiency is going to go dramatically up, right? Um you know, I I I think many people now argue like one of the most important decisions for um a company in the AI space is like how do we use the power we have between, you know, training and the most valuable use cases for any any watt. On that topic, like here in you know uh September of 26, uh where does speed win? Like where do people care enough about this already?

是的，基本上是延迟重要的应用。我的意思是，一般来说，我认为每个人都关心速度，因为如果你能给我同样的嗯质量但更快，人们总是会选择更快的解决方案。然后我们看到了这一点，甚至来自前沿模型的更快版本。人们愿意支付更多来获得更快的模型，对吧？而且我认为一旦你习惯了快速模型，就很难回头。这有点像宽带，对吧？然后越来越快，如果你能尝试，你知道，人们一旦得到快速模型就无法回头。

Yeah, it is basically applications where latency matters. I mean generally speaking I think everyone cares about speed in the sense that if you can give me the same uh quality but faster people will always pick the the faster solution. And then we're seeing it with this like faster versions of even the the models from Frontier Labs. People are willing to pay more to get access to to faster models, right? And and I think once you get used to a fast model, it's hard to go back. It's kind of like broadband, right? And then get faster and faster and if you if you were able try, you know, people cannot go back once you once you get the fast model.

嗯，有没有你可以公开谈论的客户，嗯，你知道关心这个的？今天。

Uh are there customers that you can talk about publicly that um you know care about this? Today.

有几个我们可以提到，嗯，比如在语音领域，嗯 OpenCall 是一个例子，嗯他们正在构建语音智能体，嗯你知道速度当然，管道就像你有一个 ASR 模型，你有一个 LLM，它有点像做所有的工具调用并弄清楚接下来说什么，嗯它通常必须是一个推理 LLM 才能达到最高质量，然后最后有一个文本到语音嗯组件，嗯速度对他们来说非常重要，嗯他们之前使用嗯在 Cerebras 上服务他们的 LLM。所以他们使用定制芯片来达到他们需要的速度，以向客户提供最佳体验。然后他们切换到我们的基于扩散的 LLM，因为他们基本上可以获得与在定制硬件上运行自回归模型相同的速度。如果你有一个基于扩散的 LLM，它被构建为并行的，在软件层面加速，那么你可以在 Nvidia GPU 上获得相同的速度，嗯，这意味着更多的嗯可用性。我的意思是，GPU 稀缺，但比定制芯片更多，嗯，而且成本更低，质量更高。所以，这是语音领域的一个例子。

There are a few that that we can mention uh like in the voice space for example um OpenCall is an example um they they're building like voice agents um they you know speed of course the pipeline is like you have an ASR model you have an LLM that it's kind of like doing all the tool calls and figuring out what to say next uh it has to be a reasoning LLM typically to to have the highest level of quality and then there is a text-to-speech uh component at the end um speed matters a lot to um they were previously using uh serving their L&Ms on on on Cerebras. So they were using custom chips to get to the kind of speed that they need to to deliver the best experience to their customers. And then they switched over to to our diffusion based LLMs because they can essentially get the same speed as what you would get if you were to run an autoregressive model on custom hardware. If you have a diffusion based LLM that it's built to be parallel, it's accelerating at the software level, then you can get the same speed on Nvidia GPUs, uh, which means much more, um, availability. I mean, GPUs are scarce, but there's more of that than than custom chips, uh, and lower cost, higher quality. So, that's an example in the in the voice space.

是的，我实际上正想问你，你认为这如何与嗯硬件格局互动，鉴于我们现在已经看到了足够多的用例需求，比如是的，我想要一个大的昂贵的芯片，有很多 SRAM，人们会为它的输出付费。是的，在编码和其他用例中。

Yeah, I was actually going to ask you how you think this um interacts with the uh hardware landscape as well given we've seen enough now demand from use cases who are like yes I want a big expensive chip with a lot of SRAM and people will pay for the outputs of that. Yeah, in in coding and other use cases.

是的，当然。当然。我认为硬件是加速事物的一种方式，而好处是

Yeah, for sure. For sure. And I think hardware is one way to to to to accelerate things and the good thing

如果我们能使用现有硬件，软件可能更好。

and software might be better if we can use the existing hardware.

确切。确切。特别是它们是互补的。

Exactly. Exactly. And especially they are complimentary.

这正是令人兴奋的地方：在某种程度上，你从软件中获得的收益，会与你从硬件中获得的收益相乘。

That's the exciting piece, that to some extent the gains you get from the software are multiplicative with the gains you get from the hardware.

也许有一天，人们会开发出更契合你所构建模型的硬件。

And maybe someday people will develop hardware that fits even better the models that you're building.

当然。当然。是的。是的。

For sure. For sure. Yeah. Yeah.

如果我们把视野拉远，看看整个行业的开端，我认为存在一种担忧——如果我说的不对请纠正我——如今很难投资于新架构，因为如果架构或方法上出现进步，它会被拥有大规模算力资源的大公司直接吸收。请谈谈在这种情况下，你如何思考像大卫一样去竞争。

If we just zoom out to the inception in the broader industry, I think there is a vein of concern, and correct me if I'm wrong, that it's very hard to invest in new architectures today because if there are advances in architecture or methods, it will simply be absorbed by large players with the resources to scale compute. Talk to me about how you think about going and competing as David in this situation.

是的，这是一个非常合理的观点，也是我们最关心的事情之一。我认为对我们来说，最初模式就是知识产权、商业秘密，比如我们和我们的研究员为构建这些模型并使其变得更好而拥有的想法。随着我们作为一家公司逐渐成熟，我们不仅仅做纯研究，还开发产品、拥有真实客户，并从现实世界获得模型反馈，原因之一就是通过这样做，我们也在开发对部署这些模型非常重要的组件。例如，服务引擎。如果没有服务引擎，你根本无法在生产环境中服务这些模型。所以，从一开始就强迫自己走出去，端到端地部署一些东西，我们学到了很多关于如何服务这些模型以及如何构建运行这些模型所需软件的知识，这又成为了知识产权。即使你训练了基于扩散的大语言模型，如果没有服务引擎，没有等效的 VLM 来服务它，你仍然会卡住，仍然无法使用它。同样地，我们与真实客户合作，获得模型反馈。我们弄清楚什么有效，什么无效。我们有时从他们那里收集数据。我们根据他们看到的情况创建评估。因此，这又成为了技术护城河的一部分，因为当然，这些事情更难复制。

Yeah, it's a very valid point and something that is also top of mind for us. I think initially for us for sure the mode is sort of like the IP, the trade secrets, like the ideas that we and our researchers have to build these models and make them better. As we mature as a company, and one of the reasons we are not just doing pure research but we're also developing a product and we have real customers and we are getting feedback on the models from the real world, is that by doing that we are also developing components that are also very important to deploy these models. For example, like a serving engine. If you don't have the serving engine, you can't really serve these models in production. So by forcing ourselves from the very beginning to go out and deploy something end to end, we're learning a lot about how to serve these models and how to build software that is kind of like needed to run these models, and that again becomes IP. Even if you train the diffusion-based LLM, if you don't have the serving engine, if you don't have the VLM equivalent to serve it, you're still stuck and you still cannot use it. Along the same lines, we are working with real customers and we're getting feedback on the models. We figure out what works, what doesn't. We collect data sometimes from them. We create evals based on what they're seeing. And so that again becomes part of the technical moat because, of course, those things are a little bit harder to replicate.

如果这个问题在技术上不合理，你可以告诉我，但扩散模型在图像或视频生成中从结构上受益的一点是，你复制的东西在世界上应该有一些一致的结构。语音也是如此，对吧？它是真正可能且最有可能的东西。有些领域是 AI 迄今为止最有价值的。我想说，你用来训练的很多输入数据，比如代码数据，非常杂乱，对吧？有人可能会说，其中很多实际上并没有你正在寻找的正确真实结构。当涉及人类生成的输入数据与图像、视频、语音相比时，你如何看待这一点？

You can tell me if this doesn't make sense as a question to ask technically, but one of the things that diffusion models benefit from structurally in images or video generation is you're replicating something where there should be some consistent structure in the world. Voice as well, right? It is whatever is really possible and most likely. There are some of the fields where AI has been most valuable to date. I'd say a lot of the input data you use to train, like code data for example, it's very messy, right? And one could argue that a lot of it doesn't actually have the correct real structure you're looking for. How do you think about that when it's like human-generated input data versus images, video, voice?

是的，这是个好问题。从根本上说，每当你训练一个生成模型时，无论它是自回归模型，还是在某种程度上甚至是扩散模型，你都是在通过构建压缩方案来识别数据中的结构。这可能不明显，但每当你训练这些模型时，你实际上是在试图通过寻找压缩数据的有效方法来识别共同结构。所以，你能压缩的数据越多，你识别的结构就越多，模式就越多。这就是这些模型的工作原理，神奇之处在于：就像通过预测下一个词，你正在学习关于数据结构的一些东西。无论你使用扩散模型还是自回归模型，都是一样的。两种方法本质上都是在学习压缩方案。当我提到 2024 年的原始论文时，我们展示了我们达到了与自回归模型相当的水平，我们使用的指标基本上是困惑度，它衡量的是你在数据中识别了多少结构。

Yeah, it's a good question and fundamentally if you think about whenever you train a generative model, what you're doing is, whether it's an autoregressive model or to some extent even a diffusion model, you are trying to identify structure in the data by essentially building a compression scheme. That might not be obvious, but whenever you train these models you're effectively trying to identify common structure by trying to find an efficient way of compressing the data. And so the more you can compress the data, the more structure, the more patterns you're identifying. And that's how these models work, which is the amazing thing: just like by predicting the next word, you are learning something about the structure of the data. And that's the same whether you're using a diffusion model or you're using an autoregressive model. Both methods are essentially trying to learn a compression scheme. And when I mentioned the original 2024 paper when we showed that we are achieving parity with autoregressive models, the metric that we're using is basically perplexity, which is a notion of how much structure have you identified in the data.

所以，尽管可能不明显，我们实际上能够在 GPT-2 规模上识别出与自回归模型相同数量的结构。

And so even though it might not seem obvious, we were actually able to identify at the GPT-2 scale the same amount of structure as an autoregressive model.

是的。我认为那个实证结果就在那里，但直觉上会觉得，这个代码并不是——你正在处理的数据集并不是基于物理的，对吧？里面有很多噪声。听起来你认为这是一个可管理的问题。

Yes. I think that empirical result is there, but the intuition would be like, well, this code is not—the dataset that you are working on is not grounded in physics, right? There's a lot of noise in there. And it sounds like you believe that's a manageable problem.

是的，一切都有噪声。所以，只要数字不说谎，只要你能降低困惑度，那就意味着你实际上可以构建一个压缩方案，从而获得那种级别的压缩。因此，结构一定存在，模型一定能够揭示它。然后更多是归纳偏置的问题，比如 Transformer 是识别这些模式的更好方式，还是其他方式？下一个词预测是正确的建模框架，还是更像去噪？这非常是一个实证问题，我认为目前我们甚至没有工具去理解。

Yeah, there is noise in everything. And so to the extent that the numbers don't lie, to the extent that you're able to drive the perplexity down, then it means that you can actually build a compression scheme and that will get you that sort of level of compression. And so the structure must be there and the model must have been able to uncover it. And then it's more a question of an inductive bias, like is a transformer a better way of identifying those patterns or something else? Is next-token prediction the right modeling framework or is it more like denoising? And that's very much an empirical question that I think at the moment we don't have the tools even to understand.

我能问个问题吗？因为你在这里用了语音客户作为例子。一些构建这些 AI 产品的人发现，在语音管道中间放置一个 LLM 的好处之一是，他们更懂得如何在那里更好地进行对齐，或者可控性。我想对于基于扩散的模型，这看起来一定不同。你能谈谈吗？

Can I ask a question just because you used a voice customer as the example here? One of the benefits that some people building these AI products have identified of having an LLM in the middle of this voice pipeline is they understand how to do alignment a little bit better there, or controllability. I imagine that has to look different for a diffusion-based model. Can you talk about that?

是的，这是一个关键的价值主张，他们总是关注的一点是，他们提供的很多价值在于像 harness 这样的东西，确保模型确实在做正确的事情。基于扩散的 LLM 的有趣之处在于，我们构建的一切都是向后兼容的。所以 API 仍然是一样的。它仍然与 OpenAI 兼容，文本输入，文本输出。而且碰巧我们训练的模型擅长遵循指令。它们擅长输出，如果你使用 JSON，结构化输出。它们能处理所有这些事情。而且它足够好。事实上，它比他们之前使用的模型更好。所以他们仍然能够通过使用 Mercury 为客户提供那种级别的服务。

Yes, that's a key value proposition and one of the things that they always look into is to what extent a lot of the value they provide is like the harness and making sure that the models indeed are doing the right thing. And the interesting thing about a diffusion-based LLM is that we've built everything to be backwards compatible. So it's still like the API is the same. It's still OpenAI compatible, text in, text out. And it so happens that the models we've trained are good at following instructions. They're good at outputting, if you're using JSONs, structured outputs. They can handle all of those things. And it was good enough. It was better in fact than the models they were using before. And so they are still able to provide the kind of level of service to their customers by using Mercury.

嗯，如果接口相同，你可以直接使用相同的技术栈，那就非常简单了。

Well, very simple if the interfaces are the same and you can just use your same stack.

是的。

Yeah.

我觉得这其实是个非常有意思的点：我们知道扩散模型通常比自回归模型更容易控制。原因在于，如果你想想自回归模型，你得等到生成完整个对象，才能知道它是否满足某个约束、是否对齐，或者是否符合你关心的任何目标函数。比如你在生成一个分子，你关心它的溶解度，那你就得等到整个分子生成完，才能用某个奖励函数给它打分。但扩散模型是更粗到细的生成，所以从一开始你就可以逐步引导它。

I think that's actually a very interesting point: we know that diffusion models are typically easier to control compared to autoregressive models. The reason is that if you think about an autoregressive model, you have to wait until you've generated the full object to know whether or not it satisfies a constraint, or whether or not it's aligned, or whatever objective function you care about. Maybe you're generating a molecule and you care about solubility, and then you have to wait until you have the full object to be able to score it with some reward function. But a diffusion model is more coarse-to-fine generation, so from the very beginning you can progressively steer it.

从一开始你就知道这个对象是不是你想要的那种东西，你可以根据外部奖励函数或一组约束来引导生成的方向。

From the very beginning you know kind of like is this object the kind of thing I want or not and you can steer the generation in the direction provided by an external reward function or some set of constraints.

所以至少学术文献中有大量证据表明扩散模型更容易控制，而且有不同的引导方式，这些方式在自回归模型中根本不可能实现。所以这会是模型的一种不同接口，可能对自回归模型来说甚至不存在。我认为这是我们一直在思考的问题：围绕自回归模型无法提供的新能力，我们能构建什么样的正确产品体验。

And so there is a lot of evidence in the academic literature at least that diffusion models are easier to control and there are different ways of steering them that are just not possible with autoregressive models. So that would be a different interface for the model that maybe might not even be available for autoregressive models. I think that's something that we've been thinking a lot about: what would be the right product experience that we can build around new capabilities that are just not provided by autoregressive models.

你想象 Inception 的模型在规模扩大后会具备哪些当今模型没有的能力，除了性能之外？

Are there capabilities that you imagine in Inception's models having at scale that today's models don't have beyond performance?

对，这就是我们不知道的地方，对吧？这就是涌现。是的。就像现在，楔形速度，我们知道它们快得多。这是最初的赌注，因为那更容易测试。也容易衡量，而且显然有价值，对吧？但正如我们——这就是为什么我觉得如此兴奋——随着我们对如何训练这些模型了解得越多，我们不知道会发现什么。而且学术文献中有相当多的证据表明，基于扩散的模型比自回归模型更数据高效。直觉就像，如果你想想训练一个扩散模型，你是通过去噪来学习的。你从一张图像开始，添加噪声，然后学习如何去除噪声。所以它实际上是在做数据增强，因为同一张图像被许多噪声视图增强了。

Yeah, that's the thing we don't know, right? And that's the it's emergent. Yes. Like right now the wedge speed, we know they are much faster. That was the initial bet because that was easier to test. It's also easy to measure and it's obviously valuable, right? But as we—that's why I find it so exciting—as we learn more about how to train these models, we don't know what we're gonna find. And there is a decent amount of evidence in the academic literature, for example, that diffusion-based models are more data efficient compared to autoregressive models. The intuition is just like if you think about training a diffusion model, you're learning by denoising. You start with an image, you add noise, and then you learn how to remove the noise. So it's effectively doing data augmentation in the sense that the same image is augmented by many noisy views.

好的。

Okay.

是的。所以它们往往更数据高效一些，如果这在规模上成立，那么也许我们会进入数据更少的任务。

Yeah. And so they tend to be a little bit more data efficient and so if that holds up at scale and then you believe then maybe we'll get into tasks where we have less data.

是的。当数据成为瓶颈时，那就变得更有趣了。对。

Yeah. Where data becomes the bottleneck then it becomes more interesting. Right.

所以我们拭目以待。这就是为什么它如此令人兴奋，因为事情在变化，这项技术如此重要和有价值，拥有差异化的东西我认为会创造价值。

And so we'll see. That's why it's so exciting because things are changing and this technology is so important and so valuable that having something differentiated I think will create value.

如果我们预测，你知道，五年——在 AI 世界里那其实太长了——如果我们预测两年，你觉得扩散模型和传统模型之间的工作负载划分会怎样？

If we project out, you know, five years—that's actually way too long in AI world—if we project out two years, like what do you think is the workload split between diffusion and traditional models?

我认为我们还没有达到前沿智能水平，而且我认为很多工作负载确实需要前沿智能，但根据我的估计，即使你只是去 OpenRouter，它有一种很好的方式来看所有不同的用例，你可以看到研究、对话、编码、软件工程和日志处理——他们基本上有一个很好的硬分类法——我做了一些估计，我认为大约有 20% 到 30% 的情况下延迟非常非常重要，所以至少作为下限，我认为这可以由那些在给定延迟预算内为你提供最高可能质量的模型来解决。

I think we're still not at the frontier level of intelligence and I think a lot of the workloads do require frontier level intelligence but in my estimates like even if you just go to OpenRouter has this very nice way of looking at all the different use cases and you can see the research and conversational and coding and software engineering and log processing—they have a nice hard taxonomy basically of tasks—and I was doing some estimates and I think there is like between 20 and 30% where latency is really really important and so at the very least as a lower bound I think it could be addressable by models that within a given latency budget will give you the highest possible quality.

然后你知道所有技术方法都有权衡，使用扩散模型的挑战是什么？

And then just you know all technology approaches have trade-offs what are the challenges of working with diffusion models?

是的，这是一个不同的技术栈，所以挑战之一是我们必须在内部构建很多东西，而且没有成熟的生态系统，如果你想想服务引擎或内核，很多 things 必须在内部开发，所以没有什么真正开源的，或者有一些开源模型但它们不是特别好，所以这使得部署、让客户尝试他们不习惯的东西变得更困难一些，所以这一直是挑战之一。

Yeah, it's a different stack and so one of the challenges that we had to build a lot of things in house and there is not a mature sort of ecosystem of if you think about the serving engine or like kernels like a lot of the things had to be developed in house and so there is not really anything open source or there are some open source models but they're not particularly good and so that makes it a little bit more difficult to deploy to get customers to try things they're not used to it so that's been one of the challenges.

我想这也反映了外部情况，对吧，你知道在一个人们有一定成熟度、关心成本和性能的环境中，尽管对于某些用例你从一开始就会关心成本和性能，但对后训练的兴趣越来越大，对吧，所以我想在一个新架构中那更具挑战性。

I imagine that also reflects externally right you know in a landscape where folks at some sophistication where they would care about cost and performance and though you might for certain use cases you will care about cost and performance from the beginning there's an increasing amount of interest in post training right and so I imagine in a new architecture that's even more challenging.

所以我们必须构建自己的技术栈来做 SFT、做 RLHF、做 RL——我的意思是这在某种程度上成为了知识产权。所以我们决定不完全开源的原因之一就是为了让知识产权更接近我们，不开放它。但这样也有缺点，比如社区贡献的机会更少。更难采用。很难做本地部署，所以这两种选择都有利弊。

So we had to build our own stack for doing SFT, for doing RLHF, doing RL—I mean that becomes IP to some extent. So it's one of the reasons we decided not to open source everything was really to keep the IP a little bit closer to us and not opening it. But then there are downsides like there is less opportunity for the community to contribute. It's harder to adopt. It's hard to do on-prem kind of deployments and so there are pros and cons with the two choices.

你能谈谈你们自己训练的规模吗，然后像当前或期望的，然后 50 个人，我相信你们在继续招聘，为什么研究人员或工程师或其他人应该考虑投资这个方向或在 Inception 工作？

Can you talk about you know scale of your own training and then like current or aspirational and then at 50 people I'm sure you're continuing to hire like why researchers or engineers or others should consider investing in this direction or working at Inception?

是的，所以我们无法分享太多关于训练、模型大小或算力之类的信息，这有点像商业机密，但我们正在继续推动前沿，如果你想有机会塑造这个领域的方向，Inception 是一个很棒的地方。这仍然是一个相对较小的领域。有很多东西有待发明，所以很多决定来 Inception 而不是加入其他实验室的人，真的是因为他们想要有所有权，他们喜欢发明新东西。他们喜欢在一个更绿地的空间，有更多机会尝试事物。那里已知或可用的东西更少。

Yeah so we're not able to share much about the training the size of the models or the flops or all of that it's kind of like a trade secret but we are continuing to push the frontier and Inception is a great place to be if you want to have an opportunity to shape the direction of the field. Like it's still a relatively small field. There is a lot to be invented and so a lot of the people that decide to come to Inception instead of joining one of the other labs is really that they want to have ownership and they like to invent new things. They like to be in a space where there's more of a green field and more opportunities to try things. There's less that it's known or available out there.

它更开放一些，所以我们往往会吸引那类人。我觉得有一件事既令人兴奋，又让研究圈的朋友们有些绝望，那就是在研究领域本身，利用模型进行递归式自我改进的能力。鉴于你在做一个非常不同的方向，你对此怎么看？

It's a little bit more open-ended and so we tend to attract those kind of people. I think one thing that is both exciting and causes some despair amongst research friends is the ability to use models for recursive self-improvement in the research field itself. Given you're working on a very different direction, what is your view on this?

是的。我的意思是，这是我们——

Yeah. I mean, it's something that we—

我的意思是，听起来你仍然觉得你和你的团队还有工作要做。

I mean, explicitly it sounds like you still feel there's work for you and your team to do.

哦，是的。是的。我觉得我们还没到那一步。也许我们无法接触到其他人拥有的模型，但我觉得仍然有——当然，我们大量使用模型，它加快了我们迭代、尝试想法的速度。我们使用来自前沿实验室的模型，效果很好。它大大加速了我们的开发过程。与此同时，我认为至少现在——我不知道六个月或一年后会怎样——但就目前而言，人类的创造力仍然非常重要，能够提出正确的想法、修剪空间并确定更有前景的方向，对我们来说一直非常重要。

Oh, yeah. Yeah. I think we're not there yet. Maybe we don't have access to the models that other folks have, but I feel like there is still—we use models a lot, of course, and it has accelerated the speed at which we can iterate, try ideas. We use models from frontier labs, and it's been great. It has accelerated our development process a lot. At the same time, I think at least right now—I don't know what it's going to be in six months or a year—but right now, human ingenuity is still super important, and the ability to come up with the right ideas and prune the space and identify directions that are more promising has been really important to us.

50 人对一家全栈的研究服务产品公司来说，或者不管你怎么描述它，都不算多。你们如何组织，你如何考虑在这里分配资源？

50 people is not that many people for, let's say, a full-stack research-serving product company, or however you would think about describing it. How do you organize and how do you think about how you allocate your resources here?

这是一个小团队，但每个人都很有才华，工作非常努力，而且我们可以使用智能体，这让我们效率高了很多。所以我认为人数足够做很多事情。事实上，我常常觉得瓶颈更多是算力而不是人力。但没错，团队的组织方式是这样的：实际上有一个产品团队负责平台，与客户合作，让他们成功使用我们的模型。所以基本上有一个团队在服务当前最佳版本的模型，然后有一个团队在构建下一个版本的模型，包括训练、强化学习、推理，那更偏向研究。

It's a small team, but everyone is very talented and they work very hard, and we have access to agents that are making us a lot more productive. So I think the numbers are sufficient to do a lot. In fact, often I feel like the bottleneck is more compute than people. But yeah, the team is organized like there is a product team effectively that is handling the platform and working with customers to make them successful with our models. So there is basically a team that is serving the current best version of the model, and then there is a team that is building the next version of the model, and that includes training, RL, inference, and so that's more research.

Stefano，最后一个问题。24 年那篇论文是一个非常有趣的结果，引起了很大轰动。你在这个领域工作了很长时间。很多人会说，在这个能够大规模扩展资源的时代，学术 AI 研究面临很大挑战。鉴于你也围绕它创办了一家商业公司，这在某种程度上确实如此。但在你真正取得那些 24 年的成果之前，你如何对自己所从事的方向会产生影响或很有前景抱有信心？

Stefano, one last question for you. The '24 paper was a super interesting result, made a big splash. You've been working in this field for a long time. A lot of folks would say that academic AI research is very challenged in this era of being able to scale resources a great deal. This is certainly true to some degree, given you started a commercial company around it as well. But how did you get confidence in the directions that you were working in having impact or being promising before you really had those '24 results?

是的。我认为这是一系列成果，是我在实验室里一直在做的。不一定——当然，我们在实验室做的早期扩散模型工作。我们展示了在我们能在学术规模上训练的模型规模下，我们能够击败 GAN，同时更加稳定，然后整个事情接管了，变成了 Stable Diffusion、Midjourney，所有这些都始于学术界、我的实验室里开发的想法。但这并不是唯一的一个。我是共同导师，所以例如我参与了 Flash Attention，对吧？这是另一个来自学术界的成果，最终在工业界产生了巨大影响。或者另一个例子是 DPO，那是另一个项目，最初是我组里的一个轮转项目。它是一种用于对齐 LLM 和扩散模型以及各处的算法，对吧？这同样是完全在学术界开发的，只是基于一个巧妙的洞察，即那个问题中一些有趣的数学结构，让你能够以一种非常不同且更高效的方式对这些模型进行后训练和对齐，对吧？所以那里有宝石，有很多机会找到解决重要问题的新更好的方法。学术界的好处之一是它允许你下这些逆势的赌注。正如你所说，挑战在于我们可能没有足够的资源，而且资源永远不够，如果我们有更多算力，我们就能更高效。但你可以接触到优秀的学生，每个人都在尝试开发新东西，人们不怕下赌注，这就是为什么学术界如此有影响力，我认为。即使在 AI 领域，很多重要的想法都源于学术界，甚至是在学术界创造的。

Yeah. And I think it was a collection of results that I had been working on in my lab. Not necessarily—of course there's the early diffusion work that we did in the lab. We showed that at the kind of scale of models that we could train academically, we were able to beat GANs and then while being much more stable, and then the whole thing took over and became Stable Diffusion, Midjourney, all of that started from ideas that were developed in academia in my lab. But that's not the only one. I was co-advisor, so I worked on Flash Attention for example, right? That's another thing that came out from academia that then eventually had a huge impact in industry. Or another example is DPO, that was another project started as a rotation project in my group. It's an algorithm that is used to align LLMs and diffusion models and everywhere, right? And that's again something that was developed entirely in academia and it was just based on a clever insight, some interesting mathematical structure that you have in that problem that allows you to come up with a very different and more efficient way of post-training and aligning these models, right? So there are gems, there are lots of opportunities for finding new and better ways of solving important problems. One of the nice things about academia is that it allows you to take these contrarian bets. As you said, there is the challenge that maybe we don't have enough resources, and there's never enough resources, and if we had more compute we could be more efficient. But you have access to amazing students and everyone is trying to develop the new thing, people are not scared about taking bets, and that's why academia has been so impactful, I think. Even in the AI space, a lot of the important ideas have roots or even were created in academia.

太棒了。非常鼓舞人心。非常感谢你来到这里，Stefano。

Awesome. Super inspirational. Thanks so much for being here, Stefano.

非常感谢你们邀请我。

Thank you so much for having me.

在 Twitter 上找到我们，账号是 no prior pod。如果你想看到我们的脸，请订阅我们的 YouTube 频道。在 Apple Podcasts、Spotify 或任何你收听的地方关注节目。这样，你每周都会收到新一集。并在 no-briers.com 注册邮件或查找每集的文字记录。

Find us on Twitter at no prior pod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week. And sign up for emails or find transcripts for every episode at no-briers.com.