AI2 发布完全开放的 Almo 3 模型，挑战闭源 AI

Meta 的领导层发生了重大变动，Llama 的未来未知。因此，出现了一个巨大的影响力真空，被 Qwen、DeepSeek、Kimi、Moonshot 等公司填补，它们正试图用开放模型构建东西，这是一个重大转变。我们今天发布 OLMo 3 系列，就像我们之前发布的每一个模型一样，我们不只是发布最终模型。我们公布了所有细节。这是第一个完全开放的推理模型，我们展示了在基础模型上做强化学习，以及从更大的思考模型中蒸馏。在美国内部有很多讨论，认为我们有充分的理由拥有整个技术栈，包括开放模型。有些人真的开始意识到这一点。大家好，我是 Matt Turk。欢迎收听 Mad 播客。今天我们有来自艾伦人工智能研究所的 Nathan Lambert 和 Luca Sanei 的特别节目，庆祝 OLMo 3 模型系列的发布。在大多数开源发布只是开放权重的时代，AI2 全力投入真正的开放性：模型、数据、配方和中间检查点。在这期对话中，我们将剖析 OLMo 3 的架构、思考模型的兴起，以及美国开源努力与快速发展的中国巨头（如 Qwen、DeepSeek 和 Kimi）之间日益高风险的竞赛。这是一次罕见的、完全透明的视角，展示现代 AI 模型的实际工作原理。请享受这期与 Nathan 和 Luca 的精彩节目。两位，欢迎来到播客。今天有大新闻，对开源 AI 来说是个大日子。请介绍一下你们今天发布的内容。

There was a big change in leadership at Meta and Llama's future is unknown. So there's this big vacuum of influence which has been absorbed by the likes of Qwen, DeepSeek, Kimi, Moonshot in terms of who's trying to build things with open models and that's a big shift. We're launching OLMo 3 family today and just like every single model that we released before, we're not just releasing the final models. We're putting out all the details. It's like the first fully open reasoning model where we show doing RL and base models and distilling from bigger thinking models and there's a lot of discussion within the US that there's good reason that we should own the whole technological stack and that includes open models. There are people that are really starting to wake up to this. Hi, I'm Matt Turk. Welcome to the Mad Podcast. Today we have a special episode with Nathan Lambert and Luca Sanei from the Allen Institute for AI for the release of the OLMo 3 model family. At a time when most open source releases are just open weights, AI2 is going all in on real openness, models, data, recipes, and intermediate checkpoints. In this conversation, we break down OLMo 3's architecture, the rise of thinking models, and the increasingly high stakes race between US open source efforts and fast advancing Chinese powerhouses like Qwen, DeepSeek, and Kimi. This is a rare, fully transparent look at how modern AI models actually work. Please enjoy this great episode with Nathan and Luca. Guys, welcome to the pod. Big announcement today and a big day for open source AI. Walk us through what it is that you're releasing today.

感谢邀请。是的，我们今天发布 OLMo 3 系列。这是我们最新的开源模型系列。我们有 7B 模型和 32B 模型。我们有能思考的模型，能遵循指令和使用工具的模型。就像我们之前发布的每一个模型一样，我们不只是发布最终模型。我们发布了获得这个模型的完整配方。所以数据、中间状态、评估框架、所有细节、人们需要知道的所有信息，来制作这样的模型。具体来说，是 OLMo 3 基础 7B 和 32B。那么这些是什么？我们大概有五个旗舰检查点要发布。其中两个是基础模型。这意味着这些模型在训练以响应用户指令之前的状态。所以这些非常适合那些想要利用我们在预训练这些模型上花费的大部分算力，然后根据他们的用例进行定制的人。所以这是两个基础模型。有一个较小的，更高效，大约需要一个 GPU 来针对一个用例进行微调。然后有一个更大的 32B，大约需要一盒 GPU 来微调。在此基础上，我们有针对各种用例的微调后训练模型。所以有几个模型是思考模型。有 OLMo 7B Think 和 OLMo 32B Think。这些模型，就像许多推理者或前沿模型一样，可以在推理时花费算力来思考问题并解决它，最后给你答案。我们还发布了一个 7B Instruct 模型。这是一个更直接的模型，响应更快。所以它非常适合批量数据处理或需要低延迟响应的用例。

Thanks for having us. Yeah, we're launching OLMo 3 family today. So this is our latest family of open source models. We have a 7B model, a 32B model. We have models that can think, models that can follow instruction and use tools. And just like every single model that we released before, we're not just releasing the final models. We are releasing the entire recipe we follow to get this model. So the data, the intermediate states, the evaluation frameworks, all the details, all the bits that people need to know to make models like this. Specifically, this OLMo 3 base 7B and 32B. So what are those? Probably we have say five flagship checkpoints that we're putting out. Two of them are base models. That means these are models before they get trained to respond to user instruction. So these are really good for folks who want to take the bulk of our compute that we spent in pre-training these models and then customize them for their use cases. So these are two base models. There's a smaller one, more efficient, that takes about one GPU to fine-tune for a use case. And then there's a larger 32B that takes about one box of GPUs to fine-tune. And then on top of that, we have our fine-tuned, our post-trained models for various use cases. So there are a couple of models that are thinking models. So there's OLMo 7B Think and OLMo 32B Think. These are models that, just like a lot of the reasoners or frontier models out there, they can spend compute power at inference time to think through a problem and solve it and then give you an answer at the end. And also we are releasing a 7B Instruct model. This is a more immediate model that gives you faster responses. So it's really good for bulk data processing or use cases where you want low latency in your responses.

我想为这些内容增添更多色彩。我觉得 Luca 低估了他们的基础模型。我们稍后会详细讨论，但今年有更多人发布了开放模型，尤其是大型开放模型，但有些人开始不发布基础模型。我们有一堆 DeepSeek 大小的巨型 MoE 基础模型和一堆小型基础模型。但例如，被大家视为研究和行业标准的 Qwen 3，他们没有这个 32B 基础模型。所以这个基础模型在质量上与最好的可用模型相似，比如 Qwen 2.5 32B 仍然是最好的基础模型。好处是我们拥有所有数据，所以人们可以实际进行某种持续预训练，并希望使修改和理解行为变得更容易。这对我们来说很令人兴奋，尽管它实际上是潜在的最佳同类产品。它也是完全开放的，这在 AI2 我们不常这么说。有时是‘哦，我们复制了这个，现在你可以自己做。’这实际上是一件好事。然后是 7B 模型，Luca 说它曾经是一个巨大的行业标准，有太多这样的模型。它仍然是一个标准尺寸类别，但那里的模型不像以前那么多了。而这个，尤其是指令模型，更不常见。它再次跻身该尺寸类别中世界上最好的模型之列。我只是想到 Llama 3.1 8B 是有史以来 Hugging Face 上使用最多的模型之一。而这个应该更好。在我们的测量中，我们看到它比 Llama 3.1 8B 更好。希望这对人们来说成立。我们可以发布更多并修复它。但这只是试图给出——我们可能不在前沿规模，但这些仍然是世界上广泛使用的东西，然后它是第一个完全开放的推理模型，我们展示了在基础模型上做强化学习，从更大的思考模型中蒸馏，以及今年人们已经看到很多次的所有这些东西。我认为‘再得到一个思考模型’就像这个有什么用？但我们拥有所有数据，我们向人们展示它的用途。所以我认为很多时候，尤其是我们的开放后训练，数据集变成了标准。就像我们去年用于 OLMo 2 的 Tulu 3 数据集，它在思考机器 tinker API 中，我们希望人们使用这些数据，根据需要修改它，并查看不同的训练阶段。

I want to add more color to these things. I think Luca is underselling their base model. We're going to talk more about this, but over this year a lot more people have been releasing open models and especially large open models, but some people are starting to not release base models. We have a bunch of DeepSeek-size giant MoE base models and a bunch of small base models. But for example, Qwen 3 which everyone accepts as a research standard and an industry standard, they don't have this 32B base model. So this base model is similar in quality to the best available, which is like Qwen 2.5 32B was still the best base model. The upside is that we have all the data so people can actually do some sort of continued pre-training and hopefully make it a bit easier to modify and understand the behavior. So that's exciting for us, though the actual potentially best-in-class thing. It's also a fully open thing which is not something we get to say a lot at AI2. Sometimes it's like 'oh we replicated this and now you can do it yourself.' Like this is actually a good thing. And then 7B models which Luca was saying used to be this huge industry standard where there's just so many of them. It's still a standard size category, but there aren't quite as many models there as there used to be. And this, especially the instruct models, are less common. And this is up there with one of the best in the world at that size category again. And I just think of this like Llama 3.1 8B is one of the most used models on Hugging Face of all time. And this should be better. In our measurements, we see it as being better than Llama 3.1 8B. Hope that holds up for people. We can release more and fix it. But that's just like trying to give—we might not be at this frontier scale but these are things that are still widely used in the world and then it's like the first fully open reasoning model where we show doing RL and base models and distilling from bigger thinking models and all these things that people have seen a ton of times throughout the year. I think 'get another thinking model' is like what is this one for? But we have all the data and we show people what to use for. So I think a lot of times with our especially open post-training it's just like the data sets become a standard. So it's like our Tulu 3 data set from last year which we used for OLMo 2 is in like the thinking machines tinker API and we want people to use this data, modify it how they need to, and look at the different training stages.

很好，很好，太棒了。那么关于数据点，谈谈 DOLMA 3。

Great, great, fantastic. So to the data point, talk about DOLMA 3.

DOLMA 3 是我们在 OLMo 3 预训练中使用的数据。所以它是我们用来创建基础模型的数据。它实际上由三部分组成。有一个预训练池。这就像一个大约 10 万亿 token 的池子，我们有一个算法，也是完全开源的，从中采样大约 6 万亿 token 用于训练。我们那里有一些新技术。有趣的是：我们采用这种技术，不是重复随机文档来获得更多训练 token，而是智能地重复最有价值的 token。所以我们有那部分。还有一个较小的子集，我们在中间训练阶段使用。

DOLMA 3 is the data that we use in pre-training for OLMo 3. So it's what we use to create the base model. It's really three parts. There's the sort of pre-training pool. This is like a pool of about 10 trillion tokens from which we have an algorithm, also fully open source, to sample about 6 trillion tokens that we use during training. And we have kind of new techniques there. It's interesting: we do this technique where instead of repeating random documents to get more training tokens, we intelligently repeat the tokens that have the most value. So we have that part. There's a smaller subset that we use during this mid-training phase.

所以这是一个更专注的数据集，包含大量数学、高质量代码以及一些知识片段，我们希望模型能学到这些。最后，我们还有一组文档，特别有助于让模型能够处理长上下文。我对这个非常兴奋，因为从历史上看，在公开可用的数据中，用于构建语言模型的长文档数据并不多。所以这些文档是我们自己爬取的，它们是 PDF，科学类居多，主要是科学 PDF，在互联网上公开可爬取。我们有一个处理流程，但也是开源的，所以一切都是开源的，可以将它们转换为纯文本。这些文档不像网页那样短，95%的网页低于 3000 个 token，而这些文档相当长。我们有大约 6000 亿个 token，长度超过 8000 个 token。所以这些对于人们开发让模型理解超长输入的其他方法非常有用，而这通常是目前开源社区无法做到的，除非是大型实验室，他们可以获取足够长的数据来进行这个阶段。

So this is like a more focused dataset with a lot of math, high quality code, sort of knowledge tidbits that you want the model to pick up. And then finally we have a set of documents that are particularly useful to make models able to work with long context. Really excited about this one because historically, of the data that is available openly out there for people to build their language model, you don't have a lot of long document data. So these are documents that we crawl ourselves, they're PDFs, scientifically they're mostly like science PDFs, they're openly available on the internet for crawl. We have a pipeline, but it's also open source, so everything's open source to turn them into plain text. And of those, we have instead of web pages that are kind of short, like 95% of web pages are below 3,000 tokens, these are quite long. We have about 600 billion tokens that are longer than 8,000 tokens. So these are really good for people to develop other ways for models to understand very long inputs, which is typically something that people are not able to do today in the open unless they are a big lab and they can acquire data that is long enough to do this phase.

好的，谢谢。你提到了其中一些，但请谈谈性能和效率。

Okay, great. Thank you. You alluded to some of this but talk about performance and efficiency.

性能，你知道，衡量基础模型的性能非常困难。所以对于指令模型和推理模型，Nathan 会提供更多关于对比基准的信息，但基础模型确实很好。正如 Nathan 所说，它的水平是，没有多少人会发布基础模型。所以我们在比较方面有些受限，但它达到了 Qwen 2.5 3 或 Gemma 3 的水平。某些能力上，它们可能稍好一些，有些方面则优于其他模型。但基础模型的绝对性能并不像最终的指令模型那么重要。你希望模型处于一个合适的区间，足够强大，这样你的后训练团队就能在检查点上施展魔法，让它变得非常非常好。我想说，在后训练方面，我们是不以 Qwen 3 为起点的最佳模型，而且我们有理由说它们与 Qwen 3 相当。在某些基准测试上我们击败了它们，而在另一些基准测试上它们遥遥领先。我认为很多人不知道 Qwen 3 在训练数据中具体放了什么，所以我们不知道是否有些基准测试他们最大化得比我们更努力。我的意思是，我们试图在基准测试上爬山，让我们的模型变得更好。我认为总是有一定程度的这种情况，但在很多方面它们处于同一水平。我们希望有一些用例，使用 Qwen 38B 或 32B 的人愿意切换过来，从中获得一些价值，并可能根据自己的用例进行修改。但 Qwen 也发布了很棒的模型。所以这是一场永无止境的攻坚战，激励你做得更好，试图接近并与之竞争。就像他们发布了 Qwen 3VL，他们的视觉模型，在纯文本基准测试上比他们 4 月份发布的模型好得多。所以，好吧，那是新的基线，而大多数人不知道，因为他们认为那只是一个视觉模型，但实际上它是一个更好的文本模型。好吧，标准总是在提高。但在 7B 规模上，Nvidia 有 Nemotron Nano V2，这是一个 9B 混合模型，我认为它几乎等同于我们的 7B 模型。这些都是好模型。在这个规模范围内，真正强大的模型并不多。所以我认为我们很高兴能处于这个位置，也很高兴指出其他人在做很棒的工作。我们不能忽视 Qwen。那是一个失败的战略。

Performance, you know, it's very hard to measure performance of a base model. So for the instruct and the thinking, Nathan will have more info about comparative benchmarks, but the base model is really good. It's at a level, as Nathan was saying, not that many people release the base model. So we're kind of limited there in terms of comparison, but it's at the level of Qwen 2.5 3 or Gemma 3. Certain capabilities they have maybe a little bit better on some capabilities, better than some others. But the absolute performance of the base model doesn't matter so much as the instruct model at the end. You want to be in the right band where the model is capable enough that then your post-training team can do magic on the checkpoint and make it really, really good. I would say in post-training, we're the best models that don't start with Qwen 3, and we're reasonable to say that they are comparable to Qwen 3. On some benchmarks we beat them, and on some benchmarks they're way ahead. I think a lot of people don't know what Qwen 3 puts exactly in the training data, so we don't know if some benchmarks they benchmark to max a little harder than we did. I mean, we try to hill climb on benchmarks to make our model good. I think there's always some level of this, but it's in the same ballpark in plenty of things. We're hoping that there are use cases where people that use Qwen 38B or 32B are willing to switch over and get some value out of this and maybe modify it to their own use cases. But Qwen also releases great models. So it's an ever-ending uphill battle that motivates you to do better to try to get close and compete with what they're doing. It's like they release these Qwen 3VL, their vision models, and on text-only benchmarks it's way better than the models they released in April. So it's like, okay, that's the new baseline, and most people don't know about it because they think it's just a vision model, but it's actually a much better text model. And it's like, okay, the bar is always rising. But at 7B scale, Nvidia had Nemotron Nano V2, which is a 9B hybrid model, which I think is almost equivalent to our 7B model. These are good models. There are not that many of them that are in these size spans that are really strong. So I think we're happy to be there and happy to point out other people are doing great work here. It's not like we can ignore Qwen. That's a losing strategy.

Luka，为了强调这一点，AI 中的开源概念，它的不同形式。请向我们解释这意味着什么，以及你们处于什么位置。

Luka, just to drive it home, the concept of open source in AI, the different flavors of it. Walk us through what that means and where you guys are at.

这总是一个在讨论中有点被忽视的话题。但是，当谈到模型时，人们对开源有不同的理解。大多数发布的模型，我认为最好的描述是开放权重。你的 Qwen、Gemma、Llama，还有 Kimi，发布的是对应模型最终状态的权重，这是最常见的，或者可能是指令模型的最终状态、基础模型的最终状态。在很多情况下，这已经足够了，你可以在其上构建出色的软件。但从研究到应用，同样有很多情况下这还不够。你想要模型的中间状态，以便更好地定制它。你想要访问数据，这样你就可以在注入自己数据的同时重新进行训练步骤。你可能想要访问预训练数据，因为你有一个令人难以置信的研究项目，将改变我们对语言模型的看法，但你需要知道语言模型是在什么数据上训练的。所以我们想支持这些用例。因此，对于 OMO，如果我们能发布，我们就会发布。我们不能把我们的 GPU 发布给全世界，那行不通。但是，对于数据、中间检查点、基准测试、软件，任何我们能发布的，我们都会发布。如果有人问，嘿，你描述了管道的这一部分，但你没有发布，我们也会发布那部分。就像我们一直收到关于 SFT 或其他微调阶段中间检查点的问题。现在，我们在推理和指令的监督微调过程中有中间检查点，然后在最后的多天强化学习运行中也有中间检查点。所以那些想了解检查点并对其进行研究但没有算力训练的人，现在这些都有了。

That's always a topic that gets sort of overlooked a little bit in discussion. But yeah, when it comes to models, there are different levels of what people consider open source. The majority of models that get released, I think the best term to describe them is open weights. Your Qwen, your Gemma, your Llama, you know, Kimi, what gets released is a set of weights that correspond either to the final state of the model, that's the most common, or maybe final state of the instruct model, final state of the base model. And there are plenty of cases where that's enough and you can build great software on top of it. There's an equivalent large set of cases from research to application where that's just not enough. You want to have intermediate state of the model so that you can customize it better. You want to have access to the data so you can maybe redo a step of the training while infusing your own data. You might want to have access to the pre-training data because you have this incredible research project that's going to change how we think about language models, but you need to know what a language model is trained on. So we want to support those use cases. So when it comes to OMO, if we can release it, we will release it. So we can't release, I don't know, our GPUs out to the world, that doesn't work. But when it comes to the data, the intermediate checkpoints, the benchmarks, the software, anything we can, we'll put it out. If people ask, hey, you described this part of your pipeline, but you haven't put it out, we'll release that part as well. Like we've always got questions about intermediate checkpoints during SFT or other fine-tuning stages. And like now we have intermediate checkpoints during our supervised fine-tuning for reasoning and for instruct, and then also for our multi-day RL runs at the end of these we have intermediate checkpoints. So people that are looking, a lot of people like to understand checkpoints and do research on them but don't have compute to train, and now it's like, okay, this is all there.

在我们深入细节之前，我想退一步。今年对于开源 AI 世界来说是非常紧张的一年。DeepSeek 时刻感觉像是三年前，但实际上那是在一月底，也就是 10 个月前，从那以后发生了很多事情。Nathan，你能帮我们回顾一下 2025 年的关键事件，让人们了解发生了什么吗？

Before we dive into the specifics, I'd love to take a step back. It's been a very intense year in the world of open source AI. The DeepSeek moment feels like it was three years ago, but in reality that was at the end of January, so 10 months ago, and a lot has happened since. Nathan, could you help us recap the key events of 2025 for people to understand what's happened?

是的，如果我试图列出实际模型，我会漏掉一些，因为有很多值得注意的。

Yeah, if I try to make a list of actual models, I'm going to forget some because there are so many that are notable.

我认为从 DeepSeek 说起确实很重要。如果你和在中国构建模型的人交流，大家的共识是 DeepSeek 向我们展示了 AI 可能是一件大事，然后很多公司就想，'哦，我们应该做他们做的事。'所以过去一年涌现了大量实验室。除了 DeepSeek，知名的玩家还有 Qwen，我认为 Z.AI 和 Kimi Moonshot 之前就已经存在，现在更是成为了家喻户晓的名字，尤其是如果你关注西方、旧金山为中心的讨论，这些都是人们正在使用和谈论的东西，这是一个很大的变化。但中国还有大量模型涌现，比如蚂蚁集团发布了万亿参数模型，基准测试非常强；美团，相当于中国的 DoorDash，也是另一家大型科技公司。开发语言模型的标准方式已经变成了开放发布，整个生态系统都在向前推进，同时摸索。与此同时，Meta 的领导层发生了重大变化，Llama 的未来变得不那么明朗，而 Llama 曾是开源 AI 的典范定义。那条思路就此终结。所以影响力出现了巨大的真空，被 Qwen、DeepSeek、Kimi Moonshot 等试图用开放模型构建东西的公司吸收了。这是一个重大转变。我认为美国内部有很多讨论，认为我们有充分的理由应该拥有或至少影响整个技术栈，包括开放模型。因为实际上，美国的大型科技公司会通过让研究人员近距离接触、使用相同语言和基础设施来捕获下游价值。我认为这在科技行业已经持续了几十年，所以我不需要过多解释。而且有些人真的开始意识到这一点了。我认为在六七月的时候，中国的模型提供商变得无法忽视。那时我们有 Kimi K2 instruct，Qwen 发布了很多大模型，比如 Qwen Coder，Z.AI 的 GLM 4.5，现在这种情况还在继续。所以在我们录制和发布这期播客的时候，有很多人关注美国公司将如何应对。我知道英伟达正在大张旗鼓地行动，他们投资了 Reflection 很多钱，还有其他玩家试图起步。但紧迫感，我们也没有太多的算力 AI，但如果我们能在人们实际使用的模型规模上取得进展，我认为我们应该专注于研究人员。我认为密集模型对研究人员很好，它们使用的算力和工程资源更少。而且我确实认为，如果你在未来几个月看这个播客，会有更多美国实验室参与。我的意思是，OpenAI 已经发布了一些模型。所以美国规范转变需要很长时间，因为它们的建立方式不同。

I think starting with DeepSeek as you mentioned is definitely the important thing. And then if you talk to people building models in China, a lot of the consensus is like DeepSeek showed us that AI could be a big deal, and then a lot of these companies were like, 'Oh, we should do what they did.' So there's just kind of a ton of labs that have popped up over the year. I think known players in addition to DeepSeek are Qwen, and I think Z.AI and Kimi Moonshot had already kind of existed and have really stepped up to be much more known names, especially if you're following Western, SF-centric discourse. These are things that people are using and talking about, which is a kind of big change. But there's just this huge mass of models coming from China. You have everything like Ant Group releasing trillion-parameter models with really strong benchmarks. Meituan, which is the Chinese equivalent of DoorDash, is just another big tech company in China. The standard way of developing language models has become to release them openly, and that whole ecosystem is going forward with this, figuring this out. At the same time, there was a big change in leadership at Meta, and Llama's future is less known, which was really the paradigmatic definition of open-source AI. That line of thought just ended. So there's this big vacuum of influence which has been absorbed by the likes of Qwen, DeepSeek, Kimi Moonshot in terms of who's trying to build things with open models. And that's a big shift. I think there's a lot of discussion within the US that there's good reason we should own, or at least have influence over, the whole technological stack, and that includes open models. Because realistically, it's the big tech companies in the US that'll capture the downstream value from having the researchers be in close proximity, speaking the same language, and used to the infrastructure. I think this is something we've seen for decades in the tech industry, so I don't think I need to explain it that much. And there are people that are really starting to wake up to this. I think in June-July is when the Chinese model providers were really becoming impossible to ignore. That's when we had the Kimi K2 instruct, Qwen was releasing a lot of their big models like Qwen Coder, GLM 4.5 from Z.AI, and that's kind of just continuing now. So I think when we're recording and releasing this podcast, there's a lot of interest in what the US companies are going to do to respond to this. I know that Nvidia is making a lot of noise here. They invested a lot of money in Reflection, and there are other players that are trying to get going. But urgency, and we don't have a lot of compute AI too, but if we can make a dent in this and some model sizes that people actually use, I think that we focus on researchers. I think dense models are great for researchers. They take a little bit less compute and engineering resources to use. And I do think that if you look at this podcast in the coming months, there's going to be a lot more labs in the US participating. I mean, OpenAI has released some models. So it just takes a long time for the norms to shift in the US where they're just established in a different way.

那么 Qwen 的使用广泛程度可能人们还没有完全意识到，对吧？有个轶事，几周前 Airbnb 提到他们用 Qwen 而不是 ChatGPT。但你有关于 Qwen 使用情况的任何统计数据或轶事证据吗？

And then Qwen is widely used in a way that people may not have completely realized, right? There was an anecdote, the example of Airbnb talking about using Qwen over ChatGPT a few weeks ago. But do you have any sort of stats or anecdotal evidence on the usage of Qwen?

另一个著名的引用是 Martin Casado 在《经济学人》中的话，他说 80% 的公司都在基于 Qwen 构建。后来被纠正为，使用开放模型的公司中有 80% 在使用 Qwen，这大约占他投资组合的 16% 到 24%，这仍然很多。相当多的人尝试用开放模型做事情，其中大多数使用 Qwen。还有像 Cursor 发布了自己的模型 Composer 2。大家公认它是基于某个开放发布的中国大模型构建的。有一些明显的传闻说它会切换到中文等等，但这正是那种不想自己预训练模型，但通过为用例定制模型获得巨大价值的公司，它们会基于这些优秀模型构建。而且我认为随着它们试图进入更多市场，它们会希望有更多选择。实际上，很多美国公司不想部署中国模型。目前很多公开的理由都是未知的未知和无法证明的事情。比如你无法证明模型没有某些后门，而我相当确定它们现在肯定没有。但正因为你无法证明，就造成了这种奇怪的市场舞蹈，就像这些是随机的东西，有点模糊。作为研究人员，我不喜欢处于这种中间位置。但我只想提供信息和人们真正想用的好东西，把地缘政治和其他信息留给那些可能比我更有切身利益的人。我不知道。我们在非营利组织工作。我有我的狗。

The other famous quote was a Martin Casado quote in The Economist where he said 80% of companies are building on Qwen. That has been corrected where it's 80% of companies building with open models are using Qwen, which is like 16 to 24% of his portfolio, which is still a lot. It's a meaningful amount of people are trying open models for things, and most of them are using Qwen. And then there's the likes of Cursor released their own model, Composer 2. It's accepted that it is built on a large Chinese model of some sort that was released openly. There's some obvious tales of it switching to Chinese and things like this, but that's just the sort of company that doesn't want to pre-train their own models, but has immense value in specifying models for their use case that is just going to build on these great models. And I think they would want more options to choose from as they try to sell into more markets. I think realistically it's a thing where a lot of US companies don't want to deploy Chinese models. I think currently a lot of the stated reasons are just unknown unknowns and things you can't prove. Like you can't prove that the models aren't doing certain back doors where I'm fairly certain they definitely aren't now. But just because you can't prove it makes this kind of weird market dance which is like yes these are stochastic things that are kind of amorphous. And it's like I don't love being in the middle of this as a researcher. But it's like I would like to just provide information and good things that people actually really want to use and leave all of the geopolitical and other messaging to people that have probably realistically like way more on the line than I do. Like I don't know. We work in a nonprofit. I have my dog.

你认为为什么生态系统会这样发展，美国非常商业化、闭源，而中国非常开源？

Why do you think this happened that the ecosystem developed in this way, that the US was very commercial closed source and China very open source?

从历史上看，美国更愿意为服务付费。我从比我更了解中国的人那里听到轶事，比如'是的，中国中等规模、估值超过十亿美元的公司会盗版 SaaS 软件。'我不喜欢这样，听起来比实际情况更糟，但我觉得美国公司习惯于为服务付费，API 模型和按 token 付费已被证明是非常好的业务。目前在美国卖 token 是个好生意。我认为关于盈利能力的争论很多，但这些 token 的需求和实用性很高。所以我坚信可以通过卖 token 建立盈利业务，而我认为 AI 在中国公司中的嵌入方式会非常不同。我和其中一些实验室聊过，他们说，'为了打入美国市场，他们不会付费……'他们说过，美国公司不会为服务付费。

Historically the US has a lot more willingness to pay for services. I hear anecdotes from people that know China a lot more than I do that are like, 'Yeah, mediumly large, billion-dollar-plus valuation companies in China will just pirate SaaS software.' I don't like it, that sounds worse than it is, but it's just like I think that the thing is that US companies are used to paying for services and an API model and paying for tokens has been proven as a very good business. Selling tokens is a good business in the US right now. I think there's a lot of debate over profitability but the demand and usefulness of these tokens is high. So I have a lot of belief that there can be profitable businesses from selling tokens where I think that AI will be embedded in very different ways when it comes to Chinese companies. And I've talked to a few of these labs and they're like, 'In order to sell into the US market, they will not pay for...' They've said this, US companies will not pay for services.

他们并不指望企业大规模注册 Kimi 的编程计划，但他们觉得有机会让企业使用他们的模型，这是一种实际的方式来施加影响并分一杯羹。在中国构建这些模型的人对不同的生态系统有同样的认识。这就是为什么我开始喜欢和他们交流。他们看到同样的限制，这并不复杂。他们足够聪明，知道如果他们发布非常好的模型，美国的人无法忽视，这是他们在生态系统中占有一席之地的方式。所以有 DeepSeek 标准这样的混合，他们觉得这适合他们，就继续做下去。这为他们赢得了很多关注和一些重要用途。所以我认为这很有道理。

So like they don't expect enterprises to sign up for the Kimi coding plan in mass, but they're like we have a chance that they'll use our models and it's like that is a practical way to influence and get a piece of the sharing pie. And it's like the people building these models in China know the same things about the different ecosystems. It's like that's why I've enjoyed starting to talk to them. I was like these people the same thing. They see the same constraints. It's not that complicated. So they're smart enough to know that if they drop really good models like people in the US can't ignore it and like that's their way to have a part in this ecosystem. So there's a mix of like the DeepSeek standard and then they're kind of like yeah this is something that works for us. Let's keep doing it. It's getting them a lot of mind share and some use in prominent ways. So I think it makes sense.

那么美国是否出现了更有组织的回应？我知道你参与了，或者可能主导了 Atom 项目。

And is there more of an emerging organized response in the US? I know you're involved or perhaps behind the Atom project.

我认为任何协调一致的回应只有在公开时才能看到，而且我认为不同利益相关者之间有很多投资和对话在进行，但这并不那么有用。所以我无法给你证据，但我确实认为合适的人正在讨论并希望投入更多，因为相对于数万亿美元的 AI 基础设施建设，成本并不高。比如，如果 0.01%能让我们得到更好的开放模型，我们可能应该这么做。我认为这其实并不复杂。问题是如何将一亿美元的预算分配给有才华且激励正确的人。Reflection 的消息可能在未来几年是一个好的解决方案；他们有足够的资金和强大的人才基础。这是一个重要的条件。我们需要多样性，因为 Llama 的情况可能再次发生或消失，但它看起来像一个小雪球，希望在未来几个月能壮大。

I think any concerted response you only see when it actually is public and I think there's a lot of investment at different stakeholders and conversations that are happening, but that's not that useful. So I don't have the proof for you, but I do think the right people are talking about it and want to invest more because realistically the cost is not that high relative to the trillion dollar buildout of AI infrastructure. It's like, oh, if 0.01% gets us better open models, we should probably do that. I think that's actually not that complicated. It's just how do you get the hundred million dollar line item to the right people that have the talent to do it and with the right incentives. It's like the Reflection news is probably a good solution for a couple years; they have enough money and a strong base of talent. That's a major checkbox. We need to have diversity there because the Llama thing could happen again or it goes away, but it looks like a small snowball that hopefully grows in the coming months.

今天的发布和你的工作是美国对中国在开源 AI 领域崛起的回应的一部分。

Today's release and your work is a part of that American response to China's rise in open source AI.

我会说我在七月推出了 Atom，本以为会获得更多关注，但现在我收到了大量的媒体询问和报道，每个人都想分享这个故事。所以我想，‘好吧，看来我只是早了四个月，但这就是我。’就像今天我看到彭博社发布了一篇文章，标题几乎和我七月份关于 Kimi 的帖子一样。我想，‘好吧。’我很高兴现在人们开始关注了；迟做总比不做好。

I would say I launched Atom in July and thought it would get more visibility, but now I'm getting a crazy amount of media inbound and press and everyone wants to share the plot. So I was like, 'Okay, I guess I was just four months too early, but that's what I really am.' It's like just today I saw Bloomberg published a post that pretty much had the same title as my Kimi post from July. I was like, 'Okay.' I'm glad that people are paying attention now; better late than never.

好的，恭喜。这是最好的恭维。那么，换个话题，为了让这些对话对更广泛的群体有教育意义。这次发布的一个关键方面是思考模型。你能提醒大家思考模型到底是什么，与其他形式的模型或前几代模型相比有什么不同吗？很多人听说过推理时 Scaling，这很有道理：如果你在推理时投入更多算力，你会得到更好的答案。思考模型实际上是一种训练模型以充分利用这一点的方法。所以你花费大量 token，这些 token 通常作为长思维链对用户隐藏，因此模型有一个阶跃变化，在数学任务、编程任务、智能体式任务上表现更好。我认为我们未来的计划包括为模型增加更多工具使用。所以我们不会过多讨论这个模型的智能体式搜索或即时智能体式代码执行。但构建思考模型是通往更多有趣事情的途径，比如云代码，或者明年我们可能拥有全代码。思考模型是 2025 年的趋势，每个答案消耗更多算力，但在各种任务上表现更好。

Okay. Well, congratulations. Best form of flattery. All right. Switching tracks, in an effort to make those conversations educational for a broad group of people. So one of the key aspects of the release is the thinking model. Could you remind folks what a thinking model actually is versus other forms of models or prior generations of models? A lot of people have heard about inference time scaling which makes sense: if you spend more compute at inference time, you get a better answer. A thinking model is really a way to train the model to exploit that a lot. So you spend a lot of tokens which are usually hidden from the user as a long chain of thought, and the model therefore has this step change where it's way better at math tasks, coding tasks, agentic tasks. I think our future plans include adding more tool use to the model. So we're not talking a lot about agentic search or agentic code execution on the fly for this model. But building thinking models is the gateway to doing a lot more interesting things like Claude Code or maybe we'll have all code next year. The thinking model has been the thing in 2025 that uses a lot more compute per answer and gets way better at various things.

我不喜欢思考模型，但没关系。

I don't like thinking models but it's fine.

不，它们很好，非常有用。思考模型实际上是一种工作模式，而常规的指令模型通常构建起来更有趣，可能更古怪。但确实，这就是它们的定位：90%的情况下，特别是面向用户的情况下，人们愿意花时间等待这些模型生成更好的答案。仍然有空间给那些能更快响应的模型。你看到谷歌发布的关于 Gemini Flash 采用率的数据。这就是非思考模型的价值所在，它们至少能给出一个不错的初步答案。它们构建起来也更有趣。但思考模型是未来，特别是在智能体集成方面。

No, they're good. They're very useful. Thinking models are really a work mode, and regular instruct models are usually more fun to build. They can be more quirky. But yeah, I think that's really where they are: 90% of the cases, especially user-facing cases, folks are okay spending time waiting for these models to craft a better answer. There's still a space for models that can respond faster. You see stats like Google released about adoption of Gemini Flash. That's where non-thinking models that can at least approximate a good first answer are really useful. They're also more fun to build. But yeah, thinking models are where the future is, especially when it comes to agent integration.

在我们具体讨论 Almo 系列的流程之前，因为正如你提到的，这是开源的一个奇妙之处：在这样的讨论中，我们可以真正了解模型的工作原理，而不是与商业玩家的对话。所以在进入流程之前，我想先谈谈你们，你们的背景，以及 AI2——这个生态系统中一个非常重要的参与者，人们可能听说过也可能没听说过。谁先来？

Before we go into the pipeline very specifically of the Almo family, because as you alluded to, that's one of the amazing things about open source: we can in a discussion like this truly understand how the model works versus other conversations with commercial players. So before we go into the pipeline, I'd love to talk a little bit about you guys, your backgrounds, and AI2 which is a very important player in the ecosystem that people may or may not have heard about. So who wants to go first?

我算是偶然进入这个角色的，只是选择有趣的问题。我来自意大利，后来去美国读博士。我的博士研究方向是信息检索，简单来说就是如何构建搜索引擎。我逐渐深入自然语言处理领域。毕业后我先加入了亚马逊，最初在 Alexa 团队工作，负责搜索部分，后来发现用户与 Alexa 对话的部分更有趣，于是逐渐转向那里。之后我加入了 AI2，参与了一个名为 Semantic Scholar 的项目，这个项目仍在运行，是一个学术论文搜索引擎。有趣的部分实际上是用户交互，而不是搜索的论文文本。后来我进入大语言模型和构建语言模型领域，这与 AI2 开始构建语言模型的过程紧密相连。这一切大约始于 2022 年 11 月。

I sort of stumbled into this role by just picking problems that are interesting. My background originally from Italy, moved to US for PhD. My PhD is in information retrieval, how to build search engines to simplify a lot. I slowly got into more and more of the sort of natural language. First joined after grad school I joined Amazon. I was working on Alexa at the beginning, working on the search part of Alexa, and then I got into the actual part where the users talk to Alexa is the interesting part. So slowly moving towards that. Initially joined AI2 working on a project called Semantic Scholar, still active, it's a search engine for academic papers. And there the interesting bits were actually the interacting with users and less so the actual text of the papers that you were searching on. And then the way I got into LLMs and building language models is really intertwined with how AI2 got into building language models. It all started around November of 2022.

差不多在同一时间，OLMo 发布了。AI2 的一群研究人员，都是个人贡献者，不是自上而下的指令，完全是一个草根倡议。很多研究人员对构建一个完全开放（fully open）的模型非常感兴趣。AI2 在 2017、2018 年左右已经构建了某种原型语言模型。所以大家的兴趣主要在于重新捕捉并扩展那条工作线。于是我们一群人聚在一起，开始规划，联系了几家可能支持这些倡议的公司。我们最初从 AMD 获得了一笔资助，大约 200 万 GPU 小时。所以我们有了想法，有了感兴趣的研究人员，有了算力。于是我们去找当时的领导层，跟他们说：‘嘿，我们要去做这件事了，希望你们能同意。’好的一点是，我们本质上是一个研究实验室。所以大家都说：‘当然，你们自己搞定一切，玩得开心就好。’

This is around the same time OLMo got released. A bunch of researchers at AI2, individual contributors, not direction from the top, like a very grassroots initiative. A bunch of researchers got really interested in building a model that would be fully open. AI2 had already built sort of proto language models around 2017, 2018. So a lot of the interest was in recapturing and expanding that line of work. So a bunch of us got together, started planning, got in touch with a few companies who might be interested in supporting these initiatives. We got an initial grant from AMD at the time, that was about 2 million GPU hours. So we had the idea, we had the researchers interested, we had the compute. So we went to leadership at that time and sort of told them, 'Hey, we're going to go do this thing. I hope you're okay with it.' One of the nice things about it too is that at heart we're a research lab. So everyone was like, 'Sure, you figure everything out. Just have fun.'

太好了。好的，Nathan，你呢？你是个多才多艺的人。你做研究，写一个非常有趣的博客通讯叫 Interconnects，你做播客，还做很多不同的事情。所以跟我们讲讲你的经历吧。

Great. All right, Nathan. How about you? You're a man of many talents. You do research, you write this very interesting blog newsletter called Interconnects, you do podcasts, you do a bunch of different things. So tell us about your journey.

是的，我说我身兼数职，是为了完成我想做的事情。2017 年我作为电子工程博士生来到伯克利，然后我看到人工智能正在兴起，我决定尝试做这个。一开始我去找那些大家都知道的名字，比如 Sergey Levine 和 Peter，请求加入 BAIR 小组，但他们礼貌地拒绝了。然后开始了漫长的过程，学习如何在不直接嵌入这些精英团队的情况下实际去做，这涉及机器人学和强化学习的结合，并找到自己的路。所以我的博士研究主要是基于模型的强化学习。然后我的一个研究方向的职位是加入 Hugging Face，当时他们说要做 DeepMind 的开源版本来做大量研究。实际上，我的工作在 ChatGPT 出现之前并没有太大影响力或用处，然后我想：‘哦，我或许应该学学 RLHF。’这作为试图在公开场合与团队合作的人立即获得了关注。像 Lewis Tunstall 和 Hugging Face 的其他人在这个方面仍然做得很好，我们合作了一段时间。然后我主要是远程工作感到疲惫，在夏威夷的一个有趣的会议上遇到了 Luca，心想：‘哇，我可以有现实生活中的朋友。’于是我加入了 AI2，进行面对面工作，并尝试做同样的事情，这可以说是 OLMo 故事的一个演变。我当时非常有动力去弄清楚主要是基于人类反馈的强化学习（RLHF），并公开这些后训练技术的版本。然后这通过 OLMo 和我们的后训练方法（名为 Tulu）演变而来，我们花了很长时间试图复制我们认为接近 Llama 3 的后训练，使用了多个阶段和优化器。这个项目与一群人一起提出了‘基于可验证奖励的强化学习’这个名称。所以这是 AI2 的一个不断演变的旅程，追求影响力，我们认为这正是人们实际在做的事情。然后像 EOS，很大程度上 Luca、我和 AI2 其他人填补的机会是，人工智能领域有太多资金，而且越来越多，以至于能够在公开场合谈论这些事情、教育并通过传播知识让更多人参与的人越来越少。所以我描述我的职业旅程时，很大程度上是在填补这个真空，思考什么在那里有影响力。所以当存在这样一个空白时，它会有一种引力，让你清楚自己应该做什么。

Yeah, I say I wear many hats to try to get the things that I want to do done. I showed up to Berkeley as an EE mostly PhD admit in 2017 and then I saw that AI was happening and I decided that I want to try to do this. That started by going to all the names that people know like Sergey Levine and Peter and asking to join the BAIR group, and then they respectfully said no. Then starts the long process of learning how to actually do it without being directly embedded in these elite groups, which was a mix of robotics and reinforcement learning, and finding my way there. So my PhD was mostly in model-based reinforcement learning. Then my one research avenue job was to go join Hugging Face when they said they were going to make an open source version of DeepMind to do a bunch of research. Realistically my job was not that impactful or useful at Hugging Face until ChatGPT came out, and then I was like, 'Oh, I should maybe just learn about RLHF.' That got very immediate traction as somebody trying to work in public with the team there. So like Lewis Tunstall and other people at Hugging Face are still doing a great job on this, and we worked together for a while. Then mostly I was just getting burnt out on remote work and met Luca in Hawaii at a fun conference and was like, 'Wow, I could have real life friends.' And I joined AI2 to work in person and tried to do the same thing, which kind of takes an evolution of the OLMo story. It was just like I had a lot of motivation on trying to figure out what was mostly reinforcement learning from human feedback at the time and make versions of these post-training techniques public. Then that kind of evolved through both OLMo and we have our post-training methods that was named Tulu, which is like we spent a long time to try to replicate what we thought was close to Llama 3 post-training with multiple stages and optimizers. That's the project that came up with the name reinforcement learning with verifiable rewards with a bunch of people. So it's kind of this evolving journey at AI2 to in search for impact, which is what we think people are actually doing. And then like the EOS, largely the opportunity that Luca and I and others at AI2 fill is that there's so much money in AI and it only becomes increasingly so that the amount of people that can talk about these things in public and educate and get more people involved by spreading knowledge is ever smaller. So I describe my career journey as a lot of it is filling that vacuum and thinking about what's impactful there. So it kind of pulls you when there's such a void, it has a sort of gravity to make it clear what you should be doing.

你预料到了我的问题，这有点显而易见。在一个我们看到一些商业人工智能实验室为像你这样的人提供数亿、数十亿美元薪酬的世界里，我很好奇你加入 AI2 这个非营利组织的有趣动机。但简短的回答是影响力。

You anticipated my question, which is sort of obvious. In a world where we see hundreds of millions, billion dollar packages offered by some commercial AI labs for people just like you, I was curious about your interesting motivation to join AI2 which is a nonprofit. But impact is the short answer.

是的。我的意思是，我在这里已经两年了，我加入时并不出名。所以让那些正在找新工作的人知道：你想找一份你能在其中成长的工作。我认为 AI2 对很多人来说都是一个非常好的地方，因为你有独立性，被鼓励去前进和做事，而不是成为一台更大的、只是产出语言模型的机器中的一个齿轮，这很重要，但更难获得可见性。

Yeah. I mean, I've been here for 2 years and I wasn't famous when I joined. So let that be told to people looking for new jobs: you want to find a job that you can grow into. And I think AI2 has been a really, really good place for that for many people because you have independence and are encouraged to go forth and do things and not be a cog in a broader just grind out language model machine, which is important, but it's harder to get visibility.

所以我们提到了一些，但也许简单说说 AI2。AI2 是由 Paul Allen 创立的，对吧？AI2 代表艾伦人工智能研究所。你提到了一些资助，Luca，我想在之前的对话中我们也谈到了最近的一笔资助。我看到那是来自 NSF 和 Nvidia 的 1.52 亿美元。那么 AI2 是什么？它是如何开始的？从高层次看它是如何创立的？

So we alluded to some of it but maybe a few words about AI2. AI2 was started by Paul Allen, right? AI2 stands for Allen Institute for Artificial Intelligence. You mentioned some grant, Luca, and I think earlier in the conversation we talked about a recent grant as well. I saw that it was $152 million from NSF and Nvidia. So what is AI2? How did it start? How was it founded at a high level?

AI2 大约在 2014 年由已故的 Paul Allen 创立。最初，AI2 非常专注于构建能够做科学、理解科学、解决科学问题的机器。那时 Semantic Scholar 作为科学论文的存储库开始了。慢慢地，开始形成的倡议之一是围绕语言模型如何工作、当时称为自然语言处理如何工作的更基础研究。你有像 Aristo 这样的团队从很早开始就做了出色的工作。我们一直有这样的想法：不仅发布成果或研究，还要发布工具。在过去，我们有一个非常广泛使用的仓库库叫 AllenNLP，它允许你构建和定制这些模型。

AI2 was founded around 2014 by the late Paul Allen. Initially, AI2 was very focused on building machines that can do science, can understand science, solve science problems. That's when Semantic Scholar started as a repository of science papers. Slowly, one of the initiatives that started forming was more fundamental research around how language models work, how at the time it was called natural language processing was working. You had teams like Aristo doing great work since the very early days. We always had this idea of not just releasing artifacts or research but releasing the tool. Back in the day, we had this very widely used repo library called AllenNLP that would allow you to build and customize these models.

我插一句。这很酷，因为它是我们团队名称的同名，在 AI2 已经很久了。它是 Hugging Face Transformers 的主要竞争对手，最终他们胜过了 AI2，成为人们使用的工具，因为他们有非常不同的模型和支持量。但 Luca 可以继续。

I'm going to jump in. It's cool because it's the namesake of our team name and has been for a long time at AI2. It was the main competitor to Hugging Face Transformers and they ultimately outcompeted AI2 as the thing that people use for that because they had very different model and amount of support. But Luca could keep going.

看看那些更多的好东西，但我们已经开源了一段时间。我认为这是这里的人很早之前就理解的事情，在我来之前，他们很早就理解这很重要，既在推动科学方面，也在解锁商业用例方面，作为非营利组织我们可能没有预料到。但是，你知道，你发布一个工具，人们拿起它并用它做令人惊奇的事情。

Look at those little more good thing but but but we have been at it like you know open sourcing for a while. I think it's something that folks here understood really early, before my time, understood really early that was important, both in pushing science and also in unlocking commercial use cases that as a nonprofit maybe we didn't anticipate. But folks, you know, you release a tool, people pick it up and do amazing things with it.

是的，我们最近越来越转向语言建模。目前 AI2 大概有三个主要项目。一个是模型家族，有 MoE 的变体。有些专注于完整流程，有些专注于机器人技术，有些专注于处理图像、视频和音频。

Yeah, and you know we moved on to language modeling more and more recently. Right now AI2 has maybe three main projects. One is a model family, and there are variants of MoE. Some focus on the full pipeline, some on robotics, some on processing images, video, and audio.

Molmo 是这些变体之一吗？

Is Molmo part of one of those variants?

是的，Molmo 是我们处理多模态输入的项目之一。最近我们发布了另一个叫 Molmo Act，更专注于机器人技术，接收多模态输入并能在空间中行动。我们还有一个用于自动语音识别的模型，另一个用于文档处理和 OCR。这是一个精巧的小模型家族。我们有一个关于科学任务智能体的工作组，回归我们的本源。这是 ATA 倡议家族，帮助科学家完成工作的智能体。

Yeah, Molmo is one of our projects that works on multimodal inputs. Recently we released another one called Molmo Act, which is more focused on robotics, receives multimodal input and can act in space. Then we had a model for automatic speech recognition, another for document processing and OCR. It's a nifty little family of models. We have a working group on agents for scientific tasks, arching back to our roots. This is the ATA family of initiatives, agents to help scientists do their work.

那个刚出来，大概是今年八月？

And that just came out right, like August of this year?

是的。团队从去年年中就开始酝酿，但今年才首次发布。实际上有两个发布：主要的 ASA 发布，以及最近我们宣布与 Kaya（癌症 AI 联盟）合作，使用 ASA 中的一些组件帮助研究人员在癌症研究上取得进展。还有第三个分支是关于环境 AI，构建能够理解和建模地球的模型，处理不同信号进行预测。这个我有点含糊，因为不确定是否已经公布。

Yeah. The team has been cooking since middle of last year, but finally we had our first release this year. There are actually two releases: the main ASA release and recently we announced a partnership with Kaya, the Cancer AI Alliance, using some components in ASA to help researchers make progress on cancer research. Then there is a third branch on AI for the environment, building models that can understand and model Earth, working with different signals for prediction. I'm being a bit vague on this one because I don't know if it has been announced yet.

这里是预告。MAD 播客在制造新闻。好的，非常酷。太棒了。很好的背景。所以，我们有 Molmo、Tulu、ATA。最后一个问题：规模上，大概有多少人？

It's a preview right here. The MAD podcast is making news. Okay, very cool. Awesome. All right, that's great background. So, we got Molmo, we got Tulu, we got ATA. Just maybe one last question: in terms of size, what are we talking about? How many of you guys are there?

大约 200 人，包括研究人员、工程师、顾问和其他支持角色。

200 people between research staff, engineering, consultants, and other support roles.

非常好的背景。非常感谢。好的，我们换个话题，谈谈 Molmo 3、思考、推理，不管你们最后怎么叫。我认为这是一个绝佳的机会来讨论这些推理模型实际上是如何工作的。在之前的节目中，我们与 Anthropic 或 OpenAI 的人有过精彩的对话，但毫不奇怪，他们能说的有限。你们所做的美妙之处在于让一切开放并人人可及。所以我希望我们能够讨论从预训练到后训练的整个流程，各个部分，并用通俗的语言解释每个部分的作用。你们谁能先给出一个高层次架构，概述流程中的各个子类别，然后我们逐一深入？

That's fantastic background. Thank you very much. All right, let's switch tracks and go into Molmo 3, thinking, reasoning, whatever you guys end up calling it. I think it's a perfect opportunity to talk about how those reasoning models actually work. In prior episodes, we've had great conversations with folks at Anthropic or OpenAI, but not surprisingly, there's only so much they can talk about. The beauty of what you guys do is to make it open and accessible to everyone. So I'd love for us to talk about the whole pipeline from pre-training to post-training, the different parts, and make it super educational, explaining in plain English what part does what. Could either of you start with a high-level architecture of the various subcategories of the pipeline, and then we'll go into those one by one?

当然。我最近在语言模型会议上就此做了演讲，所以记忆犹新。我可以提供一个个人动机：作为深入社区的研究人员，我们看到许多人在 DeepSeek R1 之后开始做强化学习研究。这大多发生在 Qwen 模型家族上，比如 Qwen 2.5 和 Qwen 3，参数规模在 1 到 80 亿之间。促使我们关注许多细粒度细节（我们可能没有时间深入）的原因是，在进行 RL 研究时，Qwen 使用的数据存在疑问。具体来说，有两篇论文：一篇是《虚假奖励：重新思考 RLVR 中的训练信号》，我与华盛顿大学和 AI2 的许多人共同参与；另一篇是《推理还是记忆？数据污染导致强化学习结果的不可靠》。

Sure. I recently gave a talk on this at the Conference on Language Models, so I have it fresh in my mind. I can provide a personal motivation: as researchers embedded in the community, we see many people starting to do reinforcement learning research after DeepSeek R1. Most of this happens on the Qwen model family, like Qwen 2.5 and Qwen 3, between 1 and 8 billion parameters. Something that motivated a lot of the fine-grained details we might not have time for is that there are questions hanging over the data used for Qwen when doing this RL research. Specifically, two papers: 'Spurious Rewards: Rethinking Training Signals in RLVR' which I was on with many people at UW and AI2, and another one, 'Reasoning or Memorization? Unreliable Results of Reinforcement Learning due to Data Contamination.'

是的。实际上，我们花几分钟谈谈这个。'虚假奖励'是什么意思？

Yeah. Actually, let's spend a couple minutes on that. What does 'spurious rewards' mean?

需要知道的是，Qwen——这可能会引发之后的技术吐槽。很多背景知识是关于理解这些算法是什么，但本质上问题是：Qwen 是否包含了过于接近评估目标的训练数据，以至于研究捕捉到的是模型内部的奇怪行为，而不是强化学习的基本原理？

The thing to know is that Qwen—this could trigger a rant on the technical side later. A lot of background on understanding what these algorithms are, but essentially the question is: did Qwen include training data too close to the evaluation targets, so that the research picks up on weird behaviors within the model rather than the fundamentals of what this reinforcement learning is doing?

换句话说，他们是应试教育还是实现了真正的推理？

So in other words, did they teach to the test versus enabling true reasoning?

我不认为 Qwen 明确那样做了；这是一个灰色地带。所有前沿模型实验室在某种程度上都会这样做。他们有团队成员负责改进评估，最简单的方法是在测试集上训练，但他们作为精英科学家有尊严，不会那样做。最接近的做法是对测试集进行改写以创建新的训练数据。所以技术上不算作弊，但就像在从 GitHub 抓取数学问题到改写评估集的光谱上，你在哪里划清作弊的界限？不同的人有不同的答案。我们一直回到的一个目标是，我们理解 Molmo 不是一个 6000 亿参数的模型，人们会立即下载并运行代码，但我们希望确保我们的核心受众能够自信地进行研究并进行辩论。所以我们希望让人们接触到每个阶段，这样他们就能看到这如何影响这个重要的新研究领域。我们将讨论六个阶段。一是大规模预训练，在所有互联网数据上预测下一个词。二是我们称为中期训练，它是否存在尚有争议；技术上，你以更高的学习率在更高质量的网络数据上训练。三是长上下文扩展，对这些推理模型至关重要，因为它们在分享答案之前会生成许多中间词。Luca 有很多关于这方面的实战故事。

I don't think Qwen explicitly did that; it's a gray zone. All the frontier model labs do this to some extent. They have team members tasked with improving an evaluation, and the easiest way is to train on test, but they have dignity as elite scientists and won't do that. The next closest thing is paraphrasing the test set to create new training data. So you're not technically cheating, but it's like where on the spectrum of scraping GitHub for math problems versus paraphrasing the evaluation set do you draw the line on calling it cheating? Different people have different answers. A goal we kept coming back to is that we understand Molmo is not a 600 billion parameter model that people will immediately download and run code on, but we want to ensure our core audience can do the research we want with confidence and debate. So we want to give people access to every stage, so they can see how this impacts this new important area of research. We'll talk about six stages. One is large-scale pre-training, training on all the internet predicting next tokens. Two is what we call mid-training, which is debatable whether it should exist; technically, you train on higher quality web data with a change in learning rate. Three is long context extension, absolutely essential for these reasoning models because they generate so many intermediate tokens before sharing an answer. Luca has a lot of battle stories from that.

然后我们进入后训练阶段。对我们来说，预训练的那三个构建块可以说是更固定且极其重要的。而后训练，当你处理它时，你有一堆工具，比如优化器，你根据模型的大小和想要的能力按顺序应用它们。所以我们会讨论我们做过的事情，比如指令微调、偏好微调，以及我们再次使用可验证奖励的强化学习。但如果我们训练一个规模大 10 倍的模型，所有这些后训练的东西都会改变，但预训练、中期训练和长上下文我认为实际上会变得非常相似。所以这两个训练阶段存在差异，后训练有点像一门艺术，你必须针对你的具体用例做最合适的事情，这会有变化，但我们也会逐一介绍。

And then we go into post-training which in our case like those three building blocks of pre-training are, I would say, more set and super essential. And then post-training, when you approach this, you have a bag of tools which are like optimizers and you apply them in the order that suits your model depending on size and capabilities you want. So we'll talk about things that we did which is like instruction tuning, preference tuning, and then we did some reinforcement learning with verifiable rewards again. But if we were to train a model that was 10 times as big, all this post-training stuff would change, but the pre-training and mid-training and long context I think would actually become looking pretty similar. So it's kind of a difference across the two phases of training where post-training is like a bit of an art and you have to do what is best for your specific use case and that'll change, but we'll go through these too.

好的，太好了。Luca，你是负责预训练的，Nathan，你是负责后训练的，对吗？这样说公平吗？

Okay, great. And Luca, you're the pre-training guy and Nathan, you're the post-training guy, right? Is that fair?

众多之一。

One of many.

众多之一，但就这次对话而言，在我们深入每个步骤之前，预训练加强化学习这个想法似乎是过去一年左右进展的关键。我知道这个概念更早之前就出现了，但就实现而言，对于一个试图了解这个领域的人来说，正确的思考方式是什么？一部分比另一部分更好吗？它们需要共存吗？目前强化学习带来的收益是否比预训练更多？总体上的高层次看法是什么？

One of many, but for purposes of this conversation, and before we dive into each step, this idea of pre-training plus RL seems to be the key idea in terms of progress in the last year or so. And I know the concept of it came up much before that, but in terms of implementation of it, what's the right way to think about it for somebody that's trying to learn about the space? Is one part better than the other? Do they need to exist together? Is RL currently delivering more gain than pre-training? What's the overall high-level take?

我喜欢这样想：预训练阶段实际上是一个非常昂贵的模型初始化。对吧。当我思考什么是好的最终权重集，可以传递给 Nathan 和后训练团队时，我希望模型拥有丰富的世界知识，并且能开始展现出你希望它在对话时具备的能力火花。所以，从本质上随机的参数出发，创建初始模型是一种非常昂贵且计算密集的方式。但关键在于让模型拥有大量世界事实和信息的知识，并让它开始表现得有点像聊天模型，这样当我们把它交给后训练并进行强化学习时，就有一些行为可以强化并给予奖励，让模型能够学习。我认为，目前关于人们是否应该关注预训练或后训练的讨论之所以困难，是因为我们优化预训练多年，而这类强化学习还有很多未开发的潜力。就像人们说的，OpenAI 想出了一整套技巧让 o1 工作，然后它展示了这个领域的可能性。今年则是一场争夺强化学习低垂果实的竞赛，我认为这是最大的故事，为什么会出现像 o3 这样疯狂的新模型，具备思考和工具使用能力，这些只是下游结果：哦，我们可以做非常不同的事情，因为我们有这样一个良好的平台——这些可塑的预训练模型，我们已经迭代了很长时间。这个强化学习的东西我们本可以更早地利用，但潜力巨大。所以，是的，目前强化学习的改进速度更快，但归根结底，两者之间是一种舞蹈，你需要一个更好的基础模型。人们常说，更好的基础模型和更大的基础模型更容易通过强化学习改进。所以如果你把这当作强化学习研究的核心之一，那么预训练对于实现这一点显然非常重要。

I think the way I like to think of it is the pre-training phase is really like a very expensive initialization of the model. Right. You want to, when I think of what we want to have as a good final set of weights that I can pass to Nathan and the rest of the post-training team, I want a model that has great knowledge about the world, and it also can start seeing sparks of capabilities that you will want the model to have when you chat with it. So it is a very expensive and compute-intensive way to create an initial model out of what is essentially random parameters. But it's all about having this model have a lot of knowledge of world facts and information, and let's have it so that it can start behaving a little bit like a chat model so that when we pass it to post-training and you have this reinforcement learning, there is some behavior to reinforce and to give rewards on so the model can pick it up. I would say that generally the reason why discussions are hard right now on whether or not people should care about pre-training or post-training is that we optimized pre-training for multiple years and then there's a lot of untapped potential on this type of RL where, like what is said, OpenAI figured out a whole bunch of tricks to get o1 to work and then it kind of showed that this area was possible. And then this year has been a race to capture low-hanging fruit on RL, where I think that's kind of the biggest story is why we have all these crazy new models that appear like o3 with this thinking and tool use which are just downstream of, oh, we could do very different things because we have such a good platform as these malleable pre-trained models that we've been iterating on for a long time. This RL stuff we just kind of could have tapped into it much earlier but there's a lot of potential. So yes, the rate of improvement right now in RL is higher, but in the end of the day it's going to be a dance between both of them where you need a better base model. It's said very commonly that a better base model and a bigger base model is much easier to improve with RL. So if you take that as one of the core things of doing RL research, it's pretty obvious that pre-training is very important to enabling that.

顺便提一下，最近有个与 Richard Sutton 的播客，他实际上是在说强化学习才是正道，而预训练和语言模型的前提有点缺陷，因为它本质上是对现实的模仿，按照人类描述现实的方式，而不是通过强化学习面对真实现实。你们对此有什么快速看法吗？

As a quick detour, there's been that podcast with Richard Sutton that was effectively saying that RL was the way to go and that pre-training and LMs was a little bit of a flawed premise because it was sort of an imitation of reality, basically doing the way humans described reality as opposed to being confronted with the actual reality through RL. Do you guys have any quick take on that?

我的看法是，很多人是第一次接触 Rich Sutton，Rich 有很多精彩的想法，但往往不是那些能立即实用的。这就像你创造了强化学习，但不一定能影响 GPT-6 是什么。所以在此之前很多年，我一直在批评 Rich 的生活，因为人们试图把他的想法解释为现实的。我认为 2021 或 2022 年的那篇《Reward is Enough》论文，本质上是在论证一个奖励函数足以得到你想要的任何智能体。所以我认为，与其说是技术辩论，不如说是整个社区第一次被这种观点吸引注意力，这是一种干扰。

My take is that a lot of people are being exposed to Rich Sutton for the first time and Rich is a font of wonderful ideas but often not ones that are going to be immediately practical. This is how you get things like creating reinforcement learning but not necessarily things that are going to impact what GPT-6 is. So I've been on a critiquing Rich life for many years before this in terms of making people try to interpret his ideas as realistic. I think the one from 2021 or 2022 is like 'Reward is Enough' paper which essentially is an argument that a reward function is sufficient to get any intelligent agent that you want. So I think that that's actually rather than the technical debate, as an entertainment of the whole community being nerd-sniped for the first time by that is a distraction.

好的。这个信息并不令人惊讶。实际想法和围绕它的工程之间有一条细微的界限。让语言模型工作很大程度上是工程，这不是贬低，而是说我们需要弄清楚如何将研究想法转化为实际的东西。所以预训练只是初始化这些模型的一种好方法。如果未来出现更好的想法，我们可以转向它。没有人认为语言模型是终极解决方案。在理论上描述系统和实际让它们工作之间有很大区别。否则，从最初的 GPT-3 到 ChatGPT 和 GPT-3.5 之间就不会有两年半的时间差。

Okay. The message is not that surprising. There's this fine line between the actual ideas and then there is the engineering around it. A lot of making language models work is engineering, and not in a denigratory way, but in a way it's like we got to figure out how to translate research ideas into practical things. And so pre-training is just a good way to initialize one of these models. If a better idea comes out in the future, we can switch to that. No one is married to LM being the end-all solution. There's a big difference between describing the system in theory and actually getting them to work. Otherwise there wouldn't be two and a half years between the original GPT-3 and ChatGPT and GPT-3.5.

谢谢。那么让我们逐一讨论这六个模块。先谈谈预训练。你们为这个模型具体做了什么？

Thanks for that. So let's take those six modules turn by turn. So let's talk about pre-training. What did you guys do specifically for this model?

预训练非常有趣。

Pre-training is very interesting.

我们大致是这样规划的——一个好的背景是，预训练过程中我们必须非常有条理，因为首先，预训练需要很长时间。我认为前沿实验室的标准做法是将最终的大规模预训练控制在两个月以内，不超过这个时间。但要在这两个月内让模型不崩溃，你需要做大量的准备工作。所以，所有从事预训练的人都非常有条理。简单描述一下流程：通常你清楚这次运行的时长是固定的，可用的 GPU 数量也是固定的，因此你会编写尽可能快的代码来训练模型。你可以算出模型能处理多少数据。在我们的案例中，这个数字大约是 6 万亿个 token。有了这个数字，我们回过头来确定哪些是最好的 6 万亿个 token。确定方法取决于你能访问哪些数据。我们希望最终能公开这些数据，所以只使用公开可用的数据——比如互联网文本、网上能找到的 PDF 文档或代码。在初始数据池中，我们大约有 300 万亿个 token。你需要将其缩减到目标数量，并希望只保留最好的部分。因此，你要去重，并判断哪些文档更好。我们有评估模型能力的方法。所以，如果你的评估侧重于医学文档（因为有医学测试），你就会想办法挑选包含良好医学信息的文档，可能牺牲其他领域。但这是一种微妙的平衡。在确定初始运行并开始训练后，还需要确保模型设计不会突然忘记已学内容——我们称之为语言模型中的“尖峰”，但基本上你不希望发生那种必须从头开始且无法恢复的事件。这方面有很多工作要做。经过几个月的训练，你会得到一个最终模型。但这个模型仍然缺乏一些 Nathan 团队关心的能力——比如长上下文或解决某些问题的能力。这就是长上下文扩展或中期训练发挥作用的地方。

So the way we sort of planned it—a good background is that pre-training, all that happens during it, we have to be very methodical in how we do it because first of all, it takes a long time to pre-train. I think it's standard practice among the frontier labs to try to cap your big final pre-training run to two months, not more than that. But to get to something that will not crush and burn in two months, during these two months you have to do a lot of preparation around it. So we're really—everyone who works on pre-training is fairly methodical. Just to sketch out how that works: usually you have a sense that the duration of this run is fixed, the number of GPUs I have available is fixed, and therefore you write the fastest possible code to train this model. You have these three—you can figure out how much data I can show my model. In our case, that number was like six trillion tokens. Given that number, then we go back and figure out what are the best six trillion tokens out there. The way you figure that out is a combination of what data you have access to. We want to eventually release the data, so we limit ourselves to data that is publicly available—either internet text, PDF documents you can find on the internet, or code you can find on the internet. Among this pool, our initial pool was closer to 300 trillion tokens. You shrink it down until you reach your target number, and hopefully as you shrink, you only keep the best part. So you remove duplicates, you have a way to judge whether this document is better than that document. We have a way to evaluate the capability of the model. So you pick—if your evaluations focus on medical documents because there's a medical test, you figure out how to pick documents that have good medical information, maybe at the expense of some other domains. But yes, it's a delicate balancing act to find this data. After you commit to this initial run and you do your training, there's a lot of making sure that the way you design the model doesn't suddenly start forgetting what it learned—we call these spikes in the language model, but basically you don't want an event that, if it happens, you have to restart from scratch and you can't recover. So there's a lot of work on that. But after these months of training, you get to a final model. On this final model, it will still lack some capabilities that I know Nathan's team cares about—like long context or being able to solve some problems to start with. That's where things like long context extension or mid-training happen.

好，我们来谈谈这个。第二阶段，也就是中期训练。这个术语我以前没听说过。Nathan，简单描述一下它是什么，但请深入一点。

Yeah, let's get into that. So that phase two, so mid-training. Again, a term I personally hadn't heard of before. And Nathan, briefly describe what it is, but just double-click on that.

我听说有些实验室不叫中期训练，而叫“尾部修补”，我觉得这个术语好得多。意思是，在预训练的尾部，你对模型进行修补，让它学会预训练期间没学到的东西。在这个阶段，你当然还需要确保模型不会忘记预训练的内容，所以你会混合一些预训练中的最佳数据。

I heard that some labs instead of mid-training call it tail patching, which I think is a much better term. The idea is that at the tail of pre-training, you patch the model so that the things it hasn't learned during pre-training, it will learn after. At that phase, of course, when you do that, you also need to make sure the model doesn't forget stuff from pre-training, so you mix in some of the best data from pre-training.

所以你会给它更多代码数据或数学数据之类的。

So you give it more code data, for example, or math data, that kind of stuff.

如果你想让模型推理某些数学问题，你就会这么做。这就是 Nathan 之前提到的。有时在这个阶段会出现类似测试数据的泄露。有一种不变的说法：人们可能会说有人试图通过添加这些数据来作弊，但也很容易意外泄露测试数据。我们花了很多时间来确保这种情况不会发生，因为这是一个非常微妙的平衡。你希望模型开始能够解决测试中遇到的问题，但你真的不希望测试数据意外泄露，否则你就无法衡量模型的表现。

If you want the model to reason about certain math problems, you do that. That's when Nathan mentioned earlier. Sometimes there is leakage of things that look like the test during this phase. There's an unchangeable way to describe it: people might say someone is trying to cheat by adding this data, but it's also so easy to accidentally leak your test data in there. We spend a lot of time making sure that doesn't happen because you want to add—it's a really tricky balance. You want the model to start being able to solve problems like the ones you see during the test, but you really don't want that test data to accidentally leak, otherwise you can't measure how well your model does.

然后你提到了长上下文，这是六阶段流程中的第三阶段。为什么关注长上下文，首先长上下文是什么意思？

And then you mentioned long context, which is the third stage in the six-stage pipeline. So why the focus on long context, and I guess what does long context mean in the first place?

你希望这些模型能够处理非常长的文本序列，无论是作为输入——比如你想给它一堆文档——还是输出大量文本，尤其是现在有了推理轨迹，对吧？这些思考 token。为什么我们不从一开始就训练模型做到这一点？因为模型训练的输入越长，速度就越慢——上下文长度会导致二次方减速。所以我们绝对不想在整个预训练中都使用极长的序列。但到了某个阶段，我们必须教会模型处理长序列，我们把它留到最后，以便高效地进行。

You want these models to be able to work with very long sequences of text, both as input—imagine you want to give a collection of documents—and also generate a lot of text in the output, especially now that you have reasoning traces, right? These thinking tokens. Why don't we train from the beginning the model to be able to do that? Because the longer the input a model is trained on, the slower it gets—it's a quadratic slowdown with context length. So we definitely don't want to do the entire pre-training at extremely long sequences. But at some point, we have to teach the model to actually work with long sequences, and we save it for the very end so that we can do it in an efficient way.

我记得你提到过，对于长上下文来说，数据并不重要。你是什么意思，那什么才重要？

And I think you mentioned somewhere that data doesn't matter for long context. What do you mean by that, and then what does matter?

啊，这有点深入细节了。

Ah, this is getting a little into the weeds. It's—

不，Luca 热爱数据。Luca 喜欢在暗房里埋头处理 token 来训练模型。这种痛苦的情感背景——

No, Luca loves data. Luca likes to be in the dark room grinding out tokens to train the models. The emotional backdrop for this painful—

非常技术性的东西。我用 QOR 吗？用 GQA 吗？其实并不重要。但如何设置模型存在技术决策。你可以拥有世界上最好的数据，但你的模型仍然无法处理大量 token。所以，说数据不重要，是指你不能用坏数据训练模型，但即使你有最好的数据，如果模型设置错误，你也永远无法恢复。所以遗憾的是，我不能成为用神奇 token 拯救模型的人。我们必须用正确的架构来构建模型。

Very technical stuff. Do I use QOR? Do I use GQA? Doesn't really matter. But there are technical decisions in how you set up your model. You can have the best data in the world and your model will not be able to reason over many tokens. So it doesn't matter in the sense that you can't train the model on bad data, but you can have the best in the world, but if you set up your model wrong, you're never going to recover it. So sadly, I can't be the savior with the magic tokens that makes the model good. We have to make the model with the right architecture.

好，这就是第三阶段，长上下文。也许为了更生动地说明，前后有什么区别？比如，如果你把一个 40 页的 PDF 放入上下文窗口，它会得到更快的结果还是更好的结果？

Okay, so that's stage three, long context. Maybe just to bring this to life, what's the difference between before and after? Like if you have a 40-page PDF that you fit into the window, it will just get faster results or better results.

一开始是什么情况？你不能像预训练那样只做 4000 到 8000 个 token。我们给 OMO 3 用的就是这个量，Llama 也是。这大概相当于八页纸，如果你用双倍行距的话。之后我们扩展到大约 65,000 个 token。在业界，有人扩展到百万 token。我觉得 Gemini 最近宣布了超过百万 token。到那个程度，百万 token 大概要 10 美元。所以你可以处理极长的信息量。这很好。你在用这个语言模型构建应用时，不用再考虑如何提取需要展示给模型的信息。你可以直接把所有信息给它，模型自己会搞定。所以这真的解锁了很多机会。

What happens at the beginning? You just can't do it like pre-training is something like 4,000 to 8,000 tokens. That's what we use for OMO 3. That's what Llama uses. That's about maybe eight pages if you use double spacing. And after that we extend to about 65,000. In industry you have extension of a million tokens. I think Gemini recently announced over a million tokens. At that point a million tokens is like 10 bucks. So you can work with extremely long amounts of information. It's nice. You don't have to think about, if you're building an application with this language model, you don't have to think about how to extract the information you need to show the model. You can just give it all and the model will figure it out. So it really unlocks a lot of opportunities.

好了，这就是预训练的世界，包括预训练本身、中期训练和长上下文。现在让我们切换到后训练的世界。Nathan，你来吧。先从 SFT 开始，也就是监督微调。

All right, so that's the pre-training world between pre-training itself, mid-training, and long context. So now let's switch to the post-training world. Nathan, if you will. So starting with SFT, which stands for supervised fine-tuning.

是的。我认为有一件事，尤其是对于像 Mo 这样我们逐步拼凑起来的模型，最大的变化之一是当推理模型变得流行时，行业流行的评估套件会转向加入大量新内容。所以在每个阶段都会发生的一件事是，即使很多数据有重叠，你也会以不同的方式混合它们。所以我认为 Kyle Low 和 May Chen，另一位研究员和实习生，做了这个混合流程，我们在所有阶段都使用它，就是为了提高数学、代码和推理内容的权重，确保后训练阶段更易处理，并且所有东西都设置好了。这就是我们必须做的事情，它贯穿始终。然后后训练，我认为对于这个模型，我们做的一切都基于一个假设：这是一个大约 7B 的模型。我们非常聚焦，因此我们要做很多人做过的事情，即从更大的教师推理模型中蒸馏。我认为蒸馏被描述为当你从一个模型获取输出，然后在其上进行微调。我认为社区对此有很多广泛的讨论。然后这个监督微调阶段，或者说 SFT 或指令微调，就是获取来自推理模型或社区的最佳轨迹，然后教你最近训练的基础模型非常接近地模仿这些行为。所以在我们的案例中，我们混合了现有的数据集，比如 Open Thoughts 3，并进行了修改，它来自一家初创公司 Bespoke AI Labs，然后我们还生成了大量新数据。所以我们最终使用了来自 DeepSeek R1、DeepSeek R1-0528（他们的更新版本）以及 Qwen 的推理模型 QWQ 的混合教师模型。这些往往是非常强大的教师模型。

Yeah. I think one of the things, especially for a model like Mo where we're scrappy and putting everything together over time, is that one of the biggest changes is that when reasoning models become popular, the in-vogue evaluation suite of the industry shifts to add a whole bunch more new things. So one of the things that happens at every stage is that even if a lot of the data has overlap, you mix it in a different way. So I think Kyle Low and May Chen, another researcher and intern, did this whole mixing procedure that we use across all these stages just to upweight the math, code, and reasoning stuff to make sure that what happens later in post-training is much more tractable and that all this stuff is set up. So that's the type of thing we have to do that's baked into everything. And then post-training, I think for this model everything we're doing is operating under the assumption that this is about a 7B model. We are very narrowly focused and therefore we're going to do what many people have done, which is called distilling from bigger teacher reasoning models. I think distillation is described as when you take the outputs from one model and then you fine-tune on it later. I think there's been a lot of broader discussions on this in the community. And then this supervised fine-tuning stage, or SFT or instruction tuning, is all about just getting the best traces from reasoning models out there or the community and then just teaching your recently trained base model to behave really closely to what is going on there. So in our case, we took a mix of existing datasets like Open Thoughts 3 and modified it, which is from Bespoke AI Labs, a startup, and then we also generated a whole bunch of new data. So we ended up using a mix of teachers from DeepSeek R1, DeepSeek R1-0528, which was their updated version, and then Qwen's reasoning model, QWQ. These tend to be pretty strong teachers.

这是为什么？你有一个预训练模型，但在监督微调中，你却用另一个模型进行蒸馏。简单来说，这是为什么？

Why is that? So you have a pre-trained model, but for supervised fine-tuning, you distill using a different model. Why is that in simple terms?

本质上是因为我们的小模型无法输出同样强大的文本，即使……所以过程中有一个分叉。如果我们有一个更大的模型，我们会做大量的强化学习来开始，然后模型会花时间学习这些有趣的行为并获得强大的性能。但对于小模型，天花板相当低。它没有能力从这些更难的数学问题中学习。所以常见的做法是，你获取你能得到的最好的推理模型，这些模型是公开可用的，有良好的许可证，你可以自己生成新数据，在上面训练，然后发布给社区，今年我们看到了很多这样的做法。因此，在性能上最接近前沿且拥有良好许可证的模型，今年全年恰好都是中国模型。我认为在我们的案例中，即使 GPoss 存在，我也不会用它来生成合成数据，因为那个模型主要是为工具使用设计的，而我们在项目中做了一些工具使用，但不像那个模型那样，是那种多跳的智能体式推理，带有搜索等功能。所以像 DeepSeek 和 Qwen 这样的模型在生成数学和代码答案以及其他内容方面非常强大，而且通常很稳健。这就是我们所做的。我们有大约 250 万条推理轨迹，主要集中在数学、代码和 STEM 上，但也包括聊天和其他通用能力，模型在这一点上吸收了很多。我认为如果你去年在开发 Elmo 2 时告诉我们，如果我们有一个类似规模的 Elmo 模型，在数学上能得 95 分，在 AMC 这些疯狂的数学基准上能得 70 分，那会令人惊讶，但这就是当你直接从这些非常强大的模型中提取数据并进行蒸馏时所能得到的。我认为实际上很多公司都会想这样做，因为你可以为你的领域做同样的事。我认为我们生成了一个全面的方案：我们希望所有这些来自指令遵循的基准，并确保你实际上可以和模型对话，而不会让它完全崩溃。但你可以为你想要的任何特定任务做这件事，只要 DeepSeek 覆盖了它，而且非常高效。

Essentially because our small model is not going to be able to output as strong of text even if... So there's a kind of a fork in process. If we had a bigger model, what we would do is do a lot of reinforcement learning to start, and the model then would take time to learn these interesting behaviors and have strong performance. But with a smaller model, the ceiling on that is fairly low. It just doesn't have the capacity to learn from these harder math problems. So what the common practice is, is you take the absolute best reasoning models you can get that are openly available with a good license where you can just generate new data yourself and train on it and release it to the community, which is something we've been seeing a lot of this year. And therefore the models that are closest to the frontier in performance with good license all happen to be Chinese models throughout the year for this case. And I think in our case, even if GPoss had existed, I don't think we would have used it for synthetic data in this case because that model is really designed for tool use, which is something we did a bit of in this project but not in the sense that that model is, which is like this many-hop agentic reasoning with search and stuff. So like the DeepSeek and Qwen models of the world are just powerhouses at generating math and code answers and other things and being generally robust. So that's what we do. We have about 2.5 million reasoning traces, mostly on math and code and STEM, but also on chat and other general capabilities, and the model really absorbs a lot at this point. I think if you would have told us last year when working on Elmo 2 that if we had a similarly sized Elmo model that gets like 95 on math and like 70 on AMC on these crazy math benchmarks, it would have been surprising, but this is just what you can get when you can extract data directly from these really powerful models and distill it down. And I think realistically a lot of companies are going to want to do this because you can do this for your domain. I think we generated a blanket approach: we want all of these benchmarks from instruction following and make sure that you can actually talk to the model and not have it become totally broken. But you can do this in any specific task you want if DeepSeek has coverage on it, and it's very efficient.

好的，很好。

Okay, great.

然后之后的大部分过程，那就是基础。如果你在训练一个小型推理模型，你需要这样做。之后的其他事情是如何提取更多性能，它们很快变得更技术性，或者因为我们想做而做，可能在我们的时间里效率不高。所以今年夏天我们 90% 的时间都花在让优秀的人攻克强化学习基础设施上，因为当 Luc 暗示当你生成大量 token 时，时间或计算和内存的增加是二次方的。因此，你几乎会遇到框架中的每一个可能的 bug 或每一个可能的边界情况，这些都会让你的工作停滞。但大部分性能是通过这个 SFT 和之前的偏好调优获得的。但 RL 是我们需要做的，以便为我们今年晚些时候想要构建的未来基础设施打下基础，那时模型会变得更大，它们可以用工具做更有趣的推理等等。所以这算是模型的一个细微之处。

And then most of the process after that is like that is the foundation. And if you're training a small reasoning model, you need to do this. And then the other things after are how do you extract more performance, and they quickly become more technical or done because we want to do them and maybe not efficient in our time. So like 90% of our time this summer is having great people battle reinforcement learning infrastructure because when Luc is hinting at when you generate a lot of tokens, the time or compute increase and memory increase is quadratic. Therefore, you pretty much encounter every possible bug in your framework or every possible corner case that'll make your job go to a halt. But most of the performance is through this SFT and the preference tuning that comes before it. But the RL is like we need to do this in order to build the infrastructure for many of the future that we want to build later this year where they get bigger and they can do more interesting reasoning with tools and so on. So it's kind of a nuance point of the model.

是的，我们会向你展示我们在最后通过强化学习获得了一些提升，但真正重要的是，强化学习工具对于构建接下来的模型至关重要。

Yeah, we'll show you that we got a couple points out of doing RL at the end, but really the RL tooling is something that's so crucial to doing the next models that come from here.

没错。谢谢你的解释。为了能让更多对 AI 工作原理感到好奇的人理解，我再确认一下：SFT 还不是强化学习，对吧？那是监督微调。也就是说，你给模型展示一个“黄金标准”的样本，然后让它基于这些标注数据进行训练。这样描述对吗？

Right. And thank you for that. And just to again, in an effort to make this interesting to a broad group of people who are curious to understand how AI works. So SFT is not RL yet, right? That's supervised fine-tuning. So that means that you basically show the model a golden copy of what good looks like and you train it based on that labeled data. Is that a good way to describe it?

这和预训练用的损失函数一样，都是预测下一个词。具体来说，问题可能像是一个很难的数学题，比如“列出在 x 和 k 的约束下的所有质数”。就这一句话，模型会生成 3 万个词，比如“让我想想，这样做，然后测试这个，我得用这个定理和假设”。我们之前聊过一点关于词元直觉，但用 3 万个词元解一道数学题确实很惊人。如果让我坐下来读这个，我可能要花几个小时才能读完这一个数学解答。所以这些模型在很多方面都很难理解。我认为推理模型有时会陷入反复猜测和检查，尝试几百次后才意识到不能再这样下去。这就是我们的推理模型。前沿模型可能已经解决了这个问题，但这些词元里确实有很多非常奇怪的东西。即便如此，这种下一个词预测仍然是许多人使用的强大性能基础。它并不匹配人类的推理方式或人们期望它做的事，但它教会了模型自己的一套语言，即逐步分解问题以实现目标。

So it's the same loss function as pre-training, which is you're predicting the next token. In this case, what it looks like is a question could be like, I don't know, something like an Amy style, like a really hard math question: list all the prime numbers within some constraint of x and k. It's like this one sentence that is really hard, and then the model generates 30,000 tokens of "let me think about this and do this and to test this I'll have to use this theorem and hypothesis" which is like the 30 we talked about token intuitions for a bit. But 30,000 tokens to solve a math problem is pretty mind-bending. So if I were to sit there and read this, it would be hours of me just trying to read this one math solution. So these models are very unintelligible in many ways. I think the reasoning models sometimes will go into a bout of guess and check for hundreds of attempts before realizing that they can no longer guess and check. I mean, this is like our reasoning model. I think the frontier models could have probably done this and fixed this issue, but there's just really, really odd things in these tokens. But even with that, doing this next-token prediction is an incredible foundation of performance that many people use. So it's not matching any sort of human reasoning or things that people might want it to do, but it is teaching the models kind of their own language of breaking down problems step by step in order to solve a goal.

对于 SFT 这个特定阶段，你能谈谈你们是如何为它创建数据集的吗？也就是精确地构建模型认为“好”的样本？

For this specific stage of SFT, do you want to talk about how you went about creating the dataset for it? So precisely this representation of what good looks like for the model?

好的，Luka，你想插话吗？你也有要说的吗？

Yeah, Luka, do you want to jump in? Do you have things too or you...

我想提的另一件事是，有时在前沿实验室的重大公告中，你看不到 Nathan 描述的那种情况，即必须先做 SFT 再做强化学习。这很常见，我们经常处于这种未知技术领域。所以，你一无所有，必须想办法修复流程中的某些部分，然后才能构建其余部分，之后再回头修复第一部分。对我们来说，我们想在这个更大的模型上做强化学习。好的，我们需要强化学习代码非常快、非常可靠且有用。如果我们需要迭代那部分，我们想先用较小的模型迭代，因为迭代更快。它们消耗的算力更少，所以我们可以并行做更多事情。但较小的模型不能先完成所有事情。你必须先创建数据。你必须先做 SFT。然后我们很幸运，有其他优秀的开源模型可以用来创建这些数据，否则就得花大量时间指导人类创建同样数量的数据，这会拖慢进度。所以这很大程度上就像你在火车以惊人速度行驶时铺设轨道，必须想办法修复流程的某些部分，以便继续处理其余部分。

The other thing I was going to mention is that sometimes in the big announcement of the frontier labs you don't see what Nathan was describing around like having to do SFT to then do RL. It's very common, we are in a common situation where this is uncharted technology, right? So you have nothing, you have to find ways to fix some components of your pipeline before you can build the rest of your pipeline and then go back fixing the first part. So for us, it's okay, we want to do reinforcement learning on this larger model. Okay, we need our reinforcement learning code to actually be super fast, super reliable and useful. If we need to iterate on that part, we want to iterate with a smaller model first because we can iterate faster. They take less compute to work, so we can do more things in parallel. Okay, smaller models they cannot do all first. You got to create the data first. You have to go through the SFT. And then we're lucky enough that there are other great models that are open source that we can use to create this data, versus the alternative would be, I don't know, to spend a lot of time instructing humans to create the same volume of data, slow things down. So it's a lot of this of like, you're building the tracks as the train is going down at incredible speeds and you have to figure out ways to fix some parts of your pipeline so you can work on the rest.

好的。那我们聊聊流程的下一阶段。第五阶段，DPO 和偏好调优。那是什么？它有什么作用？

All right. So let's talk about the next stage in the pipeline. Stage five, DPO and preference tuning. What is that? What does that do?

是的。这是那种“嘿，试试这个，我们不确定它会不会有效”的事情，在流程后期，当你花了很多时间在其他事情上时，它却效果很好。我认为 DPO，即直接偏好优化，并不完全是新东西。它是一种优化偏好的方法，与我们之前提到的 RLHF 有关。从技术上讲，一句话概括，它是一个解析推导出的损失函数，本质上是对 RLHF 目标应用随机梯度下降。所以它比其他方法更容易实现。我们过去在 Mo 2、Tulu 3、Tulu 2 等模型上用过。问题是，我们能否直接将其应用于推理模型？我们知道它在很多不同情况下都有效，因为不确定这些长推理轨迹被包含在损失函数中会发生什么。所以，有一个学生 Scott 一直在研究这个，他称之为 delta 学习假设，这是一种直觉，认为 DPO 更关注所选样本和拒绝样本之间的对比。偏好学习的核心是你有同一提示的多个补全对或分组。所以一个问题有多个补全，他的直觉和工作表明，这种对比比答案的绝对好坏程度更重要。所以他花了很多时间尝试为推理模型找到好的配对，这些模型是开源的，有开放权重和宽松许可证，并且包含推理轨迹，因为我们确实需要这个，而且它们需要足够多样化。所以我们花了很多时间生成这些数据，并进行一些常规的调整，比如调整学习率和小细节，结果就是，是的，这有效。我认为我们做完之后，看到 Hugging Face 用小型语言模型做了类似的事情，他们训练了一个完全开放的 3B 模型，也做了预训练。有趣的是，我们最终都使用了相同的 Qwen 32B 和 Qwen 0.6B。问题是，这些小型 Qwen 模型和小型公开推理模型实际上非常强大，以至于要获得足够的 delta 来应用这种偏好学习技术变得很困难。我们过去的技术是从一组模型中采样，但随着这些开放模型越来越好，这些样本变得过于同质化，导致学习信号无法存在。

Yeah. So this is one of the things that is thought of as "hey, let's try this. We're not sure if it'll work" kind of later in the process when you spend a lot of time on other things and it works very well. I think DPO, or Direct Preference Optimization, is not exactly new. I think it's a way of optimizing for preferences. It's related to this whole RLHF thing that we mentioned. Technically speaking, in one sentence, it's an analytically derived loss function that is essentially applying stochastic gradient descent to the RLHF objective. So it becomes much easier to implement than other things. And we used this in the past with Mo 2, with Tulu 3, Tulu 2, others. And the question was, can we apply this out of the box on top of a reasoning model? And we knew that it works in many different situations because we weren't sure what would happen with these long reasoning traces being included in the loss function and so on. So then essentially, there's a student, Scott, that has been working on this, what he calls the delta learning hypothesis, which is an intuition for understanding DPO as being more about the contrast between your chosen and rejected examples. So the core of preference learning is that you have pairs or some grouping of completions to the same prompt. So you have one question with multiple completions, and his intuition and work is showing that this contrast is more important than the exact magnitude of goodness of the answer. So what he did is he spent a lot of time trying to come up with a good pairing of reasoning models, which are open source so that they have open weights and a permissive license and they include the reasoning traces because we kind of need this, and you need them to be sufficiently well spread about. So we spent a bunch of time generating this data and doing some normal kind of like, let's fiddle with the learning rate and small things, and it's kind of just like, yes, this works. I think after we did it, we saw that Hugging Face did something similar with small LM, so they trained like a fully open 3B model where they pre-trained it as well. And the funny thing is that we converged on using the same Qwen 32B and Qwen 0.6B. So the problem is that these small Qwen models and these small public reasoning models are actually so strong that getting a sufficient delta to another model to apply this preference learning technique was kind of hard. So our past techniques, we kind of had groups of models we sampled from, but as these open models are getting better, these samples become too homogeneous for the learning signal to exist.

所以这是一个很酷的实验，因为它验证了潮汐变化的假设。想想几年前 Alpaca 那些模型，它们问题很多，但这个群体有足够的方差和对比，让我们可以做一种不同类型的偏好学习。现在你真的要仔细看生成结果，确保模型有学习信号。我们这么做了，结果全面提升了。我觉得有时候当你做了仔细的数据工作并理解优化器后，事情看起来就很简单。

So it's a kind of cool experiment because it validates the hypothesis of the changing tides. If you think about years ago with Alpaca and stuff, those models were so broken that having this group had enough variance and contrast in it where we could do a different type of preference learning. Now it's really you have to look very closely at the completions and make sure there's a learning signal for the models. We did this and it gave us a boost across the board. I think sometimes things look very easy when you've done careful data work and set up to understand your optimizers.

我觉得 Luca 把预训练描述为非常科学，而后训练则是狂野西部。有很多类似的比喻。所以我做了这个 SFT 数据集，我们有一堆云积分，快用完了，而且我们落后了。我就尽可能多地生成了生成结果。另外，我们还有 DeepSeek 周末产生的几十亿个生成结果，想着‘哦，我们之后再混合和过滤’。我做了过滤，结果是我们几乎全部保留。我们做得很少，如果有更多资源我们会做更多，但有时候有低垂的果实，做显而易见的事就能带来很多成果。

I think Luca described pre-training as very scientific and post-training as the Wild West. There are many analogies. So I made this SFT dataset where we had a bunch of cloud credits and they were running out and we were behind. I just generated as many completions as possible. Also we had a few billion completions from DeepSeek over the weekend, like 'oh we'll mix it and filter it later.' I applied filtering and the answer was we just include almost all of it. We did very little, we would have liked to do more if we had the resources for longer, but sometimes there's low-hanging fruit and doing the obvious thing yields a lot of results.

这个 SFT 和 DPO 阶段在很多意义上就是这样，而这种 RL 阶段则是极其艰难的技术打磨，周复一周地让工具能运行起来。后训练中的差异对我来说是合理的。你看到所有这些检查点飞来飞去，看起来一片混乱，然后某个极其明显的东西给你带来巨大收益。DPO 的收益就像从 Qwen 2 水平到几乎 Qwen 3 水平的差距。你用来达到那里的东西有时真的很明显。

This SFT and DPO stage in a lot of sense are that, and then this kind of RL stage is extremely hard technical grinding week in and week out to make the tools even run at all. The disparity in post-training tracks to me. You have all these checkpoints flying around and it seems like chaos, and then something extremely obvious gives you a massive gain. The DPO gains are like the difference from being about Qwen 2 level to almost Qwen 3 level. The thing you apply to get there is sometimes really obvious.

我认为前沿实验室在这条路上走得更远，他们很快就能摘到这些低垂的果实。但作为一个较小的团队，试图跟上领域不断变化的优先级，有时就是转动一个非常直接的东西的曲柄。我不记得是谁说的，但 Anthropic 的 Dario 非常直白地说：看，这里有效的是 50 到 100 行代码。他是在间谍活动的背景下说的，担心 Anthropic 的一些商业机密被泄露出去。但最终行业青睐的解决方案实际上非常简单。问题在于，有大量同样简单的解决方案，所有工作都集中在如何测试这些方案、如何尽可能快地测试、如何说服自己看起来好的结果不是因为某个 bug 导致某些东西好得难以置信。所以很大程度上，重要的不是最终解决方案，而是你迭代的速度和工具的稳健性，这样你看到好结果后就能立刻知道它是好结果。

I think the frontier labs are much further down this path where they take these low-hanging fruits so fast. But as a smaller team trying to map to the changing priorities of the field, sometimes it's just turning the crank on a really straightforward thing. I don't remember who said it, but Dario from Anthropic said very plainly: look, what works here is 50 to 100 lines of code. He was saying it in the context of espionage and being scared about some trade secret from Anthropic being exported out. But the solutions at the end that the industry favors are actually very simple. The problem is that there's a very large space of equally simple solutions, and all the work goes into how do you test these, how do you test them as fast as possible, how do you convince yourself that results that look good are not just because there's a bug somewhere causing something to be too good to be true. So a lot of it is less about the final solution and more about the speed at which you iterate and how robust your tools are so that immediately after you see a good result, you know it's a good result.

好的。为了完成我们开始讨论 RL 的旅程，RLVR——基于可验证奖励的强化学习。让我们花点时间专门谈谈这个阶段。Nathan，我知道那是你的宝贝，或者你是这个宝贝的父亲之一。你能快速带我们回顾一下历史吗？

All right. To complete the journey since we started talking about RL, RLVR - reinforcement on verifiable rewards. Let's spend a little bit of time on that stage in particular. Nathan, I understand that's your baby, or you're one of the fathers of the baby. Do you want to walk us maybe a little bit through the history quickly?

我想我是那个把它公开带给世界的人。众所周知，业界很多人已经做了好几年，然后这项技术开始变得更有影响力。它基本上是采用现有的强化学习算法或下游演化，比如近端策略优化（PPO），它是 REINFORCE 的演化，然后 DeepSeek 有了他们的组相对策略优化。我总是想说‘组鲁棒’，但我觉得是‘组相对’。所有这些算法都非常相似。你根据模型是否答对来训练它们，或者在代码的情况下，根据测试是否执行且不失败来训练。

I think I'm the person that got to bring it publicly to the world. It's well known that people across industry have been doing this for years, and then the technique started to get far more impactful. It's broadly taking existing reinforcement learning algorithms or downstream evolution of like Proximal Policy Optimization (PPO), which is an evolution of REINFORCE, and then DeepSeek had their Group Relative Policy Optimization. I always try to say 'group robust' and I think it's 'group relative'. All these algorithms are really quite similar. You're training the models with whether or not they got the answers right, or in the case of code, whether or not the tests execute and don't fail.

我认为一个著名的例子是，做太多这种 RL 方法或急于摘取低垂的果实，会导致所有这些代码模型做各种 try-except 来避免错误，因为它们接受所有错误。我认为这只是因为模型变得有用的收益远高于烦恼以及它也会做这些蠢事的事实。我们最终会修复这些蠢事。对于这个模型，没什么疯狂的。我们在 RL 数学问题上撒大网。我们做一些数据比较，看看哪些数据最适合教这些模型。我们混合代码和精确指令遵循。这种混合实际上是根据你从许多实验中知道的以及你开始的特定模型检查点来调整你拥有的大数据集。所以如果你有一个非常强的模型，给它看非常简单的数学问题，就没有学习信号。如果你有一个非常弱的模型，给它看非常难的问题，它全错，也没有学习信号。学习信号完全来自你有时对有时错的梯度。

I think one of the famous examples is that doing too much of this kind of, or racing to get the low-hanging fruit from this RL approach, is what makes all these code models do all these try-except things to avoid errors because they accept all the errors. I think that is just because the gains you get in the model being useful is so much higher than the annoyance and the fact that it also does these stupid things. We'll fix the stupid things eventually. In the case of this model, it's not anything crazy. We take a very wide net on RL math problems. We do some data comparisons to see which data we think is the best for teaching these models. We do mixing with code and precise instruction following. This mixing is effectively when you tune the big set that you have to what you've known from many experiments and to the specific model checkpoint that you're starting on. So if you have a really strong model and you show it really easy math problems, there's no learning signal. If you have a really weak model and you show it really hard problems, it gets them all wrong. There's no learning signal. The learning signal is all from the gradient of you sometimes get it right and you sometimes get it wrong.

你想给出 RLVR 与 RLHF 的通俗定义吗？

You want to give the plain English definition of RLVR versus RLHF?

RLVR 中的可验证奖励就在名字里。本质上，你从环境（即生成结果或评分器）获得的奖励是你是否答对了问题。RLHF 的奖励是某种奖励模型，它根据人类偏好的代理来评估响应的质量。所以可以说 RLVR 的奖励更容易理解，因为这些奖励模型往往有很多问题，你更容易过度优化它们，因为奖励模型会捕捉到你可能并不关心的特征，比如表情符号之类。RLVR 更匹配性能特征而不是风格。

RLVR with verifiable rewards is in the name. Essentially the reward that you get from the environment, which is the completion or the grader, is whether or not you got the problem right. RLHF, the reward is some reward model which is rating the quality of the response based on a proxy to what humans would like. So it's described as being much like the RLVR reward is much easier to understand because these reward models tend to have a lot of problems and you can overoptimize them much more easily because the reward models will pick up on features that are maybe emojis or something like this that you don't actually care about. RLVR is much better matched to performance characteristics rather than style.

还有你发推或在某处说过，非上下文推理任务中的 RL 非常困难。

And what you tweeted or said somewhere that RL in non-context reasoning tasks is very hard.

我不确定这是否是强化学习的普遍问题，但正是这种特性让它变得极其困难，并且成为当前 AI 的前沿。

I don't know if that's RL in general, specifically this type of what makes it super hard and very much the frontier of AI right now.

所以你的工具在很多方面都可能出问题。我认为目前大多数流程的设置是：你有一组生成 GPU（比如 VLM），还有一组训练 GPU（某种分布式学习框架），在那里进行实际的强化学习更新和损失函数计算，因此你需要某种系统来协调两者并来回传递信息，而这种信息传递非常麻烦。这是一个系统问题，因为涉及到分布式错误处理等。一个常见的情况是，在最基本的方法中，你会有一个生成任务，比如一个数学问题，模型一直在思考，所以所有 GPU 都在处理一个问题，实际上整个系统都在闲置等待答案。还有很多类似的小问题，比如长上下文生成会消耗大量内存，以至于你需要引入不同类型的并行化来有效进行生成，而且还有很多微妙的数值问题。所以我认为这就像是对我们已有的后训练基础设施进行压力测试，把各种可能出错的因素调到最大。我认为开源社区面临的一个问题是，VLM 和 Hugging Face 使用不同的内核来进行模型的实际内部计算。这些内核让 VLM 变得非常快，但这会导致模型生成的完成结果与计算损失函数的 log props 之间存在微妙的数值差异。如果你看这些强化学习算法的数学原理，它们假设这些来自同一分布，因此这是许多数值问题的根本原因。然后看看我们多年来所做的，很多实验室都采用了不同的修复方法来改变这些数值。我认为机器有一篇著名的博客，是他们最早的博客之一，关于确定性 VLM，使其完全确定，这非常有用，人们认为这是他们 tinker API 和进行其他强化学习工作的关键，你可以完全控制非确定性的来源，而这可能只是强化学习中数值鲁棒性不足的问题。如果我们谈论强化学习中的商业秘密之类的东西，也有关于开源实验室的算法是否比闭源实验室更差的讨论，实际上大多数人似乎都在使用某种进化版的 GRPO，它比 PPO 简单一些。所以有些实验室可能使用学习到的价值函数。细节并不重要，但每个实验室都会找到他们需要的一系列调整，以获得非常稳定的强化学习性能。在强化学习文献中，历史上对所谓“新算法”所需的改动门槛很低，但实际上这只是一个实现细节。所以每个人都会找到他们稳定的操作配置，而且这非常依赖于工具。因此，你可以说他们有不同算法，但这也不是你能轻易从实验室窃取的东西，因为它依赖于堆栈的多个层次，可能还有他们使用的芯片等各种因素。所以训练这些模型很复杂，简单的评论永远无法反映这一点。

So there's many ways that your tooling could fail. I think where most of these processes are set up right now is that you have a set of generation GPUs which look like something like VLM and you have a set of training GPUs which is some distributed learning framework which is where you actually have this RL update and loss function and therefore you need to have some sort of system that orchestrates the two and passes information back and forth and this kind of information passing back and forth is really annoying. It's a systems problem because you have distributed error handling and things like this. So like a common case is when you have the most basic approach is that you'll have like one generation this one math problem the model is thinking and thinking and thinking and thinking so you have all these GPUs working on one problem so effectively your whole system is somewhat idle waiting for the answer and there's many other small things like this which is this long context generation just uses so much memory that you then need to introduce different types of parallelism and stuff to do the generation effectively and there's just a lot subtle numerical issues. So I think it's just kind of stress testing a lot of the post-training infrastructure that we have had by turning up a lot of different things that could go wrong to the maximum. I think there's like the things that the open community struggles with is that VLM and Hugging Face use different kernels to do the actual internal computation of the model. So these kernels are the things that make things like VLM really fast. But these things, this then results in subtle numerical differences between the completions that you're generating from the model and then the log props that the thing that's doing the loss function actually generates. And if you look at the math of these RL algorithms, it's assumed that those are from the same distribution. So therefore, you have these like this is like what a big root cause of a lot of numerical problems. And then if you look at what we're doing, a lot of labs have done throughout the years, they do different fixes to change these numerics. I think machines had a famous blog, one of their first blog posts on like deterministic VLM to make it exactly deterministic and like that is really useful and people think it is key to their like tinker API and doing other sorts of RL things where you just have complete control over sources of non-determinism and that could just be like numerical lack of robustness in RL and if we talk about trade secrets or whatever in RL like there's also these discussions on if the open labs have worse algorithms than the closed labs and in reality it seems like most people are using something like an evolved version of GRPO which is a bit simpler than PPO. So some labs might be using a learned value function. It's not that important about the details but what happens is that each lab finds the set of tweaks that they need to get really stable RL performance. And in the RL literature historically there's a pretty low bar on the amount of changes that are needed to call it a quote unquote new algorithm but it's realistically like an implementation detail. So it's like a lot they everyone finds their stable configuration for operating and it's really dependent on the tool. So like, yes, you could say that they have a different algorithm, but it's also not really something that you could easily exfiltrate from a lab because it's dependent on many layers of the stack and maybe what chips they're operating on and all sorts of things. So it's just one of these things where training these models is complex and the kind of quick quips could never reflect that.

后训练的软件栈也非常新。我觉得大的进展是从 2024 年开始的，比如你同时训练模型和运行模型，而且它们必须以一定的节奏进行，而预训练方面，像谷歌 2017 年发布的 TensorFlow 中已经有了分布式预训练的种子。所以预训练的栈要成熟得多，而后训练这边需要的栈则不然。

The stack for post-training is also so new, software-wise. I feel like the big strides started happening in '24 around like, oh you do both you train the model but also you run the model at the same time and that they have to happen at a certain cadence versus pre-training the seeds of distributed pre-training like you had in TensorFlow which Google released in 2017, right? So there's a much more mature stack versus what you need on the post-training side.

好了，作为这次对话的最后一部分，你们描述的一切真的非常引人入胜且富有启发性，特别是它突显了系统的复杂性，比如多个阶段。我喜欢你刚才提到的，Nathan，几分钟前你说预训练是科学的，而后训练——这是我的说法，不是你的，但我的理解是——它更多的是修修补补，把东西拼凑在一起，希望它能工作，一方面真正深入理解这些模型如何工作，但另一方面，每次你打开报纸或上推特，每个人都在谈论 AGI，说我们快到了，它将改变一切。现实是努力让这些模型工作，尽管我们看到了所有令人难以置信的进步，但另一方面是这种讨论，两者之间存在一点认知失调。Nathan，与一些 AI 研究人员相比，你对 AI 进展的看法要温和得多。你最近写了一篇很棒的博客文章，叫做《曲线上的思考》。我很好奇你最新的想法，当然随时可以插话，但就你在那篇文章和之前一篇文章中描述的复杂性和复杂性税而言，考虑到你刚才描述的流程，人们开始理解其中涉及的纯粹复杂性。

All right, so maybe as a last part to this conversation, it's been really fascinating and illuminating, everything that you guys have described, because in particular it sort of highlights the complexity of the systems, like the multiple stages. And I love what you mentioned, Nathan, a few minutes ago when you said the pre-training is scientific and post-training, my words not yours but my interpretation of your words was like it's a lot of tinkering and putting things together in a way that you hope is going to work, and truly diving into how those models work on the one hand, but on the other hand, you know, each time you open a newspaper online or go on Twitter, everybody's talking about AGI and how we're almost there and how it's going to change everything. There's a little bit of a cognitive dissonance between the reality of trying to make those models work with all the unbelievable progress that we've seen of course, but like, you know, that on the one hand and the discourse on the other hand. Nathan, you've had a much more, I would say, tempered view of AI progress compared to some AI researchers. You had a great blog post very recently that you called "Thoughts on the Curve." And I'm curious what your latest thinking is, and look, obviously feel free to jump in any time, but in terms of what you described in that essay and a prior one as complexity and complexity tax, which again, in view of the pipeline you just described, one starts to understand the level of sheer complexity that's involved in all of this.

是的。所以最终我肯定把自己描述为轻度 AGI 信徒，我认为你必须这样，才能理解我们所处局势的规模和严重性。

Yeah. So ultimately I definitely describe myself as lightly AGI pilled and I think you have to be to appreciate the magnitude and gravity of the situation that we're in.

但我也认为，由于复杂性等因素，我非常不相信任何形式的奇点可能发生。一方面，我们谈到了所有这些改进模型的低垂果实，我对此并不怀疑。就像你在本播客和其他地方评论的那样，这些实验室仍然在许多方面看到改进模型的低垂果实。我不认为他们的方法感觉有很大不同；它们只是相对于我们所做的进行了精炼。但与此同时，随着工具和堆栈层级的增加，事情变得复杂，你必须构建一个产品来支撑它，就像要充分利用云服务，你需要使用 Claude Code 这样的神奇产品，以及相对于 GitHub Copilot 的提示工程，这是你为了获得 AGI 以及所有这些工具使用等需要正确掌握的事情之一。所以，随着任何系统变得更加复杂，变化的速度就会变慢。我认为任何科技公司都看到了这一点。然后现实地说，我们能够建设的基础设施数量将受到物理限制。所以这种信念同时给了我们这些新的数据中心，我个人希望还有新的发电能力，但有一个上限。为了让所有这些事情达到平台期，然后你增加 10 倍的算力，你得到一个大跳跃，你不能永远这样做。所以现实地说，在某个时候会有一些物理限制出现。但在复杂系统和低垂果实之间取得平衡的结果是，我认为这些研究人员会在多年内逐步改进，但永远不会以那种我们被吸引进去的加速井喷的方式。所以这有点像，我不知道，在某些方面感觉我是在鱼与熊掌兼得，但如果你看看其他类型的技术，这似乎是可能的结果。

But also I think that I'm very far from believing in any sort of singularity being possible due to these things like complexity. On one hand, we talked about all these things which are low-hanging fruit to improve the models, and I don't doubt that. I mean, as you commented on this pod and other places, at these labs they still see low-hanging fruit in improving the models in many ways. I don't think that their approach feels that different; they've just been refined relative to what we're doing. But at the same time, as things get complex with tools and adding more layers to the stack, and you have to build a product to scaffold it, it's like if the requirement to get the best out of cloud is to use Claude Code, which is some magic product, and prompting relative to GitHub Copilot, this is one thing that you're going to need to get right in order to get AGI along with all these tool uses and stuff. So it's like as any system gets more complex, the pace of change is slower. I think any tech company has seen this. And then realistically, there's going to be physical constraints on the amount of infrastructure that we could build. So this belief is simultaneously giving us these new data centers, and I personally think hopefully new power generation, but there's a cap. For having all these things plateauing, and then you 10x the compute and you get a big jump, you can't do this forever. So realistically, there's going to be some physical constraint that kicks in at some point. But balancing that with complex systems and the low-hanging fruit results in, I think, these researchers grinding out improvements for multiple years, but never in a way that results in this kind of accelerating well that we get drawn into. So it's kind of like, I don't know, some ways it feels like I'm having my cake and eating it too, but it seems like the likely outcome if you look at other types of technology.

是的。而那次对话是对 AI 2027 的特定反应，这是一个非常有趣的对话，其中，我让你总结一下前提，但简短的版本是人工智能某种程度上自我构建，因此加速。

Yeah. And that conversation was in a particular reaction to AI 2027, which is a really interesting conversation where the, I'll let you summarize the premise, but the short version is that AI sort of builds itself and therefore accelerates.

是的。我认为他们有这些里程碑，比如人工智能自动化研究工程，然后人工智能自动化人工智能研发，每一个都是性能的巨大飞跃。我认为更可能的是这种混乱的共同进化。他们当然应该因产生这种影响而受到赞扬，但即使是他们现在也说，哦，也许我们应该叫它 AI 2028 或 AI 2029。所以我认为这反映了这些真实的限制，但进步也将会是惊人的。通常这些模型的能力增长，但从事过这方面工作后，我认为在任何时候看到不连续性都是非常不可能的。这与我们是否会达到人们满意的 AGI 或超级智能的定义无关。我们会达到的。似乎不太可能那个时刻会是一个回顾性的练习，比如，哦，这些是重要的里程碑，这才是真正有效的。构建这个模型很大程度上是各种精炼的集合，以解锁下一阶段，这将是一条平滑的轨迹。无论它是在某个我们无法继续改进的点达到顶峰，还是永远加速，我不知道。如果人们希望有一天登录 Twitter 就看到 AGI，他们会失望的。这很混乱，从事其中很有趣，因为它混乱，而且带来很多满足感。

Yeah. And I think that they have these milestones like AI automates research engineering and then AI automates AI research and development, where it's like each of these are incredible jumps in performance. And I think what's more likely is this messy co-evolution. They deserve credit for getting this impact for sure, but even they are now like, oh maybe we should have called it AI 2028 or AI 2029. So I think that is the reflection of these real constraints, but the progress is also going to be incredible. Normally the growth in capability of these models, but having worked on one, I think it's very unlikely that we will see a discontinuity at any point. That has nothing to do with whether we'll get to a definition of AGI or superintelligence that people are happy with. We will get there. It seems unlikely that the moment is going to be a looking-back kind of exercise of like, oh these were the important milestones and this is what really worked. Building this model is so much a collection of refinements to unlock the next stage that it's going to be this smooth trajectory. Whether it hits at some point where we don't have more capacity to keep improving or whether it forever accelerates, I don't know. People are going to be disappointed if they want to see a moment where one day they log into Twitter and AGI is there. It's messy, and it's fun working on it because it's messy and it gives a lot of satisfaction.

所以回顾一下，你们俩都是对 AGI 说“是”，但对不连续性/奇点说“不”。第一，这样说公平吗？第二，如果这就是你们的意思，那么对于 AGI，你们是说当前的范式，基本上就是我们刚才在上一小时描述的预训练加强化学习，就能让我们达到那里。

So to play it back, you're both saying yes to AGI but no to discontinuity/singularity. And one, is it fair? And two, if that's what you're saying, then for AGI you're saying the current paradigm, basically what we just described in the last hour of pre-training plus RL, gets us there.

我认为 AGI 这个词实际上没什么用。我认为我的描述方式是，大型科技公司都集体意识到，这些语言模型加上支撑系统将释放绝对不可思议的价值。而且我有很高的概率，除非极端的地缘政治情况，大型科技公司会在两到五年内执行这一愿景，在我们物理能力限制和 LLM 能力范围内，构建出 95% 到 98% 的成果。我认为那将是极端的。到 2030 年由此带来的转变将在整个社会中非常强大。有很多长尾效应。在五年内，社会将对互联网、媒体和信息的含义进行大规模调整。这主要是我做这件事的原因。我认为争论它是否是 AGI 是次要的，重要的是这正在发生，我们希望人们研究和理解正在发生的事情。

I think the AGI word is actually pretty not useful. I think that how I describe it is that big tech has all collectively realized that these language models plus scaffolding is going to unlock absolutely incredible value. And I have very high probability, barring extreme geopolitical situations, that big tech executes on this vision across the two to five years to build 95 to 98% of the way there of what you can do with our physical power constraints and what an LLM's ability is. And I think that that will be extreme. The transformation from that by 2030 is going to be so powerful across society. There's a bunch of long tail. There's going to be mass societal readjustment to what the internet and media and information means within 5 years. And that's mostly why I do this. And I think whether debating whether or not it's AGI is kind of secondary to the fact that this is coming and we want people to study and understand what is happening.

对于最后一点，研究和准备意味着什么？你会建议人们做什么？尽管如果人们已经一路听到播客的这一点，他们已经做了很多工作。

And to that last point, what does that mean, study and prepare? What would you recommend people do? Although if people have made it all this way to this point of the podcast, they've already done a bunch of the work.

所以我认为，在世界上的计算机科学专业之外，人们对人工智能也有很大的兴趣，它正在为政策制定者提供信息。我认为信息的传播仍然需要很长时间，而且通常没有那么多人在参与这件事，纯粹是为了这种，你可以称之为对齐和担忧。有很多一般的噪音。我的意思是，我担心权力集中或各种事情。而我只是试图提升人们对人工智能的理解，以便他们能够参与进来，比如吸引听众，思考它如何影响他们的领域。另一部分，也许更积极的一点是，如果支撑系统真正将原始能力模型转化为具有实际意义的影响，那么这种支撑系统不仅仅是，哦，只有训练模型的实验室才能做到。能够为此做出贡献的人数，无论是具有技术专长的人还是非技术专长的人，都要大得多。如果支撑系统真正推动了能力，让我们实现这项不可思议的技术，那么能够为此做出贡献的人不仅仅是那些在前沿实验室工作的人。

So I think there's a lot of interest in AI outside of the CS majors of the world, where it's informing policy makers. I think it still takes a long time for information to diffuse, and there's often not that many people that are engaging in this that are doing it just purely for this kind of, you can call it alignment and concern. There's just a lot of general noise. And I mean, I worry about concentration of power or all sorts of many things. And it's just trying to upscale people into understanding AI so they can be engaged, like engage listeners and think about how it affects their domain. The other part, maybe on a more positive note, is that if the scaffolding is what really moves a lot from raw capability model to something that actually has meaningful impact, that scaffolding is not just like, oh only the labs of people trained models can do it. The number of people that can contribute to that, both in terms of people with technical expertise and people with non-technical expertise, it's much larger. If the scaffolding is what really moves capabilities, what gets us to this incredible technology being realized, then the number of people that can contribute to it is not just those who work at frontier labs.

有大量的技术工作要做，但一旦你开始将这项技术融入真实人们的生活，非技术工作也同样重要。一旦你开始从事高风险的医疗应用或其他高风险领域，那么大量人口都可以为改进这项技术做出贡献，让它为所有人服务。仅仅是基础模型，我觉得能够真正帮助这项技术为所有人服务的人数就很多。社会中的每个人都觉得自己可以做出贡献。

There's a tremendous amount of technical work to do, but also non-technical as soon as you start integrating this technology into the lives of real people. As soon as you start working on high-stakes medical applications or other high-stakes domains, then a large amount of the population can contribute to making this technology better and make it work for everyone. Just the base model, I feel like the number of people who can really help make this technology work for everyone is large. Everyone in society feels like they can contribute.

好的。这似乎是一个很好的结束点。非常感谢两位，不仅感谢这次对话，也感谢你们在开源前沿 AI 领域所做的一切工作，这非常必要且极其重要。非常感谢你们的时间和分享的想法。谢谢。

All right. Well, that feels like a wonderful place to leave it. Thank you so much both, not just for this conversation but for all the work that you're doing in open source frontier AI, which feels sorely needed and extremely important. So, really appreciate the time and all the thoughts. Thank you so much.

嗨，我是 Matt Turk。感谢收听本期 Mad Podcast。如果你喜欢这期节目，如果你还没有订阅，我们非常感激你能考虑订阅，或者在你观看或收听本期节目的任何平台上留下好评或评论。这真的有助于我们建设播客并邀请到优秀的嘉宾。谢谢，下期再见。

Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already, or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks, and see you at the next episode.