为什么 MCP 和工具调用已经过时：AI 的未来是直接与代码交互

我觉得 MCP 和工具调用很蠢。如果你面对的是复杂任务——越来越长的时域、越来越复杂的任务，不管是写代码还是别的什么——你总要与数据源打交道，对吧？你还要与安装在某种虚拟机上的东西交互。而我们在中间塞入 MCP，塞入工具调用——模型本来可以直接写代码与系统交互。我们开始看到，像 Llama 3 就经常这样做，前沿模型里也会看到——它们越来越不依赖那一套了。我们不再把 50 个工具塞进系统提示词。而是说：给你一台虚拟机，里面装好这些二进制文件，给你一个可以操作的代码库，给你一个文件夹，想的话可以写你的记忆。然后模型用代码来完成复杂任务。

I think MCP and tools are stupid. If you are looking at complex tasks — increasingly longer horizon, increasingly complex tasks, doesn't matter if it's coding or something else — you are going to be interacting with data sources, right? And you're going to be interacting with things that are installed on some form of a virtual machine. And we're putting MCP in between, putting tool calls in between, when the model can just write code and interact with the system. We're starting to see that — like Llama 3 does this a lot, and you'll see it in frontier models too — they're increasingly no longer relying on that. We're not going to stuff 50 tools into the system prompt anymore. Instead: here's a virtual machine with these binaries installed, this code base you can operate in, here's a folder where you can write your memory if you want. And the model is using code to do complex tasks.

在进入今天的正题之前，我想对听众们说几句。谢谢。如果没有你们选择点进来、收听我们的内容，我们不可能为您带来你们那么想要的 AI 工程、科学与娱乐内容。几乎每天都有赞助商找上我们。但幸运的是，有足够多的订阅者支持我们，让我们能在没有广告的情况下持续运转，而我们想保持这种状态。但我只想拜托大家一件事：你能做的最有力、完全免费的事情，就是点击订阅按钮。这也是我唯一会请求你们的事，它对我和我团队意义非凡，他们每周都非常努力地把这档节目带给你们。如果你们订阅了，我保证我们会一直努力把节目做得更好。好，我们开始吧。

Before we get into today's episode, I just have a small message for listeners. Thank you. We wouldn't be able to bring you the AI engineering, science, and entertainment content you clearly want if you didn't choose to click in and tune into our content. We've been approached by sponsors almost daily. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. I just have one favor to ask all of you. The single most powerful, completely free thing you can do is click that subscribe button. It's the only thing I'll ever ask of you, and it means everything to me and my team who work so hard to bring the show to you every week. If you do it, I promise you we'll never stop working to make the show even better. Now, let's get into it.

好，今天我们请到的是来自 PSA 的 Khan，还有 Ibu。欢迎。

All right, we're here in the studio with Khan from PSA, together with Ibu. Welcome.

谢谢，谢谢你们邀请我，很高兴来到这里。

Thanks. Thanks for having me, guys. Good to be here.

是啊，还在飞机上吧。你发短信跟我说‘嘿，我正要去旧金山’。我就问‘你现在在飞机上吧？’你知道的……

Yeah. Fresh on the plane. You texted me, you're like, "Hey, I'm on my way to SF." I was like, "You're on a plane right now, right?" You know...

我发完短信才意识到，很可能会倒时差倒得很厉害。今天会带来一些有趣的体验，但我们开始吧。

After I texted you, I realized I'd probably be coming in with major jet lag. Going to offer some fun experiences today, but let's do it.

我想对所有嘉宾说，其实你们不用准备太多。因为如果你真的每天都扑在这件事上，那么即使是你模糊记得的东西，对很多并不生活在你们这个圈子的听众来说也是新鲜的，对吧？十年前，你在 Slush 大会上做过一个关于 AI 民主化的演讲。而如今，你又开源了一个很厉害的新模型，我们稍后会聊到。但究竟是什么让你走上了 AI 民主化的道路呢？这些在你的 LinkedIn 上看不出来。

I mean, the thing I tell guests is they actually don't have to prepare that much, because if you're truly working on this every single day, even what you hazily remember is going to be new for a lot of the audience who don't live in your world every day, right? So, ten years ago you gave a talk at Slush about democratizing AI. And now here you are, open-sourcing an incredible new model that we're going to talk about. But what got you into democratizing AI? It's not obvious from your LinkedIn or anything.

完全不明显。说实话，我觉得我进入这个领域的原因并不明显。我得感谢 Andrej Karpathy。2015 年，他写了一篇文章，叫《循环神经网络的惊人有效性》。

No, it's not obvious at all. Actually, I don't think it's obvious how I got into this space. I owe getting into this space to Andrej Karpathy. In 2015, he wrote an article called 'The Unreasonable Effectiveness of Recurrent Neural Networks.'

嗯。

Yep.

我读了那篇文章，然后让当时的创业公司一夜之间彻底转向，去做 RNN，后来又做 LSTM 和 Transformer 模型，目标是让模型能写代码。如果你打开那篇文章往下翻，就能看到，这实际上是后来语言模型的雏形——至少是能真正预测字母的字符级语言模型。他给了一个例子，一个 Paul Graham 生成器，你读起来好像有点道理，但又不是真的通顺。再往下一点还有一个代码示例。

I read that article, and I pivoted my startup at the time overnight to working on RNNs, and later LSTMs and Transformer models, to be able to write code. If you go to that article and scroll down, you can start to see this was the precursor to what ended up becoming language models — at least character-level language models that were starting to actually predict letters. He has an example out here — a little Paul Graham generator — and you can kind of read it; the text kind of makes sense, but it doesn't. And there's an example of code a little further down.

上面写着莎士比亚。

It says Shakespeare.

不知道为什么，我读了这篇文章之后，就一头扎进去，拼命学习所有关于 RNN 和 LSTM 的知识。那时 Transformer 论文还没出来。我当时产生了一个完全不理性的信念：神经网络应该能泛化到任何事物，语言也应该能泛化到很多属于智能的东西，包括写代码的能力。于是我创办了 Source，一家完全开源的初创公司，想做的事情当时叫‘代码上的机器学习’——就是基于代码的语言模型。我们花了大概四五年时间，一直到 2019 年底。现在听起来很酷，但当时没人在乎。真的没人在乎。我们就像在黑暗中摸索。一路上我们试过很多东西——我们尝试把卷积神经网络用到代码结构上。注意力机制出来时，我们把它用到 LSTM 上。然后 Transformer 论文出现了。当时一点都不明显。我们整个旅程中错过的核心，是我们其实走对了路，但应该继续 Scaling 下去。今天大家都觉得缩放定律和 Scaling 是最显然不过的东西。但我花了自己生命中四五年去做代码语言模型，那时一点也不显然。我非常尊重 Google、OpenAI 以及其他一些人，他们带着那种信念继续推进。我们最后失败了，那大概是我职业生涯最大的一次失败。

And for some reason, I read this and I went down the rabbit hole of learning everything I could about RNNs and LSTMs. This was pre-Transformer paper. And I had built a completely unreasonable belief that neural nets should be able to generalize to anything and everything, and that language should be able to generalize to a lot of things that are intelligence, including the ability to write code. So I started building Source, which was a fully open-source company trying to build what we used to call 'machine learning on code' — language models on code. We spent about four or five years on this, until the end of 2019. And that sounds really cool today, but back then no one cared. No one cared. We were in the dark. We did things along the way — we tried applying convolutional neural networks to the structure of code. When attention came out, we were applying it to LSTMs. Then the Transformer paper came out. And it wasn't obvious. What we missed throughout that entire journey was that we were on the right track, but we should have just kept scaling up. And today, to all of us, the scaling laws and scaling up seem like the most obvious thing. But having spent four or five years of my life working on language models on code, it wasn't obvious. I have a lot of respect for the folks at Google and OpenAI and others who took that confidence and kept going. We ultimately failed at the time, and it was like the biggest failure of my career.

是啊，你烧掉了投资人 1200 万美元，在当时算是很多钱了。

Right. You blew $12 million of investors' money, which was a lot back then.

没错。

Yep.

到今天也很多。你带着 40 人的团队，花了几年时间，就死磕这个问题。

Still a lot. And you spent years with a group of 40 people just obsessing over this problem.

后来人生方向变了，家庭成了重心。我埋头过自己的日子，说实话，接下来两年几乎没怎么看语言模型——事后看是个大错，因为接下来几年特别精彩。然后 ChatGPT 出来了，就像一种平反。人们开始给我发短信，翻出我以前的旧工作文档和演讲。而在那整个旅程中，我们当时有一个非常坚定的观点：当你构建更强大的智能时，它应当是开放的——开源。但当我们创立 Poolside 时，情况就不是这样了。我想非常坦诚地说。我们创立 Poolside 时，有两个前提。第一：这项技术的能力不会停止复利式增长——在今天看来对多数人显而易见。但三年多前我们刚开始时，大多数人还在争论这些是不是随机鹦鹉。第二：强化学习会成为 LLM 能力最大的推动力。今天这很明显。但三年前，OpenAI、Google、Anthropic 或其他机构都没有这样的观点或方向。所以有些人有点看不起我们，觉得这真的能成吗？于是我们就只在解决问题。我们再也没有认真想过开源的事。

And life took a different turn, and family kind of became a focus. I kind of kept my head down, and frankly didn't really look at language models for the following two years — kind of a big mistake, considering the following years turned out to be really interesting. Then ChatGPT came out, and it was kind of a vindication. People started texting me, finding my old work decks and old talks. And throughout that whole journey, we had a strong point of view at the time that as you're building more capable intelligence, it should be open — open source. But when we started Poolside, that actually wasn't the case. I want to be very open about that. When we started Poolside, there was a premise with two things. One: this technology is not going to stop compounding in capabilities — I think that's obvious to most people today. But three-plus years ago when we started, most people were still arguing over whether these were stochastic parrots or not. The second: reinforcement learning was going to be the biggest driver for LLM capabilities. Today it's very obvious. Three years ago, it was not an opinion held or a direction held at OpenAI, or Google, or Anthropic, or others. So people kind of looked down on us a little bit, like, "Is this really going to work?" And so we just started working the problem. We never really thought about open source again.

我们只是埋头苦干，从零开始建立自己的知识体系，对吧？我们不是从某个现成的实验室里出来的。我们拿起论文，开始写代码，自己摸索。直到今年年初，我和联合创始人 Jason 才重新拾起开源这个讨论。如果你回头看看我们网站早期的一些内容，非常直白：我们要实现 AGI，我们要支持一个丰饶的世界，我们要成为第一家做到这一点的公司。但是我们从今年年初开始讨论这个问题，因为情况变得很明显，世界正朝着一个让我们有点不安的方向发展。这不是一夜之间发生的，而是一点一点累积的。我们看到这些，心想：好吧，世界正在走向某条路。在整个过程中，我常用一个比喻。如果回到 2015 或 2016 年那时候，我们正在做这件事。我从书架上拿下一本科幻小说。我读的是关于 2035 年实现 AGI 的故事，之后的故事会延续几十年，对吧？你会看到第一章里每个人都在试图弄清楚一切，然后会有一章聊天机器人出现，再然后会到世界站在十字路口的那一章。在其中一个分支里，三四家或一小撮公司将创造未来所有的智能。当我想到那个故事时，感觉这是一本科幻反乌托邦小说，而不是乌托邦科幻小说。而实际上我是个乌托邦科幻迷。所以我们退后一步，说：嘿，我们能在这里扮演一个角色吗？我们现在这样做很容易，因为我们处于前沿，但还没有被卷得太深——我不认为我们以后还能改变主意。我不是说等到有太多资本投入、你有太多预期的时候。我们是个小团队，只是不断改进。所以我们知道我们现在可以做出这个决定，但随着我们越来越接近前沿、赶上其他人，做决定会难得多。我们做了很多灵魂拷问，也谈了很多次，最终说：不，这说得通。即使还有巨大的未解问题，比如你如何围绕开源基础模型建立商业模式？这是一个我们还没有完全答案的大开放性问题。到什么时候你不再想发布开源模型，因为模型滥用有真实潜在风险？政府会如何回应开源？但我认为所有这一切只归结为一点——我就此打住——我宁愿生活在一个有一百家基础模型公司的世界，也不愿生活在一个只有五家的世界，哪怕我是那五家之一。而让 H 存在的最微小、最有意义的贡献，就是现在开放我们的研究，开放我们的权重，然后在这个过程中想办法做得更多。

We just kept our heads down and built our knowledge and understanding from scratch. We didn't roll out of an existing lab. We picked up the papers and started writing code and figuring things out. It wasn't until the beginning of this year that my co-founder Jason and I picked up the open source conversation again. If you go back to some of the early things on our website, they were very straightforward: we want to get to AGI, we want to support a world of abundance, and we want to be the first company that gets there. We started talking at the beginning of this year because it became obvious that the world was going in a direction that was starting to pick at us a little bit. It didn't happen overnight; it was a little bit at a time. We saw this and thought, okay, the world is going down a path. Throughout this journey, I used an analogy. Back in those days, in 2015 or 2016, we were working on this, and I picked up a sci-fi book off the shelf. I was reading about 2035, AGI is achieved, and the story would be over the following decades. You'd have that first chapter where everyone's trying to figure things out, then the chapter of chat coming out, and then the chapter where the world was at a fork in the road. In one branch, three or four or a handful of companies were going to create all of intelligence moving forward. Reading that story felt like a dystopian sci-fi book, not a utopian one. And the reality is I'm a utopian sci-fi guy. So we took a step back and said, hey, can we play a role here? Now it was easy for us to do so because we were at the frontier but not too deep into it — I don't think we could have changed our mind later. I don't mean when there's too much capital involved, too many expectations you've built up. We're a small team, just improving and improving. So we knew we could make that decision now, but it would be a lot harder as we got closer to the frontier and caught up to others. We did a lot of soul-searching and had a lot of conversations and said, no, this makes sense. Even if there are big unanswered questions: how do you build a business model for open source foundation models? That's a big open-ended question we don't fully have the answer to yet. At what point do you no longer want to release open source models because misuse of models has real potential risks? How is the government going to respond to open source? But I think it all comes down to one thing — and I'll stop the monologue — I'd rather live in a world that has a hundred foundation model companies than a world that has five, even if I were one of the five. And the smallest, most meaningful contribution we can make for H to exist is to open up our research and open up our weights right now, and figure out along the way how we can do more.

是的，我觉得如果说过去三年有什么变化的话，这个说法越来越成立。你是人们现在所说的那批 Neo-labs 之一。而且我们录制这期节目那天，Thinky 发布了他们的新模型，你在他们发布的一些基准测试上超过了他们，对吧？他们就是还没达到。所以这说明，实际上这件事确实容得下多个玩家。你看到的未来正在变大一点点——也许更像 20 个而不是 100 个——但你是那 20 个之一。

Yeah, I think if anything, over the past three years that has become a bit more true. You are one of a cohort of Neo-labs that people are now calling that. And we're doing this on the day that Thinky launched their new model, and you are outperforming them on some benchmarks they released, right? They just don't have it yet. So it goes to show that I think there actually is room for multiple players. You're seeing a little bit more of the future — maybe more like 20 not 100 — but you are one of the 20.

我真的希望如此。对，我觉得我们是。我为他们的发布感到兴奋，也为所有人的发布感到兴奋。因为归根结底，选择和竞争都会推动进步走向正确方向。但事实是，我们都在构建模型，而虽然我们都实际上从同一个数据之井里取水，但我们在模型中引入了非常不同的行为和偏见。有些是有意的偏见，有些完全是意外。如果我们塑造一个生态，让开源模型成为 token 经济的一部分——我认为这已经毫无疑问了——那么我们就希望生活在一个公司、国家和个人都能选择并说“嘿，我最认同这个提供商，在这类事情上最信任它”的世界。

I really hope so. Right. I think we are. I'm excited about their release, and I'm excited about everyone releasing. Ultimately, choice and competition are both going to drive progress in the right direction. But the fact is, we create models, and while we all drink out of the same well of data effectively, we do introduce very different behaviors and biases in our models. Some are intended biases, some are completely accidental. And if we shape an ecosystem where open models are going to be part of the token economy — I don't think there's any question about it anymore — then we want to be able to live in a world where companies, countries, and people can choose and say, hey, I am most aligned with this provider and trust it most for these kinds of things.

是的。我觉得不只是 20 个 Neo-labs 之一。直到最近，大部分开源创新都来自中国实验室，对吧？所以西方有 DeepSeek。今天是这样吗？好吧，也许还有 Thinking Machines 的 Reflection，但数量不多。有个事情是，你们最初是在法国、欧洲起步，但现在很大程度上在采取美国立场。而且不仅如此，关键是我们看到的中国模型并不是超级开放研究。你们发布的工作，我认为是最好的之一。所以每隔几个月，你们不仅带来前沿模型，还会附上一份拆解博客、论文、技术报告，说明构建前沿智能需要的一切。你们也在填补这个空白——不仅是开源权重，不仅是西方面孔，还有相当开放的研究。

Yeah. I think more than just one of the 20 Neo-labs. Up until recently, most open source innovation was coming from the Chinese labs, right? So there's the DeepSeek of the West. Is that today? Okay, maybe it's Thinking Machines' Reflection, but there aren't many. One of the things is you guys started sort of in France, Europe, but very much now you're taking that American standpoint. And more than just that, the point is the Chinese models we see are not super open research. The work you put out is, I think, some of the best. So every few months you get not only frontier models but also a breakdown blog, paper, technical report with everything needed to build frontier intelligence. You're filling that gap too — not just open weight, not just Western, but also pretty open research.

不，我很感激。听着，我认为这实际上是最有意义的贡献。权重是二进制的——我们就叫它本来的样子。是的，我们可以修改它们、改变它们，但把权重给别人，并不能让他们最终复现你所做的事情，对吧？所以现在发布数据集有挑战，发布某些东西也有挑战，但能够分享你的研究——我们是怎么做到的？我们花了数万次算力实验学到的经验是什么？我深以为然。不过我要更正一点：Vivu 和我——我会这么说是因为这几年来这件事一直在困扰我们——我们实际上从第一天起就是一家美国公司。

No, I appreciate it. Look, I think it's actually the most meaningful contribution. Weights are a binary — let's call them what they are. Yes, we can modify them and change them, but giving someone the weights does not allow them ultimately to recreate what you're doing, right? So now there are challenges around releasing datasets, challenges around releasing certain things, but being able to share your research — how did we do it? What are the lessons we learned that we spent tens of thousands of experiments of compute on? I think very much so. One correction though: Vivu and I — I say this because it's been haunting us for quite a few years — we actually from day zero were an American company.

是的，他们搬到了法国。

Yeah. They moved to France.

所以这个故事，一劳永逸地说清楚，我们是一家美国公司。我们一直是一家美国公司。早期我们做了一个非常清醒的决定：我们不会在湾区招任何研究员。我们要在世界其他地方寻找人才——从美国中部、西雅图，到塞尔维亚，再到台湾、新加坡和其他地方。因为我们当时就认为这会变成一场人才争夺战，而且我觉得这几年确实如此。

So the story, once and for all, is we're an American company. We have always been an American company. Early on, we made a very conscious decision: we're not going to hire any researchers in the Bay Area. We're going to look for talent everywhere else in the world — from middle America, Seattle, to Serbia, and to Taiwan, Singapore, and other places. It was because we took a view that this was going to become a talent war, and I think it has over the years.

三年前，这一点还不完全明显。今天我觉得已经非常明显了。我们还意识到，世界上最有能力、最具创新想法的人不会主动来这里。于是我们决定创建一家完全远程的公司。我们后来在巴黎、伦敦等地开了办公室。团队有很多成员在美国，也有很多在美国以外，但我们一直持这样的观点：我们是一家美国公司，但要想让最优秀的人才与我们合作，就必须有全球视野。如今公司发展起来了，我们在硅谷也有同事。但我认为，有一件事一开始拖慢了我们的速度，现在却加快了我们的速度。这就是为什么你会看到我们的模型在进步，发布节奏在加快——因为我们不是从现有实验室里走出来的。我们当时并没有这里那么多自由流通的信息。我们只是抱着一种心态：好，那咱们就去解决这个问题，去读那几篇已经发表的论文，然后自己把这事儿搞清楚。正因为如此，这些年来我们在模型训练上犯过一些很可笑的大错，尤其是头 12 个月。有几个错误到现在还让我心有余悸。我们以后再谈。但这种经历给团队带来了韧性和坚持。这些年来几乎没有人离开。这让我们相信：我们可以做到。当我们第一次完全从零开始编写训练代码库时，它不是任何开源项目的分支，而是“好，我们从零开始建一个”。我记得有一次，我们花了三周解决一个优化器的 bug。训练就是无法稳定下来。我们为此痴迷，甚至怀疑是不是自己错了，是不是应该直接 fork 一个现成仓库。但等到我们真正解决它时——当时公司只有五个人——我们心想：“只要愿意努力，我们也能做成事。”这种强烈的工程导向文化帮助我们走到了今天。关于开源、人才这些事，我觉得我们只是从不同的起点做了不同的选择。我认为我们很幸运——我确实想用‘幸运’这个词——但同样也离不开团队的大量努力，而现在这些开始体现在成果上了。

Now three years ago, that wasn't fully obvious yet. I think today it very much is. We also realized that some of the world's most capable people with the most interesting innovative ideas were not just going to be here. So it led us to create a fully remote company. We ended up opening an office in Paris and London and other places. We have a lot of the team in the US and a lot outside, but we always took this view: we're an American company, but if we want the best of the best to work with us, we need to take a global view. We do also have people here in Silicon Valley now that the company has grown, but one of the things that slowed us down at the beginning has sped us up now. It's why you're seeing the progress on our models and the cadence with which we release — because we didn't come out of an existing lab. We didn't have a lot of the information freely flowing around here at the time. We just took the point of view: okay, let's work the problem, let's read the few papers that are out there, and let's figure this stuff out. We made some hilarious mistakes in model training because of that, especially in the first 12 months. There are a few that still haunt me and scare me; we can talk about them later. But it created a resiliency and persistency in the team. Extremely few people have left us over the years. It told us, okay, we can do this. When we first wrote our first training codebase completely from scratch, it wasn't a fork of any open source. It was like, "Okay, let's build it from scratch." I remember we had one moment where we spent three weeks working out an optimizer bug. Training just couldn't get stable. We obsessed over it and thought maybe we were wrong, maybe we should have just forked a repo. But then when we solved it — at the time we were five people in the company — we thought, "Oh, we can do things if we're just willing to work hard." That culture, with a very strong engineering bias, has helped us get to where we are. There's this notion of open source and talent and these things; I think we just took different decisions from a different starting point. I think we are lucky — I definitely want to call it lucky — but also a lot of hard work of the team, and now that's starting to show up in results.

既然我们大概不会再聊到这里了，不过这是一个很有意思的招聘挑战，如果有人知道答案的话。那个 bug 到底是什么？我们是不会公布答案的。

Since we probably won't revisit this again, this is a fun recruiting challenge if someone knows the answer. What was the bug? And we won't tell the solution.

这个 bug——你这是要考验我的记忆啊。不过我觉得我还记得。如果用 Adam 作为优化器，分母里有一个 epsilon。我记得那时候，你看了早期的 LLaMA 论文之类的。大家都在把 epsilon 调得相当高，加了 1e-4 还是什么，反正就是很高的值。如果你想一想训练过程，这其实有点奇怪，也有点反直觉：我们等于在分母上加了一个随机数，给优化器注入噪声，而且是在小数点后面。具体那个 bug 我记不清了，但我记得的是，一旦我们解决了它，就不再需要像 LLaMA 论文里那样把 epsilon 调得那么高。那是那种根本性的时刻：我们之前相信了一篇论文，觉得‘哦不，就必须这么做，epsilon 必须这么大’。但凭直觉这完全说不通：为什么一定要调这么高呢？如果你只是为了避免除以零，为什么不能设一个极小的值？那种时刻会让你意识到，自己从零开始搞明白事情能建立更好的直觉，因为在做模型这件事上，你会很快发现，你一开始的直觉会被无情地打脸。这真的是一门实验科学，那些显而易见的东西，你很快就会发现自己错了——幸运的话你能搞清楚为什么，有时候你甚至搞不明白。嗯，就是这样。

So the bug — you're going to test my memory here — but I think I can recall. If you take Adam as an optimizer, you have an epsilon, right in the denominator. At that time, if I recall, you looked at the early LLaMA papers and things like that. People were juicing epsilon quite a bit; they were adding, I don't know, 1e-4 or whatever, a high value for epsilon. If you think about this during training, it's kind of weird and counterintuitive that we're adding noise to our optimizer by effectively adding a random number in the denominator, right behind the decimal point. I don't recall the exact bug, but what I remember is once we solved it, we no longer had to juice epsilon as much as was happening in the LLaMA paper and other places. It was one of those fundamental moments where we had trusted a paper that was out there, and we thought, oh no, it has to be this way, it has to have this high value of epsilon. But it made no sense to us intuitively: why do you have to have it so high? If you're just trying to avoid division by zero, why can't the value be extremely small? That was one of those moments where you realize that finding things out from scratch yourself builds a better intuition, because the one thing you learn very quickly in model building is that your initial intuitions are going to get beaten up so hard. It's such an experimental science that the things that seem obvious, you very quickly learn you were wrong — and hopefully you figure out why, and sometimes you don't even. Yeah.

嗯，所以你知道，你发布新模型的时候，Vivoo 真的特别兴奋。我是说，所有人都很兴奋，但 Vivu 带头在我们论文小组里专门研究了它，显然你们也看到了。或许可以聊聊你们从中得到的教训，能透露多少算多少。我们可以从模型工厂谈起，你觉得从哪儿开始都可以。

Um yeah, so you know, one of the reasons that you — when you released your new models, Vivoo got really excited. I mean, everyone got really excited, but Vivu led our paper club on it, and you guys saw it obviously. Maybe talk through some lessons learned in that, whatever you can disclose. We can focus on model factory stuff, whatever you think is a good starting point.

我想说，我们从公司成立之初就认为，做模型最终 90% 是工程。我相信业内大家都知道这一点，因为如果你看看每个研究人员把时间花在哪里，他们花在写代码、看数据，还有写代码上。所以我们当时就觉得，三年前这个领域的状况就是：用 bash 脚本、Slurm，以及一团乱麻的训练代码库，还有东拼西凑的数据管道。我们看着这些说，模型构建归根结底是一个流程。从原始数据开始——预训练的原材料，比如整个互联网——你要做大量过滤、清洗、变换和分析。如今这比三年前要复杂得多。然后你训练一个模型，这实际上是一个大规模分布式系统的问题，要跨越各种硬件；硬件可靠性今天已经好多了，但当年极其不稳定。而且每更新一代，我们都会遇到全新的挑战。接着进入下一个阶段——当时还没有中期训练，但你会做后训练，然后是强化学习。所以我们就想，这看起来像是一个工业化的流程，一个端到端的流程，每一步都有它相应的机器。如果你有大数据管道，有对互联网的爬取和摄入，有大规模分布式训练，接下来你还要面对可靠性问题。

So I would say that our view from very early on in the company was that model building is ultimately 90% engineering. And I think we all know it in the industry, because if you look at where every researcher is spending their time, they're spending it writing code, looking at data, and writing code. So we kind of said, okay, the state of the moment three years ago was bash scripts, Slurm, and spaghetti codebases for training, and data pipelines that were patched together. We looked at this and said, well, ultimately model building is a process. You go from raw data — pre-training raw material, the web, etc. You do a whole bunch of filtering, cleaning, transformations, analyzing. These days that's far more complex than it was three years ago. Then you train a model, which is effectively a large distributed systems problem across hardware that has become a lot more reliable but was extremely flaky back then. Now with every new generation we get new sets of challenges. Then you go into the next stages — there was no mid-training back then — but you go into post-training and then reinforcement learning. So we looked at this and said, this looks like an industrialized process, an end-to-end process, where every single part has its machinery. If it's your big data pipelines, if it's your crawling ingestion of the web, if it's your large-scale distributed training, and then you have your reliability.

我们说，为什么不从零开始就把我们认识的一些世界上最聪明的分布式系统工程师纳入研究过程，而不是事后再补上，而是真正从一开始呢。这就成了我们的模型工厂。我们的模型工厂最开始只有少量组件，今天已经发展成数千个组件。我总爱把它比作：如果你是富士康最早期的人，之后跟了它十年，你就有能力重建富士康，因为你看到了构建那个系统背后的每一个决策和全部复杂性。如果今天你我走进一座富士康，门都没有，

And we said, why don't we take some of the world's smartest distributed systems engineers that we knew and make them part of the research process from day zero, not retrofitting it later on, but really from the beginning. That became our model factory. Our model factory started with a handful of components. Today it's thousands of components. I like to equate it to this: if you were someone at the very early days of Foxconn, and you had been there for the following decade, you would be able to rebuild Foxconn because you saw every decision that led to building that system and all the complexity. If you and I walk into a Foxconn today, no chance,

对吧？

right?

因为我们没有那样的决策谱系和历史。所以我们从一开始就在团队的配合下构建，这个团队真正明白，我们优化的指标是一个想法从研究人员到值得信赖的实验结果的速度，然后再成为下一次模型训练的一部分。因为最终这是一门实验科学，在最开始还没那么复杂的时候，你可以用打补丁的方式绕过去。但现在，任何基础模型公司，你都在运行——我的意思是，我们是小团队，对吧？我们不到 70 名研究员，另有 35 名工程师，而我们每月运行的实验——我没看最新数字——但远远超过 1 万次，可能是 1 万到 2 万次。如果你看到那个规模，每一次模型运行——最终，你必须把它当作基础设施问题来信任。这些年来我们非常擅长这一点，就是通过持续工作、改进、并痴迷于那些端到端决策。所以现在这意味着什么：看看我们推出的 Laguna XS2，从开始预训练到发布只用了 5 周。我们今天要讨论的模型，从开始预训练到发布用了 8 周。我们昨天已经开始了下一个模型，因为我们刚刚完成了这个即将发布模型所需的后训练——你知道的，下周或到节目播出时——我们把那部分算力转移到了更大的、正在训练的 Laguna M 模型上。所以模型应该是某人流程的产物，它本身不应该真的是一件事。我们像看待 SpaceX 的工厂一样：是的，第一枚火箭真的很难造，但更难的挑战是建造工厂。现在火箭陆续出厂，没人再去想下一次发射了。它只是另一次发射——另一枚火箭出厂。这就是我们在模型构建上想做的事。另外一件不是从零开始就计划好的事——虽然我们心里隐隐觉得总有一天会发生——就是当你构建了一个真正优秀的端到端模型工厂，拥有真正好的 API 和工程系统时，它最适合做什么？它最适合智能体，因为智能体现在开始在我们的模型工厂里接管越来越多的工作。

Because we don't have the lineage and history of decisions that led to that. So from the beginning, we built with a team that really understood that the metric we are optimizing for is the speed of an idea from a researcher to an experimental result we can trust, and then that result becomes part of the next model training. And because it's such an experimental science, in the beginning when it wasn't that complex, you could kind of patch your way around it. But now, at any foundation model company, you're running—I mean, we're a small team, right? We have less than 70 researchers and another 35 engineers, and we're running—well, I haven't checked the latest count—but far more than 10,000, maybe 10 to 20,000 experiments a month. And if you look at that scale, every model run—ultimately, you need to be able to trust it as an infrastructure problem. Over the years, we've gotten really good at that, just by working on it, improving it, and obsessing over those end-to-end decisions. So what that means now is: look at Laguna XS2, which we launched—it was 5 weeks from the beginning of pre-training to launch. The model we're going to talk about today was 8 weeks from start of pre-training to launch. We started the next model literally yesterday, because we now finished the post-training required for the model we're launching—you know, next week or by the time this comes out today—and we moved that compute to the much larger Laguna M model that we're now training. So the model should be an artifact of someone's process; it shouldn't really be a thing in itself. We treat this the way you'd look at a SpaceX factory: yes, the first rocket was really hard to build, but the much harder challenge was building the factory. Now they're rolling off, and no one is really thinking about the next launch anymore. It's just another launch—another rocket comes off. That's what we're trying to do with model building. And what was not planned from day zero—though it was kind of in the back of our mind that this would happen one day—is that when you build a really good end-to-end model factory with really good APIs and really good engineering systems, what is it perfect for? It's perfect for agents, because agents are now starting to take over more and more work in our model factory.

对。

Yeah.

所以当我在办公区走动时——我们每月有线下聚会——我会走到人们屏幕后面，和研究员聊天。默认状态是，屏幕上跑着各种智能体，它们在写代码、提交任务、评估模型运行返回的结果、做修改。我们仍然坐在驾驶座上——我们仍然在提出想法、帮忙调试——但越来越多，而且这在数据管线方面现在非常明显，无论是预训练还是后训练，还有合成数据管线。也开始出现在架构一侧。你开始看到 RSI（递归自我改进）会是什么样子的苗头。坦白说，所以当你问我们的模型时，我谈的是模型工厂。我最酷的例子永远是：当我们启动一次新的运行，无论是大的预训练运行，还是我们为发布前做的十个后训练版本之一，或者许多实验，在任何给定时刻，某人前一天做出了改动并拿到了实验结果，这些改动就会进入那次运行。没有 90 天前的截止线。真的就是从那一刻开始，因为我们现在已经足够信任机器。然后你还必须在可靠性上投入。所以关于 Laguna S，我最喜欢的指标之一是没有待命告警事件——完全是零。实际上，就我记忆所及，今年一整年我们都没有一次有意义的待命告警，比如需要半夜爬起来。有一个例外：通常在启动一次新模型运行的头 6 个小时，会出点问题，因为配置设错了或出了小差错。所以通常会有一点点人工干预，但那总是在待命时段内，而不是真正的待命告警。我认为这现在开始产生复利了。所以我们现在发布的这个模型，我很喜欢，它很棒，但我们已经在做下一个了。我觉得本来就该是这样。

So when I walk around—we have monthly on-sites, and I kind of walk behind people's screens and talk to our researchers. The default is all these different agents running on their screens that are writing the code, launching the jobs, evaluating the results coming back from the model runs, and making the changes. We're still in the driver's seat—we're still coming up with ideas, still helping with debugging—but more and more, and this is right now very profound on the data side of our pipelines, both pre-training and post-training, and the synthetic data pipelines. It's starting to become more on the architecture side as well. You're starting to see those inklings of what RSI is going to look like. And that's frankly what I talk about when you ask about our models—I talk about the model factory. My coolest example is always that when we kick off a new run, whether it's a big pre-training run or one of the ten post-training versions we do for pre-release or many experiments, at any given moment, the changes somebody made the day before, with experimental results, make it into that run. There's no cut-off 90 days before. It's literally from that moment, because we can now trust the machine enough. And you also have to invest in reliability. So one of my favorite metrics about Laguna S is that there were no on-call events—completely zero. Actually, we haven't had a meaningful on-call event—like waking someone up—as far as I recall, this entire year. There is one asterisk: usually in the first six hours of launching a new model run, something breaks because you set a config wrong or made a small mistake. So there's a little bit of intervention, but that's always within on-call periods, not on-call. And I think that's starting to compound now. So the model we're releasing now—I love it, it's amazing—but we're already on to the next one. And I think that's the way it should be.

我还想再指出来——补个背景，这大概是一个月前。我们其实是在技术报告里发现的。我们当时进来，说好，新模型发布了。还没听说过他们做过这种事。我们当时很像是：啊，好吧，你看，跟 Kimi、DeepSeek 之类的差不多——小号模型，Gemma 级别。哦，这篇论文讲怎么构建的，很有意思。然后我们翻到了这一页，对吧？技术报告的第二页写着：“这个过程让我们能够把小型模型从零做到交付，只用了 5 周，应用了那些经验教训。” 我就想，哦，这篇论文不是关于基准测试和训练了多少 token 的。对于那些想深入了解、而我们不会在播客里讨论的内容，这里全写了，对吧？从智能体可以用来与训练代码交互的定制软件，到预训练数据。

I think I also just want to point out—for context, this was like a month ago. We found it, actually, in the tech report. So we just came in with, okay, new models dropped. Haven't heard about it for doing this. We were very much like, ah, okay, look, it's on par with Kimi, DeepSeek, whatnot—the small ones, Gemma level. Oh, it's a very cool paper on what goes into building. And then we hit this page, right? Literally page two of the tech report says: 'This process allowed us to build the small model from scratch to delivery within 5 weeks, applying the lessons.' And I'm like, oh, this paper is not about benchmarks and how many tokens it was trained on. For people who want to dive more into what we're not going to discuss on the podcast, it's all laid out here, right? From custom software that agents can use to interface with training code, pre-training data.

对，论文里的时间线。

Yeah. The paper clock.

是的是的，那类东西。你知道的，论文在这儿，去读吧。不过我热爱原则。我觉得那是一个很好的切入点，可以讲一些故事。也许我们可以一条一条地过这些原则。我只提一个——Dexter 刚被 Prefect 收购了。对，挺有意思的。不过是的，我对 Dexter 其实非常熟悉。呃，只要哪条能触发什么故事就行。

Yeah, yeah. All that stuff. You know, read the paper here. But I love principles. I think that's a good starting off point for maybe telling some stories. Maybe we can go one by one past the principles. I'll just call out—Dexter just called 'Bought by Prefect.' Yeah, it's kind of fun. But yes, I'm actually very familiar with Dexter. Uh, just anything where they trigger some kind of story.

嗯，我会说“实验即代码”是显而易见的，但我最喜欢的事情之一——我不知道它在报告里哪里——早期，而且我现在仍然这么认为，很多基础模型公司会准备好训练数据集，打包好，复制到训练集群，分发到所有节点上，然后才开始训练。我们三年前看到这种情况，觉得这毫无道理。因为每当你要重新物化数据集、要做改动、要修复什么问题，你都要花大量时间重新打包：分词、重新打包、搬到集群、再分发到各个节点。集群越大，你甚至要开始用那种花哨的、类似 torrent 的算法来分发数据。那为什么不把数据流式地送进训练呢？这在基础领域里是非常常见的……

Well, I would say experiments as code is obvious, but one of my favorite things — I don't know where it is in here — early on, and I still think this is the case, a lot of foundation model companies prepare their training datasets, package them up, copy them over to a training cluster, distribute across all of the nodes, and then training starts. We looked at this three years ago and thought, that makes no sense. You lose so much time because the moment you have to rematerialize a dataset, make a change, fix something, you have all this time of repackaging it: tokenizing, repacking, moving it over to a cluster, then distributing it across the nodes. The bigger your clusters are, you start using fancy torrent-like algorithms to distribute your data. So why aren't we streaming data into training? It's something very common in just basic...

即时（Just in time）。

Just in time.

即时（Just in time）——这是一个很好的计算机科学原则。那是首批解锁我们模型工厂（model factory）的事情之一。一旦你开始思考，一次训练任务——不管是大规模主训练（hero run）还是小的后训练实验——每秒消耗一定数量的 token。从数据搬运的角度看，这个量其实不大。所以我们就说：我们有自己的训练集群，现在我们还有一套类似 AWS 的内部设施，可以构建很厉害的大数据管道。我们可以搭建各种东西，底层用的是 Spark 之类的技术。

Just in time — it's a good computer science principle. That was one of the first things that unlocked the model factory. The moment you start thinking about it, a training job — whether it's a big hero run or a small post-training experiment — consumes a certain number of tokens per second. It's actually not a lot from a data-movement perspective. So we said, we have our training cluster, and now we have this AWS-like setup where we can build amazing big data pipelines. We can set things up; we use Spark underneath the hood, all these things.

但你说 AWS 的时候，不是真正的 AWS 吧？是你们内部的？

But when you say AWS, it's not actual AWS — it's your internal.

是我们内部的——就是跑我们自己的内部 Web 服务。

It's our internal — we're just running our internal web services.

跑在 AWS 账号上，或者任意硬件上，对吧？

Running on an AWS account or any hardware, right?

一旦我们转向把数据流式送进训练，你会突然发现很多东西都解锁了。现在你不需要等整个数据集物化完成。当你做数据混合实验时，它只是一个配置，因为数据源是源源不断进来的。我们有一个叫 Blender 的服务——报告里有提到——我们在里面配置：这次运行，我要 20% 来自这个数据源、10% 来自那个，重复多少个 epoch，用某种方式打乱。你的训练任务可以启动，而其余数据还在物化中。另外，因为底层我们把数据层当作不可变数据层，这一点非常重要。实验即代码加上不可变数据层，意味着你随时可以回溯，精确到单个 token，它是在哪个 cursor 位置、哪个代码版本下进入训练的。

Once we made that shift to streaming data into training, you suddenly realize a lot of things unlock. Now you don't have to wait for the whole dataset to materialize. When you're running data experiments about mixing data, it's just a config, because you have these data sources coming in. We have a service called Blender — it's in the report — where we say: for this run, I want 20% of this source, 10% of that source, this many epochs of repetition, shuffled in a certain way. And your training job can start while the rest of the data is still materializing. What it also does — because underneath it all, we treated the data layer as an immutable data layer. That was really important. Experiments as code plus immutable data layer means you can always go back and understand, down to the single token, at which cursor it went in and on which version of the code.

嗯。

Yeah.

而且我得承认，poolside 的第一年，我们明白工程能力必须变得很强，但还没真正理解这最终是为了支撑严谨的科学研究进展。我们当时人很少，所以很多想法和实验都是“YOLO”（先跑再说）式的。

And it took us — I have to admit — the first year of poolside, we understood that engineering had to get great, but we didn't yet understand that this is ultimately in support of rigorous scientific progress. We were quite a few — a very small number of people — so a lot of it was YOLO ideas and YOLO runs.

嗯。

Yeah.

我们为这些 YOLO 式实验建了很棒的基础设施。但一旦我们意识到，把数据当作不可变的、代码始终有版本，你就可以把每个实验端到端完美地追踪和溯源；你也可以完美地复现一切。也就是说，你有了完美的可复现性。如果我愿意，现在还能复现两年前的实验。这其实才是科学流程能够运转的原因。我觉得我们是在公司成立一年到一年半之后才想明白这件事的。我们还有一些很棒的新人加入，比如我们的应用研究联合负责人 Nikolai，他是从 Yandex 来的，从 2020 年代初就开始做语言模型。他把这种理念带进公司：“嘿，我们要更严谨。”一旦我们把一个能力越来越强的平台——让人们能做更多事——和这种不可变性结合起来，我们就能说：好，每个实验都是真正的消融实验，我们必须真正理解它。我觉得过去两年我们变得科学严谨多了，底层基础设施是其中的关键。然后还有一些好玩的东西——比如我喜欢……

We built great infrastructure for those YOLO runs. But once we realized we treated data as immutable and code as always versioned, you could always track and trace every experiment end to end perfectly. You could repeat everything perfectly, right? You had perfect reproducibility. I can still reproduce runs from two years ago if I wanted to. That actually enables the scientific process. I think it took us about a year, a year and a half into the company to figure that out. We also had some great hires — our co-head of applied research, Nikolai, who joined us from Yandex and had been working on language models since the early 2020s — brought that into the company: 'Hey, we want even more rigor.' Once we had the combination of an increasingly capable platform that allowed people to do more but with this immutability, we were able to say: okay, every experiment is truly an ablation; we truly need to understand it. I think we became much more scientifically rigorous in the last couple of years, and the infra underneath enabled it. And then there's just fun stuff — like I like...

是啊，很多东西都很有意思。比如你分享的关于如何挑选数据集的所有消融实验——这里面有一小段写着：哦，预训练数据，我们有一个自动混合器（automixer），它会训练八个小型模型，然后放大，挑选预训练数据集；我们甚至都不用去看。我当时想，哇，这里面有大量的工程精力。还有很多很多。

Yeah, a lot of it's fun. Even just the one where you share all the ablations for picking the datasets — there's a small paragraph in here that says, oh yeah, pre-training data: we have an automixer that trains eight small models, scales them up, picks the pre-training dataset; we don't even need to look at it. I'm like, wow, a lot of engineering vigor there. And there's just a lot.

是的，而且我想说，我们希望发布更多内容。实际上，很长一段时间里，我们都把写论文看作“我们还没挣到这个权利”的事。只有当你处在前沿时，你才有资格花时间去发表研究；因为在那之前你都在追赶，这个行业里的每一分钟、每一小时都很重要。我执着的不仅是“从想法到结果的墙上时钟时间”，还有每天被浪费的时间——那会让你追不上。但在这次，我们说，好吧，我们给自己——我记得给了团队三四天时间，同时还要继续做日常工作——把所有东西都放进去。就像你之前说的，如果你真的懂，发布出来是很容易的。所以还有很多很多我们想在未来聊的东西，我们一定会开始做。而且随着我们赢得更多发表的权利——再加上我们现在的使命里多了一条：希望出现更多基础模型公司——你会看到我们更加主动，不断分享我们一路走来学到的、能帮别人提速的东西。

Yeah, and look, we want to put out more. Actually, for a long time we treated writing papers as something we hadn't earned the right to do yet. You earn the right to spend time publishing research once you're at the frontier, because until then you're catching up, and every minute and hour in this industry matters. I obsess over not just wall-clock time from idea to result, but just general time every day that we waste, because it doesn't allow us to catch up. But in this case, we said, okay, we're going to give ourselves — I think we gave the team three or four days while still doing their work — like give everything in there. And to your point earlier, if you know your stuff, it's easy to put it out. There are so many more things we want to talk about over time, and we will definitely start doing that. And as we earn more of the right — but also now that we've added to our mission that we want more foundation model companies to exist — you'll see us be way more proactive, just trying to keep dropping some of the things we've learned along the way that can help others speed up,

这就是这件事的另一个酷炫之处，对吧？回到你的观点——这不只是“这是我们训练的基准。”如果你想复现，这里有优化器、数据集、后训练的实验。你在这里摆了一堆，旁边还写着“这是你们可以用来做这件事的系统。”所以这其实是在推广……

Which is the other cool side of this, right? Back to your point — it's not just 'here are the benchmarks of our training.' If you want to replicate, here are experiments of optimizers, datasets, post-training. You lay out a lot of it here alongside 'here's your system for how to do it.' So it's really like promoting...

让其他人也能做同样的事。

Other people can do the same.

顺便说一句，我也想明确一下：我们确实从其他人发表的所有开放研究中获益良多，对吧？而且你提到过那些中国实验室——我认为重要的是，每个国家、每种文化、每种背景，包括像我们这样的西方公司，都会有不同的模型出现，让人们可以选择信任。但我认为我们必须给予应有的认可。这些出色的中国实验室在分享他们的研究方面做得非常好，而我们绝对是受益方，享受了这些研究成果。所以当你是某种东西的受益方时，我想你也有义务回馈。你有没有特别喜爱或被低估的中国实验室想点名表扬？大家都喊一句 DeepSeek（深度求索）。

And by the way, I also want to make clear: we have taken a lot of advantage of the fact that all the open research that others have published, right? And you mentioned that the Chinese labs—and I think it's important that there are, from every country and every culture and background, including Western companies like us, different models that come out that people can choose to trust. But I think we do have to give credit where credit's due. The incredible Chinese labs have done an amazing job at sharing their research, and we have definitely been on the receiving end of that, taking advantage of it. So when you're on the receiving end of something, I think you also have an obligation to give back. Do you have a favorite or underrated Chinese lab that you want to shout out? Everyone shout out DeepSeek.

这是个好问题。Mulan 显然因为……

That's a good question. Mulan obviously for…

嗯。

Yeah.

嗯。你看，我觉得显然大家最近都在谈论智谱和 GLM 5.2。但我觉得大多数人没有意识到的是他们是什么时候起步的。

Yeah. Look, I think obviously everyone's been talking about Zhipu lately with GLM 5.2. I think what most people don't realize is when they started.

嗯。

Yeah.

对。他们在 ChatGPT 出现之前很多年就开始了。只是后来改了个名字。所以我到现在还记得，在世界其他地方对这个事情兴奋起来之前，做这些事情有多难。因此，我非常敬佩那些在模型改进还不受关注、相信大语言模型甚至会被人嘲笑的时候，就投身其中的人。我记得 2016 年，当我们用一些这样的模型做我们所谓的代码机器学习时，人们真的会当面嘲笑我们，说：‘这没有意义。你们为什么要浪费几百万美元去研究这个？’所以我想，他们可能是最值得点名表扬的，不仅仅因为他们最新的模型非常好，而是因为他们一路拼到了今天。而且我觉得每一家基础模型公司都需要时间才能走到这一步，对吧？我们花了三年时间，才做到现在即将发布的这个模型。而现在，模型发布的间隔已经缩短到以周来计算，不再是以月或年。但这件事很难。如果我们能让下一个人稍微容易一点，我们都应该这样做，因为如果不这样做，我们面临的时间窗口很小——模型很快就会对递归自我改进产生巨大影响，到那时再追赶可能就不现实了。我们应该在那个窗口期鼓励尽可能多的新实验室——或者随便我们怎么称呼它们——开始做这件事。所以我目前的一个使命就是：我想鼓励现在任何一个觉得自己真的能攻克这个问题的研究者，去另外创办一家基础模型公司，成为我的竞争对手，因为我觉得我们需要这样。否则，我们不会进入一个……我不想只是成为第五或第六家胜出的公司。我想看到一个充满选择的世界。

Right. They started years before ChatGPT. It was just rebranded. And so I have like… I remember how hard it was to work on these things before the rest of the world got excited about it. And so I have an immense amount of respect for people who were working on improving models when it wasn't the sexy thing to do, when believing in LLMs was going to get you ridiculed. I remember back in 2016, when we were doing what we'd call machine learning on code with some of these models, people would just frankly laugh at us: 'This makes no sense. Why are you wasting all these millions of dollars trying to figure this out?' And so I would say they're probably the one that deserves the shout out, not just because their latest model is very good, but because they fought to get here. And I think every foundation model company takes time to get here, right? It took us three years to get to the model that we're now going to be releasing. And now the time between models is counted in weeks, no longer counted in months or years. But this stuff is hard. And if we can make it a little bit easier for the next person, we should all do so, because if we don't, we've got a small window before models are really impacting recursive self-improvement to a level where catching up otherwise might become unfeasible. And we should try, in that window, to encourage as many new labs—or whatever we want to call them—to start. And so one of my current missions is: I want to encourage whoever is a researcher right now who thinks they can actually tackle this to go and leave and become my competitor, start another foundation company, because I think we need it. Otherwise, we're not going to be in the world where—I don't want to just be the fifth or the sixth company that wins. I want to look at a world where there's lots of choice.

创办一家基础模型公司，人们还有哪些看不到的地方？你知道，需要大量算力、大量资本、大量算力。你搭建模型工厂，设计训练方式，但这里面还有很多东西，对吧？这就是一个……

What else do people not see in starting a foundation model? You know, there's a lot of compute, a lot of capital required, a lot of compute. You lay out the model factory and how to do the training, but there's a lot there, right? That's a…

嗯，你看，这其实是一个过度简化的说法——我总是要加上这个说明——因为它可能会在人们脑海里产生一些偏差。但我真的认为你可以把 95% 的模型开发归结为只做两件事：要么改进数据，要么提高算力效率。我知道，对于人们所做的那些杰出而有技巧的工作来说，这听起来像是一种过度简化。但如果你真的去看——我们在做什么？我们在看数据。我们生成新数据。我们改进数据。而要想做到这一点，唯一的方法就是去看数据，对吧？这是基础模型开发中很大的一部分。另一方面，我们在推理、架构、新的注意力机制上取得了一些了不起的突破。但它们本质上在做什么？它们在提升算力效率。当然，这些年来我们确实有了一些能带来更强模型能力的突破，但说到底，如果你能训练一个足够大的模型——好比拥有无限的算力——我们可能明天就已经达到 AGI（通用人工智能）了。而且——让我说，无限算力加上无限快的网络能力，因为网络往往比算力更成为瓶颈。所以我认为，主要就是这些。另外要认识到，这本质上是一门工程。我觉得这一点已经越来越明显。但在相当长一段时间里，人们把基础模型公司和研究者捧上了神坛，好像你在做某种不可思议的魔法或火箭科学，好像只有诺奖物理学家才能做。不要误会，确实有一些非常难的问题需要解决。但我们大多数人日常做的大部分工作，并不是坐下来证明某个数学定理。我们做的大量工作其实就是做好基础的事：写好的代码、看数据、改进数据、跑实验、看曲线、试着塑造我们的直觉。而且，其实有更多人能够成为能力很强的研究者。我这么说可能让人感觉遥不可及。但我在我们公司内部看到过，工程师变成了研究者，因为模型工厂让他们以更低的门槛去跑实验、去尝试。我们团队里有个人，一开始是做我们智能体的工程师，现在已经是真正的强化学习研究者，取得了实实在在的进展。而这只用了大约 6 个月。我觉得这在几年前，大多数人都不会认为有可能。

Well, look, it's an oversimplification—and I always asterisk it with that—because it can land a little bit the wrong way in people's minds. But I actually think you can sum down—I sum up 95% of model building to just two things: you're improving data, or you're improving compute efficiency. And I know it feels like an oversimplification for the incredible gifted and skilled work people do. But if you really look at it—what are we doing? We are looking at data. We're generating new data. We're improving data. And the only way to do that is to look at the data, right? That's a big part of foundation model building. And on the other hand, we come up with these incredible breakthroughs in inference, in architecture, new attention mechanisms. But what are they really doing? They're bringing compute efficiency. Now, we have definitely had some breakthroughs over the years that allow for more model capabilities, but at the limit, if you could train a large enough model—like you had infinite compute—we'd be at AGI probably already tomorrow. And so—let me say that infinite compute with infinite ability of much faster networking, because networking is more of the bottleneck than compute. So I do think that those are the main things. And to just realize that this is engineering—I think it's become more obvious. But I think for quite a few years, people have held foundation model companies and researchers on this pedestal, like you're doing credible magic or rocket science, or only Nobel laureate physicists can do this. And don't get me wrong, there are some really hard problems that need to be solved. But a lot of the work that all of us are doing on a day-to-day is not sitting down trying to solve a math theorem. A lot of the work we're doing is just really doing the basics: writing good code, looking at data, improving it, running experiments, looking at plots, trying to shape our intuitions. And a lot more people could be highly capable researchers. I've seen in our own company, we've seen engineers become researchers, because the model factory allowed them to have a much lower hurdle of running experiments and trying things. And one of the guys on our team who started as an engineer building our agents is a legit reinforcement learning researcher now, making real progress. And that happened in the span of like six months. That would not have been—I think most people assumed it was possible a couple of years ago.

嗯。我觉得一个有趣的时刻是，当你能够实现一种‘自举’——你知道，对于一门编程语言来说，比如你能用这门语言去编译这门语言本身——与之对应的是：你能使用自己的工具吗？对吧。你们有 Pool CLI，有自己家的模型。大概你们不会只用自己家的模型——那不可能——但你知道，这个比例随着时间推移是多少？

Yeah. I think one of the interesting moments is when you can sort of self-host—you know, for a programming language, like if you can compile the language in the language, the equivalent is: can you use your own tools? Right. You have the Pool CLI, you have your own models. Presumably you're not only using your own models—there's no way—but, you know, what's that percentage over time?

这是我们发布的第一款开始对自身工作产生显著贡献的模型。它还不是最顶尖的模型。市面上已经有很多能力很强的模型，但 Laguna S 真的很有意思。

This is the first model that we're releasing that is starting to meaningfully contribute to our own work. It's not a state-of-the-art model yet. There are very capable models, but Laguna S is really interesting.

我要把那段话调出来。我们的应用研究联合负责人之一 Penging 上周说过，大约 10 天前模型刚出来时，坦白说比我们期待的要好得多。他说：“我感觉 Laguna S 的很多进步不是来自更强的智能，而是来自不同的行为：更多的验证、更少想当然、不提早宣布胜利，以及更加坚持不懈。”说实话，这些特质比原始智能更能预测成功——对人类在某种程度上也是如此。他是 7 月 5 号周日写给我的，从那时起这句话就深深印在我脑海里。因为 Laguna S 这个模型，你会看到，它在基准测试和日常使用中表现这么好的原因，就是它极其执着。它思考很多，我要明确指出这一点。我们在提升效率、提供不同推理模式方面还有工作要做。但这个模型做到了我从未想过它能做到的事。它是一个 118B 总参数、8B 活跃参数的模型，并不算大。它可以装进 DGX Spark，在 Spark 上仍能以每秒 30 到 40 个 token 的速度运行。它能独立解决 Erdős 397 问题，能完成复杂的编程任务。今天早上我让它帮我在 Mac 上写一个不依赖任何外部库的 Wi-Fi 扫描器，它在没有网络的情况下，靠不断坚持尝试，真正理解了核心 WLAN API。我越来越喜欢检验它。过去 10 天，我每天大概跟这个模型待 8 到 10 个小时，一点也不夸张。昨天我坐了 11 个小时的飞机，其中 10 个小时都在读这个模型的轨迹和追踪记录。

I'm going to actually pull up the quote. Penging, one of our co-heads of applied research, said something last week, as the model came out about 10 days ago. It was much better, frankly, than we had hoped or expected. And he said, 'I have the feeling that a lot of the gains in Laguna S come not from more intelligence, but more from different behavior: more verification, less taking things for granted, not declaring victory early, and being way more persistent.' To be honest, those are more predictive than raw intelligence for success — and in humans, too, to some degree. He wrote me this on July 5th, a Sunday, and it's kind of been burned into my brain ever since. Because the Laguna S model, as you'll see, does so well in benchmarks and in daily use simply because it's incredibly persistent. It reasons a lot. I definitely point that out. We have work to do on making it more efficient, and we have work to do on offering different reasoning modes. But this is the model that has been able to do things that I never thought it could do. It's a model with 118 billion total parameters and 8 billion active parameters, which is not that large. It fits on a DGX Spark and still runs at 30–40 tokens per second on a Spark. It's able to solve Erdős 397 independently. It's able to do complex programming tasks. I asked it this morning to make me a Wi-Fi scanner without using any external libraries on my Mac, and it figured out the core WLAN API by really persistently trying to understand it, without access to the internet. And more and more, I love checking. I've probably spent 8 to 10 hours a day with this model for the last 10 days — I'm not exaggerating. I was on my 11-hour flight yesterday, and I spent 10 hours reading trajectories and traces of the model.

我从中得到的收获，正是 Ping 说的那句话。我们将能够从更小的模型中榨出比我们行业原本想象的要多得多的能力。因为是的，智能是存在的，更大的模型也确实更智能，这一点毋庸置疑；我们应该继续做 Scaling。但那些行为——真正地坚持不懈、发现自己错了以后能够回溯、懂得如何与环境互动——告诉我们，我们能从它们身上得到更多。这让我在过去几天心里产生了一个疑问。想一想我们现在在哪些场景使用模型。我们使用模型，比如说，做知识型工作，这占全球经济的 25%，也就是 25 万亿美元的工作。随着模型 Scaling 到更大规模、变得更聪明，我们很兴奋能越来越多地用它们去推动科学前沿。如果你看科学前沿，真正的突破在很多地方确实跟更强大的智能联系在一起。爱因斯坦想出广义相对论，能把别人不会放到一起的想法结合起来。我认为智能的众多维度之一就是这种能力。而且我们清楚地看到，随着模型变得更大、能力更强，它们能把更多的想法和线索汇聚起来，而较小的模型做不到。

And what I take away from it is exactly what Ping said. We are going to be able to squeeze so much more out of smaller models than I think the industry had imagined. Because yes, there's intelligence, and larger models are more intelligent — no doubt about it; we should continue to scale up. But the behaviors — being really persistent, being able to backtrack when you're wrong, understanding how to interact with your environment — show us that we can get a lot more out of them. This, for me, has created a bit of a question in my mind over the last couple of days. If you think about where we're using models today, we're using them, say, for knowledge work, which represents 25% of the global economy — $25 trillion of work. As we scale up models and they become more intelligent, we're excited about using them more and more to push the frontier of science. And if you look at the frontier of science, true breakthroughs have been linked to more intelligence in many places. Einstein, figuring out general relativity, was able to bring ideas together that others would not have brought together. I think one of the many dimensions of intelligence is the ability to do that. And we clearly see that as models get larger and more capable, they can pull together more ideas and threads than a smaller model could.

我们开始看到这样的例子，出现在医学、生物学等领域。但想想我们做的大部分知识型工作——包括写软件：我内心深处首先是个软件开发者。不过现在可能不能这么说了；我很多年没写生产代码了。让我们真正变优秀的，其实是我们的坚持。是遇到问题然后回溯的能力，然后说：“我需要把这个 bug 搞清楚，需要去研究它，需要去查文档，需要尝试五种不同的方法来看能不能解决。”但这并不一定就是把来自完全不同领域的三个想法结合在一起。所以如果我们现在看到——而且我认为 Laguna S 就是一个例子——我们能做出一个相对较小的模型，其能力远超我的预期，也远超之前任何基准测试对同等甚至更大规模模型（至少是编码任务）的展示，那都是因为这些行为。所以现在我脑子里有一个还没有答案的问题：我知道在极限情况下，模型无限大，一个极大的模型运行成本会非常高。这点我们都清楚，对吧？所以更大的模型有个 ROI 问题。我知道在最极端的情况下，我不会用全世界最大的模型——哪怕到千万亿参数、无论我们扩张到什么疯狂规模——去做一个基础编码任务。今天我已经开始为某些任务选择更小的模型。这意味着存在一个最优值。有一条曲线：当我们为了知识型工作而增大模型规模时，会到达某个峰值；在那之后，用更大模型的投入产出比就不划算了。

We're starting to see examples of that in medicine, in biology, and other things. But if you think about the majority of knowledge work that we do — and it includes building software: I'm a software developer at heart, first and foremost, probably. Although I probably can't say that much anymore; I haven't written production code in years. What makes us good is actually our persistence. It's our ability to encounter a problem and backtrack, and say, 'I need to go figure out this bug. I need to research it. I need to look at the documentation. I need to try five different ways to see if I can solve it.' But it's not necessarily bringing three ideas together from radically different fields. And so if we're now seeing — and I think Laguna S is an example — that we can make a relatively small model far more capable than I had predicted, or than any previous benchmarks had shown for any model remotely this size or even larger, at least on coding tasks, it's because of these behaviors. So now the question I have — and I don't have it answered — is this: I know at the limit, with infinite model size, an extremely large model is going to be very expensive to run. We know this, right? So larger models have an ROI issue. I know that at the very limit, I'm not going to use the world's largest model — quadrillion parameters, whatever crazy scale we scale up to — to do a basic coding task. Already today, I'm starting to size down for certain tasks. So it means there is an optimum. There's some curve: as we go up in model size for knowledge work, at some point we're at the peak, and after that, the return on investment of using a bigger model just doesn't make sense.

现在，我想问题在于——以前，我会认为那个峰值非常非常遥远。这个模型对我来说是第一个信号：也许峰值在一万亿、五万亿、十万亿参数。也许我们能从这些模型中榨出多得多的能力。我不再认为我们需要把最大模型再扩大两三个数量级才能解决知识工作——会计、法律，以及我们写的代码。所以如果这一点成立，那就构成了模型商品化的论据，也是开源能在这个世界中获胜并成功的论据。当然，这是一个自私的论据，也是一个充满希望的论据。但从理论上讲，在极限情况下它是有道理的。我们只需要在未来几年去发现还能榨出多少。现在我想加一个大大的星号。这并不意味我反对 Scaling 模型。我认为，只有把模型 Scaling 到和竞争对手一样大，我们最终才能成功。我们不应该把头埋进沙子，说什么我们要做开源小模型的王者。坦率讲，那是一种逃避。那是在自己的王国里当王，却看不到外部世界在做什么。所有人都会更愿意用一个更聪明、更快、能力更强的模型。但这是一种希望的信号。所以我不想过度渲染：这是一个好模型，但离最先进水平还有很长的路要走。我希望人们使用这个模型时得到的启发是，它内部的行为——而不是参数的多少——才是推动它远比预期强大的原因。

Now, I think the question is — before, I would have thought that peak was really very far away. This model, for me, is the first sign that maybe that peak is at a trillion, five trillion, ten trillion. Maybe we can squeeze way more out of these models. I'm no longer thinking that we need two or three orders of magnitude on the largest models to solve knowledge work — accounting, legal, and the code we write. And so if that holds true, it's an argument for the commoditization of models. It's an argument that open source can win and succeed in this world. And now, of course, it's a self-serving argument and a hopeful argument. But theoretically, at the limit, it works. We just have to discover, over the next couple of years, how much more we can squeeze out. Now, I do want to put a big asterisk. This does not mean I'm against scaling models. I think we ultimately only succeed if we scale our models as large as our competition. I think we should not put our head in the sand and say we're going to be king of open-source small models. That's frankly a copout. It's trying to be king of your own kingdom but not realizing what the rest of the world is doing. All of us would rather use a smarter, faster, more capable model. But it's a sign of hope. So I don't want to overstate this: it's a good model, but we have a long way to go to get to the state of the art. What I hope people take away when they use this model is that the behaviors inside it are what push it to be far more capable — not necessarily the number of parameters.

这主要来自后训练吗？

Is that mostly post-training?

完全是后训练。

It's entirely post-training.

预训练方面还有需要改进的地方吗？预训练已经完成了？

Are we done improving anything on pre-training? Is pre-training done?

没有。

No.

好。那我们就先聊预训练，然后再聊后训练。

Okay. So I just wanted to cover pre-training, and then we go post-training.

预训练有一部分只是处理技能问题。模型技能的每一个新数量级，你都会遇到需要解决的新问题，这些归根结底是工程挑战。我有一个可以说并非普遍认同的观点：强化学习会越来越早地进入预训练阶段。

There's a part of pre-training that's just about dealing with skill. Every new order of magnitude of model skill, you're going to get new things you have to solve for. Those are ultimately engineering challenges. I have an opinion that I would say isn't commonly held: reinforcement learning will move earlier and earlier into pre-training.

是啊，这不就是中期训练（mid-training）吗？

Yeah, is that mid-training?

不是中期训练。今天的中期训练其实是……我们已经在这方面做了好几年了。我记得第一次在公开场合看到它，是 DeepSeek Zero 论文，大概一年半以前，如果我没记错的话。你可以在一开始就能在模型里诱发推理，只要它开始具备使用语言等能力。我心中的问题是：我们有一份数据集，就是互联网。而互联网，我想可以说，很可能在某个地方以不同形式编码了人类知识的全部。质量差异极其巨大——从垃圾数据开始，一旦你看过预训练数据，你真的会被互联网的样子震撼——到最伟大的科学论文、最好的博客文章、最好的转录稿等等。所以我们现在要弄清楚、并且已经投入大量工作的事情——在这个方面我们可能不像在其他事情上那么开放，但随着时间的推移我们会变得更开放——我们已经花了几年时间真正研究如何把互联网变成不只是下一个词预测，而是在训练早期就教会模型思考的一种方式。我觉得那里有大量的金子可挖。现在，我们行业里有几种“药物”。其中一种是蒸馏，另一种是更多的环境。它们很棒，让我们感觉良好，也让模型变得更好，我们都对它们上瘾，会以各种方式使用它们。但归根结底，我觉得我们从互联网中榨取的东西，还远远没达到本应得到的程度。

Not even mid-training. Mid-training today is like... Well, we've been working on this for years already. I think the first time we saw it out in public was the DeepSeek-Zero paper, a year and a half ago I think, if I recall correctly. You can induce reasoning very early in a model, as it starts to become capable of using language and so on. The question I kind of have is: we have this dataset that is the web. And the web, I think we can arguably say, probably has the totality of humanity's knowledge somewhere encoded in different places. The quality varies enormously — from garbage data, and once you look at pre-training data you really get humbled by what the web is — to the greatest scientific papers, the best blog posts, the best transcripts, and whatnot. So what we are trying to figure out, and have been doing a lot of work on — and it's a place where maybe we're not as open as we are on other things, but we'll become more open over time — we've been spending a couple of years really doing research on how we can turn the web into not just next token prediction, but a way to teach the model to think earlier in its training. And I think there's a huge amount of gold to be found there. Right now we are in a situation where we've got some drugs in the industry. One of the drugs is distillation. Another drug is more environments. They're great, they make us feel good, they make the models better, and we're all addicted to them, and we'll use them in various different ways. But ultimately, I think we are still barely squeezing out of the web what we should be getting out of the web.

我觉得仅仅做下一个词预测的预训练是不够的。所以我认为我们还会看到一些非常有意思的事情发生。而后训练中使用强化学习来诱发行为、改进表现——我觉得全世界现在都知道怎么做了。我们在规模化，每个人都在做。但我怀疑我们是否需要像今天这样在环境上走那么远。我不太确定你是不是说我们走得太远了。

I think just next token prediction pre-training is not enough. And so I think we'll see some very interesting things still happen. And that RL in post-training to induce behaviors, to improve things — I think the whole world knows how to do this now. We're scaling it up. Everyone is. But I wonder if we need to go as far as we're going today with environments. I'm not sure yet if you mean you're going too far.

我不确定通往 AGI 的道路……

I'm not sure if the path to AGI...

……只是更多的环境。

...is just more environments.

更多环境。

More environments.

这似乎是一个永无止境的过程。好吧，我想要这张桌子的说明书，对吧？我是要搞一个环境来学习组装家具，还是最终我们还是需要一个通用解决方案？

It seems like a neverending process. Okay, I want an instruction manual for this table, right? Am I going to use an environment to learn how to assemble furniture, or are we just going to end up needing some general solution?

我认为可以从互联网中实现更多泛化。但我也非常受鼓舞，因为当我看到 Llama 2 时——后训练才是产生重大影响的地方。然后我想，‘等等，只要把其中一些行为做得更好，我们就能从中得到多得多的东西。’这稍微改变了你对智能的思考方式。人们常说的类比是：强化学习阶段并不会学到多少新知识，只是重新分配。对，所以你在重新分配分布，然后让它向着你想要的方向推理。至于你提到的中期训练——很多中期训练其实仍然只是在某个领域里继续预训练，比如医学，然后再做强化学习。所以它仍然只是预训练，只是数据更好，对吧？我的意思是，中期训练——我喜欢我们发明这个词的方式。它实际上就像是第二阶段，是预训练的第二个阶段，用一种相当笨拙的方式来安排课程。最终你想要的是从第 0 个 token 到第 30 万亿甚至 40 万亿个 token 的课程，真正是模型学习的最优课程。但中期训练本质上是在互联网上的两阶段课程，因为我们没有足够的算力去彻底验证最优课程。所以我几乎可以肯定，很快会有人开始讨论别的术语，因为现在我们讨论的是阶段二、阶段三、阶段四、中期训练。但归根结底，我们做的只是为我们手里的互联网数据分配一个课程，让模型学得更好。我认为到了某个时候，随着算力发展、模型运行成本降低、下一代算力出现，这会变成一个更连续的谱系。我还觉得，之所以会有中期训练、阶段二、阶段三，其实是组织问题。这是我们在 Model Factory 中真正试图避免的：中期训练之所以存在，是因为现在有一个中期训练团队，有人决定专注于中期训练工作。但你要的其实是工程能力和实验能力，让我们拥有一个更连续的谱系，也就是现在可以有无限多个阶段。我们还没到那一步，算力还没到，组织设计也还没到。但我认为我们终究会到达。几年后我们回头看会想：‘天哪，我们以前用那么天真幼稚的方式做预训练数据。’我们几乎都没怎么排序，没真正做好构建课程这件事。我们会做到的。

I think there is an ability to generalize more from the web. But I also am very encouraged when I look at Llama 2 — just post-training is where the big impact is there. And I see like, 'Oh wait a second, just by making some of these behaviors much better, we're able to get so much more out of it.' It just changes a little bit the way you think about intelligence. The analogy people often draw is that the RL phase is where you don't learn as much new knowledge. You just reshift. Yeah. So you reshift the distribution, and you can have it reason towards what you want. On your point about mid-training — a lot of mid-training is still just continuing pre-training on a domain, say medicine, then you do RL. So it's still just pre-training. It's just better data, right? I mean, mid-training — I like how we invented this word. It's effectively just like a second phase. It's a second phase of pre-training with a really dumb way to do a curriculum. Ultimately what you'd want is a curriculum from token zero to token 30 trillion, whatever, or 40 trillion tokens, that truly is the optimal curriculum for the model to learn. But mid-training is essentially a two-stage curriculum on the web, because we do not have the compute to actually ablate the perfect curriculum. So I'm pretty sure you'll start to see people talking soon about some other term or two, because now we have this conversation — we talk about stage two, stage three, stage four, mid-training. But ultimately all we're doing is trying to assign a curriculum to the web data we have, to allow the model to learn better. I think at some point, as compute evolves and models get cheaper to run, as next generations of compute appear, this will become more of a continuous spectrum. I also think the reason you have mid-training and stage two and stage three is organizational. This is something we really try to avoid with the Model Factory: mid-training exists because there's a mid-training team now, people decide to focus on a mid-training effort. But what you really want is engineering and experimentation skill that allows for a much more continuous spectrum, so that you have infinite stages now. We're not there. Compute is not there. Organization design is not there for it yet. But I think we'll get there. We'll look back in a couple of years and think, 'Oh my god, it was so cute that we did our pre-training data in such a naive way.' We barely ordered it. We didn't really do a good job of building that curriculum. We'll get there.

而且我要确认一下——当我和一些研究者聊天时，现在大家大量关注的是：预训练如何改变，除了下一个词预测之外，下一个目标是什么？我猜你没有答案，但你有一些想法。

And I'll confirm — when I talk to some researchers, this is a lot of the focus now: how does pre-training change, and what is the next objective other than next token prediction? I assume you don't have the answers, but you have some ideas.

我们有了一些想法，但还没准备好多谈。我们多年来一直在做这些研究。而且我觉得这正是——就像你之前问到的，创办一家基础模型公司，有哪些不那么显而易见的事——你需要在很多事之间不断权衡……

We have some ideas. We're not ready to talk about it yet. We've been working on them for years. And I think that's the one thing — like you asked earlier about what's not obvious about building a foundation model company — is that you are constantly balancing...

那些基础必要的工作，也就是你知道有效的那套配方。

The table-stakes work. The recipe, you know, that works.

对。

Yeah.

与之相对的是你的突破性纯研究，以及找到这种平衡，并根据你在竞赛中所处的位置调整投入比例。

Versus your breakthrough pure research, and finding that balance, and adjusting the percentage based on where you are in the race.

……真的很重要。

...is really important.

我的意思是，这是一个很好的切入点——我本来打算某个时候提一下 auto research（自动研究）。那是 Andre 发明的另一个说法。

I mean, this is a nice way — I was going to bring up auto research at some point. That's another Andre invention, a coinage.

就是说——说真的，目标函数能有多少种，对吧？试试一千种，让它跑起来，随便什么。

Which is like — honestly, how many objective functions can there be, right? Just try a thousand of them, set it running, whatever.

还有就是……

It's also...

你知道你要找的是什么。你找的是那些明显失败的，就像这样。

You know what you're looking for. You're looking for the losers, like that.

是的，这也是人们下注的地方，对吧？当你说“更多 Neo 实验室”时，你就是在做“我们要做基础模型、扩大规模、下一个词预测”这件事。我们看到很多其他 Neo 实验室想走完全不同的路线，对吧？在某种程度上，你说得对——这一切都是算力效率，这才是最终目标——但你知道，有些人接受不同的架构，比如算力投入规模差异巨大。所以有些是不同的。他们不是 99% 都押在“普通模型然后扩大规模”上；他们是 99% 押在“这里有会改变一切的新研究”上。而且我觉得，这也取决于你什么时候开始，对吧？

Like, it's also a thing people take bets on, right? When you say more Neolabs, you're doing a version of 'we'll do foundation models, scale them up, next-token predictors.' A lot of other neolabs that we see want to take a completely different approach, right? At some level, you're right—it's all compute efficiency and that's the net objective—but you know, some are okay with different architecture, like vastly different amounts of compute spend. So some are different. They're not just 99% on the vanilla-and-scale-up approach. They're 99% on 'here's novel research that'll change everything.' And I think it depends when you started as well, right?

是的。我们刚起步的时候，做的创新事就是在代码上做强化学习。不——这早就不再新颖了，但当时没人相信强化学习，我们就是痴迷于这件事。所以你得——当你创办公司时，你必须有自己的想法。你得有能让你加速的差异化东西，对吧。对我们来说，那就是“从强化学习到大语言模型”，后来这变成常识，但一开始并不是。

Yeah. When we started, the novel thing we did was reinforcement learning on code. No—well, it's no longer novel by far, but that's where we obsessed over it when no one believed in RL. So you have to—when you start the company, you have to have your own idea. You have to have something that's different that allows you to speed up, right. For us it was RL-to-LLM, which later became common knowledge, but in the beginning it wasn't.

这很酷。你知道，这就像你 2023 年那篇初衷博文，而且你确实把一切都写出来了。那篇博客被低估了，对吧？整个“代码上的强化学习”是非常非常早的。

It's cool. You know, this was like your original 2023 blog of purpose, and you do lay it all out there. The blog is pretty underrated, right? The whole RL-on-code thing was very, very early on.

非常非常早。甚至我们还得和别人争辩。我们说，要突破当前能力，必须训练自己的基础模型。我们必须说服别人，拥有自己的基础模型很重要。你无法靠微调达到成功，对吧？重大能力源自训练一个基础模型，再在微调过程中让它变得准确和有用。

Very, very early. And even we had to argue with people. We said things like, to push beyond current capability, you have to train your own foundation model. We had to argue with people that it mattered that you had your own base model. You can't fine-tune your way to success, right? Major capabilities emerged from training a base model made accurate and useful during fine-tuning.

这放在当时来看，你知道，我们知道的闭源模型——OpenAI、Anthropic——都很庞大。我们有的开源模型大概是 Mistral 7B、30B、70B，对吧？

Which, for perspective at the time, you know, we knew closed models—OpenAI, Anthropic—were huge. The open models we had were like Mistral 7B, a 30B, a 70B, no?

但实际上——这上面的日期是错的——我们发布的时候是 4 月……

But when we actually—the date on this thing is wrong—when we published this, it was April...

2023 年 4 月？

April 2023?

我觉得这只是发生在 arXiv 的一份迁移文件上。Mistral 当时刚起步——我们在同一个月开始，对吧？所以当时没有其他东西——我觉得当时只有 Llama 出来，就这样，对吧？所以……

I think this just happened on a migration file on Arxiv. Mistral had started—we started on the same month, right? So there wasn't much else—there was only, I think, Llama out at the time, and that's it, right? And so...

嗯，但我同意——我认为我们需要尽可能多的多元想法。而且我确实认为，如果你是今天才开始，你想要能给你带来优势的东西，对吧？我确实觉得我们有时会过度——我认为每种架构在极限下都有效，RNN 也有效——只是算力效率不高，对吧？如果你有无限算力，你大概直接拿一个老式基础 RNN 就能走得很远。现在已经有意义重大的突破——注意力机制——还有其他一些东西。但我认为我们在理解这些方面仍然非常早期。最让我兴奋的事：很多人做极低精度训练，对吧？就像我们看到的 ternary（三值）之类的东西——非常酷。昨天 Bonsai 的东西也超级酷。你可以找到我在 2023 年的推文，大意是：嗯，这显然是一种权衡——更大的模型、更低的精度，按定义等于更小的模型、更高的精度，对吧？只是这实际上会怎么发展？真正的规模上限是什么？所以现在有公司在试着搞清楚这些。如果做得好，它们就是那种能改变我们行业的东西，因为我们算力的瓶颈归根结底是内存瓶颈和网络瓶颈，一旦你开始做这些事，就能改变。所以我对此很兴奋。我们没在做什么特别的——我是说，我们做常规的。Laguna S 是用 FP8 训练的。我得承认，这一轮里唯一不是 FP8 的就是我们昨天刚开始的新一轮里的 all-to-all——all-to-all 有点勉强，我们还不太放心做。Nematron 在 NVFP4 训练上有很棒的工作——我觉得他们在那里的成就是被低估了。我很期待用到 NVFP4 训练。但现在还不合理，因为我们还在 Hopper 上训练，对吧？我们现在相对较小——我们现在是 1 万张 H200 集群的公司。很快会扩大很多——如果你想申请工作的话，真的会多很多。但话说回来，我认为这里面还能榨出很多价值。希望 Laguna S 能向人们展示，一个这种尺寸的模型能做得更多，而且我们只用了 8 周就做出来了。我们认为任何模型尺寸都还有更多潜力可挖。我们现在扩大规模，因为这对我们公司来说是最优的选择，但如果时间无限，我很想在其他模型尺寸上更进一步。

Uh, but I agree—I think we want as many diversity of ideas. And I do think if you're starting today, you want something that gives you an edge, right? And what I do think we sometimes over—I think every architecture at the limit works, and RNN works—it's just not compute efficient, right? Like if you had infinite compute, you could probably just take a basic RNN from back in the day and you could get pretty far. There have been meaningful breakthroughs—attention—other things that are there. But I think we're still very early in figuring these out. The things I'm most excited about: people doing extremely low-precision training, right? So like the ternary stuff that we're seeing—and it was very cool. The Bonsai stuff yesterday was super cool to see. I think you can find tweets from me going back to 2023, which is this notion of: well, it's an obvious trade-off—bigger model, lower precision equals smaller model with higher precision by definition, right? It's just how does that actually play out? What's the actual size limit? So you now have companies trying to figure that out. Those are the kind of things that can change our industry if done right, because ultimately our bottleneck on compute is a memory bottleneck and a networking bottleneck. The moment you start doing those things—so I'm excited about that. We're not doing anything—I mean, we're doing the usual. Laguna S was trained in FP8. The only thing in this run that I have to admit wasn't FP8 was the all-to-all in the new run we just started yesterday—the all-to-all was just borderline; we're not perfectly comfortable doing it. You've got amazing work by Nematron in NVFP4 training—I think it's underrated what they've done there. I'm excited to get to NVFP4 training. It doesn't make sense yet because we're still training on Hoppers, right? We're relatively small—we're a 10K H200 cluster company right now. Will be scaling to a lot more soon—and really a lot more if someone is thinking about applying for a job. But yes, I think there's so much more juice to squeeze out of this. Hopefully Laguna S shows people that a model at this size can get a lot more, and we did this thing in 8 weeks. We think there's a lot more juice to squeeze out at any model size. We're now scaling up because it's the most optimal thing to do for us as a company, but if I had infinite time, I would love to push more capabilities at other model sizes.

我觉得你们还没有正式公布新的规格。我们有 XS，大概是 30B。旧的 medium 是 200B，看起来要废弃了。那么，新的——

I don't think we've properly announced what your new sizes are. So, we have XS, which was 30B-ish. Old medium was 200B, which is going to be deprecated, it seems. So, new—

Laguna S —— 也就是 Laguna Small。总参数量 1180 亿，激活参数 8B，所以非常稀疏。它是 XS 架构的放大版。它属于那种经典的——或者如今该说“经典”了——滑动窗口注意力与全局注意力 3:1 的比例。它这个尺寸很好，有几个原因。一是对我们来说成本效率非常高。这是一种好方式——我们想快速把进展发布出去。我们观察到的其中一点是，基础模型公司内部需要在“发布与交付”和“新研究”之间取得平衡。但有了 model factory（模型工厂），我们能把模型发布对团队的时间投入看得更轻，因为这只是在某个时间点：预训练跑完，套用最新的后训练，所以我觉得这是一个不错的重量级。它也能跑在 DJX Spark 上，我对这个小东西有点偏爱。我喜欢这个小家伙——你知道，能跑一个好模型。

Laguna S—Laguna Small. 118 billion total parameters, 8B active. So very sparse. It's a scale-up of the XS architecture. It's the kind of classic—or call it classic these days—like a 3:1 ratio of sliding window attention to global attention. It's just a nice size for a couple of reasons. One is just very cost-efficient for us. It was a good way to—we wanted to get our progress out quickly. One of the things we've seen is that inside a foundation model company, there's a balance between focus on releasing and shipping, and your novel research. But with the model factory, we can treat the release of a model as less of a time investment from the team because it's just, at this moment in time, you do the pre-training run, apply the latest post-training, and so this is, I think, a nice weight class. It's one that also will fit on the DJX Spark, which I have a small soft spot for. I love having that little thing—you know, run a good model.

是的，去年我们在 GTC 上这个播客里讲过它。

Yeah, we covered it on this pod at GTC last year.

不错。

Nice.

我觉得 GPT-OSS 12B 是第一款，因为它是一块大单 GPU，也就是 H100，对吧？租一块 H100。现在你有 128GB 的 Mac mini、Sparks。这是最舒适的家用区间。但我最兴奋的是，这个模型有望向人们展示在这个尺寸下能做到什么，因为当你看到基准测试并开始使用它时，你会发现我们超越了比它大两三倍的模型。

I think a GPT-OSS 12B was the first because it's a large single GPU, which was the H100, right? Rent one H100. Now you've got 128-gig Mac minis, Sparks. It's the home sweet spot. But I think what I'm most excited about is that this model hopefully shows people what is possible at this size, because when you look at the benchmarks and start using it, you'll realize we are outperforming models two or three times their size.

是的，他们也这么觉得，比如现在 Thinky 模型大约有 1 万亿参数。嗯——

Yeah, and they think so—for example, today's Thinky model is like a trillion parameters. Um—

所以是的，完全正确。而且你看，顺便说一句，我对它很兴奋——它刚刚发布。

So yeah, exactly. And look, by the way, I'm excited about it—it just came out.

各位听众——我是两秒前在手机上看到这条帖子的，还没来得及读。但不知怎么回事，你不仅在一项叫 Thinky 的基准上表现更好，在像 tau-bench 这样的某些基准上，你其实已经是 SOTA 了。

For those listening—I saw it on my phone two seconds ago, so I haven't even had a chance to read the post. But somehow you're not only better than Thinky, which is one of those benchmarks, but on certain benchmarks like tau-bench, you're actually state of the art.

嗯，我不确定我们在 tau-bench banking 上是不是 SOTA——我还没看排行榜——但我可以很有把握地说，在我们的体量级别里我们处于顶峰，即使面对两倍大的模型，我们很可能也是 SOTA。不过我要补充一句：现在世界上最好的模型绝对是——你知道的，Gemini Flash 和 GPT-5.6——而且正如你之前说的，我们也会用其他模型。

Well, I'm not sure if we're state of the art on tau-bench banking—I haven't checked the leaderboard—but I can confidently say we're at the top of our weight class, and even against models twice our size, we're probably state of the art. I'll caveat that: the best model in the world right now is definitely—you know, Gemini Flash and GPT-5.6—and, as you pointed out earlier, we also use other models.

对，我想你之前提到的有趣的一点是，你开始把自己大量的实际使用转到它上面，对吧？基准测试拿来比较是有用的，但它们并不是很真实。

Yeah, I think the interesting thing you mentioned earlier is that you're starting to shift a lot of your actual usage to it, right? Benchmarks are good for comparison, but they're not super realistic.

必须这样，对吧？这就是你在实践中验证一个基准的方式。你得用你自己的模型，你得有自己的内部评测和基准。有意思的是，一个新 checkpoint 出来后的前 30 分钟——也就是预训练之后的第一次后训练——你就能感觉到这个模型的走向。你还不确定，但当这个模型出来时，我们想说，‘哦，这次不一样。’我想这就是个例子。不过这有点像你的孩子——父母看自己的孩子，觉得完美、爱得不得了，看不到那些粗糙的边角。自己做的东西就是这样。最有意思的部分就是你对自己做的每个模型都会多爱一点点。我们一直试着说这句话：这是我们训练过的最差的模型。我知道团队现在已经在做下一个了，也应该如此，因为这是一场竞赛。这个模型是一个时刻，希望能向大家展示我们对这场竞赛是认真的，我们真的很想努力，我们想要反馈。它好在哪里？不好在哪里？让模型以开放的方式公之于众的一个好处就是，你会得到大量反馈。

They have to, right? This is how you dogfood a benchmark. You have to use your own models, and you have to have your own internal evals and benchmarks. The funny thing is, within the first 30 minutes of a new checkpoint coming out—the first post-training after a pre-training—you can feel where this model is going. You don't know exactly, but when this one came out, we were like, 'oh, this is different.' And I think that's the example. But it's a little bit like your kids—parents see their kid and it's perfect and they love it, and they don't see all the rough edges. You always get that when you build your own. The most fun part is you love a little bit every model you do. We try to constantly say this: it's the worst model we'll ever train. And I know the team is already on to the next one, as it should be, because this is a race. This model is a moment in time that hopefully shows people we're serious about this race, that we want to work really hard at it, that we want feedback. Where is it good? Where is it not? One of the nice things about having your models out in the world in an open way is that you get a lot of feedback.

你怎么看待在开发时配合智能体框架？比如 OpenCode、Codex，还有你自己的 CLI 工具。基本上让用户去用它——模型框架的代码设计，并在其中训练。

How do you think about building it while working with a harness? So OpenCode, Codex, you have your own CLI tool. Basically getting people to use it—the code design of the model harness, training it in.

所以你需要做一些多智能体框架训练，尤其是在这些较小的规模上。你要做一点多框架训练来获得正确的行为——不需要很多——让在你自己的框架里看到的行为能迁移到其他人可能使用的框架上。我们在内部把这叫作‘打磨’：你有自己的模型，你稍加打磨，让它能像在你自己的框架里一样，在其他框架里也表现良好。毫无疑问，它在你自己框架里会更好，因为你把强化学习算力都放在那里。你把强化学习和合成数据放在自己的框架里，因为这是你最了解、也最能推动的框架。这种端到端的控制正是让你把它做得更好的原因。而迁移这些能力，更多是要确保模型能引出适当程度的推理，并理解那些可能存在于别处的更复杂的、有点奇特的工具调用格式。所以我们确实会做所谓的多框架打磨。它不是提升能力的关键，但能改善体验。说实话，我想现在所有人大概都会这么做，但完全能理解，你自己的框架仍然会比别人的更好。我们能在所有基础模型公司身上看到这一点。当你在推动能力边界时，你不会想为了在强化学习运行里放进十个框架而牺牲这一点，因为那只是复杂度，是工程上的复杂度。当你想做好的科学、想真正理解是什么让模型变强时，你会想在你了解的东西上只改变一个变量。而别人的框架，你并不像了解自己的框架那样了解它——它们可能有不同的子智能体，或者不同的提示词。

So you need to do some multi-harness training, especially at these smaller sizes. You want a little bit of multi-harness training to get the right behaviors—just enough, not a lot—so that the behaviors you see in your harness transfer to the harness that other people might use. Internally we call this 'polishing': you've got your model, you do a little polishing so it works well on other harnesses, just as it does on your own. No doubt it's going to be better in your own harness, because that's where you put your reinforcement learning compute. You put your RL and synthetic data into your own harness because it's the one you understand best and can push the most. That end-to-end control is what lets you make it better. Transferring those capabilities is more about making sure the model induces the right amount of reasoning and understands some of the more complex, maybe weird tool-call formats that exist elsewhere. So we do some multi-harness polishing, as we call it. It's not really what drives capabilities, but it creates a better experience. Frankly, I think everyone probably does this these days, but it's totally fair to see why your own harness will still be better than others. We see this with all the foundation model companies. When you're pushing capabilities, you don't want to trade that off by putting ten harnesses in your RL runs, because it's just complexity. It's engineering complexity. When you're trying to do good science, when you're trying to really understand what makes your model improve, you want to change one variable in something you understand. A harness from someone else you don't know or understand the same way as your own—they might have different subagents or different prompts.

嗯，如果是开源的，你可以看源代码。

Well, if it's open source, you can look at the source.

对，但这是时间问题，对吧？我真的很想强调这一点。我知道我在这件事上是个怪人——我有朋友问，‘我们能见面吗？能做这个吗？能去那里吗？’我说不行，因为归根结底这是一场竞赛，时间才是唯一重要的东西。如果我看我们的团队，我会问：在我们走向能力更强、能快速泛化到其他智能体框架的总体路线上，引入这种复杂度值得吗？顺便说一句，我们的模型在其他框架里也表现良好。我真的鼓励大家去用。它表现很好。我们已经在 OpenCode、Kilocode 和其他工具里测试过了，还有 Cline——

Yeah, but it's time, right? I really can't stress this enough. I know I'm a weird person on this—I have friends who ask, 'Can we meet up? Can we do this? Can we go?' and I'm like no, because ultimately this is a race, and time is the only thing that matters. If I look at our team, I ask: what complexity is worth introducing on our general trajectory to building more capable models that generalize to other harnesses quickly? And by the way, our model works well in other harnesses. I really encourage people to do it—it works well. We've been testing in OpenCode, Kilocode, and others, and in Cline—

今天刚被收购。

Which just got bought today.

我知道，对吧？

I know, right?

没错。

Exactly.

什么都在被收购。

Everything's getting bought.

没错。

Exactly.

而我认为这其中有一部分是——你知道的，有一些很棒的——我很兴奋。我觉得 Hermes 其实是一个非常酷的智能体框架——所以……

And I think part of that is, you know, there are some amazing—I'm excited. I think Hermes is actually a ridiculously cool harness—and so...

而你知道，这个问题的一部分其实是，模型本身与‘模型加智能体框架’之间，投入到底差多少，对吧？所以像 Agentless Exam 这样的新基准，它不想只评测模型。同样，模型变得越来越智能体式，它们需要一个智能体框架才能运行，对吧？所以……

And you know part of the question was actually just how much effort is model versus model plus harness, right? So new benchmarks like the Agentless Exam don't want to just measure the model. Same with models getting more and more agentic—they need a harness to operate in, right? So...

我觉得，当你向一家模型公司问这个问题时，可以分成两部分。关于智能体框架——我们有一个非常精简的框架。你去看的话，大约就六个工具：shell、shell kill、shell wait、write、fetch web，还有，我不知道，bash——我想我漏了一个——但基本上就是这些工具。它非常简单、非常轻量。所以它不是那种为在某个基准上取得好成绩或针对某个特定子集而设计的框架。它不是深度研究型框架。

I think when you ask that question to a model company, you can separate it into two parts. The harness—we have a very slimmed-down harness. When you look at it, it's about six tools: shell, shell kill, shell wait, write, fetch web, and I don't know, bash—I think I'm missing one—but that's effectively all the tools. It's very simple and lightweight. So it's not a harness designed to do well on a benchmark or on a specific subset of tasks. It's not a deep research harness.

我认为，我们看到复杂外部框架的巨大能力——它们围绕模型构建大量提示词、接入额外数据源和其他工具，从而真正推动模型能力向前。但我们的模型在编程类任务上仍然优于某些这样做的外部框架，因为它是在这些任务上经过强化学习训练的。现在，我确实鼓励人们——顺便说一句，我认为我们的模型完全没问题、很好——差异可能小到没人会注意到，但我们最终在基准测试上还是略胜一筹。所以我觉得两种情况都成立：基础模型公司用它们的外部框架确实会推动模型进步，因为坦白说，这在运营上就是保持科学严谨、改进模型的最佳方式；但另一方面，如果有人拿我们的模型，花大量精力改进外部框架，他们也会超过我们，这很公平。那只是因为外部框架是模型能力与它需要的附加指令、数据访问和工具之间的一个过渡性补丁，对吧？而我认为这最终就是外部框架的本质。随着你构建更强大的模型，你也在改进模型的指令遵循能力。所以额外的外部框架其实就是在说：嘿，如果你遇到 X、Y 或 Z，就按这种方式行事。因此，即使你说两个模型配上两种不同的外部框架，最终都能达到你在意的同一能力，但一个真正针对某项能力定制的外部框架，会以更高效率做到这一点。这有点像一个人拿到了一本手册，告诉他如何用正确的工具和数据源以最有效率的方式完成任务，而另一个非常聪明的人被告知：你自己去搞清楚。他们都能解决任务，但一个会高效得多。所以，我非常喜欢现在世界上所有外部框架的开发工作。我们也想和更多外部框架的创作者合作，确保如果需要一些额外的训练，比如打磨，我们会去做。

So I think we see incredible ability for complex harnesses that build lots of prompts around and extra data sources and other tools to really push capabilities of models forward. But our model is still better than some other harnesses that do that in coding-like tasks because it was trained with reinforcement learning on it. Now, I do encourage people — I think our model, by the way, is perfectly fine and good — the differences are probably maybe too small for anyone to notice, but we see it ultimately still on benchmarks by a little bit. So I think both are true. Foundation model companies with their harnesses will really push them, because it's operationally, frankly, the best way to have scientific rigor and improve your models. But also, someone who takes our model and really does a lot of work on improving a harness is going to outcompete us, as they should. And that's just because the harness is the stopgap between what the model is capable of and what it needs as additional instructions and what it needs as access to data and tools, right? And that's ultimately, I think, what a harness is. As you build more capable models, you're improving the instruction-following of the models. And so an additional harness is just saying: hey, if you encounter X, Y, or Z, behave this way. And so even if you would say that two models with two different harnesses can equally reach the same capability that you care about, a harness that is really tailored towards a capability will do it more efficiently. It's kind of like a person who's getting a manual of how to do the task in the most efficient way, with the right tools and the right data sources, versus a really smart person like, go figure it out. They'll both solve the task, but one will do it a lot more efficiently. So, I'm a big fan of all the harness development that's happening in the world. And we want to work with more harness-like creators to also make sure that if it needs some additional training, like polishing, that we will do it.

我的意思是，当你说这是一场竞赛时，有个问题是你到底在奔向什么？你是在争当最好的编程模型公司，还是最好的编程模型加外部框架公司？我觉得这是不同的东西——或者你填空，或者都不是。

I mean, I think when you say it's a race, there's a question of what are you racing to? Are you racing to be the best coding model company, or the best coding model plus harness company? I think those are different things — or fill in the blank, or neither.

所以，我们——我从公司网站上线第一天起就提出 AGI（通用人工智能）编程这件事，而且我们已经反复说过很多次。我们认为，专注于编程和长周期软件任务是通往 AGI 的一条路径，因为它迫使我们解决难题。它迫使我们具备完成极其长周期复杂工作的能力，而这些工作需要大量推理、外部工具、数据等等。我可以给你看的一件事是：我们会为这个模型上线网页聊天，而我很喜欢用它做深度研究，只是用在编程任务里。它从来没有为这个训练过，甚至没有见过这类任务，但在我看来它做得很好，因为最终技能会迁移、会泛化。现在，我们当前没有专注的是把世界上最顶尖的医学知识编码进这个模型，或者世界上最顶尖的法律知识。但它实际上确实——我们不会发布这个基准测试，因为我们没有时间真正把它做好——但它在 LegalBench 上表现确实很好，至少在我们最初的几次运行里是这样。而我们发布评估时非常严谨；我们会逐项检查，运行很多次，并且尽量保持极度诚实。所以如果我们没能在某个内部使用且公开的基准测试上花足够时间，我们就不发布。所以另一种方式就是把它交给 Artificial Analysis，让它们作为第三方来跑。

So we — I raised AGI coding for us since day zero of our website, and we've said this over and over again. We think focusing on coding and long-horizon software tasks is a path towards AGI, because it forces us to solve the hard problems. It forces us to solve the ability to do extremely long-horizon complex work that requires lots of reasoning, external tools, data, etc. And one of the things I can show you: we'll have a web chat on with this model, and I've loved this model for deep research, just using it in my coding tasks. It was never trained for it, never even looked at it, but it's great at it in my opinion, because ultimately the skills transfer — they generalize. Now, where we are not focused on today is making sure the world's greatest medical knowledge is encoded in this model, or the world's greatest legal knowledge. But it actually did — we won't be publishing this benchmark because we didn't have time to really do it properly — but it did really well on LegalBench, at least on our first runs. And we are very rigorous when we publish evals; we have checked them for every little thing, run them many times, and we try to be extremely honest with it. So if we haven't spent enough time on a benchmark that we use internally, and that benchmark is public, we just said we won't publish it. And so the other way is just to give it to Artificial Analysis and let them run it like a third party.

哦，100%。我们自己也会这样做。但这仍然需要时间和精力，对吧？因为你要和人们一起搞清楚，你知道的，他们的基础设施、他们用的工具，以及这些是否配置良好。但我同意，你绝对应该这么做。我非常喜欢 Vals、Artificial Analysis 以及其他在做这类事情的公司。你是第一个提到这个的。

Oh, 100%. And we are going to be doing this as well. And still it takes time and effort, right? Because you're working with people to understand, you know, the infrastructure and the tools they're using, and are they set up well. But I agree, you absolutely want to. I'm a big fan of companies like Vals and Artificial Analysis and others that are doing this stuff. You're the first to bring it up.

是的，我觉得它们很棒。它们已经——我非常喜欢它们做出来并发布的大量工作。而且我觉得还会有更多。请创建更多评估公司，创造更多评估。我认为这对行业太有价值了。

Yeah, I think they're great. They've got — I love a lot of the work they've done and put out. And there are, I think, many more. Please create more eval companies, create more evals. I think it's so valuable for the industry.

简直是垄断了，有点——我感觉，哦，可能是双头垄断，你知道吗？

Actual monopoly, kind of — I feel like, oh, duopoly, maybe, you know?

我觉得这种格局是可以被打破的。

I think it can be broken.

是的，因为我觉得它其实很容易被打破。创造一项评估并不性感，但无论谁做了，大家都会为得到一个好评估而高兴。你从来没有——比如，如果一个评估构建得很好，人人都在庆祝，人人愿意为之付费，大家都愿意——就像基础模型市场。哦，是的，是的，我认为创造评估是好的。但就比如作为行业标准者，我们会运行这些基准测试，确保你没有作弊；而且我会以你运行它的方式去运行，用来和你的竞争对手运行的结果做比较。

Yeah, because I think it can actually be broken really easily. Creating an eval isn't sexy work, but whoever does it, everyone is happy to get a good eval. You've never — like, if an eval is well constructed, everyone's celebrating it, everyone's willing to pay for it, everyone's willing — like the foundation market. Oh yeah, yeah, I think creating evals is yes. But like in terms of being like, we are the industry standard ones that will run the benchmarks and make sure that you didn't cheat, and I'll run it the same way that you run it versus your competitor run it.

是的，这非常正确。我们需要这个。而且有几家标准机构让我们所有人都需要遵守，这其实是件好事。它让我们都保持诚实。我认为这非常重要。不过，是的，不是这样——我的目标其实是构建世界上能力最强的模型。而目前我们专注的是编程智能体的能力——长周期工作。但你会看到，这会带来很多额外好处。我总说，当我们在编程上达到最先进水平和前沿后，对我们来说要容易得多，可以说：好了，现在我们专心用模型工厂为那些我们不太强的领域添加更多数据，比如医学、法律或其他任何领域。类似地，我认为我们看到的——而且我们从推理模型中经常看到——如果你让模型获得正确的知识来源，并且它们拥有强大的推理方式，它们就能很好地进入那些它们不太熟悉、甚至在训练数据中从未出现过的领域。所以，我们是智能体式模型加外部框架吗？不，我们是模型公司。但我认为如今的模型不靠外部框架就无法训练。这是不可能的。所以这有点像以前只是容器里的权重；现在则挂载了一个智能体外部框架。但我认为，作为模型公司拥有一个智能体外部框架，与真正构建智能体公司的人之间有着巨大区别。我认为他们能做的比我们多得多。

Yeah, that is very true. And we need that. And it's actually nice that there are a few standard places that we all have to, you know, adhere to. It keeps us all honest. I think that's super important to do so. But, yeah, no, I think our goal is to build the world's most capable models. And right now we are focused on the coding agent capabilities — long-horizon work. But what you see with that is that you get a lot for free. I've always said it's a lot easier for us, as we get to state-of-the-art and frontier on coding, to then say, okay, now we're going to obsess over using the model factory to add more data for places that we're not as strong on, like medical or legal, or any other areas. And similarly, I think what we see — and we see this with reasoning models a lot — is if you give models access to the right knowledge sources and they have capable ways of reasoning, they're able to go very well into domains that are less known to them, or even unseen in their training data. So, are we an agent-like model plus harness? No, we're a model company. But I think models today cannot be trained without harnesses. It's not possible. So it kind of is just like where before it was just the weights in the container. Well, now there's an agent harness that's attached to it. But I think there's a big difference in being an agent harness as a model company than someone who's truly building an agent company. I think they can do far more than we can.

是的，明白了。我觉得这是我小小的反驳。如果你真的把自己定位为一家模型公司，那就为 OpenCode 做出最好的模型，对吧？而不是为 Pool 或其他什么。

Yeah, understood. I think that is my minor pushback. If you truly identify as a model company, then make the best model for OpenCode, right? Instead of for Pool or whatever.

嗯，我觉得那并不是——你知道，那是次要的；如果目标是 AGI（通用人工智能），真正要紧的是为 Hermes 做出最好的模型。

Um, I think that that's not — you know, that's minor compared to actually, if the goal is AGI, actually making the best model for Hermes.

对，因为这是编码之后的下一阶段。听着，我们和他们合作得非常紧密，因为我认为——你必须在乎它、投资它。这就是为什么我们做打磨、花时间在上面。而且随着时间推移，是的，你说得对，你得平衡一下，但最终你要的是通用能力，让一切在每个场景里都好用。

Right, because that is the next stage after coding. Look, we are working very closely with them, because I do think it's — and you have to care, you have to invest in it. That's why we do the polishing and we spend time on it. And I think over time, yeah, you're right that you want to kind of balance that out, but ultimately you just want general capabilities that everything works equally in every harness.

呃，就着这个话题——你们和 Hermes、OpenClaw、NanoClaw 之类的合作多吗？

Uh, just on the topic — do you guys do much with like Hermes, OpenClaw, NanoClaw, whatever?

Pi，Pi。不，Pi 是另一回事。

Pi, pi. No, Pi is a different pie.

它更偏编码。

It's more coding.

不过我是 Pi 的忠实粉丝。我得说我觉得它真的很 Pi。你在 pool 上最接近 Pi，在最简曲面这类事情上也最像 Pi。

I'm a big fan of Pi though. I have to say I think it's a really Pi, Pi. You sound closest to Pi in terms of pool, and Pi in terms of, like, the minimal surface.

最简。这是因为我不——我有——请容许我再提一个强烈的看法。

The minimal. It's because I don't — I have — allow me for one more strong opinion.

呃，这个我已经说了两年了。呃，我觉得 MCP 和工具很愚蠢。

Uh, I've been saying this now for two years. Uh, I think MCP and tools are stupid.

哦，这很酷。可是你们支持 MCP。

Ooh, that's cool. Now, you support MCP.

我支持 MCP，我们也支持工具等等。对我来说它们完全说不通。呃，我会稍微解释一下为什么，然后我想我大概能让大家认同这一点。如果你要解决复杂任务——越来越长的时间跨度、越来越复杂的任务，不管是写代码还是别的——你一定会和数据源交互，对吧？你还会和安装在某种虚拟机上的东西交互。而我们现在做的事情，是在这些东西之间加一层。我们在中间加 MCP，在中间加工具调用。这与其说是 MCP，不如说更像是工具调用：模型可以直接写代码并与系统交互。而且我们开始看到，比如 Laguna S 就经常这么做。在前沿模型里你也会看到这一点。它们越来越不再是“我们要往系统提示里塞 50 个工具”，而是“这是一台装了这些二进制的虚拟机，这是你可以操作的代码库，这是你可以写记忆的文件夹”。模型用代码来完成复杂任务。当它用代码时，不是一两次工具调用，也不是三件事串在一起。它实际上会开始用 if 语句、for 循环，让事情变得有条件分支。所以我真的认为我们正在从工具调用——其实已经开始了——转向模型直接编写代码、编写小脚本。你在用 Python 的时候会经常看到这个，你知道……

I support MCP, and we support tools and everything. They make absolutely no sense to me. Uh, and I'll explain a little bit why, and then I think I can probably get people to come along on this one. If you are looking for complex tasks — increasingly longer horizon, increasingly complex tasks, doesn't matter if it's coding or something else — you are going to be interacting with data sources, right? And you're going to be interacting with things that are installed on some form of a virtual machine. And what we are doing is that we're putting a layer in between those things. We're putting MCP in between. We're putting tool calls in between. And this is even more about tool calls than MCP: the model can just write the code and interact with the system. And we're starting to see that like Laguna S does this a lot. You'll see this as well in frontier models. They're increasingly no longer 'here, we're going to stuff 50 tools into the system prompt' — no, 'here's a virtual machine with these binaries installed, this codebase you can operate in, here's a folder where you can write your memory if you want to.' And the model is using code to do complex tasks. And when it uses code, it is not one or two tool calls or three things that are chained together. It actually starts, you know, using if statements and for loops and making things conditional. And so I actually think we're moving — we already are moving — from tool calls to effectively models writing code, little scripts. And you see this a lot when you get the Python, you know...

代码解释器。

Code interpreter.

没错。就像直接——你知道的，写下代码和文件。我不知道你们怎么叫那个 EOF。是的。

Exactly. Like in just the — you know, written code and the file. I don't know what you call the EOF. Yeah.

是的，没错。呃，你已经在更多模型身上看到这种情况了。因为当你开始用强化学习训练它们时，模型想要自由。它们想以最有效的方式做自己想做的事，而不是去调用系统提示里的 50 个工具。所以我非常赞成给模型一个最小的框架——尽可能小——给它一个容器，里面有它自己的代码库，对吧？这个模型代码库能访问 API 密钥、数据源、小库和它需要的文档，然后就让它在任务上自由发挥。嗯，我认为这就是我们要走的方向。我认为 12 个月后，我们不会再看到任何一个塞满 20、30 或 40 个工具的系统提示了。

Yeah. Exactly. Uh, you already see this happening more in models because when you start training them in RL, the models want to be free. They want to be able to do the thing they want to do in the most efficient possible way, and it is not calling one of the 50 tools in the system prompt. And so I'm a very big fan of giving the model a minimal harness — as minimal as possible — giving it a container in which it has its own codebase, right? They've got a model's codebase that has access to the API keys and data sources and little libraries and documentation that it needs, and just let it run free at the task. Um, and I think that is the way we're going. I think we will, in 12 months, not see a single system prompt that is stuffed with 20 or 30 or 40 tools anymore.

我不反对——不不，这我完全没意见。我觉得，呃，这种支持还会长期存在，因为现在很多人都是这么训练的。但也许你们未来在模型里可以不支持它。所以，呃，但是的，我是说，如果你们能做到——我确实觉得写代码更普适，而且它是达到目的的一种手段，为了……

No comment — no, no pushback there. I think, uh, I think there will be — it'll be supported for a long time just because a lot of people are trained on that now. But maybe you guys don't have to support it in your models going forward. So, uh, but yeah, I mean, if you can — I do think that writing code is more generalist, and it's a means to an end for...

而且我们确实支持工具，我们支持——这其实是我们的第一个支持并行工具调用的模型，这也是我们之前需要追赶的。所以那功能是有的，但……这其实是个人的小执念：我希望模型拥有尽可能多的自由度，就，你知道的，自由地去做各种厉害的事。

And we do support tools, and we support — and this is actually the first model we're doing parallel tool calling, in which we needed to catch up. So like that's there, and so it's there, but... it's a personal nitpick: I want the models to have as many degrees of freedom and just, you know, be free and do capable things.

是的。那么，接下来，这就不难理解——好吧，你怎么把你的 Poolide 模型和 Laguna 模型用到我的 Hermes 或我的 OpenClaw 上之类的？通常我关注的是计算机使用或视觉——这非常重要，你们有篇博客文章专门讲这个。但此外，持久性我认为是很有价值的能力，还有长上下文，你们有百万 token 的上下文。还有别的吗？

Yeah. So, and then, so that was on the path towards — okay, how do you use Poolide's models and Laguna models for my Hermes or my OpenClaw, all those things? And so typically what I look for is computer use or vision — that's a very big one, you guys have a blog post on that. But then also the persistence, I think, is very strong value, as well as long context, which you guys have — a million-token context. Anything else?

对我们来说——你看，对我们来说，视觉理解是下一件大事，对吧？我们目前没有视觉理解。

So for us — look, for us, vision understanding is the next thing, right? We don't have vision understanding.

我正要……是那个……

I was going to — is the...

这些模型里还没有视觉理解。

We don't have vision understanding in these models yet.

呃，所以这是我们已经开始投入的事情，我们认为拥有视觉理解非常重要。

Uh, and so this is something that we've started efforts on, like we think it's super important to have visual understanding.

这就是公司愿景。

That's company vision.

呃，所以不，我们在这方面还有工作要做。呃，而这实际上是我喜欢 thinking 模型的一点——我刷了大概两分钟的博客文章——它非常重视多模态，包括音频。

Uh, and so no, we've got work to do there. Uh, and this actually one of the things I loved about the thinking model — like, from the two minutes I scrolled the blog post — very committed to multimodal, including audio.

是的。它们的音频是业界顶尖的。虽然它有万亿参数，音频也是顶尖的，而且这一切都是从头训练的，对吧？可以说没有编码器。

Yeah. They're state-of-the-art audio. As much as it's a trillion-parameter, state-of-the-art audio, but also all trained from scratch, right? No encoder in the sense.

对我来说，这是为什么你需要从头训练的最有力理由之一——你会有一个不同的分词器。你会有不同的……我完全赞同，在这点上我零分歧。就像，呃，直接把模态加进来，不要加——你知道的，保持简单。呃，我们——我觉得我们很长一段时间内不会碰音频。

To me, that's one of the strongest reasons why you need to train from scratch — you would just have a different tokenizer. You'd have different... I'm fully aligned, zero disagreement from me here. Like, uh, just add the modality and don't put — you know, keep it simple. Uh, we're — I don't think we'll touch audio for a very long time.

这也有点写在名字里了，你知道——think，link，think。

It's kind of in the name too, you know — think, link, think.

确实。

True.

嗯，嗯，呃……

Yeah. Yeah. Uh...

为什么——音频难在哪里？

Why so — what's so hard about audio?

不是关于什么——再说一次，一切都归结为专注。

It's not about what's — again, it all comes down to focus.

我明白了。

I see.

对——对事情说不，意味着有研究员和算力可以用在推动通用进步上。而我们的观点是，通用进步来自于推动这些模型拥有强得多的推理能力、长得多的任务跨度。呃，我不认为音频对此有贡献。我不认为它让我们更接近 AGI。我觉得靠近 AGI 时，它是一个必要的模态。我认为视觉理解处于这两者之间。我认为视觉理解绝对能做到这一点，但它也能解锁今天就有价值的能力。呃，所以——但这就是关键，对吧？你需要更多样性。

Right — like saying no to things means that there's a researcher and compute that can go to making general progress. And our view is that general progress is going to come from the ability to push these models to far more capable reasoning, far longer-horizon tasks. Uh, I don't think audio adds to that. I don't think it pushes us close to AGI. I think it is a necessary modality as you get close to AGI. I think visual understanding sits in the middle of those things. I think visual understanding can absolutely do so, but it also unlocks capabilities that are just valuable today. Uh, so — but this is the point, right? You want more diversity.

你想要更多不同的前沿模型公司专注在不同的事情上。我觉得我们就像戴着眼罩的马，只顾着……

You want more different foundation model companies to focus on different things. I think we are just kind of like a horse with blinders on, just like...

是啊，你有你的路线。

Yeah, you have your path.

我们有我们的路线。我们要追赶前沿，不想被其他任何事分心。

We have our path. We want to catch up to the frontier, and we don't want to distract ourselves with anything else.

嗯。

Yeah.

我要指出，其中一个研究分支是深度 OCR——也就是说，你能不能直接丢掉文本的 tokenizer，只用视觉。

I will call out that one of the branches of research is deep OCR—which is, can you just throw away the text tokenizer and just only vision?

我觉得这个——我内心的极客看到这个东西就会想，好吧，你看编码里有多少比特。

I find this—the geek in me looks at this stuff and it's like, okay, look at the number of bits in code.

没错。我觉得这很酷，但我觉得我们最终还是会回到这一点：它可能是可行的，只是它是否足够高效？我想很多这类东西最终都会成功。但问题在于，文本的好处是什么？我之前提到过 Peng 和 Nikolai，他们是应用研究的联席负责人，非常出色。没有他们和整个团队，我们走不到今天。Nikolai 和我多年来一直在争论：推理应该在潜在空间中进行，还是应该在 token 中进行？但有一件事，我觉得他和我真正达成一致——我们三个人都一致——那就是语言非常了不起，因为它是一种极其稠密的方式来编码知识、信息和智能。对吧？你想一想，一篇物理论文，最后变成二三十页——要产生这份 20 页文档所包含的智能、思考和种种东西，这么少的比特里编码了太多东西。其他模态，比如视频和图像，也很棒，但它们没有同样的知识或推理密度，或者说我们试图推动的那些东西，在这个模态中编码的密度没那么高。在很多情况下它们确实存在。你可以在 YouTube 上看一场 50 分钟的精彩讲座，但如果你把它当作视频数据而不是文本数据，那么比特的信噪比、模态的计算效率就低得多。所以我们的观点是，用语言你可以走得很远，而且当你算力有限、人力有限——这两者紧密相关——我觉得我们可以推进语言，它是一项更好的投资。但我想要所有模态。我觉得它超级酷，我也很喜欢 DeepSeek 和其他团队在尝试的东西——我可以一直转发他们。但在内部，我们只是说，保持专注。

Right. I think it's super cool, but I think this is what we're going to come back down to: probably it works. It's just—is it compute efficient enough? I think so many of these things ultimately will work. It's just like, you know, what's the nice thing about text? And I referenced earlier Peng and Nikolai, my two co-heads of applied research, who are just incredible. Like, we wouldn't have gotten here without them and the entire team. And Nikolai and I have been debating for years about, should reasoning be in latent space, should reasoning be in tokens. But one thing that I think he and I really agree on—and all three of us—is that language is incredible because it's such an incredibly dense way to encode knowledge, information, and intelligence. Right? If you think about what went into a physics paper that then becomes 20 or 30 pages—the amount of intelligence and thought and whatnot to then generate that 20-page document, those little amount of bits—there's so much encoded. Other modalities like video and images are amazing, but they don't have the same density of knowledge or reasoning or whatever, the things that we're trying to push for, that are encoded in that modality. They're there in many cases. You can watch an incredible lecture for 50 minutes on YouTube, but if you treat that as video data versus text data, the bits to signal-to-noise ratio, the compute efficiency of the modality, is a lot less. So we kind of have this view that with language you can go really far, but also when you have limited compute, limited people—and they're very much linked—I think we can push language; it's a better investment. But I want all the modalities. I find it super cool, and I love what DeepSeek and others are trying—I can retweet them all the time. But internally we're just like, let's stay focused.

要我说，你看 Anthropic 就能看到这有一定道理。OpenAI 有很多视觉、多模态。Anthropic 就基本上没做，对吧？Fable 在图像处理上是一个很大的进步，但他们并不以多模态公司著称，对吧？作为语言模型编程公司，拥有多模态能力，但从来不是特别炫耀，而且你知道，走得还挺远。

Which I'll say, you know, you can see somewhat works looking at Anthropic. OpenAI has a lot of vision, multimodality. Anthropic kind of just didn't, right? Fable's a big step up in image processing, but like they're not known as the multimodal company, right? The language model coding company that has multimodal capabilities that's never super flexed, and you know, goes pretty far.

你看，在这方面，我觉得 Anthropic——我的意思是，他们做了很多对的事情——但这种近乎偏执地只专注于推动能力、Scaling 模型，我非常同意。这是第一道坎，一旦我们跨过去了，我们就能改进很多其他事情。但与此同时，在另一个极端，看到人们在构建这些空间模型，也就是世界模型，用于非常不同的用例，真的很令人兴奋。但我认为最终这一切会在某个时刻汇合。

Look, in this I think Anthropic—I mean, they've done many things right—but this maniacal focus on just pushing capabilities, scaling up models, is—I couldn't agree more. It's the first hurdle, and once we get that, then we can improve a whole bunch of other things. But at the same time, on the other end of the spectrum, it's really exciting to see people building these spatial models, right, world models, that are being built for very different use cases. But I think ultimately it all comes together at some point.

好，那来说说模型的 Scaling。Laguna S 表示小号——你们的命名很不错：extra small、small、medium、large。

Okay, so scaling models. This is Laguna S for small—you have good naming: extra small, small, medium, large.

还是在 Scaling。

Still scaling.

新的 medium 模型已经开始训练了，它比上一个 medium 大得多。M 是昨天开始训练的，是一个 39 天的预训练运行。

So the new medium started training, and it's much bigger than the last medium. M started training yesterday. It's a 39-day pre-training run.

你怎么提前知道天数？就靠计算机吗？

How do you know the days in advance? Just the computer?

模型工厂，但是——

Model factory, but—

对，而且到了这个阶段，用模型工厂，就好像……我觉得这点很有意思。在 Laguna medium 和 extra small 里，你甚至标明了 GPU 小时数、多少天什么的，针对不同尺寸。我就想，哦，你还可以倒推出那要花多少钱，对吧？什么 GPU、多少小时。

Right, and at this point, with the model factory, like it's... I thought it was interesting. So in the Laguna medium and extra small, you even quoted number of GPU hours for how many days, whatever, for different sizes. And I'm like, oh, you can also work backwards to how much that costs, right? What GPUs, how many hours.

然后你会发现其实并不多。不，确实不多。不是很多钱。而且你知道，我们起步时就像西方的 DeepSeek……我觉得那就是 DeepSeek 时刻，对吧？那一刻人们意识到，你可以在训练运行上花很少的钱，就能训练出能力极强的模型。但我认为这是一种假象。训练运行并不是最贵的部分。训练运行是一个非常平淡无奇的事件。比如，昨天我们 Slack 蹦出一条消息，说新模型开始训练了，这里是链接，你可以跟着看评估。就这些。为那一刻所做的所有工作，有点像人们说的——我对体育一窍不通——但就像运动员说的：全是准备，全是去健身房，而比赛只是比赛。我觉得模型也是这样。

And you realize it's not a lot. No, it's not. It's not a lot of money. And you know, we started as the DeepSeek of the West... and I think that's the DeepSeek moment, right? It was a moment when people realized you can train incredibly capable models for not a lot of money on the training run. But I think that's the falsehood, right? The training run is not the expensive part. The training run is a very anticlimactic event. Like, we just had a Slack message come up yesterday to say the new model is training and here are the links, so you can follow along the evals. And that's it. All the work that goes into that moment is kind of like how people talk—I know nothing about sports—but how athletes talk about: it's all the preparation, it's all going to the gym, and then the game is just a game. I think that's a little bit like with models.

是啊，人们过度看重 DeepSeek 只花了 500 万美元就训练出来这件事，对吧？但在此之前还有大量的研发投入，还要把基础设施建起来。

Yeah, people had overindexed on DeepSeek being trained for $5 million or whatever it was, right? It's like, there's the amount of R&D before that, the infrastructure is built.

或者说更早的那些东西——数据，但不是……所以 Laguna M 在训练，而且是的，会有 L，还会有 XL。你会看到 M，对吧？M 比上一个 M 大得多。所以这些名字有点像是我们的某种版本……

Or older things—the data, but no... So Laguna M is training, and yes, there will be an L, and there will be an XL. And what you'll see with M, right? M is much larger than the last M. So these monikers are a little bit our version of the...

他在嘲笑那些说 small 是 24B 的人。

He was making fun of people for saying small is 24B or something.

不，Mistral 现在的 small 已经超过 100B 了。

No, small for Mistral now is over 100B.

什么？

What?

是啊，我可以做到。我的意思是，我们的 small 是 118，所以我不想再多说什么了。

Yeah, I can pull it off. I mean, our small right is 118, so I don't want to say anything else.

我是说，我觉得也没问题，就叫‘small’……

I mean, I think it's also okay, 'you're small' as...

我们都知道，对于任何前沿模型公司来说，唯一最难的事情就是命名。我也不想说我们很擅长。我是说，这就是 Laguna，你知道，Laguna S 2.1。它……

We all know that the single hardest thing for any foundation model company is naming. I don't want to say that we're good at it either. I mean, it's this is Laguna, you know, Laguna S 2.1. It's...

但至少人们明白，medium 比 small 大。除非你把这事搞砸了——那就有问题了。

But at least people understand, you know, medium is bigger than small. Until you mess that up—then you have a problem.

我们努力，我们非常努力。趁我们还在聊命名——这本来打算放到最后，但不如现在讲了。为什么叫 poolside？为什么叫 Auna？我们创办公司的时候，本来打算叫 Snowball Apps。这个名字来自雪球效应，因为我们预期这家公司会像滚雪球一样发展，而对我们来说，它确实一直像滚雪球。

We try, we try hard. While we're on the topic of naming—this is going to be at the end, but might as well. Why poolside? Why Auna? So when we started the company, it was going to be called Snowball Apps. It was after the snowball effect, because we expected this company to become a snowball effect, and it definitely has been a snowball effect for us.

结果发现那是亚马逊的商标。

Turns out it's an Amazon trademark.

我没开玩笑，我的联合创始人下一个提出的名字是：就叫 Bedrock 吧。

I kid you not that my co-founder's next suggestion of a name was, let's call it Bedrock.

当时我心里想：好吧，如果你在亚马逊，你真的很会给东西起名字。公司成立早期，在我们还没注册的时候，我们参加了一家大型科技公司的年度大会。我们一直在跟他们谈——你要知道，这个时候公司就只有我、联合创始人兼 CEO Margarita，我们知道第一个员工马上要加入，但还没注册——我们在跟这家大科技公司谈一笔类似 OpenAI 和微软的合作：他们给我们大量算力，我们给他们永久使用权等等一堆东西。我们是在那场大会上、在谈一个其实我们根本没资格谈的合同时，发现名字已经被注册商标了——Snow Labs，对吧？我们只是几个还一无所有的小伙子，但这家大公司居然愿意考虑跟我们合作。我们就在他们年度大会的公开场合谈这件事，然后那家公司的首席科学家说：这里人多耳杂，我们换个地方吧，去泳池边的餐厅。不知为什么，我和 Jason 在那一刻对视了一下，说：哦。后来那天晚上，这个词一直留在我们脑子里，我们就说：公司就叫 Poolside 吧。那笔交易最后没有做成，而我们把它当作一个提醒：永远不要低估自己的野心，因为走那条容易的路很简单。我们选择的是困难的路——两个人赤手空拳，甚至都不在硅谷，也不是出自什么知名实验室，就想去融巨额资金。所以大家都以为 Poolside 是因为 AGI（通用人工智能）——大家都在泳池边坐着，这个名字很俏皮，我们喜欢，也有点特别。但实际上，这个名字提醒我们：永远不要低估自己的野心，每当面临选择的时候，就去选那条更难的路。

And so at this point I was like, okay, you are amazing at naming things if you were at Amazon. Early on in the company, before we were incorporated, we were at an annual conference of a very big major tech company. We had been discussing with them — and you have to realize the company at this point is me, my co-founder CEO Margarita, and we know the first person is going to join us, but we haven't incorporated yet — and we were discussing an OpenAI–Microsoft style deal with this big tech company. They were going to provide us with a lot of compute, and we would give them perpetual access and a whole bunch of things. And we found out the name was trademarked — Snow Labs — while we were at that conference and having this discussion that we had no right to have, right? We were a couple of guys who had nothing yet, but this big company was willing to entertain the fact that we might partner with them. We were discussing this at their annual conference in a public setting, and the chief scientist of that company said, "People can hear us here, we should move somewhere else. Let's go to the restaurant poolside." And for some reason, me and Jason looked at each other in that moment and said, "Oh." Then later that night, the name stuck with us, and we said, "Let's call the company Poolside." We never ended up doing that deal, and we used it as a reminder to never round down our ambitions, because that would have been the easy path. The hard path was what we did — starting and trying to raise exorbitant amounts of money when you're just a couple of guys who aren't even based in Silicon Valley and don't come from any of the known labs. Everyone assumes Poolside is because AGI, everyone sits poolside — it was a playful name and we liked it. But actually the name is a reminder for us to never round down our ambitions, and whenever you're faced with those decisions, to just pick the harder path.

是啊，这个故事很棒。我知道你之前讲过，但我想正式记录一下。我第一次见你的时候就是这样的。你让我坐下，我们在某个酒店里，你说：我们要融 5 亿美元。然后你跟我讲了整个愿景，最后你真的做到了。我当时就想：好吧，我其实没什么机会问——你是怎么做到向那些 VC 融到这么多钱的？他们在找什么？对，大概是 AGI，但他们具体想要什么？

Yeah, I mean, that's a great story. I know you told it before, but I just wanted to get it on the record. That's what I did the first time I met you. You sat me down, we were in a hotel somewhere, and you were like, "We're raising 500 million." And then you gave me the whole vision, and then you actually did it. And I was like, well, you know, I don't have that many opportunities to ask: how do you do that kind of raise to those kinds of VCs? What are they looking for? You know, yes, vaguely AGI, but like, what do they want?

世界确实变了。当我们融那轮 5 亿美元的时候，大部分投资人对话都还在努力解释：这些模型不只是随机鹦鹉，而且它们还会继续进化。我亲眼见证了世界从‘OpenAI 会赢下一切，没人能再创业’——我的意思是，Anthropic 当年融 5 亿美元都很艰难，这是公开报道过的；他们很艰难才完成。所以我觉得，我们融那轮的时候，大概一年半前，世界跟今天很不一样。如今，相信 AGI 是真实存在的人数，大概是一个超线性、甚至是指数级的增长。这很重要，因为如果你持有我们三年前、一年半前的那种信念——这项技术将从根本上支撑未来所有有经济价值、有科学意义的东西——那之后的价值函数就很好理解：如果你做到那一步，你就是能建造这种商品的玩家之一。而这么多年下来，建造这种商品已经不只是建模型，还包括建基础设施等等。所以今天，因为相信的人变多了，结果也被验证了，Anthropic 现在惊人的财务成功、OpenAI 和其他公司以及 Google 的增长，都让这不再是‘有没有产品市场契合’的问题。而两三年前，问题的一部分还是‘这些东西能做多大’。你现在跟业内人说这些营收数字，他们会把你笑出房间。

The world's definitely changed. When we were raising that $500 million round, the majority of investor conversations were still trying to explain that these models were not just stochastic parrots and that they were going to keep going. I've seen the world go from "OpenAI is going to win it all and there's no one else who can build a company" — I mean, Anthropic struggled to raise their $500 million round, it's well reported; they pulled it off gladly. So I think when we raised that, about a year and a half ago, the world was very different than it is today. Today, the number of people who believe AGI is real is probably superlinear — definitely some form of exponential function. And this is important, because if you hold the belief we had three years ago and a year and a half ago, and we looked for people who shared that belief — that this technology is going to fundamentally underpin everything that's economically valuable and scientifically interesting for the future — then the value function afterwards is easy to understand: if you get there, you are one of the players who can build this commodity. Over the years, building that commodity has become not just about building models but also about building infrastructure and other things. So today, because the number of people is bigger and the outcomes have been proven right, the incredible financial success that Anthropic is having right now, and the growth OpenAI's had and others and Google, no longer make this a question of whether there's product-market fit. A couple of years ago, part of the question was how big can these things be. If you told people in our industry these revenue numbers right now, they would laugh you out of the room.

现在我觉得，这取决于世界上有多少人相信智能会形成寡头格局，又有多少人相信这个寡头是可以被其他公司打破的。我认为，在相信 AI 的投资人当中，这是最大的分歧点。然后还有一整层人，他们主动把自己淘汰出基础模型公司的阵营，因为他们觉得：你看，我投到那里的钱，跟投到应用层公司的回报没法比。确实很不一样。我相信有非常出色的应用层公司，也应该出现很多。但我确实认为，我们现在还处在早期阶段——现在仍然可以算是早期阶段，决定哪些人能进入最终的赢家名单。在我看来，智能将成为全世界最被需要的商品。它的利润和价格会不断商品化。世界想要选择和选项。所以，认为世界上只会有两三个玩家，我觉得这样的投资人眼光太短浅。我认为，年初持有这种想法的人比现在要多得多。

Now I think it's a function of who in the world believes that there's going to be an oligopoly of intelligence, and who believes that oligopoly can be broken by other companies. And I think that divides investors more than anything else — for the ones who believe in AI. Then you've got a whole layer that's kind of self-selecting out of foundation model companies, because they think, "Look, I can't make the money I put there compared to what I can put in application companies." It's very different. I think there are incredible application companies, and there should be many built. But I do think we are still in a world right now where this is the early innings — it can still be the early innings of who is going to be part of the set of people who win this. Intelligence, in my view, is going to be the world's most demanded commodity. It will commoditize in margin and price. The world wants choice and options. So treating the world as if there's only going to be two players, I think, is very short-sighted from investors. I think that group who thought that was a lot bigger at the beginning of the year than now.

嗯。我觉得过去几个月惊醒了不少人，大家开始觉得：天哪，世界既需要更多智能，也远比想象中复杂。我们应该有多个选择、更多选项、可以被关掉的东西。现在人们对模型施加的限制，我认为是这个问题的另一个侧面，对吧？我们正在进入这样一个世界：模型公司说‘你不能用我来开发基础模型公司’。他们应该被允许这样做。这是资本主义。这是他们的业务。这是他们的劳动成果。但我们居然对此习以为常，这太疯狂了。

Mhm. I think the last couple of months have woken up a lot of people, going, "Holy — the world both can use a lot more intelligence, but also the world is far more complex." We should have multiple choices, more options, things that can be turned off. The restrictions that people put on models now, I think, is another area of this, right? The fact that we are entering into a world where model companies are saying, "You're not allowed to use me for foundation model company development." They should be allowed to do this. It's capitalism. It's their business. It's their work product. But it is insane. It is wild that we are okay with that.

你更反感 Anthropic 这么说，还是白宫这么说？你知道你是在挑……

Do you have more problem with Anthropic saying it or the White House saying it? You know that you're picking...

两种不同的限制和约束。好吧，我换个方式说。

Two different limitations and restrictions there. Look, I think I'll put it this way.

我认为，随着这项技术变得越发强大——无论好坏——我们确实需要越来越多地交由民主来解决这些问题。我认为任何单一公司单方面做决定都是危险的。这是权力集中在极少数人手中，制衡非常有限，这在历史上从未有过好的结果，无论以何种形式。这不是对现有基础模型公司的批评，而更多地是我希望世界变成什么样子的一种评论。在一个技术能力不断增强的世界里，政府需要发挥积极作用，确定真正的滥用风险在哪里。而且我确实认为，我们需要把安全区分为滥用风险和世界末日场景，后者是否会发生谁也不知道。非常实际地讲，我很高兴看到政府层面现在又开始进行大量对话，试图解决这个问题。最终决定是什么，也许我满意，也许不满意；也许我同意，也许不同意。但这在很大程度上就是民主，对吧？在任何一个时刻，我可能不会对某一方面完全满意，但人们选择了投票选出某个人来做这些决定。所以从长远看，在 20 年的时间跨度里，世界大致会朝着正确的方向前进，民主也确实在起作用。有句名言——这是所有制度中最糟糕的，诸如此类。

I think as this technology gets more capable—for better and worse—we do want to yield to democracy to figure this out more and more. I think any single company making unilateral decisions is dangerous. It's a concentration of power in a small number of people with very limited checks and balances, and that has never worked out well in history in any way, shape, or form. This is not a criticism of existing foundation model companies. It's more commentary on how I'd like the world to be. In a world where the technology gets more capable, government needs to play an active role in determining where there are real risks of misuse. And I do think we need to separate safety between misuse and doomsday scenarios, which no one knows if they're going to happen or not. Very practically, I'm glad to see a lot of conversation starting to happen again at the government level to try to figure this out. Whatever the final decisions are, maybe I'm happy about them, maybe I don't. Maybe I agree, maybe not. But that's kind of democracy, right? At any given moment I might not be perfectly happy with one or the other, but people chose to vote in someone to make those decisions. Over a 20-year time span, the world directionally goes correct, and democracy does work. There's that famous quote—it's the worst of all systems, or something like that.

这是最糟糕的组织形式，除了我们试过的其他所有形式以外。

It's the worst form of organization except for all the others that we've tried.

没错。正是——

Exactly. That's—

你总能指望我引用丘吉尔——我可是研究过不少丘吉尔。

You can always count on me for a Churchill quote—I've studied Churchill a lot.

我喜欢这句话。这就是我现在的期望。我确实认为我们处于一个关键的历史时刻，所以任何人发言都很重要。考虑创办自己基础模型公司的研究者，想要发表意见并大声说出来的人——无论是向他们的代表还是就在 X 上——都应该这样做。但具体到你的问题，我认为我们目前的能力水平还远没有到应该以任何方式限制开放模型的地步。如果我们那样做，会伤害创新。

I love that. That's what I hope for now. I do think we're in a critical moment of time, so speaking up for anyone is important. Researchers thinking about starting their own foundation model companies, people who want to share their opinion and be vocal—whether that's with their representatives or just out on X—should do so. But concretely to your point, I think we are not at a level of capability right now where we should start restricting open models in any way, shape, or form. I think it will hurt innovation if we do.

有没有一个节点会让你改变这一观点？

Is there a point at which you will change your opinion there?

有，而且必须有。你不可能板着脸说，这东西可以永远以任何形式开放。这与说反话——现在一切都必须关闭——同样荒谬。走两个极端中的一个，我们就会出错。

Yes, and there has to be. You cannot, with a straight face, say this can be open forever in every way, shape, or form. It is just as ridiculous as saying the opposite—that everything needs to be closed down right now. At either end of the extreme spectrum is where we go wrong.

对，而且在社会中，无论如何都是如此。

Right, and in society in any way, shape, or form.

所以答案总是更微妙的，从来不是非黑即白。我认为当我们遇到现实场景，必须说“嘿，我们要更加小心”时，我们需要重新评估。那可能意味着以不同的方式训练模型、开放它、提供不同版本，或者限制某些东西——这完全可以，因为我不认为任何人应该不负责任。但我特别想指出的是，自 GPT-2 以来，人们就一直在呼吁担心这些模型被滥用，对吧？我还记得当时有人说我们不能发布 GPT-2，因为整个世界会陷入集体歇斯底里。所以这不是对 Dario 的评论，而只是对这个领域整体的评论。迄今为止，我们在这一方面做得并不好，我们需要做得更好。而且我确实认为，安全研究所和更好的评测所做的努力可能是正确的方向。

And so the answer is always more nuanced; it's never black and white. I think as we encounter real-world scenarios where we have to say, “Hey, we have to be more careful,” we need to reevaluate. That may mean training a model differently, opening it up, having different versions, or restricting some things—totally okay, because I don't think anyone should be irresponsible. What I do want to call out is that people have been calling for the fear of misuse of these models since GPT-2, right? I still remember the claim that we couldn't release GPT-2 because the whole world would get mass hysteria. And so this is not a commentary on Dario, it's a commentary just in general in the space. And so we have not been very good at this so far, and we need to get better at it. And I do think that the work happening with safety institutes and better evals is probably the right direction.

是的。我想为这个说法辩护一下。宁可过度谨慎，之后再往回退，也比反过来好，因为反过来是单向决定。

Yeah. I want to say something in defense of this. It's better to err on the side of safety and then roll it back rather than the other way around, because the other way is a one-way decision.

我觉得这是对的。

I think that's true.

但这里还有个问题，就是竞争，对吧？并不是全世界都会选择过度谨慎。你在说——

The caveat there is also the competition, right? You don't have the whole world erring on the side of safety. You're talking—

是的，完全如此。所以你不可能单独搞安全，因为别人只会比你更不安全。

Yeah, exactly. So you don't get to do unilateral safety, because someone else will just be more unsafe than you.

对，完全如此。你可以在这里暂停创新，但这并不意味着创新真的会停住。

Yeah, exactly. You can pause innovation here. It doesn't mean it's actually pausing.

所以这个问题很复杂，对吧？我认为，与其用非黑即白的方式讨论模型是否可用，不如去讨论我们能够共同认可、并在国际上一致同意需要限制或不提供的某些具体能力。一旦你开始做出这些笼统的声明，你就会面临这样的风险——我总想起当年禁止香烟广告的事。那本来是好事，我不是反对，但它实际上造就了烟草公司的寡头垄断，因为其他任何人都无法竞争了。那可能是烟草行业有史以来最好的时刻。我们现在不想重蹈覆辙。如果我们在创新背后竖起高墙——这本身是自私的说法，因为我还不处于前沿——但这也与我无关，而是关系到这个领域的每个人。你正在 2026 年此刻，基于当前模型的能力，断定只有两三家能够构建它。这在我看来，像是我能读到的最反乌托邦科幻小说的第 14 章。因为从那里开始，你可以推演出世界上发生的所有场景，而这些场景中没有一个让我对未来感到兴奋。我认为这是我们都应该思考的问题：我们想要什么样的未来？我们想拥有什么？我觉得那是一个智能成为商品、人人都能获取的未来。它会变得越来越便宜，对吧？这很重要。它能影响世界上更多的人。而不是只有一个公司在输出上动手脚，或者随时开关它。我认为在这个领域里，比美国政府权力更大的实体就是 Nvidia，因为基本上谁拿到分配额度，谁就拿到了算力。你可以一路追溯到 TSMC，或者——

So, they're complex, right? And I think we are much better off talking about certain capabilities that we can commonly agree on—and internationally agree on—that we want to limit or not have available, than we should talk about it in black and white of models available yes or no. The moment you start making these big blanket statements, you start getting into the risk of—I always think back to when we banned advertising on cigarettes. Good thing. I'm not saying I'm against it, but it effectively established an oligopoly of cigarette companies because no one else could ever compete. It was probably the best moment for the tobacco industry that ever happened. And we don't want to do that right now. If we pull up walls behind innovation—and this is a self-serving comment because I'm not at the frontier yet—but it's not just related to me, it's related to everyone in the space. You are deciding right now in 2026, based on the current capabilities of models, that this is something only two or three companies can build. That reads like chapter 14 of the most dystopian sci-fi novel I could read. Because from there, you can play out all the scenarios that happen in the world, and none of those are the ones that make me excited about the future. And I think that's the thing we should all think about: what's the future we want to be excited about? What do we want to have? I think that's a future where intelligence is a commodity—everyone can access it. It becomes cheaper and cheaper, right? And that's important. It can impact more of the world. It's not one where a single company puts its thumb on the scale of what it outputs, or turns it on or off. I think the one entity that has more power than the US government here is Nvidia, because basically whoever gets the allocations gets the compute. You can take it down to TSMC or—

再往下就是 TSMC。

TSMC below that.

但我就是想抛出一些挑衅性的说法，看看你会有什么回应。

But I just want to test provocative statements to see if you have any response.

那个我得想一想。

I need to think on that one.

其实我觉得它们是被监管的，对吧？比如你可以看到政府——

Which actually I think they are regulated, right? Like you can see the government—

被监管。

Regulated.

它们能向中国发货吗？

Can they ship to China?

好吧。但你知道，它们不是中国。

Okay. But you know, they're not China.

听着，我认为这个行业之所以存在，是因为 Nvidia 的贡献。

Look, I think this industry has existed because of what Nvidia has done.

没错。

Yeah.

对吧。

Right.

我知道大家爱拿他开涮，但我也想说说：我记得 2015 年我们在 Karpathy 那篇文章之后创办了 Source{d}。后来的进步之所以能发生，正是因为我们能把消费级 GPU 放进服务器，而他们允许我们这么做。然后这条路就一直往前走。所以说，基础模型和它们的硬件、系统是紧密绑在一起的。为什么我们看到的是这种阶梯式的进步？因为它们来自下一代网络和系统的不断涌现，对吧？你在 Hopper 上能训出的模型，和你在 GB300 上能训出的模型，差别就是一个万亿参数与五六万亿参数的差别。所以我觉得这些东西真的是紧密共生的。而且我觉得，未来更有意思的问题是：当我们开始把这些东西做更深入的协同设计时，我们能用模型能力解锁什么？我们已经在下一代系统上看到这一点了。还有，我觉得这个世界——你知道，资本主义真的很擅长推动那些能带来更多竞争的东西，对吧？而 Nvidia 就是在促进竞争。不是别的——但如果政府通过监管说“别人都不能有效地做基础模型”，那就完全是另一回事了。去造一个 Nvidia 难不难？当然难。去造一个基础模型难不难？我觉得也非常难。但我们应该把赛场建成这样：如果明天有人醒来想去做，他们可以去做，而且他们可以使用相应的工具。我还觉得，在关于模型公司的讨论和关于芯片公司的讨论之间，仍然存在着很大的差别。

I know people like to give him flak, but I also want to say: I remember when we started Source{d} back in 2015, after that Karpathy article. Progress was able to happen because we were able to put consumer GPUs in servers, and they allowed us to do so. And then it kept going further. So foundation models are so closely linked to their hardware and their systems. Why do we see these stepwise progressions happening? We see them because of the next generation of networking and systems that come out, right? The difference between a model you could train on Hopper versus GB300s is the difference between a trillion-parameter model and a five- or six-trillion-parameter model. So these things really coexist very closely with each other, I think. And I think the more interesting question for the future is going to be: how do we — what can we unlock in terms of model capabilities as we start co-designing these things even more? And we're seeing that with the next generation of systems. And I think the world — you know, capitalism does a really good job of pushing toward things that allow for more competition, right? And Nvidia allows for competition. It's not — you know, but if a government says no one else can build foundation models effectively through regulation, that is very different. Is it hard to go build an Nvidia? Absolutely. Is it hard to build a foundation model? I think it's very hard to build a foundation model. But we should make the playing field one where, if someone wakes up tomorrow and wants to do so, they're allowed to do so, and they're allowed to use the tools to do so. And I think there's still a big difference between what we're seeing in the discussions about model companies versus what we're seeing with chip companies.

差距似乎还在于监管者的专业能力，对吧？政府里谁来判定什么太安全、太聪明、太危险？我们既然在抛这些辛辣的问题，你脑海里有没有什么现在就该改变的事？比如，OpenAI 和 Anthropic 应该开源他们的模型吗？是开放权重吗？还是说我们在强化学习里做的事情决定了你的安全屏障？有没有什么方面应该去改变？就——

The gap also seems to be the expertise in who regulates it, right? Who in the government decides what's too safe, too smart, too dangerous? But while we're throwing spicy questions out there, do you have anything that comes to top of mind that could be changed? So — should OpenAI, Anthropic open source their models? Is it open weights? Is it what we do in RL that determines your safety barriers? Is there anything that should be done there? Just —

嗯，是的。让我兴奋的一件事——我觉得我们谈得越来越多，但还没有人真正在做的——就是在强化学习训练里做硬件混搭，对吧？你想想这个概念，我们在推理里已经看到了：预填充和解码，用通用 GPU 配上一个更专用的芯片效果更好，对吧？比如 Groq 芯片、LPU 和 GPU 组合，行业里还有各种不同版本。而强化学习是受批次大小限制的，对吧？你最终会被批次大小卡住，因为你没有无穷无尽的任务。当你拥有整个互联网时，你可以更灵活地扩大批次大小，因为你有整个互联网的数据。但对强化学习来说，你只有几百万个要训的任务。所以你不能把批次大小大幅吹大，这意味着你在强化学习里实际上没法像预训练那样把算力扩展到同一个水平。所以任何能改善这一点的事情，我都非常兴奋。我觉得最好的改进方向之一，就是我们在推理中已经开始看到的东西：把预填充和解码拆分到不同芯片上，并把这种做法引入强化学习，对吧？我觉得我们很快就会做到。而且我认为应该有更多人做这个。因为这样一来，我们突然就能在墙钟时间上大幅提升训练大语言模型的效率。再回到“这是一场竞赛”的事实，对吧？竞赛不是看你有多少 GPU，而是看日历时间。这可能是我们现在加速整个行业所能做的最大影响之一。所以这是我在技术上特别爱深入研究、也爱跟人聊的一件事。

Um, yes. One of the things I'm excited about — that I think we're talking about more and more, but no one is doing yet — is mix-and-match hardware during RL training, right? You think about the notion, and we're seeing this in inference, right? Prefill and decode just work better with a general-purpose GPU and a more specialized chip, right? Like the Groq chip, the LPU, and the GPU combined, and there are different versions of that in the industry. And RL is batch-size constrained, right? You are ultimately batch-size constrained because you don't have infinite tasks. When you've got the entire web, you can be much more flexible in scaling up your batch size because you have the entire web. But for RL, you have, you know, X million tasks that you're going to be training on. So you can't massively blow up your batch size, which means you actually can't scale compute with RL to the same extent you could with, like, pre-training. And so I'm very excited about anything that improves that. And I think one of the best ways to start improving that is the things we're already starting to see in inference: separating prefill and decode onto different chips, and bringing that to reinforcement learning, right? And I think we'll be there soon. And I think more people should be working on this. Because then all of a sudden we're able to be way more efficient in how we train our LLMs in terms of wall-clock time. Again, coming back to the fact that it's a race, right? The race is measured not in how many GPUs, but in calendar time. And that's probably one of the biggest impacts we can have right now to speed up our industry. And so that's one — technically, I love geeking out about it and talking to people.

是的，我想和 Edged 聊聊。我参观过他们的数据中心，你能亲眼看到预填充/解码分离在数据中心里是怎么布局的。而且要做到这一点，你必须拥有自己的硬件。

Yeah, I would talk to Edged. I had a tour of their data center, and you can physically see how PD disaggregation is mapped out in the data center. And you have to own your own hardware to do that.

是的。不，你看，我觉得这个领域里更多的创新就是最酷的事。我之所以兴奋，是因为——坦率地讲，这就是为什么我们所有人都会问：我们为什么要在发布前两周才跑完这个模型的后训练？不对，抱歉——我是说从预训练之后，中间的训练、SFT，再到最终发布，这整个过程所花的时间。现在我最大的墙钟时间瓶颈就是强化学习时间，对吧？就因为我没法进一步扩展，我没法给它加更多 GPU，因为批次大小的限制。最近刚有一篇很酷的博客文章，展示了用比我们所有人做的都更低的精度来做强化学习。我觉得这真的很酷。所以——今天是几号？现在 7 月 15 日。那篇文章是 5 天前出来的。我还觉得非常酷。我认为低精度强化学习——同时还要保持稳定——我们现在还在用 FP8，所以看到他们分享并发布这个工作，我很兴奋。一旦我们迁移到 Blackwell GPU，这绝对是我很想尝试的事。

Yeah. No, look, I think more innovation in this space is just the coolest thing. And I'm excited because, frankly, that's why all of us are like: why did we finish post-training this model, whatever, two weeks before release? No, sorry — between pre-training, then mid-training SFT, and then the time it takes for release. My biggest wall-clock bottleneck right now is RL time, right? And it's just because I can't scale it up further — I can't add more GPUs to it because of that batch-size constraint. There's a really cool blog post that just came out showing RL done in even lower precision than any of us are doing. I thought this was really cool. So — what date is it today? We're on July 15. So this came out five days ago. And I thought this was very cool. I think lower-precision RL while keeping it stable — we're still doing this in FP8 — so I was excited to see them sharing this work and putting it out. It's definitely something I'm excited to be doing once we move to Blackwell GPUs.

但没错，开放研究最酷的地方就是有取有予。

But yeah, the cool part of open research is you take and you give.

没错，是的。我顺便提一句，有一篇论文对量化级别做了调研，他们大致得出结论：4-bit 是最佳点。

Exactly, yeah. I'll just quickly mention there was a paper that did a survey on levels of quantization, and they roughly concluded that 4-bit was the sweet spot.

但我不记得了，这难道不是几年之前的事吗？我总觉得好像是今年。

But I don't remember — this was a couple of years ago, right? I think I remember it was this year.

那篇文章是一年前发的。

It came out one year ago.

但我会想，好吧，也许 NVFP4 就是极致了。你没法再低了——最低也就到三值（ternary）。就这样，选项并没有那么多。

But I'm like, okay, maybe NVFP4 is it. You can't really — the lowest you can go is ternary. That's it. Like, there aren't that many.

嗯，我的意思是，NVFP4 和 4-bit 之间还是有相当大的差别，对吧？就能力上限而言。但我觉得 NVFP4 就其本身来说是被低估了。它出来的时候我就挺兴奋——正好是在取值范围和……之间的额外权衡，嗯，是的。

Well, I mean, there's still quite a difference between NVFP4 and 4-bit, right? In terms of what's possible. But I think NVFP4 is, you know, underrated in terms of what it is. I'm quite excited that when it came out — it's just getting that extra trade-off between, um, between range and — yeah.

非常酷。

It's very cool.

嗯，还有几个收尾问题。

Uh, a couple of closing questions.

我有个简短的问题。

I have a quick question.

好，快问，回到技术那边。关于从 XS 2.1 Medium 到训练新的小型模型，有没有什么大的心得？——就是从训练模型的通用角度来看。

Okay, quick question, back to the technical side. So any big takeaways from XS 2.1 Medium to training the new small — just generally in terms of training models?

你在前面讨论里提到很多次：预训练中有很多可以压榨的空间，对吧？你可以从互联网上学到更多。同时，你把 30B 扩展到了 120B。那有没有一个“多小才算太小”的门槛？我就先随便说几句，最后再问问题。你知道，Karpathy 的论点是认知核心，对吧？我们看到很多小推理模型——Nano、3B、4B——能推理很久。然后思路是把工作卸载给另一个模型，这些都是小型推理模型。所以你有没有在小尺寸上发现什么有趣的东西，比如设备端 20B、30B，单 GPU 上 100B？能压出更多吗？

You mentioned a lot in the earlier discussion that in pre-training there's a lot you can squeeze out — you can learn a lot more from the web. At the same time, you took 30B and scaled it up to 120B. Is there any gating on how small is too small? I'm going to ramble for a bit; I'll come to a question at the end. Part of Karpathy's thesis was the cognitive core, right? We've seen small reasoning models — Nano, 3B, 4B — that reason a lot. Then the idea is you offload to a different model for the work; these are small reasoning models. So have you found anything interesting in model sizes like 20B, 30B on device, 100B on a single GPU? Can you squeeze out more?

还能压榨出很多。我不想做太多过度承诺，但我认为在 XS 尺寸上我们也能压出更多。我们在 S 训练中学到了很多，这能让我们进一步改进类似 XS 的尺寸。而且从那时起，我们已经学到了一些本可以让 S 变得更好的东西。我认为在小得多的模型上，我们还有更多空间可以压榨。我不觉得这是反对 Scaling（规模扩张）的理由，只是……顺便说一句，我觉得这很有意思：把一个小模型的后训练配方直接套到大模型上，其实并没有太大帮助。

There's a lot more to squeeze out. I don't want to make too many forward promises, but I think we can squeeze a lot more out of the XS size as well. We learned a lot during S training that will allow us to improve XS-like sizes even further. Already since then, we have learned things that could have made S even better. I think there is a lot more still for our space to squeeze out of models much smaller. I don't think that's an argument against scaling; it's just... And by the way, I think this is a nice thing: it's really not very helpful to have a post-training recipe for a smaller model and try to apply it to a bigger model.

对，不管哪种情况，你都得重新思考大部分配方。但把一个更大模型的后训练配方用在较小模型上，几乎总是一个很好的改进和基线。你还可以继续调整，但我不觉得这是理所当然的。所以一旦你让更大的模型变得更好，就常常有一个快速杠杆，能再次改进你的小模型。但我们还能从小模型里压出更多吗？Laguna S 给了我很大信心，我觉得可以。我想这和我们之前讨论的有关：你要改进的是行为，而不一定是原始智能。这体现在所有维度上——有一个维度是模型能推理多久，它能坚持多久——还有效率，对吧？你理想中两个都要推进。

Yeah, just in all cases you're going to have to rethink most of the recipe. But a post-training recipe for a bigger model applied to a smaller model is almost always just a really good improvement and baseline. You can still tweak it more, but I don't think that's necessarily obvious. And so once you make your bigger models better, you often have a quick lever to quickly improve your smaller models again. But will we be able to squeeze a lot more out of smaller models? Laguna S gave me a lot of confidence that I think we can. I think it's around that discussion we had earlier: it's about the behaviors, not necessarily the raw intelligence, that you're trying to improve the models for. And that's on all axes — there's an axis of how long a model will reason, so how long can it stay on task — and there's also efficiency, right? You want to ideally push on both.

还有一点想确认：你们现在没有做的、而我们在前沿实验室看到的是蒸馏，对吧？你们有一个不会真正提供给用户的大模型，而推理时部署的是从那个大模型蒸馏出来的版本，这能带来不少收益，对吧？

And the thing to clarify: you guys aren't doing right now, which we do see at frontier labs, is the distillation, right? You have a big model that you don't really ship to users, and what you put out for inference is typically distilled from that, which gets you quite a bit of gains, right?

你看，我们现在不做蒸馏，是因为我们构建这些模型的原因。这些模型对我们来说是我们研究路径的一部分。比如，Laguna Medium 比我们发布的最新两个模型——这一个和上一个——要大得多。过去我们还训练过更大的模型。所以，更大的模型有工程层面的考量，在每一个量级的尺度上，你都会在预训练中学到关于稳定性的新东西。但在较小的模型尺寸上，你可以在内部研究上迭代得快很多。所以对我们来说，蒸馏到更小的模型其实并不能达到目的。这些模型用词不太准确，但对我们来说是双重用途的模型：既让我们看到在模型工厂里是否取得了进步，也是可以发布到世界上的东西。所以我们才不蒸馏。我们做过蒸馏实验，确实有很多很酷的事情可以做。如果你有大量用户数据，那还能走得更远。但我觉得，以很快的节奏端到端地从零训练模型，是很有价值的，这样作为研究组织就能学到经验，而不是一直等待。这是我们多年来学到的最重要教训之一。以前我们两次模型训练之间的间隔很长，比如六个月，我们就训练一个大模型，等六个月，再训练一个更大的模型。你会把太多改进叠加在一起，到训练下一个模型时，它就像一锅汤，你根本不知道哪些成分带来了最终结果。所以，当你更频繁地训练模型——这在后训练和从头开始的预训练中都成立——你就能更好地理解是什么带来了改进。我认为这很重要。归根结底，我们仍然没有真正的大语言模型深度学习科学。但我们都在试图从实验中获得洞见，因为这些洞见会带来缩放定律，而缩放定律会带来改进，让我们更省算力、获得更多能力。

Look, I think it's something we don't do right now because of why we're building these models. These models are, for us, part of our research path. For example, Laguna Medium was much larger than the last two models we've released — this one and the last one. And we've trained even bigger models in the past. So there's an engineering component to a bigger model, and at every order of magnitude size, you'll learn new things in pre-training about stability. But at smaller model sizes, you're able to iterate a lot quicker internally on your research. So for us, distilling down to a smaller model doesn't actually serve the purpose. These models are — it's not the right term — but to us they're dual-purpose models. They are progress for us to see if we improved in the model factory, and something to put out into the world. That's why we don't do it. We've done distillation experiments, and there are really cool things you can do. If you have lots of user data, then you can go even further. But I think there's something to be said for having a quick cadence of models trained end to end from scratch, so that you as a research organization can learn the lessons and not wait. That was one of the big lessons we learned over the years. When we used to have a much longer cadence between model trainings — six months — we would train a big model, wait six months, train another bigger model. You'd be compounding so many changes and improvements that by the time you're training your next model, it's a bit of a soup, and you don't really know what ingredients led to the outcomes. So when you train models far more frequently — and this holds true for both post-training and pre-training from scratch — you're much better able to understand what led to the improvements. I think that's important. Ultimately, there is still no true science of deep learning for large language models. But we are all trying to gain insights from our experiments, because it's those insights that lead to scaling laws, and scaling laws lead to improvements that let us be more compute efficient and get more capabilities.

对。太棒了。我本来想用一点历史来收尾。你曾经研究过工程团队生产力的指标。你今天怎么看待工程团队生产力？

Yeah. Amazing. I was going to end off with a little bit more history. You spent some time looking at metrics for engineering team productivity. How do you think about engineering team productivity today?

我觉得这太疯狂了。这简直是黄金时代。你可以只凭一个想法，等一夜让智能体去干活，就能把它做出来。我不知道。对我来说……

I mean, it's wild. It's like the golden age. The fact that you can just take an idea and build something by waiting overnight for an agent to do the work. I don't know. To me...

那你怎么衡量呢？你知道，因为从字面上看……你看——

Like how do you measure on — you know, because literally in a — you look —

我觉得这是个好问题。这个问题我很久没想过了。

I think it's a good question; it's one I haven't thought about in a long time.

但你很有资格回答这个问题。

But you know, you're pretty qualified to do it.

我是要说——不，你说得对。让我想一下。你看，归根结底，代码是什么？软件是什么？工程是什么？就是把对最终用户或一组最终用户有价值的东西——一个想法、一个额外的 bug 修复、一个功能——变成真正交付那个价值。我认为，随着这些模型变得越来越有能力，我们正在大规模削减中间环节，压缩交付价值所需的时间。最终，对任何初创公司或任何公司来说，正是这个迭代周期让你获胜，对吧？如果你能在两小时内解决一个 bug，而不是让它躺在 backlog 里三个星期……如果你能在客户电话里了解到“嘿，如果有这个功能，他们会愿意付更多钱，这对他们更有价值”，然后你在一周内上线，而不是一个月内上线……所以我觉得，也许我们多年前在 LLM 出现之前看的那些东西仍然适用。本质上就是周期时间的概念，但在这种情况下，它指的是从你有了一个想为某人做的有价值的事情，到它真正交付给他们的前置时间。

I'm going to — no, it's a fair point. Let me take a second to think about it. Look, ultimately, what is code, what is software, what is engineering? It's to go from something that is valuable for an end user or sets of end users — an idea, an extra bug fix, a feature — to delivering that value. And I think what we're doing with these models becoming more capable is that we are massively cutting out middlemen and compressing the time it takes to deliver that value. Ultimately, that iteration cycle for any startup or any company is what allows you to win, right? If you're able to solve a bug in two hours, versus having it sit in the backlog for three weeks... If you're able to be on a customer call and learn, hey, if this feature existed, you know, they'd be willing to pay more, and it's more valuable to them, and you ship it in a week instead of a month... So I think ultimately maybe the same things we looked at years ago pre-LLM still apply. It's just the notion of cycle time, but in this case, it's the lead time from the moment you have a valuable thing that you're looking to do for someone, to the moment it's actually shipped to them.

其他所有指标最终都是那个滞后指标的领先指标，对吧？无论你看的是代码量、Pull Request、评审还是其他这些东西，都不重要。所以我认为在这种情况下，我们在某些方面开始进展得如此之快，以至于我们可以直接坐下来，看看传统意义上的滞后指标——也就是从一张工单到最终交付结果的周期。在这个新世界里，我会关注的、可能我们以前没想过的问题是：一个人能借此做出多少东西，对吧？最重要的一点是：如果你去看 AI 原生公司，它们的组织方式并不像上一个时代的工程组织。它们往往就是由开发者本人组成，尽量贴近客户，贴近交付能力。中间不一定要有一大堆团队。我觉得这很令人兴奋——一个组织里，单个个人贡献者就能非常接近这个目标。所以我会看从价值被识别到最终交付之间，有多少人参与其中。你会希望参与的人更少，端到端的时间更短。

Every other metric is ultimately a leading indicator for that lagging indicator, right? It doesn't matter if you're looking at amounts of code, pull requests, reviews, all those kinds of things. And so I think in this case we are starting to move so quickly on some of these things that we can just sit back and look at what was traditionally the lagging indicator, which is the lead time from a ticket to an end result. What I would look at in this new world, that maybe we didn't think about before, is how much can a single person do with that, right? One of the most important things: if you look at AI native companies, they're not designed like the engineering orgs of a previous age. They're actually designed with often just the builder, right? And as close to the customer as possible, right up to the ability to ship. There isn't necessarily a huge team in between that sits there. And I think that is exciting — organizations where a single individual contributor can just get much closer to that. So I would look at from where the value is identified to the moment it's shipped, and how many people are involved in that. You want the number of people involved to be less, and you want the time end to end to be shorter.

好。那你在面试别人的时候，有没有什么方式可以评估这一点？

Okay. Is there a way to evaluate when you're interviewing somebody?

哦，嗯，因为那个，你知道，我看的是最精简的版本。我觉得对此最常见的答案是——主动性。

Oh, uh, because that is, you know, I look at the most compressed version. I think the common answer to this is agency.

对，那一个人有多少主动性呢？

Yeah, how much agency does a person have?

嗯，我觉得在 AI 能力越来越强的时代，主动性大概会成为对任何人都最重要的特质之一。而且主动性是你可以在别人过去做过的事情里找到的东西，因为主动性这种东西，如果你有，你一定会展现出来，对吧？没有人会拥有主动性却坐在那里不用它。它的定义就在于它会得到实践。所以你要去理解：人们在生活里，在职业和个人项目中，做过哪些能体现主动性的事。你的个人成长经历就展现了惊人的主动性，对吧？我觉得归根结底——这就是过去一年半左右硅谷的那句话：你就是可以去做事。这就是你要找的东西。

Uh, I think in the age of AI getting more capable, agency becomes probably one of the most important qualities for anyone. And I think agency is something you can look for in what people have done in the past, because agency is something that if you have it, you are demonstrating it, right? No one has just agency and is sitting back not exercising it. The whole definition of it is that it's exercised. And so understanding what were things that people did in their lives, in their professional and personal projects, that showed agency. Your personal backstory shows a ridiculous amount of agency, right? I think that is ultimately it — it's the Silicon Valley quote of the last year and a half or so: you can just do things. That's what you're looking for.

但我觉得，要让高主动性的人保持一致很难，因为他们都想走自己的路，这正是关键，对吧？

I think then aligning high-agency people is very hard, because they all want to go their own way, that's the whole point, right?

对，但我认为，在一个组织里，一个好的领导者的概念是能把人们汇聚到一个共同的结果上。而且我觉得对于任何高主动性的人，你要做的是——我很幸运，我带的是一个有极高主动性员工的组织。我是说，我不是构建模型的那个人，对吧？这一点我怎么强调都不为过。是团队实现了这一切，而且这个团队主动性极强。所以如果你去看，要把这些人凝聚在一起需要什么，归根结底是一个共同目标和一套共同的边界。因为如果你放任人们想做任何事，你就会变成一个探索算法——这就是我们在大型科技公司里看到的情况，对吧？在大型科技公司的研究部门，一切都是探索算法。每个人都可以做任何事，然后就变成关于争夺资源的政治。所以当你说，这是我们的共同目标，这些是我们设定的边界——对，我们不做多模态，我们专注于强化学习，我们做这些事情——而且你在人们加入公司之前就坦率地告诉他们。你知道，你会得到很多主动性。你可以去你想去的地方，但这些才是合理的赛道。

Yeah, but I think the notion of a good leader in an organization is to be able to bring people together around a common outcome. And I think what you want to do with anyone who's high agency — I feel very lucky I've got an organization with incredibly high agency people. Like, I mean, I'm not the one who built the model, right? I cannot stress this enough. It's the team that achieved this, and it's a team that is incredibly high agency. And so if you look at what it takes to bring that together, it's ultimately a common goal and a common set of boundaries. Because if you allow people to just go and do everything, you become an exploration algorithm — and this is what we see in big tech, right? In research in big tech, everything is an exploration algorithm. Everyone can do anything, and then it becomes political about gathering resources. So when you say, this is our common goal and these are the boundaries that we've set — right, we're not multimodal, we focus on RL, we do these things — and you're upfront with people before they join the company. You know, you get a lot of agency. You can run where you want, but these are the lanes that make sense.

我觉得这其实能让人发挥出最好的状态，因为……

I think it actually gets the best out of people, because...

就像创新来自于约束。我们是用相对很少的算力和相对很少的钱做到这件事的，和外面一些其他团队相比。我最近回想了很多，觉得这其实是件好事，因为那些约束在某种程度上迫使我们在其他一些维度上变得更好，而别人可能没有这些。对吧？我们购买的外部数据相对很少。

Like innovation comes from constraints. We did this with relatively little compute and relatively little money compared to some of the others out there. And I've thought back on that quite a bit recently and thought actually it was a good thing, because those constraints kind of forced us to become much better on certain other axes that others might not have. Right? We purchased relatively little external data.

我正想问这个。

I was going to ask about that.

完全正确。那是一个约束，但它是一个推动我们去其他领域改进的约束。这样的例子有很多。所以我觉得对于高主动性的人，你要赋能他们，你要让他们对自己正在做的事感到非常兴奋，但同时你也要说：嘿，如果你加入这个使命，这是需要你达成的结果，但这些是我们不会碰的地方。也许如果你在乎那些地方，那就去别处吧。

Exactly right. That was a constraint, but it's a constraint that pushed us to move on other areas to improve. And there are lots of versions of that. So I think with high-agency people, you want to empower them, you want to get them really excited about what they're doing, but you also want to say: hey, if you join this mission, this is the outcome I need you to achieve, but these are the places that we don't go. And maybe if you care about those places, go somewhere else.

好，太好了。行动号召：你们在招什么人？

Yeah, great. Call to action: who are you hiring?

我们在招收公司应用研究和工程方向的所有可能职位——从预训练一路到评估、后训练、架构。我们仍然处于一个个人能产生巨大影响的世界。我觉得我们给出的加入邀约是——我们谈了很多关于使命，关于我们怎么思考问题——但我认为我们是个人影响力比例最高的地方之一。对吧？不到 70 个人构建了这个模型。不到 150 人，工程师和研究员一起完成了这件事，这是一个非常宽泛的定义，因为我自己也算在其中。所以能够在一个你认同的使命上做这种工作——你可以拥有这些，而且每一个人仍然有巨大的影响。

We are hiring for every possible role in applied research and engineering in the company — so from pre-training all the way to eval, to post-training, architecture. We are still in a world where individuals can have massive impact. And I think our pitch to join us — we've spoken a lot about the mission, how we think about things — but I think we are one of the places with the highest ratio of individual to impact. Right? Less than 70 people built this model. Less than 150 engineers and researchers together did this effort, and that's a very broad definition because I count myself among them. And so being able to do this kind of work on a mission that you're aligned with — you can have that, and every individual still has a huge impact.

而且能够发表论文，能够开源模型。

And being able to publish, being able to open-source the model.

对，你看，所有这些都是其中一部分，但我觉得归根结底，当你今天可以选择加入一家非常大的前沿模型公司时，你只是众多人之一——这并不是他们的错，而是因为分母已经变得很大。我们的分母相当小，所以你能获得的影响力水平真的很高。而且我觉得归根结底，我们所有人——坦率地说，我所认识的最厉害的高主动性的人——他们在优化什么？他们在优化影响力。他们在优化影响力，并且会问：我是否和使命一致？如果你今天听说了这个使命，并且产生了认同，而你又在优化影响力，那我觉得我们是一个非常值得一试的地方。

Yeah, look, all of those things are part of that, but I think ultimately, when you can today pick between joining a very large foundation model company, you are one of many — not by any fault of them, but just by definition the denominator has become really big. Our denominator is quite small, so the level of impact you get to have is really high. And I think ultimately, all of us — frankly, the most incredible high-agency people I know — what are they optimizing for? They're optimizing for impact. They're optimizing for impact, and am I aligned with the mission? And if today you heard about the mission and are aligned and you're optimizing for impact, I think we're a really good place to try.

好，我想我们就到这里。太棒了。你在只睡了四个小时的情况下表现非常出色。你们这些人，你知道，播客测评绝对值得赞赏。我真的感觉眼睛开始变成这样了。我想，让你走吧，让你回去睡觉吧。谢谢你安排这次访谈。我们想把它做出来，因为我们觉得这是一个很棒的模型，也是一个很棒的故事。

Okay, I think we'll end it there. That was fantastic. You did amazing on four hours of sleep. You guys, you know, podcast eval definitely appreciated. I literally feel my eyes starting to go like this. I'm like, let you go, let you go back to sleep. Thank you for setting this up. We wanted to get this in because we think it's a great model and a great story to tell.

谢谢。

Thank you.