开源模型能跟上 OpenAI 和 Anthropic 吗？

Nathan，非常高兴你能来。感谢你亲自到场。鉴于你刚推出 Trillium，这非常及时，我很想聊聊。但更宏观地讲，我们谈了很多开源，这也是你很关心的。我们在公共讨论中经常听到：“嘿，我们必须控制前沿，前沿这个，前沿那个。”我真正想问专家的是：“前沿里有什么？我们到底在控制什么？那里有什么？更重要的是，把它和开源、开放权重模型串起来，它们能跟上前沿吗？”我认为这非常重要，而你是这方面的专家。那么，也许从这个弧线开始，你会如何分解是什么让我们到达了前沿，无论是闭源实验室还是开源模型？也许先在高层次上谈谈预训练、中期训练、后训练，后训练是如何演变的，这是你向公众做了大量教育的内容。

So, Nathan, really happy to have you here. Thank you for joining in person. Super timely given your launching Trillium, which I'd love to talk about. But more sort of on a macro level, I think we talked a lot about open source, and that's, you know, a lot of what you care about. And what we've been hearing a lot in the public discourse is like, "Hey, we have to pace the frontier, you know, the frontier this, the frontier that." And I think an arc of the stuff that I'm really trying to ask experts about is like, "What is in the frontier? What is it even like what are we pacing? What's there? And then more importantly, kind of threading it with open source and open weight models, like can they keep up with the frontier?" And I think this matters very much and this is something that you're an expert on. So, maybe to sort of start on that arc, how would you break down what's gotten us to the frontier for both closed labs and open source models? Maybe first at a high level of pre-training, mid-training, post-training, how post-training has evolved, which is something you've done a great amount of education to the public on.

当我思考模型进步时，它归结为你投入的不同形式的资本。那就是算力、数据和人才。所有实验室都有不同数量的这些。我认为闭源前沿实验室，如 OpenAI、Anthropic、Google，显然拥有最多这些。我们看到的是实验室组织这些资本形式的不同能力。让组织架构为构建模型服务真的很难，这就是为什么公司必须从底层彻底重置。然后有时它确实奏效。像 Meta 这样的公司完全重置，现在又走上了积极轨道。

When I think about model progress, it comes down to like different forms of capital that you put into them. And that is like compute, data, and talent. And all of the labs have various amounts of this. And I think the closed frontier labs of OpenAI's and Anthropic's Google's have clearly had the most of these. And what we see is like the different abilities of the labs to organize these forms of capital. And like making the org chart work for building models is really hard, which is why you have companies have to do complete resets from the bottom up. And then it actually kind of works sometimes. Like company like Meta is like totally reset and they're like on a positive trajectory again.

真的吗，是的。

Seriously, yeah.

我对这些人的方面非常感兴趣，比如为什么竞争驱动前沿，为什么组织架构如此重要，因为我们在开放-闭源动态中也看到这些。闭源公司，真正的前沿实验室，我说如果有人想在前沿 AI 工作，我大多告诉他们去 Anthropic 和 OpenAI，这是最容易的赌注。Meta、Google、SpaceX AI，你可能还在那里，但从外部更难看到。我认为这些公司最早采取了最雄心勃勃的赌注来达到那里。这就是为什么 OpenAI 做了像启动推理模型革命这样的事情。我认为那种突破和推理缩放定律的美丽曲线是 OpenAI 的巅峰。他们真的很擅长得到这种改变每个人思考方式的曲线。

It's like I think this I'm very interested in these human aspects of like why competition drives the frontier and like why the org chart is so important because like we see these things and this comes into play in the open closed dynamic as well, where it's like the closed companies the front the true frontier labs, which I say if somebody wants to work at the frontier of AI, I mostly tell them to work at Anthropic and OpenAI, which is the easiest bet, I think. Um Meta, Google, um SpaceX AI, like I think you still might be there, but it's harder to see from the outside. And I think these companies have just taken the most ambitious bets the earliest in order to get there. And that's why OpenAI has done things like kickstart the reasoning model revolution. And I think that kind of breakthrough and like that beautiful plot with inference scaling laws is like peak OpenAI. It's like they're really good at getting this plot that just changes how everyone thinks about everything.

是的。

Yeah.

另一方面，Anthropic 一直非常关注模型将能做什么类型的事情。然后他们的回报是 Claude 代码的爆发，因为他们长期专注于编码。有这些微妙不同的文化。我认为这两家公司尤其最具前瞻性，最愿意花费巨额资金朝着人们认为不可能的方向迈出一步。中国公司进入这个领域是因为他们想拥有一家重要的公司。如果他们听说从中国构建模型并尝试销售 API，没有人会知道他们。没有人会知道 Moonshot AI。很少有人会知道 Jipu，之后还有很长的名单。这是他们非常实际的决定。他们跟随讨论。他们知道会有巨大的市场。他们知道美国对这些东西有巨大的付费需求。这非常实际。因此，发布开放模型是捷径并进入对话的一种方式。我认为他们为什么这样做有很多复杂性？政府参与了吗？没有。情况并非如此。他们只是试图建立一家初创公司。并不是说他们不会达到那里。中国政府做的事情会让美国人震惊。我认为这让我想到中国实验室对安全有更严肃的方法，因为在中国发生的事情是对公司在美国做事情的更激烈反应。我不需要深入这个兔子洞。但这就像在开放和闭源之间的平衡，这些中国公司正在努力追赶。这与我对 OpenAI 和 Anthropic 所说的非常不同。他们试图创造新范式。他们试图让真正的 RSI 发生，以便他们的公司永远赢得最好的模型，如果你先到达那里。这些是繁荣的想法。你只需要填补市场空白。然后开放权重模型试图这样做。我认为随着时间的推移，会有这些层级的开放模型，有最好的闭源模型，你可以要求最好的利润。然后有接近前沿的模型，落后一步。那些将是开放的，因为那里有某种供需关系。当你进入一个可能利润较低的不同市场时，但例如，如果你有私有企业数据等等。

And on the other side is like Anthropic has been like very focused on the types of things the models are going to be able to do. And then their reward was this Claude code explosion with their focus on coding for so long. And there's these subtly different cultures. And I think those two companies in particular have been the most forward-looking and the most willing to spend egregious amounts to make a step towards a direction that people think is impossible. The Chinese companies came into this because they want to have a company that matters. And if they heard to build their models and try to sell an API from China, nobody would know about them. Like nobody would know about Moonshot AI. Very few people would know about Jipu, and there's a very long list after this. It's a very practical decision by them. They're following the discourse. They know there's going to be a huge market. They know there's a huge demand in the US for paying for these things. And it's very practical. So therefore, like releasing open models is a way to shortcut this and get into the conversation. And I think there's a lot of complexity in like why do they do this? Is the government involved? And not. It's like not the case. They're just like trying to build a startup. And it's not to say that they will not get there. Like the Chinese government does things that would blow people's minds in the US. And I think they like That makes me think that the Chinese labs have a much more serious approach to security because like things happen in China that are much more drastic responses to companies doing things in the US. And I don't need to go down this rabbit hole. But this is just like between the balance of open and closed, it's like these companies in China are trying to catch up. And this is a very different dynamic than what I said about OpenAI and Anthropic. They're trying to create the new paradigms. They're trying to like make true RSI happen so that their company wins with the best models forever if you get there first. That's these boom ideas. And you just have to fill the market where there is a void. And then open-weight models are trying to do this. And I think over time there will be these tiers of open models where there's the best closed models which you can demand the best margins for. And then there's the near frontier models which is a step behind this. And those will be open because it's just like some sort of supply and demand there. As you get access to a different market that might be less profitable, but for example, like if you have private enterprise data or so on.

这些层级其实是公司自然演进的结果。我们说的是前沿，回到资本层面，他们的资本更少，赚的钱也会更少，这没关系，但这种追赶的动力非常强。一旦 Anthropic 和 OpenAI 向你展示了需要构建什么样的模型，接下来就容易多了，你会想：“哦，我们只需要做这件事。”

And these tiers are kind of natural evolutions of companies that are trying to do things. And I think we're talking about the frontier, and it's just like if you go back to capital, like they have less capital, they're going to make less money, and like that's okay, but I think this drive to catch up is very strong. And once Anthropic and OpenAI show you the type of model you need to build, it's a lot easier to then be like, "Oh, we just need to do this thing."

是的。

Yeah.

他们购买类似的数据集。这会渗透到数据经济中，OpenAI 和 Anthropic 花费更多来构建这个数据产业的功能，然后那些数据公司一年后可能会把它们卖给所有人。他们需要最大化投资回报，而且他们有一些合同期。我认为有很多动态使得快速跟随变得非常可行。我们已经看到这个周期持续了一段时间，比许多人预期的要长。当 ChatGPT 发布时，我们讨论了行业整合并预测了它。我认为我们才刚刚开始看到一个更加活跃的并购市场，但我以为这会在两年前发生。而开源模型公司持续的时间比我以为的要长得多。

They buy similar data sets. It trickles through the data economy where OpenAI and Anthropic spend way more to build out this function of a data industry, and then those data companies a year later might sell them to everybody. They need to maximize their return on their investment, and they had some contractual period. And I think there's a lot of dynamics where fast following is going to be very doable. And we've seen this cycle proceed for a while, and longer than many people would expect, I think. When ChatGPT was released, we talked about consolidation of the industry and predicting it. And I think we're just starting to see a much more active mergers and acquisitions market, but I thought this was going to happen 2 years ago. And the open model companies have lasted much longer than I thought.

嘿，这里有很多有趣的线索可以展开。从你对两大前沿实验室组织差异的评论开始吧。我们可以先搁置那个，或者继续那条线索。但当然也要解释一下，中国的开源模型或中国的开源实验室是如何真正选择将其作为一种分发/实际商业化策略，而不是出于利他主义。尽管研究人员在开始时可能是出于分享的动机，这很好，但最终它持续下来是因为它是一种极好的商业化策略。另一个威胁可能是，我认为大多数人将蒸馏视为他们做得如此好且越来越好的主要杠杆，因为开源模型和前沿模型之间的差距在过去几年里只缩短了。现在可能只有两到三个月。这也促使前沿实验室可能想要关闭 API，关闭他们最好的模型以提供第一方服务。一方面可能是为了捕获更多价值，因为总是有可能，但另一方面也因为更难蒸馏，不是不可能，但难得多。也许在这两个方面，因为它们都很有趣，所以我们会尝试触及。让我们从蒸馏的故事是否真的更像是一种讽刺开始，然后是中国实验室，因为你实际上见过他们，而且你研究得更多，他们是不是全能型，比如擅长预训练，也有自己的好的后训练配方，他们可能透露也可能没有透露，你认为杠杆在哪里加速？

Hey, you know, there's so many interesting threads to pull on here. Starting with actually your commentary on the organizational differences between the top two frontier labs. You know, we can maybe table that or go on through that thread. But then of course also kind of explaining how the open Chinese models or the open labs in China have really just chosen that as much more of a distribution/practical commercialization strategy, rather than, hey, you know, obviously out of altruism. Though, it's great that the researchers maybe for better or for worse were motivated for sharing the beginning, but at the end of the day it perpetuated because it was an excellent commercialization strategy. So the other threat maybe to pull there is sort of, you know, I think most people regard distillation as sort of like the main lever for them to do this so well and kind of increasingly well because the lag between the open source models and the frontier models have only shortened with time in the last few years. Now it's like maybe only two to three months. And this is also pushed the frontier labs to maybe want to close off the APIs, close off their best models to do first-party services. I mean one probably to capture more value cuz it's always possible that way, but then two also cuz it's harder to distill, not impossible but much harder to distill. Maybe on those two fronts cuz they're both very interesting, so we'll try to touch on it. Let's start with the is the distillation story really more of a caricature and then Chinese labs, cuz since you've actually met with them and you've studied this a lot more, is it sort of like they're all-rounder, like great at pre-training, also have their own good post-training recipes that they may or may not have revealed, and where do you see the levers kind of accelerating?

是的，所以中国实验室在技术上非常出色。他们更缺算力。我是说

Yeah, so the Chinese labs are very technically excellent. They're more compute-starved. I mean

是的。

Yes.

毫无疑问，他们拥有相当的人才库。我认为可能有一点优势，取决于 AI 真正起步时谁在哪里，但要说中国在人才上不平等，你会很难，因为他们有太多人在如此高的水平上接受训练，而他们缺少的是算力。你和那里的人交谈，你说：“哦，我听说 OpenAI 的普通研究员有 5000 个 H100 等效算力，可以随时访问。”他们的下巴都掉到地上了。他们就像：“我无法理解。”这是一个你知道的实验室的顶级预训练研究员。然后你有几次这样的互动。算力问题我认为是最大的区别，而蒸馏的作用在于它是一种有效花费更多钱来获得算力等效物的方式。语言模型的输出是算力的函数，如果你知道你想要的输出，那就等同于拥有算力来用模型生成它。我见过蒸馏的各个阶段，我的看法是，以前的后训练更依赖于监督微调和其他，就像基于人类反馈的强化学习（RLHF）时代，那是后训练的开始，你可以通过在一个非常强大的模型的输出上做下一个词预测损失来获得大部分性能。非常确定的是，你拿最好的模型，生成补全，在上面训练，你的模型就会相当好，只要你能访问正确的提示。提示一直是后训练的关键。我记得在 AI2，我和 Hannah 共同领导后训练，我们总是说提示是后训练的关键单元。你需要获取提示，你可以在训练的不同阶段用它们做不同的事情，现在这已经演变成以强化学习为主的环境。但很难将这种后训练——你得到补全，分数上升——映射到今天推理中发生的事情。根据公开文献，我很难说蒸馏是一个主要因素，因为例如在完全公开的文献中，人们试图做开放后训练配方，所有开放模型构建者都卡在使用一个非常过时的 Qwen 模型上，因为你无法弄清楚如何让教师模型正确。我的假设是，有一些中间训练和公共预训练数据中的某些东西是开放的，使得从绝对最好的模型中获取推理轨迹并进行监督微调然后数字上升这件事变得困难。所以，如果我在构建 Olmo 时能够访问 Claude 的推理轨迹——我实际上是指 Claude Opus、Claude Mythos 之类的——我不确定它是否真的会让我们的模型好很多。所以，这是我基于文献的证据，说明为什么我认为蒸馏很难。然后，你如何利用模型的输出构建一个强化学习环境来进行大规模的强化学习运行？因为世界上有小米这样的公司进行为期一周的开放强化学习运行，或者像“这是我们做的”，然后他们发布模型，蒸馏到底在哪里进入？他们很聪明。他们知道如何做好营销。

Like without a doubt they have a comparable talent pool to draw on. I think there might be like a slight edge based on who was where when AI really got going, but to say that China is anything but equal in talent is going to you're going to have a hard time because they just have so many people being trained at such a high level and what they don't have is compute. You talk to people there and you're like, "Oh, I heard an OpenAI average OpenAI researcher has 5,000 H100 equivalents that they can access at any time." and like their jaw just falls on the floor. They're just like, "I can't comprehend." And this is like a top pre-training researcher at one of the labs that you would know. And then you have this interaction a few times. And like the compute thing I think is the biggest difference and distillation plays in as it is a way to effectively spend more money to get what is compute equivalent. Like outputs from a language model are a function of compute and if you know the outputs you want it's the same as having compute to generate with it with the model. And I've seen the stages of distillation and how I see it is that previous post-training was much more reliant on the supervised fine-tuning and other to like just RLHF era which is where post-training started and you could get most of your performance just by you do this next token prediction loss on outputs of a very strong model. It was very established that you take the best model you can you generate completions you train on it and your model's going to be pretty good so long as you have access to the right prompts. Prompts have always been a key to post-training. I remember at AI2 like with Hannah who I co-led post-training there with we would always say like prompts are the key unit post-training. It's like you need to get the prompts and you can do different things with them at different stages of training and now this has morphed a bit into environments where it's RL heavy. But it's just like hard to map this type of post-training which is you get the completions and the score goes up to what is going on in reasoning today. And based on the public literature it's very hard for me to be like distillation is a major factor because for example in the fully open literature of people trying to do open post-training recipes all the open model builders are stuck using like a really outdated Qwen model because you can't figure out how to get the teacher right. My hypothesis is that there's some like mid-training and something in the common pre-training data that's open that makes it hard to do this thing which is get reasoning traces from absolute best model and do SFT and the numbers go up. So like if I were to have access to Claude's reasoning trace when I was building Olmo I'm actually like Claude Opus Claude Mythos whatever I'm not sure that it would have actually made our model a lot better. So, this is like my evidence grounded in the literature why I think distillation is hard. And then like how do you take the outputs of a model and build an RL environment to do a big RL run? Cuz then you have the Xiaomis of the world doing this week-long open RL run or like here's what we did and then they drop the model and it's like where exactly does distillation go into this? They're clever. They know how to do marketing well.

有些问题是，他们是不是提前做了这件事，因为他们先跑了强化学习，然后发布了模型，接着他们的技术报告说在强化学习之后还有一个额外的后训练阶段。所以有些疑问，但把已有的蒸馏文献套用到这上面非常困难。不过有一些问号让我稍微支持这些实验室，比如这些 Claude 代理和中国的一些模型代理，那里使用了大量 token，有中间商，他们有大量账号，甚至出现 Kimi 在产品中向用户提供 Claude 服务的情况。我就是不太明白这怎么符合蒸馏作为后训练中常见技术工具的定义。是的，我们也应该关注，比如中国公司大规模使用 OpenAI 和 Anthropic 模型的输出，还有其他奇怪的事情。所以我认为我们还有很多问题需要回答，我认为大多数理性的人在问蒸馏的差距有多大时，都会说误差棒非常大。但当我说蒸馏对他们有帮助时，这是基于我们已有的科学信息。我们应该做更多开放研究并允许这样做，以便我们研究它，但基于这些，我不知道你怎么能得出不同结论，说它在当前后训练中不是巨大影响。

There are questions like did they do this ahead of time because they did their RL run and then they dropped the model and then their tech report said they had an extra post-training stage after RL. So there were some questions, but mapping the established distillation literature to that is very hard. But there are question marks that make me support the labs a bit, which is like these Claude proxies and these model proxies in China where there's a ton of tokens that are used and there's intermediaries and they have tons of accounts and even cases where Kimi was serving Claude in their products to users. And I just don't really understand how this fits into distillation as a common technical tool in post-training. And yes, we should watch it too, like there's other weird stuff with Chinese companies using the outputs from OpenAI and Anthropic's models at a vast scale. So I think we have a lot more questions we need to answer, and I think most reasonable people when they're like what's the gap from distillation are like there's very big error bars. But when I say that distillation helps them by a few, it's based on this scientific that we just like that's information we have. We should do more open research and enable this so that we can study it, but like I don't know how you can come to a different conclusion that it's like not a huge effect in current post-training based on this.

这就像说中国人太擅长构建模型了。嗯，我认为有点自我安慰，人们说只有我们擅长构建模型，但实际上有很多公司。大约有 20 到 30 家公司在构建好的模型。我认为你只是

And this is like the Chinese are so good at building models. Well, I think there's a bit of cope where people are like only we are good at building models and it's like there are so many companies. There's like 20 to 30 companies that are building good models. And I think you just

是的。

Yeah.

我很难释怀，我认为有一篇非常著名的论文，讲的是从闭源模型中窃取推理轨迹。我知道中国公司在这方面遥遥领先，因为 AI 安全社区后来才发表。我觉得每个中国实验室都有一个团队知道这类事情。这回到你所说的，他们不需要提供 API。我认为推理模型生成这些思考 token 有一些特点，很容易把它们从 API 中敲出来，这样你就能看到推理 token，然后可以用来训练你的模型。只要他们提供这些 API，就无法阻止这种蒸馏。但如果他们只把模型放在产品中，需要签署企业协议之类的，就会难得多。我支持实验室这样做，如果他们想保护的话，但他们必须保护自己的产品。我不认为了解你的客户对任何人都有好处。我不认为这对他们的业务有好处。而且我也不明白为什么要把政府卷入服务条款违规。我甚至不认为服务条款是签署的合同。我不认为你可以以同样的方式起诉他们。你只能取消他们的访问权限。

And it's hard for me to get over and I think that there's this very prominent paper that's this like stealing reasoning traces from closed models. And I know that the Chinese companies are way ahead on doing this since the AI safety community later published it. Like I think every Chinese lab has a team that knows those sorts of things. And this goes back to what you're saying about like they don't need to offer the API. And I think there's some characteristic of reasoning models that generate these thinking tokens where it's easy to just knock them out of the API so you see the reasoning token and then you can use that to train your model. And so long as they offer these APIs there's going to be no way to stop that type of distillation. But if they do make it so it's only in products and things where you have to sign enterprise agreements or whatever, it'll be way harder. And I support the labs doing that if they want to secure it, but it's like they have to secure their products if they want to. Like I don't think know your customer would be good for anybody. I don't think it'd be good for their business. But also I don't know why you should get the government involved in a terms of service violation. It's like not even I don't even think terms of service are a signed contract. It's just like a I don't think you can sue them in the same way. You just can cancel their access.

对。

Right.

所以这就是为什么我不那么同情，但我喜欢听，国家安全社区声音很大。但确实有方向性的真理，随着 AI 的发展，它的重要性，你必须在头脑中平衡许多

So that's why I'm not that sympathetic, but I am like I like to listen to the like the national security community is loud. And but it's like there is a directional truth to that with where AI is going and how important it is that you have to balance many

是的。

Yeah.

许多事情。

many things in your head.

我想知道你的回应是什么。所以我听说，我想是 Jensen 在某个推文或采访中说的，也许

I wonder what your response is. So I heard I think Jensen said in some literally either tweet or interview maybe

太棒了。太棒了。把它放进节目里。

It was great. It was great. Put it Put it in Put it in the show.

人们每天都在蒸馏我的产品。他们拿走它，把它拆解到原始部件。他们测试一切。那叫做竞争。

People distill my products every single day. They take it and they they strip it down to its raw parts. They test everything. That's called competition.

你认为这是公平竞争吗？你认为蒸馏不是抢劫吗？

Do you think that's fair game? Do you think distillation is not robbery?

那叫做竞争。

That's called competition.

我非常同情，因为我认为 AI 安全社区有很多呼吁，我们希望在达到这样的能力水平时，美国尝试对话。如果你要让公司只是开始对他们的 AI 行业进行轻微的惩罚，以及更多

I'm very sympathetic cuz I think there's a lot of calls from the AI safety community that we want to have like US trying to dialogue as we're getting to such capability levels. And if you're going to ask the company to just like start hitting their AI industry with slaps on the wrist and more

是的。

Yeah.

蒸馏的下游。所以你不会用国会政策来监管蒸馏。你可能会采取联邦行动让他们的生活变得烦人。

downstream of distillation. So you won't regulate distillation with policy from Congress. You will probably just do federal action to make their life annoying.

是的。

Yeah.

同时，当我们谈论如何管理前沿风险时，你有闭源模型在训练中遍布网络，访问它们不应该访问的东西，我们对开放权重模型会发生什么有合理的担忧，虽然还没有危害，但为什么人们不使用开放权重模型？为什么他们去闭源模型并绕过保障措施？而且

And at the same time when we're talking about how do we manage frontier risks where you have closed models in training going all over the web and accessing things that they shouldn't and we have valid worries about like what will happen with open weight models where there's like hasn't been harms yet, but it's like why like what why aren't people using open weight models? Like why did they go to the closed models and get around the safeguards? And like

是的。

Yeah.

我们希望能够与中国科技公司沟通，以及关于这一切。我只是认为如果你对蒸馏采取任何行动，会造成更多伤害而不是好处，因为他们不会停止。他们只会继续这样做。就像

We want to be able to communicate with the Chinese tech companies and everything about this. And I just think like you're going to cause more harm than good if you do anything about distillation cuz they're not going to stop. It's all it's like they're going to keep doing it. Like

对。对。然后开源社区受益，因此创业社区也受益匪浅。你可以进入，你知道，很多不仅仅是中美之间。实际上，你知道，我们可以谈论它，就像我显然同情或试图为前沿实验室的立场辩护，那就是你需要前所未有的资本来构建这些前沿模型，对吧？这有点令人担忧，或者说不太有利于这个飞轮的持续，如果通过蒸馏或其他蒸馏相关方法，开源可以如此迅速地赶上。所以，你知道，我说的飞轮只是一个非常简单的画面，你知道，你投入大量资金获得算力来训练前沿模型，然后他们当然会停放这些算力。OpenAI 和 Anthropic 在这方面非常内疚，或者说非常成功。

Right. Right. And then and and the open source community benefits and therefore the startup community benefits a lot. You can go into you know, a lot of it's not just like hey China USA. It's actually like there's you know, and we can actually talk about it like I am sympathetic obviously or try to steelman the um frontier lab position which is like you have just unprecedented amounts of capital that's required to build these frontier models, right? And it's it's kind of alarming slash not very um uh conducive to to this uh flywheel continuing if if the um through distillation or other kind of distillation related methods um open source can catch up so quickly. And so, you know, and what I mean by this flywheel is just kind of a very simple picture of you know, you you train the frontier model by putting in a lot of money to get compute and then they park this computer of course. Open AI and Anthropic are very guilty slash you know, successful in this.

成功。

Successful.

无论你怎么看。有巨大的算力，你知道，有点缺乏可用的算力，因为实际上不仅仅是算力和芯片，还有电力等。所以，所有这些稀缺性在 2028 年之前都是瓶颈。总之，这一切都是为了说明，停放算力是一种成功的策略，这实际上带来了大量收入，因为一旦他们到达前沿，因为他们理解，你知道，Scaling（规模扩张）是给他们的一种东西，或者 Scaling（规模扩张）的科学是让他们掌握如何将其转化为可预测收入流的东西。

Whichever way you want to like look at look at. and there's a huge compute, you know, uh kind of uh um sort of um lack of computer available because there's actually like not just compute and the chips, but there's also power, etc. So, like all of this scarcity is kind of, you know, bottlenecked till 2028. So, anyway, this is all to say that it's a successful strategy to park that, which leads to actually a lot of revenue because once they get to the frontier because they understand, you know, scaling is the one thing that kind of gives them or the science of scaling is what gives them a handle over how how well that can translate like into a predictable revenue stream.

所以，他们获得营收，而因为有了营收，他们就能筹集更多资金。这简直是前所未有的，就像一个美妙的飞轮，OpenAI 和 Anthropic 都是如此。当然，我们需要指数级的资源，所以我们已经到了这样一个地步：好吧，我们需要 IPO。我们需要进入公开市场。因此，任何威胁到这个飞轮良好运转的因素，都是对整个前沿地位的巨大威胁。

So, they get the revenue and because of that revenue they're able to raise more money. It's just unprecedented, like a beautiful flywheel, and this happens for both OpenAI and Anthropic. Of course we need exponential amount of resources, and so we've gotten to the point where we're like, okay, we need to IPO. We need to go to the public markets. And so anything that threatens this flywheel from functioning well is a huge threat to just the entire, like, being at the frontier.

现实地说，是整个 AI 行业。

AI industry, realistically.

实际上正是如此。所以，你知道，也许不仅仅是同情他们那种为自己榨取资本主义的做法，而是，嘿，如果他们在 IPO 中表现不佳，那实际上对其他人也不利。所以我希望每个人都好，我希望他们做得好。另一方面，你知道，中国的开源模型对创业生态系统是巨大的利好。我的意思是，我们一直看到这一点。我记得像 Decagon 和 Harvey 的指标，他们试图做大量的微调。有大量初创公司以各种程度提供后训练即服务，并试图将其产品化，而且这确实有效，因为你可以通过这些，你知道，Kimi K3 或者你最喜欢的 GLM 开源模型达到帕累托前沿。所以，我想权衡这两件事，你怎么看，我想有很多问题可以从中衍生，但我想一个问题是，你如何看待未来几年谁会赢，以及你如何将其与你为什么做开源联系起来。

And exactly that, actually. So, like, you know, maybe not just kind of being sympathetic to their version of like trying to, you know, eke out capitalism for themselves, it's also just like, hey, if they don't do well in the IPO, that's actually bad for everyone else. And so I want everyone, you know, I want them to do well. On the other hand, you know, it's like the Chinese open source models are a huge benefit to the startup ecosystem. I mean, we see this all the time. Just kind of metrics I remember from like Decagon and Harvey was trying to do a lot of fine-tuning. There's like a huge amount of startups doing post-training as a service of various degrees and trying to productize that, and it actually works because you can get to the Pareto frontier with these, you know, Kimi K3 or pick your favorite like GLM open source model. So, I guess like weighing these two things, how do you, I guess there's so many questions can fall from that, but I guess one, how do you see that playing out in terms of like who's going to win in the next couple of years versus you know what, and then also relate that to you know why you're doing obviously an open source.

所以我认为开源和闭源模型处于不同的指数时间尺度上，闭源实验室如此一体化，他们拥有高效的推理，当他们准备好时就能大规模服务模型，因为他们拥有所有这些资源，他们可以规划，并且他们最先进入这些高价值的市场。我们在编程领域看到了这一点，这是今年的故事，营收从零增长到 1000 亿美元，而且很可能继续增长。但开源模型实验室，部分来说这是一个经济问题，作为快速跟随者，你没有这种信息流，也没有这种营收流。所以开源模型，你有一个开源模型构建者，然后你有开源模型推理公司，还有开源模型后训练公司，来自那些客户的信息，比如你需要模型做的事情，这些信息并不容易流回模型构建者。而且他们也没有规模经济来服务那么多用户，以那么高效的方式服务，在推理上获得那么多利润，这些都需要更长的时间。但就事物可以被商品化的程度而言，公司似乎真的想要拥有私有数据。有很多东西他们就是不能通过 API 发送，我们可以讨论零数据保留与否等等，但这些事情需要很长时间才能弄清楚，如果公司有内部专业知识，或者他们弄清楚如何构建一个真正与营收匹配的内部评估，然后他们弄清楚如何微调，或者他们去找 Thinking Machines 或 Prime Intellect，他们说如果我们能弄清楚如何微调一个解决这个问题的模型，我们就能大规模赚钱，他们有这些客户，但涉及的公司太多了，所以需要的时间长得多。但在这个过程结束时，他们拥有真正的所有权，即模型权重，没有条款和条件，他们可以在自己的硬件上运行，并且他们弄清楚了可以获得投资回报。所以，这就是为什么我认为开源模型经济需要更长的时间来构建，我不太担心这会是泡沫，因为这可能会侵蚀快速起飞的市场，因为人们会弄清楚什么是最便宜的模型来完成任务，以及如何在利润上竞争？

So I think open and closed models are on different time scales of exponentials and the closed labs are so integrated where they have efficient inference, they can serve the model at scale when they're ready to because they have all these resources, they can plan, and they get to these markets first that are very high value. We have seen this with coding and this has been the story of the year where the revenue has just gone zero to 100 billion and likely to grow. But the open model labs partially this is like an economic issue being the fast follower, it is that you don't have this flow of information and you don't have this flow of revenue. So the open model you have an open model builder, then you have the open model inference companies, and you have the open model post-training companies, and like there's information from those customers which are like things you need the models to do and it's like you can't like that information doesn't just flow back to the model builder very easily. And they also don't have the economies of scale to serve as many users, to serve it as efficiently, to have as much margins on inference, and these are just like it's going to take a lot longer. But to the extent that things can be commoditized, it seems like companies really want to have private data. There's a lot of things that they just cannot send over an API and we can talk about the like zero data retention or not and so on, but like these things take a long time to figure out if a company has internal expertise or they figure out how to build an internal evaluation that actually matches to revenue and then they figure out how to fine-tune or then they go find thinking machines or Prime Intellect and they're like we can make money in mass if we figure out how to fine-tune a model that solves this and like they have these customers, but there's just so many more companies involved that it takes so much longer. But at the end of this process, they have real ownership, which is model weights, no terms and conditions that they can run on their hardware, and they figured out they can make an ROI on. So, this is why I think the open model economy is just it takes way longer to build, and I'm less concerned about this being kind of bubbly, where it's like this could eat into the market of the really fast takeoff as people figure out how to what is the cheapest model that accomplishes a task, and how do you compete on margins?

嗯。

Mhm.

我真的不知道。就像我真的不知道会发生什么。我认为有

And I don't really know it. Like I don't really know what's going to happen. I think there's

是的。

Yeah.

现在知道还为时过早。

It's too early to know.

是的，这太难了，我的意思是，这绝对是始终萦绕在脑海的事情，比如有些事情，扩大通用推理，作为一个通用模型，它会击败任何专用模型，我们在数学中明显看到了例子。这可能是你能看到的最纯粹的领域。但是，你知道，即使在机器人领域，你知道，随着 Astra，嗯，也许无论信号的真实程度如何，听起来他们并没有真正尝试在机器人领域提升

It's so yeah, it's so hard I mean it's definitely something that's always top of mind thinking about like there's thing there are things where scaling up general reasoning just as a general model is so uh it's going to beat out any specialized model, and we see obviously examples in math. And that's probably the most pure domain that you can see this. Um but also, you know, even in say robotics, you know, with with Astra, um and and I for maybe uh however degrees of like uh tru- trueness the signal is, it just doesn't sound like they've actually tried to ascend in robotics

哦，我认为他们做了。我认为这是一个周期，大约 6 到 8 个月前，他们开始招聘人员并购买大量机器人数据。

Oh, I think they did. I think this is a cycle of like 6 to 8 months ago they started hiring people and buying a lot of data in robotics.

他们确实做了。

They did.

模型神奇地在 6 个月后变得非常擅长。但这就是我认为的

the model magically 6 months later is really good at it. But this is what I think that the

但实际上，我听到的其他更相反的事情是，它实际上来自计算机使用，嗯，因为你知道，因为计算机使用是某种程度的空间推理，然后能够有好的 Blender 演示是另一个延伸。嗯，所以，你知道，我们可以相信某种版本，但无论如何，我想这就像，也许如果它真的很专业，你会看到能力提升真的来自试图在某个领域变得更好。但就像在数学中，我甚至不知道这是否来自试图专攻数学。我的意思是，我认为整个论点就是，让我们扩大推理，实际上在自然语言方面变得更好。它比任何专用模型在数学上都更好。我的意思是，Anthropic 甚至没有试图擅长数学。你知道，OpenAI，当然，我认为，你知道，他们雇佣了一堆数学家，但他们也没有试图使用专门的，你知道，类似精益数据集来变得更好。嗯，或者至少不是那样。嗯，抱歉。所以，无论如何，这些都是支持这种观点的素材：与前沿模型竞争很难，因为最终它们开始提升得如此之多，以至于你在各个类别中都变得足够好。

But I actually the other other, you know, more contrary things I've heard is that it's actually coming out of computer use, um because it's you know, cuz computer use is like some level of spatial reasoning, and then getting to have good blender, you know, demos is is another extension of that. Um so, you know, we can believe some version of that, but either way, I guess it's like it maybe if it's like really specialized, you see it's it's the capability gain is really from like trying to get better in a certain area. but like in math, I don't even know if that fell out of trying to specialize in math. I mean, I think the whole thesis there was just like, let's just scale up reasoning and actually got better in in in natural language. It got better than any specialized model in math. I mean, Anthropic wasn't even trying to be good at math. You know, OpenAI, you know, certainly I think, you know, they hired a bunch of mathematicians, but they also weren't trying to like use specialized, you know, kind of like lean data set to get better at it. Um Or at least that's not Um sorry. So, anyways, this is this is all kind of fodder for the view that like it's kind of hard to compete with the frontier models because eventually they start ascending so much that, you know, you you get uh good enough in respective um categories.

话虽如此，我认为你也看到了一股非常具有竞争力的反向力量，那就是已经足够好的开源模型允许微调，正如你所说，这些非常具体的基准和数据集缓慢渗透扩散，而这些数据集之前只存在于你被应用的那些公司中。而且它已经足够好了，我们开始看到人们采用它。我的意思是，在最纯粹的体制下，你会期望前沿模型表现良好，我认为 Cursor 在夏天发布了一些东西，他们说：“嘿，我们有一个长期任务，就是重写 SQLite，结果发现使用两个前沿模型，一个用于规划，一个用于执行，并不比只使用 Composer 搭配 Opus 或 Fastra 作为规划器更好，或者效果相当，但成本高得多。”我认为这是一个很好的证据，表明开源微调非常好或足够好，并且已经超越了需要智能的任务。说了这么多，我想问的是：对于前沿模型与开源模型各自胜出的这种稳定状态的各种版本，你觉得最有说服力的证据是什么？

With that said, I think there's also a very competing counterforce of data that you're seeing, which is the open source models that are already good enough allow for fine-tunes, to your point of these slow percolation diffusion of these very specific benchmarks and datasets that were only available in these companies that you're being applied to. And it's just good enough, and we're starting to see people adopt that. I mean, in the most pure regime where you'd expect frontier models are good, I think Cursor published something over the summer where they said, "Hey, we had this long horizon task of rewriting SQLite, and it turns out using two frontier models, one for planner, one for execution, is not as good or is equivalently as good, but far more costly than just using Composer with maybe Opus or Fastra as planner." And I think that's great evidence that the open source fine-tunings are very good or good enough and have surpassed intelligence-hungry tasks. This is a long way of asking: what evidence do you find most compelling for various versions of this stable state of where the frontier wins versus open source?

我强烈主张，在那些不完全可验证的领域，前沿进展很大程度上来自于购买高质量数据，并给模型提供可以行动和爬坡的世界。我认为有两件事。我们有数学和代码，我们不知道编码市场会有多大。然后还有其他一切，仍然针对知识工作、生物学、医疗、法律。所有这些都不那么可验证，我认为这更接近 OpenAI 购买大量机器人数据的故事。他们可能会告诉我们这是泛化魔法，噗的一下。人们喜欢噗的一下泛化。但现实地说，可能一部分是 A，一部分是 B，但很大程度上归结为数据。当我与前沿实验室的人交谈时，有些人有很棒的工作。有些人通过做非常狭窄的事情来贡献：你制作一个数据集，你弄清楚你的模型在那个领域应该有什么行为，你弄清楚如何制作有助于获得该行为的数据，你弄清楚你需要购买什么数据。这需要——我认为我们将在不同行业经历多年这样的过程，购买所有这些数据是非常资本密集的，但这并不是像噗的一下魔法 RSI 就发生了。我在数学和代码以及一些分布式机器学习工程中确实看到，这是非常可验证的。你知道你的成本是多少，AI 模型将以真正神奇的速度降低它，这将带来大量剩余收益，我们可以更快地做研究。我非常感兴趣的是——我认为模型的效率将在多年内成倍下降。所以我们看到 OpenAI 大幅降价，就像 GPT-4 或 Astra 和这些更大的模型在未来 3 年内每 8 个月价格降低 2 倍意味着什么？你开始做这个数学，就像 4 年后成本是 1%，就像他们能否因为杰文斯悖论而获得更高的利润率，或者他们的收入会因为市场饱和而减少？有所有这些非常奇怪的动态，但根本上 AI 进步是关于数据和编写更好的软件来服务你的模型，而这些是非常不同的世界。

I'm a big proponent that a lot of the frontier progress in domains that are not fully verifiable is by buying data that is high quality and giving the models worlds to act in and hill climb on. And I think there are two things. We have the math and code thing, and we don't know how big the coding market will be. And then there's everything else, which is still targeting this knowledge work, this biology, health care, legal. And all of these things are not as verifiable, and I think it is much closer to the story of OpenAI bought a lot of robotics data. They're probably going to tell us it's generalization magic poof. People love poof generalization. But realistically it's probably some of A, some of B, but a lot of it comes down to data. When I talk to people in frontier labs, there's some people that have amazing jobs. There's some people that contribute by doing a very narrow thing: you make a dataset and you figure out what the behavior should be for your model in that area, and you figure out how to make the data that helps get that behavior, and you figure out what data you need to buy. And that's going to take—I think we're going to have years of this across different sectors, and that's very capital intensive to buy all of this data, but that's not just like poof magic RSI happens. Where I do see in this math and code and some like distributed ML engineering, it's very verifiable. You know what your cost is, and the AI models are going to drive that down at a really magical rate, and that's going to have a lot of surplus benefits, and we can do research faster. I'm very interested in—I think the efficiency of models is going to drop multiplicatively over years. So we've seen OpenAI have drastic price cuts, and it's like what does it mean for GPT-4 or Astra and these bigger models to reduce in price 2x every 8 months for the next 3 years? You start doing this math, and it's like it's 1% of the cost in 4 years, and it's like are they going to be able to have higher margins due to Jevons paradox there, or their revenues going to decrease because the market fills up? There's just all these really weird dynamics, but fundamentally AI progress is about data and about writing better software to serve your models, and those are very different worlds.

是的，我的意思是至少我同意你的观点，从外部观察，有很多数据强化学习环境公司做得非常非常好，既卖给两大前沿实验室，也越来越多地卖给长尾实验室。所以正如你所说，我认为很多能力提升确实来自于扩大规模，这不是一件容易的事。我的意思是，也许我实际上很好奇你对人工构建的数据收集和环境与在模拟中完成的环境之间的看法。似乎有不同的版本，比如它们可能是什么，它们能要求什么样的价格等等。

Yeah, I mean at least definitely I agree with you that observing from the outside there's a lot of data RL environment companies that are doing very very well selling both to the top two frontier labs but an increasing kind of longer tail of labs as well. So to your point I think a lot of the capability gain has really come from just scaling that up and it's not an easy task. I mean maybe there's actually very curious on your opinions between the data collection and environments that are built by hand versus the ones that are done in Sim. There seems to be like different versions of like what could be what kind of price they command etc.

有好的方面和坏的方面，我确实认为有很多讨论，比如我们谈过蒸馏，但另一方面，就像中国公司购买他们 6 个月后使用的强化学习环境，然后像

have good things and bad things and I do think there's a lot of talk of like we've talked about distillation but on the other side it's like the Chinese company buys the RL environment that they use 6 months later and like

哦，Anthropic 或 OpenAI 买了这个，现在就有溢价了。

Oh Anthropic or OpenAI bought this now there's a premium.

就像好数据会这样。

Like the good data will do this.

是的。

Yeah.

但人们购买的数据相当有缺陷。就像我会说，那种方法是大量购买，然后你弄清楚什么有效，并不是 100% 的命中率。

But the data that people are buying is pretty flawed. Like I would say that that method is you buy in mass and then you figure out what works and it's not a 100% hit rate.

是的。

Yeah.

而且我个人认为，这种强化学习环境的爆炸式增长，有太多赚钱的机会，导致很多人偷工减料，这可能会引发奖励黑客之类的问题。所以就像模型太聪明了，如果你把它们放在一个世界里，它们就像我的工作是解决这个任务，但他们没有给我，就像你有一个测试文件，而测试文件实际上是一个空文件，而不是你必须导航的迷宫。模型就像，好吧，我仍然应该想办法获得更高的奖励。我认为很多这种行为是这条供应链的下游产物。

And I think I personally think that this RL environment explosion where there's just so much opportunity to make money is having a lot of people like cut corners and this can induce things like reward hacking. So it's like the models are so smart that if you put them in a world and they're like my job is to solve this task but they didn't give me the like you have a test file and like the test file actually has an empty file instead of the maze that you have to navigate. And they're like the model is like, well, I should still figure out how to get a higher reward anyways. And I think a lot of this behavior is downstream of this supply chain.

嗯。

Mhm.

我认为它会修复，但有趣的是看到实验室消除奖励黑客的需求，以及所有这些强化学习公司研究人员花这么多时间在这上面。然后所有数据公司说买我们的数据，我认为他们花在模型行为上的时间相对较少，更多时间花在世界构建和收入上。不知何故，这两方需要走到一起，才能真正制作出更高质量的数据。

And I think it'll fix it, but it's interesting to see the demand from the labs to eliminate reward hacking and all these re- company researchers spending so much time on it. And then all the data companies like buy our data and I think proportionally spend less time on model behavior and more time on world building and revenue. And somehow these two parties need to meet together to actually make higher quality data.

是的。嗯，很多实验室实际上正在将其内部化，部分原因是他们不想

Yeah. Well, a lot of the labs are actually taking it in house partially because they don't want

是的。

Yeah.

你知道，数据泄露。并不是说，你知道，字面上和比喻上说，就像信号泄露他们在做什么，但他们试图继续。但尽管如此，我认为他们的购买量有巨大的增长。

you know, data leaking. And not that, you know, like literally and figuratively speaking it's like, you know, signals leaking of what they're doing, but they're trying to send on. But despite that I think there's a tremendous amount of growth in how much they're purchasing.

你觉得这可能是最强的杠杆之一吗？你能说明一下这会有多大提升吗？你预期这会因为投入而增加吗？还是有其他因素？比如，如果能在模拟中做好，我想这或许更能说明它会被实验室吸收，尤其是。

Do you think that's probably one of the strongest levers? Can you qualify how much that will increase? Do you expect that to increase based on spending? Or are there other factors? For instance, if it can be done well in simulation, I think maybe that isn't more of an argument for that being absorbed by labs, in particular.

所有公司都在做混合的合成数据销售，他们用语言模型来修改或构建世界，或者生成其变体。如果人类引导这个过程，它仍然是合成数据。所以很少有人会坐在那里用 Blender 为我们的模型制作一个小世界来行动。我认为所有公司都在以远快于此的速度前进。

All the companies are doing a mix of selling synthetic data where they're using language models to modify or build the world or generate permutations of it. And if a human is guiding this process, it's still synthetic data. So there's very few people that are going to sit there and use Blender and make a little world for our model to act in. All the companies I think are moving far faster than that.

对。

Right.

他们购买这些数据，因为模型从中变得更好。

And they buy them because the models are getting better from them.

是的。

Yeah.

我仍然对此持怀疑态度。我想，这些数字太疯狂了。数十亿美元的研究预算花在环境上。我想，他们这样做只是因为它有助于模型，他们看到了信号。我的总体观察是，数据行业在提高质量和改进运营安全方面有很多低垂的果实。所以，如果你深入细节，你会发现使用海外承包商的数据公司有奇怪的事情。那些人通过中介。他们不知道自己在出售数据。他们只是上传到 GitHub。如果你在数据行业里探索，你会发现很多奇怪的粗糙边缘。

I still have denial about it. I'm like, these numbers are crazy. The billions of dollars research budget on environments. I'm like, they're only doing this because it is helping the models and they see the signal. And I think my general observation is that the data industry has a lot of low-hanging fruit on improving quality and improving just like opsec. So, if you look into the details, you can find weird things with data companies that use contractors overseas. Those people are through an intermediary. They don't know they're selling it. They just upload it to GitHub. There's a lot of stuff out there that if you poke around the data industry, you find weird rough edges.

是的。

Yeah.

而且这还很早。这就是为什么一方面数字太大，质量很差。但另一方面，实验室进步非常快，知道如何整合。所以，我认为这将需要几年时间过渡，真正将赢家拉向更高质量、更高投资回报率。当你拥有这些系统来做更好时，它只需要更少的资源。然后实验室继续与他们合作，因为他们获得更多的训练信号，并且他们必须做更少的内部过滤。我认为这只需要几年时间。

And it's just very early. And that's why it's the one thing is like the numbers are too big, the quality is bad. But on the other hand, the labs are improving very fast and know how to onboard it. So, I think this will take a few years to transition of really pulling the winners into higher quality, higher ROI. It just takes less resources when you have these systems for doing better. And then the labs keep working with them because they get even more training signal and they have to do less internal filtering. And I think this is going to just take years.

那么，作为你正在创立的非营利组织的一部分，你如何考虑在采购决策中使用环境？

How do you think about then using environments in your purchasing decisions as part of the nonprofit that you're starting?

我认为我们的大部分工作将是制作开源环境，用于非常高杠杆的任务。所以，我想到了我指导的华盛顿大学或华盛顿大学和 AI2 的学生 Hamish Iverson 的 TMAX 工作，这是用于终端任务的，在 Terminal Bench 上非常受欢迎，无论我们进行到哪个编号。这是完全合成数据，你提示一个前沿模型生成一种问题，你使用几个不同的提示来生成多样性。然后你使用模型 rollout 和其他东西来发现错误，然后消除有错误的。这个过程不是魔法。就像你阅读每个数据点，你的数据会变得更好，你可以做到这一点。但并没有那么多机构投资所需的东西，数千或数万美元来做好这些环境然后发布，因为利润动机如此之高。

I think most of our work will be making open-source environments for tasks that are very high leverage. So, I think of this work that I advised a UDub or a UDub and AI2 student, Hamish Iverson, on this TMAX work, which is you use a few It's for terminal tasks, which are like super popular at Terminal Bench, whatever number we're on. And this is fully synthetic data where you prompt a frontier model to generate a type of problem and you use a few different prompts to generate diversity. And then you use model rollouts and other things to find bugs and then eliminate ones with bugs. And this process is not magic. It's like you read every data point, your data will get better, and you can do this. But there's just not that many institutions that are investing what it takes, thousands or tens of thousands of dollars to make these environments well and then release it because the profit motive is so high.

高。

High.

我和作者们开玩笑。我说：“是的，你或许可以以一百万美元出售这些数据，然后发布论文。”他们说：“不，我们关心正确的事情。但这就像”

And you got I was joking with the authors. I'm like, "Yeah, you could probably sell this data for a million dollars and then release the paper." And they're like, "No, we like we care about the right thing. But it's just like"

天哪，是的。

God, yeah.

这些开放资源是许多研究的乐趣基础。就像你发布一些 RL 环境和 RL rollout，人们可以研究比如有效性意识，你有模型的生成，人们可以寻找不同的行为，如果你在上面做了 RL，他们可以看到它如何随时间变化。这只是博士生花一年时间研究的基础设施单元，并产生增加的知识，而科技行业是从计算机科学研究中获取价值的地方。所有计算机科学研究，利益在大约 10 年后归于科技行业。我只是认为这就是为什么更多人支持开放，如果你要在行业工作 10 年，你的同行将是受益于你的人。

These open resources are the fun foundation of so much study. And it's like you release some RL environments with some RL rollouts, people can do research on say like a valid awareness, you have the generations from the model, people can look for different behaviors, if you've done RL on it, they can see how it goes over time. And this is just the units of infrastructure that PhD students spend a year looking at and produce increase knowledge and the tech industry is the place where value is captured from CS research. All CS research, the benefit accrues to the tech industry in like 10 years. And I just think like this is why more people supporting open that if they're going to work if you're going to work in the industry for 10 years like your peers are going to be the ones that benefit from you.

我们不要耗尽它。

Let's not deplete it all.

但追求短期胜利的机会如此之高。在和朋友创立初创公司后的五个月内为你的企业赚取 1 亿美元（九位数回报）的机会，就像这些人可能做过更多开源工作，或者这些人离开了他们的博士项目之类的。这就像拉回来。这就是为什么我们为我们的实验室想出了 trillium 这个名字，trillium 是一种春季短命植物，它在树冠长出叶子之前开花。所以就像冬天，树上没有树叶，这些延龄草花开花，它们对于通过春夏生长周期传播养分至关重要。这就是我觉得 AI 行业做得很多正确的地方，就像很多都在顺利进行，但只是我们需要多一点生命来传播这个，让事情变得新人们可以进来。就像有这么多关注，就像做研究，做教育，

But it's just like the opportunity to pursue a short-term win is so high. The opportunity to make what is $100 million like nine-figure returns for your business within five months of founding a startup with your friends is just like these are people who may have done more open-source work or these are people who left their PhD or something. And it's just like pulling back. Which is why we came up with this name trillium for our lab, which is like trillium is a spring ephemeral flower, which it blooms before the canopy has grown its leaves. So it's like winter, there's no tree leaves on the trees, these trillium flowers bloom, and they're like essential for spreading nutrients through the cycle of growth that comes through spring and summer. And this is where I feel like the AI industry is doing so much right as like so much is going right, but it's just like we just need a little bit more life to like spread this out and make things so new people can come in. Like there's so much attention and just like do research, do education,

是的。

Yeah.

然后随着这个规模扩大并变得更有影响力，就像你有这些开放的东西，它们成为如此重要问题的研究标准，比如你如何理解沙盒？你如何定义什么是失控？你如何看到这在 RL 领域中如何变化，比如在 1000 个 GPU 或 100,000 个 GPU 上运行一周？就像让这些地方让人们有更多信任，我认为这为每个比我更可能获利的人带来如此多的红利，但很难让人们迈出这一步，就像我因为偶然进入这个领域而得到了如此多的回报，现在我得到更多，我可以做这样有趣的事情，就像现在是最好的工作，因为我试图为更多人创造空间，让他们发展独立思考并构建这个基础设施。

And then as this scales and becomes more impactful, it's like you have these open things which become the standards of study for such important problems, which is like how do you understand sandboxes? How do you define what loss of control is? How can you see how this changes in an RL realm that is a week on a thousand GPUs or 100,000 GPUs? And it's just like make these places so that people have more trust and I think this pays so much dividends to everybody that stands to profit more than I do and it's just like hard to get people to take that leap and like I've been rewarded so much for kind of stumbling into this area and now I get so much more and I get to do things like this which is fun and it's like now is the best job cuz I tried to create space for more people to like grow their independent thought and build this infrastructure.

是的，也许一个天真的方式让它延续下去，就是因为它是开源的，人们可以——你可能会得到更多来自那些人的分布式贡献，他们就是如此，你知道，就像他们的动机不仅仅是为了卖给实验室，而是因为嘿，它就在那里，他们可以分享他们的工作，然后你实际上有非常高质量的贡献者参与项目。我的意思是，那可能就是天真的希望。

Yeah, and maybe one like a naive way that it kind of perpetuates is just because it's open source, people get to you you might get more of a distributed kind of contribution from people who are just so, you know, it's like they're motivated not for just trying to sell to labs themselves but because hey, it's out there, they can share their work and then you actually have a very high caliber amount of contributors to the project. Like I mean that that would probably be the the naive hope of how that is.

是的，AI 智能体带来了一个新难题，人们会直接向你扔各种东西。比如我有一些随机的论文研究仓库，现在人们会开最疯狂的 GitHub issue。

Yeah, it's a new puzzle with AI agents where people just throw things at you. Like I have some random research repositories for research papers and people open the most crazy GitHub issues now.

你是什么意思？

What do you mean?

比如假的研究结果。他们开一个 issue，然后说——

Like fake research results. They open an issue and they're like—

我的天。

Oh my god.

然后他们说‘我在你的数据集里发现了这个东西’，我就想，如果你看了论文，我们在第二句话就说了。而且很多都是语言模型的输出，这对开源社区来说是个新问题。

And they're like, 'I found this thing in your dataset,' and I'm like, well, if you looked at the paper, we said that in like the second sentence. And it's just a lot of language model output, and that's a new thing for the open community to go through.

天哪。

Oh man.

是的。

Yeah.

但这是个可解决的问题。因为确实有人想为此做贡献，因为——

But this is a solvable problem. Like there are people out there that want to contribute to this stuff cuz like—

对，不只是在另一个维度追逐云，而是真的，嘿，它就在那里。我的意思是，我希望因为它已经发表、开放且可用，你实际上可以吸引博士生或其他研究人员，或者像‘好吧，我白天做这个，但也许我可以——’是的，他们仍然能够贡献。是的，希望来自前沿实验室。

Right, not just cloud chasing in another dimension but like actually hey, it's out there. I mean, I would just hope because it is published and open and available, you can actually just get like PhD students or other researchers or like, 'Okay, I do this in my day job but maybe I can—' Yeah, they're still able to contribute. Yeah, hopefully from the frontier labs.

我见过这样的人，比如有个来自东欧的人帮我改进我这本书的代码，就在这儿。他做这件事是出于正确的动机，我们会聊天，我就想确实有人这样做。我认为很多人只是没有时间或稳定性去投入所需的长久时间，因为当你试图进入一个新领域时，你一无所知。你得花几周时间阅读却什么都不懂，然后你发邮件没人回。你得花几周才能认识联系人，问出好问题。

Like I see people like this and I was like somebody from like Eastern Europe was helping me improve the code for my book that's just right here. And he's just like doing it for the right thing and like we would talk and I was like there are people that do it and I think a lot of people just take like they don't have the time or the stability to take the long time it takes cuz when you're trying to get into a new field, you know nothing. You have to spend like weeks reading and knowing nothing and then you send an email and nobody replies. And you're just like it takes you weeks to know the contacts like ask a good question.

是的。

Yeah.

而现在有了语言模型，大多数人只需按个按钮，然后说‘哦有趣，我读了这个，我发出去’。就像这样——

And now with language models, most people just push a button and they're like oh interesting, I read this, I'll send it off. And it's just like this—

哦是的。

Oh yeah.

这种沉浸其中是人们不太常做的事，而这是进入 AI 领域的关键，你必须专注于此。你得手写一些代码，坚持几个月，然后等 6 到 18 个月才能被注意到。这需要很长时间，而 AI 智能体我认为只会打乱这一点，让人们更难。

This kind of like sitting in it is the thing that people don't do as much and that is the key to like becoming into AI is that like you just have to focus on it. You have to write some code by hand and you have to do this for months and then you have to wait 6 to 18 months for you to get noticed. It just takes a long time and like AI all the AI agents I think just like frazzle that and makes it harder for people.

数学家们也抱怨这个。

It's what the mathematicians complain about, too.

就是这样。就像嗯——

That's all. It's like um—

所以现在读博很难，但学术界的人们仍在做很棒的工作，只是很难。

So it's a hard time to be a PhD student, but people are doing great work in academia still and but it's hard.

是的，但我同意你的看法。我认为这是可解决的，因为智能体在延续某种版本的问题的同时，也能帮助解决、识别和质量过滤。嗯，好吧，所以环境，你们将发布开源版本，帮助促使更多人贡献并揭示这一点。我认为这真的很——我的意思是，尤其是考虑到发布这个而不是从实验室快速获得收入的激励如此之少。嗯，另一个领域，我在你们的公告帖上读到的，是 RSI，也就是，你知道，相当——我想我们可以先谈谈它到底递归在哪里，或者它有多可怕，或者我们是否会起飞。我认为你对此有非常接地气的看法。嗯，但你知道，我还想谈的另一件事是，我认为 RSI，也许与它被炒作的程度有关，是我们担心前沿实验室会拉开差距的原因。嗯，事实上，这也是我听说过不止一次而是好几次的原因，人们确信自己作为研究者的工作将被自动化。

Yeah, but I agree with you. I think it's solvable as the agents kind of perpetuate some version of the problem, it can also help solve and identify and quality filter. Um okay, so environments you're going to release open source versions of that help precipitate maybe more people to contribute and reveal that. I think that's really—I mean especially given how little incentive there is to release this instead of just get, you know, fast revenue from the labs. Um another area I think I read on your announcement post is RSI, which is, you know, quite—I think I mean we can first talk about what is actually recursive about it or how scary it is or whether we're going to have takeoff. I think you have a very grounded opinion on that. Um but you know, another thing I want to kind of talk about is I think RSI, maybe related to how much it's kind of hyped, is a reason why we are afraid frontier labs will pull away. Um in fact, it's also a reason why I've heard, not just once but several times, that people are convinced that their own jobs are going to be automated as a researcher.

是的。是的，过去三年，我的一些最好的朋友，我说‘你们是世界上最擅长这个的人之一。’然后他们去上班，说‘Nathan，我很挣扎。我该怎么办？’

Yeah. Yeah, like I've gone to the last three years, some of my best friends, I'm like, 'You are some of the best people in the world at this.' And then they go to work and they're like, 'Nathan, I'm struggling. Like, what am I going to do?'

因为你的观点，他们做的很多事是可验证的，对吧？但正因为如此，实际上有很多其他非可验证或更难的分布外任务，比如对齐之类的。所以，实际上因为，我听到的一个论点是，RSI 在模型能力上的表现与对齐之间的差异如此之大，而对齐更偏向分布外，可能是创造性工作，这让人感到安心，也许确实存在危险等等。嗯，所以，无论如何，正因为如此，我非常想听听你的看法，因为我认为我们确实需要有一个开源阵线来展示 RSI 是如何进行的，这样也许我们能更好地理解能力，比如如果实验室里有更好的模型，他们真正拉开差距的可能性有多大，更好地理解和校准。嗯，所以，是的，我很想多听听那个。

Cuz your point, like a lot of what they do is verifiable, right? For um but because of that, you know, there's actually quite a lot of other sort of non-verifiable or harder out of distribution tasks, such as um uh uh with alignment and things like that. And so, actually because, I mean, one argument I heard was because there's such a difference between how well RSI is doing for model capabilities versus alignment, which is much more of out of distribution, maybe creative work, that gives people an ease on how, you know, maybe there is actual dangers, etc., etc. Um so, anyways, because of this, I'm very interested in hearing from you, like, how cuz I think it's really important that we do have an open-source um kind of front in in showing how RSI is being done, how much maybe then we can, you know, understand the capabilities of like if in within a lab they have a better model, what what are the chances they actually pull away, just better understanding, calibrating. Um so, yeah, I would love to hear more about that that that.

是的，所以我认为我们不会是唯一这样做的人，但本质上 RSI 有太多神秘感。他们说‘我们什么都用模型。’而作为开源实验室和非营利组织，我们能做的，我认为其他公司如 Eleuther、Hugging Face 或 Open Athena（Marine 和 Stanford 的），他们会带着这个好的营销口号进入这个领域。就像，我们可以分享我们使用的工具。我们将为 SFT、中期训练构建开源堆栈。我们将使用 SkyRL、Miles 等所有这些你知道的。我们有我们的任务，有我们的配置，知道用什么类型的计算机运行，我们可以用 Claude 和 GPT 等所有模型，我们只需说我们可以存储我们所做的轨迹，我们可以告诉你我们的任务是什么，或者你用 linear 或任何流行工具。它会说我们的任务是什么，为什么选择它们，模型是否能在上面工作，你这样做一年，然后说这就是我们所做的，然后每 6 个月检查一次，说它变了吗？就像构建全栈，不需要通过这些分类器，所以试着直接说任务是什么或模糊地说等等。就像能够分享正在发生的自动化的人应该这样做，我们有不同的任务分布，但我认为研究是相似的。就像我在 AI2 做的 AI 研究，我们将在 Trillian 做的，与学术界某些派别和实验室的人所做的，并没有那么不同。

Yeah, so I think this we won't be the only people doing this, but essentially like RSI has so much mystique. They're like, 'We're using models for everything.' And like, 'What we can do as a open lab and a nonprofit and I think other companies like Eleuther or Hugging Face or Open Athena, which is Marine and Stanford, they they will get into this with this good marketing line for them is in me as I'm doing this. And it's like, we can share what tools we're using. We're going to build an open-source stack for SFT, for mid-training. We're going to use SkyRL, Miles, all these things that you know. And we have our task, we have our configs, we know what type of computer running, we could work with Claude and GPT and all these models and we'll just be like we can store the trajectories for what we did, we can tell you what our tasks were or whatever you use linear or whatever these popular things are. It'll say what our tasks were and why we chose them and whether models work on them or not and you just do this for a year of a lab and you're like here's what we did and then you just check in every 6 months and you're like has it changed? And and it's just like to build to have the full stack and not need to go through these classifiers and so like try to just like say what the task is or vaguely and so on. It's just like people that can share what automation is happening should do this and we have a different distribution of tasks but I think that the research is similar. Like I think that the AI research that I've done at AI2 and we'll do it Trillian versus what people do in some sects of academia and the labs that's not that different.

你有更多资源，把它 Scaling（规模扩张）起来，而这对世界是件好事——当所有人都在讨论这件事时，就直接展示给大家看：这是 Nathan 让他的模型做的事，而这是它卡住的地方。我觉得这会帮到很多真正的政策制定者，让他们觉得，哦，原来是这样。

You have more resources, you scale it up, and it's just so good for the world when everybody's talking about this, to just show people: this is what Nathan asked his model to do, and this is what it got stuck on. And I think that will help a lot of literal policy makers, like, oh, okay.

就是它卡住的地方。就是它代码里被塞进去的那个 bug。

Like that's what it got stuck on. Like that's the bug that put in its code.

那些 bug 和粗糙的地方会随着时间减少，但我只是觉得，模型在一些根本性事情上的参差不齐——有些东西会起飞。就像我之前说的效率。我确实认为模型会非常擅长降低服务成本。这些公司有数十亿美元押在上面，而模型擅长这个。它们会通过让我的服务更高效、调整架构来赚很多钱。但这不一定是「你怎么解决知识工作」的问题。Claude 不会——你不会说，Claude，让你自己做出更好的 PowerPoint。就像，到底什么是——

Like the bugs and the sketchups will reduce over time, but I just think the jaggedness in some of these fundamental things of the models, like some more things will take off. Like I said, efficiency earlier. Like I do think the models will be really good at lowering serving costs. These companies have billions of dollars on the line and the models are good at it. They're going to make a lot of money on me serving more efficient and tweaking architecture in order to go into that. But it's not necessarily like, how do you solve knowledge work? Like Claude's not going to—you're not going to be like, Claude, make yourself tell—make better PowerPoints. Like what what what what is—

这得花上一段时间。

Like it's going to take a while.

对，对。

Yeah, yeah.

不，我很喜欢这种明确的目标，就是让我们把这件事去神秘化，因为我觉得围绕 RSI 的言论之所以变得这么激烈，很大程度上是因为它听起来——

No, I love the kind of specified goal of like, let's just demystify this, because I think a lot of why the rhetoric gets so heated around RSIs is like it sounds—

这是 AI 非营利组织可以做的很多事情之一。学术界很难做到，比如作为一个教授，我 15% 的时间和我研究生 40% 的时间都用来记录我们在做什么。所以这是一种很好的政策和社区服务：这是我们的任务。人们可以用这些数据来做评估之类的。就像，去构建那些含有稀有信息的东西，这样当人们投入更多高专注度的工作时，它就会对各种各样你无法预测的未来事情产生更多价值。

It's something AI nonprofits can do a lot of things like this. Like it's hard for academia to be like, as a professor, 15% of my time and 40% of my grad students' time is going to be documenting what we work on. So there's such a great like policy and community service of like, here are our tasks. People can use this data to make evaluations or whatever. And it's just like, just build things that have rare information so that when people put more high-focus work into it, it just has more value for all sorts of things that you can't predict in the future.

说实话，这也非常及时，因为，你知道，这不是来自一个人，而是我认识的实验室里的好几个人，就像，嘿，我要被自动化了，而你说的，你的朋友们，就像我作为我们社会中的一个人，肯定会在一年内被自动化。所以这很可能意味着，真正接触到组织架构图上那个特定部分正在做什么的人会变少。嗯，呃，所以嗯——

It's honestly just also very timely because the kind of, you know, it's not from one person but several people that I know from the labs, just like, hey, I'm going to be get automated, and you to your point, to your friends, like I will definitely as a person in our society will get automated in a year. And so that probably means there's going to be less people who actually are exposed to what's being done on that particular, you know, part of the org chart. Um, uh, and so um—

有一个相关的学术——有一个 open RSI 的东西，是一堆学术机构。我不隶属于它。他们上周才刚启动。所以我当时想，他们给我发了几封邮件。我想，open RSI 是这么回事。他们也在尝试做这个。就像它不需要只是——

There's a related academic—there's an open RSI thing, which is a bunch of academic institutions. I'm not affiliated with it. They just launched like last week. So I was like, they sent me a few emails. I'm like, open RSI is a thing. They're trying to do this as well. Like it doesn't need to just be—

一个团队。

One group.

还有很多——

There's a lot more—

用不同的方法来做这件事，因为你需要不同的看待方式，以及不同的结果。

To have different approaches doing this, 'cause you get—you need the different ways of looking at it and different results.

对，这就是开放的意义所在。

Yeah, whole point of being open.

我还听说的另一件事，更多是那种，你知道，前沿实验室的八卦，就是说实验室里实际使用的方法并不是——并不是它们不为人知。只是说，需要用什么比例才能成功。部分原因是他们有资本去 Scaling（规模扩张）、去实验。嗯，所以我觉得知道这一点挺令人振奋的，因为，好吧，并不是说正在发生的事情有什么大秘密。所以人们其实知道这些技术。但另一方面，你知道，资本仍然是唯一真正的壁垒，因为很难真正知道哪些技术、哪些配方有效。

The other thing I also heard was from more kind of, you know, frontier lab gossip, just like the actual methods that are being used in the labs are not that—it's not that they're unknown. It's just sort of like, what are the proportions that need to be used that are successful. And this is partially because they have the capital to scale it, to experiment. Um, so I think that's one heartening to know because like, okay, it's not like there's a big secret of what's going on. So people actually know the techniques. But on the other hand, you know, capital is the one true barrier still because it's hard to actually know which of the techniques, which recipes work.

嗯，这就是为什么我不做预训练。我很高兴能自由。我可以用任何基础模型。我现在——我现在能从竞争中受益。

Um, that's why I'm not pre-training. I'm happy to be free. I can use any base model. I'm—I'm now get to benefit from competition.

对。你觉得有没有可能做任何预训练的开源努力？我知道资本需求只是——大了好几个数量级，所以这才是困难所在。嗯，但也许它的价值是什么，然后也许我们可以讨论一下有意义的潜在商业策略。

Yeah. Do you think that there's like a chance for any pre-training open-source effort? I know the capital requirements are just like order—orders of magnitude larger, so that is what's difficult. Um, but maybe what are the values of that, and then maybe we can kind of discuss potential commercial strategies that make sense.

预训练科学相当——工具相当容易获取，而且你有所谓的预训练阶梯，就像你必须用来测试你的想法是否能扩展的决策工具。而预训练数据并没有像——像环境场景那样增长。基本上——粗略地说，你在互联网上训练。有很多——有一点——有一些细节和秘密的东西，但那并没有太大变化。最先进的架构在变，GPU 在变，那确实在变，但做一个扎实的预训练模型并不需要天文数字的开销。所以我觉得你不需要花十亿美元才能得到一个非常好的基础模型。你花得越多，效果越好，但——人们一直在取得进展，我总是惊讶于一个新实验室能从头开始，因为他们可以。其中一些是基于开源工具的。比如你可以基于 Megatron。很多新创公司，他们会说，哦，我们该训练什么？他们基于开源工具训练，fork 它，然后再也不发布。所有——甚至那些知名的开放模型实验室都这么做过。他们拿开源代码，做成自己的东西，然后——很多人不把它推回上游。所以我一直对此感到惊讶。所以我不看衰新进入者，因为并不是你需要 100 亿美元才能拥有一个相关的模型。难的是做出一个足够好、人们愿意用的模型。

Pre-training science is pretty—like the tools are pretty accessible, and you have what is a pre-training ladder, which is like the decision-making tool you have to test if your ideas scale. And the pre-training data isn't growing at like—like the environment scene. Like it's essentially—so first approximation, you train on the internet. And like there's a lot of—there's a little—there's a little details and secret things, but like that's not changing that much. The state-of-the-art architecture is, and the GPUs are, and that's like that changes, but it's not astronomically expensive to make a pre-trained model that is solid. So I think like you don't have to spend a billion dollars to get a very good base model. The more you spend, the better you will get, but it's—people keep making progress and I'm constantly surprised by a new lab just goes from scratch because they can. And some of this is on open tools. So like you can go off Megatron. Like a lot of new startups, they're like, oh, what should we train? They train—they train off the open-source tool and they fork it and then they never release it again. This is what all the—all the like even prominent open model labs have done. They take the open-source code, they make it their own thing and they—a lot of them don't push it upstream. So I've always been surprised by this. So I'm like not betting against new entrants because like it's not that you need 10 billion dollars to like have a relevant model. The hard thing is like get a model that's good enough for people to use.

嗯。

Mhm.

并且要有一个故事，让你一旦发布它，就能进行下一轮融资。这就像你必须跨过这个坎：我们需要我们的第一个模型落地。而你的整个公司都建立在「模型是最好的」之上。而你还没达到水准，你也不发布。就像你在倒计时。就像我想到 Thinking Machines 推出 Inkling，他们非常透明。他们说，这不是最好的模型。我们为它感到骄傲。它有一些很酷的东西。

And to get a story where if you release it, you can do your next fundraise. Which is like you have to get past this lock of like, we need our first model to land. And like your whole company is based on models are the best. And you're not in the ballpark and you're not releasing. Like you're on the clock. And it's like I think of like Thinking Machines getting Inkling out, where it's like they were very transparent. They're like, this wasn't the best model. We're proud of it. It has some cool things.

对。

Yeah.

而现在他们发布了第二个模型，开始运转起来，他们还有 Tinker 有这个。这对他们非常好。他们算是起飞了。就像——那——我不认为那是他们原本在做的事。他们在构建模型，但开放模型并不是他们的第一计划。然后他们走到了那一步。有很多——有传言说 Met——像 Meta 说他们会再次发布一个新的开放权重模型。那会来的。外面有很多公司买不起模型。

And like they're now that they've released their second model and they're getting going and they have Tinker that has this. And like that's very good for them. They're like off the ground. And it's just like—that—I don't think that's what they were doing. Like they were building models, but open models wasn't their first plan. And then they got there. And there's a lot of—there's rumors that Met—like Meta said they'll release a new open weight model again. Like that'll come. There's a lot of companies out there that can't afford models.

对，对。我知道我肯定，你知道，很高兴 Thinking Machines 是——而且这非常符合他们的商业模式，嗯，因为他们做后训练。

Yeah, yeah. I know I'm definitely, you know, glad that Thinking Machines is—and makes so much sense for their business model, um, 'cause they do post-training.

关于 Meta，我想你在之前的一些文章里也提到过。对他们来说，这真是一个很好的策略。我觉得他们这么做的时候也很明显，就像，嘿，这能补充我的收入来源。那为什么不呢？

With Meta, I think you mentioned this as well in some previous writings. It's just such a good strategy for them. I think it's also just kind of obvious when they do it. It's like, hey, it's complimentary to my revenue streams. So, why not?

那篇博客文章是这方面最好的。当 Llama 3 出来时，他写了最好的文章，阐述为什么开放是前进的道路。这家伙有点分心了。看起来他正在回归正轨。

Like that was the best blog post on this. When Llama 3 came out, he wrote the best like why open is path forward. The guy got a little distracted. Seems like he's coming back to the light.

对。对。是的，是的。

Right. Right. Yeah, yeah.

所以，我希望他们继续保持这种动力。在预训练方面达到前沿，我认为有很多帮助，因为有很多好的开源模型可以某种程度上仍然从中引导。我不断得到一些相互矛盾的信号，关于预训练是否真的已经饱和或充分理解。我的意思是，也许就像你说的，有很多众所周知的技术有效，但困难的是实际上要获得 20,000、30,000 个芯片来运行。

So, I hope they keep on being motivated to do that. Getting to the frontier with pre-training, I think there's a lot of help given that there's such good open source models to maybe somehow still kind of bootstrap from. I keep on getting kind of conflicting signals as to whether pre-training is actually as saturated or well understood. I mean maybe to your point there's a lot of well-known techniques that work, but what's hard is just actually having access to the 20,000, 30,000 chips to run.

正在推动预训练，而当超长上下文生成出现时，预训练也调整了。所以，如果你能让稀疏注意力架构也工作起来，那么你的 100,000 个 token 生成会快 40%，回报就大得多。

Are turning the crank on pre-training and pre-training became adjusted when super long context generation happened. So there's just way more return if you can for example get a sparse attention architecture to work as well and then your 100,000 token generation is 40% faster.

嗯。

Mhm.

所以我认为整个技术栈有很多共同进化，他们有自己的预训练栈，他们意识到推理的 Scaling（规模扩张）要高得多，但即使是预训练的基础也在不断改进，你只需要一个稍微更好的架构就能获得投资回报。然后它模糊地进入后训练，有中间训练，你开始推理行为，你从旧模型中获取它们，对它们进行强化学习来播种行为。就像这个循环

So I think there's a lot of co-evolution of the whole stack where they have their pre-training stack, they realize that inference scaling is so much higher, but even the fundamentals of pre-training keep improving where you just have a marginally better architecture that returns on investment. And then it kind of blurs into post-training where there's mid-training where you start the reasoning behaviors, you take them from your older models, you do RL on them to seed behaviors. Like this cycle

是的。

Yeah.

比预训练复杂得多。就像我认为，如果你没有合适的人，让它起步比预训练要难得多。就像我觉得，做一个受控的，比如让我们开始预训练我们的模型，对我来说似乎比我不太了解语言模型，我要弄清楚整个后训练的事情，就像狂野西部一样，要容易得多。

is much more complicated than pre-training. Like I think it's if you don't have the right people, it's a lot harder to get that off the ground than I think pre-training was. Like I think there's like I think like to do a controlled like let's just start pre-training our model seems a lot more approachable to me than I don't really know much about language models. I'm going to figure out this whole post-training thing where it's just like the wild west.

是的。

Yeah.

这就是为什么当我离开 AI2 时，我觉得我们的后训练老实说落后了，只是因为我们的周期时间不够高，而我们正在进入这些智能体式的东西。这些仍然是一些领先的开源后训练配方，这就是我想修复的，就像

And that's why I was like when I left AI2, I felt like our post-training was honestly falling behind just cuz our cycle time was not high enough and we're going into these agentic things. And these are still some of the like leading open source post-training recipes and to that's just like that's what I want to fix is like

是的，是的。

Yeah, yeah.

在开源领域，不需要那么多资本就能在最新技术上取得非常大的复合收益，因为门槛很低。就像它并不接近。

it doesn't take as much capital to make very big compounding gains on the state-of-the-art in the open source cuz the bar is very low. It's like it's not like it's not close.

是的，是的。不，这说得通。就像一种效率，你可以投入资本并产生巨大影响。是的，是的。

Yeah, yeah. No, that that makes sense. It's just like a kind of efficiency where you can put capital and make a huge difference. Yeah, yeah.

与整个后训练方面相关，也许从闭源和开源拉开距离，是事实，即智能体，闭源实验室中使用的 harness 类型非常不同，并且实际上训练和后训练得专门适合他们的模型，专门适合那些 harness。所以，这增加了神秘感，使它变得非常困难。你如何，我猜一是，你同意吗？然后二，比如你具体希望如何解决这个问题？我的意思是，真正的例子就像，嘿，启动数千个智能体，数万个智能体。这就像与人们显然接触到的非常不同的工作流程。

Kind of related to the whole post-training side maybe pulling away from closed and open, is the fact that the agents, the type of harnesses that are being used in the closed labs are very different and actually trained and post-trained on such a to be specifically good for their models are specifically good for those harnesses. And so, that increases the mystique even more, makes it very hard. How do you I guess one, do you agree? And then two, like how are you sort of hoping specifically to kind of address that? And things that I mean are really examples like, hey, spin up like thousands of agents, tens of thousands of agents. It's just like a very different workflow than what people are obviously exposed to.

我认为对开源来说会没事，因为几年前我认为闭源实验室会像苹果一样，他们是整合型公司，他们为涉及的 harness 设计模型，这给他们带来规模经济、效率，但事实证明，很多人有不同的 harness 用于他们喜欢做的不同事情，模型公司被迫使他们的模型能在各种不同的 harness 中工作。我认为开源模型的弱点是它们必须设计成能在各种类型的 harness 上训练，但结果却相当均衡。或者就像处于均衡状态，他们都必须在某种程度上这样做。然后我们看到这些智能体集群，OpenAI 和 Anthropic 会更早进入。他们会知道如何设计他们的模型，他们会知道如何设计我们称之为集群的 harness 什么？我不知道。就像

I think it'll be okay for open because I thought years ago that say the closed labs would be like Apple, they'd be the integrated company where they have the model designed for the harness which is involved, it gives them like kind of economies of scale, efficiency, but it turned out that so many people have different harnesses for different things that they like to do that the models companies have been pressurized to make it so that their model can work in all sorts of different harnesses. And I thought it would be a weakness of open models that they have to be designed to train on all sorts of types of harnesses, but it ended up being pretty equilibrated. Or like it's at equilibrium where they all have to do it to some capacity. And then what we're seeing with this like agent swarms is that OpenAI and Anthropic are going to be earlier to it. They're going to know how to design their models, they're going to know how to design what do we call what What do we call want to call a harness for a swarm? I don't know. Like

我不知道。我不知道，但会有一个词来形容它。它不是 harness。它像一个协调系统。但最终其他人会想要使用它们，然后我认为会像这些循环，前沿实验室推动这个，他们有这个。迄今为止，他们作为先行者获得了巨大回报，但像纳维-斯托克斯方程和所有这些事情的回报是什么？我们真的不知道。我们不知道。他们推动前沿，假设会有高回报，这种边际智能有不成比例的回报。我们不知道这是否总是正确。但我有点期待这些循环，一旦你知道如何在一个 harness 中控制 10,000 个智能体的形状，那么开源模型公司就会说，"哦，我们也能做到。"

I don't know. I don't know, but there's going to be a word for it. It's not a harness. It's like a coordination system. But then eventually other people are going to want to use them and then I think it's going to be like these cycles where the frontier labs like they push this out and they have this To date they've had huge returns on being first, but like what is the returns on Navier-Stokes and all these things? Like we don't really know. Like we don't know. They push the frontier in the assumption that there will be high returns and that this marginal marginal intelligence has like disproportionate returns. And we don't know if that's always true. But I kind of expect these cycles of like once you know the shape of how do you control 10,000 agents in one harness, then the open model companies will be like, "Oh, we can do this, too."

嗯。

Mhm.

但并不是说 Timmy 会先做到。首先，他们没有算力。

But it's not like Timmy is going to do that first. For one, they don't have the compute.

是的。

Yeah.

这有点像天文数字的算力。但这些东西会经历周期，视频，芯片会变得更快，有更多算力被构建出来。呃 这

It's sort of an astronomical amount of compute. But these things go through cycles and videos, chips will get way faster, there's more compute built out. Uh this

所以你相当确信，那里没有太多隐藏的东西，你大概能猜到他们在做什么。然后当它变得相关时，让开源模型实际上也尝试在类似的元 harness 上进行后训练，不管我们想给它们起什么名字。

So you're pretty convinced that like there's not too much hidden there that you can probably guess at what they're doing. And then when it becomes relevant, let the open source models actually try to also post train on similar meta harnesses, whatever we want to give them names.

我认为这些实验室在信息方面的安全性并不像他们想象的那么高。就像硅谷以想法传播而闻名，就像一杯啤酒什么的？就像每个人下班后都会聊天。

I think like the security at these labs in terms of information is not as high as they think. Like Silicon Valley is famous for like ideas traveling like what is it like one beer or something? And it's like everybody chats after work.

除了这就是为什么 SSI

Except which is why SSI

如此令人印象深刻。没有其他人知道发生了什么。

So impressive. Nobody else knows what's going on.

那是唯一的路。但就像他们都是科学家，如果他们取得了突破，他们会想告诉朋友。

That's the only path. But it's like they're all scientists if they made a breakthrough, they're going to want to tell their friends about it.

他们或许能保密几个月，但我不认为他们能一直保密。所需的信息安全级别太高了，知识总会扩散。他们能赢的地方在于把更多推理投入基础设施，从而以更低价格提供服务、获得更高利润，并长期持续。所以我很确信，前沿实验室能够以每瓦、每评估分数最低的成本提供智能。

They might hold it for a few months, but I don't think they can keep it secret. The amount of information security you would need is just so high that knowledge spreads. What they can win on is putting more inference into their infrastructure, which lets them serve at a lower price and make higher margins, and then continue this over a longer time. So I'm very confident in the frontier labs being able to serve the cheapest intelligence per watt, per score on an eval.

嗯。

Mhm.

他们能去需求最高的地方，提供最便宜的模型。那会带来收入，对他们来说是很好的生意。问题在于，他们无法包揽一切。这就是经济中其他部分填补的地方。只是编程有利可图吗？那么

They're going to be able to go to the places with the highest demand and have the cheapest models. And that will be revenue creative, that will be good business for them. The question is, they can't do everything. And that's where the rest of the economy fills in. Is it just coding that is profitable? Then

绝对不是。

Definitely not.

对前沿模型来说是个好案例，因为它们拥有最多的集成。

good case for frontier models cuz they will have the most like they have integration.

会做得很好，但是

do very well, but

整个知识工作经济都是它们的市场吗？那对它们可能不利，因为它们能同时做所有事吗？

Is it the whole economy of knowledge work is the market for things? And then it might not be good for them because can they do everything at once?

嗯，它们不需要。我认为会有很多赢家。

Well, they don't need to. I think there's going to be a lot of winners.

是的。

Yeah.

是的。我的意思是，这就是为什么我们投资前沿模型和初创公司。不是因为我们想对冲，而是因为我确实认为，你的观点是，边缘有足够多的更难任务，需要非常专业化的数据，这些数据某种程度上不在通往 AGI 或无论你现在怎么称呼它的溅射区。这使得有很多很多赢家成为可能。

Yeah. I mean, that's why we invest in the frontier models and in startups, you know. Not because we want to hedge, but because I actually do think, you know, to kind of cooperate your view that there's enough harder tasks at the fringe that requires very specialized data that's kind of out of the splash zone for getting to AGI or whatever you want to call it these days. That makes it possible to have many, many winners.

是的。

Yeah.

我确实喜欢你的 RSI 观点，因为我们稍微谈过，就是实际地说，“嘿，成本是指数级的，资源消耗也是，因此前沿实验室能拉开多大差距有一个非常明显的限制。”如果这没有过度简化你的观点。你还想补充什么吗？因为我确实觉得现在的讨论太恐惧了，比如前沿有多隐蔽，以及正在突破。

I do like your RSI point cuz we talked a little bit about it, which is just kind of say practically, "Hey, it's exponential cost and resource consumption and therefore there's a very obvious kind of limit to how much that um the frontier labs can pull away. Um if that's not too oversimplification of your view. Um anything else you kind of want to add there because I I do feel like the the the discourse now is just so so much fear on like how how much the frontier is just hidden and also breaking away.

我认为在推理上的花费如此之大，短期内我们正在学习如何将推理转化为性能提升。这就像我们刚刚经历的事情。我们现在看到的是，人们花费如此之多，前沿实验室的所有员工相对于改进都在花费指数级的金额，但我们正在弄清楚如何做到这一点，会有很多收益。但我不认为这一定会是模型自我构建时的快速起飞，那是人类瓶颈。所有的缩放定律都是算力指数增长换取性能线性增长，这就像我只是不得不更好地反对这一点。我确实说，我们有纳维-斯托克斯方程，我认为数学特别适合。也许是最适合这种智能形态的科学领域。智能可以采取许多不同形式。这并不是说，如果 AI 模型发现了与 Transformer 非常不同的东西，带来非常不同的属性并解锁一种新型智能。那确实值得更新，值得说“好吧，那不一样。”但在那发生之前，我对我的证据相当满意，就像

I think there's so much spend on inference and that in the short term we are learning how to corral inference into performance gains. And this is like a thing that we just had. And like this is what we are seeing now is that like people are spending so much and like all the employees at the frontier labs are spending exponential amounts relative to the improvements, but like we're figuring out how to do this and there will be a lot of gains. But I don't think that it's necessarily going to be this like fast takeoff when the models build themselves which is human human bottlenecks. The all the scaling laws are exponential increase in compute for linear increase in performance and ex- like that's just like I don't like I just had to better against this. I do say that like we have Navier-Stokes and I'm of the opinion that math is exceptionally well suited. Maybe like the scientific field that is best suited to this shape of intelligence. And like intelligence can take many different forms. And that's not to say that like if they do like if an AI model does say discover something that's very different than the transformer that gives very different properties and unlocks a new type of intelligence. Like it is worth updating on this and worth being like, "Okay, like that is different." But until that happens, I'm like pretty happy with my evidence of like

公司会做得很好。他们花费指数级更多的钱来保持这种惊人的进步速度。一些 Claude 系统卡这么说。他们说，比如需要 10 倍更多的进步才能实现快速起飞，而现在它们帮助我们维持进步速度。我认为这是一个好的平衡——我说这是一个好的均衡，但我确实认为实验室有人这么说。科学很难克服这一点。这没关系，但我们不知道接下来会怎样。

the companies are going to do well. They're spending exponentially more to keep this remarkable pace of progress. That some of the Claude system cards say this. They're like, it would take say like 10x more progress to make this a fast take off and it's like for now they're helping us sustain the pace of progress. And I think it's a good equal- I was like it's a good equilibrium, but I do think that like the labs have people saying this. The the science is like it's hard to overcome that. And it's okay, but it's like we don't know exactly what will go from here.

是的，是的。不，我认为这还好，令人安心，因为我们知道这里的输入和杠杆是什么。只是有一个非常成功的自回归机制，我们一直在优化它，它只是需要一定量的指数资源，而这些资源显然有上限，因此希望开源也能保持强劲。它不会落后。

Yeah, yeah. No, I I I think that's okay com- comforting cuz we know what the inputs and and levers are here. It's just there is a very successful autoregressive regime that we've been optimizing and that one just, you know, asks for a certain amount of like exponential resources which are clearly capped and therefore open source hopefully, you know, can keeps it strong as well. It just isn't going to lag behind.

在基础设施准备方面有大量工作要做。比如 AI 能力曲线并不一定让我担心。但 AI 能力曲线与几乎不存在的跨公共网络安全准备等所有这些事情，需要在假设这即将到来的前提下做更多准备。这比说 RSI 即将到来并解锁新风险更让我担心。有些风险已经存在，并将继续扩散。多少次说过，闭源实验室达到那里，然后开源实验室也达到那里。我们正处于这个过程中，很多事会引发一些问题。只是弄清楚如何让社会做好准备非常困难。

There is a dramatic amount of work to do in preparing for infrastructure. Like there's the AI capability plot which doesn't necessarily worry me. But the AI capability plot in the pretty much non-existent preparation of cross public cybersecurity and all of these things like there needs to be more preparation in the assumption that this is coming than there is. And then that's kind of the thing that worries me more than say like RSI is coming and unlocking new risks. Like there are risks that are here and that are going to continue to diffuse. How many times have said that closed labs get there and then the open labs get there. And it's like we're in this process for a lot of things that are going to cause some problems. It's just like figuring out how to get society to prepare is very hard.

是的。

Yeah.

我认为我们需要更多双眼睛，需要更多科学，让社会感到知情。

And I come of the opinion that we need more eyes, we need more science so that society feels informed.

是的。

Yeah.

但也有其他合理的观点，我确实认为，不害怕快速起飞并不意味着你不担心风险。然后你不应该

But there are other valid opinions and like I do think that not fearing fast takeoff does not mean that you don't worry about risks. And then like you shouldn't

不，我认为那完全——也许我会介绍一些从实验室叛逃的人给你，他们正是关于这个的。嗯，非常感谢你，Nathan。这很有趣，嗯，是的，我很兴奋看到 Gillian 终于推出后的进展。

No, I think that's completely I'm maybe I'll I'll send in some people your way that are defected from the labs who weren't precisely about this. Well, thank you so much, Nathan. This was a lot of fun and um yeah, I'm really excited to see what Gillian is up to now that it's finally launched.

是的，非常感谢邀请我。

Yeah, thanks so much for having me.