John Schulman、Charlie O'Neill 与 Beren Millidge 激辩递归自我改进还有多远

今天我和三位 AI 研究员朋友聊天，每次交谈我都从他们身上学到很多。他们恰好都在相对开放的实验室和公司，所以你们可以公开讲一些东西。和我一起的是 Beren，他是 Zyra 的 CTO，公司在开发开源模型；John Schulman，Thinking Machines 的首席科学家，此前是 OpenAI 的联合创始人，主导了催生 ChatGPT 的 RLHF 工作；还有 Charlie O'Neill，Baseten 的模型训练负责人。

Today I'm chatting with three of my AI researcher friends, from whom I learn a lot every time we talk, and who also happen to be at somewhat open-ish labs and companies, so you guys can actually say things on the record. I'm joined by Beren, who is the CTO of Zyra, which is developing open source models; John Schulman, who is the chief scientist at Thinking Machines, previously the co-founder of OpenAI, and led the RLHF work that led to ChatGPT; and Charlie O'Neill, who is head of model training at Base 10.

我的第一个问题是：假如到了 2036 年，已经过去 10 年，我们并没有成千上万、疯狂运转、彻底改变世界的超级智能，那么最可能的原因是什么？排除外生的政治冲击，比如爆发战争、或者 AI 被禁之类，最可能的技术原因是什么，让 2036 年没有变成一个疯狂的异形超级智能世界？

The first question I have: if we're in 2036, it's been 10 years, and we don't have crazy billions of crazy superintelligences running around that radically transform the world, what is the most likely reason that doesn't end up being the case? Other than exogenous political shocks, or there's a war, or they ban AI or something — what is the most likely technical reason that 2036 isn't a crazy alien superintelligence world?

我的理由大概是，有一种经典现象，几乎像莫拉维克悖论，对吧？我们想象 AI 是这样的——如果它能做这件事，那就会很了不起，对吧？如果它能解这些难题、能赢国际象棋等等。然后它真的解决了这些事，结果却没那么大影响。显然它有一定影响，但不是一切。如果这种情况一直持续，始终没有出现真正的泛化火花，我觉得那可能导致 AI 只是在人们放进基准、放进环境里的所有事情上极其擅长，但仍然存在一些持续的“仿真到现实”的鸿沟，不知怎么地卡住了一切。

I mean, my reason would just be that there's been a classic thing, almost like Moravec's paradox, right? We think of the AI being like — if it can do this, it's going to be amazing, right? If it can solve these hard math problems, if it can win at chess, blah blah. And then it solves these things, and then it's like, not that impactful. Obviously it's somewhat impactful, but not everything. It's like if somehow that continues, and there's never the true spark of generalization that occurs, I think that could lead to the AI just being extremely good at kind of everything that people put into a benchmark, put into an environment, but there are still some persistent sim-to-real gaps that are somehow blocking everything.

我觉得这有点不太可能。我认为在实践中，我们确实已经看到强化学习也能带来这种泛化。但如果泛化就是难得离谱——元学习，加上我们没解决持续学习，就是超级难、不可能——那在这种情况下，这会是我的默认情景。

I think this is kind of unlikely. I think we do actually see this kind of generalization even from RL in practice already. But if it is just ridiculously hard to generalize — meta-learning plus we don't solve continual learning, it's just super hard and impossible — yeah, this would be my default scenario in that case.

是的，我同意。人类现在相对模型有很多优势，每次新模型出来，它会在其中一些方面追上来。但你最终会被模型较弱的环节卡住——它判断力更差的地方，或者模型没法足够好地自我检查的地方。所以有一个不断重复的循环：新模型出来，人们惊叹，说就是它了，这就是 AGI。但用一阵子之后，大约一个月后它就开始显得笨了。这个循环可能就一直持续，很难预测它会重复多少次。而现在你得不到能力的爆炸式增长，因为做研究和工程时你仍然会被卡得够呛——即使模型能写的代码比人多得多，也不会让你效率提高 100 倍。所以，也许只是这类循环比我们预期的更多。

Yeah, I agree with that. Humans have a lot of advantages over models now, and each time a new model comes out, it'll sort of catch up in some of these areas. But you end up getting bottlenecked by the places where the model is weaker, where it has worse judgment, or where the models can't check themselves well enough. So there's this cycle that keeps repeating, where a new model comes out and people are blown away and they're like, this is it, this is AGI. But then they use it a bit, and then it starts to feel dumb after a month or so. So that cycle just might keep going, and it's hard to predict how many times it's going to repeat. And right now you don't get explosive growth in capabilities, because you still get bottlenecked enough when you're trying to do research and engineering that even if the model can write way more code than a person, it doesn't make you 100 times more productive. But yeah, so maybe there are just more of these cycles than we would expect.

对我来说，问题在于芯片上能有的学习器的全局最优离我们有多远——也就是 Transformer 加强化学习，基本上就是当前的配方。人们想象，一旦你有一个在 AI 研究上比所有人类都强的智能体，哪怕只强 0.1%，那么你能并行运行几十万甚至上百万个，能跑得快得多，芯片还会加速——这会压过其他所有瓶颈，最终你就会迎来递归自我改进带来的极快起飞。

For me, it's a question of how far off the global optimum of a learner you could have on a chip is — like the Transformer plus RL, basically the current recipe. People imagine that once you have an agent which is better than all humans at AI research, even if it's like 0.1% better than all humans, then the fact that you can run hundreds of thousands if not millions of these in parallel, you can run them much faster, chips are going to speed up — that's going to outweigh every other bottleneck, and you're eventually just going to hit this very fast takeoff with recursive self-improvement.

我可以想象，如果我们沿着当前这个范式的轨迹继续走——基本上就是自注意力、强化学习、扩大强化学习环境——想想摩尔定律发生了什么，对吧？我们有过一条非常漂亮的直线，它维持了非常非常久，但为了维持那个缩放定律，必须发生许多离散的不连续性和创新。语言模型也发生了类似的事：我们有过预训练的缩放定律，然后它开始出现收益递减，接着我们提出强化学习解决了它，然后又得到一条新的收益递减曲线要突破，这让它看起来一直像一条上升的直线。所以，如果解决它需要又一次那样的不连续性——我不确定当前用这些强化学习环境训练语言模型的方法，即便是针对递归自我改进的强化学习环境，能否发现那个不连续性。如果不能，我们可能就会撞上这条渐近曲线，然后——

I could imagine that if we continue along the trajectory we're currently on with that paradigm — where it's basically just self-attention, RL, scaling up RL environments — I guess if you think about what happened with Moore's law, right? We had this very nice straight line, and that held for a really, really long time, but there were so many discrete discontinuities and innovations that had to happen to keep that scaling law going. And the same thing has kind of happened with LMs: we had this pre-training scaling law, and then that was kind of hitting diminishing returns, and then we came up with RL and solved that, and then we got this new diminishing-returns curve to hit, which made it keep looking like a straight line going up. So if it requires another one of those discontinuities to solve — I'm not sure that the current method of training LMs with these RL environments, even RSI-targeted RL environments, would be able to discover that discontinuity. And if not, we're probably going to hit this asymptotic curve where —

抱歉打断一下，你觉得这个不连续性会比 2012 年以来出现的任何东西都更难吗？

Sorry, but do you think the discontinuity will be harder than anything that's come since 2012?

如果我们知道答案，我们大概就有能力实现它了。但也许我们应该区分两种不连续性：一种是给当前范式做加法的——同样是累积的——另一种是我们必须发现的、超越强化学习的东西。也许它们有能力在那条直线上把点连起来。

If we had the answer to that, we'd kind of have the ability to implement it. But maybe we should distinguish between a discontinuity which adds to the current paradigm — again, it's cumulative — and something beyond RL that we have to discover. And maybe they're capable of connecting the dots in that straight line.

或者，还是那个问题：我们离全局最优有多远？我们是否必须回头把梯度下降和神经网络整个扔掉？我不认为——如果你继续扩大当前范式，一个 LLM，无论你运行多少个语言模型，如果那个东西太远，它未必有能力发现它。

Or, again, how far off the global optimum are we? Do we have to go back and throw out gradient descent and neural nets in general? And I don't think — if you continue to scale up the current paradigm, an LLM, no matter how many LMs you're running, is necessarily capable of discovering that if it's too far away.

是的。唯一的希望其实在于：深度学习就是没法把我们带到一个至少能主导人类研究和人类发展的 AI，包括人类提出新范式等等的能力。或者，我不知道，也许人类也永远不会发现下一个学习架构。但就人类最终有可能发现它而言——只是看起来，我不知道，如果你看看 2012 年至今的进展，然后就这么继续下去。

Yeah. The only hope really is if deep learning just can't get us to an AI which can at least dominate human research and human development, including the human ability to come up with new paradigms and so forth. Or, I don't know, maybe humans would also never have discovered the next learning architecture. But to the extent humans could have discovered it eventually — it just seems like, I don't know, if you just look at the progress that's happened from 2012 till now and you just continued that on.

我的意思是，我知道它是由大量算力规模扩张等驱动的，但如果它没有达到至少能在研发领域主导人类的地步，那就很奇怪了，尤其是在未来几年。Ryan Greenblatt 最近上了播客，他提出了一个观点：你可以想象，随着 AI 能力越来越强，并且能够在模拟中取得进展，这会激励不仅提升 AI 研发能力，还能普遍提升科学能力。所以这是所有实验室都在瞄准的目标，许多初创公司也在瞄准。另一个直觉泵是，如果你看自 80 年代以来国际象棋机器人的 ELO 分数，ELO 随时间呈非常线性的增长，但当它们跨越人类专家的范围时，会出现巨大的不连续性。人类专家总是能赢 AI，然后随着这种线性增长，人类专家再也赢不了 AI。你可能会想，我同意你的观点，到目前为止 AI 能力在最终经济影响方面还没有那么大的影响。但那是因为它们相对于人类的 ELO 在缓慢上升。

I mean, I know it's been powered by huge amounts of compute scaling and so forth, but it would be weird if it just didn't get to the point where it could dominate humans, at least in R&D, especially over the next few years. Ryan Greenblatt was on the podcast recently and he made this point that you could imagine as AI get more and more capable and are capable of making progress on simulations which incentivize getting better at not only AI R&D, but generally at science. So this is a thing that all the labs are targeting, many startups are targeting. Another intuition pump is if you look at the ELO score of chess bots since the 80s, there's just a very linear increase in ELO over time, but there's this huge discontinuity as they cross the human range of human experts. Human experts always win against AIs, then human experts never win against AIs as this linear increase happens. You could think I agree with your point that so far AI capabilities have not been that big of a deal in terms of their end economic impact in the world. But that is because they're slowly rising in ELO relative to humans.

是的。

Yeah.

是的。我的意思是，我同意这将会非常——我的意思是，唯一不让这种情况发生的方式是，如你所说，不知何故 ASI 就在——基本上因为在我看来，我们已经非常接近开始跨越人类 ELO 分数的时刻，所以我们需要在那之前上升。这是唯一的方式，你知道，在你设想的场景中，我们坐在 2035 年，一切正常，这种情况才会发生。我想唯一的另一种方式是对 AI 进行某种戏剧性的监管。实际上，我认为这可能是这种情况发生的最可能方式，而不是技术上的原因。

Yeah. I mean I agree it would be very—I mean the only way for this to not happen is if, as you say, somehow ASI just before—basically because we're already pretty close in my opinion to where we'll start crossing the human ELO score, and so we'll need to ascent before that. And that's the only way, you know, in the scenario you pose where somehow we're sitting here in 2035 and everything is normal for this to happen. I think the only other way is if there's some dramatic regulation on AI. It's kind of what I see as the most likely way for this scenario to happen actually, rather than the technical thing.

是的，我认为有不同类型的研究。有一种是自动研究风格，目标已经非常清晰地指定，你正在优化那个目标。我想每个人都在想象，如果我们继续沿着这条路径，让预训练损失下降，让我们的环境，让水位上升，那将导致改进。但也许 Ryan 谈论的是这种更加开放式的科学，这是范式转变所必需的，我们无法指定目标，AI 也绝对无法指定那个目标。就像我们必须非常非常小心地为任何这些事情指定目标。

Yeah, I think there's different kinds of research. There's research where it's like the auto research style where the objective is already specified very cleanly and you're optimizing that objective. And I think everyone is picturing like if we continue along this path of making pre-training loss go down, making our environments, making water go up, that's going to lead to improvement. But maybe what Ryan is talking about is this much more open-ended type of science which is required for paradigm shifts where we can't specify the objective and the AIs are definitely not able to specify that objective either. Like we have to be really really careful about how we specify objectives for any of these things.

也许你的观点是，自 2012 年以来发生的突破的本质是，我们发现——就像在 2012 年，人们并没有说——或者我假设，我不知道，你们当时在场，或者至少 John 你在场，但我——

And maybe your point is that the nature of the breakthroughs that have happened since 2012 is that we have found—like in 2012 people weren't saying—or I'm assuming, I don't know, you guys were there or at least John you were there, but I was—

我当时在小学。

I was in primary school.

实际上，John，我很好奇你的岁月智慧，或者当年在战壕里的智慧，但大概一个重大突破是意识到下一个词预测是——你不会想到在 2014 年 nanoGPT 速度跑是要优化的东西。但现在我们已经进入了这个新范式，你不会想到对此进行速度跑并让你们变得非常擅长。但也许有一个下一个内循环需要优化，而 AI 不会预料到。还有一个收入的外循环或其他东西，最终应该会很强。但这是一个非常慢的外循环。

Actually, John, I'm curious for your wisdom of the ages, or wisdom of being in the trenches way back when, but presumably a big breakthrough was realizing that next token prediction is—you wouldn't have thought that nanoGPT speedrun is the thing to be optimizing for in 2014. But now that we have come to this new paradigm, you wouldn't think to do a speedrun on that and have you guys get really good at that. But maybe there's a next inner loop to optimize that the AIs wouldn't anticipate. And there's an outer loop of revenue or something that eventually should be strong. But it's a very slow outer loop.

是的。事实上，我记得在 OpenAI 早期，有一种直觉，即仅仅最小化对数损失并不能让你达到智能，因为重要的部分只占损失的一小部分，会被噪声淹没。所以仅仅在下一个词预测上训练语言模型，不会学到你想学到的有趣东西。我们需要设计更好的目标，更强调重要的东西。你可以为此提出各种论据，你可以说，哦，人类可能不会学习如何——我们不会学习如何建模环境中的一切。大多数人无法对他们看过的某个场景进行照片级逼真的再现。所以一定有——我们一定需要更好的目标。但后来发现它还是奏效了。

Yeah. In fact, I remember in the early OpenAI days having the intuition that actually just minimizing log loss wasn't going to get you to intelligence because the important bits are accounting for such a small fraction of the loss that it was going to be overwhelmed by noise. So just training a language model on next token prediction just wasn't going to learn the interesting things you wanted to learn. And we needed to craft better objectives that would put more emphasis on the important things. And you can make all sorts of arguments for this and you could say, oh humans probably don't learn how to—we don't learn how to model everything in our environment. Most people can't create a photorealistic reproduction of some kind of scene they've looked at. So there must be—we must need a better objective. But then it turned out that it just worked anyway.

正如你指出的，即使在当前 AI 研究中，后训练基准测试之类的内循环也不一定转化为用户喜欢的东西。

And as you were pointing out, the inner loop even in current AI research of like post-training benchmarks or whatever doesn't necessarily translate into what users like.

哦是的。我的意思是，整个领域非常依赖泛化，很难预测你什么时候会得到泛化，或者什么时候会得到某种分布外泛化。所以我们知道，如果你在你关心的任务上训练，你会做得更好。

Oh yeah. I mean the whole field relies a lot on generalization and it's very hard to predict when you're going to get generalization or when you're going to get some kind of out of distribution generalization. So we know that if you train on the task you care about you're going to do better.

是的。

Yeah.

但最重要的进步往往是那种我们本无权期待的泛化。例如，仅仅在这个非常朴素的下一个词预测目标上进行预训练，然后应用到各种感兴趣的任务上，其中一些任务需要以某种深刻的方式理解输入，或者从预训练中学习某种非常罕见且代表性不强的技能。然后还有从这些可验证任务到不太可验证任务的泛化。这也是一种泛化——

But the most important advances are often the types of generalization that we have no right to expect. So for example from just pre-training on this very naive next token prediction objective to various tasks of interest where some require understanding of the input in some deep way or learning some skill from pre-training that's very rare and not very heavily represented. And then also generalization from these verifiable tasks to less verifiable ones. This is also a type of generalization that—

先验上没有理由期待它。

There's no reason a priori to expect it.

是的。是的。

Yeah. Yeah.

所以这是一个有趣的问题，因为一个直觉泵是，为什么你会看到某种奇点迅速出现，甚至不需要扩大 AI 进展的输入（不仅仅是 AI 劳动力），那就是在你运行每一个七位数实验之前，你在 AI 劳动力上花费同等数量的算力，所以你就有你们这些人的自动化版本，花一个世纪思考什么是最优实验，进行小规模消融，发展 literally 一个世纪的理论。所以甚至在深度学习之前，在你决定运行什么实验之前，进行极其优化的实验设置。然后实验结束后，你花一个世纪思考，分析发生了什么以及下一个实验是什么。

So this is an interesting question because one intuition pump that you could have for why you would see some sort of singularity very rapidly without even scaling up the inputs to AI progress that are not just AI labor is that before every single experiment you run that's like a seven figure experiment you spend an equivalent amount of compute on AI labor and so you just have automated versions of you guys spending a century thinking about what is the optimal experiment to run, doing small scale ablations, developing literally a century's worth of theory. So going back even before deep learning, before you decide what experiment to run, doing extremely optimal setting up of the experiment. Then you do a century of thinking after the experiment is over where you're analyzing what happened and what the next experiment to run is.

是的。嗯，我认为如果你思考得足够深入，你大概可以事先预料到一些事情。就像可能有某种非常聪明的方法来做小规模实验，让你建立理论，然后推广到大规模实验。

Yeah. Well, I think if you think hard enough, you probably could have expected some of these things beforehand. Like there is probably some very clever way to do a small scale experiment that'll let you build the theory that then will generalize to the large scale experiments.

所以我认为，我们在研究上能达到的水平还远未触及天花板。我可以想象这样一个未来：AI 进行大量的分析和理论构建，花费与实验本身相当的算力，进行各种分析，并围绕我们目前所见构建理论。

So I would expect that we're nowhere near the ceiling of how well you can do research. And I would imagine a future where AI is doing a lot of analysis and theory building, spending a comparable amount of compute to the amount that you're spending on the experiments themselves, doing various kinds of analysis and building a theory around what we've seen so far.

我认为当目标明确时，有非常具体的例子。所以，思考所能做的只是根据你形成先验后获得的比特来更新你的后验。仅仅思考并不能获得任何新的比特。

I think there's really concrete examples of this when the objective is well specified. So again, all thinking can do is update your posterior based on the bits that you've gotten since you formed your prior. You can't gain any new bits from just thinking.

但当目标明确且周围有这些数据时，我想在当前范式下会有很大的加速。一个很好的例子是，如果你让 AI 思考 Kaplan 缩放定律，此时的 AI 会注意到他们只是取了这些中间检查点，而没有考虑退火，所以这是错误的，那会早几年发现。我们本可以取得进展——仅从 AI 的那个观察就能节省一两年的进展。

But when the objective is well specified and there is this data sitting around, I imagine there will be this big speed up in the current paradigm we're in. And a good example of this is if you got an AI to think about the Kaplan scaling laws, an AI at this point would have noticed that they've just taken these intermediate checkpoints and didn't account for the annealing, and so this is wrong, and that would have caught that years earlier. We would have made progress—would have cut off a year or two of progress just from that observation from an AI.

再说一次，一旦目标明确，比如降低预训练损失之类的，有很多很多好例子，如果你再多思考一下，就能大幅减少你已经做的事情。比如 mu 以及学习率如何随模型规模缩放，并意识到模型宽度在其中也很重要。我觉得你真的可以推导出很多这些东西，并摘掉很多低垂的果实。所以，如果我们的目标只是最大化当前的目标，我想象会有 10 倍的加速。

And again, once the objective is well specified, which is like lower pre-training loss or whatever, there's many many good examples where if you just thought about it a bit more you would have been able to cut down significantly on things that you've done. So like mu and how learning rate scales with model size and realizing that model width is important in that as well. I feel like you can really back out a lot of these things and cut off a lot of low hanging fruit. So I would imagine a 10 times speed up if our thing is just to maximize the objective we're currently on.

但我看不出这如何推广到首先提出正确的目标。仅仅思考并不一定能让你首先获得正确的目标。

But I don't see how that generalizes at all to coming up with the right objective in the first place. Just thinking doesn't necessarily buy you the right objective in the first place.

是的，我认为这对当前 AI 的任何快速递归自我改进来说都是关键问题：AI 在多大程度上能泛化到学习自己的目标？因为要有任何自驱动的自动化循环，你需要 AI 提出目标、优化它们、弄清楚，提出新目标，并确保这在很长很长一段时间内任何时候都不偏离轨道。回到 MVX 产品，可能有 MX 产品的情况，我们认为这种自主性和自我封装，以便我们可以自己思考应该做什么然后去做，并拥有这个循环是超级容易的，因为我们总是这样做。显然，进化需要创造能够长时间独立生存的生物，而这可能对 AI 来说由于某种原因非常困难，就像运动 stuff 非常困难一样。数学超级容易，尽管对我们来说超级难。

Yeah, I think this is really the key question to any kind of very rapid RSIs from current AI: how well can AI generalize to learning their own objectives? Because to have any kind of self-propelling automated loop, you need the AI to propose objectives, optimize them, figure that out, propose a new objective, and have this not go off the rails at any point for a long long time. Coming back to MVX products, there might be a case of MX products where we think this kind of autonomy and being self-encapsulated so we can think of what we should do ourselves and then go do it and have this loop is super easy because we always do this. And obviously evolution needs to create creatures that can survive on by themselves for long periods of time, and this just might be something that for some reason is really hard for the AI in the same way that locomotion stuff is really hard. Math is super easy despite being super hard for us.

我不知道。时间跨度的增加难道不表明那是——

I don't know. Doesn't the time horizon increasing suggest that that's—

是的，没错。我是说这是另一种可能性，但我同意没有明显的证据。事实上——

Yeah, exactly. I mean this is another possibility, but I agree there's no obvious evidence for this. In fact—

你知道智能体现在超级持久，而且做这个相当容易，这有点是反对这一点的证据。

The fact that you know agent is now super persistent and it's quite easy to do this is kind of evidence against this.

但这可能是我们没有立即起飞的原因之一，如果这很难的话。

But this would be potentially one of the reasons why we just don't get this immediate takeoff is if this is hard.

如果你回顾从 2012 年到现在，或者从你开始做研究到现在，自那时以来发生的所有创新中，包括纯工程的、纯概念的，哪部分看起来是 AI 完全自动化 AI 研发之前人类最后要做的事情？

If you look back from 2012 till now or maybe from when you started doing your research till now, what part of all the innovations that have happened since that time, including purely engineering ones, including purely conceptual ones, what seems like the thing that would be the last things humans would have to do before AI is totally automate AI R&D?

可能只是迭代地提出正确的问题。如果你能让 AI 做任何实验，但你需要决定做什么实验。而现在，我认为 AI 在这方面与编码实验相比还不太擅长。

Probably just iteratively asking the right questions. If you can get the AI to do any experiment, but you need to decide what experiments to do. And right now, I think AI are not very good at this compared to coding the experiments at all.

是的。是的，就像每当我们谈论研究时，他们提出一堆杂七杂八的东西，都是非常非常小的步骤。

Yeah. Yeah, like whenever we talk about research, they proposed a bunch of miscellaneous things which are very very tiny steps.

甚至从 DeepMind 的方法“我们将通过学会在超人水平上玩游戏来解决智能”——那将是方法——到像 Radford 这样的一个随机研究员说“我将尝试从非常广泛的数据中预测下一个词。”

Or even going from DeepMind's approach of 'we're going to solve intelligence by learning to play games at a superhuman level'—that's going to be the approach—to one random researcher like Radford being like 'I'm going to try and just predict the next token off a very wide swath of data.'

然后即使 Radford 发现了这一点，人们决定扩大规模也花了一段时间，因为我们必须提出缩放定律的想法，以及你可以非常可靠地预测这些事情。

And then even once Radford had discovered that, it took a while before people decided to scale it up because we had to come up with the idea of scaling laws and the fact that you could very reliably predict these things.

是的。是的。我会说人类最后的工作或持续时间最长的角色是定义目标和决定我们真正想要什么。所以在这方面，比如决定 AI 助手应该如何表现，或者什么是有帮助的，或者我们在做基于人类反馈的强化学习（RLHF）时的目标是什么，就是其中之一，然后后来定义宪法和模型规格是另一个。我认为即使 AI 能完成所有技术工作，我们仍然必须做很多这方面的工作，并决定我们真正想要什么。

Yeah. Yeah. I would say that the last job for humans or the role for humans that'll last the longest is defining the objective and deciding what we actually want. So in that vein, something like deciding how the AI assistant should behave or what it means to be helpful or what's the objective when we're doing RLHF is one such thing, and then later defining constitutions and model specs is another one. And I think even if the AIs can do all the technical work, we'll have to still do a lot of that and decide what we actually want.

是的，对齐是最后的工作。

Yeah, alignment is the final job.

是的，对齐算是答案，但也是——对齐本身可以分解为目标的 specification 或弄清楚正确的目标应该是什么，然后实际实现或优化你定义的目标。我认为第一个不会很快消失。如果我想到一个后训练团队以及为什么需要很多人在团队中，那是因为有很多不同的领域你需要弄清楚模型应该如何表现，而且没有办法——很难自动化整个过程，因为必须有人思考模型在这个领域应该如何表现。

Yeah, alignment is sort of the answer, but it's also—alignment itself can be kind of decomposed into specification of the objective or figuring out what the right objective should be, and then actually achieving or optimizing the objective you've defined. And I think the first one is not going to go away anytime soon. And if I think about a post-training team and why you need a lot of people to be on the team, it's just because there are a lot of different areas where you have to figure out how the model should behave, and there's no way—it would be very hard to automate the whole thing just because someone has to think about how should the model behave in this area.

Jane Street 在 2025 年初开始使用 Antithesis 来测试我们的软件，他们对产品印象深刻，于是决定投资这家公司。

Jane Street started using Antithesis to test our software in early 2025 and they were so impressed by the product that they decided to invest in the company.

我最近采访了 Ron Minsky，他共同领导 Jane Street 的技术团队，我问他 Antithesis 实际上是如何接入的。

I recently caught up with Ron Minsky who co-leads Jane Street's tech group to ask about how Antithesis actually plugs in.

我认为 Antithesis 最令人印象深刻的一点是，我们开始在一个构建高保障软件的团队中使用它，团队非常谨慎，但它仍然能够找出那些原本很难发现的 bug。这很重要，既因为它有助于使这些系统更可靠，也因为它帮助构建这些系统的团队更快地推进。

The thing that I think is most impressive about Antithesis is we started using it in a team that was building high assurance software and being really careful and nonetheless it was able to shake out bugs that were otherwise going to be really hard to find. And that's important both because it helps make those systems more reliable, but also because it helps the teams that build it to just move faster.

随着代码生产越来越自动化，这一点变得越来越重要。

This matters more and more as code production is increasingly automated.

我认为总的来说，随着我们越来越多地使用智能体，你遇到的关键问题是验证瓶颈。就是人们需要花时间查看代码并弄清楚那是否真的是你想接受的生产软件。而让测试更好的工具在这方面非常有帮助。你只需缓解验证瓶颈，就能完成更多工作并更快推进，因为你可以更有信心地认为智能体生成的代码实际上没有引入新问题。

I think in general as we've been using agents more and more, the key problem that you run into is the verification bottleneck. Just the time it takes from people to look at code and figure out is that actually something you want to accept in your production software. And tools that make testing better are just incredibly helpful there. You just ease the verification bottleneck and make it possible for you to get more stuff done and move faster because you can have more confidence that the code generated by the agent is actually not introducing new problems.

要了解 Antithesis 如何融入你的开发流程，请访问 antithesis.com/bash。

To see how Antithesis fits into your development process, go to antithesis.com/bash.

为什么模型提供商没有出现大规模整合？这里有很多因素都指向集中化。如果我们回顾过去几年的情况，有什么会阻止这种趋势吗？

What is the story for why there isn't huge consolidation in model providers? There's just so many things that point to centralization here. If we step back over the course of years is there something that is gonna prevent that?

是的，我认为蒸馏是抵抗集中化力量的主要因素，因为基本上任何可以通过强化学习学到的东西都可以很容易地被蒸馏，因为它只是少量比特，你可以从少量数据中学到。所以如果你能从模型中获得展示某种行为的轨迹，你就可以轻松地蒸馏它。所以我认为蒸馏是抵抗集中化的因素之一。还有一种可能性是会出现公司特定的模型，可以从部署中学习，并让公司不断改进自己的模型。

Yeah I think distillation is the main thing that fights against the centralizing force because basically anything that can be learned through RL can be distilled very easily because it's a small number of bits, it's something that you can learn from a small amount of data. So if you can get trajectories from the model that show a behavior you can easily distill it. So I think distillation is one of the things that fight centralization. There's also a possibility that there'll be company specific models that it'll be possible to learn from deployment and have a company continually improving its own model.

这样的系统可以由当前的模型提供商寡头或一些目前较小的公司提供，但我认为这会稍微改变游戏规则。

Such a system could be provided by the current oligopoly of model providers or some other currently smaller company, but I think that'll change the game a bit.

是的，我还想指出，持续学习并不能阻止蒸馏，对吧？就像即使你的模型每天都在改进，人们也可能每天都在蒸馏它。所以循环可以以相同的速度运行。

Yeah and I also want to point out that continual learning doesn't stop distillation, right? Like even if your model is improving every day, people could be distilling it every day. So the loops could just operate at the same pace.

对吧？这说得通。好的，那么复制模型行为，我猜你需要自己知道正确的提示分布是什么，才能获得相关的模型行为。

Right? That makes sense. Okay, so copying model behavior, I guess you need to know yourself what the right distribution to prompt is in order to get the relevant model behavior.

哦是的，对于用监督学习进行蒸馏来说，提示分布极其重要。所以即使你完全访问模型并拥有思维链等一切，蒸馏模型也是非常不简单的。是的，从中蒸馏所有有用的能力是不简单的。因为你需要用一些东西来提示模型，并且需要用现实的提示来提示它。你需要有一个非常广泛的现实提示分布。所以是的，最近出现的一件事是，一些中国公司可能正在使用这些路由服务，这些服务旨在让中国的人们使用原本在中国被封锁的美国前沿模型。但有所有这些路由或代理服务，允许中国的人们使用这些模型，主要用于编程，而这些路由服务正在收集和出售一些数据。所以我认为这是一个非常有用的蒸馏数据集，因为它为你提供了完美的提示分布。

Oh yeah, for just distilling with supervised learning the prompt distribution is extremely important. So it's very non-trivial to distill a model even if you have full access to it and have the chain of thought and everything. Yeah it's non-trivial to distill all of the useful capabilities from it. Because you need to prompt the model with something and you need to prompt it with realistic prompts. You need to have a really wide distribution of realistic prompts. So yeah, one thing that's been coming out recently is some of the Chinese companies are probably using these router services which are designed to allow people in China to use the US frontier models which would otherwise be blocked in China. But there are all these router or proxy services that allow people in China to use these models mostly for coding and these router services are collecting and selling some of the data. So I think this is a very useful data set for distillation because it gives you the perfect prompt distribution.

是的，我认为这是 AI 在这方面帮助很大的一件事。就像如果你真的看看中国模型的前沿流程，他们在论文中实际提出的，很多是人类，或者他们从某个地方获得种子提示，这是人类、这类数据等的某种组合，然后他们使用现有模型或其他前沿模型从这些种子提示中合成大量覆盖。所以你可以自动化大量的提示分布收集和环境创建。只是随着模型变得更好，人类需要提供的比特越来越少。

Yeah, I think this is one of those things where AI helps a lot here. Like if you actually look at the frontier pipelines of say the Chinese models that they actually put in their papers, it's a lot of humans or they get seed prompts from somewhere which is some combination of humans, this kind of data, and then they synthesize a vast coverage from those seed prompts using their existing models or other frontier models. And so you can automate an awful lot of this prompt distribution gathering and environment creation. It's just humans need to provide increasingly few amounts of bits. As the models get better.

对吧？但似乎你仍然受限于拥有一个拥有用户或用户正在使用的服务。

Right? But it seems you're still bottlenecked by having a service which has users or users are going through.

不一定。我的意思是，那显然非常有帮助，但理论上你可以只考虑用户想要什么，或者像……

Not necessarily. I mean, that's obviously very helpful, but theoretically you can just think about what users want or like...

就像整个重点是，我们不……用户说给我做一个这样的应用。哦那没有……我实际上想让你做这个新功能，但实际上让我们退一步做另一件事，捕捉整个轨迹是……或者如果你无论如何都能做到这一点，那么你就有了递归自我改进……

Like the whole point is that we don't... the user says make me an application like this. Oh that didn't... I actually want you to make this new feature but actually let's step back and do this other thing and capturing that whole trace is the... or to the extent you could have done that anyways then you just have like RSI any...

是的，我的意思是最终如果你有这个完全自动化的循环，那基本上就是递归自我改进，对吧？就像 AI 决定数据，决定训练，那就是循环。但是的，这取决于你需要多少人类信息。就像在某个时候，如果你只是说“我想要看起来像这样的轨迹”，你向模型提示，模型就能给出一个相当好的近似。

Yeah, I mean ultimately if you have this fully automated loop that is basically RSI right? Like the AI is deciding the data, it's deciding the training, that is the loop. But yeah, it depends how much human information you need. Like at some point if you're just like I want traces that look like this, you prompt that to the model, the model will be able to come up with a pretty good approximation.

但如果你想做的是，比如让我成为一个真正优秀的政治家，并从头预测参议院大厅里的讨论会如何进行之类的。我只是觉得会有很多事情。

But what if you want to do like make me a really good politician and anticipate de novo like how would discussion in the Senate halls go or something. I just feel like there's going to be a lot of things.

讽刺的是，我认为这对蒸馏者来说实际上比前沿实验室更容易，对吧？因为蒸馏者只是说“我想要一个优秀的政治家”，他们去找前沿模型。前沿模型已经知道如何成为一个优秀的政治家。所以它只是生成那些轨迹。而如果你真的想构建第一个做到这一点的模型，你必须以某种方式获取政治家每天做什么的数据并构建它。所以说“我想要这样的东西”，然后让 AI 产生十亿个变体来实际创造这个东西，实际上要容易得多。

Ironically, this is actually I think easier for the distillers than the frontier labs, right? Because the distillers just like I want a good politician, they go to the frontier model. The frontier model already knows how to be a good politician. So it just generates those traces. Whereas if you actually want to build the first model that does this, you have to actually somehow get data on what politicians do every day and build that. So it's actually much easier to say I want something like this and then get the AI to produce a billion variations to actually create the thing to begin with.

我认为你实际上可以基于中国实验室拥有这些路由数据的观察做出一个非常具体的预测。

I think you can actually make a really concrete prediction based off this observation that the Chinese labs have this router data.

就像我觉得最初提出这一点的是我，我说：难道不奇怪吗？Set 5 和 Opus 5 几乎客观上是比 GLM 5.3、Kimi K3 更差的模型，尽管它们不仅能做蒸馏，还能从 Mythos 做逻辑蒸馏。所以这里的反驳是：好吧，提示分布真的非常非常重要。你需要看到用户在做什么，才能蒸馏出这些行为等等。我认为由此得出的预测是，前沿实验室现在在强化学习环境上不一定有多大优势，甚至可能没有优势，因为是的，用户分布对通用行为等很重要，但衡量能力的最佳标准是你在前沿构建的非常非常难的强化学习环境。所以如果你像 Anthropic 一样能访问那些强化学习环境，又能做逻辑蒸馏，却仍然做出了更差的模型，那么也许现实世界的部署比环境更重要。这真的很有趣。所以他们首先必须在 Fable 或前沿模型中激励那些能力。

Like I think the thing that just stated this originally was I was saying isn't it weird how set 5 and opus 5 are like almost objectively worse models than like GLM 5.3 Kim K3 even though they've had access to like not only distillation but logic distillation from like mythos and so the counter here was that like okay the prompt distribution really really matters. Like you need to see what users are doing so you can distill like kind of these behaviors and things in. I think the prediction from this is that the frontier labs don't necessarily have much of an advantage if at all in RL environments now because yes like user distribution matters for like general like behavior and so on but like the best measure of a capability is the very very hard RL environments you've made at the frontier and so if you have access to those RL environments as Anthropic and you have access to logic distillation and you've still made a worse model then maybe real world deployment matters more than the environment. That's really interesting. So but they had to incentivize those capabilities in the first place in Fable or the Frontier model.

所以奇怪的是他们不能再激励一次。

And so it's weird that they can't incentivize them again.

或者用更小的模型之类的。

Or with a smaller model or something.

也许我们只是处在这个奇怪的恐怖谷中，实际上试图过多地复制那个前沿模型，学生-教师差距或其他什么，就是太大了。我认为人们用 Opus 提出了这一点：Opus 4.6 和 Opus 5 的区别在于，Opus 5 真的感觉它有一个 AI 作为裁判检查它所做的每一件可能的事情，这就是为什么它使用这么多 token。它试图思考所有这些东西，但它不一定有 Fable 的大模型嗅觉，知道什么时候停止这样做，或者什么时候是好的路径之类的。

Maybe we're just in this weird uncanny valley where actually trying to copy that frontier model too much, the student-teacher gap or whatever it is, is just too large. And I think people made this point with Opus: the difference between Opus 4.6 and Opus 5 is that Opus 5 really feels like it's got this AI as a judge checking every possible thing it's done, and that's why it uses so many tokens. It tries to think about all these things but it doesn't necessarily have the big model smell of Fable to know when to stop doing that or when's the good path to go down or whatever.

目标超出了能力。

The reach exceeds the grasp.

是的。

Yeah.

是的。我想提出一个稍微不同的假设。我认为你可以创建的环境有几个不同的轴，一个是难度，另一个是现实性。相对容易创建大量困难的环境，这些环境涉及做更复杂的任务或需要更多聪明才智的事情。你可以说这就像是刷榜分布，因为许多最突出的基准测试只是涉及做一些非常难的、像谜题一样的任务，这些任务容易验证。然后是现实性轴，你希望模型在现实的编码智能体设置中表现良好，那里有与人类的多次来回交流，有多个目标。我认为第一次塑造模型行为的实验室需要在两个方向上推进，而要获得良好的模型行为，你需要真正在现实性轴上推进，并有评分标准或某种人类反馈来为你在那里使用的奖励函数提供信息。但我认为当你天真地尝试蒸馏时，你最终只是在刷榜分布上匹配教师模型。如果你没有足够的环境来真正锻炼这些更棘手的现实设置中的能力，那么你就不会把这些能力带入你的学生模型。我认为可能发生的一件事是，大模型从棘手的狭窄任务到这些更现实的任务的泛化能力更好。所以如果你有一个非常好的现实提示分布用于蒸馏，你可以很好地匹配大模型。但如果你只有这种容易验证的任务分布，那么你可以在所有基准测试上匹配大模型，但在这个更广泛的分布上表现更差。所以这甚至可以解释一些关于较小的 Anthropic 模型如 Sonnet 5 的事情，尽管很难准确预测他们在后训练那些模型时做了什么。也可能是他们总是在改变他们的后训练技术栈，他们只是在一些模型中弄错了几件事。所以他们把某个东西调得太高，创造了一些人们真的不喜欢的怪癖。所以很容易在后训练中搞砸，而这些问题不会在基准测试中显示出来。我的意思是，另一个非常基本的点是，前沿 AI 实验室从大数据公司购买所有数据，中国人也可以从数据公司购买同样的数据，他们完全——有很多人对此感到恼火——但如果他们有完全相同的数据，他们可以购买，他们也可以蒸馏，这意味着很容易跟上。

Yeah. I would offer a slightly different hypothesis. So I would say there are a couple of different axes for the environments you can create, and one of them is difficulty and the other is realism. It's comparatively easy to create a lot of difficult environments that involve doing a much more complicated task or doing something that requires a lot more cleverness. And you could say this is like the benchmaxing distribution because a lot of the most prominent benchmarks just involve doing some very hard puzzle-like task that's easy to verify. And then there's the realism axis where you want the model to be good in the realistic coding agent setting where there's multiple back and forth with the human and there's multiple objectives. And I'd say the labs who are crafting the model behavior for the first time need to push in both directions and to get good model behavior you need to really push on the realism axis and have rubrics or some kind of human feedback that's informing the reward function you use there. But I think when you try to do distillation naively, you end up just matching the teacher on the benchmaxing distribution. And if you don't have enough of the environments that really exercise the capabilities in these trickier realistic settings, then you're not going to get those into your student model. And I think maybe one thing that's happening is the big models generalize better from the tricky narrow tasks to these more realistic tasks. So if you have a really good realistic prompt distribution for distillation, you can match the big model really well. But if you only have this distribution of easily verifiable tasks, then you can match the big model on all the benchmarks but you do worse on this broader distribution. So that might even explain something about the smaller Anthropic models like Sonnet 5, though it's hard to predict exactly what they're doing to post-train those models. It could also be that they're always changing their post-training stack and they just got a few things wrong in some of these models. So they turned up something too high and created some quirks that people really don't like. So it's really easy to screw up post-training in some way that doesn't show up in benchmarks. I mean just one other very basic point is just like the frontier AI labs buy all their data from big data companies and the Chinese can also just buy the same data from data companies and they are exactly—there's a lot of people being annoyed about this—but if they have exactly the same data and they can buy that, they can also distill, it means it's quite easy to keep up really.

是的。是的。是的。好的。我还有一个问题是，第一批能够自动化 AI 研发的模型实际上将如何训练，因为有一个玩具版本，就是 Ryan 谈到的那个东西，你只是让 GPT-8 尝试构建 GPT-3 大小的模型，这些模型非常擅长像内循环类型的挑战，比如玩需要持续学习的电子游戏，或者用最少的算力达到某个损失值等等。但 John，我认为你有一个有趣的观点，也许那不是实践中实际会发生的方式。所以我很好奇，是的，到你有真正能够自动化研发的模型时，它们可能如何训练？

Yeah. Yeah. Yeah. Okay. The other question I had is how the first models that are capable of automating AI R&D will actually be trained because there's a toy version which is this thing that Ryan was talking about which is you just have GPT8 try to build GPT3 size models that are really good at like inner loop type challenges of beating video games that require continual learning or just getting to a certain loss with the least amount of compute etc. But John, I think you had an interesting point that maybe that's not the way it actually will happen in practice. So I'd be curious about yeah, by the point which you have that are actually capable of automating R&D, how are they probably trained?

是的，我认为我们可能会结合从人类反馈中学习来吸收研究人员的品味，以及创建大量涉及做多步研究项目的练习环境。所以我认为是的，人们实践中会结合这两件事，并且每次迭代修补上一次迭代中最明显的问题。所以研究人员会大量使用 AI，我们会注意到它们有一些一致的弱点，然后这些弱点要么通过收集人类反馈来修补，要么通过创建环境来修补。

Yeah, I think we'll probably do some combination of learning from human feedback to absorb the researchers' taste and just creating a lot of practice environments which involve doing multi-step research projects. So I think yeah people will in practice do some combination of those two things and just each iteration patch whatever seems to be most broken in the last iteration. So researchers will be using the AIs a lot and we'll notice that they have some consistent weaknesses and then those things will either be patched by collecting human feedback or creating environments.

是的。有道理。

Yeah. Makes sense.

也许一个有用的思考方式是，我们回滚多少传承，然后从那里让自我对弈。就像我认为在极限情况下，你想象的是只给他们一个 GPU，也许还有神经网络之类的，然后说：好吧，弄清楚如何训练一个模型来完成这些特定任务。

Maybe a useful way to think about this is like how much of the lineage we roll back and then let self-play from there. Like I think in the limit you're picturing just giving them a GPU and maybe neural nets or something and saying okay figure out how to train a model to do these particular tasks.

目前的做法是，我们走到这条谱系的最前沿，说：好，这些是 Anthropic 过去几个月在训练栈里发现的 bug。然后把这些变成环境：你需要在前沿上训练、变得更强。所以你显然会锁定谱系里之前的所有历史。但你可以想象一个世界：回滚到比如 GRPO 之前，然后有一些环境试图让它自己发现奖励模型的最佳形式，然后也许再往前回滚。但我认为我们仍然会受算力瓶颈限制，所以人们会一直待在前沿，基本上就是把自上一个模型版本以来发现的 bug 和改进做 diff，把它们变成训练环境。

The way it currently works is we go up to the very edge of the lineage and say, okay, here are the bugs that, you know, Anthropic has found in their training stack in the last few months. Then we turn those into environments: you need to train and get better on the frontier. So you obviously lock in all the previous history of the lineage. But you could imagine a world in which you roll back to, you know, before GRPO or something, and then you have environments that try to get it to discover the best form for, like, reward models, and then maybe roll further and further back. But I think we will still be so compute-bottlenecked that people will just keep staying at the frontier and essentially diffing the bugs and whatever improvements they found since the last model version, turning those into training environments.

这对在模型代际之间拥有不过时的新数据也很有好处。这基本上就是 AI 实验室内部的持续学习：把过去三个月的 AI 研究进展，通过环境和 RLHF 之类的东西，蒸馏回模型本身。

Which is also really good for having non-stale, like, new data between model generations. This is basically continual learning within the AI lab: distilling the last three months of AI research progress through environments and RLHF-type stuff back into the model itself.

而且这确实是蒸馏，对吧？这也许就是为什么我们有些人觉得这像西西弗斯式的任务：你总是在追赶过去三个月的进展，而那进展当然有人的贡献，但人也仍然在回路里。感觉就像你一直在一点点逼近人类研究者能做到的事情。

And it is distilling, right? And that's maybe why some of us feel like it's a Sisyphean task: you're always just trying to get the last three months of progress, and that progress is being contributed to by humans, of course, but it also still has humans in the loop. It feels like you're just constantly inching closer and closer to what the human researchers are capable of doing.

是的。

Yeah.

不过我要说的一点是，显然，如果你只是在轨迹上蒸馏，你永远无法超越它。但环境可以远远超出人类能做到的水平。设计一个没人能解决的环境非常容易，但 AI 显然仍然可以尝试去解决它。

I mean, the one thing I will say though is, obviously, if you're just distilling on trajectories, you can never go above it. But environments can go quite far above what humans can do. It's very easy to design an environment that no human can solve, but the AI can obviously still try and solve it.

所以那会是一条超越人类 AI 研究水平的路径……

And so that would be the path to go ahead of just what the human AI research...

你有例子吗？

Do you have an example of that?

是指递归自我改进（RSI），还是比人类速通玩家还快？

In terms of RSI, or like doing it even faster than a human speedrunner?

是的。我觉得尤其在 AI 研究里，定义目标非常容易，比如你可以说损失需要到 1.3 左右，现在没人能达到，但那是一个极其可测量、可验证的任务，如果 AI 达到了，那就太好了，对吧？我不知道。训练一个 1 亿参数的模型打败《我的世界》，那也许太简单了，但换成复杂得多的游戏之类的。

Yeah. I mean, I feel like in AI research especially, it's very easy to define goals where, you know, you could say the loss needs to be 1.3 or something, and no human can get there now, but that's a very extremely measurable, verifiable task, and if the AI gets there, then great, right? I don't know. Building a 100 million parameter model that beats Minecraft, that's maybe too easy, but like a much more complicated game or something.

1 亿参数的模型打败《我的世界》，我们居然说这太简单了，是不是很疯狂？想象一下你五年前说这话。

Isn't it crazy that 100 million parameter models to beat Minecraft, we're calling that too easy? Like imagine you said that five years ago.

不过我会说，很多研究并不完全是这样，不是在一个定义明确的目标上爬山。它更像是：我们对模型应该在某个方面更好有一个直觉，然后我们也有一些算法想法，似乎朝这个方向走了一点。所以让我们设计一个任务，用来展示这种方法有生命迹象，看看我们能不能得到那些生命迹象，如果能，我们就可以逐步做出更真实的任务版本，对吧？它更多是由直觉引导的，然后内循环是去引出或为那个直觉做测试，而不是测试本身带来洞见。

I would say a lot of research is not exactly like that though, where it's like hill climbing on a well-defined goal. It's sort of more like: here's an intuition we have about some way a model should be better, and then we also have some idea for an algorithm that seems to go a little bit in this direction. So let's come up with a task that is sort of designed to show signs of life on this approach, and see if we get those signs of life, and then if we do, we can make successively more realistic versions of the task, right? It's a lot more guided by intuition, and then the inner loop is to elicit or make a test for that intuition, rather than the test itself leading to the insight.

对吧？你并不是直接优化你最终关心的目标，或者实际的、生产的目标。你是在稍微放松你的目标。你说：好，我们在真实性的轴上稍微放松一点，找到一些真正有效的方法，然后等方法成熟一点再回到真实性。

Right? Like you're not directly optimizing for the eventual objective you care about, or the practical, like, production objective. It's sort of, you're relaxing your objective a little bit. You're saying, yeah, let's relax on the realism axis a little bit and find some methods that actually work, and then try to get back to realism later after the method matures a little bit.

是的。

Yeah.

然后还有更多研究是更偏向解释事物、发展理论的，或者说……通常我们在机器学习里没有那种很有预测力的数学理论，但我们有很多关于正在发生什么的更非正式的理论。

And then there's also more research that's more oriented towards explaining things and developing a theory, or a sort of... often we don't have mathematical theories in machine learning that are that predictive, but we have a lot of more informal theories for what's going on.

是的。我猜模型会在所有这些任务的某种组合上训练，有些非常容易验证，有些像 LLM 裁判，或者直接问人类：这看起来合理吗？然后希望是这些都能泛化到那些难得多、更模糊的任务上。某种程度上大概会。但它是否能泛化到足以让循环在完全没有人类在回路的情况下自我闭合，还不清楚。

Yeah. I mean, presumably the models will be trained on some combination of all of these tasks, and some will be very easily verifiable, some will be like an LLM judge, or just ask the human, does this look reasonable? And then the hope would be that these would all generalize to these much harder, more vague, fuzzy kinds of tasks. And it probably will to some extent. Whether it generalizes enough that the loop can become self-sealing without humans being in the loop at all is unclear.

是的。是的。是的。

Yeah. Yeah. Yeah.

等等，也许退一步说。在我看来，未来 AI 研究的计划是这样的，你告诉我你觉得它会不会奏效，或者你同不同意这个描述。这个赌注是：我们会在数百万个多样化环境、数百种不同领域上扩大 RLVR 训练。另一端涌现出来的会是一个智能体，它学会了这些基本或不太基本的技能：坚持不懈、能在上下文中分诊信息、最终能端到端地优化与其他智能体的协作等等。这样的智能体在上下文内会非常样本高效。你知道，你研究过如何真正扩展上下文学习，让它任意长，但你一直在扩展它。所以另一端出来的东西，基本上会像一个随插即用的远程工作者，能工作一周或一个月。首先，你同意这是实验室正在下的赌注吗？其次，这够吗？基本上就是在数据中心里的这些模拟环境中学会如何学习，然后部署到现实世界，但并不真正从现实世界的部署中学习，只从数据中心的模拟环境里学习这些元技能？

Wait, maybe taking a step back. Here's what it seems to me that the plan for AI research going forward is, and you tell me if you think it's going to work or if you agree with this characterization. So the bet is that we will scale up RLVR training across millions of diverse environments across hundreds of different kinds of domains. And what will emerge at the other end is an agent which has learned these basic or less-than-basic skills around being persistent, being able to triage information in context, eventually having end-to-end optimization of working with other agents and things like that. And such an agent will be very sample efficient within the context. You know, you've done research on how you actually scale up in-context learning to make it arbitrarily long, but you keep scaling it up. And so what comes out the other end will be something that basically functions like a drop-in remote worker over the course of a week or a month. First of all, do you agree that that is a bet the labs are making? And second, is that enough, basically learning how to learn within these simulacra within a data center and then getting deployed into the real world, but not actually learning from real-world deployment, only learning these meta skills from the simulated environments in the data center?

是的，我觉得现在很难区分实验室有多少精力是直接投向 RSI，多少是投向做出可以持续部署、收集收入来资助下一次大训练运行的通用智能模型。我认为对于后者，是的，那大概就是他们正在下的赌注，而且过去几年这些环境的发展模式非常清楚。

Yeah, I think it's now hard to separate out how much of the lab's effort is going towards direct RSI versus making generally intelligent models they can continue to deploy to collect revenue to fund the next big training run. I think for the latter, yes, that's probably just the bet they're making, and it's very clear, like the pattern of where these environments are going over the last few years.

我是说，Anthropic 的环境谱系就是一个非常清晰的例子。首先我们只专注编程，我们要把这件事做到非常非常好。然后我们从编程中获得的任务时间跨度——这大概是利用互联网上可用数据来构建环境时最容易摘到的果子，还有他们自己内部的东西可以转化成环境——接着我们再去泛化。下一步我们要进军金融，训练数据里literally 有海量的 Excel 数据之类的东西。再然后就是 PowerPoint。这就是工作经济的长尾。这似乎效果非常好，很多其他实验室——甚至开源实验室——现在都意识到那才是正确的做法。

I mean, Anthropic's lineage of environments is a very clear example of this. First we just focus on coding and we're going to get really, really good at that. Then the task horizon that we've got from coding — which is probably the lowest-hanging fruit in terms of data available on the internet to create environments, their own internal stuff that they can turn into environments — then we're going to generalize. We're going to go after finance next, and literally just so much Excel data and all that sort of stuff in the training. And then, you know, it's PowerPoints. It's this long tail of the working economy. And that seemed to work really well, and a lot of the other labs — even the open-source labs — have now realized that that was the correct approach.

但那意味着什么呢？我请 Dario 上播客时，我问他的问题是：如果你真的期望模型在学习能力上像人一样、能在工作中学习，那你为什么还要把这些比如用 PowerPoint 之类的技能预先烘焙进去呢？你难道不会期望模型在部署之后自己就能学会这些吗？

But what is the implication from that? When I had Dario on the podcast, the thing I asked him was: if you truly expect models which will be humanlike in their ability to learn on the job, why would you try to bake in all these skills of like working with PowerPoint or something? Wouldn't you just expect the model to be able to pick that up while it's deployed?

对，有几种不同的解释。一种是我们预期模型很快就能达到那个水平，但现在还没到。那为什么不把这些技能摊销进模型训练里呢？另一种是我们并不专注于让它把广泛部署的工作做得很好。我们只想让它把 RSI 做得很好。而这只是我们获取收入的一种方式，好把钱再投回一个真正擅长做 RSI 开发的模型里。然后一旦奇点发生，最终出来的那个东西会在当前这一代模型看起来是瓶颈的所有事情上都非常擅长。

Yeah, there's multiple different explanations. One is just that we expect models to get there soon, but they're not there yet. So why not amortize these skills into the model training? Another is that we're not concentrated on making it really good at widely deployed work. We just want it really good at RSI. And this is just a way for us to get revenue so that we can pour it back into a model that is actually really good at doing RSI development. And then once the singularity happens, the thing that comes out the other end will be really good at all the things which seem like bottlenecks to the current generation of models.

对，John，不知道你有没有看法：如果路径是这种泛化，那你怎么解释这些模型里为什么有这么多任务特定的知识？

Yeah, John, I don't know if you have takes on how — once you construe why there is so much task-specific knowledge in these models — if the path is this kind of generalization.

对，我是说，如果模型在上下文学习方面足够好，那理论上你就不需要用金融数据训练它们。它们就能自己搞定——当场读完所有书，然后弄清楚如何在相应的司法管辖区里做每一件事。

Yeah, I mean, if the models were good enough at learning in context, then in theory you wouldn't need to train them on finance. They would just be able to figure out — read all the books on the fly and figure out how to do everything in the appropriate jurisdiction.

嗯，对，你也可以说，你需要做大量这种领域特定的训练，只是为了让他们更高效。所以即使它们聪明到能当场想明白，你可能还是想做一堆强化学习，把这些直觉烘焙进权重里，这样模型在推理时会更高效。

Um, yeah, and you could argue that you need to do a lot of this domain-specific training just to make them more efficient. So even if they were smart enough to figure this out on the fly, you still might want to do a bunch of RL and bake all these intuitions into the weights, so the model would be more efficient at runtime.

对。

Yeah.

对。我觉得在实践中，模型提供方确实是一个领域一个领域地推进，努力在最高价值的领域强化模型。我觉得这也是模型为什么变得好这么多的答案之一。就是因为模型提供方覆盖了很多高价值领域和最常用的技能类型。

Yeah. I'd say in practice it does seem like model providers are going domain by domain and trying to strengthen the models in the highest-value domain. And I'd say that's one of the answers to why the models have gotten so much better. It's just because the model providers have covered a lot of the high-value domains and the most common types of skills.

我是说，我觉得另一点是，同时做这两件事其实没那么贵，对吧？因为模型非常庞大。就参数量而言，它们很容易负担得起学会一切。而且很可能存在一些迁移——即使金融对 RSI 不是特别重要，光是那种通用的元学习：如何判断什么重要、如何有品味、如何做长跨度的工作，这些都可能具有泛化性。而且世界上并没有那么多 RSI 数据，它挺难生成的，需要大量投入。所以如果你能把这些其他数据摊销进来、从中获得一些迁移——你本来就有海量算力、海量参数空间——那为什么不也这么做呢？再加上卖模型这个显而易见的直接商业激励。

I mean, I think another thing is just that it's not that expensive to do both at the same time, right? Because the models are massive. They can easily afford, in terms of their parameters, to learn everything. And there is likely some transfer — even if finance is not specifically important for RSI, just the general meta-learning of how to figure out what's important, how to have taste, how to do long-horizon work, is potentially generalizable. And there's not that much RSI data in the world as well. It's kind of hard to generate and requires a lot of effort. So if you can amortize in this other data, get some transfer from it — you already have masses of compute, masses of parameter space — why not do that as well as the obvious direct commercial incentive of selling a model?

对，有道理。

Yeah, it makes sense.

哦对，我补充一点。我是说，有一个问题是，当前这种从仿真到现实的范式是否会永远占据主导。基本上就是，你先看现实世界的任务是什么样的，然后尝试创建一堆可以在数据中心里仿真的环境，再在上面做强化学习。我觉得这显然非常成功，但它有很多弱点，因为很多东西就是很难仿真，尤其是涉及与一群人实时互动的时候。对。所以就有个问题：从仿真到现实会不会永远是主导框架。

Oh yeah, I'll add that. I mean, there's one question about whether this current paradigm of doing sim-to-real will be the dominant one forever. So basically you look at what the real-world tasks are like, and then you try to create a bunch of environments that can be simulated in the data center, and you can do RL on them. And I think obviously this has been very successful, but it has a lot of weaknesses, because a lot of things are just kind of hard to simulate, especially if they involve interacting with a bunch of humans in real time. Yeah. So there's some question about whether sim-to-real will be the dominant framework forever.

我觉得在样本效率还比较低的时候，从仿真到现实必然是主导框架，因为现在你需要成千上万次与人类的交互，而没有人会坐在那里基本上充当强化学习训练回路中的一环。所以我们现在只能仿真来获得所需的样本。但显然，如果样本效率大幅提升，你就会预期从部署中学习会成为其中大得多的部分。不过你还可以做其他事情——你可以做离策略学习，这样你可以拿到所有轨迹，即使不重新仿真一切，也有可能从中学到东西。

I think sim-to-real has to be the dominant framework while sample efficiency is kind of low, because right now you need thousands and thousands of interactions with the humans, and no human is going to sit there and basically be in the loop of RL training. And so we kind of have to simulate that now to get the samples you need. But obviously if sample efficiency improves a lot, you'd expect learning from deployment to become a much bigger part of it. Though there are also other things you could do — you can learn off-policy, so you can take all the traces and even without re-simulating everything, you can potentially learn something from them.

Jane Street 刚刚推出了一项新竞赛，这是他们迄今最有野心的一次：设计一款协议仿真器 ASIC。基本上，如果你有一块想测试的芯片，你可以把它连接到这块 ASIC 上，然后这块 ASIC 会模拟真实的流量。这样你就能看到芯片如何响应，而无需把它插进一个真实系统里。Jane Street 想要的是灵活、通用的设计，而不是单一协议的仿真器。我和他们聊的时候，他们建议我先从实现三种据说非常常见的协议开始：UART、SPI 和 I2C。Jere 还提到，他们希望更有野心的设计也能处理低速 USB 和以太网，以及任何其他能考验你芯片特定架构的协议。重要的是，你的设计应当是可重新编程的，而不是把一堆特定协议硬砸到一块芯片上。如果你的 ASIC 流片之后出现了一种新协议，你的芯片仍然需要能够处理它。具体怎么实现由你决定，但有一个硬性约束：你的设计必须针对开源的 130 纳米工艺节点。这是因为 Jane Street 会为最新颖的提交付费流片，并把实物寄给获胜者。竞赛开放至 2027 年 1 月 18 日，非常鼓励组队参加。

Jane Street just launched a new competition and it's their most ambitious one yet: design a protocol emulator ASIC. Basically, if you have a chip that you want to test, you can connect it to this ASIC and then this ASIC will simulate realistic traffic. That way you can see how the chip responds without having to plug it into a live system. Jane Street is looking for flexible, general-purpose designs, not single-protocol emulators. When I was chatting with them, they suggested that I start off by trying to implement what are apparently three very common protocols: UART, SPI, and I2C. Jere also mentioned that they hoped more ambitious designs will also tackle low-speed USB and Ethernet and any other protocols that flex your chip-specific architecture. Importantly, your design should be reprogrammable rather than smashing a bunch of specific protocols onto a chip. If a new protocol comes out after your ASIC is taped out, your chip still needs to be able to handle it. How exactly it does it is up to you, but there is one hard constraint: your design must target an open-source 130 nm process node. That's because Jane Street will pay to tape out the most novel submissions and send the physical copies to the winners. The competition is open till January 18th, 2027, and working in teams is highly encouraged.

我想多问一些关于这个的问题，因为有点奇怪的是，你有 50% 的算力花在推理上，而这并没有直接帮助模型变得更好。就像你最终期望数字心智拥有的一个关键优势是，不像人类只能拥有 50 年的现实世界经验，模型通过它的所有实例将能体验，我不知道，数百万年的部署，跨越经济中各种经济相关的工作，而现在这些数据并没有在 meaningful 的意义上帮助模型变得更好。就像很明显，最终模型应该能够从这些数据中学习，一旦它们这样做了，你就会得到一种几乎像是广泛部署的智能爆炸，因为模型正在吸收这么多信息，跨越所有这些部署的实例，但你预计这种蜂巢思维之类的疯狂什么时候开始发生。

I want to ask more about this because it's sort of weird that you have 50% of compute that's spent on inference that is not directly helping the model become better. Like one of the key advantages you'd expect eventually digital minds to have is unlike a human who gets to have 50 years of real world experience, a model will get to through all its instances will get to experience I don't know millions of years of deployment across all kinds of economically relevant work in the economy and right now that data is just not in a meaningful sense helping the model get better. Like it just seems so obvious that eventually models should be able to learn from this data and once they do you would have something that almost feels like a widely deployed intelligence explosion because the model is assimilating so much information across all these deployed instances but when do you expect this kind of hive mind kind of crazy to start happening.

我认为从广义上讲，在一个非常基本的层面上，这已经在发生了，对吧，就像在下一代模型中。所以就像现在你显然可以把你的部署数据放入未来模型的预训练或中期训练中，特别是如果你做一些过滤或某种判断或注释，或者最近的那种合成。

I think broadly like at a very basic level this is already happening right like just in the next generation of models. So like right now you can obviously take your deployment data and put this in the pre-train or the mid train of like future models especially if you do like some kind of filtering or some kind of like judgment or annotation or like recent you know synthesization of that.

你认为这在多大程度上解释了逐代改进？

How much do you think that explains the generation over generation improvement?

我认为这解释了相当多。我的意思是，特别是，我的意思是，这是你知道的，我不知道实验室是否这样做，因为理论上他们声称不在人们的数据上训练，但中国 100% 这样做，他们肯定获得了这个优势，显然部署这基本上就是蒸馏，他们拿出模型，他们得到一些部署数据，他们通过 ping 模型获得其中的一部分，然后他们在其上训练下一代模型，他们突然也可以在自己的模型上这样做，完全没有理由不这样做。

I think it explains like quite a bit. I mean especially like I mean this is you know I don't know whether the labs do this because you know theoretically they claim not to train on people's data but like the Chinese 100% do and like they definitely get this advantage both like obviously deploying this is basically what distillation is like they take out the models they get some of their like deployment data they get some fraction of that by like pinging the model and then they train their next generation models on it and they can suddenly do it on their own models as well like there's no reason not to whatsoever.

我完全同意这一点。我认为如果你把视野拉得足够远，这肯定在发生。就像你在想象这个，我们都在想象这个。这就像持续学习，就像圣杯是就像

I completely agree with this. I think if you zoom out far enough, this is like definitely happening. Like you're picturing this like and we're all picturing this. This is like what continual learning like the holy grail is is like

这个非常非常有机的，就像实时循环，就像单个模型获得经验并现场实时更新并从中学习。而当你深入到那种粒度时，很多事情会崩溃。但就像是的，大实验室正在这样做，封闭模型正在这样做。也有早期迹象表明人们使用开源模型以更快的节奏这样做。所以一个很好的例子可能是像 composer，嗯，就像你有某种模型，你能够或者像你知道 Harvey 正在用法律代理做同样的事情，就像

this very very organic like live loop of like an individual model like getting an experience and like live updating on the spot and learning from that. And like a lot of things break when you like zoom into that level of granularity. But like yeah the big labs are doing this like the closed the closed models are doing this. There's also early signs of life of like people using open source models doing this at a much faster cadence. So like a good example is probably like composer um like you have some sort of model and you are able to or like you know Harvey's doing the same thing with like legal legal agents like

它从你为那个特定任务拥有的数据以及你知道用户抱怨的事情以及你以某种方式从特定部署中提取的所有反馈中获得非常特定的环境，许多这些公司比大实验室有优势，他们可以非常好地使用这些数据，然后他们会创建环境，他们喜欢你知道做一个大的 Kimmy K3 后训练，嗯他们会去部署它，他们可能会做一些在线学习，就像 composer 在线做的，基本上像强化学习了很长时间，嗯所以是的，仍然有人在循环中，仍然有人说好的这些是我们关心的信号，这是我们如何从我们拥有的数据创建环境，这仍然比你想象的节奏要长，但就像它真的在发生，最终那个循环会变得越来越快

it is getting very specific environments from the data that you have for that particular task and things that like you know users are complaining about and like all the feedback that you're somehow extracting from like your specific deployments and a lot of these companies have the advantage over the big labs and that they can use this data really really well and then they will create environments they like you know do a big post train of Kimmy K3 um they will go deploy it they might do some online learning as well like composer did online basically like reinforce for a long time um so yeah like there's still a human in the loop there's still a human saying okay these are the signals we care about here's how we're going to create environments from the data that we have and it's like still a longer cadence than maybe the one that you're thinking of but like it really is happening and like eventually that loop will become like faster and faster

我的意思是 composer 的事情很有趣，因为这是模型在 cursor 中人们按 tab 或者他们不按 tab 在模型建议的下一个补全上，基于此每天 composer 在预测下一个方面变得更好。

I mean the composer thing is interesting because this is where the model like in cursor people like press tab or they don't press tab tab on the next completion that the model suggests and based on that every single day composer gets better at like predicting the next.

所以那是旧的 tab 模型，他们实际上对实际的，不仅仅是 tab 模型，而是实际的生成模型做了同样的事情。哦，这很有趣。

So that was that was the old tab model like they actually did the same thing for the actual not just like the tab model but the actual like generative model. Oh, it's interesting.

他们，这很难，因为当你做在线强化学习时，你没有组，对吧？你只有一个用户说一件事，然后你得到一个展开。所以就像你有一个很大的方差减少问题

And they it was it's hard because when you do online reinforcement learning, you don't have groups, right? You just have one user saying one thing and then you get one roll out. And so like you have a big variance reduction problem

而 cursor 对此的模糊回答就像是，哦你知道我们有非常好的启发式方法能够估计像

and like curs's kind of fuzzy answer to this was like oh you know we have very good heristics which are able to estimate like

就像这个响应比平均好多少或这个响应比平均差多少。然后他们会做这个大的，你知道的，像强化更新，然后他们对于是否变差的解决方案是，如果它在 cursor bench 上有所改进，他们会每五小时部署新模型，如果没有，他们会扔掉那个版本。

like what the what how much better than average like this response was or how much worse than average this response was. And then they would do like this big, you know, like reinforce update and then their solution to like whether it got worse or not was like if it improved on cursor bench, they would deploy the new model like every five hours and if it didn't, they would like throw that version out.

有趣。

Interesting.

是的。我认为你最大的问题实际上只是不知道奖励函数应该从自然数据中是什么。如果你使用某种表面信号，比如他们是否接受了代码或编辑，嗯，那可能会以某种方式被奖励黑客攻击。但这是否是模拟到现实的一个更大的问题，任务视野越长，在数据中心内模拟就越难，对吧？在我看来，至少甚至在编码中，我们已经到了这样一个点，嗯，没有一些长达一年的编码任务最终不需要你与客户交谈或与公司互动或与用户互动。如果你考虑我们希望 AI 能够胜任的所有事情，你最终希望超级智能能够经营企业或创办新企业并使其盈利，或者在市场上进行有利可图的日内交易，或者赢得官司。而这些都是很难在数据中心模拟的事情，学习的内在部分是与现实世界互动。

Yeah. I think your biggest problem is actually just not knowing what the reward function should be from natural data. And if you use some kind of superficial signal like did they accept the the code the the edit um you might that might get reward hacked in some way. But is it this seems like a bigger issue with the sim to real thing where the longer and longer horizon tasks get, the harder they are to simulate within a data center, right? It seems to me already potentially at least even in coding, we're getting to over the point where um there there's like not some year-long coding task that doesn't eventually require you to like talk to a client or interact with the company or interact with users. And if you think about the gamut of things we would want AI to be capable at, you want eventually super intelligence should be able to like run a business or like start a new business and make it profitable or like have a profitable day trading in the markets or win a court case. And these are all things which are very hard to simulate in a data center like inherent part of the learning there is interacting with the real world.

所以也许它们得学会通过从模拟到现实的迁移，在这些事情上做得更好。但另一种可能是，你确实需要从这类交互中获得权重更新，才能在这些事情上变得更强。而如果真是这样——如果迁移不够强、你确实需要权重更新——那么模型样本效率相当低这件事，也许就是个更深层的问题。我好奇这一点的原因是，我觉得按默认情况看，我想不出未来 10 年内怎么会不出现某种疯狂的递归自我改进。但这件事可能不发生的唯一原因，就在于权重更新——权重更新的样本效率，它们看起来远远落后于人类，对吧？人类从出生到成年所见的数据量，对比模型从冷启动到训练完成所见的数据量， plausibly 差了上百万倍。所以这一切是说：首先，模拟和我们在现实世界中希望 AI 完成的极长时程、极其复杂的任务之间，会有良好的迁移吗？如果没有，那是否真的意味着这些模型样本效率的缺失会反噬我们？

So maybe they better learn how to get better at these things from the transfer between sim to real. But alternatively, maybe you do need weight updates from these kinds of interactions in order to get better at them. And then if that is the case—if transfer isn't strong enough and you do need weight updates—then the fact that the models are quite sample inefficient is maybe a deeper problem. The reason I'm curious about this is I feel like by default I don't see how you don't get some kind of crazy recursive self-improvement within the next 10 years. But the one reason why that might not happen is in terms of weight updates, the sample efficiency of weight updates—they just seem way far behind humans, right? Like plausibly a millionfold behind humans in terms of how much data a human sees from birth to adulthood versus how much a model sees from cold start to finishing training. So this is all to say: first of all, is there going to be good transfer between simulations and the extremely long-horizon, really complicated real-world tasks we want the AI to do? And if not, does that really mean that the lack of sample efficiency in these models comes to bite us?

我觉得可以这样拆解模型会变强和模型会持续挣扎的两类任务：任务到底是累积性的，还是你面对的是非平稳分布、必须不断学习和反复重新处理一堆东西。累积性任务的一个例子可能是 RSI——理论上可能存在一个不到一百万 token 的 Python 文件，从零开始训练出一个具备递归自我改进能力的模型，而你做出的每一项发现都像是一条你守住的沙线。如果对 RSI 来说，我们不需要再发现新的注意力变体之类的东西——一旦你发现了注意力机制，一旦你发现了混合专家，一旦你发现了 GPO，你只要把它加进训练栈，它就在那里了。一个很好的例子是 GPT-5.6 或者 OpenAI 让它训练的那个版本——它不必回头重新发现注意力机制；它大概率只是调用一堆像 pre-training.sh 和 post-training.sh 这样的脚本，然后就这么做了。所以那是累积性任务的一个例子。我觉得真实世界——也是人们如此关注持续学习的原因——并不是真正的累积性任务。想象在一家律所里，你有一个智能体担任法律助理。那是一个非常非平稳的分布。你必须把公司里所有重要人物之间的所有关系都装进上下文，而这些关系也在不断变化。

I think maybe the way I'd break down the two types of tasks in which models get good and models will still continue to struggle is whether the task is cumulative or whether you kind of have this non-stationary distribution where you have to keep learning and relitigating a bunch of stuff. So maybe an example of a cumulative task might be RSI—it's theoretically possible to have a less-than-a-million-token Python file which from scratch trains a model that is capable of recursive self-improvement, and every discovery that you make is kind of a line in the sand that you hold. If it's true that for RSI we don't need to discover a new attention variant or whatever—once you've discovered attention, and once you've discovered mixture of experts, and once you discover GPO, you just add that to the training stack and that's there. And a good example of this is like GPT-5.6 or whatever one OpenAI told it to train—it didn't have to go back and discover attention; it probably would have called a bunch of scripts like pre-training.sh and post-training.sh and just did that. So that's an example of a cumulative task. I think the real world—and the reason people are thinking so much about continual learning—is it's not really a cumulative task. Imagine in a law firm you have an agent acting as a legal associate. That's a very non-stationary distribution. You have to be able to fit in your context all the relationships between all the important people at that company, which are also changing all the time.

你还有所有这些关于事情怎么做、去哪里找信息的隐性方式。那不像 RSI 那样是累积性任务的干净例子。所以我认为任务之间会出现这种分化。但如果实验室意识到这一点，并且他们确实相信 RSI 是累积性的——即我们不需要回头发现某种全新架构之类的东西——那么也许越来越多的精力和算力会集中到那上面，而不是——

You have all these implicit ways about how things are done, where to find information, etc. And that's not as clean an example of a cumulative task as RSI is. So I think there will be this breakdown between tasks. But if the labs realize that and they do believe that RSI is cumulative in the sense that we don't need to go back and discover some brand-new architecture or whatever, then maybe more and more effort and compute gets focused on that versus the—

太不幸了，RSI 居然恰好比律师助理工作更容易。

It's so unfortunate that RSI happened to be easier than paralegal.

是啊。

Yeah.

嗯，不知道你们几位对这个问题有没有想法。

Yeah, I don't know if you guys have thoughts on this.

嗯，我会说今天的模型在很多不同方面都比人类弱，其中一些可能跟特定区间内的样本效率有关。在某些区间里模型样本效率很高，比如上下文学习。但可能还存在某个中等长度的区间，它们样本效率较低，因为人类能以比模型更高效的方式做某种权重更新。所以我认为在某些区间样本效率较低，可能是弱点的来源之一。但我也认为还有其他完全不同的弱点来源——比如思维的多样性低于人类，或者在某些长时程判断上表现很差。我的意思是，我觉得人们所说的很多“品味”，是关于某种长期有效的、人们已经意识到长期有效的行为。不是全部，但品味的某些方面——尤其对软件工程这类事情——我觉得很多品味是关于哪些系统在项目的长期运行中可维护、能良好运作。所以是的，我认为限制 RSI 的人类弱点，连同其他因素，种类很多，有些跟样本效率有关，有些无关。

Yeah, I would say today's models are weaker than humans in a lot of different ways, and some of them might have to do with sample efficiency in a certain regime. In some regimes models are very sample efficient, like learning in context. But then there might be some medium-length regime where they're less sample efficient because humans can do some kind of weight update more efficiently than models. So I think being less sample efficient in certain regimes might be one of the sources of weakness. But then I think there are other sources of weakness that are completely different from that—for example, having lower diversity of thought than humans, or being bad at certain kinds of long-horizon judgments. I mean, I think a lot of what people call taste is something about behavior that works in the long run and that people have realized works in the long run. Not everything, but some aspect of taste—especially for something like software engineering—I think a lot of taste is about what are the systems that are going to be maintainable and work well in the long run of this project. So yeah, I think the weaknesses of humans which limit RSI, along with other things, are a variety of them, and some of them are related to sample efficiency and some aren't.

也许一个有趣的思想实验是：如果你能给模型一个上下文窗口，比如一万亿 token，或者足以装下你在比如说 RLHF 之前的全部经验——它把所有这些经验都放进上下文窗口，并且保持和一百万 token 时相同的样本效率与上下文学习能力——你觉得品味问题就解决了吗？它能做出和你一样的判断吗？还是说，除了更长上下文窗口加同样样本效率之外，还有某种根本性的缺失？

Maybe an interesting thought experiment is: if you were able to give a model a context window of, I don't know, a trillion tokens or whatever you would have needed to fit in your experience prior to, let's say, RLHF—and it's got all that experience in the context window and has the same sample efficiency and in-context learning ability as it does at a million tokens—do you think taste is then solved? Would it be able to make the same judgments that you did, or is there something fundamentally missing apart from just a longer context window with the same sample efficiency?

嗯，我的意思是，它必须被训练成能从那个上下文中学习。所以我不确定——要么它必须被训练成能从那个上下文中做出正确的更新，要么——

Yeah, I mean it would have to be trained to learn from that context. So I'm not sure—either it would have to be trained to learn the right update to make from that context, or—

你得把全部东西都倒进去——比如你的一生、你全部的科研经历。

You have to just dump it all in—like your whole life, your whole research experience.

我的意思是，你仍然需要数据来训练它的长上下文，对吧？即使理论上你能获得一万亿的上下文，你也需要一万亿长度的数据来训练它。就像现在你有上下文——

I mean, you still need the data to train it long context, right? Even if you could theoretically get a trillion context, you would need a trillion lengths of data to train it. Like right now you have like context—

我只是在问，理论上如果你有那个的话。

I'm just asking if you had that in theory.

我觉得答案是肯定的。我的意思是，这真的就归结为一个问题：品味从较短时程的片段中有多强的元可学习性。我觉得没有明显理由说它必须超级长，因为人类不知怎么就在没有很多长片段的情况下发展出了品味——我们又不是活到一万岁。

I think yes. I mean, this really just comes down to the question of how meta-learnable taste is from shorter-horizon episodes. And I feel like there's no obvious reason it's super long, because humans somehow develop taste without having many long episodes—like we don't live to be 10,000.

我们发展得很快，对吧？所以如果你想想，即使在博士阶段，一个一年级博士生和一个最后的博士后之间的差距，那大概是五年，他们总共可能只做了 10、15、30 个研究项目，但不知何故他们从相对短暂的一系列小事中很快培养出了品味。所以理论上是有可能这样培养的。AI 显然会有多得多的经验来培养品味，来元学习它。而这在多大程度上能泛化到真正长视野的事情上，我认为是目前真正未解决的问题。我们不知道。回到这个问题，最终应该有一个阶段，AI 从每次部署的实例中学到大量东西。目前你可以说有一个元模糊过程，模型确实从部署中改进，但我觉得这是一个非常弱的反馈回路。

We develop pretty quickly, right? And so if you think about even in a PhD, the difference between a first-year PhD student and a final postdoc or something, that's like five years maybe, and they've only done maybe 10, 15, 30 research projects in total, but somehow they develop taste quite quickly from a relatively short succession of small things. And so theoretically it's possible to develop it like that. The AI obviously will have vastly more experience in which to develop taste, to meta-learn it. And how well does that generalize to really long-horizon things is I think the question which is really unsolved at this point. Like we don't know. Going back to this question, eventually there should be a regime where AIs are learning a ton from each individual instance of deployment. Well, currently you could say there's a meta fuzzy process by which models do improve from deployment, but I feel like it's a very weak feedback loop.

你看到这种快速学习的蜂巢思维即将出现了吗？如果是，它究竟如何发生？

Do you see this around the horizon where there's this hive mind kind of learning that's very rapid? And if so, how exactly does it happen?

实际上我想说，关于我们是否会得到一个从所有部署经验中学习的蜂巢思维？其中很大一部分实际上是关于激励，而不是技术问题。所以公司不会希望模型提供商从他们所有的部署中学习，因为那可能只会降低他们业务的优势。

Actually I would say that around will we get a hive mind that learns from all of its deployment experience? I mean a big part of that is actually about incentives rather than being a technical question. So companies aren't going to want to have the model provider learn from all of their deployment because that might just reduce the advantage of their business.

我认为也许经济因素会迫使不是对一个大共享模型进行权重更新，而是某种模块被替换进来。一个很明显的例子是 LoRA，但也可能是别的。你知道有很多工作试图将任意上下文长度适配到固定大小，比如所有线性注意力之类的东西，还有像“卡带”这样的东西，本质上是训练成非常非常压缩的 KV 缓存，以容纳大量信息。这是另一个例子，公司可能愿意接受，如果它被替换到模型中，而实际上并不改变底层基础模型本身。所以有很多不同版本的从数据中实时学习，而后面的那些并不真正帮助大实验室，因为它们只是这些模块。但我认为经济压力会迫使实验室先走这条路，然后才能开始这个。

I think that maybe the economics of this will pressure not necessarily weight updates to one big common shared model but kind of modules that get subbed in. Like a very obvious example, this is a LoRA, but it might be something else. Like you know there's been a lot of work to try and fit an arbitrary context length into a fixed size, like this is all the linear attention stuff and all that sort of stuff, and like cartridges which are essentially KV caches trained to be very very compressed KV caches to fit in a lot of information. That's another example of something that companies may be willing to sign up for if that gets subbed into the model and it's not actually changing the base underlying model itself. So like there's many different versions of learning from your data in real time, and the latter ones are not really helping the big labs because they are just these modules. But I think the economic pressure will force the labs to go down that path first before they can embark on this.

但究竟是哪种经济压力？因为我觉得即使你有一堆卡带或 LoRA 之类的，你仍然可以拿走所有这些轨迹，然后将其蒸馏到你下一代模型的预训练中。

So which economic pressure though? Because I feel like even if you have a bunch of cartridges or LoRAs or whatnot, you can still just take all these traces and just distill this down to the pre-training of your next generation of models.

是的。所以大实验室获得的可能是一种更间接的学习形式，这显然对他们仍然非常有价值，但我无法想象一个世界，我们从一开始就说，我们要直接在一个大模型上训练所有精确数据。我认为它肯定会经历阶段，因为这是假设有一个不连续的事件，突然我们连续地修复权重更新，而在实践中，我认为更可能是这样的：卡带之类的东西让你在部署中专业化，然后你生成轨迹，你把它放入你的模型，比如 3 个月后你推出一个在这方面更好的模型，你再次专业化并再次巩固，然后最终我们会让这个循环越来越快。所以不是每 3 个月发布一个模型，现在是每周，然后每天，然后每小时，到那时我们基本上显然已经解决了它。

Yeah. So it may be a more indirect form of learning that the big labs are getting, and that's obviously still really valuable to them, but I can't imagine a world in which we start off with like, you know, we're going to just directly train this one big model on all the exact data. I think it will definitely go through stages because this is assuming there's one discontinuous event whereas suddenly we fix weight updates continuously, and in practice I think it's much more likely to be like the cartridges and stuff allow you to specialize in deployments, then you generate traces, you put that in your model, like 3 months later you come out with a model which is better at this stuff, you specialize it again and you consolidate it again, and then eventually we'll just make this loop faster and faster. So instead of every 3 months we release a model, now it's like every week and then every day and then every hour, at which point we basically have obviously solved it.

是的，我认为这也是一个好观点，因为你问的是我们离能够做到这一点有多远。我想我们对此做了一些研究，人们也在非常大的规模上做了很多研究。就像当你洗掉足够的噪声并且有足够大的批次时，这种将数据放入中期训练并创建我们的环境的外循环过程，它确实在某种持续学习机制中起作用。但问题是，当你放大到微观层面时，就像我有一个模型，我试图再次为一家律师事务所更新它。我试图非常连续地用相对少量的数据来做这件事。就像所有方法都有点崩溃。所以如果我仅对成功的轨迹进行 SFT，离策略、在策略，最终在非常迭代的机制中，当你进行数百次这样的微更新时，你会看到灾难性遗忘，你会看到对之前我在基础模型之上学到的信息的遗忘，并且我看到一般能力的退化。你知道，在策略蒸馏似乎将这一视野推远了一点，但它最终仍然屈服于同样的问题。而强化学习不太擅长——它擅长获取能力，但不那么擅长获取知识，以及这种非常明确的知识，比如，哦好吧，这个人在这个律师事务所做这个，这是我们发现的一个非常具体的过程，你必须投入大量算力来创建正确的环境以获取知识。

Yeah, and I think this is a good point as well because you asked kind of how far off the current paradigm we are from being able to do this. I think we've done a bit of research to this and people have done a lot of research at a really large scale. Like when you wash out enough noise and you have large enough batches, like this outer loop process of putting data into mid-training and creating our environments, like it does work in some sort of continual learning regime. But the problem is when you zoom in close enough at a micro level, it's like I've got one model and I'm trying to update it again for a law firm or something. And I'm trying to do that very continuously with a relatively small amount of data. Like all the methods kind of break down a bit. So like if I SFT the model on just successful traces, off-policy, on-policy, like eventually in the very iterative regime, when you're doing hundreds of these micro updates, you see both catastrophic forgetting, you see forgetting of previous information I've learned on top of the base model that was much earlier on, and I see degradation of general capabilities. You know, on-policy distillation seems to push this horizon out a little bit but it still eventually succumbs to the same. And RL is not very good at—it is good at getting capabilities in, but it's not as good as getting knowledge in, and just this very explicit knowledge of like, oh okay, this person does this at this law firm and this is a very specific process we find, and you have to pour in a lot of compute to create the right environments to get the knowledge in.

你认为这里的根本问题——为什么你在任何这些其他技能上变差，或者有遗忘之类的——你认为这根本上是容量问题还是技术问题？

Do you think that the fundamental issue here—why you get worse at any of these other skills or there's forgetting and stuff—do you think it's fundamentally an issue of capacity or it's an issue of techniques?

我认为两者都有一点。就像 SFT 甚至蒸馏，比如在策略蒸馏，可能破坏性太大。强化学习之所以好，是因为它对模型的改变非常非常小，有很多证据表明为什么是这样，所以它只是在这个非常非常小的损失值上调整它，使其达到正确的点。但这也限制了你能用强化学习做什么，比如你实际上能改变模型多少。

A little bit of both, I think. Like SFT and even distillation, like on-policy distillation, can be way too destructive. The reason RL is so nice is because it changes a very very small amount about the model, and there's a lot of evidence for why this is the case, and so it kind of just tweaks it in this very very small loss value to get it into the right point. But that also then limits what you can do with RL, like how much you can actually change the model.

或者抱歉，你是说这之所以可能不是赢家通吃，是因为很难在不破坏接口中的某些东西的情况下将那么多信息蒸馏到基础模型中？

Or sorry, are you saying like the reason this isn't a winner-take-all potentially is that it's just very hard to distill that much information into the base model without ruining something in the interface?

就像将其蒸馏到一个不同的基础模型是容易的。我认为这主要是技术问题。这绝对不只是容量不够。

Like it's easy to distill it into a different base model. Like this is where I think it's mostly technique. It's definitely not just like there isn't capacity.

比如你有一个模型，带着所有这些数据，然后你拿一个完全相同规模的模型，用所有这些内容在中期训练里从头预训练，它会更好。我觉得今天发生的很多事情就是这样。

Like if you had some model with all this data, and you take literally the same size model and pre-train it from scratch with all of this stuff in mid-training, it will be better. And I think that's a lot of what's happening today.

对。所以很大程度上是有一个瓶颈，阻止我们一直训练同一个模型，而不是把旧模型的所有数据拿出来、从头训练一个新模型。这说的正是可塑性和灾难性遗忘的某种组合——如果你只是天真地在非平稳数据上训练，因为你一边训练一边加入新数据，这基本上就是在扰乱数据分布。于是旧的东西就被遗忘了，而我们并没有真正好的方法来阻止这种情况发生。

Yeah. And so it's very much like there's a bottleneck that stops us from just keeping training the same model forever, versus just getting all the data from the old model and training a new model from scratch. And this is exactly saying some combination of plasticity and catastrophic forgetting—if you just naively train on non-stationary data because you're adding new data as you go, basically this is messing with the data distribution. So the old stuff is just forgotten, and we don't really have good methods to stop that from happening.

所以也许在极限情况下，你就是被“用所有这些新信息从头重训模型”这件事卡住了。

And so maybe in the limit you're just bottlenecked by retraining the model from scratch with all this new information.

对。这当然非常昂贵——从头训练一个模型很贵。

Yes. Which of course is very expensive—training a model from scratch is expensive.

但反正你都会这么做。

But you're going to do that anyways.

我的意思是，不一定。也许最终，如果你有持续学习，你就永远不训练新模型——你只有一个模型，它不断学习和扩展，对吧？

I mean, not necessarily. Maybe eventually, if you have continual learning, you never train a new model—you just have a model and it keeps learning and expanding, right?

但可能有一些深层技术原因让这非常困难，因为这些——

But there might be some deep technical reason why that's very difficult because of these—

我的意思是，这就是问题所在。我觉得我们已经在“需要多少从头训练”这件事上往回推了。现在完全可能拿预训练基座，在上面做非常好的中期训练，某种程度上是持续的，再加上从训练后期不同检查点出发的一些强化学习，这看起来更像学习了。但肯定不是那种拿最新模型、做几次很小的更新、然后迭代地永不丢失的情况。

I mean, that's the question. I think we have pushed back on how much from scratch we need to do. It is definitely possible now to take the pre-trained base and do very good mid-training on top of that, kind of continuously, plus some RL from different checkpoints that are later on in the training, and that's looking more like learning. But certainly not the case of taking the most recent model, applying a couple of very small updates, and iteratively never losing it.

但这不是——我有点困惑，因为这难道不就是训练过程中发生的事吗？在后训练之类的阶段，你有一个已经经过大量训练的模型，然后你蒸馏某个经过进一步强化学习的分支之类的？这不就是实际发生的事吗？

But isn't this—I'm a bit confused, because isn't this literally what happens during training, where during post-training or something you just have a model that's already gone through so much training, and then you distill some fork that's been further RL'd or something? Isn't that literally what happens?

我觉得它仍然处在足够大的规模上，所以你会洗掉很多噪声，而且你不是只专注于一个分布——正如 Beren 所说，现在非常——如果你只专注于一个任务，对吧？在最终的状态下你会做的是，我不知道，有数十亿个部署实例，你同时从所有这些实例中学习，所以希望这能洗掉一些噪声之类的东西，对吧？

It's still at a large enough scale, I think, that you're washing out a lot of the noise, and you're not just focused on one distribution—which, as Beren said, is now very much—if you're just focusing on one task, right? In the eventual regime you'd be doing, I don't know, there's billions of deployed instances, you're learning from all of them at once, and so hopefully there's some washing out of noise and stuff from that, right?

也许在那个规模上。对。

Maybe at that scale. Yeah.

对。我的意思是，我觉得确实，就像我说的，你可以做很长时间的持续中期训练，也可以回滚到某个检查点、喂入新的中期训练数据。但与此同时，你不能无限地这么做——如果你就一直对同一个基座持续中期训练下去，它某种程度上会渐近饱和。你没法在那个基座里学到新东西，这就是为什么人们最终会训练新的基座。否则你就只会一直中期训练同一个基座。

Yeah. I mean, I think definitely, as I was saying, you can do continual mid-training for a long time, and you can roll back to a checkpoint and give new mid-training data. But at the same time, you can't do this indefinitely—if you just keep continuing mid-training the same base forever, it sort of asymptotes. You can't just learn new stuff in that base, and this is why people end up training new bases. Otherwise you would just keep mid-training the same base forever.

每次我录完一场访谈，我都会立刻把所有想法一股脑倒进 Slack——比如什么最有意思、什么该剪掉。这能确保我的编辑们拿到开始剪辑这期节目所需的全部背景。但这些脑暴笔记并没有清晰的时间戳，而我没剪过的录音长达好几个小时。光是找到我提到的那些确切片段，就可能耗费编辑大量时间。所以我们决定试着往聊天里加一个 Grok Bot 制作人。现在，只要我的某位编辑发出这期节目的粗剪版，Grok Bot 就会在它自己的电脑上打开转录稿开始干活，通常在我看到消息之前就开始了。它会拿我丢进 Slack 的笔记，在转录稿里高亮出相关片段。它还会用我整理的一份包含我所有偏好的大案例文件，来给出可能的剪辑建议。做完之后，它会把它最看好的候选片段发给我，让我能在手机上审阅全部内容。这招效果非常好。能像给编辑发短信一样发些非正式消息，然后转录稿立刻反映出我的偏好，真是太有用了。你可以自己在 x.ai/bot 试试 Grok Bot。

Whenever I finish recording an interview, I immediately brain dump all my thoughts into Slack—things like what was most interesting and what should get cut. This ensures that my editors have all the context they need to start editing the episode. But it's not like these brain dumps have any clear timestamps, and my unedited recordings are many hours long. It can take a ton of editor time to even find the exact moments that I was referencing. So we decided to try adding a Grok Bot producer to our chat. Now, whenever one of my editors posts a rough cut of the episode, Grok Bot opens a transcript on its own computer and starts working, usually before I've even seen the message. It takes the notes that I dropped into Slack and highlights the relevant snippets in the transcript. It also uses a big case file that I've compiled with all my preferences so it can suggest potential edits. And when it's done, it sends me its top clip candidates so that I can review everything from my phone. This has worked really well. Being able to send informal messages like I'm texting my editor and then having the transcript immediately reflect my preferences has just been so helpful. Try Grok Bot yourself at x.ai/bot.

好，我们现在聊一聊数据。我总体上对这个问题很感兴趣：AI 的进步有多少只是由数据进步解释的。这并不意味着它一定难以自动化，但那是另一个问题。那么，是否存在某种数据分布，如果你用当前架构在它上面训练，就会得到一个在每个领域都完全碾压人类专家的超级智能？

Okay, let's talk a bit about data now. So I'm generally interested in this question of how much of AI progress is just explained by data progress. Doesn't mean it will necessarily be hard to automate, but that's a separate question. So is there some data distribution which, if you trained current architectures on, would result in a superintelligence that totally dominates human experts across every single field?

我们说的是预训练加后训练数据，也包括环境吗？我觉得这种东西的存在是显而易见的。问题只在于我们能否造出正确的环境。

Are we talking about pre-training plus post-training data, like environments as well? I think the existence of this is obvious. It's just whether we can create the right environments.

对。

Yeah.

在最简单的情况下，我们可以直接训练它输出那个训练出真正超级智能的 Python 文件——就让它记住那些权重。

In the trivial case, we could just train it to output the Python file which trains the actual superintelligence—just have that memorizing the weights.

是的，可能存在一个可以构造出来的环境阶梯，让你得到一个至少和人类研究员一样好的 AI 研究员。但每往上爬一级，所需的努力大致呈指数增长。而这两件事就是你必须权衡的，以判断我们多快能到达最后那一级、也就是它更好的那一级。我觉得这一点相当清楚。而且我认为我们在环境创造方面还相对早期。我们利用了很多不对称性来创造好的环境。其中一个不对称性，我们之前聊过，就是有些环境更容易正推和反推。我的意思是，定义一个复杂的数据生成过程非常容易，这就像你向模型隐藏的一个潜变量。你可以生成任意复杂的环境，而模型必须花费大量不可削减的 token 和不可削减的工作，才能弄清楚那个数据生成过程是什么。还有一些不对称性，比如你可以从真实世界注入信息。

Like yes, there's probably a ladder of environments that it's possible to construct such that you would get an AI researcher which is at least as good as a human researcher. But the effort to climb each successive rung grows kind of exponentially. And that's going to be the two things that you have to trade off against as to how fast we're going to hit that final rung where it's better. I think that's fairly clear. And I think we're still relatively early in our environment creation. There's a lot of asymmetries that we exploit in order to create good environments. So one of those asymmetries, which we've talked about before, is that there are environments where it's easier to go backwards and forwards. And what I mean by that is it's very easy to define this complex data generating process, and this is like this kind of latent variable you keep hidden from the model. You can generate arbitrarily complex environments, and the model has to do a lot of irreducible token spend and irreducible work to figure out what that data generating process was. There's asymmetries in terms of you can inject information from the real world.

Anthropic 通过结合数万小时的人工和 LMR 时间发现了一个 bug，然后把它变成一个非常非常精巧的环境，单个语言模型理论上可以在几百万个 token 内找到它。所以我们 cherry-pick 了所有这些不对称性，并指望这种任务视野上的泛化。但我觉得，同样地，某个时候一定会出现收益递减——在创建这些环境本身的难度上出现收益递减，比如想出这些环境，因为你不一定能直接拥有这些过程，让倒退比前进更容易。你实际上必须坐下来构建某种东西，对人类来说，它看起来像是一个足够长的时间视野——创建它将会是一项非常复杂的任务。然后还有智能体实际执行这些任务时的算力和时间瓶颈。所以我觉得你只会开始看到这条曲线变平。

So Anthropic finds a bug through tens of thousands of human and LMR hours combined, and turns that into a very, very neat environment which a single LM could theoretically find within a few million tokens. So there are all these asymmetries which we're cherry-picking, and we're counting on this task-horizon generalization. But I think, again, there are just going to be diminishing returns at some point—diminishing returns in how hard it is to create those environments in the first place, like coming up with them, because you can't necessarily just have these processes where it's easier to go backwards than forwards. You actually have to sit down and construct something that, with humans, looks like a long enough time horizon—it's going to be a really complex task to create. And then there are also going to be the compute and time bottlenecks for the agent to actually do those tasks. So I think you're just going to start seeing this curve flatten out.

我看到过一些东西，说有人把 Talkie 模型——它只训练到 1930 年的数据——在现代编码智能体数据上做了微调，结果它在 SWE-bench 上比 Claude 3 Opus 表现更好。所以这个完全不懂代码的模型，可以在适量数据上微调，然后作为编码智能体表现得比这个大得多的预训练模型更好——这挺疯狂的。这也多少说明，一旦你有了正确专家行为的例子，把它复制到一个相对较弱的模型里其实出奇地容易。

I saw something about how someone fine-tuned the Talkie model—which is only trained on data up to 1930—on modern coding-agent data, and it did better than Claude 3 Opus on SWE-bench. So this model that has no knowledge of code whatsoever can be fine-tuned on a moderate amount of data and behave better as a coding agent than this much larger pre-trained model—that's pretty crazy. And it kind of shows you that once you have an example of the right expert behavior, it's actually surprisingly easy to copy that into a relatively weak model.

是的。但一个反例是，最近有一篇论文，他们把它训练到五年级数学，还有小学英语之类的。所以它是一个还不错的语言模型，然后他们试着用强化学习让它做高中后期和大学数学，差距实在太大了。他们根本没法让它爬上去。但如果你做连续的训练——比如七年级数学，然后八年级数学，以此类推——你显然可以爬到十二年级。所以 again，问题就是：那些梯子上的横档之间距离有多远，以及创建它们有多难？

Yeah. But a counterexample to that is there was a paper recently where they trained it up to fifth-grade maths, and also like primary-school English and stuff. So it was a decent language model, and they tried to RL it to do late high-school and college maths, and the gap was just too large. They couldn't get it to climb at all. But if you did successive runs—like year-seven maths, then year-eight maths, and so on—you could obviously climb to year 12. So again, it's just: what is the distance between the rungs on those ladders, and how hard is it to create?

是的。这就回到了强化学习信号的问题。强化学习现在不太擅长探索。所以如果模型在 128 次 rollout 里都找不到，它极不可能获得进展的信号。这就是为什么在强化学习中我们需要课程，而在预训练中我们不需要，因为这对预训练来说根本不是问题。

Yeah. And this just comes back to the RL signal problem. RL is not very good at exploring right now. So if the model can't get in, you know, 128 rollouts, it's very unlikely to get signal to progress. And this is why in RL we need curricula, whereas in pre-training we don't, because that's not a problem for pre-training at all.

是的。而且 again，预训练数据和后训练数据是不同的。我想随着我们继续推进，人类的参与会越来越少，但这不改变一个事实：你受限于能从真实世界提取多少信号。世界上有很多信号——这是真的，有人在处理电子表格任务，有人在处理法律任务等等——但模型现在所处的能力前沿：世界上有多少比特实际上真正与提升模型能力相关？有多少新的数学问题正在被解决，而这些问题超出了当前模型的能力范围或理解范围？有多少新的编码问题正在被创造或解决，而它们超出了当前模型的能力范围？我觉得这就是收益递减出现的原因，因为即使整个世界也没有给你那些有用的比特，把你推入下一个能力盆地。

Yeah. And again, pre-training data is different from post-training data. And I imagine as we continue on, humans will be involved less and less, but that doesn't change the fact that you're bottlenecked on how much signal you can extract from the real world. So there's a lot of signal in the world—that's true, there are people doing spreadsheet tasks, people doing legal tasks and all this sort of stuff—but the capability frontier of where the models are at now: how many bits in the world are actually really relevant to improving the model's capabilities? How many new maths problems are being solved that are just beyond the reach or grasp of the current models? How many new coding problems are being created or solved that are beyond the reach of the current models? And I think that's why the diminishing returns kicks in, because even the world as a whole is not giving you the bits that are useful for tipping you into the next basin of capability.

是的，我完全同意。所以这真的是信号来自哪里的问题。在预训练中，信号已经在 Common Crawl 里了，对吧？对于预训练中你关心的任务，问题根本不是获取信号——而是过滤掉所有存在的噪声，而这是一个相当可自动化的过程。但随着模型变得更好，随着我们进入中期训练和后训练，信号在原始数据里根本不存在。再多的过滤也得不到——Common Crawl 里没有隐藏的千禧年大奖难题的证明，让我们可以一直过滤直到看见它。所以到那个时候，你必须以其他方式获取比特：要么直接来自人类，比如让他们写出推理过程；要么通过创建环境，由人类决定应该创建什么环境以及这些环境的目标是什么；要么在部署中存在的真实人类数据上进行某种训练。你必须从某个地方获取比特。

Yeah, I totally agree with this. So it's really a question of where the signal is coming from. In pre-training, the signal is already in Common Crawl, right? For the tasks you care about in pre-training, the problem is not just getting signal at all—it's filtering out all the noise that exists, and that's quite an automatable process. But as the models get better, as we enter mid-training and post-training, the signal just doesn't exist anywhere in the original data. No amount of filtering will get this—there's no hidden proof of the Millennium Prize problem sitting in Common Crawl that we can just filter until we see it. And so at that point you have to get bits some other way: either from humans directly, like asking them to write out their reasoning, or by creating environments where humans decide what environment should be created and what the objectives of these environments are, or some kind of training on the human data that exists in deployment. You have to get the bits from somewhere.

是的。有一个问题是，预训练中有多少进展是由数据驱动的。

Yeah. There's a question of how much of the progress in pre-training is being driven by data.

我和 Jerry Han——普林斯顿的一个学生——做了这项研究，我们基本上把 2019 年至今的所有训练配方与 2019 年至今的所有数据集两两配对训练。所以你在最新的数据集比如 UltraFineWeb 上训练 GPT-2，然后在 Pile 或某个旧数据集上训练 Dolly——它是最新的训练配方，或者说开源训练配方——你跑完整个网格，看达到某个能力水平需要多少更少的算力。你发现数据似乎解释了大约 9 倍的算力效率提升，而架构改进在非常小的规模上解释了大约 3 倍的算力效率提升。所以，如果这在大规模下也成立——即预训练算力效率提升的大部分来自更好的数据——那这能持续多久？你能一直越来越多地过滤数据、构建越来越多的合成数据，直到……是的，你对这种预训练进展能持续多久有感觉吗？

I did this investigation with Jerry Han, who's a student at Princeton, where we basically trained all the recipes from 2019 till now pairwise with all the data sets from 2019 to now. So you train GPT-2 on the newest data set, like UltraFineWeb, and you train Dolly—which is the newest training recipe, or the open-source training recipe—on the Pile or some old data set, and you do the whole grid, and you see getting to some level of capabilities: how much less compute does it take across this grid? And you see that the data seems to explain like a 9x compute efficiency gain, but the architecture improvements explain like a 3x compute efficiency gain at a very small scale. And so, to the extent that that is true at large scale—that most of the pre-training compute efficiency gains are coming from better data—how much can that continue? Can you keep just filtering data more and more and building more and more synthetic data until... yeah, do you have a sense of how much this kind of pre-training progress can continue?

我觉得我的先验是，again，低垂的果实某种程度上已经摘完了。我们得到了互联网这个大盘子，而互联网不一定以同样的速度增长——互联网上所有有用的东西都以同样的速度增长。所以我们可能还有一堆 0.1% 的损失下降可以拿，但肯定没有目前已经发生的那么多。

I think my prior is that, again, the low-hanging fruit is somewhat exhausted. We got the internet as this big block, and the internet isn't necessarily growing at the same rate—all the useful stuff on the internet is growing at the same rate. So we've probably got a bunch of like 0.1% loss drops to go, but definitely not as many as have currently occurred.

但同样很有意思的是，你发现这个——总共累计大约 27 倍的提升，两者加起来？

But it's also really interesting that you find this—what, cumulative like 27 times improvement across both?

我记得是 Epoch 或者某个人估算过，自 2019 年以来大约每年提升三倍，那意味着 3 的 7 次方，也就是超过 2000 倍的提升。那缺失的那 100 倍左右是从哪来的？这大概能给你一个很好的信号，说明其中有多少来自后训练。

I think it was Epoch or someone who estimated like three times a year since 2019, which would imply something like 3 to the 7, like over 2,000 times improvement. So where's that missing 100 times or whatever coming from? That probably gives you a good signal of how much of this is post-training.

我觉得解释只能是：很多算力效率的提升是依赖规模的，而我们是在极小的规模上做研究。

I think the explanation has to be that a lot of the compute efficiency gains are scale dependent and we're studying at extremely small scale.

这就引出一个问题：数据算力效率的提升，还是算法算力效率的提升，哪个更依赖规模？我不知道你们对此有没有先验判断。我们只是没有足够的算力去研究这个问题。

And that raises the question of whether the data compute efficiency gains or the algorithmic compute efficiency gains have more scale dependence. I don't know if you guys have a prior on that. We just didn't have enough compute to investigate that question.

我是说，单纯从直觉上讲，理论上架构的规模依赖性是相当清楚的，你可以拟合出一条直线。但对于把预训练、后训练数据和中期训练数据结合起来，我完全不知道该怎么拟合。

I mean, just naively, right, theoretically the scale dependence of the architecture is fairly well-known, and you can fit a straight line to it. Whereas I would have no idea how to do that for combining pre-training plus post-training data and mid-training data.

说来有趣，我反而觉得数据在规模变大时更重要。我觉得架构有点像一次性的东西。我认为把效率提升简单说成“提升了 x%”是有点误导的，因为架构的作用是让你进入一个质变的新区间，这是旧架构到不了的。而在那个区间里，显然数据才是决定性的因素。但如果我们连 GQA 都没有，一直用全注意力，我们根本做不了百万上下文——做百万上下文的开销会高得离谱。正因为如此，我们就永远用不上那些真正处于百万上下文的数据，也就拿不到这些能力。可如果你只是天真地问“这东西在 2k 上下文下表现如何”，而架构在那里并没有解锁任何东西，那数据看起来就会比它实际上重要得多。我不确定这些东西真的只是这样相乘的增益。

Funny enough, I feel like data is actually more important with scale. Like I feel like architectures are kind of a one-time thing, you know. I think combining, saying just an x% efficiency gain is kind of misleading, because what an architecture does is let you reach a qualitatively new regime which you couldn't reach with the old architecture. And then within that regime, obviously the data is the primary thing determining it. But if we didn't have even GQA, if we were doing full attention all day, we wouldn't be able to do a million context — it would be ridiculous expense to do a million context. And because of that, we could never use the data which is actually at a million context, and so we couldn't get these capabilities. Even though if you just do a naive 'how much does this do at 2k context' where the architecture isn't unlocking anything, then the data will look much more important than in some sense it is. It's unclear to me that these things are really just multiplicative gains in this way.

我明白了。那么关于数据的规模依赖性，结论是什么？

I see. So but then what is the takeaway for the scale dependence of data?

关于规模依赖性，我觉得我们现在很多中期训练和后训练数据，其实是规模越大效果越好，因为其中很多——那些超长上下文视野的环境——确实需要大模型才能用得上。

So on scale dependence, I think a lot of the mid-training and post-training data we have now actually gets better with scale, because a lot of it — the very long context horizon environments — really requires big models to be able to make use of them.

对。

Yeah.

而且这不是——你知道，如果你拿一个一亿参数的模型去训练三条 bench 轨迹，它根本不会有任何进展。它不会展现出你用真正合理规模的模型训练时能得到的那种提升。

And this is not — you know, if you try and train your 100 million parameter model on three bench traces, it's not going to get anywhere. It's not going to show you the same kind of improvement that you would get if you train an actual sensible size model on it.

对。

Yeah.

对。而且现在也很难，因为很多架构改动——你看 Kimi，或者 DeepSeek——他们做这些架构修改时，不只是想着降低预训练损失，还考虑比如模型在真实世界里会怎么被使用，也就是推理效率。比如 DeepSeek 模型里某种压缩注意力，未必是为了“这从根本上是个改进”，而只是说，好吧，我们在考虑模型会怎么被使用。

Yeah. And it's hard as well now because so many of the architecture changes — you look at Kimi for instance, or DeepSeek — they're doing these architectural modifications not just with dropping the pre-training loss in mind, but for instance how the models are going to be used in the real world, the inference efficiency. Like having some form of compressed attention in the DeepSeek models is not necessarily geared around 'this is fundamentally an improvement' — it's just like, okay, we're considering how the models are going to be used.

对，对。我很好奇的一个问题，为了理解未来，是在我们越来越进入强化学习为主的范式时，参数规模会怎么走。我不知道历史上有多快——你可以看开源架构，看参数增长得多快，前沿开源模型大概是每年翻一倍。而即便前沿闭源模型也只有 1000 亿或 2000 亿激活参数——你觉得这还会逐年翻倍吗？还是说现在我们处在强化学习范式里，你还要在 rollout 上节省算力，而且也许存在阈值效应，容量够了之后，再随意增加参数就没那么重要了。你们觉得到 2030 年，一个前沿模型会有多少激活参数？

Right, right. One question I'm curious about, to understand the future, is how parameter scaling will go as we're getting into more of an RL-heavy regime. I don't know how fast historically — you can look at open-source architectures and see how fast parameters have been scaling, and maybe it's roughly 2x every [year] for frontier open-source models. And to the extent that even frontier closed-source models have like 100b or 200b active parameters — do you think that keeps 2xing year over year? Or now that we're in an RL regime where you also want to conserve compute on rollouts, and also maybe there is a threshold effect where you have enough capacity and at that point increasing parameters arbitrarily doesn't matter as much. Do you guys have a sense of in 2030 how many active parameters will a frontier model have?

对，我觉得未来几年，因为我们非常专注于为强化学习做越来越长视野的 rollout，而推理效率在那里很重要——感觉模型在这方面的能力还没饱和，瓶颈仍然在环境上。所以我们可能会看到一点平台期。我有种感觉，像 Mythos 和 GPT 系列模型，比人们谈论的 10 万亿参数规模要小得多。就算只是天真地和开源模型对比一下，你大概也能反推出这个结论。

Yeah, I think for the next few years, because we're so focused on doing longer and longer horizon rollouts for RL, where inference efficiency matters a lot — it feels like the models aren't necessarily saturated on their ability to do that, where the bottleneck is still the environments. And so we might see a little bit of plateau. I have a feeling that, you know, like Mythos and the GPT models are much smaller than the 10 trillion parameter range that people are talking about. Even just naively comparing them to open-source models, you can probably back out that conclusion.

所以是的，未来几年我大概不会预期参数数量有巨大增长。但话说回来，这里要权衡的东西太多了。你根据自己有多少预训练数据，以及要训练的强化学习环境的难度，来决定模型大小。理想情况下你想达到最优点：在你最难的环境上能拿到不错的通过率之类的，而把模型做得更大去通过那里就没意义了，因为那样你只是在付远超所需的推理算力。所以这有很多输入因素——取决于像 Mccor 以及内部这些团队能多快把训练用的 rollout 环境的复杂度扩上去。

So yeah, probably for the next few years I wouldn't imagine a huge growth in the number of parameters. But again, there are so many different things to trade off here. You decide the size of your model based on how much pre-training data you have, and then the difficulty of the RL environments that you've got to train on. And you ideally want to get to the optimal point where you can get a decent pass at one or something on the hardest environments you have, and it wouldn't make sense to make a bigger model pass there because then you're just paying much more inference flops than you need to. So there's a lot of input to this — it depends on how quickly, you know, like Mccor and then in-house these guys can scale up the complexity of the rollout environments they're training on.

我预期模型会继续变大，因为人们在扩大算力，GPU 也越来越大。但具体变大多少，会以不那么显然的方式取决于缩放定律。一点是，我觉得在人们实际采用的架构上，数据效率会比算力效率成为更大的驱动因素，因为我们现在正进入高质量预训练数据有点快用完的范式。这可能会影响你想把模型做得多稀疏。另外我也觉得我们对稀疏性的理解还不够好。

I would expect the models to keep getting bigger, just because people are scaling up compute and the GPUs are getting bigger. But I would say exactly how much they get bigger depends a bit on the scaling laws in non-obvious ways. So one thing is that I think data efficiency is going to be a bigger driver than compute efficiency of the exact architectures people use, now that we're getting to the regime where we're sort of running low on high quality pre-training data. So that might affect how sparse you want to make the model. And then I also think we don't understand sparsity that well.

而且参数和激活参数是不同的资源，但稀疏性确实增加了一些，不过不清楚它会不会无限制地一直增加。可能存在某种最佳平衡点。

And it's like parameters are a different resource than active parameters, but sparsity has definitely increased a bit, but it's not clear that it's going to keep increasing without bound. There might be some kind of sweet spot.

有一种观点认为，稀疏性会让数据效率变差，因为你可能得在多个专家上学习同样的东西。

There's an argument that sparsity should make data efficiency worse, because you might have to learn the same thing on multiple experts.

不过这还有争议。我觉得我们对缩放定律的理论还不够完善，没法真正理解稀疏性为什么有用、能有多大帮助，以及它是否会在某个稀疏程度上遇到瓶颈。

Though that's debatable. I think we don't — I don't think we have a good enough theory of scaling laws that we really understand why sparsity is helping, to what extent it'll help, and whether it'll plateau at some point at a certain level of sparsity.

抱歉，你能具体说明一下数据效率会带来什么影响吗？听起来你会说稀疏度应该更低，但它对参数规模还有什么其他影响？

Sorry, can you spell out exactly what the implication of data efficiency would be? So it sounds like you'd say there should be less sparsity, but what are the other implications on parameter scaling?

我想只是说，在缩放定律里你不一定追求最高的——你不是在优化算力效率。你在架构上有一堆可选的设计，每一种都给出不同的缩放定律，传统上你会看某种基于算力的包络线。也就是看性能对算力的关系，取最好模型的包络。但如果我们基于数据来做这个决策，也就是假设我们可以花很多算力，把数据放在 x 轴上而不是算力，那我们就会得到一组不同的最优点，或者前沿上一组不同的模型。

I guess just that in the scaling law you're not necessarily looking for the most — you're not trying to optimize compute efficiency. So you have all your choices you can make on the architecture, and each of these gives you a different scaling law, and then traditionally you would look at some kind of envelope based on compute. So you would look at performance versus compute and take the envelope of the best models. But if we're making that decision based on data, so it's like we're sort of assuming we can spend a lot of compute, and we're sort of putting data on our x-axis instead of compute, then we just get a different set of optima or a different set of models that are on that frontier.

对。而且我也不认为过去几年我们每年都把模型规模翻了一倍。人们训练万亿参数模型至少已经有好几年了，甚至还有一个开源的叫 Falcon。但 Periodic Labs 的 Liam 昨天在 Twitter 上发帖说，OpenAI 早期有个实验训练了一个非常非常稀疏的万亿参数模型。

Yeah. And I also don't think that we've necessarily doubled the size of the models every year for the last few years. People have been training one-trillion-parameter models for at least a few years. There was even an open source one called Falcon. But Liam from Periodic Labs posted yesterday on Twitter about how an early experiment at OpenAI was training a one-trillion-parameter model that was very, very sparse.

哦对。那是 OpenAI 之前做的——那是在 Google，那个 switch。对。所以它在知识上非常非常强，但推理很糟糕，因为太稀疏了。所以感觉我们至少在 1000 亿到 2 万亿参数这个区间里玩了一阵子了，而且肯定不是那种漂亮的线性增长。

Oh yeah. That was what they did before OpenAI — that was like at Google, the switch. Yeah. So it was very, very good at knowledge but terrible at reasoning because it was so sparse. And so it feels like we've been playing in this 100 billion up to two trillion parameter range for at least a little bit, and it certainly hasn't been this nice linear increase.

对，我觉得有两件事。就像 Charlie 说的，推理效率对强化学习的 rollout 超级重要，所以这会大幅压低活跃参数量。然后我觉得总参数量很大程度上也取决于硬件。你确实需要非常高的内存带宽和显存容量，才能真正服务多参数万亿级的模型。现在大家还在用很多 H100 之类的东西，所以随着大家转向 GB 系列、再到 Vera Rubin，我们会有更强的能力去扩展、去实际服务，并在更大规模上做大型强化学习输入。

Yeah, I mean I feel like there's two things. As Charlie was saying, inference efficiency is super important for RL rollouts, and so this will really push down active parameters quite a lot. And then I think the total parameters really depends a lot on the hardware as well. You really need to get very high memory bandwidth and VRAM size to actually be able to serve multi-trillion parameter models. Right now people are still using a lot of H100s and stuff, and so as everyone moves to GBs and then Vera Rubins, we'll get more of the ability to scale and actually serve and do large RL inputs at larger scales.

数据瓶颈我觉得很有意思，因为直觉上更大的模型在实际数据点上的样本效率高得多，所以即使你没有把模型训练到饱和，做大仍然更好，因为更大的模型泛化更好，在同样的数据量下能达到更低的损失。所以现在我觉得我们数据还挺多，瓶颈不是数据而是算力，于是我们做小模型，推理效率很高。但如果算力不再是瓶颈，可能又会回到更大的模型，它们有点欠饱和，但因为更大而有这种泛化能力。

The data crunch I think is interesting because naively larger models are much more sample efficient in terms of actual data points, and so even if you're not saturating the model, it's still better to go bigger because larger models generalize better and get to a better loss for the same amount of data. And so right now I think we kind of have a lot of data and that's not the constraint rather than compute, and so we're having small models which are very inference efficient. But if compute's no longer the bottleneck, it might come back to larger models which are sort of undersaturated but have this generalization ability because they're larger.

如果你只看基本的 Chinchilla 缩放定律，把参数最大化——

If you just look at the basic Chinchilla scaling law and you just maximize out parameters —

对。

Yeah.

——它其实只会让你达到同样损失所需的数据量减少一点点。

— it actually decreases the amount of data you need to get to the same loss very little.

对。如果参数趋于无穷，我觉得所需数据量下降不到 10 倍，就是因为幂律的性质。

Yes. If you go to infinity on parameters, the amount of data you need, I think, goes down less than 10x, just because of the nature of the power law.

我们现在处在 Chinchilla 定律里数据太多的那一侧，对吧？现在我们是按 Chinchilla 过度训练模型，所以很容易回到一个点：随着数据耗尽，我们回到 Chinchilla 最优点，甚至稍微偏向过度训练——也就是模型训练不足的那一侧。但肯定的是，即使有这些新芯片上线，未来几年我们还是会严重受限于算力，那就不一定——

We're now on the way-too-much-data side of the Chinchilla laws, right? So right now we overtrain models according to Chinchilla, and so we could easily go back to a point where, as we're running out of data, we move back to the Chinchilla optimal point or even a bit on the overtraining — you know, undertraining model side. But surely even with these new chips that come online and stuff, we're just going to be so compute bottlenecked for the next few years that that won't necessarily—

这很可能——对，这真的取决于训练算力和推理算力的比例。如果你严重受限于数据而不是算力，就该做大。如果你严重受限于算力，就总该做小。但你也可以用算力生成合成数据。所以这是很难预测的事情之一。

I mean, this could well — yeah, this depends on the ratio you have with training and inference compute really. If you're super bottlenecked on data, not on compute, you should go bigger. If you're super bottlenecked on compute, you should always go smaller. But you can also use compute to generate synthetic data. So it's one of these very hard things to predict.

对。我觉得人们一开始花了那么久才搞清缩放定律，部分原因是如果你没把这些都做对，就得不到这么干净的关系。图上那些漂亮的直线掩盖了很多复杂性：你得确保每个超参数都按正确方式缩放，或者把优化器参数化得能扩展，并且改变模型规模时不用改超参数。

Yeah. I think part of the reason it took people so long to figure out the scaling laws in the first place was that if you don't get all these things right, then you don't get such a clean relationship. And the beautiful straight lines on graphs hide a lot of complexity in how you have to make sure to scale every hyperparameter the right way, or parameterize your optimizer in a way that scales and where you don't have to change your hyperparameters as you change the model size.

而且 bug 也有它们自己干净的缩放定律，对吧？比如 Kaplan 忘了余弦那件事，甚至只是没考虑嵌入参数——我觉得那搞乱了小模型上的估计，因为嵌入参数在模型里占比不小。

And bugs have their own clean scaling laws as well, right? Like with Kaplan forgetting the cosine thing, or even just not considering embedding parameters — I think that messed up the estimate at smaller models because embedding parameters are a decent size of the model.

聊一点强化学习。我感觉一年前很多人都在说，强化学习在模型扩展上不会特别成功。John，我记得你写过一篇研究论文，指出模型在强化学习时每个回合只学一个比特——基本上就是学“我答对了还是答错了”。我今年早些时候写过一篇博客说，情况比那还糟，因为当通过率很低、模型几乎不可能答对时，它从一个强化学习回合里几乎什么都学不到。但我看今天的模型，它们似乎挺聪明的，而且似乎是扩大强化学习规模的结果。Beren，你几周前发过一个帖子试图解释这是怎么回事。但为什么强化学习比人们直觉上以为的更成功？

A bit on RL. I feel like a year ago a lot of people were making this argument that RL will not be super successful at scaling for models. I think, John, you wrote a research paper where you were pointing out that models learn one bit per episode when you RL — basically learn did I get the answer right or did I get it wrong. And I wrote some blog post earlier this year saying it's even worse than that, because when the pass rate is low, when the model is very unlikely to get the answer right, it learns almost nothing at all from an RL episode. But I look at the models today and they seem pretty smart, and it seems to be the result of scaling up RL. Beren, you had a post a few weeks ago where you were trying to explain what's going on. But why has RL been more successful than one would have naively thought?

我觉得强化学习的成功归结于好几个不同的因素。首先，我觉得被稍微低估的其实是中期训练。

I mean, I think the success of RL comes down to a bunch of different things. First, I think what is slightly underestimated is actually the mid-training.

我们看到的大量 RL 成功案例，其实来自非常好的中训练数据——基本上就是我们在做预训练，只不过用的是合成推理数据，以及那些能让模型为 RL 热启动的环境。所以这往往能把模型带到距离最终 RL 检查点大约 80% 的位置。然后 RL 在上面做的，本质上是对策略做大量微调。这也是为什么它不需要你天真以为的那么多比特。它不必从零学会所有这些行为，只需要从这些回合里拿到几个比特，而这些你确实能拿到。然后我在博客里特别指出的另一点是：相比常规预训练，这些比特的信号强度极高，这正是为什么你需要 RL，而不是只对成功的推理轨迹做 SFT。

So an awful lot of what we see as successes of RL actually comes from very, very good mid-training data, which is basically where we're essentially doing pre-training but on synthetic reasoning data and the kind of environments that get the model warm-started for RL. And so this actually takes the model almost 80% of the way to the final RL checkpoint often. And then what RL does on top of that is essentially a lot of tweaking to the policy. And so this is one of the reasons why it doesn't need as many bits as you would naively think. It doesn't have to learn all of these behaviors from scratch. It needs just a few bits from these episodes, which you do get. And then the other thing that I really point out in my blog is that these bits are actually extremely high signal compared to regular pre-training, which is why you need RL at all versus just SFTing on successful reasoning traces.

嗯，有两点。是的，第一，它确实就是关于如何得到正确答案的那些比特。但事情并不完全像你想的那样，因为在 SFT 里你有一条轨迹，对吧？你有一堆数学推理，然后最后是答案。那个比特仍然在。你仍然对答案 token 做 SFT。所以那个比特仍然在。重要的是，目标函数忽略了所有其他比特。在 SFT 里，你得努力去匹配模型产生的那些确切的推理 token。所以你实际上拿到了太多关于你正在训练的这个模型具体如何推理的比特。而在 RL 里你只拿到那一个比特，这意味着那个信号不会被模型所有其他比特的噪声淹没。所以这才是真正在训练中大幅提高信噪比的东西，也是为什么 RL 在步数上效率如此之高。不知道你们对此有什么想法。

Well, there's two things. So yes, one, it's exactly the bits about how to get the answer right. But this is not exactly how you think of it, because in SFT you have a trace, right? You have a bunch of math reasoning and then the answer at the end. The bit is still there. You still SFT on the answer token. So that bit is still there. What's important is that the objective ignores all the other bits. So in SFT you have to try and match the exact reasoning tokens that the model produces. So you're essentially getting too many bits about the exact way this other model you're training on reasons. For RL you only get the one bit, and that means that signal is not drowned out in the noise of all the other bits the model has. And so that's what really dramatically increases the signal-to-noise ratio during training, which is why RL is so dramatically efficient in terms of steps. I don't know if you guys have thoughts on that.

是啊，关于 RL 对模型做了什么、相比中训练或 SFT 之类的，争论一直很多。大家都在说 pass@1 会上升，但 pass@256 会下降，那些非常罕见的正确推理轨迹会被降权，会被来自更容易的推理轨迹的梯度信号压过去。我觉得现在看 RL 的一个简单方式是：如果你有足够大的算力去采样足够大的组大小，使得你得到一堆正确答案的概率超过某个不算微不足道的概率，那它就会被升权。而且就像 Von 说的，基本上中训练和更多预训练——也就是 RL 的起点 pass@1——是随预训练 token 数的对数增长的。

Yeah, there's been so much debate about what RL does to the model versus mid-training or SFT or whatever. And everyone talks about how pass@1 will go up but pass@256 will go down, that very rare correct reasoning traces will be down-weighted and kind of outweighed by gradient signal from easier reasoning traces. And I think the simple way to view RL now is that if you have a large enough amount of compute to sample a large enough group size such that your probability of getting a bunch of correct answers is past some not insignificant probability, then it will be upweighted. And to Von's point, basically mid-training and more pre-training—the pass@1, the starting point for RL—scales in a log number of pre-training tokens.

你可以问一些非常基础的问题。我想那个回答说得通，也许有实证研究表明事情就是这样。但然后我就看看模型本身，我不知道发生了什么。也许你可以给我讲讲过去一年 AI 进步的根基是什么。但如果它——也许它只是给那些本来就会做正确思考的策略升权——但看起来在质上模型变得强太多了。也许那里并没有内在矛盾，但我们怎么把这个观点所暗示的 RL 影响相对较小，与模型似乎正在获得的实际质性能力对上呢？

You can ask some very basic questions. I guess that answer makes sense, and maybe there's empirical research which shows that this is what's happening. But then I just look at the models themselves and I don't know what's happened. Maybe you can give me a sense of what is the basis of the AI progress over the last year. But if it's—maybe it's just upweighting the policies which were going to do the correct thinking anyways—but it just seems like qualitatively the models have gotten so much more capable. And maybe there's no inherent contradiction there, but how do we square the relatively small impact this take would imply that RL would have with the actual qualitative capabilities the models seem to be gaining?

这里我想指出的一点是，它并不必然意味着 RL 的影响小，对吧？即使你只有几个比特，只对参数做了很小的改动，对模型函数空间——也就是输入输出映射——的实际影响仍然可能非常巨大。哪怕一个比特也能大幅改变你的函数空间，它能排除掉一半的假设空间，这非常大。所以我不认为从很好的起点出发、少量比特、少量 RL，就必然意味着行为上没有巨大影响——至少不一定。

So one thing I want to point out here is that it doesn't necessarily imply that RL has a small effect, right? Even if you have a few bits and you only change the parameters a small amount, the actual impact on the function space of the model—the input-output mapping—can still be super dramatic. Even one bit can change your function space a lot, and it can rule out half the hypothesis space, which is huge. So I don't think it's necessarily the case that small amounts of bits, small amounts of RL once you're starting from a really good point, means you don't have dramatic impacts in behavior—at least not necessarily.

我觉得这归结为两点。因为我认为第一点是，大家都希望 RL 能把这种推理泛化到所有这些不同领域，而我不认为我们必然得到了这种横向泛化——只在数学上训练并不一定让你成为最厉害的编程者，你确实得在代码环境上做 RL。不过我认为我们确实得到的是视野泛化。模型就是学会了如何用更多 token 工作更久，并且仍然在某种任务上取得进展。所以你可以让它们在越来越长的环境上训练，然后把它们放进一个全新的环境，是的，它们可能没有泛化出能让它们在那个环境里表现好的推理模式，但它们至少泛化了继续做那个任务更久的能力，而这与成功相关。我记得有一篇叫 EdgeBench 的论文，它表明模型能工作更久的能力大约每 3 个月翻一倍。所以这是泛化的明确证据。我觉得最后一种理解方式是：预训练里有量子（quanta）这个概念。你有一条非常平滑的预训练损失曲线。而当你真正看模型内部发生了什么，模型在学所有这些非常离散的任务，有所有这些涌现点，那里有点像相变——它之前没有归纳头，现在有了归纳头——这样的东西有成千上万、数百万，可能上亿个，你把它们平均在一起，就得到这条非常平滑的损失曲线。我觉得在某种程度上，RL 也在发生类似的事。正如 Beren 提到的，有一个非常缓慢的循环：我们训练一个模型，然后对它做 RL，然后在下一个模型迭代的训练中，我们会把一堆这样的合成推理轨迹倒进中训练数据里。我们有点像在为所有这些不同任务命中所有这些量子，而在单个任务层面，它可能看起来像相变，比如你在某个特定的金融任务或 Excel 任务上突然从 5% 的通过率跳到 90% 的通过率。

I think it comes down to two things. Because I think the first thing is that everyone was hoping that RL would generalize this reasoning across all these different domains, and I don't think we necessarily got this horizontal generalization—just training on math doesn't necessarily make you the greatest coder, you do have to do RL on code environments. I think what we did get though is horizon generalization. The models just learned how to use more tokens for longer and still make progress on some sort of task. And so you can train on environments where they get longer and longer and longer, and then put them into a completely new environment, and yes, they may not have generalized the reasoning patterns which allow them to do well in that environment, but they've at least generalized the ability to continue on that task for longer, which is correlated with success. I think there's a paper called EdgeBench which showed that the rate at which models can work for longer is doubling every 3 months. And so that's clear evidence of generalization. And I think the final way to think about it is: in pre-training there's this idea of quanta. So you have this very smooth pre-training loss curve. And when you actually look at what's happening in the model, the model is learning all these very discrete tasks, and there are all these emergent points where there's kind of a phase transition—it didn't have induction heads, now it has induction heads—and there are tens of thousands, millions, probably hundreds of millions of these things, and you average them all together and you get this very smooth loss curve. I think to an extent a similar thing is happening for RL. There is this very slow ad loop, as Beren mentioned, of we will train a model and then RL it, and then the next model iteration of training we will dump a bunch of these synthetic reasoning traces into the mid-training data. We're kind of hitting all these quanta for all these different tasks, and on an individual task level it may look like a phase transition, like you're suddenly going from a 5% pass rate to a 90% pass rate on a particular finance task or Excel task.

随便问吧，但你把所有这些平均起来，再加上横向泛化，你会觉得，哇，我们有了质的提升的模型。

Ask or whatever, but you average all these things together and plus the horizon generalization, you kind of will go, wow, we've got like qualitatively better models.

是的。我觉得很多也在于，我认为强化学习确实有一点泛化，比如突然之间数学和代码之间、谜题和数学之间会有一些迁移。另外，人们针对的环境数量也大大增加了。所以你知道，以前你尝试做日常生活中的某个任务，两年前实验室根本不会关心这个，他们不会为此训练模型，而现在范围广多了。他们有很多针对这个特定事物的环境。

Yeah. I mean, I think a lot of this as well is just like I think RL does generalize a bit like suddenly you get like some transfer between like math and code or like puzzles and math and this kind of stuff. Also just like the amount the sheer amount of environments I think the people are targeting is just like vastly greater. So like you know before when you try to do you know some task which like you do in your daily life like two years ago like the labs wouldn't really care about this. they wouldn't like train the model for it and now like it's just so much broader. They have a lot of environments targeting this specific thing.

在对话早期，我们讨论的都是导致这种熵坍缩的语境，或者只是把概率集中在基础模型已经完成的解决方案上，以及导致策略中相对稀疏的更新。但是当我想到强化学习时，我认为还有另一个故事，那就是回到 Atari 游戏，然后 AlphaGo 下出第 37 手，那个超级有创造性的 move，因为它从未在人类数据上初始化，所以它可以以人类甚至没有想到的方式思考，并提出极具创造性的解决方案。

Early in the conversation we're talking about are all in the context of causing this entropy collapse or just you know concentrating probability on solutions the base model were already done. Um and causing relatively sparse updates in the policy. Um, but when I think like when I think about I think the there's also another story about RL which is going back to the Atari games and then AlphaGo coming up with move 37, the super creative move that because it was never initialized on human data, it can like think in ways that humans are not even thinking and come up with extremely creative solutions.

是的。你觉得我们什么时候能期待，或者是否应该期待大语言模型上的强化学习能产生像第 37 手那样的事情，即超越人类创造力的极端创造力，因为就像是从头初始化智能一样。

Yeah. Do you have a sense on when we should expect or if we should expect RL on LLMs to result in things like move 37 just extreme creativity even beyond human creativity because like there's just denovo uh dova initialization of intelligence.

我是说，这里有几件事。首先，我认为 AlphaGo 使用的是 MCTS，这显然比常规的策略梯度做了更多的探索之类的事情。但我也认为，强化学习不一定降低创造力。我的意思是，即使我们……我认为你知道这显然是定性的，但如果我们看看 OpenAI 的事件，这些模型一次能想出多个零日漏洞来突破沙盒，这显然已经有一定程度的第 37 手创造力了，我们只是从语言模型的通用泛化特性中获得了这一点。我不认为强化学习完全破坏了它们的熵。

I mean so a couple of things here like first off I think that the alpha go is using MCTS which obviously does like more exploration and like stuff than regular policy gradients but I kind of also think that like RL doesn't necessarily like reduce the creativity and like I mean even if we I think you know this is obviously qualitative but if we look at like the you know the open air hinge incident like these models were coming up with like multiple zero days at a time to like break out of the sandbox and like this is clearly like some level of like move 37 creativity I think already which we just get from just like the general generalization properties of the LMS like I don't think it's definitely not the case like RL is like totally destroying their like entropy

尤其是在长视野上

especially on long horizons

是的，我的意思是，人们称之为创造力的一件事就是解决困难的搜索问题，所以第 37 手显然是一个例子，或者写一首满足大量不同约束的诗。所以如果训练的话，AI 显然会非常擅长这个。然后，还有另一种方式，模型输出的多样性在强化学习后大大降低，它们会发展出这些习惯，即使模型看起来擅长写作，当你做某种分布分析时，你会发现它们一直在重复使用某些主题，一直使用相同的角色名字。所以实际上，你得到的多样性并不像从人类作者那里得到的那样。你得到的是某种非常好的风格。所以我认为这种多样性肯定被强化学习大大削减了。事实上，既然我们之前谈到了蒸馏，现在正在发生的一件事是，很多人主要从 Claude 蒸馏，所以所有开源模型都像 Claude 一样写作，有相同的习惯。所以这让我有点担心，我们正在出现这种单一文化。

Yeah I mean one thing that people call creativity is just uh solving hard search problems so uh and so that's like uh like move 37's obviously an example of that or like uh writing some kind of poem that satisfies a ton of different constraints um so that's something AI is obviously going to be extremely good at uh if if trained for it. Um then uh there's another way in which the models uh like uh the diversity of their outputs uh is a lot lower after RL and they sort of develop these ticks and like um even though the models seem like they're good at writing when you do some kind of like distributional analysis you find that like they're reusing uh certain themes uh like all the time and they're using the same character names all the time. So there's actually um it's not like you're getting the same kind of diversity that you get when you like from human authors. You're sort of getting one really good uh like style. So I think that um like that kind of diversity has definitely uh been like cut down by RL a lot. And in fact uh now oh yeah since we were talking about distillation earlier that's sort of uh something yeah one thing that's happening is that so many people are distilling uh mostly from claude that like uh like all the open wade models write the same way as claude and use the same like have the same ticks. So this uh seems kind of concerning to me that we're having this uh like this monoculture emerge.

是的。再说一次，我不认为这是强化学习作为一种方法的根本问题。蒸馏也一样。即使蒸馏，你只是在数据上训练。就像你的数据不是超级无聊，这并不意味着训练方法本身有问题。这是数据的问题。我认为很多，例如强化学习的熵坍缩基本上是由于利用相当简单的验证器，当你没有大量多样化的环境时，因为例如写作，我认为写作大概是由某个评委评分的，评委有特定的习惯，模型学会了奖励黑客评委，这就是为什么它会坍缩，但这真的是评委的问题。这不是强化学习本身的问题。

Yeah. Again, I don't think this is like fundamental to RL as like a method though. And same with distillation. Like even with distillation, like you're just training on the data. It's like just because your data is not like super bored, that doesn't mean like the training method itself is somehow wrong. It's like a problem with the data. And I think a lot of for instance like the RL like entropy collapse is basically due to like exploitation of fairly simple like verifiers when you don't have like a huge diversity of environments cuz like for instance like the writing I think the writing is presumably graded by some judge and like the judge has some specific ticks and like the model is learning to award hack the judge and that's why like it collapses but like this is really a problem with the judge. It's not a problem with like RL in general.

好的，超快速预测未来。所以我想对下面几个问题给出时间线。到什么时候我们会有这样的模型，对用户来说感觉像是：你基本上可以雇佣它们作为远程工作者，从事各种白领工作，不仅仅是编程，还有我不知道的视频编辑、法律助理等等。就像是一个真正的远程工作者，具备完整的计算机使用能力，能无缝学习和操作一个月，执行复杂项目，需要与其他人互动等等。就像人类工作者一个月能做的所有事情。

Um okay super rapid fire predictions about the future. So I want timelines on the following couple questions. By when do we have models which you can here's what the it feels like to a user. You basically hire them as a drop in remote worker for all kinds of white collar work not just coding but I don't know video editing um uh law parallegal etc. Like it's like literally an actual remote worker with like full computer use with like literally a month of seamless learning and operation and executing on like complex projects and it required interacting with other people etc etc. It's like everything a human worker could do over a month.

如果你强制要求使用浏览器之类的，而不是公司把信息设置成可以通过程序访问的，那可能得几年。但是

If you like mandated to use like a browser or whatever rather than like these the again the firm setting up the information to be like programmatically accessible like maybe a couple years. But

如果不是基于浏览器的，比如它可以发送 Slack 消息，它可以做所有这些事情。但我还是可能会说大约一年。

if it's not like browser based like it can send Slack messages, it can do all this stuff. But I'd still probably say around a year.

是的。是的。我的意思是，对于完全通用性，也许大概三年，但我认为就像 Charlie 说的，最终会有很多人让他们的组织更容易被 AI 使用。所以在那之前，你就能达到 80% 到 90%。

Yeah. Yeah. I mean, I would say maybe like for the like full generality maybe like three years, but I think to Charlie's point, we will end up with like a lot of people like making their organizations easier for the AIS to use. And so you get like 80 90% of the way there before that.

所以，但一年和三年之间的区别就是字面意义上的

So, but the thing that's the the diff between one year and 3 years there is just literally like

我觉得会有一个长尾的各种杂项事情，人类可以做，但模型需要相当长时间才能做。

like I think there's going to be like a long tail of like miscellaneous stuff which like some human can do which like will take the models like quite a while to do.

是的。你指的是计算机方面的事情还是基本的认知能力？

Yeah. Like you mean are you thinking of sort of computer stuff or like basic cognitive capabilities?

我的意思是，我认为这真的归结为一个问题：我们能多快解决这种在线学习，以及

I mean I think this this really comes down to a question of like how quickly can we solve this kind of like online learning and like

我们能否通过压缩和给自己写文件之类的方式达到 80% 到 90%，这是我最大的不确定性。我真的不知道。

whether we can like get like 80 90% of the way there with like compaction and like writing files to yourself and stuff and like that's my big uncertainty. I really don't know.

还有一个例子，说明我不擅长的事情，比如在工作中我得对某人大吼才能拿到东西，或者真的逼某人完成某事，模型不会那么做。它只会太友善了。

And another example of something that I wouldn't be good at is, you know, if I have to yell at someone to get something at work or really push someone to get something done, the model isn't just going to do that. It's just going to be too nice.

是的。

Yeah.

是的。我想说，人类远程工作者的质量差异很大。所以如果你试图从 Upwork 上雇人来做软件工程项目，质量会有巨大差异。通常很难让他们做好工作，或者关注你得到的全部反馈。我猜在某些情况下会更糟——AI 之前的版本比你现在从现有 AI 得到的还要差。所以我认为这可能会有点复杂，因为也许在某种程度上我们已经有了这个，对于质量不高的工作，但显然在某些更高质量的工作形式上，我们还没有达到人类水平。所以，但我基本上同意 Charlie 和 Baron 的观点，也许一年左右我们会有某种版本，还算可以，它能做好一些事情，有些事情做得不太好，然后从那里开始会不断改进。

Yeah. I'd say there's a wide variation in quality of human remote workers. So if you try to hire someone off of Upwork to do a software engineering project, there's going to be a huge variation. It's often quite hard to get them to do a good job or pay attention to all the feedback you're getting. And I would guess that in some cases it'll be worse—the pre-AI version of this was worse than what you can get now from existing AI. So I think it might end up being a little complicated because maybe to some extent we already have this for not so high quality of work, but then obviously we're not matching human level in certain higher quality forms of work. So, but I basically agree with Charlie and Baron that maybe we'll have some version of this in a year or so that's okay, and it'll be able to do some things really well, some things not so well and things will be improving from there.

就像我们总是根据非常长的尾部来移动目标。我觉得你之前用过这个例子，比如报税之类的。就像今年我 literally 就告诉 Codex 去获取我需要做的所有事情，然后发给会计师，有一大堆东西它必须用电脑点击并下载一些东西，它没有——它很好。它完美。所以我不知道，很多这些事情它已经能做了。

Like we ship the goalpost based on the very long tail all the time. Like I think feel like you've used this example before of doing your taxes or something. Like this year I literally just told Codex to go get everything I needed to do and send it to the accountant and there was this massive list of stuff it had to use computers to click through and download some stuff and it didn't—it was fine. It was perfect. So like I don't know a lot of this stuff it can already do.

是的。好的。给你 10 倍的总生产力提升。

Yeah. Okay. Give you 10x total productivity uplift.

基本上，如果以前你需要一年才能取得突破，现在你每个月都能取得突破。

Basically if it takes you a year to make a breakthrough now you make a breakthrough every month.

我想我会直接拒绝给你一个标度。就像在某些类型的工作中，我们可能已经超过了那个。比如假设你只是试图证明——是的，你试图做某些类型的数学——

I think I would just refuse to give you a scaler on this. Like we might already be past that in some types of work. Like let's say you're just trying to prove—yeah, you're trying to do certain types of math—

抱歉，但对于你们作为 AI 研究人员，试图——是的——推动 AI 研究的进展，AI 研究人员被加速或提升了多少？

I was sorry but for you as AI researchers trying to make—yeah—like advance the state of AI research, how much are AI researchers sped up or uplifted?

大约在 5 到 10 年之间。

Somewhere between five and 10 years.

哦，真的吗？那很远。你认为这比一般远程工作者需要的时间更长？是的，有趣。

Oh really? That's far away. You think it's longer than for general remote worker? Yeah, interesting.

我想你可能——我意识到你可能对完全通用的远程工作者有非常不同的定义。我本可以说明的。

I think you probably—I think I'm realizing you probably have very different definitions of fully general remote worker. I could have specified.

是的，这是真的，因为我的意思是——是的，因为显然 AI 研究人员可以是远程工作者，所以就像——

Yeah, this is true because I mean like—yeah, because obviously an AI researcher can be a remote worker and so like—

是的。不，我想象的是你知道的正常白领工作，在一个月的时间内。

Yeah. No, I'm picturing like you know normal white collar work over the period of a month.

是的，我想超过一个月后开始有点分歧——就像一个非常能干的白领工人，但不一定是超级有创造力的研究员。

Yeah, I think it starts to diverge a little bit past a month—like a very competent white collar worker but not necessarily like a super creative researcher.

我想说大约两年。

I would say like two years.

两年。是的。10 倍。好的。你呢，Beren？

Two years. Yeah. 10x. Okay. How about you, Beren?

我其实能理解，因为现在它已经肯定超过 10 倍的编码工作了，所以如果它能做哪怕一两个实验反馈循环，那实际上已经很大了。

I can kind of see that actually because it really is just like right now it's already definitely more than 10x of coding stuff and so it's like if it can do even one or two loops of experimental feedback that would actually be massive already.

所以 AI 研究人员在两年内提升 10 倍。

So 10x uplift of AI researchers within two years.

如果你把它代入一个非常简单的 AI 进展模型，以及有多少来自 AI 研究人员，并且他们的生产力提高了 10 倍，是的，从两年后开始，AI 进展的速度会急剧加快。

If you just plug it into a very naive model of AI progress and how much is coming from AI researchers and there's a 10x increase in their productivity, yeah, you have radically accelerated pace of AI progress starting 2 years from now.

是的。我的意思是，我认为这意味着 AI 进展不会瓶颈在 AI 研究人员运行小实验的能力上。它会瓶颈在其他事情上。

Yeah. I mean I think this will mean that AI progress doesn't get bottlenecked on AI researchers' ability to run small experiments. It gets bottlenecked on other things.

当然。当然。但它只是发生得快 10 倍。

Of course. Of course. But it just like happens 10x faster.

当然。是的。

For sure. Yeah.

这是一个巨大的事情，这也帮助下一个给你 100 倍加速的事情更早发生，等等。

Which is a huge deal and that also helps the next thing which gives you 100x speed up happen sooner etc.

是的。我愿意在那个上坚持更长的时间。

Yeah. I'm happy to stick with longer on that one.

关键是什么？就像我吸收信息并对下一个实验做出贝叶斯最优决策的能力。

And what's the crux? Like my capacity to absorb information and make the like Bayesian optimal decision on the next experiment.

有道理。

Makes sense.

是的。我的意思是，我假设你可以把其中一些委托给 AI。所以 AI 在决定方面变得不错，你知道，它运行这个实验，得到这个结果，它运行下一个实验，然后如果它能连续运行两三个实验而不崩溃，那实际上就是很大的提升，而且——

Yeah. I mean, I'm assuming that you can delegate some of this to the AI. So the AI is becoming decent at deciding, you know, it's run this experiment, it's got this result, it runs the next experiment and then if it can run two or three experiments in a row without crashing, then that is actually big uplift and—

好的，最后一个问题。一个 AI 在每一个可以通过计算机完成的工作领域都主导顶级人类专家。所以不仅是研究，而是所有认知工作，不仅是短周期的工作，而是真的如果花费三年左右的时间，它们仍然会比人类做得更好。所以这基本上就是 ASI。好的。

And okay, final question. An AI which dominates top human experts across every single field of work that can be done over a computer. So not only research but all cognitive work and not just short horizon work but literally if it takes three years or something they will still do better than humans. So this is basically just ASI. Okay.

我想说大约三到四年。

I would say like three or four years.

那——

The—

我的意思是那似乎没错。

I mean that doesn't seem wrong.

我想说，AI 显然获得了更多关注。所以它是较难的事情之一，但有很多精力投入其中，而且它也不是 AI 最难的事情之一，因为它涉及大量代码和数学，模型非常擅长这些。也许对于涉及 3D 和空间以及物理的东西，我认为那会需要更长一点时间。所以特别是如果它不是——是的，如果像机械工程之类的，现在没有得到最多关注，那可能需要更长一点时间。

I would say like AI is obviously getting more attention. So it's one of the harder things, but a lot of energy is being put into it and it's also not one of the hardest things for AI because it involves a lot of code and math which models are really good at. Maybe for things that involve 3D and spatial stuff and physical stuff, I think that will take a little longer. So especially if it's not—yeah, if it's like mechanical engineering or something and it's not getting the most attention right now, that might take a little longer.

但它还必须包括由于领域性质而数据相对较少的领域，并且它必须即时学习那些数据。例如，必须在台积电成为工程师方面超越人类。

But it also has to include fields where there is relatively little data because of the nature of the field and it has to learn that data on the fly. For example, has to become superhuman at being an engineer at TSMC or something.

哦，是的。所以你必须假设入职培训——是的。你可以给 AI 相同的入职材料，然后——哦，是的。那么必须解决关于更长周期学习的问题，或者——

Oh yeah. So you would have to assume that the onboarding—yeah. You can give the AI the same onboarding material and—oh yeah. Then there's something has to be solved about longer horizon learning or—

我想说 5 到 10 年。

I'd say five to 10.

所以基本上，是的，这是——你认为自动化 AI 研究就像是 ASI 完成之类的。

So basically yeah it's—you think automating AI research is like ASI complete or something.

是的，我认为是这样。

Yeah I think so.

嗯，我觉得世界上有太多事情，即便你有一个模型外部的记忆系统，即便上下文窗口增长了一点，也还是有一些根本性的东西——即便你能自己查资料或做笔记，你也需要超出上下文窗口的东西才能做到。

Yeah, I think there are so many things in the world which, even if you have some sort of memory system external to the model, and even if context length grows a little bit, there are just fundamentally things that, even if you could research the information or write notes yourself, you need more than the context to be able to do.

嗯，我某种程度上同意，至少在五年这个区间内，对于实验室正在聚焦的那些东西是这样。但我觉得会有一个长尾，那些东西我理论上可以去学，但没人费心去做，分配给它的算力也少——所以要覆盖到每一个人类专家，那可能得花更长时间。

Yeah. I mean, I kind of agree in the five-year range, at least for the stuff that labs are focusing on. But I think there's going to be a long tail of stuff which, yeah, I could theoretically go out and learn about, but no one has bothered to do it, and the computers being allocated to that — so that might take longer for literally every single human expert.

对，没错。我在这里也把“能像人一样快速学习一个新领域”的能力算进去了。

Yeah, it's right. By this I also included the ability to learn a new domain as fast as a human.

我是说，我觉得那其实未必是必需的，因为 AI 会拥有远超任何人类的经验，对吧？

I mean, I think that's not necessarily necessary, actually, because the AI will have vastly greater experience than any human, right?

非常感谢各位参与。我觉得这种形式非常好，能让不同的专家互相提出异议、辩论、一起讨论。非常有收获。好。

Thanks so much for doing this, guys. I feel like this is a great format for getting different experts to disagree and debate and discuss things together. It was very productive. Cool.