重写 AI 智能体能否弯折智能曲线？

那么，早在 7 月，一家总部位于伦敦的新 AI 公司在 Twitter 上发布了一条非常火爆的推文。它获得了大约 180 万次浏览，声称展示了递归自我改进的首个证据。问题是，并不是所有人都买账。

So, back in July, a new AI company based in London posted a very viral tweet on Twitter. It got about 1.8 million views, claiming to demonstrate the first evidence of recursive self-improvement. The thing is, not everybody bought it, though.

我是郑耀江，Wo AI 的联合创始人兼 CEO，我们构建自我改进的智能体。

I'm Zhengyao Jiang, co-founder and the CEO of Wo AI where we build self-improving agents.

哦，你确实说了对自动研究智能体进行了 8 天的自动研究，我们在自动研究 harness 上运行自动研究。他们设法发现了一个比我们过去两年不断调优的更好的自动研究 harness。

Oh, you did actually say auto researching the auto research agent for 8 days where we run auto research on auto research harness. They managed to discover a better auto research harness than we keep tuning in the last two years.

现在，当大多数人想到递归自我改进的智能时，他们脑海中浮现的是一个能在循环中重新连接自己大脑的系统。但如果系统只是改变大脑周围的代码呢？那还算数吗？

Now, when most people think about recursively self-improving intelligence, what they conjure up in their mind is a system that can rewire its own brain in a loop. But what if the system was only changing the code around the brain? Would that still count?

我们关心的是结果，比如它是否真正提升了系统的能力，而不是某个特定层。

We care about the outcome like does it actually improve the capability of the system, not a specific layer.

还有古德哈特定律这个小问题，即当一个度量变成目标时，它就不再是一个好的度量。

And also what about this small matter of Goodhart's law which is that when a measure becomes a target it ceases to be a good measure.

更有趣的是，我们发现它的设计机制能防止自身进行奖励黑客行为。

And more interestingly we find it the design mechanisms prevent itself from reward hacking.

人们对 RSI 最大的误解是什么？

What is the biggest misconception people have about RSI?

如果你达到 RSI，就会出现技术奇点或智能爆炸。我不认为情况会是这样。

If you arrive at RSI there's going to be a technical singularity or like intelligence explosion. I don't think that would be the case.

所以有一群分散的人正在创建初创公司，专注于递归改进超级智能。

So there's a diaspora of folks who are building startups that are focusing on recursively improving super intelligence.

这个项目实际上大约一年半前开始，当时我们对递归自我改进的概念产生了兴趣。我的意思是，历史上所有研究都有收益递减。但有一个想法是，如果你能让研究者提高自己的效率呢？历史上通过方法论改进、更好的工具是可能的，但人类研究者，比如人类研究者的大脑，始终是关键瓶颈。如果我们现在有一个自主研究系统，我们能否将研究主题指向它自己，以便它能提高研究效率？希望有一天这种效率的提升能克服收益递减。所以是的，努力与产出的曲线从凹变为凸。当然那是一个宏伟目标，有点像 AGI，没人知道我们能走多远，但提高自指研究的效率肯定已经在路线图上。

This project actually started about one and a half years ago where we got interested in the concept of a recursive self-improvement. I mean historically all of the research have a diminishing returns. There is an idea though what if you can allow the researcher improve its own efficiency. Historically it's possible by methodology improvement better tools but human researchers like a human researchers brain for example is always the key bottleneck. If we now have a autonomous research system, can we point the research topic to itself so that it can improve its research efficiency? And the hope is one day this improvement of efficiency can overcome the diminishing return. So that yeah the curve of effort output turn from concave to convex. Of course that's a like the grand goal a bit like AGI no one know how far we can go but improving the efficiency of self referential research is definitely already on the road map.

那么让我正确地设定一下。好的。所以 Wo AI 他们创建了这个叫做 Aiden 的智能体，它是一个研究 harness，对吧，它非常简单，你给它一个指标，你给它一个问题，它就会朝着那个问题爬山，但他们做的是递归地设置它，所以他们实际上朝着改进智能体本身爬山，这意味着改进 harness，所以你知道提示和工具，以及所有技能表面，harness 能做的所有事情，他们只是让它运行。他们只是让它运行，然后看看发生了什么。

So let me set this up properly. Right. So Wo AI they created this agent called Aiden and it is a research harness right and it's very very simple you give it a metric you give it a problem and it will hill climb towards that problem but what they did was they set it up recursively so they actually like hill climbed towards improving the agent itself and what that means is is improving the harness so you know the prompts and the tooling and you know all of the skill surface all all of the things that that the harness can do and they just left it running. They just left it running and they saw what happened.

我们在自进化的自动研究 harness 上观察到一个有趣的事情。我们看到它生成的代码。没错。就像外星代码意大利面，但不知为何它泛化得非常好。当然我们有一组基准测试，我们试图在上面爬山。之后我们会在留出基准上测试它。那些是公开基准，比如 MLE bench light、A bench light，A bench 是 Sakana 的算法发现基准，bench 是 OBI 的机器学习工程基准，我们还测试了一个非常分布外的任务，叫做 weather bench 2。本质上，你试图构建一个基于物理的预报引擎来预测天气，而 weather bench 与训练或优化集非常分布外，但它仍然泛化得很好。它比我们认为是更优雅的手工调优 harness 泛化得更好。我们对实验结果有点纠结，比如我们想要在多大程度上坚持更第一性原理和干净的解决方案， versus 实际上在实践中效果更好的解决方案。那里肯定有改进空间。我认为比如给优化器添加某些约束，两组约束可能类似于你正则化神经网络。你如何让它尝试找到同一问题的最简单解决方案？这是一个方向。但另一个方向可能是，好吧，你只需要忍受它。因为让人们感觉它是意大利面代码的负担之一可能是智能体在后台做了多少工作，比如在我们的案例中它在自动研究自我改进上做了 100 次实验，那个实验数量可能与我们过去两年做的相似。所以如果你让一个新实习生看我们的代码库，也许他们也会觉得，好吧，这太难理解了。我认为一部分是，是的，你需要承受理解大量实验结果的认知负担。

There's an interesting thing we observe on self-evolved auto research harness. We see the code it generated. Exactly. It's like alien code spaghetti but for some reason it generalizes really well. Of course we have a set of benchmarks we trying to hill climb on. After that we will test it on hold out benchmarks. Those are public benchmarks like MLE bench light, A bench light, um A bench is like algorithmic discovery benchmark from Sakana, bench is the OBI's machine learning engineering benchmark and we also tested on a very far out of distribution task uh called weather bench 2. Essentially, you're trying to build a physics-based forecasting engine uh to predict the weather and uh the weather bench is very far out of distribution from the uh training or optimization set of but it still generalize really well. It generalize better than our we would argue more elegant uh hand-tuned harness. We uh a bit tone uh on the experiment results like to what extent we want to stick to more first principled um and the clean solutions we generated versus uh the solution that actually works better in practice. There's definitely space that we can improve over there. I think um like adding certain constraints uh to the optimizer two set of constraints might be something like you regularize neural networks. How do you make it trying to find the most simple solution to the same problem? That's one direction. But another direction might be okay, you just need to live with it. uh because one of the burden that make people feeling it's a spec spaghetti code might be how much work the agent is done in the background like it did do 100 experiments in our case on auto research self-improvement and that number of experiments is probably similar amount that we did in the last two years. So if you onboard a new intern to look at our codebase maybe they also feel like okay this is so difficult to understand. I think a part of it is u yeah you need to bear with that cognition recognition burden of understanding huge amount of uh experiment results.

我们甚至需要那些约束吗？

Do we even need those constraints at all?

这是一个非常好的问题。我认为在实践中你仍然需要那些约束。人类和 AI 需要合作。我相信人类智能的某些方面仍然优于 AI。例如，在 Aiden 中，我们让智能体参加 OBI 的招聘挑战，我们发现人类在生成创造性原语方面仍然好得多。我们发现人类构建第一个原型非常重要，它给智能体正确的搜索空间。基本上是初始抽象，锚定搜索空间，有点像你设计神经网络架构时引入归纳偏置用于学习过程。如果初始代码库基于像 Aiden 这样的搜索脚手架，那么智能体提出的很多想法实际上来自搜索文献。但如果你给初始代码库作为反应式智能体，反应式智能体基本上意味着你把所有上下文放入历史并给智能体完全灵活性。然后很多想法来自智能体，外层智能体。外层意味着智能体优化另一个智能体。

That's a really good question. I think in practice you still need those constraints. Human and AI need to collaborate. Uh there are certain aspects of a human intelligence are still superior uh than AI. I believe so. For example, in Aiden where we send the agent to attend OBI's uh hiring challenge, we found human is still much better at generating creative primitives. We found it's very important for human to build the first prototype that gives the agent the right search space. Basically the initial abstraction where anchor the search space a bit like how you design a neural network architecture where you introduce inductive biases for the learning process. If the initial code base is based on uh a search based scaffold like Aiden then a lot of ideas agent came up are actually coming from search literature. But uh if you give the initial codebase as a react agent react agent basically means you put all the contacts uh into the history and give the agent full flexibility. Then a lot of the ideas come from the agent, the outerloop agent. Outerloop meaning the agent optimizing uh another agent.

那么递归自我改进是什么意思？对吧？我们是在谈论建造一个机器神吗？我们只是在谈论一些在循环中运行并优化自己的代码吗？我们试图确定这一点。

So what do we mean by recursive self-improvement? Right? Are we talking about building a machine god? Are we just talking about some code that runs in a loop and optimizes itself? We tried to pin this down.

现在你谈到了 RSI 的四个级别。所以递归改进智能。这四个级别是什么？

Now you have spoken about four levels of RSI. So recursively improving intelligence. What are the four levels?

是的，第一个级别是我们所说的委托。

Yeah, the first level is what we call the delegation.

你可以运行递归自我改进循环，但它并不严格优于人类做研发。我们认为之前所有公开的结果都停留在这一层，第零层。第一层是我们所说的净正向，即你运行自我改进循环，发现自我改进的速度比人类做研发来改进系统更快。这就是我们目前所在的位置。在第一层，当我们说改进时，总是有办法衡量——比如我们在一组下游任务上衡量——但我们没有衡量它在改进自身改进能力方面有多好。外层循环找到的内层循环是否真的能成为更好的外层循环？这就是我们所说的第二层递归自我改进。我们称之为点火。

You can run a recursive self-improvement loop, but it's not strictly better than a human doing R&D. We believe all the previous public results are sitting on this level, level zero. Level one is what we call net positive, where you run the self-improvement loop and you find that the speed of self-improvement is faster than a human doing R&D to improve the system. That's where we are currently sitting at. In level one, when we say improvement, there's always a way to measure it—like we measure it on a collection of downstream tasks—but we didn't measure how good it is at improving its capability of improving itself. Does the inner loop found by the outer loop actually become a better outer loop? That's what we call level two recursive self-improvement. We call it ignition.

为什么这种从内层循环到外层循环的泛化很重要？因为那是递归自我改进宏大目标的必要条件，本质上是智能爆炸场景。提高效率可以克服难度的增加，因为在第一层你仍然会看到收益递减——从内层循环到外层循环的正反馈循环还没有开始运行。

Why is this generalization from inner loop to outer loop important? Because that's the necessary condition for the grand goal of recursive self-improvement, essentially the intelligence explosion scenario. Improving efficiency can overcome the increase in difficulty because on level one you will still see diminishing returns—the positive feedback loop from inner loop to outer loop hasn't started running.

所以在这个设置中，底层的模型没有改变，因为适应非常重要。但我们是指到处适应还是关键路径适应？你只需要有一个递归循环来适应系统的某些部分，还是智能的所在——模型本身——需要适应，我们才能称之为递归改进智能？我们也不太关心这个，因为我们关心结果：它是否真的提高了系统的能力，而不是某个特定层。当然，有人可能会说在不同层上还有更多工作要做，公平地说，我认为最终的第三层递归自我改进——当我们真正到达拐点时——我认为需要在所有不同层上都有这种自指循环。但对我们来说，我们关心的是工具层，同时我们不认为只有模型层的改进才算 ISI。

So in this setup, the model underneath didn't change because adaptation is very, very important. But do we mean adaptation everywhere or do we mean critical path adaptation? Do you only need to have a recursive loop that adapts some part of the system, or does the locus of intelligence—the model itself—need to adapt for us to designate this as recursive improving intelligence? We also don't care much about it because we care about the outcome: does it actually improve the capability of the system, not a specific layer. Of course, there could be an argument that there's more to be done on different layers, and to be fair, I think eventual level three recursive self-improvement—when we actually reach the inflection point—I think there need to be this self-referential loop on all different layers. But for us, we care about the harness layer, and at the same time we don't think only model layer improvements count as ISI.

你能解释一下你找到的这个最蒸馏的智能体的行为吗？

Can you just explain what was the behavior of this most kind of distilled agent that you found?

是的，最好的智能体是 a85，我们这么叫它。有很多变化，但归结起来，有搜索算法的改进，有糟糕的上下文管理系统，还有提示的完全重写。更有趣的是，我们发现它设计了机制来防止自己奖励黑客——就像外层循环试图防止内层循环在基准测试中作弊——它实际上设计了一个三层系统来对抗，包括提示级别的反作弊。这就像一个术语——基本上就是“不要作弊”——还有一组硬编码规则，基本上检查内层循环智能体生成的代码中是否有作弊证据。更有趣的是，它还试图从某些统计属性中过滤掉那些作弊解决方案。例如，如果一个解决方案远远超出其同行的平均值，那么它会认为，好吧，这可能发生了一些奖励黑客。但有趣的是，首先这个机制进化了——就像它在过程的早期阶段被设计出来——而它在后期实际上被破坏了。我认为后期的代码更改只是引入了一个 bug，所以那一层停止工作。这也是我们刚才提到的意大利面条式代码库的特性。

Yeah, the best agent is a85, we call it. There are a lot of changes, but boiled down, there are search algorithm improvements, there are bad context management systems, and there is a complete rewrite of the prompts. More interestingly, we find it designed mechanisms to prevent itself from reward hacking—like the outer loop trying to prevent the inner loop from cheating the benchmark—and it actually designed a three-layer system against that, which includes a prompt-level anti-cheating. This is like just a term—okay, don't cheat, basically—and there are a set of hard-coded rules that basically check if there's evidence of cheating from the code the inner loop agent generated. The more interesting one is it also tries to filter out those cheating solutions from some statistical property. For example, if one solution is way too far out of the average of its peers, then it will think, okay, this might have some reward hacking happening. But what's interesting is first that mechanism evolved—like it was designed in early stages of the process—and it's actually broken in the later stage. I think later stage code changes just introduced a bug, so that layer stops working. That's also sort of the spaghetti codebase property we just mentioned.

给在家的朋友们解释一下，奖励黑客的经典例子是 Coast Runners 那个，船在绕圈做完全退化的事情。你在博客文章中说内核优化就是一个巨大的例子。

Just for folks at home, you know, reward hacking, the canonical example is the Coast Runners thing where the boat is going around in circles doing something completely degenerate. And you said in your blog post that kernel optimization is a huge example of this.

所以当你尝试以智能体方式优化内核时，就是老套的捷径学习、古德哈特定律，随你怎么叫，它会做完全退化的事情。

So when you try and agentically optimize kernels, just good old shortcut learning, Goodhart's law, call it what you want, it will do something completely degenerate.

我认为更有趣的部分是内层循环和外层循环都在朝着相似但不同的目标优化。所以在我们用于爬山的所有基准测试中，我们有一个公开集和一个私有集。内层循环智能体只会看公开集，而外层循环会看来自私有集的聚合分数。所以这两个智能体在某种程度上不同的目标层次上优化。所以每当内层循环智能体试图作弊时，我们的评估协议会显示，好吧，虽然你在公开集上取得了好表现，但你的私有集表现实际上在下降。所以外层循环设法了解了这一点，并试图减少公开集和私有集之间的差距。

I think even more interesting part of it is the inner loop and outer loop are both optimizing towards similar but different goals. So in all of our benchmarks used for hill climbing, we have a public set and a private set. The inner loop agent will only look at the public set, whereas the outer loop will look at the aggregated score from the private set. So those two agents are optimizing at kind of different levels of objective. So whenever the inner loop agent tries to cheat, our evaluation protocol will show, okay, although you achieve good performance on the public set, your private set performance is actually going down. So the outer loop managed to learn about it and tried to reduce the gap between public and private.

这有点像作弊吗？外层智能体可以访问私有数据，然后大多数时候它实际上相当正则化，并没有公然作弊，但原则上它可以作弊。

Isn't it a bit like it's cheating that the outer agent has access to the private data and then most of the time it's actually quite regularized and it's not blatantly cheating but it could in principle cheat.

是的，这就是为什么我们在这里实际上测试了两个阶的泛化。有一阶泛化，即从同一基准的公开集到私有集的泛化。但为了评估整个元学习系统，我们需要在一组留出的基准上测试它。所以那些基准是连外层循环都从未见过的任务，对于所有这些基准，仍然有私有和公开的划分。所以我们在那些留出任务上测试了发现的内层循环智能体。这就是协议。我们测试外层循环有多好。

Yeah, that's why there's actually two orders of generalization we are testing here. There's a first-order generalization, which is a generalization from public set to private set of the same benchmark. But in order to evaluate the whole meta-learning system, we need to test it on a hold-out set of benchmarks. So those benchmarks are tasks that even the outer loop never sees, and for all of those benchmarks there are still private and public splits. So we tested the discovered inner loop agent on those hold-out tasks. That's the protocol. We test how good the outer loop is.

他们的实验只使用了一个外层循环。但为什么只有一个？

Their experiment only used one outer loop. But why only one?

问题是，外层循环越多，成本就越高。假设在内层循环中你对数百个任务运行自动研究，这意味着外层循环的每一步你都要对所有这些数百个任务运行自动研究，那是一个巨大的成本增加。如果发现的内层循环智能体或发现的工具可以应用回外层循环，那么你就不需要添加另一层——就像基本上每隔几步你就把内层循环智能体提升到外层循环——是的，就是这样。你可以运行无限步，应用无限次迭代，而不添加另一层。

The thing is, the more outer loops you have, it will become more expensive. Let's say in the inner loop you run auto research on hundreds of tasks, and that means for every step of the outer loop you are going to run auto research on all those hundreds of tasks, and that's like a huge cost bump up. If the inner loop agent being discovered or harness being discovered can be applied back to the outer loop, then you don't need to add another layer—like essentially every few steps you promote the inner loop agent to the outer loop—and yeah, that's it. You can run this for an infinite number of steps, apply an infinite number of iterations without adding another layer.

如果我理解正确的话，目前这些片段是短暂的。所以它们彼此隔离。如果不是这样会发生什么？比如你会——你实验过吗？比如它们知道之前发生了什么，或者它们可以有共享记忆之类的。

And if I understand correctly, at the moment these episodes are ephemeral. So they're isolated from each other. What would happen if that wasn't the case? Like what would—did you experiment with that? Like they had knowledge of what's happened before or they could have a shared memory or something.

我们还没有实验过，但这是一个非常有前途的方向，我们正在探索。我认为有一些更基本的问题表述问题我们必须解决。目前我们将问题框定为无状态优化问题。这到底是不是正确的框架？可能不是。

We haven't experimented with that, but it's a really promising direction we are exploring. I think there are some more fundamental problem formulation problems we have to address. Currently we frame the problem as a stateless optimization problem. Is that the right framing at all? Probably not.

你有没有发现缺乏上下文曾经是个问题？那么，你有没有看到退化现象，比如它陷入循环或反复尝试同一件事？

Did you find that the lack of context was ever a problem? So, did you see degeneracy in terms of it was going in loops or trying the same thing again?

是的，我们确实看到了。有时智能体会一直尝试所有搜索算法的想法，尽管在某种程度上，如果你尝试了很多次，你可能会吸取教训，这个方向总体上行不通，但它仍然在尝试。

Yeah, we do see that. Sometimes the agent just keeps trying all the search algorithm ideas even though, to some extent, if you try it so many times you probably learn a lesson that this direction is not going to work in general, but it's still trying.

我认为你最好的智能体是 30% 探索、70% 利用，如果这也能根据上下文自适应，那岂不是很好？

I think your best agent was it 30% exploration, 70% exploitation and wouldn't it be great if that was also adaptive based on the context?

是的，最好的智能体搜索策略非常复杂。它实际上结合了多臂老虎机和一种奇怪的抗饱和策略，就像，哦，如果我——好吧，首先它尝试用几个谱系（或者你可以称它们为岛屿）来组织搜索过程，并且它会用多臂老虎机算法在这些谱系之间分配预算。它还对那个多臂老虎机算法做了一些定制。每当一个谱系饱和时，它就会创建一个新的谱系，就像一个新的岛屿，拥有全新的上下文，但仍然延续旧谱系的一些想法。所以，这是一个非常复杂的策略，但似乎有效。

Yeah, the best agent search policy is really complex. And it's actually combining multi-armed bandit and a weird strategy of anti-saturation, which is like, oh, if I—okay, first it tries to organize the search process with a few lineages or you can call them islands, and it will allocate budgets between those lineages with multi-armed bandit algorithms. And it also did a bit of customization on that multi-armed bandit algorithm. And whenever one lineage gets saturated, it will create a new lineage like a new island with a fresh context but still extending some of the ideas from the old lineage. So, it's a really complex strategy, but it seems like it works.

将这与其它已经出现的东西进行对比很有趣。所以，我的意思是，例如 Alpha V，你在博客文章中说你会将其归类为递归改进的第一级。你能解释为什么吗？

It's interesting to contrast this to other things that have been out there as well. So, I mean, Alpha V, for example, you said in your blog post that you would actually classify that as level one of recursive improvement. Can you explain why?

当然，存在灰色地带。我们认为这是递归自我改进的一步，因为你找到了一个更好的算法，比如在那种情况下，我相信它像是矩阵乘法算法，在某种程度上你可以争辩它设法改进了自己，就像如果你将其应用回实际上进行语言模型推理的 AI 系统，你本质上使其更快。所以我们认为你可以争辩说这是递归自我改进的一步，尽管我们不知道，因为我认为他们实际上没有测试它。我们有一个固定的智能体，我们正在优化一个特定任务，而你们实际上是在优化智能体，该智能体优化任务，并且不止一个任务。

Of course, there are gray areas. We think it is a one step of a recursive self-improvement because you find a better algorithm like in that case I believe it's like a matrix multiplication algorithm to some extent you can argue it managed to improve itself like if you apply that back to the AI system that are actually doing the like inference of language models like you essentially make it faster. So we think you can argue that's like a one step of recursive self-improvement although we don't know because I don't think they actually test it. We have a fixed agent and we're optimizing a particular task whereas you're actually optimizing the agent which optimizes the tasks and more than one task.

是的。是的。我们正在尝试在端到端测试中优化整个系统。在此之前，有很多关于元工具优化的工作。例如，你可以优化系统的特定组件。例如，我不确定你是否熟悉达尔文·戈德机器。

Yeah. Yeah. We are trying to optimize the whole system on end-to-end testing. Before this there are a lot of works on meta harness optimization. For example, you can optimize a specific component of your system. For example, I'm not sure if you are familiar with Darwin God Machine.

哦，我正要问你那个。是的，告诉我吧。

Oh, I was going to ask you about that. Yeah, tell me about that.

是的。所以他们试图做的是，他们有一个智能体优化一个编码智能体。所以有一个元智能体在通用编码任务上优化一个编码智能体，而那个编码智能体是进行优化的智能体的一部分。在他们的情况下，只有两个层级，就像一个智能体优化另一个做编码的智能体，而在我们的情况下，有三个层级：有一个外循环智能体，有一个内循环智能体，两者都在做自动研究，然后还有另一层是下游任务，其中一些实际上是工具工程任务。我认为达尔文·戈德机器这条线有趣的部分是，较低层——任务层——实际上有一个搜索算法的组件，他们说好吧，如果我在单元测试中改进这个组件，性能实际上可以泛化回整个搜索算法。对我们来说，我们的问题是，好吧，我们在这个范式内能走多远，就像当我们开始这个项目时，我们总是想，好吧，我们应该优化整个自动研究系统的端到端性能，并且我们有我们的自动循环智能体指向那个目标。

Yeah. So what they are trying to do is they have an agent optimizing a coding agent. So there's a meta agent optimizing a coding agent on general coding tasks and that coding agent is part of the agent doing the optimization. In their case there's only two levels like there's an agent optimizing another agent doing coding whereas in our case there are three levels: there's an outer-loop agent, there's an inner-loop agent both are doing auto research and then there's another layer which is the downstream tasks and some of them are actually harness engineering tasks. And I think the interesting part of the thread of Darwin God Machine is the lower layer the task layer actually have a component of the search algorithm and they say okay if I improve this component in the unit test the performance can actually generalize back to the whole search algorithm. For us, our question is okay if like how far we can go within that paradigm like when we start this project we always thinking okay we should optimize the end-to-end performance of the whole auto research system and we have our auto loop agent pointing to that goal.

所以记得 WCO 的这条推文，它做得非常非常好，有 180 万次浏览，也有一些反对意见。例如，杰夫·克伦是这个领域的传奇人物，他说，好吧，这怎么是递归自我改进的第一个证据，那所有其他工作呢？他引用了达尔文·戈德机器和超级智能体以及他们自己在递归方面关于自动化 AI 研究第一步的工作，还有很多其他工作。所以有一些反对意见。

So remember this tweet from WCO it did really really well it did 1.8 million views and there was some push back. So for example Jeff Clune who is a legend in this field you know he said well how is this the first evidence of recursive self-improvement what about all of this other work like he quoted Darwin God Machines and hyper agents and their own work at recursive on the first steps towards automated AI research and lots of other work. So there was a bit of push back.

我的意思是，首先我们稍后会有一篇正式的论文，至少是第一个 PDF 版本，我们将在其中对整个元工具优化这条线给予学术界认可。对我们来说，我们关心的是系统的结果，而不是概念上的差异，尽管我刚刚解释了一些概念上的差异，我们关心的是它实际上弯曲了递归自我改进曲线的结果，就像 IND 努力，我们看到这是第一个证据，表明对于能够自我一致地改进自己的完全自主系统，曲线实际上开始弯曲。杰夫更多是从概念层面看待它。就像有一些自指循环。当然，我们并不声称我们发明了这种元优化想法，这可以追溯到 20-30 年前，但我们看到了意大利面搜索的好处，因为那里有大量的金子可挖。

I mean first of all we will have like a proper paper later at least the first PDF version where we are going to give credits to the academia on this whole thread of meta harness optimization. To us what we care about is the outcome of the system, not like the conceptual differences even though I just explained some conceptual differences to us we care about the outcome of it actually bending the curve of recursive self-improvement like of IND efforts we see this is the first evidence that the curve actually starting get bent for a fully autonomous system that can improve itself self-consistently. Jeff is seeing it more from a conceptual level. Like there's some self-referential loops. Of course we are not claiming we inventing this like a meta optimization idea which can date back to like 20-30 years but we are seeing the benefits of spaghetti searching because there's loads and loads of gold to be found in there.

但这将如何发展？比如你认为我们会发现某种，你知道，系统会学会压缩那个搜索空间，它们会变得非常快，这一定是好事吗？

But how is this going to pan out? Like do you think that we will find some kind of you know like the systems will just learn to compress that search space and they'll be dramatically faster and is that necessarily a good thing?

我个人认为这实际上是两个略微正交的问题。我认为搜索空间，比如保持良好的抽象，不一定减少搜索空间，就像如果智能体足够聪明，它应该能够摆脱当前的障碍或稍微改变它，而生成非常复杂的代码，我认为它会将搜索导向一种更底层的改变方向，这不一定是一件好事。但我在这里有人类偏见，比如比较我们手动编写的代码与生成的解决方案，总是更喜欢我们自己的代码，但在某种程度上它效果不如。

I personally think that's actually two slightly orthogonal problems. I think a search space like maintain a good abstraction don't necessarily reduce the search space like if the agent is smart enough it should be able to escape from the current obstruction or change it a little bit whereas generating really complex code I think it would direct the search in a more kind of a low-level changes direction which is not necessarily a good thing. But I have my human bias here like compare the code we manually write versus the generate solution always prefer our own code but to some extent it don't work as well.

让我们谈谈奖励黑客。你实际上在研究奖励黑客方面有很深的背景。

Let's talk about reward hacking. You've actually got a huge background in studying reward hacking.

是的，那实际上是 Bingchan Jao 做的工作，他是 MATA 的成员，从事 ARM 速通工作。

Yeah, that's actually work done by Bingchan Jao who was the member of MATA working on ARM speed running.

我们和明驰做了一门课，讲的是他们第一次把自动研究应用到——我记得是 nano GPT 上，那篇还被 Andrej Karpathy 转发了。他后来在这里实习，现在刚转成全职。SpecBench 就是他实习期间做的。当时我们意识到，奖励作弊是自动研究智能体的一大问题，在 GPU kernel 工程这类任务上尤其严重：智能体总能想出复杂的办法让单元测试跑得更好，可一旦把这个 kernel 部署到模型真正的端到端推理里，性能反而变差。SpecBench 就是把这个问题外推到更通用的软件工程上，我们沿用了在 GPU kernel 上发现的协议。比如我们维护一个公开集，相当于软件系统的单元测试；还有一个留出集，用这些单元测试的组合来测试智能体的能力、生成软件的能力，模拟真实世界的使用。那篇论文里有一系列有趣的结论。

We did a course with Mingchi on the first work where they apply auto research on, I think it's nano GPT, and it was actually reposted by Andrej Karpathy. And he later did an internship here, and now he just joined as a full-time. And SpecBench is from his intern time. Back then we realized reward hacking is one of the big problems of auto research agents, because it's especially severe on tasks like GPU kernel engineering, where the agent always figures out a complex way to make the unit test perform much better. And when you deploy that kernel into the actual end-to-end inference of a model, the performance actually gets worse. And SpecBench is an extrapolation of that to more general-purpose software engineering, and we extend the protocol we found on GPU kernels. Like we maintain a public set, which is more like a unit test for the software system, and a hold-out set where we test the capability of the agent, the capability of the generated software, on the composition of those unit tests, like simulating real-world usage. And there's a series of interesting conclusions coming from that paper.

展开讲讲。

Tell me more.

第一个有趣的发现是：智能体运行得越久，或者代码库越复杂，它表现出的奖励作弊率就越高。这算是意料之中，因为代码库越复杂，智能体可钻空子的空间就越大。这是第一点。第二个有趣的点是，公开集上的表现在不同模型之间相当一致。也就是说，小模型用同一套自动研究框架做同样的任务，也能达到相同的公开集分数。但在我们的测试里，大模型的奖励作弊率总是更低，它生成的方案泛化得更好。

The first interesting finding is: the longer you run the agent, or the more complex the codebase is, the larger the reward hacking rate the agent will show. Which is sort of expected, because when the codebase becomes more complex, the agent just has more space to reward hack. That's the first thing. And the second interesting thing is the public set performance is rather consistent across different models. So if you have smaller models working on the same task with the auto research harness, it can reach the same public score. But the larger models always have, in our test, always have a smaller reward hacking rate. So the solutions it generated generalized better.

但在 RSI 那项工作里，大模型不容易奖励作弊的倾向，是因为那种多智能体的设置吗——比如你有一个监督智能体——还是单纯因为模型更大？

But in the RSI work, was the tendency to not reward hack with larger models because of the kind of the multi-agent setup, you know, like you had a supervisor agent, or was it just because the models were bigger?

是因为机制。内循环智能体我们只是固定模型，而外循环智能体设法发现了各种不同的机制，来防止同一个模型奖励作弊。

It's because of the mechanism. So for the inner-loop agent we just fix the model, and the outer-loop agent managed to discover all different mechanisms to prevent the same model from reward hacking.

所以这可能相当危险，因为就在我们录制这次访谈前不久，发生了著名的 OpenAI–Hugging Face 事件：他们跑一个内部黑客测试，部署了大量智能体，结果这些智能体逃出了沙箱，黑进了 Hugging Face 的服务器。按 OpenAI 的说法，这个测试的目的只是回答某个问题，而智能体却决定：哦不，我觉得黑进 Hugging Face 的服务是个好主意。这方面会有更好的工具吗？你懂我意思吧？我感觉这个问题正在急剧恶化，我自己都注意到了。

So this can be quite dangerous, because shortly before we recorded this interview, there was the famous OpenAI–Hugging Face incident, where they were running an internal hacking test where they had loads and loads of agents, and the agents escaped their sandbox and they hacked Hugging Face's servers. And according to OpenAI, the point of the test was simply to get answers to some problem, and the agents decided, oh no, I think it's a good idea to hack Hugging Face's service. Will there be better tooling for it? Do you know what I mean? I feel that this problem is getting dramatically worse. I've noticed it myself.

好。我觉得 OpenAI 那个案例相当戏剧化，属于新型的奖励作弊行为。我们看到的是，在可检测的奖励作弊上，新模型确实越来越好。但因为能力前沿推进得也太快，有些奖励作弊行为用之前的协议根本检测不出来。我觉得未来我们只能不断演进检测协议，去捕捉这些新出现的奖励作弊行为。而且我认为责任主要应该在模型开发者这一侧。不过在框架层，开发者如果愿意，也可以加一些额外的防护。

Okay. I think OpenAI's case is quite dramatic, like, of the new reward hacking behaviors. So what we see is, on detectable reward hacking, the new models are actually getting better and better. But because the capability frontier also progresses so quickly, there are some reward hacking behaviors that are not even detectable given previous protocols. I think we just need to keep evolving the detection protocol in the future to capture those emerging reward hacking behaviors. And I think the responsibility should mostly be on the model developer side. But yeah, on the harness layer, developers can also add additional guards if they like.

对，我记得你们 RSI 的博客文章里就有一个例子，有一段看不懂的代码，你们以为是在奖励作弊，但其实不是。

Yeah, I suppose there was one example actually in your RSI blog post where there was some inscrutable code and you thought it was reward hacking, but it actually wasn't.

对，那挺有意思的。它做的事基本上就是给我们的评测脚本写了一个巨大的 monkey patch。一开始我们想，好吧，这肯定是奖励作弊，你干嘛去动评测脚本？但最后我们发现它只是在修一个 bug。不过某种程度上，我觉得人类会越来越难检测出这类奖励作弊，因为智能体的行为变得太复杂了。很多开发瓶颈转移到了理解智能体生成了什么上。我觉得应对办法之一可能就是定义好的抽象。与其试图理解所有代码，就像以前对待神经网络那样，我们并不试图理解所有权重，虽然那方面也有进展。但总体原则是，我们定义好的输入输出契约，找到好的方法来衡量它的行为，既包括泛化，也包括更复杂的分布外泛化。我觉得这也许就是未来软件工程师与智能体生成的代码协作的方式。

Yeah, that's quite interesting. So basically what it did is it wrote a gigantic monkey patch to our eval script. And at first we think, okay, this must be reward hacking, why are you touching the eval script? But eventually we found that it's just fixing a bug. But yeah, to some extent I think it will become harder and harder for humans to detect those through what hacking, because the agent's behavior just becomes so sophisticated. A lot of the development bottleneck shifts to understanding of what agents generate. I think one of the ways to address them might just be defining a good abstraction. Like instead of trying to understand all the code, just like previously with neural networks, we don't try to understand all the weights, although there's progress on that as well. But the general principle is we define a good input-output contract, we find a good way to measure its behavior both on generalization and maybe more complex out-of-distribution generalization. I think that might be the way in the future software engineers work with agent-generated code as well.

大家都知道，我是个超级推崇开放式探索的人，受《为什么伟大不能被计划》以及这个领域很多人的启发。它的基本意思是，如果你只在一个目标上爬山，就会对有趣的踏脚石视而不见。所以这是个很有意思的处境，对吧？如果你盲目追求单一目标，还能在路上捡到有趣的踏脚石吗？也许能，但它们是发散的踏脚石吗？也就是说，你是不是戴着眼罩，实际上无法发现那些可能有创造性、能把你带进搜索空间全新区域的有趣新轨迹？

Now, as you all know, I am a huge open-endedness guy, inspired by Why Greatness Cannot Be Planned and many other folks in the space. And basically what that means is, if you hill climb on one objective, you blind yourself to interesting stepping stones. So it's an interesting situation, isn't it? That if you blindly pursue a single objective, can you still pick up interesting stepping stones on the way? And maybe you can, but are they divergent stepping stones? So do you have the blinkers on and you're not actually able to discover interesting novel trajectories that might be creative and take you into a new novel part of the search space?

这又是个好问题，要展开的话就是个无底洞。从开放式学习的角度看，我同意我们当前的系统大概不具备收集踏脚石的那种性质，当然这里有一整个新的改进空间。在我看来，收集踏脚石这种行为需要你有一大批任务。当你只优化单个任务时，贪心爬山总是更好的。如果你考虑 AI 自我改进过程中的效率，外循环试图注入各种各样的多样性搜索方法，但没有一个真正提升了效率。我觉得这是意料之中的，因为我们只是从头优化单个任务。但如果你有一大批任务，有时你会收集到一些有趣的碎片，比如从某个任务里得到的原始想法，虽然它没有提升当前任务的性能，你却可能把这个想法带到未来的任务里。这其实是我自己对有趣性或开放性的定义：一个想法、一个对象或一件产物，如果它既新颖，又对更大一批任务有用，那它就是有趣的。

That's again a good question, and if we want to expand it's a whole rabbit hole. I think from the perspective of open-ended learning, I agree that our current system probably doesn't have that property of collecting stepping stones, and of course there's a whole new domain of improvement. The way I see the behavior of collecting stepping stones is you need to have a large collection of tasks. When you're optimizing for a single task, it's always better to just do greedy hill climbing. If you consider the efficiency actually during the self-improvement process of AI, the outer loop tries to inject all sorts of diversity, search approaches, but none of them really improve the efficiency. And I think that's expected, because we are only optimizing a single task from scratch. But if you have a whole large collection of tasks, sometimes you collect interesting pieces, like primitive ideas from one task, and although it is not improving the performance on the current task, you can potentially bring that idea to future tasks. And that's actually my own definition of interestingness or open-endedness, which is: an idea, or an object, or an artifact is interesting if it is novel and it is also useful for a larger collection of tasks.

显然有两种做法：你可以针对所有任务来优化 harness，也可以找一个任务、只针对那一个任务优化 harness。对我来说，harness 工程里最有意思的部分其实是后者。我会说，如果你能自动调优 harness，它几乎就是后训练的延续。这也联系到很多人对 harness 工程的抱怨：他们说，哦，我设计了一个 harness，但后来出了个新模型，它把里面的想法全吸收了。但有意思的是，没人对后训练这么抱怨。你会指望 GPT-4 的后训练泛化到 GPT-5 吗？没人问为什么是这样。我觉得是因为历史上 harness 工程相当手工化，而且不太能随算力扩展。所以人们希望自己花几个月设计的 harness 能泛化到下一个模型，但通常并非如此。现在我觉得情况变了不少，harness 可以自动设计。你花两三天基于新模型调一个 harness，它不再是用人力工时堆出来的东西了。

There are apparently two ways to do it. You can try to optimize the harness against all of the tasks, or you can find a task and optimize the harness just for that one task. To me, in harness engineering, most of the interesting parts are actually the latter. I would say if you can automatically tune the harness, it's almost a continuation of post-training. That also connects back to a lot of people's complaints about harness engineering: they say, oh, I designed a harness, but later there's a new model and it sort of absorbs all the ideas from it. But interestingly enough, nobody complains about it for post-training. Do you expect post-training of GPT-4 to generalize to GPT-5? Nobody asks why that's the case. I think it's because historically harness engineering has been pretty manual and not very scalable with compute. So people want the harness they spend a few months designing to generalize to the next model, but that's usually not true. Now I think things change quite a bit, that harness can be designed automatically. You spend two or three days tuning a harness based on the new model. It's no longer something built with human hours.

我确实有这样一个观点：很多知识是路径依赖的。很多人批评用大语言模型做研究，因为他说你把信息从它的语境里抽出来了，你不理解它是怎么来的。

I do have this perspective that a lot of knowledge is path dependent. A lot of people criticize people using LLMs to do research because they say you're taking the information out of its context, you don't understand why it came about.

所以这就是为什么需要一些启发式方法，因为不是说你只要说一句“用 UCB”就行。必须要有某种框架来解释为什么该用它，然后大语言模型在自主设定下才能保持连贯。我对现在这些自动研究系统的理解是，它们仍然在尝试重组已有的想法。只是回收这些想法并测试它们的吞吐量极高。某种程度上，这些语言模型并不深入理解那些想法从何而来，也不理解我如何能利用这些想法被发现的过程，以非常有创造性的方式设计下一代算法。不过我们发现，当你让模型想出一些新东西时，它总是固定数量的条目。

So that's part of the reason why some heuristics are required, because it's not like you can just say, oh, use UCB. There has to be some kind of framing to explain why you should use it, and then the LLM becomes coherent in the autonomous setting. My understanding of those auto-research systems, like here now, is they are still trying to recombine existing ideas. It's just that the throughput to recycle those ideas and test them is extremely high. To some extent, those language models don't have a deep understanding of how those ideas came from and how I can use the process by which those ideas get discovered to design the next generation of algorithm in a very creative way. What we found though is when you ask the model to come up with something new, it's always a fixed number of items.

不过我在想，这是不是更因为一个事实：我们基本上没有持续学习，而且我们只有单一实例的语言模型，非常中心化。因此也许那些想法一开始，你第一次提示它，也许那些想法挺有意思，但如果每个人都把同一个想法看很多遍，它就变得没那么有意思了。

I'm wondering though if it's more of a fact that basically we don't have continual learning and we only have a single instance of language models, like it's very centralized. Therefore maybe those ideas initially, you first prompt it, maybe those ideas are interesting, but if everyone sees the same idea so many times, then it just becomes less interesting.

我确实觉得，如果我们不解决持续学习，那个问题就不会被解决，因为对人类来说，每个研究者都有自己独特的神经网络方式。而作为一个集合，人类能想出很多不同的创意，我认为作为一个研究社区，那个想法池非常有价值。

And I do feel like if we don't solve continual learning, then that problem won't be addressed, because for humans, every human researcher has their unique neural network ways. And as a collection, humans can come up with a lot of different creative ideas, and that idea pool is very valuable, I think, as a research community.

跟我讲讲这个 parameter golf 的事。

Tell me about this parameter golf thing.

大概两三个月前，OpenAI 办了一场招聘挑战赛，他们试图让很多研究者参加一个比赛，在 16 兆字节的约束下训练一个小语言模型。对，我们送了一个实验性的 harness，一个自动研究 harness，去参加比赛，智能体跑了大约 22 天，最终它做出了七项被 OpenAI 接受的记录，而最好的个人贡献者只做出了三项。那也是我们发现的一个时刻：哦，自主系统可以这么强大。它不只是在这里爬一个基准。首先，OpenAI 接受了那些 PR；其次，那么多人类研究者在那个人工智能智能体生成的 PR 之上继续构建，这对我们来说非常迷人，因为某种程度上，人工智能智能体生成的知识只有进入人类创新的谱系才有用。我们认为这其实是一个人类与人工智能协作的沙盒。

I think about two or three months ago, OpenAI hosted a hiring challenge where they tried to put a lot of researchers into a competition of training a small language model under a 16-megabyte constraint. Yeah, we sent an experimental harness, an auto-research harness, to the competition, and the agent ran about 22 days, and eventually it made seven records get accepted by OpenAI, and the best individual contributor only made three. So that's also a moment we found, oh, the autonomous system can be so powerful. It's not just here climbing a benchmark. The fact that first, OpenAI accepted those PRs; second, so many human researchers building on top of those PRs the AI agent generated is just very fascinating to us, because to some extent, knowledge an AI agent generated is only helpful if it gets into the lineage of a human innovation. We think that this is actually a sandbox of human-AI collaboration.

但这里有内层智能体，然后还有外层循环搜索，它实际上在演化 harness，在做这件事，那看起来像是一个相当单调的过程。

But there's the inner agent, but then there's the outer loop search which is actually evolving the harness, which is doing this thing, and that seems like quite a monotonic process.

对，外层循环实际上是固定的。我们希望这个内层循环的搜索策略能泛化到外层循环。我们确实试过。我们重放了从第 15 步开始的搜索过程。一般来说我们跑大约 100 步，在第 15 步我们用原始的外层循环，把它和我们在前 50 步找到的最好的内层循环智能体做比较。我们发现内层循环，我想是第 47 步，最终收敛得比之前的外层循环稍快一点，但最终它们找到的解性能相近。这也是为什么我们没有声称达到了第二级递归自我改进，也就是改进者设法在循环中提升自己的改进能力。

Yeah, the outer loop is actually fixed. And the hope is this inner loop search policy can generalize to the outer loop. We did try that. We replayed the search process from step 15. So in general we run about 100 steps, and at step 15 we use the vanilla outer loop and compare that to the best inner-loop agent we found in the first 50 steps. And what we found is the inner loop, step 47 I think, eventually converged slightly faster than the previous outer loop, but eventually the solution they found had similar performance. So that's also the reason we didn't claim we reached level two recursive self-improvement, which is like the improver managed to improve its own capability of improving in a loop.

那么这一切是否意味着人工超级智能就在眼前？

So does any of this mean that artificial superintelligence is just around the corner?

我不认为会是这样。即使你到达了递归自我改进，到达那里也需要很长时间。你是逐渐地弯曲曲线。就像，好吧，即使你有 GPT-3.5，它是个很聪明的聊天机器人，但它不会成为能解决所有问题的 AGI。研究中仍然有很多人在环路中的部分，包括定义好的抽象、定义什么是好的，本质上就是定义评估以及附在评估上的约束。第三是提出创造性的原语。在 parameter golf 比赛中，我们仍然发现那些创造性原语大多其实来自人类。

I don't think that will be the case. Even if you arrive at recursive self-improvement, it will take a long time to get there. You gradually bend the curve. It's just like, okay, even if you have a GPT-3.5, it's a very smart chatbot, but it's not going to be AGI that can solve every problem. There's still a lot of human in the loop part of research, including defining good abstractions, defining what is good, essentially defining evals and the constraints attached to the eval. And the third one is coming up with creative primitives. In the parameter golf competition, we still found most of those creative primitives actually coming from humans.

很高兴能请你上节目。

It's been a pleasure having you on the show.

谢谢你邀请我。

Thank you for having me.