后训练配方与 RLVR：对话 Nathan Lambert

大家好，欢迎收听 Lidden Space 播客。我是 Allesio，Desel 的合伙人兼 CTO，和我一起的是 Small AI 创始人 Swixs。你好。我们很高兴再次邀请到 AI2 的 Nathan Lambert。欢迎。

Hey everyone, welcome to the Lidden Space podcast. This is Allesio, partner and CTO Desel, and I'm joined by Swixs, founder of Small AI. Hello. And we're excited to welcome back Nathan Lambert from AI2. Welcome.

谢谢。很高兴来到这里。

Thanks. Fun to be here.

我觉得我还得提一下 Interconnects、Freedman 播客，还有你和 AIE World's Fair，过去一年半你做了很多事。

I feel like I also have to say Interconnects and the Freedman podcast and you and the AIE World's Fair, like you've just done a lot in the last year and a half.

其实没那么夸张。我拒绝了很多事情。

Not that many. I still say no to plenty of things.

你第一次上我们节目是 2024 年 1 月，那时你刚加入 AI2。后来你在 NeurIPS 又和我们一起，做了开放模型那期。其实是 Luca 主讲的，你提供了支持。最近你在旧金山参加 AIE。首先，恭喜你获得推理赛道的最佳演讲者。给你这个。我只能用表情符号表达了。

Your first episode with us was January 2024 when you just joined AI2. Then you joined us again at NeurIPS where you did the open models. Well, Luca did, and you supported. And then more recently here in SF for AIE. First of all, I wanted to congratulate you on winning the best speaker for the reasoning track. Here you go. I'm limited by emoji.

哦，是的。这是个不错的 AI 生成图。我看起来太禅意了。

Oh yeah. There's a nice AI generated. I look too zen.

这张 AI 生成的图里我看起来好禅意。我们让赛道主持人在你演讲时拍了照片，然后转成吉卜力风格，但这张你闭着眼睛。挺搞笑的。

I look so zen in this AI generated. So, we had our track host take photos of you while you're speaking and we turned them into Ghibli photos, but this one your eyes were closed. It's funny.

好吧。我们本来想请 Mochi（推理方面的 Palmsky）一起，但她好像很不安分。有点太疯了。Mochi 很不安分。好吧。

Okay. We were trying to have Mochi, the reasoning Palmsky, join us, but I think she's very restless. A little too crazy. Mochi, very restless. Okay, sure.

你一直在做很棒的工作。老实说，我们想重点聊聊 Tulu 和 RLVR。从这里开始可以吗？

So, you've been doing really good work. Honestly, I think one of the things we wanted to establish was Tulu and RLVR. Is that a good place to start?

当然。这让我们从最近的旅程开始。我们可以回顾一下 Tulu 3 的目标，以及它如何融入新的叙事。目标是压缩复杂的行业后训练方法，使其变得易于处理，让你可以自行修改，并在后训练中达到最先进水平。相对于前沿实验室，我们的任务数量可能更少。Tulu 的后训练套件大概有 10 到 15 个任务，但我猜 OpenAI 的后训练可能有数百个评估。增加评估意味着更多的数据工作和混合工作。但在基于当时 Llama 的 8B、70B 和 4B 模型套件的核心评估上，它匹配或超越了 Meta。Meta 对 Llama 3.1 有不同的优先级，那是一套很棒的模型。关键在于如何从 Llama 3.1 报告（包含复杂反馈图）和早期 Anthropic 论文（含多种模型变体和宪法 AI）中提炼复杂的后训练解释。当你进行大规模指令微调到偏好微调时，这看起来是什么样的？在讨论强化学习之前，核心贡献是我们展示了如何扩展偏好数据。学术界自 Hugging Face 模型（如 Zephyr beta）以来一直使用同一个数据集，UltraFeedback 数据集变得流行。一年后，它仍然是开放偏好微调的最先进数据集。这是那些显而易见但不必如此的事情之一。所以，关键在于让人们获得更成熟的方法。我在与 Jordan 的谈话中提到了 RLVR 的起源。在开放环境中工作，很多时候是试图匹配行业做法。我们走的是不同的路，因为我们的基础设施不同。OpenAI 在长上下文方面的一些做法对我们来说效果不好，因为我们的基础模型可能没有足够的算力，或者缺乏某些法律相关数据集。但方向上是复现。我一直想请 John Schulman 上播客。当时他获准与我聊天，他说的话证实了我关于指令微调、多任务和偏好微调的很多观点。他说：‘哦，是的，每个人都在输出上做强化学习。’这就是我们得到 RLVR 想法并将其扩展为通用方法的原因。当时也有类似的工作，比如用于数学和编程领域的 VinePPO 和 QuietStar，它们使用可验证奖励。RLVR 的重点是将其应用于通用方法。

Sure. It starts us in the recent journey. I think we can recap the story of what Tulu 3 was aiming to be and then how it got folded into the new narrative. The goal is to compress complicated industry post-training recipes into something tractable that you can modify on your own and do post-training at a state-of-the-art level. What we do relative to Frontier Labs is that we probably have a smaller number of tasks. Our post-training suite for Tulu is probably like 10 to 15 tasks, but I would guess post-training at OpenAI has hundreds of evals. Adding more evals is more data work and mixing work. But on core evals for our suite of models from 8B, 70B, and 4B based on Llama at the time, it matches or beats Meta on these core evals. Meta has different priorities for Llama 3.1, which is a great set of models. It's about how we distill complicated post-training explanations from reports like Llama 3.1 with complex feedback diagrams and earlier Anthropic papers with multiple model variants and constitutional AI. What does that look like when you're doing large-scale instruction tuning into preference tuning and what else you might add? The core contributions before we talk about reinforcement learning is that we showed how to scale up preference data. The academic community had been using one dataset since the Hugging Face models like Zephyr beta, where the UltraFeedback dataset got popular. A year later, it's still state-of-the-art for open preference tuning. It's one of those obvious things that doesn't need to be the case. So, it's about making more mature recipes available to people. I mentioned on a talk with Jordan the origin of RLVR. When you work in the open, a lot of it is trying to match what industry has done. We're on a different path because our infrastructure is different. Some things OpenAI does for long context won't work well for us because we might not have enough flops in our base model or certain datasets for legal things. But directionally, it's about reproducing things. I've long tried to get John Schulman on the pod. At the time he had approval to chat with me, and what he said confirmed a lot of things I had said on instruction tuning, multitask, and preference tuning. He said, 'Oh yeah, everyone just does RL on the outputs.' That's how we got the RLVR idea and scaled it into a general method. There were similar works at the time like VinePPO and QuietStar for math and coding domains with verifiable rewards. The RLVR thing was about doing it in general recipes.

而且这个命名后来就固定下来了。最初我们有……我想尤其是 Costa Hang，他是 AI2 的首席强化学习工程师，现在在做某个隐形创业公司。Hamish Iverson，还是华盛顿大学的学生，主导了大部分技术工作。命名原本是‘基于真实值的强化学习’，但‘可验证奖励’是一个更通用的概念，因为只有数学问题有真实值，而代码是可验证的，精确指令遵循也是可验证的。这个名字的演变很好，当你考虑更多领域时，这很合理。这就是它流行起来的原因。一旦 Jensen 开始使用它，我就想，好吧，这就是我们的目标。但它在 DeepSeek 之后才真正起飞。当像他这样的人在幻灯片上使用这个缩写时，它也非常清晰。RLHF 是四个字母。我们想进化它，有一个类似的四个字母缩写。这里面没什么魔法，但确实在这些小细节上有意图。

And the naming was something that stuck. Originally we had... I think it's especially Costa Hang, who was a lead RL engineer at AI2, now doing some stealth startup. Hamish Iverson, still a student at UDub, led most of the technical work. The naming was going to be RL from ground truths, but verifiable rewards is a more general notion because only math questions have a ground truth, while code is verifiable, precise instruction following is verifiable. It's a nice evolution of the name, which makes sense as you look at more domains. That's why it caught on. Once Jensen started using it, I was like, okay, that's our goal. But it took off after DeepSeek. When people like that have the acronym on the slides, it's also very clear. RLHF is four letters. We want to evolve that and have a similar four-letter acronym. There's not much magic to it, but there's definitely intention on these little things.

RLGT 可能就没那么有效。不知道为什么，但确实如此。

RLGT may not have worked as well. I don't know why, but yeah.

是的。这些人指出的正是我们思考的，他们做了这个名称更改，效果很好，这很有趣。

Yeah. And that's what these people called out were definitely thinking and they made that name change which works, which is fun.

你确实提到过，所以我们会展示 RLVR 图表。你提到现在想改一下，我们也会稍微预览一下关于智能体的讨论。

You did mention, so we'll show the RLVR chart. You did mention that you wanted to change it now, and we'll sort of preview a little bit of the agents discussion.

是的，当你接触 RLVR 时，实际上只有一个函数来检查语言模型的输出是否正确。没有真正的环境，因为你只是在看生成结果。现在我需要弄清楚如何正确传达多跳工具使用在这方面的样子，这确实是人们正在做的事情。我在想用什么合适的图表来概括 O3 的训练方式。它们会采取多个动作，因为下一个序列取决于来自环境的反馈，而环境是某种信息存储。所以当它搜索一条小众信息时，没有来自必应搜索之类的反馈，你就无法知道下一步动作。这一步确实在发生。随着人们尝试转向更端到端的强化学习，环境的概念变得很强：你要从多次生成中寻找稀疏信号，这正是人们想做的。我认为现在是否真的有人在这样做还有争议。深度研究博客文章暗示他们做了一堆小规模的强化学习，然后系统就工作了。我认为实际情况是，人们在许多小事情上训练，做一些提示，然后发现当你把这些片段组合起来，或者使用模型的不同微调版本时，它就能工作。所以深度研究似乎包含了 O3 的某个微调版本，你在不同的强化学习领域这样做，它就有效，而不是深度研究直接在结果上训练。这很有道理，因为对深度研究做基于结果的强化学习又会变成 RLHF：你需要两个人类来判断哪个生成的报告更好。你当然可以这样做，OpenAI 展示的整个 sick event 事件表明他们在后训练中有许多不同的奖励模型和奖励信号，但这只是其中之一。我认为让这一切成为可能的大部分进展是在信息检索、编辑和搜索任务上做强化学习。我们和 Noam Brown 讨论过深度研究和可验证奖励。他提到那是非可验证事物通过强化学习工作的一个例子。在你最近的一篇文章中，你也谈到大型实验室拥有所有这些数据，可以找到长尾事物进行强化学习，当把它们放在一起时，问题就解决了。

Yeah, when you are introduced to RLVR, there's just a function really that checks if the output from the language model is correct. There's no real environment because you're just looking at the generation. Now I need to figure out the right way to communicate what multi-hop tool use looks like for this, which is something people are definitely doing. I'm thinking about what the right diagram is to encapsulate how O3 is trained. They take multiple actions because the next sequence depends on feedback from the environment, which is some sort of information store. So when it's searching for a niche piece of information, you can't know what the next actions are without feedback from, say, Bing searches. That step is very much happening. As people try to transition to more end-to-end RL, there's a strong notion of environment: you're looking for a sparse signal from multiple generations, and that's what people want to do. I think it's debatable whether people are actually doing it now. The deep research blog post hints that they do a bunch of small-scale RL and then the system works. I think what's happening is people train on a bunch of small things, do some prompting, and see that when you put these pieces together or use a couple of different fine-tunes of a model, it works. So deep research seems to have some fine-tune of O3 in it, and you do that with some different domains of RL. It works, rather than deep research being trained on the outcome. That makes sense for why it doesn't work in deep research, because doing outcome-based RL for deep research would be RLHF again: you have to have two humans and ask which generated report is better. You can definitely do that, and the whole sick event thing OpenAI showed that they have many different reward models and reward signals in their post-training, but that's just one of them. I think a lot of the progress in making it exist is doing RL on a bunch of information retrieval, editing, and search tasks. We talked with Noam Brown about deep research and verifiable rewards. He mentioned that's an example of non-verifiable things having RL work on them. In one of your recent posts, you also talked about how the big labs have all this data to find long-tail things to RL on, and when you put them all together, it fixes it.

你是否觉得我们能够验证的东西是一个很大的瓶颈？验证只在这些较小的原子性事物上进行，所以我们无法真正扩展它。

Do you feel like what we're able to verify is a big bottleneck? That verifications are only done in these smaller atomic things, so we cannot really scale that.

我认为我的评论是关于这个的。在这篇文章中，我主要反思了智能体进步相对于模型进步会是什么样子。我们已经有了近三年的模型进步，对此已经很习惯了。这不仅仅是关于能在小事情上做强化学习，而是做任何后训练来修复奇怪的行为。如果你能得到正确的信号，强化学习是一种非常数据高效的方式。但你也可以说：它做了这个奇怪的非可验证的事情。让我们创建 100 或 1000 条指令包含在后训练中，这样模型就能正确地进行这种信息提取或软提取。这是一个我想用更多任务例子来充实的方向。如果你观察 Claude Code 的运行，它在后台做什么？它大量读取文件，甚至压缩上下文。这其实不是可验证的事情。但搞砸这一点对于长上下文动作和更长的任务来说是一个至关重要的技能：很好地压缩。这需要一些训练上的创新，关于如何有效地修改训练数据，而不是拥有所有的多轮上下文。你只需插入摘要，并希望性能保持，因为更短的上下文也能节省成本。有很多像这样的新领域。

I think my comment was on making so. In this post, I was reflecting mostly on the question of what agent progress will look like relative to modeling progress. We've had almost three years of modeling progress, and we're pretty used to the messaging on that. It wasn't just about being able to RL on small things, but doing any post-training to fix a weird behavior. RL is a very data-efficient way if you can get the right signal. But you could also just say: it does this weird non-verifiable thing. Let's create 100 or 1000 instructions to include in post-training so that the model does this type of information extraction correctly, or soft extraction. It's a space I want to flesh out more with more examples of tasks. If you watch Claude Code going, what is it doing in the background? It's a lot of reading files and even compressing context. That's not really a verifiable thing. But that being messed up is a super crucial skill for long-context actions and longer tasks: compressing well. That's going to take some training novelty on how you can effectively modify your training data instead of having all the multi-turn context. You just insert the summary and you want to make the performance stay as well, because it's also a cost saving to have shorter context. There are just a lot of new domains like that.

但你是否觉得在发布之前就能弄清楚这些事情？还是说实验室有巨大优势，因为他们有大量用户数据，可以在推理时检查这些？

But do you feel like you can figure out what these things are before you release? Or do the labs have a big advantage because they have so much user data that they can inspect this at inference?

我认为在这一点上主要是看真实世界的数据。只要有明确的基准，你就可以公开使用它们，但我们看到行业围绕不同形式的数据进行了整合。我认为这对人们来说是一个非常重要的接触点。

I think it's mostly looking at real-world data at this point. To the extent that there are clear benchmarks, you can use them in the open, but we see the industry consolidated around data in different forms. I think that's a real important touch point for people.

我很好奇谁还在收集大家使用的可靠开放数据源。

I'm curious who's still collecting reliable sources of open data that everyone uses.

这个领域有很多动作，但很难获得 traction。

There's a lot of action in the space, but hard to get traction.

是的。

Yeah.

所以我认为很长一段时间以来，偏好数据一直是人们理解拥有大型存储库会非常好的东西。如果你想要，你可以烦我尝试发布所有 Tulu 的数据。我们有一个最终数据集，但我们有来自更多模型的完成和评分。我正在和学生讨论如何标记这些，因为我们有太多的完成和 LM 作为评判的 AI 反馈数据，我们不知道如何清理。这是一件事。问题在于，我认为其中很多是任务和模型特定的。所以这个关于在策略上采用强化学习术语用于偏好数据和偏好建模的概念：你希望训练奖励模型的序列看起来像你开始微调的模型。这使得它很难开箱即用。例如，我提到的 UltraFeedback 包含很多模型。所以人们微调的大多数模型，都有一些改进的信号，我不知道这能持续多久。我们仍然没有回答人类反馈与 AI 反馈相比有多重要的问题。每次我和前沿实验室的人交流时，他们都说，‘是的，我们仍然使用人类偏好数据。’我就想，‘好吧，我无法获得那些数据，也不知道如何衡量它真正给你带来了多少好处。’可能大部分好处在于——用什么形容词来描述聊天机器人竞技场。

So I think for a long time, preference data has been something where people understand that it would be very good to have large repositories of it. If you want that, you can annoy me to try to release all for Tulu. We have a final dataset, but we have completions and ratings from more models. I'm talking to the student to figure out how to mark this down because we have so much completions and LM-as-a-judge AI feedback data that we don't know how to clean. That's one thing. The problem is that I think a lot of it is task and model specific. So this notion of on-policy to adopt an RL word for just this preference data and preference modeling: you want the sequences that you're training this reward model on to look like the model that you're starting to fine-tune. That is something that has made it hard to grab off the box. For example, UltraFeedback that I mentioned has a lot of models in it. So most of the models that people are fine-tuning, there's some signal for it to improve on, and I don't know how long that lasts. We still don't have the answered question on how important human is versus AI feedback. Every time I check in with people at frontier labs, they're like, 'Yeah, we still use human preference data.' And I'm like, 'Okay, I don't have access to that, and I don't know how to measure how much it gives you really.' It might be most of the benefit is on the what's the right adjective to describe chatbot arena.

感觉大家对 Chatbot Arena 有些失望，但人类数据可能极大地提升了留存时间和整体偏好，而大多数学者之前做的都是 multi-kill 和 AlpacaVal 这类东西，这就不那么关键了。大家都在注意力经济中竞争。

It's like people are down on Chatbot Arena, but it might be that the human data helps boost retention time and general preference a lot, where most academics were doing multi-kill and AlpacaVal type things, which is just not as crucial. Everybody's fighting in the attention economy.

你提到了 LMSys 和 Elo Arena，那是我在 Interconnects 上很喜欢的一篇文章。它们是不是不行了？竞技场还有未来吗？这会怎么发展？他们现在有 1 亿美元了，打算做什么？

You mentioned LMSys and Elo Arena, that was one of your posts on Interconnects that I really enjoyed. Are they cooked? Is there a future for arenas? How does this play out? They got $100 million now, like what are you going to do?

我不知道这笔钱对他们有什么用，但我认为 Elo 仍然有价值，尤其是在前沿领域。人们很 cynical，但在压缩竞赛中——即用多便宜的模型还能表现良好——这对很多人来说仍然非常有用。

I don't know what the money does for them, but I think that the Elo is still valuable, especially at the frontier. People are very cynical, but in the compression race of how cheap a model can be while still performing well, this is still so useful to a lot of people.

聊天是王道。

Chat is king.

是的，每个人都用这些东西聊天。这就是为什么我用的 GPT-4.5 在 Chatbot Arena 上表现没那么好。我觉得它在 Yep 上排名更高，Yep 是一个新竞争者。他们有一个 vibe 类别。

Yeah, everyone chats with these things. It's why I use GPT-4.5 isn't as good on Chatbot Arena. I think it's higher on Yep, which is a new competitor. They have a vibe category.

抱歉，Yep？

Sorry, Yep?

是的，有个 Yep。你可以查一下。它是一个竞争对手，另一家创业公司。他们有各种类别，其中一个是 vibe，GPT-4.5 排在最前面。我就想，‘好吧，这有点道理。’

Yeah, there's Yep. You can look it up. It's a competitor, another startup. They have categories, and one of their categories is vibes, and GPT-4.5 is on top. And I'm like, 'Okay, there's something this tracks.'

它是一个前沿模型。

It's a frontier model.

是的。那些无形的东西非常好。排行榜已经建立起来了。人们仍然应该使用它。它对社区来说是一种聚焦功能，跨越从工业界到学术界的各个群体。

Yeah. It's just that intangible stuff is very nice. The leaderboard is established. People still should use it. It's kind of a focusing function for the community across different batches from industry to academia.

是的。

Yeah.

我不打算替他们解决变现问题，但有明确的规范和可以永远攀登的目标是非常好的。比如这种将模型联系起来的 Elo 概念，你无法让它饱和。

I'm not going to try to solve their monetization problems for them, but having clear norms and things that could be hill climbed forever is very good. Like having this idea of an Elo linking models that you cannot saturate.

是的，但你可以钻空子。所以我认为这就是问题所在。

Yeah, but you can game it. So I think that's the issue.

是的，但每个人都在多个方面进行评估。

Yeah, but everyone evaluates on multiple things.

Sarah Hooker 出来了，我从没见过她这么公开地表达不满，但她通常不会这样公开表态。

Sarah Hooker came out, I've never seen her so public about any of her gripes, but she doesn't really go public like that.

是的。Artificial Analysis 也有一个，我觉得挺酷的。另一个与这个讨论相关的问题是，很多数据实际上是单轮的，不是多轮的。我想知道如何创建合适的多轮竞技场，因为你需要切换模型。这就是 Elo Arena 的整个前提。这取决于用户数据的价值。

Yeah. Artificial Analysis also has one which I think is kind of cool. The other thing I think is relevant to this discussion is that a lot of the data is actually single-turn, like a single round, not multi-turn. I wonder how to create proper multi-turn arenas because you have to switch the models. That's the whole premise of Elo Arena. It depends on how valuable the user data is.

如果用户数据仍然与推理同等或更有价值，就会有一个平台不断推动这件事走向更昂贵的方向。是的。所以他们会建立一个深度研究竞技场，因为如果我是 OpenAI 在做深度研究，那就是我想要的数据。而且有竞争对手，LMSys 是拥有市场地位来正确建立它的实体。

If the user data keeps being equally or more valuable than the inference, there's going to be a platform to keep pushing this into more and more expensive things. Yeah. So they're going to set up a deep research arena, because that's the data that, if I was OpenAI working on deep research, that's the data I want. And there are competitors, and LMSys is the entity that has the market placement to set it up right.

这几乎就像我看 Scale 一样。Scale 不断攀登 AI 数据处理的前沿，因为他们是名牌，他们不断攀登增量评估游戏，很多公司都有长寿性。

It's almost like how I see Scale. Scale kept climbing the edge of what AI data processes is, and because they're the name brand, they keep climbing the incremental evaluation game, and a lot of them have longevity.

是的。这在某些方面是一种网络效应。

Yeah. It's a network effect in some ways.

你提到了 Scale，这是另一个热门话题，但我们把所有的热点评论留到最后。我确实想先专注于技术层面。你还在写 RLHF 的书。现在是 RLVR 的书了吗？

You mentioned Scale, which is another hot topic, but we'll put all the hot takes at the end. I do want to focus on being technical up front. You're still writing the RLHF book. Is it an RLVR book now?

我可以解释一下。归根结底，RLVR 还不够成熟，也不够有趣，不足以写成一本书。所以从两个方面来说，这就是我不想改名的原因。还有一些个人职业策略，但这应该独立于客观上什么是好书。因为 RLVR 在未来 18 个月内会发生很大变化。我们已经看到了。有很多新算法，但我认为在如何为它做正确的预训练、数据是什么、如何使用涌现等方面，还有很多隐藏的东西。所有这些都将是 RLVR 的核心。我在观察 o3 是一个小众模型，还是成为每个人都必须遵循的路径，它有不同风格的工具使用，特别是搜索。

I can give my spiel on it. Ultimately, RLVR is not mature enough, nor is it as interesting of a book. So on two fronts, that's why I don't want to rebrand. There's also some personal career strategy, but that should be independent of what is objectively a good book. Because RLVR is going to change so much in the next 18 months. We've already seen it. There are all these new algorithms, but I think there's a lot more under the hood on how you do the right pre-training for it and what the data is, how to use emerges. All of this stuff is core to what RLVR will be seen as. I'm watching to see if o3 is like a niche model or becomes the path that everybody needs to follow, with its different style of tool use, particularly with search.

好的。

Okay.

而且我们不知道 OpenAI 是怎么做到的，这些我认为是 RLVR 书的核心，但我们没有。而 RLHF 更具跨学科性，就像 Chatbot Arena 永远不会饱和一样。RLHF 永远无法解决。我们大致知道这些对齐和过度优化的问题，以及人们用来获取数据的管道。是的，我可以在书中添加更多 RL 算法，这对我学习有好处，但这并没有真正改变。它不会改变像‘哦，奖励建模是什么，以及人们今天实现它的不同方式，无论是价值函数还是奖励模型等等’这样的问题。所以我认为 RLHF 的广度很好，我会告诉很多学者，我认为 RLHF 问题将是基础性的，并且研究速度会稳定得多，而我们正处于 RLVR 的巨大高峰上，但它可能很快就被解决，然后在学术上归零。这有点夸张，但可能有一种最佳实践可以在任何问题上达到 100% 的准确率，然后它就被解决了，而关于什么是偏好的争论将永远持续下去。

And we don't know how OpenAI did this, and these are the things that I think are core to an RLVR book that we don't have, whereas RLHF is more interdisciplinary, in the same way that Chatbot Arena can never be saturated. RLHF can never be solved. And we kind of know these problems of alignment and overoptimization and what the pipelines to getting data that people are using are. And yes, I can add more RL algorithms to the book, which is nice for me to study, but that's not really changing. It's not changing like, 'Oh, what reward modeling is and the different ways that people implement these today, whether it's a value function or reward model and stuff like this.' So I think the breadth on RLHF is nice, and I think I would tell a lot of academics that I think RLHF problems are going to be foundational and kind of just have a much more steady study rate, where we're on this massive spike of RLVR, but it might just be solved and then it goes back to zero academically. It's an embellishment, but there could just be a best practice for getting 100% accuracy on any problem that you want, and then it's solved, where the debate on what is a preference is going to go on forever.

是的，因为它是可验证的，有正确答案。

Yeah, because it's verifiable, there is a right answer.

你说未来 18 个月会有很多变化，是什么意思？你预见到什么？实际上，我们先回顾一下最近已经发生了什么。

What do you mean about over the next 18 months there'll be a lot of changes? What do you foresee? Actually, let's just catch up on what's already happened in recent history.

是的。所以我们有两类信息：模型在做什么，以及研究人员在做什么。我认为模型在什么是真正的前沿方面提供了很多灵感，比如 o3、Gemini 2.5、Claude。这些都是混合体。o3 是最 Scaling 的 RL 方法，而 Claude 和 Gemini 2.5 非常相似，都是可以开关的混合推理模型。它们以不同的方式推出。所以 Gemini 在发布时没有混合推理，但后来引入了，而 Claude 在发布时就具备了。

Yeah. So there are two categories of information that we have: what are the models doing and what are the researchers doing. I think the models provide a lot of inspiration in terms of what the actual frontier is, and that's things like o3, Gemini 2.5, Claude. These are a mix. o3 is the most scaling RL approach, and then Claude and Gemini 2.5 are very similar with hybrid reasoning models that you can turn on and off. They rolled it out in different ways. So Gemini didn't have hybrid reasoning at launch, but they've brought it in, and Claude had it at launch.

最重要的问题之一是：O3 的纯推理模型路径还是混合推理模型更有用？它们在训练方法上有分歧吗？我认为 Nvidia Llama Neatron 推理论文可能是关于混合推理最详细的论文，而 DeepSeek R1 仍然是纯推理模型的经典配方。这些方法非常不同，我不知道哪个会胜出。此外，在数据和强化学习方法方面还有很多工作。我认为有一大堆关于 GRPO 的抱怨，说某些地方的数学说不通。

One of the most important questions has got to be: is the O3 path of just a reasoning model or hybrid reasoning models more useful? Do they diverge in their methods for training them? I think the Nvidia Llama Neatron reasoning paper is probably the most detailed paper on a hybrid reasoning thing, and then DeepSeek R1 is still the canonical recipe on a reasoning-only model. Those are very different approaches, and I don't know if one will win out or not. And then there's just a lot of work on data side and RL methods. I think there's a whole list of GRPO complaints out there where the math doesn't make sense for certain things.

对我来说，我看到的每一篇论文总是对 GRPO 有一些修复。人们在做各种变体，这挺酷的，但我也不知道 DeepSeek 会不会推出 R2，然后用下一个东西惊艳所有人。

To me, every paper I see come out always has some fix to GRPO. It's kind of cool that people are taking variations on it, but also I don't know if DeepSeek is going to come out with R2 and just blow away everyone with whatever is next.

是的，我绝对不认为算法往往是最重要的。我在 AI Engineer World Fair 的演讲中说过，有点讽刺：如何训练推理模型？你拿到一个初始数据集，逐步改进它，一直做到时间不够或性能开始提升，然后你尝试所有论文中的各种开关，或者对所有这些算法变化做一大堆二元测试，再做个网格搜索，看看什么有效。

Yeah, I definitely don't think the algorithm tends to be the most important thing. I think I had this in my AI Engineer World Fair talk, which is kind of a snarky take: how do you train a reasoning model? You get a starting dataset, you incrementally improve the dataset, you do that until you're running out of time or your performance starts going up, and then you try all these switches from all the papers, or you do a whole bunch of binary tests of all these various algorithmic changes, and you do a grid search and see what works.

坦白说，这就是为什么 GRPO 刚出来时我不屑一顾，因为它被宣传为效率方面的东西。

Candidly, that's why I dismissed GRPO when it first came out, because it was sold as an efficiency thing.

是的。我当时想，‘好吧，行。’但我已经被训练得不在乎效率了，因为那只是资源问题。

Yeah. And I was like, 'Okay, fine.' But I've been trained to not care about efficiency because it's just a matter of resources.

是的。GRPO 的优势估计非常适合可验证奖励，对吧？但另一件事有点无形：它在基础设施类型的论证上效果更好。当它用于 DeepSeek Math 时，远在 RLVR 阶段之前，它确实是基于这一点来推广的。

Yeah. The GRPO advantage estimate is very well suited to verifiable rewards, right? But the other thing is kind of an intangible: it works better on the infrastructure type argument. And when it came out for DeepSeek Math, which is well before the RLVR phase, it was really marketed on that.

当你谈到混合模型时，你如何将其与 OpenAI 所说的他们想摆脱模型选择器、只用一个统一界面相协调？你觉得他们感到压力要路由到正确的东西，还是认为有其他原因？

When you talk about hybrid models, how do you reconcile that with OpenAI saying they want to move away from the model selector to just having a unified interface? Do you feel like they feel pressure to route them to the right thing, or do you think there's something else?

我认为 OpenAI 想要一个知道问题有多难的模型。我认为这必须是大多数从事推理工作的人的北极星：模型会花适当数量的 token 在上面。如果你看算力层面的讨论，那就是推理时 Scaling 意味着什么。

I would think that OpenAI wants to have a model that knows how hard the problem is. I think that has to be the north star for most people working on reasoning: the model will just spend the right amount of tokens on it. And if you look at a compute-level discussion, it's about what inference time scaling means.

我认为在很多方面，混合推理者可能只会被淘汰，除了小众应用，因为质量比推理 token 少 100 倍重要得多。你只需在算力上花钱，而且会越来越好。我觉得就像 Jensen 在最近的采访中说的：一切都会成为推理模型，因为会变得非常便宜而且更好。所以混合推理有点奇怪。

I think in plenty of ways, hybrid reasoners might just be aged out except for niche applications, because quality is so much more important than having 100x fewer inference tokens. You just pay for it in compute, and that'll get better. I think it's like what Jensen said in his most recent interview: everything's going to be a reasoning model because it's going to get so cheap and they're better. So the hybrid reasoning thing is a little bit weird.

我总是会开启推理，除非是一个非常愚蠢的查询，比如‘这是什么东西？’所以两年后大概就是这样。我认为 O3 也在我们身上烧钱：它为我搜索 80 个网站来问这是什么论文。那是很多 token，但方向上看，如果那有效，就会成为默认。

I always just will turn reasoning on unless it's a really silly query, like 'what is this thing?' So in two years that kind of tracks. I think O3 is also just burning money on us: it searches 80 websites for me asking what paper it is. That's a lot of tokens, but it seems directionally like if that's the thing that works, that'll be the default.

至少在所有这些高端领域，我们交谈的人——无论是编程还是非常高端的知识经济——都认为价值在那里。

At least in all of these high-end things, the people we talk to—whether it's coding or very high-end information economy—think the value is there.

我想深入探讨一下你似乎经常回到的一个话题：你似乎断言 O3 做了非常不同的事情，大量使用搜索，比基本上其他所有人都多。

I wanted to double-click on something that you seem to be coming back to a lot: you seem to assert that O3 does something very different by using search a lot, much more than basically everyone else.

是的。

Yeah.

现在所有模型都自带搜索引擎了吗？这是必备的吗？

Do all models come with a search engine now? Is that a must-have?

这取决于你的用例。如果你在做通用信息检索或理解，是的。

It depends on your use case. If you're doing general information retrieval or understanding, yeah.

有一些旧论文我们可以尝试找到链接。我想 Sam 提到过：DeepMind 的 RETRO 论文和其他架构，人们又开始讨论了。就像你有一个非常小的模型，有非常大的上下文窗口和非常大的检索存储。我不是那种会押注 Transformer 架构和解决长上下文问题的人，但这些是人们重新提出的想法：搜索更好。你看看推理模型的所有评估，一个趋势是简单 QA 数字都在下降。DeepSeek R1 到新 R1 下降了。Quen 2.5 到 Quen 3 的简单 QA 也下降了，至少在没有工具评估时是这样。简单 QA 被认为是一个非常好、数值上相当稳健的长尾知识评估。所有这些原始模型都在下降，但这使得长尾信息的搜索行为更有意义。

There are old papers that we could try to find the links. I think Sam was talking about it: there's this RETRO paper from DeepMind and other architectures that people have been pulling into the discussion again. It's like you have a very small model with a very big context length and a very big retrieval store. I'm not one to bet against the Transformer architecture and just figuring out long context, but those are ideas that people are bringing back: search is better. You look at all the evals from reasoning models, and one of the trends is that simple QA numbers all drop. DeepSeek R1 to the new R1 goes down. Quen 2.5 to Quen 3 simple QA goes down, at least when you're evaluating these without tools. Simple QA is considered a very nice, fairly numerically robust long-tail knowledge evaluation. All these raw models are going down, but it just makes long-tail information search behavior make a lot more sense.

好的，对此的反驳——我也经历过这个旅程——是：‘哦，为什么不做一个什么都不知道只会搜索的模型？你可以随时搜索你想学的东西。’但问题是你需要知道搜索词是什么。你需要一些基础智能来让这一切工作。

Okay, the counter argument for this—I have been through this journey too—is like, 'Oh, why don't you make a model that doesn't know anything but search? You can search up anything you want to learn just in time.' But the problem is you need to know what the search terms are. You need some baseline intelligence to make all of this work.

是的，说得好。我认为这很重要，因为有一种论点认为语言模型会永久成为在线语言模型，但并没有被大力追求。Perplexity 是最早让我注意到这一点的：他们说我们会把搜索引擎附加到语言模型上，这就是你现在得到的。我认为越来越多的人开始将其作为默认服务的一部分。Gemini 也有搜索接地功能。我的意思是，人们说这是 Anthropic 的一大限制，因为它使用 Brave Search，返回的 SEO 垃圾比……

Yeah, that's a good way to put it. I think it's important because there's this thesis of LMs becoming just online LMs permanently, and it hasn't been super pursued. Perplexity was one of the first to put it on my radar: they said we'll attach the search engine to the LM, and that's what you get now. I think more and more people are starting to offer it as part of their default services. Gemini has a search grounding thing as well. I mean, it's what people say is a big limitation of Anthropic because it uses Brave Search, which returns a bunch more SEO slop than...

这有证据吗？因为我不知道。我以为他们有自己的索引。

Is that proven? Because I don't know. I thought they had their own index.

好吧，我没有仔细看过，所以我在处理谣言。

Okay, so I haven't done a detailed look, so I'm dealing with rumors.

我认为他们最终都会做自己的索引，这是其中之一。就像谷歌应该再次拥有优势，但谁知道呢。

I think they'll all end up doing their own index, and it's one of these things. It's like Google should have an advantage again, but who knows if they do.

我在文章里也暗示过，就像 Hamish 尝试设置的那样，同一个来自 RLVR 的学生在玩搜索和 RL 模型。让模型使用工具很容易，只要你提示它，但让 RL 模型学会工具是有用的却非常难。这就是为什么会出现这种情况：80 次工具使用失败后它仍然能成功，或者它停下来，在第 81 次成功，这只是一个 RL 行为，感觉像是从一种很好的让模型学会使用工具的方式中涌现出来的。你不能通过 SFT 让这个模型做到这一点。感觉他们就是设置好了环境，对吧？这融入了他们做的深度研究这类工作，他们把问题分解成这些子 RL 任务，然后让它去做这件事。

I also hinted at this in my post, but it's like Hamish had tried to set this up, the same student from RLVR playing with like search and RL model. And it's very easy to get the model to do tools if you prompt it to, but it's very hard to get the RL model to learn that the tool is useful. And that's why it's to go through these things where it's like 80 failed tool uses and it still gets it or like it stops or it gets it on the 81st, okay, is just a RL behavior that feels emergent from having a very nice way of like getting the model to learn to use the tool. And it's not like you can't SFT this model to do this. It just really feels like they set up the environment, right? And it plugs into this deep research kind of line of work that they did and they broke down the problem into these sub RL tasks and then it kind of lets it do this thing.

有意思。

Interesting.

我不想一直当 OpenAI 的吹鼓手，但我就是一直告诉人们去玩 O3，因为它很奇怪。

I don't want to be an OpenAI shill all the time, but I just think I tell people to play with O3 all the time because it's weird.

它很棒。我想说，你对深度研究团队的工作量估计，据我所知只有三个人在做。是 Issa 和她的两个合作者。我不知道他们在 O3 之上做了多少。我从 OpenAI 得到的每一个迹象都表明，深度研究基本上只是 O3 的一个薄包装。

It's excellent. I would say like the amount of work you're imputing on the deep research team when like as far as I know it's three people did it. It was Issa and like the two other collaborators that she had. I don't know if they did that much on top of O3. Like every indication I've had from OpenAI is that deep research is more or less a thin wrapper over what just O3.

是的，可能就是一两个小东西，他们觉得‘哦，我们可以通过给训练添加这么一点数据就让深度研究工作，然后它就成功了。’

Yeah, it's probably like one or two small things that it they're like, 'Oh, we can make deep research work by adding this small amount of data to the training thing and then it just works.'

我就是这么描述的。

That is how I would describe it.

我是说，那个匿名者 Gwen 是什么？他前几天在 Twitter 上回复了我的 Q star 帖子，他说：‘为什么这一切都错了？’

I mean, what is it, Gwen the anonymous person? He replied to my Q star post on Twitter the other day and he was like, 'Why was this all wrong?'

显然，简单的事情无法 Scaling。有很多复杂性，因为当时 AI 领域还有很多其他令人兴奋的事情，OpenAI 发出了很多让人困惑的东西。但这符合这一点：深度研究是对现有 RL 轨迹的一个小改动，就像 O3 可能他们已经发现搜索会更好，然后我们就说好吧我们可以重新包装这个，这是一个简单的事情却能带来巨大差异，一旦你有了势头，大多数事情都是这样。

And it's obviously like simple things don't scale. There's a lot of complexity because there's a lot of other exciting things in the AI field at the time and OpenAI kind of sends out a lot of things that confuse people. But this would fit into that which is deep research is a minor change from an existing RL trajectory of what was like O3 probably they had already figured out that search was going to be better and then we're like okay we can repackage this and it's a simple thing that makes a big difference and most of the things are like that once you have traction.

我认为，试图在 sigmoid 上取得初始起飞是困难的 Q star 事情，但一旦它像这样，中间很多事情就感觉很明显了，这就是为什么我描述我们为 Elmo 做的一件事。很多只是有动力去做那些感觉有点明显但仍然困难的事情。比如很难获得不同的配方，或者很难让一个完整的推理配方起步。这就像是一个巨大的变化，因为你在 Val 套件上有所有这些惯性，然后你必须弄清楚是分支你的配方还是从头开始，比如我们是否直接采用 open reasoner zero 并从头开始，这完全是另一件头疼的事。让这些超过 5 到 10 人的项目在惯性下动起来很难。但一旦你开始爬山，事情就会变得非常明显。

I think once trying to get the initial takeoff on the sigmoid is the hard Q star thing but then once it's like once it's like this a lot of things in the middle feel obvious which is why I describe one of the things that we work on for Elmo. It's like a lot of it is just having motivation to do things that feel somewhat obvious, but they're still hard. Like it's hard to get different recipes or it's hard to get a full reasoning recipe off the ground. It's just like a huge change because you have all this inertia on this Val suite and then you have to figure out if you branch your recipe or do you start from like do we just take like open reasoner zero and start from scratch which is like it's a whole other headache of things. It's just hard to move these projects that are anywhere above 5 to 10 people with inertia to get stuff done. But then once you're hill climbing, things can seem really obvious.

是的。好的。你涵盖了很多内容。在我下一个问题之前，为了结束 Brave 的事情，我们的朋友 Simon Willis 写了一篇文章说 Anthropic 将 Brave Search 添加为他们产品的一个子处理器。是的。所以事情就是这样来的。至于它被使用的程度，我们不知道。

Yeah. Okay. You covered a lot there. Before my next question, just to close the brave thing, our friend Simon Willis wrote a post that Anthropic added Brave Search as one of the subprocessor in their product. Yes. So that's where the thing came from. Now to what extent it gets used, we don't know.

我们不知道。我只想评论一下他说的一些事情，然后我们再继续你的问题。有一篇关于 Qstar 回顾的非常好的文章。

We don't know. I would just kind of comment on a couple of things that he said and then we'll go on to your question. There's a very good post on just on the retrospective of Qstar.

你有一篇非常好的文章，像是‘O1 是一个 scop 吗？’这暗示了一个问题：如果 O1 是一个 scop，那么现在还有什么可能是 scop？

There's a very good post that you had which was like 'Was O1 a scop?' That does imply the question of if O1 was a scop, what else could be scops now?

是的，肯定有 scop 存在。我的意思是整个推理时间 Scaling 图就是一个 scop。你把这两个东西放在一起，带一个 x 轴，看起来很容易控制。每当你看到一个 x 轴，你就认为它很容易控制。

Yeah, there's definitely scops out there. I mean the whole inference time scaling plot is such a scop. You put these two things next to each other with an x-axis and it just looks like it's easy to control. Whenever you see an x-axis, you think it's easy to control it.

而对于左边的训练，那是训练。是的。训练很有道理。所以如果你没有，即使你去看非常古老的 RL 论文，RL 学习曲线通常是非对数 x 轴，它们看起来像这样。它们看起来像对数或指数上升，然后如果你取其中一个并把它变成对数 x，它是一条直线。所以那一侧就像‘哦好吧，我们以前在 RL 中见过这个’，但推理时间 Scaling 有一个 x 轴，这就是为什么人们觉得‘哦有一个旋钮我可以把搜索调高很多’。是的，这就是滋生所有这些奇怪想法的原因。那篇文章的核心就是他们从训练内部取点，或者存在自然方差，然后你把它们排成一行，如果你排好它们，你就会得到这个漂亮的推理时间 Scaling 行为，现在很多人已经重现了这个推理时间 Scaling 图，现在更清楚了，但当时我明白为什么我认为它是一个旋钮。就像‘哦，他们称之为推理时间 Scaling，你可以控制它。’

Whereas like for training on the left one was training. Yes. And training makes a lot of sense. So if you haven't, even if you go to really old RL papers, RL learning curves are a non-log x-axis usually and they look like this. They look like these like whatever logarithm or exponential rise and then if you take one of these and you make it a log x, it's a straight line. So like that side is like oh okay we've seen this before with RL but with inference time scaling it being an X-axis is why people are like oh there's a knob I can turn search up a lot. Yeah which is like what breeds all these weird ideas. The core of that article is just they're taking points from within training or there's a natural variance and then you line them up and if you line them up then you get this nice inference time scaling behavior which is and now people a lot of people have reproduced this plot on inference time scaling and it's much clearer now but at the time it's like I see why I thought it was a knob. It's like oh it's they called it inference time scaling you control it.

我认为最有趣的，嗯，你的博客里有很多有趣的东西。但有一个很突出，是关于 RL 和工具使用的。你说在我们的 RL 实验中，很容易告诉模型尝试搜索，但如果它用工具没有得到结果，它就会很快停止使用工具。我们能展开说说吗？那么，有没有可能有一个好工具，但模型不知道如何使用，然后它失败了，就停止使用它？有没有可能有一个坏工具，在放弃它之前应该改进？人们应该如何考虑设计工具、改进模型以及在哪里干预？

I think the most interesting, well, you have a lot of interesting things in your blogs. But one that stood out was about RL and tool use. You said that it's easy in our RL experiment to tell the model to try searching, but then if it doesn't get results with the tool, it's going to stop using the tool very rapidly. Can we unpack that? So, can there be a good tool that the model doesn't know how to use and then it kind of fails and then it stops using it? Can there be a bad tool that should be improved before giving up on it? How should people think about designing the tool, improving the model and kind of like where to intervene?

这绝对是我想要研究或已经研究过的事情中较新的一面。我认为特别是在 2026 年，尤其是在开放方面，所有基础设施模型都会迎头赶上很多，我想在这方面更深入，比如更深入的搜索风格的事情，这些非常依赖推理，需要多次调用。

This is definitely on the newer side for my things that I want to work on or have worked on. I think particularly in 2026, I think the especially in the open side, all the infrastructure models will caught up a lot where I want to go deeper on this in terms of like deeper search style things are very inference heavy multiple calls.

回答你的问题，确实可能存在糟糕的工具，模型也可能用错工具。我希望模型具备的不是创造力，而是一种开放性——它不完全清楚能从工具中得到什么，带着这种不确定性去尝试不同方法，这几乎像是经典的强化学习行为。但想想语言模型通常的做法：它们总是非常自信——不是真的自信，而是有明确的路径和方向。而推理词元带来的重大变化就是引入了回溯等概念，这是一种对工具中未知事物的开放性。模型能这样想会很好：‘哦，我试试这个会怎样？能得到什么？’尤其是在开源模型方面，如果人们想用开源模型配合工具，那通常是因为他们有私有数据存储。所以如果你训练一个像 O3 那样擅长推理但基于私有记录的开源模型，这些记录永远不会发送到云端，它就需要思考：‘在放弃之前，我可以先尝试一些东西来了解它。’看看现在的工具使用，它更像代码执行——只是顺序路径的一部分，你有一个计划，如果某一步失败，可能有备用方案。但这并不是通过反复与环境互动来制定计划的迭代过程。这可能需要人们训练模型具备这种能力，比如你可以告诉它：‘你不知道这里面有什么，但答案可能就在里面。’这是个很奇怪的提示，但也许会有帮助。

And to answer your question, there definitely can be bad tools and the model just using them wrong. Something I would want to see in a model is not necessarily creativity but an openness that it doesn't know exactly what it'll get out of all of its tools, and this uncertainty to try a few different things, which almost seems like classical RL behavior. But if you think about what a language model does, they're always very confident—not necessarily confident, but they have a path and a direction in their answer. Whereas a big change in these reasoning tokens is the notion of backtracking and things like that, which is some sort of openness to the tools having unknown things. It seems like a really nice thing for the model to have: 'Oh, what if I try this? What does it get?' Especially on the open model side, if this is going to work where people want to use open models with tools, it's going to be because people have private data stores. So if you train an open model that is a good reasoner like O3 but on private records that will never get sent to the cloud, it needs to be thinking, 'I can try some things with this to get a sense for it before saying I have to give up.' If you look at tool use right now, it seems much more similar to code execution—it's just a part of a sequential path where you have a plan and if it fails at a certain step you might have a backup. But it's not this iterative process of fiddling with the environment to come up with your plan. It's something people probably have to train into these models, like you might just tell it, 'You don't know what is in this, but your answer might be in it.' That's a very odd prompt, but maybe it'll help.

是的。我们采访过 Anthropic 的 Eric Schlans，他在 Claude Code 之前做过云智能体。他提到他们大部分时间都花在工具设计上，然后让模型学习如何使用。你通常——我不确定你在这方面做了多少工作——是把工具逐个加入强化学习过程吗？你觉得这样有帮助，还是最好把所有工具都给模型，让它自己探索？

Yeah. When we had Eric Schlans from Anthropic, who worked on the cloud agent before Claude Code, he mentioned they spent the majority of time on tool design to give to the model, and then you just kind of learn how to use it. Are you usually—well, I don't know how much you worked on this—but are you putting the tools one by one in the RL process? Do you think that helps, or is it better to give all the tools and let the model explore?

我不太确定；我们还没能让这个奏效。我觉得这取决于模型和你的起点。如果起点已经擅长使用工具，它可能泛化得更好。但如果你在做这种奇怪的基座模型强化学习，并且需要足够长的课程，那么你需要一个难度递增的课程——这很明显。所以在这种情况下，当模型解决某些问题变得太困难时，可能会添加工具。这听起来很直观，但在实践中很难管理，因为训练过程中什么自动信号告诉你该添加工具了？

I don't really know; we haven't gotten this to work. I would say it probably depends on the model and your starting point. If your starting point is already good at tools, it can probably generalize more. But if you're doing this weird base model RL and you have to have a curriculum long enough, you're going to need a curriculum of things getting harder—that's pretty obvious. So in that case, tools might get added when things become too hard for it to solve certain questions. That sounds very intuitive but also really hard to manage in practice, because what is your automated signal on your training run that it's time to do that?

这就是为什么电子游戏那么好——它们设计成随着进度解锁内容。但我觉得对于搜索之类的东西，比如你被允许访问一个小型数据存储，或者被允许访问互联网上的所有知识，情况就不同了。

That's why video games are so good because they're designed to unlock things as you progress. But I think with things like search, it's like if you're given access to a small data store or you're given access to all knowledge on the internet.

给 ARC AGI 团队关于 V3 基准测试的一个好建议是：设计一些场景，让语言模型在达到某个阈值后需要学习使用新的执行器。

Good feedback for the ARC AGI people for the V3 benchmark is to have things where the language model needs to learn to use new actuators in the world after a certain threshold.

那就会是 RAGI4 了。

That would be RAGI4 then.

是啊，我不确定。他们正在快速推出新版本。

Yeah, I don't know. They're cranking them out.

他们确实在快速推出。我记得几周后他们有个发布派对。我真的很喜欢玩 RKGI，不知道你试过没有。

They're cranking them out. They're actually doing a launch party I think in a couple weeks. So I'm actually really—it's fun to play RKGI. I don't know if you tried.

哦，我还没试过。

Oh, I haven't.

挺有趣的。这些是智商测试。我以前觉得它们不太相关，但现在随着语言模型在这些测试上显著进步，比较自己的智力反而变得更有意思了。

It's pretty fun. These are IQ tests. I used to think they weren't that relevant, but now that we have a gradient where LLMs are actually significantly climbing them, it's actually more interesting to compare your own intelligence.

我支持无限制条件。

I'm with no gnome on no harnesses.

无限制条件。是的。

No harnesses. Yeah.

是的。我的意思是，限制条件很酷，但它们是一种改变学习动态的障碍。所以作为演示很好，但我觉得核心方向应该是无限制条件。这就像说这些只是归纳偏置，它们不在模型里。没错。但任何只看结果的做法都会污染这个——它变成了不同的任务。我觉得——我在 RKGI 和 Greg 聊过这个——我告诉他：‘有条件和无条件都做，分成两个不同的类别。’

Yeah. I mean, harnesses are cool, but they're a handicap that's changing the learning dynamic substantially. So it's good demos, but I feel like the core thrust has to be no harnesses. I mean, it's always like, is it wrong to say that these are just inductive biases? They're not in the model. Sure. But anything where you're just looking at the results contaminates this—it's a different task. I think doing—or I've talked with Greg about this at RKGI—I told him, 'Do harness and no harness. You just have both different categories.'

就像你试图透明地为前沿实验室设定目标，那就两个都做。我不觉得这会稀释太多。无限制条件显然更难，这样你的基准测试性价比更高。

Just like you're trying to be transparent and build targets for frontier labs, just do both. I don't think it dilutes that much. The no harness is going to obviously be harder, and then you just get more bang for your buck on your benchmark.

嗯，是的。是同一个数据集。

Mhm. Yeah. It's the same data set.

既然我们还在聊工具这个话题，你对多工具强化学习的最新工作做了很好的总结，包括 Loop、Read Tool、Toal 等等。我觉得这个领域现在研究空间很大。我想让你重点说说你最喜欢哪些？你认为人们应该探索什么？

Staying on the topic of tools while we're at it, you had a really good summary of recent work in multi-tool RL, which had like Loop, Read Tool, Toal, and all these other things. I think this is just an area that's super rich for research right now. I just wanted to give you the space to highlight what are your favorites? What do you think that people should explore?

我可以分享一下我的中等野心：一个有趣的研究项目是创建某种竞争性动态或评估，而且它必须比工业界正在做的狭窄得多。我午餐时跟你提过：深度研究，但只针对存档论文。这样你不需要做完整索引，领域有限，你需要想办法衡量它。我认为学者研究学术工具很好，因为他们有很高的领域专业知识，已经知道情况，然后想办法让它要么对用户非常有用（如果足够好的话），要么成为可以攀登的目标。我不知道这是不是即兴头脑风暴：从论文中提取相关作品，只看文本并断开所有链接，然后做一个评估，用存档链接填充数百条相关作品。这是一个有趣的深度研究风格想法——看看你是否能用开源模型配合工具在数据存储上实现它。

I could share what my moderate ambition would be: a fun research project is to create some sort of competitive dynamic or a eval, and it has to be so much narrower than what industry is doing. So I told you this at lunch: Deep Research but only archive papers. So you don't have to do a full index. You have a limited domain. You have to figure out how to measure it or something. I think it's good for academics to work on academic tools because they have very high domain expertise. They already know what's going on, and just figure out how to make that something that is either very useful to users if it's going to be good enough, or something you could climb on. I don't know if this is brainstorming on the fly: take related works out of papers, just look at the text and break all the links, and make an eval which is filling in hundreds of related works with archive links. That's a fun Deep Research style idea—see if you could do it with open models on a data store with tools.

AI2 对此进行了大量讨论：如果你现在想在 AI 领域产生影响，作为学者，你必须从论文升级到制品——模型、数据集、评估数据集。评估更容易让人产生影响。接下来是人们实际使用的东西。在 AI2，尤其是现在研究不同类型信息智能体的 Semantic Scholar 团队，另一个我保持距离的事情是：我们能让开放模型更好地完成那个网站功能吗？你能做出人们真正关心的东西吗？如果你有实际用户，那会是一个更高的影响力层次。这对学者和小机构来说很难做到。

AI2 has gone through a lot of discussions about this: if you're trying to have impact in AI right now as an academic, you have to level up from papers to artifacts—models, datasets, eval datasets. Evals are easier for people to have impact on. The next thing is what people actually use. In AI2, especially the Semantic Scholar team now working on information agents of different types, another thing I'm distancing myself from—I don't have all the names—is: can we make open models do that site thing better? Can you make something that people actually care about? That's a whole level of impact that's much higher if you have actual users. It's hard for academics and small institutions to do that.

但如果你在研究智能体，dog feeding 是可行的。比如：我们能给自己做一个好用的 Slack 摘要机器人吗？就是让这些智能体变得非常易处理。这是一个方向。另一个方向就是带着工具在人类最后的考试上爬山。我认为作为学者我们不太可能赢得最先进的数字，因为他们会开始每个查询花费数百万个 token。那是大量的算力消耗。在翻牌等价物上击败它会非常困难。

But if you're working on agents, dog feeding is viable. It's like: can we make ourselves a good Slack summary bot that we like, or something? Just making these agents really tractable. That's one direction. The other direction is just hill climb on humanity's last exam with tools. I think it's unlikely that we're going to win as an academic and state-of-the-art number, because they're going to start spending millions of tokens per query. It's a lot of compute burn. Beating that on the flop equivalents is going to be so hard.

非结构化的想法我大多是这样的：好吧，我会处理这个。我在建模和我所谓的技能层面还有更多东西要弄清楚——就是如何做推理来引发推理时 Scaling 并获得高评估分数。一旦你知道你能做到，你就可以带着你的知识在更具体的领域去做。有技能和技能获取。我认为 Archi 对 AGI 的定义引用了它：就像高效的技能获取。效率。因为我用三个词描述过，对吧？

Unstructured thoughts is something I'm mostly like: okay, I'll get to this. I have more things to figure out on the modeling and what I call the skills level—which is just how do you do reasoning to induce inference time scaling and get high eval numbers. Once you know you can do that, you can take your knowledge with you to do it in more specific domains. There's skill and there's skill acquisition. I think the Archi definition of AGI quoted it: it's like efficient skill acquisition. Efficiency. Because I described it as three words, right?

是的。

Yeah.

你在最近的演讲中强调了技能。你想为人们重申一下这个论点吗？

Your emphasis on skills in your recent talks that you've done. Do you want to sort of reiterate that thesis for people to pick up on?

是的。所以我一直在思考如何领先于 OpenAI 等公司正在做的事情。可能现在，如果不在他们的模型里，加上所有智能体，规划似乎是一个非常关键的任务。所以问题是：你如何为需要训练到推理模型中的不同类型的事情提出一个分类法，以便在它成为瓶颈时使用？我提出了四个。基础的是技能——我认为我们已经在 o1 和 R1 上做到了：你做了很多强化学习，你展示了推理时 Scaling 有效，你得到了非常高的基准分数。接下来的三个是接下来会发生什么，其中大部分围绕规划。我列表中的第三和第四是抽象和策略——这是为了避免使用规划这个词，因为人们已经用了很多。策略是模型应该走的方向，以及技术上的计划步骤。抽象是如何将其分解成它实际能解决的事情。最后一个是校准——就是不要浪费算力，知道何时放弃并向用户提问，因为过度思考显然是个问题。通过使用更多的推理时 Scaling 来让评估分数更高很容易，但最终那不是人们想要的模型。他们想要一个更智能的训练机制，让模型在训练中成比例地变得更好，而不是——有很多关于过度思考的论文。我认为 OpenAI 想要它，因为他们要付 GPU 账单。如果 o3 对很多人无限循环，那不好。

Yeah. So, I've been thinking mostly about trying to get ahead of what OpenAI etc. are doing. Probably now, if it's not in their models and with all the agents, it seems that planning is a very critical task. So it's kind of: how do you come up with a taxonomy for different types of things you need to train into reasoning models for when it'll be a bottleneck? I came up with four. The foundational one was skills—which I would say we have already done with o1 and R1: you do a lot of RL, you show that inference time scaling works, and you get really high benchmark numbers. The next three are kind of what comes next, and most of them are around planning. What I had is three and four on my list were abstraction and strategy—which is trying to not use planning because planning is a word that people already use a lot. Strategy would be the direction the model should go in and technically what are the steps of its plan. Abstraction is how does it break it down into things it can actually solve. The fourth last thing is calibration—which is just not wasting compute and knowing when to give up and ask the user things, because overthinking is obviously a problem. It's easy to keep getting your eval scores to go higher by using more inference time scaling, but eventually that's not what people want in their models. They want a smarter training regime where the model is actually getting proportionately better for its training, and not—there are a lot of papers on overthinking and stuff like this. I think OpenAI wants it because they have to foot the GPU bill. If o3 just infinite loops itself for a bunch of people, that's not good.

它真的会吗？

Does it actually?

我不知道。但可能。我是说，这些推理方法肯定能让模型变得不稳定，只是瞎说。所以就像——

I don't know. But it might. I mean, these reasoning methods definitely can make the models just kind of unstable and just yap. So it's like—

但这也是 GPT-5 的想法：你如何得到一个模型，它只是把问题路由到正确的地方——不一定是路由器——但知道它是否需要做一个计划，或者它可以直接回答。如果你看 DeepSeek-R1，问它一个难的数学问题，它不会说‘这是我的攻击计划’。它直接开始。拥有一个知道何时说‘好的，这是我的攻击计划。我可能需要给自己做一个记忆存储。我可能需要为这个查询采用云代码方法。我要构建一个记忆存储，启动一些并行搜索器，然后回来’的模型。可以想象，这都是你可以训练到模型中的东西，因为搜索或并行模型可以像工具一样。在这种情况下，简单的描述方式是我们有类似思考 token 和答案 token 的东西。模型应该能够在思考或使用工具之前选择性地拥有计划 token。就像‘好的，这是基本要求。我需要做这些事情。’这类任务会有难有易。这似乎比一些遥远的 AI 想法更容易处理。就像：语言模型可以写出一个好的计划，只需要被要求这样做。我敢打赌 Claude Code 和 Deep Research 正在这样做：你得到一个用户提示，首先模型——Claude Code 中有一个计划工具——首先它们分解它。这是它们训练到模型中的东西。我不认为 DeepSeek 内置了它，但它可能能做到。只是思考这个接口：如果模型需要能够端到端地自己完成任务，它能做那种事情吗？

But it's also the GPT-5 idea: how do you get a model that just routes the question to the right—not necessarily a router—but just knows if it needs to do a plan or if it can just answer. If you look at DeepSeek-R1 and you ask it a hard math question, it's not like 'here's my plan of attack.' It just starts. Having a model that knows when to be like, 'okay, here's my plan of attack. I might need to make myself a memory store. I might need to take a Claude Code approach for this query. I'm going to build a memory store and spin up some parallel searchers and then come back.' Conceivably, this is all something you can train into a model, because the searches or the parallel models could be like tools. In that case, the simple way to describe it is we have something like thinking tokens and then answer tokens. And the model should be able to optionally have plan tokens before thinking or before using tools. It's like, 'okay, here are the table stakes. I need to do these things.' These sorts of tasks will be harder versus easier. It seems more tractable than some far-out ideas for AI. It's like: a language model can write a good plan, and it just needs to be asked to do so. I would bet that Claude Code and Deep Research are doing this: you get a user prompt and first the model is like—there's a plan tool in Claude Code—and first they break it down. That is something they've trained into the models. I don't think DeepSeek has it built in, but it probably could do it. Just thinking about that interface between if the model needs to be able to do the task end to end on its own, can it do that sort of thing?

我认为我在调和这种方法与‘无束缚’概念时的挑战是，我认为很多人——尤其是工程师——想要建模的方式是计划和记忆是工具，没有特殊的计划 token，没有特殊的记忆 token。它只是上下文，或者任何专门用于规划的东西，因为那样你可以分叉到其他智能体进行工具调用等。所以它不必是顺序的。但我只是觉得：这是一个分叉路口吗？我们是否必须在这里做出真正的选择，是把事情外包给工具，还是保持在模型 token 内部？我不认为这是一个主观差异。我认为规划的想法主要是为了说明人们不会免费得到东西，规划的改进可能有点平凡。就像：我们提示 Claude，它的计划在这方面很糟糕。让我们提供一些数据，让它的计划更详细或分解成更多步骤，这样它们更容易做到。

I think my challenge with reconciling this approach with the 'no harnesses' thing is that I think a lot of the way that people—especially engineers—want to model it is that the plans and the memories are tools, and there are no special plan tokens, no special memory tokens. It's just context, or whatever, specifically for planning because then you can do fan out to other agents for tool calls and stuff. So it doesn't have to be sequential. But I'm just like: is this a fork in the road? Do we have to make a real choice here as to whether we outsource things to tools or keep it native within the model's tokens? I don't think it's a subjective difference. I think mostly the planning idea is to make the point that people don't get things for free, and the planning improvements might be kind of mundane. It's just like: we were prompting Claude and its plans were bad in this way. Let's give some data where its plans are more detailed or break things down into more steps so that it's easier for them to do it.

是的，因为它实际上在一个黑箱里。所以如果没有针对性，就不清楚性能会怎样。

Yeah, because it's in a black box effectively. So if it hasn't been targeted, it's unclear what the performance will be.

或者在开放模型方面，可能只是为不同部分使用不同模型的想法。那么你实际上是在训练一个模型只擅长规划，这需要你提供数据，而且你只在那一个部分使用那个模型。感觉计划更具可重用性，也许不应该每次都生成？我觉得特别是在编码中，对于某些任务集，你希望有相似类型的计划。

Or on the open model side, it might just be the idea of having different models for different parts. Then you're really training a model to just be good at planning, and that's data you need to come up with, and you only use that model for that one part. Does it feel like plans are much more reusable and should maybe not be generated every time? I feel like especially in coding for certain sets of tasks, you want to have similar types of plans.

所以也许让模型每次都重新生成计划不是正确的方式。应该几乎像计划蓝图作为工具，然后模型填充它。你认为平衡点在哪里？

So maybe it's not the right way to ask the model to regenerate a plan every time. There should almost be like plan blueprints as tools, and then the model fills it in. Where do you think the balance should be?

我认为它们是合理的。计划显然是一个中间目标。在这种规划层面上很可能存在失败。同样的情况也适用于流行的评分标准。所谓评分标准技术中，很多做法是：你有一个提示，然后让语言模型为该提示生成一个评分标准，即几个需要做对的具体事项。这在概念上非常类似于为每个任务制定计划。无论是评分，你都会有不同的抽象类型，而不是执行。但人们看到的是，相对于效果而言，直接生成它更便宜。

I think they're reasonable. A plan is obviously an intermediate goal. It seems likely that there are failures on this kind of planning level. The same goes for these rubrics that are popular. A lot of the technique for so-called rubric things is you have a prompt and you have a language model generate a rubric for that prompt, which is a few specific things that need to get right. That's conceptually very similar to making a plan for every task. Whether it's grading, you're going to have a different type of abstraction than executing. But what people are seeing is that it's cheaper relative to the effectiveness to just generate it.

所以计划不会太长，可能没有那么多 token。所以可能就像，好吧，我们这样做。把它放进我的分类法可能有点过度推销了，它只需要是一个提示，你只需要确保你的模型在那个提示阶段不会太奇怪。顺便说一句，我觉得你的分类法非常有用。技能、校准、策略、抽象。我觉得抽象可能是最被低估或最难解决的。你介绍它的方式和你博客文章里写的不一样。你说它基本上就是不要过度思考。那是校准。

So plans are not super long and they probably don't have that many tokens. So it's probably just like, okay, we do this. Putting it in my taxonomy might be overselling it where it just needs to be a prompt and you just need to make sure your model's not too weird at that prompting stage. I think your taxonomy is super useful by the way. Skills, calibration, strategy, abstraction. I feel like maybe abstraction might be the most underrated one or hardest to solve. The way you introduced it was different than how you wrote in your blog post. You said it was basically not to overthink. That's calibration.

抽象是关于分解事物。

Abstraction is about breaking things down.

是的，我认为策略和抽象在最困难的任务上最有意义，这些任务我们不知道模型是否能完成，对吧？

Yeah, I think both strategy and abstraction make the most sense on the hardest tasks that we don't know if the model can do them, right?

所以如果你给一个模型分配一个你不知道它是否能实现的任务，策略非常重要，因为它需要非常具体和狭窄。而如果它做的是普通代码或深度研究，计划实际上没那么有趣。但当你处于前沿，比如实现一个 GPU 相关的东西，你可以接受 OpenAI 的说法：帮我在我们复杂的分布式 GPU 系统中实现这个研究想法。这是一个对人类和 AI 都很难的任务，想出正确的计划来调试并完成它是一条非常狭窄的路径。所以策略非常重要：它是否从某些测试开始，以及它如何实际构建出复杂性？很明显我需要为这个想出更好的例子，但我认为随着你推进，更自然地会发现只有少数几个计划能真正完成它，然后当你的任务变得非常大时，抽象就很重要了。

So if you're assigning a task to a model that you don't know if it can implement it, the strategy is very important because it needs to be very specific and narrow. Where if it's doing mundane code or deep research, the plan is actually not that interesting. But when you're at the frontier, like implementing a GPU thing, you could buy into the OpenAI narrative: help me implement this research idea in our complex distributed GPU thing. This is a task that's hard for a human and for an AI to come up with the right plan to debug and do this is a very narrow path. So the strategy is pretty important: does it start with certain tests and how does it actually build this out to complexity? It's obvious that I need to come up with more better examples for this, but I think as you push it, it's more natural to see that there are only a few plans that actually get it done, and then abstraction is just important as your task becomes so big.

这几乎像是一个提示工程的事情。

It's like a prompt engineering thing almost.

是的。而且你只能生成 10 万个 token。你需要确保模型分解它，这样它就不会在自己下面产生大量无限进程。我确实同意抽象是一个有趣的问题，特别是当你开始考虑这些模型可以调用其他模型来执行子任务，或者可以通过多次搜索或更多算力并行化的部分。我认为这有点融入抽象，即你如何接近问题的某个核心。我肯定说我没有构建这个的经验，只是感觉如果你要想象 AI 做最困难的软件或其他任务，这是人类非常擅长的。所以就像你如何在 10 周内提出一个研究计划？有很多关于如何优先进行哪些实验的问题。有很多归纳偏差进入其中，我认为语言模型目前还做不好。可能记忆在那里会有帮助。所以你可以直接得到我们在现实生活中做这件事的方式：我们积累经验。

Yeah. And you only have 100k tokens you can generate. You need to make sure the model breaks it down so it's not just spawning a ton of infinite processes under itself. I do agree that abstraction is an interesting one, especially when you start to think about these models that could call in other models to do subtasks for it, or parts that can be parallelized with multiple searches or just more compute. I think that kind of folds into abstraction, which is just how do you approach a certain nugget of the problem. I definitely say I don't have experience building this, it just feels like if you're going to visualize AI doing the hardest software or other tasks, it's something that humans are very good at. So it's like how do you come up with a research plan in 10 weeks? There's a lot of how do you prioritize which experiments to do. There are a lot of inductive biases that go into that that I don't think a language model would do well at right now. Probably memory would be helpful there. So you can just get the way we do this in real life: we accumulate experience.

是的，我确实想深入探讨一下并行性。有一个关于 o1 和那种 Q* 想法的案例。有一个案例在某种意义上有点被过度炒作，但现在随着 o1 Pro 和 Deep Think 又回来了。理论是，至少，如果我错了请纠正我，基本上他们运行 o1 八次，然后有一个奖励模型，对它进行评分，然后给你八个中最好的。

Yeah, one thing I did want to dive into was parallelism in general. There's one case with o1 and the sort of Q* ideas. There was one case where it was sort of overhyped in some sense, but now it's coming back with o1 Pro and Deep Think. The theory is, at least, correct me if I'm wrong, basically they run o1 eight times and then they have a reward model, rate it, and then give you the best of the eight.

类似那样。

Something like that.

类似那样。Deep Think 也一样。除此之外我们不知道任何细节。我认为有很多人在探索这一点，至少在信息提供者方面，如何并行化搜索和规划等等。我担心对它过度炒作。我认为它在逻辑上很有道理，而且这是那种 MCTS 也很有道理但我们被愚弄的事情之一。嗯，我不认为我们使用并行算力来搜索低概率 token，而是用它来获得鲁棒性。如果你喜欢 o1 Pro，它非常好，因为它有非常可预测的深度，即使在有时模型会失败的 niche 主题上也是如此。

Something like that. Deep Think also the same. We don't know any details beyond that. I think there are a lot of people exploring that, at least on the info provider side, of how do we parallelize search and planning and all that. I'm worried about getting too hyped about it. I think it makes a lot of logical sense, and this is one of those things where MCTS also made a lot of logical sense and we were fooled. Well, I don't think we're using parallel compute in a way to search over low probability tokens, but for using it to get robustness. If you like o1 Pro, it was so nice because it just had a very predictable depth to it, even on niche topics where sometimes models just fail out.

是的，你有一些数字，它从大约 10% 上升到大约 95% 之类的。

Yeah, you had some numbers that it went from like 10 to like 95% or something.

我不记得确切的数字，但感觉就是这样。感觉不像你打开 o3 Pro 就能让找到某个 niche 信息的可能性提高 10 倍。也许可能性会大一点，但我们并没有得到那种搜索式的概念，即在我们的树中获得更多广度或深度。所以我认为它有价值，我们想用这种并行性来处理我们生成的最重要的 token，或者像，好吧我知道这部分很关键，让我们多花一点时间让这些 token 更好。但这并不是变革性的。在变革性方面可能有趣的部分是，如果你能得到更好的验证器。所以我认为验证器改变了推理时间 Scaling 的斜率。你在推理时花费更多的 token。

I don't remember the exact numbers, but that's what it feels like. It doesn't feel like you turn on o3 Pro to make it 10 times more likely to find some niche piece of information. Maybe it'll be a bit more likely, but we're not getting that type of searchy notion of getting more breadth or depth into our tree. So I think there's value to it where we want to use this parallelism on either the most important tokens we're generating, or like okay I know this part is crucial, let's just spend a bit more so that those tokens are better. But it's not a transformative thing. The part that's potentially interesting on the transformative side is if you can get much better verifiers. So I think verifiers change the slope of inference time scaling. You spend more tokens at inference.

如果你在做并行计算，验证器越好，就越能提取出罕见的结果。所以现在，如果我们的验证器只擅长人类偏好，我们不需要大幅提升它。但如果我们做真正多样化的生成，并且验证器更好，它就会变得更好。我认为你可以看看奖励模型和预言机之间的极端情况：预言机是搜索越多，最终越有效，所以斜率很好；但奖励模型，至少在做这种偏好类事情时，信号是有上限的，所以斜率很小，而且收益递减。所以我确实认为，如果你能用更有趣的验证器来填补这个空缺，并行计算可能还有更多潜力，但以我目前的看法，它并没有那么大的变革性。它更像是：如果我们的任务耗时很长，并行智能体作为吞吐量引擎更有意义，而不是作为峰值性能引擎。

The better verifier you have, if you're doing parallel, it can extract a rare occurrence. So right now, if our verifiers are only good at human preference, we don't need to crank that up very much. But if we are doing really diverse generations and your verifier is better, it will get better. I think you could look at the extreme between a reward model and an oracle, where the oracle is: the more you search, eventually it works. So the slope is good, but a reward model is like there's really a capped signal, at least if you're doing this preference type of thing. So the slope is pretty minor and it has diminishing returns. So I do think that if you could fill that with more interesting verifiers, there's potentially more to get out of parallel compute, but I don't think it is as transformative right now on my outlook. It's more like parallel agents makes more sense as a throughput engine if our tasks are taking a long time, rather than as a peak performance engine.

好的，是的。

Okay, yeah.

这跟整个智能体与模型的对比很吻合：智能体更注重把事情做成、稳健和快速，而模型是一次生成：你能答对吗？

Which kind of fits with the whole agent versus model thing, where agents are much more about getting it done at all, being robust and being fast, whereas this model is one generation: can you get the answer right?

是的。我们再多花点时间在这上面，然后可以继续。我的反驳或反对意见是，这是一种将假设的未来模型拉到现在，然后从中蒸馏的方法。

Yeah. We'll spend a little bit more time on this and happy to move on. My pushback or counter to this is that it's a way to pull forward a hypothetical future model that you can then distill from.

是的。

Yeah.

这很好。

Which is nice.

嗯，我打赌人们肯定会用这些来生成合成数据。合成数据的边际收益总是很高。或者就像 Amanda Askell 会说的，更好的提示词会让你感觉拥有了下一代模型，而大多数人并不在提示词上下功夫。

Well, I bet people will surely use these for synthetic data. The marginal gain on synthetic data is always very high. Or just like Amanda Askell will say, better prompting will effectively make it seem like you have the next generation model, where most people don't put effort into their prompts.

天哪。好吧。

Oh my god. Okay.

或者她在一次 Anthropic 采访中说过类似的话，就是如果你真的能弄清楚如何让模型进入某些状态。

Or she had said something along those lines in one of her Anthropic interviews, which is just like if you can really figure out how to get into the certain states of the model.

是的。是的。好吧，不管怎样，这就是我为什么认为这值得做的理由。而且，你知道，我想写一个科幻故事，关于量子模型的世界，在那里我们可以廉价地探索多个宇宙，然后拉出正确的那一个。那会有效。这听起来太科幻了，但我觉得在一个我们能足够好地控制量子计算来探索并扩展它的世界里，那可能会很酷。并行计算也可能为有趣的创新提供基础。比如，我不知道，用扩散语言模型做并行计算，一次性生成所有 token，这意味着什么？它会显著改变某种应用吗？我真的不知道。我认为如果扩散语言模型有效，那会很有趣。你对推理时 Scaling 有更多控制。我是说，Gemini 有一个。很难弄清楚它改变了什么。但一旦我们有了所有这些旋钮，我希望它能帮助建立一些有趣的创新，因为并行计算是新的，架构可以改变。我们拭目以待。

Yeah. Yeah. Well, anyway, that's my pitch for why this is worth doing at all. And, you know, I have a science fiction story that I want to write about quantum models in a world where we could explore cheaply multiple universes, then sort of pull forward the right one. That would work. This sounds too science fictiony, but I feel like in a world where we could control quantum computing well enough to explore this and scale it up enough, it could be kind of cool. It also could be that parallel compute is grounds for interesting types of innovation. Like, I don't know, what does it mean to have parallel compute with diffusion language models that generate all their tokens at once? Does that meaningfully change some sort of application? I don't really know. I think it would be fun if the diffusion language model works. You have much more control over inference time scaling. I mean, Gemini has one. It's hard to sus out what it changes. But once we have all these knobs, I'm hopeful that it helps build some interesting types of innovation, because the parallel stuff is new, architectures can change. We'll see.

我一直在用 Codex Besan 的东西，我觉得大多数生成结果彼此之间只有 5% 的差异。

I've been using the Codex Besan thing and I feel like most of the generations are, you know, 5% different from each other.

因为你用 Ruby。

Because you use Ruby.

不，不，不。我有个 JavaScript 的。我有个 JavaScript 的。所以，我应该擅长那个。我不知道是不是 RL 编码的方式。我注意到一件事，这些模型在缺少变量时总想用 if 语句，这样运行时就不会失败。我觉得这对我来说就是 RL 的一个症状。

No, no, no. I had a JavaScript one. I have a JavaScript one. So, I should be good at that. I don't know if it's just how the RL encoding works. One thing that I've noticed, these models always want to do if statements when there's a missing variable so that it doesn't fail when it runs. And I feel like that to me is just a symptom of the RL.

是的，代码很糟糕。不，你不应该那样写代码。如果缺少变量，它不应该静默失败，而应该直接报错。是的。

Yeah, the code is terrible. No, you should not write code like that. It shouldn't silently fail if there's a missing variable. It should just raise an error. Yeah.

但我觉得 RL 正把代码推向这个方向。然后所有的生成结果都有相同的模式。你知道，我生成四个东西，它们都用 if 语句，只是放在不同的地方。

But I feel like the RL is pushing the code in this direction. And then all the generations have the same pattern. You know, I generate four things, all of them use the if statement, just in different pieces.

是的，那是我们肯定会克服的。这就像实验室在用性能的巨大提升换取可用性的小损害。

Yeah, that was something we will definitely get over. That's just like the labs are trading off massive gains in performance for small detriments in usability.

然后就是，你发布那个模型吗？是的，你直接发布，以后再处理。但我相信他们能修复。我相信那是可以修复的。我觉得对我来说问题是：你谈到有时在事情的某些部分有收益，但在整个轨迹上没有。你觉得这些是例子吗，还是你觉得随着我们变得更好，如果我们做更长的轨迹，不只是写这段代码，还要考虑以后如何维护以及如何运行，那会修复它？我很难理解。

And it's like, do you ship that model? Yeah, you just ship it and deal with it later. But I'm sure they can fix it. I'm sure that's a fixable thing. I think like to me that's the question: you talk about how you have gains in pieces of the thing but not in the full trajectory sometimes. Do you feel like these are examples of that, or do you feel like as we get better, if we did a longer trajectory where instead of just writing this piece of code you have to think about how you're going to maintain it later and how it's going to run, that's going to fix it? It's hard for me to grasp.

是的，软件方面不容易，因为可维护性几乎又感觉像是一个人类偏好类型的问题，有人看到它可能会说，'是的，那没那么好。'

Yeah, the software stuff is not easy because it's almost like maintainability almost feels like a human preference type issue again, where somebody could look at it and be like, 'Yeah, that's not as good.'

但添加启发式和权衡似乎非常混乱。

But adding heuristics and trade-offs seems very messy.

是的。

Yeah.

所以也许是的。我不知道。还有很多需要深入挖掘。我的意思是，这就是 Anthropic 说他们在做的，实际的前沿是什么？比如他们说他们只专注于代码，那实际上意味着什么？

So maybe it is. I don't know. There's a lot more to dig into. I mean, this is what Anthropic says they're doing, and what are the actual frontiers in making? Like they said they're working on code only, and what does that actually mean?

其中很多将是设计权衡，以及模型有多少自主权，相对于这些来自更长时间训练的潜在副作用，我们不知道如何消除。我的意思是，那绝对可能是一种容易消除的行为，比如它可能只是痴迷于某种代码格式，当你重新审视时它就会失败，即使就像每个人都见过它只是绕过测试用例。我认为它们会比那更微妙一些，但也可能非常简单。

A bunch of it is going to be design trade-offs and how much autonomy the model has versus these potential side effects from training longer that we don't know how to get rid of. I mean, that definitely could be the sort of behavior that is simple to remove, where it might just be obsessed with some code format that fails when you revisit it, even if it's like everyone has seen it with just bypassing test cases. I think they'll be a bit more nuanced than that, but they could probably be super simple.

这个话题对我来说有与过度优化相似的语义内容地址，这是你写过的东西。它是具有不同奖励函数的过度优化。

This topic has a similar semantic content address for me as overoptimization, which is something that you've written about. It is overoptimization with a different reward function.

我知道。好的。嗯，我想建立那个联系。我想确认我们在同一个思路上。我只是想再回顾一下你花时间思考过的具体话题。你写道有三种类型的过度优化。第一种是用于控制的 RL。第二种是 RLHF。第三种是 RLVR。它们总是发生。

I know. Okay. Well, I want to make that link. I want to verify that we are thinking on the same wavelength. I just wanted to go over again specific topics on things that you've spent some time thinking about. You write that there are three types of overoptimization. First was RL for control. Second was RLHF. And third is RLVR. They always happen.

显然，强化学习对奖励破解并不陌生，但也许你可以详细说明一下，我们作为行业在学习方面是如何演变的。

Obviously, RL is no stranger to reward hacking, but maybe you could elaborate on how things are evolving in terms of how we're learning as an industry.

是的。所以这三个方面的分解是为了让人们把历史上发生的事情拼凑起来。所有这些过度优化都是因为模型优化器足够强大，可以操纵智能体与环境的关系，或者以对其目标信号有用的方式操纵环境。另外，作为背景，我认为我们目前用语言模型和强化学习所做的是，如果有东西可以提高它的奖励信号，它会选择最容易、最直接的方式来提高那个信号。这就是我在 sycopency 上说的部分内容：这个用于用户反馈的奖励模型可能太明显了，人类就是喜欢点赞，比如人们会按下那个点赞按钮，当有填充的要点时。

Yeah. So that three-things breakdown is for people to put the pieces together for what has happened historically. All of these overoptimizations are just the model optimizer is strong enough where it can manipulate the agent with respect to the environment or manipulate the environment in a way that's useful to its target signal. Also, for context, I think with what we're doing with language models and RL in general is that if there's something that can move its reward signal up, it'll move the easiest thing, the most direct things to move that signal up. So that's part of the story that I said on sycopency which is this reward model for user feedback was probably so obvious that humans just like to like stuff that is like people press that thumbs up button when filled bullet points.

填充的要点。

Filled bullet points.

是的。所有这些事情对模型来说都非常容易提取。所以一旦他们添加了它，模型变化很大，分数上升了很多，强化学习很容易就找到了控制方法。最古老的强化学习环境通常是一个固定的模拟器，没有反馈。所以过度优化表现为非物理和荒谬的行为。比如摩托艇绕圈的例子。还有一个例子来自我作为中间作者的一个项目，实际上是过度优化了 half cheetah，这是 Mujoco 中的一个东西。它没有跑步，而是翻着跟头消失在地平线上，得到了无限分数。这显然不是预期的目的，看起来像是一个故障。所以它只是操纵了智能体与环境的接口。基于人类反馈的强化学习是一个经典案例，模型会因为奖励模型不完美而崩溃。所以在 RLHF 案例中，环境非常不完美。

Yeah. Like all those things have just been really easy for the model to extract. So once they added it, the model changed a lot and the score went up a lot and it was easy for the RL to find that in control. The oldest RL environment is normally a simulator that is fixed. There's no feedback. So the overoptimization looks like unphysical and nonsensical behaviors. There's the motorboat example going in circles. There's an example from a project I was middle author on, effectively overoptimizing half cheetah which is this Mujoco thing. Instead of running, it took cartwheels off into the sunset and got like infinite numbers. It's obviously not the intended purpose. It looks like a glitch. So it's just kind of manipulating the agent interface with the environment. RLHF is kind of a classic case where the model will just break down because the reward model is imperfect. So the environment is really imperfect in the RLHF case.

它太稀疏了，非常人为。

It's so sparse, it's very artificial.

是的，这是一个非常人为的环境。所以这些生成的 token 动作会简化为重复同一个 token，这是有道理的。比如，我们在 Hugging Face 早期的一个例子是模型只会说 JavaScript，不停地重复 JavaScript。就像一些玩具数据集，你看到时很明显。但如果你在做大量 RLHF，处于高层并做出何时停止训练的合理决策时，可能更难发现。但那是人们经历过的阶段，现在我们处于 RLVR 阶段，即当模型在数学上做对事情时给予奖励。我认为这更难过度优化，除非有工具，模型学会搜索和作弊而不是学习数学，我相信有人能在现实中看到这种情况。比如，模型会说，哦，你在用斯坦福的 CS 习题集训练我，我已经见过一千次了，那我就直接去找答案手册。我相信有人能找到确切的例子。但在代码和信息检索上，更容易作弊。代码方面，让单元测试通过的最简单方法就是放一个 pass 语句。模型学会这样做并不奇怪。因此，对于代码，你需要更多的奖励设计，我认为这是一个很好的学术课题：代码中的奖励设计如何平衡对测试用例的过度优化或避免失败等。我确信这不一定是受控环境，因为这些模型很复杂，但我猜你可以在某种程度上复现它。

Yeah, it's a very artificial environment. So it makes sense that these actions which are generated tokens will do things like reduce into just repeating one token over again. It'll be like, I think one of the early examples we had playing with this at Hugging Face was the model would just say JavaScript. It would be JavaScript JavaScript JavaScript. Just like some toy dataset and it's very obvious when you see it. It's probably harder to see when you're at the top and making decent decisions on when to stop training if you're doing a lot of RLHF. But that was kind of the phase that people have gone through and now we're in the RLVR phase which is we're giving the model reward when it does something quote unquote right for math. It's a bit harder to overoptimize, I think, unless you have tools and the model learns to search and cheat instead of learning math, which I'm sure somebody could see that out in the world. It's just like, oh, I'll just find the you're train. It's like the model's like, oh, you're training me on Stanford's problem set for CS whatever that it's seen a thousand times. So, it's like I'll just go get the solution manual, which I'm sure there's somebody can find an example where that has surely happened. But on code and maybe information retrieval, it's easier to fudge. So the code thing is like the easiest way to get a unit test to pass is just put a pass in it. And that is not too surprising that a model can learn how to do that. And there for code you need more reward design which I think would be a nice for like a substantial academic work is like what is reward design in code for balancing this sort of like understanding this overoptimization of test cases or avoiding failures or something like this. I'm sure there's it's not necessarily going to be a controlled environment cuz these models are complicated, but I would guess you can reproduce that in some ways.

再深入一点，奖励设计意味着例如，对部分正确的工作给予部分分数。

Just to double click, reward design means like for example, giving credit partial credit for partially correct work.

是的。或者如果你能检测到，就稍微惩罚模型做单元测试作弊的行为。是的，针对作弊。

Yes. Or like giving the model a slight penalty for doing the unit test thing if you can detect it. Yeah. For cheating.

是的。与数学相比，这给训练这些模型增加了很大复杂性，数学只是看答案是否正确。我的意思是，你可以看看 GRPO 数学，部分分数在那里很奇怪，因为它是按批次归一化的。我不知道我是否已经对此有完整的论述，但如果你混合领域，事情会变得非常复杂，比如代码中的部分分数是否比数学中的部分分数更好等等。奖励设计变得非常复杂，而这正是你激励模型做不同事情的方式。

Yeah. Which adds a lot of complexity to training these models compared to math which is just if the answer is right. I mean you can look at the GRPO math and partial credit is weird in that because it's kind of normalized per batch. I don't know if I have a whole spiel already on it for that but it's also just it becomes very complicated if you're mixing domains and it's like is partial credit in code better than partial credit in math or all these things. It's like reward design becomes very complicated and that's what you're incentivizing the models to do different things.

是的。有没有关于混合这些的文献或假设？假设你有代码的验证器、数学的验证器，以及其他你能想到的验证器，它们单独工作都很好。它们会冲突吗？

Yeah. Is there any literature or hypothesis about mixing these things? So let's say you have the one for code, you have the one for math, you have whatever other verifiers you can come up with and individually they work. Do they conflict?

我认为 RLVR 的部分直觉是模型擅长知道它处于哪个提示区域，这就是为什么如果你只训练数学或精确指令遵循，模型在知识基准上不会变差。所以模型会发展出对不同提示在空间中位置的直觉。因此，梯度更新会根据你的批次而不同，这也是为什么人们会说做大批次。这样模型的大部分被激活，强化学习的信号噪声更小，但很多直觉是模型能够处理这一点。关于顺序也有有趣的问题：你是否先做大规模数学和代码的强化学习来获得序列长度，然后再加入更通用的内容？是的，Deepseek 提到了这一点，但 Deepseek 报告中的一个点是：从数学和代码到更通用的强化学习。还有一个问题是，如果你要做代码执行和搜索，工具应该在哪个阶段引入？我不知道是交织在一起还是作为第二阶段。明白了。我对此没有评论。只是惊讶于有多少未知，而你需要大量算力来做消融实验。

I think part of the intuition of RLVR is that the model is good at knowing which prompt area it is, which is why the models don't get worse on knowledge benchmarks if you're training on just math or precise instruction following. So the model just kind of develops an intuition for where the different prompts are in space. So the gradient updates will be different depending on your batches, which is partially why people will just say do big batches. So a lot of the model is activated and you have a less noisy signal with RL, but a lot of the intuition is that the model just kind of handles that. And there's interesting questions on sequencing like do you do large scale math and code RL to get the sequence length and then add in more general stuff. Yeah. Which Deepseek mentioned, but that's one thing to go the Deepseek report is like math and code to more general RL. There's a question on where do you do tools if you're going to do like code execution and search within this. So I don't know if that's interweaved or if it's a second stage. Got it. Yeah. I don't have comments there. It's just like it's surprising how much is not known and you just need a lot of compute for ablations.

推理时的高生成长度肯定会破坏所有基础设施，因为 token 太多了。这增加了内存不足或其他出错的可能性。所以默认情况下，你所有的训练任务都需要更多的 GPU 来满足推理的内存需求。

The inference high inference length generations definitely just kind of breaks all infrastructure cuz it's just so many tokens. It's more opportunity for out of memory or other things to go wrong. So it's like just on a default all of your training jobs need way more GPUs for the memory of inference.

当然。

Sure.

或者只是训练，但这只会让它更麻烦。

And or just training but it's just it's just makes it more of a pain.

是的，这是成本问题。你知道，从你听过的 gnome prod 中可能有一个有争议的结论：从环境获得反馈的墙钟时间也是一个问题，尤其是如果它是真实世界的事情。

Yeah, that's a cost thing. You know, one of the maybe controversial takeaways from the gnome prod which you listened to was that there's also just wall clock time of just getting feedback from the environment whatever that is especially if it's like a real world thing.

我的意思是，在某个点上，你的训练运行不能超过一个人的寿命。所以对我来说，那就是一堵墙。他不同意这一点。但我的意思是，在某个时候，你确实希望长推理能在合理的时间内终止，不管怎样，作为用户。

I mean, there's some point at which your training runs cannot take longer than a human life. So to me, that was the wall. He disagreed with that. But that was what I meant by at some point you do want long inference to terminate within a reasonable amount of time regardless, just as a user.

是的，我们必须找到一种方法，在训练时间内内部加速，比实际宇宙的时间流逝更快。我不担心那个问题，但原则上我同意你的看法。

Yeah, we have to find a way to accelerate internally within the training time faster than the passage of time in the actual universe. I'm not worried about that problem, but I agree with you in principle.

对，我扯得太远了。我明白。

Right, so I'm stretching this all too far. I get it.

当我们开始总结时，还有哪些有趣的想法是人们应该追求的？在你的 AIE 演讲中，你说到我在考虑 Scaling RL，你提到了大型多领域数据集、难度过滤、长运行时间。有没有什么具体的东西，如果有人在做研究或想创业，这些是你不想做但希望别人探索的有趣事情？我认为大多数都不在推理领域，而演讲一直围绕推理。

As we start wrapping up, what are other interesting ideas that people should pursue? In your AIE talk, you said what I'm thinking about for scaling RL, you had big multi-domain datasets, difficulty filtering, long run times. Is there anything specific that if there are people out there doing research or wanting to start a company, these are interesting things you don't want to do but want others to explore? Most of them I think are not in the reasoning space, which the talks have been about reasoning.

我一直说角色训练是被低估的东西，我一直在指导一个学生做这个。比如个性训练。以及如何通过提示、激活或微调（比如数据工程）来改变模型个性的不同方式。就像 Joan Jen 为 OpenAI 做的那样。这有多重要？基础研究是什么？希望我能分享更多我指导学生的内容。我已经说了好一阵子了。

I've been long talking about character training as something that is under-indexed on, and I've been advising a student on that. Like personality training. And how different ways of changing the personality of the model from prompting, activation, or fine-tuning, like data engineering. Stuff that Joan Jen does for OpenAI. How much does that matter? What are the fundamental research things? Hopefully I can share more that I've been advising a student on. So I've been saying that for a while.

顺便问一下，你喜欢她做的模型规范之类的东西吗？

Do you, as a side note, like the model spec stuff that she's doing?

是的。

Yeah.

好的。那个方向。

Okay. That trajectory.

是的，我早期就是它的粉丝。我是说，她就是这样注意到我的，因为我是唯一一个在他们首次发布时就报道它的人。我想那是一年多以前了。

Yeah, I've been an early fan of that. I mean, that's how she noticed me, as the only person that covered it when they first released it. I think it was over a year ago.

我喜欢它。

I liked it.

是的，没多少人喜欢。

Yeah, not many people did.

好吧。好吧。你是第一个。

Okay. All right. You were first.

我不知道。但她是这么跟我说的。

I don't know. But that's what she said to me.

嗯，我们有一个模型规范的演讲，结束了整个会议，对吧？那是我注意这个的信号。

Well, we had a model spec talk that closed the whole conference, right? That was my sign to pay attention to this.

但它是真实的，因为它传达给开发者的信息。它对开发者有好处，知道你的模型走向，还有监管方面。我认为区分有意行为和仅仅是训练错误非常重要。

But it's real because of what it sends to developers. It has a developer benefit of knowing where your model is going, and also just regulatory. I think it is very important to distinguish intentional behavior versus just a training error.

好的。

Okay.

所以我认为对于模型透明度来说，这非常棒，而且我说过模型规范比宪法有用得多，因为宪法是你给训练算法的中间训练产物，以获得你想要的模型。它不一定是我们喜欢的模型；我们不会以宪法的形式写下模型的目标。

So I think for model transparency it's really fantastic, and I've said that the model spec is much more useful than a constitution, because a constitution is an intermediate training artifact that you give to the training algorithm to get the model you want. It is not necessarily what model we like; we don't write down our goals of the model in a constitution form.

顺便问一下，你看过宪法吗？

By the way, have you looked at the constitution?

不是他们谈到的。他们加入了苹果的设计指南，还有联合国宣言。在这个层面上我见过。我不知道他们是否更新了。这很奇怪。我希望他们会写一个模型规范。我不太乐观，但他们是下一个倒下的多米诺骨牌。

Not they talked about it. They put in Apple's design guidelines but also the UN declaration. At this level I've seen it. I don't know if they've updated it. That's very odd. I hope they would write a model spec. I'm not too optimistic, but they're the next domino to fall.

嗯，我对此的看法是：我推动这个太晚了，因为 OpenAI 已经批准了演讲，但我本来想让他们比较 OpenAI 的模型规范和 Claude 的系统提示，这是他们最接近模型规范的东西。系统提示是不完整的，因为 OpenAI 的模型规范中有他们模型目前没有做到的事情，尤其是刚开始的时候。比如‘我们希望模型能够参与敏感话题，甚至可能包括 NSFW’在他们的模型规范中，这只是他们想要它做什么的信号。他们说这很难实现，因为存在所有明显的风险，但在一个理想模型中，如果我们能解决所有问题，这就是我们要做的。我认为这对许多不同的利益相关者都有好处。所以主要是，还没有一篇好的基础研究论文关于这个，这有很多工作要做。它也涉及到个性化和个性或类似的东西，就像如果开放模型要赢，部分原因可能是每个人都能拥有他们想要的模型。我们提供 GPT-4.5；它就是这样。你可以提示它，但如果微调比提示更有效，每个人都可以拥有他们想要的模型。所以这是一个很好的学术问题或开放生态系统问题，人们在更可能获胜的领域上竞争。

Well, my take on that actually: I pushed for this too late because OpenAI already approved the talk, but I was going to ask them to compare the OpenAI model spec to the Claude system prompt, which is their closest thing to the model spec. The system prompt is incomplete because OpenAI has things in the model spec that their model doesn't currently do, especially when they started. It's like 'we want the model to be able to engage on sensitive subjects and maybe even NSFW' was in their model spec, which is just signaling of what they wanted it to do. They say this is very hard to implement because of all the obvious risks, but in an ideal model where we can solve every problem, this is what we do. I think that's good for many different stakeholders. So mostly my thing is there hasn't been a good foundational research paper on that, which is a lot to do. It also runs into personalization and personality or similar, which is like if open models are to win, part of it could be that everybody can have exactly the model they want. We're serving GPT-4.5; it's kind of its thing. You can prompt it, but if fine-tuning is more effective than prompting, everybody can have the model they want. So it's a good academic problem or an open ecosystem problem where people are fighting on the turf that feels more likely to win.

是的。

Yeah.

这是你作为 AI to OMO 想要赢的地方，还是你只是指导一个研究生做这个？

Is this somewhere where you, speaking as AI to OMO, you want to win, or is this just you advising a grad student on it?

我认为它还不是一个差异化因素，但我很愿意研究它。我认为开放模型有很强的角色扮演用例和角色个性化，所有这些，尤其是因为人们找到了他们的“老婆”并想留住她。这是一个贬义词，但……

I don't think it's a differentiating factor yet, but I'm very open to working on it. I think open models have a strong roleplay use case and character personalization, all that stuff, especially because people find their waifu and want to keep their waifu. That's the derogatory term for it, but...

我会说我们肯定讨论过这个，我想……部分应该是它是一个基础模型，容易朝你想要的方向发展，我们可能会对个性持稍微保守的看法。我的意思是，我读过 OpenAI 的模型规范，大部分我们都同意，并且在拟人化上要保守。

I would say that we've definitely discussed it and I want to... part of it should be that it is a base model that's easy to take in directions you want, and we will have an opinion that is probably slightly conservative on personality. I mean, I've gone through the OpenAI model spec and most of these we agree with, and be conservative on anthropomorphization.

你不同意什么？

You disagree with?

我不记得了；我几个月前做的。但很多是关于开放性或透明度，也就是说如果我们训练一个开放权重的模型个性，我们不会隐瞒任何东西，而且我们有不同的层级。所以大多数是那种信息交换，而不是‘要友善’。开放权重模型是相当随和的。如果你通读一遍，就像‘尊重用户’所有这些。

I don't remember; I did it a couple months ago. But a lot of it is openness or transparency, which is if we're training an open-weight model personality, we're not going to withhold anything, and we have a different hierarchy. So most of them are that type of information exchange rather than 'be kind.' An open-weight model is pretty agreeable. If you read through it, it's like 'treat the user with respect,' all these things.

他们就是这样教育孩子的。读读规范就知道了。

They're raising kids that way. Just read the spec.

是的，听起来有点蠢。

Yeah, it sounds kind of stupid.

对于做研究的人来说，最后一点是那种古怪的模型路由，你可以找出很多不同的模型来路由，因为开放模型工具比任何 OpenAI 产品都能更轻松地使用更多模型，因为 OpenAI 仅限于自己的模型。

The last thing for people doing research is wacky model routing things where you figure out a bunch of different models to route to because an open model tool could use way more models more easily than any OpenAI product, since OpenAI is restricted to its own models.

比如 OpenRouter 可能说，我要用这个做一个产品，就是一个路由器。Open Router 确实这么做了，他们说我们的聊天窗口根据我们所有的使用情况知道最好的模型。

Where if maybe OpenRouter is like, I'm going to make a product out of this which is a router. Open Router actually does it and they're like our chat window knows the best model based on all this usage that we have.

针对你的查询。

For your query.

还有其他人以其他方式开始，比如 Martian，不是 Diamonds。我不知道还有谁。他会知道。

There's people that started other way like Martian, not Diamonds. I don't know who else is. He would know.

有很多。

There's a bunch.

有很多。所以，我不知道那是否可行。Hugging Face 应该在这方面努力。

There's a bunch. So, I don't know if that would work. Hugging Face should work on it.

这是一个登月计划式的想法。你不知道它什么时候会……

It's a moonshot idea. You don't know when it'll...

鉴于你的 Hugging Face 背景，Hugging Face 如何赚钱？这是一个非常常见的梗问题。

Given your Hugging Face background, how does Hugging Face make money? This is a very common meme question.

我认为主要是企业交易。

I think mostly enterprise deals.

他们就是这么说的。他们在做自己的事。我的意思是，是的，他们很棒。他们很大。他们很赚钱。只是对大多数人来说不那么明显。

That's what they say. They're doing their thing. I mean, yeah, they're great. They're big. They're profitable. It's just not that obvious to most people.

我喜欢媒体模型的 router 想法。我觉得有很多长尾的，比如背景移除器、风格应用器等。这在技术方面实际上很难找到。我觉得除非你有延迟或价格限制，否则就用大的。你应该只用最好的模型。即使我们在做缩略图时，我也是这样，比如我想移除某人的背景，我去 Replicate，那里有 55 个背景移除器。

I like the router idea for media models. I feel like there's a long tail of a background remover, a style applier, etc. That is actually hard to find on the tech side. I feel like just use the big unless you're under some latency or price constraint. You should just use the best model. Even when we're doing thumbnails, I'm like, okay, I'm trying to remove a background of somebody and I go to Replicate and there's like 55 background removers.

是的，我就用 Adobe，因为它是一个网站。

Yeah, I just use Adobe because it's a website.

但那不管用。Photoshop 模型在某些方面很糟糕。但同样，比如我想生成一个图表来模仿某物，那么哪个模型更擅长图表呢？这些都不容易找到，因为没有基准测试是……

But that doesn't work. The Photoshop model is bad on some things. But again, it's like I want to generate a diagram to mimic something and it's like, well, which model is better for diagrams? Those are not easy to find because none of the benchmarks are...

部分论点是，如果蒸馏效果很好，我们可以不断让蒸馏的目标越来越小，也就是模型非常狭窄，对吧？它们在这些巨大的模型上模仿一些非常具体的东西，比如重新格式化表格。你能在一个 1 亿参数的模型中做一个从 markdown 到 LaTeX 的表格重新格式化器吗？如果你把它做得足够小，它在经济上真的可行，因为它几乎免费且即时。

Part of the argument is that if distillation works really well, we could just keep making the target for distillation smaller and smaller, which is you have models that are very narrow, right? And they're mimicking these huge models on something that's pretty specific, like reformatting tables. Can you do a table reformatter from markdown to LaTeX in a 100 million parameter model? If you get it small enough, it is really economically feasible because it's effectively free and instant.

我的反驳是：如果你在做图像编辑，AI 应该做所有的事情。

My pushback is on this: if you're doing image editing, AI should do all of it.

嗯，是的，但我认为它确实……只是我们还没到那一步。再给五年时间，它就能做到一切。那为什么还要研究路由器呢？你只需要扩大规模。

Well yeah, but I think it does like... it's just we're not there yet. Give it five years, it'll do it all. So why work on a router at all? You just scale up.

我想……是的。

I guess... yeah.

告诉我这里的逻辑是什么。这就像是一个临时的事情。

Tell me where the logic is here. This is like a temporary thing.

在设备上。

On device.

在设备上。我认为本地建模社区比人们认为的要小得多，因为开放模型的大部分使用仍然在 API 中。就像 DeepSeek API，如果模型不多，有人会以比大多数人自己做更便宜的价格托管它。这很现实。但确实有一个小社区需要本地化。

On device. The local modeling community, I think, is much smaller than people give it credit for because most of the use for open models is still in APIs. It's like DeepSeek API and it's like if there aren't that many models, somebody's going to host it for cheaper than most people doing it themselves. That's pretty realistic. But there is a small community that needs local.

是的。

Yeah.

最好的结果是开放模型不仅能在长尾事物上竞争，但这需要最大的转变。顺便说一句，我因为这个原因抵制自己买 GPU、建集群。我觉得 API 能解决大部分问题。人们亏本为我提供模型服务。我为什么要自己拥有呢？除了 4090 的价格在过去一年翻了一番。所以实际上你做本地模型赚了钱。

The best outcome is if open models can compete on not just longtail things, but that takes the most transformation. Side note, I resisted building my own GPUs, building my own cluster for this reason. I'm like API will solve most of it. People are losing money to serve me models. Why am I having those? Except for the fact that 4090 prices have doubled in the last year. So actually you made money doing local models.

那怎么赚钱呢？因为你的投资升值了。

How does that make you money? Because your investment goes up.

是的。而且二手 4090 也涨价了。有趣。

Yeah. And used 4090s go up. Interesting.

买了一个 4090。我得到了一个 4070。

Bought a 4090. I got a 4070.

该死。

Damn it.

我当时想这是什么？然后这让我考虑，如果 5090 广泛可用，我是否应该买一个。

I was like what is this? Well then it puts me on to like should I buy a 5090 if it ever becomes widely available.

在 GTC 上他们搞了掉落活动。太疯狂了。你就像跑去露营车买它。

At GTC they were doing the drops. It was crazy. You were like running to the camper to buy it.

我知道。

I know.

在我问结束问题之前还有其他话题吗？总的来说你的工作、ROVR、当下的话题。我认为公司应该继续考虑重新发布开放模型，主要是为了公关和吸引用户。如果 OpenAI 在发布，这似乎是一个趋势。

Any other topics before I give a closing question? Just generally your work, ROVR, topics of the day. I think companies should keep considering re-releasing open models mostly for PR and onboarding. It seems like a way it's going if OpenAI is releasing it.

你对此兴奋吗？你觉得这是一个第二次……

Are you excited about that? Do you feel like it's a second...

OpenAI 的模型会很好。我期待它。

The OpenAI model will be good. I expect it.

他们很认真。它会在某些任务子集上成为某个规模类别中最好的。OpenAI 只做那样的事情。你必须尊重他们。

They're pretty serious. It'll be best in class for some size category in some subset of tasks. That's like OpenAI only does things like that. You have to give them respect.

是的。那是一个巨大的胜利。当更多人参与时，开放就赢了。所以那是一个胜利。

Yeah. Like that is a big win. Open wins when more people are doing it. So that's a win.

是的。嗯，我的意思是希望他们真的对技术开放，而不仅仅是权重。

Yeah. Well, I mean hopefully they are actually open about the techniques and not just the weights.

我们认为开放模型的大小能告诉我们关于他们将构建的硬件的任何信息吗？

Do we think the size of the open model tells us anything about the hardware that they're going to build?

不。

No.

什么？不。他们对此非常保密。这就是为什么他们没有发布 GPT-3.5 或任何东西，因为它太容易暴露内部情况或计划了。

What? No. They're so secretive about this. That's why they haven't released GPT-3.5 or anything because it's too revealing about internal stuff or plans.

哦，好的。不，我在说 Stargate 或者什么硬件？

Oh, okay. No, I was talking about Stargate or what kind of hardware?

我认为那是不同的形态。是的，那是……是的。

I think that's a different form factor. Yeah, that's... Yeah.

我认为那东西会在云端运行。我不认为它会本地运行。

I think that thing will run on the cloud. I don't think that'll run locally anyways.

好吧，我们得谈谈这个。似乎每个播客我们都在谈。所以显然今天的新闻，我想你看到了，是一个耳戴设备，他们被起诉了之类的。但我认为耳戴形态很好。我实际上在 B 那里达到了这一点，关于它最终会走向何方？你希望 AI 听到你听到的东西，而你在耳朵上听到你听到的东西。基本上就是这样。我不知道你们对可穿戴设备及其发展方向有什么想法。

Well, okay, we have to talk about it. It seems like every podcast we talk about it. So apparently the news from today, which I think you were looking at, was that it was like an ear device that they got sued over or whatever. But I think the ear form factor is pretty good. I actually did get there with B in terms of where does this ultimately go? You want the AI to hear what you hear and where do you hear what you hear on the ear. That's pretty much it. I don't know if you guys have thoughts on wearables and where that goes.

我试着……我认为它知道得太多了。这是我的……这真的是我的……

I try to be... I think it just knows too much. That's my... that's really my...

但你想给它上下文。

But you want to give it context.

是的，我有虚假的隐私希望。我认为像很多人一样，我的意思是整件事就是人们实际上并不关心隐私。

Yeah, I have false privacy hopes. I think like a lot of people, I mean that's the whole thing is like people don't actually care about privacy.

它只是不……它只是非常好的记忆。

It's just not taking... it's just really good memory.

我认为 Meta Rayban 的形态很好。我不认为它那么大众市场。

I think the Meta Rayban form factor is good. I don't think it's as mass market.

就像如果做成 AirPods 大小的形态，显然市场会大得多。但太阳镜形态也是可行的，我觉得。

It's like if you get it in an AirPods size form factor, it's a way bigger market for obvious reasons. But the sunglasses form factor is a thing that works, I think.

好的。

Okay.

我自己不用它们做 AI，但可以把 AI 放进去用。从经验上看，这显然是可行的。

I don't use them for AI, but they can fit the AI to work it. Empirically, it obviously works.

是的。

Yeah.

酷。

Cool.

嗯，我留的最后一个问题是关于 Meta 在做什么。你四月份发了一篇很有意思的帖子，说 'Llama for：Meta 是不是按下了恐慌按钮？' 我觉得当时他们其实没按，但现在他们真的按了。

Well, the last question I was saving up was this whole what is Meta doing? You had a pretty interesting post back in April. You said 'Llama for: did Meta just push the panic button?' I feel like back then it didn't actually push the panic button, but now they really pushed the panic button.

有道理。我觉得当时的恐慌按钮是整个 LMIS 模型不是他们发布的那个模型，再加上发布日期的各种奇怪之处。但一个声称开放却不发布自己主打模型的模型，就是执行不力。执行不力。

That's fair. I think the panic button at the time was the whole LMIS model not being the model that they released thing, along with a bunch of weirdities about the day of the week they released. But to be a model that claims to be open and then not release the model that is your leading claim is just bad execution. Bad execution.

是的。

Yeah.

这倒没什么。最近的事情我觉得主要可以归结为人才比 GPU 便宜得多。归根结底，如果我们已经花了这么多钱，他们走进房间，对着镜子一看，你会想，等等，把钱花在顶尖人才身上可能也没那么离谱。不如试试看。

Which is fine. And then the recent stuff I think mostly can be boiled down to talent is cheaper than GPUs by a dramatic margin. And at the end of the day, it's like, okay, if we're spending this much, they go to the room and they stare in the mirror and you're like, wait, it might not actually be that ridiculous to spend this money on the top people. It's like, might as well try it.

他们已经在 VR 上花了那么多钱。

They already spend it on VR.

迟早会有人这么做的。而且这很合理，就像如果苹果决定要做，他们也会进来做 Meta 正在做的事。

Somebody was bound to do this eventually. And it makes sense that it's like if Apple somehow decided we're going to do this, they'd come in and do exactly what Meta is doing.

他们需要一个创始人模式的 CEO，说 '管他呢，我们认栽。' 我想到的是，Meta 与其把钱花在 VR 上，不如花在 RLVR 上。

They need a founder mode CEO who's like 'screw it, we'll take the L.' The thought that occurred to me is, you know, Meta instead of spending on VR, they should spend on RLVR.

我觉得问题是，很多研究人员，大多数人都会拿着高薪开心地跳槽……

I think the question is, I think a lot of researchers, most people will take the payday and happily move to...

每个人都有一个收买价。

Everybody has a bribe number.

对。

Right.

只是非常大。

Just really big.

是的，但我觉得有些研究人员对这种 '伟人理论' 感到不安，认为必须花这么多钱才能得到这个水平的人才。

Yeah, but I think some researchers are uncomfortable with the idea that this is a sort of great man theory of research, that you have to pay this much to get this level of talent.

人才肯定是分布式的，对吧？

The talent is definitely distributed, right?

很多他们愿意花大价钱请的人都有信心重做事情，或者只是做一些同样的事情。不管你是称之为 '感受到 AGI' 还是只是建造东西的驱动力，'感受到 AGI' 和硅谷过去传说中的很多东西没什么不同。就是那些有愿景、愿意执行、看到未来的人，这些人能带来巨大改变。我觉得你有了这些人，再消除官僚主义。获得有技术天赋的研究人员，实际上是 Meta 拥有很多或者有能力获得很多的。所以这很大程度上是循环利用，这对个人和组织士气来说很艰难，但作为一种方法是可以理解的。

A lot of the people that they would be paying this much have the confidence to redo things or to just do some of the same things. Whether you call it feeling the AGI or just drive to build things, feeling the AGI is not that different from a lot of things that have existed in Silicon Valley lore in the past. So just people with the vision that are willing to execute on it and they see something coming, and those people make a big difference. I think you have those people and you remove bureaucracy. Getting technical talented researchers is actually something that Meta has a lot of, or has the ability to get a lot of. So it's a lot of recycling, which is very hard on individuals and morale of an organization, but it's understandable as an approach.

是的，当然。酷。我就问这些。

Yeah, for sure. Cool. That's all I have.

关于你打算如何打造 '美国版 DeepSeek'，还有什么临别感言吗？那条推文很棒。

Any parting thoughts on how you're going to build the American DeepSeek? That was a nice tweet.

是的，如果我看我的十年目标，其实我只有两到五年的目标。我认为随着模型越来越向智能体转变，Scaling 正在放缓。这其中有固定成本和固定路径的一面，去实现类似 '美国版 DeepSeek' 的东西。我主要想说，如果它是完全开放的，你拥有一切并且可以修改，那它不一定是美国的。有几件事需要做。很多只是更多资源，但 Qwen 32B 如果眯眼看，相当于原始 GPT-4 水平，而且完全开放。你需要经过几个层次。那显然是一个稠密模型。需要把它变成稀疏的，你需要扩展它。你需要更多的 GPU，然后你需要做大规模推理。这就是我想做的目标。在如何与 AI 合作方面有很多复杂性。AI2 要怎么做才能达到那里？这非常困难。

Yeah, mostly if I look at what my 10-year goal is, it's like I only have a two to five year goal. I think as models are shifting more towards agents, I think that scaling is slowing. There's a side of it of a fixed cost and a fixed path to getting towards something like American DeepSeek. Mostly I would say it doesn't have to be American if it's fully open, if you have everything and you can modify it. There are a few things that need to be done. A lot of it is just more resources, but it's like Qwen 32B is, if you squint, original GPT-4 level and fully open. And there are a few levels you need to go through. That's obviously a dense model. It needs to be taken to sparse, and you need to scale it. You need to have a lot more GPUs, and then you need to do large-scale reasoning. That's the goal I want to do. There's a lot of complexity in navigating how to work with AI. What does AI2 do to get there? It's very hard.

我认为它是一个非营利组织。很难获得资源，构建模型需要协调很多不同的人。这就是 DeepSeek 的故事：他们有优秀的人才。OpenAI 长期保留了很多非常优秀的人。Anthropic 现在也吸引了很多优秀的人。这需要积累很多渐进的硬技术问题。这就是我想在未来几年里努力实现的事情，但达到那里并不容易。所以这个说法是：AI2 的最佳情况是 AI2 会做其他事情。你不能经营一个非营利组织或公司，说我们的目标是在三年内拥有一个 '美国版 DeepSeek'。没有人会为此买单，因为你需要讲一个更好的故事。但这就是我想做的，而且我相信 AI2 会在这个过程中做很多更有趣的事情，比如产品方面。

I think it's a nonprofit. It's hard to get the resources, and building a model is a lot of aligning a lot of different people. That's the DeepSeek story: they have great people. OpenAI has kept a lot of really good people for a long time. Anthropic has gotten a lot of good people right now. And it's a lot of incremental hard technical problems that you need to stack up. That's what I would like to do and make work in the next couple years, but it's not easy to get there. So that's the pitch: AI2's best case scenario is AI2 is going to do other things. You can't just run a nonprofit or a company that says our goal is in three years to have an American DeepSeek. No one's going to keep paying the bills on that because you have to tell a better story. But that's what I would like to do, and I'm sure AI2 will do many more interesting things along the way, like product stuff.

我不认为这一定是产品，而更像是 AI 中那些前沿的东西，我们可以为某些事情创造新的架构。

I don't think it's necessarily product, but more like what are cutting-edge things in AI that we could make a new architecture for certain things.

或者像开放模型更好工作的演示，无论你是否有私有数据之类，或者只是能让你脱离 Transformer 轨迹的远见想法。我认为你仍然需要做这些才能在 AI 中领先。

Or like what are demos of open models working better, whether you have private data or something, or just far-out ideas that could take you off the Transformer trajectory. I think you still need to be doing these to kind of lead in AI.

感谢你为真正开源 AI 所做的努力。

Thank you for working so hard on truly open source AI.

是的，很有趣。我觉得这很容易让价值观与你所做的一致。

Yeah, it's fun. I think it makes it easy to align values with what you're doing.

如果更多东西是开放的，世界会更好，因此很多只是靠意愿让它成为现实。

It would be better for the world if more things are open, and therefore a lot of it is just willing it into existence.

我认为看到 OpenAI 所做的或他们声称将要做的，希望很快能取得胜利。

I think seeing what OpenAI does or is saying they're going to do as hopefully a win coming soon.

是的。DeepSeek 是最意想不到的胜利，它让其他多米诺骨牌倒下，但我认为这就是前进的道路，看看需要什么。非常感谢。

Yeah. DeepSeek was the most unexpected win that made some other dominoes fall, but I think that is the path forward and see what it takes. Thank you so much.

谢谢你来参加。

Thanks for coming on.