重新思考面向智能体 AI 的预训练

在很长一段时间里，我们都在用静态基准来评估预训练。如果你希望这些模型作为智能体有用，它们就需要能够与环境交互。当我们开始关注这些智能体式任务时，预训练就需要从根本上重新思考。这不仅仅是后训练的问题，来实现我们想要在下一代模型中拥有的那套能力。而衡量这类智能所需的基准，有时今天还不存在。

For the longest time, we were measuring pre-training on static benchmarks. If you want these models to be useful as agents, they need to be able to interact with environments. And when we start caring about those agentic tasks, pre-training needs to rethink from fundamentals. This is not just a post-training problem to achieve these set of capabilities that we want in the next generation of models. And the kind of benchmarks we need for measuring this kind of intelligence is sometimes not available today.

好了，各位。欢迎收听 TwiML AI 播客的另一期节目。我是主持人 Sam Cherington。今天，我们邀请到了 Aakanksha Chowdhery。Aakanksha 是 Reflection 的技术人员。在开始之前，请务必在你收听今天节目的地方点击订阅按钮。Aakanksha，欢迎来到播客。

All right, everyone. Welcome to another episode of the TwiML AI podcast. I'm your host, Sam Cherington. Today, I'm joined by Aakanksha Chowdhery. Aakanksha is a member of technical staff at Reflection. Before we get going, be sure to hit that subscribe button wherever you're listening to today's show. Aakanksha, welcome to the podcast.

谢谢你，Sam。

Thank you, Sam.

你的背景非常有趣。你训练过一些最早的大语言模型，包括 PaLM、Gemini 1.0 和 Gemini 1.5。跟我们聊聊那些经历吧。

You have a really interesting background. You've trained some of the earliest large language models including PaLM and Gemini 1.0, Gemini 1.5. Tell us a little bit about those experiences.

我在谷歌开始接触大语言模型，当时正在构建一个分布式系统，最终训练出了 PaLM，那是我们当时最大的语言模型，有 5400 亿参数，在那之后大家就不再公布参数数量了。这让我走到了预训练的前沿，在 PaLM 前两代和 Gemini 前两代中，逐一解决规模带来的各种问题。我认为做预训练时学到的一点是，在规模下每个问题都会被放大，并且堆栈的每个部分都会变得很强。所以这总是很有趣，也很令人兴奋。

I got into large language models at Google while building one of the distributed systems that led to the training of PaLM, which was our largest language model at the time. It had 540 billion parameters and people stopped publishing the number of parameters after that one. That led me to be at the forefront of pre-training, solving one set of problems after the other that come with scale in the first two generations of PaLM models and then the first two generations of Gemini models. I think the one thing that you learn when you do pre-training is that at scale every problem magnifies and things go strong at every possible part of the stack. So, it's always fun and it's always exciting.

你说你是在基础设施这边？

And you said you were on the infrastructure side.

我做过机器学习方面，但两边都做过。预训练非常有趣的一点是，你必须能够跨堆栈思考。否则，如果你要训练一个模型两到两个半月，你就需要考虑系统的每一个部分。

I worked on the ML side, but I've done both. One thing that is super interesting about pre-training is that you have to be able to think across the stack. Otherwise, if you're going to train something for two or two and a half months, you need to be able to think about every single part of the system.

很好。那么跟我们说说 Reflection 吧。Reflection 专注于什么？

Nice. And so tell us about Reflection. What is Reflection focused on?

Reflection 的使命是构建面向智能体能力的前沿开放智能。公司一直专注于为智能体任务构建后训练堆栈，而随着最近一轮融资，我们正在做端到端的训练。所以我们正在构建前沿开放的智能体模型，这些模型在内部同时进行预训练和后训练。

The mission of Reflection is to build frontier open intelligence for agentic capabilities. The company has been focused on building the post-training stack for agentic tasks, and with the most recent fundraise, we are doing training end to end. So we're building the frontier open agentic models which are both pre-trained and post-trained in-house.

这确实是我们今天要讨论的重点。你认为采用不同的预训练方法很关键，你是这么想的吗？

And that's really going to be a focus of what we're talking about today. Some of the reasons why you think pre-training a different approach to pre-training is key. Is that kind of the way you think about it?

我真正想说的是，在很长一段时间里，我们都在用静态基准来评估预训练。例如，LAMBADA 是一个流行的基准，还有 GSM8K，或者数学奥林匹克问题如 AMC 和 MATH，等等，但都是极其静态的基准。如果你希望这些模型作为智能体有用，它们就需要能够与环境交互，并在我们可以使用它们的工作流程中对我们有用。我们今天已经开始看到的最简单的例子是编码智能体和深度研究智能体。编码智能体非常有用，因为你可以让编码智能体帮你理解大型代码库，或者它们可以帮助你，例如，重构并在多个文件中应用修复。它们可能做得不对，还不完美。或者对于深度研究智能体，我认为我们已经从单纯的搜索栏界面转向更像“这是你想要的东西”，然后让语言模型去查找多篇研究该工作流程的文章。你可以想象，任何这样的目标导向任务都可以交给语言模型，模型可以通过多个步骤实现这些目标，而不仅仅是作为聊天机器人。所以这就是我们开始关心的那种智能体任务。当我们开始关心这些智能体任务时，预训练就需要从根本上重新思考。这不仅仅是后训练的问题，来实现我们想要在下一代模型中拥有的那套能力。而衡量这类智能所需的基准，有时今天还不存在。所以我也很乐意谈谈这个。

Well, the way I really put it is that for the longest time we were measuring pre-training on static benchmarks. For example, LAMBADA is a popular one, or GSM8K, or math Olympiad problems like AMC and MATH, or you name it, but extremely static benchmarks. If you want these models to be useful as agents, they need to be able to interact with environments and be useful to us in the workflows where we can use them. The simplest version of that that we are already starting to see today are coding agents and then deep research agents. The coding agents are extremely useful in the sense that you can put a coding agent to help you understand a large code base, or they can help you, for example, to refactor and apply a fix across multiple files. They might not do it correctly; they're not perfect yet. Or in deep research agents, I think we moved away from just a search bar interface to more like 'here is what you want' and then putting a language model on the job of finding multiple articles that investigate that kind of workflow. You can imagine that any such goal-oriented task can be given to the language models, and the models can achieve these goals over multiple steps as opposed to just being chatbots. So those are the kind of agentic tasks that we start caring about. And when we start caring about those agentic tasks, pre-training needs to rethink from fundamentals. This is not just a post-training problem to achieve these set of capabilities that we want in the next generation of models. And the kind of benchmarks we need for measuring this kind of intelligence is sometimes not available today. So I'm happy to talk about that as well.

也许为了强调这一点，过去三年里我们取得的许多渐进式进步，从 ChatGPT 开始，无论是推理还是工具使用，所有这些有助于我们构建所谓智能体系统的能力，都不是来自对模型核心基础大语言模型预训练方式的根本性不同思考，而是来自附加了关于如何通过强化学习和其他技术调整或后训练它们的新思路。为什么我们不能继续这样做呢？

And to maybe underscore this point, a lot of the incremental progress we've made over the past three years since, you know, ChatGPT, whether we're talking about reasoning, whether we're talking about tool use, all these things that contribute to our ability to build so-called agentic systems, this has all come from not, you know, thinking radically differently about the way these models, the core, the base LLMs are pre-trained, but by kind of tacking on new ways of thinking about how we tune them or post-train them through reinforcement learning and other techniques. Why can't we keep doing that?

哦，我们完全可以继续这样做。但我认为这有限制性，因为它从根本上限制了你在能力方面所能达到的水平。所以我思考模型的方式是：它们能做什么？能力可以是任何事情：它们有长上下文能力吗？它们能在更长的上下文中进行检索吗？或者它们擅长，例如，解决自然语言理解问题吗？它们擅长自然语言生成问题吗？它们擅长推理吗？我认为对于像编码任务这样的智能体任务，特别重要的是，如果你试图阅读一个代码库，然后执行其中的某些部分或编写相关内容，你会随着时间的推移积累上下文，因为你阅读了大量内容，然后如果你执行了某些操作，你还会从执行中获得额外的反馈。所以你需要模型具备的能力是：拥有规划能力，能够在其可能很长的上下文长度上进行推理。所以，什么进展顺利，什么不顺利。例如，它需要能够从错误的轨迹中失败。所以它可能会有短期挫折，做了错事，但随后需要自我纠正。然后它需要能够即时学习新工具。所以它需要能够探索工具，并弄清楚如果被置于新环境中应该使用什么。所以所有这些在某种程度上是可能的，但有时是有限的，然后你看到非常有限的根本问题是上下文工程问题。

Oh, we totally can keep doing that. I think it is limiting in the sense that it fundamentally limits you in what you can achieve in terms of capabilities. So the way I think about models is: what are they capable of? The capabilities can be anything from: do they have long context capabilities? Can they do retrieval over longer context? Or are they good at, for example, solving natural language understanding problems? Are they good at natural language generation problems? Are they good at reasoning? I think what is especially important about agentic tasks like coding tasks is that if you're trying to read a codebase and then execute some parts of it or write about it, you will accumulate context over time because you're reading a lot of content, and then if you execute something, then you also have some additional feedback from execution. So what you need a model to be able to do is have planning as a capability, have ability to reason over its context length which might be very long. So, what went well, what did not go well. It needs to be able to fail from trajectories that went wrong, for example. So like it can have short-term setbacks where it went and did something wrong, but then it needs to correct course for itself. And then it needs to be able to learn new tools on the go. So it needs to be able to explore tools and figure out what it should use if it's put in new environments. So all of this is possible to some extent but it's sometimes limited, and then the fundamental problem where you see it's very limiting is the context engineering problem.

那么，当如今的公司构建智能体系统时，他们是通过大量上下文工程来实现的——基本上受限于我能塞进上下文窗口的内容，这就是我的智能体能处理的长度，然后下一个任务就交给下一个子智能体。是什么让你相信，仅通过改变预训练就能实现你试图达到的目标？我们是否需要更根本性的改变、一种新架构，或者说一种后 Transformer 的方法？你对此怎么看？

So when you have companies building agentic systems today, they are building them by doing a lot of context engineering where they tack on a lot of—like it's literally limited to this is what I can fit into context length, and that's the length my agent will do, and then the next thing will be the next sub-agent. What gives you the confidence that we can achieve the goals you're trying to achieve just by changing pre-training? Like do we need a more fundamental set of changes, a new architecture, kind of a post-Transformer approach? How do you think about that?

当我谈论重新思考预训练时，我总会从我们希望模型具备哪些能力出发。对我来说，架构只是方程的一部分。方程的另一大部分是损失目标和训练数据。正如我之前提到的，我们希望这些模型能够进行多步骤的规划和推理。它们需要回顾之前上下文中的内容，并以此为依据来指导下一步行动。要实现这一点，Transformer 中的基础机制是注意力机制。注意力机制在长上下文上是否有效，是我们正在认真审视的问题——这是否是实现长形式推理的最佳方式。我们确实知道，长形式检索——比如在数百万的上下文长度中进行检索——如今用大语言模型是可以实现的。但长上下文的推理是一个更难的问题。第二个极其重要的因素是输入这些模型的训练数据。第三个极其重要的因素是损失目标。所以，信不信由你，我们实际上一直在训练这些模型进行概率性的下一个词预测——预测最可能的下一个词。而由此，我们成功让这些模型成为了极其强大的推理者。但这是否是最好的目标呢？我们需要这些模型具备更基础的长形式推理能力，并能够学会从上下文中学习。

When I go and talk about rethinking pre-training, I always think about it in terms of what are the capabilities we want out of the models. And to me, architecture is just one part of the equation. A big part of the equation is what is the loss objective and what is the training data. So as I was alluding to before, what we want are these models to be able to do planning and reasoning over multiple steps. Look at what they did in their context before and refer to that and use that as a way to inform their next steps in terms of taking actions. And for them to be able to do that, the fundamental mechanism in Transformers that enables this is the attention mechanism. Whether the attention mechanism works well over long context or not is a question that we're definitely taking a very strong look at—as to whether that's the best way for us to achieve long-form reasoning. We do know that long-form retrieval, like if you want to retrieve over millions of context length, is achievable today with large language models. Reasoning over longer context is a harder problem. The second bit that is extremely important is the training data that goes into these models. And the third bit that's extremely important is the loss objectives. So believe it or not, we have actually trained these models to probabilistically predict the next token. What is the most likely next token? And out of that we have managed to get these models to be extremely strong reasoners. But is that the best objective one could go after? Embedding the fact that we need these models to have more fundamental long-form reasoning capabilities and need to be able to learn how to learn from their contexts.

嗯。我从中听到的是，你在预训练中包含了少许架构上的调整。你知道，当你谈论重新思考预训练时，这包括可能调整注意力机制的方式，但你不认为——好吧，你相信我们可以用现有的东西做得更好，而且你并不一定需要状态空间模型或某种花哨的下一代东西。可以说，你是在骑你现有的那匹马。

Yeah. What I heard in there was a little bit of like you include a little bit of architectural shift in pre-training. You know, when you talk about rethinking pre-training, that is inclusive of maybe tweaking the way attention is done, but it's not—you don't believe that—well, you believe that we can do better with what we have and you're not necessarily requiring state space models or some kind of funky next-generation thing. You think you're riding the horse that you have, so to speak.

说得对。我还要补充一点：我认为你必须经历过多个模型世代的预训练，才能知道该在哪里下注。预训练是一门科学，既有趣又令人兴奋，同时伴随着高资本支出的高风险。所以当你下注时，你希望尽可能确定。正因如此，根据我的经验，预训练中大多是渐进式变化，而后训练中则有更激进的改变。否则，你会花更多时间调试你改动的东西，而不是得到一个能工作的模型。我遇到过很多有趣的新架构提案，研究者把 Scaling 留给别人，因为他们没有资源，但这仍然留下一个大问题：你能否在大规模上看到预期结果。我认为，对于有 Scaling 经验并且有幸参与其中的人来说，把它做对既是一门艺术也是一门科学。这不像我们改了这个又改了那个就能直接工作。每次都是一场有趣的冒险。

That is fair. And I will add: I think you have to have gone through the pre-training experience for multiple generations of models to kind of know where to make the bets. And pre-training is one of the sciences which is both fun and exciting and high risk at high capital expenditure. So when you make those bets, you want to be as sure of your bet as possible. And that's why pre-training sees most incremental changes, and post-training sees more radical changes, in my experience. Otherwise, you spend a lot more time debugging through what you changed as opposed to getting a model that works. There are a lot of interesting proposed architectures that I come across where the researcher is leaving scaling to someone else because they don't have the resources, but it still leaves a big question as to whether you'll see the expected results at scale. I think for folks who are experienced in scaling and have had the fortune of being one, it's truly an art and a science to get it right. And it's not just like we change this thing and we change that other thing and it just works. It's a fun adventure every single time.

那么，当你考虑第一个要素——注意力时，你认为注意力需要如何演变才能更好地支持智能体？

So when you think about the first of those elements, attention—like how do you think attention needs to evolve in order to support agents better?

当我思考智能体能力时，我认为我们需要注意力机制具备的基本能力——我并不是在提出确切的解决方案——是不仅能够关注上下文长度内的内容，还能引用更早之前的内容。因为你可以想象，比如对于一个深度研究智能体，你可能收集了 100 篇文章，然后需要将它们按主题组织起来。所以你的注意力机制需要对这些文章有某种形式的摘要，需要将它们组织成一份综合报告交给用户。在这种情况下，大量繁重的工作是由注意力机制完成的。那么，你如何看待这种长形式推理？词元在关注什么？这够不够？这些都是我们提出的问题。

When I think about agentic capabilities, I think the fundamental bit that we need out of the attention mechanism—and I'm not proposing exactly the solution—is the ability to look at not just things around its context length, but also being able to refer to things that were further in the past. Because as you can expect, for specific examples of say a deep research agent, you might have collected 100 articles, and then you need to somehow organize them into topics. So your attention mechanism needs to have some form of a summary of these articles, in that it needs to somehow organize them into content that is a synthesized report that you will give to the user. So a lot of the heavy lifting in this case is being done by the attention mechanism here. So how do you think about that long-form reasoning? What is the token attending to? Is that enough or is that not enough? Those are the kind of questions we ask.

在研究和实验室中，有很多工作关注记忆架构，以及用不同的记忆思考方式来扩展或增强当前的大语言模型方法。你如何看待这一点，相对于我们关于增强注意力的讨论？

There's a lot of work happening in research as well as in labs looking at memory architectures and expanding or augmenting current LLM approaches with different ways of thinking about memory. How do you think about that relative to this conversation about enhancing attention?

谢谢。我认为记忆是注意力机制之外的一个额外工具。所以在任何给定时间，我们希望语言模型能够很好地推理该使用什么工具，而记忆是其中之一。只要它们是强大的推理者，它们就应该能够回溯到一个索引，说这是我需要查找内容的地方，这是我需要从上下文长度中回头读取的内容。如今，这些系统最终被设计成多智能体系统，你放入子智能体，然后某个智能体去获取内容。但本质上，你指的是大语言模型使用工具的能力，这个工具可以查找数据库、记忆或某些存储起来的东西。人类就是这样做的，对吧？所以那种记忆方法并不一定消除对另一种注意力方法的需求，这种注意力方法能够实现不同的回忆模型，我想。

Thank you. I think I see memory as an additional tool in addition to the attention mechanism. So at any given point in time, what we want the language models to be able to reason well about is what tool to use, and memory being one of them. And as long as they are strong reasoners, they should be able to refer back to an index and say this is where I need to look up content, and this is what I need to go back and read from the context length. Today, the way some of these systems end up being engineered are as multi-agent systems where you put sub-agents and something is going to go fetch that content. But effectively, what you're referring to is sort of the ability of the large language model to use a tool which can look up a database or memory or something that's perhaps stored away. That's how humans do it, right? So that type of memory approach doesn't necessarily remove the requirement to have a different approach to attention that enables different recall models, I guess.

是的。显然，有很多工作从提高计算效率和提升长上下文性能的角度来研究注意力机制。你觉得哪些研究比较有启发性，并且朝着你认为需要发生的方向去更好地支持智能体？

Yes. Clearly there's a lot of work being done looking at the attention mechanism both from the perspective of increasing computational efficiency and increasing long context performance. What research out there do you find kind of inspiring and in the direction of what you think needs to happen to better support agents?

你会发现我在架构问题上极其保守，因为我 Scaling 过太多模型，调试过太多模型，基本上一直在前线。

You'll find me extremely conservative on architecture stuff because I have scaled too many models and debugged and been in the trenches for too many models basically.

我训练过五个 Scaling 模型，每次都得亲自调试。我不是唯一的人，通常会有作战室和几个人，但我是那个必须在一线的人。

I've trained five scaled up models and every single time I have to debug them. I'm not the only one. There's usually a war room and a few other people, but I'm the person who has to be in the trenches.

这让你学到了什么？

What has that informed for you?

必须有充分的理由，而且必须对评估极其严格。对 Scaling 的方式也必须极其严格。那些看似无害、感觉有趣又令人兴奋的改变，在规模化时行不通。

There has to be a good reason, and one has to be extremely rigorous about evaluations. One has to be extremely rigorous about the way you do scaling. Innocuous changes that feel fun and exciting don't work at scale.

嗯。

Mhm.

所以下注必须有充分理由。每一个赌注都经过严格审查。因此，Transformer 相比下一个最好的东西，已经得到了极其充分的验证。

So there has to be good reason to make that bet. Every single bet is extremely well vetted. Consequently, Transformers have been extremely well vetted relative to the next best thing.

我认为对于像 Reflection 这样的初创公司，我们也必须让第一个赌注尽可能强大。

I think for a startup like Reflection, we also have to make the first bet to be as strong a model as possible.

我们可能想调整注意力机制，但这只是预训练配方的一部分。还有微调损失函数。你刚才谈到损失函数以及人们思考它的一些方式。你能详细说明一些损失调整如何直接与智能体性能挂钩吗？比如，如果你考虑 LLM 的下一个词预测，以及我们最初期望 LLM 完成的任务，比如生成文本，那是有道理的，很直观。但当我们考虑智能体时，我们需要这些智能体擅长更多事情。推理是你提到的一点，工具使用是另一点。有哪些方法可以调整损失函数来支持这些行为？

There are ways that we might want to tweak attention, but it's only one part of the pre-training recipe. There's also fine-tuning the loss. You talked a little bit about the loss and some of the ways that people are thinking about that. Can you elaborate on how some of the loss tweaks tie directly to agent performance? Like if you think about next-token prediction for an LLM and the initial tasks we expected LLMs to do, like generate text, that makes sense. It's intuitive. When we think about agents, there are so many more things we need these agents to be good at. Reasoning is one thing you mentioned. Tool use is another. What are some of the ways we can tune the losses to support those types of behaviors?

过去有一些做法，而且广为人知：当我们训练代码模型仅作为自动补全模型时，我们不只是给它们代码文件然后训练下一个词预测。我们采用了填充中间（fill-in-the-middle）的方法，实际上是把文件拆开，把前半部分放到后半部分，中间部分放到末尾，让模型去预测。如今我们训练这些模型用于工具使用时，实际上会屏蔽某些部分，让模型学会使用哪个工具。这不仅仅是关于使用哪个搜索查询。我认为这些方面在你训练模型的方式中被屏蔽了，这样你就可以通过屏蔽来教导模型。如果你回顾语言模型的历史，比如 BERT 等，掩码语言建模是一个被广泛探索并以有意义的方式利用的目标。所以我的看法是，你关注什么在损失目标中也很重要。屏蔽是一种特定方式，而训练数据的增强是另一种方式，让模型关注每个序列中特定部分。这教会模型更多地关注或预测那部分，而不是其他部分。

Some things that have been done in the past, and this is extremely well known, is that when we were training coding models to just be autocomplete models, we don't just train them on code files and then train them on next token. We have this fill-in-the-middle where we actually break up the file, put the first half in the second half, and then the middle part goes at the end so the model has to predict that. When we train these models for tool use today, we actually mask out certain portions so that the model learns which tool to use. It's not just about which search query to use. I think those aspects get masked out in how you train the model so that you can teach the model by masking. If you go to the history of language models, for example BERT and so on, masked language modeling was an objective that was heavily explored and leveraged in meaningful ways. So the way I look at it is that what you pay attention to is also important in the loss objective. Masking is one particular way, and how the training data is augmented is another way to get the model to pay attention to specific parts of what is there in each sequence. That teaches the model to pay attention to that part or predict that part more versus the other.

从损失函数的角度看，你似乎在说，我们在后训练中做的很多事情并不一定带来新的预训练内容，而是找到方法将后训练中的一些东西以可扩展的方式拉入预训练，使它们成为模型更基础的能力。

From a loss perspective, it sounds a little bit like what you're saying is that a lot of what we're doing in post-training has not come out with new pre-training things necessarily, but it's finding ways to pull some of the things that we're doing in post-training into pre-training in a scalable way so that they're more fundamental capabilities of the models.

是的，我会说它统一了这两个范式。我认为预训练和后训练从根本上源于这样一个世界：我们必须从预训练开始，因为那是我们看到 Scaling 在放大自然语言能力方面带来最大收益的地方。而后训练让我们能够将这些模型作为聊天机器人更易使用。当你让它们与基于人类反馈的强化学习（RLHF）配合工作时，它们变得非常强大。随着我们进入智能体式范式，推理是你想在最后阶段赋予模型的能力，还是应该提前？我们是否需要在这两个范式之间划出严格的界限？这是一个值得问的好问题。

Yes, I would say it unifies the two paradigms. I think pre and post is fundamentally coming from the world of we had to start out with pre-training because that's where we saw scaling gave us the largest set of gains in terms of amplifying natural language capabilities. While post-training allowed us to make these models more accessible as chatbots. And they were extremely strong when you got them working with say reinforcement learning with human feedback. As we move into this agentic paradigm, is reasoning a capability you want to endow in the model in the last stage or does that move up front? And do we need a very strict divide between the two paradigms? That's a great question to ask.

从训练数据的角度，谈谈你认为它需要如何演变。

From a training data perspective, talk a little bit about how you think that needs to evolve.

预训练模型从根本上依赖于大量高质量且多样化的数据。我认为我们利用得最多的一点是尽可能追求规模。所以数据越多越好，并且尽可能追求多样化的数据源。我认为我们从东方的同行那里看到的，比如 Qwen 模型、DeepSeek 模型或 Kimi 模型，是训练数据的整理质量也很重要。所以我认为这是一个轴，当你让训练数据极其高质量时，你会从预训练模型中获得更多的算力效率。第二个重要的轴更偏向推理。那么如何获得真正好的推理轨迹？这几乎就像今天问的问题：如果你在互联网上训练，推理会默认涌现，但如何获得那些人们在所有可能领域解决问题的专家轨迹？今天我们通过让这些模型在强化学习环境等中探索来做到这一点。但一旦我们对这些模型如何生成这些轨迹有了一些理解，我们应该放什么以及预训练中有什么可用的规模，就变成了一个问题。所以这暗示了一个事实：什么样的好推理轨迹可以大规模反馈给模型，是我们正在思考很多的问题。

Pre-trained models are fundamentally dependent on large volumes of data that is high quality and diverse. I think the bit that we have exploited the most is to go for as much scale as possible. So more and more data is always better, and go for as diverse data sources as possible. I think what we have seen from our counterparts in the east, like Qwen models or DeepSeek models or Kimi models, is that the quality of curation in training data matters as well. So I think that is one axis in which you get a lot more compute efficiency out of pre-trained models when you get your training data to be extremely high quality. The second axis that matters is more along the axis of reasoning. So how do you get really good reasoning traces? It's almost like asking the question today: if you train on internet, reasoning emerges kind of by default, but then how do you get those expert traces of people working on problems in all possible domains? Today we are doing them by getting these models to explore in say reinforcement learning environments and whatnot. But once we have some semblance of understanding how these models can generate those traces, what should we put and what is available at scale in pre-training starts to become a question. So this kind of alludes to the fact that what would be good reasoning traces that can feed back into the models at large scale is a question that we're thinking a lot about.

你感觉推理轨迹的 token 量与核心训练数据的 token 量相比如何？相差几个数量级，只是总训练数据中很小的一部分。如果它在预训练中被 Reddit 闲聊远远超过，你怎么让它对训练过程产生影响？

Do you have a sense for the token volume of reasoning traces compared to the token volume of core training data? It's off by several orders of magnitude. It's a very small fraction of the total training data. How do you make it so that it makes a difference in the training process if it's so outnumbered by Reddit chatter in pre-training?

这是个合理的观点。我认为值得问的问题是：主要的预训练数据源是什么，以及如何增强它以生成那种数量的数据？有一些方向上的线索。

That's a fair point. I think the question worth asking is: what is the dominant pre-training data source and how can that be augmented to generate that kind volume of data? There are some pointers in that direction.

比如？

For example?

例如，主要的训练数据源最终会是互联网上的文章。

For example, the dominant training data source would end up being say articles on the internet.

你也会利用互联网上的文章，但以有趣的方式增强它们。已有一些结果表明，你可以将它们转化为问答对或对话形式，或者如果是解决任务的问题，你也可以用有趣的方式增强它们。所以这方面的工作已经开始出现，但在我们走上完全由新生成数据源构成的道路之前，你必须利用基础数据量来生成类似规模的数据。

How you use articles on the internet too, but augment them in interesting ways. There have been some results which show that you can formulate them into question-answer pairs or into conversational things, or if you have problem-solving tasks, then you can augment them in interesting ways. So there is starting to be work in this direction, but you have to tap into the fundamental volumes of data to generate similar volumes of data before we can go down the path of completely new generated data sources.

我听到你在谈数据量，我也在试图拼凑起来：如果我们要对预训练所用训练数据的很大一部分进行推理，以提取问答对或推理轨迹之类的东西，我在想这是否可行。听起来很贵。

I hear you talking about the volumes of data, and I'm also trying to piece together: if we're talking about needing to do inference on some significant fraction of all the training data that we use for pre-training in order to extract question-answer pairs or reasoning traces or things like that, I'm trying to decide if I think that's tenable or not. Sounds expensive.

你觉得有多贵？DeepSeek 训练了 15 万亿个 token。我不能透露 Gemini 的具体数字，但假设预训练方面有几十万亿个 token。那么，如果你每百万 token 的成本是零点几美元，再加上一些小的倍数，其实并不算太贵。

How expensive do you think it is? I mean, DeepSeek trained on 15 trillion tokens. I can't say the numbers for Gemini, but let's assume it's tens of trillions of tokens on the pre-training side. Well, then it's not horribly expensive if you're doing a fraction of a dollar per million tokens in and some small multiple of that maybe.

嗯。

Yeah.

我认为值得回答的问题是，长期以来这并非重点领域，因为从代表人类工作流程的自然数据中总能获得更好的性能。合成数据存在挑战，模型表现不佳，因为你并没有以有意义的方式扩展数据分布。那么在这个特定案例中，你的基本赌注是什么？我想我暗示的是，这正是我们思考的轴心：我们真正关心的模型能力，而要获得这些能力，我们会走这条路。

I think the question worth answering is that for the longest time, that has not been a focus area because it was not where you always get better performance out of natural data that is representative of human workflows. Synthetic data has challenges in the sense that the models don't perform as well because you're not actually expanding the data distribution in any meaningful ways. So what is the fundamental bet you're making in this particular case? I think what I'm alluding to is that this is the axis along which we are thinking: here are the model capabilities we fundamentally care about, and to get those capabilities, we would go down this path.

我想在合成数据训练的背景下，另一个常被提出的挑战是“自产自吸”问题。你有没有感觉，特别是当你扩大合成数据量时，最终如何克服这个问题？

I guess the other challenge often raised in the context of training on synthetic data is the "smoking your own exhaust" problem. Do you have a sense for how, especially if you're scaling up the amount of synthetic data, that is overcome at the end of the day?

你如何判断什么有帮助、什么有害，其根本基础取决于底层数据分布。如果你大幅改变分布，使其完全由合成数据生成，我预计模型表现不会好。所以这里的目标是尽量保持底层数据分布非常接近你希望模型训练所基于的自然数据分布。

The fundamental underpinnings of how you determine what is going to help and what's going to hurt is dependent on what's the underlying data distribution. If you are changing the distribution substantially to be all synthetically generated, I expect the models will not do well. So the goal here is to very much keep the underlying data distribution to be very representative of the natural data distribution that you want the models to be trained on.

我认为这自然引出了关于这些任务的测量和基准构建的讨论。你目前看到哪些方向是正确的？

I think that leads us naturally to talking a little bit about measurement and benchmark construction for these types of tasks. What have you seen so far that you think is in the right direction?

好问题。基准的有趣之处在于，每年都会出现一些新的，然后它们被饱和，接着你又得构建新的。对于这些模型的前沿开发，我们目前形成的范式是，每个团队内部构建基准来衡量他们追求的能力集。例如，我们希望模型能更好地进行长上下文推理，能处理多步问题，所以它们不仅要解决下一步，还要考虑多步之后的行动。我们希望模型能从失败的轨迹中恢复，更好地规划，并学会使用工具。所有这些能力都需要非常具体的测量方式。尽管我给出的是总体概念，但当你将它们分解为实际的编码任务或 inbench 风格的任务时，我们开始看到一些具有不同视野的任务出现。例如，METR 有“软件原子动作”这个东西，允许你有任务长度，还有其他一些基准。你会发现，你基本上需要复杂度不同、能力维度也不同的任务来测试模型。这几乎就像把你一天的真实工作流程分解，模型可以处理哪些子问题？即使这些子问题很难，难在哪里？将其作为模型的挑战，大致就是许多基准的构建方式。

That's a great question. The fun thing about benchmarks is that every year a few new ones come out, then they get saturated, and then you have to build new ones. For a lot of the frontier development for these models, the current paradigm we've settled on is that everyone builds benchmarks in-house to measure the set of model capabilities they're going after. For example, we want these models to be able to reason better in long-form reasoning over their context length. We want them to be able to do multi-step problems, so they need to tackle not just the next step but what their actions might be multiple steps from now. We want these models to be able to recover from failed trajectories, plan better, and learn how to use tools. All of these capabilities require very specific ways of measuring. Even though I'm giving you overarching concepts, when you break them down into actual coding tasks or inbench-style tasks, we're starting to see some of these tasks with varying horizons come into picture. For example, METR has this "software atomic actions" thing that allows you to have task lengths, and there are a few other benchmarks. What you'll see is that you basically want tasks of varying complexity and also varying along these dimensions of capability to test the models. It's almost like if you were to break down your real workflow in a day, what are the sub-problems the models could go after? And even if those are hard, why is it hard? Making that a challenge for the model is roughly how a lot of these benchmarks are constructed.

为了让你明白为什么这非常重要：三四年前，我们在训练 PaLM 时，它是一个大型模型，但我们尝试了一个新的基准，这个基准是通过研究人员众包构建的，他们提出了当时大型语言模型无法完成的推理任务。我们在刚训练好、实际上只训练了一半的模型上测试了这个基准。我们发现这个基准突然出现了阶跃变化，这让我们意识到模型实际上开始展现出推理能力。这在某种程度上很迷人，但如果我们没有那么多基准可供参考，包括理解表情符号或模型如何推理电影剧本等，我们就无法获得这种洞察。所以我认为，以正确的方式收集这些问题，并理解当前一代模型难以处理什么，往往是向下一代模型取得进展的方法。

Just to give you a sense of why this is super important: back three or four years ago when we were training PaLM, it was a large model, but we tried it on a new benchmark that was crowdsourced through researchers who were coming up with reasoning tasks that they found large language models could not do back in the day. We tried our freshly trained, actually halfway trained model on this benchmark. What we found was that suddenly this benchmark had a step change, and that led us down the path of figuring out that the model was actually starting to showcase reasoning capabilities. That's fascinating to some extent, but that would not really have been available had we not had that plethora of benchmarks to look at, including understanding emojis or how the model reasons over a movie script or whatnot. So I think sourcing these questions in the right ways and understanding what is hard with the current generation of models is often the way to make progress towards the next set.

你在多大程度上看到这些——我认为这个问题适用于我们讨论过的所有关于预训练的影响，但特别是对于测量——你如何看待哪些事情社区会提供，或者当你需要时大家会朝着同一方向前进，而像 Reflection 这样的小实验室又需要承担哪些工作来将其整合到自己的模型中？

To what extent do you see these—and I think this question applies to all the implications on pre-training we've talked about, but for measurement in particular—how do you think about which of these things the community will provide or everyone's moving in the same direction by the time you need it, versus what a small lab like Reflection takes on to try to incorporate into your own models?

我谈到的基本能力集，大概三四个，一只手就能数过来，如果我们相信这些是基本路径，我们肯定会优先考虑它们。

The fundamental set of capabilities that I talked about, which is roughly three or four that I can count on one hand, are things that if we believe those are fundamental paths, we'll definitely prioritize them.

除了常规的基准测试之外，我还认为我们才刚刚开始让强化学习与大型语言模型良好配合。第一个结合强化学习的推理模型实际上是一年前才出现的。因此，我相信很多在现实世界中尝试的工作流程将在未来一年内被纳入基准测试。我们将会有更多代表真实工作流程的基准测试，以具有挑战性的方式测试模型，并帮助我们了解模型的优势与不足。

In addition to the usual array of benchmarks, I also believe that we are just at the start of getting reinforcement learning to work well with LLMs. The first reasoning model with reinforcement learning really came out a year ago. So I do believe that a lot of the workflows being tried out in the real world will make it into benchmarks over the next year. We will have more benchmarks that are representative of real-world workflows, testing models in challenging ways and helping us learn where the models are challenged versus not.

我认为规划预训练的有趣之处在于，它最终是一项长期的工作。你必须提前思考你追求的是哪些基础模型能力，并为此进行规划。当你思考这个领域相对于更大模型和缩放定律的发展方向时，一方面是对不断增大的模型的追求，另一方面我们也看到对更小、更高效模型的兴趣重新兴起——你瞄准的是哪一个方向？

I think the fun part about planning pre-training is that it ends up being a longer-term endeavor. You have to think ahead of time about what fundamental model capabilities you're shooting for and plan for that. When you think about the direction the field is heading relative to larger models and scaling laws, the march towards ever-increasing models, and then we've seen a resurgence in interest in smaller, more efficient models — which of those are you aiming for?

当你追求更大规模的模型时，你通常能比在小模型上更早地发现能力。通常，模型缩放是我们注意到以前未见过的新能力的一种方式。通常，通过正确的训练数据和更长的训练时间，在大规模上获得的能力也会在小模型上出现。例如，o1-mini 是在最大的推理模型构建之后出现的——假设是 GPT-4。你拿最大的模型，对它们进行后训练以增强推理能力，然后下一代模型即使在小模型上也拥有了这种能力。因此，我们追求的是最有能力的模型，然后是最具成本效益的能力模型。但通常，要发现这种能力，你必须达到适当的规模，然后才能回来让它们变得成本高效。这通常涉及额外的蒸馏步骤，这很有帮助。所以这就是看待它的方式。

When you go for larger-scale models, you often find capabilities much sooner than you will in smaller models. Typically, model scaling is one way in which we notice model capabilities that we haven't noticed before. Often, the model capabilities that become available at scale, with the right training data and training longer, also become available in smaller models as well. For example, o1-mini came out after the largest reasoning model was constructed — let's assume GPT-4. You take the largest models, post-train them for reasoning, and then the next generation of models had that capability even in the smaller models. So what we are shooting for is the most capable models, and then the most cost-efficient capable models. But often, to discover that capability, you do have to go to the right scale, and then you can come back and make them cost-efficient. That often involves an additional step of distillation, which helps. So that's the way to look at it.

我想你大致理解了我的意思。也许问题是：作为一个小型实验室，鉴于人们对小模型有很大兴趣，你是否可以直接瞄准构建一个小模型？我听到的回答是并非如此，因为很多能力是涌现的，你需要达到一定的规模才能看到它们。因此，你需要在资源允许的范围内尽可能做大，然后再考虑如何变小，以便获得小模型的所有好处。

I think you kind of got what I was getting at. Maybe the question is: as a small lab, given that there is a lot of interest in small models, can you just target building a small model? What I'm hearing is not really, because a lot of these capabilities are emergent in the sense that you need to hit a certain level of scale to see them. So you want to go as big as you can given your resources, and then worry about making smaller later in order to achieve all the benefits of smaller models.

是的。通常，至少根据我处理大型语言模型和训练它们的经验，涌现是指你在更大的模型中发现有趣的东西，然后你通常会在下一代中通过蒸馏或其他方式将它们放入小模型，因为你已经找到了获得这些能力的确切方法。

Yes. Often, at least in my track record and history of playing with large language models and training them, emergence is something where you find fun and interesting things in the larger models, and then you often distill them or get them in the smaller models in the next generation, because you figured out exactly the recipe it takes to get them.

我们在整个对话中随意地讨论了长形式推理这个概念。深入探讨一下它的含义——相对于短形式，长形式是什么？

We've talked about this idea of long-form reasoning casually throughout this conversation. Dig into what that means — what is long-form relative to short-form?

我会给你非常具体的例子来说明什么是长形式推理。例如，如果你想让一个 AI 模型理解你的整个代码库，包括另一个团队编写的部分，或者如果你有一个大型组织，你想查看代码库的不同部分并理解你点击的任何子集。长形式推理允许你引用你的团队正在处理的代码库部分，并像使用库一样无缝地使用它，因为 AI 可以帮助你理解和使用它。你不需要走到那个团队面前，花时间寻找团队成员。这是长形式推理的一种形式。粗略地说，这意味着如果你在上下文中放入很多东西，模型能够推理出在上下文窗口中位置相距甚远的不同事物。另一种看待方式是，检索代码库的相关部分是一个较容易的问题——大海捞针或多针捞针是一个问题。但是当你转向具有多跳推理的基准测试时，比如 MuSiQue 或 LoFT，这些基准测试直到最近才开始显示出模型正在变得更好的信号。在很长一段时间里，即使是非常长上下文的模型，这些基准测试也没有显示出强大的能力。即使你的上下文长度有数百万个 token，你也不一定能够思考所有 token。如果你放入非常混乱的文档集——比如你想为度假做深度研究，有 10 篇关于你最喜欢的度假胜地的文章——模型不一定能够思考每一篇文章并给出综合结果。这是一个更难的问题。所以这是长形式推理的一种形式。第二种形式更侧重于轨迹。当你希望模型进行多步推理时，你实际上是在要求模型提前思考，就像下棋一样，你提前思考，下一个行动应该让你朝着目标前进，并有一定的预测感。这一点以前并不存在，因为它们实际上只是生成下一个 token。你如何推理这组下一个 token 会把你带向何方？就下一个行动的轨迹而言，这意味着什么？此外，过去如何影响到现在？如果你查看一个存在失败的现有轨迹并需要恢复，今天的模型通常会陷入重复循环，因为它们并不知道自己犯了那个错误。它们会再次尝试同样的事情，因为那是最可能的选择。这些是长形式推理的一些例子，在当今的模型中非常相关。

I'll give you very specific examples of what long-form reasoning is. For example, if you wanted an AI model to understand your entire codebase, including parts written by another team, or if you have a large organization, you want to look at different parts of the codebase and understand any subset you tap on. Long-form reasoning allows you to reference that part of the codebase your team is working on and use it like a library without thinking about it, because AI can help you understand and use it seamlessly. You don't have to walk up to that team and spend time finding team members. That's one form of long-form reasoning. Roughly, it means if you put a lot of things in context, the model is able to reason about disparate things that are far apart in position in the context window. Another way to look at it is that retrieving relevant parts of the codebase is an easier problem — needle in a haystack or multiple needles in a haystack is a problem. But when you go to benchmarks that have multi-hop reasoning, like MuSiQue or LoFT, these benchmarks have only recently started to show signal that models are getting better. For the longest time, these were not benchmarks where you saw strong capabilities even in very long context models. Even with millions of tokens in your context length, you're not necessarily able to think about all of them. If you put very confusing sets of documents — say you want to do deep research for a vacation and have 10 articles about your favorite vacation spot — the model will not necessarily always be able to think about each article and give you a synthesized form from each one. That's a harder problem. So that's one form of long-form reasoning. The second form is more along the lines of trajectories. When you want models to do multi-step reasoning, you're asking them to think ahead, like in chess, where you think ahead and the next action should move you toward your goal with some sense of projection. That bit has not existed because they are realistically just generating the next token. How do you reason about where this next set of tokens will take you? What does that mean in terms of trajectories with the next action? Also, how does the past affect up to this point? If you're looking at an existing trajectory with failures and need to recover, often models get stuck in repetitive loops today because they don't really know they made that failure. They'll try the same thing again because that's the most probable thing. Those are some examples of long-form reasoning that are pertinent in today's generation of models.

我原以为你要区分的，是模型直接吐出答案，和模型经历思维痕迹与推理时 Scaling 之间的区别。但听起来，你描述的长篇推理至少有一部分指向的是抽象推理。也就是说，不一定是从 A 到 B 到 C，而是你了解 A，了解 B，你在解决某个可能受益于这些知识的独立问题 C。你如何将其融入解决方案？

I thought the distinction you were going to make was between the model spitting out an answer and the model going through thought traces and inference time scaling. But it sounds like at least part of what you're pointing to in describing it as long-form reasoning is abstract reasoning. Meaning, not necessarily going from A to B to C, but you know about A, you know about B, you're working on some separate problem C that might benefit from that knowledge. How do you incorporate that into the solution?

是的。而且我认为你提到的第一点是推理的自然演进。所以我在说的是，我们接下来该往哪里走？

Yes. And I see the first thing that you brought up is a natural evolution of reasoning. So I'm talking about like where do we go from there?

在你的描述中，你稍微谈到了失败，以及模型从失败中恢复的能力，智能体从失败中恢复的能力。深入谈谈这一点，以及它所暗示和要求的。

In your description, you talked a little bit about failure and the model's ability to recover from failure, the agent's ability to recover from failure. Dig into that a little bit and what that implies and requires.

通常当你考虑轨迹时——让我先定义一下“轨迹”这个词。假设你在 terminal bench 或 su bench 中解决一个编码任务，或者一般性地解决一个目标导向的任务。那么我们会制定一个计划：比如这些是要查看的相关文件集合，这些是有助于实现目标的变更集合，然后你会尝试这些变更，然后你会有一个验证过程，比如运行测试来验证这些变更是否能解决问题，对吧？当你收集这样的轨迹时——这里是有相关上下文，这里是我的解决方案，这里是我尝试的内容，这里是执行反馈的输出——你会发现，在多个步骤中，模型可能需要弄清楚从过去步骤中学到的东西不应该重复，或者应该在后续步骤中选择不同的行动集合。这对模型来说是一个更难的问题，既从注意力的角度——这些是离散的事物，所以它需要选择新的行动空间——这在一定程度上是一个强化学习问题，但也是一个预训练问题：模型在关注什么？它是否理解这些是失败的步骤？它是否以正确的格式传递了轨迹，从而注意到这些是需要纠正的地方？所以这既是一个强化学习目标问题，也同样是一个模型能够关注和学习的上下文长度问题。

Typically when you think about trajectories and let me define the word trajectories. So let's say you are solving a coding task in say terminal bench or in su bench or you're solving a goal-oriented task in general then we will have come up with a plan of like these are the relevant set of files to go look at, here are the set of changes that would help achieve the objective and then you would go try those changes and then you would say have some form of a verification process where you would go run tests for example to verify whether these changes will solve the problem or not. Right? When you collect such traces where here was the relevant context, here was my solution to the problem, here is what I tried and here is the output the execution feedback from trying this. What you'll find is that over multiple steps the model might need to figure out what it learned from the past step should not be redone or like it should choose different set of actions in the subsequent set of steps and that's a harder problem for the model both from the perspective of paying attention to like okay these are discrete set of things so it needs to choose new action space so that's partly a reinforcement learning problem but it's also a pre-training problem as to what was the model paying attention to? Did it understand that these were failed steps? Did it have the right trajectories passed with the right format so that it noticed that these were things to correct on. So it's both a reinforcement learning objective problem, but it's as much a problem of like what is the length of context the model is able to pay attention to and learn from.

听起来你在谈论推理时的失败，以及模型对这些失败类型的意识。是的，我们希望模型学会如何学习以及如何自我纠正。

It sounds like you're talking about inference time failure and an awareness on the part of the model of these types of failures. Yeah, we want the model to learn how to learn and how to correct itself.

我想这个问题是要将其与幻觉作为 LLM 的一个属性联系起来，以及这些失败在多大程度上源于某种幻觉。如果这是 LLM 的一个基本属性，你想象如何在不修复它的情况下克服它？

And I guess the question was going to relate that to hallucination as a property of LLMs and the degree to which often these failures come from hallucination of one type or another. And if that's like this fundamental property of LLMs, how do you imagine that you would overcome it without fixing that?

我倾向于重新表述——我的意思是，取决于业务目标，幻觉是一个正面或负面的词。

I tend to rephrase I mean depending on the business objective hallucination is a positive or a negative word.

我不是在评判它。我只是说这是会发生的事情。所以如果模型——如果我们谈论的这些失败是模型认为某件事是真的，但实际上不是，而模型没有意识到这一点。你如何赋予模型这种意识？如果我们能做到这一点，我们不是已经修复了幻觉吗？

And I'm not making a judgment about it. I'm just saying that it's something that happens. And so if the model, if these failures that we're talking about are when the model thinks something is true and it's not true but the model doesn't have an awareness of that. How do you give the model that awareness? And if we could do that, haven't we fixed hallucination?

所以我把幻觉更多地看作：从概率上讲，模型认为应该输出的最高概率的下一组 token 是什么。所以对我来说，是否有其他东西可以进入它的上下文，或者它是否可以通过与一组工具交互获得额外的反馈，从而以某种方式改变方向？对吧？所以这就是我在问的问题：给定目标和它为实现该目标所尝试的所有事物的历史，是否存在一组它可以输出的概率性 token，这些 token 有更高的概率推动它朝着目标前进？

So I see hallucination as more of like what probabilistically was the highest probable next set of tokens that the model thought it should output. So to me was there anything else that could have made it into its context or could it use could have gotten additional feedback from interacting with a set of tools to change course in certain ways. Right? So that's the question I'm asking like given the goal and given the history of all the things that it has tried to achieve that objective is there a next set of probabilistic tokens that it can output that would have a higher probability of moving it forward towards the goal.

那么当你谈到这种可恢复性时，是在推理时，对吧？

And so when you talk about this recoverability is that at inference time, right?

推理时。目标是在推理时实现它。

Inference. The goal is to get it at inference time.

那么你——

And are you

这不是一个已解决的问题。我就直说了。

It's not a solved problem. Let me just say that.

是的。不，我明白。我明白。你是否设想模型有某种意识，比如当它没有足够信息时，会要求更多信息或停止，而不是继续？

Yeah. No, I get that. I get that. Are you envisioning like the model having some awareness of when it doesn't have enough information for example and asking for more information or halting as opposed to continuing?

是的，它没有足够的信息来验证它提出的方案是否正确。所以它去尝试其他几种方式来验证它正在尝试的方法，然后回来说：好的，现在我有足够的信息来做出明智的决定。

So yes it doesn't have enough information to verify what it proposed is the correct solution. So it goes and tried a few other ways to like verify the approach that it was trying and then comes back and it's like okay now I have enough information to make an informed decision.

好的。

Okay.

或者向用户求助，给我指个方向。

Or ask the user for help point me in the right direction.

对，对，对。而不是缺乏对信息不足的意识，只是用错误信息或编造的信息硬撑下去。

Right. Right. Right. As opposed to a lack of awareness of not having enough information and just powering on through with wrong information or information that they made up.

是的。或者自信地告诉用户它得出了正确答案。

Yes. Or confidently telling the user that it came up with the right answer.

你如何看待这个特定方面——可恢复性——与我们之前讨论的预训练杠杆（注意力、训练数据、损失测量）联系起来？

How do you see this particular facet recoverability tying back to the pre-training levers that we talked about attention training data loss measurement?

所以我认为归根结底是训练非常强大的推理器，在训练数据和损失目标方面都有一些我们认为会有帮助的杠杆。这实际上归结为非常强大的推理器，能够在上下文长度上以及跨领域进行良好推理。

So I think it comes down to training very strong reasoners and there are some levers both on training data side and loss objective side that we think will help here really comes down to really strong reasoners that can reason well over context length and in general across domains.

在考虑构建能够学习新工具的智能体时，谈谈这个。我立刻想到了 MCP，以及我们看到的这场闹剧：哦，MCP 是个好东西，让我们都用 MCP 吧。哇，MCP 撑爆了上下文，用的比智能体要解决的问题还多。不如把它委托给智能体可以编写的软件程序，这样它就不需要那么多上下文了。然后，哦不，这有点难。不如让智能体能够搜索它的工具。如果这个列表中有任何东西迫切需要更基础的解决方案，那可能就是工具使用和智能体学习新工具的能力。这是你在描述时谈到的那种事情吗？

And when thinking about building agents that can learn new tools, talk a little bit about that one. MCP immediately comes to mind and like this drama that we've seen with oh hey MCP's this great thing. Let's all use MCPs. Whoa, MCPs blow up the context and use more of it than the question that the agent is trying to solve. Let's instead delegate it to software programs that the agent can write so it doesn't have all that context. And then, oh no, well that is kind of hard. Let's actually make it so that the agent can search its tools. If there's anything on this list that cries for a more foundational solution, maybe it's like tool use and an agent's ability to learn new tools. Is that the kind of thing that you're talking about when you describe this?

是的。

Yes.

所以我认为，我们目前教会模型学习的方式，本质上就是上下文里放什么，模型就学什么。如果我们能引导模型去接触终端工具，并通过交互让它们学会这些工具有用还是没用，这其实在过去的持续学习或机器人领域的模仿观察中已经有过尝试——就是观察工具是如何被使用的。那么问题又回到了：我们如何让模型学会元学习，也就是学会如何学习？探索新环境需要以什么样的格式进行，才能让模型学会在该空间中采取行动？一个有前景的方向是某种统一的领域特定语言，也许采用代码格式，能让这种推理能力涌现。这方面已经有一些论文了。但总的来说，我确实相信，通过探索新工具并从尝试中获取反馈来学习，能让模型比要么把所有东西塞进上下文窗口，要么用所有正确轨迹进行微调，要么修剪进入上下文的工具集以构建完美系统的方式更加灵活和强大。所以我认为要真正实现这一点，还有很多工作要做。目前还不能直接拿来用。

So I think what we have taught our models to learn so far is really what we put in context is what they know. If we could point them in the direction of, here are end tools, and by interaction they are able to learn how these tools are useful versus not, that is something that has been tried in the past in the form of continual learning or in the world of robotics like watch by imitation. So watch by how this tool is being used. So the question comes back to how can we get the models to metalearn, like learn how to learn? What exactly is the format in which this exploration of the new environment needs to be done so that the model learns to take actions in that space? One of the promising directions there has been some form of unified domain-specific language, perhaps in code format, that would allow for this kind of reasoning to emerge. There have been a few papers in that direction. But overall, I do believe that the ability to explore new tools and learn by exploration and feedback from trying those tools allows for more flexible and capable models than either having to fit them all in context length or having to fine-tune them with all the right traces or pruning the set of tools that goes into the context so that you have the perfect system built to go in. So I think there is a fair bit of work to happen to get that going. It's not off the shelf available today.

在整个对话中，你反复以编码智能体、代码开发智能体作为基准。你认为你为这些编码智能体产生的想法在多大程度上适用于工作流智能体？比如面向企业工作流或消费者工作流的智能体，不是代码生成，而是用一堆工具生成行程或解决一些工作流任务。同样的原则是否都适用？还是略有不同？

Throughout the conversation you've returned to coding agents, code development agents as a touchstone. To what degree do you think that the ideas that you're generating for these coding agents apply to workflow agents? You know, agents for enterprise-oriented workflows or consumer workflows, but not code generation, but you've got a bunch of tools generate an itinerary or solve some workflow tasks or set of tasks. Do all the same principles apply? Do they differ a little bit?

所以即使我们在讨论编码智能体，我仍然认为同样的原则适用，只是范围更广一些。即使在我们构建编码智能体时，甚至在那之前构建大语言模型时，我们在预训练规模上获得的一个基本能力就是模型学会了跨模态推理。这意味着它们学会了跨语言推理，学会了跨代码和文本推理，学会了在代码内部跨不同语言推理，甚至跨非文本模态推理。所以对我们来说，构建任何需要跨领域推理的东西，其基本原则都源于这种能力。因此，对我来说，基本原则仍然非常适用，可能只是后训练的最后阶段会有所不同，但模型需要能够跨领域、跨文本和语言、跨不同领域或语言等进行推理。

So even when we are talking about coding agents, I do believe the same principles apply, though slightly broader. Even when we are building coding agents, and even before that when we were building large language models, one of the fundamental capabilities that we get at pre-training scale is that the models learn to reason across modalities. What that means is they learn to reason across languages, they learn to reason across code and text, they learn to reason across different languages even within code, for example, or even modalities that are not text. So to us, the fundamental principles for building anything that requires reasoning across a crossover set of domains comes from this kind of capability. So to me, the fundamental principles still very much apply, and it's mostly a matter of maybe the last stage of post-training will be different, but the models need to be able to reason across domains, across text and language, across different domains or languages, and so on.

酷。我们聊了一点你在 Reflection 的工作。对于那些对这些想法有共鸣并想进一步探索的人，有没有相关的论文、研讨会或各种论坛？人们如何深入研究这些问题？

Cool. So we've talked a little bit about what you're working on at Reflection. For folks for whom these ideas resonate and they want to further explore this, are there papers out there or workshops or various fora? How do folks dig into these problems?

来和我们一起工作吧。

Come work with us.

我猜你们在招人。

I take it you're hiring.

是的，我们在预训练和后训练方面都在招聘。我们最近完成了一轮融资，得到了 Nvidia 的支持，我们的使命是构建前沿开放智能。我们是一个大约 60 人的研究团队，仍然非常小，希望建立一个面向开放智能体模型的前沿实验室。我们的研究人员来自 X、DeepMind、OpenAI 和顶尖大学，他们在 PaLM、Gemini、AlphaProof、AlphaGo 等方面做出了领先贡献。所以我认为这是一个你会喜欢并感到挑战的团队。

Yes, we are hiring across pre-training and post-training. We recently closed a funding round with backing from Nvidia, and our mission is to build frontier open intelligence. We are a team of about 60 researchers, a very small team still, hoping to build a frontier lab for open agentic models. We have researchers from X, DeepMind, OpenAI, top universities, who had leading contributions in PaLM, Gemini, AlphaProof, AlphaGo. So I think it's a team you would enjoy and feel challenged by.

除了来和你一起工作，他们还能如何深入研究这些问题？

And besides coming to work with you, how else can they dig into these problems?

所以，如果你想在当下这个时间点对社区产生真正的影响，很多工作都始于构建良好的测量系统。我谈到的很多能力，目前并没有真正强有力的基准来实际衡量它们。思考如何衡量智能的最佳方式是，你常常会想：“哦，如果我能把这个语言模型用于这个特定任务”，然后你发现模型并不擅长。这就是一个起点：“好吧，为什么它不擅长？当前的模型缺少什么？”能够将这一点打包成形式，这些就是目前效果不佳的评估基准。我在斯坦福也花时间；我经常和学生一起找出当前模型集合中的差距在哪里。如何将其转化为我们可以使用的评估基准，这通常是一个非常好的起点。从那里，你可以从后训练开始，这是一个相对成本较低的转变，你可以尝试看看已经能缩小模型哪些差距。如果这让你兴奋，那就加入我们，或者走你自己的路，去承担更大的冒险，从零开始构建这些模型。

So if you're trying to really make a dent in the community at this point in time, a lot of the work starts with building good systems of measurement. So a lot of what I talked about are capabilities that don't really have fundamentally strong benchmarks that actually measure them. The best way to think about measuring intelligence is that often you think, "Oh, if I could use this language model for this particular thing," and then you actually find the model is not good at it. That's a starting point: "Okay, why is it not good at it? What is missing in the models today?" And being able to create that in a packaged form, these are evaluation benchmarks that don't work well. I spend time at Stanford as well; often I'll work with students to figure out where the gaps are in the current set of models. How does that lead to an evaluation benchmark that we can use is often a very good starting point. And from there, you can start with post-training, which is a relatively inexpensive shift for you to go try and see what you can already start to close in terms of gaps for the models. And if that excites you, then come join us, or in your paths, take on a bigger adventure to build these models from scratch.

太棒了。太棒了。那么，Aakanksha，非常感谢你参与节目，分享了你对智能体的见解。

Awesome. Awesome. Well, Aakanksha, thanks so much for jumping on and sharing a bit about what you're seeing with regards to agents.

谢谢。

Thank you.