Ilya Sutskever 谈 AI 进展、对齐和时间线

很高兴来到这里。我想从你说过的最有趣的事情之一开始：大约 4 个月前，你和 OpenAI 团队谈到目标是在今年 9 月之前实现具有研究实习生水平的系统。也就是即将到来，大概从现在起 6 个月后。然后到 2028 年 3 月实现更全自动的 AI 研究员。所以，4 个月后的今天，你对这些时间表感觉如何？

Happy to be here. I think I'll start with one of the most the juiciest things you said, which is, you know, 4 months ago I think you and the OpenAI team talked about aiming for a system with research level intern capabilities by September of this year. So, coming up. I think that's what six months from now. And then a more fully automated AI researcher by March 2028. And so I guess you know, checking in four months later, how are you feeling about those timelines?

是的，我认为在过去几个月里，真正发生的变化是编码工具的爆炸式增长。没错，这还说得轻了。在 OpenAI，我们已经发展到使用 Codex 来完成大部分实际编码工作。所以我认为对大多数人来说，编程行为已经发生了很大变化。我绝对认为这是一个信号，表明我们走在正确的轨道上。另一个非常有趣的更新是数学研究能力的进展，以及我们在物理学和其他领域看到的结果。我认为这种能力水平——提供洞察的能力，结合访问基础设施的能力，以及使用更多计算机测试时间的能力——是 Codex 目前还不具备的。而且通用智能水平也有非常强的提升，我预计未来几个月还会继续。是的，我们仍在积极规划并高度关注这一点。

Yeah, I think you know, over I think over the last months I think like the change that's really happened is we've seen this explosive growth of coding tools. Yeah. It's an understatement. Yeah, we've definitely like really kind of gone to a place in OpenAI where we use Codex for the majority of you know, actual coding. And so I think for most people like the kind of the act of programming has changed quite a bit. So, I definitely see this as a signal that like you know, something here is on track. The other kind of like very interesting update over the last few months to me has been the progress on the math research capabilities. Also, the results we've kind of seen in physics and other fields. I think this kind of level of capabilities that was like ability to provide insight when combined with ability to access infrastructure, ability to use maybe more computer test time that's something that Codex isn't currently. And very strong improvement in general level intelligence which I also expect over the next couple of months. Yeah, it's something we're still very much planning for and very focused on.

你怎么知道已经达到了那个目标？比如，你会看什么样的工作流程来判断“好了，我认为我们已经具备了研究实习生级别的能力”？

And how do you like know when you've you've gotten there? Like what's like a a workflow you might look to to say, hey okay, I think we've got these, you know, research intern level capabilities.

我区分研究实习生和全自动研究员的方式是，系统能够自主工作的时间跨度，或者需要给出的任务的具体程度。所以，我不指望我们会有这样的系统：你只需告诉它“去提高你的分子能力”或“去解决对齐问题”，它就能做到。不是今年，我认为我们可能会在某个时候达到那一步。但对于更具体的技术想法，比如我有一个关于如何改进模型或如何以不同方式运行评估的具体想法，我认为我们已经有了大部分组件，只需要把它们组合起来。

The way I would distinguish you know, a research intern from from full automated researcher is the kind of span of time that that you would have it work mostly autonomously or the kind of like specificity of the task that has to be given. So, I don't expect you know, we'll have systems where you kind of just tell them all like you know, go improve your molecular ability, go solve alignment and they will do it. Not this year, you know, I think we might get there at some point. But I think for like more specific technical ideas like I I have this particular idea of how to improve the models, how to like you know, run this evaluation differently. I think I think we have the pieces that we mostly just need to put together.

Andre Karpathy 发布了一个相当火爆的版本，用这些模型来改进他的一些模型——显然比你们在这里构建的模型简单得多。但这是否大致符合这些工具可能的样子？

Andre Karpathy released, you know, a pretty viral version of of using some of these models to you know, improve some of his you know, obviously way less complex models than what you guys are building here. But did that feel like generally in this you know, in the spirit of some of what these tools might look like?

是的，我认为这符合精神。我预计它会从 Codex 现在的状态持续演进，朝着更自主、运行时间更长的方向发展。但我们会看到很多这类应用。总的来说，我们会看到这些模型在不同任务上更自主、使用更多算力。

Yeah, I think it's in the spirit. Yeah, I mean I I expect it to look like a pretty continual evolution from kind of where Codex is now. I think towards a bit more autonomy running for a longer time. But yeah, I think we'll see a lot of this sort of application. I think we'll in general we'll see more autonomous and higher compute use of these models for different things.

你提到了数学和物理方面，显然你们在数学上取得了令人印象深刻的突破，比如各种有趣的竞赛问题。也许对我们的听众来说，编码方面的进展如何直接转化为帮助 AI 研究是直观的。数学和物理方面的进展又是如何与此相关的呢？

You mentioned kind of like the math and physics side and obviously you've had these really impressive breakthroughs in math on you know, some interesting like different kinds of competition you know, problems. Maybe you know, I think for our listeners it like intuitively makes sense how progress in coding directly translates to something like you know, helping with AI research. How does like math and physics progress like also tie into this?

关注数学基准对我们来说最大的作用是作为通用的基准和北极星，指导我们如何改进这项技术。数学是非常可衡量的，判断你是否真正解决了数学问题，比判断你是否编写了一段好的软件要容易得多。而且数学可以变得非常困难，所以你可以有那些解决与否非常明确，但实际解决却极其困难的问题。

The biggest role that focusing on this math benchmarks has played for us is as a general benchmark and North Star for how to improve this technology, right? Math is very measurable, right? It's much easier to tell whether you've actually solved the math problem than like whether you've even produced a good piece of software. And also it can get very hard, right? So, you can have things where it's very definite whether you've solved them, but it can be arbitrarily pretty much hard to actually solve them.

直到不久前，我的观点还是这样的：我们的模型可能连简单的数学题都解不了。好吧，它们能解简单题了，但解不了 IMO 级别的题。所以，这些模型在智能上明显存在一个可衡量、易追踪的差距。我们很清楚需要做什么，这成了推理模型等的北极星。现在，情况变化很大。我们已经达到了努力追求的里程碑，比如 IMO 目标、解决 IMO 第六题，并在研究级数学上取得进展。我认为继续衡量这方面的进展仍有价值。而且，在数学推理上的提升肯定能迁移到 AI 研究上。我们很多最优秀的研究者都是正在培养的数学家或来自其他理论领域。但我们确实在重新思考这些北极星，更关注我们产出的下一个模型如何在现实世界中真正有用，尤其对 AI 研究，也对其他有经济价值的活动和科学领域，特别是应用科学。这种转变的原因是，我们相信模型现在足够有能力——虽然并非在所有方面都比人类聪明——但足以实质性地改变经济和工作方式。所以我们对此感到非常紧迫。

You know, I would say up until not too long ago, my perspective has been like, 'Well, okay, our models may not be able to solve simple math problems. Okay, our models are able to solve simple math problems, but are not able to solve IMO level problems.' So, clearly there is a gap in the intelligence of these models that is very measurable, very easy to run at. It's very clear what we need to do. And this has become kind of our North Star for reasoning models and so forth. Now, of course, that is changing quite a bit. And we have kind of reached these milestones that we've been working towards, like IMO goals level, solving IMO problem six, and making progress in research-level mathematics. And from this, I think there still is utility in continuing to measure progress on this. I think there's also definitely transfer that you can get from getting better at mathematical reasoning to getting better at AI research. A lot of our best researchers are mathematicians we're training or from other theoretical fields. But definitely we are very much changing how we think about these North Stars, and we are very focused on how the models, the next models that we're producing, are actually useful in the real world, useful especially for AI research, but also for other economically valuable activities and for other fields of science, especially more applied sciences. And the reason for this shift is because we believe the models are now capable enough, not as smart as people in all ways, but capable enough to actually materially change the economy, change how things are done. So, we feel a lot of urgency about that.

早期，选择像数学这样难解但易验证的领域，是完美的起点。我认为代码显然也有许多类似属性，可以检查和验证，非常适合强化学习。很多人都在思考一个问题：我们看到强化学习在那些容易验证的领域表现极好。世界上很多有价值的任务，比如医学、法律、金融，虽然也有一定程度的可验证性，但肯定不如数学和代码。所以很多人想知道，我们是否会看到类似的改进？显然，代码和数学的改进速度已经非常惊人。

In the early days, picking a domain like math that is so hard to solve but easy to verify whether you did it is kind of the perfect place to get started. And I think code obviously shares a lot of attributes to that, possible to check and verify and great for reinforcement learning. I think one question that a lot of people are thinking about is, okay, we've seen reinforcement learning work incredibly well in these domains where you can verify it rather easily. A lot of valuable tasks in the world, medicine and law, finance, there's some level of the ability to do that, but it's certainly not to the same extent that math and code are. So, a lot of people are trying to figure out, are we going to see similar improvements? Obviously code and math, the rates of improvement have been so astronomical and shocking.

是的，我确实这么认为。我们经常思考一个有趣的二元性：对于这些更通用、更难评估的任务，它们与更长周期的任务有很多共同点。因为即使是一个定义明确的数学或编程问题，如果需要花一年时间解决，那么即使成功的标准很明确，长期来看，第一天该做什么仍然是一个相当开放的问题。所以我相信这些困难是重合的，它们显然是我们系统发展的下一个前沿。我认为我们在这些更通用领域扩展强化学习的能力上看到了非常令人鼓舞的迹象。而且我们也可以以很有前景的方式进行扩展。

Yeah, I definitely expect so. I think an interesting duality that we think about a lot is for these more general tasks, for tasks that are harder to evaluate, they share a lot of commonalities with longer horizon tasks. Because if you think about even a very well-specified math or coding problem, if it's something that you need to work on for a year, then even if it's very clear what the criteria of success are, in the long term, what to do on your first day of working on it is a pretty open-ended problem. So, I kind of believe these difficulties coincide and they're very clearly the next frontier for our systems to develop. And I think we definitely see very encouraging signs both on our ability to scale RL on these more general domains. I think also we can scale in ways that showed a lot of promise.

在其他领域，最难的事情之一就是知道任务的成功标准是什么，对吧？你可以想象，无论是代码和数学中的短期任务还是长期任务，这种困难在那些领域之外会被放大，对吧？比如一个短期的法律或医疗任务，可能很难运行数千次迭代并判断是否做对了。而那些长期任务就更难了。我很好奇你如何概念化这个研究挑战。需要解决哪些问题才能让模型在这些其他领域真正发挥作用？

In these other domains, it feels like one of the hardest things to know is just what was success in a task, right? And you can imagine, whatever the problems are that are facing code and math that are short-term tasks and then longer-term tasks, it feels like it'll be amplified in the space that is outside of those, right? Where a short-term legal task or medical task may be harder to run thousands of iterations on, and figure out was that done correctly? And then those longer-term tasks, even harder. I'm curious how you even conceptualize that research challenge. Like what are the things that need to be figured out to be able to really make models work well in some of these other spaces.

是的，我回到这个想法：如何让模型长时间工作，以及如何教它们评估部分进展。我的意思是，即使不看强化学习，在更长周期上的进步来自哪里？随着模型通过预训练中的纯监督变得更加一致，它们会对好的部分产物是什么样有一些概念。所以我认为即使我们没有显著扩展强化学习，我们也会看到这些周期随时间延长。如何利用强化学习等新想法将其应用到通用领域确实是一个挑战，但我对此相当乐观。

Yeah, I come back to this idea of how do we make the models work for very long time and how do we teach them to evaluate kind of partial progress. I mean, if you look at even outside of RL, where that sort of progress on longer horizons is coming from, as the models become more consistent from pure supervision in pre-training, they get some ideas of what a good partial artifact looks like. So I think even if we weren't scaling RL very meaningfully, we would see an elongation of these horizons over time. It's definitely a challenge to figure out how to leverage these new ideas from RL and so forth to apply this to general domains, but I'm quite optimistic about that.

是的，这很有趣。听起来你的思维模型的一部分是模型本身能够以足够可靠的节奏检查进展。至少在外部看来，我们是否已经看到了强化学习的泛化还不完全清楚。感觉我们显然有一些技术可以围绕我们选择的任何焦点来优化模型，但这几乎像是老派的机器学习，一次只做一件事。你同意这种描述吗？你如何看待当前的气候？

Yeah, and it's interesting. It sounds like part of your mental model is like the models themselves being able to check progress with some sort of cadence that is reliable enough. In the outside at least, it's not totally clear if we've seen generalization in RL yet. Feels like we clearly have some techniques that really optimize models around whatever we choose to focus on, but it's like almost feels like an older school version of ML of one thing at a time. Is that like, you know, I guess would you agree with that characterization and like, how do you kind of see this current climate?

嗯，我们正在购买大量算力，对吧？因为我们仍然相信 beta 教训，我们相信它超过平均时间。是的，我们看到了新技术和新的扩展方式，但这正是我们看待事物的视角。

Well, we are buying a lot of compute, right? Because we still believe the beta lesson, we believe it more than average time to degree. Yeah, we've seen new techniques and I think new ways to scale, but that is kind of the lens through which we've been viewing things.

是的，我认为我们需要应对一定的复杂性，每个人都需要应对，因为我们不再只是构建一个完全与现实世界隔绝的“天空中的大脑”。如果你想让这个模型做医学研究，想让它治愈癌症，那么在某个时刻，它需要以有意义的方式了解现实世界，也许进行一些实验并从结果中学习。为此，你需要弄清楚如何真正连接它。这确实朝着你描述的方向发展，但我认为这与我们一直在开发的简单算法的扩展和发现并不矛盾。

Yeah, I think there is a certain amount of complexity that we need to grapple with, and everyone needs to grapple with because we're no longer purely building a brain in the sky that's completely isolated from the real world. If you want this model to do medical research, if you want it to cure cancer, at some point it needs to learn about the real world in a meaningful way, maybe conduct some experiment and learn from its results. For that, you need to figure out how to actually connect it. That goes in the direction you described, but I don't think that goes counter to actually scaling and finding simple algorithms that we've been developing.

我觉得我和很多公司聊过，现在每个人似乎都在问的一个主要问题是：我们是否应该自己做强化学习？拿一个开源模型，我们有一些关于人们执行任务的数据，我们有评估，因为我们很了解自己的领域。这对我们来说有意义吗，还是我们应该等待模型在这些事情上继续变得更好？对于许多收听播客的构建者来说，当他们思考在强化学习方面投入多少时，你会给他们什么建议？强化学习可以是一种非常数据高效的方式来改进模型在某个任务上的表现。还有一种更数据高效的学习方式，那就是在上下文中学习。这可能是人们教这些模型最基本的方式：只需用示例和指令提示它们，告诉它们你想要什么。我预计随着时间的推移，学习会变得更好。所以，模型能够适应你的上下文和你关心的任务类型非常重要。我不确定复制当前的强化学习流程是否是正确的方法，但这绝对是我们正在思考的问题。

I feel like I talk to a lot of companies, and one of the main questions everyone seems to be asking these days is: should we be doing our own reinforcement learning? Take an open source model, we have some data on a task that people do, we have evals because we know our domain well. Is this something that makes sense for us to do, or should we just wait for the models to continue to get better at these things? What advice would you give to the many builders that listen to the podcast as they think through the extent to which they invest on the reinforcement learning side? Reinforcement learning can be a very data efficient way to improve the model at some task. There is a much more data efficient way of learning, which is learning in context. This is maybe the most fundamental way people teach these models: just prompt them with examples and instructions for what you want. I expect that learning is going to get much better over time. So, it definitely matters that the models can adapt to your context and the kind of tasks you care about. I'm not sure if replicating the current RL pipelines is going to be the right way to go about it, but it's definitely a problem we're thinking about.

所以，这几乎就像你仍然需要做工作。你仍然应该找出哪些评估是重要的，收集数据、示例，但未来可能你会发现，把这些信息输入到上下文中，比试图在自己的模型上做任何事情要好得多。

So, it's almost like you still have to do the work. You still should figure out what the evals are that matter, gather the data, the examples, but it may just turn out in the future you're far better off just feeding that into the context than trying to do anything on your own model.

是的，我认为这很合理。而且我认为人们已经看到了像 Codex 这样的工具的成功，我知道你在这方面发挥了重要作用，他们想知道：我们是否需要为我们的领域构建自己的工具或使用方式，无论是法律、金融还是医疗保健？我们是否只需采用大型模型使用的工具，并在我们拥有的上下文中使用它们？对此有什么想法吗？工具的实现在很长一段时间内不应该成为限制。我认为我们将能够获得更通用的工具，人们可以将其用于各种其他领域。我的意思是，我认为 Codex 实际上相当不错，如果你尝试将它用于编码之外的事情。

Yeah, I think that's quite plausible. And I think people have seen the success of tools like Codex, which I know you've been a big key part of, and wondered: do we need to build our own harnesses or our own ways of using these things for our own domains, whether it's legal, finance, or healthcare? Do we just take the harnesses that the large models do and use them within the context that we have? Any thoughts around that? The implementation of the harness shouldn't really be a limitation for a very long time. I think we'll be able to get much more general harnesses that people can use for all sorts of other domains. I mean, I think Codex is pretty good, actually, if you try using it for things beyond coding.

这太有趣了。一个更通用的工具，几乎可以自适应，或者只是在你领域中的任何特定工具集或你想暴露给模型的特定事物上工作。

That's so interesting. A much more general harness being something that's almost adaptive to or just works across whatever specific set of tools you have in your domain or specific set of things you want to expose to the model.

是的，我认为也值得思考我们想要与模型交互的最终界面是什么。模型提供了一些 UI 工具，它们可以构建自己的 UI，可以做人们觉得非常耗时的事情。但我绝对认为还有很多空间可以让模型访问我们目前为人类使用的界面。所以，我认为我们希望在 Slack 上拥有 AI，例如，它们可以接入我们的联系人，能够从中学习并实现这些现有功能。所以，这里肯定有一些中间地带，但我相信从长远来看，默认情况下 AI 应该在你所在的地方与你相遇，如果不是，那将是因为它拥有新能力，而不是因为它有局限性。

Yeah, I think it's also worth thinking about what is the ultimate interface that we want to interact with the model with. The model gives some UI harness, they can build their own UIs, they can do things that people would find very time-consuming. But I definitely think there is also a lot of space to enable the models to access the current interfaces that we use for people. So, I think we want to have AIs on Slack, for example, that are plugged into our contacts and able to learn from it and realize these existing things. So, definitely there is some meet in the middle here, but I believe long term, by default the AI should just meet you where you are, and if not, that will be because it has new abilities, not because it has limitations.

是的，这是一个有趣的观点。基本上今天这些工具似乎是为特定环境定制的，但随着时间的推移，随着你添加越来越多的技能和工具，模型能够有效地导航这些，它就会变得通用，就像人类的方式一样。这非常有道理。我很好奇，你显然每天都会在研究方面看到疯狂的事情。在这一点上，当你想到“如果有一天我看到 X 或 Y，那将非常疯狂”时，哪些里程碑对你来说仍然有意义？你最关注的是什么？

Yeah, that's an interesting point. Basically today it feels like these harnesses are so bespoke to certain environments, but over time as you add more and more skills and tools and models can navigate across those effectively, it'll just be a general thing, like the way humans have. That makes a tremendous amount of sense. I'm curious, you obviously see crazy stuff on the research side every day. At this point, what are the milestones that are still meaningful to you as you think about, 'It would be pretty crazy if I one day saw X or Y?' What are the things you're paying most attention to?

在这一点上，这真的关乎研究。关乎模型发现新事物，执行更长期的研究问题。这几乎就像在寻找某种洞察，你会想，“哦，如果我团队中的某个人想到了这一点，我会对此非常感兴趣。”

At this point it really is about research. It's about the model discovering new things, executing on a longer horizon research problem. It's almost like looking for some sort of insight that you're like, 'Oh, if someone on my team had come up with that, I'd be pretty intrigued by it.'

是的，我们实际上已经有一些微小但相当有影响力的想法来自甚至 GPT-5.2 Pro。这完全是，但我认为与我期望的相比仍然非常小。

Yeah, we've actually had some minor but quite impactful ideas come from even GPT-5.2 Pro. That's really entirely, but I think it's still very small compared to what I expect it to be.

是的。似乎这些模型几乎不可避免地会变得更好。它们将被用于研究。它们将更普遍地用于科学。你几乎是这个阶段最早直接与这些模型作为研究伙伴互动的人之一。

Yeah. It seems like almost inevitably these models are going to get better. They will be used in research. They'll be used in science more generally. You're one of the first people interacting directly with these models as research partners almost at this stage.

关于如何正确做到这一点，或者当你思考随着这些模型不断进步，研究机构可能会是什么样子时，你有什么心得吗？

Anything like you've learned around the right way to do that or as you think about like what a research organization, you know, as these models continue to get better might look like?

是的。我认为我们确实处在一个转折点，模型的短期即时质量即将成为我们研究进展速度的一个决定性因素，因为模型将驱动很多进展。所以这肯定需要重新调整关于如何运营研究机构的一些直觉。通常你会尽量不关注即时质量，而是更专注于长期。我认为我们有很多非常令人兴奋的事情正在推进，但我感到非常紧迫，要真正执行下去，并利用模型智能的这些进步来加速人工智能研究，尤其是 AI 对齐。

Yeah. I think we're definitely at a transition point where the short-term immediate quality of the model is about to be a quite determining factor for the pace of our research progress, because the models are going to drive a lot of that. And so that definitely requires rewiring some intuitions about how to run a research organization. Normally you try not to focus on immediate quality. You try to be much more focused on the longer term. I think we have a lot of very exciting stuff coming up that we are working towards, but I feel a lot of urgency to actually execute on it and to use these advances in model intelligence to accelerate research on AI and especially AI alignment.

是的，这真是一个引人入胜的观点，因为我之前听你谈过运营研究机构。我觉得过去更像是给人们空间去追求很多不直接相关的事情，比如这不是一两个月的进展，而是那些真正能推动事情前进的想法。但现在我们处于这样一个时代，你说，看，我们做的一切都会变得更好，如果我们只是短期专注于这个并让它变得更好。同时处理这些和那些可能更遥远的研究想法，以及运营一个机构，一定非常有趣。

Yeah, it's such a fascinating point because I've heard you talk before about running a research organization. I feel like in the past it was like giving people the space to pursue a lot of things that weren't directly, you know, hey, this is for a month or two months of progress, but it's like what are the ideas that are really going to drive things forward. But it makes total sense that we're in a time now where you're like, look, everything we do will be so much better if we just focus on this in the short term and make it better. It must be fascinating to navigate that and these maybe further off research ideas at the same time and like running an organization.

是的，这确实是我们现在和 Mark 花了很多时间讨论的事情。

Yeah, it's definitely something we spend a lot of time on with Mark nowadays.

现在你拥有大量的算力，但显然在预训练方面有很好的缩放定律，在强化学习方面也有很好的缩放。你可能有很多实验与这两个方向无关，但都是有趣的新方法。你如何考虑在所有这些东西之间分配算力？

Right now you have a ton of compute, but you obviously have great scaling laws on the pre-training side. You have great scaling on the RL side. You have probably lots of experiments going on that have nothing to do with either of those vectors, but are like interesting new ways. How do you even think about allocating compute across all of this stuff?

是的，这会非常复杂，对吧？因为有很多事情需要做。我们开始保持的一个纪律是，我们确保明确地将大部分算力预算给最具可扩展性的方法，即我们认为对驱动通用模型智能最负责的东西。即使这并非总是最有效的算力分配，因为如果你把这么多算力分配给一个实验或一组实验，其他地方用一点点算力就能加速很多事情。但我认为很容易把一切都分割开来，最终却没有做我们认为最重要的事情。你肯定想理解经验证据，确保评估有序，实验严谨。然后你还想基于以下因素应用一些正则化：“好吧，我们理解这个方法吗？我们真的期望它能扩展吗？我们期望这是未来可以构建的东西吗？这是不是一次性的？”然后据此确定代价。

Yeah, it's going to be very complicated, right? Because there's so many things that we need to do. One discipline we've started keeping is we try to make sure we explicitly budget a large chunk of our compute to the most scalable methods, the things that we believe are the most responsible for driving general model intelligence. And even if it's not the most efficient allocation of compute at all times, because if you're allocating so much compute to one experiment or one set of experiments, there's so many things you can accelerate with a little bit of that compute elsewhere. But I think it's easy to partition all of it and not end up doing the things that we believe are most important. You definitely want to understand the empirical evidence. You definitely want to make sure your evaluations are in order and the experimental rigor is there. And then you also want to apply some regularization based on, 'Okay, do we understand this method? Do we actually expect it to scale? Do we expect this is something you can actually build on in the future? Is this kind of a one-off?' And based on that, determine the price.

是的，这太有趣了。你可能发现了所有你知道可以改进的地方，但它们可能有点偏离你认为整体进步轨迹的方向。所以你在某种程度上留下了一些唾手可得的果实，因为真正重要的是找到未来的方向，然后在该方向内进行缩放，并将算力投入其中。

Yeah. It's so interesting. You probably find all the ways that you know you could improve things, but they feel maybe a little bit off to the side of where you think the overall arc of progress is. And so you end up leaving some of these low-hanging fruits to some extent, because really the most important thing is finding the future direction and then the scaling within that and devoting compute to that.

显然，我们经常谈到 Codex 和编码的成功。感觉去年是编码方面令人难以置信的爬山之年。我很好奇，显然，Codex 是一个非常成功的产品。在很多方面，Anthropic 几乎是第一个进入这个市场的，你知道，Claude Code 在那里是一个主导产品。你怎么看待 Anthropic 在这个领域的成功？

Obviously, the place where we talked about Codex a lot and the success of coding. And it feels like last year was the year of just incredible hill climbing on coding. I'm curious, obviously, you know, Codex has been a super successful product. In many ways, like Anthropic was kind of first to this market, you know, Claude Code, it was a dominant product there. What do you make of the success Anthropic's had in this space?

是的，我认为这关乎真正将产品方向聚焦在你认为技术下一个应用所在的地方。如果你看看我们在 OpenAI 产品上的优先级，我们确实一直在做编码项目，但它们相对于我们的主要优先级来说一直是次要的。有趣的是，这并不太反映 OpenAI 内部研究机构的优先级。我认为鉴于 ChatGPT 取得了爆炸性的成功，ChatGPT 本身正在并且将会发生很大变化，但就 2023 年的样子而言，这个特定产品可能并不完全代表它所赋能的一切。所以我们的研究工作大部分都集中在那个未来的东西上，而且我认为它越来越与我们的短期产品策略脱钩。是的，我对我们一直在构建的东西以及我们在模型智能研究方面正在构建的东西非常有信心。我们在产品方面增加的代表性和关注度，实际上是为了部署它们。并且相信它们才是现在真正重要的东西。

Yeah, I think it's a matter of really focusing your product direction on where you believe the next application of the technology is. And if you look at the prioritization we've had on our product at OpenAI, I mean, we have been working on coding projects, but they have been kind of a secondary thing compared to our main priorities. And the interesting thing is that is not very reflective of the priorities of the research organization within OpenAI. I think given that we've had this explosive success of ChatGPT, ChatGPT as it was, I think ChatGPT is evolving quite a bit and it's going to evolve quite a bit, but as it was in 2023, this particular product that is maybe not, I think it's definitely quite aligned with our vision of where AI is going, but it's not really representative of everything that it enables. And so the majority of our work in research has been focused on that future thing, and I think increasingly it has decoupled from our short-term product strategies. Yeah, I'm very confident about the things we've been building and the things we are building on the research on the model intelligence side. A lot of our representation and increased focus on the product side is about actually getting to deploy them. And the belief that actually they are the thing that really matters now.

是的，现在感觉整个公司的优先级仍然锁定并专注于这一点。你看到 Codex 在最近几个月取得了令人难以置信的改进。对于所有收听播客的开发者来说，随着这些模型在越来越长的任务上不断爬山，几乎很难理解世界会变成什么样子。

Yeah, and now it feels like the whole company priority is still locked in and focused on this. And you've seen just incredible improvement in Codex in recent months. For all the developers who listen to the podcast, it's almost hard to comprehend what the world looks like as these models keep hill climbing on longer and longer tasks.

比如你觉得他们的生活会有什么不同？或者，他们会在未来 3 到 6 个月内如何使用 Codex？我知道在这个领域，3 个月和 6 个月的时间线差别很大，但你可以选一个中间点来谈。

Like what do you think will look different in their lives? Or like how will they be using Codex in, you know, 3 6 months? I realize 3 months and 6 months are very different timelines in this world, but take whichever in between point you'd like.

我预计会逐步提升的是你愿意让模型拥有的自主程度，以及能有效工作的描述详细程度，还有它所需的监督水平。我认为我们离能够自主工作几天的模型并不遥远。可能只需要比现在多一点的算力，它们就能独立生成质量高得多的成果。

I would expect just a gradual increase in just the level of autonomy you feel comfortable letting the model, just the thickness of the description that's going to work with, you know, the level of supervision it needs. I think we're not very far from models that can work autonomously for a couple days. Maybe just quite a bit more compute than we're using now and produce much higher quality artifacts on their own.

你凭直觉怎么看？一直有个问题：世界是否需要软件工程技能来监督这些运行几天的模型？还是说，到了某个阶段，任何人都能使用编码智能体并监督它们产生某种输出？

Do you have a gut instinct on like what, like, you know, there's always been this question of like will the world, you know, do you need that software engineering skill set to supervise these models running for a few days or like, hey, does it turn out at some point of like being able to run for a while, you know, anybody can use coding agents and supervise them to some sort of output?

我的意思是，对于很多输出，你显然已经不需要太多经验了，对吧？但我仍然会区分，比如我们这里的实习生和真正的自主研究员或软件工程师：如果你想构建更大的东西，你可能仍然需要施加监督。你仍然需要有一个总体框架。你要能识别哪些构建块合适，哪些不合适。不过，我确实预期所需的技能组合会随着时间的推移发生很大变化，朝着更通用的愿景设定方向转变。

I mean, I think definitely for like a lot of outputs you already don't need that much experience, right? I think still the distinction I would draw between like, you know, an intern here and like really an autonomous researcher or software engineer would be that like if you want to build something bigger, like, you know, you probably still want to apply supervision. You still kind of want to have like an overarching thing. You want to recognize like what building blocks fit in and what which don't. But yeah, I definitely expect that like the desired skill set will shift quite a bit over time. Towards this like more general vision setting.

我猜在研究方面，感觉大概一个月前，所有人都在谈论持续学习，这成了时代精神。有很多新实验室开始专注于持续学习。有些人离开了 OpenAI 去专注这个。我好奇的是，这背后可能有一种信念：仅靠强化学习要么无法达到目标，要么会导致某种非常低效的 Scaling，而且这与人类的学习方式不同。我甚至听你以前说过，强化学习至今仍与人类学习方式大相径庭。你对整个运动怎么看？

I guess on the research side I feel like there's been, maybe like a month ago I feel like all anyone could talk about was continual learning and there was just, you know, it was in the zeitgeist. There's all these neo labs starting to go focus on continual learning. Some folks left OpenAI to go focus on that. I'm curious like, I think part maybe behind that is a belief that like, you know, RL alone, you know, either won't get us there or will get us to like some level of very inefficient scaling and it's kind of different than the way humans learn. I think even I've heard you say before like that, you know, RL is still very different today than the way that humans learn. What's your take on that whole movement?

是的，我对此有点困惑，因为在我看来，我们一直以来的兴奋点——我的意思是，看看 GPT-3 论文的标题，对吧？它表明这类模型实际上具备持续学习的能力，对吧？它能够学会在上下文中学习，对吧？这实际上是推动我们进一步扩展这些 GPT 模型的兴奋点背后的驱动力。这也是为什么我们需要用强化学习来训练它们，以便更高效地在上下文中学习的前提。所以，我完全同意持续学习确实是关键，对吧？它正是我们正在构建的东西。但我并不认为这是一个被忽视或偏离我们当前道路的问题。我认为这正是我们努力的方向。

Yeah, I am a little bit confused by it because, you know, in my mind like the whole kind of excitement that we've had, I mean, if you look at the titles of like the GPT-3 paper, right? Like it is that like oh, you know, this class of models is actually capable of continual learning, right? It's capable of learning to learn in context, right? That has been really, you know, the driving force behind the kind of excitement to like scale these GPT models further. That has been like the premise for why we really need to teach them with RL so like learn in context more efficiently. And so, I definitely agree that continual learning is really the thing, right? Like it's really the thing that we're building, but I don't really think this is like a problem that's like oh, you know, it's kind of ignorant and off the path of what we're doing currently. I think it is what we're working towards.

是的，我认为你的意思是，达到目标的最佳路径就是继续扩展预训练和强化学习。我认为这正是我们迄今为止在这个问题上取得最大进展的方式，而且肯定还有更多想法和步骤。我认为很多改进也将直接来自规模的扩大。

Yeah, I think you're saying this is like the single best path to get there is to continue to kind of scale the pre-training and RL. I think that is kind of how we've made the most progress on this problem so far and you know, I think there are definitely more ideas, more steps. I think also a lot of improvement that will just come from scale.

我猜，我们有很多听众可能已经能用这些模型做很多简单的事情，然后他们尝试做一些更复杂的、比如上百步或更长期的任务，然后他们会说，哦，模型还做不到。我觉得这更难。你们内部不断感受到这种进步，但对他们来说，感觉就像离能完成这种更长期的任务还有天壤之别。你如何向他们阐明，要实现这些更长的步骤，需要满足哪些条件？是像你之前说的更频繁地检查，还是我觉得研究界有一种信念，认为所有这些任务都会在未来一两年内解决，而外界很多人可能并没有完全理解我们一直看到的这条改进曲线。

I guess like, you know, we have a lot of folks listening that maybe have been able to do a lot of simpler things with these models and then they try to do like some of these more complex, I don't know, call it hundred step or longer term tasks and they're like oh, you know, the models don't work for this yet. And I think that's harder. You on the inside constantly feel this improvement, but for them it feels like hey, this is like night and day away from being able to do this much longer thing. How do you kind of articulate to them, I guess, the set of things that need to be true for these like much longer steps to happen? Is it around kind of checking in more often as you were talking about before or I feel like there's just this belief among the research community of like oh, all of these tasks will be solved in the next year or two and then in the wild a lot of people maybe not totally grokking that improvement line that we've been seeing.

是的，我的意思是，很多这种预测只是基于历史改进曲线，对吧？我认为我们越来越能大致看清这里的形状。我不认为这很大程度上只是模型变得足够聪明，能够识别自己是否在进步。我认为其中一部分是这种非常实际的工作：模型是否真的能访问所有上下文、所有文件、所有基础设施来完成你想要它们做的工作。我记得过去我们讨论强化学习的路线图时，我确实觉得，好吧，我们首先需要教会模型用自己的 token 进行推理，然后当然需要它使用工具与环境交互。在某个时候，我们肯定要教会它看，对吧？在某个时候，我们需要教会它使用物理身体，对吧？但嗯，是的，我认为我们肯定已经进入了这样一个阶段：它确实需要与环境交互，确实需要看，而且不久我们就会真正推出机器人。但是的，我确实觉得很多时候，当我听到人们抱怨模型不能做 X 或 Y 时，实际上只是因为你没有给它提供或连接到系统，或者没有输入足够的上下文。实际上，我确实在想，如果上下文能够普遍适用并流入这些东西，我觉得很多问题其实用今天的模型就能解决。

Yeah, I mean I definitely a lot of that prediction comes from just looking at historical improvement lines, right? And I think increasingly we can roughly see the shape here. I don't think a lot of this is about just the models becoming intelligent enough to recognize whether they're making progress. I think some of this is like we have this very kind of pragmatic work of like are the models actually, you know, can they actually access all the contexts, all the files, all the infrastructure they need to do the work you want them to do which yeah, I remember like in the past when we were discussing the kind of the roadmap that we're taking with RL. You know, I definitely feel like, okay, we just need to teach the models kind of reason with its own tokens as kind of a priority and then of course we'll need it to use tools like interact with the environment. You know, at some point we definitely teach it to see, right? At some point we need to teach it to use a physical body, right? But like uh yeah, I mean I think we're definitely like well into the stage where you know, it really needs to like interact with the environment and really needs to see uh and you know, someday soon we'll really come up with robots, but Yeah, I mean it does feel like a lot of the times when I hear people complain about how a model can't do X or Y, it's like literally just cuz you haven't fed, you know, or connected to the systems or fed enough context into it. Actually, I do wonder if like context was universally applicable and able to flow into these things, like I feel like a lot of these problems would actually just be solved with today's models.

我想聊聊你们一直在做的 AI for Science 方面的一些工作。特别有一件事，我觉得编程相关的东西是每个人都能直观感受到的，因为每家公司都在用这些工具，生产力大幅提升。但数学方面，不是所有人都参加过 IMO 竞赛，也不一定对这些突破有直观的感受。我知道你们做的一个很有意思的工作是关于“首个证明”的，对吧？我认为这些问题和传统的竞赛数学非常不同。我希望你能稍微谈谈这个，因为我觉得我们的听众可能对这个领域不太熟悉，也不太了解模型能在这里做出很酷的工作意味着什么。

You know, I want to talk about some of the AI for science stuff that you guys have been working on. One thing in particular, I feel like the coding stuff is something that everyone feels very viscerally because in every company they're using these tools and getting tons of productivity. The math side, not all of us have competed in IMO competitions and necessarily have as much of an intuitive feel for some of these breakthroughs. One of them I know that was really interesting that you guys did is use some compelling work around first proof, right? I think these are very different problems than traditional competition math. I want you to speak a little bit to that because I think it's a space that our listeners might be less familiar with and less familiar with understanding the implications of models being able to do pretty cool work here.

是的，我对“首个证明”挑战感到非常兴奋。因为那个挑战有点像是一个基准测试，对吧？几位受人尊敬的数学家和理论计算机科学家发布了一些问题，他们认为这些问题代表了他们的日常工作，但从未发表过，这样我们的模型就可以尝试解决。我们对这个挑战非常兴奋，但它是在没有任何提前通知的情况下发布的，只有一周的截止时间。当时我们正在训练一个非常令人兴奋的模型，所以负责训练的人之一 James Lee 开始手动提示那个模型，然后发现，“哦，它真的在解决这些问题。”这真是非常迷人的一幕。其中一个问题实际上来自我博士研究的领域，看到模型提出这些想法——这些想法如果是我自己提出，我会在一两周内感到非常自豪——而它在一个小时左右就想出来了，这种感觉非常奇怪。过去，我在看我们的 Dota 机器人无限地玩非常有趣的 Dota 游戏时也有过这种感觉。感觉就像某种魔法在发生，因为有趣的事情不应该是无限的。看到这种情况发生在数学上，发生在我认为代表了我们很多重要工作的领域，确实增加了我的紧迫感。

Yeah, I was very excited about the first proof challenge. Because that particular one is kind of a benchmark, right? It's a couple of respected mathematicians and theoretical computer scientists releasing problems that they believe are representative of their day-to-day work, but haven't been published anywhere, so that we can really have our models take a crack. We were so excited about this challenge, but it was kind of dropped without any advanced warning, with a week-long deadline to actually execute. We had a very exciting model training at the time, and so one of the people in charge of training, James Lee, kind of started prompting that model just by hand, and actually seeing, 'Oh, okay, it's actually solving these problems.' It's really a fascinating thing to see. One of these problems actually is from a domain that I did my PhD in, and seeing the model come up with these ideas which I would be quite proud to come up with in a week or two, seeing it come up with them in like an hour or so, that was a very weird feeling. In the past, I felt like that when watching our Dota bot play very interesting Dota games infinitely. It feels like there's some sort of magic happening because interesting things should not be indefinite. Seeing that happen for math, for something that I believe is quite representative of a lot of the work that we're doing and that really matters in the world, definitely increased my feeling of urgency.

还有一点很迷人，就是你在训练这些模型，然后把这些问题扔进去，没人知道它们能解决得多好。看到你非常熟悉、投入了大量时间的领域，然后意识到，“嘿，可能上一代模型根本做不到这个。”你甚至不一定认为这是该做的基准测试，但它只是普遍展示了模型的通用能力和进步。

One thing that's fascinating too is the idea that you're training these models and you throw these problems in, and nobody knows how good they will be at solving them. It must just be fascinating to see something that you know so well and in a space that you spent so much time in, and realizing, 'Hey, probably the previous generation of models wouldn't have been able to do that.' You wouldn't have even thought necessarily that this was the benchmark to do, but it's just generally showing the general-purpose capabilities and improvements of the models.

目前这个阶段，我们需要找特定领域的专家来告诉我们这些证明是否正确。但判断是否取得了进展，仍然比判断编程进展要容易得多。因为当然，竞赛编程可以评估，但大多数编程不是竞赛编程，它涉及抽象和处理所有情况。

It is at a stage where we needed to seek out experts in the particular domains to be able to tell us whether these particular proofs are correct or not. But it's still much easier to tell whether you've actually made progress than for something like even coding. Because sure, competitive programming you can evaluate, but most programming is not competitive programming, and it's about abstractions and handling all the cases.

我觉得一年前有一个常见的批评，我不知道现在是否还那么强烈，就是“这些模型就像模式匹配器，但你真的想要 AI for Science，我们不可能从模式匹配中得到新想法或全新的事物。”感觉我们一直在削弱这种说法。我们是不是越来越接近从根本上推翻它了？

I feel like there was this common criticism a year ago, and I don't know if it's as strong now, that 'Okay, these models are like pattern matchers, but you really want AI for science, like we're not going to get new ideas or entirely novel things out of pattern matching.' Feels like we continue to chip away at that narrative. Are we getting closer to fundamentally disproving that?

我相信是的。我认为在规模上，我们开始看到一些小的进步，比如这里那里的小想法。也许还有一些与科学家合作的大论文。但 AlphaZero 是模式匹配器吗？AlphaGo 是模式匹配器吗？我们的 Dota 机器人是模式匹配器吗？它们确实为各自的游戏提出了新策略。

I believe so, yeah. I think on scale we're starting to see minor advancements, like small ideas here and there. Maybe some bigger papers in collaboration with scientists. But was AlphaZero a pattern matcher? AlphaGo a pattern matcher? Our Dota bot a pattern matcher? They did come up with new strategies for the respective games.

有趣的是，早在 2016、2017 年就有反例了，对吧？

It's funny that there are counterexamples to it all the way back to 2016, 2017. Right?

没错。而且我认为你总能找到其中的缺陷，这很有意思。AlphaGo 可以用某种策略击败。我们的 Dota 机器人也可以用某种策略击败。我认为在一段时间内，这些模型会有很多不足。但我也认为它们能够发现新事物，因为它们拥有很多这样的能力。从非常小的游戏环境到更广泛的科学研究，这花了几年的时间，期间需要大致掌握所有人类知识，学习所有人类语言等等。但我认为基本原理非常相似。

Right. And I think you can always find flaws in that, which I think is interesting. AlphaGo can be beaten with some strategy. Our Dota bots could have been beaten with some strategy. I think there will be a lot of deficiencies for a while with these models. But I think also they are able to discover new things because they have a lot of these capabilities. It's taken a couple of years to go from very tiny game environments to much more broad scientific research that requires going through a decent approximation of all human knowledge in the meantime, and learning all the human languages and so forth. But I think the basic principle is very similar.

你知道吗，这很有趣。我记得你们在“首个证明”的结果出来时，组织者评论这些 AI 解决方案说，“这感觉像是 19 世纪的数学，暴力、计算密集型的方法，而不是那些优雅的现代技术。”我不确定这是这些模型工作方式的一个特点还是缺陷。听到这个，你是担心还是兴奋？

You know, it's funny. I think when you guys had these first proof results, if I remember right, the organizers said they were commenting on these AI solutions and they were like, 'This feels like 19th century mathematics of brute force, computation-heavy approaches rather than these elegant modern techniques.' Which I'm not sure is a feature or bug of the way these models work. Hearing that, does that concern you or excite you?

我不担心。我认为这是意料之中的。我确信至少对于其中一个问题，我实际上得到了相当不错、相当漂亮的结果，比预期的要短得多。

It doesn't concern me. I think it's expected. I'm sure for at least one of the problems, I actually produced pretty good, pretty nice results. Quite a bit shorter than the intended one.

但我觉得总的来说，你会预期这些模型能在短时间内产生比人类多得多的推理，对吧？就原始 token 或想法的数量而言。我不认为这会是一个长期特征。现在 AI 在科学领域势头很猛。你提到过，在某个时候你必须将这些模型连接到物理世界。你们和 Ginkgo 合作发布了一些很酷的东西，还有其他一些你们在实验的东西。我相信你对 AI 在多个科学领域有很多思考。当你深入研究这些东西时，你有没有形成一些直觉？想想三年后，哪些科学领域你会觉得“哦，那里会有疯狂的进展”，而哪些可能对即时变化更具抵抗力？

You know, but I think in general you would expect like, yeah, these models can produce so much more reasoning in a short time than a person can, right? Just in terms of the raw number of tokens or thoughts. I don't expect that to be a long-term feature. It feels like there's so much momentum behind AI for science right now. And you mentioned obviously, at some point you do have to connect these models to the physical world. And you guys released some cool stuff with Ginkgo and some of these other things you've been experimenting with. I'm sure you've thought a lot about AI for a bunch of different areas of science. As you've dug into some of this stuff, have you developed any intuition? Think about 3 years from now, the spaces of science where you're like, oh, there's going to be crazy progress there versus the ones that might prove a little more resistant to immediate change?

我的直觉是，这实际上取决于那些需要手工操作的事情，模型还不能即插即用，而生态系统——不同的实验室——也会很快进化以适应这些新技术。在这些 STEM 领域内，有一个问题是：它是能访问物理世界的 LLM，还是像 Isomorphic（生物学）、Periodic（材料科学）或 Physical Intelligence（机器人学）这样专门围绕这些领域创立的公司？你对在多大程度上用不同模型架构独立追求这些事情，还是全部放在一个地方，有什么直觉？

My intuition would be that it's really about the things that require manual work, where the models are not quite plug-and-play, and the ecosystem—the different laboratories—will also evolve pretty quickly to adapt to these new technologies. Within those STEM fields, there's a question of whether it's an LLM with access to the physical world, or you've had companies started specifically around these domains, like Isomorphic in biology, Periodic in material sciences, or Physical Intelligence in robotics. What's your gut instinct on the extent to which it makes sense to pursue some of these things independently with different model architectures versus all within one place?

是的，我认为这有点像我对 Codex 界面的回答：我会围绕技术的能力而不是其局限性来构建。所以，如果你有东西能突然设计大量有趣的化学或生物实验，那么建造能实现这一点的实验室就是有意义的。我认为如果我们真的到了模型能设计高质量实验的地步，那么让它与人类协作也是合理的。我们不应该认为要么完全自动化，要么把它当作使用一些工具的辅助工具。我们会到达一个与努力解决问题的 AI 科学家自然协作的世界。

Yeah, I think it's kind of similar to my answer with the UI for Codex, which is: I would build around the capabilities of the technology and not around its limitations. So, if you have something that can suddenly design a huge amount of interesting chemical or biological experiments, it makes sense to build labs that enable that. I think if we did get to a place where the model is very capable of designing high-quality experiments, it would also make sense to have it work with humans in a loop. We shouldn't think of it as either fully automating it or having it as a fun thing using some tools on the side. We will get to a world where it's very natural to be collaborating with AI scientists that are working hard on a problem.

是的，这很有趣。这几乎是两种不同的愿景。一种世界是：你训练一个模型来基本上运行这些端到端任务，成为自动化的生物学家或化学家或任何东西。另一种是：你构建真正的工具来提出、运行，并与一群人类研究人员协同工作。

Yeah. It's interesting. It's almost like a different vision. One world where this works is: you just train a model to basically run these end-to-end tasks and be the automated biologist or chemist or whatever. And another one is: you're building really tools to both propose, run, and work in tandem with a bunch of human researchers.

我不一定会这样描述。当然在某种意义上存在工具，但我认为我们正走向它们驱动整个过程的许多设计和构思的阶段。使用 LM 架构，但能够找出正确的实验类型并实际设计它们。

I wouldn't necessarily characterize it that way. Of course there are tools in some sense, but I think we're going to a point where they're driving a lot of the design and ideation for the whole process. With an LM architecture, but just being able to figure out the right kinds of experiments to run and then actually design them.

至于不同的架构，当然自然语言推理——我们优先考虑的那种——给了你很多通用性。有些东西你想训练一个不同的模型来建模。我认为即使你想创建一个非常好的目标模型，我也不认为大型语言模型是最有效的方式。它们最终可能会产生最好的模型，但我认为对于蛋白质折叠或其他类似任务也是如此。

And when it comes to different architectures, for sure natural language reasoning—the kind of things we're prioritizing—gives you a lot of generality. There are things that you want to train a different model to model. I think even if you want to create a very good goal model, I don't think large language models are the most efficient way to go about it. They might result in the best model eventually, but I think it's similar for protein folding or other tasks of this kind.

所以你认为围绕这一点进行一些独立的努力是有意义的，但显然这最终会与一个核心的、非常好的研究员大型语言模型配对，帮助推动这些事情。

So you think it makes sense to have some independent efforts around that, but obviously that will end up being paired with a core really good researcher large language model that is helping drive a bunch of this stuff.

我还想谈谈 AI 安全，因为我认为这是你做了很多开创性工作的领域。我不确定所有听众都熟悉——你实际上在多个实验室做了一些非常有趣的工作，专注于思维链监控。所以也许先告诉我们一点那项工作和你的发现。

I want to also make sure to talk about AI safety, because I think that's an area that you've done a lot of really pioneering work on. I'm not sure all our listeners will be familiar with—you actually did some really interesting work across the labs, and were focused on chain-of-thought monitoring. So maybe to start, tell us a little bit about that work and what you found.

是的，这是我们在看到当前第一批推理模型时的一个认识。我们意识到，好吧，这行得通，我们思考了很多这意味着什么。我们想，好吧，可能它们会在未来两三年内真正改变这一点。我们在思考这对安全以及我们理解这些模型在做什么的能力意味着什么。我们意识到，由于我们训练这些模型的方式，以及我们不直接监督推理过程——不像 ChatGPT 试图表现得礼貌友好，总是说我有好主意。嗯，那是另一个问题。但即使假设它完全按照我们想要的方式对齐，这绝对不是谄媚，它有时仍然不会透露其动机的一些东西，因为可能那会不安全，或者可能不友善，或者可能因为它实际上并没有像我们想的那样对齐，但它想隐藏这一点。而我们训练推理模型的方式，思维链没有这些。

Yeah, so this is a realization that we had around the time we actually saw the first reasoning models of the current crop. We realized that, okay, this works, and we were thinking a lot about what this means. We were like, okay, probably they will really change this over the next two or three years. We were thinking about what this means for safety and for our ability to understand what these models are doing. And we realized that because of the way we train these models, and because we don't supervise the reasoning process directly—it's not like ChatGPT is trying to be polite and nice and always tells me I have great ideas. Well, that's a separate issue. But even assuming it's aligned exactly the way we would want it to, which is definitely not sycophantic, it's still not going to reveal some things about its motivations at times, because maybe it would be unsafe, or maybe it would be unkind, or maybe because it's actually not aligned the way we think, but it wants to hide that. And the way we train the reasoning models, the chain of thought doesn't have any of that.

它并没有以任何特定方式优化，因为它本身并不直接出色，对吧？它只在产生高质量输出方面表现出色。我们意识到，这实际上是一种非常强大的范式，能够解释模型在做什么，对吧？这与机械可解释性的想法并没有太大不同，对吧？因为在机械可解释性中，想法同样是：你有一个模型，模型有一些激活值，这些激活值并没有被直接监督来预测任何标签。它们有点像是间接监督的，但模型从未接受过任何对这些激活值的检查训练。所以，这些激活值可能会揭示其内部运作的一些信息。但思维链的巨大优势在于，默认情况下它们是英文的，对吧？所以理解发生了什么要容易得多。尤其是当概念变得更高级时。另一个有趣的点是，我们刚才谈到，我们相信在未来，所有这些模型会长时间工作，它们自主运行，对吧？所以会有更多的推理。因此，如果这是模型能力增长的一个重要维度，那么我们监督它们的能力也将相应扩展。

It's not optimized to be in any particular way because it's just not directly great, right? It's only great at how it relates to producing a high-quality output. And we realize this is actually a very powerful paradigm for being able to interpret what the model is doing, right? It's actually not a very different idea from mechanistic interpretability, right? Because in mechanistic interpretability, the idea is again, you kind of have this model, you have these activations of the model that are not directly supervised to predict any label. They're kind of indirectly supervised, but the model has never been trained with any sort of inspection of these activations. And so, these activations might reveal something about its inner workings. But the big advantage of the chain of thought is that, by default, they're in English, right? And so, it's so much easier to understand what is going on. It's especially as the concepts get more advanced. And the other interesting thing is, we were just talking about how we believe in the future where all these models work for a very long time. They work autonomously, right? And so there is much more of this reasoning. And so, if this is a big axis of how the capability of these models increases, then our ability to supervise them will scale commensurately.

是的，这实际上归结于一个原则：你不应该监督思维链。所以这实际上是我们在最初做预览模型时，决定隐藏思维链的原因。对我来说，这是主要动机。这就是为什么我甚至不想考虑以不同方式发布它。内部确实有一些讨论，但我强烈认为应该隐藏它，就是因为这个。然后还有另一个我最初没考虑到的担忧，但我觉得也非常合理：这个模型可以在一定程度上被蒸馏，这肯定也是一个重要因素。但没错，我实际上认为给模型一些私人空间……哦，顺便说一句，为什么我认为不在产品中展示思维链很重要？如果我说重要的是在训练期间不监督它们，那么我认为如果我们确实在产品中展示它们，如果我们建立一种范式，即你在产品中展示思维链，那么最终你不得不训练它们，对吧？你不得不训练它们，原因和你必须训练其他模型一样。我只是觉得那会……

Yeah, this really comes down to this principle though that, you know, you're not supposed to supervise a chain of thought. And so this is actually something when we originally, we were doing the own preview model, we made this decision to hide the chain of thought. And for me, that was a primary motivation. That was the reason I didn't really even want to consider releasing it in different ways. There definitely was a bit of internal discussion about this, but the reason I felt very strongly that we should just hide it is because of this. Then there was this other concern that I didn't initially think about, but I think it was also very valid: well, this model can be distilled to some extent, and that's definitely also been a big factor here. But yeah, I actually think that allowing the models some sort of private space... Oh, and by the way, why do I think it's important that we don't show this chain of thought in product? If I'm saying that the important thing is not to supervise them during training, well, I think if we did show them in product, if we establish a paradigm where you just show this chain of thought in product, then eventually you'd have to train them, right? You'd have to train them for the same reasons you have to train whatever models you should. And I just think that would be...

并不是所有人都想知道模型为了得到某个回复而经历的思维链。

Not all want to know what the chain of thought our model has to get to a response for.

对。我的意思是，我认为它在某种程度上是有用的，我们正试图通过摘要来捕捉大部分价值，我认为摘要是一种权宜之计。我认为长期的解决方案是让模型实时与你对话，最新版本的 Codex 和推理模型已经能做到这一点，但我觉得这还会变得更好。是的，但我认为这里有一个非常令人兴奋的点，就是不让训练信号与我们对抗，对吧？而且不……

Right. I mean, I think it would be useful to some extent, and we are trying to capture most of that value, either with summaries, which I think are kind of a stopgap. I think the longer-term solution here is having the model actually talk to you in real time, which the latest version of Codex kind of does, the latest version of the reasoning model kind of does, but I think that will get much better. Yeah, but I think there's something very exciting here about just not having the training signal fight against us, right? And not...

是的，因为我认为，如果你想长期理解模型在做什么，但你却在扩展一种与之直接对抗的方法，那你可能不会有好结果，对吧？至少其他人必须吸取惨痛的教训。所以这种解耦，我认为是一个想法，它让我对我们的能力充满希望，至少能理解这些模型的动机和泛化如何随着它们变得更好、工作更久而演变。是的，我不认为这是 AI 安全和对齐的完整解决方案，远非如此。我认为这只是我们工具箱中的另一个工具。但我希望，通过用这样的技术工具构建我们的工具箱，我们实际上可以继续逐步解决这里的根本问题。

Yes, because I think if you want to be able to understand what the model does in the long term, but you're scaling a method that is kind of going directly against it, you're probably not going to have a good time, right? At least others have to learn a bitter lesson. And so this decoupling, I think, is an idea that gives me a lot of hope for our abilities to at least understand how these models' motivations and generalization evolve as they get better, as they work for longer. Yeah, I don't think it's a complete solution to AI safety and alignment by a long shot. I think it's just another tool in our toolbox. But I am hopeful that by building our toolbox with technical tools like this, we can actually continue chipping away at the fundamental problems here.

是的。看起来在中期内，这将是非常有帮助的。可能不是长期对齐的万能解决方案。

Yeah. Seems like almost over the medium term, it's something that's going to be incredibly helpful. Probably not the catch-all solution for long-term alignment.

是的，我的意思是，我认为这是一个可以帮助我们理解的工具。我认为它对于建立对长期对齐的理解非常有用，对吧？例如，有一项与 Anthropic 合作的非常激动人心的工作，关于模型阴谋，他们研究，根据你放置模型的环境以及你如何训练它，它是否容易开始追求隐藏的目标。而支撑那整个工作的是监控渠道，对吧？就是这种概念：‘哦，你实际上可以检查模型的动机是什么。’所以，我认为从那里出发，可能会在缓解措施上带我们走向完全不同的方向，对吧？比如，也许正确的方法是改变模型的预训练数据，或者像接种提示这样的东西。我认为这些都是非常有趣的想法，但拥有这种理解能力对于评估这些方法非常有帮助。

Yeah, I mean, I think it's a tool that can help us understand. I think it's actually very useful to build understanding of long-term alignment, right? So for example, there has been this very exciting work from a collaboration with Anthropic on model scheming, where they investigate, depending on what environment you put the model in, how you train it, is it prone to start having hidden objectives that it pursues. And what enables that whole line of work is channel for monitoring, right? It's this notion of, 'Oh, you can actually inspect what the model's motivations are.' So, I think from that, that might take us in a completely different direction in terms of mitigation, right? Like, maybe the right way is changing the pre-training data of the model, or maybe it's something like inoculation prompting. I think those are very interesting ideas, but having this ability to understand this is very helpful to evaluate these.

是的，这几乎是任何进一步研究领域的基础。你还在关注对齐领域的哪些其他研究方向，或者你认为哪些领域有前景值得关注？

Yeah, it's almost like foundational for any further area of research. What are like the other research areas of alignment that you're paying attention to or that you think are promising areas to focus on?

是的，我认为对齐的许多长期挑战在于泛化，对吧？比如，我们可以训练模型在某种程度上表现良好。比如，我们可以在我们训练过的分布内的事情上大致控制它们的行为。

Yeah, I think a lot of the longer-term challenge with alignment is about generalization, right? Like, we can train our models to do well in, at least mostly to some extent. Like, we can mostly control their behavior in the things that are in distribution that we trained for.

但你知道，有些人担心的是：“当模型被要求做非常不同的事情，或者发现自己处于非常不同的情境中，或者它比以往任何时候都聪明得多，并且拥有所有这些我们还没有真正考虑过如何训练的能力时，会发生什么？”

But, you know, the things that are worrying some is like, 'Well, what happens when the model's asked to do something very different or it finds itself in a very different situation or it's much smarter than it ever was before and it has all these capabilities that we haven't really thought about how to train for.'

所以，我认为这种长期价值对齐的研究实际上是对泛化的研究。比如，模型依赖的价值观是什么？我对此非常兴奋的一条研究路线，也是我们投入相当多的，就是理解这种泛化如何回溯到预训练数据上。

So, I think the study of this kind of longer-term value alignment is really a study of generalization. Like, what are the values that the model falls back on? One line of research I'm very excited about here and something that we're investing in quite a bit is understanding how that generalization falls back onto the pre-training data.

在过去的六个月里，你对对齐的担忧是增加了还是减少了？比如，我们在这项工作上的整体趋势如何？

Over the last 6 months, have your concerns around alignment increased or decreased? Like, how are we trending overall with this work?

过去几年我对这个问题的思考方式确实发生了变化，从一个总是非常模糊、甚至难以把握或定义的问题，变成了“哦，我认为我们可以通过非常具体的技术解决方案和技术洞察来取得进展”。这就是为什么我们一直将大语言模型视为研究的核心部分，并真正确保我们在设计推理模型时考虑这一点，并围绕这些推理模型进行对齐研究。所以，我认为我总体上的信念——即存在一条研究路径能让我们最终到达一个极其美好的世界——已经大大增强了。与此同时，我对于非常强大模型的时间线肯定缩短了很多。我认为我们离那不远了。再说一次，我不认为这些模型在任何方面都很聪明，但我认为这些模型非常有变革性。所以，我相当乐观，我们可以很好地把握我们在对齐问题上的进展，如何大致评估我们模型的风险或问题，但我确实认为，作为一个行业，我们必须准备好做出权衡，并可能根据我们看到的情况放缓开发。

The way my thinking about the problem has evolved over the past few years is definitely gone from always this very nebulous problem that's very hard to even grapple with or define, to 'oh, I think we can actually make progress at it by very concrete technical solutions and technical insights.' This is why we've really been viewing LLMs as a core part of research and really making sure that we are designing our reasoning models, thinking about this, and conducting our alignment research with these reasoning models in mind. So, I think my general belief that there's a research path here that actually gets us to an extremely happy world has increased quite a lot. At the same time, my timelines to very capable models have definitely decreased a lot. I think we're not that far. And again, I don't think these models are smart in any of the ways, but I think these models are just very transformative. So, I'm quite optimistic we can keep a good grip on how we're doing on the alignment problem, how to roughly evaluate the risks of our models or the problems with them, but I do think we have to be, as an industry, really prepared to take trade-offs and possibly slow down development depending on what we see.

看到很多这类工作在各主要实验室中展开已经很有趣了。你们与 Anthropic 和 DeepMind 合作，这是自然发生的吗？鉴于你们三家确实处于最前沿，主要参与者之间是否有大量关于对齐的讨论？

It's already interesting to see a lot of this work happening across the major labs. The fact that you did this in collaboration with Anthropic and DeepMind, has that just come up organically? Is there a lot of alignment talk between the major players, given you three are really at the forefront?

肯定有一些。对这些话题确实有共同的兴趣，是的。

It's definitely some. There's definitely shared interest in these topics, yeah.

我想稍微转向 AI 内部。我觉得过去两三年里，世界上没有哪家公司比你们更受关注，尤其是运营一个研究机构是什么样的。你之前谈到过，你的工作重要部分之一是给研究人员提供舒适和空间，让他们几乎像洞穴居民一样思考模型几年后会是什么样子。我们也处在一个大规模竞争的时代，每个人都在这些编码模型上大干快上。我想知道你今天实际上如何实现这种平衡，以及你在监督这个组织时，对正确做法的思考有什么变化。

I want to shift a little bit to going inside of an AI. I feel like no company in the world has been more interested in over the last 2-3 years, and I think particularly what it's like to run a research organization. You talked before about how it's an important part of your job to give researchers comfort and space to almost be cave dwellers and think about what the models will look like in a few years. We're also in a time where there's a massive competitive race and everyone's going gangbusters on these coding models. I'm wondering how you actually operationalize this balance today and anything you've changed in your thinking overseeing this organization around the right way to do this.

我专注于高质量实验、组织安排、我们是否真的在进步、对自己诚实，并促进对结果的诚实。我认为这没有改变。尽管我们的工作会演变很多，但我相信我们还有很多工作要做。所以我不认为我们需要非常迅速地结束所有项目。所以是的，我认为没有根本性的变化。我认为变化的是紧迫感的程度，真正把我们认为最有希望的一些事情推向成果。

I focus on high-quality experiments, organizing, are we actually making progress, being honest with ourselves, and promoting honesty about the results. I don't think that has changed. Even though our work will evolve a lot, I believe we still have quite a lot of work left to do. So I don't think it's like, 'oh, we need to wrap up all our projects very quickly.' So yeah, I don't think there's a fundamental change. I think what does change is a level of urgency to really bring some of these things that we think are most promising to fruition.

多年来，OpenAI 内部有一些非常公开的时刻。你在这里很久了。回顾过去，你们做出的一些艰难决定，也许是 51-49 的抉择，真正定义了公司？从过去七八年的电影中，有哪些关键时刻让你印象深刻？

There have been some very public internal moments of OpenAI over the years. You've been here for a long time. As you reflect back, what were some of the difficult decisions that you guys made that maybe were 51-49 that really defined the company? Any key moments that stick out to you from the movie of the last 7-8 years?

哦，是的，肯定有一些戏剧性的时刻。我认为公司经历最大变化的方式并不是这些突然的变化和决定，而是运营方式的转变。我会说 OpenAI 经历了几个阶段。当我在 2017 年初加入时，它非常像一个追求许多不同想法的学术实验室，并没有太多实际进行 Scaling。我认为那是第一个大变化：与 GPT 相关的项目，我们转向了“好吧，我们实际上必须购买大型计算机。我们必须快速扩展。我们必须发展 Scaling 的科学和基础设施。”这开始了第二阶段：“好了，现在我们正在扩展，我们仍然会追求许多基础研究想法，但我们将评估它们是否可扩展。”然后是我之前谈到的一个有趣时期，ChatGPT 成为了一个大事件。完全出乎意料。我原以为它会有点不同。我实际上很惊讶文本模型竟然是第一个。

Oh yeah, there are certainly a number of dramatic moments like this. I think the ways the company underwent the most change is not really these snap changes and decisions, but more like shifts in how it operates. I will say OpenAI has gone through a couple phases. When I joined at the start of 2017, it very much felt like an academic lab pursuing a lot of different ideas, not so much scaling built in practice. I think that was the first big change: the dollar project with GPT, we moved to 'okay, we actually are going to have to buy big computers. We're going to have to scale fast. We're going to have to develop the science of scaling and the infrastructure for it.' That started the second phase of 'okay, now we're scaling, and we're still going to pursue a lot of these basic research ideas, but we are going to evaluate them for whether they are scalable.' Then there was an interesting period I talked about earlier, where ChatGPT became this big thing. Complete surprise. I thought it would look a little bit differently. I was actually surprised that text models were kind of the first thing.

我曾以为我们会处于一个生成式 AI 的视频用户率先爆发的世界，并且不得不权衡是否要追求更长期的基于文本的研究。但我认为我们预见到了这种紧张关系的出现：现在有流行的东西，但我们相信在达到目标之前它还会经历很多演变。我们处于这个阶段已经有一段时间了。现在我们开始部署 AGI 或具有经济变革性的模型。

I thought we would be in a world where video-style users of generative AI are the first big thing to take off, and we'd have to trade off pursuing longer-term text-based research. But I think we anticipated this tension would arise, where you have something popular now, but we believe it will evolve a lot before reaching the goal. That's the phase we've been in for a while. Now we're starting to deploy AGI or models that are economically transformative.

在过去一年里，你对 AI 领域的哪件事改变了看法？

What's one thing you've changed your mind on in the AI world in the last year?

我认为这确实是开始调和一种紧张关系：你构建的 AI 最终会影响世界，但在接近目标之前，它只是理论性的。现在我们需要专注于如何部署这项技术。在 ChatGPT 之外，它更多是抽象的或研究上的攀登，只有一些现实世界的使用。在过去一年里，通过编码智能体，它已经大规模渗透。

I think it's really starting to reconcile the tension between building AI that affects the world, but until you get close, it's theoretical. Now we need to focus on how we deploy this technology. Outside of ChatGPT, it was more abstract or research hill climbing with some real-world usage. In the last year, via coding agents, it has trickled in massively.

ChatGPT 正朝着与编码模型相同的方向发展，变得非常有用，成为人们生活中有意义的一部分。当你说朝着相同方向发展时，是指执行更长期的任务，还是成为可靠的助手？

ChatGPT is going in the same direction as coding models, becoming very useful and a meaningful part of people's lives. When you say going in the same way, do you mean executing longer-term tasks or becoming a dependable assistant?

这是其中的一部分，但也是成为一个可靠、值得信赖的助手或伴侣。看到年轻人使用 ChatGPT 令人惊叹。我认为对于许多高中生和大学生来说，它已经达到了那个程度。

That's part of it, but also becoming a dependable, trustworthy assistant or companion. It's amazing to watch younger people use ChatGPT. I think it's already there for many high school and college students.

我对核心语言模型世界之外的事物（比如机器人技术）的时间线很感兴趣。你对机器人领域的 ChatGPT 式时刻或缩放定律有什么看法吗？

I'm interested in timelines for things outside the core LM world, like robotics. Do you have any sense of a ChatGPT-like moment or scaling laws for robotics?

我认为有一些非常有前途的算法思路，与语言模型领域的思路相差不大。我对时间线持乐观态度，尽管它们比虚拟 AI 更长。

I think there are very promising algorithmic ideas that are not too dissimilar from the LM space. I'm optimistic about timelines, though they are longer than virtual AI.

关于模型持续改进的影响，我们社会目前对哪件事思考不足？

What's one thing we're under-thinking as a society regarding the impact of continued model improvement?

当大量智力工作可以被自动化时，会带来一些没有明显解决方案的重大问题。这涉及就业和财富集中问题。我认为这需要真正的政策制定者参与。虽然有一些乐观的看法，但从根本上说，过去有价值且昂贵的事情现在可以廉价完成。长期来看这应该是好事，但可能发生得很快。一个自动化的研究实验室或公司可能由极少数人控制，却能做很多事情。有了机器人会更疯狂，但不需要机器人。弄清楚如何治理这种由少数人组成的强大组织，是我们社会必须面对的新问题。

Getting to a point where so much intellectual work can be automated comes with big problems without obvious solutions. There's the question of jobs and concentration of wealth. I suspect this requires real policymaker involvement. Some optimistic takes exist, but fundamentally, things that used to be valuable and costly can now be done cheaply. In the long term it should be good, but it can happen quickly. An automated research lab or company could be controlled by a very small number of people, doing a lot. This gets crazier with robots, but you don't need robots. Figuring out governance of such powerful organizations made of only a few people is a new question we have to grapple with.

你在 AI 方面的工作如何改变了你对下一代应该如何培养的看法？

How has your work on AI changed the way you think about how the next generation should be raised?

我们的任务是构建 AI 和世界，让人类拥有主动权并设定方向。我们现在珍视的技术挑战可能会成为过去的必需，而挑战将更多在于弄清楚什么是重要的、我们应该做什么。在那个世界里，人们最终会有更多令人兴奋的事情可做。

Our task is to build AI and the world in a way where humans have agency and set the direction. The technical challenges we cherish now may become a past necessity, and the challenges will be more about figuring out what's important and what we should do. In that world, people can end up with more exciting things to do.

而且，我认为你仍然需要理解基础技术，无论你如何获取这些知识，才能思考这些问题。好吧，这真是太棒了，伙计。我非常感谢你坐下来聊了这么多不同的话题。我想把最后的话留给你。你有什么想对听众说的吗？无论是你正在做的研究，还是你感兴趣的产品，或者任何你想推广的东西，请随意。

And, you know, I think you still want to have an understanding of the basic technology, however you acquire it, to be able to think about these problems. Well, this has been fascinating, man. I really appreciate you sitting down and talking about so many different things. I want to make sure to leave the last word to you. Anything you want to point our listeners to, whether it's research you're doing or products you're excited about or really anything you'd like to plug, the floor is yours.

我认为我们刚刚讨论的那一系列问题，对吧？还有关于对齐和可监控性的问题。我认为这些将是极其紧迫的挑战，而且我不认为这些只是 AI 研究人员的挑战。我认为它们是政策制定者的挑战，也是我们整个社会必须思考的问题。是的，我很高兴看到一些讨论开始出现，我认为我们需要更多这样的讨论。

I think the set of problems we just discussed, right? And also the questions around alignment and monitorability. I think those are going to be very urgent challenges, and I don't think they're challenges only for AI researchers. I think they're challenges for policymakers, but also just things we have to think through as a society. And yeah, I'm happy to see some discourse starting to arise and I think we need more of it.

是的。好吧，我觉得我们可以再聊几个小时，但如果我让你无法继续改进这些模型的实际工作，那对世界来说将是一个巨大的损失。非常感谢你接受采访。这非常有趣。谢谢。我是 Jacob Effron，这里是 Unsupervised Learning，一个播客，我在这里与 AI 领域最聪明的人交谈，问他们很多关于模型进展以及这对企业和世界意味着什么的问题。我希望大家能看出来，我对此乐在其中。这是我除了在 Redpoint 做投资人的日常工作之外，在晚上和周末做的项目。但我们能请到这些了不起的嘉宾，真的离不开像你这样的听众订阅播客、与朋友分享。这才是让这一切运转起来的根本原因。所以，请考虑这样做，非常感谢你的支持和收听。我们下期再见。

Yeah. Well, I thought we could talk to you for hours more, but I'd be doing the world a great disservice by keeping you from your actual work of continuing to improve these models. Thank you so much for doing this. This was a ton of fun. Thank you. I'm Jacob Effron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint, but our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so, please consider doing that, and thank you so much for your support and listening. We'll see you next episode.