OpenAI 研究副总裁谈推理、思维链与 AI 未来

嗨，我是 First Mark 的 Matt Turk。欢迎收听 Med 播客。今天我的嘉宾是 OpenAI 研究副总裁 Jerry Tworek，他也是 Med 全球顶尖 AI 研究人员榜单成员。本期我们将深入探讨模型如何真正推理，还会走进 OpenAI 幕后：几个重大赌注如何组建团队，为什么每个人都知道一切，以及文化如何快速转变。请享受与 Jerry 的精彩对话。嘿 Jerry，欢迎。

Hi, I'm Matt Turk from First Mark. Welcome to the Med podcast. Today, my guest is Jerry Tworek, VP of research at OpenAI and a member of the Med list of the world's top AI researchers. In this episode, we go deep on how models actually reason. We also go behind the scenes at OpenAI. How a few big bets get staffed, why everyone knows everything, and how that culture shifts fast. Please enjoy this great conversation with Jerry. Hey Jerry, welcome.

你好，很高兴来到这里。

Hello, very happy to be here.

我们这次会大量讨论推理。从高层次看，当我们与 ChatGPT 对话，它说自己在思考时，幕后到底发生了什么？

We are going to talk about reasoning a lot in this conversation. At a high level, what does reasoning actually mean when we talk to ChatGPT and ChatGPT says it's thinking, what actually is happening behind the scenes?

我认为思考过程至少是一个很好的类比。在 AI 早期，我们一直有这样一个目标、梦想：试图教模型推理。我们思考它，花更多时间以获得更好的结果。如果一个人面临一个非常困难的问题，他们很少能立刻得到答案。有时他们需要找到答案，有时需要执行某些计算，有时需要查找信息，有时需要自学一些东西。推理的过程就是得到一个你还不知道的答案。在某种程度上可以称之为搜索，但并不是那种非常幼稚的搜索。搜索是一个含义丰富的词，但推理是得到答案的过程，以及你需要做的工作，这比通常认为的回答问题要更长。我认为区别在于：回答问题通常意味着你已经知道答案，你只是把答案引出来，对吧？而推理的过程是得到你不知道的答案，通常你花在得到这个答案上的时间越长，无论你需要做什么才能到达那里，结果就越好。

I think that the thinking process is at least a good analogy. As we were in the early days of AI, we always had this goal, dream of trying to teach models to reason. We're thinking about it, spending more time to get better results. If a human is posed with a very hard problem in front of them, very rarely they have answers straight away. Sometimes they need to find that answer, sometimes they need to perform certain computations, sometimes they need to look up some information, sometimes they need to teach themselves something. And the process of reasoning is like getting to an answer that you don't yet know. In some way it can be called search, but it's not really very naive search. Search is a loaded word, but reasoning is the process of getting to an answer and the work that you need to do that is longer than what usually is considered answering a question. I think that difference is here: answering a question usually means you already know the answer and you just elicit the answer, you know? And the process of reasoning is getting to the answer that you don't know, and usually the longer you spend on getting to this answer, for whatever you need to do to get there, the better it gets.

自从你们发布 01 以来，我想大概是一年多前，2024 年 9 月，我们都熟悉了思维链的概念。用外行的话说，就是当你查询 ChatGPT 时看到的小消息，它告诉你它展示了它的工作过程。它告诉你它做了什么。这实际上做了什么？它是一个逻辑树，逐个排除选项吗？实际发生了什么？

And we've all become familiar since you guys released 01, I guess a little over a year ago now in September 2024, with the concept of chain of thought, which is in layman's term the little messages that you see when you query ChatGPT and it tells you it shows its work. It tells you what it does. What does that actually do? Is that a logical tree and it eliminates option after option? What actually happens?

语言模型在它们的基本层面上，通常被称为下一个词预测机器。在强化学习时代这并不完全准确，但它们仍然主要操作在大多是文本的词元上。如今的语言模型也是多模态的，但简单来说，语言模型生成文本。而思维链就是它们用人类语言和人类概念表达出来的思考过程。所以我们看到的魔力，为什么这一切成为可能，是因为当你在整个互联网、大量人类知识和人类思维过程上训练时，模型开始以某种方式学习像人类一样思考，并以某种方式像人类一样得到答案，这是从大量人类预先生成的文本和基于人类训练数据中看到的。然后思维链基本上就是引出语言模型中像人类一样思考和得到答案的能力。早期思维链工作很多是在解决数学谜题。第一个最著名的引出语言模型思维链的提示就是所谓的“让我们一步步解决”。语言模型中有一个非常经典的结果：如果你问它们一个数学表达式或某个谜题，它们会试图给你一个答案，试图预测下一个词元，但会失败。这是一件难事；它们无法在一个词元跳跃中计算出来。但如果你要求它们“请一步步做”，它们就会开始思考：“好吧，我不知道答案，但得到答案的第一步是这个。”然后它们写出一系列文本，一系列词元，进行第一部分计算、第二部分、最后一部分，然后连接这些部分，最终得到答案。所以思维链基本上是用文字编码的思考过程，就像人类在纸上一步步从开始到结束解决问题一样。

Language models, on their own fundamental level, are often called next-token prediction machines. And that's not completely accurate in the age of reinforcement learning, but they still operate mostly on tokens that are mostly text. Language models these days are also multimodal, but to simplify a little bit, language models generate text. And what chain of thought is, is their thinking process verbalized using human words and human concepts. So the magic that we are seeing, why this is all possible, is that while you are training on all of internet, on a lot of human knowledge and human thinking process, the model starts learning in some ways to think how humans do and in some ways get to the answers how humans do, from seeing humans do it a lot in the text that was pre-generated and that was based in the training data on humans. And then chain of thought is basically eliciting that capability in language models of thinking and getting to an answer like humans do. A lot of what early chain of thought work was doing was kind of solving math puzzles. And the first most famous prompt to elicit chain of thought in a language model was the so-called "let's solve it step by step." There is this very classical result in language models that if you ask them something like a mathematical expression or some puzzle, they will try to give you an answer, they will try to predict the next token, but they fail. It's a hard thing; they can't compute it in one token jump. But if you ask them like "please do it step by step," they will start thinking, "Okay, I don't know the answer, but the first step of getting to the answer is this." And then they write a chain of thought, which is a series of text, a series of tokens doing the first part of the computation, the second part, the last part, then they connect those things, and then they can get to the answer. So chain of thought is basically a process of thinking encoded in words, how humans would solve a problem on a piece of paper, going step by step from start to end.

既然时间——我指的是思考所花的时间——对推理这个概念如此重要，那么当我们在 ChatGPT-5 中处于自动模式，它说会自动决定思考多长时间时，模型是如何决定思考时长的？那里发生了什么？

And since time, and by that I mean the time spent thinking, is so important to that concept of reasoning, how does a model decide how long to think when we're in ChatGPT-5 and we're in auto mode and it says that it's going to decide automatically how long to think? What happens there?

这基本上是我们优化过程的一部分，部分是为了用户的满意度和他们的期望。因为当你有一个思考过程时，你需要平衡两件事：结果的质量。正如我们所说，并且我们在发布 01 时展示了相当不错的缩放定律，模型思考的时间越长，得到的结果越好。但人们也不喜欢等待。等待是你可以做其他事情的时间损失。每个人都想尽快得到结果。有句老话：你可以得到便宜、快速或优质，但只能选两个。这也适用于语言模型。这里有一个权衡，而且很微妙。这就是为什么我们也向用户展示了一些这种权衡，你可以选择高推理模型和低推理模型。这最终是同一个模型；我们只是调整了参数，告诉它想得更长或更短。我们尝试编码一些启发式规则，关于我们认为用户会想要什么：当花更长时间得到更好的答案时，等待是否值得。但这有点像试图猜测用户的预期。

It's basically part of our optimization process, partially for the happiness of the users and what they want to expect. Because when you have a thinking process, you need to balance two things: the quality of the result. As we said, and there have been pretty great scaling laws that we demonstrated with the release of 01, the longer the model thinks, the better result you get. But also people don't like waiting. Waiting is time lost that you could do something else. Everyone wants to get results as quickly as possible. And there is this saying: you can get cheap, fast, or good, and you can take two. And that applies to language models as well. There is a trade-off, and it's delicate. That's why we also expose some of that trade-off to the users, where you can have a high reasoning model and a low reasoning model. And this is in the end the same model; we just tweak the parameter which says we want you to think longer or shorter. And we try to encode some heuristics of what we think the users will want: when getting to an answer a little bit longer and getting to a better answer is worth the wait versus not. But it's a bit of trying to guess the anticipation of the users.

在特定情况下，它们应该思考多少才合适？

What's the right amount of thinking for them in this particular situation?

嗯，很有意思。所以这更由用户驱动，更像是一种用户体验问题。归根结底，问题在于你愿意等多久才能得到答案？你总是可以等更久，得到更好的答案。

Well, fascinating. So, it's more user-driven, so it's more like a user experience kind of thing. In the end it is because the question is like how long do you want to wait for an answer? You can always wait longer and get an even better answer.

距离你领导的世界首个推理模型 01 发布已经一年多了。这期间经历了怎样的历程？从 01 到 03，再到最近的 ChatGPT-5，你如何描述过去一年中推理能力在这三个模型上的演变？

It's been a little over a year since the release of the world's first reasoning model 01, which is an effort that you led. What has been the journey since? So, there was 01, then there was 03, then most recently ChatGPT-5. How would you characterize the evolution of reasoning specifically across the three models in the last year?

在某种程度上，我描述我们推理或扩展强化学习研究项目的方式是，我们进行一系列规模越来越大的运行，每一步都更具野心。每一次我们都尝试做更多的事，更大的规模，从而得到比上一个更好的模型。显然，我们不会发布所有训练的模型。有些我们发布，有些我们认为需要再等一等，等到它们在用户手中大放异彩的时刻。但 01 是我们决定发布的第一个模型，旨在向世界展示这类模型的存在。老实说，01 主要擅长解谜，可能还有一些零星的思考问题，但它还不是一个非常有用的模型。它更像是一个技术演示，而不是真正成熟的产品。但我们觉得我们有很酷的东西，想以 OpenAI 的身份与世界分享。我认为 03 显著改变了这一点。在某种程度上，它是一个有实际用处的模型。有点自私地说，那是我开始大量使用 ChatGPT 的时刻。我现在基本上是一个完全沉迷于推理模型的用户。在 ChatGPT 中，我几乎只使用推理模型，因为只有这些模型的输出和结果我才信任。我认为 03 使用工具、利用各种来源的上下文信息并坚持得出答案的能力，真的带来了 AI 轨迹上的一次板块移动。我觉得我们在那里做了一些非常了不起的事情。GPT-5 在某种程度上可以看作是 03.1，是对同一事物和概念的迭代。我和我的团队现在追求的是下一步，那将是与模型交互方式的又一次重大飞跃——模型能力更强、思考时间更长、并能自主与更多系统和信息源交互。但与此同时，我们继续在 03 技术之上构建很多东西，比如 Codex，我认为编码智能体是目前第一个相当成功的基于 AI 的智能体产品。还有像计算机使用智能体，现在叫 ChatGPT 智能体，以及预搜索和其他一些东西，我们将继续在 03 代技术之上构建。

In some way, how I characterize our reasoning or scaling up reinforcement learning research program is we do a series of scale-up runs that are progressively more and more ambitious. Every one we try to do something more, something larger scale, something that should result in a better trained model than the last one. And obviously we don't release all the models that we train. Some we release, some we think like I need to wait a little bit longer for the moment where they will have their time to shine in the hands of the users. But 01 was the first model we decided to release as kind of like to demonstrate to the world there are those models. And 01, to be perfectly honest, it was really mostly good at solving puzzles. And like maybe a few kind of thinking problems here and there, but it wasn't yet very useful model. It was almost more like a technology demonstration than a really polished product. But we were thinking we have something cool and we wanted to share it with the world as OpenAI. 03 I think changed that pretty significantly. In some way it is a model that is meaningfully useful. And like you know, a little bit self-serving, but it was the moment when I started using ChatGPT quite a bit. And I'm basically a user completely hooked on reasoning models. In ChatGPT right now I use basically exclusively reasoning models because those are the only models that I trust the output and their result. And I think 03, its ability of using tools and getting to answer leveraging a lot of contextual information from various sources and persevering to get to that, has really been something like you know, I think there was a little bit of a tectonic shift in the trajectory of AI. And like you know, I think we did something really really great there. Like GPT-5 in some way can be considered as 03.1. It's a little bit of an iteration of the same thing and the same concept. And what I am after and my team right now is something next that would be the next pretty significant jump of how we interact with models that are even more capable, thinking even longer, and interact with even more systems and sources of information on their own journey. But separately, in the meantime we continue to build a lot of things on top of 03 technology like Codex, which I think coding agents are at the moment the first pretty successful agentic products built on top of AI. There are things like computer using agent, called ChatGPT agent right now I think, and the pre-search and a few other things that we will keep on building on 03 generation technology.

很好。

Great.

好的。我们马上会更详细地讨论所有这些。但在此之前，让我们谈谈你的经历。我认为这对我们所有人来说都是一个非常迷人的话题，你们正在改变世界。所以我很想知道，我们也都想知道，那些产生如此影响的人是谁，他们的人性一面。那么从头开始，你是在波兰长大的，对吗？

All right. So we're going to go into all of this in much greater detail in a minute. But before we do that, let's talk about your journey. I think it's a super fascinating topic for all of us, like you guys are changing the world. So I'm curious and we're all curious about the people, the human aspect of who those people are that are having such an impact. So starting from the beginning, you grew up in Poland I believe right?

是的。是的。所以你是在波兰长大的。请带我们回顾你的成长岁月，以及你是如何进入这个领域的。

Yes. Yes. So you grew up in Poland. Walk us through your formative years and how you got started in this field.

很高兴这样做。有趣的是，这几乎就像一颗水晶从某物开始，你在开头放了一点东西。有一个重要的部分，是我旅程的起点，我不知道它从何而来，因为它从我生命的最初就存在，在一个我甚至不知道何时开始的时刻。它一直伴随着我。我一直认为成为科学家、从事科学是人类最高的使命，我真的不知道这从何而来。也许我父母在我一岁时给我唱了摇篮曲之类的。无论如何，从我记事起，我就想成为一名科学家。早年我还发现自己在这些方面有天赋。我上学时发现自己比周围人学东西稍快一些，至少在波兰中部的一所普通学校是这样，这让我更喜欢学习数学和科学，因为感觉很好。感觉这天生适合我。我作为一个非常普通的孩子长大，只是一个有点书呆子气的家伙，试图平衡对科学、编程、数学的兴趣和社交生活。我生活中确实有过一些派对阶段。但我认为最重要的部分和时刻是我真正上大学的时候——华沙大学——我最终决定学习数学。在 18 岁左右，我对生活的设想是成为一名数学家，拿着铅笔坐在房间里，用一张纸解方程。这是我 18 岁时的梦想，关于生活应该如何度过以及我想在生活中做什么。我的性格也再次构建，真正欣赏扎实的科学、追求真理、伟大的工程以及所有这些方面。

Happy to do that. And interesting fact, it's almost like a crystal starts from something and you put a little bit something in the beginning. There is one part that was important and like the starting point of my journey where I didn't know where it came from because it was there with me from the very beginning of my life in a moment that I don't really know when it started. It just was always there with me. I always thought that being a scientist and doing science is the highest calling a human can have and I don't really know where it came from. You know, my parents maybe were singing the light right lullabies to me when I was one or something like that. Whatever. Basically since I remember I wanted to be a scientist. In the early years I also discovered I have talent for those things. I went to school and I saw I get things slightly faster than people around me. At least in a regular school in the middle of Poland, which made me kind of like doing those things like studying maths and science a little bit more because it felt good in a way. It felt like this is something that naturally fit me. I grew up as a very regular kid, just being a slightly nerdy guy and trying to balance my side of being interested in science, programming, maths, and having some social life. And I definitely have had some kind of party arc in my life. But I think the most important part and moment was when I actually went to university, University of Warsaw, and I decided in the end to study mathematics. At around that time of being 18, my idea of life was to be a mathematician with a pencil sitting in a room with a piece of paper and solving equations. This is kind of my 18-year-old dream of how life should be lived and what I want to do in my life. And I do have my personality built again in a way of really appreciating solid science, pursuit of truth, great engineering, and all those aspects.

但我确实有点格格不入、叛逆的倾向。学了几年数学后，我意识到自己真的很喜欢数学，而且相当擅长，但我不太喜欢学术界。我意识到自己不想留在学术界或大学里；它太死板、太结构化，我不知道自己是否会感觉良好。对于 21 岁左右的我来说，那是一次相当大的信仰危机。我曾一度失去人生目标。所以我做了一个非常简单的第一性原理思考：我拥有数学学位，需要找份工作糊口。什么工作能用到数学呢？看看 2010 年或 2011 年左右的就业市场，我决定成为一名交易员，以此为生，既能做我喜欢的事——数学，又能获得职业生涯。我在摩根大通获得了一份实习，在交易大厅的股票衍生品部门。在那里待了 6 个月，学习交易是如何运作的。完成学业后，我收到了摩根大通老板的老板的消息，说：‘嘿，Jerry，你是我见过的最好的实习生之一。我们真的很喜欢和你一起工作，我们要离开银行去创办一家新的对冲基金。你愿意跟我们来吗？’对于一个 21、22 岁的 Jerry 来说，这听起来像是一个很酷的冒险。它有足够多有趣的问题需要解决，而且那种尝试新事物、雄心勃勃的感觉正是我喜欢的。所以我去了伦敦。那家公司不幸没有成功，但那是艰难而雄心勃勃的，不是每件事都能成功。我又尝试了一次，在阿姆斯特丹和其他几个人从零开始创办了另一家对冲基金。我在那里又工作了几年，最终感到厌倦了。做交易是一个有趣且令人兴奋的问题，但几年后我感觉自己不再成长。与此同时，我和一个朋友开始聊起人工智能。真正吸引我的是强化学习，特别是 DeepMind 的人在 2013 年左右训练的 DQN 智能体，不过我是几年后才了解到这些结果的。在我看来，2012 年的 ImageNet 结果并不那么重要。大学期间，我了解了一些经典人工智能，当时神经网络并不流行，但我还是学了。我学习了支持向量机和各种训练分类器的方法。对我来说，如果你有足够的参数并努力调整，你就能将分类器拟合到任何你想要的东西上，这几乎是显而易见的。但我从不认为分类器是智能的东西。分类器学习一个从输入到输出的函数；你可以不断训练它来逼近得更好。我当时忽略的是，当你能够更好地拟合任何函数时，你就可以开始塑造行为和策略。我真正看到这一点是在 DQN 的结果中，他们将同样的方法——神经网络，虽然不是特别大或令人印象深刻——应用于经典的强化学习领域，来解决简单的电脑游戏。那些简单的神经网络和简单的学习算法开始学习相当复杂的电脑游戏，并表现出非常有趣的行为。我看到这些结果后，心想：‘这就是我余生想做的事。’我搜索了世界上哪里可以做强化学习。谷歌 DeepMind 和 OpenAI 出现了，当时它们还很小，有些知名度。

But I definitely have like a little bit of a misfit, rebellious tinge to it. After a few years of studying mathematics, I realized I really like math and I'm quite good at it, but I didn't like academia that much. I realized I don't want to stay in academia or university; it felt too rigid and structured in a way that I didn't know if I would feel good. For young me, around 21, that was a pretty big crisis of faith. I had a moment where I lost purpose of life. So I did a very simple first principles thinking: I graduated with a degree in mathematics, I need to get a job to get food. What job can I do to use mathematics? Looking at the job market around 2010 or 2011, I decided to become a trader and trade for a living, as a way to do what I like—mathematics—and get a career. I got a quick internship at JP Morgan, on the trading floor in the equity derivatives group. Spent 6 months there learning how trading works. Finished my degree, and then I got a message from my boss's boss at JP Morgan saying, 'Hey Jerry, you were one of our best interns ever. We really liked working with you and we are leaving the bank to start a new hedge fund. Would you want to come with us?' For a 21 or 22-year-old Jerry, that sounded like a cool adventure. It had enough interesting problems to solve and the kind of trying something new and ambitious that I generally liked. So I was in London. That company didn't really work out unfortunately, but it was hard and ambitious and not everything works out. I tried again starting another hedge fund from scratch with a few other people in Amsterdam. I worked there for a few more years and eventually I got bored. Working in trading is an interesting and exciting problem, but I stopped feeling I was growing after a few years. At the same time, together with a friend, we started chatting about AI. What really drew me to artificial intelligence was reinforcement learning, specifically the DQN agents trained by people at DeepMind around 2013, though I learned about those results a few years later. From my perspective, the 2012 ImageNet results weren't that significant. During my university years, I learned about classical AI and neural networks weren't very fashionable, but I still learned about them. I learned about SVMs and all kinds of methods to train classifiers. For me, it was kind of obvious that if you have enough parameters and tweak it hard enough, you will fit a classifier to whatever you want. But I never considered classifiers a smart thing. Classifiers learn a function on some set of inputs to some outputs; you can keep training it to approximate better. What I missed back then is that when you can fit any function better, you can start shaping behaviors and strategies. I really saw that in the DQN results, where they applied the same things that worked in ImageNet—neural networks, not particularly big or impressive—to the classical field of reinforcement learning to solve simple computer games. Those simple neural networks with a simple learning algorithm started learning pretty complex computer games and exhibiting very interesting behaviors. I saw those results and I was like, 'This is what I want to do for the rest of my life.' I searched where you can do reinforcement learning in this world. Google DeepMind and OpenAI came up, at that moment pretty small and somewhat known.

你在 2019 年加入 OpenAI，对吧？所以那是在早期，仍然处于 OpenAI 的非营利时代。你是怎么和他们联系上的？

You joined OpenAI in 2019, right? So very much in the early days, still very much in the non-profit era of OpenAI. How did you connect with them?

我只是通过网站申请的。我做了世界上最无聊、最无趣的事，就是打开 openai.com/jobs，申请，发送简历，希望他们回复。幸运的是，他们回复了。我不知道当时 OpenAI 收到了多少简历，但肯定比今天少得多。我去了那里，心想，只要做强化学习，做什么都行。

I just applied through the website. I did the most boring and uninteresting thing in the world, which is openai.com/jobs, apply, send resume, and hope they respond. Luckily, they did. I don't know how many resumes OpenAI was getting at that time, but definitely much less than today. I came there and I was like, it doesn't matter what I do as long as it's reinforcement learning.

所以你在 2019 年加入，对强化学习充满热情。那是在 Dota 2 时刻前后吗？因为有趣的是，在 2019 年早期，OpenAI 做了很多专注于强化学习的工作，对吧？然后后来才有了无监督学习的 GPT 时刻，但它是从强化学习的根基开始的。你具体参与了那个项目吗，还是等你出现时它已经太超前了？

So you joined in 2019 with a passion for reinforcement learning. Was that around the Dota 2 moment? Because OpenAI, interestingly, in those early days of 2019, did a lot of reinforcement learning focused work, right? And then there was a whole unsupervised learning GPT moment that happened afterwards, but it started from roots in reinforcement learning. Did you work on that project specifically or was it too advanced by the time you showed up?

我参与的项目是 OpenAI 的机器人项目，它与 Dota 项目共享相同的代码和方法。一方面，Dota 项目是 OpenAI 向世界展示扩展强化学习能力的方式。

The project I worked on was the robotics project at OpenAI, which shared the same code and same methods as the Dota project. On one hand, the Dota project was OpenAI's way to demonstrate to the world what scaling up reinforcement learning can do.

在某种程度上，这就像把 2013 年的 DQN 智能体拿来，然后下苦功让它变得越来越大，解决越来越难的问题。OpenAI 从一开始就意识到，你需要一个大规模系统才能学到真正有趣且复杂的行为，这简单但天才。这就是 Dota 试图展示的一种方式：通过扩展强化学习，我们可以解决相当复杂的环境。当时我们还在 OpenAI 指导三个强化学习项目。第二个是机器人技术，应用我们已知能解决复杂电脑游戏的相同方法。它们能解决实际问题吗？OpenAI 一直乐观且雄心勃勃，试图看看如果我们能扩展到解决 Dota，那它能不能装我的洗碗机、叠我的衣服、建一栋房子？这就是我们在做的。我参与的项目专注于灵巧操作，这在当时乃至现在对训练策略来说都是一个难以捉摸的挑战。我们最终展示了一个由神经网络控制的手能够解开魔方，这是一项相当精细和复杂的任务。

And in some way, it was like taking the 2013 DQN agents and just doing all the hard work of making it bigger and bigger and solving harder and harder problems. OpenAI generally from the very beginning was aware, and it was simple but genius, that you need to have a large-scale system to learn really interesting complex behaviors. That was one way Dota was trying to show that by scaling up reinforcement learning we can solve pretty complex environments. Then there was another project: we were directing three reinforcement learning projects at OpenAI at that time. The second one was robotics, applying the same methods that we now knew could solve pretty complex computer games. Could they solve practical problems? OpenAI was always optimistic and ambitious, trying to see if we can scale up to solve Dota, can it load my dishwasher, can it fold my clothes, can it build a house? That was what we were doing. The project I was working on focused on dexterous manipulation, which back then and still continues to be an elusive challenge for trained policies. We got to a showcase demonstrating that a hand controlled by a neural network was able to solve a Rubik's Cube, which is a pretty delicate and complex task.

那么，快进到今天，还是关于 OpenAI 幕后和那里的生活。Jerry 的一天是怎样的？像你这样的人做什么？你读论文、训练模型、管理团队。一天是什么样的？

So, fast forward to today, still in the same vein of behind the scenes at OpenAI and life there. What's a day in the life of Jerry? Like, what does somebody like you do? You read papers, you train models, you manage teams. What's a day like?

我的一天出奇地一致。送孩子上学后，我很早就到办公室。然后我整天做的就是和其他研究人员交谈。我每天每时每刻都在和其他研究人员交谈，这基本上是我唯一做的事。我从别人那里获取想法，和他们讨论，和一个人头脑风暴，然后转到另一个人那里，重复同样的过程并迭代。通过这种方式，我们不断优化我们的研究计划。有时是小组会议，小组会议有它们自己的团队动态。但这基本上就是我做的全部。唯一变化的是每次会议和每个人之间的研究主题。

My days are surprisingly uniform. I come to the office early after driving my kids to school. Then, what I do all day is basically talk to other researchers. I talk to other researchers all day every day, and this is basically exclusively what I do. I take ideas from people, bounce them, brainstorm with one partner, then move to another one and do the same thing over and over and iterate. In that way, we keep refining our research program. Sometimes those are group meetings, and group meetings have their own team dynamics. But that is basically exclusively what I do. The only thing that changes is the topics of research from meeting to meeting and from person to person.

研究中的优先级是如何确定的？这些可能的项目范围，是自上而下还是自下而上？是人们提出想法然后其他人审查吗？这是如何运作的？

How are priorities in research determined? This range of possible projects, is it top-down, is it bottom-up? Do people suggest ideas and others vet them? How does that work?

构建、组织和领导研究项目的艺术，是我在 OpenAI 的旅程和职业生涯中很快学会欣赏的东西。我们擅长的一件事就是构建研究项目。这是一种独特的混合体。你不能说它是自上而下的，也不能说它是自下而上的。它是两者的混合，平衡了所有重要方面。OpenAI 体现的一点是，我们所有人都只专注于很少的几个项目。项目并不多。OpenAI 并不试图做所有事情。我们不想拥有一个投资组合或多个不同的赌注。我们的想法是，我们把少数核心事情做得非常好，并投入大量精力，这意味着需要很多人一起工作在同一个大规模、大雄心的项目上。我们有几个这样的项目。总数大概是三到四个，取决于你怎么算，仅此而已。从这个角度来看，人们没有最终的自由。并不是人们来到 OpenAI 说，‘嘿，我想做这个’，然后就去做，因为你需要朝着这四个项目之一的目标努力。然后在那些项目内部，我们尽量相对自下而上，只要它有助于实现这些目标。研究负责人的最重要部分是确保所有研究人员朝着这个共同目标努力，这样他们就不会在思考和做事方式上分裂。这是一件极其困难的事情。这是一份非常艰难的工作，而且并不总是容易看出它有多微妙。但这很大程度上就是它的本质。我不认为自上而下的研究结构在研究组织中有效。我真的不相信它，因为你雇佣的是世界上最聪明的一些人。OpenAI 有非常聪明的人，你不能直接告诉他们该做什么。他们需要自己弄清楚该做什么。但他们不能在整个空间里找出什么是酷的事情。他们需要在项目需要的空间内，找出什么最能推进 OpenAI 的研究目标。

The art of structuring, organizing, and leading a research project is something I learned to appreciate very quickly in OpenAI's journey and in my career. There is something we are good at: structuring research projects. It's a unique mix. You can't say it's top-down, you can't say it's bottom-up. It's a mix of those two, balancing all the important aspects. One thing OpenAI embodies is that we all work on a very few projects total. There are not that many projects. OpenAI is not trying to do everything. We are not trying to have a portfolio or multiple different bets. The idea is we do a few core things really really well and put a lot of effort there, which means there needs to be a lot of people working together on the same large-scale, large-ambition project. We have a few of those. The whole number is probably three or four, depending on how you count it, and that's it. From that perspective, people don't have ultimate freedom. It's not that people come to OpenAI and say, 'Hey, I want to do this,' and they just do it, because you need to do something towards the goal of one of those four projects. Then within those projects, we try to be relatively bottom-up as long as it feeds into those goals. The most important part of the research lead is key to making sure all the researchers are working towards this one shared goal so that they don't fracture in their own ways of thinking and doing things. It's an incredibly hard thing. It's a very hard job and not always easy to see how delicate it is. But that's a lot of what it is. I don't think top-down structuring of research works in research organizations. I really don't believe in it because you are hiring some of the smartest people in the world. OpenAI has incredibly smart people to kind of tell them what to do. They need to figure out what to do. But they cannot figure out in the whole space of things what are cool things to do. They need to figure out from within the space of what the project needs and what could advance the research goals of OpenAI the most.

同时从事这三四个项目的团队之间合作到什么程度？因为我想象自己站在你的角度，OpenAI 的角度，可能有一种紧张关系：一方面想要普遍合作，但另一方面这可能是世界上最重要的知识产权。所以你大概想确保不是每个人都知道所有事情的一切。嗯，也许不是。我在这里猜测。你如何看待这种合作与某种知识产权保护之间的平衡？

To which extent is there collaboration between the teams working on those three or four projects at the same time? Because I would imagine putting myself in your shoes, OpenAI's shoes, there is probably a tension between wanting to be collaborative in general, but equally this is probably the most important IP in the world. So you probably want to make sure that not everybody knows everything about everything. Well, perhaps not. I'm speculating here. How do you think about that collaboration versus some protection of IP?

你会惊讶的，但事实是，在 OpenAI 的研究部门，目前大约不到 600 人，每个人都知道所有事情。真的，就是这样。我们一直完全透明。在某种程度上，如果有一个研究人员至少没有机会了解所有事情，那你就有点搬起石头砸自己的脚，因为他们没有最好的信息来以最佳方式完成工作。确实存在一些知识产权流失的风险，但我个人认为，不做好事、人们不了解研究、无法做最好研究的风险要高得多，这是我处理这些事情的方式。因此，我们在研究内部非常透明，这是我们的运营原则之一，因为目标是尽我们所能做最好的研究，并因此训练出最好的模型。总的来说，文化是非常协作的。

You'd be surprised, but the truth is in research at OpenAI, which is around slightly less than 600 people at the moment, everyone knows everything. Really, it does. And we always have been fully transparent. In some way, you are a little bit shooting yourself in the foot if there is a researcher that doesn't at least have the chance to learn about everything because they don't have the best information to do their job in the best way. There is some risk of losing IP, but I think the risk of not doing the right thing and of people not being informed about research and not being able to do the best research is much higher in my personal opinion and how I approach those things. So, we are extremely internally transparent within research, and that is one of our operating principles, as the goal is to do the best research we can and train the best models we can consequently. The culture generally is very collaborative.

你知道，当你有 600 个人时，总会有一个人因为另一个人奇怪地看了他一眼而不喜欢对方，或者一个人觉得另一个人有体味，或者就是不喜欢对方的想法。这种情况确实会发生。这些都是人性，是真实的人类行为。但总的来说，在大的层面上，我认为我们确实相信我们共同追求一个比我们每个人都更伟大的目标。这是一个非常正和的游戏，因为人工智能似乎变得越来越重要。而 OpenAI 的成功远非板上钉钉，它取决于我们每天做好工作。所以，我们有很多同舟共济的感觉，以及我们需要相互依赖来完成工作、实现共同使命的事实。所以，总的来说，尽管人性有时会带来一些阻碍，我认为在大的层面上，OpenAI 是非常协作的。

Like, you know, it is always the case when you have 600 people, when you have groups of people, there is always one person who doesn't like another person because they looked at them weirdly, or one person thinks the other smells bad, or just doesn't like their ideas. That does happen. Those are humans, and those are the real human things. But generally, at a large scale, I think we really have this belief that we are together in this goal that's larger than every one of us. It's a very positive-sum game because AI seems to be getting only more and more significant. And the success of OpenAI is far from guaranteed. It depends on us doing great work every day. So there is a lot of feeling of shared fate and the fact that we all need to rely on each other to do our job to achieve this shared mission. So, generally, with all the caveats of human nature getting in the way sometimes, I think on a large scale, OpenAI is very collaborative.

你们是如何保持这种发布节奏的？从外部来看，我觉得研究在某种程度上感觉像是一件长期的事情，而另一方面，你们似乎在整个组织中不断地发布产品，包括核心模型，比如你们在一年内从 01 到 03 再到 GPT-5。你们是如何平衡这一切的？为什么你们能这么快地发布？

How do you all manage to keep that pace of releases? It seems to me from the outside that there is a tension between research, which in some ways feels like it could be a long-term thing, and on the other hand, you guys seem to be just shipping and shipping across the organization, including core models, like you went from 01 to 03 to GPT-5 in like a year. How do you balance all of that? Why are you able to ship so quickly?

我认为根本原因在于，总的来说，在我的世界观里，OpenAI 是一家划时代的公司，我们身后有着不可思议的势头。我们知道过去做得很好，需要继续下去。我们有非常聪明的人。世界上最优秀的人才都想来 OpenAI 工作，这意味着每个人的产出都非常高，每个人都做了很多。所以我们有推动我们前进的势头。我们有非常优秀的人一起工作。我们有良好的研究组织方式，并且可以从硅谷借鉴很多快速完成事情的方法。而且人们通常对工作非常兴奋。每个人都感受到了我们正在做、试图做的事情的重要性和潜力，因此 OpenAI 的人倾向于非常努力地工作。让优秀的人对他们所做的事情充满热情，并且相当好地合作，通常会导致完成很多事情。我们明白，在历史上只有一次机会来构建、部署和开发人工智能，人们希望以最好的方式做到这一点。

I think the fundamental reasons are that, in general, OpenAI, in my worldview, is a generational company in a way that we have incredible momentum behind us. We know we were doing pretty great in the past and we need to continue that. We have incredibly smart people. The most talented people in the world are all coming and want to work at OpenAI right now, which means every single person's output is incredibly high, and everyone really does a whole lot. So we have momentum that carries us forward. We have really great people that work together. We have a good operating way of structuring research and can borrow a lot from Silicon Valley on how to get things done quickly. And people are generally very excited about work. Everyone feels the weight and potential of what we are doing, what we are trying to do, and because of that, people at OpenAI have a tendency to work pretty hard. Having great people excited about what they are doing, all working together reasonably well, generally results in doing a lot of things. We understand that there is only one time in history where AI is being built and deployed and developed, and people want to do it in the best way possible.

你们大量使用自己的工具吗？我记得 Fiji Seema 前几天在推特上说，你们今天在 Dev Day 上宣布的很多东西都是用 Codex 写的。这是日常体验的一部分吗？你们会用模型来提出模型的新想法吗？你们用 Codex 写代码吗？这是怎么运作的？

Do you all use a lot of your own tools? I think Fiji Seema was tweeting the other day that a lot of what you announced at Dev Day today was written by Codex. Is that part of the daily experience? Do you use a model to come up with new ideas for models? Do you use Codex to write the code? How does that work?

是的，我们确实大量使用 Codex 进行编码，而且这只会越来越好。正如我所说，我经常使用 ChatGPT，虽然不出所料，我并不是用它来想点子，而是用来回答我的很多问题。我认为我现在是 ChatGPT 的重度用户，很乐意每月支付 200 美元，而且我觉得物有所值。他们让我付费，我对此很满意，因为这样你就能获得相当慷慨的使用限制，不会真正遇到瓶颈。

Yeah, we definitely use Codex a lot for coding, and this is only getting better. As I said, I use ChatGPT a lot, although not surprisingly not that much for coming up with ideas, but for a lot of questions I have. I think I'm a pretty heavy user of ChatGPT right now, happily paying $200 a month for it, and I think I'm getting what it's worth. They are making me pay, and I'm pretty okay with this because then you get pretty generous usage limits and not really bottlenecked on it.

谢谢你说了这么多。让我们换个话题，回到这一切是如何运作的。那么，思考 OpenAI 的现代 AI 系统的正确方式是什么？我说的现代是指 2025 年 10 月，而不是 9 个月前。所以，把它看作是预训练和强化学习的结合是正确的思考方式吗？首先，这是正确的思考方式吗？其次，如果是的话，在高层面上，这两者之间是如何衔接的？之后，我想稍微深入探讨一下强化学习，让这变得对大家很有教育意义。

Thank you for all of that. Let's switch tags and go back to how all of this works. So is the right way to think about modern AI systems at OpenAI? By modern I mean as of October 2025 versus the old days of 9 months ago. So is the right way to think about it as a combination of pre-training and RL? First of all, is that the right way to think about it? And secondly, if so, just at a high level, how does the articulation between both of those work? And then after that I'd love to do a little bit of a deep dive on RL to make this very educational for folks.

今天的语言模型基本上可以理解为首先进行预训练，然后在其上进行强化学习。没有预训练，强化学习就无法工作，同样地，预训练模型有很多局限性，如果不做类似强化学习的事情就很难解决。所以我认为这两部分都会保留。我认为它们结合的方式未来可能会演变。任何事情都不应被视为教条和固定不变的。我们需要不断找出训练更好模型的方法，这正是我们正在努力做的。有趣的是，我要归功于 Ilya，他非常有远见。当我在 2019 年初加入 OpenAI 时，我记得有一次研究全员大会，Ilya 上台谈论 OpenAI 的研究计划是什么。他在 2019 年初说的是，在我们能获得的所有数据上训练一个大型生成模型，然后在其上进行强化学习。这就是 2019 年初 OpenAI 的研究计划，而这正是我们今天所做的。现在算法变了，架构变了。我不认为他当时甚至考虑过 Transformer。GPT 就像一个有人在玩的玩具例子，但在世界上所有数据上训练大型生成模型然后进行强化学习的目标已经存在于 OpenAI 的核心 DNA 中，而这正是现在正在发生的事情。

Today's language models can basically be thought of as first they are pre-trained, then you do reinforcement learning on it. The reinforcement learning would not work without pre-training, and in a similar way, pre-trained models have a lot of limitations that are very hard to resolve without doing something that looks like reinforcement learning. So I think both of those bits are here to stay. I think the way they are combined may and probably will evolve in the future. Nothing should be treated as dogmatic and fixed. We need to keep figuring out how to train better models, and this is what we are trying to do. The interesting thing, and I can credit Ilya for how much foresight he had, was that when I started at OpenAI in early 2019, I remember there was a research all-hands or something where Ilya came on stage and talked about what OpenAI's research program was. What he said at the beginning of 2019 was to train a large generative model on all data we can and then do reinforcement learning on it. That was the OpenAI research plan at the beginning of 2019, and this is exactly what we are doing today. Now the algorithms change, architectures change. I don't think he was even thinking about the transformer at that moment. GPT was like a toy example that someone was playing with, but the goal of training a large generative model on all the data in the world and then doing reinforcement learning with it was already in the core DNA of OpenAI, and that's what is happening right now.

那么，如果你愿意的话，我们来一点强化学习入门，让这对更广泛的听众来说真的很有趣。所以用非常简单的术语，像对我这个 10 岁孩子一样解释。什么是强化学习？

So let's do, if you will, a little bit of reinforcement learning 101 to make this really interesting to a broader group of people listening to this. So in very simple terms, explain it to me like I'm 10. What is reinforcement learning?

是的，是的。我通常用训练狗来比喻强化学习。

Yeah, yeah. I usually use the metaphor and analogy of reinforcement learning as training a dog.

这非常接近。我十几岁时养过一只狗。我记得我父母做的事，我对养狗一无所知。他们通过朋友的朋友请来了一位消防员，他好像是训练服务犬的。他告诉我一些训练狗的方法。大多数有雄心的狗主人都知道，口袋里随时放一袋零食非常重要。这是你一直要做的事。每当你看到狗表现好，你就微笑并给它零食。每当你看到狗做坏事，你就转移注意力，转身离开，变得难过。经过多年的繁殖，狗会发现这是不好的奖励和不好的行为。这和用模型做 RL 完全一样。我们激发模型的各种行为，把它们放在有挑战性的情境中，如果它们做了我们想要的事，就奖励它们；如果做了我们不想要的事，就给予某种惩罚或负面奖励。做好 RL 的关键是平衡这些。所以如果你一半时间给奖励，另一半时间给惩罚，但这几乎是数学层面的问题。最重要的部分是：激发行为，奖励好的，然后模型会更可能做你想要的，更少做你不想要的，通过这种方式它得到改进。这就是训练模型激发实际行为的方法，而不仅仅是下一个词预测。如果你预训练模型，你实际上是在训练模型预测下一个词。RL 是一个完全不同的梯度，或者说一套完全不同的目标。让模型按照你的意愿行事，涉及词汇和语义，你有时会听到“策略”这个词。在 RL 中，你会听到智能体、环境、动作、奖励和策略这些术语。我认为很多术语是不言自明的，但策略就是策略，是模型的行为。

It's very close. I used to have a dog when I was a teenager. Even what I remember my parents did, I didn't know anything about raising a dog. They invited through a friend of a friend a fireman who I think was working with service dogs. He came to me and told me a little bit about how to train your dog. What most ambitious dog owners know is it's always extremely important to have a bag of treats in your pocket. That's what you always do. Whenever you see your dog behave well, you should smile and give your dog a treat. Whenever you see your dog do something bad, you basically give your attention away, turn away and become sad. Before the years of breeding, the dogs discover it's bad reward and bad behavior. This is exactly doing that but with models. We elicit a lot of different behaviors in models, put them in challenging situations, and then we give them a cookie if they do something we want, if they do a good thing, and give them some kind of punishment or negative reward if they do something we don't want and don't like. A good way to do RL is if you balance those things. So if you give cookies half of the time and punish the other half, but this is almost a mathematical aspect of it. That's the most important part: elicit behaviors, reward the good ones, and going forward the model will be more likely to do what you want and less likely to do what you don't want, and through that it improves. It is the way to train models to elicit actual behaviors that is not just next-token prediction. If you pre-train a model, you literally train the model to predict the next token. RL is like a completely different gradient or a completely different set of what we want to get out of the model. Getting the model to do what you want for some vocabulary and semantics, you hear sometimes the term policy. So in RL you hear terms like agent, environment, action, reward, and policy. I think a lot of those are self-explanatory, but policy is the strategy, the behavior of the model.

是的，策略就是模型的行为，因为模型权重代表了它在不同情况下会做什么。模型最终是一个数学对象，你可以定义它，策略是一个将观测映射到动作的数学函数。你看到什么，然后你根据看到的东西做什么。所以智能体是模型，动作是模型做什么，奖励是你如何判断好坏。环境，最近你经常听到为 RL 设计合适的环境。这是什么意思？环境在某种程度上是模型看到的一切，但 RL 环境与大多数其他类型（如监督学习或无监督学习）的不同之处在于，强化学习环境是交互式的。你希望它们随着模型的行为而演变。类似地，如果你想学弹吉他，你拿起吉他弹奏，然后听到声音，然后你可以通过吉他的实际反馈来学习。这样，环境就是世界对你的动作的反应，而驱动你动作的很多因素正是环境中发生的事情。这几乎是唯一能真正教会智能体学习对环境变化做出反应的方法，通过强化学习。

Yeah, policy is the behavior of the model as the model weights represent what it does when put in a different thing. The model in the end is a mathematical object and you can define it, and policy is a mathematical function that maps observations to actions. What you see and then what you do with what you see. So agent is a model, action is what the model does, reward is how you say whether that's good or bad. Environment, you hear a lot these days about designing the right environment for RL. What does that mean? Environment is in some way everything that the model sees, but the interesting thing about RL environments compared to most other types like supervised or unsupervised learning is that reinforcement learning environments are interactive. You want them to evolve as the model does things. Similarly, if you want to learn how to play guitar, you take a guitar and strum it, and what happens is you hear the sound, and then you can learn to play with actual feedback of what is happening with the guitar. In that way, the environment is how the world reacts to your actions, and a lot of what drives your actions is what is happening in your environment. That's kind of the only way to really teach agents to learn to react to changes in the environment, through reinforcement learning.

你能给我们一个鸟瞰式的视角，看看 RL 这些年的演变吗？现代 RL 与历史上的 RL 有何不同？

Can you give us a bird's-eye view of the evolution of RL over the years? How does modern RL differ from historical RL?

是的，有非常早期的 RL，它甚至没那么古老，但主要的板块移动是将神经网络与强化学习结合起来。强化学习早于神经网络，作为一种在数学定义的环境中优化行为的通用数学方法和研究方法。所以这被称为深度强化学习。是的，然后深度强化学习，基本上是 DeepMind 将神经网络与强化学习结合起来的发明，就是我跟你说的 DQN 时刻。从那时起，关于游戏强化学习的研究非常活跃。即使在我 2019 年开始的时候，强化学习在当时有点时髦，尽管不是很成功。我们能够解决很多游戏，但瓶颈是模型没有经过任何预训练。我们训练了很多玩游戏的行为，甚至得到了 AlphaGo 时刻，很多人非常兴奋，但它仍然是在没有模型对行为有真正智能理解的情况下学习行为。仍然有很多可以称为穴居人智能的东西，模型即使经过大量强化也不是真的很聪明。这方面有长期的研究，很多很酷的结果和对 RL 的理论理解来自那些日子，因为人们积极研究 RL，但在某种程度上，没有预训练的 RL 是一条死胡同。然后当我完成机器人方面的工作，开始教语言模型编程时，拥有预训练模型是一件大事。GPT 时代的 Scaling 和大规模，大量数据真正训练出好模型，使我们当时就能开始 RL。那是我几乎立即做的第一件事。每当 GPT-3 被训练出来，我就尝试在上面做 RL，但总是有瓶颈。系统有点笨重。

Yeah, there was super historical RL, it's not even that old, but the main tectonic shift was when combining neural networks with reinforcement learning. Reinforcement learning predates neural networks as a general mathematical method of optimizing behaviors in mathematically defined environments and as a method of study. So that is known as deep reinforcement learning. Yes, then deep reinforcement learning, basically DeepMind's invention of combining neural networks with reinforcement learning, the DQN moment I talked to you about. From there, there was a pretty active area of research on reinforcement learning on games. Even when I started in 2019, reinforcement learning was kind of fashionable at that moment, although not very successful. We were able to solve a lot of games, but the bottleneck was that the models were not pre-trained in any way. We were training a lot of behaviors playing games, we even got the AlphaGo moment out of that, which a lot of people got very excited about, but it was still learning behaviors without models that were meaningfully smart about those behaviors. There was still a lot of what you might call caveman intelligence, models not being really smart even though being pretty heavily reinforced. There was a long research in that, a lot of cool results and theoretical understanding of RL comes from those days because people were researching RL actively, but it was in some way a dead end doing RL without pre-training. Then when I finished working on robotics and started working on teaching language models to code, having pre-trained models was a really big deal. The GPT era of scaling and large scale, just lots of data to really train great models, enabled us already at that moment to start RL. That was one of the first things I did almost immediately. Whenever GPT-3 was trained, I tried to do RL on it, and there were always bottlenecks. The systems were kind of clunky.

当时很难弄清楚哪些是正确的算法，哪些是正确的研究问题，以及用什么算法来训练模型。研究有点像从游戏里照搬很多东西，机器人领域也差不多。我在大语言模型上做的第一个强化学习基本上就是那个到处用的 PPO。它确实给出了一些结果，但那些早期结果在强化学习领域并不算惊艳。我们持续投入了很长时间。我个人一直相信强化学习和语言模型会有一个非常重要的时刻，但早期的试错并不太成功。我们训练 GPT-4 的时候，现在大家都觉得它是个很棒的模型，但当时内部其实挺失望的。有很多次我们想，'哦，我们花了这么多钱训练这个模型，它还挺笨的。'至少我们已经有了 GPT-3，它能做那些事，而 GPT-4 看起来并没有好多少。我们有个疑问：它在那些只有一个 token 的评估上看起来很聪明，能对复杂问题给出很详细的答案，但如果你让它说长一点，它就不太连贯了，或者给出很长的回答。我们需要回答这个问题：如何让一个权重里似乎有些智能的语言模型真正听起来聪明，并且善于对话？就在那时，一个几年前开发的技术真正大放异彩，那就是 RLHF。基本上就是在大型语言模型上做 PPO，奖励来自人类对两段文本的偏好，比如点赞和点踩。这是一个非常好的奖励，因为模型有很多方式生成糟糕的文本，而早期的 GPT-4 在很多方面都在生成糟糕的文本。RLHF 能够捕捉到这些问题并纠正它们。它强化了好的行为，强化了生成好的文本，然后惩罚了坏的文本。最终，GPT-4 加上 RLHF 这个组合给世界带来了 ChatGPT 时刻。虽然这是预训练的巨大成功，但实际上也是强化学习以 RLHF 形式取得的巨大成功。

It was hard to figure out what are the right algorithms, what are the right problems to work on, and what is the right algorithm to train it on. At that moment, research goes kind of like cargo culting a lot of things that were used for games, and almost the same things are for robotics. The first RL I was doing on large language models was kind of the same PPO we used for everything. It gave some results, but those early results weren't completely mind-blowing in RL. There was a long time where we kept investing in it. Personally, I always believed there would be a really big moment for RL and language models, but the early trials and errors weren't super successful. The moment when we trained GPT-4, everyone today thinks it's such a great model. But when we trained GPT-4, we were pretty underwhelmed internally. There were a lot of moments like, 'Oh, we trained this model, we spent a lot of money on it, and it's kind of pretty dumb.' At least, we already have GPT-3 that does all that stuff, and GPT-4 doesn't really seem to be that much better. We had this question: it seems smart on evals that were one token long. It seemed to be able to give a pretty detailed answer to a complex question where it was one token, but if you actually let it speak for longer, it wasn't very coherent or gave very long answers. We needed to answer this question: how do we actually make a language model that seems to have some smartness in its weights actually sound smart and be good in talking to it? That was the moment where a technique developed a few years earlier really shone, which was called RLHF. Basically, doing PPO on large language models with the reward given from human preferences of seeing two parts of the text, thumbs up and thumbs down. That's a very good reward because there are a lot of ways the model can generate bad text, and early GPT-4 was generating bad text in a lot of ways. RLHF was able to catch those things and correct them. It reinforced good behaviors, reinforced generating good text, and then punished bad text. In the end, GPT-4 plus RLHF together as a package delivered the ChatGPT moment to the world. As much as it is a big success of pre-training, it actually also was a pretty big success of RL in the form of RLHF.

太棒了。再深入一点，RLHF——作为用户我们都熟悉界面上的点赞和点踩，但实际的 RLHF 发生在后训练阶段。对吗？那项工作是什么样的？是不是有一群人坐在模型前面，可能是行业专家，给它反馈？实际是怎么操作的？

Amazing. And just to double-click on that, the RLHF — we all familiar as users with thumbs up and thumbs down on the interface, but the actual RLHF happened post-training. Is that right? And what did that look like as an effort? Did you have a bunch of humans sitting down in front of the model, industry specialists maybe, giving it feedback? How did that actually work?

RLHF 是一个已经在幕后进行了一段时间的研究项目。我记得至少对 GPT-2 我们就做了不少 RLHF，它已经存在了。为 RLHF 收集数据本身就是个研究领域，总是要弄清楚什么是训练模型的正确数据，什么是训练奖励的正确数据，以及如何塑造奖励。这是一个非常开放且深度很大的研究。有很多论文写过 RLHF 是什么，但它有很多深度。长话短说，你现在有我们称之为 AI 训练师的人。他们查看模型的输出并给出分数，然后你学习一个这些分数的模型，并用它来训练。

RLHF was a research program that was already happening in the background for a while. I think we did RLHF at least for GPT-2 quite a bit, and that was already there. Gathering data for RLHF is its own research domain, always figuring out what is the right data to train the model, what is the right data to train your rewards, and how to shape your rewards. It's a very open-ended and deep research in many ways. There are papers written on what RLHF is, but there is a lot of depth to it. Long story short, you have what we call AI trainers these days. They look at outputs of the models and give them scores, and then you learn a model of those scores and use that for training.

这就是整个数据标注行业的一部分——Scale AI 和其他公司——他们做的就是这些，对吧？

And that's part of the entire data labeling industry — Scale AI and a bunch of others — that's what they do, right?

是的，是的。在某种程度上，我认为随着模型越来越智能，这越来越成为过去式。这正在减少，但几年前，尤其是 GPT-4 时代，这是主流。数据标注行业有趣的一点是它必须不断自我革新，因为 AI 越来越聪明。在某个时候，如果 AI 已经能做了，你就不想再用人类来标注某些东西了。所以你要移动前沿，改变你标注的数据类型，因为你已经对前一部分做了 RLHF。

Yes, yes. In a way, I think it's getting more and more to be a thing of the past as the models get smarter. This is becoming less of a thing, but a few years back, especially in GPT-4 days, this was the thing. The interesting bit about the data labeling industry is that it has to constantly reinvent itself because the AIs are getting smarter. At some point, certain things you don't want to label with humans if AI can already do it. So you move the frontier and change the type of data you are labeling as you already RLHF the previous part.

我们一直在讨论强化学习，但这一切的第一阶段是模型的创建，即预训练，这是无监督学习，对吧？你想定义一下无监督与有监督，以及预训练在哪些方面是无监督的，还是自监督的，或者其他细微差别？

We've been talking about RL, but the first phase of all of this is the creation, the pre-training of the models, which is unsupervised learning, right? Do you want to define unsupervised versus supervised and in what way was the pre-training unsupervised versus self-supervised or whatever nuance?

我认为这些都是细微差别，并不像有些人喜欢定义的那样截然分明。预训练被称为无监督，是因为在某种定义下，你不需要给输入模型的数据添加任何额外的标签。你只是把文本原样输入。从某种意义上说，你可以认为数据已经有了标签，因为它是自标签的。如果你让模型根据文本预测下一部分文本，在某种程度上这就是一个标签，但它是自监督的，因为我们没有明确告诉模型什么是对的、什么是错的，或者我们想要什么、不想要什么。我们只是希望它预测数据的另一部分。你可以对图像做同样的事情。你可以遮住图像的一部分，让模型预测图像的下一部分。但这里有经典的机器学习概念，比如目标和标签，就像我们讨论分类器时那样。有监督学习是你对目标有一些概念，知道你的目标是什么，以及一些标签的概念。

I think those are nuances and they are not as stark and sharp as some people like to determine them. The way pre-training is called unsupervised is because in some definition of it, you don't need any extra labels to the data that you feed into the model. You just feed the text as is. In some way, you may argue that the data already has labels because it is self-labeled. If you give the model from the text predict the next part of text, in some way it is a label, but it is self-supervised because we don't clearly tell the model what is right or wrong, or what we want from it or what we don't want. We want it to just predict the other part of data. You can do the same thing with images. You can mask part of the image and tell the model to predict the next bit of image. But there is this classic machine learning notion of targets and labels, like we were talking about classifiers. Supervised learning is you have some notion of targets what your targets are and some notion of labels.

监督学习就像是从目标中预测标签，这是一种映射。但实际上有趣的是，目标中的信息比特通常远多于标签，研究目标本身的结构比学习映射本身能带来更多的学习和智能。所以把所有算力都花在无标签的数据学习上是正确的做法。这通常被称为表示学习，即研究数据及其属性。

Supervised learning was like predict those labels from targets and this is like a some type of mapping but actually what's interesting is that there are many more bits usually in the targets than in the labels and studying the structure of targets itself yields much more learning and much more intelligence than learning the mapping itself. So spending all compute on just learning the data itself without the labels is the right thing to do. And what is often called representation learning and studying the data and its properties.

好的，很好。那么回到强化学习，你前几天发推说 GRPO 的发布在很大程度上加速了大多数美国研究实验室的研究学习计划。那么什么是 GRPO？

Okay, great. All right, so going back to RL, you tweeted the other day GRPO the GRPO release has been in a large way has accelerated the research learning program of most US research labs. So, what is GRPO?

是的，这有点半开玩笑。我在这里稍微推断一下具体发生了什么，因为我没有去过大多数美国研究实验室，但我对发生的事情和方式有一些心理模型。长话短说，GRPO 是 DeepSeek 的开源发布，所有长期在线关注 AI 讨论的人都知道那个 DeepSeek 时刻。就是那家似乎在做非常出色工作的中国公司发布了新模型，也是一个预训练模型，一个推理模型。他们开源了算法，开源了他们做的很多事情。总的来说是一个非常棒、技术上非常出色的发布。有很多讨论说他们预训练模型特别便宜。这是关于那个 DeepSeek 时刻讨论的一部分。但讨论的另一部分是他们也发布了推理过程。这距离我们的 O1 发布并不远。据我所知，我们的 O1 发布让很多美国实验室措手不及。据我所知，他们没有类似先进的强化学习研究项目。基本上没有。我认为世界上唯一一家公司，我意识到可能有很多我不知道的事情，但你跟人聊天时会听到一些传言。所以这是我的版本。如果你看 DeepSeek 的其他论文，那家公司在某些方面做的强化学习研究和我们做的非常相似。我必须澄清，OpenAI 做的并不完全是 GRPO。它在很多方面略有不同。但有些部分肯定是相似的。最重要的是，两者都是大规模策略梯度算法。DeepSeek 在稍微相邻的领域做研究。他们离得不远。当我们发布 O1 并告诉世界可以通过在语言模型上扩展强化学习获得很好的结果时，我认为对 DeepSeek 来说，意识到'好吧，我们离得到类似的好结果不远了'并不是一个很大的跳跃。他们做到了。他们训练了推理模型，发布了它，并告诉了世界怎么做。比我们发布 O1 晚不了多少。我认为对于很多还不知道或没有研究项目如何训练推理模型的美国研究实验室来说，他们看到'哦，有这家中国公司。他们发布了怎么做。这帮助我们更快地启动和训练推理模型，比我们自己找出所有那些细节要快得多。'

Yeah, it was a little bit of a tongue-in-cheek moment. I am extrapolating here a little bit what exactly happens because I haven't been in most US research labs, but I have some mental model of what happened and how. Long story short, GRPO was the open-source release from DeepSeek, and there was a like everyone who is terminally online follows AI discourse knows that DeepSeek moment. Whatever it was when the Chinese company that seems to be doing really great work released new model, and it was also a pre-trained model, a reasoning model. They open-sourced the algorithm, they open-sourced a lot of things they did. Overall really great and technically excellent release. And there was a lot of discourse about that they pre-trained their model particularly cheaply. And that was part of the discussion about that DeepSeek moment. But the other part of the discussion was that they kind of released their reasoning process. It was not very far after our O1 release. And as far as I know our O1 release mostly caught a lot of US labs by surprise. They didn't have similarly advanced RL research program to my knowledge. Basically, no one. And I think the only company in the world that they get I am as I am aware there's probably a lot of things I didn't know, but you talk to people sometimes you hear rumors. So, this is my version of the world. Is that if you look at the other papers of DeepSeek like that company was doing pretty similar in some ways RL research to what we are doing. And I have to clarify what OpenAI is doing is not exactly GRPO. It is slightly different in many different ways. But some parts are definitely similar. And what's most important those are both large-scale policy gradient algorithms. And DeepSeek was doing research in a slightly adjacent area. They were not very far. And when we released O1 and we told the world that you can get great results with scaling up reinforcement learning on language models, I think it was not very big hop for DeepSeek to realize, 'Okay, we are not very far from getting similarly good results.' And they did it. They trained their reasoning model, and they released it, and they told the world how. Pretty not very much later than we released O1. And I think for a lot of US research labs that didn't yet know or didn't have a research program how to train reasoning models, they looked, 'Oh, there's this Chinese company. They released how to do it. It helped us kickstart and train reasoning models much faster than we would have to otherwise if we would have to find all those bits ourselves.'

扩展强化学习需要什么？有一个阶段 OpenAI 非常专注于预训练，然后如果我理解正确的话，过去 12 到 18 个月左右，重点放在了计划的第二部分，即扩展强化学习。这仅仅是给强化学习更多算力、更多数据、更多标注的问题吗？需要什么？

What does it take to scale RL? So, there was a phase where OpenAI was very focused on pre-training, and then if I understand correctly the last 12-18 months or whatever time period where the emphasis has been on the sort of second part of the plan, which is scaling RL. Is it a question of just giving RL more compute, more data, more labeling as we were saying? What does it take?

嗯，首先需要知道和理解的是，强化学习很难。从概念上讲，如果你思考它，它仍然有很多深度。但从非常概念性的数学角度来说，预训练非常简单。它是你能做的最简单的事情，而且已经有很多思考和优化投入其中，几年来甚至在非常大规模上优化和做好非常简单的数学运算。就强化学习而言，它要复杂得多。一次强化学习运行中有更多的事情在发生。做这件事时可能出错的地方更多，尤其是当你扩展规模时，还有更多类型的瓶颈和失败。它是一件更微妙的事情，有更多的出错空间。再次，我不想深入那个类比，因为它有点夸张，但只是给一些巧合，你可以有一个钢厂生产钢铁，过程相对标准化，你制造钢块，它们均匀、漂亮且定义明确。而建造半导体，世界上很少有公司能做到，因为有太多可能出错的地方，你必须非常注意细节才能制造出好的半导体。它内部非常复杂。在很多方面，这有点像，我不想贬低，因为在大规模上做好预训练有很多非常困难的技术难点。但强化学习堆栈中有更多活动部件和更多元素需要正确，才能让大规模运行成功。

Well, the first thing that is important to know and understand, RL is hard. Conceptually, if you think about it and there's still a lot of depth to it. But very conceptually, mathematically speaking, pre-training is dead simple. It is the kind of the simplest thing you can do, and there has been a lot of thought and a lot of optimization already put through that for a few years of even optimizing and doing very well at very large scale very simple mathematical operation. In terms of RL, it is much more complex. There are many more things going on in a reinforcement learning run. There are many more things that can go wrong in doing it, especially as you scale up, and many more types of bottlenecks, failures. It's a much more delicate thing, and there is much more room for error. In some way again, I don't want to go too deeply in that parallel because it's a little bit overblown, but just to give some coincidence, you can have a steel factory which makes steel, and the process is relatively standardized, and you make blocks of steel, and they are uniform and nice and well-defined what it is. Versus building semiconductors, which there are very few companies in the world that can do it because there are so many things that can go wrong, and you have to put a lot of attention to details to make great semiconductor. It's very complex internally. And in many ways this is kind of like you know I don't want to diminish because there is a lot of very hard technical difficulty to do pre-training well at large scale. But there are just many more moving pieces and many more elements of the reinforcement learning stack that need to be gotten right to get a large-scale run successful.

你提到过像智能体式 AI 这样的 ChatGPT 智能体。这一切如何融合？像智能体自主性与推理强化学习这样的工具。帮助我们理清什么做什么，什么影响什么？

You mentioned working on a so chat GPT agent like the agentic AI. Like what where does it all fit? The tool used like the agentic autonomy versus reasoning RL. Help us to sort of reconcile what does what and what impacts what?

我认为重要的是，我相信 AI 可以通过自动化、解决问题以及 AI 为我们做我们想要的好事，对我们的世界和生活产生很多积极影响。

Much I think what is the important thing is I believe and I think that there can be a lot of positive impact of AI on our world and on our lives through automation, through problem-solving, and through AI doing good things for us, the things that we want.

很长一段时间以来——其实也不算太长，就是过去两年左右，也许快三年了——我们一直生活在这样一个世界里：我们向 AI 提问，它给出答案。一开始是瞬间回答，现在它能思考一两分钟。这感觉很长，但想想人类解决一个问题需要多久，两分钟又能做什么？AI 在它能解决的问题上可能更快一些，但它能做的仍然有限。还有很多任务需要 AI 花更长的时间。最后，当我提示 Codex 时，它会工作一段时间，又是几分钟。我们内部有很多东西，我们正在做的是让模型能工作更长时间。我们还没有找到合适的产品来部署它们。但如今，模型在某些类型的任务和问题上可以思考 30 分钟、1 小时、2 小时，甚至更长。它们通常有能力做到这一点，我们需要弄清楚如何让这个过程更有用，能够真正解决现实生活中的各种问题，无论是编码、预订旅行、制定计划，还是设计房屋或新的电子设备，或者任何你希望模型为我们做的事情。这很大程度上来自于模型独立思考更长时间，考虑更多的替代方案、赌注，有时只是处理一长串任务。所以，智能体部分是由基础推理驱动的。

And for a long time, and for a long time again, it's not that long, but the last 2 years or so, or maybe approaching three, we've been living in this world where we kind of ask questions to AI, and it gives us an answer. At the beginning instantly, now it can think for like a minute or two. Which feels long, but in many ways, what can you do for 2 minutes if you think of how many problems humans solve? And AI is probably a little bit faster in the things it can solve. But it's still a limit of what it can do. There are still a lot of tasks that would take AI much longer. Finally, when I prompt Codex, it works for a while. Again, a few minutes. There are a lot of things we have internally and we are doing that allow the model to work for much longer. We still haven't figured out the right product to deploy them. But the models can think for like 30 minutes, an hour, 2 hours these days on certain types of tasks and problems, even longer than that. And they are generally capable of doing so, and we need to figure out how to make that process more useful and able to actually come to various problems in real life, whatever it is: coding, booking travel, making plans, or even designing houses or new electronic devices, or whatever else you would like models to do. We would like them eventually to be able to do for us. And a lot of this comes through the models thinking independently for longer periods of time and considering more alternatives, bets, and sometimes just going for a slog of very long lists of tasks. So, the agentic part is powered by fundamental reasoning.

是否存在在线强化学习的概念，即智能体在做事时从现实世界中学习，强化学习实时发生？

Is there a concept of online RL that happens where as the agent does something and learns from the real world, the RL happens in real time?

总的来说，所有强化学习——你听到的大多数关于语言模型的强化学习——都是在线进行的，但它的在线方式仍然是一次训练运行。它仍然是独立于用户进行训练的。世界上有一些模型，我最近了解到 Cursor 正试图在用户参与的情况下在线训练一些模型。理论上，可以在 ChatGPT 或任何其他产品中训练模型，只需响应用户并通过获得的任何奖励进行强化。但据我所知，至少 OpenAI 目前没有这样做。这可以很好，但也可能很危险，因为你并没有真正控制在这个循环中强化了什么，以及可能发生什么。所以，至少在我们有很好的保障措施之前，我认为我们不应该在像 ChatGPT 这样复杂且大规模的系统中尝试这样做。

So, in general, all of RL is happening like most of RL that you hear talk to language models is online, but it's done online in a way that is still a training run. It's still being trained kind of separately from the user. There have been a few models in the world, and I've learned recently that I think Cursor is trying to train some models online with their users in the loop. And it's theoretically possible to train models like in ChatGPT or every other product just responding to the users and reinforce through whatever rewards you get in there. But this is not what I am aware of, at least not what OpenAI is doing at the moment, and it can be great, but it can also be dangerous because you are not really controlling what you are reinforcing in that loop and what could happen. So, I at least until we have really good safeguards, I don't think we should try to do that in anything as complex and large scale as ChatGPT.

是的，很有趣。说到这个，我们谈谈对齐。对齐是强化学习的事情吗？我的意思是，你是通过判断模型的对错来创造对齐的吗？

Yeah, interesting. And very much on that note, talking about alignment for a minute, is alignment an RL thing? I mean, do you create alignment in the model by judging it to what is right and wrong?

有点是，也有点不是。在某种程度上，对齐是关于引导模型走向某些行为，这绝对是强化学习的事情和问题。但你也希望模型知道什么是对什么是错，并理解世界。推理和强化学习问题通常就是 AI 问题。为了对齐，模型需要知道对错才能选择正确的。我不认为你可以直接告诉模型，给它看一些好的行为，它就会照做。模型需要深刻理解其行动和后果，才能真正选择正确的事情。我认为这是一个永无止境的追求，因为即使对人类来说，定义什么是对齐也不容易。我认为随着我们文明的发展，对齐的概念和人类的目标会不断演变，我们需要不断引导模型走向这些目标，并不断向它解释我们想要什么。但这是任何 AI 研究项目中非常重要和核心的部分。

Kind of, it's a little bit of yes and a little bit of no. In a way, alignment is about steering the model to certain behaviors, and that is definitely an RL thing and an RL problem. But also you want the models to know what is right and what is wrong and understand the world. Reasoning and RL problems are very often just AI problems. In the way to be aligned, the model needs to know right or wrong to choose right. I don't think you can just tell the model, show it a few good things to do, and it will do them. The model needs to deeply understand its actions and consequences to really be able to choose the right thing. I think it's a never-ending pursuit because even for humans it's not super easy to define what we consider aligned. I think as our civilization evolves, the notion of alignment and the goals of humanity will keep evolving, and we will need to keep nudging the model towards those things and keep explaining to it the things that we want from it. But it's a very important and central part of any AI research program.

是的，这引出了一系列问题：强化学习在哪些方面高效，哪些方面不那么高效。看起来它在数学和编码方面特别有效，那么下一个明显的问题是，世界的其他部分呢？但先快速深入一下数学。就在九月，几周前，你们在 ICPC 世界总决赛上做了些不可思议的事情。你想谈谈那是什么，以及从模型技术角度来看幕后发生了什么吗？

Yeah, which brings a whole next series of questions of where RL is efficient versus less. So it seems that it's been particularly good for math and coding, and then the next obvious question is what about the rest of the world? But taking a quick sort of going down the rabbit hole a little bit about math. So just in September, just a few weeks ago, you guys did something unbelievable with the ICPC World Finals. You want to talk about what that was and what went on from a model technical perspective behind the scenes?

从我们的角度来看，从模型的角度来看，发生的事情出奇地少。我们只是有一个相当聪明的模型，当我们要求它解决编程问题时，它就能正确解答。一点背景是，我认为我们有一段时间把编程谜题作为我们想法的很好的研究测试平台。这些是很好的实验问题，它们从未被视为产品的一部分，但它们是相当复杂的问题，需要大量的思考，并且非常适合给予奖励。所以我们所有的研究人员都喜欢在这些问题上工作，作为尝试他们强化学习想法的一种方式。你总是需要一个数据集；我会拿一个编程谜题数据集来尝试。我认为正因为如此，我们的模型在竞争性编程方面一直非常擅长，这算是一种副产品。

There happened surprisingly little from our perspective, from the model perspective. We just have a pretty smart model, and then when we ask them to solve programming problems, they are correct. What's a little bit of a backstory is that I think we used programming puzzles for a while as a very nice research test bed for our ideas. Those are nice problems to experiment on, and they weren't ever considered part of the product, but they are pretty complex problems and require a whole bunch of thinking, and are very nice to give rewards to. So all of our researchers just liked working on those problems as a way of trying out their RL ideas. You always need a dataset; I'll take a dataset of programming puzzles and try it. And I think because of that, our models were always very good at competitive programming as a kind of byproduct.

我们从未刻意追求擅长这类问题，但研究人员一直在上面尝试他们的想法，因此每次训练运行最终都非常擅长这类谜题。对我们来说，提交到竞赛更像是一种形式，向世界展示这些模型的能力水平。但重要的是要承认，并非在所有领域，这种与人类基线的比较都能像编程竞赛问题那样顺利和出色，因为那些问题已经被许多研究人员长期尝试，而研究人员并不总是花足够多的时间在人们使用 ChatGPT 或模型解决的实际问题上。

We never tried to be good at it, but researchers were trying their ideas on it, and because of it, every training run ended up being very good at those types of puzzles. It was a bit of a formality for us to submit to a competition and demonstrate to the world the level of capability in those models. But it is important to acknowledge that not in all domains this type of comparison to human baseline is as nice and as good as in programming competition problems, because those were tried for a long time by many researchers, and researchers don't always spend as much time as they could on very practical problems that people go to ChatGPT or models with.

很好。所以这几乎是开箱即用的，没有针对它进行专门训练。提醒一下，我指的是 2025 年 9 月在阿塞拜疆巴库举行的 ICPC 世界总决赛，即国际大学生程序设计竞赛，OpenAI 在 5 小时内解决了 12 个复杂的算法问题，实际上在人类队伍中取得了第一名。提供一下背景。

Great. So it sort of came out of the box. There was no specific training for it. And just to remind people, what I'm referring to is the ICPC World Finals in September 2025, the International Collegiate Programming Contest in Baku, Azerbaijan, where OpenAI solved 12 complex algorithmic problems within the 5-hour time limit, effectively taking first place in front of human teams. So just for context.

我们进行了一次小型竞赛巡礼。我们参加了 ICPC，今年早些时候还参加了 IOI（国际信息学奥林匹克），以及一场 AtCoder 启发式竞赛，我们获得了第二名，仅次于一位人类，他也是波兰人，曾受雇于 OpenAI。有趣的巧合。但我们一直在寻找一个时机，让我们的模型足够聪明，能够与非常聪明和有才华的人类竞争。这从来不是我们特定的目标或重点。我们认为，如果我们为训练智能模型做了好的研究，它们就应该足够聪明去做这些事情。我们把这个里程碑当作一个节点，继续前进。我希望我们会看到，而且我认为我们已经看到，越来越多实际和具体的东西出现。每周或每隔一周，在 Twitter 上，我都会看到可信的报告，说真正的科学家正在使用我们的推理模型来帮助进行计算和解决困难的技术问题。这就是我们想要达到的。解决竞赛很酷，但人们解决竞赛是为了证明他们能够从事前沿工作并解决新的技术问题。这也是我们希望我们的模型做到的。

We did a little tour of various competitions. We did ICPC, also the IOI (International Olympiad in Informatics) earlier this year, and an AtCoder heuristics competition where we came second, behind a single human who is also Polish and used to be employed by OpenAI. Funny coincidence. But we were looking for a moment where our models are smart enough to compete with incredibly smart and talented humans. It was never our particular goal and focus. We think if we are doing good research for training smart models, they should be smart enough to do those things. We take that milestone and keep moving forward. I hope we will see, and I think we are already seeing, more practical and tangible things coming out. Every week or every other week on Twitter, I see credible reports of actual scientists using our reasoning models to help perform calculations and solve hard technical problems. That's where we want to be. Solving competitions is cool, but people solve competitions to prove they can work at the frontier level and solve new technical problems. That's what we want from our models as well.

我们研究了第二次尝试。所以从心理上讲，至少对于像我这样的人来说，我理解强化学习如何非常有效地用于针对数学问题或编程问题进行训练。我认为现在一个大问题是，如何将其应用于世界的其他部分，在那些答案不是对或错、可能更模糊的领域和学科，以及经济的其他部分。你们组织前几天提出了一个 GDP 值，这是一种评估不同行业表现的方法。对于强化学习作为世界其他部分成功之路的泛化，你有什么想法？

We looked into this second go. So mentally, at least for somebody like me, I understand how RL could be used very effectively to train against math problems or coding problems. I think one of the big questions right now is how do you do that for the rest of the world, in contexts and disciplines where the answer is not right or wrong, maybe a little more murky, and the rest of the economy. You guys as an organization came up with a GDP value the other day, which is a way of evaluating performance against different industries. What is your thinking in terms of generalization of RL as a path to success for the rest of the world?

简短的回答是，人类能够学习所有这些事情，只要有任何方式可以评估表现并判断某件事是对是错，并且你可以计算那个反馈——比如你需要能够计算出某件事做得有多好——那么你就可以优化它，并对其应用强化学习。我认为可以争论的是，如果没有对错的概念，那么人类也无法改进和学习，因为需要有一个来自某处的学习信号。这主要是一个获取反馈的便利性和容易程度的问题。每个做强化学习的人都应该努力在越来越复杂和有趣的训练信号上进行训练。经常会出现奖励破解的概念。在进行强化学习时，这种情况经常发生，这是一个重要的问题。你以某种方式塑造奖励来奖励某些行为，但有时你奖励的并不是你真正想要的。有一件事：你需要训练模型去做你奖励的行为，但你给模型的奖励和你真正想要的之间自然存在不匹配。有时模型会做你奖励的事情，但不符合你想要的意图，我们需要修复它。这几乎像是一种育儿天赋。从某种意义上说，你可以说这是强化学习的局限性，但当我思考时，我意识到这在人类系统中也经常发生。有很多激励系统和奖励系统，甚至在工作场所和所有类型的人类群体中，人类的奖励并不总是为系统的最终目标而优化，他们不断地以各种方式破解奖励。在设定正确的奖励和观察系统是否执行之间，存在一场持续的打地鼠游戏。这在任何政策制定和激励计划中都是一个巨大的问题。在强化学习研究中，同样存在这种打地鼠游戏，试图确保你的奖励越来越好地代表你真正关心模型要做什么。

The short and quick answer is that somehow humans can learn all those things, and as long as there is any way to evaluate performance and figure out if something is going right or wrong, and you can compute that feedback—like you need to be able to calculate how well something did—then you can optimize it and do reinforcement learning with it. I think there can be an argument that if there is no notion of what is right or wrong, then humans are also not able to improve and learn, because there needs to be a learning signal coming from somewhere. It's mostly a question of how convenient and easy it is to get that feedback. Everyone doing reinforcement learning should strive to train on more and more complex and interesting training signals. Very often there is the notion of reward hacking. What happens when doing reinforcement learning is that it happens a lot, and it's an important problem. You shape your reward in some way to reward certain behaviors, but sometimes it is the case that what you reward is not what you actually want. There is one thing: you need to train the model to do the behaviors you reward, but there is also a natural mismatch between the reward you give the model and what you actually want. There are moments where the model does what you reward, but it's not in the spirit of what you wanted, and we need to fix it. It's almost like a parenting talent. In some way, you can say it's a limitation of reinforcement learning, but when I thought about it, I realized a lot of that happens in human systems as well. There are a lot of incentive systems and reward systems, even in workplaces and all kinds of human groups, where humans have rewards that are not always optimized for the ultimate goals of the system, and they hack rewards constantly in many different ways. There is a constant whack-a-mole game between setting the right rewards and seeing if the system does it. It's a huge issue in any policy making and any incentives programs. This is the same kind of whack-a-mole game in reinforcement learning research, trying to make sure your rewards are better and better representing what you actually care about the model to be doing.

好的。那么也许放大视野来结束这次对话，你前几天发推文说：“我们都集体认为 AGI 昨天就应该被造出来。而它还没有被造出来的事实，主要是因为一个需要被修复的简单错误。” 这条推文非常棒。

All right. So maybe to zoom out to close this conversation, you said the other day you tweeted: "We all collectively believe AGI should have been built yesterday. And the fact that it hasn't yet is mostly because of a simple mistake that needs to be fixed." Which is super awesome as a tweet.

你认为预训练和规模化强化学习的结合会让我们达到 AGI 吗？关于什么不是预训练和强化学习，以及极限在哪里，总有一个有趣的问题。

Do you think that the combination of pre-training and scaled RL takes us to AGI? There is always an interesting question of what we consider something that is not pre-training and RL, and where is the limit?

我大体上认为，我们今天所做的预训练是必要的，强化学习也是必要的。肯定还会有更多的东西，我们在其中一些方面有很多非常雄心勃勃的研究项目。我不认为研究空间的距离问题很难说；对一些人来说，我们想要做的和计划构建的东西离那些并不远。对另一些人来说，他们可能会说完全不同。但我们正在并且想要不断改变训练模型的方式，代表我们认为正确的智能形式和最有用的学习形式。与 AGI 的距离也是一个非常复杂的问题。我真的很喜欢有人对我说过的话：如果你和 10 年前的人交谈，给他们看今天的 ChatGPT，他们可能会称之为 AGI。但我们今天还没有达到，因为它仍然有很多局限性，我们都很清楚这些局限性。我们很有信心能解决这些局限性，但未来的模型可能还会有进一步的局限性需要修复。有一个终极问题：模型何时能在没有太多外部输入、没有人类干预和修复的情况下自我改进？这是一个非常难的问题，一个人类需要努力回答的严肃问题。模型仍然会在很大程度上依赖于我们的基础设施和系统，但它们将能够开始自我修复，而无需我们修复。到那时，对 AI 能够做什么和解决什么的预测会比我们现在能做的更清晰一些。

I generally think that what we are doing today, pre-training is necessary, and RL is necessary. There will surely be a few more things, and we have a lot of very ambitious research programs on some of those things. I don't think the question of distance in research space is hard to say; for some people, what we want to do and what we are planning to build is not very far from those things. For someone else, they might say it's completely different. But we are and want to be constantly changing the way we train models, representing what we think is the right form of intelligence and the most useful form of learning. The distance from AGI is also a very complex question. I really like something someone said to me: if you talk to someone from 10 years ago and show them ChatGPT from today, they would probably call it AGI. But we are not there today because it still has a lot of limitations, and we are all very aware of those limitations. We are pretty sure we can resolve those limitations, but there will probably be further limitations of future models that need to be fixed. There is an ultimate question: when is the moment that the model can improve itself without much external input and without humans working on it and fixing it? That is a very hard question, a serious question that humanity needs to try to answer. Models will still largely depend on our infrastructure and systems, but they will be able to start fixing themselves without us having to fix them. At that point, predictions of what AI will be able to do and solve become a little more clear than what we can do right now.

从哲学上讲，你可能听过 Richard Sutton 在 Dwarkesh 播客上的发言，他实际上说通往 AGI 的唯一路径是纯强化学习，而 LLM 是一个有缺陷的前提，因为它们是对现实的模仿，而强化学习是对现实的强制。你对这个问题有什么看法吗？

Philosophically, you may have heard Richard Sutton on the Dwarkesh podcast, effectively saying that the only path to AGI was going to be pure RL, and that LLMs were a flawed premise because they were imitation of reality, whereas RL was enforcement of reality. Do you have any thoughts on that question?

我还没有机会完整听完那一集，所以不了解所有细节。但我可以说，我们现在正在语言模型上做相当严肃的强化学习。就纯强化学习而言，我认为它没有意义。强化学习需要预训练才能成功，预训练也需要强化学习才能成功。没有强化学习，我们正在进行的研究项目就没有意义。OpenAI，而且我确信所有其他 AI 实验室，都非常认真地在我们的模型上做大量的强化学习。很多人说 LLM 是通往 AGI 的入口还是出口，他们通常指的是预训练。但同样清楚的是，当前的做法还不够，也不是全部。还需要对设置进行进一步的改变。有时人们说，“如果你在做强化学习，那就不是 LLM，而是别的东西。”有时他们说，“如果你能在你的展开中写程序，那就是思维链，不仅仅是神经网络，而是神经符号系统。”有些人很容易把某样东西视为 LLM，而把另一样东西视为不是。就我个人而言，我的观点是，我们拥有的东西是下一步的很好基础。我们首先训练 Transformer 用于翻译，然后在大规模数据上预训练它们，然后对它们进行 RLHF。现在我们正在做大规模强化学习。我们还会做更多更复杂的事情。在某个时候，架构可能会开始或多或少地发生改变。我个人认为我们走在正确的道路上，这感觉不像是完全转向，更像是不断添加更多东西，也许还会溶解一些旧元素，这些元素曾把我们带到那个特定的智能水平，但不再需要了。

I haven't had a chance to fully listen to that episode yet, so I don't have all the details. But what I can say is that we are doing quite serious RL on language models these days. In terms of pure RL, I don't think it makes sense. RL needs pre-training to be successful, and pre-training needs RL to be successful as well. Without RL, the research program we are doing wouldn't make sense. OpenAI, and I'm pretty sure all other AI labs, are very serious about doing a lot of reinforcement learning on our models. A lot of people say that whether LLMs are an on-ramp or off-ramp to AGI, they often mean pre-training. But it is also clear that the current way of doing things is not yet enough and not yet everything. There will need to be further changes to the setup. Sometimes people say, "If you are doing RL, it's not LLM, it's something else." Sometimes they say, "If you can write a program in your rollout, then it's chain of thought, not a neural network only, it's a neural symbolic system." It's easy for some people to consider something an LLM and the other thing not. Personally, my view is that what we have is a pretty good foundation for the next step. We had transformers first trained for translation, then we pre-trained them on large-scale data, then we did RLHF on them. Now we are doing large-scale reinforcement learning. We'll do a few more complex things. There is a chance somewhere along the line the architecture will start changing more or less significantly. I personally think we are on the right path, and it will feel less like completely turning around and more like keep adding more things and maybe dissolving some old elements that carried us to that particular level of intelligence and were not needed anymore.

这感觉是一个很好的结束点。你非常慷慨地分享了你的时间和想法，让我们得以一窥 OpenAI、你的工作、幕后情况，以及预训练和规模化强化学习的关键方面。这是一次精彩的对话。Jerry，非常感谢你。真的很感激。

That feels like a wonderful place to leave it. You've been very generous with your time and thoughts, giving us a glimpse into OpenAI, what you work on, what it looks like behind the scenes, and the key aspects of pre-training and scaling reinforcement learning. It's been a wonderful conversation. Jerry, thank you so much. Really appreciate it.

非常感谢。我也非常享受在这里。

Thank you very much. I enjoyed being here a lot, too.

嗨，我是 Matt Turk。感谢收听本期 MAD 播客。如果你喜欢，我们将非常感激你考虑订阅（如果还没订阅的话），或者在你看或听这期节目的任何平台上留下好评或评论。这真的有助于我们建立播客并邀请到优秀的嘉宾。谢谢，下期节目再见。

Hi, it's Matt Turk again. Thanks for listening to this episode of the MAD podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already, or leaving a positive review or comment on whichever platform you're watching or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.