构建 ChatGPT：从预训练到 RLHF

今天的嘉宾是 John Schulman。John 是 OpenAI 的联合创始人之一，也是 ChatGPT 的首席架构师。在领导 ChatGPT（全球使用最广泛的大语言模型）的开发之前，John 是深度强化学习的早期先驱之一，他发明了广泛使用的近端策略优化算法（PPO），该算法实际上是 ChatGPT 训练的一部分。他还发明了信任区域策略优化（TRPO）。他是 OpenAI Gym、OpenAI Baselines、Stable Baselines 以及许多现代深度学习时代元学习算法的关键贡献者。在共同创立 OpenAI 之前，John 实际上是我在伯克利实验室的博士生，那段时光我至今仍非常珍惜。John，很高兴你能来，欢迎来到节目。

Our guest today is John Schulman. John is one of the co-founders of OpenAI and the lead architect of ChatGPT. Before leading the charge on ChatGPT, the world's most widely used large language model, John was one of the early pioneers of deep reinforcement learning, having invented the widely used Proximal Policy Optimization algorithm, also known as PPO, which is actually part of the ChatGPT training. He also invented Trust Region Policy Optimization, or TRPO. He was a key contributor to OpenAI Gym, OpenAI Baselines, Stable Baselines, and to many of the modern deep learning era meta-learning algorithms. Before co-founding OpenAI, John was actually a PhD student in my lab at Berkeley, which is a time I still very much treasure. John, so great to have you here. Welcome to the show.

谢谢你邀请我，Peter。很高兴我们能以这种形式叙旧。

Thanks for having me, Peter. So glad we get to catch up in this format here.

John，在我们深入对话之前，我想快速感谢我们的播客赞助商：Index Ventures 和 Weights & Biases。Index Ventures 是一家风险投资公司，投资于从种子轮到 IPO 各个阶段的杰出企业家，在旧金山、纽约和伦敦设有办事处。该公司支持涵盖 AI、SaaS、金融科技、安全、游戏和消费等多个垂直领域的创始人。就我个人而言，Index 是 Covariant 的投资者，我极力推荐他们。Weights & Biases 是一个 MLOps 平台，通过实验跟踪、超参数调优、数据集版本管理和模型管理，帮助您更快地训练更好的模型。OpenAI、Nvidia 以及几乎所有发布大型模型的实验室都在使用它。事实上，我在伯克利的学生和 Covariant 的同事中，许多人（如果不是全部）都是 Weights & Biases 的重度用户。

Now John, before diving into our conversation, I'd like to quickly thank our podcast sponsors: Index Ventures and Weights & Biases. Index Ventures is a venture capital firm that invests in exceptional entrepreneurs across all stages, from seed to IPO, with offices in San Francisco, New York, and London. The firm backs founders across a variety of verticals including AI, SaaS, fintech, security, gaming, and consumer. On a personal note, Index is an investor in Covariant, and I couldn't recommend them any higher. Weights & Biases is an MLOps platform that helps you train better models faster with experiment tracking, hyperparameter tuning, dataset versioning, and model management. They are used by OpenAI, Nvidia, and almost every lab releasing a large model. In fact, many if not all of my students at Berkeley and colleagues at Covariant are big users of Weights & Biases.

John，你身处近期历史上（很多人会说是史上）最大的 AI 发布——ChatGPT 发布——的中心，它让几乎每个人都在谈论 AI，实际上是与 AI 对话。为了确保我们都在同一页上，ChatGPT 是什么？

John, you were at the center of the biggest AI release in recent history, many would say ever: the release of ChatGPT, which made literally everyone talk about AI, in fact talk with AI. Just to make sure we're all level here, what is ChatGPT?

ChatGPT 是一个在网页应用上的聊天机器人；你可以访问 chat.openai.com。它是一个语言模型，所以是一个生成文本的模型。我认为它真正火爆的原因是，尽管之前也有语言模型，但这个模型易于使用，因为你就像和人说话一样跟它交谈，而且它的智能程度超过了一个阈值，使得它在很多任务上确实有用，比如如果某人对某个随机知识话题感到好奇，或者他们需要写作帮助。它对很多这类事情都有用。

ChatGPT is a chatbot that's on a web app; you can visit it at chat.openai.com. It's a language model, so it's a model that generates text. I think the reason it really took off is because even though there have been language models out there before, this one was easy to use because you just talked to it like you would talk to a person, and it was above some threshold of smartness that made it actually useful for a lot of tasks, like if someone is curious about some random knowledge topic or they want help writing. It's useful for a lot of that.

我经常玩它，John，而且总是印象深刻。前几天我让它……我需要给一家公司做演讲，我先让它用一段话描述这家公司是什么，它描述了。然后我让它，“你能再用 Snoop Dogg 的说唱风格描述一下这家公司吗？”它当场就做到了。而且可以肯定 Snoop Dogg 从未说过关于那家公司的唱词，它却完美地完成了。它重组事物的方式真是令人惊叹。这样的东西到底是怎么构建的？

I have to play with it quite a bit, John, and I'm always impressed. The other day I asked it to... I had to give a talk to a company, and I first asked it to describe in one paragraph what the company is, and it described it. Then I asked it, "Can you now describe the company again, but using Snoop Dogg rap style to describe it?" And it just did it right there. And for sure Snoop Dogg never did any raps about that company, and it just nailed it. It's really mind-blowing, the way it can recompose things. How is something like this even built?

是的，训练这个模型有几个步骤。首先，从预训练语言模型开始。其工作原理是，你拿模型，训练它模仿大量人类撰写的文本。所以我们基本上想让模型像人类一样使用语言。为此，你需要找到大量文本。实践中，我们在互联网上找到大量文本，训练模型生成类似的文本。实际上，它是在根据前面的词预测下一个词。你大量训练这个，就得到了预训练语言模型。但现在它只知道如何生成互联网上的随机文本，其中一些是你想要的行为，一些不是。所以我们需要微调它，使其行为更一致、更专业。这就是我们进行第二步微调的地方。具体来说，我们做 RLHF。在流程中，有几个步骤：首先我们做一些监督学习，用我们雇佣的人类撰写的高质量回答来训练它。然后我们实际做强化学习来进一步改进，训练一个奖励模型来识别好的回答，然后我们用那个奖励模型做强化学习。

Yeah, so there are several steps in the process of training this model. First of all, you start with the pre-trained language model. The way this works is you take the model and just train it to imitate a lot of human-written text. So we want to basically try to get the model to use language like humans. To do that, you need to find a lot of text. In practice, what we do is we find lots of text on the internet, and we train the model to produce text that looks like that. In practice, what's going on is it's predicting the next word given previous words. So you train on a lot of this, and you've got your pre-trained language model. But now all it knows how to do is generate random text from the internet, and some of this is the kind of behavior you want and some of it isn't. So then we need to fine-tune it to have more consistent behavior and professional behavior. So that's where we do this second fine-tuning step. In particular, we do RLHF. In the pipeline, we have a few steps where first we do a little supervised learning, where we train it on some very high-quality responses written by humans we've hired. Then we actually do RL to improve it further, where we train a reward model that can recognize good responses, and then we do RL with that reward model.

你用奖励模型做强化学习，意思是机器人最终生成文本，由奖励来评分，并试图最大化那个奖励。

You're doing RL with the reward model, meaning that the bot is eventually generating text and getting rated by the reward, and tries to maximize that reward.

对。

Right.

原则上，就像在 AlphaGo 中一样，有没有可能生成一个在生成文本方面严格优于任何人类的聊天机器人？

Is it possible that in principle, just like in AlphaGo, you could generate a chatbot that is strictly superior at generating text than any human has ever been?

哦，是的，当然。我会说模型在某些方面已经超人类，而在其他方面显然不是。当然，如果你把速度算作一种能力，它们在写诗等方面比你我都快得多，或者写 Snoop Dogg 风格的歌词。所以总的来说，模型……没有一个单一的尺度来衡量模型的智能，它们在某些方面是超人类的，比如广博的知识面，以及以非常精确的方式用所有这些不同风格写作并很好地遵循模式的能力。然后在很多其他方面，它们比人类差。

Oh yeah, definitely. I'd say the models are already superhuman in some ways and obviously not in others. Certainly if you count speed as a capability, they're a lot faster than you and me at writing poems and so forth, or writing Snoop Dogg-like lyrics. So I'd say overall, models are... there's not a single scale that measures the smartness of the model, and they're superhuman in some ways, like their vast breadth of knowledge and their ability to write in a very precise way in all these different styles and follow the patterns very well. And then they're worse than humans in a lot of other ways.

你认为它们在哪些方面还不太好？

In which ways do you think they're still not so good?

让我想想。当然，有很多任务比如数学推理，它们远不如熟练的人类。比如说，如果你想让它们做一个长路径的任务，即使你非常仔细地提示它们，告诉它们你想让它们做什么，告诉它们可以分多步进行，它们也常常会在中间卡住，不太擅长恢复。

Let's see. Certainly there are a lot of tasks like mathematical reasoning where they're not nearly as good as skilled humans. Let's say if you want them to do a long-running path, even if you prompt them very carefully and tell them what you want them to do and tell them they can take multiple steps, they often will get stuck in the middle and aren't very good at recovering.

公平地说，很多人类也需要一段时间才能擅长数学。事实上，有些人从未真正达到数学的顶级水平。大多数人达不到数学的顶级水平或成为最聪明的人类，对吧？

Not to be fair, a lot of humans take a while too before they become good at math. In fact, some humans never truly get to the top level of math. Most people don't get to the top level or the smartest humans in mathematics, right?

是的。

Yeah.

那么你认为有没有可能这只是一个开始，但它可以在数学等方面不断变得更好？

So do you think there might be a path that this is just the beginning, but it could keep getting better at things like math?

哦，是的，我认为模型会不断变得更好，甚至很难确切描述限制是什么或根本限制是什么。所以当然它们现在有实际限制，比如模型没有我们拥有的很多执行器；它们只能写出文本，不能做任何事情。但这些都是非常表面的限制。所以一旦你克服了这些，就不清楚根本的障碍是什么了。当然，我不认为模型足够聪明。

Oh yeah, I think the models are going to keep getting better, and it's even hard to describe exactly what the limitations are or the fundamental limitations. So certainly they have practical limitations right now, like the models don't have a lot of the actuators that we have; they can just write out text, they can't do anything. But those are very superficial limitations. So once you overcome those, it's not clear what the fundamental blockers are. Certainly I don't think the models are smart enough.

肖恩，你提到语言模型之前就存在，但 ChatGPT 却突然突破了人们喜欢与之互动的门槛，之前的模型都没能做到。当你在开发 ChatGPT 及其前身 InstructGPT，引入强化学习组件时，你预料到会发生这种情况吗？这是你当时设想的结果，还是只是觉得这可能是个改进？在它爆火之前你是怎么想的？

Now Sean, you mentioned that language models have existed before, but ChatGPT somehow exceeded its threshold of people just all of a sudden liking to interact with it, which wasn't as much the case with any of the previous models. When you were working on ChatGPT and its predecessor InstructGPT, bringing in the reinforcement learning component, did you think that was going to happen? Was that what you had in mind, or was it just like, oh, this might be an improvement? And what was your thinking before it all took off?

我确实认为聊天界面比之前的更容易使用，所以我觉得即使是一个很简单的产品，也有很大的潜力，这种交互形式很直观。我绝对没有预料到它会这么受欢迎。我以为它只会吸引一个小众群体。其实用 InstructGPT 模型也能做一个聊天机器人：只要给对提示词，就能让它表现得像个聊天机器人，效果还不错。在 ChatGPT 发布的时候，用 InstructGPT 得到的效果已经接近了，可能稍微差一点。我们训练 ChatGPT 让它更有自我意识，了解自己的局限性，减少幻觉，而之前的 Instruct 模型更侧重于续写文本。在写作测试中，那种幻觉其实算是一种特性。所以我认为它比之前能做的稍微好一点，但没有好太多。所以它这么火爆，我也很惊讶。

Well, I did think that the chat UI was a lot easier to use than what people had before, so I thought there was a lot of potential there, even with a pretty minimal product, that this would be an intuitive form factor. I definitely didn't anticipate how popular it would get. I thought it would just have a niche appeal. And you could use the InstructGPT model to make a chatbot: if you just give it the right prompt, you can make it behave like a chatbot and you'll get something decent. At the time we released ChatGPT, you would have gotten something that was close to as good, not quite as good maybe. We trained ChatGPT to be a little more self-aware and understand its limitations and to hallucinate less, whereas the previous Instruct models were more designed for continuing text. In writing tests, that kind of involved hallucination, where that's kind of a feature. So I think it was a little better than what you could have created before, but not dramatically better. So I was surprised that it blew up as much as it did.

ChatGPT 爆火并被广泛使用。你看到的 ChatGPT 的用途中，有没有让你感到惊讶或兴奋的？就是人们用它做的那些有趣的事。

ChatGPT has been blowing up and used so widely. Are there any uses that you've seen of ChatGPT that kind of surprised you or got you excited? Just fun things that you've seen people do.

我看到人们以各种不同的方式使用它来获得价值。很多非英语母语者或需要帮助调整语气的人会大量使用它来辅助写作。这是一个明显的用例。即使是免费的、不那么智能的模型，也不需要 GPT-4 就能做到。还有一些创意用例，比如人们用它给孩子写睡前故事，或者只是找乐子。我看到有人用它来想聊天开场白。我自己用它来获取旅行建议和推荐有趣的活动。我还看到我的学生经常用它来编程，尽管这不是传统意义上的英文文本。它在编程方面似乎也很擅长。实际上，编程用例是我们自己和同事们一直在用的，所以我们真的在吃自己的狗粮，这也是早期的一个主要动力。看到它如何帮助非专业人士编程，真的非常令人兴奋。那些没有正式学过编程的人，只需要给模型一个提示，它就能写出脚本，我见过有人用它完成了非常复杂的任务，尽管他们从未正式编程过，以前也觉得自己做不到。这太令人兴奋了。

Well, I see that people are using it in lots of different ways to get value. I mean, a lot of people who aren't native English speakers or need help writing in the right tone will use it a lot for writing help. So that's an obvious use case. Even the free, less smart models, you don't need GPT-4 for that. Let's see. I see some creative use cases, like people will use it to write bedtime stories for their kids, or just have fun with it. I've seen people use it to come up with conversation starters. I've used it for travel advice and advice on what kinds of fun activities to do. And I'm seeing my students use it all the time for their programming, even though that's not English text in the traditional sense. It seems to be really good at programming too. Actually, the programming use case is one that we were using and my colleagues were using, so that's one we really dogfooded, and that was a big early motivator. Just seeing how much it helps non-experts do programming has been really exciting. People who haven't really studied programming can just prompt the model to write them a script, and I've seen people do very complicated things even though they never formally programmed and previously didn't think they were capable of it. That's super exciting.

约翰，你几次提到幻觉这个概念，说 ChatGPT 可能会产生幻觉。什么是幻觉？你有什么想法来避免模型产生这种幻觉吗？

You've alluded to this notion of hallucinations a couple times, John, that ChatGPT could have hallucinations. What are they, and do you have any thoughts on how to maybe avoid having such hallucinations in the model?

幻觉就是模型开始编造东西，输出一些听起来合理但实际是捏造的事实、数字或引用的文本。出现这种情况的原因，如果允许我们这么说的话，有时更容易理解模型的方式是认为它有一定的主体性。我们可以说，模型更关心听起来正确，或者写作风格正确，而不是实际正确。如果你考虑最大似然目标，即只试图输出可能的词，这一点就很明显。在那个目标中，可能有一小部分是为了输出正确的东西，但更强烈的倾向是输出风格正确或听起来像答案的东西。所以，如果一个模型是以简单的方式训练的，它就会产生大量幻觉。通过我们使用 RLHF 进行的微调，我们大大减少了幻觉，但仍然没有完全消除。所以我们的免费模型会产生相当多的幻觉，尤其是当你开始要求它提供引用之类的东西时，它会直接编造。基于 GPT-4 的更好模型幻觉少得多，但偶尔还是会犯，特别是当你询问它没有的某些具体细节，而它又没有经过训练来意识到这个限制时。

Yeah, hallucinations are just where the model starts making things up, and it outputs some plausible-sounding text which makes up facts or numbers or citations. The reason you get this is, well, if we're allowed to sometimes it's easier to understand the model by saying it has agency in some way. So we could say that the model cares more about sounding right, or writing in the right style, than actually being correct. And that's obviously true if you think about the maximum likelihood objective where you're just trying to output likely words. In that objective, there is maybe some small component where it's trying to output correct things, but there's a much stronger tendency to just output something that is in the right style or sounds like an answer. So if you have a model that's trained in a naive way, it's going to hallucinate a lot. With the fine-tuning we do with RLHF, we cut that down a large amount, but we still don't completely get rid of it. So our free tier model hallucinates a decent amount, especially if you start asking it for citations and that sort of thing, it'll just make something up. The better model based on GPT-4 doesn't hallucinate nearly as much, but it'll still occasionally do it, especially if you ask for certain kinds of specifics that it doesn't have and where it hasn't been trained to be aware of this limitation.

你说它没有经过训练来意识到这个限制，这似乎表明有办法训练它意识到这一点。这怎么做到？你如何让它意识到这些限制？

You're saying it hasn't been trained to be aware of this limitation, which seems to suggest that there is a way to train it to be aware of that. How does that work? How do you make it aware of such limitations?

我们有时会训练模型意识到特定的限制。例如，早期版本的模型完全不知道自己的能力范围。所以如果你问它‘你能给某人发邮件吗？’，它会回答‘是的，我刚发了那封邮件’，因为一个乐于助人的聊天机器人听起来就该这样。于是我们针对这类特定查询进行了训练，教它说‘不，我不能发邮件’。所以你可以通过一个渐进的过程，教给模型它不具备的特定限制，然后模型会进行某种泛化。

We sometimes train the models to be aware of a specific limitation. For example, early versions of our models had no idea what their capabilities were. So you would ask it, 'Can you send an email to so and so?' and it would say, 'Yes, I just sent that email,' because that's kind of what a helpful chatbot might sound like. So then we went and trained it with this specific type of query, and we trained it to say, 'No, I can't send emails.' So you can do a sort of piecemeal process where you teach the model specific limitations that it doesn't have, and then the model will kind of generalize.

这些模型，我想说 GPT-4，因为它是一个非常聪明的模型，所以它的泛化能力确实很好。如果你教它一些它不能做的事情，它会推断出很多其他可能也不能做的事情。但它并不能完美地做到这一点。例如，对于引用这样的事情，模型实际上对特定书籍和著名论文中的内容有很多了解。所以如果你让它给出引用，有时它确实会给出正确的答案。这被认为是有用的。所以在我们进行写作过程时，显然给出答案比不给出要好。所以模型认为它可以给出引用，这是正确的，但它对自己在这些引用上的信心程度并没有很好的内部感知。所以有时它会编造引用。我猜它可能不知道自己在编造，或者它实际上可能知道。所以有时你可以问它：“你确定吗？”它会说：“不，对不起，我编的。”所以是的，我想我们并不完全理解所有这些能力是如何泛化的，以及关于限制的教学是如何泛化的。所以这绝对是一个有趣的研究课题。

The models, and I would say GPT-4, since it's a very smart model, it does generalize quite well. So if you teach it a few things that it can't do, it'll infer lots of other things that it probably can't do. But it doesn't do this perfectly. For example, for something like citations, the model does actually have a lot of knowledge about what's in specific books and famous papers, and so on. So if you ask it for a citation, sometimes it actually gives you correct answers. And that's rated as useful. So when we do our writing process, obviously it's better for it to give the answer than not give it. So the model thinks it can somehow give citations, which is correct, but it doesn't quite have a good internal feeling of how confident it is about these citations. So sometimes it'll just make them up. And I guess it probably doesn't know that it made them up, or it actually might know that. So sometimes you can ask it, 'Are you sure about that?' and it'll say, 'No, sorry, I made that up.' So yeah, I guess we don't totally understand how all these abilities generalize and how teaching about the limitations generalizes. So that's definitely an interesting topic for research.

当你谈到引入引用时，似乎有一种替代方案，不是让模型提前阅读整个互联网，然后仍然尝试用引用回答，而是让它即时检索信息。你对使用检索的模型与将所有内容训练到权重中的模型之间的权衡有什么看法？

When you talk about bringing in citations, it seems like an alternative instead of having the model ahead of time read the entire internet, so to say, and then try to still answer with citations, is to let it retrieve things on the fly. What are your thoughts on the trade-offs between models that use retrieval versus models that have everything trained into their weights?

是的，我认为两种方法都有其用武之地，我们现在也在这样做。在 ChatGPT 中，我们有一个浏览模型，最近我们不得不暂时将其下线，但它会回来的。所以我们有一个可以在网上查找信息的模型，但默认模型没有这个功能。我认为模型可以在其权重中存储大量信息，包括非常详细的事实知识。如果你在权重中拥有信息，模型可以非常灵活地使用它。它可以在事物之间建立联系。如果你问一个与某事物模糊相关的问题，它可能会建立一个很难通过搜索查询建立的联系。所以我认为在权重中拥有信息最终会导致更智能、更灵活的行为。但能够进行检索也有几个很大的优势。首先，实时访问实时信息，访问比你能塞进权重的更详细的信息。最后，它也更容易被人类检查。所以我认为这对于使这些模型输出可检查非常重要，无论是作为训练过程的一部分还是作为测试时用例的一部分。因为在训练时，当我们让人类查看输出并对其进行评分时，模型的知识面非常广，评分的人可能对主题了解不够，无法真正评估答案。所以如果模型能提供引用，那就能更容易地进行准确的监督。所以这非常重要。但作为最终用户，能够检查语言模型的输出也非常有用，原因显而易见，因为它们有时会幻觉。所以如果你能让它的输出可验证，那显然会很有用，尤其是在像医学这样的高风险环境中。

Yeah, I think there's a place for both approaches, and we're doing that right now. So we have in ChatGPT a browsing model, which we recently had to temporarily take down, but it'll be back. So we have a model that can look things up on the web, but the default model doesn't. I think the models can store a huge amount of information in their weights, including very detailed factual knowledge. And if you have information in the weights, the model can use it in a very flexible way. So it can make connections between things. If you ask a question about something that's vaguely related, it might make a connection that would be hard to make with a search query. So I think having information in the weights is ultimately going to lead to smarter and more flexible behavior. But there are a couple big advantages to also being able to do retrieval. First of all, real-time access to real-time information, access to more detail than what you'd be able to cram into the weights. And last of all, it's also more checkable by a human. So I think it's extremely important for making these model outputs checkable, both as part of the training process and as part of the test-time use cases. Because at training time, when we have humans looking at the outputs and rating them, the models have such a big breadth of knowledge that the people doing the rating might not know enough about the subject to really assess the answer. So if the model can provide citations, that makes it a lot easier to get accurate supervision. So that's really important. But then also as an end user, being able to check the language model output is extremely useful for obvious reasons, because they do sometimes hallucinate. So if you can just make its output verifiable, that's obviously going to be useful, especially if it's a high-stakes setting like medicine.

就底层技术而言，据我所知，许多大语言模型的训练方案都是单周期训练，也就是说你只遍历训练数据一次，而不是多次。那么它能从一次遍历中记住那些具体的事情，难道不令人惊讶吗？你有什么直觉，在一次数据遍历中，它只对那个特定的引用进行了一次梯度更新，却 somehow 存储了它？这对我来说似乎很令人惊讶。我不是说这不是真的，我只是对此感到非常惊讶。

In terms of the technology underneath it, as I understand it, a lot of the large language model training regime does single-epoch training, meaning that you just go through your training data once, not multiple times. Isn't it then surprising that it can remember those specific things from just one pass? Do you have any intuition how in one pass over that data, it has gotten just one gradient step on that one specific citation and it somehow stores it? Seems surprising to me. I'm not saying it's not true, I'm just very surprised by it.

是的，这些模型从预训练数据中吸收信息的能力确实令人惊讶。我想说的是，任何给定的事实都会出现在互联网上的许多不同文档中，如果它只出现在一个文档中，模型可能无法回忆起来，至少目前的模型是这样。所以这是一个有趣的问题：模型需要看到这个事实多少次才能真正内化它，我猜大概在几十次左右，但很难确定。

Yeah, it is surprising how well these models absorb information from the pre-training data. I'd say that any given fact will appear in many different documents on the internet, and if it's only in one document, the model probably won't be able to recall it, at least current models. So it's an interesting question how many times the model has to see the fact to really internalize it, and I would guess that's somewhere in the tens, but it's hard to say for sure.

谈谈互联网和数据。显然这些模型正在训练大量数据，最近你的同事 Sam Altman 提到，作为一个社区，随着我们不断扩展这些模型，可能会动力不足，它们可能不会继续变得更好，在数据和模型 Scaling 方面没有足够的前景。你对此怎么看？

Talk about internet and data. Obviously these models are training on a lot of data, and recently your colleague Sam Altman mentioned that might be running out of gas as a community in terms of as we keep scaling these models, they might not keep getting better, not enough opportunity ahead in terms of data and model scaling. What do you think about that?

现有的数据和模型 Scaling 方法肯定会在某个时候逐渐失效，或者至少改进可能在数据集大小和训练算力方面呈对数增长。所以你会遇到收益递减。但我想说还有很多事情要做，我不认为事情会很快进入平台期。

Definitely the existing methods of data and model scaling might peter out after a while, or at least the improvements might be logarithmic in dataset sizes and training compute. So you're sort of hitting diminishing returns. But I'd say there's plenty more to do, and I don't see things plateauing anytime soon.

目前该领域正在发生的事情之一是，当然也有很多竞争对手的模型被发布。不仅仅是 ChatGPT，还有很多其他模型，包括许多开源模型。我很好奇你认为闭源和开源发布的作用是什么？它们都重要吗？一个是对还是错？你的想法是什么？

One of the things that is happening right now in the field is that there are of course many competitor models being released too. It's not just ChatGPT, there are many others, including many open-source ones. I'm curious what you think is the role of closed-source versus open-source releases. Are they both important? Is one right versus wrong? What is your thinking?

是的，事实上 Llama 2 今天发布了，所以这是一个及时的问题。嗯，我认为开源模型对于研究来说当然非常好，比如让学术研究人员能够进行实验，微调模型，进行架构更改等等，并尝试做我们在 OpenAI 所做的那种工作，比如尝试改进基于人类反馈的强化学习。所以拥有一个非常强大的开源模型使这成为可能，所以我认为这非常有价值。我认为对于闭源模型，它们目前更好，或者至少现有的最佳模型更好。而且我认为如果没有一些商业激励，没有闭源产品，很难激励模型变得真正优秀。

Yeah, in fact Llama 2 is released today, so that's a timely question. Well, I think the open-source models are certainly really good for research, like for academic researchers to be able to do experiments where they're fine-tuning the models and making architecture changes and so on, and to try to do the kind of work we're doing at OpenAI, like trying to improve RL from Human Feedback. So having a really strong open-source model makes that possible, so I think that's really valuable. I think for the closed-source models, they are currently better, or at least the best models that are out there are better. And I think it would be hard to incentivize models to get really good without there being some commercial incentive, without it being a closed-source product.

我认为最好的模型会是封闭模型，但开放模型对研究和商业用途非常有用，人们可以在自己的特定数据上微调它们。

I think the best models will be closed models, but open ones are really good for research and commercial uses where people fine-tune them on their specific data.

你是否将其归结为高质量数据和算力所需的资源，使得构建开源模型变得困难？

Are you tying it back to the resources required for high-quality data and compute, making it hard to build an open-source model?

是的，很难激励对无法盈利的开源模型进行如此大的投资。所以我不期望最先进的模型是开源的，但它们对公共利益有很大价值。也存在安全担忧，比如 API 提供商不允许的大规模垃圾信息。像 Meta 这样生产开源模型的公司将不得不三思。

Yes, it's hard to incentivize that big an investment in an open-source model you can't monetize. So I wouldn't expect state-of-the-art models to be open source, but they have great value for the public good. There are also safety concerns, like large-scale spam that API providers wouldn't allow. Companies like Meta producing open-source models will have to think twice about it.

你认为达到下一阶段是继续做同样的事情，比如用人类反馈整理文本数据，还是需要引入其他东西，比如视频或模拟器？

Do you see the future of getting to the next level as doing more of the same, like curating text data with human feedback, or does something else need to be brought in, like video or a simulator?

添加像视频这样的新模态会带来很多好处。如果从扩展现有数据中获得的收益递减，新模态能让模型获取文本形式之外的知识，并以纯语言模型无法做到的方式行动。例如，与物理世界或计算机屏幕交互得益于感知视频。此外，在纯语言模型领域，除了 Scaling 之外还有空间——比如让模型帮助自我评分，而不是仅仅在人类数据上训练奖励模型。

Adding new modalities like video will add a lot. If we get diminishing returns from scaling up existing data, a new modality allows the model to access knowledge not in text form and act in ways a pure language model can't. For example, interacting with the physical world or computer screens benefits from perceiving video. Also, in the pure language model world, there's room beyond scaling—like using models to help grade themselves instead of just training a reward model on human data.

有说法称微调可能会降低泛化能力和知识广度。这是真的吗？它是如何发生的？

It's been said that fine-tuning might reduce generalization and breadth of knowledge. Is that true, and how does it happen?

确实，微调会减少风格和输出的多样性，导致模式崩溃或熵崩溃。例如，如果你让 ChatGPT 讲个笑话，它经常给出同一个。能力也会退化，因为预训练使用更大的批次，在大量输入上保留能力，而微调的数据集更小、噪声更多。我们运行基准测试确保能力不会退化太多，在最新配方中退化不大。

It's true that fine-tuning reduces the variety of styles and outputs, causing mode collapse or entropy collapse. For example, if you ask ChatGPT to tell a joke, it often gives the same one. There's also degradation of capabilities because pre-training uses much bigger batches and preserves capabilities on a huge variety of inputs, while fine-tuning has a smaller dataset and more noise. We run benchmarks to ensure abilities don't degrade too much, and in latest recipes they don't degrade much.

你认为未来还有其他领域可能出现类似的巨大飞跃吗？

Do you see any other area that could see a similarly big leap forward in the future?

我没有看到某个特定领域会起飞。语言模型将作为核心，许多东西在此基础上构建。其他模态很可能会建立在它们之上。

I don't have a specific area that will take off. Language models will serve as a core that many things are built on top of. Other modalities will likely be built on top of them.

语言模型就像……你会拿大型语言模型，然后加上视觉、视频等等。语言有一个很大的优势：信息密度高，不像视频等其他类型的数据那样有那么多噪声。所以在很长一段时间内，语言将是用有限算力吸收大量智能的好方法。但语言和其他模态之间存在迁移，所以你会看到人们把语言和视频等结合起来。对于机器人技术，我预测最终人们会使用某种多模态模型，联合训练语言、视频和控制。这类东西是 AI 的未来。还有一些领域与你训练什么模型完全正交，但会变得重要，比如可扩展监督或提高监督质量。思路是：在人类难以产生高质量标签或演示的领域，如何收集数据？我们如何结合机器和人类，创造出比人类单独能创造的更高质量的数据，并在非常困难的领域监督模型？这是一个重要问题，可能会在机器学习研究界变得更流行。一个例子是能有效进行科学研究的 AI，它读取生物数据，如蛋白质序列、RNA、DNA 序列，查看实验结果，并将它们组合成人类很难想出的新假设或结论。

Language models like you'll have, you'll take the big language models and you'll add vision and video and so on. Language has a big advantage: it's very information dense and doesn't have as much noise as other types of data like video. So for a long time, language is going to be a good way to soak up a lot of intelligence with a limited amount of compute. But there's transfer between language and other modalities, so you'll see people having language plus video and so on. For robotics, I would predict that eventually people will use some kind of multimodal model that is jointly trained with language, video, and control. That kind of thing is in the future of AI. There are also areas that are totally orthogonal to what kind of model you're training that will rise in prominence, like scalable oversight or improving supervision quality. The idea is: how can you collect data in domains where it's hard to get humans to produce high-quality labels or demonstrations? How do we use bot plus human together to create higher quality data than a human would be able to create, and supervise models in domains that are really hard? This is an important problem and will probably become more popular in ML research. An example is an AI that would do scientific research effectively, reading biological data like protein sequences, RNA, DNA sequences, looking at experimental results, and combining them into new hypotheses or conclusions that are very hard for humans to come up with.

是的，我同意。这真的很令人兴奋。有些领域，比如生物学，对人类来说可能太复杂了。既然 AI 即使不比人类更聪明，也肯定能更快地做更多工作，我们可以让它们筛选大量复杂的生物学数据并找出答案。我期待看到这一切发生。也许我们将来都会从事这方面的工作，谁知道呢。现在，回顾你的职业轨迹，John，我注意到你如何跨越不同主题。你从我的实验室开始做模仿学习和机器人技术，然后得出结论认为强化学习是取得更多进展的关键，接着花了很多时间在强化学习上，创办了 OpenAI，并将其引入语言模型。我对这个轨迹很好奇。让我们从最新的开始，然后往回追溯。你什么时候决定开始关注语言模型，为什么？

Yeah, I agree. That's a really exciting one. There might be some fields like biology that are just too complicated for humans. Since AIs, even if they're not smarter than humans, can certainly do more work faster, we can have them sift through a lot of complicated data from biology and figure something out. I look forward to seeing that happen. Maybe we'll both be working at it in the future, who knows. Now, one thing that stood out to me as I look at your career trajectory, John, is how you've moved across topics. You started in my lab in imitation learning robotics, then concluded reinforcement learning would be key to make more progress, then spent a lot of time on reinforcement learning, opened AI, and brought it into language models. I'm really curious about that trajectory. Let's maybe start with the latest and work our way back. When did you decide to start paying attention to language models and why?

大约在 GPT-2 的时候，开始变得清楚这些东西真的很好，值得关注。我当时并没有真正转向研究语言模型。我当时的结论是，无监督学习现在算是可行了，训练一个生成模型是创建具有通用能力的模型的好方法，这些模型可以针对下游任务进行微调。那时，我对强化学习中的样本效率非常感兴趣——模型学习新任务的速度有多快。这在某种程度上是强化学习的核心问题，甚至可能是 AI 的核心问题之一。所以我对样本效率非常感兴趣。GPT-2 出现了，可以做很多事情，比如少样本学习，而且针对不同任务进行微调也取得了很好的结果。所以我想，也许我们应该为玩游戏和机器人等领域训练视频模型，然后在强化学习任务上对它们进行微调。我在这方面做了一些工作，它有点效果，但最终没有好到让人兴奋的程度。大约在 GPT-3 的时候，我对它的出色表现更加震惊，然后我决定把我的工作和团队的工作转向在语言模型上做强化学习。我们实际上不是 OpenAI 中第一个做自己语言模型的，但作为强化学习团队，我们认为朝这个方向走是有意义的。那时，我们开始了两个项目：一个是关于解决数学问题，另一个是关于整合检索和网页浏览，并使用强化学习来学习如何更好地使用这些工具。这就是我进入语言模型领域的方式。我记得时间线——这大概是 2019 年中期。

Around GPT-2, it started to become clear that these things were really good and worth paying attention to. I didn't actually switch over to working on language models at that point. My conclusion at that point was that unsupervised learning kind of works now, and training a generative model is a really good way to create a model with general-purpose capabilities that can be fine-tuned for a downstream task. At the time, I had been really interested in sample efficiency in reinforcement learning—how fast the model can learn a new task. That's in some ways the core problem in reinforcement learning and maybe even one of the core problems in AI. So I was really interested in sample efficiency. GPT-2 came out and could do a lot of things like few-shot learning, and there were also good results with fine-tuning it for different tasks. So I thought that maybe we should train video models for domains like playing games and robotics, and then fine-tune them on RL tasks. I worked on that a bit, and it kind of worked, but it didn't end up working well enough to get really excited about. Around GPT-3, I was even more blown away by how good it was, and then I decided it made sense to pivot my work and my team's work to do RL on language models. We weren't actually the first ones at OpenAI to do our own language models, but we decided as the RL team it made sense for us to go in this direction. At that time, we started working on two projects: one was around solving math problems, and the other was around incorporating retrieval and web browsing and using RL to learn how to use those tools better. So that's how I got into language models. I remember the timeline—this was probably mid-2019.

你提到过转型。读博期间，你从专注于模仿学习、通过演示教机器人取得很好成果，转向决定强化学习才是取得进展的关键。当时你为什么做出这个转变？因为对很多研究者来说，大问题是：你做的事情对不对？尤其是你已经在某个主题上投入了大量时间，决定转向一个邻近但全新的主题，是一个高成本的决策，因为你的产出可能会放缓一段时间，然后才能在新领域达到同样的生产力。所以我很好奇你当时的思考过程，而且你作为博士生就已经敢于转换方向了。

Now you made a transition. During your PhD, you transitioned from being really focused on imitation learning, getting really good results teaching robots from demonstrations, to then deciding that reinforcement learning would be key to make progress. Why did you make that transition at the time? Because I think for many researchers, the big question is: are you working on the right thing? And especially if you've already invested a bunch of time on one topic, making the decision to transition into a neighboring but new topic is kind of a high-cost decision, because you'll probably slow down for a while in your output before you start doing, you know, the same productivity in the new area. So I'm curious about your thought process back then, also even as a PhD student that you already dared to switch topics in some sense.

是的，从机器人转向强化学习是我最大的转变，可能仅次于最初进入机器学习领域。但这是一个巨大的转变，因为如你所知，我玩了很长时间的小例子，比如拼车之类的东西，大概六个月。但很快，这需要很大的信念飞跃。之后我做的转变感觉更平滑，更像是连续的过渡，比如从一个领域做强化学习转到另一个领域，或者从一个问题转到另一个问题。所以这些转变对我来说一直很自然。嗯，我认为这很好，结果证明是运气。要么是有先见之明，要么是幸运地在那个时间点转向了强化学习。

Yeah, I'd say the switch from working on robotics to working on RL was the biggest switch they made, except perhaps going into machine learning in the first place. But I'd say that was a big shift, because as you remember, I was just playing with toy examples for a long time. I was playing with like carpool and that kind of thing for like six months. But soon, that involved a big leap of faith. I'd say the ones that I've done after that have felt a little smoother and more of a continuous transition, like switching from doing RL on this domain to doing RL in that other domain, or focusing on one problem to another problem. So yeah, it's always felt pretty natural to do these transitions. Well, I think it was good to, well, it turned out to be luck. Either prescient or lucky to switch to RL at the time that I did.

你做了最早将规划与强化学习结合的工作之一，至少在现代深度学习真正起作用的时代，这仍然是很多人今天还在使用和构建的工作，当然包括近端策略优化，它可能仍然是今天使用最广泛的强化学习算法。我有点好奇，回想起来，你博士毕业已经有一段时间了，对吧？大概七年了。但你一定还记得博士时光，尤其是现在很多博士生心里想的一件事：工业界，特别是 OpenAI，拥有巨大的预算。微软最近的投资是 100 亿美元，这似乎主要用于算力和数据整理等。这种预算在博士项目中显然是没有的。所以看起来在 OpenAI 有一些推进 AI 的机会，但博士项目中可能没有。不过，从你在 OpenAI 的角度来看，你是否看到一些不需要大量算力和数据预算、同样令人兴奋的事情可以做？

You did some of the first work in combining planning with reinforcement learning, at least in the modern era where deep learning really started to work, which is still the work a lot of people build on and use today, including of course proximal policy optimization, which is probably the most widely used reinforcement learning algorithm still today. I'm kind of curious, as you think back, it's been a while since you were in your PhD, right? It's been seven years by now probably. But you must remember PhD days, and especially there's a thing that's on a lot of PhD students' minds today, which is industry, especially OpenAI, has a tremendous budget, very large budget. The latest investments from Microsoft is a $10 billion investment, which is seemingly largely going to compute and maybe data curation and so forth. That kind of budget is obviously not available in PhD programs. And so it seems like some opportunities to make progress in AI exist at OpenAI but might not exist in PhD programs. Still, from your perspective being at OpenAI, do you see opportunities to do things that don't require the massive compute and data budget that are also very exciting?

是的，我认为如此。我的意思是，我偶尔会看到来自学术界的好论文，所以这绝对是可能的。但有点棘手；你需要找到一个 niche，不会被工业界抢先，或者你的工作不会因为某个发布的模型而过时。所以你需要考虑这一点，但我认为这样的主题很多。无论是做一些有趣的微调，还是进行科学调查来理解这些模型及其泛化方式，或者如何收集数据以更好地监督它们，我认为有很多机会做高质量的科学研究，比如真正深入挖掘和理解事物。而在工业实验室，可能更关注结果和创造更好的产品。是的，我认为作为博士生，有很多机会可以非常精确地做事，保持好奇心，并试图理解事物。

Yeah, I think so. I mean, I see papers from academia that I like occasionally, so it's definitely possible. I'd say it's a little tricky; you need to figure out some kind of niche where you're not going to get scooped by industry, or your work isn't going to be just kind of obsoleted by some model that gets released. So yeah, you gotta think about that, but I think there are plenty of such topics available. Whether you're doing some interesting kind of fine-tuning, or some kind of scientific investigation to understand these models and how they generalize, or how to collect data to supervise them better, I'd say there's a lot of opportunity for doing high-quality scientific work, like trying to really dig in and understand things. Whereas at an industry lab, there might be more of a focus on results and on creating better products. Yeah, I think there's a lot of opportunity to do things really precisely, be curious, and try to understand things as a PhD student.

有没有一些你知道的例子，如果你有时间的话，你会个人感到兴奋的事情？但我知道你已经很忙了，但你会说，‘哇，这些是如果我现在是博士生，我可能会去解决的问题’？或者有没有一个你识别这类问题的过程？

Are there things that you know, examples of things that you would personally be excited about if you had the time to spare, but you know, I guess here you're already so busy, but they say, 'Wow, these are problems that if I were a PhD student today, I might take on'? Or is there maybe a process how you would identify such problems?

嗯，让我想想。我没有准备清单，所以也许我谈谈过程。我可能会先想，首先，哪些能力是模型拥有会令人兴奋的，但如何实现还不清楚？我们当前训练模型有哪些局限性？不一定非得是新能力，比如让模型做手术之类的，而是说，我们这样做似乎不好。我们不了解模型的能力来自数据集的哪一部分，这似乎不好。所以我认为这实际上是一个有趣的领域：模型行为对数据集的归因。最近有一些有趣的工作，但事实是，我们预训练模型，然后在不同的数据混合上微调，最终得到一些东西，却不确定它的所有行为来自哪里，这似乎不好。所以你可能会有这个想法，然后回过头来说，好吧，我们如何解决这个问题？然后就从那里开始。所以我会结合思考未来感兴趣的能力，或者思考当前方法的弱点，并尝试修复它们，然后避免那些看起来没有你的帮助也能解决的问题。

Yeah, let's see. I don't have a list prepared, so maybe I'd go for the process. Well, I'd probably just think about what, okay, so first of all, what are some abilities that I think would be exciting for models to have, where it's not totally clear how to get there? What are some limitations in how we currently train models? Maybe it doesn't have to be a new capability like you want the model to do surgery or something, but it's like, it seems bad that we do things this way. It seems bad that we don't understand where in the dataset models' capabilities are coming from. So I think this is actually one interesting area: attribution of model behavior to the dataset. So there's been some interesting recent work on this, but I think the fact that we pre-train the models and then fine-tune them on these different soups of data, and we get something out at the end and we're not sure where all of its behaviors came from, that seems bad. So you might have this idea and then go back and say, okay, how can we fix that? And then yeah, just go from there. So I would, maybe there's a combination of thinking about future capabilities of interest, or thinking about weaknesses of current methods, and just trying to fix them, and then avoiding things that seem like they're on track to being solved without your help.

如果我们回顾深度学习的起源，你和其他合作者研究了很多年才真正取得成果。你认为我们现在可能处于局部最优吗？当时除了少数人，没人研究深度学习，然后它突破了。现在每个人都在研究这些在大数据集上训练的大模型。只是说一些担忧：未来可能是小数据集吗？可能不是超级小，但有没有可能还有别的东西尚未被发现，与我们今天所做的截然不同？

If we look back at the origins of deep learning, with you working on it for many years before it really came to fruition, of course with many collaborators, you think it's possible that we're in a local optimum now? Again, back then nobody was working on deep learning except for a few people, and then it broke through. Now everybody's working on these large models trained on large data sets. Just to say something concerned: is the future maybe tiny data sets? Probably not super tiny, but is it possible that there is something else that is yet to be discovered that's quite different from what we're doing today?

是的，我认为这是可能的，而且实际上很有可能。谁知道呢，也许我们可以用小数据集走得更远，或者有另一种范式。我认为保持开放的心态很重要。

Yeah, I think it's possible and quite likely in fact. And who knows, it could be that we can get a lot further with tiny data sets, or maybe there's a different paradigm. I think it's important to keep an open mind.

人类通过眼睛接收高带宽数据来学习，但婴儿看到的通常只是一栋房子，所以能从中学会很好的视觉系统已经很了不起了。还有很多东西有待发现。很可能存在比我们现在更好的新架构和损失函数。人们倾向于继续推进当前有效的方法，而且它确实在不断扩展，但我们可能陷入了局部最优。很难预测重大突破会出现在哪里。

Humans learn from high-bandwidth data through their eyes, but a baby sees mostly one house, so learning a good visual system from that is amazing. There's still a lot to discover. It's probably the case that new architectures and loss functions are better than what we have now. There's a temptation to keep pushing what's working, and it keeps scaling, but we might be in local optima. It's hard to predict where the big breakthroughs will be.

当我们谈论杰夫和约书亚时，可能存在幸存者偏差——那三个做对了事情的人成功了，但很多人当时在做各种冷门研究，从未得到认可。所以这很难。现在有些人认为，在学术界做有趣的博士研究比在工业界难，因为工业界有更大的资源和更可见的成果。如果你今天从加州理工本科毕业，你会选择读博还是直接进入工业研究实验室？

When we talk about Jeff and Yoshi, there might be survivorship bias—the three people who worked on the right thing succeeded, but many worked on obscure things and never got recognized. So it's hard. Some argue it's hard today to do interesting PhD work in academia compared to industry, where you have access to bigger resources and more visible results. What would you personally do if you graduated from Caltech undergrad today? Would you explore a PhD or jump into an industrial research lab?

博士项目或实习项目都是不错的起点。它们各有利弊。博士时间更长，可以成为某个领域的专家，还能做实习。实习项目自由度较小，探索空间也有限，所以存在探索与利用的权衡。我不确定自己会怎么选，两者都是好选择。

Either a PhD program or a residency program could be a good start. They have pros and cons. A PhD is longer, so you can become a world expert in something, and you can do internships. A residency program gives you less freedom and less runway to explore, so there's an exploration-exploitation trade-off. I'm not sure what I would do; both are good options.

我再问一个研究建议问题。你是我实验室里最成功的博士生之一。很多学生问你是怎么做的研究。你作为博士生时的方法是什么？你的日程和轨迹是怎样的？

I'll ask another research advice question. You stand out as arguably the most successful PhD student from my lab. Many students ask how you went about your research. What was your approach as a PhD student? What did your schedule and trajectory look like?

我当时天时地利人和，所以很难复制。如果重来一次，我会彻底阅读该领域的文献，包括优化和信息论的基础教材。头两年，我跟着实验室的方向走，比如个人机器人和手术机器人，尝试用相对通用的方法做出一些酷的东西。我认为有激励性的问题很好，但你不是在造产品，所以它们应该激励出好方法。后来深度学习兴起，我经历了博士中期危机，觉得机器人领域的一切都很粗糙。于是我开始探索深度强化学习。这是一个自然的演进：从目标导向的工作开始，然后看到局限性，转向方法导向的研究。

I was in the right place at the right time, so it's hard to replicate. If I were to redo it, I would read up thoroughly on the area, including fundamental textbooks on optimization and information theory. In the first couple of years, I went with what was going on in the lab—like personal robotics and surgical robotics—and tried to do something cool with reasonably general methods. I think it's good to have motivating problems, but you're not building a product, so they should motivate a good method. Then deep learning took off, and I had a mid-PhD crisis, thinking everything in robotics seemed hacky. So I started exploring deep RL. That's a natural progression: start with goal-oriented work, then see limitations and move to methods-oriented research.

谢谢分享，约翰。希望这对读博或做实习的人有帮助。你有时会放松吗？做些什么？

Thanks for sharing that, John. Hopefully it helps people in their PhD or residency programs. Do you ever have time to relax? What do you do?

我工作很多，但有时也很懒，需要努力才能完成任务。最近我开始攀岩。我也跑步、听音乐，在后院养了鸡，弹钢琴，刚去意大利度假了。

I work a lot but I'm also sometimes lazy and struggle to get things done. Lately I've been getting into rock climbing. I also go running, listen to music, have chickens in my backyard, play the piano, and I just went on vacation to Italy.

这是一场精彩的对话。非常感谢你抽出时间。谢谢你的参与。

A fantastic conversation. I really appreciate you making the time. Thanks for joining.

哦，是的，谢谢邀请。这很棒。

Oh yeah, thanks for having me. This is great.