OpenAI 联合创始人兼 ChatGPT 首席架构师 John Schulman 解释了该模型是如何通过预训练、监督微调和基于人类反馈的强化学习构建的。
John Schulman, co-founder of OpenAI and lead architect of ChatGPT, explains how the model is built through pretraining, supervised fine-tuning, and reinforcement learning from human feedback.
要点 · TL;DR
ChatGPT 通过互联网文本预训练和 RLHF 微调实现对齐。 ChatGPT is built via pre-training on internet text and fine-tuning with RLHF for alignment.
幻觉源于模型优先风格而非事实；RLHF 减少但未消除。 Hallucinations persist because models prioritize style over facts; RLHF reduces but doesn't eliminate them.
数据和算力扩展可能遇瓶颈；新模态和架构带来未来进展。 Scaling data and compute may face limits; new modalities and architectures offer future progress.
核心观点 · Key points
大型语言模型通过互联网文本预训练构建，然后使用基于人类反馈的强化学习（RLHF）进行微调以实现对齐。 LLMs are built via pre-training on internet text, then fine-tuning with RLHF for alignment.
幻觉源于模型优先考虑合理的风格而非事实准确性；基于人类反馈的强化学习（RLHF）可以减少但无法消除幻觉。 Hallucinations stem from models prioritizing plausible style over factual accuracy; RLHF reduces but doesn't eliminate them.
模型可以在权重中存储大量知识，实现灵活连接，但检索有助于实时信息和可验证性。 Models can store vast knowledge in weights, enabling flexible connections, but retrieval aids real-time info and verifiability.
Scaling（规模扩张）数据和算力可能面临收益递减，但视频等新模态和更好的训练方法提供了进步空间。 Scaling data and compute may face diminishing returns, but new modalities like video and better training methods offer progress.
闭源模型激励大规模投资，而开源模型有利于研究和微调灵活性。 Closed-source models incentivize large investments, while open-source models benefit research and fine-tuning flexibility.
反共识 · Contrarian takes
模型在速度和广度上超越人类，但在数学推理上不如熟练的人类。 Models are superhuman in speed and breadth but worse than skilled humans in mathematical reasoning.
微调可能导致模式崩溃，并因数据集较小和噪声而降低能力。 Fine-tuning can cause mode collapse and degrade capabilities due to smaller datasets and noise.
语言模型将作为其他模态的核心，而非被其取代。 Language models will serve as a core for other modalities, not be replaced by them.
使用模型帮助自我评分的可扩展监督是未来的关键方向，而不仅仅是 Scaling（规模扩张）。 Scalable oversight using models to help grade themselves is a key future direction, not just scaling.
我们可能处于局部最优；新的架构和损失函数可能超越当前方法。 We may be in a local optimum; new architectures and loss functions could surpass current methods.
本期章节 · Chapters（共 22）
引言Introduction
赞助鸣谢Sponsor Acknowledgments
什么是 ChatGPTWhat is ChatGPT
训练流程：预训练与微调Training Pipeline: Pre-training and Fine-tuning
超人能力与局限Superhuman Capabilities and Limitations
ChatGPT 意外走红ChatGPT's unexpected popularity
惊喜与实用案例Surprising and exciting use cases
理解与缓解幻觉Understanding and mitigating hallucinations
模型局限的泛化Generalization of model limitations
单周期训练与记忆Single-epoch training and memory
扩展极限与数据可用性Scaling limits and data availability
闭源与开源模型Closed-source vs open-source models
开放与封闭模型Open vs Closed Models
语言模型未来：新模态Future of Language Models: New Modalities
微调与模式崩溃Fine-Tuning and Mode Collapse
超越语言模型的突破Future Breakthroughs Beyond Language Models
多模态模型与未来方向Multimodal models and future directions
从模仿学习到强化学习Transition from imitation learning to reinforcement learning
大预算时代博士生机遇Opportunities for PhD students in the era of large budgets
超越大规模深度学习的新范式Possibility of a new paradigm beyond large-scale deep learning