Jared Fowk explains how models reason, the role of chain of thought, and what's next after GPT-5.
要点 · TL;DR
推理需要更长的思考时间，并通过思维链进行语言化表达。 Reasoning requires longer thinking time and chain-of-thought verbalization.
预训练加强化学习是现代 AI 的核心公式。 Pre-training plus RL is the core formula for modern AI.
扩展强化学习比扩展预训练更难，因为故障模式更多。 Scaling RL is harder than scaling pre-training due to more failure modes.
核心观点 · Key points
推理是得出未知答案的过程，通常需要更长的思考时间。 Reasoning is the process of getting to an answer you don't yet know, often requiring longer thinking time.
思维链用人类语言将模型的思考过程语言化，实现逐步解决问题。 Chain of thought verbalizes the model's thinking process using human language, enabling step-by-step problem solving.
预训练加强化学习是现代 AI 的核心公式，自 2019 年以来 OpenAI 一直以此为目标。 Pre-training plus reinforcement learning is the core formula for modern AI, as envisioned by OpenAI since 2019.
基于人类反馈的强化学习（RLHF）对于让 GPT-4 变得连贯有用至关重要，将一个令人失望的模型变成了 ChatGPT。 RLHF was crucial for making GPT-4 coherent and useful, turning a disappointing model into ChatGPT.
扩展强化学习比扩展预训练更难，因为涉及更多活动部件和故障模式。 Scaling reinforcement learning is harder than scaling pre-training due to more moving parts and failure modes.
反共识 · Contrarian takes
GPT-4 最初在内部令人失望；需要基于人类反馈的强化学习（RLHF）才能释放其真正潜力。 GPT-4 initially felt underwhelming internally; RLHF was needed to unlock its true potential.
OpenAI 的研究内部完全透明；每个人都知道一切，以最大化协作。 OpenAI's research is fully transparent internally; everyone knows everything to maximize collaboration.
DeepSeek 的 GRPO 发布比 OpenAI 自己的 O1 发布更能加速美国实验室的推理研究。 DeepSeek's GRPO release accelerated US labs' reasoning research more than OpenAI's own O1 release.
没有预训练的纯强化学习不可行；两者对于成功都是必要的。 Pure reinforcement learning without pre-training is not viable; both are necessary for success.
对齐是一项永无止境的追求，因为人类对是非的观念会随时间演变。 Alignment is a never-ending pursuit because human notions of right and wrong evolve over time.
本期章节 · Chapters（共 25）
引言与推理的含义Introduction and the meaning of reasoning
思维链及其工作原理Chain of thought and how it works
模型如何决定思考时间How models decide thinking time
推理模型的演进Evolution of reasoning models
背景与职业转型Background and Career Shift
加入 OpenAIJoining OpenAI
强化学习规模化：从 Dota 到机器人Scaling Reinforcement Learning: From Dota to Robotics
Jerry Tworek 的一天A Day in the Life of Jerry Tworek
研究优先级：自上而下与自下而上Determining Research Priorities: Top-Down vs. Bottom-Up
OpenAI 的协作与知识产权保护Collaboration and IP Protection at OpenAI
OpenAI 的协作文化Collaborative culture at OpenAI
RL：基于奖励的训练RL as Training with Rewards
RL 演进：从经典到深度Evolution of RL: From Classical to Deep RL
早期 RL 与 GPT-4 表现平平Early RL and GPT-4 underwhelming
自监督学习与监督学习Self-supervised learning vs supervised learning
GRPO 及其影响GRPO and its impact
RL 规模化：挑战与复杂性Scaling RL: challenges and complexity
智能体 AI 与推理 RLAgentic AI and reasoning RL
智能体 AI 与长周期任务Agentic AI and Long-Horizon Tasks
对齐与 RLAlignment and RL
RL 效率与数学/编程RL Efficiency and Math/Coding
竞赛表现与里程碑Competition Performance and Milestones
预训练与 RL 通往 AGI 之路Pre-training and RL path to AGI
Richard Sutton 的纯 RL 观点Richard Sutton's pure RL view