DeepMind 创始工程师 Giannis 探讨从 AlphaGo 历史性胜利到结合强化学习与大语言模型的 AI 智能体未来。
Giannis, a founding engineer at DeepMind, discusses the journey from AlphaGo's historic victory to the future of AI agents combining reinforcement learning and large language models.
要点 · TL;DR
从 AlphaGo 到现代 AI 智能体，扩展和规划是关键原则。 Scaling and planning are key principles from AlphaGo to modern AI agents.
强化学习使模型能够通过自生成数据超越人类数据限制而改进。 Reinforcement learning enables models to improve via self-generated data beyond human data limits.
LLM 需要像游戏 AI 那样的鲁棒性和可靠性，才能成为可信的智能体。 LLMs need robustness and reliability, like game-playing AIs, to be trusted agents.
核心观点 · Key points
Scaling（规模扩张）和规划是从 AlphaGo 到现代 AI 智能体的关键原则。 Scaling and planning are key principles from AlphaGo to modern AI agents.
强化学习使模型能够通过自生成数据超越人类数据限制来改进。 Reinforcement learning enables models to improve via self-generated data beyond human data limits.
LLM 需要像游戏 AI 那样的鲁棒性和可靠性，才能成为可信的智能体。 LLMs need robustness and reliability, like game-playing AIs, to be trusted agents.
合成数据对于克服数据墙至关重要，但需要谨慎的方法。 Synthetic data is essential to overcome the data wall, but requires careful methods.
上下文学习使模型能够通过少量示例即时适应。 In-context learning allows models to adapt on the fly with few examples.
反共识 · Contrarian takes
AlphaGo 的第 37 手最初被认为是错误，但揭示了 AI 的创造力。 AlphaGo's move 37 was initially thought an error, but revealed AI creativity.
AlphaZero 从随机权重开始优于使用人类专家数据。 Starting from random weights in AlphaZero outperformed using human expert data.
MuZero 在不了解游戏规则的情况下学习内部世界模型，可应用于现实世界。 MuZero learns an internal world model without knowing game rules, applicable to real-world.
由于环境可控，数字 AGI 将比具身 AGI 更早到来。 Digital AGI will arrive much earlier than embodied AGI due to controlled environments.
LLM 尚未迎来它们的 AlphaZero 时刻；可能在未来 5 年内到来。 LLMs haven't had their AlphaZero moment yet; it may come in 5 years.
本期章节 · Chapters（共 27）
引言与 AlphaGo 的意义Introduction and AlphaGo's significance
DeepMind 为何从游戏起步Why DeepMind started with games
游戏对现实世界的代表性Representativeness of games for real world
AlphaGo 的开发与围棋选择AlphaGo's development and the choice of Go
AlphaGo 作为里程碑AlphaGo as a milestone
强化学习与深度学习的作用Role of reinforcement learning and deep learning
工程挑战与规模Engineering challenges and scale
AlphaGo 项目起源与信念AlphaGo project origin and conviction
第 37 手与创造力Move 37 and creativity
第 78 手与盲点Move 78 and blind spots
AlphaZero 与自我对弈AlphaZero and self-play
AlphaZero：自我对弈与策略改进AlphaZero: Self-Play and Policy Improvement
MuZero：无规则学习世界模型MuZero: Learning a World Model Without Rules
强化学习的回归Reinforcement Learning's Return
LLM 世界中的强化学习Reinforcement Learning in the LLM World
合成数据的视角Perspective on Synthetic Data
推理与新颖科学发现Reasoning and Novel Scientific Discoveries
用于推理的强化学习Reinforcement Learning for Reasoning
来自 AlphaGo 和 AlphaZero 的教训Lessons from AlphaGo and AlphaZero
开放问题：鲁棒性与可靠性Open Questions: Robustness and Reliability
关键挑战：数据墙、规划、鲁棒性、上下文学习Key Challenges: Data Wall, Planning, Robustness, In-Context Learning
初创公司优势 vs 大型实验室Startup advantages vs big labs
定义里程碑与钦佩的研究者Defining milestones and admired researchers
下一个重大里程碑与 SWE-bench 预测Next big milestones and SWE-bench prediction