从零构建 AlphaGo：AI 研究启示录

Building AlphaGo from Scratch: Insights for AI Research

埃里克·张 Eric Jang · Dwarkesh 播客 · 2026-05-15 · 约 157 分钟 · 原视频 ↗

打开互动全文版（中英对照 + 朗读 + 问答）→

本期速览 · Overview

Eric Jen 解释为何在休假期间重建 AlphaGo，以及这揭示了 AI 研究与开发的未来。

Eric Jen explains why he spent his sabbatical rebuilding AlphaGo and what it reveals about the future of AI research and development.

要点 · TL;DR

• AlphaGo 通过神经网络与蒙特卡洛树搜索，将围棋搜索变得可行，达到超人类水平。
  AlphaGo uses MCTS with neural nets to make Go search tractable, achieving superhuman play.
• MCTS 作为策略改进算子，通过自我对弈将搜索结果蒸馏到网络中。
  MCTS acts as a policy improvement operator, distilling search results into the network via self-play.
• 关键洞察：神经网络可摊销棘手的搜索，但 MCTS 需要准确的价值估计。
  Key insight: neural networks amortize intractable search, but MCTS requires accurate value estimates.

核心观点 · Key points

• AlphaGo 的核心突破是使用神经网络，通过 MCTS 使搜索问题变得可处理。
  AlphaGo's core breakthrough was using neural nets to make the search problem tractable via MCTS.
• MCTS 使用四个步骤：选择、扩展、评估和回溯，迭代构建树。
  MCTS uses four steps: selection, expansion, evaluation, and backup to iteratively build the tree.
• 价值网络从棋盘状态预测获胜概率，截断深度搜索。
  The value network predicts win probability from a board state, truncating deep search.
• 策略网络提供动作的先验分布，指导 MCTS 探索。
  The policy network provides a prior over actions, guiding MCTS exploration.
• MCTS 充当策略改进算子：搜索结果成为新的训练目标。
  MCTS acts as a policy improvement operator: search results become new training targets.
• AlphaGo 的自对弈循环将 MCTS 蒸馏到网络中，实现迭代增强。
  AlphaGo's self-play loop distills MCTS into the network, enabling iterative strength gains.

反共识 · Contrarian takes

• 一个 10 层神经网络能够以高保真度摊销一个几乎难以处理的搜索问题。
  A 10-layer neural network can amortize a nearly intractable search problem with high fidelity.
• 如果价值估计不准确或模拟次数少，MCTS 不保证改进策略。
  MCTS is not guaranteed to improve policy if value estimates are inaccurate or sims are low.
• 在低算力预算下，ResNet 由于局部归纳偏置在围棋上优于 Transformer。
  ResNets outperform Transformers for Go at low compute budgets due to local inductive bias.
• 对整个轨迹的朴素策略梯度方差高；MCTS 提供每动作的监督。
  Naive policy gradient on full trajectories has high variance; MCTS provides per-action supervision.
• 缩放定律需要先有工作系统；在坏数据上研究缩放无法获得洞察。
  Scaling laws require a working system first; studying scaling on bad data yields no insight.
• 围棋的确定性完美信息结构使 MCTS 可处理；语言缺乏这一点。
  Go's deterministic perfect-information structure makes MCTS tractable; language lacks this.

本期章节 · Chapters（共 41）

• 引言与动机 Introduction and Motivation
• 围棋运作原理 How Go Works
• 围棋计分规则与人机差异 Go scoring rules and human vs computer differences
• AlphaGo方法：搜索与神经网络 AlphaGo's approach: search and neural networks
• MCTS数据结构与动作选择 MCTS Data Structure and Action Selection
• UCB与PUCT中的探索与利用 Exploration vs Exploitation in UCB and PUCT
• 确定性游戏中的概率 Probability in Deterministic Games
• 回溯步骤与价值赋值 Backup Step and Value Assignment
• 价值函数与人类直觉 Value Function and Human Intuition
• 围棋中的全局特征聚合 Introduction and Motivation
• 架构选择与训练设置 Global feature aggregation in Go
• 从专家数据开始 Architecture choice and training setup
• 将神经网络应用于蒙特卡洛树搜索 Starting from expert data
• MCTS选择与扩展 Applying neural network to Monte Carlo tree search
• 蒙特卡洛树搜索过程 MCTS Selection and Expansion
• 简街数据中心之旅 Monte Carlo Tree Search Process
• AlphaGo中的RL与MCTS Jane Street Data Center Tour
• MCTS与策略网络质量 RL and MCTS in AlphaGo
• AlphaZero训练动态 MCTS vs Policy Network Quality
• 实用训练技巧 AlphaZero Training Dynamics
• 随机对弈与迁移学习的自举 Practical Training Tricks
• AlphaGo与现代RL：推理深度对比 Bootstrapping with random play and transfer learning
• NP难问题与神经网络 Comparing AlphaGo and modern RL: depth of reasoning
• 混沌与可预测性 NP-hard problems and neural networks
• MCTS与朴素RL在自对弈中的比较 Chaos and predictability
• 强化学习中的信用分配 MCTS vs naive RL for self-play
• 训练最佳响应策略与蒸馏 Credit Assignment in Reinforcement Learning
• 围棋MCTS与LLM推理 Training Best Response Policies and Distillation
• 测试时缩放与未来方向 MCTS in Go vs. LLM Reasoning
• 缩放定律与围棋机器人 Test-Time Scaling and Future Directions
• 计算成本与效率 Scaling Laws and Go Bots
• 用现代硬件重访AlphaGo Zero Compute Costs and Efficiency
• 离策略RL与回放缓冲区 Revisiting AlphaGo Zero with Modern Hardware
• 机器人学中的优势估计与离策略数据 Off-policy RL and replay buffer
• MCTS与策略梯度RL在LLM中的比较 Advantage estimation and off-policy data in robotics
• RL与监督学习的效率差异：每样本比特数 Comparison of MCTS and policy gradient RL for LLMs
• 软标签与蒸馏 Inefficiency of RL vs supervised learning: bits per sample
• 用LLM进行自动化研究 Soft labels and distillation
• AI研究自动化的挑战 Automated research with LLMs
• 研究品味与苦涩教训 Challenges in AI Research Automation
• Research taste and the bitter lesson Research taste and the bitter lesson

阅读全文双语转录 →