Aakanksha Chowdhery discusses why pre-training needs a fundamental rethink for agentic tasks, moving beyond static benchmarks to interactive environments.
要点 · TL;DR
预训练必须从根本上重新思考，才能实现智能体 AI，而不仅仅是后训练。 Pre-training must be fundamentally rethought for agentic AI, not just post-training.
先扩展模型再蒸馏，是小实验室的可行策略。 Scaling models first, then distilling, is a viable strategy for small labs.
核心观点 · Key points
预训练必须从根本上重新思考，才能实现智能体能力，而不仅仅是后训练。 Pre-training must be rethought from fundamentals to achieve agentic capabilities, not just post-training.
智能体任务要求模型具备规划、长上下文推理、从失败中恢复以及学习新工具的能力。 Agentic tasks require models to plan, reason over long context, recover from failures, and learn new tools.
当前的静态基准测试不足；需要衡量智能体能力的新基准。 Current static benchmarks are insufficient; new benchmarks measuring agentic capabilities are needed.
Scaling（规模扩张）模型是发现涌现能力的关键，这些能力随后可以蒸馏到更小的模型中。 Scaling models is key to discovering emergent capabilities, which can later be distilled into smaller models.
高质量精选训练数据和超越下一个词预测的损失目标对推理至关重要。 High-quality curated training data and loss objectives beyond next-token prediction are crucial for reasoning.
反共识 · Contrarian takes
仅靠后训练是有限的；必须重新思考预训练才能释放完整的智能体能力。 Post-training alone is limiting; pre-training must be rethought to unlock full agentic capabilities.
Transformer仍然是智能体的最佳架构；激进的架构变革在Scaling（规模扩张）中风险很高。 Transformers remain the best architecture for agents; radical architectural changes are risky at scale.
合成数据如果保持自然数据分布则有益；纯合成数据会损害性能。 Synthetic data can help if it preserves natural data distribution; pure synthetic data harms performance.
幻觉不是根本缺陷，而是概率性输出；可通过推理提高可恢复性。 Hallucination is not a fundamental flaw but a probabilistic output; recoverability can be improved via reasoning.
工具使用应通过探索和元学习获得，而不仅仅是上下文工程或微调。 Tool use should be learned through exploration and meta-learning, not just context engineering or fine-tuning.
本期章节 · Chapters（共 23）
重新思考智能体任务的预训练Rethinking pre-training for agentic tasks
重新思考智能体能力的预训练Rethinking Pre-training for Agentic Capabilities
预训练的渐进与激进变革Incremental vs. Radical Changes in Pre-training
智能体系统的注意力进化Evolving Attention for Agentic Systems
超越注意力的记忆工具Memory as a Tool Beyond Attention
注意力研究的启发Inspirational Research on Attention
调试大规模模型与严格评估Debugging scaled models and rigorous evaluation
调整损失函数提升智能体性能Loss tweaks for agent performance
智能体训练数据的演变Evolution of training data for agents
合成数据生成与扩展Synthetic Data Generation and Scaling
合成数据与模型崩溃Synthetic Data and Model Collapse
测量与基准构建Measurement and Benchmark Construction
社区与实验室特定测量Community vs. Lab-Specific Measurement
强化学习与基准Reinforcement Learning and Benchmarks
模型扩展与能力Model Scaling and Capabilities
小实验室策略：先扩展后蒸馏Small Lab Strategy: Scale First, Then Distill
长程推理：定义与示例Long-Form Reasoning: Definition and Examples
长程推理与失败恢复Long-form reasoning and failure recovery
学习新工具与MCP上下文膨胀Learning new tools and MCP context blow-up