从语言模型到物理智能体：机器人基础模型的前沿

From Language Models to Physical Agents: The Frontier of Robot Foundation Models

谢尔盖·莱文 Sergey Levine · AI Agent Frontier · 2026-03-29 · 约 60 分钟 · 原视频 ↗

打开互动全文版（中英对照 + 朗读 + 问答）→

本期速览 · Overview

Sergey Levine 探讨了构建能与物理世界交互的 AI 智能体的挑战与机遇，从语言模型迈向能执行清洁、烹饪、组装等真实世界任务的机器人。

Sergey Levine discusses the challenge and opportunity of building AI agents that can interact with the physical world, moving beyond language models to robots that can perform real-world tasks like cleaning, cooking, and assembly.

要点 · TL;DR

• 机器人基础模型需要多样化预训练和强化学习后训练以实现鲁棒性。
  Robot foundation models need diverse pre-training and RL post-training for robustness.
• 通过汇集多个实验室的数据，通用机器人模型优于专用模型。
  Generalist robot models outperform specialists by pooling data from many labs.
• VLA 中的测试时推理提升了性能，并实现了从网络数据的迁移。
  Test-time reasoning in VLAs improves performance and enables transfer from web data.

核心观点 · Key points

• 物理世界交互对 AI 学习超越人类知识至关重要。
  Physical world interaction is crucial for AI to learn beyond human knowledge.
• 机器人基础模型需要多样化数据预训练和鲁棒性后训练。
  Robotic foundation models require pre-training on diverse data and post-training for robustness.
• 第二代 VLA 添加专用动作解码器实现连续运动控制。
  Second-generation VLAs add a dedicated action decoder for continuous motor control.
• VLA 中的测试时推理提升性能并促进网络数据迁移。
  Test-time reasoning in VLAs improves performance and transfer from web data.
• 汇集多实验室机器人数据得到的通用模型优于专用模型。
  Pooling robot data from many labs yields generalist models outperforming specialists.
• 强化学习后训练显著提升鲁棒性和速度。
  Reinforcement learning post-training significantly boosts robustness and speed.

反共识 · Contrarian takes

• 通用机器人模型在专用领域也能超越专用模型。
  Generalist robotic models can outperform specialists even in specialist domains.
• 仅高质量后训练数据若无多样化预训练则无法应对恢复。
  High-quality post-training data alone fails without diverse pre-training for recoveries.
• 自主机器人性能在复杂任务中可超越人类指导。
  Autonomous robot performance can exceed human-directed performance in complex tasks.
• 仅 3%移动机器人训练数据足以产生有效策略。
  Only 3% of training data from mobile robots suffices for effective mobile robot policies.
• 触觉传感因与视觉冗余在大规模中可能非关键。
  Touch sensing may not be critical at scale due to redundancy with vision.
• 机器人安全更多依赖预训练常识而非约束。
  Safety in robotics relies more on common sense from pre-training than on constraints.

本期章节 · Chapters（共 28）

• 引言与动机 Introduction and Motivation
• 机器人 vs 计算机智能体 Robotics vs. Computer Agents
• 端到端机器人学习的挑战 Challenges in End-to-End Robotic Learning
• 机器人基础模型 Foundation Models for Robotics
• 类比基础模型训练 Analogy to Foundation Model Training
• 后训练阶段与机器人基础模型 Post-training phase and robotic foundation models
• 三大挑战及应对方法 Three big challenges and approaches
• 现代机器人基础模型架构 Architectures of modern robotic foundation models
• 第一代视觉-语言-动作模型 First-generation vision-language-action models
• 第二代 VLA 与虚拟运动皮层 Second-generation VLAs with virtual motor cortex
• 动作解码器架构 Action Decoder Architecture
• Pi Zero 模型演示 Pi Zero Model Demonstration
• 完整洗衣任务表现 Full Laundry Task Performance
• 测试时计算与推理 VLA Test Time Compute and Reasoning VLAs
• 具身思维链 Embodied Chain of Thought
• 思考带来的性能提升 Performance Improvement with Thinking
• 更丰富模态的推理 Reasoning with Richer Modalities
• 可操控的视觉-语言-动作模型 Steerable Vision-Language-Action Models
• 数据异质性与通用模型优势 Data heterogeneity and generalist model outperformance
• 预训练与后训练阶段 Pre-training and post-training phases
• 数据池化与泛化缩放 Data pooling and generalization scaling
• 用强化学习缩小鲁棒性差距 Closing the robustness gap with RL
• 强化学习制作咖啡任务 Coffee Making Task with RL
• 总结与动机 Summary and Motivation
• 机器人触觉 Touch in Robotics
• VLA 推理与多智能体协调 Reasoning in VLAs and Multi-Agent Coordination
• 机器人安全与数据质量 Safety and Data Quality in Robotics
• 结束语 Closing Remarks

阅读全文双语转录 →