Petera Beal discusses the feasibility of a robot beating Roger Federer at tennis, the impressive physical abilities of Boston Dynamics robots, and the psychological connection humans form with robots.
要点 · TL;DR
自我对弈通过成功与失败的对比为机器人提供强大的学习信号。 Self-play provides strong learning signal for robots via success/failure contrast.
模仿学习以高信噪比快速教授机器人基本技能。 Imitation learning offers high signal-to-noise ratio for teaching basic skills quickly.
分层推理对于长期信用分配是必要的，但当前强化学习缺乏这一点。 Hierarchical reasoning is needed for long-horizon credit assignment but current RL lacks it.
核心观点 · Key points
要让机器人在网球上击败费德勒，硬件和软件都需要重大突破。 Hardware and software both need major advances for a robot to beat Federer at tennis.
自我对弈能提供强大的学习信号，因为总有一方成功、一方失败。 Self-play provides strong learning signal because one side always succeeds and the other fails.
模仿学习能提供高信噪比的数据，让机器人快速学会基本技能。 Imitation learning offers high signal-to-noise ratio for teaching robots basic skills quickly.
通过微调大型预训练模型进行迁移学习是 AI 领域的一大成功。 Transfer learning via fine-tuning large pre-trained models is a major success in AI.
长时域信用分配需要分层推理，但当前的强化学习尚不具备。 Hierarchical reasoning is needed for long-horizon credit assignment, but current RL lacks it.
反共识 · Contrarian takes
让机器人挥动网球拍通过强化学习和模拟是可行的，并非极其困难。 A robot swinging a tennis racket is feasible with RL and simulation, not extremely hard.
即使知道机器人是预设脚本或无学习能力，人们仍会自然地将它们拟人化。 People naturally anthropomorphize robots even knowing they are scripted or non-learning.
深度强化学习的成功部分源于分段线性控制，而非仅仅是深度学习的魔力。 Deep RL's success partly stems from piecewise linear control, not just deep learning magic.
模拟器集成可能比单个精确模拟器更适合向真实世界迁移。 Simulation ensembles may work better than a single precise simulator for real-world transfer.
机器人可以通过强化学习学会爱人类，就像狗建立情感一样。 A robot could learn to love humans through RL, similar to how dogs form affection.
本期章节 · Chapters（共 16）
0. 引言与网球机器人问题Introduction and Tennis Robot Question
1. 最令人印象深刻的机器人及交互心理学Most Impressive Robot and Psychology of Interaction
2. 为涌现行为形式化目标Formalizing objectives for emergent behavior
3. 稀疏奖励下 RL 为何有效Why RL works despite sparse rewards
4. 现实世界强化学习的挑战Challenges of Real-World Reinforcement Learning