Tulu 3：大语言模型开源后训练技术

Tulu 3: Open Source Post-Training for LLMs

内森·兰伯特 Nathan Lambert · The Cognitive Revolution · 2024-11-21 · 约 110 分钟 · 原视频 ↗

本期速览 · Overview

Nathan Lambert 讨论 Tulu 3，一个使用相同基础模型匹配 Meta 后训练性能的开源项目，涵盖 SFT、RLHF 和 RLVR 技术。

Nathan Lambert discusses Tulu 3, an open-source project that matches Meta's post-training performance using the same base model, covering SFT, RLHF, and RLVR techniques.

要点 · TL;DR

数据质量在后期训练中比算法选择带来 10 倍提升。
Data quality yields 10x improvement over algorithm choice in post-training.
开源后期训练只需 10-15 人即可媲美前沿模型。
Open-source post-training can match frontier models with 10-15 people.
基于可验证奖励的强化学习会涌现链式思考行为。
RL with verifiable rewards causes emergent chain-of-thought behaviors.

核心观点 · Key points

数据质量比算法选择更重要；更好的数据能带来 10 倍的改进，而算法仅能带来 1-2%。
Data quality matters more than algorithm choice; better data yields 10x improvement vs 1-2% from algorithms.
后训练包含三个阶段：监督微调、直接偏好优化和基于可验证奖励的强化学习，每个阶段都增加价值。
Post-training has three stages: SFT, DPO, and RL with verifiable rewards, each adding value.
在策略偏好数据（由当前模型生成）在直接偏好优化中优于离策略数据。
On-policy preference data (generated by current model) outperforms off-policy data for DPO.
基于可验证奖励的强化学习可以显著提升数学分数，而不会降低其他能力。
RL with verifiable rewards can boost math scores significantly without degrading other capabilities.
开源后训练可以通过 10-15 人和精心数据整理，匹配前沿模型如 Llama 3.1。
Open-source post-training can match frontier models like Llama 3.1 with 10-15 people and careful data curation.

反共识 · Contrarian takes

大语言模型作为评判的偏好数据在下游评估分数上通常优于人类数据，尽管存在高偏差。
LLM-as-judge preference data often outperforms human data in downstream eval scores, despite high bias.
基于可验证奖励的强化学习可以引发类似 o1 的思维链行为，如自我检查。
RL for verifiable rewards can cause emergent chain-of-thought behaviors like self-checking, similar to o1.
监督微调提供了 90%的性能；直接偏好优化和强化学习仅增加最后 10%，但对角色和推理至关重要。
SFT provides 90% of performance; DPO and RL add only the last 10%, but are crucial for character and reasoning.
尽管推理轨迹封闭，但通过合成数据和强化学习，开源类似 o1 的模型是可行的。
Open-source o1-like models are feasible despite closed reasoning traces, using synthetic data and RL.
模型的角色和一致性缺乏评估；前沿实验室优先考虑聊天竞技场分数而非学术指标。
Character and consistency in models lack evals; frontier labs prioritize chat arena scores over academic metrics.

本期章节 · Chapters（共 33）

开场与赞助 Introduction and Sponsor
欢迎与嘉宾介绍 Welcome and Guest Introduction
介绍与艾伦研究所背景 Introduction and Allen Institute background
后训练与开放模型格局变化 Shifting post-training and open model landscape
介绍与赞助商 Introduction and Sponsors
后训练基础理解 Baseline Understanding of Post-Training
后训练中开放与封闭配方 Open vs Closed Recipes in Post-Training
训练流程概览：SFT、DPO、RL Training Pipeline Overview: SFT, DPO, RL
计算成本对比：SFT vs DPO vs RL Compute cost comparison: SFT vs DPO vs RL
赞助商插播：甲骨文云基础设施 Sponsor break: Oracle Cloud Infrastructure
评估改进与后训练设置 Eval improvement and post-training setup
数据污染与评估挑战 Data contamination and evaluation challenges
后训练直觉：指令调优与偏好调优 Post-training intuition: instruction tuning and preference tuning
后训练中 RL 与 SFT/DPO 对比 RL vs SFT/DPO in post-training
多 token 评估与奖励模型 Multi-token evaluation and reward model
PPO 中 token 级价值真值 Ground truth for token-level value in PPO
token 价值直觉 Intuition on token values
与推理时计算的联系 Connection to inference-time compute
DPO 与 PPO 对比 DPO vs PPO comparison
微调的程序性理解 Procedural understanding of fine-tuning
项目管理与团队动态 Project management and team dynamics
开发流程与团队动态 Development process and team dynamics
合成数据与 LLM 作为裁判 Synthetic data and LLM as judge
计算资源与补偿 Compute resources and compensation
LLM 作为裁判与合成数据 LLM as a judge and synthetic data
基于可验证奖励的强化学习 Reinforcement learning from verifiable rewards
RL 训练中的涌现推理行为 Emergent reasoning behavior in RL training
无监督微调下的纯粹涌现 Purely emergent without supervised fine-tuning
与 grokking 和相变的联系 Connection to grokking and phase changes
无意中观察到的行为 Observing the behavior unintentionally
理想化的最终复制流程 Idealized final process for replication
微调策略与饱和极限 Fine-tuning strategies and saturation limits
o1 对开放与封闭模型的影响 Implications of o1 for Open vs Closed Models

阅读全文双语转录 →