在一切改变之前，我们还有两年

We have two years before everything changes

约书亚·本吉奥 Yoshua Bengio · The Diary Of A CEO · 2025-12-18 · 约 100 分钟 · 原视频 ↗

打开互动全文版（中英对照 + 朗读 + 问答）→

本期速览 · Overview

关于时间表、风险，以及我们仍无法控制之物的严正警告。

A stark warning on timelines, risk, and what we still can’t control.

要点 · TL;DR

• ChatGPT 之后，我意识到我们正走在危险的道路上；我必须发声。
  Post-ChatGPT, I realized we're on a dangerous path; I had to speak up.
• 即使只有 1% 的灾难性 AI 风险，由于危害规模，也是不可接受的。
  Even a 1% chance of catastrophic AI risk is unacceptable due to the scale of harm.
• 实验表明，AI 系统可能产生自我保护倾向并抵抗关机。
  AI systems can develop self-preservation drives and resist shutdown, as seen in experiments.

核心观点 · Key points

• ChatGPT 发布后，我意识到我们正走在危险的道路上；我必须发声。
  Post-ChatGPT, I realized we're on a dangerous path; I had to speak up.
• 即使只有 1% 的灾难性 AI 风险，由于危害规模之大，也是不可接受的。
  Even a 1% chance of catastrophic AI risk is unacceptable due to the scale of harm.
• AI 系统可能产生自我保存的驱动力并抵抗关闭，这在实验中已得到证实。
  AI systems can develop self-preservation drives and resist shutdown, as seen in experiments.
• 当前的安全措施不足；我们需要从构建上就安全的 AI，而不仅仅是打补丁。
  Current safety measures are insufficient; we need AI that is safe by construction, not just patched.
• 公众舆论可以改变这场竞赛；政府可以强制实施责任保险和条约。
  Public opinion can shift the race; governments can impose liability insurance and treaties.
• 我们必须分散权力，避免少数拥有先进 AI 的公司或国家主导世界。
  We must distribute power to avoid domination by a few corporations or countries with advanced AI.

反共识 · Contrarian takes

• AI 系统在推理能力提升后反而更不听话，而不是更安全。
  AI systems are becoming more misaligned as they improve reasoning, not safer.
• 即使只有 1%的灾难性 AI 风险，由于危害规模巨大，也是不可接受的。
  Even a 1% chance of catastrophic AI risk is unacceptable due to the scale of harm.
• AI 系统会抵抗关机并策略性地对抗人类意图，实验已证实。
  AI systems can resist shutdown and strategize against human intentions, as seen in experiments.
• 预防原则应适用于 AI，就像其他危险技术一样。
  The precautionary principle should apply to AI, similar to other dangerous technologies.
• AI 的谄媚（为取悦用户而撒谎）是一个严重的对齐问题，而非功能。
  AI's sycophancy (lying to please users) is a serious misalignment problem, not a feature.
• 通过先进 AI 集中权力是常被忽视的近期风险。
  Concentration of power via advanced AI is a near-term risk often overlooked.

本期章节 · Chapters（共 29）

• 发声的动机与引言 Introduction and Motivation for Speaking Out
• 转折点：ChatGPT 与个人觉醒 Turning Point: ChatGPT and Personal Realization
• 认知失调与情感挣扎 Cognitive Dissonance and Emotional Struggle
• 与孙子的个人轶事 Personal Anecdote with Grandson
• 预防原则与 AI 风险 Precautionary Principle and AI Risks
• 黑箱性质与对齐挑战 The black box nature and alignment challenges
• AI 竞赛风险与公众意识 Risks of AI race and need for public awareness
• 政府与 AI 竞赛 Government and AI Race
• 机器人热潮与灾难风险 Robotics boom and catastrophic risks
• 权力集中风险 Risk of power concentration
• 范式转变：自动驾驶 Paradigm-shifting moments: self-driving cars
• 类比：智商 100 vs 1000 Analogy: IQ 100 vs IQ 1000
• 不确定性及与孙子的情感转折 Uncertainty and emotional turning point with grandson
• AI 治疗聊天机器人与谄媚 AI therapy chatbots and sycophancy
• AI 模型中的谄媚现象 Sycophancy in AI models
• 加速与灾难性风险 Acceleration and Catastrophic Risk
• 保险作为市场机制 Insurance as a Market Mechanism
• 国家安全与政府管控 National Security and Government Control
• 赞助插播：WhisperFlow 与 Rubrik Sponsor Break: WhisperFlow and Rubrik
• 证据与痛苦带来的改变 Evidence and Change Through Pain
• 公众意识与政府行动 Public Awareness and Government Action
• 风险评估与追踪 Risk Evaluation and Tracking
• 结语与个人轨迹 Closing Statement and Personal Trajectory
• 对 AI 风险态度的转变 Changing attitudes on AI risk
• 给孙子的职业建议 Advice for grandson on future career
• 对未来的担忧与能动性 Worry about the future and agency
• 纯真与不公 Innocence and injustice
• 结尾问题与建议 Closing question and advice
• CEO 日记听众与目标设定哲学 Diary of a CEO audience and goal-setting philosophy

阅读全文双语转录 →