OpenAI 安全委员会主席 Ziko Culture 解释董事会如何监督模型发布、为什么更大的模型并不更安全，以及 AI 系统的简单性。
Ziko Culture, chair of OpenAI's safety committee, explains how the board oversees model releases, why bigger models aren't safer, and the simplicity of AI systems.
要点 · TL;DR
安全需要明确的训练和额外层，而不仅仅是扩展模型。 Safety requires explicit training and layers, not just scaling models.
AI 安全涵盖四类：错误、有害使用、社会影响和失控。 AI safety spans four categories: mistakes, harmful use, societal effects, loss of control.
智能体安全引入了新的漏洞，如来自第三方数据的提示注入。 Agent security introduces new vulnerabilities like prompt injection from third-party data.
核心观点 · Key points
安全性需要明确的训练和额外防护层，不能仅靠扩大模型规模。 Safety requires explicit training and additional layers, not just scaling models.
AI 安全涵盖四类：错误、有害使用、社会影响和失控。 AI safety spans four categories: mistakes, harmful use, societal effects, and loss of control.
现代 AI 安全使用多层防御：输入分类器、安全训练、输出分类器和运营安全。 Modern AI security uses multiple defense layers: input classifiers, safety training, output classifiers, and operational security.
智能体安全引入了新漏洞，如来自第三方数据的提示注入。 Agent security introduces new vulnerabilities like prompt injection from third-party data.
强化学习让模型基于自身输出进行训练，从而实现自我改进。 Reinforcement learning trains models on their own outputs, enabling self-improvement.
LLM 的核心代码仅 200-300 行；复杂性来自数据和规模扩展。 The core code for an LLM is only 200-300 lines; complexity comes from data and scaling.
反共识 · Contrarian takes
更大的模型不会自动变得更安全；安全性需要明确的努力。 Bigger models do not automatically become safer; safety requires explicit effort.
在开源模型上优化的越狱字符串可迁移到商业模型。 Jailbreak strings optimized on open-source models transfer to commercial models.
机械可解释性可能最终通过 AI 智能体进行自动化研究而成功。 Mechanistic interpretability may finally succeed via AI agents doing automated research.
Transformer 架构并非必需；其他架构也能达到类似效果。 Transformer architecture is not essential; other architectures could have achieved similar results.
加速主义 vs. 末日论标签具有贬义；大多数研究者持有细致观点。 Accelerationist vs. doomer labels are dismissive; most researchers hold nuanced views.
在适当的护栏和沙箱保护下，智能体已准备好投入生产。 Agents are ready for production with proper guardrails and sandboxing.
本期章节 · Chapters（共 34）
引言Introduction
OpenAI 角色与安全委员会Role at OpenAI and Safety Committee
OpenAI 内部安全团队Internal Safety Teams at OpenAI
大模型不会自动更安全Bigger Models Don't Get Safer Automatically
AI 系统的简单性Simplicity of AI Systems
模型发布的准备框架Preparedness Framework for Model Release
安全与能力的进展速度Pace of Progress in Safety vs Capabilities
智能体红队测试：能力与漏洞Agent Red Teaming and Capability vs Vulnerability
安全需要超越规模的显式工作Safety requires explicit work beyond scaling
安全问题的四大风险类别Sources of safety issues: four categories of risk
加速主义与末日论标签On accelerationist vs doomer labels
AI 风险与收益的研究共识Researcher consensus on AI risks and benefits
早期机器学习之旅Early journey into machine learning
深度学习的前夜At the cusp of deep learning
2015 年访问 OpenAIVisiting OpenAI in 2015
CMU 在 AI 与产业拉动中的角色CMU's role in AI and industry pull
学术界的冒险精神与 CMURisk-taking in academia and CMU's role
Grace One 创业与 AI 安全Grace One startup and AI safety
安全与安保的区别Difference between safety and security
AI 安全简介Introduction to AI Security
GCG 论文与越狱The GCG Paper and Jailbreaks
开源与商业模型的越狱Jailbreaking open-source vs. commercial models
AI 智能体安全基础AI Agent Security Basics
智能体准备好投入生产了吗？Are Agents Ready for Production?
机制可解释性与 AI 安全Mechanistic Interpretability and AI Safety
机制可解释性与自动化研究Mechanistic Interpretability and Automated Research
安全与安保展望Safety and Security Outlook
前沿 AI 研究与强化学习Frontier AI Research and RL
未来突破与持续学习Future Breakthroughs and Continual Learning