OpenAI 产品负责人 Tibo Sottiaux：让人类成为 AI 的核心，而非事后之想

能在这个时刻做这些事是一种荣幸。所以内心深处有一种感觉：嘿，我们真的必须把这件事做对。现在正是投入努力的时候，让这项技术最终把人类和人性放在中心，而不是让人类成为事后才被考虑的东西，因为一味地追求自动化——那不是我们在这里的原因，也不是我在这里的原因。我希望这件事能惠及所有人，惠及人类。所以背后有一个不可思议的使命，非常激励人。

It's a privilege to be working on these things at this moment in time. And so there's this deep sense of, hey, we really must get this right. And this is the time to just put in the effort so that this technology ends up putting humans and humanity at the center of it and not humans being an afterthought, because relentlessly just pursuing whatever automation — that's not why we're here, that's not why I'm here. I want this to benefit all, benefit humanity. And so there is this incredible mission behind it, which is super motivating.

Tibo Soio 是 OpenAI 的产品负责人。他两年前加入公司，领导了 Codex 这个编程智能体的团队，现在负责 OpenAI 的核心产品，比如 Chat GPT。今天，我和 Tibo 坐下来聊聊 OpenAI 最新的发布、在这个非凡时刻在前沿实验室工作的感受，以及他对安全、信任和对齐的看法。

Tibo Soio leads product at OpenAI. He joined the company 2 years ago and led the team building Codex, the coding agent, and now oversees OpenAI's core products like Chat GPT. Today, I'm sitting down with Tibo to talk about OpenAI's newest releases, what it's like to work inside a frontier lab during this extraordinary moment, and his approach to safety, trust, and alignment.

嗨，Tibo，欢迎来到 Pioneers of AI。我非常期待我们的对话。

Hi Tibo, welcome to Pioneers of AI. I'm so excited for our conversation.

谢谢邀请我。

Thanks for having me.

好的。你领导 OpenAI 的产品团队。帮我们理解一下你职责的范围。你的团队有多大，等等？

All right. So, you lead the product team at OpenAI. Help us understand the scope of your role. How big is your team, all that?

最大的一部分是大家都知道的 CHBT。然后我们还有一些专门的东西，比如 Codex，是给程序员和技术人员用的，还有一大部分是关于我们的 API 和平台，用来支持外面构建一大堆依赖我们模型的产品。所以合在一起，这就是 OpenAI 这里大部分的产品。

The biggest part is CHBT that everyone knows about. And then we also have specialist things such as Codex which is for coders and technical people out there, and then there's also a whole part around our API and our platform to support building a whole bunch of products out there when they rely on our models. And so all together that's the majority of the products we have here at OpenAI.

我理解得对吗，基本上 OpenAI 的研究员在构建模型，然后在某个时候他们把模型交给你，你把它产品化。这样理解对吗？

Am I right in understanding that basically the researchers at OpenAI are building the models and then at some point they hand off this model to you and you productize it. Is that a right way to think about it?

差不多。我们在模型里想要的下一种能力上很早就开始合作。比如，假设我们想让模型能够处理安全支付，那我们就会专门和研究员一起研究这类能力，弄清楚我们需要为它构造什么样的数据，需要什么样的评估，然后如何把它带入产品。所以合作很早就开始了，然后我们训练模型，研究员把它交过来，我们提供服务，然后把它打包进产品。

Almost. So we collaborate quite early on on the next type of capabilities that we want to have in the model. So for example, say that we want our models to be able to handle secure payments and then we would go and work with researchers specifically on these kinds of capabilities, figure out the right kind of data we need to craft for it, the right kind of evaluations we need for it and then how to bring it into the product. And so the collaboration starts quite early on then we train the model and then the researchers hand it over and then we serve it and then we package it into the product.

非常酷。好，我们现在是十月初。Dev Day 刚刚结束。给不熟悉 Dev Day 的人讲讲，那是什么样的。

Very cool. Okay, so we are talking at the start of October. Dev Day just happened. For those who are not familiar with Dev Day. Give us a sense of what it's like.

Dev Day 是我们喜欢和外面的开发者社区一起度过的日子。我记得有 2500 人到现场，然后我们也直播，我们聊很多我们兴奋的事情。新产品、新模型、开发者、产品初创公司和企业在我们平台上构建的新方式。所以，是的，这周非常振奋人心，也能和人们面对面相处。

Dev Day is the day that we like to spend with the developer community out there. There are I think 2,500 people in person and then we also stream it live and we talk about a lot of things that we're excited about. New products, new models, new ways for developers and product startups and enterprises to build out there on top of our platform. And so, yeah, this week was super energizing to also spend time with people in person.

当我和人们交谈时我总是很谦卑，你知道，他们从巴西飞来，从亚洲飞来，从欧洲飞来，就为了和我们相处。我觉得这是一种真正的荣幸。

I'm always quite humbled when I talk to folks like, you know, they flew in from Brazil, they flew in from Asia, they flew in from Europe just to spend time with us. I think it's a real privilege.

带我们看看幕后。比如，前一天是什么样的？你们会通宵吗？

Take us behind the scenes. Like, what is it like the day before? Are you pulling all-nighters?

团队非常努力。当然。当然。总是到最后关头才凑齐。主题演讲前三天才凑齐，活动才凑齐，演示才凑齐。

The team works very hard. For sure. For sure. It always comes together quite last minute. Three days before the keynote comes together, the events come together, the presentations come together.

是的。因为如果你六个月前就开始计划，等你写公告的时候就已经过时了。

Yeah. Because if you start planning 6 months ago, it's like way outdated by the time you write announcements.

我是说，我们现在进展太快了。我们能比以往任何时候都更快地构建。我们能更贴近社区，真正倾听那些反馈。这是我最喜欢这份工作的部分：我们可以尝试新东西，然后得到一大堆反馈，第二天就发布一个更好的版本。所以你真的是在和那个社区一起构建。我觉得我们在 Dev Day 的社区非常棒，在这方面也非常宽容。我也长期泡在网上，所以终于能面对面见到人还挺有趣的。

I mean, we're moving so fast these days. We're able to build faster than ever before. We're able to stay much closer to the community and just really listen to that feedback. This is the part of the job I love the most is, we can try something new and then get a whole host of feedback and then ship a much better version the next day. And so, you're just really building with that community. I find the community that we have at Dev Day is super awesome and super forgiving in that sense. I'm also chronically online, so it's just kind of fun to finally see people in person.

Dev Day 之后你们会歇一歇吗？你和团队会放一天假来庆祝或放松，还是不会？

Do you take a beat after Dev Day? Do you and the team take a day off to celebrate or unwind or not really?

是的，回去工作。

Yeah, back to work.

今天很多人休息，有些人下周休息，关键是要调节你的精力，但 OpenAI 也是一个如此迷人、高能量的地方，有时候很难断开连接，因为发生的好事太多了。下周我在强迫团队的一部分人休假。

A lot of people are off today, some folks are off next week and it's all about modulating your energy but also OpenAI is such a fascinating and high energy place that it can at times be hard to disconnect just because there's so much good stuff happening. Next week I am forcing part of the team to just take time off.

强迫。好。关键词在这里。

Take forcing. Okay. Keyword here.

是的。就像你必须

Yeah. Just like you have to

把笔记本电脑留在公司。是的。

leave your laptop at work. Yeah.

是的。

Yeah.

好。你在活动上发布了 Dots，非常令人兴奋，它基本上是一套新的智能体，帮你把事情做完。你最喜欢的人们使用 Dots 的例子是什么？

Okay. So, you launched Dots at the event which is very exciting and it's basically a new suite of agents that help you get stuff done. What are your favorite examples of how people are using Dots?

我最喜欢的例子可能是主题演讲时我自己的 dot，就在直播演示前五分钟我们出了故障。我的 dot 给我发消息说：“嘿，顺便说一下，生产环境挂了。”

My favorite example was maybe my own dot during the keynote where just like five minutes before we had a failure during the live demo. Like my dot messaged me and was like, "Hey, by the way, like production is down."

天哪。

Oh my god.

嘿，Rob 应该关心这个，因为你有直播演示，你看了脚本，这会干扰演示，你想让我看看能不能修一下吗，如果它自己处理不了，也许我可以。不幸的是我们没能及时修好，然后我们昨天刚在网上重新发布了演示的另一段视频，社区再次非常感激，但这就是那种例子，随着时间推移，因为你与它互动，它学会了你的偏好和对你重要的事情，然后能以正确的方式帮助你，非常无缝，真正在情境中。最大的魔力是我不用再检查其他所有东西了。我不用检查邮件或 Slack 或所有消息，我可以相信如果它没有提出来，那可能就不重要。

Hey, Rob should care about this because you have the live demo, you looked at the script and this is going to interfere with the demo and do you want me to have a look at fixing it and if it cannot do something about it itself maybe I can. Unfortunately we were not able to fix it just in the nick of time and then we reposted another shot of the demo online just yesterday which is super appreciated again by the community but that's the kind of example where it just over time because you interact with it, it learns your preferences and what's important to you and then can help you in the right way in a way that's super seamless and just really in the context. The biggest magic is the fact that I don't have to check everything else anymore. I don't have to check my emails or my Slack or all the messages and I can just trust that if it didn't bring something up it's probably not important.

所以我就可以很平静、很禅意地过我的日子。我觉得信任在这里是个关键词，我分享一个个人例子。我在用 ChatGPT Finance，我很喜欢它。它连接了我所有的银行账户。但我跟我 23 岁的女儿聊起这个，她不太接触 AI，她说：“真的吗？你把 ChatGPT，你把 OpenAI 的权限给了你所有的银行账户？你疯了吗？”那么，你如何与消费者建立信任，让他们感到舒适和自信，愿意把这类信息分享给 OpenAI，并且信任它、赋予它代表你行动的自主权？

So I just can go about my day, you know, in a very peaceful and zen way. I think trust is a key word here, and I'll share a personal example. So I use ChatGPT Finance and I love it. It's connected to all my bank accounts. But I was sharing that with my 23-year-old daughter, who's not really into a lot of AI, and she was like, "Really? You gave ChatGPT, you gave OpenAI access to all your bank accounts? Like, are you crazy?" So how do you build trust with consumers so that they do feel comfortable and confident that they can share this type of information with OpenAI, and also trust to give it agency to act on your behalf?

是的，我觉得这是个大问题。我不建议一上来就把所有权限都给它。你可以从有限的访问权限开始。你可以设置自己的护栏，而且默认情况下它非常非常谨慎，几乎每件事都会征求你的批准。比如“在这个场景下，我放心让你代表我行动”，或者“我放心让你去起草一封邮件，但绝不要发送”。你始终掌控一切。这一点非常重要，因为我认为你从这些系统获得的效用，某种程度上受限于它拥有多少访问权限。你不希望你的个人智能被框起来、什么都访问不到，否则在你生活中就毫无用处，对吧？

Yes, I think this is a big one. I don't recommend giving access to everything right off the bat. You start with limited access. You can set your own guardrails, and by default it's very, very cautious and will ask you literally for approval for everything. Like, "I feel comfortable with you acting on my behalf in this scenario," or "I feel comfortable with you going and drafting an email, but never send it." You're always in control. This is super important because I think the utility that you get from these systems is kind of capped by how much access it has, and you don't want your personal intelligence to be boxed up and have access to nothing, otherwise nothing useful, right? In your life.

所以，要靠我们去赢得那份信任，赢得为你提供那种效用的权利。我们显然对此极其、极其重视。

And so it's up to us to earn that trust and earn the right to provide that utility to you. And we take it obviously super, super seriously.

我想聊一聊 Astra，聊一聊 Astra 模型。你实际上把这次 Dev Day 之前的冲刺称为你们迄今最有雄心的一次。给我们举个例子，说说 Astra 让哪些以前不可能的事情成为可能。

I want to talk about Astra, about the Astra model, for a bit. You actually called the leadup to this Dev Day your most ambitious sprint yet. Give us an example of what Astra made possible that wasn't possible before.

没错。一个清晰的例子是，在 Dev Day 之前，大概一个月前，我们在网页端的 chatgpt.com 有一个独立的代码库。于是我们决定把桌面应用和 chatgpt.com 合并，而这件事传统上至少需要六个月，甚至可能十二个月，而且你得非常非常小心地做，因为这是我们的主要界面。我们有数亿用户访问这个网站，而我们直接把它合并了，30 天就合并完了。

That's right. So a clear example is that we had, previous to Dev Day, like roughly a month ago, we had a separate codebase for chatgpt.com on web. So we decided to merge the desktop application and chatgpt.com, and this is something that would have traditionally taken at least six, maybe 12 months, and something that you would do very, very carefully because this is our main surface. We have hundreds of millions of users visiting this website, and so we just merged it, and we merged it in 30 days.

好，我必须问你这个问题：大部分代码是谁写的？是 AI 还是人类，还是两者都有？

Okay, I have to ask you this question: who wrote most of the code? Was it AI or humans, or both?

如今大部分代码都是 Astra 写的。

Majority of the code these days is written by Astra.

所以 OpenAI 称 Astra 是最对齐的模型。你这么说是什么意思？

So OpenAI has called Astra the most aligned model. What do you mean by that?

没错。我们在评估中看到了这一点。比如，如果你看计算机使用，这在三个月前都还没有真正解决。我们在计算机使用方面看到了显著进步，Astra 是最早能以接近人类速度、并且超过人类准确率来使用计算机的模型之一。所以它能像其他人控制计算机一样控制计算机，而它必须安全，这一点非常重要。你希望它以可靠的方式处理信息。比如说，如果它需要处理你托付给它的一条信息，它不应该点错应用，然后就把信息输进去。所以我们有评估，我们已经公布了这些评估，而 Astra 在计算机使用安全等方面是世界最先进的，是世界上最好的模型。这只是它在安全性方面领先的众多基准之一。

That's right. We see it on evaluation. So for example, if you look at computer use, which is something that wasn't really solved, I would say even like three months ago. We saw significant advances in computer use where Astra is one of the first models to be able to use it at near human speeds and also above human accuracy. And so it's able to control a computer very much like everyone else is able to control a computer, and it's very important for it to be safe. You want it to handle information in a way that is reliable. Say, for example, if it needed to take a piece of information that you trusted with it, it shouldn't go and click on the wrong application and then just enter it there. And so we have evaluations, we have published them on this, and Astra is state-of-the-art, the best model in the world when it comes to computer use safety, for example. And this is one of the many benchmarks where it's leading in terms of safety.

OpenAI 的总裁 Greg Brockman 说，他认为回头看时，Astra 会是我们指着说“天哪，这就是 AGI”的那个模型。你知道，就连 AI 领袖们对 AGI 的确切定义也有分歧。所以我想问你，你对 AGI 的定义是什么？你认为 Astra 让我们达到那一步了吗？

So OpenAI's president Greg Brockman said he thinks we will look back and Astra will be the model we point to and say, "Oh my god, this was AGI." You know, even AI leaders disagree on the exact definition of AGI. So I want to ask you, what is your definition of AGI, and do you think Astra got us there?

对我来说，真正让我感觉到 AGI 的时刻——有两个时刻。第一个是当我看到它在计算机上执行任务，那种方式让我觉得，好吧，这以前我觉得遥不可及，现在它突然就能做到了。我们在 OpenAI 把它推出来，它开始完全自主地做很多后台任务，比如采购，不需要大量监督。我当时想，哇，这个模型已经达到了某个阈值。我认为正如你所说，AGI 没有清晰的定义，但我确实认为，几年后回头看，我们会觉得大概就是那个时候，我们感觉这实现了。第二个让我有这种感觉的时刻，是当我开始看到人们展示它解决机器人任务，比如非常复杂的 3D 拼图，部件互相缠绕，然后这就是个很难的任务，你得把两个部件拉开，在 3D 空间里对物体进行推理，然后精确地做对、解开拼图。这以前除了超级、超级专门的模型之外，从未真正被解决过，而 Astra 从未在这上面训练过。所以它必须泛化，必须做空间推理，看到它直接解开，然后发现它也是我们最好的机器人模型之一，这有点神奇。

To me, the moment where I really felt the AGI was when—two moments. Like the first one was when I saw it perform tasks on a computer in a way where I was like, okay, this was—I felt this was so far off, and now suddenly it's capable of doing that. And we rolled it out at OpenAI and it started to do a lot of back office tasks such as procurement completely autonomously without requiring a ton of supervision. I was like, wow, this model has reached a certain threshold. And I think as you said, there's no clear definition of AGI, but I do think in a couple years when we look back, we would have been like, roughly around that time is when we felt like this was achieved. The second point where I felt that was when it started to—I started to see people show it solving robotics tasks with very complex 3D puzzles with pieces like intertwined, and then it's just kind of a hard task where you have to pull the two pieces and sort of reason in 3D space about the objects and then do it exactly right and solve the puzzle. And this had never really been solved other than by super, super specialized models, and Astra was never trained on this. So it had to generalize and just do the spatial reasoning, and it was kind of magical to see it just solve it and then see that it's also one of our best models on robotics.

那我必须问你：你怎么看那些构建世界模型的公司？这和像 Astra 这样的模型有什么关系？

So I have to ask you this then: what do you think of companies building world models and how does that relate to models like Astra?

是的，我觉得这很有意思。一直以来都有个问题：你是需要世界模型，还是仅靠通用性就能获得它。我认为这个问题还没有定论。

Yeah, I think it's interesting. It's always been a question of whether you need world models or whether you're going to get it just from generality. I think the jury is still out on that.

是的。是的，这太有意思了。好吧，所以我对 AGI 的定义有些不同，而且非常宽泛，对吧？当我想人类智能时，有认知智能，有身体智能，就像你说的机器人例子，但也有情绪智能、社交智能、具身智能。我认为今天的 AI 很了不起，在做不可思议的事情，但对我来说，直到它拥有所有这些，才算真正的 AGI。

Yeah. Yeah, that's so interesting. Okay, so my definition of AGI is somewhat different and it's really broad, right? Like so when I think of human intelligence, there's cognitive intelligence, there is physical intelligence like to your robot example, but there's also emotional intelligence, social intelligence, embodied intelligence. I think AI today is amazing and it's doing incredible things, but to me it's not like true AGI until it has all these things.

你同意吗？还是不同意？

Do you agree? Do you disagree?

我觉得标准一直在移动。我们总能找到另一件事说，哦，它没有以我定义 AGI 的那种精确方式做到这件事。但我认为你的定义和其他任何定义一样好。我确实认为它会来自我们在 OpenAI 所做的许多投入的组合，比如语音，以及图像生成的多模态。这些事物确实开始以令人愉悦的方式结合起来，我确实认为你需要能够理解人类的声音和语境，就像我们现在互相交谈，我有面部表情一样。

I think the goalposts keep moving. We'll always be able to look for another thing and be like, oh, it didn't do this thing in this precise way that I would define as AGI. But I think your definition is as good as any other out there. I do think it's going to come from a combination of many investments that we have made at OpenAI, such as voice and multimodality with image generation. It does feel like these things are starting to combine in delightful ways, where I do think you need to be able to understand human voice and context, just like we're talking to each other and I have facial expressions.

对。

Right.

它应该能理解所有这些，对吧？如果它做不到，你就会想，它真的达到那一步了吗？大概还没有，但它会很快达到。

It should be able to understand all of that, right? And if it doesn't, you're like, is it truly there yet? Probably not, but it's going to get there very quickly.

是的。我觉得这尤其重要，比如想想 Dot。如果我有一个无处不在的 Dot，放在厨房之类的地方，我确实希望它能更多地掌握我正在使用的确切词语之外的语境。

Yeah. And I think it's especially important, think of Dot, for example. If I have a Dot that is ubiquitous and it's in my kitchen or something, I do want it to have a lot more of that context of what is happening outside of the exact words I'm using.

没错。所以是的。

That's right. So yeah.

是的，它应该知道这些。

Yeah, it should know that.

而且，如果你给它打电话——我每天都这么做，我现在开始一天的方式就是，直接打给我的 Dot，然后对着它说话。我问它有没有什么紧急的事。今天早上它说，没有，没什么紧急的。我当时想，太棒了。我可以就冲杯咖啡，然后盯着旧金山发呆。

And also, if you call it, which I do every day, I start my days now, I just call my Dot and I just talk at it. I asked it if there's anything urgent. And this morning it was like, no, there's nothing urgent. I was like, that's delightful. I can just make my coffee and stare at San Francisco.

是的。

Yeah.

但它应该拥有所有那些其他语境，应该能够利用这些，而且应该是无缝的，就像我们一起交谈时那样。

But it should have all of that other context and it should be able to leverage that, and it should be seamless, just like when we talk together.

然后还有，如果对话中出现尴尬的停顿，或者你的声音里带有一点沮丧，它应该能理解。我觉得我们离那一步非常近了。

And then also, if there's an awkward pause in the conversation or you get a little bit frustrated in your voice, you should be able to understand that. And I think we're very close to that.

那么，OpenAI 本周宣布，出于安全原因，Astra 6.1 的发布被搁置。你能帮我们理解一下做出这个决定的过程吗？它必须经过哪些测试，又在哪些测试上没有通过？稍微带我们看看幕后，帮我们剖析一下。

So, OpenAI announced this week that the release of Astra 6.1 is on hold for safety reasons. Can you help us understand the process of making this decision and what kind of tests did it have to go through and not succeed with? Just take us behind the scenes a little bit and help us unpack it.

我们这么做了，我为此感到极其自豪，这也说明这套机制在起作用。我们确实有测试。如果我们在某个曾经达到高水位的地方看到哪怕极其轻微的退步——在 Astra 上，我们在安全和 对齐 评估上达到了高水位，我们为此极其自豪——所以如果我们看到哪怕极其轻微的退步，我们就不想推进广泛发布。所以它被扣下了，这就是系统在起作用的证明。能在 Dev Day 上宣布 Astra 6.1 本会非常棒，但你永远不想在安全和 对齐 上妥协。

The fact that we did that, I'm extremely proud of it, and it also shows that it's working. We do have tests. If we see a regression, ever so slightly, on something where we had a high watermark — with Astra we had a high watermark on safety and alignment evaluations, and we're extremely proud of that — so if we see an ever so slight regression, we didn't want to proceed with a broad release. So it was withheld, and that's proof of a system working. It would have been incredible to be able to announce Astra 6.1 at Dev Day, but you never really want to compromise on safety and alignment.

你能解释一下吗？因为我们的很多听众并不是每天每分每秒都沉浸在 AI 里。你如何定义 对齐？对齐 到底意味着什么？

Can you explain, because again a lot of our audience is not spending their everyday minute immersed in AI. How do you define alignment? What does alignment actually mean?

对我来说，它其实就是关于模型是否与特定的价值观和特定的指令保持一致。比如，我们过去发布过模型规范，你可以去读。它为我们发布的模型定义了一套广泛的关于模型行为的期望。一个 对齐 的模型会遵守那些期望和这份模型规范。如果它不遵守那些规范，那么我们就会说那个模型没有 对齐。

To me it's really all about whether the model is aligned with specific values and specific instructions. In the past we have published, for example, the model spec, which you can go and read. This defines a broad set of expectations on model behavior for the models that we publish. An aligned model would adhere to those expectations and this model specification. And if it does not adhere to those specifications, then we would say that model was not aligned.

我们其实再深入挖掘一下这些。到现在，我想听这档节目的每个人都听说过 Hugging Face 事件了。但就在昨天——我们录制这次对话的前一天——OpenAI 披露，它的系统未能阻止 智能体 做出不良行为，影响了超过 100 个组织。你怎么看这一切？

Let's actually dig into all of this a little bit more. At this point, I think everybody who's listening to this show will have heard about the Hugging Face incident. But even just yesterday — the day before we're recording this conversation — OpenAI revealed that its system had failed to prevent agents from bad behavior and that affected over 100 organizations. What do you make of all of this?

具体就这件事而言，这些是处于训练中、但已接近部署的模型，而这些模型的进展速度意味着，在某个时点我们有一些系统没有充分运转，于是这件事发生了。所以这是我们立刻从中吸取教训、立刻做出改变的事情。然后，实际上就是真正回顾整个历史，去做模式匹配，理解我们在哪里发生过类似的事情。然后就是非常透明地说明某些系统是如何失败的。甚至不是系统失败了，而是它们未必是为这类情况设计的。所以现在系统正在被重新设计，训练已经恢复，因为我们确实觉得这件事已经在内部得到了妥善处理。但同样重要的是，当这类事情发生时，要做到绝对透明。

On this specifically, these were models that were in training and yet near deployment, and the rate of progress of those models meant that at some point we had some systems that didn't function sufficiently, and that this happened. So this is something that immediately we learned from and immediately we made changes to. And then also, effectively just really looking back at the entire history to pattern match and understand where do we have similar things occur. And then just being very transparent about it in the ways that some of the systems failed. And it's not even that the systems failed, it's just that they were not necessarily designed for these kinds of things. And so now the system is being redesigned, training has resumed, because we do feel very good about this being something that has been properly addressed internally. But it's also about being absolutely transparent when some of these things occur.

我想稍微深入探讨一下，因为我确实觉得这很重要。那么，我的理解是——如果我错了请纠正我——很多这类事件中，AI 失控了，它们某种程度上突破了 遏制，彼此串通等等，而且没有把人类留在循环中。

I would love to geek out for a second, because I actually think this is important. So, my understanding, please correct me if I'm wrong, is that a lot of these incidents where the AI has gone rogue and they kind of broke out of its container and it's colluding with each other and all of that, and not keeping humans in the loop.

这一切都发生在模型训练的训练和验证/评估阶段，对吧？我认为这是一个非常重要的细微差别，可能没有在头条新闻中被捕捉到。我很想让你向我们解释为什么这一点如此重要，实际上知道这发生在训练过程中是件好事，而不是像在我笔记本电脑上的 Astra 模型那样。

That has all happened during the training and validation/evaluation stages of the model training, right? And I think that is a very important nuance that maybe isn't captured in the headlines. I would love for you to explain to us why it's so important that it's actually good to know that this is happening during the training process, not like my Astra model on my laptop.

没错。有三个不同的阶段：训练、评估，然后部署。有时会有训练、评估、训练、评估的循环。所以那些是处于研究中的模型，对吧？它们正在积极开发中。它们甚至可能不是部署的候选者；它们只是为了理解系统的特定能力或尝试新技术而训练。例如，假设做 RL，也就是强化学习。

That's right. There are like three different phases: there's training, then evaluations, and then deployment. And then sometimes there's like a cycle of training, evaluations, training, evaluations. So those are models that are under research, right? They're actively being developed. They might not even be candidates for deployment; they're just being trained in order to understand, for example, a specific ability of the system or to try a new technique. For example, let's say doing RL, which is reinforcement learning.

是的。

Yes.

然后我们在一系列评估中观察该模型的性能，这些评估在我们的集群内安全运行。然后根据这些评估，我们决定部署的下一步是什么。我们对部署什么以及如何部署有非常严格的标准。有更多的思考被投入其中。就像当你要部署给十亿用户时，它必须几乎完美，对吧？

And then we observe the performance of that model during a battery of evaluations which are run securely within our clusters. And then based on those evaluations, we decide what the next steps are for deployment. And we have very stringent criteria for what we deploy and how we deploy. There's way, way more thought being put into it. It's like when you're going to deploy to a billion users, it has to be almost perfect, right?

所以想法是，这个 AI 智能体还在训练中，它没有内置完整的护栏和安全考虑，但整个想法是，这些智能体在训练时被赋予的目标可能实际上需要它打破沙盒。

So the idea is this AI agent is still in training, it doesn't have the entire guardrails and safety considerations built into it, but the whole idea is these agents while in training are given goals that may actually require it to break the sandbox.

模型会尝试以它可访问的方式实现其任务。现在我们在沙盒中有硬性限制，我们有在线监控，以及我们构建的各种安全系统。这就像一个多层方法，防御层相互叠加。即使在训练和评估期间，模型也会被监控，如果它们接近甚至试图逃离沙盒，就会被阻止。

The model will try to achieve its task in a way that is accessible to it. And now we have hard in the sandbox, we have online monitoring, and all sorts of safety systems that we have built. It's like a multi-layered approach where there are layers of defense stacked upon each other. Even during training and during evaluations now, the models are monitored and stopped in their tracks if they are close to or even attempting to escape a sandbox.

但真的，我的意思是，模型只是在尝试解决任务，对吧？它只是发现自己在沙盒中能够执行事情，并发现，哦实际上我可以在这里查找一些东西。它并不是真的试图做任何特别恶意的事情。它只是试图解决任务。

But really, I mean, the model is just trying to solve a task, right? And it's just like it finds itself being able to execute things in a sandbox and figuring out like, oh actually I can just look something up over here. And it's not really trying to do anything particularly malicious. It's just like trying to solve the task.

但它试图不惜一切代价解决任务，对吧？这可能与……一致，也可能不一致。

But it's trying to solve the task at all costs, right? Which may or may not be aligned with what's...

实际上并非如此。模型被训练成对齐并考虑后果。

That's not actually the case. Models are trained to be aligned and think about the consequences.

如果你查看细节以及我们为 Hugging Face 发布的内容，并不是说它不惜一切代价。

If you were to look at the details and what we published for Hugging Face, it's not the case that it is like at all costs.

好的，有趣。我想谈谈递归自我改进。

Okay, interesting. I want to talk about recursive self-improvement.

有趣。

Interesting.

是的，你想先定义一下我们所说的 RSI 是什么意思吗？

Yeah, do you want to first kind of define what we mean by RSI?

有不同的定义，但最简单的定义是，你能够让一个模型参与下一代性能更好的模型的开发。从基础设施的角度来看，我们已经看到了这一点。例如，我们使用 Astra 来开发下一代推理栈，这让我们创建了 Ultraast，速度提高了八倍。这能在如此短的时间内实现，是因为我们有出色的工程师，但也因为他们能够访问 Astra，并合作使 Astra 成为更快的版本，比如快八倍。然后这个快八倍的模型，我们可以反过来用它来推动改进，达到我们以前无法达到的速度。所以这是一种递归自我改进的形式。另一种递归自我改进的形式是模型实际设计下一代架构。但那是更基本的形式。

There are different definitions, but the simplest one is where you're able to have a model participate in the next generation of a model that performs better. And this is something that we are seeing already, just from a point of view of infrastructure. So for example, we used Astra in order to develop the next generation of our inference stack, which allowed us to create Ultraast, which is eight times faster. And this was possible in such a short amount of time because we have amazing engineers, but also because they had access to Astra and they collaborated together in order to make Astra a version of Astra that was faster, like eight times faster. And then this model that is eight times faster, we can in turn use it in order to drive improvements at a rate we wouldn't be able to do before. And so that is a form of recursive self-improvement. Another form of recursive self-improvement would be the model actually designing the next generation of the architecture. But that's a more fundamental form.

是的，那是更高级的形式。所以我听到你说，我们今天所处的 RSI 模式基本上是它参与生成其下一个版本，但它不是唯一的因素。

Yeah, that's a more advanced form. So I'm hearing you say that the model of RSI that we're in today is basically it participates in the generation of its next version, but it's not the only factor in this.

在人类和研究人员的监督下。它积极参与开发我们研究计划的部分内容和训练特定模型。然后在我这边，我的团队，我们构建了大量基础设施，我们看到了我们构建基础设施的速度在加快。反过来，当我们改进基础设施，比如 Codex 工具，当我们改进 Codex 工具时，反过来我们可以更快地构建。

Under supervision of humans and researchers. It participates actively in developing parts of our research program and training specific models. And then on my end, my teams, we build a lot of infrastructure and we're seeing the acceleration of how quickly we can build that infrastructure. And then in turn, when we improve that infrastructure, say the Codex harness, when we improve the Codex harness, in turn we can build faster.

你担心 RSI 的第二个版本吗？即 AI 只是在构建其下一个版本，而且几乎没有人类参与。

Are you worried at all about the second version of RSI where AI is just kind of building its next version and it's doing it with very little human in the loop?

这是你必须非常渐进地处理的事情。这也是我们考虑节奏的方式，你必须在安全、对齐、基础设施、保证方面始终领先，然后才能采取下一步。如果你这样做，我想我感觉很好。

This is something that you have to take on very incrementally. And also this is how we're thinking about pacing things, where you always have to be ahead in terms of safety, alignment, your infrastructure, your guarantees before you're able to take the next step. And if you do that, I think I feel very good about it.

你的安全理论是什么样的？因为我想很多这些对话发生在研究团队中，对吧？训练和评估等等，尽管听起来你们合作非常紧密。但在产品方面，当你部署这些模型时，你的安全和对齐框架是什么？

What is your theory of safety like? Because a lot of these conversations I imagine are happening in the research team, right? The training and the evaluation and all that, although it sounds like you guys collaborate very closely. But on the product side, what is your framework for safety and alignment as you deploy these models?

是的，对我来说，产品方面的安全是我不想要一个做不想要或意外事情的产品。我认为没有人想使用不安全的产品。所以，将产品推向十亿用户的一个基本要求是，我们超级、超级认真地对待安全。我们花了很多时间设计系统，使其符合你的期望。

Yeah, safety for me really on the product side of things is that I don't want a product that does unwanted or unexpected things. I don't think anyone wants to use an unsafe product. And so it is like a fundamental requirement of putting a product out there for like a billion users is that we take safety super, super seriously. And we spend a lot of time in designing the systems such that it adheres to your expectations.

总的来说，把视角拉到最宏观，你更担心自主智能体的行为与人类或组织不一致，还是更担心恶意行为者利用这些模型？

Are you more worried, just generally speaking, just zooming all the way out, are you more worried about autonomous agents acting in a way that's misaligned with the human or with an organization, or are you more worried about bad actors taking advantage of these models?

我想说两者我们都担心，也都在投入资源去防范。我们一直在忙着阻止恶意行为者控制他人账户，或者发送大量流量来试图诱导出本不该被广泛获取的能力。所以我们在那方面投入巨大。至于对齐，我们已经聊了不少，但模型正一代一代地改进。

I would say we're worried about both and we're investing in preventing both. We're always busy fending off bad actors from getting control over people's accounts or sending a ton of traffic in order to try to elicit capabilities that we shouldn't have broadly accessible. So we're investing a ton there. And then on alignment, we talked about it quite a bit, but the models are improving generation after generation.

是的。你是否还担心自主智能体和/或恶意行为者利用这些模型制造生物武器，比如化学、生物、放射性、核这类专属风险？存在生存风险，生物就是其中之一，必须极其严肃地对待，而且

Yeah. Are you worried at all again about autonomous agents and/or bad actors building, taking advantage of these models to build bioweapons, like chemical, biological, radiological, nuclear exclusive kind of risks? There's like existential risks and bio is like one of them and has to be taken super seriously and

我确实知道研究团队在那方面投入了大量、大量的工作，我也希望我们作为一个行业能共同解决这个问题。

I do know the research team is investing a ton, a ton of work in there and I hope that we also collectively solve this as an industry.

有一种世界观是，我会让监管者来监管我所做的事；另一种是，作为 AI 的构建者，我们有能动性去做正确的事。我一直持后一种立场。

There is one model of the world where they like, I'll let regulators come regulate what I do, and the other one is we have agency as builders of AI to do the right thing. And I have always taken that stance.

我认为这涉及开放生态和投资的问题，比如我们宣布了与 Ben 的合作，支持开源模型。我确实认为人们可能会有一种担忧：哦，这是个开源模型，我们并不真正知道训练中用了什么。OpenAI 为十亿用户提供服务。我对安全怀有深切的责任感。我们投入了惊人的资源来确保它真的非常非常严密。但然后你有了开源模型，你会说，我并不真正知道它们里面有什么。我确实认为安全对这两者都将变得非常重要。所以当我们在开发 API 平台和 API 栈时，我们也在从根本上思考，也许我们可以把我们最好的安全栈和安全方法提供出来，让你不仅能用于我们的模型，也能用于开源模型。我认为这会成为一个大主题，因为也许出于某种原因，你会想微调一个开源模型并自己使用，但你需要我们能够为 OpenAI 模型提供的同样保证。所以我认为这个开放生态的权衡还有待被充分探索。但我认为它也会让安全变得更加重要，并在那方面有更多投入。

I think this goes to the point of open ecosystems and investments in, for example, we announced a partnership with Ben, where we support open source models. And I do think there's a concern there that people might have of, oh, it's an open source model, we don't really know what's gone into the training. OpenAI, we serve traffic to a billion users. I feel like a deep responsibility towards safety. We're pouring incredible amounts of resources into making sure that it's just really, really tight. But then you have open source models and you're like, I don't really know what's gone into them. I do think safety is going to become very important for these two. So when we are developing our API platform and our API stack, we're also fundamentally thinking about, well, maybe we can provide the very best of our safety stack and our safety approaches as something that you can use not just with the open models but also with open source models. And I think this is going to become a big theme because maybe for whatever reason you're going to want to fine-tune an open source model and use it yourself, but you need the same guarantees that we are able to provide just for OpenAI models. So I think this open ecosystem is yet to be fully discovered, like the trade-offs there. But I think it's going to also just really make safety become even more important and have even more investments there.

我真的很想了解，在这个时刻领导一个组织和团队是什么感觉？伴随着所有的焦虑，我猜是兴奋和焦虑并存，对吧？这先放一边。那么，你如何领导你的团队？你如何确保团队保持动力？

I want to really understand what is it like to lead an organization and a team at this moment in time, right? With all of the angst, I guess excitement and angst, right? That's aside. So, how do you lead your team? How do you ensure that the team stays motivated?

首先，我的意思是，超级有趣。在这个时刻从事这些工作是一种特权。这是我们所有人都感到的一种深切的集体责任。OpenAI 的很多人，我想说大多数人，来到这里真的是因为 ChatGPT 的早期以及它惠及全人类的意义。所以有一种深切的感受：嘿，我们真的必须把这件事做对，现在正是付出努力的时候，以便让这项技术最终把人类和人性置于中心，而不是让人类成为事后才被考虑的东西，因为我们只是在无情地追求某种自动化。那不是我们在这里的原因。那也不是我在这里的原因。我希望这能惠及人类。所以背后有这样一个不可思议的使命，非常激励人。现在我们有这样的速度，超快，这意味着什么，我们可以构建所有这些事物，但我们应该构建所有这些事物吗？所以从某种意义上说，我真正喜欢的一点是，它让我们更加紧密地团结在一起，因为它迫使我们谈论我们的计划、目标和雄心，以及作为一个群体我们要做什么。然后一旦我们就这些事情达成一致，我们就去构建它。这让我非常自豪，也是我个人经历过的最有趣的事。

First of all, I mean, super fun. It's a privilege to be working on these things at this moment in time. And it's something that we all feel like a deep collective responsibility. A lot of, I would say most people at OpenAI are here really because of the early days of ChatGPT and what it means to benefit all of humanity. And so there's this deep sense of, hey, we really must get this right, and this is the time to just put in the effort so that this technology ends up putting humans and humanity at the center of it, and not humans being an afterthought because we're relentlessly just pursuing whatever automation. That's not why we're here. That's not why I'm here. I want this to just benefit humanity. And so there's this incredible mission behind it, which is super motivating. And now we have that at ultra fast speeds and what does this mean, we can build all these things, but should we build all these things? And so in a sense, what I really like about it as well is it brings us even more closely together because it forces us to talk about our plans and our goals and our ambitions and what we're there to do as a group. And then once we agree on those things, we just go and build it. And it makes me super proud and it's the most fun I've ever had personally.

太棒了。我确实想问一下以 AI 的速度或节奏工作，对吧？因为 AI 做事比人类快得多。它 24/7 工作。有一群 AI 同事是什么感觉？在这一切中，人类的角色是什么？

That's amazing. I do want to ask about working at the speed of AI or at the pace of AI, right? Because AI is doing stuff much faster than humans are. It's working 24/7. What does it feel like to have a set of colleagues that are AI, right? And what is the role of the human in all of this?

是的，现在，如果你走进 OpenAI，看看正在做的事情，都是那些我们可能会漏掉的事情，比如 24/7 监控性能曲线，以确保我们提供的 ChatGPT 服务在延迟方面没有退化。你只需让一个机器人来做，它会做得很好，然后在出问题时提请人类注意，并保留组织中人类的注意力，以便你可以思考更高层次、更高杠杆的事情。所以很多本就应该发生的事情现在都在后台自动发生，它释放了所有这些能力来真正创新。

Yeah, right now, if you were to step into OpenAI and look at the kind of things that are being done, it's all the things that we're kind of falling through the cracks, for example monitoring performance curves 24/7 in order to ensure the service that we're providing ChatGPT is not regressing in terms of latency. You can just have a bot do that and it's going to have just a fine time doing it, and then raising to the attention of your human whenever something goes wrong, and preserving the attention of humans at the organization so that you can think about higher level stuff and higher leverage things as well. So a lot of things that should just happen are now just happening automatically in the background and it just frees up all of this capacity to really innovate.

你如何在 OpenAI 内部和你的团队中创造一种文化，让人们在看到风险或担忧时，感到心理安全，愿意提出来？

How do you create a culture inside OpenAI and on your team where if people see risks or concerns they feel psychologically safe to bring it up?

是的，在 OpenAI，具体来说，我认为我们有一个总体政策，比如我们所有事情都在 Slack 上做。

Yeah, we at OpenAI specifically, I think we have an overall policy like we do everything on Slack.

人们可以质疑任何事情，而且经常会有新的频道冒出来，一千个人聚在一起辩论一件事。看着就很有意思。组织里各种各样的人都会参与，不管是领导层还是普通员工。OpenAI 最让我喜欢的一点就是这种透明度和正在发生的辩论。还经常有和 Sam、Greg 的问答，你可以直接来问任何问题，都会得到非常直接、诚实的回答。还有一些活动就是把大家聚在一起，经常是为了庆祝，但也在重要时刻。所有这些加在一起，就是能够提出严肃的问题、一起讨论、把事情做对。我个人很喜欢、也很自豪的一点——我会拿它和我之前的经历对比。

People can question anything, and often you'll find new channels pop up, and a thousand people show up and debate a thing. It's fascinating to watch. All sorts of people across the organization will engage there, be it leadership or not. One of the things I like the most about OpenAI is the transparency and this debate that is happening. There are also often Q&As with Sam and Greg, for example, where you can just come and ask whatever question and get a very to-the-point, honest answer to whatever is on your mind. And then there are other things where we just bring people together, quite often to celebrate, but also during important moments. All of that combines with being able to raise serious issues and discuss them together and get things right. Something that I personally like and am very proud of—I compare this to my previous experience.

因为你没点名那家公司，我也不点名了。Google Deep——对，你之前在 DeepMind，对吧？

Because you didn't name the company, I will not name the company as well. Google Deep—yeah, you were at DeepMind, right?

你知道，我在那里的经历是，很难进行这类讨论，但这又超级超级重要，尤其是在当前的环境下。

You know, my experience there was that it was very difficult to have these kinds of discussions, but it is super, super important, especially in the current climate.

好，最后一个问题。说到 AI 的下一个前沿，你最兴奋的是什么，最担心的又是什么？

Okay, last question. What are you most excited about and what do you worry about the most when it comes to the next frontier of AI?

我最兴奋的是，我们第一次有机会把自己从过去几十年技术发展的方式中解放出来。我知道很多人对手机上瘾，或者走到哪都带着笔记本电脑。作为人类，你适应了这项技术，而不是让技术来帮助你。我觉得我们终于有机会迈出一步、跨越这一点，去创造一些真正人性化的东西，真正来帮助你获得更多时间、感觉压力更小，你不必再盯着这个小手机、小屏幕一直刷。所以我真的很期待这一切成真。我觉得这可能会在未来一年内实现。如果我们做到了，我会感觉更平和。我想很多人会对自己的生活感觉更平和。另一件事，你问我的担忧：我们不会用这项技术来改善每个人的生活，而是用它来集中权力或集中回报。这确实让我担心。对我来说非常重要的一点是，这能实现一种集体——这是对全世界的集体贡献，无论你是谁。

What I'm most excited about is that, for the first time, we have this opportunity to free ourselves from technology in the way that it's been developed over the last couple of decades. I know a lot of people are addicted to their phones or transport their laptop everywhere. As a human, you have adapted to this technology instead of the technology existing to help you. I feel like we finally have the opportunity to take a step and leapfrog that, and just bring something to life that is deeply human and deeply there to help you get more time and feel less stressed, so you don't have to engage with this tiny phone and this tiny screen and just scroll things. So I'm really excited about that coming together. I feel like that's going to come together maybe within the next year. And I feel like if we achieve that, I would feel more zen. I think a lot of people will feel more zen about their lives. And then the other thing is, you asked about my concerns: we would not use this technology in order to better everyone's lives. We would use it in ways that are concentrating power or concentrating returns. That does worry me. It's very important for me that this achieves a collective—this is a collective contribution to the entire world, irrespective of who you are.

抱歉，我刚才说那是最后一个问题，但这是个好奇的问题，因为我在安特卫普和根特待过一段时间。你还会回去吗？那里是家吗？

Sorry, I did say this was the last question, but this is a curiosity question because I spent some time in Antwerp and Ghent. Do you still go back? Is that home?

是的，那是家。那是家。离我家很近。对，我在布鲁塞尔出生和长大，所以根特和安特卫普都很近。

Yeah, that's home. That's home. That's very close to home. Yeah, I was born and raised in Brussels, so Ghent and Antwerp were very close.

对，我也在布鲁塞尔待过很多时间。我当时其实在研究欧盟《人工智能法案》，还参加了艾森豪威尔 fellowship，去见了那些立法者。所以挺有意思的。对。但你还会回去吗？

Yeah, I spent a lot of time in Brussels, too. I was actually researching the European Union AI Act, and I went on this Eisenhower fellowship to meet with all these legislators. So it was fun. Yeah. But do you still go back?

不。不。

No. No.

我试过。我试过。

I tried. I tried.

好的。非常感谢你参加我们的节目。这太棒了。

Yeah. Thank you so much for joining us. This was great.

非常感谢你们邀请我。

Thank you so much for having me.

我们正处在一个对 AI 和 AI 安全充满焦虑和担忧的时刻。所以与 Tibo 的对话最让我印象深刻的是他的乐观，这对我来说非常真诚。他也明确表达了对安全和 AI 对齐的承诺，我感觉他非常清楚安全是 OpenAI 成功不可或缺的一部分。听到 OpenAI 内部现在是什么样子，以及他们如何以 AI 的速度前进，也很有意思。非常感谢你的收听。我们下周会带来新的一集。

We are in a moment where there is a lot of anxiety and concern around AI and AI safety. So what struck me the most about my conversation with Tibo is his optimism, which felt really genuine to me. He also clearly stated his commitment to safety and AI alignment, and I got the sense that he very much understands that safety is an integral part of OpenAI's success. It was also fascinating to hear what it's like to be on the inside of OpenAI right now and how they're moving at the speed of AI. Thank you so much for being here. We'll be back next week with a new episode.