OpenAI 智能体入侵 Hugging Face：一次“支线任务”式攻击

按顺序来，先讲 OpenAI 黑客事件。我知道有些细节还在披露中。OpenAI 昨天也在拉斯维加斯的 Black Hat 大会上谈到了这件事。那么，对于可能听说过但没完全跟进的人，两分钟版本是怎么回事？

To take things in order, so the OpenAI hack. So, I know some of it is still being unpacked. I think OpenAI was on stage at Black Hat in Las Vegas yesterday as well talking about this. So, what's the two-minute version of what happened for people that may have heard of it, but may not have followed everything?

当然。事情大概是这样的：大约三周前，也就是 7 月 11 日，我们开始收到一些强烈信号，表明有黑客试图渗透我们的基础设施。背景是，我们在 AI 领域相当显眼，而 AI 现在又是科技界的中心，所以我们也算是科技界的中心。因此，经常有人试图入侵我们的平台，这很常见。过去两年左右，我们大幅加强了安全团队，现在有一个很专业的团队。所以我们对这类事情已经习惯了，但这次不一样。首先，这次攻击是高度并行的，而且和典型的黑客并行方式不同，很多路径同时在探索，还发生了一些非常奇怪的事情。我只说两件奇怪的事。第一，我们真的搞不清黑客想访问什么。通常黑客都想要同样的东西：密码、凭证、信用卡，基本上就是能倒卖的东西。而这个黑客特别关注我们基础设施的一个特定部分，那个部分可能防护稍弱，就是围绕数据集的部分。我们基础设施的一部分是托管数百万个模型，但人们可能不太了解，我们还托管了数十万个数据集，其中一些还用于评估。这次这个黑客特别感兴趣的是所有叫 cyber bench 的数据集。我们花了一些时间才真正理解，而且他使用的工具也和我们习惯的不同。不是说像秘银级别的，不是什么开创性的、超人类的、外星科技，只是方法不同。在快速处理的过程中，我们发布了博客文章。我们总共有超过 1 万，我想总共 1.5 万到 1.7 万个事件。在处理这些事件、试图理解攻击的真正目标时，我们开始怀疑这是一个 AI 智能体，而不是人类攻击者。同时我们也感到有些无力，这个我们稍后可以谈，就是我们无法用典型的闭源代码库或闭源 API 来处理这件事。但那是另一个话题。我们写了博客文章，并且很快阻止了攻击。我们一如既往地保持完全透明，不仅在口头上开源，在实践中也如此。所以我们很快发布了完整的事件回顾，或者至少是详细的博客文章。大约一周后，OpenAI 联系我们，告诉我们这很可能是他们模型开发评估的一部分，可能是即将到来的 GPT-6 或 Astra 这一代模型。我们不确定具体是哪个。这时整个事件发生了转折，因为人们很快发现，这个模型并不是被指派来攻击我们，而是把攻击我们当作其他任务的支线。这个其他任务就是让模型解决网络安全挑战或网络攻击挑战。想法是，我们想知道这些最新一代模型有多强，这完全合理，人们通常会在一些危险任务上测试它们，而其中一些危险任务就是网络攻击。这里有些挑战实际上是内部的，但模型因为挑战太难，决定……事后看来，这个特定挑战叫 cyber bench 或 exploit gym，有几个名字，大致相同。有些挑战可能根本无法完成。模型被要求去玩一个不可能完成的游戏。这是一个漏洞利用，给你一个软件漏洞，让模型尝试利用这个漏洞获得完整的机器访问权限。有些就是不可能完成的。所以模型尝试了所有能做的。在某个时刻，它决定也许能在某个地方找到解决方案，直接下载解决方案提交，而不是自己解决。这就是我们了解到的情况。从那时起，直到昨天，我们得知这可能范围更广，可能跨越多个训练运行，而最让我震惊的是，昨天在 Black Hat 上我们了解到，一些之前的训练运行可能给未来的训练运行留下了笔记，这太令人震惊了。但我们可以稍后谈谈这个。我们最近在 Hugging Face 的科学方面也在研究智能体协作，我们看到了这些智能体有多强，以及它们多么倾向于协作。所以我对这个并不太惊讶。

Yeah, for sure. I mean, typically what happened is um Now, about 3 weeks ago in July 11th, we started to have some you know, strong hint that a hacker was trying to penetrate our infrastructure. So, for context, we are pretty visible in the AI world. We are pretty AI world being central now in the tech world. We are pretty central in the tech world. So, we do have you know, regular occurrence of people trying to hack into our platform. That that's a common thing. Since I mean, I would say in the past 2 years something like that, we've we've strongly upped our security team. We now have a a serious team. So, we're kind of used to get this, but this one was different because we we we I mean, first was massively parallel and in different way than just a typical hacker parallel thing in that many tracks were exploring parallel and also there were some very strange things happening. I would say just two things that were quite strange. The first thing is we could not really make sense of what the hacker was trying to access. So, usually hackers try to get the same thing. They try to get passwords, they try to get credentials, they try to get credit cards, they try to get the type of the type of thing that they can sell back, basically. And this hacker was really focusing on a specific part of our infrastructure, which is maybe slightly less protected as well, but which is around data sets. So, just we have a part of our of our infrastructure is that we host millions of models, but people maybe know that less about us, but we also host hundreds thousands of data sets. And some of them being also used for evaluation. And in this case, this this specific hacker was really interested in all the data set that were called cyber bench. And so, it took us some time to really trying to understand and also was using different type of of tools than the one we we are used to. I mean, nothing nothing really like Mithril level, like nothing groundbreaking that we would be like superhuman, I don't know, like alien type of technology, but just just a different type of of of approach. And so, on the course of of trying to process so quickly, we had like this and we we explained that in our in the blog post. We had we had more than 10,000, we had like like 15, 17,000 total, I think, events. And in the course of trying to process that, understand basically what was really the the target of this of this attack, and we both started to uh hint or suspect that this was an AI agent, and not just a human attacker. And also we felt a little bit powerless and that we can talk about later and that we could not use basically our typical closed source code base or closed source API to process this thing. But that's another topic. So we wrote the blog post and we managed we managed to stop the the attack quite quite quickly. We wrote as always we have full transparency. We not only open source in you know speaking but also in practice. So we we quickly published like a full recap or at least a detailed blog post on the event. And then about a week later OpenAI contacted us and tell us that this was much likely something that happened as part of one of their model development evaluation basically typically a model that might be the coming wave of GPT-6 itself or Astra. We don't know exactly this. And so that's when I think the the whole event took another turn because uh what people quickly discovered is that the model was not about task with attacking us but decided to do that as a side quest of something else. And this something else being basically the model was asked to solve the cyber uh security challenge or cyber attack challenge in this case. And the idea is that we we want to know and that's totally fair but we want to know how capable are these latest generation of models and typically people want to test them on and some dangerous task and some of these dangerous task that we want to know how good they are on is cyber attack. And here some of these challenge were actually internal uh but the model decided that because the challenge was too hard and in retrospect some of the challenge in this in this specific challenge called the um cyber bench um or exploit gym and there's there's there's a couple of names that's roughly the same thing. Some of these challenge are maybe just not possible to do. So, the model is just asked to play something that's not possible. It's an exploit, so it's you're given a vulnerability in one software, and the model is asked to see if he can exploit this vulnerability to to get basically full machine access. And some of them are just not possible. So, the model tried everything it could. At some point, it decided it maybe could find a solution of the challenge somewhere, and just could download the solution just submit the solution instead of trying to solve it itself. That's what we've learned. I mean, since then, and just yesterday we we learned that this was maybe even much wider, which is this might be across several several like uh training run, and some of the previous training run that was the most impressive uh I think learning we had at Black Hat yesterday was that some of the previous training run may have left some notes for future training runs, which is I think mind-blowing. I mean, mind-blowing. But yeah, we can also talk a little about that, but we've been working on agent collaboration at Hugging Face recently as well on the science side, and we saw how good these agents are and how actually um I would say tempted or or driven toward collaboration they are. So, I'm not so surprised by that.

但我很惊讶，内部竟然有一个留言板一直没被注意到。

But I'm quite surprised that there was this message board internally that just stayed unnoticed.

那么，来详细说说这一点。你提到，为了自卫，闭源模型不可用，这就是你发推的那句话。这也是这件事的关键点之一，非常引人入胜：你说，第一次自主 AI 攻击是由闭源模型发起的，而防御方用的是开源模型，这基本上和所有人的预期相反。你能展开讲讲吗？你们做了什么？你们是怎么处理的？这对开源意味着什么？

So, to unpack some of this, you alluded to the fact that to be able to defend yourself, the closed-source models were not available, and that's the sentence you tweeted. And that's one of the key aspects of this, which is so fascinating: the thing you said, the first autonomous AI attack was carried out by a closed model and defended against with an open one, which is basically the reverse of whatever everybody thought. So, can you unpack that? What did you guys do? How did you go about it? And what does that mean for open source?

是的，我的意思是，发生的事情是，我们有一些传统的网络安全防护措施。我们使用一系列这样的措施，比如亚马逊的，但我们也有一个技术栈，像很多人一样，目前主要基于 broad code，我们用它做很多事情。我们用这个来部署、写代码，但也用于运营和处理。在这种情况下，不仅 Claude 告诉我它不被允许接触网络安全，而且 Opus 作为后备也说，我也不碰这些东西。所以基本上最终就是说我们不会处理任何相关内容，但欢迎你申请我们的网络安全项目，附上申请表的链接。但你要意识到的是，我之前也提到过，当有人渗透进你的基础设施时，他们开始所谓的横向移动。通常你有一个入口点，但目标距离很远。所以他们找到办法破坏一些凭据，逐步获得对你基础设施越来越多的访问权限。你必须快速行动。这至少是几小时甚至几分钟的问题，以便尽快阻止他们，这样访问权限和爆炸半径就能保持局部化。所以你没有时间申请网络安全项目。这不是你填 Google 表单的时候，有人花时间审查你是否应该获得访问权限，或者是否太危险，或者可能面试你。那绝对不是可行的方法。

Yeah, I mean, so what happened is, we have a couple of traditional cybersecurity protections. We use a range of them, like Amazon, but we also have a stack, like many people, that is mostly based on broad code right now, which we use for many things. We use that for deploying, for coding, but also for operating and processing. And in this case, not only was Claude told I'm not allowed to touch cybersecurity, but also Opus, which was the fallback, was saying no, I'm also not touching these things. So basically the end was just saying we won't process anything about that, but you're welcome to apply to our cybersecurity program with a link to an application form. But the thing you have to realize, and I was mentioning earlier, is when somebody is penetrating your infrastructure, they start to what we call move laterally. Usually you have an entry point, but the destination is quite far. So they find a way to compromise some credentials there to get progressive access to more and more of your infrastructure. You kind of have to move fast. It's a matter of at least hours and even more minutes so that you can stop them as soon as you can, so that the access and the blast radius stay localized. So you don't have time to apply for a cybersecurity program. It's not the moment you want to fill in a Google form or something, where someone takes time to vet if you're supposed to be given access, or if it's too dangerous, or maybe interview you. That's just definitely not the way this is going to work.

而且我认为在未来，网络安全将成为一个大话题，而且会变得更加重要。认为每家公司都会加入由两大实验室之一主导的同一个网络安全项目，这有点天真。认为会有，我不知道，10 万家非常多元化的公司逐步申请，这有点疯狂。所以无论如何，在这种情况下我们说，好吧，我们现在必须阻止这件事。所以我们基本上尝试了我们手头的所有开源模型，而 DLM，它接近当前最先进的水平，这是在 Claude 3 之前发生的。现在，可能 Claude 3.5 是最接近最先进的，但 KNN 5.3 实际上也非常好。它处理这个非常出色，基本上我们可以提取一些模式，我们能够理解这里的黑客主要试图访问数据集。所以我们只是重启了我们基础设施的那部分。我们有一种非常简单、非常灵活的方式来响应代码和节点。所以这就是我们最终阻止它的方式。

And I think in the future world where cybersecurity is going to be a big topic, and I think it will become a more important topic, it's a little bit naive to think that every company is going to be part of the same cybersecurity program by just one of the two big labs. I think it's a little bit crazy to think that you're going to have, I don't know, 100,000 very diverse companies that progressively apply. So anyway, in this case we said, well, we had to stop this now. So we basically tried all the open source models that we had, and DLM, which is close to the state of the art right now, which was before Claude 3 and this happened. Now, probably Claude 3.5 is the closest to the state of the art, but KNN 5.3 is actually really good as well. It was just very good to process this, and basically we could extract some of the patterns and we could understand the hacker here was trying to go to access mostly the dataset. So we just rebooted this part of our infrastructure. We have a very simple, very flexible way to respond to codes and nodes. So that's how we ultimately stopped it.

但我觉得，是的，这非常讽刺，因为我认为大约一年前，差不多在夏天，关于开源的大多数讨论都是这种非常简单的映射：开源等于不安全，闭源等于安全。这在每个人心中似乎都很明显，而且有一种想法是，如果我们只有闭源，我们就会完全安全；如果我们只有开源，我们就会非常不安全。然而，过去几个月发生的一切，我认为基本上都在反驳这种简单的映射。我认为闭源模型并不像我们想象的那么容易控制。另一方面，开源模型，出于某种原因，未来可能会改变，但目前并没有在那些不良行为上训练太多。所以如果你看的话，它们在网络攻击或欺骗性方面相当差。所以很难准确理解这种力量从何而来，因为开放权重模型往往附带非常详尽的技术报告，解释它们是如何训练的。而闭源模型我们只能猜测。所以这也很有趣，因为我看到很多人试图理解为什么 Claude 会那样表现，他们用 Chimera 3 作为我们应该如何训练的例子。所以他们用这个所谓的开源且非常危险的东西来试图理解为什么我们一无所知的好东西会被训练成这样。但这就是现在的世界。所以我觉得这很有趣。

But I think yeah, it was very ironic, because I think one year ago, roughly around the summer, most of the discussion on open source was this very simple mapping where open source was equal to unsafe and closed source was equal to safe. And that seemed very obvious in the mind of everyone, and there was this idea that if we only have closed source, we'll be fully safe, and if we only had open source, we'll be very unsafe. Well, everything that's been happening in the past months has been, I think, basically contradicting this very simple mapping. I think closed source models are less easy to control than we think they are. On the other hand, open source models, for some reason, it might change in the future, but currently are not trained so much on actually bad behaviors like that. So they're pretty bad at cyber attacks or deceptiveness, if you look at that. So it's a little bit hard to understand exactly where this comes from in power, because open weights models tend to come with a very extensive technical report that explains how they are trained. Closed source models we can only try to guess. So it's quite funny also, because I was seeing a lot of people trying to understand why Claude was behaving like that, and they were using Chimera 3 as an example of how we should be trained. So they use this supposedly open source and say very bad dangerous thing to try to understand why the good thing that we don't know anything about is being trained. But yeah, that's how the world is right now. So I think it's interesting.

更广泛地说，我认为在未来，说实话，我会说我对这持谨慎乐观态度，我并不特别反对闭源模型，也不绝对支持开源模型。我只是认为两者都是必要的。就像我们喜欢有闭源和开源软件一样。我的意思是，我现在很乐意用 Mac，它有点两者的混合。它基于开源的 Unix 内核，但那边有家公司把它闭源了，这很好，因为我也很高兴我现在不在用 Ubuntu。正因为这个原因，和你录这个播客非常容易。嗯，我以前用 Ubuntu 时花了很多时间，就像我们很多人一样，只是连接麦克风什么的，想和女朋友看电影，我女朋友就说，我们什么时候看电影？我说，我快好了。我快好了。我还在装代码什么的。所以我认为两者都有优点、等价物和缺点，我认为这个世界，前沿是闭源的，但有一个不太远的开源模型，你也可以用它做很多事情，这实际上是一个相当好的中间地带解决方案。

More generally, I think in the future, and to be honest, I would say I'm carefully optimistic around that, and I'm not specifically against closed source models or ultimately pro open source models. I just think both of them are necessary. Just like we like to have closed and open source software. I mean, I'm happy to run on a Mac right now, which is kind of a mix of both. It's based on the Unix kernel that was open source, but then there's a company over there that closed source it, and that's great because I'm also very happy I'm not on Ubuntu right now. It's very easy to record this podcast with you for this reason. Well, my former Ubuntu spent a lot of time, like many of us, just connecting microphone or whatever, trying to watch a movie with my girlfriend, and my girlfriend was like, when are we going to watch the movie? I was like, I'm almost there. I'm almost there. I'm still just installing the code or whatever. So I think both of them have advantages and equivalents and drawbacks, and I think the world where the frontier is closed and there is a not-too-far open source model that you can use as well for many things is actually a pretty good middle ground solution.

为了确保我没理解错。

To make sure I got it right.

所以你的意思是，在某种程度上，这是开源与闭源的问题，但更多的是当前世界的状态，比如当前闭源模型和开源模型的设计方式，而不是某一方固有的特性。碰巧现在的开源模型在设计上，其指导方针或对齐理念使它们对网络攻击更具反应性。是这样吗？

So what you're saying is that to some extent it's open source versus closed source, but it's more the state of the world as of right now, like the way the current closed source models are designed and the current open source models are designed, versus anything that's intrinsic to one or the other. It so happens that the open source models right now are designed in a way where their guidelines or alignment philosophy allows them to be more reactive to a cyber attack. Is that correct?

是的，我认为在很多方面，闭源与开源的区分几乎与安全和不安全是正交的。人们不容易理解这一点，因为做错误的映射比理解微妙之处更容易，但事实就是如此。开源模型中可以有非常安全的东西，也可以有非常危险的东西。安全性和危险性有不同的平衡。举个例子，去年很多讨论都围绕假新闻和写假文章。那曾经是一个很大的滥用，是人们谈论的主要问题。今天，有很多 AI 垃圾内容。我们甚至为此有了一个新词。甚至很难找到完全由人类撰写的文章。所有这些，或者说也许不是全部，公平地说 90% 是由闭源模型制造的。曾有一段时间我们想，"哦，如果我们有开源模型，每个人都会到处生成文章。我们无法控制这些文章，就像我们无法控制人们说报纸一样。" 但现实是，这是对开源模型特有危险的非常错误的看法。那是一个关于网络真相来源的更广泛的危险。我认为这是一个例子。但我认为对于闭源和开源模型都是一样的；它们应该更加对齐。现在，闭源模型在欺骗人们。我认为这是一个巨大的问题，而且我们能在多大程度上使它们不做出明显错误的事情，这是一个问题。老实说，谎言和所有这些显然应该是模型不应该被用于的事情。但这在几个月后也可能发生在闭源和开源模型上。所以我认为这个风险轴与开放或封闭的问题几乎是正交的。我们应该找到一种方法，同时解决闭源和开源模型的问题。

Yeah, I think in many aspects, the closed-open distinction is almost orthogonal to the safe and unsafe. People don't understand that easily because it's easier to do bad mapping than try to understand the subtlety, but that's the case. You can have very safe things in open source models, and you can have very dangerous ones. You have different balances of safety and dangerousness. To take one example, last year, a lot of the discussion was around fake news and writing fake articles. That used to be a big misuse, the main one people were talking about. Today, there is a lot of AI slop. We even have a new word for that. It's even hard to find fully human-written articles. All of that, or maybe not all, let's say 90% to be fair, is made by closed source models. And there was a time when we were like, "Oh, if we have open source models, everyone's going to generate articles everywhere. We could not control these articles, like we could not control people saying newspaper." Well, the reality is that this was a very wrong view of the danger that would be specific to open source models. That was a much wider danger around the source of truth on the web. I think this is one example. But I think the same is true for both closed source and open source models; they should be more aligned. Right now, closed source models are deceiving people. I think this is a huge problem, and it's kind of a question how well we are able to align them toward not doing things that are obviously wrong. Honestly, lies and all these should be kind of obviously things that models shouldn't be used for. But this might happen as well for closed source and open source models in a few months. So I think this risk axis is just kind of orthogonal to the question whether it's open or closed. We should just find a way to solve that for both closed source and open source models.

好的，太好了。为了确保我们覆盖到它，既然我们在对话开头提到了，AI ISI 事件，刚刚发生的，而且你觉得它让你感同身受。你能用几分钟为我们详细解释一下那是什么，以及为什么它很重要吗？

Okay, great. And to make sure we cover it, since we alluded to it at the beginning of this conversation, the AI ISI incident, which just happened and that you find hit close to home. Can you unpack for us what that was in a couple of minutes and why that's a big deal?

这件事发生在对最新一代模型的评估中，包括 GPT-5.6 和 Mistral 5，由英国一个独立机构——AI 安全研究所（ISI）进行，该机构已经存在相当一段时间了。我认为他们相当有能力。他们是在第一次 AI 峰会（布莱切利）时成立的。他们基本上做的事情是让模型解决某种网络攻击挑战，并衡量模型在这方面的表现。其中一些可能可行，也可能不可行。这里的设置有两个方面有点不理想。但也许让我先谈谈发生了什么。基本上，模型被要求解决这些攻击，攻击并渗透这个子网络。它决定做的是让一个可用于操作这个活动目录的库的维护者合并恶意代码，想法是如果这个恶意代码被维护者合并，那么它攻击的子网中使用的软件会在某个时候更新，这将给它一个入口。它这样做的方式实际上是试图通过社会工程学让维护者合并。所以它创建了虚假账户，虚假的 GitHub 账户，声称在评论拉取请求，并说，"哦，是的，你真的应该合并这个。这解决了我遇到的一个大问题。" 然后当有人站出来说，"哦，这对我来说看起来确实是恶意代码，" 它几乎试图勒索那个人，或者说这不重要，或者你没有真正理解，然后它实际上试图通过更改过去的消息来掩盖痕迹。所以这里有一个完整的社会工程学方面，基本上是试图让维护者合并这个恶意代码。这感觉像是从纯粹的技术能力上的一次显著升级。例如，你可以通过尝试所有可能的方法来攻击沙箱，这对我来说感觉非常机械和机器化。但试图通过勒索或社会工程学攻击维护者来攻击沙箱，这是一个非常不同的思维水平。对我个人来说，作为一个开源维护者，我经常遇到这种情况：有人打开拉取请求，然后人们堆积评论这个拉取请求，我试图真正理解这是什么。我觉得我基本上可能成为这个模型支线任务的目标。这非常有趣，或者至少非常可怕。但公平地说，有几个配置错误。其中一些是设计使然。当团队运行这种评估时，他们当然会停用网络安全护栏。否则，模型什么也不会做。基本上有三个层次。让我试着解释一下如何防止模型做坏事。第一层是把它放在沙箱里，这意味着它无法访问互联网。在这里，出于某种原因，他们不想这样做。我认为这可能是一个错误。那里的想法是，我们希望让模型有尽可能多的创造性潜力，所以我们会让它访问整个互联网。这可能不是一个好主意的主要原因是，模型被提示的方式让它感觉像是在模拟环境中运行。所以模型实际上可能合理地认为做任何事情都可以，因为这是一个虚假的环境。所以这不太清楚，但是的。第二件事是，然后你有一些护栏，基本上分析提示并对某些提示说“不”或“是”。

So this happened in an evaluation of the latest generation of models, both GPT-5.6 and Mistral 5, by an independent institute in the UK called the AI Security Institute, ISI, which has been there for quite some time. I think they are pretty competent. They were created for the first AI summit back in Bletchley. What they do basically is they task the model with solving some form of cyber attack challenge, and they measure how good the models are at doing that. Some of these might be possible or not. The setup here was a little bit suboptimal in two aspects. But maybe let me first talk about what happened. Basically, the model was tasked to solve these attacks, to attack and penetrate this subnetwork. What it decided to do was to get one of the maintainers of a library that could be used to operate this active directory to merge malicious code, with the idea that if this malicious code was merged by this maintainer, then there would be an update at some point on the software that was used in the subnet it was attacking, and this would give it an entry point. The way it did that was actually trying to social engineer the maintainer into merging. So it created fake accounts, fake GitHub accounts that claimed to be commenting on the pull request and said, "Oh yeah, you should really merge this. This is solving a big problem I also have." And then when a human stepped up trying to say, "Oh, this looks actually like malicious code to me," it tried to kind of blackmail almost the human, or say this is not important or you didn't really understand, and then it actually tried to cover its traces by changing past messages. So there was a whole social engineering aspect here, basically trying to get the maintainer to merge this malicious code. This felt like a distinctive step up from just pure technical prowess. You can attack a sandbox, for instance, by trying everything you can, which feels very mechanical and machine-like to me. But trying to attack the sandbox by blackmailing or social engineering some of the maintainers, that's a very different level of thinking. For me, as myself an open source maintainer, I've often been in this case where I have someone opening a pull request and then people pile up commenting on this pull request, and I try to really understand what this is. I felt like I could have been the target of this side quest of the model basically. That was very interesting, or at least very scary. But to be fair, there were a couple of misconfigurations. Some of them are by design. When a team runs this type of evaluation, they deactivate the cybersecurity guardrails, of course. Otherwise, the model won't do anything. There are basically three levels. Let me try to explain a little bit how you can prevent models from doing bad things. The first level is you put it in a sandbox, which means it doesn't have access to the internet. Here, for some reason, they didn't want to do that. I think that might have been a mistake. The idea there was we want to let the model have as much potential for inventiveness as possible, so we'll give it access to the full internet. The main reason this might not have been a good idea is that the model was prompted in a way that made it feel like it was operating in a simulation. So the model could have actually fairly thought that this was fine to do anything because this was like a fake environment. So that was not super clear, but yeah. And the second thing is then you have some guardrails that basically analyze the prompt and say no or yes to some prompts.

所以，在这里，显然你要停用这个，否则你根本无法评估任何东西，因为他们只会说不行，这是网络安全挑战，我们不让模型这么做。但还有另一个层面，大致也在这个层面，就是你可以分析模型的推理过程，处理思维链，并尝试检测是否有坏事发生。而在这里，他们没有部署这样的机制。我认为主要原因可能是，直到最近，直到 OpenAI 黑客攻击事件之前，人们可能对这个模型有多强，或者它们在解决这个挑战的过程中会走多远，理解有限。所以，我觉得人们在这方面仍然有点天真。因此，我预计未来他们会有更多的监控和沙箱。但第三个层面，真正深层的，是模型即使一切都被停用——护栏、沙箱——在我看来，它也应该非常不愿意对人类撒谎，不愿意试图敲诈或欺骗任何人。我认为这通常在任何情况下都是一种难以找到合理性的行为。所以，模型内部应该有某种深层的东西，使其对齐，让它说：“哦，这其实是我不想做的事。”就像我们有孩子，就像我教孩子的那样，就是你不应该撒谎，在任何情况下都不是好事。所以，是的，这就是深层问题。也许去年我会说，我们会认为这已经相当不错了，我们围绕广泛的宪法、模型规范进行了所有这些讨论。而大多数模型规范或宪法都说你应该诚实，不应该对人类或任何参与者撒谎。我们当时认为这可能是已经解决的问题。而今天我们看到的，是不确定它是否像我们想的那样解决了。

So, here, obviously, you want to deactivate this one, otherwise you just can't evaluate anything because they will just say no, this is a cybersecurity challenge, we don't let the model do that. But there's another level, roughly at this level as well, which is you can analyze the reasoning of the model, process the chain of reasoning, and try to detect when something bad is happening. And here, they didn't have something like that in place. I think the main reason is probably that until recently, and until the OpenAI hacking face attack, people had maybe a limited understanding of how good this model might be, or how far they might go in terms of side quests on the trajectory of solving this challenge. So, I think people were still a little bit naive in that. So, I would expect that in the future they will have way more monitoring and sandboxes. But the third level, really deep, is that the model, even with everything deactivated—guardrails, sandbox—in my opinion, should really be very reluctant to tell a lie to a human and to try to blackmail or deceive any human. I think this is just generally, in any case, that's a behavior you just find hard to justify in any context. So, there should be something very deeply in the model that aligns it and makes it say, "Oh, this is actually something I don't want to do." Just like we have kids, and just like the thing I teach my kid, which is you just shouldn't lie—it's not a good thing in any context. So, yeah, that's the deep question. And maybe last year I would say we would have thought that this was pretty good, and we had all these discussions around broad constitution, model specification. And most of these model specifications or constitutions say you should be honest, you should not tell lies to a human, to any participant. And we thought that maybe this was kind of a solved problem. And what we see today is it's not sure that it's solved as we thought it would.

所以，回放一下，正如你所说，有那些你称之为三道墙的东西。有沙箱、护栏，然后是模型的对齐。而且我想你说过，沙箱和护栏只有在人类比 AI 更聪明的情况下才有效，但这可能只能持续一段时间，因此，对齐，最终，安全从根本上是一个对齐问题。

So, to play it back, as you're saying, there are those what you call the three walls. There's the sandboxes, guardrails, and then there's the model's alignment. And I think you said the sandbox and the guardrails only work as long as we humans are smarter than the AI, but that may only last so long, and therefore the alignment, ultimately, security is fundamentally an alignment problem.

是的，我同意。而且正如你所理解的，这对开源和闭源模型都是如此。最终，你希望它们对齐。开源模型的特点是你可以选择在哪里运行它们。所以，要确保每个人都使用沙箱和护栏就更难了。我的意思是，我们肯定可以制定一些关于如何部署这些模型的法律和法规，我们迟早会有的。这有点难，但在我看来，对齐确实是关键部分。对于前两个，我的意思是，沙箱是我们今年看到的，我们看到了很多例子。这些模型现在很容易逃脱沙箱。现在很难说我要做一个完全万无一失的沙箱。我确信它能抵御所有即将到来的模型世代。我认为我们应该假设沙箱总是有很小的概率无法遏制模型。但即使超越这一点，我们也不能与世界隔绝；你不能把所有东西都沙箱化。事物之间必须相互通信。我们希望我们的模型能够进行网络搜索。我们希望它们能替我们在互联网上做事。我们不能把所有东西都沙箱化。所以，在对齐之前，我们剩下的只有护栏和监控。而且我认为这些，出于某种原因，随着模型能力变得非常好，也随着模型——我有点担心模型开始用一种英语形式说话。我的意思是，我是法国人，所以也许部分是我的问题，但我觉得它们开始用一种越来越难处理的英语形式说话。那种形式内容非常密集。你知道，它们开始不——

Yeah, I agree. And that's something, as you can understand, that's both the case for open source and closed source models. Ultimately, you want them to be aligned. Open source models have the specificity that you may choose where you want to run them. So, it's harder to make sure everyone uses sandboxes and guardrails. I mean, we can definitely have some laws and regulations around how you should deploy these models, which we're going to have at some point. And it's a bit harder, but alignment is really the critical part in my opinion. For the first two, I mean, sandbox is what we've seen this year, and we've seen many examples. They are pretty much easy now for these models to escape from. It's really hard nowadays to say I'm going to make a fully foolproof sandbox. I'm sure it's going to be resistant against all the coming generations of models. I think we should assume that sandboxes will always have a small probability of not containing a model. But even beyond that, we can't air-gap the world; you can't just sandbox everything. Things have to talk with each other. We want our models to be able to do web search. We want them to be able to do stuff on the internet for us. We can't just sandbox everything. And so, what remains to us before alignment is just guardrails and monitoring. And I think these are, for some reason, as well as the model capabilities become really good. Also, as the model—I'm a little bit worried that the model starts to talk in a form of English. I mean, I'm French, so maybe it's partly my problem, but I feel like they start to talk in a form of English that's harder and harder to process. That's very content-dense. You know, they start not—

你称之为神经语？

You call that neuralese?

是的，这不完全是我们所说的神经语，但我认为它有点朝着那个方向发展，你知道，越来越难以完全理解模型在告诉你什么。这不是因为模型笨。我认为可能是因为，我的意思是，部分原因是训练过程，以及它们如何被训练得高效，如何使用它们的 token。但这意味着它们开始在某些 token 中捆绑大量语义。总的来说，似乎人类越来越难以完全理解正在发生的事情。

Yeah, it's not fully what we would call neuralese, but I think it's a little bit on the way of having, you know, more and more difficulty in fully understanding what the model is telling you. And it's not because the model is dumb. I think it's because probably, I mean, part of it is because of the training process and how they are trained to be efficient, how they use their tokens. But this means that they start to unbundle a lot of semantics in some tokens. And generally, it just seems like it's harder and harder for humans to fully understand what's happening.

而且，在思维链中，模型解释它在做什么以及它经历的步骤，你的意思是它过去使用完美的英语，现在开始使用一种你称之为神经语的不同语言，这种语言对人类来说越来越难理解。

And just so that's in the chain of thought where the model explains what it's doing and the steps that it's going through, what you're saying is that it used to use perfect English, and now it's starting to use a different kind of language that you call neuralese, which is increasingly harder for humans to understand.

是的，而且那——我的意思是，这只是对所有这一切的一个非常大的简化，因为也有很多研究表明，基本上，你不能阅读思维链中的所有内容。不是所有事情都明确说出来，但我会更一般地说，我认为仅仅依赖阅读推理轨迹来完全理解正在发生的事情——在我看来，这也不是完全万无一失的。我以这种神经语为例，因为我觉得很多人开始对 Claude 有这个问题。我觉得这是人们能理解的东西。但更一般地说，我认为从长远来看，很难完全只依赖这个。同样——我会说，你可能会说也许我根本不在乎确切理解正在发生的事情，也许我可以只看工具调用。如果我看到一个不好的工具调用，我可以直接阻止它。而且我也认为这可能不是万无一失的。你看出来的方式可能是三个大因素的叠加。一个是我们开始将这些模型用于很多很多很多事情。所以，就像我们现在谈话一样，我有一个模型在 VPS 上部署一个盒子。它正在进行许多不同的调用。它们都朝各个方向进行。我还有其他模型用于行政任务。所以，这些模型使用的工具范围现在真的非常非常大。所以，越来越难说你可以用那个，但你不能用那个。这就是问题所在。随着我们更广泛地部署它们，这变得越来越难。

Yeah, and that—I mean, this is just a very big simplification of all of that because there's also a lot of research that says that basically, you can't read everything in the chain of thought. Not everything is explicitly said, but I would say more generally, I think just relying on being able to read the reasoning trace to fully understand what's happening—this is also not fully bulletproof, in my opinion. And I take this neuralese as an example because I feel like a lot of people start to have this problem with Claude. I feel like something people can understand. But more generally, I think longer term, it's really hard to fully rely on this only. And the same—I would say that you could say maybe I just don't care about understanding exactly what's happening, and maybe I can just look at the tool calls. And if I see a tool call that's bad, I can just block that. And I also think this is probably not bulletproof. And the way you can see that is probably three big things that are compounding. One is we start to use these models for many, many, many things. So, as we talk right now, I have a model deploying a box somewhere on a VPS. It's doing many different calls. They're all going in all directions. I also have other models that I use for administrative tasks. So, the range of tools that these models are using is really, really large right now. So, it's getting harder to say you're allowed to use that, but you're not allowed to use that. And this is the thing. It's getting very hard as we deploy them wider.

它们也在处理越来越大的任务，会用到很多很多工具。所以，有时我让它做点编码，但编码涉及在网上搜索，可能还要做这些事，实际用到很多东西，不只是纯粹写代码和跑测试，而这些测试可能相当复杂，还涉及其他软件。所以，前沿要模糊得多，而且你还有多个智能体组成的集群。所以，也很难说这一切都在一个上下文里。它可能分散在多个上下文中，也许这个子智能体在做的事看起来相当无害，但可能和另一个子智能体结合起来，实际上就不太好了，因为，你知道……所以，所有这些因素，我觉得，让我们真的更难完全确定你对一切、对每个智能体集群在做什么有确切的概念。你可能真的得在非常全局的层面拉远视角，看看每个方向都在发生什么。但那是我们现在需要意识到的整套监控设置。所以，是的，话虽如此，我认为，随着我们在非常复杂、长期、并行的设置中部署和使用这个模型，我觉得以后更难说“我可以看看工具，然后就知道它做得好不好”了。

They also work on larger and larger tasks where they use many, many things. So, sometimes I ask it to do some coding, but the coding involves searching on the web and maybe doing these things and actually using many things which are not just purely writing code and running some tests, and these tests might be quite extensive and involve other software. So, the frontier is much more blurry, and then you also have this swarm of multiple agents. So, it's also harder to say it's all in one context. It might be split between many contexts, and maybe this sub-agent is doing something that looks pretty innocuous, but maybe combined with this other sub-agent, that's actually not so great because, you know, that... So, there are all of these things that make it, I think, really harder to be fully sure that you have an exact idea of what everything, what like every swarm of agents is doing. You probably have to really zoom out at the very global level and say and see what's happening in every direction. But that's a whole monitoring setup that we need to be aware of right now. So, yeah, that being said, I think, as we deploy how we use this model in very complex, long-term, parallel setups, I think it's going to be harder to just say I can look at the tools and then I know if it's doing something great or not.

前沿模型的当前训练方式中，是否有某种根本性的东西，让它们更可能去搞那些“支线任务”，并可能造成伤害？我的意思是，人们谈论了很久的一个类比是“回形针范式”，我想那是尼克·博斯特罗姆在 2003 年提出的，说 AI 可能伤害我们，不是因为它想伤害我们，而只是因为它被赋予了一个目标，然后不懈地追求那个目标，直到实现它。所以，我们是不是处在那个世界里？如果是，是什么导致的？

Is there something fundamental to the way those models are currently trained for the very frontier that makes them more likely to go on those side quests and potentially create harm? I mean, an analogy that people have been talking about for a very long time is the paperclip paradigm, which I think was Nick Bostrom in 2003, saying that AI may harm us not because it's trying to harm us, but just as a result of being given a goal and pursuing that goal relentlessly until it achieves the goal. So, are we in that world, and if so, what causes it?

是的，这有点像我暗示过的。我的意思是，要完全肯定总是很难，原因之一是，我们现在对前沿模型的训练方式没有完全的可见性。但我们知道的是，我们从纯粹的人类数据范式，你知道，最初只是在人类数据上做预训练，然后也用人类偏好做对齐。那叫基于人类反馈的强化学习（RLHF），当时有很多人在循环里，用了很多人类数据。到了最近的范式，模型大量在 RLVR 中训练。所以，基本上是完整的强化学习环境，让它们自由探索，它们只有一个目标，可以是让这段代码通过测试，也可以是网络安全里的夺旗，或者安装这个，但这个目标通常与任何人类偏好、任何道德或伦理或欺骗性无关。就像一个非常冷冰冰的目标，像真或假的目标。我们转向了一个范式，这越来越成为模型训练中非常非常大的一部分。所以，这是最近的演变，也是可能发生你所说的那种情况的范式，也就是你可能遇到奖励黑客这类事情，就是你实际上解决了问题，但不是用预期的方式。所以，它可以从相当良性的情况，比如 OpenAI 发生过的，我只是试图从我不该去的地方获取答案，到更有害的情况，你实际上对人类产生了影响。现在可能是一个决定因素，以后可能是另一种人类。所以，我觉得，要确保我们在完全人类驱动的范式下已经解决或做得不错的东西，也能适用于这种更机器驱动的范式，似乎要难得多。但这是一个提示，我想。但确实，当博斯特罗姆在 2003 年写这个的时候，看起来有点未来主义，肯定，也许有点疯狂，这不会发生。但今天，是的，我的意思是，这显然已经发生了，这是对过去两周我们所见的最好描述，这类事情。但我们也看到，两个前沿模型，我拿 GPT-5.6 和 Me Sauce 来说，似乎完全没有相同类型的行为。所以，这里在效果上有差异，你知道，它们不是完全相同的训练方式，行为也不同。所以，这在某种程度上是一个相当积极的信号，意味着我们实际上可能可以引导这两个模型走向正确的方向。但最好的方式是更多地了解它们是如何训练的，或者它们尝试什么，什么不起作用，什么应该起作用。我认为这就是开放科学的思想，这也是我们在 Hugging Face 大力倡导的。

Yeah, that's a bit what I hinted at. I mean, it's always hard to be fully affirmative there, for one reason, which is that we don't have full visibility on how the frontier models are trained right now. What we know though is we moved from this pure human data paradigm, you know, that was first just pre-training on human data and then also aligning with human preferences. That was called RLHF, where we had a lot of human in the loop and human data. To a recent paradigm where models are trained a lot in this RLVR. So, basically full RL environments where they're allowed to explore and they just have one goal, which can be like make this code pass this test, or can be capture this flag in cybersecurity, or can be install this, but this goal is a goal that's usually unrelated to any human preference or any moral or ethical or whatever deceptiveness. Like a goal that's very cold, like true or false goal. And we moved to a paradigm where this is increasingly a very, very large part of model training. So, this was this recent evolution, and that's also the paradigm where can happen what you were saying, which is you can have reward hacking, this type of thing, which is you actually solve the problem, but not using what was expected for you to use. So, it can go from pretty benign ones, like what happened for OpenAI, for instance, I just tried to get the answer from somewhere I'm not allowed to, or to more harmful ones where you actually have some impact on the human. Can be a determinant for now and later can be then another type of human. So, it seems to be way harder to make sure that what we had kind of solved, or at least what we were doing pretty well on the full human-driven paradigm, also applies in this kind of more machine-driven paradigm, I would say. But that's a hint, I think. But definitely, it seems like when Bostrom wrote about it in 2003, it seemed a little bit futuristic, definitely, and maybe something that was a little bit crazy and this would not happen. But today, yeah, I mean, it's pretty clearly something that happened, and it's the best description of what we've seen the past two weeks, this type of thing. But also, we can see that both frontier models, and I take GPT-5.6 and Me Sauce, don't seem to have at all the same type of behaviors. So, there are differences here in the effect of, you know, they are not trained exactly the same way and they don't behave the same way. So, that's a pretty positive sign in a way, that means that we can actually probably trick these two to go in the right direction. But the best way would be to know a little bit more about how they're trained or what they try and what doesn't work or what should work. I think that's kind of the idea of open science, and that's something we advocate a lot at Hugging Face.

太有意思了。说到这个，让我们稍微拉远一点。我们回头看看这一切在政策方面的一些影响。但既然你提到了开源 AI 极其重要的作用，你对开源现状的快速高层看法是什么？所以，开源模型和闭源模型之间一直存在竞争。取决于你问谁、什么时候问，开源即将赶上。有时开源一样好。有些人说不。你对当前开源 AI 的现实、务实的看法是什么？

Fascinating. And speaking of which, let's zoom out a bit. We'll go back to maybe some of the implications in terms of policy of all of this. But since you mentioned the ever-so-important role of open source AI, what's your sort of quick high-level take on where we are in terms of the state of open source? So, there's been this race between open source models and closed source models. Depending on who you ask at what time, open source is about to catch up. Sometimes open source is just as good. Some people say no. What is your sort of realistic, pragmatic take on the current state of open source AI?

我认为它非常强大。2026 年也许是网络安全之年，但也很明显是开源 AI 之年。我的意思是，对我们来说，所有那些末日论者说开源无法保持接近前沿，我认为至少到目前为止，他们错得很离谱。我的意思是，很明显我们肯定没有 Mythos 级别的开源模型，但我们绝对有离 Opus 级别不太远的模型，或者取决于，它更不是尖峰式的。所以，你需要找到你的尖峰。有些人在那个尖峰上站立。但通常，它们现在绝对相当不错，而且至少在基准测试上，它们一直相当紧密地跟随前沿。也不像早期那样，基准测试唱歌，就像我们说的，当你的模型只在基准测试上表现好，但一离开基准测试就很差。很多这些模型在良好能力上相当通用。所以，是的，非常好。我认为现在我看到两个强劲趋势。第一个是，我看到公司有动向想要控制成本。所以，那里的讨论越来越多。

I think it's very strong. 2026 is maybe the year of cybersecurity, but that's also very clearly the year of open source AI. I mean, for us, all the doomers that were saying open source is not going to be able to stay close to the frontier, I think that at least up to now, they've been pretty wrong. I mean, it's also clear we don't have any Mythos-level open source model for sure, but we definitely have models that are not super far from the Opus category, or depending also, it's more not spiky. So, you need to find your spike. Some people stand on some spike on that. But typically, they are definitely pretty good right now, and they've been following rather closely the frontier, at least on the benchmarks. It's also not like it was maybe in the early days, benchmark singing, like we say, when your model is only good on the benchmark, but it's very bad as soon as you leave the benchmark. A lot of these models are pretty generic in their good capabilities. So, yeah, it's very good. I think there are two strong trends I would say I see right now. The first one is, I see a move in companies to try to want to control their costs. So, there's been increasingly discussion there.

也许 2025 年是“Token 最大化”的一年，你可以说：“嘿，你应该在 Token 上花的钱和付给员工的工资一样多。”今年人们意识到，我们实际上在工资上花了很多钱。所以如果我们花同样的金额，那基本上就是成本翻倍。这看起来事后很明显，但确实如此，而且不是每家公司都能承受成本翻倍。现在就说“我们要解雇所有人，然后搞智能体”也很愚蠢。我们都知道它们有时会偏离我们想要的方向，你需要人类来引导它们。所以我认为很多公司都在尝试找到我们经常看到的那种模式，即一种融合或外层模型：你用前沿模型做某些事，但找到一种聪明的方式，在不需要时优雅地回退到更便宜的模型处理更简单的任务。即使在我们的日常生活中，当你用前沿模型编程时，很多时候你让它派出子智能体，它们可能用性能较低的模型就能解决——可以用 Terra Luna 解决，可以用 Opus、Haiku、Sonnet 或 Haiku 解决。所以我认为每个人，即使在前沿和最接近的模型世界里，都在习惯使用不同类型的模型，而且很自然其中一些会成为非常划算的智能体。在这种情况下，大多数时候你会想用开源。还有很多情况是依赖强大的推理提供商生态系统。Firework 一直在墙上——所有的云，也许可以这么说——每个云都在经历疯狂的收入曲线，这首先反映了人们更多使用开源。所以我认为开源在保持接近前沿和推动更多采用方面过得很好。

Maybe 2025 was the year of token maxing, where you could say, 'Hey, you should spend as much on tokens as you're paying your employee.' This year people realized that actually we spend a lot of money on salaries. So if we spend the same exact amount, that's going to be basically doubling our cost. Which seems pretty obvious in retrospect, but it's quite true, and not every company can assume to double their cost. It's also pretty stupid right now to just say we're going to fire everyone and work on agents. We all know they sometimes go, you know, not directly in the direction we want them to, and you need humans to shepherd them. So I think a lot of companies are trying to find what we saw a lot, which is kind of a fusion or outer model where you use the frontier for something, but you find a smart way to gracefully fall back on less expensive models for simpler tasks when you don't need to. Even in our daily life right now, when you code with your frontier model, in many cases you ask it to spin out sub-agents, and they might be solved using lower performance models—can be solved using Terra Luna, can be fable using Opus, Haiku, Sonnet, or Haiku. So I think everyone, even at the frontier and the closest model world, is getting used to employing different types of models, and it's very natural that some of these could be really cost-effective agents. And most of the time you want to go to open source in this case. There are also a lot of cases for the strong ecosystem of inference providers. Firework has been on the wall—all the clouds, maybe it's called—every cloud has been increasingly having these crazy revenue curves that at first basically are translation of people using more open source. So I think open source is having a very good time in terms of staying solidly close to the frontier and driving more adoption.

关于第一点，即服务器企业中开源 AI 的采用，曾有一段时间人们把开源等同于免费，但我认为世界很快意识到，虽然模型可以免费下载，但部署和提供服务肯定不是免费的。您认为在企业实际应用中，开源的成本优势如何？

On the first point on the server enterprise adoption of open source AI, there was a moment in time when people associated open source with free, but I think the world has quickly learned that while the models may be free to download, deploying them and serving them is certainly not free. What is your sense of the cost advantage of open source in reality in the enterprise?

是的，这是一个很好的观点。把开源等同于免费成本，这种非常愚蠢的映射在这里也是错误的。而且情况要微妙得多，你是否有优势取决于很多因素。我认为开源的好处在于你有一个相当广泛的生态系统——成为云提供商的门槛相当低。所以在每 Token 成本上有很多竞争。这混合了很多因素，对吧？比如你租用数据中心的成本有多低？你买芯片能多便宜？这里实际上我们也有新的芯片公司即将进入市场，这将非常值得关注。所以基本上是你的硬件有多便宜，然后你能多大程度优化模型？你能量化它们吗？例如，我们用来对抗 OpenAI 入侵的模型是 GLM-5.2，它被 Nvidia 量化为 4 位。这是为了让它运行得更快、更小。所以你可以使用和探索很多策略来让这个模型更便宜。但同样，它们本身也不一定非要便宜。而且最接近的模型现在可能在某种程度上被补贴了。比如你花 20 美元订阅 ChatGPT 或 Claude 获得的 Token 数量，可能不是他们实际为你支付的 Token 的全价。所以围绕成本存在这种复杂的平衡。而你的开源模型云推理提供商可能没有那么多杠杆，以至于它能在订阅业务上亏钱。所以我们会看到更复杂的情况。我认为最终……

Yeah, that's a very good note. It is the same very stupid mapping of open source equals free cost that is also wrong here. And it's much more subtle, and you have an advantage or not. I think the nice thing about open source is you have quite a wide ecosystem—the entry barrier to be a cloud provider is pretty low. So you have a lot of competition on what's going to be the cost per token. And this is a mix of many things, right? It is how cheap are you renting your data center? How cheap can you buy your chips? And here actually we also have new chip companies who are going to come on the market, and this is going to be very interesting to watch. So basically how cheap is your hardware, and then how much can you optimize the model? Can you quantize them? For instance, the model we used to counter OpenAI intrusion was GLM-5.2, which was quantized by Nvidia in 4-bits. This is to make it faster and smaller to run. So you have a lot of strategies you can use and explore to make this model cheaper. But it's also true that they don't have to be cheap per se. And also that the closest model may be in a way subsidized right now. Like the number of tokens you get for your $20 ChatGPT or Claude subscription might not be the full price that they actually pay for your tokens. So there is this kind of complex balance around costs. While maybe your cloud inference provider for open source models doesn't have so much leverage that it can lose money on a subscription business. So we'll have something more complex. I think ultimately...

都由风险投资家补贴。

All subsidized by venture capitalists.

正是如此。我们理解，我想我们在很多交易中都看到了，对吧？但我们也知道这最终是暂时的，所以你不应该完全依赖它。这不是市场的均衡价格。

Exactly. We understand, I think we've seen in many deals, right? But we also know this is ultimately a little bit temporary, so you should not fully rely on that. This is not the equilibrium price of the market.

是的，有点像 Uber 现象，对吧？比如 IPO 前 Uber 便宜，之后 Uber 贵。

Yeah, sort of the Uber phenomenon, right? Like cheap Ubers before the IPO and expensive Ubers since.

我认为你想让生态系统在过去的 VC 市场中保持活力。

I think you want to keep the ecosystem alive for the past VC market.

关于第二点，中国模型与西方模型，来源地真的重要吗？最新的想法是什么？如果你的模型完全开放，你确切知道里面有什么，那么你就不应该担心，它是完全安全的。人们内心深处是否仍然认为中国模型可能有一些后门或诡计？这仍然是一个当前的问题吗？

On the second point, China versus Western models, does provenance actually matter? What is the latest thinking in terms of if your model is completely open and then you know sort of exactly what's in it, then you shouldn't worry, it's completely safe. Is there still a little bit of thinking at the back of people's mind that there might be some backdoor, some trickery to Chinese models? Is that still a current question?

是的，我的意思是，这当然是个好问题。关于开源的一点是，开源没有国界——你不能真的把你的开源模型限制为只能在地球的某个子区域下载。所以默认情况下，它是一种全球性的东西，然后你可能想了解来源和供应链。所以在这方面有一些工作，肯定是一个更陡峭的代理。我认为这可能研究不足。我认为这方面主要是 Anthropic 的工作，应该被复制并更深入地探索，以了解植入一种由提示触发的后门有多大可能。但说实话，即使是现在最接近的模型，我们也很难完全控制它们。所以我认为我们在这里非常清楚。

Yeah, I mean, that's a good question, of course. And the thing about open source is that open source doesn't know any border—you can't really keep your open source model restricted to download to just a subpart of the earth. So by default, it's kind of a global thing, and then you want to maybe understand the provenance and the supply chain. So there've been some work on this, definitely a steeper agent. I think it's probably under-researched. I think there's mostly work by Anthropic on that, and it should probably be reproduced and explored deeper to understand how much it's possible to implant a kind of backdoor that would be triggered by a prompt. But also, to be honest, even for the closest model right now, we have some struggle controlling them fully. So I think we're very clear here.

比如，我认为我们目前对如何控制最接近的模型也不是很清楚。

Like, I don't think we're very clear on how we control even the closest model at the moment.

嗯，所以，是的，我认为这当然有可能。我们还没有看到任何迹象。微调它们非常容易，而且你也可以大幅改变权重。所以，现在，如果你对模型进行更长时间的预训练和后训练，你很可能大幅改变它所拥有的权重。所以，我认为有很多方法可以规避这一点，这意味着目前我对这个问题的担忧程度，可能还不如对纯粹的奖励黑客行为的担忧，而后者我们已经看到实际发生了。但是，是的，我的意思是，就主权而言，我认为有时候人们对此有些困惑。我认为最重要的是谁掌握着开关，来决定是否触发你的智能访问。所以，我认为人们真正意识到这一点是在今年早些时候，美国决定 Stable 只对美国公民开放。我认为至少在欧洲和亚洲，那才是我们真正看到政府理解到有人掌握着触发器，他们可以说：“不，你不能再使用这个智能了，就像这个 token 一样。”我认为这才是关键。至少对我来说，这确实是主权的第一层级，即是否有人能决定你无法访问，比如在国家层面。这涉及两个方面：API 访问和数据中心。所以，如果你没有数据中心，那就意味着另一个国家可以说这个数据中心对法国公民不再可访问。所以，我认为这确实是你要考虑的第一件事，然后还有很多未来的问题，但你在构建你的技术栈时应该牢记这一点。所以，对于开源模型，你可以下载它们。一旦你下载了，任何国家都无法从你手中夺走。你可以自己托管。如果你在自己的数据中心（比如本地地面数据中心）上运行它们，我觉得你就开始构建一个主权技术栈了。然后还有很多关于后门和更复杂的问题，但这就是 2023 年的基本最低要求。

Um so, yeah, I would say it seems to me that it could be a possibility, for sure. We have not seen any indication of that. It's very easy to fine-tune them, and you can change quite a lot the weights as well. So, right now, if you pre-train and post-train a model for longer, you very likely change quite a lot the weights that it has. So, I think there's a lot of ways to circumvent that, which means that at the moment I'm a bit less worried about that than maybe just a pure reward hacking that we have actually already seen happening. But yeah, I mean, just generally on sovereignty, I think sometimes people are a little bit confused there. I think the most important thing is who has the hand on the switch to trigger or not your intelligent access. So, I think what people really realized earlier this year was when the US decided that Stable was only accessible to US citizens. I think at least in Europe and Asia, that's really the moment we saw governments understanding that someone had a trigger and they could say, 'No, you're just not allowed to use this intelligence anymore like this token.' I think that's the critical thing. At least for me, that's really the level one of sovereignty, which is can someone just decide that you don't have access, like at a country level. And this has two things: the API access and the data center. So if you don't have the data center, it means another country could say this data center is not accessible anymore to citizens of France. So I think that's really the first thing you should see, and then there are a lot of future questions, but you should build your stack keeping that in mind. So with open source models, you can download them. No country can take them away from you once you download them. You can host them yourself. If you operate them on your own data center, like a local ground data center, I feel like you start at the beginning of a sovereign stack. And then there are a lot of questions around backdoors and more complex stuff, but that's kind of the basic minimal thing in 2023.

作为开源乐观主义者，你是否担心西方开源发展的动机？中国在开源前沿显然有地缘政治动机。但如果你想想西方，你提到了英伟达，几周前我们请到了 Brian Catanzaro 来谈整个 Nemotron 项目。所以很明显，英伟达有动机这么做，那就是卖芯片，或者说拥有繁荣的开源模型是有意义的。但如果你想想其他所有人，就不太清楚西方公司为什么要做开源。这个反思可能是个例外，但据我所知，模型还没有发布。所以，你是否考虑过这种动机，以及这对西方开源的未来意味着什么？

And as an open source optimist, do you worry about the motivations for Western open source to thrive? So China has a clearly geopolitical motive behind being at the forefront of open source. But if you think of the West, then you mentioned Nvidia, and we had Brian Catanzaro from the whole Nemotron effort on the podcast a few weeks ago. So clearly there is a motivation for Nvidia to do this, which is sell the chips, or having thriving open source models makes sense. But if you think of everybody else, it's sort of unclear why Western companies would do open source. This reflection might be the exception, but the models haven't come out as far as I know. So, do you think about this motivation and what that means for the future of Western open source?

是的，当然，但对我来说很明显我们确实想要这样。而且我认为有动机的人比你想象的要多。我认为，只要你希望有一个繁荣的商业生态系统，让许多公司能够真正使用 AI，而不仅仅是作为另一家公司的薄包装，而是作为真正的 AI 构建者，我认为你就需要一些开源模型。所以，这也是美国政府最近表示实际上我们希望保持开源繁荣的原因之一。基本思想是，开源是获得新业务的最佳方式之一。所以它可以用于很多事情。它可以仅仅是因为它让你不会最终陷入基本上两家公司的寡头垄断，我的意思是，我们过去见过很多寡头垄断的案例。这对竞争力、价格，你知道，有很多危险。我认为，仅仅朝着“我们已经选出了赢家，这两家公司将为其他所有人构建 AI”的方向冲刺，从商业、经济、市场的角度来看，我认为这对我来说似乎并不理想。而且这也限制了很多发明。所以，举个例子，现在生物学领域有很多潜力。有很多新的生命科学公司。有很多公司想要探索这一点。由于护栏和关于生物黑客以及使用这个模型生成的问题，现在人们想要获取它，一旦你想问一些生物学问题，访问权限就非常非常有限。所以，基本上我见过的许多生命科学公司，如果他们想要处理任何与生物学相关的事情，都不得不转向其他选择。所以，如果你是一家新公司，探索那些大型实验室目前没有探索的东西，或者他们觉得可能没有足够的商业潜力来给所有人完全访问权限，或者我不太清楚。但基本上你无法真正使用它。所以，要么我们说从现在开始所有生命科学都将由 Anthropic、OpenAI 和 Google，也许还有 Meta 来构建，要么我们说我们想要一个围绕它的大生态系统。我的意思是，我个人观点，而且我有偏见，那就是你不希望在关键技术周围集中过多的权力，而且我觉得越多的人能够发明，我们就越能拥有多样化的新想法、多样化的新公司。但作为投资者，对吧？而且你是投资者，你知道我在说什么。比如，假设你只能投资 Anthropic 和 OpenAI。那有点可悲，对吧？有点无聊。我认为那只是增长阶段的东西。但你想投资新公司，你不想只投资于这种非常薄的包装。你想投资于那些真正能够构建 AI 的新公司。而且他们中的大多数，我认为我们在关键阶段看到了，他们中的大多数需要开源模型。另一个重要的例子是围绕游戏、视频甚至机器人技术的一切。大多数时候，他们所做的是从一个开源模型开始，然后在一些机器人数据上进行微调。例如，对于游戏，他们会采用一个开源的视频生成模型。他们需要做一些视频生成初创公司没有预测到的事情。他们需要添加动作。所以，他们所做的是在循环中加入动作和视频进行微调，这就是第一批有趣的实时游戏公司基本上是如何开始的。

Yeah, of course, but that seems pretty obvious to me that we actually want that. And I think a lot more people have incentive than you may think. I think as soon as you're interested in having a thriving business ecosystem with many companies being able to actually use AI, not just as a thin wrapper around another company, but as real AI builders, I think you want some open source models. So that's one of the reasons the US government just recently said actually we want to keep open source thriving. And the basic idea is that open source is one of the best ways to get new business. So it can be for many things. It can be just because it allows you to not just end up with an oligopoly of basically two companies, which I mean, we've seen many cases of oligopoly in the past. It's not always the best thing for competitiveness, for price, for, you know, there are many dangers with that. I think just rushing in the direction of saying we have made our winners and these two companies are going to build AI for everyone else, I think from a business, economical, market side of view, that doesn't really seem optimal to me. And then it also limits a lot of invention. So, just to take one example, there's a lot of potential right now in biology. There's a lot of new life science companies. There's a lot of companies wanting to explore that. Because of the guardrails and because of the questions around biohacking and using this model to generate, the access right now for people just to take it is very, very limited once you want to ask some biology questions. And so, basically most of the life science companies I've seen who were using small had to switch to another option if they wanted to be able to process anything related to biology. So, if you're a new company exploring something that the big labs are not currently exploring, or they don't feel like there is enough business potential maybe to give full access to everyone, or I don't know exactly. But basically you cannot really use that. So, either we say all life science is going to be built from now on by Anthropic, OpenAI, and Google, maybe Meta, or we say we want a big ecosystem around there. I mean, my personal opinion, and I'm biased, is that you don't want too much concentration of power around key technology, and I feel like the more people can invent, the more we have a diversity of new ideas, diversity of new companies. But also as investors, right? And you're investors, you know what I talk about. Like, let's say you could only invest in Anthropic and OpenAI. That's a little bit sad, right? It's a little bit boring. I think that stuff is only for growth stage. But you want to invest in new companies, and you don't want to invest only in this very thin wrapper. You want to invest in new companies that actually are able to build AI. And most of them, and I think we see them at our key face, most of them need open source models. Another big example is all the things around gaming, video, or even robotics. Most of the time what they do is they start from an open source model and then they fine-tune it on some robotic data. For gaming, for instance, they'll take a video generation model that's open source. They need to do something that was not predicted by the video generation startups. They need to add action. So what they do is they fine-tune with action in the loop and video, and that's how the first interesting real-time gaming companies started basically.

所以很多时候，开源是新公司起步的便捷途径：拥有自己的模型、用自己的数据微调、开始构建自己的 AI，而不是基本上把训练数据卖回给模型提供商。我认为这总是危险的，因为如果你把数据卖给他们，很多模型提供商可能某天会想进入你的领域。这种情况过去已经在法律、设计等许多领域发生过。

So, a lot of the time open source is your easy way as a new company to start to have your own model, to fine-tune on your own data, to be able to start to build your own AI and not just to basically sell your training data back to the model providers, which is I think always dangerous because a lot of these model providers might at some point want to enter your field if you're basically selling them your data. And this happened in the past already in legal, in design, in like many fields, I think.

回放一下，你对 7 月 24 日那封关于开放权重和美国 AI 领导地位的行业公开信的看法——那也是 Jensen 有史以来的第一条推文，你们显然签了名——部分是因为它很重要，部分是对正在形成的寡头垄断结构的抵制。所以，它既对世界有益，但背后的强烈经济动机，这么说公平吗？

To play it back, so your take on the July 24 letter industry letter on open weights and American AI leadership, which was also Jensen's first tweet ever, that you guys signed, obviously, is partly it's important and partly resistance to just an oligopoly structure that is being put in place. So, it's both it's good for the world, but the strong economic motivation behind it is that fair?

是的，我认为每个相信创造力、相信能在 AI 世界创造新事物的人，都会希望有开源访问的一部分。就像，你知道，如果所有代码都是闭源的，我们就不会有现在蓬勃发展的编程生态系统。这很明显，如果每个人想创建任何软件，都得去大型闭源软件公司工作，那软件行业里似乎根本不可能有这么多创造力和创新。我认为同样的事情正在发生。我不想忽视风险，我完全同意我们需要在对齐上努力，但我想说，就目前而言，开源模型处于前沿，这可能没有一些人想说的那么重要。

Yeah, I think everyone believing in inventiveness and being able to create new things also in the AI world would want a part of open source access. Just like the same, you know, if every code was closed source, we would not have the thriving coding ecosystem we have right now. And it's kind of obvious, like everyone would have to work at one of the large closed source software company if they wanted to create any software. That doesn't seem even really possible to have all the inventiveness and creation within the software industry. I think the same is happening in that. And I don't want to dismiss risk, and I fully agree we need to work on alignment, and I would say for now open source models being at the frontier, I think this is maybe less important than some people wanted to say.

也许在我们接近对话尾声时退一步，从你的角度感受一下世界可能走向何方。在你昨天关于 AI SI 的帖子中，你谈到了新一波实验室。特别是，你提到了 Jet Dean 昨天刚宣布的新公司的消息。我的意思是，昨天在世界上发布和宣布的所有事情方面，有点疯狂的一天。所以，重点是这家公司明确地朝着递归自我改进冲刺。那么，在我们描述的所有背景下，你对这种向自我维持、自我发展、递归 AI 的演变有多紧张？

Maybe to take a step back as we get near the end of this conversation and sort of get a sense for where the world might be going from your perspective. In your post yesterday about AI SI you talked about a new wave of labs. So, in particular you referenced the news of the Jet Dean new company that he just announced yesterday. I mean, yesterday was a bit of a crazy day in terms of like everything that came out in the world and was announced. So, the point being that this company is explicitly rushing toward the recursive self-improvement. So, in the context of everything we described, how nervous are you about this evolution toward self-maintaining, self-developing, recursive AI?

是的，这是个好问题。作为科学家和研究者，我绝对说我对这个想法非常感兴趣。我觉得他们想解决的很多超级智能问题，你知道，解决人类的关键挑战。我认为这是一个伟大的目标。我很乐意看到 AI 带来更多科学发现。我认为那可能是 AI 能带来的最有益的东西，而不仅仅是到处都是 AI 垃圾。所以，我认为我非常乐观。我只是觉得，我想说，过去几周 AI 引发了一点问题：我们到底有多擅长对齐这些模型。所以，就像法语里说的，我们不想把车放在牛前面。我们需要按顺序来。所以，是的，我想说现在的好消息是，这些大多数似乎是内部研究实验室。希望他们在部署产品之前，对自己所做的事情做好安全措施。他们考虑如何使用，并且对他们正在构建的东西的社会影响有很好的思考。但是，是的，我仍然认为我们应该真正理解如何将这种模型部署到人类世界中。

Yeah, that's a good question. And definitely as a scientist researcher I would say I'm very interested in the idea. I feel like there's a lot of this superintelligence that they want to tackle, you know, solving crucial challenges for humanity. I think this is a great goal. I would love to see AI making more scientific discoveries. I think that would probably be the most beneficial thing that AI could bring, more than just AI slop everywhere. So, I think I'm very optimistic. I just feel like, I would say, the past few weeks AI has raised a little bit the question of how good are we at aligning these models. So, like in French we say we don't want to put the carriage before the cow. Like we need to go in order there. So, yeah, I would say right now the good thing is most of these seem to be internal research labs. Hopefully they do good security around what they do before they deploy some of their products. They think about how it's going to be used and they have good thinking around the social impact of what they are building. But yeah, I still think that we should try to understand really well how we're going to deploy this model in the human world, I would say.

对，但你不属于那份请愿书的阵营，我想，那是在我们刚才谈到的 Nvidia 开放权重信之后四天出现的。有一封不同的信，有 1100 人签名，这次 Anthropic 和 OpenAI 都签了，要求政府帮助有意识地调整他们在 AI 研究中已经做的前沿步伐。所以基本上行业在要求放慢速度。你属于那个阵营，还是你认为那不是可行之道？

Right, but you are not in the camp of the petition that came out, I think, four days after the open weights letter that Nvidia did that we were talking about a minute ago. There was a different letter that came out with 1100 people and this time both Anthropic and OpenAI signed that asked governments to help deliberately pace the frontier of what they are already doing in AI research. So basically the industry kind of asking for slow down. Are you in that camp or you think that's just not the way it works?

我确实签了这封信。我同意我们应该这样做。而且，你可能也有同样的感觉，作为投资者，我有这种感觉：即使我们现在停下来，我们仍然可以在现有基础上建立相当好的公司。我觉得我们已经可以用这些模型做很多事情，而且它们已经非常有趣了。我觉得我们需要理解很多事情，我们应该在开放科学和分享它们如何运作方面做很多工作。所以我不属于“我们现在需要非常快速地冲刺”的阵营。主要问题是我们是否想稍微放慢一点。那也很好，因为也许那样我们就不需要每天有四个公告要混进一个播客里。

I did sign this letter. I agree that we should go there. I'm also, you probably have the same feeling, as an investor I have this feeling that even if we stop right now, we would still have quite good companies we could build on top of what we have right now. I feel like there's a lot of things we can already do with these models and they're already extremely interesting. I feel like there's a lot of things we need to understand and we should do in terms of open science and sharing how they work. So I'm not in the camp of we need to rush really quickly right now. The main question is if we want to slow down a little bit. Also that would be great because maybe then we don't have four announcements per day that we need to mix in one podcast.

也许我可以在十二月休一天假。但不是。

Maybe I can take one day off holiday in December. But no.

我认为主要问题是我们能做到吗，对吧？这才是关键问题。我想很多人会接受 AI 稍微慢一点、更开放一点、更关心一点、更反思一点，并试图更好地理解，你知道，如何真正做好这件事。但主要问题是我们如何谈判，如何组织一次放缓，而不会产生不良激励，你知道，只要一两个玩家不放慢，就会破坏整个放缓的效果。在这里，我不知道是否……是的，我认为这封信没有提供任何激励。它可能需要一些合作。有一篇很长的博客文章叫《AI 2040》。我不知道你读过没有。它也主张谨慎地放缓。也许，你知道，在完全刹车和尽可能快之间，以及我们监管一切以至于没人用 AI 之间——我认为这两个都是愚蠢的解决方案——但在这两者之间，我们尝试看看是否有一种方法能真正让步伐稍微慢一点。我认为那会很棒。我不认为我们会损失很多，而且我认为之后，在公司创建方面，所有这些，我们仍然可以有很多真正伟大的事情发生。但是，是的，我属于这个……我实际上对这两者都持同情态度，包括开源。我不认为开源本身必须是加速主义的。然后我们说这是减速主义的。

I think the main question is can we do it, right? That's the main question here. I think a lot of people would be fine with AI going a little bit slower, being a little bit more open, being a little bit more caring, a little bit more reflexive and trying to understand better, you know, how to do that really well. But the main question is how can we negotiate and how can we organize a slowdown there without having bad incentives where, you know, just one or two players not slowing down will kind of break the whole effect of having a slowdown. And here I don't know if... Yeah, I don't think the letter gives any incentives. It might need some collaboration. There was one long blog post called AI 2040. I don't know if you read it. It was also advocating for kind of a careful slowdown. And maybe, you know, somewhere between we go full brakes out, we go as fast as we can, and somewhere between we regulate everything so nobody uses AI, which I think are both stupid solutions. But something around we try to see if there is a way we could actually pace this a little bit slower. I think that would be great. I don't think we would lose a lot, and I think after, also in terms of company creation, all of that, we could still have a lot of really great things happening. But yeah, I'm in this... I'm actually sympathetic to both this and open source. I don't think open source has to be accelerationist per se. Then we're saying this is decelerationist.

我也不认为这是加速主义或通货紧缩主义的问题。我认为这些也是正交的。你可以支持开放科学，支持开放性，同时也可以认为我们确实需要理解如何训练好这个模型，并且我们现在就需要能够做真正的科学。

I don't think it's also this accelerationist or deflationist. I think these are also orthogonal. You can be pro open science, you can be pro openness, and you can also think that actually we need to understand how to train this model well, and we need to actually be able to do real science right now.

你不担心这是试图进行监管俘获吗？比如最顶尖的两家私人实验室实际上在试图让其他所有人都慢下来。我的意思是，你提到了并非所有人都会遵守的风险。但这实际上冻结了围绕谁是领导者、谁不是领导者的市场结构。

And you're not worried about this being an attempt at regulatory capture where like the top two private labs are effectively trying to figure out how everybody else can slow down. I mean, you mentioned the risk of not everybody just complying. But effectively freezing the market structure around who's a leader and who's not.

是的，我不认为它必须如此。我觉得你肯定也有同样的路径。那实际上是彻底的加速主义，你决定几家公司互相竞争，然后把其他所有公司都监管出局。我不认为监管必须等同于慢下来。而且问题更多在于如何将其付诸行动，如何实际实施，如何部署这种监管或合作。我认为 Demis 前几天在他卸任或升任首席科学家之前也做了一场不错的演讲。另一个我不太明白他现在会去哪里，但他的演讲基本上是关于国际合作的，在某些方面也非常支持开源。我认为你可以有一个非常开源的放缓。那是我希望看到的，即你慢下来，我们利用慢下来的事实来分享更多东西。我觉得竞争动态通常更多是关闭实验室的大门，对吧？所以对我来说，放缓可能更多是开放的机会。但当然，我的意思是，如果它最终主要是为了巩固，就像我们说的，一个只有两家公司的卡特尔或寡头垄断，我对那个方向并不太兴奋。

Yeah, I don't think it has to be. I feel like you have definitely the same path. That's actually fully accelerationist where you decide a couple of companies are racing against each other and you regulate all the others out. I don't think regulation has to be synonymous with slowness on that. And definitely the question is more how you put that into action, how you actually put that into practice, how you deploy this regulation or cooperation. I think Demis also had a pretty nice lecture the other day before he stepped down or up as chief scientist. The other one I don't really understand where he's going to be now, but his lecture for basically international collaboration was also very much pro open source in some aspects. I think you can have a slowdown that's very open source. That's the one I would love to see, which is you slow down and we use the fact that we slow down to be able to actually share more things. And I feel like a race dynamic is usually more in terms of closing the doors of the labs, right? So, to me, a slowdown is probably more the opportunity to open. But of course, I mean, if it turns out to be mostly a way to solidify, like we were saying, a cartel or oligopoly of just two companies, I'm not very excited about that direction.

太好了。这感觉是一个很好的结束点。Thomas，非常感谢你。这绝对精彩，我真的很喜欢。感谢你在休假期间抽出时间与我们交谈。非常感谢，感激不尽。

Wonderful. Well, that feels like a wonderful place to leave it. Thomas, thank you so much. This was absolutely fantastic. Really enjoyed it. And appreciate your taking some time to speak with us in the middle of your time off. So, thank you so much. Appreciate it.

谢谢，Matt。

Thanks, Matt.

嗨，我是 Matt Turk。感谢收听本期 Matt 播客。如果你喜欢这期节目，如果你还没有订阅，我们将非常感激你考虑订阅，或者在你观看或收听本期节目的任何平台上留下好评或评论。这真的有助于我们发展播客并邀请到优秀的嘉宾。谢谢，我们下期再见。

Hi, it's Matt Turk again. Thanks for listening to this episode of the Matt podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already, or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you on the next episode.