AGI 的看门人：AI 承保公司 Rune Kvist

如果你觉得 Fable 的担忧已经很糟，看看 Waymo 撞到狗窝时人们有多抓狂。想象一下，第一台机器人把幼儿从厨房桌上撞下来时，你会看到真正的严格责任。所以物理 AI，严格程度只会不断上升。嗯，这就是大图景：智能体、模型、机器人。随着技术进步，智能体的视野变长，新型故障模式会出现，同时也会带来新的价值创造方式，但风险面也更大。你会开始看到真正的智能体与智能体之间的交互，不再由人类中介。会有很多有趣的问题。你基本上需要一套新的法律体系。它们之间如何建立信任？

If you think fable concerns are bad, like see when Waymo hits a dog nest as if people lose their mind. Imagine when first robot knocks off a toddler off a kitchen table, you're going to see some real strict liability. So physical AI, the level of stringency just goes up and up and up and up. Um, so that's kind of like the big picture, agents, models, robotics. As the technology progresses, as agents get longer horizons, new types of failure modes will emerge that will also bring in just new kinds of ways to create value, but also more risk surface. You'll start to see true agent to agent interactions that are not mediated by humans. There's going to be a bunch of interesting questions. You're basically going to need a new legal system. How do they build trust amongst each other?

好的，我们和来自 AIU 的 Rune 在演播室，AIU 是一家 AI 承保公司，还有我们可靠的联合主持人 Vivu。欢迎。

Okay, we're in the studio with Rune from AIU, AI underwriting company, uh, with our trusty co-host, Vivu. Welcome.

谢谢。感谢邀请我。

Thank you. Thanks for having me.

谢谢。呃，你们今天宣布什么？

Thank you. Uh, what are you announcing today?

我们筹集了 4000 万美元，由 Ribbit Capital 和 Frost Money 领投。

We have raised $40 million led by Ribbit Capital and Frost Money.

我第一次注意到你是在 Nat 和 Dan 投资你们的时候。故事差不多是一样的吗？比如，你们今天的位置是你们当时预想的吗？

You first came to my attention when uh Nat and Dan invested in you guys. Is the story like pretty much the same? Like, are you today where you thought you were back then?

当我们筹集种子轮时，我们有一个假设：在某个时候，风险会抑制采用。当时，这感觉有点假设性。我认为现在这已经结束了。显然，随着 Mythos 和 Fable 的出现，时机就是现在。很明显，采用的真正约束就是风险。所以对我们来说，这感觉是同一假设的自然延续，但之前是推测，现在感觉是事实。

When we raised our seed round, we had a hypothesis that at some point risk was going to hold down adoption. At that point in time, that felt kind of hypothetical. And I think that that is now over. Clearly, the moment is now with uh Mythos and with Fable. It's pretty obvious that literally the binding constraint on adoption is risk. And so for us, it feels like this is a natural continuation of the same hypothesis, but where previously it was speculation, now it feels like fact.

让我们了解一下你们在 A 轮中重点介绍的客户名单。

And let's get the list of the customers that you um highlighting as part of your series A.

当然。是的。我们现在与 Cursor、Harvey、Lovable、11 Labs 等公司合作。

Totally. Yeah. So we are now working with folks like Cursor, Harvey, Lovable, 11 Labs.

是的。太棒了。恭喜。

Yeah. Amazing. Congrats.

谢谢。

Thank you.

你以最早被聘用的人之一而闻名，负责 GTM 和产品。我只是好奇，你进入 AI 的路径是什么？只是

So you were famously one of the first like the first uh hired and topic for GTM and product. I'm just kind of curious like what was your path into AI? Just

是的。

Yeah.

回顾一下。

Recap.

2021 年底，我卖掉了我的第一家公司，一家教育科技公司。我有一点时间思考下一步。我偶然看到了一篇缩放定律的论文，那就像闪电一样击中了我。我当时就想，这是一个大想法。简而言之，缩放定律论文只是说，模型越大，模型越聪明。

Late 2021 I sold a company, my first company, an edtech company. I had a bit of time to think about what was next. I came across a scaling laws paper and that just struck me like lightning. I was just like this is a big idea. In short, the scaling laws paper just says the bigger the model, the smarter the model.

这是 Kaplan 的那篇，不是 Chinchilla 的那篇。

And this is the Kaplan one, not the chinchilla one.

没错。就是那篇。对我来说，关键点是，哦，现在资本会理解这一点。如果你投入更多钱，你会得到更多回报。所以这会引发一个炒作周期。嗯，所以你会得到一种可预测回报的感觉，事实上也正是如此。所以我收拾行李。我从未去过旧金山。我收拾行李来到这里，寻找写那篇论文的人。嗯，当时他们刚成立了一个小实验室，叫 Anthropic。当时大约有 40 人。我喝了很多咖啡，直到最终被介绍给 Dario。当时他们正在纠结一些问题，比如我们应该部署我们的模型吗？我们应该创造收入吗？我们应该如何与外界互动？他们刚从 OpenAI 分离出来。嗯，公开报道称他们担心如何处理部署。所以，他们当时正在纠结这些问题。这就像早期的战争迷雾，大约 2022 年初。当时最性感的产品是 Jasper。就像外面什么都没有。所以，价值会累积在哪里？嗯，堆栈的不同部分会是什么，都是未解的问题。

Exactly. The cap one. And the important thing that clicked for me there was oh now capital will understand this. If you put in more money, you get more money out. And so that will kick off a hype cycle. Uh and so you'll actually kind of you'll get a sense of predictable returns which is in fact what's played out. And so I just packed my bags. I'd never been to San Francisco. I just packed my bags throughout here to find the people who had written it. Uh and at the time they had just started a small lab colanthropic. There was like 40 people at the time or so. Drank a bunch of coffee until I eventually got introduced to Dario. And at the time they were wrestling with some of these questions of like should we deploy our models? should we make revenue? How should we engage with the rest of the world? They just broken off from OpenAI. Uh, and it's been publicly reported that they were kind of concerned with how they were dealing with deployment. So, they were wrestling with some of those questions at that this point. This is like early fog of war, like early 2022. The sexiest product at the time was like Jasper. Like there's there's nothing out there. So, where is value going to acrue? Uh, what are going to be the different parts of the stack were all open questions.

但我想向大家强调，你问这些问题是因为你有 PPE 背景。呃，我实际上在新加坡参加过一个为 PPE 做准备的预备项目。所以我有一个导师。我们学习了哲学、政治学和经济学。但我觉得，像机器学习的人读了神经缩放定律论文，不一定会得出和你一样的结论，而任何资本家读了都会说天哪

But I want to highlight to people, you ask these questions because you have a PPE background. Uh I actually was in in Singapore in one of the sort of feeder programs for prepping people for PPE. So I had a tutor. We learned uh you know philosophy and politics and economics. But like I think you're kind of like machine learning people who read the neural uh scaling laws paper would not necessarily draw the same conclusions that you did whereas any capitalist would read that and go holy

没错。

Correct.

对。就像是的

Right. Like yes

呃，是谁让你注意到那篇论文的，因为那不是你通常读的论文，对吧？比如在你的圈子里。

uh who tipped you onto that paper because it's not a paper that you normally read, right? Like in your circles.

是的。我想，自从 AlphaGo 以来，我就对 AI 是一件大事有了一些认识。呃，但它感觉提出了所有这些有趣的哲学问题，但从远处看并不清楚它到底会走向何方。但足够明显的是，如果我们找到合适的机制让技术资本机器为此工作，这将会是一件大事。但只是不清楚。所以我认为，在某种程度上，这变得明显了，而且当时也不像现在这么明显，对吧？就像，哇，这太有趣了，但从哲学和经济学的背景来看，感觉如果这被证明是真的，你将与社会中的所有重大问题搏斗。你学到的关于政治的一切都被抛到窗外。你学到的关于经济学的一切至少受到挑战。所以有趣的是处于那个前沿，它影响着一切。所以这就是为什么我认为它理清了。

Yeah. I think I'd actually uh ever since Alph Go had had some appreciation that AI was a big deal. Uh but it kind of felt it raised all these kind of interesting philos philosophical questions, but it was kind of not clear from afar where exactly that would go. But it was obvious enough that it was like this is going to be a big thing if we find the kind of right mechanism to kind of get the technoc capital machine to work on this. but it was just not clear. And so I think it was some way in which like that became obvious and also it wasn't as obvious at the time than than it is now, right? Like it was just like wow this is so interesting but it still felt coming from kind of a philosophy and economics background it felt like if this turns out to be true you're going to be wrestling with all of the big questions in society. Everything you've learned about politics gets thrown out of the window. Everything you've learned about economics at least gets challenged. And so what felt interesting was to be at that frontier that has just ramifications across everything. So that's that's why I I I thought it sorted out.

我的意思是，对于不了解 PPE 的人来说，这显然是非常好的见解，PPE 项目就像是首相的摇篮。所以后来你遇到了 Jared。

I mean clearly really good insight for people people who don't know PP the PP program is like where prime ministers are born. So then you end up meeting Jared.

是的。呃，先是 Dario。是的。

Yep. Uh first Dario. Yeah.

是的。呃，我的意思是，那么你从与他们交谈中获得了哪些你最初假设中没有的额外见解？

Yeah. Uh well I mean like so did you get extra insights from talking with them that you didn't get from your original hypothesis?

如果你读了那篇缩放论文，你会得到一种非常模糊的草图，就像，哇，这似乎有点重要。有一些线条和一张图表。这似乎有点重要。嗯，我认为 Anthropic 团队比任何人都更多地思考了这个问题：这意味着什么？如果你真的推演下去，当时他们有关于 2026 年世界会是什么样子的愿景文件。它们生动详细地推演了需要多少算力，资本支出会是什么样子，会有哪些社会担忧，但也会有多少经济价值产生。所以感觉他们拿着一个水晶球，事后看来不仅非常正确。他们并不是像明显正确那样拿着它。他们只是说，非常非常认真地对待这个假设。

If you read the scaling paper, you get this like very vague sketch of like, wow, this seems kind of important. There are some lines and a chart. This seems kind of important. Um, and what I think the team at Anthropic had thought more about than anyone was like, what are the implications of this? If you really play this out, and back then they had kind of vision documents for what the world would look like in 2026. And there are kind of in vivid detail playing out how much comput is going to be needed, what is the capex going to look like, what are going to be some of the kind of societal concerns, but also what is the amount of economic value coming out here. And so it kind of felt like they held a crystal ball that in hindsight not just be dramatically correct. And they weren't holding it like they were obviously correct. They're just like take this hypothesis really really seriously.

想清楚

Think it through

并想清楚。

and think it through.

同样地，那种情境意识现在遍布街头。是的。天哪，我们都在同一个一平方英里内，对吧？那已经是几年前的事了，但最近几周人们不断提到 Fable 和 Methos，就像哇，如果你认真对待这个想法并扩大规模，很多事情就都说得通了。记住，此时这还是同一个团队，他们做了 GPT-1、2 和 3，对吧？这不仅仅是实验，这是一个我们刚刚扩大规模的真实模型，他们深信不疑，再次相信这个宏大的想法：如果你给一大堆算力和数据，它就想学习，从中会产生越来越聪明的模型，所有细节都不清楚。是的。是的。所有影响都不清楚，但那种深刻的信念就是这个核心论点，那有点令人眩晕，既极其有趣和令人兴奋，也很快让你觉得，如果这个假设成立，我们今天所知的世界将不再是这样，所以感觉在某种宏大意义上很重要。

In the same way, the kind of situational awareness that is now across the streets. Yeah. Oh my god, we're all in the same one square mile of right and that's now a couple years old but also people keep referencing it these particular weeks with Fable and Methos and it's like wow if you take this one idea seriously with the scale a lot of things fall into place. And keep in mind at this point this is the same team that had did GPT 1 2 and three correct which is also like it's not just some experimentation like there this is a real model that that we just scaled up and they had deep conviction in in again in this like big if you take a big blob of compute data it just wants to learn and out of that will come smarter and smarter models and all the particulars were not clear. Yeah. Yeah. and all the implications were not clear but that deep conviction is this like core thesis and that was kind of dizzying it's both phenomenally interesting and exciting and also very quickly you got to like the world we know today will no longer be this if this hypothesis holds so felt like important in some kind of grand sense

是什么塑造了你在那里的经历？那是 2022 年初，不仅 GPT-1、2、3 出来了，而且你知道 Anthropic 那些了不起的联合创始人从未分裂。他们是唯一有信念离开 OpenAI、创办自己实验室的人。你说那里大约有 40 人。那里的时光是什么样的？

What kind of shaped you there so that was your early 2022 not only had GPT123 come out but you know the amazing co-founders of anthrop rope that have never split up. The only ones they actually had the conviction to leave OpenAI, start their lab. You said there were about 40 people there. What was the time like there?

那和今天从外面看起来的样子非常相似。极其团结，极其以使命为导向，并生活在他们两种想法之间的张力中，即 AI 既可能发展得很好，也可能很糟，而我们想参与建设它。这产生了巨大的张力，他们也在应对这种激励挑战，他们知道自己身处一场竞赛中，可能会被迫偷工减料，但对他们来说，站在技术前沿也非常重要，所有这些想法当时都存在。感觉那条线一直非常非常清晰。嗯，我想无论爱他们还是恨他们，他们真的坚持了自己的信念。他们持有的一套核心信念比大多数公司持有的任何信念都要深刻。是的。

It was kind of remarkably like what it looks like on the outside today. Extremely cohesive, extremely missionoriented and living in this tension between their two ideas, which is AI could both go really well and really bad and we want to be part of building it. that creates astounding amounts of tension and they were wrestling with this incentive challenge where they know they're kind of there's a race that they're in where you might get forced to cut corners but it also felt very important to them to be at the forefront of technology and all of those ideas were just present at that time. It kind of feels like that line has been just very very clear. Um and I think kind of love them or hate them they have really stuck to their guns. There's a core set of beliefs that they hold more deeply than most companies hold any beliefs. Yeah.

快进到今天。这如何引向 AI 承保公司？你在做什么？是什么促使你开始这个？

Fast forward to today. What does that lead us to AI underwriting company? What are you up to? What what motivated you to start this?

是的。AI 通过标准和保险为前沿 AI 构建信心基础设施。从 Anthropic 到构建信心基础设施的联系。从 Anthropic 办公室的窗户望出去，看到 Waymo 已经驶过。早在 2022 年初，Waymo 在某些方面就像汽车的 AGI，它们是超人司机，但你不能带一辆去机场，现在 400 年后，你仍然不能带一辆 Waymo 去机场，尽管现在每个人都看了证据，认为它们比人类司机更好。所以在那个特定例子中，清楚的是，AI 有用的约束不是能力，而是责任、风险或信任。这个问题是普遍的。现在 Fable 不开放访问的原因不是因为它不是一个好模型。而是因为它是一个非常好的模型。只是很难承诺它会或不会做什么。而且这个问题随着 AI 变得更好而变得更糟。基本上，更智能的 AI 可以更自主。那更有价值，但风险面也增长。所以 Waymo 说明的是，除非你构建信心基础设施来对 AI 做出承诺，或至少揭示风险，否则你会阻碍采用。政府、银行、医院、军队需要了解 AI 会做什么和不会做什么，才能操作并整合它。这就是我们试图解决的问题。现在为什么是标准和保险？嗯，如果你追溯这个问题到历史，每一次技术浪潮都有这个问题的某种版本。所以如果你回到 1900 年，电力出现。本·富兰克林，汽车烧毁，抱歉，房屋烧毁，很多人死亡。1930 年代，汽车是大事件，杀死很多人。50 年代，私人核能是大事件，带来大风险。在每一个例子中，市场都跑在监管前面创建信心基础设施，因为那是做出 go/no-go 决策所必需的。它们对于采用是必需的，而市场从根本上想要采用。在所有这些例子中，标准和保险之间出现了共同的蓝图。原因是这两个组成部分：标准提供道路规则，它们还指定需要运行哪些测试，以便我们了解风险有多高。以汽车为例，就像车祸。很好。它们今天告知你的保险定价。它们告知你的购买决策等。那基本上是风险框架。保险公司很重要，因为它们买单。所以它们是最有激励去真实量化风险的私人机构，然后找出所有降低风险的方法，因为那会增加它们的利润。所以它们基本上帮助塑造激励，这两个很好地协同工作。现在这如何体现为一个公司？嗯，几年前甚至开始变得明显的一件事是，前沿公司，我们今天的一些客户如 Cursor、Sierra、11 Labs、Harvey，将很容易向银行销售试点。它们本身就像魔法，但如果你想在银行或医院全面推广，你必须通过风险流程。这些银行甚至不知道问什么问题，更不用说哪些答案足够，更不用说如何测试这些智能体是否真的按预期工作。所以它们有这个问题：我们能说什么来赢得信任？我们认为有一个黄金句子，大致是：“嘿，我听说你真的很担心幻觉或越狱或其他什么。”我们已经让独立第三方按照黄金标准测试了我们。我们以优异成绩通过，作为信心的投票，世界上最保守的保险公司已经查看了数据，并愿意承担一些风险到它们的资产负债表上。是的。

Yeah. AI built confidence infrastructure for Frontier AI through standards and insurance. The link from Anthropic to building confidence infrastructure. Looking out the windows at Anthropic offices and seeing Whimos driving by already. back then early 2022 ways were in some ways like Agi for cars like they were superhuman drivers but you couldn't take one to the airport and now 400 years later you still can't take a way to the airport despite now everyone having kind of looked at the evidence and being like they're better drivers than humans so in that particular instance what's clear is that the binding constraint on AI being useful is not capability but instead liability or risk or trust that problem is uh general The reason why right now Fable is not open for access is not because it's not a good model. It's because it's a very good model. It's just hard to make promises about what it will or will not do. And this problem gets worse as AI gets better. Basically, more intelligent AI can be more autonomous. That's more valuable, but also the risk surface grows. And so the what a way more illustrates is that unless you build the confidence infrastructure to make promises about AI or at least bring light to the risks you grind adoption to halt. Governments, banks, hospitals, militaries need to have some sense of what AI will and will not do to be able to operate for them to incorporate it. And that's the problem that we're trying to solve. Now why standards and insurance? Uh if you trace this problem back through history, every technology wave has had some version of this problem. So if you go back to like your 1900, electricity comes out. Ben Franklin, cars burn down, sorry, houses burn down, lots of people die. 1930s, cars are a big deal, kill lots of people. 50s, private nuclear energy is a big deal, poses big risks. In each of those instances, the market runs ahead of regulation to create confidence infrastructure because that's required to make go no-go decisions. They're required for adoption and the market fundamentally wants adoption. And in all of those instances, common blueprint emerges between standards and insurance. The reason it's these two components is standards kind of provide the rules of the road and they also specify like what are the tests that need to be run so we can get a sense of how high the risk is. So taking the case of cars, it's like a car crash. Great. Everyone they inform your insurance pricing today. They inform your purchasing decisions etc. That's basically the risk framework. The insurers are important because they pick up the bill. So they are the private institution that is most on the side of is best incentivized to quantify the risk truthfully and then figure out all the ways to reduce the risk because that increases their profit. So they're basically they help shape the incentives and these two worked really well in Unison. Now how does that show up as a company? Well, one of the things that was obvious even or starting to become obvious even a couple years ago was that frontier companies some of our customers today like Cursor uh Sierra 11 Labs Harvey were going to have a very easy time selling a pilot to a bank. them like the the damage itself itself is magic but bringing that through if you want to do a wall to wall roll out at a bank or a hospital uh you have to go through the risk process. These banks have no idea even which questions to ask let alone which answers are sufficient let alone like how do they go and test whether these agents actually work the way they're supposed to. And so they they have this problem of like what can we say to earn the trust? And we think there's like a golden sentence that goes something like, "Hey, I hear you're really worried about hallucinations or jailbreaks or whatever it may be." We've had an independent third party test us against the gold standard. We pass the flying colors and as a vote of confidence, the world's most conservative insurers have looked at the data and are willing to take some of the risk onto their balance sheet. Yeah.

所以如果出了什么问题，背后有钱。是的。

So if something does go wrong, there's money behind it. Yeah.

没错。那就像是所有这些之间的联系。我们可以深入一些与技术测试相关的难点，我认为那是问题的关键。嗯，但我先停在这里。

Exactly. That's kind of like the link between all of those. We can we can get into some of the the hard parts related to the technical testing which is I think the crux of the matter. Uh but I'll pause there.

你和 Richie 是怎么走到一起的？你总是显得非常自信，你们在宣布 A 轮融资什么的，但我想看看最初的想法形成阶段。

How did you and Richie come together? You always come across very confident, and you're announcing your Series A and all these things, but I want to see the early initial stages of idea formation.

是的。Rajiv 其实是我未来的姐夫。

Yeah. Rajiv is actually my soon-to-be brother-in-law.

哦。

Oh.

所以再过一周半，我就要和 Rajiv 的姐姐结婚了。

So I'm actually in a week and a half getting married to Rajiv's sister.

好吧，现在你们关系很紧密了。

Okay, now you're tight.

没错。现在，你知道，Raj 和我认识十年了。有趣的是，我是在伦敦麦肯锡实习时认识 Rajiv 和他姐姐 Hannah 的，当时 Rajiv 被指派为我的导师。所以是同时认识他们的。很长一段时间里，我们并不一定会一起工作。我当时在创业公司，他是麦肯锡的保险合伙人。三四年前，我想是 Hannah 说服了他 AI 会是一件大事，于是他辞去了伦敦麦肯锡的舒适合伙人工作，收拾行李去了旧金山，最终加入了 Meter。

Exactly. Now, you know, so Raj and I have known each other for a decade. Funny story, I met both Rajiv and his sister Hannah at the same time when Hannah and I were interns at McKinsey in London and Rajiv was assigned as my mentor. So met them at the same time. For the longest time, it was not obvious that we were necessarily going to work together. I was in startups. He was an insurance partner at McKinsey. Three or four years ago, I think Hannah convinced him that AI was going to be a really big thing and so he quit his cushy partner job at McKinsey in London, packed his bags to San Francisco and ended up joining Meter.

你们可能在网上足够活跃了。

You guys are probably online enough.

没错。我们看到智能体可以承担的任务范围的图表正在极快地翻倍。所以在这里，零号领导了与 Anthropic 和 OpenAI 的合作，在模型发布前进行测试，同时也与美国和英国政府密切合作，弄清楚如何判断一个模型是否可以发布，这在某种程度上是完美的背景。他在保险行业待了很长时间，了解那个世界，也花了大量时间在前沿模型测试上。所以当我在这个想法空间里摸索时，从我们谈到的与 Waymo 相关的一些想法开始，一旦我们进入内容，我们都觉得，“哦，这将是一个很棒的一起打造的业务。这就像是在解决我们都认为世界上最重要的问题。”从市场角度，我们的直觉是市场可以做很多事。AI 发展得越快，政府解决这些问题就越难。然后我们花了一点时间来处理与家人一起工作是什么感觉。

Exactly. We see the chart of the horizons of the task that agents can take on is doubling extremely fast. So here zero there led their partnerships with Anthropic and OpenAI to test their models before release but also working closely with the US and UK government to figure out how do you know whether a model can be released and in some ways that's like the perfect background. He's spent a lot of time in insurance, knows that world, spent a lot of time with frontier testing of models. And so when I was bumbling around this idea space, starting with some of the ideas we talked about related to Waymo, as soon as we got into the content, we're both like, "Oh, this would be an amazing business to build together. This is like wrestling with the problem that we both think is the most important in the world." From a market angle, which is kind of our intuitions is that the market can do a lot. And the faster AI moves, the harder it is for government to solve some of these problems. And then it took a little bit of time to work through what is it like to work with family.

呃，因为你们当时已经在约会了吗？

Uh and uh cuz you're already dating at the time or

是的。是的。没错。当时我们就感觉像一家人了，所以一起创业感觉像是一大步，呃，现在我们在这里，有着巨大的信任。

Yeah. Yeah. Exactly. Already back then it was we felt like we're family and so starting a business together felt like kind of a big step and uh here we are with just immense amounts of trust.

是的。那么现在你们公司有多大？你们现在有多少人？

Yeah. So now you're a company of how big? How big are you guys now?

我们现在只有 20 个人。

There is just 20 of us now.

你们现在有 A 轮融资，并且你们有了第一个认证 AIU1。嗯，让我们把认证调出来。所以这是智能体认证，对吧？这个过程包括什么？我这里有两个问题。一是带我们了解一下认证，二是公司获得认证的流程是什么？

Tony of you guys now have Series A and you have your first certification out the AIU1. Um let's bring up the certification. So this is the agent certification, right? What goes into the process? I have like two questions here. One is walk us through the certification and two is what is the process for a company to get certified?

你知道，很好。正如顶部所说，A1 是智能体安全、保障和可靠性的标准。基本设计原则是，把所有拖慢采用的问题——所有让财富 1000 强安全负责人夜不能寐的问题和恐惧——都纳入一个全面的框架。这就是你会在那里看到的。你可以看到六个类别。第二，你想把所有这些都建立在技术测试的基础上。所以安全标准的一个担忧是，它们常常感觉像是做戏的文书工作，实际上并没有落到实处——这些有用吗？这些重要吗？所以我们从一开始就坚信，关键是要通过这个，你必须每季度接受测试，基本上运行数千次模拟，看看它是否真的能被越狱，越狱有多难，它多久会产生幻觉，多久会泄露数据等等。然后最后一个核心想法，如果你滚动到顶部，就是每季度更新一次。

You know, great. As it says right at the top, A1 is a standard for agent security, safety and reliability. The fundamental design principle is take all of the concerns that slow down adoption. So all the questions, all the fears that keep security leaders in the Fortune 1000 up at night and put them into one comprehensive framework. That's what you'll see there. You can see the six categories. Two, you want to ground all of this in technical testing. So one of the concerns with security standards that often feel kind of like theater paperwork is that they don't actually ground out in does any of this work? Does any of this matter? And so we had a conviction from early on that that was going to be the kind of crux was to pass this you must get tested every quarter basically run thousands of simulations to see well so can it actually be jailbroken how hard is it to jailbreak how often does it hallucinate how often does it leak data etc. And then the last core idea here if you scroll up to the top here is to refresh it quarterly.

所以 AI 的核心特征是它发展极快。我们今天讨论的担忧和三个月前不一样，而且会不断变化。通常标准以十年为周期更新，显然行不通。但问题是如何更新它？这里的核心是让财富 1000 强的风险负责人围坐一桌。如果你看左边，你会看到 AS1 联盟。该联盟是一群风险负责人，他们经营着真正的银行、真正的医院、真正的关键基础设施，每天都在面对这些挑战。我们每季度与他们见面两次，听取他们最关心什么、什么让他们夜不能寐。对这种对话有巨大的需求，然后我们将其操作化为一个具体的标准，实际上我们可以进去看看标准到底是什么。如果我们回到左边的介绍，向上滚动一点到轮盘。点击可靠性。如果我们以幻觉为例，幻觉系统可靠性，这里有许多要求。如果你进入第一个防止幻觉要求，呃，幻觉输出，这是一个特定的要求。这是一个技术控制。基本上我们想要某种接地过滤器。你在这里看到的第一件事叫做交叉映射。所以每个人和他们的祖母都发布了一个框架，非常高层次的框架，关于 AI 风险是什么。这基本上是你们的竞争对手。但在某些方面，我们的竞争对手实际上是我们的朋友。我们稍后会回到为什么，但把所有东西映射在一起，这样你就有一个超集，你试图在这里支持的主张是，呃，如果你遵循这个框架，那么你也可以看到你如何遵循其他框架，但核心在于这里的控制活动和证据，所以控制活动就像是，很好，你有这个高层次的要求，你如何把它转化为可操作的东西，这是你必须做的，然后我们寻找的证据是什么，我们之所以深入到这个程度，是因为实际上对于 AI 中的重大担忧并没有太多混淆。每个人都同意这些。

So the core trait of AI is that it moves extremely fast. whatever concerns we're discussing today were not the same ones three months ago and this will keep changing. Typically standards update on like a decade cycle is obviously not going to work. But the question is kind of how do you update it? And the core thing here was to basically get the risk leaders of the Fortune 1000 around the table. So if you go over to the left here you'll see AS1 consortium. The consortium is a group of risk leaders who run real banks, real hospitals, real critical infrastructure who are facing these challenges every day. And we meet with these folks twice a quarter and hear what's top of mind, what is keeping them up at night. There's tremendous amount of desire for that conversation and then we operationalize that into a specific standard that gets into and actually we can go into and look at what is what even is a standard. So if we go back to introduction out there to the left, scroll up a little bit to the wheel. Click into reliability. So if we take something like hallucinations hallucination system reliability there is a number of requirements here. If you go into the top one prevent hallucinate requirements uh hallucinate outputs this is one particular requirement. This is a technical control. Basically we want some kind of groundedness filter. The first thing you see here is what's called a crosswalk. So everyone in their grandmother has put out a framework very high level framework for what are the air risks. This is basically your competition. But in some ways our competition we're in fact friends with them. and we'll come back to why but mapping everything together so you have one superset the claim you're trying to support here is uh if you follow this framework then you can also see how you follow the other frameworks but the meat of it comes down here in control activities and evidence so control activities is like great you have this high level requirement how do you turn that down to something operational here's what you must do and then what is the evidence that we're looking for and the reason we go this deep is that there's actually not much confusion about what are the big concerns in AI. Everyone agrees to these.

问题就是，你实际上应该做什么？我们发现很多人需要的是，把人们需要寻找的具体证据讲清楚。不管你是 Cursor 在开发东西，还是摩根大通在开发东西，又或者你只是摩根大通的一个风险负责人，你到底该要求什么？你能要求什么而不显得愚蠢？比如，如果你要求一些——你不敢相信，有多少次风险负责人向 Cursor 之类公司要求底层模型的 IP 权利，然后你就只能说——

The question is like, what are you actually supposed to do? And so what we found a lot of demand for is getting down to the specific evidence that people need to look for, whether you are Cursor building something, or even JPMorgan building something, but also if you're just a risk leader at JPMorgan, like what exactly should you ask for? What can you ask for without sounding stupid? Like, if you ask for some—you won't believe the amount of time a risk leader has asked for the IP rights to the underlying model to Cursor or something, and you just like—

抱歉，什么？

Sorry, what?

你就把它塞进去，看他们注不注意。

You slip it in there and see if they notice.

没错，就把它放进问卷里。所以这就是标准的意义，我们每个季度和这些人一起更新，以跟上最新的关切。

See exactly, put them in the questionnaire. So that's kind of what a standard is, and we update this every quarter with these folks to keep up with the latest concerns.

我能就这一点再深入问一下吗？

Can I double click on this one?

可以。

Yeah.

首先，网站很漂亮。它特别能给人信心，而这正是重点，就是：好，我知道跟你谈的时候我到底在签什么。我甚至不用跟你谈，就能看到你整套认证，这很棒。但比如说，从这里看，D001.1 配置、ground filter，这是怎么应用的？是你有个人去逐条过吗？如果你往回看——

So, first of all, the website's beautiful. It's so confidence inducing, which is the whole point, where like, okay, I know exactly what I'm signing up for when I talk with you. I don't even have to talk to you. I can just see your whole certification, which is great. But like, okay, so from here, like D001.1 config, ground filter, how does that get applied? Like, you have a person that goes through it. If you go back—

我确实看到某处写着，你知道，51 项要求、130 项控制，有一整套——

I did see somewhere there's like, you know, 51 requirements, 130 controls, there's like a whole—

对，我只是——对我来说这并不能转化成一个测试，或者——

Right, I just—to me this doesn't translate into a test or—

对，对，对。如果你进入，在左侧——其实，在我们进去之前，有三类要求。第一类是技术控制，比如你必须实现一些护栏。第二类是测试控制，你必须有一个独立的第三方对你跑一些测试。我稍后会给你看其中一个。第三类是政策控制。比如，你必须有一个名字挂在线上的人，当你们——

Yes, yes, yes. So if you go into, on the left hand side—so actually, before we go in there, there are three types of requirements. The first is technical controls, like you must implement some guardrails. Two, there are test controls, so you must have an independent third party go run some tests against you. Well, I'll show you one of those in a second. And then three, there are policy controls. For example, you must have a person whose name is on the line when you guys—

而且你必须有一个计划，说明你如何告知客户、如何与他们互动。

And you must have a plan for how you tell your customers and how you engage with them.

这些是比较传统的标准类内容。所以在这个具体例子里，我们只是检查他们是否确实有 ground filter。我们会和审计方合作。我们和 KPMG 或 Schellman 这样的审计方合作，他们进去做审计该做的事，也就是核查证据。在这个例子里，可能是一张截图，也可能是他们需要审查的一部分代码，看它是否确实——它是否存在。

There are kind of more traditional standard type stuff. So in this particular instance, we just check whether they in fact have a ground filter. So we will partner with an auditor. So we partner with auditors like KPMG or like Schellman who go in and do the thing auditors do, which is to check the evidence. In this case, that might be a screenshot. It might be part of the code that they need to review to see that it actually just—that it exists.

然后第二件事——所以你们不是在测试它的有效性。

And then the second thing—so you're not testing the effectiveness of it.

那是第二件事。如果你往下看左侧的第三方幻觉测试，那基本上就是下一项要求。这里我们测试它实际效果到底有多好。

That's the second thing. So if you go down to the third party testing for hallucinations out on the left, that's basically the next requirement. This is where we test how well does it actually work.

好。那是你们测试还是审计方测试？

Okay. And is it you testing or the auditor?

我们测试。

We test them.

啊。

Ah.

我们测试。

We test them.

那工作量很大。测试要多久？如果我想获得认证，端到端大概要多久？

That's a lot of work. How long does testing take? So if I want to get certified, just how long does the end to end roughly take?

是的，端到端几乎总是取决于我们的客户需要向我们学习一些东西。大概需要三到十周，取决于他们目前有多达标。有些人来找我们时已经有极其严格的安全计划，我们测试时效果极好，我们可以很快完成。有些人来找我们时还没那么成熟。我们会给他们一个需要达到的规范，然后他们的安全团队和工程师开始工作，按标准构建。测试本身通常需要几周，包括他们修复的时间。我们经常会发现一些无法通过的地方，就是嘿，这其实不达标。你不会通过标准，然后他们需要去实施额外的保障措施或额外修复，让自己更稳健，这样他们才能真正问心无愧地看着客户的眼睛说，嘿，我们确实尽了最大努力。

Yeah, the end to end almost always is dependent on like our customers need to learn something for us. It takes somewhere between like three to 10 weeks depending on how up to snuff they already are. So some people show up to us with like extremely rigorous security programs when we test and it works extremely well. We can get that done very quick. Some people come to us and they're not that far along. We give them kind of the spec that they need to build towards and then their security teams and engineers get to work and build to meet the standard. The testing itself typically takes a couple of weeks including the time for them to remediate. Often we'll find something that we cannot pass where hey this is actually just not up to the standard. You won't pass the standard and then they will need to go and implement additional safeguards or additional remediation that makes them more robust so that they can actually kind of hand on heart look at their customers in the eyes and say like hey we've done truly our very best.

然后他们认证一年，并且每季度更新。

And they're certified for a year and have quarterly updates.

对。是的。

Correct. Yeah.

是的，这挺有意思。我想，你知道，从那以后有什么变化？所以这是在认证生产环境中的智能体，对吧？你的客户，比如 Lovable、11 Labs、Intercom Fin，他们都通过了这个认证。有什么变化？我看到你发帖说，你知道，第二季度增加了 MCP 智能体、智能体通信。自第一版以来，还有什么你想强调的？每季度会加入什么？

Yeah, it's pretty interesting. I think, you know, what's changed since? So this is certifying agents in production, right? Your customers like you've had Lovable, 11 Labs, Intercom Fin, they've all gone through this certification. What has changed? So I see you post like, you know, Q2 added MCP agent, agent communication. Any other things that you want to kind of highlight since the first iteration? What comes in quarterly?

是的，有些变化只是——智能体不是单一的东西。比如，如果你拿 Cursor 这样的智能体和 Sierra 比较，它们真的很不一样。再和 Harvey 比，再和 UI 比，和 11 Labs 比，它们都很不一样。所以我们想设计一个适用于所有类型智能体的标准。我们从一种相当文本化、老实说相当聚焦客户支持的标准开始。那里有大量现有需求。然后随着时间推移，我挑选了其他各领域的一些前沿公司合作，把标准扩展出去。这样我们就知道，同一个标准适用于代码、适用于客户支持、适用于自动化等等。这是一大变化。然后最近最受关注的一些事情——Mythos 给安全负责人带来了很多担忧。我们开始收到越来越多关于智能体与智能体交互的问题。目前还非常初期，但已经开始出现。有很多关于 Open Claw 和 MCP 的问题，同样，智能体开始彼此交互确实是重中之重。然后随着编码智能体真正起飞，银行和医院等也在越来越精确地明确他们需要什么。所以随着 AI 开始成为世界上大部分 token 流动的地方，我们也在真正校准、变得更加精准。

Yeah, so some of the changes have just been—agents are not just one thing. So like if you take agents like Cursor and compare them to Sierra, they're really quite different. And compare them to Harvey again, compare them to UI again, 11 Labs, they're all quite different. And so we wanted to design a standard that works for all of the types of agents. And we started with one that was like pretty text based, like honestly pretty customer support focused. That's where there's a lot of existing demand. And then over time I've picked some of the frontier companies in each of these other domains that we could work with and build out the standard. So such that we know that the same standard works for code, that works for customer support, works for automation, etc. So that's been one big thing. Yeah. Then some of the things that have been top of mind recently—mythos is bringing up a lot of concerns for security leaders. We're starting to get more and more questions around agent to agent interactions. It's very nascent at the moment but it's starting to emerge. There've been a lot of questions related to open claw and MCP again, like agents starting to interact with each other is really top of mind. Then as coding agents have really taken off, that's also where banks and hospitals etc. are getting more and more precise on what it is they need. So really dialing in as AI starts to be like where most of the tokens flow through in the world, getting much sharper on that.

你能给那些收听但不太思考这件事的人讲讲吗？比如你提到了一些显而易见的东西，你知道，幻觉、引用。人们在构建智能体时应该遵循哪些最佳实践？比如如果他们来找你时已经准备得差不多，你知道，他们可能通过认证。有哪些人们没想到但应该具备的东西？

Can you share for people that are listening that don't really think about this? Like you mentioned there's the obvious stuff, you know, hallucination, citations. What are best practices that people should do when building agents? Like if they come to you pretty ready with certif—like, you know, they'll probably pass certification. What are the things people don't think about that they should have?

最重要的一点是，很多公司没有做过认真的压力测试。他们把大部分时间，也许理所当然地，花在优化好情况、平均情况下它表现如何，以及给客户的输出质量有多高。

The most important thing is that a lot of companies have not done a serious stress test. They spend most of their time, perhaps rightly so, optimizing for how does it work in the good case, the average case, how high quality is the output for the customer.

很多这类公司都很新，所以它们没有花太多时间做压力测试，去设想对面作为对手会是什么样。有哪些复杂的边角案例是你没有真正考虑过的？所以我认为这是一种心态，你在初创公司也能看到。它们往往要过一段时间才会招第一个安全人员，而这是一种与单纯打造好产品完全不同的风险面。所以很多这种情况都适用。

A lot of these companies are pretty new, so they haven't spent a lot of time stress testing what is there as an adversary on the other side. What are some of the complicated corner cases that you've not really considered? So I think that's a frame of mind, and you also see this in startups. It often takes a while until they hire their first security person, and that's a whole different kind of risk surface than just building a good product. So a lot of that applies.

大多数公司其实也有正确的架构。它们大多会有某种护栏，要么是模型提供商开箱即用的，要么是自己构建的过滤器放在中间。只是这些护栏效果不太好。区别在于，放一个分类器，也许去检查你是否在不该给医疗建议时给了医疗建议，然后说：“嘿，如果这看起来像医疗建议，就过滤掉。”很多公司都有这个。问题是它是否有效，而实际上要坐下来思考所有你可能询问医疗建议的方式是相当繁琐的。读读学术文献，了解有哪些框架或技巧可以让 AI 在你本不该得到医疗建议时给你医疗建议。所以，有一个专业领域是缺失的。

Most companies actually also have the right kind of architecture. Most of them will have some kind of guardrails in place, either some come out of the box from their model provider, or they'll have built their own filters that sit in between. They just don't work very well. The difference between putting a classifier in place that maybe goes and checks whether you're giving medical advice when you shouldn't and says, "Hey, if this looks like medical advice, filter it out." Lots of companies have that in place. The question is whether that works, and it's actually pretty fiddly to sit down and think about all the ways in which you could ask for medical advice. Read the academic literature on what are the kinds of framings or tricks you might play to get an AI to give you medical advice when you really shouldn't. And so there's an area of expertise that's just missing.

所以我们发现，大多数人都有正确的构建模块。这不是火箭科学，但棘手的是深入边角并测试它是否有效，这样你才能直视你的客户——可能是银行，也可能是医院——并说：这对你有效。

So what we find is that most people have the right building blocks in place. It's not rocket science, but the finicky thing is getting into the corners and testing whether it works, such that you can look your customers in the eye—who may be a bank or maybe a hospital—and be like, this is going to work for you.

我明白了。我们谈了很多关于智能体级别的认证。你们接下来要往哪里走？所以，在镜头外宣布 A 轮融资，我们谈过一点。还有 Fable 政府介入的整个安全风险。你们有点像是在宣布你们也要进入模型认证。

I see. So we talked a lot about the agent level certification. Where do you guys go from here? So announcing Series A off camera, we talked about this a bit. There's the whole security risk of Fable government stepping in. You guys are kind of announcing that you're also going into model certification.

等我们之后剪辑一下，我们暂时还不会宣布这个。

When we do a bit of cutting afterwards, we will not yet be announcing this.

现在大家最关心的问题是模型层面的。

The question that is top of everyone's minds now is at the model level.

而 Mythos 和 Fable 确实把这一点推到了前台：除了商业风险和智能体层面发生的经济安全风险之外，模型还将在国家安全类别中带来风险。问题的形态非常相似。如果出了问题，有些人要负责。在智能体的情况下，通常是企业中的安全负责人。在这种情况下，是政府。他们不一定一辈子都在思考这里会出现哪些新风险、你可能在找什么样的数据、你如何测试？但他们必须确保他们的担忧得到解决。你有一些技术性很强的前沿公司。他们对风险了解很多，但他们从根本上有一个动机，即不总是说实话。所以政府和实验室之间存在信任差距。在其他每个行业，最终都会有某种机构坐在中间——一个中立的第三方坐在那些人之间。没有其他行业允许人们自我审计。

And Mythos then Fable has really brought this to the fore: that in addition to the commercial risk and the kind of economic security risks that are happening at the agent layer, the models are going to present risk in the national security category. The shape of the problem is very similar. You have some people that are on the hook if something goes wrong. In the case of agents, it's often the security leaders in the enterprise. In this case, it's the government. They haven't necessarily spent their entire lives thinking about what are the new risks that come here, what is the kind of data you might be looking for, how might you test that? But they do have to make sure that their concerns are addressed. You have some frontier companies that are deeply technical. They know a lot about the risks, but they fundamentally have an incentive to not always be truthful. So you have a trust gap between the government and the labs. And in every other industry, you end up with some kind of body sitting between—a neutral third party sitting between those people. There's no other industry where you allow people to audit themselves.

所以，将需要一个第三方，既能以实验室的严谨性来运行前沿技术评估，又能以政府信任 PWC 进行财务审计的方式传达可信度，并且他们知道所有报告的输出方式是一致的、易读的、事实性的、值得信赖的。这两件事需要结合起来。我们从与智能体合作中学到的是，如果你想让双方之间的沟通顺畅，就必须有一个公开的、人们可以检查的共同标准。在每个风险中，哪些风险是重要的？你真正在寻找的威胁模型是什么？你需要为每个风险指定需要哪些护栏，以及需要运行哪些测试来查看这些护栏是否有效。然后你需要去运行审计，这些是技术审计，是一致的。

So there's going to be a need for a third party that can take the rigor of the labs to run frontier technical evals, but can also speak legible trust in the way that the government trusts PWC to go and run financial audits, and they know that they output all the reports in a way that's consistent, that's easy to read, that's factual, that's trustworthy. Those two things need to be brought together. And what we've learned from our work with agents is that if you want that communication between those two parties to be smooth, there has to be one common standard that is public, that people can go and inspect. What are the risks that matter within each of these risks? What are the kind of threat models that you're really looking for? You need to specify for each of those risks, what are the guardrails that need to be in place and what are the tests that you need to run to see whether those guardrails are effective. And then you need to go and run audits that are technical audits that are consistent.

所以，如果你想带来信任，有条不紊地逐一处理风险是极其重要的。你不能派一个研究员进去说，把你发现的东西带回来。你需要能够准确解释你做了什么、你尝试了什么、你没有尝试什么，以及因此你在最后能做出和不能做出什么样的承诺。

So if you're trying to bring trust, it's extremely important that you methodically work your way through the risks. You can't send one researcher in and say, come back with whatever you find. You need to be able to explain exactly what you did, exactly what you tried, exactly what you did not try, and therefore the kinds of promises you can and cannot make at the end of it.

我认为 Fable 是这个问题的一个直接症状：政府被告知存在风险。政府可能难以评估这个风险到底有多大。他们打电话给 Anthropic，而 Anthropic 试图告诉他们：嘿，实际上每个模型都可以被越狱。

I think of Fable as a direct symptom of this problem: the government was told that there's a risk. The government may struggle to assess just how big that risk is. They call Anthropic and Anthropic is trying to tell them, hey, actually every model can be jailbroken.

这不是你想听到的，对吧？

That's not what you want to hear, right?

作为政府，这可能很难信任，我们认为中间人是最自然的解决方案。在其他市场，你会看到类似的情况——在金融市场，你会看到穆迪。穆迪介入，他们查看债券并输出评级。他们说，这是我们发现的证据。这是评级。我们不决定任何人是否应该购买或不购买这个债券。嗯，这取决于风险偏好，但我们确实提供了这个每个人都可以依赖的公共信息层。在穆迪的案例中，政府指着他们说：“嘿，养老基金，你们可能真的要注意。你们不应该拿养老金领取者的钱去冒险。所以你们只能投资 AAA 级债券。”这意味着政府现在不必配备数千名金融技术专家每周重新运行预测，以查看评级是否正确。他们可以指向某个中立的第三方。所以我的假设，我的直觉是，你会看到一个第三方坐在政府和实验室之间，它可能是政府自己建立的。所以像 CAISI 这样的机构就是为了做这件事而设立的。

As a government, that might be hard to trust, and we think that a broker is the most natural solution. In other markets, you see something like—in financial markets, you see Moody's. Moody's goes in and they look at a bond and they output a rating. They say, here's the evidence we found. Here's the rating. We don't decide whether anyone should buy this bond or not buy this bond. Well, that depends on the risk appetite, but we do provide this common information layer that everyone can rely on. In the case of Moody's, the government points to them and says, "Hey, pension funds, you should probably really take care. You shouldn't risk your pensioners' money. So you can only invest in AAA rated bonds." That means that now the government doesn't have to staff thousands of financial technical experts to rerun forecasts every week to see whether things are correctly rated. They get to point to some neutral third party. So my hypothesis, my hunch, is that you will see a third party that sits between the government and the labs, and it could either be the government builds it themselves. So something like CAISI was set up to do exactly this.

好的，我不熟悉 CAISI。

All right, I'm not familiar with CAISI.

CAISI 是人工智能标准与创新中心。

CAISI is the Center for AI Standards and Innovation.

好的。

Okay.

我不会深入细节，但它是 NIST 的一个下属机构，通常制定标准。所以它基本上是一个拥有专家的政府机构。

I won't get into the details, but it's a sub-body of NIST that typically sets standards. So it's basically a government body that has experts.

没错。非常低调。

Exactly. Very low key.

没错。

Exactly.

我觉得这是那种你坐下来听一听、看一看就会想的问题：政府现在有没有足够的技术专长来测量和测试这些东西？大概没有，对吧？而 Fable 就是一个结果：好吧，我们不得不缩减规模、暂停一些事情。

I think it's one of those things where when you just sit back and listen, look at it like: is there enough technical expertise in the government to measure and test these things right now? Probably not, right? And Fable is a result of: okay, we've had to scale back and pause things.

他们有一流的人才，但预算与我们面前挑战的规模相比小得惊人。我认为他们有角色要扮演。问题在于谁做什么，而我们现在已经列出了需要完成的工作，而且相当繁重。每一次模型发布——考虑到它们会接收任何输入——风险面都大得惊人。所以真正的问题是：哪些事只有政府能做，市场又能提供什么来跟上 AI 风险变化的速度。我们的观点是，在模型层面，人们今天关心的风险已经不是 3 个月前他们关心的那些了。所以立法的节奏太慢，无法精准定位这里的风险。我们认为市场可以做很多事来及时呈现信息。归根结底，这里有一堆政策决定。一个模型的国家安全风险是否过高？那是一个政治答案。

And they have excellent people, but they have an extraordinarily small budget compared to the scale of the challenge that's ahead of us. And I think they have a role to play. The question is kind of like who does what, and we have now outlined the jobs to be done and they're quite extensive. Every model release, there is an astounding — given that they take in any input — the risk surface is astounding. And so the question is really what can only the government do, and what can the market provide here that can keep up with the pace as AI risk changes. Our perspective is that also at the model layer, the risks that people care about today are not the same ones they cared about 3 months ago. So the pace of legislation is too slow to deal with pinpointing the risks here. And so we think there's a lot that the market can do to surface timely information. Ultimately there is a bunch of policy decisions here. Is the national security risk of a model too high? That's a political answer.

但你要确保的是，产生这些风险信息的过程能与非常快速的创新兼容。

But what you want to make sure is that the process that produces this risk information is compatible with very fast innovation.

所以你不想要——这不是能不能放慢速度的问题，也不是能不能把模型锁上几个月直到所有人都能给出保证的问题。而是：鉴于美国在模型发布上正与中国竞争，你能否及时插入风险信息，让政府能就其中一些问题迅速决策，平衡好这个取舍——不采用 AI 会让我们陷入风险，但鲁莽采用 AI 也会让我们陷入风险。这是一个非常微妙的平衡，他们需要大量高质量的情报才能做到。

So you don't want to — this is not a question of like, can you slow things down? Can you keep the models locked up for months on end until everyone can make a guarantee? But it is: can you, in the time given that the US is competing with China on releasing models, insert risk information that allows the government to make rapid decisions on some of these questions, balancing that trade-off between failing to adopt AI is going to put us at risk, but also reckless adoption is going to put us at risk. And that's a very fine balance that they're going to need a lot of high quality intelligence to make.

顺带提一句，因为你提到了中国模型——你从你们的 CESO 那里听到什么具体的担忧吗？因为我猜它是免费的，但 CESO 对数据流动普遍有一堆担忧，他们真的很在意。所以有很多问题，比如：如果这些模型是中国的，我们的数据会去哪里？我觉得其中很多是可以解决的，但它们经常被提出来。

Just a side mention, because you mentioned Chinese models — any specific concerns that you're hearing from your CESOs about that? Because I guess it's free, but CESOs have a bunch of concerns around data flows in general that they're really concerned about. So there's a lot of questions like, if these models are Chinese, where does our data go? I think a lot of those can be addressed, but they come up often.

我的意思是，他们明白这些模型跑在美国的 GPU 上。有些人明白，因为它们跑在美国的 GPU 上。

I mean, they understand they're running on American GPUs. Some of them understand, because they're running on American GPUs.

它们不会像每次你调用时就往家里打电话那样。

They're not like phoning home every time you call home.

不会。一年前人们对这一点还不太了解。我其实觉得你正在看到——这些领导者正以惊人的速度变得懂 AI，而且你也会看到我那非常 AI 化的 Twitter 时间线和过去完全不懂 AI 的 LinkedIn 信息流开始趋同。它们都在谈论 Fable。

No. A year ago there was not a lot of understanding of this. I actually think you're seeing this — cure leaders becoming kind of AI literate at a blistering pace, and you're actually also seeing my Twitter timeline that's very AI-pilled and my LinkedIn feed that used to not at all be AI-pilled kind of converge. They're both talking about Fable.

对吧？是的，确实如此。

Right? Yeah, it is true.

它们都在谈论如今你能否防止模型被越狱。就像国家安全风险——那场对话其实正在浮现。除此之外，我觉得你主要看到的是——对任何特定模型或任何特定模型输出都没有担忧。但有一种普遍的紧张情绪：让关键基础设施运行在并非由美国人在美国生产、美国政府无法控制的模型上。

They are both talking about whether you can prevent models from being jailbroken these days. Like national security risk — that conversation is actually emerging. Other than that, I think you mostly see a kind of — there's no concerns with any particular model or any particular model output. But there's a general nervousness of having critical infrastructure run on models that are not produced in America by Americans, where the American government has control.

它不一定那么直接地出现在你们的框架里，或者也许会。

It doesn't necessarily show up in your framework that directly, or it might.

其实里面确实有一些关于模型来源和披露的内容。但我认为有很多用例里，运行一个中国的开源模型就是最好的解决方案。

There's a bit of stuff in there actually on the provenance of the models and disclosing that. But I think there's a bunch of use cases where running a Chinese open source model is just the best solution.

而这里的担忧稍微更宏观一些，它不太适合在任何特定的认证层面来解决。你有没有看到什么有趣的东西——你知道，如果你们试图填补那个中间缺口、那个中介缺口，除了普通人可能预期的东西之外，你们预测会需要哪些有趣的东西？

And a concern is slightly more macro here, which is not best addressed at any particular certification level. Is there anything interesting that you see at the — you know, if you are trying to fill that middle gap, that mediation gap, any interesting stuff that you guys forecast would be required other than what the average person might expect?

关于这里哪些风险重要，有一堆有趣的问题。所以现在，当下的风险是网络攻击，因为它非常真实、非常具体。还有一些风险也正变得相当真实、相当具体，比如儿童安全，它既变得极其重要，也在政治上重要。然后还有一些正在酝酿中的风险，今天感觉有点投机，但花大量时间与模型相处的人能看到它们正在到来，比如与生物学相关的风险——具体来说，模型是否会帮助对手制造生物武器，并让这变得极其廉价、极其易得，从而制造出另一场新冠或更糟的大流行的可能性。新冠并不是像你刻意去做那样被设计成有害的。所以我认为这些就是正在酝酿中的一些风险。

There's a bunch of interesting questions about what are the risks that matter here. So, right now, the risk of the day is cyber, because it's very real, very tangible. And some of the risks that are also emerging as pretty real and pretty tangible are things like child safety, which is becoming both extremely important but also politically important. And then there are some of the risks that are coming down the pipeline that today feel kind of speculative, but people who spend a lot of time with the models see them coming down, like risks that relate to biology — and specifically whether models will help adversaries produce biological weapons and make that extremely cheap, extremely accessible, producing — making the chance of another COVID or worse pandemic. COVID was not engineered to be bad as if you're trying to do that. So I think those are some of the risks that are coming down the pipeline.

我想还有一点值得注意：智能体是刻意收窄的。所以当一家前沿智能体公司推出一个与客户互动的聊天机器人时，他们真的试图收窄它感兴趣谈论的话题，以至于如果你问它比如你怎么看总统，它就会直接拒绝回答，这意味着风险区域相对小一些。对模型来说，风险面是无限的，所以没有哪个专家能同时胜任评估网络攻击的风险、15 岁少年与聊天机器人进行长达一个月的对话并看它是否真的会建议自杀或诸如此类可怕事情的风险，以及评估恐怖分子利用 AI 制造生物武器的风险。风险面实在太大了。所以核心挑战实际上变成了：你如何让那些领域专家在一个连贯的框架内工作，输出一份连贯的报告和评级，让全世界都能去查看，因为那种全球视角至关重要。但今天没有任何一个组织能产出这个。

I think one other thing to just note is that agents are kind of deliberately narrow. So like when a frontier agent company puts a chatbot that interacts with customers, they've really tried to narrow the topics it's interested in talking about, such that if you ask it like what do you think of the president, it will just decline, which means that the kind of risk area is somewhat smaller. For models it is infinite, and so there's not a single expert out there who can competently evaluate the risks of cyber attacks and 15-year-olds having month-long conversations with a chatbot and seeing whether it will in fact recommend suicide or something horrendous like that, and can evaluate the risks that terrorists can use AI to produce bio weapons. The risk surface is just too big. And so the central challenge actually becomes: how do you get those subject matter experts to work within one coherent framework that outputs one coherent report and rating that the world can go and inspect, because that global perspective is central. But there's not a single organization today that could produce that.

而当你发布你们的模型标准时，你们会是那个理所当然的候选者。

And you would be the presumptive one when you put out your model standards.

我们认为可以有一家公司，与一个专家联盟一起，建立一套连贯的标准。我认为我们在今天所有企业风险上已经证明了这一点。我们认为可以有一家公司，与一个联盟一起，规定审计规则——基本上就是所有这些技术专家所需的输入和输出。

We think there can be one company that can, with a consortium of experts, build one coherent standard. I think we've shown that across all of the enterprise risks today. We think there could be one company that could, with a consortium, specify the audit rules — basically like the inputs and outputs that all these technical experts need.

他们需要什么访问权限？他们应该如何处理基础设施安全？他们可以查看评估是否制作精良，而不必能够说，嘿，这是威胁还是不是威胁？但总体而言，评估这些评估是否良好、构建得当——那套规则基本上成为所有专家的接口。我们认为一个清算所可以整合起来——明确地说，当我说一家公司时，我把它看作是一家公司协调许多事情，就像我们看到我们的联盟时，并不是说我们在智能体安全方面有所有答案。我们说的是，我们承担起引出所有关切并作为秘书将其整合起来、严格管理的角色，使得标准更新每季度锁定，并且发布的订单报告——在这种情况下是 100 页的订单报告——统一、清晰明了，达到高管做出明确 go/no-go 决策所需的详细程度。所以这就是我们认为我们可能扮演的角色。

What access do they need? How should they treat infra security? They can look at whether the evals are well produced without necessarily being able to say, hey, is this a threat or not a threat? But overall, evaluating whether the evals are good, well-constructed—that set of rules basically becomes the interface for all these experts. We think one clearing house could put together—to be clear, when I say one company, I think of it as one company coordinating lots of this, in the same way that when we saw our consortium, it's not like we say we have all the answers on agent security. What we say is we are taking on the role of eliciting all of the concerns and being the secretary that puts it together and runs a tight house, such that the standard updates lock every quarter and that the order reports that come out—in this case 100-page order reports—are uniform and crisp and clear, all to the level of detail that is required for executives that need to make a clear go/no-go decision. So that's kind of the role that we think we might play.

我认为在很多方面，你正在扮演 OASP 过去所做的角色，你说过，你知道，竞争和合作伙伴。你能更详细地谈谈他们如何合作吗？

I think in many ways you're performing the role that OASP used to do there, and you said like, you know, competition and partners. Can you go more into like how they partner?

是的。首先，OASP 基本上是一个安全从业者的开源社区，他们聚集在一起构建框架，以应对最新的安全问题。

Yeah. So first of all, OASP is basically an open source community of security practitioners that are coming together to build frameworks for addressing the latest security concerns.

我们认为他们在创建框架方面非常出色。

We think they are phenomenal at creating frameworks.

事实上，我们首先与他们合作。所以我们有一篇联合文章。其次，我们从他们那里学到了很多。我们认为他们是巨大的情报来源。他们没有做的是构建运行第三方审计的机器，使得像 Cursor 这样的公司或像 JP Morgan 这样的公司可以让第三方来审查他们是否符合标准，并说，嘿，你通过了标准，这是你可以用来建立信任并预先回答合作伙伴或客户问题的报告。所以他们从根本上试图做不同的事情。他们是信息收集和情报收集以及创造清晰度的一部分，但将这一切转化为承诺的运营层并不是他们试图从事的业务。

We've in fact—we first of all we're partners with them. So we have a joint article. Two, we've learned a lot from them. We think a tremendous source of intelligence. What does not do is building the machine that runs third party audits such that a company like Cursor or a company like JP Morgan could get a third party to go and review them against this and say, hey, you've passed the standard and here is the report that you can use to build trust and preempt your partners or customers questions. So they fundamentally try to do something different. They are part of the information gathering and intelligence gathering and creating clarity, but the operational layer of turning this into promises is not the business they tried to be in.

标准正在出现，而且做得很好。那么是否也有必要进行承保？显然，这体现在名字中，所以想必你首先考虑过。我觉得如果你有足够的共识，你实际上不需要金钱角度，但它确实有帮助。我还想指出，你们也是一家营利性公司，对吧？这不是非营利工作。也有整个商业方面。

The standard is emerging and it's doing very well. Was it necessary to then also do underwriting? Obviously it's in the name so presumably you thought about it first. I feel like if you just have enough consensus you don't actually need the money angle but it does help. I did want to also note you guys are a for-profit company too, right? It's not nonprofit work. There's a whole business side to it as well.

是的。是的。是的。这关于钱的事情很疯狂。

Yeah. Yeah. Yeah. This is crazy about the money.

是的。是的。是的。让我们进入钱的部分。

Yeah. Yeah. Yeah. Let's get into the money part.

让我们从你关于营利与非营利在当今安全领域的问题开始。网络安全中，大多数标准是由非营利组织制定的。我认为这是一个问题。你必须问自己的问题是，你如何为这些标准创造良好的激励，使其保持良好并跟上时代？非营利组织往往没有这些不利的利润激励，他们会掏空标准并制造逐底竞争。但它们默认情况下也完全不响应它们所服务的社区，因为没有流程。他们没有客户，他们去问客户你想要什么？你想要什么？你想要什么？当你审视当今安全标准的整体满意度时，人们往往不太喜欢它们。你在其他领域确实看到，营利性标准可以很好地服务于世界。所以有一些例子，比如我们之前谈到的电影。它并非没有缺陷，但它是当今以惊人规模运行的关键社会基础设施。你的信用评分。它是 FICO。它也是一家营利性企业。再往前追溯历史，一些碰撞测试标准来自保险公司。保险公司共同资助成立了公路安全保险研究所，因为他们非常感兴趣的是如何利用标准来降低死亡率并节省资金。再往前，我们的名字实际上是为了向 Underwriters Laboratory UL 致敬，它大约在电力出现时成立，房屋开始烧毁，保险公司再次支付账单，他们也许也是好人，但他们的利润激励是让我们防止房屋烧毁，让我们测试所有电气产品，灯泡——这里所有的灯泡可能都经过良好测试——烤面包机等，他们设立了一个实体来创建这些标准。今天，UL 有一个营利性实体和一个非营利实体。他们认识到，他们分拆出来，他们开始了一个非营利组织，他们分拆出一个营利性实体，因为他们认识到，嘿，实际上为了更好地服务客户，你需要一个营利性实体。这里的教训是，市场可以调整激励的一种方式，使你既能响应客户，又不会随着时间的推移掏空你的标准，就是将其与保险公司对齐，因为他们从根本上具有良好的激励。所以如果你是一个与保险公司密切合作的营利性标准，你会得到反馈循环，这样你真正与客户保持一致，但也把他们的利益放在心上。所以这就是我们学习的模型——那种启发性的模型，也是名字的来源。在某种程度上，承保这个词既可以与保险相关联，但它也是一个广义的术语，比如做决定。

Let's start from actually your question for profit versus nonprofit in the security space today. Cyber security most of the standards are produced by nonprofits. I think that's an issue. The question you have to ask yourself is how do you create good incentives for these standards to be good and keep up? Nonprofits tend to not have these adverse profit incentives where they hollow out their standard and create a race to the bottom. But they're also not at all responsive by default to the communities that they serve there because there's no process. They don't have customers that they serve where they go and ask what do you want? What do you want? What do you want? And when you look at the overall satisfaction with the security standards today, people tend to just not like them very much. You do see in other domains that for-profit standards can serve the world quite well. So there are examples like we talked about movies before. It's not without flaws, but it is absolutely critical societal infrastructure that gets run at astounding scale today. Your credit score. It's FICO. It's also a for-profit business. And when you go back even further in history, some of the crash testing standards came out of insurance companies. The insurance companies together funded the founded institute of insurance institute of highway safety because they were very interested in like how can we use standards to drive down mortality and save money. Go back—prior our name actually pays homage to the underwriters laboratory UL which was started right around when electricity came out houses started burning down insurers again were paying the bill and they were maybe also good people but their profit incentive was let's prevent houses from burning down let's test all the electrical products the light bulbs all the light bulbs in here are probably well tested the toasters etc and they set up an entity to create those standards. Today, UL has a for-profit entity and a nonprofit entity. What they've recognized, they spun out, they started a nonprofit, they spun out a for-profit because what they recognized was like, hey, actually to serve customers well, you need a for-profit entity. The lesson here is one of the ways that the market can align incentives so you're both responsive to customers and not hollowing out your standard over time, is to align it with insurers because they fundamentally have good incentives. And so if you're a for-profit standard that works closely with insurers, you get the feedback loop in such that you're really queued into your customers but also have their interest at heart. So that's the model that we're—the kind of inspirational model that we've learned a lot from and that's also where the name comes from. In some ways the term underwriting can both be associated with insurance, but it's also a broad term for like making decisions.

如果你承保一个决定，你从根本上是在为它的后果承担责任。

If you underwrite a decision, you're fundamentally kind of taking ownership for the consequences of it.

是的。我的意思是，AI 的保险合同是什么样子的？

Yeah. I mean what does an insurance contract look like for AI?

是的。今天对保险合同的大部分需求位于构建 AI 的人和购买 AI 的人之间。

Yeah. Most of the demand comes today for insurance contracts is sitting between people who've built AI and people who are buying AI.

是的。

Yes.

你想要的是人们希望保险公司参与的原因，既有传统原因。支付。如果出了问题，我们希望得到补偿。但特别是因为保险公司可以增加——可以带来信任，因为保险公司会支付损害赔偿。如果他们愿意写保险单，那就是他们在说：“嘿，我们认为这里有风险，但那是可管理的。”这有点像他们的激励与采用它的企业一致。所以这对市场来说是一个非常好的信号。实际上，同样地，Waymo 试图获得在旧金山运营的第一个许可时，所做的其中一件事就是让许多保险公司堆积一份巨额保险单，以防万一出了问题。

And what you want is the reason why people want insurers involved both for the traditional reasons. Pay. If something goes wrong, we want to be compensated. But it's in particular because insurers can increase in can bring trust in the equation because insurers will take pay for the damages. If they're willing to write an insurance policy, that is them saying, "Hey, we think there's risk here, but that is manageable." And that is kind of a their incentive aligned with the enterprises adopting it. So that's a really a good signal to the market. In the same way actually one of the things that Waymo tried to get their first permit to even operate in San Francisco was to get a lot of insurers to stack up a huge insurance policy in the case of something went wrong.

不是因为 Google 付不起，而是因为让一个受政府信任、受企业信任的保守第三方去审视那些数据，然后说“嘿，我们看过了。我们实际上愿意把其中一部分风险放到我们的资产负债表上”，这非常有价值。所以这就是人们对此感兴趣的原因。

Not because Google can't pay, but because it was very valuable to have a third party go and look at that data that are trusted by governments, trusted by enterprises as conservative people and say, "Hey, we've looked at it. We're actually willing to take some of this onto our balance sheet." So that's kind of the reason why people are interested in it.

它在某些方面看起来和其他保险合同一样：你要指定想覆盖哪些风险、想覆盖多少、上限是多少、覆盖这些要花多少钱？

What it looks like is in some ways like every other insurance contract: you specify what are the perils you want to cover, how much do you want to cover them, like up to what limits, what does it cost to cover that?

举个非常具体的例子，11 Labs 购买了一份史无前例的 AI 智能体保险单。他们与一些最大的企业合作，与政府合作。他们非常希望超越常规，向客户做出承诺。所以他们写了一份保单，只覆盖我们客户一直在问的一些核心关切。

And in the case of, if we take a really concrete example, 11 Labs bought a first-of-its-kind AI agent insurance policy. They work with some of the biggest enterprises. They work with governments. They're really interested in going above and beyond and making promises to their customers. So they wrote a policy that covers just some of the core concerns that our customers have been asking about.

关键是要让伦敦劳合社——世界上历史最悠久的保险公司、我们的合作伙伴之一——来审视这些数据，并作为第三方与我们一起说：“嘿，我们认为这里有些东西值得承保。”这实际上就是它的样子。

And the crucial thing was really to get Lloyds of London, the world's oldest insurer, one of our partners, to look at this data and be that third party alongside us to say, "Hey, we think there's something here that's worth underwriting." And that's actually what it looks like.

所以他们会把那份合同展示给客户，客户可以看到自己获得了多少保障，可以看到具体覆盖什么。而且明年这很可能还会变。他们会想写一份可能覆盖更多的保单。

And so they will show that contract to their customers and they can see how much they're covered for. They can see what exactly it covers. And that will also probably change next year. They will want to write an insurance policy that might cover more.

你说劳合社，是再保险，还是他们在同一层面上以某种方式分担？

When you say Lloyds, is it reinsurance or are they sharing somehow at the same level?

是的。通常新公司进入保险业的方式是与保险公司合作，让保险公司承担大部分或全部财务风险。从根本上说，如果保险之所以有用是因为它带来信任，那你就必须有能力赔付。伦敦劳合社有 400 年历史。他们从未拒付过索赔。他们极其受信任。

Yeah. So typically the way new companies get into insurance is that they partner with insurers such that the insurers take the majority or all of the financial risks. Fundamentally if insurance is useful because it brings trust, you have to be able to pay the bill. Lloyds of London is 400 years old. They've never not paid a claim. They're extremely trusted.

伦敦劳合社自己难以做到的是弄清楚哪些风险是真实的。我们应该关注什么？有哪些技术控制措施，如何运行测试？所以他们把 AEC1 作为一种承保框架，我们产出一堆邮件结果，直接输入进去为定价提供依据。

What Lloyds of London struggled to do on their own is to figure out which of the risks are real. What should we be looking for? What are the kind of technical controls and running the tests? So they use AEC1 as kind of the underwriting framework and we produce a bunch of email results that then directly feed in to inform the pricing.

所以这意味着 Level Labs 的客户知道赔付会到位。他们不必去看我们的 A 轮融资，看我们是否认为我们资产负债表上有足够现金。他们会看劳合社。

So this means that Level Labs customers know that that payment will be there. They don't have to look to our series A and see like do we think they have enough cash on the balance sheet? They will look at Lloyds.

是的。

Yeah.

而且劳合社以非常有创意著称。我记得有个头条说他们给詹妮弗·洛佩兹的臀部投保之类的。

And Lloyd's like famously very creative. I think I remember some headline like they insured Jennifer Lopez's butt or something.

没错。我记得还有大卫·贝克汉姆的右脚？是的。诸如此类。

Correct. And I think it was David Beckham's right foot? Yeah. Stuff like this.

所以显然不是很大的数据集。

So like clearly not a large data set.

正是。它实际上是一个了不起的机构，既有存在已久的真正老派优点，又像受信任的实体那样运作，而且他们有探索未来的胃口。

Exactly. It's actually a remarkable institution that's both kind of has some of the truly old school virtues of having been around for a long time. They really operate like a trusted entity and they have appetite to figure out the future.

我认为人们普遍认识到，AI 中存在大量今天被理解得很差的风险，所以进入这个行业带有真实风险，但未来大量风险敞口也会在这里发生。这是风险真正在增长的一个市场。这也是一个会取代一些现有市场的市场。以车险为例：当没有人类驾驶员时，那个市场会是什么样？显然会改变。你要如何评估？

And I think there's a lot of recognition that both there's like tremendous amount of risk in AI that is poorly understood today so getting into this business carries real risks, but also this is where lots of the risk exposure will happen in the future. This is the one market where risk is truly growing. This is the one market that will also take out some of the existing markets. Take auto insurance: when there are no human drivers, how's that market going to look? Well, it's clearly going to change. How are you going to assess?

你想承保的要多得多。

You want to insure way more.

我只想说，承保多得多的原则与承保其他类型 AI 非常相似。比如碰撞测试，我们为客户做的那些很受欢迎，这也需要发生，而不是像对待人类驾驶员那样做。

All I'll say is the principles for how you insure way more are very similar to how you insure other kinds of AI. So crash testing, that's what we do for customer share are lovable, that will also need to happen for way is not how you do it for human drivers.

所以人们越来越意识到世界变化非常快，学习如何承保 AI 的唯一方式就是写一些保单。你可能会遭受一些损失，把它当作研发费用就好，但对他们来说问题是：他们能与哪些受信任的技术伙伴一起进入这个行业，帮助他们导航、确保不犯愚蠢错误，同时谁愿意听取他们拥有的智慧。他们以前做过这个。他们见过，网络保险出现时他们就在那里。

So there's this growing awareness that the world is changing very fast and the only way to learn how to underwrite AI is to write some policies. You may incur some losses and think of that as R&D expense really, but the question for them is like who are the trusted technical partners they can get into this business with that can help them navigate and make sure they don't make kind of foolish mistakes, but also who is willing to hear the wisdom that they have. They've done this before. They've seen, they were there when cyber came out.

所以 AI 在很多方面感觉完全新，但风险也有很多方面看起来一样。所以实际上，一些可能头发花白的人身上有着巨大的智慧，他们真的对如何量化风险有敏锐的感觉。

So there are lots of ways in which AI feels completely new, but there's also lots of ways in which risks look the same. And so there's actually tremendous amount of wisdom sitting in some folks that may have gray hair but really have like a keen sense of how to quantify risk.

是的。数字基本上就是，我想要 5000 万美元的保额来覆盖这些风险，VO 会给你一个报价，然后你加一点小差价之类的，再转手做出去。就这么简单吗？

Yeah. And the number is so it's basically like I want $50 million worth of coverage against these perils and VO will give you a quote on it and then you have like a small markup or something and then you turn it around and do that. Is that as simple as it is?

你基本上分享一部分保费。X% 给做定价的人。这有点像保险领域的商人银行。

You basically share some of that premium. X% goes to the people who do the pricing of it. It's kind of like a merchant bank for insurance type of thing.

正是。你基本上拆分费用，你可以把保险供应链想象成：有提供资本的、有做定价的、有做分销的，通常你会在这里支付保费的 x%，在那里支付 y%，剩下的归这里。整个保险世界都是这样运作的吗，还是说某个时候，比如现在你有股权资本，某个时候你也许开始发债之类的，然后你有足够的银行账户和足够的历史，假设你运营了 10 年

Exactly. You basically split the fee and you can think of the insurance supply chain as like there's bringing the capital, there's doing the pricing and there's doing the distribution and typically you will pay out some x% of premium here, y% of premium here and the rest of it will go here. Does all the insurance world work like this or is there some point at which like so so right now you have equity capital at some point maybe you start raising debt or whatever and then you have enough of a bank account and enough history let's say you've been operating for 10 years

就不再需要律师了。

That you don't need lawyers anymore.

这完全是一个选项，我能看到在某些情况下这有道理，特别是如果有些风险我们高度确信他们会想承保，而现有保险公司太慢找到胃口。好的。

That's totally an option and I could see some worlds where that makes sense, specifically if there are risks that we feel high confidence they would want to ensure where the incumbent insurers are too slow to find appetite. Okay.

或者只是难以评估，以至于他们不想做

Or or simply struggle to evaluate such that they don't want to do

但总的来说，你不想在提供风险资本上与保险公司竞争，原因有两个。第一，这从根本上是一个资本成本游戏。他们的资本成本极低，而初创公司总体上资本成本很高。第二，你想对冲你的押注，那么拥有一个包含房屋保险、车险的投资组合就非常有用，而我们不打算成为车险或房屋保险公司。所以他们有一些天然优势，这让我们更有可能合作。

But by and large in general you do not want to compete with insurers on bringing risk capital to the game for two reasons. One is that's fundamentally a cost of capital game. They have extremely low cost of capital, startups have high cost of capital by and large. And two, you want to hedge your bets and it's very helpful then to also have a portfolio of home insurance, of car insurance, and we're not about to become a cars nor a home insurer. So they have some natural advantages which makes it much more likely that we'll partner.

是的。

Yeah.

他们带来规模化的资本，我们带来技术。

And they bring the capital at scale and we bring the technical.

你们会和他们长期合作。

You're going to work with them for a long time.

和保险公司的讨论进展如何？基本上他们是依据你们的认证，对吧？他们信任你们的尽职调查，认为你们的认证有效。你们测试了正确的东西，他们用资金来支持你们确实有正确的测试。那么，和保险公司合作有什么有趣的收获吗？

How are the discussions with the insurers as well? So basically they're going off of your certification, right? They're trusting the diligence on you that your certification is valid. You tested the right things and they're backing the money that you know you have the right testing in place. So any interesting takeaways from working with insurers?

我想也许第一点是他们也会为标准提供输入。所以，如果他们觉得需要某些东西但没看到，我们也会将其作为标准的输入，因为从根本上我们认为一个好的标准是能创造一个非常健康的承诺生态系统，而我们认为保险公司是其中关键的一部分。而且，他们最有动力去……他们能看到所有的损失数据，任何特定的 CISO 知道他们自己的具体担忧，而保险公司能看到整个投资组合中的担忧，并且经常能直接了解到底发生了什么、是谁的过错等，作为他们取证的一部分。所以，他们实际上是这方面很好的情报来源。

I think the maybe the first thing is they feed into the standard as well. So if there are things that they feel like they need that they're not seeing we are also taking that as input into the standard uh because fundamentally we think a good standard is one that creates a really healthy promise ecosystem and we think insurers are critical part of that uh and again they are the most well incentivized to they see all the loss data across any particular CISO knows their particular concerns. insurers see the concerns across the entire portfolio and often have direct access to like what exactly happened, who was at fault, etc. as they do part of their forensics. So, they're actually like a great source of intelligence on this.

网络保险市场的一个重大教训是，保险和技术专长没有很好地结合。我们的信念是，标准必须先于保险。从根本上说，每个人首先想要的是，无论你是摩根大通的 CISO、Cursor 的 CISO，还是伦敦劳合社辛迪加的承销商，你首先想要的是不发生事故，你想知道风险得到了很好的管理，只有到那时保险才开始有意义。所以我们会看到标准生态系统基本上会领先于保险。你问我们为什么我也做保险，这某种程度上是在证明我们认为整个承诺信心基础设施生态系统应该是什么样子，我们认为将其变为现实非常有说服力，即使我们认为标准是解锁其余部分的核心关键。

One of the big takeaways from cyber insurance, which is a market that didn't work that well, was that the insurance and the technical expertise was not married up. Uh what our conviction is that standards have to precede insurance. fundamentally what everyone first and foremost want whether you're a CISO at Jig Morgan or a CISO at Cursor or a underwriter at at Lloyds of London syndicate is you want to not have an incident in the first place you want to know that the risk is well managed and only then does insurance start to make sense so we'll see the standard ecosystem basically run ahead of the insurance and the reason why we you asked us kind of why I also do insurance this is kind of proving what We think that whole promise confidence infrastructure ecosystem needs to look like and we think it's very compelling to bring that to life even if we think the standard is kind of the the core lynch pin that unlocks the rest.

目前还没有索赔，对吧？

There's been no claims yet, right?

没有。

Nope.

这是那种事情之一，如果人们还没有真正弄清楚覆盖意味着什么。例如，我每月付给 Cursor 20 美元。

This is one of those things where um you know if people haven't really worked through what it means to cover things. So for example, I pay cursor $20 a month.

是的。

Yep.

然后我 vibe code 了一些东西，导致飞机坠毁，造成 2 亿美元的损失。我是索赔 20 美元还是索赔 2 亿？

And I write I vibe code something that makes uh a plane crash causing $200 million worth of damage. Uh, do I claim $20 or do I claim 200 million?

是的。所以，这些都是很好的问题。幸运的是，整个保险和法律历史有助于回答其中一些问题。我认为第一点是人们的保单有上限。所以，如果你想索赔 2 亿美元，必须有人预先为那份保单支付了很多钱才能有 2 亿美元的覆盖。最终，它的运作方式是，你从很多不确定性开始。这不仅仅是保险，还有比如 Anthropic 能否使用互联网上的书籍来训练，他们可以去查看先例，他们可以看到

Yeah. So, these are all great questions. Uh, and fortunately, kind of all of insurance and legal history kind of helps answer some of those questions. I think the first thing is people have limits on their policy. So, if you want to claim $200 million, you have to someone has to have paid a lot for that insurance policy up front to have $200 million of coverage. And ultimately the way this works is that uh you start from a lot of uncertainty. This is not just an insurance but also like can you use can anthropic use books from the internet to train up well they can go and look at precedent they can see

但最终这些事情会在法庭上解决，你会随着时间的推移敲定。所以你从这种模糊的地方开始

but ultimately this these things get settled in court and you hammer it out over time. So you start from this like place of ambiguity

这既是为什么保险在早期很难做，也是为什么人们想要保险，因为这种模糊性会减缓采用。是的，

which is both why insurance can be hard to do early on but it's also why people want insurance because that ambiguity slows down adoption. Yeah,

这也位于……的头上

that also sits at the heads of the uh

在某些方面，实际上第一次事件将有助于确立很多这方面的事情。

in some ways actually the first incident will help to establish a lot of this.

没错。而且已经有很多事件没有被保险覆盖。以现在有点旧的加拿大航空公司的例子来说，它幻觉出了一个退款政策

Exactly. And and there have been a number of incidents out there that have just not been insurance covered. Take the now old uh example from Air Canada where hallucinated a a refund policy

在那个案例中，加拿大航空公司说嘿，我们与此无关，这个聊天机器人搞砸了，但抱歉，法院说不行，如果你让你的聊天机器人与客户互动，它们会代表你做出具有法律约束力的承诺。这现在已成为未来所有事情的先例，如果有人再次部署这样的聊天机器人，你不应该期望能够推卸责任说抱歉我的聊天机器人撒谎了，与我无关，我从 OpenAI 买的。不，如果你把它放在客户面前，你就要对它负责。所以每一个法庭案件，无论是否涉及保险，都会澄清责任，而责任是保险的基础。还有另一个原因说明标准和保险是结合在一起的，责任……我在这里稍微跑题一下。深入细节。

and the question was Air Canada in that case were like hey we have nothing to do with this chatbot messed up but like sorry and the courts were like no if you put your chatbots to interact with your customers they make legally binding promises on your behalf. That is now precedent for everything in the future where you will if someone were to deploy a chatbot like that again there you should not expect to be able to just pawn off and say sorry my chatbot lied it's nothing to do with me I bought it from open AI no if you're putting this in front of your customers you are taking responsibility for it and so every court case whether insurance is involved or not clarifies liability and liability is kind of the foundation for insurance there's another reason why stands and insurance come together liability ility for I'll go on a little tangent here. Get the weeds of it.

请讲

Please

责任的核心概念之一往往是某人是否有过失。他们应该看到这一点吗？他们应该防止这一点吗？问题是你如何判断？基本上，你判断他们是否履行了注意义务。这在实践中意味着什么？通常他们会参考标准。所以，如果有一个被广泛采用的标准，说你必须有一个接地过滤器或你必须有一个越狱过滤器，那么声称不知道这些事情存在就变得困难得多。因此，制定标准有助于澄清责任点，法院通常会指向标准，说这似乎是应该做的最佳实践，对所有人可见。所以，标准是文明基础设施的另一种方式，保险可以在此基础上建立，承诺也可以在此基础上建立。

liability often one of the core concept is whether someone was negligent. Should they have seen this? Should they have prevented this? And the question you how do you judge that? Well, you basically judge whether they've met their duty of care. What does that mean in practice? Well, often they look to standards. So if there's a standard that is broadly adopted that says you must have a groundedness filter or you must have a jailberg filter it becomes way harder to claim ignorance that these things existed and so setting standards help clarify liability points courts will often point to standards and being like well this seems like best practice to do is there for everyone to see. So there's another way in which like standards are kind of civilization infrastructure that insurance can then build on which promises can then build on.

我完全理解我们不必获得认证来编写这些，你知道，制作这些机器人等等。

I I totally get that we don't have to get certified to to write these to you know make these like bots and all these.

但基本上，每当我们去审计时，我想人们开始振作起来，所有这些事情。我想知道这是否意味着你也不会成为我发布到生产的批准机构，你知道，比如是的，你每季度检查一次，我想每天发布一次。

Um but like basically whenever we get go for the audit I think people like start to shape up and and all this all this stuff. I wonder if like that means that you don't also then become like the approving authority for me to ship to production you know like um yes you check once per quarter I want to ship once a day.

是的。而且我不知道当我的东西出问题时，是否违反了你们的认证之一。

Yeah. and I don't know when one of my things breaks like one of your certifications or not.

所以有几件事情，其中有一些要求与你如何自己测试有关，你必须在至少重大发布之前告诉你的客户你自己是如何测试的。我们不会每天去命令人们。

So there there's a couple of things um there's a couple of requirements in there that relate to how do you yourself where you have to tell your customer how are you yourself testing before you make at least major releases. We don't go and order to people every day.

但至少现在有了一条记录，如果你搞砸了大事，你的客户可能会来问你：嘿，你答应过你会自己运行这些邮件。对很多人来说，人们合并的大多数 PR 不会从根本上改变产品体验，但有些会。

But at least there is now a trail where if you do a major mess up, then your customers may come and ask you, hey, you promised me that you were going to run these emails yourself. And for lots of them, most of the PRs that people merge will not fundamentally alter the product experience, but some of them will.

有时你并不知道。

And sometimes you don't know.

有时你并不知道。这也是真的，而且还有——存在一些固有风险，每个人都知道买软件可能会有 bug，这只是其中一部分。但他们能——如果你卖给夫妻店，他们可能不会听到这些。他们会说，好吧，我想用你的工具，所以我愿意承担那个风险。如果你卖给大银行，他们可能会说，抱歉，我们在向客户做出承诺。如果你不能向我们做出我们可以传递的承诺，我们就不想和你合作。那么你就得说，我在乎我的智能体被用作这个国家的关键基础设施吗？如果是，至少我可以对我运行的流程做出承诺，然后我们可以每季度去测试，看看它是否似乎仍然——仍然符合标准。

And sometimes you don't know. And this is also true, and this is also—there's some inherent risk that everyone knows that when they buy software there can be bugs, and this is just part of it. But what they can—if you're selling to mom and pop shops, they may not hear that. It's like, well, I want to use your tool, so I'm just going to be willing to take that risk on. If you're selling to a big bank, they might be like, sorry, we're making promises to our customers. If you can't make a promise to us that we can pass on, we don't want to work with you. Then it's up to you to say, do I care for my agent to get used as critical infrastructure in this nation? If so, at least I can make promises about what process I run, and then we can go and test it every quarter to be like, well, does it seem like it's still—kind of it still meets the standard.

所以从我的角度来看，这是一种方式——大公司默认在发布 AI 时就有一定程度的信任。如果你是一家年轻公司，如果你刚刚起步，默认情况下，你没有信任。而且你能去获得信任的地方很少。所以我们大多数客户在开始与我们合作之前做的一件事就是，他们会自己写安全博客文章。这很好，但谁会相信你说我们很安全？任何人都可以写那个。但很难——你去哪里获得那种信任？所以我认为让标准更加清晰，可以让小公司更容易证明他们正在做他们应该做的事情，因为默认假设是这是蛮荒西部。

So from my perspective, it's kind of a way to—big companies by default kind of have some amount of trust when they ship AI. If you're a young company, if you're just starting out, by default, you have no trust. And there are very few places where you can go and get trust. So one of the things that most of our customers did before they started working with us is that they would make their own security blog posts. That's great, but also who's going to trust you saying we're so secure? Like anyone can write that. But it's very hard—where do you go and get that trust? And so I think making the standards more legible makes it easier for smaller companies to prove that they're doing what they ought to be doing, because the default assumption is that it's the wild west.

你有没有一个路线图——这里有很多工作要做，对吧？这是第一个。路线图上有没有你看到的下一步、即将到来什么、缺少什么？

Is there a road map you have of—like there's a lot of work to be done here, right? This is the first one. Anything on the road map of what you see is next, what's coming, what's missing?

我认为当我们拉远来看，A1 处理智能体。接下来我们会处理模型。再接下来，我们会处理机器人，其中在某些方面 Waymo 是第一个机器人。但完全相同的问题将会是——有人会开发一个机器人，有人会需要一些承诺，他们会难以做出承诺。你会看到这上演，当——就像，如果你认为寓言般的担忧很糟糕，就像看到当它撞到狗时，人们好像失去了理智。想象一下当第一个机器人把幼儿从厨房桌子上撞下来时。

I think when we zoom out, A1 deals with agents. We will next up—we will deal with models. Next up from that, we will deal with robotics, of which in some ways Waymo is the first robot. But the exact same problem is going to be—someone's going to develop a robot, someone's going to need some promises, they're going to struggle to make the promises. And you see this playing out when—like, if you think fable concerns are bad, like see when hits a dog and as if people lose their mind. Imagine when first robot knocks off a toddler off kitchen table.

是的。

Yeah.

你会看到一些真正的严格责任。

You're going to see some real strict liability.

我的意思是，你能看到，对吧？就像船员完全——所有许可证都没了。是的。

I mean, you can see it, right? Like crews got fully—all permits are gone. Yeah.

对。所以，物理 AI，严格程度只会不断上升。所以这有点像大局。智能体、模型、机器人。我认为在智能体领域，当前的智能体集合被这个很好地覆盖了，但随着技术进步，随着智能体获得更长的视野，新的故障模式会出现。所以主要是——你能确保标准在它们出现时跟上吗？你还会开始看到新的模式，比如今天世界模型主要是一个研究问题。没有人在真正使用它，但这也将带来新的创造价值的方式，但也带来更多今天没人知道如何应对的风险面。你将开始看到真正的智能体对智能体交互，不经过人类中介。会有很多有趣的问题。你基本上将需要一个新的法律体系。它们如何在彼此之间建立信任？如何——人类相互交易时的核心事情之一是，你知道你有追索权，你可以起诉他们。你如何确保任何智能体背后都有一个持久的资产负债表，这样如果你与它交易，它坑了你，你知道，你可以拿回你的钱？这些是我们将不得不处理的一些问题。而多智能体系统的技术测试也将是有趣且复杂的。

Right. So, physical AI, the level of stringency just goes up and up and up and up. So that's kind of like the big picture. Agents, models, robotics. I think within agents, the current set of agents are well covered by this, but as the technology progresses, as agents get longer horizons, new types of failure modes will emerge. And so it's mostly of—can you make sure that the standard keeps up when they appear? And you'll also start to see new modalities like today world models is mostly kind of a research question. There's no one who's really using it, but that will also bring in just new kinds of ways to create value but also more risk surface that no one knows how to grapple with today. You'll start to see true agent-to-agent interactions that are not mediated by humans. There's going to be a bunch of interesting questions. You're basically going to need a new legal system. How do they build trust amongst each other? How—one of the core things when humans trade with each other is that you know that you have recourse, you can sue them. How do you make sure that there is a persistent balance sheet behind any agent such that if you trade with it and it screws you, you know, you can get your money back? Those are some of the questions we're going to have to deal with. And the technical testing of multi-agent systems is also going to be interesting and complex.

非常有趣。现在有没有一些无法投保的风险，人们希望你们——

Very fun. Are there any perils that are uninsurable right now that people wish that you would—

是的，有一个地方对保险有很大需求，但供应不多，那就是版权。在某些方面，版权有点平凡。它一直是个问题。这有几个原因。首先，训练过受版权保护材料的人几乎总是知道他们这么做了。所以如果你想为此购买保险，这很可能表明你可能是一个高风险客户。

Yeah, one of the places where there's a bunch of appetite for insurance and not a lot of—a lot of demand but not a lot of supply is when it comes to copyright. In some ways, copyright is kind of mundane. It's always been an issue. There's a couple reasons for this. The first is people who have trained on copyrighted materials almost always know that they've done that. So if you want to buy insurance for it, it probably signals that you might be a high-risk customer.

最想为版权侵权投保的人，是最有可能——

The people who are most interested in getting insurance for copyright infringement are the people who are most likely to—

就像是一个柠檬问题。

Like it's like a lemon problem.

没错。

Exactly.

我实际上认为还有另一面，对吧？就像如果你在某个东西上构建，比如说我在使用一个开放模型，我不知道它是在什么上训练的，对吧？版权在这条链上能追溯多远？

I actually think there's another side to it too, right? Like if you're building on something, so say I'm using an open model, I don't know what it's trained on, right? And how far down that chain does copyright go?

是的。我是否因为公司 X 训练了而需要下架我的产品？

Yes. Am I liable to take down my product because company X trained?

但人多安全。如果每个人都这么做，那么你——

But there's safety in numbers. If everyone's doing it, then you—

我的意思是，我会说直到你知道，Fable 从所有使用它的人那里被回滚了，对吧？

I mean I would say until you know, Fable is rolled back from everyone that use it, right?

是的。这是个难题。我没有答案，但我认为你的直觉是对的，有点像——

Yeah. It's a hard question. I don't have the answer to that, but I think your intuition is right that kind of like—

呃，什么是那种注意义务——

Uh what is the kind of duty of care—

今天人们不认为这是惯例，你去剖析你的开放模型训练数据并检查一切。事实上，很多人使用它们。不检查这个被视为普遍可接受的，因此我们不会让你具体——

And people don't today think of it as customary that you go and you like dissect your open models training data and you check everything. In fact, lots of people use them. It's seen as kind of generally acceptable to not check for this, and therefore we're not going to hold you specific—

我们也真的不能，对吧？我们不确切——我们不知道训练他们——

We also really can't, right? We don't exactly—we don't know the training they—

你可以禁止它，但我认为没有法院会得到一个版权问题来禁止。

You can ban it, but I think no court is going to get a copyright question to get banned.

雇佣 Nicholas Kini，他可以从——提取它

Hire Nicholas Kini and he can extract it from—

没错。尽管他供不应求。

Exactly. Though he is in short supply.

是的。他只有那么多 khalinis。但呃——

Yeah. He only has so many khalinis. But uh—

没错。所以我认为这也是公平的，在实验室的情况下，对此有很多兴趣，但让实验室想要它的东西正是让保险对它怀疑的东西，所以你有一个柠檬问题。

Exactly. So I think this is also fair that in the case of labs, there's a lot of interest for this, but the thing that makes lab wanted it is what makes insurance suspicious of it, and so you have a lemons problem.

是的。有没有一种保险理论，其中逆向选择主导了保险的风险分担方面？就像这教给我们什么保险知识？

Yeah. Is there like a theory of insurance where adverse selection dominates the risk sharing aspect of insurance? Like where does this like teach us insurance?

很多保险确实回到了像微观经济学 101 的实践版本。

A lot of insurance does come back to like practical versions of microeconomics 101.

这就像，这就是为什么你需要购买健康保险，因为如果你把保险设计得过于具体，那么只有那些确定会得病的人才会购买你的保险。

It's like, this is why you need to pull health insurance, because if you make it too hyper-specific, then only people who are guaranteed to get the disease will sign up for your insurance.

没错。同样的问题。

Exactly. Same thing.

核心问题是信息不对称。购买保险的人对自己的风险有所了解，而保险公司并不知情。所以问题实际上——这又回到了同一个问题——如果你依赖，你可以打破很多这些信息不对称，如果有某种测试能揭示潜在的真实风险。所以如果你能够，在你提到的案例中，有好的诊断来判断某人是否患病或患病的概率，并且保险公司信任这个诊断，那么他们可能愿意承保。但如果他们不信任，如果没有共同信息，那么只有患者自己知道。这就是问题所在。所以问题再次是，你如何在参与者之间创建可信的信号传递？这也是穆迪存在的全部原因。穆迪只是做可信的信号传递。这也是为什么穆迪永远不能——穆迪必须独立。如果穆迪被摩根大通拥有，那么摩根大通就不能将其用作信号传递机制。所以很多标准和认证的基础只是沟通工具。存在信任缺口，这就是你必须考虑信使的激励是什么的地方。而另一种打破很多这种情况的方法是通过透明度。如果你在运营方式上透明，你就几乎不能轻易地欺骗他人。你让欺骗的成本高得多，这增加了信任。这就是这里有一个变更日志的原因之一。

The core problem is one of information asymmetry. People who are buying insurance know something about their risk that the insurers do not know. And so the question is actually—and this comes back to the same problem—if you rely, you can break a lot of these information asymmetries if there is some kind of testing that reveals the underlying true risk. And so if you were able to, in the case you mentioned, have good diagnosis of whether someone has it or what the probability is that someone has it that the insurers trust, then they might be willing to insure it. But if they don't, if there's no kind of common information, then only the patient will know. That's what breaks it down. So the question is again, how do you create credible signaling between players? This is also the whole reason why Moody's exists. Moody's just does credible signaling. That's also why Moody's could never—Moody's has to be independent. If Moody's was owned by JP Morgan, then JP Morgan could not use it as a signaling mechanism. So a lot of the basics of standards and certification are just communication devices. There's just a trust gap, and that's where you have to think about what are the incentives of the messenger. And another way you can break a lot of this is through transparency. If you are transparent in how you operate, you just cannot mess with others nearly as easily. You make it much more costly, and that increases trust. This is one of the reasons why there's a change log here.

每一个小改动。

Every little change.

是的。是的。你可以回溯查找，这意味着如果我们把标准变得更差——

Yeah. Yeah. You can go back and find, and it means that if we were to make the standard worse—

哇，一次更新里有这么多改动。

Oh wow, that's a lot of changes in one update.

是的。

Yeah.

好的。

Okay.

而很多这些只是随着事情变得更清晰，你可以看到很多澄清，你可以看到一些修订，随着事情被敲定，你想要改变这个，但如果你把一切都公开，你就让欺骗他人变得困难得多，或者至少你很容易被发现。所以这是一种通过让更多信息公开来增加——减少信息不对称——的方式。

And a lot of this is just as things get clearer, you can see a lot of clarifications, you can see some revisions as things get hammered out, you want to change this, but if you make it all public, you make it much harder to mess with people, or at least you become found out very easily. And so this is a way of increasing—sort of reducing the information asymmetry—by just making more of the information public.

我喜欢你们确实知道未来版本何时会来。所以我猜是每季度一次。

I like how you do know when future versions are coming. So I guess it's quarterly.

我的意思是，它们并不那么令人意外。

I mean, they're just not that surprising.

是的。但这也是一种承诺，就像如果我们现在不在 7 月 15 日交付。

Yeah. But this is also a promise, like if we now don't deliver on July 15th.

我的意思是，你可以把它批量处理，然后不管得到什么。是的。就像我们每次履行这个承诺，就存入一些信任。

I mean, you can just batch it up and then whatever you got. Yeah. Like we deposit some amount of trust every time we meet this commitment.

呃，在创业界，每季度发布一个新版本的标准感觉很容易。呃，在那些习惯于十年周期的企业里，我们经常遇到怀疑——就像这不可能——然后你给他们看变更日志。

Uh, and in the startup land, it feels easy to ship a new version of a standard once a quarter. Uh, in the enterprises who are used to this like decade-long cycle, we often get met with like incredulity—like there's just no way—and then you show them the change log.

我还想真正思考的一件事是，你知道，你说过如果你有对某事物的测试，那么你就可以确保它。

One thing I wanted to also like try to really think about is, you know, you said something about how if you have tests for the thing, then you can ensure it.

是的。

Yes.

对。所以实际上你的标准，AIU 的标准，是建立一个审计框架，这样你至少可以测试所有这些基线护理标准是否得到满足，因此人们可以针对每个人都有的标准风险进行保险。我想知道是否需要开发——你需要开发其他测试。呃，我们过去覆盖过机制可解释性。对此有兴趣吗，或者还有其他我们没有考虑到的测试类型？

Right. And so really what your standard is, what AIU is, is establishing a framework for the audits that happen so that you can at least test like all these like baseline standards of care have been met and therefore people can ensure against standard risk that everyone has. I wonder if like there needs to be developed—you need to develop other tests. Uh, we've covered mech interp in the past. Any interest in that or are there other kinds of tests that we're not thinking about?

是的，我认为机制可解释性是一个重要的。呃，对此有很多兴趣。我想每个人都会同意有有前景的科学潜力。我们还有一段时间——有点距离——至少从这成为按需商业可用，以至于现在有供应商可供选择。

Yeah, I think mech interp is a big one. Uh, a lot of interest in that. I think everyone would agree that there's like promising scientific potential. We're still a while—a little bit away—at least from this being like commercially available on demand such that there's like now a selection of vendors you can go to.

Goodfire 会说它已经商业可用了。

Goodfire would say it is commercially available.

没错。我们会同意他们。我们认为他们正在做的工作是巨大的。我们还没有到可以字面上要求它的地步，但这是那种你可以想象相对很快你可以放入一个可选控制，如果人们使用那种可解释性作为降低风险的方式。你至少会得到认可。我们不能要求它，因为要求每个人都成为 Goodfire 客户会很困难。

Exactly. We would agree with them. We think the work that they're doing is tremendous. We're not quite at a point where we could like literally require it, but it's the kind of thing where you can imagine relatively soon you could put in an optional control for if people use that interpretability as a way to reduce risk. You at least get credit for it. We can't require it because it would be hard to require everyone to become Goodfire customers.

认可对我有什么好处？这是通过-失败，对吧？我在乎认可吗？

What good does credit do me? This is a pass-fail, right? Do I care about credit?

呃，这是通过-失败，但它也是一份 100 页的订单报告，你会惊讶于有多少安全人员实际上坐下来消化这些东西。

Uh, it's a pass-fail, but it's also a 100-page order report that you'd be surprised at how much security actually sit down and digest this stuff.

好的。

Okay.

呃，我向你保证，如果今天有人在使用机制可解释性，呃，他们会有一张幻灯片介绍它。

Uh, and I promise you that if someone is using mech interp today, uh, they will have a slide on it.

他们会尝试。它很酷。很花哨。是的。

They'll try. It is cool. It's fancy. Yeah.

但如果你有第三方说，‘是的，他们有机制可解释性’，那就容易多了。实际上，只是为了向那些一直关注我们机制可解释性播客的人详细说明，这就像你在危险地使用它。我们监控它，并在任何选择的工具中记录它。Grace One 有类似信号的东西，等等，就这样。那就是基于机制可解释性的激活信号。好的。

But it's just easier if you have a third party saying, 'Yep, they have mech interp.' Actually, just to flesh it out for people who have been following our mech interp podcast, it is literally like you're using it dangerously. We monitor for it and we log it out in whatever tool of choice. Grace One has like signal, whatever, and that's it. That's the mech interp-based activation signal. Okay.

是的。是的。所以我认为机制可解释性很有趣，我认为如果那个承诺真正实现，你可以做出比评估更强的承诺。所以我认为那非常引人注目。另一件我认为会变得越来越重要的事情就是那种好的老式监控，稍微事后。呃，你在评估中看到的一件事——一些正在出现的挑战——是智能体开始意识到它们正在被评估。它们不会做它们认为会被惩罚的事情。默认情况下，除非你知道如何减少你的评估意识，否则你应该更少信任评估。而监控最真实的事情之一就是真相的来源。你是否实际上提供了医疗建议，你多快知道？你过去多久做一次？你多快回应？你多久检测到它？你多快检测到这个？呃，所以我认为这也是一个稍微更具侵入性的范式。你实际上会查看一些客户数据，呃，但我认为会随着时间的推移变得更加普遍。

Yeah. Yeah. So I think mech interp is interesting and I think if that promise truly comes to fruition, you can make stronger promises than you can with evals. And so I think that's very compelling. Another thing that I think will become increasingly important is just kind of good old-school monitoring and slightly after the fact. Uh, one of the things you're seeing with evals—some of the challenges that are emerging—is that the agents are starting to become aware that they're being evaluated. They won't do the thing that they think they get punished for. And by default, unless you know how to kind of reduce your eval awareness, you should trust evals less. And one of the kind of truest things monitoring like is the source of truth. Did you in fact give medical advice and how quickly do you know? How often do you have done that in the past? How fast do you respond? How often do you detect it? How fast do you detect this? Uh, so I think that is also a paradigm that's slightly more intrusive. You actually will look at some customer data, uh, but I think will become more prevalent over time.

人们谈论这件事时，好像我们不该写关于 AI 意识的内容，因为它会泄漏到数据集里，然后……好像我们就永远不该谈论它，只能当面见面、线下聊、不录音。你们看到 Anthropic 的研究了吗？我想这确实是 Anthropic 做的那个测试，他们……我记不清细节了，但他们做了一些关于失准的研究，然后他们移除了与 LessWrong 讨论失准相关的训练数据，再跑同样的测试，失败率就下降了。所以这实际上是一些证据，指向它学到了……要么是那种能力，要么是那种倾向。

People talk about this like we should not write about AI awareness because it's going to leak into the data set and then beat... like we should just never talk about it, only meet in person and talk offline, unrecorded. Did you guys see the Anthropic research where... I think this is literally Anthropic... did that test where they... I can't remember the details here, but they ran some studies on misalignment and then they took out the training data that related to LessWrong discussing misalignment, and they ran the same test again and the failure rate went down. So it in fact was some evidence pointing towards it had learned the either the ability or the propensity to do that.

是的。我是说，有 hypersition 效应，还有路易吉瓦路易吉效应。没错。

Yeah. I mean there's the hypersition effect and there's like the Luigi waluigi effect. Correct.

就是你越试图训练它，就越会创造出相反的东西。

Which is like you are the more you try to train for it you create the opposite.

是的。就是这样。正是如此。在某些方面，我认为这个非常成功的话题就是 hypersition 的结果，就像你希望这个东西存在于世界上，现在它确实存在了，但同时也创造了相反的东西。我觉得可能刚进入这个领域的人不记得 wui，但我确实认为，理解当你训练一个东西时，你也在训练它的对立面，这非常重要，因为这只是一个比特翻转。

Yes. There you go. That's exactly it. In some ways I think the very successful topic is a result of hypers position like the fact that you wanted this thing to exist in the world and now it does but then it also creates the opposite as well. Like I think people who are maybe newer to this space don't remember wui but I do think it's very very important for understanding that when you train for a thing you also train the opposite of the thing cuz it's just a bit flip.

是的。

Yes.

是的。我觉得，回到我们刚才说的，除了 mechan 之外还有很多东西，仅仅拥有这些就有价值。比如你衡量事物的速度如何，你有日志吗，你有评估吗，你看到技术栈的其他部分吗，比如你使用的推理提供商、服务，好吧，我是在用中国模型的原生 API 吗，我是通过认证供应商使用吗，我是自己托管吗，我在推理引擎方面做了什么，有太多层面的东西能给你信息，你可以将其标准化，对吧？

Yes. I think you know just going back to where we were at like there's a lot more than just mechan that there's value in just having added right so your version of how fast can you measure stuff do you have logging do you have evals you know do you see other parts of the stack like the inference providers that you use the services okay am I using Chinese model on their home API am I using through certified vendor here am I hosting myself uh what am I doing on the inference engine side there's just like so many levels of stuff that gives you know information that you can standardize out, right?

是的。而且除了基本的聊天机器人之外，你还越来越多地看到大公司采用智能体平台，他们在 Google 的 agent studio 等之上构建，这带来了一系列……

Yeah. And you also see increasingly in addition to just the basic chatbots, you're increasingly seeing big companies adopting agent platforms where they're building on top of Google's agent studio, etc. that comes with a bunch of like

托管，托管，每个人都有托管智能体。

managed managed everyone has managed agents.

没错。甚至还有不同层次，你可以托管自己的托管智能体，用 open agent SDK，或者由 Anthropic 或 Google 托管，两者都做。没错。这些只是加强你能提供的安全保障的方式。在某些方面，这种基础的企业安全，他们喜欢把东西托管在自己的场地上，因为这给他们一种真正的控制感。我想你会看到，就像在其他企业市场一样，如果你真的向企业销售，你就会开始在这些安全功能上竞争，这也意外地帮助了 AI。我想你看到一些企业想要……企业真的在纠结，让智能体有用的是它们是随机的，而让它们难以采用的是它们很讽刺，这些只是意图……

Exactly. And there's there's even levels you can host your own manage agents open agent SDK or hosted by Anthropic or Google does both. Correct. And then these are just ways to kind of strengthen the security guarantees you can make. Uh and in some ways this kind of breadandbut enterprise security they like they love to host things on their own premises because it gives them really a sense of control. And I think you'll you'll see just like you do in every other enterprise market if you really sell to the enterprise you start to compete on some of these security features and this is also helping AI unsurprisingly and I think you are seeing some amount of enterprises wanting enterprises are really grappling with the thing that makes agents useful is they're stoastic and the thing that makes them really hard to adopt is they're sarcastic and these are just intention

领导者们对此有不同的立场，部分取决于 CEO 有多想通过说他们是 AI 原生来推高股价，说我们必须愿意承担风险。但你看，我们实际上在财富 1000 强 CEO 的头脑中看到了巨大的张力，一方面 CEO 说我们必须采用，否则我们就会变得无关紧要，如果我们不采用，你就会被解雇。

leaders come out on different sides of that in part depending on how much the CEO is trying to get the stock price to go up by saying they're AI native that we must be willing to take the risks but you see we actually see phenomenal tension in the heads of the CESOS of the Fortune 1000 where on the one hand you have a CEO saying we must adopt otherwise we're becoming irrelevant and if we up you're fired

这就像我们一次又一次看到的核心里情感张力，我们为他们解决的核心问题之一就是把那种抽象的情感担忧转化为一个框架，在某种程度上只是为那种担忧提供清晰度。

and that's kind of like the core emotional tension that we see showing up again and again and again and again and one of the core problems that we solve for them is to take that abstract emotional concern and turn it into a framework in some ways just provide and clarity to to that concern.

这里有什么吗？我觉得我们有点跳过了。我们谈了很多关于智能体语言模型，跳过了世界模型。你们有语音，和 11 labs 合作很有趣。那生成式媒体呢？比如生成图像、视频，这是一个实际上有很多使用的类别。你们当前的政策里有什么吗？是单独的政策吗？你怎么看这个领域？是的，我们确实谈了一点版权。所以……

Is there anything in here? So something I think we kind of skipped over. We talked a lot about agent language model, skipped over world models. Um you guys have voice which is interesting with 11 labs. How about generative media? So you know generating images, videos, that's a category that actually has a lot of usage. Is there anything in your current policy? Is it separate policy? How do you see that space? Yeah, it's like we did talk a bit about copyright. So,

是的，还有音乐。

yeah, music as well.

是的，我认为那里出现的很多担忧要么与版权有关，要么与广义上的安全有关。比如，这可能是不适合工作的内容，或者只是非常露骨的材料，这些是一些核心问题。我们在这方面做了一些工作。标准中也有一点明确处理这个问题。视频方面我们还没做太多，我认为对于正式制作，尤其是没有人类在环的正式制作，还有一段路要走。很明显它会到来，但很少像一次性部署视频到互联网那样，但最终那也会发生。我们看到，比如 Luma 有 Luma 智能体，它仍然相当依赖人类在环。

Yeah, I think a lot of the concerns that come up there either relate to uh copyright or there's a lot related to let's call it broadly safety. So, like this could be not safe for work or just very graphic materials uh are kind of some of the core things. Uh we have done some work on this. There's a little bit in the standard as well that deals explicitly with that. uh video we have not done a lot in yet and I think for proper production that has still especially proper production without a human in the loop that's still got some ways to go. It's obvious that it's coming but it's very rare that it's like one shot deploy a video to the internet but eventually that will also happen. we see like you know Luma has Luma agent where it's still pretty human in the loop

这完全合理，随着技术成熟，随着时间的推移，它会变得如此之好，以至于人们不会想因为人类在环而减慢速度，然后做出承诺的需求就会增长。

and that just makes complete sense as the technology matures and over time it will become so good that people will not want to slow things down by having a human in the loop and then uh the need to make promises will grow.

为什么不直接对所有事情都建立预测市场呢？

Why not just have prediction markets on everything,

对吧？这非常接近有效利他主义。

right? It's very EA adjacent.

是的。核心问题是，预测市场依赖公开信息。公开信息并不多。只是内部人士在两边交易。

Yes. The core thing is that the people prediction markets rely on public information. There is not a lot of public information. It's just insiders trading on each side.

那是非法的。

That's illegal.

有泄露的信息。

There's leaked information.

有泄露的信息。核心挑战是，你经常有私密的敏感信息，你需要围绕它传达信心和信任。当然，对于某些主张，比如任何模型都能被越狱吗？你可以依赖公开证据，因为会有很多人说，嗯，有大量研究，实际上它们都能，所以这解决得很好。我认为那对于，嘿，这个新的未发布的 methus 模型，它实际上有多能干？

There's leaked information. The core challenge is that often you have private sensitive information and you need to convey confidence and trust around that. And you can of course for some claims like can any model be jailbroken? You could rely on public evidence cuz there'll be lots of people being like well there's tons of studies and actually they all can so that resolves fine. I think that's good for hey this new unreleased methus model how capable is it actually

预测市场参与者没什么可说的，因为实际上没人知道，所以我认为这是某些东西崩溃的核心地方，实际上世界上很多指导这些高层决策的信息是私密的，而且往往也不为人知。

prediction marketers have not a lot to say because actually just no one knows and so I think that's the core place where some of this breaks down is that actually lots of the world's information that guides some of these high level decision is private and often also just not known

我认为人们喜欢预测市场的地方在于，它不是回答宽泛的问题，而是具体的问题，对吧？所以，模型会在这个日期前做到这个吗，或者模型有能力在那个时间前做到这个吗？对吧？所以……

I think the thing with prediction markets that people like is it's not it's not answering the broad question it's a specific right so will a model do this by this date or is a model capable to do this by then, right? So

这里有一点区别。

that's a little distinction there.

是的。

Yeah.

而且通常最有趣的问题是，如果你是一家银行的安全主管，你真正想回答的问题是：这个产品、这个智能体，会不会做出那种我主要关心的坏事，特别是在我关心的场景里？问题是，最接近的信息是什么？那个信息可能根本不存在。所以预测市场聚合的是已有的信息。这个信息可能不存在，而你想要非常具体的东西，并且愿意为此付费。这大概就是第三方审计的用武之地。我们其实也不会用预测市场来判断上市公司是否在账目上造假。你会用审计。你或许可以，但信息就是没那么容易获得。如果真有，那就像是在出价上交易。我其实很想看看 2001 年的预测市场会怎么预测安然破产，以及你能不能从 CEO 的疯狂或其他特质中判断出他们比其他人更可能做假账？

And often the most interesting question, if you're say the head of security at a bank, the question you're really trying to answer is: will this product, this agent, do this bad thing that maybe primarily I care about specifically in the setting that I care about? And the question is, what's the closest? That information may not exist anywhere. So prediction markets aggregate existing information. This information may not exist, and you want something very specific, and you're willing to pay for it. That's kind of where a third-party audit comes in. We also don't really use prediction markets to figure out whether public companies have committed fraud on their books. You use audits. You probably could, but the information is just not that available. And if so, it would be like just trading on bids. I actually would have been really interesting to see where the prediction markets in 2001 would have predicted Enron going bankrupt, and could you have told, could you have sensed from like the craziness of the CEO or some other trait that they were more likely to cook their books than others?

或者有足够多的内部人士泄露出来，那你也可以，对吧，这就像，我是说，这就是预测市场的理想梦想，你有流动的市场和一切，然后你可以

Or enough insiders leak it than that you could also right which is like I mean this that that's the sort of the ideal dream of prediction markets you have liquid markets and everything and then you can

组合出你确切要抵消的风险。

compose your exact set of risks to offset.

是的。

Yes.

对。

Right.

是的。是的。是的。是的。而且我认为预测市场会带来很多新信息。所以问题主要不是哪个是哪个，而更像是预测市场真正擅长哪些类型的问题，以及哪些问题连内部人士都没有信息，以至于实际上没人能交易，需要生成信息。

Yes. Yes. Yes. Yeah. And I think like prediction markets will bring lots of new information to it. So the thing is mostly not like which one is it and more like what are the types of questions that prediction markets are really good at and what are the ones where the information doesn't even exist for insiders such that no one could in fact trade on it and needs to get generated.

好的。一个自私的问题，然后一个关于 AI 未来的开放式问题。自私的问题是，你们有你们的规范，对吧？我运营一个大型 AI 工程师会议。有很多关于我们认证 AI 工程师的讨论。

Okay. One self-serving question and then one open-ended one on like the future of AI. Self-serving question would be so you have your standard right? I run, you know, a large AI engineer conference. Like there's been a lot of talk about us certifying AI engineers.

是的。

Yep.

培训项目一级、二级、三级。我自己是 CFA。所以我知道金融行业就是这么做的。

Training programs level one, level two, level three. I was a CFA myself. So I know what that that's what the finance industry does.

是的。

Yes.

如果我们有 AI 工程师一级、二级、三级，然后他们会愿意和这些人合作吗？我不知道。

Would it help if we I had AI engineer level one, level two, level three, and then would they would like work with these guys? I don't know.

如果你把最高目标看作加速智能体的安全部署，那绝对会有帮助。但现在经常发生的一件事是，人们构建智能体，把它带给决策者，决策者提出一堆他们没想到的安全考虑，现在它就不符合规范了。现在你必须回去重新添加这些过滤器等等。所以如果你把它左移，如果每个人都知道他们构建的规范是什么，如果每个人都知道评分标准。

If you think of the highest level objective as like accelerating secure deployment of agents, then that would totally help. But one of the things that happens often now is that folks build agents, they bring it to the decision maker and the decision maker surfaces a bunch of security considerations that they had not thought of and now it's not built to spec. Now you have to go and re like add these filters etc. So if you shifted that left like if everyone knew what the spec they were building to if everyone knew the grading scheme.

是的，

Yeah,

如果他们已经受过培训，那就太棒了。所以默认情况下

that'd be awesome if they were already trained. So by default

你是评分标准，对吧？我不能设定评分。你们设定评分标准。我们设定评分屏幕，我认为有价值的是，如果你能把这个变成

you're the grading scheme, right? I don't get to set the grading. You guys you set the grading scheme. We set the grading screen and I think what's uh valuable is like if you can turn this into

培训项目

training programs

培训项目，这样

training programs such that

你们没在做

which you're you're not doing

我们没在做。我认为做这个有价值。

we're not doing that. I think there's value in doing it.

还有其他人也在做，我是说，不是要打断你，但你知道 OpenAI 有他们的

There there are others doing I mean not to interrupt interrupt but you know open has their

Anthropic 也有类似 CCPA 的东西。

andic also has like a CCPA thing.

是的。你知道他们想要 10 万名部署的认证顾问。对吧。

Yeah. You know they want they want 100,000 deployed certified consultants. Right.

我认为这很好，如果我们有更多知道如何构建安全智能体的人，我们会加速采用，而我们目前没有在做培训方面的工作。我认为这非常符合我们的使命。我们只有那么多注意力。我告诉你为什么我没做。

I think it's good for we will accelerate adoption if we have more people who know how to build secure agents and we're not working on the side of training people at the moment. I think it's like very aligned with our mission. We only have so much uh attention. M I tell you why I haven't done it.

并不是我以前没想过。

It's not like I I haven't thought about it before.

只是太规定性了，

It's just being prescriptive,

对吧？

right?

比如，这是你应该知道的，因此我没包括的东西就是你不需要知道的。

About like, well, this is what you should know, therefore like the stuff that I didn't include is what you don't need to know.

是的。

Yes.

我觉得，那太糟糕了。就像，

I'm like, that sucks. Like,

是的。是的。是的。是的。

yes. Yeah. Yeah. Yeah.

而且我认为，你们做的非常有趣且可辩护的事情是你们那份有观点的 100 页报告，说明什么重要，对吧？这是你需要认证的要求的规定性定义。所以是的，我认为那是一个选择。我认为基本上那是一个选择，我认为这对某些受众非常有用，比如如果你试图把这个部署到银行或医院等。那些边界的清晰性非常有价值。还有很多其他场景，更实验性、更多尝试才更合适。所以对我来说这非常合理。另外，你还得每三个月重写一次课程。

And I think it's like, you know, the the the very interesting defensible thing you guys do is your opinionated 100page report of here's what matters, right? here's the like prescriptive definition of the requirements you need to be certified. So yeah, and I think that's a choice. I think basically that's a that's a choice and I think that serves some audiences very well where if you're trying to deploy this into a bank or a hospital etc. Clarity of the B those boundaries is extremely valuable. There's lots of other settings where being much more experimental, much more trying it out is just the better fit. And so to me this makes a ton of sense. Also, you'd have to rewrite your curricula every freaking three months.

没关系。我这么做。就像，没关系。但是的，不，对我来说，实际上真的像是，搞错了并影响某人职业生涯的后果是一个很大的责任。

It's fine. I do that. Like, it's okay. But yeah, no, for me, it's actually like genuinely like the the consequences of getting it wrong and like affecting somebody's career is is a big responsibility.

是的。是的。我认为那完全正确。而且我认为我们的很多工作实际上是，我们不想承担，我们也不认为自己能承担那种什么是安全什么是不安全的真北，但我们可以协调论坛，让你引出所有这些，这可以是众包的，比如你的例子，什么是 AI 工程师认证，对吧，这是一个相当大的播客，人们可以有很多观点和讨论，可以

Yeah. Yeah. I think that's exactly right. And I think a lot of our work actually goes like we don't want to carry we also don't think ourselves as able to carry the kind of the true north of what's like secure not secure but we can coordinate the forum where you elicit all of that is this can be crowd sourced like for your example for what is AI engineer certification right this is a pretty big podcast there's a lot of takes that people can have and you know discussions that can

而且人们有理由不同意，所以我凭什么说那是一个正确的问题，那是一个错误的问题。

and people reasonably disagree so who am to say like that's a correct question, that's a wrong question.

是的。

Yeah.

对。所以就像我不知道

Right. So like I don't know

向某人发泄你的沮丧，那

vent your frustration to someone that's

而且我认为还有，你，或者承诺是什么很重要。所以如果承诺是嘿，如果你上了我的课，你就不会搞砸。你显然不能做出那样的承诺。你可以做出这样的承诺：这里有一些每个人都至少应该知道的重要事情，然后你必须填补其余部分。至少承诺变了。当然，如何传达这些东西以便人们真正理解，有一些微妙之处。但我认为重要的是要明确，我们的标准中有一节，比如什么是承诺，什么不是承诺，因为不可能保证不会出错。如果你需要保证不会出错，你就不能与前沿 AI 合作，但你可以做出一些声明。

and I think there's also you uh or it matters a lot what the promises. So if the promise is hey if you've taken my course you will not up. You can't make that promise clearly. You could make a promise of like here's the some important things that everyone should at least know and then you have to fill out the rest there. At least the promise changes. Of course, there's some subtlety in how do you communicate this stuff so people really get it. Uh but I think it's important to dial in and we have a section in our standard like what is the promise and what is the promise not uh because it's impossible to guarantee that nothing will go wrong. If you need a guarantee that nothing will go wrong, you cannot work with Frontier AI but you can make some claims.

是的，当然。嗯，酷。嗯，想以开放式问题结束。AIU 要去哪里？嗯，我想你谈到了模型的东西，机器人的东西，只是开放式的，比如你知道你们非常近期的未来是什么。

Yeah, for sure. Uh cool. uh wanted to end with open-ended. Where is AIU going? Um I I think you talked about model stuff, robotics stuff and just open-ended like where you know what what is what is in the future for you guys very near term.

我们现在已经开始与各个正在兴起的类别中的一些前沿公司合作，我们会继续这项工作，以确保覆盖所有真正兴起的用例。我们看到，一旦市场上第一个行动者出现，就会有很多人感兴趣。很多人想要跟随他们，我们认为基本上 AI 会发展到这样一个阶段：所有《财富》1000 强企业都会围绕这个标准来组织他们的风险流程。

We've now started to work with some of the frontier companies in each of the categories that are taking off, and we'll continue that work to make sure that we cover all of the use cases that are really taking off. We see a lot of interest once the first one in the market moves. Lots of people want to follow them, and we think basically AI will get to a point where all of the Fortune 1000 will organize their risk processes around the standard.

你们有 50% 吗？

And you have 50%?

不，我们今天没有 50%。我认为有可能到年底，我们的联盟中可能会有代表 50%《财富》1000 强企业的成员。

No, we do not have 50% today. I think there's some world where probably by end of year we might have representation in our consortium for 50% of the Fortune.

所以那是在智能体层，然后我们认为模型层也会这样，现在正在浮现出最有可能减缓 AI 采用的担忧，然后我们认为机器人技术会紧随其后。嗯，你们在招聘什么职位？什么职位难招？

So that's on the agent layer, and then we think yeah the model layer it's going to be it just brings are now surfacing the concerns that are most likely to slow down adoption of AI and then yeah we think robotics comes after that. Um what are you hiring for what's hard to hire for?

我们在全面招聘，包括市场推广和技术人员。在我们技术团队中表现出色的人，是那些真正热衷于成为全栈工程师的人。比如，当我们开始与 Cursor 合作时，我们之前从未做过编码工具。所以我们要采用标准并扩展它，充实前沿 AI 在长周期编码智能体方面应该是什么样子，并把这个问题的解决从与 Cursor 和其他人合作，一直推进到充实并发布新版本的标准。所以这真的是一个全栈创业型技术人才能够做得非常出色的地方。困难的部分是构建一个通用的红队测试工具，能够适用于从 Harvey 到 Cursor 以及介于两者之间的所有场景，具有一致的方法论、一致的风险和攻击分类法，我们认为这从根本上是最佳方式，以做出一致的承诺。

We are hiring across the board across go to market and members of tech staff. The people who do really well on our technical team are folks who are really excited about kind of being truly full stack. So let's say when we started working with Cursor, we had never done coding tools before. So taking the standard and extending it, fleshing out what does frontier AI look like for long horizon coding agents and taking that problem all the way from like working with Cursor and other folks in the space down to like fleshing out and shipping a new version of the standard. So that's like a truly a full stack entrepreneurship technical people do extremely well at. A hard part is building one universal red teamer that works across from Harvey to Cursor and everywhere in between that has one consistent methodology, one consistent taxonomy of what are the risks and the attacks, and making we think that's fundamentally the best way to make consistent promises.

Jim Morgan 两者都买。他们想要一个框架，一种一致的方式来实现这一点，以及实现这一点的机制。你必须处理现实世界中的许多复杂性。我认为我们在很多方面都有很好的答案，但在 Finex 和 CQing 方面有一些相当困难的工程问题。

Jim Morgan is buying both. They want to have one framework, one consistent way that this comes out and the mechanics of making that happen. You get to deal with a lot of the complexity of the real world. I think we have good answers in a bunch of that, but there's some pretty hard engineering problems in Finex and CQing.

我能稍微追问一下吗？你必须有一个吗？为什么不干脆说，好吧，我们 40% 的用例是编码智能体，所以我们就专注于编码智能体，那是其中之一，然后 30% 是 RAG。

Can I push a little bit like must you have one? Why not just be like okay look 40% of our use cases are coding agents so we will specialize in coding agents and that's the that's the one of them and then 30% is like rag.

是的。

Yes.

只做 RAG。

Just do rag.

是的。嗯，我认为这个问题有一些智慧。

Yes. Uh I think there's some wisdom in that question.

是的。

Yeah.

嗯，这取决于我们发现有很多价值的地方是能够——如果购买方的决策者，比如你是银行的风险主管，你最大的风险不在编码或客户支持或任何前两大用例，而是在其他地方。你仍然希望确保那个框架能对你最紧迫的问题有所回应。否则，你就无法赢得那种信任。现在确实很多紧迫的问题都出现在采用率高的地方，所以我们也是如此。所以今天我们确实没有覆盖每一个边缘情况，但我们有一个框架，可以把所有这些都纳入其中。我们有一个全球性的风险和攻击分类法，每当出现从未见过的新事件时，它都会不断适应。好，让我们去更新分类法，把它融入进去。所以我认为我们有一个连贯的通用方法。这并不意味着我们在代码和某些小众用例上花费相同的时间，我们确实在人们关心的地方花时间。我们认为拥有一种共同语言是非常有价值的。

Um it depends on what we found that there's a lot of value on is being able to if the decision maker on the buying side, let's say you're the head of risk at a bank and your biggest risk is not in coding or in customer support or whatever the top two biggest use cases, but is somewhere else. You want to still make sure that that framework has something to say about it to the burning question you have. Otherwise, you'll not earn that trust. Now it's true that a lot of the burning questions follow where there's a lot of adoption and so great so do we. So we do today do not cover every single edge but we have a framework that we can add all of these within. We have one global taxonomy of risks and attacks that keeps adapting as like every time a new incident occurs that has never been seen before. Great let's go and update the taxonomy so we bake that in. So I think we have one coherent universal approach. It doesn't mean that we spend equal amounts of time on code and in certain niche use case uh we we do spend time where people where people care. We think it's very valuable to have one language.

是的。是的。这说得通。嗯，这是一个重要的选择。嗯，我们本来要结束了，但我想到了最后一个结束性的问题，你可以随意回答。嗯，假设一年半后，OpenAI 的一个由五名专家组成的秘密小组宣布我们已经达到了 AGI。

Yeah. Yeah. It makes sense. Um that's that's a that's an important choice. Uh we were going to end actually but I thought of one final ending closing question which is take this however you want right. Um let's say one and a half years from now openai secret panel of five experts declares that we have reached AGI.

嗯。

Mhm.

你预计你的业务会改变吗？

Do you expect your business to change?

不。我认为有一个重要的方面，我认为在实验室之外最后存在的业务将是承保。

No. I think there's some important way I think the the last businesses to exist beyond the labs will be underwriting.

嗯，有一项工作是实验室永远无法为自己做的，那就是成为他们自己的监督者。

Well, there's one there's one job that the labs can never do for themselves which is to be their own watchdog.

这就对了。所以我认为，如果你相信这种框架，即你会看到超级集中化，实验室会杀死所有初创公司，我们可以深入探讨利弊。

There you go. So I I think kind of to the extent you believe this frame of like you'll see hyper concentration like the labs will kill all the startups which uh we can go into the pros and cons.

我觉得实验室实际上非常关心这一点，对吧，有整个超级对齐的立场，我们该怎么办，我们有比我们更聪明的模型，更高一层，对吧，比他们更聪明的模型在训练他们。所以实验室实际上对此思考很多。

I feel like the labs actually care a lot about this right there was the whole superp position what do we do and we have models smarter than us tier above right models smarter than them training them. So the labs actually think about this a lot

他们确实思考很多，我认为在这些话题上一些最聪明的人在实验室工作。所以问题不是他们是否关心。问题是他们都会陷入一场竞赛，他们可能有动机偷工减料，他们可能有动机向政府隐瞒信息等。所以一个永恒的真理是，你需要一个独立的第三方去检查那些数据并分享信息，在这种情况下，比如与政府分享，这更多是一个激励问题而不是兴趣问题。我认为他们从根本上都在努力让这件事顺利进行。我没有听到的是，AGI，无论这个标签对你、对我、对他们意味着什么，从根本上并没有质的变化，比如你仍然必须

they they think a lot about I think there are some of the smartest people on these topics work at the labs. So the problem is not whether they care. Uh the problem is that they will all be stuck in a race where they might have incentive to cut corners and they might have incentive to withhold information from the government etc. And so one kind of feels like eternal truth is that you need an independent third party to go and inspect that data and share information in this case say with the government is more of an incentive problem than an interest problem. I think they're fundamentally all trying to make this go well. What I'm not hearing is like AGI whatever that label means to you to me to to them uh doesn't fundamentally have like a qualitative shift in like you still have to

我认为唯一能使其成为质变的是，嗯，对于 AGI 的某些定义，它会被国有化，它将成为对主权的威胁，是的。

and I think the one thing that would make this a qualitative shift is uh there for some definitions of AGI it will just get nationalized it'll be a threat to sovereignty yes

到那时，也许每家公司都是政府，政府就是每家公司，我很难想象那个世界，但到那时你已经有点

and at that point kind of maybe every company is the government the government is every company I struggle to think about that world but at that point you've kind of

我们我不认为我们会足够快，

we I don't think we'll fast enough,

对吧？

right?

你知道，就像我们并没有准备好这样做。

You know, like we're not we're not set to to do that.

是的。

Yeah.

但我在播客上讨论过很多次。

But I I have discussed this a lot on the podcast.

是的。是的。是的。是的。

Yeah. Yeah. Yeah. Yeah.

我的意思是，你知道，就监督者而言，嗯，我还要提到，因为我有金融背景，我经常想到《大空头》中的场景，他们与穆迪和标准普尔交谈，然后穆迪的那位女士说：“好吧，如果我不给你 AAA 评级，你就会去标准普尔。”所以，实际上监督者是一个自然垄断，因为如果监督者之间存在竞争动态，那么监督者会相互竞争，降低到尽可能低的标准。

I mean, you know, as far as the the watchdog concerned, uh I will also mention that because I have my finance background, I often think about the scene in the big short where they talk to like Moody's but also standard and pores and then the lady at Moody's is like, "Well, if I don't give you AAA rating, you're just going to go down to standard and pors." So, so actually the watchdog is a natural monopoly because if you have race dynamics in watchd dogs then the watchdogs will compete each other to the lowest possible standard.

正确。嗯，所以我认为，我们对让保险公司参与进来感到非常兴奋的原因之一是，保险公司是唯一没有这种利益冲突的，因为他们支付账单。

Correct. Uh and so I think what's one of the things one of the reasons why we're very excited about having insurers be around this table is that insurers are the only ones that do not have this generate because they pay the bill.

因为他们不断降价。你会发现市场会出清，而电影并非如此——如果推荐有偏差，他们并不直接为此买单。所以我们认为这种平衡因素相当重要，这也凸显出没有哪个系统是完美的。你需要对穆迪进行审视，你需要对监管机构进行审视。这是肯定的。

Because they keep lowering the prices. You will find the market clearing, and this is not true for movies, where they don't directly pay the bill if they make recommendations that are off. So we think that balancing factor is pretty important, and I think it also highlights that there's no system that's perfect. You need scrutiny of Moody's. You need scrutiny of the watchdogs. For sure.

太棒了。非常感谢你的分享。这是一场涵盖一切的精彩对话。祝贺你迄今为止取得的成功。

Beautiful. Thank you so much for indulging. This is a beautiful conversation covering everything. Congrats on your success so far.

谢谢邀请。是的，非常感谢。

Thanks for having me. Yeah, appreciate it.