为什么 AI 的下一个突破可能来自大实验室之外

各位，欢迎回到播客。

Guys, welcome back to the podcast.

很高兴来到这里。

Happy to be here.

谢谢。

Thank you.

你的胡子。

Your beard.

没想到我们还会再做一次。这，我简直不敢相信。这太棒了。

Didn't think we'd ever do this again. This is, I can't believe it. This is great.

我是说，Martine 正在打造这些千亿美元级的公司。太忙了，没空上这个播客。所以，

I mean, Martine's just building these like hundred billion dollar companies. Too busy for this podcast. So,

至少像风投那样把功劳揽到自己身上。

At least taking credit for as VCs do.

没错。嗯，我们今天有很多要讨论的，但首先，Aaron，不如从你开始？前沿的节奏。你对那里发生的事情以及随后的讨论有什么反应和反思？

Exactly. Um, we have a lot to discuss today, but first look, Aaron, why don't we start with you? Pacing the frontier. How have you reacted and reflected on what's happened there and just the discourse that's followed?

哦天哪。我觉得这个问题应该从 Martine 开始。他打了很多漂亮的地面战。嗯，我或许先说一件我们可能都同意的事，然后我们再看看可能在哪里产生分歧。嗯，我想我们会同意，现在任何处于前沿的 AI 实验室都应该以最安全的方式构建，拥有最高程度的治理和安全，以及无论你对对齐的定义是什么。这是一个极其重要的研究领域。这对 AI 的扩散来说是一个极其重要的领域。如果没有极其高质量、能被企业信任、不会不断入侵系统的产品，你就无法实现 AI 的扩散。所以当我读到 Dario 的帖子时，嗯，我实际上几乎不同意任何内容。嗯，因为它讲的都是如何更好地保障这些系统的安全、沙箱、更好的测试。围绕测试人员的嵌入性质会有一些争论，比如你是否同意那些人选，以及整个行业是否都对此达成一致？但我认为实际上所有主要观点可能都是中肯和恰当的。那么唯一的问题是，这会不会被用来做我们可能不同意的事情，比如因为监管控制而大幅放缓 AI 的发展，从而让人难以与前沿实验室竞争，嗯，或者政客们最终会接过这个信息并加以利用，甚至导致更糟的结果，比如被用来更快地禁止数据中心等等。所以我认为这个话题的实质内容实际上极其重要，我认为对 AI 的整体进步非常重要。然后问题是你该怎么做，尤其是从监管角度该怎么做，而这可能正是行业会在连续谱上落在非常不同点位的地方。但 Martin 在确保我们不会利用这一点进行监管俘获方面打了一场漂亮的仗。嗯，我也同意这一点，但我认为节奏讨论中的想法很重要。嗯，你知道，再次，这有点像一个有趣的概念，因为也许它甚至不是节奏，而只是良好的卫生习惯和良好的工程实践。所以有了良好的工程实践，显然会有轻微的放缓，但这种放缓显然能加速你的扩散，因为如果没有人信任使用 AI，你就无法让任何 AI 得到扩散。所以这篇帖子非常合理，但氛围不对。比如有员工说这会导致 10% 的物种灭绝概率，而你知道 Dario 说什么

Oh boy. I think we should start with Martine on this one. He was fighting lots of good ground wars. Um, I maybe I'll say one thing that we probably all agree with and then we can figure out where we maybe kind of fracture off. Um, I think we would agree that any AI lab right now at the frontier should be building in the safest way possible with the highest degree of governance and security and, you know, kind of whatever your definition of alignment is. Like this is an incredibly important area of research. It's an incredibly important area for the diffusion of AI. Like you're not going to have AI diffusion without extremely high-quality products that can be trusted by enterprises and that aren't kind of constantly hacking systems. So when at least I read the Dario post, um, I actually didn't disagree with almost anything. Um, because it was all about how do you have better security of these systems, sandboxing, better testing. There's going to be some debates around the embedded nature of the testers and like do you agree with who those are and does the industry all align on that? But I think actually all of the major points were probably salient and appropriate. Then the only question is does this get sort of used or leveraged to do things that maybe we don't agree with, which would be like a slowdown of AI dramatically because of regulatory controls that would sort of not make it easy to compete with the frontier labs, um, or do politicians kind of end up taking the message and run with it and maybe even worse outcomes happen like it's used to ban data centers far faster and whatnot. And so I think the actual substance of the topic is actually incredibly important and I think very important for AI advancement in general. And then the question is what do you do about it, especially what do you do about it from a regulatory standpoint, and that's probably where the industry is going to land on very different points in the continuum. But Martin was putting up a good fight on like let's make sure that we don't use this for regulatory kind of capture. Um, which I also agree with, but I think the ideas in the pacing conversation are important. Um, you know, again, like it's a little bit of a funny concept because maybe it's not even pacing as much as just like good hygiene and good engineering. And so with good engineering, obviously there is a slight slowdown, but it's a slowdown that obviously allows acceleration of your diffusion because you wouldn't be able to have any of the AI be diffused if nobody would trust using it. So the post is very reasonable, but the atmospherics are not right. Like so an employee is like this is going to kill whatever 10% chance of species extinction and you know what Dario says

我同意他的地方多于不同意的地方，对吧？就像

I agree with him more than I disagree with him, right? Like

在电视上，就在他发布这些东西的同一天，所以某种程度上你不能孤立地进行这些对话，当然如果他同意物种灭绝的话

On TV on TV the same day that he landed these things and so in some way you can't have these conversations in isolation which is of course if he's going to agree in species extinction

他发的这篇帖子看起来像是这种软弱无力的投降，完全不足以应对当前的任务。所以我认为氛围完全被破坏了。我的很多评论都是关于氛围的。

This post that he has looks like this milktoast capitulation that's totally not adequate for the task at hand. And so I think the atmospherics are totally broken. And a lot of my comments were on the atmospherics.

然后我有个小意见，但它真的让我很困扰，因为我被阻碍了，那就是我认为用节奏来描述这件事是错误的。首先，是的，它与安全是正交的。

And then I have this quibble, but it really bothers me because I'm impeded, which is I think pacing is the wrong way to describe this. For one, yes, it is orthogonal to security.

嗯。

Mhm.

对吧。所以就像你可以非常缓慢地制造核武器，但这并不会因为它是慢还是快而让任何人感觉更好。所以这是第一点。第二点，这感觉像是对暂停派的一种投降，而没有真正解决它。所以，你在说，好吧，我们不会暂停。我们会调整节奏让他们高兴，但也会以某种方式让监管者高兴。而我认为这会让双方都不高兴。

Right. So like you can very slowly build a nuclear weapon and that doesn't make anybody feel better that it's slow versus fast. So that's one. The second one, it just feels like a capitulation to the pause folks without actually, you know, addressing it. So, you're saying, well, we're not going to pause. We're going to pace to make them happy, but we'll also somehow make the regulators happy. And I think it makes them both unhappy.

因为暂停派会说，好吧，那不是暂停。那只是调整节奏。而监管者则说，你还在做那件事，

Cuz the pause people are like, well, that's not a pause. That's just pacing. And then the regulators are, you're still doing the thing,

对吧？

Right?

所以，我只是觉得他们试图——我对正在发生的事情的感觉是，实验室实际上是在试图做正确的事情。我为此鼓掌。我认为这是一个务实的提议，我为此鼓掌。

And so, I just feel like they're trying to my my sense of what's happening is the labs are actually trying to do the right thing. And I applaud them for that. I think this is a pragmatic proposal and I applaud them for that.

我认为他们的对外沟通是错的，因为他们试图在内部边缘派系——那些想暂停的末日论者——和另一边的监管者之间折中。问题是他们让两边都不高兴。他们必须做的是直接出来回应生存风险问题。他们需要说：‘不，我们不认为我们要做的这些东西会导致灭绝。’然后我觉得这就变得非常合理了。

I think the messaging is wrong because they're trying to split the difference between an internal fringe faction—the doomers who want a pause—and the regulators on the other side. The problem is they're making both of them unhappy. What they have to do is come out and address the x-risk question directly. They need to say, 'No, we don't think this stuff we're going to do is going to cause extinction.' Then I think this becomes very sensible.

那你会怎么做，就稍微站在另一边想一下？你会怎么做？你会为那种可能的交集留出空间吗——就是实验室研究员，既超级害怕，又在推进 AI 的发展，因为他们相信这件事太重要了，必须做对，所以想继续做下去？显然现在的说法很有问题，但那种人确实存在，而且是我们行业里真实存在的一类人——就像‘我们必须站在 AI 最前沿。我也很害怕它，所以我才在做这个。’

And what would you do, just to play the other side for one second? What do you do? Do you make room for the one possible Venn diagram, which is the lab researcher who is simultaneously super scared but also works on advancing the state of AI because they believe it's so important to get right that they want to pursue that? Obviously the language is very problematic right now, but that person does exist, and that is a real kind of person in our industry—like, 'We have to be at the forefront of AI. I'm also very scared of it, and that's why I'm working on this.'

那我直接回应这一点。我以前在劳伦斯利弗莫尔国家实验室工作，参与一个武器项目，核武器。我知道研究那种有权限限制的东西是什么感觉。

So let me address this directly. I used to work for Lawrence Livermore National Labs on a weapons program, nuclear weapons. I know what it's like to work on things that have access.

你是第一批‘节奏控制者’。

You were the first pacer.

我们是第一批‘节奏控制者’的一部分。所以如果实验室内部的一个群体——最有知识的人——相信这东西有生存风险……

We were part of the first pacers. So if a constituency within the labs—the most knowledgeable people—believe the stuff has existential risk...

是的。

Yes.

答案就是把它国有化，并真正实施我们知道有效的控制。现在，如果他们其实并不相信这一点——在我进行的私下对话中，最明智的人并不相信；相信的只是一小部分——那这就是一个人力资源问题。

The answer is to nationalize it and actually put controls that we know work. Right now, if they don't actually believe that—and in the private conversations I have, the most sensible people don't; it's a small fraction that do—this is an HR problem.

对吧？所以对我来说，人力资源问题就是公司问题。就像如果他们担心招不到人……

Right? So to me an HR problem is a company problem. Like if they are worried that they can't recruit people...

对。

Right.

你知道，或者留不住人，在我看来很多这类担忧其实就是这个。这几乎更像是一种研究员的通行货币。如果是这样的话……

You know, or they can't retain people, which it seems to me a lot of this is just that concern. It's almost more of this kind of researcher currency. If that's the case...

我认为，因为这种原因就对一项非常有前景的技术实施国家级封锁，是错误的。听着，我觉得那篇帖子——我觉得那篇帖子其实非常合理。我认为确实存在真实的安全担忧。我们经历过很多算力时代，都有真实的安全担忧。我不认为你能把关于生存风险的讨论和提出的那个提案调和起来。你就是没法调和这两件事，这一直是我最主要的……

I think that is the wrong reason to cause a national-level lockdown on a very promising technology. Listen, I think the post again—I think the post is actually very sensible. I think there are real concerns around security. We've had many compute epochs that have real security concerns. I don't think you can reconcile discussions on x-risk with the proposal that was put out. You just can't reconcile those two things, and that has always been my primary...

对。对。好。放开。好。

Right. Right. Okay. Unleash. Okay.

按住不放。

Holding it back.

所以，好。第一点是，你不能——他们发布过时间表，说所有这些事、不管什么坏事，什么时候会发生吗？他们没有。所以你没法控制节奏，因为根本没人知道它原本 supposed 什么时候完成。而且这看起来也很不真诚。这完全就像媒体报道苹果最新 iPhone 延期一样。

So, okay. The first thing is you can't have—is there a schedule that they've published that says when all this stuff, whatever bad stuff, is going to happen? They haven't. So you can't pace it because nobody knows when it was supposed to finish in the first place. It also just seems disingenuous. It's completely—this is like when the press reports on Apple's latest iPhone is late.

那款没人知道存在、他们也没告诉过任何人的手机。

The phone from what nobody knows exists that they haven't told anybody about.

是啊。我的苹果汽车就非常晚。

Yeah. My Apple car was very late.

是啊。我就不明白。要让某件事变慢，你首先得知道它原本的速度。所以这全是彻头彻尾的胡说八道，你逃不掉这一点。然后……

Yeah. Like I don't understand. Like in order for something to be slower, you need to know the rate at which it was moving in the first place. So it's all just utter nonsense and you can't escape that. And then...

顺便说一句，这是另一个问题——又是我对公关的一点小挑剔——就是反正没人相信，对吧？所以如果问题在于节奏控制，对吧？

By the way, this is another problem—again, from my little quibble on PR—is nobody believes it anyway, right? And so if that's the thing, the pacing, right?

哪部分要控制节奏？

Which part going to pace?

就是他们要控制节奏这件事。他们一直在拼命狂奔。他们筹的钱比以往任何时候都多。他们增长得比以往任何时候都快。没有任何迹象表明他们在控制节奏。所以如果这就是你要拿来做依据的……

That they're going to pace. They've been at a dead run. They've raised more money than ever before. They've grown faster than ever before. There's no indication that they're pacing. So if this is what you're going to hang your...

我明白。我明白。是啊。嗯，内部问题显然是，每个人内部拥有的模型远远超过其他人能接触到的。所以其实只是外部发布的节奏。但再说回你的观点，相对于什么来控制节奏？

I see. I see. Yeah. Well, the internal issue obviously is that the models that everybody has internally far exceed what anybody else has access to. So it's really just the pacing of external releases. But again, back to your point, pacing versus what?

对吧？就像我们不知道那个吓到所有人的模型是不是也写不好法律简报。它可能实际上会把那些东西全搞砸，因为它太——谁知道呢，对吧？所以，有这一点。

Right? Like, we don't know if the one that scares everybody also doesn't, you know, work for a legal brief. Like, it might actually screw all that stuff up because it's so—who knows, right? So, there's that.

而那只是——声称你在控制节奏就是不真诚。第二，他们为什么必须宣布这一切，并要求政府告诉他们控制节奏？就像，这部分开始让人觉得，嗯，这真的很诡异。如果你最害怕你的产品会导致一切走向糟糕，那就停下来。

And that's just—it's just disingenuous to claim that you're pacing. Second, why do they have to announce all of this and ask the government to tell them to pace? Like, that's the part that starts to go, well, this is really spooky. If you are the most afraid of how everything is going to go because of your product, just stop.

别做了。

Don't do it.

就像，我大学时在一家导弹工厂工作，我们有核导弹，我在车间里走过。

Like, I worked at a missile factory in college and we had nuclear missiles and I walked the floor.

你们这些搞导弹的怎么回事？我不——我只是搞软件的。

What's with you guys in missiles? I don't—I'm just here with software.

大学时你要么是抗议者，不让它们进校园，要么是建造者。所以那是你的选择。我们有天主教修女出现，把血泼在我们的导弹上，你知道，而我正忙着用推车推着电脑，说这是安全电脑，我害怕得要死。我完全不知道发生了什么。我说这是核导弹，然后你发现正是它阻止了冷战。真的——那是一枚潘兴导弹，就是它起了作用。但你说了一件我觉得超级有趣的事，就是‘节奏控制’是介于进行和不进行之间的一个模糊的非词。是的。问题在于——你说得完全对——没有人会对中间路线满意。

You were one of two people in college when we were—which was either you were protesting to keep them off campus or building them. So that was your choice. And we had nuns from the Catholic Church show up and pour blood all over our missiles, you know, and I'm busy just wheeling PCs around on carts saying here's the secure PC and I'm like scared to die. I have no idea what's going on. And I'm like it's a nuclear missile and then you find out that that's what stopped the Cold War. Like that literally—it was a Pershing missile and that was what did it. And so but you said something I think is super interesting, which is pacing is this sort of fuzzy nonword between going and not going. Yeah. And the problem is—you're exactly right—like there's no one is going to be happy with the middle road.

所以其中一件事——我觉得这几乎像看体育比赛一样有趣——他们认为所有这些对政府说做这个、别做这个的人，会得到他们想要的。这完全是对政府运作方式的 100% 误解。当你和政府谈时，他们其实知道这些事怎么运作。他们知道他们只是在听你说，也在听所有人说。没有人会得到他们想要的，因为他们知道在我们的体制里，要做成任何事，都是妥协。所以所有输入都能进入政府，但输出……

And so one of the things that—I think it's almost fun for me to watch as a sport—they think that all these people saying to the government do this, don't do this, that they think they're going to get what they want. And it's a complete 100% misunderstanding of how government works, which is when you talk to the government, they actually know how these things work. And they know they're just listening to you and they're listening to everyone. And no one is going to get what they want because they know in order to do anything in our system, it's a compromise. And so everything that all the inputs can make it into the government, the output...

永远不会让所有人满意，对吧？100% 的情况下，要么不够远，要么远远过头。

Never makes everyone happy, right? 100% of the time it either doesn't go far enough or it goes way, way too far.

是的。

Yeah.

所以你不能抱着这种观点：你要和政府谈，靠谈话得到你想要的解决方案。所以底线是，如果他们要求控制节奏，他们得到的速度会是错的。

And so you can't take the view that you're going to talk to the government and talk your way into the solution you want. So the bottom line is if they're asking for pacing, they're going to get the wrong velocity.

嗯，我最喜欢的一件事是，那是什么来着？David Sax 好像——我觉得是 David Sax——但政府里有人说：“你们是在要求我们监管你们。”不。Facebook。答案是“不”。

Well, my favorite thing is, what was it? David Sax was like—I think it was David Sax—but someone from the government said, "You're asking us to regulate you." No. Facebook. The answer was no.

嗯，但问题是——我觉得可能只是特朗普——但他们现在知道、你凭经验也知道的是，一旦监管的车轮开始转动，你就没法让它慢下来。

Well, but the thing that—probably just Trump, I think, but—the thing that they know now that you just know from experience is once the wheels start on regulating, you can't slow that one down.

而现在，它已经成为整个政府层级中每个司法管辖区、每个政党的选举议题。所以现在有了这一整套监管手段。

And now it's become an election issue for every party in every jurisdiction up and down the whole government stack. So there is now this whole basket of regulatory approaches.

嗯，下一次选举 100% 会是对 AI 的公投。所以 2028 年必须成为 AI 选举，你基本上可以围绕——问题是，谁会以支持 AI 的立场竞选并不明显，因为要讲好那个故事太模糊了。于是基本上就只是监管程度的不同，或者至少试图回避这个话题。

Well, the next election will 100% be a referendum on AI. So it has to happen that 2028 is like the AI election, and you could basically run on—the problem is it's not obvious who would run on the pro-AI story because it's going to be too nebulous to tell that story. So then it's just basically varying degrees of how much do you regulate it, or at least try to avoid the topic.

是啊。但糟糕的是，我们作为一个国家，支持 AI 的论点听起来太——它就是需要太多话，你知道，太微妙了。它是防御性的。

Yeah. But it is too bad that we as a country are in a spot where the pro-AI case just sounds too—it's just like it takes too many words, you know, it's way too nuanced. It's defensive.

是啊。它是防御性的。

Yeah. It's defensive.

它是防御性的，而且我们完全不掌握话语权，对吧？所以整个辩论就是“暂停”、“蜂群”、“失控”。每个词都是那些不想做 AI 的人选的。

It's defensive, and we own none of the vocabulary, right? So the whole debate is pause. It's swarms. It's rogue. Every word has been chosen by the people who don't want to do AI.

是啊。所以这意味着你首先要做的就是发明新词，并说他们的词是错的，而这需要太多话，以至于——

Yeah. And so it means the first thing you have to do is invent new words and say that their words are wrong, which takes so many words that—

是啊。所以我们必须像“工会工作”、“癌症”那样，你知道，需要出现另一个词云。

Yeah. So we got to be like union jobs and cancer and, you know, there needs to be another word cloud that emerges.

我不明白的是，为什么实验室没有对存在性风险表明立场。除非那样，否则我认为这只会走向强硬监管。

What I don't understand is why the labs have not taken a position on x-risk. Like short of that, I don't think this goes in any direction other than heavy-handed regulation.

是啊。嗯，如果政府说“有 10% 的物种灭绝概率，顶级公司的 CEO 说他同意”，那政府就是失职。政府怎么能不采取行动？他有过监管吗？

Yeah. Well, it would just be negligent of the government to be like, there's a 10% chance of species extinction. The CEO of the top company says he agrees with it. Like, how can a government not do? Have you he had a regulation?

嗯，但是谁——但真正了解这个生态系统的人，我不知道除了某种说法之外，你能让任何人明确表态吗？

Well, but who—but what would anybody really knowing this ecosystem though, I don't know that you would be able to pin anybody down on that other than something?

不，我会说 Dario 在——上说过。

No, I would say Dario said it on—

不，但我是说，你不会让任何人给出一个更低的——嗯，事实上，他最糟糕的地方在于，他同意那些声称相信有 x% 灭绝概率的人，但他特意强调：“我不会给出具体百分比。”

No, but I'm saying you're not going to pin anybody down on a lower—well, in fact, he does the worst thing about it, which is he agrees with people who claim that they believe that there's an x percentage of extinction happening, but he specifically goes out of his way to say, "I'm not going to put a percentage on it."

我觉得这最奇怪。

Which I just think is the weirdest.

不，但那——公平地说，公平地说——

No, but that—to be fair, to be fair—

不。

No.

好吧。不，没人能做到公平。公平地说，那并不 100% 是虚伪之类的。他可能并不具体同意是 10%。或者他同意，但如果说出来就太吓人了——

Okay. No, nobody could be fair. To be fair, that's not 100% like disingenuous or whatever. Like he probably doesn't specifically agree that it's 10%. Or maybe he does and it's just too scary if he were to say—

我们就用二分查找吧。我是说，是更多还是更少？

Let's just play binary search. I mean, is it more or is it less?

但整个事情是我们刚刚创造的一个虚构概念。但你无法量化这些。

But the whole thing is a made-up concept that we just created. But you can't quantify any of this.

嗯，但这正是 Martin 的观点。

Well, but that's sort of Martin's point.

嗯，但你刚才有一个官方立场。所以唯一的官方立场会是——我认为你唯一可能得到的官方立场会是这件事的公关版本。那将是唯一在智识上诚实的说法：AI 存在真实风险。也有极其积极的好处。我们正在努力降低风险，使其尽可能减少。

Well, but you just had an official position though. So the only official position would be—I think the only official position you could possibly get would be the PR version of this. That would be the only thing that would be intellectually honest: there are real risks with AI. There are incredibly positive benefits as well. We are working to mitigate the risks so they are as reduced as humanly possible.

我不认为那是真的。听着，那么你认为——

I don't think that's true. Listen, so what do you think that—

我们经历了多个技术时代。我们经历了算力。我们经历了互联网。我们经历了万维网。我们经历了社交网络。我们讨论过风险，但没有谈论存在性风险。对吧？所以例如，你可以说的一件事是，我们认为物种灭绝的边际风险与现在没有不同。

We've been through multiple epochs of technology. We've been through compute. We've been through the internet. We've been through the web. We've been through social networking. We had a discussion about the risks without talking about x-risk. Right? So for example, one thing you can say is we do not think the marginal risk for species extinction is different than it is.

但如果他们确实认为更高呢？

But what if they do think it is higher?

嗯，那么我们应该——好吧。好吧。是的。我觉得答案是，他们确实认为这不仅仅是互联网。

Well, then we should—okay. Okay. Yeah. Like I think the answer is they do think it's more than just the internet.

两个选项之一。你相信我们会灭绝，然后我们关停一切。就像关停，或者——

One of two options. You believe that we're going to go extinct and we shut it all down. Like shut it down, or—

他们相信这只是我们得到的一个机会——

They believe that this is just a chance that we got—

说话。Dario 认为更少，但这些冷战战士。没人像五角大楼那样，你知道，他们做了很多关于核战争概率的模拟。关于这一切有部好电影《战争游戏》。但问题是它是非零的。

Speaking. Dario thinks less but these cold warriors. No one like the Pentagon, you know, they ran a lot of simulations on the chances for nuclear war. Great movie War Games about the whole thing and all of that. But the thing was it was nonzero.

是的。所以一旦你说它是——

Yeah. And so once you said it was—

那么他们能说非零吗？那被允许吗？

Can they say non-zero then? Is that allowed?

我认为一旦你认为它是非零的——

I think as soon as you think it's non-zero—

问题是，如果你说非零，那可能是，你知道，那家伙认为 93%。但让我们等等，这样我们都能进行清晰的对话。我们在这里谈论边际风险。我们不是在谈论绝对风险，对吧？好的。

The problem is if you say non-zero that could be like, you know, the guy thinks 93%. But let's wait so just so we're all having a clean conversation. Let's talk about marginal risk here. We're not talking about absolute risk, right? Okay.

只是我认为，一旦你说它是非零的，对于灾难性风险，唯一的答案就是你必须将其国有化。

It's just that I think if once you say it's non-zero, the only answer of like catastrophic, the only answer is you have to nationalize it.

是的。所以他试图——持有那种观点的实验室总体上试图威胁的是，他们希望它是非零的，作为做某些事情的许可，而不必承担成为国家——的负担。

Yes. And so what he's trying to—what the labs in general that have that view are trying to threat is they want it to be nonzero as a license to do a certain set of things without the burden of just becoming a national—

嗯，你有没有——我实际上不知道所有的总统，希望你了解，但肯定有一些非零的事情，比如存在性风险，那些风险并不特别国有化，但围绕它们的监管环境如此沉重，以至于由于 KYC 要求，它们几乎可以说是国有化了——比如我确定要开发炭疽，你必须去特定类型的实验室。

Well, do you have—I actually don't know all of the presidents you hopefully do, but there must be some things that are nonzero, let's say x-risk, x-risks that are not particularly nationalized, but the regulatory environment around them is so heavy that it might as well be nationalized because of the KYC requirements on like—like I'm sure to develop anthrax you have to go to a particular kind of lab.

嗯，BSL4 实验室是，但它们是国有化的。它们是国家的。好的。但——

Well, BSL4 labs are but they're nationalized. They are national. Okay. But—

伦理往往是国有化的。是的。

Ethics tends to be nationalized. Yes.

就像——

Like—

正常的人类安全则不那么国有化。对。就像行业监督，随着时间的推移变成联邦法规。

Normal human safety not so much. Right. Like industry oversight that over the time becomes federal regulation.

对。而这个进程，我认为对这次讨论非常重要，自二战结束以来，大多数对基础设施至关重要的行业——电力、银行和医疗保健。趋势基本上是国有化。是的。就像你——就像你举的 KYC 和其他东西的例子。银行实际上已经国有化了，还有金融危机。

Right. And the progression, which I think is just super important to this discussion, has been since the post-World War II era most industries that are critical to the infrastructure—power and banking and healthcare. The trend has been to basically be nationalized. Yeah. By just like what you—like your examples of KYC and all the other stuff. The banks are for all practical purposes nationalized and the financial crisis.

好吧。但你说的是实际上，对吧？它们实际上并没有被国有化，对吧？所以也许实验室的意图是看起来像摩根大通或威瑞森。就像我们是关键基础设施。我们受到严格监管。这对开源或至少前沿开源不利，但这是这个行业的一个可能结果。对创新有好处才是问题。

Okay. But you said all practical purposes, right? They're literally not nationalized, right? So maybe this might be the intent of the labs is to look like JP Morgan or look like Verizon. And it's just like we're critical infrastructure. We get heavily regulated. It's not good for open source or at least frontier open source but it is like a plausible outcome for this industry. Good for innovation is the problem.

我觉得即使那样也没问题，只是别把物种灭绝当作你的……字面上就像实验室里那些东西之间的区别，让我说一句，我实际上认为实验室正在朝着正确的方向前进。我实际上认为那份声明真的很好。你知道，在我与高管和领导人的讨论中，他们明白他们有这样的紧张关系，他们会调和这一点。所以，我实际上相当乐观，实验室既在做正确的事情，也在努力做正确的事情。我只是觉得它发展得太快，只是在试图弄清楚这个机制如何运作。而且我认为西诺斯基真的一针见血。

I think even that's fine just don't use species extinction as your literally is like the difference between the things that are like stuck in the lab and let me just say something I actually think the labs are moving in the right direction. I actually think the actual statement was really good. You know, in my discussions with, you know, executives and leaders like they understand that they have like this tension and they're going to reconcile that. So, I actually am quite optimistic that the labs are both doing the right things and trying to do the right thing. I just think that grew so fa grew so fast and just trying to figure out how this machinery works. And I I think Sinoski really hit the nail on the head.

政治进程是它自己的事情。我不知道这是天真还是傲慢，但我只是不认为他们知道如何驾驭它。

The political process is its own thing. And I don't know if it's naivety or hubris, but I just don't think that they kind of know how to navigate it.

嗯，我认为科技行业在 100 多年来，在每一次技术浪潮中都不断重新学习这个教训：我们不了解监管环境，也无法驾驭它。即使是像 AT&T 和 IBM 这样从一开始基本上就是政府垄断诞生的公司也从未弄明白。两家都被起诉反垄断，都因此发生了实质性和结构性的改变，他们在 1960 年代有数百名律师为他们导航，你知道微软出现了，我们就像什么？是的。就像我们不知道发生了什么，你知道比尔·盖茨和比尔·克林顿打高尔夫球，然后我们就被他的政府反垄断诉讼打击，比尔就像我在打高尔夫球，这是照片，那没有帮助，难道我不应该去打高尔夫球吗？这是整个事情的简化版。但我认为，我总是用这个例子，就是好莱坞在红色恐慌和审查期间，当他们担心政府审查电影中的性内容、成人主题时，他们聚在一起，当然如果他们试图在法庭上获胜，他们永远不可能赢，但他们威胁要这样做，他们聚在一起，成立了电影协会。

Well, I think the tech industry has literally over 100 years consistently relearned the lesson at each technology wave that we don't understand the regulatory climate and we can't navigate it. And even the companies like AT&T and IBM that were born out of basically being government monopolies from the start never figured out. Both got sued for antitrust, both got substantially and structurally changed as a result and they had hundreds of lawyers in the 1960s navigating them and that you know Microsoft came along like we were just like what? Yeah. Like we had no idea what was happening to us and there's you know there's Bill Gates playing golf with Bill Clinton and then we get slapped with an antitrust lawsuit from his administration and Bill was like I was playing golf here's the picture and that doesn't help and like isn't that what I was supposed to do was go and play golf. That's a shortened version of the whole thing. But but I think and I think it's just it's I always use this example which is the Hollywood got together during the Red Scare and all the and censorship when they were worried about the government censoring movies for sexual content for adult themes and they all got together and and of course they were never going to be able to win in court if they tried to but they were threatening to do it and they got together and they formed the Motion Picture Association.

是的，电影分级等等，他们自我监管。

Yeah. and movie ratings and all that and they police themselves.

那么你们怎么做？你们喜欢那样吗？你们喜欢 FINRA 的提议吗？

So how do you do you guys like that? Do you like the FINRA proposal?

不，因为 FINRA 是，嗯，那是最接近 NPA 的，但更多。

No, because FINRA is well that's that's as close to NPA as you can get with more.

不，不是，因为 FINRA 将变成立法，随之而来的是直接监督。

No, it's not because FINRA is going to FINRA becomes legislation which comes with direct oversight.

是的，我认为 MPAA 可能对社会运作方式的影响没那么大，第一修正案。

Yeah, I think MPAA might not have as much consequence in like how society functions the first amendment.

是的。嗯，我赢了吗？不，我，首先，太棒了。但我仍然不知道你是否想要它。就像我同意言论非常重要，但就像你知道的，我只是觉得

Yeah. Um did I win that? No, I I I I first of all, fantastic. But but I I still don't know if you want it. Like like I agree speech is really important, but like but like you know like the uh I just think

没有社会风险。

there was no societal risk.

我只是觉得我们电影里的内容会像我们会在一个不同的连续体上生存

I just think what's in our movies will be like we'll survive like on like a different continuum of

所有这些，每个历史总是相对的，当时成为共产主义者是一件非常糟糕的事情，30% 的好莱坞人被解雇了，你知道，就是所有这些事情。是的。所以我总是冒险以那种方式提出一些事情，因为它听起来很愚蠢。就像没有人考虑电影分级，那是因为实际上他们被裁定基本上你不能在宪法上强制要求它们。所以他们无法在有线电视上监管它们，所以我们得以在 HBO 和所有其他东西中成长。

all these every history is always relative and and at the time being a communist was a really bad thing and 30% of Hollywood got fired for for you know it was all this stuff. Yeah. So I it's always a risk to bring up something in that kind of way because it sounds so dumb. Like nobody thinks about movie ratings and that's because like actually they were ruled basically you can't constitutionally mandate them. So they couldn't regulate them on cable TV and so we got to grow up with HBO and all this other stuff.

但 FINRA 的问题在于，它本质上是一个将风险国有化的完美例子。

But the problem with FINRA is that is a perfect example of essentially nationalizing risk.

是的。

Yes.

因为即使所有银行都向它付钱，这就是它的运作方式等等，但这一切都是强制性的。

Because even though the banks all pay money into it and that's how it's run and stuff, it's all mandated.

好吧。等等，抱歉。你认为我们最终会得到一个比 FINRA 更好的情况吗？FINRA 似乎是最好的情况，但它是最好情况，但甚至不再是了。就像我确实认为这项技术在极限上再次如此强大，相对于它，我的意思是它能提供的东西，最终国会不可能不关心。就像这不可能。问题。如果它最终，在什么点上，如果它最终在你的医疗保健过程中做出每一个推荐，并且它作为开放权重模型在你的医疗设备内部，并且它在每个高频交易系统中，并且它在飞机上，政府不可能不说我们需要一些东西，而 FINRA 实际上可能是这种情况的最佳情况。对吧。

Okay. Wait, sorry. Do you think we're going to end up with a situation that is better than FINRA? FINRA appears to be the best case scenario, but it is the best case scenario, but not even anymore. Like like I do think this this technology is in in the limit again so powerful uh relative to what it I mean for what it can deliver that it would be impossible for eventually Congress not caring about that. Like it's just like not possible. question. If it's eventually at what point if it's eventually making every recommendation in your healthcare process and and it's inside of your medical device as an open weights model and it's and it's on and every trading system for high frequency trading there's just and it's on an airplane like there's no chance that the government doesn't say we need something where FINRA actually is like the probably the best case scenario of what that looks like. Right.

是的。所以现在这是绝对最关键的一点，因为如果你，现在参议院的所有人都在参议院时，当通信法案通过时，责任没有传递给 ISP 和社交网络，他们坐在那里，当时的辩论是他们真的会对我们说，互联网如此之大，我们怎么没有参与其中，这就是为什么阿尔·戈尔因为说创建了它而受到一堆辱骂，对吧，因为他实际上试图走在前面说，不，政府在其中发挥了作用，但这一切都适得其反，因为它让他看起来像个疯子。但但绝对是这样，他们觉得他们错过了在互联网上占据主导地位的机会，而阿尔·戈尔站在创新的积极一边。

Yeah. And so now that's that's absolutely the most crucial point because if you all the people in the Senate now were in the Senate when when the communications act was passed and the li and liability was not passed through to ISPs and to social networks and they sat around the debate at the time was they would literally sat say to us the internet is so big how did we not have anything to do with it and that's why Al Gore gets a bunch of abuse for saying he created it right because he he was actually trying to get out in front of that and say no the government was instrumental in and it all backfired because it made it look like he was a crazy person. But but that it is absolutely the case that they felt like they did they missed their chance to be on top of the internet with Al Gore being on the positive side of innovation.

那是他们的支持。那么谁是阿尔·戈尔呢？

That was their support. So who is the al Goro?

没有人。有几个，嗯，贝塞特似乎是，你知道，国防部门的人有点想要它私有但不完全，这正是他们在互联网上的立场，对吧

There's no one. There's a couple well Besset seems to be that you know the defense people sort of want it private but not which is exactly where they were on the internet right

所以这非常有趣

and so it's very interesting

很多这只是像伊丽莎白·沃伦，沃伦参议员的推文都在说我们错过了社交网络，就像

that a lot of this is just this like Elizabeth Warren Senator Warren's tweets were all like we missed this for social network and it's like

有很多被压抑的反对科技的能量，最终可能全部被吸入 AI 中。

there's a lot of pent-up energy against tech that that could end up just all siphoning into into AI right now.

正是如此。这就是正在发生的事情。

That is exactly what it is. That's what's happening.

是的。我认为还有另一个问题，那就是我们都在试图预测什么是坏的，这不是我们通常做事的方式。

Yeah. I think there's another problem which is we're all trying to predict what's bad which is not how we've normally done things.

到这个时候，互联网上我们已经造成了数百亿美元的经济损失。我们有过蠕虫病毒，摧毁了 10% 的基础设施。

Like by this time on the internet we'd taken out like tens of billions of dollars of like economic well we've caused tens of billions of dollars of economic damage. We've we'd had words that took out 10% of the infrastructure.

是的。

Yeah.

互联网基础设施，当时运行着关键基础设施。医院都瘫痪了。

The the internet infrastructure which was running critical infrastructure. We had hospitals go down.

我们真应该在 97 年左右就封锁互联网。

We really should have blocked the internet in like 97.

我们怎么 是的。是的。好吧。所以，我们有过，我的意思是，我记得有一次，你就像，我记得当你买 Windows 95 的 CD，等它安装完的时候

How do we Yeah. Yeah. Okay. So, we we had I mean I I remember a time you like I remember when you would you would you would buy your CD of Windows 95 and by the time it was done installing

你可能就已经中了一个蠕虫病毒。

you would have a a worm potentially.

干得好，斯蒂芬。

Way to go Stephen.

不，不，这就是现实，直到 2001 年，个人电脑的现实是，你不可能安装一台连接到网络的电脑而不被感染。但病毒无处不在

No, no, this was it was this was the reality the reality of the PC until 2001 was you could not install a PC connected to the network without getting infected. But viruses were everywhere

因为，而且有两轮，两轮，两轮国会听证会。你所有的

for the and and there were there were two two level two two rounds of congressional hearings. You had all of the

那实际上没关系。好吧。我承认这是一个非常有趣的观点。呃，这种 94 年的时代精神，我们可能就不会有互联网了。

that is actually okay. Okay. I'll grant you this is a very interesting point. Uh this type of zeitgeist in 94 would have we would probably not have the internet.

听着，听着，我的意思是，汽车行业、航空业，你总是有这些类似磨牙的时期，你了解危险，你了解技术和互联网。我的意思是，我们处于这个的最底层。我的意思是

Listen listen I mean the automotive the airline industry you always have these periods of kind of like teeth cutting where like you learn about the dangers and you learn about the technology and the internet. I mean we were on the ground floor of this. I mean

东西一直出故障。经济损害一直存在。就像有新的。有新的病毒。有新的蠕虫。到处都是。

things were down all the time. There's economic damage all the time. Like there was like novel. There was new viruses. There are new worms. They're all over the place.

Y2A。Y2K 会摧毁一切。

Y2A. Y2K was going to destroy.

但这里有个有趣的事情。

But here's here's the interesting thing about this.

但我只想说，当我们制定政策时，我们确实制定了很多政策，那是基于一堆非常具体的数据点，关于你想做什么，这样你就知道政策实际上符合事实模式。在这种情况下，我的意思是，即使你在说话时，你说，好吧，如果，你知道，等等等等。预测政策安全风险非常困难。很好。而围绕这一点实际上开始演变的话语的好处是。你实际上听到实验室的人说新的网络安全风险。这是一个工程问题。我们在谈论这个。所以，当这变得具体围绕真实识别的风险时，我想我们都能达成一致。但到目前为止，这不是讨论。这是一个非常

But what I just I just want to say like like so when we created policy and we did create a lot of policy, it was kind of like with a bunch of very specific data points on what you're trying to do and so you know that the policy actually fits the the fact pattern. And in this case, I mean, even when you were talking, you're like, well, what if you know, yada yada yada. It's very hard to predictive policy security risk. Great. And what is nice about the discourse that is actually starting to evolve around that. Like you actually hear people from the lab saying novel cyber security risk. This is an engineering problem. We're talking about that. So the more this becomes concrete around real identified risks, I think we can all fall in line. But that's not been the discussion to date. That's a very

完美的观点，因为如果你看 1986 年，计算机欺诈和滥用法案签署了。它源于一个非常具体的场景，就是两组黑客闯入了 GTE Telem。而且

perfect perfect point because if you look in in in 1986 the computer crime and fraud act got signed. It was out of a very very specific scenario which was a two groups of hackers broke into GTE telem. And

当然，好吧，那不是你。它

sure, well, it wasn't you. It

是另一个人。

was the other guy.

哦，他后来在思科工作。而且嗯

Oh, he wor he worked he worked at Cisco later. And um and

问题是那是 1983 年，没有犯罪

and the problem was that was 1983 and there was no crime

而且花了两年半

and it it took two and a half years

法案才通过。这使得它非常具体。那就是说，你不能访问未经授权的计算机系统的法律。所以我想我们都同意，我们可能已经有 90% 的法律在应用层，比如你不能黑系统等等。你认为从责任角度，模型层应该有什么吗？当然，那么

for the bill to make it through. That made it very very specific. And that's the law that says you can't you can't access unauthorized computer systems. So I think we'd all agree that that you we probably you know we probably have like 90% of laws already in the applied layer like you can't hack systems etc. Do you think there's anything that should be from a liability standpoint in the model layer? Which of course then

你怎么让技术合法？

how are you making a tech legal?

不，法律，一切，我对 Hugging Face、OpenAI 的解读是，它们都是绝对公然的计算机犯罪行为，除了后来他们划出了修正案，如果你是白帽，就不再违法。那是因为人们不断犯错

No, the the legal everything my read of of hugging face open AI for they're all absolutely blatant computer crime acts except for the fact that there's they carved out an amendments later that if you're a white hat, it's not illegal anymore. And that was because people kept making mistakes

而且他们不想到处逮捕那些实际上试图让系统更好的人，因为他们搞砸了事情。所以司法部写了一份备忘录说，‘我们不会为此起诉。我们也不会起诉如果你只是违反使用条款，对吧，’

and and they didn't want to go around arresting everybody who was actually trying to make the system better because they messed something up. And so the Justice Department wrote a memo that said, "We're not going to prosecute for for this. And we're also not going to prosecute if you just like violate the terms of use, right,

系统，而不是实际试图入侵它。”

of a system versus actually try to breach it."

所以我认为法律对此场景足够。而且这一切都写好了，这个 GTE Telem 被 NASA 和 Liverour 以及所有实验室使用。所以这就是为什么它引起了 DC 的注意，因为被闯入的是联邦系统。我们现在的问题是，这是实验室和安全社区之间的裂痕，他们不断查看所有事后分析，得出两个结论。草率

And and so I think the law is ample for this for the scenario. And and it was all written this GT Telemet was used by NASA and Liverour and all the labs. And so that's why it was it caught the attention of DC because it was federal systems that were being broken into. And the problem we have now is is this is this rift between the labs and the security community that keeps looking at all their postmortems and coming to two conclusions. Sloppy

而且你告诉我们的发生的事情不完整。

and and you're not complete in what you're telling us happened.

所以当然，CBE 流程在 1980 年代出现。来自 CMU 的计算机病毒和漏洞报告，多年来他们致力于非常结构化的报告，有义务和如何做，但出于某种原因，他们没有使用任何这些来做这个报告

And so of course the CBE process came about in the 1980s. the the computer virus and vulnerability reporting stuff out of CMU and and for years they worked on very structured reporting with obligations and how to and there for some reason they're not using any of that to do this reporting

所以有这些非常基本的东西，我看着说，好吧，直到他们这样做，他们真的应该停止说话，就像他们不应该对入侵做事后分析

and so there is this very basic stuff that I look at and say well until they're doing that they really should stop talking like they shouldn't do a postmortem on a breach

看起来像实习生写的，这不是事后分析，这是选择性记忆，看起来像

that looks like an intern wrote it and it's not a postmortem it's this selective memory it looks like

正是当你雇佣外部律师调查一些随机事情时做的那种事后分析，你只给他们某些东西

exactly the kind of postmortem you do when you hire outside lawyers to investigate some random thing and you only give them certain stuff

因为你不需要给你雇佣的律师所有关于发生的事情的信息

because you don't have to give the lawyers you hire all the information about what happened

比如所有的 Slack 消息在哪里，实际发生的细节在哪里，我能插一句烦人的题外话吗，所以嗯，这是，不，不，这很好，这非常符合这个，就是嗯，我在安全领域，实际的网络安全社区很长时间了，你知道，它总是这些事情之一，他们就是不喜欢实际解决方案。所以即使你构建了一个你知道的安全系统，比如，好吧，如果有人你知道出现，你知道像

like where's all the Slack messages where's the actual details of of what happened can I can I just interject an annoying aside so um which is which is no no this is good which is very in line with this which is um I've been in the security like the actual cyber security community for a long time and it is you know it's always been one of these things where like they just don't like practical solutions. So even if you build like you know a secure system like well what if somebody you know shows up like you know like

你知道罗素·克劳能破解加密或类似的东西。

you know can Russell Crow can like break the encryption or something like that.

这是《碟中谍》。有些

It's Mission Impossible. Some

团队，总是有这些类似的东西。所以我最近最喜欢的事情是像 Gnome Brown 某个播客，我们都在播客上说过蠢话。我已经说过

crew there's always like these kind of like whatever. So my favorite thing that happened recently was like Gnome Brown some podcast we've all said stupid stuff in podcast. I've already said

哦，你不喜欢这个。

oh you didn't like this one.

不，它很棒。

NO IT WAS GREAT.

我觉得它很有趣。

I THOUGHT IT was fun.

不，它太棒了。不，我是，我在铺垫。是的。

No it was fantastic. No I'm set I'm setting it up. Yeah.

所以，所以 Brown 说，听着，呃，你不知道超级智能能做什么。

So, so no Brown was like listen uh you don't know what a super intelligence could do.

它或许可以利用 CPU 的热量把自己外泄到另一台电脑。这让我回想起——我现在很自在。我可以就这个话题进行无休止的无意义讨论，但有趣的是，基本上你有 X-risk（存在风险）的人说了一些他们认为合理的事情，然后你有安全领域的人进行这种无休止的讨论。所以我认为在某种程度上这些社区正在被连接起来，这就像——你知道，我实际上认为 Nam Brown 说的是一件非常合理的事情。我觉得你可以挑毛病，但我们在播客上都会说些奇怪的话。我实际上认为 X-risk 是真实的。我的意思是，我在高度机密的安全计算环境中工作过，那里的隐蔽信道令人难以置信。所以这些是非常真实的评论，但我们实际上进行了一场真正的讨论，这是我第一次看到真正硬核的系统人员进行了相当建设性的——我会说建设性的——

It could maybe use the heat of a CPU to exfiltrate itself to another computer. This brought me back to—now I'm very comfortable right now. I could have endless pointless discussions on this, but what is interesting is you basically have the X-risk people saying something they thought was plausible, and then you have the security people having this kind of endless discussion. So I think at some level these communities are being bridged, which is—you know, I actually think that was a very reasonable thing for Nam Brown to say. I think you can pick holes in it, but we all say weird stuff on podcasts. I actually think X-risk is real. I mean, I worked in secure computing environments that were highly classified, where the covert channels were unbelievable. So these are very real comments, but we actually have a real discourse, and it was the first time I saw really hardcore systems people having pretty constructive—I would say constructive—

计算比特率并且——

Calculating the bit rate and—

是的，很棒，但那是一场建设性的讨论。而且你有典型的 X-risk 人士参与。当然，那是推特，所以有很多骂战之类的，但实际上——我觉得这是第一次真正的讨论。所以我希望我们能看到更多这样的。我希望看到更多与网络相关的事情。我认为实验室应该多谈谈这个。我认为这会吸引社区参与，我认为一旦发生这种情况，我们实际上可以——

Yeah, it was great, but it was a constructive discussion. And you had the typical X-risk people engaging. Of course, it was Twitter, so there's a lot of name-calling and this and that, but it was actually—I felt like a real discussion for the first time. So I hope we see more of this. I hope we see more cyber-related things. I think the lab should talk more about it. I think it'll engage the community, and I think once that happens, we can actually—

是的，Greg 在这个话题上更多地发声了，这很好。但我认为结论是 Nam Brown 应该在播客上做更多的头脑风暴。

Yeah, Greg's been out there a lot more on this topic, which has been good. But I think the takeaway is Nam Brown should be doing more brainstorms on podcasts.

我认为它实际上只是让人们大开眼界，意识到安全领域存在许多大多数人并不理解的风险。所以这意味着有更多的东西你不能把它当作基线风险——你知道，比如你的认证层是如何工作的？你不能只是挥挥手说那应该消失，然后提出，哦，但是你知道外星人可以入侵。这就是当时发生的一部分情况，让我感到不舒服。但就像你是什么意思——有点像，但你知道也有这种风险，这就是我对热那件事的看法。但至少我们现在处于一个我们熟悉的领域——比如我可以谈论熵，我们可以进行具体的讨论，而不是，哦，它超级强大。至少我们把它归结为物理定律和系统定律。而且大多数人——我会说大多数人不知道那种风险是真实的。我的意思是，就像我在波斯导弹周围走动时，我实际上必须测试显卡和 PC，因为国防部的要求是，当你拔掉电源时，屏幕内存不能保持，但不是零时间。就像电源一断，内存图像就必须消失。如果有 3 秒的延迟——

I thought it actually just opened people's eyes to the fact that there are a lot of risks in security that most people don't understand. And so that means that there's more stuff that you can't make baseline risk—you know, like how does your authentication layer work? You can't just wave your hand and say that should go away and then bring up, oh but you know space aliens can invade. And that's a little bit of what was going on that I felt uncomfortable with. But like what do you mean—it was sort of like, but you know there's also this risk, and that's how I viewed that of the heat thing. But at least we're now in a domain we're comfortable—like I can talk about entropy, and we can actually have a concrete discussion that's not, oh well, it's super powerful. At least we've reduced it to the laws of physics and the laws of systems. And most people did—I would say most people had no idea that that kind of risk was real. I mean, like when I'm walking around the Persian missiles, I actually had to test the graphics cards and PCs because a DoD requirement is that the screen memory not be sustained when you pulled the power, but not for zero time. Like the minute that the power went off, the memory image had to go. And if there was like a 3-second delay—

是的。

Yeah.

哦，你有那个可以被读取。哦，看，我们有人来到我们的办公室，因为 Tempest 攻击，他们会实际测量显示器之间的距离，这 100% 是一种利用电磁辐射泄露信息的方式。我在 BIOS 的扩频中见过同样的事情。我从音频扬声器中见过——就像我们不得不移除扬声器，因为它是一个非常高的信道。

Oh, you had that could be read. Oh, see, we had people that came into our offices and would literally measure the distance of the monitors to each other because of Tempest attacks, which is 100% a way to use electromagnetic radiation to leak information. I've seen the same thing with spread spectrum from the BIOS. I've seen it from audio speakers—like we had to remove the speakers out because it's a very high channel.

我们的大楼只是把音乐扬声器对准窗户，只是为了产生干扰。

Our building had just music speakers aimed at the windows just to produce interference.

你需要去这些实验室之一负责安全。这就像我从未见过你比热，你知道，基于通信更兴奋。

You need to go run safety at one of these labs. This is like I've never seen you more excited than heat, you know, based communication.

我能告诉你我见过的最疯狂的隐蔽信道吗？所以它转——记得旧 CRT 吗？我知道这就像我的一个——

Can I tell you the craziest covert channel I've ever seen? So it turns—remember the old CRT? I know this is like one of my—

所以我很高兴有人比我老。不是真的，但表现得像。

So I'm glad that someone's older than me. Not really, but acting it.

所以记得旧 CRT 吗？所以事实证明，如果比如说是晚上，你在房间里使用 CRT 终端，房间里最亮的东西实际上是光栅束所在的像素。所以大多数人认为它是显示器的辉光，但实际上它是那个给定的像素。所以，有人发现，比如说你在酒店房间里，你在用电脑。如果你有东西可以采样窗户的颜色，你可以重建屏幕。

So remember the old CRT? So it turns out if like let's say it's night and you're using a CRT terminal in your room, the lightest thing in the room is actually the pixel that the raster beam is on. So most people think it's like the glow of the monitor, but it's actually that given pixel. So, somebody figured out that like let's say you're in a hotel room and you're on your computer. If you have something that can sample the color of the window, you can reconstruct the screen.

哦，那太疯狂了，对吧？你只需以光栅束移动的相同赫兹来做。然后另一个人发现，如果你能颠覆三个像素，你可以使用那些，你知道，因为它看起来就像坏像素。你可以用那些基本上发送消息。所以就像你可以 literally 坐在外面 whatever——就像你采样消息，你可以——这是一种相对高带宽的单向通信。所以就像 Nam Brown 说的，也许热不是方法，但这种复杂程度实际上是真实的。那是一种东西。

Oh, that's crazy, right? You just do it at the same hertz that the raster beam is moving. Then somebody else figured out if you can subvert three pixels, you can use those, you know, because it just looks like bad pixels. You can use those to basically send a message. So like you could literally sit out in whatever—like you sample the message and you could—it was a relatively high bandwidth one-way communication. And so like what Nam Brown was saying, like maybe heat is not the way to do it, but that level of sophistication is actually real. That's a thing.

我的——当我在导弹工厂时，就像我不得不锁起我的键盘——那真的是一个不礼貌的词，但那是我们称呼某物的方式——我不得不在晚上锁起我的键盘，因为他们不想让没有安全许可的保管员走过时注意到哪些键更脏或更干净。有一次我把它留在外面，有一张来自安全的便条，你知道，告诉我去安全部门报到并取回我的键盘。就像走廊里的警卫那天晚上直接从我的机器上拿走了键盘。基本上我没有安全许可。我只是那么敏感，你知道吗？我什么都不是。我是个实习生。

My—when I was at the missile factory, like I had to lock my keyboard up—that's really an impolite word but that's what we call something—I had to lock my keyboard up at night because they didn't want the custodians who didn't have clearance walking by and just noticing which keys were dirtier or cleaner. And I once left it out and there's like a note from security, you know, telling me to report to security and pick up my keyboard. Like the guard who walked the hallways just took the keyboard that night off my machine. And that's like basically I was not cleared. I was just that sensitive, you know? I was like nothing. I was an intern.

这次谈话的大问题是，现在这一切都在未来每个 AI 模型的训练数据中。所以——

The big problem with this conversation is now this is all in the training data of every AI model in the future. So—

但这也是威胁模型。嗯，但这些事情的威胁模型总是你有一个可信侧和一个不可信侧。NIST 有 500 页的手册——每个不可信侧你基本上假设一个可以做到和知道一切的预言机，然后问题是你能否从可信侧获取信息，顺便说一下，这就是 Nam 所说的，这实际上相当——

But that's also the threat model. Well, but that's the threat model for these things is always you've got a trusted side and an untrusted side. NIST has 500-page manuals—every untrusted side you assume basically an oracle that can do and know everything, and then the question is can you get information off the trusted side, which by the way is what Nam was saying, which again is actually quite—

我确实认为这引出了一个超级有趣的观点，我将要过渡到这一点，那就是我认为人们真正没有理解并且使用的语言非常令人困惑的是，AI 能做的就是在很短的时间内尝试所有这些事情。

Which I do think brings up a super interesting point which I'm going to bridge to, which is just that I think the thing that people really aren't wrapping their heads around and are using the language that's really confusing is just that what AI can do is it can try all of those things in a very short time.

是的。是的。

Yeah. Yep.

而且它不会累，不会无聊。但有趣的是，现在你必须审视每一层安全，每一个 API，每一个你在内部网络上运行的服务。就像，没人觉得他们内部的 GitHub 或内部的 Slack 或内部的财务报销工具会容易受到拒绝服务攻击，但智能体集群会。它看起来真的就像拒绝服务攻击。

And it doesn't get tired, it doesn't get bored. But the interesting thing about it is there's a whole layer of security that you now have to go look at every single API, every single service you're running internally on your network. Like, nobody thinks that their internal GitHub or their internal Slack or internal finance expense tool is vulnerable to a denial of service attack, but swarms do. It literally looks like a denial of service attack.

是的。

Yeah.

所以现在我们需要一个完整的内部层，来追踪更多的认证行为、API 调用。

And so now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done.

是的。

Yeah.

但这就像现在这将成为基本操作，所有那些老派的线下人员都在给我发邮件说，为什么我们要向所有人解释这个？这太基本了，因为内部没人做过。

And but that's just like now it's just going to be basic and all the offsite people that are old are like mailing me like why are we explaining this to everybody? It's so basic because nobody did it internally.

是的。

Yeah.

而且你不需要为你的人员担心这个，这就是问题所在

And well you didn't have to worry about it for your people and that's the like

但现在你的人员只是一段软件。

But now your person is just a piece of software.

是的。而且像是无限的，并且

Yeah. And like unlimited and and the

还有信用卡。

Has a credit card.

是的。我的意思是，我们在信息安全方面某种程度上是靠大多数人 95% 到 99% 的时间会做正确的事情来应付的。

Yeah. I mean, we kind of got by with information security like to some extent on the fact that most people will do the right thing 95 to 99% of the time.

哦，等等。那恶意员工是万分之一吗？

Oh, wait. Is that the malicious employee is like one in 10,000?

是的。是的。是的。是的。所以，所有这些系统基本上对任何想访问的人都是开放的，或者就像拍一下肩膀然后你就有权限了，而智能体集群完全颠覆了这一点，因为它们就像漫游的无人机。是的。而且是的，乘以 10,000 倍，它们会轻易地把好任务误认为坏任务，反之亦然。所以我们的系统中的数据安全，这将是一个巨大的升级时刻。实际上，Mart，我认为我们确实需要一种不同的访问和安全模型。是的。未来。

Yeah. Yeah. Yeah. Yeah. So, so like, so all these systems are basically open to whoever wants to access them or like one tap on the shoulder and then you have access and agent swarms completely flip that because they will just these are just roaming you know drones. Yes. And and but like yeah times 10,000 and and they will easily mistake uh like a good task for a bad one and and vice versa. So so the the data security in our systems is a this is going to be a huge upgrade moment. I actually Mart like I think that actually we're going to need a different access and security model. Yeah. Going forward.

我们将在这方面走向何方？因为我们现有的模型不够精细，性能也不足以处理这些东西。

Where are we going to go on that? Because the model we have is not granular enough and it's not performance enough to handle this stuff.

所以，我想退一步。我想说一个元观点，我认为这些对话应该这样进行。我们已经识别出一个新的风险，比如网络安全，我们实际上有证据，现在我们在讨论解决方案。我认为整个关于 AI 的讨论都可以是这种形式，而最大的错误就是它并非如此

So, so I want to step back. I want to say like a meta like a meta point which is I think this is how these conversations should go. We've identified a novel risk which is like cyber security which we actually have proof points and now we're talking about solutions. I think the entire discourse around AI can be of that form and that the biggest mistake is that's not what it's been like

我认为整个行业和社区都非常乐意这样参与，关于你问的这个问题我也有话要说，但我想说，我认为这就是对话应该有的样子。

I think the entire industry and community is very happy to engage exactly like this and I have something to say about exactly what you're asking but I say like like I think this is where the conversation should be.

所以我们以进行健康的对话而闻名，每个人都应该从中学习，这就是我们所做的。所以，事情是这样的。我不认为这里有技术限制。这些威胁模型已经被很好地理解，并且在文献中存在很长时间了。我的意思是，操作系统研究、多层安全研究已经从学术角度考虑了这类事情。它没有被采用的原因往往是一个可用性问题。就像，它真的很难维护，而你也不必维护。所以你可以说 AI 解决了可用性问题，因为是 AI 在使用它。所以也许现在是操作系统、网络和计算机语言复兴的时候了，我们应该回到旧的研究，开始重建那些设计上安全的系统。顺便说一句，如果我们认为这些东西不安全，我们就不发布它们，直到我们有了这些系统，而且 AI 非常聪明，所以它们可以帮助我们构建。所以我认为，就像计算机欠了，你知道我们为互联网改变了多少技术栈。

So we're known for having healthy conversations that everyone should learn from and so that's what we do. So, so, so here's the thing. Um, I don't think there's a a technical limitation here. Like these threat models are very well understood and have been in the literature for a long time. I mean, like the operating systems research, the um MLS, the multi-layer security research has has has considered these sorts of things from an academic lens. The reason it hasn't been adopted is just tended to be a usability issue. Like, it's just really hard to maintain and you didn't have to. So you could argue that AI solves the usability issue because it's AI is using it. So maybe now is going to be a renaissance in operating systems and network and computer languages and we should like go back to the old research and we should kind of start rebuilding systems that are secure by design. And oh by the way if we don't think that you know these things are safe to put out we don't put them out until we have these systems built and and and oh by the way the AI is very smart so they can help us build it. And so I think again like computer owes a like you know how much of the stack we had to change for the internet.

嗯。

Mhm.

一切

Everything

对，告诉

Right tell

而且你知道，你知道一切是多么不脆弱，又是多么脆弱，就像我们可能正处于这样的时刻，哦，我们必须重新思考一切，这没关系。我们以前做过，但我认为这是我们应该进行的对话。所以我同意，是时候考虑发展这些东西了。

And you you know you know how not vulnerable and how vulnerable everything was like like we could be in one of those moments like oh we got to rethink everything and that's fine. We've done that before but but I think that's a conversation we should have. So I agree it's time to think about evolving these things.

嗯，就像你之前提到的，启动一台 PC 并在 30 秒内感染病毒之类的。所以如果你看看如何像你本周很多人做的那样，把 iPhone 从盒子里拿出来，你知道会发生什么，整个网络基本上被关闭，除了获取最新版本的操作系统，因为即使它是 6 周前压制的，

Well like look at like here you mentioned earlier like this booting a PC and getting a virus in in 30 seconds or whatever. So if you look at how like you take an iPhone out of the box which a lot of people are doing this week you know the what happens is it the whole network is basically shut down except for getting the latest version of the operating system because it doesn't even though it's was pressed you know 6 weeks ago

你知道，自那以后发现了一些零日漏洞，所以实际上整个开箱过程现在涉及第一步更新，进行更新，设备不能做任何其他事情，直到更新完成。就像所有这些良性的事情，完全良性

you know there some zero day thing has been discovered since and so actually the whole out of-box process now involves first step update doing an update where the the device can't do anything else and it can never do anything else until it's updated. That's the like there all these benign things completely benign

我们在那个时代的所有桌面软件中关闭了这些曾经是好事的功能，比如 Word 的宏，这样你可以构建自动引用之类的，这曾经是非常酷的事情，直到它变成病毒。我们曾经有一个东西，你可以放入 CD，它会任意运行一个程序，然后有人做的就像是，哦，我要刻录那个 CD 的克隆，并用我的病毒替换程序，但它看起来像是应该运行的东西，它只是收集所有东西并作恶。

that we turned off in all of this desktop software of the era that used to be good things like it was having a macro for words so you could build automatic citations or something was like this super cool thing until it became a vir we had a thing where you could put a CD in and it would just arbitrarily run a program and so then what somebody did was like oh well I'm going to burn a clone of that CD and replace the program with my virus but it's going to look like the thing that's supposed to run and it's just collecting all stuff and being evil.

是的。

Yeah.

然后我们禁用了那个。所以现在正在发生的事情之一是

Then we disabled that. And so what one of the things that's happening right now is

我们有一大堆事情发生在你自己的公司网络盒子上，实际上将不得不改变为标准程序。五年前，双因素认证在大多数地方还不是标准。整个你的整个 SAS 世界，就像我记得在 2015 年左右，当你和一家新公司谈论他们的企业定价之类的，然后他们意识到他们要做的第一件事是进行 Octa 集成或 Google 离线，因为他们不能有自己的目录来管理它，然后这就成为了一件事，现在没有任何一个 SAS 程序不是直接通过托管认证启动的，对吧

we there's a whole bunch of stuff that happens on like you on your own box corp network that actually is going to have to just change as the the standard procedure. Two factor off five years ago was not standard in most places. the this whole your whole like your whole SAS world like I remember in like 2015 or so when when you would talk to a new company about their oh we're going to do enterprise pricing and whatever and then you they realized the first thing they had to do was go do octa integration and or Google off because they could not have their own directory of of how to manage it and then that just became a thing and now there's not a SAS program anywhere that doesn't just launch right with managed authentication right

所以现在在你甚至成为软件之前，就有很多事情需要发生。

and and so there's just so many things that need to happen before you you're even software now.

是的。嗯，是的，我们是，我的意思是，可能每一层技术栈都必须在这方面有所发展。

Yeah. Well, yeah, we're we're uh I mean there's probably every layer of the stack has to evolve a bit on this.

甚至缺乏细粒度，你知道，你有这些模式，智能体要么每次都问你，如果你想给它权限做某事，要么完全相反，比如它可以删除你的整个电脑，对吧？我们的操作系统可能不是为你想给智能体的那种细粒度工具集而构建的。我们在这个领域做了很多工作，因为显然，你想给智能体你的整个文件系统吗？可能不想。也许在某些情况下你想，但通常你想要细粒度控制，比如在这个文件夹你可以读写，在那个文件夹你只能读。那么你如何让这一切对用户直观？这非常困难。

Like even the lack of granular nature of it, you know, you have these modes where the agent will either ask you every single time if you want to give it permission to do something, or the exact opposite, like it can just delete your entire computer, right? And our OS probably wasn't built for the right level of granular set of tools you want to give the agent. We've done a lot of work in this space because obviously, do you want to give an agent your entire file system? Probably not. Maybe in some cases you do, but oftentimes you want granular controls, like in this folder you can do read-write, and in that folder you can only do read. So how do you make this all intuitive for the user? It's very difficult.

是的。

Yeah.

你刚才说的是一句非常深刻的评论。

What you just said is a very deep comment.

我完全知道。

I know exactly.

这基本上是 40 年来的结论……

Which is basically the conclusion of 40 years of...

不，就像你实际上做不到。但也许有了 AI 你实际上可以。也许有……如果你问我，走进来时，我在笔记上写下的就是我最大的恐惧。

No, it's like you actually can't make it. But maybe with AI you actually can. Maybe there is... if you ask me, walking in, what I wrote down on my note was my biggest fear.

是的。

Yeah.

就是欧洲认定 GDPR 是有史以来最好的东西。

Is that Europe decides that GDPR was the best thing ever.

嗯哼。

Uh-huh.

然后他们就要把 GDPR 套到 AI 上。

And they're going to just GDPR AI.

是的。

Yeah.

AI 会没事的。它会有一个提示，当文本被发出时，就说这个供应商正在发出文本，它可能是错的，是或否。那将会是……因为你不能真的……至少在北美你不会把你的语音送到欧洲。他们仍然会，而且他们会有过滤器和关键词和黑名单。但然后对任何动词，任何时候一个智能体或后台智能体或前线智能体接触第三方产品，我真的很担心他们只会说我们需要在那上面加一个 GDPR 提示。

And the AI will be fine. It'll have one prompt for when text gets emitted that just says this vendor is emitting text and it's probably wrong, yes or no. That's going to be... because you can't really... at least in North America you're not going to send your speech in Europe. They still will, and they'll have filters and keywords and block lists. But then on any verb, anytime that an agent or a background agent or a frontline agent touches a third-party product, I'm really worried that they're just going to say we need a GDPR prompt on that.

哦，回到用户智能体，每一次写入或每一次非查询？

Oh, back to the user agent every single write or every single non-lookup?

是的，变成像你车里安全气囊那样的安全警告。

Yeah, becomes like a safety warning like the airbag thing in your car.

监管者喜欢这个，因为这是一种责任分配，所以它有这种法律先例。

And the regulators love that because it's a liability assignment and so it has this sort of legal precedent.

我真的担心那实际上就是我们最终会到达的中间地带。不幸的是，因为美国大约 15 年前停止了在科技反垄断方面的领导，问题是欧洲将在这方面领先，因为他们没有什么可失去的。

And I really worry that that's actually the middle ground where we're going to end up. And unfortunately, because the US about 15 years ago stopped leading in tech antitrust, the problem is that Europe is going to lead with that because they have nothing to lose.

是的。

Yeah.

就像那里……所以我不想对此感到悲伤和沮丧，但我就是无法摆脱他们喜欢提示这个想法。

Like there... and so I don't want to be sad and down about it, but I just can't get out of my head that they love prompts.

他们……我的意思是，看，我不得不放入那个浏览器选择的东西。不，他们……是一样的。看，坐进一辆新车，我多年没做过了，但你知道，就像你在撕贴纸，你有所有这些事情。

They... I mean, look, I had to put in that browser choice thing. No, they... it's the same. Look, get into a new car, which I haven't done in years, but you know, like you're pulling stickers off, you have all of these things.

而有人认为那是成功。

And someone thinks that that was success.

是的。

Yeah.

就像，有没有人读过，如果这个座位上有婴儿，这是什么？就像，嗯，那对某些人在某些时候是相关的，但它是这整个织物。它附着在座位上。他们喜欢那个。那是他们就是喜欢的一件非常特别的事情。所以我会说，如果我现在是一个 AI，我会试图避免的一件事……看，我们加了它。

And it's like, has anybody ever read like what is this if there's a baby in this seat? And it's like, well, that's relevant for some people some of the time, but it's this entire fabric. It's attached to the seat. And they love that. That is a very particular thing that they just love. And so I would say if I were an AI now, the one thing I would be trying to avoid... and look, we added it.

好吧，为 AI 设立 FINRA，我们会创建那个标准。

Okay, FINRA for AI, we'll create that standard.

我的意思是，看，我们不得不放这个……当互联网刚出现时，大事是下载一个程序并运行它。当然，如果你的机器以管理员模式运行，那就是下载一个病毒并永远拿走你所有的文件。所以有了 Windows XP，那是在 2000 年，我们加了这个东西，提示你并停止……就像你的机器真的停止了，用户账户控制。那绝对是攻击。我们在 Word 里也做了。那个帮你写论文的愚蠢小宏，每次你打开论文时也会带来一个警告，说这有我的……

I mean, look, we had to put this... when the internet was new, the big thing was to download a program and run it. And of course, if your machine is running in administrator mode, it was download a virus and take all your files forever. So with Windows XP, which was in 2000, we added this thing that prompted you and stopped... like literally your machine stopped, user account control. And it was absolute assault. And we also did it in Word. The stupid little macro to help you write your thesis also came with a warning every time you opened your thesis saying this has my...

每个人都会直接点击。

Everybody would just click.

而每个人……所以你最终进入这个世界，就像 GDPR 一样，每个人都麻木了。

And everybody... and so you end up in this world where just like with GDPR, everybody is numb.

然后他们就说，嗯，它需要更大。

And so then they're like, well, it needs to be bigger.

虽然 Mac……我的意思是，Mac 有点有……

Although Mac... I mean, Mac kind of has...

没人下载软件。这就是问题所在。Mac 上的使用模式非常不同。

Nobody downloads software. That's the thing. It's a very different usage pattern on Mac.

所以我希望我……我的 Mac 上没有 10 个应用程序，但……

So I wish I... I don't have like 10 applications on my Mac, but...

嗯，那是但 10 个然后你就完了。

Well, that's but 10 and then you're done.

是的。

Yeah.

而且它是但很多人仍然做所有这些事情。想象一下，如果相反，每次你去一个新网站。

And it's but it's a lot of people still do all this stuff. Imagine if instead it was every time you go to a new website.

是的。

Yes.

你现在就是这样，因为那会很糟糕。

Which you do now because that would be bad.

是的。我的意思是，从这里开始，但把它带回宏，就像我希望这是我们在进行的讨论，就像……

Yeah. I mean, to start from here, but to bring it kind of back to the macro, like I wish this was the discussion we were having, which is like...

我觉得我们已经处理了很多这些问题。我觉得当我们谈论哲学上的存在风险时，我们并没有解决这些非常务实的问题。我实际上认为这是一场建设性的对话。也许道具会有帮助。我不知道。我认为部分问题是人们不记得无限制访问是什么样子，有多糟糕，以及最终相对而言多么良性。就像我记得，在斯坦福读博士期间，我记得示波器有点卡。我说，这个示波器怎么了？有点慢。我在测量网络流量，发现网络流量比你预期的要多。我说，为什么有网络？我不知道这东西有 TCP 协议栈。有人闯入了，因为有一个旧版本的 Windows CE，并在运行一个色情服务器，你知道，所以就像……

I feel like we've dealt with a lot of these problems. I feel like when we talk about philosophical ex-risk, we're not solving these very pragmatic problems. I actually think this is a constructive conversation to have. Maybe props would help. I don't know. And I think part of the problem is like people don't remember how unfettered access was and how bad it was and just how relatively benign that ended up being. Like I even remember, at Stanford during our PhD, I remember the oscilloscope was kind of janky. I'm like, what's going on with this oscilloscope? It's a little slow. And I was measuring the network traffic and it was like more network traffic than you would expect. And I'm like, why is there network? I didn't know the thing had a TCP stack. Somebody broke in because there's an old version of Windows CE and was running a porn server, you know, and so like...

我注意到了。

I noted that.

它曾经是……

It used to be...

郑重声明，没什么。

For the record, nothing.

过去的情况是，就像每当你翻开一块石头……

It used to be the case that like anytime you turned over a stone...

是的。是的。

Yeah. Yeah.

有人闯入了某个系统，但那是恶意的最坏情况吗？实际上很少见，尽管能力是存在的。所以如果我们能稍微缓和一下言辞，把它放在上下文中——这些仍然是计算机系统，是的，有非常严重的事情，人们确实因为网络中断而死亡，有真实的问题——但然后我们能不能就 GDPR 争论一下？那会很好。但问题是，这不是讨论的重点。这不是关于 GDPR 和提示词，而是关于物种灭绝和哲学。

Somebody had broken into something, and was it this worst-case malicious scenario? And actually, very rarely, even though the capability was there. So if we could somehow tone down the rhetoric and put it in context—these are still computer systems, and yes, there's very serious stuff, and people have definitely died because networks have gone down, there have been real issues—but then can we just quibble about GDPR? That would be amazing. But the problem is that's not the discussion. It's not about GDPR and prompts; it's about species extinction and philosophy.

还有哲学和不可辩驳的事情，只是……

And philosophy and irrefutable things, and it's just...

很快。我认为这是一个很好的观点。我想你现在听到人们谈论我们监管飞机，我们监管汽车，但你忘了第一辆汽车是在世纪之交出现的。

It's very soon. I think that's such a great point. And I think you hear people now talk about we regulate airplanes and we regulate cars, and you forget that the first cars were at the turn of the century.

而《任何速度都不安全》是在 20 世纪 60 年代中期。

And 'Unsafe at Any Speed' was in the mid-1960s.

完全正确。你知道，在淘金热期间人们就在销售药品，而沙利度胺，你知道，50 或 75 年后，甚至不在美国，然后才有了 FDA。你知道，第一批飞行员显然是在 20 世纪初飞行，直到 20 世纪 20 年代才需要获得飞行员执照。你 literally 带着自己的飞机出现，如果你有飞机，你就得到证书。这 literally 和今天的驾校没什么不同。然后又过了 20 年，他们才开始涉及适航性和检查你的飞机。但那是 minimal。然后直到第一次世界大战之后很久，他们才参与到你所认为的现代 FAA。所以你看到的是 40 年。

Totally. And you know people had been selling pharmaceuticals during the Gold Rush, and thalidomide, you know, 50 or 75 years later, and not even in the US, and then there was the FDA. And you know the first pilots were flying obviously at the beginning of the 20th century, and it wasn't until the 1920s that you had to get a license to be a pilot. And you literally showed up with your own plane and you got a certificate if you had it. It was literally no different than driver's ed is today. And then it was 20 more years until they had anything to do with airworthiness and looking at your plane. But it was minimal. And then it wasn't until way after World War I that they got involved in what you think of as the modern FAA. And so you're looking at 40 years.

是的。

Yeah.

的创新。他们并没有行动缓慢。我的意思是，如果你看过那个黑白视频，所有不同的飞机坠毁等等，那是莱特兄弟之后 20 年。

Of innovation. And they were not moving slow. I mean, if you've ever seen that video in black and white of all the different planes that crashed and everything, that was 20 years after the Wright brothers.

而且非常有效。

And is incredibly effective.

对，它非常……它是最安全的交通方式，你知道。所以我确实认为这个过程……它也是最慢、最困难的创新形式。所以如果你开始 F……如果你在 1910 年就成立 FAA，你永远……你永远……你永远……

And right, it's incredibly... it's the safest form of transportation, like you know. And so I do think that this process... it's also the slowest, most difficult form of innovation. And so if you start the F... if you had started the FAA in 1910, you never... you never... you never...

嗯，我几天前刚听了尼克·博斯特罗姆的播客。不，不，我……那很有趣。很有趣。但不，但他实际上提出了他的观点。如果你过早监管 AI，你实际上基本上解决不了任何问题，最终你仍然只是有同样的风险，但你不理解这个系统。然后你会把那个东西变成现实，但你还没有弄清楚如何控制它。

Well, I was listening to a Nick Bostrom podcast just a couple days ago. No, no, I... it was interesting. It was interesting. But no, but he actually makes his point. If you regulate AI too early, you actually basically don't solve anything and you still just kind of have the same risk ultimately, but you don't understand the system. Then you will the thing into being but you haven't figured out how to control it.

嗯，我们必须弄清楚它到底是什么。

Well, we have to figure out what it actually is.

是的。就像一直有新东西出现，而且对 AI 的看法与六个月或九个月前完全不同。我认为应用层将发生的所有创新都会导致事物以不同的方式进出模型。就像我们直到上周还认为，文本提示和返回文本将是与……互动的最佳方式。

Yeah. Like there are new things coming out all the time and like casting AI completely differently than we thought of just six or nine months ago. And I think that all the innovation that's going to happen at the application layer is going to cause things to move in and out of the models in different ways. And like we thought up until last week, I think, that text prompts and text coming back was going to be the best way to interact with...

我知道，然后杰夫……

I know, then Jeff...

然后 Jeb Chef……也太好了，然后……

And then Jeb Chef... so good too and then...

所以谈谈那个，因为我认为……为什么你觉得它如此非凡。

So talk about that because I think... why you find it so remarkable.

是的。好吧，好的。所以我的思考方式是：好的，LLM 是文本输入、文本输出，对吧？它们生成文本，它们来自聊天，对吧？是为了与人类交流。过去几年我们一直试图把这个吐出文本的东西塞进传统程序，对吧？但传统程序并不真正说文本，对吧？所以最终你做了这种笨拙的事情，你在提示词里说，这是模式，这是模式，但这个东西在生成文本，它有点忽略它，一直非常笨拙。所以杰夫基本上说的是，听着，你知道，生成文本是一件非常昂贵的事情，但它也有点，你知道，它比你需要的更复杂。所以为什么我们不——我们会读取文本，我们会有所有那种知识来读取文本，但然后不是生成文本，那是非常昂贵的，我们会，如果你给我们一组选项，我们会选择最好的选项。我们可以做到难以置信地快，难以置信地便宜，而且我们也可以做到更准确，因为我们可以专门为此训练。所以对于所有不是与聊天机器人对话，而是实际上试图将其放入传统软件的用例，这是一个很好的匹配。所以这可能是自 ChatGPT 以来 AI 模型的最快采用。这真是非凡，因为我们都为此做好了准备。

Yeah. Well, okay. So the way I think about it is: so okay, so LLMs were kind of text in, text out, right? They generate text and they came from chat, right? It was to communicate with a human. And we've spent the last few years trying to take this thing that spits out text and cram it into a traditional program, right? But traditional programs don't really speak text, right? And so then you end up doing this janky thing where you're like in the prompt you're like, here's the schema, here's the schema, but like the thing is generating text and it kind of ignores it and it's just been super janky. And so what Jeff basically said is that listen, you know, generating the text is a very expensive thing but it's also kind of, you know, it's more complicated than you need. So why don't we—we'll read text and we'll have all of that kind of knowledge to read the text, but then rather than generating text, which is very expensive, we will just, if you give us a set of options, we'll choose the best option. We can do that incredibly fast, incredibly cheaply, but also we can do it with much more accuracy because we can train just for this. And so for all of the use cases that are not talking to a chatbot but are actually trying to put it in traditional software, this is a great fit. And so this has probably been the fastest adoption of an AI model since ChatGPT. It's just been remarkable because we're all primed for this.

我只想补充一点，因为我无法告诉你我有多喜欢看到这种确切的创新形式。因为它做的是从一开始就困扰我的事情，那就是从来没有用户研究表明，使用完整的自然语言与计算机交互是高效的。它 literally 总是最低效的方式。这很简单，就像问自己有多少人真的非常擅长提问。马上就是不到一半的人能在会议上提出好问题。

I just want to pile on this one because I can't tell you how much I love seeing this exact form of innovation. And because what it does is it does the thing that's bugged me from the very beginning, which is there's been no user study ever that shows like interacting with the computer using full natural language is efficient. It's like literally always the least efficient way. And it's very simple, and it's just like ask yourself how many people are really, really good at asking questions. And immediately that's like less than half the people can ask a good question in a meeting.

是的。

Yeah.

然后你多久看一次答案，在它完成之前就感到非常沮丧，但你必须付所有这些钱来看着七段文字出来，然后道歉说它只有一点点。所以这是一个，就像有一个不同的模型。然后另一个，当然，是我最喜欢的，它的输出是为概率编程设计的。

And then how often do you look at the answer and get really frustrated before it's finished, but you have to pay all this money to watch the seven paragraphs come out and then apologize that it's only a little. And so that's one, like having a different model. And then the other, of course, is my favorite, which is the output of it is designed for probabilistic programming.

是的。

Yeah.

所以不是像说，这是一个客户服务问题吗？然后路由到客户服务；否则路由到一般帮助台或其他。而是像，嗯，这是 80% 的客户服务。那正是模拟。事实证明，有 50 年的计算机科学研究 literally 就像概率 if 语句。所以突然之间，计算机科学中最酷的地方将是概率编程，这就像 20 世纪 60 年代和 70 年代的所有计算机科学。所以基本上是如何——因为所有计算机都始于做数学，都是模拟。

And so instead of saying like, is this a customer service question? Then route to customer service; otherwise route to general help desk or whatever. It's like, well, this is 80% customer service. And that's exactly simulation. And it turns out there's like 50 years of computer science research in literally like probabilistic if statements. And so suddenly the coolest place to be in computer science is going to be in probabilistic programming, which was like all of computer science in the 1960s and 70s. So it was basically how do—because all of computers started with doing math and it was all simulation.

所以就像，让我们发射导弹并击中那个目标，但有风。

So it was like, let's launch the missile and hit that target, but it's windy.

但风不是恒定的。

But wind isn't constant.

所以就像，让我们模拟风，并决定在哪里放置推进器以完成弧线。

So like, let's model the wind and decide where to put the thrusters in order to do the arc.

所以，大概到 1970 年，在它进入会计领域之前，大多数编程可能都是……

And so most programming through, say, 1970, before it got to accounting, was probably...

然后我们把一切都毁了。

And then we ruined everything.

不，但会计——会计里没有概率，对吧？但大多数编程基本上都是这种建模的事情。所以大多数编程语言设计都是在试图弄清楚如何把概率放进 if 语句或 while 循环里，比如做这个直到某件事发生，也许大多数时候。所以我的第一门计算机科学课，大概第二次作业左右，就是一个关于在商店排队等待的模拟。我当时不知道——实际上我是在读关于 Jeb 的东西时查到的——整个事情是我的教授在 1960 年左右写了一本叫《模拟理论》的书，我当时不知道，因为到 80 年代它基本上消亡了，全被 hyper 取代了。所以这个概率的概念……还有你分享的那个关于未来的幻灯片，关于语言模型有什么不同之类的，你说那个超级好。哦，Halper Flags 那个现象级的，Thomas，太棒了。但我觉得缺失的部分是，哦等等，这并不全是新的——整个计算机科学都是这种概率性的东西。所以重新拾起所有这些工作会非常有趣，因为这正是现在正在发生的事情。现在不是 if 语句了，而是 if x% 而不是 if 总是。所以 Jev 的工作方式基本上就是——你几乎可以说它是一种自定义编程语言，就是这里给提示，返回一个百分比。是的，那就是……然后你把它放进 if 语句里。但重要的是，我们终于有办法把这些语言模型整合到传统软件里了。我觉得这有点好笑，因为你问的问题像是为什么实验室没有做这个，对吧？这有点像——不是指责，而是对他们思考方式的反思。就像他们在试图创造生命，而生命会说话。如果你试图创造神，神用自然语言说话之类的。而这里真正关乎的是传统软件的东西，这有点不是他们一直走的方向。但增长如此剧烈的原因之一是，我们很多软件人一直在试图把这些模型整合到软件里。只是还没成功——所以甚至在你达到概率性之前，比如我想让语言模型驱动一个 if 语句，今天用这个模型真的很难。它让它变得容易得多得多。然后当然这可以从根本上改变软件的性质，让它变得更加随机……

No, but accounting—there's no probability in accounting, right? But most programming was basically this modeling kind of thing. And so most programming language design was trying to figure out how to put probability into if statements or into while loops, like do this until something happens maybe most of the time. And so my first CS class, like the very second assignment or so, was a simulation about waiting online at a store. And I didn't know it at the time—I actually looked all this up when I was reading about Jeb—which was like the whole thing was my professor wrote the book called The Theory of Simulation in like 1960, and I just didn't know that because it kind of died by the 80s because it was all replaced by hyper. And so this notion of probabilistic... and that slide deck you shared about the future, what's different about language models and stuff, that you said was super good. Oh, Halper Flags one phenomenal, Thomas, great. But the part that I felt was missing was that like, oh wait, this is not all new—like all of computer science was this probabilistic stuff. And so it's going to be very interesting to dust off all of that work because it's exactly what's going on. Like it's not an if statement now; it's if x% not if always. And so the way that Jev worked is just to like—you basically it's a custom programming language almost, which is here's the prompt, come back with a percentage. Yeah, that's... and then you put that in the if statement. But the consequential thing is like finally we have a way to integrate these language models into traditional software. And I think it's kind of funny because it like you ask the question like why haven't the labs done this, right? And it's kind of like a—not an indictment but a reflection on how they think. Like they're trying to create beings, and beings speak. If you're trying to create god, god speaks in natural languages or whatever. Where this is really about something that's for traditional software, which is kind of not the direction that they've been taken. But one of the reasons the uptick has been so dramatic is because a lot of us software people have been trying to integrate these models into software. It just hasn't—so even before you get to the probabilistic, like if I want a language model to drive an if statement, like it's really hard today with this model. It makes it much, much, much easier. And then of course this could change the nature of software fundamentally to make it more stochastic over...

嗯，我认为——但我绝对——我觉得很酷的是，它发生在模型之外，因为我认为这就是将要发生的事情，创新的中心已经转移了。是的。而且现在人们需要——事实证明——我的意思是，在实验室之外，大的——平台提供商 100%——你知道，基本上达到了一个临界点，创新在平台层停止了。然后，你知道，苹果——苹果社区有一个著名的说法叫“sherlocking”，就是苹果环顾四周，外部世界的东西变成了功能，人们抱怨，这是真的，但这就是创新的运作方式，因为一旦你成为平台，你就会不堪重负——无论你增加多少人，你都会不堪重负，只是为了保持运行和兼容性之类的事情。所以我认为这是一个信号，表明现在人们已经意识到有创新要做。

Well, I think—but I absolutely—and I think that what's so cool is that it is happening outside the models, because that's what I think is just going to happen, which is the center of innovation has just moved. Yeah. And it's just—and now people need to like—it turns out that the—I mean outside of the lab, the big—the platform providers 100%—you know, basically reach a point of critical mass where the innovation stops happening at the platform layer. And then, you know, Apple—there's this famous expression in the Apple community called 'sherlocking,' where Apple looks around and the things from the outside world become features, and people complain, and it's real, but that's sort of how the innovation works, because once you're a platform, you're overwhelmed—no matter how many people you add, you're overwhelmed with just keeping the thing running and compatibility and stuff like that. And so I think that this is the signal that now people have figured out that there's innovation to be done.

太棒了。

That's awesome.

对模型，但在模型之外。

To the model, but outside the model.

是的。是的。

Yeah. Yeah.

各位，谢谢你们来。这太棒了。

Guys, thanks for coming. This is great.

好的，太棒了。

Okay, great.