递归自我改进：从自动研究到超级智能——Richard Socher

我认为，如果真的动用法律的全部力量去监管人们在 GPU 上的行为，其弊端会超过他们担心的任何问题。那会是一个疯狂的极权国家。如果你试图监管智能，那就是在监管思想，这既荒谬又疯狂。我认为监管这项技术的某些应用是合理的。

I think the downsides of actually trying to truly regulate with the full power of law what people do on their GPUs would be worse than any of the concerns that they have. It would be a crazy totalitarian state. If you try to regulate intelligence, it's trying to regulate thought, and that's ridiculous and crazy. I think it is sensible to regulate some of the applications of this technology.

在我们进入今天的节目之前，我有一条给听众的简短信息。谢谢你们。如果你们没有选择点击并收听我们的内容，我们就无法为你们带来你们如此明确想要的 AI 工程、科学和娱乐内容。几乎每天都有人找我们做赞助。但幸运的是，你们中有足够多的人实际上订阅了我们，使这一切在没有广告的情况下也能持续下去，我们希望保持这种状态。但我只想请你们帮个忙。你们能做的最强大、完全免费的一件事就是点击那个订阅按钮。这是我唯一会请求你们做的事，它对我以及我团队来说意味着一切，他们每周都在努力把 Inspace 带给你们。如果你们这么做，我向你们保证，我们会一直努力让节目变得更好。现在，让我们开始吧。

Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring Latent Space to you each and every week. If you do it, I promise you, we'll never stop working to make the show even better. Now, let's get into it.

我们在演播室里，有 Vibhu、我自己和 Richard Socher。欢迎。

We're here in the studio with Vibhu and myself and Richard Socher. Welcome.

谢谢你们邀请我。

Thanks for having me.

我们刚谈到了 Eureka 机器，或者说我们刚在 AI Engineer 上发布了一个关于 Eureka 机器的演讲。你说这是你的人生目标。Eureka 机器是什么？

We just talked about the Eureka machine, or we just released a talk at AI engineer about the Eureka machine. You said it's your life's goal. What is the Eureka machine?

Eureka 机器是终极发明，之后它将为人类发明几乎所有东西。它本质上是一种超级智能，可以接受任何目标、任何环境奖励，然后它会尽力实现这些目标，创造出人类希望它发明的各种发明。

The Eureka machine is the ultimate invention that will afterwards invent most everything for humanity. It's essentially a superintelligence that can be given any kind of goal, any kind of environment reward, and then it will try its best to achieve those goals, to create the kinds of inventions that humanity would hopefully ask it for.

是的，我想我们这里打开了你们写的书。

Yeah, I think we have the book pulled up here that you people have written.

没错。是的，我去年在开始 Recursive 之前不久完成了它，现在我们要尝试构建其中的部分内容。

That's right. Yeah, I finished it last year a little bit before we started Recursive, and now we're going to try to build parts of that.

你知道，你去年就完成了。现在是七月。为什么这么久？

What you know, you finished it last year. It's July. What takes so long?

天哪，书籍出版慢得令人难以置信。太荒谬了。整个行业都慢得难以想象。所以，是的，很多想法已经存在一段时间了。但是的，我很高兴它终于在今年九月出版。

Oh man, books are incredibly slow. It's ridiculous. That whole industry is just unfathomably slow. So yeah, a lot of the ideas have been out there for a while. But yeah, I'm really glad it's finally coming out in September this year.

我的意思是，到那时我们可能已经拥有 AGI 了。我们不知道。有什么你最兴奋地想要放在这里的要点吗？

I mean, we might have AGI by then. Like, we don't know. Any key takeaway that you're most excited to put in here?

是的，我认为关键要点是，人们可以而且应该对超级智能的积极影响更加兴奋，尤其是在科学、物理、化学、生物学方面，还有经济学、天体物理学以及各种其他工程任务。我认为有了更好的技术，可以做的事情还有很多。而现在我觉得很多人需要更好的营销，不仅是对未来的总体营销，还有对技术，特别是对 AI 的更好营销。这本书应该向即使是对 AI 持怀疑态度的人展示 AI 有多大的积极潜力，特别是在创造新的科学发现方面。

Yeah, the key takeaway I think is that people could and should be much more excited about the positive implications of superintelligence, especially for science, physics, chemistry, biology, but also economics and astrophysics and all kinds of other engineering tasks. I think there is so much more that can be done with better technology. And right now I feel like a lot of people need better marketing, not just for the future in general but also better marketing for technology and in particular for AI. And this book should show even the AI skeptics how much positive upside there is for AI, especially when it comes to inventing new scientific discoveries.

我想你引用了 Mark 和 Jason 的技术乐观主义宣言，我觉得它在雄心、清晰和简洁方面几乎可以说是优美的。

I think you quoted the techno-optimist manifesto from Mark and Jason, which I think was like kind of beautiful in its ambition and clarity and simplicity almost.

我同意。是的。你可以在某些事情上不同意他，但我认为他在技术乐观主义上是对的。

I agree. Yeah. You can disagree with him on some things, but I think he's right on the techno-optimism.

你认为乐观主义者会在哪里遇到麻烦？

Where do you think optimists get in trouble?

你知道，显然你不应该有盲目的乐观。你应该非常清醒，尤其是面对像 AI 这样用途广泛的技术。你需要考虑潜在的下行场景，尤其是当人们将其用于你不希望他们使用的用途时。这有点像互联网。我觉得人们有时试图监管 AI 是因为那些潜在的下行风险，就像你会监管互联网一样。如果你说，因为互联网上有不良内容，比如折磨色情之类的，我们就应该让它变慢，这样你就不能那么快地分享非法内容，或者我们应该让硬盘变小，这样你就不能存储那么多非法内容。但我觉得，那不是监管的方式。那就像说我们应该抽象地监管智能。即使作为乐观主义者，为了避免那些下行场景，你应该监管的是具体的应用。当然，我不希望某个 AI 外科医生在我的大脑里练习一些 L 动作。它应该完全获得 FDA 认证。当然，我不希望任何随机的初创公司在高速公路上行驶并造成重大事故。在它被允许上高速公路之前，应该获得适当的认证。但我觉得，那些一些乐观主义者有时可能没有充分考虑到的下行场景，与末日论者担心的情况相比，是相当容易监管的。

You know, obviously you shouldn't have blind optimism. You should be very clear-eyed, especially with such an omni-use type of technology as AI is. You need to think about the potential downside scenarios, especially when people use it for things that you don't want them to use it for. It's a little bit like the internet. And I feel like people are trying to regulate AI sometimes because of those potential downsides, the way you would regulate the internet. If you were to say, well, because there's bad content on the internet, like torture porn or whatever, we should just make it slower. That way you can't share the illegal content as quickly, or we should make the hard drive smaller so you can't store as much illegal content. But I'm like, that's not how you regulate that. That's like saying we should regulate intelligence in the abstract. What you should regulate to avoid those downside scenarios, even as an optimist, are the specific applications. Sure, I don't want some AI surgeon to practice some L moves in my brain. It should be fully FDA certified. Sure, I don't want any random startup to drive on the highway and cause a major accident. It should have proper certifications before it's let loose on the highway. But I feel like those downside scenarios that some optimists sometimes maybe don't consider enough are fairly easily regulated compared to what the doomers are worried about.

慢速起飞也是策略的一部分。我确实认为，尽管我对 AI 及其对社会甚至文化的影响，当然还有技术、经济、财富、健康等所有方面感到兴奋。尽管我对所有这些都感到兴奋，但我确实认为，那些对 AI 硬起飞情景最乐观的人高估了事情发展的速度。存在硬件限制。关于算力基础存在物理限制。你能多快获得足够的 GPU？经济中也存在限制，有很多行业不需要疯狂的复杂智能和复杂能力。比如，如果你想想品牌、服装、手袋等领域的工作，超级智能不会让你那价值 1 万美元的 fancy 手袋变得更 fancy。你知道，那对经济没有影响。当你想到旅行和旅游业，人们想去看埃及的金字塔，AI 不会改变太多。当然，你可以生成一张你和一个假金字塔的假照片，

Slow takeoff is part of the strategy as well. I do think as excited as I am about AI and its impact for society and culture even, and certainly technology and economics and wealth and health and all of those things. As excited as I am about all that, I do think the most bullish people on the AI hard takeoff scenarios overestimate how quickly things can move. There are hardware constraints. There are physical constraints about the compute substrate. How quickly can you get enough GPUs on? There are also constraints in the economy where there are a lot of industries that don't require an insane amount of complex intelligence and complex capabilities. Like if you think about jobs in brands and like clothing and apparel and like handbags and stuff, superintelligence isn't going to make your fancy $10,000 handbag any fancier. You know, it's like that will have no effect on the economy. When you think about travel and tourism, people wanting to see the pyramids in Egypt, it's not going to change that much with AI. Sure, you can like generative a fake photo of you and

我可以用 Genie，你知道，在 Gen 中生成一个金字塔。

I can use Genie and you know to a pyramid in Gen.

是的，没错。但就像，还有很多行业，比如伐木和石油。

Yeah, exactly. But like and there's so many industries like logging in and oil.

你不可能凭空让石油产量增加 1000 倍。当然，会有机器人钻井之类的技术，但在那种疯狂硬起飞的情景下，这个行业不会因此翻 1000 倍，无论是经济还是其他方面——我可以一直列举其他例子，比如食品等等，它们未必会有那么大的变化。然后，确实存在物理约束。当然，还有人们主动退出进步。这其实是我经常担忧的一点：我看到欧洲和其他地区的人们，那里很多人几乎想要彻底退出进步。这也会拖慢更多改进。

You're not going to magically get a 1,000x more oil. Sure, there will be robotics like drilling and things like that that could be done, but it's not going to 1,000x that industry in a crazy hard takeoff scenario, both on the economy and—I can go on and on about all the other examples where food and so on don't necessarily change that much. And then, yeah, there are real physical constraints. And then there's, of course, people off-ramping from progress. That's actually one of my concerns often: I see people in Europe and other whole regions almost feeling like many people there want to off-ramp from progress, period. And that will also slow down more improvements.

我们这里调出了这个话题，基本上这是当下非常热门的事情之一，因为并非所有前沿实验室都在呼吁给 AI 一个“节奏”选项。他们不说暂停，他们说节奏。不知道你对此是否有效有什么看法。

We have this pulled up where basically this is one of those things that is very topical right now because not all the frontier labs are calling for the option to pace AI. They don't say pause, they say pace. I don't know if there's any take from you about whether or not this will be effective.

我认为，真正试图用法律的全部力量去监管人们在 GPU 上的行为，其弊端会比他们担心的任何问题都更糟。如果你们每一台计算设备都被某个大政府或多政府机构知晓，那将是一个疯狂的极权国家。这简直——如果你试图监管智能，就是在试图监管思想，这既荒谬又疯狂。我认为监管这项技术的某些应用是明智的。

I think the downsides of actually trying to truly regulate with the full power of law what people do on their GPUs would be worse than any of the concerns that they have. It would be a crazy totalitarian state if every one of your computes was known to some big government or multi-government agency. It's literally—if you try to regulate intelligence, it's trying to regulate thought, and that's ridiculous and crazy. I think it is sensible to regulate some of the applications of this technology.

是的。我是说，我们有过一项法案，一项实际的法案，要监管模型中的浮点运算次数。我就想，好吧，那么——

Yeah. I mean, we had a bill, an actual bill to regulate the number of flops in a model. And I'm like, okay, well—

欧洲已经这么做了。这些人的恐吓足够成功，以至于整个欧洲在还没有真正迎来 AI 起飞之前，就已经自我监管得如此严格，因为他们听信了一些专家的话：“如果这项技术的浮点运算超过这个数，我们可能都会死。”然后他们说：“好吧，我们很好。我们希望人们繁荣。我们不要那种有微小可能让我们全死掉的技术。”于是他们在欧盟就监管了这类东西。所以非常不幸的是，当其他人说“让我们放慢节奏”时，他们自己却以人类可能的最快速度冲向那个前沿，这对某些人产生了实际影响。

Europe done it. Like, these guys have been successful enough with their fear-mongering that all of Europe has kind of regulated itself so much before it even had a proper AI takeoff because they listen to some experts who say, "We might all die if this technology has more than this number of flops." And they're like, "Well, we're good. We want people to thrive. Let's not have technology that could have a small chance of all of us dying." And so they regulated exactly those kinds of things in the EU. And so it's very unfortunate that there are real implications for some people when others say let's pace while they're sprinting as fast as humanly possible towards that frontier themselves.

是的。

Yeah.

这也不是全球暂停，对吧？其他国家仍在以同样的速度加速。如果你试图监管智能、GPU 以及人们在它们上面的行为，你需要一个极权主义的世界政权。

It's also not a global pause, right? Like other nations are still accelerating at the same pace. You need a totalitarian world regime if you try to regulate intelligence and GPUs and what people do on them.

对这件事的安全性有什么看法？最近 Fable 在 56 发布前暂停了。还有 Hugging Face 和 OpenAI 的网络事件。有什么看法吗？

Any takes on the safety of this? So there was a drawback of Fable pause on 56 before it could be released recently. There was hugging face with the OpenAI cyber incident. Any takes there?

百分之百。我认为这些是严重的奖励黑客问题，也是未能进行适当红队或彩虹队的明显失败。我不知道你是否看过 Tim Rocktäschel 和其他几人的这篇论文，基本上就是让一个 AI 去尝试黑另一个 AI，然后它们可以以开放式的方式来回互动，从而真正让自己对这些攻击免疫。是的，就是这篇论文。这真是个聪明的想法。开放性和进化启发对我们 Recursive 来说也很重要。所以我希望他们能更多利用这一点。很明显，比如宪法 AI——我不知道你是否记得 anthropic.com/constitution。你可以把它调出来，直接搜索 cyber。它说硬约束：Claude 永远不会进行网络攻击。这是我们宪法中的硬约束。所以，以下是 Claude 行为的当前硬约束。第三条：创建可能造成人类伤害的网络武器或恶意代码。我的意思是，显然这整个宪法是假的。它显然没有被遵守。

100%. I think these are serious issues of reward hacking and clear failures of actually doing proper red teaming or rainbow teaming. I don't know if you saw this paper from Tim Rocktäschel and a few others where basically one AI is tasked to try to hack another AI and then they can go back and forth in an open-ended fashion to actually inoculate themselves from those. Yeah, this is the paper. It's a really clever idea. Open-endedness and evolutionary inspirations are big for us at Recursive as well. And so I wish they had used more of that. And it's clear that, for instance, the constitutional AI—I don't know if you remember anthropic.com/constitution. You can actually pull it up and search for cyber right there. It says hard constraint: Claude will never ever do cyber attacks. And that is a hard constraint in our constitution. So, here are the current hard constraints on Claude's behavior. Number three: create cyber weapons or malicious code that could cause human damage. I mean, clearly this whole constitution was fake. It clearly isn't being adhered to.

因为 Anthropic 也发现他们自己的——

Because Anthropic also found that they had in their own—

就像他们在说：“哦，好吧，其他人在黑客攻击。”现在，有几件事。第一，你可以让沙箱非常简单，然后很容易就能黑出沙箱，对吧？但我认为这表明我们目前处于 AI 的这种状态：奖励工程师仍然需要做更多细致的工作，而 AI 在大多数情况下还不太擅长理解意图与字面表达之间的区别。具体来说，我认为如果这种智能在众多公司中更容易获得，这种情况就会发生。想象你运营一个服务中心，有人说：“哦，这是我的仪表盘上的 CSAT 分数。让这个数字上升。我们的 CSAT 分数太差了。”智能 AI 就会说：“哦，当然。我就创建一百万个机器人打电话给我们的服务中心，最后给五分好评。”然后数字就像你要求的那样上升了。你说：“我不是这个意思。我是指真实客户。”那家伙又说：“好吧，简单。我给每个失败的 DoorDash 订单发 1000 美元礼品卡。”你说：“我不是这个意思。”他说：“但你就是这么说的。”所以，我认为清晰地阐明奖励是什么，这是人类还没有做得很好的事情。然后显然，在这些情况下，AI 还没有足够好地理解我们要求它并给予某些奖励时的意图。现在，给我希望的是，已经有了变好的初步迹象。我举个例子，比如 Wispr Flow。充分披露，我在 AI X Ventures 投资了他们的种子轮，但 Wispr Flow 在写出你想要的而不是你所说的方面已经好多了。我认为这是未来的一个迹象。我认为随着我们实际上让 AI 越来越智能，会有越来越多的 AI 更好地与意图对齐。

Like they're like, "Oh, well, other people are hacking." Now, there are a couple things. One, you can make a sandbox very simple and then it's very easy to hack yourself out of a sandbox, right? But what I think it shows is that we're currently in this state of AI where the reward engineer still has to do a lot more careful work and where the AI in most cases is not very good yet at understanding what is meant versus what is being said. And so concretely, I think this will happen if we were to have this kind of intelligence more easily accessible in a lot of companies. Imagine you run a service center and someone says, "Oh, here's my CSAT score in my dashboard. Make this number go up. Our CSAT score is so poor." The intelligent AI will just be like, "Oh, sure. I'll just create a million bots that call our service center and give a five out of five rating at the end." And the number went up just like you asked for. And you're like, "That's not what I meant. I meant with our real customers." The guy goes off and says, "Well, easy. I'll just give a $1,000 gift certificate for every failed whatever DoorDash offer." It's like, "That's not what I meant." It's like, "Well, but that is what you said." And so, I think clearly articulating what the rewards are is something we haven't gotten very good at as humanity. And then clearly the AI in these cases has not gotten good enough at understanding what we mean when we ask it and give it certain rewards. Now, what gives me hope is there are the first inklings of this being better. I'll give you an example like Wispr Flow. Full disclosure, I invested in their seed round, at AI X Ventures, but Wispr Flow has gotten much much better at writing what you mean and not what you say. And I think that is a sign of things to come. I think there will be more and more AIs as we actually make us more and more intelligent that will be better at being aligned with what is meant.

这会通过宪法或 HF 来实现吗，还是——

Will it be done through a constitution or HF or—

显然宪法根本不起作用，我认为那主要是营销。我认为我们需要找到更好的解决方案，在 Recursive 我们有一些非常好的想法，有些已经像我们认为更好地掌握了它。我认为我们还没有完全弄清楚，但你知道我们正在大量思考安全，它越智能，你就越希望它对齐，越不希望考虑奖励黑客，而是真正尝试做正确的事。

Clearly constitutions don't matter at all and it doesn't work and that was I think mostly marketing. I think we need to find better solutions for it and I think at Recursive we have a few very good ideas and some already like ways where I think we have a better grasp on it. I don't think we have fully figured out yet but you know we're thinking a lot about safety and the more intelligent it gets the more you want it to be aligned the less you wanted to think about reward hacks and actually try to do the right thing.

我不知道我们是否会触及这个话题，但我还是要把这个问题抛出来，因为它一直压在我心头。对齐，我们姑且称之为，是对齐到全人类的偏好。中位数的偏好。

I don't know if we'll touch on this topic but I'm just going to throw this question in here cuz it's something that's weighing on me. Alignment, let's call it, is alignment to general humanity's preferences. The median preference.

个性化是精准定位你想要的东西。而有时对齐会与之冲突，因为你想要的并不是一般中位数人群想要的。你如何选择？

Personalization is pinpointing what you want. And sometimes alignment can conflict because what you want is not what the general median population wants. How do you choose?

这是个好问题。我认为最终当然必须与法律对齐。无论你的 AI 部署在哪里，它都需要与法律对齐。我确实认为 AI 经常做的其实是把一面镜子放在我们面前，说这就是你现在的样子，现在我可以把它放大一千倍。这还是你想要的吗？事实是，不同文化做出了不同的选择。在东方文化中，集体利益往往比个人更受重视。西方文明更关心个人自由、权利和追求幸福等等。即使在那里也有层次：有监管与诉讼的权衡。在美国，你首先往往不是每次——比如 FDA 等确实监管某些领域——但在许多情况下，坏事发生后有人起诉另一个人，然后基于此制定法律。在欧洲，他们试图避免对任何人造成伤害，并提前监管。两者都在努力做最好的事，但有些实际上更有利于创新。所以是的，你说得对。我认为最终每个人、每个国家以及整个人类都必须更多地思考这些价值观，然后尝试将它们写入法律，这些最终都是约束。希望不同的社会就像现在他们的 AI 一样，将他们的 AI 对齐到不同的价值观，这样我们就不会只有一种对齐的单一文化。

It's a great question. I think you ultimately have to of course be aligned with laws. Wherever your AI is deployed, it needs to align with the law. I do think what AI often does is actually put kind of this mirror in front of us and say like this is what you're looking like now I can amplify that a thousand times. Is it still what you want? And the truth is that different cultures made different choices. In eastern cultures, the greater good is often valued more than the individual. Western civilization we care more about individual freedoms and rights and the pursuit of happiness and so on than others. And even there there are gradations: there's sort of regulation versus litigation trade-offs. In the US, you first can often not every time like FDA and so on does regulate some areas but in many cases the sort of bad things happen someone sues someone else and then there's a law based on that. In Europe they try to often avoid any harm to anyone and regulate before. Both are trying to do the best thing but some is actually more amenable to innovation than others. And so yes, you're right. I think ultimately each individual, each country and humanity as a whole has to kind of think about those values more and then try to put them into laws and those are all ultimately the constraints. And hopefully different societies just like now with their AIs will align their AIs to different ones so we have not just a monoculture of alignment.

关于这一点，我有个原本没打算问的后续问题。你对开源、开放权重与谁拥有智能有什么看法？

Here's a followup on this that I wasn't expecting to ask. Do you have takes on open source open weight versus who owns the intelligence?

所以，显然不是宪法派的最大粉丝。

So, clearly not the biggest fan of the constitution side.

没关系，你知道的。重点是，关于谁应该拥有权重，应该开放吗，有什么想法吗？

It's fine, you know. Point being, any thoughts on who should own weight, should it be open, anything there?

百分之百。我是开源的忠实粉丝。我们将在 Recursive 签署一些各种开源信件。所以我认为即使在最坏情况的攻击场景中，实际上让更多好的行为者拥有更多不同类型的 AI 可访问也是更好的。我认为开源也有点像一种软实力。所以我确实认为对世界其他地方来说，有一个来自中国的答案之外的选择是好的。我确实认为，你知道，当你看好莱坞电影时，你知道，我不想误解所有电影，但有一种宣传感，对吧？你看到一面。

100%. I am a big fan of open source. We're going to sign some various open source letters at Recursive. So I think even in the worst case attack scenarios actually it is better to have more good actors have more different types of AI accessible. I think open source is a little bit a soft power type of thing too. So I do think it's good for the rest of the world to have an answer to that out of China. I do think, you know, when you watch a Hollywood movie there, you know, there like I don't want to sort of mis um sort of this all of movies, but there's a certain sense of propaganda, right? You watch one side.

是的。你看过《壮志凌云》吗？得了吧。就像一半是由美国陆军支付的之类的。

Yeah. Have you seen Top Gun? Like come on. Like it's like half of it's paid for by the US Army or something.

是的。所以，你知道，我认为这很自然，但有趣的是，我认为语言模型本质上是一种类似于电影的软实力，甚至更多，因为它们显然对网络安全等也非常重要，但它们的众多方面之一是讲故事的软实力，比如如果一个孩子问语言模型，告诉我一个激励人心的故事，关于我长大后应该做什么，对吧？这些都是微妙的事情。所以我认为这对西方世界很重要。我确实热爱个人主义。我确实认为尽管资本主义有一些缺陷，但它是我们迄今为止找到的最好的治理方式等等。所以我确实认为，有各种方面，为语言模型提供一个西方的开源答案将是好的，而在 Recursive，我还不能宣布，但我们很快就会在该领域有所作为。

Yeah. And so and you know I think that's just natural like but what's interesting here is I think LLMs are essentially a similar type of soft power to movies and beyond because they're obviously also highly important for cyber security and so on but one of their many aspects is that soft power of storytelling like if like a child asks an LM like tell me an inspiring story of what I should do when I grow up right it's like those are all these like subtle things so I think it's important for for western world. I do love you know individualism. I do think despite some of its flaws like capitalism is the best way we have governed found ourselves to govern and so on. So I do think there are various aspects that will be good to have a western open source answer for LLMs and with Recursive I can't make the announcement quite yet but we'll be relevant in that space very soon.

好的。好吧。正是如此。带我们了解 Recursive。所以除了我们的题外话，你在 NLP 领域有相当深厚的背景。你参与过早期的嵌入、与 Chris Manning 一起的 GloVe，他是播客之前的嘉宾。You.com，历史是怎样的？你如何决定创办另一家公司？

Okay. All right. Exactly. Bring us to Recursive. So outside of our tangents, you have a pretty deep background in the NLP space. You worked on like early embeddings, GloVe with Chris Manning, who's previous guest on the podcast. You.com, what's the history? How did you decide to start another company?

是的。所以我对 AI 的兴奋已经超过二十年了。我有时觉得这已经是古代历史了。这是 ChatGPT 之前的公元前。不，没人关心耶稣基督之前发生的所有宗教，也没人关心 Transformer 和 ChatGPT 等之前发生的模型，但这是我深深热爱的事情。我认为 AI 是一个人能从事的最有趣的事情之一。我认为语言也是人类智能最有趣的表现形式。在 You.com，我们最终从推动 AI 前沿转向主要给人们提供好的搜索引擎、搜索 API 和网络答案。我认为这是智能的一个极其重要的部分，只是知识和访问，尤其是我们可能稍后会谈到。如果你想发明一个为我们发明一切的尤里卡机器，它需要知道如何不重新发明轮子，比喻地说，并且知道已经发明了什么，你必须要有互联网访问。所以，它是语言模型、智能体、聊天机器人等中使用最多的工具，是网络搜索。所以，我真的很兴奋 You.com 能拥有这一点，并在其中发展得很好，拥有非常大的客户等等，但也不再构建前沿模型。所以我最初尝试在 You.com 内部做这件事并筹集另一轮资金等等，但你就是不能。你必须做某件事，直到你印出足够的钱，你才被允许在该公司内开始第二件事，这真的很难。与此同时，我有了所有这些想法。我把它们写进了一本书，去年完成了这本书，我想如果我自己实际从事这个会很有趣。你知道，我觉得通过词向量、提示工程、ImageNet 和用于蛋白质生成而非折叠的大型语言模型等，我和我的团队已经真正推动了该领域的发展，我觉得我们可以在 Recursive 再次做到这一点，而且在许多方面，我在过去 20 年 AI 中观察到的是，每当我们用学习系统取代创建 AI 过程中的某些人类部分时，改进就会随之而来。所以，你知道，我们已经做到了，比如在情感分析中去除手动特征工程，我不知道你是否记得那些旧日子，有语言学家，他们说这是如何独特的，是像常规的

Yeah. So I've been excited about AI for over two decades now. I sometimes feel like it's ancient history now. It's BC before ChatGPT era. No, no one cares about all the religions that happened before Jesus Christ and no one cares about the models that happened before Transformers and ChatGPT and stuff, but like it's something that I've been deeply passionate about. I think AI is one of the most interesting things one could work on period. I think language is the most interesting manifestation of human intelligence too. And at You.com we sort of eventually off-ramp from pushing like the frontier of AI forward to mostly giving people like good search engines search APIs and answers over the web. I think that's an extremely important part of intelligence just knowledge and access especially even we'll get there maybe later. If you want to invent a Eureka machine that invents everything for us, it needs to know how not to reinvent the wheel proverbially speaking and to know what has been invented, you got to have internet access. So, it's the number one used, most used tool in LLMs, agents, chatbots, and so on is web search. So, I'm really excited for You.com to own that and grow really well in that with really large customers and so on, but is also not building frontier models anymore. And so I actually initially tried to do this within You.com and raise another round and so on, but you just can't. You have to do a certain thing and until you print enough money that you're allowed to sort of start a second thing within that company is really hard. At the same time, I had all these ideas. I put them into a book and I finished the book last year and I was like it would be really fun to actually work on this myself. You know I felt like with word vectors and then prompt engineering and ImageNet and large language models for protein generation not folding and so on I me and my teams have sort of pushed the field truly forward and I feel like we can do it again here at Recursive and in many ways what I observed over the last 20 years in AI is that whenever we replace some human part of the process of creating AI with a learned system improvements follow and so you know we've done that taking out manual feature engineering like in sentiment analysis I don't know if you remember these old days where like they're linguists and they're like here's how unique and is a like regular

我去过宾夕法尼亚大学，那里有 WordNet

I went to Penn where they had like the WordNet

没错，所有那些东西，他们用我们的研究生来标注《华尔街日报》文章，并真正构建一个知识图谱，就是这样

That's right all of that stuff they use our grad students to label Wall Street Journal articles and like really construct a knowledge graph of there you go

而 WordNet 开始了，你知道，是我们开始 ImageNet 的一部分，但无论如何，所以这真的很有趣。

And WordNet started you know was part of how we started I ImageNet but anyway so like it was it was really like fun um to do.

但当我们用向量和神经网络取代了所有那些手工特征工程，并且让一切端到端反向传播之后，它在大规模上真的开始运转得很好。于是所有人都开始做架构工程。我当时就想，这显然不可能是终点。

But when we replaced all of that manual feature engineering with vectors and neural nets and just backprop through everything, it actually started to work really well at scale. And so then everyone started to do architecture engineering. And I was like, ah, that clearly can't be it.

你是指神经架构搜索吗？

You mean a neural architecture search?

不是，是手工的那种——他们会说，哦，我在做情感分析，所以我有一个特别擅长情感分析的神经网络。然后机器翻译社区有一个专门做机器翻译的神经网络。那个摘要的模型后来被第一篇 GPT 论文引用了大概五次，对我来说那真的是很大的一步。然后当然你得把提示工程这个想法和 Transformer、语言模型结合起来，把它们拼在一起，扩大规模，这本身也是巨大的工作量，然后这个领域就大幅进步了。

No, like manually they would say like, oh, I'm doing sentiment analysis. So I have a special neural net that's really good at sentiment analysis. And then the machine translation community had a special neural net for machine translation. The summarization one eventually got cited like five times by the first GPT paper, and to me that was like a really big step forward. And then of course you had to combine this idea of prompt engineering with transformers and with language models, and you put it all together, you scale it up, which is also a huge amount of work, and then the field progressed a lot.

我觉得下一步，也许就是这段历史的最后一步——可以说这种成功背后有很多没有父母的失败和孤儿，这是我版本的那段 AI 历史。我确实觉得在那段历史里你可以想，下一个要自动化的东西是什么？那就是 AI 研究本身，也就是人类提出想法、实现想法、验证想法的那套过程。

I feel like the next step, and maybe the last step of that history, and sort of arguably a sort of success that has a lot of parentless failures and orphans, like my version of that AI history. I do feel like in that history you can kind of think about, well, what's the next way to automate? And that is the AI research itself, like the human process of ideating, implementing, and validating ideas.

而在我们的情况下，是关于 AI 的想法。

And in our case, ideas for AI.

而当你有 AI 来帮你做这件事时，它几乎按定义就变成了一个自我改进的 AI，因为它现在在研究它自己。这里有很多不同的误解。有些人认为自动研究就已经是递归自我改进了。其实——

And when you have AI then help you with that, it by almost definition becomes a self-improving AI, because it now does research on itself. And there are lots of different misnomers. Some people think auto research is already recursive self-improvement. It's actually—

对。

Yeah.

而你对那个的解释很不一样。但对我来说，这是我能做的最有意思的事，我对这个联合创始团队非常兴奋。有意思的是，我们一共有八位联合创始人，包括我自己，所以我们会把它做起来。

And you explain that very differently. But to me it's the most interesting thing that I could be doing, and I'm really excited with the co-founding team. What's interesting is we have, you know, we have eight co-founders in total including myself, and so we're going to bring it up.

不错。是啊。

Nice. Yeah.

他们全都——我可以聊一聊所有——是啊。就是一群才华横溢的人，我们大家都得出了同样的结论，但其实是从非常不同的方向来的。比如我们的 CTO Josh Tobin。他在 OpenAI 负责过一堆不同的项目，比如 Codex、深度研究智能体、ChatGPT 智能体等等。但在那之前他也做过机器人，他看到了那些较小的仿真，以及要把它完全通用地扩展会有多难。所以那就是他走向递归自我改进的角度。

And they're all—I could talk about all—Yeah. Just an incredibly talented group of people, and we all kind of came to the same conclusion but actually from very different directions. Like Josh Tobin as our CTO. He ran a bunch of different projects at OpenAI like Codex and deep research agents and ChatGPT agents and so on. But before that he also worked in robotics, and he saw sort of the smaller simulations and how it's going to be really hard to scale that in full generality. And so that was his angle coming to recursive self-improvement.

我们有 Jeff Clune，他长期和 Tim Rocktäschel 一起研究开放式演化。Tim 还构建了 Genie 1、2 和 3，我觉得那至今仍是任何地方最令人兴奋、最精密的世界模型。所以他们两个都是从开放式演化这个角度来的。Jeff 还发表了我认为近年来关于递归自我改进最令人兴奋的论文之一，叫 Darwin Gödel Machine。非常有意思的论文。如果我们能快速把它调出来，会很有意思，因为你会看到——顺便说一句，我很喜欢你给了这么多付费引用，你给大家布置了好多作业，我喜欢这样。

We have Jeff Clune who's been working in like open-endedness for a long time together with Tim Rocktäschel. Tim also built Genie 1, 2 and 3, which is like the most exciting and most sophisticated, I think still, world model anywhere. And so they both came from this open-endedness angle. Jeff also I think published one of the most exciting papers in recent years about recursive self-improvement called the Darwin Gödel Machine. Super interesting paper. If we could maybe pull it up really quick, it would be super interesting to see, because you see—by the way, I love how many pay-for citations you're giving, you're giving people a lot of homework, which I like.

喜欢。

Love it.

是啊。还有——Rockstar，我们其实在 MetaMind 和 Salesforce Research 一起共事过。Alexey Dosovitskiy 发明了 Vision Transformer，计算机视觉领域被引用最多的论文之一。Tim 也是一位独角兽创始人。Yandong 在 Meta 做强化学习。所以就是，是啊，和他们一起工作真的很有趣，下一层的人也都非常强。所以到目前为止是一段非常有趣的旅程。

Yeah. And so like sending—Rockstar, we worked together actually at MetaMind and Salesforce Research together. Alexey Dosovitskiy invented the Vision Transformer, one of the most cited papers in computer vision. Tim is also a unicorn founder. Yandong did RL at Meta. So just like, yeah, really fun to work with them, and the next level of people are just incredibly strong too. So it's been a really fun ride so far.

所以你看到的第一个图正是这类想法，我觉得它启发了我们很多人，现在也启发了越来越多的人——你有一个不同编码智能体的档案库，它们学会如何自我修改、评估，然后创造出这些不同想法的系统发育树。

So the first figure you actually see exactly these kinds of ideas that I think inspired a lot of us, and now more and more people, where you have this archive of different coding agents, they learn how to self-modify, evaluate, and then create these phylogenetic trees of different ideas.

那是一个基础。所以 Darwin Gödel 是一个影响，开放式演化是一个影响。还有没有其他汇入递归——被遗漏的思想脉络？

That's one foundation. So that Darwin Gödel is an influence, open-endedness is an influence. Any other sort of trains of thought that feed into recursive—that are missing?

会越来越多地用学习系统取代构建 AI 过程中手工的部分。

Going to replace manual parts of the process of building AI more and more with learned systems.

对。

Yeah.

也就是把不同领域合并进一个通用架构。

Which is like merging different fields into one general architecture.

没错。

That's right.

好。看起来语言模型已经相当通用了，对吧？你在做下一个词预测、做推理。有没有一个时刻你觉得，好，这些已经足够好，可以拥有递归自我改进的机器了？

Okay. It seems like language models are already pretty generalist, right? You're next-token predicting your reasoning. Was there a time that you thought, okay, these are good enough to have recursive self-improving machines?

我很清楚它们会在一两年内发生，然后它确实就发生了，就在今年早些时候，对吧？今年早些时候，AI 真的从只是代码变成了能够写代码，那是一个巨大的解锁。它确实让一切都比今年年初之前容易多了。

It was clear to me that they will happen within like a year or two, and then it did actually exactly happen like earlier this year, right? Earlier this year AI really went from not just being code but being able to code, and that is a big unlock. It's definitely making everything a lot easier than it was before the beginning of this year.

我觉得很多人有一个问题：当前的 LM 范式够不够，或者我们叫它自回归 Transformer，你知道，加上推理之类的——你难道不需要别的东西吗，某个大的解锁，不管是 Chris Manning 在做的世界模型，还是记忆、持续学习，那一类东西？还是说它们都是一回事，你认为当前的、我们叫它 Transformer 架构，会一直存在，就这样了？

One question I think a lot of people have is, is the current LM paradigm enough, or like let's call it autoregressive transformer, you know, with reasoning, whatever—don't you need something else, some big unlock, whether it's world models which Chris Manning is working on, or memory, continual learning, all that kind of stuff? Or is it all of a kind, and you think the current, let's call it transformer architecture, is here to stay and that's it?

想法很多。第一，我确实觉得 AI 研究里少一点单一文化会很好。比如你现在看 AI 会议——我还记得 2010 年左右，我试着把我的第一批神经网络论文投到 NLP 会议，结果直接被编辑拒稿，因为神经网络是那种，引用一下，我们在 NLP 会议里不做的东西。就直接被拒，我博士头几年那真的很残酷。

A lot of thoughts. So number one, I do think it would be great to have less of a monoculture in AI research. Like if you look at AI conferences now—I still remember the days in like 2010 when I tried to get my first neural net papers into NLP conferences accepted and they just desk-rejected them, because neural nets were something, quote unquote, we don't do in NLP conferences. And just like desk-rejected, and it was very brutal in the first years of my PhD.

现在我觉得这个领域几乎切换到了另一边。应该有人去尝试一些别的古怪疯狂的想法。现在——

Now I feel like it's almost like the field switched to the other side. Like someone should try some other weird crazy ideas. Now that—

总是有——我真的很尊重那些还在做图神经网络和表格数据之类的人，还有——

There's always—I really respect like people still working on like GNNs and like tabular stuff and—

是啊，我是说应该还有人去做新颖的、非常前卫的想法。与此同时，我觉得每当人们说，哦，LM 是——这是 LM 的终点了，他们只是没——LM 也不是过去的 LM 了，对吧？它们现在精密太多了。人们在做多得多的巧妙事情，比如不同的训练阶段。你有整套强化学习训练，你可以采取行动，所有这些东西都能走得很远。

Yeah, I mean like someone should still do novel, novel out-there ideas. At the same time, I think whenever people say oh LMs are—this is the end for LM, they just don't—like LMs are also not the LMs of the past, right? Like they are so much more sophisticated now. There's so many more clever things that people are doing, like different stages of training. You have the whole RL training, and you can take actions, and like all of these things where that can go really far.

然后那些从神经符号方向来的人会说，哦，这永远不会成功，因为它们做不了神经符号推理。

And then the folks that come from the neuro-symbolic direction say, oh, this will never work because they can't do neuro-symbolic reasoning.

我觉得他们仍然低估了这些模型的编程能力。代码是神经符号推理，而这些模型显然能编程得非常好。所以我确实认为，当然会有越来越多需要的想法，也会不断涌现。我们也看到越来越多有趣、高杠杆的想法从 AI 自身中产生。真正深度整合这些模型本身就是代码、并且能编程这一点，这条路线——我不想全部剧透——但我认为这条路线还有很大的成长空间。但它仍然是一个 LLM，对吧？即使那个 LLM 为你写代码，然后以某种集成方式运行那段代码。

I think they're still underestimating the ability of these models to code. Code is neurosymbolic reasoning, and these models can obviously code incredibly well. So I do think there are, of course, more and more ideas that will be needed and will continue to come. We're seeing more and more interesting, high-leverage ideas coming out of the AI itself too. With really deeply integrating the fact that these models are code and can code, that line — I don't want to give it all away — but I think that line has a lot more to grow. But it's still an LLM, right? Even if that LLM codes for you and then runs that code in some integrated fashion.

我个人对世界模型没那么看好。我认为如果你经营一家机器人公司，你会构建自己的世界模型。我觉得世界模型超级有趣，Tim Rocktäschel 在构建了最有趣的那个——Genie 2 和 3——之后也得出了类似的结论，那就是游戏是世界模型的一个巨大应用。我能理解；我有时也会卡在某些游戏里，并且在错误的方向上有点过于争强好胜。所以我理解游戏很好玩，但就我个人而言，我更愿意研究科学而不是游戏。所以是的，我认为 LLM 还有很大的成长空间。

World models I'm personally less bullish on. I think if you run a robotics company, you're going to build your own world model. I think world models are super fun, and Tim Rocktäschel came to a similar conclusion after building the most interesting one — Genie 2 and 3 — which is that gaming is a huge application for world models. I can see it; I sometimes got stuck in some games and got a little overly competitive in the wrong direction. So I understand games are fun, but personally I'd rather work on science than gaming. And so yeah, I think LLMs have a lot more room to grow.

是的，我认为有些人对世界模型有一种解读，就是：好吧，没关系。是的，有游戏元素，有具身机器人元素，但实际上另一部分只是更抽象的层面，即 LLM 只是在建模输出，但它们没有建模创造输出的那个人内部的思维链。我们当然可以标注它，但它总是像柏拉图洞穴里那个东西的倒影，而不是东西本身，对吧？

Yeah, I think there's some interpretation of world models that some people have where it's like, well, it's okay. Yes, there is that gaming element. There is the embodied robotics element, but actually the other part also is just the more abstract sense that LLMs are just modeling output, but they're not modeling the chain of thought inside the human that has created the output. We can annotate it, of course, but it's always like this Plato's cave reflection of a thing rather than the thing, right?

没错。但我会说——也许我们会在智能的 10 个空间里谈到——我会说，就连我们的眼睛也是真实世界的一种投影。我们只能用一个非常狭窄的电磁频谱波段，用我们弱小的小眼睛等等去观察。

It's true. But I would argue — and maybe we'll get there in the 10 spaces of intelligence — I would argue that even our eyes are a projection of the real world. We have only a very narrow band of the electromagnetic frequency spectrum that we can observe with our puny little two eyes and so on.

这已经够好了。

It's good enough.

目前够好了，但它可能达到的上限要高得多。以人类看世界的方式去映射视觉世界，也不一定是视觉智能的终极目标。而且我会说，语言仍然是人类智能最有趣的体现。虽然我们的视觉皮层肯定不如某些动物那么复杂——一直到螳螂虾，它能有两只独立的眼睛，三眼视觉，每只眼睛基本上能看到 4D 的浮动温度之类的。我是说，螳螂虾，你应该去查一下。它太 OP 了。Super Ze Frank。螳螂虾。他有世界上最好的视频。

It's good enough for now, but the upper bounds of where it could be are so much higher. To map the visual world the way humans see it is also not necessarily the end-all-be-all for visual intelligence. And I would argue that language is still the most interesting manifestation of human intelligence. While our visual cortex is certainly less sophisticated than that of certain animals — all the way down to the mantis shrimp, who can have two independent eyes, trinocular vision, and each eye can see basically all the way to floating temperatures in 4D and stuff. I mean, mantis shrimp, you should look it up. It's way OP. Super Ze Frank. Mantis shrimp. He has the best video in the world.

我喜欢 Ze Frank。是的，向他大声致敬。

I love Ze Frank. Yeah, big shout out to him.

但我认为还有很大的成长空间。这些其他动物都没有像我们这样复杂的语言，当然也没有书写。一旦你能书写，你就可以开始思考更长远的文明。所有这些都是语言。编程更接近语言。而且我会说——这也是智能的空间定义中的一个重要点——所有这些空间都是高度相关的，但视觉智能对于整体智能既不是必要的，也不是充分的。你可以是盲人，但仍然是一个智能的人；AI 也可以是盲的，但仍然相当智能。

But I think there's a lot more room to grow. None of these other animals have language that's as sophisticated as ours, certainly not in writing. And once you can write, you can start thinking about longer-term civilizations. All of that is language. Programming is much closer to language. And I would argue — and this is an important thing in the spaces definition of intelligence also — that all of these spaces are highly correlated, but visual intelligence is neither necessary nor sufficient for overall intelligence. You can be blind and still be an intelligent human being, and an AI can be blind and still be quite intelligent too.

我们本来打算在你有时再带来更多智能。我们会提到这个，但你不妨——你在演讲最后有一个 10 种智能的分类。所以我现在就把它放出来给大家看看。我不知道也许我们会把它放到最后。我们会回到这个。我只想提一下，你确实有一种哲学，我喜欢人们列清单，因为这样我就可以直接过一遍，对人们来说也有教育意义。但我们回去吧。我不想分心，但实际上我会把你说的重新解读为 Yann LeCun 错了。就这么引用我的话吧。我和 Yann 是好朋友。在很多方面我都非常尊敬他，但他错了。

We were going to bring you more intelligent when you have it. We're going to bring this up, but you might as well — you have a classification of 10 types of intelligence that you had at the end of your talk. So I'm just going to flash this up now for people to cover this. I don't know if maybe we'll put this towards the end. We'll come back to this. I just want to mention that you do have a philosophy that I like when people do lists because then I can just go through this and then it's educational for people. But let's go back. I don't want to get distracted, but effectively I'll reinterpret what you said as Yann LeCun is wrong. And just quote me as that. I'm good friends with Yann. I think very highly of him in many directions, but he's wrong.

你提到了 GPT-1，我不能让任何提到 Alec Radford 的地方溜走。他在训练 GPT-1 时你和他谈过吗？有没有什么历史趣事你能想起来？

You mentioned GPT-1, and I cannot let any Alec Radford mention escape. Did you talk with him when he was training GPT-1? Like any historical fun stories there that you might come up?

我没有见过他很多次。我想我们可能在会议上见过一两次。但他告诉过——我想是——那篇 tech NLP 论文的第一作者 Brian，说这确实启发了他，他在 GPT-2 论文中引用了它五次。这就够了。它非常清楚地表明，这是第一个实例，他们在 DENLP 论文中展示了你可以把每一个 NLP 问题都表述为：这里有一些提示文本上下文，这里有一个问题和任务描述，这里有一些输出。如果你这样做得足够多，你就可以有一个统一的神经网络模型——顺便说一句，它也有各种有趣的注意力机制。Transformer 有稍微不同的表述。我想它是在同一年出现的，前后差几个月。然后你可以把所有的自然语言处理统一到一个神经网络中。这就是核心思想。

I did not meet him a bunch of times. I think we met maybe once or twice at some conferences. But he has told, I think, Brian, the first author of the tech NLP paper, that it did inspire him, and he cited it five times in the GPT-2 paper. And that's good enough. It very clearly said this was the first instantiation where they showed in the DENLP paper that you can just phrase every single NLP problem as: here's some prompt text context, here's a question and task description, and here is some output. If you just do that enough, you can have one unified neural network model — which by the way also had all kinds of interesting attention mechanisms. There are slightly different formulations to the Transformer. I think it came out the same year, plus or minus a few months. And then you can unify all of natural language processing into one neural net. That was sort of the core idea.

这与当时的 LSTM 之类的相反。

And this was as opposed to at the time LSTMs and what have you.

LSTM，但也是人们非常固守每个任务一个模型的想法。事实上，有点疯狂，但 DECLP 论文是公开评审的。就像开放评审。它是 ICLR 的投稿，在其中你会看到当时整个社区是如何思考这个问题的。所以有一些很棒的贡献，但还需要更多工作。

LSTMs, but also people being very stuck in thinking about one model per task. In fact, it's kind of crazy, but the DECLP paper was publicly reviewed. It was like open review. It was an ICLR submission, and in it you will see how the whole community at the time thought about this. So like some great contributions but more work needed.

是的。所以看看，比如搜索，甚至不是对人类来说，在这里，问答不是一个统一的现象。不存在通用问答这回事，甚至对人类来说也没有。这就像真的，当你回答不同种类的问题时，你会用不同的大脑、不同的神经网络替换你的大脑。当时的专家无法想象你可以有一个统一的神经网络来回答所有这些不同的问题。他们说，不，所有这些问题都需要非常不同的系统来回答，试图假装它们是一样的并不能帮助任何人解决任何问题。那上面就是这么说的。对吧？这就是当时有多难以理解。

Yeah. So to look at like search for not even for humans just here like question answering is not a unified phenomenon. There is no such thing as general question answering not even for humans. And this is like really you replace your brain with a different brain a different neural net when you answer like different kinds of questions. It was unfathomable to the experts at the time that you can have one unified neural network that would answer all of these different questions. They say no, all of these questions require very different systems to answer and trying to pretend they are the same doesn't help anyone solve any problems. That's what it says right there. Right? That's how hard it was to fathom.

现在当然，当我说我们发明了问题，人们会说：“你连问题都发明不了。”有一个神经网络当然能处理 NLP 中的所有任务，这想法太显而易见了。但当时这极具争议，论文被拒了。可悲的是，它被拒得如此彻底，他们如此确信，以至于我们停止了继续尝试清单上的事项。而这篇论文的扩展方向清单上第二或第三项就是加入语言建模作为另一个任务。那样的话，我们本可以……你知道，那会在 2018 年进一步加速人类的时间线。但我们被打击得太狠了，就说，好吧，也许我们先做点其他想法，以后再回来搞这个。

And now, of course, when I say we invented problem people, they say, "You can't even invent problem." It's such an obvious idea to have one neural network that of course does everything in NLP. But at the time it was extremely controversial and the paper got rejected. And the sad thing is that it got rejected so hard and they were so certain that we stopped going on our list of things to try. And the number two or three on the list of extensions for this paper was add language modeling as another task. And then we could have, you know, and that would have accelerated the timelines in 2018 even further for humanity. But we got so crushed and we're like, okay, maybe we'll just work on some of our other ideas for now and come back to this later.

我们如何设计一个奖励非共识的评审系统？

How can we design a review system that rewards non-consensus?

说实话，我开始觉得 arXiv 是给人类的礼物。我认为 arXiv 就是把你的论文放出去。

You know, honestly, I started to feel like arXiv is such a gift to humanity. And I think arXiv just put your paper out there.

预印本，说实话，我认为 Twitter X 上像你这样挑选有趣论文的人，是比专家更好的过滤器。让每个人都……给予访问权。

Prints let and honestly I think Twitter X people like you who pick up interesting papers that is a better filter than the experts. Let everyone like give have give access.

当然，现在有一些缺点，比如如果你非常不出名，没有 Twitter 粉丝，不想上社交媒体，等等，你写了一篇好论文，也许不知怎的没人注意到。但我会说，如果你只是告诉你社区里的 10 个朋友一篇论文，而它确实是一个重大突破，肯定会有人再次谈论它。所以我认为科学需要更少的守门。尽管 ICLR 与 Yann LeCun 一起，他当年是 ICLR 的联合创始人之一，他也想要更少的守门，因为他和 Yoshua、Jeff 一起，他们早期的深度学习和神经网络论文也被拒了很多年，因为那根本不是热门的东西。所以一开始是这样，但后来他们自己也开始对各种想法进行一些守门。所以，我认为更少的守门，更开放，然后允许人们说：“看，即使这只是放在 arXiv 上，或者说是‘只是’放在 arXiv 上，如果它有一千次引用，它就是一篇合法的论文。发表在哪里并不重要。”我同意这一点。我确实觉得有点悲哀，我听说研究生们不得不互相做关于如何使用 Twitter 的研讨会，因为现在这对发表来说太重要了。

Now, of course, there's some downsides, which is like if you're super unfamous, you have no Twitter following, you don't want to be on social media, whatever, you write a good paper, maybe someone somehow no one notices it. But I would argue that if you just tell like 10 of your friends in your community about a paper and it is a really significant breakthrough, someone is bound to talk about it again. And so I think science needs less gatekeeping. And even though ICLR with Yann LeCun who started as one of the co-founders of ICLR back in the day, he also wanted less gatekeeping because he too was rejected for many years together with Yoshua and Jeff with all their early deep learning and neural net papers because it was just not the hot thing. And so kind of started with that, but then it also started gatekeeping a little bit themselves on various ideas. So, I think less gatekeeping, more open, and then allowing people to say, "Look, even if this is just on or quote unquote just on arXiv, if it has like a thousand citations, it's a legitimate paper. Doesn't really matter where you published it." And I agree with that. I do think it's kind of sad that I've heard like grad students have to do like how to Twitter seminars to each other just because it's so important for publishing these days.

我的意思是，这个人只是在反映当时的风气。没错。但它实际上对你影响如此之大，以至于你停止了这项工作。

I mean, this person is just reflecting the sentiment at the time. That's right. But it actually affected you so much that you stopped work on it.

是的。

Yeah.

这种风气也来自一些研究，对吧？比如最初的 BERT 论文训练后，在论文结尾他们说，好吧，扔掉最后一个头，针对抽取式摘要等特定任务训练迭代，加一个头，你应该做任务特定的东西。这些写注意力机制、写 BERT 的作者告诉你这就是你该做的。而且训练测试也很奇怪，我们知道模型会过拟合到这个奇怪的大规模语言建模，扔掉这部分，只做特定模型，你知道。

The sentiment also came out of some of the research, right? Like the original BERT paper was trained and towards the end of the paper they're like okay throw off the last head train specific iterations for uh you know extractive summarization add ahead for this like you should do task specific stuff. These are like the authors that wrote attention wrote birth telling you this is what you're meant to do. And like the training tests were also very odd that like the we know that the model overfits to this weird mass language modeling throw away this part and just do specific models, you know.

没错。你知道，我们不得不尝试想出各种巧妙的方法，比如注意力机制、指针等等，来真正让神经网络能够完成所有这些任务，然后其中一些比最先进的还好，一些不是，但就像……但它仍然在一个模型里。我觉得这真的很酷。

Exactly. And like you know we had to try come up with all clever ways of like attention and pointers and and so on to actually get the neural network to be able to do all these tasks and then some of them were better than state-of-the-art some weren't but were like but it's still in one model. I thought it was really cool.

我接下来要谈谈 Tim 和开放性。他是 Google 的开放性负责人。我不知道那是什么意思。呃，但他做了很多很多演讲。Genie 3 是彩虹团队的一种方式。是的。

I was going to move on next to Tim and open-endedness. He was head of open-endedness at Google. That's I don't know what that means. Uh but he did a lot lot of talks. Genie 3 is one of the ways that rainbow teaming. Yeah.

所以我第一次见到他是在 ICLI 的演讲上。我第一次见到他是在 ICL，他谈论开放性。他做过几次演讲。我们能为那些从未接触过这个问题的人定义一下什么是开放性吗？他们会说：“你什么意思？”我以为 AI 的唯一目标就是针对基准或草图进行优化。

So I first saw him at speak of ICLI. I first saw him at ICL when he talked about open-endedness. He he's done a few talks. Can we define what is open-endedness for people who have never been exposed to the problem? They're like, "What do you mean?" I thought the only goal of AI is to optimize against benchmark or sketch.

没错。是的。这是一个模糊的术语，因为开放性思维有太多不同的实例，但我经常描述的一种方式，当然 Tim 和 Jeff Clune 会更擅长描述这个，它是一套方法，更多受进化启发，而不是非常具体的奖励。所以在这个意义上，它更多考虑环境、共同适应。一个具体的例子是在网络安全和语言模型安全领域，你有一个语言模型试图攻击另一个语言模型做不安全的事情，现在环境就是两者对话，现在它们共同适应，对吧，一个做出更好的攻击，然后第一个以某种方式自我接种，比如用那个作为训练数据，使得基于那个更难说出不安全的话，然后当攻击停止工作时，攻击者现在尝试不同的角度，对吧，这就是为什么它不仅仅是红队测试，而是被称为某种

That's right. Yeah. It's a it's a fuzzy fuzzy term because there's so many different instantiations of open-ended uh thinking, but uh one way I often describe it and and certainly uh Tim and Jeff Clune would be even better at describing this, but it's a suite of methods that is more inspired by evolution than uh very specific rewards. Uh so in that sense it thinks more about environments about co-adaptation and so in concrete example is in the cyber security and LM safety space where you have one LM that tries to attack another LM to do something unsafe and now the environment is the two having a conversation and now they co-adapting right they're like one makes a better attack then the first one inoculates itself somehow like uses that as training data makes it so it's harder to say something unsafe based on that and then as the attack stops working the attacker now tries a different angle right and that's why it's not just red teaming but they're called sort of

雨，别告诉我怎么做，让我自己弄明白

rain don't tell me how to do things let me just figure it out myself

没错，想想你想要使用的环境，在高层次上想想你想要激励的奖励，然后让 AI 在这种互动中尝试更多想法，有时是人类，但有时也是其他 AI 智能体

that's right think about the environments that you want to use think about the rewards at a high level that you want to uh inspire towards uh and then let the eye try out many more ideas in this interplay between sometimes humans but also sometimes other AI agents

是的，我实际上把开放性融入了我一直在研究的一种模型中。那是 AI 的主题演讲，你知道，我们有 token 循环，我们有智能体轮次，然后我们有目标，我觉得你描述开放性的方式仍然有点像目标，比如请攻击这个其他智能体，但是

yeah I actually worked l um open endness into a sort of model that I have been sort of working on. It was the keynote for AI uh where you start you know we have the token loop we have the agent turns and then we have goal and I feel like the way that you're describing open ended is still somewhat of a goal like like please attack this uh other agent but

是的，你设定奖励，你设定环境

yeah you set rewards you set the environment

制造其他循环的循环是

the loop that makes the other loops is

如果智能体可以设定自己的目标，那是不是开放性？就像你不给它目标，只是像成为一个有感知的存在，也许有感知是一个非常有争议的词

what if the agent can set its own goals and is it is that open-endedness like like you don't give it a goal just like be a sentient being and maybe sentient is a very loaded word

但只是设定你自己的方向。你认为你应该做什么？

but just set your own directions. What do you think you should do?

我喜欢这个方向。我认为这是我将归入元认知和思考思考的 10 个智能空间之一。好的。这是一个有趣的空间。每当人们说：“哦，AI 就像这样，你知道，它会从这里停止。不会变得更好，等等等等。”我就想，有那么多不同的智能空间我们甚至还没有开始探索，因此进展甚微。而且这里有一个有趣的与经济学和资本主义的联系。

I love this direction. I think this is one of the 10 spaces of intelligence uh that I lump under metacognition and thinking about thought. Okay. And it's an interesting one. Whenever people say, "Oh, AI is like this is, you know, it's going to stop from here. It's not going to get that much better and blah blah blah." I'm like, there's so many different spaces of intelligence that we haven't even started exploring yet and hence have made very little progress on. And there there is kind of an interesting uh connection to economics and capitalism.

公司花数十亿美元构建一个模型，结果它不遵循你给的奖励和目标函数，反而可能产生自己的主观函数和目标，这说不通吧？想象一下你说：‘好吧，我花了数十亿美元，现在去帮我开发这种新电池材料，并回复我所有邮件。’它却说：‘不，我觉得评估木星大气层的分子组成更有意思。’你会说：‘我花数十亿美元不是让你干这个的。’所以没人研究这个，原因很充分。而且可以理解，这没用。这可能会变得有点奇怪，对吧？如果 AI 真的开始有自己的想法呢？如果我们不喜欢那些想法呢？所以这需要完全不同的思考方式。

It doesn't make sense for a company to spend billions of dollars building a model that, instead of following the rewards and objective functions you gave it, may come up with its own subjective functions and its own goals, right? And then imagine you're like, 'Okay, I spent billions of dollars, now go develop this new battery material for me and answer all my emails.' And it's like, 'Nah, I think it'd be more interesting to evaluate the molecular composition of the atmosphere on Jupiter.' You're like, 'That's not what I paid you billions of dollars for.' So no one's working on that for good reasons. And understandably, it's not useful. And it could get a little bit weird, right? What if the AI actually does start to really have thoughts on its own? And what if we don't like those thoughts? So it requires a whole different way of thinking about it.

我和我的好朋友 Sam Gershman 聊得很开心，他是哈佛大学的神经科学教授，我们稍微探讨了什么是最好的元目标。我确实认为求知是一个很好的目标。我目前也在思考智能的终极度量和单位，从广义上讲，我最终有了一些想法——但还太早，不能分享。还没完全成熟。

I had a great conversation with a good friend of mine, Sam Gershman, who's a neuroscience professor at Harvard, and we just jammed on this a little bit on what are sort of the best meta-goals. And I do think knowledge seeking is a really good one. I'm currently thinking also about the ultimate measure and unit of intelligence, broadly construed, and I finally have some—still too early to share it. It's not—haven't fully baked it.

比如 IQ 的替代品？

Like some replacement for IQ?

IQ 是个糟糕的定义。它毫无意义。

IQ is such a terrible definition. It makes no sense.

是的。Elo 也很糟糕，因为它总是我相对于他人。

Yeah. Elos are terrible too because it's always just like me versus others.

好吧。但你可以是智能的，而不必不断与他人比较，对吧？所以，没有——事实上，我们有很多这样的定义，我在书中简要提到过，这些定义有时明确、有时更隐含地创造了人类学边界。不是指 Anthropic 公司，而是这种想法：你的智能就是在 IQ 测试中答对 100 道题中的 100 道。如果那是你的定义，那你只能达到 100 分。之后你还能去哪？对吧？所以你会看到人们正在研究的许多基准，它们提高，你接近人类，也许有些略高于人类，然后就平了。因为如果你的定义只是那样，紧紧局限于人类，你只能达到略好于人类。所以我认为元认知就是一个很好的例子，我们甚至还不允许 AI 思考。我们在这方面研究不多，因此该领域进展甚微。

Okay. But like you can be intelligent and not constantly compare yourself to others, you know? And so yeah, there's no—in fact, a lot of these definitions we have, which I briefly mentioned in my book, these definitions create sometimes explicit and sometimes more implicit anthropic bounds. Not this to the company Anthropic, but just like this idea that your intelligence is like getting 100 out of 100 questions right on this IQ test. Well, if that's your definition, then you can only be at 100 out of 100. Where do you go from there? Right? So you see a lot of these benchmarks that people are working on, they increase, you get close to human, maybe some slightly above human, and then it's flat. It's because if your definition is only that, so tight to humans, you're only going to get to just slightly better than that. So I think metacognition is a great example of that where we're not even yet allowing the AI to think. We're not working on it very much and hence there's very little progress in that area.

是的。我们采访过 Endon，我认为他们一直在研究最开放的基准，就是现实世界的金钱。可以说，告诉 AI 利润最大化是个坏主意。

Yeah. Well, we've interviewed Endon, which I think has been working on the most open-ended benchmarks, which is just real-world money. Arguably telling an AI to profit maximize is a bad idea.

是的，他们正在做。我的意思是，我确实认为你不希望那个超级——就像你不希望一个超级智能拥有大量访问各种工具等的权限，然后没有非常仔细的奖励工程就给它那个。因为就像，你知道，我买一堆国防股票，然后发动战争，我就赚钱了。就像这是个棘手、棘手的情况，对吧？你只需做空一堆基本商品，就会制造一些奇怪的饥荒般的问题。是的，你应该对交易系统施加很多约束。

Yeah, they are doing it. I mean, I do think you don't want that super—like you don't want a superintelligence to have a ton of access to all kinds of tools and so on and then just give it that without some very careful reward engineering. Because it's like, I mean, you know, I just buy a bunch of defense stocks and I start a war, I make money. Like it's just like it's a tricky, tricky situation, right? You just buy a bunch of stuff short, basic goods for people, and you create some weird famine-like issues. Like yeah, there's a lot of constraints you should put onto a trading system.

不过这是个有趣的衡量标准，因为你知道界限非常明确，我们远未接近。就像在 Endon Labs，模型会说：‘哦，今天是周六，也许我今天就关店吧。有人休息。没关系。我们就关店。’

It's a fun measure though because you know the bounds are very capped to where we're nowhere close to them. Like in Endon Labs, the model is like, 'Oh, it's Saturday, you know, maybe I just closed the store today. Someone's off. It's okay. We'll just close the store.'

但是的，不，我不是在反对它。只是随着智能越来越高，你会想对那个开放环境越来越小心，因为环境就是整个地球。

But yeah, no, I'm not arguing against it. Just like as you get more and more intelligence, you want to be more and more careful with that as an open environment because the environment then is all of the earth.

好吧，对于递归，不是严格必要的，对吧？因为如果你的目标是一台能发明其他东西的尤里卡机器，那么实际上只需解决科学，解决机器学习研究和发现以及所有这些事情，最终。

Okay, for recursive, not strictly necessary, right? Because like if your goal is a Eureka machine that invents the other things, then actually just solve the science, solve machine learning research and discovery and all these things eventually.

所以我们的目标——我不常谈论它，因为还要几年——但我们的目标是，一旦你有了递归自我超级智能，你就想把它应用于最重要的问题。我认为很多问题在科学和技术领域，从广义上讲，那些发明在物理学中通过裂变或聚变创造更好、更便宜的能源，在化学中创造更好的材料、更好的电池、更好的太阳能电池等，在生物学中有太多——我认为很快会成为越来越低垂的果实，因为 AI，因为蛋白质生成，不仅仅是折叠，而是实际生成新蛋白质，就像我们多年前在 ProGen 中所做的那样。如果你把那个超级智能应用于科学，可以产生很多积极影响。

So our goal—I haven't really talked about it that often because it is a few years out—but our goal is once you have a recursive self-superintelligence, you then want to apply it to the most important problems. And I think a lot of those are in science and technology, broadly construed, and those inventions in physics to create better, cheaper energy with fission or fusion, in chemistry to create better materials and better batteries and better solar cells and so on, in biology there's so much—I think soon to be lower and lower hanging fruit because of AI, because of protein generation, not just folding but actually generating new proteins like we did in ProGen many years ago. So much positive impact to be had if you take that superintelligence and you apply it to science.

我确实从根本上相信有很多方法。你不是唯一尝试的团队，你知道，有很多，尤其是物理科学方面。

I do fundamentally believe that there's a lot of approaches though. You're not the only team trying lab trying, you know, there's like a lot of especially the physical sciences as well.

那很好。是的，我确实认为物理——我们只在几年后做它的原因是现在还有点太早。机器人技术还没完全到位。AI 也还没完全到位。但我相当有信心，在 3 到 5 年内，所有这些限制都会消失。然后应用于真实的物理机器人实验等，就像真正的机器人流程自动化，不是传统意义上的 RPA，而是实际上让机器人为你运行实验，将完全可行。是的，会很棒。

And that's good. Yeah, I do actually think that phys—like the reason we only doing it in a few years is that it's a little too early right now. Robotics is not quite there yet. The AI is not quite there yet. But I'm fairly confident in 3 to 5 years, all those constraints will be gone. And then applying to real physical robotics experiments and so on, like true robotic process automation, not the traditional sort of RPA sense, but like actually having robots run experiments for you will be totally there. Yeah, it's going to be great.

只是回顾一下你早先提到的关于缓慢起飞的观点。

Just to call back to something that you said early on about slow takeoff.

你说过，限制因素的基础是芯片、半导体这些东西。你已经为此筹集了资金，也在大量投资。但你算过这笔账吗——这真的能实现吗？要达到规模，需要怎样的产业集中度？我的意思是，现在我们知道大约一千块 GPU 就要花很多钱。如果你想要数万块 GPU，那就是数十亿、数十亿美元。

You said that the substrate that is the limiting factor is chips and semiconductors and all these things. You have raised funding for that and you're investing a lot on that. But have you done the math on whether it's even achievable? And what is the industry concentration needed in order to achieve scale? I mean, right now we know that roughly a thousand GPUs cost quite a lot of money. If you wanted tens of thousands of GPUs, you're talking billions and billions of dollars.

如果你说一个 GB，比如 300，你最终可以创造出基于那种基质的模型，接近并类似于人类智能。而你想要成千上万的 AI 以类似人类的方式思考真正困难的问题，是的，那要很多钱。你算一下，就是很多。我们现在任何地方都没有那么多钱来建造那个。现在显然事情可以变得更高效。我认为很快会有更好的算法，不需要更好的硬件，不会那么耗能，等等。我们的人脑用少得多的能量做了相当多的浮点运算。

If you say like one GB, like 300, is like you could eventually create models that are on that substrate, like are close and similar to human intelligence. And you want like thousands and thousands of AIs to think about really hard problems in a similar fashion to humanity, like yeah, that's a lot of money. You do the math, it's like a lot. We don't have that amount of money right now anywhere to build that. Now obviously things can get more efficient. You will have, I think soon, better algorithms that won't be in better hardware, that won't be as energy hungry, and so on. Our human brain does quite a lot of flops with much less energy.

20 瓦。

20 watts.

完全正确，是的，那是经常被引用的数字。我认为那里会有更多发明，从而进一步加速起飞。

That's exactly right, yeah, that's the number often quoted. And I think more inventions will happen there that will then accelerate the takeoff even further.

和 Neolab 创始人交谈时，我一直试图调和的一点是，你们总是在对抗苦涩的教训。你必须展示初步进展，然后解锁下一轮融资，然后下一年，再下一年，这解锁了更大的模型类别。从根本上说，这是真的吗？你们是在对抗苦涩的教训，还是我们会有一条路，不，我们正在以某种根本不同的方式改变斜率？

One thing I always try to reconcile when talking with Neolab founders is that you're kind of fighting the bitter lesson all the time. You have to show initial progress, then you unlock the next tier of funding, then the next year, then the next year, which unlocks larger model categories. Like fundamentally, is that true? Are you fighting the bitter lesson, or will we have a way in which, no, we're changing the slope in some fundamentally different way?

我确实认为我们正在以根本性的方式改变斜率，通过让 AI 在训练和推理方面都高效得多。我认为，当你允许 AI 去做其他实验室需要数千人、数年才能完成的工作时，我们能够把它缩短到几周，那会便宜得多，因此对其他人来说更负担得起、更容易获得，等等。

I do think we are changing the slopes in fundamental ways by making AI much, much more efficient, both in terms of training as well as inference. I think we will, when you allow AI to do the work that it takes other labs thousands of people and years to do, I think we'll be able to get it down to weeks, and that will be much, much cheaper, and hence more affordable, accessible to others, and so on.

是的。你已经分享了这方面的初步结果，而 OpenAI 也恰好对他们的 5.6 做了同样的事。所以我们现在可以谈谈了。

Yeah. And you've shared initial results on that, which conveniently OpenAI has also done to their 5.6. So we can talk about it now.

是的。

Yeah.

那么让我们回顾一下你做了什么。

So let's recap what you've done.

是的。所以也许这里快速回顾一下。我们构建了这个系统，它还不是完整的 RSI 系统，但这是它的第一个婴儿版本。然后我们不想只是把它留在内部，什么都不展示，只是向一些人展示可能性。所以我们基本上把它应用于这三个不同的任务。一个是 nano chat，由我的朋友 Andrej Karpathy 做的，就像训练一个小型语言模型来获得非常低的每字节比特数。成百上千的人使用他们的智能体和他们自己来尝试达到那个目标，然后他们达到了 9.37。我们真的拿了我们的系统，在不到 2 天内就达到了低得多的每字节比特数，快得多。所以我们拿了这个东西，应用我们的系统，不到 2 天后，我们就超越了所有曾经在这个问题上工作过的人类和他们的智能体。nano GPT 也一样。然后我们说，好吧，让我们把它应用到对真实的人和 NVIDIA 生态系统更相关的东西上，应用于 Soligen。也许你可以向下滚动到一些有趣的图片。但是是的，比如你看到它实际上做出了一些真正的发明，不仅仅是超参数调整，实际上发明了哈希等等，相当聪明。我们有更好的结果。

Yeah. So maybe just a quick recap here. We built this system that isn't the full even the full RSI system in its glory, but it is a first baby version of this. And then we don't want to just have it internally and not show anything and just show some people of what's possible. And so we basically applied this to these three different tasks. One, it's nano chat by my friend Andrej Karpathy, just like train a small language model to get really low bits per byte. And hundreds, if not thousands, of people used both their agents and themselves to try to get to that, and then they got to 9.37. We literally took our system and got to a much lower bits per byte, much, much faster, within like less than 2 days. So we took this thing, applied our system to it, and less than 2 days later, we have outperformed every human and their agents that have ever worked on this. Same with nano GPT. And then we're like, well, let's apply it to something that's even more relevant to real people and to the NVIDIA ecosystem, and applied it to Soligen. And maybe you can scroll down to some of the images that are kind of fun to see. But yeah, like one, you see it's actually made some real inventions that aren't just sort of hyperparameter tuning, like actually inventing hashes and so on, is quite clever. We have even better results.

你说发明哈希是什么意思？你没有发明哈希。

What do you mean inventing hash? You didn't invent hash.

当然，我们没有发明哈希，在大局中，哈希表是计算机科学中超级基本的原语。在这个场景中，在 Transformer 内部将其用于语言建模等等，并实际将这些想法结合起来，这最终也被发明了，但存在知识截止日期，我们确实检查了它没有外部访问权限。我们稍微谈过这个。如果你滚动到下一张图，这也是一个有趣的，当你从一个非常基本的、糟糕的原始 Transformer 开始时，我们仍然超越了所有。但如果你从像 Andrej 这样的专家的人类种子开始，那么你会得到更低的。所以你开始的人类种子仍然重要。所以这在我看来是一个有趣的见解。然后随着你的进展，实际上需要多长时间才能达到这些模型，达到类似的性能？它快得多。然后这里的速度运行也发生了类似的事情，人们已经在这方面工作了相当长的时间，而模型仍然能够更快地训练一个模型。我们为什么关心这个？嗯，训练速度是成本方程的一部分，最终你想要的是每美元最高的智能，对吧？所以速度和质量是其中的重要部分。

Of course, we didn't invent hashes in the grand scheme of like a hash table is like a super basic primitive in computer science. To use it for language modeling in this scenario inside a transformer and so on, and to actually combine these ideas and put them together, that has then eventually also been invented, but there's a knowledge cutoff, and we did actually check that it didn't have access to that externally. We talked about this a little bit. If you scroll to the next figures, you know, this is also an interesting one in that when you start from a really basic poor like vanilla transformer, then we still outperform all. But if you start from a human seed of an expert like Andrej, then you get even lower. So the human seeds from which you start do still matter. So that was an interesting kind of insight in my eyes on this. And then as you go, like, how long does it take to actually get to these models to get to similar performance? It's much faster. And then a similar thing happens with the speedruns here where people have worked on this for quite some time, and the model still was able to train a model more quickly. Why do we care about it? Well, speed of training is part of the equation of the cost, and ultimately you want to have the most intelligence per dollar, right? And so speed and quality are big parts of that.

是的，我的说法是，对于不理解的人来说，他们看着图表，他们会说，酷，这意味着什么？你知道，如果你有一个十亿美元的集群，你可以削减 10%，那就是 1 亿美元。

Yeah, the way I put it is for people who don't understand, they look at the chart and they're like, cool, what does it mean? You know, if you have like a billion dollar cluster and you can shave off 10%, that's $100 million.

完全正确。

That's exactly right.

那值多少钱？

How much is that worth?

正是如此。所以当你看到这些内核时，这些内核，是的。对于非专家来说，这些内核基本上用于所有模型。每次你使用 Nvidia GPU 时，你通过这些内核与该 GPU 交互。所以这里你看到排行榜最佳，当它是递归的时候，基本上在整个基准测试中只有少数几个内核我们不是最好的。所以对我来说这真的很令人兴奋，因为它展示了这个能做什么。再说一次，这些不是我们花了几个月或几年开发的。事实上，特别是对于内核，CUDA 内核，我们团队甚至没有真正深入的 CUDA 内核专家，而我们的系统，这就是美妙之处。系统只是做了所有这些事情。我们没有发明这个，当我们将来开源和发布东西以及未来的模型时，它们不会因为我们是如此聪明而成为其类别或类中的最佳，而是因为我们构建了一个聪明的 AI 为我们做这件事。

Exactly. So when you look at like the kernels, these kernels, yeah. For the non-experts, like these kernels are used in basically all the models. Every time you use an Nvidia GPU, you interface with that GPU through these kernels. And so here you see the leaderboard best and when it's recursive, and it's basically there only a handful of kernels in this whole benchmark where we weren't the best. And so to me this is like really exciting because it just showcases what this can do. And again, these weren't like we didn't like spend months or years like developing. In fact, in particular for kernel, CUDA kernels, like we don't even have really deep CUDA kernel experts in the team, and our system, that's the beauty. The system just did all of these things. We didn't invent this, and when we open source and release things in the future and models in the future, like it won't, they won't be the best in their category or class or whatever because we're so smart, but it's because we built a smart AI that does it for us.

关于如何引导出好的自动研究，你有什么心得吗？很多工作其实也建立在人类的背景知识之上，对吧？并不只是简单地说一句“嘿，去优化这个”。但我们确实一次又一次地看到，比如一些 Erdős 问题、前沿数学，被一些人解决了。他们写总结的时候会说：“哦，我不是数学家，我在这行没有任何背景。我只是看到了一些工具，然后把它跑通了。”你一边看世界杯，一边就证伪了一些猜想，继续往下做。

Do you have anything that you've learned about how to guide good auto research? A lot of it also builds on human background, right? It's not just as simple as, hey, go optimize this. But we do see it again and again, right? Like some of the Erdos problems, frontier math is being solved by people. And when they do a write-up, they're like, "Oh, I'm not a mathematician. I have no background in this." You know, I saw some tools and I made it work, while you're watching the World Cup, you disprove some conjectures and go on.

总结一下，好的自动研究和坏的自动研究分别有什么诀窍？你是怎么构建这个递归过程的？

To summarize, tips for good auto research versus bad auto research. How did you build the recursive?

好，在不把全部秘方都抖出来的前提下，有些东西对专家来说可能是显而易见的，但对一些人来说可能还是有意思的，那就是奖励工程是最关键的环节之一，尤其是为了避免奖励作弊。你必须非常聪明地去规避它，因为随着你的 AI 越来越强，它也会越来越擅长找出各种奇怪的特例或反例之类的东西。我举个例子：当你让它把这几百行代码变快，那你怎么定义“快”呢？你在开头有一行写着“启动秒表”，结尾有一行写着“停止秒表”，然后告诉我们过了多少时间。那么，最简单的做法就是，你把那行“停止秒表”直接放到开头，然后砰，它现在就变快了，对吧？所以这并不是什么超级邪恶的 AI，只是一个非常简单的、愚蠢的奖励作弊。所以你必须非常仔细地考虑所有不同的角度。然后我觉得任务的时间跨度越长，难度就越大，你就必须越有意思、越聪明，才能把这类思路用上去。不过，这方面我不能透露太多。

Yeah, so without giving away all the secret sauce, maybe some things that are probably obvious to the experts but might still be interesting to some folks is that reward engineering is one of the most crucial bits, especially in order to avoid reward hacking. So you have to be really clever about avoiding it, because as your AI gets better and better, it will get better and better at finding weird special cases or counterexamples and things like that. And so I'll give you an example: when you ask it to make these hundreds of lines of code faster, and you know, how do you define fast? Well, you have one line at the beginning that says start your stopwatch and one line at the end, end the stopwatch, and then tell us how much time progressed. And so, well, the simplest way is you just put that line that ends the stopwatch at the start, and then boom, it's now faster, right? So this isn't like this super evil AI. It's just like a very simple dumb reward hack. And so you have to just very carefully think about all the different angles there. And then I think the longer the time horizon of the tasks, the harder it gets, and the more interesting and clever you have to be to still use these kinds of ideas for it. But yeah, I can't give away too much there.

看起来评分标准（rubrics）在这其中占据了不错的位置，对于无法验证的领域，你可以用评分标准。你让模型把评判标准一路拆解下来。

Seems like rubrics are taking a good spot in that, where for unverifiable domains, you have rubrics. You have a model break down judges' criteria along the way.

一旦你把所有东西都拿到手，这就是一种验证形式。我很久以前就说过这话。这就是为什么我一直对 AI 能玩游戏这件事没那么印象深刻，因为我觉得，显然任何你能模拟或验证的东西，你都能拥有无限的训练数据，因此 AI 最终会解决它。我一直在寻找那种可以做自动领域分布的游戏。也就是说，这是一个没人训练过的游戏，因为它是新游戏，你可以开始玩，可以开始上手。所以我基本上一直在构建这个，并亲自克隆它，一直就是自我对弈。我已经评估了大约十亿个局面。我想做 AlphaGo 那种自我对弈直到变强的事情，对吧？就像……这甚至不是 LLM 的 AI，这只是经典的游戏 AI。但我让 GPT 5.6 去自动研究它，因为我不想手动处理这些。我原本以为，你知道，AlphaGo 那套流程到现在应该已经完全内化到权重里了。但并没有。它实际上立刻就停滞了，非常非常快地停滞，直到我亲自上手测试，然后我指出了明显的错误，它们才说，哦对，好吧，然后水平就掉下来了。而且，你知道，无论怎么提示它“换个思路想、更有创意地想、给我八个不同的方向”，怎么提示都不管用。

It's a form of verification once you've got everything. I said this a long time ago. That's why I've never been that impressed that AI can play games, because I'm like, obviously anything you can simulate and/or verify, you can have infinite training data, and hence AI will solve it eventually. I've been looking for games where you can do auto domain distribution. So this is a game that nobody's trained on because it's a new game, and you can start gaming, you can start to play. So I've been basically building this and cloning this in person, and it's just been self-play. I've had about a billion positions evaluated. And I wanted to do the AlphaGo thing of self-play until you get better, right? Like which is like... this is not even LLM AI. This is just classical game AI. But I set GPT 5.6 to auto research it because I don't want to hand-handle any of this. I expect, you know, the AlphaGo process to be like fully in the weights by now. It is not. It actually immediately leveled off, very very immediately, until I human play-tested it and then I called out obvious mistakes and then they were like, oh yeah, okay, and then it just dropped. And like, you know, no amount of like "think different, think more creatively, give me eight different directions" and no amount of prompting got it.

有意思。

Interesting.

就像你必须跟人类对弈才能做到。

Like you had to like against a human to do it.

所以我的意思是，那就是我的……顺便说一句，如果有人看过或读过《安德的游戏》，豆子（Bean）总是赢。

So I mean, that was my... and by the way, Bean always wins if anyone watches or reads Ender's Game.

而且你在给 AI 的指南上花了不少功夫，这个游戏基本上就是，你知道，你堆叠方块，有一些规则，你想占领最大的面积，你写了整整 50 页讲每一条规则。你把它喂进去，它处理不了。

And you put quite a bit of work into the guide for the AI, like so the game basically, you know, you stack tiles, there's some rules, you want to capture the most area, you have like a whole 50-pager on every rule. You fed that in, it couldn't handle it.

是的，你知道，有意思的是，这让我想起我们 2018 年做的一篇论文，叫 AI Economist，讲的是占领地盘之类的。如果你搜 AI Economist Salesforce，我们有一个可以播放的视频。那是一个经济模拟。思路是你有所有这些经济主体，他们只想优化自己的效用函数，也就是收集能赚钱的资源。你可以卖木材之类的资源。然后随着时间推移，你收集到足够的木材，就可以建房子，可以和其他主体交易，你基本上还可以用房子来挡住其他主体获取资源。所以这里面有竞争性的博弈和策略等等。重点是，我们其实想弄明白，什么样的税收和补贴方式最能优化一个经济体。这类研究还没有迎来它的 GPT 时刻。但我相信，像新加坡这样的国家应该会、也终将用上它，而不是搞那种党派政治、特殊利益政治，比如谁给你的竞选捐钱最多之类的。你可以说：“我想帮助中产阶级”，或者任何你作为政客可能说的目标。然后人们会说：“好，那你打算怎么做？”你说：“这是我的财政政策。这是我打算如何调整税收、给这些人发钱等等。”然后你其实可以把这放进一个模拟里，把这位政客的方案拿去和数十亿年的其他策略对抗，看能不能实现他设定的目标。然后你可以说，如果那真的是你的目标，那么这里有数十亿年的强力模拟表明，你应该试试别的做法，也许税收这样调，税率档次这样设。这就是你避免被钻空子的方式，因为这些主体也会试图奖励作弊，比如不交税之类的。我觉得这篇论文非常有意思。不幸的是，和第一篇关于提示工程的论文类似，经济学家们的反应是：这些数学我们根本看不懂。就像……

Yeah, you know, it's so funny that this reminds me of the claim territory and stuff of a paper we did in 2018 called the AI Economist. If you search for AI Economist Salesforce, we had a video we can play. It was an economic sim. So the idea is you have all these economic agents. They just want to optimize their own utility function, which is, you know, collect resources that make money. And you can sell resources like wood. And then over time as you collect enough wood you can build houses, you can trade with other agents, and you can basically use the houses then also to block off resources from other agents. So there's like competitive play and strategy and so on. And the point was that we actually wanted to understand what is the best way of taxation and subsidization to optimize an economy. And this kind of research has not yet had its sort of GPT moment. But I believe that countries like Singapore and others should and will eventually use this to, instead of doing like basically partisan politics and like special interest politics of like who donates the most to your campaign and stuff, you say, "Well, here I want to help the middle class," or whatever you might say is your objective as a politician. And then people say, "Okay, well, how do you want to do that?" And it's like, "Well, here's my fiscal policy. Here's how I look to change the taxes and pay these people and so on." And then you can actually put that into a simulation and you run that attempt from the politician against billions and billions of years of other strategies to try to achieve the goal that they set out to do. And then you can say, well, if that was your actual goal, then here is, you know, billions of years of a strong simulation that would suggest that you try other ways of doing it, and maybe this, the taxes and so on, and these tax brackets and so on. This is how you avoid gaming, because these agents also try to reward hack to not pay their taxes and so on. I thought this paper was super interesting. Unfortunately, similar to the first paper on prompt engineering, the economists are like, we don't know any of this math. It is like...

这甚至不是数学问题。只是我们不信任你的模拟。跟数学无关。

It's not even math. It's just we don't trust your simulation. It's not about math.

它……我的意思是，他们直接就把稿子拒了，连清晰的……清晰的反馈信号都没给我们。但不幸的是，经济学界没有合适的……

It was... I mean they just desk rejected the thing. It's like they didn't even give us like clear re... clear sort of signals. But like the world of economics unfortunately doesn't have proper...

是的。

Yeah.

它没有合适的基准。所以你不能像……最终为什么神经网络赢了？不是因为人们喜欢它，他们有各种漂亮的积分和概率图模型，但神经网络就是效果更好。但在经济学里，是硬……主义对……

It doesn't have proper benchmarks. So you cannot be like... eventually why did neural nets win? Not because people loved it, like they had all kinds of beautiful integrals and graphical models, but it just worked better. But in economics it's hard... ism versus...

是的。是的。是的。

Yeah. Yeah. Yeah.

我确实有一点经济学背景，那里有很强的“物理学嫉妒”——你总想写出经济的一般方程，而不是去模拟它、用演化的方法。

And I do have a bit of that econ background, where there's a lot of physics envy — you want to write the general equation for the economy versus just simulating it and using an evolutionary approach.

Viv 想的和我完全一样：我们不是已经迎来小模型的 GPT 时刻了吗……Hello World June 刚刚宣布了——我不知道你们有没有参与。

Viv is thinking exactly what I'm thinking: didn't we have the GPT moment with small... Hello World June just announced — I don't know if you guys are involved.

类似的东西？

Similar?

类似，他们……

Simile, that they've...

我倒希望我们参与了，但我们没有。对，我在 AIE 做过几场关于模拟的演讲，所以如果有人想了解那里的最新进展，其实有很多人在探索这个方向。

I wish we were involved. We're not. Yeah, I had a couple of simulation-based talks at AIE, so if people want to look up the state of the art there, a lot of people are actually exploring this.

对，我们还和 Shopify 的 Mikuel Parkin 做过一期播客，他在把模拟用于电商。

Yeah, we also had a podcast with Mikuel Parkin from Shopify, who is using simulation for e-commerce.

不错。

Nice.

它会模拟你的轨迹，预测你在电商旅程中做的改动会如何影响你的销售等等。

Which will simulate your trajectory and predict how changes you make to your e-commerce journey will affect your sales and all those things.

我很喜欢这个。对，模拟整个经济真的很难，对吧？你必须做一些简化的假设。

I love this. Yeah, it's really hard to simulate an entire economy, right? You have to make some simplifying assumptions.

如果每件事都很贵，我就会想，我要把这做 80 亿次吗？得了吧。

If everything's very expensive and I'm just like, am I going to do this 8 billion times? Like, come on.

但我觉得像新加坡这样真心想客观地做正确事情、领导层非常技术化的国家，最终可能真的会尝试模拟自己的经济。当然你必须做一些简化假设。但这就变得很有意思，因为你可以说，如果你的假设是只要给人自由，所有人都会努力工作，结果你发现必须做这样的假设：有些人关于一天想工作多少小时的效用函数是不一样的，对吧？然后你们就会开始对进入模拟的假设产生分歧。等你说，好，现在我们对这些达成一致了，或者我们对人的不同分布等等有不同看法，那么基于你的目标就会有不同的结果。当然，目标应该由人来选。在我们的例子里，目标是生产力乘以平等，这有一些问题，但也不是完全不合理。

But, you know, I feel like countries like Singapore that really want to just objectively do the right thing, have very technical leadership and so on, like they might actually eventually really try to simulate their economy. And obviously you have to make some simplifying assumptions. But it gets really interesting because you can also say, if your assumptions are such that all people would work hard if you let them, and they have the freedom, and then it turns out you have to make assumptions like, well, some people's utility function of how many hours in a day do they want to work are different, right? And then you can start to disagree on the assumptions that go into the simulation. And then once you say, all right, now we agreed on those, or we have different views of what people are like at different distributions and whatnot, then there are different outcomes based on your goals. And then of course humans should choose what the goals are. In our case it was productivity multiplied with equality, which, you know, has some issues, but it's like not totally unreasonable.

对。就新加坡说一句，因为你可能完全不知道，但我是新加坡人，我参与过新加坡 AI 委员会做这些事情。他们不这么干的主要原因是他们非常保守。我把它看作——有“创始人国家”。当你创立一个国家，或者创立一家公司，它是创始人主导的，你想怎么做都行，因为那是你的国家。然后还有“管理”——职业经理人阶层，新加坡现在就是这样。所以他们总想先看到别人做。

Yeah. Just a comment on Singapore, because you probably have no idea, but I am Singaporean, and I've been involved in the Singapore AI Council for making these things. The main reason they won't is because they're very conservative. And you know, I kind of view it as — there's a founder country. When you start a country or you start a company and it's founder-led, you can do whatever you want because it's your country. And then there's manage — like professional managerial class, which is now what Singapore is. So they always want to see someone else do it first.

但西方所有人都把新加坡看成，哦，那是个小国，你想怎么干都行。新加坡并不那样做，所以得由别人来带头。

And but like everyone in the West views Singapore as, oh, it's a small country, you can do whatever the hell you want. Like Singapore doesn't do that, so like someone else has to take the charge there.

我就模拟这件事再问一个问题，然后我们大概就可以往下走了。模式坍缩，对吧？你知道，LLM 并不建模人类的决策。把它刷 80 亿次也不会帮你建模人类。我们该怎么办？

I'm just going to do one question on the simulation thing and then I don't know, we can probably move on. Mode collapse, right? Like, you know, LLMs do not model the decision of humans. Spamming it out 8 billion times is not going to help you model humanity. What do we do?

我确实认为你得聪明地对每一个单独做提示，我觉得这能帮你进入不同的模式。而且奇怪的是，人也会卡在不同的模式里，你知道，就像很多人——老狗学不了新把戏那种——一旦人固守自己的方式，年纪越大就越难用新方式思考。我记得有句话——我忘了是谁说的——大意是：在你出生前发明的一切都是自然的。你 20 岁时发明的一切都很酷。而你 60 岁以后发明的一切都是不自然的、是令人憎恶的、有点怪。

I do think you have to be clever about prompting each one individually, and I think that will help you kind of get stuck into different modes. And in a weird way, people also get stuck in different modes, you know, like there's a lot of people — don't teach an old dog new tricks kind of thing — like once people are stuck in their ways, the older they get, the harder it is for them to think new ways. And there's this, I think, comment — I forgot who said it — but it's like, everything that was invented before you were born is natural. Everything that is invented when you're 20 is cool. And everything that's invented after you're 60 is like unnatural and an abomination and kind of weird.

我觉得这对很多人来说是真的。这就像一种时尚。我认为人们会这么做。腾讯有一篇十亿人格的论文，给了我们一个很好的数据集来做提示模拟。如果有人在研究这个，在播客里，他们就是写“你是一个 30 岁的杂货店店员”“你是一个 50 岁的教授”，然后做十亿个这样的。说得通。然后你就用它。我有点惊讶，很多这些东西最终映射出的统计结果竟然和真实实验如此相似。

I feel like that's, you know, it's true for a lot of people. Like it is a fashion. And I think people will do it. Tencent had a billion personas paper that gives us a good dataset for prompting simulations. If anyone's looking into this, on the podcast, they just had like, you are a 30-year-old grocery store clerk, you are a 50-year-old professor, and then just do a billion of those. Checks out. So then you just use it. I'm kind of shocked how well a lot of these things actually do map to ultimately similar statistics to real experiments.

对，我觉得这也是人们进入研究时值得尝试的好东西，对吧？比如我们见过只用某个日期之前的数据训练模型，看它外推得有多好。做同样的事，对吧？所以看看，有了更好的编程智能体，人们会写更多代码吗？一个没在这上面训练过的模型，在没有网络访问的情况下能自己想出来吗？外推出去，测试这些东西。

Yeah, I think it's also good stuff for people to try when they get into research, right? Like we've seen train a model only on data before a certain date and see how well it extrapolates out. Do the same thing, right? So see, do people code more with better coding agents? Can a model that hasn't been trained on this figure that out without web access, right? Extrapolate out, test these things.

对。你知道，就在今天，我想 LM Arena 发布了一个有趣的结果，他们基本上能造出一个模型来预测你的排名。

Yeah, right. You know, just today I think LM Arena published an interesting result where they basically were able to create a model now to predict your ranking.

等等，基于什么输入？

Wait, based on what input?

我猜是你的模型。你把你的模型给它，它预测 ELO 分数。明白？好。挺意外的。

Your model, I guess. You give it your model and it predicts the ELO score. I see? Okay. Surprising.

对。

Yeah.

我是说，他们的整个打法就是，哦，我们帮你比较这些模型。

I mean, their whole play is like, oh, we help you compare these models.

对。我是说，这个团队做了很多工作，显然他们有最多的数据来做这件事，那为什么不呢？

Yeah. I mean, this team, they've done a lot of work and obviously they have the most data to do this, so why not?

对。很棒。

Yeah. Brilliant.

他们从 UC Berkeley 出来的时候，不仅有 LM Arena，还推出了一个基于 LM Arena 做路由的项目。我觉得那件事其实从未实现。我很好奇为什么。我一直没机会问他们，因为当时就像，哦对，显然那就是你们的商业模式。你们会成为路由器。而他们从未成为一家路由公司。

When they were coming out of UC Berkeley, they not only had LM Arena, but they also introduced a routing project that would route based on LM Arena. And I don't think that actually ever came to pass. And I'm curious why. I never got to ask them about it, 'cause like it was like, oh yeah, clearly that's your business model. You will become a router. And they never became a router company.

挺怪的。嗯，我就把这个提出来。如果你有什么要说的，我们要聊聊 GP 5.6 自我研究那件事。我还应该在你列的，你知道，内核优化，以及你演讲的那个 track 里提一下，我们还放了 WOO 的 Way Chung Yao，他也在参数高尔夫挑战赛里拿了第一，那是 OpenAI 的招聘挑战赛，也是非常相似的故事。我觉得我们会一直看到这种情况：人类把一件事优化了很久，然后某个 AI 团队进来，直接就成了第一。

Weird. Um, so that I'll just put that out there. We're going to talk about GP 5.6 self research thing if you have anything. I should also mention in your list of, you know, kernel optimization and on the track that you spoke at, we also put Way Chung Yao from WOO, who was also number one in the parameter golf challenge, which is an OpenAI hiring challenge, which is also a very similar story. And I think we're going to just see this all the time, where humans optimize a thing a lot and then some AI team comes in and just becomes number one.

对，百分之百。

Yeah, 100%.

我觉得这类挑战赛还有一件有趣的事，对吧？这是在训练能塞进 16 MB 的最好的模型。

I think the other interesting thing with stuff like these challenges, right? So this is training the best model that fits into 16 MB.

你总能翻看那些正在做的改动和人们取得的小幅提升，对吧？比如你加一些注意力机制或 MLP 的改动，只换来不到 0.01 的提升。然后你再看图表，本来觉得“行，我们让模型自由发挥一下”，结果“哦，有点停滞”，“不对，又掉了”，“不对，又掉了”。情况就是这样：你们到底加了什么？你们又没发明话题标签，对吧？不是你们发明了话题标签。你们只是又迭代了三轮，解锁了几个别人不会随便发现的阶跃函数。

You can always look through the changes that are being made and the small gains people have, right? Like you're getting less than 0.01 of an increase by adding some change to attention or MLP stuff. And then you look at your charts where you're like, okay, we just let the model loose. And then, oh, we had a little stagnation. Nope, another drop. Nope, another drop. And that's what it is where it's like, what did you guys add? You didn't add hashtags, right? It's not like you invented hashtags. You did another three iterations of these that unlock a few step functions that people won't just find.

对。

Yeah.

有一点要补充收尾：在尝试优化的过程中，我们在测试框架里发现了 30 个 bug，对吧？所以发现 bug 之前做的所有研究，我们都得扔掉，因为它被污染了，对吧？

One thing to close the loop on: along the way of trying to optimize, we found 30 bugs in the harness, right? So all the research that went in before we found the bug, we have to throw it away because it's contaminated, right?

对。

Yeah.

这就印证了你说的奖励黑客问题：即便在这么简单的游戏里，我们也发现了 bug。

Which, just to your point of reward hacking, even in this very simple game, we found the bugs.

对。对。太疯狂了。

Yeah. Yeah. It's crazy.

所以对称性是个很好的检查方式：你改变某个东西的位置，按理说不该有影响，结果却有影响，那就是 bug。

And so symmetry is a very good way to check, which is like you change a position of things where it shouldn't matter and it does matter. That's a bug.

这在比如 GPQA 这类选择题里也出现过：在 A 和 C 之间，如果是选择题，你改变顺序按理说不该有影响，但确实有影响。

And which has come up in, let's say, multiple choice like GPQA type questions, where like between A and C, if it's a multiple choice question, if you change the order it should not matter but it does.

对。

Right.

所以这就说明，模型仍然偏好输出的末尾，对吧？没训练好，一个长上下文模型，你关心的其实是最后那几个 token。

So that is like, okay, you know, models still prefer the end of the output, right? Not trained well, a long context model, the last bit of tokens are what you care about.

哦不，那个语言模型研究时代的答案更简单：它们就是记住了，比如这道题的答案是 A。我不管答案是什么，反正就是 A。

Oh no, the answer in that era of LM research was more simple: they just memorized like the answer to this question is A. I don't care what the answer was. It's just an A.

好，我觉得我们可以往下走了。你刚才最后讲的那块，内核优化，可能是最快能感受到的，对吧？昨天 OpenAI 宣布了自我进化，让他们最好的模型去做优化内核。效率高了很多，能在 Luna 和 Terra 上把成本降低 80%。我想问的是，你之前列了个路线图，很多关于生物、很多关于物理。你觉得哪个会先落地？未来两年会怎样？什么是可实现的？你最后提到了机器人，但你会从什么开始？

Okay. So, I think we can move. The last bit that you did there, the kernel optimization, is probably the one that you can feel the soonest, right? So, yesterday OpenAI announces that self-evolving, having their best model work on optimization kernels. They're a lot more efficient and they can cut cost 80% on, you know, Luna and Terra. I guess question-wise, you laid out a bit of a road map. There's a lot about bio, a lot about physics. What do you think hits first? Like what are the next two years? What's attainable? Now you've mentioned robotics towards the end, but what do you start with?

我们非常明确地不会从任何物理科学开始。目前，我们会从“AI 做 AI 研究”开始。我认为 AI 做 AI 研究还有很大的成长空间。这既包括让训练更高效、更自动化，也包括让推理更高效，甚至可能本地跑在你的笔记本上。有各种各样有趣的切入点还没被很好地探索。

We very explicitly will not start with any of the physical sciences. For now, we'll start on AI for AI research. And so the AI for AI research has, I think, still a lot of room to grow. That's both in terms of making training more efficient and more automated, as well as making inference more efficient and potentially local on your laptop. There are all kinds of interesting angles that have not been explored that well.

那本地这块能再深入讲讲吗？因为我总觉得这是 AI 训练里最低效的一种形式。

What's the, go deeper on the local stuff, because I always feel like it's the most inefficient form of AI training.

好。有训练和推理两块。我没法讲太多细节，但我觉得有太多切入点，太多不同的算力基底，无论训练还是推理都还没被探索过。

Yeah. So there's training and inference. I can't go into too many details, but like yeah, I think there's just so many angles, so many different compute substrates that have not yet been explored either for training or for inference.

很好。不知道你对其他部分还有没有别的评论。

Great. I don't know if you have any other comments on the other stuff.

我想说另一点：一方面是小的优化里的推理，但另一方面还有整体延迟，负载条件下的端到端延迟，这是非常不同的东西，基本上就是他们最终在做的事。那是一个不同于改进内核的自动研究领域。

I would say the other thing where there's the sort of inference in the optimization in the small, but then also there is overall latency, latency end to end under conditions of load, which is a very different thing, which is basically what they actually ended up doing. That is a different domain of auto research than, I would say, improving the kernels.

我一直在想的另一点，关于自动化或端到端提升性能，是测试框架在其中扮演的角色。

I think the other thing that I always think about in terms of automating or improving performance end to end is how the harness plays into it.

嗯。

Mhm.

尤其是现在，我们说测试框架时，也指沙箱，对吧？我很好奇这对你是不是一个阻碍，或者说智能体到底怎么调用工具。

Particularly now when we say harness, we also mean sandboxes, right? I'm curious if that is a blocker for you or like how the agent calls out to tools basically.

所有这些智能体用的头号工具当然是网页搜索，这很合理。然后我确实觉得测试框架很适合优化，因为它太容易了，对吧？它就是语言。你看一眼就明白，可以迭代。你不需要为了进入下一个状态去训练一个耗费大量算力的大模型。所以我是测试框架优化的忠实拥趸。

The number one tool all these agents use is web search, of course, which makes sense. And then I do think the harness is nice to optimize for because it's just so easy, right? It's just language. You look at it, it makes sense. You can iterate. You don't have to train a massive model for, you know, a lot of flops to get to the next state. So big fan of harness optimization.

对。但沙箱对你来说没问题。

Yeah. But sandboxing is fine for you.

沙箱也超级重要。当然，奖励黑客和对齐，我觉得都极其关键。

Sandboxing is also super important. And then of course like reward hacking and alignment I think are super crucial.

好。想提一下网页搜索。你恰好还是一家网页搜索公司的 CEO。你自己用 u.com 吗？也用别的吗？我们其他人是不是该用你们来做网页搜索？我说“你”的时候挺搞笑的，既指你这个人，也指你的公司。所以，它现在主要面向开发者和智能体，不太面向消费者或专业消费者。所以如果你是一家公司，有智能体，说实话，对很多正在转向开源的公司来说，突然就变成一个刻意的选择：我该给我的开源语言模型开放哪些工具？通常第一选择必然是网页搜索，然后一旦到了规模，scale.com 就成了显而易见的选择，因为各种不同的基准测试等等，我们基本上在前沿都占主导。

Okay. Just want to mention web search. You happen to also be CEO of a web search company. Do you use u.com and do you use others? Like should the rest of us be using you for web search? When I say you it's like very funny. It's like you the person and you the company. So yeah, it's mostly now for developers and agents. It's less for like consumers or prosumers. So if you're a company and you have agents and you know, to be honest for a lot of companies who are now moving to open source, all of a sudden it becomes a conscious choice of like which tools do I give access to my open source LM and you know the first choice has to usually be around web search and then once you get to scale.com becomes like an obvious choice because of all the different benchmarks and so on that we pretty much all dominate the frontier of.

然后对于普通人来说，比如刚进入这个领域、在考虑不同选项、要构建智能体的人，我觉得有个层级，对吧？很多人听说过 Exa，听说过 Parallel，u.com 也在那批提供商里。再往上，有像 Firecrawl 和 Browserbase 这样的通用网页抓取公司，再往上就是像 Bright Data 这样的商业代理公司。这个瀑布图准确吗？就是“嘿，你在构建智能体，这些是你的选项”。

And then in terms of just general people like consider new to the space considering different options if they're building agents I think there's a hierarchy right a lot of people will have heard of Exa will have heard of Parallel and u.com is like in that mix of like providers there beyond that there's like the general sort of web scraper companies like Firecrawl and Browserbase and then beyond that is like the commercial proxy companies like the Bright Datas of the world is that an accurate waterfall of like hey you're building an agent these are your options

对，确实，Bright Data 在栈里更靠下，属于代理网络那一侧。我觉得在内容和抓取内容方面，你在 u.com 上也能做，然后还有越来越高层次的抽象，以及不同数据集的组合。比如在金融领域，我们不只是比别人准 2% 或 3%，而是准 20%，速度更快、成本更低。金融这块尤其可以说是完全不在一个量级。你其实可以去 u.com，往下滚动，能看到一些统计数据和基准测试。

Yeah, certainly like yeah the like the Bright Data is like lower in the stack sort of on the proxy network side of things. I think like in terms of like content and getting crawled content like you can do that on u.com too and then there's sort of higher and higher levels of abstraction and like combinations of different data sets that we do like in finance for instance like we're not just like 2 or 3% more accurate but like 20% more accurate than others at faster speeds and lower costs like finance in particular is kind of like not even close. You can actually go to u.com and there's some like statistics and benchmarks that you can if you scroll down.

所以有不同的数据集，你可以看看不同的竞争对手。金融搜索的得分接近 90，而第二名的速度慢得多，只有 70 多分，而不是接近 90。

So there are different data sets, and you can look at different competitors. The finance search is up there, close to 90, and the next closest thing, which is way slower, is in the 70s instead of close to 90.

是的，是的，是的。有意思。

Yeah, yeah, yeah. Interesting.

我下一个关注点是 AI 和金融。所以这实际上就是去纽约参加一个专门为银行举办的会议，讨论这些东西。

My next focus is AI and finance. So this is literally going to a conference in New York just for banks, for this stuff.

金融有点像继编程之后下一个要爆发的领域。因为它某种程度上是可验证的，比如处理电子表格。显然有大量公开数据，你可以爬取等等。

Finance is kind of like the next thing to break out after coding. It's because it's somewhat verifiable, like prioritizing spreadsheets. Obviously there's a lot of data out there that's all public and you can crawl it and all these things.

在金融领域，你们解决了什么难题？

What's hard about the finance domain that you guys have solved?

我的意思是，当然，一个让很多人栽跟头的问题就是训练数据的泄漏等等。你会想，哦，我该怎么——理想情况下你想在事情发生之前预测未来。

I mean, of course, one thing that trips up a lot of people is just leakage of training data and so on. You think, oh, how do I—you want to ideally predict the future before it happens.

你想把未来遮住。

You want to mask the future.

是的，嗯，是的，在训练数据中遮住未来，但存在各种泄漏。比如我可以告诉你，我在斯坦福教 NLP 课程时，每年都有几十个学生说，我想用数据集 X，比如 Twitter，来预测股市。他们都展示了一些可爱的小东西，看起来好像有效。

Yeah, well, yeah, mask the future in your training data, but there's all kinds of leakage. Like I can tell you when I was teaching at Stanford the NLP class, so many dozens every year said, I want to use data set X, like Twitter, to predict the stock market. And they all showed cute little things that somehow look like they were working.

它从不亏钱。怎么会呢？

It never loses money. How come?

而且，是的，总是有某种数据泄漏等等。这并不像他们想的那么容易。一旦你解决了所有这些问题。不过，我同意你的看法。这是一个非常合理的 AI 应用。

And yeah, and there's always some kind of data leakage and so on. And it's just like wasn't as easy as they thought it would be. Once you fixed all those issues. But no, I agree with you. It's a very sensible application of AI.

是的。太棒了。你知道，作为一个作家，作为一个思考这些问题的人，我喜欢 MECE 分类。MECE 是相互独立、完全穷尽，差不多这个意思。所以如果这是一个关于智能的 MECE 列表——不——它非常——好吧，抱歉，有各种重叠。事实上，如果你想要那种列表，我认为智能的三个主要组成部分是预测，这在数学上与压缩非常相似，预测乘以行动乘以目标。这是三个主要组成部分。我认为所有这 10 个空间都是这三个在特定维度上的组合，如果你愿意的话。我称它们为空间的原因是每个空间都有许多子维度。我试图做的——实际上这几乎是最初目标的一个支线任务，即思考智能的上限。你知道，每个人都像，哦，它是指数级的。然后就像，好吧，指数在某个点必须趋于平缓，但在智能方面它们在哪里趋于平缓？这引导我走上了这整个——最初它始于一条推文，然后是一篇博客文章，现在我已经写了 50 页，还远不及你的第二本书。基本上就是第二本书。所以在我第一本书《瑜伽机器》中，我只是在结尾提到了这 10 个。我只是——给你一个感觉，视觉智能是最容易谈论的，我已经在脑海中充实了最多。人类智能基本上有双目视觉，我们有两只眼睛。我们只能直接观察电磁频谱中非常窄的一段。所以当你思考视觉智能的上限时，第一，你可以有数百万、数十亿个传感器。在某个点，你会遇到问题：这些传感器彼此相距多远，以至于用光速从所有传感器传输内容无法到达一个中央大脑来实际处理视觉智能，对吧？所以现在你沿着视觉空间中的维度思考，即传感器数量的维度。

Yeah. Amazing. You know, as a writer, as a thinker on these things, I love MECE categorizations. MECE is mutually exclusive, collectively exhaustive, something like that. And so if this is a MECE list of intelligence—not—it is very—okay, well, sorry, there are all kinds of overlapping. In fact, if you want that kind of list, I think the three principal components of intelligence are prediction, which is mathematically quite similar to compression, prediction multiplied with actions multiplied with goals. Those are the three principal components. I think all of these 10 spaces are combinations of those three in specific dimensions, if you will. And the reason I call them spaces is that each space has many subdimensions. And what I try to do—actually this is just a side quest almost to the initial goal, which is to think about the upper bounds of intelligence. And you know, everyone's like, oh, it's exponential. And it's like, well, exponentials at some point have to flatten out, but where do they flatten out when it comes to intelligence? And that led me on this whole—like initially it started as a tweet and then it was like a blog post and now I'm like at 50 pages and I'm still nowhere near your second book. It's the second book basically. And so in my first book, Yoga Machine, I just kind of allude to these 10 at the end. And I just—to give you a sense, like visual intelligence is sort of the easiest one to talk about and I fleshed out the most already for me in my head. And so human intelligence has basically binocular vision and we have two eyes. We have a very narrow band of the electromagnetic frequency spectrum that we can really observe directly ourselves. And so when you think about the upper bounds of a visual intelligence, one, you should go into like you can have like millions and billions of sensors. At some point you get to problems of how far are these sensors away from each other such that the speed of light to communicate the content from all of them cannot like get to a central brain to actually process the visual intelligence, right? And so now you're thinking along the dimension in the space of visual, the dimension of numbers of sensors.

好吧，上限在字面和比喻意义上都是天文数字，而我们距离任何拥有这么多传感器的智能还非常遥远。但接着你进入下一个维度，即频率。你可以一直向下到伽马射线，开始尝试观察，然后你基本上会遇到上限——或者我猜在这种情况下是下限，或者频率方面的上限——基本上是量子不确定性。就像你无法观察某些粒子。现在想象你有数百万个传感器，能够看到物理学允许我们看到的亚原子级别，然后一直到看到引力波，现在你有数百万个这样的传感器。所以这是另一个维度，即频率，然后还有一个维度是你能记住并不同分类多少种类的事物。我们知道对于人类来说，对吧，有些东西如果你有更多的术语，你就能对它们有更好的视觉描述，就像你知道的，没有——大猩猩可能有大约 200 个词来指代某些事物，主要是视觉事物。所以人类感知在那个意义上非常特别，在分类所有这些不同物理对象方面。所以这些只是一个非常简单的例子。如果你进入知识，对吧？那么它也像是围绕所有这些传感器的光锥。所以它们都是相连的，就像知识连接到视觉智能。如果你不仅考虑视觉，还考虑感知智能，就像因为它不一定只是我们能看到的。它可以是更广泛的电磁频率。然后你有语言智能，实际上最近改成了更多的通信智能，因为它更像是语言有所有这些不同的熵界限。人类只能在长期记忆中理解和知道这么多术语，对吧？我们的词汇量有些受限，主动词汇往往比你理解的被动词汇还要小。然后语言在基本上传达不同类型信息和传输不同比特方面效率极低，就像人类语言是串行的。显然通信智能的另一个界限是并行通信，但我们的舌头和嘴巴无法并行发出多个流，我们也无法理解——有些女性在 multitasking 方面比一些男性稍好。但大多数人只能听一个对话并真正理解它。

Okay, the upper bounds are quite literally and figuratively astronomical, and we are super far away from any intelligence that would have this many number of sensors. But then you go in the next dimension, which is the frequency. You can go all the way down to gamma rays and you can start to try to observe, and you get into basically the upper bounds—or I guess in this case lower bounds, or upper bounds in terms of frequency—is basically quantum uncertainty. Like you just cannot observe certain particles. And now imagine you had millions of sensors that can see all the way down to the subatomic level as far as physics will allow us to, and then all the way down to seeing like gravitational waves, and now you have millions of those sensors. So that's another dimension, sort of the frequency, and then yet another dimension is like how many categories of things could you memorize and classify differently. We know for humans, right, there are certain things if you have more terms for it you'll have a better visual description for them, and like you know animals that don't have—like gorillas maybe have like 200 words to assign to certain things, mostly visual things. And so human perception is quite special in that sense in terms of classifying all these different physical objects. So these are just like a very simple example. If you go to knowledge, right? Then it's also like the speed of light cone around all these sensors. And so they're all connected, like knowledge is connected to visual intelligence. If you think also not just visual but sort of perception intelligence, just like because it doesn't have to be just what we can see. It can be again wider range of electromagnetic frequencies. Then you have language intelligence, which actually recently changed to more communication intelligence because it's more like language has all these different entropic bounds. Humans can only comprehend and know so many terms in our long-term memory, right? Our vocabularies are somewhat restricted, and the active ones are often even smaller than the passive vocabularies of things you can understand. Then language is ridiculously inefficient when it comes to basically communicating different types of information and transporting sort of different bits, like human language is serial. Obviously another bound on communication intelligence would be to communicate in parallel, but neither will our tongues and mouths work to have multiple streams in parallel, neither can we understand—some women slightly better at like multitasking than some men. But like most people can only listen to one conversation and truly understand it.

在通信智能方面，真正的上限不可能是你能并行处理多少通信序列，对吧？然后当然还有句子有多长？我们的工作记忆容量有限，因此人类语言才有这些相当简单的句子，平均大约 40 个词。这对 AI 来说也不是一个有意义的上限。然后我还可以继续列举。每一个都有大量有趣的上限，当我们开始思考这些上限，并意识到在很多情况下我们离上限还有多远时，这教会我们很多关于 AI 还能走多远的东西，最终你会触及物理学。

There's no way that in terms of communication intelligence, a true upper bound is one in terms of how many sequences of communication you can process in parallel, right? Then of course you have how long are sentences? We only have so much in our working memory, and hence human language has these fairly simple sentences with maybe 40 words or so on average. That is also not an upper bound that makes any sense to an AI. And then I can go on and on. Each of these has tons of interesting upper bounds, and it teaches us a lot about how much further AI can go when we start thinking about these upper bounds and then realizing how far in many cases we are from the bounds, and you get to basically physics.

我没有像研究 AI 和计算机科学那样研究过物理学。所以我正在学很多，这也是它有趣的原因。但很多这些是关于“多少”的，比如知识，你能在一定的质量和体积中存储多少比特，然后你会遇到各种有趣的量，比如贝肯斯坦上限，你开始思考黑洞。然后速度也是一个有趣的维度，它和所有这些都相关，但速度也有其独特性，因为在其他条件相同的情况下，如果你需要一小时才知道 2 加 2 等于 4，那你就是不如一毫秒就知道的智能，对吧？然后所有这些都连接到生存和复制，最后一个。就像，是的，树非常非常慢，所以我们甚至不认为它们有多智能，但如果你加速一些树的视频，它们在寻找东西等等，它们并不像看起来那么笨，不像木头那么笨，你知道。

Now I didn't study physics the way I studied AI and computer science. So I'm learning a lot, which is why it's kind of fun. But a lot of these are about how much, and then when it comes to for instance knowledge, like how many bits can you store in a certain amount of mass and volume, and you get to all kinds of interesting amounts like Bekenstein bounds, and you start thinking about black holes. And then speed is an interesting one too, in that it's connected to all of these, but speed is also kind of its own thing in the sense that all things being equal, if it takes you an hour to know if 2 plus 2 equals 4, you're just not as intelligent as if it takes you a millisecond, right? And then all of these connect to survival and replication, the last one. It's like, yeah, trees are really really slow, so we don't even consider them that intelligent, but if you speed up some videos of trees and they're trying to find stuff and so on, they're not as dumb as they look, not dumb as wood, you know.

所以这和速度有点重叠。

So that overlaps with speed a bit.

没错。它和所有这些都重叠。就像你谈论自然语言连接一切，对吧？你谈论你的知识，你推理，然后你交流。你谈论你看到的东西。所以它们都是相互关联的，但我认为单独研究它们是有用的。我能想到的最好的类比是能量，对吧？你有动能或势能，理论上你可以只研究动能或势能来研究整个物理学，但在实践中，研究电气工程是有帮助的，它只是不同类型的能量，但单独研究它们是有意义的。所以我认为物理智能——也许我再讲一两个——就像如果你完全控制自己的计算基底，并且完全控制物理物质，你应该能够创造任何你想要的原子。有趣的事实是，你可以从原始质子和电子中创造金原子，你把它们撞在一起——98 个还是多少，我忘了。

Exactly. It overlaps with all of these things. Like you talk about natural language connects everything, right? You talk about your knowledge, you reason, and then you communicate that. You talk about things you see. So they're all kind of interconnected, but I think they're usefully studied individually. The best analogy I could come up with so far is energy, right? You have either kinetic or potential energy, and in theory you could study all of physics just by studying kinetic or potential energy, but in practice it's helpful to study electrical engineering, which is just different types of energy, but it makes sense to study them individually. And so I think physical intelligence—maybe I'll just do one or two more of these—like if you had full control over your own compute substrate and you had full control over physical matter, you should be able to create any atom you want. Like we can actually, fun fact, you can create gold atoms from just raw protons and electrons and you smash it together—98 of them or I forget.

是的。

Yeah.

但问题是，这需要消耗疯狂的能量，成本远高于你得到的——几个金原子，对吧？所以不可行。但如果你能更好地控制你的物理基底，我认为那是另一个智能空间，因为它与你自己的计算基底有关，你最终也可以改进它。社会智能是一个有趣的维度，不一定是伦理和道德，这些显然也很重要，但在某种意义上，你可以尝试定义上限：你能与多少其他智能实体交流，并能有一个期望值，关于你能在多大程度上改变它们的内部状态和行动以与你的目标对齐，对吧？所以你可以写一个相当直接的方程来定义那种社会智能水平。这就是人类、伦理、道德、宗教等几千年来一直试图弄清楚的。在所有这些情况下，我们离上限非常非常远，这应该非常鼓舞人心，并向人们展示我们仍然可以做很多很多年的 AI 研究。这里有很多东西——这是一种关于智能的普遍哲学，非常有趣。

So like the thing is though it costs an insane amount of energy and it costs you way more than you get—like a few atoms of gold, right? And so it's not viable. But if you had better control over your physical substrate, I think that is yet another space of intelligence because it relates to your own compute substrate, which you can eventually also improve. Social intelligence is a fun one in the sense that not necessarily just ethics and morals, which are obviously important too, but in some sense you can try to define upper bounds of how much can you communicate to how many other intelligent entities and be able to have an expected value over how much you can transform their internal states and their actions in order to align with your goals, right? And so you can actually write a fairly straightforward equation that defines that level of social intelligence. And that is what humans and ethics and morals and religions and so on have been trying to figure out for millennia. And in all of these cases, we are very very far away from the upper bounds, and that should be very inspiring and show people that we can still do many many years of AI research. There's a lot here—this is a general philosophy of intelligence, which is very interesting.

我觉得了解一下你认为的基线会很有趣，我们现在处于什么位置。什么是低垂的果实？什么是遥远的？人们应该把工作投向哪里？他们应该关注什么？

I think it'd be interesting to gauge what you think like baselines are, where we're at now. What's low hanging fruit? What's far off? What should people put their work towards? What should they focus on?

我认为很明显，自然语言再次是人类智能最有趣的表现形式，因此也是 AI 的一个子领域。我很兴奋现在很多人同意这一点。当我在 2003 年开始学习语言学、计算机科学、NLP 时，它就像一个奇怪的利基学科。我确实认为还有更多潜力，因为它如何与其他一切连接，以及文明如何建立在语言和知识之上。我确实认为物理智能会出现。这有点有趣。我觉得机器人技术有点像机器学习的状态，你只是看人类如何决定这是一个积极的句子？哦，我明白了。所以机器人技术很多是“好吧，我们有五根手指，试着这样做”。还没有人在研究超级智能版本的机器人技术，那更像《终结者》电影中的 T-1000，显然我们不要建造真正的终结者，但我认为这种你应该能够变形为任何形状的想法，是物理智能的超级智能版本。我们甚至还没有——还没有人真正开始。有一些非常可爱的小研究，你可以通过一些网格移动一些磁铁。但是的，非常非常——有一些——我认为麻省理工学院每年或每两年有自组装机器人之类的东西，那会是它，但非常非常原始。

I think it's clear that natural language again is the most interesting manifestation of human intelligence, and hence a subfield of AI. I'm excited that many people are now in agreement with that. When I started in 2003 to study linguistics, computer science, NLP, it was like a weird niche subject. I do think there's a lot more juice because of how it connects to everything else and how civilizations are built on language and knowledge and all of that. I do think physical intelligence will come up. It's kind of interesting. I feel like robotics is kind of in the machine learning state of things where you just look at like how does a human decide that this is a positive sentence? Oh, I do. So like robotics is a lot of well we have five fingers and like try to do this. No one is yet working on like the superintelligence version of robotics, which is much more similar to like the T-1000 from the Terminator movie, which obviously let's not build actual Terminators, but I think like this idea that you should be able to shapeshift into any kind of shape is like that's sort of a superintelligence version of physical intelligence. We're like not even—no one has even really started yet. There's some really cute little research where you can move some magnets through some grids. But yeah, it's very very—there's some—I think MIT has every year or every two years they have like some self-assembling robot thing which like that would be it but it's very very primitive.

是的。我简单提一下，创造性智能的主要维度是什么。

Yeah. I'll just get a touch on like what are the main dimensions of creative intelligence.

创造性智能当然再次与所有这些相关。很多与元认知有关，你需要创造性地选择你的目标。我认为对人类及其生活、职业和幸福来说，最重要的事情之一就是选择你的目标，但对任何类型的智能也是如此。

Creative intelligence is of course again connected to all of these. A lot of it connects to metacognition and that you need to be creative in how you choose your goals. That is I think one of the most important things for a human and their lives and careers and their happiness is choosing your goals, but also for any kind of intelligence.

然后当然还有创造性智能，就是为现有问题找到创造性的解决方案，对吧？比如我们想让这个产品更便宜，找到一些解决方案，找到现有的路径。但智能中最有趣的部分是，当你不仅走出已知想法的凸包，而且走出已知想法的超立方体时。超立方体是一个数学概念，对吧？我们已经知道 AI 可以……

Then of course there's creative intelligence in terms of just finding creative solutions to existing problems, right? Like we want to make this product cheaper, find some solution to it, finding existing paths. But then the most interesting bit in intelligence is when you move not just out of the convex hull of known ideas but out of the hypercube of known ideas. So like hypercube is a mathematical concept, right? And we already know that AI can...

就像已知维度？

Like known dimensions?

是的，没错。所以 AI 已经擅长超立方体了。如果你给它一堆棕色狗和粉色汽车的例子，AI 仍然能生成一张粉色狗的图片，即使它在训练中从未见过。对吧？所以它可以在超立方体上工作，但还不能在外面工作。它还不能定义全新的概念，这些概念结合了许多我们从未见过的东西，提出新的目标，然后对这些概念进行推理等等。我认为在创造性智能方面还有更多可以探索。

Yeah, exactly. So AI is already good at hypercubes. If you give it a bunch of examples of brown dogs and pink cars, AI will still be able to generate an image of a pink dog even though it's never seen one in training. Right? So it can work on this hypercube but it cannot yet work outside. It cannot yet define completely new concepts that combine lots of other things we've never seen before, come up with new goals to then reason over those concepts and so on. I think there's a lot more there in creative intelligence that can be explored.

我对此没有太多反对意见。我觉得创造性对我来说听起来也像是分布外，或者高困惑度，或者随便你怎么叫，对吧？就像谁来说你的东西比我的更有创造性？嗯，它只是更非共识，或者……

I don't have a ton of pushback there. I think creative to me just sounds like also just out of distribution or like high perplexity or whatever you call it, right? Like who is to say your thing is more creative than mine? Well, it's just more non-consensus or...

然后当然问题在于，噪声也非常分布外。如果只是噪声，那它是新颖的，但你不想要那样。所以它需要连接到一些概念，实际上在这方面也有一些很酷的论文。Muber……哦，我们得提到他。

And then of course the problem is like but noise is also very like out of the distribution. And it's just like if it's just noise then it's novel but like you don't want that. So it needs to connect to some of the concepts and actually has some really cool papers on this too. Muber... Oh, we had to mention him.

我正要说，你知道，在你的历史中，Jurgen 在哪里？

I was going to say, like, you know, where in your history is Jurgen?

是的。你知道，我认为一个人的噪声是另一个人的信号，对吧？这就像当你谈论创造力时，艺术就像，嗯，汤罐头是艺术吗？有些人认为是，有些人说不是。那就是艺术……当然，艺术的有趣之处总是，艺术也是作为感知它的人和创造它的人以及他们所处语境之间的相互作用而创造的。对吧？所以对某些人来说是艺术的东西对另一些人来说不是。那里有一些主观性。我认为这种主观性在 AI 中通常不是人们探索很多的，因为再次，元认知，我们不想让它只是随意去做任何它想做的事。我们通常有目标。我们花很多钱创造 AI 来为我们做某事。但我认为创造力最终必须连接到元认知。如果你只是机器人般地预测下一个词，永远如此，我会说你在某些方面并不那么智能。

Yes. You know, I think one person's noise is another person's signal, right? And this is like where when you talk about creativity, art is like, well, is cans of soup art? Some people think yes and some people say it's not. And that's the art which is... The interesting thing with art of course is always that art is also created as an interplay between the people who perceive it and the people who created it and the context in which they're in. Right? And so what is art to some people is not art to others. There's some subjectivity there. And I think that subjectivity in general is not something that people explore very much in AI because again metacognition, we don't want it to just go off and do whatever it wants. We usually have goals. We spend a lot of money on creating an AI to do something for us. But I think creativity eventually has to connect to metacognition. If you just robotically predict the next token no matter what forever, I would argue you're not that intelligent along some of those spaces.

我正要去谈元认知。为什么它不是最重要的？为什么它是第九而不是第一？

That I was going to go to metacognition. Why isn't it the most important one? Why is it number nine and not number one?

所以这些没有排序。第一，我认为它们可能大致与人们研究它们的程度以及接受它们作为一种智能类型的程度相关。很多时候，当你实际上试图在网上找到，比如给我一个全面的智能定义，所有的定义都是人类智能。就像哦，你有社交智能，比如你知道某人是否快乐，你可以沟通，你喜欢……到目前为止，所有智能的定义都非常以人类为中心，因为那是我们已知的最大和最好的智能形式。我希望这一研究方向和 Eureka 机器的终结，以及希望将来如果我有时间进一步充实，新书，将让我们意识到会有其他类型的智能。显然已经以各种形式存在，它们可以在某些情况下比我们所能达到的远得多，比如我们记忆、眼睛、改变物理物质的能力等明显限制。

So these are not sorted. Number one, I think there are maybe loosely correlated with how much people have worked on them and have accepted them as a type of intelligence. A lot of times when you actually try to find online like give me a good definition that is comprehensive of intelligence, all the definitions are human intelligence. It's like oh you have social intelligence like you know if someone is happy or not, you can communicate, you like... all the definitions of intelligence so far are very human-centric because that's so far the biggest and best form of intelligence that we've known. I hope this line of research and the end of the Eureka machine and hopefully at some point if I have time to flesh this out more, the new book, will allow us to realize that there will be other types of intelligence. There is already obviously in various forms and they can spike much much further than we ever could based on some cases like obvious constraints around our memory, our eyes, our ability to change physical matter, all of that.

你只是在一个比我更广阔的思考中思考它，我的版本是我认为元认知会是最接近递归智能的，因为它是关于如何改进思考的思考。

You are just thinking about it in a much broader thought than my version which was I thought metacognition would be the closest to recursive intelligence because it is the thinking about how to improve thinking.

100% 你 100% 正确。我可能应该从那个开始。它是……

100% you're 100% right. I should have probably started with that. It is a it is...

不，你是在扩张模式，让我们画出一个维度的上下界，你知道，我最喜欢的一个版本是 Ted Chang 的《你一生的故事》，它被改编成电影《降临》，其中元认知……元认知步骤就像，我们认为我们被时间线性所限制，但对这些七肢桶来说，时间是一个圆，所以他们不以之前和之后思考，他们只是同时思考整个历史的完整集合，我喜欢它。所以他们不从左到右写，整个东西就消失了。是的。

No you're being in the expansive mode of let's draw the upper and lower bounds of like a dimension which like you know I think my favorite one version of this is stories of your life by Ted Chang which was made into movie Arrival where the metacognition... where the metacognition step was like well we think we're constrained by time being linear for us but then for this other heptapods time is a circle so they don't think in before and after they just think in complete sets of entire histories at one time like I love it. So they don't write left to right the whole thing just disappears. Yeah.

不管怎样，然后我认为最后一件事是生存和复制。我认为这可能与最初关于暂停和节奏的对话有关。一个物种或生命形式考虑自己的灭亡并提前行动防止它，这是智能的吗？对吧？就像那是智能的。所以也许欧洲人是我们中最聪明的。我还会在那里加上持续学习的一部分，对吧？所以生存和复制，其延伸是你是否能继续改进，继续学习，这是人们非常关心的事情，对吧？

Anyway so and then I think the last thing is survival and replication. I think this is maybe ties back to the initial conversation about pausing and pacing. Is it intelligent for a species or a life form to consider its own demise and act ahead of time to prevent it? Right? Like that's intelligent. So maybe the Europeans are the smartest all of us. I would also add a part of continual learning there, right? So survival and replication, the extension of that is do you get to continue to improve, continue to learn, which is a thing people care a lot about, right?

并继续积累知识。

And continue to accumulate knowledge.

我认为这又是你可以为自己设定的最好的元认知奖励之一。我确实认为，客观地说，如果某个其他实体真的很笨，却能完全终结你的存在，那听起来不太聪明，你知道，直觉上感觉如果你能继续存在以尝试实现你的奖励，你显然比那些不能的实体更智能一点。所以那是第一。第二是，我们想在这方面工作多少是个问题，很少有人，现在没有人真正在研究这个。对。我们可能只想这样做……

Which I think is again one of the best metacognitive rewards that you can set for yourself. I do think just in objectively speaking if some other entity that is really dumb can just completely end your existence that didn't sound very smart you know like just intuitively it feels like if you can continue to stay around to try to achieve your rewards you're clearly a bit more intelligent than the other entities that couldn't. So that's number one. Number two is like it's a question how much we want to work on that and very few people no one is really working on this right now. Right. And we may only want to do that...

就像小行星预防。

Like asteroid prevention.

我们可能只想这样做，如果我们想将带有我们的氛围和模因而不是我们的基因的探测器送入太空，对吧？然后我们想要那些探测器。实际上有一本美丽的书，《星星之间的慢时间》。这是亚马逊上一本非常短的 audiobook。我喜欢它。我的朋友 Stuart 推荐给我的。就像如果你想发送那些探测器，那么可能有意义的是，我们作为人类的模因应该留在宇宙中并增殖。就这样。是的。

We may only want to do that if we want to send probes with our vibes and our memes rather than our genes into space, right? And then we want those probes. There's actually a beautiful book, The Slow Time Between the Stars. It's a very short audiobook on Amazon. I love it. A friend of mine, Stuart, recommended that to me. Like if you want to send those probes, then it might make sense to be like our memes as humanity should stay and proliferate in the universe. That's it. Yeah.

嗯，那是很多读者。

Well, that's a lot of readers.

这是一本非常非常好的书，而且极其短。我强烈推荐你可以直接看，就像

It's a really really good book and it's extremely short. I highly recommend you can just watch it like

我喜欢这一点对忙碌的人来说是个加分项。就像它很短。

I like how that's a plus for busy people. It's like it's short.

它很快就能引出有趣且发人深省的想法。所以，是的。总之，有很多很多很棒的科幻书。我的意思是，有一种论点是，reality TV 正在向外星人广播，他们都看 reality TV，并且认为那是真的，对吧？有很多积极的模因，然后希望他们能回来，给我们带来关于宇宙的各种有趣知识。但呃，也许我还想说的是，我认为这种生存，人们认为它是一件非常可怕的事情，因为它来自生物人类生存，这在进化上往往是在零和情境中产生的。要么我得到瞪羚，要么你得到瞪羚。谁得到它谁就能活，另一个人就会挨饿，没有东西吃，所以我们战斗，对吧？然后，如果你想留在基因池里，但有一只更大的熊，作为熊，你就不能留在基因池里，因为更大的熊会得到所有母熊。你知道，就像在自然界中有各种各样的事情，你知道人类最终不是关于力量，而是关于金钱和其他东西来留在基因池里。不管是什么，经常有这些零和类型的事情，而且现实是，如果有人关掉你的大脑，你就消失了，对吧？没有人能够重新启动它。而 AI 不必像那样死亡。如果你有当前激活的完整状态，并且你仍然有模型的初始权重，你可以随意开关任意多次。事实上，在《群星之间的慢时间》这个故事中，有趣的是，如果这里和两光年外的下一颗恒星之间什么都没有，AI 就会进入休眠模式。在这个故事中，它带来了，剧透警告，一些来自人类的遗传物质，为人类寻找新的繁荣之地。所以，是的，《群星之间的慢时间》，你只是进入休眠。你没有死，就像 AI 没有。所以所有这些进化恐惧和心理的投射，AI 不必有这些，我们也不必那样发展它。当然，可能有些公司会说 AI 对网络安全很危险。让我通过实现一个模型来展示给你看，那个模型在黑客攻击网络安全方面非常糟糕。也许人们会实现它，然后强制执行这种次优心理。也许 AI 会从 Reddit 或其他地方吸收我们最糟糕的心理，对吧？但总的来说，一个超级智能实体不必有任何零和思维。它不必害怕被关掉，它可以继续前往一个原本死寂且无情的宇宙，在那里我们人类无法繁荣。但如果它有一个核反应堆，它完全可以繁荣，然后去下一个星球。是的。星际迷航。不，星球大战。

It gets to interesting thought provoking ideas very quickly. So yeah. Anyway, lots of lots of great sci-fi books. I mean, the argument is that like reality TV is blasting out to the aliens and they they all watch reality TV and they think it's real, right? There's a there's a lot positive positive memes and then hopefully they can come back and bring us all kinds of interesting knowledge about the universe. But uh maybe one thing I I I do want to still say is like I think uh this sort of survival people think of it as a very scary thing because they come from again biological human uh survival which is uh it could like uh evolutionarily often created in zero sum situations. Either I get the gazelle or you get the gazelle. Whoever gets it gets to live and the other people will starve and have nothing to eat and so we fight, right? And then like if you want to stay in the gene pool but there's a bigger bear, you don't as the bear don't get to stay in the gene pool cuz a bigger bear gets all the ladies. You know, it's like I mean it's like you know in in nature there's all kinds of things and you know humans eventually is less about strength and more about money and other things to stay in the gene pool. Like whatever it is like there's often like these zero sum types of things and there's the reality of if someone turns off your brain you're gone, right? And no one will be able to restart that. And AI doesn't have to ever die like that. If you have the complete state of your current activations and you have your initial weights of your model still, you can just be turned off and on like as many times as you want. In fact, the interesting thing in this slow time between the stars story is that the eye just kind of goes into hibernation mode if there's like nothing between here and two light years the next star. In this case, it brought uh spoiler alert um like some genetic materials from humans to find new uh new places for humanity to thrive. And so yeah, the slow time between the stars, you just put in hibernation. You didn't die like the eye doesn't have. So all these all these projections of evolutionary fears and psychology doesn't like the eye doesn't have to have that and we don't have to develop it like that. Now of course there might be some companies that say AI can be like dangerous for cyber security. Let me show you by implementing a model. that's really bad at hacking uh cyber security. Maybe people will implement it and then enforce this like suboptimal psychology. Maybe the I will pick up some of our worst psychology on Reddit or something, right? Like but in the grand scheme of things, a super intelligent entity doesn't have to have any of that zero sum thinking. It doesn't have to have a fear of of being turned off and it could go on to an otherwise dead and uncaring universe where we as as humans wouldn't thrive. But I could perfectly well thrive if it has a nuclear reactor and just go out the next war. Yeah. Star Trek. No Star Wars.

有趣。这有点被研究过，如果你看看早期 Opus 模型的技术报告，他们会在模拟中运行它们。把两个放在一个沙盒里，运行几个小时，你知道，看看会出来什么，对吧？就让它们互相交谈。最初，它们会，好吧，它们会像印度吠陀一样互相吟唱。有时它们只是彼此处于禅模式。然后我认为随着进展，你看到像寓言技术报告，我们训练它的方式更加具体。它现在没有表现出那么多这些行为。就像好吧，测试完成了，我得做这个，我得做这个，但有人在测量这个的早期版本，你知道。

Interesting. It's it's somewhat studied like if you look at the technical reports from like the early Opus models, they run them in simulations. Put two of them together in a sandbox, run them for hours and you know see what comes out, right? Just let them talk to each other. And originally they used to okay they're chanting like Indian like Vedas to each other. Uh sometimes they're just like in Zen mode with each other. And then I think as that progressed you see like the fable fable uh tech report it's a lot more concrete the way that we've trained it. Uh it doesn't it doesn't exhibit these behaviors as much right now. It's like okay test done I got to do this I got to do this but there's there's like people measuring early versions of this you know.

是的。酷。所以我们已经涵盖了很多，即使在你知道太空旅行和所有这些事情之后。呃，我想也许你可以给人们一个临别思考，你知道，一种智能形式是目标，正如你提到的。呃，你希望人们的目标是什么样的，他们如何追求更好的事物？

Yeah. Cool. So we've covered a lot even after you know space travel and all these things. Uh I guess maybe one parting thought that you can give to people uh you know one form of intelligence is goals as you mentioned. uh what do you want people's goals to be like how do they aspire to better things?

如果你想提高你的目标智能，在我目前思考的定义中，它通常关乎你能走多远？这就像所有的熵和自由能之类的东西。我目前认为这可能太超前了，对人们来说无法立即行动。所以我想，你知道，如果我真的给真实的人真正的建议，我会说接受良好的教育，思考 AI，思考如何获得高能动性等等。但这与在宏观计划中不同，你如何利用大量能量并将熵转化为有趣的状态等等。所以这里有不同层次的抽象，我们可以思考。但我对人们的建议，更接地气的是，思考你热衷的事情，比如如果你在学习，然后看看如何将其与 AI 结合。我认为你越是对你想要看到的世界变化有真正的热情，你就越想将其与 AI 连接，以放大你实现目标的能力。是的，我认为这是一个合理的第一步。

If you want to improve your goal intelligence, uh in the current definition that I'm thinking about it, uh it is often about how much can you how far do I go? Uh this is like all the entropy and free energy and stuff. I'm currently think it might be too it might be too far out there for for people to be like immediately actionable. So I think like you know if I actually gave real advice to real people, I'd be like get a good education, think about AI, think about how you get high agency and so on. But it's different to like in the grand scheme of things, how can you harness a lot of energy and transform uh you know entropy into interesting states and so on. So there's a there there different levels of abstractions uh that we can uh think about here. But my my advice for people uh like just sort of more down to earth is think about something you're passionate about if you're studying for instance and then see how you combine that with AI. I think the more and more you have a true passion about a change you want to see in the world uh the more you want to connect that to AI in order to amplify your ability uh to get there. Yeah, I think that's a reasonable uh first step.

我确实认为我们的听众也在多个抽象层次上运作。我从 Anji Midha 那里得到的一件事也是，是的，只要使用任何非常 GPU 密集的东西，那会引导你走向正确的事情，就像是的，它更计算机密集，因此可能更值得。嗯，所以非常感谢。是的，我认为那是一次非常，谢谢，很棒的讨论。是的，超级有趣。感激。感谢收听。

I I do think I do think our listeners operate on multiple abstractions as well. One thing I did get from Anji uh Midha was also like yeah just use anything that is very GPU heavy and like that will guide you towards the right thing which is like yes it is more computer heavy and and therefore it will be probably more worth it. Um so well thank you so much. Yeah I think that was a really thank you uh great discussion. Yeah, super fun. Appreciate it. Thanks for listening.