从 DeepMind 到 River AI：Igor Babuschkin 谈个人 AI 与智能体的未来

欢迎回到 Unsupervised Learning，我是 Jacob Efron。今天这期我们请到了 Igor Babuschkin，非常精彩。Igor 一直在对的时间出现在对的地方，领导了很多非常有趣的 AI 工作。他曾在 DeepMind 领导了星际争霸和 AlphaCode 的很多工作；在 OpenAI 做了推理的早期研究；后来联合创立了 xAI，在 Colossus 上做了很多英雄式的工作，也帮助把模型带到了今天的高度。能跟 Igor 聊他的新公司 River AI，聊他们在个人 AI（面向企业和消费者）以及本地硬件上的押注，非常棒。我们聊了他对生态走向的所有看法、要提升模型超越编码进入不可验证领域需要什么、以及他认为什么能让我们到达那里。我们还聊了为什么他认为闭源模型提供商现在作为企业其实处境艰难，也聊了他对 AI 模型对未来世界意味着什么以及政策影响的反思。能跟站在这个领域最前沿的人聊这些最受关注的问题，真的很棒。我相信大家会很喜欢 Igor 的观点。话不多说，有请 Igor。Igor，非常感谢你来上播客，真的很高兴能请你来。

Welcome back to Unsupervised Learning. I'm Jacob Efron. We had an awesome episode today with Igor Babuschkin. Igor has been at all the right places at all the right times, leading a lot of really interesting AI work. He was at DeepMind where he led a lot of the work around StarCraft as well as AlphaCode. He was at OpenAI doing the early work on reasoning. Then he was a co-founder of xAI where he did some of the heroic work on Colossus as well as helped get the models up to where they are today. It was awesome to get to talk to Igor about his new company, River AI, the bets they're making around personal AI for companies, for consumers, as well as local hardware. We talked about all his takes on where the ecosystem is headed, what's required to improve models beyond coding into non-verifiable domains and what he thinks will get us there. We talked about why he thinks the closed source model providers are actually in a really hard place right now as businesses. And we hit on his reflections on what AI models mean for the future of the world and the policy implications of this as well. Just awesome to talk to someone who's at the forefront of the space on all the questions that are top of mind. I think folks will really enjoy Igor's perspective. Without further ado, here he is. Igor, thanks so much for coming on the podcast. Really excited to have you here.

谢谢邀请，Jacob，我们也很期待跟你聊。

Yeah, thank you for the invitation, Jacob. We're excited to chat with you.

你在 DeepMind、OpenAI、xAI 期间开创了这么多重要的研究。你早早地就触及了很多我觉得定义了今天这个时代的东西：基于验证的推理、推理时搜索。而你最近离开 xAI 创办了自己的公司 River，专注个人 AI。我觉得有很多东西我们的听众会很想听你的看法。我想确保他们了解 River，听到你对身处 AI 所有这些事件中心的反思，以及你对未来走向的预测。我在想从哪里开始时，发现一件特别引人入胜的事：你今年早些时候写了一些很有意思的虚构作品。对于那些没看过的人，其中一篇是对模型最终失控统治世界的反乌托邦式展望，结尾非常有意思——主角意识到这一切都是梦，什么都没发生，但他们仍然决定迈出第一步，开始行动。然后作为一个有趣的对比，你的第二篇作品其实是写给接下来那些机器的一封情书。所以也许从这开始，我很好奇是什么启发你写这些？

You've pioneered so much important research in your time at DeepMind, OpenAI, xAI. You were early to so many of the things that I feel like define the era we have today: verified-based reasoning, inference time search. And you recently left xAI to start your own company, River, really focused on personal AI. I feel like there are so many things that our listeners are going to be excited about to get your take on this episode. I want to make sure they understand River, they get your reflections on being at the epicenter of all these things that have happened in AI, as well as your predictions for where we're going. As I was thinking about where to begin, one thing I found really compelling is that you wrote some really interesting fiction earlier this year. For those that haven't seen, one piece was this dystopian look at how models ended up uncontrolled running the world and had this really interesting ending, right, where your protagonist kind of realizes that they dreamed all of this and none of this has happened yet, but they still make the decision to take that first step and begin. And then in kind of an interesting contrast, your second piece was really this love letter to the machines that come next. So maybe to start, I'm curious what inspired you to write these?

是的，我觉得去年 11 月、12 月左右，我们所有软件工程师和 AI 研究者都经历了一件挺大的事：编码智能体突然变得强大到无法忽视。以前你可以选择用或不用，但从那以后，随着 Claude Opus 模型的最新迭代，突然所有人都同意：是的，这些模型非常强大，是的，它们让我们的工作轻松太多了，而且哇，它们做了太多我们以前做、也享受做的软件工程。所以这对我们所有人来说都是一次真正的转变。和很多人一样，我 12 月花了很大一部分时间，用编码智能体重写了我这辈子想做或做过的所有编码项目。你会得到一种难以置信的感觉：哇，现在一切皆有可能。而且这些智能体会继续进步，会变得越来越有能力。随着这些模型越来越强，整个世界会发生什么？这启发了我写那个故事。它很大程度上是《魔法师的学徒》的现代版。《魔法师的学徒》对魔法非常着迷，开始使用它，但后来魔法失控了，就像迪士尼短片或原诗里那样。所以在我看来，这很大程度上就是那个故事的翻版。而且我觉得我们现在都成了魔法师的学徒，因为大语言模型和智能体变得越来越有能力。而且我认为这不会止步于编码。所以我非常非常感兴趣的是，我们的智能体下一步会走向哪里，还有什么会被改变。我越想越觉得一切都会被改变。当涉及到智能体时，所有计算其实都是可以争取的，因为深度理解你生活中发生了什么、工作场所发生了什么、整个世界发生了什么，然后基于此采取行动的能力，可以应用到任何问题上，应用到任何你作为企业或消费者可能想做的事情上。这就是为什么我对个人 AI 如此感兴趣，因为我觉得那是下一个迭代，是智能体将能为我们做什么的下一个体现。影响只会从这里增长。

Yeah, I think there's something pretty big that happened to all of us software engineers, AI researchers around November, December of last year, which is that coding agents suddenly became so powerful that you can't ignore them. Previously you could, some people used them, some people didn't. But from that point on, with the latest iteration of Claude Opus models, suddenly everybody agreed that yes, these models are very powerful, yes, they make our jobs much much easier, and also wow, they're doing so much of the software engineering that we used to do and used to enjoy. So it was a real transformation for all of us. And like many folks, I spent a large part of December rewriting all of the coding projects that I've ever wanted to in my life or have done, using the coding agents. And you get this incredible sense of like wow, anything is possible now. And also these agents are going to continue to improve. They're going to continue to get more and more capable. And what will happen with the world at large as these models are getting stronger and stronger? That kind of inspired the story. It's very much a modern version of the Sorcerer's Apprentice. The Sorcerer's Apprentice is very fascinated with magic and starts to use it, but then it kind of runs away from him, like in the Disney short or in the original poem. So it's very much a version of that in my mind. And I think we're all becoming the Sorcerer's Apprentices now, with LLMs and agents becoming more and more capable. And I think it's not going to stop at coding. So I'm very, very interested in figuring out where our agent is going to go next, what else is going to be transformed. And the more I think about it, the more it feels like everything will be transformed. All of computing really is up for grabs when it comes to agents, because the ability to deeply understand what's happening in your life, at your workplace, what's happening in the world at large, and then taking action based on that, is something that you can apply to any problem, to any kind of thing you might want to do as a business or as a consumer. So that's why I'm so interested in personal AI, because I feel like that's the next iteration, the next manifestation of what the agents are going to be able to do for us. The impact will only grow from here.

我是说，显然你提到的那个 12 月时刻引起了这么多人的强烈共鸣。而且我觉得实际上很多研究者都回应了你的故事，说这真的很触动人心。我们在 11 月、12 月经历了编码方面的那个时刻，我觉得很多人都在试图弄清楚，要怎么做才能让智能体在其他领域也真正工作得很好。也许在问：编码和数学，或者那些最可验证的领域，是不是有什么特别之处？我们会在其他领域看到同样的进步吗？你如何看待需要做什么才能达到更通用的智能体？

I mean, obviously it feels like that December moment you talk about resonated so strongly with so many folks. And I think actually even a lot of researchers responded to your story being like this really hits hard. We had that moment in November, December on the coding side, and I think a lot of folks are trying to figure out what's required to get agents to work really well in other domains. And maybe asking, well, is there something special about coding and math, or the most verifiable domains out there? Are we going to see the same kind of progress in other domains? How do you think about what needs to be done to get to even more general purpose agents?

是的，我觉得实际上从这里我们可以走很多下一步。有很多不同的方向会打开。编码有点像中心路径，是明显的低垂果实，因为它可验证，因为它触及这么多不同领域，它解锁了你可以用智能体做的这么多不同事情。但从这里开始，有几个不同的选择可以去。我会说科学发现是一个非常非常明确的方向。我们也开始看到智能体能够解决数学和科学中真正严肃问题的第一丝火花。这也是 xAI 背后的一个大灵感。

Yeah, I think there are actually many next steps that we could be taking from here. There are many different directions that open up. Coding was sort of the central path, the obvious low-hanging fruit, because it's verifiable, because it touches so many different domains, it unlocks so many different things that you can do with the agents. But from here, there are a few different options of where to go. I would say scientific discovery is a very, very clear one. We're also starting to see the first sparks of agents being able to solve real serious problems in math and the sciences. That was a big inspiration behind xAI as well.

所以我们当时都被这个想法深深吸引。那时候，大语言模型还不太能解决困难的推理问题，但我们觉得这会是下一个重大突破，这也是我们创办 xAI 的重要灵感来源。所以我认为智能体会在复杂编程项目和数学问题（任何可验证的事情）上继续快速进步。在数学上，你甚至可以用 Lean 这样的工具来形式化你的定理和证明，这能给你很好的奖励信号，帮助改进智能体。它还能帮你判断你是否真的找到了证明。随着这些问题越来越复杂，看到它真正在实际中发挥作用，真的非常令人着迷。因为我们 AI 研究者之前就想过，也许形式化在未来会很有用，但真正看到它在现实中发生，完全是另一回事。但除此之外，一个巨大的瓶颈是数据。对吧？所以如果你想在物理世界取得重大突破，无论是材料科学、新的基础物理发现，还是建造更好的火箭发动机——就是 Elon 可能想用智能体做的那种事——我们往往需要在现实世界中运行实验。所以你需要想办法与现实世界环境闭环。智能体有了设计某种材料的想法后，需要得到反馈，判断这个想法好不好，有没有用。所以我认为这是当前的一个大前沿，也就是用智能体做科学发现。这种科学与发现的方向，我认为是当下正在发生的一件大事。很多人都在关注智能体在这方面的应用，而且有潜力获得高度超级智能的 AI，它们比任何人类甚至任何人类群体都更有能力取得科学突破、做出决策，甚至可能预测未来。我也会把这个大致归入那个方向。然后我觉得可能还有其他方向。那么普通人会使用的 AI 是什么样的呢？他们可能不会用那种回答一个问题就要花费一百万美元的超级 AI，对吧？那么每个人都会用的 AI 是什么呢？所以我认为我们可能会看到一种分化：一边是只有少数人才能使用的超级 AI，另一边是我们日常都会使用的普通 AI，用来提高生产力、帮助更好地组织生活，基本上就是让生活更美好。所以，奖励函数就是它能在多大程度上帮助人类感觉更好、完成更多事情、对自己感觉更好。所以我认为在改善人类生活、改善我们的日常生活方面也有巨大的潜力，但这些智能体可能并不需要最大能力——它们可能不需要能证明黎曼猜想。

So we were all really fascinated by this idea. At the time, LLMs weren't really capable of solving hard reasoning problems, but we kind of felt that this was the next big breakthrough, and that was a big inspiration behind xAI and why we started it. So I think agents will continue to improve very rapidly on complex coding projects and also math problems—anything that's verifiable. So with math, you can even formalize your theorem and your proof using tools like Lean, and it gives you a good reward signal that helps you improve agents. It also helps you tell if you've actually found the proof. As these get more and more complex, seeing that actually work for real is really fascinating, because, you know, we AI researchers have thought about this for some time—like maybe formalization will be useful in the future—but actually seeing it happen in real life is quite something else. But beyond this, one huge bottleneck is data. Right? So if you want to create great breakthroughs in the physical world, whether it's material science, whether it's new fundamental physics discoveries, whether it's building a better rocket engine—the type of thing that Elon might want to do with the agents—that's where we often need to run experiments in the real world. So you need to figure out a way to close that loop with the real-world environment. So the agents, when they have an idea of what kind of material to design, they need to get some feedback as to whether that was a good idea—did it work, did it not work. So that's a big frontier right now, I would say, and using agents for scientific discovery. So this kind of direction of science and discovery—I think that's a big thing that is happening at the moment. That's something a lot of people are focusing on with the agents, and there's a potential there that we'll get these highly super intelligent AIs that are vastly more capable than any human or even any collection of humans at creating scientific breakthroughs, at making decisions, maybe at predicting the future as well. So I would also broadly categorize that in that type of direction. And then I think there might be other directions as well. So what is the AI going to be that the everyday person is going to use? They probably are not going to use the super AI that costs a million dollars to run once on a question, right? So what is that AI that everybody will be using? So I think we might see a bit of a bifurcation there, actually, where there are other super AIs that only a few people have access to, but there's also the everyday common AI that you and I will be using all the time to be more productive or to help organize our lives better—just lead better lives basically. So very much, you know, the reward function is how much did it help the human feel better, get more things done, feel better about themselves. So I think there's also tremendous potential there to improve things for humanity, improve our everyday lives, but those agents might not actually need maximum capability—they might not need to be able to prove the Riemann hypothesis.

我是说，这种分化似乎已经在发生了，对吧？最前沿的模型对很多编程和数学任务确实很有价值，但对很多 ChatGPT 用户来说，并不明显更复杂的模型真的——一年前的模型对他们想做的事情来说已经完全够用了。

I mean, it already feels like that bifurcation's happening, right? In terms of the most frontier models being really valuable for a lot of coding and mathematics tasks, but for many people that are ChatGPT users, it's not inherently clear that the more complex models have actually—the ones from a year ago were totally fine for most of what they wanted to do.

是的，没错。我觉得让个人也能感受到 AI 的好处很重要，因为越来越感觉是少数大公司创造了这些强大的 AI 模型，并控制着对它们的访问，比如决定它们的行为方式。对我来说，让每个人都觉得他们真的能亲自从 AI 中获得价值，这一点非常重要。这也是我最近创办 River 的重要灵感来源，因为在我看来，这本质上可能是一个技术问题。对吧？我们对 AI 应该如何构建和分发给人做出了一些假设。你应该进行大规模预训练。你应该建立 API，按 token 收费。这是每个人都知道、每个人都遵循的模式，但未必非得如此。所以我们可以想出新的方式来构建 AI 模型、定制它们、分发给人们，也许能让他们更好地控制自己的体验，更好地控制 AI 在做什么。这是一个研究问题，也是一个工程问题。所以我觉得这非常迷人。这就是我们决定创办这家新公司的原因。

Yeah. Yeah. Exactly. And I feel like it's important for the individual to also feel the benefits of AI, because increasingly it feels like a few large companies create these powerful AI models that control access to them, like get to say how they will behave. And for me, it's just very important that everybody feels like they're actually personally able to get value out of AI. And that's a big inspiration behind where I started River recently, because to me it feels like potentially a technology problem. Right? We've made certain assumptions about how AI should be built and distributed to people. You should do large pre-training runs. You should set up an API and charge people by the token. And that's sort of the model everybody knows about, everybody follows, but it doesn't have to be that way. So we can come up with new ways of building AI models, of customizing them, of distributing them to people that maybe will put them more in control of their experience, will put them more in control of what the AI is doing. And it's a research problem. It's an engineering problem. So it's something that I find very fascinating. So that's why we decided to start this new company.

这个想法是什么时候在你心中成形的？因为你之前的经历显然是在一些最大的闭源模型公司。所以这一切是什么时候发生的？

When did it kind of crystallize for you? Because obviously your previous experience had been at some of the largest, you know, closed source players with those kind of models. So when did this kind of all come to be?

是的。所以当我在 2025 年离开 xAI 后，我决定做一些 AI 安全公司的天使投资。所以我得以认识 AI 社区里很多不同的创始人，尤其是 AI 安全领域的。他们有很多新想法，关于如何帮助让 AI 长期对人类更有益，或者更好地控制 AI 的负面影响或风险。外面有很多很棒的想法，我觉得都非常迷人。但归根结底，我只是坐在那里，等着这些人推进他们的使命并取得成功，变得非常不耐烦。所以我就想自己开创一些新东西，这样我就能帮助推动这个方向。对我来说，帮助分配 AI 的好处、帮助分配对 AI 的控制权，是 AI 安全的一部分。这可能是我们现在最需要做的事情，因为对 AI 的权力正在相当集中，尤其是在美国，你能感觉到人们对此不太满意。他们不一定信任 AI 的构建者，也不觉得他们会考虑自己的最佳利益。所以我认为重要的是我们要展示一些方式，让人们能感受到好处，能感觉自己掌控着这一切。

Yeah. So when I left xAI in 2025, I decided to do a bit of angel investment in AI safety companies. So I got to meet a lot of different founders in the AI community and in AI safety in particular. So people having new kinds of ideas around how we could help make AI more beneficial for humanity in the long run, or to better control the downsides of AI or the risks. And there are a lot of amazing ideas out there, and I just found it all really fascinating. But at the end of the day, I'm just sitting around waiting for all these folks to make progress on their mission and to succeed, and got very impatient. So I just felt that I wanted to start something new myself so I can help push on this direction. And for me, helping distribute the benefits of AI, helping distribute control over AI, is a part of AI safety. It's maybe the most urgent thing that we need to do right now, because power over AI is concentrating quite a bit, and especially in the US, you can feel that people aren't too happy about that. They don't necessarily trust the AI builders or feel like they have their best interests in mind. So I think it's important we show some ways in which people can feel the benefits, can feel like they're in control of the whole thing.

我想快速插一句，如果你喜欢这次对话，支持节目的最好方式就是在你收听的地方点击关注或订阅。这能真正帮助我们邀请到最好的嘉宾，并帮助其他人发现这个节目。现在，回到 Igor 的访谈。

Just wanted to take a quick break to say that if you're enjoying the conversation, the single best way to support the show is hitting follow or subscribe wherever you're listening. It makes a real difference in helping us get the best guests and helping others find the show. Now, back to Igor.

嗯，也许现在是时候聊聊你对 River AI 这家公司、产品以及你希望如何实现这一目标的愿景了。

Well, maybe interesting time to just talk about your vision for River AI as a company, the product, and you know what how you hope to kind of achieve that.

是的。所以，当我们创办这家公司时，我们环顾四周，看看哪里有这些技术机会，有哪些想法被投资不足，却可能改变我们使用 AI、构建 AI 的方式，然后我们把每个想法都当作一个研究项目来推进。我们要弄清楚这是否是我们可以实现并最终推向全世界的东西。所以我们目前实际上在推进三个不同的项目。第一个是，我们意识到帮助公司掌控 AI 的工具基本上已经存在了，你只需要把它做得非常好用。这就是我们创办 River API 的原因，你可以把它看作一个强化学习和微调服务，类似于 Thinking Machines 的 Thinker 等其他玩家。但我们有自己的独特见解，而且我们真正引以为傲的是我们的工程能力。所以我们尽可能优化这个平台，利用我们多年来学到的所有技能，让训练尽可能便宜、可靠、可扩展。这就是我们在 River API 上的第一个赌注。第二个是关于个人 AI 和个性化。我们觉得有机会让 AI 智能体比现在更好地与个体对齐。今天，我们训练 AI 模型基本上是为了让它们对普通用户表现良好。你实际上是在训练过程中汇集所有用户的所有反馈和结果，然后告诉模型对每个人都表现相同。我们的想法是，如果我们打破这个假设，允许模型对它所服务的每个个体表现不同呢？所以你的 AI 智能体对你说话的方式会不同于我的智能体对我说话的方式，它们会有不同的偏好、不同的行为。这个研究领域还处于早期阶段，所以我们在这里有几个不同的方向。因为显然，市面上所有商业成功的智能体都是以这种平均人群的方式训练的。但我们觉得，一旦智能体真正直接从你那里学习，并且每次你与它互动时都变得更好，那感觉会非常棒。这是我非常兴奋的事情。然后我们下的第三个大赌注是硬件，因为现在的假设是，获得最强大智能体的唯一途径是通过数据中心。像 OpenAI 和 Tropic 这样的公司，他们建立非常大的数据中心，所有的推理和训练都在那里进行，作为想要使用 AI 的个人，你必须通过他们的 API。没有其他方式。他们可以决定如何限制、在哪里提供访问权限。所以真正的终极举措是，我们能否将推理算力本地化到用户手中，那么要真正在办公室或家里的小型设备上运行前沿模型，需要什么条件？这非常像一个研究项目，因为今天你在那里受到很大的内存限制。所以实际上能够将模型的所有权重放到单个芯片、单个设备上。这就是我们想要弄清楚的，如何将世界上最好的模型带给个人，让你真正感觉自己掌控了它。它就在你家里。你的数据也受到保护。所以这就是我们下的三个赌注，我们非常像当年 OpenAI 和 DeepMind 那样运营。我们考虑得非常长远，我们承担一些风险更大的赌注。我们假设会有研究，会有一些新想法被使用。

Yeah. So, when we started the company, we kind of looked around like where are these technological opportunities, like what are these ideas that have been underinvested into that could change the way that we use AI, that we build AI, and let's run this as a research project. Let's figure out if this is something that we can make work and then ship to everybody in the world. So we have really three different projects that we're working on at the moment. So one is we realized the tools to help companies take control of AI are pretty much already there. You just have to make it work really really well. So that's why we started the River API, which you can think of as a reinforcement learning and fine-tuning service, similar to other players like Thinker from Thinking Machines. But we have our own unique take on it, and the one thing that we're really proud of is our engineering. So we really optimize this kind of platform as much as possible. Try to make training as cheap and as reliable, as scalable as we can make it, using all these skills that we've learned over the years. So that's the first bet that we're making on the River API. The second one is around personal AI and personalization. So we feel like there's an opportunity to align AI agents much much better with the individual than what we're currently doing. So today we're training AI models to basically work well for the average user. So you're literally pulling all of the feedback, all of the results from all of your different users during training, and you're telling the model behave the same way for everybody. And our idea is what if we kind of break that assumption and we allow the model to behave differently for each individual that it's serving. So your AI agent will speak differently to you than my agent speaks to me, and they're going to have different preferences, different behaviors. This area of research is still pretty early. So there are a few different directions we're going down here. Because obviously all the agents that are out there that are commercially successful, they're all trained in this sort of average population kind of way. But we feel like it's going to feel really amazing once the agent truly learns directly from you and gets better every time you interact with it. So something that I'm pretty excited by. And then the third big bet that we're making is on hardware, because right now the assumption is that the only way to get access to the most powerful agents is through access to a data center. So companies like OpenAI and Tropic, they bring up really really large data centers, all of the inference all the training happens there, and as an individual that wants to use AI, you have to go through their APIs. There's no other way. They can decide how it's restricted, where they get access or not. So really the ultimate move here is can we bring the inference compute locally to the users, so what would it take to actually be able to run a frontier model in a small device that you can have in your office, that you can have at home. Very much a research project, because today you're very memory limited there. So actually able to fit all of the weights of the model onto a single chip, onto a single device. So that's something that we want to figure out, like how do we bring the best models in the world to the individual, so you can really feel like you've got control over it. It's in your home. Your data is protected as well. So those are the three bets that we're making, and we're very much running this the way that back in the day OpenAI and DeepMind will run. So we're thinking very long term, we're taking some riskier bets. We're assuming there's going to be research, there's going to be some new ideas that will be used.

不，这很吸引人。我觉得这三个赌注都很有意思。所以也许为了深入探讨，我们先从最后一个开始，也许是硬件方面。显然，你在这里要完成的事情范围很广。硬件方面的动机是什么？显然，最终最可控的 AI 是你能在本地运行的，对吧？没有人能决定你是否能访问。这是动机吗，还是你看到了其他对终端用户的实际好处，让你通过增加硬件组件进一步提高了复杂程度？

No, it's fascinating. I mean I think each of those three bets is really interesting. And so maybe to kind of dig into them we'll start with the last one maybe on the hardware side. Like obviously you have a really broad scope of things you're seeking to accomplish here. Is the motivation there on the hardware side? Obviously like ultimately the most controllable AI is one you can run locally, right? That no one can kind of dictate you know whether you have access or don't have access. Like is that kind of the motivation, or is there other kind of tangible benefits you see to end users that made you increase the complexity level even further by adding the hardware component to it?

完全正确。所以对我们来说，控制方面是最重要的。我们希望人们能够掌控自己的 AI 体验，真正感觉模型是他们的，而不是别人的。这是一个很大的动力。但当我们开始研究这个时，我们也意识到还有一些额外的好处。例如，在你家里本地拥有这个盒子，可以给你带来更低的延迟。所以当涉及到通过语音或视频输入、视频输出与智能体互动时，它开辟了更多的可能性。所以一旦你采用这种形态，实际上会出现一些新的消费体验可能性。这是我非常兴奋的事情，因为它甚至可能让整个体验比我们从数据中心世界得到的要好得多。是的。而且我认为隐私在个人 AI 世界中也将非常重要。这些深入了解你的个人智能体，它们会观察你所做的一切，然后主动帮助你。它们将获得大量关于你的敏感数据。基本上，你电脑上的所有文档，你在房间里说的所有话，都可能被个人智能体存储，因为如果它拥有这些上下文，它就会更有用，对吧？而将所有数据发送到数据中心，那里可能存在安全漏洞，或者即使你不喜欢，你的数据也可能被用于训练。那真的很不理想。所以我们也从隐私的角度来考虑。

Exactly. So really the control aspect is the most important to us. So we'd like people to be able to take control of their AI experience and really feel like the model is theirs and not anybody else's. That's a big motivator. But then as we started to work on this, we also realized there are a few additional benefits. So having this box in your home locally with you could give you a much lower latency, for example. So it opens up a lot more possibilities when it comes to interacting with the agents via voice or via video input, video output. So there actually some new possibilities for consumer experiences that open up once you go with this form factor. So that's something I'm very excited about, because it might even make the whole experience much better than what we're getting from the data center world. Yeah. And I think just the privacy is also going to be important in the personal AI world. So these personal agents that know you deeply, they kind of observe everything that you're doing and then help you proactively. They're going to get access to an incredible amount of sensitive data about you. So basically all of your documents on your computer, everything that you're saying in the room could be stored by a personal agent, because it just makes it more useful if it has that context for you, right? And sending all of that data out to the data center, where there might be some security leaks, or your data might be trained on even if you don't like that. That's really suboptimal. So we're also thinking of it from a privacy point of view.

我认为公司愿景的一部分是，每个人都拥有自己的模型，这种模型会不断学习改进，并为那个人定制。当人们思考如何构建这类模型时，有不同的思想流派。

I think part of the vision of the company is like everyone kind of having their own model that's like kind of a continually learning to improve and to be customized to that person. As folks think about how these types of models may be built, there's like different schools of thought.

其中一种方法是直接修改权重本身，而另一些人则说：“嘿，你或许可以通过一个非常好的记忆系统和提示来实现这一点。”你怎么看待这个问题？你们是在探索多种不同的研究方向，还是对某一种方法非常有信心？

One of them is, you know, going into the weights themselves and updating the weights, and other people saying, "Hey, you might just be able to do this through like a really good memory system and prompting." How do you think about that? And is that still like, you know, are there a bunch of different research directions you all are pursuing, or do you feel pretty convicted in the approach?

是的，对我们来说，哪些工具会成为关键工具，这个问题还相当开放。但根据我构建 AI 模型的经验，有一件事是绝对必需的，那就是端到端训练。你确实需要训练你的智能体去利用个人信息，如果有记忆系统的话，也要利用记忆系统。这非常关键。而如今，我们经常做的是采用现成的专有智能体，这些智能体在编码方面训练很多，但未必经过长时间跨度的训练，去帮助个人利用记忆来协助他们。那么，如何最大化一个人的幸福感呢？这作为奖励函数，还没有真正实现过。所以对我来说，这才是关键方面，因为我观察到，那些为特定任务训练的模型表现最好，而你确实需要在你想要优化的实际信号上进行端到端训练，也就是个人的幸福感。

Yeah. So for us, it's pretty open which of those are going to be the key tools to use. But I'd say from my experience building AI models, the one thing that you really really need is end-to-end training. So you actually want to train your agents to make use of their personal information, to make use of their memory systems if they have one. That's really crucial. And today, often what we do is we take off-the-shelf proprietary agents which have been trained a lot on coding but not necessarily trained on over long time horizons helping an individual using their memory to help them. So how do you maximize somebody's happiness? That's as the reward function, that hasn't really been done yet. So that for me is the key aspect, because I've kind of observed over time that the models that were trained for a particular task have performed the best, and you really want to train end-to-end on the actual signal that you want to optimize, which would be the individual's happiness.

你们是打算让模型在真实环境中运行，然后连接到某种自我报告的幸福感或可穿戴设备上，从而形成一个闭环吗？

Are you going to basically have the models running in the wild and hook it up to, you know, some self-reported happiness or like a wearable that, I guess, closes the loop?

是的，这确实是我们需要解决的问题。但你也可以尝试观察，比如我最终是否帮助了用户。而且模型正变得越来越智能，所以你可能能够判断：“哦，我今天把事情搞砸了，负奖励；我让事情变好了，正奖励。”所以可能性正在朝那个方向发展。这正在进入一个难以验证的领域，仍然非常具有挑战性。所以我们需要用模型来判断你的状态。但随着模型变得越来越智能，这正变得越来越可行。

Yeah, that's really something we got to figure out. But you could also just try to observe, you know, did I end up helping the user? And the models are getting more and more intelligent, so you might be able to figure out, "Oh, actually I made things worse today, negative reward; I made things better, positive reward." So the possibilities are going there. It's moving into this non-verifiable domain that's still very challenging. So we need to use models to figure out how you're doing. But increasingly, with models becoming more intelligent, it's getting possible.

当你设想 River 的未来状态时，你认为哪些模型差异是典型例子？比如，如果随着时间的推移，你和我拥有不同的模型，显然有一些能力如果通用模型具备，大家都会满意，然后还有一些人们可能想要的不同之处。你脑海中浮现的例子有哪些？

As you envision the future state of River, what are the kinds of differences in models that you think of as canonical examples? Like if you and I had different models over time, obviously there's some set of capabilities that if the general models had, everybody would be happy, and then there's different things folks might want. What are the examples that you think of top of mind?

我认为例子真的是无穷无尽的，而且很难想象，因为我们自己还没有经历过。但可能是一些非常微妙的事情，从模型回复你时写多少文字，到它使用什么样的词汇。是更随意还是更正式？所以答案的这些不同方面。今天，我们有一套固定的回复方式，模型会以固定的风格回应你。但未来，这可能会被解开，而且几乎是无限的，因为它会影响它一整天如何帮助你，比如它应该在什么时候联系你，在什么情况下，它是不是太烦人，或者帮助不够？你知道，它会自我校准以适应你。它会了解你在书籍、电影、音乐方面的品味，并能据此调整你的一天。实际上，可能性真的是无穷无尽的。如果你想获得更多灵感，可以看看今天的推荐系统能做什么。比如，如果你使用像 TikTok 或 YouTube 这样的应用，它会找出你的兴趣所在，并让你深入探索你喜欢看的内容。这种体验与给每个用户提供相同视频是完全不同的。这就是我们想要为智能体世界创造的东西。你可以把它想象成一个推荐系统与智能体的混合体，这是前所未有的。所以这是我们非常兴奋的事情。

I think the examples are really endless, and it's just hard to imagine because we haven't experienced it yet ourselves. But it could be really subtle things, from maybe how much text the model writes when it replies to you, to what kind of vocabulary it uses. Is it more casual? Is it more formal? So all of these different aspects of the answer. Today we have like one fixed set of responses, one fixed style in which the model will respond to you. But in the future, this could be untied, and it can be pretty much endless because it will affect how it will help you throughout the day, like when should it reach out to you, in which situations, is it being too annoying or is it not helping enough? You know, it will calibrate itself to you. It will know what your taste is in terms of books, movies, music, and it will be able to adapt your day to that. It's really the possibilities are actually endless. If you want to be a bit more inspired about what's possible, you can look at what recommendation systems are able to do today. So if you use an app like TikTok or YouTube, it figures out what your interests are and allows you to go really deep on what you would like to see. And the experience is totally different from giving the same set of videos to every single user. That's kind of what we want to create for the agent world. And you can kind of think of it as a recommendation system-agent hybrid, which has never been done before. So that's something we're pretty excited about.

你可以想象在企业领域也有类似的情况，你说过你开始创业时是与企业合作开发模型。我的意思是，过去几个月里显然有很多关于拥有自己的模型和训练自己模型的重要性的讨论。感觉今天很多人这样做主要是为了成本和速度，对吧？比如寻找运行更小模型的方法。你认为这会如何随时间变化？你觉得人们会真正尝试推动能力进步，还是说，显然有很多与你提到的相似之处，比如人们有不同的偏好，公司有不同的文化和做事方式。你如何想象那里的相似之处？

You could imagine obviously the parallels in the enterprise space, and you said you're kind of starting the business with working with enterprises on models. I mean, there's obviously been a ton of dialogue in the last months about owning your own models and the importance of training your own models. It feels like today a lot of people are doing that for really for cost and speed, right? Like finding ways to run smaller models. How do you think that changes over time? Do you think folks will actually try to push capabilities forward, or like, even obviously a lot of parallels to what you talked about in people having different preferences, companies have different cultures, ways of doing things. How do you imagine the parallels there?

是的，我认为今天很多人首先选择专有模型。它们拥有最强的能力，而且大家都在用。这是选择它们的一个重要原因。然后他们可能会出于速度、成本的原因定制模型，隐私也可能是一个方面，如果你必须在自己的基础设施上运行，因为你可能有其他客户的数据更敏感。所以这些是我看到的主要原因。但我认为人们选择专有模型的很大原因是这个领域发展太快了。如果你想在公司内部学习如何训练自己的模型，掌握深度学习、强化学习，并充分利用你拥有的内部数据，你必须建立团队，建立基础设施。而且要超越专有模型可能需要数月时间的项目。但到那时，又会出现一个更好的新模型。所以你可能还不如直接用专有模型，我认为这就是过去几年的趋势，但预计这种情况会改变，因为现在开放模型开始变得非常强大，而且我们正在开始利用那些不需要深入了解公司数据和问题的低垂果实。

Yeah, I think today a lot of people go to the proprietary models first. They have the strongest capabilities, and they're used by everybody else. That's a big reason to start out with them. And then they might do custom models for speed reasons, cost reasons, maybe privacy could also be an aspect if you actually have to run this on your own infrastructure because you have maybe other customers that have more sensitive data. So those are the main reasons that I'm seeing. But I think a lot of the reason why people go to the proprietary models is that the field is moving so quickly. If you wanted to learn how to train your own models internally at your company and sort of master deep learning, master reinforcement learning, and get the most out of your internal data that you have, you have to build up a team, you have to build up infrastructure. And it can be a many-month project to be able to surpass the proprietary models. But by that time, there's a new model that's even better. So what you could have just gone with the proprietary model, and I think that's been the way things have been going for the last few years, but expect that to change now that open models are starting to become extremely capable, and we're sort of starting to harness all the low-hanging fruits in terms of what you can do without deeply knowing basically the data and the problems that a company is solving.

所以，如果我是某个领域的专家，我的业务是构建某个 SaaS 工具，比如一个人力资源平台，可以想象我拥有最好的数据，我最了解客户的需求，而且我对这个领域的思考比 Anthropic 或 OpenAI 对我构建的产品的思考要多得多。所以，实际上那些公司最清楚他们的智能体应该如何工作，应该做什么、不应该做什么，以及如何最好地帮助客户。而到目前为止，这还没有真正显现出来，因为我们还在乘着越来越强的专有模型的浪潮。但一旦这股浪潮结束，你就面临一个选择：要么把你所有的数据交给 Anthropic 或 OpenAI，让他们更好地进行后训练，要么你自己开始做，试图榨出额外的能力。我认为很多人会选择后者，因为否则你就是在放弃你整个业务的基础。你放弃的是让你在竞争对手面前占据优势的东西。

So if I'm an expert in a certain area, my business is building a certain SaaS tool like an HR platform, for example, can imagine that I have the best data, I have the best understanding of what the customers need, and I've thought about the domain quite a lot, much more than Anthropic or OpenAI would think about the product that I've built. So really those companies are in the best place to decide how their agents should be working, what they should and shouldn't do, how to best help the customers. And so far that hasn't really hit yet because we're still riding the wave of stronger and stronger proprietary models. But as soon as that runs out, then you have a choice: either you give all of your data to Anthropic or OpenAI for them to post-train their models better, or you start to do it yourself to try to squeeze out additional capability. And I think a lot of people will opt to go for the second one, because you're kind of giving away the whole foundation of your business otherwise. You're giving away this thing that gives you an advantage over all your competitors.

你觉得专有模型会不会真的慢下来？因为显然有无限的数据预算，他们似乎会去购买各个领域的数据。你知道，我想我能看到几种不同的论点，但其中一个可能是，比如，拿银行来说。你可以真的去雇一群前银行家，他们可以为你标注大量数据，并在不同环境中做很多事情，然后突然你的模型在金融方面就变得相当不错了。我觉得一个有趣的问题是：第一，我们是否期望模型在其他领域继续进步；第二，企业的数据实际上有多独特，相对于外面存在的数据标注大军。

Do you think the proprietary models like do slow down because obviously there's endless amounts of data budgets to they seem to go purchase data across a bunch of different domains. You know, I guess I can see a few different arguments, but one might be like, you know, take a bank for example. It's like you can actually go hire a bunch of people that were ex-bankers and they can label a lot of data for you and do a lot of things in different environments, and suddenly your model gets pretty good at finance. And I think it's an interesting question of like, one, you know, do we expect the model progress to continue in these other domains, and two, like how unique is actually an enterprise's data versus like the army of data labelers that exists out there.

是的。所以训练这些模型实际上存在收益递减。我们想让预训练模型和整体模型变得越强，就需要把越多的 GPU 组合在一起，收集越多的高质量数据。AI 公司在 Scaling 方面做得非常出色。他们在收集数据、调优模型、不断改进、添加新的创新算法和能力方面投入了更多的努力。但最终总会有一些东西要放弃。所以，在某个时候，你不可能用 GPU 覆盖整个地球。所以最终，AI 公司能够实现的模型能力会有所放缓。我认为我们正开始接近那个点。与此同时，还有另一件事正在发生，那就是这些模型在前沿变得如此强大，以至于你可能不再想发布它们，或者你甚至可能受到批评，或者可能会有一些监管行动阻止你发布下一代模型。与此同时，开源模型变得越来越强。所以我认为作为专有模型的构建者，你开始有点被挤压了。你把模型做得太好，就不被允许发布。但开源就在你身后，每个月都在变得更好。所以我认为作为专有模型的构建者，这实际上不是最好的位置。所以我认为 Anthropic 的出路是创新。这是他们在过去几年里所做的，使他们能够达到今天的位置。所以他们必须提出一些真正新的基本想法，关于如何让模型更有用，如何让每个人都感受到他们正在构建的东西的好处。探索新的领域，探索新的影响方式和创收方式。

Yeah. So training of these models actually has diminishing returns. So the stronger we want to make the pre-trained model and the model overall, the more GPUs we have to put together, the more high-quality data we have to gather. And AI companies have done an incredible job at scaling up. They're putting in so much more effort at gathering the data, tuning the models, improving them again and again, adding new, you know, innovative algorithms and capabilities into them. But eventually something will have to give. So at some point, you know, you can't cover the entire earth with GPUs. So eventually there will be a bit of a slowdown in terms of what the AI companies are able to do in terms of the capabilities of their models. And I would argue we're starting to get close to that point. At the same time, there's another thing that's happening, which is these models are starting to get so capable at the frontier that you might not want to release them anymore, or you might even get criticized, or there might be some regulatory action that prevents you from releasing the next generation of the model. At the same time, open models are getting stronger and stronger. So I think as a proprietary model builder, you're kind of starting to get squeezed in a little bit. You make the model too good, you're not allowed to release it. But then open source is just right behind you getting better and better every month. So I think it's actually not the best place to be as a proprietary model builder. So I think that the way out for Anthropic is to innovate. So this is the same thing that they've done the past few years that's allowed them to be in the place where they are today. So they have to come up with some really new fundamental ideas about how to make the models more useful, how to make everybody feel the benefit of what they're building. Explore new domains, explore new ways of having impact, generating revenue.

因为基本上能力提升正在放缓。

Because you basically the capability improvements are slowing down.

是的，完全正确。如果他们不这样做，那么你可能实际上不得不保持模型私有，因为他们开始跨越这个关键阈值，现在你真的必须仔细考虑是否可以让任何人访问模型。然后在后训练方面，所做的努力真的令人难以置信。所以你基本上是在召集世界上每个主题的所有专家，然后让他们与你合作，将所有知识集中到一组权重中，然后交付给所有客户。所以扩大这个操作规模也非常具有挑战性。那里也有一些收益递减。我认为出路是走向分布式。所以世界上已经有所有这些特定领域的专家。他们在现有的公司里，在大学里，等等。我认为我们应该给人们一个机会，基本上独立地创建模型的改进，并基本上在本地为他们自己、他们的公司、他们的团队、个人进行后训练，而不是试图捕获所有的人类知识，然后按 token 出售。所以这只是我个人的理念，这也是我们在 Riveri 想要做的事情。弄清楚我们如何允许人们基本上在本地进行他们自己的后训练。

Yeah, exactly. And if they don't, then you might actually have to keep the model private because they're starting to cross this critical threshold where now you really have to think carefully about whether you can give anyone access to the model. And then on the post-training side, the effort that's being made is really incredible. So you're basically sort of assembling all the world's experts on every single topic and then having them work with you to centralize all that knowledge flows into one set of weights, you know, and then gets delivered to all the customers. So scaling that operation up is also incredibly challenging. There are also some diminishing returns there. And I think that the way out there is to go distributed. So there are all these experts out in the world already for their particular domains. So they're inside of the existing companies, they're at the universities, and so on and so forth. And I think we should give people an opportunity to basically independently create improvements to models and essentially post-train them locally for themselves, for their company, for their team, for the individual, rather than trying to capture all of human knowledge essentially and then sell it by the token. So just my personal philosophy, and that's the type of thing that we want to do at Riveri. Figure out how do we allow people to do their own post-training essentially locally.

有趣的是，我觉得我们正处于这样一个时刻：你在一个领域购买大量数据，专注于那个领域，然后这增加了那些能力，你是一个一个地来，这感觉与预训练世界非常不同，在预训练中，你达到某个阈值，然后开始看到全面的巨大泛化。我认为有些人已经提出，同样的事情会发生在后训练的强化学习中，最终，你知道，我不知道，你添加了第 30、40 或 50 个领域，突然你从把所有东西放在同一个地方获得了很多好处，而不是仅仅在这个地方训练金融或在这个地方训练医疗。你认为我们会在强化学习中看到这种泛化和泛化的好处吗？还是说，总是有意义的是，拿一个非常强大的预训练模型，然后专门应用一个领域？

It's interesting because I feel like we're in this moment where you know yeah you buy a lot of data in a domain and you focus on that domain and that like then adds those capabilities and you're kind of going one by one and it feels very different than you know the pre-training world where it's like you get to some certain threshold and then you start to see like massive generalization across the board right and I think some people have posited that like the same thing would happen in post-training in RL like eventually you know I don't know you're adding domain 30 or 40 or 50 and suddenly it becomes like you get a lot of benefits from having everything in the same place versus you know just training on you know finance in this place or healthcare in this place like do you think we see that generalization and that benefit of generalization down the line for RL or is it like going to always make sense to just take some really strong pre-trained model and like you know uh apply one domain specifically to that.

我认为我们正在看到泛化，从某种意义上说，这些模型对世界有着惊人的知识，并且拥有广泛的技能，它们现在可以进入任何领域，并带来来自其他所有地方的知识。

I think we're seeing generalization in the sense that these models have such an incredible knowledge of the world and they have such a wide set of skills that they can now go into any domain and bring in all the knowledge from everywhere else.

我认为编码就是一个很好的例子。至于后训练中所有不同内容之间是否存在很多协同效应，说实话我不太确定。我觉得有太多专业领域和专业技能，你只会在某个地方真正需要它们。我怀疑这些内容到底有多少真正提升了模型的整体智能。目前看来，很多智能似乎来自编码和数学训练等。

I think coding is a great example of that. And then I think whether there is a lot of synergy between all the different things that are being post-trained, I don't really know to be honest. I feel like there are so many specialized domains, so many specialized skills out there that you only really need in one place. I wonder how much of that really is contributing to the overall intelligence of the model. A lot of the intelligence seems to come from coding and math training and things like that at the moment.

是的，这似乎是一个很有趣的问题，即企业对自己的模型进行后训练有多大意义，对吧？比如制药公司拥有相当专有的药物试验数据集。还有芯片设计公司，有些情况下你会觉得：“是的，这些公司拥有相当独特的数据，这些数据可能不会公开。”但对我来说不太清楚的是，普通银行是否真的拥有那些最终不会落入闭源模型提供商手中的数据。

Yeah, this seems like a really interesting question for how much it makes sense for enterprises to post-train their own models, right? It's like there are pharmaceutical companies that have datasets of drug trials that are pretty proprietary. There are chip design companies, and there are certain things where you're like, 'Yes, those companies have quite unique data that probably isn't out there.' Maybe it's less clear to me if your average bank actually has data that isn't just going to be in, you know, that eventually your closed source model providers get access to.

是的，如果这些企业无法保持竞争优势，无法利用自己的杠杆和知识产权来改进自己的模型，我们可能会看到其中一些陷入困境。所以我认为这确实取决于行业，这也是你猜测事情可能走向以及转型可能如何发生的方式。我想说，最糟糕的情况是你的所有知识都在互联网上，对吧？如果让你公司独特和特别的一切都已经公开并且可以被抓取，那是一个非常危险的位置。

Yeah, we might see some of those businesses struggle if they're not able to retain their competitive advantage, if they're not able to use their own leverage, their own IP, to make their own models better. So I think it really depends on the industry, and that's kind of how you can guess where things might go and how the transformation might happen. I'd say the worst place you can be in is if all of your knowledge is on the internet, right? If everything that makes your company unique and special is already out there and can be scraped, that's a very dangerous place to be.

这恰好与 River 的整体理念相契合，那就是保持这种智能的本地化。你知道，就是与这些模型交互和使用它们的方式。

It kind of exactly ties into the overall thesis of River, which is like, you know, keeping this intelligence local. You know, the ways you're interacting with and using these models.

实际上，我认为我们是有选择的。对。我认为作为一家公司，你可以选择以某种方式将数据交给像 Anthropic 和 OpenAI 这样的公司，比如通过 API 或其他途径，甚至可能将数据卖给他们以获利，或者你可以开始利用这些数据来构建自己的模型。所以我看到两条不同的路径。显然，一条通向智能的更加集中化，另一条则带来 AI 的更多分布式收益。

I'd say we have a choice actually. Right. I think as a company you can choose to give your data to somebody like Anthropic and OpenAI in a way, you know, through the APIs or other means, or maybe selling your data to them even if you want to turn a profit on it, or you can start to leverage it to build your own models. So I see these two different paths. You know, one obviously leads to more centralization of the intelligence, and one leads to more distributed benefits of AI.

在你今天构建的所有东西背后，似乎任何试图构建自己的模型或微调模型的人都在使用前沿的中国开源模型。显然，过去几个月关于这些模型有很多讨论。显然，我想你认为前沿开源模型对世界非常有益。你如何看待今天这个生态系统全是中国的，以及政府讨论可能禁止这些模型的事实？

Behind all the stuff you're building today, it seems like anyone that's trying to build their own models or fine-tune models is using the cutting edge Chinese open source models. And obviously there's been a big discourse these past months about these models. Obviously, I imagine you think cutting edge open source models are really good for the world. What do you make of the fact that this ecosystem is all Chinese today and then the government talking about potentially banning these models?

是的，首先我认为拥有强大的开源模型是件好事。这确实降低了任何人进入 AI 领域、定制模型、利用模型为自己谋利并真正掌控自己命运的门槛。但同时，这也让中国实验室处于一种优先地位，他们也能施加一些控制。例如，他们将来可能停止发布开放权重，而每个人都依赖他们。这对美国经济来说不是好事。他们还可能更改这些模型的许可证。我们已经经常看到，如果你的公司收入超过某个阈值，就需要签订合作协议，规则可能有所不同。这是人们施加压力并保持对这些开放权重控制的另一种方式。还有一个更理论化但理论上可能的问题，就是权重可能以某种方式包含后门或我们不喜欢的行为。

Yeah, I think first of all it is good that we have powerful open models out there. So it really lowers the bar for anyone who wants to get into AI, who wants to customize models, who wants to use models to their own advantage and really have that control of their own destiny. At the same time, it does put the Chinese labs in a kind of preferred position where they also can exhibit some control. So for example, they might stop releasing their open weights in the future and everyone relies on them. That's not a great thing for the US economy. They could also change the licenses to those models. So we often see that already that there are certain times where if you're a company that has more than X revenue, then you need to enter into a partnership agreement where the rules might be a bit different. So that's another way people can exert a bit of pressure and stay in control of these open weights. And then one issue that's more theoretical but it is theoretically possible is for the weights to somehow have backdoors or behaviors that we would find undesirable.

你对此有多担心？

How worried about that are you?

是的，今天我并不怎么担心，因为我的感觉是，如果这类事情存在，我们至少会在野外发现一个例子。既然我们还没有看到任何相关报道，而且据我所知，将这种行为、后门植入权重的技术目前还不够先进，无法真正避免检测且不留下任何痕迹。所以我认为我们现在还处于早期阶段，我并不太担心，但理论上这种担忧是存在的。而在未来两年，随着这项技术变得更加复杂，而且可能 LLM 和智能体在各个领域（包括军事）变得更加重要，这是我们需要注意的事情。所以我认为美国拥有最好的开源模型至关重要。这是我非常支持的，我觉得我们都应该想办法让美国团队训练出绝对最好的开源模型，不仅仅是美国最好的开源模型，而是全球最好的。所以是的，我非常支持这一点。

Yeah, today I'm not very worried at all because my sense is that if these kinds of things existed, we would already have at least one example of it happening in the wild. Since we haven't seen any reports of it, also the technology that I'm aware of for planting these behaviors, planting these backdoors into weights is not advanced enough today to really avoid detection and to not leave any hints. So I think we're kind of early on this right now and I'm not too concerned, but in theory the concern is there. And over the next two years, as this tech becomes more sophisticated, and you could actually and maybe LLMs and agents become more important as well in all kinds of domains including the military, that is something that we should have on our radar. So I would say it's really critical that we have the best open models in the US. So that's something that I'm very supportive of, and I feel like we should all figure out how can we get a US team to train the very best open model period, not just the best US open model but the best one worldwide. So yeah, definitely super supportive of that.

是的，这似乎是一个有趣的商业模式问题，即你最终如何将其转化为收益，但它就像一种公共产品，也许可以找到一群人。我的意思是，Nvidia 已经在大力补贴这类事情使其成为可能，所以也许你会看到。

Yeah, seems to be an interesting business model question of how you end up turning it, but it's like such a public good that maybe find a bunch of folks. I mean Nvidia already is kind of subsidizing this stuff quite dramatically to make it possible, and so maybe you'll see.

也许也会是我们，对吧？我们有过 xAI 从零开始的经验，在不到两年的时间里，我们有了 Grok 3，然后还有 Grok 4，它们都非常接近前沿。所以我很想训练出美国最好的开源模型，但现在我们非常感兴趣的是如何围绕这些开放权重建立业务，使公司能够自我维持，而我们在 API 方面已经取得了良好进展。我们正在构建的另一个前提是什么？

Maybe it's going to be us as well, right? So we've had that experience with xAI of starting from scratch and then within less than two years we had Grok 3 and then Grok 4 as well, which were very much at the frontier. So I'd love to train the best US open model, but right now we're very interested in how to build a business around these open weights so that the company can be self-sustaining, and we're making good progress on that with the API. What's the other premise of what we're building?

好吧，既然你在这里，如果不问你一些你过去的角色，那我就太失职了，因为我认为你显然亲眼目睹并领导了过去几年发生的很多事情。你知道，也许从你所说的 AI 进展比你预期的更快开始。

Well, I'd be remiss if, you know, having you here not to ask a little bit about some of the past roles you've had, because I think you've obviously seen, you've had a front row seat and kind of led a lot of this stuff that's happened over the past years. You know, maybe to start like you've said AI progress took off faster than you expected.

过去三四年里，有没有哪个时刻让你特别惊讶？

Is there a moment or two that really surprised you these past, you know, I don't know, three or four years?

但我觉得疯狂的是，我对进展并不太惊讶。只是亲身经历这一切太疯狂了。这和你想象中完全不同，几乎像在做梦。所以当我开始对 AI 产生兴趣时，我是一名物理学家，在大型强子对撞机工作，在那里读博士。物理学教会你要有远见。我觉得这种思维也影响了物理学家，他们会开始思考未来会是什么样子。所以当我意识到，比如 AlphaGo 出现，DeepMind 创造了一个非常强大的围棋 AI，那时我就明白了 AI 领域将发生惊人的事情，我最好转行。

But I think the crazy thing is I'm not too surprised by the progress. It's just crazy living through it. It's totally different from when you imagine it. It's almost like being in a dream. So when I started to get interested in AI, I was a physicist working at the Large Hadron Collider, doing a PhD there. In physics, they teach you to really think ahead. I think some of that rubs off on physicists as well, and they start to think about how the future will look. So when I realized that, for example, AlphaGo happened, and DeepMind was able to create a Go AI that's very powerful, that's when it clicked for me that amazing things are going to happen in AI, and I better switch over.

你和很多杰出的研究者都从物理学转行过来。很有意思，这么多人都是从那个领域起步的。

You and a lot of prominent researchers made the move from physics. It's very interesting how many folks started in that world.

是的，我觉得这是学习如何思考这些难题、如何从第一性原理解决问题的一个很好的方式。所以这为从事 AI 工作做了很好的准备。我决定加入 DeepMind 担任研究工程师，这真的很有趣，因为我既看到了算法方面，也看到了如何解决这些研究问题、如何训练模型，以及如何构建分布式系统、如何提高训练和推理的效率等等。然后我参与了 WaveNet 的文本到语音生成工作，之后对强化学习产生了浓厚兴趣，最终成为 DeepMind 星际争霸项目的技术负责人。

Yeah, I think it's just a great way to learn the ropes about how to think about these tough problems and how to solve problems from first principles. So it's a great preparation for doing AI work. And I decided to join DeepMind as a research engineer, which is really fun because I got to see both the algorithms side and how to solve these research problems, how to train the models, and also how to build distributed systems, how to improve efficiency of training and inference, and so on. Then I worked on text-to-speech generation with WaveNet, and then got really interested in reinforcement learning, and ended up being a tech lead for the StarCraft project at DeepMind.

那是一个非常有挑战性的强化学习项目。在某种程度上，它超越了那个时代。

It was a really challenging RL project. It kind of in some ways it was ahead of its time.

百分之百。那似乎比我们今天优化强化学习的很多环境都要复杂得多。

100%. That seems to be a much more complex environment than a lot of things we're optimizing RL on today.

没错。我们必须想办法应对这种复杂性，而我们开发的一些策略，我们立刻就能看到它们将来如何应用到语言模型上。比如在星际争霸项目中，我们做了很多模仿学习。我们收集了各种技能水平的人玩星际争霸的回放，从初学者到接近职业选手，然后我们先进行模仿学习，之后才转向强化学习。这里的奖励是可验证的，因为这是一个双人游戏：赢的人获得正奖励，输的人获得负奖励。我们发现，这两者的结合——模仿学习，然后大规模训练模仿，再用强化学习进一步改进策略——非常强大。这也是今天这些智能体训练的方式。

Exactly. We had to figure out how to deal with that complexity, and some of the strategies we developed, we could immediately see how they could apply to language models as well down the line. So for example, with StarCraft, we did a lot of imitation learning. We got all these replays of people playing StarCraft at various skill levels, from beginners to almost professional players, and then we started with imitation learning and only then switched to RL. The rewards here are verifiable because it's a two-player game: whoever wins gets a positive reward, if you lose you get a negative reward. The combination of those two—imitation and then large-scale training for imitation, and then reinforcement learning to further improve the policy—that's really powerful, we found. And that's still the way that these agents are trained today.

所以在 DeepMind 的星际争霸项目之后，我们意识到编码将是下一个大事件。当时还早，远没有商业上可用的代码模型。我们把它看作是星际争霸的自然延伸。所以我们启动了 AlphaCode 项目。当时我强烈感觉到 LLM 在代码上表现不佳，我认为原因是它们无法思考——我们基本上没有给它们足够的时间来思考问题。当人类解决难题时，他们必须思考一会儿，然后会发生一些神奇的事情，让他们更好地理解问题并解决它。当时我们还没有机制让 LLM 进行那种推理。所以我真的觉得我们需要在这方面做研究。谁在做这个？我意识到 OpenAI 刚刚成立了一个推理团队，正是为了解决这个问题。所以我加入了 OpenAI，搬到了加州，参与了那里的推理工作，但也和 Greg Brockman 一起帮助了一些大型预训练运行。所以我也看到了大规模预训练的一面，这真的很有趣。

So really after the StarCraft project at DeepMind, we actually realized that coding is going to be the next big thing. It was a bit early at the time, way before there were any commercial capable code models out there. We saw it as a natural extension of StarCraft. So we started this project called AlphaCode at the time. I had the strong feeling that LLMs were not performing well on code at all at the time, and I thought the reason was that they're not able to think—we don't give them enough time to think about the problem basically. When humans solve tough problems, they have to think for a moment or two, and something magical happens that allows them to understand the problem much better and solve it. We didn't have a mechanism back then for LLMs to do that kind of reasoning. So I really felt like we got to do some research on this. And who's working on that? I realized OpenAI had just started up a reasoning team exactly to figure out this problem. So I joined OpenAI, moved to California, and was involved with the reasoning effort there, but also helped out on some big pre-training runs together with Greg Brockman. So I got to see the large-scale pre-training side there as well, which was really fun.

是的，很多这种推理的东西似乎在真正大规模预训练之后才起作用。所以在 OpenAI 推理工作的早期，几乎就有这种信念，即当模型扩展并变得更好时，这会成功，这真是令人印象深刻的直觉。

Yeah, a lot of this reasoning stuff seems to have worked post, you know, really large-scale pre-training. So it seems like in the early days of the reasoning work at OpenAI, there was almost just this conviction that this is going to work when models scale and get better, which is really impressive intuition.

是的，没错。当时我们还没有解决方案。解决方案最终基本上是 o1 模型，以及后来在 OpenAI 进行的研究，解锁了这种推理方法，即模型在给出最终答案之前可以生成思考令牌，然后使用强化学习来改进其思考过程和思考能力。这最终成为我们一直在寻找的关键。那是几年后的事了。然后在 OpenAI，我只是觉得这些人会做得非常好。

Yeah, exactly. At the time we didn't have the solution just yet. The solution ended up being essentially the o1 model and the research that happened later at OpenAI to unlock this reasoning approach, where the model has thinking tokens that it's allowed to generate before giving the final answer, and then use reinforcement learning to basically improve its thinking process and thinking ability. That ended up being really the key that we were looking for. That happened a few years later. And then at OpenAI, I just felt like these folks are going to do unbelievably well.

你如何将这与 DeepMind 相比？因为显然 DeepMind 有众多重量级人物。我想几乎每个人都在某个时候在 Google Brain 待过。随着人们的反思，也许那里有更多不同类型的赌注，或者更相信强化学习，而较少关注其他。我不知道。你经历过两者。所以我很好奇你是怎么想的。

How do you compare that to DeepMind? Because obviously DeepMind had a who's who of heavy hitters. I think almost everybody did a stint through Google Brain at some point. And as people have reflected on it, maybe there were more different kinds of bets made, or more of a belief in reinforcement learning, and less of a focus. I don't know. You lived through both of them. So I'm curious how you thought about that.

老实说很难说，因为 DeepMind 有这么多杰出的思想家，我认为一些正确的想法肯定在流传。但我认为 OpenAI 的优势在于他们是一个较小的团队，他们能够在内部达成很好的共识，坚信这是正确的方法。我认为这需要那种关键的人才密度，以及人们沿着正确的思路思考，而他们做到了。

It's really hard to say honestly, because there were all these extraordinary thinkers at DeepMind, and I think some of the right ideas were floating around for sure. But I think what helped OpenAI was that they were a smaller team, and they were able to get great alignment within themselves, great conviction that this is the right approach. I think it requires that kind of critical mass of talent density and people thinking along the right lines, which they were able to do.

所以即使作为弱势一方，他们也取得了巨大进展，而且我认为当时我们很清楚 OpenAI 在很多方面会赢。那时我就开始想，哇，如果世界上只有一家 AI 公司控制着最强大的模型，决定做什么，那真是太不幸了。我们甚至会考虑如何为每个人设立全民基本收入（UBI），这样人类就能在 OpenAI 控制模型的同时获得报酬，但这让我很不舒服，因为我一直是个开源爱好者，多年来也是 Linux 的重度用户。所以我觉得这才是分配利益的方式。归根结底，从哲学上讲，如果你想一想，模型的预训练是基于全人类的知识，基于我们在网上创造的所有文本。这真的是全人类应该共同拥有的东西。但事实上，没有人技术上拥有它，你可以直接下载它，这基本上让你能够压缩知识、提炼知识，把它变成可操作、有用的东西，也就是模型权重。在我看来，这才是 AI 模型构建者真正在做的事。从第一性原理出发，对我来说，让这些预训练检查点免费供人们使用是最合理的，对吧？什么才是公平的。但显然，AI 实验室在算力、专家资源方面投入了巨大努力，这些专家研究如何设置算法、如何训练模型、如何在这个领域创新。所以他们的参与肯定带来了巨大好处，但我觉得我们没有考虑到网络文本中蕴含的所有人类智慧。

So even as the underdog they were able to make huge progress and I think to us at the time it was pretty clear that OpenAI was going to win in many ways. And then already at that point I started to think like, wow, that's kind of unfortunate if there's only one AI company in the world that controls the most capable models and decides what gets done. We would even think about how to set up UBI for everybody so that humanity can be paid while OpenAI essentially controls the models, which just didn't sit right with me because I was always a big open source fan and big Linux user over the years. So I felt like that's the way to distribute the benefits. In the end, philosophically, if you think about it, the pre-training of the model happens on all of humanity's knowledge, all the text that we've created out there on the web. It's really something that humanity as a whole should own. But the fact that nobody technically owns it, and you're able to just download it, basically allows you to compress the knowledge, to refine it, to turn it into something that's actionable and useful in the form of the model weights. And that's really what the AI model builders are doing in my opinion. And to me, philosophically, it makes the most sense for those pre-trained checkpoints to go out and be free for people to use, just thinking from first principles, right? Like what's fair here. But obviously the AI labs are putting in tremendous effort in the form of compute, in the form of all these experts that figure out how to set up the algorithms, how to train the model, how to innovate in the space. So there's definitely huge benefit that comes from having them involved, but I feel like we're not accounting for all the human ingenuity that went into the text on the web.

但显然，所有数据都是开放的，这让你们 xAI 在短时间内就能接近前沿水平。我觉得看到这一切发生很有趣。这是怎么做到的？

But obviously the fact that all that data is open allowed you, in the case of xAI, in short order to kind of get back up to pretty close to the frontier. And I feel like that's been interesting to see that play out. How did that come about?

是的，基本上我是偶然遇到埃隆的。我碰巧在特斯拉办公室和团队讨论他们在做什么。他正好在附近，所以我有机会和他聊聊，这非常吸引人，因为他也是一个伟大的思想家，对人性、技术以及两者之间的一切都思考得很深。我觉得我们在所有这些问题上看法一致，比如世界上应该有另一个竞争者，另一个 AI 实验室，有着不同的使命，对模型应该如何开发以及它们在现实世界中应该如何表现有不同的理念。所以我想我们从那时起就有了联系。后来，当他想开始新事业时，他联系了我，这就是 xAI 的起源。然后从那时起，我和一些想找新事情做的老朋友谈了谈，我们成功聚集了一群很不错的人。我认为这对任何 AI 努力都至关重要——你必须获得足够的人才，合适的人，他们愿意努力工作并做出伟大的事情。这真的是我们能否做到的头号问题，取决于我们能否把人聚在一起。

Yeah, so basically I ran into Elon randomly. I happened to be at the Tesla office talking with the team there about what they're up to. And he happened to be around, so I got a chance to talk to him, which was pretty fascinating because he's such a great thinker as well, thinking very deeply about humanity, technology, and everything in between. And I think we kind of saw eye to eye on all these things, like the idea that maybe there should be another competitor in the world, another AI lab that has a bit of a different mission, different philosophy around how the models should be developed and basically how they should behave when they're out in the wild. So I think we had that connection from then. And then later on, when he wanted to start something new, he reached out to me, and that's kind of how xAI started. And then from then on, I talked to some old friends that were looking for something new to do, and we managed to get a pretty good group of people together. And I think that's critical for any AI effort—you have to get that critical mass of talent, the right kinds of people that want to work pretty hard and do something great. That was really the number one question of whether we were able to do it. It would depend on whether we got the people together.

然后从那里，几乎从零开始，你们慢慢建立了公司，训练了模型。然后在不到两年的时间里，你们就达到了前沿水平。你们做了一些非常疯狂的事情，比如著名的 Colossus 数据中心，我记得是在 120 天内就建成了。从你们能以如此快的速度完成这些事情中，你学到了什么教训？

And then from there, starting pretty much from scratch, you sort of slowly built up the company and trained the models. And then within less than two years, you were able to get to the frontier. You did some pretty crazy things, famously like getting Colossus up and running in, I think, 120 days. What lessons did you take away from just the speed at which you were able to do a lot of that stuff?

是的，我认为这只需要极大的专注和跳出框框的思考。这很大程度上是由埃隆推动的，因为他看到了我们需要快速上线更多 GPU 以增强竞争力，特别是用于训练。所以他决定，让我们建自己的数据中心，我们不想依赖任何人。在我看来，他确实是世界上最擅长做这件事的人，因为他能看到大局，知道需要做什么，有哪些单独的组件和步骤，然后以非常专注的方式，从第一性原理出发，找出如何加速项目的每个方面。然后他能够走出去，做其他人可能都不知道的事情。

Yeah, I think it just requires tremendous focus and thinking outside the box. That was really very much driven by Elon, because he saw the need for us to bring more GPUs online very quickly to be more competitive, to train in particular. And so he decided, let's build our own data center, we don't want to depend on anybody else. And he's really the best person in the world to do that, from what I've seen, because he's able to see the big picture of what needs to be done, what are the individual components and steps, and then in a very wave-focused manner, figure out from first principles how to accelerate every aspect of the project. And then he was able to go out and do things that other people might not actually be aware of.

当这个提议首次提出时，从时间线的角度来看，你觉得这很疯狂吗？

Did you think it was crazy when it was first proposed, on timeline perspective?

我确实觉得疯狂，但那时我已经足够了解埃隆，知道我们能完成。所以对他有极大的信心。然后让 Colossus 成功的关键就是质疑其他人建设数据中心的方式，因为当我们问别人建一个新数据中心并放入这么多 GPU 需要多长时间时，我们得到的报价都超过一年。原因是他们必须建一个新的外壳，比如一个新的数据中心建筑，并且有一个非常瀑布式的建设流程。而且你经常要面对分包商的分包商的分包商。所以在这个过程中会损失很多效率。埃隆有完全相反的思维方式——我们想自己做，我们想成为总承包商，这里有各种解决问题的方法，别人甚至都没意识到是可能的。所以这有点像在项目的每个方面找到矩阵中的故障。所以，和他一起做这件事真的非常迷人。

I did, but I also knew Elon well enough at that point that I knew we'd get it done. So I had tremendous confidence in him. And then the key to making Colossus work was just questioning how everybody else was building data centers, because when we talked to people about how long it would take to build a new data center and put this many GPUs inside, we would get quotes way above a year. The reason is that they had to bring up a new shell, like a new data center building, and had this very waterfall-like process for building it. And you also often deal with subcontractors of subcontractors of subcontractors. So a lot of efficiency gets lost in the process. Elon has the totally opposite mindset—we want to do it ourselves, we want to be the general contractor, and here are all these ways of solving our problems that other people haven't even realized are possible. So it's kind of how to find the glitch in the matrix on every single aspect of the project. So yeah, really fascinating working with him on this.

我相信我们的听众会非常好奇和他一起工作是什么感觉，以及你如何将那个环境与你在 DeepMind 和 OpenAI 的经历进行比较——这三个可能是非常不同的环境和文化。

I'm sure our listeners would be super curious about what it's like working with him, and obviously how you compare that environment to your previous experiences at DeepMind and OpenAI—three probably very different environments and cultures.

我想说，那次初次交谈之后，我之所以非常想和他共事，是因为我能感受到他对工程师的尊重。作为一名研究工程师，我经常在研究和工作之间切换。我并不是说这些 AI 实验室里一定有双重标准，但确实有头衔区分研究科学家和研究工程师，而我觉得自己真的受到了埃隆的尊重，感觉他就像是我在合作的另一位团队成员。他非常关注工程会议，直接与做实际工作的人交谈。他深入探究公司所做的每一件事的细节，你能感受到他对工程师的尊重，对他们所解决问题的尊重。这是其中非常有趣的一面。此外，他精力充沛，每天带着微笑来上班，想要从头再解决所有最困难的问题。这种能量你也能在团队中感受到。我认为，在一个人们真正渴望成功、带着巨大精力和动力进来的地方工作，真的令人兴奋。

I'd say that the reason I was really interested in working with him after that initial chat is that I could sense the respect that he has for engineers. So, as a research engineer, it's kind of jumping between research and engineering quite a bit. And I wouldn't say that there's a necessarily double standard at these AI labs, but definitely you've got titles that reflect your research scientist, your research engineer, and I just felt really respected by Elon and felt like we were seeing that he was another team member that I was working with. He focuses a lot on engineering meetings, so really talking directly to the people doing the actual work. He dives tremendously into the details of everything that the company does, and you can feel the respect for the engineers, the respect for the problem that they're working on. That was a really fun aspect of it. Also, he has this tremendous energy, and every day he comes to work with a smile and wants to tackle all the hardest problems once again from the beginning. That energy you can feel in the team as well. I think it's really exciting to work in a place where people really want to succeed and come in with tremendous energy and motivation.

自从你离开后，公司做了一些相当有趣的事情，显然是被 SpaceX 收购，然后又收购了 Cursor。你对收购 Cursor 怎么看？

Since you left, the company's done some pretty interesting things, obviously getting acquired by SpaceX and then acquiring Cursor. What did you make of the Cursor acquisition?

是的，我认为这是一步非常聪明的棋。它基本上让他能够在编码模型上大幅领先。Cursor 既有一个非常熟悉编码的优秀团队，又是一个很多人都在使用且重度依赖的产品。而且有大量数据可以用来真正启动训练，并大幅超越 XI 当时的状态。

Yeah, I think it's an incredibly smart move. I think it's basically allowed him to jump way ahead on coding models. So Cursor both has a great team that's very familiar with coding. There's a product that a lot of people out there use and have also used Cursor quite heavily. And there's a lot of data that you can use to really kickstart your training and tremendously improve over what the state where XI was at.

是的。那么，来自真实世界、人们实际使用的编码数据，是不是比从数据标注员和其他供应商那里获得的数据更有价值得多？

Yeah. And so is that coding data, you know, from the real world from people using it like, you know, just that much more valuable than kind of the stuff you get from data labelers and then like the other suppliers?

我认为关键在于数量，如果你有足够的数量，那真的非常宝贵。此外，构建正确的强化学习环境也有巨大价值。基本上，设置这些可验证的任务，让智能体负责做某事，比如修复代码库中的 bug，然后检查它是否真的正确完成了，以及是否以正确的方式完成。这确实是编码模型取得大量改进的关键。这是硬币的另一面。在这个过程中，你可能需要两种类型的数据。

I think it's the quantity of it that is really, if you have it in quantity, that's really really valuable. There's also a tremendous value in building the right kinds of RL environments. So basically setting up these verifiable tasks where the agent is tasked with doing something, fixing a bug in the codebase, and afterwards you check did it actually do it correctly, and also did it do it in the right way. That's really what's been able to deliver a lot of improvements in coding models. And that's kind of the other side of the coin. You kind of want both types of data in the process.

是的。但显然，看到这些部署之后，更容易弄清楚要构建什么样的环境。我敢肯定，当它实际上是自己的模型在 Cursor 框架中时，会容易得多，但即使是 Claude 或其他模型在 Cursor 框架中，可能仍然非常有价值。

Yeah. But obviously it's way easier to figure out what environments to build having seen these things deploy. I'm sure way easier when it's actually your own model in the Cursor harness, but even Claude or whatever in the Cursor harness probably still really valuable.

是的。显然，我并不知道 Cursor 团队带来了什么样的数据，以及他们是如何在模型性能上实现如此巨大的飞跃的，但我得说，他们如此迅速地提升了 Gawk 模型的编码能力，这非常令人印象深刻。而且我敢肯定，我们还没有看到最大的飞跃。所以还会有一些额外的模型即将推出。

Yeah. And obviously I don't have any insight into what kind of data the Cursor team brought in and how they were able to make such a huge leap in model performance, but I'd say it's super impressive how quickly they managed to ramp up the coding abilities of the Gawk models. And I'm sure we haven't even seen the biggest jumps yet. So there's some additional models that are coming out.

你觉得目前持续模型进步的最大约束是什么？是数据吗？还是获取组织中尚未被利用的专业知识？你是怎么看的？

What do you feel like the biggest constraint is on continued model progress right now? Is it like data? Is it like getting all this expertise that lives within organizations that hasn't been fed in, or how do you think about that?

我认为最大的解锁点在于围绕如何设置训练的新想法，使其能够处理更长的时间跨度，能够更容易地处理不可验证的奖励。我认为要让这些工作起来，存在很大的惯性和能量障碍，而继续在编码领域迭代则相对容易。所以这里的一个陷阱就是继续让编码变得更好。但我认为人们真正想做的是弄清楚如何进入那些更长期的领域。例如，一个可能成为训练瓶颈的量是，你的智能体轨迹展开需要多长时间，对吧？为了更新模型，你首先要进行试错。让你的智能体尝试许多解决方案。你收集奖励，然后可以使用强化学习来更新模型的权重。但如果你的展开需要 24 小时，那么训练模型就会非常困难，对吧？因为每一步训练都会花费很长时间。所以我们必须想办法切分展开的工作，也就是智能体所做的工作。因此，能够增量地训练它们，智能体必须根据增量进展来猜测它们做得好不好。然后进入所有那些我们没有奖励的领域，在那里我们不能有单元测试，也不能有证明或验证。所以，如果人们想改进智能体，今天就应该在这些方面发力。

I think the biggest unlock would be just new ideas around how to set up the training such that it can handle much longer time horizons, such that it can handle non-verifiable rewards much more easily. And I think there's a lot of inertia and a big energy barrier to getting these things to work, and it's kind of easier to continue to iterate on the coding regime. So I think one trap here would be just to continue to make the coding better and better. But I think what people really want to do is figure out how to move into those longer term domains. So for example, one quantity that can bottleneck your training is how long does it take to roll out your agent trajectories, right? So if you're in order to update the model, you first sort of do trial and error. You have your agent attempt many many solutions. You collect the rewards and then you can use reinforcement learning to update the weights of the model. But if your rollout takes 24 hours, you're going to have a really hard time training the model, right? Because every single training step will take very long. So we have to figure out ways of slicing up the work that the rollouts, the work that the agents are doing. And so incrementally being able to train them, agents have to guess did they do well, did they not do well based on incremental progress. And then moving into all these domains where we don't have the rewards, where we can't have a unit test or we can't have a proof that's a verification of a proof. So that's really where people should be pushing today if they want to improve agents.

你觉得我们距离解决一些不可验证领域的问题有多近？也许回到你的 AI 科学家，以及能够做出新颖物理发现之类的。你直觉上觉得我们离这些东西有多近？

How close do you feel like we are to cracking some of the non-verifiable domain problem? I guess maybe back to your AI scientists and the ability to have a novel physics discovery or something like that. Do you have a gut intuition as to how close we are to some of that stuff?

我认为现在的模型已经非常强大，它们常常能对某件事是否成功做出惊人的判断。但我们还没有看到人们大规模实施这一点来进一步改进模型的优秀例子。所以我认为，我们可能会在接下来的几个月、大约 12 个月内看到这一点。人们实际上会实现这些 LLM 评判者，即使用 LM 评判者的方法，在没有可验证奖励的情况下，我们仍然能够从真实世界的互动中学到很多。

I think that the models are so capable now that they can often make amazing judgments about whether something has worked or not. But we still haven't seen great examples of people implementing that at scale to improve the models further. So I think that's something that we might see in the next few months, next 12 months or so. People actually pulling off these LLM judges, the approaches with LM judges where really there is no verifiable reward but we're still able to learn a lot from real world interactions.

是的。但你觉得人们已经知道配方了，只是真的需要去运行那个实验。

Yeah. But you think people like the recipe is kind of known. It's just literally about running that experiment.

是的。我认为人们可能还没有完全弄清楚配方的所有细节，如何最优地做这些事情，但一直有帮助的是，模型本身变得如此强大，以至于你通常可以信任它们的判断。

Yeah. I think people have maybe haven't figured out all the details of the recipe yet, how to optimally do these things, but the thing that's been helping is just the models themselves becoming so strong now that you can often trust their judgment.

它们通常非常可靠地判断发生了什么。所以这有点像是提升了整体水平。

They're often very reliable at determining what's happened. So it's kind of something that lifts the whole thing up.

是的。我很好奇，你显然在很多想法上很早就有了信念，而且这些想法后来被证明是正确的，比如在推理等方面你非常早。那么在过去几年，或者可能在过去一年里，有没有什么事情是你曾经强烈认为会在 AI 领域发生，但结果却走向了不同方向，从而让你改变了看法？

Yeah. I'm curious, you've obviously had conviction early in a lot of ideas that proved correct, and super early on reasoning and a bunch of these things in the last few years. Or maybe in the last year, anything you've changed your mind on that you felt pretty strongly would happen in the AI world and has gone in a different way?

嗯，问题是，我认为没有人能预测到，一旦编码智能体跨越了某个能力阈值，它们会带来多么大的变革。所以我原本预期编码模型会平稳进步，它们会持续变好，最终被越来越多地使用。但这种突然的爆发式使用，真的是另一回事。现在我开始觉得，未来我们可能会看到更多这样的转变。

Well, the thing is, I think nobody could predict how transformative the coding agents would be once they crossed a certain threshold of ability. So I was expecting smooth progress on coding models. They just continuously get better and eventually they'll be used more and more. But this sudden explosion of usage was really something else. And now I'm starting to think maybe we'll see more of these transformations in the future.

是的。就像是一个阶跃变化的时刻，我觉得每个人在 11 月、12 月左右都感受到了这一点。甚至很难说清楚那是什么，但它就是开始工作得更好，你可以想象在其他领域也会达到某个点，让这种变化变得明显，但这很难预测。是的。我觉得有趣的是，即使是最接近这些模型的人，也很难先验地预测突破会在那时发生。

Yeah. There's just like a step change moment where something—I mean, I feel like everyone felt that around like November, December. And it's even hard to articulate what it is, but it just starts working so much better, and you can imagine in other domains you'll reach some point where that becomes apparent, but it's kind of hard to predict. Yeah. I think what's fascinating is even the folks closest to those models, it was hard to predict a priori that's when the breakthrough was going to happen.

我在想该在哪里结束。有一件事，也许可以回到我们开始的地方，也就是你写那部小说，思考世界走向何方。显然，这些问题激励着 River 和你的创立。你知道，我觉得你写过这个问题，显然让很多人夜不能寐的是，如果机器能做所有这些工作，那人们还能做什么？

I was thinking about where to end. And one thing, maybe to tie it back to where we started, which was really around you writing this fiction and thinking about where the world's headed. And there's obviously these questions that motivate River and your founding there. You know, I feel like you've written the question obviously keeping many people up at night is, if machines can do all this work, what's left for people?

我认为未来，如果我们想作为人类在世界上保持相关性，我们需要找到与机器正确的共生方式。所以今天已经有了一些共生。如果我在使用编码智能体，它会处理很多底层细节，处理那些我在高层面上理解但可能不想自己写代码的任务。而我仍然可以负责架构和规划。我认为这是一种很好的模式，人机协同工作。但存在一个很大的风险，即这种平衡会随着时间推移而改变。随着编码智能体和一般智能体变得越来越有能力，激励总是让它们接管更多控制，让它们做得越来越多。而这正是那个故事所讲的，因为一旦你让智能体为你做所有决定，你就不再控制自己的生活，不再控制局面了。所以我认为我们应该努力弄清楚如何保持这种共生关系。对我来说，这需要在对齐方面取得新的进展。所以我们为模型做的对齐，现在往往围绕着从人类偏好中学习。比如很多人类喜欢这种行为，不喜欢那种行为，我们就修改模型以符合这些偏好。但我认为未来，为了让人类保持相关性，我们需要更深层次的对齐，更深层次的人机融合。最终，也许会有类似神经链接的东西，让你仅通过思考就能控制大量的智能。但我认为那还需要一段时间。所以与此同时，我们可以通过改变训练方式、在算法和技术上创新来更好地对齐模型。这正是 River AI 试图做的。所以我认为这也是我鼓励任何从事 AI 研究的人去做的：思考如何让 AI 模型与我们保持亲近？如何让它们遵循我们的意图并帮助我们，而不是接管和取代我们？

I think the future, if we want to stay relevant in the world as humans, we want to figure out the right kind of symbiosis with the machines. So today there is a bit of symbiosis going on. If I'm using a coding agent, it takes care of a lot of the low-level details, takes care of tasks that I kind of understand at a high level but I might not want to type out the code myself. And I still get to do the architecture, the planning. And I think that's a great mode to be in, where human and machine are working together. But there's a big risk that the balance will shift over time. So as the coding agents and as agents in general become more capable, the incentive is always to let them take more control, let them do more and more. And this is kind of what the story is about, because once you let the agent make all your decisions for you, you're not controlling your life anymore. You're not controlling the situation anymore. And so I think what we should try to do is figure out how to keep the symbiosis going. And for me, that requires new advances in alignment. So the kinds of alignment that we do for the models are often now around learning from human preferences. So a lot of this group of humans, they like this kind of behavior, they don't like this kind of behavior, and we modify the model to comply with that. But I think the future, in order for humans to stay relevant, will need a much deeper alignment, much deeper integration between human and machine. And ultimately, maybe there will be something like neural link that allows you to control large amounts of intelligence just by thinking. But I think that's still some time away. So in the meantime, we can align the models better by changing the way we train them, by innovating on the algorithms, on the technology. And that's kind of what River AI is trying to do. So I think that's something I'd encourage anyone who's in AI research also to do: think about how do we keep the AI models close to us? How do we keep them following our intentions and helping us, rather than sort of taking over and replacing us?

是的。不，看起来在这种情况下，帮助我们基本上仍然给了我们一个角色，即使这在某种程度上可能不是最优的，比如最大化某种回形针目标。它至少是在最大化人类的繁荣。所以那——

Yeah. No, and it seems like in this context, helping us is still giving us a role basically, even if it's maybe suboptimal in some way of like maximizing whatever paperclip objective there is. It is maximizing at least human flourishing. And so that—

完全正确，我们应该训练模型去最大化人类的繁荣。这是另一个要点。

Exactly, and we should train the models to maximize human flourishing. That's another takeaway.

你知道，显然在过去 24 小时里，有一封许多研究人员签署的信，我想是昨天发布的，呼吁政府考虑放缓 AI 的发展。而且有很多关于这一切进展速度的担忧。这会引起你的共鸣吗？你怎么看待这件事？

You know, there was obviously this letter over the last 24 hours that many researchers signed, I think it was released yesterday, calling for the government to consider slowing down AI. And there's all this concern around just how fast it's all moving. Is that something that resonates with you? How do you think about that?

是的，我认为在这一点上放缓是极其困难的。AI 已经变得如此庞大。美国经济依赖于 AI 的进一步进展。此外，还有国际竞争对手。中国在 AI 能力方面正在迅速追赶，他们可能会也可能不会决定也放缓。所以我认为让人们放缓真的非常困难。我不知道那封信在那种意义上有多现实。显然，如果我们真的能稍微放缓，做更多对齐工作，我会很高兴被惊喜到。我认为那会是一件好事。如果我们无法放缓，我认为我们应该加速那些能够改善对齐、改善安全的技术，因为那将很快对我们部署的模型变得非常、非常相关。

Yeah, I think it's incredibly difficult to slow down at this point. AI has become such a big thing. The US economy is reliant on AI making further progress. As well, there are international competitors out there. China is quickly catching up in terms of AI capabilities, and they might or might not decide to slow down as well. So I think it is really, really difficult to get people to slow down. I don't know how realistic that kind of letter really is in that sense. Obviously, I would love to be surprised if we do manage to slow down a little bit, do more work on alignment. I think that would be a good thing. If we're not able to slow down, I think we should accelerate on technologies that can improve alignment, that can improve safety, because that's going to quickly become very, very relevant for the models that we deploy.

是的。那么最好的方式就是更多的开源模型，以及在闭源提供商内部投入更多努力吗？或者如果你有魔法棒，什么会帮助增加那方面的工作量？

Yeah. And is the best way to do that just more open source models and more effort within the closed source providers? Or if you had a magic wand, what would help increase that amount of work?

我认为那些接近可能开始变得危险阈值的开放模型真的非常非常有价值，因为它们允许世界上的任何人尝试他们关于如何对齐和控制这些模型的想法。所以它们不会强大到你能用它们造成真正的伤害，但它们会展现出一些在未来会变得危险的特征。

I think open models that are close to the threshold of where they might start to become dangerous are really, really valuable, because it allows anybody in the world to try out their ideas for how to align these models, control them. So they're not so powerful that you can do real harm with them, but they exhibit some of these things that will become dangerous in the future.

网络安全能力是目前最大的问题之一。所以我觉得我们不应该只让大型 AI 实验室里的几千人思考这些问题。我们应该让尽可能多的人去尝试弄清楚我们需要构建哪些算法、哪些工具来让这些方案可行。

Cybersecurity capabilities are one of the biggest ones right now. So I feel like we don't want to just have the few thousand people at the big AI labs thinking about these problems. We should have as many people as possible trying to figure out what are the algorithms, what are the tools that we need to build to make these things viable.

我的意思是，你显然比任何人都更接近这些东西。比如，你目前认为这一切最终会好起来的概率有多大？

I mean, you've obviously been closer to this stuff than anyone. Like what's your current probability that this all ends up okay?

这是个好问题。问题是，对谁来说算好？因为 AI 可能会加剧世界上的不平等。有些人可能从中受益匪浅，而另一些人可能会感到被抛在后面。我认为这是最紧迫、最直接的风险。然后，是的，未来确实存在风险，比如 AI 模型是否会接管一切，是否会从那些建造它们并控制它们的人手中夺走控制权。这也是一个风险，但很难对此做出预测。我觉得那会是在更远的未来，而我看到的更直接的问题是：我们能否让所有人都跟上这趟旅程？

That's a good question. The question is okay for whom at this point, because potentially AI could amplify inequality in the world. There could be people who benefit greatly from it, and then there could be people who kind of feel left behind. I think that's the most immediate and urgent risk. And then yes, there are risks in the future around can the AI models take over, can they take control away even from the people that have built them and are controlling them today? And that's also a risk, but it's so hard to make predictions about it. I feel like that's going to be in the further future, and the more immediate problem that I can see is: are we able to bring everybody else along on the journey?

是的。好吧，最后一个问题。在你写的那篇《向机器神祈祷》中，你有一句很美的话，我想大概是“当模型第一次回应时，你流下了意想不到的泪水”。所以我必须问：那是来自个人经历吗？你是否有过类似的感觉？

Yeah. Well, last question for you. You had in that piece you wrote, 'Prayer to Machine God,' you had this beautiful line, I think it was like the tears you didn't expect when the model first spoke back. So I have to ask: was that from a personal experience? Have you had something that felt like that?

是的，我把它写进去是因为我仍然记得第一次让这些模型工作时那种神奇的感觉。那时候和现在相比非常简陋，但我认为所有早期从事 AI 的人都感受过那种感觉，当模型第一次开始做一些非常简单的事情，但却是以前任何机器都做不到的事情时，那真的非常非常神奇。无论是写一个能生成质数的 Python 函数，还是诸如此类的事情，都让人觉得这不真实。所以，是的，我想捕捉那种感觉。我相信很多早期从事 AI 的人都有过这种感觉，但现在回头看，一切都显得那么不真实，因为模型已经取得了巨大的进步。它们现在确实在影响整个世界，而且已经远远超出了我们过去在 AI 还是一种爱好时所做的那些小实验。

Yeah, I put that in there because I still remember how magical it felt to first get these models to work. Back then it was very, very modest compared to today, but I think this is something that all the early folks in AI felt when the models first started to do some pretty simple things, but things that you couldn't do with any other machine before. It was just really, really magical. So whether it was just writing a Python function that can generate prime numbers or something like that, it just felt like this is unreal. So yeah, I wanted to capture that feeling. I'm sure that's something that a lot of folks that were early on in AI felt, but now looking back, it all seems so unreal because the models have advanced tremendously. They're really affecting the entire world at this point, and it's gone so far beyond those little experiments that we used to do when AI was more of a hobby.

哦，完全同意。好吧，我觉得这是一个完美的结束点。这太迷人了。非常感谢你来到播客，分享这一切。

Oh, totally. Well, I think that's a perfect place to end. It's been fascinating. Thank you so much for coming on the podcast and sharing all this.

谢谢你，Jacob。非常感谢。

Thank you, Jacob. Appreciate it.

我是 Jacob Efron，这里是 Unsupervised Learning，一个我能与 AI 领域最聪明的人对话的播客，我会问他们大量关于模型发展以及对世界商业影响的问题。我希望大家能看出来，我对此乐在其中。这是我除了在 Redpoint 做投资人的日常工作之外，利用晚上和周末做的项目。但我们能请到这些出色的嘉宾，真的离不开像你这样的听众订阅播客、与朋友分享。这最终才是让这一切运转起来的关键。所以，请考虑这样做。非常感谢你的支持和收听。我们下期再见。

I'm Jacob Efron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so, please consider doing that. And thank you so much for your support and listening. We'll see you next episode.