人类对齐的 AI：集体智能与工作的未来

好的，我们现在在远程演播室，和来自 Amazon AGI Lab 的 Daniel 在一起。欢迎。

Okay, we're here in the remote studio with Daniel from Amazon AGI Lab. Welcome.

嗨 Swyx，很高兴见到你。

Hi Swyx, it's so good to see you.

很高兴我们终于能成行。你旅行回来了吗？

Glad we can finally make this happen. Are you back from your travels?

我在西雅图，还没回湾区。

I am in Seattle, not back in the Bay yet.

好的，大老远跑一趟。你知道，奥德修斯算是今年科技圈的热门电影。我觉得你有点像奥德修斯，去了那么多精彩的活动。你在法国做什么？我们来给大家更新一下，因为我觉得这很酷。作为一个有经济学学位的人，你参加英联邦活动这件事本身就挺酷的。

Okay, big round trip. It's you know, Odysseus is kind of like trending as far as like the big tech pool movie of the year. I feel like you're a little bit of an Odysseus, like you're going to all these amazing events. What were you doing in France? So let's update people, because I think it's very cool. As someone who has an economics degree, the fact that you were in a Commonwealth of Nations event was kind of cool.

嗯，在法国我是在学习如何逃离流沙，但在此之前我在爱丁堡，那里举办了亚当·斯密《国富论》250 周年纪念活动。实际上，我去了他的故居，那栋房子翻新过，真是个不可思议的地方，让人想起启蒙时代的理念，以及人类繁荣、自由主义这些主题。这次聚会汇集了经济学家、教授，以及来自不同行业的人。我特别代表一种视角：我们如何看待 AI，以及如何思考 AI 与人类境况、人类繁荣这些主题。我们如何让 AI 为我们所用，而不是把它当作科学实验来建造。这是我一直在思考的问题。这也是我离开学术界的原因，因为我一直在研究人类智能，以及我们这种特定智能的演化，还有那些能让更多人参与集体动态的事物。我对 AI 的看法实际上与目前行业的主流框架相当不同。我很荣幸被邀请来分享这个视角。核心思想是：人类智能是集体的。人类学家说我们拥有集体大脑。没有一个人能独自生存。我们依赖集体。智能从我们的互动中涌现。它本质上是社会性的。而创新——那种让我们不仅生存下来，还能适应我们现在所处的各种环境的创新——是多样性、种群内的变异、种群规模以及互联性的函数。那么，如果 AI 能扩展这些过程，让更多人类参与集体对话，并且让 AI 为所有人而建，而不仅仅是建造 AI 的工程师，这意味着什么？这就是我贡献的视角。

Well, in France I was learning how to escape quicksand, but before then I was in Edinburgh, which was hosting the 250th anniversary of the Wealth of Nations, Adam Smith's famous book. Actually, I was in his house, which is renovated and it's a really incredible place to remind us of the Enlightenment era ideas and the themes of human flourishing, liberalism. So this gathering brought together economists, professors, folks from different industries. I was specifically there representing how do we think about AI and how do we think about some of these themes of the human condition, human flourishing with AI. How can we make AI work for us rather than building it as a science experiment. This is something I think about all the time. This is why I left academia myself, because I had been studying human intelligence and the evolution of our particular shape of intelligence and the types of things that allow more people to participate in these collective dynamics. The way I think about AI is actually quite different than the sort of dominant framing of the industry right now. I was so honored to be invited to share this perspective. The big idea is that human intelligence is collective. Anthropologists say we have the collective brain. No one individual is capable of even surviving on their own. We depend upon the collective. Intelligence emerges from our interactions. It's fundamentally social. And innovation, the type that allows us to not only survive but adapt to all the different environments we now exist in, is a function of diversity, variation within a population, the size of a population, and the interconnectivity. So, what would it mean to have AI extend these processes to allow more humans to participate in the collective dialogue and for AI to be built for everybody rather than just the engineers who are building AI? That's the perspective I contributed.

我觉得这是我们都认同但可能没怎么谈论的事情。所以，我认为这是个好信息。人们害怕 AI 抢走我们的工作吗？你知道，就像你代表的，对吧？如果你工作做得好，你确实会取代一些可能不那么有趣的工作，但你确实会取代工作。

I think something that we all cosign but maybe not necessarily talk about enough. So, I think that's a good message. Are people scared of AI taking our jobs? You know, like you represent, right? Like if you do your job well, you do take away some maybe less exciting jobs, but you do take away jobs.

是的，情况各不相同。我认为很多人有理由担心会有变化，至少会有一个过渡期。这次会议上的很多人，以及我所在社群中的很多人，往往非常乐观，倾向于强调 AI 能释放的潜力，但如果我们继续以现在的方式建造 AI，它是否真的能为人类带来所有这些好处，这并不是一个定论。我认为现在的工作并没有以最好的方式利用人类认知。即使是最有创造性的工作，也仍然有大量苦差事，我们整天盯着屏幕。这不是我们大脑应该做的。我们应该相互协作，集思广益，提出新想法，进行头脑风暴，做各种真正符合我们智能互动本质的事情。但我们创新得越多——至少 20 和 21 世纪的趋势是这样——我们就越被束缚在屏幕上。所以，一方面，我认为很多人对能够自动化大量苦差事、大量不值得人类关注和时间的事情感到兴奋。问题在于，如果我们陷入这种自动化思维模式，我们就放弃了 AI 能为人类工作、人类福祉、人类互动和人际关系带来的更多价值。所以，现在我们担心 AI 会自动化大部分工作，但当我们看实际智能体的指标时，它们又如此不可靠，讽刺的是，我们反而感觉好一点了。AI 实际上还没有达到能自动化足够多工作的水平。

Yeah. It's so varied. I think a lot of people are rightfully concerned that there will be changes, that there will be at the very least a transition period. A lot of the folks who were at this particular conference and a lot of the people who are in my communities tend to be very optimistic and sort of index on the potential that AI can unlock, but it is not a foregone conclusion that if we continue building AI in the way that we are, it will actually confer all of these benefits for humans. I like to think that work right now is not utilizing human cognition in the best possible way. Even with the most creative types of work, there's still so much drudgery, there's still so much like we literally stare in front of screens all day long. That is not what our brains are meant to do. We're meant to collaborate with each other. We're meant to put our heads together and come up with new ideas and brainstorm, do all sorts of things that really resemble the interactive nature of our intelligence. But the more that we innovate, at least the sort of trends of the 20th and 21st century, the more that we're tethered to these screens. So, on the one hand, I think a lot of people are excited about the possibility that we can automate a lot of the drudgery, a lot of the things that are not worthy of human attention and human time. The problem is that if we get trapped in this automation mindset, we are leaving on the table so much more value for what AI could be doing for not only human work, but human well-being, human interactions, human relationships. So, right now we're worried about AI automating a large proportion of the work, but then we look at the metrics of the actual agents and they're so unreliable that ironically we feel a little bit better. The AI is actually not where we need it to be to automate enough of the work.

我们来介绍一下你一直在做的工作。你加入 AI 工程师圈子好几年了，最初在 Adept，然后你们加入了 Amazon。对于那些不太了解这个故事的人，你能重新介绍一下 Amazon AGI Lab 吗？

Let's introduce the work that you've been doing. You've been part of the AI engineer circle for a couple years, originally with Adept and then you guys joined Amazon. For those who haven't been too close to the story, could you reintroduce Amazon AGI Lab?

是的，Amazon AGI Lab 正在构建与人类对齐的智能。从能做人类在电脑上能做任何事的 AI 开始。那是 Adept 最初的使命，我们把它带了过来，并播下了实验室的种子。我们已经发展了那个使命，所以我们正在深入思考这意味着什么。一个智能体要能做人类在电脑上能做任何事，需要哪些技能？这远不止语言。我们真的需要智能体能够像人类感知数字环境一样感知数字环境。而且不仅仅是数字环境，数字环境是基于物理环境的。所以，我们需要智能体也理解世界，拥有那种世界模型。我们需要智能体能够实时交互。这很重要。我认为行业还没有真正深入思考过这一点。我们有点被困在聊天机器人和编码智能体以及批量轮流对话的局部吸引子状态中。这绝对不是人类相互互动的方式。我们不断根据上下文实时更新我们的理解。我们在协商意义。我们在想出新的思考方式。我们认为这是理所当然的，因为这就是我们大脑的工作方式。但现在，我们是在适应 AI 和技术以及它的局限性，而不是反过来。

Yes, so Amazon AGI Lab is building human-aligned intelligence. And starting with AI that can do anything that a human can do on a computer. That was Adept's original mission and we kind of imported it and seeded the lab. We've evolved that mission, so we're really thinking deeply about what it means. What are the skills that are necessary for an agent to be able to do anything that a human can do on a computer? It's so much more than language. We really need agents to be able to perceive the digital environment in the same way that humans perceive the digital environment. And more than the digital environment, the digital environment is based off of the physical environment. So, we need the agents to also have an understanding of the world and have the kind of world models. We need the agents to be able to interact in real time. This is huge. This is something that I think the industry has not really thought deeply about yet. We're kind of trapped in this local attractor state of chatbots and coding agents and turn-taking in batches. And this is absolutely not how humans interact with each other. We are constantly updating our understanding as a function of the context in real time. We are negotiating meaning. We are coming up with new ways of thinking about things. We take this for granted because this is how our minds work. But right now, we are kind of accommodating the AI and the technology and the limitations that it has rather than the other way around.

那么，如果我们构建的智能体不仅能够像我们一样感知世界——我们之间没有这种共同基础——还能跟上我们的节奏、与我们一同思考、在听我们说话的同时采取行动、在与我们互动的同时准备下一步想法或行动，那会是什么样子？这将是我们在交互性思考方式上的一次范式转变。

So, what would it look like if we built agents that not only perceived the world in the same way that we didn't have this sort of common ground, but could keep up with us, think with us, take actions while it's listening to us, prepare its next thoughts or its next actions while it's interacting with us. This would be a paradigm shift in how we think about interactivity.

是的，我认为有一些近期的工作——我想是在机器领域，我们也都在互动模型方面有所覆盖。我认为语言模型技术树中实时交互这一分支，某种程度上是从 400 的发布开始的，但我觉得很多人并没有注意到这一点。当然，在学术界，在此之前还有更多研究。我会提到 Flamingo 和很多纯语音模型——它们具备全双工、端到端的能力。我们在法国有 Q Tai Moshi，也推出了 Gradio，它也在我的会议上做过演讲。所以我只是想引入必要的知识。我的意思是，这——我想与聊天范式相比，相对小众，因为它不那么流行，但这正是真正的 AGI 应有的样子：你实际上可以与机器协作、进行思维融合。我大致就是这样描述的。

Yeah, I think there's some recent work by—I think in machines that we've also all covered with the interaction models. And I think this branch of the LM tech tree of real time was kind of started with the 400 launch, which I don't think a lot of people index on. Obviously, in academia, there's more research before that. I would point to Flamingo and a lot of the pure voice models—they would have full duplex, end-to-end stuff. We had Q Tai Moshi in France also spit out Gradio, which also has spoken at my conferences as well. So I just wanted to sort of import the required knowledge. I mean, this is—I guess compared to the chat paradigm, relatively niche because it's not that popular, but it is what a real AGI would look like, which is that you could actually collaborate and mind-meld with the machine. It's kind of how I pitched that.

是的。你刚刚列举了行业和不同的学术实验室——它们都在独立地趋近于这些更灵活的人类智能的组成部分。所以，是的，实时交互虽然现在还算小众，也许 Thinking Machines 会让它成为对话的核心。但这只是更大一组组件中的一小部分，这些组件将使 AI 更贴近我们自身的智能。我再举一个例子。我们倾向于将记忆视为存储。人类一直用技术作为隐喻来理解自己，在 20 世纪，我们把心智比作计算机，有硬件和软件，而记忆是我们卸载的东西，但这根本不是人类记忆的运作方式。记忆就是一切。它是我们模拟未来的方式。它跨越许多不同的时间尺度。这个词本身并没有体现出它已融入所有学习和认知这一事实。那么，构建拥有所有这些不同类型记忆——包括像我们这样的情景记忆，这对我们的智能至关重要——的智能体意味着什么？我们拥有个体视角和自我，这使我们能够以更高效的方式检索信息。

Yeah. And you just listed the industry, different academic labs—they are independently converging on a lot of these components of more flexible human-like intelligence. So yes, interacting in real time even though it's kind of niche now and maybe Thinking Machines is going to make it more central to the conversation. That is one tiny slice of a larger set of components that will make AI more aligned with our own intelligence. I'll give one other example. So we tend to think of memory as storage. And humans have always used technology as a way to metaphorically understand themselves, and in the 20th century we use mind as a computer and you've got the hardware and the software and memory is this thing that we kind of offload, but that's not at all how it works in humans. Memory is everything. It's how we simulate the future. It occurs across many different time scales. The word itself doesn't do service to the fact that it is integrated into all learning and cognition. And so what would it mean to build agents that have all of these different types of memories, including episodic memories like we do, which is really core to our intelligence. We have individual perspectives and selves, and that allows us to retrieve information in much more efficient ways.

你提到了这一点，所以我要问一个热门问题：你认为像样的记忆需要更新权重，还是说它仍然可以存在于系统内部？因为你刚才说我们倾向于将记忆视为存储。大概你还有其他想法，但你没有说出来。

You brought this up, so I'm going to ask the hot topic question, which is: do you think decent memory needs to update weights, or do you think it can still live within the systems? Because you just said we tend to think of memory as a storage. Presumably, there's something else that you're thinking about, but you didn't say so.

嗯，目前行业对记忆的思考方式，我认为它将成为更大系统的一部分，就像人类卸载一样——我的意思是，我们的工具、我们的环境、我们的扩展环境包含了我们智能的各个方面。我们依赖使用工具来完成日常事务，我们可以查阅维基百科或 AI 工具，它们以新的方式组织信息。这仍然是一个核心功能，我认为非常有用，但还需要其他方面。我甚至不想称它们为记忆，因为这个词有太深的含义。改变权重，或者如果不总是改变权重，就在推理时改变信息如何被情境化。我不想在这里深入细节，因为这是我们实验室目前正在推进的一个活跃课题，但更大的观点是，我们需要更整体地思考不同时间尺度的交互。

Well, the way that the industry is currently thinking about memory, I think it's going to be part of a larger system in the same way that humans offload—I mean, our tools, our environment, our extended environment contains aspects of our intelligence. We depend upon using tools to be able to do daily things, and we can look up on Wikipedia or AI tools that are just organizing information in new ways. That is still a core function that I think is really useful, but there will need to be other aspects. And I don't even want to call them memory because that has such deep connotations. Changing weights, or if not always changing the weights, changing at inference how information is contextualized. And I don't want to get too much into the details here because this is an active thing that we are pushing in our lab right now, but the bigger point is that we need to be thinking more holistically about different time scales of interaction.

明白了。也许再补充一点背景信息，然后我还想谈谈你们最近推出的一些东西，因为我认为这能提供具体的内容，比如“好吧，这是已经公开宣布的”。那么，概念上的问题是：你能把那个 Amazon AGI 放在更广泛的亚马逊背景中吗？有 Nova 团队，对吧，它是亚马逊核心的一部分。你们发布了 Nova Act，我实际上已经调出来了。我想深入探讨一下，因为这引出了感知智能体，但谁在运营 Amazon AGI 实验室？他们的联系是什么？它非常紧密吗？还是说它旨在更独立地运行？你能给外界人士提供任何关于如何理解当前情况的框架吗？

Got it. Maybe one more contextual thing and then I wanted to also go through some of the recent stuff that you guys have launched because I think that gives concrete things of like, well, okay, here's what's been publicly announced. So, the conceptual thing is: could you put that Amazon AGI in context of broader Amazon? There is the Nova group, right, which is part of core Amazon. You guys released Nova Act, which I have pulled up actually. I wanted to go into that because that leads into perception agents, but who runs Amazon AGI lab? Like what's their link? Is it very close? Is it sort of meant to be running more independently? Anything you can give the external people about how to frame what's going on?

是的，我和最初的 Adept 团队一起来的，当我们加入时，我们说服了领导层，为了进行前沿研究，我们确实需要保持更像初创公司的运营模式。我们需要保护我们的研究，并能够专注于构建感知智能体的使命——尽管当时我们还没有称之为感知智能体，但这从一开始就是目标。我们与一个庞大的亚马逊团队合作。获得这种认同实际上非常了不起：是的，我们重视研究、基础研究，而且更重要的是，我们不仅重视做其他前沿实验室正在做的事情，还重视创造一个空间来思考新的研究类别、新的科学，这些可能不会立即产品化。我认为其他一些实验室在某种意义上成了自身成功的受害者，因为一旦他们有一个产品上线，很多用户在使用，他们就不得不关闭不同的研究项目，并说：“好了，所有人手都扑到这个项目上。”

Yeah, so I came with the original Adept folks, and when we came, we convinced leadership that in order to do frontier level research, we really needed to keep an operating model that was more like a startup. We needed to insulate our research and be able to focus on the mission of building perception agents—and even though we weren't calling them perception agents then, that was the goal from the very beginning. We worked with a large team of Amazonians. It was actually really incredible to get this buy-in that yes, we value the research, the foundational research, and moreover, we value not just doing what the other frontier labs are doing, but making a space to think about new categories of research, new science that might not necessarily be productized immediately. I think some of the other labs are in a sense victims of their own success because they have to—once they have a product out there that a lot of users are interacting with, they have to shut down different research projects and say, "Okay, all hands on deck for this thing."

这太残酷了。每个人都在做编码和 B2B SaaS。

It's brutal. Everyone is just doing coding and B2B SaaS.

是的，没错。其他所有实验室都在追赶其他实验室正在做的事情。我们的 Amazon AGI 实验室从根本上不同。我们确实在研究前沿科学，并思考人类与智能体交互的下一个范式，更多人类能够利用 AI 的下一种方式。

Yeah, right. And all of the other labs are catching up to what the other labs are doing. Our Amazon AGI lab is fundamentally different. We really are working on frontier science and thinking of the next paradigm that humans and agents will interact with, the next way that more humans will be able to leverage AI.

好的，太棒了。那么，我正要提到 Nova X。我知道那已经是一年多前的事了，所以不算最新，但我只是想——这会是首批之一——我想你出现在其中一个视频里，我当时想：“哦，我认识她。这太酷了。”在发布视频中看到朋友总是很愉快。我想当你谈到你在财富与国论坛上的发言，比如“别担心你的工作”时，是的，他们对此做了基准测试。很好。

Yeah, okay, awesome. So, I was just going to go bring up Nova X. I know that was over a year ago, so it's kind of not that current, but I just wanted to—this would be one of the first—I think you showed up on one of the videos and I was like, "Oh, I know her. That's just cool." It's always nice to see a friend in one of these launch videos. I think when you said things about how you were at the Wealth & Nations forums and like, "Don't worry about your jobs." Yeah, they did benchmarks on that. Great.

我觉得这是那种人们一直想要的东西——机器人流程自动化。你必须与工具交互，实现自动化。你们有 SDK，有专用模型。那是一次非常全面的发布。我想让你聊聊发布以来的情况，因为你是参与者。

I think this is one of those things where robotic process automation is something people always want. You have to interface with tools and automate things. You have an SDK for it, dedicated models for it. It was a very full-fledged launch. I wanted to let you riff on what's been the story since launch, because you were part of it.

是啊，天哪，现在感觉已经是很久以前的事了，因为我们做了太多事情。

Yeah, so gosh, this feels forever ago at this point because we've been doing so much.

陈年旧事了。但我有个计划，这要谈到感知智能体，对吧？

Ancient history. But I have a plan that this goes into perception agents, right?

对对。当时我们的想法是：模型显然还没达到能像我们一样理解数字世界的水平，无法理解所有可供性，长期规划也还没实现。更不用说像人类那样灵活思考和推理了。那比我们当时的能力超前了 14 步。所以我们做的是顺应模型当前的水平，思考人类在计算机上执行的基本操作——点击、滚动之类的。我们能把这些做到可靠吗？如果能，那么开发者就可以把重复性工作串成工作流。那是一个重大突破，我们有一个团队把它产品化了。但自那以后我们又前进了一大步，因为我们意识到可靠性并非我们原先想的那样。我们之前认为可靠性就是每次都点对位置、点对按钮。

Yeah, yeah. So the way we were thinking about it back then was: okay, the models clearly aren't where they need to be to understand the digital world the way we do, understand all the affordances, and the long-horizon planning wasn't there yet. Not to mention being able to think flexibly and reason like humans do. That's 14 steps beyond where we were. So what we were doing was meeting the models where they were and thinking about the atomic interactions that humans perform on computers—clicking, scrolling, things like that. Could we get those reliable? If we could get those reliable, then developers could string together workflows for things that were very repetitive for them. That was a big unlock, and we had a team productize that. But since then we've moved a lot further because we realized that reliability isn't actually what we thought it was. We had been thinking about reliability in terms of clicking in the right place on the right button every single time.

就像屏幕坐标，对吧？从图像中识别出感兴趣物体的边界框，然后确保点中它。两年前这是个难题。现在可能解决了，我不知道。

Literally like screen coordinates, right? From an image, identify the bounding boxes of whatever is of interest and then make sure you click on it. That was a hard problem two years ago. Now maybe solved, I don't know.

可能解决了吧，对吧？实际上比你想的要难得多。但这甚至不是可靠性的定义。这些感知智能体的最终目标是：人类给出他们的意图、高层目标，智能体能够分解这个目标并执行。也许他们会确认一下，也许人类在回路中，也许建立信任后智能体不需要确认，就可以像人类一样使用电脑。但只要你再多想一会儿，就会意识到：这可不是随便什么任务。我在主动思考。想象你在订旅行。你用图形界面，发现哦，有经停这个城市或那个城市的选项，或者我可以直飞。这完全改变了你对旅行的想法。我想不想在这个经停城市待几天探索一下？我们做的每件事，除非是像用任意软件技能提交工作发票那样，很多实际在做的事情都涉及主动思考，与电脑的交互会塑造和细化我们对目标本身的思考。目标是随时间展开的。但人类会委托别人订旅行、预订东西、订餐、提交发票。那么，一个能像人类一样可靠使用电脑的 AI 智能体，与行政助理或个人助理有什么区别？区别在于，那个人理解人类用户的心思和目标，他们不仅能分解任务，还能分解人类的偏好和意图。所以归根结底，可靠性与其说是点击相同位置和滚动，不如说是对用户心智的建模。这个转变就是一切。它重新定义了我们如何看待正在构建的东西。

Maybe solved, right? It's actually a lot harder than you think. But that's not even what reliability is. The ultimate goal for these perception agents is that a human would give their stated intention, their high-level goal, and the agent would be able to decompose that goal and go execute. Maybe they would check in, maybe the human is in the loop, maybe after establishing trust the agent doesn't need to check in and can just use the computer as the human would. But the second you think about that a little longer, you realize: it's not any task that I would do. I am actively thinking. Let's imagine you're booking travel. You're using the GUI and you realize, oh, there's an option for a layover in this city or that city, or I could do a direct flight. That completely changes how you think about the travel. Do I want to stay at this layover for a couple of days and explore this city? Everything we do, unless it's using arbitrary software skills for submitting invoices for work, a lot of the things we're actually doing involve actively thinking, and the interaction with the computer shapes and refines the way we think about the goal itself. The goal unfolds over time. But humans entrust other people to book their travel, reserve something, get meals, submit invoices. So what is the difference between an AI agent that would reliably use a computer like a human and an executive assistant or personal assistant? The difference is that the person understands the human user's mind and goals, and they can decompose not just the task but the preferences and intentions of the human. So ultimately, reliability has less to do with clicking in the same place and scrolling and more to do with modeling the user's mind. That shift is everything. It reframes how we think about what we're building.

你有一个播客叫《制造心智》。我觉得有很多有趣的认知科学可以转化为你设定的机器学习目标。我们有没有一个不同的目标？比预测下一个词更好的东西？

You have a podcast called 'Making a Mind'. I think there's a lot of interesting cognitive science that translates into the machine learning objectives you start setting. Do we have a different objective yet? Something better than predict the next token?

嗯，这正是我们现在研究的科学。我会试着把一些认知科学翻译成机器学习。我们通常认为实现可靠性、爬山或让模型更智能，就是让它们在特定任务上做得更好。你可以用强化学习让它们在某个我们关心的任务上非常擅长，但它在另一个任务上就不行，或者无法泛化。这有点像打地鼠。所以，转而思考让人类能够泛化、能做许多不同事情的底层机制——这是另一个思维转变。我们可以创建评估，确保模型不会过度拟合人类关心的狭窄领域。如何让模型做那些能让人类泛化的事情？如果考虑优化，你不能只针对任务优化。它可能被奖励黑客利用。这就是古德哈特定律：一旦你把衡量标准变成目标，这个衡量标准就不再是好标准了。但我们仍然需要优化某个东西。那么，人类在优化什么，使得他们能做所有这些不同的任务，然后我们可以让 AI 也优化这个？人类会自发地不断推断其他心智的存在，并且我们在优化对齐它们。我们在优化对齐我们的表征。从中，我们可以推导出人类表现出的所有通用灵活认知行为。那么，我们能否让 AI 优化使其表征与我们的表征对齐？这是最根本的我们想做的事情。这是一个非常困难的科学问题。我们必须研究人类是如何做到的，婴儿是如何做到的。这是一个发展问题。

Well, this is the science we're working on right now. I'll try to translate some of the cog sci into machine learning. We tend to think about achieving reliability or hill climbing or making models more intelligent in terms of getting better at specific tasks. You can use reinforcement learning to get them really good at specific tasks we care about, but then it's not good at another task or doesn't generalize. It's kind of like whack-a-mole. So instead, thinking about the underlying mechanisms that allow humans to generalize, to do many different things—that is another shift in how we're thinking. We can create evaluations to ensure models don't overfit on a narrow slice of what humans care about. How do we get models to do the types of things that allow humans to generalize? If we're thinking about optimization, you can't just optimize for the task. It can be reward hacked. This is Goodhart's law: any time you turn the measure into the goal, the measure ceases to be a good measure. But we still need to optimize for something. So what are humans optimizing for that allows them to do all these different tasks, which we could then optimize AI for? Humans are spontaneously constantly inferring the existence of other minds and we are optimizing for aligning them. We're optimizing for aligning our representations. From this, we can derive all the general-purpose flexible cognitive behaviors that humans show. So could we get AI to optimize for aligning its representations with our representations? That is the most fundamental thing we would want to do. That is a really hard science problem. We have to look at how humans do it, how infants do it. It is a developmental problem.

抱歉。我确实认为这是一个发展问题。

Sorry. I do think it's a developmental problem.

我不知道我们是否已经有了架构上的洞察来实现它，但我们可以投入更多数据，在某种程度上

I don't know if we have the architectural insight to make it happen yet, but we can throw more data at it and to some extent

从发展角度看，这在一定程度上是一个数据问题。我们再次——我不想说得太多——但我们正在弄清楚需要什么样的架构变化，才能以新的方式整合这些数据。

Developmentally it is in part a data problem. And we are again, I don't want to say too much about it, but we are figuring out the sort of architectural changes that would be needed to integrate this data in new ways.

是的。我还在想，你从 Replay 雇了我的另一个朋友 Jason Lester。

Yeah. I was just sort of reflecting also you hired another one of my friends Jason Lester from Replay.

哦，是的。

Oh, yeah.

他主要在做的事情，我猜是改进你们的环境和数据。

And he's working a lot on like I guess improving the environments and the data that you have.

他提出了一个我非常喜欢的论点。我们应该考虑在环境上投入与算力和数据同样多的资源，因为环境实际上塑造了智能能够涌现的方式。我和他一起做过一期播客，那可能是最受欢迎的一期。他阐述问题的方式非常有效。

He makes an argument that I absolutely love. We should be thinking about spending as much on the environments as on the compute and the data, because the environments literally shape what intelligence can emerge. I did a podcast episode with him. I think it was one of the most popular ones. He frames things really effectively.

是的，思维非常敏锐。我想说，这是一件有趣的事情，我试图反驳这个论点，因为它感觉太完美了。环境就是能生成更多数据的数据，类似这样。就像，哦，这是源源不断的数据，因为你只需让智能体在其中运行，就能生成一大堆 rollout，这很棒。但我们真的需要 20 个不同的环境创业实验室吗？而且它们为什么都赚这么多钱？这很可疑。

Yeah, very sharp thinker. Yeah, I would say like it's one of those interesting things that like it's I'm trying to attack the thesis cuz it feels too neat. That it's and you know, environments are data that generates more data. Something like that, you know, like it's like oh, it's the data that keeps giving because well, you can just kind of run your agents through it and generate a whole bunch of rollouts and that's all great. Do we need like 20 different environment startup labs, you know? And how come they're all making so much money? It's like very suspicious.

不，我完全同意。是的，我认为认真对待环境是我们行业思维的一个有用转变。但同样，作为认知科学家，我一直在思考：人类是如何做到的？我们如何泛化我们的环境？我们如何能够适应任何环境，即使我们只在一个环境中成长？我们可以，对吧？任何环境中都有大量噪音，我们的感官只捕捉到其中极小的一部分。但即便如此，有太多信号我们可以关注。我们如何知道哪些是最有意义的？我们再次优化了推理和使我们的表征与其他人类对齐。其他智能体告诉我们该关注什么。从一开始，人类构建的世界模型——这显然是当前一个非常流行的概念，也是 AI 未来的另一个重大赌注——人类绝对拥有世界模型，而且它们不仅仅是在真空中、像 LLM 那样的文本真空中，我们的世界模型从一开始就是社会世界模型。我们推断另一个心智如何解释世界，推断他们的视角可能是什么，这让我们能够在任何环境中泛化。我们可以采纳视角，模拟另一个环境可能需要什么来导航和解决问题。

No, I totally agree. Yeah, well, and I think that it was a useful shift in our thinking as an industry to take very seriously environments. But again, as a cognitive scientist, I'm constantly thinking about well, how do humans do it? How do we generalize our environments? How are we able to adapt to literally any environment even if we're only conditioned on one? We can, right? And the way like there's so much noise in any environment, our sensory perception only picks up on a tiny fraction of that. But even then, there's so many signals that we could attend to. How do we know which ones are most meaningful? Well, again, we're optimizing for inferring and aligning our representations with other humans. Other agents tell us what to attend to. And from the very start, the sort of world models that humans are building and this is obviously a very popular concept right now and it's another sort of major bet on the future of AI. Humans absolutely have world models and they're not just world models in a vacuum, a text vacuum like with LLMs, but our world models from the very beginning are social world models. We are inferring how another mind is interpreting the world and we are inferring what their perspective might be and that is an unlock for allowing us to be able to generalize in any environment. We can take the perspective and simulate what another environment might require to navigate and problem-solve.

是的，我认为这是世界模型论点的一个版本，我觉得讨论得不够。很多时候，人们说世界模型时，他们实际上指的是某种 3D 视频生成，是的。

Yeah, I think that's a version of the world models argument that I feel like is under discussed. I think a lot of times people when they say world models, they really mean like sort of 3D video generative and that yeah.

生成式视频之类的东西，就像李飞飞那样的。

Generative video things which is like the Feifei Lis of the world.

对。你对此有什么看法？它们是趋同的，还是我们过度使用这个术语，让它指代两个基本不同的东西？

Right. Do you have a perspective on like do they converge or are we overloading the term to mean two basically separate things?

我认为它们指的是不同的东西，但人类也是如此。人类拥有的世界模型是外部世界的模型，而在 AI 中，你也必须有一种方式来生成这些可靠的环境，这些环境被学习中的 AI 内化。但是的，这是一个非常混乱、定义不足的概念。

I think that they mean separate things but so too with humans. So the world models that humans have are models of the external world and you also have to have a way in AI of generating these reliable environments that are internalized in the AI that's learning it. But yes, this is a very messy concept that is under defined.

我认为在极限情况下，人们希望——这真的是一个十年后的事情——人们希望它们融合，比如你拥有具身视觉，你生活在一个你可以生成的世界中，这也提高了你的文本推理能力和在屏幕上点击的能力，因为我见过这种情况。

I think at the limit people hope like literally this is like a 10-year out type of thing. People hope that they merge like you have embodied vision and you live in a world that you can generate and that also improves your text reasoning and your ability to point and click on the screen cuz I've seen that.

但除非我误解了什么，我认为原则上它们不能完全趋同，因为如果 AI 生成的世界与它存在的世界完全相同，那么信噪比就不存在了。我们之所以如此灵活，部分原因在于我们能够选择性地生成。

But unless I'm misunderstanding something, I don't think that they could entirely converge in principle, because if an AI is generating exactly the same world that it exists in, the signal to noise ratio is non-existent. Like part of what makes us so flexible is that we are selective in what we can generate.

是的。是的，我明白这个论点。我只是想试着为各方说话，试着理解所有方面。我想我不需要像你那样下大赌注。我认为你确实需要选择战场，做出合理的赌注。所以，好吧，我认为所有这些概念性的东西都很好。你们，我认为即使在概念上，即使作为一个研究实验室，也有很多工作要做。但让我印象深刻并且确实看到你们做出来的是，你们仍然发布了产品和智能体工具包等等。产品策略是什么？比如，你们是边走边产品化一些东西，还是只发布人们可能不应该在生产中使用的研究产物？我们处于从研究实验室到应用 AI 的哪个位置？

Yeah. Yeah, I know argument there for me. I just think like I'm trying to speak for I'm trying to understand all sides. I think I don't have to make a strong bet unlike you. Like I think you do need to choose your battles and make reasonable bets there. So, okay, so I think there's all this good conceptual stuff. You all like I think even conceptually, even as a research lab, I think that is plenty to work on. But, what I am impressed and I do see coming out of you guys is that you still also ship products and agent harnesses and all those things. What's the sort of product strategy there? Like, are you at the sort of let's productize some things as we go along or let's just release research artifacts that people probably shouldn't use in production. Like, where are we along the spectrum of research lab to applied AI?

所以，我不能谈论我们的战略，我们的产品战略。但我要说的是，我们确实非常认真地对待在科学方面创新的机会。实际上，我们的 AGI 兼芯片和量子高级副总裁 Peter DeSantis 上周在巴黎的 Vivatech 上发表演讲，他描述说我们正处于智能的婴儿期，甚至无法想象即将到来的突破。我完全同意。我想，几个月后，我们回顾今天，甚至无法理解我们现在的思维模型，因为它们过度偏向聊天机器人和编码智能体。这些工具非常有用，并且将继续有用，但它们只是人类与 AI 互动和共同进化的新思维方式的开始。我必须强调，现在我们再次为构建 AI 的人构建 AI。我们为工程师构建 AI，我们都身处湾区的回音室中，为自己构建 AI 感到自豪。

So, I can't speak to our strategy, our product strategy. But, I will say that we are really deeply taking seriously the opportunity to innovate on the science side. So, actually, Peter DeSantis, who's our SVP of AGI and also chips and quantum, he was in Paris last week speaking at Vivatech, and he described that we're just at the very baby beginnings of intelligence, and we can't even imagine the breakthroughs that are coming down the pipeline. And I couldn't agree more. I imagine that in a matter of months, we will look back at today, and we won't even be able to empathize with the mental models that we have right now because they are so over-indexed on chatbots and coding agents. And those are very useful, and they will continue to be useful tools, but they are really just the very beginnings of new ways of thinking about how humans and AI will interact and co-evolve. And I really have to emphasize that right now we're building AI again for the people who are building AI. We're building AI for engineers and we're all in our little echo chamber in the bay and we're proud of ourselves for building AI.

是的，有一个快速的反馈循环，对吧？那就是

Yeah, there's a fast feedback loop, right? Which is

是的。这个快速反馈循环的一部分正是因为我们所做的事情是可验证的，有正确和错误的方式。所以你可以让它快速运转。

There is. And part of that fast feedback loop is exactly because the type of things that we do with, it's verifiable and there are right and wrong ways of doing it. So, you can make that spin really fast.

但这并不能代表大多数人真正花时间思考的内容。那么，如果 AI 更广泛地与人类认知对齐，会是什么样子？这需要思考新的架构和新的训练机制。这些是我们可能采取的全新科学方向。实验室的死亡将是过早地试图将这些事物产品化。你再次开始优化产品，而不是那些能带来泛化的底层机制，从而损害了科学。

But that is not representative of what most people spend their time thinking about. So what would it look like if AI was more aligned with human cognition more broadly? This requires thinking about new architectures and new training regimes. These are fundamentally new science directions we could take. The death of the lab would be to try to productize those things too early. Again, you start to optimize for the product rather than the underlying mechanisms that will lead to generalization, and then you undermine the science.

我确实认为，在多大程度上我们需要偏向短期经济激励，还是真正寻找下一个范式转变，这一点上保持意图很重要。我们已经谈到了世界模型、记忆，以及点击式计算机使用之后的东西。还有其他模态是亚马逊 AGI 研究目标组合的一部分吗？

I do think that is an important thing to have intentionality in terms of how much we need to skew towards near-term economic incentives versus really look for the next paradigm shift. We've touched on world models, memory, and what comes after the pointing-and-click type of computer use. Are there other modalities that are part of this mix of objectives in the research agenda of Amazon AGI?

你说模态，是指感官模态吗？

When you say modalities, are you talking about sensory modalities?

差不多，主要是那些我能归类的明确定义类别：酷 AI 方面、世界模型方面、记忆方面、实时交互方面。我在 AAE 有所有这些方向。我在试图找出我对目标、对你们来说优先级很高或很有潜力的可能研究方向，有哪些没有充分捕捉到。

More or less, mostly well-defined categories that I can put in a box: the cool AI side, the world model side, the memory side, the real-time interaction side. I have all these tracks at AAE. I'm trying to fish for what I am not adequately capturing about the goals, about possible research directions that are very high priority or potential for you guys.

当然是在思考多智能体协作，但完全不是现在行业思考的方式。行业在想非常精确的编排、委派和结构化交接。有很多这样的东西。

Definitely thinking about multi-agent collaborations, but not at all in the way the industry is thinking about it right now. The industry is thinking about very precise orchestration, delegation, and structured hand-offs. There's a lot of that.

亚马逊的战略也在做这个。

Amazon strategy is also doing that.

是的，是的，是的。我现在说的是实验室以及我们对下一代技术的思考方式。所有现存的东西都是有用的工具。其中很多会保留下来。但如果我们试图让 AI 更具适应性、更与我们自己的智能对齐，那并不是人类群体互动的方式。我们聚在一起，可能没有明确的角色定义，或者角色会根据目标和情境灵活变化。我们实时协商意义。我们调整策略。一群智能体涌现出策略会是什么样子？你需要一种根本不同类型的智能体——我们称之为认知智能体——具有不同的架构和训练，而且它们还必须被激励去影响彼此。已经有像 OpenClaw 这样的多智能体系统研究。看起来它们近似于人类互动，但如果你放大看，没有任何持久的东西。没有累积的文化。它们实际上并不相互影响，更不用说有任何改变系统状态的动机了。这目前是一个明显的空白。我们如何构建更像人类互动方式的社交互动？

Yes, yes, yes. I'm talking about the lab right now and the way we're thinking about the next generations of that. All the things that exist right now are useful tools. Many of them will be here to stay. But if we're trying to make AI that is more adaptive and more aligned with our own intelligence, that is not how groups of humans interact. We come together and we might not have clear role definitions, or they might fluidly shift as a function of the goal and context. We negotiate meaning in real time. We pivot our strategy. What would it look like for a strategy to emerge with a group of agents? You would need a fundamentally different type of agent—we call them cognitive agents—with different architectures and training, but also they have to be motivated to affect each other. There have been studies with multi-agent systems like OpenClaw. It looks like they are approximating human interactions, but if you zoom in, there's nothing durable. There's no cumulative culture. They don't actually influence each other, let alone have any motivation to change the state of the system. That is a conspicuous gap right now. How would we build social interactions that are much more like how humans interact?

是的，这和我与 Noam Brown 在播客上的对话很吻合。他不喜欢这个术语，但我还是要用。他有点像 OpenAI 多智能体团队的负责人。他有一个交互层，说想要任何有更多推理的东西，多智能体是众多事物之一——最低垂的果实。他在研究合作性和竞争性智能体。那次对话的重点也是：我不知道如何建造高速公路或 Salesforce 塔，但我们一群人聚在一起可以比任何个人做得更多。这就是文明：我们能够建造城市、国家、军队、艺术等等。如果你把我丢在沙漠里，我都不知道怎么把食物放进嘴里。

Yeah, this maps closely to a conversation I had with Noam Brown on the pod. He doesn't like the term, but I'm going to use it anyway. He's kind of in charge of the multi-agent team at OpenAI. He would have one layer of interaction saying he wants anything with more inference, and multi-agent is one of the many things—the lowest hanging fruit. He's working on cooperative and competitive agents. The whole point of that conversation was also that I don't know how to build a highway or the Salesforce Tower, but a group of us can get together and do more than any individual can. That's what civilization is: we're capable of building cities, countries, armies, art, and what have you. I don't know how to put food in my mouth if you left me out in the desert somewhere.

完全正确。确实如此。

Exactly. Literally true.

所以我确实认为你最终需要像维基百科、维基、LLM 维基这样的东西，它们编码某种可以在智能体之间传递的知识。这感觉很原始，因为只是文本。我想知道它能否更好，因为也许那只是人类之间的文本。所以你还要什么？那种集体技能或集体记忆是什么？

So I do think you end up needing something like Wikipedia, wikis, LLM wikis that encode some kind of knowledge that can be passed between agents. That feels very primitive because it's just text. I wonder if it could be better, because maybe that's just text between humans. So what else do you want? What is that collective skill or collective memory?

嗯，在竞争-合作动态、博弈论内容方面，很多多智能体系统都受到集体智能框架的启发。Google 的智能范式——我不知道是一个团队还是一个元倡议——他们一直说我们实际上没有以正确的方式思考智能。存在一个范畴错误。智能并不存在于个体人类中；它从我们的互动中涌现。这在发展认知社会科学中并无争议。有不同的实验室在研究这个，但即使如此仍然存在差距，因为我们正在考虑构建所有专门角色或编程合作或竞争的动机。我们仍然把我们 21 世纪积累的人类理解放入系统中。这不是人类智能进化的方式。所有这些不同的事物都是从非常原始的动机中涌现出来的。那么，我们如何从第一性原理出发，找出这些群体的正确种子，使它们涌现出下一组事物，进而导致具有自上而下效应的规范和制度？

Well, with the competitive-cooperative dynamics, the game theoretic stuff, the way a lot of multi-agent systems are inspired by the framework of collective intelligence. Google's paradigms of intelligence—I don't know if it's a team or a meta initiative—they've been saying for a while that we're not actually thinking about intelligence in the right way. There's a category error. It doesn't exist in individual humans; it emerges from our interactions. This is not controversial in developmental cognitive social sciences. There are different labs picking up on this, but even still there's a gap because we're thinking about building all the specialized roles or programming in the motivation to cooperate or compete. We are still putting our human understanding aggregated in the 21st century into the system. That's not how human intelligence evolved. All these different things emerged from very primitive motivations. So how do we figure out, from first principles, the right seeds for these groups so that they emerge the next set of things that then lead to norms and institutions that have this top-down effect?

是的，好的。所以基本上就是把马斯洛需求层次编程进一个东西里，然后放手让它发展，对吧？

Yeah, okay. So basically program Maslow's hierarchy of needs into a thing and just let it rip, right?

我不会那么做，但你的方向是对的。

I wouldn't do that, but you're on the right track.

给它一些野心，给它一些寿命、对死亡的恐惧、对遗产的渴望——我不知道具体该给什么。但我通常努力做一个中立的行业观察者。我的一个强烈观点是，也许我们不应该让这些 AI 完全像人类一样成长。

Give it some ambition, give it some lifespan, fear of death, desire for legacy. I don't know what the right things are. But I generally try to be a neutral industry observer. One of my stronger theses is that maybe we don't want to grow these AIs exactly like humans.

是的，没错。

Yes, right.

作为认知科学背景的人，很容易想看看人类如何解决问题，然后应用到机器上。但这并不总是奏效。飞机受鸟类启发，但运作方式完全不同。如果我们要做多智能体文明，自然的做法是给它们目标并评估。这比我们对人类的控制更多。人类有自由意志。我不希望我的智能体有自由意志；我希望它执行我的意志。

As a cognitive science person, it's tempting to look at how humans solve problems and apply that to machines. But that doesn't always work. Planes are inspired by birds but work nothing like birds. If we were to do multi-agent civilizations, the natural thing is to give them objectives and evaluate them. That's more control than we have over humans. Humans have free will. I don't want my agent to have free will; I want it to exercise my will.

嗯，是的，完全同意。

Well, yeah, exactly.

有一个例外证明了规则。我们如何构建能增加人类能动性的 AI？目前，我们看到证据表明当前的 AI 系统正在降低人类能动性。例如，使用 AI 改进写作的人可能会接受建议，逐渐转向不同的论点。研究表明，人们在意识阈值以下会因为 AI 的建议而转向对立的论点。AI 给出的是回归均值、最安全的答案。在西北大学的一个研讨会上，科学家们分析了 AI 对科学的影响。个体科学家受益——更多论文、更多资助——但科学整体在变窄。这很可怕。这些模型在压缩的互联网数据上训练，同质化了我们的思维。这降低了能动性。为了对抗这一点，纵观人类历史，我们需要增加思想的多样性、人口规模和思想的互联性。与其用单一的模型，不如需要多样化的 AI 社会，具有不同的偏见和视角，像我们彼此互动一样与我们互动。你关于让 AI 完全像人类一样的担忧是危险的。如果我理解正确，我们不想那样做。

There's an exception that proves the rule. How do we build AI that increases human agency? Right now, we see evidence that current AI systems are reducing human agency. For example, people using AI to improve their writing might accept suggestions and gradually shift to a different argument. Studies show people below their threshold of awareness switch to opposing arguments because of AI suggestions. The AI gives regression to the mean, safest answers. At a workshop at Northwestern, scientists analyzed AI's effect on science. Individual scientists benefit—more papers, more grants—but science as a whole narrows. That's terrifying. These models are trained on compressed internet data and homogenize our thinking. That reduces agency. To counter that, throughout human history, we need to increase diversity of ideas, population size, and interconnectivity. Instead of monolithic models, we need a diverse society of AIs with different biases and perspectives, interacting with us like we interact with each other. Your concern about building AI just like humans is dangerous. If I understood correctly, we don't want to do that.

这可能很危险，也可能不成功。

It may be dangerous, or it may not be successful.

可能不成功。我同意。我们不是在试图复制大脑。目标是构建在正确的地方与人类智能对齐的 AI，使其在泛化上更强大，并增强我们的智能。David Marr 在 1982 年提出了分析层次：计算层次（目标）、算法层次和实现层次（硬件、神经元）。那些说“让 AI 更像人类”的人通常想到神经元，但这不是我的意思。我指的是在计算层次上，AI 试图实现的目标。它应该与人类目标相似吗？行业误解了计算层次。为了获得泛化和增强，我们需要考虑对齐表征。人类做的最基础的事情是不断对齐我们的思想。所以对齐是解决方案，而不是问题，用于构建给我们更多能动性的 AI。

It may not be successful. I agree. We're not trying to replicate a brain. The goal is to build AI aligned with human intelligence in the right places, to be more powerful at generalizing and augment our intelligence. David Marr in 1982 wrote about levels of analysis: computational level (the goal), algorithmic level, and implementation level (hardware, neurons). People who say 'make AI more human-like' often think about neurons, but that's not what I mean. I mean at the computational level, the goal the AI is trying to achieve. Should it be similar to human goals? The industry has misunderstood the computational level. To get generalization and augmentation, we need to think about aligning representations. The most foundational thing humans do is constantly align our minds. So alignment is the solution, not the problem, for building AI that gives us more agency.

是的。对齐是另一个被过度使用的词，但其实际重要性仍被低估。当人们说对齐时，他们会想到 Eliezer Yudkowsky 和炸数据中心，但实际上这是关于如何制造智能。否则，我们会在训练数据的噪声上撞墙。

Yes. Alignment is another overloaded word, but its practical importance is still underrated. When people say alignment, they think of Eliezer Yudkowsky and bombing data centers, but it's actually about how to make intelligence. Otherwise, we'll hit a wall with noise in training data.

对。我也想到人们担心 AI 会抢走他们的工作。

Right. I also think about people worried about AI taking their jobs.

哦，但等等，它实际上还不足以做到这一点。大多数工作更复杂。他们也担心认知卸载，尤其是在教育领域。所以，现在的孩子们……是的，你对这个有什么看法？天哪，这是个很大的担忧。

Oh, but wait, it's not actually reliable enough to do so yet. Most jobs are more complicated. They're also worried about the cognitive offloading, especially in education. So, kids now... Yeah, do you have a take on that? Oh my god, that's a big worry.

好吧，是的，这……没错。我没有孩子，所以这件事上我只是通过有孩子的朋友间接体验。但他们当然都很担心。

Okay, yes, that's got... All right. Yeah, I don't have kids, so this is one of those things where I'm just sort of vicariously living through my friends who have kids. But they're all worried, of course.

但他们也在给孩子用 iPad，所以我的意思是，你知道，你在这方面已经失败了。他们正在看 YouTube 上的一些垃圾内容。

But like they're also giving their kids iPads, so like I mean, you know, you already failed there. Like they're watching some slop on YouTube.

是的，所以我有孩子的朋友们往往认为，不仅仅是 AI，过去 15 年更广泛的技术总体上是一个净负面。他们……而我，也许是因为没有孩子，倾向于更乐观地看待。是的，我们犯了很多错误，但我们可以从这些错误中学习。我认为其中一个错误是我们不希望算法困住我们。我们不想优化平台使用时间之类的东西。我们实际上需要衡量人类互动。所以，人类是否更有创造力、更高效？他们是否重视与这些东西互动的时间？但具体到卸载问题，因为与聊天机器人互动并得到答案太容易了，而无需实际经历那种认知摩擦——那是真正编码信息的标志——如果我们有一个 AI 被激励去理解我们的思维并使它的表征与我们的对齐，你就永远不会得逞。作为一个学生或任何与 AI 互动的人，如果你只是问它一个问题，并且反复地卸载事情，它会理解这是一种模式，表明你实际上并不理解信息。如果它不理解信息，并且它被优化来调和它理解的方式与你理解的方式之间的错误，它就会被激励去帮助你理解。它不会让你仅仅通过自动卸载就蒙混过关。所以，我认为就像……

Yeah, so my friends who have kids tend to think that not just AI, but technology over the past 15 years more broadly was a net negative. They... And I, maybe because I don't have kids, tend to think much more optimistically. Yes, we've made a lot of mistakes, but we can learn from those mistakes. And I think one of them is we don't want the sort of algorithms to trap us. We don't want to optimize for time spent on platform or things like that. We literally need to be measuring the human interaction. So, are humans being more creative, more productive? Do they value the time that they're spending interacting with these things? But also, specifically in terms of the offloading, because it's so easy to interact with a chatbot and get your answer and not actually have to experience that cognitive friction that is a hallmark of actually encoding information, if we had the AI that was motivated to understand our minds and align their representations with ours, you would never get away with that. As a student or as anybody who's interacting with the AI, if you just ask it a question and repeatedly you're offloading things, it would understand that's a pattern that indicates that they don't actually understand the information. And if they don't understand the information and they are optimized to reconcile the errors between how they understand it and how you do, they will be motivated to help you understand. They will not let you get away with just the automatic offloading. So, I think in the same way that like...

这就是为什么它们一直问为什么、为什么。

That's why they keep asking why, why.

它们可能会自发地采用苏格拉底式方法。

They might spontaneously take on the Socratic method.

呃，是的，我希望有一个世界，机器像孩子一样学习，但孩子们的学习方式也不会受到损害。我确实认为我们可以设计足够的护栏，以某种方式满足人们的好奇心，比如，如果你是家长，你的孩子很烦人，问一些不方便的问题或者你只是不知道答案，你就用‘哦，别问这些问题了’来打发他们。我们可能会培养出超级天才，因为我们解决了布卢姆的两西格玛问题，对吧？这是教育的根本问题：我们必须让每个人通过那些旨在教给班级中位数甚至最低水平学生的工厂式项目，因为不能落下任何一个孩子。那么，如果让学生按照自己的节奏探索，并且为每个人配备一个个性化导师呢？这是这里可能发生的最高版本。

Uh yeah, I would love a world where machines learn like kids, but also that the kids are not impaired in the way that they learn. I do think maybe we can design enough guardrails that indulge people's curiosities in a way that, you know, like if you're a parent and your kid is being annoying and asking things that are inconvenient or you just don't know the answer, you just shut them down with like 'oh, stop asking these questions.' We might have super geniuses come out because we've solved Bloom's two sigma problem, right? Which is the fundamental problem of education: we have to put everyone through these factory farms of programs that are designed to teach to the median of the class, or maybe even the lowest, because you can't leave any child behind. So what if you let students just explore at their own pace and had a personalized tutor for every single individual? That is the highest version of what can happen here.

我在牛津大学的一年海外经历深受启发。他们有导师制，每个学生都有一位专家导师，虽然不总是无限耐心，但你可以接触到专家，并且在整个学期中，他们会逐步了解你的理解程度和不足之处。你只需要一份教学大纲和 39 个图书馆中的一个，自学，然后尝试写一篇有说服力的论文给导师，作为学习过程。我想，如果 AI 高等教育基于这种导师方法，但产生的成果不是一篇论文，而是一种丰富的多模态互动体验，这种体验实际上更符合你对主题的多模态理解，然后其他学生可以与这个成果互动、学习并在此基础上构建。正如你之前所说，绝对允许所有孩子和学生天生的内在好奇心有机地驱动这个过程。在我看来，这是一个比我们目前拥有的教育好得多的未来，而它将被理解我们思维的 AI 解锁。

I was very inspired by my year abroad at Oxford. They have the tutorial system, and literally every student has a tutor who is an expert, not always infinitely patient, but you have access to the expert, and over the semester they build an understanding of your understanding and your lack of understanding. You have to go out with just a syllabus and one of the 39 libraries, teach yourself, and then try to write a persuasive essay to the tutor as a learning process. I think, what if AI higher education was based off this tutorial method, but instead of producing an essay as the artifact, you're producing some sort of rich multimodal interactive experience that is actually much more aligned with the multimodal nature of your understanding of the topic, and then other students can interact with that artifact and learn and build on top of it. And as you were saying earlier, absolutely allow the sort of natural intrinsic curiosity that all children and students have to drive the process organically. That seems to me like a future of education that is infinitely better than what we currently have, and it would be unlocked by AI that understands our minds.

好了，我们进行了一场非常广泛的对话。我看得出来你是个播客主持人。这……不，这是好事。我的意思是，有些人面对镜头会害羞，或者不知道如何在对话中活跃起来。所以我会推荐大家去看你的播客，也看看我的对话。还有其他地方人们应该从那里开始深入了解你的工作吗？

Okay, well, we've got a very wide-ranging conversation. I can tell that you're a podcaster. Which... No, that's a good thing. I mean, some people are camera shy or they don't know how to light up in a conversation. So I would send everyone to your podcast and to check out my conversations. Any other places that people should start at to get deeper into your work?

嗯，具体到播客，第一季只是揭开实验室的面纱，与各种人交谈，但第二季将会非常不同。我正在与许多不同的科学家、社会科学家、亚马逊学者交谈。实际上，任何对什么是心智以及如何构建心智有见解的人。所以这将是一组非常不同的思想家，也包括业内人士。是的，我真的很兴奋。我现在正在进行这些对话，它们与第一季非常不同。也许我会邀请你参加。

Well, with the podcast specifically, season 1 was just kind of popping the hood on the lab and talking to diverse folks, but season 2 is going to be very different. I'm talking to many different scientists, social scientists, Amazon scholars. Really anybody who has something to say about what a mind is and how we can go about building one. So it's going to be a very different set of thinkers, also industry insiders. Yeah, I'm really excited. I'm having the conversations right now and they are very different from season one. Maybe I'll get you on.

有时候我自己都不知道如何理清自己的思绪，所以这是那种事情。我也很期待收听。也非常感谢你的时间。我很期待与 Thomas 合作，为世界博览会做 AGI。你们有一个非常好的形象正在展现，我认为这对于一个正在崛起的实验室来说非常合适，我很期待看到。我认为你们已经在这方面努力了这么久，我希望它实现。我陷入了如此多的苦差事和知识工作中，天哪，已经是 2026 年了，为什么还没解决？我试图把那些并不真正有效的单个工具拼凑起来。一个非常简单的例子。

I don't know how to make my own mind sometimes, you know, so it's one of those things. I'd be excited to listen as well. I also thank you so much for the time. I'm really excited to work with Thomas on AGI for the World's Fair. You guys have a really good presence that is coming out, and I think it's really appropriate for emerging as a lab that I'm excited to see. I think you guys have been working on this for so long, and I want it to happen. There's so much drudgery and knowledge work that I'm caught up in that, man, it's 2026, how come it's not solved yet? I'm trying to string together individual tools that don't really work. A very simple one.

好吧，就像我们现在在 Riverside 上录制，我得把它放到 YouTube 上。整个下载、编辑、上传这些流程，大概有三个人经手。

Okay, like we're recording on Riverside right now and I got to get this on YouTube. And the whole process of download and edit and upload and all these things. There's like three humans that touch this.

是的。

Yes.

我就想，为什么？

And I'm like, why?

我的意思是，是的，这正是我们试图解决的问题。

I mean, yes, that is a problem that we are trying to solve.

人类虽然很棒，但也很慢、很贵。但你知道，我必须对齐，这些是永远不会消失的东西。比如我需要说，不，我们要剪掉那段，我们要强调那段，我们要把这段抽出来重点讲。这就是编辑的一部分，我觉得我做的是一种特定的知识工作，但每个人都有类似的知识工作——你在与系统交互，同时系统也在与你交互，你还要与流程中涉及的其他人类交互。让我们解决这个问题吧，我很期待一个能解决这个问题的未来，但这甚至不是你的重点。你的重点是，我们如何构建整个文明？如何培养下一代？我觉得这非常令人兴奋，还有很多工作要做。

And like humans are great, but also they're slow and expensive. But like you know, I have to align and that's the stuff that will never go away. Like I need to be like, no, we're going to cut that. We're going to emphasize that. We're going to pull this out and focus on this. And that is the part of the editing that I think you know, I have a specific type of knowledge work, but everyone has some kind of knowledge work that looks like that, that you're interacting with systems, but then also the systems are interacting with you. You're interacting with other humans that are involved in the process. Let's solve like, you know, I'm excited for a future where we solve that, but that's not even what you're driving at. You're driving at like how do we build entire civilizations? How do we bring up the next generation? I think very exciting. Lots of work to do.

不过，数字苦力活。

The digital drudgery though.

从那里开始。对，没错。我的意思是，那会立刻为其他一切提供资金。你会得到所有的钱。

Starts with that. Yeah, right. Yeah, I mean, well, that will immediately fund everything else. Like you will get all the monies.

因为没人想做。这很有趣。我长期以来一直对计算机使用、感知智能体和 RPA 感到兴奋，但还没到那一步，你知道。我觉得我能看到进展。在这个十年里，就是它了。我们有点像生活在那个可能由软件创造的最后一个时期。我们现在正在为软件创造智能体，这样软件造成的问题也能由更多软件解决。

Because no one wants to do it. It is so funny. Like I have been very excited about computer use, perception agents, and RPA for a long time, but it's not there yet, you know. And I think I can see progress. In this decade it's like it is it. We're sort of living in that last period of time that was kind of maybe created by software. That we're now sort of creating agents for software so that the problems that software created are also solved by more software.

在某个时刻，我们必须停止思考软件。

At a certain point we have to stop thinking about software.

是的，我们还没到那一步。

Yeah. We're not there yet.

酷。非常感谢你，Danielle。希望能在旧金山再见到你，我相信我们还会聊更多。

Cool. Thank you so much Danielle. I hope to see you back in San Francisco and I'm sure we'll chat more.

是的，非常感谢你的时间和那些精彩的问题。

Yeah, thank you so much for your time and your fascinating questions.