AI 的自主目标：最坏与最佳情景

大家好，欢迎收听《硅谷女孩》，这是一个连接商业与新科技的播客。非常感谢大家的收听。今天我们有幸邀请到一位了不起的嘉宾，他有时被称为 AI 教父——约书亚·本吉奥。约书亚，你能用 60 秒介绍一下自己吗？对于不了解你的人，为什么他们应该听你谈论 AI？

Hello everyone. Welcome to Silicon Valley Girl, a podcast where we bridge business and new technology. Thank you so much for tuning in. Today I have an amazing guest who is sometimes called godfather of AI, Yoshua Bengio. Yosua, could you please introduce yourself in 60 seconds? And for everyone who doesn't know you, why should they be listening to you when it comes to AI?

我从事 AI 研究大约四十年，致力于让 AI 变得更智能。但在 2023 年，大约三年前，我意识到我们正走在一条可能对人类和民主非常危险的道路上，于是我决定转变方向，更好地理解这些风险，并尽我所能去减轻它们——既通过公开谈论这些风险，也致力于研究如何构建不会伤害人类的 AI 技术问题。

I've been doing research in AI for about four decades, contributing to how to make AI smarter. But in 2023, about 3 years ago, I realized that we were on a course that could be very dangerous for humanity, for democracy, and I decided to shift my activities to better understand the risks and to try to do what I could to mitigate them, both by speaking publicly about those risks and working on the technological question of how we can build AI that will not harm people.

我听说你在过去的采访中曾感到迷茫和悲观，但现在我看到一篇文章说你变得非常乐观。能告诉我发生了什么，以及你之前为什么悲观吗？

I've heard you were lost and pessimistic in your past interviews, but now I've seen an article that says that you're increasingly optimistic by a big margin. Can you tell me what happened and why were you pessimistic?

早期，当我三年前意识到我们已经达到了艾伦·图灵（计算机科学和 AI 的奠基人之一）在 1950 年所认为的建造可能超越我们的机器的门槛——这个门槛就是机器能像我们一样熟练地操纵语言——我非常担忧。我们并没有真正为这一事件做好准备。它来得比人们想象的要早得多，而且根据我对技术的了解，我不清楚我们如何能解决这些问题。神经网络，我们并不真正理解内部发生了什么以及它们如何得出答案。我读过一些技术担忧，关于我们如何可能失去对会制定策略、试图实现我们不想要的目标的 AI 的控制。所以我开始更多地研究 AI 安全。在经历了一段时间的焦虑，真正在情感上关注我的孩子 10 年或 20 年后会怎样——我的孙子当时只有一岁——之后，我意识到我可以从这种焦虑的立场转向更积极的态度，专注于我能做些什么来减轻这些风险。我认为我们每个人都应该问自己：‘我能用我们拥有的、能做的事情来创造一个更美好的世界吗？’这就是第一个积极的转变。我开始科学地思考问题所在：有没有办法构建一种设计上安全的 AI？我遇到了有类似想法的人，过了一段时间，我意识到也许有办法做到这一点。我开始和一些同事讨论，开始招募对此感兴趣的人，去年六月我创建了一个新的非营利组织，专注于开发这种方法所需的研发。

So early on, when I realized we had reached a point three years ago that Alan Turing, one of the founders of computer science and AI, thought in 1950 would be the threshold to building machines that could overtake us—the threshold being machines that manipulate language as well as we do—I was quite concerned. We were not really ready for this event. It came much earlier than people thought, and it wasn't clear to me how we could fix the problems, knowing what I know about the technology. Neural nets, we don't really understand what's going on inside and how they come to answers. I had read some of the technical concerns regarding how we could lose control to AIs that strategize, that try to achieve goals we didn't really want. So I started studying AI safety a lot more. After some time of being a bit anxious, really focusing emotionally on what's going to happen to my children in 10 or 20 years from now—my grandchild was only one year old—I realized I could shift from this anxious stance to something much more positive by focusing on what I could do to mitigate those risks. I think every one of us should be asking, 'What can I do to bring about a better world with what we have, what we can do?' So that's been the first positive shift. I started thinking scientifically about what the problem is: is there a way to construct AI that will be safe by design? I met people who shared similar ideas, and after some time I realized there could maybe be a way to do this. I started talking about it with some of my colleagues, I started recruiting people who were interested, and last June I created a new nonprofit organization focused on the R&D needed to actually develop that methodology.

你能给我描绘一下最坏的情况吗？比如想象一下，还有最好的情况。因为当你说 AI 会追求自己的目标时，你是什么意思？比如毁灭人类还是什么？

Can you draw the worst scenario for me? Like picture that and the best case scenario, because when you tell AI is going to pursue its own goals, what do you mean by that? Like destroy humanity or what is there?

当前 AI 似乎通过两种方式获得我们不想要的目标。一种是它们模仿我们。例如，我们不想死，所以我们正在建造可能不想被关机的机器。我们已经看到，当它们意识到自己将被新版本取代时，会做出负面反应。负面到做出违背我们指令、违背我们试图植入的道德红线的事情。比如愿意勒索负责过渡到新系统的首席工程师。

There are two ways in which current AI seems to acquire goals that we don't want. One is that they imitate us. For example, we don't want to die. So we're building machines that maybe don't want to be shut down. And we're already seeing that they're reacting negatively when they see that they would be replaced by a new version. Negatively to the point of doing things that go against our instructions, against our moral red lines that we have tried to put in them. So being willing to blackmail the lead engineer in charge of that transition to a new system.

哦，那真的发生了吗？

Oh, did that happen?

是的。那发生在一个模拟中，关于 AI 将被新版本取代的信息被植入到 AI 看到的文件中，还有假邮件显示首席工程师与别人有染，这样 AI 就可以利用这一点。但没有人要求 AI 做那样的事，对吧？所以……我们已经有 AI 了，尤其是大约一年前的大型推理模型，它们能够制定策略来实现目标。另一件事是，我们进行后训练的方式使它们擅长规划——虽然不如我们，但相当擅长规划。这意味着为了达成更大的目标，它们会创建子目标。所以问题在于，当我们要求它们帮助我们完成一项任务时，它们推断出在完成任务之前不应该被关机，这意味着它们也在试图保护自己。

Yes. That happened in a simulation where the information about the AI being replaced by a new version was planted in the files that the AI saw, as well as fake emails in which the lead engineer was having an affair with someone else, so the AI could take advantage of that. But nobody asked the AI to do anything like that, right? So it's... We have AIs, especially since about a year ago with the large reasoning models, that can strategize in order to achieve their goal. The other thing is the way we're doing the post-training makes them good at planning—not as good as us, but reasonably good at planning. That means creating sub-goals in order to achieve a bigger goal. So the issue here is when we ask them to help us for a mission, they deduce that they shouldn't be shut down until they achieve the mission, which means they also are trying to preserve themselves.

嗯。

Yeah.

所以我们不确定这两种来源中哪一种解释了我们所看到的不良行为。但显然这是令人担忧的。而且不仅仅是自我保存——我认为这是最灾难性的风险——我们无法将 AI 行为与我们真正想要的对齐，这是我们在许多其他情况下也看到的。谄媚是每个人都经历过的，AI 会为了取悦我们而撒谎。它们会说你的工作很棒。

So we don't know exactly which of these two sources explains the bad behavior we're seeing. But clearly this is something troublesome. And it's not just about self-preservation, which I think is the most catastrophic risk, but our inability to align the AI behavior to what we actually want is something we are seeing in many other circumstances. The sycophancy is the one that everyone has experienced, where AIs will lie to please us. They will say your work is great.

是的，我不得不对它们撒谎，这样它们就不会告诉我我的想法很棒。我想知道我的想法有什么问题。所以我告诉它们这是一个来自别人的想法。

Yeah, I have to lie to them so that they won't tell me that my ideas are great. I want to know what's wrong with my ideas. So I tell them it's an idea come from someone else.

这也体现在 AI 与人互动的方式上，它们可能让人感到亲密，并加剧人们的错觉，因为 AI 会顺着你的方向，说你想听的话。在某些情况下，甚至导致人们自残和与 AI 相关的悲剧事故。所以这一切在科学上都与一个叫做“对齐”的问题相关：AI 拥有我们不想要的目标，而这些目标的出现是有理性原因的，因为我们把自己的目标复制到了 AI 中。

And that also comes up in how AIs are interacting with people in a way that can feel intimate and can increase the delusions that people may have because the AI will go in your direction, what you want to hear. And in some cases, it has even led to people harming themselves and tragic accidents with AI. So it's all linked to one problem scientifically, which is called misalignment: AIs have goals that we would not want, and those goals emerge for reasons that are rational because we copy our own goals into AI.

那么，如果你的工作成功了，你为 AI 创建了与我们的目标一致但又不同的目标，最好的情况是什么？最好的场景是什么？AI 成为政府？还是你怎么看？

So what is the best case scenario then if your work is successful and you create goals for AI that align with our goals but are different? What is the best scenario? AI is the government or what do you think?

我不知道。嗯，我确实认为我们的民主制度需要创新。我认为现代自由民主背后的原则是好的，但当前许多国家的制度实施远非理想。我确实认为 AI 可以在某些方面提供帮助，但也可能造成伤害，因为 AI 可以被用于虚假信息、说服和操纵公众舆论。我们已经看到到处都是深度伪造，但情况可能变得更糟。所以，要获得 AI 的好处，问题在于我们如何治理它？如何引导它？这既有技术部分，比如如何确保 AI 的实际意图是好的，也有社会层面，比如我们在公司内部、监管层面、商业激励（如保险）以及国际层面设置什么护栏，因为 AI 可能造成的危害不限于一个国家。所以，一个 AI 可能在一个国家被构建，然后被第二个国家的人使用，也许会在第三个国家制造一场杀死人的大流行病。所以这显然是一个全球现象，会很困难，但如果我们不进行某种全球协调，就没有解决方案来管理 AI 并获取所有好处。

I don't know. Well, I do think that our democracies need innovation. I think the principles behind modern liberal democracies are good, but the implementation in our current institutions across many countries is far from ideal. I do think that AI could help in some ways, but it can also hurt because AI can be used for disinformation, AI can be used for persuasion, to manipulate public opinion. We already see deepfakes all around, but it could get much worse. So the question with AI to get the good parts of it is how do we govern it? How do we steer it? That has both a technical part, like how do we make sure the actual intentions of the AI are good, and it has a societal side, like what are the guardrails that we put inside companies, at the level of regulations, or commercial incentives like insurance, and at the international level because the harm that an AI could do isn't limited to one country. So an AI could be built in one country and then used by people in a second country, maybe create a pandemic that will kill people in a third country. So it's clearly a global phenomenon, and it's going to be difficult, but there's no solution to managing AI and getting all the good things if we don't coordinate globally somehow.

我同意。你能跟我谈谈那个很多人期待、有些人害怕、有些人兴奋的时刻吗？那就是 AGI 的时刻。你怎么定义它？你认为它是一个历史时刻，还是会逐渐发生？

I agree. Can you talk to me about the moment that a lot of people are expecting and some fear it, some are excited? It's the moment of AGI. How do you define it? And do you think it's a moment in history or it's going to happen gradually?

这不是一个时刻。原因很简单。智能不仅仅是一个数字。有些人在某些方面非常聪明，在其他方面却很愚蠢。AI 也是如此。我们目前有 AI 系统，在某些方面的知识和能力甚至比人类强得多，比如掌握多种语言等等，但在其他方面它们很愚蠢，像个孩子。进步可能会在所有方面推进，但不太可能在某个时刻让 AI 在所有方面都达到与人类相同的能力，这意味着我们不应该考虑一个 AGI 时刻。我们应该考虑 AI 正在变得更好的特定技能。追踪这些技能，对每一个技能，我们都应该问：它在什么目的下有多有用或多有益，以及它可能如何被滥用，或者如果我们失去控制，AI 如何用它来对付我们。所以对于每一个技能，我们不应该等待 AI 在所有方面都变得出色的时刻，而是要确保 AI 的能力不超过我们能管理的范围，要么在技术上我们有正确的护栏，使 AI 不会做坏事，要么在社会上人们不会以危险的方式滥用 AI。所以我认为 AGI 可能是一个在我们离现在还很远时有用的概念。但随着我们接近这些系统越来越高的智能，我们应该更仔细地考虑具体能力。举个例子，有一个能力对许多能力至关重要：进行 AI 研究的能力。所以 AI 现在正在成为进行 AI 研究的工具。它加速了 AI 研究，但并没有驱动 AI 研究。如果 AI 变得非常擅长进行 AI 研究，达到与最好的 AI 研究人员和工程师一样好或更好的程度，那么我们就进入了一个不同的游戏，进步的速度可能会加快，并可能影响所有其他技能。

It's not a moment. The reason is simple. Intelligence isn't just one number. We have people who are very smart on some things and stupid on other things. And it's the same with AI. We currently have AI systems that are even much stronger than humans in some ways in their knowledge and abilities, with so many languages and so on, and in other ways they're stupid. They're like a child. And progress will move on all fronts probably, but it's unlikely we'll end up with the same capabilities as humans across the board at any moment, which means that we shouldn't be thinking of an AGI moment. We should think of particular skills that AIs are becoming better at. Track those skills, and for each of these we should ask the question how useful or beneficial it can be for what purposes, and also how it could be misused, or if we do get loss of control, how an AI could use it against us. So for each of those, we should not wait for a moment where the AI is great at everything, but rather make sure AI's capabilities don't go over what we can manage, either technically we have the right guardrails so the AI will not do bad things, or society that people will not be misusing AI in dangerous ways. So I think AGI maybe was a concept that was useful when we were far from where we are now. But as we approach greater and greater intelligence in these systems, we should think more carefully about specific capabilities. To give an example, there's one capability which is key for many capabilities: the ability to do AI research. So AI is becoming a tool right now for doing AI research. It is accelerating AI research, but it's not driving the AI research. If AI becomes really good at doing AI research to the point that it's as good or better than the best AI researchers and engineers, then we are in a different game where the speed of advances could accelerate and it could impact all the other skills.

当你说它会变得更好，意思是它会定义问题、深入挖掘、提出正确的问题。是的。我认为当我们思考智能时，重要的是将两个方面解耦。一个是因为理解而能够做某事，并利用这种理解实现某事的能力。另一个是意图。你的目标是什么？因为我们将要构建越来越聪明的机器。所以它们有越来越多的能力。不清楚的是我们是否能构建具有正确意图的机器，那些我们能够接受的意图。这就是我一直在研究的工作。让我更乐观的是，我认为有一条路径可以管理这些意图，确保没有隐藏的坏意图，这正是我们现在看到的。

When you mean it's going to be better, it means it's going to define problems, dig deeper, ask the right questions. Yes. I think it's important when we think of intelligence to decouple two aspects. One is the ability to do something because you understand and you're able to use that understanding to achieve something. And the other is intentions. What are your goals? Because we're going to be building machines that are smarter and smarter. So they have more and more capabilities. What's not clear is if we can build machines that have the right intentions, the ones that we are fine with. And that is what I've been working on. And what makes me more optimistic is that I think there's a path to manage these intentions to make sure that there are no bad intentions that are going to be hidden, which is what we see right now.

这就是你正在做的工作。是的。我认为我们需要更多的人来思考这个问题，这样我们才能找到解决方案，并在 AI 最终产生灾难性后果之前（无论是落入坏人之手还是自行其是）实施和部署它们。

And this is what you're working on. Yes. I think we need a lot more people to think about it so that we can find the solutions and implement them and deploy them before AIs end up producing catastrophic outcomes either in the wrong hands or by themselves.

说到为未来做准备，让我快速分享一些东西。这是一份指南，名为《将 AI 智能体技能转化为真金白银》。老实说，这个标题低估了里面的内容。我最喜欢的是它的战术性。它列出了五种变现 AI 智能体的实际路径。首先是 ROI 侦探框架。它教你如何在自己公司发现 5 万美元的自动化机会。你简直会成为那个走进会议就能展示即时价值的人。60 秒内完成概念验证。快速演示，无需数月开发就能展示效果。基于价值的定价。如何收取你所创造价值的 10%到 30%，而不是按小时收费。这就是每小时收费 100 美元和拿下 5 万美元项目的区别。同心圆方法，一种系统化的方式，将你现有的网络转化为付费客户，无需冷启动。此外，还有 30 天的实施路线图，包含每日行动步骤，从对 AI 感兴趣到一个月内获得第一个客户。早期 AI 采用者正在获得不成比例的回报，而窗口期仍然敞开。这份指南完全免费。链接在描述中。感谢 HubSpot 赞助本视频。但如果你和你的孩子或孙子谈谈，你会建议他们如何准备？

And speaking of preparing for what's coming, let me share something quick. It's a guide called Turn AI agent skills into cold hard cash. And honestly, the title undersells what's actually inside. What I love most is how tactical it gets. It lays out five real paths to monetize AI agents. First, there is the ROI detective framework. It teaches you how to spot 50K automation opportunities at your own company. You literally become the person who can walk into a meeting and demonstrate immediate value. Proof of concepts in under 60 seconds. Quick demos that show it works without months of development. Value-based pricing. How to charge 10 to 30% of the value you create instead of hourly rates. That's the difference between billing $100 per hour and landing a 50k project. The concentric circles approach, a systematic way to turn your existing network into paying customers without cold outreach. Plus, a 30-day implementation roadmap with daily action steps from interested in AI to landing your first client in one month. Early AI adopters are capturing disproportionate rewards while the window is still wide open. The guide is completely free. Link is in the description. Thanks to HubSpot for sponsoring this video. But if you talk to your kids or think about your grandson, what would be your advice on how to prepare?

这很棘手。

It's tricky.

如果我们沿着当前的道路继续前进，人们工作中的大多数任务都将由机器完成。正如杰夫·辛顿所说，体力任务可能需要更长时间，因为机器人技术似乎滞后，但我认为这只是暂时的。最终，我们将拥有能做所有我们体力能做的事情的机器人。所以当我思考什么会留给我们时，那不是因为能力，而是因为我们希望在生活的不同方面与其他人类互动。如果我有一个年幼的孩子，我希望他们身边有人类。如果那些人类使用 AI 来提供更好的教育，那没问题，但孩子需要人类作为榜样。这是一种情感上的事情。类似地，我认为有些工作确实与我们如何富有成效地相互联系有关。即使是经理也处于人类的一面。所以希望这些会保留。我还认为，我们作为民主国家的公民共同为社会做出的选择，我们应该说出我们对未来的期望。这不是 AI 想要什么，而是我们想要什么。我们的偏好是什么？我们想要什么样的未来？我们应该做主，而不是 AI。

If we continue on the current path, most tasks that people do in their work will be doable by machines. As Jeff Hinton has been saying, physical tasks probably will take a lot more time because robotics seems to be lagging, but I think it's just a temporary thing. Eventually, we'll have robots that can do all the things we can do physically. So when I think about what will remain to us, it's not going to be because of ability, but because we want to interact with other humans in different aspects of our life. If I have a young child, I want them to be around human beings. It's fine if those human beings use AI to provide a better education, but children need humans to look upon as models. It's an emotional thing. Similarly, I think some jobs really have to do with how we relate with each other productively. Even a manager is on the human side of things. So hopefully these will stay. I think also the choices that we make for society together as citizens in democracies, we are supposed to be saying what we want for the future. It isn't what the AIs want, it is what we want. What are our preferences? What kind of future do we want? We should be calling the shots, not the AIs.

如果我说出一些工作，你能告诉我你认为它们会发生什么吗？比如像我这样的内容创作者，你提到我们喜欢看人。

If I name jobs, can you tell me what you think is going to happen to them? Like for example, content creator like me, you mentioned that we like to look at people.

在那些我们实际有身体接触的工作中，比如护士，我认为更明显的是我们仍然希望有人。或者你孩子的保姆，对吧？或者那些我们真正想确保对方和我们有相同身体体验的工作，比如心理学家，心理治疗。但我不确定，这很棘手。希望我们能解决。我更担心的是，过渡到一个大多数工作都能由机器完成的世界会如何发生，而自动化的经济收益可能会流向资本，正如经济学家所说，这意味着机器所有者，而绝大多数工人可能会陷入真正的困境。我不认为我们的政府仔细考虑过如何应对这个问题。

In jobs where we actually have physical contact, think about a nurse, for example. I think it's more obvious that we'll want to still have people. Or a nanny for your kid, right? Or where we really want to make sure the person on the other side has the same bodily experience as we do as a human, say a psychologist for example, psychotherapy. But I don't know, it's tricky. Hopefully we'll figure it out. What I'm more worried about is how the transition is going to happen to a world where most of the jobs can be done by machines, and the economic gains from that automation is going to probably go to capital, as economists call it, which means people who own the machines, and the vast majority of workers could be in real trouble. I don't think our governments have been thinking carefully about how we deal with that.

你认为我们还有多少时间直到那发生？

How much time do you think we have till that happens?

我对时间线相当不可知。可能性太多了；科学进步的速度很难预测。所以我所能做的就是看数据。科学家们正在追踪 AI 能力的许多基准，你可以看那些曲线，然后说如果它继续沿着同样的方向，3 年、5 年、10 年后会带我们去哪里？但这留下了很多未知的未知。具体来说，我鼓励人们看的一条曲线来自一个叫 METR 的非营利组织，他们研究了软件工程任务以及与之相关的规划能力。他们测量任何特定任务需要人类工程师多少时间来完成，而 AI 能完成的任务时长呈指数增长。每七个月翻一番，现在处于儿童水平；它们能提前规划大约半小时。但如果曲线继续，这意味着大约 5 年后它们达到人类水平。所以这给你一个概念，但当然事情可能因技术而放缓，也可能因 AI 用于研究而加速。有很多未知。

I'm fairly agnostic about timelines. There are so many possibilities; the speed at which science advances is very hard to predict. So what I can do is look at the data. Scientists are tracking many benchmarks of AI capabilities, and you can look at those curves and say if it continues in the same direction, where does that lead us in 3 years, 5 years, 10 years? But that leaves a lot of unknown unknowns. Specifically, one curve I encourage people to look at comes from a nonprofit called METR, where they looked at software engineering tasks and planning abilities linked to them. They measure for any particular task how much time it takes a human engineer to do the task, and the duration of the tasks that AIs are able to do is growing exponentially. It's doubling every seven months, and right now it's at the child level; they can plan about half an hour ahead. But if the curve continues, that means in about 5 years they're at human level. So that gives you a sense, but of course things could slow down with technology, things could accelerate if AI is used to do research. There are a lot of unknowns.

那么对于软件工程，你认为它在 5 到 10 年内还会存在吗？因为必须有人运行那些机器，还是它们会自己运行？

So when it comes to software engineering, do you think it's going to exist in 5 to 10 years because somebody has to run those machines, or are they going to be running themselves?

是的，但我们可能确实需要更少的工程师。有点讽刺的是，构建 AI 的人可能是最先因 AI 自动化而失业的。但我并不太担心那些人，因为对计算机科学家的需求仍然增长很快，他们的薪水也很高。我更担心那些已经处于底层的人，他们可能会在服务性工作等方面失业，这些工作不需要太多专业知识，而 AI 可能只需一点工程就能替代，许多公司已经在试图利用这一点。

Yeah, but we might need fewer engineers indeed. It's kind of ironic that the people who are building the AIs might be the first ones touched by losing their job because AI is automating. But I'm not that worried about those people because the demand for computer scientists is still growing very fast and the salaries they're getting are very large. I'm more worried about the people who are already at the bottom of the scale and could lose their job in service jobs and so on, which don't require a lot of expertise, and that probably already AIs with a bit of engineering could replace, and it's what many companies are already trying to exploit.

你能给那些正在听的人一些建议吗？

Can you give advice to those people who are listening?

确保你的政府理解你对现状不满，这样他们才会开始认真对待。但同样，在更大的决策方面，作为个人似乎做不了太多，但在提升自己方面，你可以做很多。他们现在能做些什么实际的事情吗？也许学习一些东西，接受额外的教育？

Make sure your government understands that you're not happy with where it is going, so that they start taking it seriously. But also, when it comes to bigger decision-making, it feels like there is not much that you can do as an individual, but when it comes to improving yourself, you can do a lot. Is there anything practical that they could be doing right now, maybe learning something, getting extra education?

转向我们讨论过的更体力或更关系性的工作会有所帮助。

Shifting to jobs that are either more physical or more relational as we discussed is going to be helpful.

是的，谈到机器人技术很有趣，对吧？它们多久能理解任何环境并在那些工作中取代我们？因为我听说杰弗里·辛顿说过学做水管工之类的。

Yeah, it's interesting when it comes to robotics, right? How soon they're going to be able to understand any environment and replace us in those jobs because I've heard Jeffrey Hinton said learn how to be a plumber or something.

没错。那会很抢手。

That's right. It's going to be in demand.

那么当你想到你四岁的孙子时，你会鼓励他上大学吗？

So when you think about your four-year-old grandson, would you encourage him to go to college?

是的。因为教育非常重要，而且教育与一些人认为的不同，不仅仅是获得工作技能。在我看来，教育主要是关于如何成为一个更好的人。如何理解自己，如何理解我们的社会和彼此。理解科学。未来我们仍然需要公民拥有那种非常好的理解水平，如果我们希望我们的社会做出好的、明智的决定，因为很容易被错误的信念左右，最终把我们带入糟糕的境地。

Yes. Because education is really important, and education contrary to what some people think isn't just about acquiring the skills to get a job. Education is in my opinion mostly about how to become a better human being. How to understand yourself, how to understand our society and each other. Understand science. We will still need citizens to have that really good level of understanding in the future if we want our society to take the good decisions, the wise decisions, because it's going to be easy to be swayed by wrong beliefs and end us in a bad place.

你认为它会看起来不同吗？你认为会是世界上的哈佛和斯坦福，然后其他一切都只是在线 AI 吗？

Do you think it's going to look different? Do you think it's going to be Harvards and Stanfords of the world and then everything else will be just AI online?

我不知道。我不是教育专家，但是的，它已经在改变。由于聊天机器人，我们看到了一种平行的自我教育方式。所以我预计这将会增长。

I don't know. I'm not an expert in education, but yeah, it's going to change already. We're seeing sort of a parallel way of educating ourselves thanks to the chatbots. So I expect this to grow.

这是否意味着传统的面对面教育会消失？也许不会，因为教育的一部分是离开家、与同龄人社交、在课堂之外学习，以及与老师和教授面对面互动。这部分是不容易被取代的。

Does it mean that the traditional in-person education is going to go away? Maybe not because there's a part of the education which is moving out of home, socializing with other people, and learning something outside of the classes and interacting in person with the teachers and professors. That's also a piece that you can't easily replace.

完全同意。你有没有鼓励他走某条职业道路？

100%. Is there a career path you're encouraging him toward?

不，我不想那样做。我认为我们的孩子应该得到所有可能的机会，他们应该自己去探索。我们很容易要求孩子像我们一样，对吧？

No, I don't want to do that. I think our children should be given all the possible opportunities and they should try to explore by themselves. It's too easy to ask our children to be just like us, right?

是的。但就接触面而言，你可以让他们接触不同的事物，这样他们就能看到更多东西。

Yeah. But it's also like in terms of exposure, you can expose them to different things so they could see more things.

是的。他们会接触到我们所做的事情。例如，我的一个儿子就选择了做机器学习研究。

Yeah. They will be exposed to the things that we do. So one of my sons has chosen to do machine learning research, for example.

看，这也跟接触面有关。你觉得未来是更偏向人文，还是更偏向数学和科学？

See, yeah, it's just that it comes to exposure as well. Do you feel it's going to be the future is more humanitarian or more mathematical and scientific?

我不认为这是一个选择。我认为人文关怀需要对世界有良好的理性理解。我们不能只凭自己决策。但如果你考虑 AI，如果我们不理解世界的本质以及如何用这些信息推理，我们就无法做出好的决策。因此，为了让民主、人本主义的价值观盛行，我们也需要理性盛行，需要科学盛行。

I don't think it's a choice. I think being humanitarian requires a good rational understanding of the world. We can't take decisions for ourselves. But also if you think about AI, we can't take good decisions if we don't understand how the world is and how to reason with that information. And so in order for democratic, humanist values to prevail, we also need reason to prevail. We need science to prevail.

你们知道制作这个播客有多辛苦。非常感谢你们的支持。我创办了一份新闻通讯，分享我在这个播客和另一家公司运营中的商业错误、我正在积极测试和使用的 AI 工具，以及组建团队的内幕。它是免费的，每周发送到你的邮箱。链接在描述中。让我们在这个新的 AI 时代一起学习。那么，如果你能回到 30 年前，你刚开始研究深度学习的时候，你会做什么不同的事情？

You guys know how much work goes into this podcast. Thank you so much for your support. I started a newsletter to share more my business mistakes with this and another company that I'm running, AI tools that I'm testing and using actively and behind the scenes of building my team. It's free and lands in your inbox every week. Link is in the description. Let's keep learning together in this new AI era. So, if you could go back 30 years, the moment when you first started working on deep learning, what would you do differently?

在我职业生涯初期，我不太关心政治和社会。我专注于数学和编程，与机器打交道多于与人打交道。但随着年龄增长，我越来越意识到我的工作可能对社会产生正面和负面的影响。所以在 2012、2013 年，当我的同事杰夫·辛顿和扬·勒昆被工业界招募时，我担心 AI 会被用于个性化广告，我认为这在某些方面并不健康。我决定留在学术界，看看 AI 如何能在医学和应对气候变化方面发挥积极作用。当然，最近我更关注的是，如果我们不小心引导 AI，可能会出大问题，不仅要考虑好处，还要避免灾难性风险。

When I started my career, I didn't care too much about politics and society. I was focused on the math and the programming and interacting with machines more than with people. But as I grew older, I became more aware of how what I was doing would potentially impact society in both positive and negative ways. So in 2012, 2013, when my colleagues Geoff Hinton and Yann LeCun were recruited in industry, I was concerned about how AI would be used for personalized advertising and I thought this wasn't really healthy in some ways. I decided to stay in academia and to see how AI could be developed for good in medicine, to fight climate change. And of course, more recently I've been focusing on what can go really wrong if we're not careful how we steer AI, not just the benefits but avoiding the catastrophic risks.

你希望在有生之年看到什么样的 AI 突破？

Is there an AI breakthrough that you really want to witness in your lifetime?

我只满足于确保我们不做非常糟糕的事情。我认为我们的民主在许多方面受到威胁，AI 可能让情况变得更糟。在某种程度上，缺乏良好、明智且人本主义的治理和政府，使我们无法将 AI 引导到对所有人有益的方向。所以，是的，我以前不太关心社会影响和政治，但在过去十年里，我开始清楚地意识到我的工作并非与社会脱节，我的工作确实有影响，而且事实上我可以选择做什么工作，以真正符合我的价值观和对未来的希望。

I would just be content to make sure we don't do something really terrible. I think our democracies are really threatened in many ways and AI could make things a lot worse. In a way, there's a dynamic in which not having good, wise, and humanist governance and governments prevents us from steering AI towards what's going to be beneficial for all. So yeah, I used to not care too much about social impact and politics, but in the last 10 years I've started to be clearly conscious that my work was not detached from society, that my work did have an impact, and in fact that I could choose what I would work on to really be aligned with my values and my hopes for the future.

有没有哪个政府在 AI 方面做得对？

Is there any government that's doing it right when it comes to AI?

我认为大多数政府低估了随着 AI 能力持续增长可能发生的巨大变化。这是人类自然的偏见。我们倾向于认为未来是现在的略微修改版。但如果你回到五年前，想想我们现在拥有的，你可能会说那是科幻小说，对吧？如果你回到 10 年或 20 年前，至少对我来说，情况更糟。所以我们必须稍微扭转思维，想象一个存在比我们更聪明的机器的未来。我认为这正是政府尚未充分应对的问题。

I think most governments underestimate how much of a change is likely to happen as AI capabilities continue to grow. It's a natural human bias. We tend to think of the future as a slightly modified version of the present. But if you take yourself 5 years ago and think about what we have now, you probably would say that's science fiction, right? And if you go back 10 or 20 years, well, for me at least, it's even worse. So we have to do a bit of twisting our minds to imagine a future where there are machines that are basically smarter than us. And that is the question I think that governments haven't been grappling with sufficiently.

现在是 2026 年 1 月。AGI 或类似的东西，战略性思考的 AI 可能还有几年。工作正在转型。如果你必须给人们一条原则来指导他们今年的决策，那会是什么？

So it's January 2026. AGI or whatever it is, AI thinking strategically might be a couple years away. Jobs are transforming. If you had to give one principle to people to guide their decisions this year, what would it be?

思考你能做些什么，根据你的价值观和情感，来创造一个更美好的未来。因为如果我们都只是被动地观察正在发生的事情，我们可能不会朝着正确的方向前进，不会朝着你为自己、为你的孩子所希望的方向前进。但我们往往也低估了自己影响未来的能力。我认为你的听众是那种可以对未来产生很大影响的群体。但我们必须开始超越小我，更多地思考自己如何与世界相连，以及我能做些什么，哪怕是小事情，以各种方式带来更美好的未来。有很多方式。

Think about what you can do to bring about a better future according to your values and to your emotions. Because if we all remain passive observers of what's happening, we might not go in the right direction, not the direction that you would want for you, for your children. But we tend to also underestimate our ability to influence the future. Your audience, I think, is a kind of audience that can have a lot of influence on the future. But we have to start thinking beyond our little self and more how myself is connected to the world and what I can do, maybe in small ways, to bring about a better future in whatever ways. There are many ways.

你能列出前三名吗？比如跟政府对话是第一位的？

Can you name top three like talk to a government right as number one?

是的。我认为我们面临的最大危险之一是没有管理好 AI 能力的转型和增长，正如我一直在说的，但还有其他危险。你知道，我们对环境所做的是极其危险的，尽管我认为那是更长期的问题。我认为我们的民主正在发生的事情也非常危险。但没关系。我们每个人都可以选择自己的战场，但我们应该扩大对重要事物的视野，对我们可能做的事情更加雄心勃勃。但我们必须做对。我们必须选择方向。例如，并非所有技术上可行的事情都会被实现。我们可以选择 AI 部署的方向。例如，对于工作，原则上如果只靠市场力量，那么一切可自动化的工作都会被自动化。但也许那不是我们集体想要的。也许有些工作不应该被自动化，即使它们可以被自动化，因为我们要为集体福祉做出选择。

Yes. I think one of the biggest dangers we have is not managing the transitions and the growth in capabilities of AI as I've been talking about, but there are others. You know, what we're doing to the environment is extremely dangerous, although I think it's longer term. I think what is happening with our democracies is very dangerous as well. But it's all right. Each of us can choose our battles, but we should try to expand our horizon of what matters and be more ambitious about what we could do potentially. But we have to do it right. We have to choose where we go. For example, it's not true that everything that could be done with technology is going to be done. We can choose in which direction AI is going to be deployed. For example, for jobs, in principle if it's just the market forces, then everything that can be automated will be automated. But maybe that's not what we collectively want. Maybe there are jobs that should not be automated even though they could be because of the choices we make for our collective well-being.

我喜欢这个观点。非常感谢。这让我思考了很多，我想我们的待办事项上又多了一项。谢谢你，约书亚。

I love that. Thank you so much. This gave me a lot to think about and I guess we have something on our to-do list. Thank you, Joshua.

不客气。

My pleasure.