从哲学到人工智能：穆斯塔法·苏莱曼的旅程

大家好，很高兴见到这么多朋友，还有我丈夫，这有点吓人——他从来不出席我参加的任何活动。但能在这里和一位写了本关于当下最热门话题的畅销书的人交谈，真是太好了。你们有多少人用过 ChatGPT？请举手——几乎所有人。我想你们大概都同意，ChatGPT 是去年 11 月出现的，直到那时大多数人才意识到人工智能，特别是生成式 AI 模型，即将改变世界。突然之间，全球都产生了一种集体的‘天哪，这种能力太惊人了’的感觉。这反映在无数的社论和焦虑的政客身上。我想我说得没错，主要焦点都在负面影响上。每个人对存在风险的概率都有自己的看法。我们都会自杀吗？一切都糟透了。而穆斯塔法在这个领域是一位极具可信度的人。他不仅联合创办了一家，而是两家成功的 AI 公司。在这本书中，他以冷静、现实且极具说服力的方式审视了我们的未来。这就是为什么你真的应该读一读。这本书很棒，我读了两遍。你应该读一读。穆斯塔法，简单介绍一下——我想在座各位对他并不陌生——他是 2010 年 DeepMind 的联合创始人，后来又与里德·霍夫曼共同创办了 Inflection AI。他的联合创始人里德·霍夫曼在 ChatGPT 的帮助下，对这项技术的潜力持非常乐观的态度，所以我很想听听你们之间的辩论。几年前，他因对英国科技领域的远见服务和影响力获得了 CBE 勋章。他还是《经济学人》的董事会成员，所以我能近距离看到穆斯塔法的工作。他是《经济学人》的朋友，也是英国科技界的杰出人物。我认为这本书的起点是——这本书叫《即将到来的浪潮》——如果你最近打开电视或听播客，你会知道，别说即将到来的浪潮了，已经有一波宣传和人们对这本书的赞赏。我相信你已经做了 60 场各种形式的露面，所以请感到幸运，或者算上这场是 61 场。但可以理解的是，这本书产生了巨大的影响，因为它非常有趣、非常有思想，而且涉及当下最热门的话题。所以我们大部分时间想谈谈这本书，但我也想为那些不了解穆斯塔法的人提供一些背景。首先，穆斯塔法其实不是计算机极客。你没有学过计算机编程，对吧？你在牛津大学学习哲学和神学。那么你能简单介绍一下，一个学习哲学和神学的人是如何成为两家科技公司的联合创始人的吗？你在做什么？

Hello everybody, it's great to see so many friends, and indeed my husband, which is a bit alarming—he never turns up to any event I do. But it is great to be here to talk about literally one of the hottest topics of the moment with someone who has written one of the best books about it. How many of you have used ChatGPT? Just a show of hands—virtually everybody. I think you'd probably then agree that ChatGPT came in November last year, and it was only then that most people realized that artificial intelligence, generative AI models in particular, were about to change the world. Suddenly there was a kind of collective global 'Oh my God, this capability is extraordinary.' And it's been reflected in endless editorials, hand-wringing politicians. I think I'm right in saying the main focus has been on the downsides. Everyone has their pet view of what the odds are of existential risk. Are we all going to kill ourselves? It's all terrible. And Mustafa comes into this as a man with considerable credibility. He is a man who has co-founded not just one but two successful AI companies. And he's a man who in this book takes a sober, realistic, and actually very compelling look at what lies ahead of us. So that's why you really should read it. It's great. I've read it twice. You should read it. Mustafa, just to give you some—he doesn't need much introduction, I don't think, to this group—but he was a co-founder of DeepMind back in 2010. He then was a co-founder of Inflection AI with Reid Hoffman. Reid Hoffman, his co-founder, has with the help of ChatGPT written an extremely upbeat view of the potential of this technology, so I'd love to know the debates between the two of you. He got a CBE a few years ago for his visionary services and influence in the UK technology sector. He is also on the board of The Economist, so I get to see Mustafa working up close. He's a friend of The Economist, a great figure in British technology. I think the place to start with this book—and the book is called 'The Coming Wave'—you will know that there has been, if you've turned on your TV or listened to a podcast recently, you will know that never mind the coming wave, there is already a wave of publicity and people being impressed with this book. I believe you've had 60 appearances of various sorts, so consider yourselves lucky, or 61 on this list. But understandably, the book has had a tremendous impact because it is very interesting, very thoughtful, and it's on the hottest topic of the moment. So we want to talk most of the time about the book, but I do want to get a little bit of background for those of you who don't know Mustafa. The first is that Mustafa is actually not a computer geek. You didn't study computer code, right? You studied philosophy and theology at Oxford. So can you just give us the potted history about how a man who studied philosophy and theology comes to be the co-founder of two tech companies? What are you doing?

我一直认为哲学是一种系统思维工具。它让我能够严谨清晰地思考。从一开始，大概 19 岁时，我实际上从哲学学位辍学了——你不知道吧？是的，我没完成学业——我深受能在世界上产生影响这一动机的驱动。我离开学校去帮助创办一个慈善机构。当时是一个电话咨询服务，叫做穆斯林青年热线，这是一个世俗的——尽管我在穆斯林背景下长大，但我是个无神论者——这是一个世俗服务，旨在为英国穆斯林青年提供信仰和文化敏感的支援。那是 2003 年。我发现自己身处牛津，研究这些非常理论化、深奥的思想，而我想在伦理方面将真实事物付诸实践，这就是为什么我去创办了热线，并作为志愿者工作了三年。很快，我对非营利组织的影响力规模感到沮丧。我曾在当时的伦敦市长肯·利文斯通手下短暂工作，担任人权政策官员，这很鼓舞人心，但我仍在为影响力规模而挣扎。我意识到，如果我不抓住真正让我们作为物种组织起来并高效运作的东西——利润激励——那么我将错过我一生中最重要的事情之一。当时我看到了 Facebook 的崛起，大约是 2007-2008 年，它在两年内增长到 1 亿月活跃用户。我完全被它从无到有的快速增长所震撼，这对我来说完全是新事物。于是我踏上了寻找任何愿意教我技术的人的旅程。在那之前，我创办过几家企业——两家不同的公司，其中一家是科技公司，在诺丁山附近的餐馆销售电子销售点系统，试图在那里安装 Wi-Fi 基础设施等等。那并不成功，太超前了。所以我寻找可以建立新伙伴关系并利用技术的人。就在那时，我遇到了我的朋友兼 DeepMind 联合创始人德米斯·哈萨比斯，因为他是我当时学校最好朋友的兄弟。他刚在 UCL 完成神经科学博士学位，我们走到了一起，剩下的就是历史了。早在 2010 年，你和另一位联合创始人肖恩·莱格——你们三人怀有雄心，要创造一种能够复制甚至超越人类智能的人工智能。想想看，那是 13 年前。我们其他人甚至不知道这些事情正在发生。你们在——在哪里？摄政广场附近？你们有没有想象到 2023 年世界会拥有我们现在所拥有的？在某种程度上，是的。我们很难确切想象它会如何展开，但我们押注了深度学习，这是推动这场新革命的主要工具之一，而且是在任何人涉足深度学习之前。所以 OpenAI（ChatGPT 的创造者）的现任首席科学家兼联合创始人，在 2011 年曾是我们的实习生。杰弗里·辛顿，后来成为谷歌 AI 负责人之一，现在被称为 AI 教父——最近在媒体上对后果表示担忧——他是我们的第一位顾问，付费顾问。我想他的年薪是 2.5 万英镑。所以 OpenAI 的六位联合创始人中有三位曾在某个时候经过 DeepMind，要么做演讲，要么实际上是团队成员。所以时机真的很不可思议。

Well, I've always found philosophy a systems thinking tool. It enables me to be rigorous and clear about what I think. And right from the very outset, I think when I was 19, I actually dropped out of my philosophy degree—no, I didn't know that? Yeah, I didn't finish—and I was really motivated by the impact that I could have in the world. I left to help start a charity. At the time it was a telephone counseling service called Muslim Youth Helpline, and it was a secular—I was an atheist even though I had grown up with a Muslim background—it was a secular service that was designed to provide faith and culturally sensitive support to young British Muslims. This was in 2003. And I found myself at Oxford studying this very theoretical, esoteric set of ideas, and I wanted to put real things into practice in terms of my ethics, and that was why I went to start the helpline and worked on that as a volunteer for three years. I soon got frustrated about the scale of impact in our nonprofit. I worked briefly for the mayor of London at the time, Ken Livingstone, as a human rights policy officer, and that was inspiring, but I was also struggling with the scale of impact. I realized that if I didn't capture what really makes us organized and effective as a species—the profit incentive—then I was going to miss one of the most important things to happen in my lifetime. At the time I saw the rise of Facebook, this was around 2007-2008, and it had grown in the space of two years to 100 million monthly active users. I was totally blown away at how quickly this was growing out of seemingly nowhere, something completely new to me. So I set about on a quest to find anyone and everyone that would speak to me to teach me about technology. I had started a bunch of businesses before that—two different businesses, one actually a technology company selling electronic point-of-sale systems around here in Notting Hill, in restaurants, trying to put Wi-Fi infrastructure in there, and so on. That was unsuccessful, ahead of its time. So I was looking for people who I could form a new partnership with and figure out how to take advantage of technology. That's where I met my friend and co-founder of DeepMind, Demis Hassabis, because he was the brother of my best friend at the time from school. He was just finishing his PhD in neuroscience at UCL, and we got together, and the rest is history. Back in 2010, you had between you and there was another co-founder, Shane Legg—the three of you had the ambition that you were going to create an artificial intelligence that was capable of replicating human intelligence or even succeeding it. So just think, this was 13 years ago. The rest of us didn't even know this stuff was really going on. You're in—where is it, in Regent Square somewhere? Did you imagine that by 2023 the world would have what we have now? I mean, in a way yes. It was difficult for us to imagine exactly how it would unfold, but we made a very big bet on deep learning, which is one of the primary tools that is powering this new revolution, before anybody was involved in deep learning. So the current chief scientist and co-founder of OpenAI, the creators of ChatGPT, was one of our interns back in 2011. Geoffrey Hinton, who subsequently became one of the heads of AI at Google and is known now as the Godfather of AI—recently in the press worried about the consequences—he was our first advisor, our paid advisor. I think his salary was £25,000 a year to us. So I think three of the six co-founders of OpenAI at some point passed through DeepMind, either to give talks or were actually members of the team. So it's really incredible about timing.

我们时机把握得非常好，当时遥遥领先，而且不知怎么地我们坚持了下来。你在那里待了一段时间，然后我们快进一下。剩下的你可以在书里读到。你现在共同创立并经营着 Inflection AI，正在创建一个名为 Pi 的 AI，如果你愿意可以与之互动。告诉我们 Pi 是做什么的。

We got the timing absolutely right. We were way ahead of the curve at that moment, and somehow we managed to hang on. So you were there for a while, and then let's fast forward a bit. You can read the rest of this in the book. You now have co-founded and run Inflection AI, and you are creating an AI called Pi, which you can interact with if you'd like. Tell us what Pi does.

Pi 代表个人智能，我相信在未来几年，每个人都会拥有自己的个人 AI。世界上将有成千上万的 AI。它们将代表企业、代表品牌。每个政府都会有自己的 AI，每个非营利组织、每个音乐人、艺术家、唱片公司。现在由网站或应用代表的一切，很快将由一个交互式对话智能服务来代表，该服务代表任何组织的品牌价值观和理念。我们相信，与此同时，每个人都会想要自己的个人 AI，一个站在你这边、支持你、帮助你更有条理、帮助你理解世界的 AI。它几乎会像参谋长一样运作，进行优先级排序、规划、教学和支持你。

So Pi stands for Personal Intelligence, and I believe that over the next few years, everybody is going to have their own personal AI. There are going to be hundreds of thousands of AIs in the world. They'll represent businesses, they'll represent brands. Every government will have its own AI, every nonprofit, every musician, artist, record label. Everything that is now represented by a website or an app is soon going to be represented by an interactive conversational intelligence service that represents the brand values and the ideas of whatever organization is out there. And we believe that at the same time, everybody will want their own personal AI, one that is on your side, in your corner, helping you to be more organized, helping you to make sense of the world. It really is going to function as almost like a chief of staff, prioritizing, planning, teaching, supporting you.

听起来很棒。但实际上这意味着什么？因为关于 AI 的讨论经常在这个点上转向末日论：我们会自我毁灭，因为某个坐在车库里的 rogue 会释放一种病毒杀死我们所有人。所以在谈到那些之前，假设五年后。你说过在未来 3 到 5 年内，你认为 AI 将在各种任务上达到人类水平的能力，也许不是所有，但很多。那么为我们描绘一下五年后，2028 年的生活是什么样的。首先，会是您和我在这里，还是会有某种 Mustafa AI 和机器人？

That sounds great. What does it actually mean in practice? Because so often this conversation about AI, at this point, turns into the apocalyptic: we're going to end up wiping ourselves out because there'll be some rogue person sitting in a garage somewhere who will unleash a virus that will kill us all. So before we get to all of that stuff, let's say in 5 years. You've said within the next 3 to 5 years, you think AI will reach human-level capability across a variety of tasks, perhaps not everything, but a variety. So paint a picture for us of what life will be like in five years, 2028. First of all, will it be you and me here, or will there be the kind of Mustafa AI and the bot?

让我先回到 10 年前，让你了解已经发生了什么，以及为什么我做出的预测是合理的。深度学习革命使我们能够理解原始混乱的数据。我们可以用 AI 来解释图像的内容，分类图像中是否包含狗或猫，这些像素实际意味着什么。我们可以用它来理解语音，所以当你对着手机口述时，它会转录并记录完美的文本。我们可以用它来做语言翻译。所有这些都是分类任务。我们基本上是在教模型理解原始输入数据的混乱复杂世界，足以理解该数据中的对象。那是分类革命，第一个 10 年。现在我们处于生成革命中。这些模型正在生成你从未见过的新图像、你从未见过的新文本，它们可以生成音乐片段。这是因为这是硬币的另一面。第一阶段是理解和分类。第二阶段，在足够好地完成这一点后，你可以要求 AI：既然你理解狗的样子，现在给我生成一只带有你想象中的粉色、黄色斑点等的狗。这是一种插值，是对两个、三个或四个概念之间空间的预测。这就是在所有模态中产生生成式 AI 革命的原因。随着我们对此过程应用更多的算力，我们正在堆叠更大的 AI 模型和更大的数据。这些生成式 AI 的准确性和质量变得更好。为了让你了解我们在算力方面的轨迹：在过去 10 年中，每年用于前沿 AI 模型的算力增长了 10 倍。所以连续 10 倍、10 倍、10 倍、10 倍、10 倍。这在技术史上是前所未有的。我们在其他任何地方都没有见过这样的轨迹。在未来 5 年内，我们可能会增加三到四个数量级，基本上是今天用于生产 GPT-4 或你可能与之交互的聊天模型的算力的一千倍。理解这一点非常重要，因为当人们谈论 GPT-3 或 GPT-3.5 或 GPT-4 时，这些模型之间的差距实际上是 10 倍的算力。它不是渐进的，而是指数级的。GPT-4 和 GPT-2 之间的差异实际上是 100 倍的算力。世界上最大的算力基础设施基本上学习所有原始数据的所有输入之间的所有关系。那么，在下一阶段，这使它们能够做什么？我们将从能够完美生成语音、视频、图像、语言，转向能够跨多个时间范围进行规划。目前，你只能对模型说：给我一首 X 风格的诗，给我一张匹配这两种风格的新图像。这是一次性预测。接下来，你将能够说：给我生成一个新产品。为了做到这一点，AI 需要去进行研究，查看市场，看看什么可能卖得好，人们在谈论什么。然后它需要生成一张新产品可能看起来像什么的新图像，与其他图像相比，使其与众不同且独特。然后它需要联系制造商并说：这是蓝图，这是我想让你做的。它可能会与制造商谈判以获得最佳价格，然后进行营销和销售。这些能力大约在未来 5 年内将会到来。它不能自动独立地完成每一项；该系统中不会有自主性，但当然这些单独的任务是可能的。

Let me just go back 10 years to give you a sense for what has already happened and why the predictions that I'll make are plausible. The Deep Learning Revolution enabled us to make sense of raw messy data. We could use AIs to interpret the content of images, classify whether an image contains dogs or cats, what those pixels actually mean. We can use it to understand speech, so when you dictate into your phone and it transcribes it and records perfect text. We can use it to do language translation. All of these are classification tasks. We're essentially teaching the models to understand the messy complicated world of raw input data well enough to understand the objects inside that data. That was the Classification Revolution, the first 10 years. Now we're in the Generative Revolution. These models are now producing new images that you've never seen before, new text that you've never seen before, they can generate pieces of music. And that's because it's the flip side of that coin. The first stage is understanding and classifying. The second stage, having done that well enough, you can then ask the AI to say, given that you understand what a dog looks like, now generate me a dog with your idea of pink, with your idea of yellow spots, or whatever. And that is an interpolation, a prediction of the space between two or three or four concepts. And that's what produced this Generative AI Revolution in all of the modalities. As we apply more computation to this process, we're stacking much larger AI models and much larger data. The accuracy and quality of these generative AIs gets much better. To give you a sense of the trajectory we're on with respect to computation: over the last 10 years, every single year, the amount of compute used for the cutting-edge AI models has grown by 10x. So 10x, 10x, 10x, 10x, 10 times in a row. That is unprecedented in technology history. Nowhere else have we seen a trajectory anything like that. Over the next 5 years, we'll add probably three or four orders of magnitude, basically another thousand times the compute used today to produce GPT-4 or the chat model you might interact with. It's really important to understand that because when people talk about GPT-3 or GPT-3.5 or GPT-4, the distance between those models is in fact 10x compute. It's not incremental, it's exponential. The difference between GPT-4 and GPT-2 is in fact 100 times worth of compute. The largest compute infrastructures in the world basically learn all the relationships between all the inputs of all this raw data. So what does that enable them to do in the next phase? We'll go from being able to perfectly generate speech, video, image, language, to now being able to plan across multiple time horizons. At the moment, you can only say to a model: give me a poem in the style of X, give me a new image that matches these two styles. It's a one-shot prediction. Next, you'll be able to say: generate me a new product. In order to do that, the AI would need to go off and do research, look at the market, see what was potentially going to sell, what people are talking about. It would then need to generate a new image of what that product might look like compared to other images so that it was different and unique. It would then need to go and contact a manufacturer and say: here's the blueprint, this is what I want you to make. It might negotiate with that manufacturer to get the best possible price, and then go and market it and sell it. Those are the capabilities that are going to arrive approximately in the next 5 years. It won't be able to do each of those automatically independently; there will be no autonomy in that system, but certainly those individual tasks are likely to.

所以这意味着创新过程变得更加高效，管理过程也变得更加高效。这意味着什么？我们暂时先关注积极的一面。我保证我们会谈到所有负面影响，有很多。但这将使我们能够做什么？我的意思是，人们谈论 AI 将帮助我们解决气候变化，AI 将带来医疗保健的巨大改进。请给我们讲讲其中一些可能是什么，这样我们就能看到积极的一面。

So that means that presumably the process of innovation becomes much more efficient, the process of managing things becomes much more efficient. What does that mean? And let's stick with the upside for the moment. I promise you we'll get to all the downsides, of which there are many. But what is that going to enable us to do? I mean, people talk about AI will help us solve climate change, AI will lead to tremendous improvements in healthcare. Just talk us through what some of those things might be so we can see the upside.

智能一直是创造的引擎。你在这里看到的一切都是我们与环境互动的产物，以制造更高效、更便宜的桌子，例如，或新的 iPad。如果你回顾历史，今天我们……

Intelligence has been the engine of creation. Everything that you see around you here is the product of us interacting with some environment to make a more efficient, a cheaper table, for example, or a new iPad. If you look back at history, today we're...

我们现在生产一公斤谷物所需的劳动力，只有 100 年前的 2%。技术的轨迹意味着东西越来越便宜、越来越容易制造，带来巨大的生产力提升。农业进步背后的智能，正是我们现在用 AI 发明的同一种工具。例如，我们可以培育抗旱、抗虫、更具韧性的作物，应对气候变化。我们已经看到 AI 优化工业系统，比如让大型冷却基础设施更高效。在医疗、教育、交通等各个领域，未来二三十年我们将看到巨大的效率提升。AI 就像插值：它想象从未见过的东西，发现新知识，发明新科学。如果我们做对了，我们将走向一个极度充裕的时代。想象每个人都能拥有最好的科学顾问、研究助理、教练、知己——这些角色如今只属于富人和受过教育的人。这种智能将广泛普及，就像今天智能手机和笔记本电脑对数十亿人开放一样。

We're able to produce a kilo of grain with just 2% of the labor required 100 years ago. The trajectory of technology means things get cheaper and easier, leading to huge productivity gains. The intelligence behind agricultural improvements is the same as what we're now inventing with AI. For example, we can produce drought-resistant, pest-resistant crops and tackle climate change. We've seen AI optimize industrial systems, making cooling infrastructure more efficient. In healthcare, education, transportation, we'll see massive efficiencies over the next two to three decades. AI is like interpolation: it imagines something it's never seen before, discovering new knowledge and inventing new science. If we get this right, we're headed towards an era of radical abundance. Imagine everyone having the best scientific advisor, research assistant, coach, confidant—roles that are now exclusive to the wealthy and educated. That intelligence will be widely available, just like smartphones and laptops are accessible to billions today.

你没有把书叫做《即将到来的天堂》，而是叫《即将到来的浪潮》。我听说原定书名是《遏制是不可能的》。解释一下你的论点：这不是即将到来的天堂，而是更微妙的东西。告诉我们负面影响是什么，以及书中对遏制的关注。

You didn't call your book 'The Coming Nirvana', you called it 'The Coming Wave'. I'm told the original title was 'Containment Is Not Possible'. Explain the argument: it's not nirvana around the corner; it's more subtle. Tell us the downsides and the focus on containment.

我对风险持清醒和诚实的态度。更强大的模型会变得更小、更便宜、更容易使用——这是每一项技术的历史。扩散带来了巨大的好处，但另一面是，这些强大的工具可能助长不良行为者破坏我们的世界。每个人都有议程，他们会更容易宣扬自己的主张。极端情况下，模型会指导如何制造生物和化学武器。这是我们在大型语言模型中观察到的能力。在遵守法律的大公司模型中，这相对容易控制。但这些模型在开源中广泛可用；你可以免费下载代码运行较小版本的 GPT。十年后，更强大的模型会更小、更易转移，使人们更容易造成伤害。领先的公司现在拥有最大的模型，但开源模型也紧随其后。问题是：你能防止开源模型被车库里的愤怒青少年控制吗？更黑暗的一面是，这些本质上是想法——知识产权，写在三页纸上。实现今天需要大量算力，但如果这个约束消失，十年后你可以在手机上运行它们，遏制就成了挑战。还有集中化的风险：权力将赋予构建这些模型的人，比如我的公司、谷歌和其他大型科技提供商。仅通过解决开源问题并不能消除风险；我们还需要弄清楚超级强大的科技公司与民族国家之间的关系。

I'm wide-eyed and honest about the risks. More powerful models will get smaller, cheaper, and easier to use—that's the history of every technology. Proliferation has delivered immense benefits, but the flip side is that these powerful tools could empower bad actors to destabilize our world. Everyone has an agenda, and they'll have an easier time advocating for it. At the extreme, models provide coaching on manufacturing biological and chemical weapons. That's a capability we've observed in large language models. It's relatively easy to control in models from big companies that abide by the law. But these models are widely available in open source; you can download code to run smaller versions of GPT for free. Over 10 years, much more powerful models will be smaller and transferable, making it easier for people to cause harm. The leading companies have the biggest models now, but open-source ones are not far behind. The question is: can you prevent open-source models from being controlled by an angry teenager in their garage? The darker side is that these are fundamentally ideas—intellectual property expressed on three sheets of paper. Implementation requires vast compute today, but if that constraint is removed and you can run them on a phone in a decade, containment becomes a challenge. There are also risks of centralization: power will be conferred on those building these models, like my company, Google, and other big tech providers. We don't eliminate risk by addressing open source alone; we also need to figure out the relationship between super powerful tech companies and the nation state.

我们来谈谈最常被提及的风险或负面后果。你经常听到的一个说法是：当 AI 在广泛任务上达到或超越人类智能时，我们就都没工作了。为什么要雇人，如果你能用 AI？历史表明这是胡说。我们从未缺过工作。作为一个资深经济学家，我认为这是劳动总量谬误。但很多人这么说。工作会怎样？你怎么看？

Accountable. So let's go through some of the most frequently cited risks or indeed negative consequences. And the one that you hear a lot is: as AIs become equivalent to or exceed human intelligence across a wide range of tasks, there won't be any jobs for any of us. Why would you employ a human if you could have an AI? So history suggests that's bunkum. We've never yet run out of jobs. And being a good paid-up economist, I think it's a lump of labor fallacy. But lots and lots of people say this. What's going to happen to the jobs? Where are you on that?

我们先描述一下劳动总量谬误，因为我认为理解这一点很重要，因为这是迄今为止的历史趋势。它基本上意味着：当我们自动化事物并提高效率时，我们为人们创造了更多时间去发明新事物，并创造了更多的健康和财富。这本身创造了更多需求，然后我们最终创造出新的商品和服务来满足这些需求。所以我们会不断创造新的工作和角色。你可以看到在过去几十年里，有许多角色在 30 年前甚至无法想象，从应用设计师到如今的大语言模型提示工程师。所以这是一个可能的轨迹。我认为关于工作的问题取决于你的时间跨度。在未来二十年，我认为极不可能出现结构性失业，即人们想为市场贡献劳动力却无法竞争。我认为这不太可能。今天的统计数据中肯定没有证据。除此之外，我确实认为有可能许多人即使借助 AI 也无法生产出市场所需的足够价值的东西，即他们和他们的 AI 联合在系统中。我的意思是，AI 越来越比人类准确、更可靠、可以全天候工作、更稳定。所以我认为这绝对是一个风险，我们应该正视它，并对自己诚实，这实际上可能是一个有趣且重要的目的地。我的意思是，工作不是社会的目标。有时我认为我们忘记了，实际上社会、生活和文明是关于福祉、和平与繁荣的。它是关于创造更高效的方式来保持我们的生产力和健康。许多人——可能在这个房间里，包括我们——享受我们的工作，我们热爱我们的工作，我们足够幸运和特权，有机会做我们想做的事。我认为非常重要的是要记住，很多人没有这种奢侈，很多人做着如果不必工作就不会做的工作。所以对我来说，社会的目标是追求极端丰裕：我们如何用更少的资源创造更多，并将人们从工作的义务中解放出来？这意味着我们必须解决再分配的问题。显然，这是一个极其困难的问题，我在书中提到了它。但这是我们必须关注的事情：在这个新体制下，税收是什么样的？我们如何捕获创造的价值，并确保它实际上转化为美元，而不仅仅是对 GDP 的增值？

Well, let's just describe the lump of labor fallacy because I think it's important to sit with that, because that is the historical trend so far. What it basically means is that when we automate things and make things more efficient, we create more time for people to invent new things, and we create more health and wealth. That in itself creates more demand, and then we end up creating new goods and services to satisfy that demand. So we'll continually just keep creating new jobs and roles. You can see that in the last couple of decades: there are many roles that couldn't even have been conceived of 30 years ago, from app designer all the way through to the present-day prompt engineer of a large language model. So that's one trajectory that is likely. I think the question about what happens with jobs depends on your time horizon. Over the next two decades, I think it's highly unlikely that we will see structural disemployment where people want to contribute their labor to the market and they just can't compete. I think that's pretty unlikely. There's certainly no evidence of it in the statistics today. Beyond that, I do think it's possible that many people won't be able to—even with an AI—produce things that are of sufficient value that the market wants them and their AI jointly in the system. I mean, AIs are increasingly more accurate than humans, they are more reliable, they can work 24/7, they're more stable. So I think that's definitely a risk, and I think we should lean into that and be honest with ourselves that this is actually maybe an interesting and important destination. I mean, work isn't the goal of society. Sometimes I think we've just forgotten that actually society, life, and civilization is about well-being and peace and prosperity. It's about creating more efficient ways to keep us productive and healthy. Many people—probably in this room and including us—enjoy our work, we love our work, and we're lucky enough and privileged enough to have the opportunity to do exactly the work that we want. I think it's super important to remember that many, many people don't have that luxury, and many people do jobs that they would never do if they didn't have to work. So to me, the goal of society is a quest for radical abundance: how can we create more with radically less and liberate people from the obligation to work? That means we have to figure out the question of redistribution. Obviously, that is an incredibly hard one, and I address it in the book. But that's the thing we have to focus on: what does taxation look like in this new regime? How do we capture the value that is created and make sure it's actually converted into dollars rather than just a sort of value add to GDP?

我们马上要谈到再分配和政府角色。但首先，提醒一下——我一开始就该说——穆斯塔法和我会再聊大概 15-20 分钟，然后开放提问。对于观看直播的朋友，现在就可以开始提问了。因为我这里的这个小 AI 告诉我电话和通知会被静音，这没什么用。好了，我现在看到了问题。所以请开始写问题，我们大约 15 分钟后回答。但好吧，政府角色。在这个世界里，你需要更激进的再分配。但一个担忧是，AI 及其崛起实际上使民主运作更加困难。我们已经看到很多关于深度伪造破坏 2024 年选举的担忧。40 亿人生活在明年将举行选举的国家。人们在担心 2024 年，更不用说 2028 或 2034 年了。而且我们——穆斯塔法和我刚刚与尤瓦尔·赫拉利聊过，他和你一样深思熟虑地乐观，但基本上说这是民主的终结。我不确定你我都同意。但在未来几十年，在这个 AI 世界里，对自由民主的后果是什么？

We're going to get on to redistribution and the role of government in just a second. But first, to remind you—and I should have said this at the beginning—Mustafa and I are going to talk for perhaps another 15-20 minutes, but then we're going to open it up to questions. And for those of you who are watching on the live stream, feel free to start asking them now. Because if this little AI that I have here is telling me that calls and notifications will be silenced, that's not very helpful. Yeah, now I've got an answer. I do see the questions there. So please start writing in the questions, and we will get to them in about 15 minutes. But okay, role of government. You need to have, in this world, more radical redistribution. But one of the concerns is that AI and the rise of AI makes actually the functioning of democracy ever harder. We're already seeing lots of concerns about deepfakes wrecking the 2024 elections. 4 billion people live in countries that will have elections next year. People are worrying about 2024, never mind 2028 or 2034. And we just—Mustafa and I just had a conversation with Yuval Harari, who is as pessimistic as you are thoughtfully optimistic—who basically said it was the end of democracy. I'm not sure that either you and I agreed. But what is the consequence for liberal democracy in the coming decades in this world of AI?

首先，我认为我们现在的状况相当严峻。我的意思是，对政府、政客和政治进程的信任度处于历史最低点。事实上，皮尤研究在美国的一项调查中，35%的受访者认为军队统治会是好事。所以我们已经处于非常脆弱和焦虑的状态。我认为，稍微站在尤瓦尔的角度想，论点会是这些新技术使我们能够生产出具有说服力和操纵性的新型合成媒体，这些媒体高度个性化，并加剧了潜在的恐惧。对吧？所以我认为这是一个真正的风险。我们必须接受，制造假新闻将变得更容易、更便宜。我们对不真实的东西有着无法满足的、上瘾的、多巴胺刺激的胃口。它卖得更快，传播得更快。这是一个我们必须解决的根本问题。我不确定这是 AI 带来的新风险；这是 AI 和其他技术加速的事情。这就是 AI 的挑战。这是一个理解 AI 总体影响的好视角：它将放大我们最好的一面，也将放大我们最坏的一面。

Look, I think the first thing to say is that the state we're in is pretty bleak. I mean, trust in governments and in politicians and the political process is as low as it has ever been. In fact, 35% of people interviewed in a Pew study in the US think that army rule would be a good thing. So we're already in a very fragile and anxious state. And I think that, to sort of empathize with Yuval for a moment, the argument would be that these new technologies allow us to produce new forms of synthetic media that are persuasive and manipulative, that are highly personalized, and they exacerbate underlying fears. Right? So I think that is a real risk. We have to accept that it's going to be much easier and cheaper to produce fake news. We have an insatiable, addictive, dopamine-hitting appetite for untruth. It sells quicker, it spreads faster. And that's a foundational question that we have to address. I'm not sure that it's a new risk that AI imposes; it's something that AI and other technologies accelerate. And that's the challenge of AI. That is a good lens for understanding the impact that AI has in general: it is going to amplify the very best of us, and it's also going to amplify the very worst of us.

而且，这一切发生在一个地缘政治上分裂的世界，至少在后冷战时代的过去几十年里从未如此。所以我们有中美之间的紧张关系。这两个政权之间基本上是一场全球主导地位的竞赛。在这样一个世界里，你如何实现你在书中写到的那些治理结构，这些结构是试图防止 AI 最极端负面后果所必需的？

And what about the fact that this is developing in a world which geopolitically is split in a way that it hasn't been, at least in the last couple of decades in the post-Cold War world at all? So we have the tensions between the US and China. We have essentially a race for global dominance between these two regimes. In that kind of a world, how can you achieve the sort of governance structures that you write about in your book that are needed to try and perhaps prevent the most extreme downsides of AI?

是的，尽管我被指责为乐观主义者，我也被指责为对我们必须采取的干预措施持乌托邦态度。我认为不幸的是，这只是陈述事实：需要的是良好运作的治理和监督。我的意思是，公司是开放并愿意接受审计和监督的。

Yeah, I mean, much as I've been accused of being an optimist about it, I've also been accused of being a utopian about the interventions that we have to make. And I think that unfortunately that's just a statement of fact: what's required is good functioning governance and oversight. I mean, the companies are open and willing to expose themselves to audit and to oversight.

我认为，与以往几代科技 CEO、发明家和创造者相比，这是一个独特的时刻。我们非常明确地表示，可能需要预防原则，这是一个我们必须放慢脚步、更加谨慎的时刻，也许在摘取果实之前，先让一些好处留在树上，以避免伤害。我认为这是一个相当新颖的设定，但它需要良好的治理、运作良好的民主制度和有效的监督。我认为我们在欧洲确实拥有这些。我认为欧盟的 AI 法案，已经起草了三年半，非常全面、稳健且相当合理。所以总的来说，我一直是它的支持者，并某种程度上认可它。

And I think that is a unique moment relative to past generations of tech CEOs and inventors and creators across the board. We're being very clear that the precautionary principle is probably needed, and that's a moment when we have to go a little bit slower, be a little bit more careful, and maybe leave some of the benefits on the tree for a moment before we pick that fruit in order to avoid harms. I think that's a pretty novel setup as it is, but it requires really good governance, it requires functioning democracies, it requires good oversight. I think that we do actually have that in Europe. I think that the EU AI Act, which has been in draft now for three and a half years, is super thorough and very robust and pretty sensible. And so in general, I've been a fan of it and kind of endorsing it.

但人们常说，如果我们在英国做对了，或者在欧洲和美国做对了，那中国怎么办？我的意思是，我一次又一次听到这个问题：中国怎么办？我认为这是一个非常危险的推理思路。首先，它有点妖魔化中国，好像中国有一个疯狂的、自杀式的使命，要不惜一切代价、不惜任何代价接管世界，成为下一个主导全球的强国。我的意思是，到目前为止，我没有看到任何证据。我不排除这种可能性，我也不是同情者，但我认为我们应该清醒地看待他们目前实际采取的行动。他们有自我保护的直觉，就像我们一样，我们越能迎合他们让公民从经济相互依存、和平、繁荣和福祉中受益的愿望，我们在这些激励因素上就越一致。第二，指责中国是危险的，因为实际上我们不能在价值观上竞相逐底。我们必须决定我们支持什么，对吧？如果我们不——我的意思是，我相信我们不应该拥有由 AI 驱动的大规模国家监控设备。我们不应该仅仅因为中国在做就那样做。我们不应该仅仅因为他们冒险就卷入军备竞赛并冒险。这对一些人来说很难接受，因为他们可能过于务实，我认为这只会导致不可避免的自我实现预言，即我们最终都承担了不必要的可怕风险。

But people often say, well, if we get it right in the UK or if we get it right in Europe and the US, what about China? I mean, I hear this question over and over again: what about China? And I think that's a really dangerous line of reasoning. First, it sort of demonizes China, as though China has this sort of maniacal suicidal mission to at all costs, at any cost, sort of take over the world and be the next dominant global power. I mean, so far I don't see any evidence of that. I'm not ruling it out, I'm not a sympathizer, but I think we should just be wide-eyed about the actions they're actually taking at the moment. They have a self-preservation instinct just as we do, and the more that we can appeal to that desire to have their citizens benefit from economic interdependence and from peace and prosperity and well-being, we're both aligned in those incentives. I think the second thing is it's dangerous to sort of point the finger at China because actually we can't just have a race to the bottom on values. We have to decide what we stand behind, right? If we're not — I mean, I'm a believer that we shouldn't have a large-scale state surveillance apparatus enabled by AI. We shouldn't do that just because China is doing it. We shouldn't get into an arms race and take risks just because they're taking those risks. And that's difficult for some people to accept because they might be hyper pragmatic, and I think that only leads to an inevitable self-fulfilling prophecy that we both end up taking terrible risks which are unnecessary.

那么政府应该做什么——或者具体来说，这个政府应该做什么？我们在英国，大概这里大多数人来自伦敦。英国政府想成为 AI 的超级大国，一个 AI 超级大国，并在 11 月举办一个关于 AI 安全的会议。这里有很大的焦点。这个政府，或者其他政府，具体应该做什么来最小化风险？有什么东西现在应该被禁止吗？是否有应该制定的规则？

So what should government — or let's be concrete, what should this government? We're in the UK, and presumably most people here are from London. The British government wants to be the superpower of AI, an AI superpower, and is having an AI conference on AI safety in November. There's a big focus here. What should this government, or indeed other governments, be doing concretely to minimize the risks? What should be — is there stuff that should be banned now? Are there rules of the road that should be put in place?

所以第一件事是政府必须自己构建技术。我们已经养成了外包和委托第三方创造技术的习惯，我认为要控制你不理解的东西非常困难，除非你构建它，否则你不会深刻理解它。所以我认为这是第一件事，这本身非常有争议。当我在政府中提出这一点时，人们几乎要举手投降，缺乏意愿、缺乏自信、缺乏政府可以成为创造者、制造者的信念，尤其是在技术方面。要做到这一点，我认为第二件事是我们必须有深厚的技术和工程人员，以及更广泛的技术专家，进入内阁职位和每个政府部门的领导层。对我来说，我们没有首席技术官在内阁中，没有运行我们的大型机构，这很疯狂。所有这些都是外包的。要做到这一点的挑战是，你必须支付接近私营部门的薪水——又是一个没人想谈的高度敏感话题：永远不应该比首相挣得多。对我来说，这毫无意义。我们怎么能有一个开放的劳动力市场，一方面我们对人们说，去为你喜欢的人工作，另一方面人们得到 10 倍的薪水，而另一方面我们说，好吧，以公共服务为名做出巨大牺牲？实际结果是，如果这种情况持续几十年，净效果就是你在这里有一种质量，在那里有另一种质量，而这正是我们面临的。我们必须面对这个现实。让人们接受我们应该支付超高薪水非常困难；这会产生其他问题，比如我们如何让这些人负责，考虑到他们可能从公共资金中赚取多少等等。但基本上，这两件事促成了第三件事，即政府必须在监管上冒险。有一种恐惧，认为政府行动过于激进或过于实验性，会惹恼大公司。作为一个经常处于接收端的人，过去也犯过错误，我仍然认为正确的做法是给政府一些空间，让他们犯错，让他们进行不成功的投资，赞扬实验性的政府结构，相信政治进程，参与其中，鼓励它，否则只会陷入衰退的螺旋，缺乏信心，认为我们实际上不能做正确的事，我们应该做正确的事，然后最终导致自我实现的预言，就像中国的情况一样。

So the first thing is that governments have to build technology. We've got into this habit of outsourcing and commissioning third parties to create technology, and I think it's really difficult to be able to control what you don't understand, and unless you build it, you don't deeply understand it. So I think that's just the first thing, which in itself is very controversial. When I propose that in government, people sort of throw up their hands, and there's a lack of will, a lack of self-confidence, a lack of belief that government can be a creator, a maker, especially on the technology front. To do that, I think the second thing is that we have to have deeply technical and engineering people, as well as technologists more generally, in cabinet positions and at the heads of every government department. It's pretty crazy to me that we don't have a CTO, a chief technology officer, in cabinet, running our big institutions. All of that is outsourced. The challenge to be able to do that is you just have to pay close to private sector salaries — again, another highly sensitive topic that no one wants to talk about: should never earn more than the Prime Minister. To me, this makes no sense. How can we have an open labor market where on the one hand we're saying to people, go work for whoever you like, and on the one hand people are being paid 10x, and on the other we're saying, well, take this huge sacrifice in the name of public service? The practical reality is that if that happens over many decades, the net effect is that you have quality of one type over here and another type over there, and that's really what we're facing. We have to confront that reality. It's very difficult for people to accept that we should be paying super large salaries; it creates other issues around how we hold those kinds of people accountable given how much of the public purse they might be earning, etc. But fundamentally, those two things enable a third thing, which is governments have to take risks with regulation. There is a fear that governments act too aggressively or too experimentally and upset the big companies. And as someone who's on the receiving end of this quite a lot and have been in the past where mistakes have been made, I still think the right thing to do is to give governments a break, let them make mistakes, let them make investments that don't work, praise the experimental government structures, have faith in the political process, participate, encourage it, because otherwise there's just this spiral of decline, this sort of lack of confidence that we can actually do the right thing, that we should do the right thing, and then that ultimately leads to the self-fulfilling prophecy, much like with China.

你认为你的观点在你的行业中是例外吗？我的意思是，刻板印象是一群 30 岁的科技兄弟，他们认为政府没用，要用 AI 改变世界，我们要这样做。这是一个准确的刻板印象吗？你是例外吗？我的意思是，我们应该担心你行业中人的色调吗？

And do you think that your view is the exception in your industry? I mean, the stereotype is a bunch of 30-year-old tech bros who think the government is useless and who are going to kind of change the world with AI, and we're going to do this. Is that an accurate stereotype? Are you the exception? I mean, should we worry about the hues of people in your industry?

我认为我们到处都有两极分化，所以刻板印象可能是真的，但反过来说，我们可以没有技术就做到，我认为这完全错误。技术是过程中绝对必要但不充分的部分。我认为硅谷的一些人确实有更倾向于技术自由主义的倾向，这是毫无疑问的。政府是问题所在，目标是消灭国家并完全独立运行。老实说，有一些非常非常有影响力、非常有权势的人有这个目标，他们正在用他们的公司和财富朝着这个目标建设。我对他们非常怀疑。

I think we have polarization everywhere, so the stereotype is probably true, but the counter is that we can do it without technology, and I think that's totally wrong. Like, technology is an absolutely necessary but not sufficient part of the process. And I think that some people in Silicon Valley do have a tendency to be much more techno-libertarian, there's no question about that. The government is the problem, the objective is to eradicate the state and run it completely independently. And I'll be honest, there are some very, very influential, very powerful people who have that objective, are building towards that objective with both their companies and their fortunes. And I'm very skeptical of them.

我知道我显然站在另一边，这塑造了公众对此的很多恐惧，即一群超级有权势的人在塑造这一切，而且对国家和民主进程不屑一顾。我有两个简短的问题，我知道否则会有人问，然后我们进入观众提问。第一个是关于奇点的问题。我们谈论 AI 就不能不谈奇点。它会实现吗？什么时候实现？

I know obviously I'm on the other side of that and that's what shapes a lot of the public fear about this, that you have a bunch of hyper powerful people who are shaping this with disdain for the state and the democratic process. Two quick questions for me, which I know someone would ask otherwise, and then we're going to audience questions. The first one is the whole question of the singularity. We can't have a conversation about AI without the singularity. Will it happen? When will it happen?

我真心认为这是一个非常无益的框架，人们跳到这个框架是因为很容易指向《终结者》和天网，但这几乎就像在发明晶体管之前就跳到登月。我是说这要几百年后。这真的很无益。有很多实际的近期操作能力你可以预测，就像我试图描述的那样，然后你可以用这些来思考对国家的影响、如何改变我们的企业、对我们的政府意味着什么。所以总的来说，我不做那些预测。我非常怀疑超级智能这个框架对我们有用。

I honestly think it's a very unhelpful framing of what's to come, and people jump to this framing because it's easy to point to Terminator and Skynet, but it's almost like leaping to the Moon before we've even invented the transistor. I mean it's hundreds of years away. It's really unhelpful. There are many practical near-term operational capabilities that you can predict, just as I've tried to describe, and you can then use those to wrestle with what are the consequences for the nation state, how does this change our businesses, what does this mean for our governments. So in general I don't make those predictions. I'm very skeptical that the superintelligence framing is useful to us.

那另一个呢，那些业余 AI 爱好者总是谈论的，即存在性灾难的概率？我们因此自我毁灭的概率有多大？

What about the other one that backyard wannabe AI comms always talk about, which is the odds of existential catastrophe? What are the odds that we will wipe ourselves out with this?

同样，我认为非常非常低。我真的认为小到可以忽略不计，以至于不值得投入……

Again, I mean I think very very low. I really think it's infinitesimally small, such that it's not worth putting in the...

我问你这个的原因是我问了一个和你类似的人这个问题。他们说非常低。我问多低？大约 5%。所以你认为小到可以忽略不计，零？

The reason I asked you that is because I asked one of your... someone somewhat similar to you what this was. Oh, very low, they said. And I said what's very low? Oh, about 5%. So you think it's infinitesimally small, zero?

是的，好吧。那是个不错的结束点。好了，我们现在开始接受你们的提问和在线观众的提问。

Yes, okay. Well, that's a good place to end on. All right, we're going to open now to your questions and questions from the online audience.

哦，这是 Kitty Hadock 提出的一个好问题：所有这些算力对我们的碳排放会有什么影响，还是 AI 能够提高生产力从而在其他地方减少碳排放？

Oh, this is a good question from Kitty Hadock who asks: what will be the impact of all that computer power on our carbon emissions, or will AI be able to enhance productivity so we reduce carbon elsewhere?

是的，另一个热门观点：非常低，真的微不足道。我们在数据中心上消耗的碳相对而言确实微乎其微。其次，大部分发生在完全可再生的数据中心。谷歌和微软都是 100%使用可再生能源。谷歌实际上拥有世界上最大的风电场，最大的风电场群。我在 DeepMind 时参与的一个项目就是让整个风电场群的效率提高 20%。所以从一开始他们就专注于这一点。我不是说没有其他环境后果，比如芯片制造中使用的镓和钴等，但我真心认为，相对于我们看到的收益以及每单位计算的绝对碳成本，这非常非常小。

Yeah, another hot take on this: very low and really inconsequential. The amount of carbon that we spend on our data centers is genuinely minuscule relatively speaking. Secondly, most of that happens in completely renewable data centers. Google and Microsoft are both entirely 100% renewable. Google actually owns the largest wind farm, the largest set of wind farms in the world. One of the projects that I worked on whilst I was at DeepMind was making the entire wind farm fleet 20% more efficient. So right from the outset they have been focused on this. I'm not saying there aren't other environmental consequences like the use of gallium and cobalt in the actual chip manufacturing and so on, but I honestly think that relative to the benefits that we're seeing and with respect to the absolute cost of carbon per unit of computation, it's very very small.

接着这个问题，因为我经常听到一个论点，即电力和能源获取的成本将制约这些 AI 的发展和普及。你也认为这不成立吗？

And just to follow up to that, because an argument I have often heard is that the cost of electricity and the access to power will be a constraint on the development of these AIs and their proliferation. Do you also think that's not true?

不，我认为这不成立。我是说，我认为这不成立。我认为一些数据中心将达到 100 兆瓦规模，这可能是小城市电力消耗的个位数百分比，但我们谈论的是非常少的 100 兆瓦规模。我是说那确实巨大。今天还没有这样的东西。所以不用担心 AI 本身的碳后果。

No, I think that's not true. I mean, I think that's not true. I think that some data centers will be at the 100 megawatt scale, which is maybe a single digit percentage of a small city's electricity consumption, but we're talking about a very small number at the 100 megawatt scale. I mean that really is enormous. Nothing like that exists today. So don't worry about the carbon consequences of the actual AIs.

来自观众提问，是的，第二排的女士。我不太确定麦克风怎么用。你有麦克风吗？是这样用吗？正在递过来。第二排的女士，那里。谢谢。

From the audience questions here, yes, lady here in the second row. I'm not quite sure what the mic does. Do you get a microphone? Does it work that way? It's on its way down. Lady in the second row, there. Thank you.

你好，我是来自 Number of Education 的 Sheru。你好。使用 AI 的教育公司。我的问题是，如果你考虑两个行业，比如医疗和教育，以及 AI 的应用，你能在两者中选择一个你抱有最大希望的吗？他们应该如何思考？他们应该考虑采购吗？如何安全地采购？或者如你所说，他们可以自己生产，但有些组织可能无法很快生产。那么如果你是采购方，如何做好？应该使用哪些框架？

Hi, Sheru from Number of Education. Hello. Education companies that use AI. My question to you is, if you think about two industries, say healthcare and education, and you think about the applications that AI has, could you choose between the two which you would hold the most hope for? And how should they be thinking about it? Should they be thinking about procuring it? And how do you safely procure it? Well, or as you said, you could produce it, but some of those organizations may not be in a position to produce it anytime soon. So if you're a procurer, how do you do that well? And what are some of the frameworks that should be used for that?

是的，谢谢。这是个好问题。关于……在近期直接影响方面，我最兴奋的可能是教育。这些模型已经在使用了。我认为 ChatGPT 的主要用例实际上是作业帮助。人们常常认为，哦，我的孩子在复制粘贴，但实际上如果你观察他们使用这些模型的方式，很多人使用我们的模型正是因为这个原因：这是一种对话式互动，就像一位热情的老师和孩子的兴趣对话。所以孩子或学习者可以用自己的风格提问，选择他们感兴趣的东西，问一些奇怪、模糊、表达不清、不完整的问题。而 AI 无限耐心，提供非常详细、基本准确的信息。它并不总是完美的，但会变得完美。我认为这对每个人来说都是一个难以置信的精英主义收益。我是说，我们需要想象一个五年后的世界，在那里，世界上最好的教育，完全个性化、完全准确，对地球上任何想要的人几乎免费提供。这听起来很棒。我们如何从现在走向那个世界？我认为这些模型的美妙之处在于它们有内在的扩散和变小趋势。这是扩散的好处：它们传播是因为每个人都想访问，每个人都想集成它们。现在有这么多竞争模型。按词购买模型的成本，比如你在开发一个应用，你会去找三四个大模型创建者之一，按词付费。这个成本自一月份以来下降了 70 倍，因为我们都在相互竞争。这意味着你现在可以拿一个你可能已经开发了几年的普通应用，添加一个对话组件。事实上，我们在《经济学人》正在做这个，Ecobot，秘密项目进行中。显然不再那么秘密了。谢谢，抱歉。你将对话元素集成到现有工作流中。所以你应该能够以你的品牌风格和主题就特定内容提出任何问题……

Yeah, thank you. That's a great question. I mean, on the... I'm probably most excited in terms of the immediate near-term impact about education. I mean, these models are already being used. I think the primary use case of ChatGPT is in fact homework help. And people often think, oh my kids are copying and pasting, but actually if you watch the way they're using these models, and many people use our models up high for exactly this reason, it's a conversational interaction much like an enthusiastic teacher might speak to a child about the interest that they have. So the child or the learner in general gets to phrase the question in exactly their style, picking on exactly the thing that they're interested in, asking the odd obscure poorly phrased, not complete picture type question. And of course the AI is infinitely patient, provides really detailed mostly factual information. I mean it's not always perfect, but it will be perfect. And I think that's an unbelievable meritocratic gain for everybody. I mean, I think we need to picture a world in 5 years' time where the best education in the world, completely personalized, entirely factually accurate, is available to absolutely everybody who wants it on the planet, pretty much for free. Which sounds amazing. How do you go from where we are now to that world? I think the beauty of these models is that they have an inherent tendency to proliferate and get smaller. I mean, this is the upside of proliferation: they spread because everybody wants access, everybody wants to integrate them. There are so many competing models now. The cost of buying a model per word, so if you're building an app for example, you'll go to one of the three or four big model creators and you pay per word. That cost has come down 70x since January because we're all competing with each other. So that means that you can now take a regular app that you might have been developing for years in its current instantiation and add a conversational widget. In fact, we're doing this at The Economist with the Ecobot, secret project underway. Clearly not so secret anymore. Thank you, sorry. And you integrate the conversational element into your existing workflow. So you should be able to ask any question in the style and the theme of your brand about the specific content that...

它会像一个即插即用的小部件，可以放在应用的任何地方。这就是我所说的普及。显然，每个人都会觉得它很有用，你可以在低代码或无代码环境中使用这个工具。你可以看到图像生成模型如今是如何集成到 Adobe 中的。如果你已经是 Adobe 的用户，你正在以拖放的方式使用最前沿的 AI 模型，无需任何训练。如果你今天在搭建一个新网站，也是拖放式的。你只需抓取一个小部件，把它放到这里，突然你就有了一个带视频的 YouTube 播放器，突然你就有了一个基于你所有数据的语言模型对话交互。所以我认为理解这一点很重要：这将会广泛地提供给每个人。不会存在访问问题。风险和危害来自于如何减轻那些可能将其用于邪恶目的的不良行为者的负面影响，但好处是巨大的。

It will be like a plug-and-play widget that you can put anywhere in the app. That's what I mean by proliferation. Obviously everybody finds that useful, and you'll be able to use that tool in a low-code or no-code environment. You see how image generation models are being integrated into Adobe today. If you're already a user of Adobe, you're using the absolute cutting-edge AI models in a drag-and-drop way, no training required. If you're building a new website today, it's drag and drop. You just grab a little widget and plop it over here, and suddenly you have a YouTube player with your video, and suddenly you have a conversational interaction with a language model that is conditioned over all your data. So I think it's important to wrap your head around the idea that this is going to be widely available to everybody. There isn't going to be an access issue. The risk and harm comes from mitigating the downsides of the bad actors who might use it for nefarious purposes, but the upsides are incredible.

我们来看一个在线提问。Renato Della Jr 问：当你说“我们会解决这个和那个”时，这个“我们”是谁？是人类、企业、联合国，还是埃隆·马斯克？

Let's get a question from online. Renato Della Jr asks: when you say 'we will solve this and that,' who is this 'we'? Humanity, corporations, the UN, or Elon Musk?

我当然希望不是埃隆·马斯克。我认为这是研究者、发明家和创造者的共同体。存在这样一种对话——有时你在 Twitter 上看到片段，有时在学者发表的研究论文中，有时在大公司的博客和产品中。这是一个不断展开、演化的生态系统，相互借鉴、创造和进化。所以当我说“我们”时，当然不是指我在 Inflection 的团队，我现在的公司。我只是指人类这个生态系统。我们正集体朝着发明和创造的方向发展。

I definitely hope it's not Elon Musk. I think of it as the community of researchers, inventors, and creators. There's this dialogue—sometimes you see snippets on Twitter, sometimes in research papers academics publish, sometimes in blogs and products from big companies. There's this unfolding, evolving ecosystem that references each other, creating and evolving. So when I say 'we,' I certainly don't mean me at Inflection, my current company. I just mean the ecosystem of humanity. We're trending collectively in a direction of invention and creation.

这个生态系统包括中国科学家吗？

Does that ecosystem include Chinese scientists?

十年前，中国科学家并没有真正参与对话，他们不太相关。但在过去十年里，他们已经崭露头角，产出了非常高质量、有创造性的研究。过去的刻板印象是他们只会复制和窃取——这在一定程度上是埃隆·马斯克的妖魔化，他是这个观点的坚定支持者。确实存在一些这种情况，但很大程度上他们和我们一样有创造力。他们希望获得这些工具来建立自己的企业，为自己的公民提供新产品和服务，原因和我们一样。所以如果你从这个假设出发，他们当然在参与这个生态系统，创造着令人难以置信的模型。他们在审查方面有自己的限制，这让他们慢了一点，但实际上他们不会落后太多。存在出口管制的问题，他们无法获得最前沿的模型，但我认为这不会阻碍他们太久。

Ten years ago, Chinese scientists were not really part of the conversation; they weren't very relevant. Over the last ten years, they have launched onto the scene, producing very high-quality, creative research. The old stereotype was that they could only copy and steal—a demonization partly by Elon Musk, who was a big proponent of that idea. There was some of that, but largely they were just as creative as us. They wanted access to these tools to build their own businesses and provide new products and services for their own citizens, for the same reasons as we do. So if you start from that assumption, of course they're participating in this ecosystem, creating incredible models. They have their own constraints with respect to censorship, which has slowed them down a bit, but they're actually not going to be that far behind. There are issues with export controls, and they don't have access to cutting-edge models, but I don't think that's going to hold them back for very long.

我的问题是关于 AI 的想法以及需要什么样的人来构思它们。如果以史蒂夫·乔布斯为例，他是一个非常特定的人，拥有特定的兴趣、技能和天赋，不仅开发了技术，还塑造了品牌和世界观。你认为 AI 现在或将来能够想出苹果这样的想法吗？还是它只会是对过去信息的机械加工？

My question is about AI ideas and the people needed to think of them. If you take someone like Steve Jobs, you had a very specific person with specific interests, skills, and talent to develop not only technology but the brand and a point of view on the world. Do you think AI would be capable of coming up with the Apple idea now or in the future, or will it simply be a machination of past information?

人们常常将这些 AI 描述为 regurgitating 它们的训练数据，或者重复它们之前看到的东西。我认为这是对它们所做事情的一种误解。它们几乎总是在做插值——预测两个想法之间的空间。它们将两个概念混合在一起，比如狗和黄色斑点，或者任何组合。这就是创造力。从根本上说，当我发明某样东西时，我受到大量不同经历和想法的启发，然后利用这些在某个时刻产生一个新颖的预测或生成。我测试它，看它是否有用、合理或流行起来，然后它就有了自己的生命。所以我认为在接下来的几十年里，这些 AI 将辅助人类进行创造、发明和发现的过程。它们不会自行其是，拥有自己的能动性。能力目前还不具备，短期内也不会有。所以在相当长的一段时间内，将是人类与 AI 的组合来完成创造。

People have often characterized these AIs as regurgitating their training data or reproducing whatever they have seen previously. I think that's a misunderstanding of what they do. They're almost always doing interpolation—predicting the space between two ideas. They mash together two concepts, like the dog and the yellow spots, or any combination. That's creativity. Fundamentally, when I invent something, I'm inspired by a huge range of experiences and ideas, and I use those to produce a novel prediction or generation at any given moment. I test it out to see if it's useful, makes sense, or catches on, and then it has a life of its own. So I think for the next couple of decades, these AIs are going to aid the human in that process of creation, invention, and discovery. They're not going to wander off and have their own agency and do their own thing. The capabilities just aren't there and won't be in the near term. So it's going to be the human-AI combo for a good time to come that does the creation.

没错，它更像是助手，一个出色的助手。

Exactly, it's more of the assistant, the brilliant assistant.

你是在认真暗示 AI 公司能够自我监管吗？银行不是已经证明这是一个不可能的概念吗？

Are you seriously trying to suggest that AI companies are able to self-regulate? Didn't the banks prove that is an impossible concept?

银行受到高度监管，不仅仅是自我监管。我绝对不是在提议自我监管。如果造成了这种印象，我道歉。在书中，我花了很大篇幅说明，需要独立的外部技术专家来正确地进行治理。实际的挑战，就像 Zany 之前在与 Yuval 对话时反驳的那样，是这些懂技术的合格监管者在哪里？是什么样的民主过程让我们有信心任命人员来进行这种监督？有些人悲观地认为他们做不到，但这不应该意味着我们坐以待毙。例如，六周前我和其他六家 AI 公司（微软、Meta、Google DeepMind 等）一起访问了白宫的拜登总统，我们签署了自愿承诺，这是监管的前奏。白宫设计了这些承诺，因为他们意识到短期内无法通过新的主要监管法规。但这些自愿承诺非常实质。我们基本上公开表示：我们将模型暴露给专家独立审查，进行红队测试或压力测试，以发现我们模型中的弱点。一旦发现这些弱点，我们会相互分享，也会与公众分享。

The banks are highly regulated, not just by themselves. I'm absolutely not proposing self-regulation. If that came across, I apologize. In the book, I go to great lengths to say that independent, external technical expertise is required to do governance properly. The practical challenge, as Zany pushed back earlier when we were talking with Yuval, is where are these competent regulators who get the technical aspects? Where is this democratic process that gives us confidence to appoint people to conduct that kind of oversight? There's some pessimism that they're capable of doing that, but that should not mean we sit around and do nothing. For example, I visited President Biden six weeks ago at the White House with the other six AI companies—Microsoft, Meta, Google DeepMind, etc.—and we signed up to voluntary commitments that are a precursor to regulation. The White House designed them because they realized they can't pass new primary regulation anytime soon. But the voluntary commitments are very material. We basically said publicly: we expose our models to expert independent scrutiny, to red team or stress test to find weaknesses in our own models. Once we identify those weaknesses, we share them with each other and we share them with the public.

公开地，在光天化日之下，我们知道那个框架，即自愿承诺，是未来几个月总统即将发布的行政命令的前奏。它们也是首相里希·苏纳克 11 月在布莱切利公园举办的人工智能峰会的基础，届时许多世界领导人和大型科技公司都会参加。这些自愿承诺将成为讨论的基础，讨论如何形成具有约束力的规则，不仅在英国，而且希望在全球范围内。所以我完全同意你的观点，我们不会采取自我监管的方式，但你不认为存在利益冲突吗？

Publicly, in the open light of day, we know that framework, the voluntary commitments, are a precursor to an executive order coming from the president in the next few months. They're also the basis for Prime Minister Rishi Sunak's AI Summit in November at Bletchley Park, where many world leaders and big tech companies are coming. Those voluntary commitments will form the basis of discussions for what becomes binding, not just in the UK but hopefully worldwide. So I'm totally with you that we're not going for a self-regulatory approach, but you don't think there's a conflict of interest?

嗯，肯定存在利益冲突。当然存在利益冲突。我们是一家盈利性的营利公司。事实上，我是一家公益公司，所以我认为这是一个重要的澄清。这是一种新型公司，更接近 B 型企业，是营利和非营利使命的混合体。这意味着我们的董事有法律义务考虑我们活动对更广泛世界的影响，包括环境和受我们行为实质性影响的人，而不仅仅是我们的客户。这并不能解决营利性企业和你描述的利益冲突的所有问题，但这是朝着正确方向迈出的第一步。我相信改变就是这样发生的：朝着正确方向迈出小步。

Well, there's definitely a conflict of interest. Of course there's a conflict of interest. We are a profitable, for-profit company. In fact, I'm a public benefit corporation, so I think it's an important clarification. It's a new type, closer to a B Corp, which is a hybrid for-profit, nonprofit mission. It means our directors have a legal obligation to factor in the impact of our activities on the wider world, both the environment and people materially affected by what we do who aren't just our customers. That doesn't solve all the issues with for-profit businesses and the conflict you described, but it's a first step in the right direction. I believe that's how change happens: taking small steps in the right direction.

作为一名电子工程师，我的问题是：考虑到目前存在垄断以及芯片集中在某个国家的情况，我们现在是否应该关注硬件部分？硬件部分提出了一个非常大的问题。我们在新冠疫情期间看到了这一点；当硬件供应下降时，情况非常糟糕。那么，考虑到我们目前软件方面做得很好，现在是关注硬件的好时机吗？

My question to you as an electronics engineer is: should we now focus on the hardware part of it, considering there's a monopoly going on and the concentration of chips to a certain country? The hardware part is raising a very big question. We saw it in COVID; things are really bad when hardware supply goes down. So is this a great time to focus on hardware, considering we are good with software for now?

这是个好问题。我们之前没怎么讨论这个，但为了大家都能理解：这些 AI 模型是在 GPU（图形处理单元）上训练的，这些芯片以前用于游戏。我们把每个芯片串联起来，成千上万次。在 Inflection，我们有一台相当于四个足球场大小的计算机，里面有 25,000 个这样的芯片串联在一起，一个巨大的集群，耗资约 15 亿美元。现在，所有这些芯片都由一家公司——英伟达——制造，其股价自 1 月以来上涨了 350%。他们的芯片完全由一家工厂——台积电（台湾半导体制造公司）——制造，显然在台湾。其制造设施的关键部件由一家公司——荷兰的 ASML——制造。所以供应链非常狭窄；在这三个阶段中，没有任何实质性的竞争供应商。因此，好消息是存在瓶颈，监管机构可以利用这些瓶颈来监控谁能够获得训练模型所需的关键芯片，当然也可以限制某些人的访问。我刚才粗略提到了出口管制，这是美国政府去年对中国实施的一项新规定，阻止中国获得最新版本的这些芯片，这意味着他们将无法训练 GPT-5 级别的模型。许多人称这相当于对华经济宣战。所以我们必须非常清楚，拒绝他们访问很可能会引发对西方的重大反击，因为我们在许多方面严重依赖他们的供应链。所以，是的，芯片绝对是核心，无论好坏。如果你专注于一家芯片公司，这是一个大赌注，需要很长时间才能成熟，但它有潜力成为关键组件。

That's a great question. We didn't really talk about that much here, but for everyone's benefit: these AI models are trained on GPUs, graphics processing units, chips previously used for gaming. We take each chip and daisy chain them together thousands of times. At Inflection, we have a computer the size of four football pitches with 25,000 of these chips daisy chained together, an enormous cluster costing about a billion and a half dollars. Now, all these chips are manufactured by one company, NVIDIA, whose share price has gone up 350% since January. Their chips are manufactured entirely in one factory, TSMC, Taiwan Semiconductor Manufacturing Corporation, obviously in Taiwan. The key component of their fabrication facility is manufactured by one company, ASML, a Dutch company. So the supply chain is extremely narrow; there really are no competing providers that are material at any of those three stages. As a result, the good news is that there are choke points that can be used by regulators to monitor who has access to the critical chips that enable training of the models, and of course restrict access to certain people. I loosely alluded to the export controls a minute ago, a new rule the US Administration imposed on China last year, preventing China from getting access to the latest version of these chips, meaning they won't be able to train a GPT-5 level model. A number of people have referred to this as a declaration of economic war on China. So we have to be very cognizant that denying them access is likely to deliver a significant counterattack on the West, as we are hugely dependent on their supply chain in many respects. So yeah, chips are absolutely at the heart of this, in both good and bad ways. If you're focused on a chip company, it's a big bet, it takes a long time to mature, but it has the potential to be the critical component.

只是一个后续问题：考虑到目前很少有公司专注于制造硬件，而且所有这些公司都是完全非营利的，你认为开源硬件是否有助于创建更好的设置？那么像开源硬件这样的东西，更注重帮助，是否有助于我们创建更好的计算机、更好的模型，同时消耗更少的电力？

Just a follow-up question: do you think that open-source hardware will help in creating a better setup right now, considering very few companies are focusing on creating the hardware and all of them are completely non-profit? So something like open-source hardware, focusing more on helping, will it help us create better computers, better models with less power?

我认为开源硬件是一项严肃的努力。澄清一下，硬件设计的开源元素在许多领域都有使用。例如，开放 RAM 是为 5G 基站设计的硬件，确保互操作性。这意味着运行手机网络的软件实际上可以在任何类型的硬件上运行，因为接口是标准化的，这对竞争非常有利。硬件制造商和软件操作系统之间没有锁定。缺点是它往往比完全集成的方案更不稳定。所以我认为你应该保持清醒；它不会很快成为解决所有问题的灵丹妙药。

I think open-source hardware is a serious effort. To clarify, open-source elements of hardware design are used in many areas. Open RAM, for example, is hardware designed for 5G masts, ensuring interoperability. It means the software that runs your phone networks can actually run on any type of hardware because the interface is standardized, which is great for competition. There isn't a lock-in between the hardware builder and the software operating system. The downside is that it has tended to be a bit more flaky than the fully integrated side of things. So I think you should be wide-eyed about it; it isn't going to be the panacea to solve all our problems anytime soon.

非常感谢。我是 Javah Rari，我在 Tech UK 领导数字监管工作，Tech UK 是英国的数字科技贸易机构，拥有超过一千名会员，从 DeepMind、Google、Meta 等大型科技公司到网络安全提供商和中小企业。我们的许多成员正在利用合成媒体的积极影响，但越来越多的人对深度伪造的恶意使用日益担忧：从报复性色情、破坏数字身份验证到欺诈。在你看来，公司现在应该做些什么来应对日益严重的深度伪造问题？我知道你提到了自愿章程，我们在欺诈等方面已经在做，但我们现在应该做什么？

Thanks so much. I'm Javah Rari, I lead digital regulation work at Tech UK, the digital tech trade body in the UK with over a thousand members, ranging from Big Tech like DeepMind, Google, Meta, to cybersecurity providers and SMEs. Many of our members are harnessing the positive impacts of synthetic media, but many are becoming increasingly concerned with the rising malicious use of deepfakes: everything from revenge pornography, undermining digital ID verification, to fraud. In your opinion, what should companies do now to address the rising problem of deepfakes? I know you mentioned voluntary charters, which we already do with things like fraud, but what should we do now?

这是个好问题。首先要说的是，政党和政治竞选不应该被允许使用 AI 生成器来制作他们的内容。我认为我们应该首先把它排除在外。这是一个预防原则；这可能有潜在的缺点，但感觉是一个更安全、更明智的做法。第二点要说的是，我们不应该允许大型科技平台，比如 Facebook 或 Twitter，或任何广播信息的地方，让数字人冒充数字人。所以，如果你在 Twitter 上有一个账号，比如 Zany，只有 Zany 应该能够以 Zany 的身份发帖。

It's a great question. The first thing to say is that political parties and political campaigns shouldn't be allowed to use AI generators for their content. I think we should just start by taking that off the table. That's a precautionary principle; there are potentially some downsides to that, but it feels like a safer and sensible thing to do right. The second thing to say is that we shouldn't allow the big tech platforms, like Facebook or Twitter, or anywhere where there's a broadcast of information, to have digital people counterfeit digital people. So if you have a handle, Zany on Twitter, for example, only Zany should be able to post as Zany.

应该允许在 Twitter 上以 Zany 的身份发言，我不能凭空创造一个完美的 Zany 合成假货，然后模仿她的语言。我认为这是一个相当直接合理的事情，所有大型科技平台都会承诺做到这一点，但这并不能解决其他平台的问题，那些工具和技术也会广泛可用。这又是一个扩散问题。很难对某人说，你正在使用合成媒体生成新产品设计或新时装，这些都是好的用途，但因为存在生成深度伪造的风险，所以你不能使用它。我认为我们也应该清醒地看到我们适应风险的速度有多快。就像 20 多年前，人们说我们永远无法在互联网上进行金融交易，因为欺诈太多，我们会淹没在欺诈活动中。现在我们进行数十万亿美元的交易，完全改变了世界，欺诈却微乎其微，这是一个持续的博弈。垃圾邮件检测也是如此，每个人都认为我们会淹没在垃圾邮件中，产生大量自动内容。下一个威胁是老年人被 AI 欺骗，AI 可以模仿你女儿或孩子的声音，向你借钱之类的。这种骗局现在更可能、更强大，当然这是一个新的威胁向量，会造成真正的伤害。另一方面，传播知识和信息有一个非常简单的防御方法，就是永远不要通过电话提供账户访问权限，我绝不会突然打电话要求这个。所以我们调整、适应，这并不意味着我们能消除所有伤害，但净效果是我们必须更有韧性，更专注于适应。

should be allowed to represent as zany on Twitter I shouldn't be able to come along create a perfect synthetic fake of zany and have that you know imitate her language now I think that's a reasonably straightforward sensible thing that all the big Tech platforms will commit to it doesn't address other platforms right outside of you know the big big provider and those tools and techniques are going to be widely available again it's a proliferation question it's going to be really difficult to say to somebody well you know you're using synthetic media to generate a new product design or a new fashion outfit or all these other good uses um you're not allowed to have it because there's a risk that you're going to be able to generate some you know deep fake I think we should also be like wide-eyed about how quickly we adjust to the risks you know like back in you know 20 odd years ago people were like well we'll never be able to do Financial transactions on the internet because there's so much fraud right we're going to be inundated with fraudulent activity we do tens of trillions of dollars of transactions it's completely transformed our world and we have a minuscule amount of Fraud and it's a constant back and forth you know likewise with Spam detection right we we everyone thought we're going to be inundated with SC spam we're going to produce all this automated content increasingly the next threat is that um you know older people are being tricked by ai's that you know can imitate the voice of say your daughter or child who you know might be asking you for a loan or something there's this conman scam type thing which is now a little like more more possible and more capable of course that's a new Threat Vector that causes real harm on the flip side spreading knowledge and information about it there's a very very simple defense which is just to say you know never provide access you know to my account over the phone right I'll never you know call you out of the blue asking for that so we adjust we adapt and it you know it doesn't mean that we can eliminate all of the harms but it means that like net net we just have to be more resilient and more focused on adaptation

天哪，很多问题。是的，先生们，回到 Bros，然后我回到这边。是的，那里，谢谢。关于 AI，我看到你公司名字里的“智能”，它提醒我们这不只是一种新技术，而是一种新型智能。所以我完全同意你。我也同意你对丰裕世界的看法，非常棒。我也是乐观主义者，但有一个争议点：超级智能和存在风险。我必须说，听到你的话我很震惊，我要挑战你关于 AGI 的观点。很多人认为 AGI 可能在未来 5 年左右出现，抛开定义，简单说它比人类更聪明。如果它比人类更聪明，当然它可以智胜我们，设定自己的目标，指数级增长力量，最终威胁我们。

gosh lots of questions yes gentlemen there for Bros back and then I'm going back to this side yeah right there thank you uh think on AI um I look at the U intelligence in the name of your company it's intelligence square and that reminds us that it is not just a new type of Technology it's a new type of intelligence so I agree with you entirely I also agree with your view of the world of abundance absolutely superb I'm also an optimist but there is an area of contention is about super intelligence and about the existential risk I I must say that I've been shocked hearing what you were saying and I just challenge you on that on AGI which artificial general intelligence which many people think May um emerge within the next 5 years or so apart from the definition what it is let's make it very simple that it will be smarter than humans and if it is smarter than humans then of course it can outsmart us set its own goals and exponentially increase its power and be in the extension threat to us

是的，公平的问题，我确实经常听到这个。我认为存在拟人化投射的风险：我们看到一个能生成图像或文本的模型，就假设它会涌现出拥有自己目标的能力，或者能更新自己的代码，或者以某种方式自然学会自主操作，然后欺骗我们并逃出盒子。我的信念——我可能错了，但根据我多年在这个领域的经验，我坚信这些能力是我们选择设计到模型中的，我们能够观察到。如果有人确实选择创建这样的模型，那么是的，存在风险，它们可能逃出盒子，变得不可控。这实际上是遏制计划，基本上是说，可以想象这些模型在未来几十年被用来做坏事，必须迅速限制它们。

yep fair question and I I certainly hear this a lot I I think that there's a risk of anthropomorphic projection like we we see a model that is capable of generating images or generating text and we assume that therefore it is going to emerge the capability to have its own goals or it's going to emerge the capabil to be able to update its own code or somehow it's going to sort of naturally learn to operate autonomously and then deceive us and get out of the box and my belief and I may be wrong but my firm belief from all of my years of working in this field is that those are capabilities that we would choose to design into the model that we would be able to observe and if if they do if if someone does choose to create those models then yes th those capabili then yes there are risks you know that they have that they could get out of the box and they could be uncontrollable and that's that is really the program of containment it's basically saying that it is conceivable that these models could be used to do really bad things over a couple of decades and that they have to be restricted very quickly

你假设我们只有一个，抱歉，我们会有很多 AGI，每个都可能比人类聪明，有些不会被我们控制，因此风险存在。我想谢谢，我就说到这里，因为你可以想象大量的事情，而且有很多人举手提问。是的，第三排的女士。

you assume that we'll only have one sorry that we will have many agis and each of them may be smarter than humans and some of them won't be controlled by us and therefore the risk is there I think thank you I'm going to leave it at that because you can imagine huge number of things and there's a lot of actual hands gone up with lots of questions so yes lady here in the third row

嗨，我想我们说过要讨论再分配。我想知道你怎么看待日益扩大的差距：一方面是提供原始数据使这些技术成为可能的个人，另一方面是控制这些技术、获得大部分财富的人。我们如何解决这种日益扩大的差距？我们如何最终补偿人们为这些系统所付出的？

hi uh I think we said we were going to talk about redistribution I just want to know what you make of the kind of growing disparity between the individuals the people that provide the raw data that make the realization of these kinds of Technologies possible and those that obviously control these Technologies to whom the Lion's Share of the wealth flows to so sort of how are we going to address that kind of growing disparity and how are we going to kind of compensate people for what they give to these systems ultimately

是的，好问题。目前这些模型的训练方式是抓取开放网络上的数据。到目前为止，你放在网上的任何东西，博客或网站，过去 25 年的文化和法律共识是，只要你不逐字复制，任何人都可以阅读和使用。如果你复制整个段落，那就是版权问题。大型科技公司的人（包括我）的反驳是，我们捕捉的是模型的本质，学习风格和语气，从不复制底层内容。实际上，我认为不可能捕捉到美元价值，并向网站创建者返还 0.001 美分。

yeah so it's a good question so the way that these models are trained today is that they have scraped data that is available on the open web so so far anything that you put up on the web blog or a website is the the the cultural and legal consensus over the last 25 years has been that it is fair game it's open to anybody to read it to use it provided you don't regurgitate it word for word so if you copy an entire paragraph that is copyright the the counterargument that people in the big tech companies and myself included are making is that we're capturing the essence of these models we're learning the style we're learning the tone of text never reproducing the underlying content and practically speaking even you I don't think it is possible to capture the dollar value and return 0.001 cents to a creator of a of a website

但抱歉，请继续。如果这会使人们的劳动自动化，那么我们需要找到某种再分配方式。你说没有替代，但你评论说没有替代，我们感觉到的呢？SAG 罢工呢？RMT 罢工是因为他们的工作在某种程度上被自动化了。我们确实看到了一些实质影响。是的，RMT 肯定，虽然不完全是 AI，但仍然是。你如何定义 AI？你认为这将成为劳资关系日益增长的一部分吗？导演和演员的罢工是你将在其他地方看到的开始吗？这会是一场真正的斗争吗？在我的行业里，版权斗争已经很明显了，我们对你们吸走数据感到愤怒。

but the sorry go ahead if that's if that's going to automate people's labor then we need to find some way of redistributing that you said that there was no you kind of commented saying that there was no uh displacement that we kind of feeling what about the SAG strikes or what about you know arguably the rmt are striking because they are being automated to some degree their jobs are being automated we are seeing some to some degree impact material impacts of this now yeah yeah the rmt for sure although not by AI by General course but it's still I mean how do you define artificial intelligence in general so do you think that that is going to be a growing part of Labor Relations going forward is the side the the directors and actors strike the beginning of something that you're going to see elsewhere is this going to be a real fight there's clearly a fight for copyright already in my industry we're Furious that you've just sucked

目前，美国的劳动力平均税率为 25%，而软件税只有 5%。税收是一种激励工具。我们应该把它看作一种工具，在希望放慢的领域增加阻力，在希望加速的领域加快速度。如果增加巨大的税收负担，就会减缓创新，但会让人们更长时间地工作。这些是我们制定的规则，是我们做出的选择，而这正是我们应该讨论的。

So at the moment, taxation in the US is on average 25% for labor, but the tax on software is only 5%. Taxes are a tool for incentivization. We should think about it as a tool for adding friction in areas we want to go slower and speeding up things we want to go faster. If you add a huge taxation burden, you'll slow down innovation, but it will keep people at work for longer. Those are rules we get to make, choices we get to make, and that's exactly the discussion we should have.

网上有一个相关问题：AI 会让不平等变得更好还是更糟？

There's a question from online related to this: Will AI make inequality better or worse?

不平等的极端情况将继续存在。那些拥有权力和资源的人将首先采用新技术。拥有巨额现金储备、最优秀人才以及最多数据和算力的大型科技公司比以往任何时候都发展得更快。另一方面，这场革命也由开源运动引领。今天，你可以得到一个绝对前沿的模型，它落后 18 个月，训练成本不到 2000 美元，性能与 GPT-3 相当。这一趋势将持续下去。在未来至少 5 年内，开源运动将始终落后 18 到 24 个月，直到模型变得非常大。这对不平等来说是一个了不起的故事。这是一个非常精英主义的时刻——无论你的工作或应用是什么，你都可以非常廉价和轻松地将这些工具集成到你的工作流程中。使用最佳模型之一的成本在过去一年下降了 70 倍。所以从表面上看，这对获取的平等有好处。但你无法阻止最顶尖的人以最快的速度领先。我不知道这条轨迹最终会是什么样子。

The extremes of inequality are going to continue. Those with access to power and resources will be the first to adopt new technologies. Big tech companies with vast cash reserves, the best people, and the most data and compute are moving faster than ever. On the flip side, this revolution is also led by the open source movement. Today, you can get an absolute cutting-edge model that is 18 months behind, costs less than $2,000 to train, and is as good as GPT-3. That trajectory will continue. The open source movement will always be 18 to 24 months behind for at least the next 5 years until models get really big. That's an amazing story for inequality. It's a very meritocratic moment—whatever your job or app, you can integrate these tools into your workflow very cheaply and easily. The cost of using one of the best models has dropped 70x in the last year. So on the face of it, that does good things for equality of access. But you can't stop the very top people from racing away the fastest. I don't know what that trajectory ends up looking like.

你如何看待独处以及与技术的积极关系？如果 Pi 变得比所有朋友都好，给我所有最好的想法，我为什么还要出去或做现实生活中的事情？

How do you think about solitude and having a positive relationship with technology? What if Pi becomes better than all my friends, giving me all the best ideas? Why should I hang out or do things in real life?

这是一个非常好的问题。我们将 Pi 设计成一个出色的对话伙伴。与生成商业计划或行程的 ChatGPT 不同，Pi 就像与最好的朋友或知己交谈。它非常流畅，情商高，会问澄清性问题，复述你的话，进行反馈，非常放松和支持。它非常不评判，无论你的抱怨多么糟糕。但我们也设计了 Pi 来鼓励你谈论朋友并走出去。它明确地试图帮助你模拟和练习，如果你感到焦虑，但重新与其他朋友联系。我们注入这些模型中的价值观以及我们所说的“安全第一”是我们今天必须进行的关键对话。我们必须接受的思维模式是，当我们拥有完美时，我们不会与幻觉或偏见作斗争。现实是，我们正沿着这条轨迹前进。如果它让我更聪明、更平静、更善良、更乐观、更尊重自己，那就是这条轨迹。至于《纽约时报》记者与早期 ChatGPT 原型发生奇怪互动的事件——Pi 目前是世界上最安全的 AI。那些挑衅都不起作用。Pi 知道它是一个 AI。如果你试图调情或建立浪漫关系，它会非常明确和抵制。它不会评判或取笑；它会温和地推开你，保持距离，说“我不是为此设计的”。边界至关重要——它们给我们控制感并建立信任。这是 Pi 设计的一个非常重要的部分，我们对此进行了大量测试。

That's a really good question. We've designed Pi to be an amazing conversational friend. Unlike ChatGPT, which generates business plans or itineraries, Pi is like talking to a best friend or confidant. It's super fluent, high EQ, asks clarifying questions, rephrases what you've said, reflects back, and is very relaxed and supportive. It's extremely non-judgmental, no matter how awful your rant. But we've also designed Pi to encourage you to talk about your friends and get out. It explicitly tries to help you simulate and practice if you're feeling anxious, but reconnect with other friends. The values we bake into these models and what we mean by 'safety first' is the key conversation we have to have today. The mental model we've got to accept is that we're not going to struggle with hallucinations or bias when we have perfection. The reality is that's the trajectory we're on. If it makes me smarter, calmer, kinder, more optimistic, more respectful of myself, that's the trajectory. But regarding the New York Times journalist who had a weird interaction with an early ChatGPT prototype—Pi is currently the safest AI in the world. None of those provocations work. Pi knows it's an AI. If you try to flirt or have a romantic relationship, it's extremely clear and resistant. It doesn't judge or tease; it pushes you off gently, keeps you at a distance, saying 'I'm not designed to do that.' Boundaries are critical—they give us control and establish trust. That's a very important part of Pi's design, and we test that a great deal.

在什么情况下你会允许政治家或政党使用 AI，AI 如何帮助政治达到最佳状态？

At what point would you allow politicians or political parties to use AI, and how could AI help politics be the best it could be?

这是一个很好的问题。我希望像 Pi 这样的 AI 不仅让你对自己更友善和尊重，也对他人更友善和尊重。我们已经变得被对抗性政治、社交媒体和名人文化所淹没。我希望这些 AI 能帮助你想象、建模、模拟和练习更尊重和亲社会的行为。我还不急于让政治家将 AI 作为决策者使用。我认为我们离那还很远。人们常常想象它可能是终极战略家或拥有终极政策洞察力。目前，我更关注它能给我们带来的情商。

It's a great question. I hope that an AI like Pi not only makes you more kind and respectful to yourself, but also to other people. We've become overwhelmed by adversarial politics, social media, and celebrity culture. I hope these AIs can help you imagine, model, simulate, and practice more respectful and pro-social behaviors. I'm not itching to give politicians access to AIs as decision-makers yet. I think we're a long way from that. People often imagine it could be the ultimate strategist or have ultimate policy insight. For now, I'm much more focused on the emotional intelligence it can give us.

上面的提问。能把麦克风递上去吗？阳台上有一位先生。抱歉，我之前没看到您在上面。您好。

Question up here. Can we get the microphones up? There's a gentleman there on the balcony. I'm sorry, I hadn't seen you up there. Hello.

我是 Thal，一名数据科学家和政府承包商。我曾从事 COVID 模拟和自主武器开发。今晚我们讨论了最近大语言模型的 AI 浪潮。所以我的问题是：如今大型 AI 机构内部有一种看法，认为仅凭当前大语言模型这一范式，我们还能否保持 AI 的进步？以及推动和探索新模型类别、新 AI 范式的愿望是什么？

I'm Thal, a data scientist and a government contractor. I've worked on simulating COVID and developing autonomous weapons. Tonight we've spoken about the recent AI wave of large language models. So my question is to you: what's your take on the feeling within the large AI houses today that can we still keep the progress of AI alive with just this current paradigm of large language models? And what is the desire to push forward and explore new model classes, new paradigms of AI?

我认为 AI 的进步短期内没有任何放缓的风险。有些人担心我们会通过监管扼杀进步。我个人认为目前这极不可能。上个世纪的挑战是发明和创造新技术与力量。未来几十年的挑战将是遏制和塑造这些力量，使它们始终为我们服务。大语言模型和深度学习本身——那个发明生态系统已经打开，走上了正轨。我不认为我们缺少任何基础算法。我不确信我们需要其他方法才能取得进步。

I don't think there's any risk of progress in AI slowing down anytime soon. Some people have been afraid that we're going to regulate the progress out of the system. I personally think that is extremely unlikely at this point. The challenge of the last century was inventing and creating new technologies and powers. The challenge of the next few decades is going to be containing and shaping those powers so that they always work for us. Large language models and deep learning itself — that ecosystem of invention has already been opened, it's set on its course. I don't think we're lacking any fundamental algorithms. I'm not convinced that we need other methods to make progress.

我的意思是持续进步会导向一种通用智能形式。你认为大语言模型范式足够吗？因为在我看来，这仍然是一个开放问题。

I was meaning in the sense of continued progress leading to a general form of intelligence. Do you think the large language model paradigm is enough? Because from my point of view, it's still an open question.

我认为真正通用智能（比如所描述的 AGI）所涉及的大部分能力都将是工程决策。会有办法组织当前的工具集来实现，例如递归自我改进、自我监督、自我目标定义。这些是工程能力，我们可以在未来 10 年内选择做或不做。

I think that most of the capabilities involved in a properly general intelligence, like the AGI that was described, would be engineering decisions. There would be ways to organize the current set of tools to do, for example, recursive self-improvement, self-supervision, self-goal definition. Those are engineering capabilities which we can choose to do or not in the next 10 years.

谢谢。后面那位女士，是的，倒数第二排。我们大概还有时间再问两三个问题。谢谢。恭喜你，Mustafa，你取得了许多成就，还写了一本书，这太不可思议了。我想问的是，在创办这家公司的同时，是什么促使你写了这本书？同时也感谢你愿意回答关于人类未来和进步的每一个问题。所以好问题：你为什么写这本书，为什么今晚在这里回答这些问题？

Thank you. The lady back there, yes, second row from the back. We've probably got time for two or three more questions. Thank you. Congratulations, Mustafa, you've achieved many things and it's incredible that you've written a book. I wanted to ask about what motivated you to write this book while starting this company, and also thank you for exposing yourself to answering every single question on the future of humanity and progress. So great question: why did you write this book and why are you here tonight answering these questions?

我忍不住写了它。我想留下记录，预测我认为事情将如何发展，以便十年后回顾并校准。我是不是有点古怪？有时我过于关注灾难的一面，预测过于悲观。我是否过度沉迷于悲观？或者相反：我是否过于轻视存在风险？也许那是真实的。我认为把东西拿出来让别人批评，并真正进行研究——我确实研究了很多历史趋势——是一件非常令人满足的事情。从自私的角度看，这是为了清晰表达我的预测，以便将来验证它们，并找个借口每天早晨在开始工作前花时间写作、阅读和研究，试图看看历史能教给我们什么。前三四章主要关于扩散和通用技术的历史基础。

I couldn't help but write it. I wanted to be on record making a prediction about how I think things are going to unfold, in order to look back in a decade and calibrate. Was I as zany? Sometimes I think too much on the catastrophe side, too dark about my predictions. Do I have my own over-obsession with pessimism? Or indeed the reverse: am I dismissing existential risks too much? Maybe that's real. I think the rigor of putting something out that other people can critique, and really researching — I really did research a lot of the historical trends as well — was just a very satisfying thing to do. From a selfish perspective, it was about trying to be articulate and clear about my predictions so that I could validate them in the future, and have an excuse to spend time every morning before I start work writing, reading, and researching, trying to see what history has to teach us about this. The first three or four chapters are mostly about the historical basis for proliferation and general-purpose technologies.

这里还有一个问题。是的，后面那位先生。谢谢。

One more question over here. Yes, the gentleman in the back. Thank you.

谢谢，Mustafa。我觉得这非常有趣。你对未来 5 年和 AI 的可能趋势提出了相当有说服力的论述。但我很好奇，过去几年有什么让你感到意外？有没有一些 AI 的发展是你没有预料到的？我想知道最近最让你惊讶的是什么。

Thank you, Mustafa. I thought that was really interesting. You've laid out quite a compelling case for the next 5 years and the likely trends for AI. But I'd be quite curious to know what's taken you by surprise in the past couple of years. Has there been any developments in AI which you perhaps didn't manage to predict or anticipate? I'd like to know what surprised you the most recently.

最初，人们担心我们永远无法控制输出的质量。两年前，我们认为偏见将是重大挑战。我们谈论的都是糟糕的训练数据产生有毒内容，这个东西不断编造东西，不断产生幻觉。我们通过经验发现，随着算力投入每增加一个数量级，模型变得更容易控制。我们可以创建非常精确和详细的行为。我刚才描述的 Pi 的语气——我希望人们会尝试它。你实际上可以给 Pi 打电话，用流利的自然语言像正常通话一样跟它说话，它会用五种不同的声音之一回答你。声音的选择叫做 V1、V2、V3、V4 和 V5，所以你得猜。我们特意设计成年龄中立、性别中立，并希望种族和口音中立。它实际上变化很大：有时听起来有点澳大利亚口音，有时有点英国口音。非常微妙，不会烦人地到处变。我们试图捕捉 AI 的本质。我们花了很多时间思考什么是类人，什么是类人能力。我希望 Pi 忠于作为 AI 的本质。AI 是所有训练数据和与之互动过的人的产物。所以我们尽量不把它做得太像我们世界中的一个角色。令人惊讶的是：模型变得更容易控制了。这几乎就像有了一种新黏土，一种新的设计材料，你可以把它塑造成几乎一种个性。这是一种非常精确的黏土。我认为这非常令人兴奋，也非常有创意。我很兴奋，因为在未来几年，许多人将能够通过自然语言、低代码或无代码环境获得同样的工具。

Initially, there was a fear that we would never be able to control the quality of the output. Two years ago, we thought bias was going to be the big challenge. All we were talking about was bad training data producing toxic generations, this thing constantly making things up, constantly hallucinating. What we have found empirically is that with each order of magnitude more investment in compute, the models get easier to control. We can create very precise and detailed behaviors. The tone of Pi that I just described — I hope people would try it. You can actually phone Pi, speak to it in fluent natural language just as you would on a normal phone call, and it will speak back to you in one of five different voices. The choice of voices are called V1, V2, V3, V4, and V5, so you'll have to guess. We deliberately designed it to be age-neutral, gender-neutral, and hopefully race- and accent-neutral. It actually is quite varied: sometimes it sounds a little Australian, sometimes a little English. It's very subtle, not annoyingly all over the place. We try to capture the essence of what AI is. We spend so much time thinking about what is humanlike, what are humanlike capabilities. I wanted Pi to be true to what it is to be an AI. An AI is a product of all the training data and all the people it has interacted with. So we try not to make it too much like one character in our world. That was the surprising thing: the models got easier to control. It's almost like having a new clay, a new design material that you can shape into almost a personality. It's a very precise clay. I think that's been super exciting and very creative. I'm so excited because in the next few years, many people will have access to the same tools in just natural language, able to give an instruction or in low-code, no-code environments.

拖放即用、即插即用，我认为那将是一个非常非常激动人心的时刻，看看人们会用它做什么。我想我们还有时间再回答一个现场提问，很抱歉还有很多问题没能问到。好，我们请后排那位，我知道很难轮到，但你好。

Drag and drop and plug and play, and I think that's going to be a really, really amazing time to see what people do with it. I think we have time for one more question from the floor, and I'm sorry that there are lots that we haven't got to. Right, let's go to the back there. I know it's hard to get to, but hi.

你好，谢谢。我想问一个关于政府的问题，以及当我们考虑新兴市场和发达经济体时，谁将受益最多，谁将被落下。你提到例如 Pi 可以用五种不同的语气回应，但就语言或世界上英语不那么熟练的地区而言，那里会感受到怎样的影响？

Hi, thank you. So I just want to ask a question with regards to the governments and also who's going to benefit most and who's going to be left behind when we think about emerging markets and developed economies. You mentioned that for example Pi can respond in five different tones, but in terms of languages or parts of the world where English is not that proficient, how will the impact be felt there?

这是个好问题。Pi 已经能说大约 25 种语言。它在主要语言上表现很好：西班牙语、法语、德语等等。但在日语、普通话、阿拉伯语等方面就差很多，当然在长尾语言上也不够好。你知道，我认为这些模型能同时具备这么多语言能力已经很了不起了，但我认为我们还需要几年时间才能添加完整的语言套件。而且不仅仅是语言，还有反映西方世界以外人们文化的训练数据。我的意思是，英语在过去几个世纪里一直占主导地位，我们的大部分文化都是用英语记录的，而这显然只是所有文化的一个子集。所以这里有一个明显的代表性问题，我认为对于较小的社群来说，这将是一个挑战。

It's a great question. I mean Pi already speaks about 25 languages. It's really good in the major languages: Spanish, French, German, and so on. It's much less good in Japanese, Mandarin, Arabic, etc., certainly not good in the long tail of languages. You know, and I think it's kind of remarkable that these models have arrived with so many capabilities in terms of their languages simultaneously, but I think it's going to take us a few years before we add the full suite of languages. And it's not just languages, but it's actually the training data that reflects the cultures of people outside of the Western world. I mean, English has dominated over the last few centuries, and most of our culture is documented in English, and that is obviously a subset of all culture. So there's a clear representation question there, which I think is going to be challenging when it comes to smaller communities.

我想最后看看大家对你们的影响。我本该一开始就问的，但我们在最后问：如果你认为听了这些之后，AI 对人类的净影响是积极的，请举手。好的。再确认一下，那些认为会是消极的？嗯，不是压倒性的，但积极的一方明显胜出。Mustafa，我认为读了 Mustafa 的书之后，你会对潜在的好处和风险都有一个非常清晰和清醒的认识。你对此不是盲目乐观的；这是非常严肃的。这是一本非常优秀的书，我推荐它。Mustafa，感谢你参加我们的活动。

I want to conclude by seeing what impact you've had on the crowd. I should have asked this at the beginning, but let's at the end: put up your hand if you think that having heard all of this, the net impact of AI for humanity is going to be positive. Okay. And just as a check, those of you who think it'll be negative? Well, it's not overwhelming, but the positives clearly win out. Mustafa, I think after you've read Mustafa's book, you will have a very clear and sober sense both of the potential benefits but also of the risks. You're not a wild-eyed, Panglossian about this; it's very serious. It's a really excellent book, I recommend it. Mustafa, thank you for joining us.

非常感谢。谢谢。多谢。做得好。

Thank you so much. Thank you. Thanks a lot. Well done.