从化学到 AI：Greg Brockman 谈构建智能系统

以下是与格雷格·布罗克曼的对话。他是 OpenAI 的联合创始人兼首席技术官，OpenAI 是一个世界级的研究机构，致力于开发 AI 理念，最终目标是创造安全且友好的通用人工智能，造福并赋能人类。OpenAI 不仅是出版物、算法、工具和数据集的来源；其使命是推动关于我们与狭义和广义智能系统未来的重要公共讨论。这场对话是麻省理工学院及其他地方的 AI 播客的一部分。如果你喜欢，请在 YouTube、iTunes 上订阅，或直接在 Twitter 上联系我@LexFriedman。现在，开始我与格雷格·布罗克曼的对话。所以，在高中和之后不久，你写了一本化学教科书的草稿，涵盖了从原子基本结构到量子力学的所有内容。很明显，你对物理世界（化学，现在还有机器人学）和数字世界（AI、深度学习、强化学习等）都有直觉和热情。你认为物理世界和数字世界是不同的吗？你觉得差距在哪里？

The following is a conversation with Greg Brockman. He's the co-founder and CTO of OpenAI, a world-class research organization developing ideas in AI with the goal of eventually creating a safe and friendly artificial general intelligence, one that benefits and empowers humanity. OpenAI is not only a source of publications, algorithms, tools, and datasets; their mission is a catalyst for an important public discourse about our future with both narrow and general intelligence systems. This conversation is part of the Artificial Intelligence podcast at MIT and beyond. If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at @LexFriedman. And now, here's my conversation with Greg Brockman. So in high school and right after, you wrote a draft of a chemistry textbook that covers everything from basic structure of the atom to quantum mechanics. So it's clear you have an intuition and a passion for both the physical world with chemistry and now robotics, to the digital world with AI, deep learning, reinforcement learning, and so on. Do you see the physical world and the digital world as different, and what do you think is the gap?

实际上，这很大程度上归结于迭代速度。我认为真正激励我的是构建东西。以数学为例，你深入思考一个问题，理解它，然后用一种非常晦涩的形式（我们称之为证明）写下来，但它就存在于人类的知识库中，永远在那里。这是我们发现的某种真理。也许你所在领域只有五个人会读到它，但你在某种程度上推动了人类进步。我以前真的以为自己会成为一名数学家。然后我开始写这本化学教科书。一个朋友告诉我：‘你永远也出版不了，因为你没有博士学位。’所以我决定建一个网站，用那种方式推广我的想法。然后我发现了编程。在编程中，你深入思考一个问题，理解它，然后用一种非常晦涩的形式（我们称之为程序）写下来，但同样，它存在于人类的知识库中，任何人都能从中受益。可扩展性巨大。所以我认为数字世界真正吸引我的是，你可以拥有这种不可思议的杠杆作用。一个拥有想法的个体就能影响整个地球。这是你在移动物理原子时很难做到的事情。

A lot of it actually boils down to iteration speed. I think that what really motivates me is building things. Think about mathematics, for example, where you think really hard about a problem, you understand it, you write it down in this very obscure form that we call a proof, but then it's in humanity's library, right? It's there forever. This is some truth that we've discovered. Maybe only five people in your field will ever read it, but somehow you've moved humanity forward. I actually used to think I was going to be a mathematician. Then I started writing this chemistry textbook. One of my friends told me, 'You'll never publish it because you don't have a PhD.' So instead, I decided to build a website and try to promote my ideas that way. Then I discovered programming. In programming, you think hard about a problem, you understand it, you write it down in a very obscure form that we call a program, but then once again, it's in humanity's library, and anyone can get the benefit from it. The scalability is massive. So I think the thing that really appeals to me about the digital world is that you can have this insane leverage. A single individual with an idea is able to affect the entire planet. That's something I think is really hard to do if you're moving around physical atoms.

你提到了数学。如果你看我们的思维，你最终认为它只是数学，只是信息处理，还是有一些其他的魔力，就像你通过生物学和化学等看到的那样？

You mentioned mathematics. If you look at our mind, do you ultimately see it as just math, as just information processing, or is there some other magic, as you've seen through biology and chemistry and so on?

我认为将人类视为信息处理系统真的很有趣。这似乎是一种很好的方式来描述世界运作的很多方面，或者我们能力的很多方面。如果你回顾历史上的技术创新，从某些方面来说，我们拥有的最具变革性的创新是计算机。从某些方面来说，互联网不在于这些物理电缆；而在于我突然能够与地球上的任何其他人即时通信，并且能够检索人类曾经拥有的任何知识。这些都是不可思议的转变。

I think it's really interesting to think about humans as just information processing systems. It seems like it's actually a pretty good way of describing a lot of how the world works, or a lot of what we're capable of. If you look at technological innovations over time, in some ways the most transformative innovation we've had has been the computer. In some ways, the internet is not about these physical cables; it's about the fact that I am suddenly able to instantly communicate with any other human on the planet, and I'm able to retrieve any piece of knowledge that the human race has ever had. Those are these insane transformations.

你是否将我们的社会整体，即集体，视为人类智能的另一种延伸？如果你将人类视为信息处理系统，你提到了互联网，然后是网络。你是否认为我们作为一个文明整体，是一种智能系统？

Do you see our society as a whole, the collective, as another extension of the intelligence of the human being? If you look at the human being as an information processing system, you mentioned the internet, then networking. Do you see us all together as a civilization as a kind of intelligence system?

是的，我认为这实际上是一个非常有趣的视角。我们有整个社会的集体智能。经济本身就是一个超人的机器，在优化着某些东西。公司有自己的意志。所有这些个体都在追求自己的个人目标，并认真思考该做什么，但公司却做出了某种涌现的事情。所以这是一个非常有用的抽象。我认为在某种程度上，我们自认为是地球上最聪明、最强大的东西，但有些东西比我们更大，这些系统是我们所有人都贡献的。如果你读过阿西莫夫的《基地》，那里有心理史学的概念，即如果你有数万亿或数千万亿的个体，那么你也许真的可以预测那个巨大的宏观存在会做什么，几乎独立于个体的意愿。我实际上还有第二个角度，我觉得很有趣：思考技术决定论。在 OpenAI，我经常思考的一件事是，我们正在迎来通用智能这项极其变革性的技术，它将在某个时刻发生。问题是如何采取行动，真正引导它变得更好而不是更糟。我认为你需要问的一个问题是，作为科学家、发明家或创造者，你总体上能产生什么影响？看看像电话这样的东西，同一天被两个人发明。这对创新的形态意味着什么？我认为实际情况是，每个人都站在同一个巨人的肩膀上。你不能真的希望创造出别人永远不会创造的东西。如果爱因斯坦没有出生，别人也会提出相对论。他改变了一点时间线；也许需要再花 20 年，但人类绝不会永远发现不了这些基本真理。所以有一种无形的动力，像爱因斯坦或 OpenAI 这样的人在接入它，但其他人也可以接入。最终，这股浪潮将我们带向某个方向。

Yeah, I think this is actually a really interesting perspective. You have this collective intelligence of all of society. The economy itself is this superhuman machine that is optimizing something. A company has a will of its own. You have all these individuals pursuing their own individual goals and thinking really hard about the right things to do, but somehow the company does something that is this emergent thing. So there's a really useful abstraction. I think in some ways, we think of ourselves as the most intelligent and powerful things on the planet, but there are things that are bigger than us, these systems that we all contribute to. It's interesting to think about if you've read Asimov's Foundation, there's this concept of psychohistory, which is effectively that if you have trillions or quadrillions of beings, then maybe you could actually predict what that huge macro being will do, almost independent of what the individuals want. I actually have a second angle on this that I think is interesting: thinking about technological determinism. One thing I think a lot about with OpenAI is that we're coming onto this insanely transformational technology of general intelligence that will happen at some point. There's a question of how you can take actions that will actually steer it to go better rather than worse. I think one question you need to ask is, as a scientist, inventor, or creator, what impact can you have in general? You look at things like the telephone invented by two people on the same day. What does that mean about the shape of innovation? I think what's going on is everyone's building on the shoulders of the same giants. You can't really hope to create something no one else ever would. If Einstein wasn't born, someone else would have come up with relativity. He changed the timeline a bit; maybe it would have taken another 20 years, but it wouldn't be that humanity would never discover these fundamental truths. So there's some kind of invisible momentum that some people like Einstein or OpenAI are plugging into, but anybody else can also plug into. Ultimately, that wave takes us in a certain direction.

时间线有点关系，但如果你真的想产生影响，我认为你必须做的——你唯一真正的自由度——就是设定技术诞生的初始条件。想想互联网吧，有很多竞争对手试图构建类似的东西，而互联网赢了。初始条件是由一个真正重视人人可接入的群体创造的，这种非常学术的开放和连接心态。我认为互联网在接下来的 40 年里确实如此发展。也许今天事情开始朝不同方向转变，但我认为那些初始条件对决定接下来 40 年的进步非常重要。

The timeline a little bit, but if you really want to make a difference, I think the thing you really have to do—the only real degree of freedom you have—is to set the initial conditions under which a technology is born. So think about the internet, right? There were lots of other competitors trying to build similar things, and the internet won. The initial conditions were created by a group that really valued people being able to plug in, this very academic mindset of being open and connected. I think the internet for the next 40 years really played out that way. Maybe today things are starting to shift in a different direction, but I think those initial conditions were really important to determine the next 40 years of progress.

说得真好。另一个例子是：我最近看了维基百科的形成。我想知道如果维基百科有广告，互联网会是什么样子。关于他们为什么选择不在维基百科上放广告，有一个有趣的论点。我认为维基百科是互联网上最伟大的资源之一。它运作得如此之好，能聚合所有这些优质信息，这非常令人惊讶。维基百科的创建者基本上设定了初始条件，现在它自己向前发展。

That's really beautifully put. So another example of that I think about: I recently looked at the formation of Wikipedia. I wonder what the internet would be like if Wikipedia had ads. There's an interesting argument about why they chose not to put advertisements on Wikipedia. I think Wikipedia is one of the greatest resources we have on the internet. It's extremely surprising how well it works and how well it was able to aggregate all this good information. The creator of Wikipedia essentially set the initial conditions, and now it carries itself forward.

这真的很有趣。所以你思考 AGI 或人工智能时，你专注于为进步设定初始条件。

That's really interesting. So you're thinking about AGI or artificial intelligences, you're focused on setting the initial conditions for the progress.

没错。这很有力量。

That's right. That's powerful.

那么展望未来。如果你创建了一个 AGI 系统，比如一个能通过图灵测试、使用自然语言的系统，你认为你会与它进行怎样的互动？你会问什么问题？你会问它的第一个问题是什么？

So look into the future. If you create an AGI system, like one that can ace the Turing test, natural language, what do you think would be the interactions you would have with it? What do you think are the questions you would ask? What would be the first question you would ask it?

我认为在那个时候，如果你真的构建了一个能够塑造人类未来的强大系统，你真正应该问的第一个问题是：我们如何确保这一切进展顺利？所以这实际上是我会问一个强大 AGI 系统的第一个问题。

I think at that point, if you've really built a powerful system that is capable of shaping the future of humanity, the first question you really should ask is: how do we make sure that this plays out well? So that's actually the first question I would ask a powerful AGI system.

所以你不会问你的同事，不会问伊利亚，而是会问 AGI 系统？

So you wouldn't ask your colleague, you wouldn't ask Ilya, you would ask the AGI system?

哦，我们已经和伊利亚谈过了，对吧？还有这里的每个人。所以你希望尽可能多的视角和智慧来回答这个问题。我不认为你一定会听从你的强大系统告诉你的任何东西，但你会把它作为一个输入来尝试弄清楚该怎么做。

Oh, we've already had the conversation with Ilya, right? And everyone here. So you want as many perspectives and pieces of wisdom as you can for answering this question. I don't think you necessarily defer to whatever your powerful system tells you, but you use it as one input to try to figure out what to do.

我认为从根本上说，归根结底是：如果你构建了非常强大的东西，想想核武器创造之后不久。世界最重要的问题是：世界秩序会变成什么样？我们如何安排自己以便作为一个物种生存下去？对于 AGI，我认为问题略有不同。有一个问题是如何确保我们不会受到负面影响，但也有积极的一面。你可以想象 AGI 会是什么样子，它能做什么。我认为 AGI 能够强大且具有变革性的核心原因之一实际上是技术发展。如果你拥有一个像人类一样能干且更具可扩展性的东西，你绝对希望它去阅读整个科学文献，思考如何为所有疾病创造疗法。你希望它思考如何构建技术来帮助我们创造物质丰裕，并解决我们难以处理的社会问题，比如如何清理环境。也许你希望它发明一堆可生物降解的小机器人，出去将海洋垃圾转化为无害分子。我认为积极的一面是人们在思考 AGI 会是什么样子时有时会忽略的。所以如果你有一个能做所有这些的系统，你绝对希望它给出建议，关于如何确保我们以积极的方式为人类使用它的能力。

And I guess fundamentally, what it really comes down to is: if you built something really powerful, think about the creation of nuclear weapons shortly after their creation. The most important question for the world was: what's the world order going to be like? How do we set ourselves up so we're going to be able to survive as a species? With AGI, I think the question is slightly different. There is a question of how do we make sure we don't get the negative effects, but there's also the positive side. You imagine what AGI will be like, what it will be capable of. I think one of the core reasons an AGI can be powerful and transformative is actually due to technological development. If you have something as capable as a human and much more scalable, you absolutely want that thing to go read the whole scientific literature and think about how to create cures for all diseases. You want it to think about how to build technologies to help us create material abundance and figure out societal problems we have trouble with, like how to clean up the environment. Maybe you want it to invent a bunch of little robots that will go out and be biodegradable and turn ocean debris into harmless molecules. I think that positive side is something people sometimes miss when thinking about what an AGI will be like. So if you have a system capable of all that, you absolutely want its advice about how to make sure we're using its capabilities in a positive way for humanity.

那么你怎么看待这种心理：它看到 AGI 系统所有不同的可能轨迹，其中许多——也许是大多数——是积极的，却仍然关注负面轨迹？你和人们互动，你自己也思考这个问题。你看看山姆·哈里斯等人。似乎——抱歉这么说——思考负面可能性几乎更有趣。这深植于我们的心理。你怎么看，我们该如何应对？因为我们希望 AI 帮助我们。

So what do you think about that psychology that looks at all the different possible trajectories of an AGI system, many of which—perhaps the majority—are positive, and nevertheless focuses on the negative trajectories? You get to interact with folks, you think about this within yourself as well. You look at Sam Harris and so on. It seems to be—sorry to put it this way—almost more fun to think about the negative possibilities. That's deep in our psychology. What do you think about that, and how do we deal with it? Because we want AI to help us.

我认为这个问题包含两个问题。第一个是：你如何能想象一个拥有新技术的世界会是什么样子？想象我们在 1950 年，我试图向某人描述优步：应用程序和互联网。那将极其复杂，但可以想象。现在想象在 1950 年预测优步：你需要描述互联网、GPS、每个人口袋里都会有一部手机的事实。所以我认为第一个事实是，很难想象一项变革性技术将如何在世界上展开。我们以前在远不如 AGI 变革性的技术上也见过这种情况。所以一部分是，很难想象并真正置身于一个你能预测那个积极愿景会是什么样子的世界。第二件事是，支持负面总是比支持正面更容易。破坏总是比创造更容易——不是物理意义上，而是智力意义上。因为要创造某物，你需要把很多事情做对；要破坏，你只需要把一件事做错。所以我认为很多人的思维一看到负面故事就走进死胡同。但话虽如此，我实际上有一些希望。我认为积极愿景是我们能够谈论的。仅仅说出这个事实——是的，有积极，有消极，每个人都喜欢画它们——消极的人必须回应那个信息，说：‘嗯，你说得对，这里面有一部分……’

I think there are kind of two problems entailed in that question. The first is: how can you even picture what a world with a new technology will be like? Imagine we're in 1950 and I'm trying to describe Uber to someone: apps and the internet. That's going to be extremely complicated, but it's imaginable. Now imagine being in 1950 and predicting Uber: you need to describe the internet, GPS, the fact that everyone's going to have a phone in their pocket. So I think the first truth is that it is hard to picture how a transformative technology will play out in the world. We've seen that before with technologies far less transformative than AGI will be. So one piece is that it's just hard to imagine and to really put yourself in a world where you can predict what that positive vision would be like. The second thing is that it is always easier to support the negative side than the positive side. It's always easier to destroy than to create—less in a physical sense and more in an intellectual sense. Because to create something, you need to get a bunch of things right; to destroy, you just need to get one thing wrong. So I think a lot of people's thinking dead-ends as soon as they see the negative story. But that being said, I actually have some hope. I think the positive vision is something we can talk about. Simply saying this fact—yeah, there's positive, there's negatives, everyone likes to draw them—the negative people have to respond to that message and say, 'Huh, you're right, there's a part of this...'

我们没在谈论、没在思考的这一点，实际上我认为一直是 OpenAI 思考 AGI 的关键部分。你可以这样看：OpenAI 谈到存在风险，却仍在试图构建这个系统。你如何调和这两个事实？你是否认同一些人的直觉——从 Sam Harris 到 Elon Musk 本人——即在开发 AGI 时，很难防止它滑向存在性威胁？你对保持发展在积极轨道上的难度有什么直觉？

That we're not talking about, not thinking about. And that's actually something that I think has been a key part of how we think about AGI at OpenAI. You can look at it as: OpenAI talks about the fact that there are risks, and yet they're trying to build this system. How do you square these two facts? Do you share the intuition that some people have—from Sam Harris, even Elon Musk himself—that it's tricky as you develop AGI to keep it from slipping into existential threats? What's your intuition about how hard it is to keep development on a positive track?

要回答这个问题，你可以看看我们如何构建 OpenAI。我们有三个主要部门：能力部门，负责实际技术工作并推动这些系统的能力；安全部门，致力于技术机制以确保我们构建的系统与人类价值观对齐；政策部门，确保我们有治理机制，回答“谁的价值观”这个问题。技术安全是人们谈论最多的。许多反乌托邦 AI 电影都是关于没有良好的技术安全措施。我们发现，很多人认为技术安全问题难以解决——人类想要什么，如何写下来，我甚至能写下我想要什么吗？不可能。然后他们就停在那里了。但问题是，我们已经构建了能够学习人类无法指定的事情的系统。即使是识别图像中是否有猫或狗的规则，事实证明也无法写下来，但我们却能学习它。我们在 OpenAI 构建的系统中看到，仍处于早期概念验证阶段，你能够从数据中学习人类偏好，学习人类想要什么。这是我们技术安全团队的核心重点，而且我们在已经实现的工作方面取得了一些相当令人鼓舞的进展。

To answer the question, you can really look at how we structure OpenAI. We have three main arms: capabilities, which is actually doing the technical work and pushing forward what these systems can do; safety, which is working on technical mechanisms to ensure that the systems we build are aligned with human values; and policy, which is making sure that we have governance mechanisms, answering the question of whose values. The technical safety one is the one people talk about most. A lot of the dystopian AI movies are about not having good technical safety in place. What we've been finding is that many people look at the technical safety problem and think it's intractable—this question of what humans want, how to write that down, can I even write down what I want? No way. Then they stop there. But the thing is, we've already built systems that are able to learn things that humans can't specify. Even the rules for recognizing if there's a cat or a dog in an image turns out to be intractable to write down, yet we're able to learn it. What we're seeing with systems we build at OpenAI, still in an early proof-of-concept stage, is that you are able to learn human preferences, learn what humans want from data. That's the core focus for our technical safety team, and we've had some pretty encouraging updates in terms of what we've been able to make work.

所以你有一种直觉和希望，即从数据出发，通过数据解决价值对齐问题，我们可以构建与人类集体中更好的天使对齐的系统，与人类的伦理和道德对齐。换个说法，想想我们如何对齐人类。一个人类婴儿可以成长为邪恶的人或伟大的人，这在很大程度上来自于从数据中学习——孩子成长过程中的反馈，看到积极的例子。我们拥有的唯一一个能够从数据中学习的通用智能的例子，也是与人类价值观对齐并学习价值观的。我认为我们不应该惊讶于我们可以使用同样的技术，或者同样的技术最终会成为我们解决 AGI 价值对齐的方式。

So you have an intuition and a hope that from data, looking at the value alignment problem from data, we can build systems that align with the collective better angels of our nature, aligned with the ethics and morals of human beings. To say it differently, think about how we align humans. A human baby can grow up to be an evil person or a great person, and a lot of that is from learning from data—feedback as a child grows up, seeing positive examples. The only example we have of a general intelligence that is able to learn from data is also aligned with human values and learns values. I think we shouldn't be surprised that we can do the same sorts of techniques, or that the same sorts of techniques end up being how we solve value alignment for AGIs.

让我们再往高处走。我不知道你是否读过《人类简史》，但有一个观点是，作为一个集体，我们人类共同发展，我们持有的观念——在这种背景下没有客观真理，我们只是都同意某些观念并集体持有它们。如果你觉得世界上存在善与恶，你是否觉得大致上有些东西是好的，并且你可以教系统表现得善良？

Let's go even higher. I don't know if you've read the book Sapiens, but there's an idea that as a collective, we human beings develop together, and ideas we hold—there's no objective truth in that context, we just all agree to certain ideas and hold them as a collective. If you have a sense that there is, in the world, good and evil, do you have a sense that to a first approximation, there are some things that are good and that you could teach systems to behave to be good?

我认为这实际上涉及我们的第三个团队，即政策团队。这是人们谈论得远不够多的方面。想象一下，我们构建了超强大的系统，并设法弄清楚了所有机制，让这些系统能做操作者想做的任何事情。最重要的问题变成了：谁是操作者，他们想要什么，这将如何影响其他人？什么是好的，那些价值观是什么——我认为你甚至不需要去那些非常宏大的存在主义层面就能意识到这个问题有多难。你只需看看世界上的不同国家和文化，对世界如何运作以及社会希望如何运作就有非常不同的概念。真正核心的问题非常具体，而且我们还没有现成的答案：你如何创造一个世界，让所有不同的国家——美国、中国、俄罗斯以及其他数百个国家——能够继续以他们认为合适的方式运作，但同时，这些非常强大的系统与人类共存的世界最终会赋予人类更多力量，让人类存在更有意义，让人们更快乐、更富有，能够过上更充实的生活？一旦你拥有了那个非常强大的系统，如何设计那个世界并不显而易见。

I think this actually blends into our third team, which is the policy team. This is the aspect people talk about way less than they should. Imagine we built super-powerful systems that we've managed to figure out all the mechanisms for these things to do whatever the operator wants. The most important question becomes: who's the operator, what do they want, and how is that going to affect everyone else? This question of what is good, what are those values—I think you don't even have to go to those very grand existential places to realize how hard this problem is. You just look at different countries and cultures across the world, and there's a very different conception of how the world works and what kinds of ways society wants to operate. The really core question is very concrete, and it's not a question we have ready answers to: how do you have a world where all the different countries—United States, China, Russia, and hundreds of others—are able to continue to operate in the way they see fit, but the world that emerges with these very powerful systems operating alongside humans ends up being something that empowers humans more, makes human existence more meaningful, and people are happier, wealthier, and able to live more fulfilling lives? It's not obvious how to design that world once you have that very powerful system.

所以如果我们稍微退一步，我们正在进行一场引人入胜的对话。OpenAI 在很多方面是世界上的技术领导者，然而我们却在思考这些重大的存在性问题，这既迷人又非常重要。我认为你是那个领域的领导者，而思考 AI 如何从宏观角度影响社会是一个非常重要的领域。奥斯卡·王尔德说过：“我们都身处阴沟，但有些人仰望星空。”我认为 OpenAI 有一份仰望星空的章程：创造智能，创造通用智能，使其有益、安全且协作。你能告诉我这是如何产生的吗？这样的使命以及创造它的路径是如何形成的？

So if we take a little step back, we're having a fascinating conversation. OpenAI is in many ways a tech leader in the world, and yet we're thinking about these big existential questions, which is fascinating and really important. I think you're a leader in that space, and it's a really important space of just thinking how AI affects society in a big-picture view. Oscar Wilde said, 'We are all in the gutter, but some of us are looking at the stars.' I think OpenAI has a charter that looks to the stars: to create intelligence, to create general intelligence, make it beneficial, safe, and collaborative. Can you tell me how that came about, how a mission like that and the path to creating it came about?

我认为在某种程度上，这确实归结为审视整个格局。如果你想想 AI 的历史，在过去 60 或 70 年里，人们一直在思考这个目标：如果你能自动化人类智力劳动，会发生什么？想象你可以构建一个能够做到这一点的计算机系统。什么变得可能？从科幻小说中，我们有各种反乌托邦的故事，而且越来越多地有像《她》这样的电影，告诉你一些可能更乌托邦的愿景。如果你想想我们从拥有“心灵的自行车”——计算机——所看到的影响，计算机和互联网的影响已经远远超出了任何人真正能够想象的范围。

I think that in some ways, it really boils down to taking a look at the landscape. If you think about the history of AI, for the past 60 or 70 years, people have thought about this goal of what could happen if you could automate human intellectual labor. Imagine you can build a computer system that could do that. What becomes possible? Out of sci-fi, we have stories of various dystopias, and increasingly you have movies like Her that tell you a bit about maybe a more utopian vision. If you think about the impacts we've seen from being able to have bicycles for our minds—computers—the impact of computers and the Internet has far outstripped what anyone really could have imagined.

我认为很明显，如果你能构建出人工智能，它将成为人类有史以来最具变革性的技术。所以归根结底的问题是：有没有一条路？有没有希望？有没有办法构建这样的系统？在过去的六七十年里，人们一次次兴奋，但最终都没能兑现寄予它们的期望。经历了两次 AI 寒冬之后，人们几乎不敢再做梦了。谈论 AGI 在社区里几乎成了禁忌。但我认为人们从 AI 历史中吸取了错误的教训。回顾 1959 年，感知机发布——这是最早的神经网络之一——它被过度炒作。1959 年《纽约时报》有篇文章说感知机有一天能认出人、叫出名字、即时翻译语言。当时的人看了说，这系统什么都做不了，然后花了十年试图诋毁整个感知机方向，并且成功了。所有资金枯竭，人们转向其他方向。80 年代有一次复兴。我一直听说是因为反向传播的发明，但实际上是因为人们建造了更大的计算机。80 年代的文章说，计算能力的民主化意味着你可以运行更大的神经网络。然后人们开始做各种惊人的事情。反向传播算法被发明了，但人们运行的神经网络很小——只有 20 个神经元。用 20 个神经元能学到什么？所以当然得不到好结果。直到 2012 年，这个最简单、最自然的方法——人们在 50 年代甚至 40 年代就提出的方法——突然成了解决问题的最佳方式。我认为深度学习有三个核心特性值得关注。第一是通用性：我们只有很少的深度学习工具——SGD、深度神经网络、也许一些强化学习——但它们解决了大量问题：语音识别、机器翻译、游戏。工具集很小。所以有通用性。第二是能力：你想解决任何这些问题，投入 40 年的计算机视觉研究，换成深度神经网络，它效果更好。第三是可扩展性：一次又一次证明，如果你有更大的神经网络、更多的算力、更多的数据，它会工作得更好。这三个特性加在一起，感觉像是构建通用智能的必要部分。但这并不意味着仅仅扩展我们现有的东西就能得到 AGI——显然还有缺失的部分、缺失的想法，我们需要对推理有答案。但核心是，我们第一次有了一个范式，让我们看到通用智能是可以实现的希望。一旦你相信这一点，其他一切都变得清晰。如果你想象你也许能构建出有史以来最具变革性的技术——时间线仍不确定，但肯定在我们有生之年，可能比人们预期的短得多——你就不再那么关注自己，而是开始思考如何让世界变得更好。你需要考虑实际的问题：如何建立组织，聚集人员和资源，确保人们有动力并准备好去做。然后你开始思考，如果我们成功了，如何确保世界是我们希望存在的那个世界，在罗尔斯的意义上。这就是更广阔的图景。OpenAI 成立于 2015 年，带着那个高层次图景：AGI 可能比人们想象的更早实现，我们需要尽力确保它顺利发展。然后我们花了接下来几年时间弄清楚这意味着什么以及如何去做。通常一家公司从很小开始——联合创始人，构建产品，获得用户，产品市场匹配，然后融资，招人，扩展。然后大公司发现你存在并试图消灭你。对于 OpenAI 来说，基本上就是完全按照这个顺序。

I think it's very clear that if you can build an AI, it will be the most transformative technology that humans will ever create. So what it boils down to is a question: is there a path, is there hope, is there a way to build such a system? For 60 or 70 years, people got excited but ended up not being able to deliver on the hopes pinned on them. After two AI winters, people almost stopped daring to dream. Talking about AGI became almost taboo in the community. But I actually think people took the wrong lesson from AI history. If you look back starting in 1959, when the perceptron was released—one of the earliest neural networks—it was met with massive overhype. The New York Times in 1959 had an article saying the perceptron would one day recognize people, call out their names, instantly translate speech between languages. People at the time looked at this and said, 'This system can't do any of that,' and spent ten years trying to discredit the whole perceptron direction, and succeeded. All the funding dried up, and people went in other directions. In the 80s, there was a resurgence. I'd always heard that was due to the invention of backpropagation, but actually the causality was people building larger computers. Articles from the 80s said the democratization of computing power meant you could run larger neural networks. Then people started doing all these amazing things. The backpropagation algorithm was invented, but the neural nets people were running were tiny—like 20 neurons. What are you supposed to learn with 20 neurons? So of course they weren't able to get great results. It really wasn't until 2012 that this approach—the most simple, natural approach people came up with in the 50s, even in the 40s with Pitts and McCulloch's neuron—suddenly became the best way of solving problems. I think there are three core properties of deep learning worth paying attention to. The first is generality: we have a very small number of deep learning tools—SGD, deep neural net, maybe some RL—and they solve a huge variety of problems: speech recognition, machine translation, game playing. A small set of tools. So there's generality. The second piece is competence: you want to solve any of those problems, throw 40 years of computer vision research at it, replacing the deep neural net, it kind of works better. The third piece is scalability: the one thing shown time and time again is that if you have a larger neural network, more compute, more data, it will work better. Those three properties together feel like essential parts of building a general intelligence. That doesn't just mean scaling up what we have will give us AGI—there are clearly missing pieces, missing ideas, we need answers for reasoning. But the core is that for the first time, we have a paradigm that gives us hope that general intelligence can be achievable. As soon as you believe that, everything else comes into focus. If you imagine you may be able to build the most transformative technology that will ever exist—and the timeline remains uncertain, but certainly within our lifetimes, possibly much shorter than people expect—you stop thinking about yourself so much and start thinking about how to have a world where this goes well. You need to think about the practicalities of building an organization, gathering people and resources, making sure people feel motivated and ready. Then you start thinking about what if we succeed, and how do we make sure the world is the place we want to exist in, in the Rawlsian sense. That's the broader landscape. OpenAI was formed in 2015 with that high-level picture: AGI might be possible sooner than people think, and we need to try our best to make sure it goes well. Then we spent the next couple years figuring out what that means and how to do it. Typically with a company, you start very small—a co-founder, build a product, get users, product-market fit, then raise money, hire people, scale. Then down the road, big companies realize you exist and try to kill you. For OpenAI, it was basically everything in exactly that order.

让我暂停一下。他说了很多，我想赞叹一下 OpenAI 所代表的令人震撼的一面，那就是敢于梦想。我是说，你说得很有力。你让我措手不及，因为我觉得这非常真实。仅仅是敢于梦想以积极、安全的方式创造智能的可能性——甚至只是创造智能本身——就是 AI 社区非常需要的一剂清新催化剂。所以这是起点。好的，那么 OpenAI 的成立呢？

Let me just pause for a second. He said a lot of things, and let me just admire the jarring aspect of what OpenAI stands for, which is daring to dream. I mean, you said it's pretty powerful. You caught me off guard because I think that's very true. The step of just daring to dream about the possibilities of creating intelligence in a positive, safe way—but just even creating intelligence—is a much-needed refreshing catalyst for the AI community. So that's the starting point. Okay, so formation of OpenAI?

当我们开始创办 OpenAI 时，第一个问题是：现在开始一个由最优秀人才组成的实验室是不是太晚了？这是一个真实的问题。那是核心问题。2015 年 7 月有一次晚餐，我们整个时间都在讨论这个。因为想想 AI 当时的状态：它从学术追求转变为工业追求。所以很多最优秀的人都在那些大型研究实验室里，而我们想创办自己的实验室。无论我们能积累多少资源，与大科技公司相比都相形见绌。我们知道这一点。问题是我们是否真的能让这件事起步。你需要临界质量。你不能只靠你和联合创始人构建一个产品；你真的需要五到十个人的团队。

When we were starting OpenAI, the first question we had was: is it too late to start a lab with a bunch of the best people possible? That was an actual question. That was the core question. There was a dinner in July 2015, and that was really what we spent the whole time talking about. Because you think about where AI was: it transitioned from an academic pursuit to an industrial pursuit. So a lot of the best people were in these big research labs, and we wanted to start our own. No matter how much resources we could accumulate, it would pale in comparison to the big tech companies. We knew that. There was a question of whether we could actually get this thing off the ground. You need critical mass. You can't just do you and a co-founder build a product; you really need to have a group of five to ten people.

说到与巨头竞争，我们来谈谈你在成长过程中、在思考如何大规模开发这些系统以参与竞争时遇到的一些棘手问题。你最近成立了 OpenAI LP，一家新的 capped-profit 公司，现在以 OpenAI 的名义运营。所以 OpenAI 现在有了这家正式公司；原来的非营利组织仍然存在，并保留 OpenAI 非营利的名称。你能解释一下这家公司是什么，它成立的目的，以及你是如何做出这个决定的吗？

So speaking of competing with the big players, let's talk about some of the tricky things as you think through this process of growing, of seeing how you can develop these systems at scale that compete. You recently formed OpenAI LP, a new capped-profit company that now carries the name OpenAI. So OpenAI has now this official company; the original nonprofit company still exists and carries the OpenAI nonprofit name. So can you explain what this company is, what the purpose of its creation is, and how did you arrive at the decision?

是的，创建 OpenAI LP，整个实体和 OpenAI LP 作为一个载体，旨在实现确保通用人工智能造福所有人的使命。我们实现这一目标的主要方式是自己尝试构建通用智能，并确保其利益惠及全世界。这是主要途径。如果其他人做到了这一点，我们也可以接受，不一定非得是我们。如果其他人要构建 AGI，并确保利益不会被锁定在一家公司或一个人手中，我们实际上对此没有意见。这些想法都融入了我们的章程，这是描述我们价值观和运营方式的基础文件。但这也深深植根于 OpenAI LP 的结构中。我们设立 OpenAI LP 的方式是，如果我们成功，即我们真正构建了我们试图构建的东西，那么投资者可以获得回报，但这个回报是有上限的。所以如果你从可能创造的价值角度来考虑 AGI，你谈论的是有史以来最具变革性的技术。它将创造比任何现有公司多出几个数量级的价值，而所有这些价值都将归世界所有，在法律上归属于非营利组织以实现其使命。这就是结构。

Yeah, to create OpenAI LP, the whole entity and OpenAI LP as a vehicle is trying to accomplish the mission of ensuring that artificial general intelligence benefits everyone. The main way we're trying to do that is by actually trying to build general intelligence ourselves and make sure the benefits are distributed to the world. That's the primary way. We're also fine if someone else does this, alright? It doesn't have to be us. If someone else is going to build an AGI and make sure that the benefits don't get locked up in one company or one person with one set of people, we're actually fine with that. And so those ideas are baked into our Charter, which is kind of the foundational document that describes our values and how we operate. But it's also really baked into the structure of OpenAI LP. The way we've set up OpenAI LP is that in the case where we succeed, if we actually build what we're trying to build, then investors are able to get a return, but that return is something that is capped. So if you think of AGI in terms of the value that you could really create, you're talking about the most transformative technology ever created. It's going to create orders of magnitude more value than any existing company, and all of that value will be owned by the world, legally titled to the nonprofit to fulfill that mission. So that's the structure.

这个使命很强大，我想大多数人都会同意。这是我们希望 AI 发展的方式。那么你如何将自己与这个使命绑定？你如何确保不会偏离这个使命，确保其他利润驱动的激励不会干扰这个使命？

The mission is a powerful one, and it's one that I think most people would agree with. It's how we would hope AI progresses. So how do you tie yourself to that mission? How do you make sure you do not deviate from that mission, that other incentives that are profit-driven wouldn't interfere with the mission?

这实际上是我们过去几年面临的一个核心问题。因为我们的历史是，第一年我们刚刚起步。我们有一个宏观蓝图，但不知道具体如何实现。真正是在两年前，我们开始意识到，为了构建 AGI，我们需要筹集比作为非营利组织所能筹集的多得多的资金。我的意思是，你谈论的是数十亿美元。所以第一个问题是，你如何做到这一点并忠于这个使命？我们研究了所有现有的法律结构，并得出结论，没有一个完全适合我们想要做的事情。我想这并不太令人惊讶，如果你要做一些疯狂的前所未有的技术，你就必须想出一些疯狂的前所未有的结构来承载它。我们与 OpenAI 内部的人进行了很多讨论，这些人之所以加入，是因为他们非常相信这个使命，思考我们如何实际筹集资源来实现它，同时忠于我们的立场。你必须从真正明确我们的立场开始，我们的价值观是什么，什么对我们真正重要。所以我想说，我们花了大约一年的时间真正编写了 OpenAI 章程。它决定了——如果你看其中的第一条，它说我们预计将需要调动大量资源，但我们将确保最大限度地减少与使命的利益冲突。这种对所有部分的协调一致，是弄清楚如何构建一个能够实际筹集所需资源的公司的最重要一步。

So this was actually a really core question for us for the past couple years. Because the way our history went was that for the first year we were getting off the ground. We had this high-level picture but we didn't know exactly how we wanted to accomplish it. Really two years ago is when we first started realizing that in order to build AGI, we're just going to need to raise way more money than we can as a nonprofit. I mean, you're talking many billions of dollars. So the first question is how are you supposed to do that and stay true to this mission? We looked at every legal structure out there and concluded none of them were quite right for what we wanted to do. I guess it shouldn't be too surprising if you're going to do something like crazy unprecedented technology that you're gonna have to come up with some crazy unprecedented structure to do it in. A lot of our conversation was with people at OpenAI, the people who really joined because they believe so much in this mission, thinking about how do we actually raise the resources to do it and also stay true to what we stand for. The place you got to start is to really align on what is it that we stand for, what are those values, what's really important to us. So I'd say that we spent about a year really compiling the OpenAI Charter. That determines, if you even look at the first line item in there, it says that we expect we're gonna have to marshal huge amounts of resources, but we're going to make sure that we minimize conflicts of interest with the mission. That kind of aligning on all of those pieces was the most important step towards figuring out how do we structure a company that can actually raise the resources to do what we need to do.

我想创建 OpenAI LP 的决定非常困难，正如你提到的，进行了长达一年的讨论，有各种不同的想法，也许 OpenAI 内部有反对者，有各种不同的路径可以选择。这些担忧是什么？考虑了哪些不同的路径？做出这个决定的过程是怎样的？

I imagine the decision to create OpenAI LP was a really difficult one, and there was a lot of discussions as you mentioned for a year, and there were different ideas, perhaps detractors within OpenAI, sort of different paths that you could have taken. What were those concerns? What were the different paths considered? What was that process of making that decision like?

是的。如果你看 OpenAI 章程，里面几乎嵌入了两条路径。一条是：我们主要试图自己构建 AGI，但我们也接受别人来做。这对一家公司来说很奇怪，真的很有趣。是的，有竞争的元素，你确实希望自己是做到的那一个，但同时你也接受别人做到。我们稍后会讨论这个权衡。这真的很有趣。我认为这是我们在设计 OpenAI LP 时的核心张力，也是 OpenAI 战略的核心：你如何确保自己有机会成为主要参与者，这需要建立组织、筹集大量资源，并真正有意志去执行一个非常非常困难的愿景？你需要真正长期投入，承受很多痛苦和风险。通常要做到这一点，你只需要采用创业心态，对吧？你考虑如何执行，每个人都从竞争角度出发。但你还有第二个角度：真正的使命不是让 OpenAI 构建 AGI，真正的使命是让 AGI 为人类带来好的结果。那么你如何采取所有这些初步行动，同时确保不关闭那些实际上积极且能实现使命的结果的大门？我认为这是一个非常微妙的平衡。完全走向一个方向或另一个方向显然不是正确答案。所以我认为，即使在我们谈论和思考 OpenAI 的方式上，我脑海中始终有一件事：确保我们不只是说 OpenAI 的目标是构建 AGI，实际上它要广泛得多。首先，不仅仅是 AGI，而是安全的 AGI——这非常重要。其次，我们的目标不是成为构建它的人；我们的目标是确保它对世界有益。所以我认为，弄清楚如何平衡所有这些，并让人们真正坐到谈判桌前，编写一份包含所有这些内容的单一文件，并非易事。所以这里的部分挑战是……

Yeah. So if you look at the OpenAI Charter, there are almost two paths embedded within it. There is: we are primarily trying to build AGI ourselves, but we're also okay if someone else does it. And this is a weird thing for a company, it's really interesting. Yeah, there is an element of competition that you do want to be the one that does it, but at the same time you're okay if somebody else does. We'll talk about that trade-off a little bit. That's really interesting. I think this was the core tension as we were designing OpenAI LP, and really the OpenAI strategy: how do you make sure that you have a shot at being a primary actor, which really requires building an organization, raising massive resources, and really having the will to go and execute on some really, really hard vision? You need to really sign up for a long period to go and take on a lot of pain and a lot of risk. To do that normally you just import the startup mindset, right? You think about how to execute, everyone gives this very competitive angle. But you also have the second angle of saying that the true mission isn't for OpenAI to build AGI; the true mission is for AGI to go well for humanity. So how do you take all of those first actions and make sure you don't close the door on outcomes that would actually be positive and fulfill the mission? I think it's a very delicate balance. Going 100% one direction or the other is clearly not the correct answer. So I think that even in terms of just how we talk about OpenAI and think about it, there's one thing that's always in the back of my mind: to make sure that we're not just saying OpenAI's goal is to build AGI, that it's actually much broader than that. First of all, it's not just AGI, it's safe AGI—that's very important. But secondly, our goal isn't to be the ones to build it; our goal is to make sure it goes well for the world. So I think figuring out how to balance all of those and to get people to really come to the table and compile a single document that encompasses all of that wasn't trivial. So part of the challenge here...

你的使命，我认为是美丽、赋权，是研究界和所有思考 AI 的人们的希望灯塔。所以你的决策比普通盈利公司受到更多审视。你在制定章程和运营方式时感受到这种负担了吗？

Your mission is, I would say, beautiful, empowering, and a beacon of hope for people in the research community and just people thinking about AI. So your decisions are scrutinized more than I think a regular profit-driven company. Do you feel the burden of this in the creation of the Charter and just in the way you operate?

是的。那么为什么要通过制定这样的章程来承担负担呢？为什么不保持低调？我的意思是，这归根结底是使命，对吧？我和其他所有人在这里，是因为我们认为这是最重要的使命。敢于梦想。

Yes. So why do you lean into the burden by creating such a charter? Why not keep it quiet? I mean, it just boils down to the mission, right? I'm here and everyone else is here because we think this is the most important mission. Dare to dream, all right.

那么你怎么看？作为一家盈利公司，你能造福世界或创造有益的 AGI 系统吗？从我的角度看，我不明白为什么利润会干扰对社会的积极影响。我不明白为什么谷歌（主要靠广告赚钱）不能同时造福世界，或者其他公司如 Facebook。我不明白为什么这些必须相互干扰。在我看来，利润并不是影响公司影响力的因素。影响公司影响力的是章程、文化、内部的人，而利润只是滋养这些人的东西。你对此有什么看法？

So what do you think? You can be good for the world or create an AGI system that's good when you're a for-profit company? From my perspective, I don't understand why profit interferes with positive impact on society. I don't understand why Google, that makes most of its money from ads, can't also do good for the world, or other companies like Facebook. I don't understand why those have to interfere. You know, profit isn't the thing, in my view, that affects the impact of a company. What affects the impact of the company is the Charter, is the culture, is the people inside, and profit is the thing that just fuels those people. So what are your views there?

是的，我认为这是一个非常好的问题，其中涉及人类社会中一些长期存在的辩论。我的思考方式是：想想世界上最有影响力的非营利组织和最有影响力的营利组织。列出营利组织要容易得多，对吧？我认为这里有一些真实的道理：我们建立的体系，即当今世界组织的方式，确实允许巨大的影响力。部分原因是营利组织能够自我维持并依靠自身动力发展。我认为这是一件非常强大的事情。但当我们没有正确设置护栏时，就会引发问题。想想砍伐雨林的伐木公司——那非常糟糕，我们不希望那样。而且我特别感兴趣的是，如何从营利公司中获得积极利益的问题，与如何从 AGI 中获得积极利益非常相似。你有一个非常强大的系统，比任何人类都强大，在某些方面是自主的，在许多方面超人类，你必须设置护栏来确保好的结果。但当你这样做时，好处是巨大的。所以当我思考非营利与营利时，我认为非营利组织做得不够——它们非常纯粹，但很难做成事情。而营利组织在某些方面做得太多，但如果以正确的方式塑造，实际上可以非常积极。因此，OpenAI 选择了一条中间道路。现在，我认为非常重要的一点是，我们对 OpenAI 的看法是：在 AGI 真正实现的世界里，在我们成功的情况下，我们将构建有史以来最具变革性的技术。我们将创造的价值将是天文数字。因此，我们设定的上限将只是我们所创造价值的一小部分，而返还给投资者和员工的价值看起来与一个相当成功的初创公司相似。这正是我们优化的目标——在成功的情况下，确保我们创造的价值不会被锁定。我预计在其他营利公司中，也有可能做到类似的事情，但如何正确做到并不明显。作为一家营利公司，你对股东负有大量受托责任，有些决定你根本无法做出。在我们的结构中，我们设定为对章程负有受托责任——我们总是能够做出对章程正确的决定，即使这会损害我们自己的利益相关者。所以当我思考什么真正重要时，其实不在于非营利与营利。

Yeah, I think that's a really good question, and there are some long-standing debates in human society wrapped up in it. The way I think about it is just think about what are the most impactful nonprofits in the world, what are the most impactful for-profits in the world. It's much easier to list the for-profits, right? And I think there's some real truth here that the system we set up, the system for how today's world is organized, is one that really allows for huge impact. Part of that is that for-profits are self-sustaining and able to build on their own momentum. I think that's a really powerful thing. But when it turns out that we haven't set the guardrails correctly, it causes problems. Think about logging companies that deforest the rainforest—that's really bad, we don't want that. And it's actually really interesting to me that this question of how to get positive benefits out of a for-profit company is very similar to how to get positive benefits out of an AGI. You have this very powerful system, more powerful than any human, kind of autonomous in some ways, superhuman on a lot of axes, and somehow you have to set the guardrails to get good to happen. But when you do, the benefits are massive. So when I think about nonprofit vs. for-profit, I think not enough happens in nonprofits—they're very pure, but it's just hard to do things. In for-profits, in some ways, too much happens, but if shaped in the right way, it can actually be very positive. So with OpenAI, we're picking a road in between. Now, the thing I think is really important to recognize is that the way we think about OpenAI is that in the world where AGI actually happens, where we are successful, we build the most transformative technology ever. The amount of value we're going to create will be astronomical. So then the cap that we have will be a small fraction of the value we create, and the amount of value that goes back to investors and employees looks pretty similar to what would happen in a pretty successful startup. That's really the case we're optimizing for—in the success case, making sure the value we create doesn't get locked up. I expect that in other for-profit companies, it's possible to do something like that, but it's not obvious how to do it right. As a for-profit company, you have a lot of fiduciary duty to your shareholders, and there are certain decisions you just cannot make. In our structure, we've set it up so that we have a fiduciary duty to the Charter—we always get to make the decision that is right for the Charter, even if it comes at the expense of our own stakeholders. So when I think about what's really important, it's not really about nonprofit vs. for-profit.

营利性，这实际上是一个问题：如果你构建了 AGI，人类进入新时代，谁受益，谁的生活变得更好？我认为真正重要的是有一个答案是“每个人”，这是宪章的核心方面之一。所以人们担心的一个问题是，不仅对 OpenAI，还有谷歌、Facebook、亚马逊，任何产生这种规模影响的公司：我们如何避免，正如你的宪章所说，让 AGI 被用来过度集中权力？为什么像 OpenAI 这样的公司不把 AGI 系统的所有权力都留给自己？宪章——它如何在日常中实现？

For-profit, it's really a question of if you build AGI and humanity enters a new age, who benefits, whose lives are better? I think what's really important is to have an answer that is everyone, which is one of the core aspects of the Charter. So one concern people have not just with OpenAI but with Google, Facebook, Amazon, anybody creating impact at that scale is: how do we avoid, as your Charter says, enabling the use of or AGI to unduly concentrate power? Why would a company like OpenAI not keep all the power of an AGI system to itself? The Charter—how does it actualize itself in day-to-day?

我们构建公司的方式是，决定 OpenAI 行动的权力最终归属于非营利组织的董事会。董事会有一些限制，你可以在 OpenAI LP 的博客文章中读到，但董事会实际上是 OpenAI LP 的治理机构，董事会有责任履行非营利组织的使命。这就是我们如何把所有这些联系在一起。现在，日常执行的人是员工。这里掌握技术王国钥匙的人如何实现这一点，就像任何公司的价值观一样：你需要那些真正相信使命和宪章的人，他们愿意采取可能对自己不利但对宪章有利的行动。这已经融入了文化，我们必须努力保持它。这是我们招聘的重要部分。所以这里有员工可以站出来说，‘等等，这完全违背了我们的立场。’

The way we structure the company is so that the power dictating the actions that OpenAI takes ultimately rests with the board of the nonprofit. The board is set up with certain restrictions you can read about in the OpenAI LP blog post, but effectively the board is the governing body for OpenAI LP, and the board has a duty to fulfill the mission of the nonprofit. That's how we thread all these things together. Now, day-to-day, the people who are executing are the employees. The way people here who have the keys to the technical kingdom actualize that is like any company's values: you need people who are here because they really believe in the mission and the Charter, and are willing to take actions that may be worse for them but better for the Charter. That's baked into the culture, and we have to work to preserve it. It's an important part of how we hire. So there are people here who can speak up and say, 'Hold on, this is totally against what we stand for.'

文化上的监督，对吧？

Cultural eyes, yeah?

是的，当然。我认为这是我们运营中非常重要的一部分。即使在最初设计宪章和 OpenAI LP 时，也有很多与员工的对话，很多时候员工会说，‘等等，这似乎方向不对，我们来谈谈。’作为一家小公司，我们非常独特的一点是，如果你在一家大型科技巨头，一个认同公司价值观的员工很难去找 CEO 说，‘我认为我们做错了。’在谷歌，员工围绕 Maven 等项目有过一些集体行动。但在这里，任何人都可以很容易地把我拉到一边，把 Sam 拉到一边，把 Ilya 拉到一边，人们经常这样做。

Yeah, for sure. I think that's a pretty important part of how we operate. Even in designing the Charter and OpenAI LP in the first place, there was a lot of conversation with employees, and many times employees said, 'Wait, this seems like it's coming in the wrong direction, let's talk about it.' One thing that's very unique about us as a small company is that if you're at a massive tech giant, it's hard for an aligned employee to go talk to the CEO and say, 'I think we're doing this wrong.' At Google, there have been some collective actions from employees around things like Maven. But here, it's super easy for anyone to pull me aside, pull Sam aside, pull Ilya aside, and people do it all the time.

宪章中一个有趣的想法是在 AGI 开发的后期阶段从竞争转向合作。竞争与合作之间的这种舞蹈真的很有趣。你怎么看？

One of the interesting things in the Charter is this idea of switching from competition to collaboration in late-stage AGI development. It was really interesting, this dance between competition and collaboration. How do you think about that?

假设你实际上能解决 AGI 开发的技术问题，我认为在如何实际部署并使其顺利进行方面会有两个关键问题。第一个是构建第一个 AGI 之前的冲刺阶段。你看看自动驾驶汽车是如何开发的，那是一场竞争性竞赛。在竞争性竞赛中，总是有巨大的压力要牺牲安全。这是我们非常担心的一点：多个团队都在想办法实现目标，但如果选择更慢但更安全的路径，我们就会输，所以我们会选择快路径。我们越能让自己不陷入那种竞争性竞赛——也就是说，如果竞赛正在进行，而别人领先，我们不会试图超越，而是会与他们合作。只要他们试图实现我们的使命，我们就会帮助他们成功，那就没问题。我们不必自己构建 AGI。我认为这是我们一个非常重要的承诺，但这不能只是单方面的。其他认真构建 AGI 的参与者也需要做出类似的承诺。在某种程度上，如果每个人都相信 AGI 应该造福所有人，那么实际上由哪家公司构建并不重要，我们都应该担心的是，我们为了达到目标而过度竞争，导致出了问题。

Assuming you can actually do the technical side of AGI development, I think there will be two key problems with figuring out how to actually deploy it and make it go well. The first is the run-up to building the first AGI. You look at how self-driving cars are being developed, and it's a competitive race. What always happens in a competitive race is that you have huge amounts of pressure to get rid of safety. That's one thing we're very concerned about: multiple teams figuring out we can actually get there, but if we took the slower path that is more guaranteed to be safe, we will lose, so we're going to take the fast path. The more we can both be in a position where we don't generate that competitive race—where we say if the race is being run and someone else is further ahead, we're not going to try to leapfrog, we're going to actually work with them. We will help them succeed as long as what they're trying to do is to fulfill our mission, then we're good. We don't have to build AGI ourselves. I think that's a really important commitment from us, but it can't just be unilateral. Other players who are serious about building AGI need to make similar commitments. To the extent that everyone believes AGI should benefit everyone, it actually shouldn't matter which company builds it, and we should all be concerned about the case where we race so hard to get there that something goes wrong.

你认为政府在这个领域从研究到开发再到 AGI 开发的早期和后期阶段，在制定政策和规则方面扮演什么角色？

What role do you think government has in setting policy and rules about this domain, from research to development to early stage to late-stage AGI development?

首先，政府成为答案的一部分非常重要。归根结底，我们谈论的是构建将塑造世界运作方式的技术，政府需要参与其中。这就是为什么我们做了多次国会证词并与立法者互动。现在，我们向他们传达的主要信息是，现在不是监管的时候，而是测量的时候。我们的主要政策建议是，人们——政府也经常通过像 NIST 这样的机构这样做——花时间弄清楚技术在哪里，发展有多快，并了解应该期待什么。所以今天，答案实际上是关于测量。我认为将来会有时间和地点发生变化，但很难准确预测这个轨迹应该是什么样的。

First of all, it's really important that the government is part of the answer. At the end of the day, we're talking about building technology that will shape how the world operates, and there needs to be government as part of that. That's why we've done a number of congressional testimonies and interact with lawmakers. Right now, a lot of our message to them is that it's not the time for regulation, it's the time for measurement. Our main policy recommendation is that people—and the government does this all the time with bodies like NIST—spend time trying to figure out where the technology is, how fast it's moving, and become literate and up to speed with respect to what to expect. So today, the answer is really about measurement. I think there will be a time and place where that will change, but it's a little hard to predict exactly what that trajectory should look like.

所以会有一个点，美国联邦政府介入并帮助成为——我不想说房间里的大人——确保有严格的规则，也许是保守的规则，没有人能跨越？

So there will be a point where regulation—federal in the United States—the government steps in and helps be the, I don't want to say the adult in the room, to make sure that there are strict rules, maybe conservative rules, that nobody can cross?

嗯，我认为可能有这么两种……

Well, I think there's this kind of maybe two...

所以今天对于狭义 AI 应用，我认为已经有现有的机构负责并且应该负责监管。比如，以自动驾驶汽车为例，你希望国家公路交通安全管理局在这方面做得很好。这很合理，对吧？基本上我们在说的是，我们将拥有这些技术系统，它们将执行人类已经做得很好的应用。我们已经有了思考这些标准和安全的方法。所以我认为实际上赋予这些现有监管机构权力也非常重要。然后对于 AGI，你知道，会有一个点我们会有更好的答案。我认为可能类似的方法，先测量然后开始思考规则应该是什么。我认为非常重要的是我们不要过早地扼杀进步。我认为很容易扼杀这个新兴领域，这是必须避免的。但我认为正确的方式不是说我们只管往前冲，不涉及所有其他利益相关者。

So today with narrow AI applications, I think there are already existing bodies that are responsible and should be responsible for regulation. You think about, for example, with self-driving cars, you want the National Highway Traffic Safety Administration to be very good at that. That makes sense, right? Basically what we're saying is that we're going to have these technological systems that are going to be performing applications that humans already do great. We already have ways of thinking about standards and safety for those. So I think actually empowering those regulators today is also pretty important. And then I think for AGI, you know, there's going to be a point where we'll have better answers. And I think that maybe a similar approach of first measurement and start thinking about what the rules should be. I think it's really important that we don't prematurely squash progress. I think it's very easy to kind of smother the budding field, and I think that's something to really avoid. But I don't think the right way of doing it is to say let's just try to blaze ahead and not involve all these other stakeholders.

所以你最近发表了一篇关于 GPT-2 语言模型的论文，但没有发布完整模型，因为你担心这种模型可用性可能带来的负面影响。这个决定本身之外，非常有趣的是它在社会层面引发的讨论和话语。所以这方面很吸引人。但首先，如果你考虑具体细节，你预见到哪些负面影响，当然还有哪些正面影响？

So you've recently released a paper on GPT-2 language modeling but did not release the full model because you have concerns about the possible negative effects of the availability of such a model. It's outside of just that decision, it's super interesting because of the discussion at a societal level, the discourse it creates. So it's fascinating in that aspect. But if you think about the specifics here at first, what are some negative effects that you envisioned, and of course what are some of the positive effects?

是的，所以再次，我想退一步看，我们对 GPT-2 的看法是，在语言建模方面，我们显然处于一个轨迹上，我们扩大模型规模，得到质量上更好的性能。对吧，GPT-2 本身实际上只是我们去年 6 月发布的一个模型的放大版，对吧？我们只是以更大的规模运行它，得到了这些结果，我们突然开始写出连贯的散文，这是我们以前从未见过的。我们现在在做什么？嗯，我们要把 GPT-2 扩大 10 倍、100 倍、1000 倍，我们不知道会得到什么。所以很明显，我们去年 6 月发布的模型，我认为它有点像一个好的学术玩具。我们不认为它真的能有负面应用，或者说人们能够玩它的正面影响远远大于可能的危害。你快进到不是 GPT-2 而是 GPT-20，想想那会是什么样子，我认为能力将是实质性的。所以如果在两者之间需要一个点，你说这是我们划清界限的地方，我们需要开始考虑安全方面。我认为对于 GPT-2，我们本可以走任何一条路。事实上，当我们内部讨论时，我们有很多利弊，不清楚哪个更重。我认为当我们宣布嘿，我们决定不发布这个模型时，有很多讨论，各种人说很明显你应该发布它，其他人说很明显你不应该发布它。我认为这几乎定义性地意味着保留它是正确的决定，对吧？如果有争议，如果不清楚某件事是否有益，你可能应该默认谨慎。所以我认为我们思考的整体图景是，这个决定本可以走任何一条路。两个方向都有很好的论点。但对于未来的模型，可能比你预期的更早，因为扩大这些东西不需要那么长时间，那些模型你肯定不想发布到野外。所以我认为我们几乎把这看作一个测试案例，看看我们能否设计出如何让一个社会或系统从没有负责任披露的概念，仅仅因为安全原因不发布某事的想法是陌生的，到一个世界你说好吧，我们有一个强大的模型，至少让我们想想它，让我们经历一些过程。你想想安全社区，他们花了很长时间设计负责任披露，对吧？你想想这个问题，嗯，我有一个安全漏洞，我把它发给公司，公司试图起诉我或者只是忽略它，我该怎么办？所以替代方案是哦，我总是发布你的漏洞，那似乎也不好。所以这真的花了很长时间，而且比任何个人都大，实际上是关于建立整个社区，相信好吧，我们会有这个过程，你把它发给公司，如果他们在一定时间内不行动，那么你可以公开，你不是坏人，你做了正确的事。我认为在 AI 中，对 GPT-2 的部分反应恰恰证明我们没有任何这个概念。所以这是高层次的图景。所以我认为这是一个非常重要的举动，我们本可以推迟到 GPT-3，但我很高兴我们在 GPT-2 上做了。所以现在你看看 GPT-2 本身，想想实质内容，好吧，潜在的负面应用是什么？所以你有这个在互联网上训练的模型，它也会有很多有偏见的数据，很多非常冒犯的内容。你可以让它为你生成任何主题的内容，对吧？你给它一个提示，它就会开始写，它写的内容就像你在互联网上看到的，甚至在其生成内容中间说广告。你想想生成假新闻或滥用内容的可能性。看到人们用我们发布的较小版本 GPT-2 做了什么很有趣。人们做了像尝试生成，你知道，拿我自己的 Facebook 消息历史生成更多像我一样的 Facebook 消息，人们生成假政治家内容。有很多事情你至少必须思考，这对世界有好处吗？另一面是，我认为有很多我们真正想看到的很棒的应用程序，比如创意应用。如果科幻作家可以用这个工具想出酷点子，那似乎很棒。如果我们能通过使用这些工具写出更好的科幻小说，而且我们实际上...

Yeah, so again, I think to zoom out, the way that we thought about GPT-2 is that with language modeling, we are clearly on a trajectory right now where we scale up our models and we get qualitatively better performance. Right, GPT-2 itself was actually just a scale-up of a model that we'd released in the previous June, right? And we just ran it at much larger scale and we got these results where we're suddenly starting to write coherent prose, which was not something we'd seen previously. And what are we doing now? Well, we're gonna scale up GPT-2 by 10x, by 100x, by 1000x, and we don't know what we're going to get. And so it's very clear that the model that we released last June, I think it's kind of like a good academic toy. It's not something that we think can really have negative applications, or in the sense that the positive of people being able to play with it far far outweighs the possible harms. You fast forward to not GPT-2 but GPT-20, and you think about what that's gonna be like, and I think that the capabilities are going to be substantive. And so if there needs to be a point in between the two where you say this is something where we are drawing the line and that we need to start thinking about the safety aspects. And I think for GPT-2, we could have gone either way. And in fact, when we had conversations internally, we had a bunch of pros and cons, and it wasn't clear which one outweighed the other. And I think that when we announced that hey, we decided not to release this model, then there was a bunch of conversation where various people said it's so obvious that you should have just released it, other people said it's so obvious you should not have released it. And I think that that almost definitionally means that holding it back was the correct decision, right? If it's controversial, if it's not obvious whether something is beneficial or not, you should probably default to caution. And so I think that the overall landscape for how we think about it is that this decision could have gone either way. There are great arguments in both directions. But for future models down the road, and possibly sooner than you'd expect because scaling these things up doesn't have to take that long, those ones you're definitely not going to want to release into the wild. And so I think that we almost view this as a test case to see can we even design how you have a society or how you have a system that goes from having no concept of responsible disclosure, where the mere idea of not releasing something for safety reasons is unfamiliar, to a world where you say okay, we have a powerful model, let's at least think about it, let's go through some process. And you think about the security community, it took them a long time to design responsible disclosure, right? You think about this question of well, I have a security exploit, I send it to the company, the company tries to prosecute me or just ignores it, what do I do? And so the alternatives of oh, I just always publish your exploits, that doesn't seem good either. And so it really took a long time and it was bigger than any individual, it was really about building the whole community that believed that okay, we'll have this process where you send it to the company, if they don't act in a certain time then you can go public and you're not a bad person, you've done the right thing. And I think that in AI, part of the response to GPT-2 just proves that we don't have any concept of this. So that's the high level picture. And so I think this was a really important move to make, and we could have maybe delayed it for GPT-3, but I'm really glad we did it for GPT-2. And so now you look at GPT-2 itself and you think about the substance of okay, what are potential negative applications? So you have this model that's been trained on the Internet, which is also going to be a bunch of very biased data, a bunch of very offensive content. And you can ask it to generate content for you on basically any topic, right? You just give it a prompt and it'll just start writing, and it writes content like you see on the internet, even down to like saying advertisement in the middle of some of its generations. And you think about the possibilities for generating fake news or abusive content. And it's interesting seeing what people have done with the smaller version of GPT-2 that we released. People have done things like try to generate, you know, take my own Facebook message history and generate more Facebook messages like me, and people generating fake politician content. There's a bunch of things there where you at least have to think, is this going to be good for the world? There's the flip side, which is I think that there's a lot of awesome applications that we really want to see, like creative applications. If you have sci-fi authors that can work with this tool and come up with cool ideas, that seems awesome. If we can write better sci-fi through the use of these tools, and we've actually...

很多人写信问我们，'嘿，我们能不能把它用于各种创意应用？'所以积极的一面其实很容易想象。常见的 NLP 应用很有趣，但让我们深入探讨。想象一个世界，看看推特，那里充斥着假新闻，但越来越智能的机器人能够传播有趣、复杂且有效的信息，淹没我们这些有原创思想的普通人，这很有意思。那么你对 GPT-2 这个世界的看法是什么？我们该如何思考？这有点像 50 年代试图描述互联网或智能手机。你对那个世界怎么看？信息的本质。一种可能性是，我们总是试图设计系统来区分机器人和人类，并且会成功，这样我们就能验证自己仍然是人类。另一种世界是，我们接受我们正淹没在假新闻海洋中的事实，并学会在其中游泳。

We had a bunch of people write in to us asking, 'Hey, can we use it for a variety of different creative applications?' So the positive is actually pretty easy to imagine. The usual NLP applications are really interesting, but let's go there. It's kind of interesting to think about a world where you look at Twitter, where there's just fake news but smarter and smarter bots being able to spread interesting, complex, and working information that floods out us regular human beings with our original thoughts. So what are your views of this world with GPT-2? How do we think about it? It's like one of those things about in the 50s trying to describe the internet or the smartphone. What do you think about that world? The nature of information. One possibility is that we'll always try to design systems that identify robot versus human, and we'll do so successfully, and so we will authenticate that we're still human. The other world is that we just accept the fact that we're swimming in a sea of fake news and just learn to swim there.

你见过那个流行的表情包吗？一个机器眼睛，带着物理手臂和笔，点击'我不是机器人'按钮。我认为事实是，试图区分机器人和人类最终是一场必败之战。

Have you ever seen the popular meme of a robot eye with a physical arm and pen clicking the 'I am not a robot' button? I think the truth is that trying to distinguish between robot and human is a losing battle ultimately.

你认为这是一场必败之战？

You think it's a losing battle?

我认为最终这是一场必败之战。就内容和你能采取的行动而言，想想验证码的发展。验证码曾经非常简单：你有一张图片，我们的 OCR 很差，你加入一些干扰，人类能识别出来，而 AI 系统不能。今天，我几乎都做不了验证码了。我认为这就是趋势。验证码只是某个时期的产物。随着 AI 系统变得更强大，存在一些人类能力可以通过非常简单的自动化方式测量，而 AI 却无法做到。我认为这只是一场越来越艰难的技术战斗。但并非所有希望都破灭了。想想我们如何已经验证自己。我们有像美国社会安全号码这样的系统，有识别个人身份的方法，将现实世界身份与数字身份绑定。这似乎是朝着验证内容来源而非内容本身迈出的一步。但这存在问题：在一个只有通过查看内容来源才能信任内容的世界里，你如何拥有隐私和匿名性？建立良好的声誉网络可能是一个可能的解决方案。但这个问题并不显而易见。也许比我们想象的更快，我们将进入一个世界，今天我经常读到一条推文，心想'我觉得这是一个真人写的'，或者'我觉得这不真实'。我有点评判内容。在未来，情况将不再如此。例如，FCC 关于网络中立性的评论：后来发现数百万条是自动生成的，研究人员能够使用各种统计技术来检测。在一个这些统计技术不存在的世界里，你该怎么办？根本无法区分人类和 AI。事实上，最有说服力的论点是由 AI 写的。这不再是科幻了。GPT-2 可以提出一个很好的论点，说明为什么回收对世界有害。你读到它，会想'嗯，你说得对。'

I think it's a losing battle ultimately. In terms of the content and the actions you can take, think about how CAPTCHAs have gone. CAPTCHAs used to be very nice and simple: you have this image, our OCR is terrible, you put a couple of artifacts in it, humans are gonna be able to tell what it is, an AI system wouldn't be able to. Today, I can barely do CAPTCHAs. I think that this is just where we're going. CAPTCHAs were a moment in time thing. As AI systems become more powerful, there are human capabilities that can be measured in a very easy automated way that the AIs will not be capable of. I think it's just an increasingly hard technical battle. But it's not that all hope is lost. Think about how we already authenticate ourselves. We have systems like social security numbers in the US, ways of identifying individual people and having real-world identity tied to digital identity. That seems like a step towards authenticating the source of content rather than the content itself. Now there are problems with that: how can you have privacy and anonymity in a world where the only way you can trust content is by looking at where it comes from? Building out good reputation networks might be one possible solution. But yeah, I think this question is not an obvious one. Maybe sooner than we think, we'll be in a world where today I often read a tweet and think, 'I feel like a real human wrote this,' or 'I don't feel like this is genuine.' I feel like I judge the content a little bit. In the future, it just won't be the case. For example, the FCC comments on net neutrality: it came out later that millions of those were auto-generated, and researchers were able to do various statistical techniques to detect that. What do you do in a world where those statistical techniques don't exist? It's just impossible to tell the difference between humans and AIs. In fact, the most persuasive arguments are written by AI. That stuff is not sci-fi anymore. You have GPT-2 making a great argument for why recycling is bad for the world. You read that and think, 'Huh, you're right.'

这很有趣。我的意思是，最终归结为物理世界是证明的最后前沿。你基本上说，人与人之间的网络，人类在物理世界中为人类担保，认证就在那里结束。如果我必须让你验证，证明我怎么知道你不是机器人，你怎么知道我不是机器人？我认为到目前为止，在这个领域，我们刚刚进行的对话，我们做的物理动作，是我们和 AI 系统之间最大的差距，就是物理关系。所以也许那是最后的前沿。

That's quite interesting. I mean, ultimately it boils down to the physical world being the last frontier of proving. You said basically networks of people, humans vouching for humans in the physical world, and somehow the authentication ends there. If I had to ask you to authenticate, prove how do I know you're not a robot and how do you know I'm not a robot? I think that so far, in this space, this conversation we just had, the physical movements we did, is the biggest gap between us and AI systems is the physical relation. So maybe that's the last frontier.

另一个问题：为什么解决这个问题很重要？哪些方面对我们真正重要？我认为我们最终会专注于我们真正想从知道是否在和人类交谈中得到什么。这归结为身份。未来的互联网，我预计，将会有很多智能体在那里与你互动。但'这是一个真实的有血有肉的人类还是一个自动化系统'这个问题将变得不那么重要。

Here's another question: why is solving this problem important? What aspects are really important to us? I think that probably where we'll end up is we'll hone in on what do we really want out of knowing if we're talking to a human. This comes down to identity. The internet of the future, I expect, will have lots of agents out there that will interact with you. But the question of 'is this a real flesh-and-blood human or is this an automated system' will become less important.

让我们深入探讨。GPT-2 令人印象深刻。看看 GPT-2：为什么我所有的朋友都是 GPT-2 这么糟糕？为什么在互联网上只与人类互动如此重要？为什么我们不能生活在一个想法可以来自基于人类数据训练的模型的世界？

Let's actually go there. GPT-2 is impressive. Let's look at GPT-2: why is it so bad that all my friends are GPT-2? Why is it so important on the internet to interact with only human beings? Why can't we live in a world where ideas can come from models trained on human data?

我认为这实际上是一个非常有趣的问题。这又回到了你如何想象一个拥有新技术的世界？我认为重要的一点是诚实。如果，几乎以图灵测试的方式，有 AI 假装成人类并欺骗你，那感觉是件坏事。我们感到自己掌控环境、理解我们在和谁互动，以及它是 AI 还是人类，这一点非常重要，我们不应该被欺骗。但另一方面：我能和 AI 进行像和人类一样有意义的互动吗？我认为你可以转向科幻。电影《她》是提出这个问题的绝佳例子。我非常喜欢《她》的一点是，它几乎一开始就问人类与虚拟关系有多有意义。然后你有一个与 AI 建立关系的人类，你真的开始被吸引进去。你所有的情感按钮都被触发，就像电话那头有一个真实的人类一样。所以我认为我们可以进行有意义的互动。如果有一个有趣的笑话，某种感觉……

I think this is actually a really interesting question. It comes back to how do you even picture a world with some new technology? One thing I think is important is honesty. If you have, almost in the Turing test style, AIs that are pretending to be humans and deceiving you, that feels like a bad thing. It's really important that we feel like we're in control of our environment, that we understand who we're interacting with, and if it's an AI or a human, that's not something we're being deceived about. But the flipside: can I have as meaningful an interaction with an AI as I can with a human? I think you can turn to sci-fi. 'Her' is a great example of asking this very question. One thing I really love about 'Her' is it really starts out almost by asking how meaningful human-virtual relationships are. Then you have a human who has a relationship with an AI, and you really start to be drawn into that. All of your emotional buttons get triggered in the same way as if there was a real human on the other side of that phone. So I think that we can have meaningful interactions. If there's a funny joke, some sense...

内容是由人类还是 AI 写的并不重要，但我们应该划清界限的是欺骗。我认为，只要我们还在思考——我们为什么要构建 AI 系统？我们构建它们的目的是为了改善人类生活，让人类能做更多事情，让人类感到更充实。如果能构建出这样的 AI 系统，我举双手赞成。

It doesn't really matter if it was written by a human or an AI, but what we should draw hard lines on is deception. I think that as long as we're in a world where—why do we build AI systems at all? The reason we want to build them is to enhance human lives, to make humans able to do more things, to have humans feel more fulfilled. If we can build AI systems that do that, sign me up.

语言建模这个过程——你认为它能带我们走多远？我们来看看电影《她》。你认为像图灵测试所定义的那种基于对话的自然语言交流，能通过无监督语言建模实现吗？

The process of language modeling—how far do you think it can take us? Let's look at the movie Her. Do you think a dialogue-based natural language conversation, as formulated by the Turing test, could be achieved through unsupervised language modeling?

我认为真正的图灵测试不仅仅是关于语言，而是关于推理。要真正通过图灵测试，我应该能教对方微积分，让他们真正理解微积分并能解决新的微积分问题。所以，要真正解决图灵测试，我们需要比语言模型目前展示的更多——我们需要某种方式引入推理。这与我们已有的方法有多大不同，这是一个开放问题。可能我们需要一系列全新的激进想法，也可能只需要对现有系统稍作调整。但就语言建模能走多远而言，它已经远远超出了许多人的预期。有很多有趣的切入点，比如 GPT-2 对物理世界的理解程度。你读到 GPT-2 关于水下火的描述，它似乎并不完全理解这些东西是什么。但与此同时，你也能看到像火焰冒烟这样的现象。GPT-2 没有身体，没有物理体验——它只是静态地读取数据。我认为答案是我们还不知道，但我们开始能够向物理系统、真实存在的系统提出这些问题，这非常令人兴奋。

I think the Turing test in its real form isn't just about language; it's about reasoning. To really pass the Turing test, I should be able to teach calculus to whoever's on the other side and have them truly understand calculus and solve new calculus problems. So to really solve the Turing test, we need more than what we're seeing with language models—we need some way of plugging in reasoning. How different that will be from what we already do is an open question. It might be that we need a sequence of totally radical new ideas, or it might be that we just need to shape our existing systems in a slightly different way. But in terms of how far language modeling will go, it's already gone way further than many people expected. There are interesting angles to poke at, like how much GPT-2 understands the physical world. You read a little about fire underwater in GPT-2, and it seems like maybe it doesn't quite understand what these things are. But at the same time, you see things like smoke coming from flame. GPT-2 has no body, no physical experience—it just statically reads data. I think the answer is we don't know yet, but we're starting to be able to ask these questions to physical systems, real systems that exist, and that's very exciting.

你的直觉是什么？你认为如果只是大幅扩展语言建模，推理能否从完全相同的机制中涌现出来？

What's your intuition? Do you think if you just scale language modeling significantly, reasoning can emerge from the same exact mechanisms?

我认为仅仅扩展 GPT-2 不太可能获得成熟的推理能力。类型签名有点不对。我们有一种叫做思考的过程，我们会花费可变的算力来得到更好的答案。我多想一点，就能得到更好的答案。这种类型签名并没有完全编码在 GPT-2 中。GPT-2 在进化过程中融入了所有信息，变得非常擅长预测，然后在运行时只需一次前向传播就能生成内容。可能有一些小的调整可以让类型签名正确。例如，它实际上不是一次前向传播——你是逐个符号生成的。所以也许你生成一整个思考序列，只保留最后一部分。但至少，我预计你必须做出这样的改变。

I think it's unlikely that if we just scale GPT-2, we'll have reasoning in a full-fledged way. The type signature is a bit wrong. There's something we do called thinking, where we spend a variable amount of compute to get better answers. I think a little harder, I get a better answer. That type signature isn't quite encoded in GPT-2. GPT-2 has evolutionarily baked in all this information, getting very good at prediction, and then at runtime it just does one forward pass and generates stuff. There might be small tweaks to get the type signature right. For example, it's not really one forward pass—you generate symbol by symbol. So maybe you generate a whole sequence of thoughts and only keep the last bit. But at the very least, I'd expect you have to make changes like that.

正是如此。思考是一个念头接一个念头生成的过程，就像你说的——保留最后一部分，我们收敛到的那个东西。还有另一个有趣的方面：分布外泛化。思考让我们以某种方式做到了这一点。我们有了一个经验，却还能不断精炼我们对它的心理模型。这感觉与推理的本质紧密相连。也许只是对我们现有方法的小调整，也许需要很多新想法，耗费数十年。

Exactly. Thinking is the process of generating thought by thought, in the same kind of way—you keep the last bit, the thing we converge towards. There's another interesting piece: out-of-distribution generalization. Thinking somehow lets us do that. We have an experience, and yet we somehow keep refining our mental model of it. That feels tied to whatever reasoning is. Maybe it's a small tweak to what we do, maybe it's many ideas and will take decades.

分布外泛化的假设是，有可能创造新想法。也有可能从来没有人创造过新想法，而将 GPT-2 扩展到 GPT-20 基本上就能泛化到人类可能拥有的所有思想。我来唱个反调：自莎士比亚以来，我们想出了多少新的故事创意？无非都是不同形式的爱情和戏剧。

The assumption in out-of-distribution generalization is that it's possible to create new ideas. It's possible that nobody is ever creating new ideas, and scaling GPT-2 to GPT-20 would essentially generalize to all possible thoughts humanity would have. To play devil's advocate: how many new story ideas have we come up with since Shakespeare? It's all different forms of love and drama.

你读过 Rich Sutton 最近的博客文章《苦涩的教训》吗？他基本上说，AI 研究中最大的教训是，利用算力的通用方法最终会胜出。你同意吗？总的来说，关于你正在探索的想法——无论是 GPT 建模还是 OpenAI Five 玩 Dota——通用方法比更精细调整的专家方法更好。

Have you read the recent blog post 'The Bitter Lesson' by Rich Sutton? He basically says that the biggest lesson from AI research is that general methods that leverage computation will ultimately win out. Do you agree? In general, about the ideas you're exploring—whether it's GPT modeling or OpenAI Five playing Dota—a general method is better than a more fine-tuned expert method.

我认为那篇博客文章的一个有趣反应是，很多人将其解读为算力才是一切，这是一个威胁性的想法。我也不认为这是真的。很明显，算法思想对进步非常重要。要真正构建 AGI，你需要在算力规模和人类智慧上都尽力推进。两者都需要。但你提问的方式非常好：关键在于我们应该追求什么样的想法。当然，如果你能找到可扩展的想法——投入更多算力，投入更多数据，它就会变得更好——那才是真正的圣杯。所以是的，我们就是这样想的。我们对深度学习构建 AGI 的潜力感到兴奋的部分原因是，我们看到最成功的 AI 系统，并意识到扩展它们会让它们工作得更好。这种可扩展性给了我们构建变革性系统的希望。

I think one interesting reaction to that blog post is that many people read it as saying compute is all that matters, which is a threatening idea. I don't think it's true either. It's clear that algorithmic ideas have been very important for progress. To really build AGI, you want to push as far as you can on computational scale and on human ingenuity. You need both. But the way you phrased the question is very good: it's about what kind of ideas we should strive for. Absolutely, if you can find a scalable idea—pour more compute into it, pour more data into it, it gets better—that's the real Holy Grail. So yes, that's how we think about it. Part of why we're excited about deep learning's potential for AGI is because we look at the most successful AI systems and realize that scaling them up will make them work better. That scalability gives us hope for building transformative systems.

我跟你说，这在一定程度上是一种情绪反应。人们常常觉得算力对最先进的性能如此重要。个体开发者——也许是一个坐在堪萨斯某处的 13 岁孩子——他们坐在那里……

I'll tell you, this is partially an emotional response. People often have the feeling that compute is so important for state-of-the-art performance. Individual developers—maybe a 13-year-old sitting somewhere in Kansas—they're sitting...

他们可能连 GPU 都没有，或者只有一块 1080 之类的显卡。然后就会觉得，如果规模这么重要，我怎么可能在这个 AI 世界里竞争或做出贡献呢？所以你能谈谈这个吗？总的来说，你认为我们未来也需要像普及算法一样，更多地普及算力资源吗？

They might not even have a GPU, or may have a single GPU, a 1080 or something like that. And there's this feeling like, 'How can I possibly compete or contribute to this world of AI if scale is so important?' So if you can comment on that, and in general, do you think we need to also in the future focus on democratizing compute resources more, or as much as we democratize the algorithms?

嗯，我的想法是，存在一个可能的进步空间。这个空间里有一些想法和系统是有效的，能推动我们前进。其中有一部分，在某种程度上越来越重要，确实需要大量的算力。对于那部分，我认为答案很明确，我们之所以有现在的结构，部分原因就是我们觉得推动规模扩张、建造这些大型集群和系统非常重要。但还有另一部分空间，不依赖于大规模算力——这些想法，我认为要让 AI 真正有影响力、真正发光，它们应该是那种如果规模扩大就会比小规模时效果好得多的想法。但你可以不用大量算力就发现它们。如果你看看最近的发展历史，想想 GAN 或 VAE 这些东西——我认为你不需要大量算力就能想出来，实际上人们也确实是在没有大量算力的情况下想出来的。

Well, the way that I think about it is that there's this space of possible progress. There's a space of ideas and systems that will work and move us forward. And there's a portion of that space, to some extent an increasingly significant portion, that does just require massive compute resources. For that, I think the answer is kind of clear, and part of why we have this structure that we do is because we think it's really important to be pushing the scale and building these large clusters and systems. But there's another portion of the space that isn't about large-scale compute—these ideas that, again, I think for the AI to be really impactful and really shine, they should be ideas that if you scaled them up would work way better than they do at small scale. But you can discover them without massive computational resources. If you look at the history of recent developments, think about things like GANs or VAEs—these are ones that I think you could come up with without having, and in practice people did come up with them without having massive computational resources.

我刚刚和 Ian Goodfellow 聊过，但问题是，最初的 GAN 产生的结果相当糟糕，对吧？只是因为他们足够聪明，知道这很惊人——能生成他们知道的任何东西。你觉得想象算力资源由政府拥有并作为公共事业提供，是不是太乐观、太理想化了？

I just talked to Ian Goodfellow, but the thing is, the initial GAN produced pretty terrible results, right? Only because they were smart enough to know that this is quite surprising—can generate anything that they know. And do you see a world that's too optimistic and dreamer-like to imagine that compute resources are something that's owned by governments and provided as a utility?

在某种程度上，这个问题让我想起我在哈佛的一位前教授 Matt Welsh 的博客文章，他是一位系统学教授。我记得参加他的终身教职演讲，他刚拿到终身教职，暑假去了谷歌，然后决定不回学术界了。在他的博客文章里，他提出一个观点：作为一个系统研究者，我想出这些很酷的系统想法，然后做一个小型概念验证。我能期望的最好结果就是谷歌或雅虎（当时还存在）的人能实现它，并真正让它大规模运行。那对我来说就是梦想——我造了个小东西，他们造了个真正能运行的大东西。他说，'我受够了，我想成为那个真正建造和部署的人。'我认为这里有一个类似的二分法。我认为有些人确实觉得有价值——而且我认为这是一件有价值的事——成为那个提出想法、构建概念验证的人。是的，你无法生成最酷的 GAN 图像，但你发明了它。所以这里有一个真正的权衡，我认为这是一个非常个人化的选择，但双方都有价值。

To some extent, this question reminds me of a blog post from one of my former professors at Harvard, this guy Matt Welsh, who was a systems professor. I remember sitting in his tenure talk, and he had literally just gotten tenure, went to Google for the summer, and then decided he wasn't going back to academia. In his blog post, he makes this point: look, as a systems researcher, I come up with these cool system ideas, and I kind of do a little proof of concept. The best thing I can hope for is that the people at Google or Yahoo (which was around at the time) will implement it and actually make it work at scale. That's like the dream for me—I built the little thing, and they build the big thing that's actually working. For him, he said, 'I'm done with that, I want to be the person who's actually doing this building and deploying.' I think there's a similar dichotomy here. I think there are people who really find value—and I think it is a valuable thing to do—to be the person who produces those ideas, who builds the proof of concept. And yeah, you don't get to generate the coolest possible GAN images, but you invent it again. So there's a real trade-off there, and I think that's a very personal choice, but I think there's value in both sides.

你认为创造 AGI 或一些新模型，即使在原型阶段也能看到其卓越性的回响吗？也就是说，你可以在没有规模的情况下发展这些想法——最初的种子。

Do you think creating AGI or some new models would see echoes of the brilliance even at the prototype level? So you would be able to develop those ideas without scale—the initial seeds.

我总是喜欢看现有的例子，看真正的先例。看看我们 2018 年 6 月发布的模型，我们把它扩展成了 GPT-2。你可以看到，在小规模下，它创下了一些记录。这是奉献版 GPT——我们确实有一些很酷的生成结果，虽然远没有 GPT-2 那么惊艳，但它很有前景，很有趣。所以我认为很多这样的想法确实在小规模下就显示出潜力。但这里有一个星号，一个非常大的星号，那就是有时我们会看到一些行为涌现出来，与小规模时看到的性质完全不同，原始算法的发明者看了会说，'我没想到它能做到这个。'这就是我们在 Dota 中看到的。PPO 是由我们这里的研究员 John Schulman 创建的。在 Dota 中，我们基本上只是大规模运行 PPO。为了让它工作，我们做了一些调整，但核心还是 PPO。我们能够实现这种长期规划，这些行为在时间尺度上真正展现出来，我们原以为这是不可能的。John 看了之后说，'我没想到它能做到这个。'这就是当规模比原来大三个数量级时会发生的事情。

I always like to look at examples that exist, look at real precedent. Take a look at the June 2018 model that we released, that we scaled up to turn into GPT-2. You can see that at small scale, it set some records. This was the devotional GPT—we actually had some cool generations that weren't nearly as amazing and stunning as the GPT-2 ones, but it was promising, it was interesting. So I think it is the case that a lot of these ideas do show promise at small scale. But there is an asterisk here, a very big asterisk, which is sometimes we see behaviors that emerge that are qualitatively different from anything we saw at small scale, and the original inventor of whatever algorithm looks at and says, 'I didn't think it could do that.' This is what we saw in Dota. PPO was created by John Schulman, a researcher here. With Dota, we basically just ran PPO at massive scale. There were some tweaks to make it work, but fundamentally it's PPO at the core. We were able to get this long-term planning, these behaviors to really play out on a time scale that we just thought was not possible. John looked at that and was like, 'I didn't think it could do that.' That's what happens when you're at three orders of magnitude more scale compared to that.

但它仍然有相同的味道，至少是预期中数十亿的回响。虽然我怀疑随着 GPT 规模越来越大，你可能会得到令人惊讶的东西。

But it still has the same flavors of, at least, echoes of the expected billions. Although I suspect with GPT scaled more and more, you might get surprising things.

是的，你说得对。有趣的是，很难看出一个想法在规模扩大后能走多远。这是一个开放问题。在 Dota 和 PPO 的那个点上，我们也有一个非常具体的例子。实际上，关于 Dota 有一件非常令人惊讶的事情，我认为人们没有太多关注，那就是出现的分布外泛化程度。这个 AI 在其整个存在过程中都是与其他机器人对练的。

Yeah, you're right. It's interesting that it's difficult to see how far an idea will go when it's scaled. It's an open question. We've also at that point with Dota and PPO—here's a very concrete one. It's actually one thing that's very surprising about Dota that I think people don't really pay that much attention to is the degree of generalization out of distribution that happens. You have this AI that's trained against other bots for its entire existence.

你能讲讲 Dota 的故事吗？从开始到 OpenAI Five 以及那次通过的过程？自我对弈的过程是怎样的？训练量很大。

Can you talk through the story of Dota, the story leading up to OpenAI Five, and that pass? What was the process of self-play? It's a lot of training.

是的，Dota 是一个复杂的电子游戏。我们开始训练，尝试解决 Dota，因为我们觉得相对于国际象棋或围棋等其他游戏，这是迈向现实世界的一步。那些是非常离散的棋盘游戏，你只有一个棋盘和非常离散的移动。Dota 在时间上要连续得多，所以你有各种各样的动作，一场 45 分钟的游戏有这么多不同的单位，而且有很多混乱之处，之前的游戏都没有捕捉到。众所周知，所有硬编码的 Dota 机器人都很糟糕——根本不可能为它写出好的程序，因为它太复杂了。所以这似乎是一个推动强化学习最先进技术的好地方。我们首先专注于游戏的一对一版本，并成功解决了它。我们能够击败世界冠军，学习技能曲线是……

Yeah, with Dota, it's a complex video game. We started training, we started trying to solve Dota because we felt like this was a step towards the real world relative to other games like chess or Go. Those are very discrete board games where you just have a board and very discrete moves. Dota starts to be much more continuous in time, so you have this huge variety of different actions, a 45-minute game with all these different units, and it's got a lot of messiness to it that really hasn't been captured by previous games. Famously, all of the hard-coded bots for Dota were terrible—just impossible to write anything good for it because it's so complex. So this seemed like a really good place to push the state of the art in reinforcement learning. We started by focusing on the one-versus-one version of the game and were able to solve that. We were able to beat the world champions, and the learning skill curve was...

这种疯狂的指数增长，对吧？就像我们一直在扩大规模，修复漏洞，你看那条技能曲线，真的非常非常平滑。看到人类迭代循环如何产生稳定的指数级进步，这真的很有趣。顺便提一句：首先，这是一款非常受欢迎的视频游戏。结果是，有很多人类专家精通这款游戏，所以我们试图达到的基准非常高。另外：你能谈谈最初以及整个训练过程中，训练这些智能体玩这个游戏所使用的方法吗？

This crazy exponential, right? It was like constantly we were just scaling up, that we were fixing bugs, and you know, you look at the skill curve and it was really very, very smooth. It's actually really interesting to see how that human iteration loop yielded very steady exponential progress. And one side note: first of all, it's an exceptionally popular video game. This effect is that there's a lot of incredible human experts at that video game, so the benchmark we're trying to reach is very high. And the other: can you talk about the approach that was used initially and throughout training these agents to play this game?

是的，我们使用的方法是自我对弈。所以你有两个智能体，它们什么都不知道，它们互相战斗，发现一些好的东西，然后现在它们都知道了，它们就不断变得更好，没有上限。这是一个非常强大的想法，对吧？然后我们从游戏的一对一版本扩展到四对五，对吧？所以你可以想象像篮球这样的团队运动，需要很多协调，我们能够将同样的想法，同样的自我对弈，推广到完整的五对五版本，达到专业水平。我认为这里非常有趣的是，这些智能体在某种程度上几乎像昆虫一样的智能，对吧？它们与昆虫的训练方式有很多共同点。昆虫在环境中生活很长时间，或者昆虫的祖先已经存在很长时间，积累了大量经验，这些经验被编码到智能体中。它不像人类那样聪明，对吧？它不能去学微积分，但它能非常好地导航环境，并很好地处理环境中从未见过的意外情况。我们在 Dota 机器人中也看到了类似的情况，对吧？它们能够在这个游戏中与人类对战，而人类在其进化环境中从未存在过——人类与机器人的玩法完全不同——但它却能处理得非常好。这让我们非常惊讶，这是我们在较小规模的 PPO 中没有看到的现象。对吧？我们运行这些东西的规模，你知道，我可以用十万个 CPU 核心和数百个 GPU，大概每天相当于数百年的经验输入到机器人中。所以这个规模是巨大的，我们开始看到我们熟知和喜爱的算法产生出非常不同的行为。

Yeah, and so the approach we used is self-play. And so you have two agents, they don't know anything, they battle each other, they discover something a little bit good, and now they both know it, and they just get better and better and better without bound. And that's a really powerful idea, right? That we then went from the one-versus-one version of the game and scaled up to four versus five, right? So you think about kind of like with basketball where you have this team sport, you know, I need to do all this coordination, and we were able to push the same idea, the same self-play, to really get to the professional level at the full five-versus-five version of the game. And the things I think are really interesting here is that these agents in some ways are almost like an insect-like intelligence, right? Where there's a lot in common with how an insect is trained. An insect kind of lives in this environment for a very long time, or the ancestors of this insect have been around for a long time and had a lot of experience, it gets baked into this agent. And it's not really smart in the sense of a human, right? It's not able to go and learn calculus, but it's able to navigate its environment extremely well and handle unexpected things in the environment that it's never seen before pretty well. And we see the same sort of thing with our Dota bots, right? They're able to, within this game, play against humans, which are something that never existed in its evolutionary environment—totally different playstyles from humans versus the bots—and yet it's able to handle it extremely well. And that's something I think was very surprising to us, was something that doesn't really emerge from what we've seen with PPO at smaller scale. Right? The kind of scale we're running the stuff at, you know, I could take a hundred thousand CPU cores running with like hundreds of GPUs, it's probably about, you know, something like hundreds of years of experience going into this bot every single real day. And so that scale is massive, and we start to see very different kinds of behaviors out of the algorithms that we all know and love.

Dora 提到在一对一中击败了世界专家，但今年你们没能赢得五对五的比赛，对吧？面对世界最强。那么逆袭的故事是怎样的？首先，谈谈那个特别激动人心的事件，然后接下来的几个月和今年是什么情况？

Dora mentioned beat the world expert 1v1, and then you weren't able to win 5v5 this year, yeah? At the best in the world. So what's the comeback story? What's first of all, talk through that exceptionally exciting event, and what's the following months and this year look like?

是的，是的。有趣的一点是，我们一直在输，因为——OpenAI 的 Dota 团队，我们总是让机器人对抗比我们系统更强的玩家，至少以前是这样，对吧？第一次公开失败是在国际邀请赛的舞台上，我们对阵一些世界顶级战队，最终输掉了两场比赛，但我们让他们费了很大劲，对吧？两场比赛都打了大约 30 分钟、25 分钟，比分交替上升。所以我认为这确实表明我们达到了职业水平。回顾那些比赛，我们认为运气可能偏向另一边，我们本可以赢下一些比赛。所以这实际上对我们来说非常鼓舞人心。有趣的是，国际邀请赛是在固定时间举行的，对吧？所以我们确切知道比赛日期，我们尽可能快地推进。两周后，我们有了一个机器人，它对阵 TI 时的机器人有 80%的胜率。所以进步的历程，你应该把它看作一个快照，而不是最终状态。事实上，我们很快会宣布决赛。我实际上认为我们会在本期播客发布之前宣布最终比赛。Cassell's 应该会，将对阵世界冠军。对我们来说，这其实不那么重要——我们对未来的看法是，这是项目的最后一个里程碑，最后一个竞争里程碑，对吧？我们所有这些目标并不是要在 Dota 中击败人类；我们的目标是推动强化学习的前沿，我们已经做到了，对吧？而且我们实际上从我们的系统中学到了很多。我认为有很多令人兴奋的下一步计划。所以，作为我们构建成果的最终展示，我们将进行这场比赛，但对我们来说，成功或失败并不在于运气是否站在我们这边。

Yeah, yeah. So one thing that's interesting is that we lose all the time because we—so the Dota team at OpenAI, we played the bot against better players than our system all the time, or at least we used to, right? Like, the first time we lost publicly was we went up on stage at The International and we played against some of the best teams in the world, and we ended up losing both games, but we gave them a run for their money, right? Both games were kind of 30 minutes, 25 minutes, and they went back and forth, back and forth, back and forth. And so I think that really shows that we're at the professional level. And looking at those games, we think that the coin could have gone a different direction and it could have had some wins. And so that was actually very encouraging for us. And it's interesting because The International was at a fixed time, right? So we knew exactly what day we were going to be playing, and we pushed as far as we could as fast as we could. Two weeks later, we had a bot that had an 80% win rate versus the one that played at TI. So the march of progress, you know, you should think of as a snapshot rather than as an end state. And so in fact, we'll be announcing our finals pretty soon. I actually think that we'll announce our final match prior to this podcast being released. Cassell's should be, will be playing against the world champions. And for us, it's really less about that—the way that we think about what's upcoming is the final milestone, the final competitive milestone for the project, right? That our goal in all of this isn't really about beating humans at Dota; our goal is to push the state of the art in reinforcement learning, and we've done that, right? And we've actually learned a lot from our system. And I think a lot of exciting next steps that we want to take. And so, kind of a final showcase of what we built, we're going to do this match, but for us it's not really the success or failure to see if we have the coin flip go in our direction or against.

你认为深度学习领域在未来几年会走向何方？你认为强化学习的工作可能会走向何方？更具体地说，在 OpenAI，你正在从事的所有激动人心的项目，2019 年对你来说意味着什么？

Where do you see the field of deep learning heading in the next few years? What do you see the work in reinforcement learning perhaps heading? And more specifically with OpenAI, all the exciting projects that you're working on, what is 2019 hold for you?

大规模扩展。规模。我会强调这一点，并说，我认为这是关于想法加规模。两者都需要。所以这确实是一个很好的观点。

Massive scale. Scale. I will put an emphasis on that and just say, you know, I think that it's about ideas plus scale. You need both. So that's a really good point.

所以关于想法的问题：你有很多项目在探索智能的不同领域。问题是，当你想到规模时，你是否考虑扩大这些单个项目的规模？你是否考虑增加新项目？如果你在考虑增加新项目，或者回顾过去，提出新项目和新想法的过程是怎样的？

So the question in terms of ideas: you have a lot of projects that are exploring different areas of intelligence. And the question is, when you think of scale, do you think about growing scale those individual projects? So do you think about adding new projects? And if you are thinking about adding new projects, or if you look at the past, what's the process of coming up with new projects and new ideas?

所以我们这里有一个项目的生命周期。我们从几个人开始，只是在一个小规模的想法上工作。语言实际上是一个很好的例子，有一个人在这里长期推动语言方面的工作。然后你看到生命的迹象，对吧？所以就像，比如说最初的 GPT，我们有了一些有趣的东西，然后我们说，好吧，是时候扩大规模了，对吧？是时候投入更多的人，更多的计算资源了。然后我们就不断推进，不断推进，最终状态就像 Dota 或机器人项目，有一个 10 到 15 人的大团队，以非常大的规模运行，能够进行实质性的工程。

So we really have a life cycle of projects here. So we start with a few people just working on a small-scale idea. And language is actually a very good example of this, that it was really one person here who was pushing on language for a long time. Then you get signs of life, right? And so this is like, let's say with the original GPT, we had something that was interesting, and we said okay, it's time to scale this, right? It's time to put more people on it, put more computational resources behind it. And then we just kind of keep pushing and keep pushing, and the end state is something that looks like Dota or robotics, where you have a large team of 10 or 15 people that are running things at very large scale, and you're able to really have material engineering.

机器学习科学汇聚在一起，制造出能够工作并取得实质性成果的系统，这些成果在以前是不可能实现的。所以我们经历了整个生命周期，我们已经做了很多次。你知道，通常端到端大概需要两年左右。我，你知道，这个组织已经成立三年了，所以也许我们会找到答案。我们也有更长期的项目，但你知道，我们会逐步推进。我们有一个团队，实际上我们刚刚开始，伊利亚和我正在启动一个名为推理团队的新团队，这真的是要尝试解决如何让神经网络进行推理的问题。我们认为这将是一个长期项目，我们对此非常兴奋。

Of machine learning science coming together to make systems that work and get material results that just would've been impossible otherwise. So we do that whole lifecycle, we've done it a number of times. You know, typically end to end it's probably two years or so to do it. I, you know, the organization's been around for three years, so maybe we'll find it. We also have longer lifecycle projects, but you know, we will work up to those. We have so, so one team that we were actually just starting, Ilya and I are kicking off a new team called the reasoning team, and this is to really try to tackle how do you get neural networks to reason. And we think that this will be a long-term project, and we're very excited about it.

在推理方面，这是一个非常令人兴奋的话题。伍迪，你设想了什么样的基准测试，什么样的推理测试？如果你坐下来喝点什么，你会对这个系统能够做到的事情印象深刻，那会是什么样子？

In terms of reasoning, super exciting topic. Woody, what kind of benchmarks, what kind of tests of reasoning do you envision? What would, if you sat back with whatever drink and you would be impressed that this system is able to do something, what would that look like?

不是恐惧，它们正在改进。所以某种逻辑，尤其是数学逻辑，我想是的。对，我认为还有其他一些问题与改进是双重的。特别是，你知道，想想编程，我甚至想到代码的安全分析，这些都属于同一类核心推理，并且能够进行一定程度的分布泛化。如果 OpenAI 推理团队能够证明 P 等于 NP，那将非常令人兴奋。那会非常好。我会非常非常非常兴奋，尤其是如果结果证明 P 等于 NP，那也会很有趣。只是，这会是讽刺和幽默的，你知道。

Not fear, improving they are improving. So some kind of logic and especially mathematical logic, I think so. Right, I think that there's kind of other problems that are dual to if you're improving. In particular, you know, you think about programming, I think about even like security analysis of code, that these all kind of capture the same sorts of core reasoning and being able to do some amount of distribution generalization. It would be quite exciting if OpenAI reasoning team was able to prove that P equals NP. That would be very nice. I'd be very, very, very exciting, especially if it turns out the P equals NP, that'll be interesting too. It just, it would be ironic and humorous, you know.

那么，你认为哪个问题是最令人兴奋、最具挑战性、对我们整个社区以及 OpenAI 今年的工作影响最大的？你提到了推理，我认为这是一个大问题。

So what problem stands out to you as the most exciting and challenging, impactful to the work for us as a community in general and for OpenAI this year? You mentioned reasoning, I think that's a heck of a problem.

是的，所以我认为推理是一个重要的问题。我认为在 2019 年很难取得好的结果。你知道，再次，就像我们考虑生命周期需要时间一样。我认为对于 2019 年，语言建模似乎正处于那个斜坡上。它已经到了我们拥有一种有效的技术的地步，我们想要将其扩展 100 倍、1000 倍，看看会发生什么。

Yeah, so I think reasoning is an important one. I think it's gonna be hard to get good results in 2019. You know, again, just like we think about the lifecycle takes time. I think for 2019, language modeling seems to be kind of on that ramp right. It's at the point that we have a technique that works, we want to scale 100X, thousand X, see what happens.

太棒了。你认为我们生活在模拟中吗？

Awesome. Do you think we're living in a simulation?

我认为很难对此有真正的看法。我，你知道，这实际上很有趣。我把那些我认为可以对世界产生实质性不同预测的事情与那些只是猜测起来有趣的事情区分开来。我有点把模拟看作更像是，火星和木星之间有没有一个飞行的茶壶？也许有，但很难知道这对我的生活意味着什么。

I think it's hard to have a real opinion about it. I, you know, it's actually interesting. I separate out things that I think can have, like, you know, yield materially different predictions about the world from ones that are just kind of, you know, fun to speculate about. And I kind of view simulation it's more like, is there a flying teapot between Mars and Jupiter? Like maybe, but it's a little bit hard to know what that would mean for my life.

所以有一些可行的事情。我认为 OpenAI 做得最好的一些工作是在强化学习领域，而强化学习的一些成功来自于能够模拟你试图解决的问题。那么你对强化学习的未来和模拟的未来有希望吗？就像我们正在讨论的，自动驾驶汽车或任何类型的系统，你看到它扩展，使我们能够模拟系统并增强，能够创建一个反映我们真实世界的模拟器，并一劳永逸地证明，尽管你否认，我们生活在模拟中？

So there is something actionable. I'd say some of the best work OpenAI has done is in the field of reinforcement learning, and some of the success of reinforcement learning comes from being able to simulate the problem you're trying to solve. So do you have a hope for the future of reinforcement learning and for the future of simulation? Like what we're talking about, autonomous vehicles or any kind of system, do you see that scaling so we'll be able to simulate systems and enhance, be able to create a simulator that echoes our real world and proving once and for all, even though you're denying it, that we're living in a simulation?

对，所以你知道，其核心是，我们能否使用模拟来开发自动驾驶汽车？看看我们的机器人系统 Dactyl，它实际上是在使用 DOTA 系统的模拟中训练的，并且它转移到了物理机器人上。我认为每个人看到我们的 DOTA 系统，都会想，这只是一个游戏，你怎么能逃到现实世界？答案是，嗯，我们用物理机器人做到了，那个贵族可以编程。所以我认为答案是，如果你应用正确的技术，模拟比你想象的要走得更远。现在有一个问题，你知道，那个模拟中的存在是否会醒来并拥有意识？我认为这个问题似乎更难推理。我认为，你知道，你真的应该思考，人类意识到底从何而来，我们自己的自我意识？而且，你知道，是不是一旦你有一个足够复杂的神经网络，你就必须担心智能体会感到痛苦？我认为那里有一些有趣的猜测，但你知道，再次，我认为很难确定。

Right, so you know, kind of the core thereof, like can we use simulation for self-driving cars? Take a look at our robotic system Dactyl, right, that was trained in simulation using the DOTA system in fact, and it transfers to a physical robot. And I think everyone looks at our DOTA system, they're like, it's just a game, how are you ever going to escape to the real world? And the answer is, well, we did it with the physical robot, the noble could program. And so I think the answer is simulation goes a lot further than you think if you apply the right techniques to it. Now there's a question of, you know, are the beings in that simulation gonna wake up and have consciousness? I think that one seems a lot, a lot harder to again reason about. I think that, you know, you really should think about like, where exactly does human consciousness come from and our own self-awareness? And, you know, is it just that like once you have like a complicated enough neural net, do you have to worry about the agents feeling pain? And I think there's like interesting speculation to do there, but, you know, again, I think it's a little bit hard to know for sure.

好吧，让我继续猜测。你认为要创造智能，通用智能，你需要意识和身体吗？你认为这些元素中的任何一个都是必要的，还是智能是与这些正交的东西？

Well, let me just keep with a speculation. Do you think to create intelligence, general intelligence, you need consciousness and a body? Do you think any of those elements are needed, or is intelligence something that's orthogonal to those?

我先坚持非宏大的答案，对吧。所以非宏大的答案就是看看，你知道，我们已经让什么工作了？你知道，GPT-2，很多人会说，即使要得到这样的结果，你也需要真实世界的经验，你需要一个身体，你需要基础。你怎么能推理这些事？如果你从未经历过烟和火这些东西，你怎么能知道它们？而 GPT-2 表明，你可以比那种推理预测的走得更远。所以我认为，就意识而言，我们需要身体吗？答案似乎是否定的，对吧，我们可能可以继续推动我们现有的系统。它们已经感觉通用，它们不像 AGI 那样有能力、通用或学习得快，但你知道，它们至少在某种程度上是原始 AGI，而且它们不需要任何这些东西。现在，让我们转向宏大的答案，那就是，你知道，如果我们的神经网络已经有意识，我们会知道吗？我们怎么知道，对吧？是的，这就是猜测开始变得至少有趣或好玩，也许有点令人不安的地方，取决于你如何理解。但似乎当我们考虑动物时，存在某种意识连续体。你知道，我的猫，我认为它在某种程度上是有意识的，对吧？我，你知道，不像人类那样有意识。你可以想象你可以建造一个小意识计，对吧？你指向一只猫，它会给你一个小读数。我们指向一个人，它会给你一个大得多的读数。如果你把那个指向一个 DOTA 神经网络会怎么样？如果你训练这个巨大的模拟，神经网络会感到痛苦吗？你知道，很难知道答案是否定的，而且很难真正思考如果答案是肯定的那意味着什么。而且很有可能，你知道，例如，你可以想象，也许人类有意识的原因是因为它是一个方便的计算捷径。好吧，如果你想想，如果你有一个存在想要...

I'll stick to the kind of like the non-grand answer first, right. So the non-grand answer is just to look at, you know, what are we already making work? You know, GPT-2, a lot of people would have said that even get these kinds of results you need real-world experience, you need a body, you need grounding. How are you supposed to reason about any of these things? How are you supposed to like even kind of know about smoke and fire and those things if you've never experienced them? And GPT-2 shows it you can actually go way further than that kind of reasoning would predict. So I think that, in terms of doing any consciousness, do we need a body? It seems the answer is probably not, right, that we can probably just continue to push kind of the systems we have. They already feel general, they're not as competent or as general or able to learn as quickly as an AGI would, but you know, they're at least like kind of proto-AGI in some way, and they don't need any of those things. Now, let's move to the grand answer, which is, you know, if our neural nets are conscious already, would we ever know? How can we tell, right? Yeah, here's where the speculation starts to become, you know, at least interesting or fun, and maybe a little bit disturbing, depending on where you take it. But it certainly seems that when we think about animals, that there's some continuum of consciousness. You know, my cat, I think is conscious in some way, right? I, you know, not as conscious as a human. And you could imagine that you could build a little consciousness meter, right? You pointed at a cat, it gives you a little reading. We ran a human, it gives you much bigger reading. What would happen if you pointed one of those at a DOTA neural net? And if your training of this massive simulation, do the neural nets feel pain? You know, it becomes pretty hard to know that the answer is no, and it becomes pretty hard to really think about what that would mean if the answer were yes. And it's very possible, you know, for example, you could imagine that maybe the reason these humans have consciousness is because it's a convenient computational shortcut. All right, if you think about it, if you have a being that wants to...

避免痛苦——这似乎对在这种环境中生存非常重要。而一旦你吃东西，也许最好的方式就是拥有一个有意识的个体，对吧？为了在环境中成功，你需要具备这些属性。那么你该如何实现它们呢？也许意识就是实现这一点的方式。如果真是这样，那么我们或许应该期待真正有能力的强化学习智能体也会拥有意识。但你知道，这是一个很大的假设，而且我认为还有很多其他方向的论点可以提出。

Avoid pain, which seems pretty important to survive in this environment. And once you eat food, that may be the best way of doing it is to have a being that's conscious, right? That in order to succeed in the environment, you need to have those properties. And how are you supposed to implement them? And maybe this consciousness is a way of doing that. If that's true, then actually maybe we should expect that really competent reinforcement learning agents will also have consciousness. But you know, it's a big if, and I think there are a lot of other arguments they can make in other directions.

我认为这是一个非常有趣的想法，即使是 GPT-2 也拥有一定程度的意识。这其实并不是一个疯狂的想法。在我们思考创造狗的智能、猫的智能和人类的智能意味着什么时，思考这个问题是很有用的。

I think that's a really interesting idea, that even GPT-2 has some degree of consciousness. That's something that's actually not as crazy to think about. It's useful to think about as we think about what it means to create intelligence of a dog, intelligence of a cat, and the intelligence of a human.

那么最后一个问题：你认为我们会不会像电影《她》里那样，与人工智能系统坠入爱河，或者人工智能系统与人类相爱？

So last question: do you think we will ever fall in love, like in the movie Her, with an artificial intelligence system, or an artificial intelligence system falling in love with a human?

我希望如此。如果有什么更好的方式结束，那就是爱。那么 Greg，非常感谢你今天的分享。

I hope so. If there's any better way to end it, on love. So Greg, thanks so much for talking today.

谢谢你邀请我。

Thank you for having me.